You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
yuno is a JSON document store with fulltext search. It's meant for embedding in electron apps, focuses solely on text search, and in most cases should handle millions of documents easily.
yuno is pretty basic - it has three components:
The document store, which is just the raw JSON objects stored in leveldb
The inverted search index, powered by search-index
A customisable natural language processing pipeline that is applied to documents before adding them to the index, greatly improving speed and memory usage compared to the vanilla search-index.
None of this is revolutionary - actually it's standard in fulltext-search database engines. And all the pieces exist already in the node ecosystem. But I couldn't find a node fulltext search and document store that could handle millions of documents, persisted on disk, didn't have crazy memory requirements and could be easily bundled into an electron app.
Like, db, yuno exist already??
Install
npm install --save yunodb
## Use
Create / load a database
yuno(options, callback)
e.g.
{
if (err) throw err
// do stuff with the db
db = dbhandle
})">var yuno = require('yunodb')
var dbopts = { location: './.yuno', keyField: 'id', indexMap: ['text'] } var db = yuno(dbopts, (err, dbhandle) => { if (err) throw err
// do stuff with the db db = dbhandle })
opts configures the two persistent datastores. Possible key-value pairs are:
location (String, required) - Base directory in which both datastores will be kept.
keyField (String, required) - JSONpath specifying the field in each document to be used as a key in the document store.
indexMap (Array | Object, required) - JSONpaths specifying the fields in each document to index for fulltext searching. See index mapping below for details.
deletable (Boolean, optional) - Whether documents should be deletable. Setting to true increases index size. Default: false.
ngramLength (Integer | Array, optional) - ngram length(s) to use when building index.
Index mapping
It is quite rare that all fields in a database should be exposed to the user search. More often, we want to allow the user to search certain fields, but retrieve the full document for each result. The indexMap option allows you to specify how to index documents.
There are two ways to tell yuno how to index:
1. Pass an Array of fields
The simple option - an array of fields to index. The contents of each field will be passed through the default Natural Language Processing pipeline before being added to the search index.
2. Pass an Object mapping fields to processors
To fine-tune the processing on a per-field basis, pass an Object where each key is a field to index. Values can be one of:
true/false whether to apply the default NLP pipeline
function a custom processing function.
Custom processing take the field value as a single argument, and their return value (either a string or an array) will be tokenised and added to the index.
Add documents
db.add(documents, options, callback)
documents, array of JSON-able objects to store
options optional, can override the database-wide indexMap option
callback, function to call on completion, with a single argument to be passed an error if there was one
yuno is being built to serve my use-case of embedding pre-made databases in electron apps. If you have another use-case and would like features added, please open an issue to discuss it - I'm happy to add things that will be widely useful.
Contributions are very welcome. Please open an issue to discuss any changes you would like to PR, or mention in an existing issue that you plan to work on it.
Ideas for improving performance are particularly welcome.