DocumentStores
You can think of the DocumentStore as a "database" that:
- stores your texts and meta data
- provides them to the retriever at query time
There are different DocumentStores in Haystack to fit different use cases and tech stacks.
Initialisation
Initialising a new DocumentStore within Haystack is straight forward.
Types
Input Format
DocumentStores expect Documents in dictionary form, like that below.
They are loaded using the DocumentStore.write_documents()
method.
See Preprocessing for more information on the cleaning and splitting steps that will help you maximize Haystack's performance.
from haystack.document_store import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore()dicts = [ { 'text': DOCUMENT_TEXT_HERE, 'meta': {'name': DOCUMENT_NAME, ...} }, ...]document_store.write_documents(dicts)
Writing Documents (Sparse Retrievers)
Haystack allows for you to write store documents in an optimised fashion so that query times can be kept low.
For sparse, keyword based retrievers such as BM25 and TF-IDF,
you simply have to call DocumentStore.write_documents()
.
The creation of the inverted index which optimises querying speed is handled automatically.
document_store.write_documents(dicts)
Writing Documents (Dense Retrievers)
For dense neural network based retrievers like Dense Passage Retrieval, or Embedding Retrieval, indexing involves computing the Document embeddings which will be compared against the Query embedding.
The storing of the text is handled by DocumentStore.write_documents()
and the computation of the
embeddings is started by DocumentStore.update_embeddings()
.
document_store.write_documents(dicts)document_store.update_embeddings(retriever)
This step is computationally intensive since it will engage the transformer based encoders. Having GPU acceleration will significantly speed this up.
Choosing the Right Document Store
The Document Stores have different characteristics. You should choose one depending on the maturity of your project, the use case and technical environment:
Our Recommendations
Restricted environment: Use the InMemoryDocumentStore
, if you are just giving Haystack a quick try on a small sample and are working in a restricted environment that complicates running Elasticsearch or other databases
Allrounder: Use the ElasticSearchDocumentStore
, if you want to evaluate the performance of different retrieval options (dense vs. sparse) and are aiming for a smooth transition from PoC to production
Vector Specialist: Use the MilvusDocumentStore
, if you want to focus on dense retrieval and possibly deal with larger datasets