Persistence of DocumentSummaryIndex artifacts beyond to filesystem

[hragbalian]

Hello,

I'm wondering if there's a reason that DocumentSummaryIndex doesn't have any persistence to a docstore or vector store and only has filesystem persistence enabled? Ideally I would have a similar docstrategy (e.g. duplicates only) apply to my document summaries and my document summary vectors as I have to my document vectors but it seems like that's not currently supported.

Any guidance would be greatly appreciated.

Thanks

[dosubot[bot]]

The DocumentSummaryIndex primarily focuses on in-memory operations and real-time retrieval, which is why it doesn't inherently support persistence to a docstore or vector store for document summaries and summary vectors. This design choice is made to optimize performance and efficiency for immediate use rather than long-term storage.

However, it does interact with both a docstore and a vector_store in specific ways:

• It utilizes the docstore to store generated document summaries.
• For retrieval based on embedding similarity, it employs the vector_store to store embeddings of the summaries.

To enable persistence of document summaries and summary vectors similar to document vectors, you could consider enhancing the DocumentSummaryIndex to manage the persistence of summaries and their embeddings explicitly in the docstore and vector_store. This would involve adding or modifying methods to ensure that summaries and embeddings are not only stored temporarily but are also persisted for future retrieval.

Additionally, implementing functionality for the efficient management (retrieval, update, deletion) of persisted summaries and their embeddings would be necessary. This requires a careful approach to maintain the relationships between documents, their summaries, and embeddings in the storage schema of both the docstore and vector_store.

The FusedSummaryRetriever class demonstrates an advanced approach by combining full document summaries with chunk summaries for improved retrieval efficiency, suggesting that while the DocumentSummaryIndex is optimized for real-time operations, it can be extended to support persistent storage by integrating more deeply with the docstore and vector_store.

For implementing these strategies, you might need to delve into the source code of the DocumentSummaryIndex and related classes, potentially adapting the methodology used by the FusedSummaryRetriever for managing and retrieving document elements for long-term data management.

Details

Sources

• Saving and loading a DocumentSummaryIndex using a vector store
• llama-index-legacy/llama_index/legacy/indices/document_summary/base.py
• llama-index-core/llama_index/core/indices/document_summary/base.py
• llama-index-legacy/llama_index/legacy/indices/document_summary/retrievers.py
• llama-index-core/llama_index/core/indices/document_summary/retrievers.py
• llama-index-packs/llama-index-packs-docugami-kg-rag/llama_index/packs/docugami_kg_rag/helpers/fused_summary_retriever.py
• llama-index-core/llama_index/core/storage/docstore/types.py
• llama-index-legacy/llama_index/legacy/storage/docstore/types.py
• llama-index-integrations/vector_stores/llama-index-vector-stores-docarray/llama_index/vector_stores/docarray/in_memory.py
• llama-index-core/llama_index/core/storage/docstore/simple_docstore.py
• llama-index-core/tests/indices/document_summary/test_index.py
• llama-index-legacy/tests/indices/document_summary/test_index.py
• llama-index-core/tests/indices/document_summary/test_retrievers.py
• llama-index-legacy/tests/indices/test_loading.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.