Manually setting doc_id on Documents fails results in error with IngestionPipeline using DynamoDB docstore and LanceDB vectorstore

[hragbalian]

Hello,

I'm trying to build an ingestion pipeline on documents retrieved from an API where I'm creating the Document object by manually setting the doc_id like so:

Document(
   text=json.dumps(doc),
   doc_id=meta_id,
   metadata=meta,
)

these are going into a list of documents, let's just call it documents, and doc is a JSON object and meta_id is a hash of the metadata for that document. I want only one version of the docs to persist in my index for every unique meta_id.

I'm using an IngestionPipeline with a DynamoDB docstore and a LanceDB vector store, which I've set up like this:

            vector_store = LanceDBVectorStore(
                    uri=<the uri>,
                    table_name=<the table name>)
                
            pipeline = IngestionPipeline(
                transformations=[
                    OpenAIEmbedding()
                ],
                docstore=DynamoDBDocumentStore.from_table_name(
                        table_name=<dynamo db table name>
                    ),
                vector_store=vector_store,
                )

            pipeline.run(
                documents=documents,
                show_progress=True
                )

This pipeline does a great job of not re-embedding the same documents twice when there are no changes detected in the document on a give meta_id, but when there is a change detected in the document (through the document_hash) I'm getting the following error.

OSError: LanceError(IO): Received literal Utf8("0024abc1ff76dded6f223a5cdbcbdb0eca1aaf5181de1598c8d5326e81d199ef") and could not convert to literal of type 'Null', /Users/runner/work/lance/lance/rust/lance/src/datafusion/logical_expr.rs:38:27

where in this example "0024abc1ff76dded6f223a5cdbcbdb0eca1aaf5181de1598c8d5326e81d199ef" is the meta_id with the now outdated document that needs to be reindexed.

Any guidance on this?

Thanks!

[dosubot[bot]]

The OSError: LanceError(IO) you're encountering when trying to reindex an updated document suggests an issue related to LanceDB's handling of document updates, particularly when the document's meta_id changes. This error might be due to a mismatch in expected data types or issues with how LanceDB processes updates for existing documents. Here are some targeted steps to address and potentially resolve the issue:

1. Verify LanceDB and Environment Setup: Ensure LanceDB is correctly installed and that your environment is correctly configured to connect to LanceDB. This includes checking the URI and accessibility of the LanceDB service.

2. Document ID Handling: When updating a document, ensure that the process for deleting the old document and adding the updated one is correctly implemented. This might involve correctly using the delete and add methods with the appropriate ref_doc_id to manage document versions in LanceDB.

3. Debugging and Logging: Increase the logging level to capture more detailed information about the operations leading to the error. This can help identify if the issue is with the connection to LanceDB, query execution, or another operation.

4. Consult LanceDB Documentation: Given that the error is specific to LanceDB, consulting its documentation or support channels may provide insights into the error and how to resolve it.

The IngestionPipeline is designed to handle document updates by checking for duplicates and updating the document store with new content if a document already exists. This functionality is demonstrated in the test_pipeline_update function, which shows that the pipeline supports reindexing by updating existing documents with their new versions.

If these steps do not help, consider creating a minimal example that reproduces the error and testing it in a different environment to rule out any environment-specific issues.

Sources

• llama-index-legacy/llama_index/legacy/vector_stores/lancedb.py
• llama-index-core/llama_index/core/ingestion/pipeline.py
• llama-index-core/tests/ingestion/test_pipeline.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[hragbalian]

If I set the docstore_strategy to DUPLICATES_ONLY, I can avoid this error message, but I'd like to be able to do UPSERTS_AND_DELETE. Looking through the files linked above, I think there's an issue with how the LanceDBVectorStore module is storing the ids and also with deleting duplicates that it finds. The doc_id field in my LanceDB table only has None values and the delete method in the LanceDBVectorStore has a hardcoded field it's looking for, document_id, which doesn't exist in my LanceDB table.