qdrant indexing

[maaly77]

Hi,
I am trying to index 1.5 M sentences with their vectors from a mistral model as well as normalizing the embeddings. I am indexing it on my server using your docker image. The indexing is taking so long 7 days now and I am not sure if it should take that long or am I doing something wrong.
Thanks

[generall]

Do you mean that mistral embeddings + vector indexing together is taking more than 7 days?

[maaly77]

yes, the indexing of 1.5M sentences vectorised by mistral embedding model is taking more than 7 days. I used https://python.langchain.com/docs/integrations/vectorstores/qdrant#on-premise-server-deployment

[generall]

do you have an estimation of which fraction of time takes vectorization and which indexing?

[maaly77]

it seems that I am running out of memory while indexing. I am using this langchain function to index 5 million sentences with their metadata. It worked with 10 sentences but once I started indexing the 5 million it takes really long time (2 weeks) and the script keep running while encountering memory issues. My server is 128 GB memory. I am currently using this script in a loop where texts are fraction of the 5 million, not sure if it will work?

qdrant = Qdrant.from_documents(
texts,
embeddings,
url=url,
api_key=None,
prefer_grpc=True,
collection_name="mycollection",
#force_recreate=True,
)

[timvisee]

You may want to disable indexing while doing bulk upserts, which could significantly increase overall performance and indexing time. Please see our documentation here: https://qdrant.tech/documentation/tutorials/bulk-upload/

You likely want to upload to disk directly too, which is also described on that page.

[maaly77]

I am using langchain to create and index vectors https://qdrant.tech/documentation/frameworks/langchain/ using mistral embeddings model (On-premise server deployment). So are you implying that this approach can't be used with large datasets? and I will need to create vectors first then upload then index ....??? There is no fast way to do that with langchain framework?

[maaly77]

I am not sure how to know the indexing time. I managed to vectorize those sentences in chromadb and they took half day and the total size was 6 GB. Here are screenshots of the structure I had for qdrant collection. Not sure where to look to answer your question