How many collections

[ava57r]

Hello

I have huge vectors to insert, but the index is being rebuilt all the time.
I think if I split 1 collection and I can get more than 100 collections
How many collections can qdrant process on 1 node?

Thanks

[generall]

As much as you have free file descriptors. Not recommend to create small collections, though

[rojas-diego]

I'm interested in this reply, why is the number of collections tied to the number of open fds? We have a use-case where we don't know whether we should distribute our vectors across multiple collections or store everything in a single collection and use multi-tenancy. The number of tenants will be around 100k and the number of vectors per tenant would be around 10k with embedding size 1024. What is the recommended route?

[timvisee]

or store everything in a single collection and use multi-tenancy

If your number of collections will be greater than 10, it's very likely that you'd want to store all of it in just one of them.

Please see our documentation here: https://qdrant.tech/documentation/guides/multiple-partitions/

The more collections you have, the more file descriptors you'd be using. But we have no exact numbers on that.

[rojas-diego]

Thank you for your answer @timvisee. I took a good look at the docs and we will likely go the multi-tenancy route based on your suggestion. Still, I'm curious to know why payload-based partitioning is preferred over collection-based partitioning. If there are any technical details you could share which explain this decision I'd really appreciate it!

[generall]

each collection is an independent storage with index, configs, e.t.c All of than requires additional memory

[ava57r]

Does a new collection add RAM consumption?