Best practices for improving performance

[Dmitry-Evsyukov]

Hello, please help me which methods are most effective in improving the performance of a vector database.

Milvus - 2.3.7.

there is a collection of ~25 million records.

Entity {
id int64 
string_field  varchar(256)
vector - 128-dim float vector 
key int64
}

The table takes up about 27 GB of memory space

We' re calculating hash of key to select partition to read/write from software side (not partition key)

There are 128 partitions in total.

QPS:

1. Approximately 5-10 recording operations per second.
2. Expected number of read operations: 10k QPS

Milvus deployed in k8s (not standalone).

At the moment, it has been possible to achieve an indicator of 4k QPS of reading operations.

k8s config:

QueryNodes:

Replicas: 64
Processor: 2
Memory: 4 GB 

At the moment, the collection is loaded as 8/4 in-memory replicas. There are no changes in the performance here.

ProxyNodes: 

Replicas: 32
CPU: 1
Memory: 2GI

The rest of the architecture: RootCoord, QueryCoord, etc - under one replica

Avg CPU usage in QueryNodes like 50%
Avg CPU usage in ProxyNodes like 20-30%

May you please give any advice about tuning perfomance in this situation

I will be grateful for any answer. Thank you.

[yhmo]

Memory usage is 27GB.
Query nodes RAM = 64 * 4GB = 256GB, CPU cores = 64 * 2 = 128

Some suggestions:

1. In my opinion, no need deploy so many query nodes. 16 query nodes is better, assign 8 processes and 16GB to each node.
2. No need to deploy so many proxies, 8 proxies with 2 CPU for each is enough. Build a upper layer balancer to balance the requests to proxies.
3. Since the total RAM(256GB) of query nodes is much larger than the memory usage(27GB), you can use milvus replica to increase QPS. Milvus replica is: how many copies of the data in memory. Replica is assigned when you call the collection.load() method. Default replica number is 1, which means only 1 copy of the data in memory. To change the replica number:

collection = Collection("XXX")    
collection.release()  
collection.load(replica_number=4)

Change the replica_number to 4 and observe the QPS.

4. HNSW index is recommended for high-performance search.

[Dmitry-Evsyukov]

And I have another any important question. Is there an opportunity using batch search with some different filters (expressions like key in [1,2,3])

Client was wrote with golang-milvus SDK

[yhmo]

Each search request can only handle one filter expression. Different filters ought to be executed by different search requests.

[Dmitry-Evsyukov]

After configuring k8s quotes to query and proxy nodes QPS increased up to 6k

But now another problem with HNSW index. Could you help with parameters while creating and searching with index?

All my queries are just to find one the nearest vector in chosen partition/partitions. So I used flat index for max accuracy. Which M and ef do if prefer to maximize accuracy in search(topK always = 1)

Current values:
M = 32
efConstruction = 500

ef search = 1

[yhmo]

https://milvus.io/docs/index.md

• M defines tha maximum number of outgoing connections in the graph. Higher M leads to higher accuracy/run_time at fixed ef/efConstruction.
• efConstruction controls index search speed/build speed tradeoff. Increasing the efConstruction parameter may enhance index quality, but it also tends to lengthen the indexing time.
• ef controlling query time/accuracy trade-off. Higher ef leads to more accurate but slower search.

Decrease M value, search faster. When M decrease to a small value, the top1 result might be wrong. So, test and try different M values to find a trade-off between accuracy and performance.

[Dmitry-Evsyukov]

Thanks a lot for your responses! BTW read RPS increased up to 9.8k with new index. Now index testing:)

So I think we' re all here, thanks

[Dmitry-Evsyukov]

Hello! I need some advice to horizontal milvus scaling.

BTW I got 10k QPS with this setup. Also using eventually consistencyLevel.

And there is an interesting moment. Perfomance requirements has changed and we need 25k QPS.
After that (we have 10k QPS) multiply query nodes x3 (from 16 to 48) and multiply x3 in-memory replicas (from 4 to 12)

Expectations: 10k QPS -> 30k QPS
Real: Perfomance hasn't changed

Also increased resources for proxyNodes and queryCoords, but it like useless.
Could you explain, how to scale pods in that case please? Thank you

[yhmo]

The best way to improve QPS is to increase the number of load replicas and query nodes.
If it doesn't work, you need a monitoring system to observe the cluster state: https://milvus.io/docs/monitor.md

Observe these metrics:

[xiaofan-luan]

Hello! I need some advice to horizontal milvus scaling.

BTW I got 10k QPS with this setup. Also using eventually consistencyLevel.

And there is an interesting moment. Perfomance requirements has changed and we need 25k QPS. After that (we have 10k QPS) multiply query nodes x3 (from 16 to 48) and multiply x3 in-memory replicas (from 4 to 12)

Expectations: 10k QPS -> 30k QPS Real: Perfomance hasn't changed

Also increased resources for proxyNodes and queryCoords, but it like useless. Could you explain, how to scale pods in that case please? Thank you

can you share more details about

1. what is the cpu usage of all node?
2. is there a chance you can generate a pprof for further analysis?