Unable to create GPU_IVF_PQ index

[aping]

so I deploy a milvus cluster 2.4 to awk eks, and was hoping to try out gpu indices. however when I try to create gpu index on a tiny collection I got something like

[2024/06/04 12:37:05.472 +00:00] [INFO] [indexnode/task.go:516] ["index params are ready"] [buildID=450232097587185780] ["index params"="{\"cache_dataset_on_device\":\"false\",\"dim\":\"768\",\"index_type\":\"GPU_IVF_PQ\",\"m\":\"16\",\"metric_type\":\"IP\",\"nbits\":\"8\",\"nlist\":\"2048\"}"]
E20240604 12:37:05.594095    45 index_factory.cc:49] [KNOWHERE][Create][milvus] failed to find index GPU_IVF_PQ_fp32 in factory
[2024/06/04 12:37:05.595 +00:00] [WARN] [indexcgowrapper/helper.go:71] ["failed to create index, C Runtime Exception: index not supported\n"]

• milvus version is milvusdb/milvus:v2.4.3
• deployed to an awk eks cluster
• I don't see any gpu specific config except for

gpu:
  initMemSize: 0
  maxMemSize: 0

• the gpu nodes are g5.2xlarge, driver version 550.54.15

nvidia-smi
Wed Jun  5 08:11:43 2024       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
|  0%   20C    P8             12W /  300W |       0MiB /  23028MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

so wondering what am I missing?

[aping]

this is index node's log: indexnode.log

[Presburger]

the version not have gpu.
example if the gpu milvus started.

Version:   v2.4.5-dev-gpu

[aping]

hum looks like I need the milvusdb/milvus:v2.4.3-gpu image? I manually edited the index node and query node deploy to milvusdb/milvus:v2.4.3-gpu, should I update the helm chart like

image:
  all:
    repository: milvusdb/milvus
    tag: v2.4.4-gpu

I mean I think this will apply to all deploys, including stuff like proxy and coordinates, which runs on cpu nodes, is that ok?

[yhmo]

To deploy GPU Milvus: https://milvus.io/docs/install_cluster-helm-gpu.md

To use GPU index:

1. Milvus must be GPU edition
2. GPU index https://milvus.io/docs/index-with-gpu.md

[RakeshRaj97]

I followed this documentation and its not working.

I don't see any docker tags with -gpu when I inspect the values using the command helm show values milvus-helm/milvus

Please update the documentation.

[BabaJaguska]

I have the same issue. Having followed the above instructions, the node lists the gpu in capacity section, and nvidia-smi works, but Milvus says GPU indexes are unsupported.

[xiaofan-luan]

you have to use https://milvus.io/docs/prerequisite-gpu.md image instead of of using normal cpu image

[BabaJaguska]

If you'd please update the instructions to include this info, I'm sure it would save hundreds of cumulative people-hours. It still took me some time after I saw your response to understand where I should specify the image, and what should I write instead.
Thanks though, I think I would have given up if it wasn't for this thread.

[Presburger]

@BabaJaguska I'm very sorry for the inconvenience. We will be updating and optimizing the GPU-related documentation soon.