Ubuntu 23.10, LocalAI Not Detecting Nvidia

[pritchey]

Operating system: Ubuntu 23.10

NVidia/Cuda:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Fri_Jan__6_16:45:21_PST_2023
Cuda compilation tools, release 12.0, V12.0.140
Build cuda_12.0.r12.0/compiler.32267302_0

NVidia Card:

lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

Using the command provided in the documentation for getting LocalAI running with cublas:

docker run --rm -ti --gpus all -p 8080:8080 -e DEBUG=true -e MODELS_PATH=/models -e THREADS=1 -v /home/pritchey/models:/models quay.io/go-skynet/local-ai:v1.40.0-cublas-cuda12
Unable to find image 'quay.io/go-skynet/local-ai:v1.40.0-cublas-cuda12' locally
v1.40.0-cublas-cuda12: Pulling from go-skynet/local-ai
........
<removed for brevity>
Status: Downloaded newer image for quay.io/go-skynet/local-ai:v1.40.0-cublas-cuda12
@@@@@
Skipping rebuild
@@@@@
If you are experiencing issues with the pre-compiled builds, try setting REBUILD=true
If you are still experiencing issues with the build, try setting CMAKE_ARGS and disable the instructions set as needed:
CMAKE_ARGS="-DLLAMA_F16C=OFF -DLLAMA_AVX512=OFF -DLLAMA_AVX2=OFF -DLLAMA_FMA=OFF"
see the documentation at: https://localai.io/basics/build/index.html
Note: See also https://github.com/go-skynet/LocalAI/issues/288
@@@@@
CPU info:
model name	: Intel(R) Core(TM) i7-8700K CPU @ 3.70GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
CPU:    AVX    found OK
CPU:    AVX2   found OK
CPU: no AVX512 found
@@@@@
3:58PM INF Starting LocalAI using 1 threads, with models path: /models
3:58PM INF LocalAI version: v1.40.0 (6ef7ea2635ae5371be4e6eef516c2ad4afd9f4a3)
3:58PM DBG Model: bert-embeddings (config: {PredictionOptions:{Model:bert-MiniLM-L6-v2q4_0.bin Language: N:0 TopP:0 TopK:0 Temperature:0 Maxtokens:0 Echo:false Batch:0 F16:false IgnoreEOS:false RepeatPenalty:0 Keep:0 MirostatETA:0 MirostatTAU:0 Mirostat:0 FrequencyPenalty:0 TFZ:0 TypicalP:0 Seed:0 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:bert-embeddings F16:false Threads:0 Debug:false Roles:map[] Embeddings:true Backend:bert-embeddings TemplateConfig:{Chat: ChatMessage: Completion: Edit: Functions:} PromptStrings:[] InputStrings:[] InputToken:[] functionCallString: functionCallNameString: FunctionsConfig:{DisableNoAction:false NoActionFunctionName: NoActionDescriptionName:} FeatureFlag:map[] LLMConfig:{SystemPrompt: TensorSplit: MainGPU: RMSNormEps:0 NGQA:0 PromptCachePath: PromptCacheAll:false PromptCacheRO:false MirostatETA:0 MirostatTAU:0 Mirostat:0 NGPULayers:0 MMap:false MMlock:false LowVRAM:false Grammar: StopWords:[] Cutstrings:[] TrimSpace:[] ContextSize:0 NUMA:false LoraAdapter: LoraBase: NoMulMatQ:false DraftModel: NDraft:0 Quantization:} AutoGPTQ:{ModelBaseName: Device: Triton:false UseFastTokenizer:false} Diffusers:{PipelineType: SchedulerType: CUDA:false EnableParameters: CFGScale:0 IMG2IMG:false ClipSkip:0 ClipModel: ClipSubFolder:} Step:0 GRPC:{Attempts:0 AttemptsSleepTime:0} VallE:{AudioPath:}})
3:58PM DBG Extracting backend assets files to /tmp/localai/backend_data

 ┌───────────────────────────────────────────────────┐
 │                   Fiber v2.50.0                   │
 │               http://127.0.0.1:8080               │
 │       (bound on host 0.0.0.0 and port 8080)       │
 │                                                   │
 │ Handlers ............ 74  Processes ........... 1 │
 │ Prefork ....... Disabled  PID ................ 14 │
 └───────────────────────────────────────────────────┘

It is not picking up the GPU and leveraging that for acceleration. According to the documentation I should be seeing a line similar to the following and it's not there:

ggml_init_cublas: found 1 CUDA devices:
  Device 0: Tesla T4

What is missing/did I do wrong? Thank you! (Side note: I've previously had this running under MacOS using Metal without any issues. This is a first attempt under Linux and Nvidia.)

[pritchey]

Just to note, the supplied command for testing does not work:
docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi

Based on googling, the listed image has apparently been removed. using the following variation works:
docker run --rm --gpus all nvidia/cuda:11.0.3-base-ubuntu20.04 nvidia-smi

And provides the following output:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.146.02             Driver Version: 535.146.02   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 2080 Ti     Off | 00000000:01:00.0 Off |                  N/A |
| 24%   23C    P8              19W / 260W |      5MiB / 11264MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
+---------------------------------------------------------------------------------------+

Therefore I think everything that needs to be in installed is, and LocalAI should be able to pick up the GPU.....

[pritchey]

I think it's working.

Continuing to dig, I realized the image being pulled in the original command was an old one. I updated it to:
docker run --rm -ti --gpus all -p 8080:8080 -e DEBUG=true -e MODELS_PATH=/models -e THREADS=1 -v $PWD/models:/models quay.io/go-skynet/local-ai:v2.5.1-cublas-cuda12
Not seeing the expected output indicating GPU was seen/would be used on startup, I threw a curl command at it to see if it would show up WHEN it actually was tasked to do something, this showed up in the log:

7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr ggml_init_cublas: found 1 CUDA devices:
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr   Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
<redacted for brevity>
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr llm_load_tensors: ggml ctx size       =    0.11 MiB
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr llm_load_tensors: using CUDA for GPU acceleration
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr llm_load_tensors: system memory used  = 5272.45 MiB
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr llm_load_tensors: offloading 0 repeating layers to GPU
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr llm_load_tensors: offloaded 0/33 layers to GPU
7:22PM DBG GRPC(llama-2-7b-chat.Q6_K.gguf-127.0.0.1:45771): stderr ....................................................................................................

Based on that, I believe everything is set to go and it will use the GPU when whatever condition is met that requires it.

Summarizing:
a) I'm using a more updated image than the website docs show in the sample command.
b) Unlike the bit of text on the website, the use/recognition of a GPU being seen and available for use won't show until you actually use it.

[BadlyDrawnBoy]

I'm on Debian 12 with a RTX3070 and it took me a while to get cuda working in the docker-container. What helped was to set the --privileged and -v /dev:/dev option. I'm facing other issues now but I offload is now working.

$ docker run -ti -p 9090:8080 --gpus all --privileged -v /dev:/dev -v /home/martin/programme/ai/LocalAI_models/models:/build/models localai/localai:v2.8.2-cublas-cuda12-ffmpeg --debug --models-path /build/models --config-file /build/models/mixtral.yaml

5:10PM DBG GRPC(dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf-127.0.0.1:37909): stderr llm_load_tensors: offloading 6 repeating layers to GPU
5:10PM DBG GRPC(dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf-127.0.0.1:37909): stderr llm_load_tensors: offloaded 6/33 layers to GPU
5:10PM DBG GRPC(dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf-127.0.0.1:37909): stderr llm_load_tensors: CPU buffer size = 30735.50 MiB
5:10PM DBG GRPC(dolphin-2.7-mixtral-8x7b.Q5_K_M.gguf-127.0.0.1:37909): stderr llm_load_tensors: CUDA0 buffer size = 5727.56 MiB