Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Discoverability issue during CDI spec generation when the program lives inside a snap container #554

Open
gabrielmougard opened this issue Jun 20, 2024 · 1 comment
Open

Discoverability issue during CDI spec generation when the program lives inside a snap container #554

gabrielmougard opened this issue Jun 20, 2024 · 1 comment

Comments

@gabrielmougard
Copy link

I'm trying to generate a CDI specification in a program living inside a snap container (see canonical/lxd#13562)

When generating a CDI spec from inside a snap package, we need to tell the CDI discoverability process to start
looking after the host root file system which is defined inside the snap by /var/lib/snapd/hostfs (a.k.a, this is the same as / if we are not in a snap)

In consequence, here is how I created my cdilib object:

package cdi

import (
	"fmt"
	"os/exec"
	"path/filepath"

	"github.com/NVIDIA/nvidia-container-toolkit/pkg/nvcdi"
	"github.com/canonical/lxd/shared"
	cdpiSpec "tags.cncf.io/container-device-interface/pkg/cdi"
	"tags.cncf.io/container-device-interface/specs-go"
)

const (
	// defaultNvidiaTegraMountSpecPath is default location of CSV files that define the modifications required to the OCI spec
	defaultNvidiaTegraMountSpecPath = "/etc/nvidia-container-runtime/host-files-for-container.d"
)

// DefaultNvidiaTegraCSVFiles returns the default CSV files for the Nvidia Tegra platform.
func DefaultNvidiaTegraCSVFiles() []string {
	files := []string{
		"devices.csv",
		"drivers.csv",
		"l4t.csv",
	}

	var paths []string
	for _, file := range files {
		paths = append(paths, filepath.Join(defaultNvidiaTegraMountSpecPath, file))
	}

	return paths
}

// generateNvidiaSpec generates a CDI spec for an Nvidia vendor.
func generateNvidiaSpec(cdiID ID) (*specs.Spec, error) {
	mode := nvcdi.ModeAuto
	if cdiID.DeviceType() == IGPU {
		mode = nvcdi.ModeCSV
	}

	deviceNamer, err := nvcdi.NewDeviceNamer(nvcdi.DeviceNameStrategyIndex)
	if err != nil {
		return nil, fmt.Errorf("Failed to create device namer: %v", err)
	}

	deviceNamers := []nvcdi.DeviceNamer{deviceNamer}
	nvidiaCTKPath, err := exec.LookPath("nvidia-ctk")
	if err != nil {
		return nil, fmt.Errorf("Failed to find the nvidia-ctk binary: %v", err)
	}

	rootPath := ""
	if shared.InSnap() {
		rootPath = "/var/lib/snapd/hostfs"
	}

	cdilib, err := nvcdi.New(
		nvcdi.WithLogger(&CDILogger{}),
		nvcdi.WithDriverRoot(rootPath),
		nvcdi.WithDevRoot(rootPath),
		nvcdi.WithNVIDIACTKPath(nvidiaCTKPath),
		nvcdi.WithDeviceNamers(deviceNamers...),
		nvcdi.WithMode(mode),
		nvcdi.WithCSVFiles(DefaultNvidiaTegraCSVFiles()),
	)
	if err != nil {
		return nil, fmt.Errorf("Failed to create CDI library: %v", err)
	}

	deviceSpecs, err := cdilib.GetAllDeviceSpecs()
	if err != nil {
		return nil, fmt.Errorf("Failed to create device CDI specs: %v", err)
	}

	commonEdits, err := cdilib.GetCommonEdits()
	if err != nil {
		return nil, fmt.Errorf("Failed to create edits common for entities: %v", err)
	}

	return &specs.Spec{
		Version:        cdpiSpec.CurrentVersion,
		Kind:           fmt.Sprintf("%s/%s", cdiID.Vendor(), cdiID.Product()),
		Devices:        deviceSpecs,
		ContainerEdits: *commonEdits.ContainerEdits,
	}, nil
}

Here is the resulting log for the generation:

DEBUG  [2024-06-20T15:41:27Z] Is WSL-based system? false: could not load DXCore library: libdxcore.so: cannot open shared object file: No such file or directory 
DEBUG  [2024-06-20T15:41:27Z] Is NVML-based system? true: found NVML library 
INFO   [2024-06-20T15:41:27Z] Auto-detected mode as "nvml"                 
DEBUG  [2024-06-20T15:41:27Z] Is Tegra-based system? true: /sys/devices/soc0/family has 'tegra' prefix 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia0                        
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia0" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia0' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia0 as [/var/lib/snapd/hostfs/dev/nvidia0] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/dri/card1' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia0 as /dev/nvidia0 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/dri/card1 as /dev/dri/card1 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/card1                      
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/controlD65" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/card1" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/controlD65                 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/dri/card1 as [/var/lib/snapd/hostfs/dev/dri/card1] 
WARNING[2024-06-20T15:41:27Z] Could not locate /dev/dri/controlD65: pattern /dev/dri/controlD65 not found 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/dri/renderD128                 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/dri/renderD128" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/dri/renderD128' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/dri/renderD128 as [/var/lib/snapd/hostfs/dev/dri/renderD128] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/dri/renderD128 as /dev/dri/renderD128 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
WARNING[2024-06-20T15:41:27Z] Failed to evaluate symlink /var/lib/snapd/hostfs/dev/dri/by-path/pci-0005:09:00.0-card; ignoring 
WARNING[2024-06-20T15:41:27Z] Failed to evaluate symlink /var/lib/snapd/hostfs/dev/dri/by-path/pci-0005:09:00.0-render; ignoring 
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Locating "libcuda.so.*.*" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Using driver version 535.161.08              
DEBUG  [2024-06-20T15:41:27Z] Opening ld.conf at /var/lib/snapd/hostfs/etc/ld.so.cache 
DEBUG  [2024-06-20T15:41:27Z] Locating "*.so.535.161.08" in [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating "libcuda.so.535.161.08" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-modeset                 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-modeset as /dev/nvidia-modeset 
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-modeset" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-modeset' 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-modeset as [/var/lib/snapd/hostfs/dev/nvidia-modeset] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-uvm' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-uvm-tools               
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-uvm-tools" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-uvm-tools as /dev/nvidia-uvm-tools 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-uvm-tools as [/var/lib/snapd/hostfs/dev/nvidia-uvm-tools] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidia-uvm-tools' 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidia-uvm                     
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidia-uvm" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidiactl as /dev/nvidiactl 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidia-uvm as [/var/lib/snapd/hostfs/dev/nvidia-uvm] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/dev/nvidia-uvm as /dev/nvidia-uvm 
DEBUG  [2024-06-20T15:41:27Z] Locating /dev/nvidiactl                      
DEBUG  [2024-06-20T15:41:27Z] Locating "/dev/nvidiactl" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/dev] 
DEBUG  [2024-06-20T15:41:27Z] Located /dev/nvidiactl as [/var/lib/snapd/hostfs/dev/nvidiactl] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/dev/nvidiactl' 
DEBUG  [2024-06-20T15:41:27Z] Locating libnvidia-egl-gbm.so.*              
DEBUG  [2024-06-20T15:41:27Z] Locating "libnvidia-egl-gbm.so.*" in [/var/lib/snapd/hostfs /var/lib/snapd/hostfs/usr/lib64 /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib64 /var/lib/snapd/hostfs/lib/x86_64-linux-gnu /var/lib/snapd/hostfs/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/lib/x86_64-linux-gnu/nvidia/current /var/lib/snapd/hostfs/lib/aarch64-linux-gnu/nvidia/current] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1' 
DEBUG  [2024-06-20T15:41:27Z] Located libnvidia-egl-gbm.so.* as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0] 
DEBUG  [2024-06-20T15:41:27Z] Locating glvnd/egl_vendor.d/10_nvidia.json   
DEBUG  [2024-06-20T15:41:27Z] Locating "glvnd/egl_vendor.d/10_nvidia.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Located glvnd/egl_vendor.d/10_nvidia.json as [/var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json' 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/icd.d/nvidia_icd.json        
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/icd.d/nvidia_layers.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/icd.d/nvidia_icd.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/icd.d/nvidia_layers.json     
DEBUG  [2024-06-20T15:41:27Z] Located vulkan/icd.d/nvidia_icd.json as [/var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json as /usr/share/vulkan/icd.d/nvidia_icd.json 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/vulkan/icd.d/nvidia_icd.json' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json as /usr/share/glvnd/egl_vendor.d/10_nvidia.json 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0 as /usr/lib/aarch64-linux-gnu/libnvidia-egl-gbm.so.1.1.0 
DEBUG  [2024-06-20T15:41:27Z] Locating "vulkan/implicit_layer.d/nvidia_layers.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating vulkan/implicit_layer.d/nvidia_layers.json 
WARNING[2024-06-20T15:41:27Z] Could not locate vulkan/icd.d/nvidia_layers.json: pattern vulkan/icd.d/nvidia_layers.json not found 
DEBUG  [2024-06-20T15:41:27Z] Located vulkan/implicit_layer.d/nvidia_layers.json as [/var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json' 
DEBUG  [2024-06-20T15:41:27Z] Locating egl/egl_external_platform.d/15_nvidia_gbm.json 
DEBUG  [2024-06-20T15:41:27Z] Located egl/egl_external_platform.d/15_nvidia_gbm.json as [/var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json] 
DEBUG  [2024-06-20T15:41:27Z] Locating "egl/egl_external_platform.d/10_nvidia_wayland.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating "egl/egl_external_platform.d/15_nvidia_gbm.json" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/libglxserver_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json' 
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/libglxserver_nvidia.so.535.161.08: pattern nvidia/xorg/libglxserver_nvidia.so.535.161.08 not found 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json as /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json 
DEBUG  [2024-06-20T15:41:27Z] Locating egl/egl_external_platform.d/10_nvidia_wayland.json 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia/nvoptix.bin as [/var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin] 
WARNING[2024-06-20T15:41:27Z] Could not locate egl/egl_external_platform.d/10_nvidia_wayland.json: pattern egl/egl_external_platform.d/10_nvidia_wayland.json not found 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/nvoptix.bin                  
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/nvoptix.bin" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin' 
DEBUG  [2024-06-20T15:41:27Z] Locating "X11/xorg.conf.d/10-nvidia.conf" in [/var/lib/snapd/hostfs/etc /var/lib/snapd/hostfs/usr/local/share /var/lib/snapd/hostfs/usr/share] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/nvidia/nvoptix.bin as /usr/share/nvidia/nvoptix.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/nvidia_drv.so" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/nvidia_drv.so           
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/nvidia_drv.so: pattern nvidia/xorg/nvidia_drv.so not found 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/libglxserver_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating X11/xorg.conf.d/10-nvidia.conf      
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located X11/xorg.conf.d/10-nvidia.conf as [/var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/X11/xorg.conf.d/10-nvidia.conf as /usr/share/X11/xorg.conf.d/10-nvidia.conf 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libEGL_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/share/vulkan/implicit_layer.d/nvidia_layers.json as /usr/share/vulkan/implicit_layer.d/nvidia_layers.json 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvcuvid.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-allocator.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-cfg.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-eglcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-encode.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-fbc.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glsi.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ml.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-glvkspirv.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-opencl.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libcuda.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-nvvm.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ngx.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv1_CM_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/nvidia-fabricmanager/socket" in [/var/lib/snapd/hostfs/run /var/lib/snapd/hostfs/var/run] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-opticalflow.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-ptxjitcompiler.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08" in [/var/lib/snapd/hostfs] 
WARNING[2024-06-20T15:41:27Z] Could not locate /tmp/nvidia-mps: pattern /tmp/nvidia-mps not found 
DEBUG  [2024-06-20T15:41:27Z] Locating "/nvidia-persistenced/socket" in [/var/lib/snapd/hostfs/run /var/lib/snapd/hostfs/var/run] 
DEBUG  [2024-06-20T15:41:27Z] Locating /nvidia-persistenced/socket         
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08" in [/var/lib/snapd/hostfs] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/run/nvidia-persistenced/socket as /run/nvidia-persistenced/socket 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 
WARNING[2024-06-20T15:41:27Z] Could not locate /nvidia-fabricmanager/socket: pattern /nvidia-fabricmanager/socket not found 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvoptix.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-rtcore.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating /nvidia-fabricmanager/socket        
DEBUG  [2024-06-20T15:41:27Z] Locating "/tmp/nvidia-mps" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating /tmp/nvidia-mps                     
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/run/nvidia-persistenced/socket' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Locating "/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs] 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-tls.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Located /nvidia-persistenced/socket as [/var/lib/snapd/hostfs/run/nvidia-persistenced/socket] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libnvidia-vulkan-producer.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/535.161.08/gsp*.bin          
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/535.161.08/gsp*.bin" in [/var/lib/snapd/hostfs/lib/firmware/updates/5.15.0-1012-nvidia-tegra-igx /var/lib/snapd/hostfs/lib/firmware/updates /var/lib/snapd/hostfs/lib/firmware/5.15.0-1012-nvidia-tegra-igx /var/lib/snapd/hostfs/lib/firmware] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libcudadebugger.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin' 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin as /lib/firmware/nvidia/535.161.08/gsp_ga10x.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-smi" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 as [/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin as /lib/firmware/nvidia/535.161.08/gsp_tu10x.bin 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-smi                          
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLESv2_nvidia.so.535.161.08' 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia/535.161.08/gsp*.bin as [/var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_ga10x.bin /var/lib/snapd/hostfs/lib/firmware/nvidia/535.161.08/gsp_tu10x.bin] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-smi' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 as /usr/lib/aarch64-linux-gnu/libGLX_nvidia.so.535.161.08 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-smi as [/var/lib/snapd/hostfs/usr/bin/nvidia-smi] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-debugdump                    
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-smi as /usr/bin/nvidia-smi 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-debugdump" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-debugdump' 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-debugdump as [/var/lib/snapd/hostfs/usr/bin/nvidia-debugdump] 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-debugdump as /usr/bin/nvidia-debugdump 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-persistenced                 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-persistenced" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-persistenced as [/var/lib/snapd/hostfs/usr/bin/nvidia-persistenced] 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-persistenced' 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-persistenced as /usr/bin/nvidia-persistenced 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-cuda-mps-control" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-cuda-mps-control as [/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control] 
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-cuda-mps-server              
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control' 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-control as /usr/bin/nvidia-cuda-mps-control 
DEBUG  [2024-06-20T15:41:27Z] Checking candidate '/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server' 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia-cuda-mps-control             
DEBUG  [2024-06-20T15:41:27Z] Found 1 candidates; ignoring further candidates 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia-cuda-mps-server" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/var/lib/snapd/hostfs/snap/lxd/current/bin /var/lib/snapd/hostfs/run/bin /var/lib/snapd/hostfs/snap/lxd/x1/usr/sbin /var/lib/snapd/hostfs/snap/lxd/x1/usr/bin /var/lib/snapd/hostfs/snap/lxd/x1/sbin /var/lib/snapd/hostfs/snap/lxd/x1/bin /var/lib/snapd/hostfs/usr/local/sbin /var/lib/snapd/hostfs/usr/local/bin /var/lib/snapd/hostfs/usr/sbin /var/lib/snapd/hostfs/usr/bin /var/lib/snapd/hostfs/sbin /var/lib/snapd/hostfs/bin /var/lib/snapd/hostfs/usr/games /var/lib/snapd/hostfs/usr/local/games /var/lib/snapd/hostfs/snap/lxd/current/bin] 
DEBUG  [2024-06-20T15:41:27Z] Located nvidia-cuda-mps-server as [/var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server] 
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/nvidia_drv.so" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/nvidia_drv.so           
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/nvidia_drv.so: pattern nvidia/xorg/nvidia_drv.so not found 
DEBUG  [2024-06-20T15:41:27Z] Locating nvidia/xorg/libglxserver_nvidia.so.535.161.08 
INFO   [2024-06-20T15:41:27Z] Selecting /var/lib/snapd/hostfs/usr/bin/nvidia-cuda-mps-server as /usr/bin/nvidia-cuda-mps-server 
DEBUG  [2024-06-20T15:41:27Z] returning cached mounts                      
DEBUG  [2024-06-20T15:41:27Z] Locating "nvidia/xorg/libglxserver_nvidia.so.535.161.08" in [/var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu] 
WARNING[2024-06-20T15:41:27Z] Could not locate nvidia/xorg/libglxserver_nvidia.so.535.161.08: pattern nvidia/xorg/libglxserver_nvidia.so.535.161.08 not found

What is quite odd is the fact that the discoverability logic is looking for path starting by /var/lib/snapd/hostfs/var/lib/snapd/hostfs and not /var/lib/snapd/hostfs...
Is it a bug or is it because the cdilib object has not been well configured?

This actually leads to a problem where nvidia/xorg/libglxserver_nvidia.so.535.161.08 is not generated in the output (missing mount entry + hook entry)
The logic look for the pattern inside /var/lib/snapd/hostfs/var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu and /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu.
If it was looking in /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu, then it would have worked.

What do you think @elezar ? /cc @tomponline
@elezar
Copy link
Member

elezar commented Jun 24, 2024

This is a bug in the "discovery" code for the graphics libraries as implemented here:

nvidia-container-toolkit/internal/discover/graphics.go

Lines 268 to 281 in ddeeca3

libRoot := filepath.Dir(libcudaPath)
xorgLibs := NewMounts(
logger,
lookup.NewFileLocator(
lookup.WithLogger(logger),
lookup.WithRoot(driver.Root),
lookup.WithSearchPaths(libRoot, "/usr/lib/x86_64-linux-gnu"),
lookup.WithCount(1),
),
driver.Root,
[]string{
"nvidia/xorg/nvidia_drv.so",
fmt.Sprintf("nvidia/xorg/libglxserver_nvidia.so.%s", version),
},
:

	libRoot := filepath.Dir(libcudaPath)
	xorgLibs := NewMounts(
		logger,
		lookup.NewFileLocator(
			lookup.WithLogger(logger),
			lookup.WithRoot(driver.Root),
			lookup.WithSearchPaths(libRoot, "/usr/lib/x86_64-linux-gnu"),
			lookup.WithCount(1),
		),
		driver.Root,
		[]string{
			"nvidia/xorg/nvidia_drv.so",
			fmt.Sprintf("nvidia/xorg/libglxserver_nvidia.so.%s", version),
		},
	)

The issue in this case is that the libRoot as /var/lib/snapd/hostfs/lib/aarch64-linux-gnu and this is prepended with driver.Root (/var/lib/snapd/hostfs) to locate libglxserver_nvidia.so. Note that we allow for /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu to be searched, but not /var/lib/snapd/hostfs/usr/lib/aarch64-linux-gnu.

#548 reworks this somewhat, but from your description it seems that there is still a bug that would surface in your case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@elezar @gabrielmougard