How to keep `nvidia-smi` working?

```bash
apptainer shell --writable --fakeroot --nv debian-sandbox/
```
```
WARNING: nv files may not be bound with --writable
WARNING: Skipping mount /usr/bin/nvidia-debugdump [files]: /usr/bin/nvidia-debugdump doesn't exist in container
WARNING: Skipping mount /usr/bin/nvidia-cuda-mps-control [files]: /usr/bin/nvidia-cuda-mps-control doesn't exist in container
WARNING: Skipping mount /usr/bin/nvidia-cuda-mps-server [files]: /usr/bin/nvidia-cuda-mps-server doesn't exist in container
WARNING: Skipping mount /usr/share/vulkan/implicit_layer.d/nvidia_layers.json [files]: /usr/share/vulkan/implicit_layer.d/nvidia_layers.json doesn't exist in container
WARNING: Skipping mount /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json [files]: /usr/share/egl/egl_external_platform.d/15_nvidia_gbm.json doesn't exist in container
WARNING: Skipping mount /usr/bin/nvidia-persistenced [files]: /usr/bin/nvidia-persistenced doesn't exist in container
WARNING: Skipping mount /var/run/nvidia-persistenced/socket [files]: /var/run/nvidia-persistenced/socket doesn't exist in container
WARNING: Skipping mount /usr/share/glvnd/egl_vendor.d/10_nvidia.json [files]: /usr/share/glvnd/egl_vendor.d/10_nvidia.json doesn't exist in container
WARNING: Skipping mount /usr/share/vulkan/icd.d/nvidia_icd.json [files]: /usr/share/vulkan/icd.d/nvidia_icd.json doesn't exist in container
WARNING: Skipping mount /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json [files]: /usr/share/egl/egl_external_platform.d/10_nvidia_wayland.json doesn't exist in container
WARNING: Skipping mount /usr/share/nvidia/nvoptix.bin [files]: /usr/share/nvidia/nvoptix.bin doesn't exist in container
WARNING: Skipping mount /usr/bin/nvidia-smi [files]: /usr/bin/nvidia-smi doesn't exist in container
WARNING: Error changing the container working directory. Using '/root' instead: chdir /scratch/loison_benjamin/apptainer: no such file or directory
Apptainer> nvidia-smi
bash: nvidia-smi: command not found
Apptainer>
```
May 20, 8:57am

The issue you're encountering is due to the --writable flag conflicting with the --nv (NVIDIA GPU support) flag in Apptainer (formerly Singularity). The --writable flag makes the container's filesystem writable, but this can prevent the NVIDIA drivers and libraries from being properly mounted into the container, which is why nvidia-smi is not found.

How to Fix nvidia-smi in Apptainer

1. Remove the --writable Flag

The simplest solution is to remove the --writable flag if you don't need a writable filesystem. The --nv flag requires read-only access to the host's NVIDIA drivers and libraries.

bash
apptainer shell --fakeroot --nv debian-sandbox/

2. Use --nv with --containall (if needed)

If you need a writable filesystem, you can try using --containall instead of --writable. This ensures all necessary host files (including NVIDIA libraries) are mounted.

bash
apptainer shell --fakeroot --nv --containall debian-sandbox/

3. Manually Mount NVIDIA Directories

If the above doesn't work, manually mount the required NVIDIA directories and files using the --bind flag. For example:

bash
apptainer shell --fakeroot --nv \
  --bind /usr/bin/nvidia-smi \
  --bind /usr/lib/x86_64-linux-gnu/libnvidia-ml.so \
  debian-sandbox/

Replace the paths with the actual paths on your host system. You can find the required files with:

bash
which nvidia-smi
ldd $(which nvidia-smi)

4. Ensure NVIDIA Drivers are Installed in the Container

If the container image (debian-sandbox/) does not have the NVIDIA drivers or nvidia-smi installed, you may need to:

  • Use a base image that already includes NVIDIA support (e.g., nvidia/cuda from Docker Hub).
  • Install the NVIDIA drivers and tools inside the container.

Example for Debian-based containers:

bash
apt update && apt install -y nvidia-utils-<version> nvidia-smi

Replace <version> with your driver version (e.g., 535).

5. Check Host NVIDIA Installation

Ensure that:

  • The host system has NVIDIA drivers installed.
  • The user has permission to access /dev/nvidia* devices.

Run on the host:

bash
ls /dev/nvidia*
nvidia-smi

If these commands fail, install the NVIDIA drivers on the host first.

Summary of Recommended Command

Start with this:

bash
apptainer shell --fakeroot --nv debian-sandbox/

If you need a writable filesystem, try:

bash
apptainer shell --fakeroot --nv --containall debian-sandbox/
May 20, 8:57am

This is a shared conversation and may include unverified content that doesn't reflect Mistral AI's views. Login to resume chatting.

Le Chat can make mistakes. Check answers. Learn more