steam-headless Not Rendering On NVIDIA

Runbook for the Grafana alerts Steam Headless Not Rendering On NVIDIA (steam-headless-glx-not-nvidia), Steam Headless NVIDIA Driver Pin Stale (steam-headless-driver-pin-stale) and Steam Headless GLX Probe Stale (steam-headless-glx-probe-stale), group steam-headless-alerts.

What the alert means

steam-headless on big-boi runs a full Xorg desktop against the RTX 3070. When OpenGL inside that container is not served by the NVIDIA driver, Steam loses CUDA/GL interop, reports NVENC - No CUDA support, falls back to a VAAPI encoder that has no driver behind it, and segfaults on every Steam Link connection.

Nothing else goes red when this happens. The container stays up, the desktop comes up, nvidia-smi still lists the GPU, and the host's nvidia-gpu-exporter metrics stay healthy. The only signal is this alert.

How it is measured

/opt/big-boi/steam/glx-probe.sh runs every 5 minutes from /etc/cron.d/big-boi-steam-glx-probe. It runs docker exec steam-headless glxinfo -B and writes three gauges to node_exporter's textfile collector at /var/lib/node_exporter/textfile_collectors/steam_headless_glx.prom:

Metric	Meaning
steam_headless_running	The container is running.
steam_headless_glx_nvidia	glxinfo reported OpenGL vendor string: NVIDIA Corporation.
steam_headless_driver_mismatch	The container's NVIDIA_DRIVER_VERSION differs from the host's loaded kernel module (/proc/driver/nvidia/version).

The alert condition is steam_headless_running - steam_headless_glx_nvidia > 0.5, which is 1 only when the container is up without NVIDIA GL, and 0 when it is healthy or deliberately stopped. It has a for: 15m, the group evaluates every 5 minutes, and the notification policy repeats every 4 hours, so expect roughly 20 minutes from cause to Discord and a repeat every 4 hours after that.

steam-headless-glx-probe-stale is the companion: it fires when node_textfile_mtime_seconds for that file is more than 30 minutes behind, which means the probe itself is broken rather than the GPU. Its noDataState is Alerting, so it also fires when the series is absent entirely — see Triage: probe stale, because a current .prom file on disk is not enough.

The resolve notification does not mean anything was fixed

/opt/big-boi/steam/daily-restart.sh restarts the container at 05:00 America/New_York every day. That restart clears causes 1 and 2, so an alert that fires overnight resolves on its own in the morning whether or not anyone looked at it. If you got a resolve you did not earn, the cause is still there and it will happen again.

Triage

All three causes look identical from the metric. Xorg's own log is what separates them. It is on a bind mount, so it survives container restarts:

ssh mcp-admin@bigboi.heezy.local -t "grep -nE 'glxserver|Failed to load' /opt/big-boi/steam-headless/.cache/log/xorg.err.log | tail -20"

The file is appended across restarts, so read the last block. A healthy Xorg start looks like this:

(II) Loading /usr/lib/x86_64-linux-gnu/nvidia/xorg/libglxserver_nvidia.so
(II) Module glxserver_nvidia: vendor="NVIDIA Corporation"
(II) NVIDIA GLX Module  595.84

(II) GLX: Another vendor is already registered for screen 0 appears on every healthy start. Ignore it.

1. Device access stripped by a systemd reload

Most common. On cgroup v2 with the systemd cgroup driver, systemctl daemon-reload resets the eBPF device filter for containers Docker started, and the container silently loses /dev/nvidia*.

Xorg already holds its file descriptors, so it keeps running and logs nothing. Every process started afterwards, glxinfo and Steam's renderer included, cannot open the device and falls back to Mesa.

Signature:

• The last Xorg block in xorg.err.log loaded NVIDIA GLX cleanly.
• No Xorg restart in the container log across the whole alert window.
• systemd[1]: Reloading. in the host journal shortly before the first bad probe.

glxinfo does not always name Mesa when this happens. It can fail outright instead, which counts the same as far as the probe is concerned:

X Error of failed request:  BadValue (integer parameter out of range for operation)
  Major opcode of failed request:  151 (GLX)
  Minor opcode of failed request:  24 (X_GLXCreateNewContext)

The server side is fine in that state — glxserver_nvidia is loaded and the versions match — so this is still cause 1, not cause 3. Check xorg.err.log before assuming a driver problem.

ssh mcp-admin@bigboi.heezy.local -t "sudo journalctl --since '-12h' | grep -E 'Reloading\.|ansible-systemd'"

Any Ansible run that touches a systemd unit on big-boi does this, including playbooks/baseline.yml, which installs node_exporter with daemon_reload: yes on every run.

2. Xorg gave up and supervisord stopped retrying

If device access is lost while Xorg is starting rather than while it is running, Xorg cannot initialise the kernel module, supervisord retries four times in six seconds and gives up. The container then sits there healthy by every other measure with no X server in it.

Signature, in docker logs steam-headless:

gave up: xorg entered FATAL state, too many start retries too quickly

3. Driver mismatch after a version bump

steam-headless-driver-pin-stale fires alongside the Mesa alert. gpu.yml reads the host driver version when the big-boi playbook runs and templates it into /opt/big-boi/steam/docker-compose.yml as NVIDIA_DRIVER_VERSION. If the host driver moves without a playbook run, the container keeps installing the old userspace and Xorg segfaults loading its GLX module against the host's newer nvidia_drv.so:

(II) NVIDIA dlloader X Driver  <new version>
(II) NVIDIA GLX Module  <old version>
(EE) Segmentation fault at address 0x0

supervisord then gives up on xorg (FATAL), so there is no X server at all. A restart reuses the same pin and changes nothing.

roles/big-boi/tasks/nvidia.yml blacklists the NVIDIA packages from unattended-upgrades, so the driver should only move through that role. If this fires anyway, dispatch the big-boi playbook, then find what changed the driver.

A second, older form of this cause is the image's stale Debian GLX symlink. fix-nvidia-glxserver.sh repoints it on every container start. Its signature is a Failed to load module "glxserver_nvidia" line rather than a segfault.

Fix

Causes 1 and 2 are both fixed by a restart, which re-creates the device allowlist and starts a fresh Xorg:

ssh mcp-admin@bigboi.heezy.local -t "sudo docker restart steam-headless"

A container whose GL is broken cannot be running a game, so this is not disruptive. Give it about a minute, then confirm:

ssh mcp-admin@bigboi.heezy.local -t "sudo docker exec steam-headless glxinfo -B | grep -E 'vendor|renderer'"

Healthy output names the GPU, not Mesa or llvmpipe:

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: NVIDIA GeForce RTX 3070/PCIe/SSE2

The probe picks this up within 5 minutes and the alert clears on the next evaluation. To confirm without waiting for Grafana:

ssh mcp-admin@bigboi.heezy.local -t "sudo /opt/big-boi/steam/glx-probe.sh; cat /var/lib/node_exporter/textfile_collectors/steam_headless_glx.prom"

Cause 3 is a playbook run, not a restart. Dispatch the big-boi playbook through Gitea Actions.

Reattaching from Ansible

playbooks/big-boi.yml already handles this. Its post_tasks import roles/big-boi/tasks/gpu-verify.yml, which runs after handlers flush, checks NVML and GLX in every GPU container, and restarts only the ones that are actually broken. A healthy run restarts nothing. Running that playbook is a valid fix for causes 1 and 2 if you would rather not touch the host by hand.

playbooks/baseline.yml and playbooks/lgtm.yml also reach big-boi and reload systemd, so both run the same gpu-verify post_tasks, limited to big-boi.

Triage: probe stale

Check the metric before the file. The two can disagree, and when they do the file is the misleading one.

ssh mcp-admin@bigboi.heezy.local -t "curl -s localhost:9100/metrics | grep -e node_textfile -e steam_headless"

The series is missing entirely

If node_textfile_mtime_seconds is absent while /var/lib/node_exporter/textfile_collectors/steam_headless_glx.prom is current on disk, node_exporter is not reading the directory. Check the unit:

ssh mcp-admin@bigboi.heezy.local -t "grep ExecStart /etc/systemd/system/node_exporter.service"

It must carry --collector.textfile.directory. Without the flag the collector is off, every textfile metric vanishes, and with noDataState: Alerting the rule fires. The unit belongs to the prometheus-node-exporter role and nothing else may write it; a second writer that omits the flag is how this happened on 2026-08-25. Re-run playbooks/baseline.yml, which runs that role for every Linux host.

The series is present but stale

Then the probe really is not running. Check the cron and the script:

ssh mcp-admin@bigboi.heezy.local -t "cat /etc/cron.d/big-boi-steam-glx-probe; sudo /opt/big-boi/steam/glx-probe.sh; echo rc=$?"

The script exits non-zero without writing if the textfile directory does not exist, which is deliberate: publishing into a void would leave the alert quiet for the wrong reason.

The probe cannot tell you why

glx-probe.sh sets steam_headless_glx_nvidia=0 on any failure of timeout 30 docker exec steam-headless glxinfo -B, including an exec that fails or times out for reasons unrelated to the GPU. A broken probe and a real Mesa fallback publish the same value, which is why the triage above starts at xorg.err.log rather than at the metric.