Compute Capacity Review¶
Measured 2026-08-22, except big-boi and shared-lgtm, re-measured 2026-08-25 after the LGTM stack moved between them. Every number on this page is a point-in-time reading; re-measure before acting on an old copy.
Summary¶
| Host | Status | Notes |
|---|---|---|
| nebula-1 | 🟢 Green | Lightest pod count (12), 30% root disk from Longhorn replicas |
| nebula-2 | 🟢 Green | Normal |
| nebula-3 | 🟢 Green | Highest Longhorn share |
| nebula-4 | 🟢 Green | Normal |
| nebula-5 | 🟢 Green | Lowest memory and CPU; almost no Longhorn replicas left on it |
| big-boi | 🟡 Yellow | /mnt 89% (96 GB free) and now carries LGTM as well as Steam |
| shared-lgtm | ⚪ Idle | Stack stopped; kept as the LGTM rollback. 78 GB of old data, root 63% |
| shared-dnsmasq | 🟢 Green | Low utilization across the board |
| NFS (TrueNAS) | 🟢 Green | 35 TB / 48 TB (73%) — improved from 80% in June |
No red conditions. big-boi /mnt is the tightest thing on this page.
🟡 big-boi disks¶
/mnt breakdown:
| Path | Size | What |
|---|---|---|
/mnt/games |
424 GB | Steam library |
/mnt/steam-data |
241 GB | steam-headless home dir, including a second Steam library and shader caches |
/mnt/docker |
26 GB | Docker data-root |
/mnt/ai-data |
55 GB | Ollama models, ComfyUI checkpoints and AI working data |
/mnt/lgtm |
28 GB | LGTM config and data — Prometheus is 25 GB of it |
/mnt/nfs-media |
4 KB | empty mount point |
665 GB of the 774 GB is Steam content across two directories, so that is where the next 100 GB comes
from if it is needed. The two libraries overlap: /mnt/steam-data/.steam/debian-installation/steamapps/common
holds a second copy of at least Half-Life. Shader caches regenerate and can be cleared safely.
# What is installed, largest first
ssh mcp-admin@192.168.1.21 -t 'sudo du -sh /mnt/games/steamapps/common/* | sort -rh | head -20'
# Shader caches
ssh mcp-admin@192.168.1.21 -t 'sudo du -sh /mnt/games/steamapps/shadercache /mnt/steam-data/.steam/debian-installation/steamapps/shadercache'
Uninstalling games is the user's call.
Root is 26 GB of 114 GB, mostly /usr at 12 GB and /opt/big-boi at 13 GB — Plex's metadata library
is 6.6 GB of that and grows with the media collection. Everything that moved to big-boi writes to
/mnt: the Docker data-root, Ollama, ComfyUI, and LGTM. Keep it that way. A service pointed at
/opt will fill root long before it troubles /mnt.
MicroK8s Cluster (nebula-1 through nebula-5)¶
All nodes: 4 vCPU / 31.2 GiB RAM / 936 GB NVMe (Lenovo ThinkCentre 910q), Ubuntu 24.04.3.
CPU¶
| Node | CPU % (kubelet) | Load (1/5/15) |
|---|---|---|
| nebula-1 | 9% | 0.21 / 0.63 / 0.65 |
| nebula-2 | 17% | 1.16 / 0.85 / 0.81 |
| nebula-3 | 15% | 0.49 / 0.60 / 0.64 |
| nebula-4 | 13% | 0.58 / 0.93 / 0.96 |
| nebula-5 | 10% | 0.17 / 0.38 / 0.54 |
Every node under 20%. No node is close to saturating four cores.
Memory¶
| Node | Used (host) | % | kubectl top |
|---|---|---|---|
| nebula-1 | 4.5 GiB | 14% | 24% |
| nebula-2 | 5.3 GiB | 16% | 30% |
| nebula-3 | 5.6 GiB | 17% | 26% |
| nebula-4 | 5.1 GiB | 16% | 21% |
| nebula-5 | 3.8 GiB | 12% | 17% |
The two columns differ because kubectl top counts the kubelet's working-set, which includes page
cache the host counts separately. Both say the same thing: 25 GiB+ free per node.
Disk (root filesystem)¶
| Node | Used | Total | % |
|---|---|---|---|
| nebula-1 | 269 GB | 936 GB | 30% |
| nebula-2 | 97 GB | 936 GB | 11% |
| nebula-3 | 268 GB | 936 GB | 30% |
| nebula-4 | 173 GB | 936 GB | 20% |
| nebula-5 | 214 GB | 936 GB | 24% |
Root usage is mostly Longhorn replica data under /var/lib/longhorn/.
Longhorn Storage¶
| Node | Max | Scheduled | Available | Scheduled % |
|---|---|---|---|---|
| nebula-1 | 935 GB | 181 GB | 716 GB | 19% |
| nebula-2 | 935 GB | 68 GB | 901 GB | 7% |
| nebula-3 | 935 GB | 195 GB | 717 GB | 21% |
| nebula-4 | 935 GB | 145 GB | 819 GB | 16% |
| nebula-5 | 935 GB | 19 GB | 774 GB | 2% |
Total: 4,676 GB | Scheduled: 608 GB (13%)
Replicas never rebalanced after the 2026-08-06 nebula-5 reboot
nebula-5 holds 19 GB of replicas against 145–195 GB elsewhere. The reboot rebuilt nearly every
volume onto the other four nodes and Longhorn does not rebalance on its own unless
replica-auto-balance is enabled. Capacity is not the issue; replica placement diversity is.
Pod Distribution¶
| Node | Running pods |
|---|---|
| nebula-4 | 27 |
| nebula-2 | 26 |
| nebula-3 | 26 |
| nebula-5 | 25 |
| nebula-1 | 12 |
nebula-1 carries less than half of what the others do. No affinity rule explains it — no heezy
workload sets nodeSelector or nodeAffinity. It is scheduler history, not policy, and there is
enough headroom that it does not matter today.
big-boi (192.168.1.21)¶
Bare metal: 12 CPU / 63 GB RAM. Runs Docker: Plex, steam-headless, SWAG, Ollama, Open WebUI, Postgres, nvidia-gpu-exporter.
CPU & Memory¶
| Value | |
|---|---|
| CPUs | 12 |
| RAM used | 2.5 GB / 63 GB (4%) |
| Load (1/5/15) | 0.02 / 0.08 / 0.13 |
| Uptime | 24 days |
Massively underutilized on CPU and RAM. The constraint here is disk, not compute.
Docker Container Resource Snapshot¶
| Container | CPU % | Memory |
|---|---|---|
| steam-headless | 10.45% | 963 MB |
| ollama | 0.00% | 1.56 GB |
| open-webui | 0.15% | 554 MB |
| plex | 0.39% | 352 MB |
| swag | 0.26% | 93 MB |
| n8n-postgres | 0.01% | 44 MB |
| nvidia-gpu-exporter | 0.00% | 20 MB |
openclaw-gateway, n8n and kiro-gateway are gone — retired August 2026. steam-headless still
burns ~10% of one core while idle.
Disk¶
| Mount | Used | Total | % | Status |
|---|---|---|---|---|
/ (root) |
31 GB | 114 GB | 29% | 🟢 Improved from 66% in June |
/mnt (data disk) |
869 GB | 916 GB | 100% | 🔴 476 MB free |
/nfs (TrueNAS) |
35 TB | 48 TB | 73% | 🟢 Improved from 80% |
GPU¶
| Metric | Value |
|---|---|
| GPU | NVIDIA GeForce RTX 3070 |
| Utilization | 0% (idle) |
| VRAM Used | 346 MiB / 8,192 MiB (4%) |
| Temperature | 48°C |
| Power Draw | 23 W (idle) |
Healthy. Plenty of VRAM headroom for the models in use.
Shared VMs¶
shared-lgtm (192.168.1.10) — idle, kept as the LGTM rollback¶
| Value | |
|---|---|
| CPUs | 2 |
| RAM | 0.5 GB / 7.8 GB (6%) |
| Root disk | 88 GB / 146 GB (63%) |
| Load (1/5/15) | 0.08 / 0.02 / 0.01 |
| Uptime | 67 days |
⚪ Idle. The monitoring stack runs on big-boi now; this VM has no containers and lgtm-stack is
inactive. It stays a managed host — baseline, promtail and its DNS record — so it keeps shipping
its own logs and metrics. /opt/lgtm still holds 78 GB of the pre-move data, which is the whole
point of keeping it.
lgtm-stack is inactive but still enabled
A reboot of this VM starts a second full LGTM stack: a Prometheus scraping every target in the lab a second time, and a Loki and Mimir taking writes nothing reads. Disable the unit, or decommission the VM, before the 78 GB is no longer wanted as a rollback.
shared-dnsmasq (192.168.1.29) — DNS¶
| Value | |
|---|---|
| CPUs | 2 |
| RAM | 0.7 GB / 3.8 GB (18%) |
| Root disk | 7.9 GB / 146 GB (6%) |
| Load (1/5/15) | 0.22 / 0.07 / 0.02 |
| Uptime | 365 days |
🟢 Green. No concerns.
NFS / TrueNAS (192.168.1.200)¶
48 TB pool. Used: 35 TB (73%). Free: 14 TB.
- Media (TV, movies, music, anime, audiobooks): bulk of usage
- Backups:
/nfs/heezy/backups, 7-day rolling, weekly generation shipped to S3 - Ingest:
/nfs/heezy/ingest— statement PDFs, small
Down from 80% in June, so something was pruned or expired in the interim. 14 TB of headroom.
Recommendations¶
Priority: Low¶
-
big-boi
/mntat 89%. 96 GB free with Steam holding 665 GB of it. Not urgent, but it is the only disk in the lab without comfortable headroom, and it is now also where LGTM writes. -
shared-lgtm — decide whether the rollback is still wanted. Until then, disable the
lgtm-stackunit so a reboot cannot bring a second stack up. -
Longhorn replica rebalance — nebula-5 holds 2% of scheduled bytes. Consider enabling
replica-auto-balance: best-effortso a future node loss does not concentrate replicas further. -
NFS alert — there is still no cluster-wide filesystem alert for
/nfs.disk-alertscovers root filesystems andstorage-alertscovers only the slskd downloads PVC. An alert onnode_filesystem_avail_bytes{mountpoint="/nfs"}below 15% would have flagged/mnttoo, had the same rule covered/mnt. -
steam-headless CPU — ~10% of a core while idle. Stop the container when it is not in use.
No-action items¶
- k8s memory: 12–17% per node by host accounting, 25 GiB+ free each
- k8s CPU: 9–17%, no node over 1.16 load on 4 cores
- Longhorn: 13% scheduled of 4.6 TB
- big-boi RAM: 8% of 62 GB
- NFS: 73%, 14 TB free
- GPU: idle, 48°C, 4% VRAM
Capacity Headroom Projections¶
| Resource | Current | Concern Threshold | Estimated Time to Threshold |
|---|---|---|---|
| nebula memory | 12–17% | 80% | 12+ months |
| nebula CPU | 9–17% | 80% | 12+ months |
| Longhorn | 13% | 70% | 12+ months |
| big-boi root disk | 25% | 90% | 12+ months |
big-boi /mnt |
89% | 90% | at the threshold; Steam growth sets the pace |
| shared-lgtm root | 63% | 85% | not growing — the stack is stopped |
| NFS | 73% | 90% | 6–18 months, media growth dependent |