Compute Capacity Review¶

Last updated: 2026-06-23

Summary¶

Host	Status	Notes
nebula-1	🟢 Green	Low load, lightest node
nebula-2	🟢 Green	Normal
nebula-3	🟢 Green	Fewest pods, lower utilization
nebula-4	🟢 Green	Normal
nebula-5	🟡 Yellow	Highest load + pod count; watch memory
big-boi	🟡 Yellow	Root disk 66%, /mnt 75% — trending up
shared-lgtm	🟢 Green	Root disk 57%; RAM fine
shared-dnsmasq	🟢 Green	Low utilization across the board
NFS (TrueNAS)	🟡 Yellow	38T/48T used (80%) — 10TB remaining

No red conditions today. Three yellow items to track.

MicroK8s Cluster (nebula-1 through nebula-5)¶

All nodes: 4 vCPU / 32GB RAM / ~936GB NVMe (Lenovo ThinkCentre 910q)

CPU Utilization (10m avg at time of review)¶

Node	CPU cores	CPU %	Load (1/5/15)
nebula-1	4	8%	0.58 / 0.59 / 0.46
nebula-2	4	22%	0.41 / 0.66 / 0.71
nebula-3	4	10%	1.11 / 0.63 / 0.51
nebula-4	4	16%	0.82 / 0.69 / 0.67
nebula-5	4	24%	1.87 / 1.58 / 1.65

All nodes comfortably under 25% CPU. nebula-5 has the highest sustained load — worth watching if more workloads are added. No immediate action needed.

Memory Utilization¶

Node	Used	Total	%
nebula-1	3.9 GiB	32 GB	12%
nebula-2	8.6 GiB	32 GB	27%
nebula-3	7.5 GiB	32 GB	23%
nebula-4	7.3 GiB	32 GB	23%
nebula-5	10.4 GiB	32 GB	32%

Cluster-wide memory usage is comfortable. nebula-5 at 32% is the high point. With 32GB per node and ~20GB available on the busiest node, there's substantial headroom before memory pressure becomes a concern.

Disk Utilization (root filesystem)¶

Node	Used	Total	%
nebula-1	178 GB	936 GB	20%
nebula-2	221 GB	936 GB	25%
nebula-3	276 GB	936 GB	31%
nebula-4	125 GB	936 GB	14%
nebula-5	328 GB	936 GB	37%

Disk usage includes Longhorn replica data stored on each node's root filesystem. All nodes well within bounds. nebula-5 at 37% is highest — driven by Longhorn replica storage (185GB scheduled).

Longhorn Storage¶

Node	Max	Scheduled	Available	Scheduled %
nebula-1	935 GB	12 GB	757 GB	1%
nebula-2	935 GB	129 GB	715 GB	14%
nebula-3	935 GB	197 GB	660 GB	21%
nebula-4	935 GB	122 GB	810 GB	13%
nebula-5	935 GB	185 GB	607 GB	20%

Total Longhorn capacity: 4,675 GB | Scheduled: 645 GB (14%)

Longhorn is vastly underutilized. Capacity is not a concern for current workloads. Replication factor appears to be 3x on most volumes (replicas spread across 3 nodes).

Pod Distribution¶

Node	Pod Count
nebula-5	33
nebula-2	31
nebula-4	25
nebula-1	21
nebula-3	9

Pod distribution is uneven — nebula-3 has only 9 pods vs nebula-5's 33. This is likely due to node affinity rules or Longhorn replica placement pulling workloads away from nebula-3. Not a problem given available resources, but could be leveled out if needed via pod disruption budget adjustments or anti-affinity rules.

big-boi (192.168.1.21)¶

Bare metal: 12 CPU / 64 GB RAM. Runs Docker (Ollama, n8n, OpenClaw, Kiro, Postgres).

CPU & Memory¶

	Value
CPUs	12
RAM used	5.2 GB / 62 GB (8%)
Load (1/5/15)	0.38 / 0.33 / 0.33

CPU and RAM are not a concern. big-boi has plenty of headroom.

Docker Container Resource Snapshot¶

Container	CPU %	Memory
steam-headless	17.65%	3.6 GB
openclaw-gateway	25.53%	892 MB
ollama	0.00%	1.05 GB
open-webui	0.15%	536 MB
n8n	0.27%	340 MB
kiro-gateway	0.13%	129 MB
plex	0.18%	162 MB
swag	0.18%	71 MB
n8n-postgres	0.04%	70 MB
nvidia-gpu-exporter	0.00%	11 MB

openclaw-gateway is the most CPU-active (~25%) — expected during active sessions. steam-headless at 17.65% is notable; it persists even when idle.

Disk¶

Mount	Used	Total	%	Status
`/` (root)	70 GB	114 GB	66%	🟡 Watch
`/mnt` (data disk)	647 GB	916 GB	75%	🟡 Watch
`/nfs` (TrueNAS)	38 TB	48 TB	80%	🟡 Watch

Root disk (66%): Docker image/layer cache is the likely contributor. Run docker system prune periodically to reclaim space from unused images/layers. At current growth rate, no urgency — but worth watching.

Data disk /mnt (75%, 269 GB free): Contains Ollama models, AI data, Docker volume data. Ollama models alone can be multiple GB each. Review and prune unused models with ollama list + ollama rm.

NFS /nfs (80%, 10 TB free): Media library + backups + ingest data. 10TB remaining. Growth rate depends on media downloads and S3 archive frequency. No immediate action, but add an alert threshold at 90%.

GPU¶

Metric	Value
GPU	NVIDIA GeForce RTX 3070
Utilization	0% (idle)
VRAM Used	159 MiB / 8,192 MiB (2%)
Temperature	45°C
Power Draw	9.4 W (idle)

GPU is healthy. RTX 3070 installed (2026-07-19). VRAM headroom is fine for current models (llama3.2:3b).

Shared VMs¶

shared-lgtm (192.168.1.10) — Grafana, Loki, Prometheus, Mimir, Tempo¶

	Value
CPUs	2
RAM	1.5 GB / 8 GB (19%)
Root disk	79 GB / 146 GB (57%)
Load (1/5/15)	0.34 / 0.18 / 0.13

Status: 🟢 Green. Recently expanded from 3.8→8GB RAM (2026-06-19). Current memory usage well within bounds. Disk at 57% — Loki log retention and Mimir block storage will grow over time; monitor.

shared-dnsmasq (192.168.1.29) — DNS¶

	Value
CPUs	2
RAM	682 MB / 3.8 GB (18%)
Root disk	7.9 GB / 146 GB (6%)
Load (1/5/15)	N/A

Status: 🟢 Green. No concerns.

NFS / TrueNAS (192.168.1.200)¶

48 TB pool. Used: 38 TB (80%). Free: ~10 TB.

Usage breakdown (estimated from known data): - Media (TV, movies, music, anime, audiobooks): bulk of usage - S3 backups: /nfs/backup/heezy — archived to S3 monthly - Ingest: /nfs/heezy/ingest — statement PDFs, small

Recommendation: Set up a Grafana alert when NFS usage exceeds 90% (43.2 TB used). At typical media-centric growth, 10 TB could last 6–18 months. Consider expanding TrueNAS pool or offloading cold content to S3 Glacier when headroom drops below 20%.

Recommendations¶

Priority: Low (no immediate action required)¶

big-boi root disk — prune Docker images periodically:
```
docker system prune -f --volumes
docker image prune -a -f
```
Target: stay under 80% (currently 66%).
big-boi /mnt data disk — review Ollama model inventory:
```
ssh mcp-admin@192.168.1.21 -t 'sudo docker exec ollama ollama list'
```
Remove unused models. Target: stay under 85% (currently 75%).
NFS alert — add Grafana alert for node_filesystem_avail_bytes{mountpoint="/nfs"} / node_filesystem_size_bytes{mountpoint="/nfs"} < 0.10 on big-boi (when node_exporter metrics flow). Trigger at <10% free.
nebula-5 pod skew — if workloads grow, consider reviewing anti-affinity rules to balance pods more evenly across nodes. Not urgent at current utilization.
steam-headless CPU — consuming ~17% CPU when idle. If Trent isn't actively using it, consider stopping the container when not needed.

No-Action Items (healthy, just documenting)¶

k8s cluster memory: 12–32% per node, 20+ GB headroom per node
k8s CPU: 8–24%, no node over 1.87 load average on 4-core hosts
Longhorn: 14% scheduled capacity used, 4+ TB available
shared-lgtm RAM: 19% (after 3.8→8GB upgrade)
big-boi RAM: 8% (12 CPU / 64 GB, well underutilized for AI workloads)
GPU: idle, healthy, 3070 swap will improve performance further

Capacity Headroom Projections¶

Assuming current growth patterns and no new major workloads:

Resource	Current	Concern Threshold	Estimated Time to Threshold
nebula memory	12–32%	80%	12+ months
nebula CPU	8–24%	80%	12+ months
Longhorn	14%	70%	12+ months
big-boi root disk	66%	90%	3–6 months (Docker layer accumulation)
big-boi /mnt	75%	90%	3–6 months (model/data growth)
NFS	80%	90%	6–18 months (media growth rate dependent)

The cluster is in good shape. The three yellow items are disk-related on big-boi and NFS — routine maintenance tasks, not capacity crises.