Skip to content

Kubernetes Infrastructure

Complete operational reference for the heezy Kubernetes cluster.

Last verified against the live cluster: 2026-08-22.

Cluster Topology

Nodes: 5 (MicroK8s HA)

Node IP OS k8s Version
nebula-1 192.168.1.15 Ubuntu 24.04.3 LTS v1.32.9
nebula-2 192.168.1.16 Ubuntu 24.04.3 LTS v1.32.9
nebula-3 192.168.1.17 Ubuntu 24.04.3 LTS v1.32.9
nebula-4 192.168.1.18 Ubuntu 24.04.3 LTS v1.32.9
nebula-5 192.168.1.19 Ubuntu 24.04.3 LTS v1.32.9

Control Plane: MicroK8s HA, backed by dqlite (not etcd). Three nodes are datastore voters and two are standby; MicroK8s picks them itself and promotes a standby when a voter leaves. Which node holds which role changes over time, so never encode it anywhere. Every node runs an apiserver and can serve kubectl.

microk8s status        # prints "datastore master nodes" and "datastore standby nodes"

CNI: Calico

Node roles: kubectl get nodes shows <none> for every node. That is normal for MicroK8s — it does not apply the node-role.kubernetes.io/* labels, and it does not mean the nodes are workers only.

Namespaces

Namespace Purpose
heezy Most homelab workloads: media, finance, mailbot, SWAG, Pi-hole, promtail
gitea Gitea, its Postgres, and the act-runner
openbao OpenBao secrets manager
external-secrets External Secrets operator
secrets ExternalSecret targets and pull secrets
longhorn-system Longhorn distributed storage
metallb-system MetalLB load balancer
ingress MicroK8s' bundled nginx ingress controller DaemonSet. Not used for heezy services — SWAG in heezy is the real ingress
gpu-operator-resources NVIDIA GPU operator and node feature discovery
kube-system Kubernetes core services
default Two legacy NFS browse PVCs, nothing else

Storage

Longhorn

Distributed block storage, 3 replicas per volume by default.

  • Replication: 3 replicas spread across nodes
  • Use case: All application configs and databases
  • Provisioner: longhorn StorageClass
  • Access mode: RWO (a few volumes use RWX)
  • Resilience: Auto-rebuild of a lost replica onto a healthy node

Health check: 

kubectl get volumes.longhorn.io -n longhorn-system

Expected: all volumes state: attached, robustness: healthy. A detached/unknown row is a volume with no pod attached, not necessarily a fault.

There is no Longhorn backup target

backup-target is empty, so nothing is shipped off-cluster by Longhorn. The only recurring job is snapshot-cleanup (Sundays 05:00), which deletes system-generated snapshots left behind by replica rebuilds and purges the frozen layers. Snapshots live on the same disks as the data and are not backups. See Backups for what actually protects this data.

NFS (CSI Driver)

RWX volumes from the TrueNAS box, via csi-nfs-node on every node.

  • NAS: 192.168.1.200, dataset Arr1/SMB, mounted on hosts as /nfs
  • Use case: Media, downloads, ingest directories
  • Access mode: RWX (ReadWriteMany)
  • Provisioner: nfs.csi.k8s.io (StorageClass nfs-media), plus a set of hand-written static PVs in base/nfs-media-static-pvs.yaml

Volumes in use: nfs-media-{movies,tv,anime,music,roms,audiobooks}, nfs-downloads, nfs-slskd-downloads, nfs-receipts, nfs-heezy-ingest, nfs-statements-ingest.

Find PVCs no pod is using: 

kubectl get pvc -A -o json | jq -r '.items[] | "\(.metadata.namespace) \(.metadata.name)"' | while read ns n; do
  used=$(kubectl get pods -n "$ns" -o json | jq --arg n "$n" '[.items[].spec.volumes[]?.persistentVolumeClaim.claimName] | index($n)')
  [ "$used" = "null" ] && echo "unused: $ns/$n"
done

MetalLB VIPs

Pool default-addresspool is 192.168.1.25–192.168.1.40.

VIP Service Ports Purpose
192.168.1.25 heezy/swag 80, 443 Main ingress (HTTPS termination)
192.168.1.27 heezy/pihole-lb 53, 80 DNS + admin UI
192.168.1.28 heezy/plex 32400 Plex direct access (HomeKit, clients)
192.168.1.30 heezy/swag-plex 80, 443 Plex via reverse proxy (remote access)
192.168.1.31 gitea/gitea-lb 3000 Git + Gitea Actions
192.168.1.32 openbao/openbao-lb 8200 OpenBao, follows the active pod
192.168.1.33 heezy/uptime-kuma 3001 Direct app access; SWAG does not proxy its Socket.IO API
192.168.1.34 heezy/slskd-lb 5030, 50300 Soulseek daemon behind the gluetun VPN sidecar

Half the pool is inside the DHCP scope and unreserved

The FortiGate hands out 192.168.1.10–192.168.1.199 on internal7, which covers the whole MetalLB pool. Only .25, .27, .28 and .30 have dummy-MAC DHCP reservations holding them. .31, .32, .33 and .34 do not, so a new DHCP client could be handed a live VIP. Adding a reservation is a terraform-heezy change.

Pod Placement & Node Affinity

Current state (verified 2026-08-22): no heezy workload sets nodeSelector or nodeAffinity. The only pods with a selector are the GPU operator's node-feature-discovery workers, which use gpu-operator: enabled because that is hardware-bound.

Leaving pods unpinned buys resilience, balanced placement, and simpler troubleshooting.

HARD RULE: Never add nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution for a specific hostname unless it is hardware-required (GPU, a specific device). If you pin a workload, say why in a comment on the manifest.

Check current affinity: 

kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,NODESEL:.spec.nodeSelector --no-headers | grep -v '<none>'

Resilience Posture

Workload Replicas Notes
Most apps 1 Homelab — a brief restart is acceptable
act-runner 3 One per node, so a node reboot does not stall CI
pihole 1 DNS. Two replicas plus anti-affinity is backlogged
SWAG 1 Ingress; single point of failure, accepted

QoS: the long-running deployments now set requests, so they land Burstable or Guaranteed. What is still BestEffort is short-lived CronJob pods (mailbot, finance-sync, statement-scanner, budget-alerts, ecr-credentials-refresh, openbao-unseal) and the bundled nginx-ingress DaemonSet. The 2026-06 finding that named sabnzbd, radarr and sonarr no longer applies.

kubectl get pods -A -o custom-columns=NS:.metadata.namespace,POD:.metadata.name,QOS:.status.qosClass --no-headers | grep BestEffort

Known Issues

Services using hostNetwork

Plex — intentional and permanent. Plex must see a real LAN IP (192.168.1.x) on its interface so it registers a reachable address with plex.tv. Without it, clients receive the pod's 10.1.x.x cluster IP, fall back to the Plex relay, and show as Remote/Indirect with throttled bandwidth and forced transcoding. See Plex for full details.

Overseerr — hostNetwork: true with no matching justification. It is not pinned to a node, so the host port follows the pod around the cluster, and nothing depends on that address anyway: Overseerr has a normal ClusterIP service on 5055 and SWAG proxies overseerr.trentnielsen.me to overseerr.heezy.svc.cluster.local:5055. Dropping hostNetwork is a safe cleanup, backlogged.

Leftover debug pods

curl-rc, curl-rc2 and mailbot-probe4 are still Running in the heezy namespace. They are one-off troubleshooting pods that were never deleted and are not in any manifest.

Disaster Recovery

dqlite Quorum

MicroK8s HA runs dqlite with 3 voters out of the 5 nodes; quorum is 2 of 3.

  • One voter down: cluster keeps working, a standby is promoted
  • Two voters down at once: the API goes read-only/unavailable until a voter returns, because promotion needs quorum to commit
  • Losing two standby nodes is harmless

This is weaker than "survive any 2 node failures" — the answer depends on which two.

Longhorn Volumes

3-replica volumes tolerate losing 2 of the 3 replicas' nodes. There is no Longhorn backup target configured, so recovery from a full-cluster loss means restoring from whatever Backups covers, not from Longhorn.

Cluster Loss Recovery

  1. Rebuild 5 Ubuntu 24.04 VMs on Proxmox
  2. Install MicroK8s v1.32.9 on each and microk8s add-node / microk8s join them
  3. Enable the addons: dns, ha-cluster, gpu, dashboard
  4. Clone heezy-k8s
  5. kubectl apply -k base/ for namespaces, storage classes, MetalLB config and static NFS PVs
  6. kubectl apply -k apps/<service>/ per service, or let auto-deploy.yaml do it
  7. Restore data per Backup & Restore

Approx. recovery time: hours, not minutes. The 45-minute figure only covers steps 1–3.

kubectl Access

Everything reaches the cluster through the mcp-tools container, which bind-mounts ~/.kube. The kubeconfig context is microk8s and points at https://192.168.1.15:16443.

kubectl get pods -A

That endpoint is a single node. If nebula-1 is down, repoint the kubeconfig server at another node's :16443 — any of them serves the API.

Common Commands

Node Status

# List all nodes with details
kubectl get nodes -o wide

# Check node resource usage
kubectl top nodes

# Describe a specific node (events, conditions)
kubectl describe node nebula-1

Pod Placement & Status

# List all pods across all namespaces with node assignment
kubectl get pods -A -o wide

# Check pod resource usage
kubectl top pods -A

# Get pod logs
kubectl logs -n heezy <pod-name>

# Port-forward to a pod (for local debugging)
kubectl port-forward -n heezy svc/<service-name> 8080:80

Force Pod Restart

Useful when a pod is stuck or needs cache refresh:

# Restart all replicas of a deployment
kubectl rollout restart deployment/<name> -n heezy

# Wait for rollout to complete
kubectl rollout status deployment/<name> -n heezy

# Example: restart heezy-finance
kubectl rollout restart deployment/heezy-finance -n heezy
kubectl rollout status deployment/heezy-finance -n heezy

Longhorn Volume Health

# List all volumes with health status
kubectl get volumes.longhorn.io -n longhorn-system

# Describe a specific volume (replicas, state)
kubectl describe volume <name> -n longhorn-system

# Check the recurring jobs and the backup target
kubectl get recurringjobs.longhorn.io -n longhorn-system
kubectl get settings.longhorn.io -n longhorn-system backup-target -o jsonpath='{.value}'

Gitea DB Sync (Stale Run Status)

If a Gitea workflow shows stale status (appears "cancelled" but steps completed), manually sync the database:

kubectl exec -n gitea gitea-postgres-0 -- psql -U gitea -d gitea -c \
  "UPDATE action_run_job SET status=3 WHERE status IN (1,2) AND task_id IN (SELECT id FROM action_task WHERE stopped>0);"

This updates all incomplete jobs with completed tasks to status 3 (completed).

Check Service IPs & Endpoints

# List services with MetalLB VIPs
kubectl get svc -A -o wide | grep LoadBalancer

# Check endpoint IPs for a service
kubectl get endpoints -n heezy <service-name>

Scale Deployment

# Set replica count
kubectl scale deployment/<name> -n heezy --replicas=2

# Or use kubectl edit for full control
kubectl edit deployment/<name> -n heezy

Monitoring & Observability

There is no Prometheus Operator and no ServiceMonitor CRDs in this cluster. Prometheus runs as a Docker Compose service on big-boi (192.168.1.21) and scrapes static targets rendered by the lgtm Ansible role from inventory/hosts.yml: node-exporter on every Linux host, promtail on :9080, kubelet and cAdvisor on :10255, kube-state-metrics, the nvidia GPU exporter on big-boi, and the snmp-exporter for the FortiGate and the Cisco switch.

Adding a scrape target is an ansible-heezy change to roles/lgtm/templates/prometheus.yml.j2, not a k8s manifest.

Loki ingests logs from promtail: a systemd service on all 12 Linux hosts, plus the promtail DaemonSet in heezy for container logs.

Grafana: https://grafana.heezy.info (direct: http://192.168.1.21:3000).

See Monitoring and Capacity Review.

Best Practices

  1. Always use kubeconfig: never hardcode API endpoints in a manifest
  2. Use namespaces: separate concerns (heezy workloads vs. gitea vs. system)
  3. Set resource requests: on anything long-running, so it is not BestEffort
  4. Use Longhorn for stateful: all databases and config on Longhorn volumes, media on NFS
  5. No direct kubectl apply: push to heezy-k8s and let CI apply it (see CI/CD)
  6. Document affinity reasons: if pinning a pod, explain why in a comment
  7. Do not treat Longhorn snapshots as backups: they sit on the same disks, and no backup target is configured. See Backups