Skip to content

Kubernetes Infrastructure

Complete operational reference for the heezy Kubernetes cluster.

Cluster Topology

Nodes: 5 workers (MicroK8s HA)

Node IP OS k8s Version
nebula-1 192.168.1.15 Ubuntu 24.04.3 LTS v1.32.9
nebula-2 192.168.1.16 Ubuntu 24.04.3 LTS v1.32.9
nebula-3 192.168.1.17 Ubuntu 24.04.3 LTS v1.32.9
nebula-4 192.168.1.18 Ubuntu 24.04.3 LTS v1.32.9
nebula-5 192.168.1.19 Ubuntu 24.04.3 LTS v1.32.9

Control Plane: MicroK8s HA runs etcd cluster across all 5 nodes (no dedicated master). Every node can serve as an API endpoint.

CNI: Calico

Architecture: All nodes are workers; kubelets register with local apiserver and join the peer-to-peer etcd cluster.

Namespaces

Namespace Purpose
heezy All homelab workloads (media, finance, monitoring, ingress)
gitea Gitea + act-runner CI/CD
longhorn-system Longhorn distributed storage
kube-system Kubernetes core services
ingress SWAG ingress controller and related services
metallb-system MetalLB load balancer
openbao OpenBao secrets management
external-secrets External Secrets operator

Storage

Longhorn

Distributed block storage with 3-replica replication for resilience.

  • Replication: 3 replicas (survives 2-node failure)
  • Use case: All application configs and databases
  • Provisioner: longhorn StorageClass
  • Access mode: RWO (ReadWriteOnce)
  • Resilience: Auto-rebalance on node failure

Health check: 

kubectl get volumes.longhorn.io -n longhorn-system

Expected: All volumes have robustness: healthy and state: attached.

NFS (CSI Driver)

Distributed filesystem via csi-nfs-driver pointing to internal NAS.

  • NAS: 192.168.1.200
  • Use case: Media, downloads, ingest directories
  • Access mode: RWX (ReadWriteMany)
  • Provisioner: nfs.csi.k8s.io

Common mounts: - /media — Plex library - /downloads — SABnzbd output - /ingest — Statement parser input

Orphaned PVCs (Safe to Delete)

These PVCs exist but have no pods claiming them (2026-06-20 audit):

  • receipts-db (heezy namespace) — Longhorn 10Gi
  • sabnzbd-downloads (heezy namespace) — NFS
  • longhorn-test-pvc (longhorn-system namespace) — Longhorn test volume

Deletion is safe and will recover disk space.

MetalLB VIPs

Load balancer virtual IPs for services:

VIP Service Ports Purpose
192.168.1.25 heezy/swag 80, 443 Main ingress (HTTPS termination)
192.168.1.27 heezy/pihole-lb 53, 80 DNS + admin UI
192.168.1.28 heezy/plex 32400 Plex direct access (HomeKit, clients)
192.168.1.30 heezy/swag-plex 80, 443 Plex via reverse proxy (alt: remote access)

All VIPs are static and reserved in DHCP to prevent conflicts.

Pod Placement & Node Affinity

Current Policy (2026-06-20 audit): NO pods are pinned to specific nodes.

All workloads run on any available nebula node with no nodeAffinity constraints. This allows: - Maximum resilience (any node failure doesn't require pod reschedule) - Balanced load distribution - Simplified troubleshooting

HARD RULE: Never add nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution for a specific hostname unless hardware-required (GPU, specific device, etc.).

If you need to pin a workload, document the reason in a comment and get approval.

Check current affinity: 

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.nodeSelector}{"\n"}{end}' | grep -v '<none>'

Expected output: mostly empty (no affinity).

Resilience Posture

Workload Replicas Notes
Most apps 1 Homelab — brief restart acceptable
pihole 1 DNS; could benefit from 2 replicas + anti-affinity (backlogged)
SWAG 1 Ingress; single point of failure (acceptable for homelab)

QoS Distribution: - Guaranteed: services with both requests and limits set - Burstable: services with requests but no limits - BestEffort: 16 containers missing resource requests (2026-06-20 audit finding — backlogged for fix)

Known Issues (2026-06-20 Audit)

1. Missing Resource Requests (16 containers)

Finding: 16 containers across various deployments have no resource requests, resulting in BestEffort QoS. This can cause eviction under memory pressure.

Affected services: heezy-statement-scanner, sabnzbd, radarr, sonarr, and others.

Status: Backlogged. Fix requires: - Add .resources.requests.cpu and .resources.requests.memory to each deployment - Baseline: cpu: 100m, memory: 128Mi - Test each service post-update

2. image-prepull DaemonSet Fails on nebula-3

Finding: DaemonSet image-prepull fails on nebula-3 with error: "DinD daemon unreachable."

Impact: Cosmetic only — pod restart still works (kubelet can pull images). Does not block deployments.

Root cause: Docker-in-Docker socket misconfiguration or stale container on nebula-3.

Backlogged: Low priority; monitor for recurrence.

3. Services Using hostNetwork

Two services run with hostNetwork: true:

Plex — intentional and permanent. Plex must see a real LAN IP (192.168.1.x) on its interface so it registers a reachable address with plex.tv. Without it, clients receive the pod's 10.1.x.x cluster IP, fall back to the Plex relay, and show as Remote/Indirect with throttled bandwidth and forced transcoding. See Plex for full details.

Overseerr — binding directly to nebula-5 host port 5055. NodePort routing doesn't work; clients must connect to 192.168.1.19:5055 directly. Backlogged: consider a dedicated MetalLB VIP.

Disaster Recovery

etcd Quorum

MicroK8s HA runs etcd on all 5 nodes. Quorum requires 3 nodes.

  • Survive up to 2 node failures
  • 3+ nodes down = cluster unavailable (requires manual recovery)

Longhorn Volumes

3-replica volumes survive 2-node failure by design. If 3+ nodes fail simultaneously:

  1. Existing volumes become inaccessible (can't reach quorum replicas)
  2. Rebuild by restoring from Longhorn snapshots (if enabled)
  3. Or redeploy from git + reingest data

Cluster Loss Recovery

If the entire cluster is lost or unrecoverable:

  1. Rebuild 5 Ubuntu 24.04 VMs on Proxmox
  2. Install MicroK8s v1.32.9 on each
  3. Cluster them: microk8s join --worker <join-code>
  4. Clone heezy-k8s repo
  5. Apply manifests: kubectl apply -k infrastructure/
  6. Restore Longhorn volumes from backup (if available) or redeploy

Approx. recovery time: 45 min (VMs + MicroK8s + DNS propagation).

kubectl Access

From OpenClaw container or host machine:

export KUBECONFIG=/home/node/.openclaw/workspace/.kubeconfig
kubectl get pods -A

Kubeconfig points to internal API endpoint (192.168.1.15:16443 or similar).

Common Commands

Node Status

# List all nodes with details
kubectl get nodes -o wide

# Check node resource usage
kubectl top nodes

# Describe a specific node (events, conditions)
kubectl describe node nebula-1

Pod Placement & Status

# List all pods across all namespaces with node assignment
kubectl get pods -A -o wide

# Check pod resource usage
kubectl top pods -A

# Get pod logs
kubectl logs -n heezy <pod-name>

# Port-forward to a pod (for local debugging)
kubectl port-forward -n heezy svc/<service-name> 8080:80

Force Pod Restart

Useful when a pod is stuck or needs cache refresh:

# Restart all replicas of a deployment
kubectl rollout restart deployment/<name> -n heezy

# Wait for rollout to complete
kubectl rollout status deployment/<name> -n heezy

# Example: restart heezy-finance
kubectl rollout restart deployment/heezy-finance -n heezy
kubectl rollout status deployment/heezy-finance -n heezy

Longhorn Volume Health

# List all volumes with health status
kubectl get volumes.longhorn.io -n longhorn-system

# Describe a specific volume (replicas, state)
kubectl describe volume <name> -n longhorn-system

# Example: check heezy-postgres volume
kubectl get volumes.longhorn.io -n longhorn-system | grep postgres

Gitea DB Sync (Stale Run Status)

If a Gitea workflow shows stale status (appears "cancelled" but steps completed), manually sync the database:

kubectl exec -n gitea gitea-postgres-0 -- psql -U gitea -d gitea -c \
  "UPDATE action_run_job SET status=3 WHERE status IN (1,2) AND task_id IN (SELECT id FROM action_task WHERE stopped>0);"

This updates all incomplete jobs with completed tasks to status 3 (completed).

Check Service IPs & Endpoints

# List services with MetalLB VIPs
kubectl get svc -A -o wide | grep LoadBalancer

# Check endpoint IPs for a service
kubectl get endpoints -n heezy <service-name>

Scale Deployment

# Set replica count
kubectl scale deployment/<name> -n heezy --replicas=2

# Or use kubectl edit for full control
kubectl edit deployment/<name> -n heezy

Monitoring & Observability

Prometheus: Scrapes metrics from all services (via Prometheus Operator ServiceMonitor CRDs)

Loki: Ingests logs from all containers (via DaemonSet log shipper)

Grafana: Visualizes metrics and logs. Admin: https://192.168.1.10:3000

See Capacity Review for current resource utilization.

Best Practices

  1. Always use kubeconfig: Never hardcode API endpoints
  2. Use namespaces: Separate concerns (heezy workloads vs. gitea vs. system)
  3. Set resource requests: Prevents BestEffort pods (backlog: fix 16 containers)
  4. Use Longhorn for stateful: All databases and config must be on Longhorn volumes
  5. No direct kubectl apply: Push to heezy-k8s repo and let CI/CD handle it (see CI/CD)
  6. Document affinity reasons: If pinning a pod, explain why in a comment
  7. Regular backups: Longhorn snapshots are enabled; test restoration quarterly