Skip to content

Storage Architecture

Last verified against the live cluster: 2026-08-22.

Storage Classes

StorageClass Provisioner Reclaim Default Use Case
longhorn driver.longhorn.io Delete yes App config, databases — 3 replicas
nfs-media nfs.csi.k8s.io Retain no Shared data, media, receipts, downloads
microk8s-hostpath microk8s.io/hostpath Delete no MicroK8s addon default, unused
rook-ceph-block rook-ceph.rbd.csi.ceph.com Delete yes Orphan — see below

Two StorageClasses are marked default, and one of them has no backend

rook-ceph-block was created 2026-08-18 with storageclass.kubernetes.io/is-default-class: "true". There is no rook-ceph namespace, no Rook operator, and no Ceph pods — nothing implements that provisioner. Because it is the newer of the two defaults, the DefaultStorageClass admission plugin will pick it for any PVC created without an explicit storageClassName, and that PVC will sit Pending forever with no provisioner to answer it.

Nothing is currently bound to it, so nothing is broken yet. The fix is to drop the default annotation from rook-ceph-block (or delete the StorageClass) so longhorn is the only default.

kubectl get sc
kubectl patch sc rook-ceph-block -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"false"}}}'

Until that happens, always set storageClassName explicitly on new PVCs.

Longhorn

Distributed block storage across all five nebula nodes, disk path /var/lib/longhorn/.

  • Replication: 3 replicas per volume (default-replica-count: 3)
  • Access mode: mostly RWO; a few volumes are RWX
  • Use case: all stateful app config (sonarr, radarr, plex, gitea, openbao, …)
  • Capacity: ~935 GiB usable per node, ~4,676 GiB total, of which ~608 GiB is scheduled (13%). ~29 GiB per node is held back as reserved space.

Health check:

kubectl get volumes.longhorn.io -n longhorn-system

All attached volumes should show robustness: healthy.

Replicas are unevenly spread

Scheduled bytes per node: nebula-3 ~195 GiB, nebula-1 ~181 GiB, nebula-4 ~145 GiB, nebula-2 ~68 GiB, nebula-5 ~19 GiB. The nebula-5 reboot on 2026-08-06 rebuilt nearly every volume onto the other nodes and nothing rebalanced afterwards. Longhorn does not rebalance on its own unless replica-auto-balance is enabled. There is plenty of headroom, so this is a resilience observation rather than a capacity problem.

Snapshots are not backups, and there is no backup target

backup-target is empty. The only recurring job is snapshot-cleanup (Sundays 05:00), which deletes system-generated snapshots left over from replica rebuilds and purges the frozen layers. Snapshots live on the same disks as the data. See Backups.

NFS (TrueNAS)

  • Server: 192.168.1.200
  • Base path: /mnt/Arr1/SMB/, mounted on Linux hosts as /nfs
  • Dynamic provisioning: nfs.csi.k8s.io (csi-driver-nfs), StorageClass nfs-media, reclaim policy Retain — deleting a PVC leaves the data behind
  • Static PVs: movies, tv, anime, music, audiobooks, roms, plus heezy-ingest and slskd-downloads, defined in heezy-k8s/base/nfs-media-static-pvs.yaml
  • Access mode: RWX, shared across all pods and nodes
  • Utilisation: 35 TB used of 48 TB (73%)

NFS PV manifests require spec.nfs.server to be an IP address — DNS names are not supported by the driver. The raw 192.168.1.200 in PV manifests is intentional and immutable after creation.

Current Storage Patterns

Service Storage Access Mode Notes
Media apps (sonarr, radarr, …) Longhorn PVC RWO Config only — media files on NFS
Media files NFS static PVs RWX movies, tv, music, anime, audiobooks, roms
Receipts images NFS dynamic PVC (nfs-receipts, 100Gi) RWX /mnt/Arr1/SMB/receipts/
heezy-finance heezy-finance-config Longhorn PVC, 2Gi RWO Transactional data is in Postgres; the PVC holds app config
heezy-maintenance heezy-maintenance-config Longhorn PVC, 1Gi RWX
heezy-projects heezy-projects-data Longhorn PVC, 5Gi RWO
Gitea gitea-data 20Gi + gitea-postgres-data 5Gi RWO gitea namespace
act-runner act-runner-data 5Gi RWX NFS-backed
OpenBao data-openbao-0 5Gi RWO openbao namespace, Raft storage
Plex plex-config 50Gi RWO The largest Longhorn volume
Postgres (big-boi) Host bind mount — /opt/big-boi-ai/postgres on big-boi, outside k8s

VPN Sidecar Storage Note

soularr sets a podAffinity requiring it to schedule on the same node as slskd-vpn, because it shares that pod's RWO PVC. It is the only workload in the cluster with an affinity rule.

qbittorrent-vpn and slskd-vpn each hold their own RWO PVCs and are single-replica, so the constraint is implicit rather than declared.

Recommendations

  • Set storageClassName explicitly on every new PVC until the duplicate default is resolved
  • Never use nodeSelector to work around RWO PVC constraints — convert to NFS RWX or move state into Postgres
  • Longhorn has plenty of headroom — 13% scheduled of ~4.6 TB
  • NFS is at 73% (35 TB / 48 TB) — worth an alert at 85%, since a full media volume breaks downloads and Plex scanning at the same time