Backups¶

Backups work, but nothing runs them on a timer

Audited 2026-08-08, then re-verified the same day against a real run.

The role works. Run 667 completed ok=38 changed=19 failed=0 and wrote real archives for 22 of 24 targets — plex 12G, radarr 1.7G, receipts 1.6G, lidarr 745M, sonarr 435M, gitea-data 266M, slskd 152M, pihole 110M, uptime-kuma 75M, aurral 62M, navidrome 14M, tautulli 6.7M, prowlarr 4.9M, qbittorrent 4.1M, swag 3.9M, emulatorjs 2.4M, sabnzbd 333K, overseerr 249K, openbao 109K raft snapshot, lazylibrarian 45K, tailscale 3.7K.

But it only runs when someone pushes to roles/heezy-backup/**. playbook-heezy-backup-execution.yml has no schedule: block, so the "Daily 2am UTC" tier below has never run on a timer. Before run 667 the previous execution was 2026-07-28 — an 11 day gap with no backups at all, and that earlier run failed part way through, leaving the 23 k8s/<service>/ directories empty. That empty state is what the first pass of this audit mistook for a permanent defect.

Two things still need fixing:

Add the schedule. See Remediation Backlog item 1 — and read the generator note first, these workflow files are generated.
A volume backup cannot report failure. The type: volume branch ends with an echo, so the task's rc is always 0 regardless of what kubectl exec … tar did. Both the per-target Discord notify and the run summary key off rc, so neither can ever fire. A 0-byte tarball reports success — which is how SABnzbd's 45-byte S3 backup went unnoticed.

One restore is now proven, not assumed. OpenBao was restored from its raft snapshot on 2026-08-16, after a node failure left its Longhorn volume with an empty filesystem. Full recovery from that day's 12:50 snapshot, nothing lost. It is the only target whose restore path has been exercised against real data. Procedure: Restore OpenBao From a Raft Snapshot.

Not a defect, checked: heezy-finance archives at 105 bytes because its /data volume is genuinely empty. Its state lives in the heezy Postgres DB, which is dumped separately.

Added 2026-08-16: the network device step was a fourth instance of the same class of bug and was missed by this audit, because it reported success. It had never captured a single config. Fixed for Cisco and FortiGate, still open for Ruckus. See Network Devices.

Intended Design¶

Two-tier backup system driven by the heezy-backup Ansible role.

Tier	Schedule	Destination	Retention
Daily NFS	Daily 2am UTC (not wired up — see above)	`big-boi:/nfs/heezy/backups/`	7 days rolling
Monthly S3	1st of month 03:00 UTC	`s3://heezy-backups-025066240222/heezy/`	STANDARD_IA → 90 days, then expires

Backups run as a Gitea Actions workflow (playbook-heezy-backup-execution.yml) on the act-runner. Each target is an independent task — one failure does not stop others. Discord is notified per failure and on final summary.

What Actually Exists Today¶

Source	Covers	Cadence	Verified
`heezy-backup` Ansible role	22 of 24 targets, full list below	on push only — no timer	run 667, 2026-08-08
OpenBao raft snapshot	the whole barrier: secrets, auth methods, policies, AppRoles	with the role above	restore performed for real, 2026-08-16
`plex-config-backup` CronJob (k8s, 04:00)	`plex-config` PVC → `/nfs/heezy/backups/plex/`	daily, 3 kept	2026-08-08
`uptime-kuma-backup` CronJob (k8s, 03:00)	`uptime-kuma-data` → `/nfs/heezy/backups/uptime-kuma/`	daily, 3 kept	2026-08-08

The two empty target dirs after run 667 are nzbhydra2 (decommissioned 2026-08-08 and removed from the role, the directory is a leftover) and soularr (never in the target list — see Uncovered by oversight).

The practical risk is not that backups fail, it is that they only happen when someone edits the role. Between runs there is no fresh copy of anything, and nothing alerts on the gap.

Longhorn Has No Native Backup Layer¶

Separately from the Ansible role, Longhorn itself is configured with:

backup-target: empty — no S3 or NFS backup target is registered
recurringjobs.longhorn.io: none — no snapshot or backup schedules exist
backups.longhorn.io: none

So there is no volume-level snapshot to fall back on either. The 3-way replication protects against a single disk or node failure; it does not protect against deletion, corruption, or a bad write, because all three replicas take the change.

What's Backed Up¶

K8s PVCs (kubectl exec tar, heezy namespace)¶

Service	Path	Notes
overseerr	`/config/db`	Full `/config` causes `initialized: false` on restore
radarr	`/config`
sonarr	`/config`
lidarr	`/config`
prowlarr	`/config`
sabnzbd	`/config`
qbittorrent	`/config`	selector: `app=qbittorrent-vpn`, container: `qbittorrent`
tautulli	`/config`
navidrome	`/data`
pihole	`/etc/pihole`
uptime-kuma	`/app/data`
plex	`/config`	Large — takes several minutes
lazylibrarian	`/config`
aurral	`/app/backend/data`
swag	`/config`
slskd	`/app`	selector: `app=slskd-vpn`, container: `slskd`, excludes downloads
tailscale	`/var/lib/tailscale`
emulatorjs	`/config`
heezy-finance	`/data`
receipts	`/data/receipts`	Receipt images — not in postgres
openbao	raft snapshot	namespace: `openbao`. Restore tested end to end 2026-08-16. See Restore OpenBao From a Raft Snapshot

K8s PVCs (gitea namespace)¶

Service	Path	Notes
gitea-data	`/data`	Repos, attachments, LFS objects, app config

Postgres¶

Database	Method	Host
`heezy`	pg_dump	big-boi (192.168.1.21:5432), creds from OpenBao `production/heezy/postgres/heezy-credentials`
`gitea`	`kubectl exec pg_dump` into gitea-postgres pod	gitea namespace, creds from OpenBao `production/heezy/gitea/postgres`
`n8n`	`docker exec n8n-postgres pg_dump` via SSH	big-boi

Big-boi Docker Compose (SSH tar)¶

Service	Source Path
n8n data	`/opt/big-boi-ai/n8n/`
open-webui	`/opt/big-boi/open-webui/` (excludes cache)

Other¶

What	Method
Minecraft Bedrock	SSH tar to dmz-minecraft (192.168.3.13), stop/start around backup
Minecraft Java	SSH tar to dmz-minecraft-java (192.168.3.15)
Network configs	See Network Devices below. Cisco and FortiGate working as of 2026-08-16, Ruckus still not

Network Devices¶

Handled by backup-network.sh.j2 in the heezy-backup role, writing to /nfs/heezy/backups/network/.

Device	Model / OS	Method	State
Cisco core switch (192.168.1.26)	WS-C3750G-48PS-S, IOS 12.2(44)SE4	sshpass, interactive session, `show running-config`	Working, ~291 lines
FortiGate (192.168.1.1:8443)	FGT60E, FortiOS 7.4.3	REST API session cookie, `monitor/system/config/backup?scope=global`	Working, ~420 KB
Ruckus AP (192.168.2.66)	Unknown	Unverified session cookie	Not working

This step reported success while backing up nothing, from creation until 2026-08-16

/nfs/heezy/backups/network/ was empty from the day the directory was created (2026-07-16) until the fix landed. The step printed "All network backups complete." on every run. Three independent faults, each sufficient on its own:

The secrets were read from paths that do not exist. The role looked under production/heezy/network/{cisco,fortigate,ruckus}. All three returned 404. The credential facts rendered empty, so the template took its "not configured" branch, printed a skip notice, and exited 0.
The Cisco command could never have worked. It sent terminal length 0 ; show running-config as an ssh exec command. This IOS has no ; chaining, the service account lands at privilege 1 where show running-config is refused outright, and the stored enable secret was never read. BatchMode=yes in the shared SSH_OPTS also suppressed the very password prompt sshpass exists to answer.
The failure could not have been detected. Success was keyed off ssh's exit status, which is 0 even when IOS rejects every command it was given. The error text was written into the output file and reported with a checkmark and a line count.

Fixed in ansible-heezy PR #44. Missing Cisco or FortiGate credentials now count as an error so the Discord notify fires, rather than skipping silently.

Cisco specifics¶

The switch is a 2008-era image and needs three accommodations that are easy to mistake for bugs:

Privilege escalation is mandatory. The service account lands at privilege 1. show running-config is rejected there with Line has invalid autocommand, which looks like a syntax error but is an authorization failure. The enable secret is the third key in production/heezy/cisco-switch/secret.
No command chaining, so no ssh exec. The backup drives an interactive session and pipes commands to stdin.
Legacy crypto only. SHA-1 kex, ssh-rsa host key, CBC ciphers. Modern OpenSSH refuses all of these by default and needs them re-enabled per connection.

Output is filtered to the text between Building configuration and end, then validated for both anchors before the run is called a success. A wrong enable password now produces a 0-line capture and a reported failure.

Why the switch is backed up rather than managed as code¶

Every other device in the lab is declared in Terraform or Ansible. The switch is not, and cannot easily be:

No Terraform provider can reach it. CiscoDevNet/iosxe requires RESTCONF, which needs IOS-XE 16.8 or newer. This image has no RESTCONF and no NETCONF. Genuine Terraform management means replacing the hardware.
Most of cisco.ios will not work either. The resource modules (ios_vlans, ios_l2_interfaces, ios_interfaces) gather state with show running-config | section ..., and this IOS supports only include, exclude, and begin. | section returns % Invalid input detected. That leaves ios_config, which only needs plain show running-config and is genuinely idempotent, as the one usable path if config-as-code is ever wanted here.

So the switch is the one device where a hand-applied change is the normal case, which is exactly why a working config backup matters more here than anywhere else.

Ruckus is still not backed up¶

Blocker	Detail
No credentials	Nothing exists in OpenBao at any path. There is no `network/` tree at all
No reachable management interface	The AP (`ec:8c:a2:27:3b:f0`, DHCP-reserved at 192.168.2.66 on VLAN 200) is ARP-live but answers neither ICMP nor 443/80/8443 from the backup host. Not a firewall issue: policy 6 `allow-servers-to-users` permits SHARED to USERS for ALL
Unverified endpoints	The `_login` and `_system/backup` calls in the template are ZoneDirector-shaped and have never been exercised against this device. Unleashed uses a different scheme

To finish it: store username, password, and ip at secret/data/production/heezy/network/ruckus, confirm whether the device is Unleashed or a standalone AP, and check whether the web UI answers from a host on VLAN 200.

NFS Layout¶

/nfs/heezy/backups/
  k8s/
    <service>/          # one tar.gz per run
  gitea/
    gitea-postgres-<date>.sql.gz
  postgres/
    heezy/
    n8n/
  minecraft/
    bedrock/
    java/
  n8n/                  # n8n data dir tars
  open-webui/
  network/

What's Not Backed Up¶

Excluded by design¶

Excluded	Reason
NFS-backed media PVCs (movies, tv, music)	NFS share IS the storage
LGTM metrics (Mimir/Loki data)	~115 GB, regenerates from scrapers
ollama	Model weights are re-downloadable
heezy-blog / heezy-info	Git is the source of truth
cs16-leaderboard	Stateless, no PVC
openclaw	Low value, excluded by design. Runs as `openclaw-gateway` on big-boi, not in k8s — the empty `apps/openclaw` dir in heezy-k8s was removed 2026-08-08 as a stale deploy-trigger leftover
`gluetun-data`, `qbittorrent-gluetun-data`	VPN session state, regenerates on connect

Uncovered by oversight¶

These hold real state, have a Longhorn PVC, and are absent from the role's target list. Identified 2026-08-08; none were previously documented as gaps.

PVC	Namespace	Size	What is lost
`heezy-projects-data`	heezy	5Gi	heezy-projects application data
`heezy-maintenance-config`	heezy	1Gi	heezy-maintenance application data
`swag-plex-config`	heezy	5Gi	TLS certs + nginx config for the Plex reverse proxy (Let's Encrypt can reissue, but the config is hand-maintained)
`soularr-config`	heezy	5Gi	Previously documented as "ConfigMap only, no state" — that is wrong, a 5Gi Longhorn PVC exists. Currently detached

Known to be unprotected¶

Excluded	Reason
steam-headless	5.5 GB game saves on big-boi, not currently backed up
TrueNAS (192.168.1.200)	No backup coverage. NFS failure = loss of all NFS backup copies until the next S3 sync

Manual Trigger¶

Gitea UI: heezy-admin/ansible-heezy → Actions → playbook-heezy-backup-execution.yml → Run workflow

Via API — supply a Gitea API token from OpenBao as $GITEA_TOKEN (see Secrets & Access for the current path). Never inline the credential:

curl -sk -X POST \
  "http://192.168.1.15:30360/api/v1/repos/heezy-admin/ansible-heezy/actions/workflows/playbook-heezy-backup-execution.yml/dispatches" \
  -H "Authorization: token $GITEA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"ref":"main"}'

Verify Backup Health¶

# Recent files on NFS (last 2 days)
ssh mcp-admin@192.168.1.21 "find /nfs/heezy/backups -name '*.tar.gz' -o -name '*.sql.gz' -o -name '*.snap' | xargs ls -lt 2>/dev/null | head -30"

# S3 listing
aws s3 ls s3://heezy-backups-025066240222/heezy/ --recursive --human-readable | sort | tail -20

Restoring Gitea¶

Postgres restore¶

# Get creds from OpenBao
VAULT_TOKEN=$(curl -sk -X POST "http://192.168.1.32:8200/v1/auth/approle/login" \
  -H 'Content-Type: application/json' \
  -d '{"role_id":"<role_id>","secret_id":"<secret_id>"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")
CREDS=$(curl -sk "http://192.168.1.32:8200/v1/secret/data/production/heezy/gitea/postgres" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data']['data']; print(d['POSTGRES_USER'],d['POSTGRES_PASSWORD'],d['POSTGRES_DB'])")

# Scale down gitea, restore into gitea-postgres pod, scale back up
sudo microk8s kubectl scale deployment gitea -n gitea --replicas=0
POD=$(sudo microk8s kubectl get pod -n gitea -l app=gitea-postgres -o jsonpath='{.items[0].metadata.name}')
gunzip -c /nfs/heezy/backups/gitea/gitea-postgres-<date>.sql.gz \
  | sudo microk8s kubectl exec -i -n gitea "$POD" -- \
    env PGPASSWORD=<password> psql -U gitea gitea
sudo microk8s kubectl scale deployment gitea -n gitea --replicas=1

Data PVC restore¶

See the general K8s restore runbook. The gitea-data PVC is in the gitea namespace, label selector app=gitea.

Remediation Backlog¶

Ordered by severity, from the 2026-08-08 audit.

#	Gap	Action
1	Workflow has no `schedule:` trigger	The top priority. These workflow files are generated by `scripts/generate-workflows.py`, and the generator does emit `schedule: '0 14 * * 0'` for this playbook — the committed file has been hand-edited and lost it. Regenerating also puts several other playbooks onto schedules they do not currently run on, so review before applying. Note the generator's cadence is weekly, not the daily this page promises: pick one and make generator, workflow, and docs agree
2	~~The role writes no tarballs~~	Resolved — run 667 produced real archives for 22 of 24 targets. The earlier empty state was a partially failed 2026-07-28 run, not a permanent defect
3	Workflow cannot fail	Per-target isolation means a run where 100% of targets fail still exits 0. Add a final gate that exits non-zero if no new files landed on NFS
4	No verification that backups are restorable	Nothing checks tarball size or does a test extract. A 0-byte tarball would look identical to success
5	Longhorn has no `backup-target` and no recurring snapshot jobs	Register an S3 or NFS backup target and add a `recurringjobs.longhorn.io` snapshot schedule as a second line of defence
6	4 stateful PVCs are not in the target list	See Uncovered by oversight
7	Restore paths are undocumented-but-assumed	The Gitea restore procedure above references `/nfs/heezy/backups/gitea/gitea-postgres-<date>.sql.gz`, which does not exist. No restore procedure in this document has been tested against a real artifact
8	TrueNAS (192.168.1.200) has no backup coverage	NFS failure = loss of all NFS backup copies until the next S3 sync
9	Plex config PVC is large	Monitor that the backup completes within the 60-minute workflow timeout
10	~~Network backups silently captured nothing~~	Resolved 2026-08-16 in ansible-heezy PR #44. Cisco and FortiGate verified against the real devices. See Network Devices
11	Ruckus AP has no backup coverage	Needs credentials in OpenBao and a reachable management interface. See Ruckus is still not backed up

Item 10 is worth reading even though it is closed. It is the third instance on this page of the same defect: a step that cannot report its own failure (see also items 3 and 4, and the volume-backup echo in the warning at the top). Every one of them presented as a working backup for weeks or months.