Backups¶

Re-verified against NFS, the role defaults and the cluster on 2026-08-22.

One tool, two cadences. Restructured 2026-08-19

Everything is Ansible now. There are no backup CronJobs in heezy-k8s, no cron on big-boi, and no calendar gates buried in shell. Every capture and every upload is a task in a role under ansible-heezy, triggered by a generated Gitea Actions workflow. The only mechanism-specific case left is the OpenBao raft snapshot, which has to be an API call because a tar of a live raft store is not a restorable snapshot.

Two plays, deliberately split. Capture cadence and shipping cadence answer different questions, and fusing them meant the only lever for uploading less was capturing less:

heezy-backup.yml captures everything to NFS. Daily, 02:00 UTC.
heezy-backup-offsite.yml ships one generation to S3. Weekly, Saturday 08:00 UTC.

Plex is no longer archived whole. The old target tarred all of /config, 11.9 GiB, of which Media, Metadata and Cache are regenerable artwork and transcode scratch. It was 60% of every upload. The target is now the three paths that carry state a restore cannot recreate, ~1.6 GiB, which is what the retired plex-config-backup CronJob had been capturing separately all along.

One restore is proven, not assumed. OpenBao was restored from its raft snapshot on 2026-08-16, after a node failure left its Longhorn volume with an empty filesystem. Full recovery from that day's 12:50 snapshot, nothing lost. It is the only target whose restore path has been exercised against real data. Procedure: Restore OpenBao From a Raft Snapshot.

Not a defect, checked: heezy-finance archives at 105 bytes because its /data volume is genuinely empty. Its state lives in the heezy Postgres DB, which is dumped separately.

Design¶

Tier	Schedule	Destination	Retention	Enforced by
Local	Daily 02:00 UTC	`big-boi:/nfs/heezy/backups/`	7 days rolling	`heezy_backup_retention_days`, `heezy-backup` role
Off site	Saturday 08:00 UTC	`s3://heezy-backups-025066240222/heezy/`	30 days, STANDARD_IA	S3 lifecycle, `terraform-heezy/environments/production/aws/backup.tf`

Both run as generated Gitea Actions workflows on the act-runner. Each target is an independent task, so one failure does not stop the others, and each play ends with a gate that fails the run if any target failed. Discord is notified per failure and on final summary.

Why Saturday 08:00 UTC¶

The ask was 3am to 5am Eastern. Cron in Gitea Actions has no timezone, so the slot has to hold in both halves of the year: 08:00 UTC is 03:00 EST in winter and 04:00 EDT in summer, inside the window either way.

Why one generation and not the whole tree¶

NFS holds 7 daily snapshots of every target. aws s3 sync over the tree pushes all 7 on each run. A weekly off-site RPO needs 1, so the ship script groups files by series and uploads only the newest member of each. A series is the directory plus the filename with its date stamp removed; keying on the directory alone would be wrong for network/, which holds cisco-core, fortigate and ruckus side by side.

The retention constraint¶

Local retention must stay at or above the shipping interval. At 7 days local and a 7 day interval, every snapshot is still on disk when the weekly run comes for it. Lengthen the interval without lengthening local retention and snapshots are pruned before they are ever shipped. This is not hypothetical: the S3 step used to be gated on [ "$(date +%d)" = "01" ], the daily schedule did not exist, and nothing reached S3 for 67 days.

Why STANDARD_IA and not GLACIER¶

GLACIER Flexible Retrieval bills a 90 day minimum per object. Expiring at 30 days does not stop that clock, so a 30 day GLACIER tree is billed as a 90 day one: roughly 3x the bytes actually kept, which lands within pennies of STANDARD_IA at this size. STANDARD_IA's minimum is exactly 30 days, so the expiration sits on the boundary and pays no early deletion fee. Same money, and restores go from 3-5 hours to immediate. That matters most for the OpenBao raft snapshot, which every other restore depends on.

Retired¶

What	Replaced by	When
`plex-config-backup` CronJob (heezy-k8s, 04:00)	the `plex` target in `heezy_backup_k8s`, same three paths	2026-08-19
`uptime-kuma-backup` CronJob (heezy-k8s, 03:00)	the `uptime-kuma` target, which already tarred `/app/data` including `kuma.db`	2026-08-19
Daily `aws s3 sync` inside `heezy-backup`	`heezy-backup-offsite.yml`, weekly	2026-08-19
Full `/config` Plex tar, 11.9 GiB	three-path Plex tar, ~1.6 GiB	2026-08-19

nzbhydra2 was decommissioned 2026-08-08 and removed from the role; its directory is a leftover. soularr was never in the target list and never should be, see Uncovered by oversight — now closed.

Longhorn Has No Native Backup Layer¶

Separately from the Ansible role, Longhorn itself is configured with:

backup-target: empty — no S3 or NFS backup target is registered
recurringjobs.longhorn.io: one, snapshot-cleanup (Sundays 05:00). It deletes system-generated snapshots and purges the frozen layers left behind by replica rebuilds. It takes no snapshots
backups.longhorn.io: none

So there is no volume-level snapshot to fall back on either. The 3-way replication protects against a single disk or node failure; it does not protect against deletion, corruption, or a bad write, because all three replicas take the change.

What's Backed Up¶

K8s PVCs (kubectl exec tar, heezy namespace)¶

Service	Path	Notes
overseerr	`/config/db`	Full `/config` causes `initialized: false` on restore
radarr	`/config`
sonarr	`/config`
lidarr	`/config`
prowlarr	`/config`
sabnzbd	`/config`
qbittorrent	`/config`	selector: `app=qbittorrent-vpn`, container: `qbittorrent`
tautulli	`/config`
navidrome	`/data`
pihole	`/etc/pihole`
uptime-kuma	`/app/data`
plex	`Preferences.xml`, `Plug-in Support/Databases`, `Plug-in Support/Metadata Combination` under `/config/Library/Application Support/Plex Media Server`	Not full `/config`. `Media`, `Metadata` and `Cache` are regenerable and were 11.9 GiB of the 12 GiB archive
lazylibrarian	`/config`
aurral	`/app/backend/data`
swag	`/config`
slskd	`/app`	selector: `app=slskd-vpn`, container: `slskd`, excludes downloads
tailscale	`/var/lib/tailscale`
emulatorjs	`/config`
heezy-finance	`/data`
receipts	`/data/receipts`	Receipt images — not in postgres
openbao	raft snapshot	namespace: `openbao`. Restore tested end to end 2026-08-16. See Restore OpenBao From a Raft Snapshot

K8s PVCs (gitea namespace)¶

Service	Path	Notes
gitea-data	`/data`	Repos, attachments, LFS objects, app config

Postgres¶

Database	Method	Host
`heezy`	pg_dump	`postgres.heezy.local:5432`, creds from OpenBao `production/heezy/postgres/heezy-credentials`
`gitea`	`kubectl exec pg_dump` into gitea-postgres pod	gitea namespace, creds from OpenBao `production/heezy/gitea/postgres`. Verified: `gitea/gitea-postgres-20260822_020024.sql.gz`, 5.6 MB

The n8n database no longer exists. n8n was retired in August 2026 and its database was dropped; the container on big-boi kept the name n8n-postgres but now serves heezy, maintenance and projects. There is no n8n dump target in the role and /nfs/heezy/backups/n8n/ is a leftover directory.

Big-boi Docker Compose (SSH tar)¶

heezy_backup_compose, heezy_backup_remote_configs and heezy_backup_app_data are all empty lists. The only big-boi target is a dedicated task:

Service	Source Path	Latest archive
open-webui	`/opt/big-boi/open-webui/` (excludes cache)	8.6 KB, 2026-08-22

8.6 KB is small enough to be worth a look. Either the chat history lives somewhere the tar does not reach, or the exclude is too broad.

Game servers¶

Handled by the game-backup role, dispatched or scheduled by playbook-game-backup-execution.yml at 01:00 UTC daily. Six archives, one per world.

Archive	Volume	Quiesced
`games/cs16/`	`cs16-data`	No
`games/minecraft/bedrock/`	`bedrock-data`, both worlds in one archive	No
`games/ut2k4/`	`ut2k4-data`	No
`games/minecraft/java/`	`minecraft-java-data`	Yes
`games/minecraft/adventure/`	`minecraft-adventure-data`	Yes
`games/minecraft/cobblemon/`	`minecraft-cobblemon-data`	Yes

It reads each volume out through the container that holds it, not by mounting the volume. The volumes are ReadWriteOnce, so a backup job cannot mount one while the server has it. Every game container carries a read-only mount of its whole volume at /backup-src for this, which is also what makes the archive match the volume root, the shape the restore extracts.

It runs on a cluster node rather than the CI runner. The runner has neither cluster access nor /nfs, and a game backup written somewhere that merely looks like the destination reports success and is thrown away with the job container.

Quiesced worlds get save-off and save-all flush before the read and save-on after, whether the read worked or not. Only the Java images ship rcon-cli.

Three checks, and the second is the one that matters:

an absolute floor of 1 MB, because an archive of a few hundred bytes is tar writing an error into a file and exiting happy
a relative floor against the previous archive. Half of last night's size is not a smaller world, it is a truncated read, and that is the failure that looks most like success
that the share is actually mounted, before a byte is written

A failing archive is deleted rather than kept and counted. Failures are collected across all six and raised at the end, so one broken game does not hide the state of the other five. Retention is seven archives per world.

Other¶

What	Method
Network configs	See Network Devices below. Cisco and FortiGate working as of 2026-08-16, Ruckus still not

Network Devices¶

Handled by backup-network.sh.j2 in the heezy-backup role, writing to /nfs/heezy/backups/network/.

Device	Model / OS	Method	State
Cisco core switch (192.168.1.26)	WS-C3750G-48PS-S, IOS 12.2(44)SE4	sshpass, interactive session, `show running-config`	Working. 2026-08-22 capture: 5.5 KB
FortiGate (192.168.1.1:8443)	FGT60E, FortiOS 7.4.3	REST API session cookie, `monitor/system/config/backup?scope=global`	Working. 2026-08-22 capture: 427 KB
Ruckus AP (192.168.2.66)	Unleashed, master AP	Unverified session cookie	Not working — config export only. SNMP metrics do work

This step reported success while backing up nothing, from creation until 2026-08-16

/nfs/heezy/backups/network/ was empty from the day the directory was created (2026-07-16) until the fix landed. The step printed "All network backups complete." on every run. Three independent faults, each sufficient on its own:

The secrets were read from paths that do not exist. The role looked under production/heezy/network/{cisco,fortigate,ruckus}. All three returned 404. The credential facts rendered empty, so the template took its "not configured" branch, printed a skip notice, and exited 0.
The Cisco command could never have worked. It sent terminal length 0 ; show running-config as an ssh exec command. This IOS has no ; chaining, the service account lands at privilege 1 where show running-config is refused outright, and the stored enable secret was never read. BatchMode=yes in the shared SSH_OPTS also suppressed the very password prompt sshpass exists to answer.
The failure could not have been detected. Success was keyed off ssh's exit status, which is 0 even when IOS rejects every command it was given. The error text was written into the output file and reported with a checkmark and a line count.

Fixed in ansible-heezy PR #44. Missing Cisco or FortiGate credentials now count as an error so the Discord notify fires, rather than skipping silently.

Cisco specifics¶

The switch is a 2008-era image and needs three accommodations that are easy to mistake for bugs:

Privilege escalation is mandatory. The service account lands at privilege 1. show running-config is rejected there with Line has invalid autocommand, which looks like a syntax error but is an authorization failure. The enable secret is the third key in production/heezy/cisco-switch/secret.
No command chaining, so no ssh exec. The backup drives an interactive session and pipes commands to stdin.
Legacy crypto only. SHA-1 kex, ssh-rsa host key, CBC ciphers. Modern OpenSSH refuses all of these by default and needs them re-enabled per connection.

Output is filtered to the text between Building configuration and end, then validated for both anchors before the run is called a success. A wrong enable password now produces a 0-line capture and a reported failure.

Why the switch is backed up rather than managed as code¶

Every other device in the lab is declared in Terraform or Ansible. The switch is not, and cannot easily be:

No Terraform provider can reach it. CiscoDevNet/iosxe requires RESTCONF, which needs IOS-XE 16.8 or newer. This image has no RESTCONF and no NETCONF. Genuine Terraform management means replacing the hardware.
Most of cisco.ios will not work either. The resource modules (ios_vlans, ios_l2_interfaces, ios_interfaces) gather state with show running-config | section ..., and this IOS supports only include, exclude, and begin. | section returns % Invalid input detected. That leaves ios_config, which only needs plain show running-config and is genuinely idempotent, as the one usable path if config-as-code is ever wanted here.

So the switch is the one device where a hand-applied change is the normal case, which is exactly why a working config backup matters more here than anywhere else.

Ruckus is still not backed up¶

Blocker	Detail
No credentials	Nothing exists in OpenBao at any path. There is no `network/` tree at all
Management interface, partly	The AP (`ec:8c:a2:27:3b:f0`, DHCP-reserved at 192.168.2.66 on VLAN 200) does answer SNMP on 161 — Prometheus has polled `ruckus-ap.heezy.local` as an `if_mib` target since 2026-08-20 and the target is `up`. It still answers neither ICMP nor 443/80/8443 from the backup host, so HTTP config export remains blocked. Not a firewall issue: policy 6 `allow-servers-to-users` permits SHARED to USERS for ALL
Unverified endpoints	The `_login` and `_system/backup` calls in the template are ZoneDirector-shaped and have never been exercised against this device. Unleashed uses a different scheme

To finish it: store username, password, and ip at secret/data/production/heezy/network/ruckus, and check whether the web UI answers from a host on VLAN 200. The lgtm role treats it as an Unleashed master (only the master runs an SNMP agent; ruckus-ap-member at 192.168.2.45 does not answer 161 and is deliberately not polled), so the _login / _system/backup calls in the template need rewriting for Unleashed rather than ZoneDirector.

Both APs have had heezy.local forward records since 2026-08-20.

NFS Layout¶

/nfs/heezy/backups/
  k8s/
    <service>/          # one tar.gz per run, 25 service dirs
  gitea/
    gitea-postgres-<date>.sql.gz
  postgres/
    heezy-<date>.sql.gz
  games/
    cs16/
    ut2k4/
    minecraft/
      bedrock/
      java/
      adventure/
      cobblemon/
  network/
    cisco-core-<date>.txt
    fortigate-<date>.conf
  open-webui/

Six directories on NFS are no longer written to by any target and can be deleted: n8n/, openclaw/, lgtm/, app-data/, docker-compose/, and the old top level minecraft/, which the game backup replaced with games/minecraft/. Inside k8s/, nzbhydra2/ and soularr/ are empty leftovers.

What's Not Backed Up¶

Excluded by design¶

Excluded	Reason
NFS-backed media PVCs (movies, tv, music)	NFS share IS the storage
LGTM metrics (Mimir/Loki data)	~115 GB, regenerates from scrapers
ollama	Model weights are re-downloadable
heezy-blog / heezy-info	Git is the source of truth
cs16-leaderboard	Stateless, no PVC
openclaw	Low value, excluded by design. Runs as `openclaw-gateway` on big-boi, not in k8s — the empty `apps/openclaw` dir in heezy-k8s was removed 2026-08-08 as a stale deploy-trigger leftover
`gluetun-data`, `qbittorrent-gluetun-data`	VPN session state, regenerates on connect

Uncovered by oversight — now closed¶

The three gaps identified on 2026-08-08 were added to the target list and are capturing. Verified on NFS 2026-08-22:

PVC	Namespace	Latest archive
`heezy-projects-data`	heezy	`k8s/heezy-projects/…-20260822_021210.tar.gz`, 1.4 MB
`heezy-maintenance-config`	heezy	`k8s/heezy-maintenance/…-20260822_021211.tar.gz`, 12.5 MB
`swag-plex-config`	heezy	`k8s/swag-plex/…-20260822_021214.tar.gz`, 697 KB

soularr-config is deliberately excluded, not an oversight. soularr mounts a ConfigMap of the same name at /data/config.ini plus the nfs-slskd-downloads PVC. Nothing mounts the 5Gi Longhorn PVC, which is why it sits detached. Delete the PVC rather than back it up.

Two targets produced nothing in the 2026-08-22 capture

Every other target has an archive stamped 20260822_02xxxx. tailscale and heezy-finance stop at 20260820_125938 — two days old. Both pods are Running and both match their label selectors (app=tailscale, app=heezy-finance), so the selector is not the problem.

This is the same shape as the network-backup defect below: the run reported success while two targets silently produced nothing. The per-target failure gate catches a task that errors; it does not catch one that finds no pod and exits 0.

Known to be unprotected¶

Excluded	Reason
steam-headless	5.5 GB game saves on big-boi, not currently backed up
TrueNAS (192.168.1.200)	No backup coverage. NFS failure loses every local copy, leaving only the last weekly S3 generation. Worst-case exposure is now 7 days rather than 1 — the cost of moving the ship to weekly, and the reason local retention must not drop below the ship interval

Manual Trigger¶

Gitea UI: heezy-admin/ansible-heezy → Actions → pick the workflow → Run workflow.

Want	Workflow
Capture everything to NFS now	`playbook-heezy-backup-execution.yml`
Ship the newest generation to S3 now	`playbook-heezy-backup-offsite-execution.yml`

The off-site play reads NFS and never captures, so running it alone ships whatever the last daily capture left behind. To force a fresh off-site copy, run the capture first, then the ship.

Via API — supply a Gitea API token from OpenBao as $GITEA_TOKEN (see Secrets & Access for the current path). Never inline the credential:

curl -sk -X POST \
  "http://gitea.heezy.local:3000/api/v1/repos/heezy-admin/ansible-heezy/actions/workflows/playbook-heezy-backup-offsite-execution.yml/dispatches" \
  -H "Authorization: token $GITEA_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"ref":"main"}'

Verify Backup Health¶

# Recent files on NFS (last 2 days)
ssh mcp-admin@192.168.1.21 "find /nfs/heezy/backups -name '*.tar.gz' -o -name '*.sql.gz' -o -name '*.snap' | xargs ls -lt 2>/dev/null | head -30"

# One line per series on NFS, newest first. This is what the off-site run ships.
ssh mcp-admin@192.168.1.21 "cd /nfs/heezy/backups && sudo find . -mindepth 2 -type f -printf '%p\n' | sed -E 's/-[0-9]{8}_[0-9]{6}//' | sort | uniq -c"

# S3, newest first. Expect ~33 objects per weekly generation, four generations deep.
aws s3 ls s3://heezy-backups-025066240222/heezy/ --recursive --human-readable | sort | tail -40

Restoring Gitea¶

Postgres restore¶

# Get creds from OpenBao
VAULT_TOKEN=$(curl -sk -X POST "http://openbao.heezy.local:8200/v1/auth/approle/login" \
  -H 'Content-Type: application/json' \
  -d '{"role_id":"<role_id>","secret_id":"<secret_id>"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")
CREDS=$(curl -sk "http://openbao.heezy.local:8200/v1/secret/data/production/heezy/gitea/postgres" \
  -H "X-Vault-Token: $VAULT_TOKEN" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data']['data']; print(d['POSTGRES_USER'],d['POSTGRES_PASSWORD'],d['POSTGRES_DB'])")

# Scale down gitea, restore into gitea-postgres pod, scale back up
sudo microk8s kubectl scale deployment gitea -n gitea --replicas=0
POD=$(sudo microk8s kubectl get pod -n gitea -l app=gitea-postgres -o jsonpath='{.items[0].metadata.name}')
gunzip -c /nfs/heezy/backups/gitea/gitea-postgres-<date>.sql.gz \
  | sudo microk8s kubectl exec -i -n gitea "$POD" -- \
    env PGPASSWORD=<password> psql -U gitea gitea
sudo microk8s kubectl scale deployment gitea -n gitea --replicas=1

Data PVC restore¶

See the general K8s restore runbook. The gitea-data PVC is in the gitea namespace, label selector app=gitea.

Remediation Backlog¶

Ordered by severity, from the 2026-08-08 audit.

#	Gap	Action
1	~~Workflow has no `schedule:` trigger~~	Resolved. `SCHEDULES` in `scripts/generate-workflows.py` now carries `heezy-backup` at `0 2 * * ` and `heezy-backup-offsite` at `0 8 * 6`. Both workflow files are generated; edit the dict, never the YAML
2	~~The role writes no tarballs~~	Resolved — run 667 produced real archives for 22 of 24 targets. The earlier empty state was a partially failed 2026-07-28 run, not a permanent defect
3	~~Workflow cannot fail~~	Resolved. `tasks/main.yml` ends with a `fail:` gated on `_failed_steps \\| length > 0`, so a run with any errored target exits non-zero. It does not catch a target that finds no pod and exits 0 — see the tailscale/heezy-finance note above
4	Partially resolved: size floor, no test extract	Each volume target now enforces `MIN_BYTES` (default 200, `min_bytes: 0` for heezy-finance), so a 0-byte tarball is rejected. Nothing does a test extract, so a truncated-but-nonzero archive still looks like success
5	Longhorn has no `backup-target`	Still empty as of 2026-08-22. There is now one recurring job, `snapshot-cleanup` (Sundays 05:00), but it deletes snapshots rather than taking them. Register an S3 or NFS backup target and add a snapshot schedule as a second line of defence
6	~~4 stateful PVCs are not in the target list~~	Resolved. heezy-projects, heezy-maintenance and swag-plex are captured and verified on NFS 2026-08-22. soularr-config is deliberately excluded as an orphaned PVC. See Uncovered by oversight — now closed
7	Restore paths are documented but untested	`/nfs/heezy/backups/gitea/gitea-postgres-<date>.sql.gz` does exist now — 5.6 MB, dated 2026-08-22. The artifact the procedure needs is there; the procedure itself has still never been run. OpenBao remains the only restore proven against real data
8	TrueNAS (192.168.1.200) has no backup coverage	NFS failure = loss of all NFS backup copies until the next S3 sync
9	~~Plex config PVC is large~~	Resolved 2026-08-19. The target is now three paths instead of all of `/config`: 11.9 GiB to ~1.6 GiB. `Media`, `Metadata` and `Cache` are regenerable and were never worth archiving
10	~~Network backups silently captured nothing~~	Resolved 2026-08-16 in ansible-heezy PR #44. Cisco and FortiGate verified against the real devices. See Network Devices
11	Ruckus AP has no config backup	Needs credentials in OpenBao and an Unleashed-shaped export call. SNMP metrics now work, so the device is reachable; the HTTP management interface is not. See Ruckus is still not backed up
12	Two targets silently captured nothing on 2026-08-22	`tailscale` and `heezy-finance` last produced an archive on 2026-08-20 while every other target ran. The run reported success. Same class of defect as items 3, 4 and 10

Item 10 is worth reading even though it is closed. It is the third instance on this page of the same defect: a step that cannot report its own failure (see also items 3 and 4, and the volume-backup echo in the warning at the top). Every one of them presented as a working backup for weeks or months.