Backup & Restore¶

Overview¶

Heezy uses two backup tiers:

Tier	Schedule	Destination	Retention
Daily NFS	Daily 2am UTC	`big-boi:/nfs/heezy/backups/`	7 days
Monthly S3	1st of month 03:00 UTC	`s3://heezy-backups-025066240222/`	STANDARD_IA for 90 days, then GLACIER_IR, expire at 365 days

Backups are driven by the heezy-backup Ansible role. The NFS copies are preferred for restores — they're immediately accessible with no retrieval wait.

Covered Services¶

All services in the heezy namespace are backed up daily:

Service	Container Path	Notes
overseerr	`/config/db`	Full `/config` causes `initialized: false` on restore
radarr	`/config`
sonarr	`/config`
lidarr	`/config`
prowlarr	`/config`
sabnzbd	`/config`
qbittorrent	`/config`	selector: `app=qbittorrent-vpn`, container: `qbittorrent`
tautulli	`/config`
navidrome	`/data`
pihole	`/etc/pihole`
uptime-kuma	`/app/data`
plex	`/config`	Large PVC (~50Gi) — backup takes several minutes
lazylibrarian	`/config`
aurral	`/app/backend/data`
swag	`/config`
slskd	`/app`	selector: `app=slskd-vpn`, container: `slskd`, excludes downloads
tailscale	`/var/lib/tailscale`
emulatorjs	`/config`
heezy-finance	`/data`
openbao	raft snapshot	namespace: `openbao`

Also backed up:

What	Destination	Notes
Postgres `heezy` DB	`/nfs/heezy/backups/postgres/heezy/`	pg_dump via SSH to big-boi
Postgres `n8n` DB	`/nfs/heezy/backups/n8n/`	`docker exec n8n-postgres pg_dump` via SSH
n8n data dir	`/nfs/heezy/backups/n8n/`	tar of `/opt/big-boi-ai/n8n/`
open-webui	`/nfs/heezy/backups/open-webui/`	tar of `/opt/big-boi/open-webui/` (excludes cache)
Minecraft Bedrock	`/nfs/heezy/backups/minecraft/bedrock/`	SSH to dmz-minecraft (192.168.3.13)
Minecraft Java	`/nfs/heezy/backups/minecraft/java/`	SSH to dmz-minecraft-java (192.168.3.15)
Network configs	`/nfs/heezy/backups/network/`	FortiGate REST, Cisco sshpass, Ruckus session cookie

Restore from NFS (preferred)¶

NFS backups are at big-boi:/nfs/heezy/backups/. No retrieval wait.

Postgres (any DB)¶

ls -lt /nfs/heezy/backups/postgres/heezy/

gunzip -c /nfs/heezy/backups/postgres/heezy/heezy-YYYYMMDD_HHMMSS.sql.gz \
  | PGPASSWORD=<password> psql -h 192.168.1.21 -U heezy_app heezy

OpenBao Raft snapshot¶

Use the dedicated runbook: Restore OpenBao From a Raft Snapshot.

The two commands below only work when OpenBao is already initialized and unsealed, which is the easy case. The case you will actually hit is an uninitialized instance, where snapshot restore has nothing to authenticate against and this fails. There is also a token reviewer JWT that has to be replaced afterwards or every ExternalSecret returns 403. Both are covered in the runbook, which was walked end to end against real data on 2026-08-16.

ls -lt /nfs/heezy/backups/k8s/openbao/

sudo microk8s kubectl cp \
  /nfs/heezy/backups/k8s/openbao/openbao-TIMESTAMP.snap \
  openbao/openbao-0:/tmp/openbao.snap

sudo microk8s kubectl exec -n openbao openbao-0 -- \
  vault operator raft snapshot restore /tmp/openbao.snap

K8s PVC volume¶

Full step-by-step process:

1. Scale down the service:

sudo microk8s kubectl scale deployment/<service> -n heezy --replicas=0

2. Find which node the PVC is bound to:

sudo microk8s kubectl get pods -n heezy -l app=<service> -o wide
# Note the NODE column before scaling down, or check:
sudo microk8s kubectl get pvc -n heezy | grep <service>

3. Spin up a restore pod (use base64 to avoid SSH quoting issues):

Generate the manifest and base64-encode it locally:

cat <<'MANIFEST' | base64 -w0
apiVersion: v1
kind: Pod
metadata:
  name: <service>-restore
  namespace: heezy
spec:
  nodeSelector:
    kubernetes.io/hostname: <node-where-pvc-lives>
  restartPolicy: Never
  volumes:
  - name: config
    persistentVolumeClaim:
      claimName: <service>-config
  containers:
  - name: restore
    image: alpine
    command: ["sleep", "3600"]
    volumeMounts:
    - name: config
      mountPath: /config
MANIFEST

Apply on the node:

echo <base64> | base64 -d > /tmp/restore-pod.yaml
sudo microk8s kubectl apply -f /tmp/restore-pod.yaml
sudo microk8s kubectl wait pod <service>-restore -n heezy --for=condition=Ready --timeout=60s

4. Download backup from NFS, copy into the pod:

# SCP from big-boi to the nebula node
scp mcp-admin@192.168.1.21:/nfs/heezy/backups/k8s/<service>/<service>-TIMESTAMP.tar.gz /tmp/

# Copy into the restore pod
sudo microk8s kubectl cp /tmp/<service>-TIMESTAMP.tar.gz heezy/<service>-restore:/tmp/backup.tar.gz

5. Extract into the PVC:

sudo microk8s kubectl exec -n heezy <service>-restore -- sh -c "
  cp -r /config /config.bak.$(date +%Y%m%d) &&
  tar -xzf /tmp/backup.tar.gz -C /config --strip-components=1
"

For SQLite databases (overseerr, etc.) — remove WAL/SHM before scaling up:

sudo microk8s kubectl exec -n heezy <service>-restore -- sh -c "
  rm -f /config/db/db.sqlite3-shm /config/db/db.sqlite3-wal
"

6. Delete restore pod and scale back up:

sudo microk8s kubectl delete pod <service>-restore -n heezy
sudo microk8s kubectl scale deployment/<service> -n heezy --replicas=1
sudo microk8s kubectl wait --for=condition=ready pod -l app=<service> -n heezy --timeout=60s
sudo microk8s kubectl logs -n heezy -l app=<service> --tail=20

Overseerr — `initialized: false` fix¶

If overseerr shows the setup wizard after restore, settings.json has initialized: false. Fix via the restore pod before scaling up:

# Copy settings.json out
sudo microk8s kubectl cp heezy/overseerr-restore:/config/settings.json /tmp/overseerr-settings.json
scp mcp-admin@192.168.1.15:/tmp/overseerr-settings.json /tmp/

# Fix the flag
python3 -c "
import json
with open('/tmp/overseerr-settings.json') as f: s = json.load(f)
s['public']['initialized'] = True
with open('/tmp/overseerr-settings-fixed.json', 'w') as f: json.dump(s, f, indent=1)
"

# Push back
scp /tmp/overseerr-settings-fixed.json mcp-admin@192.168.1.15:/tmp/
sudo microk8s kubectl cp /tmp/overseerr-settings-fixed.json heezy/overseerr-restore:/config/settings.json

Then delete the restore pod and scale back up.

Docker Compose (big-boi)¶

n8n data dir¶

ssh mcp-admin@192.168.1.21 "cd /opt/big-boi-ai && sudo docker compose stop n8n"
ssh mcp-admin@192.168.1.21 "sudo tar -xzf /nfs/heezy/backups/n8n/n8n-data-TIMESTAMP.tar.gz -C /opt/big-boi-ai/n8n/"
ssh mcp-admin@192.168.1.21 "cd /opt/big-boi-ai && sudo docker compose up -d n8n"

n8n postgres¶

ssh mcp-admin@192.168.1.21 "cd /opt/big-boi-ai && sudo docker compose stop n8n"
ssh mcp-admin@192.168.1.21 "gunzip -c /nfs/heezy/backups/n8n/n8n-postgres-TIMESTAMP.sql.gz | sudo docker exec -i n8n-postgres psql -U n8n n8n"
ssh mcp-admin@192.168.1.21 "cd /opt/big-boi-ai && sudo docker compose up -d n8n"

open-webui¶

ssh mcp-admin@192.168.1.21 "cd /opt/big-boi && sudo docker compose stop open-webui"
ssh mcp-admin@192.168.1.21 "sudo tar -xzf /nfs/heezy/backups/open-webui/open-webui-TIMESTAMP.tar.gz -C /opt/big-boi/open-webui/"
ssh mcp-admin@192.168.1.21 "cd /opt/big-boi && sudo docker compose up -d open-webui"

Restore from S3¶

Use this when NFS backups have aged out (beyond 7-day retention) or NFS is unavailable.

Retrieval wait for older backups

Backups uploaded more than 90 days ago are in GLACIER_IR — retrieval takes minutes to hours. Backups within 90 days are in STANDARD_IA — immediately downloadable, no restore request needed.

# Check storage class first
aws s3api list-objects-v2 --bucket heezy-backups-025066240222 \
  --prefix heezy/k8s/<service>/ \
  --query "Contents[*].{Key:Key,StorageClass:StorageClass}" \
  --output table | tail -10

# If STANDARD_IA — download directly
aws s3 cp s3://heezy-backups-025066240222/heezy/k8s/<service>/<backup>.tar.gz /tmp/

# If GLACIER_IR — initiate restore first (minutes to hours)
aws s3api restore-object \
  --bucket heezy-backups-025066240222 \
  --key heezy/k8s/<service>/<backup>.tar.gz \
  --restore-request '{"Days":3,"GlacierJobParameters":{"Tier":"Standard"}}'

# Check restore status
aws s3api head-object \
  --bucket heezy-backups-025066240222 \
  --key heezy/k8s/<service>/<backup>.tar.gz \
  --query "Restore"
# Ready when: ongoing-request="false"

From S3, download to MCP container first (nebula nodes don't have AWS credentials), then SCP to a node, then kubectl cp into the restore pod:

# On MCP container
aws s3 cp s3://heezy-backups-025066240222/heezy/k8s/<service>/<backup>.tar.gz /tmp/
scp -i /root/.ssh/mcp_heezy /tmp/<backup>.tar.gz mcp-admin@192.168.1.15:/tmp/

# On nebula node
sudo microk8s kubectl cp /tmp/<backup>.tar.gz heezy/<service>-restore:/tmp/backup.tar.gz

Legacy S3 backup format (pre-July 2026 wipe)¶

Backups created before the July 2026 data wipe used a different format: the tarball contains a single .db file (e.g. radarr-20260612_101851.db) rather than a full /config directory tree. This affects radarr, sonarr, lidarr, and prowlarr backups from before that date.

Extraction for these old backups differs from the standard procedure:

# Extract to /tmp first — do NOT extract directly to /config
sudo microk8s kubectl exec -n heezy <service>-restore -- tar -xzf /tmp/backup.tar.gz -C /tmp/

# The extracted file will be named like <service>-TIMESTAMP.db — rename it
sudo microk8s kubectl exec -n heezy <service>-restore -- \
  sh -c 'mv /tmp/<service>-*.db /config/<service>.db'

# Remove any stale WAL/SHM
sudo microk8s kubectl exec -n heezy <service>-restore -- \
  sh -c 'rm -f /config/<service>.db-shm /config/<service>.db-wal'

All backups created by the current heezy-backup Ansible role use the full /config tar format and follow the standard restore procedure above.

Prowlarr — credential reset¶

If Prowlarr credentials are lost, reset via direct SQLite edit. There is no sqlite3 binary in the container — use a restore pod on nebula-4 (where prowlarr runs).

Prowlarr uses PBKDF2-HMAC-SHA1, 10000 iterations, 32-byte derived key, base64-encoded salt and hash stored in the Users table.

# Scale down
ssh mcp-admin@192.168.1.15 'sudo microk8s kubectl scale deployment prowlarr -n heezy --replicas=0'

# Spin up alpine pod with PVC mounted — base64 the manifest to avoid quoting issues
cat <<'EOF' | base64 -w0
apiVersion: v1
kind: Pod
metadata:
  name: prowlarr-reset
  namespace: heezy
spec:
  nodeSelector:
    kubernetes.io/hostname: nebula-4
  restartPolicy: Never
  volumes:
  - name: config
    persistentVolumeClaim:
      claimName: prowlarr-config
  containers:
  - name: reset
    image: alpine
    command: ["sleep", "600"]
    volumeMounts:
    - name: config
      mountPath: /config
EOF
# Pipe the above to: ssh mcp-admin@192.168.1.15 'base64 -d > /tmp/p.yaml && sudo microk8s kubectl apply -f /tmp/p.yaml'

# Install sqlite in the pod
ssh mcp-admin@192.168.1.15 'sudo microk8s kubectl exec -n heezy prowlarr-reset -- apk add --quiet sqlite'

# Generate hash and update — run this python3 script on the MCP container or nebula-1
# (uses subprocess to call kubectl exec, avoiding all quoting issues)
python3 - <<'PYEOF'
import subprocess, hashlib, base64, os
password = "your-new-password"
iterations = 10000
salt = os.urandom(16)
dk = hashlib.pbkdf2_hmac('sha1', password.encode('utf-8'), salt, iterations, dklen=32)
salt_b64 = base64.b64encode(salt).decode()
hash_b64 = base64.b64encode(dk).decode()
sql = f"UPDATE Users SET Username='admin', Password='{hash_b64}', Salt='{salt_b64}', Iterations={iterations} WHERE Id=1;"
r = subprocess.run(
    ["sudo","microk8s","kubectl","exec","-n","heezy","prowlarr-reset","--",
     "sqlite3","/config/prowlarr.db", sql],
    capture_output=True, text=True)
print("RC:", r.returncode, r.stderr or "OK")
PYEOF

# Clean up and scale back up
ssh mcp-admin@192.168.1.15 'sudo microk8s kubectl delete pod prowlarr-reset -n heezy'
ssh mcp-admin@192.168.1.15 'sudo microk8s kubectl scale deployment prowlarr -n heezy --replicas=1'

Key detail: dklen=32 is required. Omitting it produces a 20-byte key that won't match what Prowlarr expects.

Manual backup trigger¶

Via Gitea Actions:

Gitea → heezy-admin/ansible-heezy → Actions → playbook-heezy-backup-execution.yml → Run workflow (main branch).

Via Gitea API:

curl -sk -X POST "http://192.168.1.15:30360/api/v1/repos/heezy-admin/ansible-heezy/actions/workflows/playbook-heezy-backup-execution.yml/dispatches" \
  -u "heezy-admin:HeezyGit2026!" \
  -H "Content-Type: application/json" \
  -d '{"ref":"main"}'

Verify backup health¶

# Recent backups on NFS (files modified in the last 2 days)
ssh mcp-admin@192.168.1.21 "find /nfs/heezy/backups -name '*.tar.gz' -mtime -2 | sort"
ssh mcp-admin@192.168.1.21 "find /nfs/heezy/backups -name '*.snap' -mtime -2 | sort"

# List k8s backup dirs
ssh mcp-admin@192.168.1.21 "ls /nfs/heezy/backups/k8s/"

# S3 listing
aws s3 ls s3://heezy-backups-025066240222/heezy/ --recursive --human-readable | sort | tail -20

Known gaps¶

Plex config PVC is large (~50Gi) — backup runs but takes several minutes. Monitor to confirm it completes within the 60-minute workflow timeout.
TrueNAS (192.168.1.200) has no backup coverage. A NAS failure means loss of all NFS backup copies — there is no off-site copy of the NFS data itself beyond the monthly S3 archive.
LGTM metrics (Mimir/Prometheus data, ~115 GB) are not backed up. Only configs are preserved. Metrics will regenerate from scrapers after a restore.
qbittorrent has no S3 backup history prior to the July 2026 wipe. First usable backups will be from the first successful heezy-backup run post-wipe. (nzbhydra2 was decommissioned 2026-08-08.)
Backups work, but only run when someone pushes to the role. Verified 2026-08-08: run 667 completed failed=0 and wrote real archives for 22 of 24 targets, so the procedures below do have artifacts to restore from. However playbook-heezy-backup-execution.yml has no schedule: trigger, so artifacts can be arbitrarily stale — before run 667 the previous execution was 2026-07-28. Check the timestamp of the archive you are restoring. See Backups.
SABnzbd S3 backup was confirmed corrupt (45 bytes) prior to the July 2026 wipe. Usenet server credentials and categories must be reconfigured manually after a restore — they are not recoverable from backup.