Changelog¶

All notable changes to heezy infrastructure and services.

[2026-10-02] Daily Discord alert for watchable hockey games¶

Added¶

hockey-notify, a CronJob at 11:00 America/New_York. Posts the day's NHL slate: the Live TV channels that actually play each game (candidates by network code or guide listing, each test-played through its Threadfin), or the league's broadcasters for games the tuners do not carry. Posts every day. See Live TV
FOX affiliates on the US tuner, for Stars and Kings games on KDFW and KTTV

Fixed¶

Live TV tunnels looping on NordVPN AUTH_FAILED. gluetun picked Dedicated IP servers at random. The US Threadfin tunnel now uses standard servers only and restarts itself after about 3 minutes down

[2026-10-01] Live TV on nebula Plex through two VPN tuners¶

Added¶

Two Threadfin tuners feed nebula Plex live TV from iptv-org's public playlist, filtered to sports networks. threadfin-vpn exits in the US (69 channels: NBC, ABC and CBS affiliates, ESPN, FS1, NHL/NFL/NBA/MLB channels) and threadfin-ca-vpn exits in Canada (TSN The Ocho, TVA, MSG). Each is a separate Plex DVR. See Live TV
iptv-epg, an in-cluster iptv-org guide grabber. Its channel list is generated from the Threadfin seeds by heezy-k8s/scripts/build-iptv-epg-channels.py, so listings map to channels automatically. Local affiliates get their own station schedules, not a network feed
threadfin.heezy.info and threadfin-ca.heezy.info, LAN-only web UIs through SWAG

[2026-09-17] ut2k4's query port stops filling with ICMP errors¶

Fixed¶

ut2k4's query port held one packet at a time, and DMZ-UT2K4 flapped. The engine sets IP_RECVERR on 7778 and never reads the error queue, so every ICMP port unreachable from an internet poller stayed charged to the socket's receive buffer. Six days after the pod started the buffer was full, and 7778 dropped anything that arrived while another query was waiting. Master servers got one answer to each burst of three, and Uptime Kuma's check failed whenever it landed beside one: 18 times since 2026-09-13. An init container now drops ICMP port unreachable in the pod's network namespace. See the error queue trap

[2026-08-26] Every game moves to the cluster and the R610 is emptied¶

Added¶

All six game servers run as pods on VLAN 3. cs16 at 192.168.3.2, both bedrock worlds at .3, ut2k4 at .4, and all three Minecraft Java worlds at .5. Each holds a real DMZ address on a macvlan interface, so traffic it sends to SHARED is off-subnet and the FortiGate decides. Every volume was seeded by restoring the NFS archive rather than copying off the VM
dmz-game-pods, a /29 covering 192.168.3.0 to .7. Every game pod sits in it, so k8s-to-dmz-monitoring reaches a fifth game without being edited
cs16 runs as a pod on VLAN 3. Real DMZ address on a macvlan interface, not behind a LoadBalancer VIP, so traffic it sends to SHARED is off-subnet and the FortiGate decides. The volume was seeded by restoring the NFS backup rather than copying off the running VM: a copy proves the bytes move, a restore proves the backup is restorable, and that is the thing that has to be true before the R610 is switched off
dmz-to-shared-deny, an explicit deny for DMZ initiating into SHARED, logged. fwpolicy-implicit-log is disabled globally and should stay that way on a 60E, so until now a blocked attempt left no trace and the rule the DMZ design rests on had no evidence either way. It sits below dmz-to-lgtm-loki, still the one permitted exception
Game Server On Macvlan, covering the pattern and the two failures that look like nothing else in the lab

Fixed¶

hlds bound to the Calico address, so no client could ever join. steamclient opened its connections from 10.1.x.x while routing them out the DMZ interface, the FortiGate dropped them on reverse path check, and the server never reached Steam. With sv_lan 0 that is fatal to authenticating anyone. A2S queries are connectionless and need no Steam, so the server answered every probe from every network and looked healthy the whole time. -ip has to be passed, and passed before -game: after it, the engine accepts it and ignores it
The game-restore role could not render its own job. game_restore_nfs_path was referenced by the template and defined nowhere, so every dispatch failed before it touched a volume
The restore guard counted lost+found as data. Longhorn hands out a freshly formatted ext4 volume, so a brand new PVC is never literally empty and the guard rejected every first restore
A successful restore was reported as a failure. The completion poll filtered on [?(@.status=="True")], the command module split the argument the way a shell would, and kubectl received a filter matching nothing. The poll then ran its full 120 retries against a job that had finished in twenty seconds
game-restore no longer fires on push. It asserts that a game and a PVC were named, so a run with no inputs could only fail, and every push touching its role posted a red Discord message for a repo state that was fine

Changed¶

Minecraft Java is reachable from the internet at all, and all three worlds are. minecraft-java-external was set to deny, which is the same as having no policy: the VIP translated the address and the policy dropped the result. The host also runs three servers, 25565 java, 25566 adventure and 25567 cobblemon, and only 25565 had a VIP, so the other two worked on the LAN and nowhere else
Player traffic follows the pod. Three VIP mappedip values, and separately the dnsmasq split-DNS override for the game's public name. Both halves are needed: the VIPs carry internet players, the override carries LAN players, and the per-game USERS firewall policies carry nobody because Users to DMZ is all sources to all destinations and sits above them
Players go to a new dmz-cs16-pod address object. The dmz-cs16 one was kept for the few hours the VM stayed up as the rollback, and went with it

Removed¶

Every VM on the R610 destroyed. shared-dnsmasq, shared-lgtm, dmz-cs16, dmz-minecraft, dmz-minecraft-java and dmz-ut2k4. DNS had been a DaemonSet behind an anycast address since Phase 1 with every consumer repointed, LGTM had been on big-boi with its data since Phase 2, and cs16 became a pod today
Everything that only existed to point at them: the three DNS policies 309, 315 and 319, whose anycast equivalents already carried that traffic; big-boi-cs16-ssh; the shared-dnsmasq, shared-lgtm and dmz-cs16 address objects; the DHCP reservations; three Uptime Kuma monitors; and the inventory groups, which each held exactly one host
dmz-to-lgtm-loki. It was the single permitted DMZ to SHARED exception, for promtail on the game VMs shipping to Loki. There are no DMZ VMs left and the pods have their logs collected from the node filesystem, which never crosses the boundary. DMZ now initiates into SHARED for nothing at all, and dmz-to-shared-deny records every attempt
permit to prometheus, policy 20. It allowed USERS to tcp/9090 on 192.168.1.29, a host that ran no Prometheus. It was named for a testvm that had not existed in years and had been carried as a known problem since the legacy policies were adopted
playbooks/dnsmasq.yml, playbooks/cs16.yml and the docker-compose half of roles/dnsmasq. roles/cs16-server was kept: files/ holds the only copy of the plugin source the pod's compiled plugins were built from

Known gaps¶

No game server has a working backup. The game backup reaches four hosts over SSH and all four are destroyed. Six worlds now live on Longhorn PVCs that nothing is capturing, and Longhorn has no snapshot or backup RecurringJob either. The archives they were restored from are still on NFS, so the loss window starts today
The Minecraft adventure world has not started since 2026-04-19. It fails the same way on the VM it came from and in the pod, so it is the world data rather than the move: Overworld settings missing, with level.dat last written in April. It sat in a restart loop for four months and nothing alerted, because the monitor for that host checked node-exporter and node-exporter never stopped
Bedrock has no monitor. Both worlds are UDP only and Uptime Kuma's port monitor cannot check UDP. The node-exporter that used to stand in for it does not exist in a pod

[2026-08-25] LGTM move follow-ups¶

Fixed¶

baseline was reinstalling node_exporter over the role's unit. Both roles/baseline and roles/prometheus-node-exporter wrote /etc/systemd/system/node_exporter.service, and baseline's copy omitted --collector.textfile.directory. playbooks/baseline.yml runs the role after baseline so it always came out right; playbooks/lgtm.yml runs baseline alone, and now that the lgtm group means big-boi it stripped the textfile collector off the one host that uses it. The GLX probe kept writing its .prom file, node_exporter stopped reading the directory, and Steam Headless GLX Probe Stale fired on NoData. Node Exporter is out of baseline now, the same split already made for Promtail
steam-headless had genuinely fallen back off NVIDIA GL, which the blind probe could not report. glxinfo was failing with BadValue on X_GLXCreateNewContext while Xorg's own glxserver_nvidia was loaded cleanly, so it was device access lost to a systemd reload, not a driver mismatch. A container restart cleared it

Changed¶

LGTM data lives on /mnt/lgtm. lgtm_data_path was /opt/lgtm/data, inherited unchanged from the VM where /opt was the only disk, so the move put 27 GB of TSDB on big-boi's 114 GB root volume. Root went from 49% to 25%
Every LGTM path derives from lgtm_root_path now. The compose file, the systemd WorkingDirectory, the weekly auto-update cron, the restart handler and both retention prunes in lgtm-cleanup.sh each spelled /opt/lgtm out; repointing the data alone would have left the nightly prunes running find against an empty directory and stopped retention silently
Reclaimed 3 GB on big-boi: 1.5 GB of Steam core dumps under /var/lib/apport/coredump from the GL fallback, three orphaned Docker volumes superseded by bind mounts, and the apt lists cache

Known gaps¶

The snmp/fortigate target is down. The heezy-ro SNMP community allow-lists 192.168.1.10, so a poll from big-boi gets no reply and the scrape times out. One line in terraform-heezy/environments/production/heezy/snmp.tf
lgtm-stack on shared-lgtm is inactive but still enabled, so a reboot of that VM would start a second full stack scraping every target in the lab

[2026-08-25] LGTM moved to big-boi¶

Changed¶

LGTM runs on big-boi. grafana, prometheus, loki, tempo and mimir all resolve to 192.168.1.21. The shared-lgtm VM is stopped but intact as the rollback until the R610 is decommissioned
Mimir started fresh, 52 of the 78 GiB. Nothing read it: Prometheus is the default datasource and none of the five provisioned dashboards reference Mimir
The two DMZ firewall policies, dmz-to-lgtm-loki and 37 lgtm-dmz-node-exporter, follow it to shared-big-boi
DHCP dns_server2 on the DMZ path is unaffected; the DMZ hosts carry the address literally in inventory/group_vars/dmz.yml because they resolve through public DNS

Migration¶

Pre-seeded 26 GiB while both stacks ran, then took the blackout for a second pass that moved 38 files and 235 MB in under 33 seconds. Prometheus history survived the move.

Corrected¶

inotify does not pick up a ConfigMap update. The DNS page said a zone change needed no restart. It does. See the warning under Generated hosts files
The shared-lgtm PTR was removed rather than followed. lgtm_ip is bigboi_ip now, and keeping both would have put two PTR records on 192.168.1.21
loki_server in the dnsmasq role defaults was deleted rather than updated. Nothing read it

[2026-08-25] DNS: anycast over BGP, five instances¶

Changed¶

Internal DNS is now six dnsmasq instances, not one. A DaemonSet of five pods behind the anycast address 192.168.253.53, advertised to the FortiGate over BGP by every node with a healthy pod, plus the shared-dnsmasq VM as a secondary in a different failure domain
The address had to change. 192.168.1.29 is inside the SHARED subnet, so VLAN 1 hosts reach it by ARP and never consult a route. A BGP /32 inside a connected /24 is dead weight for exactly the clients that matter most, so the anycast address lives outside every client subnet
DHCP on SHARED and USERS hands out both resolvers. dns_server2 was 1.1.1.1, which is authoritative for nothing internal, so any client falling through to it got NXDOMAIN for every internal name rather than a slower correct answer. It is 192.168.1.29 now
CoreDNS and Pi-hole forward to both, anycast first. CoreDNS uses policy sequential; the forward plugin defaults to random, which would have sent half of every internal lookup to the VM being retired
The MetalLB default pool no longer overlaps live hosts. It was a flat .25-.40 with autoAssign: true, which let MetalLB hand out .26 (Cisco switch mgmt) or .29 (the dnsmasq VM) to any Service that did not pin its IP
The L2Advertisement is scoped. It had no spec at all, which in MetalLB matches every pool, so it would have ARPed for the anycast address alongside the BGP path

Behaviour¶

Failure	Detection	Result
Pod crashes or fails readiness	seconds	that node withdraws its path, four remain
Node reboots or dies	9s worst case	path withdrawn on hold-timer expiry
Whole cluster down	not covered	DHCP secondary, the VM, answers

Measured: deleting one pod dropped the route from five next-hops to four within seconds and restored it about 18 seconds later.

9s is the floor, not a tuning choice. MetalLB is v0.13.3 in native mode with no FRR sidecar, so there is no BFD. Sub-second detection needs FRR mode and a BFDProfile, which touches every VIP in the lab.

Two traps worth keeping¶

keepaliveTime is in the MetalLB BGPPeer CRD schema but the admission webhook rejects it in native mode. It validates client-side and fails at apply. Native mode derives it as holdTime/3
The FortiGate ASN is as_string, not as: the provider gates the numeric field to FortiOS <= 7.2.0 and this device is 7.4.3. A clean terraform plan does not prove a fortios resource will apply, because the version gate only fires once it is talking to the device

[2026-08-19] Backups: one tool, weekly off-site, 30 day retention¶

Changed¶

All backups are Ansible. The plex-config-backup and uptime-kuma-backup CronJobs are gone from heezy-k8s; both are targets in the heezy-backup role now. The only mechanism-specific case left is the OpenBao raft snapshot, which has to be an API call because a tar of a live raft store is not a restorable snapshot
Capture and shipping split into two plays. heezy-backup captures to NFS daily at 02:00 UTC. The new heezy-backup-offsite ships to S3 weekly, Saturday 08:00 UTC. While they were one task the only lever for uploading less was capturing less
Plex is three paths, not all of /config. Preferences.xml, Plug-in Support/Databases, Plug-in Support/Metadata Combination. Media, Metadata and Cache are regenerable artwork and transcode scratch that Plex rebuilds. 11.9 GiB to ~1.6 GiB
Off-site retention 365 days to 30, and storage class GLACIER to STANDARD_IA. GLACIER Flexible Retrieval bills a 90 day minimum per object, so a 30 day GLACIER tree is billed as a 90 day one: within pennies of STANDARD_IA at this size, and 3 to 5 hours slower to read. Restores from S3 no longer need a thaw
The off-site run ships one file per series, not the whole tree. NFS holds 7 dailies and a weekly RPO needs one of them

Numbers¶

	Before	After
Daily capture	~21 GB	~7.4 GB
Weekly upload	~147 GB	~7.4 GB
Off-site run time	1h+, every day	~15 min, once a week
NFS steady state	~130 GB	~52 GB
S3 stored	136.5 GiB	~30 GB

Noted while doing it¶

Local NFS retention must stay at or above the ship interval. At 7 days local and a 7 day interval every snapshot is still on disk when the weekly run comes for it. This is not hypothetical: the S3 step used to gate on date +%d == 01, the daily schedule did not exist, and nothing reached S3 for 67 days
Moving off site to weekly widens the TrueNAS blast radius. A NAS failure now leaves only the last weekly generation, up to 7 days of exposure instead of 1
heezy-backup's role defaults advertised openbao_url: http://192.168.1.32:8200 while every run actually used http://192.168.1.15:30820. An unprefixed variable in a role default is the lowest precedence Ansible has, and group_vars/all reaches the implicit localhost that hosts: localhost plays run on. Both backup roles now use role-prefixed, name-based endpoints
uptime-kuma.heezy.local resolves to SWAG (192.168.1.25), not the uptime-kuma VIP (192.168.1.33). k8s-services.j2 maps every app name to vip_swag, so an app name gives you the reverse proxy, never the backend VIP on its own port. The .33 VIP has no name

[2026-08-14] Docs: four services that were running but undocumented¶

Added¶

Maintenance and Projects — deployed 2026-08-02, no docs until now. Both are LAN-only Flask apps on their own Postgres databases (maintenance and projects, not the shared heezy one), reached through SWAG at NodePorts 30861 and 30862
Mailbot — the three Gmail CronJobs that replaced n8n on 2026-08-10. All three now run --live; they were cut over from shadow mode after the dry runs matched
CS1.6 Leaderboard — was listed on the home page with a URL and nothing else. Documents the Loki-tailer pipeline and its nanosecond-timestamp dedupe

Noted while documenting¶

Neither Maintenance nor Projects has any authentication in the app. The only access control is that the hostname resolves solely through the dnsmasq override — no public DNS record, no Cloudflare tunnel hostname, no Access application. Publishing a DNS record for either without an Access app would expose an unauthenticated CRUD API
heezy-projects orders the index by priority DESC, which is a lexical sort on text. medium sorts above low above high, so high-priority projects appear last
project_tasks and project_log exist in schema.sql and have no routes in app.py
PUT /api/projects/<id> writes every column from the payload; a partial body blanks what it omits

[2026-08-12] Receipts: concurrent uploads, lost receipts, and duplicates¶

Fixed¶

Concurrent uploads overwrote each other - The OCR poller was never cancelled when a second upload started, and there is one result panel. Two pollers raced and the last to finish owned the DOM - The quick-edit form used bare element ids (qe-total), so the Save button carried one receipt's id while getElementById could resolve to a form rendered for another. Corrections landed on the wrong receipt, which read as "the second upload didn't save" - Field ids are now suffixed with the receipt id, the previous poller is retired, and the panel carries the id it belongs to

Receipts that disappeared - /upload saved the image to NFS before inserting the row. A failed INSERT lost the receipt while the bytes stayed on disk. The image is now removed when the row cannot be written - receipts.upload_date was TEXT holding two incompatible formats — isoformat() from /upload and NOW()::text from heezy-finance. Lexicographic ordering put a space before T, so every finance-created receipt sank below the uploaded ones from the same day. Migrated to timestamptz - GET /receipts was a hardcoded LIMIT 100 with no pagination. At 157 rows, 57 receipts were unreachable from the UI. Now takes limit and offset

"Error loading items" on every receipt with line items - NUMERIC columns return Decimal, which Flask serialises as a quoted string. The items table called .toFixed() on it and threw. A NumericJSONProvider now renders Decimal as a number

The per-item edit button never worked - It inlined JSON.stringify(item) into a double-quoted onclick, so the attribute terminated at the first quote in the JSON. Every line item on every receipt was uneditable

Line item quality - A trailing number was treated as a quantity using \b, and a hyphen is a word boundary, so 997895 KS OIL OW-20 became twenty units of 997895 KS OIL OW-. UPCs were read as quantities up to 716,281,006,230. Now requires whitespace and caps at 100 - Descriptions were built by stripping trailing digits, turning 997920 OIL 5W20 into 997920 OIL 5W - 9 existing items repaired from their raw_line; total_price was correct throughout, so no financial figure moved

Category list drift - Alcohol, Transportation, Retirement Funds, Stock Funds and MittenTech were selectable in heezy-finance but missing from receipts. A test now fails the build on drift

Added¶

reconcile_orphans.py — re-imports images with no database row, dry-run by default
GET /receipts returns has_image, so the UI stops guessing from the filename
Delete control on orphaned receipts in the heezy-finance Reconcile tab, proxied to the receipts service so the image goes with the row
Receipts pod raised to 3000m CPU so concurrent OCR pipelines do not share one core

Incident¶

Recovering 52 orphaned images created 36 duplicate receipts, including seventeen copies of one Costco receipt. The recovery assumed an image without a row meant a lost receipt; in fact the delete path had been dropping rows while failing to remove files, so most orphans were receipts deleted on purpose. The duplicates were removed and the tool now rolls back any import whose amount and date already exist.

[2026-06-20] heezy-finance Phase 1-2 + Phase 4 Deployment¶

Added¶

Phase 1: Global Date Range Selector - React date picker controls all dashboard views - Persists to localStorage across sessions - Default: last 90 days

Phase 2: Bank Integration & Complete Coverage - Dedicated Bank tab with category breakdown cards - Monthly bar chart showing bank transaction trends - Top merchants table - All views (Overview, Purchases, Categories) now include bank data - 137 bank transactions integrated (debits only, transfers excluded) - Rule-based categorization: Household, Health, Travel, Food, Finance, etc.

Phase 3: Category Drill-Down Modal - Click category to see all transactions in that category - Paginated list (20 per page) - Sortable columns: Date, Source, Vendor, Item, Total, Category - Filters: date range, source (order/receipt/bank)

Phase 4: Hockey Holdem Receipt Ingestion - New receipt category for event expenses - Receipts automatically parsed and categorized - Verified: data in DB with proper timestamps and merchant normalization

Changed¶

Database schema updates: - receipts: Added date_ts (timestamptz), merchant_normalized, payment_type - bank_transactions: Added category, merchant_normalized - All existing data backfilled

Dashboard API: - /api/overview now includes bank transaction totals - /api/purchases returns all three sources with sortable columns - /api/sync reports counts from all sources - /api/category-drilldown new endpoint for drill-down modal - New /api/bank-stats endpoint for Bank page aggregations

Deployment: - k8s: heezy-finance pinned to nebula-5 (node affinity issue noted) - All code pushed to Gitea (canonical) + GitHub (backup mirror) - Commit: 45d3a4af

Fixed¶

Gas spend now accurate: $140.52 (was $36) - Root cause: Receipt dates in inconsistent text formats caused silent parse failures - Fix: Use proper date_ts column instead of parsing text dates

Bank transactions visibility: - Root cause: Zero integration in original dashboard - Fix: Full integration into all API endpoints and UI views

Receipts merchant name consistency: - Root cause: "COSTCO" vs "Costco" vs "costco" treated as different merchants - Fix: Backfilled merchant_normalized on all 43 receipts + new receipts auto-normalized

Known Issues¶

Node Affinity Pinning (heezy-finance) - Pod pinned to nebula-5; cannot reschedule if node fails - Blocking resilience work next session - Fix: Remove affinity + switch PVC to ReadWriteMany or eliminate persistent storage

June 2026 Statements Missing - BoA coverage ends May 2026 - June transactions not yet loaded - Action: Trigger statement parser or manually ingest

Capital One Backfill Incomplete - Only 1 CC statement loaded - Historical backfill pending

CI/CD Lesson Learned - Never push to GitHub origin directly - Gitea is the canonical repo; GitHub is a backup mirror only - Push to Gitea first, then GitHub - Deployments only trigger on Gitea push

See CI/CD docs and Roadmap.

Next Session¶

Phase 3: Infrastructure Resilience

Remove node affinity pinning from heezy-finance
Fix PVC storage pattern (NFS or remove if not needed)
Load June 2026 statements
Verify all features working correctly

[2026-06-18] heezy-finance Documentation & Data Audit¶

Added¶

Comprehensive data source audit: orders, receipts, bank transactions
Roadmap for 3-phase data cleanup + reconciliation
Known issues list (priority-ordered)
Daily progress log in README

Fixed¶

Identified root causes of data gaps:
Receipt dates in inconsistent formats (text field)
Bank transactions completely disconnected from dashboard
Gas spend under-reported due to category/date handling

[Earlier] Infrastructure Setup¶

k8s cluster: 5 nodes (nebula-1 through nebula-5)
Gitea: http://192.168.1.15:30360/heezy-admin
heezy-finance: Initial Flask + vanilla JS dashboard
Receipts service: AWS Textract + Ollama categorization
Bank statement parser: pdfplumber on big-boi cron job