Skip to content

heezy-finance — Daily Log

2026-08-11 (later)

Docs audit against live state

Every page in the heezy-finance and Services sections re-verified against the running cluster, the heezy database, and the source in heezy-containers. The section had last been touched on 2026-07-07 and had drifted badly.

Corrected:

  • Taxonomy. Moved to categories.py and grown from 16 documented entries to 27. Personal Care is Personal Care & Wellness. Pet Care is implemented, not backlogged. Income, Retail, Shopping, Savings, and Transfer were documented as categories and are not in CATEGORY_ICONS
  • real tech debt is gone. Every money column is numeric. The Phase 3 migration section was describing work already done
  • Counts. orders 161 → 284, receipts 43 → 155, receipt_items 342 → 821, bank_statements 64 → 80, bank_transactions 1,203 → 1,582, accounts "194+" → 27
  • boacrdNone ghost rows: 0. Fixed and documented as still open
  • Nav. Eight pages, not seven. /statements replaced /record-balances, which is still routed but unlinked. /bank, /purchases, /trends redirect to /. Added a Statements page doc
  • Spending page uses two period pickers, not a date range picker with localStorage
  • Targets page was documented as user-editable milestone goals. It is five hardcoded targets reading live account balances, with 7% projections
  • Budget page referenced a budgets table. It is spend_budgets plus annual_budgets
  • Phase 6 (remove node affinity) is done. Phase 7 shipped inside the app rather than as a separate heezy-finance-reconcile service
  • Statement parser does not run on big-boi from a host cron. It is the heezy-statement-scanner CronJob. /var/log/statement-ingest.log does not exist
  • Receipts OCR is Textract, pinned to textract (DetectDocumentText) in the deployment even though the code default is expense (AnalyzeExpense). No S3 in the image path
  • Cloudflare Access lives in terraform-heezy, not a cloudflare-terraform repo. Added statements.heezy.info and emulatorjs, plus the home-IP bypass policies
  • OpenBao secret path table was wrong on five entries. Rebuilt from the ExternalSecrets and roles

Bugs found while auditing

  • Manual balance entry is broken. statements.html and record-balances.html both POST to /api/account-balances with account_id in the body. Only /api/account-balances/<int:account_id> exists, so it 404s. Verified live
  • /sankey redirects to /money, where there is no Sankey. The chart is still in #page-overview in index.html
  • 12 receipts have a NULL date_ts and are dropped by every date-filtered query. The June backfill covered all 51 rows that existed then; volume has since tripled
  • budget_alerts.py keeps its own copy of CATEGORY_ICONS and does not import categories.py, which is the exact duplication that module exists to prevent
  • vw_sankey_spend has no Capital One branch but admits every credit card statement, so a Cap1 credit card charge would file under Direct Debit. Nothing is misfiled today, Cap1 is checking and savings only
  • get_all_purchases() still parses orders.order_date, the raw text column, against the project's own rule to use order_date_ts
  • The OpenBao doc published a live AppRole role_id and secret_id. Removed. Still in git history, and mcp-server.js uses the same values as a hardcoded fallback. Rotate

2026-08-11

Ten weeks of missing orders recovered (123 orders, 165 items)

Started as three Grafana alerts. Two were real bugs with unrelated causes; one was a false positive.

Bug 1 — sync crash-looping on invalid_grant. The heezy-finance-sync CronJob lost its envFrom: heezy-finance-gmail and silently fell back to /credentials/gmail-oauth.json, the hand-maintained copy that went stale in June (see 2026-07-08, where it was patched by hand rather than removed). The container crash-looped six times and exhausted its backoff limit at 21:00 and 22:00 UTC. Fixed by restoring envFrom, then deleting the file fallback entirely — it had caused two outages and prevented none. Credentials are now env-only, and a test asserts the fallback stays gone.

Bug 2 — the actual cause of the data gap. 28a7171 (2026-06-02, "replace hardcoded sender allowlist with broad subject keyword search") deleted the clause that was doing all the work. Amazon confirmations have the subject Ordered: "<item>", which matches none of the fourteen phrases that replaced the senders. Steam, Apple, Walmart and Grubhub went with it.

The last ingested order was dated 2026-06-01 13:29 UTC; the commit landed the next day. Running the production query unchanged over May, while the sync was believed healthy, returned 5 emails and 0 from amazon.com — the subject phrases had never matched Amazon. Fixed by making the query the union of the two clauses. A bare subject:"Ordered:" was deliberately rejected: it matches ~380 messages to the allowlist's ~124, because the forward mailbot copies every confirmation as Fwd: Ordered: from a personal address.

False positive — slskd disconnect alert. The rule counted log lines matching "Waiting about", believing it to be the reconnect backoff. It is the per-file download retry message. Over the 13 hours behind three pages, the pod logged 47 of them and zero disconnects — it connected once at 11:12:50 and stayed connected. Repointed at the connection lifecycle (Attempting to connect to|Disconnected from the Soulseek server).

Backfill

Added --dry-run to amazon_orders.py first, since a real run archives Amazon mail out of the inbox and that does not reverse. Three dry runs: 4 orders (pre-fix), 124 (post-query-fix, 1 duplicate), 123 (post-normalization, clean).

Backfilled --hours 1750: 123 orders, 165 items, covering 2026-06-02 17:29 → 2026-08-11 03:02. Gmail reconciled exactly — 122 confirmations labeled Finance/Amazon, 0 left in the inbox, 119 archived by the run, matching the 119 new Amazon orders.

Data repairs

  • Order ID normalization. Dedupe is exact string equality on a primary key, so one Woot order was about to insert twice as #212850333 and #212850333. Added _normalize_order_id(), which also made the backfill idempotent for every LLM-parsed vendor.
  • 15 legacy IDs rewritten (14 Woot with a leading #, 1 Celebration Cinema with a doubled space), 31 items repointed, 0 orphans. Needed a copy/repoint/delete because order_items.order_id has ON DELETE CASCADE and no ON UPDATE.
  • 41 grand_total values reconstructed from priced line items, $503.02. Recorded spend since June 2 went $1,094.39 → $1,597.41.

Still open

  • grand_total is unparseable on roughly a third of current-format Amazon confirmations. Line items save us, but that is luck, not design. 2 orders remain without a total because an item is unpriced.
  • Amazon Shipped:/Delivered: mail is trashed on purpose to keep the inbox clear. Not a bug.

PRs: ansible-heezy #16; heezy-containers #40, #41, #42, #43, #44

2026-07-08

  • money-history chart fixed: Trimmed leading cash-only months (Nov 2025–Apr 2026) from trend chart — these had investments=0/retirement=0 before EJ statements were loaded, causing a misleading spike from ~$20k to $1.96M. Now starts May 2026 (first complete month).
  • FIRE target fixed: annual_spend already excludes category='Mortgage'; was double-subtracting mtg_annual. Result was FIRE target $168k (wrong). Fixed to annual_spend * 25. Correct: $1.13M target, 106% funded.
  • api/uncategorized-items 500 fixed: receipts query referenced non-existent description column; orders query used wrong column names (id, title, total_cost — none exist). Fixed queries; endpoint returns 200/empty list.
  • api/reconcile/unreconciled-by-category 500 fixed: receipts.date is text, not a date object — .strftime() called on string. Fixed with hasattr check + str truncation fallback.
  • heezy-finance-sync Gmail fixed: k8s secret heezy-finance-secrets had stale refresh token. Patched with working token from n8n variables. Verified via manual test job (clean sync, exit 0).
  • Cluster hygiene: Force-deleted 9 stale Completed pods from old statements deployment rollouts. Cluster now clean — 35 Running pods, no orphaned pods.
  • All 17 heezy-finance API endpoints: 200 OK

  • Commits: heezy-containers e81381f, ce3a4c3, 23163f3; q-mcp 4875967, 2298c49, d884ae8, 14e710c

  • Data Standards doc published (heezy-finance/data-standards.md) — canonical reference for dates, numerics, categories, IDs, source tracking

  • Receipts date_ts backfill complete: 2 remaining NULL rows (NUVU FUELS 2026-06-18/19) backfilled — all 51 receipts now have normalized timestamptz
  • Known real tech debt documented: receipts.total, receipts.items_total, orders.grand_total, order_items.unit_price, receipt_items.* — scheduled for Phase 3 numeric(12,2) migration
  • Category list documented with icons and rules; Phase 3 categories FK table spec captured

2026-07-07

Statement Parser Bug Fixes (3 bugs, 3 PDFs)

Three PDFs failed to ingest with a non-descriptive "3 error(s)" response. Root cause traced through parse.py → detect_bank() → upsert_account_balance() → accounts table.

Bug 1 — Missing accounts: BoA CC last4=7503 and Chase Amazon last4=2609 had no rows in the accounts table. upsert_account_balance() silently skips balance updates when no matching account exists — no error surfaced. Fixed by inserting both accounts with display_group='debt'.

Bug 2 — Capital One false-detected as Edward Jones: Capital One PDFs can contain "EDWARD JONES" in transaction descriptions (EJ investment withdrawals). detect_bank() was checking EDJ before Cap1, causing misidentification. Fixed by swapping check order in parse.py — Cap1 is now checked before EDJ.

Bug 3 — Chase Amazon wrong closing date: CHASE_AMAZON_CLOSE_RE matched Opening/Closing Date 05/20/26 - 06/19/26 and captured the first date (05/20) instead of the closing date (06/19). Fixed with a new primary regex that specifically matches the Opening/Closing Date X - Y format and captures the second date, with the old regex kept as fallback.

DB cleanup: - Deleted stale edjbrok0000_2026-06-30 row inserted during failed Cap1 upload - Deleted bad chasecrd2609_2026-05-20 statement + 98 transactions; re-uploaded Chase CC.pdf → chasecrd2609_2026-06-19 ✅ - Deleted stale account_balances row for Chase Amazon (account 194) dated 2026-05-20 - Deactivated old account id=5 (Chase last4=7503, stale data)

New accounts added: - id=193: BoA Visa Signature (last4=7503, credit_card, debt) - id=194: Chase Amazon Visa (last4=2609, credit_card, debt)

Commits: heezy-containers — two commits fixing Cap1/EDJ detection and Chase closing date regex

Docs Updated

  • data-sources.md — rewrote Bank Transactions section: accurate parser list, upload endpoint, parser quirks table, account setup requirements
  • pages/record-balances.md — corrected upload endpoint (/api/statements/upload, field files), replaced NFS cron pipeline description with accurate synchronous in-process flow, updated supported formats

Known Issue Documented — boacrdNone (12 rows)

12 bank_statements rows with account_last4 = NULL, ending_balance = NULL, 0 transactions. Source files are eStmt_YYYY-MM-DD.pdf — BoA CC PDFs where the last4 regex failed to match. These are ghost rows that block re-upload and pollute the statements table. Separate parser bug, not fixed this session.

2026-06-21

2026-06-20

  • CI/CD: Fixed all 4 Gitea pipeline repos post .github/workflows/ removal
  • heezy-containers: Fixed [[ ]] bashism in notify job, curl -sf → curl -s in deploy job
  • heezy-k8s auto-deploy:
  • Clone from internal Gitea (not github.com)
  • kubectl served from internal Gitea release tools-v1 (dl.k8s.io → 300s timeout from DinD → 1s internal)
  • KUBE_CONFIG secret used directly (OpenBao kubeconfig path not populated)
  • All workflow steps now complete in ~31s total
  • ansible-heezy: Workflow detection confirmed ✅
  • terraform-heezy: Still blocked on one-time manual IAM bootstrap (Trent)
  • AGENTS.md: Added mandatory startup procedure (git pull q-mcp + scan skills registry)

Outage Recovery (nebula-5 offline)

  • Root cause: 13+ services had hard nodeAffinity to nebula-5; node went offline and pods couldn't reschedule
  • Fix: removed nodeAffinity from all heezy-k8s deployments (commit 3bad4fd)
  • Rule established: never add nodeAffinity unless hardware-specific requirement exists

heezy-finance Crash Fixes

  • Fixed 3 sequential startup bugs: missing typing imports, duplicate route registration, SQL column name errors
  • Category drill-down (/api/category-transactions) now working

finance.heezy.info Live

  • Added SWAG proxy conf + volumeMount entry in deployment.yaml
  • DNS: dnsmasq (192.168.1.29) + Pi-hole both updated → 192.168.1.25 (SWAG VIP)
  • Phases 1-4 verified working: date selector, lookback, drill-down, Hockey Holdem receipt

CI/CD Root Cause Found

  • Gitea act-runner was executing both .gitea/workflows/ AND .github/workflows/
  • .github/ versions used actions/checkout@v4 which failed (can't reach github.com from DinD)
  • Fix: Trent removed .github/workflows/ from all 4 repos
  • All future workflows: .gitea/workflows/ only

Terraform IAM (in progress)

  • Created productionATerraformStateBackend IAM role in terraform-state-role.tf
  • Added iam:* permissions to github-runner-user policy
  • Blocked on bootstrap: github-runner-user needs manual IAMFullAccess attachment once

2026-06-19 (evening)

  • nvidia-gpu-exporter deployed to big-boi (192.168.1.21:9835), Prometheus scraping with labels host=big-boi, gpu=gtx1070
  • Split DNS .heezy.local zone established in Pi-hole with 11 records (nfs, bigboi, lgtm, nebula1-5, vault, dnsmasq)
  • heezy-k8s manifests: hardcoded NFS IPs replaced with nfs.heezy.local DNS names; k8s coredns queries Pi-hole for resolution
  • heezy-finance: Income→Spending Sankey diagram deployed on Bank page with year/month filters (Plotly, 3-layer flow)
  • heezy-finance: Vendor drilldown feature deployed — click category cards for per-vendor breakdown + monthly stacked chart
  • New API endpoints: /api/sankey (year/month filter), /api/category-drilldown (category detail)
  • Documentation: GPU monitoring section added to infrastructure/index.md, new infrastructure/dns.md page covering split DNS architecture + Pi-hole config + all .heezy.local records

2026-06-19 (afternoon)

  • Category taxonomy overhaul planned: split Health & Personal Care → Health Care Costs + Personal Care; split Household → Household + Mortgage + Utilities; added Hockey and Pet Care categories. Implementation backlogged. See Categories.

2026-06-19

  • LGTM disk crisis: 192.168.1.10 hit 90% — root cause was lgtm-cleanup.sh targeting mimir/blocks/ instead of mimir/blocks/anonymous/. 33 blocks accumulated since April. Freed 42GB, disk 90% → 60%. Ansible role + live script both fixed.
  • Receipts OCR fix: bare 'card' keyword in skip_keywords was silently dropping items like "SHARP CARD / HOCKEY". Narrowed to compound phrases (debit card, credit card, etc.).
  • Receipts missing images: all 43 pre-NFS receipts lost images (ephemeral storage). NFS PVC working for new uploads. Added placeholder in UI for missing images.
  • Backup system: full overhaul — NFS structure, Ansible role, OpenBao policy, S3 lifecycle, monthly archive workflow. ~250GB of stale data purged from NFS.
  • Runners: comprehensive doc written covering current setup and lean runner proposal.
  • heezy-finance: added "Last Purchased" date column to Trends → Top 10 Items (All Time).

2026-06-18

Phase 1 + Phase 2 — Data consolidation + dashboard improvements

Phase 1 (completed)

  • Full audit of all data sources and dashboard code
  • Dashboard was reading orders + receipts but had zero integration with bank_transactions
  • receipts.date text column had mixed formats — causing silent parse failures; gas showed $36 instead of $140.52
  • Added date_ts (timestamptz), merchant_normalized, payment_type to receipts
  • Added category, merchant_normalized to bank_transactions
  • Backfilled date_ts on all 43 existing receipts
  • Built rule-based bank transaction categorizer (Ollama too slow on CPU for this)
  • Updated app.py to use date_ts for receipts; integrated bank_transactions as third source
  • Updated receipts service to populate date_ts + merchant_normalized on all new inserts
  • Fixed BoA parser: was hardcoded to checking — now detects credit card statements from first 3000 chars of PDF
  • Fixed Cap1 parser: transactions were assigned to wrong account due to stmt_id lookup using flushed-only map
  • Statement auto-ingest: NFS watch dir pipeline confirmed end-to-end (2 PDFs → parsed → DB)
  • docs.heezy.info: MkDocs Material site deployed to Cloudflare Pages with Gitea pipeline
  • All public heezy sites added to Cloudflare Access (Google OAuth, trentnielsen84 + brdrgrl18)

Gas spend fixed: $140.52 (was $36). June 2026 total: ~$2,226 across all sources.

Phase 2 (in progress)

  • Bank page: Added dedicated Bank tab to dashboard — category breakdown cards, Plotly monthly bar chart, top merchants table
  • Bank page "This Month": Falls back to most recent month with data instead of showing empty cards for current month (May 2026 shown)
  • Purchases table: All 6 columns now sortable (Date, Source, Vendor, Item, Total, Category) — click header to sort, ▲▼ indicators
  • Purchases date defaults: Start Date defaults to first of current month, End Date to today on first page load
  • payment_type on receipts: Dropdown added to both edit flows (post-upload + inline): cash / credit / debit / check / other
  • CI/CD: Fixed all deploy-*.yml trigger paths (were using repo name instead of k8s app dir); added rollout restart to auto-deploy.yaml so :latest images actually roll out
  • Monthly reminder: Cron fires 1st of each month at 10am ET to prompt NFS statement drop

Blocked: BoA CC statements + additional monthly statements — manual download from bank portal required.

Evening session (continued)

  • n8n Amazon forwarding fixed: Workflow had been erroring on every run since 2026-06-13. Root cause: two nodes (Log Forward to DB, Log Skip to DB) existed in DB but not in n8n's compiled in-memory graph. n8n caches the workflow graph at activation time and never reloads from DB on restart — only a REST API PUT forces a recompile. Fix: stripped the two missing nodes, rewired connections to route around them. Execs 3694+ all success.
  • Lesson: N8N_PROTOCOL=https blocks HTTP write ops (PUT/POST) on localhost with 403, even via SSH tunnel. To unblock: temporarily set N8N_PROTOCOL=https_DISABLED, restart, make API call, restore.
  • Dev agent context: Created dev-agent/CONTEXT.md — comprehensive reference for spawned dev subagents covering repos, git remotes, CI/CD gotchas, DB, SSH, services, metrics requirement, and commit style.
  • CF Access resource rename (deferred): Attempted to rename cloudflare_access_application → cloudflare_zero_trust_access_application in Terraform. Neither moved{} blocks nor state mv work in cloudflare provider v4 for cross-type renames — provider v4 can't refresh the new type, so plan shows destroy+create. Reverted access.tf to original names. Defer until provider v5 upgrade.
  • Docs updated: services/cloudflare-access.md added, statement-parser.md + data-sources.md + ci-cd.md updated via docs subagent.

2026-07-08 — Statement parser quality sweep

Root causes found and fixed (4 bugs):

  1. boacrdNone ghost statements (scan_statements.py) — When BOA_CC_ACCT_RE regex misses last4 on first parse, account_last4=None produces a ghost boacrdNone_* statement_id alongside the real one. Guard added: skip insert_bank_statement when account_last4=None and account_type=credit_card. Deleted 12 existing ghost rows. (heezy-containers ca17a25)

  2. Cap1 PDFs misdetected as BoA (parse.py detect_bank) — Cap1 statements contain "Bank of America" in transaction descriptions (transfers). detect_bank checked BOA_DETECT first, misidentifying them as boa, then parse_boa extracted 0 transactions. Fixed detection order to Cap1 → BoA → EDJ. (heezy-containers c2f4046) Reprocessed 9 files from error/ back to new/.

  3. FK violation on multi-account PDFs (scan_statements.py) — insert_transactions fired before insert_bank_statement for multi-account Cap1 PDFs (Holla checking + Our Savings). Second account's transactions hit FK constraint on bank_transactions.statement_id → bank_statements.statement_id. Fixed: insert all statement rows first, then insert transactions. (heezy-containers 41e6b3b)

  4. Cap1 Holla (8641) txns assigned to wrong account (parse.py parse_cap1) — For multi-account PDFs, transactions were assigned stmt_id = acct_stmts.get(cur_last4) at parse time, but acct_stmts only populates on flush_account() (triggered by next account section). Holla txns all got None, then the back-fill logic reassigned them all to the last-seen account (4306/savings). Fixed: store _acct_last4 per txn, resolve to stmt_id after all flushes complete. (heezy-containers 7c0efff)

Result: cap1chk8641_* — 8 months now have transactions (was 0 each). Total bank_transactions: 1,405 across 43 distinct statements.

Commits: ca17a25, c2f4046, 41e6b3b, 7c0efff (heezy-containers); 436f7bc, 4d4617f, b38d638 (ansible-heezy sync); f8ac70b + 565fc3a (heezy-k8s deploy-trigger)

Still 0-txn: boachk1897_2026-06-25 — statement row exists (inserted 2026-07-06), PDF gone. Needs Trent to re-drop eStmt_2026-06-25.pdf to NFS.