Skip to content

CI/CD

2026-06-20 — Workflow Audit + Cleanup (All 4 Repos)

  • Discord run links: Added clickable Gitea run URL to all 40 Discord curl notifications across heezy-containers, heezy-k8s, ansible-heezy, terraform-heezy
  • Catch-all playbook workflows fixed: cs16, heezy-backup, minecraft, minecraft-java, ut2k4-server were still using playbooks/** glob + playbooks/*.yml — same bug as deleted playbook-all-execution.yml. All scoped to their own playbook/role path.
  • test-gitea-detection.yml deleted: Fired on every main push (no path filter), pure migration artifact
  • Redundant AWS CLI removed: deploy + notify jobs in heezy-containers don't use Docker or AWS — removed Install AWS CLI step and DOCKER_HOST env from those jobs
  • ECR layer caching: Added docker pull :latest || true + --cache-from :latest to all 9 docker build commands. Skips pip/apt layers on unchanged base
  • POSIX if/then: Fixed [[ ]] bashism → POSIX if/then across all repos (act-runner uses /bin/sh not bash)
  • FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: Added to any workflow missing it
  • pvc-protection finalizer: Fixed k8s-cleanup playbook to patch finalizers: null before deletion (prevents silent hang)

2026-06-20 — K8s Audit + Scanner Affinity Fix

  • Full k8s audit completed: 5 nodes healthy, all node affinity removed
  • Last pinned workload: heezy-statement-scanner CronJob (was nebula-5) — fixed
  • Identified orphaned PVCs: receipts-db, sabnzbd-downloads, longhorn-test-pvc
  • 16 containers missing resource requests (backlogged for fix)
  • Overseerr hostNetwork noted (backlogged)

Canonical repo: Gitea (http://192.168.1.15:30360/heezy-admin/)
Mirror: GitHub (github.com/trentnielsen) - read-only backup

All deployments run through Gitea act-runner. Workflows are configured per repo (heezy-containers, heezy-k8s, terraform-heezy, cloudflare-terraform).

Deployment Pipeline

heezy-containers (services)
  push to gitea main
    -> .gitea/workflows/deploy-<service>.yml
    -> build job: Docker build + ECR push
    -> deploy job: push .deploy-trigger to heezy-k8s repo

heezy-k8s (Kubernetes manifests)
  .gitea/workflows/auto-deploy.yaml
    trigger: receive .deploy-trigger file from heezy-containers
    step 1: kubectl apply -k (kustomize)
    step 2: kubectl rollout restart (force pod restart)
    step 3: kubectl rollout status (wait for ready)

terraform-heezy (infrastructure)
  .gitea/workflows/<environment>.yaml per env
    terraform plan -> apply

cloudflare-terraform (CF Access + tunnel)
  .gitea/workflows/deploy.yaml
    terraform plan -> apply

Repos and Workflows

heezy-containers

Per-service deploy workflows: .gitea/workflows/deploy-<service>.yml

  • Trigger: push to gitea main
  • Build: Docker build, ECR push
  • Deploy: Write .deploy-trigger file to heezy-k8s

heezy-k8s

Auto-deploy orchestration: .gitea/workflows/auto-deploy.yaml

  • Trigger: .deploy-trigger file in repo
  • Steps:
  • kubectl apply -k (apply kustomized manifests)
  • kubectl rollout restart deployment/<name> (restart pods)
  • kubectl rollout status (wait for readiness)
  • Rollback: Git revert + push (old deploy-trigger removed, new one created on next service deploy)

terraform-heezy

Per-environment execution: .gitea/workflows/<environment>.yaml

  • Workflows: dev.yaml, staging.yaml, production.yaml
  • Trigger: push to gitea main
  • Steps: terraform plan -> terraform apply (auto-approved for now)

cloudflare-terraform

Cloudflare Access + tunnel config: .gitea/workflows/deploy.yaml

  • Manages CF Access policies, identity providers, tunnel config
  • Protected sites: finance, receipts, cs16, blog, docs, heezy.info

Hard Rules (2026-06-20)

⚠️ These are non-negotiable. Violations cause pipeline failures and deployment delays.

Repository Structure

  • .github/workflows/ must NEVER exist in any repo — Gitea act-runner executes both .gitea/workflows/ AND .github/workflows/, causing duplicate/conflicting runs and failures
  • Gitea runners cannot reach GitHub Actions (no external network in DinD)
  • All workflows must be in .gitea/workflows/ only
  • Delete .github/workflows/ from all repos immediately if found

Infrastructure Changes

  • Never run terraform apply locally — always via Gitea pipeline (.gitea/workflows/<environment>.yaml)
  • State is managed in S3; local applies cause drift

  • Pipeline enforces policy and audit trail

  • Never run Ansible locally — commit changes to repo and push to Gitea

  • Gitea runner is the single source of truth for infrastructure state

  • Never kubectl apply directly — push to heezy-k8s repo instead

  • Commit to git, push to gitea main
  • heezy-k8s auto-deploy workflow triggers: kubectl apply -k, rollout restart, rollout status
  • Rollback: git revert + push

Container Deployments

  • heezy-containers deploy trigger: Push .deploy-trigger file to heezy-k8s repo
  • heezy-containers pipeline writes it after ECR push
  • heezy-k8s auto-deploy watches for this file and deploys
  • Example: apps/heezy-finance/.deploy-trigger

Credentials and Secrets

  • AWS credentials: Always fetch from OpenBao at secret/data/production/heezy/github_runner/aws_credentials via AppRole login
  • No hardcoded credentials in repos or environment
  • No OIDC — AppRole auth only
  • Pipeline steps: vault login -method=approle -path=approle ... then vault read secret/data/production/heezy/github_runner/aws_credentials

IAM Bootstrap (One-Time)

  • Terraform IAM self-management issue: github-runner-user IAM user needs manual IAMFullAccess policy attachment exactly once before the pipeline can assume and self-manage its own IAM permissions
  • After attachment, pipeline can create/update policies via Terraform
  • This is a one-time bootstrap step per AWS account
  • Documented in terraform-heezy/iam-roles.tf

General Rules

  • Always push to gitea remote first, then origin (GitHub backup)
  • Act runner v0.6.1 does NOT support reusable workflows - inline everything
  • DOCKER_HOST=tcp://172.17.0.1:2375 required in all build jobs (Docker-in-Docker)
  • GITEA_INSTANCE_URL=http://192.168.1.15:30360 (NodePort - DinD can't resolve k8s DNS)

Key Lesson: Push to Gitea, Not GitHub

Issue: Pushing to GitHub origin instead of Gitea canonical repo breaks deployments.

Root Cause: GitHub is a mirrored backup only; act-runner watches Gitea. Pushing to origin (GitHub) doesn't trigger the pipeline.

Fix: Always push to Gitea remote first, then origin.

# ❌ Wrong — GitHub doesn't trigger act-runner
git push origin main

# ✅ Correct — Push to Gitea first, then GitHub
git push gitea main
git push origin main

See Roadmap for full deployment pipeline details (2026-06-20).

Known Issues

Gitea DB Sync Bug

Runners complete jobs but don't always update status in the database. Auto-fixed by gitea-db-sync CronJob every 5 minutes.

Manual fix if needed:

UPDATE action_task_step SET status=3 WHERE status=1 AND stopped > 0;
UPDATE action_task t SET status=3 WHERE status=1 AND stopped > 0
  AND NOT EXISTS (SELECT 1 FROM action_task_step s WHERE s.task_id=t.id AND s.status NOT IN (3,4,6));
UPDATE action_run_job j SET status=3 WHERE status IN (1,2)
  AND task_id IN (SELECT id FROM action_task WHERE status=3);
UPDATE action_run r SET status=3 WHERE status IN (1,2)
  AND NOT EXISTS (SELECT 1 FROM action_run_job j WHERE j.run_id=r.id AND j.status NOT IN (3,4,6))
  AND EXISTS (SELECT 1 FROM action_run_job j WHERE j.run_id=r.id);

Workflow Authoring Rules

Hard-won lessons from debugging act-runner failures:

Shell compatibility

Act-runner executes workflow steps with /bin/sh, not bash. The following patterns fail silently:

# ❌ Fails in sh
EMOJI=$([[ "$STATUS" == "success" ]] && echo "✅" || echo "❌")

# ✅ Works in sh
if [ "$STATUS" = "success" ]; then EMOJI="✅"; else EMOJI="❌"; fi

Replace all [[ ]] constructs with POSIX if/then/else.

curl flags

-f (--fail) silently discards the response body on HTTP errors, making failures very hard to debug:

# ❌ Hides error details
curl -sf http://openbao/v1/...

# ✅ Shows what went wrong
curl -s http://openbao/v1/...

External downloads from DinD

DinD containers in the act-runner have unreliable internet access. External downloads (e.g. from dl.k8s.io) can silently time out at 300s, causing the step to appear to hang.

Fix: Serve tools from internal Gitea releases instead.

# ❌ Flaky — dl.k8s.io times out from DinD
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

# ✅ Fast and reliable — internal LAN
curl -L "http://192.168.1.15:30360/heezy-admin/heezy-k8s/releases/download/tools-v1/kubectl" -o kubectl

Pre-built tools are stored as Gitea release assets on heezy-k8s under the tools-v1 tag.

Git clone target

Always clone from the internal Gitea NodePort, not GitHub:

# ❌ DinD can't reach github.com
git clone https://github.com/tnielsen2/heezy-k8s.git

# ✅
git clone http://192.168.1.15:30360/heezy-admin/heezy-k8s.git

Kubeconfig in heezy-k8s auto-deploy

The KUBE_CONFIG Gitea Actions secret holds the base64-encoded kubeconfig. Use it directly:

- name: Configure kubeconfig
  run: |
    mkdir -p ~/.kube
    echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
    chmod 600 ~/.kube/config
    kubectl cluster-info

DB sync lag ("cancelled" conclusion)

The gitea-db-sync CronJob runs every 5 minutes to fix stale task statuses. If a run shows conclusion: cancelled in the API but all steps completed, it's a sync lag artifact. Always check step-level statuses for ground truth.

Manual fix if needed: 

UPDATE action_run_job SET status=3 WHERE run_id=<N> AND status IN (1,2);
UPDATE action_run SET status=3 WHERE id=<N> AND status IN (1,2);