CI/CD¶

Last verified: 2026-08-22.

Canonical git host: Gitea — gitea.heezy.local:3000 from inside the lab, https://gitea.heezy.info from a browser. Both are the same server; :30360 is its NodePort and :3000 its MetalLB VIP.

Mirror: GitHub, read-only. Pushing there triggers nothing.

Every deployment runs on the Gitea act-runner in the gitea namespace. There is no GitHub Actions, no Jenkins, and nothing scheduled on big-boi.

Repos and their remotes¶

Repo	Remote name	Workflows	Branch-protected
`q-mcp` (root)	`gitea`	none	no
`ansible-heezy`	`gitea`	23	yes
`terraform-heezy`	`gitea`	13	yes
`heezy-containers`	`gitea`	20	yes
`heezy-k8s`	`gitea`	4	no, deliberately
`heezy-docs`	`origin`	2	yes

Two traps in that table:

heezy-docs calls its Gitea remote origin. Everywhere else origin is the GitHub mirror. In heezy-docs, git push origin main is the correct, pipeline-triggering push.
heezy-k8s is intentionally unprotected. Deploy workflows push .deploy-trigger commits straight to its main; adding protection breaks the pipeline. The root q-mcp repo is unprotected too, and has no GitHub mirror at all — git push origin main fails there.

There is no cloudflare-terraform repo. Cloudflare is a workspace inside terraform-heezy (environments/production/cloudflare).

Deployment Pipeline¶

heezy-containers (app code)
  push to gitea main
    -> .gitea/workflows/build.yml
         docker build + push to ECR (us-east-2)
    -> .gitea/workflows/deploy-<app>.yml
         clone heezy-k8s, commit apps/<app>/.deploy-trigger, push

heezy-k8s (manifests)
  .gitea/workflows/auto-deploy.yaml
    trigger: push touching base/** or apps/**
    kubectl apply -k base/
    kubectl apply -k apps/<changed>/
    kubectl rollout restart + rollout status for each changed deployment

terraform-heezy (infra)
  .gitea/workflows/terraform-<env>-<provider>-execution.yml
    plan -> apply on the act-runner

ansible-heezy (host config)
  .gitea/workflows/playbook-<name>-execution.yml
    AppRole login to OpenBao -> clone repo -> ansible-playbook

Repos and Workflows¶

heezy-containers¶

build.yml — builds and pushes the image for whichever dockerfiles/<app>/ changed
deploy-<app>.yml — 12 of them; each clones heezy-k8s and writes .deploy-trigger
test-<app>.yml — 6 of them: heezy-finance, heezy-mailbot, heezy-maintenance, heezy-projects, music-sweeper, receipts. pytest with a --cov-fail-under gate

Only those six apps have a test workflow. The other apps have no CI status check at all, so branch protection will happily merge a broken image.

heezy-k8s¶

auto-deploy.yaml — the main path, described above
deploy-service.yaml — manual dispatch for a single service
auto-update-images.yaml — image bumps
_discord-notify.yml — see Reusable workflows are dead files

Rollback is git revert + push; the revert is itself a change under apps/, so auto-deploy picks it up.

terraform-heezy¶

validate.yml — a 10-workspace matrix: terraform init -backend=false, validate, and fmt -check for each of environments/{dev/aws, dev/heezy, dmz/heezy, production/aws, production/cloudflare, production/gcp, production/github, production/heezy, shared/aws, shared/heezy}
terraform-<env>-<provider>-execution.yml — one per workspace, plan then apply
terraform-all-workspaces.yml — dispatch across everything

There are no dev.yaml / staging.yaml / production.yaml workflows, and no staging environment.

ansible-heezy¶

Every playbook-<name>-execution.yml is generated, never hand-written:

python3 scripts/generate-workflows.py

Run it before committing any playbook change. It is idempotent, so on a clean tree it produces no diff. If it does produce one you did not expect, the generator and the committed workflows have drifted and one of them is wrong — do not just commit the output. Per-playbook facts the generator cannot derive (cron schedules, timeouts, extra steps, title casing) live in dicts at the top of the script; add to those rather than editing the emitted YAML.

Also present: lint.yml (ansible-lint, profile: min), build.yml, build-heezy-runner.yml (builds the runner image itself), terraform-triggered.yml.

ansible-heezy is the one repo with a .github/ directory. It holds scripts/ only — no workflows.

heezy-docs¶

test.yml — on any non-main push and on PRs: mkdocs build --strict
deploy.yml — on push to main: mkdocs build, then publish to Cloudflare Pages with a token from OpenBao

Note the asymmetry: the strict build only runs pre-merge. A warning introduced directly on main will not fail the deploy.

The act-runner¶

3 replicas in the gitea namespace, one per node, so a node reboot does not stall CI
Images: gitea/act_runner:latest plus a docker:27-dind sidecar
GITEA_INSTANCE_URL=http://gitea.heezy.local:3000
Labels ubuntu-latest, ubuntu-22.04 and self-hosted all resolve to the same ECR image, heezy-runner:latest, built by build-heezy-runner.yml
Runner-level DOCKER_HOST=tcp://localhost:2375; build jobs set DOCKER_HOST=tcp://172.17.0.1:2375 to reach the DinD sidecar from inside a job container

Credentials¶

Runner secrets come from OpenBao, not from Gitea's secret store. Gitea holds only the AppRole bootstrap pair.

Gitea secret	Purpose
`OPENBAO_ROLE_ID` / `OPENBAO_SECRET_ID`	AppRole login to `http://openbao.heezy.local:8200`
`GIT_CLONE_TOKEN`	HTTP clone of sibling repos
`KUBE_CONFIG`	base64 kubeconfig, heezy-k8s only
`DISCORD_WEBHOOK_URL`	run notifications

Everything else is read at job time from secret/data/production/heezy/.... AWS credentials come from secret/data/production/heezy/github_runner/aws_credentials — AppRole only, no OIDC.

VAULT_TOKEN=$(curl -sk -X POST "$OPENBAO_ADDR/v1/auth/approle/login" \
  -H 'Content-Type: application/json' \
  -d "{\"role_id\":\"$OPENBAO_ROLE_ID\",\"secret_id\":\"$OPENBAO_SECRET_ID\"}" \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['auth']['client_token'])")

IAM bootstrap (one-time)¶

The github-runner-user IAM user needs IAMFullAccess attached manually, exactly once, before the pipeline can self-manage its own IAM permissions with Terraform. One-time per AWS account.

Hard Rules¶

Never terraform apply, kubectl apply, or ansible-playbook locally. Only Gitea Actions applies. Local is limited to plan, validate, --check, and read-only kubectl get/describe/logs
Workflows live in .gitea/workflows/, never .github/workflows/. The act-runner would execute both, producing duplicate and conflicting runs
Push to gitea first. GitHub is a mirror and triggers nothing. gh run list reads GitHub and is always wrong for this lab — use the Gitea API
Never bypass branch protection. No deleting branch_protections via the API, no force-push workarounds

git push gitea main

Known Issues¶

`gitea-db-sync` is suspended¶

The gitea-db-sync CronJob in the gitea namespace exists on a */5 * * * * schedule but is currently suspended — it last ran 12 days ago. Until it is resumed, stale run statuses do not self-heal, and a run showing cancelled while every step completed has to be fixed by hand.

kubectl get cronjob gitea-db-sync -n gitea          # SUSPEND column
kubectl patch cronjob gitea-db-sync -n gitea -p '{"spec":{"suspend":false}}'

Manual fix:

UPDATE action_task_step SET status=3 WHERE status=1 AND stopped > 0;
UPDATE action_task t SET status=3 WHERE status=1 AND stopped > 0
  AND NOT EXISTS (SELECT 1 FROM action_task_step s WHERE s.task_id=t.id AND s.status NOT IN (3,4,6));
UPDATE action_run_job j SET status=3 WHERE status IN (1,2)
  AND task_id IN (SELECT id FROM action_task WHERE status=3);
UPDATE action_run r SET status=3 WHERE status IN (1,2)
  AND NOT EXISTS (SELECT 1 FROM action_run_job j WHERE j.run_id=r.id AND j.status NOT IN (3,4,6))
  AND EXISTS (SELECT 1 FROM action_run_job j WHERE j.run_id=r.id);

Always check step-level status for ground truth before believing a run's conclusion.

Reusable workflows are dead files¶

Each repo carries _discord-notify.yml, and ansible-heezy adds _setup-openbao-token.yml and _setup-runner-aws-credentials.yml. They are valid workflow_call definitions and nothing references them — no workflow in any repo has a uses: ./.gitea/workflows/_* step. Every caller inlines the logic instead. Treat them as unused until something actually calls one.

Clone URLs have drifted¶

ansible-heezy workflows clone from gitea.heezy.local:3000. heezy-containers, heezy-k8s and heezy-docs still clone from 192.168.1.31:3000. Both work; the name is the correct form. Do not use gitea.heezy.info:3000 — that resolves to SWAG, which does not proxy git.

Workflow Authoring Rules¶

Hard-won lessons from debugging act-runner failures:

Shell compatibility¶

Act-runner executes workflow steps with /bin/sh, not bash:

# Fails in sh
EMOJI=$([[ "$STATUS" == "success" ]] && echo "ok" || echo "fail")

# Works in sh
if [ "$STATUS" = "success" ]; then EMOJI="ok"; else EMOJI="fail"; fi

curl flags¶

-f (--fail) silently discards the response body on HTTP errors, making failures very hard to debug. Use curl -s and inspect the body.

External downloads from DinD¶

DinD containers have unreliable internet access. External downloads (dl.k8s.io and friends) can silently time out at 300s, so the step appears to hang. Serve tools from internal Gitea release assets instead — they live on heezy-k8s under the tools-v1 tag.

curl -L "http://gitea.heezy.local:3000/heezy-admin/heezy-k8s/releases/download/tools-v1/kubectl" -o kubectl

`FORCE_JAVASCRIPT_ACTIONS_TO_NODE24`¶

Set it in env: on any workflow that uses a JavaScript action, or the action runs on a Node the runner image no longer ships.

Kubeconfig in heezy-k8s auto-deploy¶

- name: Configure kubeconfig
  run: |
    mkdir -p ~/.kube
    echo "${{ secrets.KUBE_CONFIG }}" | base64 -d > ~/.kube/config
    chmod 600 ~/.kube/config
    kubectl cluster-info