DNS Architecture¶

Last updated: 2026-08-25

Use the Name, Not the Address¶

This page contains a lot of IP addresses because it documents the resolver itself. That is the exception, not the pattern. Everywhere else, write the heezy.local name.

An IP in a playbook, template, script, manifest, workflow, or runbook is a bug. Addresses belong in exactly two places:

ansible-heezy/roles/dnsmasq/defaults/main.yml — the source of truth. The dnsmasq role renders this entire zone from those variables, so they are the definition of DNS, not a duplicate of it. It is not group_vars/all/infrastructure.yml: ansible.cfg sets inventory = inventory/hosts.yml, Ansible reads group_vars from beside the inventory file and beside the playbook, and the repo root was never on that search path. That directory has been deleted.
Anything that must work before or without DNS — dnsmasq's own bootstrap and its post-deploy verification step, the resolver address, FortiGate policy objects, BGP neighbour addresses, MetalLB pool ranges, DHCP reservations. A DNS check that resolves a name to find its own nameserver is circular and will pass off a stale cache.

Two traps worth naming:

{{ bigboi_ip }} is the same bug as the literal. Substituting a variable still pins the endpoint to an address instead of a record.
Prefer the VIP name over a node NodePort. openbao.heezy.local:8200 is the MetalLB VIP; nebula1.heezy.local:30820 is one node, and a single point of failure for every secret lookup in a play.

When a name does not resolve, add the record — do not fall back to the address. The fix is one line in roles/dnsmasq/templates/.

Every host in ansible-heezy/inventory/hosts.yml has a forward record, plus cisco-switch and both Ruckus APs (ruckus-ap, ruckus-ap-member). bigboi.heezy.local and big-boi.heezy.local both resolve; the hyphen-less aliases live in infra-services.j2 and stay, because callers reference them.

The Ruckus APs resolve but are not Ansible hosts and have no credentials in OpenBao, so nothing backs them up.

One name that surprises people: uptime-kuma.heezy.local resolves to SWAG (192.168.1.25), not the uptime-kuma VIP (192.168.1.33). k8s-services.j2 maps app names to vip_swag by default, so an app name usually gives you the reverse proxy on :443, not the backend VIP on its own port. Two names break that default on purpose: uptime-kuma-direct.heezy.local → 192.168.1.33, because SWAG does not proxy Kuma's Socket.IO API, and slskd.heezy.local → 192.168.1.34.

Overview¶

Internal DNS is served by six dnsmasq instances rendered from one set of templates: five pods in the cluster behind a single anycast address, and one VM outside it. Pi-hole is an ad-blocking entry point for some VLANs and holds no zone records of its own.

Resolver	Address	Role
dnsmasq (anycast)	192.168.253.53	Primary. A `DaemonSet` of five pods, one per nebula node, behind a single MetalLB service address advertised to the FortiGate over BGP by every node that has a healthy pod.
dnsmasq (VM)	192.168.1.29 (`shared-dnsmasq`)	Secondary, and a different failure domain: it is not in the cluster, so a cluster-wide fault does not take it with them.
Pi-hole	192.168.1.27 (MetalLB VIP)	Ad-blocking entry point for `pihole-vlan-201` and `prod-vlan-2000`. Holds no zone records; forwards everything to the two resolvers above.

Both dnsmasq deployments are authoritative for the same four zones and render from the same Ansible templates, so any of the six can answer any internal query identically. That is what makes the secondary a real fallback rather than a decoration.

Zones¶

Authoritative, never forwarded upstream:

heezy.local
trentnielsen.me
heezy.info
heezy.blog
1.168.192.in-addr.arpa (reverse DNS)

A name with no record returns NXDOMAIN rather than silently landing on the reverse proxy, so a typo in a hostname fails loudly.

Anycast¶

Why the address is not 192.168.1.29¶

192.168.1.29 sits inside the SHARED subnet. Every host on VLAN 1 reaches an address in its own /24 by ARP and never consults a route, so a BGP path to a /32 inside a connected /24 is dead weight for exactly the clients that matter most. An anycast address has to live outside every client subnet so the traffic is actually routed.

192.168.253.0/29 is reserved for anycast services and holds nothing else. The FortiGate has 192.168.1, 2, 3, 5, 100, 200 and 201 connected, so 253 is clear.

The cost is that internal DNS now transits the firewall, where SHARED to .29 stayed on the switch. At homelab query volume that is free, but it does mean the path is policy-governed, which is what policies 324, 325 and 326 exist for.

How a query reaches a pod¶

client
  → FortiGate (192.168.253.53 is not on any connected subnet, so it routes)
  → ECMP across the five /32 paths currently in the BGP table
  → one nebula node
  → kube-proxy → the dnsmasq pod on that node

The FortiGate runs v4-ecmp-mode: source-ip-based with ecmp-max-paths: 255, so it spreads across every advertising node and pins each client to one path. set ebgp-multipath enable is what makes it install all five rather than picking a single best path; without it the other four pods are decoration.

BGP¶

Setting	Value	Where
FortiGate ASN	64512	`terraform-heezy/environments/shared/heezy/router-bgp.tf`
Cluster ASN	64513	`heezy-k8s/base/metallb/metallb-config.yaml`
Neighbours	192.168.1.15 to .19	one per nebula node, each running a MetalLB speaker
`ebgp_multipath`	enable	FortiGate
Timers	keepalive 3, hold 9	both ends

FortiOS defaults are keepalive 60 and hold 180, which would leave a dead node blackholing its share of queries for three minutes.

keepaliveTime is not valid on the MetalLB BGPPeer

The field exists in the BGPPeer CRD schema, so it passes kustomize build and client-side validation, but the metallb admission webhook rejects it in native mode: peer 192.168.1.1 has keepalive-time set on native bgp mode. It is an FRR-mode setting. Native mode derives the keepalive interval as holdTime/3, so holdTime: 9s already yields 3s and matches the FortiGate. Checking the CRD is not sufficient for any FRR-only field.

The ASN on the FortiGate side is as_string, not as. The provider exposes both and gates the numeric one to FortiOS <= 7.2.0; this device is 7.4.3, which made the field a string so it can carry asdot notation. A clean terraform plan does not prove a fortios resource will apply, because the provider's version gate only fires once it is talking to the device.

What this protects against¶

Failure	Detection	Result
Pod crashes or fails readiness	seconds, via the control plane	that node withdraws its path, four remain
Node reboots or dies	9s worst case	path withdrawn on hold-timer expiry
Whole cluster down	not covered	DHCP hands out the VM as secondary

externalTrafficPolicy: Local on the Service is the setting the whole design rests on. A node advertises the anycast address only while its own pod is Ready. With Cluster, all five nodes would advertise regardless of pod health and one dead pod would silently blackhole a fifth of every query in the lab.

The readiness probe resolves gitea.heezy.local rather than checking the socket, because a pod holding :53 with no zone files would pass a TCP check and serve NXDOMAIN to its share of clients.

Measured on 2026-08-25: deleting one pod dropped the FortiGate route from five next-hops to four within seconds, and it returned to five about 18 seconds later when the DaemonSet replaced the pod.

9 seconds is the floor, not a tuning choice

MetalLB here is v0.13.3 in native mode: one speaker container, no FRR sidecar, so there is no BFD. Sub-second node-death detection needs FRR mode and a BFDProfile, which touches every VIP in the lab and is its own change on its own day.

What anycast does not solve¶

Five instances of the same thing inside one cluster still share the cluster's failure modes. A bad kustomize apply, an etcd problem or a full cold start takes all five together. That is precisely why DHCP hands out 192.168.1.29 as the secondary: it is a VM, not a pod, and it fails independently.

Resolution paths¶

LAN client on SHARED or USERS¶

LAN host
  → dns_server1 192.168.253.53 (anycast) → FortiGate → ECMP → dnsmasq pod
  → on failure, dns_server2 192.168.1.29 (VM)
  → per-host IP, or the SWAG VIP if that name is a SWAG-proxied app

Individual hosts are resolvable. nfs.heezy.local returns 192.168.1.200 and postgres.heezy.local returns 192.168.1.21, not the SWAG VIP. Names that do return 192.168.1.25 return it because k8s-services.j2 maps app names to vip_swag, which is a property of the record.

k8s pod¶

pod → CoreDNS → the four internal zones forward to 192.168.253.53, then 192.168.1.29

Pods never go through Pi-hole, so pod queries are not ad-filtered and do not appear in Pi-hole's query log.

DHCP assignment by VLAN¶

Interface	VLAN	`dns_server1`	`dns_server2`
`internal7`	SHARED 192.168.1.0/24	192.168.253.53	192.168.1.29
`users-vlan-200`	USERS 192.168.2.0/24	192.168.253.53	192.168.1.29
`pihole-vlan-201`	192.168.201.0/24	192.168.1.27	—
`prod-vlan-2000`	192.168.200.0/24	192.168.1.27	8.8.8.8
`dev-vlan-1000`	192.168.100.0/24	192.168.100.1	8.8.8.8
`dmz`	DMZ 192.168.3.0/24	1.1.1.1	8.8.8.8

DMZ never uses the internal resolver. It hands out public DNS, which is consistent with DMZ never initiating into SHARED.

Lease time on SHARED and USERS is 86400, so clients drain onto a changed resolver over a day rather than all at once.

Where each piece is configured¶

Component	Repo	Path
Zone source of truth	ansible-heezy	`roles/dnsmasq/defaults/main.yml` + `roles/dnsmasq/templates/`
VM deployment	ansible-heezy	`playbooks/dnsmasq.yml` → `playbook-dnsmasq-execution.yml`
Cluster ConfigMaps	ansible-heezy	`playbooks/dnsmasq-k8s.yml` → `playbook-dnsmasq-k8s-execution.yml`
DaemonSet and Service	heezy-k8s	`apps/dnsmasq/`
MetalLB pools, peer, advertisement	heezy-k8s	`base/metallb/metallb-config.yaml`
CoreDNS forwarding	heezy-k8s	`base/coredns-configmap.yaml`
Pi-hole upstreams	heezy-k8s	`apps/pihole/deployment.yaml`
BGP	terraform-heezy	`environments/shared/heezy/router-bgp.tf`
Firewall policies and objects	terraform-heezy	`environments/shared/heezy/firewall.tf`, `firewall-objects.tf`
DHCP	terraform-heezy	`environments/shared/heezy/dhcp.tf`

One source of truth, two render targets¶

roles/dnsmasq/defaults/main.yml defines the zone. The role renders it twice:

tasks/ubuntu.yml writes /opt/dnsmasq/hosts/* on the VM and runs docker-compose.
tasks/configmap.yml renders the same templates and applies them as the dnsmasq-conf and dnsmasq-zone ConfigMaps, which the DaemonSet mounts.

The ConfigMaps are deliberately not in the kustomization. A kustomize configMapGenerator would give the zone a second source that drifts from the Ansible templates. The consequence is an ordering rule: run playbook-dnsmasq-k8s before apps/dnsmasq is first applied, or the pods wait in ContainerCreating on ConfigMaps that do not exist.

playbooks/dnsmasq-k8s.yml targets a cluster node and drives microk8s kubectl locally, matching k8s-cleanup.yml. No ansible workflow needs a kubeconfig secret. Rendering from a nebula node is safe because the templates read inventory variables only (hostvars[...]['ansible_host'] and ['hostname']), never gathered facts, so the output is byte for byte what the VM produces.

Generated hosts files¶

File / ConfigMap key	Purpose
`hosts`	Per-host entries from Ansible inventory
`infra-services`	grafana, prometheus, proxmox, nfs, cisco-switch, Ruckus APs
`k8s-services`	k8s services: SWAG-proxied apps and dedicated VIPs
`heezy-info-overrides`	`*.heezy.info` → SWAG or direct VIP
`trentnielsen-overrides`	`*.trentnielsen.me` → SWAG or direct
`reverse-dns`	PTR records for 192.168.1.x

A zone change requires a rollout of the DaemonSet, and playbooks/dnsmasq-k8s.yml performs one whenever the ConfigMap actually changed.

inotify does not see a ConfigMap update

dnsmasq 2.92 is built with inotify, and the zone is mounted as a directory rather than with subPath, so it looks like a change should be picked up without a restart. It is not.

Kubelet updates a ConfigMap volume by writing a new timestamped directory and atomically re-pointing the ..data symlink at it. Every file dnsmasq opened is a symlink into ..data, so nothing it is watching is ever modified in place and no event it cares about fires. Mounting a directory rather than using subPath gets you updated files on disk; it does not get you a process that knows.

Measured on 2026-08-25, when LGTM moved to big-boi: the ConfigMap carried the new address for five minutes while all five pods kept answering with the old one. The VM resolver, rendered from the same templates, had already picked it up.

The roll is safe to take at any time. maxUnavailable is 1 and externalTrafficPolicy is Local, so the node being restarted withdraws its own BGP path before its pod goes down and four of five keep answering throughout.

If the ConfigMap is already correct but the pods are serving an older copy, which is the state any out-of-band apply leaves behind, the roll has to be forced because kubectl reports unchanged:

-e dnsmasq_force_rollout=true

The DaemonSet image is pinned by digest, not :latest. Five instances on a floating tag means one upstream push restarts the entire resolver fleet into an untested build.

CoreDNS¶

heezy-k8s/base/coredns-configmap.yaml. Each of the four internal zone blocks is:

heezy.local:53 {
    errors
    cache 30
    forward . 192.168.253.53 192.168.1.29 {
        policy sequential
    }
}

policy sequential is load-bearing. The forward plugin defaults to random, which would send half of every internal lookup in the cluster to the VM. Sequential means the VM is consulted only when the anycast address fails to answer.

Zone	Forward to
`heezy.local`, `heezy.info`, `trentnielsen.me`, `heezy.blog`	192.168.253.53, then 192.168.1.29
`cluster.local`	k8s internal service DNS
`*` (default)	`/etc/resolv.conf` (node upstream)

This points cluster DNS at something inside the cluster, which is worth being explicit about. It does not deadlock: the dnsmasq pods are pulled by digest by kubelet, which uses the node's own resolv.conf and never CoreDNS, so they start with cluster DNS completely down.

Pi-hole¶

Deployment: k8s heezy namespace, deployment/pihole, VIP 192.168.1.27
Upstreams: FTLCONF_dns_upstreams=192.168.253.53;192.168.1.29
Zone rules: none. Pi-hole resolves nothing internal itself.

Every internal name is a dnsmasq record, so all DNS changes go through the dnsmasq Ansible role. Nothing needs applying to Pi-hole.

MetalLB address pools¶

Pool	Addresses	`autoAssign`	Advertised by
`default-addresspool`	`.25`, `.27-.28`, `.30-.40`	true	`L2Advertisement`, scoped to this pool
`anycast-dns`	`192.168.253.53/32`	false	`BGPAdvertisement`, scoped to this pool

L2 and BGP coexist in one cluster because both advertisements are scoped. An L2Advertisement with no ipAddressPools selector matches every pool, which would make it ARP for the anycast address as well; that address must be reached over the routed BGP path only.

The default pool deliberately excludes .26 and .29, which are the Cisco 3560 management address and the dnsmasq VM. With a flat .25-.40 range and autoAssign: true, MetalLB was free to hand either to the next Service that asked.

VIP	Assignment	DNS name
192.168.1.25	SWAG (main ingress)	`swag.heezy.local`
192.168.1.26	Cisco switch mgmt, not MetalLB	`cisco-switch.heezy.local`
192.168.1.27	Pi-hole	`pihole.heezy.local`
192.168.1.28	Plex direct	`plex.heezy.local`, `nebula-plex.heezy.local`
192.168.1.29	dnsmasq VM, not MetalLB	`dnsmasq.heezy.local`, `dnsmasq-ui.heezy.local`
192.168.1.30	SWAG-Plex	`swag-plex.heezy.local`
192.168.1.31	Gitea	`gitea.heezy.local`
192.168.1.32	OpenBao	`openbao.heezy.local`
192.168.1.33	Uptime Kuma direct	`uptime-kuma-direct.heezy.local`
192.168.1.34	slskd (behind gluetun)	`slskd.heezy.local`
192.168.253.53	dnsmasq anycast	—

Half the pool is unreserved in DHCP

The FortiGate DHCP scope on internal7 is 192.168.1.10-192.168.1.199, which covers the whole pool. Dummy-MAC reservations hold .25, .27, .28 and .30. .31, .32, .33 and .34 have none.

Firewall¶

Policy	Source	Destination	Service
324	USERS (`all-users`)	`anycast-dns`	UDP/53, TCP/53
325	PROD	`anycast-dns`	UDP/53, TCP/53
326	SHARED (`heezy-shared-192.168.1.0-24`)	`anycast-dns`	UDP/53, TCP/53

309, 315 and 319 are the equivalents for shared-dnsmasq and stay while the VM does.

326 is redundant under SHARED intrazone = allow (the query enters and leaves internal7, since the BGP next-hop is a nebula node on that same interface) and is written out anyway to keep the path explicit and logged.

BGP itself needs no local-in policy: firewall local-in-policy is empty, and allowaccess does not govern port 179.

Maintenance¶

Add a record¶

Edit the template in ansible-heezy/roles/dnsmasq/templates/.
Infra host → infra-services.j2
k8s service → k8s-services.j2
*.heezy.info → heezy-info-overrides.j2
*.trentnielsen.me → trentnielsen-overrides.j2
Reverse DNS → reverse-dns.j2
Push to Gitea. Both playbook-dnsmasq-execution.yml and playbook-dnsmasq-k8s-execution.yml watch roles/dnsmasq/**, so the VM and the cluster ConfigMaps update from the same commit.
Verify against both resolvers.

A new heezy.info subdomain also needs a SWAG proxy config in heezy-containers/dockerfiles/swag/.

Debug¶

# anycast path (routed through the FortiGate)
dig @192.168.253.53 gitea.heezy.local
dig @192.168.253.53 receipts.heezy.info
dig @192.168.253.53 -x 192.168.1.21

# the VM, for comparison; both must agree
dig @192.168.1.29 gitea.heezy.local

# how many paths does the firewall have right now
get router info routing-table bgp
get router info bgp summary

# cluster side
kubectl -n heezy get pods -l app=dnsmasq -o wide
kubectl -n heezy get endpointslice -l kubernetes.io/service-name=dnsmasq
kubectl -n metallb-system logs -l component=speaker --tail=50

# note: `kubectl get bgppeer` resolves to Calico's CRD, not MetalLB's.
# Both projects ship a BGPPeer kind, so qualify it:
kubectl get bgppeers.metallb.io -n metallb-system

192.168.253.53 answers only 53/udp and 53/tcp. It does not answer ICMP, because kube-proxy forwards only the Service ports, so a ping check against it will fail while DNS is perfectly healthy. Uptime Kuma monitors it with a dns monitor asserting gitea.heezy.local resolves to 192.168.1.31, not with a ping monitor.

kubectl apply by hand is against the operating rules for this lab. Only Gitea Actions applies.

Known exceptions¶

NFS PersistentVolumes¶

k8s NFS PV manifests require spec.nfs.server to be an IP address; the driver does not accept DNS names, and the field is immutable after creation. nfs.heezy.local exists for human reference, but PV manifests will always carry 192.168.1.200.

Gitea act-runner (DinD)¶

The runner sets GITEA_INSTANCE_URL=http://gitea.heezy.local:3000, the name resolving to the MetalLB VIP. Job steps clone from gitea.heezy.local:3000 too. gitea.heezy.info:3000 resolves to SWAG, which does not proxy git.

The DinD container takes DNS from the node's resolver rather than CoreDNS, which is why the name has to be a dnsmasq record and not a *.svc.cluster.local service name.

Game servers point at the DMZ host, not at SWAG¶

Game servers are not web services, so their names skip the reverse proxy and resolve to the DMZ host directly: mc.trentnielsen.me and minecraft.trentnielsen.me → 192.168.3.13, minecraft-java.trentnielsen.me → 192.168.3.15, cs16.trentnielsen.me → 192.168.3.14, ut2k4.heezy.info and ut2k4.trentnielsen.me → 192.168.3.16. Publicly these are A records to the WAN address; LAN clients need the internal one or they hairpin.

cs16.trentnielsen.me is the game server. The leaderboard is cs16.heezy.info, a proxied CNAME into the Cloudflare tunnel, and a different name on purpose.

Architecture¶

FortiGate DHCP, SHARED and USERS
  dns_server1: 192.168.253.53  (anycast, in the cluster)
  dns_server2: 192.168.1.29    (VM, outside the cluster)

                        ┌──────────────────────────────┐
  LAN client ──────────▶│  FortiGate 192.168.1.1       │
  (SHARED / USERS)      │  ECMP, source-ip-based       │
                        └──────────────┬───────────────┘
                                       │ 5 x /32 BGP paths, AS 64512 ↔ 64513
             ┌─────────┬───────────────┼───────────────┬─────────┐
             ▼         ▼               ▼               ▼         ▼
          nebula-1  nebula-2       nebula-3        nebula-4  nebula-5
          dnsmasq   dnsmasq        dnsmasq         dnsmasq   dnsmasq     (DaemonSet)
             └─────────┴───────────────┴───────────────┴─────────┘
                          externalTrafficPolicy: Local
                   a node advertises only while its pod is Ready

  k8s pod ──▶ CoreDNS ──▶ 192.168.253.53, then 192.168.1.29 (policy sequential)

  pihole-vlan-201 / prod-vlan-2000 ──▶ Pi-hole 192.168.1.27
                                        upstreams 192.168.253.53 ; 192.168.1.29

  DMZ ──▶ 1.1.1.1 / 8.8.8.8   (never the internal resolver)

  All six dnsmasq instances render from ansible-heezy/roles/dnsmasq.