Game Server On Macvlan¶
How a DMZ game server runs as a Kubernetes pod with a real address on VLAN 3, and the failures that shape look like nothing else in the lab.
Manifests live in heezy-k8s/apps/<game>/. cs16 is the worked example and the pattern the rest
follow.
The shape¶
Every game pod gets two interfaces. eth0 is the ordinary Calico interface Kubernetes gives it.
net1 is a macvlan child of enp0s31f6.3, the addressless VLAN 3 sub-interface present on all five
nebula nodes, and carries a real DMZ address of its own.
That second interface is the whole point. A pod behind a LoadBalancer VIP would have its east-west traffic forwarded by the node, where no firewall can see it. A pod holding a DMZ address sends to SHARED off-subnet, so the packet goes to the gateway and the FortiGate decides.
Three pieces make it work, and all three are load bearing:
| Piece | Where | What breaks without it |
|---|---|---|
NetworkAttachmentDefinition, macvlan, static IPAM |
network.yaml |
No DMZ address at all |
default-route in the k8s.v1.cni.cncf.io/networks annotation |
deployment.yaml |
Egress to SHARED goes out eth0 over the cluster network, invisibly |
dnsPolicy: None with public resolvers |
deployment.yaml |
The pod resolves internal zones through CoreDNS, and nothing looks wrong |
The address is pinned in the NetworkAttachmentDefinition, not leased. DMZ DHCP starts at
192.168.3.10, so anything below that is safe to pin. Replicas must stay at 1: a second pod would
ask for an address already in use.
What a shared network namespace costs you¶
Containers that were separate on a VM share one namespace in a pod, and every port they bind collides, not just the one you were thinking about.
| Game | What collided | How it looked |
|---|---|---|
| bedrock | The IPv6 listener, which defaults to SERVER_PORT plus one |
Second world exited with Port [19133] may be in use, naming the port it wanted rather than the one the first world had taken |
| minecraft-java | RCON_PORT, enabled on 25575 by default whether or not anyone uses it |
Would have failed the second container on a port nobody configured |
| ut2k4 | Nothing, but port 80 needs net.ipv4.ip_unprivileged_port_start=0 |
Docker sets that inside a container, kubernetes does not, so a uid 1000 process could not bind the web admin |
Read the image's defaults, not just its documented ports, before putting two of anything in one pod.
The binding trap¶
A server that answers every probe and refuses every join
hlds, and anything else that picks its own bind address, will choose the Calico address on
eth0. Its outbound connections are then sourced from 10.1.x.x while being routed out net1.
The FortiGate sees a cluster address arrive on the dmz interface, drops it on reverse path
check, and the server never reaches the internet service it depends on.
For cs16 that service is Steam. The startup log says so directly:
With sv_lan 0 a server that cannot reach Steam cannot authenticate anyone, so every client sits on
"Retrying connection to server" and then fails.
What makes this expensive to diagnose is that nothing else is wrong. A2S queries are connectionless
and need no Steam at all, so the server answers A2S_INFO, getchallenge and ping correctly from
the DMZ, from USERS, and from the internet. Ping works. The VIP works. The firewall logs an accept.
Every test short of an actual join passes.
Confirming it¶
Two checks, both from inside the pod. The socket table is the unambiguous one:
kubectl exec -n dmz <pod> -c <container> -- sh -c "cat /proc/net/tcp | awk 'NR>1 {print \$2, \$3, \$4}'"
Local addresses are little-endian hex. F490010A is 10.1.144.244, the Calico address, and state
02 is SYN_SENT. Outbound connections stuck there, from that address, are this bug. When it is
fixed every local address reads 0203A8C0, which is 192.168.3.2.
The reachability difference on its own:
kubectl exec -n dmz <pod> -c <container> -- sh -c "nc -s 10.1.144.244 -w 4 -z 1.1.1.1 443 && echo ok || echo fail"
A socket bound to the pod's Calico address cannot reach the internet. One bound to its DMZ address can.
Fixing it¶
Pass the DMZ address as a bind address, and pass it before the argument that selects the game:
Ordering is not cosmetic. The engine reads -ip while parsing the command line and stops looking
once the game is set, so -ip placed after -game is accepted and silently ignored: the game
socket stays on 0.0.0.0 and the banner still names the Calico address.
Confirm the fix from outside rather than from the log. A2S_INFO reports a vac flag, and a server
that reached Steam reports vac=1:
The banner should also name the DMZ address:
Expect this for any game that talks to an authoritative service on the internet. It presents as a network fault and is a bind-address fault.
The error queue trap¶
A query port that holds one packet at a time
An engine that sets IP_RECVERR on a UDP socket tells the kernel to keep every ICMP error that
socket receives, charged against its receive buffer, until the application reads it back with
MSG_ERRQUEUE. A plain read reports the error once and leaves the bytes behind. If the engine
never reads the error queue, internet traffic fills the buffer a little every day, and the
socket ends up dropping any datagram that arrives while another is still waiting.
ut2k4's query socket, 7778, is one of these. Server-list pollers close their socket before the reply lands, each of those comes back as an ICMP port unreachable, and the server logs roughly 55 a day as:
Left alone, that fills the 212992 byte buffer within a week. UT2004 master servers query in bursts of
three and get one answer. An Uptime Kuma query that lands in the same few milliseconds is lost, so
DMZ-UT2K4 reports Failed all 2 attempts for one check and recovers on the next. Two seconds after
each of those, the server logs an Open myLevel from the node Uptime Kuma runs on: gamedig falling
back to the game port, which never answers a query. That connection times out 200 seconds later.
Confirming it¶
On the node running the pod, list the server's sockets from inside its network namespace:
A healthy socket reads Recv-Q 0. This one reads Recv-Q pinned just under the buffer size
(skmem:(r212608,rb212992,...,d12504)), with the d drop counter climbing, while the server keeps
answering whatever gets through. It shows from outside too: five queries sent to 7778 at once get
one reply, where a healthy socket answers all five.
Fixing it¶
Drop the errors before they reach the socket. ut2k4 runs an init container that adds this rule to the pod's network namespace:
The game image has no iptables, so the init container runs calico/node, which has it and is already
on every node. It is the only container in the pod with NET_ADMIN, and it exits before the server
starts. The rule belongs to the pod's network namespace, so it outlasts container restarts and goes
with the pod. Recreating the pod empties the buffer; the rule keeps it empty.
Only code 3 is dropped. The game socket sets IP_RECVERR as well, but ignores port unreachable while
AllowPlayerPortUnreach=False. TCP answers a closed port with a reset, and fragmentation needed still
gets through.
cs16 and both bedrock worlds do not have this problem. Their sockets sit at Recv-Q 0 with no drops.
Check any new game the same way a week after it goes live.
Cutting traffic over¶
Retargeting is three changes, and each one carries a different set of players. Missing any of them leaves part of the player base on the old host.
| Players | What steers them | Where |
|---|---|---|
| Internet | VIP mappedip |
terraform-heezy/environments/dmz/heezy/firewall-objects.tf |
| LAN, by name | dnsmasq split-DNS override | ansible-heezy/roles/dnsmasq/defaults/main.yml |
| LAN, by address | nothing, it is already allowed | policy Users to DMZ |
The firewall does not steer LAN players
Policy Users to DMZ is USERS to DMZ, all sources, all destinations, all services, and it sits
above every per-game USERS policy. Those per-game policies never match. What a LAN player
reaches is decided entirely by what dnsmasq answers, so the split-DNS override is the LAN
cutover.
The public record needs no change. Game names under trentnielsen.me are unproxied A records to the
WAN address, so they follow the VIP.
Keep the VM running and unchanged for a week after the cutover. Rollback is pointing the VIP and the override back at it.
Verification¶
Run all of these before considering a game migrated. The last one is the only one that exercises the path the binding trap breaks.
| Check | Expected |
|---|---|
kubectl get pod -o jsonpath={.metadata.annotations} |
net1 holds the pinned DMZ address |
| Delete the pod, check again | Same address, including on a different node |
ip route get <a SHARED address> in the pod |
via <DMZ gateway> dev net1 |
nc -z -w 4 <a SHARED address> 22 from the pod |
Blocked, and logged by policy dmz-to-shared-deny |
| Public name lookup in the pod | Resolves. Internal names return NXDOMAIN |
| A2S query from the DMZ and from the internet | Answers, vac=1 |
| A real client connects and plays | Joins |
ss -uamn in the server's network namespace, a week in |
Recv-Q 0 on every UDP socket |
Denied DMZ traffic is only visible because dmz-to-shared-deny logs it. fwpolicy-implicit-log is
disabled globally and should stay that way, so without that policy a blocked attempt leaves no trace
and only a packet capture shows it.
Probes¶
Nothing that reaches these pods over the cluster network can probe them. kubelet probes from the
node to the pod's cluster address; the request arrives on eth0, the server answers, and the reply
leaves by the default route, which is the DMZ interface by design. It goes to the FortiGate and
never comes back. An httpGet or tcpSocket probe therefore times out against a server that is
answering perfectly well, and kills it.
This is the same property the design rests on, met from the other side.
Probe over loopback instead. 127.0.0.1 never leaves the network namespace.
| Server | Probe |
|---|---|
| cs16, both bedrock worlds | grep -q ':<port hex> ' /proc/net/udp. UDP only, nothing to connect to |
| fastdl sidecar | wget -q -O /dev/null http://127.0.0.1/ |
| Java worlds | mc-monitor status --host 127.0.0.1 --port <port>, which speaks the server list protocol |
| ut2k4 | grep -q ':1E61 ' /proc/net/udp, carried from its compose healthcheck |
Prefer a check that speaks the game's protocol over one that proves a socket is bound. A server can hold its port open while being unable to serve, which is exactly what the adventure world did for four months.
Initial delays have to cover world load. Forge with 25 mods takes minutes, and a probe that fires first restarts the server in a loop that looks exactly like the crash it is meant to catch.
Backing up the data¶
playbook-game-backup-execution.yml, 01:00 UTC daily, one archive per world.
The volumes are ReadWriteOnce, so a backup job cannot mount one while the server holds it. Rather than stop every game nightly, the backup reads the volume out through the container:
/backup-src is a read-only mount of the whole volume that every game container carries for this
purpose. Every other mount in these pods is a subPath, which is right for the server and useless
for a backup: the restore extracts over the volume root, so that is the shape the archive needs.
It runs on a cluster node, not the CI runner, which has neither cluster access nor /nfs.
Check the size, not the exit code
tar will write an error into a file and exit happy. The backup enforces an absolute floor of
1 MB and a relative floor against the previous archive: half of last night's size is a truncated
read, not a smaller world. A failing archive is deleted rather than kept and counted.
Monitoring¶
Uptime Kuma's port monitor cannot check UDP, so a bedrock world has no port to test. Use a gamedig
monitor with game minecraftbe, which speaks the server query and returns the world name and player
count. Set gamedigGivenPortOnly, or GameDig probes a range around the port and each of the two
worlds reports the other's state.
ut2k4 is the exception: its monitor leaves gamedigGivenPortOnly off, because GameDig's unreal2
protocol queries the game port plus one (7778), and pinning the port removes that offset. When the
7778 query fails, GameDig also tries 7777, so a lost query reads Failed all 2 attempts. A single
one that clears on the next check is a dropped datagram, not an outage; see
the error queue trap.
Never monitor a stand-in. The old bedrock and cs16 monitors checked node-exporter on 9100 because it was the only TCP port on the host, and node-exporter stayed up for the four months the adventure world was down.
Restoring the data¶
The PVC is seeded from the NFS backup, never copied off the running VM. Copying proves the bytes move; only a restore proves the backup is restorable, which is what has to be true before the old host is switched off.
# dispatch playbook-game-restore-execution.yml
game_restore_name=cs16
game_restore_pvc=cs16-data
game_restore_archive= # blank takes the newest
The job refuses to run over a volume that holds anything other than lost+found, has no retries,
and prints its extract log pass or fail. Read that log rather than the run's status: confirm the
game's actual data path was extracted, not just its config directory.
Check the compose file before trusting a backup
cs16 kept its server in a Docker named volume and only its config under /opt. A backup of
/opt alone opens cleanly, lists 2,065 files, and restores a Counter-Strike server with no
Counter-Strike in it. Read each game's compose file and back up every named volume it declares.