fix(dind): open the dind host port on Windows NAT nodes too (#162) - #163
Merged
Conversation
Every `docker` command in a Windows job on the default NAT network died with
Post "http://10.88.0.1:51064/v1.47/auth": dial tcp 10.88.0.1:51064: i/o timeout
blocking the Windows CI image build since 2026-07-24.
On Windows the dind Docker API is served over TCP on the container network's
gateway (runhcs supports neither a bind-mounted unix socket nor named-pipe
sharing), so a container reaching its own daemon is making an INBOUND
connection to the host — which the host firewall default-denies. openHostPort
returned early unless network.l2bridge_egress was set, so a NAT node installed
nothing and the SYN was silently dropped. Not a regression: the NAT-era rules
#144 stripped (hostFirewallRules, hyperVEgressRules) contained only Blocks, no
Allow. dind over TCP on Windows arrived with the L2Bridge work and the NAT path
was never wired up — it has never worked.
Proven on mfl-win-amd64-101 before changing anything: with a temporary inbound
allow admitting 10.88.0.0/16 to 10.88.0.1, a `docker login` that had timed out
minutes earlier on the same node succeeded. Rule removed and removal verified.
That test also proves the host firewall matches the container's real source on
this path, which is what makes the per-container scoping below work.
Two halves:
- openHostPort/closeHostPort now resolve the rule through one shared
hostPortRuleFor, which picks the host address per path — plan.HostIP on
L2Bridge, gatewayForSubnet(cfg.Subnet) on NAT. Sharing that derivation with
Manager.GatewayIP (which is what dind binds) is deliberate: two copies that
drifted would scope the allow to an address nothing is listening on, i.e.
reproduce this bug silently. Routing both open and close through one resolver
is what guarantees teardown deletes exactly the rule setup added.
- setup() now reports the container's address on the NAT path, read back off
the created HCN endpoint (HNS allocates it; ephemerd only pins addresses on
L2Bridge). SetupResult.IP was empty there, so even with the branch fixed
there would have been no address to scope to.
The /32 scoping #152 established is preserved and extended, not relaxed. Every
job container on a NAT node shares 10.88.0.0/16 and can address the gateway, so
a subnet-wide allow would let one job port-scan the gateway's ephemeral range
and drive another job's unauthenticated Docker daemon — exactly the cross-job
break #152 closed. Malformed or missing addresses still fail closed rather than
widening. hostPortRulePrefix keeps its historical "-l2b-" token so the shutdown
sweep still finds allows leaked by the previously-running build.
Tests: the NAT path opens a correctly-scoped rule, the scope is a /32 and never
the subnet, the localip tracks a relocated container subnet and agrees with
GatewayIP, teardown targets exactly what setup added and cannot collide with a
concurrent job, the NAT path fails closed on a bad address, and the L2Bridge
path is unchanged including its no-plan no-op. Verified they fail against the
pre-fix behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #162.
On a Windows node running the default NAT container network, a job container cannot reach its own dind Docker API. Every Windows
dockercommand times out:This has blocked
build-images.yml's Windows job since ~2026-07-24 — the last successful Windows CI image build was 2026-07-12.Proven on the live node, not inferred
10.88.0.1:51098, job container endpoint at10.88.10.204(gw10.88.0.1, /16). So it is a drop, not a closed port — consistent withi/o timeoutrather thanconnection refused.dir=in action=allow protocol=TCP localip=10.88.0.1 remoteip=10.88.0.0/16), re-dispatchedbuild-images.yml, andLogin to Docker Hubwent green and proceeded toBuild and push. The run was then cancelled, the rule deleted, and its absence verified — the node is back to onlyephemerd-metrics-9090, with nothing restarted.That test established a second thing the fix depends on: on the container→gateway path the host firewall matches the container's real
10.88.x.ysource. That is what makes per-container scoping viable here, unlike egress, which is SNAT'd (#144's finding).Not a regression — it never worked on NAT
hostFirewallRulesandhyperVEgressRules, removed by #144, contained only block rules — outbound RFC1918 blocks and inbound control-port blocks, zero allows. #144 removed nothing that was serving dind inbound.openHostPortwas born L2Bridge-only in2bcf222, part of the L2Bridge work. Windows dind-over-TCP arrived in that same effort and the NAT branch was simply never wired up. One bug, always present, invisible until a Windows job actually used Docker — which is why our own release builds pass on that node whiledocker logindoes not.The change, in two halves
Fixing the branch alone would not have worked.
firewall_windows.go—openHostPort/closeHostPortnow resolve through one sharedhostPortRuleFor, which picks the host address per path:plan.HostIPon L2Bridge,gatewayForSubnet(cfg.Subnet)on NAT. That derivation is now shared withManager.GatewayIP()rather than duplicated — two copies that drifted would scope the allow to an address nothing listens on, silently reproducing this exact bug. Routing open and close through one resolver is also what guarantees teardown deletes precisely what setup added.network_windows.go—SetupResult.IPwas empty on NAT, because ephemerd only pins addresses on L2Bridge and HNS allocates on NAT.setup()now reads the address back off the created HCN endpoint, with fallbacks by ID then by name.Scoping
The
/32per-container scoping from #152 is preserved and extended to NAT, which needs it just as much: all job containers share10.88.0.0/16and can address the gateway, so a subnet-wide allow would let one job scan the gateway's ephemeral range and drive another job's unauthenticated Docker daemon. Malformed or missing addresses still fail closed.hostPortRulePrefixdeliberately keeps its-l2b-token: it is the only handle the shutdown sweep has on allows leaked by a previously running build, so renaming it would strand them.Tests
Six new tests in
hostport_windows_test.goplus one innetworking_test.go, verified to fail against the pre-fix behaviour by temporarily restoring the early return:TestHostPortRuleFor_L2BridgeUnchangedstayed green in both states, confirming the L2Bridge path is untouched.GOOS=windows go build ./...,GOOS=linux go build ./..., bothgo vets, and the fullgo test ./...are clean. (GOOS=darwin go buildfails inCode-Hex/vzfor want of cgo + the macOS SDK — pre-existing.)Behaviour change worth weighing
With
SetupResult.IPnow load-bearing on NAT, a Windows job whose endpoint address cannot be read will fail to provision rather than starting and silently losing Docker. That matches the codebase's fail-closed posture, and the address comes straight from HNS with two fallbacks — but it is a new way a Windows job can fail.Verifying after deploy
build-images.yml; the WindowsLogin to Docker Hubstep should pass.Get-NetFirewallRule -DisplayName 'ephemerd-egress-l2b-hostport-*'shows exactly one rule per running job, withRemoteAddress= the container's single address (not10.88.0.0/16) andLocalPort= the dind port.