Skip to content

fix(dind): open the dind host port on Windows NAT nodes too (#162) - #163

Merged
luthermonson merged 1 commit into
mainfrom
fix/windows-nat-dind-port
Aug 16, 2026
Merged

fix(dind): open the dind host port on Windows NAT nodes too (#162)#163
luthermonson merged 1 commit into
mainfrom
fix/windows-nat-dind-port

Conversation

@luthermonson

Copy link
Copy Markdown
Contributor

Fixes #162.

On a Windows node running the default NAT container network, a job container cannot reach its own dind Docker API. Every Windows docker command times out:

Post "http://10.88.0.1:51064/v1.47/auth": dial tcp 10.88.0.1:51064: i/o timeout

This has blocked build-images.yml's Windows job since ~2026-07-24 — the last successful Windows CI image build was 2026-07-12.

Proven on the live node, not inferred

  1. The listener exists. Captured during a real job: dind listening on 10.88.0.1:51098, job container endpoint at 10.88.10.204 (gw 10.88.0.1, /16). So it is a drop, not a closed port — consistent with i/o timeout rather than connection refused.
  2. The firewall is the cause. Added one temporary scoped rule (dir=in action=allow protocol=TCP localip=10.88.0.1 remoteip=10.88.0.0/16), re-dispatched build-images.yml, and Login to Docker Hub went green and proceeded to Build and push. The run was then cancelled, the rule deleted, and its absence verified — the node is back to only ephemerd-metrics-9090, with nothing restarted.

That test established a second thing the fix depends on: on the container→gateway path the host firewall matches the container's real 10.88.x.y source. That is what makes per-container scoping viable here, unlike egress, which is SNAT'd (#144's finding).

Not a regression — it never worked on NAT

hostFirewallRules and hyperVEgressRules, removed by #144, contained only block rules — outbound RFC1918 blocks and inbound control-port blocks, zero allows. #144 removed nothing that was serving dind inbound.

openHostPort was born L2Bridge-only in 2bcf222, part of the L2Bridge work. Windows dind-over-TCP arrived in that same effort and the NAT branch was simply never wired up. One bug, always present, invisible until a Windows job actually used Docker — which is why our own release builds pass on that node while docker login does not.

The change, in two halves

Fixing the branch alone would not have worked.

  • firewall_windows.goopenHostPort/closeHostPort now resolve through one shared hostPortRuleFor, which picks the host address per path: plan.HostIP on L2Bridge, gatewayForSubnet(cfg.Subnet) on NAT. That derivation is now shared with Manager.GatewayIP() rather than duplicated — two copies that drifted would scope the allow to an address nothing listens on, silently reproducing this exact bug. Routing open and close through one resolver is also what guarantees teardown deletes precisely what setup added.
  • network_windows.goSetupResult.IP was empty on NAT, because ephemerd only pins addresses on L2Bridge and HNS allocates on NAT. setup() now reads the address back off the created HCN endpoint, with fallbacks by ID then by name.

Scoping

The /32 per-container scoping from #152 is preserved and extended to NAT, which needs it just as much: all job containers share 10.88.0.0/16 and can address the gateway, so a subnet-wide allow would let one job scan the gateway's ephemeral range and drive another job's unauthenticated Docker daemon. Malformed or missing addresses still fail closed.

hostPortRulePrefix deliberately keeps its -l2b- token: it is the only handle the shutdown sweep has on allows leaked by a previously running build, so renaming it would strand them.

Tests

Six new tests in hostport_windows_test.go plus one in networking_test.go, verified to fail against the pre-fix behaviour by temporarily restoring the early return:

--- FAIL: TestHostPortRuleFor_NATOpensAScopedAllow
    NAT path installed no host-port allow — this is #162 ...
--- FAIL: TestHostPortRuleFor_NATNeverScopedToTheSubnet
--- FAIL: TestHostPortLocalIP_NATTracksTheConfiguredSubnet
--- FAIL: TestHostPortRuleFor_TeardownTargetsExactlyWhatSetupAdded
--- FAIL: TestHostPortRuleFor_NATFailsClosed

TestHostPortRuleFor_L2BridgeUnchanged stayed green in both states, confirming the L2Bridge path is untouched.

GOOS=windows go build ./..., GOOS=linux go build ./..., both go vets, and the full go test ./... are clean. (GOOS=darwin go build fails in Code-Hex/vz for want of cgo + the macOS SDK — pre-existing.)

Behaviour change worth weighing

With SetupResult.IP now load-bearing on NAT, a Windows job whose endpoint address cannot be read will fail to provision rather than starting and silently losing Docker. That matches the codebase's fail-closed posture, and the address comes straight from HNS with two fallbacks — but it is a new way a Windows job can fail.

Verifying after deploy

  1. Dispatch build-images.yml; the Windows Login to Docker Hub step should pass.
  2. During a job: Get-NetFirewallRule -DisplayName 'ephemerd-egress-l2b-hostport-*' shows exactly one rule per running job, with RemoteAddress = the container's single address (not 10.88.0.0/16) and LocalPort = the dind port.
  3. After the job the rule is gone; after a daemon restart none survive (prefix sweep).

Every `docker` command in a Windows job on the default NAT network died with

    Post "http://10.88.0.1:51064/v1.47/auth": dial tcp 10.88.0.1:51064: i/o timeout

blocking the Windows CI image build since 2026-07-24.

On Windows the dind Docker API is served over TCP on the container network's
gateway (runhcs supports neither a bind-mounted unix socket nor named-pipe
sharing), so a container reaching its own daemon is making an INBOUND
connection to the host — which the host firewall default-denies. openHostPort
returned early unless network.l2bridge_egress was set, so a NAT node installed
nothing and the SYN was silently dropped. Not a regression: the NAT-era rules
#144 stripped (hostFirewallRules, hyperVEgressRules) contained only Blocks, no
Allow. dind over TCP on Windows arrived with the L2Bridge work and the NAT path
was never wired up — it has never worked.

Proven on mfl-win-amd64-101 before changing anything: with a temporary inbound
allow admitting 10.88.0.0/16 to 10.88.0.1, a `docker login` that had timed out
minutes earlier on the same node succeeded. Rule removed and removal verified.
That test also proves the host firewall matches the container's real source on
this path, which is what makes the per-container scoping below work.

Two halves:

- openHostPort/closeHostPort now resolve the rule through one shared
  hostPortRuleFor, which picks the host address per path — plan.HostIP on
  L2Bridge, gatewayForSubnet(cfg.Subnet) on NAT. Sharing that derivation with
  Manager.GatewayIP (which is what dind binds) is deliberate: two copies that
  drifted would scope the allow to an address nothing is listening on, i.e.
  reproduce this bug silently. Routing both open and close through one resolver
  is what guarantees teardown deletes exactly the rule setup added.

- setup() now reports the container's address on the NAT path, read back off
  the created HCN endpoint (HNS allocates it; ephemerd only pins addresses on
  L2Bridge). SetupResult.IP was empty there, so even with the branch fixed
  there would have been no address to scope to.

The /32 scoping #152 established is preserved and extended, not relaxed. Every
job container on a NAT node shares 10.88.0.0/16 and can address the gateway, so
a subnet-wide allow would let one job port-scan the gateway's ephemeral range
and drive another job's unauthenticated Docker daemon — exactly the cross-job
break #152 closed. Malformed or missing addresses still fail closed rather than
widening. hostPortRulePrefix keeps its historical "-l2b-" token so the shutdown
sweep still finds allows leaked by the previously-running build.

Tests: the NAT path opens a correctly-scoped rule, the scope is a /32 and never
the subnet, the localip tracks a relocated container subnet and agrees with
GatewayIP, teardown targets exactly what setup added and cannot collide with a
concurrent job, the NAT path fails closed on a bad address, and the L2Bridge
path is unchanged including its no-plan no-op. Verified they fail against the
pre-fix behaviour.
@luthermonson
luthermonson merged commit 2f53386 into main Aug 16, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sec(dind): Windows NAT nodes never open the dind host port, so every Windows docker command times out

1 participant