feat(network): support --aux-address on network create

[mayur-tolexo]

Part of #5012.
Docker's network create reserves auxiliary addresses so IPAM never hands them out to containers. nerdctl had no equivalent.

host-local has no exclude list, but it allocates across every range in a set, so each reserved IP is carved out by splitting the subnet range around it. The reserved name=IP pairs are matched to the subnet that contains them (so dual-stack picks the right family), rejected when they hit the network or gateway address or fall outside every subnet, and recorded so network inspect reports AuxiliaryAddresses the same way Docker does.

[mayur-tolexo]

Checked --aux-address against Docker 29.4.0 to make sure the behaviour and output line up. Reserved addresses show up in network inspect the same way, an aux-address that matches no subnet or lands on the network/gateway address is rejected the same way, and dual-stack keeps each family's aux-address on the right subnet.

Docker:

$ docker network create --subnet 10.50.0.0/24 --aux-address host1=10.50.0.5 --aux-address host2=10.50.0.6 auxnet
$ docker network inspect auxnet --format '{{json .IPAM.Config}}'
[{"Subnet":"10.50.0.0/24","Gateway":"10.50.0.1","AuxiliaryAddresses":{"host1":"10.50.0.5","host2":"10.50.0.6"}}]

$ docker network create --subnet 10.51.0.0/24 --aux-address bad=192.168.9.9 badnet
no matching subnet for aux-address 192.168.9.9

$ docker network create --subnet 10.52.0.0/24 --gateway 10.52.0.1 --aux-address g=10.52.0.1 gwnet
Error response from daemon: failed to allocate secondary ip address (g:10.52.0.1): Address already in use

$ docker network create --ipv6 --subnet 10.55.0.0/24 --aux-address v4=10.55.0.9 --subnet fd00:55::/64 --aux-address v6=fd00:55::9 dualaux
$ docker network inspect dualaux --format '{{json .IPAM.Config}}'
[{"Subnet":"10.55.0.0/24","Gateway":"10.55.0.1","AuxiliaryAddresses":{"v4":"10.55.0.9"}},{"Subnet":"fd00:55::/64","Gateway":"fd00:55::1","AuxiliaryAddresses":{"v6":"fd00:55::9"}}]

nerdctl with this PR:

$ nerdctl network create --subnet 10.50.0.0/24 --aux-address host1=10.50.0.5 --aux-address host2=10.50.0.6 auxnet
$ nerdctl network inspect auxnet --format '{{json .IPAM.Config}}'
[{"Subnet":"10.50.0.0/24","Gateway":"10.50.0.1","AuxiliaryAddresses":{"host1":"10.50.0.5","host2":"10.50.0.6"}}]

$ nerdctl network create --subnet 10.51.0.0/24 --aux-address bad=192.168.9.9 badnet
level=fatal msg="no matching subnet for aux-address 192.168.9.9"

$ nerdctl network create --subnet 10.52.0.0/24 --gateway 10.52.0.1 --aux-address g=10.52.0.1 gwnet
level=fatal msg="failed to allocate secondary ip address (g:10.52.0.1): Address already in use"

$ nerdctl network create --ipv6 --subnet 10.55.0.0/24 --aux-address v4=10.55.0.9 --subnet fd00:55::/64 --aux-address v6=fd00:55::9 dualaux
$ nerdctl network inspect dualaux --format '{{json .IPAM.Config}}'
[{"Subnet":"10.55.0.0/24","Gateway":"10.55.0.1","AuxiliaryAddresses":{"v4":"10.55.0.9"}},{"Subnet":"fd00:55::/64","Gateway":"fd00:55::1","AuxiliaryAddresses":{"v6":"fd00:55::9"}}]

host-local has no exclude list, so the reservation is enforced by splitting the range around each reserved IP; those addresses are then never handed out. With --aux-address host1=10.50.0.5 --aux-address host2=10.50.0.9 the generated host-local ranges skip .5 and .9:

rangeStart 10.50.0.1   rangeEnd 10.50.0.4
rangeStart 10.50.0.6   rangeEnd 10.50.0.8
rangeStart 10.50.0.10  rangeEnd 10.50.0.254

[mayur-tolexo]

cc @AkihiroSuda — rebased on latest main and green, could you take a look when you get a chance?

[AkihiroSuda]

Doesn't seem implemented in CNI
https://github.com/containernetworking/plugins/blob/v1.9.1/plugins/ipam/host-local/backend/allocator/config.go#L68

nerdctl-specific information could be stored in NerdctlLabels
https://github.com/search?q=repo%3Acontainerd%2Fnerdctl+NerdctlLabels&type=code

[AkihiroSuda]

Found a similar bug

• nerdctl network create --ip-range injects a non-existent property ipRange to CNI IPAM config #5068

[mayur-tolexo]

Thanks, that makes sense — I will move it out of the ipam range entirely.

Plan before I rework the PR: keep the reservation exactly as is (host-local still enforced by splitting the range around each reserved IP), and store only the name→IP pairs in a nerdctl label instead of the CNI config. Concretely a single nerdctl/network-aux-addresses label holding a JSON {subnet: {name: ip}} map, so dual-stack keeps each aux on the right subnet. network inspect decodes that label to populate AuxiliaryAddresses as before, and I would hide the internal label from the user-facing Labels so it does not surface as a user label.

Does that match what you had in mind with NerdctlLabels? And do you prefer one grouped label like above, or a flatter key?

Separately, I will send the ipRange case (#5068) as its own PR since it is derivable from rangeStart/rangeEnd and needs no storage.

[AkihiroSuda]

Thanks, JSON label sounds good (unless it grows too much to hit the length limit)

[mayur-tolexo]

Went ahead and pushed the rework along these lines. auxiliaryAddresses is gone from the ipam range entirely — the reserved pairs now live in a nerdctl/network-aux-addresses label as {subnet: {name: ip}}, and network inspect decodes that to populate AuxiliaryAddresses (and keeps the internal label out of the user-facing Labels). The reservation itself is unchanged, still done by splitting the host-local range around each reserved IP.

Happy to switch to a flatter key or rename if you would prefer something else.

[AkihiroSuda]

#5041 (comment)

[mayur-tolexo]

Built this branch and ran it end to end against Docker 29.4.0 (nerdctl on containerd 1.7.24). Nothing nerdctl-specific ends up in the CNI config anymore — the reserved pairs are in a nerdctl label and the ranges only carry host-local's own fields:

$ cat /etc/cni/net.d/default/nerdctl-auxnet.conflist
...
      "ipam": {
        "ranges": [
          [
            { "gateway": "10.50.0.1", "rangeStart": "10.50.0.1", "rangeEnd": "10.50.0.4", "subnet": "10.50.0.0/24" },
            { "gateway": "10.50.0.1", "rangeStart": "10.50.0.7", "rangeEnd": "10.50.0.254", "subnet": "10.50.0.0/24" }
          ]
        ],
        ...
      }
...
  "nerdctlLabels": {
    "nerdctl/network-aux-addresses": "{\"10.50.0.0/24\":{\"host1\":\"10.50.0.5\",\"host2\":\"10.50.0.6\"}}"
  }

(host1=.5 and host2=.6 both carved out — the range gaps at .5/.6.) grep auxiliaryAddresses over the file returns nothing.

network inspect reports the same as before, and the internal label doesn't leak into Labels:

$ nerdctl network create --subnet 10.50.0.0/24 --aux-address host1=10.50.0.5 --aux-address host2=10.50.0.6 auxnet
$ nerdctl network inspect auxnet --format '{{json .IPAM.Config}}'
[{"Subnet":"10.50.0.0/24","Gateway":"10.50.0.1","AuxiliaryAddresses":{"host1":"10.50.0.5","host2":"10.50.0.6"}}]
$ nerdctl network inspect auxnet --format '{{json .Labels}}'
{}

Docker 29.4.0 for the same thing:

$ docker network inspect d-auxnet --format '{{json .IPAM.Config}}'
[{"Subnet":"10.60.0.0/24","Gateway":"10.60.0.1","AuxiliaryAddresses":{"host1":"10.60.0.5","host2":"10.60.0.6"}}]

The reservation still holds — requesting a reserved address fails at IPAM:

$ nerdctl run --rm --net auxnet --ip 10.50.0.5 alpine true
FATA[0000] failed to create shim task: OCI runtime create failed: runc create failed: [...] failed to call cni.Setup: plugin type="bridge" failed (add): failed to allocate all requested IPs: 10.50.0.5: unknown

Dual-stack keeps each family on its own subnet, in both the inspect output and the label:

$ nerdctl network create --ipv6 --subnet 10.55.0.0/24 --aux-address v4=10.55.0.9 --subnet fd00:55::/64 --aux-address v6=fd00:55::9 dualaux
$ nerdctl network inspect dualaux --format '{{json .IPAM.Config}}'
[{"Subnet":"10.55.0.0/24","Gateway":"10.55.0.1","AuxiliaryAddresses":{"v4":"10.55.0.9"}},{"Subnet":"fd00:55::/64","Gateway":"fd00:55::1","AuxiliaryAddresses":{"v6":"fd00:55::9"}}]

# label:
{"nerdctl/network-aux-addresses": "{\"10.55.0.0/24\":{\"v4\":\"10.55.0.9\"},\"fd00:55::/64\":{\"v6\":\"fd00:55::9\"}}"}

vs docker:

$ docker network inspect d-dualaux --format '{{json .IPAM.Config}}'
[{"Subnet":"10.65.0.0/24","Gateway":"10.65.0.1","AuxiliaryAddresses":{"v4":"10.65.0.9"}},{"Subnet":"fd00:65::/64","Gateway":"fd00:65::1","AuxiliaryAddresses":{"v6":"fd00:65::9"}}]

And the rejections still line up with docker:

$ nerdctl network create --subnet 10.51.0.0/24 --aux-address bad=192.168.9.9 badnet
FATA[0000] no matching subnet for aux-address 192.168.9.9
$ nerdctl network create --subnet 10.52.0.0/24 --gateway 10.52.0.1 --aux-address g=10.52.0.1 gwnet
FATA[0000] failed to allocate secondary ip address (g:10.52.0.1): Address already in use

# docker:
no matching subnet for aux-address 192.168.9.9
Error response from daemon: failed to allocate secondary ip address (g:10.62.0.1): Address already in use