[chruffins]

Summary

PR 5 of 6. Ingress for macOS guests, with fixed guest addresses. Base: main (#498, #499, #500 and #506 have merged). Three commits: ingress (9c956f2), per-guest vmnet networks (9f35f32), and stopping a macOS guest's shim on delete instead of killing it (0897c3f).

This replaces the earlier vsock port-forwarding design for this PR. Ingress is how callers already reach Linux guests, and native measurements showed it fits macOS guests too:

• Guest services can listen on the guest's network interface, as on Linux.
• The direct vmnet path was faster than the vsock forward: 0.22 vs 0.37 ms CDP round trip, and no per-connection setup cost.

The forwarding implementation is kept on archive/macos-guest-ports-vsock (ff629f7) if it's wanted later as a generic hypeman feature.

What changes

• Ingress targets macOS guests. The resolver returns the guest's address, and Caddy resolves it through hypeman's DNS on every request, as for Linux.
• Each networked macOS guest gets its own vmnet network with a fixed address (macOS 26+). Hypeman assigns a /29 from network.macos_subnet_cidr (default 192.168.128.0/20), stores the guest address with the instance and keeps it through stop and standby, as Linux does. vz-shim creates a shared-mode vmnet network for that subnet with a DHCP reservation for the guest's MAC (infinite lease). This replaces reading vmnet's shared lease file (/var/db/dhcpd_leases), which could return an expired address or one vmnet had since given to another guest.
• No wrong-guest routing on address reuse. Delete releases an address into a 5-minute hold, longer than Caddy's 1-minute upstream cache, before any guest can get it. If writing the hold fails, the delete fails. A memory restore whose saved address another guest now holds is refused; a cold-boot restore gets a new address. Clones and forks get new addresses.
• Delete stops the shim instead of killing it. A vz-shim owns its guest's vmnet network. Deleting a running guest SIGKILLed the shim, and macOS then kept the network idle and refused that subnet ("unable to fulfill network") until the host restarts. Delete now asks a networked macOS guest's shim to stop the VM and exit (vmm.shutdown), as stop already does, and kills it only if it doesn't exit in 2 s.
• Code-Hex/vz v4, pinned at upstream main (787f319). vmnet networks aren't in any tagged release (v3.8.0 predates them; v4 is untagged), so the darwin code moves from v3.7.1 to that commit. The only v4 API change affecting vz-shim is state notifications: the caller now owns the channel and events can be dropped or stale, so the shim re-reads the state after each one. Move to the v4.0.0 tag when it's released.
• Caddy and DNS start on macOS-only servers. CreateIngress no longer returns 501 there, and the API's macOS-target pre-check is gone.
• The Caddy daemon manager works on macOS. It found and checked its process through /proc; on darwin it now uses ps, the same way hypeman already finds vz-shims. The Linux helpers moved unchanged into daemon_linux.go.
• Caddy for macOS. macOS builds still don't embed Caddy. make build-caddy-darwin builds the same version and DNS modules as Linux into bin/caddy. Hypeman looks for it next to its own binary, then on PATH, and fails to start without it.

Linux code touched

• lib/ingress/daemon.go: isProcessRunning and findCaddyPID moved verbatim to daemon_linux.go.
• lib/instances/ingress_resolver.go: the macOS refusal is removed; Linux resolution is unchanged.
• cmd/api/main.go: ingress initialization is no longer skipped in macOS-only mode; unchanged elsewhere.
• Makefile: new build-caddy-darwin target.
• cmd/api/config: new network.macos_subnet_cidr, validated as IPv4 /8–/29 and not overlapping network.subnet_cidr.
• lib/instances/delete.go: holds a macOS guest's address before removing its data; Linux deletes are unchanged.
• go.mod: vz v3 → v4 (darwin-only code), plus the golang.org/x versions v4 requires (x/sys v0.46, x/crypto v0.53 and related).

Testing

• TestIngressResolverResolvesMacOSGuestLease: a guest resolves to its address; a guest without one has no target.
• TestMacOSAddressAssignment, TestMacOSAddressReuseHold, TestMacOSAddressAvailable: first free /29, skipping other instances, creates in progress and held addresses; reuse only after the hold.
• TestMacOSMemoryRestoreRefusesAddressNowInUse and a network-address case in the snapshot payload test; both fail without their checks.
• TestValidateMacOSSubnetCIDR, and the VZ starter test now checks the subnet and address passed to vz-shim.
• TestCaddyPIDFromPS, TestIsProcessRunningOnDarwin (darwin CI): ps parsing, and a zombie not counted as running.
• Full suite against main: the same failures, plus TestRegistryLayerCaching (Docker Hub rate limit) and TestCreateImage_InvalidTag (fails on main too when run alone).
• Native build and darwin tests on Apple silicon (lib/hypervisor/vz, cmd/vz-shim, macOS lib/instances): pass.
• Darwin type-check of lib/ingress, lib/instances and cmd/api (tests included): clean.
• Native QA (Apple silicon, macOS 27.0 host and guest): literal and {instance} pattern rules routed to a running guest's lease; an unknown instance got 503; Caddy started and stopped with hypeman. CDP round trip through Caddy was 0.27 ms p50 versus 0.22 ms direct to the guest IP, with the same large-payload throughput.
• Native QA of the vmnet networks (macOS 27.0): the guest got its assigned address with an infinite lease, router and DNS; reached the internet; kept the address through standby/restore and stop/start (a stopped guest still reports it); ingress through Caddy running as a regular user returned 200. The default pool also gets a router and internet. An earlier /30 version got no router option from vmnet's DHCP, so guests had no default route; that's why networks are /29.
• Two guests on the QA host, with each guest answering requests with its own name: the second guest got the next /29 and couldn't reach the first. Deleting it while running and recreating its name gave the new guest the next address, not the held one. Across 175 requests to that name's ingress route during the delete and recreate, every response came from the deleted guest, the new guest, or failed (timeouts while Caddy retried the held address, then 502); none came from the other guest. The other guest's route reached it every time (402/402). After the hold, a new guest reused the first subnet.
• TestDeleteStopsMacOSShimBeforeKilling: delete asks a networked macOS guest's shim to stop; Linux guests are killed as before. Fails without the fix. On the QA host, deleting running guests left their networks fully stopped, where before the fix each delete leaked its subnet.

Caveats

• Caddy needs local network access to dial guests. On the QA host, a hypeman started over SSH lost it once the session ended, and a launchd daemon running as a regular user never had it; Caddy returned 502 ("no route to host"). Allowing the guest pool with AllowedEthernetLocalNetworkAddresses (Apple TN3179) fixed it without root. Documented in docs/macos-experimental.md.

• Starting a macOS guest needs the hypeman user logged in on the host; with nobody logged in after a restart, VZ refused with Code=-9.

• Saving a macOS guest's memory (standby, memory snapshots) needs that user's screen unlocked; with it locked, VZ fails with Code=11 ("permission denied") because it cannot create its Secure Enclave key. Documented with the sysadminctl -screenLock off setting.

• Guests can reach the host's own listeners, including Caddy's ports, sshd and Screen Sharing. Hosts running guests for different tenants should not expose other host services on guest networks. Documented in docs/macos-experimental.md.

• Guests are on separate vmnet networks, so they can't reach each other's addresses.

• Instances started before this change keep the shared NAT and report no address until their next start. A memory snapshot saved on the shared NAT restores on it.

• Changing network.macos_subnet_cidr doesn't move existing guests; they keep their stored addresses.

• A shim that dies while its guest runs (crash, external SIGKILL) still leaks its network: that guest can't start and its subnet can't be reused until the host restarts. Not handled automatically yet.

Stack

Each PR builds on the one below it. macOS logic lives in its own files (macos*.go, *_darwin.go, exec_lifecycle.go); shared Linux files get small hooks such as if stored.MacOS != nil.

1. macOS guests 1/6: VZ runtime, machine images and instance lifecycle #498 VZ runtime, machine images and instance lifecycle
2. macOS guests 2/6: OCI machine-image integration #499 OCI machine images
3. macOS guests 3/6: Darwin guest service, readiness and exec #500 Darwin guest agent, readiness and exec
4. macOS guests 4/6: snapshots, schedules and stopped forks #506 snapshots, standby and stopped clones
5. macOS guests 5/6: ingress and fixed addresses for macOS guests #514 ingress for macOS guests
6. macOS guests 6/6: experimental native image builds and release tooling #503 machine-image builds (not needed for v0; can land later)

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 23c6890. Configure here.}

[sjmiller609]

a few design questions:

• generic TCP forwarding: should this support arbitrary TCP services, with an SSH-like CLI?

  # SSH today
  ssh -N -L 8080:127.0.0.1:3000 user@vm
  
  # proposed Hypeman equivalent
  hypeman port-forward my-vm 8080:3000

  both expose the remote service on local localhost:8080, without interpreting HTTP.

• comparison with exec: exec already streams stdin/stdout over WebSocket → gRPC/vsock. should forwarding follow that pattern, streaming socket bytes instead?

• our platform use case: do we actually need this feature, or could metro-api reach kernel-images-api through guest networking, as on Linux? what prevents that existing routing pattern? I like this feature for hypeman as it seems very useful, but also it seems more complicated if not actually necessary.

• performance is important: CDP is latency-sensitive; screenshots and other large payloads need both low latency and high throughput. So any new network path like this if it's used for our platform use cases and not just simple things needs careful performance comparisons.

[sjmiller609]

Is it doable to have Hypeman own the IP allocation as on Linux instead of using vmnet DHCP? The main concern is accidentally routing to the wrong VM during lifecycle changes or DHCP lease expiry. I think it's worth checking on, as it would be ideal to have static guest private IPs like it works on linux. Would be OK with DHCP if this is the only way that works, but we have to pay close attention to the potential cases for wrong-VM routing.

• mac-specific: the current DHCP lease parser can return expired/superseded addresses, including during startup.
• shared with Linux: Caddy caches upstream IPs for one minute, so address reuse can send old traffic to a replacement VM. fixed ingress rules also follow reusable names rather than immutable IDs.

macOS 26+ supports custom vmnet networks with DHCP reservations or static addressing. could we investigate that instead of discovering ownership from the lease file? reservations can’t change while a network is active, so we’d need to account for that.

ideally: persist instance→IP allocation, retain it through stop/standby, and coordinate reuse with cache expiry/invalidation.

can we test stop/delete/recreate, restore, and same-name reuse while continuously requesting the old route? it should reach the intended VM or fail, never return another VM’s response.

[chruffins]

@sjmiller609 good call. I'm going to spike-test it now: one vmnet network per guest (macOS 26+), with a hypeman-assigned subnet and a DHCP reservation for the guest's MAC, so the IP is fixed and kept through stop/standby instead of being read from the lease file. I'll report back with results, including whether save/restore works with the new attachment.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	github.com/​Code-Hex/​vz/​v4@​v4.0.0-20261028120916-787f319dcf12
	golang.org/​x/​sys@​v0.45.0 ⏵ v0.46.0	+1
	golang.org/​x/​sync@​v0.20.0 ⏵ v0.21.0

View full report

[chruffins]

@sjmiller609 the spike test worked! the only caveat is that our dependency for vz bindings doesn't have a release that supports vmnet so it's pinned to a specific commit