feat(vmm): run virtio-net on vhost-net with configurable queue pairs

[kvinwang]

Problem

A CVM's virtio-net NIC cannot exceed one core's worth of packet processing, no
matter how many vCPUs it has. The VMM builds -netdev bridge,... with no
vhost=, and a device line hardcoded to virtio-net-pci,netdev=net0,mac=...
with no mq=on. Every packet is drained by QEMU's single main-loop thread:

ceiling ≈ 1 core ÷ per-packet main-loop cost

Sampled on a production CVM (phala-tdx-prod7, pid 3543413, 16 vCPU / 32 GiB,
-netdev bridge,id=net0,br=dstack-br0):

31,822 pps in, 0 drops in the sample window, qemu main thread 91%, total 721%
→ 0.91 core / 31,822 pps = 28.6 µs/packet → ceiling ≈ 35k pps

That matches the reported symptom exactly — clean at 31k pps, 36% loss at 40k.
The three busy TAPs on that host had lifetime drop rates of 26.7% / 19.3% /
17.8%, hundreds of millions of packets each. The drops are at the TAP, so
guest-side rx_dropped and softnet_stat stay at zero and the cliff looks
like a network fault rather than a host CPU limit. Outbound guest traffic uses
the same thread, so a chatty guest pays it twice.

docs/bridge-networking.md claimed this was unavoidable: "vhost-net ... is
not enabled for bridge mode. TDX encrypts guest memory, which prevents the
host kernel from performing DMA-based packet offload." That is wrong. A TDX
guest's virtio rings and buffers live in shared, unencrypted memory precisely
so a host-side backend can reach them — which is why vhost-vsock-pci, which
dstack has always used, works.

Fix

Two settings on [cvm.networking], defaulting to vhost on and one queue pair,
each overridable per VM through the deployment RPC:

[cvm]
max_net_queues = 16         # ceiling for both the default and a request

[cvm.networking]
vhost = true

Queue pairs are not a node setting: they default to the VM's vCPU count, capped
at max_net_queues, because the useful number follows the VM rather than the
host. A deployment overrides that per VM.

vmm-cli.py deploy ... --net bridge --net-queues 4
vmm-cli.py deploy ... --net bridge --net-no-vhost

vectors is derived, never configured: 2N + 2. A single queue pair emits no
mq=on or vectors= at all, so a 1-vCPU VM and any VM deployed with
--net-queues 1 keep the historical command line byte for byte.

Each backend needed different plumbing:

Mode	netdev	notes
user	unchanged	no vhost backend, no multiqueue; a node-wide default is ignored here rather than failing the launch
bridge	tap,br=,helper=,vhost=on	QEMU's bridge netdev accepts neither vhost= nor queues=; the same setuid qemu-bridge-helper works behind a tap netdev, so the VMM still needs no CAP_NET_ADMIN
bridge, >1 queue	tap,ifname=,queues=N	the helper returns one descriptor, so netd creates a persistent multi_queue TAP — now also for unfiltered nodes
bridge + libvirt filter	tap,ifname=,vhost=on	the hardcoded vhost=off is now the configured value
macvtap	tap,fds=a:b,vhost=on	the launcher opens /dev/tapN once per queue; netd creates the link with matching numtxqueues/numrxqueues
custom	verbatim	the operator owns the netdev string; queues still drives the device line, which is what makes a hand-written netdev usable with multiqueue

Because macvtap now consumes several descriptors per NIC, the fd layout moved
into one macvtap_fd_layout() that both the launcher's open list and the
-netdev arguments derive from, replacing the 3 + index convention the two
sites previously duplicated.

QEMU aborts when vhost=on cannot open /dev/vhost-net, and it does so from
inside the launcher where the reason is easy to miss. The VMM warns when the
device node is missing, but deliberately does not pre-flight its own access to
it: QEMU need not share the VMM's credentials, and with vhost defaulting to on,
a VMM that is merely not in the kvm group would otherwise refuse every
deployment on a host where QEMU can open the device fine. stat needs no
permission on the device, so the check stays a statement about the host.

Neither field affects mr_config_id, which covers only the compose hash and
instance info, so retuning a NIC does not change app identity.

Verification

All five configurations deployed as real TDX CVMs on phala-tdx-lab (kernel
6.8, QEMU 8.2.2+tdx1.1, guest image dstack-0.6.0), against an isolated bridge
and a dedicated VMM instance.

Generated command lines, read back from /proc/<qemu>/cmdline:

A default        tap,id=net0,br=dstack-perf0,helper=/usr/lib/qemu/qemu-bridge-helper,vhost=on
                 virtio-net-pci,netdev=net0,mac=c6:29:aa:27:f7:05
B --net-no-vhost bridge,id=net0,br=dstack-perf0
                 virtio-net-pci,netdev=net0,mac=ba:6e:f8:29:d5:56
C --net-queues 4 tap,id=net0,ifname=dt4a44fb6e472b,script=no,downscript=no,vhost=on,queues=4
                 virtio-net-pci,netdev=net0,mac=4a:ea:3d:87:02:f1,mq=on,vectors=10
D macvtap, 2q    tap,id=net0,fds=3:4,vhost=on
                 virtio-net-pci,netdev=net0,mac=f6:c6:a1:17:e4:2c,mq=on,vectors=6
E libvirt filter tap,id=net0,ifname=dt851b876fcf4a,script=no,downscript=no,vhost=on

Host and guest state:

	A (default)	B (no vhost)	C (4 queues)	D (macvtap 2q)
guest ethtool -l eth0 combined	1	1	4	n/a (see below)
vhost worker threads in the QEMU thread group	2	1	5	3
DHCP lease + ssh into guest	yes	yes	yes	—

The worker count is net queue pairs + 1 for vhost-vsock in every case, which
is what confirms the guest actually negotiated that many queue pairs.

netd created what was asked for, and cleaned up on VM removal:

dt4a44fb6e472b: tun type tap ... multi_queue numqueues 4 persist on user kvin
dt7472999c6242@dstack-perf0: macvtap mode bridge ... numtxqueues 2 numrxqueues 2
/proc/212991/fd/3 -> /dev/tap1759
/proc/212991/fd/4 -> /dev/tap1759      # two opens, two queues

After removing all five VMs: no dt* interfaces and no nwfilter bindings left.

Filtering still works with vhost on. docs/libvirt-network-filter.md said
flipping this bit needed "equivalent filter integration tests", so from inside
VM E, with clean-traffic bound to the TAP and vhost=on:

ping -c 3 10.90.0.1                       → 3 received, 0% loss
ip addr add 10.90.0.222/24 dev eth0
ping -c 3 -I 10.90.0.222 10.90.0.1        → 0 received, 100% loss

The nwfilter binding lives on the host TAP interface, so packets traverse it
whether QEMU or a vhost worker wrote them.

Node policy, over the RPC:

--net-queues 9         → networking queues must not exceed 8 on this node
--net user --net-queues 4 → user-mode networking does not support multiple queues

Throughput, 64-byte UDP host→guest via kernel pktgen, same 8 vCPU / 8 GiB CVM
shape on the same host (the lab host was running other tenants' CVMs, so treat
the absolute rates as noisy and the CPU columns as the signal):

target	A: helper + vhost, 1q	B: legacy netdev, no vhost	C: netd + vhost, 4q
200k	0% loss, main 0%	0% loss, main 90%	0% loss, main 0%
500k	17.8% loss, main 0%	0.3% loss, main 93%	0% loss, main 0%
1M	26.6% loss, main 0%	5.2% loss, main 91%	0% loss, main 0%

The main thread goes from saturated to idle — that is the wall coming down, and
a whole core returned to the tenant. Note that vhost alone relocates the
ceiling rather than removing it: with one queue the guest's single receive queue
becomes the limit and drops reappear at a higher rate (the guest was at 99% CPU
in the 500k row). Multiqueue is what removes them. Hence vhost on by default,
queues raised deliberately.

A/B on the customer's own host (phala-tdx-prod7, production short-connection
workload, 4 alternating rounds to suppress noise) measured earlier in this
investigation: 10,922 → 13,933 conn/s (+27.6%), with the main thread going
94% → 0%.

Review follow-ups

A self-review pass found seven issues, all fixed in this branch:

• netd fail-open. Making filter optional so netd can build unfiltered
  multiqueue TAPs meant mode = "libvirt" with an explicit filter = "" no
  longer failed — netd skipped nwfilter-binding-create and every VM booted
  onto an unbound TAP, where it previously refused to start. Config load now
  rejects an empty filter in libvirt mode.
• One-shot mode. one_shot.rs still gated its "cannot manage TAP lifecycle"
  error on network_filter.mode == Libvirt, so queues > 1 on a bridge emitted
  -netdev tap,ifname=dt…,queues=4 for a TAP nobody created. It now uses the
  same needs_netd_interface predicate as the server path.
• Tuning forced a mode. --net-queues 4 alone was rejected despite the
  flag's "default: use global config" help, and on a node whose default backend
  is not in allowed_network_modes, restating the mode tripped node policy — so
  per-VM tuning was impossible there at all. A tuning-only request now keeps the
  node's backend; policy still governs backends a caller chooses.
• A missing bridge helper broke the node. bridge_helper() hard-failed the
  launch when none of three hardcoded paths existed on the VMM's filesystem.
  With vhost defaulting on, a working bridge node whose helper lives elsewhere
  would have lost every VM on upgrade — the same mistake the /dev/vhost-net
  probe was already corrected for. It now falls back to the non-vhost bridge
  netdev with a warning.
• Warning on every launch. The "user mode supports neither vhost nor
  multiqueue" warning tested the resolved value, and the shipped vmm.toml
  sets mode = "user" with vhost = true, so it fired on every launch of a
  stock node. Inheritance being ignored there is the documented design, not a
  silent skip; the warning is gone.
• Node tuning was pinned into the manifest. resolve_requested_networks
  persisted the merged values, so setting vhost = false node-wide to roll back
  reached VMs deployed with no override but not those deployed with
  --net bridge. Only what a deployment explicitly asks for is recorded now;
  identity-bearing fields are still pinned as before.
• max_net_queues was unvalidated. 0 rejected even queues = 1, and
  128 let a request past the node cap only to be rejected against the
  different MAX_NET_QUEUES = 64 bound.

Re-verified on phala-tdx-lab after the fixes:

--net-queues 4 with no --net   → tap,id=net0,ifname=dt…,vhost=on,queues=4
                                 virtio-net-pci,…,mq=on,vectors=10
manifest networks              → {"mode":"bridge",…,"queues":4}   # no "vhost" key
node vhost = false, restart    → runtime-networks {"vhost":false,"queues":4}
fresh VM, vhost=off + 4 queues → tap,…,vhost=off,queues=4 ; guest combined = 4
                                 1 vhost thread (vsock only) ; multi_queue numqueues 4
libvirt mode, empty filter     → cvm.network_filter.filter must not be empty…
max_net_queues = 0             → cvm.max_net_queues must be between 1 and 64

One unrelated observation from that run: these lab VMs exit after a stop/start
cycle, with dstack-prepare.service failing on the second boot. A control VM on
plain user networking — whose netdev string this PR leaves byte-for-byte
unchanged — reproduces it identically, so it is a property of the test app
(key_provider: none), not of this change.

Second review pass

Four more, all fixed:

• netd could not tear down an unfiltered TAP without libvirt. remove_interface
  decided whether to delete an nwfilter binding by checking that /usr/bin/virsh
  exists. On a mode = "none" node that has virsh installed but no reachable
  libvirtd, delete_binding fails on anything but the literal "binding not
  found", and prepare_bridge starts by calling remove_interface — so every
  multiqueue bridge VM would fail to start and leak its TAP. Remove/Check now
  carry whether the interface was created with a binding, defaulting to true on
  the wire so an older VMM's removals still clean up.
• Check reported healthy unfiltered TAPs as broken, since it ran
  nwfilter-binding-dumpxml for every non-macvtap interface. Same flag.
• netd version skew was invisible. netd is a separately deployed root
  service, and queues is #[serde(default)], so an older netd silently built a
  single-queue TAP while the VMM emitted queues=N — the exact IFF_MULTI_QUEUE
  mismatch the adjacent comment warns about, surfacing only as a QEMU failure
  inside the launcher. netd now echoes the queue count it created and the VMM
  refuses to launch on a mismatch.
• Tuning-only requests pinned a backend they were not allowed to choose. The
  first-pass fix let {"queues": 2} inherit the node's backend and skip
  allowed_network_modes, but resolve_requested_networks then pinned that
  mode, parent, and bridge into the manifest — so tuning was strictly more
  powerful than naming the backend, and a later node change no longer reached
  the VM. The rule is now uniform: pin what the caller named, inherit the rest.

Re-verified on phala-tdx-lab with virsh installed and libvirt_uri pointed at
a dead socket, which is the configuration that used to fail:

virsh --connect qemu+unix:///system?socket=/nonexistent  → failed to connect to the hypervisor

deploy --net bridge --net-queues 4
  → tap,id=net0,ifname=dte333c1530de5,…,vhost=on,queues=4 + mq=on,vectors=10
  → tun type tap … multi_queue numqueues 4 persist on user kvin
  → netd: prepared TAP … filter= queues=4
remove
  → netd: removed managed network interface … ; no leftover dt* interfaces

And the filtered path is unchanged, against a real libvirt:

deploy --net bridge (mode = "libvirt")
  → nwfilter-binding-list: dt1f60d8a14a79   clean-traffic
remove
  → binding deleted ; no leftover interfaces

Final regression on phala-tdx-lab

47 assertions against the final binary on a real TDX host (kernel 6.8, QEMU
8.2.2+tdx1.1, guest dstack-0.6.0), on a dedicated bridge and VMM instance —
config validation, RPC policy, generated command lines, host interface state,
guest-visible state, node rollback, and teardown. All pass.

A  config validation      max_net_queues 0 / 65 rejected; node queues 65 rejected;
                          libvirt mode with an empty filter rejected
B  RPC policy             queues 9 > cap rejected; user+multiqueue rejected;
                          queues 8 at the cap accepted -> 8 net vhost threads + vsock
C1 bridge default         tap,br=…,helper=/usr/lib/qemu/qemu-bridge-helper,vhost=on
                          device line unchanged, no mq=on; 1 net vhost thread
C2 --net-no-vhost         bridge,id=net0,br=dstack-perf0 ; vsock vhost thread only
C3 --net-queues 4         tap,ifname=dt…,vhost=on,queues=4 + mq=on,vectors=10
                          netd tap: multi_queue numqueues 4 ; 4 net vhost threads
C4 macvtap --net-queues 2 tap,fds=3:4,vhost=on + mq=on,vectors=6
                          fd 3 and fd 4 both -> /dev/tap1783 ; numtxqueues 2 numrxqueues 2
C5 --net-queues 2 alone   inherits the node backend: tap,ifname=dt…,vhost=on,queues=2
                          manifest: {"mode":"bridge","bridge":"","queues":2}  <- nothing pinned
G  guest ethtool -l       c1=1  c2=1  c3=4  c5=2 channels; all four got DHCP leases
D  node vhost=false       a VM deployed with --net-queues 4 comes back as
                          vhost=off,queues=4 with no net vhost threads
E  libvirt unreachable    unfiltered multiqueue VM deploys, runs and tears down;
                          netd never invokes virsh; no unowned interface left

Throughput on the final binary, 64-byte UDP host→guest via kernel pktgen, same
CVM shape on the same host:

offered	c1: helper + vhost, 1q	c2: legacy netdev, no vhost	b3: netd + vhost, multiqueue
200k	0.16% loss, main 0%	0.19% loss, main 88%	0% loss, main 0%
500k	8.7% loss, main 0%	1.4% loss, main 94%	0% loss, main 0%
1M	—	—	0% loss, main 0%

The main thread going 88–94% → 0% is the wall coming down. The c1 column is the
nuance stated above: with one queue vhost relocates the ceiling into the guest's
receive queue rather than removing it (guest CPU 121% vs 99% in the 500k row),
which is what the multiqueue column fixes.

One observation worth recording: that multiqueue VM had 2 vCPUs, and although
QEMU was given queues=8,vectors=18 the guest reported Combined: 2. The
virtio-net driver uses at most one queue pair per vCPU. Over-provisioning is
therefore inert rather than harmful, and it is not rejected at deployment
because resize can raise the vCPU count later; this is now documented.

Default queue count

Queue pairs default to min(vcpu, 16). Two consequences are worth stating
plainly, because they are behaviour changes on upgrade rather than opt-ins:

• Every VM with two or more vCPUs gets a different virtio-net device. It
  gains mq=on,vectors=2N+2 and N queue pairs where it previously had one. The
  device is not measured — mr_config_id covers the compose hash and instance
  info — so app identity is unaffected, but the guest does see a different NIC.
• Bridge nodes need netd to get it. qemu-bridge-helper returns a single
  descriptor and cannot create a multi_queue TAP. A node that has never
  deployed netd does not fail: bridge NICs fall back to one queue pair with a
  warning. A deployment that asked for a queue count explicitly still fails, so
  the caller learns their request was not met rather than silently getting less.

The measured trade-off, from an 8-vCPU TDX CVM with only the guest's channel
count changed (ethtool -L), is real in both directions:

queue pairs	short-connection throughput
1	22.3k conn/s
2	~20k conn/s
4	15–21k conn/s
8	6.2–7.7k conn/s

The same CVM moved 3.0 Mpps of 64-byte UDP with no loss at 8 queues against
roughly 600k at one. Bandwidth-bound workloads want the default; a VM serving
many short connections should set --net-queues 1. Cross-vCPU wakeups cost an
IPI and a VM exit under TDX, which is why the scaling is capped at 16 rather
than following large vCPU counts.

Verified on phala-tdx-lab, 23 assertions:

[cvm.networking] queues = 4   -> rejected: "has been removed"
 1 vCPU  -> helper netdev, no mq=on                    guest combined = 1
 4 vCPU  -> queues=4,  mq=on,vectors=10, 4+1 threads   guest combined = 4
20 vCPU  -> queues=16, mq=on,vectors=34, 16+1 threads  guest combined = 16
 8 vCPU --net-queues 1 -> helper netdev, no mq=on      guest combined = 1
 8 vCPU --net user     -> user netdev, no mq=on
no netd, default        -> helper netdev, one queue, warning logged
no netd, --net-queues 4 -> deploy fails naming netd

Surfaces and interactions

• vhost off turns the multiqueue default off too. Without vhost the QEMU main
  loop drains every queue on one thread, so extra queues buy little while still
  costing a netd interface, more MSI-X vectors, and a changed guest device.
  Anyone disabling vhost wants the old data plane, so they get the old shape.
  An explicit queue count is still honoured without vhost, since that
  combination is a deliberate request rather than a default.
• Raising the request ceiling does not raise the default. max_net_queues
  bounds what a deployment may ask for; the default's own cap is a fixed 16, so
  a larger VM never silently acquires a worse default. Hard ceiling from any
  source is 64.
• UpdateVm changes both fields, applying from the VM's next boot.
  vmm-cli.py update grew --net, --net-vhost / --net-no-vhost and
  --net-queues to reach it; the update path previously had no networking
  options at all.
• The web UI exposes both per NIC in the deploy and update dialogs, next to
  the mode selector. Leaving a control on its default emits no field, so the
  node keeps owning that value.

Verified on phala-tdx-lab (22 assertions):

8 vCPU --net-no-vhost           -> bridge,id=net0,br=…   no queues=, no mq=on, vsock vhost thread only
8 vCPU --net-no-vhost --net-queues 4 -> vhost=off,queues=4 + mq=on,vectors=10
24 queues, max_net_queues=16    -> rejected "must not exceed 16"
   after raising it to 32       -> accepted; queues=24, vectors=50
24 vCPU default, max=32         -> queues=16   (the default's cap did not move)
UpdateVm --net-queues 2 --net-no-vhost, restart -> vhost=off,queues=2, vectors=6
GetMeta.networking.max_queues   -> present for the UI to bound its input
UI payload {vhost:false,queues:2} -> accepted, applied
UI payload with both left unset   -> vCPU default, and neither field written to the manifest

Third review pass

Eight more, all fixed. Three would have stopped a working node from launching
VMs, which is the failure mode a defaulted-on feature has to be judged by:

• cvm.max_net_queues never bounded the default. Lowering it to 2 still
  handed a 16-vCPU VM sixteen queue pairs, contradicting both the vmm.toml
  comment and the docs. Raising it above 16 still only widens what a caller may
  request; lowering it below 16 now lowers the default too, because a node that
  refuses a request for four should not hand out sixteen by itself.
• One-shot dstack-vmm run broke on every bridge VM. It never applied the
  netd fallback, so with the default queue count a ≥2-vCPU bridge NIC hard-failed
  with "does not manage netd interface lifecycle" on a node that worked before.
• A dead netd made every bridge VM fail to start. Availability was probed
  with socket.exists(), but netd does not unlink its socket on shutdown, so a
  stale file read as "netd is here" and the launch then failed to connect
  instead of falling back. It now connects, the same way netd's own
  bind-time staleness check does.
• Two failure paths leaked a TAP. The new queue-mismatch bail! and the
  pre-existing response.device ? returned without the rollback loop, so a
  version-skewed netd left an interface on the bridge with nothing recorded to
  clean it up. Both now unwind through one shared rollback.
• Legacy VMs could not be stopped. The two "infer missing runtime networks"
  paths still called resolved_networks() rather than the clamped
  runtime_networks(), giving an already-running VM an unclamped queue count
  and failing stop_vm with a netd connect error.
• GetInfo reported a data plane the NIC did not get. A bridge NIC that
  fell back to the non-vhost bridge netdev for want of qemu-bridge-helper
  still reported vhost: true. The QEMU arguments and the reported status now
  read the same effective_vhost.
• The UI sent values it had hidden. Setting a queue count and then switching
  the NIC to user mode still submitted it — the deploy failed with no visible
  control to fix. queues/vhost are now scoped to non-user modes, like
  bridge_name.
• vmm-cli update --net-* replaced the whole NIC list, dropping extra
  interfaces and un-pinning a bridge. It now merges into the VM's existing NIC
  and refuses outright on a multi-NIC VM rather than guessing.

Fixing that last one surfaced a round-trip bug worth calling out on its own:
GetInfo reported parent and macvtap_mode on every interface, including
bridge NICs that had merely inherited them from [cvm.networking] — and the
deployment RPC rejects parent outside macvtap mode. Reported configuration
could be read but not sent back. Both fields are now scoped to macvtap the way
bridge_name is scoped to bridge, with a test that a reported interface
satisfies the RPC's own validation.

Verified on phala-tdx-lab (11 assertions):

16 vCPU, max_net_queues=2      -> queues=2         (the default follows the ceiling down)
stale /run/.../netd.sock       -> deploy succeeds, helper netdev, one queue, warning logged
update --net-queues 2 --net-no-vhost on a bridge VM
                               -> manifest keeps bridge=dstack-perf0, gains vhost:false queues:2
                               -> restart: tap,…,vhost=off,queues=2
GetInfo interfaces             -> {"vhost": false, "queues": 2}   matching the running QEMU
GetInfo configuration          -> {"mode":"bridge","bridge_name":"dstack-perf0","parent":""}
                                  i.e. round-trippable back into UpdateVm

Operator notes

• The account running QEMU must be able to open /dev/vhost-net
  (root:kvm 0660 — add it to kvm). On prod7 this was already true and the
  module autoloaded on first open; no root, no pre-created TAPs, no
  CAP_NET_ADMIN. If it is not, QEMU exits at startup and the VM does not
  boot.
• On host kernels older than 6.4 the vhost worker is a free-standing kernel
  thread whose CPU time escapes the VM's cgroup. Since 6.4 it is a vhost_task
  inside the QEMU thread group — verified on the 6.8 lab host, where the worker
  tids appear under /proc/<qemu>/task/ with the same cgroup — so cpu.max
  and cgroup accounting still attribute it to the CVM.
• Bridge nodes that want more than one queue pair must run netd, including
  when network_filter.mode = "none".
• --net-no-vhost restores the previous -netdev bridge,... command line
  exactly, so the old behaviour is one flag away.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Leechael]

Reviewed against next and read the netd/VMM paths end to end. Design and the on-host verification look right; the points below are the ones that would bite us on rollout.

1. max_net_queues does not bound the default.
Networking::default_queue_pairs() (config.rs:1015) is vcpu.clamp(1, DEFAULT_QUEUE_SCALING_CAP) and never reads cvm.max_net_queues; validate_resolved_network only checks the hard 64. With max_net_queues = 4, --net-queues 5 is rejected while a 16-vCPU VM deployed without the flag gets 16 queue pairs. vmm.toml:57-58 ("default to the VM's vCPU count, capped at this value") and the description say the opposite. Since cvm.networking.queues is rejected at config load, an operator has no node-level way to bound the default short of vhost = false. Suggest vcpu.clamp(1, DEFAULT_QUEUE_SCALING_CAP.min(cfg.max_net_queues)), which keeps the "raising the ceiling does not raise the default" intent.

2. Queue-echo mismatch leaks the prepared interface; upgrade order is undocumented.
In prepare_filtered_networks the rollback loop only runs on the netd::request error arm. The two post-prepare exits — missing macvtap device (app.rs:638) and the queues echo check (app.rs:644) — bail after netd has already created the interface, without Remove and before set_runtime_networks, so stop/remove never see it. Each rejected start leaves one TAP per NIC behind (plus the earlier NICs already in prepared).
The trigger is the skew you describe in the follow-ups: dstack-netd.service is a separate long-running process, so upgrading and restarting dstack-vmm alone leaves the old netd answering. On such a node every bridge (libvirt mode) or macvtap VM with more than one vCPU is refused at start until netd is restarted. Fail-closed is the right call; please route these two exits through the same rollback, and add an upgrade note (restart netd before or together with the VMM) to docs/network-data-plane.md — none of the docs mention it today.

3. Existing nodes change their QEMU command line on upgrade with no config change.
vhost is Option<bool> with unwrap_or(true), so a vmm.toml that predates this PR turns vhost on. On a stock bridge node (network_filter.mode = none, no netd socket, helper at /usr/lib/qemu/qemu-bridge-helper) the netdev goes from bridge,id=…,br=X to tap,id=…,br=X,helper=…,vhost=on; QEMU then exits if the QEMU user cannot open /dev/vhost-net, and the VMM only checks that the node exists. Nodes running netd additionally move every multi-vCPU libvirt-bridge/macvtap VM to multiqueue on restart. I understand this is intentional; the description should state it as an upgrade step (verify /dev/vhost-net access as the QEMU user, or set vhost = false first), and I'd argue the safer default for a release is vhost = false with the on-by-default flip as a separate change.

4. update_networking replaces networks wholesale.
A request that changes mode without queues/vhost clears the previous per-VM tuning; a request with only tuning stores the node's current backend in the manifest. The UI round-trips the stored values so it is unaffected, but an RPC caller has to read-modify-write. Not a defect, but worth one sentence in the proto comments ("unset means cleared, not kept").

Context that may be useful in the docs:

• Runtime vhost_net_start failure does not exit: QEMU logs falling back on userspace virtio and continues on the userspace backend, and vhostforce does not change that. Worth listing next to the vhost thread-count check as something to alert on.
• The guest kernel has no swiotlb= and CONFIG_SWIOTLB_DYNAMIC off, so the bounce pool is a fixed clamp(6% RAM, 64 MB, 1 GB). The default queue count follows vCPUs while the pool follows RAM; a 16 vCPU / 4 GB shape gets 246 MB. Your 8 vCPU / 8 GiB lab shape does not exercise it.

Separately, port_map on a non-user NIC is still accepted and silently dropped (only hostfwd_index consumes it). Not this PR's scope; I'll follow up.