Skip to content

Integrate vendor VFIO vGPUs into the instance lifecycle - #321

Open
yummybomb wants to merge 36 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu
Open

Integrate vendor VFIO vGPUs into the instance lifecycle#321
yummybomb wants to merge 36 commits into
hypeship/vendor-vfio-backendfrom
hypeship/vendor-vfio-vgpu

Conversation

@yummybomb

@yummybomb yummybomb commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Summary

Top layer of the vendor VFIO vGPU stack (generalize-vgpu-device #322hypervisor-liveness #363vendor-vfio-backend #364this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:

  • Live-instance claim guard on release — an assignment goes stale when its release succeeds but the metadata save does not (or start fails between the release and its first save). Since vendor VFIO reuses the same VF path across assignments, honoring a stale claim could clear a VF out from under the live instance that owns it now. Every release consults live instance metadata (liveness verified via Harden hypervisor process liveness checks #363's socket-ownership check) and drops stale metadata without touching the device.
  • Assignment durability — assignments are tagged with the owning instance ID, persisted before booting a started instance without carrying over the previous VMM identity, and retained in metadata when rollback release fails in create/start so later release paths can still find the device.
  • Fail-closed startup reconciliation — the protected set of live VFs is built from ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.
  • Hypervisor scope — hypervisor selection remains a caller policy, preserving the framework layer's existing mdev behavior. The production rollout continues to place vGPU instances on QEMU; the Cloud Hypervisor vendor-VFIO limitation (upstream cloud-hypervisor#7572) is documented in lib/devices/GPU.md.
  • Failure reporting — a failed create or start whose vGPU release also failed surfaces as vgpu_cleanup_pending with retained/unretained guidance; the retention record keeps identity fields (name, image, hypervisor) so it lists as a recognizable, deletable instance. The instance API exposes gpu.device_path, the assignment identity on vendor VFIO hosts where mdev_uuid is empty.

Testing

  • go build ./..., go vet ./... clean
  • go test -race passes for lib/instances targeted suites, lib/devices, lib/resources, lib/builds (TestCreateInstanceWithNetwork needs image pulls + iptables and fails in this environment on the unmodified base as well)

Note

High Risk
Changes core GPU assignment, release, and startup reconciliation with fail-closed behavior when metadata is ambiguous—mistakes could release devices in use or leave orphaned VF assignments.

Overview
Wires vendor VFIO vGPUs into create/start/stop/delete and startup reconciliation, replacing the previous hard block on that framework in CreateVGPU.

Lifecycle and safety: Assignments are persisted with GPUAssignedAt, cleared hypervisor identity on start before re-assigning, and retained in metadata when rollback cannot destroy the device (cleanupFailedCreate, start rollback). Failed create/start surfaces vgpu_cleanup_pending (500) with retained vs reconcile guidance; errors.As on VGPUCleanupPendingError is checked before wrapped causes. Vendor VFIO release scans instance metadata for live claimants on the same sysfs path (with startup grace when PID is missing/stale); mdev releases skip that scan. Startup reconcileVGPUs builds a protected path set from ListInstancesForReconcile (strict metadata load) and reschedules while grace remains.

API: Instance gpu responses add optional device_path alongside profile/mdev UUID.

Tests: Broad coverage for retention, claim guards, reconcile protection, and API error mapping; integration vGPU tests no longer skip vendor VFIO.

Reviewed by Cursor Bugbot for commit 3cc1cfd. Bugbot is set up for automated code reviews on this repo. Configure here.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 199b1d1 to a93d009 Compare July 28, 2026 21:32
@yummybomb
yummybomb changed the base branch from main to hypeship/generalize-vgpu-device July 28, 2026 21:32
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch 4 times, most recently from d893fce to de4742a Compare July 29, 2026 15:15
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from de4742a to 18e047f Compare July 29, 2026 16:09
@yummybomb

yummybomb commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design:

@yummybomb
yummybomb marked this pull request as ready for review August 5, 2026 19:47
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 7ef1f8d to bf21162 Compare August 5, 2026 19:57
Comment thread lib/instances/start.go Outdated
Comment thread integration/vgpu_test.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from bf21162 to 83c53ea Compare August 5, 2026 20:24
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 83c53ea to 79eef07 Compare August 5, 2026 20:39
Comment thread cmd/api/main.go
Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/instances/query.go
Comment thread lib/instances/create.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from ab2b0e8 to 19b9602 Compare August 6, 2026 17:08
Comment thread lib/instances/start.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from d92201d to f177335 Compare August 6, 2026 19:06
@yummybomb
yummybomb changed the base branch from hypeship/generalize-vgpu-device to hypeship/vendor-vfio-backend August 6, 2026 19:08
@yummybomb yummybomb changed the title Support vendor VFIO vGPU devices Integrate vendor VFIO vGPUs into the instance lifecycle Aug 6, 2026
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from f177335 to 1d875f2 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 1d875f2 to 23fb2f5 Compare August 6, 2026 19:40
…r seam

Replace the go:linkname shadow of devices.hostVendorVFIO with createVGPU/destroyVGPU manager fields, and wrap failed creates whose rollback release also failed in VGPUCleanupPendingError so the API can point callers at the retained instance record.
Enable vendor VFIO dispatch in CreateVGPU now that the lifecycle
persists assignments durably and guards releases.

Protect nil-PID claims in the release guard: the hypervisor PID is only
persisted after the claimant boots, so a matching assignment without a
PID must be treated as live, matching the startup reconcile protection.

Scan raw metadata instead of hydrating instances for the claim check.
Hydration derives state through hypervisor queries for every instance
on the host, which every vendor VFIO release would pay; the guard only
needs the stored assignment, PID, and socket. Unreadable metadata still
fails the release closed.

Report pending vGPU cleanup even when retaining the rollback record
fails: the destroy already failed, so the caller must learn about the
outstanding assignment either way.
The vgpu_cleanup_pending response replaced the original create error
with cleanup guidance, leaving the cause only in server logs. Prefix
the message with the wrapped error so callers see why creation failed
as well as how to recover.
ListInstancesForReconcile hydrated every instance (socket stat, UFFD
health, /vm.info per instance) before the API served and again on each
grace retry, while the protected-set scan only reads stored metadata
fields. List raw metadata fail-closed instead, matching the release
claim scan, and drop the now-unused loadInstances parameterization.

Snapshot restore preserved the source's vGPU assignment path fields but
not GPUAssignedAt, so a retained assignment lost its crash-recovery
grace timestamp across a restore. Carry the timestamp with the rest of
the assignment.
When start's vGPU create fails with a pending device-layer cleanup, the
error was returned untyped, so the API mapped it to a generic
internal_error. Create already wraps the same condition in
VGPUCleanupPendingError and surfaces vgpu_cleanup_pending with
retained/unretained guidance.

Wrap start's pending-cleanup error the same way — Retained reflects
whether the retention record was persisted — and map it in the
StartInstance handler ahead of the errors.Is cases so the wrapped cause
cannot hide the pending cleanup.
…ures

The vgpu_cleanup_pending guidance tells callers to delete the retained
instance to retry a failed vGPU release, but no test exercised delete
against the minimal GPU-fields-only stub cleanupFailedCreate writes.
Add one.

Losing the reconcile inventory disables vendor VFIO reconciliation
host-wide while releases fail closed on the same inventory, so log it
at error level instead of warn. Also document the wholesale-restore
assumption in cleanupStartVGPU.
Vendor VFIO vGPUs boot but are non-functional on Cloud Hypervisor
(upstream cloud-hypervisor#7572), and the wedged VM then blocks the VF
release until startup reconcile. Reject the combination at create and
start after the rollback handler is registered, so the rejected device
is released through the normal cleanup path. Hypervisor selection
otherwise stays caller policy and mdev on Cloud Hypervisor keeps
working.

Retain identity fields (name, image, hypervisor, data dir) on the
failed-create retention record so it lists as a recognizable, deletable
instance instead of a nameless phantom; resource claims released by
rollback stay dropped.

Expose the assigned vGPU device_path in the instance API - on vendor
VFIO hosts mdev_uuid is empty and the sysfs path is the identity an
operator needs when a release wedges.
When a later start step failed and rollback could not destroy the
freshly created vGPU, cleanupStartVGPU retained the assignment on disk
but startInstance returned the original failure untyped, so the API
reported a generic error instead of vgpu_cleanup_pending with the
retained-assignment guidance. Mirror create's named-return wrap:
cleanupStartVGPU reports retention state and start wraps the returned
error in VGPUCleanupPendingError.
When start rollback fails to destroy a vGPU and the cleanup metadata
save also fails, the assignment may still be on disk from the mid-start
save. Reporting Retained: false then misdirects callers to wait for
startup reconcile when delete or a retried start can already release
it. Check whether the surviving record still points at the device,
matching create's retention-survives check.
Drop the vendor-VFIO-on-Cloud-Hypervisor rejection from create and
start, restoring the phase-0 decision that hypervisor selection is
caller policy: production callers pin vGPU instances to QEMU, and the
Cloud Hypervisor limitation stays documented in lib/devices/GPU.md.
retainedVGPUFromCreateError built a GPU-fields-only stub, so the
retained record from a failed device-layer cleanup listed nameless and,
with GPUProfile empty, the API hid its gpu block including device_path.
The caller now supplies the identity fields and the stub picks up the
pending device's profile.
Startup reconcile protects an assignment whose PID is absent or stale
for a bounded grace window, but the release-side claim scan treated a
dead PID as unclaimed immediately. Align the two guards: a recent
assignment whose recorded hypervisor is not running fails the scan
closed so the requester retains and retries, and past the grace window
the dead claim no longer blocks the release.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-vgpu branch from 78d9812 to 3cc1cfd Compare August 13, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant