Integrate vendor VFIO vGPUs into the instance lifecycle - #321
Open
yummybomb wants to merge 36 commits into
Open
Conversation
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
July 28, 2026 21:32
199b1d1 to
a93d009
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
4 times, most recently
from
July 29, 2026 15:15
d893fce to
de4742a
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
July 29, 2026 16:09
de4742a to
18e047f
Compare
Contributor
Author
|
Trimmed over-engineering after an architectural review against the parent task (host kernel 6.8 / Ubuntu 24.04 support). Most of the removed weight came from earlier review-round additions rather than the original design: |
yummybomb
marked this pull request as ready for review
August 5, 2026 19:47
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 19:57
7ef1f8d to
bf21162
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 20:24
bf21162 to
83c53ea
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 5, 2026 20:39
83c53ea to
79eef07
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 17:08
ab2b0e8 to
19b9602
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:06
d92201d to
f177335
Compare
yummybomb
changed the base branch from
hypeship/generalize-vgpu-device
to
hypeship/vendor-vfio-backend
August 6, 2026 19:08
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:26
f177335 to
1d875f2
Compare
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 6, 2026 19:40
1d875f2 to
23fb2f5
Compare
…r seam Replace the go:linkname shadow of devices.hostVendorVFIO with createVGPU/destroyVGPU manager fields, and wrap failed creates whose rollback release also failed in VGPUCleanupPendingError so the API can point callers at the retained instance record.
Enable vendor VFIO dispatch in CreateVGPU now that the lifecycle persists assignments durably and guards releases. Protect nil-PID claims in the release guard: the hypervisor PID is only persisted after the claimant boots, so a matching assignment without a PID must be treated as live, matching the startup reconcile protection. Scan raw metadata instead of hydrating instances for the claim check. Hydration derives state through hypervisor queries for every instance on the host, which every vendor VFIO release would pay; the guard only needs the stored assignment, PID, and socket. Unreadable metadata still fails the release closed. Report pending vGPU cleanup even when retaining the rollback record fails: the destroy already failed, so the caller must learn about the outstanding assignment either way.
The vgpu_cleanup_pending response replaced the original create error with cleanup guidance, leaving the cause only in server logs. Prefix the message with the wrapped error so callers see why creation failed as well as how to recover.
ListInstancesForReconcile hydrated every instance (socket stat, UFFD health, /vm.info per instance) before the API served and again on each grace retry, while the protected-set scan only reads stored metadata fields. List raw metadata fail-closed instead, matching the release claim scan, and drop the now-unused loadInstances parameterization. Snapshot restore preserved the source's vGPU assignment path fields but not GPUAssignedAt, so a retained assignment lost its crash-recovery grace timestamp across a restore. Carry the timestamp with the rest of the assignment.
When start's vGPU create fails with a pending device-layer cleanup, the error was returned untyped, so the API mapped it to a generic internal_error. Create already wraps the same condition in VGPUCleanupPendingError and surfaces vgpu_cleanup_pending with retained/unretained guidance. Wrap start's pending-cleanup error the same way — Retained reflects whether the retention record was persisted — and map it in the StartInstance handler ahead of the errors.Is cases so the wrapped cause cannot hide the pending cleanup.
…ures The vgpu_cleanup_pending guidance tells callers to delete the retained instance to retry a failed vGPU release, but no test exercised delete against the minimal GPU-fields-only stub cleanupFailedCreate writes. Add one. Losing the reconcile inventory disables vendor VFIO reconciliation host-wide while releases fail closed on the same inventory, so log it at error level instead of warn. Also document the wholesale-restore assumption in cleanupStartVGPU.
Vendor VFIO vGPUs boot but are non-functional on Cloud Hypervisor (upstream cloud-hypervisor#7572), and the wedged VM then blocks the VF release until startup reconcile. Reject the combination at create and start after the rollback handler is registered, so the rejected device is released through the normal cleanup path. Hypervisor selection otherwise stays caller policy and mdev on Cloud Hypervisor keeps working. Retain identity fields (name, image, hypervisor, data dir) on the failed-create retention record so it lists as a recognizable, deletable instance instead of a nameless phantom; resource claims released by rollback stay dropped. Expose the assigned vGPU device_path in the instance API - on vendor VFIO hosts mdev_uuid is empty and the sysfs path is the identity an operator needs when a release wedges.
When a later start step failed and rollback could not destroy the freshly created vGPU, cleanupStartVGPU retained the assignment on disk but startInstance returned the original failure untyped, so the API reported a generic error instead of vgpu_cleanup_pending with the retained-assignment guidance. Mirror create's named-return wrap: cleanupStartVGPU reports retention state and start wraps the returned error in VGPUCleanupPendingError.
When start rollback fails to destroy a vGPU and the cleanup metadata save also fails, the assignment may still be on disk from the mid-start save. Reporting Retained: false then misdirects callers to wait for startup reconcile when delete or a retried start can already release it. Check whether the surviving record still points at the device, matching create's retention-survives check.
Drop the vendor-VFIO-on-Cloud-Hypervisor rejection from create and start, restoring the phase-0 decision that hypervisor selection is caller policy: production callers pin vGPU instances to QEMU, and the Cloud Hypervisor limitation stays documented in lib/devices/GPU.md.
retainedVGPUFromCreateError built a GPU-fields-only stub, so the retained record from a failed device-layer cleanup listed nameless and, with GPUProfile empty, the API hid its gpu block including device_path. The caller now supplies the identity fields and the stub picks up the pending device's profile.
Startup reconcile protects an assignment whose PID is absent or stale for a bounded grace window, but the release-side claim scan treated a dead PID as unclaimed immediately. Align the two guards: a recent assignment whose recorded hypervisor is not running fails the scan closed so the requester retains and retries, and past the grace window the dead claim no longer blocks the release.
yummybomb
force-pushed
the
hypeship/vendor-vfio-vgpu
branch
from
August 13, 2026 14:19
78d9812 to
3cc1cfd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Top layer of the vendor VFIO vGPU stack (
generalize-vgpu-device#322 ←hypervisor-liveness#363 ←vendor-vfio-backend#364 ← this). The backend itself is in #364; this layer wires it into the instance lifecycle and adds the release guards that make reusable-VF assignments safe:ListInstancesForReconcile, which fails on any unreadable metadata instead of silently skipping it. Recent assignments receive bounded startup protection when their PID is absent or stale; when the inventory is unavailable, vendor VFIO reconciliation is skipped entirely while mdev reconciliation still runs.vgpu_cleanup_pendingwith retained/unretained guidance; the retention record keeps identity fields (name, image, hypervisor) so it lists as a recognizable, deletable instance. The instance API exposesgpu.device_path, the assignment identity on vendor VFIO hosts wheremdev_uuidis empty.Testing
go build ./...,go vet ./...cleango test -racepasses forlib/instancestargeted suites,lib/devices,lib/resources,lib/builds(TestCreateInstanceWithNetworkneeds image pulls + iptables and fails in this environment on the unmodified base as well)Note
High Risk
Changes core GPU assignment, release, and startup reconciliation with fail-closed behavior when metadata is ambiguous—mistakes could release devices in use or leave orphaned VF assignments.
Overview
Wires vendor VFIO vGPUs into create/start/stop/delete and startup reconciliation, replacing the previous hard block on that framework in
CreateVGPU.Lifecycle and safety: Assignments are persisted with
GPUAssignedAt, cleared hypervisor identity on start before re-assigning, and retained in metadata when rollback cannot destroy the device (cleanupFailedCreate, start rollback). Failed create/start surfacesvgpu_cleanup_pending(500) with retained vs reconcile guidance;errors.AsonVGPUCleanupPendingErroris checked before wrapped causes. Vendor VFIO release scans instance metadata for live claimants on the same sysfs path (with startup grace when PID is missing/stale); mdev releases skip that scan. StartupreconcileVGPUsbuilds a protected path set fromListInstancesForReconcile(strict metadata load) and reschedules while grace remains.API: Instance
gpuresponses add optionaldevice_pathalongside profile/mdev UUID.Tests: Broad coverage for retention, claim guards, reconcile protection, and API error mapping; integration vGPU tests no longer skip vendor VFIO.
Reviewed by Cursor Bugbot for commit 3cc1cfd. Bugbot is set up for automated code reviews on this repo. Configure here.