Skip to content

Add the vendor VFIO vGPU device backend - #364

Open
yummybomb wants to merge 19 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend
Open

Add the vendor VFIO vGPU device backend#364
yummybomb wants to merge 19 commits into
hypeship/hypervisor-livenessfrom
hypeship/vendor-vfio-backend

Conversation

@yummybomb

@yummybomb yummybomb commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Layer 2 of the vendor VFIO vGPU stack (generalize-vgpu-devicehypervisor-livenessthisvendor-vfio-vgpu). Self-contained in lib/devices + lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).

Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:

  • Discovery & placement — profile discovery from the capacity-dependent creatable_vgpu_types catalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.
  • Release guards — unlike mdev (fresh UUID per assignment), vendor VFIO reuses the same VF path across assignments, so a stale release could clear a later owner's vGPU. Release is guarded by an in-process owner map (covers the window before QEMU opens the device) and an open-VFIO-handle scan (refuses to clear a VF a running VM holds).
  • Reconciliation — clears orphaned assignments on startup, skips VFs in the caller-supplied protected set, and fails closed (skips vendor VFIO entirely) when that set is unavailable, while mdev reconciliation still runs.
  • Per-VF degradation — one unreadable VF is skipped with a warning instead of failing discovery or profile listing wholesale, so a flaky sysfs read cannot blank the host's advertised GPU capacity. Only when no VF is readable does discovery fail, so a wholesale outage cannot demote a vGPU host to passthrough while assignments exist. The open-handle probe stays strict on purpose: it authorizes clearing a reused VF path, so an incomplete scan fails the release rather than risk a false "not in use".
  • Integration test — branched by discovered framework, extended to cover release on stop and reacquisition on start.

Testing

  • go build ./..., go vet clean
  • go test -race ./lib/devices/ ./lib/resources/ pass

Note

Medium Risk
Touches sysfs vGPU assignment and VFIO in-use probing on a reused VF path; vendor create remains blocked, but destroy/reconcile and resource reporting changes affect admission and cleanup behavior on GPU hosts.

Overview
Adds NVIDIA vendor VFIO vGPU support alongside mdev: profiles come from creatable_vgpu_types, assignments are written to current_vgpu_type, with least-loaded-GPU placement, create verify/rollback, and release guards (in-process owner map plus strict open-VFIO-handle scan so a reused VF path is not cleared while a VM still holds it).

Discovery and API surface replace DetectHostGPUMode with DiscoverVGPU (mdev VFs first, then vendor VFIO). ListGPUProfiles, DestroyVGPU, and new ReconcileVGPUs dispatch on VGPUFramework. Mdev discovery now ignores VFs without usable mdev_supported_types. CreateVGPU still rejects vendor VFIO until instance lifecycle integration lands; destroy/reconcile paths are wired for when metadata exists.

Resources use DiscoverVGPU for /resources GPU status and fail closed on discovery errors so a vGPU host is not mis-reported as passthrough. Profile availability is documented as a best-effort VF snapshot.

Integration test is framework-aware (device path, sysfs checks), adds stop/release and start/reacquire, and skips on vendor VFIO hosts until create is enabled. Instance cleanup passes InstanceID on VGPUAssignment for vendor VFIO ownership checks.

Reviewed by Cursor Bugbot for commit 40252f8. Bugbot is set up for automated code reviews on this repo. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go Outdated
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 2bb8e86 to 7fc3b49 Compare August 6, 2026 19:26
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7fc3b49 to f661e63 Compare August 6, 2026 19:40
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from f661e63 to d1207d0 Compare August 7, 2026 14:02
Comment thread lib/resources/gpu.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from d1207d0 to 5b47670 Compare August 7, 2026 15:04
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 5b47670 to ffbf8a0 Compare August 7, 2026 20:52
Comment thread lib/devices/vendor_vfio_linux.go
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from ffbf8a0 to 78801e7 Compare August 8, 2026 01:05
Comment thread integration/vgpu_test.go Outdated
@yummybomb

Copy link
Copy Markdown
Contributor Author

added 7f5233f: report GPUProfile.Available as the count of free VFs advertising the profile type (creatable-instance units), matching mdev's summed available_instances, the OpenAPI description, and the integration test's decrement assertion. note the gpuProfileSlots metric steps up on vendor VFIO hosts (per-GPU → per-free-VF units). go test ./lib/devices ./lib/resources green; the hardware integration test was not run locally.

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from 7f5233f to 0baacb3 Compare August 10, 2026 07:26
@github-actions

github-actions Bot commented Aug 10, 2026

Copy link
Copy Markdown
-->

✱ stlc build

go code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

114 files generated at 32bcf05 (pushed)

go get github.com/kernel/hypeman-go-staging@32bcf05dec93a1dbdf6bbdcffb9fbaa25a3054a9
typescript code · compare

Your SDK build was successful.

generate ✅bootstrap ✅format ✅

136 files generated at 0358fb8 (pushed)

Diagnostics: 💡 0 new / 5 total note
LevelCodeMessageTargets
Build metadata
Buildbd_767nlNGQ-tame-glint
Timestamp2026-08-13T14:21:59.563Z
stlc8413509
Spec hashd57cf10cee81
Config hashee536b54c599

This comment is auto-generated by stlc and is kept up to date as you push.
If you push new commits, re-run this workflow to update this comment.
Last updated: 2026-08-13 14:22:19 UTC

@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch 2 times, most recently from 49cb877 to 1438015 Compare August 10, 2026 19:53
@yummybomb yummybomb closed this Aug 10, 2026
@yummybomb yummybomb reopened this Aug 10, 2026
Comment thread lib/devices/vendor_vfio_linux.go

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 232d3f9. Configure here.

Comment thread lib/devices/vendor_vfio_linux.go
Comment thread lib/devices/mdev_linux.go
@yummybomb
yummybomb requested a review from sjmiller609 August 11, 2026 21:35
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from c29fb6e to e70b5e0 Compare August 12, 2026 15:20
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are
assigned by writing a type ID to a VF's nvidia/current_vgpu_type and
passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend
behind the existing framework dispatch: profile discovery from the
capacity-dependent creatable catalogs, least-loaded VF placement,
create/verify/rollback, and release.

Because the same VF path is reused across assignments (unlike mdev
UUIDs), release is guarded: an in-process owner map covers the window
before QEMU opens the device, and an open-VFIO-handle scan refuses to
clear a VF a running VM still holds. Reconciliation clears orphaned
assignments on startup, skipping VFs protected by the caller and
failing closed when the protected set is unavailable.

Branch the vGPU integration test by discovered framework and extend it
to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
The instance lifecycle already routes create/start/stop/delete through
CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would
activate the backend before assignment durability and release guards
exist. Reject vendor VFIO creates for now; destroy stays wired so
existing assignments remain releasable. The integration test skips on
vendor VFIO hosts at this layer and no longer asserts the transitional
stop-retention behavior.
Counting every free VF advertising a type overreports concurrent
capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q
assignment revokes the type from every other VF on that GPU. Bound each
GPU's contribution by both its free VFs and how many times the profile
framebuffer fits into the GPU's remaining framebuffer, using the largest
still-creatable profile as a lower bound on what remains.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and
GetGPUStatus turns a discovery error into a host with no GPU, so one
flaky sysfs read blanked out the host's entire GPU capacity for
admission and monitoring.

Skip unreadable VFs with a warning and keep the readable inventory: a
skipped VF is never selected for placement and never reconciled, both
safe directions. When no VF is readable, discovery still fails so a
wholesale sysfs outage cannot demote a vGPU host to passthrough while
assignments exist.

Also document that vendor VFIO vGPUs are known broken on Cloud
Hypervisor upstream and QEMU is the required hypervisor for GPU
instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read
failed, blanking every advertised profile while discoverVFs directly
above it already skips unreadable VFs for exactly that reason. Skip and
warn instead; underreporting is the safe direction for status and
admission.

Also document why openVFIOPaths stays strict where mdev's scan is lax
(it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat
in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement
wholesale when profileMetadata or selectLeastLoadedVF hit the same VF,
so /resources could advertise capacity a create then failed to use.
Skip the VF in both loops; it simply stops being a placement candidate.
@yummybomb
yummybomb force-pushed the hypeship/vendor-vfio-backend branch from e70b5e0 to 40252f8 Compare August 13, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant