Add the vendor VFIO vGPU device backend - #364
Conversation
2bb8e86 to
7fc3b49
Compare
7fc3b49 to
f661e63
Compare
f661e63 to
d1207d0
Compare
d1207d0 to
5b47670
Compare
5b47670 to
ffbf8a0
Compare
ffbf8a0 to
78801e7
Compare
78801e7 to
b4651f3
Compare
b4651f3 to
d2bf3fa
Compare
d2bf3fa to
495ab62
Compare
|
added 7f5233f: report |
7f5233f to
0baacb3
Compare
-->
✱ stlc build✅ go code · compare
✅ typescript code · compare
Diagnostics: 💡 0 new / 5 total note
Build metadata
This comment is auto-generated by stlc and is kept up to date as you push. |
49cb877 to
1438015
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 232d3f9. Configure here.
c29fb6e to
e70b5e0
Compare
Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's nvidia/current_vgpu_type and passed to QEMU as a plain VFIO PCI device. Add a vendor VFIO backend behind the existing framework dispatch: profile discovery from the capacity-dependent creatable catalogs, least-loaded VF placement, create/verify/rollback, and release. Because the same VF path is reused across assignments (unlike mdev UUIDs), release is guarded: an in-process owner map covers the window before QEMU opens the device, and an open-VFIO-handle scan refuses to clear a VF a running VM still holds. Reconciliation clears orphaned assignments on startup, skipping VFs protected by the caller and failing closed when the protected set is unavailable. Branch the vGPU integration test by discovered framework and extend it to cover release on stop and reacquisition on start.
Sort GPUs with unaccountable load last instead of rejecting placement, and stop reporting passthrough capacity when vGPU discovery fails.
…nership through releases
The instance lifecycle already routes create/start/stop/delete through CreateVGPU/DestroyVGPU, so dispatching vendor VFIO creates here would activate the backend before assignment durability and release guards exist. Reject vendor VFIO creates for now; destroy stays wired so existing assignments remain releasable. The integration test skips on vendor VFIO hosts at this layer and no longer asserts the transitional stop-retention behavior.
Counting every free VF advertising a type overreports concurrent capacity: sibling VFs share their parent GPU's framebuffer, so one 48Q assignment revokes the type from every other VF on that GPU. Bound each GPU's contribution by both its free VFs and how many times the profile framebuffer fits into the GPU's remaining framebuffer, using the largest still-creatable profile as a lower bound on what remains.
This reverts commit d6adc4c.
A single unreadable current_vgpu_type failed discoverVFs wholesale, and GetGPUStatus turns a discovery error into a host with no GPU, so one flaky sysfs read blanked out the host's entire GPU capacity for admission and monitoring. Skip unreadable VFs with a warning and keep the readable inventory: a skipped VF is never selected for placement and never reconciled, both safe directions. When no VF is readable, discovery still fails so a wholesale sysfs outage cannot demote a vGPU host to passthrough while assignments exist. Also document that vendor VFIO vGPUs are known broken on Cloud Hypervisor upstream and QEMU is the required hypervisor for GPU instances.
listProfiles failed wholesale when one VF's creatable_vgpu_types read failed, blanking every advertised profile while discoverVFs directly above it already skips unreadable VFs for exactly that reason. Skip and warn instead; underreporting is the safe direction for status and admission. Also document why openVFIOPaths stays strict where mdev's scan is lax (it authorizes clearing a reused VF path), and the 0Q/0B parsing caveat in framebufferFromProfileName.
listProfiles skips an unreadable VF but create still failed placement wholesale when profileMetadata or selectLeastLoadedVF hit the same VF, so /resources could advertise capacity a create then failed to use. Skip the VF in both loops; it simply stops being a placement candidate.
e70b5e0 to
40252f8
Compare

Summary
Layer 2 of the vendor VFIO vGPU stack (
generalize-vgpu-device←hypervisor-liveness← this ←vendor-vfio-vgpu). Self-contained inlib/devices+lib/resources; nothing in the instance lifecycle calls it yet (that's the top layer).Linux 6.8 hosts with NVIDIA R580 drop the mdev interface: vGPUs are assigned by writing a type ID to a VF's
nvidia/current_vgpu_typeand passed to QEMU as a plain VFIO PCI device. This adds that backend behind the framework dispatch introduced in #322:creatable_vgpu_typescatalogs, least-loaded-GPU VF selection, create/verify/rollback. Profile availability counts free VFs currently advertising each type as a best-effort snapshot; creating one assignment may change sibling catalogs.Testing
go build ./...,go vetcleango test -race ./lib/devices/ ./lib/resources/passNote
Medium Risk
Touches sysfs vGPU assignment and VFIO in-use probing on a reused VF path; vendor create remains blocked, but destroy/reconcile and resource reporting changes affect admission and cleanup behavior on GPU hosts.
Overview
Adds NVIDIA vendor VFIO vGPU support alongside mdev: profiles come from
creatable_vgpu_types, assignments are written tocurrent_vgpu_type, with least-loaded-GPU placement, create verify/rollback, and release guards (in-process owner map plus strict open-VFIO-handle scan so a reused VF path is not cleared while a VM still holds it).Discovery and API surface replace
DetectHostGPUModewithDiscoverVGPU(mdev VFs first, then vendor VFIO).ListGPUProfiles,DestroyVGPU, and newReconcileVGPUsdispatch onVGPUFramework. Mdev discovery now ignores VFs without usablemdev_supported_types.CreateVGPUstill rejects vendor VFIO until instance lifecycle integration lands; destroy/reconcile paths are wired for when metadata exists.Resources use
DiscoverVGPUfor/resourcesGPU status and fail closed on discovery errors so a vGPU host is not mis-reported as passthrough. Profile availability is documented as a best-effort VF snapshot.Integration test is framework-aware (device path, sysfs checks), adds stop/release and start/reacquire, and skips on vendor VFIO hosts until create is enabled. Instance cleanup passes
InstanceIDonVGPUAssignmentfor vendor VFIO ownership checks.Reviewed by Cursor Bugbot for commit 40252f8. Bugbot is set up for automated code reviews on this repo. Configure here.