Skip to content

Use kernel paging for Cloud Hypervisor restores - #400

Draft
sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/ch-kernel-paging
Draft

Use kernel paging for Cloud Hypervisor restores#400
sjmiller609 wants to merge 1 commit into
mainfrom
hypeship/ch-kernel-paging

Conversation

@sjmiller609

@sjmiller609 sjmiller609 commented Aug 11, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • boot compatible Cloud Hypervisor v51.1 guests with a private, file-backed memory zone
  • rewrite standby snapshots so restore maps memory-ranges with MAP_PRIVATE and lets the kernel fault pages in instead of eagerly copying all guest RAM
  • hardlink immutable snapshot memory across forks so siblings share one page-cache inode, with the existing reflink/copy fallback and a hypervisor-tagged fallback metric
  • allow concurrent Cloud Hypervisor fork preparation
  • leave snapshots on the eager restore path when they use ballooning, memory hotplug, hugepages, shared memory, PCI/VFIO, vhost-user, or unsupported memory layouts

No UFFD path is involved.

Benchmark

8 GiB / 8 vCPU Chromium guest; source was CDP-ready before standby. Times are request-to-CDP-ready.

backend single fork 25-way fanout 25-way p95
Cloud Hypervisor baseline 55.63 s 0/25 succeeded
Cloud Hypervisor kernel paging 317 ms 25/25 succeeded 405 ms
Firecracker file-backed restore 478 ms 25/25 succeeded 842 ms

During the 25-way runs, baseline Cloud Hypervisor read 67.25 GiB from the host block device and lost 63.5 GiB of available host memory before all requests failed. The kernel-paged run read 8.4 MiB and its maximum available-memory drop was 5.24 GiB. These I/O windows include the stress tool's hold and cleanup period, so they are comparative rather than restore-only attribution.

Known limitation: later standby I/O

This removes Cloud Hypervisor's eager restore, but Cloud Hypervisor still writes a full memory image on every snapshot. After a fork ran idle for 10 seconds, a later standby wrote 8,193 MiB according to the service cgroup's io.stat; Firecracker's dirty-page snapshot wrote 221 MiB. The Cloud Hypervisor request returned in 8.52 s but needed another 19.87 s to flush, versus 3.21 s + 96 ms for Firecracker.

With 25 simultaneous explicit standby requests, both backends hit the 60-second API deadline. Cloud Hypervisor had written 19.0 GiB by then; Firecracker had written 6.4 GiB. Matching Firecracker here requires differential snapshot support in Cloud Hypervisor; Hypeman cannot recover the missing dirty-page information after Cloud Hypervisor has already emitted the full snapshot.

Validation

  • CI: Linux tests, Darwin tests, and e2e-install
  • make build
  • go test ./lib/hypervisor/cloudhypervisor
  • go test ./lib/instances -run 'TestCloneCloudHypervisorSnapshotSharesMemoryInode|TestEnsureExclusiveSnapshotMemoryOwnership' -count=1
  • 512 MiB nginx standby → kernel-paged restore → HTTP-ready fork smoke test
  • 8 GiB Chromium single-fork, 25-way fanout, second-generation standby/restore, and cgroup/block-I/O benchmarks

@sjmiller609
sjmiller609 force-pushed the hypeship/ch-kernel-paging branch from be70d28 to 55956c4 Compare August 11, 2026 20:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant