refactor(driver-vm): replace OCI registry client with containerd shim

[ericcurtin]

Summary

The VM driver's registry pull and layer-unpack logic was a hand-rolled OCI
client (auth, manifest parsing, per-layer download, digest verification)
plus a from-scratch tar-layer merge that only understood plain files,
dirs, symlinks, and OCI whiteouts.

Replace it with containerd's own Go client libraries (core/remotes/docker,
core/content/local, pkg/archive) for OCI-correct registry resolve/fetch/auth
and layer application, including opaque dirs, xattrs, and device/fifo entries.
No containerd daemon is involved: the new goshim/ Go module links those
packages directly and builds into a small cgo shared library
(libopenshell_containerd_shim), loaded at runtime via libloading the same
way this crate already loads libkrun.

Changes

• goshim/: exports ContainerdResolveDigest, ContainerdPullImage, and
  ContainerdUnpackLayout.
• src/containerd_shim.rs: dynamic loader and safe Rust wrappers, mirroring
  ffi.rs's LibKrun pattern.
• src/driver.rs: removed the manual registry client, layer-merge, and OCI
  layout writer (~1000 net lines); both pull paths now call the shim.
• build.rs / embedded_runtime.rs: embed the shim like
  libkrun/libkrunfw/gvproxy.
• tasks/scripts/vm/build-containerd-shim.sh: builds the shim natively or
  cross-compiled, wired into vm:setup and CI.
• Dropped oci-client, flate2, sha2; added go to mise.toml/mise.lock.

Registry auth env vars are unchanged. Per-layer pull progress is now a single
event instead of one per layer, since the whole pull happens in one blocking
call into the shim.

Testing

• cargo fmt --all -- --check / cargo clippy --workspace --all-targets -- -D warnings pass
• cargo test -p openshell-driver-vm passes
• Clean release build of openshell-driver-vm with real runtime artifacts embedded
• License header check passes; markdownlint-cli2 passes on the changed README
• Verified against a live registry (docker.io/library/busybox) through the built
  shared library via libloading: resolve digest, pull into an OCI layout, unpack
  a layer, and error propagation for a nonexistent image. This caught and fixed a
  real bug: platforms.Only doesn't fail closed for cross-arch requests the way
  platforms.OnlyStrict does.
• macOS cross-build verified end-to-end with docker buildx/osxcross, producing a
  real Mach-O arm64 binary with the shim cross-compiled and embedded
• E2E tests not run here (no containerd/GPU/multi-arch runners available); the
  driver's own image-pull path is exercised by the verification above instead

Design notes (from review)

• The bootstrap image is unpacked directly on the host via the shim because that
  rootfs boots the guest, so no VM exists yet to unpack it inside of. Every other
  (less trusted, user-requested) image still goes through the existing guest-side
  umoci raw unpack path, so archive extraction of untrusted registry content never
  runs against host-owned storage. Documented in the README and in
  pull_and_unpack_registry_image's doc comment.
• The broader ask to extract ImageSource/ImageUnpacker/VmRootfsMaterializer
  interfaces before landing this, and to stage this behind docs(vm): clarify MicroVM driver use cases and limitations #2332, is a real
  architectural question I want to answer in review rather than pre-empt here.

Known follow-ups

• Go module license/SBOM scanning isn't wired into cargo-about/syft yet; this is
  the first Go code in the repo.
• Per-layer pull progress granularity could be restored via a C progress callback
  from the shim if finer-grained CLI watch detail is needed later.

Checklist

• Follows Conventional Commits
• Commit is signed off (DCO)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[ericcurtin]

There are only two implementations of the OCI protocol that have been widely battle-tested across the bazillion OCI registries in the world:

https://github.com/containerd/containerd
https://github.com/containers/image

and everything that wraps these like docker, podman, skopeo, k8s, etc.

The protocol is highly ossified, hard to test many of these incantations as they are in private data-centre's we cannot access. Using oci-client crate is a mistake. I propose a little bit of golang just for this transport component.

containerd as a library gets the nod in this PR because it's a CNCF graduated project, it's the default runtime under most Kubernetes distributions (GKE, EKS, AKS), it sits inside Docker Engine, and its image-handling code paths (the remotes resolver and the newer transfer service) are exercised billions of times a day.

containerd also supports Windows, Linux, macOS. Last I tried "containers/image" it isn't buildable or functioning on Windows.

[elezar]

Thanks for expanding on the motivation here. I agree that registry compatibility
is difficult to reproduce comprehensively and that using a widely exercised
implementation such as containerd's registry and archive libraries may be more
robust than maintaining registry resolution and layer semantics ourselves. I
also understand that this proposal links those libraries through a Go
C-shared-library shim; it does not introduce a running containerd daemon.

At the same time, I do not see a concrete registry failure or required use case
in the PR that justifies replacing the current implementation. Before taking on
the additional Go/cgo, unsafe FFI, dynamic-loading, cross-compilation, and
packaging complexity, I think we should identify the incompatibility we are
solving, the registries or OCI behaviours we need to support, and the level of
compatibility expected from the VM driver. If the VM driver's intended scope is
primarily local and experimental, its requirements may differ from those of the
Docker, Podman, or Kubernetes drivers. Issue #2332
now tracks documenting those use cases and limitations, and that positioning
should inform this trade-off.

Before replacing the current implementation, however, I would prefer that we
first reorganise the VM driver's rootfs handling around explicit interfaces.
This reorganisation is also motivated by PR #2312,
which needs the same image-acquisition capabilities but produces an OCI rootfs
artifact rather than a VM disk. Establishing the boundary first would let both
proposals build on the same model instead of introducing separate image
pipelines.

The current code combines several distinct responsibilities inside the VM
driver:

• Image reference resolution and authentication
• Manifest and platform selection
• Blob acquisition and verification
• OCI layout construction
• Layer unpacking and application
• Cache identity and progress reporting
• VM-specific rootfs preparation and ext4 materialisation

PR #2313 changes several of these responsibilities at once while they remain
embedded in the VM driver. It also adds a Go toolchain, cgo shared library,
runtime dynamic loading, embedded artifacts, and a separate dependency and SBOM
surface. Those choices may be justified, but they would be easier to evaluate,
test, replace, and potentially reuse if the architectural boundary existed
first.

Conceptually, I would like the VM rootfs pipeline to look more like:

VmRootfsProvider
├── ImageSource
│   ├── OciClientImageSource          (current)
│   └── ContainerdShimImageSource    (proposed)
├── ImageUnpacker
│   ├── RustLayerUnpacker           (current host path)
│   ├── UmociGuestUnpacker          (current guest path)
│   └── ContainerdArchiveUnpacker   (proposed host path)
└── VmRootfsMaterializer
    └── prepared ext4 image / image-preparation VM

The exact interface names are less important than keeping image acquisition and
OCI-correct unpacking separate from the VM-specific output format. For example,
an ImageSource could produce a verified OCI image layout, an ImageUnpacker
could produce an unpacked rootfs directory, and VmRootfsMaterializer could
turn that directory into the ext4 artifact consumed by libkrun.

The existing umoci dependency is relevant here as well. PR #2313 does not
replace it: the VM image-preparation guest continues to run umoci raw unpack
for OCI-layout payloads, while the new shim adds containerd's archive
implementation for host-side unpacking. OpenShell would therefore manage two
OCI unpack implementations, in addition to the registry acquisition path. That
is not necessarily wrong—the bootstrap rootfs and the need to preserve Linux
filesystem semantics may require different host and guest paths—but the reason
for each dependency and the boundary between them should be explicit. We should
decide whether the shim is intended only for acquisition, whether unpacking can
be consolidated in the preparation guest, or whether separate host and guest
unpackers are an intentional supported design.

We should also evaluate whether more of the acquisition and unpacking path can
run inside the Linux image-preparation guest. That could keep Linux-specific OCI
tooling out of the host process and avoid a host-side cgo/FFI boundary, while
still allowing the guest to use a mature implementation. It may have trade-offs
around networking, credentials, caching, and bootstrap size, but I would like to
see it compared explicitly with the proposed host-loaded shared library.

If we confirm compatibility gaps in the current Rust implementation and decide
that a Go shim is not an approach we want to maintain, another option is to
contribute the missing behaviour and regression tests to the relevant upstream
Rust project. That would preserve a native Rust integration, address the
specific compatibility problems we can demonstrate, and benefit other
Rust-based projects that need the same OCI functionality. The feasibility of
that path depends on the nature of the gaps and the upstream project's scope and
maintainer engagement, but it should be evaluated before introducing a
permanent FFI boundary.

This separation is also relevant to PR #2312.
The proposed OCI provisioner needs image acquisition and rootfs construction,
but its final artifact is an OCI mount or directory tree rather than a VM disk.
If we establish the source and unpacking contracts first, the containerd library
shim proposed here could potentially be reused by both provisioners without
making either one depend on the other's final rootfs representation. It would
also make the daemon-backed provider in #2312 clearly replaceable rather than
allowing containerd daemon concepts to become part of the compute-driver
contract.

I therefore suggest staging this work as follows:

1. Define the VM driver's intended image-compatibility requirements and document
   the concrete failures or unsupported workflows motivating this replacement.
2. Compare the host-loaded Go shim with alternatives, including retaining the
   current implementation, contributing confirmed compatibility fixes and tests
   to an upstream Rust implementation, and moving mature image tooling into the
   preparation guest.
3. Extract the existing VM image source, unpacking, caching, progress, and ext4
   materialisation boundaries without changing behaviour.
4. Keep the current oci-client and Rust unpacker as the initial implementations
   and preserve the existing tests around the new contracts.
5. Introduce the containerd Go shim as alternative ImageSource and
   ImageUnpacker implementations.
6. Compare behaviour, packaging, cross-platform support, security, build
   complexity, overlap with the existing umoci path, and SBOM/licensing
   implications before removing the existing implementation.
7. Decide whether the selected source and unpacker should be shared with the OCI
   provisioner from feat(driver-oci): add OCI compute driver #2312.

My request is not to retain the existing OCI implementation indefinitely. It is
to establish the provider and materialisation boundaries before replacing it,
so this PR becomes a focused implementation change rather than simultaneously
changing the architecture, build system, dependency model, and registry
behaviour. I would prefer that reorganisation to land first, followed by a
smaller version of this PR built against those interfaces.

[ericcurtin]

Agreed the boundary should exist eventually. I'll open a follow-up issue to extract ImageSource/ImageUnpacker once #2312 lands, so the interface is designed against two real implementations rather than speculatively. In the meantime, I've documented why umoci and the containerd archive unpacker coexist (host vs. guest trust boundary) so that's no longer an open question in this PR.

[elezar]

I think identifying these seams should be done before we land #2312. Working against an abstraction will allow us to review the compute contributions of #2312 more cleanly -- even if they are currently using containerd (or some other mechanism) for realising the rootfs. (I did create #2388 as a POC / demonstrator to start discussion. It's still a draft, but we can dig into it further if we think it's valuable).

With regards to this PR. Although the approach is interesting, I have concerns with introducing a GoLang-based library into our build and release process at this stage. Although this misgivings ot the oci-client-based approach is noted, it would be better to make this change with measurable improvements in mind. As I have called out in #2313 (comment), a better long-term approach may be to IMPROVE oci-client (or some other upstream crate) so as to provide better support for ALL rust applications that need to interact with registries.

[ericcurtin]

I think identifying these seams should be done before we land #2312. Working against an abstraction will allow us to review the compute contributions of #2312 more cleanly -- even if they are currently using containerd (or some other mechanism) for realising the rootfs. (I did create #2388 as a POC / demonstrator to start discussion. It's still a draft, but we can dig into it further if we think it's valuable).

With regards to this PR. Although the approach is interesting, I have concerns with introducing a GoLang-based library into our build and release process at this stage. Although this misgivings ot the oci-client-based approach is noted, it would be better to make this change with measurable improvements in mind. As I have called out in #2313 (comment), a better long-term approach may be to IMPROVE oci-client (or some other upstream crate) so as to provide better support for ALL rust applications that need to interact with registries.

FWIW, I ❤️ Rust. If the whole OCI ecosystem was being written in 2026, it probably would probably be written in Rust. But OCI and golang are heavily intertwined, it's a multi-year effort to reimplement all of it in Rust. We can take subsets and re-implement those in Rust bit by bit like in the case of #2312 especially in the performance critical cases, it's just something to be mindful of. I would say pushing/pulling images isn't performance critical because it's network bound, but at the same time happy to drop this if needs be.