Skip to content

Support live-sharing host directories via virtio-fs in sandbox mode (Cloud Hypervisor parity) #290

Description

@lpcox

Summary

This issue supersedes #216 with comprehensive, end-to-end requirements for live-sharing host directories with NVX microVMs using virtio-fs in sandbox mode, matching the parity and security model that github/gh-aw-firewall (AWF) maintains with its Cloud Hypervisor backend.

Currently, AWF stages the workspace into an EROFS layer and copies changes back after the VM exits (NvxWorkspaceLayer.extractAfterStop). Live-sharing directories via virtio-fs removes the need for copy-back, makes edits visible bidirectionally in real time, and eliminates the risk of host/guest merge conflicts.

Requirements

1. In-Guest Sandbox Mount (from #216)

When virtio-fs is configured via --mount, the guest initialization sequence must mount the live share inside the container rootfs before dropping privileges into the workload:

  • guest/common/nvx-init-agent (or nvx-container-enter) must mount the share at the specified guest target directory (virtfs_dir) using options matching the access mode (ro or rw, with nosuid,nodev).
  • Mount must occur after overlay assembly and survive unshare --mount / chroot.
  • Teardown must unmount cleanly before overlay unmounting.
  • If the mount fails, fail closed (abort with a sandbox error).
  • Validate guest target path (must be absolute, no .., cannot be /, /proc, /sys, /dev, or /.nvx-agent).

2. Multiple Shares with Independent RO/RW Modes

Parity with Cloud Hypervisor mount policies requires supporting at least two concurrent shares:

  1. Workspace (/workspace): Read-Write (rw).
  2. Tool Cache (/opt/hostedtoolcache or $RUNNER_TOOL_CACHE): Read-Only (ro), strictly rejecting write attempts (EROFS/EACCES).
  3. (Future) Temporary diagnostics directories (e.g. /tmp/gh-aw subdirectories).

Each share must have its own mount point, device tag, and access mode enforced on the host/VMM side (guest mount flags are not a security boundary against guest-root compromise).

3. File Ownership & Identity Mapping (--mount-owner caller)

Guest file operations must not run or create files as the VMM process identity:

  • caller identity mode: OpenVMM/HostFs should switch filesystem credentials (setfsuid/setfsgid) per-request to match the guest caller's UID/GID (e.g., workload identity 1001:1001), so created files and modified files on the host are owned by the workspace runner user.
  • Root squash: Operations requested with guest UID 0 / GID 0 must be squashed to the export root's host owner, preventing the creation of host root-owned or setuid-root files on the host filesystem.
  • Fail closed: If the credential switch fails or capabilities (CAP_SETUID/CAP_SETGID) are absent, requests must return EPERM rather than falling back to the VMM identity.

4. Read & Write Exposure Controls (Subdirectory Masking)

Parity with AWF's Cloud Hypervisor staged mount tree and sensitive-path registry (github/gh-aw-firewall#9204, github/gh-aw-firewall#9219):

  • Write narrowing: The ability to restrict write access to declared subtrees within an rw share while keeping the rest of the share read-only.
  • Read masking (hidden subtrees): Specific subdirectories inside an exported share (such as MCP gateway logs /tmp/gh-aw/mcp-logs, firewall logs, or credentials) must be maskable so they are completely unreadable to the guest workload.
    • In Cloud Hypervisor, this is achieved by over-mounting empty read-only tmpfs entries on the staged tree before launching virtiofsd.
    • NVX must support hiding/masking subdirectories on virtio-fs shares (e.g. via --mount-deny on subpaths, ensuring denied paths are absent from the container mount).
    • Explicit exemption: spilled tool payloads (mcp-payloads) must remain readable.

5. Symlink Support in Workspaces

Currently, symlink creation on the microVM profile returns ENOTSUP (lib.rs symlink() when is_microvm()).

  • Building standard projects (e.g. npm ci creating .bin symlinks, Go, Rust, or Python virtualenvs) requires symlink support in the live workspace.
  • Symlinks should be supported where the symlink target is stored verbatim and resolution is handled safely (using strict/O_NOFOLLOW traversal on host path lookups).

Acceptance Criteria

  • Sandbox mode with --mount mounts the share inside the container rootfs, accessible to the non-root workload identity.
  • Multiple shares can be attached with independent ro and rw enforcement.
  • Files created by the workload in an rw share are owned by the workload host UID/GID, not the VMM process.
  • Guest UID 0 is squashed to export root owner.
  • Denied / masked subpaths within a share are completely inaccessible inside the sandbox container.
  • Read-only shares reject writes with EROFS/EACCES.
  • Symlink creation within an rw workspace share succeeds.
  • A new v0.1.0-dev.* release (package and SOURCE-MANIFEST.json) containing these capabilities is published for integration testing in AWF.

Prior Context

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions