Skip to content

fix(mxc): keep host supervisor proxy CA files in sandbox state - #5059

Merged
shailendra-nv merged 3 commits into
NVIDIA:windowsfrom
jquesnelle:fix/5058-mxc-per-sandbox-tls-dir/jquesnelle
Oct 31, 2026
Merged

shailendra-nv merged 3 commits into
NVIDIA:windowsfrom
jquesnelle:fix/5058-mxc-per-sandbox-tls-dir/jquesnelle

Conversation

@jquesnelle

Copy link
Copy Markdown

Summary

The MXC driver starts each sandbox's host supervisor without OPENSHELL_PROXY_TLS_DIR, so every supervisor writes its proxy CA certificate and the workload's CA bundle to /etc/openshell-tls, which Windows resolves to \etc\openshell-tls on the gateway's drive. Sandboxes started together overwrite each other's files, and the files outlive the sandbox. The Linux drivers don't hit this: Docker, Podman and Kubernetes run the supervisor inside each sandbox's container or pod with OPENSHELL_PROXY_TLS_DIR=/run/openshell/proxy-tls, and the VM driver sets it to proxy-tls in the sandbox's state directory. This PR puts them in proxy-tls\ in the sandbox's host state directory, as the VM driver does, and makes the supervisor's TLS file writes work for paths longer than MAX_PATH, which that directory can exceed.

Related Issue

Closes #5058

Changes

  • openshell-driver-mxc: the host supervisor command sets OPENSHELL_PROXY_TLS_DIR=<state_dir>\<sandbox-id>\<generation>\proxy-tls. delete_sandbox already removes that directory. spawn_supervisor now calls a new supervisor_command, which builds the command, so a unit test can read its environment. The driver README says where the files are.
  • The host supervisor inherits the gateway's environment. Until now an OPENSHELL_PROXY_TLS_DIR set for the gateway reached it and moved the files (still one directory for every sandbox); the driver now always sets the variable, so that value no longer applies.
  • openshell-supervisor-network: on Windows, write_tls_output resolves the output directory with canonicalize (the \\?\ form) before creating and installing the file. If canonicalize fails, as it can on volumes that the Mount Manager does not track, the path is used as given, as before. NamedTempFile::persist passes the temporary file's path to SetFileAttributesW and both paths to MoveFileExW unchanged, and those calls fail with os error 3 beyond MAX_PATH. The new directory adds 84 characters to state_dir (\<sandbox-id>\<generation>\proxy-tls), so the CA file path is state_dir plus 101 characters. Other platforms are unchanged.
  • Files that earlier builds wrote to \etc\openshell-tls are not moved or removed.

Testing

Host: Windows 11 26340.9767 x64, MXC 1.0.1 wxc-exec (--probe: base-container, needsDaclAugmentation: false), Rust 1.95.0, VS 2022 17.14, cargo-nextest 0.9.143. The Windows commands are the ones windows:lint:x64 and windows:test:x64 run (tasks/scripts/windows-msvc.ps1), with its environment.

  • Checks appropriate to the affected code and behavior pass
    • windows:lint:x64 (workspace clippy with the Windows excludes and -D warnings, e2e/rust clippy, examples/governance-interceptor check): pass.
    • windows:test:x64 (cargo nextest run --profile ci --workspace <Windows excludes> --target x86_64-pc-windows-msvc --features openshell-server/test-support --features openshell-server/prebuilt-z3,openshell-prover/prebuilt-z3): 6,633 passed, 41 skipped. The first of two runs had one failure in a crate this PR does not touch (openshell-mxc-boundary socket_owner::tests::resolves_child_that_owns_ipv4_connection: "No Windows process owns proxy connection", bug(mxc): proxy refuses some connections from an allowed program with "No Windows process owns proxy connection" #5056); it passed alone 5 of 5 and in the second run.
    • cargo fmt --all -- --check, git diff --check: pass.
    • Linux (WSL2 Ubuntu, Rust 1.95.0): cargo clippy --locked -p openshell-supervisor -p openshell-driver-mxc --all-targets -- -D warnings and cargo nextest run --locked --profile ci -p openshell-supervisor (169 passed) pass on the driver change. The openshell-supervisor-network change and its tests are Windows-only.
  • Unit tests added/updated
    • host_supervisor_proxy_tls_dir_is_in_sandbox_host_state (openshell-driver-mxc): the host supervisor command sets OPENSHELL_PROXY_TLS_DIR to proxy-tls in the sandbox's host state directory. On windows (5d1bfbf92) it fails with left: None. It also fails if the driver points the variable at the boundary state directory or at a directory shared by the gateway's sandboxes.
    • write_ca_files_accepts_paths_longer_than_max_path (openshell-supervisor-network, Windows): writes both files into a directory whose CA file path is longer than 260 UTF-16 code units. On windows it fails with atomically install TLS output ...: The system cannot find the path specified. (os error 3).
    • extended_length_path_keeps_the_path_when_canonicalize_fails (openshell-supervisor-network, Windows): an existing directory resolves to the \\?\ form; for a directory that does not exist, canonicalize fails and the path comes back unchanged. It fails if the error is not handled.
  • E2E tests added/updated (if applicable): none added. run-mxc-e2e.ps1 -Scenario https-ca (local copy whose Test-PortListening connects to the port, because GetActiveTcpListeners() does not list the gateway's listener on this host), with %TEMP% deep enough to put the CA file path at 304 characters (the harness keeps the driver's state under %TEMP%): windows passes 2 of 2. The driver change alone fails 2 of 2 with HTTPS certificate absent (gateway log: Failed to write CA files ... (os error 3)) and passes with a short %TEMP%. With both commits it passes 2 of 2 at the same depth, and 1 of 1 with the CA file path at about 419 characters. These runs were on a host with LongPathsEnabled=1. On a host with the default 0, creating a sandbox may fail earlier once state_dir is about 186 characters or more, because create_restricted_state_dir passes the raw host state directory path to icacls.exe, which handles long paths only when that setting is on. That predates this PR and was not measured.

Live before and after with the script in #5058: one gateway (mTLS, throwaway certificates) with an explicit state_dir, 8 ProcessContainer sandboxes created at the same time; each workload saves the CA it was given and makes one HTTPS request to example.com through the proxy, curl trusting only the injected bundle.

windows 5d1bfbf92 this branch
CA files C:\etc\openshell-tls\openshell-ca.pem, ca-bundle.pem, rewritten by every run <state_dir>\<sandbox-id>\<generation>\proxy-tls\, one per sandbox
8 sandboxes at once: HTTPS 200 6 of 8 (the script); earlier, without TLS on the gateway: 3, 4 and 6 of 8 (3 runs) 8 of 8 (the script); earlier: 8, 8 and 8 of 8 (3 runs)
failures the script: 2 workloads had another sandbox's CA (curl 60). Earlier runs: 4 more had another sandbox's CA; 7 supervisors logged Failed to write CA files, TLS termination disabled and their workloads got no CA none
distinct CAs among 8 workloads 6; earlier 3, 4, 6 8 each run, each matching its own proxy-tls file
after sandbox delete files stay in C:\etc\openshell-tls no CA files left under state_dir

The overwrites depend on timing: another run of this script on windows had 8 of 8, while all eight supervisors still wrote their files to C:\etc\openshell-tls and none to sandbox state. The earlier runs used the same workload steps against a gateway without TLS. One sandbox at a time passed on both in the earlier runs (3 of 3 on windows).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable): the MXC driver README.

…ndows

write_tls_output installs each file with NamedTempFile::persist, which
passes the temporary file's path to SetFileAttributesW and both paths to
MoveFileExW unchanged. On Windows those calls fail with
ERROR_PATH_NOT_FOUND for a path longer than MAX_PATH without the \\?\
prefix, and the supervisor then disables TLS termination. A TLS directory
inside a deep state directory, such as an MXC sandbox's host state
directory, can exceed that length.

On Windows, resolve the output directory with canonicalize, which returns
the \\?\ form, before creating and installing the file. If canonicalize
fails, as it can on volumes that the Mount Manager does not track, keep
the path as given.

Signed-off-by: Jeffrey Quesnelle <emozilla@nousresearch.com>
The MXC driver starts the host supervisor without a proxy TLS directory,
so the supervisor uses its default, /etc/openshell-tls, which Windows
resolves to \etc\openshell-tls on the gateway's current drive. Every MXC
sandbox on the host writes its proxy CA certificate and trust bundle to
the same two files there and reads them back when it starts its
workload, and the files stay after the sandbox is deleted.

Set OPENSHELL_PROXY_TLS_DIR for the host supervisor to proxy-tls\ in the
sandbox's host state directory, as the VM driver does with its state
directory. delete_sandbox already removes that directory.

Signed-off-by: Jeffrey Quesnelle <emozilla@nousresearch.com>
@jquesnelle

Copy link
Copy Markdown
Author

cc @shailendra-nv

@copy-pr-bot

copy-pr-bot Bot commented Oct 30, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@shailendra-nv shailendra-nv added the test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors label Oct 30, 2026 — with ChatGPT Codex Connector

Copy link
Copy Markdown
Collaborator

/ok to test d2efbe9

Merge the current windows branch so CI includes the existing server test repairs and MXC fixes.

Signed-off-by: Shailendra Singh <shailendras@nvidia.com>

Copy link
Copy Markdown
Collaborator

/ok to test eda6cc1

@shailendra-nv shailendra-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the per-sandbox TLS directory, CA delivery and cleanup paths, and Windows extended-length atomic writes. No actionable findings.

The branch now includes the existing Windows test repairs. All 46 enabled checks pass, including native Windows x64 and ARM64 lint/tests.

@shailendra-nv
shailendra-nv merged commit e451bb3 into NVIDIA:windows Oct 31, 2026
80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test:windows Run native Windows x64 and ARM64 lint/tests on PR mirrors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(mxc): host supervisors of all MXC sandboxes share one proxy CA directory, \etc\openshell-tls

2 participants