From df3d858b10836a62ea5959c1f31328c162f777b6 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 08:51:19 +0800 Subject: [PATCH 01/15] security: restricted unpickler on the sandbox->host return boundary (#116, PR #122 stage E) Return values travel host-ward as pickle.dumps(result) and the host decoded them with plain pickle.loads. pickle reconstructs objects by invoking whatever callables a stream names, so decoding a value shaped by a less-trusted sandbox workload (a cloned repo, a generated patch, a benchmark task) is a trust boundary: the returned object can direct reconstruction on the host (#116). Keep pickle as the return codec (the msgpack-returns alternative was evaluated and closed as #118) and decode host-side through a strict allowlist (agentix/runtime/shared/safepickle.py). find_class permits only: - a reviewed set of value types whose construction has no external effect (stdlib data: datetime/decimal/fractions/uuid/collections/ pathlib; builtin containers + exceptions; numpy arrays), and - a small set of inert reconstruction helpers (copyreg, numpy _reconstruct), and refuses everything else WITHOUT importing it. A denylist was rejected as unsound: a stream can name a C-accelerator module (_operator vs operator), a callable produced by one reconstruction step is invoked by the next without passing find_class (so attribute-access helpers must never be admitted), and many ordinary constructors have side effects. A closed allowlist of value types closes all three. First-party types (agentix.*) are trusted by default: the framework and its plugins are the trusted computing base that builds the bundle and runs the sandbox, and their return types (TunnelHandle, BashResult, agent results) are inert. This keeps the framework's own paths (Proxy.start, bash.run, agent adapters) working without setup. A workload's own return types are refused by default; opt in with safepickle.allow_module(prefix) / allow_callable(module, name), or set AGENTIX_PICKLE_TRUST=1 to trust the sandbox fully. A refusal raises agentix.RestrictedUnpickleError. Only the sandbox->host direction is restricted (client._unpickle_value, both the SIO and HTTP result paths). The sandbox-side decode of host-supplied arguments/context stays plain pickle -- the trusted host->sandbox direction. Tests: non-allowlisted callables (subprocess.check_output/Popen, os.system, eval) refused; attribute-access helpers (operator/_operator attrgetter/itemgetter/methodcaller, getattr) refused; refusal does not import the named module; object-dtype numpy arrays gate nested globals; first-party return types (TunnelHandle, BashResult) and permitted value types + builtin exceptions + workload-via-allow_module round-trip; trust bypass; end-to-end refusal over a real remote() call. PROTOCOL.md documents the boundary; RestrictedUnpickleError exported. Closes #116. Co-Authored-By: Claude --- .agents/skills/agentix-ray-build/SKILL.md | 88 ++++++++ agentix/__init__.py | 2 + agentix/runtime/PROTOCOL.md | 18 ++ agentix/runtime/client/client.py | 6 +- agentix/runtime/shared/safepickle.py | 204 ++++++++++++++++++ tests/_worker_target.py | 16 ++ tests/conftest.py | 8 + tests/runtime/test_protocol.py | 24 +++ tests/runtime/test_safepickle.py | 242 ++++++++++++++++++++++ tests/test_public_exports.py | 4 +- 10 files changed, 610 insertions(+), 2 deletions(-) create mode 100644 .agents/skills/agentix-ray-build/SKILL.md create mode 100644 agentix/runtime/shared/safepickle.py create mode 100644 tests/runtime/test_safepickle.py diff --git a/.agents/skills/agentix-ray-build/SKILL.md b/.agents/skills/agentix-ray-build/SKILL.md new file mode 100644 index 0000000..4d48b8a --- /dev/null +++ b/.agents/skills/agentix-ray-build/SKILL.md @@ -0,0 +1,88 @@ +--- +name: agentix-ray-build +description: Build and run Agentix bundles on a restricted remote Ray cluster (rootless podman, no-Docker pod, egress only via a corporate HTTP proxy). Covers the working podman build recipe, sandbox run-args, and how to read job logs when `ray job logs` is blocked. Use when building/testing an Agentix bundle on a Ray box that lacks Docker. +--- + +# Building & running Agentix bundles on a restricted Ray cluster + +Context: a remote Ray pod with **no Docker**, only **rootless podman** (4.x, crun), +**read-only cgroups**, and **egress only through a corporate HTTP proxy**. Jobs are +**ephemeral** (a `uv sync` in one job doesn't persist) and **logs are gateway-blocked**, +so build + run must happen in **one self-contained job** and progress is read via the +dashboard state API. Replace every `` with your environment's value. + +## Submitting (one job, ephemeral) + +Stage a clean repo and submit `python my_script.py` with `--working-dir` = repo root: + +``` +git archive HEAD | tar -x -C # clean tree, no .venv/.git +# write /my_script.py (the driver below), then submit via the cluster's +# ray job submit (e.g. `ray job submit --address http://:8081 \ +# --working-dir --no-wait -- python my_script.py`) +``` + +The driver must **stream** subprocess output (NOT `capture_output`) so it lands in the +job's `driver.log`, and end with a sentinel like `::RESULT rc=` + `sys.exit(rc)`. + +## The podman build recipe (what actually works) + +1. **Stage as a git repo.** `agentix build` copies the *whole git repo* so a project's + `../../plugins/*` path deps resolve. A non-git tree becomes a standalone context and + those deps break (`Distribution not found at file:///plugins/...`). If you shipped via + `git archive`, run `git init -q && git add -A` in the working dir first. +2. **Build RUN steps** must skip the read-only-cgroup + netns setup: + `--container-arg --isolation=chroot --container-arg --network=host`. +3. **Nix egress.** Public `cache.nixos.org` stalls/throttles through a corp proxy. Put a + fast mirror PRIMARY and keep `cache.nixos.org` as a coverage fallback — two separate + options (`substituters` replaces the default, `extra-substituters` appends): + `--nix-arg "--option substituters https://mirrors.ustc.edu.cn/nix-channels/store"` + `--nix-arg "--option extra-substituters https://cache.nixos.org"` + (TUNA `https://mirrors.tuna.tsinghua.edu.cn/nix-channels/store` works as primary too. + The CN mirrors mirror channel snapshots and may miss a path — hence the cache fallback.) +4. **Export step.** The bundle is extracted with `podman create --network none` + copy — + it never starts, so it needs NO cgroup/netns workaround. Do **NOT** pass + `--container-run-arg --network=host` here (clashes with the default `--network none` → + "cannot set multiple networks"). +5. **Trim host-only deps** from the bundle project — provider backends (`agentix-provider-*`) + are host-side and don't belong in the sandbox bundle (their `default.nix` can also drag + heavy system binaries into the closure). + +``` +git init -q && git add -A +HTTP_PROXY= HTTPS_PROXY= NO_PROXY=127.0.0.1,localhost \ +agentix build --container-engine podman --platform linux/amd64 \ + --container-arg --isolation=chroot --container-arg --network=host \ + --nix-arg "--option substituters https://mirrors.ustc.edu.cn/nix-channels/store" \ + --nix-arg "--option extra-substituters https://cache.nixos.org" +``` + +## Running the sandbox + +The sandbox container *does* start, so it **does** need the cgroup/netns workaround as +**run-args**: `--runtime=crun --cgroups=disabled --network=host`. With `--network=host` +the runtime server binds a host port (reach it at `127.0.0.1:`, no mapping). Wire +them via `agentix deploy podman --run-arg=--runtime=crun --run-arg=--cgroups=disabled +--run-arg=--network=host`, or the provider's run-arg config when orchestrating in-process. +(Use the `--run-arg=VALUE` form — values that start with `--` break argparse otherwise.) + +> Trade-offs of these run-args: `--network=host` removes network isolation; `--cgroups=disabled` +> removes resource limits. Fine for trusted single-tenant eval/RL; not for multi-tenant. + +## Reading logs when `ray job logs` is blocked + +The dashboard proxies `ray job logs` (and `/api/jobs//logs`) to a per-node job-agent on +an ephemeral internal port that isn't reachable through `:8081` (device-auth gateway) → +`ConnectionRefusedError`. Use the status endpoint + state API on `:8081` instead: + +``` +H=http://:8081 ; JID=raysubmit_... +curl -s "$H/api/jobs/$JID" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['status'],'|',(d.get('message') or '')[:200])" +NID=$(curl -s "$H/api/v0/nodes" | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['result']['result'][0]['node_id'])") +curl -s "$H/api/v0/logs?node_id=$NID&glob=*$JID*" # confirm job-driver-$JID.log exists +curl -s "$H/api/v0/logs/file?node_id=$NID&filename=job-driver-$JID.log&lines=200" | tr -d '\000' | tail -60 +``` + +Poll status until `SUCCEEDED`/`FAILED`/`STOPPED`. Device-auth challenges are intermittent +(transient `401`); just retry. If logs are *fully* unreachable, have the driver +`raise RuntimeError(tail)` so the tail surfaces in the job `message`. diff --git a/agentix/__init__.py b/agentix/__init__.py index 476faa3..4ceb6d2 100644 --- a/agentix/__init__.py +++ b/agentix/__init__.py @@ -34,6 +34,7 @@ ) from agentix.runtime.client._sio_facade import AsyncClientNamespace, request_handler from agentix.runtime.shared.callables import RemoteCallable +from agentix.runtime.shared.safepickle import RestrictedUnpickleError from agentix.sio import Namespace, RemoteSioError, register_namespace from agentix.utils import context, log, trace from agentix.utils.log import configure_logging @@ -52,6 +53,7 @@ "RemoteCallable", "RemoteCallError", "RemoteSioError", + "RestrictedUnpickleError", "Result", "RuntimeClient", "RuntimeUnreachable", diff --git a/agentix/runtime/PROTOCOL.md b/agentix/runtime/PROTOCOL.md index e30fdc9..234ee2d 100644 --- a/agentix/runtime/PROTOCOL.md +++ b/agentix/runtime/PROTOCOL.md @@ -24,6 +24,24 @@ code behind an importable top-level function instead. Args and kwargs travel separately as `arguments = pickle.dumps((args, kwargs))`. Return values travel as `value = pickle.dumps(result)`. +The **host** decodes a return value through a restricted unpickler +(`agentix.runtime.shared.safepickle`), not plain `pickle.loads`: a sandbox may +run less-trusted workloads whose returned object directs reconstruction, and +plain unpickling reconstructs objects by invoking whatever callables the stream +names. The restricted loader decodes through a strict **allowlist** — only a +reviewed set of value types (stdlib data, builtin containers and exceptions, +numpy arrays, …) and inert reconstruction helpers are permitted, and anything +else is refused *without importing it*. First-party types (`agentix.*`) are +trusted by default — the framework's own return types (`TunnelHandle`, +`BashResult`, agent results, …) cross the boundary with no setup. A *workload's* +own return types (a project's dataclasses / pydantic models) are refused by +default; opt them in with `safepickle.allow_module(prefix)` / +`allow_callable(module, name)`, or set `AGENTIX_PICKLE_TRUST=1` to trust the +sandbox fully. A refusal raises +`agentix.RestrictedUnpickleError`. (The sandbox-side decode of host-sent +`arguments` / `context` stays plain — that is the trusted host→sandbox +direction.) + ```python from my_project.tasks import run diff --git a/agentix/runtime/client/client.py b/agentix/runtime/client/client.py index 9b07caa..2112eba 100644 --- a/agentix/runtime/client/client.py +++ b/agentix/runtime/client/client.py @@ -37,6 +37,7 @@ from agentix.runtime.shared.callables import RemoteCallable, display_name_for from agentix.runtime.shared.codec import pack, unpack from agentix.runtime.shared.models import HealthResponse, RemoteError +from agentix.runtime.shared.safepickle import restricted_loads from agentix.utils import context logger = logging.getLogger("agentix.runtime.client") @@ -119,7 +120,10 @@ def _decode_payload(raw: Any) -> dict[str, Any]: def _unpickle_value(raw: Any) -> Any: - return pickle.loads(raw) if raw is not None else None + # Sandbox→host trust boundary (#116): the return value is decoded with the + # restricted unpickler so a malicious `__reduce__` can't execute code on the + # host. `AGENTIX_PICKLE_TRUST=1` opts back into plain pickle. + return restricted_loads(raw) if raw is not None else None class RuntimeClient: diff --git a/agentix/runtime/shared/safepickle.py b/agentix/runtime/shared/safepickle.py new file mode 100644 index 0000000..a430a0d --- /dev/null +++ b/agentix/runtime/shared/safepickle.py @@ -0,0 +1,204 @@ +"""Restricted unpickling for the sandbox→host return boundary (#116). + +Return values travel host-ward as `pickle.dumps(result)` and the host decodes +them. `pickle.loads` reconstructs arbitrary objects by invoking whatever +callables a stream names, so decoding a value influenced by sandbox-side code +is a trust-boundary concern: the sandbox may run less-trusted workloads (a +cloned repository, a generated patch, a benchmark task) whose returned object +can direct reconstruction on the host. + +`restricted_loads` keeps pickle as the wire format but decodes through a strict +**allowlist**: `find_class` permits only an explicit, individually reviewed set +of data types and inert reconstruction helpers, and refuses everything else. A +refused global's module is never imported (the refusal is decided from the name), +so a stream cannot force import-time code to run on the host. Because the +permitted set contains only value-shaped types (their construction has no +external side effects) and helpers that merely rebuild those values, a decoded +stream cannot reach a callable that acts on the host. + +Why an allowlist and not a denylist of "dangerous" names: + + * A stream can reference a global's C-accelerator module (`_socket` vs + `socket`, `_operator` vs `operator`), so name-based blocking is porous. + * A callable produced by one reconstruction step sits on pickle's stack and + is invoked by the next step *without* going through `find_class`, so + admitting attribute-access helpers (`getattr`, `operator.attrgetter`, …) + lets a stream walk from any admitted object to an arbitrary callable. + * Many ordinary constructors have side effects (opening files, binding + sockets, importing modules), so "admit any class" is not safe either. + +Only a closed allowlist of value types closes all three. + +Scope: this guards the host-side decode of sandbox return values only +(`RuntimeClient._unpickle_value`). The sandbox-side decode of host-supplied +arguments and context stays plain pickle — that is the trusted host→sandbox +direction. + +First-party types (`agentix.*`) are trusted by default: the framework and its +plugins build the bundle and run the sandbox, so their own return types +(`TunnelHandle`, `BashResult`, agent results, …) are part of the trusted +computing base. The boundary defends against a *workload's* return value, and a +workload's own types are not `agentix.*` — they stay opt-in. + +Extending / relaxing: + + * `allow_module(prefix)` / `allow_callable(module, name)` add return types the + default set does not cover (e.g. a project's own dataclasses / pydantic + models). Prefer these over the full bypass. + * `AGENTIX_PICKLE_TRUST=1` restores plain `pickle.loads` for deployments where + the entire sandbox — including any workload it runs — is trusted. +""" + +from __future__ import annotations + +import io +import os +import pickle +from typing import Any + +# Inert reconstruction helpers: functions that only rebuild a value from +# following (also-gated) arguments. Each is reviewed to have no external effect. +SAFE_CALLABLES: set[tuple[str, str]] = { + ("copyreg", "_reconstructor"), + ("copyreg", "__newobj__"), + ("copyreg", "__newobj_ex__"), + ("numpy._core.multiarray", "_reconstruct"), # numpy >= 2.0 + ("numpy.core.multiarray", "_reconstruct"), # numpy < 2.0 +} + +# Value types whose construction has no external side effect. Reviewed one by +# one; modules here are stdlib/data packages that are inert to import. +SAFE_TYPES: set[tuple[str, str]] = { + ("datetime", "date"), + ("datetime", "time"), + ("datetime", "datetime"), + ("datetime", "timedelta"), + ("datetime", "timezone"), + ("decimal", "Decimal"), + ("fractions", "Fraction"), + ("uuid", "UUID"), + ("collections", "OrderedDict"), + ("collections", "defaultdict"), + ("collections", "Counter"), + ("collections", "deque"), + ("pathlib", "PurePath"), + ("pathlib", "PurePosixPath"), + ("pathlib", "PureWindowsPath"), + ("pathlib", "Path"), + ("pathlib", "PosixPath"), + ("pathlib", "WindowsPath"), + ("numpy", "ndarray"), + ("numpy", "dtype"), +} + +# Builtin value types admitted by identity (see `_SAFE_BUILTIN_TYPES`) plus every +# builtin exception subclass. Builtins are always imported, so resolving one has +# no import side effect. Builtin *functions* (getattr/eval/exec/compile/open/ +# __import__/…) are types-of `builtin_function_or_method`, not `type`, so the +# "must be an allowed type" check below excludes them. +_SAFE_BUILTIN_TYPES: frozenset[type] = frozenset( + { + complex, range, slice, bytearray, frozenset, + set, list, dict, tuple, bytes, str, int, float, bool, + } +) + +# First-party namespace, trusted by default. The #116 boundary defends against +# an *untrusted workload's* return value; the framework and its plugins are part +# of the trusted computing base that builds the bundle and runs the sandbox, and +# their return types (`TunnelHandle`, `BashResult`, agent results, …) are inert +# dataclasses/models. Trusting `agentix.*` keeps the framework's own paths +# working; a workload's own return types stay opt-in, and gadget callables +# (subprocess/os/eval/attribute-access helpers) are never first-party so remain +# refused. +_FIRST_PARTY_PREFIXES: tuple[str, ...] = ("agentix",) + +# Module prefixes the caller has explicitly opted to trust for return types the +# default set does not cover. +_ALLOWED_MODULE_PREFIXES: set[str] = set() + + +class RestrictedUnpickleError(pickle.UnpicklingError): + """A global in the stream was not on the host allowlist and was refused.""" + + +def allow_module(prefix: str) -> None: + """Trust every global whose module equals or starts with `prefix` (dotted). + Use for a package whose return types the default allowlist does not cover.""" + _ALLOWED_MODULE_PREFIXES.add(prefix) + + +def allow_callable(module: str, name: str) -> None: + """Trust one specific `module.name` type/helper by exact identity.""" + SAFE_CALLABLES.add((module, name)) + + +def _trust_enabled() -> bool: + return os.environ.get("AGENTIX_PICKLE_TRUST", "").strip().lower() in ("1", "true", "yes") + + +def _module_allowed(module: str) -> bool: + prefixes = (*_FIRST_PARTY_PREFIXES, *_ALLOWED_MODULE_PREFIXES) + return any(module == p or module.startswith(p + ".") for p in prefixes) + + +def _is_safe_type(module: str, name: str) -> bool: + """`(module, name)` is an allowlisted value type, tolerating a private + implementation submodule of an allowlisted public module — e.g. Python 3.13 + pickles `pathlib.PurePosixPath` as `pathlib._local.PurePosixPath`. Only + underscore-prefixed submodules of the public root are accepted, so a public + sibling module (`pathlib.evil`) is not.""" + if (module, name) in SAFE_TYPES: + return True + head, _, rest = module.partition(".") + if rest and all(part.startswith("_") for part in rest.split(".")): + return (head, name) in SAFE_TYPES + return False + + +class RestrictedUnpickler(pickle.Unpickler): + """`pickle.Unpickler` whose `find_class` enforces the allowlist above.""" + + def find_class(self, module: str, name: str) -> Any: + if (module, name) in SAFE_CALLABLES or _is_safe_type(module, name) or _module_allowed(module): + return super().find_class(module, name) + + if module == "builtins": + # Builtins are already imported — resolving has no import effect. + obj = super().find_class(module, name) + if isinstance(obj, type) and (obj in _SAFE_BUILTIN_TYPES or issubclass(obj, BaseException)): + return obj + raise RestrictedUnpickleError( + f"refusing builtins.{name}: only builtin value types and exceptions may " + f"cross the sandbox→host boundary. Set AGENTIX_PICKLE_TRUST=1 to trust the " + f"sandbox fully." + ) + + # Not on the allowlist — refuse WITHOUT importing `module` (importing an + # arbitrary module would itself run its top-level code on the host). + raise RestrictedUnpickleError( + f"refusing to reconstruct {module}.{name}: it is not on the host allowlist for " + f"sandbox return values. If this is a return type you trust, call " + f"agentix.runtime.shared.safepickle.allow_module({module!r}) (or allow_callable) " + f"before the call; or set AGENTIX_PICKLE_TRUST=1 to trust the sandbox fully." + ) + + +def restricted_loads(data: bytes) -> Any: + """Decode sandbox-supplied pickle bytes through the host allowlist. + + Honors `AGENTIX_PICKLE_TRUST=1` as a full-trust bypass (plain `pickle.loads`).""" + if _trust_enabled(): + return pickle.loads(data) + return RestrictedUnpickler(io.BytesIO(data)).load() + + +__all__ = [ + "RestrictedUnpickleError", + "RestrictedUnpickler", + "SAFE_CALLABLES", + "SAFE_TYPES", + "allow_callable", + "allow_module", + "restricted_loads", +] diff --git a/tests/_worker_target.py b/tests/_worker_target.py index 43a4ecb..7bf7daa 100644 --- a/tests/_worker_target.py +++ b/tests/_worker_target.py @@ -90,3 +90,19 @@ def spawn_stdin_reading_child() -> int: timeout=10, ) return proc.returncode + + +class _ReducesToSubprocess: + """A return object whose `__reduce__` directs reconstruction at + `subprocess.check_output` — a callable the host must not invoke while + decoding a sandbox return value (#116). The argument is benign; the host's + restricted unpickler refuses it before anything runs.""" + + def __reduce__(self): + import subprocess + + return (subprocess.check_output, (["true"],)) + + +def return_unsafe_reducer() -> object: + return _ReducesToSubprocess() diff --git a/tests/conftest.py b/tests/conftest.py index ace4145..ddc247d 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -16,6 +16,14 @@ import pytest +from agentix.runtime.shared import safepickle + +# The host restricts reconstruction of sandbox return values to an allowlist +# (#116). Tests return custom types from `tests._worker_target` (e.g. the +# `EchoResult` pydantic model); opt that module in, exactly as a real project +# opts in its own return types. +safepickle.allow_module("tests._worker_target") + @pytest.fixture def free_port() -> int: diff --git a/tests/runtime/test_protocol.py b/tests/runtime/test_protocol.py index 63b86a7..eac47c7 100644 --- a/tests/runtime/test_protocol.py +++ b/tests/runtime/test_protocol.py @@ -526,3 +526,27 @@ async def test_malformed_call_error_is_typed_not_keyerror(use_inprocess_worker, with pytest.raises(RemoteCallError) as ei: await task assert ei.value.error.type == "MalformedError" + + +async def test_unsafe_return_value_refused_host_side(use_inprocess_worker, live_server): + """A sandbox return value whose `__reduce__` names a non-allowlisted + callable is refused when the host decodes it (#116).""" + from agentix.runtime.shared.safepickle import RestrictedUnpickleError + + use_inprocess_worker() + base_url = await live_server() + async with RuntimeClient(base_url) as c: + with pytest.raises(RestrictedUnpickleError): + await c.remote(target.return_unsafe_reducer) + + +async def test_pydantic_return_value_round_trips_through_restricted_loads( + use_inprocess_worker, live_server +): + """The restricted host boundary must not break the common case: a custom + pydantic model returned from the sandbox still decodes.""" + use_inprocess_worker() + base_url = await live_server() + async with RuntimeClient(base_url) as c: + result = await c.remote(target.echo, "hi") + assert result.msg == "echo:hi" diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py new file mode 100644 index 0000000..4a1a0dd --- /dev/null +++ b/tests/runtime/test_safepickle.py @@ -0,0 +1,242 @@ +"""The host restricts what it will reconstruct from a sandbox return value (#116). + +`pickle.loads` reconstructs objects by invoking whatever callables a stream +names, so decoding a sandbox-influenced value is a trust boundary. `restricted_loads` +decodes through a strict allowlist: only reviewed value types and inert +reconstruction helpers are permitted; everything else is refused without +importing it. These tests verify both halves — permitted values round-trip, and +reconstruction of non-allowlisted callables/types is refused. +""" + +from __future__ import annotations + +import collections +import datetime +import decimal +import fractions +import pathlib +import pickle +import uuid +from dataclasses import dataclass + +import pytest +from pydantic import BaseModel + +from agentix.runtime.shared import safepickle +from agentix.runtime.shared.safepickle import ( + RestrictedUnpickleError, + allow_module, + restricted_loads, +) + + +@pytest.fixture(autouse=True) +def _restore_allowlist(): + """`allow_module` / `allow_callable` mutate module-global state — snapshot + and restore it so tests do not leak opt-ins into each other.""" + prefixes = set(safepickle._ALLOWED_MODULE_PREFIXES) + callables = set(safepickle.SAFE_CALLABLES) + try: + yield + finally: + safepickle._ALLOWED_MODULE_PREFIXES.clear() + safepickle._ALLOWED_MODULE_PREFIXES.update(prefixes) + safepickle.SAFE_CALLABLES.clear() + safepickle.SAFE_CALLABLES.update(callables) + + +# ── objects that direct reconstruction at non-allowlisted callables ────────── +# Each `__reduce__` names a callable that a restricted host decode must refuse. +# The referenced callables are the ones named in issue #116; the arguments are +# benign so nothing runs even if a regression let one through. + + +class _ReducesToSubprocessCheckOutput: + def __reduce__(self): + import subprocess + + return (subprocess.check_output, (["true"],)) + + +class _ReducesToSubprocessPopen: + def __reduce__(self): + import subprocess + + return (subprocess.Popen, (["true"],)) + + +class _ReducesToEval: + def __reduce__(self): + return (eval, ("1 + 1",)) + + +class _ReducesToOsSystem: + def __reduce__(self): + import os + + return (os.system, ("true",)) + + +@pytest.mark.parametrize( + "obj_cls", + [ + _ReducesToSubprocessCheckOutput, + _ReducesToSubprocessPopen, + _ReducesToEval, + _ReducesToOsSystem, + ], +) +def test_non_allowlisted_callable_is_refused(obj_cls) -> None: + blob = pickle.dumps(obj_cls()) + with pytest.raises(RestrictedUnpickleError): + restricted_loads(blob) + + +def test_refusal_does_not_import_the_named_module() -> None: + """A refusal is decided from the module name — the restricted decoder must + not import a non-allowlisted module (importing runs its top-level code on + the host).""" + import sys + + modname = "xml.dom.minidom" # importable, stdlib, not on the allowlist + sys.modules.pop(modname, None) + crafted = b"\x80\x04c" + modname.encode() + b"\nDocument\n." + with pytest.raises(RestrictedUnpickleError): + restricted_loads(crafted) + assert modname not in sys.modules, "restricted decode must not import a non-allowlisted module" + + +@pytest.mark.parametrize( + "crafted", + [ + # Attribute-access helpers — the technique that would let a stream walk + # from an admitted object to an arbitrary callable. Neither the pure- + # python nor the C-accelerator module may be on the allowlist. + b"\x80\x04coperator\nattrgetter\n.", + b"\x80\x04c_operator\nattrgetter\n.", + b"\x80\x04c_operator\nitemgetter\n.", + b"\x80\x04c_operator\nmethodcaller\n.", + b"\x80\x04cbuiltins\ngetattr\n.", + ], +) +def test_attribute_access_helpers_are_refused(crafted) -> None: + with pytest.raises(RestrictedUnpickleError): + restricted_loads(crafted) + + +# ── permitted values round-trip ───────────────────────────────────────────── + + +@pytest.mark.parametrize( + "value", + [ + 42, "hi", 3.5, True, None, b"bytes", bytearray(b"ba"), + complex(1, 2), range(0, 10, 2), slice(1, 9, 2), + [1, 2, 3], {"a": 1}, (1, 2), {1, 2}, frozenset([1]), + {"nested": [1, {"b": (2, 3)}]}, + datetime.datetime(2026, 1, 1, 12, 30), + datetime.date(2026, 1, 1), + datetime.timedelta(days=2), + datetime.datetime(2026, 1, 1, tzinfo=datetime.UTC), + decimal.Decimal("1.5"), + fractions.Fraction(3, 4), + uuid.UUID("12345678123456781234567812345678"), + collections.OrderedDict(a=1, b=2), + collections.Counter("aabbc"), + collections.deque([1, 2, 3]), + collections.defaultdict(int, {"a": 1}), + pathlib.PurePosixPath("/tmp/x"), + ], +) +def test_permitted_values_round_trip(value) -> None: + assert restricted_loads(pickle.dumps(value)) == value + + +@pytest.mark.parametrize("exc", [ValueError("boom"), KeyError("k"), RuntimeError("r")]) +def test_builtin_exceptions_round_trip(exc) -> None: + out = restricted_loads(pickle.dumps(exc)) + assert type(out) is type(exc) + assert out.args == exc.args + + +def test_numpy_array_round_trips() -> None: + np = pytest.importorskip("numpy") + arr = np.arange(6).reshape(2, 3) + out = restricted_loads(pickle.dumps(arr)) + assert np.array_equal(out, arr) + + +def test_numpy_object_array_gates_nested_globals() -> None: + """An object-dtype array pickles its elements in the same stream, so a + non-allowlisted callable referenced by an element is still refused.""" + np = pytest.importorskip("numpy") + arr = np.array([_ReducesToOsSystem()], dtype=object) + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(arr)) + + +def test_none_returns_none() -> None: + assert restricted_loads(pickle.dumps(None)) is None + + +# ── custom return types: refused by default, permitted by opt-in ───────────── + + +class _ProjectResult(BaseModel): + patch: str + score: int + + +@dataclass +class _ProjectPoint: + x: int + y: int + + +def test_first_party_return_types_round_trip_by_default() -> None: + """The framework's own shipped return types are trusted (`agentix.*`) and + must cross the boundary with no opt-in — otherwise the restriction breaks + the framework's own main paths (Proxy.start, bash.run, agent adapters).""" + from agentix.bridge.proxy import TunnelHandle + + handle = TunnelHandle(url="http://127.0.0.1:9", port=9) + assert restricted_loads(pickle.dumps(handle)) == handle + + bash = pytest.importorskip("agentix.bash") + result = bash.BashResult(stdout="o", stderr="", exit_code=0) + assert restricted_loads(pickle.dumps(result)) == result + + +def test_custom_type_refused_by_default() -> None: + # This module is not first-party (`agentix.*`) and not on the opt-in list. + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) + + +def test_allow_module_permits_custom_types() -> None: + allow_module("tests.runtime.test_safepickle") + model = _ProjectResult(patch="d", score=3) + point = _ProjectPoint(1, 2) + assert restricted_loads(pickle.dumps(model)) == model + assert restricted_loads(pickle.dumps(point)) == point + + +def test_trust_env_disables_restriction(monkeypatch) -> None: + """The escape hatch fully trusts the sandbox. Verified with a benign + reducer (json.dumps) that the restricted path refuses but is harmless.""" + monkeypatch.setenv("AGENTIX_PICKLE_TRUST", "1") + + class _ReducesToJsonDumps: + def __reduce__(self): + import json + + return (json.dumps, ([1, 2, 3],)) + + assert restricted_loads(pickle.dumps(_ReducesToJsonDumps())) == "[1, 2, 3]" + + +def test_refusal_message_points_to_opt_in() -> None: + with pytest.raises(RestrictedUnpickleError) as ei: + restricted_loads(pickle.dumps(_ProjectPoint(1, 2))) + msg = str(ei.value) + assert "allow_module" in msg or "AGENTIX_PICKLE_TRUST" in msg diff --git a/tests/test_public_exports.py b/tests/test_public_exports.py index 5c2cb1e..efc711d 100644 --- a/tests/test_public_exports.py +++ b/tests/test_public_exports.py @@ -15,6 +15,7 @@ def test_failure_vocabulary_importable_from_agentix() -> None: Failed, Ok, RemoteCallError, + RestrictedUnpickleError, Result, RuntimeUnreachable, WorkerExited, @@ -26,6 +27,7 @@ def test_failure_vocabulary_importable_from_agentix() -> None: # allow branching. assert issubclass(WorkerExited, RemoteCallError) assert issubclass(CallCancelled, RemoteCallError) + assert issubclass(RestrictedUnpickleError, Exception) assert Ok(1).value == 1 and Failed(ValueError()).error is not None assert Result[int] # generic union alias is subscriptable assert callable(configure_logging) @@ -36,7 +38,7 @@ def test_failure_vocabulary_importable_from_agentix() -> None: def test_failure_vocabulary_in_dunder_all() -> None: for name in ( "CallCancelled", "CallTimeout", "Failed", "Ok", "Result", - "RuntimeUnreachable", "WorkerExited", "configure_logging", + "RestrictedUnpickleError", "RuntimeUnreachable", "WorkerExited", "configure_logging", ): assert name in agentix.__all__ From 313d11cb7797daefb4c7fab2c7baaa79f988bd7d Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 13:08:17 +0800 Subject: [PATCH 02/15] docs: design exact-type restricted unpickling --- ...03-stage-e-exact-type-unpickling-design.md | 114 ++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-03-stage-e-exact-type-unpickling-design.md diff --git a/docs/superpowers/specs/2026-07-03-stage-e-exact-type-unpickling-design.md b/docs/superpowers/specs/2026-07-03-stage-e-exact-type-unpickling-design.md new file mode 100644 index 0000000..66cf7f3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-03-stage-e-exact-type-unpickling-design.md @@ -0,0 +1,114 @@ +# Stage E Exact-Type Restricted Unpickling Design + +## Status and scope + +This design replaces the unsafe default trust of the complete `agentix.*` +namespace in the Stage E sandbox-to-host return decoder. It is intentionally +limited to `agentix.runtime.shared.safepickle`, its public API and protocol +documentation, and directly relevant unit/integration tests. Existing issues in +providers, streaming, runner loading, packaging, documentation maturity, and +release process remain outside this change. + +The work is split sequentially. Codex owns the security-sensitive decoder +change and harmless regression tests. A later Fable pass receives only a +sanitized compatibility task: check the first-party type inventory, update +ordinary documentation, and run the full verification suite. + +## Security invariant + +Sandbox-controlled pickle data may resolve only: + +1. exact value-type identities that were individually reviewed; and +2. exact reconstruction-helper identities that were individually reviewed. + +Package provenance is not a safety property. A trusted package can contain +effectful functions, policy-mutating functions, and classes with effectful +construction hooks. Therefore neither `agentix.*` nor a workload package may be +admitted by module prefix. + +## Decoder and registration design + +`RestrictedUnpickler.find_class()` uses two closed exact registries: + +- `SAFE_TYPES: frozenset[tuple[str, str]]` for value-shaped classes; +- an internal immutable helper table for inert reconstruction helpers. + +The `_FIRST_PARTY_PREFIXES`, `_ALLOWED_MODULE_PREFIXES`, `_module_allowed()`, +and `allow_module()` surfaces are deleted. There is no compatibility alias or +deprecation shim because the repository explicitly permits breaking design +changes. + +A new `allow_type(cls: type[Any]) -> None` host-side opt-in stores the exact +`(cls.__module__, cls.__qualname__)` identity and the class object in a private +registry. It accepts only actual classes. On a match, `find_class()` returns the +registered class object rather than importing the module again, so later module +rebinding cannot change what the operator approved. + +The public `allow_callable()` surface is also deleted. It is new in Stage E, +has no repository caller, and conflicts with the value-type-only policy. The +small fixed set of pickle reconstruction helpers stays internal and immutable. + +Required first-party return classes are listed as exact string identities in +`SAFE_TYPES`. String identities preserve workspace dependency separation: core +does not import abridge or runtime-basic packages while importing safepickle. +The class module is imported only if a return pickle actually references that +exact reviewed identity. Python-version aliases such as `pathlib.*` and +`pathlib._local.*` are enumerated explicitly; no private-submodule fuzzy match +is retained. + +## Data flow and failure behavior + +The worker still serializes a result with stdlib pickle. On the host, both HTTP +and Socket.IO result paths call `restricted_loads()`. + +For each referenced global: + +1. exact safe helper/type identity is checked before importing its module; +2. approved builtin types and builtin exceptions retain their identity-based + check; +3. every other global is rejected with `RestrictedUnpickleError` without + importing the named module. + +A failed decode must not alter either registry. `AGENTIX_PICKLE_TRUST=1` remains +the explicit full-trust escape hatch and is outside the protected default. + +## Harmless regression strategy + +Security regression tests contain no real shell command, process launch, file +write, or network request. They use pickle `GLOBAL` lookups, pure functions, +monkeypatched recorders, and registry snapshots. + +The test-first cases are: + +- an unregistered function from an `agentix.*` module is rejected before it can + be invoked; +- decoder configuration functions are themselves rejected; +- one pickle load cannot change policy and then resolve a previously forbidden + pure callable; +- registry state is unchanged after rejection; +- a workload class is rejected by default and round-trips only after + `allow_type(TheClass)`; +- every exact first-party value type required by Agentix main paths round-trips + by default; +- direct non-first-party callable and attribute-access gadget regressions remain + green; +- protocol tests cover both HTTP fast-path and Socket.IO fallback decoding. + +Each new behavioral test is run against the current implementation first and +must fail for the intended missing control before production code changes. + +## Verification and handoff + +Codex runs the focused safepickle and protocol tests while implementing the +security core. After the focused suite passes, the sanitized Fable task is to: + +1. compare the exact first-party type inventory with actual public return + surfaces; +2. add any missing compatibility-only cases using inert values; +3. update protocol/public API prose without exploit descriptions; +4. run the complete pytest suite and whole-workspace pyright; and +5. create a local commit without pushing or opening a pull request. + +The final merge gate is that all intended values cross the boundary, every +unregistered global remains rejected regardless of package prefix, failed loads +cannot mutate policy, and both transport paths use the same protected decoder. From 5c3833483b68d861306e5c2ec6eca6110e3e99ab Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 13:21:52 +0800 Subject: [PATCH 03/15] security: add exact pickle type opt-in --- agentix/runtime/shared/safepickle.py | 14 ++++++++++++++ tests/runtime/test_safepickle.py | 20 +++++++++++++++----- 2 files changed, 29 insertions(+), 5 deletions(-) diff --git a/agentix/runtime/shared/safepickle.py b/agentix/runtime/shared/safepickle.py index a430a0d..9524c95 100644 --- a/agentix/runtime/shared/safepickle.py +++ b/agentix/runtime/shared/safepickle.py @@ -56,6 +56,9 @@ import pickle from typing import Any +GlobalName = tuple[str, str] +_ALLOWED_TYPES: dict[GlobalName, type[Any]] = {} + # Inert reconstruction helpers: functions that only rebuild a value from # following (also-gated) arguments. Each is reviewed to have no external effect. SAFE_CALLABLES: set[tuple[str, str]] = { @@ -133,6 +136,12 @@ def allow_callable(module: str, name: str) -> None: SAFE_CALLABLES.add((module, name)) +def allow_type(cls: type[Any]) -> None: + if not isinstance(cls, type): + raise TypeError("allow_type() requires a class") + _ALLOWED_TYPES[(cls.__module__, cls.__qualname__)] = cls + + def _trust_enabled() -> bool: return os.environ.get("AGENTIX_PICKLE_TRUST", "").strip().lower() in ("1", "true", "yes") @@ -160,6 +169,10 @@ class RestrictedUnpickler(pickle.Unpickler): """`pickle.Unpickler` whose `find_class` enforces the allowlist above.""" def find_class(self, module: str, name: str) -> Any: + allowed_type = _ALLOWED_TYPES.get((module, name)) + if allowed_type is not None: + return allowed_type + if (module, name) in SAFE_CALLABLES or _is_safe_type(module, name) or _module_allowed(module): return super().find_class(module, name) @@ -200,5 +213,6 @@ def restricted_loads(data: bytes) -> Any: "SAFE_TYPES", "allow_callable", "allow_module", + "allow_type", "restricted_loads", ] diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index 4a1a0dd..8537eaa 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -25,7 +25,6 @@ from agentix.runtime.shared import safepickle from agentix.runtime.shared.safepickle import ( RestrictedUnpickleError, - allow_module, restricted_loads, ) @@ -36,6 +35,8 @@ def _restore_allowlist(): and restore it so tests do not leak opt-ins into each other.""" prefixes = set(safepickle._ALLOWED_MODULE_PREFIXES) callables = set(safepickle.SAFE_CALLABLES) + allowed_types = getattr(safepickle, "_ALLOWED_TYPES", None) + types = dict(allowed_types) if allowed_types is not None else None try: yield finally: @@ -43,6 +44,9 @@ def _restore_allowlist(): safepickle._ALLOWED_MODULE_PREFIXES.update(prefixes) safepickle.SAFE_CALLABLES.clear() safepickle.SAFE_CALLABLES.update(callables) + if allowed_types is not None and types is not None: + allowed_types.clear() + allowed_types.update(types) # ── objects that direct reconstruction at non-allowlisted callables ────────── @@ -213,13 +217,19 @@ def test_custom_type_refused_by_default() -> None: restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) -def test_allow_module_permits_custom_types() -> None: - allow_module("tests.runtime.test_safepickle") - model = _ProjectResult(patch="d", score=3) +def test_allow_type_opt_in_is_exact() -> None: point = _ProjectPoint(1, 2) - assert restricted_loads(pickle.dumps(model)) == model + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(point)) + + allow_type = getattr(safepickle, "allow_type", None) + assert allow_type is not None + allow_type(_ProjectPoint) assert restricted_loads(pickle.dumps(point)) == point + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) + def test_trust_env_disables_restriction(monkeypatch) -> None: """The escape hatch fully trusts the sandbox. Verified with a benign From 3f17cb16ec441eecf42576dd41a19dd8013d46a5 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 13:29:47 +0800 Subject: [PATCH 04/15] test: cover exact pickle policy boundary --- tests/runtime/test_safepickle.py | 45 ++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index 8537eaa..a9bc746 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -29,6 +29,19 @@ ) +def _global_reference(module: str, name: str) -> bytes: + assert "\n" not in module and "\n" not in name + return f"c{module}\n{name}\n.".encode("ascii") + + +_POLICY_CALLS: list[str] = [] + + +def _policy_recorder(value: str) -> str: + _POLICY_CALLS.append(value) + return value + + @pytest.fixture(autouse=True) def _restore_allowlist(): """`allow_module` / `allow_callable` mutate module-global state — snapshot @@ -37,6 +50,7 @@ def _restore_allowlist(): callables = set(safepickle.SAFE_CALLABLES) allowed_types = getattr(safepickle, "_ALLOWED_TYPES", None) types = dict(allowed_types) if allowed_types is not None else None + policy_calls = list(_POLICY_CALLS) try: yield finally: @@ -47,6 +61,8 @@ def _restore_allowlist(): if allowed_types is not None and types is not None: allowed_types.clear() allowed_types.update(types) + _POLICY_CALLS.clear() + _POLICY_CALLS.extend(policy_calls) # ── objects that direct reconstruction at non-allowlisted callables ────────── @@ -96,6 +112,35 @@ def test_non_allowlisted_callable_is_refused(obj_cls) -> None: restricted_loads(blob) +def test_unregistered_first_party_function_is_refused() -> None: + with pytest.raises(RestrictedUnpickleError): + restricted_loads(_global_reference("agentix.runtime.shared.safepickle", "_trust_enabled")) + + +@pytest.mark.parametrize("name", ["allow_module", "allow_callable", "allow_type"]) +def test_policy_functions_are_refused(name: str) -> None: + with pytest.raises(RestrictedUnpickleError): + restricted_loads(_global_reference("agentix.runtime.shared.safepickle", name)) + + +def test_one_load_cannot_modify_policy_then_invoke_new_global() -> None: + _POLICY_CALLS.clear() + before_callables = set(safepickle.SAFE_CALLABLES) + before_types = dict(safepickle._ALLOWED_TYPES) + payload = ( + b"\x80\x04" + b"cagentix.runtime.shared.safepickle\nallow_callable\n" + b"(Vtests.runtime.test_safepickle\nV_policy_recorder\ntR0" + b"ctests.runtime.test_safepickle\n_policy_recorder\n" + b"(Vmarker\ntR." + ) + with pytest.raises(RestrictedUnpickleError): + restricted_loads(payload) + assert _POLICY_CALLS == [] + assert safepickle.SAFE_CALLABLES == before_callables + assert safepickle._ALLOWED_TYPES == before_types + + def test_refusal_does_not_import_the_named_module() -> None: """A refusal is decided from the module name — the restricted decoder must not import a non-allowlisted module (importing runs its top-level code on From 2c6bfe6f2d377d094168d7115155fc3faed195b9 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 13:42:32 +0800 Subject: [PATCH 05/15] security: restrict pickle globals to exact values --- agentix/runtime/shared/safepickle.py | 160 ++++++++++++--------------- tests/conftest.py | 7 +- tests/runtime/test_safepickle.py | 19 +--- 3 files changed, 78 insertions(+), 108 deletions(-) diff --git a/agentix/runtime/shared/safepickle.py b/agentix/runtime/shared/safepickle.py index 9524c95..3506ac1 100644 --- a/agentix/runtime/shared/safepickle.py +++ b/agentix/runtime/shared/safepickle.py @@ -34,17 +34,17 @@ arguments and context stays plain pickle — that is the trusted host→sandbox direction. -First-party types (`agentix.*`) are trusted by default: the framework and its -plugins build the bundle and run the sandbox, so their own return types -(`TunnelHandle`, `BashResult`, agent results, …) are part of the trusted -computing base. The boundary defends against a *workload's* return value, and a -workload's own types are not `agentix.*` — they stay opt-in. +Only the exact first-party value types listed in `SAFE_TYPES` are trusted by +default. The framework and its plugins build the bundle and run the sandbox, +so these reviewed return types (`TunnelHandle`, `BashResult`, agent results, +…) are part of the trusted computing base. Other first-party globals and a +workload's own types stay opt-in. Extending / relaxing: - * `allow_module(prefix)` / `allow_callable(module, name)` add return types the - default set does not cover (e.g. a project's own dataclasses / pydantic - models). Prefer these over the full bypass. + * `allow_type(Class)` adds one exact runtime class the default set does not + cover (e.g. a project's own dataclass / pydantic model). Prefer this over + the full bypass. * `AGENTIX_PICKLE_TRUST=1` restores plain `pickle.loads` for deployments where the entire sandbox — including any workload it runs — is trusted. """ @@ -61,38 +61,54 @@ # Inert reconstruction helpers: functions that only rebuild a value from # following (also-gated) arguments. Each is reviewed to have no external effect. -SAFE_CALLABLES: set[tuple[str, str]] = { - ("copyreg", "_reconstructor"), - ("copyreg", "__newobj__"), - ("copyreg", "__newobj_ex__"), - ("numpy._core.multiarray", "_reconstruct"), # numpy >= 2.0 - ("numpy.core.multiarray", "_reconstruct"), # numpy < 2.0 -} +_SAFE_CALLABLES: frozenset[GlobalName] = frozenset( + { + ("copyreg", "_reconstructor"), + ("copyreg", "__newobj__"), + ("copyreg", "__newobj_ex__"), + ("numpy._core.multiarray", "_reconstruct"), # numpy >= 2.0 + ("numpy.core.multiarray", "_reconstruct"), # numpy < 2.0 + } +) # Value types whose construction has no external side effect. Reviewed one by -# one; modules here are stdlib/data packages that are inert to import. -SAFE_TYPES: set[tuple[str, str]] = { - ("datetime", "date"), - ("datetime", "time"), - ("datetime", "datetime"), - ("datetime", "timedelta"), - ("datetime", "timezone"), - ("decimal", "Decimal"), - ("fractions", "Fraction"), - ("uuid", "UUID"), - ("collections", "OrderedDict"), - ("collections", "defaultdict"), - ("collections", "Counter"), - ("collections", "deque"), - ("pathlib", "PurePath"), - ("pathlib", "PurePosixPath"), - ("pathlib", "PureWindowsPath"), - ("pathlib", "Path"), - ("pathlib", "PosixPath"), - ("pathlib", "WindowsPath"), - ("numpy", "ndarray"), - ("numpy", "dtype"), -} +# one; modules here are stdlib/data packages or exact first-party value modules. +SAFE_TYPES: frozenset[GlobalName] = frozenset( + { + ("datetime", "date"), + ("datetime", "time"), + ("datetime", "datetime"), + ("datetime", "timedelta"), + ("datetime", "timezone"), + ("decimal", "Decimal"), + ("fractions", "Fraction"), + ("uuid", "UUID"), + ("collections", "OrderedDict"), + ("collections", "defaultdict"), + ("collections", "Counter"), + ("collections", "deque"), + ("pathlib", "PurePath"), + ("pathlib", "PurePosixPath"), + ("pathlib", "PureWindowsPath"), + ("pathlib", "Path"), + ("pathlib", "PosixPath"), + ("pathlib", "WindowsPath"), + ("pathlib._local", "PurePath"), + ("pathlib._local", "PurePosixPath"), + ("pathlib._local", "PureWindowsPath"), + ("pathlib._local", "Path"), + ("pathlib._local", "PosixPath"), + ("pathlib._local", "WindowsPath"), + ("numpy", "ndarray"), + ("numpy", "dtype"), + ("agentix.bridge.proxy", "TunnelHandle"), + ("agentix.bash", "BashResult"), + ("agentix.files", "UploadResult"), + ("agentix.agents.claude_code.agent", "ClaudeCodeResult"), + ("agentix.agents.qwen_code", "Result"), + ("agentix.plugins.datasets.swe.env", "PrepareEnvResult"), + } +) # Builtin value types admitted by identity (see `_SAFE_BUILTIN_TYPES`) plus every # builtin exception subclass. Builtins are always imported, so resolving one has @@ -106,37 +122,13 @@ } ) -# First-party namespace, trusted by default. The #116 boundary defends against -# an *untrusted workload's* return value; the framework and its plugins are part -# of the trusted computing base that builds the bundle and runs the sandbox, and -# their return types (`TunnelHandle`, `BashResult`, agent results, …) are inert -# dataclasses/models. Trusting `agentix.*` keeps the framework's own paths -# working; a workload's own return types stay opt-in, and gadget callables -# (subprocess/os/eval/attribute-access helpers) are never first-party so remain -# refused. -_FIRST_PARTY_PREFIXES: tuple[str, ...] = ("agentix",) - -# Module prefixes the caller has explicitly opted to trust for return types the -# default set does not cover. -_ALLOWED_MODULE_PREFIXES: set[str] = set() - class RestrictedUnpickleError(pickle.UnpicklingError): """A global in the stream was not on the host allowlist and was refused.""" -def allow_module(prefix: str) -> None: - """Trust every global whose module equals or starts with `prefix` (dotted). - Use for a package whose return types the default allowlist does not cover.""" - _ALLOWED_MODULE_PREFIXES.add(prefix) - - -def allow_callable(module: str, name: str) -> None: - """Trust one specific `module.name` type/helper by exact identity.""" - SAFE_CALLABLES.add((module, name)) - - def allow_type(cls: type[Any]) -> None: + """Trust one exact runtime class for sandbox return reconstruction.""" if not isinstance(cls, type): raise TypeError("allow_type() requires a class") _ALLOWED_TYPES[(cls.__module__, cls.__qualname__)] = cls @@ -146,34 +138,25 @@ def _trust_enabled() -> bool: return os.environ.get("AGENTIX_PICKLE_TRUST", "").strip().lower() in ("1", "true", "yes") -def _module_allowed(module: str) -> bool: - prefixes = (*_FIRST_PARTY_PREFIXES, *_ALLOWED_MODULE_PREFIXES) - return any(module == p or module.startswith(p + ".") for p in prefixes) - - -def _is_safe_type(module: str, name: str) -> bool: - """`(module, name)` is an allowlisted value type, tolerating a private - implementation submodule of an allowlisted public module — e.g. Python 3.13 - pickles `pathlib.PurePosixPath` as `pathlib._local.PurePosixPath`. Only - underscore-prefixed submodules of the public root are accepted, so a public - sibling module (`pathlib.evil`) is not.""" - if (module, name) in SAFE_TYPES: - return True - head, _, rest = module.partition(".") - if rest and all(part.startswith("_") for part in rest.split(".")): - return (head, name) in SAFE_TYPES - return False - - class RestrictedUnpickler(pickle.Unpickler): """`pickle.Unpickler` whose `find_class` enforces the allowlist above.""" def find_class(self, module: str, name: str) -> Any: - allowed_type = _ALLOWED_TYPES.get((module, name)) + global_name = (module, name) + allowed_type = _ALLOWED_TYPES.get(global_name) if allowed_type is not None: return allowed_type - if (module, name) in SAFE_CALLABLES or _is_safe_type(module, name) or _module_allowed(module): + if global_name in SAFE_TYPES: + obj = super().find_class(module, name) + if not isinstance(obj, type): + raise RestrictedUnpickleError( + f"refusing to reconstruct {module}.{name}: the allowlisted value " + f"global did not resolve to a type" + ) + return obj + + if global_name in _SAFE_CALLABLES: return super().find_class(module, name) if module == "builtins": @@ -191,9 +174,9 @@ def find_class(self, module: str, name: str) -> Any: # arbitrary module would itself run its top-level code on the host). raise RestrictedUnpickleError( f"refusing to reconstruct {module}.{name}: it is not on the host allowlist for " - f"sandbox return values. If this is a return type you trust, call " - f"agentix.runtime.shared.safepickle.allow_module({module!r}) (or allow_callable) " - f"before the call; or set AGENTIX_PICKLE_TRUST=1 to trust the sandbox fully." + f"sandbox return values. If this is a return type you trust, import its class " + f"and call agentix.runtime.shared.safepickle.allow_type(Class) before the call; " + f"or set AGENTIX_PICKLE_TRUST=1 to trust the sandbox fully." ) @@ -209,10 +192,7 @@ def restricted_loads(data: bytes) -> Any: __all__ = [ "RestrictedUnpickleError", "RestrictedUnpickler", - "SAFE_CALLABLES", "SAFE_TYPES", - "allow_callable", - "allow_module", "allow_type", "restricted_loads", ] diff --git a/tests/conftest.py b/tests/conftest.py index ddc247d..485144c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -17,12 +17,13 @@ import pytest from agentix.runtime.shared import safepickle +from tests._worker_target import EchoResult # The host restricts reconstruction of sandbox return values to an allowlist # (#116). Tests return custom types from `tests._worker_target` (e.g. the -# `EchoResult` pydantic model); opt that module in, exactly as a real project -# opts in its own return types. -safepickle.allow_module("tests._worker_target") +# `EchoResult` pydantic model); opt that exact class in, exactly as a real +# project opts in its own return types. +safepickle.allow_type(EchoResult) @pytest.fixture diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index a9bc746..c5e4645 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -44,23 +44,14 @@ def _policy_recorder(value: str) -> str: @pytest.fixture(autouse=True) def _restore_allowlist(): - """`allow_module` / `allow_callable` mutate module-global state — snapshot - and restore it so tests do not leak opt-ins into each other.""" - prefixes = set(safepickle._ALLOWED_MODULE_PREFIXES) - callables = set(safepickle.SAFE_CALLABLES) - allowed_types = getattr(safepickle, "_ALLOWED_TYPES", None) - types = dict(allowed_types) if allowed_types is not None else None + """Restore exact type opt-ins so tests do not leak policy state.""" + allowed_types = dict(safepickle._ALLOWED_TYPES) policy_calls = list(_POLICY_CALLS) try: yield finally: - safepickle._ALLOWED_MODULE_PREFIXES.clear() - safepickle._ALLOWED_MODULE_PREFIXES.update(prefixes) - safepickle.SAFE_CALLABLES.clear() - safepickle.SAFE_CALLABLES.update(callables) - if allowed_types is not None and types is not None: - allowed_types.clear() - allowed_types.update(types) + safepickle._ALLOWED_TYPES.clear() + safepickle._ALLOWED_TYPES.update(allowed_types) _POLICY_CALLS.clear() _POLICY_CALLS.extend(policy_calls) @@ -125,7 +116,6 @@ def test_policy_functions_are_refused(name: str) -> None: def test_one_load_cannot_modify_policy_then_invoke_new_global() -> None: _POLICY_CALLS.clear() - before_callables = set(safepickle.SAFE_CALLABLES) before_types = dict(safepickle._ALLOWED_TYPES) payload = ( b"\x80\x04" @@ -137,7 +127,6 @@ def test_one_load_cannot_modify_policy_then_invoke_new_global() -> None: with pytest.raises(RestrictedUnpickleError): restricted_loads(payload) assert _POLICY_CALLS == [] - assert safepickle.SAFE_CALLABLES == before_callables assert safepickle._ALLOWED_TYPES == before_types From e13d4e20f2373bdef9a327117f361cd3d4c14157 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 13:53:31 +0800 Subject: [PATCH 06/15] test: cover exact first-party pickle values --- tests/runtime/test_safepickle.py | 53 ++++++++++++++++++++++++++------ 1 file changed, 43 insertions(+), 10 deletions(-) diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index c5e4645..be1aa63 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -18,6 +18,7 @@ import pickle import uuid from dataclasses import dataclass +from typing import Any import pytest from pydantic import BaseModel @@ -168,9 +169,21 @@ def test_attribute_access_helpers_are_refused(crafted) -> None: @pytest.mark.parametrize( "value", [ - 42, "hi", 3.5, True, None, b"bytes", bytearray(b"ba"), - complex(1, 2), range(0, 10, 2), slice(1, 9, 2), - [1, 2, 3], {"a": 1}, (1, 2), {1, 2}, frozenset([1]), + 42, + "hi", + 3.5, + True, + None, + b"bytes", + bytearray(b"ba"), + complex(1, 2), + range(0, 10, 2), + slice(1, 9, 2), + [1, 2, 3], + {"a": 1}, + (1, 2), + {1, 2}, + frozenset([1]), {"nested": [1, {"b": (2, 3)}]}, datetime.datetime(2026, 1, 1, 12, 30), datetime.date(2026, 1, 1), @@ -235,14 +248,23 @@ def test_first_party_return_types_round_trip_by_default() -> None: """The framework's own shipped return types are trusted (`agentix.*`) and must cross the boundary with no opt-in — otherwise the restriction breaks the framework's own main paths (Proxy.start, bash.run, agent adapters).""" + from agentix.agents.claude_code.agent import ClaudeCodeResult + from agentix.agents.qwen_code import Result as QwenResult + from agentix.bash import BashResult from agentix.bridge.proxy import TunnelHandle + from agentix.files import UploadResult + from agentix.plugins.datasets.swe.env import PrepareEnvResult - handle = TunnelHandle(url="http://127.0.0.1:9", port=9) - assert restricted_loads(pickle.dumps(handle)) == handle - - bash = pytest.importorskip("agentix.bash") - result = bash.BashResult(stdout="o", stderr="", exit_code=0) - assert restricted_loads(pickle.dumps(result)) == result + values = [ + TunnelHandle(url="http://127.0.0.1:9", port=9), + BashResult(exit_code=0, stdout="o", stderr=""), + UploadResult(path="/workspace/a", size=1), + ClaudeCodeResult(returncode=0, stdout=b"o", stderr=b""), + QwenResult(exit_code=0, stdout="o", stderr=""), + PrepareEnvResult(ok=True, head="abc", log=""), + ] + for value in values: + assert restricted_loads(pickle.dumps(value)) == value def test_custom_type_refused_by_default() -> None: @@ -265,6 +287,17 @@ def test_allow_type_opt_in_is_exact() -> None: restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) +def test_allow_type_requires_a_class() -> None: + not_a_class: Any = object() + with pytest.raises(TypeError, match="requires a class"): + safepickle.allow_type(not_a_class) + + +@pytest.mark.parametrize("name", ["allow_module", "allow_callable"]) +def test_broad_policy_helpers_are_not_exposed(name: str) -> None: + assert not hasattr(safepickle, name) + + def test_trust_env_disables_restriction(monkeypatch) -> None: """The escape hatch fully trusts the sandbox. Verified with a benign reducer (json.dumps) that the restricted path refuses but is harmless.""" @@ -283,4 +316,4 @@ def test_refusal_message_points_to_opt_in() -> None: with pytest.raises(RestrictedUnpickleError) as ei: restricted_loads(pickle.dumps(_ProjectPoint(1, 2))) msg = str(ei.value) - assert "allow_module" in msg or "AGENTIX_PICKLE_TRUST" in msg + assert "allow_type" in msg From b75af6ae0f3ad69a283bd24dfa8c364f3135a16d Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:01:59 +0800 Subject: [PATCH 07/15] test: tighten safepickle compatibility assertions --- tests/runtime/test_safepickle.py | 54 ++++++++++++++++++-------------- 1 file changed, 31 insertions(+), 23 deletions(-) diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index be1aa63..c5c9295 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -21,6 +21,12 @@ from typing import Any import pytest +from agentix.agents.claude_code.agent import ClaudeCodeResult +from agentix.agents.qwen_code import Result as QwenResult +from agentix.bash import BashResult +from agentix.bridge.proxy import TunnelHandle +from agentix.files import UploadResult +from agentix.plugins.datasets.swe.env import PrepareEnvResult from pydantic import BaseModel from agentix.runtime.shared import safepickle @@ -244,27 +250,23 @@ class _ProjectPoint: y: int -def test_first_party_return_types_round_trip_by_default() -> None: - """The framework's own shipped return types are trusted (`agentix.*`) and - must cross the boundary with no opt-in — otherwise the restriction breaks - the framework's own main paths (Proxy.start, bash.run, agent adapters).""" - from agentix.agents.claude_code.agent import ClaudeCodeResult - from agentix.agents.qwen_code import Result as QwenResult - from agentix.bash import BashResult - from agentix.bridge.proxy import TunnelHandle - from agentix.files import UploadResult - from agentix.plugins.datasets.swe.env import PrepareEnvResult - - values = [ - TunnelHandle(url="http://127.0.0.1:9", port=9), - BashResult(exit_code=0, stdout="o", stderr=""), - UploadResult(path="/workspace/a", size=1), - ClaudeCodeResult(returncode=0, stdout=b"o", stderr=b""), - QwenResult(exit_code=0, stdout="o", stderr=""), - PrepareEnvResult(ok=True, head="abc", log=""), - ] - for value in values: - assert restricted_loads(pickle.dumps(value)) == value +@pytest.mark.parametrize( + "value", + [ + pytest.param(TunnelHandle(url="http://127.0.0.1:9", port=9), id="TunnelHandle"), + pytest.param(BashResult(exit_code=0, stdout="o", stderr=""), id="BashResult"), + pytest.param(UploadResult(path="/workspace/a", size=1), id="UploadResult"), + pytest.param( + ClaudeCodeResult(returncode=0, stdout=b"o", stderr=b""), + id="ClaudeCodeResult", + ), + pytest.param(QwenResult(exit_code=0, stdout="o", stderr=""), id="QwenResult"), + pytest.param(PrepareEnvResult(ok=True, head="abc", log=""), id="PrepareEnvResult"), + ], +) +def test_first_party_return_types_round_trip_by_default(value: object) -> None: + """The six individually registered first-party return types need no opt-in.""" + assert restricted_loads(pickle.dumps(value)) == value def test_custom_type_refused_by_default() -> None: @@ -315,5 +317,11 @@ def __reduce__(self): def test_refusal_message_points_to_opt_in() -> None: with pytest.raises(RestrictedUnpickleError) as ei: restricted_loads(pickle.dumps(_ProjectPoint(1, 2))) - msg = str(ei.value) - assert "allow_type" in msg + expected = ( + "refusing to reconstruct tests.runtime.test_safepickle._ProjectPoint: it is not " + "on the host allowlist for sandbox return values. If this is a return type you " + "trust, import its class and call " + "agentix.runtime.shared.safepickle.allow_type(Class) before the call; or set " + "AGENTIX_PICKLE_TRUST=1 to trust the sandbox fully." + ) + assert str(ei.value) == expected From ffd37ff44aab750ca99eeb1a2e855cebe513e243 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:07:25 +0800 Subject: [PATCH 08/15] style: format restricted unpickler --- agentix/runtime/shared/safepickle.py | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/agentix/runtime/shared/safepickle.py b/agentix/runtime/shared/safepickle.py index 3506ac1..33c5eef 100644 --- a/agentix/runtime/shared/safepickle.py +++ b/agentix/runtime/shared/safepickle.py @@ -117,8 +117,20 @@ # "must be an allowed type" check below excludes them. _SAFE_BUILTIN_TYPES: frozenset[type] = frozenset( { - complex, range, slice, bytearray, frozenset, - set, list, dict, tuple, bytes, str, int, float, bool, + complex, + range, + slice, + bytearray, + frozenset, + set, + list, + dict, + tuple, + bytes, + str, + int, + float, + bool, } ) @@ -151,8 +163,7 @@ def find_class(self, module: str, name: str) -> Any: obj = super().find_class(module, name) if not isinstance(obj, type): raise RestrictedUnpickleError( - f"refusing to reconstruct {module}.{name}: the allowlisted value " - f"global did not resolve to a type" + f"refusing to reconstruct {module}.{name}: the allowlisted value global did not resolve to a type" ) return obj From 4479a4a51a94824de23dc20b446158d155e1c3f5 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:07:38 +0800 Subject: [PATCH 09/15] docs: add exact-type unpickling implementation plan --- ...026-07-03-stage-e-exact-type-unpickling.md | 351 ++++++++++++++++++ 1 file changed, 351 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-03-stage-e-exact-type-unpickling.md diff --git a/docs/superpowers/plans/2026-07-03-stage-e-exact-type-unpickling.md b/docs/superpowers/plans/2026-07-03-stage-e-exact-type-unpickling.md new file mode 100644 index 0000000..f544e16 --- /dev/null +++ b/docs/superpowers/plans/2026-07-03-stage-e-exact-type-unpickling.md @@ -0,0 +1,351 @@ +# Stage E Exact-Type Restricted Unpickling Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace module-prefix pickle trust with an exact value-type registry while preserving all shipped unary Agentix return types. + +**Architecture:** Default value classes and inert pickle helpers live in immutable exact-name tables. Host code can opt in one additional class with `allow_type(cls)`, which stores the approved class object under its exact module and qualname; all module-prefix and callable opt-ins are deleted. Tests first demonstrate the missing API and the unsafe prefix behavior using only pure in-memory functions. + +**Tech Stack:** Python 3.11–3.13, stdlib `pickle`, pytest, Pydantic, uv, pyright. + +--- + +## File map + +- Modify `tests/runtime/test_safepickle.py`: test exact class opt-in, reject unregistered first-party functions, and verify one load cannot mutate decoder policy. +- Modify `agentix/runtime/shared/safepickle.py`: immutable exact registries, six first-party value identities, `allow_type`, and exact `find_class` enforcement. +- Modify `tests/conftest.py`: opt in only `tests._worker_target.EchoResult`, not its whole module. +- Leave `agentix/runtime/PROTOCOL.md` for the later sanitized Fable compatibility/documentation pass. + +### Task 1: Define exact class opt-in behavior + +**Files:** +- Modify: `tests/runtime/test_safepickle.py` +- Modify: `agentix/runtime/shared/safepickle.py` + +- [ ] **Step 1: Add a failing exact-opt-in test and registry restoration** + +Update the autouse fixture so it snapshots `safepickle._ALLOWED_TYPES` when that +attribute exists. Replace the module-wide custom-type test with: + +```python +def test_allow_type_opt_in_is_exact() -> None: + point = _ProjectPoint(1, 2) + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(point)) + + allow_type = getattr(safepickle, "allow_type", None) + assert allow_type is not None + allow_type(_ProjectPoint) + assert restricted_loads(pickle.dumps(point)) == point + + with pytest.raises(RestrictedUnpickleError): + restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) +``` + +- [ ] **Step 2: Run the test and verify RED** + +Run: + +```bash +uv run pytest -q tests/runtime/test_safepickle.py::test_allow_type_opt_in_is_exact +``` + +Expected: FAIL at `assert allow_type is not None` because Stage E has no exact +type-registration API. + +- [ ] **Step 3: Implement the minimal exact class registry** + +In `agentix/runtime/shared/safepickle.py`, add: + +```python +GlobalName = tuple[str, str] +_ALLOWED_TYPES: dict[GlobalName, type[Any]] = {} + + +def allow_type(cls: type[Any]) -> None: + if not isinstance(cls, type): + raise TypeError("allow_type() requires a class") + _ALLOWED_TYPES[(cls.__module__, cls.__qualname__)] = cls +``` + +Before other global checks in `find_class()`, return the stored class on an +exact `_ALLOWED_TYPES` match. Add `allow_type` to `__all__`. Do not remove the +legacy prefix branch yet; that is the later GREEN step for security regressions. + +- [ ] **Step 4: Verify GREEN for exact opt-in** + +Run the same pytest node. Expected: PASS, while the same-module Pydantic sibling +remains rejected. + +### Task 2: Prove prefix and policy functions are rejected + +**Files:** +- Modify: `tests/runtime/test_safepickle.py` + +- [ ] **Step 1: Add harmless global-resolution helpers and tests** + +Add a helper that creates a protocol-0 `GLOBAL` lookup without invoking the +resolved object: + +```python +def _global_reference(module: str, name: str) -> bytes: + assert "\n" not in module and "\n" not in name + return f"c{module}\n{name}\n.".encode("ascii") +``` + +Add tests for one ordinary internal function and all Stage E policy functions: + +```python +def test_unregistered_first_party_function_is_refused() -> None: + with pytest.raises(RestrictedUnpickleError): + restricted_loads( + _global_reference( + "agentix.runtime.shared.safepickle", "_trust_enabled" + ) + ) + + +@pytest.mark.parametrize("name", ["allow_module", "allow_callable", "allow_type"]) +def test_decoder_policy_function_is_refused(name: str) -> None: + with pytest.raises(RestrictedUnpickleError): + restricted_loads( + _global_reference("agentix.runtime.shared.safepickle", name) + ) +``` + +`GLOBAL + STOP` only resolves the named object. It never invokes a process, +network operation, file operation, or the function itself. + +- [ ] **Step 2: Add an in-memory one-load policy-mutation regression** + +Use a pure recorder in the test module: + +```python +_POLICY_CALLS: list[str] = [] + + +def _policy_recorder(value: str) -> str: + _POLICY_CALLS.append(value) + return value + + +def test_one_load_cannot_modify_policy_then_invoke_new_global() -> None: + _POLICY_CALLS.clear() + before_types = dict(safepickle._ALLOWED_TYPES) + payload = ( + b"\x80\x04" + b"cagentix.runtime.shared.safepickle\nallow_callable\n" + b"(Vtests.runtime.test_safepickle\nV_policy_recorder\ntR0" + b"ctests.runtime.test_safepickle\n_policy_recorder\n" + b"(Vmarker\ntR." + ) + + with pytest.raises(RestrictedUnpickleError): + restricted_loads(payload) + + assert _POLICY_CALLS == [] + assert safepickle._ALLOWED_TYPES == before_types +``` + +This uses only an in-memory list and string. On the vulnerable implementation, +the first reduction expands the callable table and the second reaches the pure +recorder. + +- [ ] **Step 3: Run all three security tests and verify RED** + +Run: + +```bash +uv run pytest -q \ + tests/runtime/test_safepickle.py::test_unregistered_first_party_function_is_refused \ + tests/runtime/test_safepickle.py::test_decoder_policy_function_is_refused \ + tests/runtime/test_safepickle.py::test_one_load_cannot_modify_policy_then_invoke_new_global +``` + +Expected: failures with `DID NOT RAISE`; the policy-mutation case also records +`marker` on the current prefix-trusting implementation. + +### Task 3: Replace prefix trust with immutable exact registries + +**Files:** +- Modify: `agentix/runtime/shared/safepickle.py` +- Modify: `tests/runtime/test_safepickle.py` +- Modify: `tests/conftest.py` + +- [ ] **Step 1: Make helper and type tables immutable and exact** + +Rename the helper table to internal `_SAFE_CALLABLES` and make both tables +`frozenset[GlobalName]`. Add the six reviewed unary Agentix return identities: + +```python +("agentix.bridge.proxy", "TunnelHandle") +("agentix.bash", "BashResult") +("agentix.files", "UploadResult") +("agentix.agents.claude_code.agent", "ClaudeCodeResult") +("agentix.agents.qwen_code", "Result") +("agentix.plugins.datasets.swe.env", "PrepareEnvResult") +``` + +Keep the existing stdlib/numpy types and add both explicit Python identities for +each pathlib type: + +```python +("pathlib", "PurePosixPath") +("pathlib._local", "PurePosixPath") +``` + +Repeat the pair for `PurePath`, `PureWindowsPath`, `Path`, `PosixPath`, and +`WindowsPath`. Delete `_is_safe_type()` and its fuzzy private-submodule rule. + +- [ ] **Step 2: Delete every broad or executable opt-in surface** + +Delete `_FIRST_PARTY_PREFIXES`, `_ALLOWED_MODULE_PREFIXES`, `_module_allowed()`, +`allow_module()`, and `allow_callable()`. Remove them from `__all__` and update +the module/error prose to direct callers to: + +```python +from agentix.runtime.shared.safepickle import allow_type +allow_type(ProjectResult) +``` + +- [ ] **Step 3: Enforce exact resolution in `find_class()`** + +Use this decision order: + +```python +key = (module, name) +registered = _ALLOWED_TYPES.get(key) +if registered is not None: + return registered +if key in SAFE_TYPES: + obj = super().find_class(module, name) + if not isinstance(obj, type): + raise RestrictedUnpickleError( + f"refusing {module}.{name}: an allowlisted value type resolved " + f"to non-type {type(obj).__name__}" + ) + return obj +if key in _SAFE_CALLABLES: + return super().find_class(module, name) +``` + +Then retain the current identity-based builtin value/exception check. Reject +everything else before importing its module. + +- [ ] **Step 4: Run the Task 2 nodes and verify GREEN** + +Before running, update the test fixture to snapshot only `_ALLOWED_TYPES` and +`_POLICY_CALLS`, because callable/default tables are now immutable and legacy +prefix state no longer exists. Remove the one-load test's `SAFE_CALLABLES` +snapshot/assert; `_ALLOWED_TYPES` is the only mutable policy registry in the +final design and remains asserted unchanged. Also replace the session-level +`allow_module("tests._worker_target")` in `tests/conftest.py` with exact +`allow_type(EchoResult)` registration so pytest can collect after the legacy API +is deleted. + +Expected: all cases raise `RestrictedUnpickleError`, the pure recorder remains +untouched, and `_ALLOWED_TYPES` retains its pre-load state. + +### Task 4: Restore test and first-party compatibility precisely + +**Files:** +- Modify: `tests/conftest.py` +- Modify: `tests/runtime/test_safepickle.py` + +- [ ] **Step 1: Replace the test module opt-in** + +In `tests/conftest.py`, replace `allow_module("tests._worker_target")` with: + +```python +from tests._worker_target import EchoResult + +safepickle.allow_type(EchoResult) +``` + +Update the nearby comment to say that only the exact Pydantic result class is +approved. + +- [ ] **Step 2: Restore registries without legacy prefixes** + +The safepickle test fixture snapshots `_ALLOWED_TYPES` and restores that dict in +`finally`. Remove all references to `_ALLOWED_MODULE_PREFIXES` and mutable +callable tables. + +- [ ] **Step 3: Expand first-party value round trips** + +Construct and round-trip these inert values in +`test_first_party_return_types_round_trip_by_default()`: + +```python +TunnelHandle(url="http://127.0.0.1:9", port=9) +BashResult(exit_code=0, stdout="o", stderr="") +UploadResult(path="/workspace/a", size=1) +ClaudeCodeResult(returncode=0, stdout=b"o", stderr=b"") +QwenResult(exit_code=0, stdout="o", stderr="") +PrepareEnvResult(ok=True, head="abc", log="") +``` + +Use the defining modules for imports when needed, matching their pickle +identities. + +- [ ] **Step 4: Add API-type validation** + +Add: + +```python +def test_allow_type_requires_a_class() -> None: + not_a_class: Any = lambda: None + with pytest.raises(TypeError, match="requires a class"): + safepickle.allow_type(not_a_class) +``` + +Import `Any` from `typing` in the test module; no source-level type suppression +is needed. + +- [ ] **Step 5: Run the complete focused suite** + +Run: + +```bash +uv run pytest -q tests/runtime/test_safepickle.py tests/runtime/test_protocol.py tests/test_public_exports.py +``` + +Expected: all focused tests pass, including HTTP remote round-trip using the +exact `EchoResult` opt-in. + +### Task 5: Type-check and commit the Codex-owned security core + +**Files:** +- Modify: `agentix/runtime/shared/safepickle.py` +- Modify: `tests/conftest.py` +- Modify: `tests/runtime/test_safepickle.py` + +- [ ] **Step 1: Run formatting and type checks for touched code** + +Run: + +```bash +uv run ruff format --check agentix/runtime/shared/safepickle.py tests/conftest.py tests/runtime/test_safepickle.py +uv run ruff check agentix/runtime/shared/safepickle.py tests/conftest.py tests/runtime/test_safepickle.py +uv run pyright agentix/runtime/shared/safepickle.py tests/conftest.py tests/runtime/test_safepickle.py +``` + +Expected: zero formatting, lint, and type errors. Fix root types rather than +adding `type: ignore` comments. + +- [ ] **Step 2: Confirm scope and working tree** + +Run `git diff --check` and inspect `git diff --stat`. Only the three Codex-owned +code/test files plus this plan should be uncommitted. + +- [ ] **Step 3: Create a local implementation commit** + +```bash +git add agentix/runtime/shared/safepickle.py tests/conftest.py \ + tests/runtime/test_safepickle.py \ + docs/superpowers/plans/2026-07-03-stage-e-exact-type-unpickling.md +git commit -m "security: restrict pickle globals to exact value types" +``` + +Do not push, open a pull request, or modify unrelated Stage E/master files. From 5f088cea7ccc4dc715943093567dce85b9600c46 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:33:00 +0800 Subject: [PATCH 10/15] test: cover warm pickle extension cache bypass --- tests/runtime/test_safepickle.py | 41 ++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index c5c9295..552d96d 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -11,6 +11,7 @@ from __future__ import annotations import collections +import copyreg import datetime import decimal import fractions @@ -169,6 +170,46 @@ def test_attribute_access_helpers_are_refused(crafted) -> None: restricted_loads(crafted) +@pytest.mark.parametrize( + ("encoded_extension", "extension_code"), + [ + pytest.param(b"\x82\xff", 0xFF, id="EXT1"), + pytest.param(b"\x83\xff\xff", 0xFFFF, id="EXT2"), + pytest.param(b"\x84\xff\xff\xff\x7f", 0x7FFFFFFF, id="EXT4"), + ], +) +def test_warm_extension_cache_cannot_bypass_exact_global_policy(encoded_extension: bytes, extension_code: int) -> None: + module = _policy_recorder.__module__ + name = _policy_recorder.__qualname__ + extension_cache: dict[int, object] = getattr(copyreg, "_extension_cache") + cache_missing = object() + previous_cached = extension_cache.pop(extension_code, cache_missing) + registered = False + try: + copyreg.add_extension(module, name, extension_code) + registered = True + payload = b"\x80\x04" + encoded_extension + b"\x8c\x16extension-cache-marker\x85R." + + _POLICY_CALLS.clear() + assert pickle.loads(payload) == "extension-cache-marker" + assert _POLICY_CALLS == ["extension-cache-marker"] + assert extension_cache[extension_code] is _policy_recorder + + _POLICY_CALLS.clear() + with pytest.raises( + RestrictedUnpickleError, + match="extension opcodes cannot be safely validated against the exact allowlist", + ): + restricted_loads(payload) + assert _POLICY_CALLS == [] + finally: + if registered: + copyreg.remove_extension(module, name, extension_code) + extension_cache.pop(extension_code, None) + if previous_cached is not cache_missing: + extension_cache[extension_code] = previous_cached + + # ── permitted values round-trip ───────────────────────────────────────────── From bd946032d0b49c8af3cf4b5a8bb270a947b1629e Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:35:09 +0800 Subject: [PATCH 11/15] security: reject pickle extension opcodes --- agentix/runtime/shared/safepickle.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/agentix/runtime/shared/safepickle.py b/agentix/runtime/shared/safepickle.py index 33c5eef..d8e9064 100644 --- a/agentix/runtime/shared/safepickle.py +++ b/agentix/runtime/shared/safepickle.py @@ -54,6 +54,7 @@ import io import os import pickle +import pickletools from typing import Any GlobalName = tuple[str, str] @@ -197,6 +198,15 @@ def restricted_loads(data: bytes) -> Any: Honors `AGENTIX_PICKLE_TRUST=1` as a full-trust bypass (plain `pickle.loads`).""" if _trust_enabled(): return pickle.loads(data) + try: + for opcode, _, _ in pickletools.genops(data): + if opcode.name in {"EXT1", "EXT2", "EXT4"}: + raise RestrictedUnpickleError( + f"refusing pickle {opcode.name}: extension opcodes cannot be safely " + f"validated against the exact allowlist" + ) + except ValueError as exc: + raise RestrictedUnpickleError(f"refusing malformed pickle opcode stream: {exc}") from exc return RestrictedUnpickler(io.BytesIO(data)).load() From f36d2a000c4b0cbbb1e779810901711d52b469b0 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 14:56:13 +0800 Subject: [PATCH 12/15] test: make restricted decode regressions harmless --- tests/_worker_target.py | 16 ----- tests/runtime/test_protocol.py | 100 ++++++++++++++++++++++++++----- tests/runtime/test_safepickle.py | 92 +++++++++++++++------------- 3 files changed, 136 insertions(+), 72 deletions(-) diff --git a/tests/_worker_target.py b/tests/_worker_target.py index 7bf7daa..43a4ecb 100644 --- a/tests/_worker_target.py +++ b/tests/_worker_target.py @@ -90,19 +90,3 @@ def spawn_stdin_reading_child() -> int: timeout=10, ) return proc.returncode - - -class _ReducesToSubprocess: - """A return object whose `__reduce__` directs reconstruction at - `subprocess.check_output` — a callable the host must not invoke while - decoding a sandbox return value (#116). The argument is benign; the host's - restricted unpickler refuses it before anything runs.""" - - def __reduce__(self): - import subprocess - - return (subprocess.check_output, (["true"],)) - - -def return_unsafe_reducer() -> object: - return _ReducesToSubprocess() diff --git a/tests/runtime/test_protocol.py b/tests/runtime/test_protocol.py index eac47c7..11f9043 100644 --- a/tests/runtime/test_protocol.py +++ b/tests/runtime/test_protocol.py @@ -9,6 +9,8 @@ import asyncio import functools +import pickle +from types import SimpleNamespace import httpx import pytest @@ -17,12 +19,31 @@ from agentix import Failed, Ok, RemoteCallError, RuntimeClient from agentix.runtime.shared.codec import pack, unpack from agentix.runtime.shared.models import RemoteRequest +from agentix.runtime.shared.safepickle import RestrictedUnpickleError from tests import _worker_target as target from tests._rpc_helpers import request_for pytestmark = pytest.mark.asyncio RPC_NAMESPACE = "/rpc" +_HOST_DECODE_CALLS: list[str] = [] + + +def _host_decode_recorder() -> None: + _HOST_DECODE_CALLS.append("called") + + +def _resolution_only_host_value() -> bytes: + module = _host_decode_recorder.__module__ + name = _host_decode_recorder.__qualname__ + return f"c{module}\n{name}\n.".encode("ascii") + + +def _assert_resolution_only_value_is_inert(raw_value: bytes) -> None: + _HOST_DECODE_CALLS.clear() + assert pickle.loads(raw_value) is _host_decode_recorder + assert _HOST_DECODE_CALLS == [] + # ── basics ───────────────────────────────────────────────────────────── @@ -158,6 +179,7 @@ async def _on_result(data): assert payload["call_id"] == call_id import pickle as _pickle + assert _pickle.loads(payload["value"]) == 1, "fn must have run exactly once" @@ -211,6 +233,7 @@ async def _on_result(data): assert payload["call_id"] == call_id import pickle as _pickle + assert _pickle.loads(payload["value"]) == 1, "fn must run exactly once" @@ -297,8 +320,7 @@ async def test_remote_accepts_script_main_function( ): script = tmp_path / "runner_like.py" script.write_text( - "async def get_patch(workdir):\n" - " return f'patch from {workdir}'\n", + "async def get_patch(workdir):\n return f'patch from {workdir}'\n", ) monkeypatch.syspath_prepend(str(tmp_path)) @@ -308,8 +330,7 @@ async def test_remote_accepts_script_main_function( monkeypatch.setattr(main_module, "__spec__", None, raising=False) namespace = {"__name__": "__main__"} exec( - "async def get_patch(workdir):\n" - " return f'patch from {workdir}'\n", + "async def get_patch(workdir):\n return f'patch from {workdir}'\n", namespace, ) @@ -528,21 +549,70 @@ async def test_malformed_call_error_is_typed_not_keyerror(use_inprocess_worker, assert ei.value.error.type == "MalformedError" -async def test_unsafe_return_value_refused_host_side(use_inprocess_worker, live_server): - """A sandbox return value whose `__reduce__` names a non-allowlisted - callable is refused when the host decodes it (#116).""" - from agentix.runtime.shared.safepickle import RestrictedUnpickleError +async def test_http_fast_path_refuses_nonallowlisted_return_without_network(): + raw_value = _resolution_only_host_value() + _assert_resolution_only_value_is_inert(raw_value) - use_inprocess_worker() - base_url = await live_server() - async with RuntimeClient(base_url) as c: + request_payload = request_for(target.add, args=[1, 2], call_id="unsafe-http").model_dump() + + def reply(request: httpx.Request) -> httpx.Response: + assert request.url.path == "/call" + assert unpack(request.content)["call_id"] == "unsafe-http" + return httpx.Response( + 200, + content=pack({"ok": True, "value": raw_value}), + headers={"content-type": "application/msgpack"}, + ) + + c = RuntimeClient("http://runtime.test") + await c._client.aclose() + c._client = httpx.AsyncClient( + transport=httpx.MockTransport(reply), + base_url="http://runtime.test", + ) + try: with pytest.raises(RestrictedUnpickleError): - await c.remote(target.return_unsafe_reducer) + await c._try_http_fast_path( + sio=SimpleNamespace(sid="in-memory-sio"), + payload=request_payload, + ) + finally: + await c.close() + assert _HOST_DECODE_CALLS == [] -async def test_pydantic_return_value_round_trips_through_restricted_loads( - use_inprocess_worker, live_server -): +async def test_socketio_result_queue_refuses_nonallowlisted_return_without_network(monkeypatch): + raw_value = _resolution_only_host_value() + _assert_resolution_only_value_is_inert(raw_value) + + c = RuntimeClient("http://runtime.test", http_sync_ms=None) + + async def emit(event: str, data: bytes, *, namespace: str) -> None: + assert namespace == RPC_NAMESPACE + if event == "cancel": + return + assert event == "call" + payload = unpack(data) + call_id = payload["call_id"] + await c._pending[call_id].put( + ("result", {"call_id": call_id, "value": raw_value}), + ) + + fake_sio = SimpleNamespace(emit=emit) + + async def ensure_sio(): + return fake_sio + + monkeypatch.setattr(c, "_ensure_sio", ensure_sio) + try: + with pytest.raises(RestrictedUnpickleError): + await c.remote(target.add, 1, 2) + finally: + await c.close() + assert _HOST_DECODE_CALLS == [] + + +async def test_pydantic_return_value_round_trips_through_restricted_loads(use_inprocess_worker, live_server): """The restricted host boundary must not break the common case: a custom pydantic model returned from the sandbox still decodes.""" use_inprocess_worker() diff --git a/tests/runtime/test_safepickle.py b/tests/runtime/test_safepickle.py index 552d96d..f903db4 100644 --- a/tests/runtime/test_safepickle.py +++ b/tests/runtime/test_safepickle.py @@ -17,6 +17,7 @@ import fractions import pathlib import pickle +import sys import uuid from dataclasses import dataclass from typing import Any @@ -64,49 +65,23 @@ def _restore_allowlist(): _POLICY_CALLS.extend(policy_calls) -# ── objects that direct reconstruction at non-allowlisted callables ────────── -# Each `__reduce__` names a callable that a restricted host decode must refuse. -# The referenced callables are the ones named in issue #116; the arguments are -# benign so nothing runs even if a regression let one through. - - -class _ReducesToSubprocessCheckOutput: - def __reduce__(self): - import subprocess - - return (subprocess.check_output, (["true"],)) - - -class _ReducesToSubprocessPopen: - def __reduce__(self): - import subprocess - - return (subprocess.Popen, (["true"],)) - - -class _ReducesToEval: - def __reduce__(self): - return (eval, ("1 + 1",)) - - -class _ReducesToOsSystem: - def __reduce__(self): - import os - - return (os.system, ("true",)) - - @pytest.mark.parametrize( - "obj_cls", + ("module", "name"), [ - _ReducesToSubprocessCheckOutput, - _ReducesToSubprocessPopen, - _ReducesToEval, - _ReducesToOsSystem, + pytest.param("subprocess", "check_output", id="subprocess-check-output"), + pytest.param("subprocess", "Popen", id="subprocess-popen"), + pytest.param("builtins", "eval", id="builtins-eval"), + pytest.param("os", "system", id="os-system"), ], ) -def test_non_allowlisted_callable_is_refused(obj_cls) -> None: - blob = pickle.dumps(obj_cls()) +def test_non_allowlisted_callable_is_refused_without_invocation(module: str, name: str) -> None: + blob = _global_reference(module, name) + + # GLOBAL + STOP resolves the object but never invokes it. Keep the + # regression harmless even if the restricted decoder accidentally admits + # the same reference in the future. + assert callable(pickle.loads(blob)) + with pytest.raises(RestrictedUnpickleError): restricted_loads(blob) @@ -267,10 +242,19 @@ def test_numpy_array_round_trips() -> None: def test_numpy_object_array_gates_nested_globals() -> None: """An object-dtype array pickles its elements in the same stream, so a non-allowlisted callable referenced by an element is still refused.""" + + class _ReducesToPolicyRecorder: + def __reduce__(self): + return (_policy_recorder, ("numpy-object-array-marker",)) + np = pytest.importorskip("numpy") - arr = np.array([_ReducesToOsSystem()], dtype=object) + arr = np.array([_ReducesToPolicyRecorder()], dtype=object) + blob = pickle.dumps(arr) + + policy_calls_before = list(_POLICY_CALLS) with pytest.raises(RestrictedUnpickleError): - restricted_loads(pickle.dumps(arr)) + restricted_loads(blob) + assert _POLICY_CALLS == policy_calls_before def test_none_returns_none() -> None: @@ -330,6 +314,32 @@ def test_allow_type_opt_in_is_exact() -> None: restricted_loads(pickle.dumps(_ProjectResult(patch="d", score=1))) +def test_safe_type_identity_must_resolve_to_a_type(monkeypatch) -> None: + policy_calls_before = list(_POLICY_CALLS) + monkeypatch.setattr(datetime, "date", _policy_recorder) + + with pytest.raises( + RestrictedUnpickleError, + match="allowlisted value global did not resolve to a type", + ): + restricted_loads(_global_reference("datetime", "date")) + assert _POLICY_CALLS == policy_calls_before + + +def test_allow_type_uses_registered_class_after_module_rebind(monkeypatch) -> None: + registered_class = _ProjectPoint + value = registered_class(1, 2) + safepickle.allow_type(registered_class) + blob = pickle.dumps(value) + + module = sys.modules[registered_class.__module__] + monkeypatch.setattr(module, registered_class.__qualname__, object) + + out = restricted_loads(blob) + assert type(out) is registered_class + assert out == value + + def test_allow_type_requires_a_class() -> None: not_a_class: Any = object() with pytest.raises(TypeError, match="requires a class"): From 442612165ac59265d019036ef1545a18ab823886 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 15:21:40 +0800 Subject: [PATCH 13/15] docs: list the six exact default return types in PROTOCOL The unpickling paragraph still described the pre-exact policy: blanket agentix.* prefix trust and the deleted allow_module()/allow_callable() surfaces. Name the six individually registered first-party return types and point extension at allow_type(Class), matching safepickle. Co-Authored-By: Claude Fable 5 --- agentix/runtime/PROTOCOL.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/agentix/runtime/PROTOCOL.md b/agentix/runtime/PROTOCOL.md index 234ee2d..4a53e60 100644 --- a/agentix/runtime/PROTOCOL.md +++ b/agentix/runtime/PROTOCOL.md @@ -31,12 +31,15 @@ plain unpickling reconstructs objects by invoking whatever callables the stream names. The restricted loader decodes through a strict **allowlist** — only a reviewed set of value types (stdlib data, builtin containers and exceptions, numpy arrays, …) and inert reconstruction helpers are permitted, and anything -else is refused *without importing it*. First-party types (`agentix.*`) are -trusted by default — the framework's own return types (`TunnelHandle`, -`BashResult`, agent results, …) cross the boundary with no setup. A *workload's* -own return types (a project's dataclasses / pydantic models) are refused by -default; opt them in with `safepickle.allow_module(prefix)` / -`allow_callable(module, name)`, or set `AGENTIX_PICKLE_TRUST=1` to trust the +else is refused *without importing it*. Six first-party return types are on +the default allowlist, each registered individually by exact module and class +name — `TunnelHandle`, `BashResult`, `UploadResult`, `ClaudeCodeResult`, the +qwen_code `Result`, and `PrepareEnvResult` — so the framework's own return +values cross the boundary with no setup. Every other global — including the +rest of `agentix.*` and a *workload's* own return types (a project's +dataclasses / pydantic models) — is refused by default; opt a class in with +`safepickle.allow_type(Class)`, which trusts exactly one class (there is no +module- or prefix-level trust), or set `AGENTIX_PICKLE_TRUST=1` to trust the sandbox fully. A refusal raises `agentix.RestrictedUnpickleError`. (The sandbox-side decode of host-sent `arguments` / `context` stays plain — that is the trusted host→sandbox From ebfbc983cc0371182122a58f2387dac457065401 Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 15:21:45 +0800 Subject: [PATCH 14/15] style: format public export list Co-Authored-By: Claude Fable 5 --- tests/test_public_exports.py | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/tests/test_public_exports.py b/tests/test_public_exports.py index efc711d..b0aa40b 100644 --- a/tests/test_public_exports.py +++ b/tests/test_public_exports.py @@ -37,8 +37,15 @@ def test_failure_vocabulary_importable_from_agentix() -> None: def test_failure_vocabulary_in_dunder_all() -> None: for name in ( - "CallCancelled", "CallTimeout", "Failed", "Ok", "Result", - "RestrictedUnpickleError", "RuntimeUnreachable", "WorkerExited", "configure_logging", + "CallCancelled", + "CallTimeout", + "Failed", + "Ok", + "Result", + "RestrictedUnpickleError", + "RuntimeUnreachable", + "WorkerExited", + "configure_logging", ): assert name in agentix.__all__ From f588f98e145d47533e90c9ca33095348c2ae239c Mon Sep 17 00:00:00 2001 From: Lingrui Mei Date: Fri, 3 Jul 2026 15:21:52 +0800 Subject: [PATCH 15/15] chore: keep agentix-ray-build skill out of the stage E diff The Ray-cluster build notes are unrelated to the restricted unpickler; untrack them so the stage E diff stays on-topic. The file stays on disk as an untracked local note. Co-Authored-By: Claude Fable 5 --- .agents/skills/agentix-ray-build/SKILL.md | 88 ----------------------- 1 file changed, 88 deletions(-) delete mode 100644 .agents/skills/agentix-ray-build/SKILL.md diff --git a/.agents/skills/agentix-ray-build/SKILL.md b/.agents/skills/agentix-ray-build/SKILL.md deleted file mode 100644 index 4d48b8a..0000000 --- a/.agents/skills/agentix-ray-build/SKILL.md +++ /dev/null @@ -1,88 +0,0 @@ ---- -name: agentix-ray-build -description: Build and run Agentix bundles on a restricted remote Ray cluster (rootless podman, no-Docker pod, egress only via a corporate HTTP proxy). Covers the working podman build recipe, sandbox run-args, and how to read job logs when `ray job logs` is blocked. Use when building/testing an Agentix bundle on a Ray box that lacks Docker. ---- - -# Building & running Agentix bundles on a restricted Ray cluster - -Context: a remote Ray pod with **no Docker**, only **rootless podman** (4.x, crun), -**read-only cgroups**, and **egress only through a corporate HTTP proxy**. Jobs are -**ephemeral** (a `uv sync` in one job doesn't persist) and **logs are gateway-blocked**, -so build + run must happen in **one self-contained job** and progress is read via the -dashboard state API. Replace every `` with your environment's value. - -## Submitting (one job, ephemeral) - -Stage a clean repo and submit `python my_script.py` with `--working-dir` = repo root: - -``` -git archive HEAD | tar -x -C # clean tree, no .venv/.git -# write /my_script.py (the driver below), then submit via the cluster's -# ray job submit (e.g. `ray job submit --address http://:8081 \ -# --working-dir --no-wait -- python my_script.py`) -``` - -The driver must **stream** subprocess output (NOT `capture_output`) so it lands in the -job's `driver.log`, and end with a sentinel like `::RESULT rc=` + `sys.exit(rc)`. - -## The podman build recipe (what actually works) - -1. **Stage as a git repo.** `agentix build` copies the *whole git repo* so a project's - `../../plugins/*` path deps resolve. A non-git tree becomes a standalone context and - those deps break (`Distribution not found at file:///plugins/...`). If you shipped via - `git archive`, run `git init -q && git add -A` in the working dir first. -2. **Build RUN steps** must skip the read-only-cgroup + netns setup: - `--container-arg --isolation=chroot --container-arg --network=host`. -3. **Nix egress.** Public `cache.nixos.org` stalls/throttles through a corp proxy. Put a - fast mirror PRIMARY and keep `cache.nixos.org` as a coverage fallback — two separate - options (`substituters` replaces the default, `extra-substituters` appends): - `--nix-arg "--option substituters https://mirrors.ustc.edu.cn/nix-channels/store"` - `--nix-arg "--option extra-substituters https://cache.nixos.org"` - (TUNA `https://mirrors.tuna.tsinghua.edu.cn/nix-channels/store` works as primary too. - The CN mirrors mirror channel snapshots and may miss a path — hence the cache fallback.) -4. **Export step.** The bundle is extracted with `podman create --network none` + copy — - it never starts, so it needs NO cgroup/netns workaround. Do **NOT** pass - `--container-run-arg --network=host` here (clashes with the default `--network none` → - "cannot set multiple networks"). -5. **Trim host-only deps** from the bundle project — provider backends (`agentix-provider-*`) - are host-side and don't belong in the sandbox bundle (their `default.nix` can also drag - heavy system binaries into the closure). - -``` -git init -q && git add -A -HTTP_PROXY= HTTPS_PROXY= NO_PROXY=127.0.0.1,localhost \ -agentix build --container-engine podman --platform linux/amd64 \ - --container-arg --isolation=chroot --container-arg --network=host \ - --nix-arg "--option substituters https://mirrors.ustc.edu.cn/nix-channels/store" \ - --nix-arg "--option extra-substituters https://cache.nixos.org" -``` - -## Running the sandbox - -The sandbox container *does* start, so it **does** need the cgroup/netns workaround as -**run-args**: `--runtime=crun --cgroups=disabled --network=host`. With `--network=host` -the runtime server binds a host port (reach it at `127.0.0.1:`, no mapping). Wire -them via `agentix deploy podman --run-arg=--runtime=crun --run-arg=--cgroups=disabled ---run-arg=--network=host`, or the provider's run-arg config when orchestrating in-process. -(Use the `--run-arg=VALUE` form — values that start with `--` break argparse otherwise.) - -> Trade-offs of these run-args: `--network=host` removes network isolation; `--cgroups=disabled` -> removes resource limits. Fine for trusted single-tenant eval/RL; not for multi-tenant. - -## Reading logs when `ray job logs` is blocked - -The dashboard proxies `ray job logs` (and `/api/jobs//logs`) to a per-node job-agent on -an ephemeral internal port that isn't reachable through `:8081` (device-auth gateway) → -`ConnectionRefusedError`. Use the status endpoint + state API on `:8081` instead: - -``` -H=http://:8081 ; JID=raysubmit_... -curl -s "$H/api/jobs/$JID" | python3 -c "import sys,json;d=json.load(sys.stdin);print(d['status'],'|',(d.get('message') or '')[:200])" -NID=$(curl -s "$H/api/v0/nodes" | python3 -c "import sys,json;print(json.load(sys.stdin)['data']['result']['result'][0]['node_id'])") -curl -s "$H/api/v0/logs?node_id=$NID&glob=*$JID*" # confirm job-driver-$JID.log exists -curl -s "$H/api/v0/logs/file?node_id=$NID&filename=job-driver-$JID.log&lines=200" | tr -d '\000' | tail -60 -``` - -Poll status until `SUCCEEDED`/`FAILED`/`STOPPED`. Device-auth challenges are intermittent -(transient `401`); just retry. If logs are *fully* unreachable, have the driver -`raise RuntimeError(tail)` so the tail surfaces in the job `message`.