Sandbox template rendering; cover the non-raising failure (#104) - #447
Conversation
Red-teaming the supported examples turned up a template injection, and
mutation-testing the acceptance suite turned up a hole it could not see.
**Parameter values were executed with Python's object graph in reach.**
A pipeline's `{{ }}` expressions are authored with the pipeline; the values
substituted into them are not, arriving from `-i name=value`, an inputs file,
or an upstream step. Those values are rendered too -- deliberately, since
`-i out_dir='{{ base }}/reports'` is useful -- but Jinja's stock `Environment`
renders them unrestricted:
orchestrator run examples/supported/01_hello_filesystem.yaml \
-i greeting='{{ "".__class__.__mro__[1].__subclasses__() | length }}'
-> output/greeting.txt == "1183, world", exit 0
That is the first hop of the standard Jinja sandbox escape, reachable from a
plain CLI argument. Every environment on the execution path is now built by
`core/template_sandbox.py` and is a `SandboxedEnvironment`. Three call sites
were building a bare `Template(...)`, bypassing every environment including
the declared-outputs renderer in `orchestrator.py`; they go through the same
factory now. Ordinary expressions, filters and `{{ step.result.field }}` are
unaffected -- the blocking suite is unchanged at 562 passed.
`TemplateManager.render` swallowed the resulting `SecurityError` and returned
the payload unrendered, leaving the refusal visible only in a log line. A
sandbox violation now propagates; every other render failure keeps its
existing fallback.
**A step can fail without raising, and nothing tested it.**
Deleting the `reported_failure` branch of `StepResult.from_task` -- the branch
that catches a tool returning `{"success": false}` while its task ends
COMPLETED -- left the entire acceptance suite green. 06 fails by raising, so
it never reaches that code. `05_reported_failure.yaml` is the sibling that
does: the run exits 1 and reports success=false while no task is FAILED.
`Case` could not express it either. It assumed status "completed" implied
success, the same conflation the runtime had; it now carries
`reported_failure` separately and asserts both fields together.
Each new assertion was mutation-tested against un-fixed code: un-sandboxing
the factory fails 7, removing the SecurityError re-raise fails 6, and dropping
the `reported_failure` branch now fails 2 where it previously survived.
Examples validating: 8/116 -> 9/117. Blocking suite: 548 -> 562 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI verification9/9 green. Legacy tally against the current post-merge
The +14 deselected is exactly the 14 tests this PR adds to the blocking layer The one number that moved unexplainedWarnings went 11 -> 12. Rather than wave that past as noise, I compared the Identical. The extra warning is another instance of a class already present, Worth noting that |
Red-teaming the supported examples with subagents found a template injection.
Mutation-testing the acceptance suite found a hole it could not have seen.
Template injection, reachable from a CLI argument
A pipeline's
{{ }}expressions are authored with the pipeline. The valuessubstituted into them are not — they arrive from
-i name=value, an inputsfile, or an upstream step's output. Those values are rendered too, which is
deliberate (
-i out_dir='{{ base }}/reports'is useful). Jinja's stockEnvironmentrendered them with Python's object graph in reach:1183is the length of__subclasses__()— the first hop of the standardJinja sandbox escape, from a plain command-line argument.
Every environment on the execution path is now built by
core/template_sandbox.pyand is aSandboxedEnvironment. Three call sitesbuilt a bare
Template(...)and bypassed every environment — including thedeclared-outputs renderer in
orchestrator.py— and now use the samefactory.
TemplateManager.renderthen swallowed theSecurityErrorand returned thepayload unrendered, leaving the refusal visible only in a log line. A sandbox
violation now propagates; every other render failure keeps its existing
fallback.
Ordinary expressions, filters and
{{ step.result.field }}are unaffected —the blocking suite is unchanged at 562 passed, and the example catalog still
compiles everything it compiled before.
A step can fail without raising, and nothing tested it
Deleting the
reported_failurebranch ofStepResult.from_task— the branchcatching a tool that returns
{"success": false}while its task endsCOMPLETED — left the entire acceptance suite green. 06 fails by raising,
so it never reaches that code.
05_reported_failure.yamlis the sibling that does. The run exits 1 andreports
success=falsewhile no task is FAILED:Casecould not express that either: it assumed statuscompletedimpliedsuccess — the same conflation the runtime had. It now carries
reported_failureseparately and asserts status and success together.Evidence
Every new assertion was mutation-tested against un-fixed code:
SecurityErrorre-raise (sandbox still on)reported_failurebranchruff check src/orchestrator(CI rule set) clean;compileallcleanTwo red-team claims I could not confirm, and did not act on
{{ }}reaching a file via the render fallback: real in code,but refuted end to end.
{{ 1/0 }}exits 1 with no artifact — theunresolved-template guard from Stop unresolved templates before they reach a tool (#153) #439 catches it first.
between
>and<mid-run. That was my own doing: a mutation-testing agentwas concurrently applying and reverting exactly that edit to that file.
Contamination from parallel red-teaming, not a defect.
Path traversal via
-i out_dir=../escapeis real but I judge it by design:this is a local CLI and the operator supplies the path. Flagging, not fixing.