Repository navigation
test(sandbox): phase 5 plan matrix, fail-closed untrusted tiers - #15
Conversation
…lation, host-network approval and credential floor Part of tinyhumansai/openhuman#7328 (phase 5). Documents that Firejail and Bubblewrap map to the namespace backend. Co-authored-by: Medulla <medulla@tinyhumans.ai>
|
Warning Review limit reached
This review includes 1 billable file and costs up to $0.25. Or wait 11 minutes for your next included review. View limit details
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
| Check name | Status | Explanation |
|---|---|---|
| Description Check |  |
Check skipped - CodeRabbit’s high-level summary is enabled. |
| Title check | |
The title clearly identifies the main change: adding the sandbox phase 5 plan matrix tests and validating fail-closed behavior for untrusted tiers. It is concise and specific. |
| Docstring Coverage | |
Docstring coverage is 80.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 3 files. |
| Linked Issues check | |
Check skipped because no linked issues were found for this pull request. |
| Out of Scope Changes check | |
Check skipped because no linked issues were found for this pull request. |
✨ Finishing Touches
-
📝 Generate docstrings
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
A rabbit checks each sandbox plan,
Through trust and settings, line by line.
Kernel levels pass the test,
Denials keep the limits set.
Then hops away, content with the design.
Comment @coderabbitai help to get the list of available commands.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b81dc1a7a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| #[cfg(test)] | ||
| #[path = "plan_matrix_tests.rs"] | ||
| mod plan_matrix_tests; |
There was a problem hiding this comment.
The required cargo fmt --all -- --check command exits with status 1 here because rustfmt expects plan_matrix_tests before tests. Since CI runs this exact formatting check, the commit cannot pass validation until the declarations are formatted.
AGENTS.md reference: AGENTS.md:L25-L35
Useful? React with 👍 / 👎.
| //! Exhaustive plan matrix: trust x deployment x switches x backend x isolation. | ||
| use super::*; |
There was a problem hiding this comment.
This new sibling test file begins with a module doc comment rather than use super::*;, contrary to the repository's explicit layout requirement for every *_tests.rs file. Move the import to the first line so the test module follows the mandated structure.
AGENTS.md reference: AGENTS.md:L117-L125
Useful? React with 👍 / 👎.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Tiny Sweeper reviewTiny Sweeper reviewed this change across 5 lane(s) and found 0 active actionable finding(s). Detailed lane evidence and any incomplete work are listed below. State: Ready for maintainer review Review snapshot
Completeness: Complete FeaturesNone identified with supported citations. TestsNo supported feature-to-test mapping was produced. Test execution is not inferred. FindingsNo active actionable findings. Before mergeNone. How this fits togetherflowchart LR
n0["supports<br/>changed"]:::changed
n1["resolve"]:::impacted
n2["plan"]:::impacted
n3["SandboxArtifact2"]:::impacted
n4["validate"]:::impacted
n5["SandboxPlanRequest2"]:::impacted
n6["select"]:::impacted
n1 -->|uses| n3
n1 -->|uses| n5
n1 -->|calls| n6
n2 -->|uses| n5
n4 -->|calls| n0
n4 -->|calls| n1
n4 -->|uses| n3
n4 -->|uses| n5
n6 -->|calls| n0
n6 -->|uses| n5
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Agent review detailscritique
security
tests
commits
description
Evidence and run details
|
Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 50619267df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assert_eq!( | ||
| response.denial, | ||
| Some(SandboxDenial2::Unsupported), | ||
| "{label}" | ||
| ); |
There was a problem hiding this comment.
For isolation-required combinations using Noop, None, or Process, this branch checks only the absent artifact and denial metadata, not response.decision.verdict. A regression in denied() that emits Verdict::Allow while retaining those fields would therefore leave the new fail-closed matrix green even though the public authorization decision permits the operation; assert the corresponding Verdict::Deny and reason here.
AGENTS.md reference: AGENTS.md:L18-L20
Useful? React with 👍 / 👎.
| /// Path-jail executor. It is planned only when it reports enforced kernel | ||
| /// isolation, so an untrusted tier never receives a best-effort jail. |
There was a problem hiding this comment.
This public documentation promises that every Jail plan reports kernel isolation, but Engine::resolve requests kernel isolation only when isolation is mandatory or enabled, and the new matrix explicitly expects trusted-local/all-disabled Jail executors reporting None or Process to be allowed. Either qualify this guarantee as applying only to isolation-required plans or enforce it for every Jail selection so the contract matches planner behavior.
AGENTS.md reference: AGENTS.md:L162-L163
Useful? React with 👍 / 👎.
…d validate Gives the planner the issue's is_suitable_for_untrusted_code check in one place so Noop and non-kernel executors can never back an untrusted or SaaS plan. Co-authored-by: Medulla <medulla@tinyhumans.ai>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a2986edf8d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| } else { | ||
| SandboxDeployment2::Local | ||
| }; | ||
| scope.origin = if saas { Origin::Saas } else { Origin::Operator }; |
There was a problem hiding this comment.
This assigns origin from the same saas flag as deployment, so the matrix never covers deployment == Local with origin == SaaS (or the reverse), even though Engine::resolve treats them as independent mandatory-isolation predicates. If the facts.origin == Origin::Saas check were removed, a trusted local-deployment scope with SaaS origin, disabled switches, and process-only isolation could be allowed while this purportedly exhaustive fail-closed suite still passed; add origin as an independent matrix dimension.
Useful? React with 👍 / 👎.
![tinysweeper[bot]](https://avatars.githubusercontent.com/in/4526585?s=60&v=4){kind=small}
There was a problem hiding this comment.
tinysweeper found nothing blocking. Approving.
$0.0018 · 314,435 in / 23,685 out · 27,776 cached (9%) · ladder/vectors-oai3, deepseek-v4.1-flash · 389 embedded
critique: $0.0002 · 150,836 in / 1,875 out · 12,288 cached (8%) · deepseek-v4.1-flash
security: $0.0005 · 71,450 in / 6,151 out · 5,376 cached (8%) · deepseek-v4.1-flash
tests: $0.0010 · 56,581 in / 13,032 out · 4,352 cached (8%) · deepseek-v4.1-flash
description: $0.0001 · 30,586 in / 922 out · 5,760 cached (19%) · deepseek-v4.1-flash
1 participant
Part of tinyhumansai/openhuman#7328
Pins the sandbox planner's fail-closed behaviour with an exhaustive matrix (4 trust tiers x SaaS/local x env/agent/config switches x 5 backends x 4 isolation levels): untrusted tiers and SaaS never get a Noop, Process or None plan, OPENHUMAN_SANDBOX=off cannot disable isolation under SaaS, host network always requires approval, and the credential-store floor (.ssh/.aws/.gnupg) is excluded from every grant even when a ceiling or trusted root names it. Documents that Firejail/Bubblewrap map to the namespace backend. No behaviour change; the planner already enforced these.
Summary by CodeRabbit
Documentation
Tests