docs(evals): track the plugin eval sandbox's hub-only read limit

[kyle-sexton]

Problem

Inside a claude plugin eval run, a skill under test can read only its hub SKILL.md. Every Read or Grep of the skill's reference/ or scripts/ files is refused with "File is in a directory that is denied by your permission settings", and the refusal lands in the trace's permission_denials. A path-scoped --allow-tools "Read(//<plugin>/skills/**)" grant is accepted by the CLI but does not lift the denial, and claude plugin eval --help lists no flag that makes a directory readable.

So any fact a case depends on must be in the hub. A skill that tells the model to "load the reference file" scores worse inside an eval than it does in real use, and every such read makes the run fail this repo's validity gate (plugins/evals/skills/plugin-eval/scripts/run-validity.py).

This is a limitation of the eval sandbox, tracked here. It can change in any Claude Code release, and it may depend on the account or the context.

Evidence

• Claude Code 2.1.287, 2026-10-02: the evals plugin's 31-case suite, job 5 baseline. 9 with-arm runs were denied reads of skills/*/reference/*.md, and 5 were denied Greps of skills/*/scripts/.
• The V1 probe: the same denial occurred with the path-scoped grant on, and the run's settings held no deny rule.
• The record, with a recheck trigger, is in plugins/evals/skills/plugin-eval/SKILL.md (the reference-read row in its facts table).

What this repo does about it

• Hubs state what a case needs. Reference files are labelled as background for a human reader.
• run-validity.py fails a run with any with-arm denial at, under or above the plugin's directory.

Recheck

Recheck when a Claude Code release note touches plugin eval or sandbox permissions, when the plugin-evals docs gain a way to make a directory readable, or when a kept trace shows a reference Read succeeding. Then run one case with --keep-temp, read the with-arm trace, and update the facts-table row in plugins/evals/skills/plugin-eval/SKILL.md.

[kyle-sexton]

This was generated by AI during triage.

Agent Brief

Type: Task
Summary: Track the claude plugin eval sandbox limit that lets a skill under test read only its hub SKILL.md.

Current behavior:
Inside a plugin eval run, a Read or Grep of a skill's reference/ or scripts/ files is denied, a path-scoped --allow-tools grant does not lift it, and no CLI flag makes a directory readable. The evals plugin already records this as a fact with its basis, as-of dates (through 2026-10-02 on 2.1.287) and a recheck trigger, and its validity gate fails a run on any with-arm denial at, under or above the plugin directory.

Desired behavior:
When the recheck trigger fires (a release note touching plugin eval or sandbox permissions, a readable-directory flag, or a kept trace showing a spoke Read succeeding), one case is re-run with --keep-temp, with and without the grant, and the facts-table row is refreshed with the outcome.

Acceptance criteria:

• The facts-table row states the outcome of the latest recheck with its Claude Code version and date

Out of scope:

• Moving reference content into hubs (already the stated practice)
• Changing the validity gate's denial rule

Routing: human-gated (proposed close)

Question for the owner: Close #5878 as already recorded? Recommended: yes. The limit, its evidence and its recheck trigger already live in the plugin-eval skill's facts table, which is the repository's chosen place for volatile upstream facts (pointer, as-of date, recheck trigger). This issue repeats that record and has no work to do until the trigger fires. Keep it open only if you want a tracker item as well as the in-skill trigger.

Note: the local Claude Code is now 2.1.289, two releases past the recorded check (2.1.287). Whether either release touched plugin eval or sandbox permissions was not checked here.

Basis: verified, plugins/evals/skills/plugin-eval/SKILL.md:338 (facts row with recheck trigger); plugins/evals/skills/plugin-eval/scripts/run-validity.py:15-22 (with-arm denial at, under or above the plugin directory always FAILs). Searched docs/out-of-scope/: no matching concept. Filed as a follow-up of PR #5895.