feat(eval): add reproducible research campaign runbook

.agents/skills/shellbench-research-runbook/SKILL.md

@@ -0,0 +1,110 @@
+---
+name: shellbench-research-runbook
+description: Plan, smoke-test, execute, checkpoint, publish, audit, and reproduce full ShellBench native benchmark campaigns across OpenClaw, Hermes, Codex, and Claude Code, including model and reasoning identity, pinned harness versions, n=3 qualification through n=6 research runs, S3 trace retention, and task-turn-tool-token-cost exports.
+---
+
+# ShellBench Research Runbook
+
+Use this skill for a real benchmark campaign, not a one-off local score.
+
+Read [references/runbook.md](references/runbook.md) before provisioning machines.
+It is the normative campaign contract and contains the commands, gates, artifact
+schema, and recovery rules.
+
+## Non-negotiable gates
+
+1. Use remote Crabbox AWS beasts for benchmark execution. Never run scored
+   trials on the operator laptop.
+2. Pin one public-task commit, runner commit or patch hash, provider model ID,
+   harness version, reasoning level, and judge route for the whole campaign.
+3. Run one `r0` qualification for every distinct harness and model-family
+   route, using exactly ten pinned representative tasks. Do not start full-suite
+   jobs until model identity, real traces, tools, usage, judge routing, and
+   artifact export pass.
+4. Retain and audit every `r0`, but force it out of leaderboard scoring. Qualify
+   with independent full-suite repetitions `r1` through `r3`. After a clean
+   audit, add `r4` through `r6`; the research result is six total repetitions.
+5. Run every provider-supported non-maximum reasoning level. Never label a
+   reasoning level as tested unless the route applies it and the trace or proxy
+   evidence proves it. Record unsupported levels instead of fabricating them.
+6. Use `gpt-5.6-sol` at `high` as the default judge. Keep the judge alias,
+   credentials, logs, and identity audit separate from the agent route.
+7. Start checkpointing after the first completed trial and continue at least
+   every ten minutes or ten new results. Verify each local archive before it
+   counts.
+8. Upload every verified checkpoint and final archive to the private S3 prefix
+   from `SHELLBENCH_TRACE_S3_URI`. Never put bucket names or credentials in git,
+   PR text, public logs, or generated reports.
+9. A run is not research-clean when traces are missing, observed model identity
+   differs from the request, reasoning is unproven, coverage is incomplete, or
+   infrastructure failures dominate.
+
+## Required commands
+
+Generate reasoning-specific plans with unique labels:
+
+```sh
+python -m scripts.native_eval.plan \
+  --tasks-root "$TASKS_ROOT" \
+  --output "$CAMPAIGN/manifests/run-index-high.json" \
+  --public-tasks-commit "$PUBLIC_TASKS_COMMIT" \
+  --run-date "$RUN_DATE" \
+  --reasoning-effort high \
+  --judge-model-id gpt-5.6-sol \
+  --judge-reasoning-effort high \
+  --repetitions 3
+```
+
+Use repeatable `--harness` and `--model` filters for r0 or phased plans.
+Set `--repetitions 6` only after the first three repetitions pass qualification.
+
+Generate each family/harness r0 separately:
+
+```sh
+python -m scripts.native_eval.plan \
+  --tasks-root "$TASKS_ROOT" \
+  --output "$CAMPAIGN/manifests/r0-openclaw-gpt56.json" \
+  --public-tasks-commit "$PUBLIC_TASKS_COMMIT" \
+  --run-date "$RUN_DATE" \
+  --phase r0 \
+  --qualification-family gpt-5.6 \
+  --harness openclaw \
+  --model gpt56-sol \
+  --reasoning-effort high \
+  --judge-model-id gpt-5.6-sol \
+  --judge-reasoning-effort high \
+  --task "<task-01>" \
+  --task "<task-02>" \
+  --task "<task-03>" \
+  --task "<task-04>" \
+  --task "<task-05>" \
+  --task "<task-06>" \
+  --task "<task-07>" \
+  --task "<task-08>" \
+  --task "<task-09>" \
+  --task "<task-10>"
+```
+
+The planner marks r0 `leaderboard_eligible=false` with exclusion reason
+`r0_non_scoring_qualification`.
+
+After extraction, produce the research tables and strict identity report:
+
+```sh
+python -m scripts.native_eval.research_audit \
+  --run-index "$CAMPAIGN/manifests/run-index.json" \
+  --extracted-root "$CAMPAIGN/extracted" \
+  --output-dir "$CAMPAIGN/summaries/research"
+```
+
+Treat any false row in `model_identity_audit.csv` as a blocker. The exporter
+does not invent prices: missing exact spend remains explicitly unavailable.
+
+## Stop conditions
+
+- Stop a route after any r0 contains the wrong or multiple model IDs.
+- Stop when the judge's observed provider model cannot be proven.
+- Stop when a harness version differs from its campaign pin.
+- Preserve and exclude infra-dominated runs, then rerun the same repetition
+  with a suffix and lower concurrency.
+- Never delete, overwrite, or silently replace an artifact or repetition.

.agents/skills/shellbench-research-runbook/agents/openai.yaml

@@ -0,0 +1,7 @@
+interface:
+  display_name: "ShellBench Research Runbook"
+  short_description: "Run reproducible, trace-audited benchmark campaigns"
+  default_prompt: "Use $shellbench-research-runbook to plan and audit a full native ShellBench research campaign."
+
+policy:
+  allow_implicit_invocation: true