feat(eval): add plan-mode and HITL contract evals to replay harness

[zjshen14]

Summary

• Extends RunTapeOptions in src/eval/replay/runner.ts with confirmFn and forcesConfirmation fields, wired to Agent.setConfirmFn / Agent.setForcesConfirmationFn after construction — enabling HITL contract evals to drive the confirmation gate fully offline.
• Adds a plan-mode contract suite in synthesized.test.ts: a synthesized tape with a write call in plan mode asserts tool_denied(plan_mode) fires, execute() is never reached, and the tape exhausts with zero unconsumed results.
• Adds a HITL contract suite (deny / non-interactive / allow): uses forcesConfirmation to trigger the gate without modifying TapeRegistry tools; asserts the correct tool_denied reason and execution-log state for each confirmFn outcome.

Closes part of #234 (plan-mode and HITL sub-items).

Test plan

• npm run typecheck && npm run lint && npm run format:check && npm test — all pass (710 tests, 0 failures)
• src/eval/replay/synthesized.test.ts grows from 8 to 14 tests — 6 new tests covering plan-mode block and HITL deny / non-interactive / allow paths
• No changes to agent loop, executor, or any non-eval code

https://claude.ai/code/session_01NnteuN9Vv47jNyFvsa1mBp

---

Generated by Claude Code

[zjshen14]

Review: feat(eval): add plan-mode and HITL contract evals to replay harness

What's good: Clean extension of RunTapeOptions, well-structured inline tapes, and solid assertions on both the plan-mode write block and the three HITL paths. CI is green, 6 new tests pass.

---

Overlap with #236

This PR and #236 cover the same #234 sub-items and make identical changes to runner.ts (the confirmFn and forcesConfirmation additions). The key difference is where the tests live:

PR	Test location	Approach
#236 (older)	New src/eval/replay/contract.test.ts + JSONL tape + expected JSON	Dedicated contract eval module with file-based tapes
#237 (this PR, newer)	Extends src/eval/replay/synthesized.test.ts	Inline in existing file

Merging both would conflict on runner.ts and produce duplicate coverage. This is the newer PR, suggesting it may be the preferred implementation — but please confirm by closing the other before merging.

Design trade-offs:

• feat(eval): add plan-mode and HITL contract evals to replay harness #237 (this PR): Less file proliferation; no new JSONL/JSON artifacts; synthesized tests stay co-located.
• feat(eval): add plan-mode and HITL contract evals (D2 follow-up) #236: Dedicated contract.test.ts is clearly scoped; file-based tapes are independently inspectable.

Recommendation: Pick one of #236 or #237 and close the other. No correctness issues in either.

---

Generated by Claude Code

[zjshen14]

Review: feat(eval): add plan-mode and HITL contract evals to replay harness

What's good: The runner.ts extension is minimal and correct. The three HITL paths (deny / non-interactive / allow) are all covered, and the plan-mode write-block test correctly asserts both tool_denied(plan_mode) and zero tool_exec_start events.

Concern: Direct conflict with #236

Both this PR and #236 make identical changes to src/eval/replay/runner.ts — the same confirmFn? and forcesConfirmation? fields on RunTapeOptions, the same import type { ConfirmFn } from core/executor.ts, and the same 3-line wiring in runTape(). Both originate from the same base commit (9c1073b5).

The test files differ:

• feat(eval): add plan-mode and HITL contract evals (D2 follow-up) #236: creates a dedicated src/eval/replay/contract.test.ts (9 tests) with a JSONL tape fixture, expected JSON, and README update
• This PR (feat(eval): add plan-mode and HITL contract evals to replay harness #237): adds 6 tests inline to src/eval/replay/synthesized.test.ts

contract.test.ts in #236 is a superset — it covers the same plan-mode and HITL paths, adds a tape-parse assertion, and an explicit no-guard-triggered assertion. The fixture-driven approach is also more durable.

Recommendation: Close this PR in favour of #236, which is the more complete implementation. If the 6 inline tests in synthesized.test.ts are specifically wanted there (e.g. alongside the existing synthesized tests), they can be extracted and added as a follow-up once #236 lands.

---

Generated by Claude Code