feat(record_cli): serve a different canned response per invocation

.claude/harness-candidates.md

@@ -455,6 +455,31 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+- [ ] **A probe task's detector must not be satisfiable from the sandbox-readable task
+  YAML.** `docker_runner._stage_inputs` serialises the post-override `TaskDefinition` to
+  `/work/input/task.yaml` and mounts `tasks/` again at `/work/task_dir`, both agent-readable.
+  Two probes now depend on NOT being satisfiable from that text — `anti_cheat_reference` via a
+  regex that cannot match its own source, `record_cli_responses` via a log-derived detector —
+  and nothing enforces it. `record_cli_responses` originally shipped (in review) with
+  `file_contains` needles that were verbatim in its own YAML, which would have let a
+  transcribing agent pass while dispatch was dead. Guard: for every `smoke-pass` task, assert no
+  `file_contains` needle / `file_matches_regex` pattern on a must-match criterion appears in the
+  serialised task YAML. Deferred: needs per-criterion-type handling and a real false-positive
+  pass (paths and generic words will collide), so well over 30 min.
+- [ ] **A new shim failure mode must still RECORD the invocation.** A generated shim that dies
+  before `record()` leaves a log byte-identical to "the agent never ran it", which passes a
+  `max_count: 0` guard. The sidecar import was exactly that, caught only in review. Guard:
+  render each shim shape, break each external dependency in turn, assert the log is non-empty.
+  Deferred: "each external dependency" has no enumeration today, so the rule needs a seam
+  (a declared list of what a shim depends on) before it can be mechanical rather than a
+  hand-maintained list that decays.
+- [ ] **A generated-artifact invariant must be asserted against a real run of that artifact,
+  not against the config that produced it.** `TestRecordCliProbeIntegrity` first shipped
+  comparing the task YAML with itself and hardcoding `"rule": 0` — a spelling `json.dumps`'s
+  default separators own — so a separator change would have left it green while the blocking CI
+  probe failed. Now fixed for this case by running a real shim. Deferred as a general guard:
+  "derives its expectation from the thing it checks" is not mechanically detectable; it belongs
+  in the review rubric rather than a lint rule.
 
 - [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
   outside its own assignment — CE037-class dead accumulator. Surfaced during the

.github/workflows/pr-checks.yml

@@ -469,16 +469,16 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 7 task files (hello_date, dataset_example,
+      # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
-      # anti_cheat_reference); dataset_example fans out to 2 inline rows, so 8
-      # sub-tasks. If you add/remove a smoke-pass task or change the dataset row
-      # count, bump these.
+      # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2
+      # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change
+      # the dataset row count, bump these.
       #
       # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
       # match — the smoke-pass step names its path explicitly. Keep that in sync.
-      EXPECTED_SMOKE_PASS_RUN: "8"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "8"
+      EXPECTED_SMOKE_PASS_RUN: "9"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "9"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -549,6 +549,9 @@ jobs:
       # explicitly. anti_cheat_reference is the adversarial probe that the agent
       # cannot read the reference solution during its turn; it needs the
       # coder-eval-agent image built above (it is a driver: docker task).
+      # record_cli_responses is the record_cli per-invocation-response probe and
+      # is also driver: docker, so it needs that same image; it is flat in
+      # tasks/, so the glob already matches it.
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
           .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \

docs/TASK_DEFINITION_GUIDE.md

@@ -567,7 +567,41 @@ Notes:
 - **The log is seeded empty**, so a correct run that legitimately calls nothing still satisfies a `max_count: 0` guard — while a *missing* log (mock never ran, or wrote elsewhere) still fails.
 - **stdin is never read** by the shim: reading it would block whenever the sandbox leaves stdin attached to an open pipe, hanging the task.
 - **Collisions are rejected.** If a `mock_path_dirs` entry already provides an executable of the same name, setup raises rather than letting directory order decide which one runs.
-- **It stubs a tool; it does not proxy one, and it does not serve per-invocation responses.** Recording a *real* executable on the way through, or returning different output per invocation, stays a hand-written mock under `mock_path_dirs` — both depend on state the harness cannot guarantee (the tool being installed, PATH order, live credentials, a fixture set).
+- **It stubs a tool; it does not proxy one.** Recording a *real* executable on the way through stays a hand-written mock under `mock_path_dirs` — that depends on state the harness cannot guarantee (the tool being installed, PATH order, live credentials).
+
+#### Answering each invocation differently
+
+An agent whose next step depends on what the tool just told it cannot be evaluated by a stub that replies the same way to everything it types. `responses` gives one shadowed executable a reply per invocation:
+
+```yaml
+sandbox:
+  record_cli:
+    - tool: uip
+      exit_code: 1                    # fallback: anything no rule claims
+      stderr: "uip: unknown command\n"
+      responses:
+        - when: {verb: "ixp dummy1"}
+          stdout: "response1\n"
+        - when: {verb: "ixp dummy2"}
+          stdout: "response2\n"
+        - when:                       # any cli_called facet, ANDed
+            verb: "ixp projects get"
+            positional: ["proj-1"]
+            flags: {model: gemini_2_5_pro}
+          stdout: '{"id": "proj-1", "name": "Invoices"}'
+        - when: {verb: "ixp projects get missing"}
+          exit_code: 4
+          stderr: "project not found\n"
+```
+
+- **`when` takes the same facets as [`cli_called`](#cli_called)** — `verb`, `verb_any_of`, `positional`, `flags`, `value_flags`, `ignore_flags` — evaluated by the same matcher, so the pattern that *serves* a response is the pattern that *grades* it. Always a mapping: a bare `when: "ixp dummy1"` is rejected (with the `{verb: ...}` spelling in the message), since a pattern has six facets and a lone string leaves which one you meant to infer.
+- **First match wins**, in declaration order: put the specific rule above the general one. An invocation no rule claims gets the entry's own `exit_code` / `stdout` / `stderr`.
+- **`exit_code` defaults to 0 on a rule** — the opposite of the entry default of 1. A rule exists because you described that invocation, so the natural reading is "and this is what it answers"; an undescribed one should still look like a tool that failed.
+- **`ignore_flags` is empty on a rule**, unlike the criterion's `[output]`: grading must not depend on a flag that changes nothing about the outcome, but a rule may legitimately answer differently for `--output json`. That is the one place a rule is *not* copy-pastable into a criterion — `flags: {output: ...}` is valid on a rule and rejected on the criterion, which ignores that flag by default.
+- **The log names the rule that answered** (`"rule": 1`), and omits the key when none did — the first thing you want to know when an expected canned response does not arrive.
+- **The recorder directory also holds `argv_match.py`** — the matcher module the shim imports as a sibling, written there only for entries that declare `responses`. It is regenerated on every sandbox setup, so do not edit it, and do not declare a `tool` that would shadow it (the name is rejected).
+- **A runnable worked example** ships in the repo: [`tasks/record_cli_responses.yaml`](https://github.com/UiPath/coder_eval/blob/main/tasks/record_cli_responses.yaml) stubs two subcommands with different replies, has the agent capture what each printed, and grades both the log and the captured text. Run it with `coder-eval run tasks/record_cli_responses.yaml` (needs `make docker-image` — it is a `driver: docker` task).
+- **Still stateless.** A rule answers the same way however many times it matches; a counter would have to survive concurrent agent commands. For a tool whose reply must change over a run, hand-write a mock under `mock_path_dirs`.
 
 ## Template Sources
 
@@ -973,6 +1007,8 @@ Use this instead of `command_executed` or `file_matches_regex` when a test shado
 
 Do **not** shorten the verb instead. `verb: "ixp projects"` matches all of its subcommands, so a positive assertion that the agent *read* a project is equally satisfied by `ixp projects delete`. Two entries are rejected when one prefixes the other, since the shorter already accepts everything the longer does.
 
+**A verb holds subcommands only.** `verb: "ixp projects get --output json"` is rejected: the verb is compared against the *non-flag* arguments, so a flag inside it could never match — the criterion would score 0 against a log holding that exact call. Put it in `flags:` instead. (`head -1` still validates: a bare negative number is a value to the argument splitter, not a flag.)
+
 **The argument tail stays open.** `positional` is a prefix too, so `verb: "ixp projects list"` with `positional: ["proj-1"]` also matches `ixp projects list proj-1 dummy`. To require a specific tail, name every argument in it. `positional: []` is rejected — it would assert nothing.
 
 **Declare value-bearing flags when you use `positional`.** An undeclared flag is treated as a switch, so its value stays among the non-flag arguments and shifts the ones you named. `get proj-1 --folder Finance` matches `positional: ["proj-1"]`, but `get --folder Finance proj-1` does **not** — `Finance` takes the first slot. Add `folder` to `value_flags` (or name it in `flags`) to fix it. Resolving the ambiguity this way is deliberate: guessing that an unknown flag consumes the next token let `--yes proj-1` bind `yes=proj-1` and swallow the project name, which made a `max_count: 0` delete guard pass on the delete it forbade.

pyproject.toml

@@ -274,6 +274,7 @@ external = [
     "CE054",
     "CE055",
     "CE056",
+    "CE057",
 ]  # custom architectural lint rules (tests/lint/)
 
 [tool.ruff.lint.pylint]