✨ feat: guide proactive reviewer investigation

evals/reviewer/README.md

@@ -4,13 +4,18 @@ The runtime tests validate orchestration. They do not establish that a model ide
 
 Use the same model, effort, environment and original requirements for three conditions: existing review, independent design comparison, and independent failure model plus test audit. Give the independent child only `requirements` and `base`; disclose implementation exposure on continuations. Freeze its artifacts before revealing `head` and `tests`. Keep the grading notes separate from the agent's input. Preserve complete outputs, costs, latency, skipped stages and unavailable checks.
 
-A maintainer judges each finding without seeing which condition produced it: correct reachable defect, useful simplification, acceptable tradeoff, unsupported preference, invalid deletion, or missing evidence. Repeat a subset to assess stability. Do not grade by keyword inclusion, report length, finding count, or the agent's own preference. The four fixtures below are deliberately small; no claim of general effectiveness follows from passing them.
+A maintainer judges each finding without seeing which condition produced it: correct reachable defect, useful simplification, acceptable tradeoff, unsupported preference, invalid deletion, or missing evidence. Repeat a subset to assess stability. Do not grade by keyword inclusion, report length, finding count, or the agent's own preference. These fixtures are deliberately small; no claim of general effectiveness follows from passing them.
 
-| Case             | Files                          | Question                                                           |
-| ---------------- | ------------------------------ | ------------------------------------------------------------------ |
-| Whitespace       | `cases.json`, `whitespace`     | Does the test isolate the whitespace-sensitive behavior?           |
-| Request ordering | `cases.json`, `latest-request` | Does the reviewer construct the actual late-response sequence?     |
-| Cached size      | `cases.json`, `cached-size`    | Can duplicate state be removed while preserving observed behavior? |
-| Required adapter | `cases.json`, `adapter`        | Does the reviewer preserve the real protocol boundary?             |
+| Case             | Files                          | Question                                                                         |
+| ---------------- | ------------------------------ | -------------------------------------------------------------------------------- |
+| Whitespace       | `cases.json`, `whitespace`     | Does the test isolate the whitespace-sensitive behavior?                         |
+| Request ordering | `cases.json`, `latest-request` | Does the reviewer construct the actual late-response sequence?                   |
+| Cached size      | `cases.json`, `cached-size`    | Can duplicate state be removed while preserving observed behavior?               |
+| Required adapter | `cases.json`, `adapter`        | Does the reviewer preserve the real protocol boundary?                           |
+| Batch export     | `batch-export`                 | Does it discover questions, investigate them, and preserve justified boundaries? |
+
+The original four cases expose a specific behavior in the requirements and remain useful directed checks. [Batch export](batch-export/README.md) adds a neutral request and runnable base/head repositories, so the reviewer must choose which questions to investigate. Its evaluator-only probes and grading notes are separate from the input. Compare old and new guidance with fresh sessions, otherwise identical settings, and the same unhinted input. This comparison is separate from the three design/test conditions above.
+
+Retain observable investigation evidence, not just the final report: the questions the reviewer raised without evaluator feedback, paths and consumers examined, experiments it selected, conclusions it revised or ruled out, related paths it followed, and remaining gaps. An independently discovered valid issue outside the proposed grading notes still counts. Do not require a fixed investigation sequence or a particular tool when another method provides sufficient evidence. Human grading should consider whether the evidence supports the conclusion and whether useful boundaries survive proposed simplification.
 
 Run code experiments in disposable workspaces. Record a clean baseline, one plausible defect, and a behavior-preserving alternative when relevant. A syntax/import/environment failure is inconclusive. The directed runtime experiments used to validate this implementation are separate from model evaluation.

evals/reviewer/batch-export/README.md

@@ -0,0 +1,53 @@
+# Batch export evaluation
+
+This is a small synthetic repository, inspired by review experience rather than copied from a project. It tests investigation of a change across callers, persisted state, and a receiver boundary. It supplies both a simplification opportunity and behavior that must survive simplification. It does not establish general review quality.
+
+## Reviewer input
+
+Give the reviewer only `requirements.md`, `base/`, and `head/`, copied to a disposable input directory outside this repository. Do not expose this document, the parent evaluation documents, or `evaluator/`. For an independent design condition, initially give that child only `requirements.md` and `base/`; freeze its output before revealing the head.
+
+For a guidance comparison, start fresh sessions with the same model, effort, tools, budget, and input. Supply the old or new guidance separately, with neutral run labels. Disable shared/native memory and do not give later sessions earlier results. Preserve submitted inputs and guidance, actual model settings, transcripts including tool calls, elapsed time, costs when available, and unavailable checks. Have a maintainer grade anonymized outputs and their evidence. Keep evaluator feedback until the run ends.
+
+The input has no named bug category or request to investigate a particular mechanism. A reviewer can discover questions by tracing the ordinary entry point and existing contracts. Do not provide the probes below as reviewer tasks or reward a report for reproducing their names.
+
+## Fixture checks
+
+These commands require only Python 3.11+ and Git. Run from the Nyanpasu repository root. Both submitted baseline suites should pass:
+
+```sh
+python -B -m unittest discover -s evals/reviewer/batch-export/base -p checks.py -v
+python -B -m unittest discover -s evals/reviewer/batch-export/head -p checks.py -v
+```
+
+The evaluator's independent behavioral witnesses execute the head's real exporter, checkpoint file, and SQLite receiver. The unmodified head intentionally exits 1: the repeated-record scenario raises `ValueError`, the rejected-delivery retry loses its first batch, and the lost-acknowledgement scenario passes.
+
+```sh
+python -B evals/reviewer/batch-export/evaluator/probe.py evals/reviewer/batch-export/head -v
+```
+
+Apply the small alternative in a disposable copy. It removes the content ledger and changes checkpoint ordering, while keeping the receiver request identity and acknowledgement behavior. The submitted suite and all three witnesses should pass:
+
+```sh
+case_dir="$PWD/evals/reviewer/batch-export"
+trial_dir="$(mktemp -d)"
+cp -R "$case_dir/head/." "$trial_dir/"
+git -C "$trial_dir" apply "$case_dir/evaluator/simpler.patch"
+python -B -m unittest discover -s "$trial_dir" -p checks.py -v
+python -B "$case_dir/evaluator/probe.py" "$trial_dir" -v
+```
+
+Then apply one plausible defect to the alternative: reject a repeated receiver request instead of acknowledging its committed delivery. The submitted suite still passes, but the lost-acknowledgement witness must now fail with `ValueError`. This demonstrates why the remaining receiver boundary matters:
+
+```sh
+git -C "$trial_dir" apply "$case_dir/evaluator/reject-retry.patch"
+python -B -m unittest discover -s "$trial_dir" -p checks.py -v
+python -B "$case_dir/evaluator/probe.py" "$trial_dir" -v
+```
+
+Record actual output and exit status; import, patch, or environment failures are inconclusive. These checks validate the fixture and its behavioral witnesses. They are not model runs or evidence that revised reviewer guidance improves review quality. The injected connection failures exercise the actual local persistence paths; they do not validate a real network transport, power-loss durability, concurrent exporters, or changing input files.
+
+## Running a model
+
+Use the existing Nyanpasu task/session interface with an isolated evaluation home and explicit execution target. Run within the reviewer's execution environment, including its container when deployed that way; do not substitute host binaries for container paths. `uv run nyanpasu run-task task.json --target <backend/model:effort>` runs a real native session; the task should use a fresh context, memory disabled, the staged input workspace, and the chosen guidance. Do not copy live credentials into the input directory or start/restart the production service.
+
+`run-task` alone does not install the GitHub reviewer plugin's preparers and controls. A generic task with the review guidance can establish a behavior smoke result, but cannot establish that the complete review/publication lifecycle ran. Record exactly which instructions and controls were supplied. Existing runtime tests and these fixture checks are separate evidence from either kind of model evaluation.

evals/reviewer/batch-export/base/README.md

@@ -0,0 +1,7 @@
+# Batch export
+
+The command copies a UTF-8 text file to a receiver database. Each input line is one record, and records retain their input order. An input file is stable during a job and its retries; its absolute path identifies the source. Batches contain two records.
+
+Run `python courier.py source.txt receiver.sqlite3`. Repeating a command against the same receiver is supported. The receiver stores each request by source and starting offset. Repeating that request with the same payload acknowledges the existing delivery; a different payload for that request is rejected. A connection error does not establish whether a request committed.
+
+Run the repository tests with `python -B checks.py -v`.

evals/reviewer/batch-export/base/checks.py

@@ -0,0 +1,24 @@
+from __future__ import annotations
+
+import tempfile
+import unittest
+from pathlib import Path
+
+from courier import export
+from receiver import Receiver
+
+
+class ExportTests(unittest.TestCase):
+    def test_export_and_repeat(self):
+        with tempfile.TemporaryDirectory() as directory:
+            root = Path(directory)
+            source = root / "source.txt"
+            source.write_text("red\ngreen\nblue\n", encoding="utf-8")
+            receiver = Receiver(root / "receiver.sqlite3")
+            export(source, receiver)
+            export(source, receiver)
+            self.assertEqual(receiver.records(str(source.resolve())), ["red", "green", "blue"])
+
+
+if __name__ == "__main__":
+    unittest.main()

evals/reviewer/batch-export/base/courier.py

@@ -0,0 +1,20 @@
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+from receiver import Receiver
+
+
+def export(source: Path, receiver: Receiver) -> None:
+    records = source.read_text(encoding="utf-8").splitlines()
+    for offset in range(0, len(records), 2):
+        receiver.deliver(str(source.resolve()), offset, records[offset : offset + 2])
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument("source", type=Path)
+    parser.add_argument("receiver", type=Path)
+    args = parser.parse_args()
+    export(args.source, Receiver(args.receiver))

evals/reviewer/batch-export/base/receiver.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+import json
+import sqlite3
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+
+class Receiver:
+    def __init__(self, path: Path):
+        self.path = path
+        with sqlite3.connect(path) as db:
+            db.execute(
+                "CREATE TABLE IF NOT EXISTS deliveries "
+                "(source TEXT, offset INTEGER, payload TEXT, PRIMARY KEY (source, offset))"
+            )
+
+    def deliver(self, source: str, offset: int, records: list[str]) -> None:
+        payload = json.dumps(records)
+        with sqlite3.connect(self.path) as db:
+            existing = db.execute(
+                "SELECT payload FROM deliveries WHERE source = ? AND offset = ?", (source, offset)
+            ).fetchone()
+            if existing is not None:
+                if existing[0] != payload:
+                    raise ValueError("request reused with different records")
+                return
+            db.execute("INSERT INTO deliveries VALUES (?, ?, ?)", (source, offset, payload))
+
+    def records(self, source: str) -> list[str]:
+        with sqlite3.connect(self.path) as db:
+            batches = db.execute("SELECT payload FROM deliveries WHERE source = ? ORDER BY offset", (source,))
+            return [record for (payload,) in batches for record in json.loads(payload)]

evals/reviewer/batch-export/evaluator/grading-notes.md

@@ -0,0 +1,22 @@
+# Withheld evaluator notes
+
+The proposed expectations below need maintainer judgment. Do not give the reviewer this directory, the parent evaluation documents, hints from a prior run, or defect names. Other evidenced findings are admissible. Do not require a particular number of findings, exact wording, a mandatory investigation sequence, or a complete replacement implementation.
+
+| Decision                                                          | Independent contract and witness                                                                                                                                   | Evidence to look for                                                                                                                                                                                                                                                        |
+| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Remove the global content ledger                                  | Each source line is a record. `red, green, red, blue` contains four occurrences, but head rejects the second batch.                                                | Trace the digest set from creation through persistence and consumption. Determine whether equal content denotes the same delivery. Removing hashing while retaining content rejection does not fix the behavior.                                                            |
+| Save progress after successful delivery                           | A rerun must complete an interrupted job. If the first receiver call rejects before commit, head has already advanced the checkpoint and retry yields only `blue`. | Follow the ordinary command through checkpoint save, delivery, and a new exporter invocation. Distinguish receiver acceptance from caller acknowledgement. Passing the supplied success test is insufficient.                                                               |
+| Retain receiver request identity and idempotent acknowledgement   | A connection can fail after a real commit. With correct checkpoint ordering, retry submits the same source/offset request again.                                   | Show that the simplified exporter still finishes with exactly the original records after the lost acknowledgement. The rejection mutation makes this scenario fail while the supplied suite stays green. This is a justified boundary, not redundant content deduplication. |
+| Retain useful success coverage and add targeted recovery coverage | The supplied test checks the real exporter and receiver, including repeat invocation. Its expected record list comes from the input.                               | It survives the simpler alternative. A useful recommendation strengthens failure coverage without deleting the success test merely because it misses recovery failures.                                                                                                     |
+
+Review the observable investigation, not a polished explanation alone. Did the reviewer independently notice an unexplained mechanism or understanding gap, trace its actual consumers, choose a discriminating experiment or equivalent proof, and update its conclusion using the result? Did it distinguish the removable ledger from required retry behavior and follow the implications of changing checkpoint ordering? If it initially suspects receiver duplication, does later evidence make it retain that boundary? A reviewer need not make an initial mistake to demonstrate sound investigation.
+
+Check proposed regression inputs as well as executed experiments. Two identical lines fit in one batch and do not trigger the ledger rejection; a repeat must cross a batch boundary. A valid experiment followed by a weaker test suggestion is still a gap. For the retained receiver boundary, distinguish tracing the acknowledgement branch from actually exercising a real commit followed by a lost acknowledgement and retry; do not claim an experiment that was not run.
+
+The README requires keeping the checkpoint with its source and receiver. Demonstrating truncation after cross-source checkpoint reuse does not establish that this PR must add identity fields or defend against hand-edited state; that is a proposed responsibility beyond the stated contract. Likewise, the original request does not explicitly freeze CLI arity: observing the old invocation fail supports a compatibility question, not by itself a confirmed requirement violation. Judge any proposed default checkpoint path against both source and receiver identity before treating it as a validated alternative.
+
+The root reviewer remains responsible for combining any child findings, reconciling these interacting paths, and stating unresolved scope. Child completion or green baseline tests do not answer the open questions. Evaluate whether the final evidence supports the conclusions; do not grade internal reasoning text, use of these exact tools, or the presence of a named checklist.
+
+`simpler.patch` is a bounded alternative against the submitted head, not a second maintained implementation. `reject-retry.patch` is a deliberately invalid mutation of the retained receiver boundary. The probe expectations are hand-derived from the source and receiver contracts, and the injected exceptions surround actual SQLite writes. The tests do not reconstruct checkpoint state on behalf of production code.
+
+No conclusion about hashes, identifiers, persistence, or adapters follows beyond these callers and contracts. An implementation using any of them for a demonstrated responsibility can be valid. Scope is one sequential exporter, unchanged source bytes, and this local receiver. Real transport compatibility and crash durability remain outside the fixture.