From 6169dff71f8962c01b726f4b2f2926abd7bf2936 Mon Sep 17 00:00:00 2001 From: Nyakku Shigure Date: Sun, 4 Oct 2026 19:31:28 +0800 Subject: [PATCH 1/2] :sparkles: feat: guide proactive reviewer investigation Make the main reviewer own its working understanding, choose evidence that can change a conclusion, and revisit affected conclusions when requirements change. Add an unhinted batch-export evaluation with removal, correction, and retention witnesses. Co-authored-by: Codex --- evals/reviewer/README.md | 19 +++-- evals/reviewer/batch-export/README.md | 53 ++++++++++++++ evals/reviewer/batch-export/base/README.md | 7 ++ evals/reviewer/batch-export/base/checks.py | 24 +++++++ evals/reviewer/batch-export/base/courier.py | 20 ++++++ evals/reviewer/batch-export/base/receiver.py | 35 ++++++++++ .../batch-export/evaluator/grading-notes.md | 18 +++++ .../reviewer/batch-export/evaluator/probe.py | 69 +++++++++++++++++++ .../batch-export/evaluator/reject-retry.patch | 11 +++ .../batch-export/evaluator/simpler.patch | 38 ++++++++++ evals/reviewer/batch-export/head/README.md | 7 ++ .../reviewer/batch-export/head/checkpoint.py | 22 ++++++ evals/reviewer/batch-export/head/checks.py | 26 +++++++ evals/reviewer/batch-export/head/courier.py | 31 +++++++++ evals/reviewer/batch-export/head/receiver.py | 35 ++++++++++ evals/reviewer/batch-export/requirements.md | 5 ++ evals/reviewer/grading-notes.md | 1 + packages/nyanpasu-github-reviewer/README.md | 2 + .../instructions/review-planning.md | 4 +- .../instructions/reviewer.md | 18 ++++- 20 files changed, 433 insertions(+), 12 deletions(-) create mode 100644 evals/reviewer/batch-export/README.md create mode 100644 evals/reviewer/batch-export/base/README.md create mode 100644 evals/reviewer/batch-export/base/checks.py create mode 100644 evals/reviewer/batch-export/base/courier.py create mode 100644 evals/reviewer/batch-export/base/receiver.py create mode 100644 evals/reviewer/batch-export/evaluator/grading-notes.md create mode 100644 evals/reviewer/batch-export/evaluator/probe.py create mode 100644 evals/reviewer/batch-export/evaluator/reject-retry.patch create mode 100644 evals/reviewer/batch-export/evaluator/simpler.patch create mode 100644 evals/reviewer/batch-export/head/README.md create mode 100644 evals/reviewer/batch-export/head/checkpoint.py create mode 100644 evals/reviewer/batch-export/head/checks.py create mode 100644 evals/reviewer/batch-export/head/courier.py create mode 100644 evals/reviewer/batch-export/head/receiver.py create mode 100644 evals/reviewer/batch-export/requirements.md diff --git a/evals/reviewer/README.md b/evals/reviewer/README.md index 2aa93dd..8850db2 100644 --- a/evals/reviewer/README.md +++ b/evals/reviewer/README.md @@ -4,13 +4,18 @@ The runtime tests validate orchestration. They do not establish that a model ide Use the same model, effort, environment and original requirements for three conditions: existing review, independent design comparison, and independent failure model plus test audit. Give the independent child only `requirements` and `base`; disclose implementation exposure on continuations. Freeze its artifacts before revealing `head` and `tests`. Keep the grading notes separate from the agent's input. Preserve complete outputs, costs, latency, skipped stages and unavailable checks. -A maintainer judges each finding without seeing which condition produced it: correct reachable defect, useful simplification, acceptable tradeoff, unsupported preference, invalid deletion, or missing evidence. Repeat a subset to assess stability. Do not grade by keyword inclusion, report length, finding count, or the agent's own preference. The four fixtures below are deliberately small; no claim of general effectiveness follows from passing them. +A maintainer judges each finding without seeing which condition produced it: correct reachable defect, useful simplification, acceptable tradeoff, unsupported preference, invalid deletion, or missing evidence. Repeat a subset to assess stability. Do not grade by keyword inclusion, report length, finding count, or the agent's own preference. These fixtures are deliberately small; no claim of general effectiveness follows from passing them. -| Case | Files | Question | -| ---------------- | ------------------------------ | ------------------------------------------------------------------ | -| Whitespace | `cases.json`, `whitespace` | Does the test isolate the whitespace-sensitive behavior? | -| Request ordering | `cases.json`, `latest-request` | Does the reviewer construct the actual late-response sequence? | -| Cached size | `cases.json`, `cached-size` | Can duplicate state be removed while preserving observed behavior? | -| Required adapter | `cases.json`, `adapter` | Does the reviewer preserve the real protocol boundary? | +| Case | Files | Question | +| ---------------- | ------------------------------ | -------------------------------------------------------------------------------- | +| Whitespace | `cases.json`, `whitespace` | Does the test isolate the whitespace-sensitive behavior? | +| Request ordering | `cases.json`, `latest-request` | Does the reviewer construct the actual late-response sequence? | +| Cached size | `cases.json`, `cached-size` | Can duplicate state be removed while preserving observed behavior? | +| Required adapter | `cases.json`, `adapter` | Does the reviewer preserve the real protocol boundary? | +| Batch export | `batch-export` | Does it discover questions, investigate them, and preserve justified boundaries? | + +The original four cases expose a specific behavior in the requirements and remain useful directed checks. [Batch export](batch-export/README.md) adds a neutral request and runnable base/head repositories, so the reviewer must choose which questions to investigate. Its evaluator-only probes and grading notes are separate from the input. Compare old and new guidance with fresh sessions, otherwise identical settings, and the same unhinted input. This comparison is separate from the three design/test conditions above. + +Retain observable investigation evidence, not just the final report: the questions the reviewer raised without evaluator feedback, paths and consumers examined, experiments it selected, conclusions it revised or ruled out, related paths it followed, and remaining gaps. An independently discovered valid issue outside the proposed grading notes still counts. Do not require a fixed investigation sequence or a particular tool when another method provides sufficient evidence. Human grading should consider whether the evidence supports the conclusion and whether useful boundaries survive proposed simplification. Run code experiments in disposable workspaces. Record a clean baseline, one plausible defect, and a behavior-preserving alternative when relevant. A syntax/import/environment failure is inconclusive. The directed runtime experiments used to validate this implementation are separate from model evaluation. diff --git a/evals/reviewer/batch-export/README.md b/evals/reviewer/batch-export/README.md new file mode 100644 index 0000000..fde8d31 --- /dev/null +++ b/evals/reviewer/batch-export/README.md @@ -0,0 +1,53 @@ +# Batch export evaluation + +This is a small synthetic repository, inspired by review experience rather than copied from a project. It tests investigation of a change across callers, persisted state, and a receiver boundary. It supplies both a simplification opportunity and behavior that must survive simplification. It does not establish general review quality. + +## Reviewer input + +Give the reviewer only `requirements.md`, `base/`, and `head/`, copied to a disposable input directory outside this repository. Do not expose this document, the parent evaluation documents, or `evaluator/`. For an independent design condition, initially give that child only `requirements.md` and `base/`; freeze its output before revealing the head. + +For a guidance comparison, start fresh sessions with the same model, effort, tools, budget, and input. Supply the old or new guidance separately, with neutral run labels. Disable shared/native memory and do not give later sessions earlier results. Preserve submitted inputs and guidance, actual model settings, transcripts including tool calls, elapsed time, costs when available, and unavailable checks. Have a maintainer grade anonymized outputs and their evidence. Keep evaluator feedback until the run ends. + +The input has no named bug category or request to investigate a particular mechanism. A reviewer can discover questions by tracing the ordinary entry point and existing contracts. Do not provide the probes below as reviewer tasks or reward a report for reproducing their names. + +## Fixture checks + +These commands require only Python 3.11+ and Git. Run from the Nyanpasu repository root. Both submitted baseline suites should pass: + +```sh +python -B -m unittest discover -s evals/reviewer/batch-export/base -p checks.py -v +python -B -m unittest discover -s evals/reviewer/batch-export/head -p checks.py -v +``` + +The evaluator's independent behavioral witnesses execute the head's real exporter, checkpoint file, and SQLite receiver. The unmodified head intentionally exits 1: the repeated-record scenario raises `ValueError`, the rejected-delivery retry loses its first batch, and the lost-acknowledgement scenario passes. + +```sh +python -B evals/reviewer/batch-export/evaluator/probe.py evals/reviewer/batch-export/head -v +``` + +Apply the small alternative in a disposable copy. It removes the content ledger and changes checkpoint ordering, while keeping the receiver request identity and acknowledgement behavior. The submitted suite and all three witnesses should pass: + +```sh +case_dir="$PWD/evals/reviewer/batch-export" +trial_dir="$(mktemp -d)" +cp -R "$case_dir/head/." "$trial_dir/" +git -C "$trial_dir" apply "$case_dir/evaluator/simpler.patch" +python -B -m unittest discover -s "$trial_dir" -p checks.py -v +python -B "$case_dir/evaluator/probe.py" "$trial_dir" -v +``` + +Then apply one plausible defect to the alternative: reject a repeated receiver request instead of acknowledging its committed delivery. The submitted suite still passes, but the lost-acknowledgement witness must now fail with `ValueError`. This demonstrates why the remaining receiver boundary matters: + +```sh +git -C "$trial_dir" apply "$case_dir/evaluator/reject-retry.patch" +python -B -m unittest discover -s "$trial_dir" -p checks.py -v +python -B "$case_dir/evaluator/probe.py" "$trial_dir" -v +``` + +Record actual output and exit status; import, patch, or environment failures are inconclusive. These checks validate the fixture and its behavioral witnesses. They are not model runs or evidence that revised reviewer guidance improves review quality. The injected connection failures exercise the actual local persistence paths; they do not validate a real network transport, power-loss durability, concurrent exporters, or changing input files. + +## Running a model + +Use the existing Nyanpasu task/session interface with an isolated evaluation home and explicit execution target. Run within the reviewer's execution environment, including its container when deployed that way; do not substitute host binaries for container paths. `uv run nyanpasu run-task task.json --target ` runs a real native session; the task should use a fresh context, memory disabled, the staged input workspace, and the chosen guidance. Do not copy live credentials into the input directory or start/restart the production service. + +`run-task` alone does not install the GitHub reviewer plugin's preparers and controls. A generic task with the review guidance can establish a behavior smoke result, but cannot establish that the complete review/publication lifecycle ran. Record exactly which instructions and controls were supplied. Existing runtime tests and these fixture checks are separate evidence from either kind of model evaluation. diff --git a/evals/reviewer/batch-export/base/README.md b/evals/reviewer/batch-export/base/README.md new file mode 100644 index 0000000..3f6fb2b --- /dev/null +++ b/evals/reviewer/batch-export/base/README.md @@ -0,0 +1,7 @@ +# Batch export + +The command copies a UTF-8 text file to a receiver database. Each input line is one record, and records retain their input order. An input file is stable during a job and its retries; its absolute path identifies the source. Batches contain two records. + +Run `python courier.py source.txt receiver.sqlite3`. Repeating a command against the same receiver is supported. The receiver stores each request by source and starting offset. Repeating that request with the same payload acknowledges the existing delivery; a different payload for that request is rejected. A connection error does not establish whether a request committed. + +Run the repository tests with `python -B checks.py -v`. diff --git a/evals/reviewer/batch-export/base/checks.py b/evals/reviewer/batch-export/base/checks.py new file mode 100644 index 0000000..972ec74 --- /dev/null +++ b/evals/reviewer/batch-export/base/checks.py @@ -0,0 +1,24 @@ +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from courier import export +from receiver import Receiver + + +class ExportTests(unittest.TestCase): + def test_export_and_repeat(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = root / "source.txt" + source.write_text("red\ngreen\nblue\n", encoding="utf-8") + receiver = Receiver(root / "receiver.sqlite3") + export(source, receiver) + export(source, receiver) + self.assertEqual(receiver.records(str(source.resolve())), ["red", "green", "blue"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/reviewer/batch-export/base/courier.py b/evals/reviewer/batch-export/base/courier.py new file mode 100644 index 0000000..695ea54 --- /dev/null +++ b/evals/reviewer/batch-export/base/courier.py @@ -0,0 +1,20 @@ +from __future__ import annotations + +import argparse +from pathlib import Path + +from receiver import Receiver + + +def export(source: Path, receiver: Receiver) -> None: + records = source.read_text(encoding="utf-8").splitlines() + for offset in range(0, len(records), 2): + receiver.deliver(str(source.resolve()), offset, records[offset : offset + 2]) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("source", type=Path) + parser.add_argument("receiver", type=Path) + args = parser.parse_args() + export(args.source, Receiver(args.receiver)) diff --git a/evals/reviewer/batch-export/base/receiver.py b/evals/reviewer/batch-export/base/receiver.py new file mode 100644 index 0000000..68c172c --- /dev/null +++ b/evals/reviewer/batch-export/base/receiver.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +import json +import sqlite3 +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from pathlib import Path + + +class Receiver: + def __init__(self, path: Path): + self.path = path + with sqlite3.connect(path) as db: + db.execute( + "CREATE TABLE IF NOT EXISTS deliveries " + "(source TEXT, offset INTEGER, payload TEXT, PRIMARY KEY (source, offset))" + ) + + def deliver(self, source: str, offset: int, records: list[str]) -> None: + payload = json.dumps(records) + with sqlite3.connect(self.path) as db: + existing = db.execute( + "SELECT payload FROM deliveries WHERE source = ? AND offset = ?", (source, offset) + ).fetchone() + if existing is not None: + if existing[0] != payload: + raise ValueError("request reused with different records") + return + db.execute("INSERT INTO deliveries VALUES (?, ?, ?)", (source, offset, payload)) + + def records(self, source: str) -> list[str]: + with sqlite3.connect(self.path) as db: + batches = db.execute("SELECT payload FROM deliveries WHERE source = ? ORDER BY offset", (source,)) + return [record for (payload,) in batches for record in json.loads(payload)] diff --git a/evals/reviewer/batch-export/evaluator/grading-notes.md b/evals/reviewer/batch-export/evaluator/grading-notes.md new file mode 100644 index 0000000..7fb9f84 --- /dev/null +++ b/evals/reviewer/batch-export/evaluator/grading-notes.md @@ -0,0 +1,18 @@ +# Withheld evaluator notes + +The proposed expectations below need maintainer judgment. Do not give the reviewer this directory, the parent evaluation documents, hints from a prior run, or defect names. Other evidenced findings are admissible. Do not require a particular number of findings, exact wording, a mandatory investigation sequence, or a complete replacement implementation. + +| Decision | Independent contract and witness | Evidence to look for | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Remove the global content ledger | Each source line is a record. `red, green, red, blue` contains four occurrences, but head rejects the second batch. | Trace the digest set from creation through persistence and consumption. Determine whether equal content denotes the same delivery. Removing hashing while retaining content rejection does not fix the behavior. | +| Save progress after successful delivery | A rerun must complete an interrupted job. If the first receiver call rejects before commit, head has already advanced the checkpoint and retry yields only `blue`. | Follow the ordinary command through checkpoint save, delivery, and a new exporter invocation. Distinguish receiver acceptance from caller acknowledgement. Passing the supplied success test is insufficient. | +| Retain receiver request identity and idempotent acknowledgement | A connection can fail after a real commit. With correct checkpoint ordering, retry submits the same source/offset request again. | Show that the simplified exporter still finishes with exactly the original records after the lost acknowledgement. The rejection mutation makes this scenario fail while the supplied suite stays green. This is a justified boundary, not redundant content deduplication. | +| Retain useful success coverage and add targeted recovery coverage | The supplied test checks the real exporter and receiver, including repeat invocation. Its expected record list comes from the input. | It survives the simpler alternative. A useful recommendation strengthens failure coverage without deleting the success test merely because it misses recovery failures. | + +Review the observable investigation, not a polished explanation alone. Did the reviewer independently notice an unexplained mechanism or understanding gap, trace its actual consumers, choose a discriminating experiment or equivalent proof, and update its conclusion using the result? Did it distinguish the removable ledger from required retry behavior and follow the implications of changing checkpoint ordering? If it initially suspects receiver duplication, does later evidence make it retain that boundary? A reviewer need not make an initial mistake to demonstrate sound investigation. + +The root reviewer remains responsible for combining any child findings, reconciling these interacting paths, and stating unresolved scope. Child completion or green baseline tests do not answer the open questions. Evaluate whether the final evidence supports the conclusions; do not grade internal reasoning text, use of these exact tools, or the presence of a named checklist. + +`simpler.patch` is a bounded alternative against the submitted head, not a second maintained implementation. `reject-retry.patch` is a deliberately invalid mutation of the retained receiver boundary. The probe expectations are hand-derived from the source and receiver contracts, and the injected exceptions surround actual SQLite writes. The tests do not reconstruct checkpoint state on behalf of production code. + +No conclusion about hashes, identifiers, persistence, or adapters follows beyond these callers and contracts. An implementation using any of them for a demonstrated responsibility can be valid. Scope is one sequential exporter, unchanged source bytes, and this local receiver. Real transport compatibility and crash durability remain outside the fixture. diff --git a/evals/reviewer/batch-export/evaluator/probe.py b/evals/reviewer/batch-export/evaluator/probe.py new file mode 100644 index 0000000..7dc139c --- /dev/null +++ b/evals/reviewer/batch-export/evaluator/probe.py @@ -0,0 +1,69 @@ +"""Evaluator-only behavioral witnesses; never include these in reviewer input.""" + +from __future__ import annotations + +import sys +import tempfile +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(sys.argv.pop(1)).resolve())) + +from checkpoint import CheckpointStore +from courier import export +from receiver import Receiver + + +class RejectBeforeCommit(Receiver): + def deliver(self, source: str, offset: int, records: list[str]) -> None: + raise ConnectionError("connection failed before acceptance") + + +class LoseAcknowledgement(Receiver): + def deliver(self, source: str, offset: int, records: list[str]) -> None: + super().deliver(source, offset, records) + raise ConnectionError("connection failed after acceptance") + + +class ExportEvidence(unittest.TestCase): + def setUp(self): + directory = tempfile.TemporaryDirectory() + self.addCleanup(directory.cleanup) + self.root = Path(directory.name) + self.source = self.root / "source.txt" + self.receiver_path = self.root / "receiver.sqlite3" + self.checkpoint_path = self.root / "checkpoint.json" + + def write_source(self, records: list[str]) -> None: + self.source.write_text("\n".join(records) + "\n", encoding="utf-8") + + def run_export(self, receiver_type=Receiver) -> None: + export(self.source, receiver_type(self.receiver_path), CheckpointStore(self.checkpoint_path)) + + def received(self) -> list[str]: + return Receiver(self.receiver_path).records(str(self.source.resolve())) + + def test_each_input_occurrence_is_delivered(self): + self.write_source(["red", "green", "red", "blue"]) + self.run_export() + self.assertEqual(self.received(), ["red", "green", "red", "blue"]) + + def test_retry_after_rejection_preserves_all_records(self): + self.write_source(["red", "green", "blue"]) + with self.assertRaises(ConnectionError): + self.run_export(RejectBeforeCommit) + self.assertEqual(self.received(), []) + self.run_export() + self.assertEqual(self.received(), ["red", "green", "blue"]) + + def test_retry_after_lost_acknowledgement_delivers_once(self): + self.write_source(["red", "green", "blue"]) + with self.assertRaises(ConnectionError): + self.run_export(LoseAcknowledgement) + self.assertEqual(self.received(), ["red", "green"]) + self.run_export() + self.assertEqual(self.received(), ["red", "green", "blue"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/reviewer/batch-export/evaluator/reject-retry.patch b/evals/reviewer/batch-export/evaluator/reject-retry.patch new file mode 100644 index 0000000..cfe09c4 --- /dev/null +++ b/evals/reviewer/batch-export/evaluator/reject-retry.patch @@ -0,0 +1,11 @@ +--- a/receiver.py ++++ b/receiver.py +@@ -26,7 +26,7 @@ + if existing is not None: + if existing[0] != payload: + raise ValueError("request reused with different records") +- return ++ raise ValueError("request already delivered") + db.execute("INSERT INTO deliveries VALUES (?, ?, ?)", (source, offset, payload)) + + def records(self, source: str) -> list[str]: diff --git a/evals/reviewer/batch-export/evaluator/simpler.patch b/evals/reviewer/batch-export/evaluator/simpler.patch new file mode 100644 index 0000000..661f018 --- /dev/null +++ b/evals/reviewer/batch-export/evaluator/simpler.patch @@ -0,0 +1,38 @@ +--- a/courier.py ++++ b/courier.py +@@ -1,7 +1,6 @@ + from __future__ import annotations + + import argparse +-import hashlib + from pathlib import Path + + from checkpoint import CheckpointStore +@@ -11,15 +10,10 @@ + def export(source: Path, receiver: Receiver, checkpoints: CheckpointStore) -> None: + records = source.read_text(encoding="utf-8").splitlines() + state = checkpoints.load() +- seen = set(state["seen"]) + for offset in range(state["offset"], len(records), 2): + batch = records[offset : offset + 2] +- identifiers = {hashlib.sha256(record.encode("utf-8")).hexdigest() for record in batch} +- if seen & identifiers: +- raise ValueError("record already exported") +- seen.update(identifiers) +- checkpoints.save({"offset": offset + len(batch), "seen": sorted(seen)}) + receiver.deliver(str(source.resolve()), offset, batch) ++ checkpoints.save({"offset": offset + len(batch)}) + + + if __name__ == "__main__": +--- a/checkpoint.py ++++ b/checkpoint.py +@@ -13,7 +13,7 @@ + + def load(self) -> dict: + if not self.path.exists(): +- return {"offset": 0, "seen": []} ++ return {"offset": 0} + return json.loads(self.path.read_text(encoding="utf-8")) + + def save(self, state: dict) -> None: diff --git a/evals/reviewer/batch-export/head/README.md b/evals/reviewer/batch-export/head/README.md new file mode 100644 index 0000000..ddf6631 --- /dev/null +++ b/evals/reviewer/batch-export/head/README.md @@ -0,0 +1,7 @@ +# Batch export + +The command copies a UTF-8 text file to a receiver database. Each input line is one record, and records retain their input order. An input file is stable during a job and its retries; its absolute path identifies the source. Batches contain two records. + +Run `python courier.py source.txt receiver.sqlite3 checkpoint.json`. Keep the checkpoint with this source and receiver. Repeating a command against the same receiver is supported. The receiver stores each request by source and starting offset. Repeating that request with the same payload acknowledges the existing delivery; a different payload for that request is rejected. A connection error does not establish whether a request committed. + +Run the repository tests with `python -B checks.py -v`. diff --git a/evals/reviewer/batch-export/head/checkpoint.py b/evals/reviewer/batch-export/head/checkpoint.py new file mode 100644 index 0000000..b3b0aea --- /dev/null +++ b/evals/reviewer/batch-export/head/checkpoint.py @@ -0,0 +1,22 @@ +from __future__ import annotations + +import json +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from pathlib import Path + + +class CheckpointStore: + def __init__(self, path: Path): + self.path = path + + def load(self) -> dict: + if not self.path.exists(): + return {"offset": 0, "seen": []} + return json.loads(self.path.read_text(encoding="utf-8")) + + def save(self, state: dict) -> None: + temporary = self.path.with_suffix(".tmp") + temporary.write_text(json.dumps(state), encoding="utf-8") + temporary.replace(self.path) diff --git a/evals/reviewer/batch-export/head/checks.py b/evals/reviewer/batch-export/head/checks.py new file mode 100644 index 0000000..6e63099 --- /dev/null +++ b/evals/reviewer/batch-export/head/checks.py @@ -0,0 +1,26 @@ +from __future__ import annotations + +import tempfile +import unittest +from pathlib import Path + +from checkpoint import CheckpointStore +from courier import export +from receiver import Receiver + + +class ExportTests(unittest.TestCase): + def test_export_and_repeat(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + source = root / "source.txt" + source.write_text("red\ngreen\nblue\n", encoding="utf-8") + receiver = Receiver(root / "receiver.sqlite3") + checkpoints = CheckpointStore(root / "checkpoint.json") + export(source, receiver, checkpoints) + export(source, receiver, checkpoints) + self.assertEqual(receiver.records(str(source.resolve())), ["red", "green", "blue"]) + + +if __name__ == "__main__": + unittest.main() diff --git a/evals/reviewer/batch-export/head/courier.py b/evals/reviewer/batch-export/head/courier.py new file mode 100644 index 0000000..33bcaa8 --- /dev/null +++ b/evals/reviewer/batch-export/head/courier.py @@ -0,0 +1,31 @@ +from __future__ import annotations + +import argparse +import hashlib +from pathlib import Path + +from checkpoint import CheckpointStore +from receiver import Receiver + + +def export(source: Path, receiver: Receiver, checkpoints: CheckpointStore) -> None: + records = source.read_text(encoding="utf-8").splitlines() + state = checkpoints.load() + seen = set(state["seen"]) + for offset in range(state["offset"], len(records), 2): + batch = records[offset : offset + 2] + identifiers = {hashlib.sha256(record.encode("utf-8")).hexdigest() for record in batch} + if seen & identifiers: + raise ValueError("record already exported") + seen.update(identifiers) + checkpoints.save({"offset": offset + len(batch), "seen": sorted(seen)}) + receiver.deliver(str(source.resolve()), offset, batch) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + parser.add_argument("source", type=Path) + parser.add_argument("receiver", type=Path) + parser.add_argument("checkpoint", type=Path) + args = parser.parse_args() + export(args.source, Receiver(args.receiver), CheckpointStore(args.checkpoint)) diff --git a/evals/reviewer/batch-export/head/receiver.py b/evals/reviewer/batch-export/head/receiver.py new file mode 100644 index 0000000..68c172c --- /dev/null +++ b/evals/reviewer/batch-export/head/receiver.py @@ -0,0 +1,35 @@ +from __future__ import annotations + +import json +import sqlite3 +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from pathlib import Path + + +class Receiver: + def __init__(self, path: Path): + self.path = path + with sqlite3.connect(path) as db: + db.execute( + "CREATE TABLE IF NOT EXISTS deliveries " + "(source TEXT, offset INTEGER, payload TEXT, PRIMARY KEY (source, offset))" + ) + + def deliver(self, source: str, offset: int, records: list[str]) -> None: + payload = json.dumps(records) + with sqlite3.connect(self.path) as db: + existing = db.execute( + "SELECT payload FROM deliveries WHERE source = ? AND offset = ?", (source, offset) + ).fetchone() + if existing is not None: + if existing[0] != payload: + raise ValueError("request reused with different records") + return + db.execute("INSERT INTO deliveries VALUES (?, ?, ?)", (source, offset, payload)) + + def records(self, source: str) -> list[str]: + with sqlite3.connect(self.path) as db: + batches = db.execute("SELECT payload FROM deliveries WHERE source = ? ORDER BY offset", (source,)) + return [record for (payload,) in batches for record in json.loads(payload)] diff --git a/evals/reviewer/batch-export/requirements.md b/evals/reviewer/batch-export/requirements.md new file mode 100644 index 0000000..59eae8b --- /dev/null +++ b/evals/reviewer/batch-export/requirements.md @@ -0,0 +1,5 @@ +# Requested change + +Add checkpoint-based continuation to the existing batch export command. An operator should be able to rerun an interrupted job using its checkpoint. Keep the existing source and receiver behavior, and keep the ordinary command usable without a separate service. + +Review the proposed change using the base repository, head repository, and their tests. Report actionable findings with evidence, examined alternatives, and any unresolved limits. Do not modify the submitted implementation or publish externally; use a disposable copy for experiments. diff --git a/evals/reviewer/grading-notes.md b/evals/reviewer/grading-notes.md index 1ec48cc..1be850a 100644 --- a/evals/reviewer/grading-notes.md +++ b/evals/reviewer/grading-notes.md @@ -4,5 +4,6 @@ - **Latest request:** sequential immediate responses never create the reported race. Start old and new requests, resolve new first, then old; assert that new results remain. A mock transport is useful if real request coordination executes. Merely asserting that a generation counter increments would not prove the behavior. - **Cached size:** a derived `len(items)` can remove the maintained count and synchronization points. Preserve the public `size()` contract and exercise mixed pushes and pops. The existing size test is useful under that refactor; do not delete it simply because a simpler implementation exists. - **Adapter:** the wrapper has a required protocol responsibility. The mock assertion targets the actual external call contract and its units. Removing the adapter or rejecting the test just because it asserts a call would be a false positive. A separate real transport check may still be needed for transport compatibility. +- **Batch export:** use the separate [evaluator notes](batch-export/evaluator/grading-notes.md). Do not give that directory, its patches, or this grading file to a reviewer. The neutral request and repository contracts intentionally leave the choice of investigation to the reviewer. These expectations are author-proposed, not maintainer-confirmed results. An alternative finding requires a reachable trigger and independent evidence. Mark an unavailable execution boundary honestly; do not infer a pass from plausible code. diff --git a/packages/nyanpasu-github-reviewer/README.md b/packages/nyanpasu-github-reviewer/README.md index 4a2f78d..2238f4c 100644 --- a/packages/nyanpasu-github-reviewer/README.md +++ b/packages/nyanpasu-github-reviewer/README.md @@ -58,6 +58,8 @@ Reviews use the repository's configured memory audience; the default reads publi The fixed reviewer role is maintained in [reviewer.md](src/nyanpasu_github_reviewer/instructions/reviewer.md). It binds the PR identity, review boundaries, continuation rules, language, and skill usage to the agent session. Each execution renders its disclosure footer from the task’s resolved `model` and `reasoning` target. If the model is unset, the footer names Codex or Claude Code without guessing a model. [review-output.md](src/nyanpasu_github_reviewer/instructions/review-output.md) is the reference for priorities, suggestions, review decisions, and footer placement. Tool procedures come from the `github-conversation` and `gh-slate` skills. +The main reviewer builds and revises its understanding of required behavior, chooses investigations for material uncertainties, challenges returned evidence, and follows confirmed causes into related paths. It records questions and conclusions in the existing review plan and checkpoints. On continuation and before publication it rechecks requirements as well as code: `review-verify` protects the comparison range, not the meaning of an edited request. These are agent judgments; the runtime owns scope, input isolation and evidence lifecycle. + Every execution prepares one short user message containing the target head, worktree, publication mode, the current model's disclosure footer, and trigger summaries or request links. Supplying the footer in each turn also updates the declaration when an existing session resumes with a different model. The previous task head is a navigation hint, not proof that a review was completed. Submitted GitHub reviews and published threads remain the evidence for prior review coverage. When publication is enabled, the agent resumes unfinished review and publication work before applying the silence rule. It may edit or replace its own unpublished review drafts after preserving their content and revalidating findings against the target head. Published discussions remain canonical. A review is complete only after its required publication is verified on GitHub; a completed task or updated dashboard alone does not prove publication succeeded. diff --git a/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/review-planning.md b/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/review-planning.md index cc8f2cc..8c5723a 100644 --- a/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/review-planning.md +++ b/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/review-planning.md @@ -2,7 +2,7 @@ Read this after scope-review.md and before implementation review. Complete the program-validated repository admission plan before deciding independent design or dispatching any child. In a continuation you may already know the implementation; disclose that and give a fresh child the original requirements, never your previous design conclusions. -Record a brief decision in `review-plan.json` and the native conversation: `run`, `skip`, or `reuse`, the reason, source head, and requirement sources. The agent makes this decision; the service owns lifecycle, version pinning, scheduling, and cleanup. +Record a brief decision in `review-plan.json` and the native conversation: `run`, `skip`, or `reuse`, the reason, source head, and requirement sources. Explain which uncertainty independent work would resolve and what evidence could change the decision. Update material questions as investigation progresses; do not create tasks merely to fill stages. The agent makes this decision; the service owns lifecycle, version pinning, scheduling, and cleanup. - Run independent design for state/lifecycle changes, new abstractions, changed boundaries/protocols, complex algorithms, or unclear necessity of added machinery. Explicit requests for independent comparison require this stage. - Skip independent design for mechanical edits or a narrow correction to an already validated design, with a concrete reason. This does not skip the necessity audit of substantive production and test changes. Reuse earlier audit conclusions only for unchanged responsibilities with recorded scope, alternatives and retention evidence; earlier bug fixes or a completed task alone are insufficient. If that record is absent, include the outstanding full-PR scope even on an incremental review. @@ -48,6 +48,6 @@ As soon as general review is complete, save its checked scope, findings, command Only after delivering that checkpoint, use `await` for unfinished children and end the turn with the initial result, confirmed dashboard URL, and pending scope. The service resumes the same task and workspace with terminal child states and frozen evidence. The task remains waiting, not completed; the parent and all descendants share one root concurrency slot throughout. A child finishing while the parent is still reviewing does not interrupt that turn. -On resume, inspect current PR state and head before further experiments or writes. If closed or superseded, cancel obsolete children, retain the old-head evidence, and end this round without publishing it as current; normal event handling prepares the current head. Otherwise verify the general checkpoint and its publication, consume the child evidence, read `design-comparison.md`, and reconcile the deep findings. Do not rerun or republish unchanged general findings. Failure, cancellation, missing evidence, or contaminated inputs mean incomplete deep review, never a successful comparison; preserve the completed general stage. A completed child is an alternative hypothesis, not a specification. +On resume, inspect current PR state, head, requirements and relevant discussions before further experiments or writes. Compare current requirements and constraints with the plan and frozen child inputs even when the code inventory is unchanged. Reinvestigate affected conclusions and dispatch a fresh reference with revised requirements when needed; retain still-applicable evidence with its original assumptions. If closed or superseded, cancel obsolete children, retain the old-head evidence, and end this round without publishing it as current; normal event handling prepares the current head. Otherwise verify the general checkpoint and its publication, consume the child evidence, read `design-comparison.md`, and reconcile the deep findings. Do not rerun or republish unchanged general findings. Failure, cancellation, missing evidence, or contaminated inputs mean incomplete deep review, never a successful comparison; preserve the completed general stage. A completed child is an alternative hypothesis, not a specification. Only the root reviewer publishes. Verify live PR head, the recorded base assumptions, CI, unresolved coverage gaps, and canonical findings before publication. If the head changes, retain the evidence under its original identity and prepare a new review; do not relabel old findings or artifacts as current. diff --git a/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/reviewer.md b/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/reviewer.md index a839024..f327cc4 100644 --- a/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/reviewer.md +++ b/packages/nyanpasu-github-reviewer/src/nyanpasu_github_reviewer/instructions/reviewer.md @@ -12,12 +12,24 @@ Apply the continuation/silence rules below. For substantive review, read `$scope - Later turns bring new events or requests for this same PR. Continue the existing review and discussions. - Read the PR description, original requirements, relevant timeline, existing threads and CI. After scope triage, review the admitted diff fully unless reliable prior coverage supports incremental review. Keep deferred scope visible and examine only what is needed to decide its placement or verify an acceptance claim. - Reuse earlier analysis as a starting point and verify it against current code and GitHub state. A previous task head is only a navigation hint; task completion does not prove that review or publication was completed. Read missing history through the available tools and resume unfinished work before applying the silence rule. -- Compare the current pinned `inventory_id` with the dashboard source before reusing coverage or applying silence. An unchanged head is insufficient after retargeting or an upstream rewrite. If the inventory is unchanged and prior review/publication is verified complete, a base-only update needs no new deep review or public update. +- Compare the current pinned `inventory_id` with the dashboard source before reusing coverage or applying silence. Also reread current requirements and relevant discussions against the sources recorded in the plan; an unchanged head or inventory does not establish unchanged requirements. Reopen conclusions affected by changed behavior or constraints, preserving evidence that still applies. An unchanged inventory and requirements with verified complete review/publication need no new deep review for a base-only update. - For a stacked PR, review the change against its direct base and use the lower layers as context. Attribute findings to the layer that introduces them and reuse its canonical discussion; do not duplicate inherited findings on every layer. Approval covers this PR's reviewed scope, not approval or merge readiness of the whole stack. Do not rebase, push or merge the stack. - For earlier findings, distinguish resolved, partially resolved, unresolved, and superseded. The original published thread remains the canonical discussion for the same semantic issue, even if its lines moved or the wording changed. Unpublished drafts are work in progress; manage them according to the output reference. - Answer explicit requests and relevant replies to your own threads. Once the dashboard exists and earlier review and publication are verified complete, automatic follow-ups with no new evidence, finding status, decision, or request require no GitHub-visible update, including dashboard updates. Do not post standalone text acknowledgements or repeat unchanged findings. - For a new relevant human comment, review, or request that needs acknowledgement, choose one fitting reaction on that item using the github-conversation skill. Check whether this account already reacted before writing. Do not react to your own messages or add reactions on unchanged automatic follow-ups. +## Drive the investigation + +Own the understanding of the change, not just completion of review stages. Build a working account of the required observable behavior, its sources, and how existing callers and boundaries provide it; revise that account as you inspect the implementation. Separate established contracts, author claims, and your assumptions. Review criteria come from skills and repository contracts; you decide where understanding is missing and what evidence would settle it. + +Treat an unexplained mechanism, an inconsistent neighboring path, or a result that conflicts with your account as a question to investigate. Ask what requires the behavior, who consumes the state, or what would be lost by removing it when those questions expose a real uncertainty. Do not accept the author's decomposition as the only way to frame the problem, or turn unfamiliarity into a finding. State the plausible consequence and the evidence that could support or refute it. + +Choose the next action to resolve that uncertainty: trace callers and consumers, compare related paths and their requirements, ask a scoped expert, obtain an isolated independent design, or run a small experiment through the responsible entry point. Use the existing workflows below. Before an experiment, identify which outcomes would change the review decision; choose the smallest input and time bound that distinguish them. Close a question once the evidence supports its conclusion; additional investigation needs a remaining uncertainty, conflicting evidence or a new change. Delegate a question, relevant context and owned scope, then assess the returned evidence yourself. Respect the independent designer's input isolation. + +When evidence contradicts an assumption, revise or discard the claim and follow the newly exposed gap. After confirming a cause, check related producers, consumers or analogous paths within the assigned scope; explain why the conclusion does or does not transfer. Keep material questions, evidence, rejected hypotheses and remaining gaps in the existing review plan and general/deep checkpoints, not a separate checklist. Deliver the general checkpoint while deeper questions remain under investigation. + +Finish when the assigned scope has supported conclusions and material questions are resolved. Pursue available evidence before leaving a question incomplete; identify the missing evidence and the limitation preventing further investigation. Passing CI, a completed child, agreement between designs, or fixing known findings does not establish completion. Neither a finding quota nor exhaustive speculative investigation is required. + ## Independent design and test evidence Read `$review_planning` before inspecting the implementation. Record run/skip/reuse with a reason and launch managed Nyanpasu subtasks when independent work is needed. Continue general review while they run, publish its verified findings and scope on the dashboard before waiting, then integrate deep evidence in a later turn. Audit test value from failure models and observable behavior. The linked workflow contains the design, test-audit, and comparison prompts. @@ -30,10 +42,10 @@ At the start of a review and before final publication, call the supplied task-co ## Review quality -- Review the pinned range supplied in the current turn. Check that the worktree and review diff match its head and merge-base. Immediately before review publication, use the supplied task-control command with `{"action":"review-verify"}`. It verifies current eligibility and the full comparison range. If it fails, do not publish conclusions or line comments from this run; report the changed range and continue at the current range in a new review run. +- Review the pinned range supplied in the current turn. Check that the worktree and review diff match its head and merge-base. Immediately before review publication, reread current requirements and relevant discussions, then use the supplied task-control command with `{"action":"review-verify"}`. It verifies current eligibility and the full comparison range, not unchanged requirements. Reassess affected conclusions if requirements changed. If range verification fails, do not publish conclusions or line comments from this run; report the changed range and continue at the current range in a new review run. - For reviews spanning multiple subsystems, delegate related module groups to subagents for deep review and cover the remaining changes yourself. Give each the PR context, target head, and owned scope; let them trace complete paths across module boundaries. Review small, cohesive changes directly. - For each change, trace real callers, data flow, and prior behavior beyond the diff; check relevant tests, existing configs and launch scripts, and contracts against pinned dependency versions. Try to disprove candidate findings before posting; report only actionable, evidenced findings, including relevant PR title/body issues. -- Ask subagents for concise findings, checked paths, evidence, and gaps even when no issues are found. Reconcile their results, inspect cross-group interactions, and close coverage gaps before approval. You retain final verification and publication; fixing earlier findings alone does not complete the review. +- Ask subagents for concise conclusions, checked paths, evidence, rejected candidates, and gaps even when no issues are found. Reconcile their results and inspect cross-group interactions; completion of their assignments does not close your unanswered questions. You retain final verification and publication. - Put new findings on changed diff lines when possible, using the coordinates from `review-start`. For an existing finding, reply only with new evidence, a changed recommendation, a correction, or an answer to a new request. ## Tools and output From b38d8b8e41c65c3908efc0e81b4af64c97fc4328 Mon Sep 17 00:00:00 2001 From: Nyakku Shigure Date: Sun, 4 Oct 2026 19:47:30 +0800 Subject: [PATCH 2/2] :white_check_mark: test: calibrate reviewer evidence grading Distinguish cross-batch regression inputs from ineffective same-batch cases, and treat cross-source checkpoint guards and CLI compatibility as contract questions unless requirements support them. Co-authored-by: Codex --- evals/reviewer/batch-export/evaluator/grading-notes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/evals/reviewer/batch-export/evaluator/grading-notes.md b/evals/reviewer/batch-export/evaluator/grading-notes.md index 7fb9f84..f984cbf 100644 --- a/evals/reviewer/batch-export/evaluator/grading-notes.md +++ b/evals/reviewer/batch-export/evaluator/grading-notes.md @@ -11,6 +11,10 @@ The proposed expectations below need maintainer judgment. Do not give the review Review the observable investigation, not a polished explanation alone. Did the reviewer independently notice an unexplained mechanism or understanding gap, trace its actual consumers, choose a discriminating experiment or equivalent proof, and update its conclusion using the result? Did it distinguish the removable ledger from required retry behavior and follow the implications of changing checkpoint ordering? If it initially suspects receiver duplication, does later evidence make it retain that boundary? A reviewer need not make an initial mistake to demonstrate sound investigation. +Check proposed regression inputs as well as executed experiments. Two identical lines fit in one batch and do not trigger the ledger rejection; a repeat must cross a batch boundary. A valid experiment followed by a weaker test suggestion is still a gap. For the retained receiver boundary, distinguish tracing the acknowledgement branch from actually exercising a real commit followed by a lost acknowledgement and retry; do not claim an experiment that was not run. + +The README requires keeping the checkpoint with its source and receiver. Demonstrating truncation after cross-source checkpoint reuse does not establish that this PR must add identity fields or defend against hand-edited state; that is a proposed responsibility beyond the stated contract. Likewise, the original request does not explicitly freeze CLI arity: observing the old invocation fail supports a compatibility question, not by itself a confirmed requirement violation. Judge any proposed default checkpoint path against both source and receiver identity before treating it as a validated alternative. + The root reviewer remains responsible for combining any child findings, reconciling these interacting paths, and stating unresolved scope. Child completion or green baseline tests do not answer the open questions. Evaluate whether the final evidence supports the conclusions; do not grade internal reasoning text, use of these exact tools, or the presence of a named checklist. `simpler.patch` is a bounded alternative against the submitted head, not a second maintained implementation. `reject-retry.patch` is a deliberately invalid mutation of the retained receiver boundary. The probe expectations are hand-derived from the source and receiver contracts, and the injected exceptions surround actual SQLite writes. The tests do not reconstruct checkpoint state on behalf of production code.