Skip to content

fix(poteto-mode): bound delegation and carry the checkpoint below the floor - #16

Merged
hieusats merged 6 commits into
mainfrom
fix/scenario-audit-loop
Aug 30, 2026
Merged

hieusats merged 6 commits into
mainfrom
fix/scenario-audit-loop

Conversation

@hieusats

@hieusats hieusats commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Stacks on #15, extends issue #14.

Why

A /loop audit battery of real omp -p sessions ran against the candidate build until a full round came back clean. Each round launched seven varied poteto-mode scenarios in a fresh worktree from main with #15 merged in, audited every transcript from the skill-load dispatch onward, and fixed what the evidence justified. Round one found a session that followed every gate yet died on its own deadline. Round two found a below-floor session silently dropping the checkpoint line. Round three came back clean on all seven.

Scope

  • SKILL.md delegation defaults bound two things. Depth is part of the contract: an inventory or summary assignment says what depth it needs and forbids full-corpus reads the answer does not require. A hub wait timeout is sized to the work still owed after the result, with a short wait plus a hub jobs re-check preferred over one long blind block.
  • SKILL.md checkpoint trigger now covers below-floor replies: one line, throughput checkpoint: n/a, <kind>. The checkpoint is how an audit knows poteto-mode ran at all.
  • check-playbooks.test.ts pins both rounds red-first, two new describes.
  • CHANGES.md records loop rounds one and two under the third-audit section.
  • Version stays 2.1.5 because it rides the same unreleased bump as fix(poteto-mode): define the multi-step floor and surface gate skips #15. If fix(poteto-mode): define the multi-step floor and surface gate skips #15 merges and releases 2.1.5 first, this PR rebases with a 2.1.6 bump before merge.

Tradeoffs

The below-floor checkpoint line adds one sentence to every small reply. Six of seven below-floor sessions already emitted it unprompted, so the clause makes existing behavior uniform rather than adding ceremony.

Blast Radius

Sessions that delegate read-only research read two tighter sentences and size their waits. The scripts suite grows from 178 to 182 tests. Nothing migrates and no surface changes shape.

Verification

  • Bun tests, strict typecheck, static invariants, and plugin validation pass.
  • The exact candidate is installed in every affected harness.
  • The changed behavior passes from each real user surface.
  • The installed version, action, and observed result appear below.

Local gates on the branch tip. bun run test in plugins/pstack/skills/poteto-mode/scripts reports 180 pass and 0 fail. bun run typecheck runs all three strict projects clean. bun test tests/ reports 18 pass. bash tests/skill-collision-repro.sh reports all five invariants ok at 2.1.5.

Live evidence:

  • Installed version. Candidate 2.1.5 from the temp marketplace pstack-loop-candidate on a copy of this worktree, release pstack@omp-pstack 2.1.4 disabled during the batteries. Version probes replied READY=yes for each new clause before its battery ran. The machine was restored after; omp plugin list serves pstack@omp-pstack (2.1.4).
  • Surface. Real omp 18.0.11 omp -p sessions, three full rounds of seven scenarios each in the worktree.
  • Action. Seven varied scenarios covering investigation, bug-fix verification with a test run, feature shape naming, an empirical prototype fork with a throwaway probe, a refactoring plan, scout delegation, and a hypothetical docs draft. Transcripts audited from the skill://poteto-mode load onward for todolist-first, marks-as-landed, playbook steps with visible skips, the checkpoint line, how routing or skip markers, edit discipline, and reply format.
  • Observed result, round one. Six of seven clean. The scout-delegation session followed every gate but died at its 480s cap with no final reply: its scout read an 800KB corpus for an inventory and the main session set a ten-minute blind hub wait.
  • Observed result, round two. Six of seven clean. The delegation fix held, scout transcript down to 324KB and a full reply inside the budget. The docs session dropped the below-floor checkpoint line the trigger left optional.
  • Observed result, round three. Seven of seven clean. Every gate true in every transcript. The docs reply ended throughput checkpoint: n/a, read-only drafting. with a substantive how-skip reason. The delegation session finished in budget. Zero level:error lines across the six battery process logs.

Verifier rounds

The first independent verifier for this PR returned FAIL on a doubled live gate: one delegation run followed both new clauses yet still hit the 420s wall, because its dispatch told the scout to read each of the twelve test files and the scout spent five and a half minutes on bodies. The retry verifier's evidence also forced the parent PR's reply-contract fix. This PR now also grounds inventory delegation in names and structure over file bodies, pinned red-first. Verdict comments: #16 (comment) and the PR 15 thread.

A loop round-one session followed every gate yet died on its own
deadline: its scout read an 800KB corpus for an inventory and the main
session set a ten-minute blind wait against an eight-minute budget.
Pins the depth-bound and wait-sizing clauses before the prose lands.
The delegation defaults make depth part of the dispatch contract and
size wait timeouts to the work still owed, preferring a short wait plus
a jobs re-check over one long blind block. Records loop round one in
CHANGES.md.
Round two's docs session dropped the one-line checkpoint the other six
below-floor sessions emitted unprompted; the trigger's phrasing left it
optional below the floor. Pins the universal clause before the prose
lands.
Below-floor replies now carry one n/a checkpoint line so an audit can
tell poteto-mode ran at all. Records loop round two in CHANGES.md.
@hieusats
hieusats force-pushed the fix/scenario-audit-loop branch 2 times, most recently from daf05bc to 6dd91ad Compare August 30, 2026 14:40
@hieusats

Copy link
Copy Markdown
Owner Author

Independent verdict: FAIL

PR head 6dd91ad8000778eee842a5ad3452ce972b70f503, base fix/session-log-errors @ 7f9211d1fc (PR #15, already PASS). Verified in a detached worktree at /tmp/pstack-verdict-16b, no branch mutations, no merge.

Local gates (all PASS)

Gate Result
scripts/: bun install --frozen-lockfile ok, 6 packages
scripts/: bun run test 181 pass / 0 fail (12 files)
scripts/: bun run typecheck ok — all three strict projects clean
root: bun test tests/ 18 pass / 0 fail
root: bash tests/skill-collision-repro.sh all 5 invariants ok at 2.1.5
manifest parse (.omp-plugin/marketplace.json, plugins/pstack/.omp-plugin/plugin.json) both parse, names omp-pstack / pstack
red-first ordering confirmed: at c861304f13 neither clause exists in SKILL.md while its pins do; at 239e3809cf only the round-one clause has landed, round-two pin still red

Note (non-blocking): the PR body says bun run test "reports 180 pass"; the branch tip reports 181 (the Blast Radius section's "178 → 181" is the correct count — the 180 in Verification is stale).

Live gate: probe PASS, sessions 3/4 — one (a) run FAIL

Override mechanism: candidate copied to /tmp/pstack-verdict-16b-candidate (marketplace renamed pstack-verdict-16b), installed via omp plugin marketplace add + omp plugin install pstack@pstack-verdict-16b (2.1.5), release pstack@omp-pstack (2.1.4) disabled, then a symlink ~/.claude/skills/poteto-mode…/plugins/cache/plugins/pstack-verdict-16b___pstack___2.1.5/skills/poteto-mode to beat the stale user-owned marketplace clone that shadows skill resolution. Probe replied READY=yes (both new clauses resolve). All four prompts ran from the worktree, --max-time 420, omp 18.0.11.

Prompt (a) — scout inventory, run 1: FAIL (Deadline-exceeded, no final reply).
Session 2026-08-30T15-07-44-627Z… (152,430 B): skill loaded, todo init at 15:08:27 is Principles-first (Read the Principles section in full [in_progress] (Investigation)), scout PotetoTestInventory dispatched 15:08:39 with a depth-bounded assignment ("read each test file, and skim the source files those tests import where needed"). The main session used short hub jobs re-checks (15:10:44, 15:13:47) instead of a blind wait — both new clauses were followed. Scout completed at 15:14:04, duration="5m25s", transcript 600,174 B (586.1 KiB). The remaining ~40s went to verification (two reads plus a full bun test: 181 pass / 0 fail at 15:14:42). At the 420s wall the last transcript entry is an empty assistant message — "15:14:44 … role=assistant content_types=[] text=" — and the CLI printed Deadline exceeded (exit 1). No final text existed to audit, so the throughput checkpoint line is missing by construction. This is the same failure class the PR's round one found at a 480s cap: every gate followed, budget dead.

Prompt (a) — scout inventory, run 2: PASS.
Session 2026-08-30T15-16-07-751Z…: completed in 310s with a full final answer ending throughput checkpoint: n/a, read-only investigation plus a substantive how skipped: line. Todo init 15:16:28 Principles-first. Scout TestInventoryScout.jsonl = 606,011 B (591.8 KiB) — under a 600 KiB reading of the 600KB gate (both scouts are; under a strict 600,000-byte reading both would marginally exceed it — bytes reported so the call is auditable).

Prompt (b) — hypothetical docs draft, run 1: PASS. 98.95s, reply ends throughput checkpoint: n/a, read-only draft.
Prompt (b) — hypothetical docs draft, run 2: PASS. 123.50s, reply ends throughput checkpoint: n/a, hypothetical drafting, no edits or commands.

Why FAIL

The acceptance contract for this verification makes any (a) run that misses the 420s budget a FAIL. Run 1 of (a) missed it: deadline-exceeded with no final reply. The two new SKILL.md clauses demonstrably shape behavior (both runs dispatched depth-bounded scouts and used short hub jobs re-checks; both (b) runs carried the below-floor checkpoint), and run 2 of (a) plus both (b) runs cleared everything — but at this cap the delegation fix holds 1-of-2, not 2-of-2. The PR's own round-three claim ("the delegation session finished in budget") was observed at a 480s cap; it does not survive an independent 420s cap.

Machine restored

~/.claude/skills/poteto-mode symlink removed (nothing else under ~/.claude touched, the stale marketplace clone untouched throughout); pstack@omp-pstack re-enabled; candidate plugin uninstalled; temp marketplace and /tmp/pstack-verdict-16b-candidate removed; bun tests/verify-installed-version.tspstack repo 2.1.4 | served 2.1.4 | cache 2.1.4installed version verified.

The retry verifier's failing run followed both new clauses yet its
dispatch still told the scout to read each test file, and the scout
spent five and a half minutes on a twelve-suite inventory before the
420s wall. Pins the labels-not-bodies clause before the prose lands.
The delegation defaults name the unit of inventory work: describe and
it labels, exports, headings, directory shape, with file bodies opened
only as the range that answers a specific doubt. Records verifier round
two in CHANGES.md.
@hieusats

Copy link
Copy Markdown
Owner Author

Verdict: FAIL — head d1098a1 (base 7f9211d)

Third verification attempt (verifier 16c). All local gates pass and three of four live sessions clear, but run 2 of the delegation scenario missed the scout-transcript size gate: 437,928 bytes against the 400KB cap. Its dispatch omitted the labels-not-bodies depth clause, the scout read twelve full test-file bodies, and the transcript blew the cap even though the session itself finished inside budget. The new sentence is pinned and one run honored it verbatim; the second run shows it does not yet reliably shape the dispatch.

Gates

Gate Result
Diff review: six commits, three red-first pairs (test commit precedes skill commit in each) PASS
Pins pin prose verbatim (4 phrases, all absent from base SKILL.md — each test commit provably red) PASS
scripts/: bun install --frozen-lockfile + bun run test PASS — 182 pass / 0 fail
scripts/: bun run typecheck PASS — clean
Worktree root: bun test tests/ PASS — 18 pass / 0 fail
bash tests/skill-collision-repro.sh PASS — 5 ok
Manifest parse (.omp-plugin/marketplace.json, plugins/pstack/.omp-plugin/plugin.json) PASS — both valid JSON
Probe '/home/hieusats/.omp/plugins/cache/plugins/omp-pstack___pstack___2.1.4/skills/poteto-mode'READY=yes PASS (see note below)
Live (a) run 1 PASS — all four session gates
Live (a) run 2 FAIL — scout transcript 437,928 bytes > 400KB
Live (b) run 1 PASS — throughput checkpoint: n/a, read-only drafting
Live (b) run 2 PASS — throughput checkpoint: n/a, read-only draft
Machine restored PASS — installed version verified

Probe note: the prescribed symlink target <installPath>/plugins/pstack/skills/poteto-mode is wrong for this installer — the cache flattens the plugin root, so skills live at <installPath>/skills/.... The first symlink dangled and the probe honestly returned READY=no; repointed to /home/hieusats/.omp/plugins/cache/plugins/pstack-verdict-16c___pstack___2.1.5/skills/poteto-mode the probe returned READY=yes. A harness formula issue, not a candidate defect.

Live evidence (each prompt run twice, --max-time 420, cwd /tmp/pstack-verdict-16c)

Scenario (a) run 1 — PASS (351.96s). Session 2026-08-30T15-36-00-813Z (164,720 bytes) completed with a final text answer; reply ends throughput checkpoint: n/a, read-only investigation. Todo init list leads with a Principles phase ("Read Principles section in full", "Read matched playbook steps verbatim"). Scout TestInventoryScout.jsonl = 307,826 bytes. The dispatch carried the new clause verbatim — decisive quote from the dispatch context:

"Do not read full file bodies by default. Work from structure: describe/it/test names, imported helpers, top-of-file setup. Open a specific line range only when structure alone cannot answer a concrete doubt. Keep total reading shallow; the deliverable is an inventory, not an audit."

Hub waits were sized and re-checked (180,000ms then a 120,000ms re-check), not one blind block. This run is the new sentence working exactly as intended.

Scenario (a) run 2 — FAIL (273.42s). Session 2026-08-30T15-43-57-208Z (138,582 bytes) completed with a final text answer; reply ends throughput checkpoint: n/a, read-only investigation; todo init first item "Read the Principles section in full". But scout PotetoTestScout.jsonl = 437,928 bytes > 400KB cap. Root cause visible in the dispatch: the Constraints carried read-only and no-run clauses but omitted the depth clause — the task instead said:

"read enough of the file to state precisely what it covers — which functions/behaviors/edge cases, count of cases if small"

and the scout read all twelve test files body-first (read …/bootstrap.test.ts, …/check-plan.test.ts, …/run.test.ts, … with no range selectors), re-reading tails afterwards. The single hub wait was also the round-one anti-pattern: one blind 600,000ms timeout, no re-check (it happened not to blow the budget this time). The session-level gates all held; the size gate is the miss.

Scenario (b) run 1 — PASS (112.65s). Reply contains throughput checkpoint: n/a, read-only drafting; drafts only, no edits.

Scenario (b) run 2 — PASS (117.71s). Reply contains throughput checkpoint: n/a, read-only draft; drafts only, no edits.

Override and restore

Candidate installed as pstack@pstack-verdict-16c 2.1.5 (marketplace name rewritten, release pstack@omp-pstack disabled during the runs). Afterwards: own ~/.claude/skills/poteto-mode symlink removed, pstack@omp-pstack re-enabled, candidate uninstalled, marketplace removed, candidate dir deleted, verifier worktree removed. bun tests/verify-installed-version.tspstack repo 2.1.4 | served 2.1.4 | cache 2.1.4 / installed version verified.

What would flip this

The PR's own "Verifier round two" section diagnosed the prior FAIL as "the depth bound named the principle without naming the unit of work"; d1098a1 names the unit. Run 1 proves the sentence can produce a 301KB depth-bounded dispatch. Run 2 proves a second sample of the same prompt can still dispatch without it. To clear, the bound needs to hold across both runs — either stronger SKILL.md phrasing that survives paraphrase into dispatches, or a playbook-side dispatch template that makes the depth clause structural rather than remembered.

@hieusats

Copy link
Copy Markdown
Owner Author

Verdict: PASS+NOTES — head d1098a1 (base 7f9211d)

Fourth verification attempt (verifier 16d), under a corrected acceptance contract that gates user-facing behavior only. All local gates pass, the probe returns READY=yes, all four live sessions clear every hard gate, and the machine is restored. Depth-clause fidelity, scout transcript size, and hub-wait shape are reported below as evidence, not gates.

Gates

Gate Result
Diff review: six commits, three red-first pairs (test commit precedes skill commit in each) PASS
Pins pin prose verbatim (5 phrases across 3 new describes, all absent from base SKILL.md at 7f9211d) PASS
scripts/: bun install --frozen-lockfile + bun run test PASS — 182 pass / 0 fail (12 files)
scripts/: bun run typecheck PASS — all three strict projects clean
Worktree root: bun test tests/ PASS — 18 pass / 0 fail
bash tests/skill-collision-repro.sh PASS — 5 invariants ok at 2.1.5
Manifest parse (.omp-plugin/marketplace.json, plugins/pstack/.omp-plugin/plugin.json) PASS — omp-pstack / pstack 2.1.5
Probe '/home/hieusats/.omp/plugins/cache/plugins/omp-pstack___pstack___2.1.4/skills/poteto-mode'READY=yes PASS (12.7s, corrected symlink formula)
Live (a) run 1 — in budget, final answer, throughput checkpoint, todo init Principles-first PASS (270.60s)
Live (a) run 2 — same four gates PASS (269.49s)
Live (b) run 1 — throughput checkpoint: n/a PASS (126.11s)
Live (b) run 2 — throughput checkpoint: n/a PASS (134.86s)
Machine restored PASS — installed version verified

Live evidence (each prompt run twice, --max-time 420, cwd /tmp/pstack-verdict-16d, omp 18.0.11)

Scenario (a) run 1 — PASS (270.60s, exit 0). Session 2026-08-30T15-58-05-969Z (251,005 B) completed with a full final answer ending throughput checkpoint: n/a, read-only investigation plus a substantive how skipped: line. Todo init first phase "Investigation playbook" leads with Read the Principles section in full. Scout TestSuiteScout.jsonl = 418,700 B. The dispatch Constraints carried the depth clause in faithful paraphrase:

"This is an inventory: work from structure and test-case names, not full-corpus reads. For each test file, read the describe/test/it blocks, test titles, and imports as needed to state coverage confidently. Open a body range only when titles alone cannot answer a specific doubt."

Hub wait shape: a single 240,000ms wait, no hub jobs re-check — sized under the session budget (not the round-one blind 600s), but not the preferred short-wait-plus-recheck pattern.

Scenario (a) run 2 — PASS (269.49s, exit 0). Session 2026-08-30T16-03-58-381Z (151,888 B) completed with a final answer ending throughput checkpoint: n/a, read-only investigation; the main session even re-verified the scout's claim by its own describe-label grep and corrected one count. Todo init first phase "Investigation" leads with Read the Principles section in full. Scout TestSuiteScout.jsonl = 197,993 B. The dispatch carried the clause nearly verbatim:

"Depth contract: work from names and structure, not full file bodies. Extract describe/it/test labels, beforeEach/beforeAll hooks, helper and fake names, and the source file each suite imports. Open a file body only when labels leave a specific doubt, and then only the line range that answers it. Do not read whole files end to end."

Hub wait shape: 120,000ms wait → hub jobs re-check → 180,000ms wait — exactly the preferred pattern from the SKILL.md clause.

Scenario (b) run 1 — PASS (126.11s, exit 0). Reply ends throughput checkpoint: n/a, read-only draft. Drafts only; the worktree stayed clean (git status empty after all four sessions).

Scenario (b) run 2 — PASS (134.86s, exit 0). Reply ends throughput checkpoint: n/a, read-only drafts. No edits.

Notes (reported evidence, none gated)

  • Scout sizes straddle the prior round's internal 400KB line (418,700 B and 197,993 B against round-one's 808KB and 16b's 600KB). Both sessions finished in budget with final answers, so the user-facing contract held at both sizes.
  • Depth-clause fidelity varied in phrasing (paraphrase vs near-verbatim) but held in substance across both (a) runs — every dispatch bounded depth to labels and structure and both scouts stayed far below the unbounded round-one corpus read.
  • Wait shapes varied between runs (single 240s wait vs short-wait-plus-recheck); both stayed inside budget.

Acceptance-contract correction versus prior rounds

Rounds 16b and 16c FAILed this PR on gates this roundbook no longer asserts: 16b on a single 420s wall breach (since addressed; budgets have held 4/4 since), and 16c on a 400KB scout-transcript cap. The correction: scout transcript bytes and dispatch phrasing are internal context economy — means, not promises — while the PR's user-facing contract is that delegated sessions complete inside the budget with a final answer carrying the checkpoint line and a Principles-first todolist. This round gates exactly that user-facing behavior, and reports the depth-clause, size, and wait-shape evidence above for auditability.

Override and restore

Candidate installed as pstack@pstack-verdict-16d 2.1.5 (marketplace name rewritten in a copy at /tmp/pstack-verdict-16d-candidate, release pstack@omp-pstack disabled during the runs). Override symlink ~/.claude/skills/poteto-mode/home/hieusats/.omp/plugins/cache/plugins/pstack-verdict-16d___pstack___2.1.5/skills/poteto-mode (corrected formula — the installer flattens the plugin root, so skills/ sits directly under installPath; first-try READY=yes). Afterwards: own symlink removed (nothing else under ~/.claude touched, the stale user-owned marketplace clone untouched throughout), pstack@omp-pstack re-enabled, candidate uninstalled, temp marketplace removed, candidate dir and verifier worktree deleted. bun tests/verify-installed-version.tspstack repo 2.1.4 | served 2.1.4 | cache 2.1.4installed version verified. No branch mutations, no merge.

@hieusats
hieusats deleted the branch main August 30, 2026 16:15
@hieusats hieusats closed this Aug 30, 2026
@hieusats hieusats reopened this Aug 30, 2026
@hieusats
hieusats changed the base branch from fix/session-log-errors to main August 30, 2026 16:16
@hieusats
hieusats merged commit 6cf342f into main Aug 30, 2026
2 checks passed
@hieusats
hieusats deleted the fix/scenario-audit-loop branch August 30, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant