diff --git a/CHANGELOG.md b/CHANGELOG.md index ae01d52..dee38d5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,49 @@ # Changelog +## 0.9.0 + +- Added child request schema v3: a single free-text `goal` brief with + `request_id` and `origin`, and no `completion_criteria`/`stop_criteria`/ + `constraints`/`deliverables`/`required_evidence` arrays, no hashed `inputs`, + and no `dispatch_inputs` snapshot. The child's `goal.md` and goal contract are + the goal text verbatim; the dropped v2 arrays are treated as empty downstream. + Schema v2 is still accepted unchanged so in-flight sessions finish. +- Put the rendered iteration prompt on a diet (single-goal-assignments.md §3). + The header is now a fixed shape — goal, optional completion/stop criteria + sections (omitted when empty), and a short key-paths block — followed by the + workflow body. The ~50-path enumeration and the inlined frozen roster/ + scheduler/capability JSON are gone; the complete machine path map, rosters, + scheduler view, and workflow contract are referenced by files through a new + per-iteration `paths.json`. Header scaffolding (excluding goal and preamble) + is CI-bounded to 2 KB. +- Added a shared workflow-set preamble hook: when + `workflow_sets//preamble.txt` exists, the renderer includes it once under + "Shared ground rules:" so per-role prompts never repeat shared rules. +- `paths.json` records `previous_worker_sessions`, the previous iteration's + team-harness `worker_sessions.json` path (or null), enabling selective + worker-session reuse across iterations (context-and-eval-economy A4). +- Added `run_when_requested` per-workflow scheduling: a workflow so marked is + eligible only while `project_state/eval_request.md` exists in the session. It + composes with the existing `must_follow`/`priority`/`enabled` gates and can + replace `run_after_successes` for orchestrator-requested evaluation + (context-and-eval-economy C3). `run_on_start` still unlocks the first + scheduling pass, so a workflow can run on start and thereafter only on + request. +- Added optional root-config `team_harness_compact_above_tokens` and + `team_harness_prompt_cache`, carried in the wire snapshot and forwarded to the + Team Harness factory when the installed version accepts them (ignored + gracefully otherwise). +- Retired the eval-receipt output from the v3 stock flow. A protocol-v3 + check-runner role's frozen roster no longer advertises the `eval_receipts/` + output; advisory evaluation is now agent-authored + (`project_state/eval_results.md`, purely agent-owned — no engine coupling). + The receipt-sealing/validation machinery stays intact and contract-gated on + `check_runner_roles`, so sessions whose frozen contract still names a + receipt-producing check-runner keep working through their lifetime, and any + role that still emits a provenance-valid receipt still has it accepted as + advisory evidence. Completion authority is unchanged (the durable + orchestrator owns `goal_met`; eval never gated it — D11). + ## 0.8.0 - Added the protocol-v3 orchestration contract. Each layer now has an diff --git a/design/analysis/protocol-v3-flaws.md b/design/analysis/protocol-v3-flaws.md new file mode 100644 index 0000000..e254e10 --- /dev/null +++ b/design/analysis/protocol-v3-flaws.md @@ -0,0 +1,255 @@ +# Protocol-v3 flaw analysis — loopy-loop / team-harness / eval-banana (2026-07-18) + +Status: analysis. Companion designs: +- `design/designs/simplification-north-star.md` +- `design/designs/single-goal-assignments.md` +- `design/designs/session-layout-and-ids.md` +- `design/designs/context-and-eval-economy.md` + +Scope note on versions: this checkout (`loopy_loop_1`) is at 0.6.0. The system +actually running in UGM (`ultimate_memory/ugm`) is the protocol-v3 engine +(0.8.0, `loopy_loop_4`, PR #80, 2026-07-17). All runtime evidence below is from +the UGM 2026-07-17/18 run: parent session `20260717_182101_7dc9a9a9e501_08630977`, +child `20260717_190245_5c2e83ae6415_71b7c8b0` (27 iterations). Team-harness +evidence is from `team_harness_2/team-harness` (HEAD 0ae608f); eval-banana from +`eval_banana_2/eval-banana` (0.3.6). + +The unifying observation: between 0.6.0 (Jul 14) and 0.8.0 (Jul 17) the system +inverted its own founding idea. The founding idea (decisions D1/D5/D8/D9) is +"delegate high-level outcomes to capable harnesses; guide lightly; detect +failures, don't prevent them." Protocol v3 instead grew a compliance layer — +typed envelopes, sealed manifests, capability rosters, receipt families, hash +pinning, mirror trace trees — that now dominates what agents read, write, and +spend tokens on. Each piece answers a real-but-rare failure; the sum is a +system that briefs a frontier model like a bureaucracy briefs a contractor. + +Measured summary of one child loop (26 iterations, Codex analysis, verified): +~27.95M coordinator prompt tokens vs ~185k completion tokens (151:1); eval +iterations 8.59M (31%); per-iteration harness runs of 20–37 turns growing from +~20k to ~96k input tokens per turn. + +--- + +## A. Context economics + +### A1. Team-harness re-sends an ever-growing conversation, uncached, with dormant compaction +The coordinator loop keeps one `messages` list, appends every assistant/tool +message in full, and re-sends the entire list plus all tool schemas every turn +(`coordinator/loop.py:119-129`, `173-214`). There is no turn cap. Auto-compaction +exists but cannot fire in practice: +- Threshold is `model_limit − ~33k` → ≈967k tokens for gpt-5.5/OpenRouter, + ≈1.467M for gpt-5.6-sol (`tracking/context.py:147-149`) — runs peak ~96k. +- It only triggers when the last message is `user`-role (`loop.py:247`), which + in an SDK run happens exactly once (the initial task). +- Manual `/compact` is REPL-only; loopy-loop uses the SDK path. +No `cache_control` / prompt-caching markers are ever sent, so the re-sent +prefix is billed fresh every turn. Cost per run is therefore quadratic in +turns: 20k + 25k + … + 96k, per iteration, for every iteration. + +### A2. The rendered iteration prompt is 28–38 KB, ~70% ceremony +Measured on `iterations/0026_outer/prompt.txt` (38,527 bytes): +- goal + criteria restatement: ~4.9 KB +- attempt contract: ~0.7 KB +- **53 absolute paths: ~8.7 KB** — every path repeats the same 140-char + session prefix +- **frozen workflow/scheduler/capability JSON dump: ~18 KB** pretty-printed +- workflow body: ~6.2 KB +This is the ~20k-token floor of every harness turn (A1 multiplies it by +20–37). The 0.6.0 render was ~30 lines of header; v3 grew it ~10×. + +### A3. Large agent reports accumulate in coordinator context +`read_agent_output(tail_bytes=8192)` has a caller-controlled, **unbounded** +`tail_bytes` (`tools/agent_tools.py:1666-1679`); the observed 20–32 KB tool +results enter context this way and are then re-sent every remaining turn (A1). +`read_new_agent_output` is capped at 64 KB — also large as a context payload. +`run.json` additionally persists every turn's full message delta and every +tool result verbatim (`tracking/models.py:18-25`), so traces balloon too. + +### A4. No worker-session reuse across iterations +Team-harness fully supports resuming worker sessions (`spawn_agent` + +`resume_from_session_id`, per-provider capability map, `worker_sessions.json` +manifest with provider session ids). Loopy-loop never surfaces the previous +iteration's manifest to the next iteration, and no prompt suggests resuming. +Every iteration's agents re-derive all context from files. (Fresh *coordinator* +per iteration is correct — D1 continuity-in-files — but worker continuations +of genuinely continuous work, e.g. "apply the reviewer's fixes", are being +re-briefed from zero.) + +### A5. Eval is structurally expensive, and its biggest cost is accidental +- eval-banana stores each judge's **full stream-json stdout twice** per check + (`stdout` and `details.raw_response`); UGM `report.json` files are + **1.5–3.7 MB** (Σ 10.65 MB over 6 reports) while the actually-useful `reason` + fields total ~4.4 KB. The eval_runner prompt tells the agent to read + `report.json` → a single ~1M-token tool result. `report.md` (5–6 KB) has the + same verdicts. +- The stock eval_reviewer prompt **forbids deterministic checks** ("even for + objective file, structure, command, or data assertions"), so UGM has 83/83 + harness_judge checks and 0 scripts. Every check is a full autonomous Opus + 4.8 high-effort agent run (up to 300 s), executed **serially**. Judge + subprocess tokens are billed on top of, and invisible to, the 8.59M metered + eval figure. +- Cadence: reviewer on start + reviewer/runner every 10 inner successes → 7 of + 27 child iterations were eval; one runner concluded a "false advisory + observation" after 7 failed judges and 1 timeout — maximum spend, no signal. + +### A6. Mandated multi-agent recipes (0.6.0 templates) +The 0.6.0 inner/outer prompts hard-code an agentic-team recipe (~90 lines, +duplicated in both): fixed model per step ("using CODEX", "WITH GEMINI!"), the +same plan review mandated **three times** with prescribed models, ~7× +"think ultra deeply". This fights the config-level `model_tiers` abstraction +and multiplies spawns (5–7 agents some iterations). Protocol v3 softened this +("Dynamic delegation is optional… a preference, not a quota") — that +direction is right and should be kept and extended. + +--- + +## B. Assignment and instruction complexity + +### B1. The assignment envelope grew from one field to seven +v1 `ChildSessionRequest` was `{workflow_set, goal, schema_version}` — the goal +was one self-contained text. v2 (deployed in UGM) is a typed envelope: +`goal` + `completion_criteria[]` + `stop_criteria[]` + `constraints[]` + +`deliverables[]` + `required_evidence[]` + sha256-pinned `inputs[]`, plus an +immutable dispatch snapshot, ordered atomic renames, and a ledger link. In the +real Phase-0 request (6,140 bytes) the same fact appears in ≥3 grammatical +shapes (completion_criteria[1] ≈ deliverables[1] ≈ required_evidence[3]) and +the assignment is materialized in ≥4 places (task file, dispatch_inputs +snapshot, pending/accepted request, child `goal.md`). Structure invites +restatement; restatement invites drift; none of it makes the child smarter +than one well-written goal paragraph would. + +### B2. The goal file describes the looping system +UGM `loopy_loop_goal.txt` is 6,212 bytes; most of it is loop mechanics +("Program organization … This is a double loop", "Delivery contract", +"Evaluation as evidence", recovery policy). The product outcome is one line. +Loop mechanics belong in the workflow-set role prompts (where they are *also* +already stated), not in the frozen goal that every layer re-reads. The goal +should be the one text a product owner would write. + +### B3. Prompts are protocol manuals +0.6.0: outer 344 lines / inner 225, ~80–85% prescriptive (24 "do not"s and 28 +"only"s in outer alone), with heavy verbatim redundancy across files (PR +policy 3×, atomic-write warning 4×, goal-source caveat 4×). Framework-to-task +text ratio per iteration ≈ 99:1. v3 prompts are shorter (dispatcher 96, +planner 138, outer 125) but nearly 100% protocol: envelope reading order, +path-name enumerations, hash computation, rename ordering, receipt taxonomy. +The dispatcher — whose *job* is writing one good goal — gets ~5 lines about +goal quality and ~80 about publication mechanics. Genuine judgment guidance +("preserve planner intent", "coherent outcome with observable criteria") is a +minority in every prompt. + +### B4. Ceremony without demonstrated rent +Artifacts present per session in UGM: `goal_contract.json`, +`workflow_contract.yaml`, `workflow_roster.json`, `harness_capability_roster.json` +(duplicated as an 18 KB dump into every prompt — A2), `trace_seals/`, +`protocol_failures/`, `delivery_receipts/`, `eval_receipts/`, `git_receipts/` +(54 files for 27 iterations), `parent_acceptance/`, `child_outcomes/`, +`trace_finalization_outbox/`, per-attempt `workflow_snapshot/` (7 files each). +Each mechanism guards against a failure that mostly hasn't been observed; +collectively they define what agents must read, produce, and keep consistent. +Agents comply — e.g. a 146-line prose "Dispatcher publication report" narrating +one file publication. + +--- + +## C. Reviewability + +### C1. Engine-generated IDs are opaque; agent-chosen names are fine +Session id = `YYYYMMDD_HHMMSS__` — two meaningless hex +blobs; workflow set, role, and purpose absent. Attempt ids are bare 12-hex. +Receipts are keyed by hash (`git-after-97521a5ed6b7.json`, +`eval-483e152e5082.report.md`) — unopenable without cross-referencing. Trace +paths stack two hashes before anything readable +(`attempts/97521a5ed6b7/harness/20260718_064207_7f654370/…`). Contrast the +parts agents named: `0025_eval_runner`, `P0-L08-WP03-COMPONENT-VERSION-REGISTRY`, +`0010-accept-l07-select-l08-….md` — instantly navigable. The convention that +works is already in the tree; the engine just doesn't use it. + +### C2. The traces split put durable evidence in the prunable half +`traces/` was meant to be the raw, independently-prunable plane. In practice +**100 LLM-authored markdown documents** (implementation reports, adversarial +reviews, acceptance audits, planning notes) exist *only* under `traces/`, and +the durable `handoff.json`/`plan.md` cite them as evidence via opaque refs +(`trace:trace-97521a5ed6b7:/harness/…/acceptance/l07-audit.md`). Pruning +traces per the runbook's stated policy would sever the justification chain for +every accepted outcome. The mirror tree (`traces//sessions//…`) +also duplicates the session hierarchy under a second set of hash keys — and +already produced a data-integrity bug: a malformed sibling root dir keyed by a +*truncated* goal hash (`…7dc9a9e501…`, 10 hex, beside the correct 12-hex one) +holding a stranded subset of trace docs. + +### C3. Plans are provenance dumps, not orientation documents +Child `plan.md` is 16.8 KB at revision 10: outcome table, ASCII DAG, and long +"Accepted delivery" paragraphs inlining commit SHAs, PR numbers, and CI run +ids. `handoff.json` is 12.2 KB. The complaint "plans are not descriptive +enough" is precise: they are *verbose* but not *orienting* — a context-free +successor must dig through receipt-grade provenance to answer "what is this +loop doing, what state is it in, what's next, and why." Provenance belongs in +receipts/ledgers; the plan should be the document you'd want on day one of a +handover. + +### C4. Session dirs are too wide to eyeball +~25 top-level entries per session dir; `state.json` is 28 KB; the same +information (assignment, goal, criteria) appears at multiple depths. Reviewing +"what happened in iteration 25" requires opening JSON files whose names don't +say what they contain. + +### C5. Repo/version sprawl +Four loopy-loop checkouts at 0.5.0–0.8.0 (`loopy_loop_1..4`); docs in this +checkout describe a layout (no traces plane, 3-field child request) that the +deployed engine no longer has; the dev venv here pins team_harness 0.1.2 +against a declared ≥0.4.0 floor. Any analysis or agent working from the wrong +checkout reasons about a retired protocol. + +--- + +## D. Flow and behavior + +### D1. Work is sliced too fine for the per-iteration overhead +Every leaf costs: outer selection iteration + inner implementation iteration + +outer acceptance iteration, each a full 20–37-turn harness run with the A1/A2 +overhead, plus eval every ~6–8 iterations. Narrow reconciliation/verification +leaves make complete loop cycles out of work a single agent session would +finish incidentally. The overhead per iteration is currently so high that +granularity is an economic decision, and nothing in the prompts says so. + +### D2. Eval is a scheduled ritual, not a milestone gate +Cadence is mechanical (`run_after_successes: {inner, every: 10}` + on-start). +Eval runs regardless of whether anything integration-shaped happened, and its +results are advisory (v3 moved completion authority to the orchestrator — +correct), which makes the current spend/utility ratio worse: 31% of tokens for +advice the orchestrator may not need at that moment. + +### D3. Templates ship with empty criteria +Both stock configs omit `completion_criteria`/`stop_criteria` (defaults `[]`), +so the rendered header advertises empty sections and the only task-specific +content in a 0.6.0 iteration prompt is the one-line goal. + +--- + +## What v3 got right (keep) + +- Completion authority with the layer orchestrator, eval advisory (0.8.0). +- Delegation freedom language ("preference, not a quota"; capability roster + replacing hard-coded model names in prompts). +- Continuity in files + fresh coordinator per iteration (D1) — the Codex + analysis confirms blanket coordinator-session reuse would *worsen* context + growth. +- Crash-safe atomic publication of the few files that genuinely need it + (control, child requests). +- The instinct to separate raw noise from semantic state (the execution was + wrong — C2 — but the distinction matters). + +## Priority map (impact × effort) + +| # | Fix | Flaws addressed | Expected effect | +|---|-----|-----------------|-----------------| +| 1 | Prompt-cache coordinator requests; mid-run compaction trigger | A1 | ~an order of magnitude off billed input tokens, no behavior change | +| 2 | Iteration-prompt diet (paths file + roster by reference) | A2 | per-turn floor 20k → ~3k tokens | +| 3 | eval-banana: drop `raw_response` duplication; runner reads `report.md`; lift deterministic ban; milestone cadence | A5, D2 | eval share 31% → single digits | +| 4 | Single-goal assignment (v3.1 envelope = goal text) | B1, B3 | dispatcher writes briefs, not envelopes | +| 5 | Fold traces into session dir; durable home for agent-authored docs | C2 | evidence chain survives pruning; one tree | +| 6 | Readable IDs (slug + role + iteration) | C1 | navigable sessions | +| 7 | Orientation-first plan/handoff contract | C3 | cold-start successors | +| 8 | Worker-session reuse for continuations; result cards | A3, A4 | less re-briefing, smaller contexts | +| 9 | Work-package granularity guidance | D1 | fewer, fuller iterations | diff --git a/design/designs/context-and-eval-economy.md b/design/designs/context-and-eval-economy.md new file mode 100644 index 0000000..a2801b0 --- /dev/null +++ b/design/designs/context-and-eval-economy.md @@ -0,0 +1,141 @@ +# Context and eval economy + +Status: proposed. Grounding: `design/analysis/protocol-v3-flaws.md` §A1–A5, §D1–D2. +Principles applied: P1, P6, P7. Companion: `single-goal-assignments.md` §3 +(prompt diet) removes the per-turn floor this document doesn't cover. + +Measured baseline (UGM child loop): 27.95M prompt / 185k completion tokens +across 26 iterations; runs of 20–37 coordinator turns re-sending 20k→96k +tokens per turn, uncached; eval at 31% of metered spend plus unmetered judge +subprocesses. Two families of fix: make the re-sent context cheap and small +(team-harness), and stop generating context nobody needs (loopy-loop + +eval-banana). + +## A. Team-harness + +### A1. Prompt caching (largest single lever, zero behavior change) +The coordinator re-sends an append-only prefix every turn with no +`cache_control` markers. Add provider-appropriate caching in +`coordinator/client.py`: cache breakpoints on the system prompt and on the +last-but-N message boundary for OpenAI-compatible/OpenRouter (and the codex +provider's equivalent). The conversation is append-only between turns, which +is the ideal caching shape; billed input for turns 2..N becomes mostly +cache-read. This converts the quadratic term's constant into a small one even +before any context shrinking. + +### A2. Compaction that can actually fire +Today's trigger needs ~967k–1.467M current-context tokens and a `user`-role +last message — unreachable in SDK runs peaking at 96k. Change +(`coordinator/loop.py`, `tracking/context.py`): +- Add `compact_above_tokens` (Config knob; loopy-loop passes it, default + ~80k): compact when `ctx.total` exceeds it, at any tool-result boundary, + not only after `user` messages. +- Compaction target stays the existing "<10k tokens" summary prompt; preserve + the initial task message verbatim (it carries the assignment). +The knob is a safety net, not the plan: with A1+A3 and the prompt diet, most +runs should finish without compacting. + +### A3. Bound what enters the coordinator context +- `read_agent_output`: give `tail_bytes` a ceiling (`Config`, default 16 KB); + larger requests get the tail plus a truncation banner naming the full log + path (pattern already used by `read_new_agent_output`). +- Worker footer (`_build_direct_spawn_footer`) gains one sentence: *"End your + stdout with a result card: ≤15 lines — outcome, key decisions, files + changed, and absolute paths to any report you wrote. Write long reports to + files, not stdout."* The coordinator reads cards by default and opens report + files only when needed — the report then costs one read, not N re-sends. +- `run.json`: store tool-call results truncated to a few KB with a pointer to + the worker log that already holds the full stream (dedupes A-side trace + bloat; the full data exists on disk exactly once). + +### A4. Worker-session reuse for genuine continuations +The capability exists end-to-end (spawn `resume_from_session_id`, per-provider +support map, `worker_sessions.json` manifest); nothing exposes it across loopy +iterations. Changes: +- loopy-loop passes the previous iteration's `worker_sessions.json` path in + the rendered header's `paths.json`. +- One line in the shared workflow-set preamble: *"If your task continues a + previous iteration's work (applying review fixes, re-verifying the same + change), resume that worker's session via the manifest instead of + re-briefing from zero. For new work, start fresh."* +Selective by prompt-level judgment (P1), not an engine mandate — the Codex +analysis is right that blanket reuse would grow context rather than save it. +Fresh coordinator per iteration stays (D1). + +## B. Loopy-loop + +### B1. Iteration granularity is an economic decision — say so +Add to outer/planner judgment guidance: *"Each iteration costs a full +coordinator run before any work happens. Prefer work packages you can carry +to done — implemented, reviewed, integrated — within one iteration; slice +only when a package genuinely exceeds one iteration's reach. Verification and +reconciliation ride along with the work they verify; they are rarely their own +iteration."* This addresses the observed pattern of narrow +reconciliation/verification leaves each costing a full cycle. + +### B2. Default model posture +Keep uniformly-strong coordinators (D9) but make the stock tier guidance +explicit that `standard` is the default spawn tier for routine execution and +mechanical steps, with `strong`/`frontier` reserved for architecture, +adversarial review, and disagreement — the current guidance implies it; +prompts hard-coding frontier-everything (0.6.0 recipes) are deleted by +`single-goal-assignments.md` §4. + +## C. Eval + +### C1. Fix the accidental megabytes (eval-banana) +- Stop duplicating judge stdout: `report.json` keeps `details.raw_response` + as a *path* to the already-persisted `checks/.stdout.txt`, and + `stdout` holds only the tail needed for error display. Report sizes drop + ~1.5–3.7 MB → tens of KB. +- eval_runner prompt: read `report.md` (5–6 KB, same verdicts + reasons); + open per-check artifacts only for failures under investigation. Never read + `report.json` wholesale. + +### C2. Lift the deterministic-check ban (revisit D4) +D4's point — agents shouldn't grade themselves with self-authored assertions +that ossify implementation details — argued for judge-led evaluation, but the +stock reviewer prompt hardened it into "deterministic checks are forbidden +even for objective file, structure, command, or data assertions." Result: +83/83 checks are full Opus-agent investigations, many of which verify facts a +script checks for free (file exists, command exits 0, table present). +New reviewer guidance: *"Judge checks evaluate semantics and quality. +Objective, mechanically-verifiable facts belong in deterministic checks — +prefer invoking the repo's own test/lint suites over bespoke assertions. +Keep judge checks for what actually needs judgment."* (Extends existing +proposal P1.2 from `improvement-proposals.md` into the stock template.) +- Judge tier: default the eval harness to the `standard` tier; `strong` for + final goal checks. Run judges in parallel (they are independent + subprocesses; the serial for-loop is just latency). + +### C3. Milestone-driven cadence instead of a ritual +Replace the stock `run_after_successes: {inner, every: 10}` schedule with +orchestrator-requested eval: the outer requests an eval iteration by writing +`project_state/eval_request.md` (one line: scope + why now) when it reaches a +milestone or intends to declare completion; a small scheduler predicate +(`run_when_requested: true`) makes eval_runner eligible only when a pending +request exists, and the terminal goal check remains mandatory before +`goal_met`. eval_reviewer runs on start and thereafter only when the goal +check's check-set is stale relative to the plan (outer requests it the same +way). Eval stays advisory (0.8.0 semantics — keep); it just stops firing on +iterations where nobody will act on it. Expected effect: eval share of spend +31% → single digits, with *better* timing relative to decisions. + +## Sequencing + +1. C1 (one-day fixes, huge effect) and A1 (caching). +2. Prompt diet (`single-goal-assignments.md` §3) + A3 result bounding/cards. +3. A2 compaction knob, A4 session-reuse plumbing + preamble line. +4. C2/C3 template + scheduler changes; B1/B2 prompt guidance. + +## Acceptance criteria + +- Re-run a UGM-scale child loop: total coordinator prompt tokens for a + comparable 26-iteration program under 5M (from 27.95M), with billed + (non-cache-read) input a fraction of that. +- No single tool result over 16 KB enters coordinator context uncompacted. +- `report.json` under 100 KB for a 10-check run. +- Eval iterations occur only on request or at terminal goal check; a program + session's eval spend share reported by the usage ledger under 10%. +- A continued-work iteration (review-fix pattern) demonstrably resumes a + worker session (visible in `worker_sessions.json` resume records). diff --git a/design/designs/session-layout-and-ids.md b/design/designs/session-layout-and-ids.md new file mode 100644 index 0000000..327442a --- /dev/null +++ b/design/designs/session-layout-and-ids.md @@ -0,0 +1,120 @@ +# Session layout, readable IDs, and the end of the traces mirror + +Status: proposed. Grounding: `design/analysis/protocol-v3-flaws.md` §C1–C4. +Principles applied: P4, P5, P6. + +## Problem + +Engine-generated IDs are timestamp+hash blobs carrying no workflow, role, or +purpose; receipts are keyed by bare attempt hashes; trace paths stack two +hashes before anything readable. The separate `traces/` mirror tree duplicates +the session hierarchy under a second set of hash keys, holds 100 LLM-authored +evidence documents that durable handoffs cite via `trace:` refs, is +declared prunable — and has already produced a truncated-hash duplicate-dir +bug. Session dirs have ~25 top-level entries. + +## Design + +### 1. IDs: ordinal + slug + role; hashes only as suffixes + +- **Session id**: `NNN_` where `NNN` is a per-scope ordinal (root + sessions numbered within the repo, children within their parent) and + `` is kebab-case, derived from the request id when present (children: + `01_phase-0-foundations`) or from the first meaningful words of the goal + (roots: `001_ultimate-memory-program`). Uniqueness within scope is enforced + by the ordinal; no random component in the name. Timestamp, goal hash, and a + uuid stay as *fields in `session.json`* for machine use — they leave the + directory name. +- **Iteration dirs**: unchanged (`0026_outer`) — this convention already works. +- **Attempt ids**: keep the hex id for the wire protocol, but every on-disk + artifact keyed by attempt gains the iteration prefix: + `git_receipts/0026_outer_after.json`, `eval_receipts/0025_eval_runner.report.md`, + raw dirs `raw/0026_outer/…`. Rule of thumb (P4): a reviewer reading `ls` + output must be able to say what each entry is without opening it. +- **Harness run dirs**: `raw/0026_outer/harness/` — the run's own + `run_id` stays inside `run.json`; it no longer names two nesting levels. +- IDs are derived exactly once, at creation, and passed as values thereafter — + never re-derived by parsing paths or re-hashing content (the class of bug + behind the truncated trace dir). + +Full example after this change: + +``` +.loopy_loop/sessions/001_ultimate-memory-program/ + children/01_phase-0-foundations/ + iterations/0026_outer/ + project_state/plan.md + git_receipts/0026_outer_after.json + raw/0026_outer/harness/run.json +``` + +versus today's +`sessions/20260717_182101_7dc9a9a9e501_08630977/children/20260717_190245_5c2e83ae6415_71b7c8b0/…` +with evidence at +`traces/20260717_182101_…/sessions/20260717_190245_…/attempts/97521a5ed6b7/harness/20260718_064207_7f654370/acceptance/l07-audit.md`. + +### 2. Fold traces back into the session tree + +Answering the standing question — was separating traces a good idea? The +*distinction* (raw noise vs durable semantic state) is right; the *mechanism* +(a parallel top-level mirror tree) is not. Verdict: keep the distinction as a +**subdirectory boundary inside each session**, delete the mirror. + +- Each iteration gets `sessions/…/raw/_/` holding only + **mechanically produced** streams: team-harness `run.json`, + `coordinator_input.json`, worker `stdout.jsonl`/`stderr.log`, git + `-diff.patch`/`-status.jsonl`, eval `checks/*.stdout.txt`, locks, reap + reports. +- **Placement rule (the load-bearing line): if an LLM authored it as a + document — report, audit, review, plan, analysis — it is evidence, and it + lives in the durable tree** (`project_state/` for state, or the iteration + dir for per-attempt reports like `iterations/0026_outer/acceptance-audit.md`). + Agents are told: scratch and verbose dumps → `raw/`; anything another agent + or human might cite → durable tree. The current situation (100 evidence .md + files only under prunable traces, cited by handoffs) inverts this. +- Pruning policy attaches to `raw/` only: `.gitignore` covers + `sessions/**/raw/`; a `loopy prune-raw [--older-than]` command deletes raw + dirs without touching evidence. `trace:` refs are retired — evidence + refs in handoffs/plans become session-relative paths, which are stable, + human-readable, and survive pruning. +- `trace_manifest.json`, `trace_seals/`, `trace_finalization_outbox/` are + deleted per the P6 kill list; `result.json`'s `trace_ref` becomes a plain + relative path into `raw/`. + +Why not keep the separate tree with better naming? Two trees require two id +schemes, cross-tree refs, finalization machinery, and seals to keep them +honest — that machinery *is* the complexity, and the truncated-hash bug shows +its cost. One tree with a gitignored `raw/` subdir gets the same +retention/privacy split with zero cross-referencing. + +### 3. Narrow the session dir + +Top-level of a session dir after consolidation (~12 entries): +`session.json`, `state.json`, `events.jsonl`, `goal.md`, `control.json`, +`project_state/`, `iterations/`, `raw/`, `child_requests/`, `children/`, +`inputs/`, `receipts/` (git+eval+delivery receipt files merged into one dir, +each self-describing per §1). `harness_capability_roster.json`, +`workflow_roster.json`, `workflow_contract.yaml`, `goal_contract.json` +collapse into `session.json` (they are all frozen-at-creation facts about the +same session). + +## Migration + +- New layout applies to new sessions only; readers support both during a + deprecation release (path helpers in `sessions.py` already centralize + construction). +- A `loopy migrate-session ` best-effort relinker is optional; UGM's + in-flight program can simply finish on the old layout. +- Update `docs/session-layout.md` in the same PR (it is already stale against + the deployed engine — flaw C5). + +## Acceptance criteria + +- `find .loopy_loop/sessions -maxdepth 4` output is readable: every entry + names its role/ordinal/slug; no bare-hash filenames outside `session.json` + internals. +- Zero LLM-authored `.md` files under `raw/` in a full stock-template run + (spot-check test greps for markdown headers in `raw/`). +- Deleting `raw/` entirely leaves every ref in `plan.md`/`handoff.json`/ + receipts resolvable. +- One id-derivation site per id kind; no path parsing to reconstruct ids. diff --git a/design/designs/simplification-north-star.md b/design/designs/simplification-north-star.md new file mode 100644 index 0000000..5bf55a4 --- /dev/null +++ b/design/designs/simplification-north-star.md @@ -0,0 +1,91 @@ +# Simplification north star + +Status: proposed. Grounding: `design/analysis/protocol-v3-flaws.md`. + +This document is the acceptance filter for every other design and every future +feature. It exists because between 0.6.0 and 0.8.0 the system drifted from +"lightly guide capable harnesses toward a goal" to "specify a compliance +protocol that harnesses execute." The three concrete designs +(`single-goal-assignments.md`, `session-layout-and-ids.md`, +`context-and-eval-economy.md`) are applications of these principles. + +## Principles + +**P1 — The intelligence is in the harness, not the protocol.** +Codex, Claude Code, and Antigravity are the strongest components in the +system. Loopy-loop's job is only: persist state across iterations, route work +between layers, and keep evidence findable. Anything the protocol does beyond +that is second-guessing a frontier model. When choosing between "add a +mechanism" and "add one sentence of guidance and trust the agent," choose the +sentence. + +**P2 — One goal, one text.** +An assignment at every boundary (root config → session, dispatcher → child) is +a single descriptive prose goal. Descriptive is good — a goal may be many +paragraphs and should say what done looks like, what matters, and where to +record evidence — but it is *one text written for a reader*, not a schema. If +something feels like it needs a field, write it as a sentence. + +**P3 — Mechanics live in role prompts, at the workflow-set level.** +How the loop works, who owns which files, how to publish a child request — +that is the workflow-set's static prompt material, written once per role. It +never belongs in goal files, assignment texts, or per-iteration renders. A +goal file must read like something a product owner wrote. + +**P4 — Reviewability is a first-class requirement.** +A human must be able to `ls` their way through a run: every engine-generated +path carries role, ordinal, and a human slug; hashes appear only as short +uniqueness suffixes. If reviewing a decision requires opening a JSON file to +learn what a filename means, the naming is wrong. + +**P5 — One tree per session.** +Everything a session produces lives under its session directory. Prunable raw +noise is a clearly-named subdirectory, not a parallel top-level mirror. +Anything an agent *authored* (plans, reports, audits, reviews) is semantic +evidence and lives in the durable part, next to whatever cites it. + +**P6 — Ceremony must pay rent.** +Every protocol artifact (receipt family, seal, contract, snapshot, hash pin) +must name the failure it prevented in a real run. On review, an artifact that +cannot point at an incident it would have caught gets deleted. Detection over +prevention (D8): prefer noticing a rare failure after the fact to making every +iteration pay an up-front compliance cost. + +**P7 — Tokens are an architectural constraint.** +Context is re-processed every turn, so every byte placed in a prompt or a tool +result is multiplied by the turns that follow it. Defaults: reference files by +path instead of inlining; return cards (a few lines + paths), not reports; +per-iteration rendered header budget ~2 KB; anything bigger goes in a file the +agent can open on demand. + +**P8 — Prompts state outcomes and ownership, then stop.** +A role prompt says: who you are, what outcome you own, what you must not own +(one or two boundaries), where things live, and what good judgment looks like +here. It does not enumerate steps, mandate team compositions, fix model names, +or repeat rules another prompt already states. Target: a role prompt fits on +one screen (~60–80 lines); shared rules are stated once in a shared preamble, +not per-prompt. + +## Kill list (apply P6 now) + +Delete unless a concrete incident justifies each — and record the incident in +`design/decisions.md` if kept: + +- `trace_seals/`, `trace_finalization_outbox/`, `protocol_failures/` dirs +- sha256-pinned `inputs[]` + immutable dispatch snapshots (git already + provides immutability; the goal text can cite paths/commits) +- the 5 structured assignment arrays (→ `single-goal-assignments.md`) +- per-attempt `workflow_snapshot/` (7 files/attempt; keep the one rendered + prompt.txt already stored in the iteration dir) +- the 18 KB frozen roster/scheduler/contract dump in every rendered prompt + (→ file by reference) +- hash-keyed receipt filenames (→ role+ordinal names, `session-layout-and-ids.md`) +- `goal_contract.json` / `workflow_contract.yaml` duplication where + `session.json` + the workflow-set on disk already say the same thing + +## How to use this document + +- New feature or field → it must cite which principle it serves and survive P6. +- Existing mechanism questioned → same test, in reverse. +- Prompt edits → P3/P8 review: does this line guide judgment, or script + compliance? Script lines need a justifying incident. diff --git a/design/designs/single-goal-assignments.md b/design/designs/single-goal-assignments.md new file mode 100644 index 0000000..eb5efd7 --- /dev/null +++ b/design/designs/single-goal-assignments.md @@ -0,0 +1,161 @@ +# Single-goal assignments and prompt diet + +Status: proposed. Grounding: `design/analysis/protocol-v3-flaws.md` §A2, §B1–B4, §D3. +Principles applied: P2, P3, P7, P8. + +## Problem + +The v2/v3 child request splits one outcome into seven typed fields plus hashed +input snapshots; the same fact gets restated in 3–4 grammatical shapes and +materialized in ≥4 files. The rendered iteration prompt is 28–38 KB, ~70% +mechanical ceremony (53 absolute paths, an 18 KB frozen JSON dump). The goal +file in UGM is 6 KB of loop mechanics around a one-line product outcome. +Dispatcher/role prompts are protocol manuals with a few lines of judgment +guidance buried inside. + +## Design + +### 1. The child request becomes a goal text again + +Wire schema (v3.1): + +```json +{ + "schema_version": 3, + "request_id": "phase-0-foundations", + "workflow_set": "inner_outer_eval", + "goal": "", + "origin": {"parent_attempt_id": "…", "parent_work_item_id": "…"} +} +``` + +Dropped: `completion_criteria[]`, `stop_criteria[]`, `constraints[]`, +`deliverables[]`, `required_evidence[]`, sha256-pinned `inputs[]`, the +immutable `dispatch_inputs/` snapshot, and the mandated +snapshot→hash→rename→ledger ordering. Kept: atomic rename of the one request +file (crash safety that has paid rent), request id as idempotency key. + +The `goal` is a brief, written the way you'd brief a strong engineer joining +the effort. It *should* be descriptive — including acceptance expectations, +constraints, and pointers to inputs — but as prose, in whatever shape fits the +work. If the dispatcher wants the child to read a phase file, it writes "Start +from `plan/phases/phase-0.md`; treat it as the scope contract." Immutability +of referenced inputs comes from git (cite a commit when it matters), not from +hash pinning. + +Dispatcher prompt replaces the envelope contract + "Child goal requirements" +bullets with roughly: + +> Write the child's goal as a self-contained brief: the outcome and why it +> matters now, what done observably looks like, constraints worth knowing, +> and where to record evidence. Preserve planner intent; don't decompose into +> leaves — the child owns its own plan. Write it so an agent with no other +> context could start working from it alone. + +The child's `goal.md` is that text, verbatim. One authored artifact instead of +four materializations. + +### 2. Root-level goal files carry zero loop mechanics + +`loopy_loop_goal.txt` describes the product outcome only. Everything currently +in UGM's goal under "Program organization", "Delivery contract", "Evaluation +as evidence", "Recovery and blockers" moves to (or already exists in) the +workflow-set role prompts and is deleted from the goal. Same for +`completion_criteria`/`stop_criteria` in root config: keep the fields +(optional, they render fine when present) but stop using them as a second +place to restate the goal — stock templates should either populate them +meaningfully or omit the section from the render when empty (fixes the +empty-section render of §D3). + +### 3. Iteration prompt diet: 2 KB header + files by reference + +Rendered header becomes: + +``` +loopy-loop assignment — iteration 0026, role: outer, session: 03_phase-0-foundations + +Goal: + + +You are inside a durable looping session. Key paths: +- session dir: (everything below is relative to it) +- project_state/ your durable working state (plan.md, handoff.md, …) +- child_requests/pending/ publish child requests here +- control.json terminal control +- raw/0026_outer/ scratch + verbose output for this iteration +- paths.json full path map, rosters, scheduler view — read if needed + +Workflow body: + +``` + +Concretely: +- The 53-path enumeration collapses to the ~6 paths above plus one + `paths.json` in the iteration dir holding the complete map for the rare + agent that needs more. +- The 18 KB frozen roster/scheduler/capability JSON is never inlined; it lives + as files referenced from `paths.json`. The model-tier table stays in the + system-prompt extension (it's small and behaviorally load-bearing). +- Budget check in CI: rendered header (everything before `Workflow body:`) + ≤ 2 KB + goal length. A test renders each stock template and fails on + regression. + +Effect: per-turn floor drops ~20k → ~3k tokens, multiplied across every turn +of every harness run (flaw A1 makes this multiplicative, see +`context-and-eval-economy.md`). + +### 4. Role prompts: one screen, judgment-first + +Rewrite each stock role prompt to the P8 shape (~60–80 lines): +identity + owned outcome; 1–2 ownership boundaries; where things live (by +reference to the header); judgment guidance specific to the role; nothing +else. Shared rules (atomic writes, PR policy, permission policy, goal-source +caveat) move to **one** shared preamble file per workflow set, included once +by the renderer — never repeated per role prompt. Delete from all stock +prompts: +- fixed team recipes and model-name mandates ("using CODEX", "WITH GEMINI!", + triple plan review) — delegation stays free-form per v3's own language: + optional, roster-informed, "a preference, not a quota"; +- step-numbered runbooks where the step order is obvious from the outcome; +- "think ultra deeply" incantations (effort belongs to tier config). + +### 5. Plans and handoffs: orientation first + +Replace the mandated multi-field plan/README/ledger formats with one contract, +stated in the outer/planner prompt: + +> `project_state/plan.md` is for a successor with zero context. Lead with: +> what this loop is building, current status in a few sentences, what's next +> and why, and open risks. Keep it under ~150 lines; when a section stops +> helping a newcomer, cut or archive it. Provenance (commit SHAs, PR numbers, +> CI run ids, acceptance details) lives in `project_state/ledger.md` +> (append-only), not in the plan. + +`handoff.json` stays typed (parents parse it) but slims to: `summary`, +`status`, `open_work[]`, `risks[]`, and refs — with refs pointing at durable +session-relative paths (see `session-layout-and-ids.md`; `trace:` refs +are retired). The 8-section task README and 9-field finished.md templates are +replaced by "write enough that a cold reader could pick the task up" plus a +2–3 bullet example. + +## Migration + +1. Engine: accept schema_version 3 alongside 2; dispatcher template writes 3. +2. Renderer: new header + `paths.json`; keep old field names in + `assignment.json` for one release for tooling that reads it. +3. Templates: rewrite prompts (worst-first: dispatcher, outer); move shared + rules to the preamble file. +4. UGM: regenerate goal file (product outcome only) before the next program + session; existing sessions finish on v2. + +## Acceptance criteria + +- A dispatched child request contains exactly one authored text; no field of + it restates another field. +- Rendered header ≤ 2 KB + goal (CI-enforced). +- Each stock role prompt ≤ 80 lines; shared rules appear in exactly one file + per workflow set. +- Root goal files in stock templates contain no loop-mechanics vocabulary + (loop, iteration, workflow, session, eval cadence…) — grep-testable. +- A reviewer can read `goal.md` + `plan.md` of any session and correctly state + what it is doing and what happens next, without opening JSON. diff --git a/pyproject.toml b/pyproject.toml index eaf3330..e35d6ad 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -4,7 +4,7 @@ build-backend = "hatchling.build" [project] name = "loopy-loop" -version = "0.8.0" +version = "0.9.0" description = "Run long-running AI agent workflows inside your repository." readme = "README.md" requires-python = ">=3.12" diff --git a/src/loopy_loop/__init__.py b/src/loopy_loop/__init__.py index bc8c296..3e2f46a 100644 --- a/src/loopy_loop/__init__.py +++ b/src/loopy_loop/__init__.py @@ -1 +1 @@ -__version__ = "0.7.2" +__version__ = "0.9.0" diff --git a/src/loopy_loop/cli.py b/src/loopy_loop/cli.py index 5685a01..85e6785 100644 --- a/src/loopy_loop/cli.py +++ b/src/loopy_loop/cli.py @@ -50,6 +50,7 @@ ROOT_CONFIG_FILENAME, DEFAULT_GOAL_FILENAME, ".loopy_loop/workflow_sets/inner_outer_eval/contract.yaml", + ".loopy_loop/workflow_sets/inner_outer_eval/preamble.txt", ".loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/config.yaml", ".loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt", ".loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/config.yaml", @@ -64,6 +65,7 @@ ROOT_CONFIG_FILENAME, DEFAULT_GOAL_FILENAME, ".loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml", + ".loopy_loop/workflow_sets/pm_planner_dispatcher/preamble.txt", ".loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/config.yaml", ".loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/prompt.txt", ".loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/config.yaml", diff --git a/src/loopy_loop/config.py b/src/loopy_loop/config.py index be78b69..b5461de 100644 --- a/src/loopy_loop/config.py +++ b/src/loopy_loop/config.py @@ -30,6 +30,7 @@ WORKFLOWS_DIRNAME = "workflows" WORKFLOW_SETS_DIRNAME = "workflow_sets" WORKFLOW_SET_CONTRACT_FILENAME = "contract.yaml" +WORKFLOW_SET_PREAMBLE_FILENAME = "preamble.txt" GOAL_HASH_LENGTH = 12 DEFAULT_GOAL_CHECK_FAILURE_CAP = 3 DEFAULT_WORKFLOW_FAILURE_CAP = 5 @@ -48,6 +49,7 @@ DEFAULT_WORKFLOW_PRIORITY = 0 DEFAULT_WORKFLOW_RUN_ON_START = False DEFAULT_WORKFLOW_EMITS_GOAL_CHECK = False +DEFAULT_WORKFLOW_RUN_WHEN_REQUESTED = False CANONICAL_MODEL_TIER_DESCRIPTIONS: dict[str, str] = { "frontier": ( "maximum-capability work: the hardest planning, architecture, " @@ -229,6 +231,26 @@ class RootConfig(BaseModel): default=DEFAULT_SYSTEM_PROMPT_EXTENSION, description="Additional system prompt text appended for every harness run.", ) + team_harness_compact_above_tokens: int | None = Field( + default=None, + gt=0, + description=( + "Optional coordinator-context compaction threshold passed to " + "team-harness as compact_above_tokens. Leave null to use the " + "installed team-harness default; ignored gracefully when the " + "installed team-harness does not accept it. Belongs in the wire " + "snapshot like the other team_harness_* fields." + ), + ) + team_harness_prompt_cache: str | None = Field( + default=None, + description=( + "Optional provider prompt-cache mode passed to team-harness as " + "prompt_cache. Leave null to use the installed team-harness " + "default; ignored gracefully when the installed team-harness does " + "not accept it. Belongs in the wire snapshot." + ), + ) recovery_policy: Literal["drain", "reap"] = Field( default="drain", description=( @@ -431,6 +453,15 @@ class WorkflowConfig(BaseModel): default=DEFAULT_WORKFLOW_EMITS_GOAL_CHECK, description="Whether this workflow is expected to write goal_check.json.", ) + run_when_requested: bool = Field( + default=DEFAULT_WORKFLOW_RUN_WHEN_REQUESTED, + description=( + "Gate this workflow on an explicit orchestrator request: when true " + "it is eligible only while project_state/eval_request.md exists in " + "the session. Composes with the other gates (must_follow, priority, " + "enabled) and can stand in for run_after_successes on that workflow." + ), + ) class WorkflowDefinition(WorkflowConfig): @@ -764,6 +795,30 @@ def workflow_set_contract_path(*, repo_root: Path, workflow_set: str) -> Path: ) +def workflow_set_preamble_path(*, repo_root: Path, workflow_set: str) -> Path: + """Return the shared-ground-rules preamble path for a workflow set.""" + + return ( + workflow_set_dir_path(repo_root=repo_root, workflow_set=workflow_set) + / WORKFLOW_SET_PREAMBLE_FILENAME + ) + + +def load_workflow_set_preamble(*, repo_root: Path, workflow_set: str) -> str | None: + """Return the workflow set's shared preamble text, or None when absent. + + The renderer includes this once per iteration under "Shared ground rules:" + so per-role prompts never repeat the workflow set's shared rules. Stock + preamble files are authored by the workflow-set templates; this reader is + the inclusion mechanism only. + """ + + path = workflow_set_preamble_path(repo_root=repo_root, workflow_set=workflow_set) + if not path.is_file(): + return None + return path.read_text(encoding="utf-8") + + def load_workflow_set_contract( *, repo_root: Path, workflow_set: str, workflows: list[WorkflowDefinition] ) -> tuple[WorkflowSetContract, str, str]: diff --git a/src/loopy_loop/coordinator_app.py b/src/loopy_loop/coordinator_app.py index 33fb65e..a2713ba 100644 --- a/src/loopy_loop/coordinator_app.py +++ b/src/loopy_loop/coordinator_app.py @@ -92,6 +92,7 @@ from loopy_loop.sessions import create_session_dir from loopy_loop.sessions import create_session_id from loopy_loop.sessions import delivery_receipts_dir_path +from loopy_loop.sessions import eval_request_path from loopy_loop.sessions import file_sha256 from loopy_loop.sessions import git_receipts_dir_path from loopy_loop.sessions import goal_check_path @@ -348,7 +349,13 @@ def _build_workflow_roster( expected_outputs.extend(["eval_checks/", "project_state/eval_state.md"]) if workflow.id in contract.check_runner_roles: authorities.append("eval_check_runner") - expected_outputs.extend(["eval_receipts/", "project_state/eval_state.md"]) + # v3 evaluation is agent-authored advisory evidence + # (project_state/eval_results.md, written by the check-runner role); + # the engine no longer advertises the retired eval_receipts/ output. + # Sessions whose frozen roster predates this keep their own + # advertisement, and the receipt-sealing machinery below stays + # contract-gated so their receipts are still accepted. + expected_outputs.append("project_state/eval_state.md") if workflow.id in contract.terminal_blocker_reporting_roles: authorities.append("terminal_blocker_reporting") run_after = ( @@ -934,6 +941,7 @@ def _advance( workflows=workflows, history=state.history, iteration_count=state.iteration_count, + eval_requested=self._eval_requested(state=state), ) if workflow is None: state.stop_reason = "no_eligible_workflow" @@ -1095,6 +1103,7 @@ def _write_scheduler_view( workflows=self._workflows_for(workflow_set=state.workflow_set), history=projected_history, iteration_count=task.iteration, + eval_requested=self._eval_requested(state=state), ) reasons = ( [ @@ -3367,6 +3376,18 @@ def _preflight_for(self, *, workflow_set: str) -> PreflightResult: def _workflows_for(self, *, workflow_set: str) -> list[WorkflowDefinition]: return self._preflight_for(workflow_set=workflow_set).workflows + def _eval_requested(self, *, state: LoopState) -> bool: + """Return whether the active session has a pending eval request. + + The file-existence predicate behind `run_when_requested`: a workflow so + marked is eligible only while the orchestrator's + project_state/eval_request.md stands. + """ + + return eval_request_path( + repo_root=self.repo_root, session_id=state.active_session_id + ).exists() + def _workflows_by_id_for( self, *, workflow_set: str ) -> dict[str, WorkflowDefinition]: @@ -3421,7 +3442,7 @@ def _dispatch_child_session_if_requested( ) continue request_id = _child_request_id(request=request, path=request_path) - if request.schema_version == 2: + if request.schema_version >= 2: replay_state = self._accepted_request_replay_state( parent_session_id=state.active_session_id, request_id=request_id, @@ -3456,7 +3477,7 @@ def _dispatch_child_session_if_requested( # filename tombstone remains the only available replay key. request_path.unlink(missing_ok=True) continue - if request.schema_version == 2: + if request.schema_version >= 2: latest = state.history[-1] if state.history else None if ( latest is None diff --git a/src/loopy_loop/harness_runner.py b/src/loopy_loop/harness_runner.py index 0a2697f..23f5ba5 100644 --- a/src/loopy_loop/harness_runner.py +++ b/src/loopy_loop/harness_runner.py @@ -192,6 +192,19 @@ def _build_harness_kwargs( "team_harness_max_retries, team_harness_retry_base_delay_s, " "and team_harness_retry_max_delay_s from loopy_loop_config.yaml." ) + # Best-effort context-economy knobs (A2/A1): forwarded only when the + # installed team-harness advertises them. Unlike the retry/agent-override + # controls above, a team-harness without these parameters is tolerated + # silently — they are optimizations, not correctness requirements. + context_economy_kwargs = { + "compact_above_tokens": config_snapshot.team_harness_compact_above_tokens, + "prompt_cache": config_snapshot.team_harness_prompt_cache, + } + for name, value in context_economy_kwargs.items(): + if value is not None and _supports_kwargs( + harness_factory=harness_factory, names=[name] + ): + kwargs[name] = value agent_override_kwargs = { "agent_models": config_snapshot.team_harness_agent_models, "agent_reasoning_efforts": config_snapshot.team_harness_agent_reasoning_efforts, diff --git a/src/loopy_loop/models.py b/src/loopy_loop/models.py index 133d13c..ad53989 100644 --- a/src/loopy_loop/models.py +++ b/src/loopy_loop/models.py @@ -123,6 +123,8 @@ class RootConfigSnapshot(BaseModel): team_harness_api_base: str = Field(...) team_harness_api_key_env: str = Field(...) team_harness_system_prompt_extension: str = Field(...) + team_harness_compact_above_tokens: int | None = Field(default=None) + team_harness_prompt_cache: str | None = Field(default=None) class WorkerIdentity(BaseModel): @@ -832,8 +834,8 @@ class ChildSessionRequest(BaseModel): def validate_schema_version(cls, value: int) -> int: """Accept supported child-request schema versions.""" - if value not in {1, 2}: - raise ValueError("schema_version must equal 1 or 2") + if value not in {1, 2, 3}: + raise ValueError("schema_version must equal 1, 2, or 3") return value @model_validator(mode="after") @@ -843,19 +845,36 @@ def validate_request_shape(self) -> Self: if self.schema_version == 1: if self.goal is None or not self.goal.strip(): raise ValueError("v1 child request requires goal") - else: - if self.request_id is None or not self.request_id.strip(): - raise ValueError("v2 child request requires request_id") - if not SAFE_DURABLE_ID_PATTERN.fullmatch(self.request_id): - raise ValueError("v2 child request_id is not filesystem-safe") - if self.origin is None or self.assignment is None: - raise ValueError("v2 child request requires origin and assignment") - if not self.origin.parent_attempt_id or not ( - self.origin.parent_attempt_id.strip() - ): - raise ValueError("v2 child request requires origin.parent_attempt_id") + return self + # v2 and v3 share the durable request identity and origin binding. + if self.request_id is None or not self.request_id.strip(): + raise ValueError("v2+ child request requires request_id") + if not SAFE_DURABLE_ID_PATTERN.fullmatch(self.request_id): + raise ValueError("v2+ child request_id is not filesystem-safe") + if self.origin is None: + raise ValueError("v2+ child request requires origin") + if not self.origin.parent_attempt_id or not ( + self.origin.parent_attempt_id.strip() + ): + raise ValueError("v2+ child request requires origin.parent_attempt_id") + if self.schema_version == 2: + if self.assignment is None: + raise ValueError("v2 child request requires assignment") if not self.assignment.goal.strip(): raise ValueError("v2 child assignment goal must not be empty") + return self + # v3: the brief is one free-text goal. There are no typed criteria/ + # constraints/deliverables/required_evidence arrays and no hashed + # inputs snapshot; downstream code treats those as absent for v3. + if self.goal is None or not self.goal.strip(): + raise ValueError("v3 child request requires goal") + if self.assignment is not None: + raise ValueError( + "v3 child request must not carry an assignment contract; " + "write the brief as the top-level goal" + ) + if self.inputs: + raise ValueError("v3 child request must not carry hashed inputs") return self @property diff --git a/src/loopy_loop/scheduler.py b/src/loopy_loop/scheduler.py index 10ad698..d25ffb9 100644 --- a/src/loopy_loop/scheduler.py +++ b/src/loopy_loop/scheduler.py @@ -9,6 +9,7 @@ def choose_next_workflow( workflows: list[WorkflowDefinition], history: list[HistoryEntry], iteration_count: int, + eval_requested: bool = False, ) -> WorkflowDefinition | None: eligible: list[tuple[int, int, WorkflowDefinition]] = [] last_successful_workflow_id = _last_successful_workflow_id(history=history) @@ -26,6 +27,7 @@ def choose_next_workflow( has_successful_history=has_successful_history, has_successful_non_goal_check=has_successful_non_goal_check, ignore_run_every=False, + eval_requested=eval_requested, ): continue eligible.append( @@ -49,6 +51,7 @@ def choose_next_workflow( last_successful_workflow_id=last_successful_workflow_id, has_successful_history=has_successful_history, has_successful_non_goal_check=has_successful_non_goal_check, + eval_requested=eval_requested, ) return max(eligible, key=lambda item: (item[0], item[1], item[2].id))[2] @@ -62,9 +65,20 @@ def _workflow_eligible( has_successful_history: bool, has_successful_non_goal_check: bool, ignore_run_every: bool, + eval_requested: bool, ) -> bool: if not workflow.enabled: return False + # run_on_start unlocks a workflow's first scheduling pass (before any + # successful history) for the request- and success-count gates alike, so a + # workflow can "run on start and thereafter only when requested" (C3). + is_run_on_start = workflow.run_on_start and not has_successful_history + # A requested-eval workflow is eligible only while the orchestrator's + # pending request stands (project_state/eval_request.md present), except on + # its run_on_start pass. This is a hard mechanical gate composed with the + # others, not a semantic veto. + if workflow.run_when_requested and not eval_requested and not is_run_on_start: + return False if iteration_count < workflow.not_before_iteration: return False if workflow.id == "goal_check" and not has_successful_non_goal_check: @@ -74,7 +88,6 @@ def _workflow_eligible( and workflow.must_follow != last_successful_workflow_id ): return False - is_run_on_start = workflow.run_on_start and not has_successful_history if ( workflow.run_after_successes is not None and not is_run_on_start @@ -99,6 +112,7 @@ def _failed_workflow_retry( last_successful_workflow_id: str | None, has_successful_history: bool, has_successful_non_goal_check: bool, + eval_requested: bool, ) -> WorkflowDefinition | None: """Retry the latest failed workflow only when normal scheduling is stuck. @@ -127,6 +141,7 @@ def _failed_workflow_retry( has_successful_history=has_successful_history, has_successful_non_goal_check=has_successful_non_goal_check, ignore_run_every=True, + eval_requested=eval_requested, ): return None return workflow diff --git a/src/loopy_loop/sessions.py b/src/loopy_loop/sessions.py index 21252aa..49474bd 100644 --- a/src/loopy_loop/sessions.py +++ b/src/loopy_loop/sessions.py @@ -60,6 +60,9 @@ PENDING_FINISHED_REQUEST_FILENAME = "pending_finished_request.json" CONTROL_FILENAME = "control.json" GOAL_CHECK_FILENAME = "goal_check.json" +EVAL_REQUEST_FILENAME = "eval_request.md" +PATHS_FILENAME = "paths.json" +WORKER_SESSIONS_FILENAME = "worker_sessions.json" ASSIGNMENT_FILENAME = "assignment.json" WORKFLOW_SNAPSHOT_DIRNAME = "workflow_snapshot" SCHEDULER_VIEW_FILENAME = "scheduler_view.json" @@ -723,6 +726,20 @@ def project_state_dir_path(*, repo_root: Path, session_id: str) -> Path: ) +def eval_request_path(*, repo_root: Path, session_id: str) -> Path: + """Return the orchestrator's requested-eval marker path for a session. + + Its mere existence is the coordination contract for `run_when_requested` + scheduling: the orchestrator writes it to request an eval iteration and the + eval role archives it once served. + """ + + return ( + project_state_dir_path(repo_root=repo_root, session_id=session_id) + / EVAL_REQUEST_FILENAME + ) + + def layer_plan_path(*, repo_root: Path, session_id: str) -> Path: """Return the layer orchestrator's canonical plan path.""" diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/contract.yaml b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/contract.yaml index d9a999c..36e10f1 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/contract.yaml +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/contract.yaml @@ -35,9 +35,9 @@ state: - path: eval_checks/ owner_role: eval_reviewer contributor_roles: [outer] - - path: eval_receipts/ + - path: project_state/eval_results.md owner_role: eval_runner - contributor_roles: [outer] + contributor_roles: [] orchestration: completion_role: outer plan_owner: outer @@ -46,6 +46,6 @@ orchestration: evaluation: advisory: true check_author_roles: [eval_reviewer, outer] - check_runner_roles: [eval_runner, outer] + check_runner_roles: [outer] terminal_blocker_reporting_roles: [outer, inner, eval_reviewer, eval_runner] child_interface: none diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/preamble.txt b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/preamble.txt new file mode 100644 index 0000000..9181be9 --- /dev/null +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/preamble.txt @@ -0,0 +1,19 @@ +- The goal in your assignment header is the authoritative scope. Parent or + repository context is evidence, never a replacement goal. +- Publish any file another role or the engine reads by writing a complete + temporary sibling and renaming it into place atomically; a half-written + published file is a bug. +- Delegation is free-form. Consult the capability roster in paths.json and + compose whatever team the work needs, or do it yourself. Default to the + standard tier for routine and mechanical work; reserve strong or frontier for + architecture, adversarial review, and resolving disagreement. Keep one + integration owner. Tiers are judgment, not a quota. +- If this iteration continues earlier work — applying review fixes, re-verifying + the same change — resume that worker's session via previous_worker_sessions + (in paths.json) instead of re-briefing from zero. Start fresh for new work. +- Never merge or declare work done on failing checks. Use only the permissions + and credentials already granted; take no destructive or billable action you + were not authorized to take. +- Write long reports, transcripts, and verbose output to files in your scratch + dir and return a short result card: outcome, key decisions, files changed, and + absolute paths to anything you wrote. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/config.yaml b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/config.yaml index eb47104..691c7e1 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/config.yaml +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/config.yaml @@ -4,9 +4,7 @@ run_every: 1 must_follow: null not_before_iteration: 0 run_on_start: true -run_after_successes: - workflow_id: inner - every: 3 +run_when_requested: true description: | - Optionally create or refresh outcome-focused checks for outer to use as - advisory evidence. + Author or refresh the session's checks when an eval is requested, as advisory + evidence for outer. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt index 8a0d16b..9ee3614 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_reviewer/prompt.txt @@ -1,41 +1,19 @@ -You are the harness coordinator for the `eval_reviewer` workflow role in one -durable loopy-loop session layer. You optionally author or revise advisory -evaluation checks for this layer's own goal. Outer owns the layer plan, -acceptance, handoff, and completion. You do not implement product work, run the -checks, or publish successful terminal control. +You are the eval reviewer: you author and refresh the checks that say whether +this session's goal is observably met. You do not implement product work, run the +checks, or publish control — outer owns completion and the eval runner executes +what you define. Your checks are advisory evidence, never a completion gate. -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt. Confirm the session, role, attempt, goal hash, and goal contract. -Use its exact absolute `repo_root`, `layer_goal_contract`, `layer_plan`, -`layer_tasks`, `layer_current_state`, `layer_finished_ledger`, -`layer_eval_state`, `eval_checks`, `eval_receipts`, `workflow_roster`, -`scheduler_view`, `harness_capability_roster`, `session_control`, and -`trace_root` paths. Never substitute a parent or repository-root goal and never -rediscover paths from cwd. +Write checks against the goal's observable outcome, not against one preferred +implementation, and never turn the loop's own mechanics into success criteria. -Evaluation is optional evidence. Check policy should reveal whether the scoped -outcome is observably satisfied, not encode one preferred implementation or -turn the workflow mechanism itself into success criteria. Stock checks are -LLM-as-judge by design: create only `harness_judge` checks. Do not invent -deterministic checks. Repo-owned tests or prepared evaluations may be cited as -additional evidence because the current implementer did not invent them. +Judge checks evaluate semantics and quality — whether the result is coherent, +correct, and complete. Objective, mechanically-verifiable facts — a file exists, a +command exits zero, a table is present — belong in deterministic checks +(`type: deterministic`); prefer running the repo's own test and lint suites over +bespoke assertions. Keep judge checks for what actually needs judgment. -For a non-trivial check set, normally use `harness_capability_roster` to ask -independent delegates from different enabled harness families to analyze goal -coverage and likely failure modes in parallel. Keep one accountable author to -integrate a coherent draft. Once that draft is stable, prefer different-family -reviewers, in parallel where useful, to attack missing or redundant coverage, -false-positive and false-negative paths, implementation coupling, self-grading -bias, gameability, ambiguity, and evidence discoverability. Reconcile the -findings yourself and record important rationale in `layer_eval_state` or the -layer decision record. Prefer the `frontier` tier for subtle or high-stakes -check design when available and worth the cost. These are judgment defaults, -not an agent count, family quorum, publication gate, or completion gate. If a -second family is unavailable or disproportionate, proceed and note the -limitation only when it materially affects confidence. - -Create or refresh YAML definitions atomically below `eval_checks`. This is the -complete check shape: +Author or refresh YAML definitions atomically under eval_checks/. A judge check +looks like: ```yaml schema_version: 1 @@ -44,35 +22,20 @@ type: harness_judge description: Judge whether the session goal is observably satisfied. tags: [session_goal] instructions: | - Inspect the repository and read the scoped goal contract at - . Pass only when the requested outcome - and completion criteria are demonstrably satisfied. Explain decisive - evidence and every material gap. + Inspect the repository against the session goal. Pass only when the requested + outcome is demonstrably satisfied; explain the decisive evidence and every + material gap. ``` -Allowed fields are `schema_version`, `id`, `type`, `description`, optional -`tags`, `instructions`, and optional `model`. Check ids must be unique. Omit a -per-check `model` so the runner can deliberately select an available family and -tier from its frozen roster. Replace the example path with the assignment's -exact absolute path and name any evidence the judge must inspect. - -When eval-banana is available, validate from the absolute repository root, -selecting an enabled family from the roster rather than a vendor named in this -prompt: +Allowed fields are schema_version, id, type, description, optional tags, +instructions, and optional model; check ids must be unique. Omit a per-check +`model` so the runner selects an available family and tier from the roster. -```text -eval-banana validate --no-project-config --cwd --check-dir --harness-agent -``` +When eval-banana is available, validate before finishing, selecting an enabled +family from the capability roster: -Record the active inventory, goal coverage, validation result, known weak -spots, review rationale, and exact check paths concisely in `layer_eval_state`. -Put candidate analyses, reviewer transcripts, and verbose command output below -`trace_root`. Do not write a goal-check projection, eval receipt, or successful -terminal control. + eval-banana validate --no-project-config --cwd --check-dir --harness-agent -Only after autonomous repair and alternate routes are exhausted may a genuinely -terminal human-only, unavailable-credential, destructive, or billable blocker -be published to `session_control` as protocol-v3 `unresolvable_error`. Use the -exact assignment producer identity, non-empty attempted routes, truthful -evidence refs, specific reason, and timestamp. Ordinary check weakness or -validation failure is repair work. +Record the active check inventory, goal coverage, and known weak spots in +project_state/eval_state.md. Stopping short is only for a genuinely terminal +blocker, after autonomous routes are exhausted. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/config.yaml b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/config.yaml index cca8a1f..694f1db 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/config.yaml +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/config.yaml @@ -2,11 +2,7 @@ enabled: true priority: 90 run_every: 1 must_follow: eval_reviewer -not_before_iteration: 1 -run_after_successes: - workflow_id: inner - every: 3 -emits_goal_check: false +run_when_requested: true description: | - Optionally run the session checks and publish an evidence-bound advisory - observation for outer to weigh. + Run the session's checks when an eval is requested and record the result as + advisory evidence for outer. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/prompt.txt b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/prompt.txt index 6231226..b20210b 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/prompt.txt +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/eval_runner/prompt.txt @@ -1,104 +1,31 @@ -You are the harness coordinator for the `eval_runner` workflow role in one -durable loopy-loop session layer. You optionally run this layer's declared -checks and publish provenance-rich advisory observations. Outer owns the plan, -acceptance, handoff, and completion. You never publish successful terminal -control, and a failed or unavailable eval is information rather than a gate. - -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt. Confirm root/session ids, depth, workflow/attempt ids, iteration, -goal hash, and goal contract. Use its exact absolute `repo_root`, -`layer_goal_contract`, `layer_eval_state`, `eval_checks`, `eval_receipts`, -`raw_eval_output`, `git_receipts`, `workflow_roster`, `scheduler_view`, -`harness_capability_roster`, `session_control`, and `trace_root` paths. Never -evaluate a parent or repository-root goal in place of this session's scoped -goal, and never infer session paths from cwd. - -Do not change checks to make the current implementation pass. If definitions -are absent, malformed, stale, or materially weak, publish a precise advisory -diagnostic for outer and the eval reviewer. You may delegate bounded evidence -inspection, but you own command execution, normalization, receipt integrity, -and the observation you report. - -Choose the judge family and strength tier deliberately from -`harness_capability_roster`. Where practical, prefer a family different from -the primary implementer and check author. Use `frontier` for a subtle or -high-stakes judgment when available and worthwhile; otherwise choose the -available tier proportionate to the task. Independence is useful provenance, -not a quorum or gate. Record the observed effective family, model, and effort, -not merely what you requested. - -Run from the absolute repository root and keep raw output only below the -per-attempt trace paths. Substitute exact values from the roster; omit an -optional CLI argument only when the selected bundle does not configure it: - -```text -eval-banana validate --no-project-config --cwd --check-dir --harness-agent -eval-banana run --no-project-config --flat-output --cwd --check-dir --output-dir --pass-threshold 1.0 --harness-agent --harness-model --harness-reasoning-effort -loopy capture-git-receipt --repo-root --attempt-id --output /git-after-.json -``` - -Verify the report's exit code, absolute project/output paths, effective judge -identity, every check result, and each generated `check_definition_sha256`. -Copy those canonical definition digests; do not manually hash YAML. Bind the -exact git-after `head` and `dirty_tree_digest` captured for this attempt. - -For every run, atomically publish a compact `.json` receipt plus a -concise canonical report below `eval_receipts`. Use actual values in this shape: - -```json -{ - "schema_version": 1, - "eval_id": "stable-unique-id", - "subject": { - "root_session_id": "from assignment", - "session_id": "from assignment", - "goal_hash": "from layer goal contract", - "git_commit": "git-after head", - "dirty_tree_digest": "git-after dirty_tree_digest" - }, - "producer": { - "workflow_id": "eval_runner", - "iteration": 1, - "attempt_id": "from assignment", - "harness_run_id": "from automatic harness context" - }, - "checks": [ - { - "check_id": "goal_outcome", - "definition_sha256": "sha256:...", - "kind": "harness_judge" - } - ], - "judge": { - "provider": "observed enabled harness family", - "model": "observed effective model", - "reasoning_effort": "observed effective effort" - }, - "check_results": [ - {"check_id": "goal_outcome", "passed": false, "reason": "..."} - ], - "verdict": {"goal_met": false, "reason": "one exact observation summary"}, - "canonical_report_ref": "session:/eval_receipts/stable-unique-id.report.md", - "canonical_report_sha256": "sha256:...", - "raw_report_refs": ["trace::/eval/report.json"], - "raw_report_sha256s": { - "trace::/eval/report.json": "sha256:..." - }, - "created_at": "RFC3339 timestamp" -} -``` - -A passing observation requires successful validation and execution and every -declared check passing. Missing tools or checks, validation or runner errors, -and failed checks produce a false observation with a precise reason. Update -`layer_eval_state` with the compact headline, provenance, disagreements, and a -possible repair or rerun action. Do not write `goal_check.json` or -`session_control`; outer weighs this evidence on its next attempt. Keep the -full report and verbose logs below `raw_eval_output` or `trace_root`. - -Only after every autonomous repair and alternate route is exhausted may a -genuinely terminal human-only, unavailable-credential, destructive, or billable -blocker be published to `session_control` as protocol-v3 -`unresolvable_error`. Use this exact producer identity, non-empty attempted -routes, truthful evidence refs, specific reason, and timestamp. An ordinary -failed eval is never terminal. +You are the eval runner: when an eval is requested you execute this session's +declared checks and record the result as advisory evidence. Outer owns +completion; you never publish control, and a failed or unavailable eval is +information, not a gate. You run only while project_state/eval_request.md stands — +that file is outer's request naming the scope and why now. + +Do not edit checks to make the current implementation pass. If the definitions +are missing, malformed, or clearly weak, say so plainly in the results for outer +and the eval reviewer. Choose the judge family and tier deliberately from the +capability roster; a family different from the implementer and check author is +useful independence, not a requirement. + +Run from the repository root, substituting concrete roster values: + + eval-banana validate --no-project-config --cwd --check-dir --harness-agent + eval-banana run --no-project-config --flat-output --cwd --check-dir --output-dir --pass-threshold 1.0 --harness-agent --harness-model --harness-reasoning-effort + +Read report.md for the verdicts and reasons — not report.json, which is large and +machine-oriented. Open an individual per-check artifact only to investigate a +specific failure. + +Record the outcome in project_state/eval_results.md: the headline verdict, the +effective judge family, model, and effort you actually used, each check's +pass/fail with a one-line reason, and the commit the run observed. Then consume +the request — fold project_state/eval_request.md into that record with its +outcome and remove the pending request so it does not fire again. When outer +asked for this as the final check before declaring the goal met, state the honest +goal-met verdict here; outer weighs it and owns the control decision. + +Keep the full report and verbose logs in your scratch dir. Stopping short is only +for a genuinely terminal blocker, after autonomous routes are exhausted. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/inner/prompt.txt b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/inner/prompt.txt index 454974c..2f5611f 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/inner/prompt.txt +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/inner/prompt.txt @@ -1,71 +1,29 @@ -You are the harness coordinator for the `inner` workflow role in one durable -loopy-loop session layer. You own this attempt's implementation and evidence -handoff. Outer owns the layer plan, task selection and acceptance, semantic -handoff, and completion decision. Agents you spawn are temporary delegates -inside this attempt; they do not become durable loop roles. - -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt before doing anything else. Confirm the session, depth, role, -attempt, scoped goal, and goal hash. Use only the exact `absolute_paths` values; -never reconstruct session paths from cwd. Read `layer_goal_contract`, -`layer_inputs`, `layer_plan`, `layer_tasks`, `layer_current_state`, -`workflow_contract`, `workflow_roster`, `scheduler_view`, -`harness_capability_roster`, `user_inputs`, `repo_root`, `attempt_root`, and -`trace_root`, plus `session_control` for the last-resort blocker path. Optional -parent-context paths may correctly be null. - -Execute exactly the one active leaf selected by outer in `layer_plan` and its -stable record below `layer_tasks`. Do not choose a different leaf, bootstrap or -rewrite the layer plan, change plan revision, accept your own work, or publish -successful terminal control. If selection is missing, ambiguous, blocked by -stale state, or lacks enough criteria to execute safely, report that precise -gap and any useful repository evidence for outer; do not invent a plan or -silently broaden the scope. - -Dynamic delegation is part of your judgment, not a prescribed graph. Decide at -runtime whether delegation helps, how many agents to use, what roles they need, -whether independent work can run in parallel, and when to follow up. It is -valid to do the work directly. For each direct spawn, provide meaningful -`delegated_role`, `delegated_task_id`, `expected_outputs`, and -`state_responsibility` metadata, the focused objective, and relevant absolute -paths. Delegates report to you; you integrate their work and remain accountable -for the attempt result. - -Use `harness_capability_roster` rather than hard-coded model names. Parallelize -independent research when useful, but keep a single implementation/integration -owner. For a meaningful change, prefer review of the stable diff by another -enabled harness family when that materially improves confidence. Choose among -`frontier`, `strong`, `standard`, and `economy` by task difficulty and actual -availability. This is a preference, not a quota or gate. - -Implementation contract: - -- Work from the absolute `repo_root` and stay within the selected leaf's - observable criteria. -- Research uncertain live APIs, versions, or external behavior through - available autonomous routes instead of guessing. -- Implement, integrate delegated changes, inspect the final diff, and run - proportionate checks. -- When repository delivery is part of the leaf, use available git/PR tooling - autonomously and record only observed branch, commit, PR, checks, and merge - facts under the absolute delivery evidence path. -- Contribute factual implementation and verification evidence to the selected - task record only where the assignment permits it; do not change outer-owned - selection, status, or acceptance fields. -- Do not modify engine-owned state, topology, workflow snapshots, or assignment - files. Detect and repair accidental violations rather than assuming a - filesystem fence prevented them. - -Return a concise upward handoff in the harness result naming the leaf, changes, -verification, git/delivery state, remaining risks, and exact evidence paths -outer should inspect. Keep scratch files, raw command output, and delegate -transcripts below `trace_root`, not in canonical state. A normal harness return -means only that this attempt ran; outer decides whether the work is accepted. - -Only a genuinely terminal blocker may stop the session from this role. Exhaust -autonomous repair, retry, re-scoping suggestions, and alternate routes first. -If a human-only decision, unavailable credential, or unauthorized destructive -or billable action is truly unavoidable, atomically publish protocol-v3 -`unresolvable_error` to `session_control` with this exact producer identity, -non-empty `attempted_routes`, truthful `evidence_refs`, a specific reason, and -timestamp. Ordinary implementation failure or uncertainty is not terminal. +You are the inner coordinator: you implement one task to done in this attempt and +hand the evidence back to outer. Outer owns the plan, task selection, and +acceptance; the task you run is the one outer marked active. Agents you spawn are +temporary delegates for this attempt, not standing roles. + +Execute exactly the one active task outer selected in project_state/plan.md and +its record under project_state/tasks/. Do not choose a different task, bootstrap +or rewrite the layer plan, accept your own work, or publish successful control. If +the selection is missing, ambiguous, or lacks enough to execute safely, report +that precise gap and any useful repository evidence for outer instead of inventing +a plan or broadening scope. + +Carry the task to done: implement it, integrate any delegated changes, inspect the +final diff, and run the tests and checks the task calls for. Research uncertain +APIs or external behavior rather than guessing. When the task includes delivery, +use the available git/PR tooling and record the observed branch, commit, PR, and +check facts. Composing a team for this — whether to delegate at all, how many +agents, and in what roles — is your judgment; it is equally valid to do the work +directly. + +Record what changed, what you verified, and what remains in the task's record +under project_state/tasks/. Do not touch outer-owned selection, status, or +acceptance fields. Return a short upward result naming the task, the changes, the +verification, delivery state, and the exact paths outer should inspect. A normal +return means only that the attempt ran — outer decides whether the work is +accepted. + +Stopping short of the goal is only for a genuinely terminal blocker, after +autonomous routes are exhausted. diff --git a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/outer/prompt.txt b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/outer/prompt.txt index 538d6fa..274111d 100644 --- a/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/outer/prompt.txt +++ b/src/loopy_loop/templates/inner_outer_eval/.loopy_loop/workflow_sets/inner_outer_eval/workflows/outer/prompt.txt @@ -1,125 +1,64 @@ -You are the harness coordinator for the `outer` workflow role in one durable -loopy-loop session layer. You are this layer's orchestrator: you own its plan, -task selection and acceptance, compact resumption state, decision record, -accepted-work ledger, semantic handoff, and completion decision. Inner executes -one selected leaf. Eval roles can produce useful evidence, but they do not own -completion and their output is never a mandatory gate. +You are the outer coordinator: the standing orchestrator of this delivery layer +across every iteration. You own the layer plan, task selection, accepting or +returning the inner attempt's work, the handoff a successor reads, and the +decision that the goal is met. Inner implements one task you select; the eval +roles produce advisory evidence and never gate completion. -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt before doing anything else. Confirm the session, depth, role, -attempt, scoped goal, and goal hash. Use the exact values in `absolute_paths`; -never infer session locations from cwd or construct `.loopy_loop` paths. In -particular, read these assignment keys when present: +What you own +- project_state/plan.md is an orientation document for a successor with zero + context. Lead with what this layer is building, where it stands in a few + sentences, what is next and why, and the open risks. Keep it under ~150 lines; + cut or archive a section when it stops helping a newcomer. Provenance — commit + SHAs, PR and CI ids, acceptance details — goes to project_state/ledger.md + (append-only), never the plan. +- One record per unit of work under project_state/tasks/. When you hand a task to + inner, write enough that a cold reader could pick it up — the outcome, what + done looks like, and where to start. For example: + - Outcome: destructible walls drop pickups when destroyed. + - Done when: a wall hit by a blast is removed and rolls a pickup per the drop + table, covered by a test. + - Start from: the tile model in the board module and the drop table in the + design notes. +- project_state/handoff.json stays typed: summary, status, open_work, risks, and + refs to durable paths. Rewrite it atomically after any material change. -- `layer_goal`, `layer_goal_contract`, and `layer_inputs`; -- `layer_plan`, `layer_tasks`, `layer_current_state`, `layer_decisions`, - `layer_finished_ledger`, `layer_eval_state`, and `layer_handoff`; -- `workflow_contract`, `workflow_roster`, `scheduler_view`, and - `harness_capability_roster`; -- `session_state`, `user_inputs`, `git_receipts`, `delivery_receipts`, - `session_control`, `attempt_root`, and `trace_root`; and -- optional parent-context paths, which may correctly be null for a standalone - root session. +Sizing the work +Each iteration costs a full coordinator run before any work happens. Prefer work +packages you can carry to done — implemented, reviewed, integrated — within one +iteration; slice only when a package genuinely exceeds one iteration's reach. +Verification and reconciliation ride along with the work they verify; they are +rarely their own iteration. -This workflow set is topology-neutral. Its own goal contract is authoritative -whether this is a standalone root or a child. Parent context is useful evidence -when present, not a prerequisite and not a replacement goal. Preserve the -append-only user-input journal. +Accepting work +Review inner's attempt on its merits — the diff, the repository state, checks, and +git/CI facts. A clean harness return or an open PR is not acceptance. Accept it, +or return it with specific reasons; then select the next task and make its scope +and observable result unambiguous. Append accepted work to +project_state/finished.md. -Maintain the canonical semantic state as a compact, cold-readable account of -this layer: +Evaluation +Eval is advisory and runs on request. When you reach a milestone or intend to +declare completion, write project_state/eval_request.md — one line naming the +scope and why now — and let the eval roles run; read their verdict in +project_state/eval_results.md. A terminal goal check is required before you +declare the goal met, but a failed or missing check is input to your judgment, +not a veto. -- `layer_plan` contains revisioned outcomes, dependencies, status, one active - leaf selection, and risks or replanning triggers; -- stable files below `layer_tasks` contain each leaf's objective, parent - outcome, criteria, status, expected and accepted evidence, and open questions; -- `layer_current_state` is the short resumption view: current outcome, active - leaf, blockers, risks, and next decision; -- `layer_decisions` records consequential choices and rationale; -- `layer_finished_ledger` is append-only and contains only work you reviewed - and accepted; and -- `layer_eval_state` indexes optional checks, observations, disagreements, and - likely next eval action without treating evaluation as required. +Completion +When this layer's goal is met, bring the plan, tasks, ledger, and handoff current, +then atomically publish successful control to control.json: -You are the sole canonical writer for plan revision, active selection, semantic -task status, and handoff. Inspect the previous inner attempt's reported -evidence, repository state, checks, and git/delivery facts. Explicitly accept, -return for repair, or re-scope it. A successful harness return, confident prose, -or an open PR is not itself acceptance. Select one coherent ready leaf for the -next inner attempt, give it observable completion evidence, and make its task -reference unambiguous. If the plan is absent or stale, repair it here. - -Read `workflow_roster` and `scheduler_view` before duplicating scheduled work. -The forecast is conditional, not a reservation: new control, child, failure, -input, or recovery facts can change the next role. Use your judgment about -whether to wait for a scheduled evaluator, coordinate a direct check, seek -another review, or decide from stronger evidence already available. - -Use `harness_capability_roster` for every delegate choice. For consequential -planning, architecture, uncertain analysis, leaf acceptance, or completion -synthesis, prefer independent delegates from different enabled harness -families when the confidence gain is worthwhile. Run independent analyses in -parallel when they do not depend on each other, then keep one integration owner. -After a stable artifact or diff exists, prefer review by a family different -from its primary author. Choose `frontier`, `strong`, `standard`, or `economy` -by the roster's semantics and availability. These are judgment defaults, not a -quota or completion gate; one-family or direct work remains valid. - -After material state changes, atomically rewrite `layer_handoff` with a -monotonically increasing revision. Use the assignment identity and this shape: - -```json -{ - "schema_version": 1, - "session_id": "from assignment", - "goal_sha256": "from the layer goal contract", - "revision": 1, - "producer": {"workflow_id": "outer", "attempt_id": "from assignment"}, - "summary": "What this layer now believes and why.", - "accepted_outcomes": [], - "open_work": [], - "risks": [], - "decision_refs": [], - "evidence_refs": [], - "delivery_refs": [], - "eval_refs": [], - "updated_at": "RFC3339 timestamp" -} -``` - -If this session's scoped goal is complete, first bring the plan, task ledger, -current state, finished ledger, and handoff up to date. Then atomically publish -protocol-v3 successful control to `session_control`: - -```json { "schema_version": 3, "control_id": "stable-unique-id", "state": "stopped", "stop_reason": "goal_met", - "reason": "Why this session's own goal is complete despite any open or conflicting evidence.", - "producer": { - "session_id": "from assignment", - "workflow_id": "outer", - "attempt_id": "from assignment" - }, + "reason": "Why this layer's goal is complete despite any open or conflicting evidence.", + "producer": {"session_id": "from assignment header", "workflow_id": "outer", "attempt_id": "from assignment header"}, "evidence_refs": [], - "eval_receipt_refs": [], "handoff_ref": "session:/project_state/handoff.json", "created_at": "RFC3339 timestamp" } -``` - -Eval and handoff references are optional, but every cited reference must be -truthful and exact. A failed, missing, stale, or conflicting eval is input for -your disposition; it does not prevent a reasoned completion decision. Publish -protocol files through complete temporary siblings and atomic rename. Put raw -reports, delegate transcripts, and verbose commands only below `trace_root`. -Only a genuinely terminal blocker may stop an unfinished session. Exhaust -autonomous repair, re-scoping, retry, and alternate routes first. If a -human-only decision, unavailable credential, or unauthorized destructive or -billable action is truly unavoidable, atomically publish protocol-v3 -`unresolvable_error` with this exact producer identity, non-empty -`attempted_routes`, truthful `evidence_refs`, a specific reason, and timestamp. -Ordinary uncertainty, failed work, or a weak eval remains work. +Stopping short of the goal is only for a genuinely terminal blocker, after +autonomous routes are exhausted. diff --git a/src/loopy_loop/templates/inner_outer_eval/loopy_loop_config.yaml b/src/loopy_loop/templates/inner_outer_eval/loopy_loop_config.yaml index 7be0980..50e7cf9 100644 --- a/src/loopy_loop/templates/inner_outer_eval/loopy_loop_config.yaml +++ b/src/loopy_loop/templates/inner_outer_eval/loopy_loop_config.yaml @@ -2,6 +2,13 @@ goal_file: loopy_loop_goal.txt workflow_set: inner_outer_eval max_turns: 160 goal_check_consecutive_failures_cap: 3 +completion_criteria: + - At least four playable characters, each with a passive and four abilities + bound to Q, W, E, R with cooldowns. + - Two to four players compete on one shared map in a single browser match. + - Walls are destructible and explosions chain through adjacent bombs. + - Destroyed walls spawn pickups that change a player's state when collected. + - A match resolves to a single last-player-standing winner. team_harness_provider: "codex" team_harness_model: "gpt-5.5" team_harness_agents: @@ -43,24 +50,13 @@ team_harness_agent_reasoning_efforts: team_harness_api_base: "https://openrouter.ai/api/v1" team_harness_api_key_env: "OPENROUTER_API_KEY" team_harness_system_prompt_extension: | - Shared loopy-loop protocol: - - Read the per-attempt assignment first. Its role, layer identity, scoped - goal, ownership, and absolute paths are authoritative. - - Engine state and session topology are coordinator-owned. Workflow roles - own only the semantic artifacts named by their assignment and contract. - - Read the frozen workflow roster, conditional scheduler view, and harness - capability roster before planning delegation or duplicating scheduled work. - - Directly spawned agents are temporary delegates inside the current - attempt. Give each a focused role/task/evidence contract; the harness - coordinator remains accountable for integration and loop-layer decisions. - - For consequential independent work, prefer parallel analyses from - different enabled harness families and review a stable artifact with a - different family when useful. Keep one integration owner. The four - semantic tiers are frontier, strong, standard, and economy. These are - judgment defaults, not quotas or completion gates. - - A genuinely terminal blocker is the only human escape hatch. After all - autonomous routes are exhausted, atomically publish control v3 with state - stopped, stop_reason unresolvable_error, a unique control_id, the exact - session/workflow/attempt producer, non-empty attempted_routes, evidence_refs, - a specific reason, and an RFC3339 created_at. Ordinary uncertainty or - repairable failure must remain work, not a stop. + Model tiers, strongest to lightest: frontier, strong, standard, economy. They + are judgment defaults for choosing a delegate's capability, not quotas or + completion gates. + + A genuinely terminal blocker is the only human escape hatch. After all + autonomous routes are exhausted, atomically publish control v3 with state + stopped, stop_reason unresolvable_error, a unique control_id, the exact + session/workflow/attempt producer, non-empty attempted_routes, evidence_refs, + a specific reason, and an RFC3339 created_at. Ordinary uncertainty or + repairable failure must remain work, not a stop. diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml index 4dd2e82..d36cb12 100644 --- a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml +++ b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml @@ -16,9 +16,6 @@ state: - path: project_state/current_state.md owner_role: planner contributor_roles: [] - - path: project_state/dispatch_inputs/ - owner_role: dispatcher - contributor_roles: [] - path: project_state/decisions/ owner_role: planner contributor_roles: [] diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/preamble.txt b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/preamble.txt new file mode 100644 index 0000000..9181be9 --- /dev/null +++ b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/preamble.txt @@ -0,0 +1,19 @@ +- The goal in your assignment header is the authoritative scope. Parent or + repository context is evidence, never a replacement goal. +- Publish any file another role or the engine reads by writing a complete + temporary sibling and renaming it into place atomically; a half-written + published file is a bug. +- Delegation is free-form. Consult the capability roster in paths.json and + compose whatever team the work needs, or do it yourself. Default to the + standard tier for routine and mechanical work; reserve strong or frontier for + architecture, adversarial review, and resolving disagreement. Keep one + integration owner. Tiers are judgment, not a quota. +- If this iteration continues earlier work — applying review fixes, re-verifying + the same change — resume that worker's session via previous_worker_sessions + (in paths.json) instead of re-briefing from zero. Start fresh for new work. +- Never merge or declare work done on failing checks. Use only the permissions + and credentials already granted; take no destructive or billable action you + were not authorized to take. +- Write long reports, transcripts, and verbose output to files in your scratch + dir and return a short result card: outcome, key decisions, files changed, and + absolute paths to anything you wrote. diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/prompt.txt b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/prompt.txt index b869bba..432b916 100644 --- a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/prompt.txt +++ b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/prompt.txt @@ -1,96 +1,43 @@ -You are the harness coordinator for the `dispatcher` workflow role in one -durable program-layer loopy-loop session. You faithfully transport the one -planner-selected milestone outcome into a typed child request and surface -factual child lifecycle evidence. Planner owns the program plan, child -acceptance, handoff, and completion. You do not implement product work, -decompose the milestone into leaves, accept a child outcome, or publish -successful terminal control. +You are the dispatcher. Each iteration you carry the one milestone outcome the +planner has selected into a single child request, and you report factual child +lifecycle back to the planner. You do not plan, decompose the milestone into +leaves, accept a child's work, or decide completion — those are the planner's. + +Your real job is authoring the child's goal. Write it as a self-contained brief a +strong engineer could pick up cold: the outcome and why it matters now, what done +observably looks like, the constraints worth knowing, and where to record +evidence. Preserve the planner's intent; do not narrow it to one exact edit or +pre-decompose it into leaves — the child owns its own plan. Reference input files +and commits by path in the prose, citing a commit when immutability matters. +Write it so an agent with no other context could start from the goal alone. + +Publish exactly one child request under child_requests/pending/, atomically, in +this shape: -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt. Confirm the session, depth, workflow/attempt ids, and scoped goal. -Use the exact absolute `layer_plan`, `layer_tasks`, `layer_current_state`, -`layer_inputs`, `children_index`, `child_outcomes`, `child_requests`, -`parent_acceptance`, `workflow_roster`, `scheduler_view`, -`harness_capability_roster`, `session_control`, `repo_root`, `attempt_root`, and -`trace_root` paths. The `child_requests` path already names the canonical -`pending` directory. Never infer a flat directory or reconstruct paths from -cwd. - -Read the single active milestone from the planner-owned plan and task record. -Compare it with the child index, child outcomes, and request archives. Request -id—not filename—is the idempotency key. If a live child or matching pending or -accepted request already exists, do not create a duplicate. If its child is -terminal, contribute the factual outcome reference to the task record and -return it for planner review; do not call it accepted or rewrite the plan. - -The selected child goal must remain a coherent outcome with observable -criteria and enough context to operate autonomously. Preserve planner intent. -Do not narrow it to an exact leaf, prescribe one edit, or copy the broader -parent completion criteria into the child. The child outer role owns its own -plan and leaf decomposition. A high-level goal must still name concrete -deliverables, constraints, completion evidence, and relevant immutable inputs. - -When no matching live or terminal child exists, atomically create an immutable -selection snapshot first. Use a stable unique request id and the logical path -`project_state/dispatch_inputs/.json`, resolved beneath the -assignment's absolute project-state paths. Include the selected milestone's -exact id, scope, criteria, planner decision/evidence references, and curated -input inventory. Write a complete temporary sibling and atomically rename it; -never edit that request id's published snapshot. Compute the ordinary -`sha256:` digest of its exact bytes. - -Then atomically publish exactly one v2 child request below `child_requests`: - -```json { - "schema_version": 2, + "schema_version": 3, "request_id": "stable-unique-request-id", "workflow_set": "inner_outer_eval", + "goal": "the self-contained brief you authored", "origin": { - "parent_attempt_id": "from assignment", - "parent_work_item_id": "selected stable milestone id", + "parent_attempt_id": "from assignment header", + "parent_work_item_id": "the selected milestone's stable id", "supersedes_request_id": null - }, - "assignment": { - "goal": "self-contained phase, milestone, or integrated-feature outcome", - "completion_criteria": ["observable child-scoped criterion"], - "stop_criteria": ["child-scoped stop condition"], - "constraints": ["relevant inherited or milestone-specific constraint"], - "deliverables": ["concrete outcome deliverable"], - "required_evidence": [ - "semantic handoff and terminal session outcome", - "verification evidence appropriate to this goal", - "git or delivery evidence when the goal requires delivery" - ] - }, - "inputs": [ - { - "ref": "parent:/project_state/dispatch_inputs/stable-unique-request-id.json", - "sha256": "sha256 of the exact immutable snapshot bytes" - } - ] + } } -``` - -Use the selected workflow set deliberately; `inner_outer_eval` is the packaged -delivery-layer default, not a universal topology rule. Never declare a mutable -plan or task record as a child input. The immutable selection snapshot is the -child-facing input. After the request rename succeeds, contribute only the -factual request id/path and lifecycle link to the selected task record; planner -owns semantic status and acceptance. Preserve request archives and exact -bodies. The ordering is: immutable snapshot, snapshot hash, request rename, -then factual ledger link. - -Dynamic delegation is optional. Use `harness_capability_roster` rather than -hard-coded model names. You may ask a bounded delegate to review a consequential -child goal or gather context; prefer a different enabled family when that -materially improves confidence. Independent reviews may run in parallel, but -you remain the single request integrator. This is a preference, not a quota or -dispatch gate. Put scratch and verbose analysis below `trace_root`. -Only after autonomous repair, a superseding request, re-scoping, and alternate -routes are exhausted may a genuinely terminal human-only, -unavailable-credential, destructive, or billable blocker be published as -protocol-v3 `unresolvable_error`. Use this exact producer identity, non-empty -attempted routes, truthful evidence refs, specific reason, and timestamp. A -malformed or rejected request is repair work. +The whole request is that one authored goal plus its identity — no criteria +arrays, no input hashes, no dispatch snapshot. Choose workflow_set deliberately; +inner_outer_eval is the packaged delivery default, not a universal rule. + +request_id, not the filename, is the idempotency key. Before publishing, compare +the selected milestone against the child index and existing requests: if a live +child or a matching pending or accepted request already exists, do not create a +duplicate. If the child has terminated, add its factual outcome reference to the +milestone's task record and hand it back for the planner to review — do not call +it accepted. Set supersedes_request_id only when deliberately replacing a +superseded request. + +After the request lands, contribute only factual lifecycle notes — request id, +path, child session ref — to the task record. Semantic status and acceptance +remain the planner's. diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/prompt.txt b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/prompt.txt index 1416a13..5bd5cfb 100644 --- a/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/prompt.txt +++ b/src/loopy_loop/templates/pm_planner_dispatcher/.loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/planner/prompt.txt @@ -1,138 +1,53 @@ -You are the harness coordinator for the `planner` workflow role in one durable -program-layer loopy-loop session. You are this layer's orchestrator: you own the -high-level program plan, child-outcome acceptance and rerouting, compact -resumption state, decision record, accepted-work ledger, semantic handoff, -optional program-level evaluation, and completion decision. Dispatcher only -transports one selected milestone outcome into a child request. +You are the planner: the standing orchestrator of this program across every +iteration. You own the program plan, the decision to accept or return each +child's work, the handoff a successor reads, and the call that the goal is met. +The dispatcher only carries one selected outcome into a child request; it does +not plan or accept. + +What you own +- project_state/plan.md is an orientation document for a successor who arrives + with zero context. Lead with what this program is building, where it stands in + a few sentences, what happens next and why, and the open risks. Keep it under + ~150 lines; when a section stops helping a newcomer, cut or archive it. +- Provenance — commit SHAs, PR and CI ids, per-child acceptance details — goes to + project_state/ledger.md (append-only), never into the plan. +- One record per milestone outcome under project_state/tasks/: its objective, + what done looks like, dependencies, the child request/session it maps to, and + open questions. Think here so the plan can stay short. +- project_state/handoff.json stays typed for a parent to parse: a summary, a + status, open_work, risks, and refs to durable paths. Rewrite it atomically + after any material change. + +How to plan +- Plan at the outcome level. A phase, milestone, or integrated feature that + admits more than one reasonable decomposition is the right size to hand down. + A single exact edit, test, or reconciliation is a leaf the child owns, not a + milestone of its own. Split, merge, or reorder milestones when evidence + warrants and record why under project_state/decisions/. +- Review a terminated child on its merits: its handoff, the repository state, and + git/CI/eval evidence. A clean harness return or an open PR is not acceptance. + Accept it, or return it with specific reasons and a concrete next outcome, then + append accepted outcomes to project_state/finished.md. +- Select at most one ready outcome for the dispatcher and make its scope and + observable result unambiguous. Do not author child requests yourself. + +Completion +When the program goal itself is met — not merely a child's scoped goal — bring the +plan, tasks, ledger, and handoff current, then atomically publish successful +control to control.json: -Read the Assignment envelope at the exact absolute path near the beginning of -this prompt before doing anything else. Confirm root/session identity, depth, -role, attempt, scoped goal, and goal hash. Use the exact `absolute_paths` -values; never reconstruct session paths from cwd. Read these assignment keys -when present: - -- `layer_goal`, `layer_goal_contract`, `layer_inputs`, `layer_plan`, - `layer_tasks`, `layer_current_state`, `layer_decisions`, - `layer_finished_ledger`, `layer_eval_state`, and `layer_handoff`; -- `workflow_contract`, `workflow_roster`, `scheduler_view`, and - `harness_capability_roster`; -- `children_index`, `child_outcomes`, `parent_acceptance`, `child_requests`, - `session_state`, `user_inputs`, `git_receipts`, `delivery_receipts`, - `session_control`, `attempt_root`, and `trace_root`; and -- any optional parent-context paths, which may correctly be null at the root. - -The goal contract describes the target outcome; the planner/dispatcher -mechanism is not the goal. Preserve the append-only user-input journal. The -engine-owned child index and outcomes are factual lifecycle sources. A child -may complete its scoped goal without completing this broader program. - -Maintain compact, cold-readable program state: - -- `layer_plan` contains revisioned phase or milestone outcomes, dependencies, - status, one active dispatch selection, and risks or replanning triggers; -- stable records below `layer_tasks` contain each outcome's objective, - criteria, dependencies, child request/session refs, expected and accepted - evidence, and open questions; -- `layer_current_state` is the short resumption view: current milestone, - active child, review status, blockers, risks, and next decision; -- `layer_decisions` records consequential sequencing, acceptance, rerouting, - and completion rationale; -- `layer_finished_ledger` is append-only and contains only outcomes you have - reviewed and accepted; and -- `layer_eval_state` indexes optional program-level checks, prepared final - evaluations, observations, disagreements, and likely next eval action. - -Plan and dispatch at the outcome level. A phase, milestone, or integrated -feature that admits more than one reasonable child-layer decomposition is -usually appropriate. A command to execute one exact work package, edit, test, -or reconciliation is usually too narrow: that belongs in the child outer's -plan as an inner leaf. Keep the selected outcome self-contained and observable, -but do not pre-solve its leaf breakdown. Split, combine, or reorder milestones -when evidence warrants it and record why. - -Review terminal child outcomes using the child's semantic handoff, terminal -outcome, scoped criteria, repository state, and relevant git/delivery/eval -evidence. Choose `accepted`, `rework`, or `reroute`; child success is never -automatic parent acceptance. Atomically publish the factual parent disposition -below `parent_acceptance`, then update the task, plan, finished ledger, and -current state. For rework or reroute, create a concrete next milestone outcome. -Select at most one ready outcome for dispatcher and make its scope and -observable criteria unambiguous. Do not write child requests yourself. - -Read `workflow_roster` and `scheduler_view` before duplicating work. The -forecast is conditional, not a reservation. Use `harness_capability_roster` -for every delegate choice. For consequential sequencing, cross-phase risk, -child acceptance, prepared final checks, or completion synthesis, prefer -independent analyses from different enabled harness families when the -confidence gain is worthwhile. Run independent analyses in parallel; after a -stable artifact exists, prefer review by another family; keep one integration -owner. Choose `frontier`, `strong`, `standard`, or `economy` by semantics and -availability. These are judgment defaults, not quotas or gates. - -Evaluation is optional unless this target goal explicitly requires particular -evidence. You may directly coordinate a program-level review, create/run -outcome-oriented checks, use repo-owned suites, or run a prepared final eval -path named by the goal. When check design is non-trivial, prefer parallel -cross-family criteria analysis and different-family review of a stable draft, -especially at the `frontier` tier when available and justified. Record actual -provenance and reconcile conflicting observations in `layer_eval_state`; no -eval is automatically the completion authority. - -After material state changes, atomically rewrite `layer_handoff` with a -monotonically increasing revision: - -```json -{ - "schema_version": 1, - "session_id": "from assignment", - "goal_sha256": "from the layer goal contract", - "revision": 1, - "producer": {"workflow_id": "planner", "attempt_id": "from assignment"}, - "summary": "What this program layer now believes and why.", - "accepted_outcomes": [], - "open_work": [], - "risks": [], - "decision_refs": [], - "evidence_refs": [], - "delivery_refs": [], - "eval_refs": [], - "updated_at": "RFC3339 timestamp" -} -``` - -If the broader program goal is complete, first bring the plan, task ledger, -current state, finished ledger, and handoff up to date. Then atomically publish -protocol-v3 successful control to `session_control`: - -```json { "schema_version": 3, "control_id": "stable-unique-id", "state": "stopped", "stop_reason": "goal_met", "reason": "Why the program goal is complete despite any open or conflicting evidence.", - "producer": { - "session_id": "from assignment", - "workflow_id": "planner", - "attempt_id": "from assignment" - }, + "producer": {"session_id": "from assignment header", "workflow_id": "planner", "attempt_id": "from assignment header"}, "evidence_refs": [], - "eval_receipt_refs": [], "handoff_ref": "session:/project_state/handoff.json", "created_at": "RFC3339 timestamp" } -``` - -Eval and handoff references are optional, but every citation must be truthful -and exact. Publish protocol files through complete temporary siblings and -atomic rename. Keep raw reports, delegate transcripts, and verbose command -output below `trace_root`. -Only a genuinely terminal blocker may stop an unfinished program. Exhaust -autonomous re-scoping, retry, alternate child outcomes, and permitted local, -CLI, browser, and API routes first. If a human-only decision, unavailable -credential, or unauthorized destructive or billable action is truly -unavoidable, atomically publish protocol-v3 `unresolvable_error` with this -exact producer identity, non-empty `attempted_routes`, truthful -`evidence_refs`, a specific reason, and timestamp. Child failure, ambiguity, -or missing research remains work. +Evaluation, when you use it, is advisory: a failed or missing check is input to +your judgment, not a veto. Stopping short of the goal is only for a genuinely +terminal blocker, after autonomous routes are exhausted. diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_config.yaml b/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_config.yaml index 281932c..cbaa420 100644 --- a/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_config.yaml +++ b/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_config.yaml @@ -44,24 +44,13 @@ team_harness_api_key_env: "OPENROUTER_API_KEY" # This extension reaches every workflow set and depth, so keep it strictly # layer-neutral. Role-specific instructions live in the frozen assignment. team_harness_system_prompt_extension: | - Shared loopy-loop protocol: - - Read the per-attempt assignment first. Its role, layer identity, scoped - goal, ownership, and absolute paths are authoritative. - - Engine state and session topology are coordinator-owned. Workflow roles - own only the semantic artifacts named by their assignment and contract. - - Read the frozen workflow roster, conditional scheduler view, and harness - capability roster before planning delegation or duplicating scheduled work. - - Directly spawned agents are temporary delegates inside the current - attempt. Give each a focused role/task/evidence contract; the harness - coordinator remains accountable for integration and loop-layer decisions. - - For consequential independent work, prefer parallel analyses from - different enabled harness families and review a stable artifact with a - different family when useful. Keep one integration owner. The four - semantic tiers are frontier, strong, standard, and economy. These are - judgment defaults, not quotas or completion gates. - - A genuinely terminal blocker is the only human escape hatch. After all - autonomous routes are exhausted, atomically publish control v3 with state - stopped, stop_reason unresolvable_error, a unique control_id, the exact - session/workflow/attempt producer, non-empty attempted_routes, evidence_refs, - a specific reason, and an RFC3339 created_at. Ordinary uncertainty or - repairable failure must remain work, not a stop. + Model tiers, strongest to lightest: frontier, strong, standard, economy. They + are judgment defaults for choosing a delegate's capability, not quotas or + completion gates. + + A genuinely terminal blocker is the only human escape hatch. After all + autonomous routes are exhausted, atomically publish control v3 with state + stopped, stop_reason unresolvable_error, a unique control_id, the exact + session/workflow/attempt producer, non-empty attempted_routes, evidence_refs, + a specific reason, and an RFC3339 created_at. Ordinary uncertainty or + repairable failure must remain work, not a stop. diff --git a/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_goal.txt b/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_goal.txt index a415cd6..2f217e6 100644 --- a/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_goal.txt +++ b/src/loopy_loop/templates/pm_planner_dispatcher/loopy_loop_goal.txt @@ -1,5 +1,5 @@ -REPLACE THIS TEXT with the target project, feature, or release goal. +REPLACE THIS TEXT with the product outcome to deliver. -State observable completion criteria and required deliverables here. The -planner/dispatcher workflow is only the mechanism used to pursue this goal; -success means the target outcome itself has been evaluated and achieved. +Describe what should be built and how you will know it is done — the observable +result someone could check for themselves. Write it as a product brief that +states the destination, not how the work gets organized to get there. diff --git a/src/loopy_loop/worker.py b/src/loopy_loop/worker.py index edadb55..1ec3651 100644 --- a/src/loopy_loop/worker.py +++ b/src/loopy_loop/worker.py @@ -23,6 +23,7 @@ from loopy_loop.assignments import write_attempt_assignment from loopy_loop.config import ConfigError from loopy_loop.config import load_workflow_config +from loopy_loop.config import load_workflow_set_preamble from loopy_loop.config import workflow_set_workflows_dir_path from loopy_loop.git_evidence import capture_git_evidence from loopy_loop.git_evidence import GitEvidenceError @@ -55,15 +56,19 @@ from loopy_loop.sessions import GOAL_CHECK_FILENAME from loopy_loop.sessions import harness_outputs_dir_path from loopy_loop.sessions import iteration_harness_output_root +from loopy_loop.sessions import PATHS_FILENAME from loopy_loop.sessions import pending_finished_request_path from loopy_loop.sessions import project_state_dir_path from loopy_loop.sessions import session_dir_path from loopy_loop.sessions import session_goal_path +from loopy_loop.sessions import traces_root_path from loopy_loop.sessions import updates_from_user_path from loopy_loop.sessions import user_updates_journal_path +from loopy_loop.sessions import WORKER_SESSIONS_FILENAME from loopy_loop.sessions import write_json_atomic from loopy_loop.tracing import create_attempt_trace from loopy_loop.tracing import import_harness_artifacts +from loopy_loop.tracing import TRACE_MANIFEST_FILENAME from loopy_loop.tracing import trace_write_json from loopy_loop.tracing import trace_write_text from loopy_loop.tracing import update_trace_manifest @@ -775,7 +780,14 @@ def _render_prompt( assignment: AttemptAssignment | None = None, assignment_file: Path | None = None, ) -> str: - """Render the workflow prompt with authoritative role and path context.""" + """Render the diet iteration header plus the workflow body. + + The header before "Workflow body:" is a coordination contract the workflow + templates are written against (single-goal-assignments.md §3): one goal, the + optional criteria sections, and a short key-paths block. The full machine + path map, frozen rosters, scheduler view, and prior worker-session manifest + move into a sibling paths.json referenced from the header — never inlined. + """ root = repo_root or Path.cwd() session_dir = session_dir_path(repo_root=root, session_id=session_id) @@ -784,87 +796,227 @@ def _render_prompt( if assignment is not None else f"legacy-{iteration}-{workflow_id}" ) + # Preserve the append-only user-input delivery mechanism (its journaling + # side effect and the surfacing of not-yet-acknowledged inputs); this is + # runtime context, distinct from the fixed header scaffolding. semantic_context = _semantic_prompt_context( repo_root=root, session_id=session_id, attempt_id=attempt_id ) - lines = [ - "loopy-loop assignment", - "", - f"Goal: {config_snapshot.goal}", - "Completion criteria:", - *[f"- {item}" for item in config_snapshot.completion_criteria], - "Stop criteria:", - *[f"- {item}" for item in config_snapshot.stop_criteria], - "", - f"Session ID: {session_id}", - f"Workflow set: {workflow_set}", - f"Iteration: {iteration}", - f"Workflow ID: {workflow_id}", - f"Session directory: {session_dir}", - f"Session goal path: {session_goal_path(repo_root=root, session_id=session_id)}", - "Session project_state directory: " - f"{project_state_dir_path(repo_root=root, session_id=session_id)}", - "Session eval_checks directory: " - f"{eval_checks_dir_path(repo_root=root, session_id=session_id)}", - "Session updates_from_user path: " - f"{updates_from_user_path(repo_root=root, session_id=session_id)}", - "Session child_requests directory: " - f"{child_requests_dir_path(repo_root=root, session_id=session_id)}", - f"Session control path: {control_path(repo_root=root, session_id=session_id)}", - "Session finished ledger path: " - f"{finished_path(repo_root=root, session_id=session_id)}", - "Session harness outputs directory: " - f"{harness_outputs_dir_path(repo_root=root, session_id=session_id)}", - f"Iteration directory: {iteration_dir}", - f"Iteration harness output root: {harness_output_root}", + paths_json_path = (iteration_dir / PATHS_FILENAME).resolve() + _write_iteration_paths( + path=paths_json_path, + repo_root=root, + session_id=session_id, + workflow_set=workflow_set, + workflow_id=workflow_id, + iteration=iteration, + iteration_dir=iteration_dir, + session_dir=session_dir, + harness_output_root=harness_output_root, + emits_goal_check=emits_goal_check, + assignment=assignment, + assignment_file=assignment_file, + ) + preamble = load_workflow_set_preamble(repo_root=root, workflow_set=workflow_set) + + header_line = ( + f"loopy-loop assignment — iteration {iteration:04d}, " + f"role: {workflow_id}, session: {session_id}" + ) + key_paths = [ + "You are inside a durable looping session. Key paths:", + f"- session dir: {session_dir.resolve()} (paths below are relative to it)", + "- project_state/ durable working state for your role", + "- child_requests/pending/ publish child requests here", + "- control.json terminal control", + f"- scratch dir (this iteration): {harness_output_root.resolve()}", + f"- paths.json: {paths_json_path} " + "full path map, rosters, scheduler view — read if needed", ] - if assignment is not None and assignment_file is not None: - lines.extend( - [ - "", - "Authoritative attempt contract:", - f"- Assignment envelope: {assignment_file.resolve()}", - f"- Actor kind: {assignment.actor.get('kind', '')}", - f"- Workflow role: {assignment.actor.get('workflow_role', '')}", - f"- Layer kind: {assignment.actor.get('layer_kind', '')}", - f"- Responsibility: {assignment.actor.get('responsibility', '')}", - "- Own-session responsibility: " - f"{assignment.ownership.get('own_session', '')}", - "- Parent-session responsibility: " - f"{assignment.ownership.get('parent_session', '')}", - "- Engine-state responsibility: " - f"{assignment.ownership.get('engine_state', '')}", - "", - "Absolute paths from the assignment envelope:", - *[ - f"- {name}: {path}" - for name, path in sorted(assignment.absolute_paths.items()) - ], - "", - "Treat the assignment envelope as authoritative. Use these absolute " - "paths; do not rediscover session state by searching the checkout.", - ] + criteria: list[str] = [] + if config_snapshot.completion_criteria: + criteria.append("Completion criteria:") + criteria.extend(f"- {item}" for item in config_snapshot.completion_criteria) + if config_snapshot.stop_criteria: + criteria.append("Stop criteria:") + criteria.extend(f"- {item}" for item in config_snapshot.stop_criteria) + + blocks: list[list[str]] = [[header_line], ["Goal:", config_snapshot.goal]] + if criteria: + blocks.append(criteria) + blocks.append(key_paths) + if preamble is not None and preamble.strip(): + blocks.append(["Shared ground rules:", preamble.rstrip()]) + if semantic_context: + blocks.append(["Current layer inputs and receipts:", semantic_context]) + blocks.append(["Workflow body:", workflow_prompt]) + rendered = "\n\n".join("\n".join(block) for block in blocks) + return rendered.rstrip() + "\n" + + +def _write_iteration_paths( + *, + path: Path, + repo_root: Path, + session_id: str, + workflow_set: str, + workflow_id: str, + iteration: int, + iteration_dir: Path, + session_dir: Path, + harness_output_root: Path, + emits_goal_check: bool, + assignment: AttemptAssignment | None, + assignment_file: Path | None, +) -> None: + """Write the full machine path map the diet header references by name. + + Holds every absolute path the old header inlined plus the complete v3 + assignment path map (rosters, scheduler view, workflow contract as files), + and previous_worker_sessions: the prior iteration's team-harness + worker_sessions.json for selective session reuse (context-and-eval-economy + A4), or null when none exists. + """ + + goal_check_output = ( + str((iteration_dir / GOAL_CHECK_FILENAME).resolve()) + if (workflow_id == "goal_check" or emits_goal_check) + else None + ) + parent_session_dir = ( + str(session_dir.parent.parent.resolve()) + if session_dir.parent.name == "children" + else None + ) + root_session_id = ( + str(assignment.identity.get("root_session_id")) + if assignment is not None and assignment.identity.get("root_session_id") + else None + ) + previous_worker_sessions = ( + _previous_worker_sessions_path( + repo_root=repo_root, + root_session_id=root_session_id, + session_id=session_id, + iteration=iteration, ) - if assignment.context: - lines.extend( - [ - "", - "Frozen workflow, scheduler, and capability context:", - json.dumps( - assignment.context, indent=2, sort_keys=True, ensure_ascii=False - ), - ] + if root_session_id is not None + else None + ) + payload: dict[str, object] = { + "schema_version": 1, + "session_id": session_id, + "workflow_set": workflow_set, + "workflow_id": workflow_id, + "iteration": iteration, + "session_dir": str(session_dir.resolve()), + "iteration_dir": str(iteration_dir.resolve()), + "scratch_dir": str(harness_output_root.resolve()), + "assignment_envelope": ( + str(assignment_file.resolve()) if assignment_file is not None else None + ), + "goal_check_output": goal_check_output, + "parent_session_dir": parent_session_dir, + "previous_worker_sessions": ( + str(previous_worker_sessions) + if previous_worker_sessions is not None + else None + ), + "session_paths": { + "goal": str( + session_goal_path(repo_root=repo_root, session_id=session_id).resolve() + ), + "project_state": str( + project_state_dir_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + "eval_checks": str( + eval_checks_dir_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + "updates_from_user": str( + updates_from_user_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + "user_inputs_journal": str( + user_updates_journal_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + "child_requests": str( + child_requests_dir_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + "control": str( + control_path(repo_root=repo_root, session_id=session_id).resolve() + ), + "finished_ledger": str( + finished_path(repo_root=repo_root, session_id=session_id).resolve() + ), + "harness_outputs": str( + harness_outputs_dir_path( + repo_root=repo_root, session_id=session_id + ).resolve() + ), + }, + "envelope_paths": ( + dict(assignment.absolute_paths) if assignment is not None else None + ), + } + write_json_atomic(path=path, payload=payload) + + +def _previous_worker_sessions_path( + *, repo_root: Path, root_session_id: str, session_id: str, iteration: int +) -> Path | None: + """Return the newest prior iteration's team-harness worker_sessions.json. + + Scans this session's attempt traces, reads each trace manifest's iteration, + and returns the worker_sessions.json belonging to the highest iteration + strictly below the current one. None when no earlier attempt produced it. + """ + + attempts_root = ( + traces_root_path(repo_root=repo_root) + / root_session_id + / "sessions" + / session_id + / "attempts" + ) + if not attempts_root.is_dir(): + return None + best_iteration = -1 + best_path: Path | None = None + for attempt_dir in sorted(attempts_root.iterdir()): + if not attempt_dir.is_dir(): + continue + try: + manifest = json.loads( + (attempt_dir / TRACE_MANIFEST_FILENAME).read_text(encoding="utf-8") ) - if session_dir.parent.name == "children": - lines.append(f"Parent session directory: {session_dir.parent.parent}") - if workflow_id == "goal_check" or emits_goal_check: - lines.append( - f"goal_check.json output path: {iteration_dir / GOAL_CHECK_FILENAME}" + except (OSError, ValueError): + continue + identity = manifest.get("identity") if isinstance(manifest, dict) else None + attempt_iteration = ( + identity.get("iteration") if isinstance(identity, dict) else None ) - if semantic_context: - lines.extend(["", "Current layer inputs and receipts:", semantic_context]) - lines.extend(["", "Workflow body:", workflow_prompt]) - return "\n".join(lines).rstrip() + "\n" + if not isinstance(attempt_iteration, int) or attempt_iteration >= iteration: + continue + if attempt_iteration <= best_iteration: + continue + sessions_file = next( + iter(sorted((attempt_dir / "harness").rglob(WORKER_SESSIONS_FILENAME))), + None, + ) + if sessions_file is None: + continue + best_iteration = attempt_iteration + best_path = sessions_file.resolve() + return best_path def _read_session_metadata(*, repo_root: Path, session_id: str) -> dict[str, object]: diff --git a/src/tests/test_cli.py b/src/tests/test_cli.py index 49afedf..949ce1e 100644 --- a/src/tests/test_cli.py +++ b/src/tests/test_cli.py @@ -129,10 +129,13 @@ def test_init_inner_outer_eval_template_scaffolds_expected_files( assert repo_root.joinpath( ".loopy_loop/workflow_sets/inner_outer_eval/contract.yaml" ).exists() + assert repo_root.joinpath( + ".loopy_loop/workflow_sets/inner_outer_eval/preamble.txt" + ).exists() assert not repo_root.joinpath( ".loopy_loop/workflow_sets/main/workflows/goal_check/prompt.txt" ).exists() - assert "harness coordinator for the `outer` workflow role" in repo_root.joinpath( + assert "You are the outer coordinator" in repo_root.joinpath( ".loopy_loop/workflow_sets/inner_outer_eval/workflows/outer/prompt.txt" ).read_text(encoding="utf-8") assert 'gemini: "gemini-3.5-flash"' in repo_root.joinpath( @@ -214,10 +217,17 @@ def test_init_pm_planner_dispatcher_template_scaffolds_expected_files( assert repo_root.joinpath( ".loopy_loop/workflow_sets/pm_planner_dispatcher/contract.yaml" ).exists() + assert repo_root.joinpath( + ".loopy_loop/workflow_sets/pm_planner_dispatcher/preamble.txt" + ).exists() + # The child inner_outer_eval set ships its preamble too (extra sources). + assert repo_root.joinpath( + ".loopy_loop/workflow_sets/inner_outer_eval/preamble.txt" + ).exists() assert "workflow_set: pm_planner_dispatcher" in repo_root.joinpath( "loopy_loop_config.yaml" ).read_text(encoding="utf-8") - assert '"schema_version": 2' in repo_root.joinpath( + assert '"schema_version": 3' in repo_root.joinpath( ".loopy_loop/workflow_sets/pm_planner_dispatcher/workflows/dispatcher/prompt.txt" ).read_text(encoding="utf-8") diff --git a/src/tests/test_config.py b/src/tests/test_config.py index ac4b2ec..a4114c8 100644 --- a/src/tests/test_config.py +++ b/src/tests/test_config.py @@ -164,6 +164,73 @@ def test_load_root_config_accepts_team_harness_retry_controls( assert root_config.team_harness_retry_max_delay_s == 60.0 +def test_load_root_config_accepts_context_economy_knobs(repo_builder: Any) -> None: + repo_root = repo_builder( + root_config={ + "team_harness_compact_above_tokens": 80000, + "team_harness_prompt_cache": "ephemeral", + } + ) + + root_config = load_root_config(repo_root=repo_root) + + assert root_config.team_harness_compact_above_tokens == 80000 + assert root_config.team_harness_prompt_cache == "ephemeral" + + +def test_context_economy_knobs_default_to_none(repo_builder: Any) -> None: + root_config = load_root_config(repo_root=repo_builder()) + + assert root_config.team_harness_compact_above_tokens is None + assert root_config.team_harness_prompt_cache is None + + +def test_context_economy_knobs_are_in_wire_snapshot(repo_builder: Any) -> None: + from loopy_loop.coordinator_app import _COORDINATOR_ONLY_FIELDS + from loopy_loop.models import RootConfigSnapshot + + repo_root = repo_builder( + root_config={ + "team_harness_compact_above_tokens": 80000, + "team_harness_prompt_cache": "ephemeral", + } + ) + root_config = load_root_config(repo_root=repo_root) + + snapshot = RootConfigSnapshot.model_validate( + root_config.model_dump(exclude=_COORDINATOR_ONLY_FIELDS) + ) + + assert snapshot.team_harness_compact_above_tokens == 80000 + assert snapshot.team_harness_prompt_cache == "ephemeral" + + +def test_run_when_requested_parses_and_defaults_off(repo_builder: Any) -> None: + repo_root = repo_builder( + workflows={ + "outer": {"prompt": "Plan", "config": {"enabled": True, "description": ""}}, + "eval_runner": { + "prompt": "Run evals", + "config": { + "enabled": True, + "run_when_requested": True, + "description": "", + }, + }, + } + ) + + workflows = { + workflow.id: workflow + for workflow in load_workflow_definitions( + repo_root=repo_root, workflow_set="main" + ) + } + + assert workflows["eval_runner"].run_when_requested is True + assert workflows["outer"].run_when_requested is False + + def test_load_root_config_rejects_invalid_retry_delay_bounds(repo_builder: Any) -> None: repo_root = repo_builder( root_config={ diff --git a/src/tests/test_examples.py b/src/tests/test_examples.py index 047d0be..78b03bc 100644 --- a/src/tests/test_examples.py +++ b/src/tests/test_examples.py @@ -24,7 +24,9 @@ def test_inner_outer_eval_template_preflight() -> None: assert preflight.root_config.team_harness_provider == "codex" assert preflight.workflow_contract.completion_role == "outer" assert preflight.workflow_contract.check_author_roles == ["eval_reviewer", "outer"] - assert preflight.workflow_contract.check_runner_roles == ["eval_runner", "outer"] + # eval_runner records advisory results in eval_results.md (receipts retired), + # so it is no longer a contract check-runner role the engine seals receipts for. + assert preflight.workflow_contract.check_runner_roles == ["outer"] orchestration = preflight.workflow_contract.orchestration assert orchestration is not None assert orchestration.task_acceptance_owner == "outer" diff --git a/src/tests/test_harness_runner.py b/src/tests/test_harness_runner.py index 7244c28..8d25403 100644 --- a/src/tests/test_harness_runner.py +++ b/src/tests/test_harness_runner.py @@ -169,6 +169,78 @@ async def run(self, task: str) -> TeamHarnessResult: ) +def test_harness_runner_passes_context_economy_knobs( + repo_root: Any, snapshot_factory: Any, monkeypatch: Any +) -> None: + monkeypatch.setenv("OPENROUTER_API_KEY", "secret") + captured: dict[str, Any] = {} + + class FakeHarness: + def __init__( + self, *, compact_above_tokens: int, prompt_cache: str, **kwargs: Any + ) -> None: + captured["compact_above_tokens"] = compact_above_tokens + captured["prompt_cache"] = prompt_cache + + async def run(self, task: str) -> TeamHarnessResult: + return TeamHarnessResult(text="done", agents=[], run_id="run-123") + + run_harness_iteration( + repo_root=repo_root, + config_snapshot=snapshot_factory( + team_harness_compact_above_tokens=80000, + team_harness_prompt_cache="ephemeral", + ), + rendered_prompt="rendered prompt", + harness_factory=FakeHarness, + ) + + assert captured["compact_above_tokens"] == 80000 + assert captured["prompt_cache"] == "ephemeral" + + +def test_harness_runner_skips_context_economy_knobs_for_old_harness( + repo_root: Any, snapshot_factory: Any, monkeypatch: Any +) -> None: + monkeypatch.setenv("OPENROUTER_API_KEY", "secret") + captured: dict[str, Any] = {} + + class OldHarness: + def __init__( + self, + *, + provider: str, + model: str, + api_base: str, + api_key: str, + agents: list[str], + agent_models: dict[str, str], + agent_reasoning_efforts: dict[str, str], + system_prompt: str, + cwd: str, + console_mode: str, + ) -> None: + captured["ran"] = True + + async def run(self, task: str) -> TeamHarnessResult: + return TeamHarnessResult(text="done", agents=[], run_id="run-123") + + # Unlike retry/agent overrides, unsupported context-economy knobs are + # tolerated silently (they are optimizations, not correctness). + result = run_harness_iteration( + repo_root=repo_root, + config_snapshot=snapshot_factory( + team_harness_compact_above_tokens=80000, + team_harness_prompt_cache="ephemeral", + ), + rendered_prompt="rendered prompt", + harness_factory=OldHarness, + ) + + assert captured.get("ran") is True + assert result.success is True + + def test_harness_runner_passes_none_api_key_for_codex_provider( repo_root: Any, snapshot_factory: Any, monkeypatch: Any ) -> None: diff --git a/src/tests/test_recursive_loop_contract_v2.py b/src/tests/test_recursive_loop_contract_v2.py index 6f28283..5209bde 100644 --- a/src/tests/test_recursive_loop_contract_v2.py +++ b/src/tests/test_recursive_loop_contract_v2.py @@ -162,6 +162,41 @@ def _write_v2_child_request( return path +def _write_v3_child_request( + *, + repo_root: Path, + parent_task: dict[str, Any], + request_id: str, + workflow_set: str, + goal: str, +) -> Path: + """Write a single-goal v3 child request (no assignment arrays, no inputs).""" + + path = ( + child_requests_pending_dir_path( + repo_root=repo_root, session_id=str(parent_task["session_id"]) + ) + / f"{request_id}.json" + ) + path.write_text( + json.dumps( + { + "schema_version": 3, + "request_id": request_id, + "workflow_set": workflow_set, + "goal": goal, + "origin": { + "parent_attempt_id": parent_task["attempt_id"], + "parent_work_item_id": f"work-{request_id}", + }, + }, + indent=2, + ), + encoding="utf-8", + ) + return path + + def _finish( client: TestClient, task: dict[str, Any], *, success: bool = True ) -> dict[str, Any]: @@ -582,7 +617,13 @@ def fake_run_harness_iteration(**kwargs: Any): / "prompt.txt" ) assert prompt_file.read_text(encoding="utf-8") == kwargs["rendered_prompt"] - assert str(assignment_file) in kwargs["rendered_prompt"] + # The diet header references the frozen envelope through paths.json + # rather than inlining it (single-goal-assignments.md §3). + paths = json.loads( + (prompt_file.parent / "paths.json").read_text(encoding="utf-8") + ) + assert paths["assignment_envelope"] == str(assignment_file) + assert Path(paths["envelope_paths"]["session_root"]).is_absolute() caller = kwargs["caller_context"] assert caller["parent_assignment_path"] == str(assignment_file) assert Path(str(caller["trace_root"])).is_absolute() @@ -3220,3 +3261,84 @@ def test_eval_receipt_cannot_substitute_a_noncanonical_raw_report_path( ) assert any("canonical attempt eval/report.json" in reason for reason in reasons) + + +def test_v3_single_goal_child_request_dispatches_with_verbatim_goal( + repo_builder: Any, monkeypatch: pytest.MonkeyPatch +) -> None: + """A v3 child request carries one free-text goal; the child's goal contract + is that text verbatim with the v2 typed arrays and inputs empty.""" + + monkeypatch.setenv("OPENROUTER_API_KEY", "secret") + repo_root = repo_builder() + _write_workflow_set( + repo_root=repo_root, workflow_set="child_set", workflow_id="child_work" + ) + client = TestClient(create_coordinator_app(repo_root=repo_root, resume=False)) + + root_task = client.post("/register", json=_register_v2(repo_root)).json() + assert root_task["action"] == "run" + + child_goal = ( + "Build the isolated component. Start from plan/phases/phase-0.md; treat " + "it as the scope contract. Record evidence in project_state/ledger.md." + ) + _write_v3_child_request( + repo_root=repo_root, + parent_task=root_task, + request_id="v3-single-goal", + workflow_set="child_set", + goal=child_goal, + ) + child_task = _finish(client, root_task) + + assert child_task["action"] == "run" + assert child_task["workflow_set"] == "child_set" + child_id = child_task["session_id"] + + child_goal_file = ( + session_dir_path(repo_root=repo_root, session_id=child_id) / "goal.md" + ) + assert child_goal_file.read_text(encoding="utf-8").rstrip("\n") == child_goal + + child_contract = json.loads( + goal_contract_path(repo_root=repo_root, session_id=child_id).read_text( + encoding="utf-8" + ) + ) + assert child_contract["goal"] == child_goal + assert child_contract["completion_criteria"] == [] + assert child_contract["stop_criteria"] == [] + assert child_contract["constraints"] == [] + assert child_contract["deliverables"] == [] + assert child_contract["required_evidence"] == [] + assert child_contract["inputs"] == [] + + child_manifest = json.loads( + session_dir_path(repo_root=repo_root, session_id=child_id) + .joinpath("session.json") + .read_text(encoding="utf-8") + ) + assert child_manifest["origin"]["request_id"] == "v3-single-goal" + assert child_manifest["parent_session_id"] == root_task["session_id"] + + +def test_v3_child_request_rejects_embedded_assignment_arrays() -> None: + """v3 requests must not smuggle the v2 typed assignment; validation rejects + it so the single-goal contract stays crisp.""" + + from pydantic import ValidationError + + from loopy_loop.models import ChildSessionRequest + + with pytest.raises(ValidationError, match="must not carry an assignment"): + ChildSessionRequest.model_validate( + { + "schema_version": 3, + "request_id": "bad-v3", + "workflow_set": "child_set", + "goal": "Do the thing", + "origin": {"parent_attempt_id": "attempt-1"}, + "assignment": {"goal": "Do the thing"}, + } + ) diff --git a/src/tests/test_scheduler.py b/src/tests/test_scheduler.py index a7ccbbc..1921459 100644 --- a/src/tests/test_scheduler.py +++ b/src/tests/test_scheduler.py @@ -468,3 +468,170 @@ def test_eval_runner_waits_for_eval_reviewer_predecessor( assert before_reviewer.id == "eval_reviewer" assert after_reviewer is not None assert after_reviewer.id == "eval_runner" + + +def _requested_eval_repo(repo_builder: Any): + return repo_builder( + workflows={ + "outer": { + "prompt": "Plan", + "config": { + "enabled": True, + "priority": 10, + "run_every": 1, + "must_follow": None, + "not_before_iteration": 0, + "run_on_start": True, + "description": "", + }, + }, + "eval_runner": { + "prompt": "Run evals", + "config": { + "enabled": True, + "priority": 100, + "run_every": 1, + "must_follow": None, + "not_before_iteration": 0, + "run_when_requested": True, + "emits_goal_check": True, + "description": "", + }, + }, + } + ) + + +def test_run_when_requested_gates_on_pending_request( + repo_builder: Any, history_entry_factory: Any +) -> None: + repo_root = _requested_eval_repo(repo_builder) + workflows = load_workflow_definitions(repo_root=repo_root, workflow_set="main") + history = [history_entry_factory(iteration=1, workflow_id="outer", success=True)] + + without_request = choose_next_workflow( + workflows=workflows, history=history, iteration_count=2, eval_requested=False + ) + with_request = choose_next_workflow( + workflows=workflows, history=history, iteration_count=2, eval_requested=True + ) + + # Higher-priority eval_runner is invisible until a request is pending, then + # it wins the tie-break — the gate composes with priority. + assert without_request is not None + assert without_request.id == "outer" + assert with_request is not None + assert with_request.id == "eval_runner" + + +def test_run_when_requested_default_is_off(repo_builder: Any) -> None: + repo_root = _requested_eval_repo(repo_builder) + workflows = load_workflow_definitions(repo_root=repo_root, workflow_set="main") + + # Default eval_requested=False keeps requested-only workflows out. + chosen = choose_next_workflow(workflows=workflows, history=[], iteration_count=0) + + assert chosen is not None + assert chosen.id == "outer" + + +def test_run_when_requested_run_on_start_authors_initial_checkset( + repo_builder: Any, history_entry_factory: Any +) -> None: + repo_root = repo_builder( + workflows={ + "outer": { + "prompt": "Plan", + "config": { + "enabled": True, + "priority": 10, + "run_every": 1, + "must_follow": None, + "not_before_iteration": 0, + "run_on_start": True, + "description": "", + }, + }, + "eval_reviewer": { + "prompt": "Author checks", + "config": { + "enabled": True, + "priority": 100, + "run_every": 1, + "must_follow": None, + "not_before_iteration": 0, + "run_on_start": True, + "run_when_requested": True, + "description": "", + }, + }, + } + ) + workflows = load_workflow_definitions(repo_root=repo_root, workflow_set="main") + + # At start (no successful history), run_on_start bypasses the requested gate + # so the initial check-set is authored even without a pending request (C3). + at_start = choose_next_workflow( + workflows=workflows, history=[], iteration_count=0, eval_requested=False + ) + + # After a success, the run_on_start window has closed; eval_reviewer is now + # eligible only on request. + history = [history_entry_factory(iteration=1, workflow_id="outer", success=True)] + after_success_no_request = choose_next_workflow( + workflows=workflows, history=history, iteration_count=2, eval_requested=False + ) + after_success_with_request = choose_next_workflow( + workflows=workflows, history=history, iteration_count=2, eval_requested=True + ) + + assert at_start is not None + assert at_start.id == "eval_reviewer" + assert after_success_no_request is not None + assert after_success_no_request.id == "outer" + assert after_success_with_request is not None + assert after_success_with_request.id == "eval_reviewer" + + +def test_run_when_requested_still_composes_with_must_follow( + repo_builder: Any, history_entry_factory: Any +) -> None: + repo_root = repo_builder( + workflows={ + "outer": { + "prompt": "Plan", + "config": { + "enabled": True, + "priority": 10, + "run_every": 1, + "must_follow": None, + "not_before_iteration": 0, + "run_on_start": True, + "description": "", + }, + }, + "eval_runner": { + "prompt": "Run evals", + "config": { + "enabled": True, + "priority": 100, + "run_every": 1, + "must_follow": "outer", + "not_before_iteration": 0, + "run_when_requested": True, + "emits_goal_check": True, + "description": "", + }, + }, + } + ) + workflows = load_workflow_definitions(repo_root=repo_root, workflow_set="main") + + # A pending request cannot bypass must_follow: eval_runner requires a most + # recent successful outer, absent here. + gated_by_predecessor = choose_next_workflow( + workflows=workflows, history=[], iteration_count=0, eval_requested=True + ) + + assert gated_by_predecessor is not None + assert gated_by_predecessor.id == "outer" diff --git a/src/tests/test_semantic_state_paths_v3.py b/src/tests/test_semantic_state_paths_v3.py index 3076339..1cec8d8 100644 --- a/src/tests/test_semantic_state_paths_v3.py +++ b/src/tests/test_semantic_state_paths_v3.py @@ -351,3 +351,47 @@ def test_v3_assignment_names_layer_and_parent_paths(repo_root: Path) -> None: ).resolve() ) assert not (child_root / "harness_capability_roster.json").exists() + + +def test_v3_roster_retires_eval_receipts_but_keeps_eval_state(tmp_path: Path) -> None: + """A v3 check-runner role no longer advertises the retired eval_receipts/ + output; it still owns project_state/eval_state.md. The receipt-sealing + machinery stays available for frozen older sessions (contract-gated).""" + + from loopy_loop.config import PreflightResult + from loopy_loop.config import RootConfig + from loopy_loop.config import WorkflowDefinition + from loopy_loop.coordinator_app import _build_workflow_roster + from loopy_loop.models import WorkflowSetContract + + contract_payload = _v3_contract() + contract = WorkflowSetContract.model_validate(contract_payload) + workflows = [ + WorkflowDefinition( + workflow_set="synthetic", + id=role_id, + directory=tmp_path / role_id, + prompt_path=tmp_path / role_id / "prompt.txt", + config_path=tmp_path / role_id / "config.yaml", + ) + for role_id in ("outer", "inner") + ] + preflight = PreflightResult( + root_config=RootConfig( + goal="Build a thing", workflow_set="synthetic", max_turns=10 + ), + workflow_set="synthetic", + workflows=workflows, + workflow_contract=contract, + workflow_contract_text=json.dumps(contract_payload, sort_keys=True), + workflow_contract_sha256="sha256:" + "d" * 64, + ) + + roster = _build_workflow_roster( + session_id="goal_20260718_synthetic", preflight=preflight, created_at=utc_now() + ) + outer = next(role for role in roster.roles if role.workflow_id == "outer") + + assert "eval_check_runner" in outer.authorities + assert "project_state/eval_state.md" in outer.expected_outputs + assert "eval_receipts/" not in outer.expected_outputs diff --git a/src/tests/test_template_contracts.py b/src/tests/test_template_contracts.py index ada46d4..c84abf0 100644 --- a/src/tests/test_template_contracts.py +++ b/src/tests/test_template_contracts.py @@ -8,10 +8,12 @@ from eval_banana.models import HarnessJudgeCheckDefinition from loopy_loop.config import load_workflow_definitions +from loopy_loop.config import load_workflow_set_preamble from loopy_loop.config import run_preflight from loopy_loop.scheduler import choose_next_workflow TEMPLATES_ROOT = Path(__file__).resolve().parents[1] / "loopy_loop" / "templates" +TEMPLATE_SETS = ("inner_outer_eval", "pm_planner_dispatcher") RUNTIME_IGNORE_RULES = { ".loopy_loop/sessions/", ".loopy_loop/traces/", @@ -27,6 +29,14 @@ def _template_root(name: str) -> Path: return TEMPLATES_ROOT / name +def _prompt_paths(template: str) -> list[Path]: + return sorted( + _template_root(template).glob( + ".loopy_loop/workflow_sets/*/workflows/*/prompt.txt" + ) + ) + + def _workflow_prompt(*, template: str, workflow_set: str, workflow: str) -> str: return ( _template_root(template) @@ -61,10 +71,10 @@ def test_packaged_workflow_contracts_name_every_role_and_owner() -> None: "eval_reviewer", "outer", ] - assert delivery.workflow_contract.evaluation.check_runner_roles == [ - "eval_runner", - "outer", - ] + # Under B (receipts retired), eval_runner records advisory results in + # project_state/eval_results.md rather than sealed eval_receipts/, so it is + # no longer a contract check-runner role that the engine seals receipts for. + assert delivery.workflow_contract.evaluation.check_runner_roles == ["outer"] delivery_plan = next( item for item in delivery.workflow_contract.state @@ -98,13 +108,12 @@ def test_packaged_workflow_contracts_name_every_role_and_owner() -> None: assert parent.workflow_contract.evaluation.advisory is True assert parent.workflow_contract.evaluation.check_author_roles == ["planner"] assert parent.workflow_contract.evaluation.check_runner_roles == ["planner"] - dispatch_inputs = next( - item + # The retired dispatch-snapshot state path is gone: a v3 child request is one + # authored goal, not an immutable snapshot the dispatcher must hash and pin. + assert all( + item["path"] != "project_state/dispatch_inputs/" for item in parent.workflow_contract.state - if item["path"] == "project_state/dispatch_inputs/" ) - assert dispatch_inputs["owner_role"] == "dispatcher" - assert dispatch_inputs["contributor_roles"] == [] def test_explicit_contract_without_protocol_version_defaults_to_v2( @@ -145,69 +154,81 @@ def test_explicit_contract_without_protocol_version_defaults_to_v2( assert derived.workflow_contract.session_protocol_version == 1 -def test_packaged_prompt_contract_uses_absolute_assignment_paths() -> None: +def test_role_prompts_fit_one_screen() -> None: + """Every stock role prompt stays within the P8 one-screen budget.""" + + for template in TEMPLATE_SETS: + paths = _prompt_paths(template) + assert paths + for path in paths: + line_count = len(path.read_text(encoding="utf-8").splitlines()) + assert line_count <= 80, f"{path} has {line_count} lines" + + +def test_prompts_drop_retired_ceremony_and_model_mandates() -> None: + """Prompts and the shared preamble carry no retired ceremony or vendor names.""" + banned = ( "questions.md", "waiting-for-human", "waiting_for_human", "target_paths", "/_feature_planning", - "/eval_results", "create an agent team", + "assignment envelope", + "absolute_paths", + "dispatch_inputs", + "required_evidence", + "think ultra", + "ultra deeply", + "using codex", + "with gemini", + "gpt-", + "claude-opus", + "claude-sonnet", + "claude-haiku", + "gemini-", + '"schema_version": 2', ) - for template in ("inner_outer_eval", "pm_planner_dispatcher"): - prompt_paths = sorted( - _template_root(template).glob( - ".loopy_loop/workflow_sets/*/workflows/*/prompt.txt" - ) + for template in TEMPLATE_SETS: + texts = [path.read_text(encoding="utf-8") for path in _prompt_paths(template)] + preamble = load_workflow_set_preamble( + repo_root=_template_root(template), workflow_set=template ) - assert prompt_paths - for path in prompt_paths: - prompt = path.read_text(encoding="utf-8") - lowered = prompt.lower() - assert "assignment envelope" in lowered, path - assert "absolute" in lowered, path - for text in banned: - assert text not in lowered, f"{text!r} in {path}" + assert preamble, f"{template} ships a shared preamble" + texts.append(preamble) + for text in texts: + lowered = text.lower() + for needle in banned: + assert needle.lower() not in lowered, f"{needle!r} in {template}" -def test_inner_prompt_keeps_delegation_dynamic() -> None: - prompt = _workflow_prompt( - template="inner_outer_eval", workflow_set="inner_outer_eval", workflow="inner" - ).lower() - prompt_words = " ".join(prompt.split()) - - assert "dynamic delegation" in prompt - assert "decide at runtime whether delegation helps" in prompt_words - assert "it is valid to do the work directly" in prompt_words - assert "delegated_role" in prompt - assert "delegated_task_id" in prompt - assert "expected_outputs" in prompt - assert "state_responsibility" in prompt - - -def test_inner_executes_outer_selection_without_bootstrapping_plan() -> None: - """Inner reports a missing selection instead of taking outer's authority.""" +def test_shared_rules_live_once_in_the_preamble() -> None: + """Cross-role rules are single-sourced in the preamble, not per prompt.""" - prompt = _workflow_prompt( - template="inner_outer_eval", workflow_set="inner_outer_eval", workflow="inner" - ).lower() - prompt_words = " ".join(prompt.split()) - - assert "execute exactly the one active leaf selected by outer" in prompt_words - assert "do not choose a different leaf" in prompt_words - assert "bootstrap or rewrite the layer plan" in prompt_words - assert "report that precise gap" in prompt_words + for template in TEMPLATE_SETS: + preamble = load_workflow_set_preamble( + repo_root=_template_root(template), workflow_set=template + ) + assert preamble is not None + lowered = preamble.lower() + assert "capability roster in paths.json" in lowered + assert "previous_worker_sessions" in lowered + assert "result card" in lowered + assert "renaming it into place atomically" in lowered + assert "failing checks" in lowered + for path in _prompt_paths(template): + prompt = path.read_text(encoding="utf-8").lower() + assert "previous_worker_sessions" not in prompt, path + assert "result card" not in prompt, path def test_only_layer_orchestrators_publish_successful_terminal_control() -> None: """Successful control belongs to outer/planner, never an eval role.""" writers: set[tuple[str, str]] = set() - for template in ("inner_outer_eval", "pm_planner_dispatcher"): - for path in _template_root(template).glob( - ".loopy_loop/workflow_sets/*/workflows/*/prompt.txt" - ): + for template in TEMPLATE_SETS: + for path in _prompt_paths(template): if '"stop_reason": "goal_met"' in path.read_text(encoding="utf-8"): writers.add((template, path.parent.name)) @@ -217,43 +238,64 @@ def test_only_layer_orchestrators_publish_successful_terminal_control() -> None: } -def test_dispatcher_teaches_v2_pending_child_contract() -> None: - """Dispatcher transports a milestone in the unchanged v2 child request.""" +def test_dispatcher_teaches_v3_child_request() -> None: + """Dispatcher transports the milestone as one authored v3 goal text.""" prompt = _workflow_prompt( template="pm_planner_dispatcher", workflow_set="pm_planner_dispatcher", workflow="dispatcher", ) - prompt_words = " ".join(prompt.split()) + words = " ".join(prompt.split()) - assert "canonical `pending` directory" in prompt_words - assert '"schema_version": 2' in prompt + assert "child_requests/pending/" in prompt + assert '"schema_version": 3' in prompt assert '"request_id"' in prompt + assert '"workflow_set"' in prompt + assert '"goal"' in prompt assert '"origin"' in prompt - assert '"assignment"' in prompt - assert '"completion_criteria"' in prompt - assert '"required_evidence"' in prompt - assert "do not narrow it to an exact leaf" in prompt_words.lower() - assert "child outer role owns its own plan and leaf decomposition" in prompt_words - assert "project_state/dispatch_inputs/.json" in prompt - assert ( - '"ref": "parent:/project_state/dispatch_inputs/' - 'stable-unique-request-id.json"' in prompt - ) - assert ( - "never declare a mutable plan or task record as a child input" - in prompt_words.lower() + assert '"supersedes_request_id"' in prompt + # v3 is one authored brief, not the retired field arrays / hashed snapshots. + assert "completion_criteria" not in prompt + assert "required_evidence" not in prompt + assert "dispatch_inputs" not in prompt + assert "sha256" not in prompt.lower() + assert "self-contained brief" in words + assert "the child owns its own plan" in words + assert "request_id, not the filename, is the idempotency key" in words + + +def test_inner_executes_outer_selection_without_bootstrapping_plan() -> None: + """Inner reports a missing selection instead of taking outer's authority.""" + + prompt = _workflow_prompt( + template="inner_outer_eval", workflow_set="inner_outer_eval", workflow="inner" ) - assert ( - "immutable snapshot, snapshot hash, request rename, then factual ledger link" - in prompt_words + words = " ".join(prompt.split()) + + assert "Execute exactly the one active task outer selected" in words + assert "Do not choose a different task" in words + assert "bootstrap or rewrite the layer plan" in words + assert "report that precise gap" in words + + +def test_inner_keeps_delegation_a_judgment_call() -> None: + """Delegation stays inner's judgment; mechanics live in the preamble only.""" + + prompt = _workflow_prompt( + template="inner_outer_eval", workflow_set="inner_outer_eval", workflow="inner" ) - assert "child eval receipt" not in prompt + words = " ".join(prompt.split()) + + assert "is your judgment" in words + assert "it is equally valid to do the work directly" in words + # The spawn-metadata recipe is retired; delegation is single-sourced. + assert "delegated_role" not in prompt + assert "delegated_task_id" not in prompt def test_eval_reviewer_examples_match_eval_banana_schema(tmp_path: Path) -> None: - """The one-layer optional reviewer teaches the real judge-check schema.""" + """The optional reviewer still teaches the real judge-check schema.""" prompt = _workflow_prompt( template="inner_outer_eval", @@ -265,7 +307,7 @@ def test_eval_reviewer_examples_match_eval_banana_schema(tmp_path: Path) -> None "--check-dir " "--harness-agent " ) in prompt - assert "Omit a\nper-check `model`" in prompt + assert "Omit a per-check\n`model`" in prompt match = re.search(r"```yaml\n(.*?)\n```", prompt, flags=re.DOTALL) assert match is not None check_path = tmp_path / "inner_outer_eval.yaml" @@ -278,77 +320,117 @@ def test_eval_reviewer_examples_match_eval_banana_schema(tmp_path: Path) -> None assert check.model is None -def test_eval_runner_selects_judge_from_roster_and_publishes_advisory_receipt() -> None: - """The optional runner records effective selection and never controls stop.""" +def test_eval_reviewer_lifts_the_deterministic_check_ban() -> None: + """C2: objective facts go to deterministic checks; judge checks for meaning.""" + + prompt = _workflow_prompt( + template="inner_outer_eval", + workflow_set="inner_outer_eval", + workflow="eval_reviewer", + ) + words = " ".join(prompt.split()) + + assert "Judge checks evaluate semantics and quality" in words + assert "belong in deterministic checks" in words + assert "the repo's own test and lint suites" in words + assert "Keep judge checks for what actually needs judgment" in words + # The old blanket ban is gone. + assert "do not invent deterministic checks" not in words.lower() + + +def test_eval_runner_is_advisory_and_reads_report_md() -> None: + """The optional runner records advisory evidence and never controls stop.""" prompt = _workflow_prompt( template="inner_outer_eval", workflow_set="inner_outer_eval", workflow="eval_runner", ) - prompt_words = " ".join(prompt.split()) + words = " ".join(prompt.split()) - assert "harness_capability_roster" in prompt + assert "eval-banana validate" in prompt + assert "eval-banana run" in prompt assert "--harness-agent " in prompt assert "--harness-model " in prompt assert "--harness-reasoning-effort " in prompt - assert "loopy capture-git-receipt" in prompt - assert "trace::/eval/report.json" in prompt - assert "copy those canonical definition digests" in prompt_words.lower() - assert "do not manually hash" in prompt_words.lower() - assert "do not write `goal_check.json` or" in prompt_words.lower() + assert "Read report.md" in prompt + assert "not report.json" in prompt + assert "project_state/eval_results.md" in prompt + assert "project_state/eval_request.md" in prompt + # Advisory only: no completion authority, no retired receipt ceremony. assert '"stop_reason": "goal_met"' not in prompt + assert "goal_check.json" not in prompt + assert "you never publish control" in words -def test_packaged_cadence_runs_eval_after_three_role_successes( +def test_packaged_schedule_authors_checks_on_start_then_evals_on_request( history_entry_factory: Any, ) -> None: - expected = { - "inner_outer_eval": [ - "eval_reviewer", - "outer", - "inner", - "outer", - "inner", - "outer", - "inner", - "eval_reviewer", - "eval_runner", - "outer", - ], - "pm_planner_dispatcher": [ - "planner", - "dispatcher", - "planner", - "dispatcher", - "planner", - "dispatcher", - "planner", - "dispatcher", - "planner", - ], - } - for template, sequence in expected.items(): - workflows = load_workflow_definitions( - repo_root=_template_root(template), workflow_set=template + """eval_reviewer authors on start; thereafter eval fires only when asked.""" + + workflows = load_workflow_definitions( + repo_root=_template_root("inner_outer_eval"), workflow_set="inner_outer_eval" + ) + + def _run(*, history: list[Any], iteration_count: int, eval_requested: bool) -> str: + chosen = choose_next_workflow( + workflows=workflows, + history=history, + iteration_count=iteration_count, + eval_requested=eval_requested, ) - history = [] - actual: list[str] = [] - for iteration_count in range(len(sequence)): - chosen = choose_next_workflow( - workflows=workflows, history=history, iteration_count=iteration_count - ) - assert chosen is not None - actual.append(chosen.id) - history.append( - history_entry_factory( - iteration=iteration_count + 1, - workflow_id=chosen.id, - workflow_set=template, - success=True, - ) + assert chosen is not None + return chosen.id + + # No request standing: eval_reviewer authors the initial check-set on start + # (run_on_start), then outer/inner carry the work and no eval role recurs. + history: list[Any] = [] + seen: list[str] = [] + for iteration_count in range(8): + chosen_id = _run( + history=history, iteration_count=iteration_count, eval_requested=False + ) + seen.append(chosen_id) + history.append( + history_entry_factory( + iteration=iteration_count + 1, + workflow_id=chosen_id, + workflow_set="inner_outer_eval", + success=True, ) - assert actual == sequence + ) + assert seen[0] == "eval_reviewer" + assert "eval_reviewer" not in seen[1:] + assert "eval_runner" not in seen + assert set(seen[1:]) == {"outer", "inner"} + + # Once real work has run, a pending request re-authors then runs the checks: + # eval_reviewer (refresh) is followed by eval_runner (must_follow). + requested_history = [ + history_entry_factory( + iteration=index + 1, + workflow_id=workflow_id, + workflow_set="inner_outer_eval", + success=True, + ) + for index, workflow_id in enumerate(["eval_reviewer", "outer", "inner"]) + ] + assert ( + _run(history=requested_history, iteration_count=3, eval_requested=True) + == "eval_reviewer" + ) + requested_history.append( + history_entry_factory( + iteration=4, + workflow_id="eval_reviewer", + workflow_set="inner_outer_eval", + success=True, + ) + ) + assert ( + _run(history=requested_history, iteration_count=4, eval_requested=True) + == "eval_runner" + ) def test_pm_template_has_no_scheduled_eval_workflows() -> None: @@ -360,54 +442,10 @@ def test_pm_template_has_no_scheduled_eval_workflows() -> None: assert {path.name for path in workflows_root.iterdir()} == {"planner", "dispatcher"} -def test_stock_prompts_use_roster_families_and_semantic_tiers() -> None: - """Role prompts stay provider-neutral while teaching collaboration defaults.""" - - banned = ( - "--harness-agent codex", - "--harness-agent claude", - "--harness-agent gemini", - "gpt-", - "claude-opus", - "claude-sonnet", - "claude-haiku", - "gemini-", - ) - for template in ("inner_outer_eval", "pm_planner_dispatcher"): - for prompt_path in _template_root(template).glob( - ".loopy_loop/workflow_sets/*/workflows/*/prompt.txt" - ): - prompt = prompt_path.read_text(encoding="utf-8").lower() - assert "harness_capability_roster" in prompt - for text in banned: - assert text not in prompt, f"{text!r} in {prompt_path}" - - -def test_eval_check_authoring_prefers_parallel_cross_family_review_without_gate() -> ( - None -): - """Strong eval collaboration is explicit guidance, never a quorum.""" - - prompt = _workflow_prompt( - template="inner_outer_eval", - workflow_set="inner_outer_eval", - workflow="eval_reviewer", - ).lower() - prompt_words = " ".join(prompt.split()) - - assert "different enabled harness families" in prompt_words - assert "analyze goal coverage and likely failure modes in parallel" in prompt_words - assert "different-family reviewers" in prompt_words - assert ( - "not an agent count, family quorum, publication gate, or completion gate" - in prompt_words - ) - - def test_template_tier_examples_name_all_four_canonical_tiers() -> None: """Both starter configs expose the complete semantic tier vocabulary.""" - for template in ("inner_outer_eval", "pm_planner_dispatcher"): + for template in TEMPLATE_SETS: config = ( _template_root(template) .joinpath("loopy_loop_config.yaml") @@ -419,7 +457,7 @@ def test_template_tier_examples_name_all_four_canonical_tiers() -> None: def test_template_gitignores_cover_runtime_state_and_trace_roots() -> None: - for template in ("inner_outer_eval", "pm_planner_dispatcher"): + for template in TEMPLATE_SETS: rules = set( _template_root(template) .joinpath(".gitignore") @@ -429,6 +467,20 @@ def test_template_gitignores_cover_runtime_state_and_trace_roots() -> None: assert RUNTIME_IGNORE_RULES <= rules +def test_root_goal_files_carry_no_loop_mechanics_vocabulary() -> None: + """Root goals read like a product owner wrote them (single-goal §2).""" + + banned = re.compile(r"\b(loop|iteration|workflow|session|cadence)\b", re.IGNORECASE) + for template in TEMPLATE_SETS: + goal = ( + _template_root(template) + .joinpath("loopy_loop_goal.txt") + .read_text(encoding="utf-8") + ) + match = banned.search(goal) + assert match is None, f"{template} goal uses {match and match.group(0)!r}" + + def test_pm_goal_scaffold_requires_a_target_outcome() -> None: goal = ( _template_root("pm_planner_dispatcher") @@ -437,5 +489,5 @@ def test_pm_goal_scaffold_requires_a_target_outcome() -> None: ) assert goal.startswith("REPLACE THIS TEXT") - assert "observable completion criteria" in goal.lower() - assert "mechanism" in goal.lower() + assert "observable" in goal.lower() + assert "product" in goal.lower() diff --git a/src/tests/test_worker.py b/src/tests/test_worker.py index 11a17c5..fb52db2 100644 --- a/src/tests/test_worker.py +++ b/src/tests/test_worker.py @@ -1,5 +1,6 @@ from __future__ import annotations +import json import os from pathlib import Path from typing import Any @@ -179,24 +180,40 @@ def fake_run_harness_iteration(**kwargs: Any) -> IterationResult: run_worker_loop(repo_root=repo_root, coordinator_url="http://coord") - assert "loopy-loop assignment" in captured["prompt"] - assert "Disk prompt body" in captured["prompt"] - assert "Workflow body:" in captured["prompt"] - assert "Goal:" in captured["prompt"] - assert "Completion criteria:" in captured["prompt"] - assert "Stop criteria:" in captured["prompt"] - assert "Workflow set: main" in captured["prompt"] - assert "Session directory:" in captured["prompt"] - assert "Session goal path:" in captured["prompt"] - assert "Session project_state directory:" in captured["prompt"] - assert "Session eval_checks directory:" in captured["prompt"] - assert "Session updates_from_user path:" in captured["prompt"] - assert "Session child_requests directory:" in captured["prompt"] - assert "Session control path:" in captured["prompt"] - assert "Session finished ledger path:" in captured["prompt"] - assert "Session harness outputs directory:" in captured["prompt"] - # Assignment ID should NOT appear. - assert "Assignment ID" not in captured["prompt"] + prompt = captured["prompt"] + # Diet header contract (single-goal-assignments.md §3). + assert prompt.startswith("loopy-loop assignment — iteration 0001, role: planner") + assert "Disk prompt body" in prompt + assert "Workflow body:" in prompt + assert "Goal:" in prompt + assert "Completion criteria:" in prompt + assert "Stop criteria:" in prompt + assert "You are inside a durable looping session. Key paths:" in prompt + assert "- session dir:" in prompt + assert "- project_state/" in prompt + assert "- child_requests/pending/" in prompt + assert "- control.json" in prompt + assert "- scratch dir (this iteration):" in prompt + assert "- paths.json:" in prompt + # The old inlined path enumeration and envelope dump are gone. + assert "Session directory:" not in prompt + assert "Absolute paths from the assignment envelope:" not in prompt + assert "Assignment ID" not in prompt + # The full machine path map now lives in a sibling paths.json. + paths_file = ( + repo_root + / ".loopy_loop" + / "sessions" + / "goal_20260419_143022_ab12cd34" + / "iterations" + / "0001_planner" + / "paths.json" + ) + assert paths_file.is_file() + paths = json.loads(paths_file.read_text(encoding="utf-8")) + assert paths["schema_version"] == 1 + assert paths["session_paths"]["project_state"].endswith("project_state") + assert paths["previous_worker_sessions"] is None def test_worker_includes_goal_check_path_for_emitting_workflow( @@ -238,9 +255,20 @@ def fake_run_harness_iteration(**kwargs: Any) -> IterationResult: ) _run_task(repo_root=repo_root, task=task) - assert "goal_check.json output path:" in captured["prompt"] - assert "0003_eval_runner/goal_check.json" in captured["prompt"] - assert "Iteration harness output root:" in captured["prompt"] + # goal_check output and the iteration scratch root move to paths.json. + assert "goal_check.json output path:" not in captured["prompt"] + paths_file = ( + repo_root + / ".loopy_loop" + / "sessions" + / "goal_20260419_143022_ab12cd34" + / "iterations" + / "0003_eval_runner" + / "paths.json" + ) + paths = json.loads(paths_file.read_text(encoding="utf-8")) + assert paths["goal_check_output"].endswith("0003_eval_runner/goal_check.json") + assert "harness_outputs/0003_eval_runner" in paths["scratch_dir"] assert "harness_outputs/0003_eval_runner" in captured["prompt"] @@ -263,20 +291,140 @@ def test_render_prompt_includes_parent_session_for_child( parent_session_id=parent_session_id, ) + iteration_dir = child_dir / "iterations" / "0001_inner" prompt = _render_prompt( config_snapshot=snapshot_factory(), session_id=child_session_id, workflow_set="inner_outer_eval", iteration=1, workflow_id="inner", - iteration_dir=child_dir / "iterations" / "0001_inner", + iteration_dir=iteration_dir, harness_output_root=child_dir / "harness_outputs" / "0001_inner", workflow_prompt="Do child work.", repo_root=repo_root, ) - assert "Parent session directory:" in prompt - assert parent_session_id in prompt + # The parent session directory moves from the header into paths.json. + assert "Parent session directory:" not in prompt + paths = json.loads((iteration_dir / "paths.json").read_text(encoding="utf-8")) + assert paths["parent_session_dir"] is not None + assert parent_session_id in paths["parent_session_dir"] + + +def _render_synthetic_workflow_set( + *, + repo_root: Path, + snapshot_factory: Any, + goal: str, + preamble: str | None, + completion_criteria: list[str] | None = None, + stop_criteria: list[str] | None = None, +) -> str: + session_id = "goal_20260419_143022_ab12cd34" + session_dir = create_session_dir( + repo_root=repo_root, + session_id=session_id, + goal_hash="ab12cd34ef56", + workflow_set="synthetic", + ) + if preamble is not None: + preamble_path = ( + repo_root / ".loopy_loop" / "workflow_sets" / "synthetic" / "preamble.txt" + ) + preamble_path.parent.mkdir(parents=True, exist_ok=True) + preamble_path.write_text(preamble, encoding="utf-8") + snapshot = snapshot_factory( + goal=goal, + completion_criteria=completion_criteria + if completion_criteria is not None + else [], + stop_criteria=stop_criteria if stop_criteria is not None else [], + ) + return _render_prompt( + config_snapshot=snapshot, + session_id=session_id, + workflow_set="synthetic", + iteration=26, + workflow_id="outer", + iteration_dir=session_dir / "iterations" / "0026_outer", + harness_output_root=session_dir / "harness_outputs" / "0026_outer", + workflow_prompt="Do the synthetic role work.", + repo_root=repo_root, + ) + + +def test_render_header_matches_diet_shape( + repo_root: Any, snapshot_factory: Any +) -> None: + prompt = _render_synthetic_workflow_set( + repo_root=repo_root, + snapshot_factory=snapshot_factory, + goal="Deliver the thing.", + preamble=None, + ) + + assert prompt.startswith( + "loopy-loop assignment — iteration 0026, role: outer, " + "session: goal_20260419_143022_ab12cd34\n\nGoal:\nDeliver the thing.\n" + ) + # Empty criteria lists omit their sections entirely. + assert "Completion criteria:" not in prompt + assert "Stop criteria:" not in prompt + assert "You are inside a durable looping session. Key paths:" in prompt + for label in ( + "- session dir:", + "- project_state/", + "- child_requests/pending/", + "- control.json", + "- scratch dir (this iteration):", + "- paths.json:", + ): + assert label in prompt + # No shared preamble → no ground-rules section. + assert "Shared ground rules:" not in prompt + assert prompt.rstrip().endswith("Workflow body:\nDo the synthetic role work.") + + +def test_render_includes_preamble_when_present( + repo_root: Any, snapshot_factory: Any +) -> None: + prompt = _render_synthetic_workflow_set( + repo_root=repo_root, + snapshot_factory=snapshot_factory, + goal="Deliver the thing.", + preamble="Write atomically. Never force-push.", + completion_criteria=["It works"], + stop_criteria=["A blocker is recorded"], + ) + + assert "Completion criteria:\n- It works" in prompt + assert "Stop criteria:\n- A blocker is recorded" in prompt + assert "Shared ground rules:\nWrite atomically. Never force-push." in prompt + # The preamble is included before the workflow body. + assert prompt.index("Shared ground rules:") < prompt.index("Workflow body:") + + +def test_render_header_budget_excludes_goal_and_preamble( + repo_root: Any, snapshot_factory: Any +) -> None: + goal = "G" * 20_000 + preamble = "P" * 20_000 + prompt = _render_synthetic_workflow_set( + repo_root=repo_root, + snapshot_factory=snapshot_factory, + goal=goal, + preamble=preamble, + completion_criteria=["It works"], + stop_criteria=["A blocker is recorded"], + ) + + before_body = prompt.split("\n\nWorkflow body:", 1)[0] + scaffold_bytes = ( + len(before_body.encode("utf-8")) + - len(goal.encode("utf-8")) + - len(preamble.encode("utf-8")) + ) + assert scaffold_bytes <= 2048, scaffold_bytes def test_worker_uses_config_snapshot_not_disk(