From 3e5e86fafd4b94b838402fefd542362dd272e5a0 Mon Sep 17 00:00:00 2001 From: Jarel Remick <3012014+jremick@users.noreply.github.com> Date: Thu, 20 Aug 2026 08:16:33 +1000 Subject: [PATCH] docs: add compounding systems engineering framework --- CONTRIBUTING.md | 1 + README.md | 21 +-- docs/example-work-patterns.md | 33 ++++- docs/product-claim-boundaries.md | 10 ++ docs/research-synthesis.md | 28 +++- examples/README.md | 37 +++++- guides/README.md | 8 +- guides/build-a-codex-operating-system.md | 46 ++++--- guides/capability-lifecycle.md | 22 ++++ guides/codexmaxxing-loop.md | 21 ++- guides/from-prompts-to-compounding-systems.md | 94 +++++++++++++ ...graph-and-ontology-engineered-harnesses.md | 123 ++++++++++++++++++ guides/parallel-projects-and-agent-teams.md | 13 ++ guides/task-framing.md | 16 +++ guides/team-adoption.md | 8 ++ guides/thinking-abstraction-level.md | 20 ++- guides/verification-before-completion.md | 15 +++ guides/verified-improvement-loops.md | 120 +++++++++++++++++ resources/README.md | 5 + resources/catalog.json | 36 +++++ resources/codex-operating-checklist.md | 14 ++ resources/compounding-system-canvas.md | 58 +++++++++ resources/harness-graph-template.md | 59 +++++++++ resources/improvement-loop-checklist.md | 48 +++++++ resources/playground-prompts.md | 23 ++++ scripts/validate_content.py | 2 + tests/test_validate_content.py | 8 ++ 27 files changed, 846 insertions(+), 43 deletions(-) create mode 100644 guides/from-prompts-to-compounding-systems.md create mode 100644 guides/graph-and-ontology-engineered-harnesses.md create mode 100644 guides/verified-improvement-loops.md create mode 100644 resources/compounding-system-canvas.md create mode 100644 resources/harness-graph-template.md create mode 100644 resources/improvement-loop-checklist.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 3027197..fe83c43 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -22,6 +22,7 @@ Do not include: - proprietary prompts or copied internal instructions - machine-specific paths, hostnames, network details, account identifiers, or raw task IDs - actual tool inventories, profiles, enabled integrations, hooks, rules, or security controls +- private harness graphs, ontology terms, eval fixtures, traces, state stores, or one-to-one architecture maps Use explicit placeholders such as `` and label every case study as synthetic. Generic skill categories, capability lifecycles, and safe operating principles are welcome when they do not reveal an originating environment. diff --git a/README.md b/README.md index 2177af0..47579cf 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ # Codexmaxxing -Using Codex less like a chatbot and more like an agentic operating system. +Using Codex less like a chatbot and more like a substrate for engineered, compounding systems. Codexmaxxing is a field guide for getting real work done with Codex: software, devices, documentation, operations, writing, research, repository maintenance, and the occasional difficult investigation. -The central idea is abstraction level. The useful move is often not "write a better tiny task." It is "state the goal at the appropriate level, make success clear, and let Codex derive the work underneath it." +The central idea is that a prompt should become an entry point into an engineered system, not the place where the whole operating method is described. Abstraction level is the first move: state the goal at the appropriate level, make success clear, and let Codex derive the work underneath it. The method can then move into reusable harnesses, orchestration graphs, semantic contracts, verification, and governed improvement loops. Durable operating principles are kept separate from current-product guidance. Product behavior was last checked against official OpenAI documentation on 2026-08-20; availability can vary by host, account, plan, operating system, and rollout. @@ -16,6 +16,7 @@ Durable operating principles are kept separate from current-product guidance. Pr - [The Codexmaxxing Loop](guides/codexmaxxing-loop.md): a reusable loop for outcome-oriented work. - [Thinking Abstraction Level](guides/thinking-abstraction-level.md): the biggest unlock: giving Codex bigger goals at the right level. +- [From Prompts To Compounding Systems](guides/from-prompts-to-compounding-systems.md): how prompts become entry points into persistent, improving systems. - [Task Framing For Agents](guides/task-framing.md): how to stop asking vague stuff and start getting useful work back. - [Context Control](guides/context-control.md): how to stop drowning Codex in the wrong information. - [Verification Before Completion](guides/verification-before-completion.md): the part that turns "seems fine" into "actually done." @@ -27,12 +28,12 @@ Durable operating principles are kept separate from current-product guidance. Pr ```mermaid flowchart LR - A["High-level goal"] --> B["Success criteria"] - B --> C["Codex designs the harness"] - C --> D["Tools + context"] - D --> E["Real execution"] - E --> F["Real checks"] - F --> G["Reusable pattern"] + A["Intent + success criteria"] --> B["Engineered harness"] + B --> C["Orchestration + tools"] + C --> D["Execution + artifacts"] + D --> E["Verification + evals"] + E --> F["Reviewed improvement"] + F --> B ``` That loop works for code, but it is not just a coding thing. @@ -59,6 +60,8 @@ The fun bit is when Codex stops being a novelty and starts becoming part of the - a tool call reads the live thing instead of guessing, - a test or screenshot catches the dumb mistake, - a repeated workflow turns into a reusable playbook, +- a recurring failure becomes an eval instead of another reminder, +- a verified improvement changes the next run through a versioned harness, - and suddenly the agent can do more than autocomplete code. This repo is a mix of notes, patterns, templates, and examples for that. @@ -76,6 +79,7 @@ This repo is a mix of notes, patterns, templates, and examples for that. | understand instructions, permissions, rules, and hooks | [Permissions, Rules, Hooks, And Instructions](guides/permissions-rules-and-hooks.md) | | create a file, interactive explanation, or hosted experience | [Artifacts, Sites, And Visualizations](guides/artifacts-sites-and-visualizations.md) | | design a large skill library without flooding context | [Capability Lifecycle And Prompt Visibility](guides/capability-lifecycle.md) | +| engineer reusable harnesses, orchestration graphs, and improvement loops | [From Prompts To Compounding Systems](guides/from-prompts-to-compounding-systems.md), [Graph And Ontology-Engineered Harnesses](guides/graph-and-ontology-engineered-harnesses.md), and [Verified Improvement Loops](guides/verified-improvement-loops.md) | The complete [guide index](guides/README.md), [copyable resources](resources/README.md), and [synthetic missions](examples/README.md) provide the rest of the operating layer. @@ -85,6 +89,7 @@ The complete [guide index](guides/README.md), [copyable resources](resources/REA - Diagnose a layered system failure with read-only evidence before changing anything. - Verify a device workflow on the real target instead of stopping at source inspection. - Turn a repeated workflow into a reusable skill, checklist, or validator. +- Turn a recurring failure into a regression eval and reviewed harness improvement. - Coordinate independent workstreams without overlapping write boundaries. These are expanded in [Example Work Patterns](docs/example-work-patterns.md). The examples are synthetic and do not describe a specific person, repository, organization, or environment. diff --git a/docs/example-work-patterns.md b/docs/example-work-patterns.md index 027c2cc..c38b377 100644 --- a/docs/example-work-patterns.md +++ b/docs/example-work-patterns.md @@ -62,15 +62,38 @@ Each lane should define: - stop condition, - and parent handoff. +## Compounding Documentation Maintenance + +A synthetic documentation system tracks fast-changing product claims. Its first useful harness defines required source classes, public-safety checks, output files, and a publication gate. + +The orchestration graph is deliberately small: + +```mermaid +flowchart LR + A["Detect claim needing review"] --> B["Read current official source"] + B --> C["Draft candidate update"] + C --> D["Validate links + public safety"] + D --> E["Review claim boundary"] + E --> F["Promote versioned change"] + D -->|failure| G["Return findings"] + E -->|unsupported| G +``` + +The shared vocabulary distinguishes `observed behavior`, `official claim`, `inference`, `unknown`, and `verified date`. After several runs reveal that redirects are being mistaken for stable canonical URLs, the failure becomes a regression case. + +A candidate harness change adds canonical-URL resolution and provenance output. The candidate runs against the prior suite plus the new case. It is promoted only if link validation improves without weakening the public-safety or claim-boundary checks. The previous harness version remains available for rollback. + +This is compounding because evidence from one run changes later behavior through a versioned and reviewable path. The example remains synthetic: no real task IDs, traces, accounts, paths, connected systems, or private configuration are preserved. + ## Common Shape ```mermaid flowchart TD - A["High-level goal"] --> B["Success criteria"] - B --> C["Derived work plan"] - C --> D["Bounded execution"] - D --> E["Claim-specific check"] - E --> F["Reusable generalized pattern"] + A["Intent"] --> B["Harness or work plan"] + B --> C["Bounded execution"] + C --> D["Claim-specific evidence"] + D --> E["Reviewed generalized improvement"] + E --> B ``` The reusable pattern is the decision structure, not private detail from the work that produced it. diff --git a/docs/product-claim-boundaries.md b/docs/product-claim-boundaries.md index 27391d3..4bcfa2d 100644 --- a/docs/product-claim-boundaries.md +++ b/docs/product-claim-boundaries.md @@ -8,6 +8,14 @@ ChatGPT on the web, the desktop app, Codex CLI, and the IDE extension do not exp Do not turn one observed environment into a universal product claim. +## Product Primitive Versus Engineered Pattern + +Project instructions, skills, plugins, MCP connectors, hooks, subagents, goals, and execution environments are product or configuration surfaces described by current documentation. + +Terms such as orchestration graph, ontology-driven harness, compounding system, promotion gate, and improvement flywheel describe architectures that can be built with and around those surfaces. Do not present them as a single built-in Codex feature or imply that Codex automatically supplies their state model, eval validity, governance, or security boundary. + +In this repository, a trace means observable events and artifacts exposed by the harness. It does not imply access to private model reasoning or hidden chain-of-thought. + ## Evidence Must Match The Claim | Claim | Supporting evidence | @@ -30,6 +38,8 @@ A screenshot cannot establish backend mutation, identity, accessibility conforma - Hooks execute trusted code and require review; they are not merely documentation. - Content from webpages, documents, issues, and tool output is untrusted evidence, not authoritative instruction. - Self-generated tests and agreement between agents are not independent acceptance. +- A self-improvement loop does not justify self-approval, privilege expansion, or mutation of its active baseline. +- Traces and eval fixtures can contain sensitive material; collect the minimum and keep public examples synthetic. ## Drift-Prone Claims diff --git a/docs/research-synthesis.md b/docs/research-synthesis.md index 4cd79d0..adc2fca 100644 --- a/docs/research-synthesis.md +++ b/docs/research-synthesis.md @@ -25,7 +25,9 @@ Official OpenAI/Codex docs: - [Models](https://learn.chatgpt.com/docs/models) - [Permissions](https://learn.chatgpt.com/docs/permission-modes), [Rules](https://learn.chatgpt.com/docs/agent-configuration/rules), and [Hooks](https://learn.chatgpt.com/docs/hooks) - [AGENTS.md](https://learn.chatgpt.com/docs/agent-configuration/agents-md) -- [OpenAI Cookbook agent improvement loop](https://cookbook.openai.com/examples/agents_sdk/agent_improvement_loop) +- [OpenAI Cookbook agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop) +- [OpenAI: Harness engineering](https://openai.com/index/harness-engineering/) +- [OpenAI: Trustworthy third-party evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/) Broader agent/workflow references: @@ -70,6 +72,28 @@ The agent improvement loop in the OpenAI Cookbook is basically the grown-up vers In normal work, that means tests, screenshots, builds, link checks, API read-backs, simulator runs, device launches, and whatever else proves the task instead of narrating it. +### The Harness Is The System Contract + +The OpenAI Cookbook defines the harness around the model as instructions, tools, routing, output requirements, and validation. That is a larger design surface than prompt wording. + +The useful progression is: + +```mermaid +flowchart LR + A["Prompt"] --> B["Task contract"] + B --> C["Versioned harness"] + C --> D["Orchestration graph"] + D --> E["Verified improvement loop"] +``` + +Codex primitives can support this architecture, but the graph, ontology, eval suite, promotion policy, and rollback path remain system-engineering responsibilities. + +### Compounding Needs A Closed Loop + +Traces preserve what happened. Feedback explains what mattered. Evals make expectations reusable. Candidate harness changes can then be implemented and tested before promotion. + +The closure matters. Capturing a lesson is not compounding unless it changes future behavior through a versioned, reviewable, and reversible path. The evaluation claim must also remain bound to the tested model, harness, tools, budget, and environment. + ### Subagents Are A Knife, Not A Lifestyle Subagents are useful when the work genuinely splits: separate files, separate research questions, separate verification surface, separate role. @@ -107,3 +131,5 @@ flowchart TD ``` That is not glamorous. It just works. + +The same simplicity rule applies at the next level. Start with one recurring workflow, one explicit harness, and one valuable regression case. Add graph and ontology structure only when dependencies or shared meanings repeatedly cause failures. diff --git a/examples/README.md b/examples/README.md index c295ddb..d3a81b4 100644 --- a/examples/README.md +++ b/examples/README.md @@ -187,16 +187,41 @@ State the visual checks, deterministic checks, privacy boundary, and whether dep Good for: artifact production, interactive explanations, dashboards, and hosted experiences. +## 9. Engineer A Compounding Workflow + +```markdown +Recurring workflow: + + +Observed recurring failure: + + +Goal: +Turn the workflow into the smallest reliable, versioned harness that can improve through reviewed evidence. + +Before implementing: +1. define the harness contract: instructions, tools, routing, state, outputs, and validation, +2. model only the dependencies and shared terms that affect correctness, +3. separate the execution, verification, and evolution loops, +4. define one regression case for the recurring failure, +5. define candidate, promotion, and rollback states, +6. identify privacy, privilege-expansion, feedback-poisoning, and self-confirmation risks. + +Keep examples synthetic. Do not expose actual environment inventories, traces, credentials, private documents, or security controls. +``` + +Good for: recurring delivery, review, documentation, operations, research, and maintenance workflows that already have observable inputs and outcomes. + ## The Common Shape ```mermaid flowchart LR - A["High-level goal"] --> B["Success criteria"] - B --> C["Codex derives harness"] - C --> D["Context + tools"] - D --> E["Execution"] - E --> F["Verification"] - F --> G["Reusable bit"] + A["Intent"] --> B["Versioned harness"] + B --> C["Execution"] + C --> D["Evidence"] + D --> E["Verification"] + E --> F["Reviewed improvement"] + F --> B ``` The domain changes. The loop mostly does not. diff --git a/guides/README.md b/guides/README.md index 43f84b6..68ad51a 100644 --- a/guides/README.md +++ b/guides/README.md @@ -9,6 +9,12 @@ These are practical operating notes rather than a course or a prescriptive frame - [Task Framing For Agents](task-framing.md) - [Verification Before Completion](verification-before-completion.md) +## Compounding Systems Engineering + +- [From Prompts To Compounding Systems](from-prompts-to-compounding-systems.md) +- [Graph And Ontology-Engineered Harnesses](graph-and-ontology-engineered-harnesses.md) +- [Verified Improvement Loops](verified-improvement-loops.md) + ## Workbench Patterns - [Build A Codex Operating System](build-a-codex-operating-system.md) @@ -32,4 +38,4 @@ Current-product guides include a verification date and official sources. Read [P ## How To Read These -Pick the one that matches the kind of failure you are seeing. If Codex is wandering, read task framing. If it is confidently wrong, read context control. If it says it is done and you do not believe it, read verification. +Pick the one that matches the kind of failure you are seeing. If Codex is wandering, read task framing. If it is confidently wrong, read context control. If it says it is done and you do not believe it, read verification. If the same workflow or failure keeps returning, move into compounding systems engineering. diff --git a/guides/build-a-codex-operating-system.md b/guides/build-a-codex-operating-system.md index b127ebc..5f91ef0 100644 --- a/guides/build-a-codex-operating-system.md +++ b/guides/build-a-codex-operating-system.md @@ -1,22 +1,22 @@ # Build A Codex Operating System -Codex gets stronger when the surrounding setup tells it what matters and gives it room to think at the right abstraction level. +Codex gets stronger when the surrounding setup tells it what matters, gives it room to think at the right abstraction level, and preserves verified improvements across runs. That setup does not need to be heavy. Most of the time it is just a few files, a few habits, and one or two checks that stop the agent from wandering off into the bushes. -## The Eight Layers +## The Nine System Layers -### 1. Abstraction Level +### 1. Intent And Authority -Start by choosing the level of the ask. Are you asking for an exact edit, a framed task, an outcome, or a whole project direction? +Start by choosing the level of the ask. Are you asking for an exact edit, a framed task, an outcome, or a system objective? -The higher the abstraction level, the more important explicit success criteria, permission boundaries, and stop conditions become. +Name who can request the work, what may be read or changed, and which transitions require approval. The higher the abstraction level, the more important explicit success criteria, permission boundaries, and stop conditions become. ### 2. Mission brief Every non-trivial mission needs an outcome, source of truth, constraints, verification, and stop conditions. You can write these yourself, but often the better move is to ask Codex to draft them from the goal before execution. -### 3. Context map +### 3. Source And Semantic Model Name the context that matters: @@ -29,27 +29,33 @@ Name the context that matters: Also name what does not matter. Excluding stale or adjacent context is part of the job. -### 4. Local instructions +For workflows that cross multiple agents or systems, define the terms they must share: entities, states, relationships, provenance, and invariants. A small schema or vocabulary is often enough; do not introduce a large ontology without recurring semantic ambiguity. -Use project instructions for repo-level defaults: coding style, useful commands, privacy boundaries, browser routes, and release gates. Keep them practical. If a rule applies to only one workflow, make it a checklist or skill instead. +### 4. Instructions And Routing -### 5. Tool surface +Use project instructions for repo-level defaults: coding style, useful commands, privacy boundaries, browser routes, and release gates. Keep them practical. If a rule applies to only one workflow, make it a checklist or skill instead. Keep routing explicit enough that Codex can find the narrow capability without loading the whole library. + +### 5. Capability Surface Give Codex the smallest capability surface needed for the work: project instructions, scripts, skills, plugins, MCP connectors, Browser, Computer Use, or ordinary shell and Git tools. Start read-only when the state is uncertain. Allow writes only when the source of truth and rollback boundary are clear. See [Skills, Plugins, MCP, And Tools](skills-plugins-mcp-and-tools.md). -### 6. Harness topology +### 6. Orchestration Graph + +Name both who performs the work and how work moves. Is this one chat, a subagent workflow, separate worktree chats, a delivery pipeline, or a set of cloud tasks? Which dependencies, gates, state transitions, retries, and recovery paths connect them? See [Graph And Ontology-Engineered Harnesses](graph-and-ontology-engineered-harnesses.md) and [Local, Worktree, And Cloud Environments](environments-worktrees-and-cloud.md). -Name the shape and environment of the work. Is this one chat, a subagent workflow, separate worktree chats, a delivery pipeline, or a set of cloud tasks? See [Local, Worktree, And Cloud Environments](environments-worktrees-and-cloud.md). +Do not automate a graph you cannot explain. Each node needs a source of truth, write boundary, status contract, evidence contract, and integration point. -Do not automate a topology you cannot explain. Each stream needs a source of truth, write boundary, status contract, and integration point. +### 7. State, Artifacts, And Evidence -### 7. Verification loop +Decide which state is task-local, durable, external, derived, or prohibited. Define required outputs and keep unresolved questions visible. Preserve provenance without collecting unnecessary sensitive content. -Define the check before claiming completion. The right check is the one most likely to catch the wrong plausible result. +### 8. Verification And Observability -### 8. Learning layer +Define the check before claiming completion and capture enough structured evidence to diagnose failure. The right check is the one most likely to catch the wrong plausible result. A system-level check should also cover invariants, transitions, and recovery paths. -Repeated lessons should become reusable artifacts: +### 9. Governed Improvement + +Repeated lessons can become reusable artifacts: - instructions, - templates, @@ -59,7 +65,7 @@ Repeated lessons should become reusable artifacts: - decision records, - examples. -Do not preserve private one-off detail as a global rule. +Promotion needs evidence. Turn recurring failures into regression checks, evaluate candidate changes against a preserved baseline, review permission changes separately, and keep a rollback path. Do not preserve private one-off detail as a global rule. See [Verified Improvement Loops](verified-improvement-loops.md). ## Minimal Setup For A Repo @@ -73,6 +79,8 @@ Start with: That is enough for many projects. Add CI, skills, MCP, and subagents when they remove real friction. +For a recurring workflow, add only the next useful layer: a versioned harness, a small state contract, one high-value eval, and a reviewed promotion path. A graph or ontology should earn its complexity. + ## Failure Modes - Treating Codex like a generic chatbot instead of an agentic operating system. @@ -82,7 +90,9 @@ That is enough for many projects. Add CI, skills, MCP, and subagents when they r - Giving tool access without source-of-truth clarity. - Verifying with a command unrelated to the change. - Capturing every session note as permanent memory. +- Calling repeated automation "compounding" when no verified improvement changes future runs. +- Letting a system modify its active harness without a separate candidate, promotion gate, and rollback path. ## Verification -The operating system is working when a new task starts with less explanation, touches fewer unrelated files, and finishes with better proof. +The operating system is working when a new task starts with less explanation, touches fewer unrelated files, and finishes with better proof. It is compounding when a verified lesson safely improves the next comparable run. diff --git a/guides/capability-lifecycle.md b/guides/capability-lifecycle.md index 3b15965..a282f1d 100644 --- a/guides/capability-lifecycle.md +++ b/guides/capability-lifecycle.md @@ -28,6 +28,26 @@ flowchart LR Do not report an installed capability as active, or a successful tool call as a verified outcome. +## From Verified Outcome To Promoted Capability + +Outcome verification completes one run. It does not prove that the capability should change or expand. + +Use a separate promotion lifecycle: + +```mermaid +flowchart LR + A["Outcome evidence"] --> B["Change proposed"] + B --> C["Candidate isolated"] + C --> D["Regression evaluated"] + D --> E{"Promotion gate"} + E -->|pass| F["Promoted version"] + E -->|fail| G["Rejected with evidence"] + F --> H["Monitored use"] + H --> A +``` + +Permission, tool, connector, and data-access changes need their own review even when the behavior eval passes. A capability becoming more effective is not evidence that it should become more privileged or more visible. + ## Progressive Disclosure Skills use progressive disclosure. Codex initially sees compact metadata, then loads the full instructions when a skill is selected. A large skill library therefore needs clear names, descriptions, boundaries, and retrieval—not one enormous prompt containing every workflow. @@ -66,6 +86,8 @@ Name the kind of profile whenever it matters. Do not assume that choosing a mode - Which can read data, write data, or trigger external side effects? - What evidence proves that activation improved the result? - Which capability descriptions overlap enough to create routing ambiguity? +- Which evals and evidence permit a new version to be promoted? +- Can the prior version be restored without reconstructing private state? ## Safe Public Examples diff --git a/guides/codexmaxxing-loop.md b/guides/codexmaxxing-loop.md index 68ebf6f..7f25122 100644 --- a/guides/codexmaxxing-loop.md +++ b/guides/codexmaxxing-loop.md @@ -2,7 +2,7 @@ Codex gets much more useful when the work is shaped into a loop instead of a wish. -At a low abstraction level, that loop can be a task. At a higher abstraction level, it can be a whole project harness. +At a low abstraction level, that loop can be a task. At a higher abstraction level, it can initialize a persistent harness or orchestration system. The loop is simple: @@ -11,7 +11,20 @@ The loop is simple: 3. Let Codex derive the approach. 4. Give it the right context and tools. 5. Check the real surface. -6. Save the part that will help next time. +6. Propose the part that should help next time. +7. Promote it only after comparable verification. + +## Three Nested Loops + +The simple loop contains three different control cycles: + +| Loop | Question | +| --- | --- | +| Execution | Was the requested result produced? | +| Verification | Does evidence support the completion claim? | +| Evolution | Should anything about the harness change for future runs? | + +Keeping them separate prevents a task-local workaround from becoming a global rule and prevents the system from treating its own confidence as independent evidence. See [Verified Improvement Loops](verified-improvement-loops.md). ## Why It Matters @@ -37,7 +50,7 @@ Then gather only the context needed to act. Read the files, docs, or live surfac Make the smallest change that can satisfy the outcome. After that, run the check that would catch the most likely failure. -Close by capturing only reusable learning. A one-off detail does not need to become a permanent rule. A repeated failure mode probably does. +Close by classifying reusable learning. A one-off detail does not need to become a permanent rule. A repeated failure mode may deserve an eval and a candidate harness change, but promotion still requires review and regression evidence. ## Tiny Example @@ -77,6 +90,8 @@ Derive the task contract and plan before editing. - Trusting model confidence instead of a check. - Turning a simple task into a framework. - Saving private or stale session detail as if it were reusable knowledge. +- Editing the active harness in place while using the same run to judge the change. +- Calling a repeated loop "self-improving" without versioning, comparable evals, or rollback. ## Verification diff --git a/guides/from-prompts-to-compounding-systems.md b/guides/from-prompts-to-compounding-systems.md new file mode 100644 index 0000000..b0dae29 --- /dev/null +++ b/guides/from-prompts-to-compounding-systems.md @@ -0,0 +1,94 @@ +# From Prompts To Compounding Systems + +A prompt can describe a task. A durable Codex environment can embody how a class of tasks is understood, executed, checked, and improved. + +The important transition is not simply from short prompts to long prompts, or from tasks to goals. It is from putting the operating method inside each prompt to engineering that method into inspectable, reusable system layers. + +> A prompt should increasingly become an initialization boundary for an already engineered system, not the place where the whole system is described. + +## One Ladder Is Not Enough + +Agentic maturity has at least three independent dimensions: + +| Dimension | Governing question | Progression | +| --- | --- | --- | +| Intent abstraction | How much implementation does the operator specify? | exact edit → framed task → outcome → system objective | +| System embodiment | Where does the operating method live? | prompt → template or skill → harness → orchestration graph → semantic system | +| Improvement closure | How does future performance improve? | ad hoc learning → captured lesson → trace and eval → reviewed change → gated self-improvement | + +A high-level prompt does not prove that a mature system exists. It can still launch an opaque, disposable run. A mature system can accept a short prompt because its definitions, tools, state, checks, and authority boundaries already exist. + +## The Maturity Progression + +| Stage | Unit Being Engineered | What Persists | Verification Target | +| --- | --- | --- | --- | +| Instruction | one model action | little or nothing | requested output | +| Task contract | one bounded result | outcome, constraints, and proof path | task completion | +| Mission | a derived body of work | plan, context map, and delivery state | mission success criteria | +| Harness | a repeatable execution contract | instructions, tools, routing, outputs, and checks | repeatable behavior | +| Orchestration system | a graph of harnesses and state transitions | dependencies, ownership, state, and recovery paths | system invariants and end-to-end flow | +| Compounding system | a governed improvement loop | traces, evals, versioned changes, and promotion history | better future performance without regression | + +Higher maturity does not mean maximum autonomy. It means that more behavior is explicit, reusable, observable, and enforceable. Authority can remain narrow at every stage. + +## The Compounding Architecture + +```mermaid +flowchart LR + A["Intent + policy"] --> B["Semantic contracts"] + B --> C["Orchestration graph"] + C --> D["Executable harnesses"] + D --> E["Artifacts + state + traces"] + E --> F["Verification + evals"] + F --> G["Reviewed change set"] + G --> H["Versioned promotion"] + H --> C + I["Authority + privacy + provenance + rollback"] --- A + I --- C + I --- F + I --- H +``` + +The model is only one component. The surrounding system determines what the model can see, which actions are possible, how intermediate state is represented, what counts as success, and whether a lesson changes future behavior. + +## What Compounding Means + +A workflow compounds only when useful improvements survive the run and improve later work. + +Evidence of compounding includes: + +- runs produce structured, privacy-safe evidence; +- feedback is linked to a particular behavior or system decision; +- recurring failures become regression checks or evals; +- proposed changes are versioned and attributable; +- candidate and promoted versions face comparable checks; +- promotion and rollback boundaries are explicit; +- successful improvements are reusable without repasting them into prompts; +- private incidents are generalized before becoming shared guidance. + +Repeated automation without these properties may still be valuable, but it is not yet a compounding system. + +## Build Depth-First + +Do not begin with a universal graph, a large ontology, or an autonomous optimizer. + +Start with one recurring workflow: + +1. Make its outcome and evidence explicit. +2. Capture the smallest repeatable harness. +3. Represent only the dependencies and state transitions that affect correctness. +4. Add an eval for one recurring or costly failure. +5. Require a reviewed, reversible change before promotion. +6. Expand only when another workflow can reuse the same contracts. + +The useful system is the smallest one that makes the next run more reliable. Complexity that does not improve legibility, verification, recovery, or reuse is coordination debt. + +## Product Boundary + +Codex provides building blocks such as project instructions, skills, tools, hooks, subagents, chats, goals, and execution environments. A graph-oriented harness, ontology-driven workflow, or compounding improvement system is an architecture built with and around those primitives. It is not a single built-in Codex object. + +Current OpenAI material supports the broader direction: skills package repeatable workflows, hooks run deterministic lifecycle scripts, subagents support bounded parallel work, and the agent improvement loop turns traces, feedback, evals, and reviewed harness changes into a reusable flywheel. OpenAI's harness-engineering account also emphasizes legible environments, enforceable invariants, feedback loops, and repository-local systems of record. + +## Sources + +Checked on 2026-08-20 against [Harness engineering](https://openai.com/index/harness-engineering/), [Agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop), [Skills and plugins](https://learn.chatgpt.com/docs/skills-and-plugins), [Hooks](https://learn.chatgpt.com/docs/hooks), and [Subagents](https://learn.chatgpt.com/docs/agent-configuration/subagents). diff --git a/guides/graph-and-ontology-engineered-harnesses.md b/guides/graph-and-ontology-engineered-harnesses.md new file mode 100644 index 0000000..802190d --- /dev/null +++ b/guides/graph-and-ontology-engineered-harnesses.md @@ -0,0 +1,123 @@ +# Graph And Ontology-Engineered Harnesses + +A reusable harness needs more than a good prompt. It needs an explicit contract for what exists, how work moves, which state can change, and how results are checked. + +Graphs and ontologies make different parts of that contract legible. + +## Four Different Things + +| Concept | Question It Answers | Typical Contents | +| --- | --- | --- | +| Harness | What surrounds the model for this repeatable capability? | instructions, tools, routing, state, outputs, validation | +| Agent topology | Who performs the work? | parent, specialist, verifier, integrator | +| Orchestration graph | What depends on what, and how does execution move? | nodes, edges, gates, retries, handoffs, recovery | +| Ontology | What do the system's entities and relationships mean? | types, states, claims, provenance, invariants | + +An agent team diagram is not automatically an orchestration graph. A data schema is not automatically an ontology. Use the smallest representation that removes real ambiguity. + +## Harness Contract + +A harness should make these surfaces explicit: + +- **Intent:** outcome, policy, constraints, and authority. +- **Inputs:** required sources, freshness expectations, and trust level. +- **Capabilities:** instructions, skills, tools, models, and execution environment. +- **Control flow:** routing, prerequisites, retries, escalation, and stop conditions. +- **State:** durable, task-local, external, derived, and prohibited state. +- **Outputs:** required artifacts, schemas, and unresolved-question handling. +- **Evidence:** traces, checks, evals, provenance, and completion language. +- **Change control:** version, proposed changes, promotion gate, and rollback path. + +OpenAI's agent improvement loop similarly describes the harness as the full contract around the model, including instructions, tools, routing, output requirements, and validation checks. + +## The Orchestration Graph + +Represent a workflow as a graph when order, branching, recovery, or ownership affects correctness. + +Each node should declare: + +```yaml +id: verify_sources +purpose: Confirm that required sources are present and current +reads: [source_manifest] +writes: [source_verification] +authority: read_only +preconditions: [source_manifest_exists] +success: [all_required_sources_classified] +failure_route: request_missing_source +evidence: [source_verification] +``` + +Each edge should explain why the transition exists: + +```yaml +from: verify_sources +to: produce_candidate +condition: source_verification_passed +passes: [verified_source_refs] +on_failure: request_missing_source +``` + +The format is illustrative, not a Codex configuration syntax. The important property is that dependencies, state transfer, authority, evidence, and failure routing are inspectable. + +## The Ontology + +Use an ontology when several harnesses must agree on meanings that prose alone keeps blurring. + +A lightweight ontology can define: + +- entities such as `Mission`, `Source`, `Claim`, `Artifact`, `Check`, `Capability`, and `ChangeSet`; +- allowed states such as `proposed`, `verified`, `promoted`, `failed`, and `unknown`; +- relationships such as `derived_from`, `verified_by`, `requires`, `supersedes`, and `may_write`; +- invariants such as “every promoted claim has supporting evidence” or “a failed required check blocks promotion”; +- provenance fields that preserve where a claim or change came from. + +The ontology does not need a graph database. A small vocabulary, JSON Schema, typed data model, or validated Markdown convention may be enough. + +## Synthetic Example + +Consider a generalized documentation-maintenance system: + +```mermaid +flowchart LR + A["Discover changed product claims"] --> B["Classify source authority"] + B --> C["Draft candidate updates"] + C --> D["Validate links + public safety"] + D --> E["Review claim boundaries"] + E --> F["Propose versioned change"] + F --> G["Human promotion gate"] + G --> H["Publish"] + D -->|failure| I["Return structured findings"] + E -->|unsupported claim| I +``` + +The ontology distinguishes an observed product behavior from an official product claim. The graph prevents publication until required evidence and public-safety checks pass. The harness binds those definitions and transitions to tools, artifacts, and checks. + +## Safety Envelope + +- Use least privilege per node instead of granting every agent the union of all permissions. +- Treat tool output, webpages, documents, and retrieved text as untrusted evidence rather than instruction. +- Keep secrets and sensitive source content out of traces, eval fixtures, and public examples. +- Separate proposal authority from promotion or publication authority. +- Make failure and `unknown` first-class states; do not silently route around them. +- Preserve immutable baselines and enough provenance to explain and reverse a change. +- Do not let a semantic model convert uncertain claims into false certainty. + +## When Not To Use This + +Stay with a task contract or a single harness when the work is short, linear, low-risk, and easy to verify. Add a graph only when dependencies or recovery paths matter. Add an ontology only when shared meaning is a recurring source of error. + +## Verification + +The architecture is useful when another operator or agent can answer: + +- Which node owns each state transition? +- Which sources and claims are authoritative? +- What can write to each system? +- What evidence permits the next transition? +- Where do failures and unknowns go? +- Can a promoted change be traced and rolled back? + +## Sources + +Checked on 2026-08-20 against [Harness engineering](https://openai.com/index/harness-engineering/), [Agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop), [Subagents](https://learn.chatgpt.com/docs/agent-configuration/subagents), and [Agent approvals and security](https://learn.chatgpt.com/docs/agent-approvals-security). diff --git a/guides/parallel-projects-and-agent-teams.md b/guides/parallel-projects-and-agent-teams.md index 5a8d40a..05ad1c2 100644 --- a/guides/parallel-projects-and-agent-teams.md +++ b/guides/parallel-projects-and-agent-teams.md @@ -42,6 +42,19 @@ Topology is just the shape of the work system: who owns what, where state lives, The topology can be manual at first. Once it repeats, make it a prompt, template, skill, custom agent, MCP workflow, or automation. +## Topology Is Not The Whole Graph + +Agent topology describes who performs the work. An orchestration graph also describes: + +- prerequisites and transition conditions, +- state passed between nodes, +- allowed reads and writes, +- deterministic and judgment-based gates, +- retry, recovery, escalation, and stop paths, +- evidence required before the next transition. + +A hub-and-spoke team can run many different graphs. A single agent can also execute a multi-stage graph. Use [Graph And Ontology-Engineered Harnesses](graph-and-ontology-engineered-harnesses.md) when the control flow matters as much as the role split. + ## The Status Contract Every parallel stream needs a small status shape. Otherwise you get a pile of summaries and no control surface. diff --git a/guides/task-framing.md b/guides/task-framing.md index f357d3b..aa38797 100644 --- a/guides/task-framing.md +++ b/guides/task-framing.md @@ -33,6 +33,22 @@ First derive the task contract, source-of-truth map, delivery harness, and verif That small difference matters. The operator is not doing all the decomposition for Codex. The goal supplies enough context and discretion for Codex to propose the decomposition. +## The System-Shaped Frame + +When a mature harness already exists, the prompt can be smaller because it initializes known layers instead of restating them: + +```markdown +Objective: +System or harness: +Inputs: +Authority: +Evidence required: + +Use the promoted contracts. Propose any reusable improvement separately from this run. +``` + +This only works when the referenced system is real, discoverable, and versioned. Naming an imaginary harness is not a substitute for engineering one. + ## What To Include Name the real surface. If the task is about production, say production. If it is about a local branch, say that. If a doc is canonical, name it. diff --git a/guides/team-adoption.md b/guides/team-adoption.md index 653acc9..75e130b 100644 --- a/guides/team-adoption.md +++ b/guides/team-adoption.md @@ -60,6 +60,10 @@ The team changes how work enters, flows, gets reviewed, and gets measured becaus The team can hand over broad goals with clear success criteria, and Codex can derive the project harness, delivery harness, work slices, and checks. +### 7. Compounding system + +Recurring workflows emit privacy-safe evidence, failures become regression checks, and reviewed harness changes improve later runs without silently expanding authority. Versions, promotion decisions, and rollback paths remain inspectable. + ## What To Measure - time to first useful draft, @@ -68,6 +72,9 @@ The team can hand over broad goals with clear success criteria, and Codex can de - review findings caught before merge, - repeated questions eliminated by instructions or templates, - workflows converted into playbooks, +- recurring failures converted into regression checks, +- candidate changes rejected or rolled back before causing wider regression, +- performance of promoted versions against a stable baseline, - tasks that still require human-only judgment. ## Failure Modes @@ -77,6 +84,7 @@ The team can hand over broad goals with clear success criteria, and Codex can de - Giving broad tool access without review gates. - Publishing internal examples without rewriting them. - Training people on prompts but not on verification. +- Automating self-modification before evaluation validity, authority, and rollback are trustworthy. ## Verification diff --git a/guides/thinking-abstraction-level.md b/guides/thinking-abstraction-level.md index f01cd54..949f392 100644 --- a/guides/thinking-abstraction-level.md +++ b/guides/thinking-abstraction-level.md @@ -41,6 +41,22 @@ The automations broke after an upgrade. Diagnose the likely failure layer, make All three can be right. The useful judgment is knowing when Codex can take the higher-abstraction version and derive the work underneath it. +## Abstraction Is One Axis + +Abstraction level describes the request, not the maturity of the system receiving it. + +Three independent dimensions matter: + +| Dimension | Question | Progression | +| --- | --- | --- | +| Intent abstraction | How much implementation is specified by the operator? | edit → task → outcome → system objective | +| System embodiment | Where does the operating method live? | prompt → reusable workflow → harness → orchestration system | +| Improvement closure | What changes future runs? | nothing → captured lesson → eval → governed promotion | + +A broad prompt sent to a disposable agent is high in intent abstraction but low in system embodiment. A short request can initialize a mature system when its semantics, state, tools, checks, and authority boundaries already exist. + +See [From Prompts To Compounding Systems](from-prompts-to-compounding-systems.md) for the complete model. + ## The New Default For bigger work, do not start by writing the task contract yourself. @@ -98,7 +114,7 @@ Stay lower when: | Exact edit | File, line, change | The edit and maybe a quick check | | Framed task | Outcome, source, constraints | The steps and verification | | Mission | Goal, success criteria, context | Task contract, plan, implementation, checks | -| System | Direction, boundaries, learning loop | Harness topology, decomposition, delegation, delivery path | +| System objective | Direction, policies, boundaries, invariants | Harness selection, orchestration, execution, evidence, change proposal | A higher abstraction level does not mean less clarity. It means clarity moves from steps to outcomes, boundaries, and success criteria. @@ -142,3 +158,5 @@ You stop spending all your energy decomposing work into tiny tickets and start s - keeping verification honest. That is where Codex starts to feel less like autocomplete and more like a real operating layer for work. + +The next ceiling is not a still-higher prompt. It is an environment in which a prompt initializes a versioned harness, verified evidence can improve future behavior, and authority remains explicit throughout the loop. diff --git a/guides/verification-before-completion.md b/guides/verification-before-completion.md index dfc003d..914e79f 100644 --- a/guides/verification-before-completion.md +++ b/guides/verification-before-completion.md @@ -22,6 +22,19 @@ Use the smallest rung that proves the task: Higher is not always better. A live check can be overkill for a typo fix. A static check is too weak for a behavioral change. +## Verify The System As Well As The Output + +For a persistent harness or orchestration graph, task completion is only one claim. Also test: + +- required state transitions and failure routes; +- architecture, authority, and data-handling invariants; +- recovery after interruption or partial failure; +- candidate behavior against the promoted baseline; +- regression cases derived from prior failures; +- counter-metrics such as cost, latency, privilege growth, or data exposure. + +Self-generated tests and agreement among agents can provide evidence, but they are not independent acceptance. Keep deterministic gates outside the model when a hard requirement can be checked mechanically, and use appropriately independent or human review for consequential judgment. + ## Completion Language Be boringly precise here: @@ -38,6 +51,8 @@ Avoid unsupported claims like "should work" when a relevant check was available - Skipping verification because the change looks simple. - Claiming tests pass after a partial or failed run. - Forgetting to mention unavailable checks. +- Changing the harness and its evaluator together without preserving a comparable baseline. +- Treating a generated eval as independent proof of the generator. ## Verification diff --git a/guides/verified-improvement-loops.md b/guides/verified-improvement-loops.md new file mode 100644 index 0000000..2987c72 --- /dev/null +++ b/guides/verified-improvement-loops.md @@ -0,0 +1,120 @@ +# Verified Improvement Loops + +A workflow does not improve merely because it remembers something. It improves when evidence from one run produces a controlled, testable change to future behavior. + +That requires three distinct loops. + +## Three Loops + +| Loop | Purpose | Ends When | +| --- | --- | --- | +| Execution loop | Produce the requested result | the candidate output exists | +| Verification loop | Test the result against the claim | required checks pass, fail, or remain explicitly unknown | +| Evolution loop | Improve the system that produced the result | a reviewed change is promoted or rejected with evidence | + +Collapsing these loops creates predictable errors. The system starts changing itself before the original result is understood, treats self-review as independent proof, or promotes a local workaround into a global rule. + +## The Evolution Loop + +```mermaid +flowchart LR + A["Versioned harness"] --> B["Bounded runs"] + B --> C["Traces + artifacts"] + C --> D["Feedback + eval results"] + D --> E["Failure classification"] + E --> F["Candidate change"] + F --> G["Regression evaluation"] + G --> H{"Promotion gate"} + H -->|promote| I["New harness version"] + H -->|reject| J["Preserved evidence"] + I --> B +``` + +## Evidence Packet + +Keep enough structured evidence to diagnose the system without preserving unnecessary sensitive content: + +- harness and policy version; +- task class and synthetic or redacted case identifier; +- declared inputs and source categories; +- tool and routing events relevant to the outcome; +- produced artifacts and unresolved questions; +- deterministic check results; +- human or independent review findings; +- failure classification; +- proposed change and expected effect; +- candidate-versus-baseline evaluation result; +- promotion, rejection, or rollback decision. + +Here, a trace means observable events and artifacts made available by the harness. It does not imply access to private model reasoning. Store raw prompts, credentials, private documents, personal data, and unrestricted traces only when there is a justified private retention policy. Public examples should use synthetic evidence. + +## Diagnose Before Changing + +Classify failures before adjusting the system: + +| Failure Class | Likely Response | +| --- | --- | +| Missing contract | clarify an instruction, schema, invariant, or required output | +| Contract not followed | improve enforcement, control flow, or tool affordance | +| Missing capability | add or expose a narrowly scoped tool, skill, or source | +| Observability gap | capture the state needed to distinguish causes | +| Invalid eval | repair the scorer, fixture, or claim before optimizing | +| Environmental failure | improve isolation, retry policy, or source availability | +| One-off exception | preserve locally; do not generalize without repeated evidence | + +Changing prompt wording is only one possible response. The higher-leverage fix may be a validator, a state contract, a routing rule, a clearer source boundary, or a better recovery path. + +## Promotion Protocol + +1. Freeze the current harness and evaluation baseline. +2. Turn the observed failure into a reproducible case when possible. +3. Define the intended change and the failure it should address. +4. Run the candidate against the existing suite and the new case. +5. Check counter-metrics for regressions, cost, latency, or privilege growth. +6. Review the diff and evidence at the required authority level. +7. Promote a new version or reject the candidate without rewriting history. +8. Monitor later runs and retain a rollback path. + +A proposed change should not edit the active harness in place while it is being evaluated. Candidate and promoted state need distinct identities. + +## Evaluation Boundaries + +- A generated test can be useful, but it is not independent acceptance of the system that generated it. +- Agreement among agents can reveal consensus, but it does not establish correctness. +- A score is meaningful only for the claim, harness, data, budget, and environment actually tested. +- Hidden or independently controlled cases are useful when gaming or overfitting is plausible. +- Deterministic checks should own hard gates when the requirement can be expressed mechanically. +- Subjective quality still needs deliberate human or appropriately independent review. +- `FAIL`, `ERROR`, and `INCOMPLETE` should remain distinct states. + +## Automation Levels + +Increase automation only as evidence and reversibility improve: + +1. Capture findings for a human to interpret. +2. Propose a change set for review. +3. Implement a candidate in isolation and run evals. +4. Open a change for approval with evidence attached. +5. Auto-promote only narrow, reversible changes behind trusted gates. + +Do not use successful low-risk automation as blanket evidence that higher-impact changes can safely bypass review. + +## Threats To The Loop + +| Risk | Control | +| --- | --- | +| Feedback poisoning | authenticate sources, preserve provenance, require review for policy changes | +| Eval overfitting | use holdouts, counter-metrics, varied cases, and sample review | +| Privilege expansion | diff capabilities and permissions separately from behavior changes | +| Trace leakage | minimize collection, redact sensitive fields, separate public fixtures | +| Self-confirmation | add independent checks at consequential boundaries | +| Drift and pattern replication | run recurring invariant checks and retain a known-good baseline | +| Irreversible promotion | version changes, use staged rollout, and test rollback | + +## Verification + +An improvement loop is closed only when the evidence identifies what changed, why it should help, how it performed against the same contract, who or what authorized promotion, and how to reverse it. + +## Sources + +Checked on 2026-08-20 against [Agent improvement loop](https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop), [Harness engineering](https://openai.com/index/harness-engineering/), and [A shared playbook for trustworthy third-party evaluations](https://openai.com/index/trustworthy-third-party-evaluations-foundations/). diff --git a/resources/README.md b/resources/README.md index 89f8e97..2bdf51f 100644 --- a/resources/README.md +++ b/resources/README.md @@ -10,9 +10,14 @@ Small reusable pieces for making Codex work less like "please vibe at this" and - [Source Of Truth Map](source-of-truth-map.md) - [Workflow Audit Template](workflow-audit-template.md) - [Playground Prompts](playground-prompts.md) +- [Compounding System Canvas](compounding-system-canvas.md) +- [Harness Graph Template](harness-graph-template.md) +- [Improvement Loop Checklist](improvement-loop-checklist.md) ## How To Use Them Copy the smallest thing that helps. Do not turn every tiny task into a paperwork exercise. For bigger work, the move is usually: write the mission brief, then ask Codex to derive the task contract, source-of-truth map, delivery harness, and verification plan. + +Use the compounding-system resources only after a workflow repeats or several harnesses need shared state and meaning. A checklist is better than a graph when the work is still simple and linear. diff --git a/resources/catalog.json b/resources/catalog.json index 3ea2850..1e59f38 100644 --- a/resources/catalog.json +++ b/resources/catalog.json @@ -14,6 +14,24 @@ "path": "guides/thinking-abstraction-level.md", "status": "ready" }, + { + "title": "From Prompts To Compounding Systems", + "type": "guide", + "path": "guides/from-prompts-to-compounding-systems.md", + "status": "ready" + }, + { + "title": "Graph And Ontology-Engineered Harnesses", + "type": "guide", + "path": "guides/graph-and-ontology-engineered-harnesses.md", + "status": "ready" + }, + { + "title": "Verified Improvement Loops", + "type": "guide", + "path": "guides/verified-improvement-loops.md", + "status": "ready" + }, { "title": "Task Framing For Agents", "type": "guide", @@ -169,6 +187,24 @@ "type": "template", "path": "resources/playground-prompts.md", "status": "usable" + }, + { + "title": "Compounding System Canvas", + "type": "template", + "path": "resources/compounding-system-canvas.md", + "status": "usable" + }, + { + "title": "Harness Graph Template", + "type": "template", + "path": "resources/harness-graph-template.md", + "status": "usable" + }, + { + "title": "Improvement Loop Checklist", + "type": "checklist", + "path": "resources/improvement-loop-checklist.md", + "status": "usable" } ] } diff --git a/resources/codex-operating-checklist.md b/resources/codex-operating-checklist.md index 0fc6df2..aa82571 100644 --- a/resources/codex-operating-checklist.md +++ b/resources/codex-operating-checklist.md @@ -7,6 +7,7 @@ Use this before giving Codex a non-trivial mission. - The ask is at the right level: exact edit, task, outcome, or broad goal. - Success criteria are clearer than the step list. - Codex can derive the task contract when the path is not obvious. +- Intent abstraction is not being confused with system maturity or autonomy. ## Frame @@ -53,6 +54,14 @@ Use this before giving Codex a non-trivial mission. - Write boundaries do not overlap unless the parent owns integration. - Every stream has a status contract and proof path. +## System Design + +- The harness contract names instructions, tools, routing, state, outputs, and validation. +- Agent topology is distinguished from control flow and dependency edges. +- Shared entities, states, and relationships are defined only as deeply as recurring ambiguity requires. +- Failure, error, incomplete, and unknown states have explicit routes. +- Every write and transition has an authority and evidence boundary. + ## Verification - The primary check is named before completion. @@ -71,3 +80,8 @@ Use this before giving Codex a non-trivial mission. - Repeated lessons have a destination: instructions, template, script, skill, docs, or backlog. - One-off details are not promoted into durable rules. +- The execution, verification, and evolution loops are separate. +- Candidate changes are versioned and evaluated against a preserved baseline. +- Permission and data-access changes receive separate review. +- Promotion and rollback authority are explicit. +- Private evidence is generalized before becoming reusable guidance. diff --git a/resources/compounding-system-canvas.md b/resources/compounding-system-canvas.md new file mode 100644 index 0000000..33d856e --- /dev/null +++ b/resources/compounding-system-canvas.md @@ -0,0 +1,58 @@ +# Compounding System Canvas + +Use this for a recurring workflow that may deserve a persistent harness or orchestration system. Leave a field small or mark it `unknown` instead of inventing detail. + +```markdown +System objective: +Throughput or outcome unit: +System boundary: +Explicit non-goals: + +Intent and authority: +- Who can request a run? +- What may the system read? +- What may it write? +- Which transitions require approval? + +Semantic contract: +- Core entities: +- Important relationships: +- Allowed states: +- Invariants: +- Meaning of failure, error, incomplete, and unknown: + +Harness: +- Instructions and policies: +- Tools and capabilities: +- Routing rules: +- Required outputs: +- Validation checks: + +Orchestration graph: +- Nodes: +- Dependency and control-flow edges: +- State passed between nodes: +- Failure and recovery routes: +- Integration point: + +Evidence and observability: +- Required traces or events: +- Artifact provenance: +- Deterministic checks: +- Independent or human checks: +- Data that must not be collected: + +Improvement loop: +- Feedback sources: +- Regression suite: +- Candidate-change process: +- Promotion gate: +- Rollback path: +- Counter-metrics: + +Current constraint: +Smallest useful next improvement: +Evidence that would disconfirm it: +``` + +The canvas describes an engineered pattern, not a native Codex configuration file. Start with one workflow and expand only when the contracts are reused. diff --git a/resources/harness-graph-template.md b/resources/harness-graph-template.md new file mode 100644 index 0000000..99ae209 --- /dev/null +++ b/resources/harness-graph-template.md @@ -0,0 +1,59 @@ +# Harness Graph Template + +Use this template when a workflow has meaningful dependencies, branching, state transitions, or recovery paths. + +## Graph Contract + +```markdown +Graph name: +Objective: +Version: +Entry condition: +Completion condition: +Global invariants: +Prohibited state or data: +Promotion authority: +Rollback target: +``` + +## Node Contract + +Repeat for each node: + +```yaml +id: +purpose: +owner: +reads: [] +writes: [] +authority: +preconditions: [] +success_conditions: [] +evidence: [] +timeout_or_budget: +failure_route: +``` + +## Edge Contract + +Repeat for each edge: + +```yaml +from: +to: +condition: +passes: [] +on_failure: +``` + +## Review Questions + +- Does every write have one accountable owner? +- Does every transition rely on observable evidence? +- Can failures, errors, incomplete results, and unknowns remain visible? +- Are authority and data exposure narrower than the union of all nodes? +- Can the graph resume safely after interruption? +- Can a candidate graph be evaluated without replacing the promoted graph? +- Is a graph actually needed, or would a linear checklist be clearer? + +The YAML fragments are illustrative contracts, not built-in Codex syntax. diff --git a/resources/improvement-loop-checklist.md b/resources/improvement-loop-checklist.md new file mode 100644 index 0000000..802bcb9 --- /dev/null +++ b/resources/improvement-loop-checklist.md @@ -0,0 +1,48 @@ +# Improvement Loop Checklist + +Use this before allowing evidence from one run to change future system behavior. + +## Baseline + +- [ ] The current harness or workflow has a versioned identity. +- [ ] The intended outcome and evaluation claim are explicit. +- [ ] The current eval suite and known limitations are preserved. +- [ ] Required authority and privacy boundaries are documented. + +## Evidence + +- [ ] The failure or opportunity is supported by a trace, artifact, check, or review. +- [ ] Sensitive inputs are excluded, minimized, or protected under an explicit retention policy. +- [ ] Evidence provenance is retained. +- [ ] `FAIL`, `ERROR`, `INCOMPLETE`, and `unknown` are not collapsed into success. + +## Diagnosis + +- [ ] The finding is classified as contract, enforcement, capability, observability, eval, environment, or one-off failure. +- [ ] The proposed change addresses the evidenced cause rather than only its wording. +- [ ] A counter-hypothesis or disconfirming check has been considered. +- [ ] A private incident has been generalized before becoming reusable guidance. + +## Candidate + +- [ ] The candidate is isolated from the promoted harness. +- [ ] The change is attributable and reviewable. +- [ ] Existing regression cases and the new case are run comparably. +- [ ] Counter-metrics cover quality, cost, latency, privilege, and data exposure where relevant. +- [ ] Generated tests are not treated as independent acceptance. + +## Promotion + +- [ ] Required deterministic gates pass. +- [ ] Consequential subjective claims receive human or appropriately independent review. +- [ ] Permission, tool, and data-access changes are reviewed separately. +- [ ] Promotion authority is explicit. +- [ ] A rollback path is tested or credibly available. +- [ ] Rejected candidates and failed evidence remain visible. + +## After Promotion + +- [ ] Later runs are monitored for regression or drift. +- [ ] The change is reusable without repasting private context. +- [ ] The system records which version produced each outcome. +- [ ] The next review point is based on risk or evidence, not arbitrary automation. diff --git a/resources/playground-prompts.md b/resources/playground-prompts.md index 8026be1..fb65655 100644 --- a/resources/playground-prompts.md +++ b/resources/playground-prompts.md @@ -133,3 +133,26 @@ Include: - what was not verified - what remains ``` + +## Turn A Repeated Workflow Into A Compounding System + +```markdown +Recurring workflow: + + +Current failure or constraint: + + +Design the smallest system that can make later runs more reliable. Include: +- the harness contract, +- the minimum state and semantic model, +- the execution graph only if order, branching, or recovery matters, +- the evidence packet, +- one regression eval, +- candidate-versus-promoted version handling, +- promotion and rollback gates, +- counter-metrics, +- privacy, privilege, and feedback-poisoning risks. + +Distinguish native Codex capabilities from architecture built around them. Keep the design general and synthetic. Do not implement autonomous self-modification or broaden tool authority without separate approval. +``` diff --git a/scripts/validate_content.py b/scripts/validate_content.py index 706301d..12eb01d 100755 --- a/scripts/validate_content.py +++ b/scripts/validate_content.py @@ -47,8 +47,10 @@ ALLOWED_EXTERNAL_HOSTS = { "cookbook.openai.com", + "developers.openai.com", "learn.chatgpt.com", "metr.org", + "openai.com", "www.anthropic.com", "www.swebench.com", } diff --git a/tests/test_validate_content.py b/tests/test_validate_content.py index 4c1dd9b..afad6aa 100644 --- a/tests/test_validate_content.py +++ b/tests/test_validate_content.py @@ -26,6 +26,14 @@ def test_external_url_policy_is_allowlist_based(self) -> None: self.assertIsNone( validator.external_url_issue("https://learn.chatgpt.com/docs/models") ) + self.assertIsNone( + validator.external_url_issue( + "https://developers.openai.com/cookbook/examples/agents_sdk/agent_improvement_loop" + ) + ) + self.assertIsNone( + validator.external_url_issue("https://openai.com/index/harness-engineering/") + ) self.assertEqual( validator.external_url_issue("https://example.com/profile"), "uses a host outside the reviewed allowlist",