From b40e893ed4c7261a9791611f0f06f9f7a4509b91 Mon Sep 17 00:00:00 2001 From: Jarel Remick <3012014+jremick@users.noreply.github.com> Date: Thu, 20 Aug 2026 09:46:17 +1000 Subject: [PATCH] docs: add advanced harness work patterns --- README.md | 8 +- docs/advanced-work-patterns.md | 352 ++++++++++++++++++ docs/example-work-patterns.md | 2 + examples/README.md | 73 ++++ guides/README.md | 1 + guides/from-prompts-to-compounding-systems.md | 20 + ...graph-and-ontology-engineered-harnesses.md | 8 + resources/README.md | 2 + resources/catalog.json | 6 + resources/compounding-system-canvas.md | 37 ++ 10 files changed, 508 insertions(+), 1 deletion(-) create mode 100644 docs/advanced-work-patterns.md diff --git a/README.md b/README.md index 48b7132..9145cca 100644 --- a/README.md +++ b/README.md @@ -14,6 +14,8 @@ If you want to use Codex better today, start with [The Codexmaxxing Loop](guides If the same workflow or failure keeps returning, move into [From Prompts To Compounding Systems](guides/from-prompts-to-compounding-systems.md). That is where the guide gets into reusable harnesses, workflow graphs, shared vocabularies, evals, and controlled improvement. +If you are already designing systems around whole classes of work, use [Advanced Work Patterns](docs/advanced-work-patterns.md). It works through meta-harness factories, ontology-driven decision systems, graph-governed artifact systems, portfolio control planes, and harness evolution. + If you are trying to understand a current Codex feature—such as projects, scheduled tasks, skills, plugins, subagents, worktrees, Browser, or Computer Use—use the [complete guide index](guides/README.md). Product-specific pages are dated and link back to current official sources. ## The Shape Of It @@ -76,6 +78,7 @@ This repo is a mix of notes, patterns, templates, and examples for that. | create a file, interactive explanation, or hosted experience | [Artifacts, Sites, And Visualizations](guides/artifacts-sites-and-visualizations.md) | | design a large skill library without flooding context | [Capability Lifecycle And Prompt Visibility](guides/capability-lifecycle.md) | | turn repeated work into a system that can improve safely | [From Prompts To Compounding Systems](guides/from-prompts-to-compounding-systems.md), [Workflow Graphs, Shared Vocabulary, And Harnesses](guides/graph-and-ontology-engineered-harnesses.md), and [Verified Improvement Loops](guides/verified-improvement-loops.md) | +| design systems that create, coordinate, evaluate, or improve other harnesses | [Advanced Work Patterns](docs/advanced-work-patterns.md) and [Compounding System Canvas](resources/compounding-system-canvas.md) | The complete [guide index](guides/README.md), [copyable resources](resources/README.md), and [synthetic missions](examples/README.md) provide the rest. @@ -89,8 +92,11 @@ The complete [guide index](guides/README.md), [copyable resources](resources/REA - Diagnose a layered system failure with read-only evidence before changing anything. - Turn a repeated workflow into a reusable skill, checklist, or validator. - Turn a recurring failure into a regression eval and a reviewed workflow improvement. +- Use a shared ontology to keep claims, sources, decisions, and artifacts consistent across several workflows. +- Build a target-neutral meta-harness that generates candidates but cannot approve its own output. +- Coordinate a portfolio through explicit dependencies, evidence, budgets, and integration gates instead of an uncontrolled agent swarm. -These are expanded in [Example Work Patterns](docs/example-work-patterns.md). The examples are synthetic and do not describe a specific person, repository, organization, or environment. +These are expanded in [Example Work Patterns](docs/example-work-patterns.md) and [Advanced Work Patterns](docs/advanced-work-patterns.md). The examples are synthetic and do not describe a specific person, repository, organization, or environment. ## Status And Support diff --git a/docs/advanced-work-patterns.md b/docs/advanced-work-patterns.md new file mode 100644 index 0000000..76845d8 --- /dev/null +++ b/docs/advanced-work-patterns.md @@ -0,0 +1,352 @@ +# Advanced Work Patterns + +These patterns start where a normal task or single harness stops. The object being designed is the operating system around a class of work: how harnesses are created, how state moves, what important concepts mean, how evidence is judged, and how later versions are adopted. + +All examples are synthetic. They describe architectures that can be built with many agent tools, including Codex, but they are not native Codex configuration formats or claims about built-in product behavior. + +Use [From Prompts To Compounding Systems](../guides/from-prompts-to-compounding-systems.md) for the progression, [Workflow Graphs, Shared Vocabulary, And Harnesses](../guides/graph-and-ontology-engineered-harnesses.md) for the building blocks, and [Verified Improvement Loops](../guides/verified-improvement-loops.md) for change control. + +## What Makes This Higher Abstraction? + +The request is no longer only "do this work." It asks the agent to help design the system that will repeatedly decide how the work should be done and checked. + +| Scope | Operates On | Produces | +| --- | --- | --- | +| Task | one input and outcome | one checked result | +| Harness | a repeatable class of tasks | reusable instructions, tools, state, outputs, and checks | +| Meta-harness | harness definitions and assurance rules | versioned candidate harnesses and evidence packages | +| Control plane | several harnesses, missions, and dependencies | routing, coordination, state, budgets, and integration evidence | +| Improvement system | current and proposed harness versions | evaluated adoption, rejection, or rollback decisions | + +A meta-harness is not automatically better than a normal harness. It earns its complexity when several harnesses need a common definition, generation path, assurance policy, or target adapter. + +Some terms recur throughout the examples: + +| Term | Plain meaning | +| --- | --- | +| Assurance policy | the evidence, checks, and approvals required before a result can be trusted or adopted | +| Target binding | the explicit translation from a general harness definition into one agent environment | +| Promotion gate | the decision point that adopts or rejects a candidate version | +| Ontology | a shared model of important entities, relationships, states, and rules | +| Control plane | the part that chooses harnesses, tracks shared state, and integrates their results | + +The pieces have different jobs. The ontology says what the system's important things and states mean. The graph says how work and state are allowed to move. The harness binds those contracts to real instructions, tools, outputs, and checks. A meta-harness can then create or govern those harnesses without collapsing all four layers into one prompt. + +## 1. Meta-Harness Factory + +### The Mission + +```markdown +Design a system that can turn a project objective, operating constraints, and assurance policy into a versioned candidate harness for a chosen agent environment. + +Keep the core definition independent of any one agent product. Put product-specific files, tools, and configuration in an explicit target binding. + +The system should validate the definition, compile the candidate in isolation, run deterministic and independent checks, preserve unknowns and failures, and produce a reviewable evidence package. It must not promote its own output merely because generation completed. +``` + +This is meta-harness work because the system operates on harnesses rather than directly performing the final domain task. + +### System Shape + +```mermaid +flowchart LR + A["Intent + constraints"] --> B["Target-neutral definition"] + P["Assurance policy"] --> B + B --> C["Validate semantics"] + C --> D["Apply target binding"] + D --> E["Compile candidate harness"] + E --> F["Run in isolated fixture"] + F --> G["Independent evaluator"] + G --> H["Evidence package"] + H --> I{"Promotion gate"} + I -->|approve| J["Versioned harness registry"] + I -->|reject or unknown| K["Preserved findings"] +``` + +### Shared Model + +Useful entities might include: + +- `HarnessDefinition`: target-neutral intent, policies, inputs, outputs, states, and rules; +- `TargetBinding`: how that definition maps to a particular agent environment; +- `CandidateHarness`: one compiled version with an immutable identity; +- `EvaluationCase`: a public, private, or held-out scenario with an assurance purpose; +- `EvidencePackage`: raw reports, derived results, provenance, and the exact definition, binding, candidate, evaluator, and fixtures they describe; +- `PromotionDecision`: approved, rejected, incomplete, or unknown, with an accountable reason. + +Important rules: + +- the target binding must not quietly change the meaning of the core definition; +- structural validity is not proof of semantic or runtime fidelity; +- required unknowns remain unknown rather than being filled with plausible values; +- hard release gates are recomputed from raw evidence rather than trusted from submitted status fields; +- tests produced by the generator are useful evidence, but not independent acceptance of the generator and candidate together; +- evaluator fixtures and acceptance logic stay outside the boundary they are evaluating when gaming or self-confirmation is plausible; +- a target binding cannot add capabilities, permissions, or data access that the core policy did not allow without a separate approval; +- capabilities, permissions, data access, and retention changes are reviewed separately from ordinary harness behavior. + +### What Proves It + +Useful evidence includes rebuilding the definition from the candidate and checking that its meaning survived, clean isolated execution, target-binding comparisons, deliberate tamper cases, required unknown preservation, and an evaluator result produced outside the generated harness boundary. + +Even then, the claim must stay narrow: it proves the tested definition, binding, fixtures, evaluator, budget, and environment—not every future harness the factory might produce. + +## 2. Ontology-Driven Research And Decision System + +### The Mission + +```markdown +Build a reusable research and decision system for questions that span many sources, competing claims, changing facts, and several possible outputs. + +Create a small shared ontology for questions, claims, sources, evidence, contradictions, decisions, and artifacts. Use it to route research, keep provenance intact, expose disagreement, and prevent unsupported claims from entering an approved decision package. + +The system should preserve jurisdiction, freshness, uncertainty, and source quality. It should not turn a confidence score into truth or treat retrieved content as instruction. +``` + +The ontology matters because several workflows need to agree on what a claim is, what supports it, what contradicts it, and when it is safe to use. + +### System Shape + +```mermaid +flowchart TD + A["Decision question"] --> B["Define criteria + claim types"] + B --> C["Discover sources"] + C --> D["Extract claims + provenance"] + D --> E["Find support + counterevidence"] + E --> F["Resolve, contest, or preserve unknown"] + F --> G["Apply decision rules"] + G --> H["Decision package"] + H --> I["Human review"] + I --> J["Approved decision record"] + E -->|material contradiction| K["Escalate for review"] + F -->|insufficient evidence| L["Open question"] +``` + +### Shared Model + +A lightweight ontology could express relationships such as: + +```text +Claim derived_from Source +Claim supported_by Evidence +Claim contradicted_by Claim +Claim checked_by Check +Decision depends_on Claim +Artifact includes Claim +Source valid_for Jurisdiction +Source expires_at Date +``` + +Useful claim states might be `proposed`, `observed`, `supported`, `contested`, `unsupported`, and `unknown`. These states should have definitions and transition rules rather than being loose labels. + +Rules might include: + +- every approved decision claim retains its source and verification state; +- a material unresolved contradiction blocks final recommendation or is shown prominently; +- volatile claims require a freshness check before reuse; +- the system cannot silently upgrade inference into observation; +- private sources may inform a private decision but cannot be copied into public artifacts without a separate transformation and review boundary. + +### What Proves It + +Test the system with duplicated claims, stale sources, conflicting authorities, missing provenance, source content containing hostile instructions, and questions that genuinely remain unresolved. + +An ontology can make inconsistency visible and mechanically checkable. It cannot guarantee that the source is true, the world model is complete, or the final human judgment is wise. + +## 3. Graph-Governed Artifact System + +### The Mission + +```markdown +Design a system that can turn mixed source material into the right reviewable artifact: a document, spreadsheet, presentation, PDF, diagram, dashboard, or interactive explanation. + +Create one canonical content model so facts, calculations, claims, assets, and citations do not drift across formats. Use a workflow graph to select the output, route specialist production, run deterministic checks, inspect the rendered result, and stop at an approval gate before sharing or deployment. + +The system must preserve private-source boundaries and report what each verification layer can and cannot prove. +``` + +This is more than file generation. It is an artifact control system with shared content, format-specific harnesses, and separate publication authority. + +### System Shape + +```mermaid +flowchart TD + A["Sources + intended use"] --> B["Classify sensitivity + audience"] + B --> C["Canonical content model"] + C --> D{"Choose artifact route"} + D --> E["Document harness"] + D --> F["Spreadsheet harness"] + D --> G["Presentation harness"] + D --> H["Visualization harness"] + E --> I["Structural checks"] + F --> I + G --> I + H --> I + I --> J["Rendered or interactive review"] + J --> K{"Share or deploy approval"} + K -->|approve| L["External publication"] + K -->|revise| C +``` + +### Shared Model + +Useful entities include `Source`, `Fact`, `Claim`, `Calculation`, `Narrative`, `Asset`, `Audience`, `Artifact`, `Review`, and `Publication`. + +Rules might include: + +- every displayed number is derived from a named calculation or source value; +- all artifact variants reference the same canonical claim identity; +- private or restricted material has an allowed-output policy; +- structure, calculations, links, and required fields use deterministic checks where possible; +- layout, clarity, accessibility, and visual quality receive rendered or human review; +- permission to create an artifact does not grant permission to send, share, host, or deploy it. + +### What Proves It + +Run a cross-format consistency case, a calculation-change case, a missing-citation case, a private-source redaction case, and a rendered-layout review. Confirm that changing a canonical fact either updates every dependent artifact or blocks release as stale. + +A valid file is not necessarily a good artifact. A screenshot is not proof that calculations are correct. Each claim needs the check that can actually support it. + +## 4. Portfolio Control Plane + +### The Mission + +```markdown +Design a control plane that can take a portfolio objective, identify distinct missions, choose the right harness for each, manage dependencies and budgets, and integrate evidence into one accountable result. + +Keep execution isolated by ownership and write boundary. Model blockers, incomplete work, stale evidence, and integration decisions as durable state. Do not create an all-powerful agent or treat the number of active agents as progress. +``` + +The control plane owns routing and integration. Individual harnesses still own their domain work and verification. + +### System Shape + +```mermaid +flowchart TD + A["Portfolio objective"] --> B["Mission decomposition"] + B --> C["Dependency + risk graph"] + C --> D["Harness selection"] + D --> E["Isolated workstream A"] + D --> F["Isolated workstream B"] + D --> G["Isolated workstream C"] + E --> H["Evidence intake"] + F --> H + G --> H + H --> I["Integration checks"] + I --> J{"Portfolio decision"} + J -->|complete| K["Integrated outcome"] + J -->|blocked| L["Escalation or replanning"] + J -->|incomplete| M["Visible open state"] +``` + +### Shared Model + +Useful entities include `Objective`, `Mission`, `Workstream`, `Harness`, `Dependency`, `Owner`, `Budget`, `Artifact`, `Evidence`, `Decision`, and `Blocker`. + +Rules might include: + +- every writable surface has one accountable owner at a time; +- a dependency is satisfied, explicitly waived, or visibly blocking; +- child completion does not imply integration completion; +- evidence has freshness and provenance, not just a completion label; +- budgets and stop conditions are enforced at the workstream boundary; +- the control plane may propose replanning but cannot broaden permissions without approval. + +### What Proves It + +Test overlapping write requests, a failed dependency, a timed-out workstream, contradictory evidence from two harnesses, and a child that reports success without the required artifact. Confirm that the portfolio stays incomplete until integration evidence exists. + +This architecture helps with coordination. It does not make independent workstreams correct, eliminate the need for domain review, or justify parallelism when the work is actually serial. + +## 5. Harness Evolution System + +### The Mission + +```markdown +Design a system that improves a recurring harness from evidence without editing the active version in place. + +Capture privacy-safe run evidence, classify failures, produce the smallest candidate change, and evaluate it against the current baseline, the observed regression case, and held-out cases. Review quality, cost, latency, permissions, and data exposure separately. + +Adopt only a versioned, reversible change that passes the required gates. Preserve rejected candidates, failures, errors, incomplete results, and unknowns. +``` + +This system operates on the lifecycle of a harness rather than only on its executions. + +### System Shape + +```mermaid +flowchart LR + A["Current harness version"] --> B["Bounded runs"] + B --> C["Privacy-safe evidence"] + C --> D["Failure classification"] + D --> E["Isolated candidate version"] + E --> F["Baseline + regression + holdout evals"] + F --> G["Independent or human review"] + G --> H{"Promotion decision"} + H -->|adopt| I["Staged rollout"] + H -->|reject| J["Preserved candidate + evidence"] + I --> K["Drift monitoring"] + K -->|regression| L["Rollback"] + K -->|healthy| A +``` + +### Shared Model + +Useful entities include `HarnessVersion`, `Run`, `Finding`, `FailureClass`, `EvaluationCase`, `CandidateChange`, `EvaluationResult`, `PromotionDecision`, and `RollbackEvent`. + +Rules might include: + +- current and candidate harnesses have distinct immutable identities; +- the observed failure is reproduced before a claimed fix is credited; +- the candidate faces the existing suite as well as the new case; +- feedback keeps its source and provenance, and untrusted feedback cannot directly change policy; +- evaluation uses fresh isolated or immutable snapshots when shared writable state could contaminate results; +- generated tests are not counted as independent acceptance; +- capability, permission, retention, and cost changes are reviewed separately from output quality; +- `FAIL`, `ERROR`, `INCOMPLETE`, and `unknown` remain distinct; +- rejection and rollback do not rewrite the earlier evidence. + +### What Proves It + +Run a candidate that fixes the new case but breaks a baseline case, one that improves quality by using broader permissions, one that lowers cost while losing source fidelity, and one whose evaluator cannot reach a required source. The system should reject or preserve uncertainty rather than optimizing a single score. + +This is controlled improvement, not autonomous self-modification. A closed loop still needs trustworthy evidence, appropriate independence, explicit adoption authority, and a credible rollback path. + +## The Common Shape + +```mermaid +flowchart LR + A["System objective"] --> B["Shared model + policy"] + B --> C["Meta-harness or control plane"] + C --> D["Versioned executable harnesses"] + D --> E["Bounded runs"] + E --> F["Evidence + evaluation"] + F --> G["Adopt, reject, or roll back"] + G --> C +``` + +The pattern underneath is simple even when the system is not: + +- define the important concepts and rules; +- make work and state transitions inspectable; +- keep model judgment separate from deterministic authority; +- preserve provenance, failure, and uncertainty; +- evaluate proposed changes outside the boundary they can manipulate; +- require explicit authority for broader permissions, external publication, and adoption. + +## Start Smaller Than This + +Do not begin with a meta-harness because the phrase sounds advanced. Begin with one useful harness. Add a workflow graph when branching or recovery keeps causing mistakes. Add a shared ontology when several workflows repeatedly disagree about the same concepts. Add a meta-harness only when multiple harnesses need a common definition, generator, adapter, or assurance policy. + +The goal is not to maximize machinery. It is to make increasingly capable work reliable, understandable, and reversible. + +## Claim Limits + +- These are reference architectures, not complete implementations or security boundaries. +- A graph does not prove correct routing, and an ontology does not prove true knowledge. +- Schema validation proves structure, not semantic fidelity, runtime behavior, or real-world quality. +- An evaluator is independent only to the extent that its data, code, runtime, incentives, and acceptance logic are outside the evaluated boundary. +- A workflow wrapper, temporary directory, or fixture is not an operating-system security sandbox. Use an appropriately controlled process, container, virtual machine, or host when the threat model requires it. +- "Privacy-safe evidence" still requires a retention policy and data minimization. Do not collect raw prompts, secrets, private documents, account data, or hidden evaluator material merely because traces are useful. +- Deterministic checks should own mechanically expressible hard gates; subjective quality still needs deliberate review. +- Codex and other agent tools can supply useful primitives, but the state model, graph, ontology, evaluator, promotion policy, and rollback system remain architecture that must be designed and tested. diff --git a/docs/example-work-patterns.md b/docs/example-work-patterns.md index b924729..e3b7bfe 100644 --- a/docs/example-work-patterns.md +++ b/docs/example-work-patterns.md @@ -2,6 +2,8 @@ The examples on this page are synthetic. Their names, systems, paths, and evidence are fictional and do not describe a specific person, repository, organization, or environment. +For systems that create, coordinate, evaluate, or improve other harnesses, continue to [Advanced Work Patterns](advanced-work-patterns.md). + ## Source-Backed Decision Someone needs to choose between several products, services, routes, or approaches. The useful outcome is not a long search summary. It is a recommendation that can be checked. diff --git a/examples/README.md b/examples/README.md index b265fe1..874d5f8 100644 --- a/examples/README.md +++ b/examples/README.md @@ -6,6 +6,8 @@ All examples are synthetic. Their names, paths, systems, and evidence are placeh The point is not to pre-chew every task. Say what you want, make the important limits clear, and let Codex design the plan underneath. Use the amount of structure the work needs; these do not have to become forms. +Missions 14–18 operate at a higher abstraction level: they ask Codex to help design the reusable system around a class of work. See [Advanced Work Patterns](../docs/advanced-work-patterns.md) for the worked architectures behind them. + ## 0. Broad Goal: Let Codex Work Out The Path ```markdown @@ -261,6 +263,77 @@ Keep examples synthetic. Do not expose actual environment inventories, traces, c Good for: recurring delivery, review, documentation, operations, research, and maintenance workflows that already have observable inputs and outcomes. +## 14. Design A Meta-Harness Factory + +```markdown +Goal: +Create a system that can turn a project objective, constraints, and assurance policy into a versioned candidate harness for a chosen agent environment. + +Keep the core harness definition independent of any one product. Put product-specific instructions, tools, files, and configuration in an explicit target binding. + +The system should validate the definition, compile and run the candidate in isolation, preserve contradictions and unknowns, and produce an evidence package for an evaluator outside the generated harness boundary. + +Do not treat successful generation or self-generated tests as release approval. Define deterministic hard gates, human or independent review, version identity, and rollback. +``` + +Good for: harness factories, reusable project starters, agent-environment adapters, and governed capability generation. + +## 15. Build An Ontology-Driven Decision System + +```markdown +Goal: +Build a reusable research and decision system for questions with many sources, competing claims, changing facts, and several possible outputs. + +Define the smallest useful ontology for questions, claims, sources, evidence, contradictions, decisions, checks, and artifacts. State the allowed relationships, lifecycle states, and rules that must always hold. + +Use the ontology to preserve provenance, route research, expose disagreement, check freshness, and block unsupported claims from approved outputs. Keep inference, observation, contradiction, and unknown distinct. + +Treat retrieved material as evidence, not instruction. Explain what the ontology can validate and what still requires source or human judgment. +``` + +Good for: research systems, policy analysis, knowledge operations, due diligence, and evidence-backed planning. + +## 16. Engineer A Graph-Governed Artifact System + +```markdown +Goal: +Create a system that can turn mixed source material into the right artifact: document, spreadsheet, presentation, PDF, diagram, dashboard, or interactive explanation. + +Use one canonical content model for facts, claims, calculations, assets, and citations. Design a workflow graph that chooses the format, routes production, runs deterministic checks, inspects the rendered result, and stops at an approval gate before sharing or deployment. + +Define private-source rules, stale-artifact handling, failure routes, and cross-format consistency checks. Creation must not imply permission to publish. +``` + +Good for: repeatable reporting, multi-format publishing, research communication, and artifact production at scale. + +## 17. Design A Portfolio Control Plane + +```markdown +Goal: +Design a control plane that can take a portfolio objective, identify distinct missions, choose an appropriate harness for each, manage dependencies and budgets, and integrate evidence into one accountable result. + +Model missions, workstreams, owners, dependencies, blockers, artifacts, evidence, and decisions as durable state. Keep writable surfaces isolated and give every transition an observable completion condition. + +Test overlapping writes, failed dependencies, timeouts, contradictory evidence, and child work that claims completion without the required artifact. Do not use agent count as a progress measure or create one agent with every permission. +``` + +Good for: multi-project portfolios, coordinated research and delivery, complex programs, and several specialized harnesses working toward one outcome. + +## 18. Build A Harness Evolution System + +```markdown +Goal: +Create a controlled system for improving a recurring harness from evidence without editing the active version in place. + +Capture privacy-safe evidence, classify failures, create the smallest isolated candidate change, and compare it with the current version on the existing suite, the observed regression case, and held-out cases. + +Review quality, source fidelity, cost, latency, permissions, and data exposure separately. Preserve failure, error, incomplete, unknown, rejection, and rollback states. + +Adopt only a versioned, reversible change that passes its real approval gates. Do not call this autonomous self-improvement or treat the system's own tests as independent proof. +``` + +Good for: eval-driven maintenance, recurring workflows, skill evolution, routing improvement, and safely compounding operating systems. + ## The Common Shape ```mermaid diff --git a/guides/README.md b/guides/README.md index 151d990..af98d91 100644 --- a/guides/README.md +++ b/guides/README.md @@ -14,6 +14,7 @@ These are practical operating notes, not a course or a framework that every task - [From Prompts To Compounding Systems](from-prompts-to-compounding-systems.md) - [Workflow Graphs, Shared Vocabulary, And Harnesses](graph-and-ontology-engineered-harnesses.md) - [Verified Improvement Loops](verified-improvement-loops.md) +- [Advanced Work Patterns](../docs/advanced-work-patterns.md) ## Build A Reusable Codex Setup diff --git a/guides/from-prompts-to-compounding-systems.md b/guides/from-prompts-to-compounding-systems.md index f20a434..d079cec 100644 --- a/guides/from-prompts-to-compounding-systems.md +++ b/guides/from-prompts-to-compounding-systems.md @@ -51,6 +51,26 @@ flowchart LR The model is only one part. The surrounding setup determines what it can see and change, how work moves, what counts as success, and whether a useful lesson changes later runs. +## Harnesses And Meta-Harnesses + +A harness surrounds a repeatable capability. A meta-harness operates on harness definitions: it can help design, validate, compile, compare, package, or govern other harnesses. + +```mermaid +flowchart LR + A["Intent + assurance policy"] --> B["Meta-harness"] + B --> C["Target-neutral harness definition"] + C --> D["Agent-environment binding"] + D --> E["Candidate harness"] + E --> F["Evaluation outside candidate boundary"] + F --> G{"Approve, reject, or preserve unknown"} +``` + +This is a change in what the system operates on, not a mandatory maturity stage. One carefully built harness is better than a harness factory with no real repeated need. A meta-harness becomes useful when several harnesses need a common contract, target adapters, assurance policy, evidence format, or promotion path. + +The same boundaries still apply: hard gates should be deterministic where possible, unknowns must remain visible, the evaluator must not simply trust the candidate's own status, and broader permissions require separate approval. + +See [Advanced Work Patterns](../docs/advanced-work-patterns.md) for worked examples of meta-harness factories, ontology-driven decision systems, graph-governed artifact systems, portfolio control planes, and harness evolution systems. + ## What Compounding Means A workflow compounds only when useful improvements survive the run and improve later work. diff --git a/guides/graph-and-ontology-engineered-harnesses.md b/guides/graph-and-ontology-engineered-harnesses.md index 9efb196..6c982d8 100644 --- a/guides/graph-and-ontology-engineered-harnesses.md +++ b/guides/graph-and-ontology-engineered-harnesses.md @@ -93,6 +93,14 @@ flowchart LR The shared vocabulary distinguishes an observed product behavior from an official product claim. The graph prevents publication until required evidence and public-safety checks pass. The harness connects those definitions and transitions to tools, artifacts, and checks. +## From One Harness To A System Of Harnesses + +The same pieces can operate one level higher. A meta-harness can accept a target-neutral definition, apply a product-specific binding, produce a candidate harness, and send it through an evaluator and promotion gate. A portfolio control plane can route several missions to different harnesses while preserving their dependencies, permissions, evidence, and integration state. + +That extra layer should not erase the contracts underneath it. Each generated or selected harness still needs its own purpose, inputs, states, permissions, outputs, verification, and failure routes. The outer system also needs rules for version identity, evaluator independence, adoption, and rollback. + +See [Advanced Work Patterns](../docs/advanced-work-patterns.md) for concrete system shapes across research, artifact production, portfolio coordination, harness generation, and controlled improvement. + ## Safety Envelope - Use least privilege per node instead of granting every agent the union of all permissions. diff --git a/resources/README.md b/resources/README.md index 6e9fe6c..613afbf 100644 --- a/resources/README.md +++ b/resources/README.md @@ -21,3 +21,5 @@ Copy the smallest thing that helps. Do not turn every tiny task into a paperwork For bigger work, write a short mission brief and ask Codex to work out the plan, relevant sources, checks, and stop conditions. Use the compounding-system resources only after a workflow repeats or several parts need shared state and language. A checklist is better than a graph when the work is still simple and linear. + +If the system will create, coordinate, evaluate, or improve other harnesses, read [Advanced Work Patterns](../docs/advanced-work-patterns.md) before using the meta-harness extension in the canvas. diff --git a/resources/catalog.json b/resources/catalog.json index 0e93ee8..63adfc3 100644 --- a/resources/catalog.json +++ b/resources/catalog.json @@ -128,6 +128,12 @@ "path": "docs/example-work-patterns.md", "status": "usable" }, + { + "title": "Advanced Work Patterns", + "type": "examples", + "path": "docs/advanced-work-patterns.md", + "status": "usable" + }, { "title": "Field Synthesis", "type": "research", diff --git a/resources/compounding-system-canvas.md b/resources/compounding-system-canvas.md index f04c4ee..da634bf 100644 --- a/resources/compounding-system-canvas.md +++ b/resources/compounding-system-canvas.md @@ -68,4 +68,41 @@ Smallest useful next improvement: Evidence that would disconfirm it: ``` +## Meta-Harness Extension + +Use this only when the system will create, adapt, evaluate, or govern more than one harness. + +```markdown +What the meta-harness operates on: +Target-neutral harness definition: +Supported target bindings: +Meaning that every binding must preserve: + +Generation boundary: +- What the generator may decide: +- What must be declared by policy or a human: +- What remains prohibited or unknown: + +Evaluation boundary: +- Deterministic structural and policy checks: +- Runtime or end-to-end checks: +- Independent or human review: +- Private or held-out cases: +- Evidence the candidate cannot rewrite: + +Version and registry model: +- Definition identity: +- Binding identity: +- Candidate harness identity: +- Evaluator and fixture identity: +- Evidence package identity: + +Promotion: +- Required gates: +- Who can approve adoption: +- Rollout boundary: +- Rollback target: +- Reasons to preserve rejection or incomplete state: +``` + The canvas describes an engineered pattern, not a native Codex configuration file. Start with one workflow and expand only when the pieces are genuinely reused.