From 51818f190aa77ae356c83a6af08a453e96ac562d Mon Sep 17 00:00:00 2001 From: bigboateng Date: Sat, 8 Aug 2026 22:42:43 +0100 Subject: [PATCH] docs: clarify AgentTask judgment boundary --- README.md | 23 +++-- docs/README.md | 3 + docs/primitives/README.md | 14 +-- docs/primitives/agent-task.md | 151 +++++++++++++++++++++++------ docs/tutorials/code-review.md | 4 +- evals/results/latest.json | 4 +- examples/release-checklist/main.ts | 2 +- scripts/readme.test.mjs | 100 +++++++++++++++++-- sdk/python/README.md | 22 +++-- sdk/rust/README.md | 22 +++-- sdk/yield/README.md | 22 +++-- 11 files changed, 287 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index 11e3d9a..6cd55fc 100644 --- a/README.md +++ b/README.md @@ -40,6 +40,9 @@ programs. The canonical workflow stays inside your repository beside the code and dependencies it uses. Generated `SKILL.md` files only help coding agents discover it. +Write the workflow in code. Use `AgentTask` only where a bounded step needs +coding-agent judgment, then continue with structured data in normal code. + Verified with Cursor, Codex, and Claude Code. Registry-backed project paths are available for 73 more coding agents. @@ -81,7 +84,9 @@ A release skill often starts as prose: > Run the tests. Review the release. Stop if the review finds a critical issue. > Ask me before publishing. Publish the package, then verify the registry. -Yield makes the order and stopping rules executable: +Yield makes the order and stopping rules executable. The coding agent reviews +what the deterministic check may miss; the program still owns the gate, +approval, publish, and verification steps: @@ -101,7 +106,7 @@ defineSkill((ctx) => { // coding agent's response at runtime before this workflow can continue. const review = ctx.agentTask( "review-release", - "Review this release. Report critical findings and a short summary.", + "Review this release for correctness problems that the test command may miss. Report critical findings and a short summary.", { stdout: tests.stdout, stderr: tests.stderr }, { type: "object", @@ -267,13 +272,13 @@ The agent follows the generated adapter, runs the canonical workflow in If replay produces a different operation, the run fails instead of silently forking. Every side effect crosses one of these primitives: -| Primitive | Purpose | -| --------------------- | ------------------------------------------------------ | -| `runCommand` | Execute a command and record its exit code and output. | -| `agentTask` | Ask the coding agent for schema-valid JSON. | -| `askUser` | Request an explicit human decision. | -| `require` | Bind a required claim to recorded evidence. | -| `blocked` / `refused` | Stop honestly when work cannot or must not continue. | +| Primitive | Purpose | +| --------------------- | ----------------------------------------------------------------------- | +| `runCommand` | Execute a command and record its exit code and output. | +| `agentTask` | Delegate one bounded judgment; an optional schema validates the result. | +| `askUser` | Request an explicit human decision. | +| `require` | Bind a required claim to recorded evidence. | +| `blocked` / `refused` | Stop honestly when work cannot or must not continue. | See the [primitive guides](https://github.com/operatorstack/yield/blob/main/docs/primitives/README.md) and [runtime reference](https://github.com/operatorstack/yield/blob/main/docs/reference/cli.md) for the full contract. diff --git a/docs/README.md b/docs/README.md index 0fc6a98..877942f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,9 @@ code. The coding agent still investigates, reviews, edits, and explains. Your program decides which operation comes next, what evidence must exist, and when the run is finished. +When one step needs judgment, call the coding agent with `AgentTask`, receive +structured data, then continue in normal code. + A **skill workflow** is a portable, executable process that combines agent skills with deterministic code, state, and verification. The canonical workflow is the source you edit. Generated adapters let coding agents discover diff --git a/docs/primitives/README.md b/docs/primitives/README.md index 43e4839..0a0e0b6 100644 --- a/docs/primitives/README.md +++ b/docs/primitives/README.md @@ -2,13 +2,13 @@ Yield has a deliberately small API. Each primitive has one clear owner. -| Primitive | What it does | Who performs it | -| ------------------------------ | ---------------------------------------------------- | --------------------- | -| [`RunCommand`](run-command.md) | Runs a command and records its real output | `yskill` | -| [`AgentTask`](agent-task.md) | Requests model judgment with an optional JSON schema | coding agent | -| [`AskUser`](ask-user.md) | Pauses for a human answer | coding agent and user | -| [`Require`](require.md) | Prevents completion unless a claim passes | skill workflow | -| [Outcomes](outcomes.md) | Completes, blocks, or refuses with a recorded reason | skill workflow | +| Primitive | What it does | Who performs it | +| ------------------------------ | ----------------------------------------------------------------------- | --------------------- | +| [`RunCommand`](run-command.md) | Runs a command and records its real output | `yskill` | +| [`AgentTask`](agent-task.md) | Delegates one bounded judgment; an optional schema validates the result | coding agent | +| [`AskUser`](ask-user.md) | Pauses for a human answer | coding agent and user | +| [`Require`](require.md) | Prevents completion unless a claim passes | skill workflow | +| [Outcomes](outcomes.md) | Completes, blocks, or refuses with a recorded reason | skill workflow | Ordinary language features provide the rest. Use `if` for choices, `for` or `while` for bounded retries, functions for reusable flows, and your language's diff --git a/docs/primitives/agent-task.md b/docs/primitives/agent-task.md index 5a9d44b..24503b6 100644 --- a/docs/primitives/agent-task.md +++ b/docs/primitives/agent-task.md @@ -1,37 +1,130 @@ -# `AgentTask`: keep judgment with the model - -Use `AgentTask` for work that needs interpretation: reviewing a diff, -diagnosing a failure, comparing designs, extracting a policy, or proposing a -fix. - -```ts -type Diagnosis = { cause: string; confidence: number } - -const diagnosis = ctx.agentTask( - "diagnose", - "Find the most likely cause of this test failure.", - { stdout: test.stdout, stderr: test.stderr }, - { - type: "object", - required: ["cause", "confidence"], - properties: { - cause: { type: "string" }, - confidence: { type: "number" }, +# AgentTask + +Use the coding agent as a typed judgment step inside your workflow. + +The agent interprets. With a JSON Schema, Yield checks the returned shape. Your +program still owns what happens next. + +```text +workflow code + ↓ +AgentTask + ↓ +coding-agent judgment + ↓ +JSON result, schema-validated when requested + ↓ +workflow code continues +``` + +Use `AgentTask` when one step needs interpretation: review what a check may +have missed, diagnose captured output, compare designs against criteria, or +extract structured information from repository material. + +## One workflow, three roles + +| Role | Primitive | Responsibility | +| ------------------ | -------------------------- | ------------------------------------------------------ | +| deterministic work | `RunCommand` and `Require` | run commands and enforce requirements | +| agent judgment | `AgentTask` | interpret a bounded request and return structured data | +| human authority | `AskUser` | make a decision before a protected effect | + +Not every workflow needs all three. Yield lets them work together without +asking the coding agent to rediscover order, retry limits, or completion rules +on every run. + +## Example: check, review, approve, publish + +This tested workflow runs a deterministic check, asks the coding agent to +review what that check may miss, requires a safe review result, then asks a +person before publishing. + + + +```typescript +import { defineSkill } from "@operatorstack/yield" + +type Review = { critical: number; summary: string } + +defineSkill((ctx) => { + // Yield runs commands itself and records their output and exit status. + const tests = ctx.runCommand("test", "echo tests-ok", 300) + + // A failed requirement stops the workflow and keeps its evidence. + ctx.require(tests.exit_code === 0, "the test command succeeds", tests) + + // Review gives TypeScript its compile-time type. The JSON schema checks the + // coding agent's response at runtime before this workflow can continue. + const review = ctx.agentTask( + "review-release", + "Review this release for correctness problems that the test command may miss. Report critical findings and a short summary.", + { stdout: tests.stdout, stderr: tests.stderr }, + { + type: "object", + required: ["critical", "summary"], + properties: { + critical: { type: "integer", minimum: 0 }, + summary: { type: "string", minLength: 1 }, + }, }, - }, -) + ) + ctx.require(review.critical === 0, "the review has no critical findings", review) + + // Yield emits these fixed choices. A supported host may show native controls; + // otherwise the coding agent asks through its normal interface. + const approval = ctx.askUser("approve-publish", "Publish this package?", [ + { value: "yes", label: "Publish" }, + { value: "no", label: "Stop" }, + ]) + if (approval !== "yes") ctx.refused("the operator declined publication") + + // Publishing cannot start before approval. Verification is a separate step, + // so completion requires evidence that the registry contains the release. + const publish = ctx.runCommand("publish", "echo publish-ok", 600) + ctx.require(publish.exit_code === 0, "the publish command succeeds", publish) + + const registry = ctx.runCommand("verify-registry", "echo registry-ok", 300) + ctx.require(registry.exit_code === 0, "the registry contains the release", registry) + + return { published: true, summary: review.summary } +}) ``` -The arguments are: + + +## Give the task the evidence it needs + +The third argument is explicit workflow context. Pass the result that the +judgment depends on, such as command output, a diff summary, or a previous +structured result. Explicit context is easier to understand, test, and replay. + +Yield sends the instruction and explicit context in the request. It does not +promise access to a complete conversation, the repository, or any hidden host +context. A coding agent may have additional working capabilities in its host, +but those capabilities are host-dependent. + +Cursor, Codex, and Claude Code are verified integrations. The host still owns +the UI and working capabilities available to the agent. + +## What schema validation proves + +When you provide a JSON Schema, Yield validates the returned JSON before the +workflow continues. It proves that required fields and declared structural +constraints are present. It does not prove that the analysis is correct, files +were inspected, or the requested work happened. + +Use `RunCommand` for machine-observed output, `Require` to control +continuation, and `AskUser` for human authority. Another `AgentTask` can offer +another judgment, but it is still model judgment. + +## Test the surrounding workflow -1. a stable operation ID; -2. the instruction; -3. optional structured context; -4. an optional JSON Schema for the response. +In production, `AgentTask` waits for the coding agent. In a test, +`yskill test` reads a deterministic fixture response instead. The same +surrounding workflow logic still runs, including commands and requirements. -The Yield CLI validates the response schema before accepting it. Schema-valid -does not mean true; use `RunCommand`, human approval, or another explicit check -when the workflow needs stronger evidence. +See [testing fixtures](../testing-fixtures.md) for +`fixtures/responses.json` and test-only effects. ## Common mistake diff --git a/docs/tutorials/code-review.md b/docs/tutorials/code-review.md index a04526f..67b8085 100644 --- a/docs/tutorials/code-review.md +++ b/docs/tutorials/code-review.md @@ -13,8 +13,8 @@ ctx.require(check.exit_code === 0, "typecheck passes", check) const review = ctx.agentTask( "review", - "Review the branch for correctness, security, and data-loss risks.", - undefined, + "Review the branch for correctness, security, and data-loss risks that the typecheck may miss.", + { exit_code: check.exit_code, stdout: check.stdout, stderr: check.stderr }, reviewSchema, ) diff --git a/evals/results/latest.json b/evals/results/latest.json index d774971..9085650 100644 --- a/evals/results/latest.json +++ b/evals/results/latest.json @@ -1,8 +1,8 @@ { "schema_version": 2, "methodology_version": "1.1", - "generated_at": "2026-08-08T20:17:45.135Z", - "source_digest": "a5e563abbe348dc60c3e8643beb2778acec8382d880c1cc08504158f8a5cb244", + "generated_at": "2026-08-08T21:45:59.521Z", + "source_digest": "f064d931095318d00a02d4b407f8b784dd6a0a24d190f2260f4158e50187515a", "status": "passed", "workflow_conformance": { "passed": 40, diff --git a/examples/release-checklist/main.ts b/examples/release-checklist/main.ts index 68f1614..ff47db0 100644 --- a/examples/release-checklist/main.ts +++ b/examples/release-checklist/main.ts @@ -16,7 +16,7 @@ defineSkill((ctx) => { // coding agent's response at runtime before this workflow can continue. const review = ctx.agentTask( "review-release", - "Review this release. Report critical findings and a short summary.", + "Review this release for correctness problems that the test command may miss. Report critical findings and a short summary.", { stdout: tests.stdout, stderr: tests.stderr }, { type: "object", diff --git a/scripts/readme.test.mjs b/scripts/readme.test.mjs index 62c1a44..f9dbf2a 100644 --- a/scripts/readme.test.mjs +++ b/scripts/readme.test.mjs @@ -10,23 +10,105 @@ async function text(path) { } test("README release example matches the tested TypeScript source", async () => { - const [readme, source] = await Promise.all([ + const [readme, agentTaskDocs, source] = await Promise.all([ text("README.md"), + text("docs/primitives/agent-task.md"), text("examples/release-checklist/main.ts"), ]) - const readmeMatch = readme.match( - /\s*```typescript\n([\s\S]*?)\n```\s*/, - ) - assert.ok(readmeMatch, "README release example markers are missing") + const example = + /\s*```typescript\n([\s\S]*?)\n```\s*/ const sourceMatch = source.match(/\/\/ README_EXAMPLE_START\n([\s\S]*?)\n\/\/ README_EXAMPLE_END/) assert.ok(sourceMatch, "TypeScript release example markers are missing") - const readmeProgram = readmeMatch[1] - .replace(/^import \{ defineSkill \} from "@operatorstack\/yield";?\n+/, "") - .trim() - assert.equal(readmeProgram, sourceMatch[1].trim()) + for (const [name, document] of [ + ["README", readme], + ["AgentTask guide", agentTaskDocs], + ]) { + const match = document.match(example) + assert.ok(match, `${name} release example markers are missing`) + const program = match[1] + .replace(/^import \{ defineSkill \} from "@operatorstack\/yield";?\n+/, "") + .trim() + assert.equal(program, sourceMatch[1].trim()) + } +}) + +test("AgentTask documentation preserves the typed judgment boundary", async () => { + const [ + agentTask, + primitiveIndex, + docsIndex, + tutorial, + readme, + pythonReadme, + goReadme, + rustReadme, + ] = await Promise.all([ + text("docs/primitives/agent-task.md"), + text("docs/primitives/README.md"), + text("docs/README.md"), + text("docs/tutorials/code-review.md"), + text("README.md"), + text("sdk/python/README.md"), + text("sdk/yield/README.md"), + text("sdk/rust/README.md"), + ]) + + const normalized = (document) => document.replace(/\s+/g, " ") + const normalizedAgentTask = normalized(agentTask) + assert.match( + normalizedAgentTask, + /Use the coding agent as a typed judgment step inside your workflow\./, + ) + assert.match( + normalizedAgentTask, + /The agent interprets\. With a JSON Schema, Yield checks the returned shape\./, + ) + assert.match( + normalizedAgentTask, + /It does not prove that the analysis is correct, files were inspected, or the requested work happened\./, + ) + assert.match( + normalizedAgentTask, + /does not promise access to a complete conversation, the repository, or any hidden host context/, + ) + assert.match(normalizedAgentTask, /Cursor, Codex, and Claude Code are verified integrations/) + assert.match(normalizedAgentTask, /`yskill test` reads a deterministic fixture response instead/) + assert.match( + primitiveIndex, + /Delegates one bounded judgment; an optional schema validates the result/, + ) + assert.match( + normalized(docsIndex), + /call the coding agent with `AgentTask`, receive structured data, then continue in normal code/, + ) + assert.match( + tutorial, + /\{ exit_code: check\.exit_code, stdout: check\.stdout, stderr: check\.stderr \}/, + ) + assert.match( + normalized(readme), + /Use `AgentTask` only where a bounded step needs coding-agent judgment/, + ) + + for (const [name, document] of [ + ["Python", pythonReadme], + ["Go", goReadme], + ["Rust", rustReadme], + ]) { + const normalizedDocument = normalized(document) + assert.match( + normalizedDocument, + /Delegate one bounded judgment; an optional schema validates the result/, + ) + assert.match(normalizedDocument, /Host workspace and conversation access are host-dependent\./) + assert.match( + normalizedDocument, + /With its schema, Yield checks the returned JSON shape before the workflow continues/, + ) + } }) test("Python README example matches the tested environment doctor", async () => { diff --git a/sdk/python/README.md b/sdk/python/README.md index 384cd89..69bd2f7 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -205,13 +205,21 @@ for each required agent or user response. Replay must produce the same operation sequence. Yield reports divergence instead of giving a recorded response to a different operation. -| Python primitive | Purpose | -| --------------------------------- | ------------------------------------------------------ | -| `ctx.run_command()` | Execute a command and record its exit code and output. | -| `ctx.agent_task()` | Ask the coding agent for schema-valid JSON. | -| `ctx.ask_user()` | Request an explicit human decision. | -| `ctx.require()` | Bind a required claim to recorded evidence. | -| `ctx.blocked()` / `ctx.refused()` | Stop honestly when work cannot or must not continue. | +## Where AgentTask fits + +`ctx.agent_task()` delegates one bounded judgment to the coding agent. Pass +important evidence explicitly, as this workflow passes the probe result. With +its schema, Yield checks the returned JSON shape before the workflow continues; +it does not prove the diagnosis is correct. Host workspace and conversation +access are host-dependent. + +| Python primitive | Purpose | +| --------------------------------- | ----------------------------------------------------------------------- | +| `ctx.run_command()` | Execute a command and record its exit code and output. | +| `ctx.agent_task()` | Delegate one bounded judgment; an optional schema validates the result. | +| `ctx.ask_user()` | Request an explicit human decision. | +| `ctx.require()` | Bind a required claim to recorded evidence. | +| `ctx.blocked()` / `ctx.refused()` | Stop honestly when work cannot or must not continue. | See the [primitive guides](https://yield.operatorstack.systems/docs/primitives/) and [CLI reference](https://github.com/operatorstack/yield/blob/main/docs/reference/cli.md) diff --git a/sdk/rust/README.md b/sdk/rust/README.md index 4b94a8f..0739bc7 100644 --- a/sdk/rust/README.md +++ b/sdk/rust/README.md @@ -215,13 +215,21 @@ each required agent or user response. Replay must produce the same operation sequence. Yield reports divergence instead of giving a recorded response to a different operation. -| Rust primitive | Purpose | -| --------------------------------- | ------------------------------------------------------ | -| `ctx.run_command()` | Execute a command and record its exit code and output. | -| `ctx.agent_task()` | Ask the coding agent for schema-valid JSON. | -| `ctx.ask_user()` | Request an explicit human decision. | -| `ctx.require()` | Bind a required claim to recorded evidence. | -| `ctx.blocked()` / `ctx.refused()` | Stop honestly when work cannot or must not continue. | +## Where AgentTask fits + +`ctx.agent_task()` delegates one bounded judgment to the coding agent. Pass +important evidence explicitly, as this workflow passes its dry-run output. +With its schema, Yield checks the returned JSON shape before the workflow +continues; it does not prove the summary is correct. Host workspace and +conversation access are host-dependent. + +| Rust primitive | Purpose | +| --------------------------------- | ----------------------------------------------------------------------- | +| `ctx.run_command()` | Execute a command and record its exit code and output. | +| `ctx.agent_task()` | Delegate one bounded judgment; an optional schema validates the result. | +| `ctx.ask_user()` | Request an explicit human decision. | +| `ctx.require()` | Bind a required claim to recorded evidence. | +| `ctx.blocked()` / `ctx.refused()` | Stop honestly when work cannot or must not continue. | See the [primitive guides](https://yield.operatorstack.systems/docs/primitives/) and [CLI reference](https://github.com/operatorstack/yield/blob/main/docs/reference/cli.md) diff --git a/sdk/yield/README.md b/sdk/yield/README.md index 36b8ecd..64e82b4 100644 --- a/sdk/yield/README.md +++ b/sdk/yield/README.md @@ -284,13 +284,21 @@ each required agent response. Replay must produce the same operation sequence. Yield reports divergence instead of giving a recorded response to a different operation. -| Go primitive | Purpose | -| --------------------------------- | ------------------------------------------------------ | -| `ctx.RunCommand()` | Execute a command and record its exit code and output. | -| `ctx.AgentTask()` | Ask the coding agent for schema-valid JSON. | -| `ctx.AskUser()` | Request an explicit human decision. | -| `ctx.Require()` | Bind a required claim to recorded evidence. | -| `ctx.Blocked()` / `ctx.Refused()` | Stop honestly when work cannot or must not continue. | +## Where AgentTask fits + +`ctx.AgentTask()` delegates one bounded judgment to the coding agent. Pass +important evidence explicitly, as this workflow passes its earlier structured +results. With its schema, Yield checks the returned JSON shape before the +workflow continues; it does not prove the diagnosis is correct. Host workspace +and conversation access are host-dependent. + +| Go primitive | Purpose | +| --------------------------------- | ----------------------------------------------------------------------- | +| `ctx.RunCommand()` | Execute a command and record its exit code and output. | +| `ctx.AgentTask()` | Delegate one bounded judgment; an optional schema validates the result. | +| `ctx.AskUser()` | Request an explicit human decision. | +| `ctx.Require()` | Bind a required claim to recorded evidence. | +| `ctx.Blocked()` / `ctx.Refused()` | Stop honestly when work cannot or must not continue. | See the [Go reference](https://pkg.go.dev/github.com/operatorstack/yield/sdk/yield), [primitive guides](https://yield.operatorstack.systems/docs/primitives/), and