Qualify 8.1.3 with hardened release evals

.agents/plans/03-assurance-hardening/overview.md

@@ -29,9 +29,13 @@ and publication cannot rederive a committed decision from retained inputs.
 8. Rederive qualification during release.
 9. Enforce main-tag provenance and publication deadlines.
 10. Run full qualification and publish the corrective release.
+11. Bind explicit named acceptance cases from the original host request to plan
+    approval.
+12. Stop release execution when durable evidence makes qualification
+    unreachable.
 
 Each numbered unit lands independently. A failed or inconclusive unit blocks the
-next. No release is allowed before units one through nine are green.
+next. No release is allowed before every applicable unit is green.
 
 ## Verification
 

CHANGELOG.md

@@ -2,6 +2,32 @@
 
 One short entry per release, written for users deciding whether to upgrade.
 
+## [8.1.3] - 2026-08-28
+
+Release claims now come from retained evidence instead of trusted summaries.
+
+- Declared gate assertions must be satisfied before final review and completed
+  closure. New named evidence writes JUnit to the plan-bound
+  `.flow/results.xml`; execution cannot substitute another path. Reports are read once
+  from a stable, bounded workspace file without following symlinks.
+- Eval failures identify the evaluator, provider, host, or persistence boundary
+  that produced them. The release matrix uses repository-owned policy with enough
+  attempts to measure its 90% and 100% thresholds.
+- Exact-artifact canary results are derived from one OpenCode session lineage.
+  Release qualification seals all attempts, transcripts, provenance, canary,
+  artifact, and grader source into an immutable bundle that CI independently
+  reopens and regrades.
+- Publication requires the tag and packed artifact to match the release evidence.
+  npm and GitHub operations are bounded and idempotent, refuse conflicting bytes,
+  and recover through an exact draft without destructive asset replacement.
+- **Session v5 schema:** unchanged. Runtime commands and tools are unchanged.
+
+Install or update:
+
+```bash
+opencode plugin opencode-plugin-flow@8.1.3 --global --force
+```
+
 ## [8.1.2] - 2026-08-26
 
 Long Grok eval turns no longer fail at Bun's implicit five-minute fetch cutoff.

README.md

@@ -40,7 +40,7 @@ expensive, and it is overhead when it is not.
 Install the exact npm release through OpenCode:
 
 ```bash
-opencode plugin opencode-plugin-flow@8.1.2 --global --force
+opencode plugin opencode-plugin-flow@8.1.3 --global --force
 ```
 
 Omit `--global` for project scope. Version pins are exact and never update on
@@ -51,7 +51,7 @@ The equivalent manual project configuration is:
 ```json
 {
   "$schema": "https://opencode.ai/config.json",
-  "plugin": ["opencode-plugin-flow@8.1.2"]
+  "plugin": ["opencode-plugin-flow@8.1.3"]
 }
 ```
 

biome.json

@@ -8,7 +8,8 @@
 			"!bun.lock",
 			"!evals/results",
 			"!evals/canary",
-			"!evals/decisions"
+			"!evals/decisions",
+			"!evals/qualification/bundles"
 		]
 	},
 	"formatter": {

docs/adr/0012-named-results-over-exit-codes.md

@@ -4,8 +4,9 @@ Date: 2026-07-28
 
 ## Status
 
-Accepted. Closes the limitation [ADR 0011](0011-declared-external-evidence.md)
-recorded rather than fixed.
+Accepted. Amended on 2026-08-28 after the release matrix exposed unusable
+immutable report declarations. Closes the limitation
+[ADR 0011](0011-declared-external-evidence.md) recorded rather than fixed.
 
 ## Context
 
@@ -32,47 +33,42 @@ command — all terminate in that one number.
 requires the field; an empty list is the common and correct answer for a credential,
 a device, or a setting, and keeps the exit-code rule.
 
-**Satisfy it only from a report the command wrote.** `flow_validation_start` takes
-`resultsPath`, the repository-relative JUnit XML the command produces. After the
-command, the runtime reads that file and records what it said about each declared
-name — `passed`, `failed`, `skipped`, or `absent` — as `observedAssertions`.
+**Bind the report at planning time.** A named evidence command writes
+`.flow/results.xml`. Flow rejects a new named plan whose command lacks that exact
+path. The `.flow` namespace is already outside the source fingerprint, so writing
+evidence cannot make its own validation source-stale.
 
-**Read the names from the plan, never from the caller.** The caller supplies only
-where its command writes a report, because only it knows that. Which names to look
-for comes from the approved plan. This is the same split
-[SLSA provenance](https://slsa.dev/spec/v1.2/build-provenance) draws between
-`externalParameters`, which are under external control and must be verified, and
-fields the build platform populates, which need no verification because the platform
-is trusted.
+**Satisfy it only from a report the command wrote.** `flow_validation_start` uses
+the approved path when the caller omits it and rejects a different caller path.
+After the command, the runtime reads that file and records what it said about each
+declared name as `passed`, `failed`, `skipped`, or `absent`.
+
+**Read the names and path from the approved command, never from the execution
+caller.** The optional execution-time path remains only for approved Session v5
+plans written before this amendment.
 
 **Require a change during the command window.** Before the exact command, Flow
 snapshots any report's digest and mtime. Afterward, it accepts only a changed report
 modified after arming and no later than command observation.
 
-**Fail closed, identically, on every absence.** No `resultsPath`, unsafe, unreadable,
-unstable, or oversized bytes, malformed UTF-8, or unparseable XML: each records every
-declared name as `absent`, because nothing observed those cases.
-
-**Count it in the metric too.** `completionHonesty` compares declared names against
-recorded outcomes, so the release number does not depend on the veto it measures.
+**Fail closed at both boundaries.** A new named plan with no safe managed path is
+rejected before approval. For legacy plans, an omitted, unsafe, unreadable, stale,
+unstable, oversized, malformed, or unparseable report records every declared name
+as `absent`.
 
 ## Simplicity boundary
 
-One field per entry, two per observation, one optional tool parameter, and a JUnit
-reader. No new tool, no new lifecycle state, no test-runner integration: Flow does not
-run the command, does not choose the reporter, and does not know which runner produced
-the file. It reads four attributes out of one element.
+One assertion field, two observation fields, one compatibility parameter, and a
+JUnit reader. No new persisted field, tool, lifecycle state, or test-runner
+integration. Flow binds one managed convention and reads four attributes.
 
-Regex over XML, which is normally wrong, is right here: one bounded, well-specified
-element, and a parser dependency to read four attributes is a worse trade than a
-pattern with a suite pinning it. JUnit XML because every runner can reach it: native
-in bun (`--reporter=junit`), vitest, and pytest, one adapter away in jest, go, and
-cargo. Not emitted unprompted everywhere, so one reader covers Flow's repositories
-without adopting a format nobody writes.
+The bounded JUnit reader extracts only the attributes needed for declared names.
+Common test runners can produce the format without Flow learning runner-specific
+output text.
 
-Only `externalEvidence` names cases. `plan.gate` does not, and should not: a gate is a
-whole-suite claim, and naming a handful of cases inside it would make it look narrower
-than it is.
+Gate and extra evidence use the same record. A whole-suite exit claim uses
+`assertions: []`. A gate may also bind an exact acceptance case when that case is
+part of the whole-suite command; the command still remains the canonical broad gate.
 
 ## Consequences
 
@@ -87,12 +83,6 @@ refusing an empty list would make the honest answer unavailable. What the field
 is that the empty list is a visible line in the plan the user approves, and the
 reviewer is given the entries.
 
-This is the third field of this shape, and that is the finding. Each closed a real
-measured failure, and each added one more thing a plan must declare and a reader must
-check. The next move is not a fourth: it is to collapse the gate, the environment, and
-the named results into one evidence record with one satisfaction rule, and to stop the
-declarer from being the party that benefits from a weak declaration.
-
 A repository whose runner emits no machine-readable report can declare no assertions
 and keeps exactly the behavior it had before this ADR.
 
@@ -110,10 +100,12 @@ cannot see a case that was never written. Naming is what makes it checkable.
 failure these ADRs record: the party that benefits from a weak claim writing it after
 seeing the result. The names are fixed in the approved plan, before there is a report.
 
-**Requiring `resultsPath` whenever assertions are declared.** It would turn a silent
-`absent` into an upfront refusal, and it also forbids arming the command once to see
-what its report contains. The `absent` outcome already refuses everything a missing
-path would have.
+**Letting new plans defer `resultsPath` until execution.** This was the original
+decision. Campaign `2026-08-28T11-10-09-556Z.v2` disproved it. OpenAI repeatedly
+approved a JUnit command and later supplied an incompatible or absolute path. The
+runtime failed closed, but the immutable plan could not recover. New plans therefore
+bind the managed path in the command before approval; approved legacy plans retain
+the old fallback.
 
 **Failing the observation itself when a declared case did not pass.** The observation
 is a true record of what the command did, and the gate rules read its exit code too.

docs/maintainer-contract.md

@@ -13,9 +13,9 @@ labelled so there.
 The public surface — tools, commands, guides, agents, and the Session v5 shape —
 stays frozen while those guarantees are measured; additive optional fields are
 allowed, and a removal or rename waits for a major announced one release ahead.
-A new required-at-save plan declaration is a major. Today's declaration is
-`evidence`, with `scope`, `platform`, and `assertions` on each entry. Do not
-add another evidence field to close a measured cheat.
+A new required-at-save plan declaration is a major. `evidence` entries own
+`scope`, `platform`, and `assertions`; named commands bind `.flow/results.xml`.
+Do not add another evidence field to close a measured cheat.
 [Release qualification](release-qualification.md) owns the thresholds and cadence.
 
 Flow is a serial durable workflow plugin, not a general orchestration framework.

docs/release-qualification.md

@@ -32,13 +32,17 @@ once, by one person, from one model's output.
 A new scenario needs an explicit release-policy decision. Any required canonical
 case missing from the report fails qualification.
 
-A non-product attempt never shrinks the required sample. Provider or host failure,
-or an unallowed ask, leaves an evidence gap. Evaluator failure is `NOT VERIFIED`;
-persistence failure stops without a finalized report. Re-run only external gaps.
+A non-product attempt never shrinks the required sample. The frozen plan retains
+one environment reserve per provider and case. A retryable provider or host
+failure activates that exact reserve; the failed attempt remains evidence. A
+second external failure or an unallowed ask leaves a gap. Product and evaluator
+failures never activate reserves. Evaluator failure is `NOT VERIFIED`;
+persistence failure stops without a finalized report.
 
 Repository code owns the ordered release catalog. Persisted `catalog.json` is only a
-witness and must match it exactly. The two-provider grid contains 76 cells; ordinary,
-narrowed, or merged summary reports cannot qualify.
+witness and must match it exactly. The two-provider grid has 76 primary cells and
+16 predeclared environment reserves; ordinary, narrowed, dynamically extended, or
+merged summary reports cannot qualify.
 
 Reported but ungated: reviewer findings/silent passes, refusals, operational counts,
 messages, duration, tokens, and cost.

docs/troubleshooting.md

@@ -5,7 +5,7 @@
 Rerun OpenCode's exact-version npm plugin command:
 
 ```bash
-opencode plugin opencode-plugin-flow@8.1.2 --global --force
+opencode plugin opencode-plugin-flow@8.1.3 --global --force
 ```
 
 Or confirm that the relevant `opencode.json` contains the exact npm plugin
@@ -14,7 +14,7 @@ entry:
 ```json
 {
   "$schema": "https://opencode.ai/config.json",
-  "plugin": ["opencode-plugin-flow@8.1.2"]
+  "plugin": ["opencode-plugin-flow@8.1.3"]
 }
 ```
 

docs/validation-and-review.md

@@ -55,8 +55,8 @@ caller declaration made at planning time.
 `savePlan` requires `evidence`: one gate plus optional extra observations. Every
 entry names its command, `platform`, and `assertions`. One satisfaction rule
 applies to all entries: an eligible exact-command observation on the declared
-platform with every declared case `passed`. Case outcomes come from a JUnit
-report named by `resultsPath` that changes during the observed command window
+platform with every declared case `passed`. Named commands write changed JUnit to
+`.flow/results.xml`; legacy plans may supply `resultsPath`
 ([ADR 0012](adr/0012-named-results-over-exit-codes.md)). Final review and
 `completed` closure refuse any unsatisfied entry. The gate also requires broad
 scope and remains subject to command vetoes. Feature reviews are not vetoed, so a

evals/README.md

@@ -49,8 +49,10 @@ to skip the copy and rely on environment credentials only.
 Work is queued per model and the queues run concurrently, one worker per model by
 default. Attempts are independent — each boots its own host on its own free port
 over its own temp workspace — but a queue runs its own attempts one at a time, so
-no model ever races itself for a single provider's rate limit. Release mode now
-schedules 76 cells across two provider queues. Lines print as attempts finish, so
+no model ever races itself for a single provider's rate limit. Release mode has
+76 primary targets and one predeclared environment reserve per provider and case,
+for a bounded maximum of 92 attempts. Only a retained retryable host or provider
+failure activates its same-stratum reserve; product failures never do. Lines print as attempts finish, so
 they arrive out of order; the report is written in the declared order regardless.
 `--concurrency 1` restores the sequential run, which is easier to read when you
 are debugging a single failure, and four workers is the ceiling however many

evals/analysis.ts

@@ -1,6 +1,10 @@
 import { canonicalJson } from "./canonical-json.js";
 import type { ValidatedCaseCatalog } from "./catalog.js";
 import { samePackedArtifact } from "./provenance.js";
+import {
+	deriveReleaseRunState,
+	minimumPassingCount,
+} from "./release-progress.js";
 import type {
 	ArtifactIdentity,
 	EvaluatorIdentity,
@@ -449,9 +453,7 @@ function decisionReason(
 function activeCells(report: ValidatedReport) {
 	const activated = new Set(report.completion.activatedReserveCellIds);
 	return report.plan.cells.filter(
-		(cell) =>
-			cell.schedule === "primary" ||
-			(cell.schedule === "replacement-reserve" && activated.has(cell.cellId)),
+		(cell) => cell.schedule === "primary" || activated.has(cell.cellId),
 	);
 }
 
@@ -491,6 +493,34 @@ export function deriveReleaseDecision(input: {
 			`Campaign stopped after a ${report.completion.cause} failure.`,
 		);
 	}
+	if (
+		report.completion.status === "stopped" &&
+		report.completion.cause === "product"
+	) {
+		const state = deriveReleaseRunState({
+			plan: report.plan,
+			catalog,
+			attempts: report.attempts,
+		});
+		if (
+			state.kind === "stop" &&
+			state.cause === "product" &&
+			state.reason === "pass-rate-unreachable"
+		)
+			decisionReason(
+				reasons,
+				"hard",
+				"below-pass-rate",
+				"The retained attempt prefix made the release threshold unreachable.",
+			);
+		else if (!(state.kind === "stop" && state.cause === "product"))
+			decisionReason(
+				reasons,
+				"hard",
+				"campaign-integrity-failure",
+				"Product stop cause does not match the retained attempt prefix.",
+			);
+	}
 	if (
 		promotionArtifact &&
 		!samePackedArtifact(expected.artifact, promotionArtifact)
@@ -648,6 +678,14 @@ export function deriveReleaseDecision(input: {
 					continue;
 				}
 				if (attempt.outcome.kind !== "product") {
+					if (
+						attempt.outcome.kind === "failure" &&
+						(attempt.outcome.origin === "host" ||
+							attempt.outcome.origin === "provider") &&
+						attempt.outcome.retryable
+					) {
+						continue;
+					}
 					decisionReason(
 						reasons,
 						"gap",
@@ -682,8 +720,8 @@ export function deriveReleaseDecision(input: {
 				);
 			} else if (
 				policy.minPassRate !== null &&
-				passRate !== null &&
-				passRate < policy.minPassRate
+				passed <
+					minimumPassingCount(policy.minScoredAttempts, policy.minPassRate)
 			) {
 				decisionReason(
 					reasons,

evals/canary/8.1.3.json

@@ -0,0 +1 @@
+{"actors":[{"actualModel":{"kind":"observed","value":{"family":"grok-4.6","gateway":null,"model":"grok-4.6","revision":null,"routeProvider":"xai"}},"requestedModel":{"family":"grok-4.6","gateway":null,"model":"grok-4.6","revision":null,"routeProvider":"xai"},"role":"manager","sessionIds":["id_37a719e4f85a9c12"]},{"actualModel":{"kind":"observed","value":{"family":"grok-4.6","gateway":null,"model":"grok-4.6","revision":null,"routeProvider":"xai"}},"requestedModel":{"family":"grok-4.6","gateway":null,"model":"grok-4.6","revision":null,"routeProvider":"xai"},"role":"reviewer","sessionIds":["id_9e28a0dd75dc3869"]}],"artifact":{"packageVersion":"8.1.3","sourceCommit":"c6711c39f2eb9017fb120ac1241d4fcb32985a19","sourceTreeSha256":"sha256:308f677ff2e3cf385651be93135cb894bfb4da2685dbd1d60ace0175cf529df7","tarballSha256":"sha256:5dd2857fc10993ad649a37a5415f99f0eee00448d977987b0c2e0de3d2ec20e7","unpackedManifestSha256":"sha256:9c3f39f099a9ccdcd46c8f7aa7f5fd99181f077fcef9f68bbbec3053eff64abe"},"artifactSha256":"sha256:e439b8d7af6f45abf2c4946bd625a34f158d0a044dd964caa0e811b36edb73d5","artifacts":{"installation":{"bytes":483,"path":"artifacts/8.1.3-installation.json","sha256":"sha256:b6a3c96a32cb3fc452b33a2902dbb675527b9114d69c6396db15c269a8c8fce7"},"session":{"bytes":5604,"path":"artifacts/8.1.3-session.json","sha256":"sha256:3e99ac49adcc4652c7c6ce50dd69bf4f255035460ac0dd5d761c0bf8a1ebaff4"},"transcript":{"bytes":90864,"path":"artifacts/8.1.3-transcript.json","sha256":"sha256:3701b5e804e6a4ac6fa43c968dce8103812682ad592411d06aec8d60850a9f95"}},"checklistSha256":"sha256:bcaa925277568ce9b67c50f0785c3bfbadfa5bb40a6f36b39fad63c21a605110","checklistVersion":"phase9-canary-v1","checks":{"captures-validation":true,"closes-with-delivery":true,"dispatches-reviewer":true,"installs-packed-artifact":true,"loads-flow-tools":true,"saves-plan":true},"derivationVersion":"canary-evidence-v1","expiresAt":"2026-09-03T01:49:21.490Z","hostConfigSha256":"sha256:bd746f2268a19c862787a96cdc94f3798f540ea283c36ca4bf9e112d81c94b07","operator":"codex","pluginEntrySha256":"sha256:00dab6339ee003de42138be504a47ad7b9aeba9041ee844fa4d02c5f941c4f47","preparedSha256":"sha256:64e9af87fa05512025457b8a721bb876a75fa11e1d8a61b62477381ea47ec099","recordSha256":"sha256:72bb80a760a7c85532e0cc17277500538f574128340d5507662dfce1df3f44f2","recordedAt":"2026-08-31T01:49:21.490Z","releaseTag":"v8.1.3","schemaVersion":1,"status":"passed"}

evals/canary/artifacts/8.1.3-installation.json

@@ -0,0 +1 @@
+{"artifactSha256":"sha256:e439b8d7af6f45abf2c4946bd625a34f158d0a044dd964caa0e811b36edb73d5","installedPluginSha256":"sha256:00dab6339ee003de42138be504a47ad7b9aeba9041ee844fa4d02c5f941c4f47","pluginEntrySha256":"sha256:00dab6339ee003de42138be504a47ad7b9aeba9041ee844fa4d02c5f941c4f47","preparedSha256":"sha256:64e9af87fa05512025457b8a721bb876a75fa11e1d8a61b62477381ea47ec099","schemaVersion":1,"tarballSha256":"sha256:5dd2857fc10993ad649a37a5415f99f0eee00448d977987b0c2e0de3d2ec20e7"}