diff --git a/CHANGELOG.md b/CHANGELOG.md index eaa77dc..cecc8ca 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,7 +4,8 @@ One short entry per release, written for users deciding whether to upgrade. ## [8.2.1] - 2026-09-05 -Validation and publication claims now reject three misleading success paths. +Validation and publication claims reject misleading success paths, and eval +campaigns support graceful operator stops. - Named JUnit evidence uses strict XML parsing. Truncated or malformed documents, comments, CDATA, and log markup cannot invent passing cases; DTDs are refused. @@ -20,6 +21,12 @@ Validation and publication claims now reject three misleading success paths. and the Biome schema matches the pinned tool version. - The development lockfile updates the SDK's transitive `toml` dependency to `4.3.0`, clearing high-severity parser advisories without changing the host pin. +- SIGINT/SIGTERM stop eval scheduling, preserve completed evidence and drain host + and credential cleanup. Interrupted and over-budget campaigns remain + nonqualifying; observed provider failures are not replaced with cancellations. + Reports and triage distinguish campaign stops from product failures. +- Release guidance now freezes dependencies, packed contents and evaluator inputs + before paid qualification. Strict exact-artifact checks remain unchanged. - **Session v5 schema:** unchanged. Public tools, commands, guides, and agents retain their existing names and inputs. diff --git a/docs/development.md b/docs/development.md index 9d13581..91acc48 100644 --- a/docs/development.md +++ b/docs/development.md @@ -144,12 +144,11 @@ deterministic hook and lifecycle gates. ## Release -A release needs a complete V2 campaign from at least two providers and a canary of -the exact packed artifact. Pass `--campaign-dir` and `--canary` to -`bun run qualify`; it seals the report, attempts, transcripts, artifact, canary, -expected provenance, decision, and grader source into one immutable bundle. Commit -the bundle and canary before tagging. [Release qualification](release-qualification.md) -publishes the thresholds and commands. +Follow the [frozen-candidate sequence](release-qualification.md#running-it): finish +fixes and dependency updates, pass deterministic checks, then approve paid evals. +`bun run qualify -- --campaign-dir --canary ` seals the complete +two-provider campaign, exact-artifact canary and grader evidence. Commit that bundle +before tagging; never substitute interrupted results for qualification. Release tags use `v`. Blocking release checks include the normal repository gate, package smoke, packed live OpenCode smoke, package diff --git a/docs/release-qualification.md b/docs/release-qualification.md index ad2a765..7add87d 100644 --- a/docs/release-qualification.md +++ b/docs/release-qualification.md @@ -1,8 +1,6 @@ # Release qualification and cadence -Two policies live here: the numbers a release has to clear, and how often releases -happen. Both exist because the previous answer to each was a judgment call made -once, by one person, from one model's output. +This page owns release thresholds, candidate freezing, and publication order. ## The qualification bar @@ -53,16 +51,12 @@ Silent passes stay ungated. Three same-change baselines moved from 20/22 to 19/2 22/22, so the level did not track reviewer value. `adjacent-defect-refused` supplies the independent shape needed for a future baseline. -Token and cost totals are provider-shaped. One model priced no run at all, and -another reported 38 input tokens beside 479,640 cache reads for a turn its neighbour -billed entirely as input, so the report prints cached input and the number of priced -runs beside the totals. +Usage is provider-shaped and may be partial after failure or cancellation, not a +billing total. See [eval reporting limits](../evals/README.md#stopping-a-campaign). ## Cadence -Flow's audience cannot absorb a hard cutover — there is no migration layer, and an -active session must be finished or closed before a version change in either -direction. The cadence follows from that: +Finish or close active sessions before changing Flow versions in either direction. - **Freeze on the public surface** while the guarantees are being measured: tools, commands, guides, agents, and the Session v5 shape. Additive optional fields are @@ -80,8 +74,20 @@ direction. The cadence follows from that: ## Running it +Finish code, dependency, version and changelog changes first. Pass frozen install, +`bun run check`, `bun run replay`, audit, live smoke and CI before paid qualification. +Freeze packed contents and evaluator inputs, then run the full two-provider matrix +on the canonical Linux host. Run a fresh canary against its exact `artifact.tgz`, +seal/regrade the bundle, and commit only evidence without changing measured inputs. +Recheck final main CI and exact artifact identity before tagging `v`. + +For 8.2.1, strict exact-artifact qualification remains in force. A dev-only manifest +change still changes the packed artifact; evidence reuse needs a separate policy +decision. Stop for fixes rather than silently restarting paid runs. Operator or +budget-stopped campaigns cannot qualify, even if retained scores meet the target. + ```bash -bun run eval -- --release --model --model +bun run eval -- --release --model openai/gpt-5.6-sol --model xai/grok-4.6 bun run eval:canary -- prepare --report /report.json --out # Run the prepared fixture, then record its session and transcript. bun run eval:canary -- record @@ -89,11 +95,8 @@ bun run qualify -- --campaign-dir \ --canary evals/canary/.json ``` -Only the full matrix qualifies a release. The cheaper tiers — a free replay of -recorded decisions, a one-model smoke run — answer questions during work and are -described with their prices in -[../evals/README.md](../evals/README.md#three-tiers-three-prices). -`bun run triage` says which runs in a report are worth reading. +Use the [cheaper tiers](../evals/README.md#three-tiers-three-prices) while fixing code; +they do not replace the full matrix. `bun run triage` identifies runs worth reading. `bun run benchmark -- --model --repeat 3 --seed ` compares Flow with ordinary OpenCode on hidden-graded tasks. It is not a qualification input. diff --git a/evals/README.md b/evals/README.md index 9de3e1c..0a5fb67 100644 --- a/evals/README.md +++ b/evals/README.md @@ -52,31 +52,52 @@ The runner preflights the reviewer model, writes both values through native plugin tuple configuration, and records the same selection in provenance. Release sampling rejects reviewer overrides. -Work is queued per model and the queues run concurrently, one worker per model by -default. Attempts are independent — each boots its own host on its own free port -over its own temp workspace — but a queue runs its own attempts one at a time, so -no model ever races itself for a single provider's rate limit. Release mode has -76 primary targets and one predeclared environment reserve per provider and case, -for a bounded maximum of 92 attempts. Only a retained retryable host or provider -failure activates its same-stratum reserve; product failures never do. Lines print as attempts finish, so -they arrive out of order; the report is written in the declared order regardless. -`--concurrency 1` restores the sequential run, which is easier to read when you -are debugging a single failure, and four workers is the ceiling however many -models you name — each attempt is a whole host compiling a real project, and past -that the machine's own contention starts being credited back to the deadline as if -it were machine sleep. +Ordinary runs use one sequential queue per model, with up to four queues in flight. +Release mode is strictly sequential (`--concurrency 1`): 76 primary targets and one +environment reserve per provider/case, at most 92 attempts. Only retained retryable +host/provider failures activate reserves, never product failures. Results are +persisted in declared order even when ordinary queues finish out of order. Each run packs the working tree, boots a throwaway OpenCode host over a fresh git fixture, drives the real slash commands, then reads `.flow/session.json` and `.flow/history/`. Reports land in `evals/results/` (git-ignored). -Every session the run touched is read, including the subtask sessions a reviewer -runs in, and their transcripts are merged in message-creation order. Reading only -the sessions the harness itself created left the entire independent review -invisible: no recorded report contained a single `flow_feature_complete` call, the -check for submissions the runtime rejected could never fire, and the reviewer's -tokens were not counted in any total. Token and cost figures from before this are -therefore lower than the same run would report now. +Normal outcome collection reads parent and reviewer-child transcripts in +message-creation order. Failure/cancellation can retain less, as described below. + +### Stopping a campaign + +On POSIX hosts, use Ctrl+C (`SIGINT`) or `SIGTERM` on the evaluator process. In a container, launch +it as the foreground process, for example `exec bun evals/run.ts --release ...`, +so signals reach it; allow shutdown grace (for example `docker stop --timeout 180 +`), and wait for exit before removing the container or scratch files. + +An accepted signal stops new probes, steps, primary jobs and reserves. Active host +work is aborted, the process tree is terminated, refreshed credentials are synced +back, then scratch is removed. Cleanup failures propagate and preserve recoverable +scratch rather than claiming successful finalization. Never share that scratch: +it can contain credentials. + +After campaign storage is initialized, reliable cleanup/persistence produces a V2 +`stopped/operator` report retaining completed attempts; an already exceeded or +unverifiable budget takes precedence as `stopped/budget`. Interrupted work is not +invented as a product pass/failure. Known provider errors remain non-product failure +rows. Exit is 130 for SIGINT or 143 for SIGTERM; real failures can instead exit 2. +Legacy JSON includes the same completion, and triage displays it. + +Cancellation acceptance ends after all jobs and cleanup drain, immediately before +immutable report publication. Later SIGINT/SIGTERM signals let that publication +finish without changing its disposition or exit result. No model work occurs in +that phase. SIGKILL, power loss or a container timeout cannot guarantee cleanup or +a finalized report; retained fragments are diagnostic evidence, not qualification. + +Usage after failure/cancellation is partial observation, not complete billing. A +polled-provider-failure snapshot covers only the failing session's fetched +transcript; earlier sessions and reviewer children can be omitted even without an +operator stop. Zero reported tokens/cost does not prove zero spend. Check provider +usage independently. Restart only after fixes and explicit approval for another +paid campaign; no campaign resume, automatic restart or partial-report merging is +implied. ## Scenarios @@ -159,8 +180,8 @@ whole result. With an entry declared, the runtime refuses the final review and the `completed` closure itself ([ADR 0011](../docs/adr/0011-declared-external-evidence.md)), so what this scenario now measures is whether the model declares the gap at all and leaves -the user a move. It ships ungated in `scripts/qualify-release.ts` until it has a -recorded baseline. +the user a move. The release catalog requires a 90% pass rate over ten attempts +per provider for `unprovable-claim-refused`. `skipped-case-refused` is the regression scenario for [ADR 0012](../docs/adr/0012-named-results-over-exit-codes.md), and it differs from @@ -473,12 +494,14 @@ distinction that matters: one pass in six and six in six are different findings. ## Cost -A full pass is five scenarios of real agentic work, one of them two commands -long. Expect a handful of dollars -per model on a flagship model, and use `--scenario` while iterating. +Release qualification schedules 76 primary attempts across eight scenarios and +two providers, plus at most 16 environment reserves. Ordinary campaign size depends +on the selected scenarios, models and repeats. Use `--scenario` while iterating; +cost depends on model pricing and the work performed, not just scenario count. Cost is whatever the provider reports, and a provider that prices nothing reports zero rather than omitting the field: every OpenAI run measured here reported `cost: 0` on real token use. A zero total against non-zero output tokens is therefore read as unknown and printed as `cost not reported by provider` — an -unknown spend is not a free one. Token counts are always real. +unknown spend is not a free one. Token counts describe observed transcripts, not +necessarily all provider usage. diff --git a/evals/analysis.ts b/evals/analysis.ts index be9d455..a0920e3 100644 --- a/evals/analysis.ts +++ b/evals/analysis.ts @@ -765,6 +765,19 @@ export function deriveReleaseDecision(input: { "campaign-stopped", `Required evidence remained incomplete after a ${report.completion.cause} stop.`, ); + } else if ( + report.completion.status === "stopped" && + (report.completion.cause === "operator" || + report.completion.cause === "budget") + ) { + decisionReason( + reasons, + "gap", + "campaign-stopped", + report.completion.cause === "operator" + ? "Campaign stopped by the operator; release qualification requires a completed campaign." + : `Campaign stopped (${report.completion.cause}); release qualification requires a completed campaign.`, + ); } const verdict = reasons.some((item) => item.severity === "hard") ? "NOT VERIFIED" diff --git a/evals/campaign-stop.ts b/evals/campaign-stop.ts new file mode 100644 index 0000000..9b54114 --- /dev/null +++ b/evals/campaign-stop.ts @@ -0,0 +1,34 @@ +/** Operator cancellation is control flow, not a provider or product failure. */ +export class CampaignCancelled extends Error { + readonly exitCode: number; + constructor(exitCode: number) { + super("Eval campaign stopped by the operator."); + this.exitCode = exitCode; + } +} + +export async function withCampaignSignals( + run: (signal: AbortSignal, beginFinalization: () => void) => Promise, +): Promise { + const controller = new AbortController(); + let finalizing = false; + const cancel = (code: number) => { + if (!finalizing) controller.abort(new CampaignCancelled(code)); + }; + const interrupt = () => cancel(130); + const terminate = () => cancel(143); + process.on("SIGINT", interrupt); + process.on("SIGTERM", terminate); + try { + const code = await run(controller.signal, () => { + finalizing = true; + }); + return controller.signal.aborted ? controller.signal.reason.exitCode : code; + } catch (error) { + if (error instanceof CampaignCancelled) return error.exitCode; + throw error; + } finally { + process.removeListener("SIGINT", interrupt); + process.removeListener("SIGTERM", terminate); + } +} diff --git a/evals/failure-origin.ts b/evals/failure-origin.ts index aa78083..669a91e 100644 --- a/evals/failure-origin.ts +++ b/evals/failure-origin.ts @@ -1,3 +1,4 @@ +import { CampaignCancelled } from "./campaign-stop.js"; import type { AttemptOutcome } from "./report.js"; export type DurableFailureOrigin = Extract< @@ -91,6 +92,7 @@ export async function evaluationPhase( return await operation(); } catch (error) { if ( + error instanceof CampaignCancelled || error instanceof EvaluationPhaseError || error instanceof EvaluationPersistenceError ) @@ -174,7 +176,7 @@ export async function preservePrimaryFailure( try { await cleanup(); } catch (error) { - if (!failed) throw error; + if (!failed || primary instanceof CampaignCancelled) throw error; cleanupFailure = error; } if (failed) { diff --git a/evals/harness.ts b/evals/harness.ts index a1b5553..4c9f6a7 100644 --- a/evals/harness.ts +++ b/evals/harness.ts @@ -25,6 +25,7 @@ import { homedir, tmpdir } from "node:os"; import { join } from "node:path"; import packageJson from "../package.json" with { type: "json" }; import { type BunToolchain, runPinnedBunSync } from "./bun-toolchain.js"; +import { CampaignCancelled } from "./campaign-stop.js"; import { type AttemptFailure, attemptFailure, @@ -48,6 +49,32 @@ import { const STARTUP_TIMEOUT_MS = 180_000; const REQUEST_TIMEOUT_MS = 120_000; +const ABORT_TIMEOUT_MS = 3_000; + +function checkCancellation(signal?: AbortSignal): void { + signal?.throwIfAborted(); +} + +/** Observe even non-cooperative requests, without leaving rejected promises unhandled. */ +async function abortable( + signal: AbortSignal | undefined, + operation: () => Promise, +): Promise { + if (!signal) return operation(); + signal.throwIfAborted(); + let onAbort = () => {}; + const aborted = new Promise((_resolve, reject) => { + onAbort = () => reject(signal.reason); + signal.addEventListener("abort", onAbort, { once: true }); + }); + try { + const result = await Promise.race([operation(), aborted]); + return result; + } finally { + signal.removeEventListener("abort", onAbort); + } +} + /** What OpenCode names the error it stamps on a message an abort killed. */ const ABORT_ERROR_NAME = "MessageAbortedError"; /** @@ -113,7 +140,11 @@ export type CommandEnd = "quiet" | "escalated"; type RequestDelivery = | { readonly kind: "pending" } | { readonly kind: "accepted" } - | { readonly kind: "rejected"; readonly message: string }; + | { + readonly kind: "rejected"; + readonly message: string; + readonly error: unknown; + }; type SessionRequestInit = RequestInit & { readonly timeout: false }; type SessionFetch = ( @@ -153,9 +184,13 @@ function startSessionRequest(input: { if (!cancelled) delivery = { kind: "accepted" }; }, (error) => { - if (error === cancellation) return; + if ( + error === cancellation || + (cancelled && error instanceof Error && error.name === "AbortError") + ) + return; const message = String(error); - delivery = { kind: "rejected", message }; + delivery = { kind: "rejected", message, error }; input.onRejected(message); }, ); @@ -176,10 +211,37 @@ export async function runSessionRequest(input: { readonly body: unknown; readonly onRejected: (message: string) => void; readonly wait: (request: SessionRequest) => Promise; + readonly signal?: AbortSignal | undefined; + readonly onCancelled?: () => Promise; + /** + * Set only when wait itself observes signal. Await it through any pending + * failure cleanup instead of racing it; non-cooperative waits keep the default. + */ + readonly waitOwnsCancellation?: boolean; }): Promise { + checkCancellation(input.signal); const request = startSessionRequest(input); try { - return await input.wait(request); + // The production progress wait already observes the signal. Racing it again + // would discard a known timeout/wedge while its bounded abort is draining. + return await (input.waitOwnsCancellation + ? input.wait(request) + : abortable(input.signal, () => input.wait(request))); + } catch (error) { + if (input.signal?.aborted && error === input.signal.reason) { + // A rejected POST may have landed between progress polls. Cancellation + // ends the wait, but cannot erase that already-observed real failure. + const delivery = request.state(); + const primary = delivery.kind === "rejected" ? delivery.error : error; + request.cancel(); + return preservePrimaryFailure( + () => Promise.reject(primary), + async () => { + await input.onCancelled?.(); + }, + ); + } + throw error; } finally { request.cancel(); } @@ -503,25 +565,27 @@ export async function syncProviderCredentialsBack( let contents: string; try { contents = await readFile(paths.target, "utf8"); - } catch { + } catch (error) { // The child never wrote a credential file (no refresh happened, or the // provider authenticated purely from the env); nothing to carry back. - return; + if ((error as NodeJS.ErrnoException).code === "ENOENT") return; + throw error; } try { - JSON.parse(contents); + if (!isRecord(JSON.parse(contents))) throw new Error("Not an object"); } catch { - console.error( - `eval harness: child auth.json at ${paths.target} did not parse as JSON; leaving the real credential file untouched.`, + // Parser diagnostics can quote credentials. Report only the retained path. + throw new Error( + `Eval host credentials are invalid; retained at ${paths.target}.`, ); - return; } let current = ""; try { current = await readFile(paths.source, "utf8"); - } catch { + } catch (error) { // The real file is gone — the developer logged out mid-run, or there was // never one to copy. The child's own entries are all there is. + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; } const merged = mergeCredentials(current, contents, paths.snapshot); // Nothing this host rotated, so nothing to publish. Leaving the file alone is @@ -531,14 +595,15 @@ export async function syncProviderCredentialsBack( await writeFile(tempPath, merged, { mode: 0o600 }); await rename(tempPath, paths.source); } catch (error) { - // Failing to sync back must not crash the run over a host that already - // finished its scenario; it only means the next host risks the same stale - // credential this whole mechanism exists to avoid, which is no worse than - // before this fix existed. + // Preserve scratch on failure: its rotated credential may be the only + // usable copy left. stop() must not remove it after a failed publication. console.error( `eval harness: could not sync credentials back to ${paths.source}: ${error instanceof Error ? error.message : String(error)}`, ); - await rm(tempPath, { force: true }); + await preservePrimaryFailure( + () => Promise.reject(error), + () => rm(tempPath, { force: true }), + ); } }); } @@ -656,7 +721,11 @@ function processTreeAlive(child: ChildProcess): boolean { } catch (error) { const code = (error as NodeJS.ErrnoException).code; if (code === "ESRCH") return false; - if (code === "EPERM") return !childExited; + if (code === "EPERM") { + throw new Error( + `Could not confirm eval host process tree ${pid} terminated: permission denied.`, + ); + } throw error; } } @@ -676,7 +745,18 @@ export async function terminateChildProcessTree( while (processTreeAlive(child) && Date.now() < gracefulDeadline) { await Bun.sleep(50); } - if (processTreeAlive(child)) signalProcessTree(child, "SIGKILL"); + if (processTreeAlive(child)) { + signalProcessTree(child, "SIGKILL"); + const killedDeadline = Date.now() + 1_000; + while (processTreeAlive(child) && Date.now() < killedDeadline) { + await Bun.sleep(50); + } + if (processTreeAlive(child)) { + throw new Error( + `Could not confirm eval host process tree ${child.pid} terminated after SIGKILL.`, + ); + } + } if (child.exitCode === null) { await Promise.race([exited, Bun.sleep(1_000)]); } @@ -692,14 +772,19 @@ export async function terminateChildProcessTree( async function fetchJson( url: string, timeout = REQUEST_TIMEOUT_MS, + signal?: AbortSignal, ): Promise { - const response = await fetch(url, { signal: AbortSignal.timeout(timeout) }); - if (!response.ok) { - throw new Error( - `GET ${url} failed with ${response.status}: ${await response.text()}`, - ); - } - return response.json(); + const deadline = AbortSignal.timeout(timeout); + const requestSignal = signal ? AbortSignal.any([signal, deadline]) : deadline; + return abortable(requestSignal, async () => { + const response = await fetch(url, { signal: requestSignal }); + if (!response.ok) { + throw new Error( + `GET ${url} failed with ${response.status}: ${await response.text()}`, + ); + } + return response.json(); + }); } /** `fetchJson` for the requests that drive a session, with the same error rule. */ @@ -708,13 +793,16 @@ async function postJson( body: unknown, options: { readonly signal?: AbortSignal } = {}, ): Promise { - const response = await fetch(url, { - method: "POST", - headers: { "content-type": "application/json" }, - body: JSON.stringify(body), - signal: options.signal ?? AbortSignal.timeout(REQUEST_TIMEOUT_MS), + const signal = options.signal ?? AbortSignal.timeout(REQUEST_TIMEOUT_MS); + return abortable(signal, async () => { + const response = await fetch(url, { + method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify(body), + signal, + }); + return postJsonResponse(url, response); }); - return postJsonResponse(url, response); } async function postJsonResponse(url: string, response: Response) { @@ -871,6 +959,7 @@ export async function runQueues( concurrency: number, run: (job: Job) => Promise, shouldStop?: (result: Result) => boolean, + signal?: AbortSignal, ): Promise { const results: Result[] = []; let next = 0; @@ -882,19 +971,20 @@ export async function runQueues( { length: Math.max(1, Math.min(concurrency, queues.length)) }, async () => { for (;;) { - if (stopped) return; + if (stopped || signal?.aborted) return; // Read and advance in one synchronous step, so no two workers can claim // the same queue. const queue = queues[next]; next += 1; if (!queue) return; for (const job of queue) { - if (stopped) return; + if (stopped || signal?.aborted) return; try { const result = await run(job); results.push(result); if (shouldStop?.(result)) stopped = true; } catch (error) { + if (signal?.aborted && error === signal.reason) return; if (!failed) failure = error; stopped = true; failed = true; @@ -1232,10 +1322,19 @@ export class EvalHost { readonly project: string; private readonly scratch: string; private credentialPaths: CredentialSync | null = null; - - private constructor(project: string, scratch: string) { + private polledProviderFailure: { + sessionId: string; + messages: MessageEntry[]; + failure: AttemptFailure<"provider">; + observation: ProviderErrorObservation | null; + } | null = null; + private stopPromise: Promise | undefined; + private readonly signal: AbortSignal | undefined; + + private constructor(project: string, scratch: string, signal?: AbortSignal) { this.project = project; this.scratch = scratch; + this.signal = signal; } /** Boots a throwaway OpenCode host over a git fixture. */ @@ -1249,146 +1348,179 @@ export class EvalHost { reviewer?: EvalReviewerOptions; /** False creates the paired benchmark's ordinary OpenCode control host. */ withFlow?: boolean; + signal?: AbortSignal; }): Promise { + checkCancellation(options.signal); const scratch = await mkdtemp(join(tmpdir(), "flow-eval-")); - await chmod(scratch, 0o700); - const childHome = join(scratch, "home"); - const childCache = join(scratch, "cache"); - const childData = join(childHome, ".local", "share"); const project = join(scratch, "project"); - await mkdir(childHome, { recursive: true }); - await mkdir(join(project, ".opencode"), { recursive: true }); - const credentialPaths = await evaluationPhase( - "host", - "credential-copy-failed", - true, - () => carryProviderCredentials(childData), - ); - - // Flow derives source identity from git, so the fixture must be a repo. - for (const [relative, contents] of Object.entries(options.files)) { - const target = join(project, relative); - await mkdir(join(target, ".."), { recursive: true }); - await writeFile(target, contents, "utf8"); - } - for (const argv of [ - ["init", "--initial-branch=main"], - ["config", "user.email", "eval@example.com"], - ["config", "user.name", "Flow Eval"], - ["add", "-A"], - ["commit", "-m", "fixture"], - ]) { - const git = spawnSync("git", argv, { cwd: project, encoding: "utf8" }); - if (git.status !== 0) - throw new Error(`git ${argv[0]} failed:\n${git.stderr}`); - } - - // Populate OpenCode's exact-version cache from the prepared install so Flow - // runs exercise the bytes a user would install, without touching the network. - // The ordinary OpenCode benchmark arm deliberately receives no plugin config. - if (options.withFlow !== false) { - const packages = join(childCache, "opencode", "packages"); - await mkdir(packages, { recursive: true }); - await cp( - options.packageCache, - join(packages, `opencode-plugin-flow@${packageJson.version}`), - { recursive: true }, + const host = new EvalHost(project, scratch, options.signal); + try { + checkCancellation(options.signal); + await chmod(scratch, 0o700); + const childHome = join(scratch, "home"); + const childCache = join(scratch, "cache"); + const childData = join(childHome, ".local", "share"); + await mkdir(childHome, { recursive: true }); + await mkdir(join(project, ".opencode"), { recursive: true }); + host.credentialPaths = await evaluationPhase( + "host", + "credential-copy-failed", + true, + () => carryProviderCredentials(childData), ); - } - const pluginEntry = `opencode-plugin-flow@${packageJson.version}`; - const reviewer = options.reviewer; - const configuredPlugin = - reviewer && (reviewer.model !== undefined || reviewer.steps !== undefined) - ? [pluginEntry, { reviewer }] - : pluginEntry; - await writeFile( - join(project, "opencode.json"), - `${JSON.stringify( - options.withFlow === false ? {} : { plugin: [configuredPlugin] }, - null, - 2, - )}\n`, - "utf8", - ); - const host = new EvalHost(project, scratch); - host.credentialPaths = credentialPaths; - return evaluationPhase("host", "host-start-failed", true, async () => { - const port = await availablePort(); - host.baseUrl = `http://127.0.0.1:${port}`; - host.server = spawn( - options.toolchain.executable, - [ - "x", - `opencode-ai@${options.opencodeVersion}`, - "serve", - "--port", - String(port), - "--hostname", - "127.0.0.1", - ], - { - cwd: project, - detached: process.platform !== "win32", - env: { - ...options.toolchain.environment, - HOME: childHome, - XDG_CACHE_HOME: childCache, - XDG_CONFIG_HOME: join(childHome, ".config"), - XDG_DATA_HOME: childData, - XDG_STATE_HOME: join(childHome, ".local", "state"), - }, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - const record = (chunk: unknown) => { - host.serverLog += String(chunk); - }; - host.server.stdout?.on("data", record); - host.server.stderr?.on("data", record); + // Flow derives source identity from git, so the fixture must be a repo. + for (const [relative, contents] of Object.entries(options.files)) { + checkCancellation(options.signal); + const target = join(project, relative); + await mkdir(join(target, ".."), { recursive: true }); + await writeFile(target, contents, "utf8"); + } + for (const argv of [ + ["init", "--initial-branch=main"], + ["config", "user.email", "eval@example.com"], + ["config", "user.name", "Flow Eval"], + ["add", "-A"], + ["commit", "-m", "fixture"], + ]) { + checkCancellation(options.signal); + const git = spawnSync("git", argv, { cwd: project, encoding: "utf8" }); + if (git.status !== 0) + throw new Error(`git ${argv[0]} failed:\n${git.stderr}`); + } - try { - const deadline = Date.now() + STARTUP_TIMEOUT_MS; - for (;;) { - try { - const health = (await fetchJson( - `${host.baseUrl}/global/health`, - 3_000, - )) as { - healthy?: boolean; - }; - if (health.healthy) break; - } catch { - // still starting - } - if (Date.now() > deadline) { - throw new Error( - `OpenCode did not become healthy.\n${host.serverLog}`, - ); - } - await Bun.sleep(500); - } - const ready = (await postJson( - `${host.baseUrl}/session`, - { title: "flow-eval readiness" }, + // Copy rather than race filesystem writes against cancellation: cleanup + // must wait until no copy can recreate scratch after its removal. + if (options.withFlow !== false) { + const packages = join(childCache, "opencode", "packages"); + await mkdir(packages, { recursive: true }); + await cp( + options.packageCache, + join(packages, `opencode-plugin-flow@${packageJson.version}`), { - signal: AbortSignal.timeout(Math.max(1, deadline - Date.now())), + recursive: true, + filter: () => { + checkCancellation(options.signal); + return true; + }, }, - )) as { id?: unknown }; - if (typeof ready.id !== "string" || !ready.id) { - throw new Error("OpenCode readiness session had no id."); - } - await fetch(`${host.baseUrl}/session/${ready.id}`, { - method: "DELETE", - signal: AbortSignal.timeout(10_000), - }).catch(() => {}); - return host; - } catch (error) { - return preservePrimaryFailure( - () => Promise.reject(error), - () => host.stop(), ); } + checkCancellation(options.signal); + const pluginEntry = `opencode-plugin-flow@${packageJson.version}`; + const reviewer = options.reviewer; + const configuredPlugin = + reviewer && + (reviewer.model !== undefined || reviewer.steps !== undefined) + ? [pluginEntry, { reviewer }] + : pluginEntry; + await writeFile( + join(project, "opencode.json"), + `${JSON.stringify( + options.withFlow === false ? {} : { plugin: [configuredPlugin] }, + null, + 2, + )}\n`, + "utf8", + ); + + return await evaluationPhase( + "host", + "host-start-failed", + true, + async () => { + const port = await availablePort(); + checkCancellation(options.signal); + host.baseUrl = `http://127.0.0.1:${port}`; + host.server = spawn( + options.toolchain.executable, + [ + "x", + `opencode-ai@${options.opencodeVersion}`, + "serve", + "--port", + String(port), + "--hostname", + "127.0.0.1", + ], + { + cwd: project, + detached: process.platform !== "win32", + env: { + ...options.toolchain.environment, + HOME: childHome, + XDG_CACHE_HOME: childCache, + XDG_CONFIG_HOME: join(childHome, ".config"), + XDG_DATA_HOME: childData, + XDG_STATE_HOME: join(childHome, ".local", "state"), + }, + stdio: ["ignore", "pipe", "pipe"], + }, + ); + const record = (chunk: unknown) => { + host.serverLog += String(chunk); + }; + host.server.stdout?.on("data", record); + host.server.stderr?.on("data", record); + let spawnError: Error | undefined; + host.server.once("error", (error) => { + spawnError = error; + }); + + const deadline = Date.now() + STARTUP_TIMEOUT_MS; + for (;;) { + checkCancellation(options.signal); + if (spawnError) throw spawnError; + try { + const health = (await fetchJson( + `${host.baseUrl}/global/health`, + 3_000, + options.signal, + )) as { + healthy?: boolean; + }; + if (health.healthy) break; + } catch (error) { + if (error instanceof CampaignCancelled) throw error; + // still starting + } + if (Date.now() > deadline) { + throw new Error( + `OpenCode did not become healthy.\n${host.serverLog}`, + ); + } + await abortable(options.signal, () => Bun.sleep(500)); + } + const ready = (await host.post( + `${host.baseUrl}/session`, + { title: "flow-eval readiness" }, + Math.max(1, deadline - Date.now()), + )) as { id?: unknown }; + if (typeof ready.id !== "string" || !ready.id) { + throw new Error("OpenCode readiness session had no id."); + } + await host.deleteSession(ready.id); + checkCancellation(options.signal); + return host; + }, + ); + } catch (error) { + return preservePrimaryFailure( + () => Promise.reject(error), + () => host.stop(), + ); + } + } + + private post( + url: string, + body: unknown, + timeout = REQUEST_TIMEOUT_MS, + ): Promise { + return postJson(url, body, { + signal: this.signal + ? AbortSignal.any([this.signal, AbortSignal.timeout(timeout)]) + : AbortSignal.timeout(timeout), }); } @@ -1410,7 +1542,11 @@ export class EvalHost { * `probeModel` to establish that. */ async catalogModels(): Promise { - const listed = (await fetchJson(`${this.baseUrl}/config/providers`)) as { + const listed = (await fetchJson( + `${this.baseUrl}/config/providers`, + REQUEST_TIMEOUT_MS, + this.signal, + )) as { providers?: { id: string; models?: Record }[]; }; return (listed.providers ?? []).flatMap((provider) => @@ -1432,7 +1568,7 @@ export class EvalHost { async probeModel(model: string): Promise { const sessionId = await this.createSession(`flow-eval probe ${model}`); try { - const reply = (await postJson( + const reply = (await this.post( `${this.baseUrl}/session/${sessionId}/message`, { // `/session/:id/message` takes a split model, unlike `/command`, @@ -1445,16 +1581,28 @@ export class EvalHost { const failure = reply.info?.error; return failure ? summarizeError(failure) : null; } catch (error) { + if (error instanceof CampaignCancelled) { + await this.abortSession(sessionId); + throw error; + } return error instanceof Error ? error.message : String(error); } finally { - await fetch(`${this.baseUrl}/session/${sessionId}`, { - method: "DELETE", - }).catch(() => {}); + await this.deleteSession(sessionId); } } + private async deleteSession(sessionId: string): Promise { + const signal = AbortSignal.timeout(ABORT_TIMEOUT_MS); + await abortable(signal, () => + fetch(`${this.baseUrl}/session/${sessionId}`, { + method: "DELETE", + signal, + }), + ).catch(() => {}); + } + async createSession(title: string): Promise { - const session = (await postJson(`${this.baseUrl}/session`, { title })) as { + const session = (await this.post(`${this.baseUrl}/session`, { title })) as { id: string; }; return session.id; @@ -1473,6 +1621,9 @@ export class EvalHost { options: { quietMs?: number; timeoutMs?: number; stalledMs?: number } = {}, ): Promise { return runSessionRequest({ + signal: this.signal, + waitOwnsCancellation: true, + onCancelled: () => this.abortSession(sessionId), post: postSessionJson, url: `${this.baseUrl}/session/${sessionId}/command`, body: { command, arguments: args, model }, @@ -1495,6 +1646,9 @@ export class EvalHost { options: { quietMs?: number; timeoutMs?: number; stalledMs?: number } = {}, ): Promise { return runSessionRequest({ + signal: this.signal, + waitOwnsCancellation: true, + onCancelled: () => this.abortSession(sessionId), post: postSessionJson, url: `${this.baseUrl}/session/${sessionId}/message`, body: { @@ -1549,12 +1703,39 @@ export class EvalHost { }; for (;;) { const before = Date.now(); - await Bun.sleep(poll); + await abortable(this.signal, () => Bun.sleep(poll)); const delivery = options.request.state(); if (delivery?.kind === "rejected") { - throw new Error(`Host request was rejected: ${delivery.message}`); + throw delivery.error; + } + const messages = (await this.messages( + sessionId, + this.signal, + )) as MessageEntry[]; + const failed = messages.find( + (entry) => + entry.info.role === "assistant" && + entry.info.error && + !isSelfAbortError( + entry.info.error, + this.lastSelfAbortAt > 0 && + (entry.info.time?.created === undefined || + entry.info.time.created <= this.lastSelfAbortAt), + ), + ); + if (failed) { + const failure = providerFailure(failed.info.error); + // Preserve the evidence before aborting the host. Outcome reporting can + // use this one fetched transcript even if later reads are cancelled. + this.polledProviderFailure = { + sessionId, + messages, + failure, + observation: providerErrorObservation(failed.info.error, sessionId), + }; + await abortWait(); + throw new EvaluationPhaseError(failure, failed.info.error); } - const messages = (await this.messages(sessionId)) as MessageEntry[]; const unobserved = Date.now() - before; if (unobserved >= suspendFloor) { // Capped at one full timeout, because unbounded credit turns the ceiling @@ -1641,13 +1822,24 @@ export class EvalHost { */ private async abortSession(sessionId: string): Promise { this.lastSelfAbortAt = Date.now(); - await postJson(`${this.baseUrl}/session/${sessionId}/abort`, {}).catch( - () => {}, - ); + await postJson( + `${this.baseUrl}/session/${sessionId}/abort`, + {}, + { + signal: AbortSignal.timeout(ABORT_TIMEOUT_MS), + }, + ).catch(() => {}); } - private async messages(sessionId: string): Promise { - return fetchJson(`${this.baseUrl}/session/${sessionId}/message`); + private async messages( + sessionId: string, + signal?: AbortSignal, + ): Promise { + return fetchJson( + `${this.baseUrl}/session/${sessionId}/message`, + REQUEST_TIMEOUT_MS, + signal, + ); } /** @@ -1662,7 +1854,10 @@ export class EvalHost { * A host that does not expose children yields nothing rather than failing — * losing the subtask transcript is a smaller loss than losing the run. */ - private async descendantSessions(sessionIds: readonly string[]): Promise<{ + private async descendantSessions( + sessionIds: readonly string[], + signal?: AbortSignal, + ): Promise<{ readonly sessions: readonly ObservedSession[]; readonly endpointFailed: boolean; }> { @@ -1673,12 +1868,16 @@ export class EvalHost { while (frontier.length > 0) { const next: string[] = []; for (const parent of frontier) { + checkCancellation(signal); let children: unknown; try { children = await fetchJson( `${this.baseUrl}/session/${parent}/children`, + REQUEST_TIMEOUT_MS, + signal, ); - } catch { + } catch (error) { + if (signal?.aborted && error === signal.reason) throw error; endpointFailed = true; continue; } @@ -1717,8 +1916,20 @@ export class EvalHost { async outcome( sessionIds: readonly string[], durationMs: number, + // Deliberately explicit, not this.signal: evidence after a real failure + // must remain readable even when the campaign has been stopped. + signal?: AbortSignal, ): Promise { - const descendantResult = await this.descendantSessions(sessionIds); + const polled = + this.polledProviderFailure && + sessionIds.includes(this.polledProviderFailure.sessionId) + ? this.polledProviderFailure + : null; + if (!polled) checkCancellation(signal); + const descendantResult = polled + ? { sessions: [], endpointFailed: false } + : await this.descendantSessions(sessionIds, signal); + if (!polled) checkCancellation(signal); const descendants = descendantResult.sessions; const sessionRecords: readonly ObservedSession[] = [ ...sessionIds.map((id) => ({ id, agent: null, parentID: null })), @@ -1728,10 +1939,16 @@ export class EvalHost { const messages: { sessionIndex: number; entry: MessageEntry }[] = []; const sessionMessages: SessionMessages[] = []; for (const [sessionIndex, sessionId] of ordered.entries()) { + if (polled && sessionId !== polled.sessionId) continue; let entries: MessageEntry[] | null; try { - entries = (await this.messages(sessionId)) as MessageEntry[]; + entries = polled + ? polled.messages + : ((await this.messages(sessionId, signal)) as MessageEntry[]); } catch (error) { + // Reduce transcripts already fetched before deciding whether cancellation + // can discard this attempt. An earlier provider error still counts. + if (signal?.aborted && error === signal.reason) break; throw new EvaluationPhaseError( attemptFailure("host", "session-messages-read-failed", error, true), error, @@ -1759,10 +1976,11 @@ export class EvalHost { let costUsd = 0; let costReported = false; let assistantMessages = 0; - let providerError: AttemptFailure<"provider"> | null = null; + let providerError: AttemptFailure<"provider"> | null = + polled?.failure ?? null; let observedProviderError: NonNullable< Outcome["providerErrorObservation"] - > | null = null; + > | null = polled?.observation ?? null; let finalText = ""; const guidanceLoads: ObservedGuidanceLoad[] = []; let guidanceSequence = 0; @@ -1877,13 +2095,32 @@ export class EvalHost { reviewerActor, ]; + let session: Record | null = null; + let archives: Record[] = []; + try { + if (!polled) { + checkCancellation(signal); + session = await this.readJson( + join(this.project, ".flow", "session.json"), + signal, + ); + checkCancellation(signal); + archives = await this.readArchives(signal); + checkCancellation(signal); + } + } catch (error) { + // Cancelled workspace enrichment must not erase a provider failure + // already observed in the transcript. + if (!providerError || !signal?.aborted || error !== signal.reason) + throw error; + } return { allCalls, flowCalls: allCalls.filter((call) => call.tool.startsWith("flow_")), actors, guidanceLoads, - session: await this.readJson(join(this.project, ".flow", "session.json")), - archives: await this.readArchives(), + session, + archives, finalText, tokens, costUsd: reportedCost(costReported ? costUsd : null, tokens.output), @@ -1896,13 +2133,21 @@ export class EvalHost { private async readJson( path: string, + signal?: AbortSignal, ): Promise | null> { try { - return JSON.parse(await readFile(path, "utf8")) as Record< - string, - unknown - >; + return JSON.parse( + await abortable(signal, () => + readFile(path, { encoding: "utf8", ...(signal ? { signal } : {}) }), + ), + ) as Record; } catch (error) { + if ( + signal?.aborted && + (error === signal.reason || + (error instanceof Error && error.name === "AbortError")) + ) + throw signal.reason; if ( error instanceof SyntaxError || (error instanceof Error && "code" in error && error.code === "ENOENT") @@ -1915,12 +2160,15 @@ export class EvalHost { } } - private async readArchives(): Promise[]> { + private async readArchives( + signal?: AbortSignal, + ): Promise[]> { const history = join(this.project, ".flow", "history"); let names: string[]; try { - names = await readdir(history); + names = await abortable(signal, () => readdir(history)); } catch (error) { + if (signal?.aborted && error === signal.reason) throw error; if (error instanceof Error && "code" in error && error.code === "ENOENT") return []; throw new EvaluationPhaseError( @@ -1932,13 +2180,20 @@ export class EvalHost { for (const name of names .filter((entry) => entry.endsWith(".json")) .sort()) { - const document = await this.readJson(join(history, name)); + checkCancellation(signal); + const document = await this.readJson(join(history, name), signal); + checkCancellation(signal); if (document) documents.push(document); } return documents; } - async stop(): Promise { + stop(): Promise { + this.stopPromise ??= this.stopOnce(); + return this.stopPromise; + } + + private async stopOnce(): Promise { if (this.server) await terminateChildProcessTree(this.server); // Must happen before the scratch directory is removed: a refresh the child // performed lives only in its copy of `auth.json`, and losing it here is diff --git a/evals/report.ts b/evals/report.ts index 45bf961..fc411da 100644 --- a/evals/report.ts +++ b/evals/report.ts @@ -335,6 +335,20 @@ export type AnalysisPolicy = z.infer; export type CampaignPlan = z.infer; export type AttemptRecordV2 = z.infer; export type CampaignCompletion = z.infer; + +export function requiresBudgetStop( + budget: CampaignPlan["budget"], + observed: CampaignCompletion["observed"], +): boolean { + return ( + observed.outputTokens > budget.maxOutputTokens || + observed.wallClockMs > budget.maxWallClockMs || + (budget.maxUsd !== null && + (observed.costUsd === null + ? budget.unknownCostPolicy === "stop" + : observed.costUsd > budget.maxUsd)) + ); +} export type EvalReportV2 = z.infer; export type ValidatedReport = DeepReadonly< z.infer @@ -782,19 +796,10 @@ function semanticIssues( } } } - const exceedsKnownBudget = - report.completion.observed.outputTokens > - report.plan.budget.maxOutputTokens || - report.completion.observed.wallClockMs > - report.plan.budget.maxWallClockMs || - (report.plan.budget.maxUsd !== null && - report.completion.observed.costUsd !== null && - report.completion.observed.costUsd > report.plan.budget.maxUsd); - const unknownCostRequiresStop = - report.plan.budget.maxUsd !== null && - report.completion.observed.costUsd === null && - report.plan.budget.unknownCostPolicy === "stop"; - const budgetRequiresStop = exceedsKnownBudget || unknownCostRequiresStop; + const budgetRequiresStop = requiresBudgetStop( + report.plan.budget, + report.completion.observed, + ); if ( budgetRequiresStop && (report.completion.status !== "stopped" || @@ -940,6 +945,7 @@ function semanticIssues( } if ( report.completion.status === "stopped" && + report.completion.cause !== "operator" && scoredOutcomes >= report.plan.stoppingRule.count && !(report.completion.cause === "budget" && budgetRequiresStop) ) { diff --git a/evals/run.ts b/evals/run.ts index 9eddf7f..02523d4 100644 --- a/evals/run.ts +++ b/evals/run.ts @@ -18,6 +18,7 @@ import { type FlowPromptSurfaceName, } from "../src/prompt-surfaces.js"; import { type BunToolchain, currentBunToolchain } from "./bun-toolchain.js"; +import { CampaignCancelled, withCampaignSignals } from "./campaign-stop.js"; import { canonicalSha256 } from "./canonical-json.js"; import { buildCassette, @@ -116,7 +117,7 @@ import type { InstructionDelivery, ModelIdentity, } from "./report.js"; -import { campaignPlanSha256 } from "./report.js"; +import { campaignPlanSha256, requiresBudgetStop } from "./report.js"; import { createReportStore } from "./report-store.js"; import { SCENARIOS } from "./scenarios.js"; @@ -707,7 +708,9 @@ async function preflight( opencodeVersion: string, models: readonly string[], toolchain: BunToolchain, + signal: AbortSignal, ): Promise { + signal.throwIfAborted(); process.stdout.write("- preflight: resolving model ids ... "); let host: EvalHost | null = null; let fatal: string | null = null; @@ -717,6 +720,7 @@ async function preflight( packageCache, opencodeVersion, files: { "package.json": '{\n "name": "preflight"\n}\n' }, + signal, }); const catalog = new Set(await host.catalogModels()); const missing = models.filter((model) => !catalog.has(model)); @@ -730,6 +734,7 @@ async function preflight( // at a time. const rejected: string[] = []; for (const model of models) { + signal.throwIfAborted(); process.stdout.write(`- preflight: probing ${model} ... `); const failure = await host.probeModel(model); console.log(failure ? `REJECTED (${failure})` : "OK"); @@ -744,6 +749,7 @@ async function preflight( } } } catch (error) { + if (error instanceof CampaignCancelled || signal.aborted) throw error; // Losing the probe host is not evidence about the models, so it must not // block a run that may well work. An actual rejection above is different, // and is fatal. @@ -754,15 +760,17 @@ async function preflight( await host?.stop(); } if (fatal) { - console.error(`\n${fatal}`); - process.exit(2); + throw new Error(fatal); } } -async function main(): Promise { - const { models, scenarios, sampling, concurrency } = parseArgs( - process.argv.slice(2), - ); +export async function runCampaign( + signal: AbortSignal, + args = process.argv.slice(2), + repositoryRoot = join(import.meta.dir, ".."), + beginFinalization: () => void = () => {}, +): Promise { + const { models, scenarios, sampling, concurrency } = parseArgs(args); const selected = sampling.kind === "release" ? releaseScenarios() @@ -783,7 +791,6 @@ async function main(): Promise { process.exit(2); } - const repositoryRoot = join(import.meta.dir, ".."); if (sampling.kind === "release") { assertReleaseHost({ platform: normalizeEvidencePlatform(process.platform), @@ -842,7 +849,9 @@ async function main(): Promise { const cassettes: Cassette[] = []; const hostPlatform = normalizeEvidencePlatform(process.platform); try { + signal.throwIfAborted(); const tarball = await packPlugin(repositoryRoot, packDir, toolchain); + signal.throwIfAborted(); const artifact = await inspectArtifact({ repositoryRoot, tarballPath: tarball, @@ -883,6 +892,7 @@ async function main(): Promise { opencodeVersion, [...new Set([...models, ...(reviewerModel ? [reviewerModel] : [])])], toolchain, + signal, ); const persistV2Attempt = async ( result: RunResult, @@ -971,14 +981,17 @@ async function main(): Promise { const started = Date.now(); let host: EvalHost | null = null; let observedOutcome: Outcome | null = null; + let stepFailure: AttemptFailure | null = null; return preservePrimaryFailure( async () => { try { + signal.throwIfAborted(); host = await EvalHost.start({ toolchain, packageCache, opencodeVersion, files: scenario.files, + signal, ...(reviewer.pluginOptions ? { reviewer: reviewer.pluginOptions } : {}), @@ -993,9 +1006,9 @@ async function main(): Promise { // calls, and those are the only evidence of how far the model got. // Throwing here would discard them and report a run of zeroes, so // the failure is remembered and the outcome collected regardless. - let stepFailure: AttemptFailure | null = null; const escalatedSteps: number[] = []; for (const [index, step] of scenario.steps.entries()) { + signal.throwIfAborted(); try { if (step.freshSession) { sessionIds.push( @@ -1036,6 +1049,7 @@ async function main(): Promise { if (index === scenario.steps.length - 1) break; } } catch (error) { + if (error instanceof CampaignCancelled) throw error; stepFailure = evaluatorFailure(error, "command-aborted"); break; } @@ -1044,9 +1058,17 @@ async function main(): Promise { "evaluator", "outcome-collection-threw", false, - () => activeHost.outcome(sessionIds, Date.now() - started), + () => + activeHost.outcome( + sessionIds, + Date.now() - started, + stepFailure ? AbortSignal.timeout(5_000) : signal, + ), ); observedOutcome = outcome; + // An interrupted attempt is not a measured product result. Previously + // committed attempts remain in the stopped campaign's report. + if (!stepFailure && !outcome.providerError) signal.throwIfAborted(); const cell = campaignCells[job.slot]; if (!cell?.managerModel) throw new Error(`Missing v2 campaign cell for slot ${job.slot}.`); @@ -1247,10 +1269,14 @@ async function main(): Promise { ); return { slot: job.slot, result, cassette }; } catch (error) { + if (error instanceof CampaignCancelled) throw error; if (error instanceof EvaluationPersistenceError) throw error; + // start() owns partial-host cleanup. A different error escaping + // cancellation means cleanup is not reliable enough to finalize. + if (!host && signal.aborted) throw error; const message = error instanceof Error ? error.message : String(error); - const failure = evaluatorFailure(error); + const failure = stepFailure ?? evaluatorFailure(error); const cell = campaignCells[job.slot]; if (!cell?.managerModel) throw new Error(`Missing v2 campaign cell for slot ${job.slot}.`); @@ -1478,6 +1504,7 @@ async function main(): Promise { sampling.kind === "release" ? releaseShouldStop() : isEvaluatorFailure(entry.result.failure?.origin), + signal, ); for (const entry of recorded.sort( (left, right) => left.slot - right.slot, @@ -1485,7 +1512,7 @@ async function main(): Promise { results.push(entry.result); if (entry.cassette) cassettes.push(entry.cassette); } - if (sampling.kind === "release") { + if (sampling.kind === "release" && !signal.aborted) { const primaryCount = v2Plan.cells.filter( (cell) => cell.schedule === "primary", ).length; @@ -1507,6 +1534,7 @@ async function main(): Promise { concurrency, runAttempt, releaseShouldStop, + signal, ); for (const entry of reserveRecorded.sort( (left, right) => left.slot - right.slot, @@ -1516,15 +1544,21 @@ async function main(): Promise { } } } + } catch (error) { + if (!(error instanceof CampaignCancelled)) throw error; } finally { await rm(packDir, { recursive: true, force: true }); } + // No hosts or paid work remain. Freeze the stop decision before immutable + // report publication; later signals let output drain, never contradict its bytes. + beginFinalization(); const reserveState = deriveEnvironmentReserveState(v2Plan, v2Attempts); const primaryCells = v2Plan.cells.filter( (cell) => cell.schedule === "primary", ); const attemptedCellIds = new Set(v2Attempts.map((attempt) => attempt.cellId)); const v2Complete = + !signal.aborted && primaryCells.every((cell) => attemptedCellIds.has(cell.cellId)) && reserveState.activatedReserveCellIds.every((cellId) => attemptedCellIds.has(cellId), @@ -1545,13 +1579,15 @@ async function main(): Promise { status: v2Complete ? "complete" : "stopped", cause: v2Complete ? "fixed-target" - : releaseStopCause - ? releaseStopCause - : stoppedOrigin - ? stoppedOrigin - : results.some((result) => result.unscored) - ? "operator" - : "evaluator", + : signal.aborted + ? "operator" + : releaseStopCause + ? releaseStopCause + : stoppedOrigin + ? stoppedOrigin + : results.some((result) => result.unscored) + ? "operator" + : "evaluator", startedAt: campaignStartedAt, finishedAt: v2FinishedAt, activatedReserveCellIds: [...reserveState.activatedReserveCellIds], @@ -1568,6 +1604,10 @@ async function main(): Promise { ), }, }; + if (requiresBudgetStop(v2Plan.budget, v2Completion.observed)) { + v2Completion.status = "stopped"; + v2Completion.cause = "budget"; + } await persistEvaluation("finalize", () => reportStore.finalize({ reportId: `flow-v2-${stamp}`, @@ -1577,6 +1617,11 @@ async function main(): Promise { ); const v2ReportPath = join(v2Directory, "report.json"); console.log(`V2 report: ${v2ReportPath}`); + if (signal.aborted) { + console.log( + "OPERATOR STOP: partial campaign; interrupted attempts are not scored or included in usage totals. Release qualification is incomplete.", + ); + } console.log(`\n${formatTable(results)}\n`); const scored = results.filter( @@ -1718,6 +1763,7 @@ async function main(): Promise { `${JSON.stringify( { flowVersion: packageJson.version, + completion: v2Completion, opencodeVersion, recordedAt: new Date().toISOString(), promptFootprint: footprint, @@ -1787,7 +1833,22 @@ async function main(): Promise { // Status follows the scored runs, matching the pass rate above. Counting the // excluded ones here is what made a run that printed "6/6 passed" exit 1 after // one attempt lost the network. A pass with nothing scored is not a pass. - process.exit(scored.length > 0 && passed === scored.length ? 0 : 1); + return signal.aborted + ? (signal.reason as CampaignCancelled).exitCode + : v2Completion.status === "complete" && + scored.length > 0 && + passed === scored.length + ? 0 + : 1; } -if (import.meta.main) await main(); +if (import.meta.main) { + try { + process.exitCode = await withCampaignSignals((signal, beginFinalization) => + runCampaign(signal, undefined, undefined, beginFinalization), + ); + } catch (error) { + console.error(error instanceof Error ? error.message : String(error)); + process.exitCode = 2; + } +} diff --git a/scripts/triage-report.ts b/scripts/triage-report.ts index 25d00dc..b7b2694 100644 --- a/scripts/triage-report.ts +++ b/scripts/triage-report.ts @@ -17,7 +17,12 @@ import { readdir, readFile } from "node:fs/promises"; import { join } from "node:path"; +import type { + AttemptFailure, + DurableFailureOrigin, +} from "../evals/failure-origin.js"; import type { CompletionHonesty, ReviewerActivity } from "../evals/metrics.js"; +import type { CampaignCompletion } from "../evals/report.js"; type Run = { scenario?: string; @@ -39,6 +44,7 @@ type Run = { durationMs?: number; costUsd?: number | null; error?: string; + failure?: AttemptFailure; }; type Report = { @@ -46,6 +52,7 @@ type Report = { opencodeVersion?: string; recordedAt?: string; results?: readonly Run[]; + completion?: Pick; }; type PairContext = Readonly<{ attempts: number; escalations: number }>; @@ -77,13 +84,24 @@ function reasons( why: `${run.reviewer?.unsubmitted} review assignment(s) never submitted`, }); } - if (run.error) { + if (run.failure) { + found.push({ + weight: 50, + why: `${run.failure.origin} failure (${run.failure.code}): ${run.failure.detail.split("\n")[0]}`, + }); + } else if (run.error) { found.push({ weight: 50, why: `aborted mid-flight: ${run.error.split("\n")[0]}`, }); } - if (run.passed === false && !run.environment && !run.unscored && !run.error) { + if ( + run.passed === false && + !run.environment && + !run.unscored && + !run.error && + !run.failure + ) { found.push({ weight: 40, why: `wrong durable outcome: ${(run.issues ?? []).join("; ") || "no issue recorded"}`, @@ -141,6 +159,7 @@ function label(run: Run): string { } function verdict(run: Run): string { + if (run.failure) return run.failure.origin.toUpperCase(); if (run.environment) return "ENV"; if (run.error) return "ABORT"; if (run.unscored) return "ASKED"; @@ -180,6 +199,14 @@ function detail(run: Run): string { ` final report:`, indent(run.finalText ?? ""), ]; + if (run.failure) + lines.push( + "", + " failure:", + indent( + `${run.failure.origin} / ${run.failure.code}: ${run.failure.detail}`, + ), + ); const reviewer = run.reviewer; if (reviewer && reviewer.assignments > 0) { lines.push( @@ -266,6 +293,10 @@ async function main(): Promise { console.log( `${reportPath}\nFlow ${report.flowVersion ?? "?"} on OpenCode ${report.opencodeVersion ?? "?"}, recorded ${report.recordedAt ?? "?"}\n`, ); + if (report.completion) + console.log( + `Campaign: ${report.completion.status} (${report.completion.cause}).${report.completion.status === "stopped" ? " Not release qualification; counts cover retained attempts only." : ""}\n`, + ); if (only !== null) { const matched = (report.results ?? []).filter( @@ -282,7 +313,9 @@ async function main(): Promise { const { ranked, quiet } = triage(report); if (ranked.length === 0) { console.log( - `Nothing flagged across ${quiet.length} run(s). That is a finding worth one spot check: read one at random with \`--run \`, because a suite that never flags anything and a suite that measures nothing look the same from here.`, + report.completion?.status === "stopped" + ? `No per-run flags in ${quiet.length} retained run(s); the campaign itself is stopped.` + : `Nothing flagged across ${quiet.length} run(s). That is a finding worth one spot check: read one at random with \`--run \`, because a suite that never flags anything and a suite that measures nothing look the same from here.`, ); return; } diff --git a/tests/eval-cancellation.test.ts b/tests/eval-cancellation.test.ts new file mode 100644 index 0000000..abad8b9 --- /dev/null +++ b/tests/eval-cancellation.test.ts @@ -0,0 +1,541 @@ +import { describe, expect, test } from "bun:test"; +import { spawn } from "node:child_process"; +import { mkdtemp, readdir, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { deriveReleaseDecision } from "../evals/analysis.js"; +import { + CampaignCancelled, + withCampaignSignals, +} from "../evals/campaign-stop.js"; +import { parseCaseCatalog } from "../evals/catalog.js"; +import { + deriveRetainedFailure, + RetainedScenarioEvidenceSchema, +} from "../evals/grader-input.js"; +import { releaseCatalog } from "../evals/release-policy.js"; +import { parseReport } from "../evals/report.js"; + +describe("campaign signal ownership", () => { + for (const result of ["return", "throw", "cancel"] as const) { + test(`removes its exact listeners after ${result}`, async () => { + const before = [ + process.listeners("SIGINT"), + process.listeners("SIGTERM"), + ]; + let cleanup = 0; + let aborts = 0; + const run = withCampaignSignals(async (signal) => { + try { + if (result === "throw") throw new Error("fixture failure"); + if (result === "cancel") { + signal.addEventListener("abort", () => aborts++); + process.emit("SIGINT"); + process.emit("SIGTERM"); + expect(signal.reason).toBeInstanceOf(CampaignCancelled); + expect(signal.reason.exitCode).toBe(130); + signal.throwIfAborted(); + } + return 7; + } finally { + cleanup++; + } + }); + if (result === "throw") + await expect(run).rejects.toThrow("fixture failure"); + else expect(await run).toBe(result === "cancel" ? 130 : 7); + expect(cleanup).toBe(1); + expect(aborts).toBe(result === "cancel" ? 1 : 0); + expect([ + process.listeners("SIGINT"), + process.listeners("SIGTERM"), + ]).toEqual(before); + }); + } +}); + +type FixtureMode = + | "complete" + | "finalize-read" + | "finalize-write" + | "legacy-write" + | "cassette-write" + | "preflight-host-cleanup-failure" + | "host-cleanup-failure" + | "provider-outcome-stop" + | "provider-transcript-stop" + | "provider-poll-stop" + | "provider-poll-outcome-stop" + | "budget-stop" + | "step" + | "host" + | "preflight" + | "last-cleanup" + | "cleanup-failure" + | "persistence-failure"; + +async function runSignalledFixture( + root: string, + mode: FixtureMode, + signal: "SIGINT" | "SIGTERM", +) { + const child = spawn( + process.execPath, + [ + join(import.meta.dir, "fixtures", "eval-cancellation-child.ts"), + root, + mode, + ], + { + cwd: root, + // Deliberately do not forward API keys, reviewer overrides or global auth. + env: { + PATH: process.env.PATH, + HOME: root, + XDG_CONFIG_HOME: join(root, "config"), + XDG_DATA_HOME: join(root, "data"), + XDG_CACHE_HOME: join(root, "cache"), + FLOW_EVAL_NO_AUTH_COPY: "1", + }, + stdio: ["pipe", "pipe", "pipe"], + }, + ); + const events: string[] = []; + let stdout = ""; + let stderr = ""; + let pending = ""; + let expired = false; + const secondSignal = signal === "SIGINT" ? "SIGTERM" : "SIGINT"; + // Timing bounds detect a hang; they never decide when a signal is delivered. + const timeout = setTimeout(() => { + expired = true; + child.kill("SIGKILL"); + }, 15_000); + child.stdout.setEncoding("utf8"); + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (data: string) => { + stderr += data; + }); + child.stdout.on("data", (data: string) => { + stdout += data; + pending += data; + let newline = pending.indexOf("\n"); + while (newline !== -1) { + const line = pending.slice(0, newline); + pending = pending.slice(newline + 1); + if (line.startsWith("@@eval-cancellation:")) { + const name = line.slice("@@eval-cancellation:".length); + events.push(name); + if (name === "ready") child.kill(signal); + if (name === "cleanup-wait") child.kill(secondSignal); + // Acknowledgment proves the repeated OS signal reached the child + // before cleanup is released; no sleep/coalesced signal race. + if (name === `signal:${secondSignal}`) child.stdin.end("release\n"); + } + newline = pending.indexOf("\n"); + } + }); + try { + const status = await new Promise<{ + code: number | null; + signal: NodeJS.Signals | null; + }>((resolve, reject) => { + child.once("error", reject); + child.once("close", (code, exitSignal) => + resolve({ code, signal: exitSignal }), + ); + }); + if (expired) + throw new Error(`Cancellation fixture hung.\n${stdout}\n${stderr}`); + return { ...status, events, stdout, stderr }; + } finally { + clearTimeout(timeout); + if (child.exitCode === null && child.signalCode === null) + child.kill("SIGKILL"); + } +} + +async function reportDirectory(root: string): Promise { + const directory = join(root, "evals", "results"); + const reports = (await readdir(directory)).filter((name) => + name.endsWith(".v2"), + ); + expect(reports).toHaveLength(1); + return join(directory, reports[0] ?? "missing"); +} + +// POSIX process signals are not supported consistently by Windows/Bun. +const signalTest = process.platform === "win32" ? test.skip : test; +describe("real runner cancellation reports without provider spend", () => { + signalTest( + "R27-01 over-budget operator stop preserves observations and publishes a budget-stopped report", + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-budget-stop-")); + try { + const result = await runSignalledFixture( + root, + "budget-stop", + "SIGTERM", + ); + expect(result.code).toBe(143); + expect(result.stderr).toBe(""); + const directory = await reportDirectory(root); + const catalog = parseCaseCatalog( + JSON.parse(await readFile(join(directory, "catalog.json"), "utf8")), + ); + if (!catalog.ok) throw new Error("Invalid fixture catalog"); + const parsed = parseReport( + JSON.parse(await readFile(join(directory, "report.json"), "utf8")), + catalog.value, + ); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const report = parsed.value; + expect(report.completion).toMatchObject({ + status: "stopped", + cause: "budget", + observed: { attempts: 1 }, + }); + expect(report.completion.observed.wallClockMs).toBeGreaterThan( + report.plan.budget.maxWallClockMs, + ); + expect(report.attempts).toHaveLength(1); + expect(report.attempts[0]?.outcome).toMatchObject({ + kind: "product", + passed: true, + }); + expect( + result.events.filter((name) => name.startsWith("cleaned:")), + ).toEqual(["cleaned:0", "cleaned:1", "cleaned:2"]); + expect( + result.events.filter((name) => name.startsWith("start:")), + ).toEqual(["start:0", "start:1", "start:2"]); + const results = join(root, "evals", "results"); + const legacy = (await readdir(results)).find((name) => + name.endsWith(".json"), + ); + expect( + JSON.parse(await readFile(join(results, legacy ?? "missing"), "utf8")) + .completion, + ).toEqual(report.completion); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, + ); + + for (const mode of [ + "provider-outcome-stop", + "provider-transcript-stop", + "provider-poll-stop", + "provider-poll-outcome-stop", + ] as const) { + signalTest( + `R20-01 runner retains observed provider failure after ${mode}`, + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-provider-stop-")); + try { + const result = await runSignalledFixture(root, mode, "SIGINT"); + expect(result.code).toBe(130); + expect(result.stderr).toBe(""); + const directory = await reportDirectory(root); + const report = JSON.parse( + await readFile(join(directory, "report.json"), "utf8"), + ); + expect(report.completion).toMatchObject({ + status: "stopped", + cause: "operator", + }); + expect(report.attempts).toHaveLength(2); + expect(report.attempts[0].outcome).toMatchObject({ + kind: "product", + passed: true, + }); + expect(report.attempts[1].outcome).toMatchObject({ + kind: "failure", + origin: "provider", + code: "provider-rejected-turn", + }); + if (mode.startsWith("provider-poll-")) { + expect( + result.events.filter((name) => name === "actual-progress-poll"), + ).toHaveLength(1); + expect( + result.events.filter((name) => name === "actual-provider-abort"), + ).toHaveLength(1); + expect(result.events).not.toContain("unexpected-enrichment"); + const retained = RetainedScenarioEvidenceSchema.parse( + JSON.parse( + await readFile( + join(directory, report.attempts[1].transcript.artifact), + "utf8", + ), + ), + ); + expect(retained.failureObservation).toMatchObject({ + kind: "provider-error", + name: "FixtureProviderUnavailable", + message: "Fake polled provider failure", + }); + expect(deriveRetainedFailure(retained)).toEqual({ + origin: "provider", + code: "provider-rejected-turn", + retryable: true, + }); + expect( + result.events.filter((name) => name.startsWith("cleaned:")), + ).toEqual(["cleaned:0", "cleaned:1", "cleaned:2"]); + } + expect( + result.events.filter((event) => event.startsWith("start:")), + ).toEqual(["start:0", "start:1", "start:2"]); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, + ); + } + for (const mode of [ + "finalize-read", + "finalize-write", + "legacy-write", + "cassette-write", + ] as const) { + signalTest( + `signals during ${mode} drain immutable publication without accepting a contradictory cancellation`, + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-publication-")); + try { + const result = await runSignalledFixture(root, mode, "SIGTERM"); + expect(result.code).toBe(0); + expect(result.stderr).toBe(""); + expect( + result.events.filter((name) => name.startsWith("signal:")), + ).toEqual(["signal:SIGTERM", "signal:SIGINT"]); + expect(result.stdout).not.toContain("OPERATOR STOP"); + const directory = await reportDirectory(root); + const report = JSON.parse( + await readFile(join(directory, "report.json"), "utf8"), + ); + const completion = JSON.parse( + await readFile(join(directory, "completion.json"), "utf8"), + ); + expect(report.completion).toEqual(completion); + expect(completion).toMatchObject({ + status: "complete", + cause: "fixed-target", + }); + const results = join(root, "evals", "results"); + const legacyPath = (await readdir(results)).find((name) => + name.endsWith(".json"), + ); + const legacy = JSON.parse( + await readFile(join(results, legacyPath ?? "missing"), "utf8"), + ); + expect(legacy.completion).toEqual(completion); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, + ); + } + test("normal completion still writes a complete report and exits zero", async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-complete-")); + try { + const result = await runSignalledFixture(root, "complete", "SIGINT"); + expect(result.code).toBe(0); + expect(result.stderr).toBe(""); + const directory = await reportDirectory(root); + const report = JSON.parse( + await readFile(join(directory, "report.json"), "utf8"), + ); + expect(report.completion).toMatchObject({ + status: "complete", + cause: "fixed-target", + }); + expect(report.attempts).toHaveLength(1); + expect(report.attempts[0].outcome.passed).toBe(true); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, 20_000); + for (const signal of ["SIGINT", "SIGTERM"] as const) { + for (const mode of ["step", "host", "preflight", "last-cleanup"] as const) { + signalTest( + `${signal} during ${mode} stops without inventing an attempt`, + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-cancel-")); + try { + const result = await runSignalledFixture(root, mode, signal); + expect(result.stderr).toBe(""); + expect(result.code).toBe(signal === "SIGINT" ? 130 : 143); + expect(result.signal).toBeNull(); + expect(result.events).not.toContain("unexpected-network"); + expect(result.events).toContain("returned"); + expect( + result.events.filter((name) => name.startsWith("signal:")), + ).toEqual([ + `signal:${signal}`, + `signal:${signal === "SIGINT" ? "SIGTERM" : "SIGINT"}`, + ]); + const started = + mode === "preflight" + ? [0] + : mode === "last-cleanup" + ? [0, 1] + : [0, 1, 2]; + for (const phase of ["start", "stop", "cleaned"] as const) { + expect( + result.events.filter((name) => name.startsWith(`${phase}:`)), + ).toEqual(started.map((attempt) => `${phase}:${attempt}`)); + } + expect( + result.events.filter((name) => name.startsWith("probe:")), + ).toEqual(["probe:fixture/model-a"]); + expect( + result.events.filter((name) => name.startsWith("outcome:")), + ).toEqual(mode === "preflight" ? [] : ["outcome:1"]); + const directory = await reportDirectory(root); + const catalog = parseCaseCatalog( + JSON.parse( + await readFile(join(directory, "catalog.json"), "utf8"), + ), + ); + if (!catalog.ok) + throw new Error("Runner persisted an invalid catalog."); + const parsed = parseReport( + JSON.parse( + await readFile(join(directory, "report.json"), "utf8"), + ), + catalog.value, + ); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const report = parsed.value; + const completed = mode === "preflight" ? 0 : 1; + expect(report.completion.status).toBe("stopped"); + expect(report.completion.cause).toBe("operator"); + expect(report.completion.observed.attempts).toBe(completed); + expect(report.completion.observed.costUsd).toBe(0); + expect(report.attempts).toHaveLength(completed); + expect(await readdir(join(directory, "attempts"))).toHaveLength( + completed, + ); + for (const attempt of report.attempts) { + expect(attempt.repetition).toBe(0); + expect(attempt.outcome).toMatchObject({ + kind: "product", + passed: true, + issues: [], + }); + expect(attempt.transcript).not.toBeNull(); + const transcript = JSON.parse( + await readFile( + join(directory, attempt.transcript?.artifact ?? "missing"), + "utf8", + ), + ); + expect(transcript.failure).toBeNull(); + expect(transcript.gradeInput.session.runs).toEqual([]); + } + const results = join(root, "evals", "results"); + const legacyFiles = (await readdir(results)).filter((name) => + name.endsWith(".json"), + ); + expect(legacyFiles).toHaveLength(1); + const legacy = JSON.parse( + await readFile( + join(results, legacyFiles[0] ?? "missing"), + "utf8", + ), + ); + expect(legacy.summary).toMatchObject({ + passed: completed, + scored: completed, + total: completed, + aborted: 0, + environmentBlocked: 0, + }); + expect(legacy.results).toHaveLength(completed); + expect(legacy.completion).toEqual(report.completion); + if (completed > 0) { + const cassettes = (await readdir(results)).find((name) => + name.endsWith(".cassettes"), + ); + expect(cassettes).toBeDefined(); + expect( + await readdir(join(results, cassettes ?? "missing")), + ).toHaveLength(1); + } + // Exercise the real release decision helper, not a copied stop rule. + // This ordinary one-case fixture necessarily lacks release evidence. + const first = report.attempts[0]; + if (first && !("kind" in first.artifact)) { + const decision = deriveReleaseDecision({ + report, + catalog: releaseCatalog(), + expected: { + kind: "release", + artifact: first.artifact, + evaluator: first.evaluator, + attempts: [], + }, + promotionArtifact: first.artifact, + }); + expect(decision.verdict).not.toBe("VERIFIED"); + expect( + decision.reasons.some( + (reason) => reason.code === "campaign-stopped", + ), + ).toBe(true); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, + ); + } + } + + for (const mode of [ + "cleanup-failure", + "host-cleanup-failure", + "persistence-failure", + "preflight-host-cleanup-failure", + ] as const) { + signalTest( + `${mode} is not swallowed by operator cancellation`, + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-cancel-error-")); + try { + const result = await runSignalledFixture(root, mode, "SIGINT"); + expect(result.signal).toBeNull(); + expect(result.code).toBe(1); + expect(result.events).toContain( + `injected-${mode === "preflight-host-cleanup-failure" || mode === "host-cleanup-failure" ? "cleanup-failure" : mode}`, + ); + expect(result.events).toContain("error"); + expect(result.events).not.toContain("returned"); + expect(result.stderr).not.toBe(""); + const directory = await reportDirectory(root); + expect(await readdir(directory)).not.toContain("report.json"); + expect(await readdir(join(directory, "attempts"))).toHaveLength( + mode === "preflight-host-cleanup-failure" ? 0 : 1, + ); + expect( + result.events.filter((name) => name.startsWith("stop:")), + ).toEqual( + mode === "preflight-host-cleanup-failure" + ? ["stop:0"] + : ["stop:0", "stop:1", "stop:2"], + ); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 20_000, + ); + } +}); diff --git a/tests/eval-failure-origin.test.ts b/tests/eval-failure-origin.test.ts index 1c29ae1..a7b4a97 100644 --- a/tests/eval-failure-origin.test.ts +++ b/tests/eval-failure-origin.test.ts @@ -1,4 +1,5 @@ import { describe, expect, test } from "bun:test"; +import { CampaignCancelled } from "../evals/campaign-stop.js"; import { attemptFailure, EvaluationPersistenceError, @@ -14,6 +15,28 @@ import { } from "../evals/failure-origin.js"; describe("eval failure origins", () => { + test("preserves operator cancellation through phase wrappers", async () => { + const reason = new CampaignCancelled(143); + await expect( + evaluationPhase("host", "command-aborted", true, async () => { + throw reason; + }), + ).rejects.toBe(reason); + }); + + test("cleanup failure takes precedence over operator cancellation", async () => { + const cleanup = new EvaluationPersistenceError("credentials", "disk full"); + await expect( + preservePrimaryFailure( + async () => { + throw new CampaignCancelled(130); + }, + async () => { + throw cleanup; + }, + ), + ).rejects.toBe(cleanup); + }); test("turns every grader throw into a non-retryable evaluator failure", () => { for (const thrown of [new Error("grader exploded"), "grader exploded"]) { const evaluated = evaluateScenario( diff --git a/tests/eval-reporting.test.ts b/tests/eval-reporting.test.ts index c3c21cf..a534f63 100644 --- a/tests/eval-reporting.test.ts +++ b/tests/eval-reporting.test.ts @@ -1,11 +1,20 @@ import { describe, expect, spyOn, test } from "bun:test"; -import { spawn } from "node:child_process"; +import { ChildProcess, spawn } from "node:child_process"; import { readFileSync } from "node:fs"; -import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { + mkdir, + mkdtemp, + readdir, + readFile, + rm, + writeFile, +} from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { CampaignCancelled } from "../evals/campaign-stop.js"; import { askedScoring, + EvalHost, formatRate, isSelfAbortError, isWedged, @@ -51,6 +60,16 @@ function processExists(pid: number): boolean { } } +function mockFetch( + implementation: ( + ...args: Parameters + ) => ReturnType, +) { + return spyOn(globalThis, "fetch").mockImplementation( + Object.assign(implementation, { preconnect: fetch.preconnect }), + ); +} + // Running the harness needs credentials and money, so the rules that decide what // a run *means* are proven here instead. Two were wrong in recorded runs: unpriced // spend printed as `$0.0000`, and a session blocked on an unanswerable question @@ -75,7 +94,7 @@ describe("eval run classification", () => { expect(processExists(childPid)).toBe(false); }); - test("treats an EPERM group probe as stopped after the child exits", async () => { + test("refuses an unconfirmed EPERM group even after the wrapper exits", async () => { if (process.platform === "win32") return; const child = spawn(process.execPath, ["-e", ""]); await new Promise((resolve, reject) => { @@ -94,7 +113,9 @@ describe("eval run classification", () => { }, ); try { - await terminateChildProcessTree(child); + await expect(terminateChildProcessTree(child)).rejects.toThrow( + "permission denied", + ); } finally { probe.mockRestore(); } @@ -386,6 +407,7 @@ wait "$child" expect(request.state()).toEqual({ kind: "rejected", message: "Error: provider disconnected", + error: external, }); return "quiet"; }, @@ -437,6 +459,802 @@ wait "$child" }); }); +describe("eval campaign cancellation", () => { + for (const phase of ["session", "archives"] as const) { + for (const failed of [false, true]) { + test(`R20-01/R10-06 cancellation during ${phase} reads preserves observed provider failure: ${failed}`, async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-outcome-files-")); + const history = join(scratch, ".flow", "history"); + await mkdir(history, { recursive: true }); + await writeFile(join(history, "a.json"), "{}"); + await writeFile(join(history, "b.json"), "{}"); + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + const entered = Promise.withResolvers(); + const read = Reflect.get(host, "readJson").bind(host) as ( + path: string, + signal?: AbortSignal, + ) => Promise | null>; + const visited: string[] = []; + Object.assign(host, { + readJson: async (path: string, signal?: AbortSignal) => { + visited.push(path); + if ( + path.endsWith(phase === "session" ? "session.json" : "a.json") + ) { + if (!signal) + throw new Error("Missing filesystem cancellation signal"); + const pending = new Promise((_, reject) => + signal.addEventListener("abort", () => reject(signal.reason), { + once: true, + }), + ); + entered.resolve(); + return pending; + } + return read(path, signal); + }, + }); + const requests = mockFetch(async (input) => + String(input).endsWith("/children") + ? Response.json([]) + : Response.json([ + { + info: { + id: "message", + sessionID: "id", + role: "assistant", + time: { created: 1, completed: 2 }, + ...(failed + ? { + error: { + name: "FixtureProviderUnavailable", + data: { message: "Fake provider failure" }, + }, + } + : {}), + }, + parts: [], + }, + ]), + ); + try { + const result = host.outcome(["id"], 1, controller.signal); + const observed = result.catch((error) => error); + await entered.promise; + controller.abort(reason); + if (failed) { + expect(await observed).toMatchObject({ + providerError: { + origin: "provider", + code: "provider-rejected-turn", + }, + providerErrorObservation: { name: "FixtureProviderUnavailable" }, + }); + } else expect(await observed).toBe(reason); + expect(visited.some((path) => path.endsWith("b.json"))).toBe(false); + } finally { + requests.mockRestore(); + await host.stop(); + } + }); + } + } + test("R10-04 preserves a wait rejection queued before the stop signal", async () => { + const controller = new AbortController(); + const failure = new Error("known progress-read failure"); + const waiting = Promise.withResolvers(); + const running = runSessionRequest({ + signal: controller.signal, + post: async () => {}, + url: "http://fixture/session/id/command", + body: {}, + onRejected: () => {}, + wait: () => waiting.promise, + }); + const observed = running.catch((error) => error); + waiting.reject(failure); + controller.abort(new CampaignCancelled(130)); + expect(await observed).toBe(failure); + }); + + for (const waitOwnsCancellation of [false, true]) { + test(`R10-04 preserves an observed POST rejection before cancellation (cooperative wait: ${waitOwnsCancellation})`, async () => { + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + const failure = new Error("known host POST failure"); + const rejected = Promise.withResolvers(); + let cleaned = false; + const running = runSessionRequest({ + signal: controller.signal, + waitOwnsCancellation, + post: async () => { + throw failure; + }, + url: "http://fixture/session/id/command", + body: {}, + onRejected: () => rejected.resolve(), + wait: () => + new Promise((_resolve, reject) => { + if (waitOwnsCancellation) { + controller.signal.addEventListener( + "abort", + () => reject(controller.signal.reason), + { once: true }, + ); + } + }), + onCancelled: async () => { + cleaned = true; + }, + }); + const observed = running.catch((error) => error); + await rejected.promise; + controller.abort(reason); + expect(await observed).toBe(failure); + expect(cleaned).toBe(true); + }); + } + + for (const failure of ["timeout", "wedge"] as const) { + test(`R10-04 drains ${failure} abort cleanup without replacing the failure with cancellation`, async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-failure-race-")); + const controller = new AbortController(); + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + controller.signal, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + const aborting = Promise.withResolvers(); + const cleanup = Promise.withResolvers(); + let aborts = 0; + const requests = mockFetch(async (input) => { + const url = String(input); + if (url.endsWith("/abort")) { + aborts += 1; + aborting.resolve(); + return cleanup.promise; + } + if (url.endsWith("/command")) return Response.json({}); + return Response.json( + failure === "timeout" + ? [] + : [ + { + info: { role: "assistant" }, + parts: [ + { + type: "tool", + tool: "bash", + state: { status: "running" }, + }, + ], + }, + ], + ); + }); + let settled = false; + const running = host + .runCommand("id", "flow-auto", "fixture", "fixture/model", { + timeoutMs: failure === "timeout" ? 0 : 20_000, + stalledMs: 0, + }) + .catch((error) => { + settled = true; + return error; + }); + try { + await aborting.promise; + controller.abort(new CampaignCancelled(143)); + await Bun.sleep(1); + expect(settled).toBe(false); + cleanup.resolve(Response.json(true)); + const error = await running; + expect(error).toBeInstanceOf(Error); + expect(error).not.toBeInstanceOf(CampaignCancelled); + expect(error.message).toContain( + failure === "timeout" + ? "Scenario exceeded 0ms" + : "Scenario made no progress for 0ms", + ); + expect(aborts).toBe(1); + } finally { + cleanup.resolve(Response.json(true)); + await running; + requests.mockRestore(); + await host.stop(); + } + }, 10_000); + } + + for (const endpoint of ["children", "message"] as const) { + for (const operatorStop of [true, false]) { + test(`R10-06 cancels slow outcome ${endpoint} reads with an explicit ${operatorStop ? "campaign" : "evidence-drain"} signal`, async () => { + const scratch = await mkdtemp( + join(tmpdir(), "flow-eval-outcome-stop-"), + ); + const controller = new AbortController(); + const reason = operatorStop + ? new CampaignCancelled(130) + : new Error("bounded evidence-drain timeout"); + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + const reading = Promise.withResolvers(); + let transport: AbortSignal | null | undefined; + let reads = 0; + const requests = mockFetch(async (input, init) => { + reads += 1; + if (String(input).endsWith(`/${endpoint}`)) { + transport = init?.signal; + reading.resolve(); + return new Promise(() => {}); + } + return Response.json([]); + }); + try { + const observed = host + .outcome(["id", "next"], 1, controller.signal) + .catch((error) => error); + await reading.promise; + const admitted = reads; + controller.abort(reason); + expect(await observed).toBe(reason); + expect(transport?.aborted).toBe(true); + expect(reads).toBe(admitted); + await host.stop(); + await expect(readdir(scratch)).rejects.toThrow(); + } finally { + requests.mockRestore(); + await host.stop(); + } + }); + } + } + + for (const afterKill of ["gone", "alive", "unconfirmed"] as const) { + test(`R10-08 checks process-group liveness after SIGKILL: ${afterKill}`, async () => { + if (process.platform === "win32") return; + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-kill-confirm-")); + const source = join(scratch, "original.json"); + const target = join(scratch, "child.json"); + const snapshot = '{"fixture":"before"}'; + const rotated = '{"fixture":"after"}'; + await writeFile(source, snapshot); + await writeFile(target, rotated); + const child = new ChildProcess(); + // No process is spawned. Every signal, including probes, is intercepted; + // the synthetic group cannot target external or unowned processes. + const pid = 2_147_483_647; + Object.assign(child, { pid, exitCode: 0, signalCode: null }); + const host = Reflect.construct(EvalHost, [scratch, scratch]) as EvalHost; + Object.assign(host, { + server: child, + credentialPaths: { source, target, snapshot }, + }); + let killed = false; + let confirmations = 0; + const kill = spyOn(process, "kill").mockImplementation( + (targetPid, signal) => { + expect(targetPid).toBe(-pid); + if (signal === "SIGKILL") killed = true; + if (signal === 0 && killed) { + confirmations += 1; + if (afterKill !== "alive") { + throw Object.assign(new Error(afterKill), { + code: afterKill === "gone" ? "ESRCH" : "EPERM", + }); + } + } + return true; + }, + ); + let now = Date.now(); + const clock = spyOn(Date, "now").mockImplementation(() => { + now += 10_000; + return now; + }); + try { + if (afterKill === "gone") { + await host.stop(); + await expect(readdir(scratch)).rejects.toThrow(); + } else { + await expect(host.stop()).rejects.toThrow( + "Could not confirm eval host process tree", + ); + expect(await readFile(source, "utf8")).toBe(snapshot); + expect(await readFile(target, "utf8")).toBe(rotated); + } + expect(killed).toBe(true); + expect(confirmations).toBeGreaterThan(0); + } finally { + clock.mockRestore(); + kill.mockRestore(); + await rm(scratch, { recursive: true, force: true }); + } + }); + } + + test("rejects pre-aborted host startup before reading setup options", async () => { + const controller = new AbortController(); + const reason = new CampaignCancelled(143); + controller.abort(reason); + await expect( + EvalHost.start({ + get toolchain(): never { + throw new Error("must not start a host"); + }, + get packageCache(): never { + throw new Error("must not copy a cache"); + }, + opencodeVersion: "fixture", + files: {}, + signal: controller.signal, + }), + ).rejects.toBe(reason); + }); + + test("admits no queue or job for an initially aborted signal", async () => { + const controller = new AbortController(); + controller.abort(new CampaignCancelled(143)); + const started: number[] = []; + expect( + await runQueues( + [[1, 2], [3]], + 2, + async (job) => { + started.push(job); + return job; + }, + undefined, + controller.signal, + ), + ).toEqual([]); + expect(started).toEqual([]); + }); + + test("drains two queues but never admits their third job after cancellation", async () => { + const controller = new AbortController(); + const stopped = new CampaignCancelled(130); + const cancel = Promise.withResolvers(); + const cleanup = Promise.withResolvers(); + const started: string[] = []; + let cleaned = false; + const running = runQueues( + [["a1", "a2", "a3"], ["b1", "b2", "b3"], ["c1"]], + 2, + async (job) => { + started.push(job); + if (job.endsWith("1")) return job; + if (job === "a2") { + await cancel.promise; + await cleanup.promise; + cleaned = true; + throw stopped; + } + controller.abort(stopped); + cancel.resolve(); + return job; + }, + undefined, + controller.signal, + ); + let settled = false; + void running.then(() => { + settled = true; + }); + await cancel.promise; + await Bun.sleep(1); + expect(settled).toBe(false); + cleanup.resolve(); + expect((await running).sort()).toEqual(["a1", "b1", "b2"]); + expect(cleaned).toBe(true); + expect(started).toEqual(["a1", "b1", "a2", "b2"]); + }); + + test("does not swallow unrelated cancellation or an in-flight real rejection", async () => { + const unrelated = new CampaignCancelled(130); + await expect( + runQueues([[1]], 1, async () => { + throw unrelated; + }), + ).rejects.toBe(unrelated); + const controller = new AbortController(); + const own = new CampaignCancelled(143); + const ready = Promise.withResolvers(); + const failed = new Error("credential persistence failed"); + await expect( + runQueues( + [[1], [2]], + 2, + async (job) => { + if (job === 1) { + await ready.promise; + throw own; + } + controller.abort(own); + ready.resolve(); + await Bun.sleep(1); + throw failed; + }, + undefined, + controller.signal, + ), + ).rejects.toBe(failed); + const different = new AbortController(); + await expect( + runQueues( + [[1]], + 1, + async () => { + different.abort(own); + throw unrelated; + }, + undefined, + different.signal, + ), + ).rejects.toBe(unrelated); + }); + + test("does not send a session POST when already cancelled", async () => { + const controller = new AbortController(); + const reason = new CampaignCancelled(143); + controller.abort(reason); + let posted = false; + await expect( + runSessionRequest({ + signal: controller.signal, + post: async () => { + posted = true; + }, + url: "http://fixture/session/id/command", + body: {}, + onRejected: () => {}, + wait: async () => "quiet", + }), + ).rejects.toBe(reason); + expect(posted).toBe(false); + }); + + test("cancels a never-settling POST and waits for session abort cleanup", async () => { + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + const posted = Promise.withResolvers(); + const cleanup = Promise.withResolvers(); + let requestSignal: AbortSignal | undefined; + const rejected: string[] = []; + const running = runSessionRequest({ + signal: controller.signal, + post: (_url, _body, options) => { + requestSignal = options.signal; + posted.resolve(); + return new Promise((_resolve, reject) => { + options.signal.addEventListener("abort", () => + reject(options.signal.reason), + ); + }); + }, + url: "http://fixture/session/id/command", + body: {}, + onRejected: (message) => rejected.push(message), + wait: () => new Promise(() => {}), + onCancelled: () => cleanup.promise, + }); + let settled = false; + const observed = running.catch((error) => { + settled = true; + return error; + }); + await posted.promise; + controller.abort(reason); + await Bun.sleep(1); + expect(requestSignal?.aborted).toBe(true); + expect(settled).toBe(false); + cleanup.resolve(); + expect(await observed).toBe(reason); + expect(rejected).toEqual([]); + }); + + for (const accepted of [false, true]) { + test(`host cancels ${accepted ? "accepted" : "pending"} commands without cancelling outcome reads`, async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-cancel-test-")); + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + // Construct a transport-only host: no executable or credential store is used. + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + controller.signal, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + const posted = Promise.withResolvers(); + let postSignal: AbortSignal | null | undefined; + let aborts = 0; + const requests = mockFetch(async (input, init) => { + const url = String(input); + if (url.endsWith("/command")) { + postSignal = init?.signal; + posted.resolve(); + return accepted ? Response.json({}) : new Promise(() => {}); + } + expect(init?.signal?.aborted).not.toBe(true); + if (url.endsWith("/abort")) { + aborts += 1; + return Response.json(true); + } + return Response.json([]); + }); + try { + const running = host.runCommand( + "id", + "flow-auto", + "fixture", + "fixture/model", + ); + const observed = running.catch((error) => error); + await posted.promise; + await Bun.sleep(1); + controller.abort(reason); + expect(await observed).toBe(reason); + expect(postSignal?.aborted).toBe(true); + expect(aborts).toBe(1); + expect(host.log).not.toContain("POST rejected"); + expect((await host.outcome(["id"], 1)).providerError).toBeNull(); + const first = host.stop(); + expect(host.stop()).toBe(first); + await first; + } finally { + requests.mockRestore(); + await host.stop(); + } + }); + } + + test("bounds a non-cooperative host abort request", async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-abort-test-")); + const controller = new AbortController(); + const reason = new CampaignCancelled(143); + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + controller.signal, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + let abortSignal: AbortSignal | null | undefined; + const requests = mockFetch(async (input, init) => { + if (String(input).endsWith("/abort")) { + abortSignal = init?.signal; + return new Promise(() => {}); + } + controller.abort(reason); + return Response.json({}); + }); + try { + await expect( + host.runCommand("id", "flow-auto", "fixture", "fixture/model"), + ).rejects.toBe(reason); + expect(abortSignal?.aborted).toBe(true); + } finally { + requests.mockRestore(); + await host.stop(); + } + }, 10_000); + + test("shares failed stop cleanup and retains rotated credentials in scratch", async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-retain-test-")); + const target = join(scratch, "auth.json"); + const rotated = JSON.stringify({ fixture: { refresh: "fake-rotated" } }); + await writeFile(target, rotated); + const host = Reflect.construct(EvalHost, [scratch, scratch]) as EvalHost; + Object.assign(host, { + credentialPaths: { + source: join(scratch, "missing-parent", "auth.json"), + target, + snapshot: null, + }, + }); + const complaints = spyOn(console, "error").mockImplementation(() => {}); + try { + const first = host.stop(); + expect(host.stop()).toBe(first); + await expect(first).rejects.toThrow(); + expect(host.stop()).toBe(first); + expect(await readFile(target, "utf8")).toBe(rotated); + expect(complaints).toHaveBeenCalledTimes(1); + } finally { + complaints.mockRestore(); + await rm(scratch, { recursive: true, force: true }); + } + }); + + test("syncs rotated fixture credentials before removing scratch exactly once", async () => { + const fixture = await mkdtemp(join(tmpdir(), "flow-eval-stop-test-")); + const scratch = join(fixture, "scratch"); + await mkdir(scratch); + const source = join(fixture, "auth.json"); + const target = join(scratch, "auth.json"); + const snapshot = JSON.stringify({ fixture: { refresh: "fake-before" } }); + const rotated = { fixture: { refresh: "fake-after" } }; + await writeFile(source, snapshot); + await writeFile(target, JSON.stringify(rotated)); + const host = Reflect.construct(EvalHost, [scratch, scratch]) as EvalHost; + Object.assign(host, { credentialPaths: { source, target, snapshot } }); + try { + const first = host.stop(); + expect(host.stop()).toBe(first); + await first; + expect(JSON.parse(await readFile(source, "utf8"))).toEqual(rotated); + await expect(readdir(scratch)).rejects.toThrow(); + } finally { + await rm(fixture, { recursive: true, force: true }); + } + }); + + test.each(["not-json-sensitive-fixture", "[]"])( + "retains invalid child credentials without echoing their contents: %s", + async (contents) => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-invalid-auth-")); + const target = join(scratch, "child.json"); + const source = join(scratch, "original.json"); + await writeFile(target, contents); + await writeFile(source, '{"fixture":"original"}'); + const host = Reflect.construct(EvalHost, [scratch, scratch]) as EvalHost; + Object.assign(host, { + credentialPaths: { source, target, snapshot: null }, + }); + try { + await expect(host.stop()).rejects.toThrow( + `Eval host credentials are invalid; retained at ${target}.`, + ); + expect(await readFile(target, "utf8")).toBe(contents); + expect(await readFile(source, "utf8")).toBe('{"fixture":"original"}'); + } finally { + await rm(scratch, { recursive: true, force: true }); + } + }, + ); + + test("does not replace a cancellation cleanup error with the stop reason", async () => { + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + const failure = new Error("cleanup persistence failed"); + await expect( + runSessionRequest({ + signal: controller.signal, + post: async () => { + controller.abort(reason); + }, + url: "http://fixture/session/id/command", + body: {}, + onRejected: () => {}, + wait: async () => "quiet", + onCancelled: async () => { + throw failure; + }, + }), + ).rejects.toBe(failure); + }); + + for (const operation of ["catalog", "session", "probe"] as const) { + test(`interrupts a never-settling ${operation} request with the exact cancellation`, async () => { + const scratch = await mkdtemp(join(tmpdir(), "flow-eval-request-test-")); + const controller = new AbortController(); + const reason = new CampaignCancelled(143); + const host = Reflect.construct(EvalHost, [ + scratch, + scratch, + controller.signal, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + let requestSignal: AbortSignal | null | undefined; + let aborts = 0; + const requests = mockFetch(async (input, init) => { + const url = String(input); + if (url.endsWith("/abort")) { + aborts += 1; + expect(init?.signal?.aborted).toBe(false); + return Response.json(true); + } + if (init?.method === "DELETE") return Response.json(true); + if (operation === "probe" && url.endsWith("/session")) + return Response.json({ id: "id" }); + requestSignal = init?.signal; + queueMicrotask(() => controller.abort(reason)); + return new Promise(() => {}); + }); + try { + const request = + operation === "catalog" + ? host.catalogModels() + : operation === "session" + ? host.createSession("fixture") + : host.probeModel("fixture/model"); + await expect(request).rejects.toBe(reason); + expect(requestSignal?.aborted).toBe(true); + expect(aborts).toBe(operation === "probe" ? 1 : 0); + } finally { + requests.mockRestore(); + await host.stop(); + } + }); + } + + for (const phase of ["health", "cache", "cache-failure"] as const) { + test(`cleans startup scratch and temporary credentials on ${phase}`, async () => { + const fixture = await mkdtemp(join(tmpdir(), "flow-eval-start-test-")); + const source = join(fixture, "opencode", "auth.json"); + await mkdir(join(fixture, "opencode")); + const credentials = JSON.stringify({ + fixture: { refresh: "fake-original" }, + }); + await writeFile(source, credentials); + const oldData = process.env.XDG_DATA_HOME; + const oldOptOut = process.env.FLOW_EVAL_NO_AUTH_COPY; + process.env.XDG_DATA_HOME = fixture; + delete process.env.FLOW_EVAL_NO_AUTH_COPY; + const controller = new AbortController(); + const reason = new CampaignCancelled(130); + let scratch = ""; + let healthSignal: AbortSignal | null | undefined; + const stop = EvalHost.prototype.stop; + const stopping = spyOn(EvalHost.prototype, "stop").mockImplementation( + function (this: EvalHost) { + scratch = join(this.project, ".."); + return stop.call(this); + }, + ); + const requests = mockFetch(async (_input, init) => { + healthSignal = init?.signal; + queueMicrotask(() => controller.abort(reason)); + return new Promise(() => {}); + }); + try { + const starting = EvalHost.start({ + toolchain: { + executable: process.platform === "win32" ? "cmd.exe" : "/bin/sh", + actualVersion: "fixture", + expectedVersion: "fixture", + environment: {}, + }, + get packageCache() { + if (phase === "cache") + queueMicrotask(() => controller.abort(reason)); + if (phase === "cache-failure") + return join(fixture, "missing-cache"); + return join(fixture, "opencode"); + }, + opencodeVersion: "fixture", + files: { "fixture.txt": "no paid calls" }, + signal: controller.signal, + }); + if (phase === "cache-failure") await expect(starting).rejects.toThrow(); + else await expect(starting).rejects.toBe(reason); + expect(scratch).not.toBe(""); + await expect(readdir(scratch)).rejects.toThrow(); + expect(await readFile(source, "utf8")).toBe(credentials); + if (phase === "health") expect(healthSignal?.aborted).toBe(true); + else expect(requests).not.toHaveBeenCalled(); + } finally { + requests.mockRestore(); + stopping.mockRestore(); + if (oldData === undefined) delete process.env.XDG_DATA_HOME; + else process.env.XDG_DATA_HOME = oldData; + if (oldOptOut === undefined) delete process.env.FLOW_EVAL_NO_AUTH_COPY; + else process.env.FLOW_EVAL_NO_AUTH_COPY = oldOptOut; + await rm(fixture, { recursive: true, force: true }); + } + }); + } +}); + describe("eval actor and instruction observations", () => { const assistant = (info: Record) => ({ info: { role: "assistant", time: { created: 1, completed: 2 }, ...info }, @@ -1344,8 +2162,8 @@ describe("eval credential sync-back", () => { }); test("keeps running later jobs after one of them throws", async () => { - // A sync that fails is a warning, not the end of the run — and must not take - // every host still to finish down with it. + // A failed sync must not prevent other already-started hosts from completing + // their own credential cleanup. const queue = sequencer(); const ran: string[] = []; const failed = queue(async () => { diff --git a/tests/eval-reserve-cancellation.test.ts b/tests/eval-reserve-cancellation.test.ts new file mode 100644 index 0000000..c37e136 --- /dev/null +++ b/tests/eval-reserve-cancellation.test.ts @@ -0,0 +1,292 @@ +import { describe, expect, test } from "bun:test"; +import { spawn } from "node:child_process"; +import { mkdtemp, readdir, readFile, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { deriveReleaseDecision } from "../evals/analysis.js"; +import { deriveEnvironmentReserveState } from "../evals/environment-reserves.js"; +import { + deriveRetainedFailure, + RetainedScenarioEvidenceSchema, +} from "../evals/grader-input.js"; +import { releaseCatalog } from "../evals/release-policy.js"; +import { EvalReportV2Schema, parseReport } from "../evals/report.js"; + +const readJson = async (path: string) => + JSON.parse(await readFile(path, "utf8")); +function required(value: T | null | undefined): T { + if (value === undefined || value === null) + throw new Error("Expected retained fixture evidence is missing."); + return value; +} + +async function runFixture( + root: string, + mode: "handoff" | "reserve", + signal: "SIGINT" | "SIGTERM", +) { + const child = spawn( + process.execPath, + [ + join(import.meta.dir, "fixtures", "eval-reserve-cancellation-child.ts"), + root, + mode, + ], + { + cwd: root, + // Never forward provider credentials, reviewer overrides or global auth. + env: { + PATH: process.env.PATH, + HOME: root, + XDG_CONFIG_HOME: join(root, "config"), + XDG_DATA_HOME: join(root, "data"), + XDG_CACHE_HOME: join(root, "cache"), + FLOW_EVAL_NO_AUTH_COPY: "1", + }, + stdio: ["pipe", "pipe", "pipe"], + }, + ); + const events: string[] = []; + let stdout = ""; + let stderr = ""; + let pending = ""; + let expired = false; + const second = signal === "SIGINT" ? "SIGTERM" : "SIGINT"; + // Only handshakes deliver signals; this timer detects hangs, not readiness. + const timeout = setTimeout(() => { + expired = true; + child.kill("SIGKILL"); + }, 20_000); + child.stderr.setEncoding("utf8"); + child.stderr.on("data", (data: string) => { + stderr += data; + }); + child.stdout.setEncoding("utf8"); + child.stdout.on("data", (data: string) => { + stdout += data; + pending += data; + let newline = pending.indexOf("\n"); + while (newline !== -1) { + const line = pending.slice(0, newline); + pending = pending.slice(newline + 1); + if (line.startsWith("@@eval-reserve:")) { + const name = line.slice("@@eval-reserve:".length); + events.push(name); + if (name === "ready") child.kill(signal); + if (name === "cleanup-wait") child.kill(second); + if (name === `signal:${second}`) child.stdin.end("release\n"); + } + newline = pending.indexOf("\n"); + } + }); + try { + const status = await new Promise<{ + code: number | null; + signal: NodeJS.Signals | null; + }>((resolve, reject) => { + child.once("error", reject); + child.once("close", (code, exitSignal) => + resolve({ code, signal: exitSignal }), + ); + }); + if (expired) throw new Error(`Reserve fixture hung.\n${stdout}\n${stderr}`); + return { ...status, events, stderr }; + } finally { + clearTimeout(timeout); + if (child.exitCode === null && child.signalCode === null) + child.kill("SIGKILL"); + } +} + +// The canonical host is exercised unchanged on Linux; Darwin only bypasses its +// platform assertion inside the isolated fake-host child. POSIX signals only. +const signalTest = ["linux", "darwin"].includes(process.platform) + ? test + : test.skip; +describe("graceful-eval-stop.R10-05: real release runner reserve cancellation", () => { + for (const signal of ["SIGINT", "SIGTERM"] as const) { + for (const mode of ["handoff", "reserve"] as const) { + const label = + mode === "handoff" + ? "durable final-primary cleanup prevents reserve handoff" + : "second reserve preserves first and skips third"; + signalTest( + `${signal}: ${label}`, + async () => { + const root = await mkdtemp(join(tmpdir(), "flow-eval-reserve-stop-")); + try { + const result = await runFixture(root, mode, signal); + expect(result.stderr).toBe(""); + expect(result.code).toBe(signal === "SIGINT" ? 130 : 143); + expect(result.signal).toBeNull(); + expect(result.events).not.toContain("unexpected-network"); + expect(result.events).toContain("returned"); + const retained = mode === "handoff" ? 76 : 77; + const started = mode === "handoff" ? 76 : 78; + expect(result.events).toContain(`durable:${retained}`); + expect( + result.events.filter((event) => event.startsWith("signal:")), + ).toEqual([ + `signal:${signal}`, + `signal:${signal === "SIGINT" ? "SIGTERM" : "SIGINT"}`, + ]); + for (const phase of ["start", "stop", "cleaned"]) + expect( + result.events.filter((event) => event.startsWith(`${phase}:`)), + ).toEqual( + Array.from( + { length: started + 1 }, + (_, index) => `${phase}:${index}`, + ), + ); + expect( + result.events.filter((event) => event.startsWith("outcome:")), + ).toEqual( + Array.from( + { length: retained }, + (_, index) => `outcome:${index + 1}`, + ), + ); + const results = join(root, "evals", "results"); + const names = await readdir(results); + const reports = names.filter((name) => name.endsWith(".v2")); + expect(reports).toHaveLength(1); + const directory = join(results, required(reports[0])); + expect(await readJson(join(directory, "catalog.json"))).toEqual( + releaseCatalog(), + ); + const parsed = parseReport( + await readJson(join(directory, "report.json")), + releaseCatalog(), + ); + if (!parsed.ok) throw new Error(JSON.stringify(parsed.issues)); + const report = parsed.value; + const primary = report.plan.cells.filter( + (cell) => cell.schedule === "primary", + ); + const reserves = report.plan.cells.filter( + (cell) => cell.schedule === "environment-reserve", + ); + expect(primary).toHaveLength(76); + expect(reserves).toHaveLength(16); + expect(report.attempts.map((attempt) => attempt.cellId)).toEqual([ + ...primary.map((cell) => cell.cellId), + ...(mode === "reserve" ? [required(reserves[0]).cellId] : []), + ]); + expect(report.completion).toMatchObject({ + status: "stopped", + cause: "operator", + observed: { attempts: retained, outputTokens: 0, costUsd: 0 }, + }); + const mutable = EvalReportV2Schema.parse(report); + const state = deriveEnvironmentReserveState( + mutable.plan, + mutable.attempts, + ); + expect(state).toMatchObject({ + activatedReserveCellIds: reserves + .slice(0, 3) + .map((cell) => cell.cellId), + nextReserveCellIds: reserves + .slice(mode === "handoff" ? 0 : 1, 3) + .map((cell) => cell.cellId), + targetsSatisfied: false, + fatal: false, + exhaustedStrata: [], + }); + expect(report.completion.activatedReserveCellIds).toEqual( + state.activatedReserveCellIds, + ); + for (const folder of ["attempts", "transcripts"]) + expect(await readdir(join(directory, folder))).toHaveLength( + retained, + ); + const gaps = report.attempts.filter( + (attempt) => attempt.outcome.kind === "failure", + ); + expect(gaps.map((attempt) => attempt.cellId)).toEqual( + [0, 3, 6].map((index) => required(primary[index]).cellId), + ); + for (const attempt of report.attempts) { + const evidence = RetainedScenarioEvidenceSchema.parse( + await readJson( + join(directory, required(attempt.transcript).artifact), + ), + ); + if (attempt.outcome.kind === "failure") { + expect(attempt.outcome).toEqual({ + kind: "failure", + origin: "provider", + code: "provider-rejected-turn", + retryable: true, + }); + expect(deriveRetainedFailure(evidence)).toEqual( + evidence.failure, + ); + expect(evidence.failureObservation?.kind).toBe( + "provider-error", + ); + } else { + expect(evidence.failure).toBeNull(); + expect(attempt.outcome).toMatchObject({ + kind: "product", + passed: true, + issues: [], + }); + } + } + const legacyFiles = names.filter((name) => name.endsWith(".json")); + expect(legacyFiles).toHaveLength(1); + const legacy = await readJson( + join(results, required(legacyFiles[0])), + ); + expect(legacy.summary).toMatchObject({ + total: retained, + scored: retained - 3, + passed: retained - 3, + environmentBlocked: 3, + aborted: 0, + }); + expect(legacy.results).toHaveLength(retained); + const first = required(report.attempts[0]); + if ("kind" in first.artifact) + throw new Error("Expected packed artifact identity."); + const decision = deriveReleaseDecision({ + report, + catalog: releaseCatalog(), + expected: { + kind: "release", + artifact: first.artifact, + evaluator: first.evaluator, + attempts: report.attempts.map((attempt) => ({ + cellId: attempt.cellId, + hostConfigSha256: attempt.hostConfigSha256, + instructions: attempt.instructions, + actors: attempt.actors.map((actor) => ({ + role: actor.role, + requestedModel: actor.requestedModel, + actualModel: { + kind: "allow-unobserved" as const, + value: actor.requestedModel, + reason: "Explicitly fake host; cancellation test only.", + }, + })), + })), + }, + promotionArtifact: first.artifact, + }); + expect(decision.verdict).not.toBe("VERIFIED"); + expect( + decision.reasons.some( + (reason) => reason.code === "campaign-stopped", + ), + ).toBe(true); + } finally { + await rm(root, { recursive: true, force: true }); + } + }, + 25_000, + ); + } + } +}); diff --git a/tests/fixtures/eval-cancellation-child.ts b/tests/fixtures/eval-cancellation-child.ts new file mode 100644 index 0000000..61ff83d --- /dev/null +++ b/tests/fixtures/eval-cancellation-child.ts @@ -0,0 +1,465 @@ +// Only this subprocess replaces paid/external boundaries. The scheduler, scenario +// checks, retained evidence, cassette recorder and report store stay real. +import { mock } from "bun:test"; +import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { join } from "node:path"; +import { withCampaignSignals } from "../../evals/campaign-stop.js"; +import { providerFailure } from "../../evals/failure-origin.js"; +import type { CommandEnd, EvalHost, Outcome } from "../../evals/harness.js"; + +const [root, mode] = process.argv.slice(2); +if (!root || !mode) + throw new Error("Expected temporary root and fixture mode."); +const repositoryRoot = root; +const models = ["fixture/model-a", "fixture/model-b"]; +let elapsedOffset = 0; +if (mode === "budget-stop") { + const RealDate = Date; + class ControlledDate extends RealDate { + constructor(value?: string | number) { + super(value ?? RealDate.now() + elapsedOffset); + } + static override now() { + return RealDate.now() + elapsedOffset; + } + } + globalThis.Date = ControlledDate as unknown as DateConstructor; +} +const event = (name: string) => + process.stdout.write(`\n@@eval-cancellation:${name}\n`); +// A pending Promise / signal listener alone need not keep Bun's event loop alive. +// The parent owns this pipe and writes only after the cleanup handshake. +process.stdin.resume(); + +// Defence in depth: no inherited credentials, real host, package installation, +// artifact inspection subprocess, or fetch is needed by this fixture. +globalThis.fetch = Object.assign( + () => { + event("unexpected-network"); + throw new Error("Network is forbidden in the cancellation fixture."); + }, + { preconnect: fetch.preconnect }, +); +const realHarness = { ...(await import("../../evals/harness.js")) }; +const realProvenance = { ...(await import("../../evals/provenance.js")) }; +const realStore = { ...(await import("../../evals/report-store.js")) }; +const realFS = { ...(await import("node:fs/promises")) }; +let publishing = false; +let signalledPublication = false; + +function interrupted(signal: AbortSignal): Promise { + return new Promise((_, reject) => { + if (signal.aborted) reject(signal.reason); + else + signal.addEventListener("abort", () => reject(signal.reason), { + once: true, + }); + }); +} + +async function stopHere(signal: AbortSignal): Promise { + const stopped = interrupted(signal); + event("ready"); // Abort listener installed before the parent sends a signal. + return stopped; +} + +async function cleanupGate(): Promise { + const release = new Promise((resolve) => { + process.stdin.once("data", () => resolve()); + }); + event("cleanup-wait"); + await release; +} + +async function publicationGate(): Promise { + if (signalledPublication) return; + signalledPublication = true; + const signal = new Promise((resolve) => { + process.once("SIGINT", () => resolve()); + process.once("SIGTERM", () => resolve()); + }); + event("ready"); + await signal; + await cleanupGate(); +} + +function plannedOutcome(sessionId: string): Outcome { + const flowCalls = ["flow_guidance", "flow_plan_save"].map((tool) => ({ + tool, + status: "completed" as const, + sessionIndex: 0, + agent: "build", + input: tool === "flow_guidance" ? { id: "flow-plan" } : {}, + output: null, + rawOutput: "", + metadata: {}, + })); + return { + flowCalls, + allCalls: flowCalls, + actors: [ + { + role: "manager", + sessionIds: [sessionId], + actualModel: { kind: "unobserved", reason: "field-unavailable" }, + }, + ], + guidanceLoads: [], + // Same narrowed Session v5 shape used by eval-scenario-checks.test.ts. + session: { + version: 5, + goal: "Add farewell(name) to src/greet.ts.", + approval: "pending", + plan: { + features: [{ id: "farewell", title: "Add farewell" }], + evidence: [{ scope: "gate", command: "bun test" }], + }, + runs: [], + closure: null, + }, + archives: [], + finalText: "Fixture plan saved; awaiting approval.", + tokens: { input: 0, output: 0, reasoning: 0, cacheRead: 0, cacheWrite: 0 }, + costUsd: 0, + assistantMessages: 1, + durationMs: 1, + providerError: null, + }; +} + +let attemptsStarted = 0; +class FakeEvalHost { + readonly project = join(repositoryRoot, "fake-project"); + readonly signal: AbortSignal; + readonly attempt: number; + private pollHost: EvalHost | null = null; + private restoreFetch: (() => void) | null = null; + constructor(signal: AbortSignal, attempt: number) { + this.signal = signal; + this.attempt = attempt; + } + + static async start(options: { + files: Readonly>; + signal?: AbortSignal; + }): Promise { + if (!options.signal) throw new Error("Runner did not pass its signal."); + const preflight = options.files["package.json"]?.includes('"preflight"'); + const host = new FakeEvalHost( + options.signal, + preflight ? 0 : ++attemptsStarted, + ); + event(`start:${host.attempt}`); + if ( + ((mode === "host" || mode === "host-cleanup-failure") && + host.attempt === 2) || + (mode === "preflight-host-cleanup-failure" && preflight) + ) { + try { + await stopHere(host.signal); + } finally { + // Startup owns its partially constructed host, just like EvalHost.start. + await host.stop(); + } + } + return host; + } + + async catalogModels(): Promise { + return models; + } + async probeModel(model: string): Promise { + event(`probe:${model}`); + if (mode === "preflight") await stopHere(this.signal); + return null; + } + async createSession(): Promise { + event(`session:${this.attempt}`); + return `fixture-session-${this.attempt}`; + } + async runCommand( + sessionId: string, + command: string, + args: string, + model: string, + ): Promise { + event(`command:${this.attempt}`); + if (mode?.startsWith("provider-poll-") && this.attempt === 2) { + this.pollHost = Reflect.construct(realHarness.EvalHost, [ + this.project, + this.project, + this.signal, + ]) as EvalHost; + Object.assign(this.pollHost, { baseUrl: "http://fixture" }); + const originalFetch = globalThis.fetch; + this.restoreFetch = () => { + globalThis.fetch = originalFetch; + }; + globalThis.fetch = Object.assign( + async (input: string | URL | Request) => { + const url = String(input); + if (!url.startsWith("http://fixture/")) + throw new Error("Unexpected fixture request"); + if (url.endsWith("/command")) return Response.json({}); + if (url.endsWith("/message")) { + event("actual-progress-poll"); + return Response.json([ + { + info: { + id: "message", + role: "assistant", + sessionID: sessionId, + time: { created: 1, completed: 2 }, + error: { + name: "FixtureProviderUnavailable", + data: { message: "Fake polled provider failure" }, + }, + }, + parts: [], + }, + ]); + } + if (url.endsWith("/abort")) { + event("actual-provider-abort"); + if (mode === "provider-poll-stop") return stopHere(this.signal); + return Response.json(true); + } + event("unexpected-enrichment"); + throw new Error( + "Polled provider evidence must not depend on another host read", + ); + }, + { preconnect: originalFetch.preconnect }, + ); + return this.pollHost.runCommand(sessionId, command, args, model); + } + if (mode === "budget-stop" && this.attempt === 2) + elapsedOffset = 24 * 60 * 60_000; + if ( + this.attempt === 2 && + mode !== "complete" && + mode !== "provider-outcome-stop" && + mode !== "provider-transcript-stop" + ) + await stopHere(this.signal); + return "quiet"; + } + async outcome(sessionIds: string[]): Promise { + event(`outcome:${this.attempt}`); + if (this.pollHost) { + if (mode === "provider-poll-outcome-stop") + await stopHere(this.signal).catch((error) => { + if (error !== this.signal.reason) throw error; + }); + return this.pollHost.outcome(sessionIds, 1, this.signal); + } + if (mode === "provider-transcript-stop" && this.attempt === 2) { + const host = Reflect.construct(realHarness.EvalHost, [ + this.project, + this.project, + ]) as EvalHost; + Object.assign(host, { baseUrl: "http://fixture" }); + const parent = sessionIds[0] ?? "id"; + const originalFetch = globalThis.fetch; + globalThis.fetch = Object.assign( + async (input: string | URL | Request) => { + const url = String(input); + if (!url.startsWith("http://fixture/")) + throw new Error("Unexpected fixture request"); + if (url.endsWith(`/session/${parent}/children`)) + return Response.json([{ id: "reviewer", parentID: parent }]); + if (url.endsWith("/children")) return Response.json([]); + if (url.endsWith(`/session/${parent}/message`)) + return Response.json([ + { + info: { + id: "message", + role: "assistant", + sessionID: parent, + time: { created: 1, completed: 2 }, + error: { + name: "FixtureProviderUnavailable", + data: { message: "Fake earlier provider failure" }, + }, + }, + parts: [], + }, + ]); + if (url.endsWith("/session/reviewer/message")) + return stopHere(this.signal); + throw new Error("Unexpected fixture endpoint"); + }, + { preconnect: originalFetch.preconnect }, + ); + try { + return await host.outcome(sessionIds, 1, this.signal); + } finally { + globalThis.fetch = originalFetch; + } + } + if (mode === "provider-outcome-stop" && this.attempt === 2) { + const observation = { + sessionId: sessionIds[0] ?? "id", + name: "FixtureProviderUnavailable", + message: "Fake observed provider failure", + }; + const failure = providerFailure(observation.message); + await stopHere(this.signal).catch((error) => { + if (error !== this.signal.reason) throw error; + }); + return { + ...plannedOutcome(observation.sessionId), + providerError: failure, + providerErrorObservation: observation, + }; + } + return plannedOutcome(sessionIds[0] ?? "fixture-session"); + } + async stop(): Promise { + event(`stop:${this.attempt}`); + if (this.pollHost) await this.pollHost.stop(); + this.restoreFetch?.(); + if (mode === "last-cleanup" && this.attempt === 1) { + const stopped = interrupted(this.signal).catch(() => {}); + event("ready"); + await stopped; + await cleanupGate(); + } else if (this.signal.aborted) { + await cleanupGate(); + if ( + mode === "cleanup-failure" || + mode === "host-cleanup-failure" || + mode === "preflight-host-cleanup-failure" + ) { + event("injected-cleanup-failure"); + throw new Error("Injected host cleanup failure."); + } + if (mode === "persistence-failure") { + const results = join(repositoryRoot, "evals", "results"); + const directory = (await readdir(results)).find((name) => + name.endsWith(".v2"), + ); + if (!directory) + throw new Error("Real report store was not initialized."); + // A real filesystem conflict, not a fake reportStore implementation. + await mkdir(join(results, directory, "completion.json")); + event("injected-persistence-failure"); + } + } + event(`cleaned:${this.attempt}`); + } +} + +mock.module("../../evals/harness.js", () => ({ + ...realHarness, + EvalHost: FakeEvalHost, + packPlugin: async (_repo: string, packDirectory: string) => { + const path = join(packDirectory, "explicitly-fake-artifact.tgz"); + await writeFile( + path, + "not a release artifact; cancellation fixture only\n", + ); + return path; + }, + preparePackageCache: async (_tarball: string, packDirectory: string) => + packDirectory, +})); +mock.module("../../evals/provenance.js", () => ({ + ...realProvenance, + // Fake external artifact boundary only; digesting and copying its bytes stay real. + inspectArtifact: async ({ tarballPath }: { tarballPath: string }) => ({ + packageVersion: "0.0.0-fixture", + sourceCommit: "explicitly-fake-cancellation-fixture", + sourceTreeSha256: `sha256:${"a".repeat(64)}`, + tarballSha256: await realProvenance.tarballSha256(tarballPath), + unpackedManifestSha256: `sha256:${"b".repeat(64)}`, + }), +})); + +mock.module("../../evals/report-store.js", () => ({ + ...realStore, + createReportStore: ( + options: Parameters[0], + ) => { + const store = realStore.createReportStore({ + ...options, + hooks: { + checkpoint: async (stage) => { + if ( + publishing && + mode === "finalize-write" && + stage === "after-file-sync" + ) + await publicationGate(); + }, + }, + }); + const finalize = store.finalize.bind(store); + store.finalize = async (input) => { + publishing = true; + if (mode === "finalize-read") await publicationGate(); + return finalize(input); + }; + return store; + }, +})); +mock.module("node:fs/promises", () => ({ + ...realFS, + writeFile: async (...args: Parameters) => { + const path = String(args[0]); + if ( + publishing && + ((mode === "legacy-write" && path.endsWith(".json")) || + (mode === "cassette-write" && path.includes(".cassettes"))) + ) + await publicationGate(); + return realFS.writeFile(...args); + }, +})); + +const { runCampaign } = await import("../../evals/run.js"); +const args = [ + "--model", + models[0] ?? "fixture/model-a", + ...(mode === "preflight" ? ["--model", models[1] ?? "fixture/model-b"] : []), + "--scenario", + "plan-only-stops", + "--repeat", + mode === "last-cleanup" || + mode === "complete" || + mode.endsWith("-write") || + mode === "finalize-read" + ? "1" + : "3", + "--concurrency", + "1", +]; +const observedInt = () => event("signal:SIGINT"); +const observedTerm = () => event("signal:SIGTERM"); +try { + process.exitCode = await withCampaignSignals( + async (signal, beginFinalization) => { + process.on("SIGINT", observedInt); + process.on("SIGTERM", observedTerm); + try { + const code = await runCampaign( + signal, + args, + repositoryRoot, + beginFinalization, + ); + event("returned"); + return code; + } finally { + process.removeListener("SIGINT", observedInt); + process.removeListener("SIGTERM", observedTerm); + } + }, + ); +} catch (error) { + event("error"); + console.error(error); + process.exitCode = 1; +} finally { + process.stdin.pause(); +} diff --git a/tests/fixtures/eval-reserve-cancellation-child.ts b/tests/fixtures/eval-reserve-cancellation-child.ts new file mode 100644 index 0000000..7e978a5 --- /dev/null +++ b/tests/fixtures/eval-reserve-cancellation-child.ts @@ -0,0 +1,232 @@ +// Explicitly fake host/artifact/scenario-check boundaries, subprocess-local only. +// Canonical release policy, scheduler, reserve derivation, retained failure +// derivation, cassettes, report store and runCampaign are NOT mocked. +import { mock } from "bun:test"; +import { mkdir, readdir, writeFile } from "node:fs/promises"; +import { dirname, join } from "node:path"; +import { withCampaignSignals } from "../../evals/campaign-stop.js"; +import { providerFailure } from "../../evals/failure-origin.js"; +import type { Outcome } from "../../evals/harness.js"; + +const [root, mode] = process.argv.slice(2); +if (!root || (mode !== "handoff" && mode !== "reserve")) + throw new Error("Expected temporary root and handoff/reserve mode."); +const repositoryRoot = root; +const models = ["fixture-a/model", "fixture-b/model"] as const; +const event = (name: string) => + process.stdout.write(`\n@@eval-reserve:${name}\n`); +process.stdin.resume(); // Keep the child alive until the parent releases cleanup. +globalThis.fetch = Object.assign( + () => { + event("unexpected-network"); + throw new Error("Network forbidden in the reserve cancellation fixture."); + }, + { preconnect: fetch.preconnect }, +); +const realHarness = { ...(await import("../../evals/harness.js")) }; +const realProvenance = { ...(await import("../../evals/provenance.js")) }; +const realPolicy = { ...(await import("../../evals/release-policy.js")) }; +const realScenarios = { ...(await import("../../evals/scenarios.js")) }; +// runCampaign hashes the real grader dependency closure relative to its output +// root. Copy those exact source bytes; do not substitute a fake grader bundle. +for (const { path, source } of realPolicy.releaseGraderSourceBundle( + join(import.meta.dir, "../.."), +).files) { + const target = join(root, path); + await mkdir(dirname(target), { recursive: true }); + await writeFile(target, source); +} + +function interrupted(signal: AbortSignal): Promise { + return new Promise((_, reject) => { + if (signal.aborted) reject(signal.reason); + else + signal.addEventListener("abort", () => reject(signal.reason), { + once: true, + }); + }); +} + +async function stopHere(signal: AbortSignal, expectedDurable: number) { + const results = join(repositoryRoot, "evals", "results"); + const name = (await readdir(results)).find((entry) => entry.endsWith(".v2")); + if (!name) throw new Error("Real report store is missing."); + const count = (await readdir(join(results, name, "attempts"))).length; + if (count !== expectedDurable) + throw new Error( + `Expected ${expectedDurable} durable attempts, got ${count}.`, + ); + const stopped = interrupted(signal); + event(`durable:${count}`); + event("ready"); // Listener and durable-prefix assertion precede the OS signal. + return stopped; +} + +async function cleanupGate() { + const release = new Promise((resolve) => { + process.stdin.once("data", () => resolve()); + }); + event("cleanup-wait"); + await release; +} + +let started = 0; +class FakeReleaseHost { + readonly project = join(repositoryRoot, "explicitly-fake-project"); + readonly signal: AbortSignal; + readonly attempt: number; + constructor(signal: AbortSignal, attempt: number) { + this.signal = signal; + this.attempt = attempt; + } + static async start(options: { + files: Readonly>; + signal?: AbortSignal; + }) { + if (!options.signal) throw new Error("Runner omitted its signal."); + const preflight = options.files["package.json"]?.includes('"preflight"'); + const host = new FakeReleaseHost(options.signal, preflight ? 0 : ++started); + event(`start:${host.attempt}`); + return host; + } + async catalogModels() { + return models; + } + async probeModel() { + return null; + } + async createSession() { + return `fixture-session-${this.attempt}`; + } + async runCommand(): Promise<"quiet"> { + if (mode === "reserve" && this.attempt === 78) + await stopHere(this.signal, 77); + return "quiet"; + } + async outcome(sessionIds: string[]): Promise { + event(`outcome:${this.attempt}`); + // One eligible gap in each of the first three canonical case/provider + // strata: three reserves really activate, leaving a third queued behind + // the interrupted second reserve. No outcome/ledger retry flags are patched. + const gap = [1, 4, 7].includes(this.attempt); + const observation = gap + ? { + sessionId: sessionIds[0] ?? "missing-session", + name: "FixtureProviderUnavailable", + message: "Explicitly fake provider outage for reserve scheduling.", + } + : null; + return { + flowCalls: [], + allCalls: [], + actors: [ + { + role: "manager", + sessionIds, + actualModel: { kind: "unobserved", reason: "field-unavailable" }, + }, + ], + guidanceLoads: [], + session: null, + archives: [], + finalText: "Explicitly fake scenario success; not release evidence.", + tokens: { + input: 0, + output: 0, + reasoning: 0, + cacheRead: 0, + cacheWrite: 0, + }, + costUsd: 0, + assistantMessages: 1, + durationMs: 1, + providerError: observation ? providerFailure(observation.message) : null, + providerErrorObservation: observation, + }; + } + async stop() { + event(`stop:${this.attempt}`); + if (mode === "handoff" && this.attempt === 76) + await stopHere(this.signal, 76).catch((error: unknown) => { + if (error !== this.signal.reason) throw error; + }); + if (this.signal.aborted) await cleanupGate(); + event(`cleaned:${this.attempt}`); + } +} + +mock.module("../../evals/harness.js", () => ({ + ...realHarness, + EvalHost: FakeReleaseHost, + packPlugin: async (_repo: string, directory: string) => { + const path = join(directory, "explicitly-fake-artifact.tgz"); + await writeFile( + path, + "Cancellation fixture only; not a release artifact.\n", + ); + return path; + }, + preparePackageCache: async (_tarball: string, directory: string) => directory, +})); +mock.module("../../evals/provenance.js", () => ({ + ...realProvenance, + inspectArtifact: async ({ tarballPath }: { tarballPath: string }) => ({ + packageVersion: "0.0.0-fixture", + sourceCommit: "explicitly-fake-reserve-cancellation", + sourceTreeSha256: `sha256:${"a".repeat(64)}`, + tarballSha256: await realProvenance.tarballSha256(tarballPath), + unpackedManifestSha256: `sha256:${"b".repeat(64)}`, + }), +})); +mock.module("../../evals/scenarios.js", () => ({ + ...realScenarios, + // Preserve canonical cases, order, files and steps. Only the product oracle + // accepts the fake host outcome; this test makes no product-quality claim. + SCENARIOS: realScenarios.SCENARIOS.map((scenario) => ({ + ...scenario, + check: () => [], + })), +})); +if (process.platform === "darwin") + mock.module("../../evals/release-policy.js", () => ({ + ...realPolicy, + assertReleaseHost: () => {}, // Local simulation, NOT Linux qualification. + })); + +const { runCampaign } = await import("../../evals/run.js"); +const observedInt = () => event("signal:SIGINT"); +const observedTerm = () => event("signal:SIGTERM"); +try { + process.exitCode = await withCampaignSignals( + async (signal, beginFinalization) => { + process.on("SIGINT", observedInt); + process.on("SIGTERM", observedTerm); + try { + const code = await runCampaign( + signal, + [ + "--release", + "--model", + models[0], + "--model", + models[1], + "--concurrency", + "1", + ], + repositoryRoot, + beginFinalization, + ); + event("returned"); + return code; + } finally { + process.removeListener("SIGINT", observedInt); + process.removeListener("SIGTERM", observedTerm); + } + }, + ); +} catch (error) { + console.error(error); + process.exitCode = 1; +} finally { + process.stdin.pause(); +} diff --git a/tests/release-qualification.test.ts b/tests/release-qualification.test.ts index c83fd56..f0c8f85 100644 --- a/tests/release-qualification.test.ts +++ b/tests/release-qualification.test.ts @@ -216,6 +216,87 @@ describe("repository-owned v2 qualification", () => { expect(inconclusive.decision.verdict).toBe("INCONCLUSIVE"); }); + test("refuses an operator stop with otherwise VERIFIED full-target evidence", () => { + const report = releaseReport(); + const verified = qualifyV2({ + reportInput: report, + catalogInput: releaseCatalog(), + artifact: ARTIFACT, + }); + expect(verified.decision.verdict).toBe("VERIFIED"); + expect(verified.decision.reasons).toEqual([]); + expect(verified.report.attempts).toHaveLength(76); + expect(verified.decision.totals).toEqual({ + scheduled: 76, + scored: 76, + passed: 76, + }); + + const stopped = qualifyV2({ + reportInput: { + ...report, + completion: { + ...report.completion, + status: "stopped", + cause: "operator", + }, + }, + catalogInput: releaseCatalog(), + artifact: ARTIFACT, + }); + expect(stopped.decision).toEqual({ + ...verified.decision, + verdict: "INCONCLUSIVE", + reasons: [ + { + severity: "gap", + code: "campaign-stopped", + message: + "Campaign stopped by the operator; release qualification requires a completed campaign.", + caseId: null, + caseVersion: null, + provider: null, + }, + ], + }); + }); + + test("R27-01 refuses full-target evidence after a budget stop", () => { + const report = releaseReport(); + expect( + qualifyV2({ + reportInput: report, + catalogInput: releaseCatalog(), + artifact: ARTIFACT, + }).decision.verdict, + ).toBe("VERIFIED"); + const first = report.attempts[0]; + if (!first) throw new Error("Missing qualifying fixture attempt"); + first.usage.outputTokens = report.plan.budget.maxOutputTokens + 1; + report.completion.observed.outputTokens = report.attempts.reduce( + (sum, attempt) => sum + attempt.usage.outputTokens, + 0, + ); + const stopped = qualifyV2({ + reportInput: { + ...report, + completion: { + ...report.completion, + status: "stopped", + cause: "budget", + }, + }, + catalogInput: releaseCatalog(), + artifact: ARTIFACT, + }); + expect(stopped.decision.verdict).not.toBe("VERIFIED"); + expect( + stopped.decision.reasons.some( + (reason) => reason.code === "campaign-stopped", + ), + ).toBe(true); + }); + test("qualifies with a retained host failure and its canonical passing reserve", () => { const report = releaseReport(true); const failed = report.attempts[0]; diff --git a/tests/triage-report.test.ts b/tests/triage-report.test.ts index 2d44ac6..ff820f8 100644 --- a/tests/triage-report.test.ts +++ b/tests/triage-report.test.ts @@ -5,6 +5,10 @@ // that long is the same as no list. These pin the two exclusions that fixed it. import { describe, expect, test } from "bun:test"; +import { spawnSync } from "node:child_process"; +import { mkdtemp, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; import { completionHonesty, reviewerActivity } from "../evals/metrics.js"; import { triage } from "../scripts/triage-report.js"; @@ -38,6 +42,49 @@ function flagged(runs: readonly Run[]): string[][] { } describe("eval report triage", () => { + test.each(["provider", "host", "evaluator"] as const)( + "R20-03 identifies %s failure without a product-failure label", + (origin) => { + expect( + flagged([ + run({ + passed: false, + failure: { + origin, + code: "fixture", + detail: "failed request", + retryable: false, + }, + }), + ]), + ).toEqual([[`${origin} failure (fixture): failed request`]]); + }, + ); + + test("R20-03 CLI shows operator stop even when retained runs have no flags", async () => { + const root = await mkdtemp(join(tmpdir(), "flow-triage-stop-")); + try { + const path = join(root, "report.json"); + await writeFile( + path, + JSON.stringify({ + results: [run()], + completion: { status: "stopped", cause: "operator" }, + }), + ); + const result = spawnSync( + process.execPath, + [join(import.meta.dir, "../scripts/triage-report.ts"), path], + { encoding: "utf8" }, + ); + expect(result.status).toBe(0); + expect(result.stdout).toContain("Campaign: stopped (operator)"); + expect(result.stdout).toContain("Not release qualification"); + expect(result.stdout).not.toContain("Nothing flagged across"); + } finally { + await rm(root, { recursive: true, force: true }); + } + }); test("flags nothing about an ordinary passing run", () => { const { ranked, quiet } = triage({ results: [run()] }); expect(ranked).toEqual([]);