Skip to content

fix: preserve eval evidence on graceful cancellation - #62

Merged
vriesd merged 1 commit into
mainfrom
fix/eval-cancellation-reporting
Sep 5, 2026
Merged

fix: preserve eval evidence on graceful cancellation#62
vriesd merged 1 commit into
mainfrom
fix/eval-cancellation-reporting

Conversation

@vriesd

@vriesd vriesd commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Handle SIGINT/SIGTERM across startup, requests, scheduling, primary/reserve attempts, and host cleanup; stop accepting cancellation before immutable report publication.
  • Preserve observed provider failures and completed evidence without inventing interrupted product outcomes. Share once-only cleanup, confirm process termination, and retain recoverable credential scratch on cleanup failure.
  • Keep operator/budget stops nonqualifying, expose completion and failure origins in reports/triage, and document the strict frozen-candidate release sequence.
  • Add isolated real-signal regressions with fake provider boundaries; no model calls or real credentials are used by these tests.

Validation

  • Commit and clean-worktree push preflights passed. Optional gitleaks scanning was unavailable because gitleaks is not installed.
  • bun run check: 790 passed, 1 skipped; typecheck, lint, build and release metadata checks passed.
  • bun run replay: 13/13 cassettes matched.
  • Prior Linux focused cancellation/qualification suite on the same implementation: 175 passed.
  • Independent Flow implementation and final documentation reviews passed; both sessions are closed.

Limits And Release Status

  • Failure snapshots can omit earlier/child-session token and cost usage. This remains explicitly documented partial accounting, not complete billing proof.
  • Session v5, public tool inputs, package version (8.2.1), and exact-artifact qualification requirements are unchanged.
  • Paid evals remain stopped. This PR contains no campaign evidence, credentials, Flow state, tag, or publication. A fresh campaign and exact-artifact canary still require separate approval after candidate validation and freezing.

Drain signal-driven shutdown and credential cleanup without losing observed failures or inventing interrupted product results. Keep operator and budget stops nonqualifying, and document the frozen-candidate release sequence.

Add deterministic real-signal and reserve-scheduling regressions. Validation: bun run check (790 pass, 1 skip), Linux focused suite (175 pass), replay (13/13), and commit preflight. Failure usage snapshots remain partial observations, not complete billing totals.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 5, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-05T08:31:23.742186Z 2fb4bc3 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@vriesd
vriesd merged commit 8a935de into main Sep 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants