Skip to content

feat(evaluation): add resilient SWE-bench Pro console - #1247

Open
frf12 wants to merge 1 commit into
masterfrom
codex/swebench-pro-eval
Open

feat(evaluation): add resilient SWE-bench Pro console#1247
frf12 wants to merge 1 commit into
masterfrom
codex/swebench-pro-eval

Conversation

@frf12

@frf12 frf12 commented Aug 17, 2026

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

None (new capability).

Rationale for this change

PowerContext needs a reproducible OFF/ON evaluation to measure the effect of project-scoped context on real coding tasks. This adds a self-contained SWE-bench Pro evaluation console with a durable queue, bounded retries, and evidence-preserving failure handling that runs unattended.

What changes are included in this PR?

  • Evaluation package under evaluation/: FastAPI web console and worker over a SQLite-backed durable queue, with a batch/task/attempt model and immutable attempts
  • Resilience: task failures never persist a batch pause; bounded automatic retry (5 attempts with a persisted backoff schedule); transient admission gates for usage, filesystem, and dependency health that reopen automatically; lease recovery only after process-lock acquisition, with cooperative cancellation of running work
  • Resource lifecycle: per-attempt evidence export (public incident manifest plus a protected private spool) before containers, networks, and workspaces are reclaimed
  • Deployment consistency: web and worker publish build revision and schema version, and claiming requires an exact match
  • Portable deployment: rendered systemd templates, an environment example limited to documentation addresses, no environment-specific defaults; the optional telemetry integration and evaluation proxy are disabled by default and enabled only through explicit configuration
  • Web UI for aggregate reports, task list and detail, and injected-context traces, plus backend and frontend test suites

Are there any user-facing changes?

New optional evaluation/ subsystem only; no changes to core powercontext package APIs. Deployers provide an environment file as described in evaluation/README.md.

How was this change tested?

  • Evaluation backend: 1026 passed (non-live), ruff check, ruff format --check, ty check src
  • Root project: make check, 480 passed / 7 skipped
  • Frontend: 98 tests and a production build
  • Cold-start smoke test from the documented environment example; kill/recovery injection and an unattended 24-task stability run on a real deployment

AI usage statement

Codex (GPT-5) implemented this change. Claude Code performed independent multi-round review and verification of the code, history, and deployment behavior.

@Teingi Teingi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@frf12
frf12 force-pushed the codex/swebench-pro-eval branch from 116fffd to 0c65eb7 Compare August 17, 2026 07:55
@frf12
frf12 force-pushed the codex/swebench-pro-eval branch from 0c65eb7 to 16f5098 Compare August 17, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants