Add codex and change cost calc

.github/workflows/test.yaml

@@ -65,10 +65,16 @@ jobs:
         with:
           filter: blob:none
           fetch-depth: 0
+      - name: Set up Node.js
+        uses: actions/setup-node@v6
+        with:
+          node-version: 22
       - name: Install uv
         uses: astral-sh/setup-uv@v7
         with:
           python-version: ${{ matrix.env.python }}
+      - name: Install agent CLIs
+        run: npm install --global @anthropic-ai/claude-code @openai/codex
       - name: create hatch environment
         run: uvx hatch env create ${{ matrix.env.name }}
       - name: list all all installed package versions

.gitignore

@@ -11,6 +11,10 @@ __pycache__/
 # Distribution / packaging
 /dist/
 
+# uv's resolved dependency set. acumen is a library — consumers resolve their own tree, and
+# CI resolves fresh through hatch — so a committed lock would pin nothing that is checked.
+/uv.lock
+
 # Tests and coverage
 /data/
 /node_modules/

.vscode/settings.json

@@ -15,4 +15,5 @@
     "python.analysis.typeCheckingMode": "basic",
     "python.testing.pytestEnabled": true,
     "python.testing.pytestArgs": ["-vv", "--color=yes"],
+    "cursorpyright.analysis.typeCheckingMode": "basic",
 }

CHANGELOG.md

@@ -9,3 +9,69 @@ and this project adheres to [Semantic Versioning][].
 [semantic versioning]: https://semver.org/
 
 ## [0.0.1dev]
+
+### Added
+
+- Run Claude and Codex models side by side in benchmark matrices and use either
+  provider for drafting, improving, task generation, and shipping.
+- Compute `cost_usd` from each run's token breakdown rather than the provider's own
+  figure, so both providers are priced by one arithmetic path and cached input is billed
+  at its own rate. The rates used are frozen into `result.json`.
+- Add `acumen prices` to show the rate table and `acumen prices --refresh` to diff it
+  against the providers' published pricing, plus a `prices:` config key to override it.
+- Ship no rate table at all: rates are read from the providers' pricing pages on every
+  command that prices something, since a table compiled into a release is wrong from
+  whatever date prices next move, and each run's cost is frozen when written rather than
+  corrected later. `prices:` in `config.yaml` still overrides, and still wins.
+- Fail `acumen bench` when the pricing pages cannot be read, before anything is spent:
+  cost is a headline metric of the report, so a pass that cannot establish rates should
+  not run. `draft`, `improve`, `tasks`, and `ship` degrade to unpriced instead, warning
+  that Codex's `max_usd` cannot be enforced without rates.
+- Record `price_source` and `price_rates_as_of` next to `price_rates` in every
+  `result.json`, so runs benchmarked months apart remain individually attributable and a
+  single report can mix them without restating either.
+- Flag arms in a report that were priced on different dates: the cost difference between
+  them includes any change in provider pricing, not only the skill's effect.
+- Render an HTML transcript for Codex runs too, from the `codex exec` event stream.
+- Run the whole comparison from one `acumen bench`: with no arm selected it now covers
+  every arm the project has — the baseline plus each version in `skills/` — benching them
+  one after another against a single prepared target, with per-arm counts and tallies and a
+  combined total. `--dry-run` plans the same set for free, and `--no-skill` / `--skill vN`
+  still restrict the pass to one arm. A version in `skills/` that fails to load stops the
+  pass at planning rather than being dropped from the comparison.
+
+### Changed
+
+- Make both backends optional, so a Claude-only and a Codex-only install are each complete:
+  the Claude Agent SDK moves to the `claude` extra (`pip install acumen[claude]`, or
+  `acumen[all]`) and Codex needs only its CLI on `PATH`. Selecting a model whose backend is
+  not installed fails preflight with the command that installs it.
+- Enforce `max_turns` and `max_usd` for Codex, which has no cap of its own, from its event
+  stream. Turns are counted in completed model actions rather than `codex exec` invocations —
+  one invocation is a single Codex turn, so the old count was always 1 — and the run is
+  stopped at the cap. `max_usd` can only mark the outcome: Codex reports usage when a turn
+  ends, so an over-budget run is recorded as a `budget` failure after the spend.
+
+- Allow `acumen bench --auth {auto,session,api}`, defaulting to the provider's subscription
+  like every other command. The old API-only rule existed because a per-run `cost_usd` needed
+  metered billing; cost is now derived from token counts, which a subscription run reports
+  just as fully. `result.json` records the run's `auth_mode`, since under `session` the figure
+  is what the run would have cost at API rates rather than metered spend.
+- Treat exhausted provider subscription usage and API credit as benchmark-invalid
+  infrastructure failures: print the provider error, cancel only that provider's remaining
+  cells while other providers finish, exit non-zero, and refuse to report or improve from the
+  invalid evidence. Resuming automatically retries the invalid and cancelled cells after the
+  credential is replenished.
+
+### Fixed
+
+- Let `acumen tasks` generate over the untouched `tasks.yaml` placeholder that `acumen init`
+  writes, instead of demanding `--force` — the two documented first steps of the loop
+  contradicted each other. A file the user has edited is still protected.
+- Stop feeding Codex transcripts to `claude-code-log`, which reads the SDK-native format
+  only: it skipped every line, exited 0, and wrote an empty page that was then recorded as a
+  successfully rendered transcript.
+- Drop `Claude` from the drafting and improving prompts, which described the artifact as a
+  "Claude Skill" even when a Codex agent was writing it for a non-Claude skills directory.
+- Remove `check_auth`/`auth_available`, which only ever looked for Claude credentials and
+  would report a Codex-only setup as unauthenticated. `resolve_auth_mode` replaced them.

README.md

@@ -48,6 +48,7 @@ acumen draft                     # generate skills/v1 from the package source, o
 acumen bench --skill v1          # benchmark the skill against the baseline
 acumen improve                   # generate skills/v2 from v1's train results, or write by hand
 acumen bench --skill v2
+acumen bench                     # or: every arm at once (baseline + each skills/vN)
 acumen report                    # aggregate every run into report.html
 
 # 4. Once a version proves out, ship it into the package itself:
@@ -67,6 +68,95 @@ train/test isolation, and for `draft`/`improve` it is recorded in the version's
 shown in the report. (Don't paste held-out test answers into `improve` feedback — that would
 defeat the split.)
 
+Claude and Codex can run side by side. Put both model families in `models` to compare them
+in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`,
+`o3`, `o4`, and `codex-*` use Codex:
+
+```yaml
+models:
+  - claude-opus-5
+  - claude-sonnet-5
+  - claude-haiku-4-5-20251001
+  - gpt-5.6-sol
+  - gpt-5.6-terra
+  - gpt-5.6-luna
+```
+
+This spans each provider's quality/cost range; it is not a claim that the tiers are
+one-to-one equivalents.
+
+Neither backend is required. Claude is an optional dependency and Codex is an external CLI,
+so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the
+`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the
+install command, before acumen prepares a target or spends anything.
+
+Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or
+`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through their
+`*_model` config keys or `--model`.
+
+Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to
+the provider's logged-in subscription, falling back to its API key. Both billing modes report
+tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that
+estimate is what the run *would* have cost at API rates, not money billed — so each run records
+its `auth_mode` alongside the figure.
+
+If the selected subscription runs out of usage or the API account runs out of credit, Acumen
+invalidates the pass instead of scoring that as an agent failure: it prints the provider error,
+cancels remaining cells for that provider, lets other providers finish all running and queued
+cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume
+retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit
+evidence.
+
+`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex,
+which has no cap of its own — acumen enforces both against its event stream:
+
+- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work
+  happens inside it, so turns are counted in completed model actions (a message, a command, a
+  file change, a tool or search call) and the agent is stopped at the cap.
+- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only
+  visible after the money is spent. The run is recorded as a budget failure — the same outcome
+  Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass.
+
+**Reports compare inferred cost per run.** Every run records its token breakdown — fresh
+input, cache reads, cache writes, and output — and Acumen prices it with the rate table
+stored in `result.json`. That gives Claude and Codex one comparable basis and prevents an old
+benchmark from being silently re-priced. The result itself retains both
+`provider_cost_usd` (when the backend supplies one) and `inferred_cost_usd`; its compatibility
+field `cost_usd` remains provider-first. The report's sidecar CSV calls the former
+`recorded_cost_usd` and keeps it separate from `inferred_cost_usd`, while every displayed cost
+and comparison uses the inferred value.
+
+**Rates are read from the providers' pricing pages, never shipped with the package.** Prices
+move, and each run's cost is frozen into its `result.json` and never recomputed, so a table
+compiled into a release would store numbers that were already wrong. `bench` resolves rates
+before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline
+metric, and a benchmark that cannot establish rates has not earned the numbers it would print.
+`draft`, `improve`, `tasks`, and `ship` fetch too but degrade to unpriced instead — their cost
+line is progress reporting, not stored evidence.
+
+Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and
+`price_rates_as_of`, so a pass run in August and another in October stay individually
+attributable and one report can cover both without restating either. When arms in a report were
+priced on different dates, the report says so: the cost gap between them includes the price
+change, not only the skill's effect.
+
+```bash
+acumen prices              # the rates in use today, and where each came from
+acumen prices --refresh    # check pinned rates against what the providers publish
+```
+
+Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish,
+to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you
+know what you are billed. They are also the only rates that can drift unnoticed, which is what
+`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because
+picking the wrong tier or context band would silently misprice future runs. A model no layer
+prices records its tokens and leaves report cost unavailable — never zero, which would read as
+free.
+
+> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an
+> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`,
+> or pin the rates.
+
 `draft`, `improve`, `tasks`, and `ship` each drive a long autonomous agent. Every run writes a
 live `logs/acumen-<command>-<datetime>.jsonl` (one event per step, flushed as it goes — so you
 can watch progress by reading the file) and a rendered `.html` transcript. Add `--stream` to
@@ -82,6 +172,15 @@ in particular, the [API documentation][].
 You need to have Python 3.12 or newer installed on your system.
 If you don't have Python installed, we recommend installing [uv][].
 
+Install the backend you actually run — both are optional, and either alone is a complete
+install:
+
+| you run | install | also needs |
+|---|---|---|
+| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login |
+| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key |
+| both | `pip install acumen[all]` | both of the above |
+
 <!--
 1) Install the latest release of `acumen` from [PyPI][]:
 

docs/api.md

@@ -81,6 +81,7 @@ so `from acumen import build_report` works.
     :toctree: generated
 
     PlannedRun
+    BenchmarkInvalidError
     build_matrix
     pending
     run_matrix
@@ -133,7 +134,9 @@ conversion.
 
     LiveLog
     locate_transcript
+    render_agent_transcript
     render_transcript
+    render_codex_transcript
 ```
 
 ## Reporting

pyproject.toml

@@ -22,8 +22,6 @@ classifiers = [
   "Programming Language :: Python :: 3.14",
 ]
 dependencies = [
-  "claude-agent-sdk>=0.2.116",
-  "claude-code-log",
   "matplotlib",
   "numpy",
   "pandas",
@@ -33,12 +31,20 @@ dependencies = [
   # for debug logging (referenced from the issue template)
   "session-info2",
 ]
+# Backends are optional and independent: install the ones you actually run. Claude needs a
+# Python SDK (and claude-code-log to render its transcripts); Codex is an external CLI, so it
+# adds no Python dependency at all — `codex` on PATH is the whole requirement.
+optional-dependencies.all = [ "acumen[claude]" ]
+optional-dependencies.claude = [
+  "claude-agent-sdk>=0.2.116",
+  "claude-code-log",
+]
 # https://docs.pypi.org/project_metadata/#project-urls
 urls.Documentation = "https://acumen.readthedocs.io/"
 urls.Homepage = "https://github.com/scverse/acumen"
 urls.Source = "https://github.com/scverse/acumen"
 scripts.acumen = "acumen.cli:main"
-scripts."acumen-install-skills" = "acumen._skills.install:main"
+scripts.acumen-install-skills = "acumen._skills.install:main"
 
 [dependency-groups]
 dev = [
@@ -73,6 +79,9 @@ envs.docs.scripts.build = "sphinx-build -M html docs docs/_build -W {args}"
 envs.docs.scripts.clean = "git clean -fdX -- {args:docs}"
 envs.docs.scripts.open = "python -m webbrowser -t docs/_build/html/index.html"
 envs.docs.dependency-groups = [ "doc" ]
+# The test suite covers both backends, so it installs every optional one. The Codex-only and
+# Claude-only import paths are covered from within the suite, by hiding a backend at import time.
+envs.hatch-test.features = [ "all" ]
 envs.hatch-test.matrix = [
   # Test the lowest and highest supported Python versions with normal deps
   { deps = [ "stable" ], python = [ "3.12", "3.14" ] },

src/acumen/__init__.py

@@ -1,18 +1,26 @@
-"""Build, benchmark, and optimize Claude skills for Python packages."""
+"""Build, benchmark, and optimize agentic skills for Python packages."""
 
 from importlib.metadata import version
 
-from acumen.bench import PlannedRun, build_matrix, pending, run_matrix, summarize
+from acumen.agents import (
+    AgentError,
+    AgentOptions,
+    AgentProvider,
+    AgentResult,
+    check_agent_cli,
+    claude_sdk_available,
+    provider_for_model,
+    run_agent,
+)
+from acumen.bench import BenchmarkInvalidError, PlannedRun, build_matrix, pending, run_matrix, summarize
 from acumen.config import Config, ConfigError, load_config, parse_config
 from acumen.draft import DraftError, DraftResult, draft_skill
 from acumen.env import (
     AuthMode,
     EnvError,
     Target,
     api_auth_available,
-    auth_available,
     build_agent_env,
-    check_auth,
     prepare_target,
     resolve_auth_mode,
     scrubbed_env,
@@ -66,10 +74,20 @@
     make_skill_guard,
 )
 from acumen.tasks import Task, TaskError, TaskSplit, load_tasks, parse_tasks
-from acumen.transcript import locate_transcript, render_transcript
+from acumen.transcript import (
+    locate_transcript,
+    render_agent_transcript,
+    render_codex_transcript,
+    render_transcript,
+)
 
 __all__ = [
     "AuthMode",
+    "AgentError",
+    "AgentOptions",
+    "AgentProvider",
+    "AgentResult",
+    "BenchmarkInvalidError",
     "Config",
     "ConfigError",
     "DraftError",
@@ -104,9 +122,9 @@
     "api_auth_available",
     "arm_metrics",
     "arm_name",
-    "auth_available",
     "available_versions",
-    "check_auth",
+    "check_agent_cli",
+    "claude_sdk_available",
     "build_agent_env",
     "build_filtered_source",
     "build_matrix",
@@ -137,10 +155,14 @@
     "parse_tasks",
     "pending",
     "prepare_target",
+    "provider_for_model",
+    "render_agent_transcript",
+    "render_codex_transcript",
     "render_transcript",
     "resolve_auth_mode",
     "resolve_palette",
     "run_dir",
+    "run_agent",
     "run_matrix",
     "run_once",
     "sandbox",