diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index f292f95..56e8223 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -65,10 +65,16 @@ jobs: with: filter: blob:none fetch-depth: 0 + - name: Set up Node.js + uses: actions/setup-node@v6 + with: + node-version: 22 - name: Install uv uses: astral-sh/setup-uv@v7 with: python-version: ${{ matrix.env.python }} + - name: Install agent CLIs + run: npm install --global @anthropic-ai/claude-code @openai/codex - name: create hatch environment run: uvx hatch env create ${{ matrix.env.name }} - name: list all all installed package versions diff --git a/.gitignore b/.gitignore index eea4877..fc17263 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,10 @@ __pycache__/ # Distribution / packaging /dist/ +# uv's resolved dependency set. acumen is a library — consumers resolve their own tree, and +# CI resolves fresh through hatch — so a committed lock would pin nothing that is checked. +/uv.lock + # Tests and coverage /data/ /node_modules/ diff --git a/.vscode/settings.json b/.vscode/settings.json index e034b91..7c09f6d 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -15,4 +15,5 @@ "python.analysis.typeCheckingMode": "basic", "python.testing.pytestEnabled": true, "python.testing.pytestArgs": ["-vv", "--color=yes"], + "cursorpyright.analysis.typeCheckingMode": "basic", } diff --git a/CHANGELOG.md b/CHANGELOG.md index 6700dde..aca4e2f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,3 +9,69 @@ and this project adheres to [Semantic Versioning][]. [semantic versioning]: https://semver.org/ ## [0.0.1dev] + +### Added + +- Run Claude and Codex models side by side in benchmark matrices and use either + provider for drafting, improving, task generation, and shipping. +- Compute `cost_usd` from each run's token breakdown rather than the provider's own + figure, so both providers are priced by one arithmetic path and cached input is billed + at its own rate. The rates used are frozen into `result.json`. +- Add `acumen prices` to show the rate table and `acumen prices --refresh` to diff it + against the providers' published pricing, plus a `prices:` config key to override it. +- Ship no rate table at all: rates are read from the providers' pricing pages on every + command that prices something, since a table compiled into a release is wrong from + whatever date prices next move, and each run's cost is frozen when written rather than + corrected later. `prices:` in `config.yaml` still overrides, and still wins. +- Fail `acumen bench` when the pricing pages cannot be read, before anything is spent: + cost is a headline metric of the report, so a pass that cannot establish rates should + not run. `draft`, `improve`, `tasks`, and `ship` degrade to unpriced instead, warning + that Codex's `max_usd` cannot be enforced without rates. +- Record `price_source` and `price_rates_as_of` next to `price_rates` in every + `result.json`, so runs benchmarked months apart remain individually attributable and a + single report can mix them without restating either. +- Flag arms in a report that were priced on different dates: the cost difference between + them includes any change in provider pricing, not only the skill's effect. +- Render an HTML transcript for Codex runs too, from the `codex exec` event stream. +- Run the whole comparison from one `acumen bench`: with no arm selected it now covers + every arm the project has — the baseline plus each version in `skills/` — benching them + one after another against a single prepared target, with per-arm counts and tallies and a + combined total. `--dry-run` plans the same set for free, and `--no-skill` / `--skill vN` + still restrict the pass to one arm. A version in `skills/` that fails to load stops the + pass at planning rather than being dropped from the comparison. + +### Changed + +- Make both backends optional, so a Claude-only and a Codex-only install are each complete: + the Claude Agent SDK moves to the `claude` extra (`pip install acumen[claude]`, or + `acumen[all]`) and Codex needs only its CLI on `PATH`. Selecting a model whose backend is + not installed fails preflight with the command that installs it. +- Enforce `max_turns` and `max_usd` for Codex, which has no cap of its own, from its event + stream. Turns are counted in completed model actions rather than `codex exec` invocations — + one invocation is a single Codex turn, so the old count was always 1 — and the run is + stopped at the cap. `max_usd` can only mark the outcome: Codex reports usage when a turn + ends, so an over-budget run is recorded as a `budget` failure after the spend. + +- Allow `acumen bench --auth {auto,session,api}`, defaulting to the provider's subscription + like every other command. The old API-only rule existed because a per-run `cost_usd` needed + metered billing; cost is now derived from token counts, which a subscription run reports + just as fully. `result.json` records the run's `auth_mode`, since under `session` the figure + is what the run would have cost at API rates rather than metered spend. +- Treat exhausted provider subscription usage and API credit as benchmark-invalid + infrastructure failures: print the provider error, cancel only that provider's remaining + cells while other providers finish, exit non-zero, and refuse to report or improve from the + invalid evidence. Resuming automatically retries the invalid and cancelled cells after the + credential is replenished. + +### Fixed + +- Let `acumen tasks` generate over the untouched `tasks.yaml` placeholder that `acumen init` + writes, instead of demanding `--force` — the two documented first steps of the loop + contradicted each other. A file the user has edited is still protected. +- Stop feeding Codex transcripts to `claude-code-log`, which reads the SDK-native format + only: it skipped every line, exited 0, and wrote an empty page that was then recorded as a + successfully rendered transcript. +- Drop `Claude` from the drafting and improving prompts, which described the artifact as a + "Claude Skill" even when a Codex agent was writing it for a non-Claude skills directory. +- Remove `check_auth`/`auth_available`, which only ever looked for Claude credentials and + would report a Codex-only setup as unauthenticated. `resolve_auth_mode` replaced them. diff --git a/README.md b/README.md index cecf89e..54cf871 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,7 @@ acumen draft # generate skills/v1 from the package source, o acumen bench --skill v1 # benchmark the skill against the baseline acumen improve # generate skills/v2 from v1's train results, or write by hand acumen bench --skill v2 +acumen bench # or: every arm at once (baseline + each skills/vN) acumen report # aggregate every run into report.html # 4. Once a version proves out, ship it into the package itself: @@ -67,6 +68,95 @@ train/test isolation, and for `draft`/`improve` it is recorded in the version's shown in the report. (Don't paste held-out test answers into `improve` feedback — that would defeat the split.) +Claude and Codex can run side by side. Put both model families in `models` to compare them +in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`, +`o3`, `o4`, and `codex-*` use Codex: + +```yaml +models: + - claude-opus-5 + - claude-sonnet-5 + - claude-haiku-4-5-20251001 + - gpt-5.6-sol + - gpt-5.6-terra + - gpt-5.6-luna +``` + +This spans each provider's quality/cost range; it is not a claim that the tiers are +one-to-one equivalents. + +Neither backend is required. Claude is an optional dependency and Codex is an external CLI, +so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the +`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the +install command, before acumen prepares a target or spends anything. + +Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or +`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through their +`*_model` config keys or `--model`. + +Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to +the provider's logged-in subscription, falling back to its API key. Both billing modes report +tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that +estimate is what the run *would* have cost at API rates, not money billed — so each run records +its `auth_mode` alongside the figure. + +If the selected subscription runs out of usage or the API account runs out of credit, Acumen +invalidates the pass instead of scoring that as an agent failure: it prints the provider error, +cancels remaining cells for that provider, lets other providers finish all running and queued +cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume +retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit +evidence. + +`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex, +which has no cap of its own — acumen enforces both against its event stream: + +- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work + happens inside it, so turns are counted in completed model actions (a message, a command, a + file change, a tool or search call) and the agent is stopped at the cap. +- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only + visible after the money is spent. The run is recorded as a budget failure — the same outcome + Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass. + +**Reports compare inferred cost per run.** Every run records its token breakdown — fresh +input, cache reads, cache writes, and output — and Acumen prices it with the rate table +stored in `result.json`. That gives Claude and Codex one comparable basis and prevents an old +benchmark from being silently re-priced. The result itself retains both +`provider_cost_usd` (when the backend supplies one) and `inferred_cost_usd`; its compatibility +field `cost_usd` remains provider-first. The report's sidecar CSV calls the former +`recorded_cost_usd` and keeps it separate from `inferred_cost_usd`, while every displayed cost +and comparison uses the inferred value. + +**Rates are read from the providers' pricing pages, never shipped with the package.** Prices +move, and each run's cost is frozen into its `result.json` and never recomputed, so a table +compiled into a release would store numbers that were already wrong. `bench` resolves rates +before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline +metric, and a benchmark that cannot establish rates has not earned the numbers it would print. +`draft`, `improve`, `tasks`, and `ship` fetch too but degrade to unpriced instead — their cost +line is progress reporting, not stored evidence. + +Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and +`price_rates_as_of`, so a pass run in August and another in October stay individually +attributable and one report can cover both without restating either. When arms in a report were +priced on different dates, the report says so: the cost gap between them includes the price +change, not only the skill's effect. + +```bash +acumen prices # the rates in use today, and where each came from +acumen prices --refresh # check pinned rates against what the providers publish +``` + +Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish, +to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you +know what you are billed. They are also the only rates that can drift unnoticed, which is what +`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because +picking the wrong tier or context band would silently misprice future runs. A model no layer +prices records its tokens and leaves report cost unavailable — never zero, which would read as +free. + +> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an +> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`, +> or pin the rates. + `draft`, `improve`, `tasks`, and `ship` each drive a long autonomous agent. Every run writes a live `logs/acumen--.jsonl` (one event per step, flushed as it goes — so you can watch progress by reading the file) and a rendered `.html` transcript. Add `--stream` to @@ -82,6 +172,15 @@ in particular, the [API documentation][]. You need to have Python 3.12 or newer installed on your system. If you don't have Python installed, we recommend installing [uv][]. +Install the backend you actually run — both are optional, and either alone is a complete +install: + +| you run | install | also needs | +|---|---|---| +| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | +| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | +| both | `pip install acumen[all]` | both of the above | +