Skip to content

Add Claude Opus: decide the cadence and the budget it needs #16

Description

@leggetter

From Gareth, on the first draft:

How much would adding Opus add? That's probably a lot of people's go-to so having
that in the list would be good even if it's similar to Sonnet.

The reasoning is sound: Opus is a common default, and a scoreboard without it invites
"but how does Opus do?" from every reader.

Opus has to run twice, like every other model: once with Hookdeck's skills and
once without, or it cannot appear in the skills comparison the page is built around
(#17). So the unit of cost is a pair of passes, not one.

The figures

A Sonnet 5 pass over fifteen scenarios is about $22, measured. Opus 5 is $5/$25
per million against Sonnet's $3/$15, so an Opus pass is roughly $37, or $55 if
the Sonnet figure was measured during its introductory pricing.

With and without skills, one cycle: $73 to $110.

Cadence Opus adds New monthly total
Weekly +$318 to $477 $503 to $662
Fortnightly +$159 to $238 $344 to $423
Monthly +$73 to $110 $258 to $295
Quarterly +$24 to $37 $209 to $222

Current spend is $185 a month against a $200 ceiling.

Every cadence breaks the ceiling, including quarterly. An earlier version of this
issue said monthly "fits the budget". That was wrong: $185 plus $73 is $258.

So something has to give

  1. Raise the ceiling to about $300 and run Opus monthly. The simplest option, and
    a decision rather than an engineering problem. Weekly is indefensible at $500 plus;
    quarterly is too stale to be worth a column.
  2. Move the existing weekly to fortnightly, freeing roughly $90 a month, and fund
    Opus monthly from it. Lands near $250. Costs regression-detection speed on the
    models we already track.
  3. Drop scenarios. Seven of fifteen pass every experiment. They are a floor and
    worth keeping for that reason, but they are also the cheapest thing to cut. Decide which models and scenarios earn a place in the weekly run #5.

The page already renders a per-row timestamp, so a model on a slower cadence shows an
older date rather than silently implying it was run this week.

Blocked on #14

Adding the most expensive model in the matrix while single-attempt results are
unreliable buys an extra column of numbers we cannot trust. Loop 1 closed as a
negative result for exactly this reason.

Done when Opus is in the matrix at an agreed cadence with the budget to match, or
we have recorded why not.

Metadata

Metadata

Assignees

No one assigned

    Labels

    harnessRunner, scorers, provisioner and CI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions