You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
How much would adding Opus add? That's probably a lot of people's go-to so having
that in the list would be good even if it's similar to Sonnet.
The reasoning is sound: Opus is a common default, and a scoreboard without it invites
"but how does Opus do?" from every reader.
Opus has to run twice, like every other model: once with Hookdeck's skills and
once without, or it cannot appear in the skills comparison the page is built around
(#17). So the unit of cost is a pair of passes, not one.
The figures
A Sonnet 5 pass over fifteen scenarios is about $22, measured. Opus 5 is $5/$25
per million against Sonnet's $3/$15, so an Opus pass is roughly $37, or $55 if
the Sonnet figure was measured during its introductory pricing.
With and without skills, one cycle: $73 to $110.
Cadence
Opus adds
New monthly total
Weekly
+$318 to $477
$503 to $662
Fortnightly
+$159 to $238
$344 to $423
Monthly
+$73 to $110
$258 to $295
Quarterly
+$24 to $37
$209 to $222
Current spend is $185 a month against a $200 ceiling.
Every cadence breaks the ceiling, including quarterly. An earlier version of this
issue said monthly "fits the budget". That was wrong: $185 plus $73 is $258.
So something has to give
Raise the ceiling to about $300 and run Opus monthly. The simplest option, and
a decision rather than an engineering problem. Weekly is indefensible at $500 plus;
quarterly is too stale to be worth a column.
Move the existing weekly to fortnightly, freeing roughly $90 a month, and fund
Opus monthly from it. Lands near $250. Costs regression-detection speed on the
models we already track.
Adding the most expensive model in the matrix while single-attempt results are
unreliable buys an extra column of numbers we cannot trust. Loop 1 closed as a
negative result for exactly this reason.
Done when Opus is in the matrix at an agreed cadence with the budget to match, or
we have recorded why not.
From Gareth, on the first draft:
The reasoning is sound: Opus is a common default, and a scoreboard without it invites
"but how does Opus do?" from every reader.
Opus has to run twice, like every other model: once with Hookdeck's skills and
once without, or it cannot appear in the skills comparison the page is built around
(#17). So the unit of cost is a pair of passes, not one.
The figures
A Sonnet 5 pass over fifteen scenarios is about $22, measured. Opus 5 is $5/$25
per million against Sonnet's $3/$15, so an Opus pass is roughly $37, or $55 if
the Sonnet figure was measured during its introductory pricing.
With and without skills, one cycle: $73 to $110.
Current spend is $185 a month against a $200 ceiling.
Every cadence breaks the ceiling, including quarterly. An earlier version of this
issue said monthly "fits the budget". That was wrong: $185 plus $73 is $258.
So something has to give
a decision rather than an engineering problem. Weekly is indefensible at $500 plus;
quarterly is too stale to be worth a column.
Opus monthly from it. Lands near $250. Costs regression-detection speed on the
models we already track.
worth keeping for that reason, but they are also the cheapest thing to cut. Decide which models and scenarios earn a place in the weekly run #5.
The page already renders a per-row timestamp, so a model on a slower cadence shows an
older date rather than silently implying it was run this week.
Blocked on #14
Adding the most expensive model in the matrix while single-attempt results are
unreliable buys an extra column of numbers we cannot trust. Loop 1 closed as a
negative result for exactly this reason.
Done when Opus is in the matrix at an agreed cadence with the budget to match, or
we have recorded why not.