Deferred until a clean full matrix existed. That precondition is now met: the
13 August run is the first with no contaminated rows.
Open question 3 in the delivery plan: three models cost about $50 a week, five
about $150. The current cadence is weekly for the frontier agents plus the weak
pair and monthly for the -no-skills twins, at roughly $185 a month against a
$200 budget.
What the first clean run says, which was not knowable before:
- Seven of fifteen scenarios pass every experiment. That is a floor and worth
keeping, but it is also seven scenarios' worth of spend producing no signal
each week.
- The
-no-skills twins now do produce signal, unlike when they were cut to
monthly: the delta is +1, 0 and -3 depending on the model, not zero.
- The weak model is where most failures live and is the cheapest arm to run.
Supersedes the reasoning in #1, which was written before any of this was
measurable.
Done when the cadence in .github/workflows/eval-refresh.yml reflects a
decision made against this data rather than against the estimates it was
originally set from.
Deferred until a clean full matrix existed. That precondition is now met: the
13 August run is the first with no contaminated rows.
Open question 3 in the delivery plan: three models cost about $50 a week, five
about $150. The current cadence is weekly for the frontier agents plus the weak
pair and monthly for the
-no-skillstwins, at roughly $185 a month against a$200 budget.
What the first clean run says, which was not knowable before:
keeping, but it is also seven scenarios' worth of spend producing no signal
each week.
-no-skillstwins now do produce signal, unlike when they were cut tomonthly: the delta is +1, 0 and -3 depending on the model, not zero.
Supersedes the reasoning in #1, which was written before any of this was
measurable.
Done when the cadence in
.github/workflows/eval-refresh.ymlreflects adecision made against this data rather than against the estimates it was
originally set from.