Repository navigation
docs(grading): compare Sonnet 5.5 and Haiku 5.5 with Opus 5.5 as first grader - #133
Merged
Merged
Conversation
Anthropic prices Claude Haiku 5.5 by prompt length and lists it in two rows, which rates.py refused as ambiguous. The catalog holds short-context prices, so the parser now skips a row for prompts over a token count and reads the other. The new entry is the row for prompts up to 100,000 tokens, read on 2026-10-12; its source text gives the higher prices. Metering still prices a session from one entry, so a Haiku total is a lower bound when a request's prompt passes 100,000 tokens. bench/rates.json, which grade.py reads for a Claude grader, is unchanged.
… on the trial's twelve batches The owner asked how much the grader's model matters. Claude Sonnet 5.5 and Claude Haiku 5.5, at high effort, labelled the 493 claims that Opus 5.5 and Codex GPT-6.1 Sol labelled in the trial's rerun on the final text. Each session was prepared from the checkout and scratch records of that rerun, and dispatched only after its key and prompt hash equalled the saved session's. The record gives label, first-fact and credit agreement beside the saved Opus against Sol figures, every credit difference, ten differences read against the rule text, contract health, and cost and time at list price. It recommends no grader. Nothing was mapped and no live record changed. Part of #30
…r-model-test * origin/main: feat(grading): take the review-code reviews into the published selection and drop three setups (#132)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Before authorizing the regrade of 198 batches, the owner asked how much the grader's model matters: "Let's do a small batch test with Opus, Sonnet, and Haiku. All 5.5 high", on the trial's twelve batches, reusing the saved Opus 5.5 and Sol answers. This records that test. The owner then chose Opus 5.5 for the regrade.
Scope
docs/research/grader-model-test-2026-10-12/: the method, the scripts (pinned_prepare.py,run.py,compare.py,usage.py), the proof that each batch's inputs equal what the saved Opus session pinned (inputs.json), the 24 saved results and one failed attempt,comparison.json,usage.v1.jsonand the plan-meter readings.bench/rates.current.json: the Claude Haiku 5.5 rate (the short-prompt row; the page lists a second row above 100,000 prompt tokens).bench/tools/rates.py: the parser skips a model's over-N-tokens row, which had made the model "ambiguous"; one test.Result on 493 claims and 430 review and known-problem pairs:
Tradeoffs
bench/rates.jsonis not changed: the rates tool treats it as immutable, and no Haiku grading is planned.Blast Radius
A research record, one rate entry in the current-rates file and a parser rule for a pricing-page row. No live grading record changed;
grades.jsonstill holds no batch. No pinned grading file changed.Verification
compare.pyreproduces the trial's published Opus-against-Sol figures and stops if it does not (run).test_rates.py,rates.py check --model claude-haiku-5-5,bun run verify:import,bun run verify:current: pass.Part of #30