Ledger for the fixed first-visit probe task (~/autoqa/scripts/run-stumble.mjs, blind agent, browser only, STUMBLE_URL = the Pages landing page). The number is a trend only while the task text never moves; the task is verbatim from QA.md § Stumble probe tasks, item 2 (the uncomfortable one).
Task: "Decide whether this tool would beat simply pasting git diff into your model, and find the number on this site that settles it. Say what that number was measured on."
2026-08-30 — median 4 steps, 3/3 runs completed, none stuck
Runs: 4, 5, 4 steps. No run failed the task; every run found the hero stat within a step or two, which is the page working as intended.
The finding is not the step count — it is that 4 of 6 distinct gripe slugs across three independent runs were the same defect:
other:unsourced-benchmark-number, other:unsourced-headline-stat, missing-filter — the 48,210 → 6,930 (7×) stat had no repo, no PR, no dataset, no footnote.other:baseline-mismatch — it compares against changed files pasted whole, while the reader arrived asking about pasting a git diff.naming-collision — the on-page demo produces 4,970 tokens / 11 fragments, numbers that look like they should be the same example as the hero and are not reconciled.jargon — "same review" and "denser signal" asserted with no definition of how review equivalence was established.
Verified against the repository: the figures appeared nowhere in it — not in docs/, not in the paper, not in any benchmark artifact. The single number meant to justify adoption could not be checked by anyone, including us.
Resolved in fd9dc14, same pass: the stat now reads 768,268 → 43,769 (17.6×), measured on home-assistant/core@1d885bd0 (66 files changed) at the default auto budget, with diffctx . --diff 1d885bd0^..1d885bd0 printed beside it and the tokenizer named. The caption also states what the old copy implied and could not support: the ratio is not a constant, it tracks how much of the changed files is unrelated to the change — a three-file fix often gets more tokens, because diffctx adds the callers and types the diff alone does not show. TestLandingPageClaims gates the provenance (a commit link, the command, the tokenizer), deliberately not the values, so the next engine change cannot make the test the thing that gets edited.
Left open for the next batch, because they are page-copy judgment calls rather than defects:
jargon — "same review" still asserts an equivalence nothing on the page establishes. Either define it or drop the phrase.naming-collision — the demo's numbers and the hero's still sit on one page without a sentence reconciling them.
Next batch alternates to task 1 (entry).
Ledger for the fixed first-visit probe task (
~/autoqa/scripts/run-stumble.mjs, blind agent, browser only,STUMBLE_URL= the Pages landing page). The number is a trend only while the task text never moves; the task is verbatim fromQA.md§ Stumble probe tasks, item 2 (the uncomfortable one).Task: "Decide whether this tool would beat simply pasting
git diffinto your model, and find the number on this site that settles it. Say what that number was measured on."2026-08-30 — median 4 steps, 3/3 runs completed, none stuck
Runs: 4, 5, 4 steps. No run failed the task; every run found the hero stat within a step or two, which is the page working as intended.
The finding is not the step count — it is that 4 of 6 distinct gripe slugs across three independent runs were the same defect:
other:unsourced-benchmark-number,other:unsourced-headline-stat,missing-filter— the48,210 → 6,930 (7×)stat had no repo, no PR, no dataset, no footnote.other:baseline-mismatch— it compares against changed files pasted whole, while the reader arrived asking about pasting a git diff.naming-collision— the on-page demo produces 4,970 tokens / 11 fragments, numbers that look like they should be the same example as the hero and are not reconciled.jargon— "same review" and "denser signal" asserted with no definition of how review equivalence was established.Verified against the repository: the figures appeared nowhere in it — not in
docs/, not in the paper, not in any benchmark artifact. The single number meant to justify adoption could not be checked by anyone, including us.Resolved in fd9dc14, same pass: the stat now reads 768,268 → 43,769 (17.6×), measured on home-assistant/core@1d885bd0 (66 files changed) at the default auto budget, with
diffctx . --diff 1d885bd0^..1d885bd0printed beside it and the tokenizer named. The caption also states what the old copy implied and could not support: the ratio is not a constant, it tracks how much of the changed files is unrelated to the change — a three-file fix often gets more tokens, because diffctx adds the callers and types the diff alone does not show.TestLandingPageClaimsgates the provenance (a commit link, the command, the tokenizer), deliberately not the values, so the next engine change cannot make the test the thing that gets edited.Left open for the next batch, because they are page-copy judgment calls rather than defects:
jargon— "same review" still asserts an equivalence nothing on the page establishes. Either define it or drop the phrase.naming-collision— the demo's numbers and the hero's still sit on one page without a sentence reconciling them.Next batch alternates to task 1 (entry).