Skip to content

Reconcile the documentation with the tree, and unbreak CI - #1

Merged
Ethan-Arrowood merged 6 commits into
mainfrom
docs-reconcile
Oct 6, 2026
Merged

Ethan-Arrowood merged 6 commits into
mainfrom
docs-reconcile

Conversation

@Ethan-Arrowood

Copy link
Copy Markdown
Member

We pushed to the public repo at a decent point in the implementation, so I did a pass over every document in it to check what still matched reality. Four things, in order of how much they cost us.

CI has been red since the repo went public

The conformance job asserted Harper's pinned version by reading /usr/local/lib/node_modules/harper/package.json. That is npm's default prefix, not a guarantee — the runner resolves elsewhere, so every run failed at the version check with MODULE_NOT_FOUND, which reads as "Harper is missing" rather than "we looked in the wrong place". It now asks npm root -g.

No document knew this. docs/plan.md still listed the GitHub repo as a blocker and CI as "written, never executed".

The harness README claimed a guard the harness does not have

It listed Correctness guards under what it implements, on the grounds that the quote is deterministic. Determinism is necessary and is checked, but it is not a ground-truth check, and docs/plan.md has had this right all along: a consistently wrong answer is deterministic too. For a repo whose whole discipline is not claiming what it cannot support, this was the one that mattered. Moved to the not-implemented list with what closing it takes.

Three other bullets listed gaps that have since been built — the containerized path, trial restore, and the cold/warm/hot split all exist in the orchestrator. Replaced with the gap actually left: one target means "containerized identically for every target" is vacuous rather than satisfied.

Also dropped its ladder table, which described a dataset an order of magnitude smaller than the current one and a capacity an order of magnitude higher. It was a second copy of numbers whose first copy moved.

The plan was four days and five commits behind

AGENTS.md says to keep it current. It reported the conformance suite as mostly stubs after the stubs were gone, listed filling it as the next task, called FRESH_MS "deliberately TBD" after SPEC.md set it provisionally, and carried loyaltyBalance as unapplied after QUOTE-004 was rewritten around it.

One item was backwards rather than merely stale, and the corrected version is a real finding: the product endpoint omits the compute phase OBS-001 requires of both endpoints, and the coverage gate passes anyway because the quote's test also names OBS-001. That is the gate satisfied by the letter rather than the requirement, which is the failure mode the gate exists to prevent.

Related, and also now recorded: scripts/verify-run-record.mjs declares coverage of the run-shaped requirements and nothing executes it — not npm run check, not CI, not run-benchmark.sh. The gate scans its COVERS: line statically, so those requirements currently pass on a comment. The split is right; the wiring is missing.

Remaining drift

  • README told every newcomer to regenerate the committed dataset in place as step one of the quickstart — two sections above the line explaining that implementations do not re-derive it, in a repo whose CONTRIBUTING calls regenerating it a heavy change.
  • docs/data-model.md explained at length why promotions carry ['*'] sentinel key arrays, then listed a Promotion type without them, with @indexed still on the authoritative arrays that are deliberately not indexed. It also omitted @sealed throughout and omitted ProductView entirely — the cache is the one piece of the Harper design SPEC.md does not describe, so this document was the only place it was going to be described, and it wasn't. The block now matches schemas/store.graphql field for field.
  • docs/structure.md called the repo root the Next.js app root, and asserted measurement scaffolding "does not live here at all" with bench/ and containers/ sitting beside it.
  • docs/future-work.md said the Next.js wiring was retained but inert; structure.md said it was removed. Removed is right.

On counts

I took tallies out of the docs — requirements, tests, stubs, checks, row totals — and pointed each at a source that cannot drift: SPEC.md and the registry, npm run check, each dataset's MANIFEST.json. Every one of those numbers was wrong within days of being written, and the dev row count had drifted silently when the generator was fixed.

Measurements keep their numbers: the ladder latencies, the 2.5% → 77.7% cache-hit finding, the scrypt memory table. Those are dated experimental records, not status.

Two things I did not decide

  • SPEC.md §8 has Self-review unticked but Cross-model ticked, which is backwards in sequence. The plan used to claim self-review had passed; the two disagreed in both directions. Gate status now lives only in SPEC.md §8, and I left the boxes exactly as they were rather than adjudicate.
  • The CI fix is unverifiable until it runs, so the plan says "not yet confirmed green by a run" rather than claiming green. This PR is what tests that.

npm run check passes.

🤖 Generated with Claude Code

Ethan-Arrowood and others added 6 commits October 6, 2026 08:55
The conformance job asserted the pinned version by reading
/usr/local/lib/node_modules/harper/package.json. That path is npm's default
prefix, not a guarantee: the runner resolves somewhere else, so every run since
the repo went public has failed at the version check with MODULE_NOT_FOUND —
which reads as "Harper is missing" rather than "we looked in the wrong place".

`npm root -g` is the question we actually meant to ask.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It listed "Correctness guards" under what it implements, on the grounds that the
quote is deterministic. Determinism is necessary and is checked, but it is not a
ground-truth check and this file was the one place in the repo saying otherwise
— docs/plan.md has had it right all along: a consistently wrong answer is
deterministic too. Moved to the not-implemented list, with what closing it takes.

Three other gaps were listed as missing but have since been built: the
containerized path, trial restore, and the cold/warm/hot split all exist in the
orchestrator. Replaced with the gap that is actually left — one target means
"identical for every target" is vacuous, not satisfied — and a note that running
run.mjs directly gets you none of it.

Dropped the ladder table. It described a dataset an order of magnitude smaller
than the current one and a capacity an order of magnitude higher, because it was
a second copy of numbers whose first copy moved. Observations now live only in
docs/plan.md, dated and next to what may not be claimed from them. Row counts go
the same way: MANIFEST.json is the contract, and a second copy of a contract is
just a second thing to be wrong.

Recorded the third time the generator was the bottleneck.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
AGENTS.md says to keep this file current. It was four days and five commits
behind: it reported the conformance suite as mostly stubs after the stubs were
eliminated, listed filling it as the next task, called FRESH_MS "deliberately
TBD" after SPEC.md set it provisionally, carried loyaltyBalance as unapplied
after QUOTE-004 was rewritten around it, and said CI had never run while CI was
failing on every push since the repo went public.

One open item was not merely stale but backwards: it said the quote endpoint
emits only `total` while the product endpoint decomposes. It is the other way
round, and the real finding is sharper — the product aggregate omits the compute
phase that OBS-001 requires of both endpoints, and the gate passes anyway
because the quote's test also names OBS-001. Recorded as what it is: the
coverage gate satisfied by the letter rather than the requirement.

Also recorded: scripts/verify-run-record.mjs declares coverage of the
run-shaped requirements and nothing executes it, so those requirements
currently pass on a comment.

Review-gate status was tracked both here and in SPEC.md §8, and the two
disagreed in both directions. SPEC.md §8 is now the only place it lives.

Counts are gone — of requirements, of tests, of stubs, of rows. Every one of
them was wrong within days of being written, and every one has a source that
cannot drift. Measurements keep their numbers; tallies do not.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
README said the conformance suite was incomplete after it was completed, stated
a requirement count that had moved, and opened its quickstart by telling every
newcomer to regenerate the committed dataset in place — two sections above the
line explaining that implementations do not re-derive it, and in a repo whose
CONTRIBUTING calls regenerating it a heavy change. The dataset is committed;
there is nothing to generate.

data-model.md's schema block had drifted out of agreement with its own prose:
it explained at length why promotions carry ['*'] sentinel key arrays, then
listed a Promotion type without them, with @indexed still on the authoritative
arrays that are deliberately not indexed. It also omitted @Sealed throughout and
omitted ProductView entirely — the cache is the one piece of the Harper design
SPEC.md does not describe, so this document is the only place it was going to be
described, and it wasn't. The block now matches schemas/store.graphql field for
field.

structure.md called the repo root the Next.js app root, and asserted that
measurement scaffolding "does not live here at all" with bench/ and containers/
sitting beside it. Both corrected; the second kept as a recorded decision with
the reason it changed, since the original intent still holds and the location is
the part that moved. Its layout block had also fallen behind the tree.

future-work.md said the Next.js wiring was retained but inert; structure.md said
it was removed. Removed is right — there is no next.config.ts and nothing in
package.json.

CONTRIBUTING counted the checks and got a number that is no longer true. It now
says what they check instead.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Several documents explained a design by narrating the thing it replaced: "an
earlier version drew keys uniformly", "that single fact broke this twice", "the
first version scheduled every arrival up front". A reader who was not here for
that learns a sequence of states and has to work out which one is current.

Each is now stated as the constraint it is. The uniform-draw story becomes why
the access distribution is skewed; the two broken promotion lookups become the
two obvious implementations and why neither works; the harness's three memory
failures become the three properties that keep the generator out of the
measurement. Nothing substantive is dropped — the reasoning is the point, and it
reads better without the chronology.

data-model.md no longer opens by relitigating a reversed decision: resolve-on-
read is argued on its own merits, with the condition that would change it.

structure.md no longer justifies the e2e exclusion by a Next.js version that is
no longer in the tree. Optional peer dependencies defeating --omit=dev is the
durable fact, and it holds for any framework that declares one.

future-work.md described deferred scope as things that "was" — now it says what
each is and what decision applies when it returns, which is what the file is
actually for.

plan.md drops its Done list. The status table above it already states what state
every piece is in, and a changelog beside a status table is two places to
disagree. Git has the history.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both jobs green on the pull request: the repo checks and the conformance suite
against the dev dataset.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Ethan-Arrowood
Ethan-Arrowood merged commit ec0603b into main Oct 6, 2026
2 checks passed
@Ethan-Arrowood
Ethan-Arrowood deleted the docs-reconcile branch October 6, 2026 18:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant