Answer where the cursor is, without reparsing the file

[samuelduchesne]

Adds a syntax layer to @idfkit/core and a language service in a new opt-in package, @idfkit/language. Both are synchronous and free of input and output, so the same code runs in Node, a browser, a browser worker and behind an editor server.

Blocked on idfkit-conformance#2. This branch pins governance-2026.11, which does not exist on main there yet. Every governance-reading job here fails to resolve the ref until that PR merges and the tag is cut. Merge order is conformance, then this.

The observation the design turns on

IDF is a flat sequence of statements terminated by a semicolon, with no nesting, no string literals and no escapes, and a comment runs from an exclamation mark to the end of its line. So the statement containing an offset is found by scanning backwards to the nearest semicolon not inside a comment, at a cost proportional to the statement rather than to the file. That is what lets a cursor be answered with no reparse, no incremental parser, no cache and therefore no hidden state.

Two cost classes rather than one. Classifying the text and positioning findings are whole-file work, run when the document settles. Asking what completes here, what this means and what this points at are bounded local work, run on a keystroke. contextAt does not import scanIdf.

Measured, not hoped

bench/corpus.mjs generates 10,001 statements and 40,002 lines from a fixed seed. bench/budget.mjs gates on ratios measured inside one run rather than on wall-clock milliseconds, because a runner varies by more than the margin being defended and a gate that fails randomly is a gate somebody disables.

gate	measured	budget
contextAt p95 / parseIdf median	0.032%	2%
completionsAt p95 / parseIdf median	0.035%	2%
scanIdf median / parseIdf median	0.49x	1.25x
parseIdf median / lex median	2.78x	3.6x
contextAt, 641,576 vs 6,860 bytes	1.16x	3x
contextAt, last vs first statement, no comments	1.40x	3x

The last two pin the design, on different axes. A ratio against parseIdf could be met by a merely fast reparse; independence from file size could not. But size independence compares two files and cannot see a cost that grows with the offset, because both readings sit at the same place in their own file.

That second axis is here because it caught a real defect, not a hypothetical one. insideComment searched backwards for an exclamation mark, which reads the whole prefix in a file that holds none, so every answer's cost grew with how far into the file the cursor sat. Measured at 8,385x between the two ends of an 800 KB comment-free file. The reference model could not show it, because a comment on nearly every line stops that search within one line, and all six other gates stayed green throughout. Both gates are demonstrated to fail on purpose.

One scanner, two modes

The character rules lived in two near-copies, in lex and in the internal fieldLine helper. Both are now callers of parse/scan.ts, which stays internal. If the layer and the reader disagreed by one character about where a comment ends, findings would land on the wrong field, and that failure is invisible until a file puts a comment somewhere unusual. A corpus test asserts the fields scanIdf reports are positionally identical to the values lex reports.

The tiling invariant is what the corpus test checks, not byte-identical reconstruction: the layer holds the source text, so reconstruction returns the text by construction and a test of it proves nothing. A failure names which of the six clauses broke and at which token index.

No token yielded by classify crosses a line boundary, because no editor token encoding can express one that does. A stored value region can, since the format lets a field be written across two lines and real files do it; classify splits at the newline and moves no boundary.

Not modified

validateDocument, parseIdf and IdfDocument. Correlation happens afterwards over the syntax layer, so a caller who never asks for a position receives exactly what it received before. Committed snapshots over the corpus assert that, which is what keeps positions additive rather than merely intended.

Two facts correlation rests on are asserted in their own tests rather than trusted: addRaw throws on a duplicate name so a parsed document never holds two objects sharing a folded type and name, and IdfCollection preserves insertion order.

Why a sixth package

Arithmetic rather than taste. Everything in @idfkit/core is installed by everyone who runs npm install idfkit, so shipping the service there charges every model-reading install for an editor they do not have. It is an optional peer reached as idfkit/language, on the mechanism @idfkit/weather already established: a dynamic import caught behind a top-level await, because a static export * links before any local code runs and the guard would never execute.

Measured cost, and it is not what the plan predicted. The plan expected the layer to emit roughly 30 KB. It adds 61,493 bytes, taking the install from 1,734,364 to 1,795,857 against a budget of 1,835,008. The gate passes with no amendment, which is what SC-015 asked, but headroom falls from 98.3 KB to 38.2 KB. The budget is not moved to make it pass, and the next addition to core will have this conversation rather than this one.

Review fixes carried in the second commit

• nameKey joined on a NUL written as an escaped backslash inside a template literal, so the separator was six printable characters that a name may perfectly well contain. Two objects bracketing that literal would have shared a key, and a finding about one would have underlined the other, silently.
• publish.yml never learned about the new package: absent from the publish loop, its core peer placeholder never rewritten, and the facade's peerDependencies.@idfkit/language left at 0.0.0. A release would have published nothing under that name and shipped a facade asking for a version that has never existed.
• clean-install.mjs redirected every scoped name to a local tarball except the new one, so every distribution gate resolved @idfkit/language from the real registry. It passed only because npm skips an unresolvable optional peer in silence; under --offline or --strict-peer-deps the gates would exit 2.
• .gitattributes marks the syntax fixtures -text. Three differ only in line endings and the matrix runs windows-latest, where core.autocrlf would have collapsed them into one while the tests kept passing against text no longer containing the case they were written for.
• check-facade.mjs still said four subpaths after ./language made it five, so a real breach would have reported the wrong number.

One open decision for the reviewer

publish.yml rewrites @idfkit/language's core peer to a caret, where contracts/language-service.md asks for an exact range. The contract's reasoning is right and the range does not deliver it: the facade depends on core with a caret, so core's first patch release makes an exact peer unsatisfiable for anyone installing the facade. Exact buys no safety there and breaks every install one release later. Tightening it means pinning the facade's own core dependency in the same change, which is a decision about the repository rather than about this package. The reasoning is recorded where the rewrite happens.

Gates

typecheck, test (699 tests, 31 files), format:check, all 11 check:* scripts, and 8 budget gates are green locally.

check:publication refuses, on a precondition that predates this branch: the conformance pin is conformance-2026.8 while feature 004's FR-044 names conformance-2026.7 as the level both libraries publish at. All six distribution preconditions pass. That question is feature 004's to settle and gates the shared name rather than this feature.