Answer where the cursor is, without reparsing the file

.gitattributes

@@ -0,0 +1,10 @@
+# The syntax fixture corpus is bytes, not text.
+#
+# `packages/core/tests/fixtures/syntax/` holds three files that differ from one another only in
+# their line endings, and the CI matrix runs the suite on windows-latest, where git's default
+# `core.autocrlf=true` rewrites every LF on checkout. Under that rewrite `line-endings-lf.idf` and
+# `line-endings-mixed.idf` both arrive as CRLF, the three fixtures become one fixture, and the tests
+# written for the distinction keep passing while measuring nothing.
+#
+# `-text` disables all end-of-line conversion for these paths in both directions.
+packages/core/tests/fixtures/syntax/*.idf -text

.github/workflows/main.yml

@@ -229,6 +229,50 @@ jobs:
 
       - run: npm test
 
+  budget:
+    name: The performance budget
+    runs-on: ubuntu-latest
+
+    # ITS OWN JOB, and deliberately not a step inside `test`. The language service exists so that
+    # an answer about the cursor costs the statement rather than the file, and a requirement with
+    # no measurement behind it is a hope: bench/budget.mjs is the measurement, and it fails.
+    #
+    # Why it is not in the test suite. A timing threshold wired into the same run as the
+    # correctness tests means a noisy neighbour on the runner blocks a documentation typo, and a
+    # gate that fails work it has nothing to do with is a gate somebody switches off within the
+    # month. Here a breach reports against itself and names the ratio that moved.
+    #
+    # Why it can fail on a shared runner at all. Every enforced number is a RATIO between two
+    # figures measured in the same process on the same machine in the same run: a cursor answer
+    # against parseIdf over the same text, scanIdf against parseIdf, parseIdf against lex, the
+    # same cursor answer on the reference model against a file one hundredth its size, and the
+    # same answer at the two ends of one comment-free file. A slower runner moves both halves
+    # together and the ratio does not move. Absolute milliseconds are printed for a human to read
+    # and nothing is held against them.
+    #
+    # The last two are the ones that pin the design, and they pin different axes. A ratio against
+    # parseIdf could be met by a merely fast reparse; independence from file SIZE could not, and a
+    # reparse shows a hundredfold difference on any machine. But size independence compares two
+    # files and cannot see a cost that grows with the OFFSET, because both of its readings sit at
+    # the same place in their own file. That is a real defect rather than a hypothetical one: a
+    # backward search for an exclamation mark reads the whole prefix in a file that holds none,
+    # and it measured 925x here while every other gate on this list stayed green.
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+
+      - run: npm ci
+
+      # After the build: the benchmark reads dist, which is what an npm consumer receives, rather
+      # than the sources a bundler would transform.
+      - run: npx tsc --build
+
+      - run: npm run check:bench
+
   docs:
     name: Docs
     runs-on: ubuntu-latest

.github/workflows/publish.yml

@@ -133,6 +133,28 @@ jobs:
             npm pkg set "peerDependencies.@idfkit/core=>=$VERSION" -w "$pkg"
           done
 
+          # @idfkit/language peer-depends on core and carries the "0.0.0" placeholder in the
+          # repository for the same reason the type packages do. Published unchanged it would ask
+          # the registry for @idfkit/core@0.0.0, a version that has never existed, and every
+          # install of the language service would fail to resolve its peer.
+          #
+          # EXACT, not a caret, and contracts/language-service.md is explicit about why: the
+          # service reads the syntax layer core builds, so a pair that disagrees about the layer's
+          # shape puts findings on the wrong characters SILENTLY. Nothing fails; the underline is
+          # simply in the wrong place. A caret would permit exactly that pairing across a core
+          # patch release, and the whole argument for splitting the two packages rests on there
+          # being no supported install in which they differ.
+          #
+          # This sits beside, and does not contradict, the facade's caret on core a few lines
+          # below, which contracts/distribution.md requires so that a patch of core reaches a
+          # consumer without a facade release. npm resolves peers as part of building the tree,
+          # so `idfkit`'s "^X.Y.Z" and this "X.Y.Z" are satisfied together by X.Y.Z, and the
+          # narrow case where they cannot be is one where some other dependency has already
+          # forced core above the version this service was built against. Refusing to resolve is
+          # the right answer there: it is the pairing the exact range exists to reject, and a
+          # failed install is better than an underline in the wrong place.
+          npm pkg set "peerDependencies.@idfkit/core=$VERSION" -w @idfkit/language
+
           # The facade's own dependency specs, which `npm version` does NOT touch: it rewrites
           # the "version" field of each workspace and nothing else. In the repository they read
           # "0.0.0", which is what makes the workspace link resolve. Published unchanged, they
@@ -145,6 +167,7 @@ jobs:
           npm pkg set "dependencies.@idfkit/core=^$VERSION" -w idfkit
           npm pkg set "dependencies.@idfkit/schemas=^$VERSION" -w idfkit
           npm pkg set "peerDependencies.@idfkit/weather=^$VERSION" -w idfkit
+          npm pkg set "peerDependencies.@idfkit/language=^$VERSION" -w idfkit
 
       - name: Publish
         if: env.PUBLISH == 'true'
@@ -157,7 +180,8 @@ jobs:
           # already-published package, so a release that failed partway can
           # never be completed by re-running it.
           # schemas first: core depends on it by exact version. weather has no
-          # workspace dependencies, so its position does not matter. The type packages go last:
+          # workspace dependencies, so its position does not matter. language peer-depends on
+          # core and so follows it. The type packages go last:
           # their peer range resolves against a core that is on the registry by then, so a
           # consumer installing one never sees an unsatisfiable peer.
           # npm applies the `latest` dist-tag on every publish unless told otherwise, and it
@@ -172,7 +196,7 @@ jobs:
           esac
           echo "Publishing $VERSION under the '$NPM_TAG' dist-tag."
 
-          for pkg in @idfkit/schemas @idfkit/core @idfkit/weather \
+          for pkg in @idfkit/schemas @idfkit/core @idfkit/weather @idfkit/language \
                      @idfkit/types-v26-1 @idfkit/types-v9-4; do
             if npm view "$pkg@$VERSION" version >/dev/null 2>&1; then
               echo "$pkg@$VERSION is already published; skipping."
@@ -241,6 +265,7 @@ jobs:
           npm pkg set "dependencies.@idfkit/core=^$VERSION" -w idfkit
           npm pkg set "dependencies.@idfkit/schemas=^$VERSION" -w idfkit
           npm pkg set "peerDependencies.@idfkit/weather=^$VERSION" -w idfkit
+          npm pkg set "peerDependencies.@idfkit/language=^$VERSION" -w idfkit
           # npm applies the `latest` dist-tag on every publish unless told otherwise, and it
           # does so for a semver prerelease exactly as for a stable version: nothing about the
           # `-rc.1` in a version reaches the registry's idea of what `npm install <pkg>` should

CHANGELOG.md

@@ -10,6 +10,113 @@ The packages in this repository, `@idfkit/core`, `@idfkit/schemas`, and
 
 ## [Unreleased]
 
+This release stays on `conformance-2026.8`, the corpus level `CONFORMANCE_LEVEL`
+reports, and adds no cases to it. A capability that exists in one language
+asserts no cross-language agreement, so there is nothing for the corpus to
+compare; the parity ledger carries the absence instead. The governance level
+moves to `governance-2026.11`, which is where the names below are registered and
+where that ledger entry lives.
+
+### Added
+
+- A syntax layer in `@idfkit/core`. `scanIdf(text)` returns a `SyntaxLayer`:
+  every statement, with the region of its type name and of each field it was
+  written with, plus every meaningful token in source order, packed. It takes
+  text and nothing else, because the layer records what the text says and never
+  what it means, and it never throws for any input. Text that breaks the grammar
+  is represented rather than stopped at: an unterminated final statement runs to
+  the end of the input and says so, a statement written with no type name still
+  gets a region, and empty text produces an empty layer.
+
+  Nothing builds it implicitly. `parseIdf` and `lex` read the same characters
+  through the same scan and construct none of it, so **a caller who never names
+  `scanIdf` pays neither its time nor its memory**, and reading a file costs what
+  it did before.
+
+  `classify(layer)` is that layer read for drawing: the stored tokens with the
+  gaps between them filled as `trivia`, so the sequence tiles the whole text with
+  no hole and no overlap, and no token crosses a line boundary, because no token
+  encoding in use can express one that does. It is a generator, so a consumer
+  colouring a viewport stops where it stops rather than materialising every token
+  in the file. `lineColumnAt` and `offsetAt` convert between an offset and a
+  1-based line and column.
+
+- `@idfkit/language`, the opt-in language service for IDF text, reachable under
+  the shared name at `idfkit/language`. It answers a cursor and positions
+  findings, and does nothing else:
+
+  - `contextAt(text, offset, schema?)` reports what the cursor is on: the
+    statement it is in, whether the offset falls on the type name, in a field,
+    inside a comment, or between statements, and which field it is.
+  - `completionsAt`, `explainAt`, and `declarationAt` answer what may be written
+    here, what this means in the schema's own words, and where the name under the
+    cursor is declared. Each returns a discriminated union rather than a list
+    that is sometimes empty, because "the schema permits anything here" and "I
+    could not consult a schema" are different states, and an editor that rendered
+    them alike would look broken in the first case and be silently wrong in the
+    second.
+  - `findingsIn(text, schema)` reads, validates, and gives every finding a region
+    plus the precision of that region, `field` or `statement`.
+    `position(findings, layer, schema)` does the same for findings a caller
+    already holds, so a consumer with its own parse pays for one scan rather than
+    a second read. No validator changed and no finding is filtered or reworded:
+    correlation is a separate step over the layer, and a caller that never asks
+    for a position receives exactly what it received before.
+
+  Everything here is synchronous, free of input and output, and holds no state.
+  There is no service object to construct, because a service object is where
+  state would accumulate, and every answer takes the text itself rather than a
+  path, since an editor's buffer differs from the file on disk whenever there are
+  unsaved changes. The same code therefore runs unchanged in Node, in a browser,
+  in a browser worker, and behind an editor server. Nothing here imports or names
+  a type from any editor protocol, and nothing here ever will; a consumer
+  translates.
+
+  An answer costs the statement rather than the file. The statement containing an
+  offset is found by scanning backwards to the nearest semicolon that is not
+  inside a comment, so there is no reparse, no incremental parser, and no cache.
+  A committed measurement under `bench/` holds that to ratios rather than to
+  milliseconds, which is the form that survives being run on someone else's
+  machine: a cursor answer at most 2 percent of `parseIdf` over the same text,
+  and `scanIdf` at most 1.25 times it.
+
+  **It is not installed by default.** `npm install idfkit` places zero bytes of
+  the service on disk, exactly as it places no weather code. Add
+  `@idfkit/language` by name to get it, and importing `idfkit/language` without
+  it names the package to install rather than failing with
+  `ERR_MODULE_NOT_FOUND`.
+
+  There is no Python counterpart, and there is not going to be one. The answers
+  are computed from byte offsets into the source text, and a second
+  implementation of that arithmetic is the drift the corpus is least able to
+  police, since it compares findings on `(code, line, typeName)` and never on a
+  column. The decision is on the parity ledger as `idf-language-service`, at
+  `never`, which is terminal: adding a counterpart takes a constitutional
+  amendment rather than an edit. What it costs a reader is stated rather than
+  implied. These answers need a JavaScript runtime, and `pip install idfkit`
+  alone does not provide them.
+
+### Changed
+
+- `ParseDiagnostic.column` is filled. It was declared in 0.2.0-rc.2 and left
+  undefined because the lexer counted lines and not columns; the shared scan
+  counts both, and a reading finding now reports the column its statement begins
+  at, taken at the first non-blank character and counted from 1. The field is
+  still optional, so nothing that treated it as absent breaks.
+
+  A column is the one location the two libraries measure in different units, and
+  that is registered rather than left to be found: Python counts code points and
+  JavaScript counts UTF-16 code units, so the two agree everywhere except in text
+  containing an astral character, which in practice means an emoji in a comment.
+  Each unit is the one its own ecosystem's editors want, so neither is converted.
+  Nothing compares columns across the two libraries today; the corpus matches
+  findings on `(code, line, typeName)`.
+
+- `lex` and `parseIdf` read through the same scan the syntax layer is built from.
+  Both keep their surface and their behaviour; there is now one scanner rather
+  than two, which is what keeps a position the layer reports and a position a
+  finding reports from drifting apart.
+
 ## [0.2.0-rc.2] - 2026-09-04
 
 ### Added
@@ -51,11 +158,14 @@ The packages in this repository, `@idfkit/core`, `@idfkit/schemas`, and
   on `message`: the corpus compares findings on `(code, line, typeName)` and
   never on wording.
 
-  `column` and `filepath` are declared but not yet filled: the lexer counts
-  lines and not columns, and `parseIdf` takes text rather than a path, so
-  neither value exists at the point a finding is built. They are optional, so a
-  reader must treat them as absent until the lexer tracks a column and the
-  file-reading edge attaches the path it read from.
+  Both are optional, and each is filled at the one place that knows the value.
+  `parseIdf` takes text and cannot know where the text came from, so `filepath`
+  is stamped by the Node file-reading edge: `loadIdf` and
+  `loadIdfWithDiagnostics` attach the path they read to every finding, on the
+  result and on the error alike. A caller parsing a string still gets none,
+  because a string names no file. `column` is filled by the reader itself, from
+  the statement's first non-blank character, once the shared scan arrived to
+  count it (see Unreleased).
 
 ### Fixed