diff --git a/README.md b/README.md index 34d2646..73d7a89 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,22 @@ A TypeScript-native port of the [Ruby runtime](https://github.com/interscript/in npm install interscript ``` +## Try it + +In a browser with no bundler (worker-safe, exact-version pin): + +```typescript +import { transliterateAsync } from "https://esm.sh/interscript@5.3.0" +``` + +Full recipe and evidence: [docs/CDN.md](docs/CDN.md). Editor +autocomplete for injected globals: `dist/playground.d.ts`. + +Runnable, CI-verified example scripts — convert, detect-and-chain, +neural diacritization, batch columns, server mode — live in +[examples/](examples/); the test suite runs them on every CI pass, so +they cannot rot. + ## Quick start ```typescript diff --git a/TODO.impl/06-catalogue-detection.md b/TODO.impl/06-catalogue-detection.md new file mode 100644 index 0000000..f8e87e4 --- /dev/null +++ b/TODO.impl/06-catalogue-detection.md @@ -0,0 +1,33 @@ +# 06 — Catalogue-driven detection (P1) + +## Goal +Detection that can rank a supplied candidate catalogue instead of only +the already-loaded maps, including an async form that loads ISC +candidates on demand. + +## Why +`DetectOptions` exposes only `mapPattern`; `detectInMaps`'s +`knownMaps` parameter is unreachable from the public API. Ranking a +289-system catalogue today requires preloading every map first. The +detector is the right place for the candidate-set semantics (it +already takes the iterable internally) — the public API just never +let callers reach them. + +## Spec + +1. `DetectOptions.systems?: readonly SystemCode[]` — constrains the + candidate set (sync `detect()` ranks the intersection with + loadable/loaded maps, unchanged ordering). +2. `detectAsync(input, output, opts?: DetectOptions): Promise` + — same ranking, loads each candidate via `loadMapAsync` (skipping + codes that fail to load, matching the existing detector semantics + of skipping non-executable candidates). +3. Pure delegation: `runtime().detectAsync` mirrors `detect`; + `detectInMaps` gains an async twin that awaits loads — no change to + the sync path's behavior or signature (OCP: additive). + +## Acceptance (TDD) +- Failing tests first: `test/detect.test.ts` — systems-constrained + sync ranking, async ranking over an unloaded ISC-backed fixture + strategy, skip-on-load-failure, mapPattern still composes. +- Existing detector/ruby-parity behavior untouched (full suite green). diff --git a/TODO.impl/07-sync-miss-hint.md b/TODO.impl/07-sync-miss-hint.md new file mode 100644 index 0000000..35a29b7 --- /dev/null +++ b/TODO.impl/07-sync-miss-hint.md @@ -0,0 +1,31 @@ +# 07 — Actionable sync-miss error (P1) + +## Goal +When a sync `load()`/`transliterate()` misses but the configured +strategy stack contains async-capable strategies, the error says what +to do next. + +## Why +`MapNotFoundError` fires bare ("Map not found: X") even when +`loadMapAsync`/`transliterateAsync` would succeed — a real trap: this +exact confusion cost a debugging cycle during the examples work (sync +`loadMap` against the ISC strategy). The contract `/Map not found/` +is asserted by tests and matched by callers — the message prefix +stays; the guidance appends. + +## Spec + +1. `MapNotFoundError` gains an optional `hint` property; when present + the message reads `Map not found: X (async loaders configured — + use loadMapAsync/transliterateAsync)`. +2. `MapLoader.load` sets the hint when the sync loop resolved nothing + AND the stack's strategies returned promises (or: any strategy is + function-typed returning a thenable — detect via the actual miss, + no strategy-type registry needed). +3. No behavior change otherwise; async path's MapNotFoundError stays + hint-free (nothing to suggest there). + +## Acceptance (TDD) +- Failing test first: sync load over an async-only strategy rejects + with `/Map not found/` AND the hint; genuinely-unknown map on a + sync-only stack keeps the bare message. diff --git a/TODO.impl/08-ml-cdn-session-probe.md b/TODO.impl/08-ml-cdn-session-probe.md new file mode 100644 index 0000000..26da4c6 --- /dev/null +++ b/TODO.impl/08-ml-cdn-session-probe.md @@ -0,0 +1,31 @@ +# [COMPLETE-WITH-FINDINGS 2026-09-08] 08 — /ml CDN session probe (P2) + +## Goal +The last unverified claim in `docs/CDN.md` closed with evidence: a +real ONNX session created from CDN-imported `interscript/ml` + +`onnxruntime-web` pinned from the same origin, decoding a real model +in a worker. + +## Why +CDN.md documents that hosts "provide" onnxruntime-web — import of the +namespace was probed, session creation was not. The maps path is +CI-proven; the neural path deserves one live proof, kotoshu-style +(with the failure mode honestly recorded if ORT-from-CDN breaks). + +## Spec + +1. One-time local probe (Playwright): worker imports + `esm.sh/interscript@5.3.0/ml` + `esm.sh/onnxruntime-web@`, + resolves `tha-g2p-small-1.0-int4` (193 MB, the smallest tier), + decodes one Thai input, asserts IPA output. +2. Findings land in `docs/CDN.md` — the working ORT pin recipe, or + the failure mode with its cause. Not a CI test (193 MB); the + document is the artifact, the probe log the evidence. + +## Acceptance +- CDN.md's neural section states only what the probe observed — and it + now does: two timeouts on the 193 MB fetch (10 and 25 min); import + and resolve verified; session-from-CDN documented as unproven with + the import-map remedy named. The claim in CDN.md matches the + evidence; completing the live proof needs a faster probe host and + stays open as an optional follow-up, not a blocker. diff --git a/TODO.impl/09-cross-runtime-parity.md b/TODO.impl/09-cross-runtime-parity.md new file mode 100644 index 0000000..87438b0 --- /dev/null +++ b/TODO.impl/09-cross-runtime-parity.md @@ -0,0 +1,30 @@ +# [RUBY COMPLETE 2026-09-08; PYTHON HALF SUPERSEDED BY 11] 09 — Cross-runtime example parity (P2) + +## Goal +The example gallery's core conversions expressed as secryst (Ruby) +and secryst-py (Python) specs — same inputs, same expected outputs. + +## Why +"Same bytes from every runtime" is the project's core promise; the +gallery currently demonstrates it in TypeScript only. Parity specs +make the claim example-backed and regression-guarded in all three +runtimes. Model-driven: one shared conversion table per repo, specs +iterate it (adding a case = one line). + +## Spec + +1. `secryst` (Ruby): `spec/secryst/gallery_parity_spec.rb` — loads + maps through the gem's map path, asserts the gallery cases + (bgnpcgn-ukr name/column set). Cases as a frozen table constant. +2. `secryst-py`: `tests/test_gallery_parity.py` — same table, same + assertions. +3. Only conversions the runtime genuinely supports (map layer; the + neural + server-mode examples stay TS-only — they are + playground-specific). + +## Acceptance +- Ruby: green locally (3/3) and in CI. +- Python: blocked on item 11 (the runtime cannot parse the corpus at + all) — its half of the table lands with the ISC port. +- The conversion table matches `examples/` outputs exactly (Anton + Olehovych et al.). diff --git a/TODO.impl/10-readme-quickstart.md b/TODO.impl/10-readme-quickstart.md new file mode 100644 index 0000000..00bdb0e --- /dev/null +++ b/TODO.impl/10-readme-quickstart.md @@ -0,0 +1,21 @@ +# 10 — README quickstart wiring (P3) + +## Goal +The new surfaces discoverable: interscript-ts README links the +example gallery and the CDN recipe. + +## Why +`examples/` and `docs/CDN.md` exist but nothing in the README points +at them; a script-first visitor (the audience this lane serves) lands +on the README first. + +## Spec + +README gains a short "Try it" section: the CDN one-liner from +docs/CDN.md, a pointer to `examples/` ("runnable, CI-verified"), and +the ambient-types note for editors. No other edits. + +## Acceptance +- The section's code block matches the CDN.md recipe verbatim (DRY by + reference, not duplication — keep the snippet minimal and say "see + docs/CDN.md"). diff --git a/TODO.impl/11-py-isc-support.md b/TODO.impl/11-py-isc-support.md new file mode 100644 index 0000000..cf71ddb --- /dev/null +++ b/TODO.impl/11-py-isc-support.md @@ -0,0 +1,35 @@ +# 11 — interscript-py ISC support (P1) + +## Goal +The Python map runtime reads the ISC corpus — the only map format the +maps repository still ships — instead of the legacy `.imp` DSL it was +built for. + +## Why +Discovered completing item 09: `_find_map` accepts `.imp`/`.isc`, but +the parser is `.imp`-only, and the corpus has **zero** `.imp` files +left. Every map call raises `ValueError: line 1: cannot parse` — the +whole Python runtime has been dark against the current corpus, and its +CI last ran 2026-08-20 so nobody saw it. TypeScript and Ruby both +parse the current ISC (the TS examples convert it live; the Ruby gem +returns identical bytes). + +## Spec + +Port the reference TypeScript implementation (`interscript-ts +src/isc/`: parser 765 lines, types, converter) to Python: + +1. `src/interscript/isc_parser.py` — the ISC grammar: `system + "id" {` headers, metadata/tests/dependency/stage blocks, sub + tables, escapes; produces the same plain-tree shape the existing + engine consumes. +2. Dispatch by extension in the loader: `.isc` → the new parser, + `.imp` → the legacy parser (both stay — OCP). +3. Tests: port the TS parser's unit fixtures; the gallery parity + table (item 09) becomes the end-to-end gate — `bgnpcgn-ukr` + returns the same three strings TS and Ruby return. + +## Acceptance +- `INTERSCRIPT_MAPS_PATH= pytest` green on the parity table + plus the existing suite. +- The parity table passes in all three runtimes with identical bytes. diff --git a/TODO.impl/README.md b/TODO.impl/README.md index 2e4a690..48c8192 100644 --- a/TODO.impl/README.md +++ b/TODO.impl/README.md @@ -18,6 +18,12 @@ repo, one verifies the API). | 03 | [Worker-mode guarantee](03-worker-mode-guarantee.md) | P2 | interscript.github.io | | 04 | [Playground ambient types](04-playground-ambient-types.md) | P2 | interscript-ts | | 05 | [API server-mode CORS](05-api-server-mode-cors.md) | P3 | verify live; fix in api if broken | +| 06 | [Catalogue-driven detection](06-catalogue-detection.md) | P1 | interscript-ts | +| 07 | [Actionable sync-miss error](07-sync-miss-hint.md) | P1 | interscript-ts | +| 08 | [/ml CDN session probe](08-ml-cdn-session-probe.md) | P2 | probe + docs | +| 09 | [Cross-runtime example parity](09-cross-runtime-parity.md) | P2 | ruby done; py via 11 | +| 10 | [README quickstart wiring](10-readme-quickstart.md) | P3 | interscript-ts | +| 11 | [interscript-py ISC support](11-py-isc-support.md) | P1 | the dark runtime | Standing rules (unchanged from the campaign): every claim measured before it ships; staged sets verified before every commit; no diff --git a/docs/CDN.md b/docs/CDN.md index e8416bc..3141bb5 100644 --- a/docs/CDN.md +++ b/docs/CDN.md @@ -42,6 +42,21 @@ the only working path). Our package has no such entanglement on the maps path: plain ESM with `fflate`/`js-yaml` dependencies, which esm.sh resolves. No raw-file fallback is needed. +## The neural session path — probed, not yet proven + +Two attempts to complete a full CDN-side session (worker imports +`esm.sh/.../ml`, resolves `tha-g2p-small-1.0-int4`, creates the ONNX +session) timed out at 10 and 25 minutes on the 193 MB model fetch +through the worker — the import and resolve layers never failed; the +download did not complete in the probe budget. What is verified: +namespace import (above) and, via the bundler path, a complete +browser session (the `/neural` demo in production). What remains +unverified from CDN alone: onnxruntime-web resolving through esm.sh's +transform of the bare peer specifier inside `interscript/ml`. If a +CDN session fails there, the import-map fallback is the documented +remedy: map `onnxruntime-web` to a pinned jsDelivr build on the host +page before importing. + ## Pins and rules - **Pin exact versions** (`@5.3.0`, never `@5`): CDN transforms are diff --git a/src/detector.ts b/src/detector.ts index 44d6cce..f2097c1 100644 --- a/src/detector.ts +++ b/src/detector.ts @@ -7,9 +7,9 @@ * Levenshtein distance to `output`, return ranked candidates. */ -import type { DetectionResult, DetectOptions, SystemCode } from "./types.js" +import type { CompiledMap, DetectionResult, DetectOptions, SystemCode } from "./types.js" import type { MapLoader } from "./loader.js" -import { executeStage } from "./runtime/interpreter.js" +import { executeStage, executeStageAsync } from "./runtime/interpreter.js" import { InterscriptError } from "./errors.js" /** @@ -60,7 +60,7 @@ export function detectInMaps( ): DetectionResult[] { const candidates: DetectionResult[] = [] const filter = opts.mapPattern ? globToRegExp(opts.mapPattern) : null - const systems = knownMaps ?? loader.loadedMaps() + const systems = knownMaps ?? opts.systems ?? loader.loadedMaps() for (const systemCode of systems) { if (filter && !filter.test(systemCode)) continue @@ -81,3 +81,40 @@ export function detectInMaps( return candidates.sort((a, b) => a.distance - b.distance) } + +/** + * Async detection: loads each candidate through `loadMapAsync` before + * ranking it, so ISC/HTTP-backed catalogues work without preloading. + * Candidates that fail to load are skipped — the same semantics the + * sync form applies to non-executable candidates. + */ +export async function detectInMapsAsync( + input: string, + output: string, + loader: MapLoader, + opts: DetectOptions = {}, + loadMap: (systemCode: SystemCode) => Promise = (code) => loader.loadAsync(code), +): Promise { + const candidates: DetectionResult[] = [] + const filter = opts.mapPattern ? globToRegExp(opts.mapPattern) : null + const systems = opts.systems ?? loader.loadedMaps() + + for (const systemCode of systems) { + if (filter && !filter.test(systemCode)) continue + + let transliterated: string + try { + const map = await loadMap(systemCode) + transliterated = await executeStageAsync(map, "main", input, loader) + } catch (e) { + if (e instanceof InterscriptError) continue + throw e + } + candidates.push({ + mapName: systemCode, + distance: levenshtein(transliterated, output), + }) + } + + return candidates.sort((a, b) => a.distance - b.distance) +} diff --git a/src/errors.ts b/src/errors.ts index 2e9eb75..b2bb920 100644 --- a/src/errors.ts +++ b/src/errors.ts @@ -18,8 +18,11 @@ export class InterscriptError extends Error { export class MapNotFoundError extends InterscriptError { readonly systemCode: string - constructor(systemCode: string) { - super(`Map not found: ${systemCode}`) + constructor(systemCode: string, options?: { asyncLoadersConfigured?: boolean }) { + const hint = options?.asyncLoadersConfigured + ? " (async loaders configured — use loadMapAsync/transliterateAsync)" + : "" + super(`Map not found: ${systemCode}${hint}`) this.systemCode = systemCode } } diff --git a/src/index.ts b/src/index.ts index 6789fb3..28d4462 100644 --- a/src/index.ts +++ b/src/index.ts @@ -20,7 +20,7 @@ import { MapNotFoundError, SystemConversionError, } from "./errors.js" -import { detectInMaps } from "./detector.js" +import { detectInMaps, detectInMapsAsync } from "./detector.js" export { InterscriptError, @@ -209,6 +209,16 @@ class InterscriptRuntime { ): DetectionResult[] { return detectInMaps(input, output, this.loader, opts, knownMaps) } + + /** Async detection — loads candidates through `loadMapAsync`. */ + async detectAsync( + input: string, + output: string, + opts: DetectOptions = {}, + ): Promise { + // The runtime's dep-aware loader, so maps with dependencies load whole. + return detectInMapsAsync(input, output, this.loader, opts, (code) => this.loadMapAsync(code)) + } } let defaultRuntime: InterscriptRuntime | undefined @@ -257,6 +267,19 @@ export function detect(input: string, output: string, opts?: DetectOptions): Det return runtime().detect(input, output, opts) } +/** + * Public API — async detection over a candidate catalogue. Loads each + * candidate (`opts.systems`, else everything ever loaded) through the + * async strategies, so ISC/HTTP-backed catalogues need no preloading. + */ +export function detectAsync( + input: string, + output: string, + opts?: DetectOptions, +): Promise { + return runtime().detectAsync(input, output, opts) +} + /** Reset the default runtime (mainly for tests). */ export function reset(): void { defaultRuntime = undefined diff --git a/src/loader.ts b/src/loader.ts index 294edb7..954f473 100644 --- a/src/loader.ts +++ b/src/loader.ts @@ -53,10 +53,16 @@ export class MapLoader { return known } + let sawAsyncStrategy = false for (const strategy of this.strategies) { const result = strategy(systemCode) - // Promise results can't be handled synchronously — skip. - if (result && typeof (result as Promise).then !== "function") { + // Promise results can't be handled synchronously — skip, but + // remember: the async API could still load this map. + if (result && typeof (result as Promise).then === "function") { + sawAsyncStrategy = true + continue + } + if (result) { const map = result as CompiledMap this.cache.set(systemCode, map) this.known.set(systemCode, map) @@ -64,7 +70,7 @@ export class MapLoader { return map } } - throw new MapNotFoundError(systemCode) + throw new MapNotFoundError(systemCode, { asyncLoadersConfigured: sawAsyncStrategy }) } /** diff --git a/src/types.ts b/src/types.ts index f028b75..833d245 100644 --- a/src/types.ts +++ b/src/types.ts @@ -31,6 +31,8 @@ export interface DetectionResult { /** Options for detect(). */ export interface DetectOptions { mapPattern?: string + /** Constrain detection to a caller-supplied candidate catalogue. */ + systems?: readonly SystemCode[] } /** diff --git a/test/detect-catalogue.test.ts b/test/detect-catalogue.test.ts new file mode 100644 index 0000000..f8c6952 --- /dev/null +++ b/test/detect-catalogue.test.ts @@ -0,0 +1,60 @@ +/** + * Catalogue-driven detection: the candidate set comes from the caller + * (opts.systems), and the async form loads candidates on demand + * through async strategies. RED first — the option and detectAsync + * don't exist yet. + */ +import { describe, expect, it } from "vitest" +import { configure, detect, detectAsync, reset } from "../src/index.js" +import { filesystemStrategy } from "../src/loaders.node.js" +import { resolve, dirname } from "node:path" +import { fileURLToPath } from "node:url" + +const MAPS_DIR = resolve(dirname(fileURLToPath(import.meta.url)), "fixtures", "maps") +const UKR = "bgnpcgn-ukr-Cyrl-Latn-2019" +const RUS = "bgnpcgn-rus-Cyrl-Latn-1947" +const fromDisk = filesystemStrategy(MAPS_DIR) + +describe("catalogue-driven detection", () => { + it("ranks a caller-supplied candidate set", () => { + reset() + configure({ strategies: [filesystemStrategy(MAPS_DIR)] }) + const results = detect("Антон", "Anton", { systems: [UKR, RUS] }) + expect(results[0]?.mapName).toBe(UKR) + expect(results).toHaveLength(2) + }) + + it("detectAsync loads unloaded candidates through async strategies", async () => { + reset() + const loaded: string[] = [] + configure({ + strategies: [ + async (code) => { + const map = fromDisk(code) + if (!map) return undefined + loaded.push(code) + return map + }, + ], + }) + const results = await detectAsync("Антон", "Anton", { systems: [UKR, RUS] }) + expect(loaded).toEqual(expect.arrayContaining([UKR, RUS])) + expect(results[0]?.mapName).toBe(UKR) + }) + + it("detectAsync skips candidates that fail to load", async () => { + reset() + configure({ + strategies: [async (code) => (code === RUS ? fromDisk(RUS) : undefined)], + }) + const results = await detectAsync("Антон", "Anton", { systems: [RUS, "missing-system"] }) + expect(results.map((r) => r.mapName)).toEqual([RUS]) + }) + + it("mapPattern composes with systems", () => { + reset() + configure({ strategies: [filesystemStrategy(MAPS_DIR)] }) + const results = detect("Антон", "Anton", { systems: [UKR, RUS], mapPattern: "*rus*" }) + expect(results.map((r) => r.mapName)).toEqual([RUS]) + }) +}) diff --git a/test/sync-miss-hint.test.ts b/test/sync-miss-hint.test.ts new file mode 100644 index 0000000..8b5034d --- /dev/null +++ b/test/sync-miss-hint.test.ts @@ -0,0 +1,21 @@ +/** + * Actionable sync-miss error: when the sync loader misses but the + * configured stack is async-capable, the error points at the async + * API. RED first — the hint doesn't exist yet. + */ +import { describe, expect, it } from "vitest" +import { MapNotFoundError } from "../src/errors.js" + +describe("sync-miss hint", () => { + it("carries the async guidance when provided", () => { + const error = new MapNotFoundError("some-system", { + asyncLoadersConfigured: true, + }) + expect(error.message).toMatch(/^Map not found: some-system/) + expect(error.message).toMatch(/loadMapAsync|transliterateAsync/) + }) + + it("stays bare without a hint", () => { + expect(new MapNotFoundError("x").message).toBe("Map not found: x") + }) +})