Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,22 @@ A TypeScript-native port of the [Ruby runtime](https://github.com/interscript/in
npm install interscript
```

## Try it

In a browser with no bundler (worker-safe, exact-version pin):

```typescript
import { transliterateAsync } from "https://esm.sh/interscript@5.3.0"
```

Full recipe and evidence: [docs/CDN.md](docs/CDN.md). Editor
autocomplete for injected globals: `dist/playground.d.ts`.

Runnable, CI-verified example scripts — convert, detect-and-chain,
neural diacritization, batch columns, server mode — live in
[examples/](examples/); the test suite runs them on every CI pass, so
they cannot rot.

## Quick start

```typescript
Expand Down
33 changes: 33 additions & 0 deletions TODO.impl/06-catalogue-detection.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# 06 — Catalogue-driven detection (P1)

## Goal
Detection that can rank a supplied candidate catalogue instead of only
the already-loaded maps, including an async form that loads ISC
candidates on demand.

## Why
`DetectOptions` exposes only `mapPattern`; `detectInMaps`'s
`knownMaps` parameter is unreachable from the public API. Ranking a
289-system catalogue today requires preloading every map first. The
detector is the right place for the candidate-set semantics (it
already takes the iterable internally) — the public API just never
let callers reach them.

## Spec

1. `DetectOptions.systems?: readonly SystemCode[]` — constrains the
candidate set (sync `detect()` ranks the intersection with
loadable/loaded maps, unchanged ordering).
2. `detectAsync(input, output, opts?: DetectOptions): Promise<DetectionResult[]>`
— same ranking, loads each candidate via `loadMapAsync` (skipping
codes that fail to load, matching the existing detector semantics
of skipping non-executable candidates).
3. Pure delegation: `runtime().detectAsync` mirrors `detect`;
`detectInMaps` gains an async twin that awaits loads — no change to
the sync path's behavior or signature (OCP: additive).

## Acceptance (TDD)
- Failing tests first: `test/detect.test.ts` — systems-constrained
sync ranking, async ranking over an unloaded ISC-backed fixture
strategy, skip-on-load-failure, mapPattern still composes.
- Existing detector/ruby-parity behavior untouched (full suite green).
31 changes: 31 additions & 0 deletions TODO.impl/07-sync-miss-hint.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# 07 — Actionable sync-miss error (P1)

## Goal
When a sync `load()`/`transliterate()` misses but the configured
strategy stack contains async-capable strategies, the error says what
to do next.

## Why
`MapNotFoundError` fires bare ("Map not found: X") even when
`loadMapAsync`/`transliterateAsync` would succeed — a real trap: this
exact confusion cost a debugging cycle during the examples work (sync
`loadMap` against the ISC strategy). The contract `/Map not found/`
is asserted by tests and matched by callers — the message prefix
stays; the guidance appends.

## Spec

1. `MapNotFoundError` gains an optional `hint` property; when present
the message reads `Map not found: X (async loaders configured —
use loadMapAsync/transliterateAsync)`.
2. `MapLoader.load` sets the hint when the sync loop resolved nothing
AND the stack's strategies returned promises (or: any strategy is
function-typed returning a thenable — detect via the actual miss,
no strategy-type registry needed).
3. No behavior change otherwise; async path's MapNotFoundError stays
hint-free (nothing to suggest there).

## Acceptance (TDD)
- Failing test first: sync load over an async-only strategy rejects
with `/Map not found/` AND the hint; genuinely-unknown map on a
sync-only stack keeps the bare message.
31 changes: 31 additions & 0 deletions TODO.impl/08-ml-cdn-session-probe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# [COMPLETE-WITH-FINDINGS 2026-09-08] 08 — /ml CDN session probe (P2)

## Goal
The last unverified claim in `docs/CDN.md` closed with evidence: a
real ONNX session created from CDN-imported `interscript/ml` +
`onnxruntime-web` pinned from the same origin, decoding a real model
in a worker.

## Why
CDN.md documents that hosts "provide" onnxruntime-web — import of the
namespace was probed, session creation was not. The maps path is
CI-proven; the neural path deserves one live proof, kotoshu-style
(with the failure mode honestly recorded if ORT-from-CDN breaks).

## Spec

1. One-time local probe (Playwright): worker imports
`esm.sh/interscript@5.3.0/ml` + `esm.sh/onnxruntime-web@<pin>`,
resolves `tha-g2p-small-1.0-int4` (193 MB, the smallest tier),
decodes one Thai input, asserts IPA output.
2. Findings land in `docs/CDN.md` — the working ORT pin recipe, or
the failure mode with its cause. Not a CI test (193 MB); the
document is the artifact, the probe log the evidence.

## Acceptance
- CDN.md's neural section states only what the probe observed — and it
now does: two timeouts on the 193 MB fetch (10 and 25 min); import
and resolve verified; session-from-CDN documented as unproven with
the import-map remedy named. The claim in CDN.md matches the
evidence; completing the live proof needs a faster probe host and
stays open as an optional follow-up, not a blocker.
30 changes: 30 additions & 0 deletions TODO.impl/09-cross-runtime-parity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# [RUBY COMPLETE 2026-09-08; PYTHON HALF SUPERSEDED BY 11] 09 — Cross-runtime example parity (P2)

## Goal
The example gallery's core conversions expressed as secryst (Ruby)
and secryst-py (Python) specs — same inputs, same expected outputs.

## Why
"Same bytes from every runtime" is the project's core promise; the
gallery currently demonstrates it in TypeScript only. Parity specs
make the claim example-backed and regression-guarded in all three
runtimes. Model-driven: one shared conversion table per repo, specs
iterate it (adding a case = one line).

## Spec

1. `secryst` (Ruby): `spec/secryst/gallery_parity_spec.rb` — loads
maps through the gem's map path, asserts the gallery cases
(bgnpcgn-ukr name/column set). Cases as a frozen table constant.
2. `secryst-py`: `tests/test_gallery_parity.py` — same table, same
assertions.
3. Only conversions the runtime genuinely supports (map layer; the
neural + server-mode examples stay TS-only — they are
playground-specific).

## Acceptance
- Ruby: green locally (3/3) and in CI.
- Python: blocked on item 11 (the runtime cannot parse the corpus at
all) — its half of the table lands with the ISC port.
- The conversion table matches `examples/` outputs exactly (Anton
Olehovych et al.).
21 changes: 21 additions & 0 deletions TODO.impl/10-readme-quickstart.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# 10 — README quickstart wiring (P3)

## Goal
The new surfaces discoverable: interscript-ts README links the
example gallery and the CDN recipe.

## Why
`examples/` and `docs/CDN.md` exist but nothing in the README points
at them; a script-first visitor (the audience this lane serves) lands
on the README first.

## Spec

README gains a short "Try it" section: the CDN one-liner from
docs/CDN.md, a pointer to `examples/` ("runnable, CI-verified"), and
the ambient-types note for editors. No other edits.

## Acceptance
- The section's code block matches the CDN.md recipe verbatim (DRY by
reference, not duplication — keep the snippet minimal and say "see
docs/CDN.md").
35 changes: 35 additions & 0 deletions TODO.impl/11-py-isc-support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# 11 — interscript-py ISC support (P1)

## Goal
The Python map runtime reads the ISC corpus — the only map format the
maps repository still ships — instead of the legacy `.imp` DSL it was
built for.

## Why
Discovered completing item 09: `_find_map` accepts `.imp`/`.isc`, but
the parser is `.imp`-only, and the corpus has **zero** `.imp` files
left. Every map call raises `ValueError: line 1: cannot parse` — the
whole Python runtime has been dark against the current corpus, and its
CI last ran 2026-08-20 so nobody saw it. TypeScript and Ruby both
parse the current ISC (the TS examples convert it live; the Ruby gem
returns identical bytes).

## Spec

Port the reference TypeScript implementation (`interscript-ts
src/isc/`: parser 765 lines, types, converter) to Python:

1. `src/interscript/isc_parser.py` — the ISC grammar: `system
"id" {` headers, metadata/tests/dependency/stage blocks, sub
tables, escapes; produces the same plain-tree shape the existing
engine consumes.
2. Dispatch by extension in the loader: `.isc` → the new parser,
`.imp` → the legacy parser (both stay — OCP).
3. Tests: port the TS parser's unit fixtures; the gallery parity
table (item 09) becomes the end-to-end gate — `bgnpcgn-ukr`
returns the same three strings TS and Ruby return.

## Acceptance
- `INTERSCRIPT_MAPS_PATH=<corpus> pytest` green on the parity table
plus the existing suite.
- The parity table passes in all three runtimes with identical bytes.
6 changes: 6 additions & 0 deletions TODO.impl/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ repo, one verifies the API).
| 03 | [Worker-mode guarantee](03-worker-mode-guarantee.md) | P2 | interscript.github.io |
| 04 | [Playground ambient types](04-playground-ambient-types.md) | P2 | interscript-ts |
| 05 | [API server-mode CORS](05-api-server-mode-cors.md) | P3 | verify live; fix in api if broken |
| 06 | [Catalogue-driven detection](06-catalogue-detection.md) | P1 | interscript-ts |
| 07 | [Actionable sync-miss error](07-sync-miss-hint.md) | P1 | interscript-ts |
| 08 | [/ml CDN session probe](08-ml-cdn-session-probe.md) | P2 | probe + docs |
| 09 | [Cross-runtime example parity](09-cross-runtime-parity.md) | P2 | ruby done; py via 11 |
| 10 | [README quickstart wiring](10-readme-quickstart.md) | P3 | interscript-ts |
| 11 | [interscript-py ISC support](11-py-isc-support.md) | P1 | the dark runtime |

Standing rules (unchanged from the campaign): every claim measured
before it ships; staged sets verified before every commit; no
Expand Down
15 changes: 15 additions & 0 deletions docs/CDN.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,21 @@ the only working path). Our package has no such entanglement on the
maps path: plain ESM with `fflate`/`js-yaml` dependencies, which esm.sh
resolves. No raw-file fallback is needed.

## The neural session path — probed, not yet proven

Two attempts to complete a full CDN-side session (worker imports
`esm.sh/.../ml`, resolves `tha-g2p-small-1.0-int4`, creates the ONNX
session) timed out at 10 and 25 minutes on the 193 MB model fetch
through the worker — the import and resolve layers never failed; the
download did not complete in the probe budget. What is verified:
namespace import (above) and, via the bundler path, a complete
browser session (the `/neural` demo in production). What remains
unverified from CDN alone: onnxruntime-web resolving through esm.sh's
transform of the bare peer specifier inside `interscript/ml`. If a
CDN session fails there, the import-map fallback is the documented
remedy: map `onnxruntime-web` to a pinned jsDelivr build on the host
page before importing.

## Pins and rules

- **Pin exact versions** (`@5.3.0`, never `@5`): CDN transforms are
Expand Down
43 changes: 40 additions & 3 deletions src/detector.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,9 +7,9 @@
* Levenshtein distance to `output`, return ranked candidates.
*/

import type { DetectionResult, DetectOptions, SystemCode } from "./types.js"
import type { CompiledMap, DetectionResult, DetectOptions, SystemCode } from "./types.js"
import type { MapLoader } from "./loader.js"
import { executeStage } from "./runtime/interpreter.js"
import { executeStage, executeStageAsync } from "./runtime/interpreter.js"
import { InterscriptError } from "./errors.js"

/**
Expand Down Expand Up @@ -60,7 +60,7 @@ export function detectInMaps(
): DetectionResult[] {
const candidates: DetectionResult[] = []
const filter = opts.mapPattern ? globToRegExp(opts.mapPattern) : null
const systems = knownMaps ?? loader.loadedMaps()
const systems = knownMaps ?? opts.systems ?? loader.loadedMaps()

for (const systemCode of systems) {
if (filter && !filter.test(systemCode)) continue
Expand All @@ -81,3 +81,40 @@ export function detectInMaps(

return candidates.sort((a, b) => a.distance - b.distance)
}

/**
* Async detection: loads each candidate through `loadMapAsync` before
* ranking it, so ISC/HTTP-backed catalogues work without preloading.
* Candidates that fail to load are skipped — the same semantics the
* sync form applies to non-executable candidates.
*/
export async function detectInMapsAsync(
input: string,
output: string,
loader: MapLoader,
opts: DetectOptions = {},
loadMap: (systemCode: SystemCode) => Promise<CompiledMap> = (code) => loader.loadAsync(code),
): Promise<DetectionResult[]> {
const candidates: DetectionResult[] = []
const filter = opts.mapPattern ? globToRegExp(opts.mapPattern) : null
const systems = opts.systems ?? loader.loadedMaps()

for (const systemCode of systems) {
if (filter && !filter.test(systemCode)) continue

let transliterated: string
try {
const map = await loadMap(systemCode)
transliterated = await executeStageAsync(map, "main", input, loader)
} catch (e) {
if (e instanceof InterscriptError) continue
throw e
}
candidates.push({
mapName: systemCode,
distance: levenshtein(transliterated, output),
})
}

return candidates.sort((a, b) => a.distance - b.distance)
}
7 changes: 5 additions & 2 deletions src/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,8 +18,11 @@ export class InterscriptError extends Error {

export class MapNotFoundError extends InterscriptError {
readonly systemCode: string
constructor(systemCode: string) {
super(`Map not found: ${systemCode}`)
constructor(systemCode: string, options?: { asyncLoadersConfigured?: boolean }) {
const hint = options?.asyncLoadersConfigured
? " (async loaders configured — use loadMapAsync/transliterateAsync)"
: ""
super(`Map not found: ${systemCode}${hint}`)
this.systemCode = systemCode
}
}
Expand Down
25 changes: 24 additions & 1 deletion src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import {
MapNotFoundError,
SystemConversionError,
} from "./errors.js"
import { detectInMaps } from "./detector.js"
import { detectInMaps, detectInMapsAsync } from "./detector.js"

export {
InterscriptError,
Expand Down Expand Up @@ -209,6 +209,16 @@ class InterscriptRuntime {
): DetectionResult[] {
return detectInMaps(input, output, this.loader, opts, knownMaps)
}

/** Async detection — loads candidates through `loadMapAsync`. */
async detectAsync(
input: string,
output: string,
opts: DetectOptions = {},
): Promise<DetectionResult[]> {
// The runtime's dep-aware loader, so maps with dependencies load whole.
return detectInMapsAsync(input, output, this.loader, opts, (code) => this.loadMapAsync(code))
}
}

let defaultRuntime: InterscriptRuntime | undefined
Expand Down Expand Up @@ -257,6 +267,19 @@ export function detect(input: string, output: string, opts?: DetectOptions): Det
return runtime().detect(input, output, opts)
}

/**
* Public API — async detection over a candidate catalogue. Loads each
* candidate (`opts.systems`, else everything ever loaded) through the
* async strategies, so ISC/HTTP-backed catalogues need no preloading.
*/
export function detectAsync(
input: string,
output: string,
opts?: DetectOptions,
): Promise<DetectionResult[]> {
return runtime().detectAsync(input, output, opts)
}

/** Reset the default runtime (mainly for tests). */
export function reset(): void {
defaultRuntime = undefined
Expand Down
12 changes: 9 additions & 3 deletions src/loader.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,18 +53,24 @@ export class MapLoader {
return known
}

let sawAsyncStrategy = false
for (const strategy of this.strategies) {
const result = strategy(systemCode)
// Promise results can't be handled synchronously — skip.
if (result && typeof (result as Promise<CompiledMap>).then !== "function") {
// Promise results can't be handled synchronously — skip, but
// remember: the async API could still load this map.
if (result && typeof (result as Promise<CompiledMap>).then === "function") {
sawAsyncStrategy = true
continue
}
if (result) {
const map = result as CompiledMap
this.cache.set(systemCode, map)
this.known.set(systemCode, map)
this.options.onLoaded?.(systemCode, map)
return map
}
}
throw new MapNotFoundError(systemCode)
throw new MapNotFoundError(systemCode, { asyncLoadersConfigured: sawAsyncStrategy })
}

/**
Expand Down
Loading