From 27a84625f7d5e9617d85e348a6aaf540a6dcc057 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Sat, 12 Sep 2026 06:16:49 +0800 Subject: [PATCH] fix(golden): classify parity by artifact precision, not filename; scope the parity claims The golden test classified models by precision substring in the GOLDEN FILENAME - but tha-g2p-small-1.0 resolves to an int8 zip whose golden name drops the suffix, so a quantized artifact got byte-exact assertions and failed on a near-tie flip ('taj' vs 'ta:j' class). Classification now reads the loaded manifest's precision field (fp32/fp16 exact; int8/int4 decode-health length floor). Full GOLDEN_DIR run: 12 models x 25 rows, green. Paper A, paper C, and publication notes now state the parity guarantee precision-scoped: byte-identical for fp32/fp16 on all hardware tested; measured quality parity for quantized artifacts. RESULTS.md gains the speculative-decode acceptance entry (0.9886 mean, 8.85 tokens/verifier pass, output-preserving). --- docs/PUBLICATION-NOTES.md | 11 +++++++---- docs/RESULTS.md | 22 ++++++++++++++++++++++ docs/paper-c.adoc | 15 +++++++++++---- docs/paper.adoc | 10 +++++----- runtime/tests/test_model.py | 22 ++++++++++++++++++---- 5 files changed, 63 insertions(+), 17 deletions(-) diff --git a/docs/PUBLICATION-NOTES.md b/docs/PUBLICATION-NOTES.md index 03525d3..45aa90b 100644 --- a/docs/PUBLICATION-NOTES.md +++ b/docs/PUBLICATION-NOTES.md @@ -9,10 +9,13 @@ un-landed number. ### 1. The artifact contract (IMF v1) and cross-runtime parity One zip, three ONNX graphs, fixed byte table, per-member SHA-256, -byte-identical output from Ruby/Python/TypeScript; opset-14 floor for -old consumer runtimes. Paper: sections 3–4 (section-imf). The framing -contribution — neural models under the same discipline as -deterministic transliteration maps. +byte-identical output from Ruby/Python/TypeScript for fp32/fp16 +artifacts; quantized artifacts carry quality parity (per-model +cer_delta gates) because near-tie decisions flip across hardware +(measured 2026-09-07); opset-14 floor for old consumer runtimes. +Paper: sections 3–4 (section-imf). The framing contribution — neural +models under the same discipline as deterministic transliteration +maps. ### 2. The decode-protocol correction Beam-4 with length normalization inflates flat byte-student PER 4.2× diff --git a/docs/RESULTS.md b/docs/RESULTS.md index ba608a1..5847c8d 100644 --- a/docs/RESULTS.md +++ b/docs/RESULTS.md @@ -626,3 +626,25 @@ hardware for fp32/fp16; for quantized artifacts, quality parity (the published per-model cer_delta gates) plus prefix-consistency.** golden-v1's quantized rows are reference outputs (documented hardware/ORT provenance), not byte-assertions. + +## Speculative decode across the tier ladder: acceptance 0.9886, output-preserving (2026-09-11) + +The shipped artifacts form a drafter/verifier pair: ara-diac-layerdrop-1.0-int4 +(190M) drafts, ara-diac-small-2.1-int8 (300M) verifies. Measured over +all 25 golden-v1 Arabic rows (CPU, K=8, greedy verification): + +- acceptance 0.9886 mean (min 0.955, median 0.991) — the int4 drafter's + argmax matches the int8 verifier's at ~99% of positions +- 8.85 tokens per verifier pass (K=8 plus the bonus token on full + acceptance): ~9x fewer verifier invocations than token-by-token + decode +- output preservation holds by construction and by measurement: every + exactness-checked row (18 of 25; the O(T^2) plain-path reference is + capped to <=600B rows) is byte-identical to the verifier's + plain-path greedy; plain and KV paths agreed on all checked rows + +Source: TODO.qwen-next/10 (DeepSeek-V4.1-Flash learnings, DSpark +pattern), probe at scripts/probe_speculative.py. Runtime: +interscript-ts SpeculativeModel (PR #77). The lite tier stays the +standalone fast path; this is a middle tier — 2.1 outputs at a +fraction of the decode calls. diff --git a/docs/paper-c.adoc b/docs/paper-c.adoc index 1adf66b..69a2742 100644 --- a/docs/paper-c.adoc +++ b/docs/paper-c.adoc @@ -17,9 +17,13 @@ Model Format (IMF v1) and its delivery channel: content-addressed zips (per-member SHA-256) resolved through a tag-pinned index with sidecar verification, served from release assets through a CORS front door, cached once per client, and executed byte-identically across -TypeScript, Python, and Ruby runtimes. We measure the cost of this -discipline across three tiers — node, server, and browser — and find -it effectively free: whole-file hashing adds ~0.1 s against 7-13 s +TypeScript, Python, and Ruby runtimes for full-precision artifacts; +for quantized artifacts the guarantee is quality-level (per-model +measured deltas against full precision), because near-tie decisions +flip across hardware — a boundary we measure, not assume. We measure +the cost of this discipline across three tiers — node, server, and +browser — and find it effectively free: whole-file hashing adds ~0.1 s +against 7-13 s session creation, verified reloads complete in ~0.4 s from cache, and a 202 MB int8 model runs end-to-end in a headless browser over WASM. Since the benchmark round, a depth-cut student (encoder 12->6, @@ -41,7 +45,10 @@ tag-pinned index channel (index-v5). == 2. The artifact contract (IMF v1) - zip + metadata.yaml + per-member sha256; fixed ByT5 byte table (byte+3, trailing EOS); opset-14 floor; KV decoder graphs -- cross-runtime parity: byte-identical decode in TS/Py/Ruby +- cross-runtime parity, precision-scoped: fp32/fp16 byte-identical + decode in TS/Py/Ruby on all hardware tested; int8/int4 quality + parity via per-model measured deltas (near-tie flips across CPU + architectures, measured 2026-09-07) - provenance: labels sha256 recorded in every verdict; per-paragraph predictions saved with every evaluation diff --git a/docs/paper.adoc b/docs/paper.adoc index de58f8f..5f2456b 100644 --- a/docs/paper.adoc +++ b/docs/paper.adoc @@ -12,7 +12,7 @@ toc::[] Interoperable script conversion (OGC Abstract Standard topic _Interoperable Script Conversion Systems_) has matured for *deterministic* transliteration: authority-published romanization systems encoded as machine-readable maps that produce byte-identical output across runtimes. A complementary class of conversions — diacritization restoration, grapheme-to-phoneme conversion for scripts without a phonemic orthography, homograph disambiguation — has no authority to publish a map, because the mapping is not a rule set but a learned function. This paper presents the phonological layer of Interscript: a complete pipeline for producing, validating, and serving distilled byte-level sequence-to-sequence models under the same engineering discipline as deterministic maps. -We contribute: (1) the Interscript Model Format (IMF v1), a checksummed, runtime-agnostic artifact contract that carries models across Ruby, Python, and JavaScript with byte-identical output; (2) a distillation methodology in which students are trained from frozen, independently evaluated teachers under pre-registered error budgets; (3) a measured size–quality frontier for client-tier models demonstrating that sub-100M randomly-initialized byte-level students fail catastrophically, while a pretrained 300M backbone quantized to 4 bits (193 MiB) retains full utility; and (4) a decode-protocol study showing that beam search with conventional length normalization is *counterproductive* on distilled byte-level students, inflating phoneme error rate by a factor of four relative to greedy decoding on the same artifact — a calibration artifact with implications for any evaluation of low-confidence autoregressive models. +We contribute: (1) the Interscript Model Format (IMF v1), a checksummed, runtime-agnostic artifact contract that carries models across Ruby, Python, and JavaScript with byte-identical output at full precision and measured quality parity when quantized; (2) a distillation methodology in which students are trained from frozen, independently evaluated teachers under pre-registered error budgets; (3) a measured size–quality frontier for client-tier models demonstrating that sub-100M randomly-initialized byte-level students fail catastrophically, while a pretrained 300M backbone quantized to 4 bits (193 MiB) retains full utility; and (4) a decode-protocol study showing that beam search with conventional length normalization is *counterproductive* on distilled byte-level students, inflating phoneme error rate by a factor of four relative to greedy decoding on the same artifact — a calibration artifact with implications for any evaluation of low-confidence autoregressive models. Twelve models across four languages are released under BSD-3-Clause, every one traceable to a documented protocol. Where public benchmarks permit protocol-matched comparison, the teacher tier ranks among the best dedicated systems measured — second only to a frontier proprietary LLM on the SadeedDiac-25 Arabic diacritization leaderboard, and above the published SentenceBench homograph accuracy for Persian. @@ -32,7 +32,7 @@ Our goal is to serve the second class under the guarantees of the first. This im |=== |Requirement |Consequence -|Byte-identical output across runtimes |A single artifact contract with per-member checksums and cross-runtime golden sets +|Byte-identical output across runtimes (full precision) |A single artifact contract with per-member checksums and cross-runtime golden sets; quantized artifacts carry measured quality deltas instead — near-tie decisions flip across hardware |No training code in consumers |Self-describing zips (metadata + ONNX graphs) loadable by any runtime with a zip reader, SHA-256, and an ONNX runtime |Verifiable quality claims |Every metric traceable to a documented harness; metadata generated from, and checked against, a public results log |Old-runtime compatibility |ONNX opset pinned to 14 (the floor of the oldest supported consumer runtime) @@ -88,8 +88,8 @@ lower error rate. Published evaluations that decode such models with beams risk reporting the decode, not the model. [[fig-architecture]] -.The phonological-layer pipeline: frozen teachers, gated students, one checksummed artifact, three runtimes, byte-identical output. -[image::paper-assets/architecture.png[The IMF v1 pipeline]] Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output. +.The phonological-layer pipeline: frozen teachers, gated students, one checksummed artifact, three runtimes, byte-identical output at full precision. +[image::paper-assets/architecture.png[The IMF v1 pipeline]] Teachers are frozen before any student trains; students pass pre-registered error budgets and a precision-aware parity gate before the artifact enters the index; three runtimes resolve the same checksummed zip and produce byte-identical output at full precision (quantized artifacts: measured quality parity). == The artifact contract (IMF v1) [[section-imf]] @@ -394,7 +394,7 @@ The general lesson: when a file "reads corrupt remotely and clean locally," susp *Benchmarks.* Cross-paper comparison in this space is unreliable by construction: protocols differ in evaluator, reference projection, skipped-example handling, and domain (SadeedDiac-25's own motivation). We therefore claim externally only where the full protocol is reproduced (<>) and otherwise publish the harness itself — the Thai test set ships with this work as a citable benchmark (`benchmarks/thai-kaikki-g2p/`). -*Determinism.* Byte-identical output across three runtimes is verified by golden sets generated from the reference implementation. Floating-point nondeterminism across ONNX runtime versions is not addressed beyond opset pinning; the parity gate bounds it at the artifact level. +*Determinism.* Byte-identical output across three runtimes is verified by golden sets generated from the reference implementation, for fp32/fp16 artifacts on all hardware tested. Quantized artifacts (int8, int4) are byte-stable on like hardware but flip near-tie decisions across CPU architectures, so their guarantee is quality-level: each artifact ships its measured delta against full precision, and golden rows assert decode health rather than byte equality. Floating-point nondeterminism across ONNX runtime versions is not addressed beyond opset pinning; the parity gate bounds it at the artifact level. == Conclusion diff --git a/runtime/tests/test_model.py b/runtime/tests/test_model.py index 925796c..d77d0d5 100644 --- a/runtime/tests/test_model.py +++ b/runtime/tests/test_model.py @@ -159,9 +159,16 @@ def test_golden_set_e2e() -> None: def test_golden_matrix() -> None: - """golden-v1 corpus: every model's released zip must reproduce its - golden rows byte-identically. Set GOLDEN_DIR (release checkout); - each golden file maps to a model id resolved from the index.""" + """golden-v1 corpus: released zips must reproduce their golden + rows. fp32/fp16 artifacts byte-identically (cross-hardware byte + stability, measured 2026-09-07); quantized artifacts decode-health + only — int8/int4 diverge at near-ties across hardware, so their + contract is the per-model cer_delta gate, not byte parity. The + precision comes from the loaded artifact's manifest, NOT the + golden filename: some golden names drop the precision suffix + (tha-g2p-small-1.0 resolves to an int8 zip). Set GOLDEN_DIR + (release checkout); each golden file maps to a model id resolved + from the index.""" import glob golden_dir = os.environ.get("GOLDEN_DIR") @@ -174,7 +181,14 @@ def test_golden_matrix() -> None: model = Model.load(model_id) rows = [json.loads(line) for line in Path(path).read_text(encoding="utf-8").splitlines() if line.strip()] assert rows, model_id + exact = model.manifest.precision in ("fp32", "fp16") for row in rows: got = model.translate(row["input"], max_len=max(256, 4 * len(row["input"]))) - assert got == row["output"], (model_id, row["input"]) + if exact: + assert got == row["output"], (model_id, row["input"]) + else: + # Length floor from the measured cross-hardware spread + # (RESULTS.md 2026-09-07): stop points move, mid-text + # stays substantially intact. + assert len(got) >= 0.25 * len(row["output"]), (model_id, len(got)) del model