From fc8bb2cb2745d5344e2a61405c915c03b7855503 Mon Sep 17 00:00:00 2001 From: Ronald Tse Date: Wed, 9 Sep 2026 11:42:06 +0200 Subject: [PATCH] docs: Libraries and How-we-measure pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Libraries: install and first-run code for all three runtimes, the CLI (deterministic + neural), the REST API, the playground, and the model catalog pointer — one entry point for every way to use Interscript. How we measure: where numbers come from, how to re-derive them (benchmark, tool, predictions, teachers), the corrected-number example, the cross-runtime guarantee and its measured boundary, and the negative results — plain language, no claims without a link. --- src/content/docs/How_we_measure.adoc | 70 +++++++++++++++++ src/content/docs/Libraries.adoc | 110 +++++++++++++++++++++++++++ 2 files changed, 180 insertions(+) create mode 100644 src/content/docs/How_we_measure.adoc create mode 100644 src/content/docs/Libraries.adoc diff --git a/src/content/docs/How_we_measure.adoc b/src/content/docs/How_we_measure.adoc new file mode 100644 index 0000000..9c07fd9 --- /dev/null +++ b/src/content/docs/How_we_measure.adoc @@ -0,0 +1,70 @@ += How we measure + +Every quality number we publish is measured, recorded, and can be +re-derived by anyone. This page explains how. + +== Where numbers come from + +Each model's accuracy figure comes from a fixed evaluation protocol, +run on the complete test set, and is recorded in a public results log: +https://github.com/interscript/interscript-ml/blob/main/docs/RESULTS.md[docs/RESULTS.md]. +The log records the number, a confidence interval where the protocol +supports one, and a checksum of the exact training labels used. + +When a number turned out to be wrong, we corrected it in public. One +example: the Arabic model ara-diac-small-2.0 was first published with a +score of 4.82. Two independent re-measurements of the released files +agreed on a different value, 5.08, and the original number was +withdrawn (the correction entry in the results log explains what +changed and what it affected). + +== Re-deriving our results + +The pieces are public: + +* The benchmark: https://huggingface.co/datasets/Misraj/SadeedDiac-25[SadeedDiac-25] +* The scoring tool: `pip install interscript-ml-tools[sadeed]` +* Our raw predictions for every published run: the + https://github.com/interscript/interscript-ml/releases/tag/frontier-predictions-v1[frontier-predictions-v1] + release +* The teacher checkpoints behind the training results: the + https://github.com/interscript/interscript-ml/releases/tag/teachers-arabic-v1[teachers-arabic-v1] + release + +To re-score one of our runs: + +``` +pip install interscript-ml-tools[sadeed] +interscript-sadeed-eval score \ + --preds \ + --data Misraj/SadeedDiac-25 +``` + +The tool reproduces the published numbers exactly; that is how the +incorrect 4.82 above was found. + +== Cross-runtime correctness + +The same model file must produce the same output in TypeScript, +Python, and Ruby. We check this with a shared corpus of reference +inputs and outputs (the +https://github.com/interscript/interscript-ml/releases/tag/golden-v1[golden-v1] +release), generated from the released artifacts themselves. + +The guarantee has a measured boundary. Models stored at full precision +(fp32, fp16) produce byte-identical output on every machine we tested. +Quantized models (int8, int4) run faster and smaller, but on different +CPU types their outputs can differ slightly: tiny numerical +differences flip decisions that the model is nearly undecided on. For +those models the guarantee is quality-level, not byte-level, and each +artifact carries its measured quality difference from the full +precision version in its metadata. + +== Negative results + +Not every approach worked. Attempts to close the quality gap between +the small student models and their teachers by adding more classical +training text, changing the text mixture, or training on the model's +own mistakes all failed, with measurements recorded in the results +log. We publish these alongside the successes because they define what +the current models are and are not. diff --git a/src/content/docs/Libraries.adoc b/src/content/docs/Libraries.adoc new file mode 100644 index 0000000..8d0fb66 --- /dev/null +++ b/src/content/docs/Libraries.adoc @@ -0,0 +1,110 @@ += Libraries + +Interscript ships as libraries for three languages, a command-line +tool, and a web API. All of them run the same transliteration systems +and the same neural models, and all model artifacts are +checksum-verified when they load. + +== TypeScript (Node and browsers) + +Install: + +``` +npm install interscript +``` + +Transliterate with one of the 289 authority systems: + +```ts +import { transliterateAsync } from "interscript" + +const out = await transliterateAsync( + "bgnpcgn-ukr-Cyrl-Latn-2019", + "Антон", +) +``` + +Use a neural model (for example, Arabic diacritization). The model +downloads once, is verified against its checksum, and is cached: + +```ts +import { imf } from "interscript/ml" + +const resolved = await imf.resolve("ara-diac-layerdrop-1.0-int4") +const model = await imf.IMFModel.fromZipBytes(resolved.bytes) +const vocalized = await model.translate("السلام عليكم") +``` + +== Python + +Install: + +``` +pip install interscript-ml +``` + +Load a model and run it: + +```python +from interscript_ml import Model + +model = Model.load("ara-diac-small-2.1-int8") +print(model.translate("قوله فحكمها في الوفاة")) +``` + +== Ruby + +Install: + +``` +gem install secryst +``` + +Load and translate: + +```ruby +require "secryst" + +model = Secryst::Model.load("ara-diac-small-2.1-int8") +model.translate("قوله فحكمها في الوفاة") +``` + +== Command line + +The deterministic systems: + +``` +echo "Антон" | npx interscript t bgnpcgn-ukr-Cyrl-Latn-2019 +``` + +The neural layer: + +``` +echo "السلام عليكم" | npx interscript m ara-diac-layerdrop-1.0-int4 +``` + +Run `npx interscript --help` for the full command list, including bulk +processing and script detection. + +== Web API + +A hosted REST API serves both layers: +https://api.interscript.org[api.interscript.org]. Its documentation, +including batch endpoints and asset serving, lives at +https://api.interscript.org/docs[/docs]. + +== Try it in the browser + +The link:/playground[playground] runs the full stack in your browser: +pick any system, optionally vocalize Arabic text first, and copy the +exact code for the configuration you selected. Nothing is sent to a +server. + +== Which models exist + +The model catalog with measured quality numbers for every artifact is +on the link:/ml[neural layer page]. The machine-readable index that the +libraries resolve against is +https://github.com/interscript/interscript-ml/blob/main/models.yaml[models.yaml] +in the https://github.com/interscript/interscript-ml[interscript-ml] +repository.