feat: initial ml-models framework skeleton

Unified training framework for Interscript ML-powered maps. One repo
trains every ML model (rababa_arabic, rababa_hebrew, secryst_thai_ipa,
and future tasks). Each task is a config + data module; the framework
is shared.

Layout:
- src/framework/: MECE abstractions (config, registry, data, model,
  trainer, evaluator, exporter, pipeline). Adding a new task = one
  directory under src/tasks/, zero framework edits.
- src/tasks/{rababa_arabic,rababa_hebrew,secryst_thai_ipa}/: each
  has config.yaml + data.py + student.py + metrics.py.
- src/cli.py: `interscript-ml {list,train,evaluate,export,publish}`.
- tests/: 37 tests, all green (CPU-only; no torch required).
- scripts/: setup_env.sh, fetch_data.sh, train.sh, export.sh,
  benchmark.sh, publish.sh.
- docs/{architecture,model_card,training_log}.md.

Design follows project conventions: OCP (registry plugins), MECE
(each module owns one concern), DRY (epoch loop, edit-distance, ONNX
export written once), no hand-rolled serialization, no require_relative
equivalent (Python has no such concept; tasks register via decorator).

Author: Ronald Tse

.gitattributes

@@ -0,0 +1,5 @@
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,34 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.venv/
+venv/
+build/
+dist/
+
+# Data (fetched via scripts/fetch_data.sh)
+data/raw/
+data/processed/
+data/augmented/
+
+# Model artifacts (Git-LFS tracked if committed at all)
+models/*.onnx
+models/*.safetensors
+models/*.ckpt
+*.tmp
+
+# Notebooks
+.ipynb_checkpoints/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Secrets
+.env
+.env.local
+hf_token.txt

README.md

@@ -0,0 +1,102 @@
+# interscript/ml-models
+
+Unified training framework for Interscript ML-powered maps.
+
+**Status:** skeleton (P0 in TODO.rababa/10). Framework abstractions
+implemented and tested. Task data modules + configs ready; full
+training requires GPU + dataset fetch.
+
+## What this is
+
+One training repo for every ML map in Interscript:
+
+- **rababa_arabic** — Arabic diacritization (adds harakat)
+- **rababa_hebrew** — Hebrew diacritization (adds nikud)
+- **secryst_thai_ipa** — Thai → IPA transliteration
+
+Each task is a **config + data module**. The training framework is
+shared. Adding a new transliteration pair (Khmer → IPA, Japanese →
+Romaji) is one new directory under `src/tasks/` — zero edits to
+framework code.
+
+## Architecture
+
+```
+src/
+├── framework/          # SHARED abstractions (MECE)
+│   ├── config.py       # TaskConfig loaded from YAML
+│   ├── registry.py     # Plugin registry (OCP)
+│   ├── data.py         # DataModule ABC
+│   ├── model.py        # ModelModule ABC (teacher + student)
+│   ├── trainer.py      # BaseTrainer + FineTune + Distill (DRY)
+│   ├── evaluator.py    # BaseEvaluator + edit_distance + DER/PER utils
+│   ├── exporter.py     # OnnxExporter ABC
+│   └── pipeline.py     # TrainingPipeline orchestrator
+├── tasks/
+│   ├── rababa_arabic/  # config.yaml + data.py + student.py + metrics.py
+│   ├── rababa_hebrew/
+│   └── secryst_thai_ipa/
+└── cli.py              # python -m src.cli train --task rababa_arabic
+```
+
+## Design principles (project conventions)
+
+- **OCP** — adding a task = one new directory. Adding a metric, model
+  architecture, or data source = one new subclass + `@register_*`
+  decorator. Framework code is never edited.
+- **MECE** — each module owns one concern. Data has no knowledge of
+  model architecture. Model has no knowledge of trainer. Trainer has
+  no knowledge of evaluator.
+- **DRY** — the epoch loop, edit-distance math, and ONNX export
+  wrapper are written once.
+- **Model-driven, semantically-driven** — class names mirror domain
+  concepts (`RababaArabicData`, `DEREvaluator`, `SecrystThaiIpaStudent`).
+- **Performance** — frozen dataclasses for config; lazy imports for
+  torch so framework tests run without GPU deps.
+
+## Quick start
+
+```bash
+scripts/setup_env.sh               # creates .venv, installs deps
+scripts/fetch_data.sh              # fetch raw datasets (set env vars first)
+scripts/train.sh rababa_arabic     # full training pipeline
+scripts/export.sh rababa_arabic    # export student to ONNX
+scripts/publish.sh rababa_arabic   # upload to HuggingFace Hub
+```
+
+Or via the CLI directly:
+
+```bash
+python -m src.cli list
+python -m src.cli train --task rababa_arabic --data-root data --out-root models
+python -m src.cli evaluate --task secryst_thai_ipa
+python -m src.cli export --task rababa_hebrew
+```
+
+## Adding a new task
+
+1. Create `src/tasks/<name>/config.yaml` (copy from an existing task).
+2. Create `src/tasks/<name>/data.py` extending `DataModule`, decorated
+   with `@register_data_module("<name>_data")`.
+3. Create `src/tasks/<name>/student.py` extending `ModelModule`,
+   decorated with `@register_model_module("<name>_student")`.
+4. Create `src/tasks/<name>/metrics.py` extending `BaseEvaluator`,
+   decorated with `@register_evaluator("<metric>")`.
+5. Run `python -m src.cli train --task <name>`.
+
+That's it. No framework edits.
+
+## Tests
+
+```bash
+pytest -v
+```
+
+Framework tests run without torch (CPU-only, fast). Training and ONNX
+export tests are gated behind `@pytest.mark.gpu` and require the
+`[train]` and `[export]` extras.
+
+## License
+
+MIT. Model weights are released under their own licenses (see
+`docs/model_card.md` per task).

configs/defaults.yaml

@@ -0,0 +1,27 @@
+# Default training hyperparameters. Override per-task in config.yaml.
+
+default:
+  optimizer: adamw
+  scheduler: cosine
+  warmup_ratio: 0.03
+  fp16: true
+  seed: 42
+  log_every: 50
+  save_every: 1000
+  grad_clip: 1.0
+
+distillation:
+  # Mix of cross-entropy (gold labels) and KL divergence (teacher logits)
+  alpha: 0.5          # weight on CE; (1-alpha) on KL
+  temperature: 4.0    # softmax temperature for KL
+
+finetune:
+  # LoRA config for teacher fine-tune
+  lora_r: 16
+  lora_alpha: 32
+  lora_dropout: 0.05
+  target_modules:
+    - q_proj
+    - v_proj
+    - k_proj
+    - o_proj

docs/architecture.md

@@ -0,0 +1,48 @@
+# Architecture decision record: LLM teacher + distilled student
+
+## Context
+
+The legacy rababa model used a Tacotron CBHG encoder-decoder with
+attention. Tacotron is a speech synthesis architecture repurposed for
+text diacritization. Quantizing it (the original "make it small"
+proposal) makes a bad architecture smaller but not better.
+
+## Decision
+
+Replace Tacotron with a two-stage training pipeline:
+
+1. **Teacher**: Qwen3.5-4B-Instruct fine-tuned via LoRA on Tashkeela++.
+   The LLM already has strong Arabic language understanding from
+   pretraining; fine-tuning teaches it the diacritization task.
+2. **Student**: A 6M-parameter character-level transformer distilled
+   from the teacher. Small enough to ship in the browser (~6MB ONNX).
+
+## Why this works
+
+- LLMs already model Arabic morphotactics — diacritization is a
+  downstream capability, not a from-scratch learning problem.
+- Distillation transfers the teacher's softened output distribution
+  to the student. The student learns richer signal than gold labels
+  alone (the "dark knowledge" of near-misses).
+- Character-level (not subword) is the right granularity for
+  diacritization: harakat attach to single characters, and a 6MB model
+  is fast enough for in-browser inference.
+
+## Alternatives considered
+
+- **Quantized Tacotron.** Rejected: makes a bad model smaller, not
+  better. DER barely improves; latency barely drops.
+- **ByT5-small fine-tune as teacher.** Plausible fallback if Qwen3.5
+  proves too large. ByT5 is character-native, smaller, and already
+  proven for text-infilling tasks. Kept as a backup in `models/`.
+- **Distillation from a speech model (Nemotron, Whisper).** Rejected:
+  those models output speech features, not text. We'd be converting
+  between modalities unnecessarily.
+
+## Risks
+
+- LLM teacher may hallucinate plausible-but-wrong diacritization on
+  out-of-domain text. Mitigated by training-data augmentation with
+  adversarial examples.
+- Student may underfit if teacher logits collapse to one-hot.
+  Mitigated by KL temperature = 4.0 + alpha = 0.5 (equal CE and KL).

docs/model_card.md

@@ -0,0 +1,46 @@
+# Model Card: rababa_arabic
+
+## Model details
+
+- **Name:** `interscript/rababa_arabic`
+- **Type:** Character-level transformer (4 layers, 256 dim, 4 heads)
+- **Trained from:** distilled from Qwen3.5-4B-Instruct teacher
+- **License:** MIT (code), CC-BY-4.0 (model weights)
+
+## Training data
+
+- Tashkeela++ (~2M verses of diacritized Arabic)
+- OpenDiacritizer (augmentation set)
+- Synthetic labels generated by fine-tuned Qwen3.5-4B teacher
+
+## Intended use
+
+- Diacritization of Modern Standard Arabic text
+- Inference in browser via ONNX runtime (~6MB)
+- Powering the `var-ara-Arab-Arab-rababa` map in interscript.org
+
+## Out of scope
+
+- Dialectal Arabic (Egyptian, Gulf, Levantine) — not in training data
+- Classical Arabic poetry — different conventions
+
+## Metrics
+
+- DER (Diacritic Error Rate) on held-out set
+- Target: ≤ 5%
+- p95 inference latency: ≤ 30ms per word
+
+## Ethical considerations
+
+This model reproduces biases in its training data. Tashkeela++ is
+heavily Quranic + religious text; diacritization choices on secular
+text may differ from speaker expectations.
+
+## How to cite
+
+```
+@software{interscript_rababa_arabic,
+  title = {Interscript rababa_arabic diacritization model},
+  url = {https://huggingface.co/interscript/rababa_arabic}
+}
+```

docs/training_log.md

@@ -0,0 +1,17 @@
+# Training log
+
+Append new entries at the bottom. Format:
+
+```
+## YYYY-MM-DD — task_name — notes
+- params: {key: value}
+- DER/PER: x.xx
+- wall time: Xh
+```
+
+## 2026-08-01 — framework skeleton committed
+
+- Initial framework: config, registry, data, model, trainer,
+  evaluator, exporter, pipeline.
+- Task packages: rababa_arabic, rababa_hebrew, secryst_thai_ipa.
+- CPU-only unit tests pass. No GPU training yet.

pyproject.toml

@@ -0,0 +1,71 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "interscript-ml-models"
+version = "0.1.0"
+description = "Unified training framework for Interscript ML models (rababa + secryst)"
+readme = "README.md"
+license = { text = "MIT" }
+requires-python = ">=3.10"
+authors = [{ name = "Interscript Project" }]
+keywords = ["nlp", "transliteration", "diacritization", "onnx", "distillation"]
+
+dependencies = [
+  "numpy>=1.26",
+  "pyyaml>=6.0",
+  "tqdm>=4.66",
+  "pydantic>=2.5",
+]
+
+[project.optional-dependencies]
+train = [
+  "torch>=2.2",
+  "transformers>=4.42",
+  "datasets>=2.18",
+  "accelerate>=0.27",
+  "peft>=0.10",
+  "sentencepiece>=0.2",
+]
+export = [
+  "onnx>=1.16",
+  "onnxruntime>=1.17",
+]
+publish = [
+  "huggingface_hub>=0.23",
+  "omegaconf>=2.3",
+]
+dev = [
+  "pytest>=8.0",
+  "pytest-cov>=5.0",
+  "ruff>=0.4",
+  "mypy>=1.10",
+]
+
+[project.scripts]
+interscript-ml = "src.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["framework*", "tasks*"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra -q --strict-markers"
+markers = [
+  "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+  "gpu: marks tests requiring a GPU",
+]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "B", "UP", "SIM", "C4"]
+
+[tool.mypy]
+python_version = "3.10"
+strict = true
+warn_unused_configs = true

scripts/benchmark.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Run all benchmarks on a trained model.
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+
+TASK="${1:-}"
+if [[ -z "$TASK" ]]; then
+  echo "Usage: $0 <task_name>" >&2
+  exit 1
+fi
+
+# shellcheck disable=SC1091
+source .venv/bin/activate 2>/dev/null || true
+
+python -m src.cli evaluate --task "$TASK" --data-root data --out-root "models/$TASK"

scripts/export.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+# Export a trained task to ONNX. Usage: scripts/export.sh <task_name>
+set -euo pipefail
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+cd "$ROOT"
+
+TASK="${1:-}"
+if [[ -z "$TASK" ]]; then
+  echo "Usage: $0 <task_name>" >&2
+  exit 1
+fi
+
+# shellcheck disable=SC1091
+source .venv/bin/activate 2>/dev/null || true
+
+python -m src.cli export --task "$TASK" --data-root data --out-root "models/$TASK"