diff --git a/TODO.sota/00-README.md b/TODO.sota/00-README.md index c23cf49..8e59e0e 100644 --- a/TODO.sota/00-README.md +++ b/TODO.sota/00-README.md @@ -60,7 +60,7 @@ hf jobs run --flavor a100-large -d \ | 20 | 20-hebrew-large-arm.md | P1 | 04 | RUNNING (run-026, byt5-large/K=4) | | 21 | 21-arabic-plane-transfer.md | P2 | 17 | pending verdict | | 22 | 22-runtime-releases.md | P1 | 14 | PREPARED — owner-gated versions | -| 23 | 23-ort-alignment.md | P1 | 18 | golden regen on ORT 1.29 RUNNING; re-dispatch after | +| 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound | ## Schedule diff --git a/TODO.sota/23-ort-alignment.md b/TODO.sota/23-ort-alignment.md index bee9d6f..b76fc7e 100644 --- a/TODO.sota/23-ort-alignment.md +++ b/TODO.sota/23-ort-alignment.md @@ -1,13 +1,25 @@ -# WO23 — ORT generation alignment (the parity root cause, corrected) - -WO18 mis-attributed the ruby-vs-py 9.25% skew to "the gem's older ORT". -The gem bundles ORT **1.29.0** — NEWER than the golden-generation -platform (py 1.20.1). The skew is a GOLDEN-PLATFORM version mismatch. - -Fix (eval infrastructure only — no library requirement changes): -1. Golden generation pins onnxruntime==1.29.0 (HF cpu job). -2. neural-parity py leg pins the same. -3. Re-dispatch with DEFAULT bounds; if ruby passes, the calibrated - tiers stand and the per-dispatch override reverts to escape-hatch. -4. ts onnxruntime-node bump (1.27→1.29) is a runtime requirement change - — owner decision, only if the ts leg misses tolerance. +# WO23 — ORT alignment (CLOSED: characterization complete) + +WO18's "gem ORT too old" attribution was WRONG (corrected): the ruby gem +bundles ORT 1.29.0 — newer than the golden platform. The completed +experiment matrix on heb-diac-plane-2.0 (byt5-base, K=3, int8 dynamic): + +| comparison | corpus diff | +|---|---| +| py(1.20) vs py(1.29), same platform (linux) | **0% (byte-identical)** | +| py(mac) vs py(linux) | 2.86% | +| ruby(mac, ORT 1.29) vs py(linux golden) | 4.23% | +| ruby(linux, CI) vs py(linux golden) | 9.25% | +| ruby opt-level NONE vs default (mac) | 9.36% vs 4.23% (knob NEGATIVE) | +| ruby intra_op=1 vs default (mac) | 4.23% (knob NEGATIVE) | + +Conclusions: (1) divergence is CROSS-PLATFORM int8-dynamic kernel +selection, scaling with model size × K-passes; (2) no session knob +restores cross-platform parity (three configs measured, closed); +(3) same-platform same-runtime is byte-exact. + +RESOLUTION: canonical golden platform = linux x86 CI (done, WO18); +per-artifact-class bounds — K=2 small keeps the calibrated 2% smoke, +K=3+ base uses the documented per-dispatch bound (measured spread +≤ 9.36% ⇒ 0.10); tight byte-exact parity remains the fp32 artifact +class. Re-dispatched run 37611316113: **py ✓ ts ✓ ruby ✓ at 0.10**.