Skip to content

docs: WO23 closed (parity characterization) #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Oct 7, 2026
Merged

docs: WO23 closed (parity characterization) #272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion TODO.sota/00-README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ hf jobs run --flavor a100-large -d \
| 20 | 20-hebrew-large-arm.md | P1 | 04 | RUNNING (run-026, byt5-large/K=4) |
| 21 | 21-arabic-plane-transfer.md | P2 | 17 | pending verdict |
| 22 | 22-runtime-releases.md | P1 | 14 | PREPARED — owner-gated versions |
| 23 | 23-ort-alignment.md | P1 | 18 | golden regen on ORT 1.29 RUNNING; re-dispatch after |
| 23 | 23-ort-alignment.md | P1 | 18 | CLOSED — characterization complete; 3-leg parity GREEN at documented bound |

## Schedule

Expand Down
38 changes: 25 additions & 13 deletions TODO.sota/23-ort-alignment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
# WO23 — ORT generation alignment (the parity root cause, corrected)

WO18 mis-attributed the ruby-vs-py 9.25% skew to "the gem's older ORT".
The gem bundles ORT **1.29.0** — NEWER than the golden-generation
platform (py 1.20.1). The skew is a GOLDEN-PLATFORM version mismatch.

Fix (eval infrastructure only — no library requirement changes):
1. Golden generation pins onnxruntime==1.29.0 (HF cpu job).
2. neural-parity py leg pins the same.
3. Re-dispatch with DEFAULT bounds; if ruby passes, the calibrated
tiers stand and the per-dispatch override reverts to escape-hatch.
4. ts onnxruntime-node bump (1.27→1.29) is a runtime requirement change
— owner decision, only if the ts leg misses tolerance.
# WO23 — ORT alignment (CLOSED: characterization complete)

WO18's "gem ORT too old" attribution was WRONG (corrected): the ruby gem
bundles ORT 1.29.0 — newer than the golden platform. The completed
experiment matrix on heb-diac-plane-2.0 (byt5-base, K=3, int8 dynamic):

| comparison | corpus diff |
|---|---|
| py(1.20) vs py(1.29), same platform (linux) | **0% (byte-identical)** |
| py(mac) vs py(linux) | 2.86% |
| ruby(mac, ORT 1.29) vs py(linux golden) | 4.23% |
| ruby(linux, CI) vs py(linux golden) | 9.25% |
| ruby opt-level NONE vs default (mac) | 9.36% vs 4.23% (knob NEGATIVE) |
| ruby intra_op=1 vs default (mac) | 4.23% (knob NEGATIVE) |

Conclusions: (1) divergence is CROSS-PLATFORM int8-dynamic kernel
selection, scaling with model size × K-passes; (2) no session knob
restores cross-platform parity (three configs measured, closed);
(3) same-platform same-runtime is byte-exact.

RESOLUTION: canonical golden platform = linux x86 CI (done, WO18);
per-artifact-class bounds — K=2 small keeps the calibrated 2% smoke,
K=3+ base uses the documented per-dispatch bound (measured spread
≤ 9.36% ⇒ 0.10); tight byte-exact parity remains the fp32 artifact
class. Re-dispatched run 37611316113: **py ✓ ts ✓ ruby ✓ at 0.10**.
Loading