Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/RESULTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -752,3 +752,37 @@ Arabic adapted-init collapses under the strong one). The lite tier
remains run-009 (5.78); init-source closes negative and the depth
axis now reads 2-of-3 negative. Remaining architecture lever:
TODO.impl/10 (lexical memory), gated as before.

## Framing completes as a distribution×precision matrix (2026-09-12, final)

The same single-vs-batched greedy test (5 golden rows x 96 steps, same
machine, ORT 1.23 everywhere), run to its endpoints:

| runtime | precision | single==batched |
|---|---|---|
| Python ORT | fp32 | 480/480 |
| Python ORT | dynamic int8 | 480/480 |
| Python ORT | static int8 | 480/480 |
| onnxruntime-node | fp32 | **101/485** |
| onnxruntime-node | dynamic int8 | 97/485 |
| onnxruntime-node | static int8 | 89/485 |

The framing instability is NOT a quantization property: the node build
diverges across batch shapes at fp32, and static activation scales
(pre-computed into the graph — TODO.impl/11's proposed fix) do not
repair it. It is a property of the onnxruntime-node kernel paths
(multi-token inputs take numerically different code paths than
single-token inputs), absent from the Python build at every precision.

Contract consequences, final form:
- cross-FRAMING parity holds under the Python reference at all
precisions and does not hold under onnxruntime-node at ANY precision
- the shipped TS runtime is unaffected in practice: every shipped path
(translate, worker, CLI) is single-framing; framing becomes a
parity variable exactly when a runtime mixes batch shapes
(speculative decode, batched serving) — which is why the speculative
tier degraded and was pulled
- static int8 remains a POSITIVE byproduct: quality-clean vs fp32 on
this sample (0/480 drift) and ~8% faster than dynamic on CPU —
candidate for the export path on its own merits, decided by
full-set quality, not framing
18 changes: 12 additions & 6 deletions docs/paper-c.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -96,12 +96,18 @@ cross-runtime nor cross-hardware studies cover:
drafting, batched verification) — acceptance figures are
framing-relative and must disclose the framing

Contract consequence: our quantized quality-parity guarantee is
framing-scoped (the framing is part of the protocol that a cer_delta
gate pins). The proposed structural fix is static activation scales
(quantize_static with calibration over both framings), which removes
the per-run scale computation entirely — pre-registered as
TODO.impl/11 with framing-equality as the gate.
Contract consequence, measured to its endpoints: framing stability is
a property of the RUNTIME DISTRIBUTION, not the graph. The same
single-vs-batched test passes 480/480 under the Python build at fp32,
dynamic-int8, AND static-int8 — and fails under onnxruntime-node at
every precision including fp32 (89-101/485). The proposed static-scale
fix (calibrated quantize_static) does not repair it: the node build's
kernels differ across batch shapes at float precision, before
quantization enters. Shipped paths are single-framing by construction,
so byte parity as published is unaffected; framing becomes a parity
variable precisely when a runtime mixes batch shapes (speculative
decode, batched serving) — and acceptance or parity figures measured
under one framing do not transfer to another.

== 6. Reproducibility discipline
- subset-overstatement: five instances, up to 3.2x inflation
Expand Down
Loading