Skip to content

fix(playground): pull the speculative tier - quantized framing degrades output - #185

Merged
ronaldtse merged 1 commit into
mainfrom
fix/playground-speculative-pull
Sep 12, 2026
Merged

ronaldtse merged 1 commit into
mainfrom
fix/playground-speculative-pull

Conversation

@ronaldtse

Copy link
Copy Markdown
Contributor

Pulls the speculative vocalization option introduced yesterday. New measurement (interscript-ml RESULTS.md, 2026-09-12) found two disqualifiers on quantized artifacts:

  1. Decode framing changes the decode. Dynamic-int8 ONNX graphs compute activation quantization scales per fed tensor, so single-step and batched decode paths produce materially different numerics on the same machine. The batched-verifier path measured a word-short output on a canonical row ("امُ عليكم" vs "السلام عليكم") - a quality degradation, not byte noise.
  2. Acceptance was framing-relative. The 0.9886 probe acceptance was measured under uniform plain-path framing; under the runtime's real framing it measured 0.4614. And int4 CPU kernels are slower than int8 here, so there was no speed story either.

The SpeculativeModel API stays in interscript@5.4.x as measurement infrastructure with the constraint documented; the playground serves only the verified tiers. All 255 tests green.

…es output

Measurement behind the tier (interscript-ml RESULTS.md 2026-09-12):
dynamic-int8 decoder graphs quantize activations per fed tensor, so
single-step and batched decode framings produce materially different
numerics - the batched-verifier output lost a word on a canonical
test row, and runtime acceptance measured 0.46 vs the probe's 0.99
(uniform-framing artifact). int4 CPU kernels are also slower than
int8 here, so the tier had no speed story either. SpeculativeModel
stays in the npm API as measurement infrastructure with the
constraint documented; the playground serves only the verified tiers.
@ronaldtse
ronaldtse merged commit 7d8e2f4 into main Sep 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant