[BlivionIaG]

What changes

Add native Metal image preparation, vision encoder and projector for LFM3-VL-2B from the official MLX 4/6/8-bit checkpoints. Projected image rows feed the LFM3 decoder and existing lane engine, including image order/content handling, prompt caching, verification, snapshots and concurrent serving. Python is used only for development references and clients.

Refs #1491. Depends on admission #1513 and text #1514. The PRs target upstream main until their foundations land; review the vision-only difference.

flowchart LR
    I[Image pixels] --> V[Vision encoder and projector]
    V --> E[Decoder embeddings]
    T[Text tokens] --> E
    E --> D[LFM3 decoder and existing lane engine]

Loading

Code pointers

• Native image integration.
• Image preparation.
• Vision encoder and projector.
• Projected image rows supplied to the decoder.
• Vision kernels.
• Vision contract checks.
• Image host tests.

Validation and numerical evidence

Pixel normalization correction: fused float32 normalization now matches the cited transformers/torchvision processor patch hash. The new regression fails before the change and passes after; host build 12/12 and image tests 26/26. Corrected-pixel vision checks reproduce all 143,360 projected words for VL4 and VL6 using the named rounded graph. Actual vision tensor, parameter, input and output identity supports VL6-to-VL8 vision-only transfer. New ideal FP64 metrics remain recorded. Earlier image receipts retain their original pixel scope; Corrected-pixel serving checks pass 183/183 at VL4/6/8, including 27 cancellation checks and synthetic ordered-image requests. Corrected-input decoder references reproduce all 34,560,000 native BF16 words under the prescribed rounded graph across three independent 90-row runs. Ideal FP64 all-row top differences/disjoint sets are 2/1, 4/4 and 7/2 at 4/6/8 bits; suffix counts are 0/0, 1/1 and 1/0. Every difference and both margins remain listed; no ideal-FP64 acceptance is asserted.

Recipe and checkpoint revisions · Receipts and reproduction tools

• The combined range-correction/norm-fusion candidate passed locked local M4 qualification across all six checkpoints: 408/408 state checks, 303/303 serving checks, including sampled/cancellation/image cases, and 394,752,000 logit bytes identical to the previous public runtime on the retained checkpoint fixtures. The range receipt separates candidate execution from the matching applied-source build: 78/78 steps, formatting passed, and zero new lean findings over 428 inherited.
• The attention correction preserves finite contributions lost by FP32 underflow/overflow. Registered original-fail/new-pass cases use unchanged FP64 bounds; disabling repair or clearing risk makes the failures return. Guarded M5 testing separately passed the current 70-kernel LFM3 contract set and normal limits. These targeted results are not a complete-backend pass.
• Paired projections, shared convolution and fused post-add/next-norm dispatch retain the original rounding boundaries. Their individual receipts identify tested source/binary hashes, exact-output checks and controls. Live CI status is shown by the PR checks.

Independent prompt evidence retains all 5,581 teacher-forced rows, changed top tokens and both margins. Named production-rounding replay resolves disjoint maximum sets; 19 lowest-index differences remain overlapping maxima. All six genuine one-row decode captures after 63 identical history tokens have zero top differences. No new tie policy or tolerance is asserted.

The frozen real-cat whole-pipeline comparison matches all 16 suffix tops at 4/6 bits but has two disjoint maxima against ideal FP64 at 8 bits. An independent CPU arithmetic replay, explicitly delegating named scalar functions to Metal, reproduces all native vision/decoder stages and logits byte-for-byte. This establishes rounding causality, not independent-FP64 acceptance. The FP32 supplement retains both differences in pure FP32; with model BF16 boundaries it has zero disjoint suffix maxima, while differences remain in the full 90-row scope. All original comparisons remain available for review.

The earlier frozen full M5 backend run at 8955faee0 reports 310 kernels: 309 pass, one inherited Qwen4 HC failure, two known findings; all 317 controls red; zero limit problems. The Qwen4 failure reproduces on pinned base b30d5b9d. It is not a global pass and does not qualify later kernels by implication.

Measurements and qualification

Current M5 long-context checks: all 26 requests completed: VL6/8 text prompts at 32k/64k, text4/6/8 at 131072/262128 prompt tokens, and VL4/6/8 at 127984 prompt tokens, each repeated in a fresh process. All return 16 tokens with zero cached input; all 13 repeated output-token hashes match. All 16 servers exit zero, all three packet leases are released, and the controller exits zero. These are native text-context checks, including VL checkpoints; they do not establish image-context, FP64 or comparative speed acceptance.

Extracted text artifact: 15 focused state checks, 120 serving checks and 18 cancellation checks passed across text 4/6/8-bit. Six prompt/decode comparisons match 49,920,000 bytes against the combined artifact. Public native/forward/exact build: 65/65 steps.

Final 6/8-bit checks: 272 state checks, 202 serving checks and 30 cancellation checks passed. Current outputs match 145,981,440 retained native bytes. Six distinct VL6 numerical contexts reproduce native logits exactly under the prescribed rounded graph; ideal FP64 differences remain listed. Long-context checks are recorded below.

Current M5 backend: 315/316 kernels passed, one Qwen4 HC failure matching the recorded baseline case, two known affine nonfinite classes, 321/321 controls red, zero limit problems. Actual contract exit: 1.

M5 four-bit code decode, solo / four streams: text 315.75 / 861.85 tok/s, standard 344.20 / 995.0; VL 298.9 / 835.65, standard 325.8 / 947.75. Requests produce 256 tokens. Cold requests produce 16 tokens with zero cached input and a 70k request budget. Cache allocation differs between engines; standard reply hashes are unavailable.

Four-bit M5 measurements, 6/8-bit M5 measurements, M4 measurements. The 6/8-bit packet records 192 repetitions, 480 primary replies and 32 cold replies. Native cold 2k / 8k TTFT spans 0.85–0.96 / 4.51–4.99 s; standard spans 0.15–0.17 / 0.38–0.39 s.

Final local artifact: build 72/72; tests 7,232 passed, 45 skipped; LFM3 contracts 70/70, two controls red, zero limit problems; four-bit serving 101/101, including 15 cancellation checks. Formatting passes; lean has zero new findings over 428 inherited. Full-backend execution on M4 stops at a tensor-unit requirement. Final 6/8-bit short-model qualification is recorded in the linked receipt; long-context checks are recorded above.

Four-bit numerical ledger: all four distinct observed disjoint contexts are reproduced by the independent prescribed-rounding graph. This includes all 70 projected image rows and both disputed image-placeholder heads. Ideal FP64 token differences and margins remain recorded. 32k/64k checks cover text4 and VL4 text prompts, with matching fresh-process token hashes.

Remaining checks: shared prompt/decode path requirements, merged-chain integration, and maintainer review of recorded FP64 differences and performance comparisons. The shared prompt/decode implementation is being verified in a private overlay and is not enabled in this PR. Its projection checkpoint checks passed all 66 retained classes at 1, 4, and 64 rows (198 cases; 65,716,704 output words per path). Tile and decode outputs were identical and matched the independently dequantized NumPy FP64 reference rounded to BF16; the current public kernel differed in 4,699 words, all corrected by repair. All cases passed aggregate RMS, worst-column RMS, and maximum absolute error comparisons against that public kernel. The observed repair rate was 33.3853%. The first cost screen stopped at its calibration bound: the one-row text4 attention-Q case measured approximately 7.19 microseconds per public-kernel operation versus 2.90 milliseconds for the tile route and 2.70 milliseconds for the decode twin during calibration. The balanced timing rounds and remaining five cases were not run; these are calibration observations, not qualified performance measurements. A cooperative exact-integer repair replacement passed a 24-case numerical screen; its first calibration case measured about 325 microseconds for the tile route and 128 microseconds for the decode twin versus 7.80 microseconds for the public kernel. Balanced timing rounds did not run because the unchanged calibration cap stopped the screen. The restored-part shared-attention operator passed numerical, identity, and negative-control checks on M5; its ten stable operator-cost cases achieved 0.2595–0.5536 times public-baseline throughput. These private routes remain disabled. Full model and MLX performance qualification remain open.

Weights: LFM Open License v1.0. Checked machines: M4 Pro 48 GB and M5 Ultra 256 GB; these measurements do not establish smaller-memory qualification. ATC issue 1491 remains held.

Prefill optimization status

Performance qualification remains open. Private replay-splitting passed 35 synthetic attention cases on M4 and M5 with identical intermediate buffers, final BF16 output and replay decisions, plus planted controls. Five balanced M5 timing rounds found no useful gain: the complete 64-row attention operation measured 5.686 ms public versus 5.682 ms split at a 2k history, and 3.701 ms versus 3.766 ms at 8k. This experiment remains disabled.

An intentionally unsafe diagnostic that suppresses range-risk predicates and replay measured 15.93x and 2.72x public throughput on the same two synthetic fixtures. It is not a production candidate or an end-to-end speedup; the measurement combines predicate, replay and compiler effects. The next candidate preserves the original risk decisions while reducing their common-case cost. No new prefill/decode/TTFT/ITL serving result against MLX-VLM is claimed yet.

The disabled risk-predicate experiment is now committed in 1543227fc. The native, kernel-contract and developer-tool builds passed, the CPU predicate-equivalence proof passed, and the public M5 targeted contract passed both full/sliding kernels with zero failures or limit problems. The receipt retains the negative M5 timing result: 5.707 to 6.417 ms at 2k and 3.708 to 6.385 ms at 8k. -Dlfm3-risk-fast=true is a developer opt-in; the default remains unchanged. This is a draft experiment, not performance qualification. Current optimization work prioritizes prompt FFN projections; no new matched serving result is claimed.

A subsequent branch-free fused-predicate candidate passed the 35-case M5 attention suite with canonical output identity and both planted controls. Five balanced timing rounds were stable: complete partial-plus-merge operation time fell from 5.678570 to 5.563094 ms at 2k history and from 3.703492 to 3.289172 ms at 8k (1.02162x and 1.12523x throughput). These are synthetic operator results, not full-model or MLX-VLM serving improvements. The default-off candidate and sanitized timing receipt are committed and pushed in de7cf96ed. Native, targeted-contract runner and all developer tools build successfully; full-model validation and FFN optimization continue.

Default-off deferred prompt heads are committed and pushed in 380f2a4b7. Private M4 VL4 checks retain identical state/logits, snapshot/resume and decode behavior at 2k/8k and with injected actual-image projected rows; fresh vision tower execution is not covered. A two-sample-per-variant ABBA serving comparison observed cold TTFT 3.38926→3.37424 s at 2k and 17.14382→17.07869 s at 8k (0.44%/0.38% lower), within the declared baseline drift bound. Both public build modes and host checks pass; public device validation remains pending. This is a small isolated improvement, not MLX parity or full performance qualification.

Isolated M5 VL4 served attention ABBA results are retained in 34c4857a5: cold 2k TTFT 0.859114→0.847420 s and 8k 4.647194→4.433517 s (1.36%/4.60% lower). Both variants use the same source revision with only the attention option differing; deferred heads and projection experiments are disabled. Exact prompt IDs, output text hashes, clean server exits and baseline drift checks pass. Separate full-model evidence covers state/logits and 16 decode IDs. No new decode/ITL measurement or matched MLX-VLM result is claimed. Public deferred-head M4 checks also pass, recorded in 1f41f3c89.

Combined default-off attention plus deferred-head results are retained in fb7ed9681: M5 VL4 cold TTFT 0.861714→0.846410 s at 2k and 4.649977→4.429145 s at 8k (1.78%/4.75% lower). Server prefill time is available only as the aggregate of both requests per leg: median 5.498918→5.262589 s; this is not a per-length prefill measurement. Cross-binary full state/logit hashes and 16 decode IDs match for 2k, 8k and corrected projected-image inputs. Isolated M4 attention results in 3a6511bd3 show 1.08%/2.77% lower TTFT. Full MLX parity and performance qualification remain open; no new decode or ITL claim.

Final matched M5 VL4/6/8 measurements are committed in 74558833c. Composed native versus MLX-VLM 8k TTFT is 4.421767/0.374483 s (4-bit), 4.767471/0.394543 s (6-bit), and 4.293734/0.383868 s (8-bit). The large prefill gap remains. VL4 MLX 2k timing exceeded the unchanged 5% drift limit (5.95%); its ratio is not qualified. The receipt distinguishes server-reported prefill semantics from client TTFT, decode delivery, ITL proxy and makespan; it does not claim true token ITL, MLX token-array parity, real-image serving comparison or full performance qualification. All 18 legs completed; owned cleanup was 12 native exit-zero and six MLX controlled SIGTERM (-15).

The source-free bounded M5 FFN experiment ledger preserves the producer/repair optimization chain, negative results, conditional arithmetic assumptions and audit corrections. Its best private route measures 110.966 µs against a 67.278 µs public baseline on one 64-row, 2048-column gate slice, so it remains disabled and is not a full-model performance result. Portable reuse variants also remain unpromoted; their stable negative measurements are retained separately.

A genuine 128-row, full-7680-column TEXT4 gate now has a five-round operator confirmation: public 478.098 µs versus private matrix candidate 448.585 µs (6.17% less time). All 983,040 candidate BF16 outputs match rounded FP64, with no worse columns; eight fault controls are rejected by the original gates. This is one actual layer-0 gate input, not all layers/classes or serving qualification. Private model integration must include repair-failure detection and the cost of losing the paired gate/up dispatch; no runtime default is changed.

Latest WIP FFN progress: joint gate/up receipt records five balanced M5 operator rounds: full R128 gate+up+activation fell from 955.848 to 921.676 µs (3.57% less GPU time). Both projections match the independently qualified primary outputs; sticky-error injection and its negative control pass. This is a private layer-0 operator experiment, not a serving improvement. The earlier gate-only integration regressed cold TTFT by 0.63% at 2k and 0.78% at 8k and is not promoted. Joint full-model validation, downstream FP64 fidelity and serving timing remain open. This PR stays draft/WIP; no MLX parity is claimed.

[ashhart]

Thanks @BlivionIaG for a careful port and an honest receipt. The same-engine checks are thorough (231 served, with cancellation and recovery), and the rotary fix comes with a test that fails on the old formula.

The merge with main is clean. Here is our answer on the gates.

Fidelity:

• Compare teacher-forced against an FP64 forward of the dequantized checkpoint. MLX is a second opinion, not the reference.
• Compare prompt rows and one-row decode rows separately, on identical input rows, at 4, 6 and 8 bits.
• List every top-1 difference with both tokens and both margins, and report the worst rows, not only an average.
• A difference passes only as a near-tie that a reference rounding where the engine rounds reproduces, as lfm3: serve LFM3-1.7B on CUDA: NVFP4 and FP8 weight-only, MLX 4-bit (stacked on #1513, #1492 and #1503) #1508 does for the CUDA side. The 18 rows with disjoint top sets need tracing first.
• For VL: the projected rows of real images against FP64, then teacher-forced text after them.

Performance, on the same Mac and checkpoint against mlx-lm (mlx-vlm for VL), at each width:

• one-stream decode and the 4-stream aggregate, with outputs of equal length;
• cold prefill at 2k, 8k, 32k and 64k. Decode should be at least the standard server's. Cold prefill should take the prompt tiles, as below; a remaining prefill gap is fine to track as its own issue, as we do for the 27B.

Before this leaves draft:

1. Prompt rows go through the row kernels and the decode attention (zig/src/families/lfm3/forward.zig:39-49, :196-216). That is why cold prefill is 7 to 18 times slower than mlx-lm's. LFM2's Metal family takes tile_qmm's prompt tiles with a decode twin of the same bits (zig/src/families/lfm2_moe/forward.zig:30), and the tensor-unit prompt attention on an M5 (:244). Please build on those, and on LFM2: shared-round decode #1377's shared rounds for the 4-stream aggregate.
2. Please split the PR: the admission and the text decoder first, with VL stacked on them. lfm3: serve LFM3-1.7B on CUDA: NVFP4 and FP8 weight-only, MLX 4-bit (stacked on #1513, #1492 and #1503) #1508 already builds on the admission.
3. Please state the weights' licence, and the Macs it was checked on, in the recipe.

Once the gates hold, we'll run the full Metal contract and the served checks on our M5 Ultra.