Add hy_v3 (Tencent Hy3) native MTP backend

[PhilipJohnBasile]

What

Adds the hy_v3 (Tencent Hy3) native MTP backend — closes the recognized-backend-pending gap for hy-v3-mtp (#141).

• mtplx/backends/hy_v3_mtp.py — facade on the glm_mtp pattern (contract-gated load, health metadata, drafting via the shared sampler in generation.py)
• mtplx/hy_v3_mtp_patch.py — is_hy_v3_mtp_config + inject_hy_v3_mtp_support. hy_v3's mlx-lm MTP revision loads the head natively, so injection installs the runtime surface (mtp_forward / mtp_update_cache / make_mtp_cache, return_hidden on the trunk forward) as a subclass wrapper — no weight rewriting. Guards: raises clearly on AR-only exports; rejects non-embedding_hidden concat order and non-pre_norm hidden variants (Hy3's fixed semantics); depth 1 only.
• mtplx/runtime.py dispatch, import-smoke registration, registry upgrade to experimental-native-contract-gated, pending-markers test updated.

Verification

• Applies on the v2.0.1 tag; tests/test_artifacts.py 76/76 pass with the change.
• The full runtime surface (forward+hidden, draft with/without hidden, cache update, make_mtp_cache, guard rejections) exercised against a tiny randomly-initialized hy_v3 model — shapes and guards verified.

What this deliberately does NOT claim

• Not live-tested against the 295B checkpoint — blocked on hy_v3 reaching the pinned mlx-lm (Add Hy3 ml-explore/mlx-lm#1211 + the MTP-surface follow-up). Draft PR until then.
• No runtime contract fixtures yet — I'll run mtplx bench/exactness on an M5 Max 128 GB against the real checkpoint as soon as the model loads, and post receipts here.

Architecture facts encoded

One appended NextN layer (depth 1) with its own 192-expert MoE (sigmoid top-8 + expert bias); draft input eh_proj(concat[enorm(embedding), hnorm(pre-final-norm hidden)]); shared embeddings/head. Loader note for whenever the model class lands: MLX-converted checkpoints carry MTP weights in a model-mtp.safetensors sidecar named mtp.* (not model.layers.80.*).

Motivation measurement (why MTPLX's batched verify matters for this model): mlx-lm's per-token draft→verify loop runs Hy3 MTP at exact output parity but ~13.7× slower than AR (0.54 vs 7.4 tok/s warm, M5 Max). Receipts: https://github.com/PhilipJohnBasile/hy3-demolition-mlx

[PhilipJohnBasile]

Pushed a fix commit after an adversarial self-audit of the initial version found three blocking bugs — worth flagging since they'd have crashed on your first load attempt:

1. Contract-default guard mismatch (the critical one): my injector guards rejected hidden_variant="post_norm", but runtime.draft_mtp resolves None/auto/contract to contract.hidden_variant, which is "post_norm" for a bare MTPContract() — so every draft call raised ValueError. Now the backend accept-and-ignores the variant/concat strings like glm_mtp_patch and step3p5_mtp_patch do (Hy3 has exactly one native draft wiring, so the string is informational).
2. validate_mtp_support failure: it requires model.mtp.layers to be truthy; the native mlx-lm MTPBlock exposes its decoder as .layer. Added a .layers alias (list wrapping the single layer — a child-module container, not a weight copy).
3. Registry gaps: hy-v3-mtp was in the catalog but absent from SUPPORTED_ARCH_IDS and the appended-layer branch of _passes_family_runtime_gate — added to both.

Also: depth≥2 now reuses the single NextN layer instead of raising, and dropped a dead contract.note() call.

Added tests/test_hy_v3_mtp_backend.py (4 tests, incl. one that reproduces the exact post_norm crash path through the real contract default). Artifact suite still 76/76 on the v2.0.1 tag.

Caveats unchanged: still not live-tested against the 295B (blocked on hy_v3 reaching your mlx-lm pin), still draft. But the load path is now correct against your runtime's actual call sites, not just my reading of them.

[davidtai]

Looking forward to this.

[PhilipJohnBasile]

@davidtai thanks — good timing, there's real progress to show, and it sharpens why this backend matters.

Two verified MLX Hy3 artifacts are now public — the actual targets this backend would serve, both running on stock mlx_lm today (AR-only view, num_nextn_predict_layers=0), receipts for every claim in the source repo:

• lite-v1 (unpruned reference, 104 GB): https://huggingface.co/philipjohnbasile/hy3-demolition-mlx-lite-v1
• reap25 (soul-preserving 25% expert prune + heal, 80 GB / ~87 GB peak): https://huggingface.co/philipjohnbasile/hy3-demolition-mlx-reap25-v1

Source + build scripts + receipts: https://github.com/PhilipJohnBasile/hy3-demolition-mlx

Why the MTPLX backend matters — measured, not hand-waved. On an M5 Max 128 GB, mlx-lm's per-token self-speculative MTP loop runs Hy3 at 1.59 tok/s vs 7.43 tok/s AR — 4.7× slower, with exact output parity (receipt: hy3_path_benchmark.json). The NextN heads are correct; the per-token draft→verify loop with cache trims is the bottleneck. That is exactly what MTPLX's batched verify is engineered to fix.

Independent cross-validation that the approach works on this hardware. A sibling project forged its own MTP artifact from Qwen3.6-35B-A3B and served it on MTPLX at 24.7 concurrent turns/min — a tie with MTPLX's own optimized build, same M-series class machine, with better quality. Same runtime family; batched verify rescues MTP. So this isn't speculative — the backend should do for hy_v3 what it already does for Qwen.

PR status (unchanged and ready): the backend is staged and audit-hardened — 76/76 artifact tests on the v2.0.1 tag, the three blocking bugs from my self-audit fixed with regression coverage (tests/test_hy_v3_mtp_backend.py, incl. one reproducing the exact post_norm contract-default crash path). Still honestly draft: not live-tested against the 295B until hy_v3 reaches your mlx-lm pin (ml-explore/mlx-lm#1211 + the MTP surface). The moment it loads I'll run bench + exactness fixtures on the M5 Max and post receipts here.

Happy to walk through the design doc or pair on the integration whenever it's useful. Everything's Apache-2.0.

[PhilipJohnBasile]

Concrete data point from your own tooling, @davidtai / @youssofal: I ran mtplx inspect (release v2.0.1) against the MTP-equipped Hy3 base checkpoint, and the release already recognizes the family — it's only the runtime backend that's missing:

compatibility.recognized:            true
compatibility.mtp_supported:         "recognized"
compatibility.message:  "HY V3 MTP markers recognized, but MTPLX does not
                          yet have a native MLX runtime backend for this family."

So the arch-detection half is done on your side; PR #142 supplies exactly the missing piece — the hy_v3 MLX runtime backend the message points at. (For completeness: my two public fused artifacts, lite-v1 and reap25, inspect as tier: no-MTP — they're AR-only daily-drivers by design; the MTP path is the base checkpoint's NextN sidecar, which is what #142 targets.)

Full mtplx inspect --json receipts are committed in the source repo under eval/receipts/mtplx_inspect_*.json. Happy to forge an MTP-equipped fused artifact for you to test against the moment the backend + the mlx-lm pin line up.

[PhilipJohnBasile]

Pushed an extension: qwen3_5_mtp (Qwen3.5/3.6-MoE) native MTP backend — this also resolves #147 (forge probe recognized qwen3_5_mtp, but the load path had no handler).

Two-part fix, mirroring the step3p5/hy_v3 backends:

• Trunk: the MTP export differs from the AR export only in the top-level model_type string (qwen3_5_mtp vs qwen3_5_moe) — text_config is identical. A sys.modules shim loads the trunk as qwen3_5_moe. (The shim also strips mtp.* before sanitize — mlx-lm's qwen3_5 sanitize shifts trunk norms +1.0 when mtp weights are present, which double-shifts an already-final-convention export into gibberish. GPU bring-up caught this.)
• Head: one appended NextN predictor (pre_fc_norm_embedding/hidden → fc → one full-attention qwen3_5.DecoderLayer → norm → shared lm_head), grafted with the standard mtp_forward/mtp_update_cache/make_mtp_cache surface.

Hardware-verified on Qwen3.6-35B-A3B (M5 Max 128 GB):

• 46 mtp.* tensors load with exact coverage (strict=True)
• trunk generates coherently (Paris/Tokyo); return_hidden logits bit-identical to the plain forward (max|Δ| = 0.0)
• ~90% 1-step greedy draft acceptance on a structured smoke — well clear of the ~3% "donkey band", confirming the pre_norm + [embedding, hidden] wiring

Hermetic unit tests added (detection/shim/remap). A full acceptance sweep across diverse text is the remaining production-tuning step. Happy to iterate on review.

[PhilipJohnBasile]

@youssofal please push :)

[PhilipJohnBasile]

Ran this branch's hy_v3_mtp_patch.py / backends/hy_v3_mtp.py end-to-end against a real, shipped hy_v3 checkpoint on hardware (hy3-demolition-mlx-reap25-v1-mtp, 87GB, M5 Max 128GB — receipts: https://github.com/PhilipJohnBasile/hy3-demolition-mlx). Found the backend logic itself is sound, but there's a real bug plus several pieces of plumbing missing before mtplx forge build / mtplx tune fully support it. Details in case useful before merge:

1. _MTPLXHyV3Model has no make_cache(). hy_v3 doesn't define one natively (unlike the hybrid-attention Qwen backends), and MTPLXRuntime.make_cache() calls model.make_cache() directly rather than going through mlx_lm.models.cache.make_prompt_cache's fallback. Crashes with AttributeError: '_MTPLXHyV3Model' object has no attribute 'make_cache'. Fix — build the default per-layer KVCache list directly (calling make_prompt_cache(self) recurses, since it dispatches back to this same method):

def make_cache(self):
    from mlx_lm.models.cache import KVCache
    return [KVCache() for _ in self.model.layers]

2. Registry never leaves recognized-backend-pending once a real backend exists. backends/registry.py's "hy-v3-mtp" entry has support_level/runtime_compatibility hardcoded to "recognized-backend-pending" — matches the mtplx inspect output referenced on ml-explore/mlx-lm#1211 ("recognized-backend-pending... the backend PR is staged"). With this branch's backend actually present, flipping to the same shape as the other native-contract-gated entries unblocks it:

support_level="experimental-native-contract-gated",
runtime_compatibility="native-contract-gated",
can_run_verified=True,

runtime.py's load() also needs the dispatch branch added (it currently has no is_hy_v3_mtp_config case, silently falling to the generic inject_mtp_support):

elif is_hy_v3_mtp_config(config):
    mtp_enabled = inject_hy_v3_mtp_support(model, path, config, contract)

3. mtplx tune doesn't know hy_v3 exists as a family. tune_policy_for_model / model_family_from_inspection in backends/descriptors.py only recognize qwen3_5/qwen3_6/gemma4/step/deepseek/glm — anything else reports model_family: "unknown" and the hardcoded "Tune is supported for Qwen 3.5, Qwen 3.6, and Gemma 4 MTPLX models only." message. Needs an hy_v3 branch in both.

4. The family marker gate reuses DeepSeek's naming convention, which doesn't match hy_v3's real layout. _passes_appended_layer_gate looks for markers under mtp.layers.{idx}.*. hy_v3's actual weight keys (checked directly against the shipped checkpoint's model.safetensors.index.json) are flat under mtp.: mtp.enorm.weight, mtp.hnorm.weight, mtp.eh_proj.weight, mtp.final_layernorm.weight, mtp.layer.* — no per-index nesting, no shared_head.*. Needs its own gate function rather than reuse.

5. Real gap, not a quick fix: mtplx tune's depth-sweep path assumes hybrid attention. Once 1–4 are addressed, AR baseline runs clean through mtplx tune, but D1/D2/D3 all fail identically:

AttributeError: 'HYV3Model' object has no attribute 'fa_idx'

generate_mtpk → rt.forward_ar_capture → forward_with_gdn_capture (gdn_capture.py) indexes the cache via inner.fa_idx — a "full-attention-layer-index" concept specific to Qwen 3.5/3.6's hybrid linear/full-attention layout. hy_v3 is uniform full-attention MoE with no such concept. This capture path needs a non-hybrid branch (or a dedicated hy_v3 forward-capture) rather than a workaround — didn't attempt to fake fa_idx, since a wrong cache-slot index there could silently produce incorrect depth-sweep numbers rather than an honest failure.

Happy to share the exact patched files or re-run against #1 in this list's diff if useful — everything above was verified against the real checkpoint, not synthetic.

[PhilipJohnBasile]

Pushed a fix (cda4605) for the fa_idx crash mentioned in my earlier comment on this PR. Full picture, including two separate issues it surfaced but does not fix:

Fixed: forward_ar_capture crashed on any non-hybrid-attention model. It unconditionally delegated to forward_with_gdn_capture (gdn_capture.py), which indexes cache[inner.fa_idx] / cache[inner.ssm_idx] and branches per-layer on layer.is_linear — built for Qwen 3.5/3.6's hybrid linear/full-attention layout. hy_v3 is plain uniform causal attention with no hybrid layers, so every mtplx tune MTP-depth candidate (D1/D2/D3) crashed with AttributeError: 'HYV3Model' object has no attribute 'fa_idx'.

Fix branches on whether the trunk actually defines fa_idx/ssm_idx; if not, it uses the plain forward_ar path and returns an empty captures dict (which commit_captured_prefix already handles correctly — falls through to trimming the standard KV cache directly, the right behavior for pure-attention layers). No fa_idx value guessed or hardcoded anywhere — didn't want to risk silently-wrong depth-sweep numbers. The hybrid path is completely untouched (verified with a synthetic fake-hybrid-model unit test that it still routes into the original gdn_capture code unchanged).

Verified on real hardware: ran the full mtplx forge build reproduce against the actual 87GB hy3-demolition-mlx-reap25-v1-mtp checkpoint. D1/D2/D3 all completed with no crash — previously 100% reproducible failure.

But two separate, pre-existing issues surfaced once the crash was out of the way — flagging clearly rather than glossing over them:

1. Real quality problem on D1/D2. Generated text is on-topic (legitimate Python code continuation, not garbage), but hit a genuine repetition-loop failure — the model got stuck restating the same function signature/docstring 4–5 times verbatim. D3 alone passed the quality check. Acceptance rates were also low across all depths (~11% at position 0, dropping to ~1–2% deeper) — well under what we measured on a different (Qwen) MTP backend earlier this week. Possibly a hidden_variant/concat_order contract mismatch specific to hy_v3's MTP head, or a deeper drafter-quality issue — genuinely unconfirmed, needs separate investigation. Not something this fix touches.

2. The AR-baseline candidate in the depth-sweep silently produces no output — no crash, no traceback, no OOM signal in the system log, just an empty result file. That candidate never calls forward_ar_capture at all, so it's unrelated to this fix, but it means multiplier_vs_ar: 0.0 and the resulting no_mtp_depth_beat_ar verdict are artifacts of a missing baseline comparison, not real evidence MTP underperforms AR on this model. Worth knowing before drawing conclusions from that verdict field.

Happy to dig into either of those as follow-ups if useful — wanted to get the crash fix in first since it's clean, isolated, and blocking any real signal at all.

[PhilipJohnBasile]

Follow-up with the confirmed numbers on the two issues flagged in my last comment — ran a clean re-test (fresh calibration + full AR/D1/D2/D3 sweep, valid AR baseline this time) to get a real comparison rather than leaving it as "unconfirmed."

AR-baseline candidate: the earlier silent failure (returncode: -9, no output) didn't reproduce on a clean re-run — looks like it was a transient SIGKILL from memory pressure during the calibration→tune handoff (other processes were competing for memory on the test machine at that moment), not a code bug. Re-running with headroom gave a clean AR result.

Full depth-sweep, with a valid AR baseline this time:

depth	tok/s	vs AR	quality_passed	acceptance
AR (0)	8.31	1.0×	✅	—
D1	8.32	1.001× (break-even)	❌ (repetition loop)	11.2%
D2	5.12	0.616× (38% slower)	❌ (repetition loop)	11.0% / 1.8%
D3	6.02	0.725× (27% slower)	✅	10.1% / 1.7% / 0.2%

MTPLX's own verdict message: "MTP measured faster, but the quality gate rejected the fast depth. AR 8.31 tok/s; best MTP D1 8.32 tok/s."

This confirms the repetition-loop is a real MTP-specific defect, not a general checkpoint/prompt quirk — same checkpoint, same prompt, same settings; AR alone passes quality cleanly, only the MTP-drafted depths loop. Given calibrate.json already reported calibration_status: "no_agreement_signal" / best_agreement: 0.0 at every probed depth, that's the most likely root cause — the contract calibration isn't converging on a working drafter configuration for hy_v3, not anything touched by the fa_idx/make_cache fixes.

Net result across two consistent runs: there's no depth where hy_v3 MTP is both fast and quality-clean — the depth that's competitive on speed (D1) fails quality, the depth that passes quality (D3) is 27% slower than AR. Given this matches an independent finding on Hy3's native MTP path too (measured 4.7-13.6x slower in our own testing, unrelated codebase), I'm not planning to chase the calibration-convergence issue further myself — flagging it clearly here in case it's useful signal for anyone who does, since the fa_idx fix at least makes it possible to measure this now, which it wasn't before.

[PhilipJohnBasile]

Cross-link in case useful: there's a parallel investigation of Hy3 MTP on the mainline-track native self-speculative path (not this backend's contract/drafter approach) — ml-explore/mlx-lm#1485 (comment). Their finding: the checkpoint's aggressive quantization (2-3 bit experts in both target and drafter) is the likely driver of low acceptance, not the algorithm — plus a real bug where the old implementation silently forced greedy decoding regardless of requested temperature, understating acceptance for any real (temp>0) usage.

That greedy-forcing bug doesn't apply here (this backend's failure mode is calibration_status: no_agreement_signal — the contract never converges, a different mechanism than a sampling bug), but the checkpoint-quantization angle may be worth keeping in mind: this MTPLX run and their PR both used the same w2q3exp (2-3 bit expert) release.

[PhilipJohnBasile]

Aside from the fa_idx work above — a user of this backend (via the mtplx CLI/app) reported the CLI feeling ~10x faster than the GUI, so I ran a controlled comparison in case it's useful signal for anyone chasing serving-path overhead.

Setup: Qwen3.6-35B-A3B-Mimosa-MTP-4bit (a small, fully mtplx-native model — verified-native per mtplx inspect), single-shot 64-token completions, same prompt, cold-ish server start each time (M5 Max 128GB).

path	config	tok/s	vs CLI
mtplx ask (CLI, in-process)	sustained, MTP on	82.7	1.0x
HTTP /v1/chat/completions (what the GUI's MTPLXChatClient hits)	sustained, MTP on	76.8	0.93x
HTTP	sustained, MTP off (AR)	56.9	0.69x
HTTP	stable profile, MTP on	43.4	0.52x
HTTP	turbo profile, MTP on	36.6	0.44x

Couldn't reproduce a 10x gap. The CLI-vs-server protocol overhead is ~7% — expected, since ChatClient.swift posts to the same local daemon ask calls internally (confirmed by reading MTPLXChatClient.swift, not just inferring it). MTP-on-vs-off is ~1.35x here; profile choice (stable/sustained/turbo) spans up to ~2.1x on a single request, though that's likely inflated for turbo specifically since it's documented as the verify-kernel-compiled fast path and a one-shot request doesn't amortize the compile cost — worth re-checking turbo over a longer/warm run before trusting that number.

Given mtplx start's onboarding asks "mode" (Sustained/Turbo/Sustained Max/Burst) and "where" (Web/CLI/Pi/...) as two independent questions, and defaults fan_mode to default (unboosted) unless a max-fan mode is explicitly chosen — my best guess is the reported 10x came from two separate onboarding sessions landing on different profile/fan choices, not the GUI vs CLI surface itself. But I couldn't fully confirm that without the original reporter's exact config, and I didn't test: the native SwiftUI app's actual request pipeline (only raw HTTP, which the app itself also uses per its own source), or thermal throttling over a genuinely long/sustained session (my tests were all short one-shot bursts). Flagging in case someone with more surface area on the scheduler/thermal code wants to chase it further.