Mac (MPS) support: CUDA templates run unchanged on Apple Silicon

[dkackman]

Makes the catalog's CUDA-authored workflows run on Apple Silicon without editing them, and fixes the defects the Mac review turned up. The engine now adapts each CUDA-only setting to the serving machine and logs a warning each time, so templates stay backend-agnostic and an MCP agent doesn't need to know which backend is serving. CUDA behaviour is unchanged. Every adaptation is keyed on the serving device, and each has a test pinning the CUDA path.

Verified end to end on an M5 Pro with 64 GB:

• ltx2/text-to-video (LTX-2.5, SDNQ uint4 + group offload)
• minimax/music (Music 3)
• text-to-image (SD 1.5)
• segment
• audio-trim-fade
• transcribe-audio

Plan: docs/superpowers/plans/2026-09-25-mac-mps-support.md. Its Task 6 was reversed during verification; see Decisions below.

Engine: CUDA settings adapted to the serving device

Setting	What happened on a Mac	Now
SDNQ quantization_device: "cuda" (all 31 ltx2/* + minimax/* files)	AssertionError: Torch not compiled with CUDA enabled during load	resolved through resolve_device() (portable_quantization_arguments, config_objects.py)
SDNQ use_quantized_matmul(_conv)	falls back to torch._int_mm: 1096 ms against 2.1 ms for a bf16 matmul (~500x)	turned off on MPS only
group_offload.use_stream / record_stream	diffusers refuses streams off CUDA/XPU	both dropped together when the onload device isn't CUDA/XPU
vram_estimate	a 64 GB Mac was refused against the catalog's 24 GB RTX 3090 entry	checks the serving backend's own cost entries, else device_capacity_gb() (Metal's recommended working set), else every entry. So an unreadable capacity never turns the guard off. The #265 refusal had no tests until now
HF_ENABLE_PARALLEL_LOADING	segfault (exit 139): the four loader threads were all inside copy_ onto MPS at once	defaults to off on macOS (_parallel_loading_default). diffusers reads the variable at import, before dw can ask torch for a device, so the choice is by platform. An explicit setting still wins. CUDA boxes keep the parallel load
LTX-2 RoPE double_precision	Cannot convert a MPS Tensor to float64 on the first step	on MPS, any module exposing the flag runs its RoPE in float32 (apply_mps_rope_precision). diffusers already does this for Wan, Lumina2, SkyReels-V2, ChronoEdit and Sana-Video; LTX-2's transformer and connectors never got the check. Upstream gap, not filed

Also:

• device_memory_stats() reports real MPS figures (allocated, driver-reserved, recommended max) and the chip name. They were zeros before, so pipeline_released events, the REPL and the UI showed 0 GB.
• torch's "fall back to run on the CPU" UserWarning now gets through dw's blanket UserWarning filter. A silent CPU fallback now shows in the log.
• The fp16-on-MPS warning is reworded (it ran two sentences together) and now recommends bfloat16.

Fixes that apply on every backend

• segment never worked. Two separate causes:
  • It read the SAM v1 key reshaped_input_sizes, which the SAM2 processor doesn't return, so every run raised KeyError. The mocked tests supplied the key; a new test uses the real Sam2Processor, which builds offline.
  • GroundingDINO only scores a lowercase, period-terminated query: "cat" scored 0.20 against a photo of a cat, under the 0.3 threshold. grounding_query() normalizes the prompt; the template now detects at 0.64.
• transcribe_audio failed on any clip over 30 s. Whisper needs return_timestamps=True for long-form audio. It's passed only past 30 s, since CTC models read the flag differently.
• Template defaults. Eleven fixed, and a new catalog test pins the rule (tests/test_template_defaults.py):
  • example.com placeholders: upscale-diffusion, upscale-spandrel, audio-trim-fade, transcribe-audio.
  • Dead URLs: segment, segment-and-inpaint (HF 404), controlnet-component (host gone), image-variation (deleted tweet).
  • image-to-image pulled its default from a third-party repo named sage-permission-probe.
  • flux-torchao spelled FLUX.1-dev two ways, which means two HF cache entries and ~23 GB downloaded twice.
  • runwayml/stable-diffusion-v1-5 in two templates.
  • dw.test's workflow ran fp16; it now runs bf16.
  • Replacements come from Hugging Face's own docs datasets. Every remaining http default in the catalog returned 200.

Mac results

Template	Wall clock	Peak memory	Notes
ltx2/text-to-video, defaults (960x544, 121 frames)	6.5 min (RTX 3090 cost: 1.8)	23.5 GB	Denoising 8 steps at ~21.5 s each. The video shows the prompt; 48 kHz soundtrack peaks at -9.5 dBFS
minimax/music, defaults	11.7 min	28.6 GB	110 s stereo mp3 peaking at -2.7 dBFS. Its lyrics transcribe back through transcribe-audio on MPS
text-to-image (SD 1.5)	13.5 s		Warm cache
segment			Correct cat mask via SAM2 on MPS

Wall clock is with weights already downloaded.

The ltx-2.5 and minimax-music3 skills told agents to stop on an mps server. They now say those templates run there, slower than the CUDA cost figures, and name what hasn't been run on a Mac.

Decisions a reviewer should weigh

• Attention slicing stays automatic on MPS. The plan made it opt-in after a micro-benchmark showed slicing making SDXL-shaped attention 2.4x slower. End to end, though, SD 1.5 ran 17.3 s without slicing and 14.1 s with it.
  • The cause: MPS SDPA on torch 2.14 is slow at head dims 40/48/160 and fast at 32 and 64-128.
  • So slicing helps SD 1.5 and hurts SDXL. The opt-in change was reverted, and the docs point SDXL-on-Mac users at disable_attention_slicing.
  • A head-dim-aware policy is a follow-up.
• Already-quantized SDNQ checkpoints bypass the matmul switch-off. The catalog's only one, Disty0/Z-Image-Turbo-SDNQ-uint4-svd-r32, ships with it off in both components, and sdnq_optimize was already limited to CUDA/XPU.
• The engine adapts; templates aren't forked or parameterized per backend. A device variable was considered, but it would push the backend choice onto every caller, an MCP agent included.

Deferred (from the final review, all minor)

• Two of the MPS memory calls sit outside the try in device_memory_stats. Every caller catches the error, and torch >= 2.13 has both calls.
• portable_quantization_arguments uses the process device, so a cpu-pinned SDNQ step on a Mac also loses quantized matmul. It runs slower but stays correct.
• device_capacity_gb() is computed on every validation even when a matching cost entry exists.
• Two stale comments: dw/worker.py:708 and dw/server/observed_cost.py:302.
• dw.test in bf16 is slow on pre-Ampere CUDA cards (T4/V100).

Follow-ups

• Run an H3 template on a Mac. The minimax-h3 skill still says CUDA-only.
• Choose slicing per model by attention head dim.
• An upstream diffusers issue for LTX-2's float64 RoPE on MPS.
• GGUF/SDNQ paths for the bitsandbytes and TorchAO-int4 templates (flux2-dev, multi-image-reference, community-pipeline, flux-torchao).
• Whether install.sh should keep auto-loading fp4-fp8-for-torch-mps. It overrides mm/linear/copy_ on MPS and sets PYTORCH_ENABLE_MPS_FALLBACK=1, and nothing in the catalog uses it.
• Quote Mac cost from observed history. Estimates quoted on a Mac are still the CUDA figures (basis: "other_device"), so they're optimistic.

Release-note items

• segment, segment-and-inpaint and transcribe-audio on clips over 30 s now work. None did before.
• On macOS, parallel checkpoint loading is off by default.
• disable_attention_slicing is the switch for SDXL-family models on a Mac.
• dw.test's workflow runs in bf16.

Testing

• Full suite: 6673 passed, 14 skipped, 1 xfailed on the Mac.
• New tests:
  • tests/test_vram_estimate.py
  • tests/test_template_defaults.py
  • tests/test_attention_slicing.py
  • tests/test_mps_fallback_warning.py
  • tests/test_parallel_loading_default.py
  • tests/test_mps_rope_precision.py
• Additions to:
  • test_config_objects.py
  • test_device_helpers.py
  • test_segment.py, including one on the real SAM2 processor
  • test_transcribe_audio.py
• A tiny random-weight transformer was loaded through the engine's builders with the LTX template's exact SDNQ + group-offload settings and ran a finite forward pass on MPS. The same load without the adaptation fails with Torch not compiled with CUDA enabled.
• Not yet run on CUDA hardware (lem was busy). The CUDA paths are covered by the unit tests and a whole-branch review.

🤖 Generated with Claude Code

[dkackman]

Review of the code changes (the plan doc and other docs-only changes skipped).

Fixed in 171fae9: transcribe_audio sent return_timestamps=True for every clip over 30 s, whatever the model. transformers raises for a ctc pipeline unless the value is "char"/"word", and for any non-Whisper seq2seq pipeline. So a wav2vec2-style model_name on a 45 s clip failed where it used to return text. It's now gated on pipe.type == "seq2seq_whisper", with a CTC test. Full suite: 6674 passed, 14 skipped, 1 xfailed.

These parts are correct as written:

• the cost entries filtered by device: every catalog entry carries device, so CUDA checks the same set as before
• resolve_device still returns strings for SDNQ
• the SAM2 post_process_masks call matches its signature
• the sdnq_optimize path already skips non-CUDA devices
• the MPS memory-ceiling message uses no missing fields

Before merging: the branch is meant for macOS, but once it's on develop it deploys to lem, because the loop keeps lem on origin/develop. Several changes also change behavior on CUDA:

• template default input URLs
• segment's GroundingDINO query and SAM2 call
• vram_estimate checks the serving device
• the group-offload streams fix
• transcription
• workflow_schema.json

The loop's tester will pick up the skill edits too.

Follow-ups in the harness: dkackman/harnest#15 (an MCP target other than lem) and dkackman/harnest#16 (an MPS regression level).

🤖 Generated with Claude Code