Repository navigation
feat(sdxl): run SDXL attention on SageAttention when enabled - #9663
Open
Pfannkuchensack wants to merge 44 commits into
Open
Pfannkuchensack wants to merge 44 commits into
Pfannkuchensack wants to merge 44 commits into
Conversation
New `attention_backend: auto | sage` setting; with `sage`, eligible PyTorch SDPA calls inside a denoising scope run on SageAttention 2.2 and every other call keeps SDPA unchanged. Each GPU's first SageAttention call is checked against SDPA and a failing GPU falls back for the session; no model family enters the scope yet. Adds the SageAttention docs page and the regenerated config schema.
With `attention_backend: sage`, FLUX.1 denoising, including ControlNets inside the loop, enters the SageAttention scope. RTX 4090, FLUX.1 dev FP8: 16.1 s → 14.0 s at 1024², 84.2 s → 54.8 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
With `attention_backend: sage`, FLUX.2 denoising enters the SageAttention scope. RTX 4090, FLUX.2 Klein 9B FP8: 15.6 s → 15.1 s at 1024², 27.1 s → 23.1 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
With `attention_backend: sage`, Z-Image denoising enters the SageAttention scope; Z-Image Control masks its attention and keeps SDPA. RTX 4090, Z-Image FP8: 10.4 s → 10.0 s at 1024², 33.7 s → 25.9 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
With `attention_backend: sage`, Qwen-Image denoising enters the SageAttention scope. RTX 4090, Qwen-Image 2512 GGUF Q4_K_M: 78.8 s → 73.5 s at 1024², 208.9 s → 150.3 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
With `attention_backend: sage`, Krea-2 denoising enters the SageAttention scope unless INVOKE_KREA2_SDPA_BACKEND pins a PyTorch kernel. RTX 4090, Krea-2 Turbo GGUF Q4_K_M: 11.5 s → 11.0 s at 1024², 42.7 s → 36.2 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
With `attention_backend: sage`, SDXL denoising and tiled multi-diffusion enter the SageAttention scope; SD1.5 and SD2 do not. RTX 4090: 4.4 s → 4.2 s at 1024², 19.6 s → 16.1 s at 2048², peak VRAM unchanged; a blind review judged no image worse.
This was referenced Oct 3, 2026
github-actions
Bot
added
python
…ent SageAttention failures sm86 now runs the Triton kernel upstream's sageattn picks there, and sm87 gets none; the Windows build's CUDA routing for both is unexplained and unmeasured here. Out-of-memory and Triton cache errors fall back to SDPA for that call (three retire the device), and the first-call check runs per precision and head size. The kernel runs with the tensors' device current, so a single-device install on cuda:1 uses SageAttention too.
An out-of-memory error now moves only the rest of that generation to SDPA, logged once, instead of counting toward retiring the device; out-of-memory is recognised by the shared is_oom_error. Only Triton cache failures in a row retire a device, and a success resets the count.
# Conflicts: # invokeai/frontend/api/openapi.json
# Conflicts: # invokeai/frontend/api/openapi.json # tests/test_config.py
Pfannkuchensack
marked this pull request as ready for review
October 12, 2026 20:44
…lower with SageAttention On Windows the driver moves memory to shared system memory instead of raising out-of-memory, so the fallback never triggers and the extra memory per call slows every step; issue invoke-ai#9831 tracks a fix.
# Conflicts: # invokeai/app/services/config/config_default.py # invokeai/frontend/api/openapi.json # invokeai/frontend/api/schema.ts
lstein
added a commit
that referenced
this pull request
Oct 16, 2026
…led (#9664) > [!NOTE] > Stacked PR 7 of 8: merge after #9663. It targets `main`, so until the PRs before it merge, its diff also shows their commits and the merges that carry them forward. This PR's own changes are two commits, [`9507eccfb6`](Pfannkuchensack@9507ecc) (the scope) and [`b3c3e4300a`](Pfannkuchensack@b3c3e4300a) (the audio result in the docs). ## Summary With `attention_backend: sage`, LTX-2 denoising enters the SageAttention scope from #9657. Its video self-attention runs on SageAttention; the audio attention is below the size threshold and the text cross-attention passes a mask, so both stay on PyTorch SDPA. The default `auto` is unchanged. The docs page lists LTX-2 under Supported models. ## Related Issues / Discussions Stacked on #9657 (SageAttention backend) and #9663, the PR before it. ## QA Instructions LTX-2.5 distilled int8, model loaded, three arms per seed as described in #9657: | | 1248×704, 121 frames | | --- | --- | | Step, default SDPA | 5263 ms | | Step, cuDNN-first SDPA | 4726 ms | | Step, SageAttention | 4178 ms (−21 %) | | Whole generation | 72.0 → 63.4 s (−12 %) | | Peak VRAM in the denoise scope | unchanged | | Per-frame LPIPS against default SDPA, median (cuDNN-first) | 0.142 (0.090) | | Flicker ratio against default SDPA | 1.001 | - Per step, the 48 video self-attention calls run on SageAttention. Audio self-attention and the two audio–video cross-attentions (3 × 48) are below the size threshold, and the text cross-attention (2 × 48) passes a mask; all of these stay on SDPA. Dropping the text mask where it masks nothing is a possible follow-up. - Accuracy on real activations, every call of step 0 (130 calls): worst cosine 0.9993, relative L2 0.027. - Frames were compared per frame by LPIPS and by frame-to-frame flicker. - **Audio.** It reads the video through cross-attention, so it changes even though its own attention stays on SDPA. Measured with the code of this PR on 8 clips (4 prompts × 2 seeds, 1248×704, 121 frames; two prompts with clear sync events: four hand claps, a spoken sentence), against default SDPA, with cuDNN-first as the exact-kernel noise floor: | | SageAttention | cuDNN-first (floor) | | --- | --- | --- | | Log-mel distance, median (max) | 0.92 dB (1.58) | 0.74 dB (1.36) | | Onset-envelope cosine, median | 0.980 | 0.986 | | Shift against the SDPA audio | 0 ms in all 8 | 0 ms | | Loudness | within ±0.2 dB | within ±0.26 dB | | Clipping / silence | none | none | Sync to the picture: on the clap and speech clips, motion and sound onsets correlate clearly (0.4–0.65), and the lag between them is identical in all three arms on every clip, so SageAttention does not move the sound against the picture. On the other two prompts there are no sync events to measure. Nobody listened to the clips; the numbers are the evidence. - At 1792×1024 all three arms ran out of memory on 24 GB without partial loading, so that size was not measured. - The timings and LPIPS come from the measurement prototype, which still used `sageattn()`'s default K smoothing; the selection #9657 ships is faster per call. To try it: install SageAttention as the docs page from #9657 describes, set `attention_backend: sage`, restart and generate with LTX-2. The first generation logs `SageAttention ... serves diffusion-model attention on cuda:0 (...)` with the GPU and kernel name, and with `log_level: debug` every generation logs how many calls ran on SageAttention and why the others did not. With `attention_backend: auto` the output is bit-identical to `main`. Checks: ruff clean; `pytest tests/app/invocations` passes with the whole stack applied (2090 passed). No call-site test was added: this change only enters the scope, whose behavior #9657 tests. ## Review Resolved: the audio was not compared at first. It now is (see QA), and the docs state the result. ## Compatibility / Rollout Merge after #9663. No API or persisted-state change; only `attention_backend: sage` behaves differently. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Meaningful regression coverage added / updated where needed; obsolete tests/code removed_ - [ ] _Persisted-state and API changes include required migrations / compatibility validation_ - [x] _Relevant performance/efficiency opportunities considered; material claims have evidence_ - [x] _Material review findings resolved and relevant checks rerun_ - [x] _Documentation added / updated (if applicable)_ - [ ] _Updated `What's New` copy (if doing a release after this PR)_
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked PR 6 of 8: merge after #9662. It targets
main, so until the PRs before it merge, its diff also shows their commits and the merges that carry them forward. This PR's own change is one commit,1968eb9c58.Summary
With
attention_backend: sage, SDXL denoising and SDXL tiled multi-diffusion enter the SageAttention scope from #9657, so eligible attention calls run on SageAttention, including ControlNets that run inside the loop. SD1.5 and SD2 do not enter it: SD1.5's head sizes (40, 80, 160) never qualify, and SD2 is unmeasured. Withattention_type: sliced, SDXL's attention does not go through SDPA, so nothing changes there. The defaultautois unchanged. The docs page lists SDXL under Supported models and notes the sliced case.Related Issues / Discussions
Stacked on #9657 (SageAttention backend) and #9662, the PR before it.
QA Instructions
SDXL, model loaded, three arms per seed as described in #9657:
sageattn()'s default K smoothing; the selection feat(attention): add opt-in SageAttention backend for diffusion models #9657 ships is faster per call.To try it: install SageAttention as the docs page from #9657 describes, set
attention_backend: sage, restart and generate with SDXL. The first generation logsSageAttention ... serves diffusion-model attention on cuda:0 (...)with the GPU and kernel name, and withlog_level: debugevery generation logs how many calls ran on SageAttention and why the others did not. Withattention_backend: autothe output is bit-identical tomain.Checks: ruff clean;
pytest tests/app/invocationspasses with the whole stack applied (2090 passed). No call-site test was added: this change only enters the scope, whose behavior #9657 tests.Review
Resolved: the scope first opened for every UNet base, including the unmeasured SD2; it now opens for SDXL only.
Compatibility / Rollout
Merge after #9662. No API or persisted-state change; only
attention_backend: sagebehaves differently.Checklist
What's Newcopy (if doing a release after this PR)