feat(flux2): tile the FLUX.2 VAE encode and decode

[Pfannkuchensack]

Summary

The FLUX.2 VAE nodes were the only VAE family not wired to the shared tiling code. force_tiled_decode and auto_tiled_decode skipped them, and an encode or decode that ran out of memory failed instead of retrying tiled. An untiled FLUX.2 decode costs about 4.3 GB at 1024² and grows with the pixel area, so large images could not be decoded on most cards.

flux2_vae_encode and flux2_vae_decode (1.0.0 → 1.1.0) now behave like the FLUX.1 nodes:

• New fields: tiled and tile_size, with the same defaults and the same 0 = default-tile sentinel.
• force_tiled_decode: tiles both nodes.
• auto_tiled_decode: the decode tiles up front when the untiled decode would take most of the GPU.
• Out of memory: an untiled encode or decode retries once with the default tile.
• Tiling state: set per call through scoped_vae_tiling, so the cached VAE is left as it was.

estimate_vae_working_memory_flux2 now handles tiles the way the FLUX.1 estimate does:

• It resolves the 0 sentinel itself.
• A tile that covers the whole image is priced as the single pass the VAE runs.
• A tile larger than the image on one side is cut to the image on that side.
• Before, a 2048 tile on a 1024² decode reserved 21.5 GiB instead of 4.3 GiB. The new fields would have made that reachable, and the cache would have offloaded the transformer for it.

Next step: an Upscale-widget entry for FLUX.2 that runs the pass at 2048²–4096².

Related Issues / Discussions

Prerequisite for FLUX.2 in the Upscale widget (follow-up to #9825 and #9889).

QA Instructions

Automated

• pytest on test_flux2_vae_tiling.py (new), test_flux2_working_memory.py, test_flux_vae_encode_tiling.py, test_flux_vae_decode_tiling.py, test_ideogram4_vae_decode.py, test_frontend_graph_node_types.py and tests/app/services/config: 956 passed, 3 skipped.
• The node tests run a small real AutoencoderKLFlux2 on CPU. The image is non-square and no side is a multiple of the tile. They check:
  • output size, tiled and untiled;
  • that the shared VAE's tiling state is restored after success and after an exception;
  • force_tiled_decode, auto pre-tiling on and off, and the OOM retry with the default tile;
  • the reservation handed to the model cache.
• The estimator cases use hand-computed values: a tile that covers the image, an elongated image, the 0 sentinel, a tile below the minimum, and the score-matrix path.
• Each of these, when removed, fails at least one test: the clamp, the covering-tile check, force_tiled_decode, the state restore, the retry tile.
• ruff@0.11.2 check / format --check: pass. openapi.json, schema.ts and docs/src/generated/settings.json are regenerated.

GPU (RTX 4090, FLUX.2 Klein 4B fp8, 4 steps, seed 1234, model cache emptied before each graph). Baseline is main's FLUX.2 VAE code.

Graph	main	This PR
txt2img 1024², fields unset	11.2 s	10.6 s, bit-identical
img2img 2048², fields unset	18.6 s	18.2 s, bit-identical
img2img 2048², tiled encode and decode	—	19.6 s
img2img 4096², tiled encode and decode	—	93.5 s

• At 2048²: the tiled and untiled outputs look the same, with no visible seams.
• At 4096²: composition and lettering survive, and a 100% crop is clean.
• At 1024²: a tiled decode differs from the untiled one by texture-level noise spread over the frame (mean 5.6/255), with no seam lines. The config description already says a tiled decode is not pixel-identical.

Review

Resolved: the estimator over-reserved for tiles as large as the image or larger (see Summary).

Remaining, shared with the FLUX.1 and Z-Image nodes, and left out of this PR:

• The encode never tiles up front. On Windows, an untiled FLUX.2 encode at 4096² pages instead of running out of memory, so the retry never fires: it was still running after 10 minutes. Callers at that size, such as the Upscale widget, set tiled on the encode.
• Pre-tiling uses the node's tile_size. The OOM retry uses the default tile instead. A tile size left over from an earlier run can therefore make the pre-tile a single pass.

Compatibility / Rollout

• Saved workflows: the nodes go 1.0.0 → 1.1.0 and the two new fields have defaults. Older workflows load with tiling off, as before.
• Generate and Canvas: they don't set the new fields. The untiled path is unchanged (bit-identical above). A decode now tiles only where its untiled working memory would take most of the GPU. With bf16 on CUDA, that is roughly above 2290² on 24 GB, 1870² on 16 GB and 1330² on 8 GB. Before, such a decode paged or ran out of memory.
• force_tiled_decode users: FLUX.2 encode and decode now tile too, as the setting describes.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[JPPhoto]

The documentation should probably mention the different result from a tiled decode. Otherwise, this is good.