Repository navigation
fix(pid): hand PiD the normalized latent it was trained on - #9888
Open
Pfannkuchensack wants to merge 4 commits into
Open
Pfannkuchensack wants to merge 4 commits into
Pfannkuchensack wants to merge 4 commits into
Conversation
PiD decode nodes no longer denormalize the latent; upstream trains PiD on the VAE-normalized latent. FLUX.2 PiD decode patchifies and BN-normalizes the stored latent with the VAE's statistics, so its VAE input is now required. Extract get_flux2_bn_stats for FLUX.2 denoise and PiD; webv2 wires the VAE only into the FLUX.2 decode.
…latent # Conflicts: # invokeai/frontend/webv2/src/features/generation/core/pidGraph.ts
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every PiD decode node denormalized the latent before handing it to PiD (
z / scale + shift,z * std + mean). PiD is trained on the normalized latent: each upstream tokenizer in nv-tlabs/PiD normalizes at encode (tokenizers/flux_vae.py,sdxl_vae.py,qwenimage_vae.py,flux2_vae.py), and the vendoredpipeline_registry.pysays the latent goes to PiD "directly — no extra denormalization". Fed the raw latent, PiD returned oversaturated, posterized images with visible banding.flux2_denoisestores the BN-denormalized, unpacked(B, 32, H/8, W/8)latent. The node now patchifies it with the(c ph pw)order and BN-normalizes it with the VAE's running statistics, which is exactly what upstream's FLUX.2 tokenizer encodes to (pack_flux2_latent_for_pid). The node'svaeinput is therefore required.Flux2DenoiseInvocation._get_bn_statsintoget_flux2_bn_stats(backend/flux2/sampling_utils.py), shared by denoise and PiD.vaeinputs on the Qwen-Image, SDXL and Z-Image decode nodes are no longer read. They stay as optional fields: existing workflows that connect a VAE still validate, and Generate/Canvas keep wiring the decode VAE there because Canvas img2img (findVaeSource, feat(canvas): restore PiD decoding and the Anima inpaint adapter on Canvas #9805) uses that edge to encode with the same VAE, including an SDXL VAE override or seamless.flux2_pid_decode1.1.0, the other six PiD nodes 1.0.1.Related Issues / Discussions
QA Instructions
A/B, old (
main) vs this branch, RTX 4090. Four test images (512 px) per backbone, each encoded with that backbone's VAE, then decoded with the matching PiD node. PSNR of the PiD output, box-downscaled 4×, against the input image. VAE round trip shown as a ceiling. Same graphs, isolated root,use_cache: false, PiD seed 0.mainAll 28 per-image comparisons improve. Visually,
mainis oversaturated with posterized skies and fur; this PR matches the VAE's colors and adds detail.Browser (webv2 Generate): FLUX.2 Klein 4B + PiD 1.5 FLUX.2 and Z-Image Turbo + PiD 1.5 FLUX, Fit mode at 512 px. Both completed. The FLUX.2 graph carries the
model_loader.vae → flux2_pid_decode.vaeedge, and the Z-Image graph has no VAE edge on the decode. Output is clean at 2048 px.Checks run
uv run --no-sync pytest tests/backend/pid tests/app/invocations tests/backend/flux2 -n 4(after mergingmain): 2921 passed.tests/backend/pid/test_pid_latent_normalization.py:BatchNorm2d);flux2_denoise's output path (unpack_flux2of the BN-denormalized latent);ruff checkandruff format --checkon the changed files.vitest run --config vitest.config.mts src/features/generation/core(1044 passed),tsc --noEmit,lint:oxc,oxfmt --check,architecture:check.frontend/api:pnpm test.Review
No material findings. Two small follow-ups came out of review:
main: Canvas img2img picks its encode VAE from the PiD decode'svaeedge, so dropping that edge for SDXL would have silently ignored a VAE override or seamless. The edge stays on every PiD decode that takes one, and regression tests cover override and seamless.Pre-existing and not changed here:
sd3_i2l/sd3_l2iignore the SD3 VAE'sshift_factor(0.0609). PiD decodes from the SD3 denoiser are unaffected, but a latent fromsd3_i2lcarries a constant offset of about 0.093.Compatibility / Rollout
flux2_pid_decode1.1.0:vaeis now a required connection.RequiredConnectionExceptionrather than producing a wrong image.openapi.jsonandschema.tsare regenerated. No persisted-data changes.Checklist
What's Newcopy (if doing a release after this PR)