diff --git a/docs/src/content/docs/Users Guide/04.Image Generation/upscale.mdx b/docs/src/content/docs/Users Guide/04.Image Generation/upscale.mdx index 6a3f574c23b..cf21a90f328 100644 --- a/docs/src/content/docs/Users Guide/04.Image Generation/upscale.mdx +++ b/docs/src/content/docs/Users Guide/04.Image Generation/upscale.mdx @@ -1,6 +1,6 @@ --- title: Upscaling -lastUpdated: 2026-10-12 +lastUpdated: 2026-10-19 sidebar: order: 8 --- @@ -19,30 +19,32 @@ The panel is split into sections, top to bottom: - **Scale**: the output size is the source size multiplied by this value. The panel shows the resulting input and output size and warns when the output is large enough to need much more time and VRAM. - **Treatment preset**, or **Creativity** and **Structure** directly. Higher Creativity invents more detail and gives the prompt more influence. Higher Structure keeps the source composition more strictly; it is only shown when a Tile ControlNet is in use. - **Detail guidance**: the prompt. It is the project prompt shared with Generate. For upscaling, describe the medium, texture and style rather than new content. -- **Generation**: the **Main model** for the detail pass (**SD1.5**, **SDXL**, **FLUX.1**, **Z-Image**, **Anima** or **Krea-2**), steps, CFG scale, scheduler, seed, and LoRAs that match the main model. FLUX.1, Z-Image and Krea-2 always run **Euler**, so they show no scheduler. +- **Generation**: the **Main model** for the detail pass (**SD1.5**, **SDXL**, **FLUX.1**, **FLUX.2**, **Z-Image**, **Anima**, **Krea-2** or **Qwen Image**), steps, CFG scale, scheduler, seed, and LoRAs that match the main model. FLUX.1, FLUX.2, Z-Image, Krea-2 and Qwen Image always run **Euler**, so they show no scheduler. - **Advanced**: the components the main model needs (see below), tile size and tile overlap, VAE and VAE precision, and CLIP skip for SD1.5. ## What each model family needs -| | SD1.5 / SDXL | FLUX.1 | Z-Image | Anima | Krea-2 | -|---|---|---|---|---|---| -| Tile ControlNet | Required: a Tile or Union ControlNet for the same family, under **Advanced** | Not used; the Structure slider is hidden | Optional: **Z-Image ControlNet Tile**, under **Advanced**. The Union ControlNet has no tile mode and is not offered | Not used; the Structure slider is hidden | Not used; the Structure slider is hidden | -| Negative prompt | Available | Not used; FLUX.1 ignores it, so it is hidden and not recorded | Used only when CFG is greater than 1 | Used only when CFG is greater than 1 | Used only when CFG is greater than 1 | -| CFG scale | Kept as you set it; back to 2 when you switch from Z-Image, Anima or Krea-2 | Kept as you set it, and used as guidance; back to 2 when you switch from Z-Image, Anima or Krea-2 | Set to the model's default when you pick the model: 1 for Z-Image Turbo, 4 for Z-Image Base | Set to the model's default when you pick the model: 4.5, or 6 for Anima-3.8B | Set to the model's default when you pick the model: 1 for Turbo, 5.5 for Raw | -| Other components | Taken from the main model | Every FLUX.1 model except an SDNQ pipeline needs a **T5 Encoder**, **CLIP Embed** and **VAE**, all under **Advanced**. SDNQ pipelines bring their own. FLUX Fill models cannot be used | A **Qwen3 Encoder** (the 4B) and a FLUX **VAE**, unless the model is a Diffusers or complete SDNQ install that brings its own | A **Qwen3 Encoder** (the 0.6B) and a Qwen-Image **VAE**; Anima-3.8B also a **Qwen3.5 Encoder** | A **Qwen3-VL Encoder** (the 4B) and a Qwen-Image **VAE**, unless the model is a Diffusers install that brings its own | -| Detail pass | Tile by tile | One pass over the whole image | One pass over the whole image | One pass over the whole image | One pass over the whole image | +| | SD1.5 / SDXL | FLUX.1 | FLUX.2 | Z-Image | Anima | Krea-2 | Qwen Image | +|---|---|---|---|---|---|---|---| +| Tile ControlNet | Required: a Tile or Union ControlNet for the same family, under **Advanced** | Not used; the Structure slider is hidden | Not used; the Structure slider is hidden | Optional: **Z-Image ControlNet Tile**, under **Advanced**. The Union ControlNet has no tile mode and is not offered | Not used; the Structure slider is hidden | Not used; the Structure slider is hidden | Not used; the Structure slider is hidden | +| Negative prompt | Available | Not used; FLUX.1 ignores it, so it is hidden and not recorded | Not used; FLUX.2 ignores it, so it is hidden and not recorded | Used only when CFG is greater than 1 | Used only when CFG is greater than 1 | Used only when CFG is greater than 1 | Used only when CFG is greater than 1 | +| CFG scale | Kept as you set it; back to 2 when you switch from a family that sets its own | Kept as you set it, and used as guidance; back to 2 when you switch from a family that sets its own | Set to the model's default when you pick the model, and used as guidance: 3.5 for FLUX.2 [dev], 1 for Klein, which ignores it | Set to the model's default when you pick the model: 1 for Z-Image Turbo, 4 for Z-Image Base | Set to the model's default when you pick the model: 4.5, or 6 for Anima-3.8B | Set to the model's default when you pick the model: 1 for Turbo, 5.5 for Raw | Set to the model's default when you pick the model: 4 | +| Other components | Taken from the main model | Every FLUX.1 model except an SDNQ pipeline needs a **T5 Encoder**, **CLIP Embed** and **VAE**, all under **Advanced**. SDNQ pipelines bring their own. FLUX Fill models cannot be used | Klein needs a **Qwen3 Encoder** of its size (the 4B for Klein 4B, the 8B for Klein 9B), [dev] a **Mistral Encoder**, and both a FLUX.2 **VAE**, unless the model is a Diffusers install that brings its own; for Klein a complete SDNQ install does too | A **Qwen3 Encoder** (the 4B) and a FLUX **VAE**, unless the model is a Diffusers or complete SDNQ install that brings its own | A **Qwen3 Encoder** (the 0.6B) and a Qwen-Image **VAE**; Anima-3.8B also a **Qwen3.5 Encoder** | A **Qwen3-VL Encoder** (the 4B) and a Qwen-Image **VAE**, unless the model is a Diffusers install that brings its own | A **Qwen VL Encoder** and a Qwen-Image **VAE**, unless the model is a Diffusers install that brings its own. Qwen Image Edit models cannot be used: without a reference image they would carry a blank one the size of the output through every step, for about 2.5 to 3 times the time and nothing gained | +| Detail pass | Tile by tile | One pass over the whole image | One pass over the whole image | One pass over the whole image | One pass over the whole image | One pass over the whole image | One pass over the whole image | The panel says what is missing before you can queue, for example "Select a compatible Tile or Union ControlNet." or "This model needs a T5 encoder selected." :::note[Output size] -FLUX.1, Z-Image and Krea-2 need sizes that are a multiple of 16. When the enlarged image is not, the panel shrinks it slightly to the nearest fitting size before the detail pass — for example 2040 px becomes 2032 px. +FLUX.1, FLUX.2, Z-Image, Krea-2 and Qwen Image need sizes that are a multiple of 16. When the enlarged image is not, the panel shrinks it slightly to the nearest fitting size before the detail pass — for example 2040 px becomes 2032 px. ::: -### Creativity on Z-Image, Anima and Krea-2 +### Creativity on Z-Image, Anima, Krea-2 and Qwen Image -Z-Image, Anima and Krea-2 count the start of the detail pass in steps rather than in noise level, so the same Creativity setting would add much more noise there than on FLUX.1. The panel translates Creativity for them: it starts the pass on the step whose noise level is closest to where FLUX.1 would start. A preset therefore means the same thing on all four, and the pass runs fewer steps than the Steps setting suggests. For example, **Balanced** at 30 steps on a 2048 px result runs the last 7 or 8, and each needs at least 2 steps. Krea-2 Turbo and Raw both keep your Steps setting: Turbo, distilled for 8 steps, then runs the end of a 30-step schedule, and Raw, the undistilled model, gets the steps it needs. +Z-Image, Anima, Krea-2 and Qwen Image count the start of the detail pass in steps rather than in noise level, so the same Creativity setting would add much more noise there than on FLUX.1. The panel translates Creativity for them: it starts the pass on the step whose noise level is closest to where FLUX.1 would start. A preset therefore means the same thing on every family, and the pass runs fewer steps than the Steps setting suggests. For example, **Balanced** at 30 steps on a 2048 px result runs the last 7 to 9, and each needs at least 2 steps. Krea-2 Turbo and Raw both keep your Steps setting: Turbo, distilled for 8 steps, then runs the end of a 30-step schedule, and Raw, the undistilled model, gets the steps it needs. -The panel offers only schedulers that start where Creativity puts the pass: **Euler** on FLUX.1, Z-Image and Krea-2, and **Euler**, **DPM++ 2M**, **DPM++ 2M SDE** and **ER-SDE** on Anima. Heun and LCM build their own noise schedule, so they would treat the enlarged image as pure noise or leave noise in it. +FLUX.2 starts where FLUX.1 does and then spreads all of your Steps over the rest of the pass, so 30 steps run 30 times. Picking a FLUX.2 model therefore sets its own step count: 4 for Klein 4B and 9B, 28 for Klein Base and [dev]. Switching to another family afterwards sets Steps back to 30. + +The panel offers only schedulers that start where Creativity puts the pass: **Euler** on FLUX.1, FLUX.2, Z-Image, Krea-2 and Qwen Image, and **Euler**, **DPM++ 2M**, **DPM++ 2M SDE** and **ER-SDE** on Anima. Heun and LCM build their own noise schedule, so they would treat the enlarged image as pure noise or leave noise in it. The Z-Image Tile ControlNet is optional because the detail pass runs over the whole image at once: there are no tile seams for it to hold together. Adding it lets **Structure** pull the result back toward the enlarged image, at the cost of a second network's memory and time. @@ -50,6 +52,6 @@ The Z-Image Tile ControlNet is optional because the detail pass runs over the wh For SD1.5 and SDXL the detail pass works in tiles, so the output can be much larger than what the model generates in one go. Larger tiles can improve continuity but need more VRAM; more tile overlap reduces seams but costs time and memory. -For FLUX.1, Z-Image, Anima and Krea-2 the detail pass runs over the whole image at once, so its memory use grows with the output size. There, **Tile size** only sets the tiles for the VAE encode and decode, and **Tile overlap** has no effect. Anima's decode picks its own tiles. +For FLUX.1, FLUX.2, Z-Image, Anima, Krea-2 and Qwen Image the detail pass runs over the whole image at once, so its memory use grows with the output size. There, **Tile size** only sets the tiles for the VAE encode and decode, and **Tile overlap** has no effect. Anima's decode picks its own tiles. For all model families, encoding and decoding the enlarged image with the VAE runs in tiles. See [Low-VRAM mode](/configuration/optimization/low-vram-mode/) if you still run out of memory. diff --git a/invokeai/frontend/webv2/public/locales/en.json b/invokeai/frontend/webv2/public/locales/en.json index e94c6a760df..72e8a0f565c 100644 --- a/invokeai/frontend/webv2/public/locales/en.json +++ b/invokeai/frontend/webv2/public/locales/en.json @@ -1005,6 +1005,12 @@ "heading": "LoRA Weight", "paragraphs": ["Weight of the LoRA. Higher weight will lead to larger impacts on the final image."] }, + "mistralEncoder": { + "heading": "Mistral Encoder", + "paragraphs": [ + "FLUX.2 [dev] reads your prompt with the Mistral Small 3.1 text encoder. A single-file or GGUF checkpoint does not carry one, so it has to be selected here." + ] + }, "model": { "heading": "Model", "paragraphs": [ @@ -1043,7 +1049,7 @@ "qwen3Encoder": { "heading": "Qwen3 Encoder", "paragraphs": [ - "Z-Image and Anima read your prompt with a Qwen3 text encoder: Z-Image with the 4B, Anima with the 0.6B. A bare checkpoint does not carry one, so it has to be selected here." + "Z-Image, FLUX.2 Klein and Anima read your prompt with a Qwen3 text encoder: Z-Image and Klein 4B with the 4B, Klein 9B with the 8B, Anima with the 0.6B. A bare checkpoint does not carry one, so it has to be selected here." ] }, "qwen3VlEncoder": { @@ -1052,6 +1058,12 @@ "Krea-2 reads your prompt with the Qwen3-VL 4B text encoder. A single-file or GGUF checkpoint does not carry one, so it has to be selected here." ] }, + "qwenVlEncoder": { + "heading": "Qwen VL Encoder", + "paragraphs": [ + "Qwen Image reads your prompt with the Qwen2.5-VL text encoder. A GGUF or single-file checkpoint does not carry one, so it has to be selected here." + ] + }, "referenceImage": { "heading": "Global Reference Image", "paragraphs": ["Applies a reference image to influence the entire generation."] @@ -2935,6 +2947,12 @@ "qwen3VlEncoder": "Qwen3-VL Encoder", "qwen3VlEncoderRequired": "This model needs a Qwen3-VL encoder selected.", "selectQwen3VlEncoder": "Select a Qwen3-VL encoder", + "qwenVlEncoder": "Qwen VL Encoder", + "qwenVlEncoderRequired": "This model needs a Qwen VL encoder selected.", + "selectQwenVlEncoder": "Select a Qwen VL encoder", + "mistralEncoder": "Mistral Encoder", + "mistralEncoderRequired": "This model needs a Mistral encoder selected.", + "selectMistralEncoder": "Select a Mistral encoder", "negativePromptCfgGated": "Used only when CFG is greater than 1.", "vaeRequired": "This model needs a VAE selected.", "tileControlNet": "Tile ControlNet", diff --git a/invokeai/frontend/webv2/src/features/generation/settings.ts b/invokeai/frontend/webv2/src/features/generation/settings.ts index 608fe1915ae..8835da181c6 100644 --- a/invokeai/frontend/webv2/src/features/generation/settings.ts +++ b/invokeai/frontend/webv2/src/features/generation/settings.ts @@ -77,6 +77,8 @@ export { isAnimaQwen3Encoder, isBundledMainForBase, isDiffusersMainForBase, + isFlux2MistralEncoder, + isFlux2Qwen3EncoderForModel, isKrea2Qwen3VlEncoder, isNonAnimaQwen3Encoder, isVaeAcceptedByBase, diff --git a/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleGraphNodeTypes.json b/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleGraphNodeTypes.json index 0edbbb13917..96b3e4a9e2e 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleGraphNodeTypes.json +++ b/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleGraphNodeTypes.json @@ -14,12 +14,16 @@ "collect collection anima_denoise negative_conditioning", "collect collection anima_denoise positive_conditioning", "collect collection anima_lora_collection_loader loras", + "collect collection flux2_denoise positive_text_conditioning", + "collect collection flux2_dev_lora_collection_loader loras", + "collect collection flux2_klein_lora_collection_loader loras", "collect collection flux_denoise positive_text_conditioning", "collect collection flux_lora_collection_loader loras", "collect collection krea2_denoise negative_conditioning", "collect collection krea2_denoise positive_conditioning", "collect collection krea2_lora_collection_loader loras", "collect collection lora_collection_loader loras", + "collect collection qwen_image_lora_collection_loader loras", "collect collection sdxl_lora_collection_loader loras", "collect collection tiled_multi_diffusion_denoise_latents control", "collect collection z_image_denoise negative_conditioning", @@ -29,10 +33,31 @@ "compel conditioning tiled_multi_diffusion_denoise_latents positive_conditioning", "controlnet control collect item", "core_metadata metadata anima_l2i metadata", + "core_metadata metadata flux2_vae_decode metadata", "core_metadata metadata flux_vae_decode metadata", "core_metadata metadata l2i metadata", "core_metadata metadata qwen_image_l2i metadata", "core_metadata metadata z_image_l2i metadata", + "flux2_denoise latents flux2_vae_decode latents", + "flux2_dev_lora_collection_loader mistral_encoder flux2_dev_text_encoder mistral_encoder", + "flux2_dev_lora_collection_loader transformer flux2_denoise transformer", + "flux2_dev_model_loader max_seq_len flux2_dev_text_encoder max_seq_len", + "flux2_dev_model_loader mistral_encoder flux2_dev_lora_collection_loader mistral_encoder", + "flux2_dev_model_loader transformer flux2_dev_lora_collection_loader transformer", + "flux2_dev_model_loader vae flux2_denoise vae", + "flux2_dev_model_loader vae flux2_vae_decode vae", + "flux2_dev_model_loader vae flux2_vae_encode vae", + "flux2_dev_text_encoder conditioning collect item", + "flux2_klein_lora_collection_loader qwen3_encoder flux2_klein_text_encoder qwen3_encoder", + "flux2_klein_lora_collection_loader transformer flux2_denoise transformer", + "flux2_klein_model_loader max_seq_len flux2_klein_text_encoder max_seq_len", + "flux2_klein_model_loader qwen3_encoder flux2_klein_lora_collection_loader qwen3_encoder", + "flux2_klein_model_loader transformer flux2_klein_lora_collection_loader transformer", + "flux2_klein_model_loader vae flux2_denoise vae", + "flux2_klein_model_loader vae flux2_vae_decode vae", + "flux2_klein_model_loader vae flux2_vae_encode vae", + "flux2_klein_text_encoder conditioning collect item", + "flux2_vae_encode latents flux2_denoise latents", "flux_denoise latents flux_vae_decode latents", "flux_lora_collection_loader clip flux_text_encoder clip", "flux_lora_collection_loader t5_encoder flux_text_encoder t5_encoder", @@ -48,6 +73,7 @@ "i2l latents tiled_multi_diffusion_denoise_latents latents", "img_resize height core_metadata height", "img_resize image anima_i2l image", + "img_resize image flux2_vae_encode image", "img_resize image flux_vae_encode image", "img_resize image qwen_image_i2l image", "img_resize image z_image_control image", @@ -55,9 +81,11 @@ "img_resize width core_metadata width", "integer value anima_denoise seed", "integer value core_metadata seed", + "integer value flux2_denoise seed", "integer value flux_denoise seed", "integer value krea2_denoise seed", "integer value noise seed", + "integer value qwen_image_denoise seed", "integer value z_image_denoise seed", "krea2_denoise latents qwen_image_l2i latents", "krea2_lora_collection_loader qwen3_vl_encoder krea2_text_encoder qwen3_vl_encoder", @@ -75,7 +103,16 @@ "main_model_loader vae i2l vae", "main_model_loader vae l2i vae", "noise noise tiled_multi_diffusion_denoise_latents noise", + "qwen_image_denoise latents qwen_image_l2i latents", "qwen_image_i2l latents krea2_denoise latents", + "qwen_image_i2l latents qwen_image_denoise latents", + "qwen_image_lora_collection_loader transformer qwen_image_denoise transformer", + "qwen_image_model_loader qwen_vl_encoder qwen_image_text_encoder qwen_vl_encoder", + "qwen_image_model_loader transformer qwen_image_lora_collection_loader transformer", + "qwen_image_model_loader vae qwen_image_i2l vae", + "qwen_image_model_loader vae qwen_image_l2i vae", + "qwen_image_text_encoder conditioning qwen_image_denoise negative_conditioning", + "qwen_image_text_encoder conditioning qwen_image_denoise positive_conditioning", "sdxl_compel_prompt conditioning tiled_multi_diffusion_denoise_latents negative_conditioning", "sdxl_compel_prompt conditioning tiled_multi_diffusion_denoise_latents positive_conditioning", "sdxl_lora_collection_loader clip sdxl_compel_prompt clip", @@ -91,8 +128,11 @@ "string value compel prompt", "string value core_metadata negative_prompt", "string value core_metadata positive_prompt", + "string value flux2_dev_text_encoder prompt", + "string value flux2_klein_text_encoder prompt", "string value flux_text_encoder prompt", "string value krea2_text_encoder prompt", + "string value qwen_image_text_encoder prompt", "string value sdxl_compel_prompt prompt", "string value sdxl_compel_prompt style", "string value z_image_text_encoder prompt", @@ -229,6 +269,45 @@ "string", "unsharp_mask" ] + }, + "flux2": { + "nodeTypes": [ + "collect", + "core_metadata", + "flux2_denoise", + "flux2_dev_lora_collection_loader", + "flux2_dev_model_loader", + "flux2_dev_text_encoder", + "flux2_klein_lora_collection_loader", + "flux2_klein_model_loader", + "flux2_klein_text_encoder", + "flux2_vae_decode", + "flux2_vae_encode", + "img_resize", + "integer", + "lora_selector", + "spandrel_image_to_image_autoscale", + "string", + "unsharp_mask" + ] + }, + "qwen-image": { + "nodeTypes": [ + "collect", + "core_metadata", + "img_resize", + "integer", + "lora_selector", + "qwen_image_denoise", + "qwen_image_i2l", + "qwen_image_l2i", + "qwen_image_lora_collection_loader", + "qwen_image_model_loader", + "qwen_image_text_encoder", + "spandrel_image_to_image_autoscale", + "string", + "unsharp_mask" + ] } }, "fieldsByNodeType": { @@ -480,13 +559,17 @@ 0 ], "guidance": [ - 2 + 1, + 2, + 3.5 ], "is_intermediate": [ true ], "loras": [], + "mistral_encoder": [], "model": [], + "qwen_image_qwen_vl_encoder": [], "qwen3_5_encoder": [], "qwen3_encoder": [], "qwen3_vl_encoder": [], @@ -498,7 +581,9 @@ "kdpm_2" ], "steps": [ - 30 + 28, + 30, + 4 ], "structure": [ 0 @@ -671,6 +756,208 @@ "latents" ] }, + "flux2_denoise": { + "inputs": [ + "latents", + "positive_text_conditioning", + "seed", + "transformer", + "vae" + ], + "literalInputs": { + "denoising_end": [ + 1 + ], + "denoising_start": [ + 0.49900000000000005 + ], + "guidance": [ + 1, + 3.5 + ], + "height": [ + 400 + ], + "is_intermediate": [ + true + ], + "num_steps": [ + 28, + 4 + ], + "scheduler": [ + "euler" + ], + "use_cache": [ + true + ], + "width": [ + 800 + ] + }, + "outputs": [ + "latents" + ] + }, + "flux2_dev_lora_collection_loader": { + "inputs": [ + "loras", + "mistral_encoder", + "transformer" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "mistral_encoder", + "transformer" + ] + }, + "flux2_dev_model_loader": { + "inputs": [], + "literalInputs": { + "is_intermediate": [ + true + ], + "mistral_encoder_model": [], + "model": [], + "use_cache": [ + true + ], + "vae_model": [] + }, + "outputs": [ + "max_seq_len", + "mistral_encoder", + "transformer", + "vae" + ] + }, + "flux2_dev_text_encoder": { + "inputs": [ + "max_seq_len", + "mistral_encoder", + "prompt" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "conditioning" + ] + }, + "flux2_klein_lora_collection_loader": { + "inputs": [ + "loras", + "qwen3_encoder", + "transformer" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "qwen3_encoder", + "transformer" + ] + }, + "flux2_klein_model_loader": { + "inputs": [], + "literalInputs": { + "is_intermediate": [ + true + ], + "model": [], + "qwen3_encoder_model": [], + "use_cache": [ + true + ], + "vae_model": [] + }, + "outputs": [ + "max_seq_len", + "qwen3_encoder", + "transformer", + "vae" + ] + }, + "flux2_klein_text_encoder": { + "inputs": [ + "max_seq_len", + "prompt", + "qwen3_encoder" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "conditioning" + ] + }, + "flux2_vae_decode": { + "inputs": [ + "latents", + "metadata", + "vae" + ], + "literalInputs": { + "is_intermediate": [ + false + ], + "tile_size": [ + 1024 + ], + "tiled": [ + true + ], + "use_cache": [ + false + ] + }, + "outputs": [] + }, + "flux2_vae_encode": { + "inputs": [ + "image", + "vae" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "tile_size": [ + 1024 + ], + "tiled": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "latents" + ] + }, "i2l": { "inputs": [ "image", @@ -934,6 +1221,45 @@ "noise" ] }, + "qwen_image_denoise": { + "inputs": [ + "latents", + "negative_conditioning", + "positive_conditioning", + "seed", + "transformer" + ], + "literalInputs": { + "cfg_scale": [ + 1, + 4 + ], + "denoising_end": [ + 1 + ], + "denoising_start": [ + 0.6 + ], + "height": [ + 400 + ], + "is_intermediate": [ + true + ], + "steps": [ + 30 + ], + "use_cache": [ + true + ], + "width": [ + 800 + ] + }, + "outputs": [ + "latents" + ] + }, "qwen_image_i2l": { "inputs": [ "image", @@ -979,6 +1305,59 @@ }, "outputs": [] }, + "qwen_image_lora_collection_loader": { + "inputs": [ + "loras", + "transformer" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "transformer" + ] + }, + "qwen_image_model_loader": { + "inputs": [], + "literalInputs": { + "is_intermediate": [ + true + ], + "model": [], + "qwen_vl_encoder_model": [], + "use_cache": [ + true + ], + "vae_model": [] + }, + "outputs": [ + "qwen_vl_encoder", + "transformer", + "vae" + ] + }, + "qwen_image_text_encoder": { + "inputs": [ + "prompt", + "qwen_vl_encoder" + ], + "literalInputs": { + "is_intermediate": [ + true + ], + "use_cache": [ + true + ] + }, + "outputs": [ + "conditioning" + ] + }, "sdxl_compel_prompt": { "inputs": [ "clip", diff --git a/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleIndexClippedStarts.json b/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleIndexClippedStarts.json index 2fa5056b50d..5c238a8fa37 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleIndexClippedStarts.json +++ b/invokeai/frontend/webv2/src/features/upscale/core/__snapshots__/upscaleIndexClippedStarts.json @@ -1,5 +1,5 @@ { - "_comment": "Generated by src/features/upscale/core/indexClippedStarts.test.ts; regenerate with `vitest -u`. Consumed by tests/app/invocations/test_upscale_index_clipped_starts.py, which drives the real z_image_denoise, anima_denoise and krea2_denoise with each denoisingStart and checks the pass enters at the step nearest fluxEntrySigma, and that the schedulers offered are the ones that start there.", + "_comment": "Generated by src/features/upscale/core/indexClippedStarts.test.ts; regenerate with `vitest -u`. Consumed by tests/app/invocations/test_upscale_index_clipped_starts.py, which drives the real z_image_denoise, anima_denoise, krea2_denoise and qwen_image_denoise with each denoisingStart and checks the pass enters at the step nearest fluxEntrySigma, and that the schedulers offered are the ones that start there.", "cases": [ { "base": "anima", @@ -649,6 +649,438 @@ "steps": 30, "variant": "krea2_base" }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.875, + "fluxEntrySigma": 0.0019999999999998908, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.75, + "fluxEntrySigma": 0.25149999999999995, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.5, + "fluxEntrySigma": 0.5009999999999999, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.25, + "fluxEntrySigma": 0.7505, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.125, + "fluxEntrySigma": 0.9002, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 512, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.9666666666666667, + "fluxEntrySigma": 0.0019999999999998908, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.8, + "fluxEntrySigma": 0.25149999999999995, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.6, + "fluxEntrySigma": 0.5009999999999999, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.36666666666666664, + "fluxEntrySigma": 0.7505, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.16666666666666666, + "fluxEntrySigma": 0.9002, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 512, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.875, + "fluxEntrySigma": 0.0019999999999998908, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.75, + "fluxEntrySigma": 0.25149999999999995, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.625, + "fluxEntrySigma": 0.5009999999999999, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.375, + "fluxEntrySigma": 0.7505, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.125, + "fluxEntrySigma": 0.9002, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 1024, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.9666666666666667, + "fluxEntrySigma": 0.0019999999999998908, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.8333333333333334, + "fluxEntrySigma": 0.25149999999999995, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.6333333333333333, + "fluxEntrySigma": 0.5009999999999999, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.4, + "fluxEntrySigma": 0.7505, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.16666666666666666, + "fluxEntrySigma": 0.9002, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 1024, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.875, + "fluxEntrySigma": 0.0019999999999998908, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.75, + "fluxEntrySigma": 0.25149999999999995, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.625, + "fluxEntrySigma": 0.5009999999999999, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.375, + "fluxEntrySigma": 0.7505, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.125, + "fluxEntrySigma": 0.9002, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 2048, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.9666666666666667, + "fluxEntrySigma": 0.0019999999999998908, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.8666666666666667, + "fluxEntrySigma": 0.25149999999999995, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.7, + "fluxEntrySigma": 0.5009999999999999, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.43333333333333335, + "fluxEntrySigma": 0.7505, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.2, + "fluxEntrySigma": 0.9002, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 2048, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.875, + "fluxEntrySigma": 0.0019999999999998908, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.75, + "fluxEntrySigma": 0.25149999999999995, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.625, + "fluxEntrySigma": 0.5009999999999999, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.375, + "fluxEntrySigma": 0.7505, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.125, + "fluxEntrySigma": 0.9002, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 4096, + "steps": 8, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -10, + "denoisingStart": 0.9666666666666667, + "fluxEntrySigma": 0.0019999999999998908, + "side": 4096, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": -5, + "denoisingStart": 0.8666666666666667, + "fluxEntrySigma": 0.25149999999999995, + "side": 4096, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 0, + "denoisingStart": 0.7, + "fluxEntrySigma": 0.5009999999999999, + "side": 4096, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 5, + "denoisingStart": 0.43333333333333335, + "fluxEntrySigma": 0.7505, + "side": 4096, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 8, + "denoisingStart": 0.2, + "fluxEntrySigma": 0.9002, + "side": 4096, + "steps": 30, + "variant": null + }, + { + "base": "qwen-image", + "creativity": 10, + "denoisingStart": 0, + "fluxEntrySigma": 1, + "side": 4096, + "steps": 30, + "variant": null + }, { "base": "z-image", "creativity": -10, @@ -1092,6 +1524,9 @@ "krea-2": [ "euler" ], + "qwen-image": [ + "euler" + ], "z-image": [ "euler" ] diff --git a/invokeai/frontend/webv2/src/features/upscale/core/graph.test.ts b/invokeai/frontend/webv2/src/features/upscale/core/graph.test.ts index 157dedbb9c3..3ef6f356cdf 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/graph.test.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/graph.test.ts @@ -476,6 +476,178 @@ describe('compileUpscaleGraph for Krea-2', () => { }); }); +const createFlux2Values = (variant: string, format = 'checkpoint') => { + const models = [ + { ...withVariant(model('main', 'main', 'flux2'), variant), format } as ModelConfig, + model('spandrel', 'spandrel_image_to_image', 'any'), + model('lora', 'lora', 'flux2'), + model('flux2vae', 'vae', 'flux2'), + // The 8B and the Ministral sort ahead of the encoders Klein 4B and [dev] take. + withVariant(model('qwen3-8b', 'qwen3_encoder', 'any'), 'qwen3_8b'), + withVariant(model('qwen3-4b', 'qwen3_encoder', 'any'), 'qwen3_4b'), + withVariant(model('ministral', 'mistral_encoder', 'any'), 'ministral3_3b'), + withVariant(model('mistral', 'mistral_encoder', 'any'), 'mistral3_24b'), + ]; + + return { + ...createDefaultUpscaleWidgetValues(models), + cfgScale: 4, + inputImage: { height: 101, image_name: 'input.png', width: 203 }, + loras: [{ isEnabled: true, model: models[2] as GenerateLora['model'], weight: 0.7 }], + negativePrompt: 'blur', + positivePrompt: 'detail', + }; +}; + +describe('compileUpscaleGraph for FLUX.2', () => { + it('drives Klein with its own nodes, the guidance and the VAE statistics, tiled at both ends', () => { + const { edges, nodes } = compileUpscaleGraph(createFlux2Values('klein_4b'), 'gallery', { + useCpuNoise: true, + }).backendGraph; + + expect(nodes.model_loader).toMatchObject({ + qwen3_encoder_model: { key: 'qwen3-4b' }, + type: 'flux2_klein_model_loader', + vae_model: { key: 'flux2vae' }, + }); + expect(nodes.i2l).toMatchObject({ tile_size: 1024, tiled: true, type: 'flux2_vae_encode' }); + expect(nodes.upscale_output).toMatchObject({ tile_size: 1024, tiled: true, type: 'flux2_vae_decode' }); + // 203x101 at 4x: Spandrel gives 808x400, and flux2_denoise takes multiples of 16 only. A new Klein runs at + // the 4 steps it is served with. + expect(nodes.denoise_latents).toMatchObject({ + guidance: 4, + height: 400, + num_steps: 4, + scheduler: 'euler', + type: 'flux2_denoise', + width: 800, + }); + // CFG stays at the node's 1, as in Generate: the slider is guidance here, and there is no negative branch. + expect(nodes.denoise_latents).not.toHaveProperty('cfg_scale'); + expect(nodes.neg_cond).toBeUndefined(); + expect(nodes.core_metadata).toMatchObject({ guidance: 4, qwen3_encoder: { key: 'qwen3-4b' } }); + expect(nodes.core_metadata).not.toHaveProperty('cfg_scale'); + expect(hasEdge(edges, 'negative_prompt', 'value', 'core_metadata', 'negative_prompt')).toBe(false); + // The denoise normalizes the encoded latents with the VAE's BN statistics. + expect(hasEdge(edges, 'model_loader', 'vae', 'denoise_latents', 'vae')).toBe(true); + expect(hasEdge(edges, 'model_loader', 'max_seq_len', 'pos_cond', 'max_seq_len')).toBe(true); + expect(hasEdge(edges, 'pos_cond_collect', 'collection', 'denoise_latents', 'positive_text_conditioning')).toBe( + true + ); + expect(hasEdge(edges, 'fit_to_grid', 'image', 'i2l', 'image')).toBe(true); + expect(nodes.pos_cond).toMatchObject({ type: 'flux2_klein_text_encoder' }); + expect(Object.values(nodes).some((node) => node.type === 'flux2_klein_lora_collection_loader')).toBe(true); + }); + + it('hands Klein 9B the 8B encoder it was trained on', () => { + const { nodes } = compileUpscaleGraph(createFlux2Values('klein_9b_base'), 'gallery', { + useCpuNoise: true, + }).backendGraph; + + expect(nodes.model_loader?.qwen3_encoder_model).toMatchObject({ key: 'qwen3-8b' }); + }); + + it('drives [dev] with its Mistral encoder, never the Ministral, and records it', () => { + const { edges, nodes } = compileUpscaleGraph(createFlux2Values('dev', 'gguf_quantized'), 'gallery', { + useCpuNoise: true, + }).backendGraph; + const loraLoader = Object.entries(nodes).find(([, node]) => node.type === 'flux2_dev_lora_collection_loader'); + + expect(nodes.model_loader).toMatchObject({ + mistral_encoder_model: { key: 'mistral' }, + type: 'flux2_dev_model_loader', + }); + expect(nodes.model_loader).not.toHaveProperty('qwen3_encoder_model'); + expect(nodes.pos_cond).toMatchObject({ type: 'flux2_dev_text_encoder' }); + expect(loraLoader).toBeDefined(); + expect(hasEdge(edges, loraLoader![0], 'mistral_encoder', 'pos_cond', 'mistral_encoder')).toBe(true); + expect(hasEdge(edges, 'model_loader', 'max_seq_len', 'pos_cond', 'max_seq_len')).toBe(true); + expect(nodes.core_metadata).toMatchObject({ mistral_encoder: { key: 'mistral' } }); + expect(nodes.core_metadata).not.toHaveProperty('qwen3_encoder'); + }); + + it('lets a Diffusers install supply its own encoder and VAE', () => { + const values = { ...createFlux2Values('klein_4b', 'diffusers'), vae: null }; + const { nodes } = compileUpscaleGraph(values, 'gallery', { useCpuNoise: true }).backendGraph; + + expect(nodes.model_loader?.qwen3_encoder_model).toBeUndefined(); + expect(nodes.model_loader?.vae_model).toBeUndefined(); + }); +}); + +const createQwenImageValues = (options: { cfgScale?: number; format?: string } = {}) => { + const models = [ + { ...withVariant(model('main', 'main', 'qwen-image'), 'generate'), format: options.format ?? 'gguf_quantized' }, + model('spandrel', 'spandrel_image_to_image', 'any'), + model('lora', 'lora', 'qwen-image'), + model('qwenvae', 'vae', 'qwen-image'), + model('qwen-vl', 'qwen_vl_encoder', 'any'), + ] as ModelConfig[]; + + return { + ...createDefaultUpscaleWidgetValues(models), + cfgScale: options.cfgScale ?? 4, + inputImage: { height: 101, image_name: 'input.png', width: 203 }, + loras: [{ isEnabled: true, model: models[2] as GenerateLora['model'], weight: 0.7 }], + negativePrompt: 'blur', + positivePrompt: 'detail', + }; +}; + +describe('compileUpscaleGraph for Qwen Image', () => { + it('drives the second pass with qwen_image nodes, tiled at both ends, on the shift the node picks', () => { + const { edges, nodes } = compileUpscaleGraph(createQwenImageValues(), 'gallery', { + useCpuNoise: true, + }).backendGraph; + + expect(nodes.model_loader).toMatchObject({ + qwen_vl_encoder_model: { key: 'qwen-vl' }, + type: 'qwen_image_model_loader', + vae_model: { key: 'qwenvae' }, + }); + expect(nodes.i2l).toMatchObject({ tile_size: 1024, tiled: true, type: 'qwen_image_i2l' }); + expect(nodes.upscale_output).toMatchObject({ tile_size: 1024, tiled: true, type: 'qwen_image_l2i' }); + expect(nodes.denoise_latents).toMatchObject({ + cfg_scale: 4, + height: 400, + steps: 30, + type: 'qwen_image_denoise', + width: 800, + }); + // A shift would replace the held dynamic one that Creativity is translated against. + expect(nodes.denoise_latents).not.toHaveProperty('shift'); + expect(nodes.denoise_latents).not.toHaveProperty('scheduler'); + // The denoise takes one conditioning per branch, not a collected list. + expect(hasEdge(edges, 'pos_cond', 'conditioning', 'denoise_latents', 'positive_conditioning')).toBe(true); + expect(hasEdge(edges, 'neg_cond', 'conditioning', 'denoise_latents', 'negative_conditioning')).toBe(true); + expect(nodes.pos_cond_collect).toBeUndefined(); + expect(hasEdge(edges, 'model_loader', 'qwen_vl_encoder', 'neg_cond', 'qwen_vl_encoder')).toBe(true); + expect(Object.values(nodes).some((node) => node.type === 'qwen_image_lora_collection_loader')).toBe(true); + expect(nodes.core_metadata).toMatchObject({ + cfg_scale: 4, + qwen_image_qwen_vl_encoder: { key: 'qwen-vl' }, + scheduler: 'euler', + }); + }); + + it('wires the negative prompt only above CFG 1, where the node runs an unconditional pass', () => { + const { edges, nodes } = compileUpscaleGraph(createQwenImageValues({ cfgScale: 1 }), 'gallery', { + useCpuNoise: true, + }).backendGraph; + + expect(nodes.neg_cond).toBeUndefined(); + expect(hasEdge(edges, 'negative_prompt', 'value', 'core_metadata', 'negative_prompt')).toBe(false); + }); + + it('lets a Diffusers install supply its own encoder and VAE', () => { + const values = { ...createQwenImageValues({ format: 'diffusers' }), vae: null }; + const { nodes } = compileUpscaleGraph(values, 'gallery', { useCpuNoise: true }).backendGraph; + + expect(nodes.model_loader?.qwen_vl_encoder_model).toBeUndefined(); + expect(nodes.model_loader?.vae_model).toBeUndefined(); + }); +}); + describe('Creativity on index-clipping architectures', () => { // Expected starts come from the backend's own schedules (`ZImageDenoiseInvocation._get_sigmas` // with `_calculate_shift`, `AnimaDenoiseInvocation._get_sigmas`) and the nodes' clipping line, @@ -521,8 +693,25 @@ describe('Creativity on index-clipping architectures', () => { expect(startFor(createKrea2Values(variant), creativity, side)).toBeCloseTo(expected, 6); }); - it('leaves the value-clipping FLUX pass on the legacy formula', () => { - expect(startFor(createFluxValues(), 0, 1024)).toBeCloseTo(0.499, 6); + // `qwen_image_denoise` rounds as well. Its scheduler shifts by `exp(mu)`, mu = 0.5 + 0.4 * (tokens - 256) / 7936 + // held at 0.9 past 8192 tokens, then stretches every sigma to `1 - (1 - s) / k`, `k = (1 - s(1/30)) / 0.98`, so + // the last step lands on 0.02. Expected steps were worked out by hand from that schedule at `t = 1 - i / 30`. + it.each([ + // 1024px result, 4096 tokens: mu 0.694, k 0.9545. Step 19 sits at 0.515 and step 20 at 0.476; unstretched + // they would be 0.537 and 0.500, and the pass would start a step later. + [512, 19 / 30], + // 4096px result: held at mu 0.9, k 0.9406. Step 20 sits at 0.523, step 21 at 0.482. + [2048, 21 / 30], + ])('starts Qwen Image from a %ipx input where FLUX starts', (side, expected) => { + expect(startFor(createQwenImageValues(), 0, side)).toBeCloseTo(expected, 6); + }); + + it.each([ + ['FLUX.1', () => createFluxValues()], + // `flux2_denoise` clips by value too, then re-spaces the window to the requested steps. + ['FLUX.2', () => createFlux2Values('klein_4b')], + ] as const)('leaves the value-clipping %s pass on the legacy formula', (_label, create) => { + expect(startFor(create(), 0, 1024)).toBeCloseTo(0.499, 6); }); }); @@ -531,6 +720,7 @@ describe('schedulers on the upscale pass', () => { // narrowed -- would start the pass from pure noise, or on Anima below the blend. it.each([ ['flux', () => createFluxValues(), 'lcm', 'euler'], + ['flux2', () => createFlux2Values('klein_4b'), 'heun', 'euler'], ['z-image', () => createZImageValues(), 'heun', 'euler'], ['anima', () => createAnimaValues(), 'heun', 'euler'], ['anima', () => createAnimaValues(), 'dpmpp_2m', 'dpmpp_2m'], @@ -552,6 +742,8 @@ describe('upscale metadata', () => { ['z-image', () => createZImageValues()], ['anima', () => createAnimaValues()], ['krea-2', () => createKrea2Values('krea2_base')], + ['flux2', () => createFlux2Values('klein_4b')], + ['qwen-image', () => createQwenImageValues()], ] as const)('records the size of the image the %s pass encoded, which is the size of the result', (_base, create) => { // Where a pass resizes Spandrel's frame, the result does not have Spandrel's size: a 1x // restoration model leaves the frame unscaled, and its multiple of 8 is not FLUX's or Z-Image's diff --git a/invokeai/frontend/webv2/src/features/upscale/core/graph.ts b/invokeai/frontend/webv2/src/features/upscale/core/graph.ts index a1be3eacd88..bba85261cc2 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/graph.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/graph.ts @@ -38,28 +38,25 @@ export const getUpscaleDenoisingStart = (creativity: number): number => ((creati * The start an index-clipping node needs to enter where a value-clipping one would. * * Creativity is defined by where FLUX enters: `flux_denoise` clips the schedule by *value*, so a start - * of 0.499 enters at sigma 0.501 whatever the shift or step count. Z-Image and Anima clip by *index* - * instead, so the same number enters far noisier there -- 0.499 starts Z-Image at sigma 0.78. All of - * them build `sigma = shift * t / (1 + (shift - 1) * t)` at `t = 1 - i / steps`. This picks the `i` - * whose sigma is nearest `1 - valueStart`. Z-Image and Anima begin at `i = int(start * steps)`, so the - * start aims at the middle of that step, where float error in the node's product cannot floor it one - * step short. Krea-2 rounds instead, half to even, so its start is the step itself. This goes away when - * those nodes clip by value; `test_upscale_index_clipped_starts.py` fails when they do. + * of 0.499 enters at sigma 0.501 whatever the shift or step count. Z-Image, Anima, Krea-2 and Qwen Image + * clip by *index* instead, so the same number enters far noisier there -- 0.499 starts Z-Image at sigma + * 0.78. Given the node's own `sigmas` (`indexClipping.sigmas`), this picks the step `i` whose sigma is + * nearest `1 - valueStart`. Z-Image and Anima begin at `i = int(start * steps)`, so the start aims at the + * middle of that step, where float error in the node's product cannot floor it one step short. Krea-2 and + * Qwen Image round instead, half to even, so their start is the step itself. This goes away when those + * nodes clip by value; `test_upscale_index_clipped_starts.py` fails when they do. */ export const getIndexClippedDenoisingStart = ( valueStart: number, - steps: number, - shift: number, + sigmas: readonly number[], rounding: UpscaleIndexClipping['rounding'] ): number => { const target = 1 - valueStart; let entryIndex = 0; let entryDistance = Number.POSITIVE_INFINITY; - // The last index would leave nothing to denoise, so it is never a candidate. - for (let index = 0; index < steps; index += 1) { - const t = 1 - index / steps; - const distance = Math.abs((shift * t) / (1 + (shift - 1) * t) - target); + for (const [index, sigma] of sigmas.entries()) { + const distance = Math.abs(sigma - target); if (distance < entryDistance) { entryIndex = index; @@ -71,7 +68,7 @@ export const getIndexClippedDenoisingStart = ( return 0; } - return rounding === 'floor' ? (entryIndex + 0.5) / steps : entryIndex / steps; + return rounding === 'floor' ? (entryIndex + 0.5) / sigmas.length : entryIndex / sigmas.length; }; export const getUpscaleControlNetValues = (structure: number) => { @@ -128,6 +125,11 @@ const addUpscaleMetadata = ( ...(components.qwen3EncoderModel ? { qwen3_encoder: toModelIdentifier(components.qwen3EncoderModel) } : {}), ...(components.qwen35EncoderModel ? { qwen3_5_encoder: toModelIdentifier(components.qwen35EncoderModel) } : {}), ...(components.qwen3VlEncoderModel ? { qwen3_vl_encoder: toModelIdentifier(components.qwen3VlEncoderModel) } : {}), + ...(components.mistralEncoderModel ? { mistral_encoder: toModelIdentifier(components.mistralEncoderModel) } : {}), + // Generate records Qwen Image's encoder under its own key, and Remix reads it back from there. + ...(components.qwenVlEncoderModel + ? { qwen_image_qwen_vl_encoder: toModelIdentifier(components.qwenVlEncoderModel) } + : {}), creativity: settings.creativity, id: 'core_metadata', // Runs after the path builder, so the graph already holds the LoRAs this architecture loads. @@ -459,6 +461,110 @@ const buildFluxUpscalePath: UpscalePathBuilder = ({ return { frame: fitted, output, title: `${model.name} upscale` }; }; +/** + * FLUX.2 runs the pass whole like FLUX.1, both VAE ends tiled. Klein and [dev] differ in their loader, text + * encoder and LoRA loader. As in Generate, CFG stays off and the slider drives the guidance [dev] is distilled + * on; Klein ignores it. + */ +const buildFlux2UpscalePath: UpscalePathBuilder = ({ + graph, + settings, + model, + components, + destination, + positivePrompt, + seed, + frame, + denoiseSize, + denoisingStart, +}) => { + const isDev = model.variant === 'dev'; + const encoderField = isDev ? 'mistral_encoder' : 'qwen3_encoder'; + const vaeModel = settings.vae ? toModelIdentifier(settings.vae) : undefined; + const modelLoader = addNode( + graph, + isDev + ? { + id: 'model_loader', + mistral_encoder_model: components.mistralEncoderModel + ? toModelIdentifier(components.mistralEncoderModel) + : undefined, + model: toModelIdentifier(model), + type: 'flux2_dev_model_loader', + vae_model: vaeModel, + } + : { + id: 'model_loader', + model: toModelIdentifier(model), + qwen3_encoder_model: components.qwen3EncoderModel + ? toModelIdentifier(components.qwen3EncoderModel) + : undefined, + type: 'flux2_klein_model_loader', + vae_model: vaeModel, + } + ); + const activeLoras = getActiveCompatibleLoras({ loras: settings.loras }, model); + const loraSource = activeLoras.length + ? addTransformerLoraCollectionLoader( + graph, + activeLoras, + isDev ? 'flux2_dev_lora_collection_loader' : 'flux2_klein_lora_collection_loader', + modelLoader, + ['transformer', encoderField] + ) + : modelLoader; + const posCond = addNode(graph, { + id: 'pos_cond', + type: isDev ? 'flux2_dev_text_encoder' : 'flux2_klein_text_encoder', + }); + const posCondCollect = addNode(graph, { id: 'pos_cond_collect', type: 'collect' }); + const imageToLatents = addNode(graph, { + id: 'i2l', + tile_size: settings.tileSize, + tiled: true, + type: 'flux2_vae_encode', + }); + const denoise = addNode(graph, { + denoising_end: 1, + denoising_start: denoisingStart, + guidance: settings.cfgScale, + height: denoiseSize.height, + id: 'denoise_latents', + num_steps: settings.steps, + scheduler: coerceUpscaleScheduler(model, settings.scheduler), + type: 'flux2_denoise', + width: denoiseSize.width, + }); + const output = addNode(graph, { + id: 'upscale_output', + is_intermediate: destination === 'canvas', + tile_size: settings.tileSize, + tiled: true, + type: 'flux2_vae_decode', + use_cache: false, + }); + + addEdge(graph, loraSource, encoderField, posCond, encoderField); + addEdge(graph, modelLoader, 'max_seq_len', posCond, 'max_seq_len'); + addEdge(graph, positivePrompt, 'value', posCond, 'prompt'); + addEdge(graph, posCond, 'conditioning', posCondCollect, 'item'); + addEdge(graph, posCondCollect, 'collection', denoise, 'positive_text_conditioning'); + addEdge(graph, loraSource, 'transformer', denoise, 'transformer'); + addEdge(graph, seed, 'value', denoise, 'seed'); + + const fitted = fitFrameToDenoiseSize(graph, frame, denoiseSize); + + addEdge(graph, fitted, 'image', imageToLatents, 'image'); + addEdge(graph, modelLoader, 'vae', imageToLatents, 'vae'); + addEdge(graph, imageToLatents, 'latents', denoise, 'latents'); + // The denoise normalizes the encoded latents with this VAE's BN statistics. + addEdge(graph, modelLoader, 'vae', denoise, 'vae'); + addEdge(graph, denoise, 'latents', output, 'latents'); + addEdge(graph, modelLoader, 'vae', output, 'vae'); + + return { frame: fitted, output, title: `${model.name} upscale` }; +}; + /** * Z-Image runs the pass whole like FLUX.1, both VAE ends tiled. Its tile ControlNet is optional: a * pass that does not tile has no seams to hold together, so the ControlNet only lets Structure pull @@ -720,18 +826,99 @@ const buildKrea2UpscalePath: UpscalePathBuilder = ({ }; /** - * Z-Image's, Anima's and Krea-2's text encoders take their encoders on named fields and hand the denoise a - * collected list; the negative branch exists only where the caller decided it is read. + * Qwen Image runs the pass whole like FLUX.1, through the VAE nodes Krea-2 uses, tiled at both ends. The graph + * sends no shift, so `qwen_image_denoise` keeps its dynamic one, held past 8192 tokens, which is the schedule + * Creativity is translated against. Validation refuses Edit models, which would run against a blank reference. + */ +const buildQwenImageUpscalePath: UpscalePathBuilder = ({ + graph, + settings, + model, + components, + destination, + positivePrompt, + negativePrompt, + seed, + frame, + denoiseSize, + denoisingStart, +}) => { + const modelLoader = addNode(graph, { + id: 'model_loader', + model: toModelIdentifier(model), + qwen_vl_encoder_model: components.qwenVlEncoderModel ? toModelIdentifier(components.qwenVlEncoderModel) : undefined, + type: 'qwen_image_model_loader', + vae_model: settings.vae ? toModelIdentifier(settings.vae) : undefined, + }); + const activeLoras = getActiveCompatibleLoras({ loras: settings.loras }, model); + const loraSource = activeLoras.length + ? addTransformerLoraCollectionLoader(graph, activeLoras, 'qwen_image_lora_collection_loader', modelLoader, [ + 'transformer', + ]) + : modelLoader; + const imageToLatents = addNode(graph, { + id: 'i2l', + tile_size: settings.tileSize, + tiled: true, + type: 'qwen_image_i2l', + }); + const denoise = addNode(graph, { + cfg_scale: settings.cfgScale, + denoising_end: 1, + denoising_start: denoisingStart, + height: denoiseSize.height, + id: 'denoise_latents', + steps: settings.steps, + type: 'qwen_image_denoise', + width: denoiseSize.width, + }); + const output = addNode(graph, { + id: 'upscale_output', + is_intermediate: destination === 'canvas', + tile_size: settings.tileSize, + tiled: true, + type: 'qwen_image_l2i', + use_cache: false, + }); + + addPromptConditioning(graph, { + collect: false, + denoise, + encoderFields: [{ field: 'qwen_vl_encoder', source: modelLoader }], + encoderType: 'qwen_image_text_encoder', + negativePrompt: wiresUpscaleNegativePrompt(model, settings.cfgScale) ? negativePrompt : null, + positivePrompt, + }); + addEdge(graph, loraSource, 'transformer', denoise, 'transformer'); + addEdge(graph, seed, 'value', denoise, 'seed'); + + const fitted = fitFrameToDenoiseSize(graph, frame, denoiseSize); + + addEdge(graph, fitted, 'image', imageToLatents, 'image'); + addEdge(graph, modelLoader, 'vae', imageToLatents, 'vae'); + addEdge(graph, imageToLatents, 'latents', denoise, 'latents'); + addEdge(graph, denoise, 'latents', output, 'latents'); + addEdge(graph, modelLoader, 'vae', output, 'vae'); + + return { frame: fitted, output, title: `${model.name} upscale` }; +}; + +/** + * Z-Image's, Anima's, Krea-2's and Qwen Image's text encoders take their encoders on named fields. All but Qwen + * Image's hand the denoise a collected list; its denoise takes the one conditioning. The negative branch exists + * only where the caller decided it is read. */ const addPromptConditioning = ( graph: BackendGraphContract, { + collect = true, denoise, encoderFields, encoderType, negativePrompt, positivePrompt, }: { + collect?: boolean; denoise: BackendInvocationContract; encoderFields: readonly { field: string; source: BackendInvocationContract }[]; encoderType: string; @@ -746,14 +933,20 @@ const addPromptConditioning = ( for (const { conditioningField, prefix, prompt } of branches) { const encoder = addNode(graph, { id: `${prefix}_cond`, type: encoderType }); - const collect = addNode(graph, { id: `${prefix}_cond_collect`, type: 'collect' }); for (const { field, source } of encoderFields) { addEdge(graph, source, field, encoder, field); } addEdge(graph, prompt, 'value', encoder, 'prompt'); - addEdge(graph, encoder, 'conditioning', collect, 'item'); - addEdge(graph, collect, 'collection', denoise, conditioningField); + + if (collect) { + const collector = addNode(graph, { id: `${prefix}_cond_collect`, type: 'collect' }); + + addEdge(graph, encoder, 'conditioning', collector, 'item'); + addEdge(graph, collector, 'collection', denoise, conditioningField); + } else { + addEdge(graph, encoder, 'conditioning', denoise, conditioningField); + } } }; @@ -764,7 +957,9 @@ const addPromptConditioning = ( const UPSCALE_PATH_BUILDERS = { anima: buildAnimaUpscalePath, flux: buildFluxUpscalePath, + flux2: buildFlux2UpscalePath, 'krea-2': buildKrea2UpscalePath, + 'qwen-image': buildQwenImageUpscalePath, 'sd-1': buildSdUpscalePath, sdxl: buildSdUpscalePath, 'z-image': buildZImageUpscalePath, @@ -805,8 +1000,7 @@ export const compileUpscaleGraph = ( const denoisingStart = clipping ? getIndexClippedDenoisingStart( valueClippedStart, - settings.steps, - clipping.shift(model, denoiseSize), + clipping.sigmas(model, denoiseSize, settings.steps), clipping.rounding ) : valueClippedStart; diff --git a/invokeai/frontend/webv2/src/features/upscale/core/indexClippedStarts.test.ts b/invokeai/frontend/webv2/src/features/upscale/core/indexClippedStarts.test.ts index c036cbfc25d..689ba642dd3 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/indexClippedStarts.test.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/indexClippedStarts.test.ts @@ -11,12 +11,13 @@ import { UPSCALE_ARCHITECTURES } from './settings'; const CREATIVITIES = [-10, -5, 0, 5, 8, 10]; const STEP_COUNTS = [8, 30]; -// Z-Image's shift follows the frame up to one megapixel and Krea-2 Raw's up to 1280x1280, held above; -// Anima's and Krea-2 Turbo's are fixed. +// Z-Image's shift follows the frame up to one megapixel, Krea-2 Raw's up to 1280x1280 and Qwen Image's up to +// 8192 tokens (about 1448x1448), each held above; Anima's and Krea-2 Turbo's are fixed. const MODELS: readonly { base: UpscaleBase; variant: string | null; sides: readonly number[] }[] = [ { base: 'anima', sides: [1024], variant: null }, { base: 'krea-2', sides: [1024], variant: 'krea2_turbo' }, { base: 'krea-2', sides: [512, 1024, 2048, 4096], variant: 'krea2_base' }, + { base: 'qwen-image', sides: [512, 1024, 2048, 4096], variant: null }, { base: 'z-image', sides: [512, 1024, 2048, 4096], variant: null }, ]; const BASES = [...new Set(MODELS.map((model) => model.base))]; @@ -38,12 +39,12 @@ describe('index-clipped denoising starts', () => { const valueStart = getUpscaleDenoisingStart(creativity); const clipping = UPSCALE_ARCHITECTURES[base].indexClipping!; const model = { base, variant } as MainModelConfig; - const shift = clipping.shift(model, { height: side, width: side }); + const sigmas = clipping.sigmas(model, { height: side, width: side }, steps); return { base, creativity, - denoisingStart: getIndexClippedDenoisingStart(valueStart, steps, shift, clipping.rounding), + denoisingStart: getIndexClippedDenoisingStart(valueStart, sigmas, clipping.rounding), fluxEntrySigma: 1 - valueStart, side, steps, @@ -57,9 +58,9 @@ describe('index-clipped denoising starts', () => { _comment: 'Generated by src/features/upscale/core/indexClippedStarts.test.ts; regenerate with `vitest -u`. ' + 'Consumed by tests/app/invocations/test_upscale_index_clipped_starts.py, which drives the real ' + - 'z_image_denoise, anima_denoise and krea2_denoise with each denoisingStart and checks the pass ' + - 'enters at the step nearest fluxEntrySigma, and that the schedulers offered are the ones that ' + - 'start there.', + 'z_image_denoise, anima_denoise, krea2_denoise and qwen_image_denoise with each denoisingStart and ' + + 'checks the pass enters at the step nearest fluxEntrySigma, and that the schedulers offered are the ' + + 'ones that start there.', cases, schedulers: Object.fromEntries(BASES.map((base) => [base, UPSCALE_ARCHITECTURES[base].schedulers])), }; diff --git a/invokeai/frontend/webv2/src/features/upscale/core/settings.test.ts b/invokeai/frontend/webv2/src/features/upscale/core/settings.test.ts index 21de7ee21b0..ee67a52626c 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/settings.test.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/settings.test.ts @@ -24,6 +24,7 @@ import { syncUpscaleWidgetValuesWithModels, UPSCALE_ARCHITECTURES, UPSCALE_PRESETS, + upscaleMaxCfgScale, upscaleSchedulerOptions, } from './settings'; @@ -86,6 +87,19 @@ describe('upscale settings', () => { expect(normalizeUpscaleWidgetValues({})?.seedMode).toBe('random'); }); + it('reads values saved before FLUX.2 and Qwen Image with their encoders unset, and checks the type of saved ones', () => { + const mistral = { ...model('m', 'mistral_encoder', 'any'), variant: 'mistral3_24b' }; + + expect(normalizeUpscaleWidgetValues({ t5EncoderModel: null })).toMatchObject({ + mistralEncoderModel: null, + qwenVlEncoderModel: null, + }); + expect(normalizeUpscaleWidgetValues({ mistralEncoderModel: mistral, qwenVlEncoderModel: mistral })).toMatchObject({ + mistralEncoderModel: { key: 'm' }, + qwenVlEncoderModel: null, + }); + }); + it('normalizes partial persisted values and calculates multiple-of-eight output dimensions', () => { expect( normalizeUpscaleWidgetValues({ negativePromptHeightPx: 4, positivePrompt: 'detail', positivePromptHeightPx: 999 }) @@ -222,6 +236,20 @@ describe('architecture-driven validation', () => { expect(getUpscaleValidationReasons(values)).toContain('Upscale cannot use a FLUX Fill model.'); }); + it('refuses a Qwen Image Edit model, which would run every step against a blank reference', () => { + const installed = (variant: string) => [ + { ...model('main', 'main', 'qwen-image'), format: 'gguf_quantized', variant } as ModelConfig, + model('spandrel', 'spandrel_image_to_image', 'any'), + model('qwen-vl', 'qwen_vl_encoder', 'any'), + model('vae', 'vae', 'qwen-image'), + ]; + + expect(getUpscaleValidationReasons(upscaleReady(installed('edit')))).toEqual([ + 'Upscale cannot use a Qwen Image Edit model.', + ]); + expect(getUpscaleValidationReasons(upscaleReady(installed('generate')))).toEqual([]); + }); + it('blocks the run when a required component has been uninstalled', () => { // Without this the Invoke button stays enabled and the graph dies after Spandrel has run. const installed = [ @@ -236,6 +264,28 @@ describe('architecture-driven validation', () => { expect(getUpscaleValidationReasons(values, withoutT5)).toContain('t5 is no longer installed.'); }); + + it("refuses a CFG above the served guidance cap of the model's denoise node, and only there", () => { + // `flux2_denoise` takes guidance up to 20 and refuses more when the graph is queued; `flux_denoise` has no cap. + const dev = upscaleReady([ + { ...model('dev', 'main', 'flux2'), variant: 'dev' } as ModelConfig, + model('spandrel', 'spandrel_image_to_image', 'any'), + { ...model('mistral', 'mistral_encoder', 'any'), variant: 'mistral3_24b' } as ModelConfig, + model('flux2-vae', 'vae', 'flux2'), + ]); + const flux = upscaleReady([ + model('main', 'main', 'flux'), + model('spandrel', 'spandrel_image_to_image', 'any'), + model('t5', 't5_encoder', 'any'), + model('clip', 'clip_embed', 'any'), + model('fluxvae', 'vae', 'flux'), + ]); + + expect([upscaleMaxCfgScale(dev.model), upscaleMaxCfgScale(flux.model)]).toEqual([20, 100]); + expect(getUpscaleValidationReasons({ ...dev, cfgScale: 25 })).toEqual(['CFG scale must be between 1 and 20.']); + expect(getUpscaleValidationReasons({ ...dev, cfgScale: 20 })).toEqual([]); + expect(getUpscaleValidationReasons({ ...flux, cfgScale: 25 })).toEqual([]); + }); }); describe('the architecture table', () => { @@ -256,7 +306,7 @@ const QWEN3_8B = withVariant(model('qwen3-8b', 'qwen3_encoder', 'any'), 'qwen3_8 const QWEN3_06B = withVariant(model('qwen3-06b', 'qwen3_encoder', 'any'), 'qwen3_06b'); const QWEN35 = withVariant(model('qwen35', 'qwen3_5_encoder', 'any'), 'qwen3_5_4b'); -describe('Z-Image and Anima components', () => { +describe('text encoder and VAE components', () => { const ready = (models: ModelConfig[]) => ({ ...createDefaultUpscaleWidgetValues(models), inputImage: { height: 64, image_name: 'in.png', width: 64 }, @@ -366,6 +416,81 @@ describe('Z-Image and Anima components', () => { ).toEqual([]); }); + it('gives each FLUX.2 the encoder its loader takes: Klein the Qwen3 of its size, [dev] a Mistral', () => { + // Each sorts ahead of the encoder that fits, as it can by name in a real install. + const ministral = withVariant(model('ministral', 'mistral_encoder', 'any'), 'ministral3_3b'); + const mistral = withVariant(model('mistral', 'mistral_encoder', 'any'), 'mistral3_24b'); + const shared = [ + model('spandrel', 'spandrel_image_to_image', 'any'), + model('flux2-vae', 'vae', 'flux2'), + QWEN3_8B, + QWEN3_06B, + QWEN3_4B, + ministral, + mistral, + ]; + const pick = (variant: string) => ready([withVariant(model('f', 'main', 'flux2'), variant), ...shared]); + + expect( + ['klein_4b', 'klein_4b_base', 'klein_9b', 'klein_9b_base', 'dev'].map((variant) => [ + pick(variant).qwen3EncoderModel?.key ?? null, + pick(variant).mistralEncoderModel?.key ?? null, + ]) + ).toEqual([ + ['qwen3-4b', null], + ['qwen3-4b', null], + ['qwen3-8b', null], + ['qwen3-8b', null], + [null, 'mistral'], + ]); + expect(getUpscaleValidationReasons(pick('klein_9b'))).toEqual([]); + expect(getUpscaleValidationReasons(pick('dev'))).toEqual([]); + }); + + it('names the FLUX.2 encoder that is missing or of the wrong size', () => { + const base = [model('spandrel', 'spandrel_image_to_image', 'any'), model('flux2-vae', 'vae', 'flux2')]; + const klein9b = withVariant(model('f', 'main', 'flux2'), 'klein_9b'); + const klein4b = ready([withVariant(model('f', 'main', 'flux2'), 'klein_4b'), ...base, QWEN3_4B]); + const dev = withVariant(model('f', 'main', 'flux2'), 'dev'); + + expect(getUpscaleValidationReasons(ready([klein9b, ...base, QWEN3_4B]))).toEqual([ + 'Upscale needs a Qwen3 8B encoder for this main model.', + ]); + expect(getUpscaleValidationReasons({ ...klein4b, qwen3EncoderModel: QWEN3_8B as never })).toContain( + 'qwen3-8b is not a Qwen3 4B encoder this main model can use.' + ); + expect( + getUpscaleValidationReasons( + ready([dev, ...base, withVariant(model('ministral', 'mistral_encoder', 'any'), 'ministral3_3b')]) + ) + ).toEqual(['Upscale needs a Mistral encoder for this main model.']); + }); + + it('lets a Diffusers FLUX.2 bring its own encoder and VAE', () => { + const spandrel = model('spandrel', 'spandrel_image_to_image', 'any'); + const diffusers = (variant: string) => + ({ ...withVariant(model('f', 'main', 'flux2'), variant), format: 'diffusers' }) as ModelConfig; + + expect(getUpscaleValidationReasons(ready([diffusers('klein_9b'), spandrel]))).toEqual([]); + expect(getUpscaleValidationReasons(ready([diffusers('dev'), spandrel]))).toEqual([]); + }); + + it('asks a GGUF Qwen Image for its Qwen VL encoder and a VAE, and a Diffusers one for neither', () => { + const spandrel = model('spandrel', 'spandrel_image_to_image', 'any'); + const gguf = { ...model('q', 'main', 'qwen-image'), format: 'gguf_quantized' } as ModelConfig; + const diffusers = { ...model('q', 'main', 'qwen-image'), format: 'diffusers' } as ModelConfig; + // The Qwen Image VAE is often installed under the Anima base; the served row accepts both. + const values = ready([gguf, spandrel, model('qwen-vl', 'qwen_vl_encoder', 'any'), model('vae', 'vae', 'anima')]); + + expect(getUpscaleValidationReasons(ready([gguf, spandrel]))).toEqual([ + 'Upscale needs a Qwen VL encoder for this main model.', + 'Upscale needs a VAE for this main model.', + ]); + expect([values.qwenVlEncoderModel?.key, values.vae?.key]).toEqual(['qwen-vl', 'vae']); + expect(getUpscaleValidationReasons(values)).toEqual([]); + expect(getUpscaleValidationReasons(ready([diffusers, spandrel]))).toEqual([]); + }); + it('keeps a hand-picked encoder through a switch to an architecture that does not read it', () => { const models = [ model('flux', 'main', 'flux'), @@ -399,6 +524,10 @@ describe('picking a main model', () => { withVariant(model('anima-38', 'main', 'anima'), 'anima_qwen35'), withVariant(model('krea2-turbo', 'main', 'krea-2'), 'krea2_turbo'), withVariant(model('krea2-raw', 'main', 'krea-2'), 'krea2_base'), + withVariant(model('klein', 'main', 'flux2'), 'klein_4b'), + withVariant(model('klein-base', 'main', 'flux2'), 'klein_9b_base'), + withVariant(model('flux2-dev', 'main', 'flux2'), 'dev'), + withVariant(model('qwen', 'main', 'qwen-image'), 'generate'), ]; const cfgAfterPicking = (key: string) => selectUpscaleMainModel( @@ -416,6 +545,41 @@ describe('picking a main model', () => { expect(['krea2-turbo', 'krea2-raw'].map(cfgAfterPicking)).toEqual([1, 5.5]); }); + it('runs FLUX.2 at the guidance it is served with and Qwen Image at its CFG', () => { + // [dev] is distilled on guidance 3.5; Klein reads none, and its served 1 says so. + expect(['klein', 'klein-base', 'flux2-dev', 'qwen'].map(cfgAfterPicking)).toEqual([1, 1, 3.5, 4]); + + const dev = selectUpscaleMainModel( + createDefaultUpscaleWidgetValues(MODELS_BY_BASE), + MODELS_BY_BASE.find((candidate) => candidate.key === 'flux2-dev') as never, + MODELS_BY_BASE + ); + + expect(selectUpscaleMainModel(dev, MODELS_BY_BASE[0] as never, MODELS_BY_BASE).cfgScale).toBe(2); + }); + + const pick = (values: Parameters[0], key: string) => + selectUpscaleMainModel(values, MODELS_BY_BASE.find((candidate) => candidate.key === key) as never, MODELS_BY_BASE); + + it("runs FLUX.2 at the steps it is served with, and leaves every other family at the user's", () => { + // `flux2_denoise` runs every requested step at upscale resolution; the others only those below the entry. + const stepsAfterPicking = (key: string) => + pick({ ...createDefaultUpscaleWidgetValues(MODELS_BY_BASE), steps: 12 }, key).steps; + + expect(['klein', 'klein-base', 'flux2-dev', 'z-turbo', 'krea2-raw', 'flux'].map(stepsAfterPicking)).toEqual([ + 4, 28, 28, 12, 12, 12, + ]); + }); + + it("gives the widget's 30 steps back after FLUX.2, and keeps the steps the user set after the change", () => { + const klein = pick({ ...createDefaultUpscaleWidgetValues(MODELS_BY_BASE), steps: 12 }, 'klein'); + const tuned = { ...klein, steps: 8 }; + + expect(pick(klein, 'sd15').steps).toBe(30); + expect(syncUpscaleWidgetValuesWithModels(tuned, MODELS_BY_BASE).steps).toBe(8); + expect(pick(tuned, 'klein').steps).toBe(8); + }); + it('leaves SD and FLUX at the CFG the user set', () => { expect(['sd15', 'flux'].map(cfgAfterPicking)).toEqual([3, 3]); }); @@ -494,8 +658,11 @@ describe('schedulers per architecture', () => { expect(offered('flux')).toEqual(['euler']); expect(offered('z-image')).toEqual(['euler']); expect(offered('anima')).toEqual(['euler', 'dpmpp_2m', 'dpmpp_2m_sde', 'er_sde']); - // `krea2_denoise` has no scheduler input. + // `krea2_denoise` and `qwen_image_denoise` have no scheduler input. expect(offered('krea-2')).toEqual(['euler']); + expect(offered('qwen-image')).toEqual(['euler']); + // `flux2_denoise` steps with Euler whenever the pass starts above pure noise. + expect(offered('flux2')).toEqual(['euler']); // SD's pass adds noise with the scheduler itself, so every one of its schedulers starts there. expect(offered('sdxl')).toEqual( getSchedulerOptions(model('sdxl', 'main', 'sdxl') as never).map((option) => option.value) diff --git a/invokeai/frontend/webv2/src/features/upscale/core/settings.ts b/invokeai/frontend/webv2/src/features/upscale/core/settings.ts index a2505e81d82..629782ce93a 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/settings.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/settings.ts @@ -13,11 +13,14 @@ import { DEFAULT_NEGATIVE_PROMPT_HEIGHT_PX, DEFAULT_POSITIVE_PROMPT_HEIGHT_PX, getDefaultGenerateSettings, + getGenerationUiPolicy, getSchedulerOptions, hasArchitectureCapabilities, isAnimaQwen35Encoder, isAnimaQwen3Encoder, isBundledMainForBase, + isFlux2MistralEncoder, + isFlux2Qwen3EncoderForModel, isKrea2Qwen3VlEncoder, isLoraCompatibleWithModel, isLoraModelConfig, @@ -35,10 +38,22 @@ import { isSeedMode, SEED_MAX } from '@platform/core/seed'; import type { SpandrelModelConfig, TileControlNetModelConfig, UpscaleWidgetValues } from './types'; -// The schedule shifts of the denoise nodes that clip `denoising_start` by step index, for the table's +// The schedules of the denoise nodes that clip `denoising_start` by step index, for the table's // `indexClipping`; `getIndexClippedDenoisingStart` in graph.ts uses them to start those nodes where // FLUX would. They go away when those nodes clip by value. +/** + * The first `steps` sigmas of a flow schedule shifted by `shift`, `shift * t / (1 + (shift - 1) * t)` at + * `t = 1 - i / steps`, without the final 0. Z-Image, Anima and Krea-2 step exactly this; diffusers' exponential + * time shift at `mu` is the same curve with `shift = exp(mu)`. + */ +const shiftedSigmas = (shift: number, steps: number): number[] => + Array.from({ length: steps }, (_, index) => { + const t = 1 - index / steps; + + return (shift * t) / (1 + (shift - 1) * t); + }); + /** `z_image_denoise._calculate_shift`, held at the fit's 4096-token end since the clamp (#378). */ const Z_IMAGE_SHIFT_MIN_SEQ_LEN = 256; const Z_IMAGE_SHIFT_MAX_SEQ_LEN = 4096; @@ -83,19 +98,54 @@ const krea2ScheduleShift = (model: MainModelConfig, { width, height }: { width: return Math.exp(seqLen * slope + KREA2_BASE_SHIFT - slope * KREA2_SHIFT_MIN_SEQ_LEN); }; +/** + * `qwen_image_denoise`'s stock scheduler: mu on a line from 256 tokens (0.5) to 8192 (0.9), held at the end since + * #9892, and a schedule stretched to end at sigma 0.02. A Diffusers install samples with its own + * `scheduler/scheduler_config.json`; Qwen's ships these same numbers, and one that did not would move every start + * without the backend contract noticing. + */ +const QWEN_IMAGE_SHIFT_MIN_SEQ_LEN = 256; +const QWEN_IMAGE_SHIFT_MAX_SEQ_LEN = 8192; +const QWEN_IMAGE_BASE_SHIFT = 0.5; +const QWEN_IMAGE_MAX_SHIFT = 0.9; +const QWEN_IMAGE_SHIFT_TERMINAL = 0.02; + +/** + * Qwen Image's sigmas as diffusers' `FlowMatchEulerDiscreteScheduler` builds them for the node: the exponential + * shift at the fit's mu, one token per 16x16 pixels, then `stretch_shift_to_terminal`, which moves every sigma + * toward 0 so the last lands on the terminal rather than on the shifted `1 / steps`. Without the stretch the pass + * would start a step off at some sizes. + */ +const qwenImageSigmas = ({ width, height }: { width: number; height: number }, steps: number): number[] => { + const seqLen = Math.min(Math.floor(width / 16) * Math.floor(height / 16), QWEN_IMAGE_SHIFT_MAX_SEQ_LEN); + const slope = + (QWEN_IMAGE_MAX_SHIFT - QWEN_IMAGE_BASE_SHIFT) / (QWEN_IMAGE_SHIFT_MAX_SEQ_LEN - QWEN_IMAGE_SHIFT_MIN_SEQ_LEN); + const shifted = shiftedSigmas( + Math.exp(seqLen * slope + (QWEN_IMAGE_BASE_SHIFT - slope * QWEN_IMAGE_SHIFT_MIN_SEQ_LEN)), + steps + ); + const scale = (1 - (shifted[steps - 1] ?? 0)) / (1 - QWEN_IMAGE_SHIFT_TERMINAL); + + return shifted.map((sigma) => 1 - (1 - sigma) / scale); +}; + /** A text encoder the widget can be asked for; each is its own persisted field. */ export type UpscaleComponentField = | 't5EncoderModel' | 'clipEmbedModel' | 'qwen3EncoderModel' | 'qwen35EncoderModel' - | 'qwen3VlEncoderModel'; + | 'qwen3VlEncoderModel' + | 'qwenVlEncoderModel' + | 'mistralEncoderModel'; const UPSCALE_COMPONENT_TYPES: Record = { clipEmbedModel: 'clip_embed', + mistralEncoderModel: 'mistral_encoder', qwen35EncoderModel: 'qwen3_5_encoder', qwen3EncoderModel: 'qwen3_encoder', qwen3VlEncoderModel: 'qwen3_vl_encoder', + qwenVlEncoderModel: 'qwen_vl_encoder', t5EncoderModel: 't5_encoder', }; @@ -146,6 +196,32 @@ const KREA2_QWEN3_VL_SLOT: UpscaleComponentSlot = { field: 'qwen3VlEncoderModel', label: 'Qwen3-VL 4B encoder', }; +// FLUX.2 Klein reads the Qwen3 its size was trained on, and `flux2_klein_model_loader` refuses the other one. +// Which fits is Generate's filter; this only names it. A Klein whose size was not recorded takes either. +const FLUX2_KLEIN_QWEN3_LABELS: Partial> = { + klein_4b: 'Qwen3 4B encoder', + klein_4b_base: 'Qwen3 4B encoder', + klein_9b: 'Qwen3 8B encoder', + klein_9b_base: 'Qwen3 8B encoder', +}; +const flux2KleinQwen3Slot = (model: MainModelConfig): UpscaleComponentSlot => ({ + accepts: isFlux2Qwen3EncoderForModel(model), + field: 'qwen3EncoderModel', + label: (typeof model.variant === 'string' && FLUX2_KLEIN_QWEN3_LABELS[model.variant]) || 'Qwen3 encoder', +}); +// FLUX.2 [dev] reads Mistral Small 3.1. ERNIE-Image's Ministral 3B is the same model type, and [dev] was not +// trained on it. +const FLUX2_MISTRAL_SLOT: UpscaleComponentSlot = { + accepts: isFlux2MistralEncoder, + field: 'mistralEncoderModel', + label: 'Mistral encoder', +}; +// Qwen Image reads Qwen2.5-VL, the only model of its type. +const QWEN_IMAGE_VL_SLOT: UpscaleComponentSlot = { + accepts: (candidate) => candidate.type === 'qwen_vl_encoder', + field: 'qwenVlEncoderModel', + label: 'Qwen VL encoder', +}; /** What a main model's loader must be handed. A model that bundles its own parts needs none of it. */ export interface UpscaleComponents { @@ -176,8 +252,15 @@ const nameIncludes = /** How a denoise node that clips `denoising_start` by step index lands on its schedule. */ export interface UpscaleIndexClipping { - /** The linear schedule shift the node runs this main model at, for the frame it denoises. */ - readonly shift: (model: MainModelConfig, size: { width: number; height: number }) => number; + /** + * The sigmas the node steps this main model through for the frame it denoises at `steps` steps, noisiest + * first and without the final 0, which would leave nothing to denoise. + */ + readonly sigmas: ( + model: MainModelConfig, + size: { width: number; height: number }, + steps: number + ) => readonly number[]; /** * Whether the node keeps `int(start * steps)` or Python's `int(round(start * steps))`, which rounds a * half to even and so cannot be aimed at the middle of a step. @@ -236,11 +319,18 @@ export interface UpscaleArchitecture { /** * Whether a new main model sets CFG to that model's served default. Z-Image Turbo and Krea-2 Turbo are * distilled for CFG 1, and anything above runs a second, unconditional pass per step at upscale - * resolution; Z-Image Base, Anima and Krea-2 Raw want 4 to 6. SD and FLUX keep the user's CFG -- SD's - * multi-diffusion pass is tuned to the widget's 2 -- and get that 2 back when the user comes from one - * of the others. + * resolution; Z-Image Base, Anima, Krea-2 Raw and Qwen Image want 4 to 6, and FLUX.2 [dev] a guidance of + * 3.5 (Klein reads none). SD and FLUX.1 keep the user's CFG -- SD's multi-diffusion pass is tuned to the + * widget's 2 -- and get that 2 back when the user comes from one of the others. */ readonly takesModelGuidance: boolean; + /** + * Whether a new main model sets Steps to that model's served default, as `takesModelGuidance` does for CFG. + * Only FLUX.2: `flux2_denoise` re-spaces the clipped window to `num_steps`, so every step runs at full upscale + * resolution, where the index-clipped nodes and FLUX.1 run only the steps below the entry. The widget's 30 + * would run a Klein distilled for 4 more than seven times over. Leaving such a model restores the 30. + */ + readonly takesModelSteps: boolean; /** Shown in validation messages, so a user reads "FLUX.1" rather than "flux". */ readonly label: string; } @@ -256,6 +346,7 @@ export const UPSCALE_ARCHITECTURES = { negativePrompt: 'always', schedulers: null, takesModelGuidance: false, + takesModelSteps: false, tileControlNet: SD_TILE_CONTROLNET, }, sdxl: { @@ -266,6 +357,7 @@ export const UPSCALE_ARCHITECTURES = { negativePrompt: 'always', schedulers: null, takesModelGuidance: false, + takesModelSteps: false, tileControlNet: SD_TILE_CONTROLNET, }, flux: { @@ -282,6 +374,29 @@ export const UPSCALE_ARCHITECTURES = { // when the backend contract finds Z-Image's Heun starting at the blend, this list is due as well. schedulers: ['euler'], takesModelGuidance: false, + takesModelSteps: false, + tileControlNet: null, + }, + flux2: { + // Both loaders fall back to a bundled main model's own encoder and VAE; [dev]'s counts only a Diffusers + // install as bundled, Klein's a complete SDNQ pipeline too. + components: (model) => + model.variant === 'dev' + ? model.format === 'diffusers' + ? NO_COMPONENTS + : { slots: [FLUX2_MISTRAL_SLOT], vae: true } + : isBundledMainForBase('flux2')(model) + ? NO_COMPONENTS + : { slots: [flux2KleinQwen3Slot(model)], vae: true }, + denoiseGrid: 16, + // `flux2_denoise` clips by sigma value like FLUX.1, then re-spaces the window to the requested steps. + indexClipping: null, + label: 'FLUX.2', + negativePrompt: 'never', + // With a start above zero `flux2_denoise` steps with Euler, whatever scheduler it is handed. + schedulers: ['euler'], + takesModelGuidance: true, + takesModelSteps: true, tileControlNet: null, }, 'z-image': { @@ -289,11 +404,15 @@ export const UPSCALE_ARCHITECTURES = { components: (model) => isBundledMainForBase('z-image')(model) ? NO_COMPONENTS : { slots: [Z_IMAGE_QWEN3_SLOT], vae: true }, denoiseGrid: 16, - indexClipping: { rounding: 'floor', shift: (_model, size) => zImageScheduleShift(size) }, + indexClipping: { + rounding: 'floor', + sigmas: (_model, size, steps) => shiftedSigmas(zImageScheduleShift(size), steps), + }, label: 'Z-Image', negativePrompt: 'cfg-gated', schedulers: ['euler'], takesModelGuidance: true, + takesModelSteps: false, tileControlNet: { matchesName: nameIncludes('tile'), required: false }, }, anima: { @@ -303,11 +422,12 @@ export const UPSCALE_ARCHITECTURES = { vae: true, }), denoiseGrid: 8, - indexClipping: { rounding: 'floor', shift: () => ANIMA_SCHEDULE_SHIFT }, + indexClipping: { rounding: 'floor', sigmas: (_model, _size, steps) => shiftedSigmas(ANIMA_SCHEDULE_SHIFT, steps) }, label: 'Anima', negativePrompt: 'cfg-gated', schedulers: ['euler', 'dpmpp_2m', 'dpmpp_2m_sde', 'er_sde'], takesModelGuidance: true, + takesModelSteps: false, tileControlNet: null, }, 'krea-2': { @@ -315,12 +435,30 @@ export const UPSCALE_ARCHITECTURES = { // single-file or GGUF transformer without both. components: (model) => (model.format === 'diffusers' ? NO_COMPONENTS : { slots: [KREA2_QWEN3_VL_SLOT], vae: true }), denoiseGrid: 16, - indexClipping: { rounding: 'round', shift: krea2ScheduleShift }, + indexClipping: { + rounding: 'round', + sigmas: (model, size, steps) => shiftedSigmas(krea2ScheduleShift(model, size), steps), + }, label: 'Krea-2', negativePrompt: 'cfg-gated', // `krea2_denoise` has no scheduler input: it steps with Euler. schedulers: ['euler'], takesModelGuidance: true, + takesModelSteps: false, + tileControlNet: null, + }, + 'qwen-image': { + // `qwen_image_model_loader` takes a Diffusers install's own encoder and VAE; the widget sends no component + // source, so any other format needs both. + components: (model) => (model.format === 'diffusers' ? NO_COMPONENTS : { slots: [QWEN_IMAGE_VL_SLOT], vae: true }), + denoiseGrid: 16, + indexClipping: { rounding: 'round', sigmas: (_model, size, steps) => qwenImageSigmas(size, steps) }, + label: 'Qwen Image', + negativePrompt: 'cfg-gated', + // `qwen_image_denoise` has no scheduler input: it steps with Euler. + schedulers: ['euler'], + takesModelGuidance: true, + takesModelSteps: false, tileControlNet: null, }, } as const satisfies Record; @@ -367,6 +505,18 @@ export const wiresUpscaleNegativePrompt = (model: MainModelConfig | null, cfgSca export const upscaleMinSteps = (model: MainModelConfig | null): number => upscaleArchitectureFor(model)?.indexClipping ? 2 : 1; +const UPSCALE_CFG_SCALE_MAX = 100; + +/** + * The highest CFG the pass can run at: the widget's 100, or the served cap on the denoise node's guidance, + * which Generate's control reads the same way. `flux2_denoise` refuses guidance above 20 when the graph is queued. + */ +export const upscaleMaxCfgScale = (model: MainModelConfig | null): number => { + const served = model ? getGenerationUiPolicy(model).guidanceMax : null; + + return served === null ? UPSCALE_CFG_SCALE_MAX : Math.min(UPSCALE_CFG_SCALE_MAX, served); +}; + /** The base's schedulers, less those that would ignore Creativity; see `schedulers`. */ export const upscaleSchedulerOptions = (model: MainModelConfig | null) => { const options = getSchedulerOptions(model ?? undefined); @@ -462,9 +612,11 @@ const pickComponents = ( export const activeUpscaleComponents = (values: Pick) => { const active: UpscaleComponentValues = { clipEmbedModel: null, + mistralEncoderModel: null, qwen35EncoderModel: null, qwen3EncoderModel: null, qwen3VlEncoderModel: null, + qwenVlEncoderModel: null, t5EncoderModel: null, }; @@ -553,13 +705,13 @@ const normalizeLoras = (value: unknown): GenerateLora[] => ) : []; -/** What SD and FLUX run at until the user changes it; SD's multi-diffusion pass is tuned to it. */ +/** What SD and FLUX.1 run at until the user changes it; SD's multi-diffusion pass is tuned to it. */ const DEFAULT_UPSCALE_CFG_SCALE = 2; /** * The CFG a change of main model leaves, see `takesModelGuidance`. Leaving a model that took its * served guidance for one that keeps the user's restores the widget's default, so SD's tuned pass - * does not inherit Turbo's 1, nor FLUX Anima-3.8B's 6. + * does not inherit Turbo's 1, nor FLUX.1 Anima-3.8B's 6. * * The default is applied once, at the change, so the change must not happen before the served table * has loaded: the widget is gated on it, and validation refuses to run without it. @@ -576,6 +728,25 @@ const guidanceAfterModelChange = ( return upscaleArchitectureFor(previous)?.takesModelGuidance ? DEFAULT_UPSCALE_CFG_SCALE : current; }; +/** What every architecture but FLUX.2 runs at until the user changes it. */ +const DEFAULT_UPSCALE_STEPS = 30; + +/** + * The steps a change of main model leaves, see `takesModelSteps`: the CFG's rule, applied once at the change + * on the same served table. Leaving FLUX.2 restores the widget's 30, so SD does not inherit Klein's 4. + */ +const stepsAfterModelChange = ( + previous: MainModelConfig | null, + next: MainModelConfig | null, + current: number +): number => { + if (next && upscaleArchitectureFor(next)?.takesModelSteps) { + return getDefaultGenerateSettings(next).steps; + } + + return upscaleArchitectureFor(previous)?.takesModelSteps ? DEFAULT_UPSCALE_STEPS : current; +}; + export const createDefaultUpscaleWidgetValues = (models: readonly ModelConfig[] = []): UpscaleWidgetValues => { const model = (models.find(isSupportedUpscaleMainModel) as MainModelConfig | undefined) ?? null; const components = upscaleComponentsFor(model); @@ -598,7 +769,7 @@ export const createDefaultUpscaleWidgetValues = (models: readonly ModelConfig[] scheduler: 'kdpm_2', seed: 0, seedMode: 'random', - steps: 30, + steps: stepsAfterModelChange(null, model, DEFAULT_UPSCALE_STEPS), structure: 0, tileControlnetModel: pickTileControlNet(null, models, model), tileOverlap: 128, @@ -624,6 +795,9 @@ export const normalizeUpscaleWidgetValues = (value: unknown): UpscaleWidgetValue batchCount: sanitizeBatchCount(value.batchCount), cfgScale: isFiniteNumber(value.cfgScale) ? value.cfgScale : defaults.cfgScale, clipEmbedModel: isComponentModelOfType(value.clipEmbedModel, 'clip_embed') ? value.clipEmbedModel : null, + mistralEncoderModel: isComponentModelOfType(value.mistralEncoderModel, 'mistral_encoder') + ? value.mistralEncoderModel + : null, qwen35EncoderModel: isComponentModelOfType(value.qwen35EncoderModel, 'qwen3_5_encoder') ? value.qwen35EncoderModel : null, @@ -633,6 +807,9 @@ export const normalizeUpscaleWidgetValues = (value: unknown): UpscaleWidgetValue qwen3VlEncoderModel: isComponentModelOfType(value.qwen3VlEncoderModel, 'qwen3_vl_encoder') ? value.qwen3VlEncoderModel : null, + qwenVlEncoderModel: isComponentModelOfType(value.qwenVlEncoderModel, 'qwen_vl_encoder') + ? value.qwenVlEncoderModel + : null, clipSkip: isFiniteNumber(value.clipSkip) ? value.clipSkip : defaults.clipSkip, creativity: isFiniteNumber(value.creativity) ? value.creativity : defaults.creativity, inputImage: @@ -692,9 +869,11 @@ export const syncUpscaleWidgetValuesWithModels = ( ? storedMain : ((models.find(isSupportedUpscaleMainModel) as MainModelConfig | undefined) ?? null); // A fresh widget, or one whose model was uninstalled, gets its model here rather than from a pick, - // and so the CFG a pick would have given it. + // and so the CFG and steps a pick would have given it. const cfgScale = model?.key === values.model?.key ? values.cfgScale : guidanceAfterModelChange(values.model, model, values.cfgScale); + const steps = + model?.key === values.model?.key ? values.steps : stepsAfterModelChange(values.model, model, values.steps); const storedSpandrel = values.upscaleModel ? modelsByKey.get(values.upscaleModel.key) : undefined; const upscaleModel: SpandrelModelConfig | null = isSpandrelModelConfig(storedSpandrel) ? storedSpandrel @@ -731,6 +910,7 @@ export const syncUpscaleWidgetValuesWithModels = ( if ( values.model === model && values.cfgScale === cfgScale && + values.steps === steps && values.upscaleModel === upscaleModel && values.tileControlnetModel === tileControlnetModel && UPSCALE_COMPONENT_FIELDS.every((field) => values[field] === components[field]) && @@ -741,10 +921,10 @@ export const syncUpscaleWidgetValuesWithModels = ( return values; } - return { ...values, ...components, cfgScale, loras, model, tileControlnetModel, upscaleModel, vae }; + return { ...values, ...components, cfgScale, loras, model, steps, tileControlnetModel, upscaleModel, vae }; }; -/** A main model the user picked: everything synced to it, and the CFG it should run at. */ +/** A main model the user picked: everything synced to it, and the CFG and steps it should run at. */ export const selectUpscaleMainModel = ( values: UpscaleWidgetValues, model: MainModelConfig, @@ -754,7 +934,11 @@ export const selectUpscaleMainModel = ( return synced.model?.key === values.model?.key ? synced - : { ...synced, cfgScale: guidanceAfterModelChange(values.model, synced.model, values.cfgScale) }; + : { + ...synced, + cfgScale: guidanceAfterModelChange(values.model, synced.model, values.cfgScale), + steps: stepsAfterModelChange(values.model, synced.model, values.steps), + }; }; const addRangeReason = (reasons: string[], label: string, value: number, min: number, max: number): void => { @@ -792,6 +976,12 @@ export const getUpscaleValidationReasons = (values: UpscaleWidgetValues, models? if (values.model?.base === 'flux' && values.model.variant === 'dev_fill') { reasons.push('Upscale cannot use a FLUX Fill model.'); } + // Without a reference image `qwen_image_denoise` gives an Edit model a zero reference the size of the output, + // which doubles the image tokens of every transformer call, the negative pass's too: about 2.5 to 3 times the + // time, and activation memory the cache does not reserve, for nothing the pass uses. + if (values.model?.base === 'qwen-image' && values.model.variant === 'edit') { + reasons.push('Upscale cannot use a Qwen Image Edit model.'); + } const tileControlNet = architecture?.tileControlNet ?? null; @@ -830,7 +1020,7 @@ export const getUpscaleValidationReasons = (values: UpscaleWidgetValues, models? addRangeReason(reasons, 'Tile size', values.tileSize, UPSCALE_TILE_SIZE_MIN, UPSCALE_TILE_SIZE_MAX); addRangeReason(reasons, 'Tile overlap', values.tileOverlap, UPSCALE_TILE_OVERLAP_MIN, UPSCALE_TILE_OVERLAP_MAX); addRangeReason(reasons, 'Steps', values.steps, upscaleMinSteps(values.model), 1000); - addRangeReason(reasons, 'CFG scale', values.cfgScale, 1, 100); + addRangeReason(reasons, 'CFG scale', values.cfgScale, 1, upscaleMaxCfgScale(values.model)); addRangeReason(reasons, 'Seed', values.seed, 0, SEED_MAX); if (models) { diff --git a/invokeai/frontend/webv2/src/features/upscale/core/types.ts b/invokeai/frontend/webv2/src/features/upscale/core/types.ts index 32796ac88a7..982b2386cfe 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/types.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/types.ts @@ -43,6 +43,8 @@ export interface UpscaleWidgetValues { qwen3EncoderModel: ComponentModelConfig | null; qwen35EncoderModel: ComponentModelConfig | null; qwen3VlEncoderModel: ComponentModelConfig | null; + qwenVlEncoderModel: ComponentModelConfig | null; + mistralEncoderModel: ComponentModelConfig | null; tileControlnetModel: TileControlNetModelConfig | null; tileSize: number; diff --git a/invokeai/frontend/webv2/src/features/upscale/core/upscaleGraphCoverage.test.ts b/invokeai/frontend/webv2/src/features/upscale/core/upscaleGraphCoverage.test.ts index 012649cb4da..bf2f1bcfb7b 100644 --- a/invokeai/frontend/webv2/src/features/upscale/core/upscaleGraphCoverage.test.ts +++ b/invokeai/frontend/webv2/src/features/upscale/core/upscaleGraphCoverage.test.ts @@ -34,10 +34,14 @@ const INSTALLED = [ model('t5', 't5_encoder', 'any'), model('clip', 'clip_embed', 'any', { variant: 'large' }), model('qwen3-4b', 'qwen3_encoder', 'any', { variant: 'qwen3_4b' }), + model('qwen3-8b', 'qwen3_encoder', 'any', { variant: 'qwen3_8b' }), model('qwen3-06b', 'qwen3_encoder', 'any', { variant: 'qwen3_06b' }), model('qwen35', 'qwen3_5_encoder', 'any', { variant: 'qwen3_5_4b' }), model('qwen3vl-4b', 'qwen3_vl_encoder', 'any', { variant: 'qwen3_vl_4b' }), + model('qwen-vl', 'qwen_vl_encoder', 'any'), + model('mistral', 'mistral_encoder', 'any', { variant: 'mistral3_24b' }), model('flux-vae', 'vae', 'flux'), + model('flux2-vae', 'vae', 'flux2'), model('qwen-vae', 'vae', 'qwen-image'), model('sd1-tile', 'controlnet', 'sd-1', { name: 'sd1 tile' }), model('sdxl-tile', 'controlnet', 'sdxl', { name: 'sdxl tile' }), @@ -46,7 +50,8 @@ const INSTALLED = [ /** * One case per path a builder can take: CFG above 1 where the negative branch is CFG-gated, the - * optional Z-Image ControlNet, a bundled Z-Image or Krea-2 install, and Anima-3.8B's second encoder. + * optional Z-Image ControlNet, a bundled Z-Image, Krea-2, FLUX.2 or Qwen Image install, Anima-3.8B's + * second encoder, and FLUX.2 Klein at either encoder size and [dev] with its own loaders. */ const CASES: { label: string; main: ModelConfig; patch?: Partial }[] = [ { label: 'sd-1', main: model('sd1', 'main', 'sd-1') }, @@ -77,6 +82,31 @@ const CASES: { label: string; main: ModelConfig; patch?: Partial { diff --git a/invokeai/frontend/webv2/src/features/upscale/ui/UpscaleWidgetView.tsx b/invokeai/frontend/webv2/src/features/upscale/ui/UpscaleWidgetView.tsx index e86c4d6ac10..2cd9107d9f7 100644 --- a/invokeai/frontend/webv2/src/features/upscale/ui/UpscaleWidgetView.tsx +++ b/invokeai/frontend/webv2/src/features/upscale/ui/UpscaleWidgetView.tsx @@ -71,6 +71,7 @@ import { UPSCALE_TILE_SIZE_MAX, UPSCALE_TILE_SIZE_MIN, upscaleComponentsFor, + upscaleMaxCfgScale, upscaleMinSteps, upscaleSchedulerOptions, } from '@features/upscale/core/settings'; @@ -124,6 +125,13 @@ const COMPONENT_FIELD_UI: Record< placeholder: 'widgets.upscale.selectClipEmbed', required: 'widgets.upscale.clipEmbedRequired', }, + mistralEncoderModel: { + hint: 'mistralEncoder', + label: 'widgets.upscale.mistralEncoder', + modelTypes: ['mistral_encoder'], + placeholder: 'widgets.upscale.selectMistralEncoder', + required: 'widgets.upscale.mistralEncoderRequired', + }, qwen35EncoderModel: { hint: 'qwen35Encoder', label: 'widgets.upscale.qwen35Encoder', @@ -145,6 +153,13 @@ const COMPONENT_FIELD_UI: Record< placeholder: 'widgets.upscale.selectQwen3VlEncoder', required: 'widgets.upscale.qwen3VlEncoderRequired', }, + qwenVlEncoderModel: { + hint: 'qwenVlEncoder', + label: 'widgets.upscale.qwenVlEncoder', + modelTypes: ['qwen_vl_encoder'], + placeholder: 'widgets.upscale.selectQwenVlEncoder', + required: 'widgets.upscale.qwenVlEncoderRequired', + }, t5EncoderModel: { hint: 't5Encoder', label: 'widgets.upscale.t5Encoder', @@ -378,7 +393,7 @@ export const UpscaleWidgetView = () => { ); const errors = useMemo( () => ({ - cfgScale: getRangeError(t('widgets.upscale.cfgScale'), values.cfgScale, 1, 100), + cfgScale: getRangeError(t('widgets.upscale.cfgScale'), values.cfgScale, 1, upscaleMaxCfgScale(values.model)), creativity: getRangeError( t('widgets.upscale.creativity'), values.creativity, @@ -568,6 +583,7 @@ export const UpscaleWidgetView = () => { // graph will really run. Where one is left there is nothing to choose, and the field goes. const schedulerOptions = useMemo(() => upscaleSchedulerOptions(values.model), [values.model]); const scheduler = coerceUpscaleScheduler(values.model, values.scheduler); + const cfgScaleMax = upscaleMaxCfgScale(values.model); const gateLabels = useMemo( () => ({ loadFailed: t('widgets.upscale.capabilitiesLoadFailed'), @@ -719,9 +735,9 @@ export const UpscaleWidgetView = () => { { expect(result.sourceValid).toBe(false); expect(result.validationReasons).toContain( - 'Upscale supports only SD1.5, SDXL, FLUX.1, Z-Image, Anima or Krea-2 main models.' + 'Upscale supports only SD1.5, SDXL, FLUX.1, FLUX.2, Z-Image, Anima, Krea-2 or Qwen Image main models.' ); }); diff --git a/tests/app/invocations/test_frontend_graph_node_types.py b/tests/app/invocations/test_frontend_graph_node_types.py index 0ef9b284972..7f7c0d1e6e5 100644 --- a/tests/app/invocations/test_frontend_graph_node_types.py +++ b/tests/app/invocations/test_frontend_graph_node_types.py @@ -103,15 +103,15 @@ def test_every_connection_the_panel_compiles_is_one_the_queue_will_accept(contra # The number of architectures each panel supports. A contract that silently stopped being # regenerated would make every assertion below vacuous, so the count is pinned: a new architecture # updates this number and the fixture in the same commit. -SUPPORTED_BASE_COUNTS = {"generate": 15, "video": 3, "upscale": 6} +SUPPORTED_BASE_COUNTS = {"generate": 15, "video": 3, "upscale": 8} # Floors separating "the fixture records literals" from "it records a fraction of them". Set just # under the current numbers (generate: 58 node types carrying 252 scalar values; video: 25 carrying -# 142; upscale: 45 carrying 189) rather than far below them: a recording bug that dropped most of the +# 142; upscale: 58 carrying 236) rather than far below them: a recording bug that dropped most of the # values while keeping the names would otherwise leave every annotation check below passing over a # short list. They are not per-architecture expectations, so removing an architecture means lowering # them deliberately. -LITERAL_FLOORS = {"generate": (55, 240), "video": (23, 135), "upscale": (43, 180)} +LITERAL_FLOORS = {"generate": (55, 240), "video": (23, 135), "upscale": (56, 225)} @pytest.mark.parametrize("contract", sorted(CONTRACTS), ids=lambda contract: contract) diff --git a/tests/app/invocations/test_upscale_index_clipped_starts.py b/tests/app/invocations/test_upscale_index_clipped_starts.py index 940137203d5..b768fb0bfeb 100644 --- a/tests/app/invocations/test_upscale_index_clipped_starts.py +++ b/tests/app/invocations/test_upscale_index_clipped_starts.py @@ -1,11 +1,12 @@ -"""The Upscale widget's Creativity translation holds against the real Z-Image, Anima and Krea-2 nodes. +"""The Upscale widget's Creativity translation holds against the real Z-Image, Anima, Krea-2 and Qwen Image nodes. -`flux_denoise` clips `denoising_start` by sigma value; `z_image_denoise`, `anima_denoise` and -`krea2_denoise` clip it by step index on a shifted schedule, so the same number enters them far +`flux_denoise` clips `denoising_start` by sigma value; `z_image_denoise`, `anima_denoise`, `krea2_denoise` +and `qwen_image_denoise` clip it by step index on a shifted schedule, so the same number enters them far noisier. The widget therefore does not send them FLUX's start: `getIndexClippedDenoisingStart` (`features/upscale/core/graph.ts`) picks the step whose sigma is nearest the one FLUX would enter at, -from its own copy of each node's shift and clipping rule (`indexClipping` in `settings.ts`). Krea-2's -shift depends on the variant, which each case records. +from its own copy of each node's schedule and clipping rule (`indexClipping` in `settings.ts`). Krea-2's +shift depends on the variant, which each case records; Qwen Image's schedule is also stretched to end at +its scheduler's terminal sigma. `upscaleIndexClippedStarts.json`, written by `indexClippedStarts.test.ts` (regenerate with `vitest -u`), records what the widget sends. Each case here drives the real `_run_diffusion` -- the real shift, the @@ -17,6 +18,9 @@ The blend only helps if the scheduler then starts where it left the latents. The fixture also records which schedulers the widget offers each node, and that has to be exactly the set that does: one that builds its own schedule would denoise the frame as if it were pure noise, or leave noise in it. + +FLUX.2 gets FLUX's start untranslated, which holds only while `flux2_denoise` enters at exactly that value +and steps it with its own Euler; the last test here pins both. """ import json @@ -30,11 +34,13 @@ from diffusers.schedulers.scheduling_flow_match_euler_discrete import FlowMatchEulerDiscreteScheduler from invokeai.app.invocations.anima.anima_denoise import AnimaDenoiseInvocation +from invokeai.app.invocations.flux2.flux2_denoise import Flux2DenoiseInvocation from invokeai.app.invocations.krea2.krea2_denoise import Krea2DenoiseInvocation +from invokeai.app.invocations.qwen_image.qwen_image_denoise import QwenImageDenoiseInvocation from invokeai.app.invocations.z_image.z_image_denoise import ZImageDenoiseInvocation from invokeai.backend.anima.scheduler_driver import AnimaSchedulerDriver -from invokeai.backend.flux.schedulers import ANIMA_SCHEDULER_MAP, ZIMAGE_SCHEDULER_MAP -from invokeai.backend.model_manager.taxonomy import Krea2VariantType +from invokeai.backend.flux.schedulers import ANIMA_SCHEDULER_MAP, FLUX_SCHEDULER_MAP, ZIMAGE_SCHEDULER_MAP +from invokeai.backend.model_manager.taxonomy import BaseModelType, Flux2VariantType, Krea2VariantType, ModelType FIXTURE = ( Path(__file__).parents[3] @@ -54,6 +60,8 @@ Z_IMAGE_MODULE = "invokeai.app.invocations.z_image.z_image_denoise" ANIMA_MODULE = "invokeai.app.invocations.anima.anima_denoise" KREA2_MODULE = "invokeai.app.invocations.krea2.krea2_denoise" +QWEN_IMAGE_MODULE = "invokeai.app.invocations.qwen_image.qwen_image_denoise" +FLUX2_MODULE = "invokeai.app.invocations.flux2.flux2_denoise" class _Stop(Exception): @@ -303,7 +311,77 @@ def capture(latents, *_args): return schedule["sigmas"], float(blended.flatten()[0]) -ENTRY = {"z-image": z_image_entry, "anima": anima_entry, "krea-2": krea2_entry} +class _TinyQwenImageTransformer: + """Stands in for `QwenImageTransformer2DModel`: the config the node reads before it builds the schedule.""" + + config = SimpleNamespace(patch_size=2, out_channels=16) + + +def qwen_image_entry(case: dict) -> tuple[list[float], float]: + """Run `qwen_image_denoise._run_diffusion` with its real scheduler to the blend, which it packs next. + + The model path has no `scheduler/`, so the node builds the stock scheduler, terminal stretch included; no + shift is sent, so it takes its own held dynamic one for the frame. The blend is caught on its way into + `_pack_latents`, so the tensors can stay small whatever the frame. + """ + node = QwenImageDenoiseInvocation.model_construct( + id="upscale", + latents=MagicMock(latents_name="latents"), + reference_latents=None, + denoise_mask=None, + denoising_start=case["denoisingStart"], + denoising_end=1.0, + transformer=MagicMock(transformer="transformer", loras=[]), + positive_conditioning=SimpleNamespace(conditioning_name="positive"), + negative_conditioning=None, + cfg_scale=1.0, + width=case["side"], + height=case["side"], + steps=case["steps"], + seed=0, + shift=None, + ) + schedule: dict[str, list[float]] = {} + + class _RecordingScheduler(FlowMatchEulerDiscreteScheduler): + def set_timesteps(self, *args, **kwargs): + super().set_timesteps(*args, **kwargs) + schedule["sigmas"] = self.sigmas.tolist() + + captured: dict[str, torch.Tensor] = {} + + def capture(latents, *_args): + captured["latents"] = latents + raise _Stop + + context = MagicMock() + context.tensors.load.return_value = torch.zeros(1, 16, 8, 8) + context.models.load.return_value = SimpleNamespace(model=_TinyQwenImageTransformer()) + context.models.get_absolute_path.return_value = Path("no-such-qwen-image-model") + with ExitStack() as stack: + stack.enter_context(patch(f"{QWEN_IMAGE_MODULE}.QwenImageTransformer2DModel", _TinyQwenImageTransformer)) + stack.enter_context( + patch(f"{QWEN_IMAGE_MODULE}.TorchDevice.choose_torch_device", return_value=torch.device("cpu")) + ) + stack.enter_context( + patch( + "diffusers.schedulers.scheduling_flow_match_euler_discrete.FlowMatchEulerDiscreteScheduler", + _RecordingScheduler, + ) + ) + stack.enter_context(patch.object(node, "_load_text_conditioning", return_value=(torch.zeros(1, 4, 8), None))) + # float32 noise: the node works in bfloat16, which would keep three digits of the entry sigma. + stack.enter_context(patch.object(node, "_get_noise", return_value=torch.ones(1, 16, 8, 8))) + stack.enter_context(patch.object(node, "_pack_latents", capture)) + with pytest.raises(_Stop): + node._run_diffusion(context) + + blended = captured["latents"] + assert torch.allclose(blended, blended.flatten()[:1].expand_as(blended)), "the blend should be uniform" + return schedule["sigmas"], float(blended.flatten()[0]) + + +ENTRY = {"z-image": z_image_entry, "anima": anima_entry, "krea-2": krea2_entry, "qwen-image": qwen_image_entry} SCHEDULER_ENTRY = {"z-image": z_image_scheduler_entry, "anima": anima_scheduler_entry} SCHEDULERS = {"z-image": list(ZIMAGE_SCHEDULER_MAP), "anima": list(ANIMA_SCHEDULER_MAP)} # Under half the narrowest step at any fixture entry (0.011), so no scheduler passes a step off. DPM++ @@ -317,7 +395,9 @@ def test_the_fixture_covers_every_index_clipping_node(): assert {case["base"] for case in CASES} == set(ENTRY) # Krea-2's shift depends on the variant: Turbo's is fixed, Raw's follows the frame and is held. assert {case["variant"] for case in CASES if case["base"] == "krea-2"} == {"krea2_turbo", "krea2_base"} - assert len(CASES) >= 120 + # Qwen Image's shift follows the frame up to 8192 tokens and is held above; the fixture has to reach both. + assert {case["side"] for case in CASES if case["base"] == "qwen-image"} >= {1024, 2048} + assert len(CASES) >= 168 @pytest.mark.parametrize( @@ -339,10 +419,13 @@ def test_the_pass_enters_at_the_step_nearest_where_flux_enters(case: dict) -> No ) -def test_krea2_has_no_scheduler_to_offer() -> None: - # The widget's Euler-only list for Krea-2 stands for the node having no scheduler input at all. - assert "scheduler" not in Krea2DenoiseInvocation.model_fields - assert OFFERED_SCHEDULERS["krea-2"] == ["euler"] +@pytest.mark.parametrize( + ("base", "node"), [("krea-2", Krea2DenoiseInvocation), ("qwen-image", QwenImageDenoiseInvocation)] +) +def test_a_node_without_a_scheduler_is_offered_only_euler(base: str, node: type) -> None: + # The widget's Euler-only list stands for the node having no scheduler input at all. + assert "scheduler" not in node.model_fields + assert OFFERED_SCHEDULERS[base] == ["euler"] @pytest.mark.parametrize("base", sorted(SCHEDULER_ENTRY)) @@ -368,3 +451,104 @@ def test_the_widget_offers_exactly_the_schedulers_that_start_where_the_blend_lef + "). Update `schedulers` in `features/upscale/core/settings.ts`, and FLUX.1's with Z-Image's: " + "`flux/denoise.py` uses the same diffusers schedulers." ) + + +def flux2_entry(denoising_start: float) -> tuple[float, list[str]]: + """Run `flux2_denoise._run_diffusion` with its real schedule past the preblend to where it reserves memory. + + The init latents are zeros and the noise is ones, so the preblend `t_0 * noise + (1 - t_0) * init` is the + entry sigma in every element. Returns it with the schedulers the node built on the way, which by then it has. + """ + node = Flux2DenoiseInvocation.model_construct( + id="upscale", + latents=MagicMock(latents_name="latents"), + noise=None, + denoise_mask=None, + denoising_start=denoising_start, + denoising_end=1.0, + add_noise=True, + transformer=MagicMock(transformer="transformer", loras=[]), + positive_text_conditioning=SimpleNamespace(conditioning_name="positive", mask=None), + negative_text_conditioning=None, + guidance=3.5, + cfg_scale=1.0, + width=1024, + height=1024, + num_steps=30, + # What the widget sends for FLUX.2, whatever is stored. + scheduler="euler", + seed=0, + vae=MagicMock(), + kontext_conditioning=None, + ) + captured: dict[str, torch.Tensor] = {} + built: list[str] = [] + + def capture(_context, blended): + captured["latents"] = blended + return None + + def recording(name, scheduler_class): + def make(**kwargs): + built.append(name) + return scheduler_class(**kwargs) + + return make + + regional = SimpleNamespace( + regional_text_conditioning=SimpleNamespace(txt_embeddings=torch.zeros(1, 4, 8), txt_ids=torch.zeros(1, 4, 4)), + restricted_attn_mask=None, + ) + context = MagicMock() + context.tensors.load.return_value = torch.zeros(1, 32, 8, 8) + context.models.get_config.return_value = SimpleNamespace( + base=BaseModelType.Flux2, type=ModelType.Main, variant=Flux2VariantType.Klein4B + ) + with ExitStack() as stack: + stack.enter_context(patch(f"{FLUX2_MODULE}.TorchDevice.choose_torch_device", return_value=torch.device("cpu"))) + stack.enter_context( + patch(f"{FLUX2_MODULE}.Flux2RegionalPromptingExtension.from_text_conditionings", return_value=regional) + ) + stack.enter_context( + patch.dict(FLUX_SCHEDULER_MAP, {name: recording(name, cls) for name, cls in FLUX_SCHEDULER_MAP.items()}) + ) + # No VAE to read BN statistics from; the node then blends in raw latent space, which is what is read here. + stack.enter_context(patch.object(node, "_get_bn_stats", return_value=None)) + stack.enter_context(patch.object(node, "_load_text_conditioning", return_value=[])) + # float32 noise: the node works in bfloat16, which would keep three digits of the entry sigma. + stack.enter_context(patch.object(node, "_prepare_noise_tensor", return_value=torch.ones(1, 32, 8, 8))) + stack.enter_context(patch.object(node, "_prep_inpaint_mask", capture)) + # The first call after the scheduler is decided. + stack.enter_context(patch.object(node, "_estimate_working_memory", side_effect=_Stop)) + with pytest.raises(_Stop): + node._run_diffusion(context) + + blended = captured["latents"] + assert torch.allclose(blended, blended.flatten()[:1].expand_as(blended)), "the blend should be uniform" + return float(blended.flatten()[0]), built + + +@pytest.mark.parametrize( + ("creativity", "denoising_start", "schedulers_built"), + [ + # `getUpscaleDenoisingStart` for these, as JavaScript computes it; 0.49900000000000005 is also what + # `upscaleGraphNodeTypes.json` records for `flux2_denoise`. + (-10, 0.9980000000000001, []), + (0, 0.49900000000000005, []), + # From pure noise the node builds the scheduler it is handed, which the widget keeps at Euler. + (10, 0, ["euler"]), + ], +) +def test_flux2_enters_exactly_where_the_widget_starts_it_and_steps_it_with_its_own_euler( + creativity: int, denoising_start: float, schedulers_built: list[str] +) -> None: + # FLUX.2 gets FLUX's start untranslated: that holds only while `flux2_denoise` clips the shifted schedule by + # value and enters at exactly that value, and steps the frame with its own Euler instead of a scheduler that + # would build a schedule of its own. + entry, built = flux2_entry(denoising_start) + + assert entry == pytest.approx(1 - denoising_start, abs=1e-6), ( + f"Creativity {creativity} sends denoising_start={denoising_start}, which should enter at sigma " + f"{1 - denoising_start:.6f}; flux2_denoise enters at {entry:.6f}" + ) + assert built == schedulers_built