diff --git a/docs/src/content/docs/Users Guide/03.Models/Local Models/flux-2.mdx b/docs/src/content/docs/Users Guide/03.Models/Local Models/flux-2.mdx index 4c7471c6ed3..3844e5adab5 100644 --- a/docs/src/content/docs/Users Guide/03.Models/Local Models/flux-2.mdx +++ b/docs/src/content/docs/Users Guide/03.Models/Local Models/flux-2.mdx @@ -1,7 +1,7 @@ --- title: FLUX.2 description: Generate and edit images with FLUX.2 Klein (4B and 9B, distilled and Base) and FLUX.2 [dev], including multi-reference image editing. -lastUpdated: 2026-09-30 +lastUpdated: 2026-10-16 sidebar: order: 5 --- @@ -169,3 +169,14 @@ Decode](/users-guide/image-generation/advanced-topics/pid-decode/). **FLUX2 Denoise** also has a **CFG Scale** and a negative conditioning input, which the Generate tab leaves off (CFG 1.0). + +:::note[Tiled VAE decode] +**Latents to Image - FLUX2** and **Image to Latents - FLUX2** have **Tiled** and **Tile Size** fields. +Without them, Invoke still tiles a decode that is too large for the GPU (`auto_tiled_decode`) or that ran +out of memory, and `force_tiled_decode` tiles every decode and encode; see [Tiled VAE decode and +encode](/configuration/optimization/low-vram-mode/#tiled-vae-decode-and-encode). A tiled result is not +pixel-identical to an untiled one: the difference is fine, texture-level noise spread over the whole image, +not seams, so the same seed does not reproduce an untiled image exactly. On a 1024×1024 FLUX.2 Klein image, +a tiled decode differed from the untiled one by 5.6 of 255 per pixel value on average (about 2%), with no +visible seams; at 2048×2048 by 3.3 of 255. +::: diff --git a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx index 12466be76d2..408b8ce1b1e 100644 --- a/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx +++ b/docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx @@ -202,19 +202,19 @@ clear_vram_after_session: true Without tiling, the memory a VAE needs grows with the square of the image size. For FLUX.1, an untiled 3072² encode peaks at about 18.8 GiB on an RTX 4090, and a tiled one at 0.8 GiB, while tiled memory stays roughly flat as the image grows. Tiling costs some speed on smaller images. -Invoke tiles large decodes by itself: when an untiled decode would claim most of the memory the GPU keeps resident, it decodes in tiles instead, for the FLUX.1, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2, Anima and Wan decodes, and for Qwen-Image-2.1's encodes as well. Because a tiled decode is not pixel-identical to an untiled one, this only kicks in for images too large for the card: for Z-Image on a 16GB GPU, for example, from about 1850x1850 on Nvidia and from about 1500x1500 on AMD, whose convolution library needs more memory. On a ROCm build under Windows the comparison uses the video-memory budget Windows grants Invoke, which is lower than the card's total and lower again while another program uses the GPU. Anima tiles a little earlier, because its untiled decode pushes the Anima model itself out of VRAM on 8GB cards. Wan videos are tiled from 832x480 on 8GB cards and from 720p on 16GB cards, where their untiled decodes need about as much memory as the card holds, or more. +Invoke tiles large decodes by itself: when an untiled decode would claim most of the memory the GPU keeps resident, it decodes in tiles instead, for the FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2, Anima and Wan decodes, and for Qwen-Image-2.1's encodes as well. Because a tiled decode is not pixel-identical to an untiled one, this only kicks in for images too large for the card: for Z-Image on a 16GB GPU, for example, from about 1850x1850 on Nvidia and from about 1500x1500 on AMD, whose convolution library needs more memory. On a ROCm build under Windows the comparison uses the video-memory budget Windows grants Invoke, which is lower than the card's total and lower again while another program uses the GPU. Anima tiles a little earlier, because its untiled decode pushes the Anima model itself out of VRAM on 8GB cards. Wan videos are tiled from 832x480 on 8GB cards and from 720p on 16GB cards, where their untiled decodes need about as much memory as the card holds, or more. -To always decode untiled unless you ask for tiling, turn this off. A decode too large for the card then runs out of memory: the FLUX.1 and Z-Image decodes retry in tiles, the others fail, and on Windows the driver may move the decode into system memory and run it very slowly instead. Anima is not affected by this setting: there, tiling a large decode is what keeps the Anima model in VRAM, and turning it off would only make those decodes slower. +To always decode untiled unless you ask for tiling, turn this off. A decode too large for the card then runs out of memory: the FLUX.1, FLUX.2 and Z-Image decodes retry in tiles, the others fail, and on Windows the driver may move the decode into system memory and run it very slowly instead. Anima is not affected by this setting: there, tiling a large decode is what keeps the Anima model in VRAM, and turning it off would only make those decodes slower. ```yaml auto_tiled_decode: false ``` -- **Always tile:** set `force_tiled_decode: true` in `invokeai.yaml`. It covers the VAE decode and encode for SD1.5/SDXL, FLUX.1, Z-Image, Qwen-Image and Qwen-Image-2.1, the Anima encode, the control image of a Z-Image ControlNet, and the Krea-2 style reference. Other model families (for example SD3, FLUX.2, Wan and LTX-2) ignore it. -- **Per node:** the SD, FLUX.1, Z-Image, Qwen-Image and Qwen-Image-2.1 **Latents to Image** and **Image to Latents** nodes, and **Image to Latents - Anima**, have **Tiled** and **Tile Size** fields for workflows. So does the **Z-Image ControlNet** node, for the control image the denoise encodes. The [Upscale panel](/users-guide/image-generation/upscale/) turns tiling on by itself. -- **Automatic retry:** if a FLUX.1, Z-Image or Anima decode, or a FLUX.1 encode, runs out of memory, InvokeAI retries it once with tiling and logs that it did. +- **Always tile:** set `force_tiled_decode: true` in `invokeai.yaml`. It covers the VAE decode and encode for SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1, the Anima encode, the control image of a Z-Image ControlNet, and the Krea-2 style reference. Other model families (for example SD3, Wan and LTX-2) ignore it. +- **Per node:** the SD, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 **Latents to Image** and **Image to Latents** nodes, and **Image to Latents - Anima**, have **Tiled** and **Tile Size** fields for workflows. So does the **Z-Image ControlNet** node, for the control image the denoise encodes. The [Upscale panel](/users-guide/image-generation/upscale/) turns tiling on by itself. +- **Automatic retry:** if a FLUX.1, FLUX.2, Z-Image or Anima decode, or a FLUX.1 or FLUX.2 encode, runs out of memory, InvokeAI retries it once with tiling and logs that it did. -A tiled decode is not pixel-identical to a single-pass one. The VAE looks at the whole image at once, so the difference is spread over the image rather than limited to the tile seams; it is small, but the same seed will not reproduce an untiled result exactly. +A tiled decode is not pixel-identical to a single-pass one. The VAE looks at the whole image at once, so the difference is spread over the image rather than limited to the tile seams; it is small (for a 1024x1024 FLUX.2 Klein image, 5.6 of 255 per pixel value on average, with no visible seams), but the same seed will not reproduce an untiled result exactly. ### PiD decode optimization diff --git a/docs/src/generated/settings.json b/docs/src/generated/settings.json index 53fd70676c7..97c353c671b 100644 --- a/docs/src/generated/settings.json +++ b/docs/src/generated/settings.json @@ -815,7 +815,7 @@ { "category": "GENERATION", "default": false, - "description": "Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.", + "description": "Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.", "env_var": "INVOKEAI_FORCE_TILED_DECODE", "literal_values": [], "name": "force_tiled_decode", @@ -826,7 +826,7 @@ { "category": "GENERATION", "default": true, - "description": "Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.", + "description": "Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.", "env_var": "INVOKEAI_AUTO_TILED_DECODE", "literal_values": [], "name": "auto_tiled_decode", diff --git a/invokeai/app/invocations/vae/flux2_vae_decode.py b/invokeai/app/invocations/vae/flux2_vae_decode.py index f0852f1880f..e6d2404fbe4 100644 --- a/invokeai/app/invocations/vae/flux2_vae_decode.py +++ b/invokeai/app/invocations/vae/flux2_vae_decode.py @@ -21,7 +21,9 @@ from invokeai.app.services.shared.invocation_context import InvocationContext from invokeai.backend.model_manager.load.load_base import LoadedModel from invokeai.backend.util.devices import TorchDevice -from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_flux2 +from invokeai.backend.util.oom import is_oom_error +from invokeai.backend.util.vae_tiling_scope import MIN_TILE_SAMPLE_SIZE, scoped_vae_tiling +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_flux2, should_pretile_vae_decode @invocation( @@ -29,7 +31,7 @@ title="Latents to Image - FLUX2", tags=["latents", "image", "vae", "l2i", "flux2", "klein"], category="latents", - version="1.0.0", + version="1.1.0", classification=Classification.Prototype, ) class Flux2VaeDecodeInvocation(BaseInvocation, WithMetadata, WithBoard): @@ -43,30 +45,79 @@ class Flux2VaeDecodeInvocation(BaseInvocation, WithMetadata, WithBoard): description=FieldDescriptions.vae, input=Input.Connection, ) + tiled: bool = InputField(default=False, description=FieldDescriptions.tiled) + # 0 is the "use the model's default" sentinel shared with the other VAE nodes: the workflow UI + # cannot represent None in a number field. + tile_size: int = InputField( + default=0, + multiple_of=8, + description=f"{FieldDescriptions.vae_tile_size} Values between 1 and " + f"{MIN_TILE_SAMPLE_SIZE} are raised to {MIN_TILE_SAMPLE_SIZE}.", + ) - def _vae_decode(self, vae_info: LoadedModel, latents: torch.Tensor) -> Image.Image: + def _vae_decode(self, context: InvocationContext, vae_info: LoadedModel, latents: torch.Tensor) -> Image.Image: """Decode latents to image using FLUX.2 VAE. Input latents should already be in the correct space after BN denormalization was applied in the denoiser. The VAE expects (B, 32, H, W) format. """ + config = context.config.get() + use_tiling = self.tiled or config.force_tiled_decode + # Decoding at FLUX.2 resolutions costs multiple GB of activations (~4.3GB at 1024x1024), # far above the default working memory the cache would otherwise reserve. Tell it up front so # it offloads enough of the (possibly still resident) transformer to leave room. - estimated_working_memory = estimate_vae_working_memory_flux2( - operation="decode", image_tensor=latents, vae=vae_info.model, device=vae_info.compute_device - ) + def estimate(tiled: bool) -> int: + return estimate_vae_working_memory_flux2( + operation="decode", + image_tensor=latents, + vae=vae_info.model, + tile_size=self.tile_size if tiled else None, + device=vae_info.compute_device, + ) + + estimated_working_memory = estimate(use_tiling) + # Where the untiled decode would claim most of what the device keeps resident, tile before it runs: on Windows + # an allocation that does not fit is paged rather than refused, so the retry below never fires. + if ( + not use_tiling + and config.auto_tiled_decode + and should_pretile_vae_decode(vae_info.compute_device, estimated_working_memory) + ): + use_tiling = True + estimated_working_memory = estimate(use_tiling) with vae_info.model_on_device(working_mem_bytes=estimated_working_memory) as (_, vae): vae_dtype = next(iter(vae.parameters())).dtype # Use the VAE's intended compute device (CUDA/MPS, or CPU if configured cpu_only). Do NOT infer it from # current param residency: partial loading may have temporarily offloaded all weights to RAM, which would # wrongly place the latents (and thus the whole decode) on the CPU (see #9373). - device = vae_info.compute_device - latents = latents.to(device=device, dtype=vae_dtype) - - # Decode using diffusers API - decoded = vae.decode(latents, return_dict=False)[0] + latents = latents.to(device=vae_info.compute_device, dtype=vae_dtype) + + def decode() -> torch.Tensor: + return vae.decode(latents, return_dict=False)[0] + + # The VAE is the cache's shared instance, so its tiling state is set for this decode only and restored. + try: + with scoped_vae_tiling(vae, self.tile_size if use_tiling else None): + decoded = decode() + except RuntimeError as e: + if use_tiling or not is_oom_error(e): + raise + # The working-memory estimate was insufficient on this system. Retry once with + # tiling, which caps the peak allocation regardless of resolution. + context.util.signal_progress("VAE decode ran out of memory, retrying tiled") + context.logger.warning( + "VAE decode ran out of memory; retrying with tiling. The tiled result is not identical to an " + "untiled decode -- the decoder's normalisation and attention are global, so the difference is " + "spread over the image rather than confined to the seams." + ) + # The traceback pins the failed attempt's activations; see the FLUX.1 decode node. + e.__traceback__ = None + TorchDevice.empty_cache() + # 0, not `self.tile_size`: the field may hold a size left over from a run where `tiled` was off. + with scoped_vae_tiling(vae, 0): + decoded = decode() # Convert from [-1, 1] to [0, 1] then to [0, 255] PIL image img = (decoded / 2 + 0.5).clamp(0, 1) @@ -96,7 +147,7 @@ def invoke(self, context: InvocationContext) -> ImageOutput: vae_info = context.models.load(self.vae.vae) context.util.signal_progress("Running VAE") - image = self._vae_decode(vae_info=vae_info, latents=latents) + image = self._vae_decode(context=context, vae_info=vae_info, latents=latents) TorchDevice.empty_cache() image_dto = context.images.save(image=image) diff --git a/invokeai/app/invocations/vae/flux2_vae_encode.py b/invokeai/app/invocations/vae/flux2_vae_encode.py index 926f3a7c043..8e731d888cd 100644 --- a/invokeai/app/invocations/vae/flux2_vae_encode.py +++ b/invokeai/app/invocations/vae/flux2_vae_encode.py @@ -18,6 +18,9 @@ from invokeai.app.services.shared.invocation_context import InvocationContext from invokeai.backend.model_manager.load.load_base import LoadedModel from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor +from invokeai.backend.util.devices import TorchDevice +from invokeai.backend.util.oom import is_oom_error +from invokeai.backend.util.vae_tiling_scope import MIN_TILE_SAMPLE_SIZE, scoped_vae_tiling from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_flux2 @@ -26,7 +29,7 @@ title="Image to Latents - FLUX2", tags=["latents", "image", "vae", "i2l", "flux2", "klein"], category="latents", - version="1.0.0", + version="1.1.0", classification=Classification.Prototype, ) class Flux2VaeEncodeInvocation(BaseInvocation): @@ -39,8 +42,18 @@ class Flux2VaeEncodeInvocation(BaseInvocation): description=FieldDescriptions.vae, input=Input.Connection, ) + tiled: bool = InputField(default=False, description=FieldDescriptions.tiled) + # 0 is the "use the model's default" sentinel shared with the other VAE nodes. + tile_size: int = InputField( + default=0, + multiple_of=8, + description=f"{FieldDescriptions.vae_tile_size} Values between 1 and " + f"{MIN_TILE_SAMPLE_SIZE} are raised to {MIN_TILE_SAMPLE_SIZE}.", + ) - def _vae_encode(self, vae_info: LoadedModel, image_tensor: torch.Tensor) -> torch.Tensor: + def _vae_encode( + self, vae_info: LoadedModel, image_tensor: torch.Tensor, tiled: bool = False, tile_size: int = 0 + ) -> torch.Tensor: """Encode image to latents using FLUX.2 VAE. The VAE encodes to 32-channel latent space. @@ -53,10 +66,12 @@ def _vae_encode(self, vae_info: LoadedModel, image_tensor: torch.Tensor) -> torc # dispatch probe about the wrong device and push the tensors onto a device the weights are # not on (see #9373, and the same note in the decode node). device = vae_info.compute_device + effective_tile_size = tile_size if tiled else None estimated_working_memory = estimate_vae_working_memory_flux2( operation="encode", image_tensor=image_tensor, vae=vae_info.model, + tile_size=effective_tile_size, device=device, ) @@ -64,9 +79,9 @@ def _vae_encode(self, vae_info: LoadedModel, image_tensor: torch.Tensor) -> torc vae_dtype = next(iter(vae.parameters())).dtype image_tensor = image_tensor.to(device=device, dtype=vae_dtype) - # Encode using diffusers API - # The VAE.encode() returns a DiagonalGaussianDistribution-like object - latent_dist = vae.encode(image_tensor, return_dict=False)[0] + # The VAE is the cache's shared instance, so its tiling state is set for this encode only. + with scoped_vae_tiling(vae, effective_tile_size): + latent_dist = vae.encode(image_tensor, return_dict=False)[0] # Sample from the distribution (or use mode for deterministic output) # Using mode() for deterministic encoding @@ -94,7 +109,26 @@ def invoke(self, context: InvocationContext) -> LatentsOutput: image_tensor = einops.rearrange(image_tensor, "c h w -> 1 c h w") context.util.signal_progress("Running VAE Encode") - latents = self._vae_encode(vae_info=vae_info, image_tensor=image_tensor) + use_tiling = self.tiled or context.config.get().force_tiled_decode + try: + latents = self._vae_encode( + vae_info=vae_info, image_tensor=image_tensor, tiled=use_tiling, tile_size=self.tile_size + ) + except RuntimeError as e: + if use_tiling or not is_oom_error(e): + raise + # An encode that does not fit should become a slower encode, not a failed generation. + context.util.signal_progress("VAE encode ran out of memory, retrying tiled") + context.logger.warning( + "VAE encode ran out of memory; retrying with tiling. The tiled result is not identical to an " + "untiled encode -- the encoder's normalisation and attention are global, so the difference is " + "spread over the latent rather than confined to the seams." + ) + # The traceback pins the failed attempt's activations; see the FLUX.1 decode node. + e.__traceback__ = None + TorchDevice.empty_cache() + # 0, not `self.tile_size`: the field may hold a size left over from a run where `tiled` was off. + latents = self._vae_encode(vae_info=vae_info, image_tensor=image_tensor, tiled=True, tile_size=0) latents = latents.to("cpu") name = context.tensors.save(tensor=latents) diff --git a/invokeai/app/services/config/config_default.py b/invokeai/app/services/config/config_default.py index 09820e2ad3b..f0f68078a75 100644 --- a/invokeai/app/services/config/config_default.py +++ b/invokeai/app/services/config/config_default.py @@ -137,8 +137,8 @@ class InvokeAIAppConfig(BaseSettings): attention_type: Attention type.
Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp` attention_slice_size: Slice size, valid when attention_type=="sliced".
Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8` attention_backend: Attention kernel for the attention inside diffusion models. `auto` lets PyTorch choose among its exact SDPA kernels and never selects a quantized one. `sage` uses SageAttention 2 for the models listed in the SageAttention docs: faster, most at high resolutions and for video, but quantized, so images differ slightly from `auto` at the same seed. SageAttention is installed separately and needs an NVIDIA GPU with compute capability 8.0 or newer; masked attention, text encoders and VAEs keep PyTorch SDPA.
Valid values: `auto`, `sage` - force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. - auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. + force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. + auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. pil_compress_level: The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = no compression, 1 = fastest with slightly larger filesize, 9 = slowest with smallest filesize. 1 is typically the best setting. max_queue_size: Maximum number of items in the session queue. session_queue_mode: Session queue mode. Use 'FIFO' for traditional first-in-first-out, or 'round_robin' to serve each user's jobs in turn. In single-user mode, FIFO is always used regardless of this setting.
Valid values: `FIFO`, `round_robin` @@ -272,8 +272,8 @@ class InvokeAIAppConfig(BaseSettings): attention_type: ATTENTION_TYPE = Field(default="auto", description="Attention type.") attention_slice_size: ATTENTION_SLICE_SIZE = Field(default="auto", description='Slice size, valid when attention_type=="sliced".') attention_backend: ATTENTION_BACKEND = Field(default="auto", description="Attention kernel for the attention inside diffusion models. `auto` lets PyTorch choose among its exact SDPA kernels and never selects a quantized one. `sage` uses SageAttention 2 for the models listed in the SageAttention docs: faster, most at high resolutions and for video, but quantized, so images differ slightly from `auto` at the same seed. SageAttention is installed separately and needs an NVIDIA GPU with compute capability 8.0 or newer; masked attention, text encoders and VAEs keep PyTorch SDPA.") - force_tiled_decode: bool = Field(default=False, description="Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.") - auto_tiled_decode: bool = Field(default=True, description="Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.") + force_tiled_decode: bool = Field(default=False, description="Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.") + auto_tiled_decode: bool = Field(default=True, description="Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.") pil_compress_level: int = Field(default=1, description="The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = no compression, 1 = fastest with slightly larger filesize, 9 = slowest with smallest filesize. 1 is typically the best setting.") max_queue_size: int = Field(default=10000, gt=0, description="Maximum number of items in the session queue.") session_queue_mode: SESSION_QUEUE_MODE = Field(default="round_robin", description="Session queue mode. Use 'FIFO' for strict first-in-first-out, or 'round_robin' to serve each user's jobs in turn. In round-robin mode, priority orders each user's own jobs, but the user rotation takes precedence: one user's high-priority job does not preempt another user's turn. In single-user mode, jobs are served in submission order either way — except that on multi-GPU systems the default 'round_robin' allows same-priority jobs to be reordered slightly so a freed GPU prefers jobs whose models it already has loaded. Set 'FIFO' to disable that reordering and enforce strict submission order.") diff --git a/invokeai/backend/util/vae_working_memory.py b/invokeai/backend/util/vae_working_memory.py index 4c92894f1ee..23673a6a275 100644 --- a/invokeai/backend/util/vae_working_memory.py +++ b/invokeai/backend/util/vae_working_memory.py @@ -399,9 +399,13 @@ def estimate_vae_working_memory_flux2( diffusers' attention dispatcher -- ``AttnProcessor2_0`` calls ``F.scaled_dot_product_attention`` itself -- so torch's own answer is the whole answer here.) - When tiling is enabled the peak is bounded by a single tile instead of the full image (measured - ~0.55GB flat at a 512px tile, from 1024px up to the 2024px reference-image cap), and the score - matrix, if one is materialized at all, is bounded by the tile too. + `tile_size` is in output pixels; None prices a single pass, and `tile_size <= 0` is the nodes' + "use the default" sentinel (see `resolve_tile_size`). A tiled peak is bounded by the largest tile + instead of the full image (measured ~0.55GB flat at a 512px tile, from 1024px up to the 2024px + reference-image cap), and so is the score matrix, if one is materialized at all. That tile is + clamped to the image per axis, as the slices `AutoencoderKLFlux2` takes are, and a tile covering + the image on both axes is priced as the single pass it runs: the VAE tiles only when a side is + strictly larger than the tile. Both terms are per sample. `vae.decode` takes whatever batch the latents carry, and a ``LatentsField`` is not pinned to one, so the batch has to multiply through: measured at 1024px @@ -417,16 +421,22 @@ def estimate_vae_working_memory_flux2( scaling_constant = _flux_vae_scaling_constant(operation, device) batch_size = image_tensor.shape[0] if image_tensor.dim() >= 4 else 1 - if tile_size is not None: + latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 + out_h = latent_scale_factor_for_operation * image_tensor.shape[-2] + out_w = latent_scale_factor_for_operation * image_tensor.shape[-1] + + # Resolved against the module-level default, never read back from the cached VAE, whose tile + # attributes are whatever the previous invocation left there. + tile = resolve_tile_size(tile_size) if tile_size is not None else None + if tile is not None and (tile < out_h or tile < out_w): + tile_h = min(tile, out_h) + tile_w = min(tile, out_w) # Add 25% for tile overlap and the blending buffers, mirroring the SD1/SDXL estimate. - working_memory = tile_size * tile_size * element_size * scaling_constant * 1.25 - mid_block_seq_len = (tile_size // _FLUX2_VAE_SPATIAL_COMPRESSION) ** 2 + working_memory = tile_h * tile_w * element_size * scaling_constant * 1.25 else: - latent_scale_factor_for_operation = LATENT_SCALE_FACTOR if operation == "decode" else 1 - out_h = latent_scale_factor_for_operation * image_tensor.shape[-2] - out_w = latent_scale_factor_for_operation * image_tensor.shape[-1] + tile_h, tile_w = out_h, out_w working_memory = out_h * out_w * element_size * scaling_constant - mid_block_seq_len = (out_h // _FLUX2_VAE_SPATIAL_COMPRESSION) * (out_w // _FLUX2_VAE_SPATIAL_COMPRESSION) + mid_block_seq_len = (tile_h // _FLUX2_VAE_SPATIAL_COMPRESSION) * (tile_w // _FLUX2_VAE_SPATIAL_COMPRESSION) working_memory *= batch_size score_matrix_bytes = sdpa_score_matrix_bytes( diff --git a/invokeai/frontend/api/openapi.json b/invokeai/frontend/api/openapi.json index d1eb0565dbf..683424ebbfb 100644 --- a/invokeai/frontend/api/openapi.json +++ b/invokeai/frontend/api/openapi.json @@ -37598,6 +37598,27 @@ "input": "connection", "orig_required": true }, + "tiled": { + "default": false, + "description": "Processing using overlapping tiles (reduce memory consumption)", + "field_kind": "input", + "input": "any", + "orig_default": false, + "orig_required": false, + "title": "Tiled", + "type": "boolean" + }, + "tile_size": { + "default": 0, + "description": "The tile size for VAE tiling in pixels (image space). If set to 0, the default tile size for the model will be used. Larger tile sizes generally produce better results at the cost of higher memory usage. Values between 1 and 128 are raised to 128.", + "field_kind": "input", + "input": "any", + "multipleOf": 8, + "orig_default": 0, + "orig_required": false, + "title": "Tile Size", + "type": "integer" + }, "type": { "const": "flux2_vae_decode", "default": "flux2_vae_decode", @@ -37610,7 +37631,7 @@ "tags": ["latents", "image", "vae", "l2i", "flux2", "klein"], "title": "Latents to Image - FLUX2", "type": "object", - "version": "1.0.0", + "version": "1.1.0", "output": { "$ref": "#/components/schemas/ImageOutput" } @@ -37676,6 +37697,27 @@ "input": "connection", "orig_required": true }, + "tiled": { + "default": false, + "description": "Processing using overlapping tiles (reduce memory consumption)", + "field_kind": "input", + "input": "any", + "orig_default": false, + "orig_required": false, + "title": "Tiled", + "type": "boolean" + }, + "tile_size": { + "default": 0, + "description": "The tile size for VAE tiling in pixels (image space). If set to 0, the default tile size for the model will be used. Larger tile sizes generally produce better results at the cost of higher memory usage. Values between 1 and 128 are raised to 128.", + "field_kind": "input", + "input": "any", + "multipleOf": 8, + "orig_default": 0, + "orig_required": false, + "title": "Tile Size", + "type": "integer" + }, "type": { "const": "flux2_vae_encode", "default": "flux2_vae_encode", @@ -37688,7 +37730,7 @@ "tags": ["latents", "image", "vae", "i2l", "flux2", "klein"], "title": "Image to Latents - FLUX2", "type": "object", - "version": "1.0.0", + "version": "1.1.0", "output": { "$ref": "#/components/schemas/LatentsOutput" } @@ -60299,13 +60341,13 @@ "force_tiled_decode": { "type": "boolean", "title": "Force Tiled Decode", - "description": "Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.", + "description": "Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.", "default": false }, "auto_tiled_decode": { "type": "boolean", "title": "Auto Tiled Decode", - "description": "Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.", + "description": "Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.", "default": true }, "pil_compress_level": { @@ -60626,7 +60668,7 @@ "additionalProperties": false, "type": "object", "title": "InvokeAIAppConfig", - "description": "Invoke's global app configuration.\n\nTypically, you won't need to interact with this class directly. Instead, use the `get_config` function from `invokeai.app.services.config` to get a singleton config object.\n\nAttributes:\n host: IP address to bind to. Use `0.0.0.0` to serve to your local network.\n port: Port to bind to.\n allow_origins: Allowed CORS origins.\n allow_credentials: Allow CORS credentials.\n allow_methods: Methods allowed for CORS.\n allow_headers: Headers allowed for CORS.\n ssl_certfile: SSL certificate file for HTTPS. See https://www.uvicorn.dev/settings/#https.\n ssl_keyfile: SSL key file for HTTPS. See https://www.uvicorn.dev/settings/#https.\n show_donation_link: Show the Donate to InvokeAI link in the frontend menus.\n log_tokenization: Enable logging of parsed prompt tokens.\n patchmatch: Enable patchmatch inpaint code.\n models_dir: Path to the models directory.\n convert_cache_dir: Path to the converted models cache directory (DEPRECATED, but do not delete because it is needed for migration from previous versions).\n download_cache_dir: Path to the directory that contains dynamically downloaded models.\n legacy_conf_dir: Path to directory of legacy checkpoint config files.\n db_dir: Path to InvokeAI databases directory.\n db_url: URL of a MySQL 8.4+ or MariaDB 10.11+ database to use instead of the SQLite database in `db_dir`, e.g. `mariadb+pymysql://invokeai:password@db.example/invokeai`. Needs the `mysql` extra. One InvokeAI process uses a database at a time. Read at startup only.\n db_synchronous: SQLite durability setting. `full`, the default and what InvokeAI has always used, flushes every commit to disk. `normal` acknowledges commits without waiting for that flush - measured at roughly 12x shorter commits on an SSD - and cannot corrupt the database under WAL, which is why it is refused, with a warning, when WAL is unavailable for the database file. What `normal` gives up is the most recent transactions on a power loss or OS crash: a just-written image record or queue status, not the image file itself.
Valid values: `full`, `normal`\n outputs_dir: Path to directory for outputs.\n image_subfolder_strategy: Strategy for organizing images into subfolders. 'flat' stores all images in a single folder. 'date' organizes by YYYY/MM/DD. 'type' organizes by image category. 'hash' uses first 2 characters of UUID for filesystem performance.
Valid values: `flat`, `date`, `type`, `hash`\n custom_nodes_dir: Path to directory for custom nodes.\n style_presets_dir: Path to directory for style presets.\n workflow_thumbnails_dir: Path to directory for workflow thumbnails.\n log_handlers: Log handler. Valid options are \"console\", \"file=\", \"syslog=path|address:host:port\", \"http=\".\n log_format: Log format. Use \"plain\" for text-only, \"color\" for colorized output, \"legacy\" for 2.3-style logging and \"syslog\" for syslog-style.
Valid values: `plain`, `color`, `syslog`, `legacy`\n log_level: Emit logging messages at this level or higher.
Valid values: `debug`, `info`, `warning`, `error`, `critical`\n log_sql: Log SQL queries. `log_level` must be `debug` for this to do anything. Extremely verbose.\n log_level_network: Log level for network-related messages. 'info' and 'debug' are very verbose.
Valid values: `debug`, `info`, `warning`, `error`, `critical`\n use_memory_db: Use in-memory database. Useful for development.\n dev_reload: Automatically reload when Python sources are changed. Does not reload node definitions.\n profile_graphs: Enable graph profiling using `cProfile`.\n profile_prefix: An optional prefix for profile output files.\n profiles_dir: Path to profiles output directory.\n max_cache_ram_gb: The maximum amount of CPU RAM to use for model caching in GB. If unset, the limit will be configured based on the available RAM. In most cases, it is recommended to leave this unset.\n max_cache_vram_gb: The amount of VRAM to use for model caching in GB. If unset, the limit will be configured based on the available VRAM and the device_working_mem_gb. In most cases, it is recommended to leave this unset.\n reserve_vram_gb: The amount of VRAM (in GB) to subtract from the model cache's available memory budget. Defaults to 0.\n log_memory_usage: If True, a memory snapshot will be captured before and after every model cache operation, and the result will be logged (at debug level). There is a time cost to capturing the memory snapshots, so it is recommended to only enable this feature if you are actively inspecting the model cache's behaviour.\n model_cache_keep_alive_min: How long to keep models in cache after last use, in minutes. A value of 0 (the default) means models are kept in cache indefinitely. If no model generations occur within the timeout period, the model cache is cleared using the same logic as the 'Clear Model Cache' button.\n device_working_mem_gb: The amount of working memory to keep available on the compute device (in GB). Has no effect if running on CPU. If you are experiencing OOM errors, try increasing this value.\n enable_partial_loading: Enable partial loading of models. This enables models to run with reduced VRAM requirements (at the cost of slower speed) by streaming the model from RAM to VRAM as its used. In some edge cases, partial loading can cause models to run more slowly if they were previously being fully loaded into VRAM.\n keep_ram_copy_of_weights: Whether to keep a full RAM copy of a model's weights when the model is loaded in VRAM. Keeping a RAM copy increases average RAM usage, but speeds up model switching and LoRA patching (assuming there is sufficient RAM). Set this to False if RAM pressure is consistently high.\n offload_on_architecture_switch: When a session uses main models of another architecture than the previous one on the same GPU, move every model it does not use from VRAM to RAM before it runs. Models both sessions use stay. Switching back reloads the moved models from RAM, which costs a few seconds per switch when both architectures would have fit in VRAM together; turn this off if you alternate between such models.\n clear_vram_after_session: After every queue item (each image of a batch), move the models held in VRAM to RAM and return the freed memory to the GPU driver. The models stay cached in RAM; the next item reloads them from there, which costs a few seconds (more with keep_ram_copy_of_weights off). Changes nothing where VRAM is system memory (CPU, MPS, integrated GPUs). Turn this on if a generation that runs fine the first time runs out of memory, or slows down sharply because the GPU spills into shared memory, when it is repeated.\n fp8_compute: Keep ComfyUI 'scaled fp8' checkpoints quantized instead of dequantizing them at load, and run their matmuls on the fp8 tensor cores (requires an Ada/SM 8.9 or newer NVIDIA GPU, or a ROCm GPU that passes an fp8 matmul probe; falls back automatically otherwise). Speeds up denoising and keeps the transformer at fp8 size - FP8 Storage alone also keeps it at that size, without the speedup - but quantizes activations as well, so images will differ from previous versions at the same seed. Reproducibility also requires the model to be FULLY resident in VRAM: a layer whose weights are still in RAM falls back to the dequantized path, and since which layers are resident shifts from run to run, the same seed then yields visibly different images. For repeatable output, ensure the model loads at 100% (e.g. enable_partial_loading=false with enough free VRAM).\n fp8_compute_full_precision_hints: Honor the per-layer 'full_precision_matrix_mult' flags that some scaled-fp8 checkpoints ship. Those layers then dequantize on every forward instead of using the fp8 tensor cores, which can cost a large part of the fp8_compute speedup - on checkpoints that mark many layers, most of it. Set to false to run every quantized layer on the fp8 tensor cores, ignoring the producer's instruction; faster, but the marked layers were flagged as numerically sensitive, so quality may suffer. Only has an effect when fp8_compute is enabled.\n ram: DEPRECATED: This setting is no longer used. It has been replaced by `max_cache_ram_gb`, but most users will not need to use this config since automatic cache size limits should work well in most cases. This config setting will be removed once the new model cache behavior is stable.\n vram: DEPRECATED: This setting is no longer used. It has been replaced by `max_cache_vram_gb`, but most users will not need to use this config since automatic cache size limits should work well in most cases. This config setting will be removed once the new model cache behavior is stable.\n lazy_offload: DEPRECATED: This setting is no longer used. Lazy-offloading is enabled by default. This config setting will be removed once the new model cache behavior is stable.\n pytorch_cuda_alloc_conf: Configure the Torch CUDA memory allocator. This will impact peak reserved VRAM usage and performance. Setting to \"backend:cudaMallocAsync\" works well on many systems. The optimal configuration is highly dependent on the system configuration (device type, VRAM, CUDA driver version, etc.), so must be tuned experimentally. Unset on a ROCm build of PyTorch on Windows, \"expandable_segments:True\" is used.\n device: Preferred execution device. `auto` will choose the device depending on the hardware platform and the installed torch capabilities.
Valid values: `auto`, `cpu`, `cuda`, `mps`, `xpu`, `cuda:N`, `xpu:N` (where N is a device number)\n precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.
Valid values: `auto`, `float16`, `bfloat16`, `float32`\n rocm_aotriton_experimental: Use AOTriton's fused (flash and memory-efficient) attention kernels on AMD GPUs that PyTorch marks experimental for them, such as the RX 7600/7700/7800 series, Ryzen AI 300/Max and the RX 9060 series, by setting TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL. Without them, attention on those GPUs runs on the slower math kernel, which also needs far more memory. `auto` turns them on only with a ROCm 10 build of PyTorch, and only when every GPU used for generation is one they were measured correct on (so far gfx1200, the RX 9060 series); `on` and `off` decide for any build and GPU. A TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL value already set in the environment takes precedence. Has no effect on other GPUs.
Valid values: `auto`, `on`, `off`\n sequential_guidance: Whether to calculate guidance in serial instead of in parallel, lowering memory requirements.\n noise_dtype: The dtype seeded noise is drawn in for SD1.5/SDXL, SD3, FLUX.1, FLUX.2, CogView4 and the Z-Image seed variance enhancer. `float32` draws the same noise on every platform. `float16` is the half-precision draw of earlier versions: on macOS it keeps the images your seeds gave before this update; on Windows and Linux both values give (nearly) the same images, and images made before the update cannot be reproduced there.
Valid values: `float32`, `float16`\n wan_memory_optimization: Enable experimental Wan memory optimizations at the cost of slower generation.\n pid_memory_optimization: Enable experimental PiD decode memory optimizations. Roughly halves the peak activation memory of a PiD decode; in exchange the decoded image changes slightly, because neither the chunked pixel pathway nor the float32 sampler intermediates are bit-exact with the default path.\n attention_type: Attention type.
Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp`\n attention_slice_size: Slice size, valid when attention_type==\"sliced\".
Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`\n attention_backend: Attention kernel for the attention inside diffusion models. `auto` lets PyTorch choose among its exact SDPA kernels and never selects a quantized one. `sage` uses SageAttention 2 for the models listed in the SageAttention docs: faster, most at high resolutions and for video, but quantized, so images differ slightly from `auto` at the same seed. SageAttention is installed separately and needs an NVIDIA GPU with compute capability 8.0 or newer; masked attention, text encoders and VAEs keep PyTorch SDPA.
Valid values: `auto`, `sage`\n force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.\n auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.\n pil_compress_level: The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = no compression, 1 = fastest with slightly larger filesize, 9 = slowest with smallest filesize. 1 is typically the best setting.\n max_queue_size: Maximum number of items in the session queue.\n session_queue_mode: Session queue mode. Use 'FIFO' for traditional first-in-first-out, or 'round_robin' to serve each user's jobs in turn. In single-user mode, FIFO is always used regardless of this setting.
Valid values: `FIFO`, `round_robin`\n clear_queue_on_startup: Empties session queue on startup. If true, disables `max_queue_history`.\n max_queue_history: Keep the last N completed, failed, and canceled queue items. Older items are deleted on startup. Set to 0 to prune all terminal items. Ignored if `clear_queue_on_startup` is true.\n allow_nodes: List of nodes to allow. Omit to allow all.\n deny_nodes: List of nodes to deny. Omit to deny none.\n node_cache_size: How many cached nodes to keep in memory.\n hashing_algorithm: Model hashing algorthim for model installs. 'blake3_multi' is best for SSDs. 'blake3_single' is best for spinning disk HDDs. 'random' disables hashing, instead assigning a UUID to models. Useful when using a memory db to reduce model installation time, or if you don't care about storing stable hashes for models. Alternatively, any other hashlib algorithm is accepted, though these are not nearly as performant as blake3.
Valid values: `blake3_multi`, `blake3_single`, `random`, `md5`, `sha1`, `sha224`, `sha256`, `sha384`, `sha512`, `blake2b`, `blake2s`, `sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`, `shake_128`, `shake_256`\n remote_api_tokens: List of regular expression and token pairs used when downloading models from URLs. The download URL is tested against the regex, and if it matches, the token is provided in as a Bearer token.\n scan_models_on_startup: Scan the models directory on startup, registering orphaned models. This is typically only used in conjunction with `use_memory_db` for testing purposes.\n allow_private_download_urls: Allow the download queue to fetch from loopback, link-local and private-network addresses. Disabled by default so that a download URL cannot be used to reach services that are only reachable from the server. Enable this only if you install models from a mirror on your own network.\n download_proxy: Optional HTTP proxy for model downloads. The proxy must enforce the public-address policy because proxy-side DNS cannot be checked by InvokeAI.\n unsafe_disable_picklescan: UNSAFE. Disable the picklescan security check during model installation. Recommended only for development and testing purposes. This will allow arbitrary code execution during model installation, so should never be used in production.\n allow_unknown_models: Allow installation of models that we are unable to identify. If enabled, models will be marked as `unknown` in the database, and will not have any metadata associated with them. If disabled, unknown models will be rejected during installation.\n multiuser: Enable multiuser support. When disabled, the application runs in single-user mode using a default system account with administrator privileges. When enabled, requires user authentication and authorization.\n strict_password_checking: Enforce strict password requirements. When True, passwords must contain uppercase, lowercase, and numbers. When False (default), any password is accepted but its strength (weak/moderate/strong) is reported to the user.\n image_index_enabled: Maintain a semantic embedding index of gallery images and videos, used by the image map and semantic search features.\n image_index_model: Name of the installed CLIP Vision or SigLIP model used to embed gallery images and video thumbnails. Changing the model discards embeddings computed by the previous model.\n image_index_device: Set to `cpu` to compute image embeddings on the CPU with a service-local copy of the model - avoids VRAM use and lets indexing run during generations. Any other value is ignored: embeddings otherwise run on the model cache's device, pausing while generations are in progress.\n image_index_batch_size: Number of gallery items embedded per batch by the image index worker.\n external_alibabacloud_api_key: API key for Alibaba Cloud DashScope image generation.\n external_alibabacloud_base_url: Base URL override for Alibaba Cloud DashScope image generation.\n external_atlascloud_api_key: API key for Atlas Cloud image generation.\n external_atlascloud_base_url: Base URL override for Atlas Cloud image generation.\n external_gemini_api_key: API key for Gemini image generation.\n external_openai_api_key: API key for OpenAI image generation.\n external_gemini_base_url: Base URL override for Gemini image generation.\n external_openai_base_url: Base URL override for OpenAI image generation.\n external_seedream_api_key: API key for Seedream image generation.\n external_seedream_base_url: Base URL override for Seedream image generation.\n base_url: Public base path when running behind a reverse proxy under a sub-path, e.g. `/invoke`. Set only when the proxy PRESERVES the sub-path (the backend receives `/invoke/api/...`). Leave unset when the proxy strips the sub-path or when serving at the domain root.\n forwarded_allow_ips: Comma-separated list of IPs (or `*`) allowed to set X-Forwarded-* headers. Set to the reverse proxy's IP. Only used when `base_url` is set.\n http_compression_level: Compression level for gzipped HTTP API responses. 0 disables response compression entirely, 1 is fastest, 9 (the default) is smallest. Compression runs on the event loop and blocks the whole server while it works, and level 9 costs about 5.5x the time of level 1 for 0.4 percentage points of extra compression, so lowering this makes the app noticeably more responsive on large libraries. Set to 0 when a reverse proxy already compresses responses." + "description": "Invoke's global app configuration.\n\nTypically, you won't need to interact with this class directly. Instead, use the `get_config` function from `invokeai.app.services.config` to get a singleton config object.\n\nAttributes:\n host: IP address to bind to. Use `0.0.0.0` to serve to your local network.\n port: Port to bind to.\n allow_origins: Allowed CORS origins.\n allow_credentials: Allow CORS credentials.\n allow_methods: Methods allowed for CORS.\n allow_headers: Headers allowed for CORS.\n ssl_certfile: SSL certificate file for HTTPS. See https://www.uvicorn.dev/settings/#https.\n ssl_keyfile: SSL key file for HTTPS. See https://www.uvicorn.dev/settings/#https.\n show_donation_link: Show the Donate to InvokeAI link in the frontend menus.\n log_tokenization: Enable logging of parsed prompt tokens.\n patchmatch: Enable patchmatch inpaint code.\n models_dir: Path to the models directory.\n convert_cache_dir: Path to the converted models cache directory (DEPRECATED, but do not delete because it is needed for migration from previous versions).\n download_cache_dir: Path to the directory that contains dynamically downloaded models.\n legacy_conf_dir: Path to directory of legacy checkpoint config files.\n db_dir: Path to InvokeAI databases directory.\n db_url: URL of a MySQL 8.4+ or MariaDB 10.11+ database to use instead of the SQLite database in `db_dir`, e.g. `mariadb+pymysql://invokeai:password@db.example/invokeai`. Needs the `mysql` extra. One InvokeAI process uses a database at a time. Read at startup only.\n db_synchronous: SQLite durability setting. `full`, the default and what InvokeAI has always used, flushes every commit to disk. `normal` acknowledges commits without waiting for that flush - measured at roughly 12x shorter commits on an SSD - and cannot corrupt the database under WAL, which is why it is refused, with a warning, when WAL is unavailable for the database file. What `normal` gives up is the most recent transactions on a power loss or OS crash: a just-written image record or queue status, not the image file itself.
Valid values: `full`, `normal`\n outputs_dir: Path to directory for outputs.\n image_subfolder_strategy: Strategy for organizing images into subfolders. 'flat' stores all images in a single folder. 'date' organizes by YYYY/MM/DD. 'type' organizes by image category. 'hash' uses first 2 characters of UUID for filesystem performance.
Valid values: `flat`, `date`, `type`, `hash`\n custom_nodes_dir: Path to directory for custom nodes.\n style_presets_dir: Path to directory for style presets.\n workflow_thumbnails_dir: Path to directory for workflow thumbnails.\n log_handlers: Log handler. Valid options are \"console\", \"file=\", \"syslog=path|address:host:port\", \"http=\".\n log_format: Log format. Use \"plain\" for text-only, \"color\" for colorized output, \"legacy\" for 2.3-style logging and \"syslog\" for syslog-style.
Valid values: `plain`, `color`, `syslog`, `legacy`\n log_level: Emit logging messages at this level or higher.
Valid values: `debug`, `info`, `warning`, `error`, `critical`\n log_sql: Log SQL queries. `log_level` must be `debug` for this to do anything. Extremely verbose.\n log_level_network: Log level for network-related messages. 'info' and 'debug' are very verbose.
Valid values: `debug`, `info`, `warning`, `error`, `critical`\n use_memory_db: Use in-memory database. Useful for development.\n dev_reload: Automatically reload when Python sources are changed. Does not reload node definitions.\n profile_graphs: Enable graph profiling using `cProfile`.\n profile_prefix: An optional prefix for profile output files.\n profiles_dir: Path to profiles output directory.\n max_cache_ram_gb: The maximum amount of CPU RAM to use for model caching in GB. If unset, the limit will be configured based on the available RAM. In most cases, it is recommended to leave this unset.\n max_cache_vram_gb: The amount of VRAM to use for model caching in GB. If unset, the limit will be configured based on the available VRAM and the device_working_mem_gb. In most cases, it is recommended to leave this unset.\n reserve_vram_gb: The amount of VRAM (in GB) to subtract from the model cache's available memory budget. Defaults to 0.\n log_memory_usage: If True, a memory snapshot will be captured before and after every model cache operation, and the result will be logged (at debug level). There is a time cost to capturing the memory snapshots, so it is recommended to only enable this feature if you are actively inspecting the model cache's behaviour.\n model_cache_keep_alive_min: How long to keep models in cache after last use, in minutes. A value of 0 (the default) means models are kept in cache indefinitely. If no model generations occur within the timeout period, the model cache is cleared using the same logic as the 'Clear Model Cache' button.\n device_working_mem_gb: The amount of working memory to keep available on the compute device (in GB). Has no effect if running on CPU. If you are experiencing OOM errors, try increasing this value.\n enable_partial_loading: Enable partial loading of models. This enables models to run with reduced VRAM requirements (at the cost of slower speed) by streaming the model from RAM to VRAM as its used. In some edge cases, partial loading can cause models to run more slowly if they were previously being fully loaded into VRAM.\n keep_ram_copy_of_weights: Whether to keep a full RAM copy of a model's weights when the model is loaded in VRAM. Keeping a RAM copy increases average RAM usage, but speeds up model switching and LoRA patching (assuming there is sufficient RAM). Set this to False if RAM pressure is consistently high.\n offload_on_architecture_switch: When a session uses main models of another architecture than the previous one on the same GPU, move every model it does not use from VRAM to RAM before it runs. Models both sessions use stay. Switching back reloads the moved models from RAM, which costs a few seconds per switch when both architectures would have fit in VRAM together; turn this off if you alternate between such models.\n clear_vram_after_session: After every queue item (each image of a batch), move the models held in VRAM to RAM and return the freed memory to the GPU driver. The models stay cached in RAM; the next item reloads them from there, which costs a few seconds (more with keep_ram_copy_of_weights off). Changes nothing where VRAM is system memory (CPU, MPS, integrated GPUs). Turn this on if a generation that runs fine the first time runs out of memory, or slows down sharply because the GPU spills into shared memory, when it is repeated.\n fp8_compute: Keep ComfyUI 'scaled fp8' checkpoints quantized instead of dequantizing them at load, and run their matmuls on the fp8 tensor cores (requires an Ada/SM 8.9 or newer NVIDIA GPU, or a ROCm GPU that passes an fp8 matmul probe; falls back automatically otherwise). Speeds up denoising and keeps the transformer at fp8 size - FP8 Storage alone also keeps it at that size, without the speedup - but quantizes activations as well, so images will differ from previous versions at the same seed. Reproducibility also requires the model to be FULLY resident in VRAM: a layer whose weights are still in RAM falls back to the dequantized path, and since which layers are resident shifts from run to run, the same seed then yields visibly different images. For repeatable output, ensure the model loads at 100% (e.g. enable_partial_loading=false with enough free VRAM).\n fp8_compute_full_precision_hints: Honor the per-layer 'full_precision_matrix_mult' flags that some scaled-fp8 checkpoints ship. Those layers then dequantize on every forward instead of using the fp8 tensor cores, which can cost a large part of the fp8_compute speedup - on checkpoints that mark many layers, most of it. Set to false to run every quantized layer on the fp8 tensor cores, ignoring the producer's instruction; faster, but the marked layers were flagged as numerically sensitive, so quality may suffer. Only has an effect when fp8_compute is enabled.\n ram: DEPRECATED: This setting is no longer used. It has been replaced by `max_cache_ram_gb`, but most users will not need to use this config since automatic cache size limits should work well in most cases. This config setting will be removed once the new model cache behavior is stable.\n vram: DEPRECATED: This setting is no longer used. It has been replaced by `max_cache_vram_gb`, but most users will not need to use this config since automatic cache size limits should work well in most cases. This config setting will be removed once the new model cache behavior is stable.\n lazy_offload: DEPRECATED: This setting is no longer used. Lazy-offloading is enabled by default. This config setting will be removed once the new model cache behavior is stable.\n pytorch_cuda_alloc_conf: Configure the Torch CUDA memory allocator. This will impact peak reserved VRAM usage and performance. Setting to \"backend:cudaMallocAsync\" works well on many systems. The optimal configuration is highly dependent on the system configuration (device type, VRAM, CUDA driver version, etc.), so must be tuned experimentally. Unset on a ROCm build of PyTorch on Windows, \"expandable_segments:True\" is used.\n device: Preferred execution device. `auto` will choose the device depending on the hardware platform and the installed torch capabilities.
Valid values: `auto`, `cpu`, `cuda`, `mps`, `xpu`, `cuda:N`, `xpu:N` (where N is a device number)\n precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.
Valid values: `auto`, `float16`, `bfloat16`, `float32`\n rocm_aotriton_experimental: Use AOTriton's fused (flash and memory-efficient) attention kernels on AMD GPUs that PyTorch marks experimental for them, such as the RX 7600/7700/7800 series, Ryzen AI 300/Max and the RX 9060 series, by setting TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL. Without them, attention on those GPUs runs on the slower math kernel, which also needs far more memory. `auto` turns them on only with a ROCm 10 build of PyTorch, and only when every GPU used for generation is one they were measured correct on (so far gfx1200, the RX 9060 series); `on` and `off` decide for any build and GPU. A TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL value already set in the environment takes precedence. Has no effect on other GPUs.
Valid values: `auto`, `on`, `off`\n sequential_guidance: Whether to calculate guidance in serial instead of in parallel, lowering memory requirements.\n noise_dtype: The dtype seeded noise is drawn in for SD1.5/SDXL, SD3, FLUX.1, FLUX.2, CogView4 and the Z-Image seed variance enhancer. `float32` draws the same noise on every platform. `float16` is the half-precision draw of earlier versions: on macOS it keeps the images your seeds gave before this update; on Windows and Linux both values give (nearly) the same images, and images made before the update cannot be reproduced there.
Valid values: `float32`, `float16`\n wan_memory_optimization: Enable experimental Wan memory optimizations at the cost of slower generation.\n pid_memory_optimization: Enable experimental PiD decode memory optimizations. Roughly halves the peak activation memory of a PiD decode; in exchange the decoded image changes slightly, because neither the chunked pixel pathway nor the float32 sampler intermediates are bit-exact with the default path.\n attention_type: Attention type.
Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp`\n attention_slice_size: Slice size, valid when attention_type==\"sliced\".
Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`\n attention_backend: Attention kernel for the attention inside diffusion models. `auto` lets PyTorch choose among its exact SDPA kernels and never selects a quantized one. `sage` uses SageAttention 2 for the models listed in the SageAttention docs: faster, most at high resolutions and for video, but quantized, so images differ slightly from `auto` at the same seed. SageAttention is installed separately and needs an NVIDIA GPU with compute capability 8.0 or newer; masked attention, text encoders and VAEs keep PyTorch SDPA.
Valid values: `auto`, `sage`\n force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it.\n auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there.\n pil_compress_level: The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = no compression, 1 = fastest with slightly larger filesize, 9 = slowest with smallest filesize. 1 is typically the best setting.\n max_queue_size: Maximum number of items in the session queue.\n session_queue_mode: Session queue mode. Use 'FIFO' for traditional first-in-first-out, or 'round_robin' to serve each user's jobs in turn. In single-user mode, FIFO is always used regardless of this setting.
Valid values: `FIFO`, `round_robin`\n clear_queue_on_startup: Empties session queue on startup. If true, disables `max_queue_history`.\n max_queue_history: Keep the last N completed, failed, and canceled queue items. Older items are deleted on startup. Set to 0 to prune all terminal items. Ignored if `clear_queue_on_startup` is true.\n allow_nodes: List of nodes to allow. Omit to allow all.\n deny_nodes: List of nodes to deny. Omit to deny none.\n node_cache_size: How many cached nodes to keep in memory.\n hashing_algorithm: Model hashing algorthim for model installs. 'blake3_multi' is best for SSDs. 'blake3_single' is best for spinning disk HDDs. 'random' disables hashing, instead assigning a UUID to models. Useful when using a memory db to reduce model installation time, or if you don't care about storing stable hashes for models. Alternatively, any other hashlib algorithm is accepted, though these are not nearly as performant as blake3.
Valid values: `blake3_multi`, `blake3_single`, `random`, `md5`, `sha1`, `sha224`, `sha256`, `sha384`, `sha512`, `blake2b`, `blake2s`, `sha3_224`, `sha3_256`, `sha3_384`, `sha3_512`, `shake_128`, `shake_256`\n remote_api_tokens: List of regular expression and token pairs used when downloading models from URLs. The download URL is tested against the regex, and if it matches, the token is provided in as a Bearer token.\n scan_models_on_startup: Scan the models directory on startup, registering orphaned models. This is typically only used in conjunction with `use_memory_db` for testing purposes.\n allow_private_download_urls: Allow the download queue to fetch from loopback, link-local and private-network addresses. Disabled by default so that a download URL cannot be used to reach services that are only reachable from the server. Enable this only if you install models from a mirror on your own network.\n download_proxy: Optional HTTP proxy for model downloads. The proxy must enforce the public-address policy because proxy-side DNS cannot be checked by InvokeAI.\n unsafe_disable_picklescan: UNSAFE. Disable the picklescan security check during model installation. Recommended only for development and testing purposes. This will allow arbitrary code execution during model installation, so should never be used in production.\n allow_unknown_models: Allow installation of models that we are unable to identify. If enabled, models will be marked as `unknown` in the database, and will not have any metadata associated with them. If disabled, unknown models will be rejected during installation.\n multiuser: Enable multiuser support. When disabled, the application runs in single-user mode using a default system account with administrator privileges. When enabled, requires user authentication and authorization.\n strict_password_checking: Enforce strict password requirements. When True, passwords must contain uppercase, lowercase, and numbers. When False (default), any password is accepted but its strength (weak/moderate/strong) is reported to the user.\n image_index_enabled: Maintain a semantic embedding index of gallery images and videos, used by the image map and semantic search features.\n image_index_model: Name of the installed CLIP Vision or SigLIP model used to embed gallery images and video thumbnails. Changing the model discards embeddings computed by the previous model.\n image_index_device: Set to `cpu` to compute image embeddings on the CPU with a service-local copy of the model - avoids VRAM use and lets indexing run during generations. Any other value is ignored: embeddings otherwise run on the model cache's device, pausing while generations are in progress.\n image_index_batch_size: Number of gallery items embedded per batch by the image index worker.\n external_alibabacloud_api_key: API key for Alibaba Cloud DashScope image generation.\n external_alibabacloud_base_url: Base URL override for Alibaba Cloud DashScope image generation.\n external_atlascloud_api_key: API key for Atlas Cloud image generation.\n external_atlascloud_base_url: Base URL override for Atlas Cloud image generation.\n external_gemini_api_key: API key for Gemini image generation.\n external_openai_api_key: API key for OpenAI image generation.\n external_gemini_base_url: Base URL override for Gemini image generation.\n external_openai_base_url: Base URL override for OpenAI image generation.\n external_seedream_api_key: API key for Seedream image generation.\n external_seedream_base_url: Base URL override for Seedream image generation.\n base_url: Public base path when running behind a reverse proxy under a sub-path, e.g. `/invoke`. Set only when the proxy PRESERVES the sub-path (the backend receives `/invoke/api/...`). Leave unset when the proxy strips the sub-path or when serving at the domain root.\n forwarded_allow_ips: Comma-separated list of IPs (or `*`) allowed to set X-Forwarded-* headers. Set to the reverse proxy's IP. Only used when `base_url` is set.\n http_compression_level: Compression level for gzipped HTTP API responses. 0 disables response compression entirely, 1 is fastest, 9 (the default) is smallest. Compression runs on the event loop and blocks the whole server while it works, and level 9 costs about 5.5x the time of level 1 for 0.4 percentage points of extra compression, so lowering this makes the app noticeably more responsive on large libraries. Set to 0 when a reverse proxy already compresses responses." }, "InvokeAIAppConfigWithSetFields": { "properties": { diff --git a/invokeai/frontend/api/schema.ts b/invokeai/frontend/api/schema.ts index 8d0fad766f7..5e8cb075413 100644 --- a/invokeai/frontend/api/schema.ts +++ b/invokeai/frontend/api/schema.ts @@ -14667,6 +14667,18 @@ export type components = { * @default null */ vae?: components["schemas"]["VAEField"] | null; + /** + * Tiled + * @description Processing using overlapping tiles (reduce memory consumption) + * @default false + */ + tiled?: boolean; + /** + * Tile Size + * @description The tile size for VAE tiling in pixels (image space). If set to 0, the default tile size for the model will be used. Larger tile sizes generally produce better results at the cost of higher memory usage. Values between 1 and 128 are raised to 128. + * @default 0 + */ + tile_size?: number; /** * type * @default flux2_vae_decode @@ -14706,6 +14718,18 @@ export type components = { * @default null */ vae?: components["schemas"]["VAEField"] | null; + /** + * Tiled + * @description Processing using overlapping tiles (reduce memory consumption) + * @default false + */ + tiled?: boolean; + /** + * Tile Size + * @description The tile size for VAE tiling in pixels (image space). If set to 0, the default tile size for the model will be used. Larger tile sizes generally produce better results at the cost of higher memory usage. Values between 1 and 128 are raised to 128. + * @default 0 + */ + tile_size?: number; /** * type * @default flux2_vae_encode @@ -22877,8 +22901,8 @@ export type components = { * attention_type: Attention type.
Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp` * attention_slice_size: Slice size, valid when attention_type=="sliced".
Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8` * attention_backend: Attention kernel for the attention inside diffusion models. `auto` lets PyTorch choose among its exact SDPA kernels and never selects a quantized one. `sage` uses SageAttention 2 for the models listed in the SageAttention docs: faster, most at high resolutions and for video, but quantized, so images differ slightly from `auto` at the same seed. SageAttention is installed separately and needs an NVIDIA GPU with compute capability 8.0 or newer; masked attention, text encoders and VAEs keep PyTorch SDPA.
Valid values: `auto`, `sage` - * force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. - * auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. + * force_tiled_decode: Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. + * auto_tiled_decode: Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. * pil_compress_level: The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = no compression, 1 = fastest with slightly larger filesize, 9 = slowest with smallest filesize. 1 is typically the best setting. * max_queue_size: Maximum number of items in the session queue. * session_queue_mode: Session queue mode. Use 'FIFO' for traditional first-in-first-out, or 'round_robin' to serve each user's jobs in turn. In single-user mode, FIFO is always used regardless of this setting.
Valid values: `FIFO`, `round_robin` @@ -23355,13 +23379,13 @@ export type components = { attention_backend?: "auto" | "sage"; /** * Force Tiled Decode - * @description Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, Z-Image and Qwen-Image VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. + * @description Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty). A tiled decode is not pixel-identical to a single-pass one: a VAE decoder normalises and attends over the whole image, so the difference is spread across it rather than confined to the tile seams. Also tiles the encode. Read by the SD1.5/SDXL, FLUX.1, FLUX.2, Z-Image, Qwen-Image and Qwen-Image-2.1 VAE nodes, the Anima encode, the Z-Image control encode and the Krea-2 style reference; other model families ignore it. * @default false */ force_tiled_decode?: boolean; /** * Auto Tiled Decode - * @description Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, Z-Image, Qwen-Image, Krea-2 and Wan decodes). Turn off to decode untiled unless tiling is requested; the FLUX.1 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. + * @description Decode large images in tiles when an untiled decode's estimated working memory would take most of the GPU's memory (FLUX.1, FLUX.2, Z-Image, Qwen-Image, Qwen-Image-2.1, Krea-2 and Wan decodes, and Qwen-Image-2.1 encodes). Turn off to decode untiled unless tiling is requested; the FLUX.1, FLUX.2 and Z-Image decodes then still retry tiled after running out of memory. Anima keeps tiling a decode too large for its GPU either way, because tiling is what makes that case fast there. * @default true */ auto_tiled_decode?: boolean; diff --git a/tests/app/invocations/test_flux2_vae_tiling.py b/tests/app/invocations/test_flux2_vae_tiling.py new file mode 100644 index 00000000000..7e1ddd29367 --- /dev/null +++ b/tests/app/invocations/test_flux2_vae_tiling.py @@ -0,0 +1,286 @@ +"""Tiling on the FLUX.2 image-to-latents and latents-to-image nodes, and the OOM fallbacks. + +The VAE is a real, tiny `AutoencoderKLFlux2` on the CPU: four blocks, so the same 8x compression +`scoped_vae_tiling` derives its geometry from, and the genuine `_tiled_encode`/`tiled_decode`. The +assertions are about tiles that were actually encoded and assembled, not about recorded flags. + +**The images are 528x784 on purpose.** Diffusers tiles only when a side is strictly greater than the +tile, so the default 512px tile has to be smaller than the image for a "tiled" cell to take the +tiled path; neither side is a multiple of any tile used here, so the assembly has partial edge tiles. +""" + +from unittest.mock import MagicMock, patch + +import numpy as np +import pytest +import torch +from diffusers.models.autoencoders.autoencoder_kl_flux2 import AutoencoderKLFlux2 + +from invokeai.app.invocations.vae.flux2_vae_decode import Flux2VaeDecodeInvocation +from invokeai.app.invocations.vae.flux2_vae_encode import Flux2VaeEncodeInvocation +from invokeai.backend.util.vae_tiling_scope import DEFAULT_TILE_SAMPLE_MIN_SIZE, MIN_TILE_SAMPLE_SIZE +from invokeai.backend.util.vae_working_memory import estimate_vae_working_memory_flux2 + +ENCODE_MODULE = "invokeai.app.invocations.vae.flux2_vae_encode" +DECODE_MODULE = "invokeai.app.invocations.vae.flux2_vae_decode" +CPU = torch.device("cpu") +IMAGE_SHAPE = (1, 3, 528, 784) +LATENT_SHAPE = (1, 32, 66, 98) + + +def _tiling_state(vae: AutoencoderKLFlux2) -> tuple: + return (vae.use_tiling, vae.tile_sample_min_size, vae.tile_latent_min_size, vae.tile_overlap_factor) + + +@pytest.fixture +def vae() -> AutoencoderKLFlux2: + """~15k parameters. `sample_size=64` makes the VAE's own idle tile geometry differ from every tile the + nodes ask for, so a leaked tile shows up as a changed attribute.""" + with torch.random.fork_rng(): + torch.manual_seed(0) + vae = AutoencoderKLFlux2( + block_out_channels=(4, 4, 4, 4), + layers_per_block=1, + latent_channels=32, + norm_num_groups=2, + sample_size=64, + ).eval() + # Spies on the tiled paths, and a record of the tiling state each top-level call ran under. + vae._tiled_encode = MagicMock(wraps=vae._tiled_encode) + vae.tiled_decode = MagicMock(wraps=vae.tiled_decode) + vae.seen_tiling = [] + for name in ("encode", "decode"): + real = getattr(vae, name) + + def record(*args, _real=real, **kwargs): + vae.seen_tiling.append(_tiling_state(vae)) + return _real(*args, **kwargs) + + setattr(vae, name, MagicMock(side_effect=record)) + return vae + + +def _fail_first_call_with_oom(vae: AutoencoderKLFlux2, name: str) -> None: + passthrough = getattr(vae, name).side_effect + calls = [] + + def fail_then_pass(*args, **kwargs): + calls.append(None) + result = passthrough(*args, **kwargs) + if len(calls) == 1: + raise torch.cuda.OutOfMemoryError("CUDA out of memory") + return result + + getattr(vae, name).side_effect = fail_then_pass + + +def _context( + vae: AutoencoderKLFlux2, *, force_tiled_decode: bool = False, auto_tiled_decode: bool = True, latents=None +): + vae_info = MagicMock() + vae_info.model = vae + vae_info.compute_device = CPU + vae_info.model_on_device.return_value.__enter__.return_value = (None, vae) + + context = MagicMock() + context.models.load.return_value = vae_info + # Explicit: a bare MagicMock attribute reads as truthy and would tile everything. + context.config.get.return_value.force_tiled_decode = force_tiled_decode + context.config.get.return_value.auto_tiled_decode = auto_tiled_decode + context.tensors.save.return_value = "latents" + context.tensors.load.return_value = latents + context.images.save.side_effect = lambda image: MagicMock( + image_name="out.png", width=image.width, height=image.height + ) + return context + + +def _encode(context, image: torch.Tensor, **fields) -> torch.Tensor: + invocation = Flux2VaeEncodeInvocation.model_construct( + image=MagicMock(image_name="in.png"), vae=MagicMock(vae=MagicMock()), **fields + ) + with patch(f"{ENCODE_MODULE}.image_resized_to_grid_as_tensor", return_value=image[0]): + invocation.invoke(context) + return context.tensors.save.call_args.kwargs["tensor"] + + +def _decode(context, **fields) -> np.ndarray: + invocation = Flux2VaeDecodeInvocation.model_construct( + latents=MagicMock(latents_name="latents"), vae=MagicMock(vae=MagicMock()), **fields + ) + invocation.invoke(context) + return np.asarray(context.images.save.call_args.kwargs["image"]) + + +@pytest.fixture +def image() -> torch.Tensor: + return torch.rand(IMAGE_SHAPE, generator=torch.Generator().manual_seed(1)) * 2 - 1 + + +@pytest.fixture +def latents() -> torch.Tensor: + return torch.randn(LATENT_SHAPE, generator=torch.Generator().manual_seed(2)) + + +def _reservations(context) -> list[int]: + vae_info = context.models.load.return_value + return [c.kwargs["working_mem_bytes"] for c in vae_info.model_on_device.call_args_list] + + +def _expected_reservation(operation: str, tensor: torch.Tensor, vae, tile_size: int | None) -> int: + return estimate_vae_working_memory_flux2( + operation=operation, image_tensor=tensor, vae=vae, tile_size=tile_size, device=CPU + ) + + +class TestEncodeTiling: + def test_a_tiled_encode_assembles_the_untiled_latent_shape(self, vae, image): + untiled = _encode(_context(vae), image) + assert vae._tiled_encode.call_count == 0 + + tiled = _encode(_context(vae), image, tiled=True, tile_size=128) + + assert vae._tiled_encode.call_count == 1 + assert tiled.shape == untiled.shape == LATENT_SHAPE + # The encoder's normalisation and attention are global, so a tiled encode cannot match. + assert not torch.allclose(tiled, untiled) + + def test_force_tiled_decode_tiles_the_encode_without_the_field(self, vae, image): + _encode(_context(vae, force_tiled_decode=True), image) + + assert vae._tiled_encode.call_count == 1 + assert vae.seen_tiling[0][1] == DEFAULT_TILE_SAMPLE_MIN_SIZE + + def test_the_shared_vae_keeps_its_tiling_state_after_a_tiled_encode(self, vae, image): + before = _tiling_state(vae) + _encode(_context(vae), image, tiled=True, tile_size=256) + assert _tiling_state(vae) == before + + def test_an_oom_while_already_tiled_is_raised_once_and_the_tiling_state_restored(self, vae, image): + before = _tiling_state(vae) + vae._tiled_encode.side_effect = torch.cuda.OutOfMemoryError("CUDA out of memory") + context = _context(vae) + + with pytest.raises(torch.cuda.OutOfMemoryError): + _encode(context, image, tiled=True) + + assert vae.encode.call_count == 1 + assert _tiling_state(vae) == before + + def test_an_untiled_oom_retries_once_tiled_and_returns_the_tiled_latent(self, vae, image): + directly_tiled = _encode(_context(vae), image, tiled=True) + vae.seen_tiling.clear() + _fail_first_call_with_oom(vae, "encode") + context = _context(vae) + + after_retry = _encode(context, image, tiled=False, tile_size=256) + + # The retry takes the default tile, not the field's leftover 256. + assert [state[0] for state in vae.seen_tiling] == [False, True] + assert vae.seen_tiling[1][1] == DEFAULT_TILE_SAMPLE_MIN_SIZE + assert torch.equal(after_retry, directly_tiled) + assert _reservations(context) == [ + _expected_reservation("encode", image, vae, None), + _expected_reservation("encode", image, vae, DEFAULT_TILE_SAMPLE_MIN_SIZE), + ] + context.util.signal_progress.assert_any_call("VAE encode ran out of memory, retrying tiled") + assert "not identical to an untiled encode" in context.logger.warning.call_args[0][0] + + @pytest.mark.parametrize( + "fields,expected_tile", + [ + ({}, None), + ({"tiled": True}, DEFAULT_TILE_SAMPLE_MIN_SIZE), + ({"tiled": True, "tile_size": 256}, 256), + ({"tiled": True, "tile_size": 64}, MIN_TILE_SAMPLE_SIZE), + ], + ids=["untiled", "default-tile", "explicit-tile", "below-minimum-tile"], + ) + def test_the_reservation_prices_the_encode_that_runs(self, vae, image, fields, expected_tile): + context = _context(vae) + _encode(context, image, **fields) + assert _reservations(context) == [_expected_reservation("encode", image, vae, expected_tile)] + + +class TestDecodeTiling: + def test_a_tiled_decode_assembles_the_untiled_image_size(self, vae, latents): + untiled = _decode(_context(vae, latents=latents)) + assert vae.tiled_decode.call_count == 0 + + tiled = _decode(_context(vae, latents=latents), tiled=True, tile_size=128) + + assert vae.tiled_decode.call_count == 1 + assert tiled.shape == untiled.shape == (IMAGE_SHAPE[2], IMAGE_SHAPE[3], 3) + assert not np.array_equal(tiled, untiled) + + def test_force_tiled_decode_tiles_without_the_field(self, vae, latents): + _decode(_context(vae, force_tiled_decode=True, latents=latents)) + + assert vae.tiled_decode.call_count == 1 + assert vae.seen_tiling[0][1] == DEFAULT_TILE_SAMPLE_MIN_SIZE + + def test_the_shared_vae_keeps_its_tiling_state_after_a_tiled_decode(self, vae, latents): + before = _tiling_state(vae) + _decode(_context(vae, latents=latents), tiled=True, tile_size=256) + assert _tiling_state(vae) == before + + def test_an_oom_while_already_tiled_is_raised_once_and_the_tiling_state_restored(self, vae, latents): + before = _tiling_state(vae) + vae.tiled_decode.side_effect = torch.cuda.OutOfMemoryError("CUDA out of memory") + + with pytest.raises(torch.cuda.OutOfMemoryError): + _decode(_context(vae, latents=latents), tiled=True) + + assert vae.decode.call_count == 1 + assert _tiling_state(vae) == before + + def test_an_untiled_oom_retries_once_tiled_and_returns_the_tiled_image(self, vae, latents): + directly_tiled = _decode(_context(vae, latents=latents), tiled=True) + vae.seen_tiling.clear() + _fail_first_call_with_oom(vae, "decode") + context = _context(vae, latents=latents) + + after_retry = _decode(context, tiled=False, tile_size=256) + + # The retry takes the default tile, not the field's leftover 256. + assert [state[0] for state in vae.seen_tiling] == [False, True] + assert vae.seen_tiling[1][1] == DEFAULT_TILE_SAMPLE_MIN_SIZE + assert np.array_equal(after_retry, directly_tiled) + context.util.signal_progress.assert_any_call("VAE decode ran out of memory, retrying tiled") + assert "not identical to an untiled decode" in context.logger.warning.call_args[0][0] + + @pytest.mark.parametrize( + "fields,force,expected_tile", + [ + ({}, False, None), + ({"tiled": True, "tile_size": 256}, False, 256), + ({"tile_size": 256}, True, 256), + ({}, True, DEFAULT_TILE_SAMPLE_MIN_SIZE), + ({"tiled": True, "tile_size": 64}, False, MIN_TILE_SAMPLE_SIZE), + ], + ids=["untiled", "field", "forced-with-tile-size", "forced-default-tile", "below-minimum-tile"], + ) + def test_the_reservation_prices_the_decode_that_runs(self, vae, latents, fields, force, expected_tile): + context = _context(vae, force_tiled_decode=force, latents=latents) + _decode(context, **fields) + assert _reservations(context) == [_expected_reservation("decode", latents, vae, expected_tile)] + + @pytest.mark.parametrize("auto", [True, False], ids=["auto-tiled-decode-on", "auto-tiled-decode-off"]) + def test_a_decode_too_large_for_its_gpu_is_tiled_up_front_unless_switched_off(self, vae, latents, auto): + """On Windows an allocation that does not fit is paged rather than refused, so the OOM retry never fires + and the decision has to be made from the estimate alone.""" + context = _context(vae, auto_tiled_decode=auto, latents=latents) + with patch(f"{DECODE_MODULE}.should_pretile_vae_decode", return_value=True) as pretile: + _decode(context) + + untiled_estimate = _expected_reservation("decode", latents, vae, None) + if auto: + pretile.assert_called_once_with(CPU, untiled_estimate) + assert vae.tiled_decode.call_count == 1 + assert _reservations(context) == [ + _expected_reservation("decode", latents, vae, DEFAULT_TILE_SAMPLE_MIN_SIZE) + ] + else: + pretile.assert_not_called() + assert vae.tiled_decode.call_count == 0 + assert _reservations(context) == [untiled_estimate] diff --git a/tests/app/invocations/test_flux2_working_memory.py b/tests/app/invocations/test_flux2_working_memory.py index 286b6b44511..935fa8fcf42 100644 --- a/tests/app/invocations/test_flux2_working_memory.py +++ b/tests/app/invocations/test_flux2_working_memory.py @@ -25,8 +25,6 @@ FLUX2_REFERENCE_HIDDEN_SIZE, Flux2DenoiseInvocation, ) -from invokeai.app.invocations.vae.flux2_vae_decode import Flux2VaeDecodeInvocation -from invokeai.app.invocations.vae.flux2_vae_encode import Flux2VaeEncodeInvocation from invokeai.backend.patches.layer_patcher import LayerPatcher from invokeai.backend.util.attention import ( SDPA_MATH_BYTES_PER_SCORE_ELEMENT, @@ -329,6 +327,32 @@ def test_tiled_estimate_is_bounded_by_the_tile_not_the_image(self): ) assert estimates[0] < untiled / 4 + def _decode_estimate(self, height, width, tile_size): + return estimate_vae_working_memory_flux2( + operation="decode", + image_tensor=torch.zeros(1, 32, height // 8, width // 8), + vae=self._mock_bf16_vae(), + tile_size=tile_size, + device=FUSED, + ) + + @pytest.mark.parametrize("tile_size", [1024, 2048, 4096]) + def test_a_tile_covering_the_image_is_priced_as_the_single_pass_it_runs(self, tile_size): + """The VAE tiles only when a side is strictly larger than the tile. Pricing the tile instead + reserved 85.9 GiB for a 1024px decode that peaks at 4.3 GiB, evicting the transformer for nothing.""" + assert self._decode_estimate(1024, 1024, tile_size) == 1024 * 1024 * 2 * 2200 + + def test_an_elongated_image_prices_the_tile_clamped_per_axis(self): + """A 2048px tile on a 1024x4096 image is tiled along the long side only, and every slice it + takes is at most 1024x2048.""" + assert self._decode_estimate(1024, 4096, 2048) == int(1024 * 2048 * 2 * 2200 * 1.25) + + @pytest.mark.parametrize("tile_size, resolved", [(0, 512), (64, 128)], ids=["default-sentinel", "below-minimum"]) + def test_the_node_tile_field_is_resolved_before_pricing(self, tile_size, resolved): + """The nodes pass their `tile_size` field through: 0 means the default tile, and anything under + the minimum is raised to it -- pricing the raw value would reserve nothing or too little.""" + assert self._decode_estimate(1024, 1024, tile_size) == int(resolved * resolved * 2 * 2200 * 1.25) + class TestVaeConstantsFollowTheConvBackend: """The pixel-area constant is not one number. MIOpen's convolution workspaces cost far more than @@ -533,74 +557,6 @@ def test_the_score_matrix_scales_with_the_batch(self): assert self._decode_estimate(3, px=1536, device=MATERIALIZING) == 3 * score -class TestFlux2VaeInvocationsRequestWorkingMemory: - """The estimate is worthless unless it reaches `model_on_device()`.""" - - def _mock_vae_info(self): - vae = MagicMock(spec=AutoencoderKLFlux2) - vae.parameters.return_value = iter([torch.zeros(1, dtype=torch.bfloat16)]) - - vae_info = MagicMock() - vae_info.model = vae - vae_info.compute_device = torch.device("cpu") - cm = MagicMock() - cm.__enter__ = MagicMock(return_value=(None, vae)) - cm.__exit__ = MagicMock(return_value=None) - vae_info.model_on_device = MagicMock(return_value=cm) - return vae_info - - def test_decode_requests_working_memory(self): - vae_info = self._mock_vae_info() - context = MagicMock() - context.models.load.return_value = vae_info - context.tensors.load.return_value = torch.zeros(1, 32, 128, 128) - - expected = 10 * GB - with patch( - "invokeai.app.invocations.vae.flux2_vae_decode.estimate_vae_working_memory_flux2", return_value=expected - ) as estimate: - invocation = Flux2VaeDecodeInvocation.model_construct( - latents=MagicMock(latents_name="latents"), vae=MagicMock(vae=MagicMock()) - ) - try: - invocation.invoke(context) - except Exception: - # The mocked decode math fails downstream; we only care that the cache was asked to - # reserve the estimate before the device context was entered. - pass - - estimate.assert_called_once() - assert estimate.call_args.kwargs["operation"] == "decode" - vae_info.model_on_device.assert_called_once_with(working_mem_bytes=expected) - - def test_encode_requests_working_memory(self): - vae_info = self._mock_vae_info() - context = MagicMock() - context.models.load.return_value = vae_info - - expected = 4 * GB - with ( - patch( - "invokeai.app.invocations.vae.flux2_vae_encode.estimate_vae_working_memory_flux2", return_value=expected - ) as estimate, - patch( - "invokeai.app.invocations.vae.flux2_vae_encode.image_resized_to_grid_as_tensor", - return_value=torch.zeros(3, 1024, 1024), - ), - ): - invocation = Flux2VaeEncodeInvocation.model_construct( - image=MagicMock(image_name="image"), vae=MagicMock(vae=MagicMock()) - ) - try: - invocation.invoke(context) - except Exception: - pass - - estimate.assert_called_once() - assert estimate.call_args.kwargs["operation"] == "encode" - vae_info.model_on_device.assert_called_once_with(working_mem_bytes=expected) - - class _StopBeforeLoad(Exception): """Raised in place of entering the transformer's device context, to end _run_diffusion early.""" @@ -900,6 +856,21 @@ def test_tiling_makes_the_score_matrix_irrelevant(self): # Tiling turns a 100GB+ reservation at the 2024px reference cap into under 2GB. assert estimates[0] < untiled / 50 + def test_the_score_matrix_of_an_elongated_image_follows_the_clamped_tile(self): + """A 2048px tile on a 1024x4096 image slices at most 1024x2048, so the mid-block attends over + 128x256 tokens, not the 256x256 an unclamped tile implies -- a quarter of the scores.""" + with _materializing(): + estimate = estimate_vae_working_memory_flux2( + operation="encode", + image_tensor=torch.zeros(1, 3, 1024, 4096), + vae=self._mock_bf16_vae(), + tile_size=2048, + device=MATERIALIZING, + ) + tokens = 128 * 256 + linear = int(1024 * 2048 * 2 * 1100 * 1.25) + assert estimate == max(linear, tokens * tokens * SDPA_MATH_BYTES_PER_SCORE_ELEMENT) + class TestVaeTermsDoNotAdd: """The two VAE terms peak in different phases of the same forward, so the estimate takes the