diff --git a/docs/src/content/docs/Users Guide/03.Models/01.introduction.mdx b/docs/src/content/docs/Users Guide/03.Models/01.introduction.mdx index 71708b62470..a59840a4415 100644 --- a/docs/src/content/docs/Users Guide/03.Models/01.introduction.mdx +++ b/docs/src/content/docs/Users Guide/03.Models/01.introduction.mdx @@ -98,6 +98,23 @@ information on how to generate these keys. Once a valid API key has been provided, the list of models hosted by that service will appear as selections in the Image Generation panel. +Opening an external model in the Model Manager shows an **External +Provider** section where you can change the provider (one of the +registered providers) and provider model ID it routes to, the modes +(text to image, image to image, inpaint) and +inputs (seed, reference images, negative prompt) it advertises, its +request limits, and the size and image count the provider is asked for +by default. The Generate panel and the Canvas only offer what the +record advertises, so keep it in step with the provider's documentation. +The Canvas compiles only text to image and image to image for external +models and refuses inpaint, because its keep-mask does not match any +provider's edit-mask contract; an advertised inpaint mode serves the +workflow editor. A model whose source matches a starter catalog entry is +shown read-only, including its provider and provider model ID, because +Invoke applies that catalog entry's capabilities to it on every load, so +a changed route would keep the original entry's capabilities. Models +added outside the starter catalog get the full editor. + :::note[Gated Downloads] The API Keys section is also used to provide keys for services that require authentication to download certain models, such as those diff --git a/docs/src/content/docs/Users Guide/03.Models/External Models/gemini.mdx b/docs/src/content/docs/Users Guide/03.Models/External Models/gemini.mdx index 53067376488..ee09997d3b0 100644 --- a/docs/src/content/docs/Users Guide/03.Models/External Models/gemini.mdx +++ b/docs/src/content/docs/Users Guide/03.Models/External Models/gemini.mdx @@ -39,6 +39,19 @@ Reference-image input is used to condition generation but counts as txt2img — All Gemini models are single-image-per-request — batch size is fixed at 1. To generate multiple variations, queue multiple invocations. +## Provider-Specific Options + +Selecting a Gemini model adds a **Provider options** section to the Generate panel: + +- **Temperature** — Override the sampling temperature (0–2; the provider default is 1). +- **Thinking level** — `minimal` or `high`, or the provider default. + +Gemini 3 models list their resolution presets under **Provider size** in the Size section. Picking one sets the width and height to the preset and sends its named size (`1K`, `2K`, `4K`) to the API. The named size and the sampling options are recorded in the image metadata and restored by **Recall All** and **Remix**. + +## Canvas + +Gemini models generate only from text, so on the Canvas the bounding box must be free of raster content: hide raster layers or move their content out of the box to generate. Invoke asks for the size whose aspect ratio is closest to the bounding box's; for Gemini 3 that is the smallest named size covering the box at that ratio, such as `2K` for a 2000×2000 box. The result is scaled to cover the bounding box and cropped to it from the center, so it is never stretched. + ## Tips diff --git a/docs/src/content/docs/Users Guide/03.Models/External Models/openai.mdx b/docs/src/content/docs/Users Guide/03.Models/External Models/openai.mdx index 2ee4628ebd3..7081175ccc0 100644 --- a/docs/src/content/docs/Users Guide/03.Models/External Models/openai.mdx +++ b/docs/src/content/docs/Users Guide/03.Models/External Models/openai.mdx @@ -44,13 +44,24 @@ Inpainting (mask-based editing) is not currently supported for any OpenAI model ## Provider-Specific Options -For **GPT Image** models, Invoke surfaces two provider-specific options in the parameters panel: +Selecting a GPT Image model adds a **Provider options** section to the Generate panel: - **Quality** — `low`, `medium`, `high`, or `auto`. Higher quality costs more and takes longer. - **Background** — `auto`, `transparent`, or `opaque`. Use `transparent` for PNG output with an alpha channel. +- **Input fidelity** — `low` or `high`, offered only for models that take image inputs (img2img or reference images). Leave it on the provider default to let OpenAI choose. DALL·E 2 and DALL·E 3 do not expose these options. +These are recorded in the image metadata, so **Recall All** and **Remix** restore them when the image's model is selected. + +The **Size** section lists the provider's fixed sizes per aspect ratio (**Provider size**). Any other size is mapped to the closest one the provider accepts. + +## Canvas + +GPT Image models that advertise `img2img` can generate from the Canvas: an empty bounding box runs text-to-image, and a filled one sends the composited bounding box as the init image. Invoke asks for the provider size whose aspect ratio is closest to the bounding box's, such as 1536×1024 for a 640×512 box. A text-to-image result is scaled to cover the bounding box and cropped to it from the center, so it is never stretched; when the shapes differ, a strip at two opposite edges is cut off. For image-to-image, the init image is resized to the provider size and the result is resized back to the exact bounding box. The render size, denoising strength, control layers and regional guidance do not apply and are not offered. Inpainting and outpainting are not available for OpenAI models. + +DALL·E 3 generates only from text: on the Canvas, hide raster layers or move their content out of the bounding box to generate with it. + ## How Requests Are Routed - Pure text-to-image requests hit `/v1/images/generations`. diff --git a/docs/src/content/docs/Users Guide/03.Models/External Models/seedream.mdx b/docs/src/content/docs/Users Guide/03.Models/External Models/seedream.mdx index de25273d289..800052bf362 100644 --- a/docs/src/content/docs/Users Guide/03.Models/External Models/seedream.mdx +++ b/docs/src/content/docs/Users Guide/03.Models/External Models/seedream.mdx @@ -48,11 +48,17 @@ All Seedream models share the same aspect ratio set: `1:1`, `2:3`, `3:2`, `3:4`, ## Provider-Specific Options -Seedream exposes two provider-specific toggles in the parameters panel: +Selecting a Seedream model adds a **Provider options** section to the Generate panel with two toggles: - **Watermark** — When enabled, BytePlus adds a small watermark to the output. Off by default. - **Optimize Prompt** — When enabled, BytePlus rewrites your prompt server-side for better generation quality. Useful for short prompts; disable if you want the exact wording preserved. +Both are recorded in the image metadata and restored by **Recall All** and **Remix**. The Size section lists the provider's 2K sizes per aspect ratio under **Provider size**. + +## Canvas + +Seedream models advertise `img2img`, so they can generate from the Canvas: an empty bounding box runs text-to-image, and a filled one sends the composited bounding box as the init image. Invoke asks for the provider size whose aspect ratio is closest to the bounding box's. A text-to-image result is scaled to cover the bounding box and cropped to it from the center, so it is never stretched; when the shapes differ, a strip at two opposite edges is cut off. For image-to-image, the init image is resized to the provider size and the result is resized back to the exact bounding box. The render size, denoising strength, control layers and regional guidance do not apply and are not offered. Inpainting and outpainting are not available. + **Seed** and **guidance scale** are not accepted by the 4.x / 5.x family. ## Reference Images diff --git a/docs/src/content/docs/contributing/external-providers.md b/docs/src/content/docs/contributing/external-providers.md index ab62fb886e4..0b132d69004 100644 --- a/docs/src/content/docs/contributing/external-providers.md +++ b/docs/src/content/docs/contributing/external-providers.md @@ -81,12 +81,14 @@ Starter overrides are matched by `source` (`external://provider/model-id`). Keep - runtime capability/default overrides depend on it - installation detection in starter-model APIs depends on it +- the model manager's External Provider section treats a record whose `source` matches a starter as catalog-managed and shows its routing, capabilities and defaults read-only, because a changed route would keep the matched entry's capabilities `STARTER_MODELS` enforces unique `source` values with an assertion. ### Install behavior notes - External starter models are managed in **External Providers** setup (not the regular Starter Models tab). +- Canvas compiles `txt2img` and `img2img` for external models; `inpaint` is refused on Canvas because its keep-mask does not match any provider's edit-mask contract (the workflow editor nodes still accept `mask_image`). For a model with `resolution_presets`, `aspect_ratio_sizes` or `allowed_aspect_ratios`, the Canvas requests one of the model's own sizes (the closest aspect ratio, then the smallest size covering the bounding box). `_bucket_request` passes a fixed size or an exact multiple of an allowed ratio through unchanged; a resolution preset whose rounded size is off its exact ratio may be nudged by a pixel, which is harmless because the provider sizes its output from the ratio and the preset's name. A `txt2img` result is scaled to cover the bounding box and center-cropped; an `img2img` init image is resized to the request size and the result resized back. Keep these capabilities accurate: the crop is planned from them. - External starter models auto-install when a provider is configured. - Removing a provider API key removes installed external models for that provider. diff --git a/invokeai/frontend/webv2/.oxfmtrc.json b/invokeai/frontend/webv2/.oxfmtrc.json index b4548ee4cab..534184baaa0 100644 --- a/invokeai/frontend/webv2/.oxfmtrc.json +++ b/invokeai/frontend/webv2/.oxfmtrc.json @@ -10,6 +10,7 @@ "stats.html", "src/features/generation/core/__snapshots__/generateGraphNodeTypes.json", "src/features/generation/core/__snapshots__/animaLLLiteCanvasGraphs.json", + "src/features/generation/core/__snapshots__/externalCanvasGraphs.json", "src/features/video/core/__snapshots__/videoGraphNodeTypes.json", "src/features/upscale/core/__snapshots__/upscaleGraphNodeTypes.json", "src/features/upscale/core/__snapshots__/upscaleIndexClippedStarts.json", diff --git a/invokeai/frontend/webv2/performance/architecture-baseline.json b/invokeai/frontend/webv2/performance/architecture-baseline.json index edcdf690391..285496f99a2 100644 --- a/invokeai/frontend/webv2/performance/architecture-baseline.json +++ b/invokeai/frontend/webv2/performance/architecture-baseline.json @@ -554,7 +554,9 @@ "source:src/features/generation/core/catalogSearch.ts", "source:src/features/generation/core/componentCompatibility.ts", "source:src/features/generation/core/conditioningRebalance.ts", + "source:src/features/generation/core/dimensionLimits.ts", "source:src/features/generation/core/dynamicPrompts.ts", + "source:src/features/generation/core/externalProvider.ts", "source:src/features/generation/core/graph.ts", "source:src/features/generation/core/graphBuilder.ts", "source:src/features/generation/core/krea2Conditioning.ts", diff --git a/invokeai/frontend/webv2/performance/browser-baseline.json b/invokeai/frontend/webv2/performance/browser-baseline.json index f83b5c8d8f2..0f02b90cdfa 100644 --- a/invokeai/frontend/webv2/performance/browser-baseline.json +++ b/invokeai/frontend/webv2/performance/browser-baseline.json @@ -29,12 +29,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 0, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 29, - "scriptRawBytes": 1868300, + "scriptRawBytes": 1882471, "scriptRequestCount": 25, - "totalRawBytes": 2120308 + "totalRawBytes": 2139551 }, "routeReadyMedianMs": 215.30000019073486, "scriptSourceOwnerSet": "launchpad-static", @@ -67,12 +67,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 0, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 29, - "scriptRawBytes": 1868300, + "scriptRawBytes": 1882471, "scriptRequestCount": 25, - "totalRawBytes": 2120308 + "totalRawBytes": 2139551 }, "routeReadyMedianMs": 225.39999961853027, "scriptSourceOwnerSet": "launchpad-static", @@ -105,12 +105,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 87, - "scriptRawBytes": 4199867, + "scriptRawBytes": 4250746, "scriptRequestCount": 82, - "totalRawBytes": 4496273 + "totalRawBytes": 4552224 }, "routeReadyMedianMs": 360.90000009536743, "scriptSourceOwnerSet": "editor-minimal-static", @@ -143,12 +143,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 87, - "scriptRawBytes": 4199867, + "scriptRawBytes": 4250746, "scriptRequestCount": 82, - "totalRawBytes": 4496273 + "totalRawBytes": 4552224 }, "routeReadyMedianMs": 367.80000019073486, "scriptSourceOwnerSet": "editor-minimal-static", @@ -181,12 +181,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 96, - "scriptRawBytes": 4718489, + "scriptRawBytes": 4775202, "scriptRequestCount": 91, - "totalRawBytes": 5014895 + "totalRawBytes": 5076680 }, "routeReadyMedianMs": 1095.3000001907349, "scriptSourceOwnerSet": "editor-canvas-static", @@ -219,12 +219,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 96, - "scriptRawBytes": 4718489, + "scriptRawBytes": 4775202, "scriptRequestCount": 91, - "totalRawBytes": 5014895 + "totalRawBytes": 5076680 }, "routeReadyMedianMs": 1042.5, "scriptSourceOwnerSet": "editor-canvas-static", @@ -257,12 +257,12 @@ "cssRawBytes": 17572, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 94, - "scriptRawBytes": 4644510, + "scriptRawBytes": 4702888, "scriptRequestCount": 88, - "totalRawBytes": 4956329 + "totalRawBytes": 5019779 }, "routeReadyMedianMs": 1093.0999999046326, "scriptSourceOwnerSet": "editor-workflow-static", @@ -295,12 +295,12 @@ "cssRawBytes": 17572, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 94, - "scriptRawBytes": 4644510, + "scriptRawBytes": 4702888, "scriptRequestCount": 88, - "totalRawBytes": 4956329 + "totalRawBytes": 5019779 }, "routeReadyMedianMs": 1105.0999999046326, "scriptSourceOwnerSet": "editor-workflow-static", @@ -333,12 +333,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 87, - "scriptRawBytes": 4199867, + "scriptRawBytes": 4250746, "scriptRequestCount": 82, - "totalRawBytes": 4496273 + "totalRawBytes": 4552224 }, "routeReadyMedianMs": 1324.5, "scriptSourceOwnerSet": "editor-minimal-static", @@ -371,12 +371,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752413, - "otherRawBytes": 226185, + "largestAssetRawBytes": 752422, + "otherRawBytes": 231257, "requestCount": 87, - "scriptRawBytes": 4199867, + "scriptRawBytes": 4250746, "scriptRequestCount": 82, - "totalRawBytes": 4496273 + "totalRawBytes": 4552224 }, "routeReadyMedianMs": 1301.5, "scriptSourceOwnerSet": "editor-minimal-static", @@ -981,7 +981,9 @@ "source:src/features/generation/core/catalogSearch.ts", "source:src/features/generation/core/componentCompatibility.ts", "source:src/features/generation/core/conditioningRebalance.ts", + "source:src/features/generation/core/dimensionLimits.ts", "source:src/features/generation/core/dynamicPrompts.ts", + "source:src/features/generation/core/externalProvider.ts", "source:src/features/generation/core/graph.ts", "source:src/features/generation/core/graphBuilder.ts", "source:src/features/generation/core/krea2Conditioning.ts", @@ -1028,6 +1030,7 @@ "source:src/features/generation/ui/GenerateHeaderActions.tsx", "source:src/features/generation/ui/GenerateModelCard.tsx", "source:src/features/generation/ui/GeneratePresetsPopover.tsx", + "source:src/features/generation/ui/GenerateProviderOptionsSection.tsx", "source:src/features/generation/ui/GenerateRefinerSection.tsx", "source:src/features/generation/ui/GenerateRenderSection.tsx", "source:src/features/generation/ui/GenerateSettingsForm.tsx", @@ -1078,6 +1081,7 @@ "source:src/features/generation/ui/shared/ConceptRow.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceBars.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceSparkline.tsx", + "source:src/features/generation/ui/shared/ExternalSizeSelect.tsx", "source:src/features/generation/ui/shared/GenerateCollapsibleSection.tsx", "source:src/features/generation/ui/shared/GenerateFieldContextMenu.tsx", "source:src/features/generation/ui/shared/GenerateToggleSwitch.tsx", @@ -2080,7 +2084,9 @@ "source:src/features/generation/core/catalogSearch.ts", "source:src/features/generation/core/componentCompatibility.ts", "source:src/features/generation/core/conditioningRebalance.ts", + "source:src/features/generation/core/dimensionLimits.ts", "source:src/features/generation/core/dynamicPrompts.ts", + "source:src/features/generation/core/externalProvider.ts", "source:src/features/generation/core/graph.ts", "source:src/features/generation/core/graphBuilder.ts", "source:src/features/generation/core/krea2Conditioning.ts", @@ -2127,6 +2133,7 @@ "source:src/features/generation/ui/GenerateHeaderActions.tsx", "source:src/features/generation/ui/GenerateModelCard.tsx", "source:src/features/generation/ui/GeneratePresetsPopover.tsx", + "source:src/features/generation/ui/GenerateProviderOptionsSection.tsx", "source:src/features/generation/ui/GenerateRefinerSection.tsx", "source:src/features/generation/ui/GenerateRenderSection.tsx", "source:src/features/generation/ui/GenerateSettingsForm.tsx", @@ -2177,6 +2184,7 @@ "source:src/features/generation/ui/shared/ConceptRow.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceBars.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceSparkline.tsx", + "source:src/features/generation/ui/shared/ExternalSizeSelect.tsx", "source:src/features/generation/ui/shared/GenerateCollapsibleSection.tsx", "source:src/features/generation/ui/shared/GenerateFieldContextMenu.tsx", "source:src/features/generation/ui/shared/GenerateToggleSwitch.tsx", @@ -3339,7 +3347,9 @@ "source:src/features/generation/core/catalogSearch.ts", "source:src/features/generation/core/componentCompatibility.ts", "source:src/features/generation/core/conditioningRebalance.ts", + "source:src/features/generation/core/dimensionLimits.ts", "source:src/features/generation/core/dynamicPrompts.ts", + "source:src/features/generation/core/externalProvider.ts", "source:src/features/generation/core/graph.ts", "source:src/features/generation/core/graphBuilder.ts", "source:src/features/generation/core/krea2Conditioning.ts", @@ -3386,6 +3396,7 @@ "source:src/features/generation/ui/GenerateHeaderActions.tsx", "source:src/features/generation/ui/GenerateModelCard.tsx", "source:src/features/generation/ui/GeneratePresetsPopover.tsx", + "source:src/features/generation/ui/GenerateProviderOptionsSection.tsx", "source:src/features/generation/ui/GenerateRefinerSection.tsx", "source:src/features/generation/ui/GenerateRenderSection.tsx", "source:src/features/generation/ui/GenerateSettingsForm.tsx", @@ -3436,6 +3447,7 @@ "source:src/features/generation/ui/shared/ConceptRow.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceBars.tsx", "source:src/features/generation/ui/shared/ConditioningRebalanceSparkline.tsx", + "source:src/features/generation/ui/shared/ExternalSizeSelect.tsx", "source:src/features/generation/ui/shared/GenerateCollapsibleSection.tsx", "source:src/features/generation/ui/shared/GenerateFieldContextMenu.tsx", "source:src/features/generation/ui/shared/GenerateToggleSwitch.tsx", diff --git a/invokeai/frontend/webv2/public/locales/en.json b/invokeai/frontend/webv2/public/locales/en.json index 6b51cf19e59..f8846ef359c 100644 --- a/invokeai/frontend/webv2/public/locales/en.json +++ b/invokeai/frontend/webv2/public/locales/en.json @@ -1498,7 +1498,41 @@ "emptyCacheConfirmBody": "Unloads all models from RAM and VRAM. The next generation reloads them from disk.", "emptyCacheConfirmTitle": "Empty model cache?", "providerId": "Provider", + "providerIdHelp": "Every registered provider, configured or not, plus this record's saved provider. Configure the provider's API key before generating.", "providerModelId": "Provider Model ID", + "providerModelIdHelp": "The model name the provider expects in requests.", + "externalProvider": "External Provider", + "externalProviderHelp": "Where requests for this model go, what the provider accepts, and the size and count it is asked for by default.", + "saveExternalSettings": "Save Provider Settings", + "externalSettingsSaved": "Provider settings saved", + "failedToSaveExternalSettings": "Failed to save provider settings.", + "externalModes": { + "txt2img": "Text to image", + "img2img": "Image to image", + "inpaint": "Inpaint" + }, + "externalFields": { + "supportedModes": "Supported modes", + "supportedModesHelp": "The modes the provider accepts. Canvas supports text to image and image to image; provider masks for inpaint are not yet supported.", + "supportsSeed": "Accepts a seed", + "supportsReferenceImages": "Accepts reference images", + "supportsNegativePrompt": "Accepts a negative prompt", + "maxImagesPerRequest": "Images per request", + "maxReferenceImages": "Max reference images", + "maxImageWidth": "Max width", + "maxImageHeight": "Max height", + "defaultWidth": "Default width", + "defaultHeight": "Default height", + "defaultNumImages": "Default image count", + "unlimited": "No limit", + "providerDefault": "Provider default", + "supported": "Supported", + "notSupported": "Not supported", + "starterManaged": "Managed by the starter catalog. Routing, capabilities, and defaults cannot be edited here.", + "lockedDraftBlocked": "These unsaved values are not the effective configuration and cannot be saved while this record is locked. Reset to discard them.", + "catalogUnavailable": "Could not check the starter catalog. Editing is unavailable until this check succeeds.", + "catalogLoading": "Checking the starter catalog before enabling edits…" + }, "queueAttentionCount_one": "{{count}} needs attention", "queueAttentionCount_other": "{{count}} need attention", "queueActionFailed": "Queue action failed", @@ -2532,6 +2566,8 @@ "deletePresetNamed": "Delete preset {{name}}", "deletePromptHistoryItem": "Delete prompt history item", "denoisingStrength": "Denoising strength", + "dimensionsAdjusted": "Dimensions adjusted", + "dimensionsAdjustedDescription": "Set to {{width}}×{{height}} for {{name}}.", "disableConcept": "Disable {{name}}", "disableReferenceImage": "Disable reference image", "dropImageToPrompt": "Describe this image", @@ -2761,6 +2797,7 @@ "searchPresets": "Search presets", "secondsPerRun": "~{{value}}s/run", "sizePreviewHandle": "Drag to resize the generation size", + "switchModelAdjustsDimensions": "It also sets the dimensions to {{width}}×{{height}}.", "switchModelBody": "Switching to {{name}} turns off or removes: {{labels}}.", "switchModelConfirm": "Switch model", "switchModelTitle": "Switch model?", @@ -2897,6 +2934,49 @@ "pidHelp": "Replaces the VAE decode with a 4x super-resolution decode.", "pidFitHelp": "Generates at the requested size, then decodes 4x and scales back down.", "pidNativeHelp": "The requested size is the 4x target; generation runs at a quarter of it.", + "providerSize": "Provider size", + "providerSizeCanvasCropHelp": "The Canvas generates {{width}}×{{height}}, then fits it to the frame: text to image is cropped at the edges, edits are resized.", + "providerSizeCanvasCropTextHelp": "The Canvas generates {{width}}×{{height}}, then crops it to the frame.", + "providerSizeCanvasResizeHelp": "The Canvas generates {{width}}×{{height}} and resizes it to the frame.", + "providerSizeHelp": "Sizes this provider renders natively. Other sizes are mapped to the closest one.", + "providerSizeCustom": "Custom", + "providerOptions": { + "title": "Provider options", + "providerDefault": "Provider default", + "openaiQuality": "Quality", + "openaiQualityHelp": "Higher quality costs more and takes longer.", + "openaiQualityOptions": { + "auto": "Auto", + "high": "High", + "medium": "Medium", + "low": "Low" + }, + "openaiBackground": "Background", + "openaiBackgroundHelp": "Transparent keeps an alpha channel in the PNG output.", + "openaiBackgroundOptions": { + "auto": "Auto", + "transparent": "Transparent", + "opaque": "Opaque" + }, + "openaiInputFidelity": "Input fidelity", + "openaiInputFidelityHelp": "How closely edits and reference images are preserved.", + "openaiInputFidelityOptions": { + "low": "Low", + "high": "High" + }, + "geminiTemperature": "Temperature", + "geminiTemperatureHelp": "Sampling temperature from 0 to 2; the provider default is 1.", + "geminiThinkingLevel": "Thinking level", + "geminiThinkingLevelHelp": "How much the model reasons about the prompt before rendering.", + "geminiThinkingLevelOptions": { + "minimal": "Minimal", + "high": "High" + }, + "seedreamWatermark": "Watermark", + "seedreamWatermarkHelp": "Adds the provider's small watermark to the output.", + "seedreamOptimizePrompt": "Optimize prompt", + "seedreamOptimizePromptHelp": "Lets the provider rewrite the prompt before generating. Turn off to keep the exact wording." + }, "presetSettingsCleared": "Settings cleared by preset", "presetSettingsClearedDescription": "Cleared: {{labels}}.", "refiner": "Refiner", diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs index 82432a94d55..61a76b490c0 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs @@ -398,7 +398,155 @@ const MODEL_TYPES = ['main', 'lora', 'controlnet', 'vae', 'embedding']; // Main models cycle through their own bases so Generate sees SDXL mains and refiners, not only SD 1.5. const MAIN_MODEL_BASES = ['sd-1', 'sdxl', 'sdxl-refiner']; -const createModels = (count) => +/** The source of the hosted record the mock starter catalog also lists, so the Model Manager shows it read-only. */ +export const CATALOG_MANAGED_EXTERNAL_SOURCE = 'external://gemini/gemini-2.5-flash-image'; + +/** + * Hosted-provider records with the capability shapes the Generate panel, Canvas, and Model Manager read: a named + * resolution table (Gemini), fixed sizes per ratio with an edit mode (OpenAI), provider toggles (Seedream), and a + * record the starter catalog manages. No shipped provider accepts a mask, so none advertises inpaint. + */ +const EXTERNAL_MODEL_FIXTURES = [ + { + base: 'external', + capabilities: { + allowed_aspect_ratios: ['1:1', '16:9', '9:16'], + max_image_size: { height: 4096, width: 4096 }, + max_images_per_request: 1, + max_reference_images: 14, + modes: ['txt2img'], + resolution_presets: ['1K', '2K'].flatMap((imageSize) => + [ + ['1:1', 1, 1], + ['16:9', 16, 9], + ['9:16', 9, 16], + ].map(([ratio, w, h]) => { + const base = imageSize === '1K' ? 1024 : 2048; + const width = w >= h ? base : Math.round((base * w) / h); + const height = w >= h ? Math.round((base * h) / w) : base; + return { + aspect_ratio: ratio, + height, + image_size: imageSize, + label: `${ratio} (${imageSize}) — ${width}×${height}`, + width, + }; + }) + ), + supports_reference_images: true, + supports_seed: true, + }, + cover_image: null, + default_settings: { height: 1024, num_images: 1, width: 1024 }, + description: 'Fixture hosted model with named resolution presets and sampling options.', + file_size: 0, + format: 'external_api', + hash: 'external:gemini:fixture-gemini-image', + key: 'fixture-external-gemini', + name: 'Fixture Gemini Image', + path: 'external://gemini/fixture-gemini-image', + provider_id: 'gemini', + provider_model_id: 'fixture-gemini-image', + source: 'external://gemini/fixture-gemini-image', + source_type: 'external', + source_url: null, + trigger_phrases: null, + type: 'external_image_generator', + }, + { + base: 'external', + capabilities: { + allowed_aspect_ratios: ['1:1', '3:2', '2:3'], + aspect_ratio_sizes: { + '1:1': { height: 1024, width: 1024 }, + '3:2': { height: 1024, width: 1536 }, + '2:3': { height: 1536, width: 1024 }, + }, + max_images_per_request: 10, + modes: ['txt2img', 'img2img'], + supports_reference_images: true, + }, + cover_image: null, + default_settings: { height: 1024, num_images: 1, width: 1024 }, + description: 'Fixture hosted model with fixed ratio sizes and canvas image-to-image.', + file_size: 0, + format: 'external_api', + hash: 'external:openai:fixture-gpt-image', + key: 'fixture-external-openai', + name: 'Fixture GPT Image', + path: 'external://openai/fixture-gpt-image', + provider_id: 'openai', + // A GPT Image model id, so Generate offers OpenAI's quality, background, and input-fidelity options; the + // fixture source keeps the record out of the starter catalog and editable in the Model Manager. + provider_model_id: 'gpt-image-1', + source: 'external://openai/fixture-gpt-image', + source_type: 'external', + source_url: null, + trigger_phrases: null, + type: 'external_image_generator', + }, + { + base: 'external', + capabilities: { + allowed_aspect_ratios: ['1:1', '16:9'], + aspect_ratio_sizes: { '1:1': { height: 2048, width: 2048 }, '16:9': { height: 1600, width: 2848 } }, + max_images_per_request: 15, + max_reference_images: 14, + modes: ['txt2img', 'img2img'], + supports_reference_images: true, + }, + cover_image: null, + default_settings: { height: 2048, num_images: 1, width: 2048 }, + description: 'Fixture hosted model with provider toggles.', + file_size: 0, + format: 'external_api', + hash: 'external:seedream:fixture-seedream', + key: 'fixture-external-seedream', + name: 'Fixture Seedream', + path: 'external://seedream/fixture-seedream', + provider_id: 'seedream', + provider_model_id: 'fixture-seedream', + source: 'external://seedream/fixture-seedream', + source_type: 'external', + source_url: null, + trigger_phrases: null, + type: 'external_image_generator', + }, + { + base: 'external', + capabilities: { + allowed_aspect_ratios: ['1:1', '16:9', '9:16'], + max_images_per_request: 1, + max_reference_images: 3, + modes: ['txt2img'], + supports_reference_images: true, + }, + cover_image: null, + default_settings: { height: 1024, num_images: 1, width: 1024 }, + description: 'Fixture hosted model whose capabilities the starter catalog manages.', + file_size: 0, + format: 'external_api', + hash: 'external:gemini:gemini-2.5-flash-image', + key: 'fixture-external-gemini-catalog', + name: 'Fixture Gemini Flash (Catalog)', + path: CATALOG_MANAGED_EXTERNAL_SOURCE, + provider_id: 'gemini', + provider_model_id: 'gemini-2.5-flash-image', + source: CATALOG_MANAGED_EXTERNAL_SOURCE, + source_type: 'external', + source_url: null, + trigger_phrases: null, + type: 'external_image_generator', + }, +]; + +/** The last slots of the model library hold the hosted-provider records so the profile counts stay as declared. */ +const createModels = (count) => [ + ...createLocalModels(Math.max(0, count - EXTERNAL_MODEL_FIXTURES.length)), + ...EXTERNAL_MODEL_FIXTURES.slice(0, count).map((model) => structuredClone(model)), +]; + +const createLocalModels = (count) => range(count, (index) => { const id = ordinal(index); const type = MODEL_TYPES[index % MODEL_TYPES.length]; diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs index fda932d3f20..c50a9ae4054 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs @@ -1469,6 +1469,89 @@ test('runtime config PATCH keeps the other writable field and validates the queu }); }); +test('hosted-provider credentials and the starter catalog agree with the hosted fixture records', async () => { + await withRepresentativeBackend(async (backend) => { + const { models } = await getJson(backend, '/api/v2/models/'); + const hosted = models.filter((model) => model.type === 'external_image_generator'); + const configs = await getJson(backend, '/api/v1/app/external_providers/config'); + + // Every backend provider is listed; those the hosted records route to have a key. + assert.deepEqual( + configs.map((config) => config.provider_id), + ['alibabacloud', 'atlascloud', 'gemini', 'openai', 'seedream'] + ); + assert.deepEqual( + configs.filter((config) => config.api_key_configured).map((config) => config.provider_id), + [...new Set(hosted.map((model) => model.provider_id))].sort() + ); + assert.ok(configs.every((config) => config.base_url === null)); + + // Exactly one hosted record carries a starter's source, so the Model Manager shows it catalog-managed. + const { starter_models: starters } = await getJson(backend, '/api/v2/models/starter_models'); + const starterSources = new Set(starters.map((starter) => starter.source)); + const managed = hosted.filter((model) => starterSources.has(model.source)); + assert.deepEqual( + managed.map((model) => model.key), + ['fixture-external-gemini-catalog'] + ); + // The editable OpenAI record routes to a GPT Image model, the only OpenAI models that take provider options. + assert.equal(hosted.find((model) => model.key === 'fixture-external-openai').provider_model_id, 'gpt-image-1'); + assert.equal(starters.find((starter) => starter.source === managed[0].source).is_installed, true); + assert.ok(starters.filter((starter) => starter.type !== 'external_image_generator').every((s) => !s.is_installed)); + + // Removing a provider's key removes its hosted records, as the backend does. + const reset = await getJson(backend, '/api/v1/app/external_providers/config/openai', { method: 'DELETE' }); + assert.deepEqual(reset, { api_key_configured: false, base_url: null, provider_id: 'openai' }); + const remaining = (await getJson(backend, '/api/v2/models/')).models; + assert.equal(remaining.length, models.length - hosted.filter((model) => model.provider_id === 'openai').length); + assert.equal( + (await fetch(`${backend.origin}/api/v1/app/external_providers/config/unknown`, { method: 'DELETE' })).status, + 404 + ); + + // A body naming neither field is refused, and changes nothing. + const refused = await fetch(`${backend.origin}/api/v1/app/external_providers/config/gemini`, { + body: JSON.stringify({}), + method: 'POST', + }); + assert.equal(refused.status, 400); + assert.equal( + (await getJson(backend, '/api/v1/app/external_providers/config')).find((c) => c.provider_id === 'gemini') + .api_key_configured, + true + ); + + // Saving an empty key removes the key and the provider's hosted records, like a reset. + const cleared = await getJson(backend, '/api/v1/app/external_providers/config/gemini', { + body: JSON.stringify({ api_key: ' ' }), + method: 'POST', + }); + assert.equal(cleared.api_key_configured, false); + const afterClear = (await getJson(backend, '/api/v2/models/')).models; + assert.equal(afterClear.filter((model) => model.provider_id === 'gemini').length, 0); + assert.equal(afterClear.length, remaining.length - hosted.filter((model) => model.provider_id === 'gemini').length); + + const empty = await startMockBackend(0, { profile: 'empty' }); + try { + const emptyConfigs = await getJson(empty, '/api/v1/app/external_providers/config'); + assert.equal(emptyConfigs.length, 5); + assert.ok(emptyConfigs.every((config) => !config.api_key_configured)); + } finally { + await empty.close(); + } + + const configured = await getJson(backend, '/api/v1/app/external_providers/config/openai', { + body: JSON.stringify({ api_key: 'sk-fixture', base_url: 'https://example.invalid/v1' }), + method: 'POST', + }); + assert.deepEqual(configured, { + api_key_configured: true, + base_url: 'https://example.invalid/v1', + provider_id: 'openai', + }); + }); +}); + test('board_id=all lists unboarded items and those on active readable boards, alike in items, names and location', async () => { await withRepresentativeBackend(async (backend) => { const query = 'categories=general&is_intermediate=false'; diff --git a/invokeai/frontend/webv2/scripts/mock-backend.mjs b/invokeai/frontend/webv2/scripts/mock-backend.mjs index a48aeba84c8..635aad27394 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend.mjs @@ -6,6 +6,7 @@ import { resolve } from 'node:path'; import { assertMockBackendFixture, assertMockBackendProfileName, + CATALOG_MANAGED_EXTERNAL_SOURCE, collectCanvasLeaves, createMockBackendFixture, getFixtureProjectWorkflowDocument, @@ -236,6 +237,17 @@ const STARTER_MODEL = { type: 'main', }; +/** The hosted starter whose source a fixture record carries, so that record reads as catalog-managed. */ +const EXTERNAL_STARTER_MODEL = { + base: 'external', + description: 'Fixture hosted starter model', + format: 'external_api', + is_installed: false, + name: 'Fixture Gemini Flash (Catalog)', + source: CATALOG_MANAGED_EXTERNAL_SOURCE, + type: 'external_image_generator', +}; + const MOCK_APP_VERSION = 'fixture'; const STARTER_MODELS_RESPONSE = { @@ -246,7 +258,52 @@ const STARTER_MODELS_RESPONSE = { name: 'SDXL', }, }, - starter_models: [STARTER_MODEL], + starter_models: [STARTER_MODEL, EXTERNAL_STARTER_MODEL], +}; + +/** Starters read as installed when a library record carries their source, as the backend reports them. */ +const getStarterModelsResponse = (state) => { + const installedSources = new Set([...state.models.values()].map((model) => model.source)); + const withInstalled = (starter) => ({ ...starter, is_installed: installedSources.has(starter.source) }); + + return { + starter_bundles: Object.fromEntries( + Object.entries(STARTER_MODELS_RESPONSE.starter_bundles).map(([base, bundle]) => [ + base, + { ...bundle, models: bundle.models.map(withInstalled) }, + ]) + ), + starter_models: STARTER_MODELS_RESPONSE.starter_models.map(withInstalled), + }; +}; + +/** The backend's `EXTERNAL_PROVIDER_FIELDS`, in its order: the config route lists each, configured or not. */ +const EXTERNAL_PROVIDER_IDS = ['alibabacloud', 'atlascloud', 'gemini', 'openai', 'seedream']; + +/** + * Hosted-provider credentials as the config route reports them. A provider the profile's hosted records route to has a + * key, since the backend removes a provider's records when its key is removed; the rest have none. + */ +const createExternalProviderConfigs = (models) => { + const routed = new Set( + models.filter((model) => model.type === 'external_image_generator').map((model) => model.provider_id) + ); + + return new Map( + EXTERNAL_PROVIDER_IDS.map((providerId) => [ + providerId, + { api_key_configured: routed.has(providerId), base_url: null, provider_id: providerId }, + ]) + ); +}; + +/** The backend drops a provider's hosted records when its key is removed or reset. */ +const removeExternalModelsForProvider = (state, providerId) => { + for (const [key, model] of state.models) { + if (model.type === 'external_image_generator' && model.provider_id === providerId) { + state.models.delete(key); + } + } }; const DELIVERY_CLAIM_TTL_MS = 120_000; @@ -287,6 +344,7 @@ const createState = (profile) => { JSON.stringify({ alphaNoticeAcknowledged: true, whatsNewSeenVersion: MOCK_APP_VERSION }), ], ]), + externalProviders: createExternalProviderConfigs(fixture.models), images: new Map(fixture.images.map((image) => [image.image_name, clone(image)])), models: new Map(fixture.models.map((model) => [model.key, clone(model)])), /** What re-identify fills in per model key; only `POST /__models` seeds entries. */ @@ -1238,8 +1296,42 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { } } - if (path.startsWith('/api/v1/app/external_providers')) { - return json(200, []); + if (method === 'GET' && path === '/api/v1/app/external_providers/config') { + return json(200, [...state.externalProviders.values()].map(clone)); + } + + const externalProviderMatch = /^\/api\/v1\/app\/external_providers\/config\/([^/]+)$/.exec(path); + if (externalProviderMatch && (method === 'POST' || method === 'DELETE')) { + const providerId = decodeURIComponent(externalProviderMatch[1]); + const config = state.externalProviders.get(providerId); + + if (!config) { + return json(404, { detail: `Unknown external provider '${providerId}'` }); + } + if (method === 'DELETE') { + Object.assign(config, { api_key_configured: false, base_url: null }); + removeExternalModelsForProvider(state, providerId); + return json(200, clone(config)); + } + + const update = await readJsonBody(request); + const apiKey = typeof update.api_key === 'string' ? update.api_key.trim() : null; + const baseUrl = typeof update.base_url === 'string' ? update.base_url.trim() : null; + + if (apiKey === null && baseUrl === null) { + return json(400, { detail: 'No external provider config fields provided' }); + } + if (baseUrl !== null) { + config.base_url = baseUrl || null; + } + if (apiKey !== null) { + config.api_key_configured = apiKey !== ''; + if (apiKey === '') { + removeExternalModelsForProvider(state, providerId); + } + } + + return json(200, clone(config)); } if (path === '/api/v1/projects/' || path === '/api/v1/projects') { @@ -1822,7 +1914,7 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { return json(200, faults.capabilities === 'empty' ? [] : ARCHITECTURE_CAPABILITIES); } if (method === 'GET' && path === '/api/v2/models/starter_models') { - return json(200, STARTER_MODELS_RESPONSE); + return json(200, getStarterModelsResponse(state)); } if (method === 'GET' && path === '/api/v2/models/hf_login') { return json(200, 'unknown'); @@ -1849,7 +1941,18 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { const key = decodeURIComponent(modelMatch[1]); const model = state.models.get(key); - return model ? json(200, model) : json(404, { detail: 'Model not found' }); + if (!model) { + return json(404, { detail: 'Model not found' }); + } + if (method === 'PATCH') { + // Mirrors ModelRecordService.update_model: each submitted field replaces the stored one wholesale. + const changes = await readJsonBody(request); + const updated = { ...model, ...changes }; + state.models.set(key, updated); + return json(200, updated); + } + + return json(200, model); } if (method === 'GET' && (path === '/api/v2/models' || path === '/api/v2/models/')) { diff --git a/invokeai/frontend/webv2/src/features/generation/core/__snapshots__/externalCanvasGraphs.json b/invokeai/frontend/webv2/src/features/generation/core/__snapshots__/externalCanvasGraphs.json new file mode 100644 index 00000000000..cae6c5dd4df --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/__snapshots__/externalCanvasGraphs.json @@ -0,0 +1,538 @@ +{ + "_comment": "Generated by compileCanvasGraph.test.ts; regenerate with vitest -u. Validated by tests/app/invocations/test_frontend_external_canvas_graphs.py.", + "txt2img": { + "edges": [ + { + "destination": { + "field": "prompt", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "value", + "node_id": "positive_prompt" + } + }, + { + "destination": { + "field": "seed", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "value", + "node_id": "seed" + } + }, + { + "destination": { + "field": "seed", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "seed" + } + }, + { + "destination": { + "field": "positive_prompt", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "positive_prompt" + } + }, + { + "destination": { + "field": "negative_prompt", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "negative_prompt" + } + }, + { + "destination": { + "field": "metadata", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "metadata", + "node_id": "core_metadata_0_i" + } + }, + { + "destination": { + "field": "collection", + "node_id": "canvas_provider_image" + }, + "source": { + "field": "collection", + "node_id": "canvas_processing_output" + } + }, + { + "destination": { + "field": "image", + "node_id": "canvas_provider_metadata" + }, + "source": { + "field": "item", + "node_id": "canvas_provider_image" + } + }, + { + "destination": { + "field": "image", + "node_id": "canvas_provider_cover" + }, + "source": { + "field": "item", + "node_id": "canvas_provider_image" + } + }, + { + "destination": { + "field": "image", + "node_id": "canvas_output" + }, + "source": { + "field": "image", + "node_id": "canvas_provider_cover" + } + }, + { + "destination": { + "field": "metadata", + "node_id": "canvas_output" + }, + "source": { + "field": "metadata", + "node_id": "canvas_provider_metadata" + } + } + ], + "id": "external_graph_0_i", + "nodes": { + "negative_prompt": { + "is_intermediate": true, + "use_cache": true, + "id": "negative_prompt", + "type": "string" + }, + "positive_prompt": { + "is_intermediate": true, + "use_cache": true, + "id": "positive_prompt", + "type": "string" + }, + "seed": { + "is_intermediate": true, + "use_cache": true, + "id": "seed", + "type": "integer" + }, + "core_metadata_0_i": { + "is_intermediate": true, + "use_cache": true, + "cfg_scale": 7, + "cfg_rescale_multiplier": 0, + "height": 512, + "id": "core_metadata_0_i", + "model": { + "base": "external", + "capabilities": { + "allowed_aspect_ratios": [ + "1:1", + "3:2", + "2:3" + ], + "aspect_ratio_sizes": { + "1:1": { + "height": 1024, + "width": 1024 + }, + "3:2": { + "height": 1024, + "width": 1536 + }, + "2:3": { + "height": 1536, + "width": 1024 + } + }, + "modes": [ + "txt2img", + "img2img" + ], + "supports_seed": true + }, + "format": "external_api", + "key": "external-model", + "name": "OpenAI Image", + "provider_id": "openai", + "type": "external_image_generator", + "provider_model_id": "gpt-image-1", + "hash": "blake3:external" + }, + "rand_device": "cpu", + "seamless_x": false, + "seamless_y": false, + "steps": 30, + "type": "core_metadata", + "width": 640, + "openai_quality": "auto", + "openai_background": "auto", + "clip_skip": 0 + }, + "canvas_processing_output": { + "is_intermediate": true, + "use_cache": false, + "height": 1024, + "id": "canvas_processing_output", + "image_size": null, + "mode": "txt2img", + "model": { + "base": "external", + "capabilities": { + "allowed_aspect_ratios": [ + "1:1", + "3:2", + "2:3" + ], + "aspect_ratio_sizes": { + "1:1": { + "height": 1024, + "width": 1024 + }, + "3:2": { + "height": 1024, + "width": 1536 + }, + "2:3": { + "height": 1536, + "width": 1024 + } + }, + "modes": [ + "txt2img", + "img2img" + ], + "supports_seed": true + }, + "format": "external_api", + "key": "external-model", + "name": "OpenAI Image", + "provider_id": "openai", + "type": "external_image_generator", + "provider_model_id": "gpt-image-1", + "hash": "blake3:external" + }, + "num_images": 1, + "type": "openai_image_generation", + "width": 1536, + "quality": "auto", + "background": "auto" + }, + "canvas_provider_image": { + "is_intermediate": true, + "use_cache": false, + "id": "canvas_provider_image", + "type": "iterate" + }, + "canvas_provider_metadata": { + "is_intermediate": true, + "use_cache": true, + "id": "canvas_provider_metadata", + "type": "metadata_from_image" + }, + "canvas_provider_cover": { + "is_intermediate": true, + "use_cache": false, + "height": 512, + "id": "canvas_provider_cover", + "type": "img_resize", + "width": 768 + }, + "canvas_output": { + "is_intermediate": true, + "use_cache": false, + "id": "canvas_output", + "height": 512, + "type": "img_crop", + "width": 640, + "x": 64, + "y": 0 + } + } + }, + "img2img": { + "edges": [ + { + "destination": { + "field": "prompt", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "value", + "node_id": "positive_prompt" + } + }, + { + "destination": { + "field": "seed", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "value", + "node_id": "seed" + } + }, + { + "destination": { + "field": "seed", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "seed" + } + }, + { + "destination": { + "field": "positive_prompt", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "positive_prompt" + } + }, + { + "destination": { + "field": "negative_prompt", + "node_id": "core_metadata_0_i" + }, + "source": { + "field": "value", + "node_id": "negative_prompt" + } + }, + { + "destination": { + "field": "metadata", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "metadata", + "node_id": "core_metadata_0_i" + } + }, + { + "destination": { + "field": "init_image", + "node_id": "canvas_processing_output" + }, + "source": { + "field": "image", + "node_id": "canvas_resize_image_to_request" + } + }, + { + "destination": { + "field": "collection", + "node_id": "canvas_provider_image" + }, + "source": { + "field": "collection", + "node_id": "canvas_processing_output" + } + }, + { + "destination": { + "field": "image", + "node_id": "canvas_provider_metadata" + }, + "source": { + "field": "item", + "node_id": "canvas_provider_image" + } + }, + { + "destination": { + "field": "image", + "node_id": "canvas_output" + }, + "source": { + "field": "item", + "node_id": "canvas_provider_image" + } + }, + { + "destination": { + "field": "metadata", + "node_id": "canvas_output" + }, + "source": { + "field": "metadata", + "node_id": "canvas_provider_metadata" + } + } + ], + "id": "external_graph_0_i", + "nodes": { + "negative_prompt": { + "is_intermediate": true, + "use_cache": true, + "id": "negative_prompt", + "type": "string" + }, + "positive_prompt": { + "is_intermediate": true, + "use_cache": true, + "id": "positive_prompt", + "type": "string" + }, + "seed": { + "is_intermediate": true, + "use_cache": true, + "id": "seed", + "type": "integer" + }, + "core_metadata_0_i": { + "is_intermediate": true, + "use_cache": true, + "cfg_scale": 7, + "cfg_rescale_multiplier": 0, + "height": 512, + "id": "core_metadata_0_i", + "model": { + "base": "external", + "capabilities": { + "allowed_aspect_ratios": [ + "1:1", + "3:2", + "2:3" + ], + "aspect_ratio_sizes": { + "1:1": { + "height": 1024, + "width": 1024 + }, + "3:2": { + "height": 1024, + "width": 1536 + }, + "2:3": { + "height": 1536, + "width": 1024 + } + }, + "modes": [ + "txt2img", + "img2img" + ], + "supports_seed": true + }, + "format": "external_api", + "key": "external-model", + "name": "OpenAI Image", + "provider_id": "openai", + "type": "external_image_generator", + "provider_model_id": "gpt-image-1", + "hash": "blake3:external" + }, + "rand_device": "cpu", + "seamless_x": false, + "seamless_y": false, + "steps": 30, + "type": "core_metadata", + "width": 640, + "openai_quality": "auto", + "openai_background": "auto", + "clip_skip": 0 + }, + "canvas_resize_image_to_request": { + "is_intermediate": true, + "use_cache": true, + "height": 1024, + "id": "canvas_resize_image_to_request", + "image": { + "image_name": "canvas-composite.png" + }, + "type": "img_resize", + "width": 1536 + }, + "canvas_processing_output": { + "is_intermediate": true, + "use_cache": false, + "height": 1024, + "id": "canvas_processing_output", + "image_size": null, + "mode": "img2img", + "model": { + "base": "external", + "capabilities": { + "allowed_aspect_ratios": [ + "1:1", + "3:2", + "2:3" + ], + "aspect_ratio_sizes": { + "1:1": { + "height": 1024, + "width": 1024 + }, + "3:2": { + "height": 1024, + "width": 1536 + }, + "2:3": { + "height": 1536, + "width": 1024 + } + }, + "modes": [ + "txt2img", + "img2img" + ], + "supports_seed": true + }, + "format": "external_api", + "key": "external-model", + "name": "OpenAI Image", + "provider_id": "openai", + "type": "external_image_generator", + "provider_model_id": "gpt-image-1", + "hash": "blake3:external" + }, + "num_images": 1, + "type": "openai_image_generation", + "width": 1536, + "quality": "auto", + "background": "auto" + }, + "canvas_provider_image": { + "is_intermediate": true, + "use_cache": false, + "id": "canvas_provider_image", + "type": "iterate" + }, + "canvas_provider_metadata": { + "is_intermediate": true, + "use_cache": true, + "id": "canvas_provider_metadata", + "type": "metadata_from_image" + }, + "canvas_output": { + "is_intermediate": true, + "use_cache": false, + "id": "canvas_output", + "height": 512, + "type": "img_resize", + "width": 640 + } + } + } +} diff --git a/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.test.ts b/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.test.ts index f4f3ea64af3..3fc9bef48dd 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.test.ts @@ -637,6 +637,83 @@ describe('component policies', () => { ); }); + it('validates the external mode being compiled against the modes the model advertises', () => { + const model: GenerateModelConfig = { + ...externalModel, + capabilities: { modes: ['txt2img', 'inpaint'] }, + }; + + expect(getGenerationValidationReasons(model, createSettings(model))).toEqual([]); + expect(getGenerationValidationReasons(model, createSettings(model), 'inpaint')).toEqual([]); + expect(getGenerationValidationReasons(model, createSettings(model), 'img2img')).toContain( + 'External Model does not support image-to-image generation.' + ); + // Modes default to txt2img only when the record omits them. + expect( + getGenerationValidationReasons({ ...externalModel, capabilities: null }, createSettings(externalModel), 'inpaint') + ).toContain('External Model does not support inpainting.'); + }); + + it('accepts provider sizes off the local model grids for external generators', () => { + expect(getGenerationValidationReasons(externalModel, createSettings(externalModel, { width: 1365 }))).toEqual([]); + }); + + it('rejects dimensions above the provider maximum on either axis', () => { + const model: GenerateModelConfig = { + ...externalModel, + capabilities: { modes: ['txt2img'], max_image_size: { width: 2048, height: 1536 } }, + }; + expect(getGenerationValidationReasons(model, createSettings(model, { width: 2048, height: 1536 }))).toEqual([]); + for (const dimensions of [ + { width: 2049, height: 1536 }, + { width: 2048, height: 1537 }, + ]) { + expect(getGenerationValidationReasons(model, createSettings(model, dimensions))).toContain( + 'External Model supports a maximum size of 2048×1536.' + ); + } + }); + + it('counts the canvas init and output against the Seedream total image budget', () => { + const model: GenerateModelConfig = { + ...externalModel, + provider_id: 'seedream', + provider_model_id: 'seedream-4-5-251128', + capabilities: { modes: ['txt2img', 'img2img'], supports_reference_images: true, max_reference_images: 14 }, + }; + const settings = createSettings(model, { + referenceImages: Array.from({ length: 14 }, (_, i) => ({ + id: `reference-${i}`, + isEnabled: true, + config: { + type: 'external_reference_image', + image: { + imageName: `reference-${i}.png`, + width: 512, + height: 512, + original: { image: { image_name: `reference-${i}.png`, width: 512, height: 512 } }, + }, + }, + })), + }); + expect(getGenerationValidationReasons(model, settings, 'txt2img')).toEqual([]); + expect(getGenerationValidationReasons(model, settings, 'img2img')).toEqual([ + 'External Model supports at most 15 images total (reference + init + output), got 16.', + ]); + settings.referenceImages[0]!.isEnabled = false; + expect(getGenerationValidationReasons(model, settings, 'img2img')).toEqual([]); + settings.referenceImages[0]!.isEnabled = true; + // A record that does not take references sends none, so the budget counts none; the references are refused + // on their own terms instead. + const reasons = getGenerationValidationReasons( + { ...model, capabilities: { modes: ['txt2img', 'img2img'], supports_reference_images: false } }, + settings, + 'img2img' + ); + expect(reasons).toContain('Reference Image #1 is not supported by external model.'); + expect(reasons.filter((reason) => reason.includes('images total'))).toEqual([]); + }); + it('renders no component slots for bases without extra requirements', () => { expect(getComponentSectionPolicy(createModel('cogview4'), createSettings(createModel('cogview4'))).slots).toEqual( [] @@ -685,6 +762,86 @@ describe('component policies', () => { expect(result.clearedLabels).toEqual(['LoRAs', 'T5 Encoder', 'CLIP skip', 'CFG rescale']); }); + it('initializes a provider size on model changes while preserving supported and same-model dimensions', () => { + const model: GenerateModelConfig = { + base: 'external', + format: 'external_api', + key: 'gemini', + name: 'Gemini', + provider_id: 'gemini', + type: 'external_image_generator', + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: 'Square', width: 1024 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: 'Landscape', width: 2048 }, + ], + }, + }; + const select = (settings: GenerateSettings) => + getGenerateModelSelectionResult({ currentValues: settings, model, models: [] }).settings; + + expect(select(createSettings(createModel('sd-1'), { height: 512, width: 512 }))).toMatchObject({ + externalImageSize: '1K', + height: 1024, + width: 1024, + }); + expect(select(createSettings(createModel('sdxl'), { height: 576, width: 1024 }))).toMatchObject({ + aspectRatioId: '16:9', + aspectRatioValue: 2048 / 1152, + externalImageSize: '2K', + height: 1152, + width: 2048, + }); + expect( + select(createSettings(createModel('sdxl'), { externalImageSize: 'stale', height: 1152, width: 2048 })) + ).toMatchObject({ + externalImageSize: '2K', + height: 1152, + width: 2048, + }); + expect(select(createSettings(model, { height: 1200, width: 1600 }))).toMatchObject({ + externalImageSize: null, + height: 1200, + width: 1600, + }); + const local = getGenerateModelSelectionResult({ + currentValues: createSettings(model, { externalImageSize: '2K', height: 1152, width: 2048 }), + model: createModel('sdxl'), + models: [], + }).settings; + + expect(local).toMatchObject({ externalImageSize: null, height: 1152, width: 2048 }); + }); + + it('prefers the nearer pixel budget among presets of one nominal aspect ratio', () => { + // Rounded 16:9 presets have slightly different pixel ratios (1.75 vs 1.78), so an exact ratio tie never occurs. + const model: GenerateModelConfig = { + base: 'external', + format: 'external_api', + key: 'gemini', + name: 'Gemini', + provider_id: 'gemini', + type: 'external_image_generator', + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, + { aspect_ratio: '16:9', height: 768, image_size: '1K', label: '16:9 (1K)', width: 1344 }, + { aspect_ratio: '4:3', height: 768, image_size: '1K', label: '4:3 (1K)', width: 1024 }, + ], + }, + }; + + const result = getGenerateModelSelectionResult({ + currentValues: createSettings(createModel('sdxl'), { height: 576, width: 1024 }), + model, + models: [], + }); + + expect(result.settings).toMatchObject({ externalImageSize: '1K', height: 768, width: 1344 }); + }); + it('turns HiDiffusion off when changing to an unsupported model family', () => { const result = getGenerateModelSelectionResult({ currentValues: createSettings(createModel('sd-1'), { hiDiffusionEnabled: true }), @@ -765,14 +922,85 @@ describe('component policies', () => { expect(result.settings.componentSourceModel).toBe(devSource); }); - it('reconciles dimensions to the new model grid when the selected model changes', () => { + it('reconciles dimensions to the new model grid as an adjustment, not a removed setting', () => { const model = createModel('cogview4'); const settings = createSettings(createModel('sdxl'), { height: 520, width: 520 }); const result = getGenerateModelSelectionResult({ currentValues: settings, model, models: [] }); expect(result.settings.width).toBe(512); expect(result.settings.height).toBe(512); - expect(result.clearedLabels).toContain('Dimensions'); + expect(result.clearedLabels).toEqual([]); + expect(result.dimensionsAdjusted).toBe(true); + }); + + it('reports a snap to a provider size as an adjustment, so the switch is not lossy', () => { + const seedream: GenerateModelConfig = { + ...externalModel, + key: 'seedream', + name: 'Seedream 4.5', + provider_id: 'seedream', + capabilities: { + modes: ['txt2img', 'img2img'], + aspect_ratio_sizes: { + '1:1': { width: 2048, height: 2048 }, + '16:9': { width: 2848, height: 1600 }, + }, + }, + }; + const result = getGenerateModelSelectionResult({ + currentValues: createSettings(createModel('sdxl'), { height: 1024, width: 1024 }), + model: seedream, + models: [], + }); + + expect(result.settings).toMatchObject({ height: 2048, width: 2048 }); + expect(result.clearedLabels).toEqual([]); + expect(result.dimensionsAdjusted).toBe(true); + }); + + it('keeps the Canvas frame when switching to a provider with fixed sizes', () => { + const gptImage: GenerateModelConfig = { + ...externalModel, + key: 'gpt-image', + capabilities: { + modes: ['txt2img', 'img2img'], + aspect_ratio_sizes: { '1:1': { width: 1024, height: 1024 }, '3:2': { width: 1536, height: 1024 } }, + }, + }; + const currentValues = createSettings(createModel('sdxl'), { height: 512, width: 640 }); + + // Generate snaps to the provider's size; the Canvas sync would carry that to the frame, so a frame is kept. + expect(getGenerateModelSelectionResult({ currentValues, model: gptImage, models: [] }).settings).toMatchObject({ + height: 1024, + width: 1024, + }); + const canvas = getGenerateModelSelectionResult({ currentValues, isCanvasFrame: true, model: gptImage, models: [] }); + expect(canvas.settings).toMatchObject({ height: 512, width: 640 }); + expect(canvas.dimensionsAdjusted).toBe(false); + }); + + it('reports the grid clamp from an off-grid provider size back to a local model as an adjustment', () => { + const provider: GenerateModelConfig = { ...externalModel, key: 'gemini', provider_id: 'gemini' }; + const result = getGenerateModelSelectionResult({ + currentValues: createSettings(provider, { height: 1024, width: 683 }), + model: createModel('sdxl'), + models: [], + }); + + expect(result.settings.width % 8).toBe(0); + expect(result.settings.height).toBe(1024); + expect(result.clearedLabels).toEqual([]); + expect(result.dimensionsAdjusted).toBe(true); + }); + + it('reports no adjustment when the size already fits the new model', () => { + const result = getGenerateModelSelectionResult({ + currentValues: createSettings(createModel('sd-1'), { height: 1024, width: 1024 }), + model: createModel('sdxl'), + models: [], + }); + + expect(result.dimensionsAdjusted).toBe(false); }); it('keeps selected VAE when applying model defaults without a VAE override', () => { diff --git a/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.ts b/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.ts index f2fe327b4fa..6c39f59426b 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/baseGenerationPolicies.ts @@ -18,6 +18,7 @@ import { import { SEED_MAX } from '@platform/core/seed'; import type { + ExternalGenerationMode, GenerateModelConfig, GenerateReferenceImage, GenerateReferenceImageAsset, @@ -48,7 +49,16 @@ import { isZImageQwen3Encoder, type GenerateComponentFilter, } from './componentCompatibility'; +import { MAX_DIMENSION, MIN_DIMENSION } from './dimensionLimits'; import { DYNAMIC_PROMPTS_DEFAULT_MAX_PROMPTS } from './dynamicPrompts'; +import { + getExternalAspectRatioSizes, + getExternalReferenceImages, + getExternalResolutionPresets, + isSeedreamBatchModel, + resolveExternalImageSize, + SEEDREAM_BATCH_MAX_TOTAL_IMAGES, +} from './externalProvider'; import { DEFAULT_PID_STEPS, getIsPidActive, @@ -95,8 +105,6 @@ import { isValidKrea2RebalanceWeights, isWanLoraTargetingMain, KREA2_REBALANCE_WEIGHT_COUNT, - MAX_DIMENSION, - MIN_DIMENSION, normalizeGenerateSettings, } from './settings'; @@ -210,11 +218,12 @@ const getBaseGenerationConfig = ( return getArchitectureGenerationConfig(model.base, model.variant) ?? FALLBACK_GENERATION_CONFIG; }; -/** Use variant-aware grids; return null for undescribed local architectures. External providers use a generic grid. */ +/** Pixel-exact: hosted providers bucket requests to their own size tables, which are not grid-aligned. */ +const EXTERNAL_DIMENSION_GRID = 1; + +/** Use variant-aware grids; return null for undescribed local architectures. External providers have no grid. */ export const getDimensionGrid = (base: string, variant?: unknown): number | null => - base === 'external' - ? FALLBACK_GENERATION_CONFIG.dimensions.grid - : (getArchitectureFeatures(base, variant)?.dimension_grid ?? null); + base === 'external' ? EXTERNAL_DIMENSION_GRID : (getArchitectureFeatures(base, variant)?.dimension_grid ?? null); /** A served architecture row does not imply frontend graph support. */ export const isArchitectureDescribed = (model: Pick): boolean => @@ -237,7 +246,7 @@ export const getGenerationDimensions = ( const { grid, optimalSide } = getPidDimensionOverrides( pidMode, model?.base, - config.dimensions.grid, + model?.type === 'external_image_generator' ? EXTERNAL_DIMENSION_GRID : config.dimensions.grid, config.dimensions.optimalSide ); @@ -510,6 +519,14 @@ export const getDefaultGenerateSettings = (model?: GenerateModelConfig): Generat pidDecoderModel: null, gemma2EncoderModel: null, pidSteps: DEFAULT_PID_STEPS, + externalImageSize: null, + openaiQuality: 'auto', + openaiBackground: 'auto', + openaiInputFidelity: null, + geminiTemperature: null, + geminiThinkingLevel: null, + seedreamWatermark: false, + seedreamOptimizePrompt: false, referenceImages: [], refinerModel: null, refinerSteps: DEFAULT_REFINER_STEPS, @@ -611,7 +628,10 @@ export interface ComponentSectionPolicy { export interface GenerateModelSelectionResult { settings: GenerateSettings; + /** Settings the switch turns off or removes, which callers confirm or report. */ clearedLabels: readonly string[]; + /** The size moved to the model's grid or provider sizes. Nothing is lost and it stays on screen, so it is reported. */ + dimensionsAdjusted: boolean; } const TYPE_CLIP_EMBED: ModelTaxonomyType[] = ['clip_embed']; @@ -1462,7 +1482,8 @@ export const getCompatibleReferenceImages = ( const getSettingsWithCompatibleModelSelections = ( settings: GenerateSettings, model: GenerateModelConfig, - models: readonly ReferenceModelCandidate[] + models: readonly ReferenceModelCandidate[], + isCanvasFrame: boolean ): GenerateModelSelectionResult => { const nextSettings: GenerateSettings = { ...settings, modelKey: model.key }; const clearedLabels: string[] = []; @@ -1476,7 +1497,40 @@ const getSettingsWithCompatibleModelSelections = ( nextSettings.height = height; nextSettings.aspectRatioId = deriveAspectRatioId(width, height); nextSettings.aspectRatioValue = height > 0 ? width / height : 1; - addClearedLabel(clearedLabels, 'Dimensions'); + } + + if (settings.modelKey !== model.key) { + const sizes = getExternalResolutionPresets(model); + const supportedSizes = sizes.length > 0 ? sizes : getExternalAspectRatioSizes(model).map(([, size]) => size); + const matchesSize = supportedSizes.some((size) => size.width === width && size.height === height); + + // The Canvas frame is kept: the Canvas asks a provider for its own size nearest the frame and restores the result. + if (!matchesSize && supportedSizes.length > 0 && !isCanvasFrame) { + // Keep the frame's shape when possible, then its pixel budget within that shape. + const distance = (size: { width: number; height: number }) => ({ + area: Math.abs(Math.log((size.width * size.height) / (width * height))), + ratio: Math.abs(Math.log(size.width / size.height / (width / height))), + }); + // Rounded presets of one nominal ratio differ by ~0.02 in log ratio; distinct ratios by 0.1 or more. + const RATIO_TOLERANCE = 0.05; + const selected = supportedSizes.reduce((best, size) => { + const candidateDistance = distance(size); + const bestDistance = distance(best); + const sameRatio = Math.abs(candidateDistance.ratio - bestDistance.ratio) < RATIO_TOLERANCE; + + return (!sameRatio && candidateDistance.ratio < bestDistance.ratio) || + (sameRatio && candidateDistance.area < bestDistance.area) + ? size + : best; + }); + + nextSettings.width = selected.width; + nextSettings.height = selected.height; + nextSettings.aspectRatioId = deriveAspectRatioId(selected.width, selected.height); + nextSettings.aspectRatioValue = selected.width / selected.height; + } + + nextSettings.externalImageSize = resolveExternalImageSize(nextSettings, model); } if (compatibleLoras.length !== settings.loras.length) { @@ -1577,23 +1631,30 @@ const getSettingsWithCompatibleModelSelections = ( } } - return { settings: nextSettings, clearedLabels }; + return { + clearedLabels, + dimensionsAdjusted: nextSettings.width !== settings.width || nextSettings.height !== settings.height, + settings: nextSettings, + }; }; /** Normalize, reconcile, and apply component policy before persisting model selection. */ export const getGenerateModelSelectionResult = ({ currentValues, + isCanvasFrame = false, model, models, }: { currentValues: unknown; + /** The width and height are the Canvas frame (a Canvas source), which the Canvas sync resizes to follow them. */ + isCanvasFrame?: boolean; model: GenerateModelConfig; models: readonly ModelConfig[]; }): GenerateModelSelectionResult => { const currentSettings = isGenerateSettings(currentValues) ? currentValues : (normalizeGenerateSettings(currentValues) ?? getDefaultGenerateSettings(model)); - const result = getSettingsWithCompatibleModelSelections(currentSettings, model, models); + const result = getSettingsWithCompatibleModelSelections(currentSettings, model, models, isCanvasFrame); const componentSourceModel = getAutoFlux2ComponentSourceModel(model, result.settings, models); if (componentSourceModel === undefined || componentSourceModel?.key === result.settings.componentSourceModel?.key) { @@ -1671,6 +1732,11 @@ const getDimensionValidationReasons = (model: GenerateModelConfig, settings: Gen reasons.push(`Generate height must be a multiple of ${dimensions.grid}.`); } + const maxSize = model.type === 'external_image_generator' ? model.capabilities?.max_image_size : null; + if (maxSize && (settings.width > maxSize.width || settings.height > maxSize.height)) { + reasons.push(`${model.name} supports a maximum size of ${maxSize.width}×${maxSize.height}.`); + } + return reasons; }; @@ -1797,7 +1863,24 @@ const getModelFamilyValidationReasons = (model: MainModelConfig, settings: Gener return reasons; }; -export const getGenerationValidationReasons = (model: GenerateModelConfig, settings: GenerateSettings): string[] => { +const EXTERNAL_MODE_LABELS: Record = { + img2img: 'image-to-image generation', + inpaint: 'inpainting', + txt2img: 'text-to-image generation', +}; + +/** Why an external model cannot run `mode`, or null when it advertises it. */ +export const getExternalModeReason = (model: GenerateModelConfig, mode: ExternalGenerationMode): string | null => + model.type === 'external_image_generator' && !(model.capabilities?.modes ?? ['txt2img']).includes(mode) + ? `${model.name} does not support ${EXTERNAL_MODE_LABELS[mode]}.` + : null; + +/** `mode` is the external generation mode being compiled; local models are validated the same way for every mode. */ +export const getGenerationValidationReasons = ( + model: GenerateModelConfig, + settings: GenerateSettings, + mode: ExternalGenerationMode = 'txt2img' +): string[] => { // All compilers share this validator; check capabilities before model-specific policy. if (!hasArchitectureCapabilities()) { // Keep core independent of the load-state store; failures direct callers to retry. @@ -1822,14 +1905,27 @@ export const getGenerationValidationReasons = (model: GenerateModelConfig, setti ]; if (model.type === 'external_image_generator') { - if (model.capabilities?.modes && !model.capabilities.modes.includes('txt2img')) { - reasons.push(`${model.name} does not support text-to-image generation.`); + const modeReason = getExternalModeReason(model, mode); + + if (modeReason) { + reasons.push(modeReason); } if (!getExternalProviderNodeType(model.provider_id)) { reasons.push(`No invocation node registered for external provider '${model.provider_id ?? ''}'.`); } + // Every request emits one output, and the Canvas edit modes add the composite as one init image. + if (isSeedreamBatchModel(model)) { + const totalImages = getExternalReferenceImages(settings, model).length + (mode === 'txt2img' ? 0 : 1) + 1; + + if (totalImages > SEEDREAM_BATCH_MAX_TOTAL_IMAGES) { + reasons.push( + `${model.name} supports at most ${SEEDREAM_BATCH_MAX_TOTAL_IMAGES} images total (reference + init + output), got ${totalImages}.` + ); + } + } + return reasons; } diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.test.ts index 9602f44b41c..61c732b40a6 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.test.ts @@ -10,6 +10,7 @@ const BBOX: Rect = { height: 100, width: 100, x: 0, y: 0 }; const input = (overrides: Partial = {}): CanvasModeInput => ({ bbox: BBOX, bboxFullyCovered: false, + bboxHasContent: false, contentBounds: null, hasActiveInpaintMask: false, ...overrides, diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.ts index 07ed4dc0e62..e1d176dd717 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasMode.ts @@ -13,6 +13,11 @@ export interface CanvasModeInput { contentBounds: Rect | null; /** Opaque coverage is alpha-scanned and contains no holes. */ bboxFullyCovered: boolean; + /** + * Some composited pixel in the bbox is not fully transparent. Local modes ignore it: content whose bounds overlap + * the frame keeps it an outpaint, as before. A hosted model's strategy reads it instead. + */ + bboxHasContent: boolean; /** True when an active mask contains inpaint content. */ hasActiveInpaintMask: boolean; } diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.test.ts index b29d7bf78b3..d8b850239f9 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.test.ts @@ -9,8 +9,37 @@ seedArchitectureCapabilities(); const sd1 = { base: 'sd-1', type: 'main' } as GenerateModelConfig; const sdxl = { base: 'sdxl', type: 'main' } as GenerateModelConfig; +const external = { base: 'external', type: 'external_image_generator' } as GenerateModelConfig; describe('resolveCanvasProcessingSize', () => { + it('asks a free-form hosted provider for the bbox itself, whatever the scaling method', () => { + const bbox = { height: 300, width: 500 }; + + expect(resolveCanvasProcessingSize(external, 'off', bbox, undefined)).toEqual(bbox); + expect(resolveCanvasProcessingSize(external, 'off', bbox, { height: null, method: 'auto', width: null })).toEqual( + bbox + ); + expect(resolveCanvasProcessingSize(external, 'off', bbox, { height: 2048, method: 'manual', width: 2048 })).toEqual( + bbox + ); + }); + + it('asks a fixed-size hosted provider for one of its sizes, whatever the scaling method', () => { + const fixed = { + ...external, + capabilities: { aspect_ratio_sizes: { '1:1': { height: 1024, width: 1024 } }, modes: ['txt2img'] }, + } as GenerateModelConfig; + + expect( + resolveCanvasProcessingSize( + fixed, + 'off', + { height: 300, width: 500 }, + { height: 64, method: 'manual', width: 64 } + ) + ).toEqual({ height: 1024, width: 1024 }); + }); + it('keeps the bbox on the model grid when scaling is off or unset', () => { expect(resolveCanvasProcessingSize(sd1, 'off', { height: 300, width: 500 }, undefined)).toEqual({ height: 304, diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.ts index a3ea9431b43..8897fe77680 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/canvasProcessingSize.ts @@ -1,12 +1,16 @@ /** * None snaps the bbox to the model grid; auto grows undersized content toward optimal area at its aspect ratio; - * manual uses the selected size. Restore final output to the exact bbox. + * manual uses the selected size. Restore final output to the exact bbox. Hosted providers ignore the scaling method: + * they are asked for one of their own sizes nearest the bbox (or the bbox itself when free-form), and the compiler + * restores their output to the bbox. */ import type { GenerateModelConfig, PidMode } from '@features/generation/core/types'; import { getGenerationDimensions } from '@features/generation/core/baseGenerationPolicies'; -import { clampDimension, MAX_DIMENSION } from '@features/generation/core/settings'; +import { MAX_DIMENSION } from '@features/generation/core/dimensionLimits'; +import { resolveExternalCanvasRequestSize } from '@features/generation/core/externalProvider'; +import { clampDimension } from '@features/generation/core/settings'; import type { CanvasScalingSettings } from './types'; @@ -74,12 +78,15 @@ const growToOptimalArea = (size: CanvasSize, optimal: number, grid: number): Can }; export const resolveCanvasProcessingSize = ( - // Keep the variant: Wan 5B and A14B use different grids. - model: Pick & { variant?: unknown }, + // Keep the variant (Wan 5B and A14B use different grids) and a provider's capabilities (its sizes). + model: GenerateModelConfig, pidMode: PidMode, bbox: CanvasSize, scaling: CanvasScalingSettings | undefined ): CanvasSize => { + if (model.type === 'external_image_generator') { + return resolveExternalCanvasRequestSize(model, bbox); + } const { grid, optimal } = getGenerationDimensions(model, pidMode); switch (scaling?.method) { case 'auto': { diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts index ea3a660c815..0e193d394e3 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts @@ -431,13 +431,318 @@ describe('compileCanvasGraph', () => { ); }); - describe('unsupported models', () => { - it('rejects external image generators for txt2img', () => { - expect(() => compile(externalModel, 'txt2img')).toThrow('does not support canvas generation'); + describe('external image generators', () => { + const editModel: GenerateModelConfig = { + ...externalModel, + capabilities: { modes: ['txt2img', 'img2img', 'inpaint'], supports_seed: true }, + }; + const smallBbox: Rect = { height: 512, width: 640, x: 10, y: 20 }; + // The starter catalog's GPT Image size table. + const gptImage: GenerateModelConfig = { + ...editModel, + capabilities: { + allowed_aspect_ratios: ['1:1', '3:2', '2:3'], + aspect_ratio_sizes: { + '1:1': { height: 1024, width: 1024 }, + '3:2': { height: 1024, width: 1536 }, + '2:3': { height: 1536, width: 1024 }, + }, + modes: ['txt2img', 'img2img'], + supports_seed: true, + }, + provider_model_id: 'gpt-image-1', + }; + + it('writes hosted canvas graphs for backend invocation contract validation', async () => { + const clock = vi.spyOn(Date, 'now').mockReturnValue(0); + const random = vi.spyOn(Math, 'random').mockReturnValue(0.5); + const graphs: Record = {}; + try { + for (const mode of ['txt2img', 'img2img'] as const) { + graphs[mode] = compile({ ...gptImage, hash: 'blake3:external' }, mode, { bbox: smallBbox }).backendGraph; + } + } finally { + clock.mockRestore(); + random.mockRestore(); + } + const fixture = { + _comment: + 'Generated by compileCanvasGraph.test.ts; regenerate with vitest -u. Validated by tests/app/invocations/test_frontend_external_canvas_graphs.py.', + ...graphs, + }; + await expect(`${JSON.stringify(fixture, null, 2)}\n`).toMatchFileSnapshot( + '../__snapshots__/externalCanvasGraphs.json' + ); + }); + + it.each(['canvas', 'gallery'] as const)('restores provider output to the exact bbox for %s', (destination) => { + const { backendGraph, mode } = compile(externalModel, 'txt2img', { + bbox: smallBbox, + compositeImageName: null, + destination, + scaling: { height: null, method: 'auto', width: null }, + }); + const output = backendGraph.nodes.canvas_processing_output; + + expect(mode).toBe('txt2img'); + expect(output).toMatchObject({ + height: 512, + is_intermediate: true, + mode: 'txt2img', + type: 'openai_image_generation', + width: 640, + }); + expect(output).not.toHaveProperty('init_image'); + expect(output).not.toHaveProperty('mask_image'); + expect(getEdge(backendGraph, 'canvas_processing_output', 'prompt')?.source.node_id).toBe('positive_prompt'); + expect(getEdge(backendGraph, 'canvas_processing_output', 'seed')?.source.node_id).toBe('seed'); + expect(backendGraph.nodes.canvas_i2l).toBeUndefined(); + expect(backendGraph.nodes.canvas_output).toMatchObject({ + type: 'img_resize', + height: 512, + width: 640, + is_intermediate: destination === 'canvas', + }); + expect(backendGraph.nodes.canvas_provider_image.type).toBe('iterate'); + expect(getEdge(backendGraph, 'canvas_provider_image', 'collection')?.source).toEqual({ + node_id: 'canvas_processing_output', + field: 'collection', + }); + expect(getEdge(backendGraph, 'canvas_output', 'image')?.source).toEqual({ + node_id: 'canvas_provider_image', + field: 'item', + }); + expect(backendGraph.nodes.canvas_provider_metadata.type).toBe('metadata_from_image'); + expect(getEdge(backendGraph, 'canvas_provider_metadata', 'image')?.source).toEqual({ + node_id: 'canvas_provider_image', + field: 'item', + }); + expect(getEdge(backendGraph, 'canvas_output', 'metadata')?.source.node_id).toBe('canvas_provider_metadata'); + expect(getEdge(backendGraph, 'canvas_processing_output', 'metadata')).toBeDefined(); + expect(getNodeByType(backendGraph, 'core_metadata')).toMatchObject({ height: 512, width: 640 }); + }); + + it('asks a fixed-size provider for its nearest-ratio size and center-crops a text-to-image result to the bbox', () => { + const { backendGraph } = compile(gptImage, 'txt2img', { bbox: smallBbox, compositeImageName: null }); + + // 640×512 is nearest 3:2 among the table's ratios; the provider receives that exact entry. + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ height: 1024, width: 1536 }); + // 1536×1024 scaled to cover 640×512 is 768×512, then 64 px come off each side. + expect(backendGraph.nodes.canvas_provider_cover).toMatchObject({ + height: 512, + is_intermediate: true, + type: 'img_resize', + width: 768, + }); + expect(backendGraph.nodes.canvas_output).toMatchObject({ + height: 512, + type: 'img_crop', + width: 640, + x: 64, + y: 0, + }); + expect(getEdge(backendGraph, 'canvas_provider_cover', 'image')?.source).toEqual({ + node_id: 'canvas_provider_image', + field: 'item', + }); + expect(getEdge(backendGraph, 'canvas_output', 'image')?.source.node_id).toBe('canvas_provider_cover'); + // Each image keeps the metadata of the provider image it was cropped from. + expect(getEdge(backendGraph, 'canvas_provider_metadata', 'image')?.source.node_id).toBe('canvas_provider_image'); + expect(getEdge(backendGraph, 'canvas_output', 'metadata')?.source.node_id).toBe('canvas_provider_metadata'); + expect(getNodeByType(backendGraph, 'core_metadata')).toMatchObject({ height: 512, width: 640 }); + }); + + it('crops a portrait frame vertically', () => { + const { backendGraph } = compile(gptImage, 'txt2img', { + bbox: { height: 1000, width: 700, x: 0, y: 0 }, + compositeImageName: null, + }); + + // 1024×1536 scaled to cover 700×1000 is 700×1050, then 25 px come off the top and bottom. + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ height: 1536, width: 1024 }); + expect(backendGraph.nodes.canvas_provider_cover).toMatchObject({ height: 1050, width: 700 }); + expect(backendGraph.nodes.canvas_output).toMatchObject({ + height: 1000, + type: 'img_crop', + width: 700, + x: 0, + y: 25, + }); + }); + + it('resizes without cropping when the provider size has the bbox’s shape', () => { + const { backendGraph } = compile(gptImage, 'txt2img', { + bbox: { height: 512, width: 512, x: 0, y: 0 }, + compositeImageName: null, + }); + + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ height: 1024, width: 1024 }); + expect(backendGraph.nodes.canvas_provider_cover).toBeUndefined(); + expect(backendGraph.nodes.canvas_output).toMatchObject({ height: 512, type: 'img_resize', width: 512 }); + expect(getEdge(backendGraph, 'canvas_output', 'image')?.source.node_id).toBe('canvas_provider_image'); + }); + + it('resizes an image-to-image init to the provider size and its result plainly back, without cropping', () => { + const { backendGraph } = compile(gptImage, 'img2img', { bbox: smallBbox, compositeImageName: 'bbox.png' }); + + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ height: 1024, width: 1536 }); + expect(backendGraph.nodes.canvas_processing_output).not.toHaveProperty('init_image'); + expect(backendGraph.nodes.canvas_resize_image_to_request).toMatchObject({ + height: 1024, + image: { image_name: 'bbox.png' }, + type: 'img_resize', + width: 1536, + }); + expect(getEdge(backendGraph, 'canvas_processing_output', 'init_image')?.source.node_id).toBe( + 'canvas_resize_image_to_request' + ); + expect(backendGraph.nodes.canvas_provider_cover).toBeUndefined(); + expect(backendGraph.nodes.canvas_output).toMatchObject({ height: 512, type: 'img_resize', width: 640 }); + }); + + it('sends the smallest named size covering the frame, so a 2000 px square gets 2K', () => { + const gemini: GenerateModelConfig = { + ...externalModel, + capabilities: { + allowed_aspect_ratios: ['1:1', '16:9'], + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + { aspect_ratio: '1:1', height: 4096, image_size: '4K', label: '1:1 (4K)', width: 4096 }, + ], + }, + provider_id: 'gemini', + }; + const { backendGraph } = compile(gemini, 'txt2img', { + bbox: { height: 2000, width: 2000, x: 0, y: 0 }, + compositeImageName: null, + }); + + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ + height: 2048, + image_size: '2K', + type: 'gemini_image_generation', + width: 2048, + }); + expect(backendGraph.nodes.canvas_output).toMatchObject({ height: 2000, type: 'img_resize', width: 2000 }); + expect(getNodeByType(backendGraph, 'core_metadata')).toMatchObject({ + height: 2000, + image_size: '2K', + width: 2000, + }); + }); + + it('sends the composited bbox as the init image for img2img when the model advertises it', () => { + const { backendGraph } = compile({ ...editModel, capabilities: { modes: ['img2img'] } }, 'img2img', { + compositeImageName: 'bbox.png', + }); + + expect(backendGraph.nodes.canvas_processing_output).toMatchObject({ + height: 1024, + init_image: { image_name: 'bbox.png' }, + is_intermediate: true, + mode: 'img2img', + width: 768, + }); + expect(backendGraph.nodes.canvas_processing_output).not.toHaveProperty('mask_image'); + // The edited whole image is restored to the bbox like a txt2img result. + expect(backendGraph.nodes.canvas_output).toMatchObject({ height: 1024, type: 'img_resize', width: 768 }); + expect(getEdge(backendGraph, 'canvas_output', 'image')?.source).toEqual({ + node_id: 'canvas_provider_image', + field: 'item', + }); + expect(getNodeByType(backendGraph, 'core_metadata')).toMatchObject({ height: 1024, width: 768 }); + // No local denoise path is grafted: the provider edits the whole image. + expect(backendGraph.nodes.denoise_latents).toBeUndefined(); + expect(backendGraph.nodes.create_gradient_mask).toBeUndefined(); }); - it('rejects external image generators for img2img', () => { - expect(() => compile(externalModel, 'img2img')).toThrow('does not support canvas generation'); + it('rejects provider inpaint until Canvas mask semantics and compositing are supported', () => { + expect(() => + compile(editModel, 'inpaint', { + compositeImageName: 'bbox.png', + maskImageName: 'mask.png', + }) + ).toThrow('OpenAI Image does not support inpainting on the canvas yet. Remove or disable the inpaint mask.'); + }); + + it('requires the composite for img2img', () => { + expect(() => compile(editModel, 'img2img', { compositeImageName: null })).toThrow( + 'Canvas generation requires a composited source image.' + ); + }); + + it('names raster content in the frame as the cause for a text-only model, in every edit mode', () => { + for (const mode of ['img2img', 'outpaint', 'inpaint'] as const) { + expect(() => compile(externalModel, mode, { maskImageName: mode === 'inpaint' ? 'mask.png' : null })).toThrow( + 'OpenAI Image cannot use image content in the generation frame. Hide or move raster content out of the frame to generate.' + ); + } + }); + + it('rejects text-to-image for a model that does not advertise it, naming the mode', () => { + expect(() => + compile({ ...externalModel, capabilities: { modes: ['img2img'] } }, 'txt2img', { compositeImageName: null }) + ).toThrow('OpenAI Image does not support text-to-image generation.'); + }); + + it('rejects outpaint even for a model that advertises every provider mode', () => { + expect(() => compile(editModel, 'outpaint', { maskImageName: null })).toThrow( + 'OpenAI Image does not support outpainting on the canvas.' + ); + }); + + it('rejects control layers and regional guidance by name instead of failing inside a graft', () => { + const layer: ControlLayerGraphInput = { + beginEndStepPct: [0, 1], + controlMode: 'balanced', + id: 'control', + imageName: 'control.png', + kind: 'controlnet', + model: { base: 'sd-1', key: 'canny', name: 'Canny', type: 'controlnet' }, + weight: 1, + }; + + expect(() => compile(editModel, 'txt2img', { compositeImageName: null, controlLayers: [layer] })).toThrow( + 'OpenAI Image does not support control layers.' + ); + expect(() => + compile(editModel, 'txt2img', { + compositeImageName: null, + regionalGuidance: [ + { + autoNegative: false, + id: 'r1', + maskImageName: 'r1-mask.png', + negativePrompt: null, + positivePrompt: 'a cat', + referenceImages: [], + }, + ], + }) + ).toThrow('OpenAI Image does not support regional guidance.'); + }); + + it('ignores the denoising strength: the provider edits at its own, so none is validated, sent or recorded', () => { + const { backendGraph } = compile(editModel, 'img2img', { compositeImageName: 'bbox.png', strength: 0 }); + const metadata = getNodeByType(backendGraph, 'core_metadata'); + + expect(backendGraph.nodes.canvas_output).not.toHaveProperty('strength'); + expect(backendGraph.nodes.canvas_output).not.toHaveProperty('denoising_start'); + expect(metadata).not.toHaveProperty('strength'); + expect(metadata).not.toHaveProperty('generation_mode'); + }); + + it('asks a free-form provider for the bbox itself even when the render size is set to auto or a custom size', () => { + const custom = compile(editModel, 'img2img', { + bbox: smallBbox, + compositeImageName: 'bbox.png', + scaling: { height: 2048, method: 'manual', width: 2048 }, + }); + + expect(custom.backendGraph.nodes.canvas_output).toMatchObject({ height: 512, width: 640 }); + expect(custom.backendGraph.nodes.canvas_processing_output).toMatchObject({ height: 512, width: 640 }); }); }); @@ -854,12 +1159,6 @@ describe('compileCanvasGraph — outpaint per base', () => { expect(backendGraph.nodes.create_gradient_mask?.minimum_denoise).toBe(0.3); expect(backendGraph.nodes.expand_mask?.fade_size_px).toBe(24); }); - - it('rejects an external model for every image mode', () => { - expect(() => compile(externalModel, 'outpaint', { maskImageName: null })).toThrow( - 'does not support canvas generation' - ); - }); }); // Exercise control grafting through the real compiler entry point. diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.ts index 1a8f66b1128..273b4f84483 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.ts @@ -1,6 +1,7 @@ /** - * Pure canvas compiler over uploaded inputs. Resize through the processing grid and restore the bbox; reject - * external models and unsupported encode modes. + * Pure canvas compiler over uploaded inputs. Resize through the processing grid and restore the bbox; hosted + * providers are asked for one of their own sizes, take the bbox composite for edits, and have their result restored + * to the bbox; reject unsupported modes. */ import type { SupportedGenerateBase } from '@features/generation/core/baseGenerationPolicies'; @@ -8,11 +9,17 @@ import type { BackendGraphContract, BackendInvocationContract } from '@features/ import type { ComponentModelConfig, GenerateModelConfig, GenerateSettings } from '@features/generation/core/types'; import { getGenerationValidationReasons } from '@features/generation/core/baseGenerationPolicies'; -import { findVaeSource, GRAPH_BUILDERS } from '@features/generation/core/graph'; +import { + describeExternalCanvasRefusal, + isExternalGenerationMode, + isExternalTextOnlyCanvasModel, +} from '@features/generation/core/externalProvider'; +import { buildExternalGraph, findVaeSource, GRAPH_BUILDERS } from '@features/generation/core/graph'; import { addEdge, addNode, addOutputPostprocessors, + renameNode, replaceCanvasOutput, toGraphContract, } from '@features/generation/core/graphBuilder'; @@ -112,8 +119,21 @@ const getCanvasValidationReasons = (input: CompileCanvasGraphInput, processingSi const reasons: string[] = []; if (model.type === 'external_image_generator') { - reasons.push(`${model.name} does not support canvas generation.`); - return reasons; + if (mode !== 'txt2img' && isExternalTextOnlyCanvasModel(model)) { + // Every edit mode comes from raster content in the frame, which names the cause better than the mode does. + reasons.push(describeExternalCanvasRefusal(model, { kind: 'raster_content' })); + } else if (mode === 'inpaint') { + // The Canvas keep-mask does not match provider edit-mask contracts, and there is no masked compositing tail. + reasons.push(`${model.name} does not support inpainting on the canvas yet. Remove or disable the inpaint mask.`); + } else if (mode === 'outpaint') { + reasons.push(`${model.name} does not support outpainting on the canvas.`); + } + if (input.controlLayers && input.controlLayers.length > 0) { + reasons.push(`${model.name} does not support control layers.`); + } + if (input.regionalGuidance && input.regionalGuidance.length > 0) { + reasons.push(`${model.name} does not support regional guidance.`); + } } if (getIsPidActive(settings.pidMode, model.base)) { @@ -147,7 +167,8 @@ const getCanvasValidationReasons = (input: CompileCanvasGraphInput, processingSi reasons.push('Canvas generation requires a composited source image.'); } - if (!Number.isFinite(strength) || strength <= 0 || strength > 1) { + // A provider edits at its own strength; only the local denoise reads this one. + if (model.type !== 'external_image_generator' && (!Number.isFinite(strength) || strength <= 0 || strength > 1)) { reasons.push('Canvas denoising strength must be greater than 0 and at most 1.'); } } @@ -234,6 +255,69 @@ const resizeCanvasOutputToBbox = ( addEdge(graph, retired, 'image', output, 'image'); }; +/** + * Hosted nodes return an image collection; restore each image to the bbox with its own provider metadata. A + * text-to-image result at another aspect ratio is scaled to cover the bbox and center-cropped, never stretched. + * Scaling before cropping keeps the crop centered when a provider answers at another scale of the requested shape + * (named sizes such as Gemini's `2K`). An edit result is resized plainly, undoing its init image's resize. + */ +const restoreExternalCanvasOutputToBbox = ( + graph: BackendGraphContract, + input: Pick & { mode: 'txt2img' | 'img2img' }, + requestSize: CanvasSize +): void => { + const { bbox, destination, mode } = input; + const provider = renameNode(graph, 'canvas_output', 'canvas_processing_output'); + provider.is_intermediate = true; + const image = addNode(graph, { id: 'canvas_provider_image', type: 'iterate', use_cache: false }); + const metadata = addNode(graph, { id: 'canvas_provider_metadata', type: 'metadata_from_image' }); + const output = { id: 'canvas_output', is_intermediate: destination === 'canvas', use_cache: false }; + // Keep core metadata on the provider so its output adds request identity, defaults and actual seed to it. + addEdge(graph, provider, 'collection', image, 'collection'); + addEdge(graph, image, 'item', metadata, 'image'); + + if (mode === 'img2img' || requestSize.width * bbox.height === requestSize.height * bbox.width) { + const resize = addNode(graph, { ...output, height: bbox.height, type: 'img_resize', width: bbox.width }); + addEdge(graph, image, 'item', resize, 'image'); + addEdge(graph, metadata, 'metadata', resize, 'metadata'); + return; + } + + const scale = Math.max(bbox.width / requestSize.width, bbox.height / requestSize.height); + const cover = addNode(graph, { + height: Math.max(bbox.height, Math.round(requestSize.height * scale)), + id: 'canvas_provider_cover', + is_intermediate: true, + type: 'img_resize', + use_cache: false, + width: Math.max(bbox.width, Math.round(requestSize.width * scale)), + }); + const crop = addNode(graph, { + ...output, + height: bbox.height, + type: 'img_crop', + width: bbox.width, + x: Math.floor((cover.width - bbox.width) / 2), + y: Math.floor((cover.height - bbox.height) / 2), + }); + addEdge(graph, image, 'item', cover, 'image'); + addEdge(graph, cover, 'image', crop, 'image'); + addEdge(graph, metadata, 'metadata', crop, 'metadata'); +}; + +/** Feeds the provider the composite at the request size, as the backend would have resized it to its own bucket. */ +const resizeExternalInitImage = (graph: BackendGraphContract, imageName: string, requestSize: CanvasSize): void => { + const provider = graph.nodes.canvas_output as BackendInvocationContract; + delete provider.init_image; + addEdge( + graph, + addInputResizeNode(graph, 'canvas_resize_image_to_request', imageName, requestSize), + 'image', + provider, + 'init_image' + ); +}; + /** Keeps saved-image metadata aligned to the bbox footprint, not the internal processing size. */ const setCanvasMetadataDimensions = (graph: BackendGraphContract, bbox: CompileCanvasGraphInput['bbox']): void => { const metadata = findMetadataNode(graph); @@ -288,9 +372,6 @@ const addInputResizeNode = ( /** Resolves the encode node type for a supported base (throws otherwise). */ const requireI2lType = (model: GenerateModelConfig): string => { - if (model.type === 'external_image_generator') { - throw new Error(`${model.name} does not support canvas generation.`); - } const i2lType = CANVAS_I2L_NODE_TYPES[model.base as SupportedGenerateBase]; if (!i2lType) { throw new Error(`Canvas generation is not supported for ${model.name}.`); @@ -740,25 +821,57 @@ export const compileCanvasGraph = (input: CompileCanvasGraphInput): CompiledCanv const validationReasons = [ ...getCanvasValidationReasons(input, processingSize), - ...getGenerationValidationReasons(model, settings), + ...getGenerationValidationReasons(model, settings, isExternalGenerationMode(mode) ? mode : 'txt2img'), ]; if (validationReasons.length > 0) { throw new Error(validationReasons[0]); } - const builder = GRAPH_BUILDERS[model.base as SupportedGenerateBase]; - - if (!builder || model.type === 'external_image_generator') { - throw new Error(`${model.name} does not support canvas generation.`); - } - // Canvas outputs are intermediate; gallery outputs are durable. const outputIsIntermediate = destination === 'canvas'; const runtimeProjectSettings = { ...projectSettings, randDevice: input.randDevice ?? (projectSettings.useCpuNoise ? 'cpu' : 'cuda'), }; + + if (model.type === 'external_image_generator') { + // Validation above already named the mode; this narrows the provider inputs to the two modes Canvas compiles. + if (mode !== 'txt2img' && mode !== 'img2img') { + throw new Error(`${model.name} does not support ${mode}ing on the canvas.`); + } + + // The provider is asked for the processing size, one of its own sizes, and the returned whole image is restored + // to the exact bbox footprint before staging or saving. No local denoising strength applies. + const backendGraph = buildExternalGraph( + settings, + model, + outputIsIntermediate, + runtimeProjectSettings, + mode === 'txt2img' ? { mode } : { initImageName: compositeImageName as string, mode } + ); + if (mode === 'img2img' && !sizesMatch(processingSize, bbox)) { + resizeExternalInitImage(backendGraph, compositeImageName as string, processingSize); + } + restoreExternalCanvasOutputToBbox(backendGraph, { bbox, destination, mode }, processingSize); + setCanvasMetadataDimensions(backendGraph, bbox); + + return { + backendGraph, + graph: toGraphContract(backendGraph, `${model.name} ${mode}`), + mode, + negativePromptNodeId: 'negative_prompt', + positivePromptNodeId: 'positive_prompt', + seedNodeId: 'seed', + }; + } + + const builder = GRAPH_BUILDERS[model.base as SupportedGenerateBase]; + + if (!builder) { + throw new Error(`Canvas generation is not supported for ${model.name}.`); + } + const backendGraph = builder(settings, model, outputIsIntermediate, runtimeProjectSettings); const { compositing } = input; diff --git a/invokeai/frontend/webv2/src/features/generation/core/dimensionLimits.ts b/invokeai/frontend/webv2/src/features/generation/core/dimensionLimits.ts new file mode 100644 index 00000000000..ada0b2f0394 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/dimensionLimits.ts @@ -0,0 +1,3 @@ +/** The generation size limits every model and the Canvas share; a leaf so provider policy can read them too. */ +export const MIN_DIMENSION = 64; +export const MAX_DIMENSION = 4096; diff --git a/invokeai/frontend/webv2/src/features/generation/core/externalProvider.test.ts b/invokeai/frontend/webv2/src/features/generation/core/externalProvider.test.ts new file mode 100644 index 00000000000..38deea345dc --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/externalProvider.test.ts @@ -0,0 +1,258 @@ +import { describe, expect, it } from 'vitest'; + +import type { ExternalImageGeneratorModelConfig, GenerateModelConfig } from './types'; + +import { + findExternalResolutionPreset, + getExternalAspectRatioSizes, + getExternalProviderOptionKeys, + getExternalReferenceImages, + resolveExternalCanvasRequestSize, + resolveExternalImageSize, +} from './externalProvider'; + +const external = ( + providerId: string, + capabilities: ExternalImageGeneratorModelConfig['capabilities'], + providerModelId = providerId === 'openai' ? 'gpt-image-1' : `${providerId}-model` +): GenerateModelConfig => ({ + base: 'external', + capabilities, + format: 'external_api', + key: `${providerId}-model`, + name: providerId, + provider_id: providerId, + provider_model_id: providerModelId, + type: 'external_image_generator', +}); + +const presets = [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, +]; + +describe('external provider options', () => { + it('offers each provider only the options its node declares', () => { + expect(getExternalProviderOptionKeys(external('gemini', { modes: ['txt2img'] }))).toEqual([ + 'geminiTemperature', + 'geminiThinkingLevel', + ]); + expect(getExternalProviderOptionKeys(external('seedream', { modes: ['txt2img'] }))).toEqual([ + 'seedreamWatermark', + 'seedreamOptimizePrompt', + ]); + // Alibaba Cloud and Atlas Cloud nodes add nothing beyond the shared fields. + expect(getExternalProviderOptionKeys(external('alibabacloud', { modes: ['txt2img'] }))).toEqual([]); + expect(getExternalProviderOptionKeys({ base: 'sdxl', key: 'sdxl', name: 'SDXL', type: 'main' })).toEqual([]); + expect(getExternalProviderOptionKeys(undefined)).toEqual([]); + }); + + it('offers OpenAI input fidelity only to models that take image inputs', () => { + expect(getExternalProviderOptionKeys(external('openai', { modes: ['txt2img'] }))).toEqual([ + 'openaiQuality', + 'openaiBackground', + ]); + expect(getExternalProviderOptionKeys(external('openai', { modes: ['txt2img', 'img2img'] }))).toContain( + 'openaiInputFidelity' + ); + expect( + getExternalProviderOptionKeys(external('openai', { modes: ['txt2img'], supports_reference_images: true })) + ).toContain('openaiInputFidelity'); + }); + + it('offers no OpenAI options to DALL·E, which the provider sends none of', () => { + for (const modelId of ['dall-e-2', 'dall-e-3']) { + expect(getExternalProviderOptionKeys(external('openai', { modes: ['txt2img', 'img2img'] }, modelId))).toEqual([]); + } + expect(getExternalProviderOptionKeys(external('openai', { modes: ['txt2img'] }, 'gpt-image-2'))).toEqual([ + 'openaiQuality', + 'openaiBackground', + ]); + }); + + it('sends enabled reference images with an image, and none to a model that takes no references', () => { + const image = { + height: 64, + imageName: 'reference.png', + original: { image: { height: 64, image_name: 'reference.png', width: 64 } }, + width: 64, + }; + const settings = { + referenceImages: [ + { config: { image, type: 'external_reference_image' as const }, id: 'on', isEnabled: true }, + { config: { image, type: 'external_reference_image' as const }, id: 'off', isEnabled: false }, + { config: { image: null, type: 'external_reference_image' as const }, id: 'empty', isEnabled: true }, + ], + }; + + expect( + getExternalReferenceImages( + settings, + external('seedream', { modes: ['txt2img'], supports_reference_images: true }) + ) + ).toEqual([image]); + expect(getExternalReferenceImages(settings, external('seedream', { modes: ['txt2img'] }))).toEqual([]); + }); + + it('resolves a named size only from the selected model’s own presets, at the preset’s exact size', () => { + const gemini = external('gemini', { modes: ['txt2img'], resolution_presets: presets }); + + expect(resolveExternalImageSize({ externalImageSize: '2K', height: 1152, width: 2048 }, gemini)).toBe('2K'); + expect(resolveExternalImageSize({ externalImageSize: '4K', height: 1152, width: 2048 }, gemini)).toBe('2K'); + expect(resolveExternalImageSize({ externalImageSize: null, height: 1152, width: 2048 }, gemini)).toBe('2K'); + // A size typed by hand after picking a preset is custom: the provider buckets the dimensions instead. + expect(resolveExternalImageSize({ externalImageSize: '2K', height: 1152, width: 2000 }, gemini)).toBeNull(); + expect( + resolveExternalImageSize( + { externalImageSize: '2K', height: 1152, width: 2048 }, + external('openai', { modes: ['txt2img'] }) + ) + ).toBeNull(); + }); + + it('recognizes typed preset dimensions while keeping unsupported dimensions custom', () => { + const gemini = external('gemini', { modes: ['txt2img'], resolution_presets: presets }); + + expect(findExternalResolutionPreset({ externalImageSize: '2K', height: 1152, width: 2048 }, gemini)?.label).toBe( + '16:9 (2K)' + ); + expect(findExternalResolutionPreset({ externalImageSize: '2K', height: 1152, width: 2000 }, gemini)).toBeNull(); + expect( + findExternalResolutionPreset({ externalImageSize: null, height: 1024, width: 1024 }, gemini)?.image_size + ).toBe('1K'); + }); + + it('breaks a tie between presets of one size by the stored name, else the first', () => { + const duplicate = external('gemini', { + modes: ['txt2img'], + resolution_presets: [ + ...presets, + { aspect_ratio: '1:1', height: 1024, image_size: 'HD', label: '1:1 (HD)', width: 1024 }, + ], + }); + + expect(findExternalResolutionPreset({ externalImageSize: 'HD', height: 1024, width: 1024 }, duplicate)?.label).toBe( + '1:1 (HD)' + ); + expect(resolveExternalImageSize({ externalImageSize: 'HD', height: 1024, width: 1024 }, duplicate)).toBe('HD'); + expect(resolveExternalImageSize({ externalImageSize: null, height: 1024, width: 1024 }, duplicate)).toBe('1K'); + }); + + it('lists fixed ratio sizes in the provider’s order', () => { + const openai = external('openai', { + aspect_ratio_sizes: { '1:1': { height: 1024, width: 1024 }, '3:2': { height: 1024, width: 1536 } }, + modes: ['txt2img'], + }); + + expect(getExternalAspectRatioSizes(openai)).toEqual([ + ['1:1', { height: 1024, width: 1024 }], + ['3:2', { height: 1024, width: 1536 }], + ]); + expect(getExternalAspectRatioSizes(external('gemini', { modes: ['txt2img'] }))).toEqual([]); + }); +}); + +describe('Canvas request size', () => { + // The starter catalog's GPT Image and Seedream 2K size tables. + const gptImage = external('openai', { + aspect_ratio_sizes: { + '1:1': { height: 1024, width: 1024 }, + '3:2': { height: 1024, width: 1536 }, + '2:3': { height: 1536, width: 1024 }, + }, + modes: ['txt2img', 'img2img'], + }); + const seedream = external('seedream', { + aspect_ratio_sizes: { + '1:1': { height: 2048, width: 2048 }, + '4:3': { height: 1728, width: 2304 }, + '16:9': { height: 1600, width: 2848 }, + '21:9': { height: 1344, width: 3136 }, + }, + modes: ['txt2img', 'img2img'], + }); + + it('asks a fixed-size provider for its size at the ratio nearest the frame, whatever the frame’s size', () => { + expect(resolveExternalCanvasRequestSize(gptImage, { height: 512, width: 640 })).toEqual({ + height: 1024, + width: 1536, + }); + expect(resolveExternalCanvasRequestSize(gptImage, { height: 500, width: 520 })).toEqual({ + height: 1024, + width: 1024, + }); + expect(resolveExternalCanvasRequestSize(gptImage, { height: 3000, width: 1800 })).toEqual({ + height: 1536, + width: 1024, + }); + expect(resolveExternalCanvasRequestSize(seedream, { height: 1080, width: 1920 })).toEqual({ + height: 1600, + width: 2848, + }); + }); + + it('picks the smallest named size covering the frame at the nearest ratio, else the largest', () => { + const gemini = external('gemini', { + allowed_aspect_ratios: ['1:1', '16:9'], + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '16:9', height: 576, image_size: '1K', label: '16:9 (1K)', width: 1024 }, + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + { aspect_ratio: '1:1', height: 4096, image_size: '4K', label: '1:1 (4K)', width: 4096 }, + ], + }); + + expect(resolveExternalCanvasRequestSize(gemini, { height: 2000, width: 2000 })).toEqual({ + height: 2048, + width: 2048, + }); + expect(resolveExternalCanvasRequestSize(gemini, { height: 512, width: 512 })).toEqual({ + height: 1024, + width: 1024, + }); + expect(resolveExternalCanvasRequestSize(gemini, { height: 5000, width: 5000 })).toEqual({ + height: 4096, + width: 4096, + }); + // The only 16:9 size is too small for this frame, yet the shape wins over a larger size of another shape. + expect(resolveExternalCanvasRequestSize(gemini, { height: 1080, width: 1920 })).toEqual({ + height: 576, + width: 1024, + }); + }); + + it('asks a ratio-limited provider for an exact multiple of the nearest ratio covering the frame, within its maximum', () => { + const ratios = external('custom', { + allowed_aspect_ratios: ['1:1', '21:9', 'wide'], + max_image_size: { height: 1024, width: 2048 }, + modes: ['txt2img'], + }); + + expect(resolveExternalCanvasRequestSize(ratios, { height: 270, width: 630 })).toEqual({ height: 270, width: 630 }); + expect(resolveExternalCanvasRequestSize(ratios, { height: 500, width: 1000 })).toEqual({ + height: 504, + width: 1176, + }); + expect(resolveExternalCanvasRequestSize(ratios, { height: 1500, width: 1500 })).toEqual({ + height: 1024, + width: 1024, + }); + }); + + it('keeps an exact ratio multiple within the generation limit when the provider declares no maximum', () => { + const ratios = external('custom', { allowed_aspect_ratios: ['21:9'], modes: ['txt2img'] }); + + // The covering multiple would be 4116×1764; the largest one within 4096 px is 195 × (21, 9). + expect(resolveExternalCanvasRequestSize(ratios, { height: 1755, width: 4096 })).toEqual({ + height: 1755, + width: 4095, + }); + }); + + it('asks a free-form provider for the frame itself', () => { + expect( + resolveExternalCanvasRequestSize(external('atlascloud', { modes: ['txt2img'] }), { height: 333, width: 777 }) + ).toEqual({ height: 333, width: 777 }); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/core/externalProvider.ts b/invokeai/frontend/webv2/src/features/generation/core/externalProvider.ts new file mode 100644 index 00000000000..a0d95c1eeda --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/core/externalProvider.ts @@ -0,0 +1,346 @@ +/** + * Pure policy for hosted image providers: which provider-specific options a selected external model exposes, the + * named sizes and fixed-size ratios it advertises, and the invocation fields those settings compile to. + */ + +import type { + ExternalGenerationMode, + ExternalImageSize, + ExternalModelCapabilities, + ExternalResolutionPreset, + GeminiThinkingLevel, + GenerateModelConfig, + GenerateReferenceImageAsset, + GenerateSettings, + OpenAIImageBackground, + OpenAIImageQuality, + OpenAIInputFidelity, +} from './types'; + +import { MAX_DIMENSION } from './dimensionLimits'; + +export const OPENAI_IMAGE_QUALITIES: readonly OpenAIImageQuality[] = ['auto', 'high', 'medium', 'low']; +export const OPENAI_IMAGE_BACKGROUNDS: readonly OpenAIImageBackground[] = ['auto', 'transparent', 'opaque']; +export const OPENAI_INPUT_FIDELITIES: readonly OpenAIInputFidelity[] = ['low', 'high']; +export const GEMINI_THINKING_LEVELS: readonly GeminiThinkingLevel[] = ['minimal', 'high']; +/** `GeminiImageGenerationInvocation.temperature` is `ge=0, le=2`; the provider default is 1. */ +export const GEMINI_TEMPERATURE_MIN = 0; +export const GEMINI_TEMPERATURE_MAX = 2; +export const DEFAULT_GEMINI_TEMPERATURE = 1; + +export const isOpenAIImageQuality = (value: unknown): value is OpenAIImageQuality => + OPENAI_IMAGE_QUALITIES.includes(value as OpenAIImageQuality); +export const isOpenAIImageBackground = (value: unknown): value is OpenAIImageBackground => + OPENAI_IMAGE_BACKGROUNDS.includes(value as OpenAIImageBackground); +export const isOpenAIInputFidelity = (value: unknown): value is OpenAIInputFidelity => + OPENAI_INPUT_FIDELITIES.includes(value as OpenAIInputFidelity); +export const isGeminiThinkingLevel = (value: unknown): value is GeminiThinkingLevel => + GEMINI_THINKING_LEVELS.includes(value as GeminiThinkingLevel); +export const isExternalGenerationMode = (value: unknown): value is ExternalGenerationMode => + value === 'txt2img' || value === 'img2img' || value === 'inpaint'; + +/** + * Seedream's batch endpoint counts inputs and outputs against one total. Mirrors `_SEEDREAM_BATCH_PREFIXES` and + * `_SEEDREAM_BATCH_MAX_TOTAL_IMAGES` in `invokeai/app/services/external_generation/providers/seedream.py`. + */ +const SEEDREAM_BATCH_MODEL_PREFIXES = [ + 'seedream-5', + 'seedream-4.5', + 'seedream-4.0', + 'seedream-4-5', + 'seedream-4-0', + 'seedream-5-0', +] as const; +export const SEEDREAM_BATCH_MAX_TOTAL_IMAGES = 15; + +export const isSeedreamBatchModel = (model: GenerateModelConfig): boolean => + model.type === 'external_image_generator' && + model.provider_id === 'seedream' && + typeof model.provider_model_id === 'string' && + SEEDREAM_BATCH_MODEL_PREFIXES.some((prefix) => (model.provider_model_id as string).startsWith(prefix)); + +export type ExternalProviderOptionKey = keyof Pick< + GenerateSettings, + | 'openaiQuality' + | 'openaiBackground' + | 'openaiInputFidelity' + | 'geminiTemperature' + | 'geminiThinkingLevel' + | 'seedreamWatermark' + | 'seedreamOptimizePrompt' +>; + +/** OpenAI applies its options only to these models; mirrors `_GPT_IMAGE_MODELS` in `providers/openai.py`. */ +const OPENAI_GPT_IMAGE_MODEL_IDS: readonly unknown[] = [ + 'gpt-image-1', + 'gpt-image-1.5', + 'gpt-image-1-mini', + 'gpt-image-2', +]; + +const getCapabilities = (model: GenerateModelConfig | undefined): ExternalModelCapabilities | null => + model?.type === 'external_image_generator' ? (model.capabilities ?? null) : null; + +const getProviderId = (model: GenerateModelConfig | undefined): string | null => + model?.type === 'external_image_generator' && typeof model.provider_id === 'string' ? model.provider_id : null; + +/** True when the model accepts image inputs in any form (an edit mode or reference images). */ +const acceptsImageInputs = (capabilities: ExternalModelCapabilities | null): boolean => + capabilities?.supports_reference_images === true || + (capabilities?.modes ?? []).some((mode) => mode === 'img2img' || mode === 'inpaint'); + +/** + * The provider options the selected model's invocation accepts, in display order. Each provider's node declares its + * own fields, so the provider decides. OpenAI sends them only for GPT Image models (DALL·E takes none), and input + * fidelity only with image inputs, so a model that takes none does not offer it. + */ +export const getExternalProviderOptionKeys = (model: GenerateModelConfig | undefined): ExternalProviderOptionKey[] => { + const capabilities = getCapabilities(model); + + switch (getProviderId(model)) { + case 'openai': + if (!OPENAI_GPT_IMAGE_MODEL_IDS.includes(model?.provider_model_id)) { + return []; + } + return [ + 'openaiQuality', + 'openaiBackground', + ...(acceptsImageInputs(capabilities) ? (['openaiInputFidelity'] as const) : []), + ]; + case 'gemini': + return ['geminiTemperature', 'geminiThinkingLevel']; + case 'seedream': + return ['seedreamWatermark', 'seedreamOptimizePrompt']; + default: + return []; + } +}; + +export const getExternalResolutionPresets = (model: GenerateModelConfig | undefined): ExternalResolutionPreset[] => { + const presets = getCapabilities(model)?.resolution_presets; + + return Array.isArray(presets) ? presets : []; +}; + +/** Fixed sizes per ratio, as `[ratio, size]` pairs in the provider's order; empty for free-form providers. */ +export const getExternalAspectRatioSizes = ( + model: GenerateModelConfig | undefined +): Array<[ratio: string, size: ExternalImageSize]> => { + const sizes = getCapabilities(model)?.aspect_ratio_sizes; + + return sizes ? Object.entries(sizes) : []; +}; + +/** Dimensions select the preset; the stored name breaks ties if the provider advertises duplicate sizes. */ +export const findExternalResolutionPreset = ( + settings: Pick, + model: GenerateModelConfig | undefined +): ExternalResolutionPreset | null => { + const presets = getExternalResolutionPresets(model).filter( + (preset) => preset.width === settings.width && preset.height === settings.height + ); + + return presets.find((preset) => preset.image_size === settings.externalImageSize) ?? presets[0] ?? null; +}; + +/** + * The named size to send: the selected model's preset the current size matches exactly, including a hand-typed size. A + * pick left over from another model never overrides the dimensions at the provider. + */ +export const resolveExternalImageSize = ( + settings: Pick, + model: GenerateModelConfig | undefined +): string | null => findExternalResolutionPreset(settings, model)?.image_size ?? null; + +/** A "W:H" ratio of whole numbers, the only form the backend sizes from (`_size_for_ratio`). */ +const parseWholeRatio = (ratio: string): ExternalImageSize | null => { + const match = /^(\d+):(\d+)$/.exec(ratio); + const width = Number(match?.[1]); + const height = Number(match?.[2]); + + return width > 0 && height > 0 ? { height, width } : null; +}; + +/** + * The sizes a Canvas request may name, keyed by nominal ratio: the resolution presets, else the fixed size per ratio, + * else per whole-number allowed ratio its smallest exact multiple covering the frame within the maximum size and the + * generation limit. The backend's bucketing (`_bucket_request`) passes a fixed size (its own table entry) and an exact + * multiple (it reduces to a declared ratio) through unchanged. A preset whose rounded size is off its exact ratio may + * be nudged by a pixel or so, which is harmless: the provider sizes its output from the ratio and the preset's name, + * and the compiler restores whatever comes back to the bbox. + */ +const getCanvasRequestSizes = ( + model: GenerateModelConfig, + frame: ExternalImageSize +): Array<{ ratio: string; size: ExternalImageSize }> => { + const presets = getExternalResolutionPresets(model); + + if (presets.length > 0) { + return presets.map((preset) => ({ ratio: preset.aspect_ratio, size: preset })); + } + + const sizes = getExternalAspectRatioSizes(model); + + if (sizes.length > 0) { + return sizes.map(([ratio, size]) => ({ ratio, size })); + } + + const capabilities = getCapabilities(model); + const maxSize = capabilities?.max_image_size; + + return (capabilities?.allowed_aspect_ratios ?? []).flatMap((ratio) => { + const unit = parseWholeRatio(ratio); + + if (!unit) { + return []; + } + + const scale = Math.min( + Math.ceil(Math.max(frame.width / unit.width, frame.height / unit.height)), + Math.floor(Math.min(maxSize?.width ?? MAX_DIMENSION, MAX_DIMENSION) / unit.width), + Math.floor(Math.min(maxSize?.height ?? MAX_DIMENSION, MAX_DIMENSION) / unit.height) + ); + + return scale >= 1 ? [{ ratio, size: { height: unit.height * scale, width: unit.width * scale } }] : []; + }); +}; + +/** + * The size a Canvas frame asks a hosted provider for. A provider limited to certain shapes gets one of its own sizes, + * so the compiler knows the shape it gets back: the ratio closest to the frame's, then the smallest size of that ratio + * covering the frame, else the largest. A free-form provider gets the frame. + */ +export const resolveExternalCanvasRequestSize = ( + model: GenerateModelConfig, + frame: ExternalImageSize +): ExternalImageSize => { + const candidates = getCanvasRequestSizes(model, frame); + + if (candidates.length === 0) { + return { height: frame.height, width: frame.width }; + } + + const frameRatio = frame.width / frame.height; + const ratioDistance = ({ size }: (typeof candidates)[number]) => + Math.abs(Math.log(size.width / size.height / frameRatio)); + const closest = candidates.reduce((best, candidate) => + ratioDistance(candidate) < ratioDistance(best) ? candidate : best + ); + const area = ({ size }: (typeof candidates)[number]) => size.width * size.height; + const sameRatio = candidates.filter((candidate) => candidate.ratio === closest.ratio); + const covering = sameRatio.filter(({ size }) => size.width >= frame.width && size.height >= frame.height); + const { size } = + covering.length > 0 + ? covering.reduce((best, candidate) => (area(candidate) < area(best) ? candidate : best)) + : sameRatio.reduce((best, candidate) => (area(candidate) > area(best) ? candidate : best)); + + return { height: size.height, width: size.width }; +}; + +/** The reference images a request sends: none unless the model takes them. */ +export const getExternalReferenceImages = ( + settings: Pick, + model: GenerateModelConfig +): GenerateReferenceImageAsset[] => + getCapabilities(model)?.supports_reference_images === true + ? settings.referenceImages.flatMap((referenceImage) => + referenceImage.isEnabled && + referenceImage.config.type === 'external_reference_image' && + referenceImage.config.image !== null + ? [referenceImage.config.image] + : [] + ) + : []; + +/** + * Canvas content a hosted model cannot take, worded once for readiness and for the invoke-time check that measures + * pixel coverage. Raster content in the frame selects an edit mode, which a model without image-to-image cannot run. + */ +export const describeExternalCanvasRefusal = ( + model: Pick, + refusal: { kind: 'control_layer' | 'regional_guidance'; layerName: string } | { kind: 'raster_content' } +): string => { + switch (refusal.kind) { + case 'control_layer': + return `${model.name} does not support control layer "${refusal.layerName}". Disable it to generate.`; + case 'regional_guidance': + return `${model.name} does not support regional guidance layer "${refusal.layerName}". Disable it to generate.`; + case 'raster_content': + return `${model.name} cannot use image content in the generation frame. Hide or move raster content out of the frame to generate.`; + } +}; + +/** + * Whether a hosted model only generates from text on the Canvas. Image-to-image is the only provider edit mode the + * Canvas compiles, so a model without it (even one advertising inpaint) cannot use content in the frame. + */ +export const isExternalTextOnlyCanvasModel = (model: GenerateModelConfig): boolean => + model.type === 'external_image_generator' && !(model.capabilities?.modes ?? ['txt2img']).includes('img2img'); + +/** + * Provider-specific invocation fields for the selected model, keyed as the node declares them. Optional fields are + * omitted when unset so the node applies its own default. + */ +export const getExternalProviderInvocationOptions = ( + settings: GenerateSettings, + model: GenerateModelConfig +): Record => { + const keys = new Set(getExternalProviderOptionKeys(model)); + const options: Record = {}; + + if (keys.has('openaiQuality')) { + options.quality = settings.openaiQuality; + } + if (keys.has('openaiBackground')) { + options.background = settings.openaiBackground; + } + if (keys.has('openaiInputFidelity') && settings.openaiInputFidelity !== null) { + options.input_fidelity = settings.openaiInputFidelity; + } + if (keys.has('geminiTemperature') && settings.geminiTemperature !== null) { + options.temperature = settings.geminiTemperature; + } + if (keys.has('geminiThinkingLevel') && settings.geminiThinkingLevel !== null) { + options.thinking_level = settings.geminiThinkingLevel; + } + if (keys.has('seedreamWatermark')) { + options.watermark = settings.seedreamWatermark; + } + if (keys.has('seedreamOptimizePrompt')) { + options.optimize_prompt = settings.seedreamOptimizePrompt; + } + + return options; +}; + +/** Metadata keys mirror what the provider nodes record themselves, so Generate and Canvas outputs recall alike. */ +export const EXTERNAL_METADATA_KEYS: Record = { + geminiTemperature: 'gemini_temperature', + geminiThinkingLevel: 'gemini_thinking_level', + openaiBackground: 'openai_background', + openaiInputFidelity: 'openai_input_fidelity', + openaiQuality: 'openai_quality', + seedreamOptimizePrompt: 'seedream_optimize_prompt', + seedreamWatermark: 'seedream_watermark', +}; + +export const getExternalProviderMetadata = ( + settings: GenerateSettings, + model: GenerateModelConfig +): Record => { + const metadata: Record = {}; + const imageSize = resolveExternalImageSize(settings, model); + + if (imageSize !== null) { + metadata.image_size = imageSize; + } + for (const key of getExternalProviderOptionKeys(model)) { + const value = settings[key]; + + if (value !== null) { + metadata[EXTERNAL_METADATA_KEYS[key]] = value; + } + } + + return metadata; +}; diff --git a/invokeai/frontend/webv2/src/features/generation/core/graph.test.ts b/invokeai/frontend/webv2/src/features/generation/core/graph.test.ts index e0be63ac823..ca7166c3bf0 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/graph.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/graph.test.ts @@ -3,6 +3,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest'; import type { ComponentModelConfig, + ExternalImageGeneratorModelConfig, GenerateModelConfig, GenerateSettings, LoraModelConfig, @@ -634,6 +635,135 @@ describe('compileGenerateGraph', () => { expect(graph.nodes.canvas_output?.reference_images).toEqual([{ image_name: 'reference.png' }]); }); + describe('external provider options', () => { + const providerSettings: Partial = { + externalImageSize: '2K', + geminiTemperature: 0.4, + geminiThinkingLevel: 'high', + openaiBackground: 'transparent', + openaiInputFidelity: 'high', + openaiQuality: 'low', + seedreamOptimizePrompt: true, + seedreamWatermark: true, + }; + const providerModel = ( + providerId: string, + capabilities: ExternalImageGeneratorModelConfig['capabilities'] + ): GenerateModelConfig => + ({ + ...externalModel, + capabilities, + provider_id: providerId, + provider_model_id: providerId === 'openai' ? 'gpt-image-1' : `${providerId}-model`, + }) as GenerateModelConfig; + + it('compiles OpenAI quality, background and input fidelity, recording them in metadata', () => { + const model = providerModel('openai', { modes: ['txt2img', 'img2img'] }); + const graph = compile(model, providerSettings); + + expect(graph.nodes.canvas_output).toMatchObject({ + background: 'transparent', + input_fidelity: 'high', + quality: 'low', + }); + // Fields of other providers' nodes never reach this one. + expect(graph.nodes.canvas_output).not.toHaveProperty('temperature'); + expect(graph.nodes.canvas_output).not.toHaveProperty('watermark'); + expect(getNodeByType(graph, 'core_metadata')).toMatchObject({ + openai_background: 'transparent', + openai_input_fidelity: 'high', + openai_quality: 'low', + }); + }); + + it('omits OpenAI input fidelity for a model with no image inputs, and when unset', () => { + const noInputs = compile(providerModel('openai', { modes: ['txt2img'] }), providerSettings); + + expect(noInputs.nodes.canvas_output).toMatchObject({ background: 'transparent', quality: 'low' }); + expect(noInputs.nodes.canvas_output).not.toHaveProperty('input_fidelity'); + + const unset = compile(providerModel('openai', { modes: ['txt2img', 'img2img'] }), { + ...providerSettings, + openaiInputFidelity: null, + }); + + expect(unset.nodes.canvas_output).not.toHaveProperty('input_fidelity'); + expect(getNodeByType(unset, 'core_metadata')).not.toHaveProperty('openai_input_fidelity'); + }); + + it('compiles Gemini temperature and thinking level only when set', () => { + const model = providerModel('gemini', { modes: ['txt2img'] }); + const graph = compile(model, providerSettings); + + expect(graph.nodes.canvas_output).toMatchObject({ temperature: 0.4, thinking_level: 'high' }); + expect(graph.nodes.canvas_output).not.toHaveProperty('quality'); + expect(getNodeByType(graph, 'core_metadata')).toMatchObject({ + gemini_temperature: 0.4, + gemini_thinking_level: 'high', + }); + + const defaults = compile(model, { ...providerSettings, geminiTemperature: null, geminiThinkingLevel: null }); + + expect(defaults.nodes.canvas_output).not.toHaveProperty('temperature'); + expect(defaults.nodes.canvas_output).not.toHaveProperty('thinking_level'); + }); + + it('compiles Seedream watermark and prompt optimization as explicit booleans', () => { + const model = providerModel('seedream', { modes: ['txt2img'] }); + + expect(compile(model, providerSettings).nodes.canvas_output).toMatchObject({ + optimize_prompt: true, + watermark: true, + }); + expect( + compile(model, { ...providerSettings, seedreamOptimizePrompt: false, seedreamWatermark: false }).nodes + .canvas_output + ).toMatchObject({ optimize_prompt: false, watermark: false }); + expect(getNodeByType(compile(model, providerSettings), 'core_metadata')).toMatchObject({ + seedream_optimize_prompt: true, + seedream_watermark: true, + }); + }); + + it('sends the named size only when the model lists it in a resolution preset', () => { + const presets = [ + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + ]; + const listed = compile(providerModel('gemini', { modes: ['txt2img'], resolution_presets: presets }), { + ...providerSettings, + height: 2048, + width: 2048, + }); + + expect(listed.nodes.canvas_output).toMatchObject({ height: 2048, image_size: '2K', width: 2048 }); + expect(getNodeByType(listed, 'core_metadata')).toMatchObject({ image_size: '2K' }); + + // A size picked for another model, or a provider without named sizes, falls back to the provider default. + const unlisted = compile(providerModel('gemini', { modes: ['txt2img'] }), providerSettings); + + expect(unlisted.nodes.canvas_output?.image_size).toBeNull(); + expect(getNodeByType(unlisted, 'core_metadata')).not.toHaveProperty('image_size'); + + // Dimensions typed after the pick are custom: the named size would otherwise override them at the provider. + const typed = compile(providerModel('gemini', { modes: ['txt2img'], resolution_presets: presets }), { + ...providerSettings, + height: 2048, + width: 1536, + }); + + expect(typed.nodes.canvas_output).toMatchObject({ height: 2048, image_size: null, width: 1536 }); + expect(getNodeByType(typed, 'core_metadata')).not.toHaveProperty('image_size'); + }); + + it('accepts provider sizes that are not multiples of a local model grid', () => { + const model = providerModel('gemini', { modes: ['txt2img'] }); + const graph = compile(model, { height: 2048, width: 1365 }); + + expect(graph.nodes.canvas_output).toMatchObject({ height: 2048, width: 1365 }); + }); + }); + it('wires SD IP Adapter reference images into denoise', () => { const graph = compile(sd1Model, { referenceImages: [ diff --git a/invokeai/frontend/webv2/src/features/generation/core/graph.ts b/invokeai/frontend/webv2/src/features/generation/core/graph.ts index 21abacc9494..d1000418497 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/graph.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/graph.ts @@ -21,6 +21,7 @@ import type { import { coerceSchedulerForGraph, + getExternalModeReason, getExternalProviderNodeType, getFlux2DiffusersComponentSource, getGenerationDimensions, @@ -42,6 +43,12 @@ import { isZImageQwen3Encoder, type GenerateComponentFilter, } from './componentCompatibility'; +import { + getExternalProviderInvocationOptions, + getExternalProviderMetadata, + getExternalReferenceImages, + resolveExternalImageSize, +} from './externalProvider'; import { addEdge, addMetadata, @@ -1591,18 +1598,29 @@ const buildWanGraph = ( return graph; }; -const buildExternalGraph = ( +/** Canvas inputs per external mode: the composited bbox is the init image for img2img. */ +export type ExternalCanvasInputs = { mode: 'txt2img' } | { mode: 'img2img'; initImageName: string }; + +/** + * Hosted providers take the whole request on one node: prompt, seed, size, provider options, and (on the canvas) the + * init image. The Canvas keep-mask does not match any provider's edit-mask contract, so inpaint is not compiled. + */ +export const buildExternalGraph = ( settings: GenerateSettings, model: GenerateModelConfig, outputIsIntermediate: boolean, - projectSettings: GenerationProjectSettings + projectSettings: GenerationProjectSettings, + canvas: ExternalCanvasInputs | null = null ): BackendGraphContract => { if (model.type !== 'external_image_generator') { throw new Error('Selected model is not an external image generator.'); } - if (model.capabilities?.modes && !model.capabilities.modes.includes('txt2img')) { - throw new Error(`${model.name} does not support text-to-image generation.`); + const mode = canvas?.mode ?? 'txt2img'; + const modeReason = getExternalModeReason(model, mode); + + if (modeReason) { + throw new Error(modeReason); } const providerId = typeof model.provider_id === 'string' ? model.provider_id : ''; @@ -1617,14 +1635,16 @@ const buildExternalGraph = ( const output = addNode(graph, { height: settings.height, id: 'canvas_output', - image_size: null, + image_size: resolveExternalImageSize(settings, model), is_intermediate: outputIsIntermediate, - mode: 'txt2img', + mode, model, num_images: 1, type: nodeType, use_cache: false, width: settings.width, + ...getExternalProviderInvocationOptions(settings, model), + ...(canvas?.mode === 'img2img' ? { init_image: { image_name: canvas.initImageName } } : {}), }); addEdge(graph, positivePrompt, 'value', output, 'prompt'); @@ -1635,17 +1655,15 @@ const buildExternalGraph = ( if (model.capabilities?.supports_seed === true) { addEdge(graph, seed, 'value', output, 'seed'); } - if (model.capabilities?.supports_reference_images === true) { - const referenceImages = getEnabledReferenceImages(settings, 'external_reference_image') - .map((referenceImage) => referenceImage.config.image) - .filter((image): image is GenerateReferenceImageAsset => image !== null) - .map(toImageField); - - if (referenceImages.length > 0) { - (output as Record).reference_images = referenceImages; - } + const referenceImages = getExternalReferenceImages(settings, model).map(toImageField); + + if (referenceImages.length > 0) { + (output as Record).reference_images = referenceImages; } - addMetadata(graph, output, settings, model, null, projectSettings, { scheduler: undefined }); + addMetadata(graph, output, settings, model, null, projectSettings, { + scheduler: undefined, + ...getExternalProviderMetadata(settings, model), + }); addReferenceImageMetadata(graph, output, settings); return graph; diff --git a/invokeai/frontend/webv2/src/features/generation/core/graphBuilder.ts b/invokeai/frontend/webv2/src/features/generation/core/graphBuilder.ts index 163d568b082..4eddd539358 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/graphBuilder.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/graphBuilder.ts @@ -42,7 +42,7 @@ export const addEdge = ( }; /** Rename every matching node ID and edge reference. */ -const renameNode = (graph: BackendGraphContract, oldId: string, newId: string): BackendInvocationContract => { +export const renameNode = (graph: BackendGraphContract, oldId: string, newId: string): BackendInvocationContract => { const node = graph.nodes[oldId]; if (!node) { throw new Error(`Could not find the "${oldId}" node to rename.`); diff --git a/invokeai/frontend/webv2/src/features/generation/core/settings.test.ts b/invokeai/frontend/webv2/src/features/generation/core/settings.test.ts index f5d373e07d4..5b16119586c 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/settings.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/settings.test.ts @@ -279,6 +279,64 @@ describe('normalizeGenerateSettings', () => { expect(normalized?.vaePrecision).toBe('fp32'); }); + it('keeps valid hosted-provider options and drops the rest back to provider defaults', () => { + const kept = normalizeGenerateSettings({ + ...legacyStoredValues, + externalImageSize: '2K', + geminiTemperature: 0.5, + geminiThinkingLevel: 'minimal', + openaiBackground: 'opaque', + openaiInputFidelity: 'high', + openaiQuality: 'medium', + seedreamOptimizePrompt: true, + seedreamWatermark: true, + }); + + expect(kept).toMatchObject({ + externalImageSize: '2K', + geminiTemperature: 0.5, + geminiThinkingLevel: 'minimal', + openaiBackground: 'opaque', + openaiInputFidelity: 'high', + openaiQuality: 'medium', + seedreamOptimizePrompt: true, + seedreamWatermark: true, + }); + + const dropped = normalizeGenerateSettings({ + ...legacyStoredValues, + externalImageSize: '', + geminiTemperature: 2.5, + geminiThinkingLevel: 'max', + openaiBackground: 'clear', + openaiInputFidelity: 'medium', + openaiQuality: 'ultra', + seedreamOptimizePrompt: 'yes', + seedreamWatermark: 1, + }); + + expect(dropped).toMatchObject({ + externalImageSize: null, + geminiTemperature: null, + geminiThinkingLevel: null, + openaiBackground: 'auto', + openaiInputFidelity: null, + openaiQuality: 'auto', + seedreamOptimizePrompt: false, + seedreamWatermark: false, + }); + }); + + it.each([0, 2])('preserves the Gemini temperature boundary %s', (geminiTemperature) => { + expect(normalizeGenerateSettings({ ...legacyStoredValues, geminiTemperature })?.geminiTemperature).toBe( + geminiTemperature + ); + }); + + it('drops a negative Gemini temperature back to the provider default', () => { + expect(normalizeGenerateSettings({ ...legacyStoredValues, geminiTemperature: -0.1 })?.geminiTemperature).toBeNull(); + }); + it('bounds the model-family tuning fields and defaults the rest', () => { const normalized = normalizeGenerateSettings({ ...legacyStoredValues, @@ -599,6 +657,8 @@ describe('dimension helpers', () => { expect(getGenerationDimensions({ base: 'flux2', type: 'main' }).grid).toBe(16); expect(getGenerationDimensions({ base: 'qwen-image', type: 'main' }).grid).toBe(16); expect(getGenerationDimensions({ base: 'cogview4', type: 'main' }).grid).toBe(32); + // Hosted providers bucket sizes themselves, so their presets (e.g. 1365 wide) are kept pixel-exact. + expect(getGenerationDimensions({ base: 'external', type: 'external_image_generator' }).grid).toBe(1); expect(clampDimension(888, getGenerationDimensions({ base: 'flux2', type: 'main' }).grid)).toBe(896); }); }); diff --git a/invokeai/frontend/webv2/src/features/generation/core/settings.ts b/invokeai/frontend/webv2/src/features/generation/core/settings.ts index c3dcc50eb61..542996cab97 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/settings.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/settings.ts @@ -29,7 +29,16 @@ import { isValidKrea2RebalanceWeights, KREA2_REBALANCE_WEIGHT_COUNT, } from './conditioningRebalance'; +import { MAX_DIMENSION, MIN_DIMENSION } from './dimensionLimits'; import { isDynamicPromptsSeedBehaviour, sanitizeMaxPrompts, sanitizeSampleSeed } from './dynamicPrompts'; +import { + GEMINI_TEMPERATURE_MAX, + GEMINI_TEMPERATURE_MIN, + isGeminiThinkingLevel, + isOpenAIImageBackground, + isOpenAIImageQuality, + isOpenAIInputFidelity, +} from './externalProvider'; import { clampPidSteps, DEFAULT_PID_STEPS, isPidMode } from './pid'; import { isCanonicalPromptTemplateSnapshot, sanitizePromptTemplateSnapshot } from './promptTemplates'; import { cloneCroppableImage, normalizeCroppableImage } from './referenceImage'; @@ -85,8 +94,6 @@ export const isAspectRatioId = (value: unknown): value is AspectRatioId => value === 'Free' || (typeof value === 'string' && value in ASPECT_RATIO_MAP); export const DIMENSION_GRID = 8; -export const MIN_DIMENSION = 64; -export const MAX_DIMENSION = 4096; /** The expansion config the settings describe, under the names the expansion route and preview use. */ export const getDynamicPromptsConfig = ( @@ -925,6 +932,20 @@ export const normalizeGenerateSettings = (values: unknown): GenerateSettings | n pidDecoderModel: getModelIdentifierOrNull(values.pidDecoderModel), gemma2EncoderModel: getModelIdentifierOrNull(values.gemma2EncoderModel), pidSteps: clampPidSteps(hasFiniteNumber(values, 'pidSteps') ? (values.pidSteps as number) : DEFAULT_PID_STEPS), + externalImageSize: + typeof values.externalImageSize === 'string' && values.externalImageSize !== '' ? values.externalImageSize : null, + openaiQuality: isOpenAIImageQuality(values.openaiQuality) ? values.openaiQuality : 'auto', + openaiBackground: isOpenAIImageBackground(values.openaiBackground) ? values.openaiBackground : 'auto', + openaiInputFidelity: isOpenAIInputFidelity(values.openaiInputFidelity) ? values.openaiInputFidelity : null, + geminiTemperature: + hasFiniteNumber(values, 'geminiTemperature') && + (values.geminiTemperature as number) >= GEMINI_TEMPERATURE_MIN && + (values.geminiTemperature as number) <= GEMINI_TEMPERATURE_MAX + ? (values.geminiTemperature as number) + : null, + geminiThinkingLevel: isGeminiThinkingLevel(values.geminiThinkingLevel) ? values.geminiThinkingLevel : null, + seedreamWatermark: values.seedreamWatermark === true, + seedreamOptimizePrompt: values.seedreamOptimizePrompt === true, refinerModel: getRefinerModelOrNull(values.refinerModel), refinerSteps: hasFiniteNumber(values, 'refinerSteps') ? Math.max(MIN_REFINER_STEPS, Math.round(values.refinerSteps as number)) diff --git a/invokeai/frontend/webv2/src/features/generation/core/types.ts b/invokeai/frontend/webv2/src/features/generation/core/types.ts index 208c3ffa964..f83e57491ad 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/types.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/types.ts @@ -30,18 +30,43 @@ export type MainModelConfig = ModelIdentifierConfig & { } | null; }; +export type ExternalGenerationMode = 'txt2img' | 'img2img' | 'inpaint'; + +export interface ExternalImageSize { + width: number; + height: number; +} + +/** A provider's named size at one aspect ratio, e.g. Gemini's `1K` at `16:9`. */ +export interface ExternalResolutionPreset { + label: string; + aspect_ratio: string; + image_size: string; + width: number; + height: number; +} + +/** The capability matrix the backend records for an external model (`ExternalModelCapabilities`). */ +export interface ExternalModelCapabilities { + modes?: string[]; + supports_seed?: boolean; + supports_negative_prompt?: boolean; + supports_reference_images?: boolean; + max_reference_images?: number | null; + max_images_per_request?: number | null; + max_image_size?: ExternalImageSize | null; + allowed_aspect_ratios?: string[] | null; + /** Fixed output size per aspect ratio, for providers that accept no free-form dimensions. */ + aspect_ratio_sizes?: Record | null; + resolution_presets?: ExternalResolutionPreset[] | null; +} + export type ExternalImageGeneratorModelConfig = ModelIdentifierConfig & { base: 'external'; type: 'external_image_generator'; format: 'external_api'; provider_id?: string; - capabilities?: { - modes?: string[]; - supports_seed?: boolean; - supports_negative_prompt?: boolean; - supports_reference_images?: boolean; - max_reference_images?: number | null; - } | null; + capabilities?: ExternalModelCapabilities | null; default_settings?: { width?: number | null; height?: number | null; @@ -294,6 +319,17 @@ export interface GenerateSettings { /** The shared Gemma-2 caption encoder PiD conditions its decode on. */ gemma2EncoderModel: ComponentModelConfig | null; pidSteps: number; + /** A provider's named size (e.g. `1K`); sent only when the selected model lists it in a resolution preset. */ + externalImageSize: string | null; + openaiQuality: OpenAIImageQuality; + openaiBackground: OpenAIImageBackground; + /** Null lets OpenAI pick; only meaningful with image inputs. */ + openaiInputFidelity: OpenAIInputFidelity | null; + /** Null uses the provider default. */ + geminiTemperature: number | null; + geminiThinkingLevel: GeminiThinkingLevel | null; + seedreamWatermark: boolean; + seedreamOptimizePrompt: boolean; /** SDXL only: a refiner-family main model that finishes the schedule from `refinerStart`; null runs none. */ refinerModel: MainModelConfig | null; refinerSteps: number; @@ -306,6 +342,11 @@ export interface GenerateSettings { refinerStart: number; } +export type OpenAIImageQuality = 'auto' | 'high' | 'medium' | 'low'; +export type OpenAIImageBackground = 'auto' | 'transparent' | 'opaque'; +export type OpenAIInputFidelity = 'low' | 'high'; +export type GeminiThinkingLevel = 'minimal' | 'high'; + /** * - `off`: ordinary VAE decode. * - `fit`: generate at the requested size, decode 4x, downscale back to it. diff --git a/invokeai/frontend/webv2/src/features/generation/react.ts b/invokeai/frontend/webv2/src/features/generation/react.ts index ff7232243b9..abb17daaeed 100644 --- a/invokeai/frontend/webv2/src/features/generation/react.ts +++ b/invokeai/frontend/webv2/src/features/generation/react.ts @@ -10,7 +10,7 @@ export { flushGenerateDrafts, useRegisterGenerateDraftFlusher } from './ui/gener export { useDebouncedDraftValue } from './ui/useDebouncedDraftValue'; export { useDynamicPrompts, type DynamicPromptsExpansion } from './ui/useDynamicPrompts'; export { createGenerateFormValuesSelector } from './ui/generateFormViewModel'; -export { notifyGenerateModelSelectionCleared } from './ui/modelSelectionNotice'; +export { notifyGenerateDimensionsAdjusted, notifyGenerateModelSelectionCleared } from './ui/modelSelectionNotice'; export { adjustFocusedPromptAttention } from './ui/promptFields/promptAttentionHotkeys'; export { focusPositivePrompt } from './ui/promptFields/promptFocus'; export { promptHistoryNavigation } from './ui/promptFields/promptHistoryNavigation'; diff --git a/invokeai/frontend/webv2/src/features/generation/settings.ts b/invokeai/frontend/webv2/src/features/generation/settings.ts index d4a29593997..d7e17c305ea 100644 --- a/invokeai/frontend/webv2/src/features/generation/settings.ts +++ b/invokeai/frontend/webv2/src/features/generation/settings.ts @@ -15,6 +15,7 @@ export { hasModelDefaultVae, isAspectRatioLocked, isFluxDypePreset, + isGenerateModelConfig, isLoraCompatibleWithModel, isLoraModelConfig, isMainModelConfig, @@ -66,6 +67,19 @@ export { type GenerateModelSelectionResult, } from './core/baseGenerationPolicies'; export { hasArchitectureCapabilities } from './core/architectureCapabilities'; +export { + describeExternalCanvasRefusal, + GEMINI_TEMPERATURE_MAX, + GEMINI_TEMPERATURE_MIN, + getExternalProviderOptionKeys, + getExternalResolutionPresets, + isExternalTextOnlyCanvasModel, + isGeminiThinkingLevel, + isOpenAIImageBackground, + isOpenAIImageQuality, + isOpenAIInputFidelity, + resolveExternalCanvasRequestSize, +} from './core/externalProvider'; export { resolveGenerateWidgetValues, type ResolvedGenerateWidgetValues, diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.browser.test.tsx index d8fd6865b3f..1782377a1e7 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.browser.test.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.browser.test.tsx @@ -1,5 +1,5 @@ /* oxlint-disable react-perf/jsx-no-new-object-as-prop */ -import type { GenerateSettings, MainModelConfig } from '@features/generation/core/types'; +import type { GenerateModelConfig, GenerateSettings, MainModelConfig } from '@features/generation/core/types'; import { ChakraProvider } from '@chakra-ui/react'; import { getDefaultGenerateSettings } from '@features/generation/core/baseGenerationPolicies'; @@ -11,13 +11,23 @@ import { createRoot, type Root } from 'react-dom/client'; import { I18nextProvider, initReactI18next } from 'react-i18next'; import { afterEach, describe, expect, it, vi } from 'vitest'; +import type { CanvasRenderSizeProps } from './GenerationUiContext'; + import { GenerateDimensionFields } from './GenerateDimensionFields'; +const harness = vi.hoisted(() => ({ invocationSourceId: 'generate' })); + vi.mock('./GenerationUiContext', () => ({ useGenerationQueueInsights: (select: (insights: unknown) => unknown) => select({ secondsPerRun: null, seedHistory: [] }), useGenerationUi: () => ({ - project: { invocationSourceId: 'generate' }, + // Stands in for the canvas's render-size slot, which lays out the Size section around its controls. + CanvasRenderSize: ({ children, frame }: CanvasRenderSizeProps) => + children({ + controls:
, + size: { height: frame.height * 2, width: frame.width * 2 }, + }), + project: { invocationSourceId: harness.invocationSourceId }, sectionPreferences: { sectionsOpen: { dimensions: true }, setSectionOpen: vi.fn() }, }), })); @@ -37,6 +47,7 @@ void i18n.use(initReactI18next).init({ height: 'Height', lockAspectRatio: 'Lock aspect ratio', megapixelsValue: '{{value}} MP', + providerSize: 'Provider size', size: 'Size', swapWidthAndHeight: 'Swap width and height', unlockAspectRatio: 'Unlock aspect ratio', @@ -49,6 +60,15 @@ void i18n.use(initReactI18next).init({ }); const sd1Model: MainModelConfig = { base: 'sd-1', key: 'sd1', name: 'SD 1.5', type: 'main' }; +const gptImage: GenerateModelConfig = { + base: 'external', + capabilities: { modes: ['txt2img', 'img2img'] }, + format: 'external_api', + key: 'gpt-image', + name: 'GPT Image', + provider_id: 'openai', + type: 'external_image_generator', +}; let host: HTMLDivElement | null = null; let root: Root | null = null; @@ -62,7 +82,7 @@ const settle = (action: () => void): Promise => }); }); -const render = async (settings: Partial) => { +const render = async (settings: Partial, model: GenerateModelConfig = sd1Model) => { host = document.createElement('div'); document.body.append(host); root = createRoot(host); @@ -73,9 +93,9 @@ const render = async (settings: Partial) => { @@ -94,12 +114,91 @@ const bracket = () => host?.querySelector('[data-part="bracket"]')?.parentElemen const presetTrigger = () => host?.querySelector('[role="combobox"]') ?? null; afterEach(async () => { + harness.invocationSourceId = 'generate'; await settle(() => root?.unmount()); host?.remove(); host = null; root = null; }); +describe('GenerateDimensionFields canvas render size', () => { + it('lays out the render-size controls and the resized badge for a local model on the canvas', async () => { + harness.invocationSourceId = 'canvas'; + await render({ height: 512, width: 512 }); + + expect(host?.querySelector('[data-testid="render-size-controls"]')).not.toBeNull(); + expect(host?.textContent).toContain('512x512 → 1024x1024'); + }); + + it('shows a hosted provider only the frame, which is the size the provider is asked for', async () => { + harness.invocationSourceId = 'canvas'; + await render({ height: 512, width: 512 }, gptImage); + + expect(host?.querySelector('[data-testid="render-size-controls"]')).toBeNull(); + expect(host?.textContent).not.toContain('→'); + expect(host?.textContent).toContain('512x512'); + }); +}); + +describe('GenerateDimensionFields provider size', () => { + const geminiImage: GenerateModelConfig = { + ...gptImage, + provider_id: 'gemini', + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, + ], + }, + }; + const providerTrigger = () => { + const providerLabel = [...(host?.querySelectorAll('label') ?? [])].find( + (label) => label.textContent === 'Provider size' + ); + + return host?.querySelector(`[role="combobox"][aria-labelledby~="${providerLabel?.id}"]`) ?? null; + }; + + it('selects the preset from the panel dimensions, breaking a same-size tie by the stored name', async () => { + await render({ externalImageSize: null, height: 1152, width: 2048 }, geminiImage); + expect(providerTrigger()?.textContent).toContain('16:9 (2K)'); + + await settle(() => root?.unmount()); + host?.remove(); + + const duplicateSizes: GenerateModelConfig = { + ...geminiImage, + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + ...(geminiImage.capabilities?.resolution_presets ?? []), + { aspect_ratio: '1:1', height: 1024, image_size: 'HD', label: '1:1 (HD)', width: 1024 }, + ], + }, + }; + await render({ externalImageSize: 'HD', height: 1024, width: 1024 }, duplicateSizes); + expect(providerTrigger()?.textContent).toContain('1:1 (HD)'); + }); + + it('commits the hosted preset through the dimension panel with its dimensions and name', async () => { + const onCommit = await render({ externalImageSize: null, height: 1024, width: 1024 }, geminiImage); + + expect(providerTrigger()?.textContent).toContain('1:1 (1K)'); + await settle(() => providerTrigger()?.click()); + await settle(() => document.querySelector('[role="option"][data-value="16:9|2K"]')?.click()); + + expect(onCommit).toHaveBeenCalledWith({ + aspectRatioId: '16:9', + aspectRatioIsLocked: true, + aspectRatioValue: 2048 / 1152, + externalImageSize: '2K', + height: 1152, + width: 2048, + }); + }); +}); + describe('GenerateDimensionFields aspect ratio', () => { it('reshapes the size to a chosen preset and locks it', async () => { const onCommit = await render({ aspectRatioId: 'Free', aspectRatioIsLocked: false, height: 1024, width: 1024 }); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.tsx index ec98116da29..be5cfd452c0 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateDimensionFields.tsx @@ -3,14 +3,13 @@ import type { AspectRatioId, GenerateModelConfig, GenerateSettings } from '@feat import { Badge, Box, HStack, Icon, Stack, Text } from '@chakra-ui/react'; import { getDefaultGenerateSettings, getGenerationDimensions } from '@features/generation/core/baseGenerationPolicies'; +import { MAX_DIMENSION, MIN_DIMENSION } from '@features/generation/core/dimensionLimits'; import { ASPECT_RATIO_MAP, calculateNewSize, clampDimension, deriveAspectRatioId, isAspectRatioLocked, - MAX_DIMENSION, - MIN_DIMENSION, } from '@features/generation/core/settings'; import { useExternalStoreSelector } from '@platform/state/selectors'; import { Button, IconButton, Tooltip } from '@platform/ui'; @@ -27,6 +26,7 @@ import { useGenerationUi, } from './GenerationUiContext'; import { AspectRatioLockButton, AspectRatioSelect } from './shared/AspectRatioSelect'; +import { ExternalSizeSelect } from './shared/ExternalSizeSelect'; import { GenerateCollapsibleSection } from './shared/GenerateCollapsibleSection'; import { GenerateFieldContextMenu } from './shared/GenerateFieldContextMenu'; @@ -248,6 +248,7 @@ const selectDimensionSettings = pickGenerateSettings([ 'aspectRatioId', 'aspectRatioIsLocked', 'aspectRatioValue', + 'externalImageSize', 'height', 'width', ]); @@ -520,6 +521,18 @@ export const GenerateDimensionFields = ({ + {selectedModel?.type === 'external_image_generator' ? ( + + ) : null} {renderSection} ) : ( renderSection(null) diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateModelCard.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateModelCard.tsx index dcfb5e10cf9..dfe9e3c4a52 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateModelCard.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateModelCard.tsx @@ -18,9 +18,22 @@ import { useTranslation } from 'react-i18next'; import type { GenerateDraft } from './generateDebounce'; import { GenerationModelSelect as ModelSelect, useGenerationUi } from './GenerationUiContext'; +import { notifyGenerateDimensionsAdjusted } from './modelSelectionNotice'; const MAIN_MODEL_TYPES = ['main', 'external_image_generator']; -const NO_CLEARED_LABELS: readonly string[] = []; + +/** What a pending switch would change in the live draft: settings it removes, and the size it moves to, if any. */ +interface PendingSwitchPreview { + clearedLabels: readonly string[]; + adjustedSize: { height: number; width: number } | null; +} + +const NO_PENDING_SWITCH: PendingSwitchPreview = { adjustedSize: null, clearedLabels: [] }; + +const arePreviewsEqual = (left: PendingSwitchPreview, right: PendingSwitchPreview) => + areArraysEqual(left.clearedLabels, right.clearedLabels) && + left.adjustedSize?.width === right.adjustedSize?.width && + left.adjustedSize?.height === right.adjustedSize?.height; interface GenerateModelCardProps { draft: GenerateDraft; @@ -45,35 +58,60 @@ export const GenerateModelCard = ({ const { i18n, t } = useTranslation(); const ui = useGenerationUi(); const { openManager } = ui.models; + // On a Canvas source the width and height are the frame, which a switch keeps for a hosted provider. + const isCanvasFrame = ui.project.invocationSourceId === 'canvas'; const openManagerForMainModels = useCallback(() => openManager({ modelType: 'main' }), [openManager]); // Retain only the selected model and recompute against the current draft on confirmation. const [pendingSwitchModel, setPendingSwitchModel] = useState(null); // Constant until a switch awaits confirmation; then the open dialog lists what the latest draft would lose. - const pendingSwitchClearedLabels = useExternalStoreSelector( + const pendingSwitch = useExternalStoreSelector( draft.subscribe, draft.getSnapshot, - (settings: GenerateSettings) => - pendingSwitchModel - ? getGenerateModelSelectionResult({ currentValues: settings, model: pendingSwitchModel, models }).clearedLabels - : NO_CLEARED_LABELS, - areArraysEqual + (settings: GenerateSettings): PendingSwitchPreview => { + if (!pendingSwitchModel) { + return NO_PENDING_SWITCH; + } + + const result = getGenerateModelSelectionResult({ + currentValues: settings, + isCanvasFrame, + model: pendingSwitchModel, + models, + }); + + return { + adjustedSize: result.dimensionsAdjusted + ? { height: result.settings.height, width: result.settings.width } + : null, + clearedLabels: result.clearedLabels, + }; + }, + arePreviewsEqual ); const commitModelSelection = (model: GenerateModelConfig) => { - onCommitSettings(getGenerateModelSelectionResult({ currentValues: draft.getSnapshot(), model, models }).settings); + onCommitSettings( + getGenerateModelSelectionResult({ currentValues: draft.getSnapshot(), isCanvasFrame, model, models }).settings + ); }; const selectModel = (model: GenerateModelConfig) => { - const result = getGenerateModelSelectionResult({ currentValues: draft.getSnapshot(), model, models }); + const result = getGenerateModelSelectionResult({ + currentValues: draft.getSnapshot(), + isCanvasFrame, + model, + models, + }); - // Lossy switches confirm before committing; lossless ones stay instant. + // Lossy switches confirm before committing; lossless ones stay instant and report a moved size. if (result.clearedLabels.length > 0) { setPendingSwitchModel(model); return; } onCommitSettings(result.settings); + notifyGenerateDimensionsAdjusted({ modelName: model.name, notifications: ui.notifications, result, t }); }; const hasNoSupportedModels = !isLoadingModels && !loadError && supportedModels.length === 0; @@ -116,14 +154,21 @@ export const GenerateModelCard = ({ - {t('widgets.generate.switchModelBody', { - labels: new Intl.ListFormat(i18n.resolvedLanguage, { style: 'long', type: 'conjunction' }).format( - pendingSwitchClearedLabels - ), - name: pendingSwitchModel?.name ?? '', - })} - + <> + + {t('widgets.generate.switchModelBody', { + labels: new Intl.ListFormat(i18n.resolvedLanguage, { style: 'long', type: 'conjunction' }).format( + pendingSwitch.clearedLabels + ), + name: pendingSwitchModel?.name ?? '', + })} + + {pendingSwitch.adjustedSize ? ( + + {t('widgets.generate.switchModelAdjustsDimensions', pendingSwitch.adjustedSize)} + + ) : null} + } confirmLabel={t('widgets.generate.switchModelConfirm')} isOpen={pendingSwitchModel !== null} diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.browser.test.tsx index b50b9f8e40e..e615c1fe077 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.browser.test.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.browser.test.tsx @@ -130,6 +130,36 @@ describe('GeneratePresetsPopover', () => { } ); + it('names the size a preset moved to when its stored size does not fit the model', async () => { + const model: MainModelConfig = { base: 'sdxl', key: 'sdxl', name: 'SDXL', type: 'main' }; + // 1028 is off SDXL's 8 px grid, as a preset saved from a hosted model's size can be. + const preset = { ...getDefaultGenerateSettings(model), height: 1024, width: 1028 }; + // The current size differs from the preset's, so the preset is not shown as active. + const currentValues = { ...preset, model, width: 1024 }; + const info = vi.fn(); + const patchGenerateSettings = vi.fn(); + const adapter = { + ...ADAPTER, + generateValues: { getSnapshot: () => currentValues, subscribe: () => () => {} }, + models: { ...ADAPTER.models, catalog: [model] }, + presets: { ...ADAPTER.presets, presets: [{ id: 'off-grid', label: 'Off grid', values: preset }] }, + notifications: { ...ADAPTER.notifications, info }, + settings: { ...ADAPTER.settings, patchGenerateSettings }, + } as unknown as GenerationUiAdapter; + await renderAdapter(adapter); + await act(() => userEvent.click(document.querySelector('[aria-label="widgets.generate.presets"]')!)); + const presetButton = [...document.querySelectorAll('button')].find( + (button) => button.textContent === 'Off grid' + ); + await act(() => presetButton!.click()); + + expect(patchGenerateSettings).toHaveBeenCalledWith(expect.objectContaining({ width: 1032 }), 'project-1'); + expect(info).toHaveBeenCalledWith( + 'widgets.generate.dimensionsAdjusted', + 'widgets.generate.dimensionsAdjustedDescription' + ); + }); + it('animates the rename dialog out instead of unmounting it on close', async () => { await openRename(0); expect(renameInput()?.value).toBe('Portrait'); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.tsx index 9c92a0c5fd4..70fe6b0d383 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GeneratePresetsPopover.tsx @@ -28,7 +28,7 @@ import type { GeneratePresetRecord } from './GenerationUiContext'; import { flushGenerateDrafts } from './generateDraftRegistry'; import { getGenerateFormCommitPatch } from './generateFormViewModel'; import { useGenerateValues, useGenerationUi } from './GenerationUiContext'; -import { notifyGenerateModelSelectionCleared } from './modelSelectionNotice'; +import { notifyGenerateDimensionsAdjusted, notifyGenerateModelSelectionCleared } from './modelSelectionNotice'; import { PANEL_HEADER_CONTROL_HEIGHT, PromptPanelHeader } from './promptFields/PromptPanelHeader'; const POPOVER_POSITIONING = { placement: 'bottom-end' } as const; @@ -223,6 +223,7 @@ export const GeneratePresetsPopover = () => { reason: 'preset', t, }); + notifyGenerateDimensionsAdjusted({ modelName: model.name, notifications: ui.notifications, result, t }); setIsOpen(false); }, [i18n.resolvedLanguage, models, projectId, settings, supportedModels, t, ui.notifications, ui.settings] diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.browser.test.tsx new file mode 100644 index 00000000000..c839470cab1 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.browser.test.tsx @@ -0,0 +1,191 @@ +/* oxlint-disable react-perf/jsx-no-new-object-as-prop */ +import type { GenerateModelConfig, GenerateSettings, MainModelConfig } from '@features/generation/core/types'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { getDefaultGenerateSettings } from '@features/generation/core/baseGenerationPolicies'; +import { createExternalStoreCore } from '@platform/state/externalStoreCore'; +import { system } from '@theme/system'; +import { createInstance } from 'i18next'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { I18nextProvider, initReactI18next } from 'react-i18next'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { GenerateProviderOptionsSection } from './GenerateProviderOptionsSection'; + +vi.mock('./GenerationUiContext', () => ({ + useGenerationUi: () => ({ + project: { invocationSourceId: 'generate' }, + sectionPreferences: { sectionsOpen: { providerOptions: true }, setSectionOpen: vi.fn() }, + }), +})); + +const i18n = createInstance(); +void i18n.use(initReactI18next).init({ + fallbackLng: 'en', + initAsync: false, + lng: 'en', + resources: { + en: { + translation: { + common: { scrubber: { editValue: 'Edit {{label}}' } }, + widgets: { + generate: { + override: 'Override', + overridesCount_one: '{{count}} override', + overridesCount_other: '{{count}} overrides', + providerOptions: { + geminiTemperature: 'Temperature', + geminiThinkingLevel: 'Thinking level', + geminiThinkingLevelOptions: { high: 'High', minimal: 'Minimal' }, + openaiBackground: 'Background', + openaiBackgroundOptions: { auto: 'Auto', opaque: 'Opaque', transparent: 'Transparent' }, + openaiInputFidelity: 'Input fidelity', + openaiInputFidelityOptions: { high: 'High', low: 'Low' }, + openaiQuality: 'Quality', + openaiQualityOptions: { auto: 'Auto', high: 'High', low: 'Low', medium: 'Medium' }, + providerDefault: 'Provider default', + seedreamOptimizePrompt: 'Optimize prompt', + seedreamWatermark: 'Watermark', + title: 'Provider options', + }, + }, + }, + }, + }, + }, +}); + +const external = ( + providerId: string, + capabilities: Record, + providerModelId = providerId === 'openai' ? 'gpt-image-1' : `${providerId}-model` +): GenerateModelConfig => + ({ + base: 'external', + capabilities, + format: 'external_api', + key: `${providerId}-model`, + name: providerId, + provider_id: providerId, + provider_model_id: providerModelId, + type: 'external_image_generator', + }) as GenerateModelConfig; +const sd1Model: MainModelConfig = { base: 'sd-1', key: 'sd1', name: 'SD 1.5', type: 'main' }; + +let host: HTMLDivElement | null = null; +let root: Root | null = null; +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const settle = (action: () => void): Promise => + act(async () => { + action(); + await new Promise((resolve) => { + globalThis.setTimeout(resolve, 50); + }); + }); + +const render = async (model: GenerateModelConfig, settings: Partial = {}) => { + host = document.createElement('div'); + document.body.append(host); + root = createRoot(host); + const draft = createExternalStoreCore({ ...getDefaultGenerateSettings(model), ...settings }); + // Commits land in the draft as the form's would, so the section re-renders from what it committed. + const onCommit = vi.fn((patch: Partial) => draft.setSnapshot({ ...draft.getSnapshot(), ...patch })); + + await settle(() => { + root?.render( + + + + + + ); + }); + + return onCommit; +}; + +const selectTriggers = () => [ + ...(host?.querySelectorAll('[data-scope="select"][data-part="trigger"]') ?? []), +]; +const selectNames = () => + selectTriggers().map((trigger) => + (trigger.getAttribute('aria-labelledby') ?? '') + .split(' ') + .map((id) => document.getElementById(id)?.textContent?.trim() ?? '') + .join(' ') + .trim() + ); +const switches = () => [...(host?.querySelectorAll('input[type="checkbox"]') ?? [])]; +const option = (label: string) => + [...document.querySelectorAll('[role="option"]')].find((item) => item.textContent === label) ?? null; + +afterEach(async () => { + await settle(() => root?.unmount()); + host?.remove(); + host = null; + root = null; +}); + +describe('GenerateProviderOptionsSection', () => { + it('renders nothing for local models and providers without options', async () => { + await render(sd1Model); + expect(host?.childElementCount).toBe(0); + + await settle(() => root?.unmount()); + await render(external('alibabacloud', { modes: ['txt2img'] })); + expect(host?.childElementCount).toBe(0); + + // OpenAI sends its options to GPT Image models only. + await settle(() => root?.unmount()); + await render(external('openai', { modes: ['txt2img'] }, 'dall-e-3')); + expect(host?.childElementCount).toBe(0); + }); + + it('offers OpenAI quality and background, adding input fidelity only for image inputs', async () => { + await render(external('openai', { modes: ['txt2img'] })); + expect(selectNames()).toEqual(['Quality', 'Background']); + + await settle(() => root?.unmount()); + const onCommit = await render(external('openai', { modes: ['txt2img', 'img2img'] })); + expect(selectNames()).toEqual(['Quality', 'Background', 'Input fidelity']); + + await settle(() => selectTriggers()[1]?.click()); + await settle(() => option('Transparent')?.click()); + expect(onCommit).toHaveBeenCalledWith({ openaiBackground: 'transparent' }); + + await settle(() => selectTriggers()[2]?.click()); + await settle(() => option('High')?.click()); + expect(onCommit).toHaveBeenCalledWith({ openaiInputFidelity: 'high' }); + }); + + it('lets Gemini temperature be overridden and thinking level return to the provider default', async () => { + const onCommit = await render(external('gemini', { modes: ['txt2img'] }), { geminiThinkingLevel: 'high' }); + + expect(selectNames()).toEqual(['Thinking level']); + // Temperature starts at the provider default: no slider until overridden. + expect(host?.querySelector('[role="slider"]')).toBeNull(); + + await settle(() => switches()[0]?.click()); + expect(onCommit).toHaveBeenCalledWith({ geminiTemperature: 1 }); + expect(host?.querySelector('[role="slider"]')?.getAttribute('aria-valuenow')).toBe('1'); + + await settle(() => selectTriggers()[0]?.click()); + await settle(() => option('Provider default')?.click()); + expect(onCommit).toHaveBeenCalledWith({ geminiThinkingLevel: null }); + }); + + it('toggles the Seedream flags and counts them as overrides', async () => { + const onCommit = await render(external('seedream', { modes: ['txt2img'] }), { seedreamWatermark: true }); + + expect(host?.textContent).toContain('1 override'); + const [watermark, optimizePrompt] = switches(); + + expect(watermark?.checked).toBe(true); + expect(optimizePrompt?.checked).toBe(false); + + await settle(() => optimizePrompt?.click()); + expect(onCommit).toHaveBeenCalledWith({ seedreamOptimizePrompt: true }); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.tsx new file mode 100644 index 00000000000..56a12cdfdb7 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateProviderOptionsSection.tsx @@ -0,0 +1,260 @@ +/* oxlint-disable react-perf/jsx-no-new-object-as-prop, react-perf/jsx-no-new-function-as-prop, react-perf/jsx-no-new-array-as-prop, react-perf/jsx-no-jsx-as-prop */ +import type { GenerateModelConfig, GenerateSettings } from '@features/generation/core/types'; + +import { Badge, createListCollection, Stack } from '@chakra-ui/react'; +import { + DEFAULT_GEMINI_TEMPERATURE, + GEMINI_TEMPERATURE_MAX, + GEMINI_TEMPERATURE_MIN, + GEMINI_THINKING_LEVELS, + getExternalProviderOptionKeys, + isGeminiThinkingLevel, + isOpenAIImageBackground, + isOpenAIImageQuality, + isOpenAIInputFidelity, + OPENAI_IMAGE_BACKGROUNDS, + OPENAI_IMAGE_QUALITIES, + OPENAI_INPUT_FIDELITIES, +} from '@features/generation/core/externalProvider'; +import { useExternalStoreSelector } from '@platform/state/selectors'; +import { Field } from '@platform/ui/Field'; +import { ScrubberField } from '@platform/ui/ScrubberField'; +import { Select } from '@platform/ui/Select'; +import { useMemo } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { type GenerateDraft, pickGenerateSettings } from './generateDebounce'; +import { GenerateCollapsibleSection } from './shared/GenerateCollapsibleSection'; +import { GenerateToggleSwitch } from './shared/GenerateToggleSwitch'; + +interface GenerateProviderOptionsSectionProps { + draft: GenerateDraft; + selectedModel: GenerateModelConfig | undefined; + onCommit: (patch: Partial) => void; +} + +const selectProviderSettings = pickGenerateSettings([ + 'geminiTemperature', + 'geminiThinkingLevel', + 'openaiBackground', + 'openaiInputFidelity', + 'openaiQuality', + 'seedreamOptimizePrompt', + 'seedreamWatermark', +]); + +/** The empty value stands for "let the provider decide" on the optional selects. */ +const PROVIDER_DEFAULT = ''; + +/** Options a hosted provider's node accepts beyond the shared prompt/size/seed fields; hidden for local models. */ +export const GenerateProviderOptionsSection = ({ + draft, + onCommit, + selectedModel, +}: GenerateProviderOptionsSectionProps) => { + const { t } = useTranslation(); + const settings = useExternalStoreSelector(draft.subscribe, draft.getSnapshot, selectProviderSettings); + const optionKeys = getExternalProviderOptionKeys(selectedModel); + const collections = useMemo( + () => ({ + background: createListCollection({ + items: OPENAI_IMAGE_BACKGROUNDS.map((value) => ({ + label: t(`widgets.generate.providerOptions.openaiBackgroundOptions.${value}`), + value, + })), + }), + inputFidelity: createListCollection({ + items: [ + { label: t('widgets.generate.providerOptions.providerDefault'), value: PROVIDER_DEFAULT }, + ...OPENAI_INPUT_FIDELITIES.map((value) => ({ + label: t(`widgets.generate.providerOptions.openaiInputFidelityOptions.${value}`), + value, + })), + ], + }), + quality: createListCollection({ + items: OPENAI_IMAGE_QUALITIES.map((value) => ({ + label: t(`widgets.generate.providerOptions.openaiQualityOptions.${value}`), + value, + })), + }), + thinkingLevel: createListCollection({ + items: [ + { label: t('widgets.generate.providerOptions.providerDefault'), value: PROVIDER_DEFAULT }, + ...GEMINI_THINKING_LEVELS.map((value) => ({ + label: t(`widgets.generate.providerOptions.geminiThinkingLevelOptions.${value}`), + value, + })), + ], + }), + }), + [t] + ); + + if (optionKeys.length === 0) { + return null; + } + + // Count decisions away from the provider defaults, as the Advanced section does. + const overrideCount = optionKeys.filter((key) => { + switch (key) { + case 'openaiQuality': + case 'openaiBackground': + return settings[key] !== 'auto'; + case 'seedreamWatermark': + case 'seedreamOptimizePrompt': + return settings[key]; + default: + return settings[key] !== null; + } + }).length; + + return ( + 0 ? {t('widgets.generate.overridesCount', { count: overrideCount })} : null + } + sectionId="providerOptions" + > + + {optionKeys.includes('openaiQuality') ? ( + + { + if (isOpenAIImageBackground(value[0])) { + onCommit({ openaiBackground: value[0] }); + } + }} + /> + + ) : null} + {optionKeys.includes('openaiInputFidelity') ? ( + + { + const level = value[0]; + + if (level === PROVIDER_DEFAULT) { + onCommit({ geminiThinkingLevel: null }); + } else if (isGeminiThinkingLevel(level)) { + onCommit({ geminiThinkingLevel: level }); + } + }} + /> + + ) : null} + {optionKeys.includes('seedreamWatermark') ? ( + + onCommit({ seedreamWatermark: checked })} + /> + + ) : null} + {optionKeys.includes('seedreamOptimizePrompt') ? ( + + onCommit({ seedreamOptimizePrompt: checked })} + /> + + ) : null} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.browser.test.tsx index 79ae3ab7727..63500f16357 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.browser.test.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.browser.test.tsx @@ -1,5 +1,5 @@ /* oxlint-disable react-perf/jsx-no-new-object-as-prop */ -import type { GenerateSettings, MainModelConfig } from '@features/generation/core/types'; +import type { GenerateModelConfig, GenerateSettings, MainModelConfig } from '@features/generation/core/types'; import { ChakraProvider } from '@chakra-ui/react'; import { seedArchitectureCapabilities } from '@features/generation/core/architectureCapabilities.testing'; @@ -172,7 +172,7 @@ const settle = (action: () => void): Promise => }); }); -const render = async (model: MainModelConfig, settings: Partial = {}) => { +const render = async (model: GenerateModelConfig, settings: Partial = {}) => { host = document.createElement('div'); document.body.append(host); root = createRoot(host); @@ -317,6 +317,22 @@ describe('GenerateRenderSection denoising strength', () => { expect(guidanceIndex).toBeGreaterThan(-1); expect(controls[guidanceIndex + 1]?.getAttribute('data-testid')).toBe('strength-field'); }); + + it('adds no strength for a hosted provider, which edits the canvas composite at its own', async () => { + harness.invocationSourceId = 'canvas'; + await render({ + base: 'external', + capabilities: { modes: ['txt2img', 'img2img'] }, + format: 'external_api', + key: 'gpt-image', + name: 'GPT Image', + provider_id: 'openai', + type: 'external_image_generator', + }); + + expect(host?.querySelector('[data-testid="strength-field"]')).toBeNull(); + expect(host?.querySelector('[data-testid="strength-badge"]')).toBeNull(); + }); }); /** Resolve names through both aria-labelledby and aria-label. */ diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.tsx index 34e3d698896..5c2da8d7a9c 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateRenderSection.tsx @@ -686,8 +686,9 @@ export const GenerateRenderSection = ({ ); - // Switching source remounts the section's subtree, as the Size section does. - return project.invocationSourceId === 'canvas' ? ( + // Switching source remounts the section's subtree, as the Size section does. A hosted provider edits the canvas + // composite at its own strength, so the canvas adds none for it. + return project.invocationSourceId === 'canvas' && selectedModel?.type !== 'external_image_generator' ? ( {renderSection} ) : ( renderSection(null) diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.browser.test.tsx index 8d4e2cfa291..9876b97bdf0 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.browser.test.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.browser.test.tsx @@ -685,6 +685,25 @@ describe('GenerateSettingsForm render isolation', () => { expect(stepsValue()).toBe(31); }); + it('switches at once when only the size moves, reporting the new size', async () => { + const info = vi.fn(); + stableGroups.notifications.info = info; + stableGroups.models.catalog = [MODEL, FLUX]; + await renderAdapter(buildAdapter()); + // 1032 sits on SDXL's 8 px grid but not FLUX's 16 px one. + await pressRight(sliderNamed('widgets.generate.width')); + + await settle(() => host!.querySelector('[data-model-pick="flux"]')?.click()); + + expect(document.querySelector('[role="alertdialog"]')).toBeNull(); + expect(patches.at(-1)?.patch).toMatchObject({ modelKey: 'flux' }); + expect((patches.at(-1)?.patch.width as number) % 16).toBe(0); + expect(info).toHaveBeenCalledWith( + 'widgets.generate.dimensionsAdjusted', + 'widgets.generate.dimensionsAdjustedDescription' + ); + }); + it('confirms a lossy switch against the draft, pending edits included', async () => { stableGroups.models.catalog = [MODEL, FLUX, ink]; storedValues().patchSnapshot({ loras: [{ isEnabled: true, model: ink, weight: 0.75 }] }); @@ -696,7 +715,9 @@ describe('GenerateSettingsForm render isolation', () => { await settle(() => host!.querySelector('[data-model-pick="flux"]')?.click()); const dialog = document.querySelector('[role="alertdialog"]'); - expect(dialog?.textContent).toContain('widgets.generate.switchModelBody: Dimensions and LoRAs'); + // The size moves to the FLUX grid, which the dialog reports beside the loss rather than as one. + expect(dialog?.textContent).toContain('widgets.generate.switchModelBody: LoRAs'); + expect(dialog?.textContent).toContain('widgets.generate.switchModelAdjustsDimensions'); expect(patches).toEqual([]); const confirm = [...dialog!.querySelectorAll('button')].find( @@ -707,7 +728,7 @@ describe('GenerateSettingsForm render isolation', () => { // The switch lands as the dialog closes, and the dialog animates out still listing what was confirmed. expect(frames).not.toHaveLength(0); for (const frame of frames) { - expect(frame.text).toContain('widgets.generate.switchModelBody: Dimensions and LoRAs'); + expect(frame.text).toContain('widgets.generate.switchModelBody: LoRAs'); } await settle(noop, 400); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.tsx b/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.tsx index 981f5770eb3..31d9533907a 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.tsx +++ b/invokeai/frontend/webv2/src/features/generation/ui/GenerateSettingsForm.tsx @@ -22,6 +22,7 @@ import { useRegisterGenerateDraftFlusher } from './generateDraftRegistry'; import { getSettingsWithLatestPromptFields } from './generateFormViewModel'; import { GenerateGuidanceSection } from './GenerateGuidanceSection'; import { GenerateModelCard } from './GenerateModelCard'; +import { GenerateProviderOptionsSection } from './GenerateProviderOptionsSection'; import { GenerateRefinerSection } from './GenerateRefinerSection'; import { GenerateRenderSection } from './GenerateRenderSection'; import { GeneratePromptFields } from './promptFields'; @@ -279,6 +280,8 @@ export const GenerateSettingsForm = ({ onCommitImmediate={commitPatchImmediately} /> + + vi.fn((key: string, options?: Record) => { @@ -71,3 +72,38 @@ describe('notifyGenerateModelSelectionCleared', () => { ); }); }); + +describe('notifyGenerateDimensionsAdjusted', () => { + const t = vi.fn((key: string, options?: Record) => + options ? `${key} ${JSON.stringify(options)}` : key + ) as unknown as TFunction; + + it('names the size the switch moved to', () => { + const info = vi.fn(); + + notifyGenerateDimensionsAdjusted({ + modelName: 'Seedream 4.5', + notifications: { info }, + result: { dimensionsAdjusted: true, settings: { height: 1600, width: 2848 } as GenerateSettings }, + t, + }); + + expect(info).toHaveBeenCalledWith( + 'widgets.generate.dimensionsAdjusted', + 'widgets.generate.dimensionsAdjustedDescription {"height":1600,"name":"Seedream 4.5","width":2848}' + ); + }); + + it('stays quiet when the size already fit', () => { + const info = vi.fn(); + + notifyGenerateDimensionsAdjusted({ + modelName: 'SDXL', + notifications: { info }, + result: { dimensionsAdjusted: false, settings: { height: 1024, width: 1024 } as GenerateSettings }, + t, + }); + + expect(info).not.toHaveBeenCalled(); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/modelSelectionNotice.ts b/invokeai/frontend/webv2/src/features/generation/ui/modelSelectionNotice.ts index b64052a88df..396b41d1e85 100644 --- a/invokeai/frontend/webv2/src/features/generation/ui/modelSelectionNotice.ts +++ b/invokeai/frontend/webv2/src/features/generation/ui/modelSelectionNotice.ts @@ -1,3 +1,4 @@ +import type { GenerateModelSelectionResult } from '@features/generation/core/baseGenerationPolicies'; import type { TFunction } from 'i18next'; interface GenerateModelSelectionNotifier { @@ -39,3 +40,29 @@ export const notifyGenerateModelSelectionCleared = ({ ) ); }; + +/** A switch that only moves the size is not confirmed, so this names the size it moved to. */ +export const notifyGenerateDimensionsAdjusted = ({ + modelName, + notifications, + result, + t, +}: { + modelName: string; + notifications: GenerateModelSelectionNotifier; + result: Pick; + t: TFunction; +}): void => { + if (!result.dimensionsAdjusted) { + return; + } + + notifications.info( + t('widgets.generate.dimensionsAdjusted'), + t('widgets.generate.dimensionsAdjustedDescription', { + height: result.settings.height, + name: modelName, + width: result.settings.width, + }) + ); +}; diff --git a/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.browser.test.tsx b/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.browser.test.tsx new file mode 100644 index 00000000000..c128494d519 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.browser.test.tsx @@ -0,0 +1,182 @@ +/* oxlint-disable react-perf/jsx-no-new-object-as-prop */ +import type { GenerateModelConfig } from '@features/generation/core/types'; + +import { ChakraProvider } from '@chakra-ui/react'; +import { system } from '@theme/system'; +import { createInstance } from 'i18next'; +import { act } from 'react'; +import { createRoot, type Root } from 'react-dom/client'; +import { I18nextProvider, initReactI18next } from 'react-i18next'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { ExternalSizeSelect } from './ExternalSizeSelect'; + +const i18n = createInstance(); +void i18n.use(initReactI18next).init({ + fallbackLng: 'en', + initAsync: false, + lng: 'en', + resources: { + en: { + translation: { + widgets: { + generate: { + providerSize: 'Provider size', + providerSizeCustom: 'Custom', + providerSizeCanvasCropHelp: 'Canvas generates {{width}}×{{height}}, cropped to the frame; edits resized.', + providerSizeCanvasCropTextHelp: 'Canvas generates {{width}}×{{height}}, cropped to the frame.', + providerSizeCanvasResizeHelp: 'Canvas generates {{width}}×{{height}}, resized to the frame.', + providerSizeHelp: 'Sizes this provider renders natively.', + }, + }, + }, + }, + }, +}); + +const external = (capabilities: Record): GenerateModelConfig => + ({ + base: 'external', + capabilities, + format: 'external_api', + key: 'external', + name: 'External', + provider_id: 'gemini', + type: 'external_image_generator', + }) as GenerateModelConfig; + +const presetModel = external({ + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, + ], +}); +const ratioModel = external({ + aspect_ratio_sizes: { '1:1': { height: 1024, width: 1024 }, '7:4': { height: 1024, width: 1792 } }, + modes: ['txt2img'], +}); + +let host: HTMLDivElement | null = null; +let root: Root | null = null; +(globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; + +const settle = (action: () => void): Promise => + act(async () => { + action(); + await new Promise((resolve) => { + globalThis.setTimeout(resolve, 50); + }); + }); + +const render = async ( + model: GenerateModelConfig, + settings: { externalImageSize: string | null; height: number; width: number }, + isCanvasFrame = false +) => { + host = document.createElement('div'); + document.body.append(host); + root = createRoot(host); + const onCommit = vi.fn(); + + await settle(() => { + root?.render( + + + + + + ); + }); + + return onCommit; +}; + +const trigger = () => host?.querySelector('[data-scope="select"][data-part="trigger"]') ?? null; +const option = (label: string) => + [...document.querySelectorAll('[role="option"]')].find((item) => item.textContent === label) ?? null; + +afterEach(async () => { + await settle(() => root?.unmount()); + host?.remove(); + host = null; + root = null; +}); + +describe('ExternalSizeSelect', () => { + it('renders nothing for providers without a size table', async () => { + await render(external({ modes: ['txt2img'] }), { externalImageSize: null, height: 1024, width: 1024 }); + + expect(host?.childElementCount).toBe(0); + }); + + it('applies a named preset as a locked size with its image size', async () => { + const onCommit = await render(presetModel, { externalImageSize: '1K', height: 1024, width: 1024 }); + + expect(trigger()?.textContent).toContain('1:1 (1K)'); + await settle(() => trigger()?.click()); + await settle(() => option('16:9 (2K)')?.click()); + + expect(onCommit).toHaveBeenCalledWith({ + aspectRatioId: '16:9', + aspectRatioIsLocked: true, + aspectRatioValue: 2048 / 1152, + externalImageSize: '2K', + height: 1152, + width: 2048, + }); + }); + + it('recognizes dimensions typed to a provider preset without a stored name', async () => { + await render(presetModel, { externalImageSize: null, height: 1152, width: 2048 }); + + expect(trigger()?.textContent).toContain('16:9 (2K)'); + }); + + it('shows a hand-typed size as custom', async () => { + await render(presetModel, { externalImageSize: '1K', height: 1000, width: 1024 }); + + expect(trigger()?.textContent).toContain('Custom'); + expect(host?.textContent).toContain('Sizes this provider renders natively.'); + }); + + it('tells a Canvas frame off the size table which size is generated and how it is fitted', async () => { + // A text-only model is never edited, so the hint leaves edits out. + await render(ratioModel, { externalImageSize: null, height: 512, width: 640 }, true); + expect(host?.textContent).toContain('Canvas generates 1024×1024, cropped to the frame.'); + + await settle(() => root?.unmount()); + host?.remove(); + await render( + external({ + aspect_ratio_sizes: { '1:1': { height: 1024, width: 1024 }, '7:4': { height: 1024, width: 1792 } }, + modes: ['txt2img', 'img2img'], + }), + { externalImageSize: null, height: 512, width: 640 }, + true + ); + expect(host?.textContent).toContain('Canvas generates 1024×1024, cropped to the frame; edits resized.'); + + await settle(() => root?.unmount()); + host?.remove(); + await render(ratioModel, { externalImageSize: null, height: 512, width: 896 }, true); + expect(host?.textContent).toContain('Canvas generates 1792×1024, resized to the frame.'); + }); + + it('applies a fixed ratio size, falling back to a free ratio id for shapes outside the preset list', async () => { + const onCommit = await render(ratioModel, { externalImageSize: null, height: 1024, width: 1024 }); + + expect(trigger()?.textContent).toContain('1:1 (1024×1024)'); + await settle(() => trigger()?.click()); + await settle(() => option('7:4 (1792×1024)')?.click()); + + expect(onCommit).toHaveBeenCalledWith({ + aspectRatioId: 'Free', + aspectRatioIsLocked: true, + aspectRatioValue: 1792 / 1024, + externalImageSize: null, + height: 1024, + width: 1792, + }); + }); +}); diff --git a/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.tsx b/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.tsx new file mode 100644 index 00000000000..55760f14609 --- /dev/null +++ b/invokeai/frontend/webv2/src/features/generation/ui/shared/ExternalSizeSelect.tsx @@ -0,0 +1,125 @@ +/* oxlint-disable react-perf/jsx-no-new-object-as-prop, react-perf/jsx-no-new-function-as-prop, react-perf/jsx-no-new-array-as-prop */ +import type { GenerateModelConfig, GenerateSettings } from '@features/generation/core/types'; + +import { createListCollection } from '@chakra-ui/react'; +import { + findExternalResolutionPreset, + getExternalAspectRatioSizes, + getExternalResolutionPresets, + isExternalTextOnlyCanvasModel, + resolveExternalCanvasRequestSize, +} from '@features/generation/core/externalProvider'; +import { isAspectRatioId } from '@features/generation/core/settings'; +import { Field } from '@platform/ui/Field'; +import { Select } from '@platform/ui/Select'; +import { useMemo } from 'react'; +import { useTranslation } from 'react-i18next'; + +type SizeSettings = Pick; + +interface ExternalSizeSelectProps { + /** The size is the Canvas frame, which the Canvas generates at the provider size nearest it. */ + isCanvasFrame: boolean; + model: GenerateModelConfig | undefined; + settings: SizeSettings; + onCommit: (patch: Partial) => void; +} + +const presetKey = (preset: { aspect_ratio: string; image_size: string }): string => + `${preset.aspect_ratio}|${preset.image_size}`; + +/** The size patch a provider entry implies; its ratio is locked so the width/height fields keep it. */ +const sizePatch = (ratio: string, width: number, height: number): Partial => ({ + aspectRatioId: isAspectRatioId(ratio) ? ratio : 'Free', + aspectRatioIsLocked: true, + aspectRatioValue: height > 0 ? width / height : 1, + height, + width, +}); + +/** + * A hosted provider's own size table: named sizes per ratio (Gemini 3's `1K`/`2K`/`4K`), or one fixed size per + * ratio. Renders nothing for free-form providers and local models. A size typed by hand shows as custom. + */ +export const ExternalSizeSelect = ({ isCanvasFrame, model, onCommit, settings }: ExternalSizeSelectProps) => { + const { t } = useTranslation(); + // One table per model: the policy reads fresh arrays off the record, so the record is the memo key. + const { collection, presets, ratioSizes } = useMemo(() => { + const presets = getExternalResolutionPresets(model); + const ratioSizes = getExternalAspectRatioSizes(model); + + return { + collection: createListCollection({ + items: + presets.length > 0 + ? presets.map((preset) => ({ label: preset.label, value: presetKey(preset) })) + : ratioSizes.map(([ratio, size]) => ({ label: `${ratio} (${size.width}×${size.height})`, value: ratio })), + }), + presets, + ratioSizes, + }; + }, [model]); + + if (presets.length === 0 && ratioSizes.length === 0) { + return null; + } + + const currentPreset = presets.length > 0 ? findExternalResolutionPreset(settings, model) : null; + const currentRatio = + presets.length > 0 + ? null + : (ratioSizes.find(([, size]) => size.width === settings.width && size.height === settings.height)?.[0] ?? null); + const value = currentPreset ? [presetKey(currentPreset)] : currentRatio ? [currentRatio] : []; + // Hosted providers have no grid, so on a Canvas route the width and height are the frame itself. + const canvasRequest = + isCanvasFrame && model && value.length === 0 ? resolveExternalCanvasRequestSize(model, settings) : null; + const canvasHelpKey = + canvasRequest && canvasRequest.width * settings.height === canvasRequest.height * settings.width + ? 'widgets.generate.providerSizeCanvasResizeHelp' + : model && isExternalTextOnlyCanvasModel(model) + ? 'widgets.generate.providerSizeCanvasCropTextHelp' + : 'widgets.generate.providerSizeCanvasCropHelp'; + + return ( + + { + if (value[0] !== undefined) { + setField('providerId', value[0]); + } + }} + /> + ) : ( + setField('providerId', event.currentTarget.value)} + /> + )} + + + setField('providerModelId', event.currentTarget.value)} + /> + + + {!canEdit ? ( + + {starters === null ? null : ( + + {t('models.externalFields.starterManaged')} + + )} + + values[key]) + .map(({ labelKey }) => t(labelKey)) + .join(', ')} + /> + {SUPPORT_FLAGS.map(({ key, labelKey }) => ( + + ))} + {LIMIT_FIELDS.map(({ key, labelKey }) => ( + + ))} + {DEFAULT_FIELDS.map(({ key, labelKey }) => ( + + ))} + + + ) : ( + <> + + + {t('models.externalFields.supportedModes')} + + + {MODE_FLAGS.map(({ key, labelKey }) => ( + setField(key, event.checked === true)} + > + + + {t(labelKey)} + + ))} + + {modesError ? ( + + {modesError} + + ) : ( + + {t('models.externalFields.supportedModesHelp')} + + )} + + + {SUPPORT_FLAGS.map(({ key, labelKey }) => ( + + + + {t(labelKey)} + + setField(key, event.checked)} + > + + + + + {t(labelKey)} + + + + ))} + + + {LIMIT_FIELDS.map((field) => numberField(field, 'models.externalFields.unlimited'))} + + + {DEFAULT_FIELDS.map((field) => numberField(field, 'models.externalFields.providerDefault'))} + + + )} + + ); +}; diff --git a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx index 7069f44e8bf..b8fb328736d 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx +++ b/invokeai/frontend/webv2/src/features/models/ui/detail/ModelDetail.tsx @@ -4,6 +4,10 @@ import type { ReactNode } from 'react'; import { chakra, DataList, HStack, Icon, Menu, Portal, Separator, Stack, Text } from '@chakra-ui/react'; import { isConvertibleToDiffusers } from '@features/models/core/baseIdentity'; +import { + supportsExternalModelSettings, + type ExternalModelSettingsModel, +} from '@features/models/core/externalModelFields'; import { isLinkableType } from '@features/models/core/relationships'; import { isAbsoluteModelPath, resolveModelAbsolutePath } from '@features/models/core/schemas'; import { getModelSourceHref, getUnrecordedModelFact } from '@features/models/core/taxonomy'; @@ -29,6 +33,7 @@ import { useTranslation } from 'react-i18next'; import { CpuOnlySetting, supportsCpuOnlySetting } from './CpuOnlySetting'; import { supportsDefaultSettings, type DefaultSettingsModel } from './defaultSettingsFields'; import { DefaultSettingsSection } from './DefaultSettingsSection'; +import { ExternalModelSection } from './ExternalModelSection'; import { MissingFileBadge, ModelBaseBadge, ModelFormatBadge } from './ModelBadges'; import { ModelEditForm } from './ModelEditForm'; import { ModelImageUpload } from './ModelImageUpload'; @@ -130,6 +135,23 @@ const selectDefaultSettingsModel = (snapshot: ModelsSnapshot, modelKey: string): : null; }; +const selectExternalModel = (snapshot: ModelsSnapshot, modelKey: string): ExternalModelSettingsModel | null => { + const model = findModel(snapshot, modelKey); + + return model + ? { + capabilities: model.capabilities, + default_settings: model.default_settings, + key: model.key, + name: model.name, + provider_id: model.provider_id, + provider_model_id: model.provider_model_id, + source: model.source, + type: model.type, + } + : null; +}; + const selectTriggerPhrasesModel = (snapshot: ModelsSnapshot, modelKey: string): TriggerPhrasesModel | null => { const model = findModel(snapshot, modelKey); @@ -182,6 +204,13 @@ export const ModelDetail = ({ ) : null} + {supportsExternalModelSettings(model) ? ( + <> + + + + ) : null} + {supportsDefaultSettings(model) ? ( <> @@ -380,6 +409,24 @@ const DefaultSettingsSectionContainer = memo(function DefaultSettingsSectionCont ); }); +const ExternalModelSectionContainer = memo(function ExternalModelSectionContainer({ modelKey }: { modelKey: string }) { + const notify = useNotify(); + const { t } = useTranslation(); + const model = useModelsSelector((snapshot) => selectExternalModel(snapshot, modelKey)); + + if (!model || !supportsExternalModelSettings(model)) { + return null; + } + + return ( + notify.error(t('models.externalProvider'), message)} + onSaved={() => notify.success(t('models.externalSettingsSaved'))} + /> + ); +}); + const CpuOnlySettingContainer = memo(function CpuOnlySettingContainer({ modelKey }: { modelKey: string }) { const notify = useNotify(); const { t } = useTranslation(); diff --git a/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.test.ts b/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.test.ts index ae4cff9f30e..0078aacbe4b 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.test.ts +++ b/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.test.ts @@ -1,5 +1,6 @@ import type { ModelConfig } from '@features/models/core/types'; +import { toExternalModelValues } from '@features/models/core/externalModelFields'; import { setModelsSnapshotForTests } from '@features/models/data/modelsStore'; import { accountLifecycle } from '@platform/state/accountLifecycle'; import { afterEach, beforeEach, describe, expect, it } from 'vitest'; @@ -47,6 +48,21 @@ describe('model drafts store', () => { expect(getModelDraft('m')?.identity?.fields).toEqual({ name: 'Draft 2' }); }); + it('counts hosted-provider record edits as unsaved changes and discards them like defaults', () => { + recordModelDraftFields( + 'm', + 'external', + { modeInpaint: true, providerId: 'openai' }, + { modeInpaint: false, providerId: 'openai' } + ); + + expect(getModelDraft('m')?.external?.fields).toEqual({ modeInpaint: true }); + expect(hasUnsavedModelChanges(getModelDraft('m'))).toBe(true); + + discardModelDraft('m', 'external'); + expect(getModelDraft('m')).toBeUndefined(); + }); + it('treats unset and null default settings as the same value', () => { recordModelDraftFields('m', 'defaults', { cfgScale: null, steps: 30 }, { steps: 30 }); @@ -197,6 +213,28 @@ describe('model drafts store', () => { expect(getModelDraft('other')).toBeUndefined(); }); + it('drops hosted-provider edits a refreshed record comes to agree with', () => { + const hosted = { + capabilities: { modes: ['txt2img'] }, + key: 'hosted', + name: 'hosted', + provider_id: 'openai', + provider_model_id: 'gpt-image-1', + type: 'external_image_generator', + } as ModelConfig; + const saved = toExternalModelValues(hosted); + + setModelsSnapshotForTests({ models: [hosted], status: 'loaded' }); + recordModelDraftFields('hosted', 'external', { ...saved, maxReferenceImages: 4, modeImg2img: true }, saved); + + setModelsSnapshotForTests({ + models: [{ ...hosted, capabilities: { modes: ['txt2img', 'img2img'] } }], + status: 'loaded', + }); + + expect(getModelDraft('hosted')?.external?.fields).toEqual({ maxReferenceImages: 4 }); + }); + it('drops drafts of models that leave the loaded library and keeps the rest', () => { setModelsSnapshotForTests({ models: [model('kept'), model('gone')], status: 'loaded' }); recordModelDraftFields('kept', 'defaults', { steps: 20 }, { steps: 30 }); diff --git a/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.ts b/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.ts index 1265343f89a..3448a62f34f 100644 --- a/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.ts +++ b/invokeai/frontend/webv2/src/features/models/ui/modelDraftsStore.ts @@ -1,6 +1,7 @@ import type { ModelEditFormValues } from '@features/models/core/schemas'; import type { ModelConfig } from '@features/models/core/types'; +import { toExternalModelValues } from '@features/models/core/externalModelFields'; import { getModelsSnapshot, subscribeModels } from '@features/models/data/modelsStore'; import { registerAccountOwnedResource } from '@platform/state/accountLifecycle'; import { createKeyedTransientStore } from '@platform/state/externalStore'; @@ -36,6 +37,8 @@ export interface ModelIdentityDraft extends ModelFieldsDraft { export interface ModelDraft { /** Default-settings edits; absent when nothing differs from the server and no save is in flight. */ readonly defaults?: ModelFieldsDraft; + /** Hosted-provider record edits, kept like `defaults`. */ + readonly external?: ModelFieldsDraft; /** Present exactly while the identity editor is open, even before anything is edited. */ readonly identity?: ModelIdentityDraft; } @@ -68,9 +71,12 @@ export const toModelIdentityValues = (model: ModelIdentitySource): ModelEditForm const SERVER_VALUES: Record object> = { defaults: (model) => model.default_settings ?? {}, + external: toExternalModelValues, identity: toModelIdentityValues, }; +const FIELD_SURFACES = ['identity', 'defaults', 'external'] as const satisfies readonly ModelDraftSurface[]; + // Unset and null are the same "no value" to the backend and to every form here. const isSameValue = (left: unknown, right: unknown): boolean => Object.is(left ?? null, right ?? null); @@ -93,7 +99,7 @@ export const hasModelDraftConflict = (server: object, draft: ModelFieldsDraft | ); export const hasUnsavedModelChanges = (draft: ModelDraft | undefined): boolean => - hasDraftFields(draft?.identity) || hasDraftFields(draft?.defaults); + FIELD_SURFACES.some((surface) => hasDraftFields(draft?.[surface])); /** * Whether a user value is still an edit: it differs from the server, or from a save in flight (a field typed back @@ -123,7 +129,7 @@ const evictBeyondBound = (): void => { }; const writeDraft = (modelKey: string, draft: ModelDraft, edited = false): void => { - if (draft.identity === undefined && draft.defaults === undefined) { + if (FIELD_SURFACES.every((surface) => draft[surface] === undefined)) { deleteDraft(modelKey); return; } @@ -224,7 +230,7 @@ const syncWithLibrary = (): void => { let draft = store.get(key); - for (const surface of ['identity', 'defaults'] as const) { + for (const surface of FIELD_SURFACES) { const surfaceDraft = draft?.[surface]; if (surfaceDraft !== undefined) { diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts index 258771e6078..6e13dcf1537 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts @@ -698,6 +698,7 @@ export { DEFAULT_CHECKER_COLORS } from './render/compositor'; export { getBaseRasterContentBounds, getCompositeLayerBounds, + isBaseRasterContentLayer, planBaseRasterComposite, type CompositeEntry, type CompositeLayerRef, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/rasterComposite.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/rasterComposite.ts index 4f4202a2ab8..82a9098ab15 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/rasterComposite.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/rasterComposite.ts @@ -2,6 +2,7 @@ import type { CanvasAdjustmentsContract, CanvasBlendMode, CanvasDocumentContractV3, + CanvasLayerContract, CanvasLayerSourceContract, CanvasRasterLayerContractV2, } from '@workbench/canvas-engine/contracts'; @@ -72,9 +73,9 @@ const defaultWriteImageData = (surface: RasterSurface, imageData: ImageData, x: surface.ctx.putImageData(imageData, x, y); /** True when a leaf is a contributing raster layer with rasterizable, non-empty pixels. */ -const isBaseRasterLeaf = (leaf: SemanticLeaf): leaf is SemanticLeaf & { layer: CanvasRasterLayerContractV2 } => { - const { layer } = leaf; - if (!leaf.contributionEnabled || layer.type !== 'raster') { +/** A raster layer whose source the generation composite draws: an image, painted pixels, or non-empty text. */ +export const isBaseRasterContentLayer = (layer: CanvasLayerContract): layer is CanvasRasterLayerContractV2 => { + if (layer.type !== 'raster') { return false; } if (layer.source.type === 'image') { @@ -86,6 +87,9 @@ const isBaseRasterLeaf = (leaf: SemanticLeaf): leaf is SemanticLeaf & { layer: C return layer.source.type === 'text' && layer.source.content.length > 0; }; +const isBaseRasterLeaf = (leaf: SemanticLeaf): leaf is SemanticLeaf & { layer: CanvasRasterLayerContractV2 } => + leaf.contributionEnabled && isBaseRasterContentLayer(leaf.layer); + /** JSON with recursively sorted object keys, so text edits produce deterministic identities. */ const stableSourceKey = (value: unknown): string => JSON.stringify(value, (_key, entry: unknown) => diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.test.ts index 4c08a0f6527..2387df857fa 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.test.ts @@ -195,6 +195,23 @@ describe('executeCompositePlan — mode geometry', () => { expect(result.bboxFullyCovered).toBe(false); }); + it('reports whether any composited pixel is visible, cached with the plan key', async () => { + const empty = makeHarness(() => uniformImageData(100, 100, 0)); + const plan = planComposites(makeDoc([rasterLayer('a')]), BBOX); + expect(await executeCompositePlan(plan, empty.deps)).toMatchObject({ + bboxFullyCovered: false, + bboxHasContent: false, + }); + // The second run reuses the cached entry and still knows the frame is empty. + expect(await executeCompositePlan(plan, empty.deps)).toMatchObject({ bboxHasContent: false }); + + const faint = makeHarness(() => uniformImageData(100, 100, 1)); + expect(await executeCompositePlan(plan, faint.deps)).toMatchObject({ + bboxFullyCovered: false, + bboxHasContent: true, + }); + }); + it('computes contentBounds as the union of layer bounds in document space', async () => { const harness = makeHarness(() => uniformImageData(100, 100, 255)); const doc = makeDoc([ diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.ts index 505386f3854..12728c34413 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/compositeForGeneration.ts @@ -50,6 +50,8 @@ export interface CompositeCacheEntry { height: number; pixelHash: string; bboxFullyCovered: boolean; + /** Whether any pixel is not fully transparent; masks store their coverage here too. */ + bboxHasContent: boolean; } /** Creates an empty {@link CompositeDedupeCache}. */ @@ -158,6 +160,8 @@ export interface CompositeResult { contentBounds: Rect | null; /** Whether the composited bbox surface is fully opaque (no transparent holes). */ bboxFullyCovered: boolean; + /** Whether any composited pixel in the bbox is not fully transparent. */ + bboxHasContent: boolean; } /** Document→bbox translate matrix (the "view" the entry is composited under). */ @@ -196,18 +200,20 @@ export const computeCompositeContentBounds = ( return getCompositeLayerBounds(layers); }; -/** True when every pixel of `imageData` is fully opaque (alpha === 255). Empty → false. */ -const isFullyOpaque = (imageData: ImageData): boolean => { +/** One alpha pass: every pixel fully opaque (empty → false), and any pixel not fully transparent. */ +const scanCoverage = (imageData: ImageData): { fullyOpaque: boolean; hasContent: boolean } => { const { data, height, width } = imageData; if (width <= 0 || height <= 0) { - return false; + return { fullyOpaque: false, hasContent: false }; } - for (let i = 3; i < data.length; i += 4) { - if (data[i] < 255) { - return false; - } + let fullyOpaque = true; + let hasContent = false; + for (let i = 3; i < data.length && (fullyOpaque || !hasContent); i += 4) { + const alpha = data[i] ?? 0; + fullyOpaque &&= alpha === 255; + hasContent ||= alpha > 0; } - return true; + return { fullyOpaque, hasContent }; }; /** @@ -232,7 +238,7 @@ const flattenControlSurfaceOverBlack = (surface: RasterSurface): void => { const executeRasterEntry = async ( entry: CompositeEntry, deps: ExecuteCompositePlanDeps -): Promise => { +): Promise => { const hashBlob = deps.hashBlob ?? defaultHashBlob; const readImageData = deps.readImageData ?? defaultReadImageData; @@ -240,6 +246,7 @@ const executeRasterEntry = async ( if (cached) { return { bboxFullyCovered: cached.bboxFullyCovered, + bboxHasContent: cached.bboxHasContent, height: cached.height, imageName: cached.imageName, key: entry.key, @@ -259,12 +266,16 @@ const executeRasterEntry = async ( try { const surface = await renderRasterComposite(entry, deps); let bboxFullyCovered: boolean; + let bboxHasContent: boolean; if (entry.kind === 'control-layer') { flattenControlSurfaceOverBlack(surface); bboxFullyCovered = surface.width > 0 && surface.height > 0; + bboxHasContent = bboxFullyCovered; } else { const fullRect: Rect = { height: surface.height, width: surface.width, x: 0, y: 0 }; - bboxFullyCovered = isFullyOpaque(readImageData(surface, fullRect)); + const coverage = scanCoverage(readImageData(surface, fullRect)); + bboxFullyCovered = coverage.fullyOpaque; + bboxHasContent = coverage.hasContent; } const blob = await deps.backend.encodeSurface(surface); @@ -280,6 +291,7 @@ const executeRasterEntry = async ( deps.dedupe.byKey.set(entry.key, { bboxFullyCovered, + bboxHasContent, height: upload.height, imageName: upload.imageName, pixelHash, @@ -288,6 +300,7 @@ const executeRasterEntry = async ( return { bboxFullyCovered, + bboxHasContent, height: upload.height, imageName: upload.imageName, key: entry.key, @@ -310,9 +323,9 @@ export const executeCompositePlan = async ( } const contentBounds = getCompositeLayerBounds(entry.layers); - const { bboxFullyCovered, ...base } = await executeRasterEntry(entry, deps); + const { bboxFullyCovered, bboxHasContent, ...base } = await executeRasterEntry(entry, deps); - return { base, bboxFullyCovered, contentBounds }; + return { base, bboxFullyCovered, bboxHasContent, contentBounds }; }; /** Composite one control layer over the bbox; the shared cache avoids re-uploading unchanged controls. */ @@ -320,7 +333,11 @@ export const executeControlComposite = async ( entry: CompositeEntry, deps: ExecuteCompositePlanDeps ): Promise => { - const { bboxFullyCovered: _bboxFullyCovered, ...result } = await executeRasterEntry(entry, deps); + const { + bboxFullyCovered: _bboxFullyCovered, + bboxHasContent: _bboxHasContent, + ...result + } = await executeRasterEntry(entry, deps); return result; }; @@ -329,7 +346,11 @@ export const executeRegionalMaskComposite = async ( entry: CompositeEntry, deps: ExecuteCompositePlanDeps ): Promise => { - const { bboxFullyCovered: _bboxFullyCovered, ...result } = await executeRasterEntry(entry, deps); + const { + bboxFullyCovered: _bboxFullyCovered, + bboxHasContent: _bboxHasContent, + ...result + } = await executeRasterEntry(entry, deps); return result; }; @@ -484,6 +505,7 @@ export const executeMaskComposite = async ( // Reuse the `bboxFullyCovered` slot to persist `hasContent` for this key. deps.dedupe.byKey.set(entry.key, { bboxFullyCovered: hasContent, + bboxHasContent: hasContent, height: upload.height, imageName: upload.imageName, pixelHash, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.test.ts index 11d074dac3a..2d44ae27b51 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.test.ts @@ -338,6 +338,7 @@ describe('composeForGeneration', () => { expect(detectMode).toHaveBeenCalledWith({ bbox: { height: 64, width: 64, x: 0, y: 0 }, bboxFullyCovered: true, + bboxHasContent: true, contentBounds: { height: 64, width: 64, x: 0, y: 0 }, hasActiveInpaintMask: false, }); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.ts index 53e6affb4e9..847e8af19ea 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/generationComposite.ts @@ -47,6 +47,8 @@ export interface GenerationModeFacts { contentBounds: Rect | null; /** Whether the composited bbox surface is fully opaque (no transparent holes). */ bboxFullyCovered: boolean; + /** Whether any composited pixel in the bbox is not fully transparent. */ + bboxHasContent: boolean; /** Whether an enabled inpaint mask with masked (non-white) content exists. */ hasActiveInpaintMask: boolean; } @@ -218,7 +220,6 @@ export const composeForGeneration = async ( if (contentBounds && intersect(contentBounds, bbox) !== null) { const result = await executeCompositePlan(plan, deps); - baseImageName = result.base.imageName; // Composite the denoise-limit mask before mode detection: even opaque raster content becomes inpaint when // the mask has coverage. @@ -228,9 +229,12 @@ export const composeForGeneration = async ( mode = options.detectMode({ bbox, bboxFullyCovered: result.bboxFullyCovered, + bboxHasContent: result.bboxHasContent, contentBounds, hasActiveInpaintMask: maskResult?.hasContent ?? false, }); + // A strategy may still settle on txt2img (no visible pixels in the frame), which takes no base image. + baseImageName = mode === 'txt2img' ? null : result.base.imageName; if (mode === 'inpaint' || mode === 'outpaint') { // A real (non-white) mask feeds create_gradient_mask; outpaint without one diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts index 0f721ed5178..b7d381aad1f 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts @@ -123,6 +123,14 @@ const setModel = (project: Project, base: GenerateWidgetValues['model']['base']) ideogram4GuidanceScale: null, ideogram4Mu: null, ideogram4ColorPalette: [], + externalImageSize: null, + openaiQuality: 'auto', + openaiBackground: 'auto', + openaiInputFidelity: null, + geminiTemperature: null, + geminiThinkingLevel: null, + seedreamWatermark: false, + seedreamOptimizePrompt: false, krea2RebalanceEnabled: false, krea2RebalanceMultiplier: 4, krea2RebalanceWeights: '1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.5,5.0,1.1,4.0,1.0', diff --git a/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.test.ts b/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.test.ts index deae8b74f6d..00b59bc37d8 100644 --- a/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.test.ts +++ b/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.test.ts @@ -43,6 +43,7 @@ describe('selectProjectGenerateModel', () => { }); const result = selectProjectGenerateModel({ + isCanvasFrame: false, currentValues, generation: { setSettings }, model, @@ -71,6 +72,7 @@ describe('selectProjectGenerateModel', () => { const setSettings = vi.fn(); const result = selectProjectGenerateModel({ + isCanvasFrame: false, currentValues: {}, generation: { setSettings }, model, diff --git a/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.ts b/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.ts index f63e37f4034..009cdf8f57d 100644 --- a/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.ts +++ b/invokeai/frontend/webv2/src/workbench/generationSettingsOrchestration.ts @@ -11,17 +11,19 @@ import { getGenerateModelSelectionResult, getPromptHistoryRecallPatch } from '@f export const selectProjectGenerateModel = ({ currentValues, generation, + isCanvasFrame, model, models, projectId, }: { currentValues: unknown; generation: Pick; + isCanvasFrame: boolean; model: GenerateModelConfig; models: readonly GenerationModelCatalogItem[]; projectId: string; }): GenerateModelSelectionResult => { - const result = getGenerateModelSelectionResult({ currentValues, model, models }); + const result = getGenerateModelSelectionResult({ currentValues, isCanvasFrame, model, models }); generation.setSettings({ ...result.settings, model }, projectId); diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.ts b/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.ts index ebfe6bafff8..26e2e3385bf 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.ts +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.ts @@ -192,6 +192,7 @@ export const useRegisterFirstPartyCommands = () => { commands, generateValues: getProjectWidgetValues(activeProject, 'generate'), image, + isCanvasFrame: activeProject.invocation.sourceId === 'canvas', kind, models: getAvailableModels() ?? [], owner, diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/executeImageRecall.ts b/invokeai/frontend/webv2/src/workbench/image-actions/executeImageRecall.ts index 3a93d0cacd0..54bc36c72e9 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/executeImageRecall.ts +++ b/invokeai/frontend/webv2/src/workbench/image-actions/executeImageRecall.ts @@ -296,6 +296,7 @@ export const executeImageRecall = async ({ generateValues, getGenerateValues, image, + isCanvasFrame, kind, models, owner: callerOwner, @@ -306,6 +307,8 @@ export const executeImageRecall = async ({ generateValues: Record; getGenerateValues?: () => Record; image: GalleryImage; + /** The project's route is Canvas, whose frame the Generate size is. */ + isCanvasFrame?: boolean; kind: ImageRecallKind; models: ModelConfig[]; /** Caller-captured identity lifetime; direct synchronous callers may omit it. */ @@ -350,6 +353,7 @@ export const executeImageRecall = async ({ const result = buildImageRecallSettings({ currentValues: currentGenerateValues, image, + isCanvasFrame, kind, metadata, models, diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.test.ts b/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.test.ts index 800ef1f7e97..89dabe39a73 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.test.ts +++ b/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.test.ts @@ -154,6 +154,14 @@ const createValues = (overrides: Partial = {}): GenerateWi ideogram4GuidanceScale: null, ideogram4Mu: null, ideogram4ColorPalette: [], + externalImageSize: null, + openaiQuality: 'auto', + openaiBackground: 'auto', + openaiInputFidelity: null, + geminiTemperature: null, + geminiThinkingLevel: null, + seedreamWatermark: false, + seedreamOptimizePrompt: false, krea2RebalanceEnabled: false, krea2RebalanceMultiplier: 4, krea2RebalanceWeights: '1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.5,5.0,1.1,4.0,1.0', @@ -587,8 +595,9 @@ describe('image recall', () => { expect(result?.values.aspectRatioId).toBe('2:3'); }); - it('recalls the size for an external generator, which has no architecture row to wait for', () => { - // External providers have no capability row and must remain eligible for dimension recall. + it('recalls the exact size for an external generator, which has no architecture row or grid', () => { + // External providers have no capability row and must remain eligible for dimension recall; their providers + // bucket sizes themselves, so the image's pixel size is kept rather than snapped to a local model grid. const externalModel = { base: 'external', capabilities: { modes: ['txt2img'], supports_seed: true }, @@ -609,11 +618,11 @@ describe('image recall', () => { vaeModels: [], }); - expect(recall()?.values).toMatchObject({ height: 768, width: 512 }); + expect(recall()?.values).toMatchObject({ height: 770, width: 513 }); resetArchitectureCapabilities(); try { - expect(recall()?.values).toMatchObject({ height: 768, width: 512 }); + expect(recall()?.values).toMatchObject({ height: 770, width: 513 }); } finally { setArchitectureCapabilities(architectureCapabilitiesFixture); } @@ -1206,6 +1215,283 @@ describe('image recall', () => { expect(capabilities.remix).toBe(true); }); }); + + describe('external provider options', () => { + const openaiModel = { + base: 'external', + capabilities: { modes: ['txt2img', 'img2img'], supports_reference_images: true }, + format: 'external_api', + key: 'gpt-image', + name: 'GPT Image', + provider_id: 'openai', + provider_model_id: 'gpt-image-1', + type: 'external_image_generator', + } as GenerateWidgetValues['model']; + const geminiModel = { + ...openaiModel, + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + ], + }, + key: 'gemini-3', + name: 'Gemini 3', + provider_id: 'gemini', + } as GenerateWidgetValues['model']; + const seedreamModel = { + ...openaiModel, + capabilities: { modes: ['txt2img'] }, + key: 'seedream', + name: 'Seedream', + provider_id: 'seedream', + } as GenerateWidgetValues['model']; + // What the provider nodes record (`_build_output_provider_metadata`) plus the named size the base node records. + const providerMetadata = { + gemini_temperature: 0.4, + gemini_thinking_level: 'high', + image_size: '2K', + openai_background: 'transparent', + openai_input_fidelity: 'high', + openai_quality: 'low', + seedream_optimize_prompt: true, + seedream_watermark: true, + }; + const recall = (model: GenerateWidgetValues['model'], metadata: Record, kind: 'all' | 'remix') => + buildImageRecallSettings({ + currentValues: createValues(), + image, + kind, + metadata: { ...metadata, model: { key: model.key } }, + models: [], + supportedModels: [model], + vaeModels: [], + }); + + it('restores only the options the recalled OpenAI model accepts', () => { + const result = recall(openaiModel, providerMetadata, 'all'); + + expect(result?.values).toMatchObject({ + openaiBackground: 'transparent', + openaiInputFidelity: 'high', + openaiQuality: 'low', + }); + // Another provider's keys and a named size the model does not list stay at the current values. + expect(result?.values).toMatchObject({ + externalImageSize: null, + geminiTemperature: null, + geminiThinkingLevel: null, + seedreamOptimizePrompt: false, + seedreamWatermark: false, + }); + expect(result?.fields).toContain('externalProvider'); + }); + + it('restores Gemini sampling and the named size the model lists, on a remix too', () => { + const result = recall(geminiModel, providerMetadata, 'remix'); + + expect(result?.values).toMatchObject({ + externalImageSize: '2K', + height: 2048, + width: 2048, + geminiTemperature: 0.4, + geminiThinkingLevel: 'high', + openaiQuality: 'auto', + }); + expect(result?.fields).toContain('externalProvider'); + }); + + it('skips a named size that is ambiguous without dimensions or an aspect ratio', () => { + const ambiguousModel = { + ...geminiModel, + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, + ], + }, + } as GenerateWidgetValues['model']; + const ambiguous = recall(ambiguousModel, { image_size: '2K' }, 'all'); + const resolved = recall(ambiguousModel, { image_size: '2K', aspect_ratio: '16:9' }, 'all'); + + expect(ambiguous?.values.externalImageSize).toBeNull(); + expect(ambiguous?.fields).not.toContain('externalProvider'); + expect(resolved?.values).toMatchObject({ externalImageSize: '2K', height: 1152, width: 2048 }); + }); + + it('restores the named size a Canvas output was generated at, though it records its frame', () => { + const canvasModel = { + ...geminiModel, + capabilities: { + modes: ['txt2img'], + resolution_presets: [ + { aspect_ratio: '1:1', height: 1024, image_size: '1K', label: '1:1 (1K)', width: 1024 }, + { aspect_ratio: '1:1', height: 2048, image_size: '2K', label: '1:1 (2K)', width: 2048 }, + { aspect_ratio: '16:9', height: 1152, image_size: '2K', label: '16:9 (2K)', width: 2048 }, + ], + }, + } as GenerateWidgetValues['model']; + + // A 2000×2000 frame asked for 2K at 1:1; a 1900×1000 one for 2K at 16:9, its nearest ratio. + expect(recall(canvasModel, { height: 2000, image_size: '2K', width: 2000 }, 'all')?.values).toMatchObject({ + externalImageSize: '2K', + height: 2048, + width: 2048, + }); + expect(recall(canvasModel, { height: 1000, image_size: '2K', width: 1900 }, 'all')?.values).toMatchObject({ + externalImageSize: '2K', + height: 1152, + width: 2048, + }); + + // On a Canvas route the size is the frame: it is restored as recorded, and the Canvas asks for 2K from it. + const onCanvas = buildImageRecallSettings({ + currentValues: createValues(), + image, + isCanvasFrame: true, + kind: 'all', + metadata: { height: 1000, image_size: '2K', model: { key: canvasModel.key }, width: 1900 }, + models: [], + supportedModels: [canvasModel], + vaeModels: [], + }); + expect(onCanvas?.values).toMatchObject({ height: 1000, width: 1900 }); + expect(onCanvas?.fields).toContain('size'); + }); + + it.each([0, 2])('recalls Gemini temperature boundary %s', (temperature) => { + expect(recall(geminiModel, { gemini_temperature: temperature }, 'all')?.values.geminiTemperature).toBe( + temperature + ); + }); + + it('rejects a negative recalled Gemini temperature while preserving the current choice', () => { + const result = buildImageRecallSettings({ + currentValues: createValues({ model: geminiModel, modelKey: geminiModel.key, geminiTemperature: 0.4 }), + image, + kind: 'all', + metadata: { external_provider: 'gemini', gemini_temperature: -0.1 }, + models: [], + supportedModels: [geminiModel], + vaeModels: [], + }); + + expect(result?.values.geminiTemperature).toBe(0.4); + }); + + it.each(['all', 'remix'] as const)('restores omitted nullable provider defaults during %s', (kind) => { + for (const model of [geminiModel, openaiModel]) { + const result = buildImageRecallSettings({ + currentValues: createValues({ + model, + modelKey: model.key, + geminiTemperature: 0.4, + geminiThinkingLevel: 'high', + openaiInputFidelity: 'high', + }), + image, + kind, + metadata: { external_provider: model.provider_id }, + models: [], + supportedModels: [model], + vaeModels: [], + }); + + if (model.provider_id === 'gemini') { + expect(result?.values).toMatchObject({ + geminiTemperature: null, + geminiThinkingLevel: null, + openaiInputFidelity: 'high', + }); + } else { + expect(result?.values).toMatchObject({ + geminiTemperature: 0.4, + geminiThinkingLevel: 'high', + openaiInputFidelity: null, + }); + } + expect(result?.fields).toContain('externalProvider'); + } + }); + + it('preserves optional provider choices when unrelated partial metadata omits them', () => { + const result = buildImageRecallSettings({ + currentValues: createValues({ + model: geminiModel, + modelKey: geminiModel.key, + geminiTemperature: 0.4, + geminiThinkingLevel: 'high', + }), + image, + kind: 'all', + metadata: { positive_prompt: 'a sunset', external_provider: 'openai' }, + models: [], + supportedModels: [geminiModel], + vaeModels: [], + }); + + expect(result?.values).toMatchObject({ geminiTemperature: 0.4, geminiThinkingLevel: 'high' }); + expect(result?.fields).not.toContain('externalProvider'); + }); + + it('restores Seedream toggles and drops values the node would reject', () => { + const result = recall( + seedreamModel, + { ...providerMetadata, seedream_optimize_prompt: 'yes', seedream_watermark: false }, + 'all' + ); + + expect(result?.values.seedreamWatermark).toBe(false); + expect(result?.values.seedreamOptimizePrompt).toBe(false); + expect(result?.fields).toContain('externalProvider'); + + const invalid = recall(geminiModel, { gemini_temperature: 9, gemini_thinking_level: 'max' }, 'all'); + + expect(invalid?.values.geminiTemperature).toBeNull(); + expect(invalid?.values.geminiThinkingLevel).toBeNull(); + expect(invalid?.fields).not.toContain('externalProvider'); + }); + + it('skips provider metadata when the image was made with a local model', () => { + const result = buildImageRecallSettings({ + currentValues: createValues(), + image, + kind: 'all', + metadata: { ...providerMetadata, model: { key: sdxlModel.key } }, + models: [], + supportedModels: [sdxlModel], + vaeModels: [], + }); + + expect(result?.values).toMatchObject({ + externalImageSize: null, + geminiTemperature: null, + geminiThinkingLevel: null, + openaiBackground: 'auto', + openaiInputFidelity: null, + openaiQuality: 'auto', + seedreamOptimizePrompt: false, + seedreamWatermark: false, + }); + expect(result?.fields).not.toContain('externalProvider'); + }); + + it('offers recall for an image whose only recallable delta is a provider option', () => { + const capabilities = getImageRecallCapabilities({ + currentValues: createValues({ model: seedreamModel, modelKey: seedreamModel.key }), + image, + metadata: { seedream_watermark: true }, + models: [], + supportedModels: [], + vaeModels: [], + }); + + expect(capabilities.all).toBe(true); + expect(capabilities.remix).toBe(true); + }); + }); }); describe('model-family tuning recall', () => { diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.ts b/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.ts index 8d42e3fd3dc..8f9b49dac3c 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.ts +++ b/invokeai/frontend/webv2/src/workbench/image-actions/imageRecall.ts @@ -16,6 +16,8 @@ import { deriveAspectRatioId, getCompatibleReferenceImages, getDimensionGrid, + getExternalProviderOptionKeys, + getExternalResolutionPresets, getGenerateModelSelectionResult, getGenerationUiPolicy, isComponentCompatibleWithModel, @@ -29,6 +31,12 @@ import { isMainModelConfig, isModelIdentifierConfig, isVaeCompatibleWithGenerateModel, + isGeminiThinkingLevel, + isOpenAIImageBackground, + isOpenAIImageQuality, + isOpenAIInputFidelity, + GEMINI_TEMPERATURE_MAX, + GEMINI_TEMPERATURE_MIN, isRefinerModel, isFluxDypePreset, isQwenImageEncoderQuantization, @@ -91,6 +99,7 @@ type RecalledField = | 'loras' | 'referenceImages' | 'krea2Rebalance' + | 'externalProvider' | 'refiner' | 'krea2SeedVariance' | 'modelTuning' @@ -572,6 +581,131 @@ const hasModelTuning = (metadata: unknown, model: GenerateModelConfig | undefine /** The canvas inpaint and outpaint graphs record the adapter key whenever the adapter ran. */ const ANIMA_INPAINT_ADAPTER_MODES: readonly string[] = ['anima_inpaint', 'anima_outpaint']; +/** + * Provider options and the named size, gated on what the effective model's provider accepts: the keys are recorded + * by the provider nodes (`openai_quality`, `gemini_temperature`, ...), so a value only applies to the same provider. + */ +const getMetadataExternalProvider = ( + metadata: unknown, + model: GenerateModelConfig | undefined, + isCanvasFrame: boolean +): Partial => { + // Local models advertise no provider options or presets, so every lookup below answers nothing for them. + const patch: Partial = {}; + const imageSize = getString(metadata, 'image_size'); + + if (imageSize !== null) { + const width = getNumber(metadata, 'width'); + const height = getNumber(metadata, 'height'); + const ratio = getString(metadata, 'aspect_ratio'); + const named = getExternalResolutionPresets(model).filter( + (preset) => preset.image_size === imageSize && (ratio === null || preset.aspect_ratio === ratio) + ); + const exact = named.filter( + (preset) => (width === null || preset.width === width) && (height === null || preset.height === height) + ); + // A Canvas output records its frame, not the provider size: the Canvas asked for the named size at the ratio + // nearest the frame, so Generate recall picks that one. A Canvas route keeps the recorded frame instead, and the + // Canvas derives the same named size from it. + const ratioDistance = (preset: { width: number; height: number }) => + width !== null && height !== null && width > 0 && height > 0 + ? Math.abs(Math.log(preset.width / preset.height / (width / height))) + : Number.NaN; + const nearest = + !isCanvasFrame && exact.length === 0 && named.length > 0 && !Number.isNaN(ratioDistance(named[0]!)) + ? named.reduce((best, preset) => (ratioDistance(preset) < ratioDistance(best) ? preset : best)) + : undefined; + const preset = exact.length === 1 ? exact[0] : nearest; + + // A name alone can describe many aspect ratios. Restore only a size the record pins down, with its dimensions. + if (preset) { + patch.externalImageSize = preset.image_size; + patch.width = preset.width; + patch.height = preset.height; + patch.aspectRatioId = deriveAspectRatioId(preset.width, preset.height); + patch.aspectRatioValue = preset.width / preset.height; + } + } + + // Provider outputs omit nullable fields when their default was used. The provider stamp distinguishes those + // omissions from partial or unrelated metadata, where the current choice should remain untouched. + const usesProviderDefaults = getString(metadata, 'external_provider') === model?.provider_id; + const isDefault = (key: string): boolean => + isRecord(metadata) && (metadata[key] === null || (usesProviderDefaults && !(key in metadata))); + + for (const key of getExternalProviderOptionKeys(model)) { + switch (key) { + case 'openaiQuality': { + const value = getString(metadata, 'openai_quality'); + + if (isOpenAIImageQuality(value)) { + patch.openaiQuality = value; + } + break; + } + case 'openaiBackground': { + const value = getString(metadata, 'openai_background'); + + if (isOpenAIImageBackground(value)) { + patch.openaiBackground = value; + } + break; + } + case 'openaiInputFidelity': { + const value = getString(metadata, 'openai_input_fidelity'); + + if (isOpenAIInputFidelity(value)) { + patch.openaiInputFidelity = value; + } else if (isDefault('openai_input_fidelity')) { + patch.openaiInputFidelity = null; + } + break; + } + case 'geminiTemperature': { + const value = getNumber(metadata, 'gemini_temperature'); + + if (value !== null && value >= GEMINI_TEMPERATURE_MIN && value <= GEMINI_TEMPERATURE_MAX) { + patch.geminiTemperature = value; + } else if (isDefault('gemini_temperature')) { + patch.geminiTemperature = null; + } + break; + } + case 'geminiThinkingLevel': { + const value = getString(metadata, 'gemini_thinking_level'); + + if (isGeminiThinkingLevel(value)) { + patch.geminiThinkingLevel = value; + } else if (isDefault('gemini_thinking_level')) { + patch.geminiThinkingLevel = null; + } + break; + } + case 'seedreamWatermark': { + const value = getBoolean(metadata, 'seedream_watermark'); + + if (value !== null) { + patch.seedreamWatermark = value; + } + break; + } + case 'seedreamOptimizePrompt': { + const value = getBoolean(metadata, 'seedream_optimize_prompt'); + + if (value !== null) { + patch.seedreamOptimizePrompt = value; + } + break; + } + } + } + + return patch; +}; + +const hasExternalProvider = (metadata: unknown, model: GenerateModelConfig | undefined): boolean => + Object.keys(getMetadataExternalProvider(metadata, model, false)).length > 0; + /** * The adapter switch and weight; only an Anima image can carry them. A recorded adapter means it ran, so the switch * turns on. `core_metadata` drops `null` on write, so an inpaint or outpaint record without the key ran without one: @@ -1060,6 +1194,7 @@ export const getImageRecallCapabilities = ({ recordedReferenceImages !== null && (recordedReferenceImages.length > 0 || currentValues.referenceImages.length > 0); const hasModel = supportedMetadataModel !== null; + const hasProviderOptions = hasExternalProvider(metadata, clipSkipModel); const hasRebalance = hasKrea2Rebalance(metadata, clipSkipModel) || hasKrea2SeedVariance(metadata, clipSkipModel); const hasTuning = hasModelTuning(metadata, clipSkipModel); const hasRefiner = @@ -1080,6 +1215,7 @@ export const getImageRecallCapabilities = ({ hasReferenceImages || hasConcepts || hasRebalance || + hasProviderOptions || hasRefiner || hasTuning || hasAdapterSettings; @@ -1095,6 +1231,7 @@ export const getImageRecallCapabilities = ({ hasReferenceImages || hasConcepts || hasRebalance || + hasProviderOptions || hasRefiner || hasTuning || hasAdapterSettings; @@ -1113,6 +1250,7 @@ export const getImageRecallCapabilities = ({ export const buildImageRecallSettings = ({ currentValues, image, + isCanvasFrame = false, kind, metadata, supportedModels, @@ -1121,6 +1259,8 @@ export const buildImageRecallSettings = ({ }: { currentValues: GenerateWidgetValues; image: GalleryImage; + /** The Generate size is the Canvas frame (a Canvas route), which recall sets as recorded. */ + isCanvasFrame?: boolean; kind: ImageRecallKind; metadata: unknown; supportedModels: GenerateModelConfig[]; @@ -1134,6 +1274,8 @@ export const buildImageRecallSettings = ({ const fields: RecalledField[] = []; const skipped: ImageRecallSkip[] = []; let values: GenerateWidgetValues = cloneGenerateWidgetValues(currentValues); + // A restored provider size is what the provider was asked for; the recorded size may be a Canvas frame instead. + let restoredProviderSize = false; if (kind === 'all' || kind === 'remix') { const model = getSupportedMetadataModel(metadata, supportedModels); @@ -1167,6 +1309,14 @@ export const buildImageRecallSettings = ({ fields.push('krea2Rebalance'); } + const providerPatch = getMetadataExternalProvider(metadata, values.model, isCanvasFrame); + + if (Object.keys(providerPatch).length > 0) { + values = { ...values, ...providerPatch }; + fields.push('externalProvider'); + restoredProviderSize = providerPatch.width !== undefined; + } + const seedVariancePatch = getMetadataKrea2SeedVariance(metadata, values.model); if (Object.keys(seedVariancePatch).length > 0) { @@ -1264,6 +1414,8 @@ export const buildImageRecallSettings = ({ values = withDimensions(values, size); fields.push('size'); } + } else if (restoredProviderSize) { + fields.push('size'); } else if (kind === 'all' || kind === 'remix') { const size = getMetadataSize(metadata, values.model); diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts b/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts index f19ed50f3b1..db73fb3c529 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts +++ b/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts @@ -269,11 +269,13 @@ export const useImageActions = ({ return board ? getGalleryBoardLabel(board, t) : t('widgets.gallery.uncategorized'); }; - const getLatestWidgetValues = (widgetId: 'gallery' | 'generate' | 'video') => { + const getLatestProject = () => { const snapshot = queries.getSnapshot(); - const project = projectId - ? snapshot.projects.find((candidate) => candidate.id === projectId) - : snapshot.activeProject; + + return projectId ? snapshot.projects.find((candidate) => candidate.id === projectId) : snapshot.activeProject; + }; + const getLatestWidgetValues = (widgetId: 'gallery' | 'generate' | 'video') => { + const project = getLatestProject(); return project ? getProjectWidgetValues(project, widgetId) : {}; }; @@ -1038,6 +1040,7 @@ export const useImageActions = ({ generateValues, getGenerateValues: getLatestGenerateValues, image, + isCanvasFrame: getLatestProject()?.invocation.sourceId === 'canvas', kind, models, projectId, diff --git a/invokeai/frontend/webv2/src/workbench/invocation.test.ts b/invokeai/frontend/webv2/src/workbench/invocation.test.ts index b3ecc0d7928..de38fa8a137 100644 --- a/invokeai/frontend/webv2/src/workbench/invocation.test.ts +++ b/invokeai/frontend/webv2/src/workbench/invocation.test.ts @@ -6,7 +6,12 @@ import type { VaeModelConfig, } from '@features/generation/contracts'; import type { ModelConfig } from '@features/models'; -import type { CanvasControlLayerContract, CanvasLayerContract } from '@workbench/canvas-engine/api'; +import type { + CanvasControlLayerContract, + CanvasLayerContract, + CanvasNodeContract, + CanvasRasterLayerContractV2, +} from '@workbench/canvas-engine/api'; import type { InvocationRoute } from '@workbench/invocationContracts'; import type { WidgetId } from '@workbench/widgetContracts'; @@ -271,6 +276,7 @@ describe('resolveInvocationRoute — canvas source', () => { mountedWidgetIds?: WidgetId[]; bbox?: { width: number; height: number }; layers?: CanvasLayerContract[]; + rasterInFrame?: boolean; } = {} ) => { const project = getActiveProject(overrides.generateValues ?? validGenerateValues); @@ -279,6 +285,7 @@ describe('resolveInvocationRoute — canvas source', () => { ...getInvocationRouteInput(project), canvasBbox: overrides.bbox ?? { height: 512, width: 512 }, canvasLayers: overrides.layers ?? [], + canvasRasterInFrame: overrides.rasterInFrame ?? false, mountedWidgetIds: overrides.mountedWidgetIds ?? (['canvas', 'generate'] as WidgetId[]), }; }; @@ -307,6 +314,173 @@ describe('resolveInvocationRoute — canvas source', () => { expect(route.validationReasons).toEqual([]); }); + it('allows Canvas preparation for an img2img-only provider while Generate still requires txt2img', () => { + const model: GenerateModelConfig = { + ...unknownExternalModel, + provider_id: 'openai', + capabilities: { modes: ['img2img'] }, + }; + const input = canvasInputFor({ generateValues: createGenerateValues(model) }); + expect(resolveInvocationRouteInput(input, 'global', canvasRoute).sourceValid).toBe(true); + expect( + resolveInvocationRouteInput(input, 'global', { ...canvasRoute, sourceId: 'generate' }).validationReasons + ).toContain('Unknown External Provider does not support text-to-image generation.'); + }); + + it('names unsupported provider control layers in readiness even without a loaded adapter catalog', () => { + const model: GenerateModelConfig = { ...unknownExternalModel, provider_id: 'openai' }; + const layer = modellessControlLayer(); + const input = canvasInputFor({ generateValues: createGenerateValues(model), layers: [layer] }); + expect(resolveInvocationRouteInput(input, 'global', canvasRoute).validationReasons).toEqual([ + 'Unknown External Provider does not support control layer "Control Layer 1". Disable it to generate.', + ]); + expect( + resolveInvocationRouteInput({ ...input, canvasLayers: [{ ...layer, isEnabled: false }] }, 'global', canvasRoute) + .sourceValid + ).toBe(true); + }); + + it('validates provider dimensions against the Canvas bbox rather than the Generate panel size', () => { + const model: GenerateModelConfig = { + ...unknownExternalModel, + provider_id: 'openai', + capabilities: { modes: ['txt2img'], max_image_size: { width: 1024, height: 1024 } }, + }; + const generateValues = createGenerateValues(model, { width: 2048, height: 2048 }); + expect(resolveInvocationRouteInput(canvasInputFor({ generateValues }), 'global', canvasRoute).sourceValid).toBe( + true + ); + expect( + resolveInvocationRouteInput( + canvasInputFor({ generateValues, bbox: { width: 1025, height: 512 } }), + 'global', + canvasRoute + ).validationReasons + ).toContain('Unknown External Provider supports a maximum size of 1024×1024.'); + }); + + it('validates a fixed-size provider against the size it is asked for, not the frame', () => { + const model: GenerateModelConfig = { + ...unknownExternalModel, + provider_id: 'openai', + capabilities: { + aspect_ratio_sizes: { '1:1': { height: 1024, width: 1024 } }, + max_image_size: { height: 1024, width: 1024 }, + modes: ['txt2img'], + }, + }; + const input = canvasInputFor({ bbox: { height: 3000, width: 3000 }, generateValues: createGenerateValues(model) }); + + expect(resolveInvocationRouteInput(input, 'global', canvasRoute).validationReasons).toEqual([]); + }); + + it('blocks a text-only provider while raster content overlaps the frame, naming the content', () => { + const textOnly: GenerateModelConfig = { ...unknownExternalModel, provider_id: 'openai' }; + const editing: GenerateModelConfig = { ...textOnly, capabilities: { modes: ['txt2img', 'img2img'] } }; + + expect( + resolveInvocationRouteInput( + canvasInputFor({ generateValues: createGenerateValues(textOnly), rasterInFrame: true }), + 'global', + canvasRoute + ).validationReasons + ).toEqual([ + 'Unknown External Provider cannot use image content in the generation frame. Hide or move raster content out of the frame to generate.', + ]); + expect( + resolveInvocationRouteInput( + canvasInputFor({ generateValues: createGenerateValues(textOnly) }), + 'global', + canvasRoute + ).sourceValid + ).toBe(true); + expect( + resolveInvocationRouteInput( + canvasInputFor({ generateValues: createGenerateValues(editing), rasterInFrame: true }), + 'global', + canvasRoute + ).sourceValid + ).toBe(true); + }); + + it('measures each visible raster layer against the frame, for a text-only provider on a Canvas route', () => { + const textOnly: GenerateModelConfig = { ...unknownExternalModel, provider_id: 'openai' }; + let project = getActiveProject(createGenerateValues(textOnly)); + const raster = ( + x: number, + y: number, + overrides: Partial = {} + ): CanvasRasterLayerContractV2 => ({ + blendMode: 'normal', + id: `raster-${x}-${y}`, + isEnabled: true, + isLocked: false, + name: `Raster ${x},${y}`, + opacity: 1, + source: { image: { height: 64, imageName: 'raster.png', width: 64 }, type: 'image' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x, y }, + type: 'raster', + ...overrides, + }); + const withRaster = (layers: CanvasNodeContract[], sourceId: InvocationRoute['sourceId'] = 'canvas') => + getInvocationRouteInput({ + ...project, + canvas: { + ...project.canvas, + document: { + ...project.canvas.document, + bbox: { height: 100, width: 100, x: 200, y: 0 }, + stacks: { ...project.canvas.document.stacks, raster: layers }, + }, + }, + invocation: { ...project.invocation, sourceId }, + }).canvasRasterInFrame; + + expect(withRaster([])).toBe(false); + expect(withRaster([raster(250, 50)])).toBe(true); + // Layers on either side of an empty frame: their union spans it, but neither reaches it. + expect(withRaster([raster(0, 0), raster(400, 200)])).toBe(false); + // Touching the frame's edge is not overlap; hidden or fully transparent content contributes nothing. + expect(withRaster([raster(300, 0)])).toBe(false); + expect(withRaster([raster(250, 50, { isEnabled: false })])).toBe(false); + expect(withRaster([raster(250, 50, { opacity: 0 })])).toBe(false); + expect( + withRaster([ + { + children: [raster(250, 50)], + id: 'group', + isEnabled: true, + isLocked: false, + name: 'Group', + opacity: 0, + type: 'group', + }, + ]) + ).toBe(false); + // Readiness only estimates a text layer's extent, so it leaves text to the invoke-time pixel check. + expect( + withRaster([ + raster(250, 50, { + source: { + align: 'left', + color: '#000', + content: 'Hello', + fontFamily: 'Inter', + fontSize: 32, + fontWeight: 400, + lineHeight: 1.2, + type: 'text', + }, + }), + ]) + ).toBe(false); + + // Other routes and other models never measure. + expect(withRaster([raster(250, 50)], 'generate')).toBe(false); + project = getActiveProject(validGenerateValues); + expect(withRaster([raster(250, 50)])).toBe(false); + }); + it('invalidates a zero-area generation frame', () => { const route = resolveInvocationRouteInput( canvasInputFor({ bbox: { height: 0, width: 512 } }), diff --git a/invokeai/frontend/webv2/src/workbench/invocation.ts b/invokeai/frontend/webv2/src/workbench/invocation.ts index cf562fb4756..c0f8a58da5a 100644 --- a/invokeai/frontend/webv2/src/workbench/invocation.ts +++ b/invokeai/frontend/webv2/src/workbench/invocation.ts @@ -13,10 +13,14 @@ import type { Project } from '@workbench/projectContracts'; import type { WidgetId } from '@workbench/widgetContracts'; import { + describeExternalCanvasRefusal, getGenerationModelAvailabilityReasons, getGenerationValidationReasons, + isExternalTextOnlyCanvasModel, + isGenerateModelConfig, isSupportedGenerateModel, normalizeGenerateWidgetValues, + resolveExternalCanvasRequestSize, sanitizeBatchCount, } from '@features/generation/settings'; import { getUpscaleValidationReasons, normalizeUpscaleWidgetValues } from '@features/upscale'; @@ -25,7 +29,14 @@ import { getProjectGraphReadiness } from '@features/workflow/graph'; import { getInvocationTemplatesSnapshot } from '@features/workflow/react'; import { localizeForLoopValidationReason } from '@features/workflow/utility'; import { areArraysEqual, createStableSelector } from '@platform/state/selectors'; -import { compileContributingLayers } from '@workbench/canvas-engine/api'; +import { + compileContributingLayers, + compileDocumentLeaves, + getSourceBounds, + isBaseRasterContentLayer, + lookupDocumentNode, +} from '@workbench/canvas-engine/api'; +import { isCompositableControlLayer, isCompositableRegionalGuidanceLayer } from '@workbench/canvasLayerContent'; import { describeControlLayerIssue, getBlockingControlLayerIssues } from './controlLayerChecks'; import { getActiveProjectGraph } from './projectWorkflows'; @@ -99,6 +110,11 @@ export interface InvocationRouteInput { canvasBbox: { width: number; height: number }; /** The canvas layer stack (identity-stable) — control layers gate a canvas invoke. */ canvasLayers: readonly CanvasLayerContract[]; + /** + * Whether a visible raster layer's bounds overlap the frame, so a text-only hosted model is blocked before invoke + * without reading pixels. Measured only for a Canvas route with such a model; false otherwise. + */ + canvasRasterInFrame: boolean; } const getMountedWidgetIds = (project: Project): WidgetId[] => { @@ -126,21 +142,69 @@ const getMountedWidgetIds = (project: Project): WidgetId[] => { return Array.from(mountedWidgetIds).sort(); }; -export const getInvocationRouteInput = (project: Project): InvocationRouteInput => ({ - canvasBbox: { - height: project.canvas.document.bbox.height, - width: project.canvas.document.bbox.width, - }, - canvasLayers: compileContributingLayers(project.canvas.document), - generateValues: getProjectWidgetValues(project, 'generate'), - upscaleValues: getProjectWidgetValues(project, 'upscale'), - videoValues: getProjectWidgetValues(project, 'video'), - workflowValues: getProjectWidgetValues(project, 'workflow'), - invocation: project.invocation, - mountedWidgetIds: getMountedWidgetIds(project), - projectGraph: getActiveProjectGraph(project), - projectId: project.id, -}); +/** + * Whether a raster layer the generation composite draws overlaps the frame's interior with its own bounds. Invoke + * decides from the composite's pixels, so this errs toward not blocking where geometry is uncertain: text is skipped + * (readiness only has an estimate of its extent) and so is anything at zero opacity, its own or a group's. The one + * remaining approximation is transparent pixels inside a layer's bounds, which this treats as content. + */ +const hasRasterContentInFrame = (document: Project['canvas']['document']): boolean => { + const { bbox } = document; + + if (bbox.width <= 0 || bbox.height <= 0) { + return false; + } + + return compileDocumentLeaves(document).some(({ contributionEnabled, layer, parentIds }) => { + if ( + !contributionEnabled || + !isBaseRasterContentLayer(layer) || + layer.source.type === 'text' || + layer.opacity <= 0 || + parentIds.some((id) => { + const group = lookupDocumentNode(document, id); + return group?.type === 'group' && (group.opacity ?? 1) <= 0; + }) + ) { + return false; + } + + const bounds = getSourceBounds(layer, document); + + return ( + bounds.x < bbox.x + bbox.width && + bbox.x < bounds.x + bounds.width && + bounds.y < bbox.y + bbox.height && + bbox.y < bounds.y + bounds.height + ); + }); +}; + +export const getInvocationRouteInput = (project: Project): InvocationRouteInput => { + const generateValues = getProjectWidgetValues(project, 'generate'); + // This runs on every project change, so only a Canvas route with the model it gates measures layers. + const measuresRaster = + project.invocation.sourceId === 'canvas' && + isGenerateModelConfig(generateValues.model) && + isExternalTextOnlyCanvasModel(generateValues.model); + + return { + canvasBbox: { + height: project.canvas.document.bbox.height, + width: project.canvas.document.bbox.width, + }, + canvasLayers: compileContributingLayers(project.canvas.document), + canvasRasterInFrame: measuresRaster && hasRasterContentInFrame(project.canvas.document), + generateValues, + upscaleValues: getProjectWidgetValues(project, 'upscale'), + videoValues: getProjectWidgetValues(project, 'video'), + workflowValues: getProjectWidgetValues(project, 'workflow'), + invocation: project.invocation, + mountedWidgetIds: getMountedWidgetIds(project), + projectGraph: getActiveProjectGraph(project), + projectId: project.id, + }; +}; export const areInvocationRouteInputsEqual = (left: InvocationRouteInput, right: InvocationRouteInput): boolean => left.projectId === right.projectId && @@ -153,6 +217,7 @@ export const areInvocationRouteInputsEqual = (left: InvocationRouteInput, right: left.canvasBbox.width === right.canvasBbox.width && left.canvasBbox.height === right.canvasBbox.height && left.canvasLayers === right.canvasLayers && + left.canvasRasterInFrame === right.canvasRasterInFrame && areArraysEqual(left.mountedWidgetIds, right.mountedWidgetIds); export const createInvocationRouteInputSelector = () => @@ -163,7 +228,8 @@ const isWidgetMounted = (input: InvocationRouteInput, widgetId: WidgetId): boole const getGenerateSnapshotValidationReasons = ( generateValues: Record, - models?: readonly ModelConfig[] + models?: readonly ModelConfig[], + canvasBbox?: InvocationRouteInput['canvasBbox'] ): string[] => { const values = normalizeGenerateWidgetValues(generateValues); @@ -171,8 +237,23 @@ const getGenerateSnapshotValidationReasons = ( return ['Generate needs a supported model before it can be invoked.']; } + // Canvas discovers the mode from pixel coverage during preparation and asks a provider for its size nearest the + // bbox, so readiness validates that size and permits an advertised whole-image mode; the compiler validates the + // captured one. Generate itself remains strictly text-to-image. + const providerModes = + canvasBbox !== undefined && values.model.type === 'external_image_generator' + ? (values.model.capabilities?.modes ?? ['txt2img']) + : null; + const mode = providerModes !== null && !providerModes.includes('txt2img') ? 'img2img' : 'txt2img'; + return [ - ...getGenerationValidationReasons(values.model, values), + ...getGenerationValidationReasons( + values.model, + providerModes !== null && canvasBbox + ? { ...values, ...resolveExternalCanvasRequestSize(values.model, canvasBbox) } + : values, + mode + ), ...(models ? getGenerationModelAvailabilityReasons(values.model, values, models) : []), ]; }; @@ -243,15 +324,32 @@ export const resolveInvocationRouteInput = ( if (sourceId === 'canvas') { // Canvas shares Generate readiness and additionally requires a nonzero generation frame. - validationReasons.push(...getGenerateSnapshotValidationReasons(input.generateValues, models)); + validationReasons.push(...getGenerateSnapshotValidationReasons(input.generateValues, models, input.canvasBbox)); if (input.canvasBbox.width <= 0 || input.canvasBbox.height <= 0) { validationReasons.push('Canvas generation frame must have a positive area.'); } - // Wait for models before validating control layers to avoid false missing-model errors. - if (models) { - const values = normalizeGenerateWidgetValues(input.generateValues); + const values = normalizeGenerateWidgetValues(input.generateValues); + if (values?.model.type === 'external_image_generator') { + const { model } = values; + + if (input.canvasRasterInFrame && isExternalTextOnlyCanvasModel(model)) { + validationReasons.push(describeExternalCanvasRefusal(model, { kind: 'raster_content' })); + } + for (const layer of input.canvasLayers) { + if (isCompositableControlLayer(layer)) { + validationReasons.push( + describeExternalCanvasRefusal(model, { kind: 'control_layer', layerName: layer.name }) + ); + } else if (isCompositableRegionalGuidanceLayer(layer)) { + validationReasons.push( + describeExternalCanvasRefusal(model, { kind: 'regional_guidance', layerName: layer.name }) + ); + } + } + } else if (models) { + // Wait for models before validating local adapters to avoid false missing-model errors. if (values && isSupportedGenerateModel(values.model)) { validationReasons.push( ...getBlockingControlLayerIssues({ diff --git a/invokeai/frontend/webv2/src/workbench/palette/WorkbenchCommandPaletteDialog.tsx b/invokeai/frontend/webv2/src/workbench/palette/WorkbenchCommandPaletteDialog.tsx index f59956c49d2..2dc79cbe55a 100644 --- a/invokeai/frontend/webv2/src/workbench/palette/WorkbenchCommandPaletteDialog.tsx +++ b/invokeai/frontend/webv2/src/workbench/palette/WorkbenchCommandPaletteDialog.tsx @@ -6,7 +6,7 @@ import type { getWidgetsForRegion as GetWidgetsForRegion } from '@workbench/widg import type { TFunction } from 'i18next'; import { imageIndexAvailabilityOptions } from '@features/gallery/queries'; -import { flushGenerateDrafts, notifyGenerateModelSelectionCleared } from '@features/generation/react'; +import { flushGenerateDrafts } from '@features/generation/react'; import { getModelsSnapshot } from '@features/models'; import { getQueueQueryScope, getQueueReadModelOptions } from '@features/queue/queries'; import { queryClient } from '@platform/query/client'; @@ -161,6 +161,7 @@ const WorkbenchCommandPaletteDialog = ({ return { currentValues: getProjectWidgetValues(project, 'generate'), + isCanvasFrame: project.invocation.sourceId === 'canvas', projectId: project.id, }; }, [workbenchQueries]); @@ -201,23 +202,19 @@ const WorkbenchCommandPaletteDialog = ({ t, }), createModelsProvider({ - applyModel: (model, catalogModels) => { - const { currentValues, projectId } = readCurrentGenerateContext(); - const result = selectProjectGenerateModel({ + locale: i18n.resolvedLanguage, + notifications: notify, + selectModel: (model, catalogModels) => { + const { currentValues, isCanvasFrame, projectId } = readCurrentGenerateContext(); + + return selectProjectGenerateModel({ currentValues, generation, + isCanvasFrame, model, models: catalogModels, projectId, }); - - notifyGenerateModelSelectionCleared({ - clearedLabels: result.clearedLabels, - locale: i18n.resolvedLanguage, - modelName: model.name, - notifications: notify, - t, - }); }, openGenerateWidget: () => openWidget('generate'), openModelManager: () => void executeCommand('app.selectModelsTab'), diff --git a/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.test.ts b/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.test.ts index 848ddfebf37..e67cf0c5182 100644 --- a/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.test.ts +++ b/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.test.ts @@ -1,20 +1,29 @@ import type { GalleryBoard, GalleryImage } from '@features/gallery/contracts'; import type { GenerationModelCatalogItem } from '@features/generation/contracts'; +import type { GenerateModelSelectionResult } from '@features/generation/settings'; import type { ModelConfig } from '@features/models'; import type { TFunction } from 'i18next'; import { ApiError } from '@platform/transport/http'; import { beforeEach, describe, expect, it, vi } from 'vitest'; -const { ensureModelsLoaded, fetchQuery, getModelsSnapshot, listPaletteImages, listPaletteSemanticImages } = vi.hoisted( - () => ({ - ensureModelsLoaded: vi.fn(() => Promise.resolve()), - fetchQuery: vi.fn(), - getModelsSnapshot: vi.fn(), - listPaletteImages: vi.fn(), - listPaletteSemanticImages: vi.fn(), - }) -); +const { + ensureModelsLoaded, + fetchQuery, + getModelsSnapshot, + listPaletteImages, + listPaletteSemanticImages, + notifyGenerateDimensionsAdjusted, + notifyGenerateModelSelectionCleared, +} = vi.hoisted(() => ({ + ensureModelsLoaded: vi.fn(() => Promise.resolve()), + fetchQuery: vi.fn(), + getModelsSnapshot: vi.fn(), + listPaletteImages: vi.fn(), + listPaletteSemanticImages: vi.fn(), + notifyGenerateDimensionsAdjusted: vi.fn(), + notifyGenerateModelSelectionCleared: vi.fn(), +})); vi.mock('@features/models', () => ({ ensureModelsLoaded, @@ -22,7 +31,11 @@ vi.mock('@features/models', () => ({ getModelsSnapshot, })); -vi.mock('@features/generation/react', () => ({ focusPositivePrompt: vi.fn() })); +vi.mock('@features/generation/react', () => ({ + focusPositivePrompt: vi.fn(), + notifyGenerateDimensionsAdjusted, + notifyGenerateModelSelectionCleared, +})); vi.mock('@features/gallery/paletteSearch', () => ({ ALL_READABLE_BOARDS_ID: 'all', listPaletteImages: (...args: unknown[]) => listPaletteImages(...args), @@ -189,23 +202,51 @@ describe('createModelsProvider', () => { model('vae', 'sdxl', 'vae'), ]; const order: string[] = []; - const applyModel = vi.fn((_selected: GenerationModelCatalogItem, _models: readonly ModelConfig[]) => { - order.push('apply'); + const result = { + clearedLabels: ['LoRAs'], + dimensionsAdjusted: true, + settings: { height: 1600, width: 2848 }, + } as unknown as GenerateModelSelectionResult; + const selectModel = vi.fn((_selected: GenerationModelCatalogItem, _models: readonly ModelConfig[]) => { + order.push('select'); + return result; }); const openGenerateWidget = vi.fn(() => { order.push('open'); }); const openModelManager = vi.fn(); + const notifications = { info: vi.fn() }; getModelsSnapshot.mockReturnValue({ models, status: 'loaded' }); - const provider = createModelsProvider({ applyModel, openGenerateWidget, openModelManager, t }); + const provider = createModelsProvider({ + locale: 'en', + notifications, + openGenerateWidget, + openModelManager, + selectModel, + t, + }); const entries = await provider.search({ text: '' }, searchContext()); expect(entries.map((entry) => entry.title)).toEqual(['supported-sdxl', 'external-provider']); entries[0]?.run(); - expect(applyModel).toHaveBeenCalledWith(supported, models); - expect(order).toEqual(['apply', 'open']); + expect(selectModel).toHaveBeenCalledWith(supported, models); + expect(order).toEqual(['select', 'open']); + // The switch reports what it cleared and the size it moved to, as the Generate widget's own picker does. + expect(notifyGenerateModelSelectionCleared).toHaveBeenCalledWith({ + clearedLabels: ['LoRAs'], + locale: 'en', + modelName: 'supported-sdxl', + notifications, + t, + }); + expect(notifyGenerateDimensionsAdjusted).toHaveBeenCalledWith({ + modelName: 'supported-sdxl', + notifications, + result, + t, + }); entries[0]?.secondary?.run(); expect(openModelManager).toHaveBeenCalledOnce(); @@ -215,9 +256,11 @@ describe('createModelsProvider', () => { const models = [model('cinematic-xl', 'sdxl'), model('fast-flux', 'flux')]; getModelsSnapshot.mockReturnValue({ models, status: 'loaded' }); const provider = createModelsProvider({ - applyModel: vi.fn(), + locale: 'en', + notifications: { info: vi.fn() }, openGenerateWidget: vi.fn(), openModelManager: vi.fn(), + selectModel: vi.fn(), t, }); diff --git a/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.ts b/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.ts index dddfbeb8c72..2269b113fd4 100644 --- a/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.ts +++ b/invokeai/frontend/webv2/src/workbench/palette/paletteProviders.ts @@ -1,4 +1,5 @@ import type { GenerateModelConfig } from '@features/generation/contracts'; +import type { GenerateModelSelectionResult } from '@features/generation/settings'; import type { ModelConfig } from '@features/models'; import type { QueueReadModel } from '@features/queue/contracts'; import type { PromptHistoryItem } from '@workbench/projectContracts'; @@ -7,7 +8,11 @@ import type { TFunction } from 'i18next'; import { getGalleryBoardLabel, type GalleryBoard, type GalleryImage } from '@features/gallery/contracts'; import { ALL_READABLE_BOARDS_ID, listPaletteImages, listPaletteSemanticImages } from '@features/gallery/paletteSearch'; import { galleryBoardsOptions, imageIndexAvailabilityOptions } from '@features/gallery/queries'; -import { focusPositivePrompt } from '@features/generation/react'; +import { + focusPositivePrompt, + notifyGenerateDimensionsAdjusted, + notifyGenerateModelSelectionCleared, +} from '@features/generation/react'; import { isGenerateModelSelectable } from '@features/generation/settings'; import { ensureModelsLoaded, getModelBaseLabel, getModelsSnapshot } from '@features/models'; import { extractGenerationMeta, getResultImageName } from '@features/queue/contracts'; @@ -174,15 +179,20 @@ export const createPromptHistoryProvider = ({ }, }); +/** Picking a model switches Generate to it, reports what the switch cleared or resized, then opens the widget. */ export const createModelsProvider = ({ - applyModel, + locale, + notifications, openGenerateWidget, openModelManager, + selectModel, t, }: { - applyModel: (model: GenerateModelConfig, models: readonly ModelConfig[]) => void; + locale: string | undefined; + notifications: { info(title: string, message?: string): void }; openGenerateWidget: () => void; openModelManager: () => void; + selectModel: (model: GenerateModelConfig, models: readonly ModelConfig[]) => GenerateModelSelectionResult; t: TFunction; }): PaletteSearchProvider => ({ contextKey: 'global', @@ -206,7 +216,16 @@ export const createModelsProvider = ({ isPersistentRecent: false, keywords: model.base, run: () => { - applyModel(model, models); + const result = selectModel(model, models); + + notifyGenerateModelSelectionCleared({ + clearedLabels: result.clearedLabels, + locale, + modelName: model.name, + notifications, + t, + }); + notifyGenerateDimensionsAdjusted({ modelName: model.name, notifications, result, t }); openGenerateWidget(); }, secondary: { label: t('commandPalette.actions.openModelManager'), run: openModelManager }, diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/GenerateCanvasRenderSize.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/GenerateCanvasRenderSize.tsx index 472f761fe54..38b92e174a7 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/GenerateCanvasRenderSize.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/GenerateCanvasRenderSize.tsx @@ -12,7 +12,7 @@ import type { Project } from '@workbench/projectContracts'; import { Box, chakra, createListCollection, Grid, HStack, NumberInput, Text } from '@chakra-ui/react'; import { resolveCanvasProcessingSize } from '@features/generation/canvasProcessingSize'; import { getArchitectureCapabilitiesSnapshot, subscribeArchitectureCapabilities } from '@features/generation/runtime'; -import { clampDimension, getGenerationDimensions } from '@features/generation/settings'; +import { clampDimension, getGenerationDimensions, isGenerateModelConfig } from '@features/generation/settings'; import { useExternalStoreSelector } from '@platform/state/selectors'; import { FieldLabel, Select, Tooltip } from '@platform/ui'; import { @@ -40,30 +40,19 @@ const scalingEqual = (a: CanvasScalingSettings, b: CanvasScalingSettings): boole const readSize = (value: unknown): number | null => typeof value === 'number' && Number.isFinite(value) ? value : null; -/** - * Use model base/type/variant and PiD mode matching compileCanvasGraph so displayed size follows identical - * grid/area policy. - */ +/** Use the stored model and PiD mode as compileCanvasGraph does, so the displayed size follows the same policy. */ const selectProcessingContext = (project: Project) => { const generate = getProjectWidgetValues(project, 'generate') as { height?: unknown; - model?: { base?: string; type?: string; variant?: unknown } | null; + model?: unknown; pidMode?: PidMode; width?: unknown; }; - const model = generate.model; return { bbox: project.canvas.document.bbox, committedFrame: { height: readSize(generate.height), width: readSize(generate.width) }, lockedRatio: readLockedAspectRatio(generate), - model: - model && typeof model.base === 'string' - ? { - base: model.base, - type: model.type ?? 'main', - variant: typeof model.variant === 'string' ? model.variant : null, - } - : null, + model: isGenerateModelConfig(generate.model) ? generate.model : null, pidMode: generate.pidMode ?? 'off', }; }; @@ -76,9 +65,7 @@ const processingContextEqual = (a: ProcessingContext, b: ProcessingContext): boo a.committedFrame.width === b.committedFrame.width && a.committedFrame.height === b.committedFrame.height && a.lockedRatio === b.lockedRatio && - a.model?.base === b.model?.base && - a.model?.type === b.model?.type && - a.model?.variant === b.model?.variant && + a.model === b.model && a.pidMode === b.pidMode; export const GenerateCanvasRenderSize = ({ children, frame }: CanvasRenderSizeProps) => { @@ -99,7 +86,7 @@ export const GenerateCanvasRenderSize = ({ children, frame }: CanvasRenderSizePr subscribeArchitectureCapabilities, getArchitectureCapabilitiesSnapshot, useCallback(() => { - const model = context.model as Parameters[0] | null; + const { model } = context; return { dimensions: getGenerationDimensions(model ?? undefined, context.pidMode), processingSize: model diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/bboxGrid.test.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/bboxGrid.test.ts index 1dddb0ec5d0..c549ecc0b27 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/bboxGrid.test.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/bboxGrid.test.ts @@ -65,8 +65,10 @@ describe('resolveModelGrid before the capability table arrives', () => { it('answers for an external generator too, which will never get a row to wait for', () => { // A writer holds off on `null`, so answering it here would stop the bbox <-> dims sync for - // external providers for the whole session. - expect(resolveModelGrid('external')).toBe(DEFAULT_MODEL_GRID); + // external providers for the whole session. Providers bucket to their own size tables, so the + // bbox is pixel-exact rather than snapped to a local model grid. + expect(resolveModelGrid('external')).toBe(1); + expect(gridSizeForModelBase('external')).toBe(1); }); it('keeps the default for readers that must show something', () => { diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts index a343e591c06..20b64d84c98 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts @@ -792,7 +792,8 @@ describe('runCanvasInvocation', () => { ); it('records a notice and dispatches no snapshot when the graph fails validation', async () => { - // External image generators are rejected by the graph compiler. + // The raster snapshot makes this an image-to-image run, and the record lists text-to-image only, so the notice + // names the frame's content as the cause. const harness = makeHarness({ model: externalModel }); await runCanvasInvocation(harness.deps); @@ -800,12 +801,136 @@ describe('runCanvasInvocation', () => { expect(harness.submittedGraphs()).toHaveLength(0); const notices = harness.notices(); expect(notices).toHaveLength(1); - expect(notices[0]?.message).toContain('does not support canvas generation'); + expect(notices[0]?.message).toBe( + 'OpenAI Image cannot use image content in the generation frame. Hide or move raster content out of the frame to generate.' + ); expect(harness.dedupe.byKey).toHaveLength(0); expect(harness.dedupe.byHash).toHaveLength(0); expect(harness.releaseRasterSnapshot).toHaveBeenCalledTimes(1); }); + it('generates from text for a text-only provider when the frame holds no visible pixels', async () => { + // The raster layer's bounds overlap the frame, but every composited pixel there is transparent. + const harness = makeHarness({ alpha: 0, model: externalModel }); + + await runCanvasInvocation(harness.deps); + + expect(harness.notices()).toEqual([]); + const submitted = harness.submittedGraphs(); + expect(submitted).toHaveLength(1); + expect(submitted[0]?.graph.backendGraph?.nodes.canvas_processing_output).toMatchObject({ mode: 'txt2img' }); + expect(submitted[0]?.graph.backendGraph?.nodes.canvas_processing_output).not.toHaveProperty('init_image'); + }); + + it('generates from text for an editing provider when the frame holds no visible pixels', async () => { + // Without this, the partial bounds overlap would read as outpaint, which hosted providers refuse. + const model: GenerateModelConfig = { ...externalModel, capabilities: { modes: ['txt2img', 'img2img'] } }; + const harness = makeHarness({ alpha: 0, model }); + + await runCanvasInvocation(harness.deps); + + expect(harness.notices()).toEqual([]); + expect(harness.submittedGraphs()[0]?.graph.backendGraph?.nodes.canvas_processing_output).toMatchObject({ + mode: 'txt2img', + }); + }); + + it('refuses a text-only provider when any pixel in the frame is visible, even partly', async () => { + const harness = makeHarness({ alpha: 1, model: externalModel }); + + await runCanvasInvocation(harness.deps); + + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual([ + 'OpenAI Image cannot use image content in the generation frame. Hide or move raster content out of the frame to generate.', + ]); + }); + + it.each(['control', 'regional'] as const)( + 'rejects contributing external %s layers by name before enqueue', + async (kind) => { + const layer = kind === 'control' ? controlLayer('Canny guide') : regionalLayer('Hat region'); + const harness = makeHarness({ model: externalModel, document: docWithLayers([layer]) }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual([ + kind === 'control' + ? 'OpenAI Image does not support control layer "Canny guide". Disable it to generate.' + : 'OpenAI Image does not support regional guidance layer "Hat region". Disable it to generate.', + ]); + expect(harness.dedupe.byKey.size).toBe(0); + expect(harness.releaseRasterSnapshot).toHaveBeenCalledTimes(1); + } + ); + + it.each(['control', 'regional'] as const)('ignores disabled external %s layers', async (kind) => { + const layer = kind === 'control' ? controlLayer('Canny guide') : regionalLayer('Hat region'); + const harness = makeHarness({ model: externalModel, document: docWithLayers([{ ...layer, isEnabled: false }]) }); + await runCanvasInvocation(harness.deps); + expect(harness.notices()).toEqual([]); + expect(harness.submittedGraphs()).toHaveLength(1); + }); + + it('submits an img2img-only provider through the real preparation path', async () => { + const model: GenerateModelConfig = { ...externalModel, capabilities: { modes: ['img2img'] } }; + const harness = makeHarness({ model }); + await runCanvasInvocation(harness.deps); + expect(harness.notices()).toEqual([]); + expect(harness.submittedGraphs()).toHaveLength(1); + }); + + it('blocks advertised provider inpainting before enqueue', async () => { + const model: GenerateModelConfig = { ...externalModel, capabilities: { modes: ['inpaint', 'img2img'] } }; + const harness = makeHarness({ model, document: docWithLayers([rasterLayer('base'), inpaintMaskLayer('mask')]) }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual([ + 'OpenAI Image does not support inpainting on the canvas yet. Remove or disable the inpaint mask.', + ]); + }); + + it('blocks oversized provider requests before enqueue', async () => { + const model: GenerateModelConfig = { + ...externalModel, + capabilities: { modes: ['img2img'], max_image_size: { width: 32, height: 64 } }, + }; + const harness = makeHarness({ model }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual([ + 'OpenAI Image supports a maximum size of 32×64.', + ]); + }); + + it('counts the captured init image against the Seedream budget before enqueue', async () => { + const model: GenerateModelConfig = { + ...externalModel, + name: 'Seedream', + provider_id: 'seedream', + provider_model_id: 'seedream-4-5-251128', + capabilities: { modes: ['txt2img', 'img2img'], max_reference_images: 14, supports_reference_images: true }, + }; + const harness = makeHarness({ + model, + generateValues: { + ...generateValuesFor(model), + referenceImages: Array.from({ length: 14 }, (_, i) => ({ + id: `reference-${i}`, + isEnabled: true, + config: { + type: 'external_reference_image', + image: { imageName: `reference-${i}.png`, width: 64, height: 64 }, + }, + })), + }, + }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual([ + 'Seedream supports at most 15 images total (reference + init + output), got 16.', + ]); + }); + it('discards provisional dedupe entries when synchronous queue dispatch fails', async () => { const dispatch = vi.fn((action: WorkbenchAction) => { if (action.type === 'submitCanvasInvocationSnapshot') { diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts index ae525b4b8e3..0ddb5f308fc 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts @@ -1,7 +1,7 @@ /** * Resolve the active engine, flush paint uploads, compose the fresh snapshot, compile, and enqueue. Invalid - * controls block; rejected regions skip. Publish dedupe only after dispatch succeeds and report failures as - * notices. A per-project in-flight guard prevents overlapping preparation. + * controls and unsupported hosted-provider regions block. Publish dedupe only after dispatch succeeds and report + * failures as notices. A per-project in-flight guard prevents overlapping preparation. */ import type { @@ -34,7 +34,11 @@ import { type RegionalReferenceImageInput, resolveGenerateSeed, } from '@features/generation/graph'; -import { normalizeGenerateWidgetValues, syncGenerateWidgetValuesWithModels } from '@features/generation/settings'; +import { + describeExternalCanvasRefusal, + normalizeGenerateWidgetValues, + syncGenerateWidgetValuesWithModels, +} from '@features/generation/settings'; import { getGenerationDevicesSnapshot, resolveRandDeviceMetadata } from '@features/queue/devices'; import { captureAccountScope, @@ -139,6 +143,9 @@ const createControlLayerCollector = ( return { shouldComposite: (layer) => { + if (model.type === 'external_image_generator') { + throw new Error(describeExternalCanvasRefusal(model, { kind: 'control_layer', layerName: layer.name })); + } const { adapter } = layer; const resolved = adapter.model ? models?.find((candidate) => candidate.key === adapter.model) : undefined; const rejection = validate({ @@ -243,8 +250,8 @@ export const resolveRegionalReferenceImages = ( }; /** - * Resolve regional references and silently skip unsupported/noncontributing regions. Join uploaded masks with - * captured prompt/reference metadata for graph inputs. + * Reject hosted-provider regions; resolve local references and skip unsupported/noncontributing local regions. + * Join uploaded masks with captured prompt/reference metadata for graph inputs. */ const createRegionalGuidanceCollector = ( model: GenerateModelConfig @@ -256,6 +263,9 @@ const createRegionalGuidanceCollector = ( return { shouldComposite: (layer) => { + if (model.type === 'external_image_generator') { + throw new Error(describeExternalCanvasRefusal(model, { kind: 'regional_guidance', layerName: layer.name })); + } const referenceImages = resolveRegionalReferenceImages(layer, model.base); const rejection = getRegionalGuidanceRejectionReason({ hasContent: true, @@ -321,7 +331,10 @@ export const runCanvasInvocation = async (deps: RunCanvasInvocationDeps): Promis // Mode-union drift guard: `detectCanvasMode`'s facts/mode types must stay // structurally identical to canvas-operations' `GenerationModeFacts` / // `GenerationCompositeMode` — tsc fails here on drift in either direction. - detectMode: (facts) => detectCanvasMode(facts), + // A hosted model takes the frame's pixels as they are, with no masked tail: bounds that overlap the frame with + // nothing visible inside it leave an empty frame, which it generates from text. + detectMode: (facts) => + model.type === 'external_image_generator' && !facts.bboxHasContent ? 'txt2img' : detectCanvasMode(facts), shouldCompositeControlLayer: controls.shouldComposite, shouldCompositeRegionalMask: regions.shouldComposite, signal: deps.signal, diff --git a/invokeai/frontend/webv2/src/workbench/workbenchState.test.ts b/invokeai/frontend/webv2/src/workbench/workbenchState.test.ts index 7014f1b0f36..fb9f8e847d5 100644 --- a/invokeai/frontend/webv2/src/workbench/workbenchState.test.ts +++ b/invokeai/frontend/webv2/src/workbench/workbenchState.test.ts @@ -262,6 +262,14 @@ const createGenerateValues = (overrides: Partial = {}): Ge ideogram4GuidanceScale: null, ideogram4Mu: null, ideogram4ColorPalette: [], + externalImageSize: null, + openaiQuality: 'auto', + openaiBackground: 'auto', + openaiInputFidelity: null, + geminiTemperature: null, + geminiThinkingLevel: null, + seedreamWatermark: false, + seedreamOptimizePrompt: false, krea2RebalanceEnabled: false, krea2RebalanceMultiplier: 4, krea2RebalanceWeights: '1.0,1.0,1.0,1.0,1.0,1.0,1.0,2.5,5.0,1.1,4.0,1.0', diff --git a/tests/app/invocations/test_frontend_external_canvas_graphs.py b/tests/app/invocations/test_frontend_external_canvas_graphs.py new file mode 100644 index 00000000000..1e513fd2b52 --- /dev/null +++ b/tests/app/invocations/test_frontend_external_canvas_graphs.py @@ -0,0 +1,233 @@ +"""Validate the frontend's hosted Canvas graphs against the real invocation fields and edge types, and run them. + +The fixture is generated by compileCanvasGraph.test.ts. Graph.validate_self checks the collection-to-item conversion +and the image/metadata inputs of the output nodes. The execution test runs a graph through GraphExecutionState with +real image storage and a fake provider in place of the network call. +""" + +import json +import logging +from pathlib import Path +from types import SimpleNamespace +from typing import Any + +import pytest +from PIL import Image + +from invokeai.app.invocations.primitives import ImageOutput +from invokeai.app.services.config.config_default import InvokeAIAppConfig +from invokeai.app.services.external_generation.external_generation_base import ExternalProvider +from invokeai.app.services.external_generation.external_generation_common import ( + ExternalGeneratedImage, + ExternalGenerationRequest, + ExternalGenerationResult, +) +from invokeai.app.services.external_generation.external_generation_default import ExternalGenerationService +from invokeai.app.services.image_files.image_files_disk import DiskImageFileStorage +from invokeai.app.services.invoker import Invoker +from invokeai.app.services.model_records.model_records_sql import ModelRecordServiceSQL +from invokeai.app.services.names.names_default import SimpleNameService +from invokeai.app.services.session_queue.session_queue_common import SessionQueueItem +from invokeai.app.services.shared.database.database import Database +from invokeai.app.services.shared.graph import EdgeConnection, Graph, GraphExecutionState +from invokeai.app.services.shared.invocation_context import InvocationContextData, build_invocation_context +from invokeai.app.services.urls.urls_default import LocalUrlService +from invokeai.backend.model_manager.configs.external_api import ExternalApiModelConfig, ExternalModelCapabilities + +FIXTURE_PATH = ( + Path(__file__).parents[3] + / "invokeai/frontend/webv2/src/features/generation/core/__snapshots__/externalCanvasGraphs.json" +) +GRAPHS: dict[str, Any] = { + name: graph for name, graph in json.loads(FIXTURE_PATH.read_text(encoding="utf-8")).items() if name != "_comment" +} + + +# The Canvas bbox is 640x512; GPT Image 1's size table answers its 5:4 frame with the 3:2 size 1536x1024. +BBOX = (640, 512) +PROVIDER_SIZE = (1536, 1024) + + +def _sources(graph: Graph) -> dict[tuple[str, str], EdgeConnection]: + return {(edge.destination.node_id, edge.destination.field): edge.source for edge in graph.edges} + + +@pytest.mark.parametrize("mode", ["txt2img", "img2img"]) +def test_hosted_canvas_graph_is_valid_for_the_backend(mode: str) -> None: + graph = Graph.model_validate(GRAPHS[mode]) + graph.validate_self() + for node_id, raw in GRAPHS[mode]["nodes"].items(): + node = graph.nodes[node_id] + unknown = set(raw) - set(type(node).model_fields) + if type(node).model_config.get("extra") == "allow": + # Core metadata intentionally preserves provider-specific fields beyond its declared schema. + assert {key: node.model_dump()[key] for key in unknown} == {key: raw[key] for key in unknown} + else: + assert not unknown, f"{raw['type']} has no field(s) {sorted(unknown)}" + + provider = graph.nodes["canvas_processing_output"] + assert provider.mode == mode + assert provider.num_images == 1 + assert provider.is_intermediate + # The provider is asked for a size from its own table, so the backend has nothing to re-bucket. + assert (provider.width, provider.height) == PROVIDER_SIZE + + +def test_text_to_image_covers_then_center_crops_the_provider_image_to_the_bbox() -> None: + graph = Graph.model_validate(GRAPHS["txt2img"]) + sources = _sources(graph) + assert graph.nodes["canvas_processing_output"].init_image is None + assert ("canvas_processing_output", "init_image") not in sources + + output = graph.nodes["canvas_output"] + assert output.get_type() == "img_crop" + assert (output.width, output.height) == BBOX + cover = graph.nodes[sources[("canvas_output", "image")].node_id] + assert cover.get_type() == "img_resize" + # 1536x1024 scaled by max(640/1536, 512/1024) = 0.5 covers the bbox; the 128 px overflow is split evenly. + assert (cover.width, cover.height) == (768, 512) + assert (output.x, output.y) == (64, 0) + assert sources[(cover.id, "image")] == EdgeConnection(node_id="canvas_provider_image", field="item") + + +def test_image_to_image_sends_the_composite_at_the_request_size_and_resizes_the_result_to_the_bbox() -> None: + graph = Graph.model_validate(GRAPHS["img2img"]) + sources = _sources(graph) + assert graph.nodes["canvas_processing_output"].init_image is None + request_image = graph.nodes[sources[("canvas_processing_output", "init_image")].node_id] + assert request_image.get_type() == "img_resize" + assert (request_image.width, request_image.height) == PROVIDER_SIZE + assert request_image.image is not None and request_image.image.image_name == "canvas-composite.png" + + output = graph.nodes["canvas_output"] + assert output.get_type() == "img_resize" + assert (output.width, output.height) == BBOX + assert sources[("canvas_output", "image")] == EdgeConnection(node_id="canvas_provider_image", field="item") + + +@pytest.mark.parametrize("mode", ["txt2img", "img2img"]) +def test_canvas_output_retains_the_provider_output_metadata(mode: str) -> None: + graph = Graph.model_validate(GRAPHS[mode]) + sources = _sources(graph) + metadata_reader = graph.nodes[sources[("canvas_output", "metadata")].node_id] + assert metadata_reader.get_type() == "metadata_from_image" + # The reader and the output's image chain both start at the same iterate item, which pairs them per image. + image_source = sources[("canvas_output", "image")] + while image_source.field != "item": + image_source = sources[(image_source.node_id, "image")] + assert sources[(metadata_reader.id, "image")] == image_source + assert graph.nodes[image_source.node_id].get_type() == "iterate" + # Core dimensions enter the provider, which adds its authoritative request identity, seed and options. + provider_metadata = graph.nodes[sources[("canvas_processing_output", "metadata")].node_id] + assert provider_metadata.get_type() == "core_metadata" + assert (provider_metadata.width, provider_metadata.height) == BBOX + + +PROVIDER_COLORS = [(200, 30, 30), (30, 200, 30)] + + +class _FakeOpenAIProvider(ExternalProvider): + """Stands in for the network call: one solid-colored image per requested image, each with its own seed.""" + + provider_id = "openai" + + def __init__(self, app_config: InvokeAIAppConfig, logger: logging.Logger) -> None: + super().__init__(app_config, logger) + self.requests: list[ExternalGenerationRequest] = [] + + def is_configured(self) -> bool: + return True + + def generate(self, request: ExternalGenerationRequest) -> ExternalGenerationResult: + self.requests.append(request) + return ExternalGenerationResult( + images=[ + ExternalGeneratedImage(image=Image.new("RGB", PROVIDER_SIZE, PROVIDER_COLORS[index]), seed=1000 + index) + for index in range(request.num_images) + ], + provider_request_id="req-canvas", + provider_metadata={"revised_prompt": "canvas"}, + ) + + +@pytest.mark.parametrize("num_images", [1, 2]) +def test_each_provider_image_reaches_the_canvas_at_bbox_size_with_its_own_metadata( + num_images: int, mock_invoker: Invoker, mock_sqlite_database: Database, tmp_path: Path +) -> None: + services = mock_invoker.services + services.image_files = DiskImageFileStorage(tmp_path / "images") + services.names = SimpleNameService() + services.urls = LocalUrlService() + provider = _FakeOpenAIProvider(services.configuration, logging.getLogger()) + services.external_generation = ExternalGenerationService({"openai": provider}, logging.getLogger()) + services.image_files.start(mock_invoker) + services.images.start(mock_invoker) + + raw_graph = GRAPHS["txt2img"] + raw_model = raw_graph["nodes"]["canvas_processing_output"]["model"] + store = ModelRecordServiceSQL(mock_sqlite_database, logging.getLogger()) + store.add_model( + ExternalApiModelConfig( + key=raw_model["key"], + name=raw_model["name"], + provider_id=raw_model["provider_id"], + provider_model_id=raw_model["provider_model_id"], + capabilities=ExternalModelCapabilities.model_validate(raw_model["capabilities"]), + ) + ) + services.model_manager = SimpleNamespace(store=store) # type: ignore[assignment] + + raw_nodes = {**raw_graph["nodes"]} + raw_nodes["canvas_processing_output"] = {**raw_nodes["canvas_processing_output"], "num_images": num_images} + session = GraphExecutionState(graph=Graph.model_validate({**raw_graph, "nodes": raw_nodes})) + queue_item = SessionQueueItem( + item_id=1, + batch_id="batch-1", + session_id=session.id, + created_at="2026-01-01T00:00:00", + updated_at="2026-01-01T00:00:00", + started_at=None, + completed_at=None, + queue_id="default", + session=session, + ) + + executed_sources: list[str] = [] + while (invocation := session.next()) is not None: + source_id = session.prepared_source_mapping[invocation.id] + context = build_invocation_context( + services=services, + data=InvocationContextData(queue_item=queue_item, invocation=invocation, source_invocation_id=source_id), + is_canceled=lambda: False, + ) + session.complete(invocation.id, invocation.invoke(context)) + executed_sources.append(source_id) + + assert not session.has_error() + assert [(request.width, request.height) for request in provider.requests] == [PROVIDER_SIZE] + # One provider request, then the iterate expands every node downstream of it once per returned image. + assert executed_sources.count("canvas_processing_output") == 1 + assert executed_sources.count("canvas_provider_image") == num_images + assert executed_sources.count("canvas_provider_metadata") == num_images + + outputs = [ + session.results[node_id] + for node_id, source_id in session.prepared_source_mapping.items() + if source_id == "canvas_output" + ] + assert len(outputs) == num_images + seeds_by_color = {color: 1000 + index for index, color in enumerate(PROVIDER_COLORS[:num_images])} + output_seeds = [] + for output in outputs: + assert isinstance(output, ImageOutput) + assert (output.width, output.height) == BBOX + image = services.images.get_pil_image(output.image.image_name).convert("RGB") + assert image.size == BBOX + metadata = services.images.get_metadata(output.image.image_name) + assert metadata is not None + # The metadata travels with the image it was read from: its seed names the provider image of these pixels. + assert metadata.root["external_seed"] == seeds_by_color[image.getpixel((320, 256))] + assert metadata.root["external_request_id"] == "req-canvas" + assert metadata.root["external_provider_metadata"] == {"revised_prompt": "canvas"} + output_seeds.append(metadata.root["external_seed"]) + assert sorted(output_seeds) == sorted(seeds_by_color.values()) diff --git a/tests/app/services/external_generation/test_external_generation_service.py b/tests/app/services/external_generation/test_external_generation_service.py index 2b6ef3b6ca5..07c0a79b203 100644 --- a/tests/app/services/external_generation/test_external_generation_service.py +++ b/tests/app/services/external_generation/test_external_generation_service.py @@ -1,4 +1,5 @@ import logging +from fractions import Fraction import pytest from PIL import Image @@ -291,28 +292,31 @@ def test_generate_snaps_unlisted_ratio_when_model_has_no_bucket_sizes() -> None: assert (provider.last_request.width, provider.last_request.height) == (1360, 765) -def _starter_sizes() -> list[tuple[str, str, int, int]]: +def _starter_sizes() -> list[tuple[str, str, int, int, bool]]: """Every fixed size a txt2img external starter model offers: its resolution presets and ratio buckets.""" - cases: list[tuple[str, str, int, int]] = [] + cases: list[tuple[str, str, int, int, bool]] = [] for starter in STARTER_MODELS: capabilities = starter.capabilities if capabilities is None or "txt2img" not in capabilities.modes: continue cases.extend( - (starter.source, preset.label, preset.width, preset.height) + (starter.source, preset.aspect_ratio, preset.width, preset.height, False) for preset in capabilities.resolution_presets or [] ) cases.extend( - (starter.source, ratio, size.width, size.height) + (starter.source, ratio, size.width, size.height, True) for ratio, size in (capabilities.aspect_ratio_sizes or {}).items() ) return cases -@pytest.mark.parametrize("source, label, width, height", _starter_sizes()) -def test_starter_model_offered_sizes_are_accepted(source: str, label: str, width: int, height: int) -> None: +@pytest.mark.parametrize("source, ratio, width, height, is_bucket_size", _starter_sizes()) +def test_starter_model_offered_sizes_are_accepted( + source: str, ratio: str, width: int, height: int, is_bucket_size: bool +) -> None: """A size a model offers in its own presets must be accepted at (nearly) that size, even when its declared ratio is not in lowest terms ("21:9") and the preset is only approximately that ratio (1024x439).""" + # The frontend's Canvas picks its provider request size from these capabilities and relies on this acceptance. starter = next(model for model in STARTER_MODELS if model.source == source) assert starter.capabilities is not None provider_id, provider_model_id = source.removeprefix("external://").split("/", 1) @@ -328,10 +332,18 @@ def test_starter_model_offered_sizes_are_accepted(source: str, label: str, width service.generate(_build_request(model=model, width=width, height=height)) - assert provider.last_request is not None - # Snapping to an exact 21:9 moves in 21x9 px steps, so a small preset (512x219 -> 504x216) can shift ~2%. - assert provider.last_request.width == pytest.approx(width, rel=0.02) - assert provider.last_request.height == pytest.approx(height, rel=0.02) + sent = provider.last_request + assert sent is not None + if is_bucket_size: + # Bucket sizes pass unchanged: the Canvas frames its output crop around this exact size. + assert (sent.width, sent.height) == (width, height) + else: + # An approximate preset may snap onto its exact declared ratio (1024x819 -> 1025x820), never onto another + # ratio. Exact 21:9 moves in 21x9 px steps, so a small preset (512x219 -> 504x216) can shift ~2%. + ratio_width, ratio_height = (int(part) for part in ratio.split(":")) + assert Fraction(sent.width, sent.height) == Fraction(ratio_width, ratio_height) + assert sent.width == pytest.approx(width, rel=0.02) + assert sent.height == pytest.approx(height, rel=0.02) def test_generate_matches_declared_ratio_not_in_lowest_terms() -> None: