diff --git a/docs/src/content/docs/Users Guide/07.Canvas/01.introduction.mdx b/docs/src/content/docs/Users Guide/07.Canvas/01.introduction.mdx index abe71656d1a..4c8733dd1ba 100644 --- a/docs/src/content/docs/Users Guide/07.Canvas/01.introduction.mdx +++ b/docs/src/content/docs/Users Guide/07.Canvas/01.introduction.mdx @@ -1,7 +1,7 @@ --- title: Crafting images with the Canvas description: An introduction to the Canvas — layers and masks, sending generations to the Canvas, inpainting, outpainting, regional prompts, control layers, drawing and adding gallery images. -lastUpdated: 2026-10-05 +lastUpdated: 2026-10-24 --- import { Image } from 'astro:assets'; @@ -132,26 +132,34 @@ Regional guidance works with SD 1.5, SDXL, FLUX.1, FLUX.2, Krea-2, Z-Image and A ## Control layers -A control layer shapes the composition with a guide image: the edges of a sketch, the depth of a scene, or a person's pose. A control model makes the generated image follow it. +A control layer shapes the composition with a guide image: the edges of a sketch, the depth of a scene, or a person's pose. A control model makes the generated image follow it. You choose what the layer controls, and InvokeAI detects the matching map from your image and picks the model. -A control layer's settings in the Layer section of Properties: Adapter type ControlNet, Model Depth Map, and Weight 0.75. +A control layer's settings in the Layer section of Properties. -1. **Put the guide image on a control layer.** A control layer's image is its own pixels; there is no separate image picker. Do one of these: - * Right-click a raster layer and choose **Copy to → Copy to control layer**, or **Convert to → Convert to control layer**. - * Drag an image from the Gallery onto the canvas and drop it on **Add as Control Layer** (see [below](#adding-a-gallery-image-to-a-layer)). - * Add an empty **Control layer** from the **+** menu and draw on it. +1. **Add a control layer with an image.** Do one of these: + * Choose **+ → Control layer**, then **Upload** an image, use the **Frame** (what is visible inside the generation frame), or **Draw** the guide yourself. These buttons are in the layer's Properties, in its **Image** row. + * Right-click an image in the Gallery and choose **Use as Control**. It fills the selected control layer if that layer has no image yet, and otherwise adds a new control layer. + * Right-click a raster layer and choose **Use as control layer**, which copies its pixels to a new control layer. + * Drag an image from the Gallery onto the canvas and drop it on **Add as Control Layer**. As with **Use as Control**, it fills the selected control layer if that layer is still empty. + * Select an empty control layer and drop an image file from your computer onto the canvas, where the layer shows *Drop an image here*. The file is uploaded into that layer. -2. **Choose the model.** Select the layer, open **Properties** and scroll to its **Layer** section. Choose the **Adapter type** (ControlNet, T2I Adapter, Control LoRA or Z-Image ControlNet) and the **Model**. A compatible installed model is chosen for you when the layer is created. +2. **Choose the Type**: **Pose**, **Depth**, **Edges**, **Soft edges**, **Line art**, **Sketch**, **Color** or **Tile**. Next to each type, the menu says how many installed models are made for it, *via Union* when only a Union model covers it, or *Not installed*. -3. **Process the image, if needed.** Most control models expect a processed image, such as an edge map or a depth map, rather than a photograph. Choosing a model starts its recommended **Filter** automatically; you can also click **Filter**, pick a filter, and **Apply** it. +3. **Check the Detection.** The type's detector turns your image into the map the model expects, such as a depth map or an edge map, in the background. If your image is already such a map, choose the **None** option, for example *None: image is already a depth map*. See [Control layer](/users-guide/canvas/layers-panel/#control-layer) for the details. -4. **Adjust the influence** with **Weight** (how strongly the guide is followed), **Step range** (which part of the generation it applies to) and, for ControlNet, **Control mode**. Then press Invoke. +4. **Check the Model.** InvokeAI picks it for you, and the line under the **Model** row says why. Choose another model from the list if you prefer. A model made for the type, or a Union model that covers it, is then remembered for that type; any other model is used for this layer only. + +5. **Adjust the influence** with **Weight** (how strongly the guide is followed), **Steps** (which part of the generation it applies to) and, for ControlNet, **Priority**. Then press Invoke. -Control layers work with SD 1.5, SDXL, FLUX.1 and Z-Image, and need a control model for the same family, installed from the Model Manager's starter models. If the **Model** list is empty, no compatible model is installed for the model selected in the Generate panel. A control layer without a model stops Invoke with the message *Select a control model before generating.* +Once a map has been detected, a switch above the canvas shows the layer as its **Original** image, the detected **Map**, or an **Overlay** of the two. The switch changes only what you see; generation always uses the map. + +An empty control layer, or one with neither a type nor a model, only shows a warning and contributes nothing; Invoke stays available. A control layer with an image blocks Invoke if no model can be chosen for it, or if its map is out of date. **Invoke** then shows the number of issues, for example *Invoke · 1 issue*; click it to select the first layer that needs attention. + +Control layers work with SD 1.5, SDXL, FLUX.1, FLUX.2 [dev], Qwen Image, Z-Image and Anima, and need a control model for the same family, installed from the Model Manager's starter models. If no model fits, the **Model** row says so, with a link to the Model Manager when you can manage models. ## Drawing on a raster layer diff --git a/docs/src/content/docs/Users Guide/07.Canvas/02.canvas-guide.mdx b/docs/src/content/docs/Users Guide/07.Canvas/02.canvas-guide.mdx index 6f2ffb72585..785bafa34f1 100644 --- a/docs/src/content/docs/Users Guide/07.Canvas/02.canvas-guide.mdx +++ b/docs/src/content/docs/Users Guide/07.Canvas/02.canvas-guide.mdx @@ -1,7 +1,7 @@ --- title: Canvas guide description: A detailed guide to the Canvas — its tools and their options, navigation, the header buttons, the generation frame, selections, transform, filters, the staging area, menus, settings and keyboard shortcuts. -lastUpdated: 2026-10-18 +lastUpdated: 2026-10-24 --- import { Image } from 'astro:assets'; @@ -112,7 +112,7 @@ To crop a layer to the generation frame, choose **Crop layer to bbox** from its ## Filters and object selection -**Filter**, in a layer's menu or its Properties, processes a raster or control layer. Starting it opens the Layers panel's **Properties** tab, headed *Operation: Filter*, where you choose and preview the filter. The filters are grouped by purpose: +**Filter**, in a raster layer's menu or its Properties, processes the layer. A control layer has no Filter: its map comes from the **Detection** row in its [Properties](/users-guide/canvas/layers-panel/#control-layer). Starting Filter opens the Layers panel's **Properties** tab, headed *Operation: Filter*, where you choose and preview the filter. The filters are grouped by purpose: * **Edge detection** — Canny, HED, Lineart, Lineart Anime, MLSD, PiDiNet and Scribble. * **Depth & normal** — Depth Anything and Normal map. diff --git a/docs/src/content/docs/Users Guide/07.Canvas/03.layers-panel.mdx b/docs/src/content/docs/Users Guide/07.Canvas/03.layers-panel.mdx index 84d61ac098a..767ac24af25 100644 --- a/docs/src/content/docs/Users Guide/07.Canvas/03.layers-panel.mdx +++ b/docs/src/content/docs/Users Guide/07.Canvas/03.layers-panel.mdx @@ -1,7 +1,7 @@ --- title: Layers panel guide description: A detailed guide to the Canvas Layers panel — the color picker, managing layers, history, the layer menu, and every layer type's settings in the Properties pane. -lastUpdated: 2026-09-30 +lastUpdated: 2026-10-24 --- import { Image } from 'astro:assets'; @@ -17,7 +17,7 @@ The **Layers** panel sits to the right of the Canvas in the **Edit** layout. It Drag the dividers between the parts to resize them, and use the chevron at the right of the Color and Properties tabs to collapse them. The sizes are saved with the project. -The Layers panel: the color wheel at the top, the layer list grouped into Inpaint Masks, Regional Guidance, Control Layers and Raster Layers, and the Properties pane showing a control layer's Adapter type, Model and Weight. +The Layers panel: the color wheel at the top, the layer list grouped into Inpaint Masks, Regional Guidance, Control Layers and Raster Layers, and the Properties pane showing a control layer's settings. For an introduction to the layer types, see [Crafting images with the Canvas](/users-guide/canvas/introduction/#canvas-concepts). @@ -89,9 +89,10 @@ The icons at the top **Arrange** the layer (move it to the front, forward, backw * **Add adjustment** — Brightness/Contrast, Exposure, Levels, Curves, Saturation, Hue, Invert, Texture, Clarity, Dehaze or Noise. Adjustments are non-destructive and appear as child rows. Texture, Clarity and Dehaze change detail and contrast; Noise adds film grain. * **Add regenerate region** — treats the layer's own shape as an inpaint mask. * **Filter**, **Select object** and **Run workflow** — process the layer; see the [Canvas guide](/users-guide/canvas/canvas-guide/#filters-and-object-selection). + * **Use as control layer** — copies the layer's pixels to a new control layer, whose Properties then ask for its **Type**. * **Boolean operations** — combine the layer with the one below: intersect, cut out, cut away or exclude. - * **Convert to** — make it a control layer, inpaint mask or regional guidance layer. Text, shape and gradient layers can also be rasterized into plain pixels. Control layers have no adjustments, so converting or copying a layer with adjustments to a control layer bakes them, with the layer's scaling and rotation, into its pixels; undo restores the raster layer with its adjustments. -* **Control layers:** **Transform**, **Filter**, **Select object**, **Run workflow**, **Enable transparent preview**, and **Convert to raster layer**. + * **Convert to** — make it a control layer, inpaint mask or regional guidance layer. Text, shape and gradient layers can also be rasterized into plain pixels. Control layers have no adjustments, so converting a layer with adjustments to a control layer, or using it as one, bakes them, with the layer's scaling and rotation, into its pixels; undo restores the raster layer with its adjustments. +* **Control layers:** **Transform**, **Select object**, **Run workflow**, **Enable transparent preview**, and **Convert to raster layer**. A control layer's map comes from its **Detection** row in Properties, so it has no **Filter**. * **Inpaint masks:** **Add noise**, **Add denoise limit**, **Extract masked area** (copy the masked part of the image to a new layer), and **Hide layer**. * **Regional guidance:** **Enable auto-negative**, **Add reference image**, and **Hide layer**. @@ -160,15 +161,25 @@ Things to know about Texture, Clarity and Dehaze: ### Control layer -* **Adapter type** — ControlNet, T2I Adapter, Control LoRA, Z-Image ControlNet or Anima ControlNet-LLLite, depending on the model selected in the Generate panel. -* **Model** — the installed control models of that type for the same model family. If the list is empty, install one from the starter models. For Anima, the list holds the LLLite control adapters but not the Inpainting adapter, which is chosen as the **Inpaint Adapter** in the Generate panel's Components section instead, and each LLLite model can drive only one control layer per generation. -* **Weight** — how strongly the guide is followed (0 to 2; the default is 0.75 for ControlNet). -* **Step range** — the part of the generation, from start to finish, in which the guide applies. -* **Control mode** (ControlNet only) — **Balanced**, **More prompt**, **More control** or **Unbalanced**. -* **Transparent preview** — shows the layer semi-transparently over the image. -* **Filter** — process the layer's image into a guide, such as an edge or depth map. +A control layer's Layer section always has the same rows, in this order: **Image**, **Type**, **Detection** and **Model**, then **Weight**, **Steps** and **Priority**. The rows stay in place; only their values change. -The control image is the layer's own pixels. To use a picture, copy or convert a raster layer, or drop a gallery image on **Add as Control Layer**. +* **Image** — the picture the guide comes from. On an empty layer, the row has three buttons: **Upload** an image from your computer, use the **Frame** (what is visible inside the generation frame), or **Draw** the control image yourself on the layer. You can also drop an image file from your computer onto the canvas while the empty layer is selected: the canvas shows *Drop an image here* over the layer, and the file is uploaded into it. Once the layer has an image, the row shows its thumbnail and a label, such as the file name, and **Replace** offers the same three choices. The layer keeps this original image beside the detected map, so detection can run again on it. +* **Type** — what the layer controls: **Pose**, **Depth**, **Edges**, **Soft edges**, **Line art**, **Sketch**, **Color** or **Tile**. In the menu, each type shows how many installed models are made for it, *via Union* when only a Union model covers it, or *Not installed*. Changing the type sets **Model** back to automatic, and on a layer with an image it runs the new type's default detector unless the current one also belongs to the new type. +* **Detection** — the detector that turns the image into the type's map, for example Depth Anything or a normal map for Depth, Canny or MLSD for Edges, and DW Openpose or Mediapipe Face for Pose. The **None** option, such as *None: image is already a depth map*, uses the image as it is. Until the layer has an image, the row reads *Chosen when an image is added* and its detectors cannot be chosen; the first image gets the type's default detector. **Sketch**, and an image you draw yourself, use **None**; a drawn image counts once it has a stroke. **Tile** has no detector and uses the image directly. + * The **Detector settings** button next to it opens the detector's options. Changing the detector or its settings runs detection again in the background. A spinner shows while it is waiting, with *Waiting to run* and the detector's name, and while it runs, with *Running* and the detector's name. + * If detection fails, a **Run again** button appears; hover over it for the reason. If the map is out of date, a **Detect again** button appears. +* **Model** — chosen for you, in this order: the model you picked for this layer; the model you last picked for this type on the same main model family; the only installed model made for the type; a Union model that covers the type. When several models fit equally, the row asks you to choose one. The line under the row says which case applies, for example *ControlNet · your only depth model*. The menu groups the models under **Made for** the type, **Union** and **Purpose not set**; a layer without a type lists every installed control model for the model family under **Installed models**. Picking a model made for the type, or a Union model that covers it, remembers it for every layer of that type and model family, in every project; a **Purpose not set** model, or one made for another type, is used for this layer only. **Reset** drops this layer's own pick, so the remembered model, if any, or the automatic choice takes over; when the remembered model is the one in use, Reset forgets it and the layer returns to automatic. If no model fits, the line says so. For Anima, the list holds the ControlNet-LLLite control adapters but not the Inpainting adapter, which is chosen as the **Inpaint Adapter** in the Generate panel's Components section instead, and each LLLite model can drive only one control layer per generation. +* **Weight** — how strongly the guide is followed. The slider runs from 0 to 2; for ControlNet, T2I Adapter and Control LoRA, the number field also accepts values down to −1. The default is 0.75 for ControlNet. +* **Steps** — the part of the generation, from start to finish, in which the guide applies. +* **Priority** — **Prompt**, **Balanced** or **Control**: whether the generation leans toward the prompt or toward the guide. It applies to SD 1.5 and SDXL ControlNet models only. A layer saved with the earlier **Unbalanced** mode shows it as a fourth, selected option until you choose another; new layers do not offer it. + +The layer's row in the list reads *Type · Detector · weight*, for example *Depth · Depth Anything · 0.75*, or *No image* or *Choose a type* while the layer is incomplete. A spinner shows on the row while its detector is waiting or running, and a warning icon when the layer needs attention; hover over the icon for the reason. + +Once a map has been detected, a switch above the canvas shows the layer as its **Original** image, the detected **Map**, or an **Overlay** of the two. It changes only the view. **Enable transparent preview**, in the layer menu, shows the layer semi-transparently over the image. + +A control layer with no image, or with neither a type nor a model, only shows a warning: Invoke stays available and the layer contributes nothing. A layer with an image blocks Invoke when no model can be chosen for it or when its map is out of date. **Invoke** then shows the number of issues, and clicking it selects the first layer that needs attention. + +A control layer made in an earlier version of Invoke keeps its model and works without a type; its row reads its adapter and weight. Choosing a **Type** for it sets its model back to automatic and can run the type's default detector on its pixels. ### Inpaint mask diff --git a/docs/src/content/docs/development/Front End/project-files.mdx b/docs/src/content/docs/development/Front End/project-files.mdx index 5dcf56162fb..21a501637a1 100644 --- a/docs/src/content/docs/development/Front End/project-files.mdx +++ b/docs/src/content/docs/development/Front End/project-files.mdx @@ -44,10 +44,12 @@ would drop part of it on load and lose it on the next autosave. Schema 4 marks c typography (a text source with `fontRef`, `fontStyle` or `fontVariations`). Schema 5 marks a noise, clarity, texture or dehaze adjustment, enabled or not, on any layer or group at any depth; an older reader would drop the -whole adjustment stack holding it. The live document and each saved canvas snapshot are versioned +whole adjustment stack holding it. Schema 6 marks a control layer carrying any derivation field +(`controlType`, `detection`, `original` or `originalTransform`), which an older reader would strip, +losing the kept original. The live document and each saved canvas snapshot are versioned this way, and the canvas state takes the newest of them. Adoption is sticky: removing the feature -does not lower the version, as the server floor cannot be lowered either. A canvas with neither -stays at 3, so older builds keep opening it. A build refuses a canvas that declares a schema above +does not lower the version, as the server floor cannot be lowered either. A canvas with none of +these stays at 3, so older builds keep opening it. A build refuses a canvas that declares a schema above its maximum as an unsupported version before parsing it, and once the floor is raised the server refuses the project to a client declaring a lower maximum. diff --git a/invokeai/backend/flux/controlnet/instantx_controlnet_flux.py b/invokeai/backend/flux/controlnet/instantx_controlnet_flux.py index 1af5fbdfc09..e981ba074d9 100644 --- a/invokeai/backend/flux/controlnet/instantx_controlnet_flux.py +++ b/invokeai/backend/flux/controlnet/instantx_controlnet_flux.py @@ -143,8 +143,8 @@ def forward( controlnet_mode_emb = self.controlnet_mode_embedder(controlnet_mode) txt = torch.cat([controlnet_mode_emb, txt], dim=1) txt_ids = torch.cat([txt_ids[:, :1, :], txt_ids], dim=1) - else: - assert controlnet_mode is None + # A non-union model has no mode embedder, so a supplied controlnet_mode is ignored. Callers pick the mode from + # the model name, and some checkpoints keep "union" in the name after their mode embedding was removed. ids = torch.cat((txt_ids, img_ids), dim=1) pe = self.pe_embedder(ids) diff --git a/invokeai/frontend/webv2/AGENTS.md b/invokeai/frontend/webv2/AGENTS.md index b29403c8e85..5d5930fe3d5 100644 --- a/invokeai/frontend/webv2/AGENTS.md +++ b/invokeai/frontend/webv2/AGENTS.md @@ -32,7 +32,7 @@ Shared generated API contracts and their CI gates: [../api/AGENTS.md](../api/AGE No Python backend: `pnpm dev:mock` serves the app against the in-repo mock backend, which picks its own free port. Arguments go to `vite dev`. Default Vite port 5174 taken: pick a spare, `pnpm dev:mock --port 5273 --strictPort`. `pnpm dev` alone proxies to a real backend at `http://127.0.0.1:9090`. - `INVOKEAI_MOCK_BACKEND_PROFILE`: `representative` (default) or `empty`. `POST /__reset` on the mock origin printed at startup restores fixtures; `?profile=` switches them. Unimplemented endpoints return 404. -- `POST /__models` with `{"set": "anima-lllite"}` or `{"set": "spandrel"}` adds opt-in model seeds (adapters and upscalers installed before their channels or scale were recorded); re-identify fills those fields. `/__reset` drops them. +- `POST /__models` with `{"set": "anima-lllite"}` or `{"set": "spandrel"}` adds opt-in model seeds (adapters and upscalers installed before their channels or scale were recorded); re-identify fills those fields. `{"set": "control-sdxl"}` adds SDXL ControlNets naming their preprocessor in both id forms, a Union, an unpurposed model and a T2I-Adapter; `{"set": "control-flux"}` adds FLUX dev with a depth Control LoRA and a Union ControlNet. `/__reset` drops them. - Stop servers by PID (`ss -ltnp`), not `pkill -f`: its pattern can match the invoking shell. Restart after merges that add mock endpoints. - Provider/context error from a `?t=` module after source edits: full reload before calling it a product bug. HMR can split context modules. diff --git a/invokeai/frontend/webv2/performance/architecture-baseline.json b/invokeai/frontend/webv2/performance/architecture-baseline.json index 201a7296cb6..0126e029233 100644 --- a/invokeai/frontend/webv2/performance/architecture-baseline.json +++ b/invokeai/frontend/webv2/performance/architecture-baseline.json @@ -837,6 +837,7 @@ "source:src/workbench/canvas-engine/document-model/screenComposition.ts", "source:src/workbench/canvas-engine/document-model/semanticLeaf.ts", "source:src/workbench/canvas-engine/document-model/semanticNode.ts", + "source:src/workbench/canvas-engine/document/controlDerivation.ts", "source:src/workbench/canvas-engine/document/documentIndex.ts", "source:src/workbench/canvas-engine/document/documentTree.ts", "source:src/workbench/canvas-engine/document/insertionAnchors.ts", @@ -873,6 +874,8 @@ "source:src/workbench/components/WorkbenchUnavailableScreen.tsx", "source:src/workbench/controlAdapters.ts", "source:src/workbench/controlLayerChecks.ts", + "source:src/workbench/controlModelResolution.ts", + "source:src/workbench/controlTypes.ts", "source:src/workbench/deferredResource.ts", "source:src/workbench/diagnostics/loggingPreferences.ts", "source:src/workbench/extensions/contributionSource.ts", @@ -1086,8 +1089,10 @@ "source:src/workbench/widgets/generate/manifest.ts", "source:src/workbench/widgets/image-map/manifest.ts", "source:src/workbench/widgets/image-map/settingsContribution.ts", + "source:src/workbench/widgets/layers/layerPropertiesRequestStore.ts", "source:src/workbench/widgets/layers/manifest.ts", "source:src/workbench/widgets/layers/panes/editorPaneLayout.ts", + "source:src/workbench/widgets/layers/revealLayerProperties.ts", "source:src/workbench/widgets/manifests.ts", "source:src/workbench/widgets/notifications/manifest.ts", "source:src/workbench/widgets/preview/livePreviewFollow.tsx", diff --git a/invokeai/frontend/webv2/performance/browser-baseline.json b/invokeai/frontend/webv2/performance/browser-baseline.json index a39c5f61d08..0182034957a 100644 --- a/invokeai/frontend/webv2/performance/browser-baseline.json +++ b/invokeai/frontend/webv2/performance/browser-baseline.json @@ -29,12 +29,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 0, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 29, - "scriptRawBytes": 1897694, + "scriptRawBytes": 1884943, "scriptRequestCount": 25, - "totalRawBytes": 2158882 + "totalRawBytes": 2149807 }, "routeReadyMedianMs": 215.30000019073486, "scriptSourceOwnerSet": "launchpad-static", @@ -67,12 +67,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 0, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 29, - "scriptRawBytes": 1897694, + "scriptRawBytes": 1884943, "scriptRequestCount": 25, - "totalRawBytes": 2158882 + "totalRawBytes": 2149807 }, "routeReadyMedianMs": 225.39999961853027, "scriptSourceOwnerSet": "launchpad-static", @@ -105,12 +105,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 87, - "scriptRawBytes": 4312524, + "scriptRawBytes": 4326894, "scriptRequestCount": 82, - "totalRawBytes": 4618110 + "totalRawBytes": 4636156 }, "routeReadyMedianMs": 360.90000009536743, "scriptSourceOwnerSet": "editor-minimal-static", @@ -143,12 +143,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 87, - "scriptRawBytes": 4312524, + "scriptRawBytes": 4326894, "scriptRequestCount": 82, - "totalRawBytes": 4618110 + "totalRawBytes": 4636156 }, "routeReadyMedianMs": 367.80000019073486, "scriptSourceOwnerSet": "editor-minimal-static", @@ -159,12 +159,12 @@ "cssRawBytes": 0, "fontRawBytes": 0, "imageRawBytes": 0, - "largestAssetRawBytes": 317101, + "largestAssetRawBytes": 341949, "otherRawBytes": 0, "requestCount": 9, - "scriptRawBytes": 540817, + "scriptRawBytes": 567866, "scriptRequestCount": 9, - "totalRawBytes": 540817 + "totalRawBytes": 567866 }, "domContentLoadedMedianMs": 100.40000009536743, "id": "editor-canvas", @@ -181,12 +181,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 96, - "scriptRawBytes": 4853620, + "scriptRawBytes": 4894760, "scriptRequestCount": 91, - "totalRawBytes": 5159206 + "totalRawBytes": 5204022 }, "routeReadyMedianMs": 1095.3000001907349, "scriptSourceOwnerSet": "editor-canvas-static", @@ -197,12 +197,12 @@ "cssRawBytes": 0, "fontRawBytes": 0, "imageRawBytes": 0, - "largestAssetRawBytes": 317101, + "largestAssetRawBytes": 341949, "otherRawBytes": 0, "requestCount": 9, - "scriptRawBytes": 540817, + "scriptRawBytes": 567866, "scriptRequestCount": 9, - "totalRawBytes": 540817 + "totalRawBytes": 567866 }, "domContentLoadedMedianMs": 97.7000002861023, "id": "editor-canvas", @@ -219,12 +219,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 96, - "scriptRawBytes": 4853620, + "scriptRawBytes": 4894760, "scriptRequestCount": 91, - "totalRawBytes": 5159206 + "totalRawBytes": 5204022 }, "routeReadyMedianMs": 1042.5, "scriptSourceOwnerSet": "editor-canvas-static", @@ -235,12 +235,12 @@ "cssRawBytes": 15413, "fontRawBytes": 0, "imageRawBytes": 0, - "largestAssetRawBytes": 208808, + "largestAssetRawBytes": 208717, "otherRawBytes": 0, "requestCount": 7, - "scriptRawBytes": 452154, + "scriptRawBytes": 454188, "scriptRequestCount": 6, - "totalRawBytes": 467567 + "totalRawBytes": 469601 }, "domContentLoadedMedianMs": 101.40000009536743, "id": "editor-workflow", @@ -257,12 +257,12 @@ "cssRawBytes": 17572, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 94, - "scriptRawBytes": 4772177, + "scriptRawBytes": 4781082, "scriptRequestCount": 88, - "totalRawBytes": 5093176 + "totalRawBytes": 5105757 }, "routeReadyMedianMs": 1093.0999999046326, "scriptSourceOwnerSet": "editor-workflow-static", @@ -273,12 +273,12 @@ "cssRawBytes": 15413, "fontRawBytes": 0, "imageRawBytes": 0, - "largestAssetRawBytes": 208808, + "largestAssetRawBytes": 208717, "otherRawBytes": 0, "requestCount": 7, - "scriptRawBytes": 452154, + "scriptRawBytes": 454188, "scriptRequestCount": 6, - "totalRawBytes": 467567 + "totalRawBytes": 469601 }, "domContentLoadedMedianMs": 99.89999961853027, "id": "editor-workflow", @@ -295,12 +295,12 @@ "cssRawBytes": 17572, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 94, - "scriptRawBytes": 4772177, + "scriptRawBytes": 4781082, "scriptRequestCount": 88, - "totalRawBytes": 5093176 + "totalRawBytes": 5105757 }, "routeReadyMedianMs": 1105.0999999046326, "scriptSourceOwnerSet": "editor-workflow-static", @@ -333,12 +333,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 87, - "scriptRawBytes": 4312524, + "scriptRawBytes": 4326894, "scriptRequestCount": 82, - "totalRawBytes": 4618110 + "totalRawBytes": 4636156 }, "routeReadyMedianMs": 1324.5, "scriptSourceOwnerSet": "editor-minimal-static", @@ -371,12 +371,12 @@ "cssRawBytes": 2159, "fontRawBytes": 23664, "imageRawBytes": 44398, - "largestAssetRawBytes": 752431, - "otherRawBytes": 235365, + "largestAssetRawBytes": 752422, + "otherRawBytes": 239041, "requestCount": 87, - "scriptRawBytes": 4312524, + "scriptRawBytes": 4326894, "scriptRequestCount": 82, - "totalRawBytes": 4618110 + "totalRawBytes": 4636156 }, "routeReadyMedianMs": 1301.5, "scriptSourceOwnerSet": "editor-minimal-static", @@ -1381,7 +1381,9 @@ "source:src/workbench/canvas-engine/color.ts", "source:src/workbench/canvas-engine/contracts.ts", "source:src/workbench/canvas-engine/controllers/booleanMergeController.ts", + "source:src/workbench/canvas-engine/controllers/controlDerivationController.ts", "source:src/workbench/canvas-engine/controllers/controlPixelController.ts", + "source:src/workbench/canvas-engine/controllers/controlViewController.ts", "source:src/workbench/canvas-engine/controllers/copyLayerController.ts", "source:src/workbench/canvas-engine/controllers/cropLayerController.ts", "source:src/workbench/canvas-engine/controllers/editSteps.ts", @@ -1421,6 +1423,7 @@ "source:src/workbench/canvas-engine/document-model/semanticLeaf.ts", "source:src/workbench/canvas-engine/document-model/semanticNode.ts", "source:src/workbench/canvas-engine/document/bitmapStore.ts", + "source:src/workbench/canvas-engine/document/controlDerivation.ts", "source:src/workbench/canvas-engine/document/documentIndex.ts", "source:src/workbench/canvas-engine/document/documentMirror.ts", "source:src/workbench/canvas-engine/document/documentTree.ts", @@ -1526,6 +1529,7 @@ "source:src/workbench/canvas-engine/transform/transformMath.ts", "source:src/workbench/canvas-engine/types.ts", "source:src/workbench/canvas-engine/viewport.ts", + "source:src/workbench/canvas-operations/ControlDetectionCoordinator.ts", "source:src/workbench/canvas-operations/FilterOperationCoordinator.ts", "source:src/workbench/canvas-operations/SelectObjectCoordinator.ts", "source:src/workbench/canvas-operations/api.ts", @@ -1533,7 +1537,10 @@ "source:src/workbench/canvas-operations/backend/canvasImages.ts", "source:src/workbench/canvas-operations/canvasImportNotice.ts", "source:src/workbench/canvas-operations/compositeForGeneration.ts", - "source:src/workbench/canvas-operations/controlRecommendations.ts", + "source:src/workbench/canvas-operations/controlDetectionRunner.ts", + "source:src/workbench/canvas-operations/controlDetectionStatusStore.ts", + "source:src/workbench/canvas-operations/controlLayerIntents.ts", + "source:src/workbench/canvas-operations/controlSourceActions.ts", "source:src/workbench/canvas-operations/createCanvasEngine.ts", "source:src/workbench/canvas-operations/createCanvasFromImages.ts", "source:src/workbench/canvas-operations/createFromBbox.ts", @@ -1565,6 +1572,8 @@ "source:src/workbench/components/WorkbenchUnavailableScreen.tsx", "source:src/workbench/controlAdapters.ts", "source:src/workbench/controlLayerChecks.ts", + "source:src/workbench/controlModelResolution.ts", + "source:src/workbench/controlTypes.ts", "source:src/workbench/deferredResource.ts", "source:src/workbench/diagnostics/loggingPreferences.ts", "source:src/workbench/extensions/contributionSource.ts", @@ -1798,10 +1807,11 @@ "source:src/workbench/widgets/image-map/ImageMapDataRuntime.tsx", "source:src/workbench/widgets/image-map/manifest.ts", "source:src/workbench/widgets/image-map/settingsContribution.ts", - "source:src/workbench/widgets/layers/controlModelOptions.ts", "source:src/workbench/widgets/layers/layerOps.ts", + "source:src/workbench/widgets/layers/layerPropertiesRequestStore.ts", "source:src/workbench/widgets/layers/manifest.ts", "source:src/workbench/widgets/layers/panes/editorPaneLayout.ts", + "source:src/workbench/widgets/layers/revealLayerProperties.ts", "source:src/workbench/widgets/layers/selectedModel.ts", "source:src/workbench/widgets/manifests.ts", "source:src/workbench/widgets/notifications/NotificationsHeaderActions.tsx", @@ -2495,7 +2505,9 @@ "source:src/workbench/canvas-engine/color.ts", "source:src/workbench/canvas-engine/contracts.ts", "source:src/workbench/canvas-engine/controllers/booleanMergeController.ts", + "source:src/workbench/canvas-engine/controllers/controlDerivationController.ts", "source:src/workbench/canvas-engine/controllers/controlPixelController.ts", + "source:src/workbench/canvas-engine/controllers/controlViewController.ts", "source:src/workbench/canvas-engine/controllers/copyLayerController.ts", "source:src/workbench/canvas-engine/controllers/cropLayerController.ts", "source:src/workbench/canvas-engine/controllers/editSteps.ts", @@ -2535,6 +2547,7 @@ "source:src/workbench/canvas-engine/document-model/semanticLeaf.ts", "source:src/workbench/canvas-engine/document-model/semanticNode.ts", "source:src/workbench/canvas-engine/document/bitmapStore.ts", + "source:src/workbench/canvas-engine/document/controlDerivation.ts", "source:src/workbench/canvas-engine/document/documentIndex.ts", "source:src/workbench/canvas-engine/document/documentMirror.ts", "source:src/workbench/canvas-engine/document/documentTree.ts", @@ -2640,6 +2653,7 @@ "source:src/workbench/canvas-engine/transform/transformMath.ts", "source:src/workbench/canvas-engine/types.ts", "source:src/workbench/canvas-engine/viewport.ts", + "source:src/workbench/canvas-operations/ControlDetectionCoordinator.ts", "source:src/workbench/canvas-operations/FilterOperationCoordinator.ts", "source:src/workbench/canvas-operations/SelectObjectCoordinator.ts", "source:src/workbench/canvas-operations/api.ts", @@ -2647,7 +2661,10 @@ "source:src/workbench/canvas-operations/backend/canvasImages.ts", "source:src/workbench/canvas-operations/canvasImportNotice.ts", "source:src/workbench/canvas-operations/compositeForGeneration.ts", - "source:src/workbench/canvas-operations/controlRecommendations.ts", + "source:src/workbench/canvas-operations/controlDetectionRunner.ts", + "source:src/workbench/canvas-operations/controlDetectionStatusStore.ts", + "source:src/workbench/canvas-operations/controlLayerIntents.ts", + "source:src/workbench/canvas-operations/controlSourceActions.ts", "source:src/workbench/canvas-operations/createCanvasEngine.ts", "source:src/workbench/canvas-operations/createCanvasFromImages.ts", "source:src/workbench/canvas-operations/createFromBbox.ts", @@ -2683,6 +2700,8 @@ "source:src/workbench/components/WorkbenchUnavailableScreen.tsx", "source:src/workbench/controlAdapters.ts", "source:src/workbench/controlLayerChecks.ts", + "source:src/workbench/controlModelResolution.ts", + "source:src/workbench/controlTypes.ts", "source:src/workbench/deferredResource.ts", "source:src/workbench/diagnostics/loggingPreferences.ts", "source:src/workbench/extensions/contributionSource.ts", @@ -2919,6 +2938,7 @@ "source:src/workbench/widgets/canvas/CanvasSurface.tsx", "source:src/workbench/widgets/canvas/CanvasSurfaceContextLayout.tsx", "source:src/workbench/widgets/canvas/CanvasWidgetView.tsx", + "source:src/workbench/widgets/canvas/ControlLayerCanvasOverlay.tsx", "source:src/workbench/widgets/canvas/DenoisingStrengthWave.tsx", "source:src/workbench/widgets/canvas/GenerateCanvasCompositingSection.tsx", "source:src/workbench/widgets/canvas/GenerateCanvasRenderSize.tsx", @@ -2993,8 +3013,13 @@ "source:src/workbench/widgets/image-map/settingsContribution.ts", "source:src/workbench/widgets/layers/AdjustmentSettings.tsx", "source:src/workbench/widgets/layers/AdjustmentStatusIcon.tsx", + "source:src/workbench/widgets/layers/ControlAdapterRows.tsx", + "source:src/workbench/widgets/layers/ControlDetectionRow.tsx", "source:src/workbench/widgets/layers/ControlLayerSettings.tsx", - "source:src/workbench/widgets/layers/ControlLayerWarningIcon.tsx", + "source:src/workbench/widgets/layers/ControlLayerStatusIcon.tsx", + "source:src/workbench/widgets/layers/ControlModelRow.tsx", + "source:src/workbench/widgets/layers/ControlSourceRow.tsx", + "source:src/workbench/widgets/layers/ControlTypeRow.tsx", "source:src/workbench/widgets/layers/InpaintMaskSettings.tsx", "source:src/workbench/widgets/layers/LayerActiveDot.tsx", "source:src/workbench/widgets/layers/LayerBlendRow.tsx", @@ -3017,6 +3042,7 @@ "source:src/workbench/widgets/layers/LayersTree.tsx", "source:src/workbench/widgets/layers/LayersWidgetView.tsx", "source:src/workbench/widgets/layers/MaskModifierSettings.tsx", + "source:src/workbench/widgets/layers/OptionNameSeparator.tsx", "source:src/workbench/widgets/layers/RasterLayerFilterSection.tsx", "source:src/workbench/widgets/layers/ReferenceImageSettings.tsx", "source:src/workbench/widgets/layers/RegionalGuidanceSettings.tsx", @@ -3024,6 +3050,7 @@ "source:src/workbench/widgets/layers/addLayerMenu.ts", "source:src/workbench/widgets/layers/colorLabels.ts", "source:src/workbench/widgets/layers/controlModelOptions.ts", + "source:src/workbench/widgets/layers/controlPanelModel.ts", "source:src/workbench/widgets/layers/curveEditorMath.ts", "source:src/workbench/widgets/layers/implementation.ts", "source:src/workbench/widgets/layers/layerActionSession.ts", @@ -3059,10 +3086,12 @@ "source:src/workbench/widgets/layers/panes/TransformPane.tsx", "source:src/workbench/widgets/layers/panes/editorPaneLayout.ts", "source:src/workbench/widgets/layers/psdExportNotice.ts", + "source:src/workbench/widgets/layers/revealLayerProperties.ts", "source:src/workbench/widgets/layers/runLayerWorkflow.ts", "source:src/workbench/widgets/layers/selectedModel.ts", "source:src/workbench/widgets/layers/useAddLayer.ts", "source:src/workbench/widgets/layers/useCanvasHistoryHotkeys.ts", + "source:src/workbench/widgets/layers/useControlSourceActions.ts", "source:src/workbench/widgets/layers/useLayerSelectionCommands.ts", "source:src/workbench/widgets/layers/useLayerStackActions.ts", "source:src/workbench/widgets/layers/useSelectedMainModel.ts", @@ -3833,7 +3862,9 @@ "source:src/workbench/canvas-engine/color.ts", "source:src/workbench/canvas-engine/contracts.ts", "source:src/workbench/canvas-engine/controllers/booleanMergeController.ts", + "source:src/workbench/canvas-engine/controllers/controlDerivationController.ts", "source:src/workbench/canvas-engine/controllers/controlPixelController.ts", + "source:src/workbench/canvas-engine/controllers/controlViewController.ts", "source:src/workbench/canvas-engine/controllers/copyLayerController.ts", "source:src/workbench/canvas-engine/controllers/cropLayerController.ts", "source:src/workbench/canvas-engine/controllers/editSteps.ts", @@ -3873,6 +3904,7 @@ "source:src/workbench/canvas-engine/document-model/semanticLeaf.ts", "source:src/workbench/canvas-engine/document-model/semanticNode.ts", "source:src/workbench/canvas-engine/document/bitmapStore.ts", + "source:src/workbench/canvas-engine/document/controlDerivation.ts", "source:src/workbench/canvas-engine/document/documentIndex.ts", "source:src/workbench/canvas-engine/document/documentMirror.ts", "source:src/workbench/canvas-engine/document/documentTree.ts", @@ -3978,6 +4010,7 @@ "source:src/workbench/canvas-engine/transform/transformMath.ts", "source:src/workbench/canvas-engine/types.ts", "source:src/workbench/canvas-engine/viewport.ts", + "source:src/workbench/canvas-operations/ControlDetectionCoordinator.ts", "source:src/workbench/canvas-operations/FilterOperationCoordinator.ts", "source:src/workbench/canvas-operations/SelectObjectCoordinator.ts", "source:src/workbench/canvas-operations/api.ts", @@ -3985,7 +4018,10 @@ "source:src/workbench/canvas-operations/backend/canvasImages.ts", "source:src/workbench/canvas-operations/canvasImportNotice.ts", "source:src/workbench/canvas-operations/compositeForGeneration.ts", - "source:src/workbench/canvas-operations/controlRecommendations.ts", + "source:src/workbench/canvas-operations/controlDetectionRunner.ts", + "source:src/workbench/canvas-operations/controlDetectionStatusStore.ts", + "source:src/workbench/canvas-operations/controlLayerIntents.ts", + "source:src/workbench/canvas-operations/controlSourceActions.ts", "source:src/workbench/canvas-operations/createCanvasEngine.ts", "source:src/workbench/canvas-operations/createCanvasFromImages.ts", "source:src/workbench/canvas-operations/createFromBbox.ts", @@ -4017,6 +4053,8 @@ "source:src/workbench/components/WorkbenchUnavailableScreen.tsx", "source:src/workbench/controlAdapters.ts", "source:src/workbench/controlLayerChecks.ts", + "source:src/workbench/controlModelResolution.ts", + "source:src/workbench/controlTypes.ts", "source:src/workbench/deferredResource.ts", "source:src/workbench/diagnostics/loggingPreferences.ts", "source:src/workbench/extensions/contributionSource.ts", @@ -4250,10 +4288,11 @@ "source:src/workbench/widgets/image-map/ImageMapDataRuntime.tsx", "source:src/workbench/widgets/image-map/manifest.ts", "source:src/workbench/widgets/image-map/settingsContribution.ts", - "source:src/workbench/widgets/layers/controlModelOptions.ts", "source:src/workbench/widgets/layers/layerOps.ts", + "source:src/workbench/widgets/layers/layerPropertiesRequestStore.ts", "source:src/workbench/widgets/layers/manifest.ts", "source:src/workbench/widgets/layers/panes/editorPaneLayout.ts", + "source:src/workbench/widgets/layers/revealLayerProperties.ts", "source:src/workbench/widgets/layers/selectedModel.ts", "source:src/workbench/widgets/manifests.ts", "source:src/workbench/widgets/notifications/NotificationsHeaderActions.tsx", diff --git a/invokeai/frontend/webv2/public/locales/en.json b/invokeai/frontend/webv2/public/locales/en.json index 668e35f542f..90314723d15 100644 --- a/invokeai/frontend/webv2/public/locales/en.json +++ b/invokeai/frontend/webv2/public/locales/en.json @@ -371,7 +371,11 @@ "preparing": "Preparing generation…", "unavailable": "Invoke unavailable: {{reason}}", "unrunnable": "route is not runnable", - "unableToQueue": "Unable to queue" + "unableToQueue": "Unable to queue", + "issueCount_one": "Invoke · {{count}} issue", + "issueCount_other": "Invoke · {{count}} issues", + "showIssue_one": "Invoke unavailable, {{count}} issue. Show it: {{reason}}", + "showIssue_other": "Invoke unavailable, {{count}} issues. Show the first: {{reason}}" }, "iterations": { "fixedModelOnly": "Model only runs once: the upscaler gives the same image every time.", @@ -862,14 +866,6 @@ "Default is 4.0. Very high values may over-saturate or distort the output." ] }, - "controlLayerWeight": { - "heading": "Control Layer Weight", - "paragraphs": [ - "Adjusts how strongly the layer influences the generation process.", - "• Higher Weight (.75-2): Creates a more significant impact on the final result.", - "• Lower Weight (below .75): Creates a smaller impact on the final result. The slider stops at 0; models that allow it accept typed weights down to -1." - ] - }, "creativity": { "heading": "Creativity", "paragraphs": [ @@ -3471,7 +3467,8 @@ "staleDocument": "The canvas changed while the images were being prepared. Try again.", "staleProject": "The destination project was closed before the images were ready.", "success_one": "Added {{count}} image as canvas layers", - "success_other": "Added {{count}} images as canvas layers" + "success_other": "Added {{count}} images as canvas layers", + "controlFilled": "Filled {{name}}" }, "nextStagedCandidate": "Next staged candidate", "previousStagedCandidate": "Previous staged candidate", @@ -3642,6 +3639,24 @@ "text": "Text", "transform": "Transform", "view": "View" + }, + "controlHint": { + "title": "Drop an image here", + "body": "or pick one in Properties", + "uploading": "Uploading…" + }, + "fileDrop": { + "oneFile": "Drop one image at a time.", + "notImage": "Only image files can be dropped on the canvas.", + "noTarget": "Select an empty control layer to drop an image on it." + }, + "controlView": { + "label": "Control layer view", + "original": "Original", + "map": "Map", + "overlay": "Overlay", + "failed": "Couldn't show the original image", + "overBudget": "Not enough canvas memory to show the original image" } }, "workflow": { @@ -3861,7 +3876,6 @@ "busy": "The canvas is busy. Try again when the current action finishes.", "copyFailed": "The layer could not be copied.", "copyLayerToClipboard": "Copy layer to clipboard", - "copyToControl": "Copy to control layer", "copyToInpaintMask": "Copy to inpaint mask", "copyToRasterLayer": "Copy to raster layer", "copyToRegionalGuidance": "Copy to regional guidance", @@ -3955,7 +3969,8 @@ "showModifiers": "Show modifiers", "addDenoiseLimit": "Add denoise limit", "addNoise": "Add noise", - "addRegenerateRegion": "Add regenerate region" + "addRegenerateRegion": "Add regenerate region", + "useAsControlLayer": "Use as control layer" }, "control": { "apply": "Apply", @@ -4056,7 +4071,6 @@ "lllite_inpaint_adapter": "Inpainting ControlNet-LLLite adapters are hidden: they need an inpaint mask, not a control image." }, "invalidLayer": "Control layer \"{{name}}\": {{reason}}", - "kind": "Adapter type", "kinds": { "anima_lllite": "Anima ControlNet-LLLite", "control_lora": "Control LoRA", @@ -4066,24 +4080,15 @@ "t2i_adapter": "T2I Adapter", "z_image_control": "Z-Image ControlNet" }, - "mode": "Control mode", "model": "Model", - "modes": { - "balanced": "Balanced", - "more_control": "More control", - "more_prompt": "More prompt", - "unbalanced": "Unbalanced" - }, "needsAttention": "Control layer needs attention", "needsReidentify": "Needs re-identify", "preview": "Preview filter", - "selectModel": "Select a model", - "stepRange": "Step range", - "switchKind": "Switch to {{kind}}", - "transparencyEffect": "Transparent preview", "validation": { "capabilities_unavailable": "waiting for model capabilities to load.", + "choose_model": "Several installed models fit this control type; choose one.", "control_lora_limit": "Only one Control LoRA can be used at a time.", + "detection_pending": "Detection is running or out of date; wait for it, or run it again from the layer's Properties.", "duplicate_lllite_model": "Another control layer already uses this ControlNet-LLLite model; each can be applied once per generation.", "flux2_control_limit": "Only one FLUX.2 [dev] ControlNet can be used at a time.", "flux2_klein_control": "FLUX.2 [dev] ControlNet is not compatible with FLUX.2 Klein models.", @@ -4094,13 +4099,167 @@ "lllite_channels_unknown": "This ControlNet-LLLite model's adapter type is unknown. Re-identify it in the Model Manager.", "lllite_inpaint_adapter": "This is an inpainting ControlNet-LLLite adapter; a control layer needs a control adapter.", "missing_model": "Select a control model before generating.", + "missing_type": "Choose what this layer should control.", + "models_unavailable": "Waiting for the model list to load.", + "no_control_models": "No control models are installed for the selected main model.", + "no_model_for_type": "No installed control model handles this type for the selected main model.", "qwen_image_control_limit": "Only one Qwen Image ControlNet can be used at a time.", "qwen_image_edit_control": "Qwen Image ControlNet is not compatible with Qwen Image Edit models.", - "switch_adapter_kind": "Switch the adapter type to {{kind}} to use this layer with the selected model.", + "switch_adapter_kind": "Choose a control type so a model for the selected main model can be picked.", "unsupported_adapter": "This control adapter is not supported by the selected base model.", "z_image_control_limit": "Only one Z-Image control can be used at a time." }, - "weight": "Weight" + "weight": "Weight", + "detecting": "Running {{detector}}…", + "detectionQueued": "Waiting to run {{detector}}…", + "detectionErrors": { + "process-failed": "the detector could not run", + "apply-failed": "the map could not be applied", + "unknown-detector": "this detector is not available", + "output-dimensions": "the map came back at the wrong size", + "source-not-ready": "the image was still loading", + "over-budget": "there is not enough canvas memory", + "commit-busy": "the canvas stayed busy", + "unrepresentable": "the result could not be stored" + }, + "detectionFailed": "{{detector}} failed: {{reason}}", + "detectionNeedsImage": "Add an image to detect from", + "detectionChosenOnImage": "Chosen when an image is added", + "detectionAddImageFirst": "Add an image first", + "detectionNoInput": "Replace the image to detect again", + "detectionNoSettings": "No settings for {{detector}}", + "detectionNone": { + "pose": "None: image is already a pose map", + "depth": "None: image is already a depth map", + "edges": "None: image is already an edge map", + "softedge": "None: image is already a soft edge map", + "lineart": "None: image is already line art", + "sketch": "None: image is already a sketch", + "color": "None: image is already a color map", + "tile": "None: Tile uses the image directly" + }, + "detectionRetry": "Run again", + "detectionRun": "Detect again", + "detectionSetByType": "Set by type", + "detectionSettings": "Detector settings", + "detectionStale": "The map is out of date.", + "families": { + "openpose": "pose", + "face": "face", + "depth": "depth", + "normal": "normal map", + "canny": "Canny edge", + "mlsd": "MLSD line", + "softedge": "soft edge", + "lineart": "line art", + "lineart_anime": "anime line art", + "scribble": "scribble", + "color": "color map", + "shuffle": "content shuffle", + "tile": "tile" + }, + "modelAction": { + "reset": "Reset", + "findOne": "Find one", + "installStarters": "Install starter models" + }, + "modelCaption": { + "only": "{{kind}} · your only {{type}} model", + "union": "{{kind}} · via Union, which covers {{type}}", + "pick": "{{kind}} · your pick for {{type}}", + "remembered": "{{kind}} · remembered for {{type}}", + "ambiguous_other": "{{count}} models fit: {{models}}", + "noneForBase": "No installed model handles {{type}} for {{base}}.", + "noModels": "No control models are installed for {{base}}.", + "noModelsAnyBase": "No control models are installed.", + "mismatch": "{{kind}} · made for {{madeFor}}, not {{family}}", + "unknownPurpose": "{{kind}} · purpose not set; it may not follow {{type}}", + "unionUncovered": "{{kind}} · Union; it may not cover {{type}}", + "chooseType": "Choose a type and the model follows." + }, + "modelGroups": { + "purpose": "Made for {{type}}", + "union": "Union", + "unknown": "Purpose not set", + "installed": "Installed models" + }, + "modelItemCaption": "{{kind}} · {{origin}}", + "modelOrigin": { + "starter": "starter", + "downloaded": "downloaded" + }, + "modelRememberedHint": "Reset also forgets the {{type}} model remembered for {{base}}, in every project.", + "modelValue": { + "choose": "Choose a {{type}} model", + "noneForBase": "None for {{base}}", + "noModels": "No control models", + "setByType": "Set by type" + }, + "priorityModes": { + "more_prompt": "Prompt", + "balanced": "Balanced", + "more_control": "Control", + "unbalanced": "Unbalanced" + }, + "priorityUnavailable": "Priority applies to ControlNet models only.", + "priorityUnavailableFlux": "FLUX ControlNets have no priority setting.", + "rows": { + "image": "Image", + "type": "Type", + "detection": "Detection", + "steps": "Steps", + "priority": "Priority" + }, + "source": { + "upload": "Upload", + "frame": "Frame", + "draw": "Draw", + "replace": "Replace", + "uploadHint": "Upload an image from your computer", + "frameHint": "Use what's visible inside the frame", + "drawHint": "Paint the control image yourself", + "fromFrame": "From frame", + "drawn": "Drawn", + "imageSize": "Image · {{width}}×{{height}}", + "frameEmpty": "Nothing is visible inside the frame", + "frameNotReady": "The canvas is still preparing image data. Try again shortly.", + "frameFailed": "Couldn't use the frame", + "uploadFailed": "Couldn't use that image", + "uploadPending": "An image is still uploading to this layer." + }, + "stepsReadout": "{{begin}}–{{end}}%", + "summary": { + "noImage": "No image", + "noType": "Choose a type" + }, + "typeHint": { + "models_one": "{{count}} model", + "models_other": "{{count}} models", + "union": "via Union", + "notInstalled": "Not installed" + }, + "typeMissing": "Choose what this image should control.", + "typeNouns": { + "pose": "pose", + "depth": "depth", + "edges": "edge", + "softedge": "soft edge", + "lineart": "line art", + "sketch": "sketch", + "color": "color", + "tile": "tile" + }, + "typePlaceholder": "Choose a type", + "types": { + "pose": "Pose", + "depth": "Depth", + "edges": "Edges", + "softedge": "Soft edges", + "lineart": "Line art", + "sketch": "Sketch", + "color": "Color", + "tile": "Tile" + } }, "rasterFilter": { "aborted": "The filter was cancelled before it was applied.", @@ -4750,7 +4909,8 @@ "otherFactors": "Other factors", "metadataFailed": "Could not read the metadata of {{name}}. Nothing was queued.", "loadFailed": "Couldn't start the upscale. Reload the page and try again." - } + }, + "useAsControl": "Use as Control" }, "itemsAriaLabel": "Gallery items", "contentAriaLabel": "Gallery content", diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs index 61a76b490c0..8d10eca82c9 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.mjs @@ -594,6 +594,18 @@ const seedModel = ({ key, name, ...fields }) => ({ ...fields, }); +/** A control adapter seed keyed and named by its install name; a ControlNet unless `fields` says otherwise. */ +const controlSeed = (name, base, preprocessor, fields = {}) => ({ + record: seedModel({ + base, + default_settings: preprocessor === null ? null : { preprocessor }, + key: `seed-${name}`, + name, + type: 'controlnet', + ...fields, + }), +}); + /** * Opt-in model sets for browser checks, added by `POST /__models` so the profiles' fixture counts stay fixed. Each * entry is the record as installed plus `probe`: the fields `POST /api/v2/models/i/{key}/reidentify` fills in, as @@ -652,6 +664,36 @@ export const MODEL_SEED_SETS = Object.freeze({ }), }, ], + // SDXL control models for the control-layer panel: two depth models naming their preprocessor in the legacy and + // the node form, a pose model, a Union model, one whose purpose nothing names, and a T2I-Adapter. + 'control-sdxl': [ + controlSeed('controlnet-depth-sdxl-1.0', 'sdxl', 'depth_anything_image_processor'), + controlSeed('controlnet-zoe-depth-sdxl', 'sdxl', 'depth_anything_depth_estimation'), + controlSeed('thibaud-openpose-sdxl', 'sdxl', 'dw_openpose_image_processor'), + controlSeed('xinsir-controlnet-union-sdxl-promax', 'sdxl', null), + controlSeed('my-sdxl-control', 'sdxl', null), + controlSeed('t2i-adapter-canny-sdxl-1.0', 'sdxl', 'canny_image_processor', { + format: 'diffusers', + type: 't2i_adapter', + }), + ], + // A FLUX dev main with a depth Control LoRA and a Union ControlNet. + 'control-flux': [ + { + record: seedModel({ + base: 'flux', + key: 'seed-flux-dev', + name: 'FLUX.1 dev', + type: 'main', + variant: 'dev', + }), + }, + controlSeed('flux-depth-control-lora', 'flux', 'depth_anything_image_processor', { + format: 'lycoris', + type: 'control_lora', + }), + controlSeed('flux-controlnet-union-pro', 'flux', null), + ], }); const createNodePacks = (nodeCount) => diff --git a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs index 8f2f79ff058..19288f5940b 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend-fixtures.test.mjs @@ -1442,6 +1442,56 @@ test('model seed sets are opt-in, and re-identify fills what the probe would rec }); }); +test('control model seed sets cover both preprocessor id forms, and a PATCH changes their default settings', async () => { + const records = [...MODEL_SEED_SETS['control-sdxl'], ...MODEL_SEED_SETS['control-flux']].map(({ record }) => record); + const allKeys = Object.values(MODEL_SEED_SETS).flatMap((entries) => entries.map(({ record }) => record.key)); + assert.equal(new Set(allKeys).size, allKeys.length); + const preprocessors = new Set(records.map((record) => record.default_settings?.preprocessor)); + // Installed models name their preprocessor by the legacy processor id or by the detection node's id. + assert.ok(preprocessors.has('depth_anything_image_processor')); + assert.ok(preprocessors.has('depth_anything_depth_estimation')); + assert.deepEqual( + MODEL_SEED_SETS['control-flux'].map(({ record }) => [record.type, record.base]), + [ + ['main', 'flux'], + ['control_lora', 'flux'], + ['controlnet', 'flux'], + ] + ); + + await withRepresentativeBackend(async (backend) => { + const listed = async () => (await getJson(backend, '/api/v2/models/')).models; + const before = (await listed()).length; + const sendJson = (path, init, body) => + fetch(`${backend.origin}${path}`, { + ...init, + body: JSON.stringify(body), + headers: { 'Content-Type': 'application/json' }, + }); + const post = (path, body) => sendJson(path, { method: 'POST' }, body); + const patch = (path, body) => sendJson(path, { method: 'PATCH' }, body); + for (const set of ['control-sdxl', 'control-flux']) { + assert.equal((await post('/__models', { set })).status, 200); + } + const byKey = new Map((await listed()).map((model) => [model.key, model])); + assert.equal(byKey.size, before + records.length); + assert.equal(byKey.get('seed-t2i-adapter-canny-sdxl-1.0').type, 't2i_adapter'); + assert.equal(byKey.get('seed-my-sdxl-control').default_settings, null); + + const patched = await patch('/api/v2/models/i/seed-my-sdxl-control', { + default_settings: { preprocessor: 'canny_image_processor' }, + }); + assert.equal(patched.status, 200); + assert.deepEqual((await getJson(backend, '/api/v2/models/i/seed-my-sdxl-control')).default_settings, { + preprocessor: 'canny_image_processor', + }); + assert.equal((await patch('/api/v2/models/i/missing', { name: 'x' })).status, 404); + + await getJson(backend, '/__reset', { method: 'POST' }); + assert.equal((await listed()).length, before); + }); +}); + test('runtime config PATCH keeps the other writable field and validates the queue history limit', async () => { await withRepresentativeBackend(async (backend) => { await getJson(backend, '/api/v1/app/runtime_config', { diff --git a/invokeai/frontend/webv2/scripts/mock-backend.mjs b/invokeai/frontend/webv2/scripts/mock-backend.mjs index df7a6fb6082..39f64511940 100644 --- a/invokeai/frontend/webv2/scripts/mock-backend.mjs +++ b/invokeai/frontend/webv2/scripts/mock-backend.mjs @@ -1940,18 +1940,15 @@ export const startMockBackend = async (port, { profile = 'empty' } = {}) => { if (modelMatch) { const key = decodeURIComponent(modelMatch[1]); const model = state.models.get(key); - if (!model) { return json(404, { detail: 'Model not found' }); } if (method === 'PATCH') { - // Mirrors ModelRecordService.update_model: each submitted field replaces the stored one wholesale. - const changes = await readJsonBody(request); - const updated = { ...model, ...changes }; + // Mirrors ModelRecordService.update_model: submitted fields replace stored ones wholesale; the key stays. + const updated = { ...model, ...(await readJsonBody(request)), key }; state.models.set(key, updated); - return json(200, updated); + return json(200, clone(updated)); } - return json(200, model); } diff --git a/invokeai/frontend/webv2/scripts/run-accessibility-journeys.mjs b/invokeai/frontend/webv2/scripts/run-accessibility-journeys.mjs index f7deb627f0f..55f8f54beb6 100644 --- a/invokeai/frontend/webv2/scripts/run-accessibility-journeys.mjs +++ b/invokeai/frontend/webv2/scripts/run-accessibility-journeys.mjs @@ -231,6 +231,46 @@ const surfaces = [ await page.getByRole('tabpanel').getByRole('group', { exact: true, name: 'Layer' }).waitFor(); }, }, + { + id: 'workbench-canvas-control-layer', + path: representativeProjectPath, + ready: async (page) => { + // Seed after the reset in openRepresentativePage, then reload so the model list includes the seed. + const seeded = await fetch(`${backendOrigin}/__models`, { + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ set: 'control-sdxl' }), + }); + assert.ok(seeded.ok, `Could not seed control models: ${String(seeded.status)}.`); + await page.reload({ waitUntil: 'domcontentloaded' }); + await waitForWorkbench(page); + await selectLayoutPreset(page, 'Edit', 'Canvas'); + // The representative main model is SD 1.x; the seeded control models are SDXL, so switch to an SDXL main + // (Fixture Model 006) through the Generate panel's model picker. + await page.getByText('Fixture Model 001', { exact: true }).click(); + const search = page.getByRole('combobox', { name: /^Search / }); + await search.fill('Fixture Model 006'); + await page.getByRole('option', { name: /Fixture Model 006(?!\d)/ }).click(); + // Once the picker has closed, the trigger is the only place naming the new model. + await search.waitFor({ state: 'hidden' }); + await page.getByText('Fixture Model 006', { exact: true }).waitFor(); + await page.getByRole('button', { exact: true, name: 'Add layer' }).click(); + await page.getByRole('menuitem', { exact: true, name: 'Control layer' }).click(); + // The new layer's Properties panel and the canvas hint over its empty source are both scanned. + const type = page.getByRole('combobox', { exact: true, name: 'Type' }); + await type.waitFor(); + await page.getByRole('note').filter({ hasText: 'Drop an image here' }).waitFor(); + await waitForSettledDocument(page); + await assertNoAxeViolations(page, 'workbench-canvas-control-layer-untyped'); + // Typed as Depth: the seed holds two depth models, so the panel asks for one; the runner scans this state. + await type.click(); + await page.getByRole('option', { name: /^Depth/ }).click(); + await page + .getByRole('combobox', { exact: true, name: 'Model' }) + .filter({ hasText: 'Choose a depth model' }) + .waitFor(); + }, + }, { id: 'workbench-gallery-representative', path: representativeProjectPath, diff --git a/invokeai/frontend/webv2/src/architecture/workbenchOwnershipManifest.json b/invokeai/frontend/webv2/src/architecture/workbenchOwnershipManifest.json index f726a7f0831..738d4ee234a 100644 --- a/invokeai/frontend/webv2/src/architecture/workbenchOwnershipManifest.json +++ b/invokeai/frontend/webv2/src/architecture/workbenchOwnershipManifest.json @@ -97,6 +97,8 @@ "canvasStagingView.ts": "workbench", "controlAdapters.ts": "workbench", "controlLayerChecks.ts": "workbench", + "controlModelResolution.ts": "workbench", + "controlTypes.ts": "workbench", "deferredResource.ts": "workbench", "floatingWindows.ts": "workbench", "focusRegions.testing.ts": "workbench", diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.test.ts index 7312b1a8147..feba2b53bc8 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.test.ts @@ -198,6 +198,30 @@ describe('addControlLayers — controlnet on flux', () => { }); }); +describe('addControlLayers — FLUX InstantX Union mode', () => { + it('sets instantx_control_mode on a FLUX controlnet layer that carries one, mode 0 included', () => { + const graph = run({ + base: 'flux', + layers: [ + layer({ id: 'depth', instantxControlMode: 2, model: model('flux') }), + layer({ id: 'canny', instantxControlMode: 0, model: model('flux') }), + layer({ id: 'dedicated', instantxControlMode: null, model: model('flux') }), + ], + }); + + expect(graph.nodes['control_net_depth']?.instantx_control_mode).toBe(2); + expect(graph.nodes['control_net_canny']?.instantx_control_mode).toBe(0); + expect(graph.nodes['control_net_dedicated']).not.toHaveProperty('instantx_control_mode'); + }); + + it('never sets it outside FLUX, even when a layer carries one', () => { + const graph = run({ base: 'sdxl', layers: [layer({ instantxControlMode: 2, model: model('sdxl') })] }); + + expect(graph.nodes['control_net_layer1']?.type).toBe('controlnet'); + expect(graph.nodes['control_net_layer1']).not.toHaveProperty('instantx_control_mode'); + }); +}); + describe('addControlLayers — t2i_adapter on sdxl', () => { it('creates a t2i_adapter node with exact fields and wires collector to denoise.t2i_adapter', () => { const m = model('sdxl', 't2i_adapter'); diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.ts index fe080f9c8ac..a8f36198bf6 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/addControlLayers.ts @@ -40,6 +40,8 @@ export interface ControlLayerGraphInput { controlMode: 'balanced' | 'more_prompt' | 'more_control' | 'unbalanced' | null; /** The resolved model's `cond_in_channels`; required to validate an Anima ControlNet-LLLite layer. */ modelCondInChannels?: number | null; + /** FLUX InstantX Union only: the mode for the layer's control type; other models and bases ignore it. */ + instantxControlMode?: number | null; } /** Kind support is separate from control-mode/model compatibility. */ @@ -195,8 +197,12 @@ export const addControlLayers = (graph: BackendGraphContract, options: AddContro image: { image_name: layer.imageName }, resize_mode: 'just_resize', type: controlNetNodeType(base), - // FLUX ControlNet has no control_mode; SD-family carries it. - ...(base === 'flux' ? {} : { control_mode: layer.controlMode ?? 'balanced' }), + // FLUX ControlNet has no control_mode; SD-family carries it. A FLUX union model takes its InstantX mode. + ...(base === 'flux' + ? typeof layer.instantxControlMode === 'number' + ? { instantx_control_mode: layer.instantxControlMode } + : {} + : { control_mode: layer.controlMode ?? 'balanced' }), }); addEdge(graph, node, 'control', ensureControlNetCollector(), 'item'); } else if (layer.kind === 't2i_adapter') { diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts index 0e193d394e3..5f76c47e13d 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/compileCanvasGraph.test.ts @@ -1173,6 +1173,27 @@ describe('compileCanvasGraph — control layers (integration)', () => { weight: 0.65, }; + it('carries the InstantX mode of a FLUX union layer onto its flux_controlnet node, and none onto other bases', () => { + const unionLayer: ControlLayerGraphInput = { + ...controlNetLayer, + controlMode: null, + instantxControlMode: 4, + model: { base: 'flux', key: 'flux-union', name: 'FLUX ControlNet Union Pro', type: 'controlnet' }, + }; + const flux = compile(fluxModel, 'txt2img', { controlLayers: [unionLayer] }).backendGraph; + const sd1 = compile(sd1Model, 'img2img', { + controlLayers: [{ ...controlNetLayer, instantxControlMode: 4 }], + }).backendGraph; + + expect(flux.nodes['control_net_control-layer-1']).toMatchObject({ + instantx_control_mode: 4, + type: 'flux_controlnet', + }); + expect(flux.nodes['control_net_control-layer-1']).not.toHaveProperty('control_mode'); + expect(sd1.nodes['control_net_control-layer-1']?.type).toBe('controlnet'); + expect(sd1.nodes['control_net_control-layer-1']).not.toHaveProperty('instantx_control_mode'); + }); + it('grafts an enabled control layer through a real compiled canvas graph: control node args, collector → denoise.control wiring, base graph intact', () => { const { backendGraph, graph, mode } = compile(sd1Model, 'img2img', { controlLayers: [controlNetLayer] }); diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.test.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.test.ts index 3ca83ae2300..f116cb53eec 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.test.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.test.ts @@ -282,6 +282,31 @@ describe('Anima ControlNet-LLLite validation', () => { }); }); +describe('unresolved control models', () => { + it('reports why resolution found no model in place of missing_model, after kind and value checks', () => { + expect(getControlValidationReason({ ...valid, adapterModel: null, unresolvedModel: 'choose_model' })).toBe( + 'choose_model' + ); + expect(getControlValidationReason({ ...valid, adapterModel: null, unresolvedModel: null })).toBe('missing_model'); + expect( + getControlValidationReason({ ...valid, adapterModel: null, unresolvedModel: 'no_model_for_type', weight: 5 }) + ).toBe('invalid_adapter_values'); + expect(getControlValidationReason({ ...valid, unresolvedModel: 'no_control_models' })).toBeNull(); + }); + + it('carries the reason through the sequence without claiming a slot', () => { + const validate = createControlValidationSequence({ base: 'flux' }); + const controlLora = { + adapterModel: { base: 'flux', key: 'lora', type: 'control_lora' }, + beginEndStepPct: [0, 1] as [number, number], + kind: 'control_lora' as const, + weight: 1, + }; + expect(validate({ ...controlLora, adapterModel: null, unresolvedModel: 'missing_type' })).toBe('missing_type'); + expect(validate(controlLora)).toBeNull(); + }); +}); + describe('createControlValidationSequence', () => { const anima = (key: string, cond_in_channels: number | null = 3) => ({ adapterModel: { base: 'anima', cond_in_channels, key, type: 'controlnet' }, diff --git a/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.ts b/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.ts index 677036fa7dc..40d84e8297e 100644 --- a/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.ts +++ b/invokeai/frontend/webv2/src/features/generation/core/canvas/controlValidation.ts @@ -19,6 +19,11 @@ export type ControlAdapterKind = (typeof CONTROL_ADAPTER_KINDS)[number]; export const CONTROL_VALIDATION_REASONS = [ 'capabilities_unavailable', 'missing_model', + 'missing_type', + 'choose_model', + 'no_model_for_type', + 'no_control_models', + 'models_unavailable', 'switch_adapter_kind', 'unsupported_adapter', 'incompatible_base', @@ -118,6 +123,8 @@ export const getControlValidationReason = (params: { flux2ControlIndex?: number; /** An earlier contributing Anima LLLite layer already applies this model; the denoiser applies each once. */ lliteModelInUse?: boolean; + /** Why model resolution found no model, reported in place of `missing_model`. */ + unresolvedModel?: ControlValidationReason | null; }): ControlValidationReason | null => { const { adapterModel, @@ -129,6 +136,7 @@ export const getControlValidationReason = (params: { mainBase, mainVariant, qwenImageControlIndex = 0, + unresolvedModel, weight, zImageControlIndex = 0, } = params; @@ -147,7 +155,7 @@ export const getControlValidationReason = (params: { return 'invalid_adapter_values'; } if (!adapterModel) { - return 'missing_model'; + return unresolvedModel ?? 'missing_model'; } // Unavailable capability data is distinct from unsupported adapters. if (!capabilitiesLoaded) { @@ -203,6 +211,7 @@ export const createControlValidationSequence = (main: { base: string; variant?: beginEndStepPct: [number, number]; kind: ControlAdapterKind; weight: number; + unresolvedModel?: ControlValidationReason | null; }): ControlValidationReason | null => { const { adapterModel, kind } = layer; const reason = getControlValidationReason({ @@ -215,6 +224,7 @@ export const createControlValidationSequence = (main: { base: string; variant?: mainBase: main.base, mainVariant: main.variant ?? undefined, qwenImageControlIndex: kind === 'qwen_image_control' ? qwenImageControlCount : 0, + unresolvedModel: layer.unresolvedModel, weight: layer.weight, zImageControlIndex: kind === 'z_image_control' ? zImageControlCount : 0, }); diff --git a/invokeai/frontend/webv2/src/features/generation/graph.ts b/invokeai/frontend/webv2/src/features/generation/graph.ts index 5aac879f7f3..4858cfb9bbf 100644 --- a/invokeai/frontend/webv2/src/features/generation/graph.ts +++ b/invokeai/frontend/webv2/src/features/generation/graph.ts @@ -26,7 +26,6 @@ export { getControlAdapterMinWeight, getControlModelUnusableReason, getControlValidationReason, - getSuggestedControlKind, isControlKindSupportedForBase, isControlModelUsableForKind, } from './core/canvas/controlValidation'; diff --git a/invokeai/frontend/webv2/src/platform/ui/Select.browser.test.tsx b/invokeai/frontend/webv2/src/platform/ui/Select.browser.test.tsx index 4ba67da02f7..0d895204cea 100644 --- a/invokeai/frontend/webv2/src/platform/ui/Select.browser.test.tsx +++ b/invokeai/frontend/webv2/src/platform/ui/Select.browser.test.tsx @@ -2,7 +2,8 @@ import { ChakraProvider, createListCollection } from '@chakra-ui/react'; import { system } from '@theme/system'; import { act } from 'react'; import { createRoot, type Root } from 'react-dom/client'; -import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; +import { userEvent } from 'vitest/browser'; import { Select } from './Select'; @@ -11,6 +12,14 @@ const RATIO_ITEMS = [ { label: '1:1', value: '1:1' }, ]; const RATIO_VALUE = ['16:9']; +const DETECTOR_ITEMS = [ + { group: 'detectors', label: 'Canny', value: 'canny' }, + { group: 'detectors', label: 'MLSD', value: 'mlsd' }, + { group: 'none', label: 'None: image is already an edge map', value: 'none' }, +]; +const DETECTOR_VALUE = ['canny']; +const getDetectorGroup = (item: { group: string }) => item.group; +const renderDetectorGroupLabel = (group: string) => (group === 'detectors' ? 'Detectors' : null); (globalThis as typeof globalThis & { IS_REACT_ACT_ENVIRONMENT: boolean }).IS_REACT_ACT_ENVIRONMENT = true; @@ -80,4 +89,62 @@ describe('Select', () => { await act(() => trigger().click()); await expect.poll(() => document.querySelectorAll('[role="option"]').length).toBe(1); }); + + it('puts a header-less group under a divider and moves the keyboard across it', async () => { + const onValueChange = vi.fn(); + await act(() => { + root.render( + + + + ); + }); + await act(() => trigger().click()); + await expect.poll(() => document.querySelectorAll('[role="option"]').length).toBe(3); + + const listbox = document.querySelector('[role="listbox"]')!; + expect(listbox.querySelectorAll('[role="separator"]')).toHaveLength(0); + expect(listbox.querySelector('[role="option"]')?.textContent).toContain('None'); + }); }); diff --git a/invokeai/frontend/webv2/src/platform/ui/Select.tsx b/invokeai/frontend/webv2/src/platform/ui/Select.tsx index 4e77fc62611..a41ed05c9bf 100644 --- a/invokeai/frontend/webv2/src/platform/ui/Select.tsx +++ b/invokeai/frontend/webv2/src/platform/ui/Select.tsx @@ -9,8 +9,8 @@ import type { } from '@chakra-ui/react'; import type { Key, ReactNode } from 'react'; -import { Portal, Select as ChakraSelect, useFieldContext } from '@chakra-ui/react'; -import { useCallback, useRef } from 'react'; +import { Portal, Select as ChakraSelect, Separator, useFieldContext } from '@chakra-ui/react'; +import { Fragment, useCallback, useRef } from 'react'; import { Scrollable } from './Scrollable'; @@ -56,7 +56,10 @@ export interface SelectProps extends Omit string; - /** The visible header for a group key; defaults to the key itself. */ + /** + * The visible header for a group key; defaults to the key itself. `null` renders the group without a header, + * under a divider unless it opens the list. + */ renderGroupLabel?: (group: string) => ReactNode; getItemKey?: (item: T, index: number) => Key; indicatorGroupProps?: SelectIndicatorGroupProps; @@ -122,19 +125,28 @@ export const Select = ({ {groupBy - ? partitionByGroup(collection.items, groupBy).map(({ group, items, startIndex }) => ( - - - {renderGroupLabel ? renderGroupLabel(group) : group} - - {items.map((item, offset) => ( - - {renderItem(item)} - {itemIndicator ? : null} - - ))} - - )) + ? partitionByGroup(collection.items, groupBy).map(({ group, items, startIndex }) => { + const label = renderGroupLabel ? renderGroupLabel(group) : group; + const groupItems = items.map((item, offset) => ( + + {renderItem(item)} + {itemIndicator ? : null} + + )); + // A header-less group is not a labelled listbox group: its options follow a divider, which + // stays out of the accessibility tree because a listbox owns only options and groups. + return label === null ? ( + + {startIndex > 0 ? : null} + {groupItems} + + ) : ( + + {label} + {groupItems} + + ); + }) : collection.items.map((item, index) => ( {renderItem(item)} diff --git a/invokeai/frontend/webv2/src/platform/ui/hints/hintRegistry.ts b/invokeai/frontend/webv2/src/platform/ui/hints/hintRegistry.ts index 8df0fe0660c..f3679ac2042 100644 --- a/invokeai/frontend/webv2/src/platform/ui/hints/hintRegistry.ts +++ b/invokeai/frontend/webv2/src/platform/ui/hints/hintRegistry.ts @@ -25,7 +25,6 @@ export const FEATURE_HINTS = { concepts: { href: SUPPORT_CONCEPTS }, conditioningRebalance: {}, conditioningRebalanceGain: {}, - controlLayerWeight: { href: 'https://support.invoke.ai/support/solutions/articles/151000178148' }, creativity: {}, denoisingStrength: { href: 'https://support.invoke.ai/support/solutions/articles/151000094998-image-to-image', diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/ARCHITECTURE.md b/invokeai/frontend/webv2/src/workbench/canvas-engine/ARCHITECTURE.md index 63f9e74342f..6b989017cbe 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/ARCHITECTURE.md +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/ARCHITECTURE.md @@ -67,7 +67,7 @@ Current extracted ownership includes: - `RasterMemoryBudgetController`: base, derived, group, decoded, detached, reserved and overage byte accounting plus generation-scoped reservations and operation-owned reservations and pins. - `RenderController`: render targets, scheduler construction, previews, render lifecycle. - `HistoryController`: engine-owned history, guarded async replay with failure reporting, inactive byte trimming. -- `PersistenceController`: dirty-bitmap flush barriers and disposal. +- `PersistenceController`: dirty-bitmap flush barriers (whole document, or one layer through `applicationHost.flushLayerUploads`) and disposal. - `EditingController`: selection state and exclusive edit-lease lifecycle. - `InteractionController`: public tool commands and their disposal boundary. - `LayerController`: guarded layer mutations and thumbnail capability ports. @@ -76,6 +76,8 @@ Current extracted ownership includes: - `StagedResultController`: guarded staged-candidate acceptance and its history entry. - `PixelEditController`, `SelectionImageController`, `LayerMutationController`: atomic pixel/document publication at their respective boundaries. Pixel editor materializes control layers and raster-image layers in place before destructive edits, keeping conversion plus pixels in one undoable transaction. - `MaskResultController`, `FilterResultController`, `GeneratedResultController`: guarded application-result adoption through narrow host ports. +- `ControlDerivationController`: control-layer intent patches and detected-map commits (see "Control derivation"). +- `ControlViewController`: the display-only Original / Overlay view of one control layer (see "Control layer view"). Controller-local tests instantiate these boundaries with fakes. Integration tests in `engine.test.ts` cover composition, document routing, cross-controller invariants. @@ -200,6 +202,47 @@ Continue-staging variant uses same guarded transaction but inserts candidate as Undo removes accepted layer and restores prior selection. Redo restores identical layer and selection without recreating event or staging candidate. History entry replays failure-atomically; successful commit publishes through an admitted transaction, which clears any previous redo stack. Undo intentionally does not return accepted candidate to staging. +## Control derivation + +A typed control layer can keep the pixels it was detected from (`original`, framed by `originalTransform`) beside the derived map in `source`. The invariants are documented on `CanvasControlLayerContract`; `document/controlDerivation.ts` owns them as pure rules, and every engine path that changes a control layer's pixels or frame applies one of them so no edit leaves a map paired with an original it no longer derives from: + +| Operation | Rule | +| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | +| Materialize before a pixel edit (`buildMaterializedPixelLayer`) | `rebaseControlOriginal` with the baked transform; a frame the bake would shear detaches | +| Transform Apply baking a paint map (`TransformEditingController`) | the same rebase in the bake's own step, so undo restores the whole before-contract | +| Crop to bbox | detach (the cropped pixels become the image) | +| Generated or Select Object result replacing a control layer | detach and drop `filter`: the result is the image, not a filter's map | +| Filter result on a control layer | refused `unsupported`: Filter runs on raster layers only; a control map comes from detection | +| Filter result copied from a raster layer to a new control layer | a fresh control layer; `controlCopy.controlType` types the copy with `detection: null` | +| Paint on a map that needs no materialize | only `source` changes; the map stays current | + +`ControlDerivationController` (`engine.controlDerivation`) is the only writer of derivation fields besides those rules. The application's detection coordinator calls `commitControlLayerPatch` for an intent (type, detector, settings, image) and `commitControlMap` for a detection result, guarded by `captureControlDetectionGuard`: same `documentRevision` and same `controlDetectionInputKey`, which ignores weight, name and paint on the map, so those never discard a running detection. Neither commit passes the operation edit owner. A detection runs in the background while the user keeps working, so its commit must yield to whatever holds the document: an operation's lease (Filter, Select Object, Transform), a gesture, a replay or a session holding the layer's pixels all refuse it `busy`, and the coordinator retries once the lock releases. The guard capture and the map commit's freshness checks (before decoding, after it, and once admitted) read the reducer document, so a background job never ends the user's structural preview; only an admitted commit does, and the map is built from the layer as it stands after that. A map whose input is the layer's own unpersisted paint is also `busy` until `flushLayerUploads` lands those strokes in the `original` it records. The commit promotes the map image through the caller's `makeDurable` only once its first check passes, just before decoding, and promotes it once across retries: promoted maps are exempt from server-side intermediate cleanup, so a result that is refused, superseded or dropped as stale before then stays collectible. A refusal after promotion (stale after decoding, over-budget, a history refusal, a busy timeout) still leaves that map durable and unreferenced; this is accepted because detector graphs cache their results, so a re-run can return a map the document or history still references and deleting a dropped one could remove a live map. A document change (undo, redo) that leaves a layer needing no detection, or changes its input other than by its own paint landing, cancels that layer's run and its backend queue item, except while that run's own map commit is in flight, whose publish reconciles synchronously. + +History pairing keeps one undo per user intent. A patch with a `coalesceKey` replaces the newest entry while that is the controller's patch of the same layer and key, so a settings scrub is one step. A map replaces the newest entry while that is the controller's entry for the same layer that requested this input key (a patch that changed the detection input, or an earlier map), so one undo reverts "set type" and its map together; otherwise the map records its own "Detect " entry, and a weight or name patch in between is never undone together with a later map. The controller's record of its newest entry, including any unpersisted before-pixels, is dropped when that entry is disposed (evicted, cleared, discarded from redo or replaced), so it never outlives the entry. Patches route as user edits; maps publish as `system` so a background result never changes the invocation route. Undo restores the exact before-state: a contract whose source is persisted re-rasterizes from it, and unpersisted paint the step replaced is captured and replayed with the entry. A source change discards the outgoing source's pending persistence. Undo and redo never schedule detection; a restored stale map shows as needing detection until the user asks again. + +### Control layer view + +`previews.setControlLayerView(layerId, 'original' | 'map' | 'overlay')` lets the user compare a control layer's map with the picture it was detected from. `ControlViewController` owns the one view that can exist. It refuses while an operation holds the document (`documentEditingLocked`) or a generation or staging locks the canvas (`setInteractionLocked`). Otherwise it reserves raster memory for the original's decoded pixels, rasterizes the `original` source with the layer rasterizers, and once the surface exists accounts it as detached pixels until the view ends; the reservation ends with the decode. `map` ends the view and resolves `shown`. + +| Outcome | Meaning | +| ------------- | -------------------------------------------------------------------------------------- | +| `shown` | the view (or the map) is displayed | +| `missing` | no control layer with a kept original that has pixels | +| `locked` | an operation or a generation holds the canvas | +| `stale` | a newer request, a reset below or an edit of the layer superseded the decode in flight | +| `over-budget` | the original's pixels do not fit the raster budget; nothing was decoded | +| `failed` | the original could not be fetched or decoded | + +The view records the contract it was built from and publishes `{ layerId, view }` as the `controlLayerView` interaction state. It returns to the map (`null`, surface released, layer invalidated), ending any decode in flight, on: + +- any change to the layer's contract (every edit, undo, redo and detection commit) and its removal; +- another selection; +- a stroke or other pixel edit admitted on the layer (strokes paint the map live), a direct repaint of it, and its cache invalidation (including `clearCaches`); +- the editing lock or the interaction lock engaging; +- document replacement, the project going away, cooldown and disposal. + +The compositor draws the original as layer-local pixels, like a floating selection: through the layer matrix and `originalTransform`, at the layer's opacity and blend, without the control-transparency effect. `original` draws it in place of the map, even when the layer has no map; `overlay` draws the map over it at `CONTROL_OVERLAY_MAP_ALPHA` of the layer's opacity. The original's landing joins the leaf's bounds, so culling and bounded damage cover it. Control layers never sit in a composited group scope (only raster-stack groups composite in isolation), so group surfaces never see a view. Isolation suppresses it, and only screen frames pass it: history, persistence, thumbnails, exports and generation composites read the map. + ## PSD export PSD export is a planner/executor transaction over immutable snapshots. `PsdExportController` captures one `CanvasDocumentSnapshot`, walks raster tree into planner inputs (groups as pass-through folders, exportable leaves with their captured content rects), then obtains a `CanvasRasterSnapshot` including disabled layers. `planPsdExport` reverses every level of top-first tree into ag-psd's bottom-to-top order, writes each node's own visibility flag (disabled group is a hidden folder), flattens merged preview from the leaves that contribute (enabled with every ancestor enabled), which is what Photoshop renders for the same flags. Folders whose subtree exports nothing are dropped. Execution reads only detached surfaces; cannot mix live layer caches from different document generations. diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/api.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/api.ts index 7a6d5a77e5d..47e0e4c5764 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/api.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/api.ts @@ -6,3 +6,12 @@ * the core dependency direction. */ export * from './capabilities'; +export { CONTROL_TYPES } from './contracts'; +export { + controlDetectionInputKey, + getControlDetectionInput, + isControlMapCurrent, + needsControlDetection, + restoreControlOriginal, +} from './document/controlDerivation'; +export { createControlLayer, type CreateControlLayerInput, nextControlLayerName } from './document/layerFactories'; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/applicationHost.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/applicationHost.ts index 72fecad528e..48c39f4cceb 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/applicationHost.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/applicationHost.ts @@ -15,6 +15,7 @@ import type { CommitMaskImageResult, CommitMaskImageResultOptions, } from '@workbench/canvas-engine/controllers/maskResultController'; +import type { LayerFlushOptions, LayerFlushResult } from '@workbench/canvas-engine/controllers/persistenceController'; import type { CanvasCompositeExecutorDeps } from '@workbench/canvas-engine/rasterTransactions'; import type { AdjustmentRefusal } from '@workbench/canvas-engine/render/exactAdjustments'; import type { RasterSurface } from '@workbench/canvas-engine/render/raster'; @@ -43,6 +44,8 @@ export interface CanvasApplicationHost { decodeSelectObjectPreview(result: { image: CanvasImageRef; rect: Rect }, signal: AbortSignal): Promise; encodeSurface(surface: RasterSurface): Promise; exportBakedLayerBlob(layerId: string): Promise; + /** Persists one layer's unsaved paint, so work reading its persisted source sees every stroke. */ + flushLayerUploads(layerId: string, options: LayerFlushOptions): Promise; exportLayerPixels( layerId: string, options?: ExportLayerPixelsOptions diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts index 97cc9ec71d4..b9e9de7e020 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/capabilities.ts @@ -1,4 +1,6 @@ import type { + CanvasControlDetectionContract, + CanvasControlLayerContract, CanvasDocumentContractV3, CanvasImageRef, CanvasLayerContract, @@ -6,6 +8,7 @@ import type { CanvasStagingCandidateContract, CanvasStateContractV3, CanvasTextFontRef, + ControlType, } from '@workbench/canvas-engine/contracts'; import type { CanvasMutationOrigin } from '@workbench/canvas-engine/mutationContracts'; import type { StrokeCommittedEvent } from '@workbench/canvas-engine/tools/tool'; @@ -29,6 +32,7 @@ import type { CanvasEditGate } from './editGate'; import type { TextStylePatch, ActiveColorPairState, + ControlLayerView, BboxToolOptions, BrushOptions, CheckerColors, @@ -95,9 +99,82 @@ export interface CommitRasterFilterOptions { mode: 'replace' | 'copy'; filter?: RasterFilterSettings; target?: RasterFilterCommitTarget; + /** + * For `target: 'control'` copies: the control type the filter implies, or null for an untyped copy. A typed copy + * uses its pixels as-is (`detection: null`); either way the copy keeps no derivation of its source layer. + */ + controlCopy?: { controlType: ControlType | null }; requireExactImageDimensions?: boolean; signal?: AbortSignal; } + +/** What a control-map commit must still match: the document session and the detection input it ran on. */ +export interface ControlDetectionGuard { + readonly projectId: string; + readonly layerId: string; + readonly documentRevision: number; + /** `controlDetectionInputKey` of the layer when the run started. */ + readonly inputKey: string; +} + +export type ControlLayerPatchResult = + | { status: 'committed' } + | { + status: 'stale' | 'missing' | 'locked' | 'busy' | 'not-ready' | 'over-budget' | 'failed'; + message?: string; + }; + +export type ControlMapCommitResult = + | { status: 'committed' } + | { + status: + | 'stale' + | 'missing' + | 'locked' + | 'busy' + | 'not-ready' + | 'over-budget' + | 'aborted' + | 'unrepresentable' + | 'failed'; + message?: string; + }; + +/** + * Edits that keep a control layer's derived map in step with its type, detector and original. Neither commit takes + * the operation edit lease: both refuse `busy` while a modal operation, gesture, replay or layer session is running, + * and the caller retries later. + */ +export interface CanvasControlDerivationCapability { + /** + * Replaces one control layer's contract as one user-origin history entry; refused `stale` unless the live layer + * still equals `expected`. Consecutive patches of the same layer with the same `coalesceKey` replace the top entry. + */ + commitControlLayerPatch(options: { + label: string; + expected: CanvasControlLayerContract; + next: CanvasControlLayerContract; + coalesceKey?: string; + }): ControlLayerPatchResult; + /** What a detection run must still match to commit; null when the layer is missing or not a control layer. */ + captureControlDetectionGuard(layerId: string): ControlDetectionGuard | null; + /** + * Installs a detected map (a persisted image) as the layer's `source`, keeping the original, as a system-origin + * entry. Paired with the intent entry that requested it when that is still the newest step. + */ + commitControlMap(options: { + guard: ControlDetectionGuard; + image: CanvasImageRef; + rect: Rect; + detection: CanvasControlDetectionContract; + /** + * Makes `image` durable. Awaited only once the guard holds and nothing holds the document, just before decoding, + * so a refused or stale result is never promoted; a rejection refuses the commit `failed`. + */ + makeDurable(): Promise; + signal?: AbortSignal; + }): Promise; +} export type MaskImageResultTarget = 'inpaint_mask' | 'regional_guidance'; export interface CommitMaskImageResultOptions { guard: LayerExportGuard; @@ -121,6 +198,8 @@ export interface CanvasInteractionState { clipToBbox: boolean; /** The mirrored workbench pair; the engine reads it at gesture start, never writes it. */ colorPair: ActiveColorPairState; + /** The control layer shown as its kept original or as original under its map; null shows every map. */ + controlLayerView: ControlLayerView | null; documentEditingLocked: boolean; eraserOptions: EraserOptions; gradientOptions: GradientToolOptions; @@ -638,6 +717,16 @@ export interface CanvasEngineExportCapability extends CanvasExportCapability { extractMaskedArea(maskLayerId: string): Promise; } +/** How a control layer displays: its detected map (the default), the kept original, or the original under the map. */ +export type ControlLayerViewMode = 'original' | 'map' | 'overlay'; + +/** + * `missing`: no control layer with a kept original; `locked`: an operation or a generation holds the canvas; + * `stale`: a newer request or an edit of the layer superseded it; `over-budget`: the original's pixels do not fit the + * raster budget; `failed`: the original could not be decoded. + */ +export type ControlLayerViewOutcome = 'shown' | 'missing' | 'locked' | 'stale' | 'over-budget' | 'failed'; + export interface CanvasEnginePreviewCapability extends CanvasPreviewCapability { /** * Sizes `target` and draws the whole document fitted to `maxSizePx`. Returns its drawn rect, or null without a @@ -651,6 +740,13 @@ export interface CanvasEnginePreviewCapability extends CanvasPreviewCapability { guard: LayerExportGuard ): Promise<'shown' | SubsetOf | SubsetOf>; setStagedPreview(input: StagedPreviewInput | null): void; + /** + * Display-only: shows a control layer as its kept original, the original under its map at half strength, or + * (`map`) as usual. One layer holds a view at a time; refused while the canvas is locked. Any edit or stroke on + * the layer, a selection change, a lock, a document replacement and the other resets in ARCHITECTURE.md return it + * to the map. Never recorded in history, persisted or exported. + */ + setControlLayerView(layerId: string, view: ControlLayerViewMode): Promise; } /** Public capability-only handle. Construction and mutable stores are intentionally absent. */ @@ -685,6 +781,7 @@ export type { BboxToolOptions, BrushOptions, CheckerColors, + ControlLayerView, EraserOptions, GradientStop, GradientToolOptions, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/contracts.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/contracts.ts index fcc3a86471d..9061599e913 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/contracts.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/contracts.ts @@ -170,6 +170,9 @@ export type CanvasLayerSourceContract = span?: number; }; +/** The pixel-backed sources: an uploaded image or a paint bitmap. */ +export type CanvasPixelSourceContract = Extract; + /** Photoshop's layer-color palette; also the PSD `layerColor` vocabulary. */ export const CANVAS_COLOR_LABELS = ['red', 'orange', 'yellow', 'green', 'blue', 'violet', 'gray'] as const; export type CanvasColorLabel = (typeof CANVAS_COLOR_LABELS)[number]; @@ -343,17 +346,46 @@ export interface CanvasRasterLayerContractV2 extends CanvasLayerBaseContract { filter?: { type: string; settings: Record }; } +/** What a control layer's image should control; a closed set, so an unknown value refuses the document. */ +export const CONTROL_TYPES = ['pose', 'depth', 'edges', 'softedge', 'lineart', 'sketch', 'color', 'tile'] as const; +export type ControlType = (typeof CONTROL_TYPES)[number]; + +/** A detector (filter node type) and its settings; the type string stays open so newer detectors survive a load. */ +export interface CanvasControlDetectionContract { + type: string; + settings: Record; +} + +/** + * Control layers may derive their map from a kept original. Invariants (`document/controlDerivation.ts`): + * I1 no `controlType` ⇒ no `detection`, `original` or `originalTransform`; + * I2 `detection === null` ⇒ no `original`: `source` is used as-is; + * I3 `original` present ⇒ `detection` is an object and `source` is the map derived from it; + * I4 the map is current when `detection` is null or absent, or an `original` is kept and `filter` structurally equals + * `detection`; + * I5 detection runs on `original ?? (filter ? null : source)`, so map-only layers cannot re-detect. + */ export interface CanvasControlLayerContract extends CanvasLayerBaseContract { type: 'control'; + /** The pixels generation uses: the derived map once detection ran. */ source: CanvasLayerSourceContract; adapter: CanvasControlAdapterContract; withTransparencyEffect: boolean; - filter?: { type: string; settings: Record }; + /** The filter that produced `source`'s pixels; absent when they are not a filter result. */ + filter?: CanvasControlDetectionContract; /** * Display-only visibility for overlay layers; generation is unchanged. Absence means visible. Raster layers * instead use `isEnabled` for both visibility and participation. */ isHidden?: boolean; + /** What the image controls. Absent means no type chosen (legacy or new); never written as null. */ + controlType?: ControlType; + /** The detector for `controlType`. `null` means "None: use the image as-is"; absent means legacy (no choice). */ + detection?: CanvasControlDetectionContract | null; + /** The pre-detection pixels, kept so detection can re-run; a persisted image reference like `source`. */ + original?: CanvasPixelSourceContract; + /** Maps `original`'s local frame into the layer-local frame of `source`; absent means identity. */ + originalTransform?: CanvasLayerBaseContract['transform']; } export interface CanvasRegionalGuidanceLayerContract extends CanvasLayerBaseContract { @@ -447,11 +479,12 @@ export const CANVAS_MAX_NODE_COUNT = 10_000; export interface CanvasDocumentContractV3 { /** * Schema this document needs: 3 for plain content, 4 for custom-font typography, 5 for noise or spatial - * (clarity, texture, dehaze) adjustments. Each document mutation raises it to the newer of its content's need and - * the canvas state's version. A restored snapshot's document keeps its own, possibly lower, version until the next - * mutation; the state's version and the project's compatibility floor stay put meanwhile. + * (clarity, texture, dehaze) adjustments, 6 for control-layer derivation fields. Each document mutation raises it to + * the newer of its content's need and the canvas state's version. A restored snapshot's document keeps its own, + * possibly lower, version until the next mutation; the state's version and the project's compatibility floor stay + * put meanwhile. */ - version: 3 | 4 | 5; + version: 3 | 4 | 5 | 6; width: number; height: number; background: 'transparent' | { color: string }; @@ -475,10 +508,10 @@ export interface CanvasStagingAreaContractV2 extends CanvasStagingAreaContract { export interface CanvasStateContractV3 { /** * Newest schema the state needs: the newest of its previous version, the live document's and every snapshot's - * (3 plain, 4 custom-font typography, 5 noise or spatial adjustments). Sticky: removing that content never lowers - * it. + * (3 plain, 4 custom-font typography, 5 noise or spatial adjustments, 6 control-layer derivation fields). Sticky: + * removing that content never lowers it. */ - version: 3 | 4 | 5; + version: 3 | 4 | 5 | 6; document: CanvasDocumentContractV3; /** * Monotonic wholesale-replacement counter. The mirror clears pixel history whenever it changes, including diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controlLayerView.engine.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controlLayerView.engine.test.ts new file mode 100644 index 00000000000..6a0821fa651 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controlLayerView.engine.test.ts @@ -0,0 +1,518 @@ +import type { CanvasProjectMutationPort } from '@workbench/canvasProjectMutationPort'; +import type { Project } from '@workbench/projectContracts'; + +import { createEmptyCanvasState } from '@workbench/canvasMigration'; +import { applyCanvasProjectMutation } from '@workbench/canvasProjectMutations'; +import { createInitialWorkbenchState } from '@workbench/workbenchState.testing'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import type { CanvasControlLayerContract, CanvasDocumentContractV3, CanvasPixelSourceContract } from './contracts'; +import type * as RasterControllerModule from './controllers/rasterController'; +import type { RasterController } from './controllers/rasterController'; +import type { BitmapStore } from './document/bitmapStore'; +import type { RasterCallLogEntry, StubRasterSurface } from './render/raster.testStub'; + +import { documentFrom, layerContract } from './document-model/documentFixtures.testStub'; +import { createCanvasEngine } from './engine'; +import { createTestStubRasterBackend } from './render/raster.testStub'; + +const owners = vi.hoisted(() => ({ raster: null as RasterController | null })); + +// Captures the engine's real raster owner to read its reservations; its behavior is unchanged. +vi.mock('./controllers/rasterController', async (importOriginal) => { + const actual = await importOriginal(); + class CapturedRasterController extends actual.RasterController { + constructor(options: ConstructorParameters[0]) { + super(options); + owners.raster = this; + } + } + return { ...actual, RasterController: CapturedRasterController }; +}); + +const DEPTH = { settings: {}, type: 'depth_anything_depth_estimation' }; +// Sizes no other surface in these tests shares, so the original's surface is recognizable. +const PHOTO = { height: 32, imageName: 'photo', width: 48 }; +const MAP = { height: 10, imageName: 'map', width: 10 }; +const PHOTO_BYTES = PHOTO.width * PHOTO.height * 4; + +const controlLayer = (overrides: Partial = {}): CanvasControlLayerContract => ({ + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + controlType: 'depth', + detection: DEPTH, + filter: DEPTH, + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Control Layer 1', + opacity: 1, + original: { image: PHOTO, type: 'image' }, + source: { bitmap: MAP, offset: { x: 0, y: 0 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: false, + ...overrides, +}); + +const bitmapStore: BitmapStore = { + discardLayer: () => undefined, + dispose: () => undefined, + flushPendingUploads: () => Promise.resolve(), + hasPendingClear: () => false, + hasPendingWork: () => false, + isSelfEcho: () => false, + markLayerDirty: () => undefined, + reset: () => undefined, + suspendLayer: () => () => undefined, +}; + +const createHarness = (layer: CanvasControlLayerContract = controlLayer()) => { + const document: CanvasDocumentContractV3 = documentFrom([layer, layerContract('raster')], layer.id); + const initial = createInitialWorkbenchState().projects[0]!; + let project: Project = { ...initial, canvas: { ...createEmptyCanvasState(), document } }; + let projectOpen = true; + const listeners = new Set<() => void>(); + const notify = (): void => { + for (const listener of listeners) { + listener(); + } + }; + const apply = (mutation: Parameters[0]): boolean => { + const next = applyCanvasProjectMutation(project, mutation); + if (next.canvas === project.canvas) { + return false; + } + project = next; + notify(); + return true; + }; + // Image fetches wait until the test settles them, so a request can be superseded or fail mid-decode. + const pendingImages: { resolve: () => void; reject: () => void }[] = []; + const resolveImage = vi.fn( + (_name: string) => + new Promise((resolve, reject) => { + pendingImages.push({ reject: () => reject(new Error('unreachable')), resolve: () => resolve(new Blob()) }); + }) + ); + const surfaces: StubRasterSurface[] = []; + const stub = createTestStubRasterBackend(); + const { engine } = createCanvasEngine({ + backend: { + ...stub, + createSurface: (width, height) => { + const surface = stub.createSurface(width, height); + surfaces.push(surface); + return surface; + }, + }, + bitmapStore, + fonts: null, + imageResolver: resolveImage, + mutationPort: { + commitEdit: () => undefined, + dispatch: apply, + getCanvasState: () => (projectOpen ? project.canvas : null), + subscribe: (listener) => { + listeners.add(listener); + return () => listeners.delete(listener); + }, + }, + projectId: project.id, + reportError: () => undefined, + uploadImage: () => Promise.resolve({ height: 1, imageName: 'image', width: 1 }), + uploadIntermediateImage: () => Promise.resolve({ height: 1, imageName: 'intermediate', width: 1 }), + }); + const raster = owners.raster!; + const settleImages = async (outcome: 'resolve' | 'reject'): Promise => { + for (let round = 0; round < 10; round += 1) { + pendingImages.splice(0).forEach((image) => image[outcome]()); + await new Promise((resolve) => { + setTimeout(resolve, 0); + }); + } + }; + /** Lets every waiting image fetch finish, then the decodes behind them. */ + const releaseImages = () => settleImages('resolve'); + const show = async (view: 'original' | 'overlay' | 'map', layerId = 'control') => { + const outcome = engine.previews.setControlLayerView(layerId, view); + await releaseImages(); + return outcome; + }; + const live = () => + project.canvas.document.stacks.control.find((node) => node.id === 'control') as CanvasControlLayerContract; + return { + apply, + canvasDocument: () => project.canvas.document, + engine, + failImages: () => settleImages('reject'), + /** What the view holds: decoding reserves, a shown original is retained as detached pixels. */ + held: () => { + const { detachedBytes, reservedBytes } = raster.memory.snapshot(); + return { detached: detachedBytes, reserved: reservedBytes }; + }, + leaveProject: () => { + projectOpen = false; + notify(); + }, + live, + mapCanvas: () => raster.layers.peek('control')?.surface.canvas, + originalSurfaces: () => surfaces.filter((s) => s.width === PHOTO.width && s.height === PHOTO.height), + releaseImages, + resolveImage, + show, + view: () => engine.interaction.get('controlLayerView'), + }; +}; + +type Harness = ReturnType; + +const NOTHING_HELD = { detached: 0, reserved: 0 }; +const ORIGINAL_HELD = { detached: PHOTO_BYTES, reserved: 0 }; + +/** A canvas element whose recorded pointer listeners `fire` invokes. */ +const fakeCanvas = () => { + const surface = createTestStubRasterBackend().createSurface(100, 100); + const listeners = new Map void>>(); + const element = { + addEventListener: (type: string, handler: (event: Event) => void) => { + listeners.set(type, (listeners.get(type) ?? new Set()).add(handler)); + }, + getBoundingClientRect: () => ({ bottom: 100, height: 100, left: 0, right: 100, top: 0, width: 100, x: 0, y: 0 }), + getContext: () => surface.ctx, + height: 100, + releasePointerCapture: () => undefined, + removeEventListener: (type: string, handler: (event: Event) => void) => { + listeners.get(type)?.delete(handler); + }, + setPointerCapture: () => undefined, + width: 100, + } as unknown as HTMLCanvasElement; + const fire = (type: string, x: number, y: number): void => { + const event = { + altKey: false, + button: 0, + buttons: 1, + clientX: x, + clientY: y, + ctrlKey: false, + metaKey: false, + pointerId: 1, + pointerType: 'mouse', + preventDefault: () => undefined, + pressure: 0.5, + shiftKey: false, + timeStamp: 0, + } as unknown as Event; + for (const handler of listeners.get(type) ?? []) { + handler(event); + } + }; + return { element, fire, surface }; +}; + +const draws = (log: RasterCallLogEntry[]) => log.filter((entry) => entry.op === 'drawImage').map((entry) => entry.args); + +/** Attaches recording canvases; `frame()` renders the queued frame and returns its `drawImage` arguments. */ +const attachScreen = async (h: Harness) => { + const frames: FrameRequestCallback[] = []; + vi.stubGlobal('requestAnimationFrame', (callback: FrameRequestCallback) => frames.push(callback)); + vi.stubGlobal('cancelAnimationFrame', () => undefined); + const screen = fakeCanvas(); + const overlay = fakeCanvas(); + h.engine.surface.attach(screen.element, overlay.element); + const frame = () => { + screen.surface.callLog.length = 0; + frames.splice(0).forEach((callback) => callback(0)); + return draws(screen.surface.callLog); + }; + const sources = () => frame().map((args) => args[0]); + frame(); + await h.releaseImages(); + frame(); + return { frame, overlay, sources }; +}; + +afterEach(() => { + vi.unstubAllGlobals(); +}); + +describe('setControlLayerView', () => { + it('shows an image original, switches to overlay without decoding again, and ends on map', async () => { + const h = createHarness(); + + const shown = h.engine.previews.setControlLayerView('control', 'original'); + // Decoding holds a reservation for the surface it is about to allocate. + expect(h.held()).toEqual({ detached: 0, reserved: PHOTO_BYTES }); + await h.releaseImages(); + await expect(shown).resolves.toBe('shown'); + expect(h.view()).toEqual({ layerId: 'control', view: 'original' }); + expect(h.held()).toEqual(ORIGINAL_HELD); + expect(h.originalSurfaces()).toHaveLength(1); + + const fetches = h.resolveImage.mock.calls.length; + await expect(h.show('overlay')).resolves.toBe('shown'); + expect(h.view()).toEqual({ layerId: 'control', view: 'overlay' }); + expect(h.resolveImage).toHaveBeenCalledTimes(fetches); + expect(h.held()).toEqual(ORIGINAL_HELD); + + await expect(h.show('map')).resolves.toBe('shown'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + }); + + it('places a painted original at its offset', async () => { + const original: CanvasPixelSourceContract = { bitmap: PHOTO, offset: { x: -4, y: 2 }, type: 'paint' }; + const h = createHarness(controlLayer({ original })); + const { frame } = await attachScreen(h); + + await expect(h.show('original')).resolves.toBe('shown'); + + const [surface] = h.originalSurfaces(); + expect( + frame() + .find((args) => args[0] === surface!.canvas) + ?.slice(1) + ).toEqual([-4, 2]); + }); + + it.each([ + ['a control layer without an original', 'control'], + ['a raster layer', 'raster'], + ['an unknown layer', 'gone'], + ])('reports %s as missing', async (_case, layerId) => { + const untyped = controlLayer(); + delete untyped.original; + const h = createHarness(untyped); + + await expect(h.show('original', layerId)).resolves.toBe('missing'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + }); + + it('reports an original that cannot be decoded as failed', async () => { + const h = createHarness(); + + const shown = h.engine.previews.setControlLayerView('control', 'original'); + await h.failImages(); + + await expect(shown).resolves.toBe('failed'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + }); + + it.each<[string, (h: Harness) => void]>([ + ['an operation holds the document', (h) => h.engine.stores.documentEditingLocked.set(true)], + ['a generation locks the canvas', (h) => h.engine.tools.setInteractionLocked(true)], + ])('refuses a view while %s', async (_case, lock) => { + const h = createHarness(); + lock(h); + + await expect(h.show('original')).resolves.toBe('locked'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + expect(h.resolveImage).not.toHaveBeenCalledWith('photo', expect.anything()); + }); + + it('drops a decode a newer request superseded', async () => { + const h = createHarness(); + + const first = h.engine.previews.setControlLayerView('control', 'original'); + const second = h.engine.previews.setControlLayerView('control', 'overlay'); + await h.releaseImages(); + + await expect(first).resolves.toBe('stale'); + await expect(second).resolves.toBe('shown'); + expect(h.view()).toEqual({ layerId: 'control', view: 'overlay' }); + expect(h.held()).toEqual(ORIGINAL_HELD); + }); + + it.each<[string, (h: Harness) => Promise | unknown]>([ + [ + 'an edit of the layer', + (h) => + h.apply({ + config: { adapter: { weight: 0.5 }, layerType: 'control' }, + id: 'control', + type: 'updateCanvasLayerConfig', + }), + ], + ['a map request', (h) => h.engine.previews.setControlLayerView('control', 'map')], + ['selecting another layer', (h) => h.apply({ id: 'raster', type: 'setCanvasSelectedLayer' })], + ['cooldown', (h) => h.engine.lifecycle.beginCooldown()], + ['disposal', (h) => h.engine.lifecycle.dispose()], + ])('drops a decode in flight on %s', async (_case, act) => { + const h = createHarness(); + + const pending = h.engine.previews.setControlLayerView('control', 'original'); + await act(h); + await h.releaseImages(); + + await expect(pending).resolves.toBe('stale'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + }); + + it('refuses an original the raster budget cannot hold, before decoding it', async () => { + const huge = { height: 12_000, imageName: 'huge', width: 12_000 }; + const h = createHarness(controlLayer({ original: { image: huge, type: 'image' } })); + + await expect(h.show('original')).resolves.toBe('over-budget'); + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + expect(h.resolveImage).not.toHaveBeenCalledWith('huge', expect.anything()); + }); + + it('never records history or touches the document', async () => { + const h = createHarness(); + const entries = h.engine.history.getEntries(); + const document = h.canvasDocument(); + + await h.show('original'); + await h.show('overlay'); + await h.show('map'); + + expect(h.engine.history.getEntries()).toEqual(entries); + expect(h.canvasDocument()).toBe(document); + }); + + it.each<[string, (h: Harness) => Promise | unknown]>([ + [ + 'an edit of the layer', + (h) => + h.engine.controlDerivation.commitControlLayerPatch({ + expected: h.live(), + label: 'Weight', + next: { ...h.live(), adapter: { ...h.live().adapter, weight: 0.5 } }, + }), + ], + [ + 'an opacity change of the layer', + (h) => h.apply({ id: 'control', patch: { opacity: 0.5 }, type: 'updateCanvasLayer' }), + ], + ['undo', (h) => h.engine.history.undo()], + [ + 'redo', + async (h) => { + await h.engine.history.undo(); + await h.show('original'); + expect(h.view()).not.toBeNull(); + await h.engine.history.redo(); + }, + ], + [ + 'a detection map commit', + async (h) => { + const guard = h.engine.controlDerivation.captureControlDetectionGuard('control')!; + const committed = h.engine.controlDerivation.commitControlMap({ + detection: DEPTH, + guard, + image: { height: 10, imageName: 'map-2', width: 10 }, + makeDurable: () => Promise.resolve(), + rect: { height: 10, width: 10, x: 0, y: 0 }, + }); + await h.releaseImages(); + expect(await committed).toEqual({ status: 'committed' }); + }, + ], + ['selecting another layer', (h) => h.apply({ id: 'raster', type: 'setCanvasSelectedLayer' })], + ['removing the layer', (h) => h.apply({ ids: ['control'], type: 'removeCanvasLayers' })], + [ + 'replacing the document', + (h) => h.apply({ document: documentFrom([controlLayer()], 'control'), type: 'replaceCanvasDocument' }), + ], + ['leaving the project', (h) => h.leaveProject()], + ['cooldown', (h) => h.engine.lifecycle.beginCooldown()], + ['a cache clear', (h) => h.engine.diagnostics.clearCaches()], + ['an operation taking the document', (h) => h.engine.stores.documentEditingLocked.set(true)], + ['a generation locking the canvas', (h) => h.engine.tools.setInteractionLocked(true)], + ['disposal', (h) => h.engine.lifecycle.dispose()], + ])('returns to the map on %s', async (_case, act) => { + const h = createHarness(); + // Undo needs a step to revert; recording it before the view leaves the view on the current layer. + h.engine.controlDerivation.commitControlLayerPatch({ + expected: h.live(), + label: 'Name', + next: { ...h.live(), name: 'Depth' }, + }); + await h.show('original'); + expect(h.view()).not.toBeNull(); + + await act(h); + + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + }); + + it('keeps the view while another layer is edited', async () => { + const h = createHarness(); + await h.show('original'); + + h.apply({ id: 'raster', patch: { opacity: 0.5 }, type: 'updateCanvasLayer' }); + + expect(h.view()).toEqual({ layerId: 'control', view: 'original' }); + }); + + it('draws the original on screen while export and thumbnails keep reading the map', async () => { + const h = createHarness(); + const { sources } = await attachScreen(h); + + await h.show('original'); + const [original] = h.originalSurfaces(); + expect(sources()).toContain(original!.canvas); + + const exported = h.engine.exports.exportBakedLayerBlob('control'); + await h.releaseImages(); + const result = await exported; + expect(result.status === 'ok' && (await result.blob.text())).toBe('stub-surface-10x10'); + + const thumbnail = fakeCanvas(); + expect(h.engine.previews.drawLayerThumbnail('control', thumbnail.element, 64)).toBe(true); + expect(draws(thumbnail.surface.callLog).map((args) => args[0])).not.toContain(original!.canvas); + }); + + it('returns to the map when an operation takes the document, and stays there after it ends', async () => { + const h = createHarness(); + const { sources } = await attachScreen(h); + await h.show('original'); + const [original] = h.originalSurfaces(); + expect(sources()).not.toContain(h.mapCanvas()); + + // What the operation controller publishes when Filter or Select Object acquires the edit lease. + h.engine.stores.documentEditingLocked.set(true); + + expect(h.view()).toBeNull(); + const locked = sources(); + expect(locked).toContain(h.mapCanvas()); + expect(locked).not.toContain(original!.canvas); + + h.engine.stores.documentEditingLocked.set(false); + await h.releaseImages(); + + expect(h.view()).toBeNull(); + expect(sources()).not.toContain(original!.canvas); + }); + + it('returns to the map as a stroke starts on the layer, so the stroke paints visibly', async () => { + vi.stubGlobal( + 'Path2D', + class { + closePath() {} + lineTo() {} + moveTo() {} + quadraticCurveTo() {} + } + ); + const h = createHarness(); + const { overlay, sources } = await attachScreen(h); + await h.show('original'); + expect(sources()).not.toContain(h.mapCanvas()); + h.engine.tools.setTool('brush'); + + overlay.fire('pointerdown', 5, 5); + + expect(h.view()).toBeNull(); + expect(h.held()).toEqual(NOTHING_HELD); + expect(sources()).toContain(h.mapCanvas()); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/bakedControlLayer.browser.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/bakedControlLayer.browser.test.ts index 5b4d15cd5c4..eace59dede8 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/bakedControlLayer.browser.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/bakedControlLayer.browser.test.ts @@ -31,7 +31,7 @@ const ADJUSTMENTS: CanvasAdjustmentsContract = [ ]; const TRANSFORM = { rotation: Math.PI / 9, scaleX: 0.6, scaleY: 1.3, x: 11, y: 5 }; const CONTENT: Rect = { height: 30, width: 40, x: -6, y: -4 }; -const CONTROL_MODEL = { base: 'sdxl', model: 'union-model' } as const; +const CONTROL_BASE = 'sdxl'; const LAYER: CanvasRasterLayerContractV2 = { adjustments: ADJUSTMENTS, @@ -119,7 +119,7 @@ const scene = (runSpatial?: SpatialRunner) => { }, ctx: h.ctx, exportBaked, - getControlModel: () => CONTROL_MODEL, + getMainModelBase: () => CONTROL_BASE, isGuardCurrent, }); const unused = () => { @@ -139,7 +139,7 @@ const scene = (runSpatial?: SpatialRunner) => { ctx: h.ctx, discardPersisted: () => undefined, exportBaked, - getControlModel: () => CONTROL_MODEL, + getMainModelBase: () => CONTROL_BASE, getDuplicateRasterPlan: unused, getSelectedLayerIds: unused, hasPendingPixelWork: () => false, @@ -174,7 +174,7 @@ const scene = (runSpatial?: SpatialRunner) => { }; const bakedControl = (id: string, name: string, rect: Rect) => ({ - adapter: createDefaultControlAdapter(CONTROL_MODEL.base, CONTROL_MODEL.model), + adapter: createDefaultControlAdapter(CONTROL_BASE, null), blendMode: LAYER.blendMode, id, isEnabled: LAYER.isEnabled, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.test.ts new file mode 100644 index 00000000000..0962da62476 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.test.ts @@ -0,0 +1,482 @@ +import type { + CanvasControlLayerContract, + CanvasDocumentContractV3, + CanvasLayerContract, +} from '@workbench/canvas-engine/contracts'; +import type { Rect } from '@workbench/canvas-engine/types'; + +import { groupContract, stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; +import { describe, expect, it, vi } from 'vitest'; + +import { ControlDerivationController } from './controlDerivationController'; +import { createLayerOperationHarness } from './layerOperationHarness.testing'; + +const DEPTH = { settings: { model_size: 'small' }, type: 'depth_anything_depth_estimation' }; +const PHOTO = { height: 64, imageName: 'photo', width: 64 }; +const MAP = { height: 32, imageName: 'depth-map', width: 32 }; +const MAP_RECT: Rect = { height: 32, width: 32, x: 4, y: 8 }; +const CACHE_RECT: Rect = { height: 20, width: 20, x: 0, y: 0 }; + +const controlLayer = (overrides: Partial = {}): CanvasControlLayerContract => ({ + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Control Layer 1', + opacity: 1, + source: { image: PHOTO, type: 'image' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + ...overrides, +}); + +const rasterLayer: CanvasLayerContract = { + blendMode: 'normal', + id: 'raster', + isEnabled: true, + isLocked: false, + name: 'Raster', + opacity: 1, + source: { bitmap: null, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'raster', +}; + +const createHarness = ( + layer: CanvasControlLayerContract = controlLayer(), + options: { historyBytes?: number; pendingPixels?: boolean; lockedParent?: boolean } = {} +) => { + const document: CanvasDocumentContractV3 = { + background: 'transparent', + bbox: { height: 64, width: 64, x: 0, y: 0 }, + height: 64, + selectedLayerId: layer.id, + stacks: stacksFrom([ + rasterLayer, + options.lockedParent ? groupContract('group', [layer], { isLocked: true }) : layer, + ]), + version: 3, + width: 64, + }; + const endStructuralPreview = vi.fn(); + const h = createLayerOperationHarness(document, { endStructuralPreview, historyBytes: options.historyBytes }); + h.layers.getOrCreateRect(layer.id, CACHE_RECT); + const fake = { documentRevision: 1, pendingPixels: options.pendingPixels ?? false, sessionHeld: false }; + const discardPersisted = vi.fn(); + const decodeImage = vi.fn((image: { width: number; height: number }) => + Promise.resolve({ + decodedHeight: image.height, + decodedWidth: image.width, + status: 'ok' as const, + surface: h.backend.createSurface(image.width, image.height), + }) + ); + const controller = new ControlDerivationController({ + captureCache: (target) => { + const entry = h.layers.get(target.id); + if (!entry) { + return null; + } + const pixels = h.backend.createSurface(entry.rect.width, entry.rect.height); + pixels.ctx.drawImage(entry.surface.canvas, 0, 0); + return { pixels, rect: { ...entry.rect } }; + }, + ctx: h.ctx, + decodeImage, + discardPersisted, + getDocumentRevision: () => fake.documentRevision, + hasPendingPixelWork: () => fake.pendingPixels, + isLayerSessionHeld: () => fake.sessionHeld, + }); + const live = (): CanvasControlLayerContract => getDocumentLayer(h.document(), layer.id) as CanvasControlLayerContract; + /** Commits `patch` against the live layer, as the coordinator's intents do. */ + const patch = (label: string, change: Partial, coalesceKey?: string) => + controller.commitControlLayerPatch({ coalesceKey, expected: live(), label, next: { ...live(), ...change } }); + const makeDurable = vi.fn(() => Promise.resolve()); + const commitMap = (guard = controller.captureControlDetectionGuard(layer.id)!) => + controller.commitControlMap({ detection: DEPTH, guard, image: MAP, makeDurable, rect: MAP_RECT }); + return { + ...h, + commitMap, + controller, + decodeImage, + discardPersisted, + endStructuralPreview, + fake, + live, + makeDurable, + patch, + }; +}; + +describe('ControlDerivationController.commitControlLayerPatch', () => { + it('commits one user entry whose undo and redo restore the exact contracts', async () => { + const h = createHarness(); + const before = h.live(); + + expect(h.patch('Set control type', { controlType: 'depth', detection: DEPTH })).toEqual({ status: 'committed' }); + const after = h.live(); + expect(after).toMatchObject({ controlType: 'depth', detection: DEPTH }); + expect(h.history.entries().past).toEqual(['Set control type']); + expect(h.committedEdits).toHaveLength(1); + + await h.history.undo(); + expect(h.live()).toEqual(before); + await h.history.redo(); + expect(h.live()).toEqual(after); + }); + + it('coalesces consecutive patches with the same key into the top entry', async () => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + const before = h.live(); + + h.patch('Detection settings', { detection: { ...DEPTH, settings: { model_size: 'base' } } }, 'settings'); + h.patch('Detection settings', { detection: { ...DEPTH, settings: { model_size: 'large' } } }, 'settings'); + expect(h.history.entries().past).toEqual(['Detection settings']); + + h.patch('Detection settings', { detection: DEPTH }, 'other'); + expect(h.history.entries().past).toHaveLength(2); + + await h.history.undo(); + await h.history.undo(); + expect(h.live()).toEqual(before); + }); + + it('refuses a patch prepared against a layer that has changed since', () => { + const h = createHarness(); + const expected = h.live(); + h.patch('Weight', { adapter: { ...expected.adapter, weight: 0.5 } }); + + expect( + h.controller.commitControlLayerPatch({ expected, label: 'Set type', next: { ...expected, controlType: 'pose' } }) + ).toEqual({ status: 'stale' }); + expect(h.history.entries().past).toEqual(['Weight']); + }); + + it('keeps the outgoing unpersisted pixels when it replaces the source', async () => { + const painted = controlLayer({ source: { bitmap: null, type: 'paint' } }); + const h = createHarness(painted, { pendingPixels: true }); + + h.patch('Replace image', { source: { image: PHOTO, type: 'image' } }); + expect(h.discardPersisted).toHaveBeenCalledWith('control'); + const installedBeforeUndo = h.installed.length; + + await h.history.undo(); + expect(h.live()).toEqual(painted); + expect(h.installed).toHaveLength(installedBeforeUndo + 1); + expect(h.layers.get('control')?.rect).toEqual(CACHE_RECT); + }); + + it('leaves pending persistence alone when the source does not change', () => { + const h = createHarness(controlLayer(), { pendingPixels: true }); + + h.patch('Set control type', { controlType: 'depth', detection: DEPTH }); + expect(h.discardPersisted).not.toHaveBeenCalled(); + }); + + it('is refused while an operation holds the edit lease or a session holds the layer', () => { + const h = createHarness(); + h.setEditingLocked(true); + expect(h.patch('Set control type', { controlType: 'depth' })).toEqual({ status: 'busy' }); + h.setEditingLocked(false); + h.fake.sessionHeld = true; + expect(h.patch('Set control type', { controlType: 'depth' })).toEqual({ status: 'busy' }); + expect(h.history.entries().past).toEqual([]); + expect(h.dispatched).toEqual([]); + }); + + it('is refused when its entry could never be kept', () => { + const h = createHarness(controlLayer(), { historyBytes: 16 }); + + expect(h.patch('Set control type', { controlType: 'depth' })).toEqual({ status: 'over-budget' }); + expect(h.dispatched).toEqual([]); + }); +}); + +describe('ControlDerivationController.commitControlMap', () => { + it('lands in the intent entry, so one undo restores the pre-intent layer', async () => { + const h = createHarness(); + const untyped = h.live(); + h.patch('Set control type', { controlType: 'depth', detection: DEPTH }); + + await expect(h.commitMap()).resolves.toEqual({ status: 'committed' }); + const mapped = h.live(); + expect(mapped).toMatchObject({ + detection: DEPTH, + filter: DEPTH, + original: { image: PHOTO, type: 'image' }, + source: { bitmap: MAP, offset: { x: 4, y: 8 }, type: 'paint' }, + }); + expect(h.history.entries().past).toEqual(['Set control type']); + expect(h.layers.get('control')?.rect).toEqual(MAP_RECT); + expect(h.discardPersisted).toHaveBeenCalledWith('control'); + // The patch routed as a user edit; the background map did not. + expect(h.committedEdits).toHaveLength(1); + + await h.history.undo(); + expect(h.live()).toEqual(untyped); + await h.history.redo(); + expect(h.live()).toEqual(mapped); + expect(h.layers.get('control')?.rect).toEqual(MAP_RECT); + }); + + it('restores the unpersisted map paint the intent and its map replaced', async () => { + const painted = controlLayer({ + controlType: 'depth', + detection: DEPTH, + filter: DEPTH, + original: { image: PHOTO, type: 'image' }, + source: { bitmap: { height: 20, imageName: 'old-map', width: 20 }, type: 'paint' }, + }); + const h = createHarness(painted, { pendingPixels: true }); + h.patch('Detection settings', { detection: { ...DEPTH, settings: { model_size: 'base' } } }, 'settings'); + + await expect(h.commitMap()).resolves.toEqual({ status: 'committed' }); + expect(h.layers.get('control')?.rect).toEqual(MAP_RECT); + + await h.history.undo(); + expect(h.live()).toEqual(painted); + expect(h.layers.get('control')?.rect).toEqual(CACHE_RECT); + }); + + it('records its own entry when the intent is no longer the newest step', async () => { + const h = createHarness(); + h.patch('Set control type', { controlType: 'depth', detection: DEPTH }); + const intent = h.live(); + const guard = h.controller.captureControlDetectionGuard('control')!; + h.patch('Weight', { adapter: { ...intent.adapter, weight: 0.4 } }); + + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'committed' }); + expect(h.history.entries().past).toEqual(['Set control type', 'Weight', 'Detect depth']); + + await h.history.undo(); + expect(h.live()).toEqual({ ...intent, adapter: { ...intent.adapter, weight: 0.4 } }); + }); + + it('is not staled by weight, name or painting on the map', async () => { + const h = createHarness( + controlLayer({ + controlType: 'depth', + detection: DEPTH, + original: { image: PHOTO, type: 'image' }, + source: { bitmap: { height: 20, imageName: 'old-map', width: 20 }, type: 'paint' }, + }) + ); + const guard = h.controller.captureControlDetectionGuard('control')!; + h.patch('Rename', { adapter: { ...h.live().adapter, weight: 0.2 }, name: 'Depth 1' }); + h.ctx.dispatch({ + id: 'control', + source: { bitmap: { height: 20, imageName: 'painted-map', width: 20 }, type: 'paint' }, + type: 'updateCanvasLayerSource', + }); + + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'committed' }); + expect(h.live()).toMatchObject({ adapter: { weight: 0.2 }, name: 'Depth 1', source: { bitmap: MAP } }); + }); + + it.each([ + ['the document was replaced', (h: ReturnType) => (h.fake.documentRevision += 1)], + [ + 'the detection input changed', + (h: ReturnType) => + h.patch('Detection settings', { detection: { ...DEPTH, settings: { model_size: 'large' } } }), + ], + ])('records nothing when %s', async (_case, change) => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + const guard = h.controller.captureControlDetectionGuard('control')!; + change(h); + const document = h.document(); + const entries = h.history.entries(); + + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'stale' }); + expect(h.document()).toBe(document); + expect(h.history.entries()).toEqual(entries); + }); + + it('is refused while an operation holds the edit lease or a session holds the layer', async () => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + const guard = h.controller.captureControlDetectionGuard('control')!; + + h.setEditingLocked(true); + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'busy' }); + h.setEditingLocked(false); + h.fake.sessionHeld = true; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'busy' }); + expect(h.dispatched).toEqual([]); + + h.fake.sessionHeld = false; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'committed' }); + }); + + it('waits for unpersisted paint on its own input to reach the original it records', async () => { + const drawn = controlLayer({ + controlType: 'depth', + detection: DEPTH, + source: { bitmap: { height: 20, imageName: 'drawn', width: 20 }, type: 'paint' }, + }); + const h = createHarness(drawn, { pendingPixels: true }); + + await expect(h.commitMap()).resolves.toEqual({ status: 'busy' }); + expect(h.dispatched).toEqual([]); + }); + + it('refuses maps it cannot represent and pixels it cannot prepare', async () => { + const h = createHarness(); + await expect(h.commitMap()).resolves.toEqual({ status: 'unrepresentable' }); + + const typed = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + typed.state.rasterBytes = 0; + await expect(typed.commitMap()).resolves.toEqual({ status: 'over-budget' }); + expect(typed.dispatched).toEqual([]); + expect(typed.history.entries().past).toEqual([]); + }); + + it('does not decode for a caller that already aborted', async () => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + const guard = h.controller.captureControlDetectionGuard('control')!; + + await expect( + h.controller.commitControlMap({ + detection: DEPTH, + guard, + image: MAP, + makeDurable: h.makeDurable, + rect: MAP_RECT, + signal: AbortSignal.abort(), + }) + ).resolves.toEqual({ status: 'aborted' }); + expect(h.decodeImage).not.toHaveBeenCalled(); + expect(h.makeDurable).not.toHaveBeenCalled(); + }); + + it('promotes the map only once nothing refuses it, before decoding', async () => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + const guard = h.controller.captureControlDetectionGuard('control')!; + + h.setEditingLocked(true); + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'busy' }); + h.setEditingLocked(false); + h.fake.documentRevision += 1; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'stale' }); + expect(h.makeDurable).not.toHaveBeenCalled(); + + h.fake.documentRevision -= 1; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'committed' }); + expect(h.makeDurable).toHaveBeenCalledOnce(); + expect(h.makeDurable.mock.invocationCallOrder[0]).toBeLessThan(h.decodeImage.mock.invocationCallOrder[0]!); + }); + + it('refuses a map it could not promote without decoding or recording it', async () => { + const h = createHarness(controlLayer({ controlType: 'depth', detection: DEPTH })); + h.makeDurable.mockRejectedValueOnce(new Error('PATCH 500')); + + await expect(h.commitMap()).resolves.toEqual({ message: 'PATCH 500', status: 'failed' }); + expect(h.decodeImage).not.toHaveBeenCalled(); + expect(h.dispatched).toEqual([]); + expect(h.history.entries().past).toEqual([]); + }); +}); + +describe('ControlDerivationController pairing record', () => { + // The record's before-pixels have no other observable owner once their entry is gone; read it directly. + const recorded = (h: ReturnType) => h.controller['last']; + /** Another owner's step, as a brush stroke or a move would publish. */ + const publishForeignEntry = (h: ReturnType) => + h.history.admit(256)!.publish({ + bytes: 256, + heldAssetRefs: { images: [], videos: [] }, + label: 'Brush stroke', + redo: () => undefined, + undo: () => undefined, + }); + + it.each([ + ['history is cleared', (h: ReturnType) => h.history.clear()], + ['the byte budget evicts it', (h: ReturnType) => h.history.trimToBytes(0)], + [ + 'another step discards it from the redo stack', + async (h: ReturnType) => { + await h.history.undo(); + expect(recorded(h)).not.toBeNull(); + publishForeignEntry(h); + }, + ], + ])('releases the unpersisted before-pixels once %s', async (_case, dropEntry) => { + const h = createHarness(controlLayer({ source: { bitmap: null, type: 'paint' } }), { pendingPixels: true }); + h.patch('Replace image', { source: { image: PHOTO, type: 'image' } }); + expect(recorded(h)?.beforePixels).not.toBeNull(); + + await dropEntry(h); + + expect(recorded(h)).toBeNull(); + }); +}); + +describe('ControlDerivationController refusals', () => { + const typed = () => controlLayer({ controlType: 'depth', detection: DEPTH }); + + it('refuses both commits under a locked ancestor, before decoding', async () => { + const h = createHarness(typed(), { lockedParent: true }); + const guard = h.controller.captureControlDetectionGuard('control')!; + + expect(h.patch('Set control type', { controlType: 'pose' })).toEqual({ status: 'locked' }); + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'locked' }); + expect(h.decodeImage).not.toHaveBeenCalled(); + expect(h.dispatched).toEqual([]); + }); + + it('reports a removed layer as missing', async () => { + const h = createHarness(typed()); + const expected = h.live(); + const guard = h.controller.captureControlDetectionGuard('control')!; + h.ctx.dispatch({ + enabledUpdates: [], + removeIds: ['control'], + selectedLayerId: null, + type: 'applyCanvasLayerStackMutation', + }); + + expect( + h.controller.commitControlLayerPatch({ expected, label: 'Set type', next: { ...expected, controlType: 'pose' } }) + ).toEqual({ status: 'missing' }); + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'missing' }); + expect(h.history.entries().past).toEqual([]); + }); + + it('retries a decode the edit lock interrupted as busy', async () => { + const h = createHarness(typed()); + h.decodeImage.mockResolvedValueOnce({ status: 'stale' } as never); + + await expect(h.commitMap()).resolves.toEqual({ status: 'busy' }); + expect(h.dispatched).toEqual([]); + }); + + it('leaves an open structural preview alone until a commit is admitted', async () => { + const h = createHarness(typed()); + const guard = h.controller.captureControlDetectionGuard('control')!; + h.fake.sessionHeld = true; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'busy' }); + expect(h.endStructuralPreview).not.toHaveBeenCalled(); + + h.fake.sessionHeld = false; + await expect(h.commitMap(guard)).resolves.toEqual({ status: 'committed' }); + expect(h.endStructuralPreview).toHaveBeenCalled(); + }); +}); + +describe('ControlDerivationController.captureControlDetectionGuard', () => { + it('captures only control layers', () => { + const h = createHarness(); + + expect(h.controller.captureControlDetectionGuard('control')).toMatchObject({ + documentRevision: 1, + layerId: 'control', + projectId: h.ctx.projectId, + }); + expect(h.controller.captureControlDetectionGuard('raster')).toBeNull(); + expect(h.controller.captureControlDetectionGuard('absent')).toBeNull(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.ts new file mode 100644 index 00000000000..72c41433b99 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlDerivationController.ts @@ -0,0 +1,441 @@ +import type { + CanvasControlDerivationCapability, + ControlDetectionGuard, + ControlLayerPatchResult, + ControlMapCommitResult, +} from '@workbench/canvas-engine/capabilities'; +import type { + CanvasControlLayerContract, + CanvasDocumentContractV3, + CanvasImageRef, +} from '@workbench/canvas-engine/contracts'; +import type { LayerMutationControllerOptions } from '@workbench/canvas-engine/controllers/layerMutationController'; +import type { DecodeImageResult } from '@workbench/canvas-engine/controllers/rasterController'; +import type { HistoryEntryToken } from '@workbench/canvas-engine/history/history'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; +import { lookupDocumentLeaf } from '@workbench/canvas-engine/document-model/documentModel'; +import { + controlDetectionInputKey, + getControlDetectionInput, + withDetectedMap, +} from '@workbench/canvas-engine/document/controlDerivation'; +import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; +import { collectHistoryMediaRefs, HISTORY_ENTRY_OVERHEAD_BYTES } from '@workbench/canvas-engine/history/history'; + +import type { CanvasMutationContext, EditStep, EditTransaction } from './mutationContext'; + +import { + guardedResultRefusal, + layerEditRefusal, + type LayerPixels, + replaceLayerStep, + replayWithPixels, + rgbaBytes, +} from './editSteps'; + +export interface ControlDerivationControllerOptions { + readonly ctx: Pick< + CanvasMutationContext, + | 'applyStep' + | 'begin' + | 'capturePermit' + | 'getDocument' + | 'getReducerDocument' + | 'historyTop' + | 'installPrepared' + | 'isPermitCurrent' + | 'preparePixels' + | 'projectId' + | 'reserveRaster' + >; + readonly captureCache: LayerMutationControllerOptions['captureCache']; + readonly decodeImage: ( + image: CanvasImageRef, + options: { signal?: AbortSignal; isCurrent?: () => boolean; scaleToImage?: boolean } + ) => Promise; + readonly discardPersisted: (layerId: string) => void; + /** The reducer state's wholesale-replacement counter; null without canvas state. */ + readonly getDocumentRevision: () => number | null; + /** Whether the layer's live pixels are not yet represented by its persisted source. */ + readonly hasPendingPixelWork: (layerId: string) => boolean; + /** Whether a session (transform, text, float, pixel edit, operation) owns the layer's live pixels. */ + readonly isLayerSessionHeld: (layerId: string) => boolean; +} + +/** The newest entry this controller recorded, so a later patch or map can amend it while it is still on top. */ +interface RecordedEntry { + readonly token: HistoryEntryToken; + readonly layerId: string; + readonly label: string; + readonly coalesceKey: string | undefined; + /** `controlDetectionInputKey` of the entry's after-state. */ + readonly inputKey: string; + /** + * Whether a map for `inputKey` belongs to this entry: it changed what detection runs on (an intent that requested + * the map), or it is a map itself. A weight or name patch is not, so undoing it never also drops a later map. + */ + readonly requestsMap: boolean; + /** The layer the entry's undo restores, with its unpersisted pixels when its source cannot reproduce them. */ + readonly before: CanvasControlLayerContract; + readonly beforePixels: LayerPixels | null; +} + +type BeforeState = + | { readonly before: CanvasControlLayerContract; readonly pixels: LayerPixels | null } + | { readonly refusal: 'not-ready' | 'over-budget' }; + +const asControlLayer = ( + document: CanvasDocumentContractV3 | null, + layerId: string +): CanvasControlLayerContract | null => { + const layer = getDocumentLayer(document, layerId); + return layer?.type === 'control' ? layer : null; +}; + +const sameSource = (a: CanvasControlLayerContract, b: CanvasControlLayerContract): boolean => + areJsonValuesStructurallyEqual(a.source, b.source); + +/** + * Commits control-layer derivation edits: intent patches (type, detector, settings, image) and background detection + * maps. Neither takes the operation edit owner, so both yield to a modal operation, gesture, replay or layer session + * and the caller retries. A map lands in the same undo step as the intent that requested it while that intent is + * still the newest step. + */ +export class ControlDerivationController implements CanvasControlDerivationCapability { + private last: RecordedEntry | null = null; + + constructor(private readonly options: ControlDerivationControllerOptions) {} + + commitControlLayerPatch({ + coalesceKey, + expected, + label, + next, + }: Parameters[0]): ControlLayerPatchResult { + const { ctx, ...o } = this.options; + if (next.id !== expected.id || next.type !== 'control') { + return { message: 'A control layer patch must keep the layer id and type.', status: 'failed' }; + } + const layerId = expected.id; + const document = ctx.getDocument(); + const live = asControlLayer(document, layerId); + if (!document || !live) { + return { status: 'missing' }; + } + if (live !== expected && !areJsonValuesStructurallyEqual(live, expected)) { + return { status: 'stale' }; + } + if (lookupDocumentLeaf(document, layerId)?.effectiveLocked !== false) { + return { status: 'locked' }; + } + if (o.isLayerSessionHeld(layerId)) { + return { status: 'busy' }; + } + const txn = ctx.begin({ historyBytes: HISTORY_ENTRY_OVERHEAD_BYTES }); + if (!('publish' in txn)) { + return { status: layerEditRefusal(txn.status) }; + } + try { + const amended = + coalesceKey !== undefined ? this.amendable(layerId, (entry) => entry.coalesceKey === coalesceKey) : null; + const resolved = this.resolveBefore(txn, document, live, next, amended); + if ('refusal' in resolved) { + return { status: resolved.refusal }; + } + const { before, pixels: beforePixels } = resolved; + const after = structuredClone(next); + const recorded = this.recordOnPublish(); + const result = txn.publish( + amended?.label ?? label, + this.replaceStep(after, null, live), + { + bytes: (beforePixels ? rgbaBytes(beforePixels.rect) : 0) + HISTORY_ENTRY_OVERHEAD_BYTES, + dispose: recorded.dispose, + heldAssetRefs: collectHistoryMediaRefs(before, after), + redo: () => ctx.applyStep(this.replaceStep(after, null, before)), + undo: () => this.restoreBefore(before, beforePixels, after), + }, + { origin: 'user', replacing: amended?.token } + ); + if (result.status !== 'committed') { + return result.status === 'dispatch-rejected' || result.status === 'postcondition-failed' + ? { status: 'failed' } + : { status: layerEditRefusal(result.status) }; + } + const inputKey = controlDetectionInputKey(after); + recorded.record({ + before, + beforePixels, + coalesceKey, + inputKey, + label: amended?.label ?? label, + layerId, + requestsMap: controlDetectionInputKey(before) !== inputKey, + token: result.token, + }); + return { status: 'committed' }; + } finally { + txn.end(); + } + } + + captureControlDetectionGuard(layerId: string): ControlDetectionGuard | null { + // A background read: the reducer document, so an open structural preview stays open. + const layer = asControlLayer(this.options.ctx.getReducerDocument(), layerId); + const documentRevision = this.options.getDocumentRevision(); + if (!layer || documentRevision === null) { + return null; + } + return { + documentRevision, + inputKey: controlDetectionInputKey(layer), + layerId, + projectId: this.options.ctx.projectId, + }; + } + + async commitControlMap({ + detection, + guard, + image, + makeDurable, + rect, + signal, + }: Parameters[0]): Promise { + const { ctx, ...o } = this.options; + const permit = ctx.capturePermit(); + if (!permit) { + return { status: 'busy' }; + } + if (signal?.aborted) { + return { status: 'aborted' }; + } + /** + * Everything the map was computed against must still hold. It reads the reducer document, so a refusal leaves an + * open structural preview alone; only an admitted commit ends it. + */ + const check = (): + | { document: CanvasDocumentContractV3; live: CanvasControlLayerContract } + | ControlMapCommitResult => { + if (signal?.aborted) { + return { status: 'aborted' }; + } + if (!ctx.isPermitCurrent(permit)) { + return { status: 'busy' }; + } + const document = ctx.getReducerDocument(); + const live = asControlLayer(document, guard.layerId); + if (!document || !live) { + return { status: 'missing' }; + } + if (o.getDocumentRevision() !== guard.documentRevision || controlDetectionInputKey(live) !== guard.inputKey) { + return { status: 'stale' }; + } + if (lookupDocumentLeaf(document, live.id)?.effectiveLocked !== false) { + return { status: 'locked' }; + } + // Pixels painted on the input itself after the run started are not in the original it names yet. + const input = getControlDetectionInput(live); + const inputIsPendingSource = live.original === undefined && input !== null && o.hasPendingPixelWork(live.id); + if (o.isLayerSessionHeld(live.id) || inputIsPendingSource) { + return { status: 'busy' }; + } + return { document, live }; + }; + // Refusals the caller retries are cheap to find; find them before decoding. + const early = check(); + if ('status' in early) { + return early; + } + const mapRect = { ...rect }; + const map = { image: structuredClone(image), rect: mapRect }; + if (!withDetectedMap(early.live, map, detection)) { + return { status: 'unrepresentable' }; + } + try { + // Promoted only once nothing refuses it, so a result that is dropped stays collectible. + await makeDurable(); + const decoded = await o.decodeImage(image, { + isCurrent: () => ctx.isPermitCurrent(permit), + scaleToImage: false, + signal, + }); + if (decoded.status !== 'ok') { + return { status: decoded.status === 'aborted' ? 'aborted' : 'busy' }; + } + const decodedCheck = check(); + if ('status' in decodedCheck) { + return decodedCheck; + } + const afterPixels: LayerPixels = { pixels: decoded.surface, rect: mapRect }; + const txn = ctx.begin({ historyBytes: rgbaBytes(mapRect) + HISTORY_ENTRY_OVERHEAD_BYTES }); + if (!('publish' in txn)) { + return { status: layerEditRefusal(txn.status) }; + } + try { + // Admission ended any structural preview; the map lands on the committed layer, never on previewed values. + const checked = check(); + if ('status' in checked) { + return checked; + } + const { document, live } = checked; + const after = withDetectedMap(live, map, structuredClone(detection)); + if (!after) { + return { status: 'unrepresentable' }; + } + if (!txn.reserveRaster(rgbaBytes(mapRect))) { + return { status: 'over-budget' }; + } + const paired = this.amendable(live.id, (entry) => entry.requestsMap && entry.inputKey === guard.inputKey); + const resolved = this.resolveBefore(txn, document, live, after, paired); + if ('refusal' in resolved) { + return { status: resolved.refusal }; + } + const { before, pixels: beforePixels } = resolved; + const prepared = ctx.preparePixels(live.id, mapRect, afterPixels.pixels); + const label = paired?.label ?? `Detect ${after.controlType ?? 'control'}`; + const recorded = this.recordOnPublish(); + const result = txn.publish( + label, + this.replaceStep(after, prepared, live), + { + bytes: + (beforePixels ? rgbaBytes(beforePixels.rect) : 0) + rgbaBytes(mapRect) + HISTORY_ENTRY_OVERHEAD_BYTES, + dispose: recorded.dispose, + heldAssetRefs: collectHistoryMediaRefs(before, after), + redo: () => + replayWithPixels(ctx, after.id, afterPixels, (replayed) => this.replaceStep(after, replayed, before)), + undo: () => this.restoreBefore(before, beforePixels, after), + }, + // A background result is not a user edit: it must never change the invocation route. + { origin: 'system', replacing: paired?.token } + ); + if (result.status !== 'committed') { + return { status: guardedResultRefusal(result) }; + } + recorded.record({ + before, + beforePixels, + coalesceKey: undefined, + inputKey: controlDetectionInputKey(after), + label, + layerId: after.id, + requestsMap: true, + token: result.token, + }); + return { status: 'committed' }; + } finally { + txn.end(); + } + } catch (error) { + if (signal?.aborted || (error instanceof Error && error.name === 'AbortError')) { + return { status: 'aborted' }; + } + return { message: error instanceof Error ? error.message : String(error), status: 'failed' }; + } + } + + dispose(): void { + this.last = null; + } + + /** + * Ties the record of an entry about to publish to that entry's lifetime: its `dispose` (eviction, a cleared redo + * stack, replacement by an amending commit) drops the record, so its before-pixels never outlive the entry. + */ + private recordOnPublish(): { dispose: () => void; record: (entry: RecordedEntry) => void } { + let recorded: RecordedEntry | null = null; + return { + dispose: () => { + if (recorded && this.last === recorded) { + this.last = null; + } + recorded = null; + }, + record: (entry) => { + recorded = entry; + this.last = entry; + }, + }; + } + + /** + * The recorded entry a new commit may replace: still the newest step, for this layer, and matching. A record whose + * entry is not on top is dropped with the pixels it holds, trading pairing for memory: if a redo later puts that + * entry back on top, its map records its own entry instead of joining it. + */ + private amendable(layerId: string, matches: (entry: RecordedEntry) => boolean): RecordedEntry | null { + const entry = this.last; + if (!entry) { + return null; + } + if (entry.token !== this.options.ctx.historyTop()) { + this.last = null; + return null; + } + return entry.layerId === layerId && matches(entry) ? entry : null; + } + + /** + * The layer one undo returns to, and the pixels it needs: an amended entry's own before-state, else the live layer. + * Pixels are kept only when the step replaces a source whose pixels are not persisted yet; otherwise the restored + * contract's persisted source reproduces them exactly. + */ + private resolveBefore( + txn: EditTransaction, + document: CanvasDocumentContractV3, + live: CanvasControlLayerContract, + after: CanvasControlLayerContract, + amended: RecordedEntry | null + ): BeforeState { + const before = amended?.before ?? structuredClone(live); + if (amended?.beforePixels) { + return { before, pixels: amended.beforePixels }; + } + if (sameSource(before, after) || !sameSource(before, live) || !this.options.hasPendingPixelWork(live.id)) { + return { before, pixels: null }; + } + const pixels = this.options.captureCache( + live, + document, + (rect) => txn.growHistory(rgbaBytes(rect)) && txn.reserveRaster(rgbaBytes(rect)) + ); + if (pixels === 'not-ready' || pixels === 'over-budget') { + return { refusal: pixels }; + } + // Unpersisted pixels without a cache cannot be restored by undo. + return pixels ? { before, pixels } : { refusal: 'not-ready' }; + } + + /** + * Swaps `layer` in. A source change drops the outgoing source's pending persistence, which would otherwise land + * its old pixels over the new source. + */ + private replaceStep( + layer: CanvasControlLayerContract, + prepared: Parameters[2], + restore: CanvasControlLayerContract, + persist = false + ): EditStep { + return replaceLayerStep(this.options.ctx, layer, prepared, { + beforeInstall: sameSource(layer, restore) ? undefined : () => this.options.discardPersisted(layer.id), + persist, + restore, + }); + } + + private restoreBefore( + before: CanvasControlLayerContract, + pixels: LayerPixels | null, + after: CanvasControlLayerContract + ): void { + const { ctx } = this.options; + if (!pixels) { + ctx.applyStep(this.replaceStep(before, null, after)); + return; + } + // Restored pixels were never persisted, so they persist again. + replayWithPixels(ctx, before.id, pixels, (prepared) => this.replaceStep(before, prepared, after, true)); + } +} diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlViewController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlViewController.ts new file mode 100644 index 00000000000..bca2df91fc7 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/controlViewController.ts @@ -0,0 +1,164 @@ +import type { ControlLayerViewMode, ControlLayerViewOutcome } from '@workbench/canvas-engine/capabilities'; +import type { + CanvasControlLayerContract, + CanvasDocumentContractV3, + CanvasPixelSourceContract, +} from '@workbench/canvas-engine/contracts'; +import type { ControlLayerView } from '@workbench/canvas-engine/engineStores'; +import type { RasterSurface } from '@workbench/canvas-engine/render/raster'; +import type { RasterizeResult } from '@workbench/canvas-engine/render/rasterizers'; +import type { Mat2d, Rect } from '@workbench/canvas-engine/types'; + +import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; +import { IDENTITY_TRANSFORM, layerTransformMatrix } from '@workbench/canvas-engine/transform/transformMath'; + +import type { RasterMemoryLease, RasterMemoryReservationResult } from './rasterMemoryBudgetController'; + +/** A shown view: the kept original's pixels, placed in the layer-local frame of the map. */ +export interface ControlLayerViewState extends Readonly { + readonly surface: RasterSurface; + /** The original's pixels in its own local frame. */ + readonly rect: Rect; + /** Original-local → layer-local (`originalTransform`). */ + readonly matrix: Mat2d; + /** The contract the view was built from; any other identity means the view no longer describes the layer. */ + readonly layer: CanvasControlLayerContract; +} + +export interface ControlViewControllerOptions { + readonly getDocument: () => CanvasDocumentContractV3 | null; + readonly rasterize: ( + source: CanvasPixelSourceContract, + document: CanvasDocumentContractV3, + signal: AbortSignal + ) => Promise; + /** Reserves raster memory for the original's surface while it is decoded. */ + readonly reserve: (bytes: number) => RasterMemoryReservationResult; + /** Accounts the decoded surface as retained pixels until the view releases it. */ + readonly trackDetached: (bytes: number) => RasterMemoryLease; + /** Whether an operation or a generation holds the canvas; views are refused meanwhile. */ + readonly isLocked: () => boolean; + readonly invalidateLayer: (layerId: string) => void; + readonly publish: (view: ControlLayerView | null) => void; +} + +const asControlLayer = ( + document: CanvasDocumentContractV3 | null, + layerId: string +): CanvasControlLayerContract | null => { + const layer = getDocumentLayer(document, layerId); + return layer?.type === 'control' ? layer : null; +}; + +/** RGBA bytes of the original's decoded pixels; null when it has none. */ +const originalBytes = (original: CanvasPixelSourceContract): number | null => { + const image = original.type === 'image' ? original.image : original.bitmap; + return image && image.width > 0 && image.height > 0 ? image.width * image.height * 4 : null; +}; + +/** + * Owns the display-only control-layer view: at most one layer shows its kept original (alone, or under its map), + * decoded under a raster reservation and retained as detached pixels until the view ends. A newer request supersedes + * an older decode. Nothing here touches the document, history or exports; callers end the view whenever the layer + * stops matching it. + */ +export class ControlViewController { + private state: ControlLayerViewState | null = null; + private lease: RasterMemoryLease | null = null; + private pending: { readonly layerId: string; readonly abort: AbortController } | null = null; + + constructor(private readonly options: ControlViewControllerOptions) {} + + get(): ControlLayerViewState | null { + return this.state; + } + + async set(layerId: string, view: ControlLayerViewMode): Promise { + this.pending?.abort.abort(); + this.pending = null; + if (view === 'map') { + this.clear((id) => id === layerId); + return 'shown'; + } + if (this.options.isLocked()) { + return 'locked'; + } + const document = this.options.getDocument(); + const layer = asControlLayer(document, layerId); + const bytes = layer?.original ? originalBytes(layer.original) : null; + if (!document || !layer?.original || bytes === null) { + return 'missing'; + } + // Original and Overlay share one decoded surface. + if (this.state?.layer === layer) { + this.show({ ...this.state, view }, this.lease); + return 'shown'; + } + const reservation = this.options.reserve(bytes); + if (reservation.status !== 'ok') { + return 'over-budget'; + } + const pending = { abort: new AbortController(), layerId }; + this.pending = pending; + try { + const { rect, surface } = await this.options.rasterize(layer.original, document, pending.abort.signal); + if (this.pending !== pending || asControlLayer(this.options.getDocument(), layerId) !== layer) { + return 'stale'; + } + const matrix = layerTransformMatrix(layer.originalTransform ?? IDENTITY_TRANSFORM); + this.show({ layer, layerId, matrix, rect, surface, view }, this.options.trackDetached(bytes)); + return 'shown'; + } catch { + // The map keeps showing either way. + return this.pending === pending ? 'failed' : 'stale'; + } finally { + reservation.lease.release(); + if (this.pending === pending) { + this.pending = null; + } + } + } + + /** Ends views and in-flight requests for the layers `which` names; every layer when omitted. */ + clear(which: (layerId: string) => boolean = () => true): void { + if (this.pending && which(this.pending.layerId)) { + this.pending.abort.abort(); + this.pending = null; + } + const state = this.state; + if (!state || !which(state.layerId)) { + return; + } + this.state = null; + this.lease?.release(); + this.lease = null; + this.options.publish(null); + this.options.invalidateLayer(state.layerId); + } + + /** Ends the view once its layer is gone or is no longer the contract it was built from. */ + reconcile(): void { + const state = this.state; + if (state && asControlLayer(this.options.getDocument(), state.layerId) !== state.layer) { + this.clear(); + } + } + + dispose(): void { + this.clear(); + } + + private show(next: ControlLayerViewState, lease: RasterMemoryLease | null): void { + const previous = this.state; + if (previous && previous.layerId !== next.layerId) { + this.options.invalidateLayer(previous.layerId); + } + if (this.lease !== lease) { + this.lease?.release(); + } + this.state = next; + this.lease = lease; + this.options.publish({ layerId: next.layerId, view: next.view }); + this.options.invalidateLayer(next.layerId); + } +} diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/copyLayerController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/copyLayerController.ts index db1dcd484a7..eba639a4ed2 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/copyLayerController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/copyLayerController.ts @@ -57,8 +57,8 @@ export interface CopyLayerControllerOptions { anchor: CanvasNodeInsertionAnchor ) => boolean; readonly exportBaked: (layerId: string) => Promise; - /** The main model's base and its default control model, for a control copy's adapter. */ - readonly getControlModel: () => { readonly base: string | null; readonly model: string | null }; + /** The main model's base, for a control copy's adapter kind and defaults. */ + readonly getMainModelBase: () => string | null; readonly isGuardCurrent: (guard: LayerExportGuard) => boolean; } @@ -105,7 +105,7 @@ export class CopyLayerController { return { status: 'not-ready' }; } const layer = createControlLayerFrom(sourceLayer, { - ...this.deps.getControlModel(), + base: this.deps.getMainModelBase(), content: { source: sourceLayer.source }, id: ctx.createLayerId(), name: `${sourceLayer.name} copy`, @@ -148,7 +148,7 @@ export class CopyLayerController { const layer = target === 'control' ? createControlLayerFrom(sourceLayer, { - ...this.deps.getControlModel(), + base: this.deps.getMainModelBase(), content: { rect: baked.rect }, id, name, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.test.ts index 2ad09f6f55a..32bf5ebb81b 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.test.ts @@ -23,13 +23,13 @@ const layer: CanvasLayerContract = { const CACHE_RECT: Rect = { height: 50, width: 50, x: -10, y: -10 }; const BBOX: Rect = { height: 20, width: 20, x: 0, y: 0 }; -const createHarness = (historyBytes?: number) => { +const createHarness = (historyBytes?: number, target: CanvasLayerContract = layer) => { const document: CanvasDocumentContractV3 = { background: 'transparent', bbox: BBOX, height: 100, - selectedLayerId: 'paint', - stacks: stacksFrom([layer]), + selectedLayerId: target.id, + stacks: stacksFrom([target]), version: 3, width: 100, }; @@ -60,7 +60,7 @@ const createHarness = (historyBytes?: number) => { isGuardCurrent: () => true, isSupportedSource: () => true, }); - return { ...h, controller, discardPersisted, original: getDocumentLayer(h.document(), 'paint')! }; + return { ...h, controller, discardPersisted, original: getDocumentLayer(h.document(), target.id)! }; }; describe('CropLayerController', () => { @@ -80,6 +80,38 @@ describe('CropLayerController', () => { expect(h.layers.get('paint')?.rect).toEqual(BBOX); }); + it("keeps a control layer's type but drops the original its cropped pixels no longer derive from", async () => { + const depth = { settings: {}, type: 'depth_anything_depth_estimation' }; + const h = createHarness(undefined, { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + controlType: 'depth', + detection: depth, + filter: depth, + id: 'paint', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + original: { image: { height: 50, imageName: 'photo', width: 50 }, type: 'image' }, + originalTransform: { rotation: 0, scaleX: 1, scaleY: 1, x: 2, y: 2 }, + source: { bitmap: { height: 50, imageName: 'map', width: 50 }, offset: { x: -10, y: -10 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }); + + await expect(h.controller.crop('paint')).resolves.toEqual({ status: 'cropped' }); + const cropped = getDocumentLayer(h.document(), 'paint'); + expect(cropped).toMatchObject({ controlType: 'depth', detection: null }); + expect(cropped).not.toHaveProperty('original'); + expect(cropped).not.toHaveProperty('originalTransform'); + expect(cropped).not.toHaveProperty('filter'); + + await h.history.undo(); + expect(getDocumentLayer(h.document(), 'paint')).toEqual(h.original); + }); + it('refuses a crop it could never undo before anything changes', async () => { const h = createHarness(1024); const before = h.document(); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.ts index ff89084cd5d..08f7281ed16 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/cropLayerController.ts @@ -8,6 +8,7 @@ import type { AdjustmentRefusal } from '@workbench/canvas-engine/render/exactAdj import type { RasterBackend, RasterSurface } from '@workbench/canvas-engine/render/raster'; import type { Rect } from '@workbench/canvas-engine/types'; +import { detachControlDerivation } from '@workbench/canvas-engine/document/controlDerivation'; import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; import { renderableSourceOf } from '@workbench/canvas-engine/document/sources'; import { collectHistoryMediaRefs, HISTORY_ENTRY_OVERHEAD_BYTES } from '@workbench/canvas-engine/history/history'; @@ -148,8 +149,9 @@ export class CropLayerController { const { adjustments: _adjustments, ...rest } = before; after = { ...rest, source: paint, transform: identity }; } else if (before.type === 'control') { + // The cropped pixels become the image itself: no original's frame survives the crop. const { filter: _filter, ...rest } = before; - after = { ...rest, source: paint, transform: identity }; + after = detachControlDerivation({ ...rest, source: paint, transform: identity }); } else { after = { ...before, mask: { ...before.mask, bitmap: null, offset: paint.offset }, transform: identity }; } diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.test.ts new file mode 100644 index 00000000000..8dee302f00d --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.test.ts @@ -0,0 +1,130 @@ +import type { + CanvasControlLayerContract, + CanvasDocumentContractV3, + CanvasLayerContract, +} from '@workbench/canvas-engine/contracts'; + +import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; +import { describe, expect, it } from 'vitest'; + +import type { CommitRasterFilterOptions } from './filterResultController'; + +import { FilterResultController } from './filterResultController'; +import { createLayerOperationHarness } from './layerOperationHarness.testing'; + +const DEPTH = { settings: {}, type: 'depth_anything_depth_estimation' }; +const CANNY = { settings: { low_threshold: 100 }, type: 'canny_edge_detection' }; +const RESULT = { height: 16, imageName: 'filtered', width: 16 }; + +const derivedControl: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: 'depth-model', weight: 0.8 }, + blendMode: 'normal', + controlType: 'depth', + detection: DEPTH, + filter: DEPTH, + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + original: { image: { height: 16, imageName: 'photo', width: 16 }, type: 'image' }, + originalTransform: { rotation: 0, scaleX: 1, scaleY: 1, x: 2, y: 2 }, + source: { bitmap: { height: 16, imageName: 'map', width: 16 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, +}; + +const rasterLayer: CanvasLayerContract = { + blendMode: 'normal', + id: 'raster', + isEnabled: true, + isLocked: false, + name: 'Photo', + opacity: 1, + source: { bitmap: null, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'raster', +}; + +const createHarness = () => { + const document: CanvasDocumentContractV3 = { + background: 'transparent', + bbox: { height: 16, width: 16, x: 0, y: 0 }, + height: 16, + selectedLayerId: 'control', + stacks: stacksFrom([rasterLayer, derivedControl]), + version: 3, + width: 16, + }; + const h = createLayerOperationHarness(document); + h.layers.getOrCreateRect('control', { height: 16, width: 16, x: 0, y: 0 }); + const controller = new FilterResultController({ + captureCache: (target) => { + const entry = h.layers.get(target.id); + return entry ? { pixels: h.backend.createSurface(16, 16), rect: { ...entry.rect } } : null; + }, + ctx: h.ctx, + decodeImage: () => + Promise.resolve({ decodedHeight: 16, decodedWidth: 16, status: 'ok', surface: h.backend.createSurface(16, 16) }), + discardPersisted: () => undefined, + getMainModelBase: () => 'sdxl', + needsPixelPersistence: () => true, + }); + const commit = (layerId: string, options: Partial) => + controller.commit({ + filter: CANNY, + guard: { layerId } as CommitRasterFilterOptions['guard'], + image: RESULT, + mode: 'copy', + rect: { height: 16, width: 16, x: 0, y: 0 }, + ...options, + }); + return { ...h, commit }; +}; + +const committedLayer = (h: ReturnType, result: Awaited>) => { + expect(result.status).toBe('committed'); + return getDocumentLayer(h.document(), 'layerId' in result ? result.layerId : null); +}; + +describe('FilterResultController', () => { + it('refuses a control source, whose map comes from detection, and leaves the document alone', async () => { + const h = createHarness(); + const before = h.document(); + + expect(await h.commit('control', { mode: 'replace' })).toEqual({ status: 'unsupported' }); + expect(await h.commit('control', { controlCopy: { controlType: 'edges' }, target: 'control' })).toEqual({ + status: 'unsupported', + }); + expect(await h.commit('control', { target: 'raster' })).toEqual({ status: 'unsupported' }); + expect(h.document()).toBe(before); + }); + + it('copies a raster result to a new control layer, typed by the filter when the caller says so', async () => { + const h = createHarness(); + + const typed = committedLayer( + h, + await h.commit('raster', { controlCopy: { controlType: 'edges' }, target: 'control' }) + ); + expect(typed).toMatchObject({ + controlType: 'edges', + detection: null, + filter: CANNY, + name: 'Photo filtered', + source: { bitmap: RESULT, type: 'paint' }, + type: 'control', + }); + + const untyped = committedLayer( + h, + await h.commit('raster', { controlCopy: { controlType: null }, target: 'control' }) + ); + expect(untyped).toMatchObject({ adapter: { model: null }, filter: CANNY, type: 'control' }); + for (const key of ['controlType', 'detection', 'original', 'originalTransform']) { + expect(untyped).not.toHaveProperty(key); + } + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.ts index 0734757b66c..9a256fa689b 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/filterResultController.ts @@ -1,10 +1,5 @@ import type { CommitRasterFilterOptions, CommitRasterFilterResult } from '@workbench/canvas-engine/capabilities'; -import type { - CanvasControlLayerContract, - CanvasImageRef, - CanvasLayerContract, - CanvasRasterLayerContractV2, -} from '@workbench/canvas-engine/contracts'; +import type { CanvasImageRef, CanvasLayerContract } from '@workbench/canvas-engine/contracts'; import type { LayerMutationControllerOptions } from '@workbench/canvas-engine/controllers/layerMutationController'; import type { DecodeImageResult } from '@workbench/canvas-engine/controllers/rasterController'; import type { PreparedLayerCacheReplacement } from '@workbench/canvas-engine/render/layerCache'; @@ -60,7 +55,6 @@ export interface FilterResultControllerOptions { } ) => Promise; readonly discardPersisted: (layerId: string) => void; - readonly getDefaultControlModel: (base: string | null) => string | null; readonly getMainModelBase: () => string | null; readonly needsPixelPersistence: (layer: CanvasLayerContract) => boolean; } @@ -108,15 +102,17 @@ export class FilterResultController { if (liveLayer.isLocked) { return { status: 'locked' }; } - if (liveLayer.type !== 'raster' && liveLayer.type !== 'control') { - return { status: 'unsupported' }; - } if (!o.ctx.isPermitCurrent(permit) || o.ctx.isGestureActive()) { return { status: 'busy' }; } + // A source that changed type since the guard was taken is stale, not unsupported. if (!o.ctx.isGuardCurrent(options.guard)) { return { status: 'stale' }; } + // Filter reads raster pixels only; a control layer's map comes from detection. + if (liveLayer.type !== 'raster') { + return { status: 'unsupported' }; + } if (options.signal?.aborted) { return { status: 'aborted' }; } @@ -141,13 +137,12 @@ export class FilterResultController { return { status: 'stale' }; } const before = structuredClone(liveLayer); - const after: CanvasLayerContract = - liveLayer.type === 'raster' - ? (() => { - const { adjustments: _adjustments, ...base } = liveLayer; - return structuredClone({ ...base, filter: options.filter, source: paintSource }); - })() - : structuredClone({ ...liveLayer, filter: options.filter, source: paintSource }); + const { adjustments: _adjustments, ...unadjusted } = liveLayer; + const after: CanvasLayerContract = structuredClone({ + ...unadjusted, + filter: options.filter, + source: paintSource, + }); const discard = (): void => o.discardPersisted(liveLayer.id); const prepared = o.ctx.preparePixels(liveLayer.id, rect, pixels); const interrupted = this.interrupted(options); @@ -181,39 +176,21 @@ export class FilterResultController { const layerId = o.ctx.createLayerId(); let copy: CanvasLayerContract; if (options.target === 'control') { - const buildControlBase = (): CanvasControlLayerContract => { - const mainBase = o.getMainModelBase(); - return createControlLayer( - `${liveLayer.name} filtered`, - layerId, - mainBase, - o.getDefaultControlModel(mainBase) - ); - }; - const base = liveLayer.type === 'control' ? structuredClone(liveLayer) : buildControlBase(); - copy = { - ...base, - filter: options.filter, - id: layerId, - name: `${liveLayer.name} filtered`, - source: paintSource, - transform: structuredClone(liveLayer.transform), - }; - } else if (options.target === 'raster' && liveLayer.type === 'control') { + // A typed copy is a map used as-is; an untyped one waits for the user to choose its type. + const controlType = options.controlCopy?.controlType ?? null; copy = { - blendMode: liveLayer.blendMode, + ...createControlLayer({ + base: o.getMainModelBase(), + id: layerId, + name: `${liveLayer.name} filtered`, + source: paintSource, + }), + ...(controlType === null ? {} : { controlType, detection: null }), filter: options.filter, - id: layerId, - isEnabled: true, - isLocked: false, - name: `${liveLayer.name} filtered`, - opacity: liveLayer.opacity, - source: paintSource, transform: structuredClone(liveLayer.transform), - type: 'raster', }; } else { - const { adjustments: _adjustments, ...base } = structuredClone(liveLayer as CanvasRasterLayerContractV2); + const { adjustments: _adjustments, ...base } = structuredClone(liveLayer); copy = { ...base, filter: options.filter, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.test.ts new file mode 100644 index 00000000000..0f2d7015ea6 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.test.ts @@ -0,0 +1,92 @@ +import type { CommitGeneratedImageOptions } from '@workbench/canvas-engine/capabilities'; +import type { CanvasControlLayerContract, CanvasDocumentContractV3 } from '@workbench/canvas-engine/contracts'; + +import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; +import { describe, expect, it } from 'vitest'; + +import { GeneratedResultController } from './generatedResultController'; +import { createLayerOperationHarness } from './layerOperationHarness.testing'; + +const DEPTH = { settings: {}, type: 'depth_anything_depth_estimation' }; + +const derivedControl: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + controlType: 'depth', + detection: DEPTH, + filter: DEPTH, + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + original: { image: { height: 16, imageName: 'photo', width: 16 }, type: 'image' }, + source: { bitmap: { height: 16, imageName: 'map', width: 16 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 3, y: 4 }, + type: 'control', + withTransparencyEffect: true, +}; + +const createHarness = () => { + const document: CanvasDocumentContractV3 = { + background: 'transparent', + bbox: { height: 16, width: 16, x: 0, y: 0 }, + height: 16, + selectedLayerId: 'control', + stacks: stacksFrom([derivedControl]), + version: 3, + width: 16, + }; + const h = createLayerOperationHarness(document); + h.layers.getOrCreateRect('control', { height: 16, width: 16, x: 0, y: 0 }); + const controller = new GeneratedResultController({ + captureCache: (target) => { + const entry = h.layers.get(target.id); + return entry ? { pixels: h.backend.createSurface(16, 16), rect: { ...entry.rect } } : null; + }, + clearPreview: () => undefined, + ctx: h.ctx, + decodeImage: () => + Promise.resolve({ decodedHeight: 16, decodedWidth: 16, status: 'ok', surface: h.backend.createSurface(16, 16) }), + discardPersisted: () => undefined, + getMainModelBase: () => 'sdxl', + needsPixelPersistence: () => true, + }); + const commit = (target: CommitGeneratedImageOptions['target']) => + controller.commit({ + guard: { layerId: 'control' } as CommitGeneratedImageOptions['guard'], + image: { height: 16, imageName: 'workflow-result', width: 16 }, + origin: { x: 0, y: 0 }, + target, + }); + return { ...h, commit }; +}; + +describe('GeneratedResultController control derivation', () => { + it('replacing a derived control map keeps the result as the image itself', async () => { + const h = createHarness(); + + await expect(h.commit('replace')).resolves.toEqual({ layerId: 'control', status: 'committed' }); + + const replaced = getDocumentLayer(h.document(), 'control'); + expect(replaced).toMatchObject({ controlType: 'depth', detection: null }); + expect(replaced).not.toHaveProperty('original'); + // Not map-only: the result itself is what a later detection runs on. + expect(replaced).not.toHaveProperty('filter'); + await h.history.undo(); + expect(getDocumentLayer(h.document(), 'control')).toEqual(derivedControl); + }); + + it('copies a result to an untyped control layer', async () => { + const h = createHarness(); + + const result = await h.commit('copy-control'); + + const copy = getDocumentLayer(h.document(), 'layerId' in result ? result.layerId : null); + expect(copy?.type).toBe('control'); + for (const key of ['controlType', 'detection', 'original', 'originalTransform']) { + expect(copy).not.toHaveProperty(key); + } + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.ts index 7a02709622f..d9e91a28374 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/generatedResultController.ts @@ -5,6 +5,7 @@ import type { DecodeImageResult } from '@workbench/canvas-engine/controllers/ras import type { PreparedLayerCacheReplacement } from '@workbench/canvas-engine/render/layerCache'; import type { LayerTransform } from '@workbench/canvas-engine/transform/transformMath'; +import { detachControlDerivation } from '@workbench/canvas-engine/document/controlDerivation'; import { getDocumentLayer, getDocumentLeaves } from '@workbench/canvas-engine/document/documentIndex'; import { createControlLayer, nextControlLayerName } from '@workbench/canvas-engine/document/layerFactories'; import { collectHistoryMediaRefs, HISTORY_ENTRY_OVERHEAD_BYTES } from '@workbench/canvas-engine/history/history'; @@ -44,7 +45,6 @@ export interface GeneratedResultControllerOptions { options: { signal?: AbortSignal; isCurrent?: () => boolean } ) => Promise; readonly discardPersisted: (layerId: string) => void; - readonly getDefaultControlModel: (base: string | null) => string | null; readonly getMainModelBase: () => string | null; readonly needsPixelPersistence: (layer: CanvasLayerContract) => boolean; } @@ -131,7 +131,9 @@ export class GeneratedResultController { const { adjustments: _adjustments, ...base } = structuredClone(liveLayer); after = { ...base, source, transform: identityTransform }; } else { - after = { ...structuredClone(liveLayer), source, transform: identityTransform }; + // A workflow result replaces the pixels outright: no filter produced them and no kept original derives them. + const { filter: _filter, ...base } = structuredClone(liveLayer); + after = detachControlDerivation({ ...base, source, transform: identityTransform }); } const clearPreview = (): void => o.clearPreview(liveLayer.id); const publishAfter = (): void => { @@ -170,14 +172,13 @@ export class GeneratedResultController { const layerId = o.ctx.createLayerId(); const selectedLayerId = document.selectedLayerId; const buildControlCopy = (): CanvasLayerContract => { - const mainBase = o.getMainModelBase(); return { - ...createControlLayer( - options.copyLayerName ?? nextControlLayerName(getDocumentLeaves(document).map((layer) => layer.name)), - layerId, - mainBase, - o.getDefaultControlModel(mainBase) - ), + ...createControlLayer({ + base: o.getMainModelBase(), + id: layerId, + name: + options.copyLayerName ?? nextControlLayerName(getDocumentLeaves(document).map((layer) => layer.name)), + }), source, transform: identityTransform, }; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerController.test.ts index 9205d03d0e8..534607619cb 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerController.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerController.test.ts @@ -95,7 +95,7 @@ describe('LayerController', () => { commitLayerCopy: () => false, ctx: {} as never, exportBaked: vi.fn(), - getControlModel: () => ({ base: null, model: null }), + getMainModelBase: () => null, isGuardCurrent: () => true, }; const newRasterLayer = { backend: {} as never, ctx: {} as never, layers: {} as never, selection: {} as never }; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerMutationController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerMutationController.ts index 80cb3a35f4a..2b789c288dd 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerMutationController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerMutationController.ts @@ -88,8 +88,8 @@ export interface LayerMutationControllerOptions { readonly backend: RasterBackend; /** A layer's pixels with its own adjustments and transform baked in, disabled layers included. */ readonly exportBaked: (layerId: string) => Promise; - /** The main model's base and its default control model, for a converted control layer's adapter. */ - readonly getControlModel: () => { readonly base: string | null; readonly model: string | null }; + /** The main model's base, for a converted control layer's adapter kind and defaults. */ + readonly getMainModelBase: () => string | null; readonly isGuardCurrent: (guard: LayerExportGuard) => boolean; /** Copies a layer's live cache; `admit` accounts its bytes first and refuses the copy as `over-budget`. */ readonly captureCache: ( @@ -500,7 +500,7 @@ export class LayerMutationController { } if (baked.status === 'empty') { const plain = createControlLayerFrom(current, { - ...o.getControlModel(), + base: o.getMainModelBase(), content: { source: current.source }, id: current.id, name: current.name, @@ -543,7 +543,7 @@ export class LayerMutationController { const pixels = o.backend.createSurface(baked.rect.width, baked.rect.height); pixels.ctx.drawImage(baked.surface.canvas, 0, 0); const after = createControlLayerFrom(current, { - ...o.getControlModel(), + base: o.getMainModelBase(), content: { rect: baked.rect }, id: current.id, name: current.name, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerOperationHarness.testing.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerOperationHarness.testing.ts index 7272c725c14..6e64e04175e 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerOperationHarness.testing.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/layerOperationHarness.testing.ts @@ -1,5 +1,5 @@ import type { CanvasDocumentContractV3 } from '@workbench/canvas-engine/contracts'; -import type { CanvasProjectMutation } from '@workbench/canvas-engine/mutationContracts'; +import type { CanvasEditIntent, CanvasProjectMutation } from '@workbench/canvas-engine/mutationContracts'; import type { PreparedLayerCacheReplacement } from '@workbench/canvas-engine/render/layerCache'; import type { RasterBackend } from '@workbench/canvas-engine/render/raster'; @@ -15,6 +15,8 @@ export interface LayerOperationHarnessOptions { /** History byte budget; an edit whose entry can never fit is refused. */ readonly historyBytes?: number; readonly gestureActive?: boolean; + /** Observes where the context ends an open structural preview (`getDocument`, `begin`). */ + readonly endStructuralPreview?: () => void; /** Real canvases for pixel checks in browser tests; the stub backend otherwise. */ readonly backend?: RasterBackend; } @@ -37,6 +39,11 @@ export const createLayerOperationHarness = ( }); const listeners = new Set<() => void>(); const dispatched: CanvasProjectMutation[] = []; + /** User edits routed to the invocation controller; system-origin publications never appear here. */ + const committedEdits: CanvasEditIntent[] = []; + /** An operation's edit lease: while held, edits without the operation's owner are refused. */ + let editingLocked = false; + const lockListeners = new Set<() => void>(); const installed: PreparedLayerCacheReplacement[] = []; const state = { gestureActive: options.gestureActive ?? false, @@ -48,7 +55,9 @@ export const createLayerOperationHarness = ( let reserved = 0; let nextId = 0; const ctx = createCanvasMutationContext({ - commitEdit: () => undefined, + commitEdit: (intent) => { + committedEdits.push(intent); + }, createLayerId: () => `new-${(nextId += 1)}`, dispatch: (action) => { dispatched.push(action); @@ -65,7 +74,14 @@ export const createLayerOperationHarness = ( return true; }, editOwner: Symbol('harness'), - editingLocked: { get: () => false, subscribe: () => () => undefined }, + endStructuralPreview: options.endStructuralPreview, + editingLocked: { + get: () => editingLocked, + subscribe: (listener) => { + lockListeners.add(listener); + return () => lockListeners.delete(listener); + }, + }, getDocument: () => project.canvas.document, getReducerDocument: () => project.canvas.document, history, @@ -103,6 +119,7 @@ export const createLayerOperationHarness = ( }); return { backend, + committedEdits, ctx, dispatched, document: () => project.canvas.document, @@ -110,6 +127,12 @@ export const createLayerOperationHarness = ( installed, layers, reservedBytes: () => reserved, + setEditingLocked: (locked: boolean) => { + editingLocked = locked; + for (const listener of lockListeners) { + listener(); + } + }, state, }; }; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.test.ts index 43b3f6f4b02..ad4879e2c71 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.test.ts @@ -1,10 +1,56 @@ -import type { BitmapStore } from '@workbench/canvas-engine/document/bitmapStore'; +import type { CanvasLayerSourceContract } from '@workbench/canvas-engine/contracts'; +import type { BitmapStore, BitmapStoreTimers } from '@workbench/canvas-engine/document/bitmapStore'; +import type { CanvasImageUploadResult } from '@workbench/canvas-engine/document/imageUpload'; +import { createBitmapStore } from '@workbench/canvas-engine/document/bitmapStore'; +import { createTestStubRasterBackend } from '@workbench/canvas-engine/render/raster.testStub'; import { describe, expect, it, vi } from 'vitest'; import { PersistenceController } from './persistenceController'; -const createStore = (): BitmapStore => ({ +/** Debounce timers that never fire, so only a barrier flushes. */ +const idleTimers: BitmapStoreTimers = { clearTimeout: () => undefined, setTimeout: () => 0 }; + +/** A real bitmap store over two paint layers whose uploads the test resolves. */ +const createPersistence = () => { + const surface = createTestStubRasterBackend().createSurface(4, 4); + const sources = new Map([ + ['a', { bitmap: null, type: 'paint' }], + ['b', { bitmap: null, type: 'paint' }], + ]); + const uploads: { content: string; resolve: (result: CanvasImageUploadResult) => void }[] = []; + const store = createBitmapStore({ + dispatch: (action) => { + if (action.type === 'updateCanvasLayerSource') { + sources.set(action.id, action.source); + } + return true; + }, + encodeSurface: () => Promise.resolve(new Blob([`pixels-${String(uploads.length)}`])), + getLayerSource: (layerId) => sources.get(layerId) ?? null, + getLayerSurface: () => ({ offset: { x: 0, y: 0 }, surface }), + hashBlob: (blob) => blob.text(), + timers: idleTimers, + uploadImage: (blob) => + blob.text().then( + (text) => + new Promise((resolve) => { + uploads.push({ content: text, resolve }); + }) + ), + }); + const controller = new PersistenceController(store); + /** Settles every upload started so far. */ + const settleUploads = async (): Promise => { + await vi.waitFor(() => expect(uploads.length).toBeGreaterThan(0)); + for (const upload of uploads.splice(0)) { + upload.resolve({ height: 4, imageName: `${upload.content}.png`, width: 4 }); + } + }; + return { controller, settleUploads, sources, store, uploads }; +}; + +const createStoreDouble = (): BitmapStore => ({ discardLayer: vi.fn(), dispose: vi.fn(), flushPendingUploads: vi.fn(() => Promise.resolve()), @@ -18,7 +64,7 @@ const createStore = (): BitmapStore => ({ describe('PersistenceController', () => { it('provides a flush barrier and idempotent disposal', async () => { - const store = createStore(); + const store = createStoreDouble(); const controller = new PersistenceController(store); await controller.flush(); expect(store.flushPendingUploads).toHaveBeenCalledOnce(); @@ -26,4 +72,57 @@ describe('PersistenceController', () => { controller.dispose(); expect(store.dispose).toHaveBeenCalledOnce(); }); + + describe('flushLayer', () => { + it('reports a layer without unsaved pixels as clean', async () => { + const p = createPersistence(); + + await expect(p.controller.flushLayer('a')).resolves.toBe('clean'); + expect(p.uploads).toEqual([]); + }); + + it('persists only the named layer', async () => { + const p = createPersistence(); + p.store.markLayerDirty('b'); + p.store.markLayerDirty('a'); + + const flushed = p.controller.flushLayer('a'); + await p.settleUploads(); + + await expect(flushed).resolves.toBe('flushed'); + expect(p.sources.get('a')).toMatchObject({ bitmap: { imageName: 'pixels-0.png' } }); + expect(p.store.hasPendingWork('a')).toBe(false); + expect(p.store.hasPendingWork('b')).toBe(true); + expect(p.sources.get('b')).toEqual({ bitmap: null, type: 'paint' }); + }); + + it('stops waiting when aborted', async () => { + const p = createPersistence(); + p.store.markLayerDirty('a'); + const abort = new AbortController(); + + const flushed = p.controller.flushLayer('a', { signal: abort.signal }); + await vi.waitFor(() => expect(p.uploads).toHaveLength(1)); + abort.abort(); + + await expect(flushed).resolves.toBe('aborted'); + await expect(p.controller.flushLayer('a', { signal: AbortSignal.abort() })).resolves.toBe('aborted'); + }); + + it('stops waiting for a suspended layer when aborted, and flushes it once released', async () => { + const p = createPersistence(); + p.store.markLayerDirty('a'); + const release = p.store.suspendLayer('a'); + const abort = new AbortController(); + + const flushed = p.controller.flushLayer('a', { signal: abort.signal }); + abort.abort(); + await expect(flushed).resolves.toBe('aborted'); + + release(); + const next = p.controller.flushLayer('a'); + await p.settleUploads(); + await expect(next).resolves.toBe('flushed'); + }); + }); }); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.ts index ede6df96c6d..15b0b92152c 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/persistenceController.ts @@ -1,5 +1,15 @@ import type { BitmapStore, FlushPendingUploadsOptions } from '@workbench/canvas-engine/document/bitmapStore'; +/** + * How a one-layer flush ended: its pixels persisted, there was nothing to persist, they could not be, or the wait + * aborted. + */ +export type LayerFlushResult = 'flushed' | 'clean' | 'failed' | 'aborted'; + +export interface LayerFlushOptions { + readonly signal?: AbortSignal; +} + export class PersistenceController { readonly store: BitmapStore; private disposed = false; @@ -12,6 +22,25 @@ export class PersistenceController { return this.store.flushPendingUploads(options); } + /** + * The paint barrier scoped to one layer: waits only for that layer's dirty and in-flight uploads, including while an + * open edit holds its pixels. + */ + async flushLayer(layerId: string, { signal }: LayerFlushOptions = {}): Promise { + if (signal?.aborted) { + return 'aborted'; + } + if (!this.store.hasPendingWork(layerId)) { + return 'clean'; + } + try { + await this.store.flushPendingUploads({ layerIds: [layerId], signal }); + return 'flushed'; + } catch (error) { + return signal?.aborted || (error instanceof Error && error.name === 'AbortError') ? 'aborted' : 'failed'; + } + } + dispose(): void { if (this.disposed) { return; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.test.ts index 0f49ad5bab3..90b79e58183 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.test.ts @@ -1,13 +1,15 @@ -import type { CanvasLayerContract } from '@workbench/canvas-engine/contracts'; +import type { CanvasControlLayerContract, CanvasLayerContract } from '@workbench/canvas-engine/contracts'; import type { TransformSession } from '@workbench/canvas-engine/engineStores'; import type { CanvasProjectMutation } from '@workbench/canvas-engine/mutationContracts'; import { createCanvasMutationContext } from '@workbench/canvas-engine/controllers/mutationContext'; import { documentFrom, layerContract } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { restoreControlOriginal, withDetectedMap } from '@workbench/canvas-engine/document/controlDerivation'; import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; import { createHistory } from '@workbench/canvas-engine/history/history'; import { createLayerCacheStore } from '@workbench/canvas-engine/render/layerCache'; import { createTestStubRasterBackend } from '@workbench/canvas-engine/render/raster.testStub'; +import { composeLayerTransforms } from '@workbench/canvas-engine/transform/transformMath'; import { applyCanvasProjectMutation } from '@workbench/canvasProjectMutations'; import { createInitialWorkbenchState } from '@workbench/workbenchState'; import { describe, expect, it, vi } from 'vitest'; @@ -184,4 +186,78 @@ describe('TransformEditingController', () => { expect(h.history.entries().past).toEqual(['Transform layer']); expect(h.calls.restoreCache).toHaveBeenCalledOnce(); }); + + it("moves a detected control layer's original with the baked map, in the same undo step", async () => { + const depth = { settings: {}, type: 'depth_anything_depth_estimation' }; + const originalTransform = { rotation: 0, scaleX: 1, scaleY: 1, x: 2, y: 3 }; + const detected: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + controlType: 'depth', + detection: depth, + filter: depth, + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + original: { image: { height: 4, imageName: 'photo', width: 4 }, type: 'image' }, + originalTransform, + source: { bitmap: { height: 4, imageName: 'map.png', width: 4 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 10, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }; + const applied = { rotation: 0, scaleX: 2, scaleY: 2, x: 12, y: 4 }; + const h = createHarness(detected); + h.controller.begin('control'); + h.controller.update(applied); + h.controller.apply(); + + const baked = h.layer() as CanvasControlLayerContract; + expect(baked.transform).toEqual({ rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }); + expect(baked).toMatchObject({ controlType: 'depth', detection: depth, original: detected.original }); + // Re-detection and "None" both land the original where the applied transform put it. + const placed = composeLayerTransforms(applied, originalTransform); + expect(restoreControlOriginal(baked).transform).toEqual(placed); + const map = { image: { height: 4, imageName: 'next-map', width: 4 }, rect: { height: 4, width: 4, x: 0, y: 0 } }; + expect(withDetectedMap(baked, map, depth)?.transform).toEqual(placed); + expect(h.history.entries().past).toEqual(['Transform layer']); + + await h.history.undo(); + expect(h.layer()).toEqual(detected); + }); + + it('holds the original a shearing bake detaches, so undo can restore it', async () => { + const depth = { settings: {}, type: 'depth_anything_depth_estimation' }; + const detected: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + controlType: 'depth', + detection: depth, + filter: depth, + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + original: { image: { height: 4, imageName: 'photo', width: 4 }, type: 'image' }, + originalTransform: { rotation: Math.PI / 4, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + source: { bitmap: { height: 4, imageName: 'map.png', width: 4 }, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }; + const h = createHarness(detected); + h.controller.begin('control'); + h.controller.update({ rotation: 0, scaleX: 2, scaleY: 1, x: 0, y: 0 }); + h.controller.apply(); + + expect(h.layer()).not.toHaveProperty('original'); + expect(h.layer()).toMatchObject({ detection: null }); + expect(h.history.heldAssetRefs().images).toContain('photo'); + + await h.history.undo(); + expect(h.layer()).toEqual(detected); + }); }); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.ts index 3708e9ef7a2..45e9e2abb7c 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/controllers/transformEditingController.ts @@ -1,5 +1,5 @@ import type { CanvasEditRefusal, StructuralCommitResult } from '@workbench/canvas-engine/capabilities'; -import type { CanvasDocumentContractV3 } from '@workbench/canvas-engine/contracts'; +import type { CanvasDocumentContractV3, CanvasLayerContract } from '@workbench/canvas-engine/contracts'; import type { TransformSession } from '@workbench/canvas-engine/engineStores'; import type { CanvasProjectMutation } from '@workbench/canvas-engine/mutationContracts'; import type { LayerCacheEntry } from '@workbench/canvas-engine/render/layerCache'; @@ -8,22 +8,27 @@ import type { LayerTransform } from '@workbench/canvas-engine/transform/transfor import type { Rect } from '@workbench/canvas-engine/types'; import { lookupDocumentLeaf } from '@workbench/canvas-engine/document-model/documentModel'; +import { rebaseControlOriginal } from '@workbench/canvas-engine/document/controlDerivation'; import { getDocumentLayer } from '@workbench/canvas-engine/document/documentIndex'; import { isLeafEditable } from '@workbench/canvas-engine/document/layerEligibility'; import { isRenderableLayer } from '@workbench/canvas-engine/document/sources'; -import { HISTORY_ENTRY_OVERHEAD_BYTES, NO_HELD_ASSET_REFS } from '@workbench/canvas-engine/history/history'; +import { + collectHistoryMediaRefs, + HISTORY_ENTRY_OVERHEAD_BYTES, + NO_HELD_ASSET_REFS, +} from '@workbench/canvas-engine/history/history'; import { isEmpty, roundOut, transformBounds } from '@workbench/canvas-engine/math/rect'; import { hittableLayerSize } from '@workbench/canvas-engine/tools/moveHitTest'; import { bakeMatrix } from '@workbench/canvas-engine/transform/transformMath'; import type { CanvasMutationContext, EditStep } from './mutationContext'; -import { rgbaBytes, withReplayReservation } from './editSteps'; +import { replaceLayerStep, rgbaBytes, withReplayReservation } from './editSteps'; export interface TransformEditingControllerOptions { readonly session: { get(): TransformSession | null; set(value: TransformSession | null): void }; readonly backend: RasterBackend; - readonly ctx: Pick; + readonly ctx: Pick; readonly getDocument: () => CanvasDocumentContractV3 | null; readonly getCache: (layerId: string) => LayerCacheEntry | null; readonly setOverride: (layerId: string, transform: LayerTransform | null) => void; @@ -42,6 +47,19 @@ export interface TransformEditingControllerOptions { const IDENTITY: LayerTransform = { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }; +/** + * What a bake step changes on the layer: only its transform, or the whole contract when more than the transform moves + * with the pixels (a control layer's kept original). + */ +type BakeChange = + | { readonly kind: 'transform'; readonly from: LayerTransform; readonly to: LayerTransform } + | { readonly kind: 'contract'; readonly from: CanvasLayerContract; readonly to: CanvasLayerContract }; + +const reversed = (change: BakeChange): BakeChange => + change.kind === 'transform' + ? { from: change.to, kind: 'transform', to: change.from } + : { from: change.to, kind: 'contract', to: change.from }; + const isRefusal = (status: string): status is Extract => status === 'busy' || status === 'gesture-active' || status === 'not-ready' || status === 'over-budget'; @@ -102,8 +120,12 @@ export class TransformEditingController { this.deps.invalidate({ layers: [session.layerId], overlay: true }); } - /** A step that sets the layer's transform and installs matching whole-layer pixels, or rolls the transform back. */ - private bakeStep(layerId: string, from: LayerTransform, to: LayerTransform, rect: Rect, pixels: ImageData): EditStep { + /** A step that makes `change` and installs matching whole-layer pixels, or rolls the change back. */ + private bakeStep(layerId: string, change: BakeChange, rect: Rect, pixels: ImageData): EditStep { + const install = (): void => this.deps.restoreCache(layerId, rect, pixels); + if (change.kind === 'contract') { + return { ...replaceLayerStep(this.deps.ctx, change.to, null, { restore: change.from }), install }; + } const hasTransform = (transform: LayerTransform) => (document: CanvasDocumentContractV3 | null): boolean => { @@ -111,12 +133,12 @@ export class TransformEditingController { return !!current && unchanged(current.transform, transform); }; return { - accepted: hasTransform(to), - install: () => this.deps.restoreCache(layerId, rect, pixels), - mutation: { id: layerId, patch: { transform: to }, type: 'updateCanvasLayer' }, + accepted: hasTransform(change.to), + install, + mutation: { id: layerId, patch: { transform: change.to }, type: 'updateCanvasLayer' }, rollback: { - mutation: { id: layerId, patch: { transform: from }, type: 'updateCanvasLayer' }, - restored: hasTransform(from), + mutation: { id: layerId, patch: { transform: change.from }, type: 'updateCanvasLayer' }, + restored: hasTransform(change.from), }, }; } @@ -190,12 +212,21 @@ export class TransformEditingController { context.drawImage(cache.surface.canvas, beforeRect.x, beforeRect.y); context.setTransform(1, 0, 0, 1, 0, 0); const after = context.getImageData(0, 0, afterRect.width, afterRect.height); - const oldTransform = { ...session.startTransform }; - const forward = this.bakeStep(layer.id, oldTransform, IDENTITY, afterRect, after); - const backward = this.bakeStep(layer.id, IDENTITY, oldTransform, beforeRect, before); + // A kept control original moves with the baked pixels (detaching when its frame would shear), in the same step. + const rebased = + layer.type === 'control' && layer.original !== undefined + ? rebaseControlOriginal({ ...layer, transform: { ...IDENTITY } }, session.transform) + : null; + const change: BakeChange = rebased + ? { from: layer, kind: 'contract', to: rebased } + : { from: { ...session.startTransform }, kind: 'transform', to: IDENTITY }; + const forward = this.bakeStep(layer.id, change, afterRect, after); + const backward = this.bakeStep(layer.id, reversed(change), beforeRect, before); const result = txn.publish('Transform layer', forward, { bytes: before.data.byteLength + after.data.byteLength + HISTORY_ENTRY_OVERHEAD_BYTES, - heldAssetRefs: NO_HELD_ASSET_REFS, + // A contract bake that detaches drops the original from the document; undo restores it, so the entry holds it. + heldAssetRefs: + change.kind === 'contract' ? collectHistoryMediaRefs(change.from, change.to) : NO_HELD_ASSET_REFS, redo: () => this.replay(forward, afterRect), undo: () => this.replay(backward, beforeRect), }); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/bitmapStore.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/bitmapStore.ts index 99341dd82af..dec3c0386c3 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/bitmapStore.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/bitmapStore.ts @@ -53,6 +53,10 @@ export interface FlushPendingUploadsOptions { * that the user cannot see through pass false and fail fast instead. */ readonly waitForHeldPixels?: boolean; + /** Limits the barrier to these layers' dirty and in-flight work; absent means every layer. */ + readonly layerIds?: readonly string[]; + /** Stops waiting with an `AbortError`; uploads already started still settle. */ + readonly signal?: AbortSignal; } /** Injectable timer seam (defaults to the global timers). */ @@ -140,7 +144,7 @@ export interface BitmapStore { suspendLayer(layerId: string): () => void; /** Cancels pending persistence and invalidates an in-flight result for one layer. */ discardLayer(layerId: string): void; - /** Flushes every dirty layer immediately and resolves once all in-flight uploads settle. */ + /** Flushes dirty layers (all, or `options.layerIds`) now and resolves once their in-flight uploads settle. */ flushPendingUploads(options?: FlushPendingUploadsOptions): Promise; /** * Detects the most recently applied paint ref so the engine skips self-echo rasterization. Different refs @@ -227,9 +231,20 @@ export const createBitmapStore = (deps: BitmapStoreDeps): BitmapStore => { resolve(); } }; - const waitForSuspensionChange = (): Promise => - new Promise((resolve) => { - suspensionWaiters.add(resolve); + const abortError = (): DOMException => new DOMException('Canvas pixel persistence wait aborted.', 'AbortError'); + /** Resolves on the next suspension change; an abort rejects and drops the waiter. */ + const waitForSuspensionChange = (signal?: AbortSignal): Promise => + new Promise((resolve, reject) => { + const onAbort = (): void => { + suspensionWaiters.delete(waiter); + reject(abortError()); + }; + const waiter = (): void => { + signal?.removeEventListener('abort', onAbort); + resolve(); + }; + suspensionWaiters.add(waiter); + signal?.addEventListener('abort', onAbort, { once: true }); }); const clearTimer = (layerId: string): void => { @@ -665,24 +680,50 @@ export const createBitmapStore = (deps: BitmapStoreDeps): BitmapStore => { /** Safety net against a genuine infinite loop; real barrier calls settle in a handful of rounds. */ const MAX_BARRIER_ITERATIONS = 10_000; - const flushPendingUploads = async ({ waitForHeldPixels = true }: FlushPendingUploadsOptions = {}): Promise => { + /** Settles with `promise`, or rejects with an `AbortError` once `signal` aborts. */ + const untilAborted = (promise: Promise, signal: AbortSignal | undefined): Promise => { + if (!signal) { + return promise; + } + return new Promise((resolve, reject) => { + const onAbort = (): void => reject(abortError()); + if (signal.aborted) { + onAbort(); + return; + } + signal.addEventListener('abort', onAbort, { once: true }); + promise.then(resolve, reject).finally(() => signal.removeEventListener('abort', onAbort)); + }); + }; + + const flushPendingUploads = async ({ + layerIds, + signal, + waitForHeldPixels = true, + }: FlushPendingUploadsOptions = {}): Promise => { // Flush dirty layers and await in-flight work until newer strokes also persist. Do not retry failures within // this barrier; poll deferred ownership until pixels are released, unless the caller cannot wait. + const scope = layerIds ? new Set(layerIds) : null; + const inScope = (layerId: string): boolean => scope === null || scope.has(layerId); const blockedThisBarrier = new Set(); for (let iteration = 0; iteration < MAX_BARRIER_ITERATIONS; iteration += 1) { - const toFlush = Array.from(dirty).filter((layerId) => !blockedThisBarrier.has(layerId) && !isSuspended(layerId)); + if (signal?.aborted) { + throw abortError(); + } + const scopedDirty = Array.from(dirty).filter(inScope); + const toFlush = scopedDirty.filter((layerId) => !blockedThisBarrier.has(layerId) && !isSuspended(layerId)); for (const layerId of toFlush) { clearTimer(layerId); void runFlush(layerId); } - const ops = [...inFlight.values()]; + const ops = [...inFlight].filter(([layerId]) => inScope(layerId)).map(([, op]) => op); if (ops.length === 0) { - const suspendedLayerIds = Array.from(dirty).filter((layerId) => isSuspended(layerId)); + const suspendedLayerIds = scopedDirty.filter((layerId) => isSuspended(layerId)); if (suspendedLayerIds.length > 0) { if (!waitForHeldPixels) { throw new BitmapPersistenceError(suspendedLayerIds, 'held'); } - await waitForSuspensionChange(); + await waitForSuspensionChange(signal); continue; } const unpersistedLayerIds = Array.from(blockedThisBarrier).filter( @@ -693,7 +734,7 @@ export const createBitmapStore = (deps: BitmapStoreDeps): BitmapStore => { } return; } - await Promise.all(ops); + await untilAborted(Promise.all(ops), signal); const deferredLayerIds: string[] = []; for (const layerId of toFlush) { if (dirty.has(layerId) && dirtyReason.get(layerId) === 'failure') { @@ -706,7 +747,7 @@ export const createBitmapStore = (deps: BitmapStoreDeps): BitmapStore => { if (!waitForHeldPixels) { throw new BitmapPersistenceError(deferredLayerIds, 'held'); } - await sleep(DEFAULT_DEFERRED_RETRY_MS); + await untilAborted(sleep(DEFAULT_DEFERRED_RETRY_MS), signal); } } throw new Error('Canvas pixel persistence barrier exceeded its iteration limit.'); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.test.ts new file mode 100644 index 00000000000..0e95efd6e42 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.test.ts @@ -0,0 +1,264 @@ +import type { CanvasControlLayerContract } from '@workbench/canvas-engine/contracts'; + +import { applyToPoint } from '@workbench/canvas-engine/math/mat2d'; +import { layerTransformMatrix } from '@workbench/canvas-engine/transform/transformMath'; +import { describe, expect, it } from 'vitest'; + +import { + controlDetectionInputKey, + detachControlDerivation, + getControlDetectionInput, + isControlMapCurrent, + needsControlDetection, + rebaseControlOriginal, + restoreControlOriginal, + withDetectedMap, +} from './controlDerivation'; +import { createControlLayer } from './layerFactories'; + +const image = (imageName: string) => ({ height: 64, imageName, width: 64 }); +const photo = { image: image('photo.png'), type: 'image' } as const; +const map = { bitmap: image('map.png'), offset: { x: 0, y: 0 }, type: 'paint' } as const; +const canny = { settings: { high_threshold: 200, low_threshold: 100 }, type: 'canny_edge_detection' }; +const IDENTITY = { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }; + +const layer = (patch: Partial = {}): CanvasControlLayerContract => ({ + ...createControlLayer({ id: 'c1', name: 'Control Layer 1' }), + ...patch, +}); +/** A typed layer whose photo was detected into `map`. */ +const detected = (patch: Partial = {}) => + layer({ controlType: 'edges', detection: canny, filter: canny, original: photo, source: map, ...patch }); + +describe('control derivation state', () => { + it.each([ + ['an empty legacy layer', layer(), { current: true, input: layer().source, needs: false }], + [ + 'a legacy filtered map (map-only)', + layer({ filter: canny, source: map }), + { current: true, input: null, needs: false }, + ], + [ + 'a typed layer using its image as-is', + layer({ controlType: 'depth', detection: null, source: photo }), + { current: true, input: photo, needs: false }, + ], + [ + 'a typed photo waiting for detection', + layer({ controlType: 'edges', detection: canny, source: photo }), + { current: false, input: photo, needs: true }, + ], + [ + 'a typed empty paint layer', + layer({ controlType: 'edges', detection: canny }), + { current: false, input: layer().source, needs: false }, + ], + ['a detected layer', detected(), { current: true, input: photo, needs: false }], + [ + 'a hand-painted detected map', + detected({ source: { ...map, bitmap: image('painted.png') } }), + { current: true, input: photo, needs: false }, + ], + [ + 'a detected layer whose detector settings changed', + detected({ detection: { ...canny, settings: { ...canny.settings, low_threshold: 50 } } }), + { current: false, input: photo, needs: true }, + ], + [ + 'a typed map-only layer (filter copy) choosing a detector', + layer({ controlType: 'edges', detection: canny, filter: canny, source: map }), + { current: false, input: null, needs: false }, + ], + ])('reads %s', (_label, subject, expected) => { + expect(getControlDetectionInput(subject)).toEqual(expected.input); + expect(isControlMapCurrent(subject)).toBe(expected.current); + expect(needsControlDetection(subject)).toBe(expected.needs); + }); + + it('compares the producing filter with the detection structurally', () => { + const reordered = { settings: { low_threshold: 100, high_threshold: 200 }, type: 'canny_edge_detection' }; + expect(isControlMapCurrent(detected({ filter: reordered }))).toBe(true); + }); +}); + +describe('controlDetectionInputKey', () => { + const key = controlDetectionInputKey(detected()); + + it('ignores weight, name and paint on the derived map', () => { + const subject = detected(); + expect(controlDetectionInputKey({ ...subject, adapter: { ...subject.adapter, weight: 1.4 } })).toBe(key); + expect(controlDetectionInputKey({ ...subject, name: 'Edges 1' })).toBe(key); + expect(controlDetectionInputKey({ ...subject, source: { ...map, bitmap: image('painted.png') } })).toBe(key); + }); + + it('is independent of detector settings key order', () => { + const reordered = { settings: { low_threshold: 100, high_threshold: 200 }, type: 'canny_edge_detection' }; + expect(controlDetectionInputKey(detected({ detection: reordered }))).toBe(key); + }); + + it.each([ + ['the input image', detected({ original: { image: image('other.png'), type: 'image' } })], + ['the original frame', detected({ originalTransform: { ...IDENTITY, x: 4 } })], + ['a detector setting', detected({ detection: { ...canny, settings: { ...canny.settings, low_threshold: 1 } } })], + ['the detector', detected({ detection: { settings: {}, type: 'mlsd_detection' } })], + ['the type', detected({ controlType: 'lineart' })], + ])('changes with %s', (_label, subject) => { + expect(controlDetectionInputKey(subject)).not.toBe(key); + }); + + it('follows the paint of a layer whose own pixels are the input', () => { + const drawn = layer({ controlType: 'edges', detection: canny, source: map }); + expect(controlDetectionInputKey({ ...drawn, source: { ...map, bitmap: image('more-strokes.png') } })).not.toBe( + controlDetectionInputKey(drawn) + ); + }); +}); + +describe('detachControlDerivation', () => { + it('keeps the map as the image and makes the detection "None"', () => { + const subject = detected({ originalTransform: { ...IDENTITY, x: 3 } }); + const detached = detachControlDerivation(subject); + expect(detached).toEqual({ ...layer({ controlType: 'edges', filter: canny, source: map }), detection: null }); + expect(detached).not.toHaveProperty('original'); + expect(detached).not.toHaveProperty('originalTransform'); + expect(isControlMapCurrent(detached)).toBe(true); + }); + + it('returns a layer without derivation unchanged', () => { + const legacy = layer({ filter: canny, source: map }); + expect(detachControlDerivation(legacy)).toBe(legacy); + const none = layer({ controlType: 'depth', detection: null, source: photo }); + expect(detachControlDerivation(none)).toBe(none); + }); +}); + +describe('rebaseControlOriginal', () => { + const point = { x: 7, y: -3 }; + const docPoint = (subject: CanvasControlLayerContract) => + applyToPoint( + layerTransformMatrix(subject.transform), + applyToPoint(layerTransformMatrix(subject.originalTransform ?? IDENTITY), point) + ); + + it('moves the original frame with a bake so the original lands where it was', () => { + const before = detected({ + originalTransform: { rotation: 0.3, scaleX: 0.5, scaleY: 0.5, x: 2, y: 1 }, + transform: { rotation: 0.2, scaleX: 2, scaleY: 2, x: 30, y: 40 }, + }); + const after = rebaseControlOriginal({ ...before, transform: IDENTITY }, before.transform); + expect(after.original).toBe(photo); + const expected = docPoint(before); + const actual = docPoint(after); + expect(actual.x).toBeCloseTo(expected.x, 9); + expect(actual.y).toBeCloseTo(expected.y, 9); + }); + + it('omits an identity frame and leaves layers without an original alone', () => { + const identity = rebaseControlOriginal(detected({ originalTransform: { ...IDENTITY, x: 5 } }), { + ...IDENTITY, + x: -5, + }); + expect(identity).not.toHaveProperty('originalTransform'); + const plain = layer({ controlType: 'edges', detection: canny, source: photo }); + expect(rebaseControlOriginal(plain, { ...IDENTITY, x: 9 })).toBe(plain); + }); + + it('detaches when the rebased frame would shear', () => { + const subject = detected({ originalTransform: { ...IDENTITY, rotation: Math.PI / 4 } }); + const after = rebaseControlOriginal(subject, { ...IDENTITY, scaleX: 2 }); + expect(after).toEqual(detachControlDerivation(subject)); + expect(after.detection).toBeNull(); + }); +}); + +describe('withDetectedMap', () => { + const result = { image: image('detected.png'), rect: { height: 64, width: 64, x: -2, y: 5 } }; + + it('keeps the input as the original and installs the map as a paint source', () => { + const pending = layer({ controlType: 'edges', detection: canny, source: photo }); + const next = withDetectedMap(pending, result, canny); + expect(next).toEqual({ + ...pending, + filter: canny, + original: photo, + source: { bitmap: image('detected.png'), offset: { x: -2, y: 5 }, type: 'paint' }, + }); + expect(next && isControlMapCurrent(next)).toBe(true); + expect(next && needsControlDetection(next)).toBe(false); + }); + + it('re-detects from the kept original, never from the previous map', () => { + const next = withDetectedMap(detected(), result, { settings: {}, type: 'mlsd_detection' }); + expect(next?.original).toBe(photo); + expect(next?.filter).toEqual({ settings: {}, type: 'mlsd_detection' }); + }); + + it('folds the original frame into the layer transform', () => { + const next = withDetectedMap( + detected({ originalTransform: { ...IDENTITY, scaleX: 2, scaleY: 2 }, transform: { ...IDENTITY, x: 10 } }), + result, + canny + ); + expect(next?.transform).toEqual({ rotation: 0, scaleX: 2, scaleY: 2, x: 10, y: 0 }); + expect(next).not.toHaveProperty('originalTransform'); + }); + + it.each([ + ['an untyped layer', layer({ source: photo })], + ['a layer without input pixels', layer({ controlType: 'edges', detection: canny })], + ['a map-only layer', layer({ controlType: 'edges', detection: canny, filter: canny, source: map })], + [ + 'a frame that would shear', + detected({ + originalTransform: { ...IDENTITY, rotation: Math.PI / 4 }, + transform: { ...IDENTITY, scaleX: 2 }, + }), + ], + ])('cannot land on %s', (_label, subject) => { + expect(withDetectedMap(subject, result, canny)).toBeNull(); + }); +}); + +describe('restoreControlOriginal', () => { + it('returns the original to the source in its own frame and undoes a detection exactly', () => { + const pending = layer({ + controlType: 'edges', + detection: canny, + source: photo, + transform: { rotation: 0.4, scaleX: 1.5, scaleY: 1.5, x: 8, y: 9 }, + }); + const landed = withDetectedMap( + pending, + { image: image('detected.png'), rect: { height: 64, width: 64, x: 0, y: 0 } }, + canny + )!; + expect(restoreControlOriginal(landed)).toEqual({ ...pending, detection: null }); + }); + + it('applies the original frame when it differs from the map', () => { + const restored = restoreControlOriginal( + detected({ originalTransform: { ...IDENTITY, x: 4 }, transform: { ...IDENTITY, scaleX: 2, scaleY: 2 } }) + ); + expect(restored).toMatchObject({ detection: null, source: photo, transform: { scaleX: 2, scaleY: 2, x: 8, y: 0 } }); + expect(restored).not.toHaveProperty('filter'); + expect(restored).not.toHaveProperty('original'); + }); + + it('keeps the map as the image when the original frame would shear', () => { + const subject = detected({ + originalTransform: { ...IDENTITY, rotation: Math.PI / 4 }, + transform: { ...IDENTITY, scaleX: 2 }, + }); + const restored = restoreControlOriginal(subject); + expect(restored).toMatchObject({ detection: null, source: map, transform: subject.transform }); + expect(restored).not.toHaveProperty('original'); + expect(restored).not.toHaveProperty('filter'); + }); + + it('only switches detection to "None" without an original, and leaves untyped layers alone', () => { + const pending = layer({ controlType: 'edges', detection: canny, source: photo }); + expect(restoreControlOriginal(pending)).toEqual({ ...pending, detection: null }); + const untyped = layer({ source: photo }); + expect(restoreControlOriginal(untyped)).toBe(untyped); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.ts new file mode 100644 index 00000000000..13f56b7fe8b --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/controlDerivation.ts @@ -0,0 +1,186 @@ +/** + * Pure rules for control layers that derive their map from a kept original; the invariants are documented on + * `CanvasControlLayerContract`. Every function returns its input unchanged when nothing changes. + */ + +import type { + CanvasControlDetectionContract, + CanvasControlLayerContract, + CanvasImageRef, + CanvasLayerSourceContract, + CanvasPixelSourceContract, +} from '@workbench/canvas-engine/contracts'; +import type { LayerTransform } from '@workbench/canvas-engine/transform/transformMath'; +import type { Rect } from '@workbench/canvas-engine/types'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; +import { composeLayerTransforms } from '@workbench/canvas-engine/transform/transformMath'; + +const asPixelSource = (source: CanvasLayerSourceContract): CanvasPixelSourceContract | null => + source.type === 'image' || source.type === 'paint' ? source : null; + +const hasPixels = (source: CanvasPixelSourceContract | null): source is CanvasPixelSourceContract => + source !== null && (source.type === 'image' || source.bitmap !== null); + +const isIdentityTransform = (transform: LayerTransform): boolean => + transform.x === 0 && + transform.y === 0 && + transform.rotation === 0 && + transform.scaleX === 1 && + transform.scaleY === 1; + +const withoutOriginal = (layer: CanvasControlLayerContract): CanvasControlLayerContract => { + const next = { ...layer }; + delete next.original; + delete next.originalTransform; + return next; +}; + +/** + * The pixels detection runs on: the kept original, else the layer's own source unless a filter produced it. Null for + * map-only layers (legacy filtered layers, filter copies) and for sources without pixels of their own. + */ +export const getControlDetectionInput = (layer: CanvasControlLayerContract): CanvasPixelSourceContract | null => + layer.original ?? (layer.filter ? null : asPixelSource(layer.source)); + +/** Whether `source` already reflects the chosen detection: no detection, or the kept original detected with it. */ +export const isControlMapCurrent = (layer: CanvasControlLayerContract): boolean => + layer.detection === null || + layer.detection === undefined || + (layer.original !== undefined && areJsonValuesStructurallyEqual(layer.filter, layer.detection)); + +/** Whether a typed layer's map is stale and there are pixels to detect it from. */ +export const needsControlDetection = (layer: CanvasControlLayerContract): boolean => + layer.controlType !== undefined && + layer.detection !== null && + layer.detection !== undefined && + !isControlMapCurrent(layer) && + hasPixels(getControlDetectionInput(layer)); + +const canonicalJson = (value: unknown): string => { + if (Array.isArray(value)) { + return `[${value.map(canonicalJson).join(',')}]`; + } + if (value !== null && typeof value === 'object') { + const record = value as Record; + const entries = Object.keys(record) + .filter((key) => record[key] !== undefined) + .sort() + .map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`); + return `{${entries.join(',')}}`; + } + return JSON.stringify(value) ?? 'null'; +}; + +const inputRef = (input: CanvasPixelSourceContract | null): string => { + if (input === null) { + return 'none'; + } + if (input.type === 'image') { + return `image:${input.image.imageName}`; + } + return input.bitmap + ? `paint:${input.bitmap.imageName}@${String(input.offset?.x ?? 0)},${String(input.offset?.y ?? 0)}` + : 'paint:empty'; +}; + +/** + * Identifies what a detection run consumes: its input pixels and their frame, the detector with its settings, and + * the type. Weight, name and painting on the derived map leave it unchanged, so they never stale a running job. + */ +export const controlDetectionInputKey = (layer: CanvasControlLayerContract): string => + canonicalJson([ + inputRef(getControlDetectionInput(layer)), + layer.originalTransform ?? null, + layer.detection ?? null, + layer.controlType ?? null, + ]); + +/** Keeps the current pixels as the image itself: drops the original and makes a chosen detection "None". */ +export const detachControlDerivation = (layer: CanvasControlLayerContract): CanvasControlLayerContract => { + if (layer.original === undefined && layer.originalTransform === undefined && !layer.detection) { + return layer; + } + const next = withoutOriginal(layer); + if (next.detection !== undefined) { + next.detection = null; + } + return next; +}; + +/** + * Follows a bake of `bakedTransform` into the map's pixels (the layer transform then resets to identity) by moving + * the original's frame with it. A frame the bake would shear cannot be kept, so the layer detaches instead. + */ +export const rebaseControlOriginal = ( + layer: CanvasControlLayerContract, + bakedTransform: LayerTransform +): CanvasControlLayerContract => { + if (layer.original === undefined) { + return layer; + } + const originalTransform = layer.originalTransform + ? composeLayerTransforms(bakedTransform, layer.originalTransform) + : bakedTransform; + if (!originalTransform) { + return detachControlDerivation(layer); + } + if (isIdentityTransform(originalTransform)) { + if (layer.originalTransform === undefined) { + return layer; + } + const next = { ...layer }; + delete next.originalTransform; + return next; + } + return { ...layer, originalTransform: { ...originalTransform } }; +}; + +/** + * Lands a detected map. The map's `rect` is in the detection input's local frame, so a kept `originalTransform` folds + * into the layer transform. Null when the layer has no type (I1), no input pixels, or the fold would shear. + */ +export const withDetectedMap = ( + layer: CanvasControlLayerContract, + map: { image: CanvasImageRef; rect: Rect }, + detection: CanvasControlDetectionContract +): CanvasControlLayerContract | null => { + const original = getControlDetectionInput(layer); + if (layer.controlType === undefined || !hasPixels(original)) { + return null; + } + const transform = layer.originalTransform + ? composeLayerTransforms(layer.transform, layer.originalTransform) + : layer.transform; + if (!transform) { + return null; + } + const next: CanvasControlLayerContract = { + ...layer, + detection, + filter: detection, + original, + source: { bitmap: { ...map.image }, offset: { x: map.rect.x, y: map.rect.y }, type: 'paint' }, + transform, + }; + delete next.originalTransform; + return next; +}; + +/** + * Detection "None": the original becomes the source again in its own frame. When that frame would shear, the map + * stays as the as-is image and only the derivation is dropped. + */ +export const restoreControlOriginal = (layer: CanvasControlLayerContract): CanvasControlLayerContract => { + if (layer.original === undefined) { + return layer.controlType === undefined || layer.detection === null ? layer : { ...layer, detection: null }; + } + const transform = layer.originalTransform + ? composeLayerTransforms(layer.transform, layer.originalTransform) + : layer.transform; + const next = withoutOriginal( + transform ? { ...layer, detection: null, source: layer.original, transform } : { ...layer, detection: null } + ); + delete next.filter; + return next; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.test.ts index 84405552977..63f282d5efd 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.test.ts @@ -1,17 +1,49 @@ import { CONTROL_ADAPTER_DEFAULTS } from '@workbench/controlAdapters'; import { describe, expect, it } from 'vitest'; -import { createControlLayer } from './layerFactories'; +import { createControlLayer, nextControlLayerName } from './layerFactories'; + +describe('createControlLayer', () => { + it('creates an empty, untyped, model-less layer on the legacy ControlNet defaults with transparency on', () => { + const layer = createControlLayer({ id: 'c1', name: 'Control Layer 1' }); + expect(layer).toEqual({ + adapter: { beginEndStepPct: [0, 0.75], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 0.75 }, + blendMode: 'normal', + id: 'c1', + isEnabled: true, + isLocked: false, + name: 'Control Layer 1', + opacity: 1, + source: { bitmap: null, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }); + // The defaults' mutable step range is copied, never shared. + expect(layer.adapter.beginEndStepPct).not.toBe(CONTROL_ADAPTER_DEFAULTS.controlnet.beginEndStepPct); + }); -describe('createControlLayer (engine results and filters)', () => { it.each([ ['anima', 'anima_lllite'], ['z-image', 'z_image_control'], ['sdxl', 'controlnet'], - ] as const)('starts a %s layer on its kind with that kind’s defaults and the given model', (base, kind) => { - expect(createControlLayer('Control Layer 1', 'c1', base, 'model-key').adapter).toEqual({ + ] as const)('starts a %s layer on its kind’s defaults without a model', (base, kind) => { + expect(createControlLayer({ base, id: 'c1', name: 'Control Layer 1' }).adapter).toEqual({ ...CONTROL_ADAPTER_DEFAULTS[kind], - model: 'model-key', + model: null, }); }); + + it('uses the given pixel source', () => { + const source = { image: { height: 48, imageName: 'photo.png', width: 64 }, type: 'image' } as const; + expect(createControlLayer({ id: 'c1', name: 'C', source }).source).toBe(source); + }); +}); + +describe('nextControlLayerName', () => { + it('returns the first free "Control Layer N" gap', () => { + expect(nextControlLayerName([])).toBe('Control Layer 1'); + expect(nextControlLayerName(['Control Layer 1', 'Control Layer 3', 'Layer 2'])).toBe('Control Layer 2'); + expect(nextControlLayerName(['Control Layer 1', 'Control Layer 2'])).toBe('Control Layer 3'); + }); }); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.ts index 539418f77a3..b382ebcdc05 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/document/layerFactories.ts @@ -5,6 +5,7 @@ import type { CanvasLayerBaseContract, CanvasLayerSourceContract, CanvasMaskFillContract, + CanvasPixelSourceContract, CanvasRegionalGuidanceLayerContract, } from '@workbench/canvas-engine/contracts'; import type { Rect } from '@workbench/canvas-engine/types'; @@ -90,29 +91,37 @@ export const createRegionalGuidanceFromImage = ( type: 'regional_guidance', }); -export const createControlLayer = ( - name: string, - id: string, - base?: string | null, - model?: string | null -): CanvasControlLayerContract => { - return { - adapter: createDefaultControlAdapter(base, model ?? null), - blendMode: 'normal', - id, - isEnabled: true, - isLocked: false, - name, - opacity: 1, - source: { bitmap: null, type: 'paint' }, - transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, - type: 'control', - withTransparencyEffect: true, - }; -}; +export interface CreateControlLayerInput { + name: string; + id: string; + /** Main model base whose adapter kind and defaults the layer starts on. */ + base?: string | null; + /** Defaults to an empty paint source. */ + source?: CanvasPixelSourceContract; +} + +/** The one control-layer factory: untyped, with no model, so generation resolves the model from the type. */ +export const createControlLayer = ({ + base, + id, + name, + source, +}: CreateControlLayerInput): CanvasControlLayerContract => ({ + adapter: createDefaultControlAdapter(base, null), + blendMode: 'normal', + id, + isEnabled: true, + isLocked: false, + name, + opacity: 1, + source: source ?? { bitmap: null, type: 'paint' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, +}); /** - * A control layer made from another layer, keeping its appearance and flags with a default adapter. `content` either + * An untyped, model-less control layer made from another layer, keeping its appearance and flags. `content` either * places baked pixels at `rect` under an identity transform, or reuses a source and transform as they are. */ export const createControlLayerFrom = ( @@ -121,11 +130,10 @@ export const createControlLayerFrom = ( id: string; name: string; base: string | null; - model: string | null; content: { readonly rect: Rect } | { readonly source: CanvasLayerSourceContract }; } ): CanvasControlLayerContract => ({ - ...createControlLayer(options.name, options.id, options.base, options.model), + ...createControlLayer({ base: options.base, id: options.id, name: options.name }), blendMode: from.blendMode, ...(from.colorLabel ? { colorLabel: from.colorLabel } : {}), isEnabled: from.isEnabled, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.test.ts index 46550bf1407..9e5c000716b 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.test.ts @@ -178,6 +178,47 @@ describe('buildMaterializedPixelLayer', () => { expect(before.source.type).toBe('image'); }); + it("moves a control layer's kept original with the baked transform", () => { + const depth = { settings: {}, type: 'depth_anything_depth_estimation' }; + const before = control({ + controlType: 'depth', + detection: depth, + filter: depth, + original: { image: { height: 10, imageName: 'photo', width: 10 }, type: 'image' }, + originalTransform: { rotation: 0, scaleX: 1, scaleY: 1, x: 5, y: 0 }, + transform: { rotation: 0, scaleX: 2, scaleY: 2, x: 10, y: 20 }, + }); + + const after = buildMaterializedPixelLayer(before, { height: 20, width: 20, x: 10, y: 20 }); + + expect(after).toMatchObject({ controlType: 'depth', detection: depth, original: before.original }); + expect(after.type === 'control' && after.originalTransform).toEqual({ + rotation: 0, + scaleX: 2, + scaleY: 2, + x: 20, + y: 20, + }); + }); + + it('detaches a control layer whose original the bake would shear', () => { + const depth = { settings: {}, type: 'depth_anything_depth_estimation' }; + const before = control({ + controlType: 'depth', + detection: depth, + filter: depth, + original: { image: { height: 10, imageName: 'photo', width: 10 }, type: 'image' }, + originalTransform: { rotation: Math.PI / 4, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + transform: { rotation: 0, scaleX: 2, scaleY: 1, x: 0, y: 0 }, + }); + + const after = buildMaterializedPixelLayer(before, { height: 10, width: 20, x: 0, y: 0 }); + + expect(after).toMatchObject({ controlType: 'depth', detection: null, filter: depth }); + expect(after).not.toHaveProperty('original'); + expect(after).not.toHaveProperty('originalTransform'); + }); + it('clears raster adjustments after baking them into a paint source', () => { const before: CanvasLayerContract = { adjustments: [ diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.ts index d826b436e80..67b9d59df04 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/editing/controlPixelEdit.ts @@ -5,6 +5,7 @@ import type { SemanticLeaf } from '@workbench/canvas-engine/document-model/seman import type { RasterBackend, RasterSurface } from '@workbench/canvas-engine/render/raster'; import type { Rect } from '@workbench/canvas-engine/types'; +import { rebaseControlOriginal } from '@workbench/canvas-engine/document/controlDerivation'; import { isLeafEditable } from '@workbench/canvas-engine/document/layerEligibility'; import { isEmptyPolygonShape } from '@workbench/canvas-engine/document/sources'; import { roundOut, transformBounds } from '@workbench/canvas-engine/math/rect'; @@ -70,12 +71,19 @@ export const decidePixelEdit = ({ return { status: 'materialize' }; }; +/** + * The layer after its transform is baked into paint pixels at `rect`. A control layer's kept original moves with the + * bake, or detaches when the bake would shear its frame. + */ export const buildMaterializedPixelLayer = (layer: PixelEditableLayer, rect: Rect): PixelEditableLayer => { const materialized: PixelEditableLayer = { ...structuredClone(layer), source: { bitmap: null, offset: { x: rect.x, y: rect.y }, type: 'paint' }, transform: { ...IDENTITY_TRANSFORM }, }; + if (materialized.type === 'control') { + return rebaseControlOriginal(materialized, layer.transform); + } if (materialized.type === 'raster') { // Raster adjustments are applied before the layer transform by the normal // compositor. Materialization bakes those displayed pixels so the eraser diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.test.ts index 4699a6065b2..0c915605349 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.test.ts @@ -9405,111 +9405,6 @@ describe('commitRasterFilterResult', () => { engine.lifecycle.dispose(); }); - it('atomically applies a filter and nonzero local origin to a control layer with exact undo/redo', async () => { - const source: CanvasLayerContract = { - adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, - blendMode: 'normal', - filter: { settings: { radius: 1 }, type: 'old_filter' }, - id: 'control', - isEnabled: true, - isLocked: false, - name: 'Control', - opacity: 1, - source: { image: { height: 10, imageName: 'control-source', width: 10 }, type: 'image' }, - transform: { rotation: 12, scaleX: 2, scaleY: 3, x: 40, y: 50 }, - type: 'control', - withTransparencyEffect: true, - }; - const { projectId, store } = createReducerBackedStore(filterDoc([source])); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - bitmapStore: createSpyBitmapStore(), - imageResolver: () => Promise.resolve(new Blob()), - projectId, - store, - }); - const before = structuredClone(source); - const exported = await engine.exports.exportLayerPixels(source.id); - if (exported.status !== 'ok') { - throw new Error('expected a filterable control export'); - } - - const result = await engine.layers.commitRasterFilterResult({ - filter: { settings: { radius: 8 }, type: 'content_shuffle' }, - guard: exported.guard, - image: durableImage, - mode: 'replace', - rect: { height: 10, width: 10, x: 7, y: -3 }, - target: 'apply', - }); - - expect(result).toEqual({ layerId: source.id, status: 'committed' }); - const after = getDocumentLeaves(engine.document.getDocument()!)[0]!; - expect(after).toMatchObject({ - filter: { settings: { radius: 8 }, type: 'content_shuffle' }, - source: { bitmap: durableImage, offset: { x: 7, y: -3 }, type: 'paint' }, - transform: source.transform, - type: 'control', - }); - await engine.history.undo(); - expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toEqual(before); - await engine.history.redo(); - expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toEqual(after); - engine.lifecycle.dispose(); - }); - - it.each(['raster', 'control'] as const)( - 'saves a filtered result as a %s layer above a control source', - async (target) => { - const source: CanvasLayerContract = { - adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, - blendMode: 'normal', - id: 'control', - isEnabled: true, - isLocked: false, - name: 'Control', - opacity: 1, - source: { image: { height: 10, imageName: 'control-source', width: 10 }, type: 'image' }, - transform: { rotation: 12, scaleX: 2, scaleY: 3, x: 40, y: 50 }, - type: 'control', - withTransparencyEffect: true, - }; - const { projectId, store } = createReducerBackedStore(filterDoc([source])); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - bitmapStore: createSpyBitmapStore(), - imageResolver: () => Promise.resolve(new Blob()), - projectId, - store, - }); - const exported = await engine.exports.exportLayerPixels(source.id); - if (exported.status !== 'ok') { - throw new Error('expected a filterable control export'); - } - - const result = await engine.layers.commitRasterFilterResult({ - filter: { settings: {}, type: 'canny_edge_detection' }, - guard: exported.guard, - image: durableImage, - mode: 'copy', - rect: { height: 10, width: 10, x: 7, y: -3 }, - target, - }); - - expect(result.status).toBe('committed'); - const saved = engine.document.getDocument()!.stacks[target][0]!; - expect(getDocumentLeaves(engine.document.getDocument()!).map((layer) => layer.id)).toEqual( - target === 'control' ? [saved.id, 'control'] : ['control', saved.id] - ); - expect(saved).toMatchObject({ - filter: { settings: {}, type: 'canny_edge_detection' }, - source: { offset: { x: 7, y: -3 }, type: 'paint' }, - transform: source.transform, - }); - engine.lifecycle.dispose(); - } - ); - it('preserves raster source placement when saving the filtered result as control', async () => { const source = filterLayer('source'); const { projectId, store } = createReducerBackedStore(filterDoc([source])); @@ -10954,46 +10849,12 @@ describe('commitGeneratedImageResult', () => { } const created = getDocumentLeaves(engine.document.getDocument()!).find((layer) => layer.id === result.layerId); expect(created?.type === 'control' ? created.adapter.kind : null).toBe(expectedKind); - // Without a getDefaultControlModel option the layer starts model-less. + // New control layers never carry a model; generation resolves it. expect(created?.type === 'control' ? created.adapter.model : 'wrong-type').toBeNull(); engine.lifecycle.dispose(); } ); - it('seeds workflow Copy To Control layers with the injected default control model', async () => { - const source = workflowRaster(); - const { projectId, store } = createReducerBackedStore(workflowDocument([source]), 'sd-1'); - const getDefaultControlModel = vi.fn((base: string | null) => (base === 'sd-1' ? 'sd1-union' : null)); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - bitmapStore: createSpyBitmapStore(), - getDefaultControlModel, - imageResolver: () => Promise.resolve(new Blob()), - projectId, - store, - }); - const exported = await engine.exports.exportLayerPixels(source.id); - if (exported.status !== 'ok') { - throw new Error('expected an exportable workflow source'); - } - - const result = await engine.layers.commitGeneratedImageResult({ - guard: exported.guard, - image: generatedImage, - origin: generatedOrigin, - target: 'copy-control', - }); - - expect(result.status).toBe('committed'); - if (result.status !== 'committed') { - throw new Error('expected a committed control copy'); - } - expect(getDefaultControlModel).toHaveBeenCalledWith('sd-1'); - const created = getDocumentLeaves(engine.document.getDocument()!).find((layer) => layer.id === result.layerId); - expect(created?.type === 'control' ? created.adapter.model : null).toBe('sd1-union'); - engine.lifecycle.dispose(); - }); - it('replaces a raster at the generated origin and native size, clears baked adjustments, and replays exactly', async () => { const source = workflowRaster(); const before = structuredClone(source); @@ -11057,8 +10918,13 @@ describe('commitGeneratedImageResult', () => { engine.lifecycle.dispose(); }); - it('replaces a control while preserving adapter, filter, transparency effect, and layer presentation', async () => { + it('replaces a control while preserving adapter, transparency effect, and layer presentation', async () => { const source = workflowControl(); + if (source.type !== 'control') { + throw new Error('expected a control fixture'); + } + // The result is the image itself, not a filter's map, so it drops the filter that made the old pixels. + const { filter: _filter, ...unfiltered } = source; const { projectId, store } = createReducerBackedStore(workflowDocument([source])); const engine = createCanvasEngine({ backend: createTestStubRasterBackend(), @@ -11082,7 +10948,7 @@ describe('commitGeneratedImageResult', () => { ).resolves.toEqual({ layerId: source.id, status: 'committed' }); expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toEqual({ - ...source, + ...unfiltered, source: { bitmap: generatedImage, offset: generatedOrigin, type: 'paint' }, transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, }); @@ -11090,8 +10956,7 @@ describe('commitGeneratedImageResult', () => { expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toEqual(source); await engine.history.redo(); expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toMatchObject({ - adapter: source.type === 'control' ? source.adapter : undefined, - filter: source.type === 'control' ? source.filter : undefined, + adapter: source.adapter, withTransparencyEffect: true, }); engine.lifecycle.dispose(); @@ -12922,13 +12787,12 @@ describe('guarded filter previews', () => { engine.lifecycle.dispose(); }); - it('starts an unfiltered layer with a recommendation but preserves an existing manual filter and Cancel', async () => { - const unfiltered = guardableLayer('recommended'); + it('starts a filtered layer from its existing filter, and Cancel leaves it as it was', async () => { const manual = { ...guardableLayer('manual'), filter: { settings: { coarse: true }, type: 'lineart_edge_detection' }, }; - const document = { ...emptyDoc(), stacks: stacksFrom([unfiltered, manual]) }; + const document = { ...emptyDoc(), stacks: stacksFrom([manual]) }; const { store } = createReactiveStore(document); const engine = createCanvasEngine({ backend: createTestStubRasterBackend(), @@ -12936,15 +12800,7 @@ describe('guarded filter previews', () => { projectId: 'p1', store, }); - expect((await engine.exports.exportLayerPixels('recommended')).status).toBe('ok'); - expect(getCanvasOperations(engine).startFilterOperation('recommended', 'normal_map')).toBe('started'); - expect(getCanvasOperations(engine).stores.filterSession.get()?.draft).toEqual({ settings: {}, type: 'normal_map' }); - getCanvasOperations(engine).cancelFilterOperation(); - expect(engine.document.getDocument()).toEqual({ ...document, selectedLayerId: 'recommended' }); - expect((await engine.exports.exportLayerPixels('manual')).status).toBe('ok'); - expect(getCanvasOperations(engine).startFilterOperation('manual', 'normal_map')).toBe('not-ready'); - expect(getCanvasOperations(engine).stores.filterSession.get()).toBeNull(); expect(getCanvasOperations(engine).startFilterOperation('manual')).toBe('started'); expect(getCanvasOperations(engine).stores.filterSession.get()?.draft).toEqual(manual.filter); getCanvasOperations(engine).cancelFilterOperation(); @@ -12952,112 +12808,26 @@ describe('guarded filter previews', () => { engine.lifecycle.dispose(); }); - it('does not replace another layer filter session with a recommendation', async () => { - const first = guardableLayer('first'); - const second = guardableLayer('second'); - const document = { ...emptyDoc(), stacks: stacksFrom([first, second]) }; - const { store } = createReactiveStore(document); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - imageResolver: () => Promise.resolve(new Blob()), - projectId: 'p1', - store, - }); - expect((await engine.exports.exportLayerPixels('first')).status).toBe('ok'); - expect(getCanvasOperations(engine).startFilterOperation('first')).toBe('started'); - getCanvasOperations(engine).updateFilterOperation({ settings: { coarse: true }, type: 'lineart_edge_detection' }); - const active = getCanvasOperations(engine).stores.filterSession.get(); - - expect((await engine.exports.exportLayerPixels('second')).status).toBe('ok'); - expect(getCanvasOperations(engine).startFilterOperation('second', 'normal_map')).toBe('not-ready'); - expect(getCanvasOperations(engine).stores.filterSession.get()).toEqual(active); - engine.lifecycle.dispose(); - }); - - it('does not overwrite a same-layer draft or preview on rapid recommendations', async () => { - const layer = guardableLayer('recommended'); - const document = { ...emptyDoc(), stacks: stacksFrom([layer]) }; - const { store } = createReactiveStore(document); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - filterDeps: { - runGraph: vi.fn(() => - Promise.resolve({ height: 10, imageName: 'filtered', origin: 'canvas', output: {}, width: 10 }) - ), - uploadIntermediate: vi.fn(() => Promise.resolve({ imageName: 'input' })), - }, - imageResolver: () => Promise.resolve(new Blob()), - projectId: 'p1', - store, - }); - expect((await engine.exports.exportLayerPixels(layer.id)).status).toBe('ok'); - expect(getCanvasOperations(engine).startFilterOperation(layer.id, 'normal_map')).toBe('started'); - getCanvasOperations(engine).updateFilterOperation({ settings: { coarse: true }, type: 'lineart_edge_detection' }); - await getCanvasOperations(engine).processFilterOperation(); - const active = getCanvasOperations(engine).stores.filterSession.get(); - - expect(getCanvasOperations(engine).startFilterOperation(layer.id, 'canny_edge_detection')).toBe('not-ready'); - expect(getCanvasOperations(engine).stores.filterSession.get()).toEqual(active); - engine.lifecycle.dispose(); - }); - - it('does not replace a different active canvas operation with a recommendation', async () => { - const first = guardableLayer('first'); - const second = guardableLayer('second'); - const document = { ...emptyDoc(), stacks: stacksFrom([first, second]) }; - const { store } = createReactiveStore(document); - const engine = createCanvasEngine({ - backend: createTestStubRasterBackend(), - imageResolver: () => Promise.resolve(new Blob()), - projectId: 'p1', - store, - }); - expect((await engine.exports.exportLayerPixels('first')).status).toBe('ok'); - expect((await engine.exports.exportLayerPixels('second')).status).toBe('ok'); - expect(getCanvasOperations(engine).startSelectObject('first')).toBe('started'); - const operation = getCanvasOperations(engine).controller.getSnapshot(); - - expect(getCanvasOperations(engine).startFilterOperation('second', 'normal_map')).toBe('not-ready'); - expect(getCanvasOperations(engine).controller.getSnapshot()).toEqual(operation); - expect(getCanvasOperations(engine).stores.samSession.get()?.sourceRect).toEqual({ - height: 10, - width: 10, - x: 0, - y: 0, - }); - engine.lifecycle.dispose(); + // Filter starts only on raster layers; a control layer's map comes from detection. + const filterFlowLayer = (): Extract => ({ + blendMode: 'multiply', + filter: { settings: { radius: 2 }, type: 'content_shuffle' }, + id: 'filter-source', + isEnabled: true, + isLocked: false, + name: 'Filter source', + opacity: 0.6, + source: { + bitmap: { height: 10, imageName: 'filter-source.png', width: 10 }, + offset: { x: 7, y: -3 }, + type: 'paint', + }, + transform: { rotation: 12, scaleX: 2, scaleY: 3, x: 40, y: 50 }, + type: 'raster', }); - const filterFlowLayer = ( - type: 'raster' | 'control' - ): Extract => { - const base = { - blendMode: 'multiply' as const, - filter: { settings: { radius: 2 }, type: 'content_shuffle' }, - id: 'filter-source', - isEnabled: true, - isLocked: false, - name: 'Filter source', - opacity: 0.6, - source: { - bitmap: { height: 10, imageName: 'filter-source.png', width: 10 }, - offset: { x: 7, y: -3 }, - type: 'paint' as const, - }, - transform: { rotation: 12, scaleX: 2, scaleY: 3, x: 40, y: 50 }, - }; - return type === 'raster' - ? { ...base, type: 'raster' } - : { - ...base, - adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, - type: 'control', - withTransparencyEffect: true, - }; - }; - - const createFilterFlowHarness = async (sourceType: 'raster' | 'control') => { - const source = filterFlowLayer(sourceType); + const createFilterFlowHarness = async () => { + const source = filterFlowLayer(); const document = { ...emptyDoc(), stacks: stacksFrom([source]), selectedLayerId: source.id }; const { projectId, store } = createReducerBackedStore(document); const engine = createCanvasEngine({ @@ -13084,7 +12854,7 @@ describe('guarded filter previews', () => { }; it('blocks Filter actions called directly after an external lock and still allows Cancel', async () => { - const { document, engine } = await createFilterFlowHarness('raster'); + const { document, engine } = await createFilterFlowHarness(); const makeDurable = vi.fn(() => Promise.resolve()); const before = getCanvasOperations(engine).stores.filterSession.get(); @@ -13111,7 +12881,7 @@ describe('guarded filter previews', () => { }); it('blocks Filter mutation when an external lock begins during durability', async () => { - const { document, engine } = await createFilterFlowHarness('raster'); + const { document, engine } = await createFilterFlowHarness(); const durable = createDeferred(); const pending = getCanvasOperations(engine).commitFilterOperation('apply', () => durable.promise); await vi.waitFor(() => expect(getCanvasOperations(engine).stores.filterSession.get()?.status).toBe('committing')); @@ -13130,7 +12900,7 @@ describe('guarded filter previews', () => { }); it('interrupts Filter processing on an external lock without losing the draft or operation', async () => { - const source = filterFlowLayer('raster'); + const source = filterFlowLayer(); const { projectId, store } = createReducerBackedStore({ ...emptyDoc(), stacks: stacksFrom([source]), @@ -13196,7 +12966,7 @@ describe('guarded filter previews', () => { ] as const)( 'preserves $type/$target output geometry through preview, commit, and one-entry undo/redo', async ({ expectedRect, output, settings, target, type }) => { - const source = filterFlowLayer('raster'); + const source = filterFlowLayer(); const document = { ...emptyDoc(), stacks: stacksFrom([source]), selectedLayerId: source.id }; const { projectId, store } = createReducerBackedStore(document); const engine = createCanvasEngine({ @@ -13246,63 +13016,60 @@ describe('guarded filter previews', () => { } ); - it.each(['raster', 'control'] as const)( - 'processes, retries durability, and applies %s with one origin-preserving history entry', - async (sourceType) => { - const { document, engine, source } = await createFilterFlowHarness(sourceType); - const makeDurable = vi - .fn<(imageName: string) => Promise>() - .mockRejectedValueOnce(new Error('promotion failed')) - .mockResolvedValueOnce(); - - await getCanvasOperations(engine).commitFilterOperation('apply', makeDurable); - expect(getCanvasOperations(engine).stores.filterSession.get()).toMatchObject({ - error: { code: 'apply-failed', detail: 'promotion failed' }, - preview: { imageName: 'filtered.png' }, - status: 'error', - }); - expect(getCanvasOperations(engine).controller.getSnapshot()).toMatchObject({ - identity: { kind: 'filter' }, - status: 'active', - }); + it('processes, retries durability, and applies with one origin-preserving history entry', async () => { + const { document, engine, source } = await createFilterFlowHarness(); + const makeDurable = vi + .fn<(imageName: string) => Promise>() + .mockRejectedValueOnce(new Error('promotion failed')) + .mockResolvedValueOnce(); - await getCanvasOperations(engine).commitFilterOperation('apply', makeDurable); - expect(makeDurable).toHaveBeenCalledTimes(2); - expect(getCanvasOperations(engine).stores.filterSession.get()).toBeNull(); - expect(getCanvasOperations(engine).controller.getSnapshot()).toEqual({ status: 'idle' }); - expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toMatchObject({ - filter: source.filter, - opacity: source.opacity, - blendMode: source.blendMode, - source: { offset: { x: 7, y: -3 }, type: 'paint' }, - transform: source.transform, - type: sourceType, - }); - expect(engine.stores.canUndo.get()).toBe(true); - await engine.history.undo(); - expect(engine.document.getDocument()).toEqual(document); - expect(engine.stores.canUndo.get()).toBe(false); - engine.lifecycle.dispose(); - } - ); + await getCanvasOperations(engine).commitFilterOperation('apply', makeDurable); + expect(getCanvasOperations(engine).stores.filterSession.get()).toMatchObject({ + error: { code: 'apply-failed', detail: 'promotion failed' }, + preview: { imageName: 'filtered.png' }, + status: 'error', + }); + expect(getCanvasOperations(engine).controller.getSnapshot()).toMatchObject({ + identity: { kind: 'filter' }, + status: 'active', + }); + + await getCanvasOperations(engine).commitFilterOperation('apply', makeDurable); + expect(makeDurable).toHaveBeenCalledTimes(2); + expect(getCanvasOperations(engine).stores.filterSession.get()).toBeNull(); + expect(getCanvasOperations(engine).controller.getSnapshot()).toEqual({ status: 'idle' }); + expect(getDocumentLeaves(engine.document.getDocument()!)[0]).toMatchObject({ + filter: source.filter, + opacity: source.opacity, + blendMode: source.blendMode, + source: { offset: { x: 7, y: -3 }, type: 'paint' }, + transform: source.transform, + type: 'raster', + }); + expect(engine.stores.canUndo.get()).toBe(true); + await engine.history.undo(); + expect(engine.document.getDocument()).toEqual(document); + expect(engine.stores.canUndo.get()).toBe(false); + engine.lifecycle.dispose(); + }); it.each(['raster', 'control'] as const)( 'processes and saves as %s above the source with one history entry', async (target) => { - const { engine, source } = await createFilterFlowHarness('control'); + const { engine, source } = await createFilterFlowHarness(); await getCanvasOperations(engine).commitFilterOperation(target, () => Promise.resolve()); expect(getCanvasOperations(engine).stores.filterSession.get()).toBeNull(); const saved = engine.document.getDocument()!.stacks[target][0]!; - expect(getDocumentLeaves(engine.document.getDocument()!).map((layer) => layer.id)).toEqual( - target === 'control' ? [saved.id, source.id] : [source.id, saved.id] - ); + expect(getDocumentLeaves(engine.document.getDocument()!).map((layer) => layer.id)).toEqual([saved.id, source.id]); expect(saved).toMatchObject({ - blendMode: source.blendMode, - opacity: source.opacity, + // A raster copy keeps the source's compositing; a control copy starts from a new control layer's. + ...(target === 'raster' ? { blendMode: source.blendMode, opacity: source.opacity } : { opacity: 1 }), + name: 'Filter source filtered', source: { offset: { x: 7, y: -3 }, type: 'paint' }, transform: source.transform, + type: target, }); expect(engine.stores.canUndo.get()).toBe(true); await engine.history.undo(); @@ -13313,7 +13080,7 @@ describe('guarded filter previews', () => { ); it('cancels an in-flight promotion when a replacement operation starts', async () => { - const { engine, source } = await createFilterFlowHarness('raster'); + const { engine, source } = await createFilterFlowHarness(); let resolvePromotion!: () => void; const promotion = new Promise((resolve) => { resolvePromotion = resolve; @@ -13336,7 +13103,7 @@ describe('guarded filter previews', () => { }); it('aborts an in-flight upload when a newer Process request wins', async () => { - const source = filterFlowLayer('raster'); + const source = filterFlowLayer(); const document = { ...emptyDoc(), stacks: stacksFrom([source]), selectedLayerId: source.id }; const { projectId, store } = createReducerBackedStore(document); const uploadSignals: AbortSignal[] = []; @@ -14405,8 +14172,7 @@ describe('document mirror wiring: prop vs source change (paint-pixel survival)', const engine = createCanvasEngine({ backend: createTestStubRasterBackend(), createSpatialExecutor, - getDefaultControlModel: (base) => `${base}-control`, - getMainModelBase: () => 'sdxl', + getMainModelBase: () => 'anima', imageResolver: () => Promise.resolve(new Blob()), projectId, store, @@ -14423,7 +14189,7 @@ describe('document mirror wiring: prop vs source change (paint-pixel survival)', expect(await engine.layers.convertLayerToControl('Convert to control', layer)).toBe('converted'); expect(getDocumentLayer(engine.document.getDocument(), 'a')).toMatchObject({ - adapter: { model: 'sdxl-control' }, + adapter: { kind: 'anima_lllite', model: null }, isEnabled: false, source: { bitmap: null, offset: { x: 0, y: 0 }, type: 'paint' }, transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, @@ -14444,7 +14210,7 @@ describe('document mirror wiring: prop vs source change (paint-pixel survival)', expect(copied.status).toBe('copied'); const copyId = copied.status === 'copied' ? copied.layerId : ''; expect(getDocumentLayer(engine.document.getDocument(), copyId)).toMatchObject({ - adapter: { model: 'sdxl-control' }, + adapter: { kind: 'anima_lllite', model: null }, name: 'a copy', source: { bitmap: null, type: 'paint' }, type: 'control', @@ -19624,3 +19390,44 @@ describe('color picker loupe', () => { engine.lifecycle.dispose(); }); }); + +describe('control derivation composition', () => { + it('commits control-layer patches without the operation lease, so a running operation refuses them', async () => { + const control: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 1], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 1 }, + blendMode: 'normal', + id: 'control', + isEnabled: true, + isLocked: false, + name: 'Control Layer 1', + opacity: 1, + source: { image: { height: 10, imageName: 'photo', width: 10 }, type: 'image' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }; + const { store } = createReactiveStore({ ...makeDoc(), stacks: stacksFrom([control]) }); + const { applicationHost, engine } = createCoreCanvasEngine({ + backend: createTestStubRasterBackend(), + imageResolver: () => Promise.resolve(new Blob()), + mutationPort: createTestMutationPort(store, 'p1'), + projectId: 'p1', + reportError: () => undefined, + uploadImage: () => Promise.resolve({ height: 1, imageName: 'durable', width: 1 }), + uploadIntermediateImage: () => Promise.resolve({ height: 1, imageName: 'transient', width: 1 }), + }); + const patch = { expected: control, label: 'Set control type', next: { ...control, controlType: 'depth' as const } }; + + engine.stores.documentEditingLocked.set(true); + expect(engine.controlDerivation.commitControlLayerPatch(patch)).toEqual({ status: 'busy' }); + engine.stores.documentEditingLocked.set(false); + expect(engine.controlDerivation.commitControlLayerPatch(patch)).toEqual({ status: 'committed' }); + expect(engine.controlDerivation.captureControlDetectionGuard('control')).toMatchObject({ + documentRevision: 0, + layerId: 'control', + projectId: 'p1', + }); + await expect(applicationHost.flushLayerUploads('control', {})).resolves.toBe('clean'); + engine.lifecycle.dispose(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.ts index ac9460a57fb..c5108ae2dae 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/engine.ts @@ -1,4 +1,5 @@ import type { + CanvasControlDerivationCapability, CanvasEditRefusal, CanvasHistoryReplayStatus, CanvasHistoryCapability, @@ -95,7 +96,9 @@ import type { CanvasProjectMutationPort } from '@workbench/canvasProjectMutation import { createItemId } from '@platform/browser/randomUuid'; import { areJsonValuesStructurallyEqual } from '@platform/core/json'; +import { ControlDerivationController } from '@workbench/canvas-engine/controllers/controlDerivationController'; import { PixelEditController } from '@workbench/canvas-engine/controllers/controlPixelController'; +import { ControlViewController } from '@workbench/canvas-engine/controllers/controlViewController'; import { EditingController } from '@workbench/canvas-engine/controllers/editingController'; import { FilterResultController } from '@workbench/canvas-engine/controllers/filterResultController'; import { GeneratedResultController } from '@workbench/canvas-engine/controllers/generatedResultController'; @@ -271,8 +274,6 @@ export interface CanvasEngineOptions { uploadIntermediateImage(blob: Blob): Promise<{ height: number; imageName: string; width: number }>; /** Supplies the currently selected model base for core-created control layer contracts. */ getMainModelBase?: () => string | null; - /** Supplies the default control model key for core-created control layer contracts. */ - getDefaultControlModel?: (base: string | null) => string | null; /** Supplies the Layers panel's transient multi-selection for grouped moves. */ getSelectedLayerIds?: () => readonly string[]; /** Publishes engine history's transient multi-selection changes back to the Layers panel. */ @@ -317,6 +318,8 @@ export interface CanvasEngineInternalExportCapability extends CanvasEngineExport /** Private engine composition shape used only inside the Canvas implementation and its tests. */ export interface CanvasEngineImplementation extends CanvasEngine { + /** Control-layer intent and detection-map commits, for the application's detection coordinator. */ + readonly controlDerivation: CanvasControlDerivationCapability; readonly exports: CanvasEngineInternalExportCapability; readonly stores: EngineStores; } @@ -383,6 +386,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC checkerColors: stores.checkerColors, clipToBbox: stores.clipToBbox, colorPair: stores.colorPair, + controlLayerView: stores.controlLayerView, documentEditingLocked: stores.documentEditingLocked, eraserOptions: stores.eraserOptions, gradientOptions: stores.gradientOptions, @@ -763,6 +767,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC if (renderController.previews.hasFilter(layerId)) { clearFilterPreview(layerId); } + controlView.clear((id) => id === layerId); scheduler.invalidate({ layers: [layerId] }); }; @@ -852,6 +857,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC if (renderController.previews.hasFilter(layerId)) { clearFilterPreview(layerId); } + controlView.clear((id) => id === layerId); }; // Engine-owned selection masks clip strokes and drive fill/erase. Marching ants redraw only the overlay while @@ -1502,6 +1508,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC groupSurfaces.get(scope, members, matrices, content, { frame: true }), getCheckerboardTile, getMaskPatternTile, + getControlView: () => controlView.get(), layerCache, previews: renderController.previews, raster: rasterController, @@ -1557,6 +1564,16 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC setStagedPreview, } = previewPublisher; + const controlView = new ControlViewController({ + getDocument: () => mirror.getDocument(), + invalidateLayer: (layerId) => scheduler.invalidate({ layers: [layerId] }), + isLocked: () => interactionLocked || stores.documentEditingLocked.get(), + publish: (view) => stores.controlLayerView.set(view), + rasterize: (source, document, signal) => rasterizeSource(source, rasterizeDeps(document, signal)), + reserve: (bytes) => rasterController.memory.reserveOperation(bytes, { purpose: 'layer-operation' }), + trackDetached: (bytes) => rasterController.memory.trackDetached(bytes), + }); + // ---- Document mirror ---------------------------------------------------- /** Actual last non-null selection, retained across deselect/reselect and updated by engine selections too. */ @@ -1590,6 +1607,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC // document — a swap can reuse a layer id with different content, so // pruning only "missing" ids isn't enough; drop them all. cleanup.run(clearAllFilterPreviews); + cleanup.run(() => controlView.clear()); // Drop document-scoped selection/lasso state and cancel outgoing floats; committing would target a replaced // layer. cleanup.run(() => floatingSelection.cancel()); @@ -1697,6 +1715,8 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC cleanup.run(() => invalidateLayerCache(id)); } } + // Any edit of the viewed layer (undo, redo and a detection commit included) returns it to its map. + cleanup.run(() => controlView.reconcile()); cleanup.run(() => scheduler.invalidate({ layers: ids })); cleanup.throwIfFailed(); }, @@ -1723,6 +1743,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC // A float belongs to exactly one layer; selecting another banks it rather // than leaving pixels in flight over a layer the user is no longer on. cleanup.run(() => floatingSelection.commit()); + cleanup.run(() => controlView.clear((id) => id !== selectedLayerId)); const session = stores.transformSession.get(); if (session && session.layerId !== selectedLayerId) { cleanup.run(() => cancelTransform()); @@ -1797,6 +1818,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC for (const layerId of ids) { cleanup.run(() => clearFilterPreview(layerId)); } + cleanup.run(() => controlView.clear()); projectWasPresent = false; cleanup.throwIfFailed(); return; @@ -1831,6 +1853,12 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC const unsubscribeBrushOptions = subscribeSizeChange(stores.brushOptions, 'brush'); const unsubscribeEraserOptions = subscribeSizeChange(stores.eraserOptions, 'eraser'); const unsubscribeCheckerboard = stores.checkerboard.subscribe(() => scheduler.invalidate({ all: true })); + // An operation taking the document (Filter, Select Object) works on the map; the original view ends with it. + const unsubscribeControlViewLock = stores.documentEditingLocked.subscribe(() => { + if (stores.documentEditingLocked.get()) { + controlView.clear(); + } + }); const unsubscribeCheckerColors = stores.checkerColors.subscribe(() => { checkerboardTile = null; scheduler.invalidate({ all: true }); @@ -2021,6 +2049,8 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC } interactionLocked = locked; if (locked) { + // A generation or staging holds the canvas; the user cannot switch the view back meanwhile. + controlView.clear(); pipeline.cancelActiveGesture(); setTool('view', { temporary: true }); } @@ -2062,6 +2092,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC editingController.cooldown(); clearStagedPreview(); clearStagedPreviewCache(); + controlView.clear(); detach(); cancelAllLayerRasterizations(); rasterController.cancelSpatialAdjustments(); @@ -2178,13 +2209,6 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC const getMainModelBase = (): string | null => { return opts.getMainModelBase?.() ?? null; }; - const getDefaultControlModel = (base: string | null): string | null => { - return opts.getDefaultControlModel?.(base) ?? null; - }; - const getControlModel = (): { base: string | null; model: string | null } => { - const base = getMainModelBase(); - return { base, model: getDefaultControlModel(base) }; - }; /** Conversion reducers clone contracts, so their publication postcondition compares by value. */ const documentHasLayerContract = ( @@ -2230,7 +2254,15 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC } }, }); - const beginPixelEdit = pixelEditController.begin.bind(pixelEditController); + const pixelEdits = pixelEditController; + /** Strokes paint the map live, so a stroke on a viewed layer first returns it to the map. */ + const beginPixelEdit: PixelEditController['begin'] = (layerId, options) => { + const transaction = pixelEdits.begin(layerId, options); + if (transaction) { + controlView.clear((id) => id === layerId); + } + return transaction; + }; const captureLayerCache = ( layer: CanvasLayerContract, @@ -2307,7 +2339,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC ctx: mutationContext, discardPersisted: (layerId) => bitmapStore.discardLayer(layerId), exportBaked: (layerId) => exportBakedLayerPixelsForStructural(layerId, { includeDisabled: true }), - getControlModel, + getMainModelBase, getDuplicateRasterPlan, getSelectedLayerIds: resolveSelectedLayerIds, hasPendingPixelWork: (layerId) => bitmapStore.hasPendingWork(layerId), @@ -2344,19 +2376,27 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC ctx: mutationContext, decodeImage: (image, options) => rasterController.decodeImage(image, options), discardPersisted: (layerId) => bitmapStore.discardLayer(layerId), - getDefaultControlModel, getMainModelBase, needsPixelPersistence: layerNeedsPixelPersistence, }); const commitRasterFilterResult = filterResultController.commit.bind(filterResultController); + const controlDerivationController = new ControlDerivationController({ + captureCache: captureLayerCache, + ctx: mutationContext, + decodeImage: (image, options) => rasterController.decodeImage(image, options), + discardPersisted: (layerId) => bitmapStore.discardLayer(layerId), + getDocumentRevision: () => mutationPort.getCanvasState()?.documentRevision ?? null, + hasPendingPixelWork: (layerId) => bitmapStore.hasPendingWork(layerId), + isLayerSessionHeld: isLayerHeldBySession, + }); + const generatedResultController = new GeneratedResultController({ captureCache: captureLayerCache, clearPreview: clearFilterPreview, ctx: mutationContext, decodeImage: (image, options) => rasterController.decodeImage(image, options), discardPersisted: (layerId) => bitmapStore.discardLayer(layerId), - getDefaultControlModel, getMainModelBase, needsPixelPersistence: layerNeedsPixelPersistence, }); @@ -2533,6 +2573,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC cleanup.run(cancelOpenPixelEdit); cleanup.run(() => pixelEditController?.dispose()); cleanup.run(() => filterResultController.dispose()); + cleanup.run(() => controlDerivationController.dispose()); cleanup.run(() => generatedResultController.dispose()); cleanup.run(() => stagedResultController.dispose()); cleanup.run(() => editingController.dispose()); @@ -2553,6 +2594,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC cleanup.run(unsubscribeBrushOptions); cleanup.run(unsubscribeEraserOptions); cleanup.run(unsubscribeCheckerboard); + cleanup.run(unsubscribeControlViewLock); cleanup.run(unsubscribeCheckerColors); cleanup.run(unsubscribeShowGrid); cleanup.run(unsubscribeBboxGrid); @@ -2568,6 +2610,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC cleanup.run(() => persistenceController.dispose()); cleanup.run(() => mirror.dispose()); cleanup.run(() => previewPublisher.dispose()); + cleanup.run(() => controlView.dispose()); cleanup.run(() => renderController.dispose()); cleanup.run(() => rasterController.dispose()); cleanup.run(() => fontLoader.dispose()); @@ -2735,7 +2778,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC commitLayerCopy, ctx: mutationContext, exportBaked: (layerId) => exportBakedLayerPixelsForStructural(layerId, { includeDisabled: true }), - getControlModel, + getMainModelBase, isGuardCurrent: isLayerExportGuardCurrent, }, extractMaskedArea: { @@ -2972,6 +3015,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC ...layerController.previews, drawDocumentOverview, preloadStagedPreview, + setControlLayerView: (layerId, view) => controlView.set(layerId, view), setGuardedFilterPreview, setStagedPreview, }; @@ -2992,6 +3036,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC encodeSurface: (surface) => backend.encodeSurface(surface, 'image/png'), exportBakedLayerBlob: (layerId) => exportBakedLayerBlob(layerId, { includeDisabled: true }), exportLayerPixels: rasterizeLayerPixelsForStructural, + flushLayerUploads: (layerId, options) => persistenceController.flushLayer(layerId, options), getCompositeExecutorDeps, getDocument: () => mirror.getDocument(), isGuardCurrent: isLayerExportGuardCurrent, @@ -3037,6 +3082,7 @@ export const createCanvasEngine = (opts: CanvasEngineOptions): CanvasEngineCoreC }; const engine: CanvasEngineImplementation = { + controlDerivation: controlDerivationController, diagnostics: diagnosticsCapability, document: documentCapability, edits: editingController.edits, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/engineStores.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/engineStores.ts index 734eef51393..7943b0056ea 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/engineStores.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/engineStores.ts @@ -267,6 +267,12 @@ export const DEFAULT_ERASER_OPTIONS: Readonly = { size: 50, }; +/** The control layer displayed as its kept original, alone or under its map. */ +export interface ControlLayerView { + layerId: string; + view: 'original' | 'overlay'; +} + /** * Transform sessions span gestures until Apply/Cancel. `startTransform` is the committed undo/cancel baseline; * `transform` drives preview and numeric controls. @@ -533,6 +539,8 @@ export interface EngineStores { documentEditingLocked: ScalarStore; /** The layer targeted by that operation, or null when no operation is active. */ documentEditingLayerId: ScalarStore; + /** The control layer displayed as its original (or original under map); null displays every map. */ + controlLayerView: ScalarStore; } const brushOptionsEqual = (a: BrushOptions, b: BrushOptions): boolean => @@ -698,6 +706,7 @@ export const createEngineStores = (initialTool: ToolId = 'view'): EngineStores = checkerboard: createScalarStore(true), clipToBbox: createScalarStore(false), checkerColors: createScalarStore({ ...DEFAULT_CHECKER_COLORS }, checkerColorsEqual), + controlLayerView: createScalarStore(null), cursor: createScalarStore('default'), eraserOptions: createScalarStore({ ...DEFAULT_ERASER_OPTIONS }, eraserOptionsEqual), documentEditingLayerId: createScalarStore(null), diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/fontReferences.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/fontReferences.test.ts index 788df6c41f7..98945bd6b86 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/fontReferences.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/fontReferences.test.ts @@ -123,6 +123,19 @@ describe('canvas font references', () => { expect(noOp).toMatchObject({ replacedCount: 0, document }); expect(sameCanvasTextFontRef(from, { ...from })).toBe(true); }); + it('raises a v3 document to schema 4 and keeps a v5 document at 5 when it remaps a font', () => { + const remap = (version: 3 | 4 | 5) => + replaceCanvasFontReferences({ ...documentFrom([textLayer('text', from)]), version }, from, { + fontRef: to, + style: 'normal', + weight: 400, + axes: [], + }).document.version; + + expect(remap(3)).toBe(4); + expect(remap(5)).toBe(5); + }); + it('uses target axis defaults when replacing a static face and synchronizes CSS typography', () => { const document = documentFrom([textLayer('text', from)]); const result = replaceCanvasFontReferences(document, from, { diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.budget.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.budget.test.ts index 95b6f69fe20..e21b5f9bbe4 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.budget.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.budget.test.ts @@ -53,6 +53,7 @@ const scene = (budgetBytes = SURFACE_BYTES * 2) => { diagnostics, getAdjustedSurface: (layer, entry) => raster.getAdjustedSurface(layer, entry), getCheckerboardTile: () => backend.createSurface(1, 1), + getControlView: () => null, getGroupSurface: (scope, members, matrices, content) => raster.groups.get(scope, members, matrices, content), getMaskPatternTile: () => null, layerCache: raster.layers, @@ -122,6 +123,7 @@ describe('frame description over 2,000 nodes', () => { diagnostics: createCanvasDiagnostics(true), getAdjustedSurface: () => null, getCheckerboardTile: () => backend.createSurface(1, 1), + getControlView: () => null, getGroupSurface: () => null, getMaskPatternTile: () => null, layerCache: raster.layers, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.test.ts index d11ec3f2254..0696d76bcbc 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.test.ts @@ -46,6 +46,7 @@ const documentOf = (layers: CanvasLayerContract[]): CanvasDocumentContractV3 => interface Harness { deps: CreateCompositeFrameDeps; + controlView: ReturnType; stores: ReturnType; previews: PreviewStateController; overrides: Map; @@ -70,8 +71,9 @@ const makeHarness = (): Harness => { const rasterizeLayer = vi.fn(); const beginFrame = vi.fn(() => FRAME_USAGE); const enforce = vi.fn(() => ({ evictedBaseLayerIds: [] as string[], overageBytes: 0 })); - return { + const harness: Harness = { beginFrame, + controlView: null, deps: { backend: {} as CreateCompositeFrameDeps['backend'], derivedSurfaceCache: { byteSize: () => 0 } as CreateCompositeFrameDeps['derivedSurfaceCache'], @@ -79,6 +81,7 @@ const makeHarness = (): Harness => { getAdjustedSurface: () => null, getGroupSurface: () => null, getCheckerboardTile: () => surface(8, 8), + getControlView: () => harness.controlView, getMaskPatternTile: () => null, layerCache: { byteSize: () => 0, @@ -105,6 +108,7 @@ const makeHarness = (): Harness => { rasterizeLayer, stores, }; + return harness; }; const draw = ( @@ -161,7 +165,7 @@ describe('SAM isolation', () => { expect(lastCompositeOptions().clipRect).toEqual({ height: 8, width: 8, x: 0, y: 0 }); }); - it.each(['layerPreviews', 'stagedPreview', 'floatingSelection', 'transformOverrides'])( + it.each(['layerPreviews', 'stagedPreview', 'floatingSelection', 'controlView', 'transformOverrides'])( 'suppresses %s so nothing but the isolated layer is judged', (field) => { harness.previews.publishFilter('a', harness.previews.beginGuardedFilter('a'), { @@ -175,6 +179,7 @@ describe('SAM isolation', () => { width: 8, } as never); harness.overrides.set('a', { x: 5, y: 5 }); + harness.controlView = controlViewOf('a'); const float = { composite: { layerId: 'a' } } as unknown as FloatingSelectionFrame; draw(documentOf([layer('a')]), float, isolated()); expect(lastCompositeOptions()[field]).toBeNull(); @@ -190,6 +195,26 @@ describe('SAM isolation', () => { }); }); +const controlViewOf = (layerId: string): Harness['controlView'] => ({ + layerId, + matrix: { a: 1, b: 0, c: 0, d: 1, e: 0, f: 0 }, + rect: { height: 8, width: 8, x: 0, y: 0 }, + surface: surface(8, 8), + view: 'original', +}); + +describe('the control layer view', () => { + it('hands the shown original to the compositor', () => { + draw(); + expect(lastCompositeOptions().controlView).toBeNull(); + + const view = controlViewOf('a'); + harness.controlView = view; + draw(); + expect(lastCompositeOptions().controlView).toBe(view); + }); +}); + describe('filter previews', () => { it('allocates no snapshot when no layer carries one', () => { const snapshot = vi.spyOn(harness.previews, 'filterSnapshot'); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.ts index 1801a5fb0a9..4b142a946b0 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositeFrame.ts @@ -42,6 +42,8 @@ export interface CreateCompositeFrameDeps { readonly getGroupSurface: NonNullable; readonly getMaskPatternTile: (style: string, color: string) => RasterSurface | null; readonly getCheckerboardTile: () => RasterSurface; + /** The control layer shown as its kept original, if any. */ + readonly getControlView: () => CompositeOptions['controlView']; /** Starts (or joins) the rasterization of a layer whose cache is stale. */ readonly rasterizeLayer: (layer: CanvasLayerContract, doc: CanvasDocumentContractV3) => void; } @@ -74,8 +76,8 @@ const sameOptionalRect = (a: Rect | undefined, b: Rect | undefined): boolean => /** * Composite only for pixel/order/view changes. Rasterize demanded visible layers and protect them during post-draw - * eviction. SAM isolation restricts drawing and clipping to its layer/rect, suppressing unrelated previews, floats - * and transform overrides. + * eviction. SAM isolation restricts drawing and clipping to its layer/rect, suppressing unrelated previews, floats, + * control views and transform overrides. */ export const createCompositeFrame = (deps: CreateCompositeFrameDeps): CompositeFrame => { const { derivedSurfaceCache, diagnostics, layerCache, previews, raster, stores, transformOverrides, viewport } = deps; @@ -211,6 +213,7 @@ export const createCompositeFrame = (deps: CreateCompositeFrameDeps): CompositeF backend: deps.backend, checkerboardTile: stores.checkerboard.get() ? deps.getCheckerboardTile() : null, clipRect: isolatedGuard && samPreview ? samPreview.rect : null, + controlView: isolatedGuard ? null : deps.getControlView(), derivedSurfaces: derivedSurfaceCache, diagnostics, // Pixels in flight, drawn directly above the layer they were cut from. diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.test.ts index c8cc7a84ef1..0fab7dbdc6e 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.test.ts @@ -919,3 +919,133 @@ describe('compositeDocument — damage', () => { expect(groupSurface).not.toHaveBeenCalled(); }); }); + +describe('compositeDocument — control layer view', () => { + // The original sits 4px right of the map's frame and is drawn at half size. + const ORIGINAL_MATRIX: Mat2d = { a: 0.5, b: 0, c: 0, d: 0.5, e: 4, f: 0 }; + const LAYER_AT = { rotation: 0, scaleX: 1, scaleY: 1, x: 10, y: 20 }; + + /** Each `drawImage` with the alpha, blend and transform in effect when it ran. */ + const draws = (log: RasterCallLogEntry[]) => { + let alpha: unknown = 1; + let blend: unknown = 'source-over'; + let transform: unknown[] = []; + const result: { source: unknown; alpha: unknown; blend: unknown; transform: unknown[] }[] = []; + for (const entry of log) { + if (entry.op === 'set' && entry.args[0] === 'globalAlpha') { + alpha = entry.args[1]; + } else if (entry.op === 'set' && entry.args[0] === 'globalCompositeOperation') { + blend = entry.args[1]; + } else if (entry.op === 'setTransform') { + transform = entry.args; + } else if (entry.op === 'drawImage') { + result.push({ alpha, blend, source: entry.args[0], transform }); + } + } + return result; + }; + + const scene = (view: 'original' | 'overlay', layerOverrides: Record = {}) => { + const backend = createTestStubRasterBackend(); + const caches = createLayerCacheStore(backend); + const map = caches.getOrCreate('control', 10, 10); + const original = backend.createSurface(40, 40); + const layer = { + ...controlLayer('control'), + blendMode: 'multiply', + opacity: 0.8, + transform: LAYER_AT, + withTransparencyEffect: true, + ...layerOverrides, + } as CanvasLayerContract; + const target = backend.createSurface(200, 200); + const controlView = { + layerId: 'control', + matrix: ORIGINAL_MATRIX, + rect: { height: 40, width: 40, x: 0, y: 0 }, + surface: original, + view, + }; + return { backend, caches, controlView, layer, map, original, target }; + }; + + it('draws the original through the layer and original transforms in place of the map, without the effect', () => { + const { backend, caches, controlView, layer, original, target } = scene('original'); + const derivedSurfaces = createDerivedSurfaceCache(); + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, controlView, derivedSurfaces }); + + expect(draws(target.callLog)).toEqual([ + { alpha: 0.8, blend: 'multiply', source: original.canvas, transform: [0.5, 0, 0, 0.5, 14, 20] }, + ]); + // The transparency effect is a property of the map; it never runs on the original. + expect(derivedSurfaces.byteSize()).toBe(0); + }); + + it('shows the original of a layer that has no map yet', () => { + const { backend, controlView, layer, original, target } = scene('original'); + + compositeDocument(target, makeDoc([layer]), createLayerCacheStore(backend), VIEW, { backend, controlView }); + + expect(draws(target.callLog).map((draw) => draw.source)).toEqual([original.canvas]); + }); + + it('draws the map over the original at half the layer opacity in the overlay view', () => { + const { backend, caches, controlView, layer, map, original, target } = scene('overlay', { + withTransparencyEffect: false, + }); + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, controlView }); + + expect(draws(target.callLog)).toEqual([ + { alpha: 0.8, blend: 'multiply', source: original.canvas, transform: [0.5, 0, 0, 0.5, 14, 20] }, + { alpha: 0.4, blend: 'multiply', source: map.surface.canvas, transform: [1, 0, 0, 1, 10, 20] }, + ]); + }); + + it('keeps the transparency effect on the map in the overlay view', () => { + const { backend, caches, controlView, layer, map, original, target } = scene('overlay'); + const derivedSurfaces = createDerivedSurfaceCache(); + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, controlView, derivedSurfaces }); + + const [under, over] = draws(target.callLog); + expect(under?.source).toBe(original.canvas); + // The map draws as its transparency-effect copy, not its raw cache. + expect(over?.source).not.toBe(map.surface.canvas); + expect(over?.alpha).toBe(0.4); + expect(derivedSurfaces.byteSize()).toBeGreaterThan(0); + }); + + it('repaints damage that reaches only the original, beyond the map', () => { + const { backend, caches, controlView, layer, original, target } = scene('original'); + // The map covers document x 10..20; the original reaches x 14..34. + const damage = { + kind: 'regions' as const, + regions: [{ layerId: 'control', rect: { height: 4, width: 4, x: 20, y: 2 } }], + }; + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, damage }); + expect(draws(target.callLog)).toEqual([]); + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, controlView, damage }); + expect(draws(target.callLog).map((draw) => draw.source)).toEqual([original.canvas]); + }); + + it('shows a grouped control layer the same way', () => { + const { backend, caches, controlView, layer, original, target } = scene('original'); + const doc = makeDoc([], { stacks: stacksFrom([groupContract('g', [layer])]) }); + + compositeDocument(target, doc, caches, VIEW, { backend, controlView, groupSurface: () => null }); + + expect(draws(target.callLog).map((draw) => draw.source)).toEqual([original.canvas]); + }); + + it('is suppressed while a layer is isolated', () => { + const { backend, caches, controlView, layer, map, target } = scene('original', { withTransparencyEffect: false }); + + compositeDocument(target, makeDoc([layer]), caches, VIEW, { backend, controlView, isolationLayerId: 'control' }); + + expect(draws(target.callLog).map((draw) => draw.source)).toEqual([map.surface.canvas]); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.ts index d061d94dc2e..5ec152f2d58 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/render/compositor.ts @@ -56,6 +56,9 @@ export const DEFAULT_CHECKER_COLORS: CheckerColors = { a: '#2a2a2a', b: '#363636 /** Opacity of the display-only regenerate tint drawn over a raster layer's coverage. */ export const REGION_OVERLAY_ALPHA = 0.5; +/** Strength of a control layer's map over its original in the Overlay view, relative to the layer's opacity. */ +export const CONTROL_OVERLAY_MAP_ALPHA = 0.5; + /** Dashed outline drawn around a staged-generation preview so it reads as pending, not committed. */ const STAGED_PREVIEW_OUTLINE_COLOR = '#3b82f6'; const STAGED_PREVIEW_OUTLINE_WIDTH = 2; @@ -120,6 +123,17 @@ export interface CompositeOptions { * matrix are layer-local and compose with the layer transform. */ floatingSelection?: { layerId: string; surface: RasterSurface; rect: Rect; matrix: Mat2d } | null; + /** + * Display-only control layer view: the kept original (layer-local through `matrix`, at the layer's opacity and + * blend) in place of the map, or under the map at {@link CONTROL_OVERLAY_MAP_ALPHA}. Suppressed under isolation. + */ + controlView?: { + layerId: string; + view: 'original' | 'overlay'; + surface: RasterSurface; + rect: Rect; + matrix: Mat2d; + } | null; /** Optional deterministic render counters; omitted in the normal zero-overhead path. */ diagnostics?: CanvasDiagnostics | null; /** @@ -367,11 +381,12 @@ const drawCachedLayer = ( matrix: Mat2d, entry: LayerCacheEntry, view: Mat2d, - opts: CompositeOptions + opts: CompositeOptions, + alpha = 1 ): void => { const { layer } = leaf; ctx.save(); - ctx.globalAlpha = layer.opacity; + ctx.globalAlpha = layer.opacity * alpha; ctx.globalCompositeOperation = blendToComposite(layer.blendMode); setTransformFromMat(ctx, multiply(view, matrix)); @@ -469,27 +484,46 @@ const drawRegionCoverage = ( }; /** - * Draws the floating selection over the layer it was cut from. Its pixels are - * layer-local, so they go through the layer's (possibly overridden) matrix and - * then the float's own — never resampled into document space and back. + * Draws layer-local pixels that are not the layer's cache (a floating selection, a control layer's original) with + * the layer's opacity and blend. They go through the layer's (possibly overridden) matrix and then their own, + * never resampled into document space and back. */ -const drawFloatingSelection = ( +const drawLayerLocalSurface = ( ctx: Ctx, leaf: SemanticLeaf, matrix: Mat2d, view: Mat2d, - float: NonNullable + pixels: { surface: RasterSurface; rect: Rect; matrix: Mat2d } ): void => { const { layer } = leaf; ctx.save(); ctx.globalAlpha = layer.opacity; ctx.globalCompositeOperation = blendToComposite(layer.blendMode); - setTransformFromMat(ctx, multiply(multiply(view, matrix), float.matrix)); - ctx.drawImage(float.surface.canvas, float.rect.x, float.rect.y); + setTransformFromMat(ctx, multiply(multiply(view, matrix), pixels.matrix)); + ctx.drawImage(pixels.surface.canvas, pixels.rect.x, pixels.rect.y); ctx.restore(); }; -/** Document-space bounds a leaf can cover this frame: committed pixels joined with its preview and float. */ +/** `bounds` joined with where layer-local `pixels` land through the leaf's matrix. */ +const unionLanding = ( + bounds: Rect | null, + matrix: Mat2d, + pixels: { rect: Rect; matrix: Mat2d } | null | undefined +): Rect | null => { + if (!pixels || isEmpty(pixels.rect)) { + return bounds; + } + const landing = transformBounds(multiply(matrix, pixels.matrix), pixels.rect); + return bounds ? union(bounds, landing) : landing; +}; + +const controlViewOf = (leaf: SemanticLeaf, opts: CompositeOptions): CompositeOptions['controlView'] => + !isIsolated(opts) && leaf.layer.type === 'control' && opts.controlView?.layerId === leaf.id ? opts.controlView : null; + +/** + * Document-space bounds a leaf can cover this frame: committed pixels joined with its preview, float and shown + * control original. + */ const leafBounds = (leaf: SemanticLeaf, matrix: Mat2d, committed: Rect | null, opts: CompositeOptions): Rect | null => { const preview = isIsolated(opts) ? null : (opts.layerPreviews?.get(leaf.id) ?? null); let bounds = committed; @@ -498,11 +532,7 @@ const leafBounds = (leaf: SemanticLeaf, matrix: Mat2d, committed: Rect | null, o bounds = bounds ? union(bounds, previewBounds) : previewBounds; } const float = !isIsolated(opts) && opts.floatingSelection?.layerId === leaf.id ? opts.floatingSelection : null; - if (float && !isEmpty(float.rect)) { - const landing = transformBounds(multiply(matrix, float.matrix), float.rect); - bounds = bounds ? union(bounds, landing) : landing; - } - return bounds; + return unionLanding(unionLanding(bounds, matrix, float), matrix, controlViewOf(leaf, opts)); }; /** Whether document-space `bounds` reach the screen region; non-finite geometry counts as visible. */ @@ -580,13 +610,18 @@ export const compositeDocument = ( } const matrix = matrices[index]!; const entry = caches.get(leaf.id); + // The original replaces the map, or lies under it in the Overlay view; it shows even when there is no map. + const controlView = controlViewOf(leaf, opts); + if (controlView) { + drawLayerLocalSurface(ctx, leaf, matrix, view, controlView); + } // A float still draws over an emptied source: its pixels are detached. - if (entry && entry.rect.width > 0 && entry.rect.height > 0) { - drawCachedLayer(ctx, leaf, matrix, entry, view, opts); + if (entry && entry.rect.width > 0 && entry.rect.height > 0 && controlView?.view !== 'original') { + drawCachedLayer(ctx, leaf, matrix, entry, view, opts, controlView ? CONTROL_OVERLAY_MAP_ALPHA : 1); opts.diagnostics?.increment('layersDrawn'); } if (float?.layerId === leaf.id) { - drawFloatingSelection(ctx, leaf, matrix, view, float); + drawLayerLocalSurface(ctx, leaf, matrix, view, float); } }; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.test.ts index 5b51333dbc9..f5c416df688 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.test.ts @@ -8,6 +8,7 @@ import { applyRotate, applyScale, bakeMatrix, + composeLayerTransforms, IDENTITY_TRANSFORM, layerTransformMatrix, localHandlePoint, @@ -505,3 +506,92 @@ describe('IDENTITY_TRANSFORM', () => { expect(IDENTITY_TRANSFORM).toEqual({ rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }); }); }); + +describe('composeLayerTransforms', () => { + const points = [ + { x: 0, y: 0 }, + { x: 13, y: -7 }, + { x: -4.5, y: 21 }, + ]; + const expectSameMapping = (composed: LayerTransform | null, outer: LayerTransform, inner: LayerTransform) => { + expect(composed).not.toBeNull(); + for (const point of points) { + const expected = applyToPoint(layerTransformMatrix(outer), applyToPoint(layerTransformMatrix(inner), point)); + const actual = applyToPoint(layerTransformMatrix(composed!), point); + expect(actual.x).toBeCloseTo(expected.x, 9); + expect(actual.y).toBeCloseTo(expected.y, 9); + } + }; + + it('is neutral with the identity on either side', () => { + const t = { rotation: 0.4, scaleX: 2, scaleY: 0.5, x: 10, y: -3 }; + for (const composed of [ + composeLayerTransforms(IDENTITY_TRANSFORM, t), + composeLayerTransforms(t, IDENTITY_TRANSFORM), + ]) { + expect(composed!.rotation).toBeCloseTo(t.rotation, 12); + expect(composed!.scaleX).toBeCloseTo(t.scaleX, 12); + expect(composed!.scaleY).toBeCloseTo(t.scaleY, 12); + expect(composed!.x).toBeCloseTo(t.x, 12); + expect(composed!.y).toBeCloseTo(t.y, 12); + } + }); + + it('scales the inner translation by the outer transform and adds rotations', () => { + expect( + composeLayerTransforms( + { rotation: 0, scaleX: 2, scaleY: 3, x: 5, y: 7 }, + { rotation: 0, scaleX: 1, scaleY: 1, x: 10, y: 10 } + ) + ).toEqual({ rotation: 0, scaleX: 2, scaleY: 3, x: 25, y: 37 }); + const rotated = composeLayerTransforms( + { rotation: 0.3, scaleX: 1.5, scaleY: 1.5, x: 0, y: 0 }, + { rotation: 0.2, scaleX: 2, scaleY: 1, x: 0, y: 0 } + ); + expect(rotated!.rotation).toBeCloseTo(0.5, 12); + expect(rotated!.scaleX).toBeCloseTo(3, 12); + expect(rotated!.scaleY).toBeCloseTo(1.5, 12); + }); + + it('keeps a flip a flip instead of turning it into a half-turn', () => { + const flip = { rotation: 0, scaleX: -1, scaleY: 1, x: 0, y: 0 }; + const composed = composeLayerTransforms(IDENTITY_TRANSFORM, flip); + expect(composed!.scaleX).toBe(-1); + expect(composed!.scaleY).toBe(1); + expect(composed!.rotation).toBeCloseTo(0, 12); + }); + + it.each([ + [ + 'uniform scale around a rotation', + { rotation: 1.1, scaleX: 2, scaleY: 2, x: 4, y: -9 }, + { rotation: -0.7, scaleX: 0.5, scaleY: 3, x: -6, y: 2 }, + ], + [ + 'flips on both sides', + { rotation: 0.25, scaleX: -1.5, scaleY: 1.5, x: 1, y: 2 }, + { rotation: 2.5, scaleX: 1, scaleY: -2, x: 3, y: 4 }, + ], + [ + 'non-uniform scale over an unrotated inner', + { rotation: 0.9, scaleX: 3, scaleY: 0.25, x: 0, y: 8 }, + { rotation: 0, scaleX: -2, scaleY: 5, x: 7, y: 1 }, + ], + [ + 'a quarter turn swapping non-uniform axes', + { rotation: 0, scaleX: 2, scaleY: 3, x: 0, y: 0 }, + { rotation: Math.PI / 2, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + ], + ] as const)('maps points exactly like inner then outer: %s', (_label, outer, inner) => { + expectSameMapping(composeLayerTransforms(outer, inner), outer, inner); + }); + + it('refuses a composite that shears', () => { + expect( + composeLayerTransforms( + { rotation: 0, scaleX: 2, scaleY: 1, x: 0, y: 0 }, + { rotation: Math.PI / 4, scaleX: 1, scaleY: 1, x: 0, y: 0 } + ) + ).toBeNull(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.ts b/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.ts index 80ea399960a..64cddea3291 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-engine/transform/transformMath.ts @@ -7,7 +7,7 @@ import type { CanvasLayerBaseContract } from '@workbench/canvas-engine/contracts'; import type { Mat2d, Vec2 } from '@workbench/canvas-engine/types'; -import { applyToPoint, fromTRS } from '@workbench/canvas-engine/math/mat2d'; +import { applyToPoint, fromTRS, multiply } from '@workbench/canvas-engine/math/mat2d'; import { snapMovedPoint } from '@workbench/canvas-engine/math/snapping'; /** The editable layer transform (translate + per-axis scale + rotation in radians). */ @@ -390,3 +390,29 @@ export const bakeMatrix = (transform: LayerTransform): Mat2d => layerTransformMa /** The identity transform (a reset layer transform after a bake). */ export const IDENTITY_TRANSFORM: LayerTransform = { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }; + +/** Relative tolerance for treating a composite's axes as perpendicular. */ +const SHEAR_TOLERANCE = 1e-9; + +/** + * The transform that applies `inner`, then `outer`; null when the composite shears (a non-uniform scale outside a + * rotation) or collapses an axis, which translate/scale/rotate cannot express. A composite flip keeps the sign of + * `outer.scaleX × inner.scaleX` on `scaleX`, so composing a flip with the identity returns the flip rather than a + * half-turn. + */ +export const composeLayerTransforms = (outer: LayerTransform, inner: LayerTransform): LayerTransform | null => { + const { a, b, c, d, e, f } = multiply(layerTransformMatrix(outer), layerTransformMatrix(inner)); + const xAxis = Math.hypot(a, b); + const yAxis = Math.hypot(c, d); + if (xAxis === 0 || yAxis === 0 || Math.abs(a * c + b * d) > SHEAR_TOLERANCE * xAxis * yAxis) { + return null; + } + const scaleX = Math.sign(outer.scaleX * inner.scaleX) < 0 ? -xAxis : xAxis; + return { + rotation: Math.atan2(b / scaleX, a / scaleX), + scaleX, + scaleY: (a * d - b * c) / scaleX, + x: e, + y: f, + }; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.test.ts new file mode 100644 index 00000000000..5ef0f624bc9 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.test.ts @@ -0,0 +1,654 @@ +import type { ControlMapCommitResult } from '@workbench/canvas-engine/api'; +import type { CanvasControlLayerContract, CanvasDocumentContractV3 } from '@workbench/canvas-engine/contracts'; +import type { LayerFlushResult } from '@workbench/canvas-engine/controllers/persistenceController'; +import type { BackendGraphContract } from '@workbench/graphContracts'; + +import { createAccountLifecycle } from '@platform/state/accountLifecycle'; +import { controlDetectionInputKey, createControlLayer, isControlMapCurrent } from '@workbench/canvas-engine/api'; +import { documentFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { withDetectedMap } from '@workbench/canvas-engine/document/controlDerivation'; +import { createCanvasEditGate, type CanvasEditLease } from '@workbench/canvas-engine/editGate'; +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +import { CONTROL_DETECTION_COMMIT_TIMEOUT_MS, createControlDetectionCoordinator } from './ControlDetectionCoordinator'; + +const DEBOUNCE_MS = 400; +const image = (imageName: string) => ({ height: 64, imageName, width: 64 }); +const photoLayer = (id: string, patch: Partial = {}): CanvasControlLayerContract => ({ + ...createControlLayer({ id, name: id }), + source: { image: image(`${id}.png`), type: 'image' }, + ...patch, +}); + +interface PendingGraph { + imageName: string; + signal: AbortSignal; + resolve(output: { height: number; imageName: string; width: number }): void; + reject(error: unknown): void; +} + +const createHarness = (initial: CanvasControlLayerContract[] = [photoLayer('a')]) => { + const layers = new Map(initial.map((layer) => [layer.id, layer])); + let document: CanvasDocumentContractV3 = documentFrom([...layers.values()]); + let revision = 0; + const publish = (): void => { + document = documentFrom([...layers.values()]); + }; + const setLayer = (layer: CanvasControlLayerContract): void => { + layers.set(layer.id, layer); + publish(); + }; + const removeLayer = (id: string): void => { + layers.delete(id); + publish(); + }; + + const gate = createCanvasEditGate(); + let lease: CanvasEditLease | null = null; + let mapCommitOverride: (() => ControlMapCommitResult) | null = null; + let onCommitPublished: (() => void) | null = null; + const derivation = { + captureControlDetectionGuard: vi.fn((layerId: string) => { + const layer = layers.get(layerId); + return layer + ? { documentRevision: revision, inputKey: controlDetectionInputKey(layer), layerId, projectId: 'p1' } + : null; + }), + commitControlLayerPatch: vi.fn( + (options: { next: CanvasControlLayerContract }): { status: 'committed' } | { status: 'busy' } => { + setLayer(options.next); + return { status: 'committed' }; + } + ), + commitControlMap: vi.fn( + async (options: { + guard: { layerId: string; documentRevision: number; inputKey: string }; + image: { height: number; imageName: string; width: number }; + rect: { height: number; width: number; x: number; y: number }; + detection: { type: string; settings: Record }; + makeDurable(): Promise; + }): Promise => { + if (mapCommitOverride) { + return mapCommitOverride(); + } + // The engine refuses while a modal operation holds the edit lease. + if (lease?.isCurrent()) { + return { status: 'busy' }; + } + const layer = layers.get(options.guard.layerId); + if (!layer) { + return { status: 'missing' }; + } + if (options.guard.documentRevision !== revision || options.guard.inputKey !== controlDetectionInputKey(layer)) { + return { status: 'stale' }; + } + // Like the engine, promote only once nothing refuses the map. + try { + await options.makeDurable(); + } catch (error) { + return { message: (error as Error).message, status: 'failed' }; + } + setLayer(withDetectedMap(layer, { image: options.image, rect: options.rect }, options.detection)!); + // The engine publishes synchronously, and its document change reconciles before the commit returns. + onCommitPublished?.(); + return { status: 'committed' }; + } + ), + }; + + const graphs: PendingGraph[] = []; + const runFilterGraph = vi.fn( + (options: { graph: BackendGraphContract; signal?: AbortSignal }) => + new Promise<{ height: number; imageName: string; width: number }>((resolve, reject) => { + const flatten = options.graph.nodes.control_filter_flatten as { image?: { image_name: string } } | undefined; + graphs.push({ + imageName: flatten?.image?.image_name ?? '', + reject, + resolve, + signal: options.signal!, + }); + }) + ); + const lockListeners = new Set<() => void>(); + const accounts = createAccountLifecycle(); + const flushLayerUploads = vi.fn((_layerId: string, _options: { signal?: AbortSignal }) => + Promise.resolve('clean') + ); + const makeDurable = vi.fn((_imageName: string) => Promise.resolve()); + const coordinator = createControlDetectionCoordinator({ + accounts: { capture: accounts.capture, register: accounts.register }, + derivation, + flushLayerUploads, + getDocument: () => document, + makeDurable, + projectId: 'p1', + runFilterGraph, + subscribeLockChanges: (listener) => { + lockListeners.add(listener); + return () => lockListeners.delete(listener); + }, + }); + + let mapCount = 0; + /** Completes the oldest pending graph with a 64×64 map. */ + const finishGraph = async (index = 0): Promise => { + const [pending] = graphs.splice(index, 1); + mapCount += 1; + pending!.resolve({ height: 64, imageName: `map-${String(mapCount)}.png`, width: 64 }); + await vi.advanceTimersByTimeAsync(0); + }; + + return { + accounts, + acquireLease: () => { + lease = gate.tryAcquire({ kind: 'filter', layerId: 'raster' }); + return lease; + }, + bumpRevision: () => { + revision += 1; + }, + coordinator, + derivation, + finishGraph, + flushLayerUploads, + graphs, + layer: (id = 'a') => layers.get(id)!, + lockListeners, + makeDurable, + emitLockChange: () => { + for (const listener of lockListeners) { + listener(); + } + }, + removeLayer, + runFilterGraph, + setLayer, + setMapCommit: (override: (() => ControlMapCommitResult) | null) => { + mapCommitOverride = override; + }, + onCommitPublished: (listener: (() => void) | null) => { + onCommitPublished = listener; + }, + status: (id = 'a') => coordinator.getStatus(id), + }; +}; + +const settle = (ms = DEBOUNCE_MS) => vi.advanceTimersByTimeAsync(ms); + +beforeEach(() => { + vi.useFakeTimers(); +}); +afterEach(() => { + vi.useRealTimers(); +}); + +describe('ControlDetectionCoordinator', () => { + it('debounces intents, runs one graph on the input and commits a durable map', async () => { + const h = createHarness(); + + expect(h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' })).toBe('updated'); + expect(h.status()).toEqual({ status: 'queued' }); + await settle(DEBOUNCE_MS - 1); + expect(h.coordinator.applyIntent('a', { kind: 'set-detector-settings', settings: { low_threshold: 50 } })).toBe( + 'updated' + ); + await settle(DEBOUNCE_MS - 1); + expect(h.runFilterGraph).not.toHaveBeenCalled(); + await settle(1); + + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + expect(h.graphs[0]!.imageName).toBe('a.png'); + expect(h.status()).toEqual({ detector: 'canny_edge_detection', status: 'processing' }); + expect(h.derivation.commitControlLayerPatch).toHaveBeenLastCalledWith( + expect.objectContaining({ coalesceKey: 'a:canny_edge_detection:settings' }) + ); + + await h.finishGraph(); + + expect(h.makeDurable).toHaveBeenCalledExactlyOnceWith('map-1.png'); + expect(h.layer().source).toMatchObject({ bitmap: { imageName: 'map-1.png' }, type: 'paint' }); + expect(isControlMapCurrent(h.layer())).toBe(true); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('maps refused patches without scheduling, and settles layers that need no detection', async () => { + const h = createHarness(); + h.derivation.commitControlLayerPatch.mockReturnValueOnce({ status: 'busy' }); + + expect(h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' })).toBe('blocked'); + expect(h.coordinator.applyIntent('missing', { kind: 'draw' })).toBe('missing'); + expect(h.coordinator.applyIntent('a', { controlType: 'tile', kind: 'set-type' })).toBe('updated'); + expect(h.coordinator.applyIntent('a', { controlType: 'tile', kind: 'set-type' })).toBe('unchanged'); + await settle(); + + expect(h.runFilterGraph).not.toHaveBeenCalled(); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('aborts a running job when a newer intent changes its input, and keeps it otherwise', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const first = h.graphs.shift()!; + + expect(h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type', name: 'Edges 1' })).toBe('updated'); + await settle(); + expect(first.signal.aborted).toBe(false); + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + + h.coordinator.applyIntent('a', { detector: 'mlsd_detection', kind: 'set-detector' }); + expect(first.signal.aborted).toBe(true); + await settle(); + expect(h.runFilterGraph).toHaveBeenCalledTimes(2); + + first.resolve({ height: 64, imageName: 'late.png', width: 64 }); + await settle(0); + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + await h.finishGraph(); + expect(h.makeDurable).toHaveBeenCalledExactlyOnceWith('map-1.png'); + expect(h.layer().filter?.type).toBe('mlsd_detection'); + }); + + it.each([ + ['layer removal', (h: ReturnType) => h.removeLayer('a')], + ['a document swap', (h: ReturnType) => h.bumpRevision()], + [ + 'a different detector (undo)', + (h: ReturnType) => + h.setLayer({ ...h.layer(), detection: { settings: {}, type: 'mlsd_detection' } }), + ], + [ + 'an undo of a replaced image', + (h: ReturnType) => + h.setLayer({ ...h.layer(), source: { image: image('before-replace.png'), type: 'image' } }), + ], + ])('drops a result made stale by %s before preserving it', async (_label, staleIt) => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + + staleIt(h); + await h.finishGraph(); + await settle(); + + expect(h.makeDurable).not.toHaveBeenCalled(); + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('restarts after the layer’s own paint lands under a run, debounced and at most three times', async () => { + const paint = (name: string) => ({ bitmap: image(name), offset: { x: 0, y: 0 }, type: 'paint' }) as const; + const h = createHarness([photoLayer('a', { controlType: 'edges', detection: null, source: paint('strokes.png') })]); + h.coordinator.applyIntent('a', { detector: 'canny_edge_detection', kind: 'set-detector' }); + await settle(); + + for (let run = 1; run <= 4; run += 1) { + h.setLayer({ ...h.layer(), source: paint(`strokes-v${String(run)}.png`) }); + // The upload landing is a document change; it keeps the run, which restarts once it finishes. + h.coordinator.reconcileWithDocument(); + expect(h.graphs[0]!.signal.aborted).toBe(false); + await h.finishGraph(); + if (run < 4) { + expect(h.status()).toEqual({ status: 'queued' }); + await settle(DEBOUNCE_MS - 1); + expect(h.runFilterGraph).toHaveBeenCalledTimes(run); + await settle(1); + expect(h.graphs[0]!.imageName).toBe(`strokes-v${String(run)}.png`); + } + } + await settle(); + + expect(h.runFilterGraph).toHaveBeenCalledTimes(4); + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + expect(h.makeDurable).not.toHaveBeenCalled(); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('lets a modal operation take the edit lease during a run, then commits after it releases', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + + const lease = h.acquireLease(); + expect(lease).not.toBeNull(); + await h.finishGraph(); + expect(h.derivation.commitControlMap).toHaveBeenCalledOnce(); + expect(h.status()).toEqual({ status: 'queued' }); + + lease!.release(); + h.emitLockChange(); + await settle(0); + + expect(h.derivation.commitControlMap).toHaveBeenCalledTimes(2); + expect(h.makeDurable).toHaveBeenCalledOnce(); + expect(isControlMapCurrent(h.layer())).toBe(true); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('retries a busy commit with backoff and reports commit-busy after the timeout', async () => { + const h = createHarness(); + h.setMapCommit(() => ({ status: 'busy' })); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + await h.finishGraph(); + + const attempts = (): number => h.derivation.commitControlMap.mock.calls.length; + expect(attempts()).toBe(1); + await settle(249); + expect(attempts()).toBe(1); + await settle(1); + expect(attempts()).toBe(2); + await settle(500); + expect(attempts()).toBe(3); + await settle(1_000); + expect(attempts()).toBe(4); + await settle(2_000); + expect(attempts()).toBe(5); + expect(h.status()).toEqual({ status: 'queued' }); + + await settle(CONTROL_DETECTION_COMMIT_TIMEOUT_MS); + expect(h.status()).toEqual({ error: { code: 'commit-busy' }, status: 'error' }); + // A result that never landed stays an intermediate the server collects. + expect(h.makeDurable).not.toHaveBeenCalled(); + const final = attempts(); + await settle(10_000); + expect(attempts()).toBe(final); + }); + + it.each([ + [{ message: 'too big', status: 'over-budget' } as const, { code: 'over-budget', detail: 'too big' }], + [{ status: 'unrepresentable' } as const, { code: 'unrepresentable' }], + [{ message: 'decode failed', status: 'failed' } as const, { code: 'apply-failed', detail: 'decode failed' }], + ])('reports a %o commit as a typed error', async (result, error) => { + const h = createHarness(); + h.setMapCommit(() => result); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + await h.finishGraph(); + + expect(h.status()).toEqual({ error, status: 'error' }); + }); + + it('reports graph failures and unknown detectors, and Retry runs again', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + h.graphs.shift()!.reject(new Error('queue offline')); + await settle(0); + expect(h.status()).toEqual({ error: { code: 'process-failed', detail: 'queue offline' }, status: 'error' }); + + expect(h.coordinator.retry('a')).toBe('scheduled'); + await settle(0); + expect(h.runFilterGraph).toHaveBeenCalledTimes(2); + await h.finishGraph(); + expect(h.coordinator.retry('a')).toBe('not-needed'); + expect(h.coordinator.retry('missing')).toBe('missing'); + + h.coordinator.applyIntent('a', { detector: 'retired_detector', kind: 'set-detector' }); + await settle(); + expect(h.status()).toEqual({ error: { code: 'unknown-detector', detail: 'retired_detector' }, status: 'error' }); + expect(h.runFilterGraph).toHaveBeenCalledTimes(2); + }); + + it('flushes the layer’s own paint before detecting it', async () => { + const h = createHarness([ + photoLayer('a', { controlType: 'edges', detection: null, source: { bitmap: null, type: 'paint' } }), + ]); + h.flushLayerUploads.mockImplementation(() => { + h.setLayer({ ...h.layer(), source: { bitmap: image('strokes.png'), offset: { x: 3, y: 4 }, type: 'paint' } }); + return Promise.resolve('flushed'); + }); + + h.coordinator.applyIntent('a', { detector: 'canny_edge_detection', kind: 'set-detector' }); + await settle(); + + expect(h.flushLayerUploads).toHaveBeenCalledExactlyOnceWith('a', { signal: expect.any(AbortSignal) }); + expect(h.graphs[0]!.imageName).toBe('strokes.png'); + await h.finishGraph(); + expect(h.layer().original).toMatchObject({ bitmap: { imageName: 'strokes.png' } }); + }); + + it('never schedules detection from document changes', async () => { + const h = createHarness(); + // An undo back to a typed layer whose map is stale. + h.setLayer({ ...h.layer(), controlType: 'edges', detection: { settings: {}, type: 'canny_edge_detection' } }); + h.coordinator.reconcileWithDocument(); + await settle(10_000); + + expect(h.runFilterGraph).not.toHaveBeenCalled(); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('runs at most two graphs at once', async () => { + const h = createHarness([photoLayer('a'), photoLayer('b'), photoLayer('c')]); + for (const id of ['a', 'b', 'c']) { + h.coordinator.applyIntent(id, { controlType: 'edges', kind: 'set-type' }); + } + await settle(); + + expect(h.runFilterGraph).toHaveBeenCalledTimes(2); + expect(h.status('c')).toEqual({ status: 'queued' }); + await h.finishGraph(); + expect(h.runFilterGraph).toHaveBeenCalledTimes(3); + expect(h.graphs.map((graph) => graph.imageName)).toEqual(['b.png', 'c.png']); + }); + + it('suspends on cooldown and resumes interrupted layers on activation', async () => { + const h = createHarness([photoLayer('a'), photoLayer('b')]); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const interrupted = h.graphs[0]!; + + h.coordinator.suspend(); + expect(interrupted.signal.aborted).toBe(true); + expect(h.status()).toEqual({ status: 'queued' }); + h.coordinator.applyIntent('b', { controlType: 'edges', kind: 'set-type' }); + await settle(10_000); + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + + h.coordinator.resume(); + await settle(0); + expect(h.runFilterGraph).toHaveBeenCalledTimes(3); + }); + + it('drops every job and status when the account changes', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const running = h.graphs[0]!; + + h.accounts.invalidate(); + expect(running.signal.aborted).toBe(true); + await h.finishGraph(); + + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('drops jobs and statuses of removed layers', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + h.removeLayer('a'); + h.coordinator.reconcileWithDocument(); + await settle(); + + expect(h.status()).toEqual({ status: 'idle' }); + expect(h.runFilterGraph).not.toHaveBeenCalled(); + }); + + it('settles a leftover error once the document no longer needs detection, without scheduling', async () => { + const h = createHarness(); + const untyped = h.layer(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + h.graphs.shift()!.reject(new Error('queue offline')); + await settle(0); + expect(h.status()).toMatchObject({ status: 'error' }); + + // Unrelated document changes keep the error while the map is still stale. + h.coordinator.reconcileWithDocument(); + expect(h.status()).toMatchObject({ status: 'error' }); + + // Undo of the type change. + h.setLayer(untyped); + h.coordinator.reconcileWithDocument(); + await settle(10_000); + + expect(h.status()).toEqual({ status: 'idle' }); + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + }); + + it('reports a failed paint flush as source-not-ready', async () => { + const h = createHarness([ + photoLayer('a', { controlType: 'edges', detection: null, source: { bitmap: null, type: 'paint' } }), + ]); + h.flushLayerUploads.mockResolvedValue('failed'); + + h.coordinator.applyIntent('a', { detector: 'canny_edge_detection', kind: 'set-detector' }); + await settle(); + + expect(h.status()).toEqual({ error: { code: 'source-not-ready' }, status: 'error' }); + expect(h.runFilterGraph).not.toHaveBeenCalled(); + }); + + it('reports a result that cannot be preserved as apply-failed, without committing it', async () => { + const h = createHarness(); + h.makeDurable.mockRejectedValue(new Error('PATCH 500')); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + await h.finishGraph(); + + expect(h.status()).toEqual({ error: { code: 'apply-failed', detail: 'PATCH 500' }, status: 'error' }); + expect(isControlMapCurrent(h.layer())).toBe(false); + }); + + it('gives up a queued slot when its job is cancelled', async () => { + const h = createHarness([photoLayer('a'), photoLayer('b'), photoLayer('c'), photoLayer('d')]); + for (const id of ['a', 'b', 'c', 'd']) { + h.coordinator.applyIntent(id, { controlType: 'edges', kind: 'set-type' }); + } + await settle(); + expect(h.runFilterGraph).toHaveBeenCalledTimes(2); + + // `c` waits for a slot; None cancels it. + h.coordinator.applyIntent('c', { detector: null, kind: 'set-detector' }); + expect(h.status('c')).toEqual({ status: 'idle' }); + await h.finishGraph(); + + expect(h.runFilterGraph).toHaveBeenCalledTimes(3); + expect(h.graphs.map((graph) => graph.imageName)).toEqual(['b.png', 'd.png']); + }); + + it('never promotes a result superseded while it waits for the document', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const lease = h.acquireLease(); + await h.finishGraph(); + expect(h.status()).toEqual({ status: 'queued' }); + + h.coordinator.applyIntent('a', { detector: 'mlsd_detection', kind: 'set-detector' }); + lease!.release(); + h.emitLockChange(); + await settle(); + await h.finishGraph(); + + expect(h.makeDurable).toHaveBeenCalledExactlyOnceWith('map-2.png'); + expect(h.layer().source).toMatchObject({ bitmap: { imageName: 'map-2.png' } }); + }); + + it('keeps a cancelled job from overwriting its successor’s status', async () => { + const h = createHarness([photoLayer('a'), photoLayer('b'), photoLayer('c')]); + for (const id of ['a', 'b', 'c']) { + h.coordinator.applyIntent(id, { controlType: 'edges', kind: 'set-type' }); + } + await settle(); + expect(h.status('c')).toEqual({ status: 'queued' }); + + // `a` finishing hands its slot to `c`; before `c` resumes, a new detector replaces its job. + const capture = h.derivation.captureControlDetectionGuard.getMockImplementation()!; + h.derivation.captureControlDetectionGuard.mockImplementationOnce((layerId) => { + h.coordinator.applyIntent('c', { detector: 'mlsd_detection', kind: 'set-detector' }); + return capture(layerId); + }); + await h.finishGraph(); + + expect(h.status('c')).toEqual({ status: 'queued' }); + await settle(); + expect(h.status('c')).toEqual({ detector: 'mlsd_detection', status: 'processing' }); + }); + + it.each([ + [ + 'an undo of its type', + (layer: CanvasControlLayerContract) => ({ ...layer, controlType: undefined, detection: null }), + ], + [ + 'an undo to another detector', + (layer: CanvasControlLayerContract) => ({ ...layer, detection: { settings: {}, type: 'mlsd_detection' } }), + ], + ])('cancels the backend run after %s, without scheduling another', async (_label, undo) => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const running = h.graphs[0]!; + expect(h.status()).toMatchObject({ status: 'processing' }); + + h.setLayer(undo(h.layer())); + h.coordinator.reconcileWithDocument(); + + expect(running.signal.aborted).toBe(true); + expect(h.status()).toEqual({ status: 'idle' }); + await h.finishGraph(); + await settle(10_000); + expect(h.runFilterGraph).toHaveBeenCalledOnce(); + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + }); + + it('keeps a job whose own commit reconciles the document while it publishes', async () => { + const h = createHarness(); + h.onCommitPublished(() => h.coordinator.reconcileWithDocument()); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const running = h.graphs[0]!; + + await h.finishGraph(); + + expect(running.signal.aborted).toBe(false); + expect(isControlMapCurrent(h.layer())).toBe(true); + expect(h.status()).toEqual({ status: 'idle' }); + }); + + it('keeps a run through document changes that leave its input alone', async () => { + const h = createHarness(); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + + h.setLayer({ ...h.layer(), name: 'Edges 1' }); + h.coordinator.reconcileWithDocument(); + + expect(h.graphs[0]!.signal.aborted).toBe(false); + expect(h.status()).toMatchObject({ status: 'processing' }); + }); + + it('disposes idempotently, cancelling work and releasing subscriptions', async () => { + const h = createHarness(); + const listener = vi.fn(); + h.coordinator.subscribeStatus('a', listener); + h.coordinator.applyIntent('a', { controlType: 'edges', kind: 'set-type' }); + await settle(); + const running = h.graphs[0]!; + + h.coordinator.dispose(); + h.coordinator.dispose(); + + expect(running.signal.aborted).toBe(true); + expect(h.lockListeners.size).toBe(0); + expect(h.status()).toEqual({ status: 'idle' }); + expect(h.coordinator.applyIntent('a', { kind: 'draw' })).toBe('blocked'); + expect(h.coordinator.retry('a')).toBe('missing'); + await h.finishGraph(); + expect(h.derivation.commitControlMap).not.toHaveBeenCalled(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.ts new file mode 100644 index 00000000000..412dff97840 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/ControlDetectionCoordinator.ts @@ -0,0 +1,626 @@ +/** + * Background detection for control layers. Commands start it (a Properties intent, Retry, resuming after cooldown); + * document changes never do, so undo and redo cannot re-run detection. One job per layer, at most + * {@link CONTROL_DETECTION_MAX_CONCURRENT} graphs per engine; results land through the engine's guarded map commit, + * which refuses while a modal operation, gesture or layer session holds the document, so a finished result waits + * and retries instead of blocking them. The commit promotes a result to durable pixels only once it is about to + * land, so a dropped result stays an intermediate the server collects. + */ + +import type { AccountOwnedResource, AccountScope } from '@platform/state/accountLifecycle'; +import type { CanvasControlDerivationCapability, ControlDetectionGuard } from '@workbench/canvas-engine/api'; +import type { + CanvasControlDetectionContract, + CanvasControlLayerContract, + CanvasDocumentContractV3, +} from '@workbench/canvas-engine/contracts'; +import type { LayerFlushResult } from '@workbench/canvas-engine/controllers/persistenceController'; +import type { + ControlDetectionErrorCode, + ControlDetectionStatus, + ControlLayerIntent, + ControlLayerIntentResult, + RetryControlDetectionResult, +} from '@workbench/canvas-operations/operationTypes'; +import type { BackendGraphContract } from '@workbench/graphContracts'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; +import { captureAccountScope, registerAccountOwnedResource } from '@platform/state/accountLifecycle'; +import { + controlDetectionInputKey, + getControlDetectionInput, + isControlMapCurrent, + lookupDocumentLeaf, + needsControlDetection, +} from '@workbench/canvas-engine/api'; +import { + createControlDetectionStatusStore, + IDLE_CONTROL_DETECTION_STATUS, + type ControlDetectionStatusStore, +} from '@workbench/canvas-operations/controlDetectionStatusStore'; +import { applyControlLayerIntent } from '@workbench/canvas-operations/controlLayerIntents'; +import { buildFilterDefaults, getFilterDefinition } from '@workbench/canvas-operations/filterGraphs'; +import { FILTER_AUTO_PROCESS_DEBOUNCE_MS } from '@workbench/canvas-operations/filterOperationSession'; + +import { ControlDetectionFailure, runControlDetection, type ControlDetectionMap } from './controlDetectionRunner'; + +export const CONTROL_DETECTION_MAX_CONCURRENT = 2; +export const CONTROL_DETECTION_COMMIT_RETRY_MS = { initial: 250, max: 2_000 } as const; +/** How long a finished result keeps waiting for the document before it reports `commit-busy`. */ +export const CONTROL_DETECTION_COMMIT_TIMEOUT_MS = 30_000; +/** Restarts after the input pixels changed under a running job (painting, an upload landing). */ +export const CONTROL_DETECTION_MAX_RESTARTS = 3; + +/** The engine's control derivation capability. */ +export type ControlDerivationPort = Pick< + CanvasControlDerivationCapability, + 'captureControlDetectionGuard' | 'commitControlLayerPatch' | 'commitControlMap' +>; + +export interface ControlDetectionCoordinatorDeps { + readonly projectId: string; + readonly derivation: ControlDerivationPort; + getDocument(): CanvasDocumentContractV3 | null; + /** Persists the layer's pending paint, waiting for an edit that holds it. */ + flushLayerUploads(layerId: string, options: { signal?: AbortSignal }): Promise; + runFilterGraph(options: { + graph: BackendGraphContract; + outputNodeId: string; + signal?: AbortSignal; + }): Promise<{ height: number; imageName: string; width: number }>; + /** Adopts a result as canvas-owned pixels; called by the map commit once nothing refuses it. */ + makeDurable(imageName: string): Promise; + /** Notifies when the edit lease, interaction lock or an editing session may have been released. */ + subscribeLockChanges(listener: () => void): () => void; + readonly statuses?: ControlDetectionStatusStore; + /** Account lifetime; the platform lifecycle by default. */ + readonly accounts?: { + capture(): AccountScope; + register(resource: AccountOwnedResource): () => void; + }; +} + +export interface ControlDetectionCoordinator { + applyIntent(layerId: string, intent: ControlLayerIntent): ControlLayerIntentResult; + retry(layerId: string): RetryControlDetectionResult; + getStatus(layerId: string): ControlDetectionStatus; + subscribeStatus(layerId: string, listener: () => void): () => void; + /** Cancels every job and remembers its layer, for {@link resume}. */ + suspend(): void; + /** Restarts the layers {@link suspend} interrupted that still need a map. */ + resume(): void; + /** + * Follows a document change without scheduling work: drops jobs and statuses of removed layers, cancels a job whose + * layer no longer needs it or whose input changed other than by its own paint landing (undo, redo), and settles a + * leftover error or queued status once the layer needs no detection (an undo past the failed edit). + */ + reconcileWithDocument(): void; + dispose(): void; +} + +const INTENT_LABELS: Readonly> = { + draw: 'Draw control image', + 'replace-image': 'Replace control image', + 'set-detector': 'Change detector', + 'set-detector-settings': 'Change detector settings', + 'set-type': 'Change control type', +}; + +const PATCH_REFUSALS: Readonly< + Record< + Exclude['status'], 'committed'>, + ControlLayerIntentResult + > +> = { + busy: 'blocked', + failed: 'failed', + locked: 'locked', + missing: 'missing', + 'not-ready': 'blocked', + 'over-budget': 'over-budget', + stale: 'blocked', +}; + +const QUEUED: ControlDetectionStatus = Object.freeze({ status: 'queued' }); + +type JobOutcome = + | { kind: 'committed' | 'drop' | 'restart' } + | { kind: 'error'; code: ControlDetectionErrorCode; detail?: string }; + +interface Job { + readonly layerId: string; + readonly controller: AbortController; + /** The input key it detects (updated once a flush settles the paint); a command for the same key keeps it. */ + inputKey: string; + /** The layer `inputKey` was read from, to tell the layer's own paint landing apart from other input changes. */ + recipe: CanvasControlLayerContract; + readonly restarts: number; + timer: ReturnType | null; + running: boolean; + /** + * While its map commit runs: the commit's own publish changes the document and reconciles synchronously, and the + * engine's checks already settle what the job's result may still do. + */ + committing: boolean; + /** Ends a commit retry wait early (a lock was released). */ + wake: (() => void) | null; +} + +const detectorDefaults = (detector: string): Record => { + const definition = getFilterDefinition(detector); + return definition ? buildFilterDefaults(definition) : {}; +}; + +/** Whether the detection input is the layer's own paint, whose newest strokes may not be uploaded yet. */ +const isOwnPaintInput = (layer: CanvasControlLayerContract): boolean => + layer.original === undefined && !layer.filter && layer.source.type === 'paint'; + +/** + * Whether a run could produce a map: the document says the map is stale, or the input is the layer's own paint + * whose pixels may still be in flight (a flush settles which). + */ +const wantsDetection = (layer: CanvasControlLayerContract): boolean => + needsControlDetection(layer) || + (layer.controlType !== undefined && !!layer.detection && !isControlMapCurrent(layer) && isOwnPaintInput(layer)); + +/** + * Whether `after` differs from `before` only by its own paint landing (an upload, or strokes on the input): same type + * and detector, and both detect their own paint source. An image swap (undo or redo of a replaced image) does not + * qualify. + */ +const onlyOwnPaintChanged = (before: CanvasControlLayerContract, after: CanvasControlLayerContract): boolean => + isOwnPaintInput(before) && + isOwnPaintInput(after) && + before.controlType === after.controlType && + areJsonValuesStructurallyEqual(before.detection ?? null, after.detection ?? null); + +const failureDetail = (error: unknown): string | undefined => { + const detail = error instanceof Error ? error.message : typeof error === 'string' ? error : undefined; + return detail || undefined; +}; + +const errorOutcome = (code: ControlDetectionErrorCode, detail?: string): JobOutcome => + detail ? { code, detail, kind: 'error' } : { code, kind: 'error' }; + +let coordinatorCount = 0; + +export const createControlDetectionCoordinator = ( + deps: ControlDetectionCoordinatorDeps +): ControlDetectionCoordinator => { + const statuses = deps.statuses ?? createControlDetectionStatusStore(); + const accounts = deps.accounts ?? { capture: captureAccountScope, register: registerAccountOwnedResource }; + const jobs = new Map(); + const interrupted = new Set(); + let suspended = false; + let disposed = false; + + let runningGraphs = 0; + const slotWaiters: (() => void)[] = []; + const acquireSlot = (signal: AbortSignal): Promise => { + signal.throwIfAborted(); + if (runningGraphs < CONTROL_DETECTION_MAX_CONCURRENT) { + runningGraphs += 1; + return Promise.resolve(); + } + return new Promise((resolve, reject) => { + const grant = (): void => { + signal.removeEventListener('abort', onAbort); + resolve(); + }; + const onAbort = (): void => { + slotWaiters.splice(slotWaiters.indexOf(grant), 1); + reject(signal.reason); + }; + slotWaiters.push(grant); + signal.addEventListener('abort', onAbort, { once: true }); + }); + }; + /** Hands the slot straight to the next waiter, so the count never dips below a queued job. */ + const releaseSlot = (): void => { + const next = slotWaiters.shift(); + if (next) { + next(); + } else { + runningGraphs -= 1; + } + }; + + const readLayer = (layerId: string): CanvasControlLayerContract | null => { + const layer = lookupDocumentLeaf(deps.getDocument(), layerId)?.layer; + return layer?.type === 'control' ? layer : null; + }; + + /** Writes only while `job` is its layer's job, so a cancelled job never overwrites its successor's status. */ + const setJobStatus = (job: Job, status: ControlDetectionStatus): void => { + if (jobs.get(job.layerId) === job) { + statuses.set(job.layerId, status); + } + }; + + const cancelJob = (layerId: string): void => { + const job = jobs.get(layerId); + if (!job) { + return; + } + jobs.delete(layerId); + if (job.timer !== null) { + clearTimeout(job.timer); + job.timer = null; + } + job.controller.abort(); + }; + + const waitForRetry = (job: Job, delayMs: number, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + const settle = (): void => { + clearTimeout(timer); + job.wake = null; + signal.removeEventListener('abort', onAbort); + }; + const done = (): void => { + settle(); + resolve(); + }; + const onAbort = (): void => { + settle(); + reject(signal.reason); + }; + const timer = setTimeout(done, delayMs); + job.wake = done; + signal.addEventListener('abort', onAbort, { once: true }); + }); + + /** + * Whether `guard` still describes the layer. The layer's own paint landing under the job (an upload, more strokes) + * runs again; anything else (removal, a document swap, an image swapped by undo or redo, another command) drops + * the result, so history never re-runs detection. + */ + const checkGuard = ( + guard: ControlDetectionGuard, + recipe: CanvasControlLayerContract + ): 'current' | 'drop' | 'restart' => { + const current = deps.derivation.captureControlDetectionGuard(guard.layerId); + if (!current || current.documentRevision !== guard.documentRevision) { + return 'drop'; + } + if (current.inputKey === guard.inputKey) { + return 'current'; + } + const layer = readLayer(guard.layerId); + return layer && needsControlDetection(layer) && onlyOwnPaintChanged(recipe, layer) ? 'restart' : 'drop'; + }; + + const commitWithRetry = async ( + job: Job, + guard: ControlDetectionGuard, + recipe: CanvasControlLayerContract, + detection: CanvasControlDetectionContract, + map: ControlDetectionMap, + signal: AbortSignal + ): Promise => { + const startedAt = Date.now(); + let delay: number = CONTROL_DETECTION_COMMIT_RETRY_MS.initial; + let durable = false; + const makeDurable = async (): Promise => { + if (!durable) { + await deps.makeDurable(map.image.imageName); + durable = true; + } + }; + for (;;) { + job.committing = true; + let result: Awaited>; + try { + result = await deps.derivation.commitControlMap({ + detection, + guard, + image: map.image, + makeDurable, + rect: map.rect, + signal, + }); + } finally { + job.committing = false; + } + switch (result.status) { + case 'committed': + return { kind: 'committed' }; + case 'stale': { + const check = checkGuard(guard, recipe); + return { kind: check === 'restart' ? 'restart' : 'drop' }; + } + case 'missing': + case 'aborted': + signal.throwIfAborted(); + return { kind: 'drop' }; + case 'over-budget': + return errorOutcome('over-budget', result.message); + case 'unrepresentable': + return errorOutcome('unrepresentable', result.message); + case 'failed': + return errorOutcome('apply-failed', result.message); + case 'busy': + case 'locked': + case 'not-ready': { + const remaining = CONTROL_DETECTION_COMMIT_TIMEOUT_MS - (Date.now() - startedAt); + if (remaining <= 0) { + return errorOutcome('commit-busy', result.message); + } + setJobStatus(job, QUEUED); + await waitForRetry(job, Math.min(delay, remaining), signal); + delay = Math.min(delay * 2, CONTROL_DETECTION_COMMIT_RETRY_MS.max); + break; + } + } + } + }; + + const detect = async (job: Job, signal: AbortSignal): Promise => { + let layer = readLayer(job.layerId); + if (!layer || !wantsDetection(layer)) { + return { kind: 'drop' }; + } + if (isOwnPaintInput(layer)) { + const flushed = await deps.flushLayerUploads(job.layerId, { signal }); + signal.throwIfAborted(); + if (flushed === 'aborted') { + return { kind: 'drop' }; + } + if (flushed === 'failed') { + return errorOutcome('source-not-ready'); + } + layer = readLayer(job.layerId); + } + if (!layer || !needsControlDetection(layer) || !layer.detection) { + return { kind: 'drop' }; + } + const input = getControlDetectionInput(layer); + const guard = deps.derivation.captureControlDetectionGuard(job.layerId); + if (!input || !guard) { + return { kind: 'drop' }; + } + job.inputKey = guard.inputKey; + job.recipe = layer; + const recipe = layer; + const { detection } = layer; + + await acquireSlot(signal); + let map: ControlDetectionMap; + try { + setJobStatus(job, { detector: detection.type, status: 'processing' }); + map = await runControlDetection({ detection, input, runFilterGraph: deps.runFilterGraph, signal }); + } finally { + releaseSlot(); + } + + const check = checkGuard(guard, recipe); + if (check !== 'current') { + return { kind: check }; + } + return commitWithRetry(job, guard, recipe, detection, map, signal); + }; + + const schedule = (layerId: string, delayMs: number, layer: CanvasControlLayerContract, restarts = 0): void => { + if (disposed) { + return; + } + if (suspended) { + interrupted.add(layerId); + statuses.set(layerId, QUEUED); + return; + } + const inputKey = controlDetectionInputKey(layer); + const existing = jobs.get(layerId); + if (existing?.running && existing.inputKey === inputKey) { + return; + } + cancelJob(layerId); + const job: Job = { + committing: false, + controller: new AbortController(), + inputKey, + layerId, + recipe: layer, + restarts, + running: false, + timer: null, + wake: null, + }; + jobs.set(layerId, job); + statuses.set(layerId, QUEUED); + job.timer = setTimeout(() => { + job.timer = null; + void run(job); + }, delayMs); + }; + + const finish = (job: Job, outcome: JobOutcome): void => { + if (jobs.get(job.layerId) !== job) { + return; + } + jobs.delete(job.layerId); + if (outcome.kind === 'error') { + statuses.set( + job.layerId, + outcome.detail + ? { error: { code: outcome.code, detail: outcome.detail }, status: 'error' } + : { error: { code: outcome.code }, status: 'error' } + ); + return; + } + const layer = outcome.kind === 'restart' ? readLayer(job.layerId) : null; + if (layer && job.restarts < CONTROL_DETECTION_MAX_RESTARTS) { + schedule(job.layerId, FILTER_AUTO_PROCESS_DEBOUNCE_MS, layer, job.restarts + 1); + return; + } + statuses.set(job.layerId, IDLE_CONTROL_DETECTION_STATUS); + }; + + const run = async (job: Job): Promise => { + job.running = true; + const signal = AbortSignal.any([job.controller.signal, accounts.capture().signal]); + let outcome: JobOutcome; + try { + outcome = await detect(job, signal); + } catch (error) { + if (signal.aborted) { + // A cancel already settled the status; an account switch alone leaves nothing to show. + if (jobs.get(job.layerId) === job) { + jobs.delete(job.layerId); + statuses.set(job.layerId, IDLE_CONTROL_DETECTION_STATUS); + } + return; + } + outcome = + error instanceof ControlDetectionFailure + ? errorOutcome(error.code, error.detail) + : errorOutcome('process-failed', failureDetail(error)); + } + finish(job, outcome); + }; + + const cancelAll = (): void => { + for (const layerId of jobs.keys()) { + cancelJob(layerId); + } + }; + + const unsubscribeLocks = deps.subscribeLockChanges(() => { + for (const job of jobs.values()) { + job.wake?.(); + } + }); + const unregisterAccount = accounts.register({ + clear: () => { + cancelAll(); + interrupted.clear(); + statuses.clear(); + }, + name: `canvas-control-detection:${deps.projectId}:${String((coordinatorCount += 1))}`, + }); + + const settleWithoutDetection = (layerId: string): void => { + cancelJob(layerId); + interrupted.delete(layerId); + statuses.set(layerId, IDLE_CONTROL_DETECTION_STATUS); + }; + + return { + applyIntent: (layerId, intent) => { + if (disposed) { + return 'blocked'; + } + const layer = readLayer(layerId); + if (!layer) { + return 'missing'; + } + const next = applyControlLayerIntent(layer, intent, { detectorDefaults }); + if (next === 'no-detection-input') { + return next; + } + if (next === layer) { + return 'unchanged'; + } + const result = deps.derivation.commitControlLayerPatch({ + coalesceKey: + intent.kind === 'set-detector-settings' + ? `${layerId}:${layer.detection?.type ?? 'none'}:settings` + : undefined, + expected: layer, + label: INTENT_LABELS[intent.kind], + next, + }); + if (result.status !== 'committed') { + return PATCH_REFUSALS[result.status]; + } + if (wantsDetection(next)) { + schedule(layerId, FILTER_AUTO_PROCESS_DEBOUNCE_MS, next); + } else { + settleWithoutDetection(layerId); + } + return 'updated'; + }, + dispose: () => { + if (disposed) { + return; + } + disposed = true; + cancelAll(); + interrupted.clear(); + unsubscribeLocks(); + unregisterAccount(); + statuses.clear(); + }, + getStatus: statuses.get, + reconcileWithDocument: () => { + const tracked = new Set([...jobs.keys(), ...interrupted, ...statuses.layerIds()]); + for (const layerId of tracked) { + const layer = readLayer(layerId); + if (!layer) { + settleWithoutDetection(layerId); + continue; + } + const job = jobs.get(layerId); + if (job?.committing) { + continue; + } + if (job) { + // History moved the layer away from what the job detects: its result would be dropped, so stop the graph + // now. The layer's own paint landing keeps it, to restart once it finishes. + const inputChanged = + controlDetectionInputKey(layer) !== job.inputKey && !onlyOwnPaintChanged(job.recipe, layer); + if (!wantsDetection(layer) || inputChanged) { + settleWithoutDetection(layerId); + } + continue; + } + // Only a status left behind by a failed or interrupted run is stale here. + const { status } = statuses.get(layerId); + if ((status === 'error' || status === 'queued') && !wantsDetection(layer)) { + interrupted.delete(layerId); + statuses.set(layerId, IDLE_CONTROL_DETECTION_STATUS); + } + } + }, + resume: () => { + if (disposed || !suspended) { + return; + } + suspended = false; + const layerIds = [...interrupted]; + interrupted.clear(); + for (const layerId of layerIds) { + const layer = readLayer(layerId); + if (layer && wantsDetection(layer)) { + schedule(layerId, 0, layer); + } else { + statuses.set(layerId, IDLE_CONTROL_DETECTION_STATUS); + } + } + }, + retry: (layerId) => { + const layer = disposed ? null : readLayer(layerId); + if (!layer) { + return 'missing'; + } + if (!wantsDetection(layer)) { + settleWithoutDetection(layerId); + return 'not-needed'; + } + schedule(layerId, 0, layer); + return 'scheduled'; + }, + subscribeStatus: statuses.subscribe, + suspend: () => { + if (disposed || suspended) { + return; + } + suspended = true; + for (const layerId of jobs.keys()) { + interrupted.add(layerId); + cancelJob(layerId); + statuses.set(layerId, QUEUED); + } + }, + }; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.test.ts index 5b6b962581e..e4fae3d9e22 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.test.ts @@ -87,7 +87,7 @@ describe('FilterOperationCoordinator', () => { expect(session.commit).toHaveBeenCalledWith('apply'); }); - it('rejects missing, unsupported, disabled, locked, and busy starts deterministically', () => { + it('rejects missing, unsupported (mask and control), disabled, locked, and busy starts deterministically', () => { const { coordinator, deps } = createHarness(); deps.getDocument.mockReturnValueOnce(null); expect(coordinator.start(layer.id)).toBe('missing'); @@ -96,6 +96,12 @@ describe('FilterOperationCoordinator', () => { selectedLayerId: null, }); expect(coordinator.start(layer.id)).toBe('unsupported'); + deps.getDocument.mockReturnValueOnce({ + stacks: stacksFrom([{ ...layer, type: 'control' } as unknown as CanvasLayerContract]), + selectedLayerId: null, + }); + expect(coordinator.start(layer.id)).toBe('unsupported'); + expect(deps.createSession).not.toHaveBeenCalled(); deps.getDocument.mockReturnValueOnce({ stacks: stacksFrom([{ ...layer, isEnabled: false } as unknown as CanvasLayerContract]), selectedLayerId: null, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.ts index 11fb66a43e8..1f66602a738 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/FilterOperationCoordinator.ts @@ -56,7 +56,7 @@ export interface FilterOperationCoordinatorDeps { } export interface FilterOperationCoordinator { - start(layerId: string, recommendedFilterType?: string | null): StartFilterOperationResult; + start(layerId: string): StartFilterOperationResult; updateDraft(draft: LayerFilterSettings): CanvasOperationMutationResult; setAutoProcess(value: boolean): CanvasOperationMutationResult; process(): Promise; @@ -90,25 +90,20 @@ export const createFilterOperationCoordinator = (deps: FilterOperationCoordinato deps.stores.filterSession.set(null); }; - const start = (layerId: string, recommendedFilterType?: string | null): StartFilterOperationResult => { + const start = (layerId: string): StartFilterOperationResult => { if (deps.isInteractionLocked()) { return 'locked'; } - if (recommendedFilterType && deps.controller.getSnapshot().status !== 'idle') { - return 'not-ready'; - } const document = deps.getDocument(); const leaf = lookupDocumentLeaf(document, layerId); const layer = leaf?.layer; if (!document || !leaf || !layer) { return 'missing'; } - if (layer.type !== 'raster' && layer.type !== 'control') { + // Control layers derive their map through detection; Filter (and its save-as-control copy) is raster-only. + if (layer.type !== 'raster') { return 'unsupported'; } - if (recommendedFilterType && layer.filter) { - return 'not-ready'; - } if (!leaf.contributionEnabled) { return 'disabled'; } @@ -119,8 +114,7 @@ export const createFilterOperationCoordinator = (deps: FilterOperationCoordinato if (!guard) { return 'not-ready'; } - const initialType = - layer.filter?.type ?? recommendedFilterType ?? readLastUsedFilterType() ?? DEFAULT_CONTROL_FILTER_TYPE; + const initialType = layer.filter?.type ?? readLastUsedFilterType() ?? DEFAULT_CONTROL_FILTER_TYPE; const initialFilter = layer.filter ? structuredClone(layer.filter) : null; const definition = getFilterDefinition(initialType); const draft = initialFilter ?? { diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/api.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/api.ts index e80771b31f5..e1677a21336 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/api.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/api.ts @@ -15,7 +15,7 @@ export { type GalleryCanvasImportDestination, type ImportGalleryImagesResult, } from './importGalleryImages'; -export { getCanvasImportNotice } from './canvasImportNotice'; +export { getCanvasImportNotice, getSendAsControlNotice, type CanvasImportNotice } from './canvasImportNotice'; export { createCanvasFromImages, type CreateCanvasFromImagesResult } from './createCanvasFromImages'; export { createFromBbox, @@ -24,9 +24,33 @@ export { type CreateFromBboxResult, } from './createFromBbox'; export { getCreateFromBboxNotice, type CreateFromBboxNotice } from './createFromBboxNotice'; +export { + fillControlFromBbox, + getControlIntentNotice, + getControlSourceSnapshot, + getControlUploadNotice, + resolveControlFileDrop, + sendGalleryImageAsControl, + subscribeControlSourceSnapshot, + uploadControlImageFile, + type ControlFileDropResolution, + type ControlSourceNotice, + type ControlSourceOrigin, + type ControlSourceSnapshot, + type FillControlFromBboxResult, + type SendGalleryImageAsControlResult, + type UploadControlImageResult, +} from './controlSourceActions'; +export type { CanvasCompositeExportEngine } from './exportCanvasComposite'; export type { + ControlDetectionError, + ControlDetectionErrorCode, + ControlDetectionStatus, + ControlLayerIntent, + ControlLayerIntentResult, FilterOperationSessionState, FilterSessionErrorCode, + RetryControlDetectionResult, SamInput, SamModel, SamSessionError, @@ -34,6 +58,7 @@ export type { SamSessionSnapshot, } from './operationTypes'; export { getCanvasOperations } from './operationAccess'; +export { IDLE_CONTROL_DETECTION_STATUS } from './controlDetectionStatusStore'; export { getCanvasEngine } from './engineRegistry'; export { saveCanvasToGallery, type CanvasGallerySaveRegion } from './saveCanvasToGallery'; export { @@ -66,5 +91,4 @@ export { type FilterParamSpec, } from './filterGraphs'; export { readLastUsedFilterType, recordLastUsedFilterType } from './filterPreferences'; -export { resolveDefaultFilterForModel } from './controlRecommendations'; export { buildSamGraph, documentToExportLocalSamInput, isSamDocumentInputValid, isSamInputValid } from './samGraph'; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.test.ts index adaa8e9b770..b40a38c0feb 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { getCanvasImportNotice } from './canvasImportNotice'; +import { getCanvasImportNotice, getSendAsControlNotice } from './canvasImportNotice'; describe('getCanvasImportNotice', () => { it('maps a complete import to a localized success notice', () => { @@ -58,3 +58,22 @@ describe('getCanvasImportNotice', () => { ); }); }); + +describe('getSendAsControlNotice', () => { + it('names a filled layer, reads a new layer as a one-image import, and passes refusals through', () => { + expect(getSendAsControlNotice({ layerId: 'control-1', layerName: 'Depth 1', status: 'filled' })).toEqual({ + kind: 'success', + options: { name: 'Depth 1' }, + titleKey: 'widgets.canvas.import.controlFilled', + }); + expect(getSendAsControlNotice({ layerId: 'control-2', status: 'created' })).toEqual({ + kind: 'success', + options: { count: 1 }, + titleKey: 'widgets.canvas.import.success', + }); + expect(getSendAsControlNotice({ status: 'stale-document' })).toEqual({ + kind: 'error', + titleKey: 'widgets.canvas.import.staleDocument', + }); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.ts index f617274adfa..3a4af8f61a0 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/canvasImportNotice.ts @@ -1,3 +1,4 @@ +import type { SendGalleryImageAsControlResult } from './controlSourceActions'; import type { ImportGalleryImagesResult } from './importGalleryImages'; export interface CanvasImportNotice { @@ -5,12 +6,13 @@ export interface CanvasImportNotice { titleKey: | 'widgets.canvas.import.blocked' | 'widgets.canvas.import.capabilitiesUnavailable' + | 'widgets.canvas.import.controlFilled' | 'widgets.canvas.import.empty' | 'widgets.canvas.import.partial' | 'widgets.canvas.import.staleDocument' | 'widgets.canvas.import.staleProject' | 'widgets.canvas.import.success'; - options?: { count: number } | { count: number; failedCount: number; successCount: number }; + options?: { count: number } | { count: number; failedCount: number; successCount: number } | { name: string }; } const assertNever = (value: never): never => { @@ -50,3 +52,15 @@ export const getCanvasImportNotice = (result: ImportGalleryImagesResult): Canvas return assertNever(result); } }; + +/** A gallery image used as control: the layer it filled is named; a new layer reads as any one-image import. */ +export const getSendAsControlNotice = (result: SendGalleryImageAsControlResult): CanvasImportNotice => { + switch (result.status) { + case 'filled': + return { kind: 'success', options: { name: result.layerName }, titleKey: 'widgets.canvas.import.controlFilled' }; + case 'created': + return getCanvasImportNotice({ failedImageNames: [], layerIds: [result.layerId], status: 'imported' }); + default: + return getCanvasImportNotice(result); + } +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/contracts.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/contracts.ts index af4ae913e8d..a58590347e0 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/contracts.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/contracts.ts @@ -18,6 +18,9 @@ import type { ComposeForGenerationResult, } from '@workbench/canvas-operations/generationComposite'; import type { + ControlDetectionStatus, + ControlLayerIntent, + ControlLayerIntentResult, FilterCommitTarget, FilterOperationSessionState, LayerFilterSettings, @@ -25,6 +28,7 @@ import type { SamModel, SamSessionError, SamSessionSnapshot, + RetryControlDetectionResult, } from '@workbench/canvas-operations/operationTypes'; import type { BackendGraphContract } from '@workbench/graphContracts'; import type { WorkbenchState } from '@workbench/projectContracts'; @@ -67,7 +71,7 @@ export interface CanvasOperationCapability { getSamSessionState(): SamSessionSnapshot | null; subscribeSamSession(listener: () => void): () => void; startSelectObject(layerId: string): StartSelectObjectSessionResult; - startFilterOperation(layerId: string, recommendedFilterType?: string | null): StartFilterOperationResult; + startFilterOperation(layerId: string): StartFilterOperationResult; updateFilterOperation(draft: LayerFilterSettings): CanvasOperationMutationResult; setFilterOperationAutoProcess(value: boolean): CanvasOperationMutationResult; processFilterOperation(): Promise; @@ -86,6 +90,12 @@ export interface CanvasOperationCapability { ): Promise; resetSelectObjectSession(): CanvasOperationMutationResult; cancelSelectObjectSession(): void; + /** Applies a Properties panel edit to a control layer, then detects its map in the background when needed. */ + applyControlLayerIntent(layerId: string, intent: ControlLayerIntent): ControlLayerIntentResult; + /** Runs detection again for a layer whose map is stale (after a failure, or undo back to a stale state). */ + retryControlDetection(layerId: string): RetryControlDetectionResult; + getControlDetectionStatus(layerId: string): ControlDetectionStatus; + subscribeControlDetectionStatus(layerId: string, listener: () => void): () => void; } /** Private composition shape; mutable stores and controller never cross the Canvas interface. */ @@ -165,7 +175,7 @@ export interface CanvasFilterCoordinatorDeps { } export interface CanvasFilterOperationCoordinator { - start(layerId: string, recommendedFilterType?: string | null): StartFilterOperationResult; + start(layerId: string): StartFilterOperationResult; updateDraft(draft: LayerFilterSettings): CanvasOperationMutationResult; setAutoProcess(value: boolean): CanvasOperationMutationResult; process(): Promise; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.test.ts new file mode 100644 index 00000000000..ecd79389115 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.test.ts @@ -0,0 +1,90 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { ControlDetectionFailure, runControlDetection } from './controlDetectionRunner'; +import { FILTER_NODE_ID } from './filterGraphs'; + +const canny = { settings: { high_threshold: 200, low_threshold: 100 }, type: 'canny_edge_detection' }; +const output = (width: number, height: number) => ({ height, imageName: 'map.png', width }); + +describe('runControlDetection', () => { + it('builds the graph on the persisted image at its content size and keeps the map in its frame', async () => { + const runFilterGraph = vi.fn(() => Promise.resolve(output(30, 20))); + const result = await runControlDetection({ + detection: canny, + input: { bitmap: { height: 20, imageName: 'paint.png', width: 30 }, offset: { x: -4, y: 6 }, type: 'paint' }, + runFilterGraph, + signal: new AbortController().signal, + }); + + const [{ graph, outputNodeId }] = runFilterGraph.mock.calls[0] as unknown as [ + { graph: { nodes: Record> }; outputNodeId: string }, + ]; + // Processed over a white input-sized background, with the input's alpha restored on the output. + expect(graph.nodes[FILTER_NODE_ID]).toMatchObject({ high_threshold: 200, type: 'canny_edge_detection' }); + expect(graph.nodes.control_filter_background).toMatchObject({ height: 20, width: 30 }); + expect(graph.nodes.control_filter_flatten).toMatchObject({ image: { image_name: 'paint.png' } }); + expect(graph.nodes.control_filter_source_alpha).toMatchObject({ image: { image_name: 'paint.png' } }); + expect(outputNodeId).toBe('control_filter_restore_alpha'); + expect(result).toEqual({ + image: { height: 20, imageName: 'map.png', width: 30 }, + rect: { height: 20, width: 30, x: -4, y: 6 }, + }); + }); + + it('places an image input at its origin', async () => { + const result = await runControlDetection({ + detection: canny, + input: { image: { height: 48, imageName: 'photo.png', width: 64 }, type: 'image' }, + runFilterGraph: () => Promise.resolve(output(64, 48)), + signal: new AbortController().signal, + }); + + expect(result.rect).toEqual({ height: 48, width: 64, x: 0, y: 0 }); + }); + + it.each([ + [ + 'unknown-detector', + { detection: { settings: {}, type: 'retired_detector' }, output: output(64, 64), image: true }, + ], + ['source-not-ready', { detection: canny, output: output(64, 64), image: false }], + ['output-dimensions', { detection: canny, output: output(32, 32), image: true }], + ] as const)('reports %s', async (code, scenario) => { + const runFilterGraph = vi.fn(() => Promise.resolve(scenario.output)); + const run = runControlDetection({ + detection: scenario.detection, + input: scenario.image + ? { image: { height: 64, imageName: 'photo.png', width: 64 }, type: 'image' } + : { bitmap: null, type: 'paint' }, + runFilterGraph, + signal: new AbortController().signal, + }); + + await expect(run).rejects.toBeInstanceOf(ControlDetectionFailure); + await expect(run).rejects.toMatchObject({ code }); + }); + + it('wraps graph failures with their message and lets aborts through', async () => { + const input = { image: { height: 64, imageName: 'photo.png', width: 64 }, type: 'image' } as const; + await expect( + runControlDetection({ + detection: canny, + input, + runFilterGraph: () => Promise.reject(new Error('queue offline')), + signal: new AbortController().signal, + }) + ).rejects.toMatchObject({ code: 'process-failed', detail: 'queue offline' }); + + const controller = new AbortController(); + const run = runControlDetection({ + detection: canny, + input, + runFilterGraph: () => { + controller.abort(); + return Promise.reject(new Error('cancelled')); + }, + signal: controller.signal, + }); + await expect(run).rejects.toMatchObject({ name: 'AbortError' }); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.ts new file mode 100644 index 00000000000..2cc3628667c --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionRunner.ts @@ -0,0 +1,122 @@ +/** Runs one detector graph on a control layer's detection input and places the map in the input's local frame. */ + +import type { + CanvasControlDetectionContract, + CanvasImageRef, + CanvasPixelSourceContract, +} from '@workbench/canvas-engine/contracts'; +import type { Rect } from '@workbench/canvas-engine/types'; +import type { ControlDetectionErrorCode } from '@workbench/canvas-operations/operationTypes'; +import type { BackendGraphContract } from '@workbench/graphContracts'; + +import { buildFilterGraph, getFilterDefinition } from '@workbench/canvas-operations/filterGraphs'; +import { + LayerFilterOutputDimensionError, + resolveFilterOutputRect, +} from '@workbench/canvas-operations/layerFilterRunner'; + +/** + * An expected detection failure with its typed code; `detail` is raw backend or engine text, never copy. Aborts + * propagate as `AbortError` instead. + */ +export class ControlDetectionFailure extends Error { + readonly code: ControlDetectionErrorCode; + readonly detail: string | undefined; + + constructor(code: ControlDetectionErrorCode, detail?: string) { + super(detail ?? code); + this.name = 'ControlDetectionFailure'; + this.code = code; + this.detail = detail; + } +} + +export interface ControlDetectionRunOptions { + input: CanvasPixelSourceContract; + detection: CanvasControlDetectionContract; + signal: AbortSignal; + runFilterGraph(options: { + graph: BackendGraphContract; + outputNodeId: string; + signal?: AbortSignal; + }): Promise<{ height: number; imageName: string; width: number }>; +} + +export interface ControlDetectionMap { + image: CanvasImageRef; + /** Where the map sits in the detection input's local frame. */ + rect: Rect; +} + +/** The persisted image and content rect of a detection input; null for a paint source without uploaded pixels. */ +export const getControlInputImage = (input: CanvasPixelSourceContract): { imageName: string; rect: Rect } | null => { + if (input.type === 'image') { + return { + imageName: input.image.imageName, + rect: { height: input.image.height, width: input.image.width, x: 0, y: 0 }, + }; + } + if (!input.bitmap) { + return null; + } + const offset = input.offset ?? { x: 0, y: 0 }; + return { + imageName: input.bitmap.imageName, + rect: { height: input.bitmap.height, width: input.bitmap.width, x: offset.x, y: offset.y }, + }; +}; + +const isAbortError = (error: unknown): boolean => error instanceof DOMException && error.name === 'AbortError'; + +/** Rejects as soon as `signal` aborts, so a cancelled run frees its concurrency slot without waiting on the queue. */ +const raceAbort = (promise: Promise, signal: AbortSignal): Promise => + new Promise((resolve, reject) => { + const onAbort = (): void => reject(signal.reason); + signal.addEventListener('abort', onAbort, { once: true }); + promise.then(resolve, reject).finally(() => signal.removeEventListener('abort', onAbort)); + }); + +export const runControlDetection = async (options: ControlDetectionRunOptions): Promise => { + options.signal.throwIfAborted(); + const { detection } = options; + if (!getFilterDefinition(detection.type)) { + throw new ControlDetectionFailure('unknown-detector', detection.type); + } + const source = getControlInputImage(options.input); + if (!source) { + throw new ControlDetectionFailure('source-not-ready'); + } + const built = buildFilterGraph(detection.type, source.imageName, detection.settings, { + height: source.rect.height, + preserveTransparency: true, + width: source.rect.width, + }); + let output: { height: number; imageName: string; width: number }; + try { + output = await raceAbort( + options.runFilterGraph({ graph: built.graph, outputNodeId: built.outputNodeId, signal: options.signal }), + options.signal + ); + } catch (error) { + options.signal.throwIfAborted(); + if (isAbortError(error)) { + throw error; + } + throw new ControlDetectionFailure('process-failed', error instanceof Error ? error.message : String(error)); + } + options.signal.throwIfAborted(); + try { + const rect = resolveFilterOutputRect({ + filterType: detection.type, + output, + settings: detection.settings, + source: source.rect, + }); + return { image: { height: output.height, imageName: output.imageName, width: output.width }, rect }; + } catch (error) { + if (error instanceof LayerFilterOutputDimensionError) { + throw new ControlDetectionFailure('output-dimensions', error.message); + } + throw error; + } +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.test.ts new file mode 100644 index 00000000000..0e7a90aa731 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.test.ts @@ -0,0 +1,49 @@ +import { describe, expect, it, vi } from 'vitest'; + +import { createControlDetectionStatusStore } from './controlDetectionStatusStore'; + +describe('controlDetectionStatusStore', () => { + it('notifies only the changed layer and keeps snapshots stable between changes', () => { + const store = createControlDetectionStatusStore(); + const first = vi.fn(); + const second = vi.fn(); + store.subscribe('a', first); + store.subscribe('b', second); + + store.set('a', { detector: 'canny_edge_detection', status: 'processing' }); + const snapshot = store.get('a'); + store.set('a', { detector: 'canny_edge_detection', status: 'processing' }); + + expect(first).toHaveBeenCalledOnce(); + expect(second).not.toHaveBeenCalled(); + expect(store.get('a')).toBe(snapshot); + expect(store.get('b')).toEqual({ status: 'idle' }); + }); + + it('forgets idle layers and clears every layer with a notification', () => { + const store = createControlDetectionStatusStore(); + const listener = vi.fn(); + store.set('a', { status: 'queued' }); + store.set('b', { error: { code: 'process-failed', detail: 'boom' }, status: 'error' }); + store.subscribe('b', listener); + + store.set('a', { status: 'idle' }); + expect(store.layerIds()).toEqual(['b']); + + store.clear(); + expect(listener).toHaveBeenCalledOnce(); + expect(store.layerIds()).toEqual([]); + expect(store.get('b')).toEqual({ status: 'idle' }); + }); + + it('stops notifying after unsubscribe', () => { + const store = createControlDetectionStatusStore(); + const listener = vi.fn(); + const unsubscribe = store.subscribe('a', listener); + + unsubscribe(); + store.set('a', { status: 'queued' }); + + expect(listener).not.toHaveBeenCalled(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.ts new file mode 100644 index 00000000000..2dcedfad338 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlDetectionStatusStore.ts @@ -0,0 +1,82 @@ +/** Per-layer detection status for control layers; React-free and `useSyncExternalStore`-compatible. */ + +import type { ControlDetectionStatus } from '@workbench/canvas-operations/operationTypes'; + +export interface ControlDetectionStatusStore { + /** The layer's status; a layer without an entry is idle. Snapshots are stable until the next `set`. */ + get(layerId: string): ControlDetectionStatus; + /** Setting idle removes the entry, so the store only holds layers with something to show. */ + set(layerId: string, status: ControlDetectionStatus): void; + subscribe(layerId: string, listener: () => void): () => void; + /** Layer ids that currently hold a non-idle status. */ + layerIds(): string[]; + /** Resets every layer to idle, notifying their listeners. */ + clear(): void; +} + +export const IDLE_CONTROL_DETECTION_STATUS: ControlDetectionStatus = Object.freeze({ status: 'idle' }); + +const sameStatus = (a: ControlDetectionStatus, b: ControlDetectionStatus): boolean => { + if (a.status !== b.status) { + return false; + } + if (a.status === 'processing' && b.status === 'processing') { + return a.detector === b.detector; + } + if (a.status === 'error' && b.status === 'error') { + return a.error.code === b.error.code && a.error.detail === b.error.detail; + } + return true; +}; + +export const createControlDetectionStatusStore = (): ControlDetectionStatusStore => { + const statuses = new Map(); + const listeners = new Map void>>(); + + const notify = (layerId: string): void => { + for (const listener of listeners.get(layerId) ?? []) { + listener(); + } + }; + + const set = (layerId: string, status: ControlDetectionStatus): void => { + const previous = statuses.get(layerId) ?? IDLE_CONTROL_DETECTION_STATUS; + if (sameStatus(previous, status)) { + return; + } + if (status.status === 'idle') { + statuses.delete(layerId); + } else { + statuses.set(layerId, status); + } + notify(layerId); + }; + + return { + clear: () => { + const ids = [...statuses.keys()]; + statuses.clear(); + for (const layerId of ids) { + notify(layerId); + } + }, + get: (layerId) => statuses.get(layerId) ?? IDLE_CONTROL_DETECTION_STATUS, + layerIds: () => [...statuses.keys()], + set, + subscribe: (layerId, listener) => { + let layerListeners = listeners.get(layerId); + if (!layerListeners) { + layerListeners = new Set(); + listeners.set(layerId, layerListeners); + } + layerListeners.add(listener); + return () => { + const current = listeners.get(layerId); + current?.delete(listener); + if (current?.size === 0) { + listeners.delete(layerId); + } + }; + }, + }; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.test.ts new file mode 100644 index 00000000000..ed24616c1c1 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.test.ts @@ -0,0 +1,225 @@ +import type { CanvasControlLayerContract } from '@workbench/canvas-engine/contracts'; + +import { createControlLayer, needsControlDetection } from '@workbench/canvas-engine/api'; +import { describe, expect, it } from 'vitest'; + +import { applyControlLayerIntent } from './controlLayerIntents'; + +const image = (imageName: string, size = 64) => ({ height: size, imageName, width: size }); +const photo = { image: image('photo.png'), type: 'image' } as const; +const mapSource = { bitmap: image('map.png'), offset: { x: 0, y: 0 }, type: 'paint' } as const; +const depth = { settings: { model_size: 'small_v2' }, type: 'depth_anything_depth_estimation' }; +const options = { + detectorDefaults: (detector: string): Record => + ({ + canny_edge_detection: { high_threshold: 200, low_threshold: 100 }, + depth_anything_depth_estimation: { model_size: 'small_v2' }, + dw_openpose_detection: { draw_body: true }, + hed_edge_detection: { scribble: false }, + normal_map: {}, + pidi_edge_detection: { quantize_edges: false, scribble: false }, + })[detector] ?? {}, +}; +const SHIFTED = { rotation: 0, scaleX: 1, scaleY: 1, x: 40, y: 8 }; + +const layer = (patch: Partial = {}): CanvasControlLayerContract => ({ + ...createControlLayer({ id: 'c1', name: 'Control Layer 1' }), + adapter: { + beginEndStepPct: [0, 1], + controlMode: 'balanced', + kind: 'controlnet', + model: 'explicit-model', + weight: 1, + }, + ...patch, +}); +/** A depth layer whose photo was detected into a map. */ +const detected = (patch: Partial = {}) => + layer({ controlType: 'depth', detection: depth, filter: depth, original: photo, source: mapSource, ...patch }); + +const apply = (target: CanvasControlLayerContract, intent: Parameters[1]) => { + const next = applyControlLayerIntent(target, intent, options); + if (next === 'no-detection-input') { + throw new Error('unexpected refusal'); + } + return next; +}; + +describe('applyControlLayerIntent: set-type', () => { + it('types an image with the default detector and returns to automatic model choice', () => { + const next = apply(layer({ source: photo }), { controlType: 'depth', kind: 'set-type' }); + + expect(next.controlType).toBe('depth'); + expect(next.detection).toEqual(depth); + expect(next.adapter.model).toBeNull(); + expect(next.source).toEqual(photo); + expect(needsControlDetection(next)).toBe(true); + }); + + it('chooses None for an empty layer and for a map without an original', () => { + expect(apply(layer(), { controlType: 'depth', kind: 'set-type' }).detection).toBeNull(); + const legacyMap = layer({ filter: depth, source: mapSource }); + const typed = apply(legacyMap, { controlType: 'depth', kind: 'set-type' }); + + expect(typed.detection).toBeNull(); + expect(typed.source).toEqual(mapSource); + }); + + it('keeps a detector the new type also offers, settings included', () => { + const custom = { settings: { quantize_edges: true, scribble: true }, type: 'pidi_edge_detection' }; + const next = apply(layer({ controlType: 'softedge', detection: custom, source: photo }), { + controlType: 'sketch', + kind: 'set-type', + }); + + expect(next.detection).toBe(custom); + }); + + it("switches to the new type's default and keeps the original for re-detection", () => { + const next = apply(detected(), { controlType: 'pose', kind: 'set-type' }); + + expect(next.detection).toEqual({ settings: { draw_body: true }, type: 'dw_openpose_detection' }); + expect(next.original).toEqual(photo); + expect(needsControlDetection(next)).toBe(true); + }); + + it('restores the original for a type without a default detector', () => { + const next = apply(detected({ originalTransform: SHIFTED }), { controlType: 'tile', kind: 'set-type' }); + + expect(next).toMatchObject({ controlType: 'tile', detection: null, source: photo }); + expect(next.transform).toMatchObject({ x: 40, y: 8 }); + expect(next.original).toBeUndefined(); + expect(next.filter).toBeUndefined(); + }); + + it('renames in the same patch and reports an unchanged type as the same object', () => { + const typed = detected(); + + expect(apply(typed, { controlType: 'depth', kind: 'set-type' })).toBe(typed); + const renamed = apply(typed, { controlType: 'depth', kind: 'set-type', name: 'Depth 1' }); + expect(renamed.name).toBe('Depth 1'); + expect(renamed.adapter.model).toBe('explicit-model'); + expect(apply(layer({ source: photo }), { controlType: 'depth', kind: 'set-type', name: 'Depth 1' }).name).toBe( + 'Depth 1' + ); + }); +}); + +describe('applyControlLayerIntent: set-detector', () => { + it("starts from the detector's defaults merged with the type's fixed settings", () => { + const next = apply(layer({ controlType: 'sketch', detection: null, source: photo }), { + detector: 'pidi_edge_detection', + kind: 'set-detector', + }); + + expect(next.detection).toEqual({ + settings: { quantize_edges: false, scribble: true }, + type: 'pidi_edge_detection', + }); + }); + + it('keeps the original so the new detector runs on it', () => { + const next = apply(detected(), { detector: 'normal_map', kind: 'set-detector' }); + + expect(next.detection).toEqual({ settings: {}, type: 'normal_map' }); + expect(next.original).toEqual(photo); + expect(needsControlDetection(next)).toBe(true); + }); + + it('None restores the original', () => { + const next = apply(detected(), { detector: null, kind: 'set-detector' }); + + expect(next).toMatchObject({ detection: null, source: photo }); + expect(next.original).toBeUndefined(); + }); + + it('refuses a detector for a map without an original', () => { + expect( + applyControlLayerIntent( + layer({ controlType: 'depth', detection: null, filter: depth, source: mapSource }), + { detector: 'normal_map', kind: 'set-detector' }, + options + ) + ).toBe('no-detection-input'); + }); + + it('leaves the same detector and an untyped layer unchanged', () => { + const typed = detected(); + const untyped = layer({ source: photo }); + + expect(apply(typed, { detector: depth.type, kind: 'set-detector' })).toBe(typed); + expect(apply(untyped, { detector: depth.type, kind: 'set-detector' })).toBe(untyped); + }); +}); + +describe('applyControlLayerIntent: set-detector-settings', () => { + it('changes only the settings and keeps fixed ones', () => { + const sketch = layer({ + controlType: 'sketch', + detection: { settings: { quantize_edges: false, scribble: true }, type: 'pidi_edge_detection' }, + source: photo, + }); + const next = apply(sketch, { kind: 'set-detector-settings', settings: { quantize_edges: true, scribble: false } }); + + expect(next).toEqual({ + ...sketch, + detection: { settings: { quantize_edges: true, scribble: true }, type: 'pidi_edge_detection' }, + }); + }); + + it('leaves a layer without a detector, or with equal settings, unchanged', () => { + const none = layer({ controlType: 'depth', detection: null, source: photo }); + const typed = detected(); + + expect(apply(none, { kind: 'set-detector-settings', settings: { a: 1 } })).toBe(none); + expect(apply(typed, { kind: 'set-detector-settings', settings: { model_size: 'small_v2' } })).toBe(typed); + }); +}); + +describe('applyControlLayerIntent: replace-image and draw', () => { + const upload = image('upload.png', 128); + + it('replaces the source, drops derived pixels and stales the map', () => { + const next = apply(detected({ originalTransform: SHIFTED }), { + image: upload, + kind: 'replace-image', + transform: SHIFTED, + }); + + expect(next.source).toEqual({ image: upload, type: 'image' }); + expect(next.transform).toEqual(SHIFTED); + for (const key of ['original', 'originalTransform', 'filter'] as const) { + expect(next).not.toHaveProperty(key); + } + expect(next.detection).toEqual(depth); + expect(needsControlDetection(next)).toBe(true); + }); + + it("gives a first image the type's default detector when None was chosen only for lack of pixels", () => { + const next = apply(layer({ controlType: 'depth', detection: null }), { image: upload, kind: 'replace-image' }); + + expect(next.detection).toEqual(depth); + }); + + it('keeps None when the layer already had an image, or the type has no default', () => { + expect( + apply(layer({ controlType: 'depth', detection: null, source: photo }), { image: upload, kind: 'replace-image' }) + .detection + ).toBeNull(); + expect( + apply(layer({ controlType: 'sketch', detection: null }), { image: upload, kind: 'replace-image' }).detection + ).toBeNull(); + expect(apply(layer(), { image: upload, kind: 'replace-image' })).not.toHaveProperty('detection'); + }); + + it('draw empties the image and uses it as-is', () => { + const next = apply(detected(), { kind: 'draw' }); + + expect(next.source).toEqual({ bitmap: null, type: 'paint' }); + expect(next.detection).toBeNull(); + for (const key of ['original', 'originalTransform', 'filter'] as const) { + expect(next).not.toHaveProperty(key); + } + expect(apply(layer({ source: photo }), { kind: 'draw' })).not.toHaveProperty('detection'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.ts new file mode 100644 index 00000000000..510bcf2d05e --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlLayerIntents.ts @@ -0,0 +1,180 @@ +/** + * Pure rules for the control layer Properties panel: each intent maps the live layer to the next one. Detection + * itself never runs here; the coordinator schedules it when the result needs a fresh map. + */ + +import type { + CanvasControlDetectionContract, + CanvasControlLayerContract, + CanvasLayerSourceContract, + ControlType, +} from '@workbench/canvas-engine/contracts'; +import type { ControlLayerIntent } from '@workbench/canvas-operations/operationTypes'; + +import { areJsonValuesStructurallyEqual } from '@platform/core/json'; +import { getControlDetectionInput, restoreControlOriginal } from '@workbench/canvas-engine/api'; +import { CONTROL_TYPE_SPECS } from '@workbench/controlTypes'; + +export interface ControlLayerIntentOptions { + /** A detector's default settings; an unknown detector has none. */ + detectorDefaults(detector: string): Record; +} + +const hasPixels = (source: CanvasLayerSourceContract | null): boolean => + source !== null && (source.type === 'image' || (source.type === 'paint' && source.bitmap !== null)); + +/** Whether there are pixels detection could run on (I5): map-only and empty layers have none. */ +const hasDetectionPixels = (layer: CanvasControlLayerContract): boolean => hasPixels(getControlDetectionInput(layer)); + +const settingsMatch = (required: Record, actual: Record): boolean => + Object.entries(required).every(([key, value]) => areJsonValuesStructurallyEqual(actual[key], value)); + +/** The type's entry for `detector`; a type may fix some of its settings (PiDi as a sketch detector). */ +const findDetectorSpec = (type: ControlType, detector: string) => + CONTROL_TYPE_SPECS[type].detectors.find((entry) => entry.type === detector); + +const belongsToType = (type: ControlType, detection: CanvasControlDetectionContract): boolean => + CONTROL_TYPE_SPECS[type].detectors.some( + (entry) => entry.type === detection.type && settingsMatch(entry.settings ?? {}, detection.settings) + ); + +const detectionFor = ( + type: ControlType, + detector: string, + options: ControlLayerIntentOptions +): CanvasControlDetectionContract => ({ + settings: { ...options.detectorDefaults(detector), ...findDetectorSpec(type, detector)?.settings }, + type: detector, +}); + +/** Drops every pixel the layer derived from or was filtered from; the caller sets the new source. */ +const withoutDerivedPixels = (layer: CanvasControlLayerContract): CanvasControlLayerContract => { + const next = { ...layer }; + delete next.original; + delete next.originalTransform; + delete next.filter; + return next; +}; + +const setType = ( + layer: CanvasControlLayerContract, + controlType: ControlType, + options: ControlLayerIntentOptions +): CanvasControlLayerContract => { + if (layer.controlType === controlType) { + return layer; + } + const typed: CanvasControlLayerContract = { + ...layer, + adapter: { ...layer.adapter, model: null }, + controlType, + }; + const spec = CONTROL_TYPE_SPECS[controlType]; + let detection: CanvasControlDetectionContract | null; + if (!hasDetectionPixels(layer)) { + detection = null; + } else if (layer.detection && belongsToType(controlType, layer.detection)) { + detection = layer.detection; + } else { + detection = spec.defaultDetector ? detectionFor(controlType, spec.defaultDetector, options) : null; + } + return detection === null ? restoreControlOriginal(typed) : { ...typed, detection }; +}; + +const setDetector = ( + layer: CanvasControlLayerContract, + detector: string | null, + options: ControlLayerIntentOptions +): CanvasControlLayerContract | 'no-detection-input' => { + if (layer.controlType === undefined) { + return layer; + } + if (detector === null) { + return restoreControlOriginal(layer); + } + if (getControlDetectionInput(layer) === null) { + return 'no-detection-input'; + } + if (layer.detection?.type === detector) { + return layer; + } + return { ...layer, detection: detectionFor(layer.controlType, detector, options) }; +}; + +const setDetectorSettings = ( + layer: CanvasControlLayerContract, + settings: Record +): CanvasControlLayerContract => { + if (layer.controlType === undefined || !layer.detection) { + return layer; + } + const fixed = findDetectorSpec(layer.controlType, layer.detection.type)?.settings; + return { + ...layer, + detection: { settings: { ...layer.detection.settings, ...settings, ...fixed }, type: layer.detection.type }, + }; +}; + +const replaceImage = ( + layer: CanvasControlLayerContract, + intent: Extract, + options: ControlLayerIntentOptions +): CanvasControlLayerContract => { + const next: CanvasControlLayerContract = { + ...withoutDerivedPixels(layer), + source: { image: { ...intent.image }, type: 'image' }, + }; + if (intent.transform) { + next.transform = { ...intent.transform }; + } + // None was chosen only because there was nothing to detect; a first image gets the type's detector. + const wasEmpty = layer.original === undefined && !hasPixels(layer.source); + const defaultDetector = layer.controlType ? CONTROL_TYPE_SPECS[layer.controlType].defaultDetector : null; + if (layer.controlType && layer.detection === null && wasEmpty && defaultDetector) { + next.detection = detectionFor(layer.controlType, defaultDetector, options); + } + return next; +}; + +const draw = (layer: CanvasControlLayerContract): CanvasControlLayerContract => { + const next: CanvasControlLayerContract = { ...withoutDerivedPixels(layer), source: { bitmap: null, type: 'paint' } }; + if (next.controlType !== undefined) { + next.detection = null; + } + return next; +}; + +/** + * The layer after `intent`, or the same object when nothing changes. `'no-detection-input'` refuses a detector for a + * layer whose pixels are a map without an original (a legacy filter result): replace the image first. + */ +export const applyControlLayerIntent = ( + layer: CanvasControlLayerContract, + intent: ControlLayerIntent, + options: ControlLayerIntentOptions +): CanvasControlLayerContract | 'no-detection-input' => { + let next: CanvasControlLayerContract | 'no-detection-input'; + switch (intent.kind) { + case 'set-type': { + const typed = setType(layer, intent.controlType, options); + next = intent.name === undefined || intent.name === typed.name ? typed : { ...typed, name: intent.name }; + break; + } + case 'set-detector': + next = setDetector(layer, intent.detector, options); + break; + case 'set-detector-settings': + next = setDetectorSettings(layer, intent.settings); + break; + case 'replace-image': + next = replaceImage(layer, intent, options); + break; + case 'draw': + next = draw(layer); + break; + } + if (next === 'no-detection-input') { + return next; + } + return areJsonValuesStructurallyEqual(next, layer) ? layer : next; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.test.ts deleted file mode 100644 index 0d43066f142..00000000000 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.test.ts +++ /dev/null @@ -1,21 +0,0 @@ -import { describe, expect, it } from 'vitest'; - -import { resolveDefaultFilterForModel } from './controlRecommendations'; - -describe('resolveDefaultFilterForModel', () => { - it.each([ - ['canny_edge_detection', 'canny_edge_detection'], - ['canny_image_processor', 'canny_edge_detection'], - ['normalbae_image_processor', 'normal_map'], - ['zoe_depth_image_processor', 'depth_anything_depth_estimation'], - ['lineart_anime_image_processor', 'lineart_anime_edge_detection'], - ])('resolves current ids and legacy alias %s', (preprocessor, expected) => { - expect(resolveDefaultFilterForModel({ default_settings: { preprocessor } })).toBe(expected); - }); - - it('returns null for missing or unknown metadata', () => { - expect(resolveDefaultFilterForModel(null)).toBeNull(); - expect(resolveDefaultFilterForModel({ default_settings: null })).toBeNull(); - expect(resolveDefaultFilterForModel({ default_settings: { preprocessor: 'future_processor' } })).toBeNull(); - }); -}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.ts deleted file mode 100644 index 4285d42e338..00000000000 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlRecommendations.ts +++ /dev/null @@ -1,32 +0,0 @@ -import type { ModelConfig } from '@features/models'; - -import { isSupportedFilterType } from './filterGraphs'; - -export const PROCESSOR_TO_FILTER_MAP: Readonly> = { - canny_image_processor: 'canny_edge_detection', - color_map_image_processor: 'color_map', - content_shuffle_image_processor: 'content_shuffle', - depth_anything_image_processor: 'depth_anything_depth_estimation', - dw_openpose_image_processor: 'dw_openpose_detection', - hed_image_processor: 'hed_edge_detection', - lineart_anime_image_processor: 'lineart_anime_edge_detection', - lineart_image_processor: 'lineart_edge_detection', - mediapipe_face_processor: 'mediapipe_face_detection', - mlsd_image_processor: 'mlsd_detection', - normalbae_image_processor: 'normal_map', - pidi_image_processor: 'pidi_edge_detection', - zoe_depth_image_processor: 'depth_anything_depth_estimation', -}; - -export const resolveDefaultFilterForModel = ( - model: Pick | null | undefined -): string | null => { - const preprocessor = model?.default_settings?.preprocessor; - if (!preprocessor) { - return null; - } - if (isSupportedFilterType(preprocessor)) { - return preprocessor; - } - return PROCESSOR_TO_FILTER_MAP[preprocessor] ?? null; -}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.test.ts new file mode 100644 index 00000000000..b783ee2a09e --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.test.ts @@ -0,0 +1,225 @@ +import type { GalleryImage } from '@features/gallery'; +import type { CanvasControlLayerContract, CanvasLayerContract } from '@workbench/canvas-engine/api'; +import type { Project } from '@workbench/projectContracts'; + +import { accountLifecycle } from '@platform/state/accountLifecycle'; +import { createControlLayer } from '@workbench/canvas-engine/api'; +import { groupContract, stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { createTestInsertionAnchorCapture } from '@workbench/canvas-engine/document/insertionAnchors.testStub'; +import { createInitialWorkbenchState } from '@workbench/workbenchState.testing'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import type { uploadCanvasImage } from './backend/canvasImages'; +import type { ControlLayerIntent, ControlLayerIntentResult } from './operationTypes'; + +import { fillControlFromBbox, sendGalleryImageAsControl } from './controlSourceActions'; +import { attachCanvasOperations } from './operationAccess'; + +const BBOX = { height: 512, width: 768, x: 31, y: 47 }; + +const engineWith = (composite: unknown) => { + const intents: ControlLayerIntent[] = []; + const engine = { + document: { getDocument: () => ({ bbox: BBOX }) }, + exports: { exportRasterComposite: () => composite }, + lifecycle: { flushPendingUploads: () => Promise.resolve() }, + projectId: 'project-1', + } as unknown as Parameters[0]['engine']; + attachCanvasOperations(engine, { + applyControlLayerIntent: (_layerId: string, intent: ControlLayerIntent) => { + intents.push(intent); + return 'updated'; + }, + } as never); + return { engine, intents }; +}; + +afterEach(() => { + accountLifecycle.invalidate(); +}); + +describe('fillControlFromBbox', () => { + it('uploads the bbox composite as a control image and places it at the bbox origin', async () => { + const { engine, intents } = engineWith({ blob: new Blob(['png']), rect: BBOX, status: 'ok' }); + const uploads: Parameters[1][] = []; + const uploadImage: typeof uploadCanvasImage = (_blob, options) => { + uploads.push(options); + return Promise.resolve({ height: 512, imageName: 'bbox.png', width: 768 }); + }; + + await expect(fillControlFromBbox({ engine, layerId: 'control-1', uploadImage })).resolves.toEqual({ + imageName: 'bbox.png', + result: 'updated', + status: 'applied', + }); + expect(uploads[0]).toMatchObject({ imageCategory: 'control', projectId: 'project-1' }); + expect(intents).toEqual([ + { + image: { height: 512, imageName: 'bbox.png', width: 768 }, + kind: 'replace-image', + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 31, y: 47 }, + }, + ]); + }); + + it('reports an empty bbox without uploading', async () => { + const { engine, intents } = engineWith({ status: 'empty' }); + const uploadImage = () => Promise.reject(new Error('should not upload')); + + await expect(fillControlFromBbox({ engine, layerId: 'control-1', uploadImage })).resolves.toEqual({ + status: 'empty', + }); + expect(intents).toEqual([]); + }); + + it('drops an upload that finishes after the account changed', async () => { + accountLifecycle.activate('control-source-a'); + const { engine, intents } = engineWith({ blob: new Blob(['png']), rect: BBOX, status: 'ok' }); + const uploadImage: typeof uploadCanvasImage = () => { + accountLifecycle.activate('control-source-b'); + return Promise.resolve({ height: 1, imageName: 'late.png', width: 1 }); + }; + + await expect(fillControlFromBbox({ engine, layerId: 'control-1', uploadImage })).resolves.toEqual({ + status: 'stale', + }); + expect(intents).toEqual([]); + }); +}); + +describe('sendGalleryImageAsControl', () => { + const IMAGE = { + boardId: 'none', + height: 160, + imageCategory: 'general', + imageName: 'gallery.png', + imageUrl: '/gallery.png', + queuedAt: '2026-01-01T00:00:00.000Z', + sourceQueueItemId: 'queue-1', + starred: false, + thumbnailUrl: '/thumb/gallery.png', + width: 320, + } as GalleryImage; + const emptyControl = createControlLayer({ id: 'control-1', name: 'Depth 1' }); + + const projectWith = (layers: CanvasLayerContract[], selectedLayerId: string | null): Project => { + const state = createInitialWorkbenchState(); + const current = state.projects.find((candidate) => candidate.id === state.activeProjectId)!; + const document = { ...current.canvas.document, bbox: BBOX, selectedLayerId, stacks: stacksFrom(layers) }; + return { ...current, canvas: { ...current.canvas, document } }; + }; + + /** A live canvas for `project`; `hasPixels` stands for strokes only the engine's cache holds. */ + const liveEngine = ( + project: Project, + { hasPixels = false, intentResult = 'updated' as ControlLayerIntentResult } = {} + ) => { + const intents: [string, ControlLayerIntent][] = []; + const commitStructural = vi.fn(() => ({ status: 'committed' as const })); + const engine = { + document: { captureInsertionAnchor: createTestInsertionAnchorCapture(project.id) }, + exports: { hasExportableLayerContent: () => hasPixels }, + layers: { canCommitStructural: () => true, commitStructural }, + projectId: project.id, + } as unknown as Parameters[0]['engine'] & object; + attachCanvasOperations(engine, { + applyControlLayerIntent: (layerId: string, intent: ControlLayerIntent) => { + intents.push([layerId, intent]); + return intentResult; + }, + } as never); + return { commitStructural, engine, intents }; + }; + + const send = ( + project: Project, + engine: Parameters[0]['engine'], + latest = project + ) => { + const applyCanvasMutation = vi.fn(); + const result = sendGalleryImageAsControl({ + applyCanvasMutation, + engine, + getProject: (projectId) => (projectId === latest.id ? latest : null), + image: IMAGE, + isActiveProject: (projectId) => projectId === project.id, + project, + }); + return { applyCanvasMutation, result }; + }; + + it('fills the selected control layer that has no image yet, at the bbox origin', async () => { + const project = projectWith([emptyControl], 'control-1'); + const { commitStructural, engine, intents } = liveEngine(project); + + await expect(send(project, engine).result).resolves.toEqual({ + layerId: 'control-1', + layerName: 'Depth 1', + status: 'filled', + }); + expect(intents).toEqual([ + [ + 'control-1', + { + image: { height: 160, imageName: 'gallery.png', width: 320 }, + kind: 'replace-image', + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 31, y: 47 }, + }, + ], + ]); + expect(commitStructural).not.toHaveBeenCalled(); + }); + + it.each([ + ['the selected control layer has an image', [{ ...emptyControl, source: { image: IMAGE, type: 'image' } }], false], + ['the selected control layer holds strokes not saved yet', [emptyControl], true], + ] as const)('adds a new control layer when %s', async (_label, layers, hasPixels) => { + const project = projectWith([...layers] as CanvasControlLayerContract[], 'control-1'); + const { commitStructural, engine, intents } = liveEngine(project, { hasPixels }); + + const result = await send(project, engine).result; + + expect(intents).toEqual([]); + expect(commitStructural).toHaveBeenCalledOnce(); + const [, forward] = commitStructural.mock.calls[0] as unknown as [ + string, + { add: { nodes: CanvasLayerContract[] }[] }, + ]; + const added = forward.add[0]!.nodes[0]!; + expect(added).toMatchObject({ source: { image: { imageName: 'gallery.png' }, type: 'image' }, type: 'control' }); + expect(result).toEqual({ layerId: added.id, status: 'created' }); + }); + + it('adds a new control layer instead of filling an empty one inside a locked group', async () => { + const lockedGroup = groupContract('group-1', [emptyControl], { isLocked: true }); + const project = projectWith([lockedGroup as unknown as CanvasLayerContract], 'control-1'); + const { commitStructural, engine, intents } = liveEngine(project); + + const result = await send(project, engine).result; + + expect(intents).toEqual([]); + expect(commitStructural).toHaveBeenCalledOnce(); + expect(result).toMatchObject({ status: 'created' }); + }); + + it('adds a new control layer through ingestion when the project has no live canvas', async () => { + const project = projectWith([emptyControl], 'control-1'); + + const { applyCanvasMutation, result } = send(project, null); + + await expect(result).resolves.toMatchObject({ status: 'created' }); + expect(applyCanvasMutation).toHaveBeenCalledOnce(); + }); + + it('refuses a document that changed since the caller read it, and reports a refused fill', async () => { + const project = projectWith([emptyControl], 'control-1'); + const { engine, intents } = liveEngine(project); + const latest = { ...project, canvas: { ...project.canvas, document: { ...project.canvas.document } } }; + + await expect(send(project, engine, latest).result).resolves.toEqual({ status: 'stale-document' }); + expect(intents).toEqual([]); + + const busy = liveEngine(project, { intentResult: 'blocked' }); + await expect(send(project, busy.engine).result).resolves.toEqual({ status: 'blocked' }); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.ts new file mode 100644 index 00000000000..b4241c176e7 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/controlSourceActions.ts @@ -0,0 +1,352 @@ +import type { AccountScope } from '@platform/state/accountLifecycle'; +import type { + AdjustmentRefusal, + CanvasDocumentCapability, + CanvasDocumentContractV3, + CanvasExportCapability, +} from '@workbench/canvas-engine/api'; +import type { CanvasProjectMutation } from '@workbench/canvasProjectMutations'; +import type { Project } from '@workbench/projectContracts'; + +import { galleryTransfers, type GalleryImage } from '@features/gallery'; +import { + assertAccountScopeCurrent, + captureAccountScope, + isAccountScopeCurrent, +} from '@platform/state/accountLifecycle'; +import { getDocumentLayer, lookupDocumentLeaf } from '@workbench/canvas-engine/api'; +import { getControlImageState } from '@workbench/canvasLayerContent'; + +import type { ControlLayerIntentResult } from './operationTypes'; + +import { uploadCanvasImage } from './backend/canvasImages'; +import { exportCanvasComposite, type CanvasCompositeExportEngine } from './exportCanvasComposite'; +import { + importGalleryImagesToCanvas, + type GalleryImportEngine, + type ImportGalleryImagesResult, +} from './importGalleryImages'; +import { getCanvasOperations } from './operationAccess'; + +/** Where a control layer's image came from, when this canvas placed it: a file (by name) or the frame. */ +export type ControlSourceOrigin = { kind: 'file'; name: string } | { kind: 'frame' }; + +export interface ControlSourceSnapshot { + /** Control layers with an image upload on its way; another upload to one is refused until it lands. */ + readonly uploading: ReadonlySet; + /** Origins of images placed from a file or the frame, by image name; others are described by their size. */ + readonly origins: Readonly>; +} + +const EMPTY_SOURCE_SNAPSHOT: ControlSourceSnapshot = { origins: {}, uploading: new Set() }; + +interface ControlSourceState { + snapshot: ControlSourceSnapshot; + readonly listeners: Set<() => void>; +} + +// Kept per engine, as its operations are: an account or project switch replaces the engine, and this state with it. +const sourceStates = new WeakMap(); + +const sourceStateOf = (engine: object): ControlSourceState => { + let state = sourceStates.get(engine); + if (!state) { + state = { listeners: new Set(), snapshot: EMPTY_SOURCE_SNAPSHOT }; + sourceStates.set(engine, state); + } + return state; +}; + +const updateSourceState = (engine: object, next: (snapshot: ControlSourceSnapshot) => ControlSourceSnapshot) => { + const state = sourceStateOf(engine); + state.snapshot = next(state.snapshot); + for (const listener of state.listeners) { + listener(); + } +}; + +export const getControlSourceSnapshot = (engine: object): ControlSourceSnapshot => + sourceStates.get(engine)?.snapshot ?? EMPTY_SOURCE_SNAPSHOT; + +export const subscribeControlSourceSnapshot = (engine: object, listener: () => void): (() => void) => { + const { listeners } = sourceStateOf(engine); + listeners.add(listener); + return () => { + listeners.delete(listener); + }; +}; + +const recordOrigin = (engine: object, imageName: string, origin: ControlSourceOrigin) => + updateSourceState(engine, (snapshot) => ({ ...snapshot, origins: { ...snapshot.origins, [imageName]: origin } })); + +const setUploading = (engine: object, layerId: string, uploading: boolean) => + updateSourceState(engine, (snapshot) => { + const next = new Set(snapshot.uploading); + if (uploading) { + next.add(layerId); + } else { + next.delete(layerId); + } + return { ...snapshot, uploading: next }; + }); + +export type UploadControlImageResult = + | { status: 'applied'; result: ControlLayerIntentResult; imageName: string } + /** Another upload to the layer is still on its way. */ + | { status: 'uploading' } + | { status: 'stale' } + | { status: 'failed'; message: string }; + +/** + * Uploads an image file and makes it a control layer's image at the bbox origin (`replace-image`, which detects its + * map), recording the file's name as the image's origin. The Properties Upload button and a file dropped on the + * canvas share it. One upload per layer at a time; `onUploaded` runs once the file is stored, for the caller to + * refresh the gallery it now appears in. The upload stays with the account that started it. + */ +export const uploadControlImageFile = async (options: { + engine: { readonly document: Pick }; + layerId: string; + file: File; + onUploaded?: (owner: AccountScope) => void; + uploadImage?: typeof galleryTransfers.upload; +}): Promise => { + const { engine, file, layerId, onUploaded, uploadImage = galleryTransfers.upload } = options; + if (getControlSourceSnapshot(engine).uploading.has(layerId)) { + return { status: 'uploading' }; + } + const owner = captureAccountScope(); + setUploading(engine, layerId, true); + try { + const uploaded = await uploadImage(file, 'none', { signal: owner.signal }); + assertAccountScopeCurrent(owner); + onUploaded?.(owner); + const bbox = engine.document.getDocument()?.bbox; + const result = getCanvasOperations(engine).applyControlLayerIntent(layerId, { + image: { height: uploaded.height, imageName: uploaded.imageName, width: uploaded.width }, + kind: 'replace-image', + // New images land at the bbox origin, as every other canvas import does. + transform: bbox ? { rotation: 0, scaleX: 1, scaleY: 1, x: bbox.x, y: bbox.y } : undefined, + }); + if (result === 'updated') { + recordOrigin(engine, uploaded.imageName, { kind: 'file', name: file.name }); + } + return { imageName: uploaded.imageName, result, status: 'applied' }; + } catch (error) { + return isAccountScopeCurrent(owner) + ? { message: error instanceof Error ? error.message : String(error), status: 'failed' } + : { status: 'stale' }; + } finally { + setUploading(engine, layerId, false); + } +}; + +/** A notice for the user: locale keys for the title and message, or a message as the server gave it. */ +export interface ControlSourceNotice { + kind: 'info' | 'error'; + titleKey: string; + messageKey?: string; + message?: string; +} + +/** Why a control layer intent did not land; `updated`, `unchanged` and `missing` (the layer went away) say nothing. */ +export const getControlIntentNotice = (result: ControlLayerIntentResult): ControlSourceNotice | null => { + switch (result) { + case 'blocked': + return { kind: 'info', titleKey: 'widgets.layers.actions.busy' }; + case 'locked': + return { kind: 'info', titleKey: 'widgets.layers.actions.locked' }; + case 'no-detection-input': + return { kind: 'info', titleKey: 'widgets.layers.control.detectionNoInput' }; + case 'over-budget': + return { + kind: 'error', + messageKey: 'widgets.layers.actions.overBudget', + titleKey: 'widgets.layers.actions.actionFailed', + }; + case 'failed': + return { + kind: 'error', + messageKey: 'widgets.layers.actions.operationFailed', + titleKey: 'widgets.layers.actions.actionFailed', + }; + default: + return null; + } +}; + +export const getControlUploadNotice = (outcome: UploadControlImageResult): ControlSourceNotice | null => { + switch (outcome.status) { + case 'applied': + return getControlIntentNotice(outcome.result); + case 'uploading': + return { kind: 'info', titleKey: 'widgets.layers.control.source.uploadPending' }; + case 'failed': + return { kind: 'error', message: outcome.message, titleKey: 'widgets.layers.control.source.uploadFailed' }; + case 'stale': + // The account changed on the way; the result belongs to nobody here. + return null; + } +}; + +export type ControlFileDropResolution = + | { status: 'accepted'; layerId: string } + | { status: 'refused'; notice: ControlSourceNotice }; + +/** + * Whether files dragged onto the canvas can fill the selected control layer: exactly one image file, the canvas + * accepting edits, and the selected layer a control layer that is not locked (by itself or a group) and still empty. + * `files` are the drag's file items while it hovers, and its files once dropped. + */ +export const resolveControlFileDrop = (input: { + document: CanvasDocumentContractV3; + engine: { readonly exports: Pick }; + files: readonly { readonly type: string }[]; + isCanvasLocked: boolean; +}): ControlFileDropResolution => { + const { document, engine, files, isCanvasLocked } = input; + const refuse = (titleKey: string): ControlFileDropResolution => ({ + notice: { kind: 'info', titleKey }, + status: 'refused', + }); + if (files.length !== 1) { + return refuse('widgets.canvas.fileDrop.oneFile'); + } + if (!files[0]!.type.startsWith('image/')) { + return refuse('widgets.canvas.fileDrop.notImage'); + } + if (isCanvasLocked) { + return refuse('widgets.layers.actions.busy'); + } + const layer = getDocumentLayer(document, document.selectedLayerId); + if ( + layer?.type !== 'control' || + getControlImageState(layer, engine.exports.hasExportableLayerContent(layer.id)) !== 'empty' + ) { + return refuse('widgets.canvas.fileDrop.noTarget'); + } + if (lookupDocumentLeaf(document, layer.id)?.effectiveLocked !== false) { + return refuse('widgets.layers.actions.locked'); + } + return { layerId: layer.id, status: 'accepted' }; +}; + +export type FillControlFromBboxResult = + | { status: 'applied'; result: ControlLayerIntentResult; imageName: string } + | { status: 'empty' | 'stale' | 'not-ready' | 'over-budget' } + | AdjustmentRefusal; + +/** + * Uses the visible raster content inside the bbox as a control layer's image: the composite is uploaded as a control + * image and replaces the layer's image at the bbox origin, which detects its map again. The upload stays with the + * account that started it; a sign-out on the way reports `stale`. + */ +export const fillControlFromBbox = async (options: { + engine: CanvasCompositeExportEngine & { readonly projectId: string }; + layerId: string; + uploadImage?: typeof uploadCanvasImage; +}): Promise => { + const { engine, layerId, uploadImage = uploadCanvasImage } = options; + const owner = captureAccountScope(); + try { + const exported = await exportCanvasComposite(engine, 'bbox'); + assertAccountScopeCurrent(owner); + if (exported.status !== 'ok') { + return exported; + } + const uploaded = await uploadImage(exported.blob, { + fileName: 'control.png', + imageCategory: 'control', + projectId: engine.projectId, + signal: owner.signal, + }); + assertAccountScopeCurrent(owner); + const result = getCanvasOperations(engine).applyControlLayerIntent(layerId, { + image: { height: uploaded.height, imageName: uploaded.imageName, width: uploaded.width }, + kind: 'replace-image', + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: exported.rect.x, y: exported.rect.y }, + }); + if (result === 'updated') { + recordOrigin(engine, uploaded.imageName, { kind: 'frame' }); + } + return { imageName: uploaded.imageName, result, status: 'applied' }; + } catch (error) { + if (!isAccountScopeCurrent(owner)) { + return { status: 'stale' }; + } + throw error; + } +}; + +export type SendGalleryImageAsControlResult = + | { status: 'filled'; layerId: string; layerName: string } + | { status: 'created'; layerId: string } + | Exclude; + +type SendAsControlEngine = GalleryImportEngine & { + readonly exports: Pick; +}; + +/** + * Uses a gallery image as a control layer's image. The selected control layer takes it when that layer has no image + * yet and the project is the one on the live canvas (`filled`, detected like any replaced image); otherwise a new + * control layer holds it (`created`), as the canvas import makes it. Every refusal of either path is reported, and a + * `project` older than the latest document is `stale-document`. + */ +export const sendGalleryImageAsControl = async (options: { + applyCanvasMutation: (projectId: string, mutation: CanvasProjectMutation) => boolean | void; + engine: SendAsControlEngine | null; + getProject: (projectId: string) => Project | null; + image: GalleryImage; + isActiveProject: (projectId: string) => boolean; + project: Project; +}): Promise => { + const { engine, getProject, image, isActiveProject, project } = options; + const latest = getProject(project.id); + if (!latest) { + return { status: 'stale-project' }; + } + if (latest.canvas.document !== project.canvas.document) { + return { status: 'stale-document' }; + } + + const { document } = project.canvas; + const selected = getDocumentLayer(document, document.selectedLayerId); + const liveEngine = engine !== null && engine.projectId === project.id && isActiveProject(project.id) ? engine : null; + if ( + liveEngine && + selected?.type === 'control' && + // Locked by itself or by a group above it: the image goes to a new layer instead. + lookupDocumentLeaf(document, selected.id)?.effectiveLocked === false && + getControlImageState(selected, liveEngine.exports.hasExportableLayerContent(selected.id)) === 'empty' + ) { + const result = getCanvasOperations(liveEngine).applyControlLayerIntent(selected.id, { + image: { height: image.height, imageName: image.imageName, width: image.width }, + kind: 'replace-image', + // At the bbox origin, as every other canvas import places an image. + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: document.bbox.x, y: document.bbox.y }, + }); + switch (result) { + case 'updated': + case 'unchanged': + case 'no-detection-input': + return { layerId: selected.id, layerName: selected.name, status: 'filled' }; + case 'missing': + return { status: 'stale-document' }; + case 'blocked': + case 'over-budget': + return { status: 'blocked' }; + case 'failed': + throw new Error(`Could not use ${image.imageName} as the image of control layer ${selected.id}.`); + case 'locked': + // Locked by a group above it: the image goes to a new layer instead. + break; + } + } + + const imported = await importGalleryImagesToCanvas({ ...options, destination: 'control', images: [image] }); + if (imported.status !== 'imported') { + return imported; + } + const [layerId] = imported.layerIds; + return layerId === undefined ? { status: 'empty' } : { layerId, status: 'created' }; +}; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.controlDetection.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.controlDetection.test.ts new file mode 100644 index 00000000000..c998ebf3beb --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.controlDetection.test.ts @@ -0,0 +1,184 @@ +import type { + CanvasControlLayerContract, + CanvasLayerContract, + CanvasRasterLayerContractV2, +} from '@workbench/canvas-engine/contracts'; +import type { StubRasterBackend } from '@workbench/canvas-engine/render/raster.testStub'; +import type { CanvasUtilityGraphResult } from '@workbench/canvas-operations/contracts'; + +import { createControlLayer, getDocumentLeaves } from '@workbench/canvas-engine/api'; +import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; +import { createTestStubRasterBackend } from '@workbench/canvas-engine/render/raster.testStub'; +import { createEmptyCanvasDocument } from '@workbench/canvasMigration'; +import { createCanvasProjectMutationPort } from '@workbench/canvasProjectMutationPort'; +import { createWorkbenchStore } from '@workbench/workbenchStore'; +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import { createCanvasEngine } from './createCanvasEngine'; +import { getCanvasOperations } from './operationAccess'; + +const DEBOUNCE_MS = 400; +const SIZE = 64; + +const photoControl = (id: string): CanvasControlLayerContract => ({ + ...createControlLayer({ id, name: id }), + source: { image: { height: SIZE, imageName: `${id}.png`, width: SIZE }, type: 'image' }, +}); + +const shapeRaster = (id: string): CanvasRasterLayerContractV2 => ({ + blendMode: 'normal', + id, + isEnabled: true, + isLocked: false, + name: id, + opacity: 1, + source: { fill: '#fff', height: SIZE, kind: 'rect', stroke: null, strokeWidth: 0, type: 'shape', width: SIZE }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'raster', +}); + +/** Decodes every image at the test size, so a detected map lands on its input. */ +const sizedBackend = (): StubRasterBackend => ({ + ...createTestStubRasterBackend(), + createImageBitmap: () => + Promise.resolve({ close: () => undefined, height: SIZE, width: SIZE } as unknown as ImageBitmap), +}); + +/** A real engine; graphs stay pending unless `runGraph` answers them. */ +const setup = ( + layers: CanvasLayerContract[] = [photoControl('c1')], + runGraph: (options: { signal?: AbortSignal }) => Promise = () => new Promise(() => {}) +) => { + const store = createWorkbenchStore(); + const projectId = store.getState().activeProjectId; + store.commands.canvas.apply(projectId, { + document: { ...createEmptyCanvasDocument(), selectedLayerId: layers[0]!.id, stacks: stacksFrom(layers) }, + type: 'replaceCanvasDocument', + }); + const graphSignals: AbortSignal[] = []; + const graph = vi.fn((options: { signal?: AbortSignal }) => { + graphSignals.push(options.signal!); + return runGraph(options); + }); + const makeDurable = vi.fn((_imageName: string) => Promise.resolve()); + const engine = createCanvasEngine({ + backend: sizedBackend(), + controlDetectionDeps: { makeDurable }, + ensureProjectOnServer: () => Promise.resolve(), + filterDeps: { runGraph: graph, uploadIntermediate: () => Promise.resolve({ imageName: 'filter-input.png' }) }, + imageResolver: () => Promise.resolve(new Blob()), + mutationPort: createCanvasProjectMutationPort(store, projectId), + projectId, + reportError: () => undefined, + }); + const operations = getCanvasOperations(engine); + const leaves = () => getDocumentLeaves(store.queries.getProject(projectId)!.canvas.document); + const layer = (id = 'c1') => leaves().find((leaf) => leaf.id === id) as CanvasControlLayerContract | undefined; + return { engine, graph, graphSignals, layer, leaves, makeDurable, operations }; +}; + +afterEach(() => { + vi.useRealTimers(); +}); + +describe('createCanvasEngine control detection', () => { + it('detects after the debounce, lands the map and undoes the intent with its map in one step', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + const { engine, graph, layer, makeDurable, operations } = setup(undefined, () => + Promise.resolve({ height: SIZE, imageName: 'edges-map.png', width: SIZE }) + ); + const before = layer(); + + expect(operations.applyControlLayerIntent('c1', { controlType: 'edges', kind: 'set-type' })).toBe('updated'); + expect(operations.getControlDetectionStatus('c1')).toEqual({ status: 'queued' }); + await vi.advanceTimersByTimeAsync(DEBOUNCE_MS - 1); + expect(graph).not.toHaveBeenCalled(); + await vi.advanceTimersByTimeAsync(1); + await vi.waitFor(() => expect(operations.getControlDetectionStatus('c1')).toEqual({ status: 'idle' })); + + expect(graph).toHaveBeenCalledOnce(); + expect(makeDurable).toHaveBeenCalledExactlyOnceWith('edges-map.png'); + expect(layer()).toMatchObject({ + controlType: 'edges', + detection: { type: 'canny_edge_detection' }, + filter: { type: 'canny_edge_detection' }, + original: before!.source, + source: { bitmap: { imageName: 'edges-map.png' }, type: 'paint' }, + }); + + expect(await engine.history.undo()).toBe('applied'); + expect(layer()).toEqual(before); + expect(engine.stores.canUndo.get()).toBe(false); + expect(graph).toHaveBeenCalledOnce(); + engine.lifecycle.dispose(); + }); + + it('suspends detection on cooldown and resumes it on activation', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + const { engine, graph, graphSignals, operations } = setup(); + operations.applyControlLayerIntent('c1', { controlType: 'edges', kind: 'set-type' }); + await vi.advanceTimersByTimeAsync(DEBOUNCE_MS); + + void engine.lifecycle.beginCooldown(); + expect(graphSignals[0]!.aborted).toBe(true); + expect(operations.getControlDetectionStatus('c1')).toEqual({ status: 'queued' }); + + engine.lifecycle.activate(); + await vi.advanceTimersByTimeAsync(0); + expect(graph).toHaveBeenCalledTimes(2); + engine.lifecycle.dispose(); + }); + + it('drops the status of a deleted layer', () => { + const { engine, operations } = setup(); + operations.applyControlLayerIntent('c1', { controlType: 'edges', kind: 'set-type' }); + + const remove = engine.document.model()!.prepare({ ids: ['c1'], type: 'remove' }); + if (remove.status !== 'prepared') { + throw new Error('expected a prepared removal'); + } + expect(engine.layers.commitPrepared('Delete', remove.edit).status).toBe('committed'); + expect(operations.getControlDetectionStatus('c1')).toEqual({ status: 'idle' }); + engine.lifecycle.dispose(); + }); + + it('aborts running detection on dispose, idempotently', async () => { + vi.useFakeTimers({ toFake: ['setTimeout', 'clearTimeout'] }); + const { engine, graphSignals, operations } = setup(); + const listener = vi.fn(); + operations.subscribeControlDetectionStatus('c1', listener); + operations.applyControlLayerIntent('c1', { controlType: 'edges', kind: 'set-type' }); + await vi.advanceTimersByTimeAsync(DEBOUNCE_MS); + expect(graphSignals).toHaveLength(1); + listener.mockClear(); + + engine.lifecycle.dispose(); + engine.lifecycle.dispose(); + + expect(graphSignals[0]!.aborted).toBe(true); + expect(operations.getControlDetectionStatus('c1')).toEqual({ status: 'idle' }); + expect(listener).toHaveBeenCalledOnce(); + expect(operations.applyControlLayerIntent('c1', { kind: 'draw' })).toBe('blocked'); + }); + + it('types a canny filter copy saved as control as an edges layer used as-is', async () => { + const { engine, leaves, operations } = setup([shapeRaster('r1')], () => + Promise.resolve({ height: SIZE, imageName: 'canny.png', width: SIZE }) + ); + expect((await engine.exports.exportLayerPixels('r1')).status).toBe('ok'); + expect(operations.startFilterOperation('r1')).toBe('started'); + operations.updateFilterOperation({ + settings: { high_threshold: 200, low_threshold: 100 }, + type: 'canny_edge_detection', + }); + await expect(operations.processFilterOperation()).resolves.toBe('completed'); + + await expect(operations.commitFilterOperation('control', () => Promise.resolve())).resolves.toBe('committed'); + + const copies = leaves().filter((leaf) => leaf.type === 'control'); + expect(copies).toHaveLength(1); + expect(copies[0]).toMatchObject({ controlType: 'edges', detection: null }); + expect(copies[0]).not.toHaveProperty('original'); + engine.lifecycle.dispose(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.ts index c92f73125a2..db293bed9c7 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/createCanvasEngine.ts @@ -6,12 +6,15 @@ import type { CanvasUtilityGraphResult } from '@workbench/canvas-operations/cont import type { GenerationCompositeHost } from '@workbench/canvas-operations/generationComposite'; import type { BackendGraphContract } from '@workbench/graphContracts'; +import { galleryDurability } from '@features/gallery'; import { assertAccountScopeCurrent, captureAccountScope } from '@platform/state/accountLifecycle'; import { createCanvasEngine as createCanvasEngineCore } from '@workbench/canvas-engine/engine'; import { canvasApplicationPort } from '@workbench/canvas-operations/applicationPort'; import { isUploadProjectNotFound } from '@workbench/canvas-operations/backend/canvasImages'; import { createBoundedCompositeDedupeCache } from '@workbench/canvas-operations/compositeForGeneration'; +import { createControlDetectionCoordinator } from '@workbench/canvas-operations/ControlDetectionCoordinator'; import { composeForGeneration } from '@workbench/canvas-operations/generationComposite'; +import { controlTypeForDetector } from '@workbench/controlTypes'; import type { CanvasOperationCapability, CanvasOperationImplementation } from './contracts'; @@ -35,6 +38,10 @@ export interface CanvasEngineOptions extends Omit< signal?: AbortSignal; }): Promise; }; + controlDetectionDeps?: { + /** Adopts a detected map as canvas-owned pixels; the gallery's canvas-asset transition by default. */ + makeDurable(imageName: string): Promise; + }; filterDeps?: { uploadIntermediate(blob: Blob, signal?: AbortSignal): Promise<{ imageName: string }>; runGraph(options: { @@ -63,7 +70,7 @@ export type CanvasEngine = CoreCanvasEngineImplementation; /** Application composition root: owns SAM/filter sessions, queues, uploads, and their core capability adapters. */ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine => { - const { ensureProjectOnServer, filterDeps, selectObjectDeps, ...coreOptions } = options; + const { controlDetectionDeps, ensureProjectOnServer, filterDeps, selectObjectDeps, ...coreOptions } = options; const owner = captureAccountScope(); const uploadLifetime = new AbortController(); let projectMissingAt = -Infinity; @@ -120,6 +127,14 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = edits: core.edits, isGuardCurrent: host.isGuardCurrent, }); + const runFilterGraph = async (runOptions: { + graph: BackendGraphContract; + outputNodeId?: string; + signal?: AbortSignal; + }): Promise => { + const output = await (filterDeps?.runGraph(runOptions) ?? canvasApplicationPort.runGraph(runOptions)); + return { height: output.height, imageName: output.imageName, width: output.width }; + }; let filterCoordinator!: ReturnType; const selectObjectCoordinator = canvasApplicationPort.createSelectObjectCoordinator({ @@ -203,6 +218,7 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = return Promise.resolve({ status: 'locked' }); } return host.commitFilter({ + ...(target === 'control' ? { controlCopy: { controlType: controlTypeForDetector(draft) } } : {}), filter: draft, guard, image, @@ -219,10 +235,7 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = host.publishFilterPreview(guard.layerId, imageName, rect, guard, filterType), }), isInteractionLocked: host.isInteractionLocked, - runFilterGraph: async (runOptions) => { - const output = await (filterDeps?.runGraph(runOptions) ?? canvasApplicationPort.runGraph(runOptions)); - return { height: output.height, imageName: output.imageName, width: output.width }; - }, + runFilterGraph, selectLayer: host.selectLayer, setViewTool: host.setViewTool, stores, @@ -233,6 +246,35 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = }, }); + const interactionLockListeners = new Set<() => void>(); + // Any of these may be what refused a detection commit; each release lets a waiting result retry. + const lockStores = [ + core.stores.documentEditingLocked, + core.stores.transformSession, + core.stores.textEditSession, + core.stores.documentEpoch, + ]; + const controlDetection = createControlDetectionCoordinator({ + derivation: core.controlDerivation, + flushLayerUploads: host.flushLayerUploads, + getDocument: host.getDocument, + makeDurable: controlDetectionDeps?.makeDurable ?? galleryDurability.makeCanvasAsset, + projectId: options.projectId, + runFilterGraph, + subscribeLockChanges: (listener) => { + interactionLockListeners.add(listener); + const unsubscribers = lockStores.map((store) => store.subscribe(listener)); + return () => { + interactionLockListeners.delete(listener); + for (const unsubscribe of unsubscribers) { + unsubscribe(); + } + }; + }, + }); + // Statuses follow the document (deleted layers, undo past a failure); it never schedules detection. + const unsubscribeControlReconcile = core.stores.documentEpoch.subscribe(controlDetection.reconcileWithDocument); + const syncEditingLock = (): void => { const operation = controller.getSnapshot(); core.stores.documentEditingLocked.set(operation.status === 'active'); @@ -281,12 +323,14 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = }; const operations: CanvasOperationImplementation = { + applyControlLayerIntent: controlDetection.applyIntent, applySelectObjectSession: selectObjectCoordinator.apply, cancelFilterOperation: filterCoordinator.cancel, cancelSelectObjectSession: selectObjectCoordinator.cancel, commitFilterOperation: filterCoordinator.commit, composeForGeneration: (composeOptions) => composeForGeneration(generationCompositeHost, composeOptions), controller, + getControlDetectionStatus: controlDetection.getStatus, getFilterSessionState: stores.filterSession.get, getOperationState: controller.getSnapshot, getSamSessionState: stores.samSession.get, @@ -294,11 +338,13 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = processSelectObjectSession: selectObjectCoordinator.process, resetFilterOperation: filterCoordinator.reset, resetSelectObjectSession: selectObjectCoordinator.reset, + retryControlDetection: controlDetection.retry, saveSelectObjectSession: selectObjectCoordinator.save, setFilterOperationAutoProcess: filterCoordinator.setAutoProcess, startFilterOperation: filterCoordinator.start, startSelectObject: selectObjectCoordinator.start, stores, + subscribeControlDetectionStatus: controlDetection.subscribeStatus, subscribeFilterSession: stores.filterSession.subscribe, subscribeOperation: controller.subscribe, subscribeSamSession: stores.samSession.subscribe, @@ -331,6 +377,9 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = if (!locked && selectObjectCoordinator.isActive()) { host.setSamTool(); } + for (const listener of interactionLockListeners) { + listener(); + } }, }; @@ -346,6 +395,8 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = unsubscribeToolChanges(); unsubscribeSamInteraction(); unsubscribeEditingLock(); + unsubscribeControlReconcile(); + controlDetection.dispose(); filterCoordinator.dispose(); selectObjectCoordinator.dispose(); controller.dispose(); @@ -355,9 +406,16 @@ export const createCanvasEngine = (options: CanvasEngineOptions): CanvasEngine = }; const lifecycle = { ...core.lifecycle, + activate: () => { + core.lifecycle.activate(); + if (!disposed) { + controlDetection.resume(); + } + }, beginCooldown: () => { filterCoordinator.cancel(); selectObjectCoordinator.cancel(); + controlDetection.suspend(); controller.invalidateDocument(options.projectId); return core.lifecycle.beginCooldown(); }, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts index b7d381aad1f..82fed8402bf 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.test.ts @@ -15,11 +15,10 @@ import { seedArchitectureCapabilities, } from '@features/generation/core/architectureCapabilities.testing'; import { accountLifecycle } from '@platform/state/accountLifecycle'; -import { getDocumentLeaves } from '@workbench/canvas-engine/api'; +import { createControlLayer, getDocumentLeaves } from '@workbench/canvas-engine/api'; import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; import { createTestInsertionAnchorCapture } from '@workbench/canvas-engine/document/insertionAnchors.testStub'; import { - createControlLayer, createEmptyPaintLayer, createInpaintMaskFromImage, createRegionalGuidanceFromImage, @@ -226,7 +225,7 @@ const expectedLayer = ( }; case 'control': return { - ...createControlLayer(`Control Layer ${index + 1}`, 'expected', base), + ...createControlLayer({ base, id: 'expected', name: `Control Layer ${index + 1}` }), id: expect.any(String) as string, source: { image: ref, type: 'image' }, transform: { rotation: 0, scaleX: 1, scaleY: 1, x: bbox.x, y: bbox.y }, @@ -313,13 +312,9 @@ describe('importGalleryImagesToCanvas', () => { }); }); - it('seeds new control layers with the default control model for the selected base', async () => { + it('imports control layers untyped and without a model, so generation resolves the model', async () => { const { project, state } = withProject((value) => setModel(value, 'sd-1')); const canvasEngine = engine(project.id); - const models = [ - { base: 'sd-1', key: 'sd1-canny', name: 'Canny', type: 'controlnet' }, - { base: 'sd-1', key: 'sd1-union', name: 'Union Pro', type: 'controlnet' }, - ] as unknown as Parameters[0]['models']; const result = await importGalleryImagesToCanvas({ destination: 'control', @@ -327,14 +322,16 @@ describe('importGalleryImagesToCanvas', () => { engine: canvasEngine, ...queriesFor(() => state), images: [image('a.png')], - models, project, }); expect(result.status).toBe('imported'); const [, forward] = vi.mocked(canvasEngine.layers.commitStructural).mock.calls[0]!; const layer = getForwardLayers(forward)[0]!; - expect(layer.type === 'control' ? layer.adapter.model : null).toBe('sd1-union'); + expect(layer).toMatchObject({ adapter: { kind: 'controlnet', model: null }, type: 'control' }); + expect(layer).not.toHaveProperty('controlType'); + expect(layer).not.toHaveProperty('detection'); + expect(layer).not.toHaveProperty('original'); }); it('fetches and uploads each resized control once with optimal model dimensions and durable hidden policy', async () => { @@ -390,13 +387,13 @@ describe('importGalleryImagesToCanvas', () => { const layers = getForwardLayers(forward); expect(layers).toEqual([ { - ...createControlLayer('Control Layer 1', 'expected', 'flux2'), + ...createControlLayer({ base: 'flux2', id: 'expected', name: 'Control Layer 1' }), id: expect.any(String), source: { image: { height: 768, imageName: 'resized-wide.png', width: 1360 }, type: 'image' }, transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 31, y: 47 }, }, { - ...createControlLayer('Control Layer 2', 'expected', 'flux2'), + ...createControlLayer({ base: 'flux2', id: 'expected', name: 'Control Layer 2' }), id: expect.any(String), source: { image: { height: 1024, imageName: 'resized-square.png', width: 1024 }, type: 'image' }, transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 31, y: 47 }, diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.ts index 4ac9fa5bc6f..2978d1a707a 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/importGalleryImages.ts @@ -1,5 +1,4 @@ import type { GalleryImage } from '@features/gallery'; -import type { ModelConfig } from '@features/models'; import type { CanvasDocumentCapability, CanvasLayerCapability, LayerStackKind } from '@workbench/canvas-engine/api'; import type { CanvasImageRef, CanvasLayerContract } from '@workbench/canvas-engine/contracts'; import type { CanvasProjectMutation } from '@workbench/canvasProjectMutations'; @@ -13,24 +12,25 @@ import { isArchitectureDescribed, normalizeGenerateWidgetValues, } from '@features/generation/settings'; -import { getModelsSnapshot } from '@features/models'; import { mapWithConcurrency } from '@platform/core/concurrency'; import { assertAccountScopeCurrent, captureAccountScope, registerAccountOwnedResource, } from '@platform/state/accountLifecycle'; -import { captureInsertionAnchor, getDocumentLeaves } from '@workbench/canvas-engine/api'; -import { resolveDefaultControlModelForBase } from '@workbench/widgets/layers/controlModelOptions'; import { + captureInsertionAnchor, createControlLayer, + getDocumentLeaves, + nextControlLayerName, +} from '@workbench/canvas-engine/api'; +import { createEmptyPaintLayer, createInpaintMaskFromImage, createLayerId, createRegionalGuidanceFromImage, createRegionalGuidanceLayerWithRefImage, DEFAULT_INPAINT_MASK_FILL, - nextControlLayerName, nextInpaintMaskName, nextRegionalGuidanceFillColor, nextRegionalGuidanceName, @@ -55,7 +55,6 @@ export type ImportGalleryImagesResult = interface BuildLayerContext { bbox: Project['canvas']['document']['bbox']; - defaultControlModel: string | null; existingLayers: readonly CanvasLayerContract[]; modelBase: string | null; } @@ -102,10 +101,12 @@ const buildLayer = ( case 'control': case 'control-resized': return withBboxOrigin( - { - ...createControlLayer(nextControlLayerName(names), id, context.modelBase, context.defaultControlModel), + createControlLayer({ + base: context.modelBase, + id, + name: nextControlLayerName(names), source: { image: ref, type: 'image' }, - }, + }), context.bbox ); case 'inpaint-mask': @@ -152,17 +153,14 @@ const buildLayer = ( const buildLayers = ( images: readonly LayerImage[], destination: GalleryCanvasImportDestination, - project: Project, - models: readonly ModelConfig[] + project: Project ): CanvasLayerContract[] => { const layers: CanvasLayerContract[] = []; const modelBase = getSelectedModelBase(project); - const defaultControlModel = resolveDefaultControlModelForBase(models, modelBase); for (const image of images) { layers.push( buildLayer(image, destination, { bbox: project.canvas.document.bbox, - defaultControlModel, existingLayers: [...getDocumentLeaves(project.canvas.document), ...layers], modelBase, }) @@ -229,7 +227,7 @@ const resizeImages = async ( }; }; -type GalleryImportEngine = { +export type GalleryImportEngine = { readonly projectId: string; readonly document: Pick; readonly layers: CanvasLayerCapability; @@ -244,8 +242,6 @@ export const importGalleryImagesToCanvas = async (options: { images: readonly GalleryImage[]; project: Project; fetchImage?: typeof fetch; - /** Model list used to pick a default control model; defaults to the loaded models snapshot. */ - models?: readonly ModelConfig[]; uploadImage?: typeof uploadCanvasImage; }): Promise => { const owner = captureAccountScope(); @@ -257,7 +253,6 @@ export const importGalleryImagesToCanvas = async (options: { getProject, images, isActiveProject, - models = getModelsSnapshot().models, project, uploadImage = uploadCanvasImage, } = options; @@ -298,7 +293,7 @@ export const importGalleryImagesToCanvas = async (options: { failedImageNames = resized.failedImageNames; } - const layers = buildLayers(layerImages, destination, project, models); + const layers = buildLayers(layerImages, destination, project); const previousSelectedLayerId = capturedDocument.selectedLayerId; const anchorFor = (stack: LayerStackKind) => matchingProjectEngine diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/operationTypes.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/operationTypes.ts index d31988fcbff..0ae42e57455 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/operationTypes.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/operationTypes.ts @@ -1,10 +1,12 @@ import type { LayerExportGuard } from '@workbench/canvas-engine/api'; import type { GuardedMutationRefusal } from '@workbench/canvas-engine/capabilities'; +import type { CanvasImageRef, ControlType } from '@workbench/canvas-engine/contracts'; import type { RasterFilterCommitTarget, RasterFilterSettings, } from '@workbench/canvas-engine/controllers/filterResultController'; import type { SamVisualInput } from '@workbench/canvas-engine/samInteraction'; +import type { LayerTransform } from '@workbench/canvas-engine/transform/transformMath'; import type { Rect, Vec2 } from '@workbench/canvas-engine/types'; export type { SamVisualInput } from '@workbench/canvas-engine/samInteraction'; @@ -99,3 +101,50 @@ export interface FilterOperationSessionState { status: 'ready' | 'processing' | 'committing' | 'error'; } export type FilterCommitTarget = RasterFilterCommitTarget; + +/** + * A control layer edit from the Properties panel. `set-type` may rename in the same history entry; `set-detector` + * starts from the detector's defaults with the type's fixed settings; `draw` empties the image for painting. + */ +export type ControlLayerIntent = + | { kind: 'set-type'; controlType: ControlType; name?: string } + | { kind: 'set-detector'; detector: string | null } + | { kind: 'set-detector-settings'; settings: Record } + | { kind: 'replace-image'; image: CanvasImageRef; transform?: LayerTransform } + | { kind: 'draw' }; +/** + * How an intent ended. `unchanged`: the layer already matched. `no-detection-input`: the layer's pixels are a map + * without an original, so a detector cannot run until the image is replaced. `missing`: no such control layer. + * `locked`: the layer or a group above it is locked. `blocked`: an operation, gesture or session holds the + * document; try again once it ends. `over-budget` and `failed`: the engine could not apply the edit. + */ +export type ControlLayerIntentResult = + | 'updated' + | 'unchanged' + | 'no-detection-input' + | 'missing' + | 'locked' + | 'blocked' + | 'over-budget' + | 'failed'; +/** `not-needed`: the map is current or there is nothing to detect from. */ +export type RetryControlDetectionResult = 'scheduled' | 'not-needed' | 'missing'; +/** Typed detection failures; the view translates the code and shows `detail` as technical detail. */ +export type ControlDetectionErrorCode = + | 'process-failed' + | 'apply-failed' + | 'unknown-detector' + | 'output-dimensions' + | 'source-not-ready' + | 'over-budget' + | 'commit-busy' + | 'unrepresentable'; +export interface ControlDetectionError { + code: ControlDetectionErrorCode; + detail?: string; +} +export type ControlDetectionStatus = + | { status: 'idle' } + | { status: 'queued' } + | { status: 'processing'; detector: string } + | { status: 'error'; error: ControlDetectionError }; diff --git a/invokeai/frontend/webv2/src/workbench/canvas-operations/useCanvasEngine.ts b/invokeai/frontend/webv2/src/workbench/canvas-operations/useCanvasEngine.ts index 1d82499ae70..198e5cf85ed 100644 --- a/invokeai/frontend/webv2/src/workbench/canvas-operations/useCanvasEngine.ts +++ b/invokeai/frontend/webv2/src/workbench/canvas-operations/useCanvasEngine.ts @@ -2,10 +2,8 @@ import type { CanvasEngine, ImageResolver } from '@workbench/canvas-engine/api'; import { useFontRuntime } from '@features/fonts/react'; import { galleryImageUrls } from '@features/gallery/utility'; -import { getModelsSnapshot } from '@features/models'; import { createCanvasProjectMutationPort } from '@workbench/canvasProjectMutationPort'; import { publishLayerPanelSelection, readLayerPanelState } from '@workbench/layerPanelState'; -import { resolveDefaultControlModelForBase } from '@workbench/widgets/layers/controlModelOptions'; import { getSelectedModelBase } from '@workbench/widgets/layers/selectedModel'; import { useActiveProjectId, @@ -86,7 +84,6 @@ export const useCanvasEngine = (): CanvasEngineHandle | null => { } }, liveEngines, - getDefaultControlModel: (base) => resolveDefaultControlModelForBase(getModelsSnapshot().models, base), getMainModelBase: () => { const project = store.getState().projects.find((candidate) => candidate.id === projectId); return project ? getSelectedModelBase(project) : null; diff --git a/invokeai/frontend/webv2/src/workbench/canvasLayerContent.test.ts b/invokeai/frontend/webv2/src/workbench/canvasLayerContent.test.ts index a58e9ca3ba0..11e5a0e6a41 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasLayerContent.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasLayerContent.test.ts @@ -7,6 +7,7 @@ import type { import { describe, expect, it } from 'vitest'; import { + getControlImageState, hasControlLayerContent, hasRegionalGuidanceMaskContent, isCompositableControlLayer, @@ -89,3 +90,15 @@ describe('isCompositableRegionalGuidanceLayer', () => { expect(isCompositableRegionalGuidanceLayer(controlLayer('not-regional') as CanvasLayerContract)).toBe(false); }); }); + +describe('getControlImageState', () => { + const photo = { image: { height: 64, imageName: 'photo.png', width: 96 }, type: 'image' } as const; + const emptyPaint = { bitmap: null, type: 'paint' } as const; + + it('reads the original before the map, and live strokes the document has not saved yet', () => { + const detectedMap = { ...controlLayer('detected', { source: emptyPaint }), original: photo }; + expect(getControlImageState(detectedMap, true)).toBe('image'); + expect(getControlImageState(controlLayer('drawn', { source: emptyPaint }), true)).toBe('drawn'); + expect(getControlImageState(controlLayer('empty', { source: emptyPaint }), false)).toBe('empty'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvasLayerContent.ts b/invokeai/frontend/webv2/src/workbench/canvasLayerContent.ts index a6e7e2bede5..1406b4deb18 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasLayerContent.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasLayerContent.ts @@ -5,8 +5,8 @@ import type { } from '@workbench/canvas-engine/api'; /** - * Shared pixel-content predicates for composite planning and invocation readiness; depends only on layer - * contracts. + * Shared pixel-content predicates for composite planning, invocation readiness and control-image state; depends only + * on layer contracts. */ /** True when a control layer holds an image source or committed paint pixels. */ @@ -17,6 +17,28 @@ export const hasControlLayerContent = (layer: CanvasControlLayerContract): boole return layer.source.type === 'paint' && layer.source.bitmap !== null; }; +/** `drawn`: paint (or another non-image source) with pixels; `empty`: nothing to detect or use yet. */ +export type ControlImageState = 'empty' | 'image' | 'drawn'; + +/** + * What a control layer holds to detect from or use as-is. `sourceHasPixels` is the engine's answer for strokes the + * document has not persisted yet. + */ +export const getControlImageState = ( + layer: Pick, + sourceHasPixels: boolean +): ControlImageState => { + const input = layer.original ?? layer.source; + if (input.type === 'image') { + return 'image'; + } + if (input.type === 'paint' && input.bitmap !== null) { + return 'drawn'; + } + // Unpersisted strokes (or a parametric source) only the engine's cache knows about. + return !layer.original && sourceHasPixels ? 'drawn' : 'empty'; +}; + /** True when a regional-guidance layer holds a persisted (non-empty) mask. */ export const hasRegionalGuidanceMaskContent = (layer: CanvasRegionalGuidanceLayerContract): boolean => layer.mask.bitmap !== null; diff --git a/invokeai/frontend/webv2/src/workbench/canvasMigration.test.ts b/invokeai/frontend/webv2/src/workbench/canvasMigration.test.ts index 9e271c9b5fc..a31bdbf1491 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasMigration.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasMigration.test.ts @@ -1,5 +1,6 @@ import type { CanvasStateContractV3 } from '@workbench/canvas-engine/api'; +import { createControlLayer } from '@workbench/canvas-engine/api'; import { groupContract, stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; import { describe, expect, it } from 'vitest'; @@ -14,10 +15,10 @@ import { loadCanvasState, normalizeCanvasDocumentContract, placementToTransform, + writableCanvasSchemaVersion, } from './canvasMigration'; import { MAX_SUPPORTED_CANVAS_SCHEMA_VERSION } from './canvasSchemaVersion'; import { - createControlLayer, createEmptyPaintLayer, createInpaintMaskLayer, createRegionalGuidanceLayer, @@ -81,7 +82,7 @@ describe('loadCanvasState', () => { { beginEndStepPct: [0.1, 0.9], controlMode: null, kind: 'flux2_control', model: 'flux2-union', weight: 0 }, ] as const; const layers = adapters.map((adapter, index) => ({ - ...createControlLayer(`Control ${index}`, `control-${index}`), + ...createControlLayer({ id: `control-${index}`, name: `Control ${index}` }), adapter, })); @@ -97,7 +98,7 @@ describe('loadCanvasState', () => { // unsupported. Load cannot tell an LLLite key from a ControlNet key without the catalog, so it keeps the kind; // the settings panel reports it and switching to LLLite keeps the model. const saved = { - ...createControlLayer('Anima Control', 'anima-control'), + ...createControlLayer({ id: 'anima-control', name: 'Anima Control' }), adapter: { beginEndStepPct: [0, 0.75], controlMode: 'balanced', @@ -114,11 +115,11 @@ describe('loadCanvasState', () => { it('refuses a control layer whose adapter kind this build does not know, naming the layer', () => { const future = { - ...createControlLayer('Future Control', 'future-control'), + ...createControlLayer({ id: 'future-control', name: 'Future Control' }), adapter: { beginEndStepPct: [0, 1], controlMode: null, kind: 'future_adapter', model: 'm', weight: 1 }, }; - const refused = refusal(withNodes([createControlLayer('Kept', 'kept'), future], 'control')); + const refused = refusal(withNodes([createControlLayer({ id: 'kept', name: 'Kept' }), future], 'control')); expect(refused.status).toBe('invalid'); expect(refused.status === 'invalid' ? refused.diagnostics : []).toEqual([ @@ -129,7 +130,12 @@ describe('loadCanvasState', () => { it('normalizes an incomplete persisted Z-Image control with backend defaults', () => { const loaded = load( withNodes( - [{ ...createControlLayer('Z Control', 'z-control'), adapter: { kind: 'z_image_control', model: 'z-control' } }], + [ + { + ...createControlLayer({ id: 'z-control', name: 'Z Control' }), + adapter: { kind: 'z_image_control', model: 'z-control' }, + }, + ], 'control' ) ); @@ -144,13 +150,40 @@ describe('loadCanvasState', () => { }); }); - it('repairs a persisted Qwen-Image control weight below the node floor with the kind default', () => { + it('keeps a control layer mode and negative weight whatever kind it stores, live, in snapshots and in session', () => { + // Created under Z-Image, then run on SDXL with a negative weight: the stored kind is advisory, the resolved one + // governs, so the document must load and save as it is. + const adapter = { + beginEndStepPct: [0.1, 0.9], + controlMode: 'more_control', + kind: 'z_image_control', + model: null, + weight: -0.3, + }; + const state = withNodes([{ ...createControlLayer({ id: 'z-control', name: 'Z Control' }), adapter }], 'control'); + const loaded = load({ + ...state, + snapshots: [{ createdAt: 'now', document: state.document, id: 'snapshot', name: 'Snapshot' }], + }); + const adapterOf = (document: (typeof loaded)['document']) => { + const layer = document.stacks.control[0]; + return layer?.type === 'control' ? layer.adapter : null; + }; + + expect(adapterOf(loaded.document)).toEqual(adapter); + expect(adapterOf(loaded.snapshots[0]!.document)).toEqual(adapter); + // Snapshot save and restore and document replace normalize in session; a refusal there would silently no-op. + const normalized = normalizeCanvasDocumentContract(loaded.document); + expect(normalized && adapterOf(normalized)).toEqual(adapter); + }); + + it('repairs a persisted control weight outside what any kind runs with the kind default', () => { const loaded = load( withNodes( [ { - ...createControlLayer('Q Control', 'q-control'), - adapter: { kind: 'qwen_image_control', model: 'qwen-union', weight: -0.5 }, + ...createControlLayer({ id: 'q-control', name: 'Q Control' }), + adapter: { kind: 'qwen_image_control', model: 'qwen-union', weight: -1.5 }, }, ], 'control' @@ -443,8 +476,8 @@ describe('loadCanvasState', () => { it('normalizes control adapters in both the live document and saved snapshots', () => { const invalidLayer = { - ...createControlLayer('Z Control', 'z-control'), - adapter: { beginEndStepPct: [0.8, 0.2], controlMode: null, kind: 'z_image_control', model: null, weight: -1 }, + ...createControlLayer({ id: 'z-control', name: 'Z Control' }), + adapter: { beginEndStepPct: [0.8, 0.2], controlMode: null, kind: 'z_image_control', model: null, weight: -1.5 }, }; const state = withNodes([invalidLayer], 'control'); const loaded = load({ @@ -464,7 +497,7 @@ describe('loadCanvasState', () => { const state = createEmptyCanvasState(); const validDocument = { ...state.document, - stacks: stacksFrom([createControlLayer('Snapshot Control', 'snapshot-control')]), + stacks: stacksFrom([createControlLayer({ id: 'snapshot-control', name: 'Snapshot Control' })]), }; const valid = { createdAt: 'now', document: validDocument, id: 'valid', name: 'Valid' }; const base = withNodes([createEmptyPaintLayer('Live', 'live')]); @@ -493,7 +526,7 @@ describe('loadCanvasState', () => { ['missing discriminant and base fields', { id: 'broken' }], ['unknown layer type', { ...createEmptyPaintLayer('Mystery', 'mystery'), type: 'mystery' }], ['raster with invalid source', { ...createEmptyPaintLayer('Raster', 'raster'), source: null }], - ['a leaf of another stack', createControlLayer('Control', 'control')], + ['a leaf of another stack', createControlLayer({ id: 'control', name: 'Control' })], ['a group without children', { ...groupContract('g'), children: null }], ['a group with a malformed child', groupContract('g', [{ ...createEmptyPaintLayer('Bad', 'bad'), opacity: 4 }])], ])('refuses a snapshot containing a malformed node: %s', (_label, malformedNode) => { @@ -521,7 +554,7 @@ describe('loadCanvasState', () => { it('round-trips valid snapshots containing every layer type and nested groups', () => { const state = createEmptyCanvasState(); const stacks = { - control: [groupContract('cg', [createControlLayer('Control', 'control')], { isHidden: true })], + control: [groupContract('cg', [createControlLayer({ id: 'control', name: 'Control' })], { isHidden: true })], inpaint_mask: [createInpaintMaskLayer('Mask', 'mask')], raster: [groupContract('rg', [createEmptyPaintLayer('Raster', 'raster'), groupContract('inner', [])])], regional_guidance: [createRegionalGuidanceLayer('Region', 0, 'region')], @@ -561,7 +594,7 @@ describe('loadCanvasState', () => { it.each([ [ 'a duplicated id across stacks', - { control: [createControlLayer('C', 'dup')], raster: [createEmptyPaintLayer('R', 'dup')] }, + { control: [createControlLayer({ id: 'dup', name: 'C' })], raster: [createEmptyPaintLayer('R', 'dup')] }, 'document.stacks.control[0].id', ], [ @@ -574,7 +607,11 @@ describe('loadCanvasState', () => { { raster: [groupContract('g', [], { isHidden: true })] }, 'document.stacks.raster[0].isHidden', ], - ['a leaf in the wrong stack', { raster: [createControlLayer('C', 'c')] }, 'document.stacks.raster[0]'], + [ + 'a leaf in the wrong stack', + { raster: [createControlLayer({ id: 'c', name: 'C' })] }, + 'document.stacks.raster[0]', + ], [ 'a group nested past the depth limit', { @@ -871,6 +908,151 @@ describe('loadCanvasState', () => { expect(refusal(legacy)).toMatchObject({ scope: 'document', status: 'invalid' }); }); + describe('control layer derivation fields', () => { + const image = (imageName: string) => ({ height: 64, imageName, width: 64 }); + const detection = { settings: { high_threshold: 200, low_threshold: 100 }, type: 'canny_edge_detection' }; + const typed = { + ...createControlLayer({ id: 'edges', name: 'Edges 1' }), + controlType: 'edges', + detection, + filter: detection, + original: { image: image('photo.png'), type: 'image' }, + originalTransform: { rotation: 0, scaleX: 2, scaleY: 2, x: 10, y: 20 }, + source: { bitmap: image('map.png'), offset: { x: 0, y: 0 }, type: 'paint' }, + }; + const legacy = { + ...createControlLayer({ id: 'legacy', name: 'Control Layer 1' }), + filter: detection, + source: { bitmap: image('map.png'), type: 'paint' }, + }; + const omit = (layer: Record, ...keys: string[]) => + Object.fromEntries(Object.entries(layer).filter(([key]) => !keys.includes(key))); + const declaring = (canvas: ReturnType, version: number) => ({ + ...canvas, + document: { ...canvas.document, version }, + version, + }); + /** Loads one control leaf, asserting no diagnostics and returning it with its exact key set. */ + const loadControl = (layer: unknown) => { + const result = loadCanvasState(withNodes([layer], 'control')); + expect(result).toMatchObject({ diagnostics: [], status: 'loaded' }); + return result.status === 'loaded' ? result.value.document.stacks.control[0] : undefined; + }; + const expectExactly = (actual: unknown, expected: Record) => { + expect(actual).toEqual(expected); + expect(Object.keys(actual as object).sort()).toEqual(Object.keys(expected).sort()); + }; + + it('round-trips a typed, detected layer and writes a v3, v4 or v5 canvas as version 6', () => { + for (const declared of [3, 4, 5]) { + const loaded = load(declaring(withNodes([typed], 'control'), declared)); + expectExactly(loaded.document.stacks.control[0], typed); + expect(loaded.version).toBe(6); + expect(loaded.document.version).toBe(6); + } + expect( + load(withNodes([groupContract('outer', [groupContract('inner', [typed as never])])], 'control')).version + ).toBe(6); + }); + + it.each([ + ['controlType', { controlType: 'edges' }], + ['detection', { detection: null }], + ['original', { original: { image: image('photo.png'), type: 'image' } }], + ['originalTransform', { originalTransform: { rotation: 0, scaleX: 2, scaleY: 2, x: 0, y: 0 } }], + ])('requires version 6 for a control leaf carrying only %s', (_key, fields) => { + const stacks = withNodes([{ ...legacy, ...fields }], 'control').document.stacks; + expect(writableCanvasSchemaVersion([3], { stacks })).toBe(6); + }); + + it('requires version 6, not 5, when a group adjustment and a typed child share a document', () => { + const spatial = { amount: 40, id: 'clarity', isEnabled: true, size: 40, type: 'clarity' }; + const graded = groupContract('graded', [typed as never], { adjustments: [spatial as never] }); + expect(load(withNodes([graded], 'control')).version).toBe(6); + + const adjusted = withNodes([{ ...createEmptyPaintLayer('Detail', 'detail'), adjustments: [spatial] }]); + const stacks = { ...adjusted.document.stacks, control: [typed] }; + expect(load({ ...adjusted, document: { ...adjusted.document, stacks } }).version).toBe(6); + }); + + it('keeps a legacy control layer unchanged and at its declared version', () => { + const loaded = load(withNodes([legacy], 'control')); + expectExactly(loaded.document.stacks.control[0], legacy); + expect(loaded.version).toBe(3); + expect(loaded.document.version).toBe(3); + }); + + it('does not adopt version 6 for derivation keys present only as undefined', () => { + const undefinedKeys = { ...legacy, controlType: undefined, detection: undefined, original: undefined }; + expect(writableCanvasSchemaVersion([3], { stacks: withNodes([undefinedKeys], 'control').document.stacks })).toBe( + 3 + ); + expect(load(withNodes([undefinedKeys], 'control')).version).toBe(3); + }); + + it('never lowers a declared version 6 once the fields are gone', () => { + const loaded = load(declaring(withNodes([legacy], 'control'), 6)); + expect(loaded.version).toBe(6); + expect(loaded.document.version).toBe(6); + }); + + it('writes version 6 when only a snapshot holds a typed layer, leaving the live document at its own version', () => { + const loaded = load({ + ...createEmptyCanvasState(), + snapshots: [{ createdAt: 'now', document: withNodes([typed], 'control').document, id: 's', name: 'S' }], + }); + expect(loaded.version).toBe(6); + expect(loaded.document.version).toBe(3); + expect(loaded.snapshots[0]?.document.version).toBe(6); + }); + + it('keeps an unknown detector type and its settings', () => { + const future = { settings: { strength: 3 }, type: 'future_detector' }; + expectExactly(loadControl({ ...typed, detection: future, filter: future }), { + ...typed, + detection: future, + filter: future, + }); + }); + + it('refuses an unknown control type rather than rewriting it', () => { + expect(refusal(withNodes([{ ...typed, controlType: 'future_type' }], 'control'))).toMatchObject({ + scope: 'document', + status: 'invalid', + }); + }); + + const untyped = omit(typed, 'controlType', 'detection', 'original', 'originalTransform'); + const detached = { ...omit(typed, 'original', 'originalTransform'), detection: null }; + it.each([ + ['a null type (dropped, with everything it carried)', { ...typed, controlType: null }, untyped], + ['derivation fields without a type', omit(typed, 'controlType'), untyped], + [ + 'an original paint source without pixels (detached)', + { ...typed, original: { bitmap: null, type: 'paint' } }, + detached, + ], + [ + 'an original that is not a pixel source (detached)', + { ...typed, original: { content: 'x', type: 'text' } }, + detached, + ], + ['an original beside a "None" detection', { ...typed, detection: null }, detached], + [ + 'an original without a detection', + omit(typed, 'detection'), + omit(typed, 'detection', 'original', 'originalTransform'), + ], + [ + 'an original transform without an original', + omit(typed, 'original'), + omit(typed, 'original', 'originalTransform'), + ], + ])('silently repairs %s', (_label, input, expected) => { + expectExactly(loadControl(input), expected); + }); + }); + it('passes a valid state through normalized, keeping every persisted field', () => { const canvas = { ...withNodes([ diff --git a/invokeai/frontend/webv2/src/workbench/canvasMigration.ts b/invokeai/frontend/webv2/src/workbench/canvasMigration.ts index fa555671db8..96a0ee5b8ab 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasMigration.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasMigration.ts @@ -12,16 +12,20 @@ import type { LayerStackKind, } from '@workbench/canvas-engine/api'; -import { getControlAdapterMinWeight } from '@features/generation/graph'; import { createItemId } from '@platform/browser/randomUuid'; -import { CANVAS_MAX_NODE_COUNT, CANVAS_MAX_NODE_DEPTH, LAYER_STACK_ORDER } from '@workbench/canvas-engine/api'; +import { + CANVAS_MAX_NODE_COUNT, + CANVAS_MAX_NODE_DEPTH, + CONTROL_TYPES, + LAYER_STACK_ORDER, +} from '@workbench/canvas-engine/api'; import { repairSelectedLayerId } from '@workbench/canvas-engine/document/selectionRepair'; import { z } from 'zod'; import type { CanvasLoadDiagnostic, CanvasLoadResult, CanvasVersionScope } from './canvasLoadContracts'; import { MAX_SUPPORTED_CANVAS_SCHEMA_VERSION, MIN_SUPPORTED_CANVAS_SCHEMA_VERSION } from './canvasSchemaVersion'; -import { normalizeControlAdapter } from './controlAdapters'; +import { isStorableControlStepRange, isStorableControlWeight, normalizeControlAdapter } from './controlAdapters'; export const DEFAULT_CANVAS_DOCUMENT_WIDTH = 1024; export const DEFAULT_CANVAS_DOCUMENT_HEIGHT = 1024; @@ -42,6 +46,8 @@ const zPaintSource = z.object({ offset: zCoordinate.optional(), type: z.literal('paint'), }); +const zImageSource = z.object({ image: zImageRef, type: z.literal('image') }); +const zPixelSource = z.discriminatedUnion('type', [zPaintSource, zImageSource]); const zTextFontRef = z.object({ contentHash: z.string().regex(/^[a-f0-9]{64}$/u), family: z.string(), @@ -50,7 +56,7 @@ const zTextFontRef = z.object({ }); const zLayerSource = z.discriminatedUnion('type', [ zPaintSource, - z.object({ image: zImageRef, type: z.literal('image') }), + zImageSource, z.object({ align: z.enum(['left', 'center', 'right']), color: z.string(), @@ -156,11 +162,10 @@ const zControlAdapter = z model: z.string().nullable(), weight: zFiniteNumber, }) - .refine(({ beginEndStepPct }) => { - const [begin, end] = beginEndStepPct; - return begin >= 0 && end <= 1 && begin < end; - }) - .refine(({ kind, weight }) => weight >= getControlAdapterMinWeight(kind) && weight <= 2); + // The same bounds for every kind: the stored kind is advisory (the resolved kind governs in the panel, readiness + // validation and the graph), and a refusal here would reject the whole document. + .refine(({ beginEndStepPct }) => isStorableControlStepRange(beginEndStepPct)) + .refine(({ weight }) => isStorableControlWeight(weight)); const zMaskFill = z.object({ color: z.string(), style: z.enum(['solid', 'grid', 'crosshatch', 'diagonal', 'horizontal', 'vertical']), @@ -255,7 +260,12 @@ const zCanvasLayer = z.discriminatedUnion('type', [ }), zLayerBase.extend({ adapter: zControlAdapter, + // An unknown type refuses the document like an unknown adapter kind; detector strings stay open. + controlType: z.enum(CONTROL_TYPES).optional(), + detection: zFilter.nullable().optional(), filter: zFilter.optional(), + original: zPixelSource.optional(), + originalTransform: zTransform.optional(), // Missing display-only visibility means visible for older documents; only overlay types carry it. isHidden: z.boolean().optional(), source: zLayerSource, @@ -404,21 +414,37 @@ const hasV5Adjustment = (node: Record): boolean => Array.isArray(node.adjustments) && node.adjustments.some((entry) => isRecord(entry) && V5_ADJUSTMENT_TYPES.has(entry.type)); -/** v4: custom-font typography. v5: a noise or spatial adjustment, enabled or not, which a v4 reader would drop. */ -const nodeSchemaRequirement = (node: unknown): CanvasSchemaVersion => { - if (!isRecord(node)) { - return 3; +/** The control derivation fields a v5 reader would strip on load, losing the kept original on its next save. */ +const CONTROL_DERIVATION_KEYS = ['controlType', 'detection', 'original', 'originalTransform'] as const; + +const controlLayerRequiresV6 = (node: Record): boolean => + node.type === 'control' && CONTROL_DERIVATION_KEYS.some((key) => node[key] !== undefined); + +const ownSchemaRequirement = (node: Record): CanvasSchemaVersion => { + if (controlLayerRequiresV6(node)) { + return 6; } if (hasV5Adjustment(node)) { return 5; } - let required: CanvasSchemaVersion = textSourceRequiresV4(node.source) ? 4 : 3; + return textSourceRequiresV4(node.source) ? 4 : 3; +}; + +/** + * The newest schema a node or any descendant needs. v4: custom-font typography. v5: a noise or spatial adjustment, + * enabled or not, which a v4 reader would drop. v6: a control layer carrying any derivation field. + */ +const nodeSchemaRequirement = (node: unknown): CanvasSchemaVersion => { + if (!isRecord(node)) { + return 3; + } + let required = ownSchemaRequirement(node); if (Array.isArray(node.children)) { for (const child of node.children) { - required = Math.max(required, nodeSchemaRequirement(child)) as CanvasSchemaVersion; if (required === MAX_SUPPORTED_CANVAS_SCHEMA_VERSION) { break; } + required = Math.max(required, nodeSchemaRequirement(child)) as CanvasSchemaVersion; } } return required; @@ -498,15 +524,7 @@ type LoadStep = Extract, { status: 'loaded' }> | Refusal; type DeclaredVersion = { kind: 'current' } | { kind: 'other'; version: number } | { kind: 'malformed' }; const classifyVersion = (value: unknown): DeclaredVersion => { - if (value === MAX_SUPPORTED_CANVAS_SCHEMA_VERSION) { - return { kind: 'current' }; - } - if ( - typeof value === 'number' && - Number.isInteger(value) && - value >= MIN_SUPPORTED_CANVAS_SCHEMA_VERSION && - value < MAX_SUPPORTED_CANVAS_SCHEMA_VERSION - ) { + if (isSupportedCanvasSchemaVersion(value)) { return { kind: 'current' }; } if (typeof value === 'number' && Number.isInteger(value) && value >= 1) { @@ -572,15 +590,47 @@ interface ParseContext { count: number; } +const isPixelSourceWithPixels = (source: unknown): boolean => + isRecord(source) && (source.type === 'image' || (source.type === 'paint' && isRecord(source.bitmap))); + +/** + * Restores the control derivation invariants (see `CanvasControlLayerContract`) on a fresh leaf copy instead of + * refusing the document: each repair keeps the pixels generation uses and drops only bookkeeping that no longer holds. + */ +const repairControlDerivation = (layer: Record): Record => { + if (layer.controlType === null) { + delete layer.controlType; + } + if (layer.controlType === undefined) { + delete layer.detection; + delete layer.original; + delete layer.originalTransform; + return layer; + } + if (layer.original !== undefined && !isPixelSourceWithPixels(layer.original)) { + delete layer.original; + if (layer.detection !== undefined) { + layer.detection = null; + } + } + if (layer.original !== undefined && !isRecord(layer.detection)) { + delete layer.original; + } + if (layer.original === undefined) { + delete layer.originalTransform; + } + return layer; +}; + /** Fills the known optional defaults on a leaf, then validates it strictly. */ const parseLeaf = (value: Record, path: string, context: ParseContext): CanvasLayerContract | null => { let candidate: Record = value; if (value.type === 'control') { - candidate = { + candidate = repairControlDerivation({ ...value, adapter: normalizeControlAdapter(value.adapter), withTransparencyEffect: value.withTransparencyEffect === undefined ? true : value.withTransparencyEffect, - }; + }); } else if (value.type === 'regional_guidance') { candidate = { ...value, diff --git a/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.test.ts b/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.test.ts index 029f3d2bf2d..b6f7c234d1d 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.test.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest'; -import { getProjectCanvasSchemaRequirement } from './canvasSchemaVersion'; +import { getProjectCanvasSchemaRequirement, isCanvasSchemaVersionSupported } from './canvasSchemaVersion'; describe('getProjectCanvasSchemaRequirement', () => { it('uses only the live canvas schema version', () => { @@ -19,3 +19,9 @@ describe('getProjectCanvasSchemaRequirement', () => { expect(() => getProjectCanvasSchemaRequirement(document)).toThrow(); }); }); + +describe('isCanvasSchemaVersionSupported', () => { + it('opens projects needing schema 3 through 6, including control-layer derivation, and refuses the rest', () => { + expect([2, 3, 4, 5, 6, 7].map(isCanvasSchemaVersionSupported)).toEqual([false, true, true, true, true, false]); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.ts b/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.ts index b40b2068e9a..c9017c6fdb9 100644 --- a/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.ts +++ b/invokeai/frontend/webv2/src/workbench/canvasSchemaVersion.ts @@ -4,9 +4,10 @@ export const MIN_SUPPORTED_CANVAS_SCHEMA_VERSION = 3; /** * Newest canvas document schema this build can safely read, edit, and write. A document adopts a newer schema only * when it carries something older builds would drop on load and then lose on save: 4 for custom-font typography, 5 - * for noise, clarity, texture or dehaze adjustments. Older builds refuse the newer schema instead. + * for noise, clarity, texture or dehaze adjustments, 6 for control-layer derivation fields (`controlType`, + * `detection`, `original`, `originalTransform`). Older builds refuse the newer schema instead. */ -export const MAX_SUPPORTED_CANVAS_SCHEMA_VERSION = 5; +export const MAX_SUPPORTED_CANVAS_SCHEMA_VERSION = 6; /** Whether a project whose documents need at least `minimumCanvasSchemaVersion` can be opened here. */ export const isCanvasSchemaVersionSupported = (minimumCanvasSchemaVersion: number): boolean => diff --git a/invokeai/frontend/webv2/src/workbench/controlAdapters.test.ts b/invokeai/frontend/webv2/src/workbench/controlAdapters.test.ts index 8a907d59662..8d369babc98 100644 --- a/invokeai/frontend/webv2/src/workbench/controlAdapters.test.ts +++ b/invokeai/frontend/webv2/src/workbench/controlAdapters.test.ts @@ -3,6 +3,7 @@ import { describe, expect, it } from 'vitest'; import { CONTROL_ADAPTER_DEFAULTS, createDefaultControlAdapter, + deriveControlKind, getDefaultControlAdapterKind, normalizeControlAdapter, } from './controlAdapters'; @@ -21,43 +22,53 @@ describe('normalizeControlAdapter', () => { expect(normalizeControlAdapter(adapter)).toEqual(adapter); }); + it.each([null, 'z-image-union'])( + 'keeps the mode and a negative weight whatever kind is stored (model %s): the resolved kind governs', + (model) => { + // Created under Z-Image, now resolved to an SDXL ControlNet; a stored pick may belong to the other base. + const adapter = { + beginEndStepPct: [0.1, 0.8], + controlMode: 'more_control', + kind: 'z_image_control', + model, + weight: -0.5, + }; + expect(normalizeControlAdapter(adapter)).toEqual(adapter); + } + ); + it.each([ ['controlnet', Number.NaN], ['controlnet', Number.POSITIVE_INFINITY], ['controlnet', -1.01], ['controlnet', 2.01], - ['z_image_control', -0.01], + ['z_image_control', -1.5], ['z_image_control', 2.01], ['anima_lllite', 2.01], - ['qwen_image_control', -0.01], - ['qwen_image_control', 2.01], - ['flux2_control', -0.01], + ['qwen_image_control', -1.01], ['flux2_control', 2.01], - ] as const)('repairs invalid %s weight %s with its default', (kind, weight) => { - const adapter = { ...CONTROL_ADAPTER_DEFAULTS[kind], weight }; + ] as const)('repairs a %s weight of %s, outside what any kind runs, with its default', (kind, weight) => { + const adapter = { ...CONTROL_ADAPTER_DEFAULTS[kind], model: 'm', weight }; expect(normalizeControlAdapter(adapter)).toMatchObject({ kind, weight: CONTROL_ADAPTER_DEFAULTS[kind].weight }); }); - it.each(['z_image_control', 'qwen_image_control', 'flux2_control'] as const)( - 'rebuilds a %s adapter from the contract fields', - (kind) => { - const persisted = { - beginEndStepPct: [0.1, 0.9], - controlMode: 'balanced', - extra: true, - kind, - model: 'm', - weight: 1.5, - }; - expect(normalizeControlAdapter(persisted)).toEqual({ - beginEndStepPct: [0.1, 0.9], - controlMode: null, - kind, - model: 'm', - weight: 1.5, - }); - } - ); + it('maps the contract fields only, and repairs an unknown mode with the kind default', () => { + const persisted = { + beginEndStepPct: [0.1, 0.9], + controlMode: 'sideways', + extra: true, + kind: 'controlnet', + model: 7, + weight: 1.5, + }; + expect(normalizeControlAdapter(persisted)).toEqual({ + beginEndStepPct: [0.1, 0.9], + controlMode: 'balanced', + kind: 'controlnet', + model: null, + weight: 1.5, + }); + }); it.each([ [Number.NaN, 1], @@ -108,3 +119,20 @@ describe('default control adapters', () => { expect(CONTROL_ADAPTER_DEFAULTS.anima_lllite.beginEndStepPct).toEqual([0, 1]); }); }); + +describe('deriveControlKind', () => { + it.each([ + [{ base: 'sdxl', type: 'controlnet' }, 'controlnet'], + [{ base: 'flux', type: 'controlnet' }, 'controlnet'], + [{ base: 'sdxl', type: 't2i_adapter' }, 't2i_adapter'], + [{ base: 'flux', type: 'control_lora' }, 'control_lora'], + [{ base: 'z-image', type: 'controlnet' }, 'z_image_control'], + [{ base: 'qwen-image', type: 'controlnet' }, 'qwen_image_control'], + [{ base: 'flux2', type: 'controlnet' }, 'flux2_control'], + [{ base: 'anima', type: 'controlnet' }, 'anima_lllite'], + [{ base: 'sdxl', type: 'main' }, null], + [{ base: 'sdxl', type: 'ip_adapter' }, null], + ] as const)('runs %j as %s', (model, kind) => { + expect(deriveControlKind(model)).toBe(kind); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/controlAdapters.ts b/invokeai/frontend/webv2/src/workbench/controlAdapters.ts index 8e146c3a127..f3d168251dc 100644 --- a/invokeai/frontend/webv2/src/workbench/controlAdapters.ts +++ b/invokeai/frontend/webv2/src/workbench/controlAdapters.ts @@ -1,7 +1,5 @@ import type { CanvasControlAdapterContract } from '@workbench/canvas-engine/api'; -import { areControlAdapterValuesValid, getControlAdapterMinWeight } from '@features/generation/graph'; - export const CONTROL_ADAPTER_DEFAULTS: Readonly< Record > = { @@ -75,6 +73,20 @@ export const getDefaultControlAdapterKind = (base: string | null | undefined): C (kind) => CONTROL_KIND_BASE[kind] === base ) ?? 'controlnet'; +/** + * The adapter kind a control model runs as on its own base: T2I-Adapter and Control LoRA by type, a `controlnet` + * model as its base's own kind (Z-Image, Qwen-Image, FLUX.2, Anima LLLite) or plain ControlNet. Null for other types. + */ +export const deriveControlKind = (model: { + type: string; + base: string; +}): CanvasControlAdapterContract['kind'] | null => { + if (model.type === 't2i_adapter' || model.type === 'control_lora') { + return model.type; + } + return model.type === 'controlnet' ? getDefaultControlAdapterKind(model.base) : null; +}; + /** A fresh copy of the default adapter for `base`, carrying `model`. */ export const createDefaultControlAdapter = ( base: string | null | undefined, @@ -89,29 +101,49 @@ const isRecord = (value: unknown): value is Record => typeof va const isControlAdapterKind = (value: unknown): value is CanvasControlAdapterContract['kind'] => typeof value === 'string' && Object.hasOwn(CONTROL_ADAPTER_DEFAULTS, value); -/** Repairs persisted numeric fields while preserving valid values and adapter kinds. */ +const CONTROL_MODES: readonly unknown[] = ['balanced', 'more_prompt', 'more_control', 'unbalanced']; + +/** + * Whether a stored weight can be kept: finite and within -1..2, the widest range any kind accepts. A stored adapter's + * kind is advisory (the layer runs as the kind its model resolves to, which the panel, readiness validation and the + * graph all use), so persistence must not judge the weight by it: a Fun kind's 0 floor would refuse or reset a + * negative weight a layer created under that base may now validly run with. + */ +export const isStorableControlWeight = (weight: unknown): weight is number => + typeof weight === 'number' && Number.isFinite(weight) && weight >= -1 && weight <= 2; + +/** Whether a stored begin/end step range can be kept: finite, within 0..1 and ordered; the same for every kind. */ +export const isStorableControlStepRange = (range: unknown): range is [number, number] => + Array.isArray(range) && + range.length === 2 && + typeof range[0] === 'number' && + typeof range[1] === 'number' && + Number.isFinite(range[0]) && + Number.isFinite(range[1]) && + range[0] >= 0 && + range[1] <= 1 && + range[0] < range[1]; + +/** + * Maps a persisted adapter to the contract, keeping every stored value persistence can hold and repairing the rest + * with the stored kind's defaults. No kind-specific reset applies (see {@link isStorableControlWeight}): the stored + * kind and model are kept as they are, and the resolved kind governs what runs. + */ export const normalizeControlAdapter = (value: unknown): unknown => { if (!isRecord(value) || !isControlAdapterKind(value.kind)) { return value; } const defaults = CONTROL_ADAPTER_DEFAULTS[value.kind]; - const minWeight = getControlAdapterMinWeight(value.kind); - const weight = - typeof value.weight === 'number' && Number.isFinite(value.weight) && value.weight >= minWeight && value.weight <= 2 - ? value.weight - : defaults.weight; - const beginEndStepPct: [number, number] = areControlAdapterValuesValid(value.kind, weight, value.beginEndStepPct) - ? [value.beginEndStepPct[0], value.beginEndStepPct[1]] - : [defaults.beginEndStepPct[0], defaults.beginEndStepPct[1]]; - - if (value.kind !== 'z_image_control' && value.kind !== 'qwen_image_control' && value.kind !== 'flux2_control') { - return { ...defaults, ...value, beginEndStepPct, weight } satisfies CanvasControlAdapterContract; - } return { - beginEndStepPct, - controlMode: null, + beginEndStepPct: isStorableControlStepRange(value.beginEndStepPct) + ? [value.beginEndStepPct[0], value.beginEndStepPct[1]] + : [defaults.beginEndStepPct[0], defaults.beginEndStepPct[1]], + controlMode: + value.controlMode === null || CONTROL_MODES.includes(value.controlMode) + ? (value.controlMode as CanvasControlAdapterContract['controlMode']) + : defaults.controlMode, kind: value.kind, model: typeof value.model === 'string' ? value.model : null, - weight, + weight: isStorableControlWeight(value.weight) ? value.weight : defaults.weight, } satisfies CanvasControlAdapterContract; }; diff --git a/invokeai/frontend/webv2/src/workbench/controlLayerChecks.test.ts b/invokeai/frontend/webv2/src/workbench/controlLayerChecks.test.ts index 3890a9c947b..53b63b9dabe 100644 --- a/invokeai/frontend/webv2/src/workbench/controlLayerChecks.test.ts +++ b/invokeai/frontend/webv2/src/workbench/controlLayerChecks.test.ts @@ -1,22 +1,29 @@ import type { ModelConfig } from '@features/models'; import type { CanvasControlLayerContract } from '@workbench/canvas-engine/api'; +import { resetArchitectureCapabilities } from '@features/generation/core/architectureCapabilities'; import { seedArchitectureCapabilities } from '@features/generation/core/architectureCapabilities.testing'; import { CONTROL_ADAPTER_KINDS, CONTROL_VALIDATION_REASONS } from '@features/generation/graph'; import { createInstance } from 'i18next'; import { describe, expect, it } from 'vitest'; import { + createControlLayerCheckContext, + describeControlLayer, describeControlLayerIssue, describeControlLayerReason, getBlockingControlLayerIssues, - getControlLayerAttentionReason, - getControlLayerReasonInSequence, hasControlLayerContent, } from './controlLayerChecks'; const model = (key: string, base: string, type: string): ModelConfig => ({ base, key, name: key, type }) as ModelConfig; +type MainModel = { base: string; variant?: string }; + +/** The ambient warning the panel and the layer row show: per-generation limits left to Invoke. */ +const ambientReason = (layer: CanvasControlLayerContract, mainModel: MainModel, models: readonly ModelConfig[]) => + describeControlLayer(layer, { ...createControlLayerCheckContext(mainModel.base), mainModel, models }).reason; + const controlLayer = ( id: string, overrides: { @@ -25,8 +32,10 @@ const controlLayer = ( model?: string | null; source?: CanvasControlLayerContract['source']; weight?: number; + derivation?: Pick; } = {} ): CanvasControlLayerContract => ({ + ...overrides.derivation, adapter: { beginEndStepPct: [0, 1], controlMode: null, @@ -64,19 +73,35 @@ describe('hasControlLayerContent', () => { }); describe('getBlockingControlLayerIssues', () => { - it('reports a content-bearing model-less control layer with a human sentence', () => { + it('leaves out a content-bearing layer with neither a type nor a model: it warns but contributes nothing', () => { + const layer = controlLayer('Control Layer 1'); + + expect(getBlockingControlLayerIssues({ layers: [layer], mainModel: sdxl, models })).toEqual([]); + expect(ambientReason(layer, sdxl, models)).toBe('missing_type'); + }); + + it('blocks an untyped layer whose model is no longer installed', () => { + expect( + getBlockingControlLayerIssues({ + layers: [controlLayer('legacy', { model: 'uninstalled' })], + mainModel: sdxl, + models, + }) + ).toMatchObject([{ code: 'missing_model', layerId: 'legacy' }]); + }); + + it('reports a typed content-bearing layer no model resolves, in structured form', () => { const issues = getBlockingControlLayerIssues({ - layers: [controlLayer('Control Layer 1')], + layers: [controlLayer('Control Layer 1', { derivation: { controlType: 'pose', detection: null } })], mainModel: sdxl, models, }); expect(issues).toEqual([ { - code: 'missing_model', + code: 'no_model_for_type', layerId: 'Control Layer 1', layerName: 'Control Layer 1', - suggestedKind: null, }, ]); }); @@ -120,7 +145,6 @@ describe('getBlockingControlLayerIssues', () => { code: 'control_lora_limit', layerId: 'second', layerName: 'second', - suggestedKind: null, }, ]); }); @@ -145,10 +169,10 @@ describe('getBlockingControlLayerIssues', () => { models: animaModels, }); - expect(issues.map(({ code, layerId, suggestedKind }) => [layerId, code, suggestedKind])).toEqual([ - ['again', 'duplicate_lllite_model', null], - ['inpaint', 'lllite_inpaint_adapter', null], - ['saved-as-controlnet', 'switch_adapter_kind', 'anima_lllite'], + expect(issues.map(({ code, layerId }) => [layerId, code])).toEqual([ + ['again', 'duplicate_lllite_model'], + ['inpaint', 'lllite_inpaint_adapter'], + ['saved-as-controlnet', 'switch_adapter_kind'], ]); }); @@ -191,8 +215,9 @@ describe('getBlockingControlLayerIssues', () => { }); it('reports issues in document order', () => { + const pose = { derivation: { controlType: 'pose', detection: null } } as const; const issues = getBlockingControlLayerIssues({ - layers: [controlLayer('top'), controlLayer('bottom')], + layers: [controlLayer('top', pose), controlLayer('bottom', pose)], mainModel: sdxl, models, }); @@ -201,46 +226,50 @@ describe('getBlockingControlLayerIssues', () => { }); }); -describe('getControlLayerAttentionReason', () => { - it('warns about a missing model even without content', () => { +describe('describeControlLayer ambient warnings', () => { + it('warns that a fresh layer needs a type, even without content', () => { const layer = controlLayer('fresh', { source: { bitmap: null, type: 'paint' } }); - expect(getControlLayerAttentionReason(layer, sdxl, models)).toBe('missing_model'); + expect(ambientReason(layer, sdxl, models)).toBe('missing_type'); }); it('returns null for a healthy layer and ignores per-kind limits', () => { - expect(getControlLayerAttentionReason(controlLayer('ok', { model: 'sdxl-control' }), sdxl, models)).toBeNull(); + expect(ambientReason(controlLayer('ok', { model: 'sdxl-control' }), sdxl, models)).toBeNull(); }); - it('warns that a Klein model cannot run the FLUX.2 control, even on a fresh layer without a model', () => { - const fresh = controlLayer('fresh', { kind: 'flux2_control', source: { bitmap: null, type: 'paint' } }); + it('warns that a Klein model cannot run the FLUX.2 control, even on a fresh typed layer without a model', () => { + const fresh = controlLayer('fresh', { + derivation: { controlType: 'depth', detection: null }, + source: { bitmap: null, type: 'paint' }, + }); const picked = controlLayer('picked', { kind: 'flux2_control', model: 'flux2-union', weight: 0.75 }); const flux2Models = [model('flux2-union', 'flux2', 'controlnet')]; - expect(getControlLayerAttentionReason(fresh, { base: 'flux2', variant: 'klein_4b' }, flux2Models)).toBe( - 'flux2_klein_control' - ); - expect(getControlLayerAttentionReason(picked, { base: 'flux2', variant: 'dev' }, flux2Models)).toBeNull(); + expect(ambientReason(fresh, { base: 'flux2', variant: 'klein_4b' }, flux2Models)).toBe('flux2_klein_control'); + expect(ambientReason(picked, { base: 'flux2', variant: 'dev' }, flux2Models)).toBeNull(); }); it('warns that an Edit model cannot run the Qwen-Image control', () => { const qwenModels = [model('qwen-union', 'qwen-image', 'controlnet')]; const layer = controlLayer('qwen', { kind: 'qwen_image_control', model: 'qwen-union', weight: 0.85 }); - expect(getControlLayerAttentionReason(layer, { base: 'qwen-image' }, qwenModels)).toBeNull(); - expect(getControlLayerAttentionReason(layer, { base: 'qwen-image', variant: 'edit' }, qwenModels)).toBe( - 'qwen_image_edit_control' - ); + expect(ambientReason(layer, { base: 'qwen-image' }, qwenModels)).toBeNull(); + expect(ambientReason(layer, { base: 'qwen-image', variant: 'edit' }, qwenModels)).toBe('qwen_image_edit_control'); }); }); -describe('getControlLayerReasonInSequence', () => { +describe('describeControlLayer after earlier layers', () => { const fluxModels = [model('flux-lora', 'flux', 'control_lora'), model('sdxl-lora', 'sdxl', 'control_lora')]; const target = controlLayer('target', { kind: 'control_lora', model: 'flux-lora' }); it('lets only an earlier layer that passes claim the one Control LoRA slot, as the invocation does', () => { const judge = (earlier: CanvasControlLayerContract[]) => - getControlLayerReasonInSequence({ earlier, layer: target, mainModel: { base: 'flux' }, models: fluxModels }); + describeControlLayer(target, { + ...createControlLayerCheckContext('flux'), + earlier, + mainModel: { base: 'flux' }, + models: fluxModels, + }).reason; expect(judge([])).toBeNull(); expect(judge([controlLayer('invalid', { kind: 'control_lora', model: 'sdxl-lora' })])).toBeNull(); @@ -248,6 +277,143 @@ describe('getControlLayerReasonInSequence', () => { }); }); +describe('typed control layers', () => { + const depthModels = [model('sdxl-depth-a', 'sdxl', 'controlnet'), model('sdxl-depth-b', 'sdxl', 'controlnet')]; + const typed = ( + id: string, + derivation: Partial> = {}, + overrides: Parameters[1] = {} + ) => controlLayer(id, { ...overrides, derivation: { controlType: 'depth', detection: null, ...derivation } }); + const depthDetection = { settings: {}, type: 'depth_anything_depth_estimation' }; + + it('asks for a model when several fit, and resolves with the remembered pick', () => { + const layers = [typed('depth')]; + + expect(getBlockingControlLayerIssues({ layers, mainModel: sdxl, models: depthModels })).toMatchObject([ + { code: 'choose_model' }, + ]); + expect( + getBlockingControlLayerIssues({ + context: createControlLayerCheckContext('sdxl', { 'sdxl:depth': 'sdxl-depth-b' }), + layers, + mainModel: sdxl, + models: depthModels, + }) + ).toEqual([]); + }); + + it('tells no control models apart from none for the type', () => { + expect(getBlockingControlLayerIssues({ layers: [typed('depth')], mainModel: sdxl, models: [] })).toMatchObject([ + { code: 'no_control_models' }, + ]); + }); + + it('never blocks on an empty layer, though it still warns', () => { + const empty = typed('empty', {}, { source: { bitmap: null, type: 'paint' } }); + + expect(getBlockingControlLayerIssues({ layers: [empty], mainModel: sdxl, models: depthModels })).toEqual([]); + expect(ambientReason(empty, sdxl, depthModels)).toBe('choose_model'); + }); + + it('blocks on a map detection has not caught up with, after the model reasons', () => { + const pending = typed('pending', { detection: depthDetection }); + + expect( + getBlockingControlLayerIssues({ layers: [pending], mainModel: sdxl, models: [depthModels[0]!] }) + ).toMatchObject([{ code: 'detection_pending' }]); + expect(getBlockingControlLayerIssues({ layers: [pending], mainModel: sdxl, models: depthModels })).toMatchObject([ + { code: 'choose_model' }, + ]); + expect(ambientReason(pending, sdxl, [depthModels[0]!])).toBe('detection_pending'); + }); + + it('counts per-generation limits by the kind the resolved model runs, not the stored kind', () => { + const fluxDepth = [model('flux-depth-lora', 'flux', 'control_lora')]; + const issues = getBlockingControlLayerIssues({ + layers: [typed('first'), typed('second')], + mainModel: { base: 'flux' }, + models: fluxDepth, + }); + + expect(issues.map(({ code, layerId }) => [layerId, code])).toEqual([['second', 'control_lora_limit']]); + }); + + it('still resolves models before capabilities load, and reports that they have not', () => { + const context = createControlLayerCheckContext('flux'); + const judge = () => + getBlockingControlLayerIssues({ + context, + layers: [typed('depth')], + mainModel: { base: 'flux' }, + models: [model('flux-depth-t2i', 'flux', 't2i_adapter')], + }); + + // Loaded, FLUX runs no T2I-Adapter, so no model fits. + expect(judge()).toMatchObject([{ code: 'no_control_models' }]); + // The context reads capabilities when it judges, not when it is made. + resetArchitectureCapabilities(); + expect(judge()).toMatchObject([{ code: 'capabilities_unavailable' }]); + }); +}); + +describe('describeControlLayer', () => { + const sdxlDepth = [model('sdxl-depth', 'sdxl', 'controlnet')]; + const ctx = { ...createControlLayerCheckContext('sdxl'), mainModel: sdxl, models: sdxlDepth }; + const detection = { settings: {}, type: 'depth_anything_depth_estimation' }; + const original = { image: { height: 64, imageName: 'photo.png', width: 64 }, type: 'image' } as const; + const depth = (derivation: Pick, empty = false) => + controlLayer('depth', { + derivation: { controlType: 'depth', ...derivation }, + ...(empty ? { source: { bitmap: null, type: 'paint' } } : {}), + }); + + it.each([ + ['empty', depth({ detection }, true)], + ['none', controlLayer('untyped', { model: 'sdxl-depth' })], + ['none', depth({ detection: null })], + ['stale', depth({ detection })], + ['current', depth({ detection, filter: detection, original })], + ['no-input', depth({ detection, filter: { settings: {}, type: 'canny_edge_detection' } })], + ] as const)('reports a %s map', (map, layer) => { + expect(describeControlLayer(layer, ctx).map).toBe(map); + }); + + it('resolves across every base without a main model, reporting only resolution and detection', () => { + const noMain = { + ...createControlLayerCheckContext(null), + mainModel: null, + models: [model('sd1-depth', 'sd-1', 'controlnet'), model('flux-t2i-depth', 'flux', 't2i_adapter')], + }; + + expect(describeControlLayer(depth({ detection: null }), noMain)).toMatchObject({ + reason: 'choose_model', + resolution: { candidates: noMain.models, how: 'ask' }, + }); + expect(describeControlLayer(depth({ detection }), { ...noMain, models: [noMain.models[0]!] })).toMatchObject({ + map: 'stale', + reason: 'detection_pending', + resolution: { how: 'only' }, + }); + }); + + it('carries the resolution, and with earlier layers judges per-generation limits as Invoke does', () => { + const flux = { + ...createControlLayerCheckContext('flux'), + mainModel: { base: 'flux' }, + models: [model('flux-depth-lora', 'flux', 'control_lora')], + }; + const layer = depth({ detection: null }); + + expect(describeControlLayer(layer, flux)).toMatchObject({ + reason: null, + resolution: { how: 'only', kind: 'control_lora' }, + }); + expect(describeControlLayer(layer, { ...flux, earlier: [depth({ detection: null })] }).reason).toBe( + 'control_lora_limit' + ); + }); +}); + const englishCatalogModules = import.meta.glob('../../public/locales/en.json', { eager: true, import: 'default' }); const english = createInstance(); await english.init({ @@ -260,21 +426,23 @@ const t = english.t.bind(english) as (key: string, options?: Record { it('has an English string for every reason code, adapter kind and hidden-model note', () => { const keys = [ - ...CONTROL_VALIDATION_REASONS.map((code) => `widgets.layers.control.validation.${code}`), + ...[...CONTROL_VALIDATION_REASONS, 'detection_pending'].map( + (code) => `widgets.layers.control.validation.${code}` + ), ...CONTROL_ADAPTER_KINDS.map((kind) => `widgets.layers.control.kinds.${kind}`), 'widgets.layers.control.hiddenModels.lllite_inpaint_adapter', 'widgets.layers.control.hiddenModels.lllite_channels_unknown', - 'widgets.layers.control.switchKind', 'widgets.layers.control.invalidLayer', ]; expect(keys.filter((key) => !english.exists(key))).toEqual([]); }); - it('names the kind to switch to and the layer, from the locale', () => { - expect(describeControlLayerReason(t, 'switch_adapter_kind', 'anima_lllite')).toContain( - t('widgets.layers.control.kinds.anima_lllite') + it('words a reason and names the layer, from the locale', () => { + // The Adapter type control is gone: a layer the main model cannot run recovers by choosing a type. + expect(describeControlLayerReason(t, 'switch_adapter_kind')).toBe( + 'Choose a control type so a model for the selected main model can be picked.' ); - expect(describeControlLayerIssue(t, { code: 'missing_model', layerName: 'Sketch', suggestedKind: null })).toBe( + expect(describeControlLayerIssue(t, { code: 'missing_model', layerName: 'Sketch' })).toBe( `Control layer "Sketch": ${t('widgets.layers.control.validation.missing_model')}` ); }); diff --git a/invokeai/frontend/webv2/src/workbench/controlLayerChecks.ts b/invokeai/frontend/webv2/src/workbench/controlLayerChecks.ts index 4958d679917..f42d4d833b0 100644 --- a/invokeai/frontend/webv2/src/workbench/controlLayerChecks.ts +++ b/invokeai/frontend/webv2/src/workbench/controlLayerChecks.ts @@ -3,125 +3,175 @@ import type { CanvasControlLayerContract, CanvasLayerContract } from '@workbench import { createControlValidationSequence, - getControlValidationReason, - getSuggestedControlKind, + isControlKindSupportedForBase, type ControlAdapterKind, type ControlValidationReason, } from '@features/generation/graph'; -import { isCompositableControlLayer } from '@workbench/canvasLayerContent'; +import { hasArchitectureCapabilities } from '@features/generation/settings'; +import { getControlDetectionInput, isControlMapCurrent, needsControlDetection } from '@workbench/canvas-engine/api'; +import { hasControlLayerContent, isCompositableControlLayer } from '@workbench/canvasLayerContent'; + +import { resolveControlModel, type ControlModelMemory, type ControlModelResolution } from './controlModelResolution'; + +/** Everything that can hold a control layer back: a validation reason, or a map detection has not caught up with. */ +export type ControlLayerIssueCode = ControlValidationReason | 'detection_pending'; /** A control layer that blocks generation; the shell words it from the locale (see `controlLayerMessages`). */ export interface ControlLayerIssue { - code: ControlValidationReason; + code: ControlLayerIssueCode; layerId: string; layerName: string; - /** For `switch_adapter_kind`: the kind the selected base runs. */ - suggestedKind: ControlAdapterKind | null; } -export { hasControlLayerContent } from '@workbench/canvasLayerContent'; +/** What model resolution needs beyond the layer: remembered picks and the kinds the main model's base runs. */ +export interface ControlLayerCheckContext { + remembered: ControlModelMemory; + supportsKind: (kind: ControlAdapterKind) => boolean; +} + +/** + * Before capabilities load every kind counts as supported, so resolution still finds models; validation then + * reports `capabilities_unavailable` instead of a misleading missing model. Without a main model (`base` null) no + * base restricts kinds either. + */ +export const createControlLayerCheckContext = ( + base: string | null, + remembered: ControlModelMemory = {} +): ControlLayerCheckContext => ({ + remembered, + supportsKind: (kind) => base === null || !hasArchitectureCapabilities() || isControlKindSupportedForBase(base, kind), +}); -const resolveAdapterModel = (adapterModel: string | null, models: readonly ModelConfig[]): ModelConfig | null => - (adapterModel ? models.find((candidate) => candidate.key === adapterModel) : undefined) ?? null; +export { hasControlLayerContent } from '@workbench/canvasLayerContent'; type MainModel = { base: string; variant?: string | null }; -/** Validates control layers in generation order, as the invocation does. */ -const createLayerValidator = (mainModel: MainModel, models: readonly ModelConfig[]) => { +export interface ControlLayerJudgement { + resolution: ControlModelResolution; + reason: ControlLayerIssueCode | null; + /** A layer with no type and no model warns but contributes nothing, so it never blocks. */ + skip: boolean; +} + +/** + * Judges control layers in generation order, as the invocation does: per-generation limits count only layers that + * passed. Invoke readiness, the settings panel and the invocation collector share it. + */ +export const createControlLayerSequence = ( + mainModel: MainModel, + models: readonly ModelConfig[], + context: ControlLayerCheckContext = createControlLayerCheckContext(mainModel.base) +): ((layer: CanvasControlLayerContract) => ControlLayerJudgement) => { const validate = createControlValidationSequence(mainModel); - return ({ adapter }: CanvasControlLayerContract): ControlValidationReason | null => - validate({ - adapterModel: resolveAdapterModel(adapter.model, models), - beginEndStepPct: adapter.beginEndStepPct, - kind: adapter.kind, - weight: adapter.weight, + return (layer) => { + const resolution = resolveControlModel({ + base: mainModel.base, + layer, + models, + remembered: context.remembered, + supportsKind: context.supportsKind, }); + // A layer with no type and no model keeps its reason for the panel but is skipped; having no model, it never + // passes, so it never claims a per-generation slot. + const reason = + validate({ + adapterModel: resolution.model, + beginEndStepPct: layer.adapter.beginEndStepPct, + kind: resolution.kind, + unresolvedModel: resolution.unresolved, + weight: layer.adapter.weight, + }) ?? (needsControlDetection(layer) ? 'detection_pending' : null); + return { reason, resolution, skip: resolution.unresolved === 'missing_type' }; + }; }; /** * Validate the same enabled, content-bearing layers as createControlLayerCollector; invalid layers do not consume - * per-generation limits. + * per-generation limits, and layers without a type or model are left out. */ export const getBlockingControlLayerIssues = (params: { layers: readonly CanvasLayerContract[]; mainModel: MainModel; models: readonly ModelConfig[]; + context?: ControlLayerCheckContext; }): ControlLayerIssue[] => { const { layers, mainModel, models } = params; - const validate = createLayerValidator(mainModel, models); + const judge = createControlLayerSequence(mainModel, models, params.context); const issues: ControlLayerIssue[] = []; for (const layer of layers) { if (!isCompositableControlLayer(layer)) { continue; } - const code = validate(layer); - if (code) { - issues.push({ - code, - layerId: layer.id, - layerName: layer.name, - suggestedKind: code === 'switch_adapter_kind' ? getSuggestedControlKind(mainModel.base) : null, - }); + const { reason, skip } = judge(layer); + if (reason && !skip) { + issues.push({ code: reason, layerId: layer.id, layerName: layer.name }); } } return issues; }; /** - * The reason one layer would block generation, judged exactly as the invocation judges it: `earlier` are the - * contributing, content-bearing control layers above it in generation order, which may claim limited slots first. + * Where a layer's control map stands. `empty`: no pixels yet. `none`: no type or no detection, so the pixels are the + * map. `current`: the map matches its detection. `stale`: detection must run again. `no-input`: the map is all the + * layer has, so it cannot be detected again until its image is replaced. */ -export const getControlLayerReasonInSequence = (params: { - earlier: readonly CanvasControlLayerContract[]; - layer: CanvasControlLayerContract; - mainModel: MainModel; - models: readonly ModelConfig[]; -}): ControlValidationReason | null => { - const validate = createLayerValidator(params.mainModel, params.models); - for (const earlier of params.earlier) { - validate(earlier); +export type ControlMapState = 'none' | 'current' | 'stale' | 'no-input' | 'empty'; + +const getControlMapState = (layer: CanvasControlLayerContract): ControlMapState => { + if (!hasControlLayerContent(layer) && !layer.original) { + return 'empty'; + } + if (!layer.controlType || !layer.detection) { + return 'none'; + } + if (isControlMapCurrent(layer)) { + return 'current'; } - return validate(params.layer); + return getControlDetectionInput(layer) ? 'stale' : 'no-input'; }; -/** Ambient warnings include empty layers but omit per-kind limits, which belong in the Invoke tooltip. */ -export const getControlLayerAttentionReason = ( +/** + * One layer as the settings panel, the layer row and Invoke present it. With `earlier` (the contributing layers + * above it) the reason includes per-generation limits, as Invoke judges it; without, it is the ambient warning. + * Without a main model the layer resolves across every base and only resolution and detection give a reason. + */ +export const describeControlLayer = ( layer: CanvasControlLayerContract, - mainModel: MainModel, - models: readonly ModelConfig[] -): ControlValidationReason | null => - getControlValidationReason({ - adapterModel: resolveAdapterModel(layer.adapter.model, models), - beginEndStepPct: layer.adapter.beginEndStepPct, - controlLoraIndex: 0, - flux2ControlIndex: 0, - kind: layer.adapter.kind, - mainBase: mainModel.base, - mainVariant: mainModel.variant ?? undefined, - qwenImageControlIndex: 0, - weight: layer.adapter.weight, - zImageControlIndex: 0, - }); + ctx: ControlLayerCheckContext & { + mainModel: MainModel | null; + models: readonly ModelConfig[]; + earlier?: readonly CanvasControlLayerContract[]; + } +): { resolution: ControlModelResolution; map: ControlMapState; reason: ControlLayerIssueCode | null } => { + const context = { remembered: ctx.remembered, supportsKind: ctx.supportsKind }; + let judgement: { resolution: ControlModelResolution; reason: ControlLayerIssueCode | null }; + if (!ctx.mainModel) { + const resolution = resolveControlModel({ ...context, base: null, layer, models: ctx.models }); + judgement = { + reason: resolution.unresolved ?? (needsControlDetection(layer) ? 'detection_pending' : null), + resolution, + }; + } else { + // Without `earlier` the layer comes first in a fresh sequence, so no per-generation limit can apply. + const judge = createControlLayerSequence(ctx.mainModel, ctx.models, context); + for (const earlier of ctx.earlier ?? []) { + judge(earlier); + } + judgement = judge(layer); + } + return { map: getControlMapState(layer), reason: judgement.reason, resolution: judgement.resolution }; +}; type Translate = (key: string, options?: Record) => string; /** The one user-facing wording of a reason, shared by the settings panel, the layer row and the Invoke tooltip. */ -export const describeControlLayerReason = ( - t: Translate, - code: ControlValidationReason, - suggestedKind: ControlAdapterKind | null -): string => - t(`widgets.layers.control.validation.${code}`, { - kind: suggestedKind ? t(`widgets.layers.control.kinds.${suggestedKind}`) : '', - }); +export const describeControlLayerReason = (t: Translate, code: ControlLayerIssueCode): string => + t(`widgets.layers.control.validation.${code}`); /** A blocking issue as the Invoke tooltip and the failed-invoke notice show it. */ -export const describeControlLayerIssue = ( - t: Translate, - issue: Pick -): string => +export const describeControlLayerIssue = (t: Translate, issue: Pick): string => t('widgets.layers.control.invalidLayer', { name: issue.layerName, - reason: describeControlLayerReason(t, issue.code, issue.suggestedKind), + reason: describeControlLayerReason(t, issue.code), }); diff --git a/invokeai/frontend/webv2/src/workbench/controlModelResolution.test.ts b/invokeai/frontend/webv2/src/workbench/controlModelResolution.test.ts new file mode 100644 index 00000000000..2f21ac401c6 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/controlModelResolution.test.ts @@ -0,0 +1,332 @@ +import type { ControlAdapterKind } from '@features/generation/graph'; +import type { ModelConfig } from '@features/models'; +import type { CanvasControlLayerContract, ControlType } from '@workbench/canvas-engine/api'; + +import { describe, expect, it } from 'vitest'; + +import { + getTypeAvailability, + isRememberableControlPick, + resolveControlModel, + type ControlModelMemory, +} from './controlModelResolution'; + +const model = (key: string, base: string, type = 'controlnet', extra: Partial = {}): ModelConfig => + ({ base, key, name: key, type, ...extra }) as ModelConfig; + +type Layer = Pick; + +const layer = ( + controlType: ControlType | undefined, + overrides: { detector?: string | null; model?: string | null; kind?: ControlAdapterKind } = {} +): Layer => ({ + adapter: { + beginEndStepPct: [0, 1], + controlMode: null, + kind: overrides.kind ?? 'controlnet', + model: overrides.model ?? null, + weight: 0.75, + }, + ...(controlType ? { controlType } : {}), + ...(overrides.detector === undefined + ? {} + : { detection: overrides.detector === null ? null : { settings: {}, type: overrides.detector } }), +}); + +const supportsAll = (): boolean => true; + +const resolve = ( + target: Layer, + base: string | null, + models: readonly ModelConfig[], + remembered: ControlModelMemory = {}, + supportsKind: (kind: ControlAdapterKind) => boolean = supportsAll +) => resolveControlModel({ base, layer: target, models, remembered, supportsKind }); + +const depthA = model('sdxl-depth-a', 'sdxl'); +const depthB = model('sdxl-depth-b', 'sdxl', 'controlnet', { + default_settings: { preprocessor: 'depth_anything_image_processor' }, +}); +const normal = model('sdxl-normal', 'sdxl'); +const pose = model('sdxl-openpose', 'sdxl'); +const sdxlUnion = model('xinsir-controlnet-union-sdxl-promax', 'sdxl'); +const sd1Depth = model('sd15-depth', 'sd-1'); + +describe('resolveControlModel without a type', () => { + it('applies the stored adapter as it is', () => { + const resolution = resolve(layer(undefined, { kind: 't2i_adapter', model: pose.key }), 'sdxl', [pose]); + + expect(resolution).toMatchObject({ how: 'none', kind: 't2i_adapter', model: pose, unresolved: null }); + expect(resolution.candidates).toEqual([]); + }); + + it('lists every model it could run on the base, for a direct pick, and no candidates', () => { + const resolution = resolve(layer(undefined), 'sdxl', [ + depthA, + pose, + sdxlUnion, + sd1Depth, + model('sdxl-main', 'sdxl', 'main'), + model('my-sdxl-control', 'sdxl'), + ]); + + expect(resolution.usable.map((entry) => entry.key)).toEqual([ + depthA.key, + pose.key, + sdxlUnion.key, + 'my-sdxl-control', + ]); + expect(resolution.candidates).toEqual([]); + }); + + it('reports a missing type when the layer names no model', () => { + expect(resolve(layer(undefined), 'sdxl', [pose])).toMatchObject({ + how: 'none', + model: null, + unresolved: 'missing_type', + }); + }); + + it('reports a missing model, not a missing type, when an untyped layer names one that is not installed', () => { + expect(resolve(layer(undefined, { model: 'deleted' }), 'sdxl', [pose])).toMatchObject({ + how: 'none', + model: null, + unresolved: 'missing_model', + }); + }); +}); + +describe('resolveControlModel with a type', () => { + it('runs an explicit pick for this base as the kind that model runs', () => { + const t2i = model('t2i-depth', 'sdxl', 't2i_adapter'); + const resolution = resolve(layer('depth', { model: t2i.key }), 'sdxl', [depthA, t2i]); + + expect(resolution).toMatchObject({ how: 'explicit', kind: 't2i_adapter', model: t2i, unresolved: null }); + }); + + it('ignores a pick made for another base, and one that was deleted', () => { + expect(resolve(layer('depth', { model: sd1Depth.key }), 'sdxl', [sd1Depth, depthA])).toMatchObject({ + how: 'only', + model: depthA, + }); + expect(resolve(layer('depth', { model: 'deleted' }), 'sdxl', [depthA])).toMatchObject({ + how: 'only', + model: depthA, + }); + }); + + it('runs the only model made for the type', () => { + expect(resolve(layer('pose'), 'sdxl', [depthA, pose, sdxlUnion])).toMatchObject({ + how: 'only', + kind: 'controlnet', + model: pose, + }); + }); + + it('asks when several models fit, listing those made for the type first, then unions', () => { + const fluxDepth = model('flux-depth-lora', 'flux', 'control_lora'); + const fluxUnion = model('flux-union-pro', 'flux'); + const fluxUnion2 = model('flux-union-alpha', 'flux'); + + expect(resolve(layer('depth'), 'sdxl', [depthA, depthB])).toMatchObject({ + candidates: [depthA, depthB], + how: 'ask', + model: null, + unresolved: 'choose_model', + }); + expect(resolve(layer('depth'), 'flux', [fluxUnion, fluxDepth]).candidates).toEqual([fluxDepth, fluxUnion]); + expect(resolve(layer('edges'), 'flux', [fluxUnion, fluxUnion2])).toMatchObject({ + candidates: [fluxUnion, fluxUnion2], + how: 'ask', + unresolved: 'choose_model', + }); + }); + + it('settles a choice with the remembered pick for this base and type, only while it is a candidate', () => { + const models = [depthA, depthB, pose]; + + expect(resolve(layer('depth'), 'sdxl', models, { 'sdxl:depth': depthB.key })).toMatchObject({ + how: 'remembered', + model: depthB, + }); + expect(resolve(layer('depth'), 'sdxl', models, { 'sdxl:depth': pose.key }).how).toBe('ask'); + expect(resolve(layer('depth'), 'sdxl', models, { 'sdxl:depth': 'deleted' }).how).toBe('ask'); + expect(resolve(layer('depth'), 'sdxl', models, { 'sd-1:depth': depthB.key }).how).toBe('ask'); + }); + + it('prefers the remembered covering union over the only model made for the type', () => { + const models = [depthA, sdxlUnion]; + + expect(resolve(layer('depth'), 'sdxl', models)).toMatchObject({ how: 'only', model: depthA }); + expect(resolve(layer('depth'), 'sdxl', models, { 'sdxl:depth': sdxlUnion.key })).toMatchObject({ + how: 'remembered', + model: sdxlUnion, + }); + }); + + it('matches the family the chosen detector produces', () => { + const canny = model('sdxl-canny', 'sdxl'); + const mlsd = model('sdxl-mlsd', 'sdxl'); + + expect( + resolve(layer('depth', { detector: 'depth_anything_depth_estimation' }), 'sdxl', [depthA, normal]).model + ).toBe(depthA); + expect(resolve(layer('depth', { detector: 'normal_map' }), 'sdxl', [depthA, normal]).model).toBe(normal); + expect(resolve(layer('edges', { detector: 'mlsd_detection' }), 'sdxl', [canny, mlsd]).model).toBe(mlsd); + expect(resolve(layer('edges', { detector: null }), 'sdxl', [canny, mlsd]).model).toBe(canny); + }); + + it('falls back to a model made for the type when none matches the family', () => { + expect(resolve(layer('depth'), 'sdxl', [normal])).toMatchObject({ how: 'only', model: normal }); + }); + + it('uses the one union model that covers the family', () => { + const zUnion = model('z-image-fun-controlnet', 'z-image'); + + expect(resolve(layer('pose'), 'sdxl', [sdxlUnion])).toMatchObject({ how: 'union', model: sdxlUnion }); + expect(resolve(layer('color'), 'sdxl', [sdxlUnion]).unresolved).toBe('no_model_for_type'); + + expect(resolve(layer('depth'), 'z-image', [zUnion])).toMatchObject({ + how: 'union', + kind: 'z_image_control', + model: zUnion, + }); + expect(resolve(layer('lineart'), 'z-image', [zUnion])).toMatchObject({ + how: 'missing', + kind: 'z_image_control', + unresolved: 'no_model_for_type', + }); + }); + + it('tells no control models apart from none for this type', () => { + expect(resolve(layer('pose'), 'sdxl', [depthA, sd1Depth])).toMatchObject({ + how: 'missing', + unresolved: 'no_model_for_type', + }); + expect(resolve(layer('pose'), 'sdxl', [sd1Depth])).toMatchObject({ + how: 'missing', + kind: 'controlnet', + unresolved: 'no_control_models', + }); + // An SD1.5 union carries no coverage until the backend reports its control types. + expect(resolve(layer('pose'), 'sd-1', [model('controlnet-union-sd15', 'sd-1')]).unresolved).toBe( + 'no_model_for_type' + ); + }); + + it('only counts models the base runs and validation would accept', () => { + const t2i = model('t2i-depth', 'sdxl', 't2i_adapter'); + const noT2i = (kind: ControlAdapterKind) => kind !== 't2i_adapter'; + const lllite = model('anima-lllite-depth', 'anima', 'controlnet', { cond_in_channels: 3 }); + const inpaint = model('anima-lllite-depth-inpaint', 'anima', 'controlnet', { cond_in_channels: 4 }); + + expect(resolve(layer('depth'), 'sdxl', [t2i], {}, noT2i).unresolved).toBe('no_control_models'); + expect(resolve(layer('depth', { model: t2i.key }), 'sdxl', [t2i, depthA], {}, noT2i).model).toBe(depthA); + expect(resolve(layer('depth'), 'anima', [lllite, inpaint])).toMatchObject({ + how: 'only', + kind: 'anima_lllite', + model: lllite, + }); + }); + + it('considers every base when no main model is selected', () => { + expect(resolve(layer('depth'), null, [sd1Depth]).model).toBe(sd1Depth); + }); +}); + +describe('resolveControlModel model lists and family match', () => { + it('lists every model the layer could run on this base, beside the candidates for its type', () => { + const resolution = resolve(layer('depth'), 'sdxl', [ + depthA, + pose, + sdxlUnion, + sd1Depth, + model('sdxl-main', 'sdxl', 'main'), + ]); + + expect(resolution.candidates).toEqual([depthA, sdxlUnion]); + expect(resolution.usable).toEqual([depthA, pose, sdxlUnion]); + }); + + it('says whether the model matches the detector family or only the type', () => { + const fluxUnion = model('flux-union-pro', 'flux'); + + expect(resolve(layer('depth'), 'sdxl', [depthA]).familyMatch).toBe(true); + expect(resolve(layer('depth'), 'sdxl', [normal])).toMatchObject({ familyMatch: false, how: 'only', model: normal }); + expect(resolve(layer('depth'), 'sdxl', [normal], { 'sdxl:depth': normal.key }).familyMatch).toBe(false); + expect(resolve(layer('depth'), 'flux', [fluxUnion])).toMatchObject({ familyMatch: true, how: 'union' }); + expect(resolve(layer('depth', { model: pose.key }), 'sdxl', [pose])).toMatchObject({ + familyMatch: false, + how: 'explicit', + }); + expect(resolve(layer('depth'), 'sdxl', [depthA, depthB]).familyMatch).toBe(false); + }); +}); + +describe('isRememberableControlPick', () => { + it('remembers only a candidate for the type and detector family', () => { + const unpurposed = model('my-sdxl-control', 'sdxl'); + const models = [depthA, normal, pose, sdxlUnion, unpurposed]; + const resolution = resolve(layer('depth', { detector: 'depth_anything_depth_estimation' }), 'sdxl', models); + + expect(isRememberableControlPick(resolution, depthA)).toBe(true); + expect(isRememberableControlPick(resolution, sdxlUnion)).toBe(true); + expect(isRememberableControlPick(resolution, normal)).toBe(false); + expect(isRememberableControlPick(resolution, pose)).toBe(false); + expect(isRememberableControlPick(resolution, unpurposed)).toBe(false); + expect(isRememberableControlPick(resolve(layer(undefined), 'sdxl', models), depthA)).toBe(false); + }); +}); + +describe('resolveControlModel InstantX modes', () => { + const fluxUnion = model('FLUX.1-dev-Controlnet-Union-Pro', 'flux'); + + it.each([ + ['edges', 0], + ['tile', 1], + ['depth', 2], + ['pose', 4], + ] as const)('gives a FLUX union %s layer mode %s', (type, mode) => { + expect(resolve(layer(type), 'flux', [fluxUnion])).toMatchObject({ instantxControlMode: mode, model: fluxUnion }); + }); + + it('leaves the mode out for families InstantX lacks, dedicated FLUX models and other bases', () => { + expect( + resolve(layer('edges', { detector: 'mlsd_detection', model: fluxUnion.key }), 'flux', [fluxUnion]) + ).toMatchObject({ + how: 'explicit', + instantxControlMode: null, + }); + expect( + resolve(layer('depth'), 'flux', [model('flux-depth-lora', 'flux', 'control_lora')]).instantxControlMode + ).toBeNull(); + expect(resolve(layer('depth'), 'z-image', [model('z-image-fun', 'z-image')]).instantxControlMode).toBeNull(); + }); +}); + +describe('getTypeAvailability', () => { + const byType = (availability: ReturnType) => + Object.fromEntries(availability.map(({ count, state, type }) => [type, [state, count]])); + + it('hints how each type would resolve on a fresh layer', () => { + expect(byType(getTypeAvailability('sdxl', [depthA, depthB, pose, sdxlUnion, sd1Depth], supportsAll))).toEqual({ + color: ['missing', 0], + depth: ['ask', 2], + edges: ['union', 1], + lineart: ['union', 1], + pose: ['ready', 1], + sketch: ['union', 1], + softedge: ['union', 1], + tile: ['union', 1], + }); + expect( + byType( + getTypeAvailability( + 'flux', + [model('flux-depth-lora', 'flux', 'control_lora'), model('flux-union', 'flux')], + supportsAll + ) + ) + ).toMatchObject({ depth: ['ready', 1], edges: ['union', 1], lineart: ['missing', 0], tile: ['union', 1] }); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/controlModelResolution.ts b/invokeai/frontend/webv2/src/workbench/controlModelResolution.ts new file mode 100644 index 00000000000..5f2e317e47a --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/controlModelResolution.ts @@ -0,0 +1,203 @@ +import type { ControlAdapterKind } from '@features/generation/graph'; +import type { ModelConfig } from '@features/models'; +import type { CanvasControlLayerContract, ControlType } from '@workbench/canvas-engine/api'; + +import { getControlModelUnusableReason } from '@features/generation/graph'; +import { CONTROL_TYPES } from '@workbench/canvas-engine/api'; + +import { deriveControlKind, getDefaultControlAdapterKind } from './controlAdapters'; +import { CONTROL_TYPE_SPECS, classifyControlModel, getDetectorFamily, type ControlFamily } from './controlTypes'; + +/** The user's last model pick per main-model base and control type: `${base}:${controlType}` → model key. */ +export type ControlModelMemory = Readonly>; + +export const controlModelMemoryKey = (base: string, type: ControlType): string => `${base}:${type}`; + +/** + * Why no model drives a layer; validation reports it in place of `missing_model`. `missing_type`: no type and no + * model, so the layer contributes nothing. `missing_model`: an untyped layer names a model that is not installed. + */ +export type ControlModelUnresolvedReason = + | 'missing_type' + | 'missing_model' + | 'choose_model' + | 'no_model_for_type' + | 'no_control_models'; + +export interface ControlModelResolution { + /** + * `explicit`: the layer's own pick. `remembered`: the last pick for this base and type, still among `candidates`; + * it applies to every layer of that base and type without an explicit pick. `only`: the one model made for the + * type. `union`: the one union model covering it. `ask`: several fit. `missing`: none fit. `none`: the layer has + * no type, so its stored adapter applies as it is. + */ + how: 'explicit' | 'remembered' | 'only' | 'union' | 'ask' | 'missing' | 'none'; + model: ModelConfig | null; + /** The kind the layer runs as: the model's own, else the base's default (the stored kind for untyped layers). */ + kind: ControlAdapterKind; + /** Models that fit the type: those made for it first, then covering unions. */ + candidates: readonly ModelConfig[]; + /** Every model the layer could run on this base, whatever it was made for, typed or not. */ + usable: readonly ModelConfig[]; + /** + * Whether the model was made for (or, as a union, covers) the family the detector produces; false when it only + * shares the type's purpose (a normal-map model on a depth layer), and for untyped or unresolved layers. + */ + familyMatch: boolean; + /** FLUX InstantX Union mode for this type's family; null for every other model. */ + instantxControlMode: number | null; + unresolved: ControlModelUnresolvedReason | null; +} + +type SupportsKind = (kind: ControlAdapterKind) => boolean; + +interface CandidateSets { + usable: ModelConfig[]; + dedicated: ModelConfig[]; + unions: ModelConfig[]; +} + +/** A model a layer on `base` can run: same base, a control kind the base supports, and nothing about it refused. */ +const usableKind = (model: ModelConfig, base: string | null, supportsKind: SupportsKind): ControlAdapterKind | null => { + if (base !== null && model.base !== base) { + return null; + } + const kind = deriveControlKind(model); + return kind && supportsKind(kind) && getControlModelUnusableReason(model, kind) === null ? kind : null; +}; + +const usableModels = (base: string | null, models: readonly ModelConfig[], supportsKind: SupportsKind) => + models.filter((model) => usableKind(model, base, supportsKind) !== null); + +const collectCandidates = ( + type: ControlType, + family: ControlFamily, + base: string | null, + models: readonly ModelConfig[], + supportsKind: SupportsKind +): CandidateSets => { + const usable = usableModels(base, models, supportsKind); + const classes = new Map(usable.map((model) => [model, classifyControlModel(model)])); + const byFamily = usable.filter((model) => classes.get(model)!.family === family); + return { + dedicated: byFamily.length > 0 ? byFamily : usable.filter((model) => classes.get(model)!.purpose === type), + unions: usable.filter((model) => { + const modelClass = classes.get(model)!; + return modelClass.union && modelClass.covers.includes(family); + }), + usable, + }; +}; + +/** + * Which model drives a control layer: an explicit pick for this base, the remembered pick for (base, type), the + * only model made for the type, a choice between several, the one covering union model, or none. A pick made for + * another base is ignored, not cleared, so switching back restores it. An untyped layer runs its stored adapter as + * it is; its `usable` still lists every model it could run on the base, for a direct pick. + */ +export const resolveControlModel = (input: { + layer: Pick; + base: string | null; + models: readonly ModelConfig[]; + remembered: ControlModelMemory; + supportsKind: SupportsKind; +}): ControlModelResolution => { + const { base, layer, models, remembered, supportsKind } = input; + const picked = layer.adapter.model ? (models.find((model) => model.key === layer.adapter.model) ?? null) : null; + const type = layer.controlType; + + if (!type) { + return { + candidates: [], + familyMatch: false, + how: 'none', + instantxControlMode: null, + kind: layer.adapter.kind, + model: picked, + unresolved: layer.adapter.model ? (picked ? null : 'missing_model') : 'missing_type', + usable: usableModels(base, models, supportsKind), + }; + } + + const family = getDetectorFamily(type, layer.detection?.type ?? null); + const { dedicated, unions, usable } = collectCandidates(type, family, base, models, supportsKind); + const candidates = [...dedicated, ...unions]; + const settle = ( + how: ControlModelResolution['how'], + model: ModelConfig | null, + unresolved: ControlModelUnresolvedReason | null = null + ): ControlModelResolution => { + const modelClass = model ? classifyControlModel(model) : null; + return { + candidates, + familyMatch: + modelClass !== null && + (modelClass.family === family || (modelClass.union && modelClass.covers.includes(family))), + how, + instantxControlMode: + base === 'flux' && modelClass?.union ? (CONTROL_TYPE_SPECS[type].instantxMode[family] ?? null) : null, + kind: (model ? deriveControlKind(model) : null) ?? getDefaultControlAdapterKind(base), + model, + unresolved, + usable, + }; + }; + + if (picked && usableKind(picked, base, supportsKind)) { + return settle('explicit', picked); + } + const rememberedKey = base === null ? undefined : remembered[controlModelMemoryKey(base, type)]; + const rememberedModel = rememberedKey ? candidates.find((model) => model.key === rememberedKey) : undefined; + if (rememberedModel) { + return settle('remembered', rememberedModel); + } + if (dedicated.length === 1) { + return settle('only', dedicated[0]!); + } + if (dedicated.length > 1) { + return settle('ask', null, 'choose_model'); + } + if (unions.length === 1) { + return settle('union', unions[0]!); + } + if (unions.length > 1) { + return settle('ask', null, 'choose_model'); + } + return settle('missing', null, usable.length === 0 ? 'no_control_models' : 'no_model_for_type'); +}; + +/** + * Whether picking `model` should become the remembered pick for the layer's base and type. Resolution honours a + * remembered pick only among `candidates`, and it then drives every unpicked layer of that base and type, so a model + * made for another family, or one nothing identifies, stays this layer's own pick. + */ +export const isRememberableControlPick = (resolution: ControlModelResolution, model: ModelConfig): boolean => + resolution.candidates.some((candidate) => candidate.key === model.key); + +/** How a control type would resolve on a fresh layer, for the Type menu's hints. */ +export interface ControlTypeAvailability { + type: ControlType; + /** `ready`: one model made for it. `ask`: several. `union`: only union models cover it. `missing`: nothing. */ + state: 'ready' | 'ask' | 'union' | 'missing'; + /** Models behind the state: those made for the type, or the covering unions. */ + count: number; +} + +export const getTypeAvailability = ( + base: string | null, + models: readonly ModelConfig[], + supportsKind: SupportsKind +): readonly ControlTypeAvailability[] => + CONTROL_TYPES.map((type) => { + const { dedicated, unions } = collectCandidates( + type, + CONTROL_TYPE_SPECS[type].primaryFamily, + base, + models, + supportsKind + ); + if (dedicated.length > 0) { + return { count: dedicated.length, state: dedicated.length === 1 ? 'ready' : 'ask', type }; + } + return unions.length > 0 ? { count: unions.length, state: 'union', type } : { count: 0, state: 'missing', type }; + }); diff --git a/invokeai/frontend/webv2/src/workbench/controlTypes.test.ts b/invokeai/frontend/webv2/src/workbench/controlTypes.test.ts new file mode 100644 index 00000000000..6e05904c272 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/controlTypes.test.ts @@ -0,0 +1,158 @@ +import type { ModelConfig } from '@features/models'; + +import { CONTROL_TYPES } from '@workbench/canvas-engine/api'; +import { CONTROL_FILTERS } from '@workbench/canvas-operations/filterGraphs'; +import { describe, expect, it } from 'vitest'; + +import { + CONTROL_TYPE_SPECS, + classifyControlModel, + controlTypeForDetector, + getDetectorFamily, + PREPROCESSOR_FAMILY, + type ControlModelClass, +} from './controlTypes'; + +describe('CONTROL_TYPE_SPECS', () => { + it('covers every control type with detectors the filter catalog runs', () => { + const filterTypes = new Set(CONTROL_FILTERS.map((filter) => filter.type)); + + expect(Object.keys(CONTROL_TYPE_SPECS).sort()).toEqual([...CONTROL_TYPES].sort()); + for (const [type, spec] of Object.entries(CONTROL_TYPE_SPECS)) { + expect(spec.id).toBe(type); + expect(spec.detectors.filter((detector) => !filterTypes.has(detector.type))).toEqual([]); + if (spec.defaultDetector !== null) { + expect(spec.detectors.map((detector) => detector.type)).toContain(spec.defaultDetector); + } + } + }); + + it('maps every generic detector to the family its spec names, so model and detector families agree', () => { + for (const spec of Object.values(CONTROL_TYPE_SPECS)) { + for (const detector of spec.detectors.filter((entry) => !entry.settings)) { + expect(PREPROCESSOR_FAMILY[detector.type], detector.type).toBe(detector.family); + } + } + }); +}); + +describe('getDetectorFamily', () => { + it.each([ + ['depth', 'normal_map', 'normal'], + ['depth', 'depth_anything_depth_estimation', 'depth'], + ['depth', null, 'depth'], + ['edges', 'mlsd_detection', 'mlsd'], + ['sketch', 'pidi_edge_detection', 'scribble'], + ['sketch', null, 'scribble'], + ['tile', null, 'tile'], + ['pose', 'canny_edge_detection', 'openpose'], + ] as const)('%s with %s produces %s', (type, detector, family) => { + expect(getDetectorFamily(type, detector)).toBe(family); + }); +}); + +describe('controlTypeForDetector', () => { + it.each([ + [{ settings: { scribble: true }, type: 'pidi_edge_detection' }, 'sketch'], + [{ settings: { scribble: false }, type: 'pidi_edge_detection' }, 'softedge'], + [{ type: 'pidi_edge_detection' }, 'softedge'], + [{ settings: {}, type: 'scribble_edge_detection' }, 'sketch'], + [{ settings: {}, type: 'normal_map' }, 'depth'], + [{ settings: {}, type: 'mediapipe_face_detection' }, 'pose'], + [{ settings: {}, type: 'content_shuffle' }, 'color'], + [{ settings: {}, type: 'img_blur' }, null], + ])('%j belongs to %s', (detector, type) => { + expect(controlTypeForDetector(detector)).toBe(type); + }); +}); + +const model = ( + name: string, + base: string, + type = 'controlnet', + preprocessor: string | null = null +): Pick => + ({ base, default_settings: preprocessor ? { preprocessor } : null, name, type }) as ModelConfig; + +const made = (purpose: ControlModelClass['purpose'], family: ControlModelClass['family']): ControlModelClass => ({ + covers: [], + family, + purpose, + union: false, +}); + +const FUN_COVERS = ['canny', 'mlsd', 'softedge', 'depth', 'openpose']; + +describe('classifyControlModel', () => { + it.each([ + [ + 'legacy preprocessor id', + model('My ControlNet', 'sdxl', 'controlnet', 'depth_anything_image_processor'), + made('depth', 'depth'), + ], + [ + 'legacy zoe id', + model('My ControlNet', 'sdxl', 'controlnet', 'zoe_depth_image_processor'), + made('depth', 'depth'), + ], + ['node id', model('My ControlNet', 'sdxl', 'controlnet', 'normal_map'), made('depth', 'normal')], + ['legacy mlsd id', model('Lines', 'sd-1', 'controlnet', 'mlsd_image_processor'), made('edges', 'mlsd')], + ['tile preprocessor', model('Custom', 'sdxl', 'controlnet', 'tile'), made('tile', 'tile')], + ['tile name', model('xinsir-controlnet-tile-sdxl', 'sdxl'), made('tile', 'tile')], + [ + 'scribble name over the backend lineart guess', + model('control_v11p_sd15_scribble', 'sd-1', 'controlnet', 'lineart_image_processor'), + made('sketch', 'scribble'), + ], + [ + 'anime line art over the preprocessor', + model('control_v11p_sd15s2_lineart_anime', 'sd-1', 'controlnet', 'lineart_image_processor'), + made('lineart', 'lineart_anime'), + ], + ['face name', model('controlnet mediapipe face', 'sd-1'), made('pose', 'face')], + ['name word', model('control_v11p_sd15_canny', 'sd-1'), made('edges', 'canny')], + ['name word prefix', model('control_v11p_sd15_normalbae', 'sd-1'), made('depth', 'normal')], + ['camelCase name', model('controlnetSoftEdge', 'sdxl'), made('softedge', 'softedge')], + ['openpose name', model('controlnet-openpose-sdxl-1.0', 'sdxl'), made('pose', 'openpose')], + ['no match inside unrelated words', model('cached-surface-model', 'sdxl'), made(null, null)], + ['unknown model', model('my-sdxl-control', 'sdxl'), made(null, null)], + ['T2I-Adapter', model('t2iadapter_canny_sd15v2', 'sd-1', 't2i_adapter'), made('edges', 'canny')], + ['Control LoRA', model('flux1-depth-dev-lora', 'flux', 'control_lora'), made('depth', 'depth')], + ['Anima LLLite', model('anima-lllite-sketch', 'anima', 'controlnet'), made('sketch', 'scribble')], + [ + 'saved preprocessor over a weak name word', + model('depth-something', 'sdxl', 'controlnet', 'canny_image_processor'), + made('edges', 'canny'), + ], + ] as const)('%s', (_label, input, expected) => { + expect(classifyControlModel(input)).toEqual(expected); + }); + + it.each([ + [ + 'an SDXL union, covering the xinsir ProMax conditions', + model('xinsir-controlnet-union-sdxl-promax', 'sdxl'), + ['openpose', 'depth', 'canny', 'mlsd', 'lineart', 'lineart_anime', 'scribble', 'softedge', 'normal', 'tile'], + ], + ['an SD1.5 union, offered only as an explicit pick', model('controlnet-union-sd15', 'sd-1'), []], + [ + 'a FLUX union, covering the InstantX modes', + model('FLUX.1-dev-Controlnet-Union-Pro', 'flux', 'controlnet', 'canny_image_processor'), + ['canny', 'tile', 'depth', 'openpose'], + ], + ['a Z-Image Fun union whatever its name', model('Z-Image-Turbo-Fun-Controlnet', 'z-image'), FUN_COVERS], + ['a Qwen-Image Fun union', model('Qwen-Image-2512-Fun-Controlnet-Union', 'qwen-image'), FUN_COVERS], + ['a FLUX.2 Fun union', model('FLUX.2-dev-Fun-Controlnet', 'flux2'), FUN_COVERS], + ] as const)('classifies %s', (_label, input, covers) => { + expect(classifyControlModel(input)).toEqual({ covers, family: null, purpose: null, union: true }); + }); + + it('classifies a replaced model config afresh', () => { + const config = model('control_v11p_sd15_canny', 'sd-1'); + expect(classifyControlModel(config).family).toBe('canny'); + + // A renamed model arrives as a new config object; the class must follow the new name. + expect(classifyControlModel({ ...config, name: 'control_v11f1p_sd15_depth' }).family).toBe('depth'); + expect(classifyControlModel(config).family).toBe('canny'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/controlTypes.ts b/invokeai/frontend/webv2/src/workbench/controlTypes.ts new file mode 100644 index 00000000000..a81e54756bd --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/controlTypes.ts @@ -0,0 +1,328 @@ +import type { ModelConfig } from '@features/models'; +import type { ControlType } from '@workbench/canvas-engine/api'; + +/** + * What a control map encodes. A control type groups the families its detectors produce; a control model is trained + * on one family (or, for union models, several). + */ +export type ControlFamily = + | 'openpose' + | 'face' + | 'depth' + | 'normal' + | 'canny' + | 'mlsd' + | 'softedge' + | 'lineart' + | 'lineart_anime' + | 'scribble' + | 'color' + | 'shuffle' + | 'tile'; + +/** A detector a control type offers: a filter type from `CONTROL_FILTERS`, with settings it fixes. */ +export interface ControlDetectorSpec { + type: string; + family: ControlFamily; + /** Settings this entry requires; a detector input matches it only when every one is equal. */ + settings?: Record; +} + +export interface ControlTypeSpec { + id: ControlType; + detectors: readonly ControlDetectorSpec[]; + /** The detector a new image starts with; null when the image is usually already the map. */ + defaultDetector: string | null; + /** The family of a map that skipped detection. */ + primaryFamily: ControlFamily; + /** FLUX InstantX Union `instantx_control_mode` per family; absent families run without a mode. */ + instantxMode: Partial>; +} + +export const CONTROL_TYPE_SPECS: Readonly> = { + color: { + defaultDetector: 'color_map', + detectors: [ + { family: 'color', type: 'color_map' }, + { family: 'shuffle', type: 'content_shuffle' }, + ], + id: 'color', + instantxMode: {}, + primaryFamily: 'color', + }, + depth: { + defaultDetector: 'depth_anything_depth_estimation', + detectors: [ + { family: 'depth', type: 'depth_anything_depth_estimation' }, + { family: 'normal', type: 'normal_map' }, + ], + id: 'depth', + instantxMode: { depth: 2 }, + primaryFamily: 'depth', + }, + edges: { + defaultDetector: 'canny_edge_detection', + detectors: [ + { family: 'canny', type: 'canny_edge_detection' }, + { family: 'mlsd', type: 'mlsd_detection' }, + ], + id: 'edges', + instantxMode: { canny: 0 }, + primaryFamily: 'canny', + }, + lineart: { + defaultDetector: 'lineart_edge_detection', + detectors: [ + { family: 'lineart', type: 'lineart_edge_detection' }, + { family: 'lineart_anime', type: 'lineart_anime_edge_detection' }, + ], + id: 'lineart', + instantxMode: {}, + primaryFamily: 'lineart', + }, + pose: { + defaultDetector: 'dw_openpose_detection', + detectors: [ + { family: 'openpose', type: 'dw_openpose_detection' }, + { family: 'face', type: 'mediapipe_face_detection' }, + ], + id: 'pose', + instantxMode: { openpose: 4 }, + primaryFamily: 'openpose', + }, + // Sketches are usually drawn as the map already, so a new sketch layer skips detection. + sketch: { + defaultDetector: null, + detectors: [ + { family: 'scribble', type: 'scribble_edge_detection' }, + { family: 'scribble', settings: { scribble: true }, type: 'pidi_edge_detection' }, + ], + id: 'sketch', + instantxMode: {}, + primaryFamily: 'scribble', + }, + softedge: { + defaultDetector: 'hed_edge_detection', + detectors: [ + { family: 'softedge', type: 'hed_edge_detection' }, + { family: 'softedge', type: 'pidi_edge_detection' }, + ], + id: 'softedge', + instantxMode: {}, + primaryFamily: 'softedge', + }, + tile: { + defaultDetector: null, + detectors: [], + id: 'tile', + instantxMode: { tile: 1 }, + primaryFamily: 'tile', + }, +}; + +const settingsMatch = (required: Record, actual: Record | undefined): boolean => + Object.entries(required).every(([key, value]) => actual?.[key] === value); + +/** The family a type's detector produces; no detector (or one the type does not list) means its primary family. */ +export const getDetectorFamily = (type: ControlType, detectorType: string | null): ControlFamily => { + const spec = CONTROL_TYPE_SPECS[type]; + return spec.detectors.find((detector) => detector.type === detectorType)?.family ?? spec.primaryFamily; +}; + +/** + * The control type a detector run belongs to, for maps made outside a control layer (a raster filter saved as + * control). An entry that fixes settings beats a generic one, so PiDi with `scribble` is a sketch. + */ +export const controlTypeForDetector = (detector: { + type: string; + settings?: Record; +}): ControlType | null => { + let generic: ControlType | null = null; + for (const spec of Object.values(CONTROL_TYPE_SPECS)) { + for (const entry of spec.detectors) { + if (entry.type !== detector.type) { + continue; + } + if (entry.settings) { + if (settingsMatch(entry.settings, detector.settings)) { + return spec.id; + } + } else { + generic ??= spec.id; + } + } + } + return generic; +}; + +/** + * Model `default_settings.preprocessor` values mapped to the family they produce: the legacy `*_image_processor` + * ids older installs carry, the filter node ids the model manager saves today, and `tile`. + */ +export const PREPROCESSOR_FAMILY: Readonly> = { + canny_edge_detection: 'canny', + canny_image_processor: 'canny', + color_map: 'color', + color_map_image_processor: 'color', + content_shuffle: 'shuffle', + content_shuffle_image_processor: 'shuffle', + depth_anything_depth_estimation: 'depth', + depth_anything_image_processor: 'depth', + dw_openpose_detection: 'openpose', + dw_openpose_image_processor: 'openpose', + hed_edge_detection: 'softedge', + hed_image_processor: 'softedge', + lineart_anime_edge_detection: 'lineart_anime', + lineart_anime_image_processor: 'lineart_anime', + lineart_edge_detection: 'lineart', + lineart_image_processor: 'lineart', + mediapipe_face_detection: 'face', + mediapipe_face_processor: 'face', + midas_depth_image_processor: 'depth', + mlsd_detection: 'mlsd', + mlsd_image_processor: 'mlsd', + normal_map: 'normal', + normalbae_image_processor: 'normal', + openpose_image_processor: 'openpose', + pidi_edge_detection: 'softedge', + pidi_image_processor: 'softedge', + scribble_edge_detection: 'scribble', + tile: 'tile', + zoe_depth_image_processor: 'depth', +}; + +const FAMILY_PURPOSE: Readonly> = { + canny: 'edges', + color: 'color', + depth: 'depth', + face: 'pose', + lineart: 'lineart', + lineart_anime: 'lineart', + mlsd: 'edges', + normal: 'depth', + openpose: 'pose', + scribble: 'sketch', + shuffle: 'color', + softedge: 'softedge', + tile: 'tile', +}; + +/** What a control model was trained for, as far as its name and saved preprocessor tell. */ +export interface ControlModelClass { + /** The control type it is made for; null for union models and models nothing identifies. */ + purpose: ControlType | null; + family: ControlFamily | null; + union: boolean; + /** Families a union model handles without an explicit pick. */ + covers: readonly ControlFamily[]; +} + +/** + * The Fun-ControlNet-Union models these bases install as `controlnet`. Both coverage lists follow the model cards' + * advertised conditions and still need confirming against them. + */ +const FUN_UNION_BASES: ReadonlySet = new Set(['z-image', 'qwen-image', 'flux2']); +const FUN_UNION_COVERS: readonly ControlFamily[] = ['canny', 'mlsd', 'softedge', 'depth', 'openpose']; +/** InstantX FLUX.1 Union modes the control types map onto. */ +const FLUX_UNION_COVERS: readonly ControlFamily[] = ['canny', 'tile', 'depth', 'openpose']; +/** The xinsir SDXL Union (ProMax) conditions; upscale also takes an SDXL union as its tile ControlNet. */ +const SDXL_UNION_COVERS: readonly ControlFamily[] = [ + 'openpose', + 'depth', + 'canny', + 'mlsd', + 'lineart', + 'lineart_anime', + 'scribble', + 'softedge', + 'normal', + 'tile', +]; +const UNION_COVERS_BY_BASE: Readonly> = { + flux: FLUX_UNION_COVERS, + sdxl: SDXL_UNION_COVERS, +}; + +/** Lowercase name words, split at punctuation and camelCase boundaries. */ +const nameTokens = (name: string): string[] => + name + .replace(/([a-z0-9])([A-Z])/g, '$1 $2') + .toLowerCase() + .split(/[^a-z0-9]+/) + .filter(Boolean); + +/** Prefix matching catches `normalbae` and `softedge` without matching inside unrelated words. */ +const hasToken = (tokens: readonly string[], ...keywords: string[]): boolean => + tokens.some((token) => keywords.some((keyword) => token.startsWith(keyword))); + +/** Names that outrank a saved preprocessor: the backend guesses `lineart` for scribble models, for one. */ +const strongNameFamily = (tokens: readonly string[]): ControlFamily | null => { + if (hasToken(tokens, 'tile')) { + return 'tile'; + } + if (hasToken(tokens, 'scribble', 'sketch')) { + return 'scribble'; + } + if (hasToken(tokens, 'lineart') && hasToken(tokens, 'anime')) { + return 'lineart_anime'; + } + if (hasToken(tokens, 'mediapipe', 'face')) { + return 'face'; + } + return null; +}; + +const WEAK_NAME_FAMILIES: readonly (readonly [ControlFamily, readonly string[]])[] = [ + ['canny', ['canny']], + ['depth', ['depth']], + ['openpose', ['openpose', 'pose']], + ['softedge', ['hed', 'softedge', 'soft']], + ['lineart', ['lineart']], + ['mlsd', ['mlsd']], + ['normal', ['normal']], + ['color', ['color']], + ['shuffle', ['shuffle']], +]; + +const weakNameFamily = (tokens: readonly string[]): ControlFamily | null => + WEAK_NAME_FAMILIES.find(([, keywords]) => hasToken(tokens, ...keywords))?.[0] ?? null; + +const preprocessorFamily = (model: Pick): ControlFamily | null => { + const preprocessor = model.default_settings?.preprocessor; + return preprocessor && Object.hasOwn(PREPROCESSOR_FAMILY, preprocessor) ? PREPROCESSOR_FAMILY[preprocessor]! : null; +}; + +type ClassifiableModel = Pick; + +const classify = (model: ClassifiableModel): ControlModelClass => { + const tokens = nameTokens(model.name); + if (model.type === 'controlnet' && FUN_UNION_BASES.has(model.base)) { + return { covers: FUN_UNION_COVERS, family: null, purpose: null, union: true }; + } + if (hasToken(tokens, 'union')) { + // Other bases' union models need the backend to expose their control types; until then they are explicit picks. + return { covers: UNION_COVERS_BY_BASE[model.base] ?? [], family: null, purpose: null, union: true }; + } + const family = strongNameFamily(tokens) ?? preprocessorFamily(model) ?? weakNameFamily(tokens); + return { covers: [], family, purpose: family ? FAMILY_PURPOSE[family] : null, union: false }; +}; + +/** + * Resolution, row renders and readiness ticks classify the same model configs over and over. Configs are replaced, + * never mutated, so a config object keys its class for as long as anything holds it. + */ +const classCache = new WeakMap(); + +/** + * Classify a control model: union models first (a base's Fun union; a FLUX or SDXL union, covering its known + * conditions; any other `union` name, covering nothing), then strong name words, the saved preprocessor, and the + * remaining name words. + */ +export const classifyControlModel = (model: ClassifiableModel): ControlModelClass => { + let modelClass = classCache.get(model); + if (!modelClass) { + modelClass = classify(model); + classCache.set(model, modelClass); + } + return modelClass; +}; diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/WorkbenchHotkeyRuntime.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/hotkeys/WorkbenchHotkeyRuntime.browser.test.tsx index 2ec6ba7df28..1b34d91a1f3 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/WorkbenchHotkeyRuntime.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/WorkbenchHotkeyRuntime.browser.test.tsx @@ -8,6 +8,7 @@ import { ChakraProvider, createListCollection, Menu, Portal, Select } from '@cha import { PromptTextarea } from '@features/generation/components'; import { adjustFocusedPromptAttention } from '@features/generation/react'; import { MenuContent } from '@platform/ui/Menu'; +import { SegmentedControl } from '@platform/ui/SegmentedControl'; import { AppToaster, createActionToast, toaster } from '@platform/ui/toaster'; import { system } from '@theme/system'; import { createExtensionRegistry } from '@workbench/extensions/extensionRegistry'; @@ -514,3 +515,69 @@ describe('WorkbenchHotkeyRuntime and prompt weighting', () => { expect(textarea.value).toBe('a cat'); }); }); + +const VIEW_OPTIONS = [ + { label: 'Original', value: 'original' }, + { label: 'Map', value: 'map' }, + { label: 'Overlay', value: 'overlay' }, +]; + +describe('WorkbenchHotkeyRuntime and a focused segmented control', () => { + let controlRoot: Root | null = null; + + afterEach(async () => { + await act(() => controlRoot?.unmount()); + controlRoot = null; + }); + + const Segments = () => { + const [value, setValue] = useState('map'); + return ( + + ); + }; + + it('leaves arrow keys to the radio group while Gallery, which binds them, holds focus', async () => { + // Gallery's grid binds the plain arrows to its own navigation. + contribute('gallery.galleryNavRight', 'arrowright', 'focused-region', { + instanceId: 'gallery', + projectId: projectId(), + region: 'right', + typeId: 'gallery', + }); + await pressOn('right'); + const controlHost = document.createElement('div'); + host!.append(controlHost); + controlRoot = createRoot(controlHost); + await act(() => { + controlRoot?.render( + + + + ); + }); + const radio = (value: string) => controlHost.querySelector(`input[value="${value}"]`)!; + const prevented: boolean[] = []; + // Registered after the runtime's listener, so it sees the press once the runtime has handled it. + const record = (event: KeyboardEvent) => prevented.push(event.defaultPrevented); + window.addEventListener('keydown', record); + + try { + await act(async () => { + radio('map').focus(); + await userEvent.keyboard('{ArrowRight}'); + }); + } finally { + window.removeEventListener('keydown', record); + } + + expect(prevented).toEqual([false]); + expect(ran).toEqual([]); + expect(radio('overlay').checked).toBe(true); + expect(document.activeElement).toBe(radio('overlay')); + + // Outside the radio group the same press is Gallery's. + await pressKey('ArrowRight', host!.querySelector('[data-testid="right"]')!); + expect(ran).toEqual(['gallery.galleryNavRight']); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.browser.test.tsx index 8fb25c1da56..80d3dfe19ab 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/firstPartyCommands.browser.test.tsx @@ -80,9 +80,7 @@ it('words a blocked control layer from the locale when Invoke runs from its hotk }); const [args] = submitActiveInvocation.mock.calls[0] ?? []; - expect( - args?.formatControlLayerError({ code: 'switch_adapter_kind', layerName: 'Sketch', suggestedKind: 'anima_lllite' }) - ).toBe( - 'Control layer "Sketch": Switch the adapter type to Anima ControlNet-LLLite to use this layer with the selected model.' + expect(args?.formatControlLayerError({ code: 'switch_adapter_kind', layerName: 'Sketch' })).toBe( + 'Control layer "Sketch": Choose a control type so a model for the selected main model can be picked.' ); }); diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/keys.browser.test.ts b/invokeai/frontend/webv2/src/workbench/hotkeys/keys.browser.test.ts index 63a17ed8e4b..ccf3d9366ec 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/keys.browser.test.ts +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/keys.browser.test.ts @@ -1,6 +1,6 @@ import { afterEach, describe, expect, it } from 'vitest'; -import { isEditableHotkeyTarget } from './keys'; +import { isEditableHotkeyTarget, isNativeControlKeyPress } from './keys'; describe('isEditableHotkeyTarget', () => { const host = document.createElement('div'); @@ -34,3 +34,40 @@ describe('isEditableHotkeyTarget', () => { expect(isEditableHotkeyTarget(null)).toBe(false); }); }); + +describe('isNativeControlKeyPress', () => { + const host = document.createElement('div'); + + document.body.append(host); + + afterEach(() => { + host.replaceChildren(); + }); + + const press = (html: string, init: KeyboardEventInit): boolean => { + host.innerHTML = html; + const event = new KeyboardEvent('keydown', init); + host.querySelector('[data-target]')!.dispatchEvent(event); + + return isNativeControlKeyPress(event); + }; + + it('leaves arrows, Home and End to a focused radio group or slider', () => { + expect(press('', { key: 'ArrowRight' })).toBe(true); + expect(press('', { key: 'Home' })).toBe(true); + expect(press('', { key: 'ArrowUp' })).toBe(true); + expect(press('
x
', { key: 'End' })).toBe(true); + expect(press('
', { key: 'ArrowLeft' })).toBe(true); + }); + + it('leaves other keys and chords to shortcuts', () => { + expect(press('', { key: 'a' })).toBe(false); + expect(press('', { ctrlKey: true, key: 'ArrowRight' })).toBe(false); + expect(press('', { altKey: true, key: 'ArrowRight' })).toBe(false); + }); + + it('is not about text fields, which isEditableHotkeyTarget covers', () => { + expect(press('', { key: 'ArrowRight' })).toBe(false); + expect(press('', { key: 'ArrowRight' })).toBe(false); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/keys.ts b/invokeai/frontend/webv2/src/workbench/hotkeys/keys.ts index 339b6f5f23e..8dc6b016478 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/keys.ts +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/keys.ts @@ -257,6 +257,23 @@ const NON_TEXT_INPUT_TYPES = new Set([ 'submit', ]); +/** What a focused radio group or slider does with a key itself: move its selection or value. */ +const NATIVE_CONTROL_KEYS = new Set(['ArrowDown', 'ArrowLeft', 'ArrowRight', 'ArrowUp', 'End', 'Home']); + +/** + * Whether a press is the focused radio group's or slider's own navigation (arrows, Home, End without Ctrl, Meta or + * Alt). These controls take no typed text, so {@link isEditableHotkeyTarget} lets chords through from them, but a + * plain shortcut on the same key (Gallery's arrow navigation) would prevent the default and freeze the control. + */ +export const isNativeControlKeyPress = (event: KeyboardEvent): boolean => + !event.ctrlKey && + !event.metaKey && + !event.altKey && + NATIVE_CONTROL_KEYS.has(event.key) && + typeof Element !== 'undefined' && + event.target instanceof Element && + event.target.closest('input[type="radio"], input[type="range"], [role="radiogroup"], [role="slider"]') !== null; + export const isEditableHotkeyTarget = (target: EventTarget | null): boolean => { if (typeof HTMLElement === 'undefined') { return false; diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.browser.test.ts b/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.browser.test.ts index 8a52b280b28..5f5adeeaae3 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.browser.test.ts +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.browser.test.ts @@ -131,3 +131,61 @@ describe('resolveHotkey inside an open popup', () => { ); }); }); + +describe('resolveHotkey on a focused radio group or slider', () => { + const host = document.createElement('div'); + + document.body.append(host); + + afterEach(() => { + host.replaceChildren(); + }); + + const context: HotkeyContext = { + activeInstanceId: 'gallery', + activeWidgetTypeId: 'gallery', + focusedRegion: 'right', + isModalPresent: false, + projectId: 'project-1', + }; + // Gallery's arrow navigation, as its grid registers it. + const galleryRight: RegisteredHotkey = { + category: 'gallery', + commandId: 'gallery.galleryNavRight', + defaultKeys: ['arrowright'], + id: 'gallery.galleryNavRight', + implemented: true, + keys: ['arrowright'], + scope: { kind: 'widget', typeIds: ['gallery'] }, + title: 'Navigate right', + }; + const editableRight: RegisteredHotkey = { + allowInEditable: true, + category: 'app', + commandId: 'editable', + defaultKeys: ['arrowright'], + id: 'editable', + implemented: true, + keys: ['arrowright'], + scope: { kind: 'global' }, + title: 'Editable', + }; + + const pressRightIn = (html: string, hotkey: RegisteredHotkey) => { + host.innerHTML = html; + const event = new KeyboardEvent('keydown', { bubbles: true, key: 'ArrowRight' }); + host.querySelector('[data-target]')!.dispatchEvent(event); + + return resolveHotkey({ context, event, hotkeys: [hotkey], matchedKey: 'arrowright' })?.commandId ?? null; + }; + + it('leaves the arrow to the radio group instead of moving the gallery selection', () => { + expect(pressRightIn('', galleryRight)).toBeNull(); + expect(pressRightIn('', galleryRight)).toBeNull(); + expect(pressRightIn('', galleryRight)).toBe('gallery.galleryNavRight'); + }); + + it('still resolves a binding that opts into editable targets', () => { + expect(pressRightIn('', editableRight)).toBe('editable'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.ts b/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.ts index 5419a5d4c63..bc3d5f5ac74 100644 --- a/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.ts +++ b/invokeai/frontend/webv2/src/workbench/hotkeys/resolve.ts @@ -1,6 +1,6 @@ import type { HotkeyContext, RegisteredHotkey } from './types'; -import { IS_MAC_OS, isEditableHotkeyTarget, normalizeHotkeyString } from './keys'; +import { IS_MAC_OS, isEditableHotkeyTarget, isNativeControlKeyPress, normalizeHotkeyString } from './keys'; /** * The keys a binding presses on this platform: `mod` is Cmd on macOS and Ctrl elsewhere. The runtime fires the first @@ -90,12 +90,13 @@ export const resolveHotkey = ({ } const pressed = toPlatformHotkey(matchedKey); - const isEditable = isEditableHotkeyTarget(event.target); + // A focused text field, radio group or slider keeps the press unless the binding opts into editable targets. + const isOwnedByTarget = isEditableHotkeyTarget(event.target) || isNativeControlKeyPress(event); return ( hotkeys .filter((hotkey) => hotkey.implemented !== false && hotkey.keys.some((key) => toPlatformHotkey(key) === pressed)) - .filter((hotkey) => hotkey.allowInEditable || !isEditable) + .filter((hotkey) => hotkey.allowInEditable || !isOwnedByTarget) .filter((hotkey) => isClaimedTarget(hotkey, event.target)) .filter((hotkey) => hotkey.allowInModal || !context.isModalPresent) .map((hotkey) => ({ hotkey, priority: getScopePriority(hotkey, context) })) diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.browser.test.tsx index c32649f0fcb..a467d6a3cd8 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.browser.test.tsx @@ -120,6 +120,7 @@ const createActions = (deleteItems: ImageActions['deleteItems']): ImageActions = recallVideoData: vi.fn(() => Promise.resolve()), selectForCompare: vi.fn(), sendToCanvas: vi.fn(() => Promise.resolve()), + sendImageAsControl: vi.fn(() => Promise.resolve()), setItemsStarred: vi.fn(() => Promise.resolve()), setImagesStarred: vi.fn(() => Promise.resolve()), savePromptAsTemplate: vi.fn(), @@ -474,6 +475,26 @@ describe('ImageContextMenu canvas destinations', () => { expect(calls[0]?.[0].map((entry) => entry.imageName)).toEqual(images.map((entry) => entry.imageName)); expect(actions.sendToCanvas).not.toHaveBeenCalled(); }); + + it('offers Use as Control beside Use as Reference Image and hands it the image', async () => { + const actions = createActions(vi.fn()); + await renderMenu(actions, [image('image-0.png')]); + const sequence = menuSequence(); + + expect(sequence[sequence.indexOf('Use as Reference Image') + 1]).toBe('widgets.gallery.itemActions.useAsControl'); + await interact(() => getMenuItem('widgets.gallery.itemActions.useAsControl').click()); + + expect(vi.mocked(actions.sendImageAsControl).mock.calls.map(([entry]) => entry.imageName)).toEqual(['image-0.png']); + }); + + it('leaves Use as Control out of a multi-image menu: one control layer takes one image', async () => { + await renderMenu( + createActions(vi.fn()), + Array.from({ length: 2 }, (_, index) => image(`image-${index}.png`)) + ); + + expect(menuSequence()).not.toContain('widgets.gallery.itemActions.useAsControl'); + }); }); describe('ImageContextMenu video placement', () => { diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.tsx b/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.tsx index a21af6291ba..c6f1ab1bbaa 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.tsx +++ b/invokeai/frontend/webv2/src/workbench/image-actions/ImageContextMenu.tsx @@ -47,6 +47,7 @@ import { ScanIcon, ScissorsIcon, ShuffleIcon, + SlidersHorizontalIcon, SproutIcon, StarIcon, Trash2Icon, @@ -769,6 +770,7 @@ const SingleImageMenuItems = ({ const handleSelectForCompare = useSelectForCompareHandler(actions, image); const handleSavePromptAsTemplate = useCallback(() => void actions.savePromptAsTemplate(image), [actions, image]); const handleUseAsReferenceImage = useUseAsReferenceImageHandler(actions, image); + const handleUseAsControl = useCallback(() => void actions.sendImageAsControl(image), [actions, image]); const handleLoadWorkflow = useCallback(() => void actions.loadImageWorkflow(image), [actions, image]); const { generation, widgets } = useWorkbenchCommands(); const openWidget = useOpenWorkbenchWidget(); @@ -868,6 +870,12 @@ const SingleImageMenuItems = ({ value="use-as-reference-image" onClick={handleUseAsReferenceImage} /> + ({ patchGalleryItemCaches: vi.fn((..._args: unknown[]) => vi.fn()), removeFromBoard: vi.fn(), reportError: vi.fn(), + revealLayerProperties: vi.fn(), + sendGalleryImageAsControl: vi.fn(), requestDeletionConfirmation: vi.fn(), resolveItem: vi.fn(), setStarred: vi.fn(), @@ -166,15 +168,22 @@ vi.mock('@platform/browser/downloadBlob', () => ({ vi.mock('@workbench/canvas-operations/api', async () => { // The notice mapper is pure; the rest of the canvas-operations surface stays doubled. - const { getCanvasImportNotice } = await import('@workbench/canvas-operations/canvasImportNotice'); + const { getCanvasImportNotice, getSendAsControlNotice } = + await import('@workbench/canvas-operations/canvasImportNotice'); return { createCanvasFromImages: (...args: unknown[]) => mocks.createCanvasFromImages(...args), getCanvasEngine: (...args: unknown[]) => mocks.getCanvasEngine(...args), getCanvasImportNotice, + getSendAsControlNotice, importGalleryImagesToCanvas: (...args: unknown[]) => mocks.importGalleryImagesToCanvas(...args), + sendGalleryImageAsControl: (...args: unknown[]) => mocks.sendGalleryImageAsControl(...args), }; }); +vi.mock('@workbench/widgets/layers/revealLayerProperties', () => ({ + useRevealLayerProperties: () => mocks.revealLayerProperties, +})); + vi.mock('@workbench/WorkbenchContext', () => { // The workbench store owns one commands object, so it is the same on every read. const commands = { @@ -451,6 +460,56 @@ describe('canvas destinations', () => { // The canvas the images went to is no longer on screen; the one that is must not be brought forward. expect(mocks.openWorkbenchWidget).not.toHaveBeenCalled(); }); + + it('names the control layer an image filled and shows its Properties', async () => { + mocks.getProject.mockImplementation((id: string) => makeMockProject(id)); + mocks.sendGalleryImageAsControl.mockResolvedValueOnce({ + layerId: 'control-1', + layerName: 'Depth 1', + status: 'filled', + }); + + await act(() => actionsRef.current!.sendImageAsControl(galleryImage('a.png'))); + + expect(mocks.sendGalleryImageAsControl).toHaveBeenCalledWith( + expect.objectContaining({ image: expect.objectContaining({ imageName: 'a.png' }) }) + ); + expect(mocks.notificationsAdd).toHaveBeenCalledWith({ + kind: 'success', + title: 'widgets.canvas.import.controlFilled', + }); + expect(mocks.revealLayerProperties).toHaveBeenCalledWith('control-1'); + // The menu has no trigger to restore focus to: the focusing open hands it to the Canvas first. + expect(mocks.openWorkbenchWidget).toHaveBeenCalledWith('canvas', { + preferredRegions: ['center'], + requireCenterView: true, + }); + expect(mocks.openWorkbenchWidget.mock.invocationCallOrder[0]).toBeLessThan( + mocks.revealLayerProperties.mock.invocationCallOrder[0]! + ); + }); + + it('reports a new control layer as an import, and a refusal without moving the layout', async () => { + mocks.getProject.mockImplementation((id: string) => makeMockProject(id)); + mocks.sendGalleryImageAsControl.mockResolvedValueOnce({ layerId: 'control-2', status: 'created' }); + + await act(() => actionsRef.current!.sendImageAsControl(galleryImage('a.png'))); + + expect(mocks.notificationsAdd).toHaveBeenLastCalledWith({ + kind: 'success', + title: 'widgets.canvas.import.success', + }); + expect(mocks.revealLayerProperties).toHaveBeenLastCalledWith('control-2'); + + mocks.revealLayerProperties.mockClear(); + mocks.openWorkbenchWidget.mockClear(); + mocks.sendGalleryImageAsControl.mockResolvedValueOnce({ status: 'blocked' }); + await act(() => actionsRef.current!.sendImageAsControl(galleryImage('a.png'))); + + expect(mocks.notificationsAdd).toHaveBeenLastCalledWith({ kind: 'info', title: 'widgets.canvas.import.blocked' }); + expect(mocks.revealLayerProperties).not.toHaveBeenCalled(); + expect(mocks.openWorkbenchWidget).not.toHaveBeenCalled(); + }); }); describe('compare selection handler', () => { diff --git a/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts b/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts index db73fb3c529..29cde6223d3 100644 --- a/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts +++ b/invokeai/frontend/webv2/src/workbench/image-actions/useImageActions.ts @@ -51,11 +51,14 @@ import { createCanvasFromImages, getCanvasImportNotice, getCanvasEngine, + getSendAsControlNotice, importGalleryImagesToCanvas, + sendGalleryImageAsControl, type GalleryCanvasImportDestination, } from '@workbench/canvas-operations/api'; import { useWorkbenchPreferenceSelector } from '@workbench/settings/store'; import { useOpenWorkbenchWidget } from '@workbench/useOpenWorkbenchWidget'; +import { useRevealLayerProperties } from '@workbench/widgets/layers/revealLayerProperties'; import { getProjectWidgetInstance, getProjectWidgetValues } from '@workbench/widgetState'; import { useWorkbenchCommands, useWorkbenchQueries, useWorkbenchSelector } from '@workbench/WorkbenchContext'; import { useCallback, useMemo } from 'react'; @@ -139,6 +142,8 @@ export interface ImageActions extends GalleryItemActions { savePromptAsTemplate: (image: GalleryImage) => Promise; selectForCompare: (image: GalleryImage) => void; sendToCanvas: (images: readonly GalleryImage[], destination: GalleryCanvasImportDestination) => Promise; + /** Fills the selected control layer when it has no image yet, else adds a control layer; shows its Properties. */ + sendImageAsControl: (image: GalleryImage) => Promise; setImagesStarred: (imageNames: string[], starred: boolean) => Promise; useAsReferenceImage: (image: GalleryImage) => void; useAsReferenceVideo: (item: GalleryVideoItem) => void; @@ -209,6 +214,7 @@ export const useImageActions = ({ requestDeletionConfirmation: RequestDeletionConfirmation; }): ImageActions => { const openWorkbenchWidget = useOpenWorkbenchWidget(); + const revealLayerProperties = useRevealLayerProperties(); const commands = useWorkbenchCommands(); const { gallery, generation, notifications } = commands; const queries = useWorkbenchQueries(); @@ -1131,6 +1137,45 @@ export const useImageActions = ({ }); } }, + sendImageAsControl: async (image) => { + const owner = captureAccountScope(); + + try { + const targetProjectId = projectId ?? queries.getSnapshot().activeProject.id; + const project = queries.getProject(targetProjectId); + const result = project + ? await sendGalleryImageAsControl({ + applyCanvasMutation: commands.canvas.apply, + engine: getCanvasEngine(project.id) ?? null, + getProject: queries.getProject, + image, + isActiveProject: queries.isActiveProject, + project, + }) + : ({ status: 'stale-project' } as const); + + assertAccountScopeCurrent(owner); + const notice = getSendAsControlNotice(result); + notifications.add({ kind: notice.kind, title: t(notice.titleKey, notice.options ?? {}) }); + + if ((result.status === 'filled' || result.status === 'created') && queries.isActiveProject(targetProjectId)) { + // The menu that asked has no trigger to hand focus back to, so the Canvas takes it, as for any import. + openWorkbenchWidget('canvas', { preferredRegions: ['center'], requireCenterView: true }); + revealLayerProperties(result.layerId); + } + } catch (error: unknown) { + if (!isAccountScopeCurrent(owner)) { + return; + } + + recordCanvasImportError({ + error, + localizedMessage: t('widgets.canvas.import.failed'), + notifications, + projectId, + }); + } + }, setImagesStarred: (imageNames, starred) => setItemsStarred( imageNames.map((name) => ({ kind: 'image', name })), @@ -1240,6 +1285,7 @@ export const useImageActions = ({ queryClient, queries, requestDeletionConfirmation, + revealLayerProperties, selectForCompare, supportedModels, t, diff --git a/invokeai/frontend/webv2/src/workbench/invocation.test.ts b/invokeai/frontend/webv2/src/workbench/invocation.test.ts index de38fa8a137..ae297aa9690 100644 --- a/invokeai/frontend/webv2/src/workbench/invocation.test.ts +++ b/invokeai/frontend/webv2/src/workbench/invocation.test.ts @@ -22,11 +22,27 @@ import { queryClient } from '@platform/query/client'; import { accountLifecycle, captureAccountScope } from '@platform/state/accountLifecycle'; import { beforeEach, describe, expect, it, vi } from 'vitest'; +import type * as settingsStoreModule from './settings/store'; + // Dynamic prompt expansion is the only transport `submitResolvedInvocation` touches. const parseDynamicPromptsMock = vi.hoisted(() => vi.fn()); vi.mock('@features/generation/data/promptUtilities', () => ({ parseDynamicPrompts: parseDynamicPromptsMock })); +// Remembered control-model picks, which readiness reads imperatively from the account preferences. +const preferences = vi.hoisted(() => ({ controlModelMemory: {} as Record })); + +vi.mock('./settings/store', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + getWorkbenchPreferences: () => ({ + ...actual.getWorkbenchPreferences(), + controlModelMemory: preferences.controlModelMemory, + }), + }; +}); + import { seedArchitectureCapabilities } from '@features/generation/core/architectureCapabilities.testing'; import { @@ -514,9 +530,9 @@ describe('resolveInvocationRoute — canvas source', () => { expect(route.validationReasons).toContain('The Canvas widget is not mounted in this project.'); }); - it('blocks a content-bearing control layer with no model once models are loaded', () => { + it('blocks a typed content-bearing control layer no model resolves once models are loaded', () => { const route = resolveInvocationRouteInput( - canvasInputFor({ layers: [modellessControlLayer()] }), + canvasInputFor({ layers: [{ ...modellessControlLayer(), controlType: 'depth', detection: null }] }), 'global', canvasRoute, loadedModels @@ -526,14 +542,43 @@ describe('resolveInvocationRoute — canvas source', () => { // Kept structured so the shell words it from the locale. expect(route.validationReasons).toContainEqual({ controlLayerIssue: { - code: 'missing_model', + code: 'no_control_models', layerId: expect.any(String), layerName: 'Control Layer 1', - suggestedKind: null, }, }); }); + it('does not block on a content-bearing layer with neither a type nor a model', () => { + const route = resolveInvocationRouteInput( + canvasInputFor({ layers: [modellessControlLayer()] }), + 'global', + canvasRoute, + loadedModels + ); + + expect(route.validationReasons).toEqual([]); + }); + + it('re-evaluates a choice between control models against the remembered pick', () => { + const lllite = (key: string) => + ({ base: 'anima', cond_in_channels: 3, key, name: key, type: 'controlnet' }) as unknown as ModelConfig; + const models = [...loadedModels, lllite('anima-depth-a'), lllite('anima-depth-b')]; + const input = canvasInputFor({ layers: [{ ...modellessControlLayer(), controlType: 'depth', detection: null }] }); + const codes = () => + resolveInvocationRouteInput(input, 'global', canvasRoute, models).validationReasons.map((reason) => + typeof reason === 'object' && 'controlLayerIssue' in reason ? reason.controlLayerIssue.code : reason + ); + + try { + expect(codes()).toEqual(['choose_model']); + preferences.controlModelMemory = { 'anima:depth': 'anima-depth-b' }; + expect(codes()).toEqual([]); + } finally { + preferences.controlModelMemory = {}; + } + }); + it('does not block content-less control layers or check layers before models load', () => { const emptyLayerRoute = resolveInvocationRouteInput( canvasInputFor({ layers: [modellessControlLayer({ bitmap: null, type: 'paint' })] }), diff --git a/invokeai/frontend/webv2/src/workbench/invocation.ts b/invokeai/frontend/webv2/src/workbench/invocation.ts index c0f8a58da5a..dd34e277217 100644 --- a/invokeai/frontend/webv2/src/workbench/invocation.ts +++ b/invokeai/frontend/webv2/src/workbench/invocation.ts @@ -38,8 +38,13 @@ import { } from '@workbench/canvas-engine/api'; import { isCompositableControlLayer, isCompositableRegionalGuidanceLayer } from '@workbench/canvasLayerContent'; -import { describeControlLayerIssue, getBlockingControlLayerIssues } from './controlLayerChecks'; +import { + createControlLayerCheckContext, + describeControlLayerIssue, + getBlockingControlLayerIssues, +} from './controlLayerChecks'; import { getActiveProjectGraph } from './projectWorkflows'; +import { getWorkbenchPreferences } from './settings/store'; import { getProjectWidgetValues } from './widgetState'; export interface InvocationSourceMeta { @@ -287,7 +292,7 @@ export const resolveInvocationRouteInput = ( const sourceId = route.sourceId; const destination = route.destination; const sourceWidgetId = sourceWidgetIds[sourceId]; - // Route-rendering surfaces subscribe to templates because readiness reads them imperatively. + // Route-rendering surfaces subscribe to templates and preferences because readiness reads them imperatively. const projectGraphReadiness = sourceId === 'workflow' ? getProjectGraphReadiness(input.projectGraph, getInvocationTemplatesSnapshot(), { @@ -353,6 +358,7 @@ export const resolveInvocationRouteInput = ( if (values && isSupportedGenerateModel(values.model)) { validationReasons.push( ...getBlockingControlLayerIssues({ + context: createControlLayerCheckContext(values.model.base, getWorkbenchPreferences().controlModelMemory), layers: input.canvasLayers, mainModel: values.model, models, diff --git a/invokeai/frontend/webv2/src/workbench/projects/README.md b/invokeai/frontend/webv2/src/workbench/projects/README.md index ea0c88aefa4..9b957447684 100644 --- a/invokeai/frontend/webv2/src/workbench/projects/README.md +++ b/invokeai/frontend/webv2/src/workbench/projects/README.md @@ -74,9 +74,9 @@ Cold-start offline editing not supported. When backend cannot load, unavailable New value in a closed canvas set, such as a control layer's adapter kind (`anima_lllite`), is added without raising canvas schema version or project's compatibility floor. Builds that know the value load and save it unchanged; builds that do not refuse the canvas instead of rewriting it, since coercing an unknown kind to a known one would replace the user's adapter on next autosave. In an older build the project is not opened and user sees that it "contains canvas data this version of Invoke cannot read"; saved copy is untouched and exportable, and a build that knows the value opens it as saved. Raise schema version instead when older builds could misread a document without refusing it. -Adjustment stacks are such a case: load drops a stack holding any entry it cannot parse, so an older build would load a document with a new adjustment type minus that layer's whole stack and autosave the loss. A new adjustment type therefore raises the canvas schema for documents that use it. A noise, clarity, texture or dehaze adjustment, enabled or not, on any layer or group at any depth of the live document or a saved snapshot, makes the canvas schema 5 (`canvasMigration.ts`, `writableCanvasSchemaVersion`); a document without one keeps its version. Adoption is sticky: removing the last such entry leaves the document at 5, matching the server's monotonic floor. A build whose maximum is 4 refuses the project at the server and the canvas at load as an unsupported version. +Adjustment stacks are such a case: load drops a stack holding any entry it cannot parse, so an older build would load a document with a new adjustment type minus that layer's whole stack and autosave the loss. A new adjustment type therefore raises the canvas schema for documents that use it. A noise, clarity, texture or dehaze adjustment, enabled or not, on any layer or group at any depth of the live document or a saved snapshot, makes the canvas schema at least 5 (`canvasMigration.ts`, `writableCanvasSchemaVersion`); control-layer derivation fields raise it to 6 (see [Control layer derivation](#control-layer-derivation)). A document with neither keeps its version. Adoption is sticky: removing the last such entry leaves the document at its adopted schema, matching the server's monotonic floor. A build whose maximum is lower refuses the project at the server and the canvas at load as an unsupported version. -Load does not reinterpret a known value either. Anima control layer saved as ControlNet before ControlNet-LLLite was supported keeps its kind and model; layer settings name the adapter type to switch to and offer a one-step switch that keeps model, weight and step range where the new kind can use them. +Load does not reinterpret a known value either. A control layer saved with a kind the selected main model cannot run (an Anima layer saved as ControlNet before ControlNet-LLLite was supported) keeps its kind and model. Choosing a control type for it re-resolves the model for the selected main model, and the kind follows the resolved model. ## Font dependencies @@ -86,6 +86,12 @@ Project archives record font dependencies by content hash. Export defaults to re Import verifies declared entries and checksums, then validates all embedded fonts before creating server resources. Fonts uploaded privately; canonical document's references remapped to destination IDs. Duplicate uploads reuse existing private records. Rollback removes only newly created fonts and only when project creation was not attempted or has confirmed absence; ambiguous create response keeps the resources. Font quota failures offer explicit references-only retry. Changing accounts cancels the operation and prevents cleanup under another user's credentials. +## Control layer derivation + +A control layer may keep the image it derives its map from, so detection can re-run. Four optional fields on the control leaf (`canvas-engine/contracts.ts`) carry it: `controlType` (what the image controls; a closed set, so an unknown value refuses the canvas), `detection` (the detector and its settings; `null` means "use the image as-is"; the detector name stays open), `original` (the pre-detection image or paint bitmap, a persisted image reference) and `originalTransform` (the original's frame inside the map's layer frame; absent means identity). `source` stays the pixels generation uses (the map once detected) and `filter` what produced them. Invariants, owned by `canvas-engine/document/controlDerivation.ts`: no type means none of the other three; a `null` detection keeps no original; an original implies a detection and a derived `source`. Load repairs violations silently by dropping the bookkeeping that no longer holds, never the pixels. + +A build that predates these fields would strip them on load and lose the original on its next save, so a control leaf carrying any of them makes the canvas schema 6 (`canvasMigration.ts`, `writableCanvasSchemaVersion`), one above the schema 5 that noise, clarity, texture and dehaze adjustments raise: a schema 5 build cannot read the fields and refuses the canvas instead. Documents without them keep their version, and adoption is sticky like the server's floor. `original` names an image by `imageName` like every other reference, so asset holding, cleanup and `.invk` export cover it unchanged. + ## Verification Run `pnpm lint`, `pnpm test`, `pnpm test:browser`, and `pnpm run test:performance:build` from webv2. Queue receipt tests also cover backend admission, account isolation, project deletion, partial acceptance, idempotent retries. Browser tests exercise real IndexedDB transactions and Web Locks. diff --git a/invokeai/frontend/webv2/src/workbench/projects/api.test.ts b/invokeai/frontend/webv2/src/workbench/projects/api.test.ts index acde0b97948..c921eacf3ec 100644 --- a/invokeai/frontend/webv2/src/workbench/projects/api.test.ts +++ b/invokeai/frontend/webv2/src/workbench/projects/api.test.ts @@ -42,7 +42,7 @@ describe('createProjectSettled', () => { const post = () => ({ body: JSON.stringify({ ...request, - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, minimum_canvas_schema_version: 3, }), @@ -93,7 +93,7 @@ describe('createProjectSettled', () => { expect(transport.apiFetchJson).toHaveBeenNthCalledWith(2, '/api/v1/projects/', post()); expect(transport.apiFetchJson).toHaveBeenNthCalledWith( 3, - '/api/v1/projects/project-1?max_canvas_schema_version=5&max_document_schema_version=4', + '/api/v1/projects/project-1?max_canvas_schema_version=6&max_document_schema_version=4', expect.anything() ); }); @@ -173,7 +173,7 @@ describe('createProjectSettled', () => { await expect(createProjectSettled(request, captureAccountScope())).resolves.toMatchObject(project); expect(transport.apiFetchJson).toHaveBeenNthCalledWith( 3, - '/api/v1/projects/project-1?max_canvas_schema_version=5&max_document_schema_version=4', + '/api/v1/projects/project-1?max_canvas_schema_version=6&max_document_schema_version=4', expect.anything() ); @@ -265,7 +265,7 @@ describe('canvas schema compatibility declarations', () => { await getProject('project/one'); expect(transport.apiFetchJson).toHaveBeenCalledWith( - '/api/v1/projects/project%2Fone?max_canvas_schema_version=5&max_document_schema_version=4', + '/api/v1/projects/project%2Fone?max_canvas_schema_version=6&max_document_schema_version=4', { signal: undefined } ); }); @@ -280,7 +280,7 @@ describe('canvas schema compatibility declarations', () => { data: {}, minimum_document_schema_version: 4, name: 'Project', - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, minimum_canvas_schema_version: 3, }), @@ -305,7 +305,7 @@ describe('canvas schema compatibility declarations', () => { expected_revision: 4, minimum_canvas_schema_version: 3, name: 'Project', - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, }), method: 'PUT', @@ -507,7 +507,7 @@ describe('project workspaces', () => { body: JSON.stringify({ data: { layout: {} }, expected_revision: 1, - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, }), method: 'PUT', @@ -535,7 +535,7 @@ describe('project workspaces', () => { minimum_document_schema_version: 4, name: 'Project', workspace, - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, }), method: 'PUT', @@ -581,7 +581,7 @@ describe('project workspaces', () => { minimum_document_schema_version: 4, name: 'Project', workspace: { layout: {} }, - max_canvas_schema_version: 5, + max_canvas_schema_version: 6, max_document_schema_version: 4, minimum_canvas_schema_version: 3, }), diff --git a/invokeai/frontend/webv2/src/workbench/projects/projectAssets.test.ts b/invokeai/frontend/webv2/src/workbench/projects/projectAssets.test.ts index c35b6c2b412..b3a7e35ccc3 100644 --- a/invokeai/frontend/webv2/src/workbench/projects/projectAssets.test.ts +++ b/invokeai/frontend/webv2/src/workbench/projects/projectAssets.test.ts @@ -246,6 +246,23 @@ describe('collectLiveAssetRefs', () => { expect(images).toEqual(new Set(['mask.png', 'reference.png', 'graph-image.png', 'upscale-input.png'])); }); + it("holds a control layer's pre-detection original beside its derived map", () => { + const control = { + controlType: 'depth', + detection: { settings: {}, type: 'depth_anything_depth_estimation' }, + id: 'control-1', + original: { image: imageRef('photo.png'), type: 'image' }, + source: { bitmap: imageRef('depth-map.png'), type: 'paint' }, + type: 'control', + }; + const document = projectDocument({ canvas: { document: { stacks: { control: [control] } } } }); + + expect(collectLiveAssetRefs(document).images).toEqual( + new Set(['photo.png', 'depth-map.png', 'graph-image.png', 'upscale-input.png']) + ); + expect(collectHeldAssetRefs([document]).images.has('photo.png')).toBe(true); + }); + it('ignores empty names and non-string values', () => { const refs = collectLiveAssetRefs({ canvas: { imageName: '' }, settings: { image_name: 7, video_name: null } }); diff --git a/invokeai/frontend/webv2/src/workbench/projects/projectFile.test.ts b/invokeai/frontend/webv2/src/workbench/projects/projectFile.test.ts index 3771878389f..80abc4a87ad 100644 --- a/invokeai/frontend/webv2/src/workbench/projects/projectFile.test.ts +++ b/invokeai/frontend/webv2/src/workbench/projects/projectFile.test.ts @@ -1,3 +1,4 @@ +import { createControlLayer } from '@workbench/canvas-engine/api'; import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; import { MAX_SUPPORTED_CANVAS_SCHEMA_VERSION } from '@workbench/canvasSchemaVersion'; import { createDraftProject } from '@workbench/workbenchState'; @@ -782,6 +783,43 @@ describe('importProjectFile', () => { expect(createRequest.data.canvas.document.stacks.raster[0]?.adjustments).toEqual([noise]); }); + it('carries a typed control layer through export and import at canvas schema 6', async () => { + const typed = { + ...createControlLayer({ base: null, id: 'pose', name: 'Pose' }), + controlType: 'pose' as const, + detection: null, + }; + const draft = createDraftProject([]); + const project = { + ...draft, + canvas: { + ...draft.canvas, + document: { ...draft.canvas.document, stacks: stacksFrom([typed]), version: 6 as const }, + version: 6 as const, + }, + name: 'Pose', + }; + + acceptCreate(); + await projectFile.exportOpenProject(project); + const { readArchive, readEntryText } = await import('./invk/archive'); + const entries = await readArchive(new Uint8Array(await capturedArchive().arrayBuffer())); + expect(JSON.parse(readEntryText(entries.get('manifest.json')!))).toMatchObject({ minimumCanvasSchemaVersion: 6 }); + + await projectFile.importProjectFile(capturedArchive()); + + const createRequest = api.createProjectSettled.mock.calls[0]![0] as { + data: { canvas: { document: { stacks: { control: Record[] } }; version: number } }; + minimum_canvas_schema_version: number; + }; + expect(createRequest.minimum_canvas_schema_version).toBe(6); + expect(createRequest.data.canvas.version).toBe(6); + expect(createRequest.data.canvas.document.stacks.control[0]).toMatchObject({ + controlType: 'pose', + detection: null, + }); + }); + describe('Auto-switch of an imported project', () => { /** A project as a build before Auto-switch saved it: the old progress toggle, no mode. */ const legacySettingsProject = () => { diff --git a/invokeai/frontend/webv2/src/workbench/queue-integration/useQueueItemRecall.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/queue-integration/useQueueItemRecall.browser.test.tsx index cb489bf722e..cd23d347607 100644 --- a/invokeai/frontend/webv2/src/workbench/queue-integration/useQueueItemRecall.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/queue-integration/useQueueItemRecall.browser.test.tsx @@ -47,7 +47,10 @@ vi.mock('@features/models', async (importOriginal) => ({ vi.mock('@workbench/useNotify', () => ({ useNotify: () => ({ error: vi.fn(), info: mocks.notifyInfo, success: mocks.notifySuccess }), })); -vi.mock('@workbench/useOpenWorkbenchWidget', () => ({ useOpenWorkbenchWidget: () => mocks.openWidget })); +vi.mock('@workbench/useOpenWorkbenchWidget', async (importOriginal) => ({ + ...(await importOriginal>()), + useOpenWorkbenchWidget: () => mocks.openWidget, +})); vi.mock('react-i18next', () => ({ useTranslation: () => ({ t: (key: string) => key }) })); seedArchitectureCapabilities(); diff --git a/invokeai/frontend/webv2/src/workbench/settings/contracts.ts b/invokeai/frontend/webv2/src/workbench/settings/contracts.ts index e1a04184969..4fef471f54d 100644 --- a/invokeai/frontend/webv2/src/workbench/settings/contracts.ts +++ b/invokeai/frontend/webv2/src/workbench/settings/contracts.ts @@ -127,6 +127,11 @@ export interface WorkbenchPreferences { krea2RebalancePresets: StoredRebalancePreset[]; /** User-saved Generate settings snapshots ("recipes"), in the order they were saved. */ generatePresets: StoredGeneratePreset[]; + /** + * The last control model picked per main-model base and control type (`${base}:${controlType}` → model key); + * control layers left on automatic use it. Not shown in Settings. + */ + controlModelMemory: Record; } export type SettingsSectionId = string; diff --git a/invokeai/frontend/webv2/src/workbench/settings/store.test.ts b/invokeai/frontend/webv2/src/workbench/settings/store.test.ts index b48ccfbbb1f..b0b4cbf0975 100644 --- a/invokeai/frontend/webv2/src/workbench/settings/store.test.ts +++ b/invokeai/frontend/webv2/src/workbench/settings/store.test.ts @@ -1,6 +1,7 @@ import type * as accountLifecycleModule from '@platform/state/accountLifecycle'; import { LOG_NAMESPACES } from '@platform/logging/contracts'; +import { CONTROL_TYPES } from '@workbench/canvas-engine/api'; import { beforeEach, describe, expect, it, vi } from 'vitest'; import type { ProjectSettings } from './contracts'; @@ -556,3 +557,70 @@ describe('normalizeWorkbenchPreferences language', () => { expect(store.normalizeWorkbenchPreferences({ language: 'missing' as never }).language).toBe('en'); }); }); + +describe('controlModelMemory preference', () => { + it('starts empty and keeps only well-formed picks', () => { + expect(store.normalizeWorkbenchPreferences({}).controlModelMemory).toEqual({}); + expect( + store.normalizeWorkbenchPreferences({ controlModelMemory: ['sdxl:depth'] as never }).controlModelMemory + ).toEqual({}); + expect( + store.normalizeWorkbenchPreferences({ + controlModelMemory: { + ':depth': 'no-base', + 'sdxl:depth': 'depth-model', + 'sdxl:flux': 'unknown-type', + 'sdxl:pose': '', + 'sdxl:tile': 'x'.repeat(256), + 'sd-1:edges': 7 as never, + sdxl: 'no-type', + }, + }).controlModelMemory + ).toEqual({ 'sdxl:depth': 'depth-model' }); + }); + + it('accepts every control type, so the spelled-out key pattern stays in step with the canvas contract', () => { + const memory = Object.fromEntries(CONTROL_TYPES.map((type) => [`sdxl:${type}`, `${type}-model`])); + + expect(store.normalizeWorkbenchPreferences({ controlModelMemory: memory }).controlModelMemory).toEqual(memory); + }); + + it('keeps the newest 256 picks', () => { + const memory = Object.fromEntries(Array.from({ length: 300 }, (_, index) => [`base-${index}:depth`, `m${index}`])); + const kept = store.normalizeWorkbenchPreferences({ controlModelMemory: memory }).controlModelMemory; + + expect(Object.keys(kept)).toHaveLength(256); + expect(kept['base-299:depth']).toBe('m299'); + expect(kept['base-43:depth']).toBeUndefined(); + }); + + it('keeps the same record across unrelated preference patches, so readers that compare it do not re-run', async () => { + await store.loadWorkbenchSettings(); + await store.patchControlModelMemory('sdxl:depth', 'depth-a'); + const memory = store.getWorkbenchPreferences().controlModelMemory; + + await store.patchWorkbenchPreferences({ reduceMotion: true }); + + expect(store.getWorkbenchPreferences().controlModelMemory).toBe(memory); + expect(store.normalizeWorkbenchPreferences({ controlModelMemory: memory }).controlModelMemory).toBe(memory); + }); + + it('remembers, replaces and forgets a pick through the persisted preferences, and resets with the rest', async () => { + await store.loadWorkbenchSettings(); + + await store.patchControlModelMemory('sdxl:depth', 'depth-a'); + await store.patchControlModelMemory('flux:pose', 'flux-union'); + await store.patchControlModelMemory('sdxl:depth', 'depth-b'); + + expect(readBackendPreferences().controlModelMemory).toEqual({ 'flux:pose': 'flux-union', 'sdxl:depth': 'depth-b' }); + + await store.patchControlModelMemory('flux:pose', null); + + expect(store.getWorkbenchPreferences().controlModelMemory).toEqual({ 'sdxl:depth': 'depth-b' }); + expect(readBackendPreferences().controlModelMemory).toEqual({ 'sdxl:depth': 'depth-b' }); + + await store.clearWorkbenchSettings(); + + expect(store.getWorkbenchPreferences().controlModelMemory).toEqual({}); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/settings/store.ts b/invokeai/frontend/webv2/src/workbench/settings/store.ts index 1c8a7b205c0..e25526f3f53 100644 --- a/invokeai/frontend/webv2/src/workbench/settings/store.ts +++ b/invokeai/frontend/webv2/src/workbench/settings/store.ts @@ -50,6 +50,7 @@ export const DEFAULT_PREFERENCES: WorkbenchPreferences = { alphaNoticeAcknowledged: false, autoSwitchInvocationRoute: true, confirmImageDeletion: true, + controlModelMemory: {}, customHotkeys: {}, developerConsoleOutputEnabled: false, developerLogEnabled: true, @@ -265,6 +266,38 @@ const normalizeGeneratePresets = (values: unknown): StoredGeneratePreset[] => { return presets; }; +/** + * `${base}:${controlType}`. The control types are spelled out because importing them from the canvas engine would + * pull it into Launchpad's bundle; a store test keeps the list in step with `CONTROL_TYPES`. + */ +const CONTROL_MODEL_MEMORY_KEY = /^[^:]+:(?:pose|depth|edges|softedge|lineart|sketch|color|tile)$/; +const CONTROL_MODEL_MEMORY_MAX_ENTRIES = 256; +const CONTROL_MODEL_KEY_MAX_LENGTH = 255; + +/** + * Keeps well-formed entries, the newest {@link CONTROL_MODEL_MEMORY_MAX_ENTRIES} (a pick is re-inserted last), so the + * synced blob stays bounded. An already-valid record comes back as is, so patching another preference keeps its + * identity and does not re-run selectors that read it (Invoke readiness). + */ +const normalizeControlModelMemory = (values: unknown): Record => { + if (!values || typeof values !== 'object' || Array.isArray(values)) { + return {}; + } + + const entries = Object.entries(values); + const kept = entries + .filter( + (entry): entry is [string, string] => + CONTROL_MODEL_MEMORY_KEY.test(entry[0]) && + typeof entry[1] === 'string' && + entry[1].length > 0 && + entry[1].length <= CONTROL_MODEL_KEY_MAX_LENGTH + ) + .slice(-CONTROL_MODEL_MEMORY_MAX_ENTRIES); + + return kept.length === entries.length ? (values as Record) : Object.fromEntries(kept); +}; + export const isViewerAutoSwitchMode = (value: unknown): value is ViewerAutoSwitchMode => value === 'off' || value === 'latest' || value === 'progress'; @@ -308,6 +341,7 @@ export const normalizeWorkbenchPreferences = (preferences?: WorkbenchPreferences typeof preferences?.confirmImageDeletion === 'boolean' ? preferences.confirmImageDeletion : DEFAULT_PREFERENCES.confirmImageDeletion, + controlModelMemory: normalizeControlModelMemory(preferences?.controlModelMemory), customHotkeys: normalizeCustomHotkeys(preferences?.customHotkeys), developerLogEnabled: typeof preferences?.developerLogEnabled === 'boolean' @@ -700,6 +734,16 @@ export const patchWorkbenchPreferences = async (preferences: Partial => { + const controlModelMemory = { ...store.getSnapshot().preferences.controlModelMemory }; + delete controlModelMemory[key]; + if (modelKey !== null) { + controlModelMemory[key] = modelKey; + } + return patchWorkbenchPreferences({ controlModelMemory }); +}; + export const clearWorkbenchSettings = async (): Promise => { const owner = captureAccountScope(); const storageSuffix = owner.accountId === null ? getUserStorageScope() : owner.storageSuffix; diff --git a/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.browser.test.tsx index abaa3d74b60..cb638878dcb 100644 --- a/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.browser.test.tsx @@ -14,6 +14,7 @@ import type * as UseTopbarShortcutModule from './useTopbarShortcut'; // icon. Mock queue plumbing to isolate that gate. const harness = vi.hoisted(() => ({ bindings: {} as Record, + revealed: [] as string[], progress: { activeItemIndex: 1, completedItemCount: 0, message: '', percentage: 0.42 as number | null }, summary: { current: 1, remaining: 1, runningQueueItemId: 'item-1', total: 1 }, })); @@ -26,9 +27,17 @@ vi.mock('./useTopbarShortcut', async (importOriginal) => ({ ...(await importOriginal()), useTopbarShortcutBinding: (commandId: string) => harness.bindings[commandId] ?? null, })); +vi.mock('@workbench/widgets/layers/revealLayerProperties', () => ({ + useRevealLayerProperties: () => (layerId: string) => harness.revealed.push(layerId), +})); vi.mock('react-i18next', () => ({ useTranslation: () => ({ - t: (key: string, options?: { shortcut?: string }) => (options?.shortcut ? `${key}: ${options.shortcut}` : key), + t: (key: string, options?: { count?: number; reason?: string; shortcut?: string }) => + options?.shortcut + ? `${key}: ${options.shortcut}` + : options?.count !== undefined + ? `${key}(${[options.count, options.reason].filter((value) => value !== undefined).join(', ')})` + : key, }), })); @@ -47,9 +56,9 @@ let root: Root | null = null; const makeInvocationState = (overrides: Partial = {}): InvocationState => ({ batchCount: 1, - blockingReasons: [], invocation, invoke: vi.fn(() => Promise.resolve()), + issues: [], isPreparing: false, isValid: true, placedTypeIds: new Set(), @@ -82,6 +91,7 @@ const hasSpinner = (button: HTMLButtonElement): boolean => button.querySelector( beforeEach(() => { harness.bindings = {}; + harness.revealed = []; harness.progress.percentage = 0.42; harness.summary.total = 1; harness.summary.runningQueueItemId = 'item-1'; @@ -165,19 +175,23 @@ describe('InvokeButton front of queue', () => { expect(invoke).toHaveBeenNthCalledWith(2, { prepend: true }); }); - it('ignores both clicks while Invoke is blocked', async () => { + it('ignores both clicks while Invoke is blocked by issues no layer can show', async () => { const invoke = vi.fn(() => Promise.resolve()); const { button } = await renderInvokeButton({ - blockingReasons: ['The backend is disconnected.'], invoke, isValid: false, + issues: [{ layerId: null, message: 'The backend is disconnected.' }], }); + expect(button.getAttribute('aria-disabled')).toBe('true'); + expect(button.getAttribute('aria-label')).toBe('topbar.invoke.unavailable'); + expect(getComputedStyle(button).opacity).toBe('0.55'); // Dispatched: a blocked Invoke is not actionable for a pointer driver. await act(() => button.dispatchEvent(new MouseEvent('click', { bubbles: true }))); await act(() => button.dispatchEvent(new MouseEvent('click', { bubbles: true, shiftKey: true }))); expect(invoke).not.toHaveBeenCalled(); + expect(harness.revealed).toEqual([]); }); it('names the live front-of-queue binding and Shift+click, and lists both bindings as shortcuts', async () => { @@ -201,3 +215,41 @@ describe('InvokeButton front of queue', () => { await expect.element(page.getByText('topbar.invoke.invokeFrontHintNoShortcut')).toBeVisible(); }); }); + +describe('InvokeButton issues', () => { + const LAYER_ISSUES = [ + { layerId: 'control-2', message: 'Control layer "Depth 1": Choose a model.' }, + { layerId: null, message: 'The backend is disconnected.' }, + ]; + + it('counts the issues in place of the shortcut and keeps listing them in the tooltip', async () => { + harness.bindings = { 'app.invoke': { aria: 'Control+Enter', display: 'Ctrl+Enter', parts: ['ctrl', 'enter'] } }; + const { button } = await renderInvokeButton({ isValid: false, issues: LAYER_ISSUES }); + + expect(button.textContent).toContain('topbar.invoke.issueCount(2)'); + expect(button.querySelector('kbd')).toBeNull(); + await act(() => userEvent.hover(button)); + + await expect.element(page.getByText('Control layer "Depth 1": Choose a model.')).toBeVisible(); + await expect.element(page.getByText('backend is disconnected.')).toBeVisible(); + }); + + it('shows the first issue’s layer on click or Enter, and never invokes', async () => { + const invoke = vi.fn(() => Promise.resolve()); + const { button } = await renderInvokeButton({ invoke, isValid: false, issues: LAYER_ISSUES }); + + expect(button.getAttribute('aria-disabled')).not.toBe('true'); + // Actionable, so not dimmed like a blocked Invoke. + expect(getComputedStyle(button).opacity).toBe('1'); + expect(button.getAttribute('aria-label')).toBe( + 'topbar.invoke.showIssue(2, Control layer "Depth 1": Choose a model.)' + ); + + await act(() => userEvent.click(button)); + button.focus(); + await act(() => userEvent.keyboard('{Enter}')); + + expect(harness.revealed).toEqual(['control-2', 'control-2']); + expect(invoke).not.toHaveBeenCalled(); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.tsx b/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.tsx index b69721a6a98..1d53d1a3522 100644 --- a/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.tsx +++ b/invokeai/frontend/webv2/src/workbench/shell/topbar/InvokeButton.tsx @@ -6,6 +6,7 @@ import { Button } from '@platform/ui/Button'; import { Tooltip } from '@platform/ui/Tooltip'; import { getDestinationLabel } from '@workbench/invocation'; import { useActiveQueueProgress } from '@workbench/queue-integration/useActiveQueueProgress'; +import { useRevealLayerProperties } from '@workbench/widgets/layers/revealLayerProperties'; import { PlayIcon } from 'lucide-react'; import { useCallback, useMemo, useState } from 'react'; import { useTranslation } from 'react-i18next'; @@ -44,7 +45,8 @@ const plural = (count: number, noun: string): string => `${count} ${noun}${count */ export const InvokeButton = ({ state }: { state: InvocationState }) => { const { t } = useTranslation(); - const { blockingReasons, invoke, isPreparing, isValid } = state; + const { invoke, isPreparing, isValid, issues } = state; + const revealLayerProperties = useRevealLayerProperties(); const shortcutBinding = useTopbarShortcutBinding('app.invoke'); const frontBinding = useTopbarShortcutBinding('app.invokeFront'); const shortcutParts = shortcutBinding?.parts ?? null; @@ -76,6 +78,14 @@ export const InvokeButton = ({ state }: { state: InvocationState }) => { [invoke] ); const canInvoke = isValid && !isPreparing; + // Blocked by issues: the count replaces the shortcut, and when the first names a layer, activation shows it. + const showsIssues = !isValid && !isPreparing && issues.length > 0; + const issueLayerId = showsIssues ? (issues[0]?.layerId ?? null) : null; + const handleShowIssue = useCallback(() => { + if (issueLayerId !== null) { + revealLayerProperties(issueLayerId); + } + }, [issueLayerId, revealLayerProperties]); const iconMode = isPreparing ? ({ mode: 'progress', value: null } as const) @@ -88,25 +98,28 @@ export const InvokeButton = ({ state }: { state: InvocationState }) => { return ( @@ -135,6 +174,7 @@ afterEach(async () => { root = null; resetArchitectureCapabilities(); templates.reset(); + preferences.reset(); }); describe('useInvocationState and the architecture capability table', () => { @@ -196,3 +236,44 @@ describe('useInvocationState invoke', () => { expect(harness.submit.mock.calls[1]![0].prepend).toBeUndefined(); }); }); + +describe('useInvocationState and the remembered control-model picks', () => { + it('re-resolves a canvas route when a remembered pick settles a choice between control models', async () => { + const depthLayer: CanvasControlLayerContract = { + adapter: { beginEndStepPct: [0, 0.75], controlMode: 'balanced', kind: 'controlnet', model: null, weight: 0.75 }, + blendMode: 'normal', + controlType: 'depth', + detection: null, + id: 'depth-layer', + isEnabled: true, + isLocked: false, + name: 'Depth 1', + opacity: 1, + source: { image: { height: 64, imageName: 'depth.png', width: 64 }, type: 'image' }, + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: 0, y: 0 }, + type: 'control', + withTransparencyEffect: true, + }; + const project = buildGenerateProject(); + setArchitectureCapabilities(architectureCapabilitiesFixture); + await renderProbe({ + ...project, + canvas: { ...project.canvas, document: { ...project.canvas.document, stacks: stacksFrom([depthLayer]) } }, + invocation: { ...project.invocation, sourceId: 'canvas' }, + }); + + expect(blockingReasons()).toContain('widgets.layers.control.invalidLayer'); + // The control layer's issue names the layer, so Invoke can show it; the others name none. + const layers = host!.querySelector('[data-testid="issue-layers"]')!.textContent!.split(' | '); + expect(layers.filter((layerId) => layerId !== '-')).toEqual(['depth-layer']); + expect(blockingReasons().split(' | ')[layers.indexOf('depth-layer')]).toContain( + 'widgets.layers.control.invalidLayer' + ); + + await act(() => { + preferences.set({ 'sdxl:depth': 'sdxl-depth-b' }); + }); + + expect(blockingReasons()).not.toContain('widgets.layers.control.invalidLayer'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/shell/topbar/useInvocationState.ts b/invokeai/frontend/webv2/src/workbench/shell/topbar/useInvocationState.ts index 67bc74097ff..2bce8d3ebc4 100644 --- a/invokeai/frontend/webv2/src/workbench/shell/topbar/useInvocationState.ts +++ b/invokeai/frontend/webv2/src/workbench/shell/topbar/useInvocationState.ts @@ -26,6 +26,7 @@ import { resolveInvocationRouteInput, } from '@workbench/invocation'; import { useIsInvocationPreparing } from '@workbench/invocationPreparation'; +import { getWorkbenchPreferences, subscribeWorkbenchPreferences } from '@workbench/settings/store'; import { useActiveProjectSelector, useWorkbenchCommands, @@ -38,31 +39,39 @@ import { useTranslation } from 'react-i18next'; const selectInvocationRouteInput = createInvocationRouteInputSelector(); /** - * Include capability and template snapshots in route selection so either load independently clears its readiness - * gate. + * Include capability, template and remembered control-model snapshots in route selection so each change + * independently re-evaluates its readiness gate. */ const subscribeRouteSources = (listener: () => void): (() => void) => { const unsubscribeCapabilities = subscribeArchitectureCapabilities(listener); const unsubscribeTemplates = subscribeInvocationTemplates(listener); + const unsubscribePreferences = subscribeWorkbenchPreferences(listener); return () => { unsubscribeCapabilities(); unsubscribeTemplates(); + unsubscribePreferences(); }; }; let routeSources: { capabilities: ReturnType; + controlModelMemory: ReturnType['controlModelMemory']; templates: ReturnType; } | null = null; -/** Referentially stable until one of the two stores publishes a new snapshot. */ +/** Referentially stable until one of the stores publishes a new snapshot. */ const getRouteSourcesSnapshot = () => { const capabilities = getArchitectureCapabilitiesSnapshot(); + const controlModelMemory = getWorkbenchPreferences().controlModelMemory; const templates = getInvocationTemplatesSnapshot(); - if (routeSources?.capabilities !== capabilities || routeSources.templates !== templates) { - routeSources = { capabilities, templates }; + if ( + routeSources?.capabilities !== capabilities || + routeSources.controlModelMemory !== controlModelMemory || + routeSources.templates !== templates + ) { + routeSources = { capabilities, controlModelMemory, templates }; } return routeSources; @@ -91,9 +100,16 @@ export const isBatchCountFixed = (sourceId: InvocationRoute['sourceId'], values: export const getBatchCount = (sourceId: InvocationRoute['sourceId'], values: Record): number => isBatchCountFixed(sourceId, values) ? 1 : sanitizeBatchCount(values.batchCount); +/** Why Invoke cannot run, worded for the user; `layerId` names the canvas layer to show for it. */ +export interface InvocationIssue { + message: string; + layerId: string | null; +} + export interface InvocationState { batchCount: number; - blockingReasons: string[]; + /** Every reason Invoke cannot run right now, in the order the tooltip lists them. */ + issues: InvocationIssue[]; invocation: InvocationRoute; /** Sessions one workflow run produces through batch nodes; absent without batch nodes, null while unresolved. */ workflowBatchSize?: number | null; @@ -163,11 +179,14 @@ export const useInvocationState = (): InvocationState => { ? 'The prompt could not be expanded.' : (promptExpansion.error ?? null); - const blockingReasons = useMemo( - () => [ - ...(isConnected ? [] : ['The backend is disconnected.']), - ...(expansionReason === null ? [] : [expansionReason]), - ...resolvedRoute.validationReasons.map((reason) => localizeInvocationValidationReason(reason, t)), + const issues = useMemo( + (): InvocationIssue[] => [ + ...(isConnected ? [] : [{ layerId: null, message: 'The backend is disconnected.' }]), + ...(expansionReason === null ? [] : [{ layerId: null, message: expansionReason }]), + ...resolvedRoute.validationReasons.map((reason) => ({ + layerId: typeof reason === 'object' && 'controlLayerIssue' in reason ? reason.controlLayerIssue.layerId : null, + message: localizeInvocationValidationReason(reason, t), + })), ], [expansionReason, isConnected, resolvedRoute.validationReasons, t] ); @@ -189,9 +208,9 @@ export const useInvocationState = (): InvocationState => { return { batchCount: getBatchCount(invocation.sourceId, sourceValues), - blockingReasons, invocation, invoke, + issues, isPreparing, isValid, placedTypeIds, diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.browser.test.tsx index f2ca20ea2de..3375470473d 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.browser.test.tsx @@ -1,4 +1,11 @@ import type * as DndKitCore from '@dnd-kit/core'; +import type { + CanvasControlLayerContract, + CanvasLayerContract, + CanvasNodeContract, + ControlLayerView, + ControlLayerViewOutcome, +} from '@workbench/canvas-engine/api'; import type { CanvasStagingCandidateContract } from '@workbench/canvas-engine/contracts'; import type { CanvasEngine } from '@workbench/canvas-operations/createCanvasEngine'; import type { Project } from '@workbench/projectContracts'; @@ -7,8 +14,19 @@ import type { WidgetViewProps } from '@workbench/widgetContracts'; import type * as WorkbenchContextModule from '@workbench/WorkbenchContext'; import { ChakraProvider } from '@chakra-ui/react'; +import { galleryTransfers, type GalleryImage } from '@features/gallery'; +import { auditAccessibility } from '@platform/browser/auditAccessibility.testing'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; import { system } from '@theme/system'; +import { createControlLayer } from '@workbench/canvas-engine/api'; +import { + groupContract, + layerContract, + stacksFrom, +} from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; import { createTestStubRasterBackend } from '@workbench/canvas-engine/render/raster.testStub'; +import { getCanvasOperations, getControlSourceSnapshot } from '@workbench/canvas-operations/api'; +import { createControlDetectionStatusStore } from '@workbench/canvas-operations/controlDetectionStatusStore'; import { createCanvasEngine } from '@workbench/canvas-operations/createCanvasEngine'; import { createCanvasProjectMutationPort } from '@workbench/canvasProjectMutationPort'; import { createInitialWorkbenchState } from '@workbench/workbenchState'; @@ -18,19 +36,32 @@ import { act } from 'react'; import { createRoot, type Root } from 'react-dom/client'; import { I18nextProvider, initReactI18next } from 'react-i18next'; import { afterEach, beforeAll, describe, expect, it, vi } from 'vitest'; -import { page } from 'vitest/browser'; +import { page, userEvent } from 'vitest/browser'; + +import type * as EngineStoreHooks from './engineStoreHooks'; /** The view reads the real store and drives its real commands; only unrelated chrome is stubbed. */ const harness = vi.hoisted(() => ({ + /** Drag data dnd-kit reports as active; null leaves the real (idle) context. */ + dragData: null as unknown, engine: null as CanvasEngine | null, store: null as ReturnType | null, })); const activeProject = () => harness.store!.getState().projects[0]!; -vi.mock('@dnd-kit/core', async (importOriginal) => ({ - ...(await importOriginal()), - useDndMonitor: () => undefined, -})); +vi.mock('@dnd-kit/core', async (importOriginal) => { + const actual = await importOriginal(); + return { + ...actual, + useDndContext: () => { + const context = actual.useDndContext(); + return harness.dragData === null + ? context + : { ...context, active: { data: { current: harness.dragData }, id: 'gallery-drag' } }; + }, + useDndMonitor: () => undefined, + }; +}); vi.mock('@workbench/WorkbenchContext', async (importOriginal) => ({ ...(await importOriginal()), useActiveProjectId: () => activeProject().id, @@ -43,7 +74,10 @@ vi.mock('@workbench/WorkbenchContext', async (importOriginal) => ({ vi.mock('@workbench/useCanvasProjectMutationDispatch', () => ({ useCanvasProjectMutationDispatch: () => () => undefined, })); -vi.mock('./engineStoreHooks', () => ({ useCanvasOperation: () => null })); +vi.mock('./engineStoreHooks', async (importOriginal) => ({ + ...(await importOriginal()), + useCanvasOperation: () => null, +})); vi.mock('@workbench/canvas-operations/react', () => ({ useCanvasEngine: () => harness.engine })); vi.mock('./useCanvasGallerySave', () => ({ useCanvasGallerySave: () => ({ isSaving: false, save: () => undefined }), @@ -79,7 +113,7 @@ afterEach(async () => { await act(() => root?.unmount()); host?.remove(); harness.engine?.lifecycle.dispose(); - Object.assign(harness, { engine: null, store: null }); + Object.assign(harness, { dragData: null, engine: null, store: null }); root = null; host = null; }); @@ -96,10 +130,40 @@ const candidate: CanvasStagingCandidateContract = { width: 64, }; -/** Stages the first image of a three-image batch whose remaining images are `status`; unstaged, the surface is live. */ -const renderView = async (status: QueueItem['status'], { staged = true } = {}) => { +/** + * Stages the first image of a three-image batch whose remaining images are `status`; unstaged, the surface is live. + * `selectedLayer` starts the document with that layer selected, alone or among `nodes`; `beforeRender` sees the + * engine before the view subscribes to it. + */ +const renderView = async ( + status: QueueItem['status'], + { + beforeRender, + nodes, + selectedLayer, + staged = true, + }: { + beforeRender?: (engine: CanvasEngine) => void; + nodes?: CanvasNodeContract[]; + selectedLayer?: CanvasLayerContract; + staged?: boolean; + } = {} +) => { const initial = createInitialWorkbenchState(); - const base = initial.projects[0]!; + const created = initial.projects[0]!; + const base = selectedLayer + ? { + ...created, + canvas: { + ...created.canvas, + document: { + ...created.canvas.document, + selectedLayerId: selectedLayer.id, + stacks: stacksFrom(nodes ?? [selectedLayer]), + }, + }, + } + : created; const batch = { backendBatchId: 'batch-eyes-backend', backendItemIds: [1, 2, 3], @@ -130,6 +194,7 @@ const renderView = async (status: QueueItem['status'], { staged = true } = {}) = projectId, reportError: () => undefined, }); + beforeRender?.(harness.engine); host = document.createElement('div'); host.style.height = '600px'; host.style.width = '1000px'; @@ -149,9 +214,11 @@ const renderView = async (status: QueueItem['status'], { staged = true } = {}) = root!.render( - + + + ) @@ -251,3 +318,348 @@ describe('CanvasWidgetView clipboard shortcuts', () => { expect(store.getState().notifications[0]?.message).toBeUndefined(); }); }); + +describe('CanvasWidgetView control layer overlay', () => { + const emptyControlLayer = createControlLayer({ id: 'control-1', name: 'Control Layer 1' }); + const hint = () => page.getByRole('note').filter({ hasText: 'Drop an image here' }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('points an empty control layer at its image sources until the brush or eraser takes over', async () => { + await renderView('completed', { selectedLayer: emptyControlLayer, staged: false }); + + await expect.element(hint()).toBeVisible(); + await expect.element(hint().getByText('or pick one in Properties', { exact: true })).toBeVisible(); + + await act(() => harness.engine!.tools.setTool('brush')); + await expect.element(hint()).not.toBeInTheDocument(); + + await act(() => harness.engine!.tools.setTool('eraser')); + await expect.element(hint()).not.toBeInTheDocument(); + + await act(() => harness.engine!.tools.setTool('move')); + await expect.element(hint()).toBeVisible(); + }); + + it('shows no hint while staging locks the surface against drops', async () => { + await renderView('completed', { selectedLayer: emptyControlLayer }); + + await expect.element(page.getByRole('button', { exact: true, name: 'Accept to Layer' })).toBeVisible(); + await expect.element(hint()).not.toBeInTheDocument(); + }); + + it('gives way to the drop zones while a gallery image is dragged', async () => { + harness.dragData = { items: [{ kind: 'image', name: 'photo.png' }], kind: 'gallery-item' }; + await renderView('completed', { selectedLayer: emptyControlLayer, staged: false }); + + await expect.element(page.getByText('Drop an image here')).toBeInTheDocument(); + await expect.element(hint()).not.toBeInTheDocument(); + }); + + it('shows no hint once the control layer has an image', async () => { + const filled: CanvasControlLayerContract = { + ...emptyControlLayer, + source: { image: { height: 64, imageName: 'photo.png', width: 96 }, type: 'image' }, + }; + await renderView('completed', { selectedLayer: filled, staged: false }); + + expect(page.getByRole('note').elements()).toHaveLength(0); + expect(page.getByText('Drop an image here').elements()).toHaveLength(0); + }); + + const depthLayer: CanvasControlLayerContract = { + ...emptyControlLayer, + controlType: 'depth', + detection: { settings: {}, type: 'depth_anything_depth_estimation' }, + source: { image: { height: 64, imageName: 'photo.png', width: 96 }, type: 'image' }, + }; + + /** Renders `layer` with its detection status driven by the returned store. */ + const renderWithStatuses = async (layer: CanvasControlLayerContract = depthLayer) => { + const statuses = createControlDetectionStatusStore(); + await renderView('completed', { + beforeRender: (engine) => { + const operations = getCanvasOperations(engine); + vi.spyOn(operations, 'getControlDetectionStatus').mockImplementation(statuses.get); + vi.spyOn(operations, 'subscribeControlDetectionStatus').mockImplementation(statuses.subscribe); + }, + selectedLayer: layer, + staged: false, + }); + return statuses; + }; + + it('announces a background detection run while it is queued or processing, below the header row', async () => { + const statuses = await renderWithStatuses(); + const layer = depthLayer; + // The shell reserves this much of the center view for its header row (zoom, fit, undo). + host!.style.setProperty('--wb-center-chrome-inset', '3rem'); + const status = page.getByRole('status').filter({ hasText: /^(Running|Waiting to run) / }); + + expect(status.elements()).toHaveLength(0); + + await act(() => statuses.set(layer.id, { status: 'queued' })); + await expect.element(status).toHaveTextContent('Waiting to run Depth Anything…'); + await expect.element(status).toHaveAttribute('aria-live', 'polite'); + const headerBottom = host!.getBoundingClientRect().top + 48; + expect(status.element().getBoundingClientRect().top).toBeGreaterThanOrEqual(headerBottom); + + await act(() => statuses.set(layer.id, { detector: 'normal_map', status: 'processing' })); + await expect.element(status).toHaveTextContent('Running Normal map…'); + + await act(() => statuses.set(layer.id, { status: 'idle' })); + await expect.element(status).not.toBeInTheDocument(); + }); + + it('announces a failed detection run with its reason and no spinner', async () => { + const statuses = await renderWithStatuses(); + + await act(() => statuses.set(depthLayer.id, { error: { code: 'process-failed' }, status: 'error' })); + + const status = page.getByRole('status').filter({ hasText: /failed/ }); + await expect.element(status).toHaveTextContent('Depth Anything failed: the detector could not run'); + expect(status.element().querySelector('.chakra-spinner')).toBeNull(); + + await act(() => statuses.set(depthLayer.id, { status: 'idle' })); + await expect.element(status).not.toBeInTheDocument(); + }); + + /** A depth layer whose map was detected from the photo it keeps. */ + const readyLayer: CanvasControlLayerContract = { + ...depthLayer, + filter: depthLayer.detection!, + original: { image: { height: 64, imageName: 'photo.png', width: 96 }, type: 'image' }, + source: { bitmap: { height: 64, imageName: 'depth.png', width: 96 }, offset: { x: 0, y: 0 }, type: 'paint' }, + }; + const viewSwitch = () => page.getByRole('radiogroup', { name: 'Control layer view' }); + const viewOption = (name: 'Original' | 'Map' | 'Overlay') => viewSwitch().getByRole('radio', { name }); + + it('switches a ready layer between its original, map and overlay, following the engine', async () => { + await renderWithStatuses(readyLayer); + const engine = harness.engine!; + const setView = vi.spyOn(engine.previews, 'setControlLayerView'); + + await expect.element(viewSwitch()).toBeVisible(); + await expect.element(viewOption('Map')).toBeChecked(); + expect(await auditAccessibility(viewSwitch().element())).toEqual([]); + + await act(() => userEvent.click(viewSwitch().getByText('Original', { exact: true }))); + expect(setView).toHaveBeenLastCalledWith('control-1', 'original'); + await expect.element(viewOption('Original')).toBeChecked(); + + // The engine ends the view (an edit, another selection): the switch follows it back to Map. + await act(() => engine.previews.setControlLayerView('control-1', 'map')); + await expect.element(viewOption('Map')).toBeChecked(); + + await act(async () => { + (viewOption('Map').element() as HTMLElement).focus(); + await userEvent.keyboard('{ArrowRight}'); + }); + expect(setView).toHaveBeenLastCalledWith('control-1', 'overlay'); + await expect.element(viewOption('Overlay')).toBeChecked(); + await expect.element(viewOption('Overlay')).toHaveFocus(); + }); + + it.each<[ControlLayerViewOutcome, string]>([ + ['failed', "Couldn't show the original image"], + ['over-budget', 'Not enough canvas memory to show the original image'], + ])('shows the request at once, then returns to Map with a notice when it is %s', async (outcome, title) => { + const { store } = await renderView('completed', { selectedLayer: readyLayer, staged: false }); + let settle!: (value: ControlLayerViewOutcome) => void; + vi.spyOn(harness.engine!.previews, 'setControlLayerView').mockReturnValueOnce( + new Promise((resolve) => { + settle = resolve; + }) + ); + + await act(() => userEvent.click(viewSwitch().getByText('Original', { exact: true }))); + await expect.element(viewOption('Original')).toBeChecked(); + + await act(() => settle(outcome)); + await expect.element(viewOption('Map')).toBeChecked(); + expect(store.getState().notifications[0]).toMatchObject({ kind: 'error', title }); + }); + + it('shows Map while the engine holds a view of another layer', async () => { + await renderView('completed', { selectedLayer: readyLayer, staged: false }); + const view: ControlLayerView = { layerId: 'other-control', view: 'original' }; + + const { stores } = harness.engine as unknown as { + stores: { controlLayerView: { set(value: ControlLayerView | null): void } }; + }; + await act(() => stores.controlLayerView.set(view)); + + await expect.element(viewOption('Map')).toBeChecked(); + }); + + it.each([ + ['queued', { status: 'queued' as const }, /^Waiting to run /], + ['failed', { error: { code: 'process-failed' as const }, status: 'error' as const }, /failed/], + ])('gives the slot to the detection chip while a run is %s', async (_case, runStatus, chipText) => { + const statuses = await renderWithStatuses(readyLayer); + await expect.element(viewSwitch()).toBeVisible(); + + await act(() => statuses.set(readyLayer.id, runStatus)); + await expect.element(page.getByRole('status').filter({ hasText: chipText })).toBeVisible(); + expect(viewSwitch().elements()).toHaveLength(0); + + await act(() => statuses.set(readyLayer.id, { status: 'idle' })); + await expect.element(viewSwitch()).toBeVisible(); + }); + + it.each([ + ['an empty untyped layer', emptyControlLayer], + ['a typed layer with no original', depthLayer], + ['a layer whose map awaits detection', { ...readyLayer, filter: { settings: {}, type: 'canny_edge_detection' } }], + ['a disabled layer', { ...readyLayer, isEnabled: false }], + ['a hidden layer', { ...readyLayer, isHidden: true }], + ])('offers no view switch for %s', async (_case, layer) => { + await renderWithStatuses(layer); + + await expect.element(page.getByRole('status')).toBeInTheDocument(); + expect(viewSwitch().elements()).toHaveLength(0); + }); + + it('hides the view switch while staging locks the surface', async () => { + await renderView('completed', { selectedLayer: readyLayer }); + + await expect.element(page.getByRole('button', { exact: true, name: 'Accept to Layer' })).toBeVisible(); + expect(viewSwitch().elements()).toHaveLength(0); + }); + + it('hides the view switch while an operation holds the document', async () => { + await renderView('completed', { selectedLayer: readyLayer, staged: false }); + await expect.element(viewSwitch()).toBeVisible(); + + const { stores } = harness.engine as unknown as { + stores: { documentEditingLocked: { set(value: boolean): void } }; + }; + await act(() => stores.documentEditingLocked.set(true)); + + await expect.element(viewSwitch()).not.toBeInTheDocument(); + }); + + it('gives way to the drop zones while a gallery image is dragged', async () => { + harness.dragData = { items: [{ kind: 'image', name: 'photo.png' }], kind: 'gallery-item' }; + await renderView('completed', { selectedLayer: readyLayer, staged: false }); + + await expect.element(page.getByRole('status')).toBeInTheDocument(); + expect(viewSwitch().elements()).toHaveLength(0); + }); +}); + +describe('CanvasWidgetView desktop file drops', () => { + const emptyControlLayer = createControlLayer({ id: 'control-1', name: 'Control Layer 1' }); + const PHOTO_FILE = new File([new Uint8Array([137, 80, 78, 71])], 'photo.png', { type: 'image/png' }); + const UPLOADED = { height: 64, imageName: 'dropped.png', width: 96 } as GalleryImage; + const hint = () => page.getByRole('note'); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + /** Sends a native drag carrying `transfer` over the canvas and drops it; returns whether each event was taken. */ + const drag = async (transfer: DataTransfer) => { + const surface = page.getByRole('region', { exact: true, name: 'Canvas surface' }).element(); + const over = new DragEvent('dragover', { bubbles: true, cancelable: true, dataTransfer: transfer }); + const drop = new DragEvent('drop', { bubbles: true, cancelable: true, dataTransfer: transfer }); + await act(() => surface.dispatchEvent(over)); + await act(() => surface.dispatchEvent(drop)); + return { dropTaken: drop.defaultPrevented, overTaken: over.defaultPrevented }; + }; + const dragFiles = (files: File[]) => { + const transfer = new DataTransfer(); + for (const file of files) { + transfer.items.add(file); + } + return drag(transfer); + }; + const lastNotice = (store: Awaited>['store']) => store.getState().notifications[0]; + + it('fills an empty control layer from a dropped image, says it is uploading, and refuses a second drop meanwhile', async () => { + let finishUpload!: (image: GalleryImage) => void; + const upload = vi.spyOn(galleryTransfers, 'upload').mockReturnValue( + new Promise((resolve) => { + finishUpload = resolve; + }) + ); + const { store } = await renderView('completed', { + beforeRender: (engine) => { + vi.spyOn(getCanvasOperations(engine), 'applyControlLayerIntent').mockReturnValue('updated'); + }, + selectedLayer: emptyControlLayer, + staged: false, + }); + const apply = vi.mocked(getCanvasOperations(harness.engine!).applyControlLayerIntent); + + expect(await dragFiles([PHOTO_FILE])).toEqual({ dropTaken: true, overTaken: true }); + await expect.element(hint()).toHaveTextContent('Uploading…'); + await expect.element(hint()).toHaveAttribute('aria-busy', 'true'); + + // A second drop while the first is on its way would race it: refused, with the reason. + await dragFiles([PHOTO_FILE]); + expect(upload).toHaveBeenCalledTimes(1); + await vi.waitFor(() => + expect(lastNotice(store)).toMatchObject({ title: 'An image is still uploading to this layer.' }) + ); + + await act(() => finishUpload(UPLOADED)); + await vi.waitFor(() => expect(apply).toHaveBeenCalledTimes(1)); + const { bbox } = activeProject().canvas.document; + expect(upload).toHaveBeenCalledWith(PHOTO_FILE, 'none', expect.anything()); + expect(apply).toHaveBeenCalledWith('control-1', { + image: { height: 64, imageName: 'dropped.png', width: 96 }, + kind: 'replace-image', + transform: { rotation: 0, scaleX: 1, scaleY: 1, x: bbox.x, y: bbox.y }, + }); + // Properties names the image after the file, as for its own Upload. + expect(getControlSourceSnapshot(harness.engine!).origins['dropped.png']).toEqual({ + kind: 'file', + name: 'photo.png', + }); + // The upload has landed: the card no longer says it is uploading. + await vi.waitFor(() => expect(hint().element().getAttribute('aria-busy')).toBe('false')); + expect(hint().element().textContent).toContain('Drop an image here'); + }); + + it('refuses a drop on a control layer locked by its group before uploading anything', async () => { + const upload = vi.spyOn(galleryTransfers, 'upload'); + const { store } = await renderView('completed', { + nodes: [groupContract('group-1', [emptyControlLayer], { isLocked: true })], + selectedLayer: emptyControlLayer, + staged: false, + }); + + expect(await dragFiles([PHOTO_FILE])).toEqual({ dropTaken: true, overTaken: true }); + + expect(lastNotice(store)).toMatchObject({ title: 'Unlock the layer before using this action.' }); + expect(upload).not.toHaveBeenCalled(); + }); + + it('says what a drop needs when no empty control layer is selected, or the files do not fit', async () => { + const upload = vi.spyOn(galleryTransfers, 'upload'); + const { store } = await renderView('completed', { + selectedLayer: layerContract('raster-1', 'raster'), + staged: false, + }); + + await dragFiles([PHOTO_FILE]); + expect(lastNotice(store)).toMatchObject({ title: 'Select an empty control layer to drop an image on it.' }); + await dragFiles([new File(['notes'], 'notes.txt', { type: 'text/plain' })]); + expect(lastNotice(store)).toMatchObject({ title: 'Only image files can be dropped on the canvas.' }); + await dragFiles([PHOTO_FILE, PHOTO_FILE]); + expect(lastNotice(store)).toMatchObject({ title: 'Drop one image at a time.' }); + + expect(upload).not.toHaveBeenCalled(); + }); + + it('leaves a drag that carries no files alone', async () => { + await renderView('completed', { selectedLayer: emptyControlLayer, staged: false }); + const transfer = new DataTransfer(); + transfer.setData('text/plain', 'not a file'); + + expect(await drag(transfer)).toEqual({ dropTaken: false, overTaken: false }); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.test.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.test.tsx index f25a59dccfe..210c3582def 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.test.tsx @@ -4,6 +4,7 @@ import type { Project } from '@workbench/projectContracts'; import type { WidgetViewProps } from '@workbench/widgetContracts'; import { ChakraProvider } from '@chakra-ui/react'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; import { system } from '@theme/system'; import { createWorkbenchStore } from '@workbench/workbenchStore'; import { createInstance } from 'i18next'; @@ -84,12 +85,14 @@ const renderView = (): string => renderToStaticMarkup( - + + + ); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.tsx index 30e09df87cd..db21096a470 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/CanvasWidgetView.tsx @@ -1,13 +1,20 @@ import type { WidgetViewProps } from '@workbench/widgetContracts'; -import type { MouseEvent as ReactMouseEvent } from 'react'; +import type { DragEvent as ReactDragEvent, MouseEvent as ReactMouseEvent } from 'react'; import { Box } from '@chakra-ui/react'; import { useDndMonitor, type DragEndEvent } from '@dnd-kit/core'; +import { invalidateGallery } from '@features/gallery/queries'; import { useQueueItemProgressImage } from '@features/queue/react'; import { ClipboardUnavailableError, copyImage } from '@platform/browser/clipboard'; import { useMountEffect } from '@platform/react/useMountEffect'; +import { useQueryClient } from '@tanstack/react-query'; import { preloadCanvasInvocation } from '@workbench/activeInvocationSubmission'; -import { getCanvasImportNotice } from '@workbench/canvas-operations/api'; +import { + getControlUploadNotice, + resolveControlFileDrop, + uploadControlImageFile, + type ControlSourceNotice, +} from '@workbench/canvas-operations/api'; import { useCanvasEngine, type CanvasEngineHandle } from '@workbench/canvas-operations/react'; import { getCanvasStagingSlots } from '@workbench/canvasStagingView'; import { useRegisterShortcutHintSource } from '@workbench/hotkeys/hintSources'; @@ -55,6 +62,7 @@ import { CanvasSurfaceContextLayout } from './CanvasSurfaceContextLayout'; import { resolveCheckerColors } from './checkerColors'; import { CanvasColorFeed } from './color-system/CanvasColorFeed'; import { useActiveColorCommands } from './color-system/useActiveColors'; +import { ControlLayerCanvasOverlay } from './ControlLayerCanvasOverlay'; import { useCanvasOperation } from './engineStoreHooks'; import { executeCanvasImageDropImport } from './executeCanvasImageDropImport'; import { createCanvasShortcutHintSource } from './shortcutHints'; @@ -68,6 +76,8 @@ import { useCreateFromBbox } from './useCreateFromBbox'; import { useStagedResultGallerySave } from './useStagedResultGallerySave'; import { reportLayerOperation, reportPreparedCommit, reportStructuralCommit } from './useStructuralCommit'; +const isFileDrag = (dataTransfer: DataTransfer): boolean => dataTransfer.types.includes('Files'); + const MissingFontsDialog = lazy(() => import('./MissingFontsDialog').then((module) => ({ default: module.MissingFontsDialog })) ); @@ -202,6 +212,57 @@ export const CanvasWidgetView = ({ runtime, instance }: WidgetViewProps) => { operationKind, }); const isInteractionLocked = interactionCapabilities.isSurfaceInteractionLocked; + + // A desktop file dropped on the canvas fills the selected control layer while it is empty, as the hint over it + // says. Every file drag is taken here, since the browser's default would navigate away from the app; a drop the + // canvas cannot use says why instead of doing nothing. + const queryClient = useQueryClient(); + const showNotice = useCallback( + (notice: ControlSourceNotice | null) => { + if (notice) { + notifications.add({ + kind: notice.kind, + message: notice.messageKey ? t(notice.messageKey) : notice.message, + title: t(notice.titleKey), + }); + } + }, + [notifications, t] + ); + const handleFileDragOver = useCallback((event: ReactDragEvent) => { + if (isFileDrag(event.dataTransfer)) { + event.preventDefault(); + } + }, []); + const handleFileDrop = useCallback( + (event: ReactDragEvent) => { + if (!isFileDrag(event.dataTransfer)) { + return; + } + event.preventDefault(); + if (!engine) { + return; + } + const files = [...event.dataTransfer.files]; + const resolution = resolveControlFileDrop({ + document, + engine, + files, + isCanvasLocked: isInteractionLocked || interactionCapabilities.isDocumentEditingLocked, + }); + if (resolution.status === 'refused') { + showNotice(resolution.notice); + return; + } + void uploadControlImageFile({ + engine, + file: files[0]!, + layerId: resolution.layerId, + onUploaded: (owner) => void invalidateGallery(queryClient, owner), + }).then((outcome) => showNotice(getControlUploadNotice(outcome))); + }, + [document, engine, interactionCapabilities.isDocumentEditingLocked, isInteractionLocked, queryClient, showNotice] + ); const handleSurfaceContextMenu = useCallback( (event: ReactMouseEvent) => { // Preserve native menus in inline editors. Canvas context actions target the selected layer without changing @@ -236,7 +297,7 @@ export const CanvasWidgetView = ({ runtime, instance }: WidgetViewProps) => { const execute = async (): Promise => { try { - const result = await executeCanvasImageDropImport({ + const notice = await executeCanvasImageDropImport({ destination: resolution.destination, canvas: canvasCommands, engine: mountedEngine, @@ -244,7 +305,6 @@ export const CanvasWidgetView = ({ runtime, instance }: WidgetViewProps) => { imageNames: resolution.imageNames, project, }); - const notice = getCanvasImportNotice(result); notifications.add({ kind: notice.kind, title: t(notice.titleKey, notice.options ?? {}) }); } catch (error: unknown) { recordCanvasImportError({ @@ -554,6 +614,8 @@ export const CanvasWidgetView = ({ runtime, instance }: WidgetViewProps) => { position="relative" role="region" w="full" + onDragOver={handleFileDragOver} + onDrop={handleFileDrop} > {engine ? ( { {engine ? ( <> + ; + +// The document index hands back the same node until it changes, so other layers' edits do not re-render this. +const selectSelectedControlLayer = (project: { + canvas: { document: Pick }; +}): CanvasControlLayerContract | null => { + const node = getDocumentNode(project.canvas.document, project.canvas.document.selectedLayerId); + return node?.type === 'control' ? node : null; +}; + +// Whether the selected layer draws at all: enabled and shown, with every ancestor. +const selectSelectedLayerDisplayed = (project: { canvas: { document: CanvasDocumentContractV3 } }): boolean => { + const { document } = project.canvas; + const leaf = document.selectedLayerId ? lookupDocumentLeaf(document, document.selectedLayerId) : null; + return !!leaf && leaf.contributionEnabled && !leaf.documentHidden; +}; + +// Below the center view's header row, as the tool strip sits. +const TOP_SLOT_TOP = 'var(--wb-center-chrome-inset, var(--chakra-spacing-2))'; + +/** + * Canvas chrome for the selected control layer: a hint over an empty layer pointing at the image sources, a + * top-center chip while its map is detected in the background, and in its place once the map is current, an + * Original / Map / Overlay switch comparing the map with the picture it was detected from. + */ +export const ControlLayerCanvasOverlay = ({ + engine, + isInteractionLocked, +}: { + engine: OverlayEngine; + isInteractionLocked: boolean; +}) => { + const layer = useActiveProjectSelector(selectSelectedControlLayer); + return layer ? ( + + ) : null; +}; + +const SelectedControlLayerOverlay = ({ + engine, + isInteractionLocked, + layer, +}: { + engine: OverlayEngine; + isInteractionLocked: boolean; + layer: CanvasControlLayerContract; +}) => { + const { t } = useTranslation(); + const activeTool = useCanvasActiveTool(engine); + const status = useControlDetectionStatus(engine, layer.id); + const { active } = useDndContext(); + const subscribeSource = useCallback( + (listener: () => void) => subscribeControlSourceSnapshot(engine, listener), + [engine] + ); + const getSourceSnapshot = useCallback(() => getControlSourceSnapshot(engine), [engine]); + const isUploading = useExternalStoreSelector( + subscribeSource, + getSourceSnapshot, + useCallback((snapshot: ControlSourceSnapshot) => snapshot.uploading.has(layer.id), [layer.id]) + ); + const { notifications } = useWorkbenchCommands(); + const isDocumentEditingLocked = useCanvasDocumentEditingLocked(engine); + const isDisplayed = useActiveProjectSelector(selectSelectedLayerDisplayed); + const controlView = useControlLayerView(engine); + // The requested view shows at once; the engine's view takes over when the request settles. + const [pendingView, setPendingView] = useState<{ layerId: string; mode: ControlLayerViewMode } | null>(null); + const viewOptions = useMemo( + () => [ + { label: t('widgets.canvas.controlView.original'), value: 'original' }, + { label: t('widgets.canvas.controlView.map'), value: 'map' }, + { label: t('widgets.canvas.controlView.overlay'), value: 'overlay' }, + ], + [t] + ); + const showView = useCallback( + (value: string) => { + const request = { layerId: layer.id, mode: value as ControlLayerViewMode }; + setPendingView(request); + void engine.previews.setControlLayerView(request.layerId, request.mode).then((outcome) => { + setPendingView((current) => (current === request ? null : current)); + // Other refusals explain themselves: the switch is hidden, or the engine's view already reflects them. + if (outcome === 'failed' || outcome === 'over-budget') { + notifications.add({ + kind: 'error', + title: t( + outcome === 'failed' ? 'widgets.canvas.controlView.failed' : 'widgets.canvas.controlView.overBudget' + ), + }); + } + }); + }, + [engine, layer.id, notifications, t] + ); + const shownView = + pendingView?.layerId === layer.id ? pendingView.mode : controlView?.layerId === layer.id ? controlView.view : 'map'; + const persistedState = getControlImageState(layer, false); + // Only a source whose strokes may be unsaved asks the engine cache; every other layer reads a constant. + const imageState = useCanvasEngineRead(engine, () => + persistedState === 'empty' && !layer.original + ? getControlImageState(layer, engine.exports.hasExportableLayerContent(layer.id)) + : persistedState + ); + const isGalleryDrag = isGalleryImageDragData(active?.data.current); + // Draw hides the hint: the user is about to give the layer its pixels by hand. A locked surface (staging, a running + // canvas generation) refuses drops, and a gallery drag shows the drop zones in its place. + // An upload on its way (from Properties or a dropped file) keeps the card up, saying so. + const isHintVisible = + imageState === 'empty' && + (isUploading || (!isInteractionLocked && !isGalleryDrag && activeTool !== 'brush' && activeTool !== 'eraser')); + // Idle status keeps the switch and the detection chip apart: one or the other occupies the slot. The engine refuses + // a view while the canvas is locked, and a layer that does not draw has nothing to compare. + const isViewSwitchVisible = + layer.original !== undefined && + !!layer.detection && + isControlMapCurrent(layer) && + isDisplayed && + status.status === 'idle' && + !isInteractionLocked && + !isDocumentEditingLocked && + !isGalleryDrag; + const detector = + status.status === 'processing' + ? status.detector + : status.status === 'idle' + ? null + : (layer.detection?.type ?? null); + const detectorLabel = detector ? t(`widgets.layers.control.filters.${detector}`, detector) : null; + + return ( + <> + {imageState === 'empty' ? ( + + + + {isUploading ? ( + + ) : ( + + )} + + + {t(isUploading ? 'widgets.canvas.controlHint.uploading' : 'widgets.canvas.controlHint.title')} + + {isUploading ? null : ( + + {t('widgets.canvas.controlHint.body')} + + )} + + + + + ) : null} + {/* The top-center slot; the live region stays mounted so a run that starts or fails is announced. */} + + + {detectorLabel && status.status !== 'idle' ? ( + + + {status.status === 'error' ? ( + + ) : ( + + )} + + {status.status === 'error' + ? t('widgets.layers.control.detectionFailed', { + detector: detectorLabel, + reason: t(`widgets.layers.control.detectionErrors.${status.error.code}`), + }) + : t( + status.status === 'queued' + ? 'widgets.layers.control.detectionQueued' + : 'widgets.layers.control.detecting', + { detector: detectorLabel } + )} + + + + ) : null} + + {isViewSwitchVisible ? ( + + + + ) : null} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasHotkeyCommands.test.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasHotkeyCommands.test.ts index 266efaf4a2c..4a35ce97902 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasHotkeyCommands.test.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasHotkeyCommands.test.ts @@ -9,10 +9,9 @@ import type { } from '@workbench/canvas-engine/api'; import type { CanvasProjectMutation } from '@workbench/canvasProjectMutations'; -import { createDocumentModel } from '@workbench/canvas-engine/api'; +import { createControlLayer, createDocumentModel } from '@workbench/canvas-engine/api'; import { groupContract, stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; import { stackTopAnchor } from '@workbench/canvas-engine/document/insertionAnchors.testStub'; -import { createControlLayer } from '@workbench/widgets/layers/layerOps'; import { beforeEach, describe, expect, it, vi, type Mock } from 'vitest'; import { executeCanvasHotkeyCommand, type CanvasHotkeyContext } from './canvasHotkeyCommands'; @@ -161,7 +160,7 @@ describe('staging precedence', () => { describe('layer navigation without staging', () => { const select = (id: string): CanvasProjectMutation => ({ id, type: 'setCanvasSelectedLayer' }); // Document order puts the control stack above the raster stack: b, a, c. - const layers = [rasterLayer('a'), createControlLayer('Control', 'b'), rasterLayer('c')]; + const layers = [rasterLayer('a'), createControlLayer({ id: 'b', name: 'Control' }), rasterLayer('c')]; it.each([ ['canvas.nextEntity', 'a', 'c'], @@ -659,7 +658,7 @@ describe('group selection', () => { ...documentOf([], selectedLayerId), stacks: stacksFrom(nodes), }); - const control = (id: string): CanvasLayerContract => createControlLayer(id, id); + const control = (id: string): CanvasLayerContract => createControlLayer({ id, name: id }); it('reorders, deletes and duplicates a selected group', () => { const document = nodeDocument([groupContract('g', [rasterLayer('a')]), rasterLayer('b')], 'g'); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasImageDropLayout.test.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasImageDropLayout.test.ts index 127c7ce973f..2c710378821 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasImageDropLayout.test.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/canvasImageDropLayout.test.ts @@ -1,4 +1,5 @@ import type { GalleryImage } from '@features/gallery'; +import type { sendGalleryImageAsControl } from '@workbench/canvas-operations/api'; import type { CanvasEngine } from '@workbench/canvas-operations/createCanvasEngine'; import type { Project } from '@workbench/projectContracts'; import type { WorkbenchCanvasCommands } from '@workbench/workbenchStore'; @@ -115,7 +116,7 @@ describe('executeCanvasImageDropImport', () => { const result = await executeCanvasImageDropImport({ canvas: { apply: () => false } as unknown as WorkbenchCanvasCommands, - destination: 'control', + destination: 'control-resized', engine, getGalleryImages: (imageNames) => { lookupCalls.push([...imageNames]); @@ -133,13 +134,72 @@ describe('executeCanvasImageDropImport', () => { expect(lookupCalls).toEqual([['first.png', 'second.png']]); expect(importCalls).toHaveLength(1); expect(importCalls[0]).toMatchObject({ - destination: 'control', + destination: 'control-resized', engine, getProject: queries.getProject, images: [first, second], isActiveProject: queries.isActiveProject, project, }); - expect(result).toEqual({ failedImageNames: [], layerIds: ['layer-1', 'layer-2'], status: 'imported' }); + expect(result).toEqual({ kind: 'success', options: { count: 2 }, titleKey: 'widgets.canvas.import.success' }); + }); + + describe('one image on the control zone', () => { + const image = { imageName: 'one.png' } as GalleryImage; + const project = { id: 'captured-project' } as Project; + const drop = ( + destination: 'control' | 'control-resized', + sent: Awaited> + ) => { + const sendCalls: unknown[] = []; + const importCalls: unknown[] = []; + const result = executeCanvasImageDropImport({ + canvas: { apply: () => false } as unknown as WorkbenchCanvasCommands, + destination, + engine: null, + getGalleryImages: () => Promise.resolve([image]), + imageNames: ['one.png'], + importGalleryImages: (options) => { + importCalls.push(options); + return Promise.resolve({ failedImageNames: [], layerIds: ['new-layer'], status: 'imported' }); + }, + project, + queries: { getProject: () => project, isActiveProject: () => true }, + sendAsControl: (options) => { + sendCalls.push(options); + return Promise.resolve(sent); + }, + }); + return { importCalls, result, sendCalls }; + }; + + it('fills the selected empty control layer and names it, as Use as Control does', async () => { + const { importCalls, result, sendCalls } = drop('control', { + layerId: 'control-1', + layerName: 'Depth 1', + status: 'filled', + }); + + await expect(result).resolves.toEqual({ + kind: 'success', + options: { name: 'Depth 1' }, + titleKey: 'widgets.canvas.import.controlFilled', + }); + expect(sendCalls).toEqual([expect.objectContaining({ image, project })]); + expect(importCalls).toEqual([]); + }); + + it('reports a new layer as an import, and leaves the resized zone to the import', async () => { + await expect(drop('control', { layerId: 'control-2', status: 'created' }).result).resolves.toEqual({ + kind: 'success', + options: { count: 1 }, + titleKey: 'widgets.canvas.import.success', + }); + + const resized = drop('control-resized', { layerId: 'unused', status: 'created' }); + await resized.result; + expect(resized.sendCalls).toEqual([]); + expect(resized.importCalls).toEqual([expect.objectContaining({ destination: 'control-resized' })]); + }); }); }); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/engineStoreHooks.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/engineStoreHooks.ts index 0629fcae787..0bd2c2fae84 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/engineStoreHooks.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/engineStoreHooks.ts @@ -20,7 +20,9 @@ import type { import { getCanvasOperations, + IDLE_CONTROL_DETECTION_STATUS, type CanvasOperationState, + type ControlDetectionStatus, type FilterOperationSessionState, type SamSessionSnapshot, } from '@workbench/canvas-operations/api'; @@ -106,6 +108,10 @@ export const useCanvasZoom = (engine: CanvasCoreStoreCapability): number => useC export const useCanvasViewportReady = (engine: CanvasCoreStoreCapability): boolean => useCanvasInteractionState(engine, 'viewportReady'); +/** The control layer the canvas shows as its original (or original under map); null while every map shows. */ +export const useControlLayerView = (engine: CanvasCoreStoreCapability): CanvasInteractionState['controlLayerView'] => + useCanvasInteractionState(engine, 'controlLayerView'); + /** The active tool id for `engine`. */ export const useCanvasActiveTool = (engine: CanvasCoreStoreCapability): ToolId => useCanvasInteractionState(engine, 'activeTool'); @@ -142,6 +148,20 @@ export const useFilterSession = (engine: object): FilterOperationSessionState | ); }; +/** One control layer's background detection status; a null engine reads as idle without subscribing. */ +export const useControlDetectionStatus = (engine: object | null, layerId: string): ControlDetectionStatus => { + const subscribe = useCallback( + (listener: () => void) => + engine ? getCanvasOperations(engine).subscribeControlDetectionStatus(layerId, listener) : () => undefined, + [engine, layerId] + ); + const getSnapshot = useCallback( + () => (engine ? getCanvasOperations(engine).getControlDetectionStatus(layerId) : IDLE_CONTROL_DETECTION_STATUS), + [engine, layerId] + ); + return useSyncExternalStore(subscribe, getSnapshot, getSnapshot); +}; + /** Whether the engine-owned canvas history has an entry to undo (enables the header undo button). */ export const useCanvasCanUndo = (engine: CanvasCoreStoreCapability): boolean => useCanvasInteractionState(engine, 'canUndo'); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/executeCanvasImageDropImport.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/executeCanvasImageDropImport.ts index 67d216f907f..a75df9da0f6 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/executeCanvasImageDropImport.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/executeCanvasImageDropImport.ts @@ -4,15 +4,19 @@ import type { WorkbenchCanvasCommands, WorkbenchQueries } from '@workbench/workb import { galleryImages } from '@features/gallery'; import { + getCanvasImportNotice, + getSendAsControlNotice, importGalleryImagesToCanvas, + sendGalleryImageAsControl, + type CanvasImportNotice, type GalleryCanvasImportDestination, - type ImportGalleryImagesResult, } from '@workbench/canvas-operations/api'; import { orderCanvasImageDropImages } from './canvasImageDnd'; -type CanvasImageDropEngine = Pick; +type CanvasImageDropEngine = Pick; +/** Imports the dropped gallery images and says how it went, as the gallery menu's matching action does. */ export const executeCanvasImageDropImport = async ({ destination, canvas, @@ -22,6 +26,7 @@ export const executeCanvasImageDropImport = async ({ imageNames, importGalleryImages = importGalleryImagesToCanvas, project, + sendAsControl = sendGalleryImageAsControl, }: { destination: GalleryCanvasImportDestination; canvas: WorkbenchCanvasCommands; @@ -31,17 +36,23 @@ export const executeCanvasImageDropImport = async ({ imageNames: readonly string[]; importGalleryImages?: typeof importGalleryImagesToCanvas; project: Project; -}): Promise => { + sendAsControl?: typeof sendGalleryImageAsControl; +}): Promise => { const fetchedImages = await getGalleryImages([...imageNames]); const images = orderCanvasImageDropImages(imageNames, fetchedImages); - - return importGalleryImages({ + const shared = { applyCanvasMutation: canvas.apply, - destination, engine, getProject: queries.getProject, - images, isActiveProject: queries.isActiveProject, project, - }); + }; + + // One image dropped as a control layer fills the selected control layer still waiting for one, as Use as Control + // does; the resized zone always adds a layer, since its image is made for the generation size. + const [image] = images; + if (destination === 'control' && images.length === 1 && image) { + return getSendAsControlNotice(await sendAsControl({ ...shared, image })); + } + return getCanvasImportNotice(await importGalleryImages({ ...shared, destination, images })); }; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts index c13a1a6bbc7..95fffc232a4 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.test.ts @@ -151,6 +151,7 @@ interface HarnessOptions { outputOnlyMaskedRegions?: boolean; scaling?: RunCanvasInvocationDeps['scaling']; generateValues?: Record; + controlModelMemory?: RunCanvasInvocationDeps['controlModelMemory']; } const makeHarness = (options: HarnessOptions = {}): Harness => { @@ -258,6 +259,7 @@ const makeHarness = (options: HarnessOptions = {}): Harness => { const deps: RunCanvasInvocationDeps = { composeForGeneration: (composeOptions) => composeForGeneration(host, composeOptions), + controlModelMemory: options.controlModelMemory ?? {}, compositing: { ...DEFAULT_CANVAS_COMPOSITING, outputOnlyMaskedRegions: options.outputOnlyMaskedRegions ?? true, @@ -275,8 +277,7 @@ const makeHarness = (options: HarnessOptions = {}): Harness => { }, flushPendingUploads: flushPendingUploads as () => Promise, // Production words rejections from the locale; the harness records the structured rejection it was handed. - formatControlLayerError: ({ code, layerName, suggestedKind }) => - `${layerName}: ${code}${suggestedKind ? ` -> ${suggestedKind}` : ''}`, + formatControlLayerError: ({ code, layerName }) => `${layerName}: ${code}`, generateValues: { ...generateValuesFor(model), ...options.generateValues }, inFlight: options.inFlight ?? new Set(), models: options.models, @@ -990,13 +991,136 @@ describe('runCanvasInvocation', () => { expect(harness.releaseRasterSnapshot).toHaveBeenCalledTimes(1); }); - it('blocks invocation and hands a nonempty model-less control layer to the formatter', async () => { + it('blocks a content-bearing control layer while the model list is still loading', async () => { const harness = makeHarness({ document: docWithLayers([controlLayer('control')]) }); await runCanvasInvocation(harness.deps); expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices().map((notice) => notice.message)).toEqual(['control: models_unavailable']); + }); + + it('blocks an untyped control layer whose model is no longer installed', async () => { + const harness = makeHarness({ + document: docWithLayers([controlLayer('control', { model: 'uninstalled' })]), + models: [], + }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); expect(harness.notices().map((notice) => notice.message)).toEqual(['control: missing_model']); }); + it('leaves out a nonempty control layer with neither a type nor a model, without blocking', async () => { + const harness = makeHarness({ document: docWithLayers([controlLayer('control')]), models: [] }); + await runCanvasInvocation(harness.deps); + expect(harness.notices()).toHaveLength(0); + expect(harness.submittedGraphs()).toHaveLength(1); + expect(harness.submittedGraphs()[0]?.graph.backendGraph?.nodes).not.toHaveProperty('control_net_control'); + }); + + describe('typed control layers', () => { + const controlModel = (key: string, base: string, type = 'controlnet', name = key) => ({ + base, + file_size: 1, + format: 'checkpoint', + hash: `${key}-hash`, + key, + name, + path: key, + source: key, + source_type: 'path' as const, + type, + }); + const typed = ( + id: string, + controlType: NonNullable, + detection: CanvasControlLayerContract['detection'] = null + ): CanvasControlLayerContract => ({ ...controlLayer(id), controlType, detection }); + const fluxMain = { ...sd1Model, base: 'flux' as const, key: 'flux', name: 'FLUX' }; + + it('runs the only model made for the type, as the kind that model runs', async () => { + const harness = makeHarness({ + document: docWithLayers([typed('depth-layer', 'depth')]), + models: [controlModel('sd1-depth', 'sd-1', 'controlnet', 'control_v11f1p_sd15_depth')], + }); + + await runCanvasInvocation(harness.deps); + + expect(harness.notices()).toHaveLength(0); + expect(harness.submittedGraphs()[0]?.graph.backendGraph?.nodes['control_net_depth-layer']).toMatchObject({ + control_model: { key: 'sd1-depth' }, + type: 'controlnet', + }); + }); + + it('runs the remembered pick when several models fit, and asks without one', async () => { + const models = [ + controlModel('depth-a', 'sd-1', 'controlnet', 'depth a'), + controlModel('depth-b', 'sd-1', 'controlnet', 'depth b'), + ]; + const asking = makeHarness({ document: docWithLayers([typed('depth-layer', 'depth')]), models }); + await runCanvasInvocation(asking.deps); + expect(asking.submittedGraphs()).toHaveLength(0); + expect(asking.notices()[0]?.message).toBe('depth-layer: choose_model'); + + const remembered = makeHarness({ + controlModelMemory: { 'sd-1:depth': 'depth-b' }, + document: docWithLayers([typed('depth-layer', 'depth')]), + models, + }); + await runCanvasInvocation(remembered.deps); + expect(remembered.notices()).toHaveLength(0); + expect( + remembered.submittedGraphs()[0]?.graph.backendGraph?.nodes['control_net_depth-layer']?.control_model + ).toMatchObject({ key: 'depth-b' }); + }); + + it('gives a FLUX union model the InstantX mode of the layer type', async () => { + const components = { + clipEmbedModel: { base: 'any', key: 'clip', name: 'CLIP Embed', type: 'clip_embed', variant: 'large' }, + t5EncoderModel: { base: 'any', key: 't5', name: 'T5 Encoder', type: 't5_encoder' }, + vae: { base: 'flux', key: 'flux-vae', name: 'FLUX VAE', type: 'vae' }, + }; + const harness = makeHarness({ + document: docWithLayers([typed('pose-layer', 'pose')]), + generateValues: components, + model: fluxMain, + models: [ + controlModel('flux-union', 'flux', 'controlnet', 'FLUX.1-dev-Controlnet-Union-Pro'), + ...(Object.values(components) as unknown as NonNullable), + ], + }); + + await runCanvasInvocation(harness.deps); + + expect(harness.notices()).toEqual([]); + expect(harness.submittedGraphs()[0]?.graph.backendGraph?.nodes['control_net_pose-layer']).toMatchObject({ + instantx_control_mode: 4, + type: 'flux_controlnet', + }); + }); + + it('blocks a layer whose type has no installed model', async () => { + const harness = makeHarness({ + document: docWithLayers([typed('pose-layer', 'pose')]), + models: [controlModel('sd1-depth', 'sd-1', 'controlnet', 'sd15 depth')], + }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices()[0]?.message).toBe('pose-layer: no_model_for_type'); + }); + + it('blocks a layer whose map detection has not caught up', async () => { + const harness = makeHarness({ + document: docWithLayers([ + typed('depth-layer', 'depth', { settings: {}, type: 'depth_anything_depth_estimation' }), + ]), + models: [controlModel('sd1-depth', 'sd-1', 'controlnet', 'sd15 depth')], + }); + await runCanvasInvocation(harness.deps); + expect(harness.submittedGraphs()).toHaveLength(0); + expect(harness.notices()[0]?.message).toBe('depth-layer: detection_pending'); + }); + }); + it('blocks invocation when a nonempty control layer has malformed numeric settings', async () => { const model = { base: 'sd-1', @@ -1400,9 +1524,9 @@ describe('runCanvasInvocation', () => { ['the same model twice', () => [lllite('a', sketch.key), lllite('b', sketch.key)], 'b: duplicate_lllite_model'], ['an inpainting adapter', () => [lllite('a', inpainting.key)], 'a: lllite_inpaint_adapter'], [ - 'a layer saved as ControlNet before LLLite was supported, naming the kind to switch to', + 'a layer saved as ControlNet before LLLite was supported', () => [controlLayer('old', { kind: 'controlnet', model: sketch.key })], - 'old: switch_adapter_kind -> anima_lllite', + 'old: switch_adapter_kind', ], ])('blocks %s before submitting', async (_label, layers, message) => { const harness = await run(layers()); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts index ced828a3c9c..eacb2e38ff3 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/invoke/prepareCanvasInvocation.ts @@ -16,20 +16,18 @@ import type { RegionalGuidanceReferenceImage, } from '@workbench/canvas-engine/api'; import type { ComposeForGenerationOptions, ComposeForGenerationResult } from '@workbench/canvas-operations/api'; +import type { ControlLayerIssueCode } from '@workbench/controlLayerChecks'; +import type { ControlModelMemory } from '@workbench/controlModelResolution'; import type { ExpandedPositivePrompts, ResultDestination } from '@workbench/invocationContracts'; import type { WorkbenchNotificationKind } from '@workbench/projectContracts'; import type { WorkbenchCommands, WorkbenchNotificationCommands } from '@workbench/workbenchStore'; import { compileCanvasGraph, type CanvasCompileMode } from '@features/generation/canvasGraph'; import { - type ControlAdapterKind, - createControlValidationSequence, detectCanvasMode, - getSuggestedControlKind, getRegionalGuidanceRejectionReason, getRegionalGuidanceSupport, type ControlLayerGraphInput, - type ControlValidationReason, type RegionalGuidanceInput, type RegionalReferenceImageInput, resolveGenerateSeed, @@ -47,6 +45,8 @@ import { type AccountScope, } from '@platform/state/accountLifecycle'; import { getCanvasEngine, getCanvasOperations } from '@workbench/canvas-operations/api'; +import { createControlLayerCheckContext, createControlLayerSequence } from '@workbench/controlLayerChecks'; +import { getWorkbenchPreferences } from '@workbench/settings/store'; import { DEFAULT_CANVAS_COMPOSITING, type CanvasCompositingSettings, @@ -59,17 +59,15 @@ export const CANVAS_INVOKE_ERROR_TITLE = 'Canvas generation failed'; /** A control layer that blocks the invocation, as `formatControlLayerError` words it for the notice. */ export interface ControlLayerRejection { - code: ControlValidationReason; + code: ControlLayerIssueCode; layerName: string; - suggestedKind: ControlAdapterKind | null; } /** Structured control-layer rejection: callers word it from the locale; the message is for logs only. */ export class ControlLayerValidationError extends Error implements ControlLayerRejection { constructor( - readonly code: ControlValidationReason, - readonly layerName: string, - readonly suggestedKind: ControlAdapterKind | null + readonly code: ControlLayerIssueCode, + readonly layerName: string ) { super(`Control layer "${layerName}" is invalid: ${code}.`); this.name = 'ControlLayerValidationError'; @@ -104,6 +102,8 @@ export interface RunCanvasInvocationDeps { generateValues: Record; /** Loaded models, for the same value/model sync the generate path performs. */ models?: readonly ModelConfig[]; + /** Remembered control-model picks, which resolve control layers left on automatic. */ + controlModelMemory: ControlModelMemory; /** The project settings the compiler consults. */ projectSettings: GenerationProjectSettings; /** Runtime-derived accelerator family recorded in generation metadata. */ @@ -128,18 +128,22 @@ const recordNotice = ( }; /** - * Validate enabled content-bearing controls in z-order and resolve adapters; invalid controls throw and block - * invocation. Join uploaded names with captured model metadata for graph inputs. + * Validate enabled content-bearing controls in z-order with the same judge Invoke readiness uses, resolving each + * layer's model; invalid controls throw and block invocation, and layers with no type or model are left out. Join + * uploaded names with captured model metadata for graph inputs. */ const createControlLayerCollector = ( model: GenerateModelConfig, - models: readonly ModelConfig[] | undefined + models: readonly ModelConfig[] | undefined, + remembered: ControlModelMemory ): { shouldComposite(layer: CanvasControlLayerContract): boolean; toGraphInputs(images: readonly { layerId: string; imageName: string }[]): ControlLayerGraphInput[]; } => { const metadata = new Map>(); - const validate = createControlValidationSequence(model); + const judge = models + ? createControlLayerSequence(model, models, createControlLayerCheckContext(model.base, remembered)) + : null; return { shouldComposite: (layer) => { @@ -147,25 +151,24 @@ const createControlLayerCollector = ( throw new Error(describeExternalCanvasRefusal(model, { kind: 'control_layer', layerName: layer.name })); } const { adapter } = layer; - const resolved = adapter.model ? models?.find((candidate) => candidate.key === adapter.model) : undefined; - const rejection = validate({ - adapterModel: resolved ?? null, - beginEndStepPct: adapter.beginEndStepPct, - kind: adapter.kind, - weight: adapter.weight, - }); - if (rejection || !resolved) { - throw new ControlLayerValidationError( - rejection ?? 'missing_model', - layer.name, - rejection === 'switch_adapter_kind' ? getSuggestedControlKind(model.base) : null - ); + // Readiness skips control checks until models load; resolving against nothing would drop or misjudge layers. + if (!judge) { + throw new ControlLayerValidationError('models_unavailable', layer.name); + } + const { reason, resolution, skip } = judge(layer); + if (skip) { + return false; + } + const resolved = resolution.model; + if (reason || !resolved) { + throw new ControlLayerValidationError(reason ?? 'missing_model', layer.name); } metadata.set(layer.id, { beginEndStepPct: adapter.beginEndStepPct, controlMode: adapter.controlMode, id: layer.id, - kind: adapter.kind, + instantxControlMode: resolution.instantxControlMode, + kind: resolution.kind, model: { base: resolved.base, key: resolved.key, @@ -335,7 +338,7 @@ export const runCanvasInvocation = async (deps: RunCanvasInvocationDeps): Promis // Flush persisted paint before capture so planning sees current bitmap references and bbox. await deps.flushPendingUploads(); - const controls = createControlLayerCollector(model, deps.models); + const controls = createControlLayerCollector(model, deps.models, deps.controlModelMemory); const regions = createRegionalGuidanceCollector(model); const composed = await deps.composeForGeneration({ // Mode-union drift guard: `detectCanvasMode`'s facts/mode types must stay @@ -484,6 +487,7 @@ export const prepareCanvasInvocation = async (args: PrepareCanvasInvocationArgs) commands: args.commands, expansion: args.expansion, composeForGeneration: (composeOptions) => operations.composeForGeneration(composeOptions), + controlModelMemory: getWorkbenchPreferences().controlModelMemory, flushPendingUploads: () => engine.lifecycle.flushPendingUploads(), formatControlLayerError: args.formatControlLayerError, generateValues: args.generateValues, diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-options/SelectionOptionsRow.test.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-options/SelectionOptionsRow.test.tsx index b648af9c6aa..4adf1fe4896 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-options/SelectionOptionsRow.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-options/SelectionOptionsRow.test.tsx @@ -5,8 +5,9 @@ import type { ReactNode } from 'react'; import { ChakraProvider } from '@chakra-ui/react'; import { system } from '@theme/system'; +import { createControlLayer } from '@workbench/canvas-engine/api'; import { stacksFrom } from '@workbench/canvas-engine/document-model/documentFixtures.testStub'; -import { createControlLayer, createEmptyPaintLayer } from '@workbench/widgets/layers/layerOps'; +import { createEmptyPaintLayer } from '@workbench/widgets/layers/layerOps'; import { createDraftProject } from '@workbench/workbenchState'; import { renderToStaticMarkup } from 'react-dom/server'; import { beforeEach, describe, expect, it, vi } from 'vitest'; @@ -81,14 +82,14 @@ beforeEach(() => { describe('SelectionActions select all', () => { it('stays available with no selection and no eligible layer', () => { hasSelection.current = false; - const captured = renderRow(createControlLayer('Control', 'control-1', 'sd-1', null)); + const captured = renderRow(createControlLayer({ base: 'sd-1', id: 'control-1', name: 'Control' })); expect(captured.get('widgets.canvas.toolOptions.selectAll')?.['aria-disabled']).toBe(false); }); }); describe('SelectionActions pixel target eligibility', () => { const rasterPaint = createEmptyPaintLayer('Raster', 'raster'); - const controlPaint = createControlLayer('Control', 'control'); + const controlPaint = createControlLayer({ id: 'control', name: 'Control' }); const rasterImage: CanvasLayerContract = { ...createEmptyPaintLayer('Raster image', 'raster-image'), source: { image: { height: 10, imageName: 'raster-image', width: 10 }, type: 'image' }, diff --git a/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-presentation/PropertyPrimitives.tsx b/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-presentation/PropertyPrimitives.tsx index 05c550029c3..f478076dc5b 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-presentation/PropertyPrimitives.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/canvas/tool-presentation/PropertyPrimitives.tsx @@ -1,9 +1,10 @@ -import type { ReactNode } from 'react'; +import type { PointerEvent as ReactPointerEvent, ReactNode } from 'react'; import { Badge, chakra, Flex, Grid, Icon, Stack, Switch, Text } from '@chakra-ui/react'; import { SegmentedControl } from '@platform/ui/SegmentedControl'; +import { Tooltip } from '@platform/ui/Tooltip'; import { ChevronDownIcon } from 'lucide-react'; -import { useCallback } from 'react'; +import { useCallback, useState } from 'react'; import { useTranslation } from 'react-i18next'; import { setPropertyGroupCollapsed, usePropertyGroupCollapsed } from './propertyGroupStore'; @@ -11,14 +12,46 @@ import { setPropertyGroupCollapsed, usePropertyGroupCollapsed } from './property const GROUP_HEADER_HOVER = { color: 'fg' } as const; /** - * Use fixed label/trailing columns around a flexible control so sliders and numbers never wrap; blank labels - * preserve continuation alignment. + * A row label that names itself in a tooltip while the pointer is on it and its column cuts it short. Only then is a + * tooltip (and its portal) mounted, so the many rows of a Properties pane cost none at rest. */ -export const PropertyControlRow = ({ children, label }: { children: ReactNode; label?: string }) => ( - - +const PropertyRowLabel = ({ label, labelId }: { label: string; labelId?: string }) => { + const [isTipShown, setIsTipShown] = useState(false); + const showIfCut = useCallback((event: ReactPointerEvent) => { + const element = event.currentTarget; + setIsTipShown(element.scrollWidth > element.clientWidth); + }, []); + const hide = useCallback(() => setIsTipShown(false), []); + const text = ( + {label} + ); + return isTipShown ? ( + + {text} + + ) : ( + text + ); +}; + +/** + * Use fixed label/trailing columns around a flexible control so sliders and numbers never wrap; blank labels + * preserve continuation alignment. `labelId` names the label so a control in the row can point + * `aria-labelledby` at it. + */ +export const PropertyControlRow = ({ + children, + label, + labelId, +}: { + children: ReactNode; + label?: string; + labelId?: string; +}) => ( + + {label ? : } {children} ); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlAdapterRows.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlAdapterRows.tsx new file mode 100644 index 00000000000..cd91960a873 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlAdapterRows.tsx @@ -0,0 +1,229 @@ +import type { SliderValueChangeDetails } from '@chakra-ui/react'; +import type { CanvasControlAdapterContract } from '@workbench/canvas-engine/api'; + +import { Box, Text } from '@chakra-ui/react'; +import { getControlAdapterMinWeight } from '@features/generation/graph'; +import { SegmentedControl } from '@platform/ui/SegmentedControl'; +import { Slider } from '@platform/ui/Slider'; +import { Tooltip } from '@platform/ui/Tooltip'; +import { + FormNumberField, + useNumberCommit, + useSliderGesture, +} from '@workbench/widgets/canvas/tool-presentation/FormControls'; +import { PropertyControlRow } from '@workbench/widgets/canvas/tool-presentation/PropertyPrimitives'; +import { useCallback, useId, useMemo, useState } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { formatPanelText, type ControlPanelModel, type ControlPriority } from './controlPanelModel'; +import { CONTROL_WEIGHT_BOUNDS } from './layerOps'; + +const STEP_RANGE_STEP = 0.05; +const PRIORITIES: readonly ControlPriority[] = ['more_prompt', 'balanced', 'more_control']; +// A layer that stored the retired `unbalanced` mode sees it as a fourth segment, so the value is visible and can be left. +const PRIORITIES_WITH_UNBALANCED: readonly (ControlPriority | 'unbalanced')[] = [...PRIORITIES, 'unbalanced']; + +const formatWeight = (value: number): string => value.toFixed(2); +const formatPercent = (value: number): string => `${Math.round(value * 100)}%`; +const clamp = (value: number, min: number, max: number): number => Math.min(max, Math.max(min, value)); + +type AdapterPatch = Partial>; + +export interface ControlAdapterEditing { + /** Shows a gesture's value live; the commit that ends it records one undo step. */ + preview(patch: AdapterPatch): void; + commit(patch: AdapterPatch, label: string): void; + /** Ends a gesture that came back to where it began, restoring what it previewed. */ + cancel(): void; +} + +/** Weight, Steps and Priority: the adapter's own values, edited the same way whatever drives the layer. */ +export const ControlAdapterRows = ({ + adapter, + disabled, + editing, + model, +}: { + adapter: CanvasControlAdapterContract; + disabled: boolean; + editing: ControlAdapterEditing; + model: ControlPanelModel['adapter']; +}) => { + const { i18n, t } = useTranslation(); + const weightLabel = t('widgets.layers.control.weight'); + const stepsLabel = t('widgets.layers.control.rows.steps'); + const priorityLabel = t('widgets.layers.control.rows.priority'); + const priorityReasonId = useId(); + + // Weight: the slider covers 0–2; typed values reach the kind's lower bound. + const commitWeight = useCallback((weight: number) => editing.commit({ weight }, weightLabel), [editing, weightLabel]); + const previewWeight = useCallback((weight: number) => editing.preview({ weight }), [editing]); + const weightGesture = useSliderGesture(adapter.weight, commitWeight, previewWeight); + const weightValue = useMemo( + () => [clamp(weightGesture.value, CONTROL_WEIGHT_BOUNDS.sliderMin, CONTROL_WEIGHT_BOUNDS.sliderMax)], + [weightGesture.value] + ); + const weightAria = useMemo(() => [weightLabel], [weightLabel]); + const handleWeightChange = useCallback( + ({ value }: SliderValueChangeDetails) => weightGesture.onChange(value[0]!), + [weightGesture] + ); + const handleWeightChangeEnd = useCallback( + ({ value }: SliderValueChangeDetails) => { + const next = value[0]!; + weightGesture.onChangeEnd(next); + if (next === adapter.weight) { + editing.cancel(); + } + }, + [adapter.weight, editing, weightGesture] + ); + const minWeight = getControlAdapterMinWeight(model.kind); + const handleWeightTyped = useNumberCommit( + useCallback( + (value: number) => commitWeight(clamp(value, minWeight, CONTROL_WEIGHT_BOUNDS.inputMax)), + [commitWeight, minWeight] + ) + ); + + // Steps: the readout follows the thumbs; one step is recorded on release. + const [rangeDraft, setRangeDraft] = useState<[number, number] | null>(null); + const range = rangeDraft ?? adapter.beginEndStepPct; + const rangeValue = useMemo(() => [range[0], range[1]], [range]); + const rangeAria = useMemo(() => [t('widgets.layers.control.beginStep'), t('widgets.layers.control.endStep')], [t]); + const handleRangeChange = useCallback( + ({ value }: SliderValueChangeDetails) => { + if (value.length === 2) { + const next: [number, number] = [value[0]!, value[1]!]; + setRangeDraft(next); + editing.preview({ beginEndStepPct: next }); + } + }, + [editing] + ); + const handleRangeChangeEnd = useCallback( + ({ value }: SliderValueChangeDetails) => { + setRangeDraft(null); + if (value.length !== 2) { + return; + } + const [begin, end] = adapter.beginEndStepPct; + if (value[0] === begin && value[1] === end) { + editing.cancel(); + return; + } + editing.commit({ beginEndStepPct: [value[0]!, value[1]!] }, stepsLabel); + }, + [adapter.beginEndStepPct, editing, stepsLabel] + ); + const stepsReadout = rangeDraft + ? t('widgets.layers.control.stepsReadout', { + begin: Math.round(rangeDraft[0] * 100), + end: Math.round(rangeDraft[1] * 100), + }) + : formatPanelText(t, model.stepsReadout, i18n.language); + + const isLegacyUnbalanced = model.priority.value === 'unbalanced'; + const priorityOptions = useMemo( + () => + (isLegacyUnbalanced ? PRIORITIES_WITH_UNBALANCED : PRIORITIES).map((value) => ({ + label: t(`widgets.layers.control.priorityModes.${value}`), + value, + })), + [isLegacyUnbalanced, t] + ); + const handlePriority = useCallback( + (value: string) => { + if (value !== model.priority.value) { + editing.commit( + { controlMode: value as NonNullable }, + priorityLabel + ); + } + }, + [editing, model.priority.value, priorityLabel] + ); + const priorityUnavailable = model.priority.unavailable + ? t( + model.priority.unavailable === 'flux' + ? 'widgets.layers.control.priorityUnavailableFlux' + : 'widgets.layers.control.priorityUnavailable' + ) + : null; + + return ( + <> + + + + + + + + {stepsReadout} + + + + {/* Disabled radios take no focus, so the group itself does while unavailable: its reason shows on focus too. */} + + + + {priorityUnavailable ? ( + + ) : null} + + + + + ); +}; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlDetectionRow.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlDetectionRow.tsx new file mode 100644 index 00000000000..0bffefd8e8c --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlDetectionRow.tsx @@ -0,0 +1,213 @@ +import type { SelectValueChangeDetails } from '@chakra-ui/react'; + +import { createListCollection, HStack, Icon, Spinner, Stack, Text } from '@chakra-ui/react'; +import { IconButton } from '@platform/ui/Button'; +import { Select } from '@platform/ui/Select'; +import { getFilterDefinition } from '@workbench/canvas-operations/api'; +import { + setPropertyGroupCollapsed, + usePropertyGroupCollapsed, +} from '@workbench/widgets/canvas/tool-presentation/propertyGroupStore'; +import { PropertyControlRow } from '@workbench/widgets/canvas/tool-presentation/PropertyPrimitives'; +import { CircleAlertIcon, RotateCcwIcon, ScanSearchIcon, SlidersHorizontalIcon } from 'lucide-react'; +import { useCallback, useId, useMemo } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { DETECTION_NONE, formatPanelText, type ControlPanelModel } from './controlPanelModel'; +import { LayerFilterControls } from './LayerFilterControls'; +import { OptionNameSeparator } from './OptionNameSeparator'; + +const SELECT_POSITIONING = { placement: 'bottom-end', sameWidth: true } as const; +const SETTINGS_GROUP_ID = 'control-detection-settings'; +const ACTION_SIZE = '7'; + +interface DetectionItem { + disabled: boolean; + hint: string | null; + label: string; + value: string; +} + +const groupDetection = (item: DetectionItem): string => (item.value === DETECTION_NONE ? 'none' : 'detectors'); +// Neither group needs a header: the detectors open the list and None follows a divider. +const noGroupLabel = (): null => null; +const ignoreFilterTypeChange = (): void => undefined; + +/** + * The Detection row: the type's detectors and "None: use the image as it is", a spinner while one runs, Run again + * after a failure, Detect again for an out-of-date map, and the detector's settings inline under the row. + */ +export const ControlDetectionRow = ({ + disabled, + model, + onDetect, + onDetectorChange, + onSettingsChange, +}: { + disabled: boolean; + model: ControlPanelModel['detection']; + onDetect(): void; + onDetectorChange(detector: string | null): void; + onSettingsChange(settings: Record): void; +}) => { + const { i18n, t } = useTranslation(); + const label = t('widgets.layers.control.rows.detection'); + const regionId = useId(); + const collapsed = usePropertyGroupCollapsed(SETTINGS_GROUP_ID, true); + const format = useCallback( + (text: Parameters[1]) => formatPanelText(t, text, i18n.language), + [i18n.language, t] + ); + + const collection = useMemo( + () => + createListCollection({ + items: model.options.map((option) => ({ + disabled: option.disabled, + hint: option.hint ? format(option.hint) : null, + label: format(option.label), + value: option.value, + })), + }), + [format, model.options] + ); + const value = useMemo(() => (model.value ? [model.value] : []), [model.value]); + const handleChange = useCallback( + ({ value: next }: SelectValueChangeDetails) => { + const choice = next[0]; + if (choice && choice !== model.value) { + onDetectorChange(choice === DETECTION_NONE ? null : choice); + } + }, + [model.value, onDetectorChange] + ); + + const hasParams = model.settings ? (getFilterDefinition(model.settings.detector)?.params.length ?? 0) > 0 : false; + const settingsUnavailable = model.settingsUnavailable + ? format(model.settingsUnavailable) + : hasParams + ? undefined + : t('widgets.layers.control.detectionNoSettings', { detector: format(model.valueText) }); + const open = !collapsed && settingsUnavailable === undefined; + const toggleSettings = useCallback(() => setPropertyGroupCollapsed(SETTINGS_GROUP_ID, !collapsed), [collapsed]); + + const valueText = useMemo( + () => ( + + {model.busy ? : null} + {model.failed ? : null} + + {format(model.valueText)} + + + ), + [format, model.busy, model.failed, model.tone, model.valueText] + ); + + const actionLabel = + model.action === 'retry' + ? t('widgets.layers.control.detectionRetry') + : model.action === 'detect' + ? t('widgets.layers.control.detectionRun') + : null; + + return ( + <> + + - - - 0 - ? hiddenModelReasons.map((reason) => t(`widgets.layers.control.hiddenModels.${reason}`)).join(' ') - : undefined - } - label={t('widgets.layers.control.model')} - > - - - ) : null} - - - - - - - {t('widgets.layers.control.transparencyEffect')} - - - + - {visibleValidationReason ? ( - - - {describeControlLayerReason(t, visibleValidationReason, suggestedKind)} - - {visibleValidationReason === 'switch_adapter_kind' && suggestedKind ? ( - - ) : null} - - ) : null} {showCapabilitiesFailure ? ( diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.browser.test.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.browser.test.tsx similarity index 54% rename from invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.browser.test.tsx rename to invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.browser.test.tsx index 77e6224b3a8..d6a716b5804 100644 --- a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.browser.test.tsx +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.browser.test.tsx @@ -1,5 +1,5 @@ import type { ArchitectureCapabilitiesRow } from '@features/generation/core/architectureCapabilities'; -import type { CanvasLayerContract } from '@workbench/canvas-engine/api'; +import type { CanvasControlLayerContract } from '@workbench/canvas-engine/api'; import { ChakraProvider } from '@chakra-ui/react'; import { architectureCapabilitiesFixture } from '@features/generation/core/architectureCapabilities.testing'; @@ -7,19 +7,24 @@ import { ensureArchitectureCapabilitiesLoaded } from '@features/generation/data/ import { accountLifecycle } from '@platform/state/accountLifecycle'; import { applyThemeToRoot } from '@theme/applyTheme'; import { system } from '@theme/system'; +import { createControlDetectionStatusStore } from '@workbench/canvas-operations/controlDetectionStatusStore'; +import { attachCanvasOperations } from '@workbench/canvas-operations/operationAccess'; import { createInstance } from 'i18next'; import { act } from 'react'; import { createRoot, type Root } from 'react-dom/client'; import { I18nextProvider, initReactI18next } from 'react-i18next'; import { afterEach, describe, expect, it, vi } from 'vitest'; -import { ControlLayerWarningIcon } from './ControlLayerWarningIcon'; +import { ControlLayerStatusIcon } from './ControlLayerStatusIcon'; // Stable identities, as the real stores hand out: a fresh array or model per render would change an // input React Compiler memoises on, and re-read the table by accident rather than by subscription. const catalog = vi.hoisted(() => ({ mainModel: { base: 'sd-1', key: 'sd1-main', name: 'SD 1.5', type: 'main' }, - models: [{ base: 'sd-1', key: 'sd1-controlnet', name: 'SD 1.5 ControlNet', type: 'controlnet' }], + models: [ + { base: 'sd-1', key: 'sd1-controlnet', name: 'SD 1.5 ControlNet', type: 'controlnet' }, + { base: 'sd-1', key: 'sd1-depth', name: 'SD 1.5 Depth', type: 'controlnet' }, + ], })); vi.mock('@features/models', async (importOriginal) => ({ @@ -55,8 +60,24 @@ const layer = { isLocked: false, name: 'Control 1', opacity: 1, + source: { image: { height: 64, imageName: 'control.png', width: 64 }, type: 'image' }, type: 'control', -} as unknown as CanvasLayerContract; +} as unknown as CanvasControlLayerContract; + +/** Typed Depth with a detector whose map has not been produced yet: the map is stale. */ +const pendingDepthLayer = { + ...layer, + adapter: { ...layer.adapter, model: 'sd1-depth' }, + controlType: 'depth', + detection: { settings: {}, type: 'depth_anything_depth_estimation' }, +} as CanvasControlLayerContract; + +const statuses = createControlDetectionStatusStore(); +const engine = {}; +attachCanvasOperations(engine, { + getControlDetectionStatus: statuses.get, + subscribeControlDetectionStatus: statuses.subscribe, +} as never); const settle = async (run: () => void = () => undefined) => { await act(async () => { @@ -67,7 +88,7 @@ const settle = async (run: () => void = () => undefined) => { }); }; -const render = async () => { +const render = async (shown: CanvasControlLayerContract = layer) => { applyThemeToRoot('classic'); host = document.createElement('div'); document.body.append(host); @@ -76,7 +97,7 @@ const render = async () => { root?.render( - + ) @@ -91,11 +112,13 @@ afterEach(async () => { host = null; root = null; getArchitectureCapabilities.mockReset(); + statuses.clear(); + i18n.removeResourceBundle('en', 'translation'); // Returns the capability store and the core registry to their unloaded state between tests. accountLifecycle.invalidate(); }); -describe('ControlLayerWarningIcon and the capability table', () => { +describe('ControlLayerStatusIcon and the capability table', () => { it('flags nothing about a valid adapter while the table is still loading', async () => { // Missing capabilities must not label a valid adapter unsupported. await render(); @@ -118,3 +141,47 @@ describe('ControlLayerWarningIcon and the capability table', () => { expect(warning()).toBeNull(); }); }); + +describe('ControlLayerStatusIcon detection and new reasons', () => { + it('shows a labelled spinner while the layer detects, then the warning detection left behind', async () => { + getArchitectureCapabilities.mockResolvedValueOnce(architectureCapabilitiesFixture); + await settle(ensureArchitectureCapabilitiesLoaded); + await render(pendingDepthLayer); + + // A stale map blocks Invoke, so the row says so until detection catches up. + expect(warning()?.getAttribute('aria-label')).toContain('widgets.layers.control.validation.detection_pending'); + + await settle(() => + statuses.set('control-1', { detector: 'depth_anything_depth_estimation', status: 'processing' }) + ); + + expect(warning()).toBeNull(); + expect(host!.querySelector('.chakra-spinner')).not.toBeNull(); + expect(host!.textContent).toBe('widgets.layers.control.detecting'); + + await settle(() => statuses.set('control-1', { status: 'idle' })); + + expect(host!.querySelector('.chakra-spinner')).toBeNull(); + expect(warning()).not.toBeNull(); + }); + + it('shows the spinner, not the pending warning, while detection waits to run', async () => { + getArchitectureCapabilities.mockResolvedValueOnce(architectureCapabilitiesFixture); + await settle(ensureArchitectureCapabilitiesLoaded); + i18n.addResource('en', 'translation', 'widgets.layers.control.detectionQueued', 'Waiting to run {{detector}}…'); + i18n.addResource( + 'en', + 'translation', + 'widgets.layers.control.filters.depth_anything_depth_estimation', + 'Depth Anything' + ); + await render(pendingDepthLayer); + + await settle(() => statuses.set('control-1', { status: 'queued' })); + + expect(warning()).toBeNull(); + expect(host!.querySelector('.chakra-spinner')).not.toBeNull(); + // The queued status names no detector; the label uses the one the layer asks for, and does not claim it runs. + expect(host!.textContent).toBe('Waiting to run Depth Anything…'); + }); +}); diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.tsx new file mode 100644 index 00000000000..b08dc0a3fb6 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerStatusIcon.tsx @@ -0,0 +1,113 @@ +import type { ArchitectureCapabilitiesSnapshot } from '@features/generation/runtime'; +import type { CanvasControlLayerContract } from '@workbench/canvas-engine/api'; +import type { WorkbenchPreferences } from '@workbench/settings/contracts'; + +import { Box, Icon, Spinner, VisuallyHidden } from '@chakra-ui/react'; +import { getArchitectureCapabilitiesSnapshot, subscribeArchitectureCapabilities } from '@features/generation/runtime'; +import { useModelsSelector } from '@features/models'; +import { useExternalStoreSelector } from '@platform/state/selectors'; +import { Tooltip } from '@platform/ui/Tooltip'; +import { + createControlLayerCheckContext, + describeControlLayer, + describeControlLayerReason, +} from '@workbench/controlLayerChecks'; +import { useWorkbenchPreferenceSelector } from '@workbench/settings/store'; +import { useControlDetectionStatus } from '@workbench/widgets/canvas/engineStoreHooks'; +import { TriangleAlertIcon } from 'lucide-react'; +import { useCallback } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { useSelectedMainModel } from './useSelectedMainModel'; + +const selectCapabilitiesStatus = (snapshot: ArchitectureCapabilitiesSnapshot) => snapshot.status; +const selectControlModelMemory = (preferences: WorkbenchPreferences) => preferences.controlModelMemory; + +/** + * A control layer row's status: a spinner while its map is queued for or being detected, else a warning when the + * layer needs attention, judged as its Properties panel judges it. `contributing` is the effective enablement; a + * layer gated by a group is not validated. + */ +export const ControlLayerStatusIcon = ({ + layer, + contributing = layer.isEnabled, + engine, +}: { + contributing?: boolean; + /** The canvas engine whose detection status the row reflects. */ + engine: object | null; + layer: CanvasControlLayerContract; +}) => { + const { t } = useTranslation(); + const detection = useControlDetectionStatus(engine, layer.id); + const models = useModelsSelector((snapshot) => snapshot.models); + const mainModel = useSelectedMainModel(); + const remembered = useWorkbenchPreferenceSelector(selectControlModelMemory); + const capabilitiesStatus = useExternalStoreSelector( + subscribeArchitectureCapabilities, + getArchitectureCapabilitiesSnapshot, + selectCapabilitiesStatus + ); + // Read inside the store's selector: whether a kind is supported comes from the capability table, and a call + // memoised on the layer and model would keep its first answer after the table loads. + const reason = useExternalStoreSelector( + subscribeArchitectureCapabilities, + getArchitectureCapabilitiesSnapshot, + useCallback( + () => + contributing + ? describeControlLayer(layer, { + ...createControlLayerCheckContext(mainModel?.base ?? null, remembered), + mainModel, + models, + }).reason + : null, + [contributing, layer, mainModel, models, remembered] + ) + ); + + // A queued run (debounce, a free slot, a commit waiting for the document) is as busy as a running one; it names + // the detector the layer asks for, since it has not started one yet. + const detector = + detection.status === 'processing' + ? detection.detector + : detection.status === 'queued' + ? layer.detection?.type + : undefined; + if (detector) { + const label = t( + detection.status === 'queued' ? 'widgets.layers.control.detectionQueued' : 'widgets.layers.control.detecting', + { detector: t(`widgets.layers.control.filters.${detector}`, detector) } + ); + return ( + + + + {label} + + + ); + } + + // Distinguish loading/failed capabilities from unsupported adapters; failures should point to retry. + if (!reason || (reason === 'capabilities_unavailable' && capabilitiesStatus !== 'error')) { + return null; + } + + const message = + reason === 'capabilities_unavailable' + ? `${t('widgets.layers.control.capabilitiesLoadFailed')} ${t('widgets.layers.control.capabilitiesRetryHint')}` + : describeControlLayerReason(t, reason); + + return ( + + + + ); +}; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.tsx deleted file mode 100644 index cecc57a1b28..00000000000 --- a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlLayerWarningIcon.tsx +++ /dev/null @@ -1,75 +0,0 @@ -import type { ArchitectureCapabilitiesSnapshot } from '@features/generation/runtime'; -import type { CanvasLayerContract } from '@workbench/canvas-engine/api'; - -import { Icon } from '@chakra-ui/react'; -import { getSuggestedControlKind } from '@features/generation/graph'; -import { getArchitectureCapabilitiesSnapshot, subscribeArchitectureCapabilities } from '@features/generation/runtime'; -import { useModelsSelector } from '@features/models'; -import { useExternalStoreSelector } from '@platform/state/selectors'; -import { Tooltip } from '@platform/ui'; -import { describeControlLayerReason, getControlLayerAttentionReason } from '@workbench/controlLayerChecks'; -import { TriangleAlertIcon } from 'lucide-react'; -import { useCallback } from 'react'; -import { useTranslation } from 'react-i18next'; - -import { useSelectedMainModel } from './useSelectedMainModel'; - -const selectCapabilitiesStatus = (snapshot: ArchitectureCapabilitiesSnapshot) => snapshot.status; - -/** `contributing` is the effective enablement; a layer gated by a group is not validated. */ -export const ControlLayerWarningIcon = ({ - layer, - contributing = layer.isEnabled, -}: { - layer: CanvasLayerContract; - contributing?: boolean; -}) => { - const { t } = useTranslation(); - const models = useModelsSelector((snapshot) => snapshot.models); - const mainModel = useSelectedMainModel(); - const capabilitiesStatus = useExternalStoreSelector( - subscribeArchitectureCapabilities, - getArchitectureCapabilitiesSnapshot, - selectCapabilitiesStatus - ); - // Read inside the store's selector: whether the adapter kind is supported comes from the capability - // table, and a call memoised on the layer and model would keep its first answer after the table loads. - const reason = useExternalStoreSelector( - subscribeArchitectureCapabilities, - getArchitectureCapabilitiesSnapshot, - useCallback( - () => - layer.type === 'control' && contributing && mainModel - ? getControlLayerAttentionReason(layer, mainModel, models) - : null, - [contributing, layer, mainModel, models] - ) - ); - const suggestedKind = useExternalStoreSelector( - subscribeArchitectureCapabilities, - getArchitectureCapabilitiesSnapshot, - useCallback(() => (mainModel ? getSuggestedControlKind(mainModel.base) : null), [mainModel]) - ); - - // Distinguish loading/failed capabilities from unsupported adapters; failures should point to retry. - if (!reason || (reason === 'capabilities_unavailable' && capabilitiesStatus !== 'error')) { - return null; - } - - const message = - reason === 'capabilities_unavailable' - ? `${t('widgets.layers.control.capabilitiesLoadFailed')} ${t('widgets.layers.control.capabilitiesRetryHint')}` - : describeControlLayerReason(t, reason, suggestedKind); - - return ( - - - - ); -}; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlModelRow.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlModelRow.tsx new file mode 100644 index 00000000000..a68fbcc2821 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlModelRow.tsx @@ -0,0 +1,257 @@ +import type { SelectValueChangeDetails } from '@chakra-ui/react'; +import type { ModelConfig } from '@features/models'; +import type { ControlType } from '@workbench/canvas-engine/api'; +import type { ControlModelResolution } from '@workbench/controlModelResolution'; + +import { Box, createListCollection, HStack, Stack, Text } from '@chakra-ui/react'; +import { ensureStartersLoaded } from '@features/models'; +import { Button } from '@platform/ui/Button'; +import { Select } from '@platform/ui/Select'; +import { Tooltip } from '@platform/ui/Tooltip'; +import { deriveControlKind } from '@workbench/controlAdapters'; +import { classifyControlModel } from '@workbench/controlTypes'; +import { PropertyControlRow } from '@workbench/widgets/canvas/tool-presentation/PropertyPrimitives'; +import { useCallback, useMemo, useRef } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { formatPanelText, type ControlPanelModel, type PanelTone } from './controlPanelModel'; +import { OptionNameSeparator } from './OptionNameSeparator'; + +const SELECT_POSITIONING = { placement: 'bottom-end', sameWidth: true } as const; +const TONE_COLORS: Record = { + default: undefined, + error: 'fg.error', + muted: 'fg.muted', + warning: 'fg.warning', +}; + +type ModelGroup = 'purpose' | 'union' | 'unknown' | 'installed'; + +interface ModelItem { + caption: string; + disabled: boolean; + group: ModelGroup; + label: string; + value: string; +} + +const groupOf = (item: ModelItem): string => item.group; + +/** + * The models a typed layer can pick, grouped: made for its type (or its detector's family), every union model on the + * base (one that does not cover the type is still a deliberate pick), then models whose purpose nothing names. + * Models made for another type stay out. An untyped layer (one from before types) lists every usable model. + */ +const groupModels = ( + resolution: ControlModelResolution, + controlType: ControlType | null +): { group: ModelGroup; model: ModelConfig }[] => { + if (!controlType) { + return resolution.usable.map((model) => ({ group: 'installed' as const, model })); + } + const unions = resolution.usable.filter((model) => classifyControlModel(model).union); + const purpose = resolution.candidates.filter((model) => !unions.includes(model)); + const unknown = resolution.usable.filter( + (model) => !purpose.includes(model) && !unions.includes(model) && classifyControlModel(model).family === null + ); + return [ + ...purpose.map((model) => ({ group: 'purpose' as const, model })), + ...unions.map((model) => ({ group: 'union' as const, model })), + ...unknown.map((model) => ({ group: 'unknown' as const, model })), + ]; +}; + +/** The Model row and its two-line caption: what was resolved, why, and the one action that fixes or undoes it. */ +export const ControlModelRow = ({ + canManageModels, + captionId, + controlType, + disabled, + model, + onFindModels, + onPick, + onReset, + reidentifyPending, + rememberedHint, + resolution, + starterSources, +}: { + canManageModels: boolean; + /** The caption's id, so the Model (and, when it warns, the Type) choice is described by it. */ + captionId: string; + controlType: ControlType | null; + disabled: boolean; + model: ControlPanelModel['model']; + onFindModels(): void; + onPick(model: ModelConfig): void; + onReset(): void; + /** Listed disabled after the rest: they need re-identifying before a layer can use them. */ + reidentifyPending: readonly ModelConfig[]; + /** Said on Reset when it also forgets the remembered pick, which applies to other layers and projects too. */ + rememberedHint: string | null; + resolution: ControlModelResolution; + /** Sources of the starter models; null until the catalog has loaded, when items leave their origin out. */ + starterSources: ReadonlySet | null; +}) => { + const { i18n, t } = useTranslation(); + const label = t('widgets.layers.control.model'); + const selectRef = useRef(null); + const format = useCallback( + (text: Parameters[1]) => formatPanelText(t, text, i18n.language), + [i18n.language, t] + ); + + const collection = useMemo(() => { + const itemFor = (entry: ModelConfig, group: ModelGroup, disabled: boolean): ModelItem => { + const kind = deriveControlKind(entry); + const kindLabel = kind ? t(`widgets.layers.control.kinds.${kind}`) : entry.type; + return { + caption: disabled + ? t('widgets.layers.control.needsReidentify') + : starterSources + ? t('widgets.layers.control.modelItemCaption', { + kind: kindLabel, + origin: t( + `widgets.layers.control.modelOrigin.${starterSources.has(entry.source) ? 'starter' : 'downloaded'}` + ), + }) + : kindLabel, + disabled, + group, + label: entry.name, + value: entry.key, + }; + }; + return createListCollection({ + items: [ + ...groupModels(resolution, controlType).map(({ group, model: entry }) => itemFor(entry, group, false)), + ...reidentifyPending.map((entry) => itemFor(entry, 'unknown', true)), + ], + }); + }, [controlType, reidentifyPending, resolution, starterSources, t]); + const renderGroupLabel = useCallback( + (group: string) => + group === 'purpose' + ? t('widgets.layers.control.modelGroups.purpose', { + type: controlType ? t(`widgets.layers.control.typeNouns.${controlType}`) : '', + }) + : t(`widgets.layers.control.modelGroups.${group}`), + [controlType, t] + ); + + const value = useMemo(() => (resolution.model ? [resolution.model.key] : []), [resolution.model]); + const handleChange = useCallback( + ({ value: next }: SelectValueChangeDetails) => { + const picked = resolution.usable.find((entry) => entry.key === next[0]); + if (picked && picked.key !== resolution.model?.key) { + onPick(picked); + } + }, + [onPick, resolution.model?.key, resolution.usable] + ); + const handleAction = useCallback(() => { + if (model.action === 'reset') { + onReset(); + // The Reset link goes away with the pick; the choice it undid is where focus belongs. + selectRef.current?.querySelector('[data-part="trigger"]')?.focus(); + } else { + onFindModels(); + } + }, [model.action, onFindModels, onReset]); + + // Whether a model came with Invoke is read from the starter catalog, fetched the first time the list opens. + const handleOpenChange = useCallback(({ open }: { open: boolean }) => { + if (open) { + ensureStartersLoaded(); + } + }, []); + const triggerProps = useMemo(() => ({ 'aria-describedby': captionId }), [captionId]); + // The Select disables an empty list; an untyped layer then says there is nothing to pick rather than "Set by type". + const valueText = + model.name ?? + (!controlType && collection.items.length === 0 + ? t('widgets.layers.control.modelValue.noModels') + : model.valueText + ? format(model.valueText) + : ''); + const valueTextProps = useMemo(() => ({ color: TONE_COLORS[model.tone] }), [model.tone]); + const showAction = model.action === 'reset' || (model.action !== null && canManageModels); + const actionLabel = + model.action === 'reset' + ? t('widgets.layers.control.modelAction.reset') + : model.action === 'find-one' + ? t('widgets.layers.control.modelAction.findOne') + : t('widgets.layers.control.modelAction.installStarters'); + const actionButton = showAction ? ( + + ) : null; + + return ( + + + + + + ); +}; + +const SourceActionButton = ({ + action, + busy, + disabled, + icon, + onRun, +}: { + action: ControlSourceAction; + busy: ControlSourceAction | null; + disabled: boolean; + icon: typeof UploadIcon; + onRun(action: ControlSourceAction): void; +}) => { + const { t } = useTranslation(); + const handleClick = useCallback(() => onRun(action), [action, onRun]); + const label = t(`widgets.layers.control.source.${action}`); + return ( + + {/* Named explicitly: while loading, the spinner replaces the visible text. */} + + + ); +}; diff --git a/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlTypeRow.tsx b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlTypeRow.tsx new file mode 100644 index 00000000000..86d3b5f8a97 --- /dev/null +++ b/invokeai/frontend/webv2/src/workbench/widgets/layers/ControlTypeRow.tsx @@ -0,0 +1,116 @@ +import type { SelectValueChangeDetails } from '@chakra-ui/react'; +import type { ControlType } from '@workbench/canvas-engine/api'; + +import { createListCollection, HStack, Text } from '@chakra-ui/react'; +import { Select } from '@platform/ui/Select'; +import { CONTROL_TYPES } from '@workbench/canvas-engine/api'; +import { PropertyControlRow } from '@workbench/widgets/canvas/tool-presentation/PropertyPrimitives'; +import { useCallback, useMemo } from 'react'; +import { useTranslation } from 'react-i18next'; + +import { formatPanelText, type ControlPanelModel, type PanelTone } from './controlPanelModel'; +import { OptionNameSeparator } from './OptionNameSeparator'; + +const SELECT_POSITIONING = { placement: 'bottom-end', sameWidth: true } as const; +const TONE_COLORS: Record = { + default: 'fg', + error: 'fg.error', + muted: 'fg.muted', + warning: 'fg.warning', +}; +const PLACEHOLDER_PROPS = { color: 'fg.muted' } as const; +const INVALID_PLACEHOLDER_PROPS = { color: 'fg.warning' } as const; +// The recipe paints `aria-invalid` triggers as errors; a missing type only warns, so its rule is outranked here. +// The theme has no `border.warning`; warning borders use the warning foreground, as the warning input shell does. +const WARNING_TRIGGER_CSS = { + '&[aria-invalid=true]': { borderColor: 'fg.warning' }, + '&[aria-invalid=true]:hover': { bg: 'bg.warning', borderColor: 'fg.warning' }, + '&[aria-invalid=true]:focus-visible': { + '--focus-ring-color': 'var(--chakra-colors-fg-warning) !important', + borderColor: 'fg.warning', + }, +} as const; + +interface TypeItem { + hint: string; + hintColor: string; + label: string; + value: ControlType; +} + +/** The Type row: what the image controls, each choice hinted with what is installed for it. */ +export const ControlTypeRow = ({ + describedBy, + disabled, + model, + onChange, +}: { + /** What explains a missing type, read while the row warns. */ + describedBy: string; + disabled: boolean; + model: ControlPanelModel['type']; + onChange(type: ControlType, label: string): void; +}) => { + const { i18n, t } = useTranslation(); + const label = t('widgets.layers.control.rows.type'); + const collection = useMemo( + () => + createListCollection({ + items: CONTROL_TYPES.map((type) => ({ + hint: formatPanelText(t, model.hints[type].text, i18n.language), + hintColor: TONE_COLORS[model.hints[type].tone], + label: t(`widgets.layers.control.types.${type}`), + value: type, + })), + }), + [i18n.language, model.hints, t] + ); + const value = useMemo(() => (model.value ? [model.value] : []), [model.value]); + const triggerProps = useMemo( + () => + model.invalid ? { 'aria-describedby': describedBy, 'aria-invalid': true, css: WARNING_TRIGGER_CSS } : undefined, + [describedBy, model.invalid] + ); + const handleChange = useCallback( + ({ value: next }: SelectValueChangeDetails) => { + const type = next[0] as ControlType | undefined; + if (type && type !== model.value) { + onChange(type, t(`widgets.layers.control.types.${type}`)); + } + }, + [model.value, onChange, t] + ); + return ( + +