feat(canvas): type-first control layers with background detection and derived models

docs/src/content/docs/Users Guide/07.Canvas/01.introduction.mdx

@@ -1,7 +1,7 @@
 ---
 title: Crafting images with the Canvas
 description: An introduction to the Canvas — layers and masks, sending generations to the Canvas, inpainting, outpainting, regional prompts, control layers, drawing and adding gallery images.
-lastUpdated: 2026-10-05
+lastUpdated: 2026-10-24
 ---
 
 import { Image } from 'astro:assets';
@@ -132,26 +132,34 @@ Regional guidance works with SD 1.5, SDXL, FLUX.1, FLUX.2, Krea-2, Z-Image and A
 
 ## Control layers
 
-A control layer shapes the composition with a guide image: the edges of a sketch, the depth of a scene, or a person's pose. A control model makes the generated image follow it.
+A control layer shapes the composition with a guide image: the edges of a sketch, the depth of a scene, or a person's pose. A control model makes the generated image follow it. You choose what the layer controls, and InvokeAI detects the matching map from your image and picks the model.
 
-<Image src={controlLayer} alt="A control layer's settings in the Layer section of Properties: Adapter type ControlNet, Model Depth Map, and Weight 0.75." width={420} />
+<Image src={controlLayer} alt="A control layer's settings in the Layer section of Properties." width={420} />
 
 <Steps>
 
-1. **Put the guide image on a control layer.** A control layer's image is its own pixels; there is no separate image picker. Do one of these:
-   * Right-click a raster layer and choose **Copy to → Copy to control layer**, or **Convert to → Convert to control layer**.
-   * Drag an image from the Gallery onto the canvas and drop it on **Add as Control Layer** (see [below](#adding-a-gallery-image-to-a-layer)).
-   * Add an empty **Control layer** from the **+** menu and draw on it.
+1. **Add a control layer with an image.** Do one of these:
+   * Choose **+ → Control layer**, then **Upload** an image, use the **Frame** (what is visible inside the generation frame), or **Draw** the guide yourself. These buttons are in the layer's Properties, in its **Image** row.
+   * Right-click an image in the Gallery and choose **Use as Control**. It fills the selected control layer if that layer has no image yet, and otherwise adds a new control layer.
+   * Right-click a raster layer and choose **Use as control layer**, which copies its pixels to a new control layer.
+   * Drag an image from the Gallery onto the canvas and drop it on **Add as Control Layer**. As with **Use as Control**, it fills the selected control layer if that layer is still empty.
+   * Select an empty control layer and drop an image file from your computer onto the canvas, where the layer shows *Drop an image here*. The file is uploaded into that layer.
 
-2. **Choose the model.** Select the layer, open **Properties** and scroll to its **Layer** section. Choose the **Adapter type** (ControlNet, T2I Adapter, Control LoRA or Z-Image ControlNet) and the **Model**. A compatible installed model is chosen for you when the layer is created.
+2. **Choose the Type**: **Pose**, **Depth**, **Edges**, **Soft edges**, **Line art**, **Sketch**, **Color** or **Tile**. Next to each type, the menu says how many installed models are made for it, *via Union* when only a Union model covers it, or *Not installed*.
 
-3. **Process the image, if needed.** Most control models expect a processed image, such as an edge map or a depth map, rather than a photograph. Choosing a model starts its recommended **Filter** automatically; you can also click **Filter**, pick a filter, and **Apply** it.
+3. **Check the Detection.** The type's detector turns your image into the map the model expects, such as a depth map or an edge map, in the background. If your image is already such a map, choose the **None** option, for example *None: image is already a depth map*. See [Control layer](/users-guide/canvas/layers-panel/#control-layer) for the details.
 
-4. **Adjust the influence** with **Weight** (how strongly the guide is followed), **Step range** (which part of the generation it applies to) and, for ControlNet, **Control mode**. Then press Invoke.
+4. **Check the Model.** InvokeAI picks it for you, and the line under the **Model** row says why. Choose another model from the list if you prefer. A model made for the type, or a Union model that covers it, is then remembered for that type; any other model is used for this layer only.
+
+5. **Adjust the influence** with **Weight** (how strongly the guide is followed), **Steps** (which part of the generation it applies to) and, for ControlNet, **Priority**. Then press Invoke.
 
 </Steps>
 
-Control layers work with SD 1.5, SDXL, FLUX.1 and Z-Image, and need a control model for the same family, installed from the Model Manager's starter models. If the **Model** list is empty, no compatible model is installed for the model selected in the Generate panel. A control layer without a model stops Invoke with the message *Select a control model before generating.*
+Once a map has been detected, a switch above the canvas shows the layer as its **Original** image, the detected **Map**, or an **Overlay** of the two. The switch changes only what you see; generation always uses the map.
+
+An empty control layer, or one with neither a type nor a model, only shows a warning and contributes nothing; Invoke stays available. A control layer with an image blocks Invoke if no model can be chosen for it, or if its map is out of date. **Invoke** then shows the number of issues, for example *Invoke · 1 issue*; click it to select the first layer that needs attention.
+
+Control layers work with SD 1.5, SDXL, FLUX.1, FLUX.2 [dev], Qwen Image, Z-Image and Anima, and need a control model for the same family, installed from the Model Manager's starter models. If no model fits, the **Model** row says so, with a link to the Model Manager when you can manage models.
 
 ## Drawing on a raster layer
 

docs/src/content/docs/Users Guide/07.Canvas/02.canvas-guide.mdx

@@ -1,7 +1,7 @@
 ---
 title: Canvas guide
 description: A detailed guide to the Canvas — its tools and their options, navigation, the header buttons, the generation frame, selections, transform, filters, the staging area, menus, settings and keyboard shortcuts.
-lastUpdated: 2026-10-18
+lastUpdated: 2026-10-24
 ---
 
 import { Image } from 'astro:assets';
@@ -112,7 +112,7 @@ To crop a layer to the generation frame, choose **Crop layer to bbox** from its
 
 ## Filters and object selection
 
-**Filter**, in a layer's menu or its Properties, processes a raster or control layer. Starting it opens the Layers panel's **Properties** tab, headed *Operation: Filter*, where you choose and preview the filter. The filters are grouped by purpose:
+**Filter**, in a raster layer's menu or its Properties, processes the layer. A control layer has no Filter: its map comes from the **Detection** row in its [Properties](/users-guide/canvas/layers-panel/#control-layer). Starting Filter opens the Layers panel's **Properties** tab, headed *Operation: Filter*, where you choose and preview the filter. The filters are grouped by purpose:
 
 * **Edge detection** — Canny, HED, Lineart, Lineart Anime, MLSD, PiDiNet and Scribble.
 * **Depth & normal** — Depth Anything and Normal map.

docs/src/content/docs/Users Guide/07.Canvas/03.layers-panel.mdx

@@ -1,7 +1,7 @@
 ---
 title: Layers panel guide
 description: A detailed guide to the Canvas Layers panel — the color picker, managing layers, history, the layer menu, and every layer type's settings in the Properties pane.
-lastUpdated: 2026-09-30
+lastUpdated: 2026-10-24
 ---
 
 import { Image } from 'astro:assets';
@@ -17,7 +17,7 @@ The **Layers** panel sits to the right of the Canvas in the **Edit** layout. It
 
 Drag the dividers between the parts to resize them, and use the chevron at the right of the Color and Properties tabs to collapse them. The sizes are saved with the project.
 
-<Image src={layersPanel} alt="The Layers panel: the color wheel at the top, the layer list grouped into Inpaint Masks, Regional Guidance, Control Layers and Raster Layers, and the Properties pane showing a control layer's Adapter type, Model and Weight." width={420} />
+<Image src={layersPanel} alt="The Layers panel: the color wheel at the top, the layer list grouped into Inpaint Masks, Regional Guidance, Control Layers and Raster Layers, and the Properties pane showing a control layer's settings." width={420} />
 
 For an introduction to the layer types, see [Crafting images with the Canvas](/users-guide/canvas/introduction/#canvas-concepts).
 
@@ -89,9 +89,10 @@ The icons at the top **Arrange** the layer (move it to the front, forward, backw
   * **Add adjustment** — Brightness/Contrast, Exposure, Levels, Curves, Saturation, Hue, Invert, Texture, Clarity, Dehaze or Noise. Adjustments are non-destructive and appear as child rows. Texture, Clarity and Dehaze change detail and contrast; Noise adds film grain.
   * **Add regenerate region** — treats the layer's own shape as an inpaint mask.
   * **Filter**, **Select object** and **Run workflow** — process the layer; see the [Canvas guide](/users-guide/canvas/canvas-guide/#filters-and-object-selection).
+  * **Use as control layer** — copies the layer's pixels to a new control layer, whose Properties then ask for its **Type**.
   * **Boolean operations** — combine the layer with the one below: intersect, cut out, cut away or exclude.
-  * **Convert to** — make it a control layer, inpaint mask or regional guidance layer. Text, shape and gradient layers can also be rasterized into plain pixels. Control layers have no adjustments, so converting or copying a layer with adjustments to a control layer bakes them, with the layer's scaling and rotation, into its pixels; undo restores the raster layer with its adjustments.
-* **Control layers:** **Transform**, **Filter**, **Select object**, **Run workflow**, **Enable transparent preview**, and **Convert to raster layer**.
+  * **Convert to** — make it a control layer, inpaint mask or regional guidance layer. Text, shape and gradient layers can also be rasterized into plain pixels. Control layers have no adjustments, so converting a layer with adjustments to a control layer, or using it as one, bakes them, with the layer's scaling and rotation, into its pixels; undo restores the raster layer with its adjustments.
+* **Control layers:** **Transform**, **Select object**, **Run workflow**, **Enable transparent preview**, and **Convert to raster layer**. A control layer's map comes from its **Detection** row in Properties, so it has no **Filter**.
 * **Inpaint masks:** **Add noise**, **Add denoise limit**, **Extract masked area** (copy the masked part of the image to a new layer), and **Hide layer**.
 * **Regional guidance:** **Enable auto-negative**, **Add reference image**, and **Hide layer**.
 
@@ -160,15 +161,25 @@ Things to know about Texture, Clarity and Dehaze:
 
 ### Control layer
 
-* **Adapter type** — ControlNet, T2I Adapter, Control LoRA, Z-Image ControlNet or Anima ControlNet-LLLite, depending on the model selected in the Generate panel.
-* **Model** — the installed control models of that type for the same model family. If the list is empty, install one from the starter models. For Anima, the list holds the LLLite control adapters but not the Inpainting adapter, which is chosen as the **Inpaint Adapter** in the Generate panel's Components section instead, and each LLLite model can drive only one control layer per generation.
-* **Weight** — how strongly the guide is followed (0 to 2; the default is 0.75 for ControlNet).
-* **Step range** — the part of the generation, from start to finish, in which the guide applies.
-* **Control mode** (ControlNet only) — **Balanced**, **More prompt**, **More control** or **Unbalanced**.
-* **Transparent preview** — shows the layer semi-transparently over the image.
-* **Filter** — process the layer's image into a guide, such as an edge or depth map.
+A control layer's Layer section always has the same rows, in this order: **Image**, **Type**, **Detection** and **Model**, then **Weight**, **Steps** and **Priority**. The rows stay in place; only their values change.
 
-The control image is the layer's own pixels. To use a picture, copy or convert a raster layer, or drop a gallery image on **Add as Control Layer**.
+* **Image** — the picture the guide comes from. On an empty layer, the row has three buttons: **Upload** an image from your computer, use the **Frame** (what is visible inside the generation frame), or **Draw** the control image yourself on the layer. You can also drop an image file from your computer onto the canvas while the empty layer is selected: the canvas shows *Drop an image here* over the layer, and the file is uploaded into it. Once the layer has an image, the row shows its thumbnail and a label, such as the file name, and **Replace** offers the same three choices. The layer keeps this original image beside the detected map, so detection can run again on it.
+* **Type** — what the layer controls: **Pose**, **Depth**, **Edges**, **Soft edges**, **Line art**, **Sketch**, **Color** or **Tile**. In the menu, each type shows how many installed models are made for it, *via Union* when only a Union model covers it, or *Not installed*. Changing the type sets **Model** back to automatic, and on a layer with an image it runs the new type's default detector unless the current one also belongs to the new type.
+* **Detection** — the detector that turns the image into the type's map, for example Depth Anything or a normal map for Depth, Canny or MLSD for Edges, and DW Openpose or Mediapipe Face for Pose. The **None** option, such as *None: image is already a depth map*, uses the image as it is. Until the layer has an image, the row reads *Chosen when an image is added* and its detectors cannot be chosen; the first image gets the type's default detector. **Sketch**, and an image you draw yourself, use **None**; a drawn image counts once it has a stroke. **Tile** has no detector and uses the image directly.
+  * The **Detector settings** button next to it opens the detector's options. Changing the detector or its settings runs detection again in the background. A spinner shows while it is waiting, with *Waiting to run* and the detector's name, and while it runs, with *Running* and the detector's name.
+  * If detection fails, a **Run again** button appears; hover over it for the reason. If the map is out of date, a **Detect again** button appears.
+* **Model** — chosen for you, in this order: the model you picked for this layer; the model you last picked for this type on the same main model family; the only installed model made for the type; a Union model that covers the type. When several models fit equally, the row asks you to choose one. The line under the row says which case applies, for example *ControlNet · your only depth model*. The menu groups the models under **Made for** the type, **Union** and **Purpose not set**; a layer without a type lists every installed control model for the model family under **Installed models**. Picking a model made for the type, or a Union model that covers it, remembers it for every layer of that type and model family, in every project; a **Purpose not set** model, or one made for another type, is used for this layer only. **Reset** drops this layer's own pick, so the remembered model, if any, or the automatic choice takes over; when the remembered model is the one in use, Reset forgets it and the layer returns to automatic. If no model fits, the line says so. For Anima, the list holds the ControlNet-LLLite control adapters but not the Inpainting adapter, which is chosen as the **Inpaint Adapter** in the Generate panel's Components section instead, and each LLLite model can drive only one control layer per generation.
+* **Weight** — how strongly the guide is followed. The slider runs from 0 to 2; for ControlNet, T2I Adapter and Control LoRA, the number field also accepts values down to −1. The default is 0.75 for ControlNet.
+* **Steps** — the part of the generation, from start to finish, in which the guide applies.
+* **Priority** — **Prompt**, **Balanced** or **Control**: whether the generation leans toward the prompt or toward the guide. It applies to SD 1.5 and SDXL ControlNet models only. A layer saved with the earlier **Unbalanced** mode shows it as a fourth, selected option until you choose another; new layers do not offer it.
+
+The layer's row in the list reads *Type · Detector · weight*, for example *Depth · Depth Anything · 0.75*, or *No image* or *Choose a type* while the layer is incomplete. A spinner shows on the row while its detector is waiting or running, and a warning icon when the layer needs attention; hover over the icon for the reason.
+
+Once a map has been detected, a switch above the canvas shows the layer as its **Original** image, the detected **Map**, or an **Overlay** of the two. It changes only the view. **Enable transparent preview**, in the layer menu, shows the layer semi-transparently over the image.
+
+A control layer with no image, or with neither a type nor a model, only shows a warning: Invoke stays available and the layer contributes nothing. A layer with an image blocks Invoke when no model can be chosen for it or when its map is out of date. **Invoke** then shows the number of issues, and clicking it selects the first layer that needs attention.
+
+A control layer made in an earlier version of Invoke keeps its model and works without a type; its row reads its adapter and weight. Choosing a **Type** for it sets its model back to automatic and can run the type's default detector on its pixels.
 
 ### Inpaint mask
 

docs/src/content/docs/development/Front End/project-files.mdx

@@ -44,10 +44,12 @@ would drop part of it on load and lose it on the next autosave. Schema 4 marks c
 typography (a text source with `fontRef`, `fontStyle` or `fontVariations`). Schema 5 marks a noise,
 clarity, texture or dehaze adjustment, enabled or not, on any layer or group at any depth; an older
 reader would drop the
-whole adjustment stack holding it. The live document and each saved canvas snapshot are versioned
+whole adjustment stack holding it. Schema 6 marks a control layer carrying any derivation field
+(`controlType`, `detection`, `original` or `originalTransform`), which an older reader would strip,
+losing the kept original. The live document and each saved canvas snapshot are versioned
 this way, and the canvas state takes the newest of them. Adoption is sticky: removing the feature
-does not lower the version, as the server floor cannot be lowered either. A canvas with neither
-stays at 3, so older builds keep opening it. A build refuses a canvas that declares a schema above
+does not lower the version, as the server floor cannot be lowered either. A canvas with none of
+these stays at 3, so older builds keep opening it. A build refuses a canvas that declares a schema above
 its maximum as an unsupported version before parsing it, and once the floor is raised the server
 refuses the project to a client declaring a lower maximum.
 

invokeai/backend/flux/controlnet/instantx_controlnet_flux.py

@@ -143,8 +143,8 @@ def forward(
                 controlnet_mode_emb = self.controlnet_mode_embedder(controlnet_mode)
             txt = torch.cat([controlnet_mode_emb, txt], dim=1)
             txt_ids = torch.cat([txt_ids[:, :1, :], txt_ids], dim=1)
-        else:
-            assert controlnet_mode is None
+        # A non-union model has no mode embedder, so a supplied controlnet_mode is ignored. Callers pick the mode from
+        # the model name, and some checkpoints keep "union" in the name after their mode embedding was removed.
 
         ids = torch.cat((txt_ids, img_ids), dim=1)
         pe = self.pe_embedder(ids)