Skip to content

feat(controlnet): support the InstantX Qwen Image ControlNet Union - #9935

Open
Pfannkuchensack wants to merge 5 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/qwen-image-instantx-controlnet
Open

Pfannkuchensack wants to merge 5 commits into
invoke-ai:mainfrom
Pfannkuchensack:feat/qwen-image-instantx-controlnet

Conversation

@Pfannkuchensack

@Pfannkuchensack Pfannkuchensack commented Oct 19, 2026 •

Copy link
Copy Markdown
Member

Summary

The InstantX Qwen-Image ControlNet Union (canny, soft edge, depth, pose; Apache-2.0) did not work in Invoke: the file was identified as a Qwen-Image main model (and also matched the FLUX InstantX ControlNet probe). This PR recognises it as a Qwen-Image ControlNet and runs it.

  • Detection: the Qwen-Image ControlNet config now carries control_architecture (fun_union or instantx_union, default fun_union so existing records load unchanged). The Qwen-Image main probe and the FLUX InstantX probe no longer claim the InstantX layout; one shared predicate owns its key signature. The separate InstantX inpainting model is refused with a clear message (not supported yet).
  • Loading: the one Qwen-Image ControlNet loader dispatches on the field; InstantX loads strictly into diffusers' QwenImageControlNetModel, geometry read from the keys.
  • Denoise: as in diffusers' QwenImageControlNetPipeline, the ControlNet runs once per active step with the positive prompt and its scaled residuals feed both CFG forwards (controlnet_block_samples); inactive steps skip it. InstantX is control-only, so inpaint inputs are refused for it, and Canvas inpaint/outpaint does not wire them.
  • Canvas: the Qwen Image ControlNet adapter type lists both unions.
  • Earlier installs: a file that an earlier version recorded as a main model now fails to load with a message pointing to Re-identify model in the Model Manager, which turns it into a ControlNet.
  • Starter model: the Comfy-Org repack of the union.

Related Issues / Discussions

Builds on #9914 (Qwen-Image Fun-ControlNet-Union); merge after it.

QA Instructions

RTX 4090, Qwen-Image 2512 Q4_K_M + Qwen2.5-VL fp8, 30 steps, CFG 4, ~1 MP; both models fully resident:

Run Result
Install the union file registered as a Qwen-Image ControlNet, instantx_union; the existing Fun union record reads as fun_union
Scale 0 vs no control, same seed bit-identical
Interleaved A/B, 3 seeds each 82 s vs 80 s wall per image with/without control; VRAM peak unchanged within the residual list (~125 MB at 768×1344)
DWPose skeleton, weight 1.0 / 1.5 / 2.0 on 2512 the pose barely changes at 1.0 and clearly follows at 1.5–2.0
Canny (weight 1.0) edge overlap 0.54 vs 0.51 without control
Canvas the control layer's model list offers both unions
Original Qwen-Image Q4_K_M, DWPose, weight 1.0 / 0.8, 2 seeds the pose follows the skeleton on every seed
Original Qwen-Image, canny, weight 1.0 edge overlap 0.86 vs 0.62 without control

InstantX was trained on the original Qwen-Image: there the card's 0.8–1.0 works as published, while on 2512 it needs about 1.5–2.0. The docs say so and point to the Fun union, which targets 2512.

CPU checks: the node path equals a restatement of the diffusers pipeline step on tiny randomly initialised models (torch.equal); config detection yields exactly one match for the released Fun and InstantX headers (also under a ComfyUI prefix for the main probe) and FLUX InstantX stays FLUX; identification fixtures pass. pytest on model-manager, identification, FLUX ControlNet and Qwen control/denoise tests: passed. webv2 pnpm lint, affected unit tests and the ControlLayerSettings browser test: passed. Ruff clean.

Review

Resolved: the InstantX working-memory term now counts exactly the residual list kept alive through both CFG forwards instead of a fitted coefficient; earlier installs get an actionable error; the key signature has one owner; unreachable checks and duplicate tests removed; the refusal for the inpainting model is in plain language. Remaining: the InstantX inpainting model is not supported; on Qwen-Image 2512 the union needs a higher weight than its card suggests (on the original Qwen-Image the card's range holds).

Compatibility / Rollout

New field control_architecture on Qwen-Image ControlNet records, default fun_union (records from #9914 load unchanged; older builds ignore the field). qwen_image_denoise behaviour for the Fun union is unchanged. openapi.json/schema.ts regenerated. Files previously installed as a Qwen-Image main model need Re-identify model. No migration.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

…trol layers

Add alibaba-pai's VACE-style control branch for Qwen-Image: identification, loader, the qwen_image_control node and control/inpaint conditioning in the denoise, injected through forward hooks on the diffusers transformer. The canvas gets a Qwen Image ControlNet layer kind with inpaint/outpaint feeding and a grayscale filter; the denoise now reserves working memory for its activations.
The union file now installs as a Qwen Image ControlNet instead of a main model and feeds diffusers-style residuals to both CFG passes.
Earlier installs fail to load with a pointer to Re-identify; the InstantX inpainting model is refused for now.
@github-actions github-actions Bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files frontend PRs that change frontend files python-tests PRs that change python tests docs PRs that change docs labels Oct 19, 2026
@Pfannkuchensack

Copy link
Copy Markdown
Member Author
qix_sheet qix_scale_sheet

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend PRs that change backend files docs PRs that change docs frontend PRs that change frontend files invocations PRs that change invocations python PRs that change python files python-tests PRs that change python tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants