Repository navigation
feat(controlnet): support the InstantX Qwen Image ControlNet Union - #9935
Open
Pfannkuchensack wants to merge 5 commits into
Open
Pfannkuchensack wants to merge 5 commits into
Pfannkuchensack wants to merge 5 commits into
Conversation
…trol layers Add alibaba-pai's VACE-style control branch for Qwen-Image: identification, loader, the qwen_image_control node and control/inpaint conditioning in the denoise, injected through forward hooks on the diffusers transformer. The canvas gets a Qwen Image ControlNet layer kind with inpaint/outpaint feeding and a grayscale filter; the denoise now reserves working memory for its activations.
The union file now installs as a Qwen Image ControlNet instead of a main model and feeds diffusers-style residuals to both CFG passes. Earlier installs fail to load with a pointer to Re-identify; the InstantX inpainting model is refused for now.
Member
Author
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
The InstantX Qwen-Image ControlNet Union (canny, soft edge, depth, pose; Apache-2.0) did not work in Invoke: the file was identified as a Qwen-Image main model (and also matched the FLUX InstantX ControlNet probe). This PR recognises it as a Qwen-Image ControlNet and runs it.
control_architecture(fun_unionorinstantx_union, defaultfun_unionso existing records load unchanged). The Qwen-Image main probe and the FLUX InstantX probe no longer claim the InstantX layout; one shared predicate owns its key signature. The separate InstantX inpainting model is refused with a clear message (not supported yet).QwenImageControlNetModel, geometry read from the keys.QwenImageControlNetPipeline, the ControlNet runs once per active step with the positive prompt and its scaled residuals feed both CFG forwards (controlnet_block_samples); inactive steps skip it. InstantX is control-only, so inpaint inputs are refused for it, and Canvas inpaint/outpaint does not wire them.Related Issues / Discussions
Builds on #9914 (Qwen-Image Fun-ControlNet-Union); merge after it.
QA Instructions
RTX 4090, Qwen-Image 2512 Q4_K_M + Qwen2.5-VL fp8, 30 steps, CFG 4, ~1 MP; both models fully resident:
instantx_union; the existing Fun union record reads asfun_unionInstantX was trained on the original Qwen-Image: there the card's 0.8–1.0 works as published, while on 2512 it needs about 1.5–2.0. The docs say so and point to the Fun union, which targets 2512.
CPU checks: the node path equals a restatement of the diffusers pipeline step on tiny randomly initialised models (
torch.equal); config detection yields exactly one match for the released Fun and InstantX headers (also under a ComfyUI prefix for the main probe) and FLUX InstantX stays FLUX; identification fixtures pass.pyteston model-manager, identification, FLUX ControlNet and Qwen control/denoise tests: passed. webv2pnpm lint, affected unit tests and theControlLayerSettingsbrowser test: passed. Ruff clean.Review
Resolved: the InstantX working-memory term now counts exactly the residual list kept alive through both CFG forwards instead of a fitted coefficient; earlier installs get an actionable error; the key signature has one owner; unreachable checks and duplicate tests removed; the refusal for the inpainting model is in plain language. Remaining: the InstantX inpainting model is not supported; on Qwen-Image 2512 the union needs a higher weight than its card suggests (on the original Qwen-Image the card's range holds).
Compatibility / Rollout
New field
control_architectureon Qwen-Image ControlNet records, defaultfun_union(records from #9914 load unchanged; older builds ignore the field).qwen_image_denoisebehaviour for the Fun union is unchanged.openapi.json/schema.tsregenerated. Files previously installed as a Qwen-Image main model need Re-identify model. No migration.Checklist
What's Newcopy (if doing a release after this PR)