Skip to content

FLUX 3 Image Klein onboarding (distilled core) - #930

Merged
dkackman merged 20 commits into
developfrom
feat/flux3-klein
Oct 16, 2026
Merged

dkackman merged 20 commits into
developfrom
feat/flux3-klein

Conversation

@dkackman

Copy link
Copy Markdown
Owner

Adds FLUX 3 Image Klein (black-forest-labs/flux-3-image-klein, distilled, released 2026-10-15) to the catalog. It includes BFL's prompt upsampler as a pinned prior step, the flux3-klein plugin skill, and the model-agnostic text_generation options the upsampler needs. Running the templates requires diffusers at or after 7ecb1e1b (PR #15046, not yet released).

Audit: docs/proposals/audits/2026-10-15-flux-3-klein-audit.md. Design: docs/superpowers/specs/2026-10-15-flux-3-klein-catalog-design.md. Plan: docs/superpowers/plans/2026-10-15-flux3-klein-catalog.md.

Engine (dw/tasks/text_generation.py, nothing model-specific)

  • chat_template_kwargs takes scalar values only; the pipeline's and apply_chat_template's own argument names are refused. The upsampler needs enable_thinking: false.
  • revision pins the model weights.
  • image may be a list. Each element goes through the existing URL and path validation.
  • system_prompt can be {"from_hub": {repo, file, revision, key}}, so a vendor's prompt file is read at a pinned commit instead of being copied into the repo. The revision must be a full 40-hex commit. The repo and file go through the shared validators. Before every use the file is checked anonymously with the Hub, and the check fails closed, so a gated or private file the server has cached is never readable. It is downloaded with token=False and capped at 1 MiB.
  • Security fix, pre-existing (since a0e8cb1): generate_kwargs was passed to generate() unfiltered. An untrusted workflow could set custom_generate + trust_remote_code and run code from any Hub repo. Those two keys now require trusted workflows (dw/trust.py). Structural keys and non-scalar values are refused.

Catalog

  • Builtins flux3_klein_upsample and flux3_klein_upsample_edit run flux-3-image-klein-pup, with weights and system prompts both pinned at 3f3edb52.
  • Templates in workflows/templates/flux3/: text-to-image, upsample-then-generate, image-edit, multi-reference and layout-boxes. They use BFL's own example captions, images and seeds. A model entry, workflows/models/flux3-klein.json, and a family README are included.
  • Every Klein step places transformer, text encoder and VAE on cuda explicitly. An unplaced component leaves _execution_device on the CPU, which breaks both the encode and the noise draw.
  • Plugin skill: plugins/dw/skills/flux3-klein/.

Measured on lem (RTX 3090, bf16, everything resident)

Probe Result
text-to-image 1024², 16 steps 24 s, 0.8 s/step, 22.1 GB reserved peak
two-reference edit 1024² 45 s, 2.6 s/step, 22.1 GB reserved peak
upsampler through text_generation 22 s; runs in thinking mode unless enable_thinking is off

Still open before merge

  • Task 9: warm cost fields from GPU runs of each template once this branch is deployed to lem.
  • Task 11: cold drill with the plugin, plus a control without it.
  • tests/test_catalog_structure.py::test_the_compact_listing_fits_the_budget fails at 11256 vs a budget of 10850. The overage is in the new templates' summary lines; trimming saves only about 40 tokens. A budget raise is the owner's call, best sized after Task 9 adds cost blocks.
  • Not on this branch: dw/tasks/speech_generation.py:219 (generate_kwargs/forward_params) has the same custom_generate remote-code exposure. dw/tasks/image_utils.py:103/114/124 pass workflow kwargs to detectors unaudited.

Tests

The full suite passes (14927), except the budget test above. Per-task reviews plus a final whole-branch review were run. Three background security findings were fixed during the work: an SSRF through a kwargs splat, disclosure through the server token, and a fail-open cache.

🤖 Generated with Claude Code

dkackman and others added 18 commits October 15, 2026 19:45
Launch-day audit of the vendor and diffusers sources, and the design for
the distilled core: probes P1/P2 on lem (bf16 fits 24 GB at 1024^2 with
every component placed; the pup runs through text_generation but needs
enable_thinking off), and the from_hub system-prompt reference.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mages

A Qwen3.5 upsampler must run with enable_thinking off, an edit sends every
reference, and a pinned prompt wants pinned weights.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…template_kwargs

Workflows are untrusted and the settings are splatted into the pipeline and
apply_chat_template calls, so images/tools/etc. would reach transformers
unvalidated. Also pins that image list elements get the URL check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…only, size cap

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… file anonymously

A full commit is answered from the local cache without an auth check, so
token=False alone left cached gated files readable. Ask the Hub first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The skill picks among the five FLUX 3 Klein templates by request shape,
states the 16-step no-guidance schedule, the 16-pixel grid, the 4096
ceiling as stated rather than checked, and the 10-reference training
limit the pipeline does not enforce. Prompts go through the upsampler
builtins, or BFL's prompting guide for hand-written captions; the box
format is in references/layout-boxes.md. Named in the plugin README and
CLAUDE.md; the tests read the step default off the distilled denoise
block.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The edit templates always re-upsample and take no caption, so a corrected
caption is reused through a one-step workflow built from the edit
template's Klein step, not by passing prompt= or falling back to
text-to-image. dw's reference sizing for the upsampler is not BFL's, so
BFL's budget is no longer quoted. One memory figure for Apple Silicon,
the guidance rule as BFL's reference code states it, and CLAUDE.md
rewrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… unsized from_hub

generate_kwargs reached model.generate() unfiltered, so an untrusted workflow
could set custom_generate (transformers downloads and runs the repo's
custom_generate/generate.py) with trust_remote_code, or pass
tokenizer_encode_kwargs around the chat_template_kwargs checks.

- dw/trust.py: require_trusted_generate_arguments refuses custom_generate and
  trust_remote_code unless workflows are trusted
- generate_kwargs: the call's structural keys are refused on both paths, and
  values must be scalars or lists of scalars; all checked before any load
- _RESERVED_CHAT_TEMPLATE_KEYS gains the calls' remaining named parameters
- system_prompt.from_hub refuses a file the Hub reports no size for

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…age need

- text-to-image, upsample-then-generate and the README declare only the
  multiple-of-16 rule; BFL's 64 to 4096 range is stated as not checked
- flux3_klein_upsample_edit says at least one image is required, and what an
  empty list produces
- tests: flux3 step-cache rows in alphabetical order; the upsample step's
  result is asserted to be an intermediate

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… built

- TASKS.md: chat_template_kwargs and generate_kwargs filtering, the
  custom_generate/trust_remote_code trust rule, and that an image list is one
  prompt with one answer (also for judge and image_to_text)
- SECURITY.md and ARCHITECTURE.md: the generate_kwargs gate and the anonymous,
  pinned, fail-closed from_hub fetch
- Klein design spec: from_hub asks the Hub before every use and fails closed
  offline; the 10-reference cap is stated, not enforced; follow-ups for the
  other tasks' generate_kwargs and revision validation parity

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dkackman

Copy link
Copy Markdown
Owner Author

Decisions needed from @dkackman

  1. speech_generation remote-code hole (pre-existing, in released versions). dw/tasks/speech_generation.py:219 passes generate_kwargs unfiltered, and :190/:218 do the same with forward_params. An untrusted workflow can set custom_generate + trust_remote_code there, which is the hole this PR closes in text_generation. Two decisions: whether it gets a GHSA, and whether to fix it here or separately. dw/tasks/image_utils.py:103/114/124 also pass workflow kwargs to detectors unaudited.
  2. Listing budget. test_the_compact_listing_fits_the_budget measures 11256 against a budget of 10850. The overage is in the new templates' summary lines, and trimming them saves only about 40 tokens. Task 9's cost blocks will add about 15 tokens per entry. A raise sized after Task 9 is the realistic fix.
  3. Deploy to lem for Task 9 (warm costs, each template run twice) and Task 11 (cold drill). lem's dw.serve isn't in HF offline mode, so the upsampler templates' fail-closed Hub check will pass there.

Rulings made during implementation

Each line gives the ruling, then what it costs if wrong.

  1. Step-count tests read the library's real default. They read Flux3DistilledCoreDenoiseStep().inputs, where the default is 16, instead of checking that a 16 appears in the source. If wrong: a slightly more involved test.
  2. The failing budget test was recorded and left for you. Work continued past it. If wrong: the branch carries a red test until you rule.
  3. The skill (Task 10) was written before the GPU costs (Task 9), because Task 9 waits on the deploy. If wrong: the skill gets a second edit once costs exist.
  4. SSRF fix: chat_template_kwargs values must be scalars, and the pipeline's and apply_chat_template's own argument names are refused. If wrong: a legitimate template setting could be refused.
  5. from_hub is public only: it downloads with token=False and caps files at 1 MiB. If wrong: a vendor that ships prompts in a gated repo can't use it.
  6. The plan's own file-path check was replaced with the shared validate_weight_name, which is the validator CodeQL models. If wrong: nothing is lost beyond the swap.
  7. A non-string key crash was fixed in the same fix round as the other from_hub findings. If wrong: nothing.
  8. from_hub fails closed: it checks the file with the Hub anonymously before every use, so a pinned-commit cache hit can't bypass auth. If wrong: offline runs can't use from_hub, even for public files already cached.
  9. Family rules are stated once, in text-to-image. The other templates describe only their own idea. If wrong: an agent reading one template alone misses a rule. The skill and validation still carry them.
  10. layout-boxes uses BFL's own box-format caption from examples/distilled/balloons.json at bbfe317d, byte for byte, instead of prose we'd write. If wrong: nothing.
  11. The final review ran before Tasks 9 and 11. If wrong: the cost diff needs a small second look.
  12. The older generate_kwargs hole is fixed on this branch, because it undid this branch's own chat_template_kwargs checks. If wrong: generate_kwargs is just stricter.
  13. The final fixes went out as one round. Path-checking revision against the pipelines' rules is deferred to a follow-up. If wrong: a slightly larger fix diff.
  14. custom_generate / trust_remote_code refusals raise UntrustedWorkflowError, matching every other trust check, not ValueError. If wrong: a caller catching only ValueError misses it.
  15. max_length is refused inside generate_kwargs; max_new_tokens is the supported control. If wrong: a workflow using max_length must switch.
  16. Nested lists are refused in generate_kwargs, so bad_words_ids can't be set. If wrong: a workflow that needs it is blocked until this is relaxed.
  17. One docs/TASKS.md line is wrong until Task 9. The generate_kwargs row still says it overrides everything above it, though max_new_tokens is now reserved. The fix lands with the Task 9 commit. If wrong: that row misleads until then.

🤖 Generated with Claude Code

dkackman and others added 2 commits October 15, 2026 22:40
… 3 Klein (#930)

Approved by Don 2026-10-15: measured 11_256 with the five Klein templates,
about 85 tokens held for the Task 9 cost blocks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…xtures

Brings test_dw_patch_targets back under the ratchet for #930.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dkackman
dkackman merged commit f4f49e6 into develop Oct 16, 2026
10 checks passed
@dkackman
dkackman deleted the feat/flux3-klein branch October 16, 2026 11:09
dkackman added a commit that referenced this pull request Oct 16, 2026
…low-ups

The text_generation fixture refactor in #930 removed 44 patch sites.
The plan lists what develop still owes: the remaining generate()/detector
kwargs splats, Klein Tasks 9 and 11 on lem, the downloads_required and
update_diffusers gaps, #928, and the 0.14.0 release body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dkackman added a commit that referenced this pull request Oct 18, 2026
Both reach model.generate() for a generative text-to-audio model, where
custom_generate / trust_remote_code download and run Hub code. Untrusted
workflows now get the same refusal text_generation does (#930), before the
model loads; each block must be an object. Open in every release through
0.14.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dkackman dkackman mentioned this pull request Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant