Skip to content
Merged

H3 #33

Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,10 @@ The REPL uses a **persistent worker subprocess** for workflow execution to maint

**Result References:** Use `"previous_result:step_name"` to pipe outputs between steps. System automatically generates all combinations when multiple results exist.

**Constant References:** Use `"constant:module.NAME"` to read a value declared in Python instead of copying it into JSON.

**Prompt References:** Use `"prompt:name"` or `"prompt:folder/name"` to load a stored prompt's text from the `prompts/` library at run time (see `dw/prompts.py`).

**Pipeline Configuration:**
```json
{
Expand Down
10 changes: 5 additions & 5 deletions .vscode/launch.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,10 @@
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
//
// The examples tree has subfolders (flux/, tasks/, archive/, projects/), so
// The workflows tree has subfolders (flux/, ltx2/, minimax/, tasks/, archive/), so
// the "current workflow file" configurations below are the ones to reach for:
// open the JSON and run it, wherever it lives. The prompting configurations
// take a path relative to examples/, subfolders included - "flux/FluxDev".
// take a path relative to workflows/, subfolders included - "flux/FluxDev".
"version": "0.2.0",
"configurations": [
{
Expand Down Expand Up @@ -71,7 +71,7 @@
"cwd": "${workspaceFolder}/",
"justMyCode": true,
"args": [
"./examples/${input:workflowName}.json"
"./workflows/${input:workflowName}.json"
]
},
{
Expand Down Expand Up @@ -105,15 +105,15 @@
"args": [
"--log_level",
"INFO",
"./examples/${input:workflowName}.json"
"./workflows/${input:workflowName}.json"
]
}
],
"inputs": [
{
"id": "workflowName",
"type": "promptString",
"description": "Workflow path relative to examples/, without .json (e.g. ZImage or flux/FluxDev):",
"description": "Workflow path relative to workflows/, without .json (e.g. ZImage or flux/FluxDev):",
"default": "??"
}
]
Expand Down
21 changes: 15 additions & 6 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
bash ./install.sh && source ./activate

# Run a workflow
python -m dw.run examples/ZImage.json
python -m dw.run examples/ZImage.json prompt="a cat" num_images_per_prompt=4
python -m dw.run workflows/ZImage.json
python -m dw.run workflows/ZImage.json prompt="a cat" num_images_per_prompt=4

# Validate a workflow against schema
python -m dw.validate examples/ZImage.json
python -m dw.validate workflows/ZImage.json

# Basic system test (torch, diffusers import check)
python -m dw.test
Expand All @@ -28,7 +28,7 @@ python -m dw.repl
# HTTP server + web UI (http://127.0.0.1:8765, API docs at /docs)
python -m dw.serve

# Run all tests (580+ tests)
# Run all tests (1,600+ tests)
pytest -v

# Run a single test file or test
Expand All @@ -52,7 +52,10 @@ over SSE, and persists history to `~/.diffusers_helper/jobs.sqlite`. The SPA liv
`ui/` (Svelte 5 + Vite; `npm run build` outputs `ui/dist`, which the server serves).
Introspection endpoints (`dw/introspection.py`) describe pipeline/class signatures for
the editor's forms. `dw/hub_cache.py` inventories and deletes from the HF hub cache
(the UI's Models page). Front-end checks from `ui/`: `npm run check`, `npm run lint`,
(the UI's Models page). `dw/prompts.py` + `dw/server/enhancers.py` back the
Prompts page: a stored-prompt library (CRUD under `--prompt-dir`) with an
enhance-with-AI panel that queues an inline workflow as an ordinary job.
Front-end checks from `ui/`: `npm run check`, `npm run lint`,
`npm test`, `npx playwright test` (e2e, starts its own server). See docs/SERVER.md.

Packaging: `pyproject.toml` (console scripts dw-run/dw-validate/dw-repl/dw-serve/dw-test);
Expand All @@ -75,6 +78,11 @@ The REPL (`dw/repl.py`) uses a **persistent worker subprocess** (`dw/worker.py`)
- Values prefixed with `constant:` read a value declared in python rather than copying it
into JSON: `"constant:diffusers.pipelines.ltx2.utils.DISTILLED_SIGMA_VALUES"`. Resolved
in `realize_args`, validated by `validate_constant_name()`; anything callable is refused
- Values prefixed with `prompt:` load a stored prompt's `text` from the prompt library:
`"prompt:name"` or `"prompt:folder/name"`. Resolved in `realize_args` (`dw/prompts.py`),
rooted at the library rather than the workflow file. The library is `DW_PROMPT_DIR` /
`--prompt-dir`, else `./prompts` if it exists, else found by walking up from the
workflow file's directory

### Quantization Support

Expand Down Expand Up @@ -115,6 +123,7 @@ All entry points use `dw/security.py`. When adding features:
- **Built-in workflows** need explicit argument mapping: `"prompt": "variable:prompt"`
- **MPS differences from CUDA**: no autocast, no bitsandbytes, no flash_attn, no triton, no torch.compile. Model offloading has less benefit on unified memory.
- **`{}`-escaped strings** in JSON arguments: `"{nf4}"` stays as string `"nf4"`, without braces it would try to load as a type
- **A stored prompt's `text` may not begin with a reference prefix** (`variable:`, `previous_result:`, `constant:`, `prompt:`) — the engine rejects it to prevent double resolution or iteration expansion
- **Audio+video muxing**: pipelines that generate audio alongside video (LTX-2) have the two muxed into one `video/mp4` file with PyAV in `result.py`

## JSON Workflow Structure
Expand Down Expand Up @@ -146,4 +155,4 @@ Steps can also have `"task"` (with `command` + `arguments`) or `"workflow"` (wit

Pipeline configuration options include: `pre_load_modules` (import modules before loading), `sdnq_optimize` (SDNQ quantized matmul), `enable_attention_slicing`, `disable_attention_slicing`, `attention_backend`, `group_offload`, `enable_layerwise_casting`.

File paths in workflows are relative to the workflow file. Built-in workflows use `"builtin:filename.json"` (resolves to `dw/workflows/`).
File paths in workflows are relative to the workflow file. Built-in workflows use `"builtin:filename.json"` (resolves to the packaged `dw/workflows/` — distinct from the top-level `workflows/` folder of runnable examples).
12 changes: 8 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ A declarative workflow engine and web UI for the [Hugging Face Diffusers library
- **Quantization** — BitsAndBytes, TorchAO, GGUF, SDNQ, optimum-quanto
- **Inference acceleration** — TeaCache, FirstBlockCache, FasterCache, MagCache, TaylorSeerCache
- **Prompt weighting** — A1111-style `(word:1.5)` syntax with long prompt support
- **Prompt library** — store prompts once in `prompts/` and reference them from any workflow as `prompt:name` or `prompt:folder/name`, with a web UI for browsing, editing, and AI-enhancing them
- **LoRA and IP-Adapter** support
- **Composable workflows** from multiple JSON files with `builtin:` references
- **Utility tasks** — upscaling, face restoration, segmentation, captioning, frame interpolation, QR codes, and more
Expand Down Expand Up @@ -64,6 +65,8 @@ Everything the engine does, in a browser backed by a persistent GPU worker — m

![The gallery with generated images and videos](docs/img/ui-gallery.jpg)

**A prompt library shared by every workflow.** Store a prompt once, reference it anywhere as `prompt:name` — the Prompts page browses, edits, and filters the library, and an *Enhance with AI* panel expands an idea into a full prompt with a local language model.

**A model manager for the disk your models actually consume.** The Hugging Face hub cache, inventoried: sizes, revisions, last-used dates, free space — download new models by id with live progress, delete with one click.

![The model manager listing cached models with sizes](docs/img/ui-models.png)
Expand All @@ -75,14 +78,14 @@ Jobs queue, stream progress live (per denoising step), cancel cooperatively, and
### Run a Workflow

```bash
python -m dw.run examples/flux/FluxDev.json
python -m dw.run examples/flux/FluxDev.json prompt="a cat" num_images_per_prompt=4
python -m dw.run workflows/flux/FluxDev.json
python -m dw.run workflows/flux/FluxDev.json prompt="a cat" num_images_per_prompt=4
```

### Validate a Workflow

```bash
python -m dw.validate examples/flux/FluxDev.json
python -m dw.validate workflows/flux/FluxDev.json
```

### Interactive REPL
Expand Down Expand Up @@ -242,7 +245,7 @@ a (photorealistic:1.4) portrait with (bright red hair:1.3) and [freckles]

**Interactive schema browser:** [View Schema](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2Fdkackman%2Fdiffusers-workflow%2Frefs%2Fheads%2Fmaster%2Fdw%2Fworkflow_schema.json)

See [examples/](examples/) for more workflow files.
See [workflows/](workflows/) for more workflow files.

## Documentation

Expand All @@ -256,6 +259,7 @@ See [examples/](examples/) for more workflow files.
- [LoRA](docs/LORAS.md) — Loading and stacking LoRA adapters
- [IP-Adapter](docs/IP_ADAPTER.md) — Image-prompt conditioning
- [Prompt Weighting](docs/PROMPT_WEIGHTING.md) — A1111-style syntax
- [Prompt References](docs/WORKFLOW_GUIDE.md#prompt-references) — The stored prompt library and `prompt:` references
- [Tasks](docs/TASKS.md) — Image processing, ControlNet preprocessors, utilities

### Reference
Expand Down
10 changes: 5 additions & 5 deletions docs/ACCELERATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ Simplest and broadest support. Compares first-block residuals to decide whether

Higher threshold = more speedup, more quality loss. Start with `0.05` and increase to taste.

**Example:** [FluxDevFirstBlockCache.json](../examples/flux/FluxDevFirstBlockCache.json)
**Example:** [FluxDevFirstBlockCache.json](../workflows/flux/FluxDevFirstBlockCache.json)

### MagCache

Expand Down Expand Up @@ -165,7 +165,7 @@ Some models have multiple variants with different coefficients:
}
```

**Example:** [FluxDevTeaCache.json](../examples/flux/FluxDevTeaCache.json)
**Example:** [FluxDevTeaCache.json](../workflows/flux/FluxDevTeaCache.json)

## Cache vs TeaCache

Expand Down Expand Up @@ -206,7 +206,7 @@ A component can also pin its backend persistently instead, via `set_attention_ba

Prefer the pinned form for a compiled component - the per-call context manager switches implementations under the compiled graph and forces a recompile on every run.

**Example:** [Flux2Dev.json](../examples/flux/Flux2Dev.json), [hunyuan15.json](../examples/archive/hunyuan15.json), [Wan22TI2V5B.json](../examples/archive/Wan22TI2V5B.json)
**Example:** [Flux2Dev.json](../workflows/flux/Flux2Dev.json), [hunyuan15.json](../workflows/archive/hunyuan15.json), [Wan22TI2V5B.json](../workflows/archive/Wan22TI2V5B.json)

## Attention Slicing

Expand Down Expand Up @@ -252,7 +252,7 @@ Typical gains are 1.3-1.5x on diffusion transformers, and compilation stacks wit
- **Don't combine `fullgraph` with a `cache`**: the cache hooks decide skip-or-compute per step, a data-dependent branch diffusers wraps in `torch.compiler.disable` - it needs the graph break that `fullgraph: true` forbids. Compile with the default (partial) graph mode when a cache is active.
- **TorchAO quantization needs compile to be fast** - see [QUANTIZATION.md](QUANTIZATION.md#torchao).

**Example:** [FluxDevFast.json](../examples/flux/FluxDevFast.json), [FluxTorchAO.json](../examples/flux/FluxTorchAO.json)
**Example:** [FluxDevFast.json](../workflows/flux/FluxDevFast.json), [FluxTorchAO.json](../workflows/flux/FluxTorchAO.json)

## Layerwise Casting

Expand All @@ -277,7 +277,7 @@ Both `storage_dtype` and `compute_dtype` are required. Applied via the component

`"residency": "on_demand"` on a component is the cheap case of the same trade: the model rests in system memory and is moved to the device whole around each of its own calls. That is a bad deal for anything called once per step, and a good one for a VAE called twice a run - it frees the VAE's VRAM for the denoise loop at the cost of two transfers, where group offloading the same VAE would restream it once per decode tile. See [On-demand components](WORKFLOW_GUIDE.md#on-demand-components).

**Example:** [FluxDev.json](../examples/flux/FluxDev.json) (`"offload": "model"`), [ZImage.json](../examples/ZImage.json) (`"offload": "sequential"`), [MiniMaxH3.json](../examples/MiniMaxH3.json) (`group_offload` per component), [MiniMaxH3Ref2VA.json](../examples/MiniMaxH3Ref2VA.json) (`group_offload` for the transformer, `on_demand` for the VAEs)
**Example:** [FluxDev.json](../workflows/flux/FluxDev.json) (`"offload": "model"`), [ZImage.json](../workflows/ZImage.json) (`"offload": "sequential"`), [MiniMaxH3.json](../workflows/minimax/MiniMaxH3.json) (`group_offload` per component), [MiniMaxH3Ref2VA.json](../workflows/minimax/MiniMaxH3Ref2VA.json) (`group_offload` for the transformer, `on_demand` for the VAEs)

## TF32 and cuDNN

Expand Down
8 changes: 4 additions & 4 deletions docs/IP_ADAPTER.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Some base models (e.g. Stable Diffusion 3.5) don't ship a default image encoder,
}
```

See [sd35ip.json](../examples/archive/sd35ip.json) for the full workflow.
See [sd35ip.json](../workflows/archive/sd35ip.json) for the full workflow.

## Image Argument

Expand Down Expand Up @@ -105,6 +105,6 @@ Can also reference a previous step's output:

## Examples

- [FluxIP.json](../examples/flux/FluxIP.json) — Flux with IP-Adapter for style transfer
- [sd35ip.json](../examples/archive/sd35ip.json) — SD 3.5 (quantized) with an explicit `image_encoder`/`feature_extractor` pair
- [ip-adapter.json](../examples/archive/ip-adapter.json) — SD 1.5 and SDXL, each with an `AutoPipelineForText2Image` step
- [FluxIP.json](../workflows/flux/FluxIP.json) — Flux with IP-Adapter for style transfer
- [sd35ip.json](../workflows/archive/sd35ip.json) — SD 3.5 (quantized) with an explicit `image_encoder`/`feature_extractor` pair
- [ip-adapter.json](../workflows/archive/ip-adapter.json) — SD 1.5 and SDXL, each with an `AutoPipelineForText2Image` step
6 changes: 3 additions & 3 deletions docs/LORAS.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,6 @@ python -m dw.run workflow.json lora="other-user/other-lora"

## Examples

- [FluxLora.json](../examples/flux/FluxLora.json) — Flux with realism LoRA and variables
- [lora.json](../examples/lora.json) — SD 3.5 with yarn art style LoRA
- [bnb_quant.json](../examples/archive/bnb_quant.json) — Quantized model with LoRA
- [FluxLora.json](../workflows/flux/FluxLora.json) — Flux with realism LoRA and variables
- [lora.json](../workflows/lora.json) — SD 3.5 with yarn art style LoRA
- [bnb_quant.json](../workflows/archive/bnb_quant.json) — Quantized model with LoRA
4 changes: 2 additions & 2 deletions docs/PROMPT_WEIGHTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,8 +71,8 @@ Any other pipeline whose class name starts with `Flux` (e.g. `FluxKontextPipelin
## Example

```bash
python -m dw.run examples/flux/FluxSchnellWeighted.json \
python -m dw.run workflows/flux/FluxSchnellWeighted.json \
prompt="a (cinematic:1.5) shot of a (dragon:1.3) breathing [smoke] over a (medieval:0.8) castle"
```

See [FluxSchnellWeighted.json](../examples/flux/FluxSchnellWeighted.json).
See [FluxSchnellWeighted.json](../workflows/flux/FluxSchnellWeighted.json).
10 changes: 5 additions & 5 deletions docs/QUANTIZATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ For 8-bit:
"arguments": { "load_in_8bit": true }
```

**Example:** [bnb_quant.json](../examples/archive/bnb_quant.json)
**Example:** [bnb_quant.json](../workflows/archive/bnb_quant.json)

## TorchAO

Expand All @@ -78,7 +78,7 @@ Common choices: `Int8WeightOnlyConfig` (any CUDA card), `Int4WeightOnlyConfig` (
`Float8DynamicActivationFloat8WeightConfig` (fastest, requires compute capability 8.9+ -
RTX 40-series/Ada or newer).

**Example:** [FluxTorchAO.json](../examples/flux/FluxTorchAO.json)
**Example:** [FluxTorchAO.json](../workflows/flux/FluxTorchAO.json)

**Pair TorchAO with `torch.compile`.** Int8 weight-only and float8 dynamic-activation
quant types get their fused-kernel speedups only under compilation - uncompiled they are
Expand Down Expand Up @@ -117,7 +117,7 @@ GGUF models load from single files using `from_single_file`:
}
```

**Example:** [FluxGGUF.json](../examples/flux/FluxGGUF.json)
**Example:** [FluxGGUF.json](../workflows/flux/FluxGGUF.json)

## SDNQ (SD.Next Quantization)

Expand Down Expand Up @@ -146,7 +146,7 @@ examples use for their large transformers and text encoders:
- `quantization_device` / `return_device` — where the quantization pass runs and where the finished component lands; quantizing on `cuda` is much faster than on CPU
- `use_quantized_matmul` — quantized matmul kernels (CUDA/XPU only)

**Examples:** [LTX2.json](../examples/LTX2.json), [MiniMaxH3.json](../examples/MiniMaxH3.json)
**Examples:** [LTX2.json](../workflows/ltx2/LTX2.json), [MiniMaxH3.json](../workflows/minimax/MiniMaxH3.json)

### Pre-quantized models

Expand All @@ -171,7 +171,7 @@ Pre-quantized SDNQ repos load as complete pipelines. The `sdnq` module must be i
- `pre_load_modules` — Imports sdnq before pipeline loading (registers quantization method)
- `sdnq_optimize` — Applies quantized matmul to listed components (CUDA/XPU only, skipped on MPS/CPU)

**Example:** [ZImageSDNQ.json](../examples/ZImageSDNQ.json)
**Example:** [ZImageSDNQ.json](../workflows/ZImageSDNQ.json)

## Modular Pipelines

Expand Down
Loading
Loading