diff --git a/.gitignore b/.gitignore index 47c8200f..0c9a77e2 100644 --- a/.gitignore +++ b/.gitignore @@ -148,6 +148,9 @@ test_outputs/ # built SPA copied into the package by scripts/build_dist.sh dw/server/ui/ + +# the guides dw_mcp serves, copied into the package by the same script +dw/docs/ venv # generated media - regenerable, archived outside the repo diff --git a/CLAUDE.md b/CLAUDE.md index a4cc6676..365c1e6c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,16 +12,16 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co # Install bash ./install.sh && source ./activate -# Run a workflow - sd15.json uses a small, ungated model and a literal +# Run a workflow - templates/text-to-image.json uses a small, ungated model and a literal # prompt, so it needs no Hugging Face login and downloads only a few GB -python -m dw.run workflows/sd15.json -python -m dw.run workflows/sd15.json prompt="a cat" num_images_per_prompt=4 +python -m dw.run workflows/templates/text-to-image.json +python -m dw.run workflows/templates/text-to-image.json prompt="a cat" num_images_per_prompt=4 -# A gated model (e.g. workflows/flux/FluxDev.json) needs Hugging Face auth +# A gated model (e.g. workflows/models/flux-dev.json) needs Hugging Face auth # first: huggingface-cli login # Validate a workflow against schema -python -m dw.validate workflows/ZImage.json +python -m dw.validate workflows/models/z-image.json # Basic system test (torch, diffusers import check) python -m dw.test diff --git a/README.md b/README.md index 470c9429..25558ff6 100644 --- a/README.md +++ b/README.md @@ -160,14 +160,14 @@ the pages and the HTTP API. ## The command line The engine runs standalone, with no server involved. -`workflows/sd15.json` is the smallest starting point — a small, ungated model +`workflows/templates/text-to-image.json` is the smallest starting point — a small, ungated model and a literal prompt, so the first run needs no Hugging Face login and downloads only a few GB: ```bash -python -m dw.run workflows/sd15.json -python -m dw.run workflows/sd15.json prompt="a cat" num_images_per_prompt=4 -python -m dw.validate workflows/sd15.json +python -m dw.run workflows/templates/text-to-image.json +python -m dw.run workflows/templates/text-to-image.json prompt="a cat" num_images_per_prompt=4 +python -m dw.validate workflows/templates/text-to-image.json ``` Most of the workflows under `workflows/` (Flux, LTX-2, MiniMax...) use **gated** @@ -177,7 +177,7 @@ then: ```bash huggingface-cli login -python -m dw.run workflows/flux/FluxDev.json +python -m dw.run workflows/models/flux-dev.json ``` Without this, the run fails partway through with an HTTP 401/403 from the Hub. @@ -186,7 +186,7 @@ An interactive REPL keeps models resident between runs for 2-4x faster iteration: ```text -dw> workflow load flux/FluxDev +dw> workflow load models/flux-dev dw> arg set prompt="a beautiful sunset" dw> workflow run [... models load once ...] diff --git a/docs/ACCELERATION.md b/docs/ACCELERATION.md index 36124df6..0bc0721d 100644 --- a/docs/ACCELERATION.md +++ b/docs/ACCELERATION.md @@ -42,7 +42,7 @@ Simplest and broadest support. Compares first-block residuals to decide whether Higher threshold = more speedup, more quality loss. Start with `0.05` and increase to taste. -**Example:** [FluxDevFirstBlockCache.json](../workflows/flux/FluxDevFirstBlockCache.json) +**Example:** [step-caching.json](../workflows/templates/step-caching.json) ### MagCache @@ -165,7 +165,7 @@ Some models have multiple variants with different coefficients: } ``` -**Example:** [FluxDevTeaCache.json](../workflows/flux/FluxDevTeaCache.json) +**Example:** [step-caching.json](../workflows/templates/step-caching.json) ## Cache vs TeaCache @@ -206,7 +206,7 @@ A component can also pin its backend persistently instead, via `set_attention_ba Prefer the pinned form for a compiled component - the per-call context manager switches implementations under the compiled graph and forces a recompile on every run. -**Example:** [Flux2Dev.json](../workflows/flux/Flux2Dev.json), [hunyuan15.json](../workflows/archive/hunyuan15.json), [Wan22TI2V5B.json](../workflows/archive/Wan22TI2V5B.json) +**Example:** [flux2-dev.json](../workflows/models/flux2-dev.json) ## Attention Slicing @@ -252,7 +252,7 @@ Typical gains are 1.3-1.5x on diffusion transformers, and compilation stacks wit - **Don't combine `fullgraph` with a `cache`**: the cache hooks decide skip-or-compute per step, a data-dependent branch diffusers wraps in `torch.compiler.disable` - it needs the graph break that `fullgraph: true` forbids. Compile with the default (partial) graph mode when a cache is active. - **TorchAO quantization needs compile to be fast** - see [QUANTIZATION.md](QUANTIZATION.md#torchao). -**Example:** [FluxDevFast.json](../workflows/flux/FluxDevFast.json), [FluxTorchAO.json](../workflows/flux/FluxTorchAO.json) +**Example:** [flux-dev-fast.json](../workflows/models/flux-dev-fast.json), [flux-torchao.json](../workflows/models/flux-torchao.json) ## Layerwise Casting @@ -296,7 +296,7 @@ Three older per-component knobs, set in the pipeline `configuration` beside `"residency": "on_demand"` on a component is the cheap case of the same trade: the model rests in system memory and is moved to the device whole around each of its own calls. That is a bad deal for anything called once per step, and a good one for a VAE called twice a run - it frees the VAE's VRAM for the denoise loop at the cost of two transfers, where group offloading the same VAE would restream it once per decode tile. See [On-demand components](WORKFLOW_GUIDE.md#on-demand-components). -**Example:** [FluxDev.json](../workflows/flux/FluxDev.json) (`"offload": "model"`), [ZImage.json](../workflows/ZImage.json) (`"offload": "sequential"`), [MiniMaxH3.json](../workflows/minimax/MiniMaxH3.json) (`group_offload` per component), [MiniMaxH3Ref2VA.json](../workflows/minimax/MiniMaxH3Ref2VA.json) (`group_offload` for the transformer, `on_demand` for the VAEs) +**Example:** [flux-dev.json](../workflows/models/flux-dev.json) (`"offload": "model"`), [z-image.json](../workflows/models/z-image.json) (`"offload": "sequential"`), [video-with-audio.json](../workflows/templates/minimax/video-with-audio.json) (`group_offload` per component), [reference-to-video.json](../workflows/templates/minimax/reference-to-video.json) (`group_offload` for the transformer, `on_demand` for the VAEs) ## TF32 and cuDNN diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index f5a45d5e..606fe19f 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -82,7 +82,7 @@ bash ./install.sh huggingface-cli login ``` -`workflows/sd15.json` uses an ungated model and needs no login - start there if you just want to confirm the install works. +`workflows/templates/text-to-image.json` uses an ungated model and needs no login - start there if you just want to confirm the install works. **Package conflicts:** Re-run the install script — it recreates the venv from scratch. diff --git a/docs/IP_ADAPTER.md b/docs/IP_ADAPTER.md index dfcd6d46..5524fce7 100644 --- a/docs/IP_ADAPTER.md +++ b/docs/IP_ADAPTER.md @@ -77,7 +77,7 @@ Some base models (e.g. Stable Diffusion 3.5) don't ship a default image encoder, } ``` -See [sd35ip.json](../workflows/archive/sd35ip.json) for the full workflow. +See [ip-adapter.json](../workflows/templates/ip-adapter.json) for the full workflow. ## Image Argument @@ -105,6 +105,5 @@ Can also reference a previous step's output: ## Examples -- [FluxIP.json](../workflows/flux/FluxIP.json) — Flux with IP-Adapter for style transfer -- [sd35ip.json](../workflows/archive/sd35ip.json) — SD 3.5 (quantized) with an explicit `image_encoder`/`feature_extractor` pair -- [ip-adapter.json](../workflows/archive/ip-adapter.json) — SD 1.5 and SDXL, each with an `AutoPipelineForText2Image` step +- [ip-adapter.json](../workflows/templates/ip-adapter.json) — Flux with IP-Adapter for style transfer +- [ip-adapter.json](../workflows/templates/ip-adapter.json) — FLUX.1 dev with an IP-Adapter, and the face-model variant in its description diff --git a/docs/LORAS.md b/docs/LORAS.md index d598a7a9..79269366 100644 --- a/docs/LORAS.md +++ b/docs/LORAS.md @@ -126,6 +126,6 @@ python -m dw.run workflow.json lora="other-user/other-lora" ## Examples -- [FluxLora.json](../workflows/flux/FluxLora.json) — Flux with realism LoRA and variables -- [lora.json](../workflows/lora.json) — SD 3.5 with yarn art style LoRA -- [bnb_quant.json](../workflows/archive/bnb_quant.json) — Quantized model with LoRA +- [lora.json](../workflows/templates/lora.json) — Flux with realism LoRA and variables +- [lora.json](../workflows/templates/lora.json) — SD 3.5 with yarn art style LoRA +- [lora.json](../workflows/templates/lora.json) — A LoRA adapter on FLUX.1 dev, with the SD 3.5 variant in its description diff --git a/docs/MCP.md b/docs/MCP.md index 6f13f935..c5627390 100644 --- a/docs/MCP.md +++ b/docs/MCP.md @@ -176,7 +176,7 @@ Nothing in this sequence costs GPU time. ## Tool reference -50 tools in six groups. Names and arguments below are transcribed from +52 tools in six groups. Names and arguments below are transcribed from `dw_mcp/server.py` — nothing here is renamed or reshaped for the docs. ### Catalog (read-only) @@ -188,9 +188,19 @@ pass it, without fetching every candidate's definition. Reusing a stored workflow is a preference, not a rule; `run_workflow` still takes an `inline_workflow` for a request nothing on disk covers. +A request usually names a *subject* ("a lego movie trailer set in the marvel +universe") while the catalog is written in *shapes* - a single image, an image +set, one shot, a multi-shot cut sequence, video with speech. Nothing in a +catalog entry will match the subject, so the shape is what has to be decided +first and matched against. `list_guides` indexes the shipped documentation by +section for exactly that, and `list_tasks` is what a shape gets composed from +when no single workflow covers it. + | Tool | Arguments | Purpose | | --- | --- | --- | -| `list_workflows()` | — | List stored workflows, each with its description, output kinds, step count, variable names, the stored prompts it references, and its `origin`/`writable` - a workflow from a read-only examples directory can be read and run but not saved over or deleted. The first call to make for a request an existing workflow might cover | +| `list_guides()` | — | List the documentation shipped with the engine: each guide's name, what it covers, and its section headings. The index is the routing table - match a request's shape against a heading rather than guessing | +| `get_guide(name, section=None)` | `name`, `section` | Get one guide whole, or one section of it. Prefer a section: a guide runs to thousands of lines. Section names match loosely, so a heading copied approximately still resolves | +| `list_workflows()` | — | List stored workflows, each with its description, output kinds, step count, variable names, the stored prompts it references, and its `origin`/`writable` - a workflow from a read-only examples directory can be read and run but not saved over or deleted. A `models/` entry also carries `configures`, the template it configures, or `configures_missing` naming it when that template does not resolve. The first call to make for a request an existing workflow might cover | | `get_workflow(name)` | `name` | Get one stored workflow's full JSON definition | | `get_schema()` | — | Get the JSON schema every workflow definition must satisfy | | `list_pipelines()` | — | List every diffusers pipeline class this installation provides | diff --git a/docs/PROMPT_WEIGHTING.md b/docs/PROMPT_WEIGHTING.md index ccbeb1a9..8ba6912a 100644 --- a/docs/PROMPT_WEIGHTING.md +++ b/docs/PROMPT_WEIGHTING.md @@ -71,8 +71,8 @@ Any other pipeline whose class name starts with `Flux` (e.g. `FluxKontextPipelin ## Example ```bash -python -m dw.run workflows/flux/FluxSchnellWeighted.json \ +python -m dw.run workflows/templates/prompt-weighting.json \ prompt="a (cinematic:1.5) shot of a (dragon:1.3) breathing [smoke] over a (medieval:0.8) castle" ``` -See [FluxSchnellWeighted.json](../workflows/flux/FluxSchnellWeighted.json). +See [prompt-weighting.json](../workflows/templates/prompt-weighting.json). diff --git a/docs/QUANTIZATION.md b/docs/QUANTIZATION.md index 64d7af90..0243f394 100644 --- a/docs/QUANTIZATION.md +++ b/docs/QUANTIZATION.md @@ -55,7 +55,7 @@ For 8-bit: "arguments": { "load_in_8bit": true } ``` -**Example:** [bnb_quant.json](../workflows/archive/bnb_quant.json) +**Example:** [flux2-dev.json](../workflows/models/flux2-dev.json) (a pre-quantized 4-bit checkpoint) ## TorchAO @@ -78,7 +78,7 @@ Common choices: `Int8WeightOnlyConfig` (any CUDA card), `Int4WeightOnlyConfig` ( `Float8DynamicActivationFloat8WeightConfig` (fastest, requires compute capability 8.9+ - RTX 40-series/Ada or newer). -**Example:** [FluxTorchAO.json](../workflows/flux/FluxTorchAO.json) +**Example:** [flux-torchao.json](../workflows/models/flux-torchao.json) **Pair TorchAO with `torch.compile`.** Int8 weight-only and float8 dynamic-activation quant types get their fused-kernel speedups only under compilation - uncompiled they are @@ -117,7 +117,7 @@ GGUF models load from single files using `from_single_file`: } ``` -**Example:** [FluxGGUF.json](../workflows/flux/FluxGGUF.json) +**Example:** [flux-gguf.json](../workflows/models/flux-gguf.json) ## SDNQ (SD.Next Quantization) @@ -146,7 +146,7 @@ examples use for their large transformers and text encoders: - `quantization_device` / `return_device` — where the quantization pass runs and where the finished component lands; quantizing on `cuda` is much faster than on CPU - `use_quantized_matmul` — quantized matmul kernels (CUDA/XPU only) -**Examples:** [LTX2.json](../workflows/ltx2/LTX2.json), [MiniMaxH3.json](../workflows/minimax/MiniMaxH3.json) +**Examples:** [text-to-video.json](../workflows/templates/ltx2/text-to-video.json), [video-with-audio.json](../workflows/templates/minimax/video-with-audio.json) ### Pre-quantized models @@ -171,7 +171,7 @@ Pre-quantized SDNQ repos load as complete pipelines. The `sdnq` module must be i - `pre_load_modules` — Imports sdnq before pipeline loading (registers quantization method) - `sdnq_optimize` — Applies quantized matmul to listed components (CUDA/XPU only, skipped on MPS/CPU) -**Example:** [ZImageSDNQ.json](../workflows/ZImageSDNQ.json) +**Example:** [z-image-sdnq.json](../workflows/models/z-image-sdnq.json) ## Modular Pipelines diff --git a/docs/RECIPES_24GB.md b/docs/RECIPES_24GB.md index 87eaad29..9de52b3d 100644 --- a/docs/RECIPES_24GB.md +++ b/docs/RECIPES_24GB.md @@ -20,26 +20,26 @@ The bf16 transformer is ~24GB - it does not fit alongside the T5 encoder. Two go | **float8 TorchAO** (RTX 40-series+) | same, with `Float8DynamicActivationFloat8WeightConfig` | Fastest, but fp8 needs compute capability 8.9+ (Ada). | | **GGUF Q8** | `from_single_file` Q8_0 transformer + `offload: "model"` | Simplest, best quality retention, slower than TorchAO+compile. | -**Examples:** [FluxDevFast.json](../workflows/flux/FluxDevFast.json) (the int8 recipe - swap in `Float8DynamicActivationFloat8WeightConfig` on Ada or newer), [FluxGGUF.json](../workflows/flux/FluxGGUF.json), [FluxDevFirstBlockCache.json](../workflows/flux/FluxDevFirstBlockCache.json) +**Examples:** [flux-dev-fast.json](../workflows/models/flux-dev-fast.json) (the int8 recipe - swap in `Float8DynamicActivationFloat8WeightConfig` on Ada or newer), [flux-gguf.json](../workflows/models/flux-gguf.json), [step-caching.json](../workflows/templates/step-caching.json) ## Qwen-Image (20B) Too large for bf16 on 24GB. Quantize the transformer (TorchAO int8/float8 or GGUF Q4/Q5) and model-offload the rest; add `compile` + `first_block` cache as for Flux. The Qwen2.5-VL text encoder is large - quantize it too, or group-offload it (`"text_encoder.model"` with `leaf_level`). -**Example:** [QwenImageEdit.json](../workflows/archive/QwenImageEdit.json) +The catalog no longer ships a Qwen-Image workflow - author one from [text-to-image.json](../workflows/templates/text-to-image.json) with the configuration above. ## Wan 2.2 - **TI2V-5B**: fits in bf16 on 24GB. No quantization needed - just `compile` + `cache` + VAE tiling for longer clips. - **T2V/I2V-A14B**: two 14B transformers (`transformer` + `transformer_2`). Quantize both (GGUF Q4/Q5 or TorchAO int8) and use `offload: "model"`; enable `vae.enable_tiling`. -**Examples:** [Wan22TI2V5B.json](../workflows/archive/Wan22TI2V5B.json), [Wan22T2V14B.json](../workflows/archive/Wan22T2V14B.json) +The catalog no longer ships Wan workflows - author one from [minimax/image-to-video.json](../workflows/templates/minimax/image-to-video.json) with the configuration above. ## HunyuanVideo (13B) Same shape as Flux: quantize the transformer (GGUF Q6/Q8, or TorchAO int8/float8 per the Flux table) + `offload: "model"` + `vae.enable_tiling`. The Llama text encoder benefits from group offload. Use `first_block` cache - video steps are expensive, caching pays off more than on images. -**Examples:** [HunyuanVideoGguf.json](../workflows/archive/HunyuanVideoGguf.json), [hunyuan15.json](../workflows/archive/hunyuan15.json) +The catalog no longer ships a HunyuanVideo workflow - author one from [ltx2/text-to-video.json](../workflows/templates/ltx2/text-to-video.json) with the configuration above. ## MiniMax-H3 @@ -92,7 +92,7 @@ model first (Z-Image drawing a subject, Music3 writing a song) must free it with `release_pipeline` before H3 loads - with it, the multi-model digital-short workflows below fit; without it, the load is an OOM kill, not a slowdown. -**Examples:** [MiniMaxH3Ref2VA.json](../workflows/minimax/MiniMaxH3Ref2VA.json), [MiniMaxH3Ref2VAChained.json](../workflows/minimax/MiniMaxH3Ref2VAChained.json), [MiniMaxH3I2V.json](../workflows/minimax/MiniMaxH3I2V.json), [MiniMaxH3SitcomShort.json](../workflows/minimax/MiniMaxH3SitcomShort.json) (five ref2va shots + two Z-Image portraits in ~35 minutes end to end) +**Examples:** [reference-to-video.json](../workflows/templates/minimax/reference-to-video.json), [chain-matched-to-audio.json](../workflows/templates/minimax/chain-matched-to-audio.json), [image-to-video.json](../workflows/templates/minimax/image-to-video.json), [dialogue-short.json](../workflows/templates/minimax/dialogue-short.json) (five ref2va shots + two Z-Image portraits in ~35 minutes end to end) ## LTX-2.5 (22B, video + audio) @@ -135,19 +135,19 @@ upsample the latents 2x, and the result is sharper than a single pass at 1536x89 fits where that would not. The base step shares its `vae` into the upsampler and sets `release_pipeline`, which frees the 11GB transformer before the 2x decode runs. -**Examples:** [LTX2.json](../workflows/ltx2/LTX2.json) (t2v), -[LTX2TwoStage.json](../workflows/ltx2/LTX2TwoStage.json) (base -> latent upsample -> mux), -[LTX2Keyframes.json](../workflows/ltx2/LTX2Keyframes.json) (first and -last frame), [LTX2Extend.json](../workflows/ltx2/LTX2Extend.json) (continue a clip), -[LTX2ICLora.json](../workflows/ltx2/LTX2ICLora.json) (generative 2x upscale via IC-LoRA), -[LTX2I2VEnhancePrompt.json](../workflows/ltx2/LTX2I2VEnhancePrompt.json) (native prompt +**Examples:** [text-to-video.json](../workflows/templates/ltx2/text-to-video.json) (t2v), +[two-stage.json](../workflows/templates/ltx2/two-stage.json) (base -> latent upsample -> mux), +[keyframes.json](../workflows/templates/ltx2/keyframes.json) (first and +last frame), [extend-clip.json](../workflows/templates/ltx2/extend-clip.json) (continue a clip), +[generative-upscale.json](../workflows/templates/ltx2/generative-upscale.json) (generative 2x upscale via IC-LoRA), +[enhance-prompt.json](../workflows/templates/ltx2/enhance-prompt.json) (native prompt enhancer and duration head) ## SDXL (2.6B UNet) Fits several times over in 24GB. Skip quantization and offloading entirely; `compile` the UNet if you generate many images per session. Use `num_images_per_prompt` batching with `vae.enable_slicing`. -**Example:** [sdxl.json](../workflows/archive/sdxl.json) +**Example:** [base-and-refiner.json](../workflows/templates/base-and-refiner.json) ## Multi-model workflows diff --git a/docs/REPL_COMMANDS.md b/docs/REPL_COMMANDS.md index c47a0de9..faa27d08 100644 --- a/docs/REPL_COMMANDS.md +++ b/docs/REPL_COMMANDS.md @@ -32,8 +32,8 @@ workflow restart Restart the worker process (clears GPU cache) ``` `workflow load` resolves names against the workflow directory (`./workflows` -by default), including subfolders - `workflow load flux/FluxDev` loads -`workflows/flux/FluxDev.json`. `workflow list` shows the available names. +by default), including subfolders - `workflow load models/flux-dev` loads +`workflows/models/flux-dev.json`. `workflow list` shows the available names. `workflow run ask ` prompts you interactively for ``'s value (the value is not saved to shell/readline history) before running — a shortcut @@ -101,7 +101,7 @@ cached; a second Ctrl+C stops the worker process itself. ## Typical Session ```bash -dw> workflow load flux/FluxDev +dw> workflow load models/flux-dev dw> arg show dw> arg set prompt="a majestic mountain landscape" dw> workflow run diff --git a/docs/REPL_WORKER_GUIDE.md b/docs/REPL_WORKER_GUIDE.md index c0e27e6d..d68f2b0b 100644 --- a/docs/REPL_WORKER_GUIDE.md +++ b/docs/REPL_WORKER_GUIDE.md @@ -13,7 +13,7 @@ python -m dw.repl ``` ```text -dw> workflow load flux/FluxDev +dw> workflow load models/flux-dev dw> arg set prompt="a cat wearing a hat" dw> workflow run # first run — loads model (~30-60s) dw> arg set prompt="a dog in a park" diff --git a/docs/SECURITY.md b/docs/SECURITY.md index 4299c833..244bea8a 100644 --- a/docs/SECURITY.md +++ b/docs/SECURITY.md @@ -88,7 +88,7 @@ own modeling code) and `custom_pipeline` (fetches and imports a pipeline module from the Hub or a local path). Neither goes through an `importlib` call of ours, so the dotted-name gate alone would leave diffusers' remote-code paths open. A workflow that needs them -(`workflows/Krea2Edit.json`, say) needs `--trust-workflows`. +(`workflows/templates/image-edit.json`, say) needs `--trust-workflows`. `--trust-workflows` is a blanket, process-wide choice - it is not scoped per-workflow or per-request. A `dw-serve` instance that accepts jobs from diff --git a/docs/SERVER.md b/docs/SERVER.md index 32c34732..dc446af0 100644 --- a/docs/SERVER.md +++ b/docs/SERVER.md @@ -191,9 +191,12 @@ The editor's forms come from these; they are just as usable from scripts: - `GET /api/workflows` — the stored workflow names, plus a `details` entry per workflow: `description`, `kinds` (the output content types' top-level halves), `steps`, `variables` (a count) and `variable_names`, and - `prompt_refs` naming the stored prompts it leans on. Enough to choose a - workflow and know what to pass it without reading each one; the variable - defaults are deliberately left out, being an order of magnitude more + `prompt_refs` naming the stored prompts it leans on, and `configures` - for + a workflow under `models/`, the `templates/` name it is a tuned + configuration of, empty when it is a template itself or when the name does + not resolve (then `configures_missing` carries what was written). Enough + to choose a workflow and know what to pass it without reading each one; the + variable defaults are deliberately left out, being an order of magnitude more payload on a listing the UI reloads. Cached by file mtime - `GET/PUT/DELETE /api/workflows/{name}` — read, save, delete workflow files (confined to `--workflow-dir`) diff --git a/docs/TASKS.md b/docs/TASKS.md index c814331b..4eac74b9 100644 --- a/docs/TASKS.md +++ b/docs/TASKS.md @@ -305,7 +305,7 @@ the joined length, not the sum. When every input carries audio, the tracks are crossfaded over exactly the seam's span so they stay in step with the picture; when any input is silent the result is, and `pair_audio` puts a score under it. -**Example:** [GyreDissolve.json](../workflows/gyre/GyreDissolve.json) +**Example:** [dissolve-between-shots.json](../workflows/templates/dissolve-between-shots.json) ### stabilize_video @@ -335,7 +335,7 @@ before the task ever saw it. Frames are shifted back and the result is cropped to the region every frame covers, then resized to the original size; a soundtrack passes through untouched. -**Example:** [GyreDissolve.json](../workflows/gyre/GyreDissolve.json) +**Example:** [dissolve-between-shots.json](../workflows/templates/dissolve-between-shots.json) ### video_frames @@ -364,7 +364,7 @@ An argument that goes through diffusers' video processor instead - LTX-2's IC-LoRA references - wants the `[0, 1]` frames the pipeline returned rather than this array; hand those over with `previous_result:step.frames`. -**Example:** [LTX2Extend.json](../workflows/ltx2/LTX2Extend.json) +**Example:** [extend-clip.json](../workflows/templates/ltx2/extend-clip.json) ### pair_audio @@ -392,7 +392,7 @@ returns frames without it, and this puts it back: | `audio` | Yes | The soundtrack - a waveform, or the earlier step whose video carried one, which brings its sample rate along | | `sample_rate` | No | Sample rate of the waveform. Required unless `audio` carries one; given here it wins | -**Example:** [LTX2TwoStage.json](../workflows/ltx2/LTX2TwoStage.json) +**Example:** [two-stage.json](../workflows/templates/ltx2/two-stage.json) ### slice_audio @@ -470,7 +470,7 @@ ending. The curve is the equal-power cosine the seam joins use: | `fade_out_ms` | No | Length of the fade out, to the tail of the track (default: 0) | | `sample_rate` | With a waveform | Sample rate of a directly passed waveform (files carry their own) | -**Example:** [TrimFadeAudio.json](../workflows/tasks/TrimFadeAudio.json) — slice a generated track to length, then fade the cut into an ending. +**Example:** [audio-trim-fade.json](../workflows/templates/audio-trim-fade.json) — slice a generated track to length, then fade the cut into an ending. ### normalize_audio @@ -500,7 +500,7 @@ changes, so the dynamics survive: A silent track is returned unchanged. -**Example:** [GyreDissolve.json](../workflows/gyre/GyreDissolve.json) +**Example:** [dissolve-between-shots.json](../workflows/templates/dissolve-between-shots.json) ### mix_audio @@ -533,7 +533,7 @@ shorter than the picture leaves the tail dry rather than cutting the picture down to fit. Summing can push peaks past full scale and the sum is *not* rescaled - follow it with `normalize_audio` to bring the peak back down. -**Example:** [GyreDissolve.json](../workflows/gyre/GyreDissolve.json) — a +**Example:** [dissolve-between-shots.json](../workflows/templates/dissolve-between-shots.json) — a generated score mixed under the shots' own audio. ### resample_audio @@ -563,7 +563,7 @@ supplied recording once, up front, feeds it what it already wants: A track already at the target rate is returned untouched. The conversion is PyAV's, which dw already needs for video - no torchaudio dependency. -**Example:** [GyreAssemble.json](../workflows/gyre/GyreAssemble.json) +**Example:** [assemble-and-score.json](../workflows/templates/assemble-and-score.json) ## Data Gathering @@ -653,8 +653,8 @@ Upscale images using spandrel-compatible super-resolution models (ESRGAN, SwinIR Large images are automatically tiled to avoid GPU memory issues. Models can be loaded from HuggingFace Hub repos or local `.pth`/`.safetensors` files. **Examples:** -- [SpandrelUpscale.json](../workflows/tasks/SpandrelUpscale.json) — Generate at 512px, then 4x upscale to 2048px. -- [UpscaleImage.json](../workflows/tasks/UpscaleImage.json) — Upscale an image you already have; there is no generation step, so the input is a path or URL. +- [upscale-spandrel.json](../workflows/templates/upscale-spandrel.json) — Upscale any existing image 4x. +- [upscale-spandrel.json](../workflows/templates/upscale-spandrel.json) — Upscale an image you already have; there is no generation step, so the input is a path or URL. ## Diffusion Upscaling @@ -690,8 +690,8 @@ Two modes are available: | `noise_level` | No | Noise level for x4 mode (default: 20, ignored for x2) | **Examples:** -- [DiffusionUpscale.json](../workflows/tasks/DiffusionUpscale.json) — Generate at 512px, then upscale. `mode` selects which: `x4` (the default) reaches 2048px, `x2` reaches 1024px through the latent upscaler. -- [DiffusionUpscaleImage.json](../workflows/tasks/DiffusionUpscaleImage.json) — Prompt-guided upscale of an image you already have, with no generation step. +- [upscale-diffusion.json](../workflows/templates/upscale-diffusion.json) — Upscale any existing image. `mode` selects which: `x4` (the default) reaches 2048px, `x2` reaches 1024px through the latent upscaler. +- [upscale-diffusion.json](../workflows/templates/upscale-diffusion.json) — Prompt-guided upscale of an image you already have, with no generation step. ## Face Restoration @@ -725,7 +725,7 @@ Restore and enhance faces in images using spandrel-compatible face restoration m Models are loaded via spandrel, so any `.pth`/`.safetensors` face restoration weights work. CodeFormer requires `pip install spandrel-extra-arches` (non-commercial license). -**Example:** [FaceRestore.json](../workflows/tasks/FaceRestore.json) — Generate a portrait, then restore faces with GFPGAN v1.4. +**Example:** [restore-faces.json](../workflows/templates/restore-faces.json) — Generate a portrait, then restore faces with GFPGAN v1.4. ### Combining with Upscaling @@ -801,8 +801,8 @@ Returns a grayscale PIL Image (mode "L") — white (255) for detected objects, b **Examples:** -- [Segment.json](../workflows/tasks/Segment.json) — Segment an object from an image -- [SegmentAndInpaint.json](../workflows/tasks/SegmentAndInpaint.json) — Segment, then inpaint the masked region +- [segment.json](../workflows/templates/segment.json) — Segment an object from an image +- [segment-and-inpaint.json](../workflows/templates/segment-and-inpaint.json) — Segment, then inpaint the masked region ## Image Captioning @@ -849,9 +849,9 @@ For Florence-2's advanced task-token captioning (detailed captions, object detec **Examples:** -- [ImageToText.json](../workflows/tasks/ImageToText.json) — Basic captioning with the default model, saves as `.txt` -- [ImageToTextVLM.json](../workflows/tasks/ImageToTextVLM.json) — Larger VLM answering a specific question -- [CaptionToImage.json](../workflows/tasks/CaptionToImage.json) — Caption an image, then regenerate with Flux +- [image-to-text.json](../workflows/templates/image-to-text.json) — Basic captioning with the default model, saves as `.txt` +- [image-to-text.json](../workflows/templates/image-to-text.json) — Larger VLM answering a specific question +- [describe-and-regenerate.json](../workflows/templates/describe-and-regenerate.json) — Describe an image, expand the caption, then regenerate it ## Extracting Sections @@ -950,12 +950,63 @@ It is merged after everything else, so it can override `repetition_penalty` and **Examples:** -- [ExpandPrompt.json](../workflows/tasks/ExpandPrompt.json) — Expand a short prompt and save as `.txt` -- [ExpandAndGenerate.json](../workflows/tasks/ExpandAndGenerate.json) — Expand prompt, then generate with Flux +- [expand-prompt.json](../workflows/templates/expand-prompt.json) — Expand a short prompt and save as `.txt` +- [expand-prompt.json](../workflows/templates/expand-prompt.json) — Expand prompt, then generate with Flux + +## Speech Generation + +Speak a line of text with a local text-to-speech model. The result is a waveform carrying the rate its model generated at, so it composes with [`slice_audio`](#slice_audio), [`fade_audio`](#fade_audio) and [`pair_audio`](#pair_audio) directly ([`concat_videos`](#concat_videos) and `dissolve_videos` join videos — pair the track onto a video first). + +```json +{ + "task": { + "command": "generate_speech", + "arguments": { + "text": "The way ahead is longer still.", + "voice_preset": "v2/en_speaker_6" + } + }, + "result": { "content_type": "audio/wav" } +} +``` + +| Argument | Required | Description | +| -------- | -------- | ----------- | +| `text` | Yes | The line to speak | +| `model_name` | No | HuggingFace model ID (default: `suno/bark-small`) | +| `voice_preset` | No | The speaker, for a model with presets — `v2/en_speaker_0` through `v2/en_speaker_9` for Bark. A model with no processor (a single-voice model such as `facebook/mms-tts-eng`) refuses a `voice_preset` with an error rather than ignoring it | +| `forward_params` | No | Passed to the model's forward/generate call | +| `generate_kwargs` | No | Ad-hoc generation settings for a generative model — `temperature`, `do_sample` | + +The default is Bark because its voice presets give distinct speakers, which is what two characters in a scene need; `facebook/mms-tts-eng` is a quarter the size and a good override where one voice will do. `voice_preset` is a preprocessing argument — it selects the speaker before generation rather than parameterizing it — so naming it here is what makes it reach the processor. Passed through `forward_params` it would be dropped and every character would sound the same. + +The result needs no `sample_rate`. A generated track carries the rate its model produced it at, and that beats the 44100 default; declaring one still wins over both, for a track whose rate was reported wrong. Every TTS model runs at a different rate, so a declared rate that does not match plays the speech at the wrong speed and pitch without ever failing. + +### Generating a voice to condition on + +The role this earns its place in is voice *timbre reference*, not the track a mouth follows. MiniMax H3 lip-syncs well when it generates the speech itself and poorly when it must follow supplied audio, so its `MiniMaxH3AudioReference` takes a few seconds of a voice to fix timbre, pitch and delivery while H3 still generates the line. Build the reference with `from_previous_result` and the clip's own sample rate comes across with it: + +```json +"references": [ + { + "reference_type": "diffusers.modular_pipelines.minimax_h3.MiniMaxH3AudioReference", + "from_previous_result": "voice" + } +] +``` + +Referencing the same preset in every shot of a scene makes a character's voice a conditioning signal rather than a prose description that has to land identically a dozen times. The other honest uses are a voice that must be matched — a specific delivery H3 will not produce from description alone — and narration over shots where nothing has to lip-sync to it, muxed with [`pair_audio`](#pair_audio). + +A speech model is worth releasing before a video model loads — set `release_models` on the step, as in the example below. + +**Examples:** + +- [generate-speech.json](../workflows/templates/generate-speech.json) — Speak a line and save it as a `.wav` +- [voice-timbre-reference.json](../workflows/templates/minimax/voice-timbre-reference.json) — Generate a voice, then condition H3's `