Release 0.11.0

.env.example

@@ -0,0 +1,132 @@
+# diffusers-workflow environment knobs.
+#
+# Copy to .env and uncomment what you need. dw loads .env from the directory
+# a process starts in (load_dotenv() in dw/__init__.py), before diffusers or
+# torch is imported, so every line here reaches dw.serve, dw.run and each
+# spawned worker. A variable already set in the environment wins over this file.
+# Everything below is commented out, so copying this file as is changes nothing.
+# Workers read .env when they start, so restart dw.serve (or its idle workers)
+# after editing it.
+#
+# Never commit .env: it is git-ignored because tokens go in it.
+
+# --- dw: server and clients --------------------------------------------------
+
+# Bearer token for dw.serve. Set, every API, UI and MCP request must carry it.
+# dw.run and the stdio MCP server read the same variable, so one value
+# configures both ends. Required before binding to anything but localhost.
+# DW_API_TOKEN=some-long-random-string
+
+# This server's public origin, for links handed to someone not at the box
+# (docs/REMOTE.md). Overrides the 'public_url' setting.
+# DW_PUBLIC_URL=http://lem:8765
+
+# Where dw.run and the stdio MCP server find dw.serve.
+# DW_MCP_URL=http://127.0.0.1:8765
+
+# The workspace the stdio MCP session starts in - a name on the server.
+# DW_MCP_WORKSPACE=default
+
+# The cap on one wait_for_job call, in seconds (default 55). Raise it only if
+# your MCP client holds a tool call open longer than that.
+# DW_MCP_MAX_WAIT_SECONDS=55
+
+# --- dw: engine ----------------------------------------------------------------
+
+# The device for a CLI run: cuda, cuda:1, mps or cpu (default: detected).
+# dw.serve picks cards with --devices instead.
+# DW_DEVICE=cuda
+
+# The workspace directory: workflows, assets, outputs, prompts
+# (default ~/diffusers-workspace; dw.serve --workspace wins over it).
+# DW_WORKSPACE=~/diffusers-workspace
+
+# 'flat' restores the pre-run-directory output layout for scripts that glob
+# the output folder (default: one directory per run).
+# DW_OUTPUT_LAYOUT=flat
+
+# Where settings.json, the job database and the log live
+# (default ~/.diffusers_helper).
+# DIFFUSERS_HELPER_ROOT=~/.diffusers_helper
+
+# --- Defaults dw sets itself (shown so you know they exist) -----------------
+# Each is set at import only when absent; set it here to override.
+
+# Lets the CUDA allocator grow segments instead of fragmenting fixed ones.
+# PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+
+# Loads sharded checkpoints on several threads (default true; false on macOS,
+# where it segfaulted an SDNQ load on MPS).
+# HF_ENABLE_PARALLEL_LOADING=true
+
+# MPS only: let PyTorch use all unified memory.
+# PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0
+
+# torch.compile's cache, and with it SDNQ's Triton autotune results - the
+# first-step stall of a new size or frame count is paid once per box while
+# this survives reboots (default $XDG_CACHE_HOME/dw/torchinductor, or
+# ~/.cache/dw/torchinductor; PyTorch's own default under /tmp does not).
+# TORCHINDUCTOR_CACHE_DIR=~/.cache/dw/torchinductor
+
+# --- Hugging Face Hub (all of diffusers and transformers download through it) --
+
+# Token for gated models such as FLUX.1-dev (or run `huggingface-cli login` once).
+# HF_TOKEN=hf_...
+
+# Where models are cached (default ~/.cache/huggingface). Point HF_HUB_CACHE
+# at a bigger disk without moving the token and settings in HF_HOME.
+# HF_HOME=~/.cache/huggingface
+# HF_HUB_CACHE=/mnt/models/hub
+
+# Never touch the network: every model must already be cached. Also stops
+# diffusers checking Hub kernel publishers (see DIFFUSERS_TRUST_REMOTE_KERNELS).
+# HF_HUB_OFFLINE=1
+
+# Faster large downloads through the Xet backend, at the cost of more CPU
+# and memory while downloading.
+# HF_XET_HIGH_PERFORMANCE=1
+
+# Seconds before a slow download is abandoned (default 10).
+# HF_HUB_DOWNLOAD_TIMEOUT=60
+
+# HF_HUB_DISABLE_TELEMETRY=1
+
+# --- diffusers -----------------------------------------------------------------
+
+# Allow attention kernels from Hub publishers diffusers does not list as
+# trusted. Needed for "attention_backend": "sage_hub" (SageAttention 2 from
+# SageAttention/sage-attention) - about 30% faster per MiniMax-H3 step on an
+# RTX 3090. It downloads and runs remote code from any such publisher, not
+# only that one, so enable it deliberately. CUDA only; no use on MPS.
+# DIFFUSERS_TRUST_REMOTE_KERNELS=true
+
+# The attention backend for every model that sets none (default native). A
+# workflow's per-component "attention_backend" overrides it. Same values as
+# that setting: native, flash_hub, sage_hub, _native_cudnn, ...
+# DIFFUSERS_ATTN_BACKEND=native
+
+# Refuse any pipeline or component that would load custom code from the Hub.
+# dw's own trust gate (--trust-workflows) already covers workflow files; this
+# closes the same door at the diffusers layer.
+# DIFFUSERS_DISABLE_REMOTE_CODE=true
+
+# --- transformers --------------------------------------------------------------
+
+# Log level for transformers (debug, info, warning, error, critical).
+# TRANSFORMERS_VERBOSITY=error
+
+# Silence the tokenizers fork-safety warning after a tokenizer has been used.
+# TOKENIZERS_PARALLELISM=false
+
+# --- SDNQ quantization (CUDA) -----------------------------------------------------
+
+# SDNQ's int matmuls run on autotuned Triton kernels by default. 0 falls back
+# to PyTorch's kernels: no first-step autotune stall at a new shape, but
+# slower steps. Measure before turning it off.
+# SDNQ_USE_TRITON_MM=0
+
+# Narrow the autotune search to shorten that stall, at the risk of a slower
+# kernel. Comma lists; the defaults are shown.
+# SDNQ_TRITON_MM_BLOCK_SIZE_M_LIST=64,128,256
+# SDNQ_TRITON_MM_BLOCK_SIZE_N_LIST=64,128,256
+# SDNQ_TRITON_MM_BLOCK_SIZE_K_LIST=32,64

.github/codeql/dw-security/DwPathSanitizers.qll

@@ -127,14 +127,22 @@ private class UnconfinedPath extends Path::PathNormalization::Range {
  * own `os.path.realpath` / `os.path.exists` of the value it is in the
  * middle of checking. This is the "trusted rather than checked" half of
  * the trade described at the top of this file, and it is scoped to the
- * definitions in `dw/security.py`.
+ * definitions in `dw/security.py`, plus `validate_media_path` in
+ * `dw/locations.py`: it confines against several roots at once (the
+ * workflow directory, the asset libraries, the output root - so even with
+ * no base directory it confines), a check the query cannot see, and its
+ * own existence test of the confined path was reported as an access.
+ * tests/test_locations.py is where that containment is established.
  */
 private class ValidatorParameter extends PathInjection::Sanitizer {
   ValidatorParameter() {
     exists(Function validator |
-      pathValidatorName(validator.getName()) or nameValidatorName(validator.getName())
+      (pathValidatorName(validator.getName()) or nameValidatorName(validator.getName())) and
+      validator.getLocation().getFile().getRelativePath() = "dw/security.py"
+      or
+      validator.getName() = "validate_media_path" and
+      validator.getLocation().getFile().getRelativePath() = "dw/locations.py"
     |
-      validator.getLocation().getFile().getRelativePath() = "dw/security.py" and
       this.asExpr() = validator.getArg(_)
     )
   }

.github/workflows/ci.yml

@@ -37,23 +37,31 @@ jobs:
       - name: Install torch (CPU)
         run: pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
       - name: Install dependencies
+        id: install
         # requirements.txt resolves pyproject.toml's dependencies + server
         # extra; requirements-test.txt the dev extra. diffusers from git
         # goes on afterwards so the resolver doesn't replace it with a
         # release
         run: |
           pip install -c constraints-openapi.txt -r requirements.txt -r requirements-test.txt
           pip install git+https://github.com/huggingface/diffusers
+      # Each check below runs whenever the install succeeded, so one failing
+      # check never hides another: the ratchet was red for three days behind
+      # a failing test (#790)
+      # Guards the architecture metrics against drifting past the committed
+      # baseline (modules, size ceiling, cycles); see docs/stabilization/
+      - name: Architecture ratchet
+        if: ${{ !cancelled() && steps.install.outcome == 'success' }}
+        run: python scripts/arch_metrics.py --check docs/stabilization/baseline.json
       - name: Format check
+        if: ${{ !cancelled() && steps.install.outcome == 'success' }}
         run: ruff format --check dw dw_mcp tests scripts
       - name: Lint
+        if: ${{ !cancelled() && steps.install.outcome == 'success' }}
         run: ruff check dw dw_mcp tests scripts
       - name: Tests
+        if: ${{ !cancelled() && steps.install.outcome == 'success' }}
         run: pytest -q
-      # Guards the architecture metrics against drifting past the committed
-      # baseline (modules, size ceiling, cycles); see docs/stabilization/
-      - name: Architecture ratchet
-        run: python scripts/arch_metrics.py --check docs/stabilization/baseline.json
 
   ui:
     runs-on: ubuntu-latest

docs/ACCELERATION.md

@@ -134,7 +134,7 @@ Select the attention implementation diffusers uses for the duration of each pipe
 }
 ```
 
-Common values: `"flash"`, `"flash_hub"`, `"sage"`, `"sage_hub"`, `"native"`, `"flex"`. The full set is diffusers' `AttentionBackendName` enum - availability depends on what's installed (`flash-attn`, `sageattention`, etc.) and the platform. `_hub`-suffixed backends are fetched from the Hugging Face Hub kernel registry on first use, which needs the `kernels` package installed (`pip install kernels`) - it is not a dw dependency, and no bundled workflow sets a backend, so each runs on a plain install.
+Common values: `"flash"`, `"flash_hub"`, `"sage"`, `"sage_hub"`, `"native"`, `"flex"`. The full set is diffusers' `AttentionBackendName` enum - availability depends on what's installed (`flash-attn`, `sageattention`, etc.) and the platform. `_hub`-suffixed backends are fetched from the Hugging Face Hub kernel registry on first use, which needs the `kernels` package installed (`pip install kernels`) - it is not a dw dependency, and no bundled workflow sets a backend, so each runs on a plain install. diffusers also refuses a Hub kernel whose publisher it does not list as trusted - `sage_hub` among them, since SageAttention 2 comes from `SageAttention/sage-attention` - until `DIFFUSERS_TRUST_REMOTE_KERNELS=true`, which allows remote code from any such publisher. On an RTX 3090 `sage_hub` measured about 30% faster per MiniMax-H3 step (13.8 s against 19.2 s at 960x544x124, 54.7 s against 81.2 s at 345 frames); the local `sage` backend needs `sageattention>=2.1.1`, which PyPI does not carry.
 
 A component can also pin its backend persistently instead, via `set_attention_backend`:
 
@@ -186,7 +186,7 @@ Compile a component once it is fully configured - the graph captures final dtype
 
 Typical gains are 1.3-1.5x on diffusion transformers, and compilation stacks with the caches above. Notes:
 
-- **First run pays the compile cost.** The [server](SERVER.md)'s persistent worker keeps compiled pipelines loaded between runs, so the cost is paid once per session rather than once per generation.
+- **First run pays the compile cost.** The [server](SERVER.md)'s persistent worker keeps compiled pipelines loaded between runs, so the cost is paid once per session rather than once per generation. A new session still retraces, but the kernels it built come back from the on-disk cache, which dw keeps across reboots (`TORCHINDUCTOR_CACHE_DIR`, see [Environment Defaults](#environment-defaults)).
 - **Pin the attention backend** on a compiled component (`"attention_backend"` in the same `components` entry) rather than using the pipeline-level per-call context manager, which forces recompiles.
 - **Composes with offloading**: apply `group_offload` and `compile` on the same component and the offload hooks are installed first, as required. Skipped with a warning on MPS.
 - **Don't combine `fullgraph` with a `cache`**: the cache hooks decide skip-or-compute per step, a data-dependent branch diffusers wraps in `torch.compiler.disable` - it needs the graph break that `fullgraph: true` forbids. Compile with the default (partial) graph mode when a cache is active.
@@ -272,13 +272,14 @@ Device-level settings, read once at startup from `~/.diffusers_helper/settings.j
 
 ## Environment Defaults
 
-Set automatically at import unless already present in the environment (export your own value to override):
+Set automatically at import unless already present in the environment (export your own value to override, or put it in a `.env` file in the directory dw starts from - [.env.example](../.env.example) lists these and the other knobs worth knowing: the Hub, diffusers, transformers and SDNQ):
 
 | Variable | Default | Effect |
 | -------- | ------- | ------ |
 | `PYTORCH_CUDA_ALLOC_CONF` | `expandable_segments:True` | Lets the CUDA allocator grow segments instead of fragmenting fixed-size ones. Multi-step workflows churn differently-shaped allocations (generate, upscale, interpolate); fragmentation is what OOMs a card that nominally has room. |
 | `HF_ENABLE_PARALLEL_LOADING` | `true` | Loads sharded checkpoints in parallel - faster cold starts. |
 | `PYTORCH_MPS_HIGH_WATERMARK_RATIO` | `0.0` | MPS only - use all available unified memory. |
+| `TORCHINDUCTOR_CACHE_DIR` | `$XDG_CACHE_HOME/dw/torchinductor` (`~/.cache/dw/torchinductor`) | Keeps `torch.compile`'s on-disk cache across reboots - Inductor's own default is under `/tmp`. A new worker still retraces, but a cache hit skips codegen and kernel builds, the bulk of a cold compile. Entries are per shape, so a new size or frame count compiles once more. |
 
 For faster model downloads, optionally `pip install hf_transfer` and set `HF_HUB_ENABLE_HF_TRANSFER=1`. Not enabled automatically - it bypasses the Python HTTP stack and breaks some proxy setups.