feat(working-memory): reserve Z-Image denoise working memory from measured lines

AGENTS.md

@@ -84,6 +84,7 @@ Main agent also owns machine-wide resources: dependency installs, generated API
 - `-n logical` fits CI runners that own the machine. Dev box: cap workers (`-n 4`, or plain `pytest`). Set `OMP_NUM_THREADS` when a run must stay light. Never start a suite during a generation.
 - `uv run --no-sync mypy scripts/invokeai-web.py` uses current exclusions. Optional diagnostic only: not a CI gate, not full type coverage.
 - Pick checks by behavior. Run required milestone gates before review. Rerun unchanged checks only with a reason.
+- Family E2E runs (`scripts/family_e2e`, guide: `docs/src/content/docs/development/Guides/family-e2e.mdx`) after changes to what a model computes or how it loads: model-manager loading, probing, configs; quantization; attention; model cache, working memory; invocations that run models; webv2 graph builders. Offer a run of the affected families, never start one unasked; none for docs-, test- or UI-only changes. Offer only where a manifest is set up; otherwise name the missing run under open limits in the final report. After a yes: run the base commit and `accept` it as reference (its bundle then reports BASELINE), then the branch, with `expect: changed` on the cases the PR is meant to change. PR description gets the `summary.md` table; attach the `bundle` zip. Until A/B runs exist, time differences under about 10 % between sessions are noise.
 - Once test dependencies exist, install hooks per checkout: `uv run --no-sync pre-commit install`. Committed config does not install hooks.
 
 ## Scratch files and repository hygiene

docs/src/content/docs/configuration/Optimization/low-vram-mode.mdx

@@ -172,6 +172,19 @@ device_working_mem_gb: 4
   Once decoding completes, the model manager "reclaims" the extra VRAM allocated as working memory for future model loading operations.
 :::
 
+#### Calibrating a model's working memory
+
+Invoke's decode estimates are measured on a few reference GPUs. Your GPU, driver and settings can need more or less. A calibration measures what a model's VAE decode really takes on your GPU, and Invoke uses those measurements from then on.
+
+- **What it runs**: a few short text-to-image generations of the model at three sizes, up to the largest your GPU can decode in one piece with the VAE fully loaded in the memory free on it when the calibration is planned, in each VAE precision the model can decode in. Before each measured operation, Invoke moves the other models off the GPU, so the measured model is fully loaded. Each size runs once to warm up and twice more. The generations go through the normal queue, so they wait behind queued work and appear there like any other generation.
+- **Who and where**: administrators only. It needs exactly one GPU set as the generation device; with several, it is not available.
+- **What it changes**: decodes up to the largest size measured use the measured values, plus the same 5% margin the built-in values carry. Measurements that a fragmented GPU memory pool inflated, or taken while the VAE was only partly on the GPU, are left out, and the Model Manager shows those sizes as not counted. Larger decodes use whichever of the measured and the built-in estimate is larger. Other models that decode the same way (for example, SD1.5 and SDXL models when you calibrate an SDXL model) use the measurements too. Because the estimate decides when a large decode is split into tiles up front, a calibration can change which large images are decoded in tiles.
+- **Where it applies**: on this kind of GPU only (identical cards share it), and only while the software that affects memory stays the same: the operating system, the PyTorch and GPU runtime versions, the memory-related environment variables and Invoke's memory settings. After such a change Invoke goes back to the built-in values until you calibrate again. A driver update keeps the measurements but marks them as possibly out of date until you calibrate again.
+- **What it does not cover**: image-to-image and inpainting encodes keep the built-in values.
+- **Reset**: remove a model's calibration from the Model Manager, or every calibration of this GPU; the change applies right away. Resetting one model removes only its own measurements: where other models' measurements cover the same decode, those still apply, and the Model Manager shows which models measured it. Otherwise the built-in values apply again.
+
+The measurements are stored next to the database in `working_memory/<device class id>.json`.
+
 ### Keeping a RAM weight copy
 
 Invoke has the option of keeping a RAM copy of all model weights, even when they are loaded onto the GPU. This optimization is _on_ by default, and enables faster model switching and LoRA patching. Disabling this feature will reduce the average RAM load while running Invoke (peak RAM likely won't change), at the cost of slower model switching and LoRA patching. If you have limited RAM, you can disable this optimization:

docs/src/content/docs/contributing/new-model-integration.mdx

@@ -601,7 +601,7 @@ Packages are discovered automatically. A directory of node modules without an `_
            ...
    ```
 
-   Report progress with `context.util.sd_step_callback(state, BaseModelType.NewModel)`; it renders previews through the architecture's `LatentSpaceFacet`. The denoise node is also where these are checked: interrupts (`context.util.is_canceled()`), inpainting, img2img through `denoising_start`, and a working-memory estimate passed to `model_on_device(working_mem_bytes=<int>)`, or `working_memory=` for an estimator that declares an `EstimatorSpec`.
+   Report progress with `context.util.sd_step_callback(state, BaseModelType.NewModel)`; it renders previews through the architecture's `LatentSpaceFacet`. The denoise node is also where these are checked: interrupts (`context.util.is_canceled()`), inpainting, img2img through `denoising_start`, and a working-memory estimate passed to `model_on_device()`: `working_memory=` from an estimator that declares an `EstimatorSpec`, so that calibration runs measure the forward (the Z-Image denoise's is in `invokeai/backend/z_image/denoise_working_memory.py`), or `working_mem_bytes=<int>` for a plain reservation. A model locked while the transformer is, such as a ControlNet adapter, reserves the estimate's `reservation_bytes` with `working_mem_bytes=`.
 
 4. **VAE encode and decode** (`vae/new_model_image_to_latents.py`, `vae/new_model_latents_to_image.py`)