[Pfannkuchensack]

Summary

End-to-end checks per model family are run by hand for almost every PR today. Several recent defects would have shown up in such a run early: a tokenizer that silently loaded a one-token vocabulary, missing working-memory reservations, and samplers that ignored the img2img start point.

This adds a developer tool, python -m scripts.family_e2e, that runs each locally installed family end to end through the real API and compares the result with an explicitly accepted reference and with the previous run:

• Cases from a local manifest: family, model keys, mode (txt2img, img2img, ControlNet), sizes, steps, seed, LoRAs, reference images. The manifest stays local because it depends on the installed models; the repo ships scripts/family_e2e/manifest.example.yaml.
• Graphs the UI would send: a new offline frontend script, pnpm graphs:export, builds the enqueue batches with the same graph builders and component policy as the Generate and Canvas widgets. The generate submission planning (prompt templates, negative-prompt suppression, dynamic prompts, seed plan, batch data) moved into Generation and Queue and is now shared by the workbench and the export, so the two cannot drift.
• Own server per run: a scratch root with a SQLite backup-API snapshot of the template root's database. use_cache=false on every node, the model cache is emptied before each family, then 1 warm-up and 3 measured runs per case and size.
• Measurements: PSNR against the reference image, total time and time per step, and VRAM peak per process and per node on driver level (D3DKMT, Windows; covers NVIDIA and AMD/ROCm). Model residency comes from the server log. Time and VRAM count only at 100 % residency; with fp8 also PSNR. Paging to shared memory and foreign GPU load invalidate a run and trigger a retry.
• Report: static HTML with reference, current run, previous run and an amplified difference image side by side, plus a Markdown summary. Outliers, invalid metrics, graph changes and environment changes (torch, driver, allocator, working memory, attention backend) are marked. References are written only by accept.
• Long runs: detached start that survives the agent harness, status, stop, resume, a per-GPU lock, and a kill-on-close job object so the server never outlives its runner.

Records keep per-node baseline and peak plus the raw memory trace, so several sizes per family can later feed working-memory calibration without a second run.

• Shareable results: bundle packs a run into a zip that opens on any machine. It holds the report and summary, rendered for the zip, with every shown image (this run, reference, previous run, difference) linked relatively, plus run.json, records.jsonl, graphs/ and traces/. A bundle.json holds the zip's format version, tool commit and creation time, so readers can refuse layouts they don't know.
• What the zip removes:
  • Absolute paths of the machine are cut to their last component or replaced by placeholders, model file paths included.
  • Environment variables keep their names; values of credential-like variables and credential settings are removed.
  • URLs lose credentials, query and fragment; private hosts never count as a shareable model source.
  • PNGs keep their pixel data byte for byte, so PSNR can be recomputed. Their text metadata is dropped, because InvokeAI embeds whole model configs there.
  • Logs, the heartbeat and scratch roots stay out. A run whose runner is still alive is refused.
• Model identity: run.json records name, base, type, format, variant, hash and a shareable source for every model a case's graph uses. The report and summary list them per case, since the local model key means nothing on another machine. Older runs without this field still load, report and bundle.
• Contributor guidance: AGENTS.md asks agents to offer a family run of the affected families after changes to model loading, quantization, attention, the model cache, model-running invocations or the frontend graph builders. They should never start one unasked, and the result goes into the PR as the summary table plus the bundle zip.

QA Instructions

Checks

• uv run --no-sync pytest tests/family_e2e tests/app/test_database_access_guard.py: 173 passed. The slow real-GPU sampler test passed on an RTX 4090 (-m slow).
• The bundle tests unpack the zip and check that:
  • every src/href resolves inside it;
  • decoded pixels equal the sources and no text chunk survives;
  • no entry, scanned as raw bytes with traces decompressed, contains a machine root, the home directory, fake credential values, or a model path with spaces;
  • verdicts match the on-disk report;
  • old runs without model identity still bundle;
  • a live runner is refused.
• uv tool run ruff@0.11.2 check / format --check on scripts/family_e2e tests/family_e2e: clean.
• Frontend (invokeai/frontend/webv2):
  • pnpm lint passes, including architecture:check.
  • pnpm test:fixtures: 35 passed.
  • pnpm test:performance:build passes with no baseline or pin changes.
• Frontend pnpm test: 10,276 passed. Known failures on this machine, not caused by this change:
  • Three digit-grouping tests fail under a German Windows locale: clusterStats, indexProgress, modelCacheStore.
  • Under full-suite load, the architecture scan tests (dependencyPolicy, typographyPolicy, workbenchOwnership) intermittently hit their 30 s / 60 s timeouts. They pass alone and inside pnpm lint.
• Exported batches validated with Graph.model_validate + validate_self() and Batch.model_validate: txt2img for every coverage shape of all 14 generate bases, img2img for 12 bases, ControlNet for 5. ERNIE-Image and Ideogram 4 correctly report that canvas generation is not supported.

Real hardware

• RTX 4090, Z-Image Turbo txt2img + img2img: the baseline matches the resident models plus context (14,190 MiB). A second run against the accepted reference gives PSNR ∞, time +0.4 %, VRAM +0.7 %. The batches stayed byte-identical (same graph hash) after the submission refactor.
• Fault injection on the 4090:
  • Forced partial loading marks time and VRAM invalid with the residency reasons.
  • A second start on the same GPU is refused by the lock.
  • Hard-killing the runner also kills the server.
  • stop and resume repeat the warm-up and add only the missing measurements.
  • The scratch root is removed after the run.
• RX 9060 XT (ROCm, Windows), SDXL: the server starts from the ROCm venv and the sampler reads the AMD adapter. Load from another process on the card was detected and the run was retried.
• Bundle on real runs: a fresh RTX 4090 run records model identity from the server: the main model, Qwen3 encoder and VAE with blake3 hashes. Its zip is 3.3 MB with 13 entries.
  • A second zip was built from PNGs whose metadata carries a local model path. Its PNGs contain only IHDR/IDAT/IEND, and a scan of all entries found no machine path.
  • Both zips, unpacked elsewhere and opened in a browser, show every image and resolve every link.
• UI: generated an image from the Generate widget with the refactored submission path. The enqueue payload carries seed and prompts in batch.data at the expected node paths.

Try it

1. Copy manifest.example.yaml, fill in your model keys, and stop the dev server for the template root.
2. Run python -m scripts.family_e2e start --manifest <file>, then status, accept <run>, report <run>, and bundle <run> for a shareable zip.
3. See the docs page Development → Guides → Family E2E Runs.

First findings from the tool

• Z-Image img2img is not reproducible with a fixed seed: the VAE encode is unseeded.
• RX 9060 XT, SDXL at 1024²:
  • The VAE decode reserves 14,650 MB of working memory, while the process peaks at 7.4–7.6 GB.
  • The UNet is therefore evicted and reloaded on every run.
  • Not investigated further in this PR.

Review

Resolved

• Stopping the server could kill an unrelated process after PID reuse. Kills now go through the job object or create-time-checked process handles.
• Time per step could be off by an integer factor when throttled preview frames fell on even steps.
• The scratch server could delete font files in the template root. Database-owned stores are now scratch paths. The database is snapshotted with the SQLite backup API, and a database held open by another process is refused.
• A server could outlive a crashed runner, and the scratch root could survive cleanup.
• The report hid partially invalid metrics and judged errors across superseded attempts.
• The batch assembly was duplicated between the workbench and the export. It is now shared.
• A server crash during a step now records an error, restarts the server and continues.
• The tool snapshotted the template database with its own SQLite driver. It now goes through the database layer's backup, and the template's files are only read.
• Bundle sanitization:
  • Text metadata in PNGs carried local model paths. It is now dropped while the pixel data stays identical.
  • Replacing environment values in the zip falsely changed the allocator verdict. Now only credential-like variables are replaced, and the reference and previous environments go through the same sanitizer.
  • Paths under the home directory and model paths with spaces could keep their tail.
  • LAN hosts passed as public model sources.

Remaining limits

• Memory sampling is Windows-only; Linux (NVML, amdgpu fdinfo) comes later.
• On ROCm/Windows, HIP keeps freed blocks at driver level, so per-node peaks after the first large node are upper bounds.
• Times drift by about 10 % between sessions on the same machine. Comparing two code states should use the interleaved A/B mode planned as a follow-up.
• The WMI fallback for detached starts (when job breakaway is refused) was not exercised on this machine.
• Free-text path matching in the bundle covers known roots and drive-letter paths exactly. UNC paths in free text, and on POSIX paths outside the known roots, are not rewritten. image_sha256 in a bundled record refers to the original PNG, before its metadata was removed.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)