feat(models): calibrate VAE working memory per model from the Model Manager

[Pfannkuchensack]

Stacked on #9899 (and through it #9894, #9896, #9897, #9898). Until those merge, the diff also shows their commits. This PR's own changes are 87079a62d9, cb91aab513 and 7714be86b6; the rest are merges.

Summary

The shipped VAE working-memory estimates come from a few reference GPUs. A user's card, driver or allocator can need less, or more. Admins can now calibrate a model's VAE decode on their own GPU from the Model Manager, and estimates on that GPU then use the measurement.

Model detail, "Working memory" section (actions are admin-only)

• Calibrate: a dialog shows the planned runs (sizes per dtype, 1 warm-up + 2 measured each). The runs then go through the queue as short generations, with progress and Stop.
• Reset: removes this model's measurements on this GPU.
• Per line (operation · dtype): a table with Measured, Built-in and Calibrated estimate per size. Each column header has an info popover. Sizes that did not count are muted with their reason.
• States: never calibrated, in use, may be outdated, not used, too few sizes, running, incomplete, failed, unsupported, no GPU, more than one generation device, store unreadable, and measurement method not verified on this platform.
• Shared lines: models whose decode prices on the same line share the calibration. The section lists who measured it.

Estimates

• Within the measured sizes: the user line, an upper envelope over the counted measurements plus 5 % (the shipped margin).
• Above the largest measured size: the larger of the user line and built-in.
• No usable fit: built-in exactly as before.
• Environment changes: a hard change (backend, allocator, torch, settings) stops using the fit. A soft change (driver) keeps it and marks it "may be outdated".
• Below the working-memory floor (device_working_mem_gb), nothing changes.

Robustness found in hardware runs

• Finished attempts only: a calibration attempt's windows join no fit until every run of that attempt has finished. Larger cases are never priced from the same attempt's smaller ones.
• Fragmentation: allocator fragmentation in a warm server is detected per attempt and excluded. This happens on ROCm with expandable segments: reserved 1.3–1.47× allocated while allocated stays flat. The shipped-data builder uses the same rule (invokeai/backend/working_memory/fragmentation.py).
• Full offload before the measured lock: before a measured calibration lock, other unlocked models are moved off the GPU whole. The cache's partial offload otherwise leaves the measured model partly resident ([bug]: model cache leaves the requested model partly resident after splitting the last offload candidate #9907). Normal locks are unchanged.
• Plan capacity: the plan sizes cases for the capacity the cache will grant, including driver memory kept after earlier runs (HIP's retained pool).

Before/after on an RX 9060 XT (SDXL, fp32 decode)

	1024² reservation
Built-in	~6.1 GiB
User fit without the fragmentation rule (first hardware run)	8.3 GiB, which would evict the UNet again
This PR	follows the counted line up to 1152², below built-in (1152²: measured 6.95, calibrated 7.3, built-in 7.48 GB)

Related Issues / Discussions

• Builds on feat(e2e): add per-family end-to-end regression runner #9894, feat(working-memory): ship VAE working-memory coefficients as data #9896, feat(working-memory): measure working memory in admin calibration runs #9897, fix(working-memory): reserve VAE working memory per dtype and allocator #9898, feat(e2e): calibrate working memory and build shipped data from probe runs #9899.
• Found while testing: [bug]: model cache leaves the requested model partly resident after splitting the last offload candidate #9907 (the model cache leaves the requested model partly resident after a partial offload) and [bug]: first load after an update shows raw translation keys (service worker serves stale locales) #9908 (the first load after an update shows raw translation keys).

QA Instructions

Tests

• uv run --no-sync pytest -n 4: 4818 passed, 158 skipped, 5 xfailed. Covered: working-memory backend, calibration service, router and runner, session processor, tests/backend/model_manager/load, tests/app/invocations, the builder, tests/family_e2e.
• uv tool run ruff@0.11.2 check and format --check.
• API regenerated (make frontend-openapi/frontend-typegen equivalent); pnpm lint and pnpm test in invokeai/frontend/api.
• Frontend:
  • pnpm build (format, oxlint, tsc, architecture);
  • vitest src/features/models 255;
  • browser tests for the section and the graph compiler 24;
  • node --test scripts/mock-backend-fixtures.test.mjs 28;
  • pnpm test:performance:architecture passes.
• Full pnpm test here: 3 failures from the de-DE locale of this machine (number formatting) and 2 load timeouts that pass alone. All unrelated.

Mutation checks, each caught:

• the margin applied to the whole reservation;
• a user fit outside its range;
• live-attempt windows joining a fit;
• attempts pooled in the fragmentation rule;
• the calibration offload disabled, or applied outside calibration;
• splitting instead of whole moves;
• the plan's outside-usage floor removed;
• reset scope, stale-failure display, uncounted-row display.

Hardware, SDXL from the UI (18 runs per attempt) on an RTX 4090 (CUDA, cudaMallocAsync) and an RX 9060 XT (ROCm 10, Windows, expandable segments):

• Final attempt: 36 of 36 windows fully resident on both cards.
• 9060 XT: fp32 1152² counts; the earlier fragmented 1216² is excluded.
• 4090: calibrated below built-in at every size (0.91–0.96×).

Browser

• Every state on the mock backend (pnpm dev:mock; fault scenarios through POST /__faults) and on both real servers.
• Keyboard flows: Calibrate, Stop and Reset confirmations; focus lands on Stop after start and returns after each dialog; popovers open with Enter and close with Escape.
• Screenshots: to be attached.

Not verified on hardware: the Linux NVIDIA/AMD, MPS and XPU counter readers. The UI marks them as unverified.

Review

Resolved material findings

• Shared lines: they showed as "built-in". Reset copy now says other models' measurements still apply.
• Stale state: it never cleared after a recalibration. The header now ignores other environments and superseded estimator versions.
• Reset scope: Reset removed less than the summary attributed to the model.
• Fit refresh during an attempt: fits were refreshed mid-attempt and the plan was built from stale fits. Now held back per attempt.
• Fragmentation: fragmented windows set the user line, and windows from different attempts were pooled.
• Margin test: it could not catch a margin on analytic terms.
• UI states: a stale failure was shown after a lost connection, and the table columns were misaligned.
• Plan sizes: they ran partly resident on 16 GB cards.

Remaining limits

• First calibration on ROCm: the plan reads capacity live, so its largest size may not count until the retained pool has been recorded once. Partly resident windows never count.
• Growth rule margin: a clean attempt's first-to-largest growth was 1.095×, close to the 1.10 threshold. A false flag falls back towards built-in.
• Plan capacity formula: it re-implements the cache's budget rule in the app layer (without the reclaimable-block credit, so it is conservative).
• Encodes: image-to-image and inpainting encodes keep built-in estimates.

Compatibility / Rollout

• Store: the store file per device class under <db_dir>/working_memory/ comes from feat(working-memory): measure working memory in admin calibration runs #9897. New optional fields (attempt_id, run soft) are left out when absent, so older files load. No database migration.
• API: new GET /api/v2/models/i/{key}/working_memory/plan and /summary (404 for unknown models). attempt_id on calibrate. Line fields measured_by, own_windows, and size fields counted, excluded_reason. Generated openapi.json and schema.ts are updated.
• Without calibration: estimates are unchanged; the resolver falls back to the shipped values.
• Locale: +3.8 KB of strings. browser-baseline.json records the measured otherRawBytes (215114 B on every route, main 211323 B); nothing else in the baseline changed.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)