[lstein]

Summary

A model load could read its record and then wait for the model-load lock. Meanwhile a load-affecting edit (e.g. cpu_only, fp8_storage) could commit and invalidate the cache. The load then built and admitted the pre-edit model, and later requests reused its stale weights.

What a cold load does now:

• On a cache miss, the load checks the caller's config against the stored record.
• If an edit has superseded it, the load raises StaleModelConfigError instead of building. Nothing is reserved or admitted.
• ModelLoadService.load_model then waits for the edit to finish, re-reads the record and retries once. Every caller gets this: invocations, the prompt utility routes, the image index and convert.
• A load whose files are missing because the model was moved and its record updated to the new path is retried the same way. A missing file under an unchanged record still raises.
• model_load_complete is sent however the load ends, so UIs never keep showing a load that has finished or failed.

No database access under MODEL_LOAD_LOCK: a long DB transaction (e.g. admin VACUUM) never stalls unrelated model builds or VRAM moves.

• Record writers that can change how a model loads (PATCH /i/{key} and reidentify) wrap their write and any cache invalidation in ModelLoadService.record_edit(key). They report on the yielded RecordEdit whether the write changed anything that loads.
• Before taking the lock, a cold load waits until no edit of its model is in progress, then reads the record.
• Under the lock, it only consults in-memory edit state. It is rejected if an edit of that model is still in progress, or if a load-affecting edit ended since its read. Edits that change nothing that loads, such as a rename, don't reject it.

Other details:

• The load-affecting predicate (Fix model manager cache and recovery failures #9669's config fingerprint) lives in services/model_load/model_load_common.py, so cache invalidation and the load check share one definition. The load check resolves path against the models directory, because the loader rewrites config.path to absolute in place.
• ModelLoader.config_check is an optional instance attribute that ModelLoadService sets. It is not a constructor argument, so registered loaders that override __init__ are unaffected.
• Model loaders must subclass ModelLoader, as the docs already required, because that is where the check runs. ModelLoaderRegistry.register rejects any other class with a TypeError, and ModelLoadService refuses one supplied by a custom registry.
• Loads that start after an edit has finished, and edits that don't affect loading, behave as before.

Related Issues / Discussions

Closes #9674. Follow-up to #9669. Remaining pre-existing race tracked in #9711.

QA Instructions

• uv run --no-sync pytest tests/app/services/model_load/test_stale_model_load_fence.py passes 14 tests. A fixture records, for every record read made by a loading thread, whether that thread held MODEL_LOAD_LOCK; the overlap tests assert none did. The tests cover:
  • a load whose record was superseded before it reached the lock loads the updated record, built once, with one started/complete event pair;
  • a load queued on the lock while an edit commits and invalidates (the [bug]: Prevent stale in-flight model loads from reentering cache #9674 ordering) loads the updated record;
  • an edit that starts while the load is queued and invalidates after the load holds the lock: the load is not built from the old record, whether the edit commits before the rejection or while the rejected load waits;
  • a rename landing while a load is queued doesn't reject it (built once, no retry);
  • a load overlapping an edit that is stuck behind a DB transaction waits without holding MODEL_LOAD_LOCK, so another thread can take it, and then loads the edit;
  • a DB transaction starting while a cold load is queued doesn't leave the load holding MODEL_LOAD_LOCK;
  • a second supersession during the retry raises after exactly two checks and still sends model_load_complete;
  • a model moved with its record loads from its new path, while missing files under an unchanged record still fail;
  • a config object reused after eviction, whose path the loader made absolute, still counts as current.
  • per-model edit tracking is dropped once no edit or load needs it, but kept while a queued load still watches it (a rename ending must not hide a load-affecting edit that follows).
  • a loader built directly on ModelLoaderBase is refused before it can cache anything, with the started/complete pair still sent.
• tests/backend/model_manager/load/test_model_loader_registry_entries.py checks that register rejects a function and a direct ModelLoaderBase subclass without changing the registry.
• tests/app/routers/test_model_manager.py and test_update_model_record_cache_invalidation.py check that PATCH /i/{key} commits and drops inside record_edit, that reidentify replaces inside it, and that the edit is flagged load-affecting exactly for fields in the fingerprint.
• Mutation checks. Each of these makes at least one test fail:
  • removing the retry, or retrying through a second started/complete pair;
  • re-reading the record under the lock;
  • not waiting for edits before the check, or before the retry's read;
  • counting every edit as load-affecting;
  • ignoring in-progress edits at the final check;
  • routes never clearing the flag, or the flag defaulting to not load-affecting;
  • removing either route's bracket;
  • no terminal event on failure;
  • no moved-file retry;
  • comparing paths without the models directory.
  • removing the registry's class check, or the service's.
• OMP_NUM_THREADS=4 uv run --no-sync pytest -n 4 tests/app/services/model_load tests/backend/model_manager tests/app/routers tests/app/services/model_records tests/app/services/model_install gave 3424 passed and 113 skipped, plus one failure in test_model_install.py::test_restore_metadata_timeout_preserves_partial_download_for_retry. That test fails intermittently on its own as well and exercises code this PR doesn't touch.
• ruff check and ruff format --check are clean.
• I did not do a manual reproduction against a running server.

Review

Findings resolved during review:

• The check is an attribute rather than a constructor argument, so it doesn't break third-party loaders.
• The retry moved from context.models.load() into the load service. Direct callers such as /expand-prompt and /image-to-prompt no longer return 500, and the loading indicator no longer sticks after a retry.
• Nothing touches the database under MODEL_LOAD_LOCK, including when a load overlaps an edit of the same model. Reidentify writes are bracketed like edits.
• Rename-only edits no longer reject overlapping loads.
• A load whose model was moved with its record retries instead of failing, and failed loads send model_load_complete.
• Per-model edit tracking no longer grows with every model ever edited.

Remaining limitations, which predate this change:

• An edit commits its record before taking the lock. A load that passes its final check just before that commit is admitted, and then dropped by the invalidation. If that entry is held, the drop only marks it stale, and ModelCache.get keeps serving it to same-cache requests until the hold is released. Tracked in Model settings edit can leave pre-edit cache entries serving requests (commit-before-invalidate race) #9711.
• A cold load of a model waits while that same model is being edited. That is usually brief, but if the edit is itself waiting on a long DB transaction, the load waits as long. Loads of other models are unaffected.

Compatibility / Rollout

Any writer of load-affecting model-record fields must use record_edit and report load_affecting (see the ModelLoadServiceBase.record_edit docstring). model_load_complete now also marks a failed load's end; both web UIs only use it to clear the loading indicator, and webv2 also refreshes its model list.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

🤖 Generated with Claude Code

[lstein]

If 9669 merges first, this PR will need conflict resolution.

[JPPhoto]

Merge blockers:

• None.

Other findings/issues:

• invokeai/app/services/shared/invocation_context.py:642: In the settings-edit race, the first load emits model_load_started but raises before model_load_complete; the retry emits another start. Webv2 appends both and removes only one matching label, leaving the queue progress rail showing a load after the invocation succeeds (invokeai/frontend/webv2/src/features/models/data/modelLoadStore.ts:27-38, QueueProgressRail.tsx:20-23). Effect: stale loading indicator. Likelihood: the exact invocation race this PR handles. Recovery: a connection or account reset clears it. Test: extend tests/app/services/model_load/test_stale_model_load_fence.py to assert no active load remains after the retry.

• invokeai/backend/model_manager/load/load_default.py:383: The new stale-config error reaches the direct /expand-prompt and /image-to-prompt callers, which return HTTP 500 through generic exception handlers (invokeai/app/api/routers/utilities.py:543-547, 659-663) instead of retrying with the current record. Effect: a valid prompt request fails and must be resubmitted. Likelihood: a settings edit overlaps a cache-miss load. Recovery: manual retry. Test: race each endpoint against a cpu_only or fp8_storage edit; assert retry succeeds or the endpoint returns an intentional non-500 conflict.

• invokeai/app/services/model_load/model_load_default.py:120: _config_is_current() reads SQLite while the cache-miss path holds MODEL_LOAD_LOCK.write_lock() (invokeai/backend/model_manager/load/load_default.py:370-383). get_model() takes the database lock, which /database/vacuum holds for the full VACUUM (invokeai/app/services/shared/sqlite/sqlite_database.py:112-130, 241-250). If vacuum begins after a caller reads the record, this load holds the global model lock while waiting; unrelated model builds and VRAM moves stall until vacuum finishes. Effect: unrelated generation/model operations pause. Likelihood: supported admin vacuum overlapping a cache miss. Recovery: automatic when vacuum completes. Test: hold the database lock after the caller reads its config, start a cold load, and verify another MODEL_LOAD_LOCK reader remains blocked until release.

Suggested follow-up work (pre-existing):

• invokeai/app/api/routers/model_manager.py:643: The record update commits before taking MODEL_LOAD_LOCK, as it did on the base. A load that passes the fence just before that commit can still admit the old config before invalidation; the PR body notes held stale entries may remain available. Suggested issue: close the commit/fence race and prevent held stale entries from serving later requests.

[lstein]

Thanks for the review. Addressed in e3ae73a.

1. Stale loading indicator (confirmed, fixed). The one-shot retry has moved out of ModelsInterface and into ModelLoadService.load_model. It now runs between model_load_started and model_load_complete, so a superseded load sends a single started/complete pair instead of started, started, complete. test_stale_model_load_fence.py now checks the event sequence exactly, both for a load that read the record before the edit and for one queued on the construction lock while the edit landed.

2. /expand-prompt and /image-to-prompt returning 500 (confirmed, fixed by the same change). Because the retry now lives in the service, every direct caller gets it: those two routes and also the image index worker. The worker holds onto its config object, so before this fix every cold load after a cpu_only edit failed its health probe until something triggered a re-activation. If the record changes again during the retry, the load still raises StaleModelConfigError and does not loop; a test covers this.

3. Record read under MODEL_LOAD_LOCK during VACUUM (confirmed, leaving as is). The mechanism is as you describe, and the _config_is_current docstring already calls it out. The read has to happen under the lock, because that is what makes it a fence. Reading earlier would let an edit's invalidation slip in between the read and the build. The only way I see to avoid the DB read is an in-memory generation counter, and that only works if callers capture it before they read the record, which means changing every call site. As for cost: VACUUM holds the single connection lock that every DB user in the process shares, so sessions on other GPUs stall at their next queue or image write anyway. The extra effect here is that they may stall a little earlier, at a VRAM move. For an admin-triggered, infrequent operation I don't think that justifies the extra machinery.

4. Commit-before-lock race / held stale entries (pre-existing). Agreed. I'll open a follow-up issue rather than widen this PR.

[JPPhoto]

Other findings/issues:

• invokeai/app/services/model_load/model_load_default.py:127-132: _config_is_current() reads SQLite while holding MODEL_LOAD_LOCK’s write lock. If a cold load reaches this check during admin VACUUM, unrelated model builds and VRAM moves wait for the database lock. Effect: unrelated model operations pause; Likelihood: rare overlap; Recovery: automatic when VACUUM ends. Test: no current test covers this lock overlap.

Corner/impossible cases to note (but don't spend time on fixing):

• invokeai/app/services/model_load/model_load_default.py:104-109: if a second settings edit supersedes the retry, the error propagates before model_load_complete; webv2 can retain the initial loading indicator. The added second-supersession test confirms the error path. Effect: stale progress indicator; Likelihood: rare, requiring another edit during retry; Recovery: connection or account reset. The missing terminal event on load errors is a pre-existing limitation.

Suggested follow-up work (pre-existing, make a new issue):

• invokeai/app/api/routers/model_manager.py:643-653: the record update still commits before acquiring MODEL_LOAD_LOCK, as it does in base. A load can pass its freshness check before the commit and race with invalidation. Suggested issue: close the commit/invalidation race and prevent held stale entries from serving later requests.

[lstein]

Thanks. Addressed in 615c3b9.

Record read under MODEL_LOAD_LOCK during VACUUM (fixed). A cold load now checks its record before taking the construction lock. A long DB transaction therefore delays only that load, not unrelated builds or VRAM moves.

Simply moving the read early would reopen the race this PR closes: an edit could commit after the early read but invalidate only after the load holds the lock. To prevent that, writers of load-affecting settings (PATCH /i/{key} and reidentify) wrap their commit and any cache drop in ModelLoadService.record_edit(key). Under the lock, a load re-reads its record only if such an edit was in progress at its check or started since. So the DB is read under the lock only while that same model is being edited.

New tests:

• an edit commits while the load is queued, and its invalidation lands after the load holds the lock;
• a DB transaction starts after the load read its record, while it is queued for the lock (another thread can still take MODEL_LOAD_LOCK);
• both routes keep the write inside the bracket.

Removing either part of the check, or either route's bracket, makes a test fail.

Second supersession during the retry. Acknowledged and not changed, as you suggested. It is the existing behaviour of every failed load, which sends no terminal load event. It is listed under the PR's remaining limitations.

Commit/invalidation race and held stale entries. Filed as #9711.

[JPPhoto]

Merge blockers:

• invokeai/app/services/model_load/model_load_default.py:170-175: When an edit is active or its generation changes, still_current() rereads the database. invokeai/backend/model_manager/load/load_default.py:374-385 calls it while holding MODEL_LOAD_LOCK. If a same-key edit is waiting on SQLite during a long transaction, the cold load holds the global lock while waiting too; unrelated model builds and VRAM moves stall. Effect: model loading pauses until the transaction ends. Likelihood: rare overlap of a queued cold load, same-key edit, and long transaction. Recovery: automatic when the transaction finishes. Test: the existing database-wait test has no active record_edit; add a barrier test for that overlap and verify unrelated lock users can proceed.

Other findings/issues:

• invokeai/backend/model_manager/load/load_default.py:281-286: The path-existence check runs before the new stale-config fence. If an in-flight request holds the old config after files are moved and the record is updated to a valid new path, the missing old path raises FileNotFoundError before the service can retry. This check predates the PR, but the new retry promise does not cover path edits. Effect: that request fails despite the valid updated record. Likelihood: plausible during a concurrent path update. Recovery: a fresh request uses the new path. Test: move the model, update its record, then load with the stale config and verify it retries successfully.

• invokeai/app/services/model_load/model_load_default.py:129-140: If a second edit supersedes the retry, _load() raises after model_load_started; model_load_complete is skipped. Effect: webv2 can keep showing the load until a connection or account reset. Likelihood: rare second edit during the retry. Recovery: reconnect or reset the account. Test: extend test_record_superseded_again_during_retry_raises to verify a terminal event clears the loading state.

[lstein]

Thanks. All three addressed in 26f58ea.

Blocker: record re-read under MODEL_LOAD_LOCK when a same-key edit overlaps (confirmed, fixed). Nothing touches the database under the lock any more.

• Before taking the lock, a cold load waits until no edit of its model is in progress, then reads the record. It holds no locks while it waits.
• Under the lock, the check only consults in-memory edit state. It rejects the load if an edit of that model is still in progress, or if a load-affecting edit has ended since the read.
• The rejected load then waits for the edit, re-reads the record and retries.
• Edits now report on the yielded RecordEdit whether they changed anything that loads. A rename, for example, no longer rejects an overlapping load. The flag defaults to load-affecting, so a write that fails partway stays conservative.

Tests:

• The overlap you described: an edit stuck behind a DB transaction while a cold load of the same model starts. Another thread can still take MODEL_LOAD_LOCK, and the load then picks up the edit.
• A new fixture records, for each record read by a loading thread, whether that thread held MODEL_LOAD_LOCK. The queued-edit tests assert it never did; HEAD's re-read fails them.
• A rename landing while a load is queued doesn't reject it.

Path edits and the existence check (fixed). If a load fails with FileNotFoundError and the stored record now loads differently (e.g. the model was moved and its path updated), the service retries with the current record. A missing file under an unchanged record still raises.

• Test: move the model, update its record, load with the stale config. It loads from the new path.

Missing terminal event after a second supersession (fixed). model_load_complete is now sent however the load ends. Both UIs use it only to clear their loading state, and webv2 also refreshes its model list, so this also fixes the stale indicator for any failed load.

• test_record_superseded_again_during_retry_raises and the missing-file test assert the started/complete pair.

The PR description is updated, including the remaining limitation: a cold load waits while that same model is being edited.

[JPPhoto]

Other findings/issues:

• invokeai/app/services/model_load/model_load_default.py:39-41 - _generations retains an entry for every key that has had a load-affecting edit, even after the model is deleted. Effect: gradual memory growth with high unique-key churn. Likelihood: low for typical use; it grows with the number of distinct edited keys. Recovery: restart; consider safe entry retirement. Test: edit and delete many distinct keys, then verify the registry stays bounded.

[lstein]

Thanks. Addressed in 774aa8e.

Unbounded edit tracking (confirmed, fixed). A model's edit count only matters to a load that has read the record but not yet done its check under MODEL_LOAD_LOCK. A load that starts later waits for in-progress edits and reads their result. So entries are now kept only while they can matter:

• The load check is a context manager that registers the load as a watcher of its model. It is held from before the lock until after it is released, so the registration is dropped on every exit path: a cache hit after waiting, a rejection, a construction error, or a database error during the record read.
• A model's entry is deleted as soon as it has no edits in progress and no watchers. The registry is bounded by the models being edited or loaded at that moment.

Tests:

• After edits to 100 distinct keys, plus a load and an edit of the test model, the registry is empty.
• A rename that finishes while a load is queued must not retire the entry that load watches, so a load-affecting edit that follows still rejects it. Retiring entries while they are watched fails this test, as do never retiring and never releasing a watcher.

The same commit also fixes the two fence tests that broke when main's #9702 removed ModelRecordServiceSQL._db (the linux-cpu failure on the merge commit). They now hold a write transaction through Database.begin(write=True), which on SQLite blocks reads as well. Both still fail if the check reads the database under the lock.

[JPPhoto]

Other findings/issues:

• invokeai/app/services/model_load/model_load_default.py:196-198 - The service installs the stale-load check only on ModelLoader. The registry contract accepts subclasses of ModelLoaderBase (model_loader_registry.py:43-47), so a custom loader registered directly from that base gets no fence. If a load-affecting edit overlaps its cold load, it can still admit a model built from the superseded record. Likelihood is low for documented integrations, which inherit ModelLoader; impact is stale loads for direct ModelLoaderBase extensions. Either enforce the narrower loader contract or expose the fence through the base contract. Add a regression test for a directly registered ModelLoaderBase implementation.

[lstein]

Thanks. Addressed in f588938.

Loaders built directly on ModelLoaderBase skip the check (confirmed, fixed). I enforced the narrower contract rather than moving the hook onto the base, because a direct implementer could still ignore a hook there. The docs already tell loader authors to inherit from ModelLoader, and all 106 registered loaders in the tree do.

• ModelLoaderRegistry.register raises TypeError for anything that isn't a ModelLoader subclass, so such a loader fails at import rather than with a stale load later. The registry's type hints are narrowed to match.
• ModelLoadService accepts any registry, so it also refuses a non-ModelLoader implementation before calling it. The refusal comes before any lock, edit registration or cache entry, and model_load_complete is still sent.

Tests:

• register rejects a direct ModelLoaderBase subclass, and the misplaced-decorator case (a bare function), without changing the registry. These replace the two post-import registry sweeps, which this check now covers.
• A registry that hands the service a direct ModelLoaderBase loader whose load_model caches a model: the load raises and nothing is cached.

Removing either check makes its test fail.

Separately, the windows-cpu failure on the previous run is test_workflow_last_run_at.py::test_update_last_run_at_sets_timestamp, which came in with #9702 and is flaky on main. Its workflow often gets the same millisecond created_at as the 43 bundled workflows. Windows' ~15 ms clock tick makes that tie almost certain there, and the workflow_id tie-break then usually pushes it off the 10-item page. That needs a separate fix on main.

[JPPhoto]

Everything is addressed and the race is tracked in #9711. Approved.