Add max_loaded_models server option with LRU eviction of idle models - #298
Merged
Conversation
The server previously kept every model resident in memory forever once loaded, with no way to enforce a residency bound. Add a native max_loaded_models setting (server.json key or --max-loaded-models CLI flag): 0 keeps the old unlimited behavior (default), 1 enforces a single loaded model at a time, and N keeps the N most recently used models warm. Enforcement lives in ensure_model_loaded_locked(), the single funnel all loads go through (startup eager loads, lazy first-request loads, and WebUI-driven loads). Before loading past the limit, the least recently used idle model is unloaded first, freeing VRAM on GPU backends; its next request reloads it. A model mid-inference is never evicted: victim busy guards are only try-acquired (a blocking wait could deadlock two loads evicting each other's target), and when every loaded model is busy the request fails fast with the existing 503 busy path so clients can retry. Loads are serialized while the limit is active so concurrent lazy loads cannot overshoot it. At startup, non-lazy models beyond the limit are registered but deferred to first use instead of churning through loads that would be immediately evicted. Also fold the duplicated inline teardown blocks into LoadedModel::unload() and make it clear the loaded flag, fixing /v1/models reporting stale "loaded": true after /v1/tasks/unload_models and unload_all_models.
Owner
|
@SelfRef Merged. Thanks! |
NairoDorian
added a commit
to NairoDorian/speech.cpp
that referenced
this pull request
Aug 22, 2026
Brings speech.cpp up to date with latest audio.cpp upstream main (4d383be): - Community models: MOSS-VoiceGenerator (PR 0xShug0#278), MMS-300M-1130 forced aligner (PR 0xShug0#279), F5-TTS (PR 0xShug0#275). - SenseASR encoder refactored to framework SAN-M modules (PR 0xShug0#285). - Server: max_loaded_models limit with LRU eviction (PR 0xShug0#298) and opt-in session options listing. - WebUI: reverse proxy hash routing (PR 0xShug0#297), Music3 precision packages, HeartMuLa options. - CUDA & Memory: CUDA graph-cache eviction and idle pool trimming (PR 0xShug0#293), Supertonic vector arena reduction. - GGML: tracked CUDA clear_graph and trim_pools as patch 0007. - Build & CI: native model manager build flags, C++17 cleanups, CMake model-link guards preserved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The server has no way to enforce a bound on loaded models. Once a model is
used it stays resident in memory forever (documented as a warning in both
READMEs), so a multi-model config on a device that only fits one model at a
time cannot be served safely. The only "one model at a time" behavior today
lives client-side in the WebUI, which manually unloads other models before
loading a new one - nothing enforces it natively on the server, and API
clients get no protection at all.
Solution
A new
max_loaded_modelssetting, following the existingbusy_timeout_msoption pattern (config field, JSON parsing and validation, CLI override,
help text, tests):
server.json:"max_loaded_models": 1, or CLI:--max-loaded-models 10(default): no limit, exactly the current behavior1: enforces a single loaded model at a timeN: keeps the N most recently used models warm; loading one more firstunloads the least recently used idle model (freeing VRAM on GPU
backends), and the evicted model transparently reloads on its next
request
How enforcement works
ensure_model_loaded_locked(), the single funnelevery load passes through (startup eager loads, lazy first-request
loads, WebUI
/v1/models/load), so no path can bypass it.last_used_msstamp on every load/run;evict_for_model_limit()unloads the oldest idle models until theincoming one fits.
try-acquired via a new non-blocking
BusyGuard::try_acquire()(ablocking wait could deadlock two loads evicting each other's target).
If the limit is reached and every resident model is busy, the request
fails fast through the existing 503
server_busypath so the clientcan retry.
model_load_mutex_while the limit isactive, so two concurrent lazy loads cannot both pass the residency
check and overshoot the limit. With the limit off (0), loads stay
concurrent as before.
deferred to first use (with a log note), instead of churning through
loads that would be immediately evicted.
Also in this PR
LoadedModel::unload()never cleared theloadedflag, so/v1/modelskept reporting
"loaded": trueafter/v1/tasks/unload_modelsand/v1/tasks/unload_all_models. Eviction reusesunload(), so this fixesthe flag there and folds the duplicated inline teardown blocks in
handle_model_load(reconfigure path) andhandle_model_unloadinto it.Docs and tests
app/server/README.mdand the rootREADME.md(including updating the "never unloads" warnings), plus
--help.are rejected (top-level and CLI validation mirror
busy_timeout_ms).Validation
server_config_testandserver_busy_guard_testpass.max_loaded_modelsomitted, behavior isunchanged and no new locking is engaged on the load path.