Skip to content

Add max_loaded_models server option with LRU eviction of idle models - #298

Merged
0xShug0 merged 1 commit into
0xShug0:mainfrom
SelfRef:feat/max-loaded-models
Aug 22, 2026
Merged

Add max_loaded_models server option with LRU eviction of idle models#298
0xShug0 merged 1 commit into
0xShug0:mainfrom
SelfRef:feat/max-loaded-models

Conversation

@SelfRef

@SelfRef SelfRef commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Problem

The server has no way to enforce a bound on loaded models. Once a model is
used it stays resident in memory forever (documented as a warning in both
READMEs), so a multi-model config on a device that only fits one model at a
time cannot be served safely. The only "one model at a time" behavior today
lives client-side in the WebUI, which manually unloads other models before
loading a new one - nothing enforces it natively on the server, and API
clients get no protection at all.

Solution

A new max_loaded_models setting, following the existing busy_timeout_ms
option pattern (config field, JSON parsing and validation, CLI override,
help text, tests):

  • server.json: "max_loaded_models": 1, or CLI: --max-loaded-models 1
  • 0 (default): no limit, exactly the current behavior
  • 1: enforces a single loaded model at a time
  • N: keeps the N most recently used models warm; loading one more first
    unloads the least recently used idle model (freeing VRAM on GPU
    backends), and the evicted model transparently reloads on its next
    request

How enforcement works

  • Enforcement sits in ensure_model_loaded_locked(), the single funnel
    every load passes through (startup eager loads, lazy first-request
    loads, WebUI /v1/models/load), so no path can bypass it.
  • Each model gets a steady-clock last_used_ms stamp on every load/run;
    evict_for_model_limit() unloads the oldest idle models until the
    incoming one fits.
  • A model mid-inference is never evicted: victim busy guards are only
    try-acquired via a new non-blocking BusyGuard::try_acquire() (a
    blocking wait could deadlock two loads evicting each other's target).
    If the limit is reached and every resident model is busy, the request
    fails fast through the existing 503 server_busy path so the client
    can retry.
  • Loads are serialized by a new model_load_mutex_ while the limit is
    active, so two concurrent lazy loads cannot both pass the residency
    check and overshoot the limit. With the limit off (0), loads stay
    concurrent as before.
  • At startup, non-lazy models beyond the limit are registered but
    deferred to first use (with a log note), instead of churning through
    loads that would be immediately evicted.

Also in this PR

LoadedModel::unload() never cleared the loaded flag, so /v1/models
kept reporting "loaded": true after /v1/tasks/unload_models and
/v1/tasks/unload_all_models. Eviction reuses unload(), so this fixes
the flag there and folds the duplicated inline teardown blocks in
handle_model_load (reconfigure path) and handle_model_unload into it.

Docs and tests

  • Documented in app/server/README.md and the root README.md
    (including updating the "never unloads" warnings), plus --help.
  • New config tests: default is 0, values 1 and N parse, negative values
    are rejected (top-level and CLI validation mirror busy_timeout_ms).

Validation

  • CPU-only build compiles clean; server_config_test and
    server_busy_guard_test pass.
  • Backward compatible: with max_loaded_models omitted, behavior is
    unchanged and no new locking is engaged on the load path.

The server previously kept every model resident in memory forever once
loaded, with no way to enforce a residency bound. Add a native
max_loaded_models setting (server.json key or --max-loaded-models CLI
flag): 0 keeps the old unlimited behavior (default), 1 enforces a single
loaded model at a time, and N keeps the N most recently used models warm.

Enforcement lives in ensure_model_loaded_locked(), the single funnel all
loads go through (startup eager loads, lazy first-request loads, and
WebUI-driven loads). Before loading past the limit, the least recently
used idle model is unloaded first, freeing VRAM on GPU backends; its next
request reloads it. A model mid-inference is never evicted: victim busy
guards are only try-acquired (a blocking wait could deadlock two loads
evicting each other's target), and when every loaded model is busy the
request fails fast with the existing 503 busy path so clients can retry.
Loads are serialized while the limit is active so concurrent lazy loads
cannot overshoot it.

At startup, non-lazy models beyond the limit are registered but deferred
to first use instead of churning through loads that would be immediately
evicted.

Also fold the duplicated inline teardown blocks into LoadedModel::unload()
and make it clear the loaded flag, fixing /v1/models reporting stale
"loaded": true after /v1/tasks/unload_models and unload_all_models.
@0xShug0
0xShug0 merged commit 84453c0 into 0xShug0:main Aug 22, 2026
6 checks passed
@0xShug0

0xShug0 commented Aug 22, 2026

Copy link
Copy Markdown
Owner

@SelfRef Merged. Thanks!

NairoDorian added a commit to NairoDorian/speech.cpp that referenced this pull request Aug 22, 2026
Brings speech.cpp up to date with latest audio.cpp upstream main (4d383be):
- Community models: MOSS-VoiceGenerator (PR 0xShug0#278), MMS-300M-1130 forced aligner (PR 0xShug0#279), F5-TTS (PR 0xShug0#275).
- SenseASR encoder refactored to framework SAN-M modules (PR 0xShug0#285).
- Server: max_loaded_models limit with LRU eviction (PR 0xShug0#298) and opt-in session options listing.
- WebUI: reverse proxy hash routing (PR 0xShug0#297), Music3 precision packages, HeartMuLa options.
- CUDA & Memory: CUDA graph-cache eviction and idle pool trimming (PR 0xShug0#293), Supertonic vector arena reduction.
- GGML: tracked CUDA clear_graph and trim_pools as patch 0007.
- Build & CI: native model manager build flags, C++17 cleanups, CMake model-link guards preserved.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants