Expose ONNX Runtime memory management options (arena settings, GPU memory limits) through the CLI and SDK [SDK] & [CLI]

[FALLEN-01]

I'm running Foundry Local as part of a local-first RAG application (Python SDK, also tested via CLI). Over extended runtime, the loaded model's memory usage grows and eventually causes inference failures — consistent with known ONNX Runtime memory arena behavior (e.g. enable_cpu_mem_arena pre-allocation, GPU arena growth not being fully released between runs; see microsoft/onnxruntime#11627 and microsoft/onnxruntime#19445).
Currently, neither the CLI nor the SDK exposes any way to tune this. The only available mitigation is:

SDK: calling model.unload() followed by model.load() to fully reset the session
CLI: attempting foundry service restart, which in my case fails outright and isn't a reliable substitute

Both are blunt workarounds that discard the whole session/service rather than tuning the underlying memory behavior.

SDK: Expose a way to pass ONNX Runtime SessionOptions and execution-provider options (e.g. enable_cpu_mem_arena, enable_mem_pattern, gpu_mem_limit, arena_extend_strategy) through the Configuration object or model.load().Even a minimal passthrough — e.g. Configuration(session_options_overrides={...}) — would help.

CLI: Expose equivalent flags on foundry model load (and/or foundry service start)

CLI: Expose equivalent flags on foundry model load (and/or foundry service start

Manually unloading/reloading the model (SDK) or restarting the service (CLI) on a request counter or timer — functional, but doesn't address root cause, adds latency spikes on recycle, and in our case foundry service restart fails and isn't usable at all.
Bypassing Foundry Local entirely and loading the ONNX model directly via onnxruntime.InferenceSession — loses Foundry Local's hardware auto-detection, execution provider management, and lifecycle handling, which is the main reason we adopted it in the first place.

[samuel100]

Thanks @FALLEN-01 for the feedback!. After tracing Foundry v2 → onnxruntime-genai → ORT, here's a proposal we'd love feedback on.

There are two tiers, both reachable today through onnxruntime-genai; Foundry just doesn't surface either.

Tier 1 — Load-time options (immutable; retune = reload). session_options (arena strategy, enable_cpu_mem_arena, enable_mem_pattern, thread counts) and per-EP provider_options (gpu_mem_limit). These are consumed when the model/session is created, so changing them requires unload+reload:

import foundry_local_sdk as fl

manager = fl.FoundryLocalManager("my-rag-app")
model = manager.catalog.get_model("phi-4-mini")

model.load(
    session_options={                       # ORT session-level (global/arena)
        "enable_cpu_mem_arena": "0",
        "arena_extend_strategy": "kSameAsRequested",
    },
    provider_options={                      # execution-provider-specific
        "cuda": {"gpu_mem_limit": "2147483648"},
    },
)

Tier 2 — Runtime options (per-run; no reload). ORT exposes memory.enable_memory_arena_shrinkage as a RunOption, and onnxruntime-genai forwards run options to every Run(). This reclaims arena memory after runs — directly targeting the "memory grows over extended runtime" case without recycle latency spikes. Because the arena is model-scoped, the same call applies whether you subsequently use chat, embeddings, or audio:

# Modality-agnostic — memory is a property of the loaded model, not the session type
model.set_runtime_option("memory.enable_memory_arena_shrinkage", "cpu:0;gpu:0")
# ... run inference (chat / embeddings / audio) ...
model.set_runtime_option("memory.enable_memory_arena_shrinkage", "")   # disable once settled

Feedback we'd like:

1. Runtime surface shape: flat model.set_runtime_option(...) vs. a generic handle model.runtime / model.get_runtime() that groups memory controls (shrink_arena(), etc.). We're deliberately avoiding a per-modality getter (e.g. get_chat_session()), since memory management is model-scoped and identical across modalities. Preference?
2. Is Tier 2 alone enough for you? Periodic arena shrinkage may address the root cause without any reload — arguably a better fix than the load-time overrides. Does that fit how your RAG service manages memory, or do you also need the Tier 1 load-time overrides?
3. Shrinkage semantics: it attaches to subsequent runs, so in a token-generation loop it fires per token unless toggled. We're considering a one-shot "shrink after next generation" helper. Useful?
4. String passthrough + open forwarding: all options cross our C ABI as string key/value pairs (same pattern as today's additional_settings), forwarded to ORT/GenAI and parsed there — tiny surface, inherited for free by the Python/C#/C++ SDKs. We'd forward any key (with logging) rather than gate on an allowlist. Objections?

CLI equivalents would mirror this: foundry model load phi-4-mini --session-option enable_cpu_mem_arena=0 --provider-option cuda.gpu_mem_limit=2147483648, plus a runtime foundry model set-runtime-option <model> memory.enable_memory_arena_shrinkage=cpu:0;gpu:0.

[FALLEN-01]

Thanks for digging into this so quickly, and for the detailed breakdown — really appreciate it.

1. API shape: Prefer the grouped handle (model.runtime / model.get_runtime()) over flat string calls. Memory management feels like a distinct concern from chat/embed/audio logic, so a dedicated object with named methods (shrink_arena(), etc.) is more discoverable and less error-prone than remembering exact ORT key strings.

2. Is Tier 2 alone enough? Yes, for our use case Tier 2 alone would fully replace our current workaround (manual unload/reload on a request counter). Periodic arena shrinkage gives us the same practical outcome — bounded memory growth — without the reload latency spike (re-init cost, cold start on the next request). We don't have a strong need for Tier 1 load-time overrides beyond nice-to-have; Tier 2 solves the actual pain point.

3. One-shot shrinkage: Yes please — a shrink_after_next_run() (or similar) would map directly onto our pattern: we want to shrink once every N requests, not on every single generation/token. Continuous shrinkage would likely add unnecessary overhead for our case.

4. Open passthrough vs. allowlist: No objection to open passthrough with logging. An allowlist just becomes a queue of future feature requests every time someone needs a new ORT/GenAI key — logging gives you visibility without gatekeeping.

One addition worth flagging: our app ships to end-user devices with mixed hardware — some run CPU-only, others have a discrete GPU — so we need both paths covered equally, not just GPU as the primary case:

• On Tier 1, we'd want session_options to explicitly cover enable_cpu_mem_arena (arena on/off) and arena_extend_strategy (kSameAsRequested vs NextPowerOfTwo) for the CPU path, alongside provider_options.cuda.gpu_mem_limit for the GPU path — the doubling behavior under NextPowerOfTwo is the main driver of unbounded RAM growth we've seen on CPU-only devices.
• On Tier 2, your own example already includes both targets in memory.enable_memory_arena_shrinkage("cpu:0;gpu:0") — glad to see it's modality- and hardware-agnostic out of the box, since our embedding + generation workloads land on whichever backend is available per device.
• If there's ever a CPU-equivalent to a hard gpu_mem_limit (i.e. OrtArenaCfg.max_mem for a real ceiling rather than just extend-strategy tuning), that would be valuable too — right now the closest CPU lever is disabling the arena outright, which is a blunter tool than an actual cap.

CLI mirroring the SDK shape sounds good too — having --session-option / --provider-option / set-runtime-option match the Python surface means we can prototype on CLI and port straight to SDK without relearning key names.

Happy to test an early build against our RAG pipeline on both CPU-only and GPU-equipped hardware if that's useful for validating shrinkage behavior across configurations.