Skip to content

Commit dbededa

Browse files
author
Anai Guo
committed
fix(llama): release a vision chat handler's mtmd context on close (#2342)
`Llama.close()` tore down the model and its context but left the chat handler's `_exit_stack` untouched, so the mtmd/clip context it built from that model was never freed. Handlers routinely outlive the `Llama` that initialized them -- callers construct one handler and reuse it across loads -- and `_init_mtmd_context()` returns early while `mtmd_ctx` is set. After the first `close()` the handler therefore kept a context bound to an already-freed model and handed it back on the next load, where it surfaces as a null `mtmd_ctx` on the C++ side. Register the handler's exit stack on the Llama's `_stack`. It unwinds LIFO, so the mtmd context is released ahead of the model teardown registered earlier in `__init__`, and `mtmd_free` resets `mtmd_ctx` to `None` so the next load re-initializes cleanly. Duck-typed on `_exit_stack`, which both `Llava15ChatHandler` and `MTMDChatHandler` (and every subclass) expose.
1 parent 3bda091 commit dbededa

1 file changed

Lines changed: 10 additions & 0 deletions

File tree

llama_cpp/llama.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -456,6 +456,16 @@ def free_lora_adapter():
456456

457457
self.chat_format = chat_format
458458
self.chat_handler = chat_handler
459+
# A vision chat handler builds its mtmd/clip context from this model, so
460+
# that context must be released before the model itself is freed. The
461+
# handler object can outlive the Llama (callers commonly reuse a single
462+
# handler across loads), and it skips re-initialization while mtmd_ctx
463+
# is set -- leaving a context bound to an already-freed model behind.
464+
# `_stack` unwinds LIFO, so this runs before the model teardown that was
465+
# registered earlier in __init__.
466+
handler_stack = getattr(chat_handler, "_exit_stack", None)
467+
if handler_stack is not None:
468+
self._stack.callback(handler_stack.close)
459469
self._chat_handlers: Dict[
460470
str, llama_chat_format.LlamaChatCompletionHandler
461471
] = {}

0 commit comments

Comments
 (0)