Vision/multimodal silently dropped with use_tokenizer_template on the python backends (sglang, vllm): images reach the engine but the prompt carries no media placeholder

[pos-ei-don]

LocalAI version

v4.7.1, image localai/localai:v4.7.1-nvidia-l4t-arm64-cuda-13.
Backends cuda13-nvidia-l4t-arm64-sglang (sglang 0.5.17) and cuda13-nvidia-l4t-arm64-vllm (vLLM 0.24.0), both from the gallery.

Environment, CPU architecture, OS, and Version

NVIDIA DGX Spark (GB10, Blackwell sm_121), arm64, Ubuntu 24.04, CUDA 13, driver 595.71.05, Docker with the NVIDIA runtime.

Describe the bug

For a model configured with template.use_tokenizer_template: true, an image sent as an image_url content part is silently ignored. No error, no warning — the model simply answers as if no image had been attached. The same model, image and prompt work correctly when the engine is driven directly (standalone sglang.launch_server and its own /v1/chat/completions), so it is neither the model nor the engine.

The images do reach the backend. What does not reach it is the media placeholder in the prompt:

1. core/http/middleware/request.go decodes the image_url parts into Messages[i].StringImages, and then — for UseTokenizerTemplate — deliberately writes only the text back into StringContent:

// When the backend handles templating itself (UseTokenizerTemplate),
// it also injects media markers server-side (see
// oaicompat_chat_params_parse in llama.cpp). ...
if config.TemplateConfig.UseTokenizerTemplate {
    input.Messages[i].StringContent = textContent
} else {
    input.Messages[i].StringContent, _ = templates.TemplateMultiModal(...)
}

That assumption holds for llama.cpp's server, which injects the markers itself. It does not hold for the python backends: they call tokenizer.apply_chat_template() on plain string content, and a chat template only emits vision tokens when the content is a list of parts.

2. core/schema/message.go (Messages.ToProto()) then drops the image parts entirely and keeps only the concatenated .text.

3. message Message in backend.proto has no media field at all, so images can only travel out-of-band in the global PredictOptions.Images — the image↔message association is lost on the wire.

4. In backend/python/sglang/backend.py, _messages_to_dicts() builds {"role": …, "content": msg.content or ""} and _build_prompt() renders that through apply_chat_template(). For a Qwen3.5-VL model the rendered prompt therefore contains no <|vision_start|><|image_pad|><|vision_end|>.

5. The images themselves are forwarded correctly — backend.py does image_data = list(request.Images) → llm.async_generate(..., image_data=image_data). But sglang's multimodal processor locates images by scanning the prompt for the model's image token (sglang/srt/multimodal/processors/qwen_vl.py: image_token="<|vision_start|><|image_pad|><|vision_end|>" plus the matching regex). With no placeholder present nothing is split out and image_data is discarded without a message.

backend/python/vllm/backend.py has the identical gap: its _messages_to_dicts() is the same string-content version and apply_chat_template() is applied the same way. Its load_image() / multi_modal_data / LimitImagePerPrompt machinery only carries the pixels, not the placeholder.

Rendering the model's own chat template confirms the mechanism directly (Qwen3.5-VL template, jinja2, no model load):

message content	rendered user turn
"Wie hoch steht das Wasser?" (what the backend builds today)	<|im_start|>user\nWie hoch steht das Wasser?<|im_end|> — no placeholder
[{"type":"image"},{"type":"text","text":"Wie hoch steht das Wasser?"}]	<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Wie hoch steht das Wasser?<|im_end|>

To Reproduce

1. Serve any VLM through the sglang backend with the tokenizer template:

name: vlm
backend: cuda13-nvidia-l4t-arm64-sglang
parameters:
  model: <a Qwen3.5-VL-family checkpoint>
template:
  use_tokenizer_template: true
engine_args:
  model_path: /models/<checkpoint>
  trust_remote_code: true

2. POST /v1/chat/completions with an image_url content part (data URI) and a question about the image.
3. → the model replies that no image was attached. HTTP 200, nothing in the log.
4. Control: send the same request to a standalone python3 -m sglang.launch_server with the same checkpoint → correct answer about the image.

Expected behavior

The image is coupled to the prompt and the model sees it — with use_tokenizer_template: true, on the sglang and vllm backends, the same way it already works on llama-cpp.

Additional context

Two ways to fix it, and they are not mutually exclusive:

(a) Backend-local, small, no protocol change. In _build_prompt(), rebuild the OpenAI content parts for the last user message from request.Images / request.Videos before templating, so the chat template emits the model's own placeholders. The pixels keep travelling via image_data / multi_modal_data:

n_img = len(request.Images) if request.Images else 0
n_vid = len(request.Videos) if request.Videos else 0
if n_img or n_vid:
    idx = next((i for i in range(len(messages_dicts) - 1, -1, -1)
                if messages_dicts[i].get("role") == "user"), None)
    if idx is not None:
        text = messages_dicts[idx].get("content") or ""
        parts = [{"type": "image"}] * n_img + [{"type": "video"}] * n_vid
        if text:
            parts.append({"type": "text", "text": text})
        messages_dicts[idx]["content"] = parts

The existing except TypeError around apply_chat_template() needs widening to except Exception so that a text-only template falls back to string content instead of failing the request. Text-only requests are unaffected — with no images the whole path is a no-op. The same patch applies verbatim to the vllm backend. This covers every single-image and last-turn request, which is effectively all real vision traffic.

(b) Protocol-level, complete. Add repeated string images (and videos/audios) to message Message in backend.proto, stop discarding the parts in Messages.ToProto(), and let the backends read them per message. This is the only way to get multi-turn conversations with images in different turns right, and it fixes every python backend at once.

Happy to send a PR for (a) — that is the change we are running locally.

Related: #10945 (same class of failure — marker↔bitmap coupling — but on the llama-cpp/mtmd path).

[pos-ei-don]

Reporter here. Two things that may save you time, @owezzy:

There is already an open PR for exactly this: #11738 takes the backend-local route — a shared attach_media_parts() in backend/python/common/python_utils.py, wired into both sglang/backend.py and vllm/backend.py, with tests. Worth coordinating with @Anai-Guo before starting from scratch.

I verified that patch against the original repro. Cherry-picked onto a vLLM 0.28.0 backend (Ornith 1.5 35B-A3B FP8, DGX Spark GB10, same setup as the report):

• before: 0/3 on a three-image colour probe; the rendered prompt contains no <|vision_start|><|image_pad|><|vision_end|> while multi_modal_data carries the image
• after: 3/3, correct on all three

One update to the original report: the symptom depends on the vLLM version. On 0.24 the image was silently dropped (HTTP 200, model answers as if nothing was attached) — that is what the issue describes. On 0.28 the same request fails hard instead:

HTTP 500  rpc error: code = Unknown desc = Unexpected
          <class 'vllm.v1.engine.exceptions.EngineGenerateError'>

Same root cause, louder failure. Worth knowing when reproducing, because "no error in the log" no longer holds on current vLLM.

On backend-local vs protocol-level: backend-local is the right first step and fixes the common case, but it cannot close everything under point 3 of the report. message Message in backend.proto still has no media field, so images travel out-of-band in the global PredictOptions.Images and the image↔message association is lost on the wire. attach_media_parts() consequently attaches all media to the last user turn — correct for single-turn requests, wrong for a conversation whose earlier turns carried images. That gap needs the protocol change; it just should not block the fix that unblocks the common case today.