fix(providers): activate vision sidecar for NVIDIA NIM text-only models - #964
fix(providers): activate vision sidecar for NVIDIA NIM text-only models#964Yuxin-Qiao wants to merge 1 commit into
Conversation
Add noVisionModels to the nvidia registry entry (issue lidge-jun#956). The NIM /v1/models catalog carries no input-modality metadata, so the registry is the only source of truth: without the list, the vision sidecar never runs and text-only NIM models either 400 on raw image parts upstream or are advertised text-only so the app blocks attachments client-side. Covers the text-only chat/code families from the live catalog (2026-08-04) plus the documented kimi family. Vision-capable NIM models (llama-3.2-*- vision-instruct, phi-3-vision, neva-22b, vila, nemotron-*-vl, ...) and non-chat endpoints stay out. Regression tests: registry list contents/exclusions, bare-config router inheritance, planVisionSidecar activation, and catalog inputModalities advertisement.
📝 WalkthroughWalkthroughThe NVIDIA NIM registry now classifies text-only models through ChangesNVIDIA NIM vision routing
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Request
participant NVIDIA_NIM_Registry
participant Vision_Sidecar_Planner
participant Catalog
Request->>Vision_Sidecar_Planner: Submit parsed request
NVIDIA_NIM_Registry->>Vision_Sidecar_Planner: Provide noVisionModels
Vision_Sidecar_Planner->>Vision_Sidecar_Planner: Plan sidecar when images target text-only models
NVIDIA_NIM_Registry->>Catalog: Provide model capability metadata
Catalog-->>Request: Advertise image input for text-only routed models
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/providers/registry.ts`:
- Line 555: Remove minimaxai/minimax-m3, moonshotai/kimi-k2.6, and
thinkingmachines/inkling from NVIDIA_NIM_NO_VISION_MODELS in the provider
registry. Update the corresponding nvidia-nim-hardening tests so all three model
IDs are asserted as excluded from the no-vision classification and therefore
retain native image handling.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 4f6170b5-d339-4ff4-abf8-e683f8a62dac
📒 Files selected for processing (2)
src/providers/registry.tstests/nvidia-nim-hardening.test.ts
| "meta/llama-3.3-70b-instruct", | ||
| "meta/llama2-70b", | ||
| "microsoft/phi-3.5-moe-instruct", | ||
| "minimaxai/minimax-m3", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Remove native vision models from NVIDIA_NIM_NO_VISION_MODELS.
minimaxai/minimax-m3, moonshotai/kimi-k2.6, and thinkingmachines/inkling accept image input. Their current classification activates planVisionSidecar and routes image requests through the sidecar instead of the model’s native vision path. It also contradicts the stated exclusion rule for vision-capable models. NVIDIA documents image input for all three models. (build.nvidia.com)
Remove these IDs. Update tests/nvidia-nim-hardening.test.ts so MiniMax M3, Kimi K2.6, and Inkling are asserted as exclusions.
Proposed fix
- "minimaxai/minimax-m3",
...
- "moonshotai/kimi-k2.6",
...
- "thinkingmachines/inkling",Also applies to: 563-563, 588-588
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/providers/registry.ts` at line 555, Remove minimaxai/minimax-m3,
moonshotai/kimi-k2.6, and thinkingmachines/inkling from
NVIDIA_NIM_NO_VISION_MODELS in the provider registry. Update the corresponding
nvidia-nim-hardening tests so all three model IDs are asserted as excluded from
the no-vision classification and therefore retain native image handling.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 220b4f728a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "openai/gpt-oss-20b", | ||
| "poolside/laguna-xs-2.1", | ||
| "stepfun-ai/step-3.7-flash", | ||
| "thinkingmachines/inkling", |
There was a problem hiding this comment.
When an NVIDIA user sends an image to thinkingmachines/inkling, this entry makes planVisionSidecar replace the native image with a lossy description—or strip it entirely when no sidecar is available—even though the same exact model ID is already recorded as text,image in BASE10_MODEL_INPUT_MODALITIES at src/providers/registry.ts:638-643. Remove image-capable entries such as Inkling from this list and add a regression that verifies their images remain untouched.
Useful? React with 👍 / 👎.
| // Free pricing, but an API key is still required (free key from build.nvidia.com). | ||
| freeTier: true, | ||
| parallelToolCalls: false, | ||
| noVisionModels: NVIDIA_NIM_NO_VISION_MODELS, |
There was a problem hiding this comment.
This registry flag changes visible NVIDIA behavior: affected models now advertise image input and may consume a configured OpenAI or Anthropic vision sidecar before every image-bearing request, but the NVIDIA provider documentation remains only a base-URL table entry. Add an NVIDIA-specific note identifying the classified model families and the sidecar/auth implications so users are not surprised by the extra upstream call.
AGENTS.md reference: src/AGENTS.md:L28-L28
Useful? React with 👍 / 👎.
| "nvidia/nemotron-4-340b-instruct", | ||
| "nvidia/nemotron-mini-4b-instruct", | ||
| "nvidia/nemotron-nano-3-30b-a3b", | ||
| "nvidia/nvidia-nemotron-nano-9b-v2", |
There was a problem hiding this comment.
For the live text-only model nvidia/nemotron-nano-9b-v2, this entry never matches because it contains an extra nvidia- segment. modelInList performs exact matching for slash-namespaced IDs, so image requests to this model continue bypassing the sidecar and reach the text-only upstream unchanged; correct the ID and add it to the registry coverage test.
Useful? React with 👍 / 👎.
|
Please put your Pull-Request on Ready for Review, once you are finished. |
|
Confirmed as a real defect and left open for its own review rather than folded into the current review stack (#951–#973). The reasoning: this is a provider-registry change — Worth flagging for whoever reviews it: the registry change is small, but the blast radius is every NIM text-only model listed in the PR, so the useful evidence is a live NIM request with an image part before and after. Fixes #956. |
|
Superseded by #980 (stack 7/7), commit Your finding is correct and it is now fixed. The What I changed, and why I owe you the detail. Six ids in the submitted list are natively image-capable per NVIDIA's own documentation:
Listing a vision-capable model there is a silent defect: the model could read the image, but the proxy substitutes another model's text description instead. No error, worse answers, extra cost, and no test fails. This is not a lapse on your part. Issue #956's own body lists minimax-m3 and kimi-k2.6 as text-only, so the reporter and the PR shared the same premise, and review passed it through. It is what an unaudited list does over an open domain. So every id was verified individually against NVIDIA per-model documentation ( The 16 vision-capable ids also needed explicit Where I did worse than you. My first two replacement designs were falsified at the audit gate. I proposed deriving text-only as the complement of a vision list (an unclassified id still ends up in neither list, so nothing changes), then a provider-level default-on rule (which would have advertised image input for every embedding, reranker, guard and OCR endpoint NIM serves, since What still is not fixed: an id NVIDIA ships after this snapshot stays unclassified. NIM publishes no modality metadata, so an unknown id carries no signal at all. #956 closes with that bounded scope stated rather than implied. Thank you for the report — the underlying bug was real and is fixed for every model named in the issue. |
3 participants
Summary
Fixes #956. The built-in
nvidiaprovider entry insrc/providers/registry.tsdid not declarenoVisionModels, so the vision sidecar never activated for NVIDIA NIM text-only models (deepseek-ai/deepseek-v4-flash,z-ai/glm-5.2,minimaxai/minimax-m3,moonshotai/kimi-k2.6,nvidia/nemotron-3-ultra-550b-a55b, ...). Raw image parts were forwarded to a text-only upstream (400 / silently ignored) or the catalog advertised the model as text-only, making the Codex app block attachments client-side.NIM's
/v1/modelscarries no input-modality metadata, so the registry is the only source of truth for capability classification.Changes
src/providers/registry.ts: addNVIDIA_NIM_NO_VISION_MODELSand wire it into thenvidiaentry asnoVisionModels.meta/llama-3.2-11b/90b-vision-instruct,microsoft/phi-3-vision-128k-instruct,adept/fuyu-8b,google/deplot,microsoft/kosmos-2,nvidia/llama-3.1-nemotron-nano-vl-8b-v1,nvidia/nemotron-nano-12b-v2-vl,nvidia/neva-22b,nvidia/vila,nvidia/nemotron-3-nano-omni-30b-a3b-reasoning(multimodal), plus non-chat endpoints (embeddings, guards, translators, detectors, image/video generators).modelInListsemantics: exact-id matching for slash-namespaced NIM ids; new text-only ids must be appended individually (same convention as the existing NIM kimi lists).With the list in place,
planVisionSidecarreturns a plan for image-carrying requests to these models and the catalog advertisesinputModalitiesincluding"image"(so the app lets attachments reach the sidecar).Tests
tests/nvidia-nim-hardening.test.ts: 4 new regression testsnoVisionModelsvia the router mergeplanVisionSidecarplans for a text-only NIM model with an image, staysundefinedwithout an image, and staysundefinedfor a vision-capable NIM modelVerification
bun test tests/nvidia-nim-hardening.test.ts→ 18 pass / 0 failbun test tests/provider-registry-parity.test.ts tests/vision-anthropic.test.ts tests/claude-sidecar-override.test.ts tests/codex-catalog.test.ts tests/nvidia-nim-hardening.test.ts→ 176 pass / 0 failbun x tsc --noEmit→ cleandevin the same sandbox — the failures are identical and environmental (sandbox blocks outbound network and socket binds; e.g. OAuth replay,/v1/liverelay, andEADDRINUSEon ephemeral ports), so this change introduces no regressions.Notes
The NIM catalog is dynamic; new text-only ids added upstream must be appended to
NVIDIA_NIM_NO_VISION_MODELS(noted in a registry comment). Vision-capable NIM models keep their previous behavior.Summary by CodeRabbit
New Features
Bug Fixes
Tests