Skip to content

fix(catalog): preserve provider capabilities on custom models #963

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MarcTCruz wants to merge 1 commit into from
+70 −49
Closed

fix(catalog): preserve provider capabilities on custom models #963

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions src/codex/catalog/provider-fetch.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -780,26 +780,26 @@ async function gatherRoutedModelsUncached(
replaceLastComboCatalogOmissions(localOmissions);
all.sort((a, b) => (a.provider === b.provider ? a.id.localeCompare(b.id) : a.provider.localeCompare(b.provider)));
// Enriched (registry-hydrated) provider clones, keyed by name — the same view used above so
// custom rows get the same noVisionModels / inputModalities treatment as discovered rows.
// custom rows get the same capability hints as discovered rows.
const enrichedByName = new Map(activeProviders);
const customModels = (config.customModels ?? []).map(cm => {
const rawProvider = config.providers[cm.provider];
const supportsReasoningSummaries = configuredReasoningSummarySupport(rawProvider, cm.modelId);
const enrichedProvider = enrichedByName.get(cm.provider) ?? rawProvider;
const providerCap = cm.contextWindow === undefined ? providerContextCap(config, cm.provider) : undefined;
const hints = enrichedProvider
? catalogHintsFromProviderConfig(cm.provider, enrichedProvider, cm.modelId, providerCap)
: {};
const base: CatalogModel = {
...hints,
id: cm.modelId,
provider: cm.provider,
// Display-only label: never feeds routing (customModels are keyed by routedSlug below).
...(cm.displayName ? { displayName: cm.displayName } : {}),
...(cm.contextWindow ? { contextWindow: cm.contextWindow } : {}),
...(cm.inputModalities ? { inputModalities: cm.inputModalities } : {}),
...(typeof supportsReasoningSummaries === "boolean" ? { supportsReasoningSummaries } : {}),
};
// Vision-sidecar coverage ONLY: if the custom model is in the enriched provider's
// noVisionModels, advertise image input so the Codex app lets images reach the sidecar
// (#349/#344). Deliberately NOT the full applyProviderConfigHints pass — custom rows are a
// user override, so their explicit contextWindow / inputModalities / reasoning fields must be
// preserved verbatim (the hint pass would cap context and overwrite modalities from registry).
const enrichedProvider = enrichedByName.get(cm.provider) ?? rawProvider;
// Explicit custom modalities win over hints, but noVisionModels still needs to append image
// so the Codex app permits attachments to reach the proxy's vision sidecar.
if (enrichedProvider && modelInList(enrichedProvider.noVisionModels, base.id)) {
const current = base.inputModalities ?? ["text"];
if (!current.includes("image")) {
Expand Down
42 changes: 42 additions & 0 deletions tests/catalog-custom-model-metadata.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
import { expect, test } from "bun:test";
import { buildCatalogEntries, gatherRoutedModels } from "../src/codex/catalog";

test("custom rows retain declared metadata while inheriting provider capability hints", async () => {
const models = await gatherRoutedModels({
port: 10100,
defaultProvider: "ollama",
providers: {
ollama: {
adapter: "openai-chat",
baseUrl: "http://127.0.0.1:11434/v1",
liveModels: false,
models: ["qwen-coder-3b"],
modelContextWindows: { "qwen-coder-3b": 32_768 },
modelInputModalities: { "qwen-coder-3b": ["text", "image"] },
modelReasoningEfforts: { "qwen-coder-3b": [] },
},
Comment on lines +16 to +17

@coderabbitai coderabbitai Bot Aug 3, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Cover the noReasoningModels regression path.

Line 16 sets modelReasoningEfforts to [], but this fixture does not declare noReasoningModels. A regression that stops inheriting noReasoningModels metadata can still pass.

Add noReasoningModels: ["qwen-coder-3b"] to this provider fixture. Keep the assertions for no supported reasoning levels and no default reasoning level.

Based on PR objectives: models in noReasoningModels with an empty effort list must expose neither generic reasoning levels nor a default reasoning level.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/catalog-custom-model-metadata.test.ts` around lines 16 - 17, Add
noReasoningModels containing "qwen-coder-3b" to the provider fixture alongside
modelReasoningEfforts in the relevant test setup. Preserve the existing
assertions that this model exposes no supported reasoning levels and no default
reasoning level, ensuring the regression path verifies inherited
noReasoningModels metadata.

},
customModels: [{
id: "qwen-coder-3b-custom",
provider: "ollama",
modelId: "qwen-coder-3b",
displayName: "Qwen Coder 3B",
contextWindow: 65_536,
inputModalities: ["text"],
addedAt: "2026-08-03T00:00:00.000Z",
}],
});

const custom = models.filter(model => model.provider === "ollama" && model.id === "qwen-coder-3b");
expect(custom).toHaveLength(1);
expect(custom[0]).toMatchObject({
displayName: "Qwen Coder 3B",
contextWindow: 65_536,
inputModalities: ["text"],
reasoningEfforts: [],
});

const entry = buildCatalogEntries(null, [], models).find(candidate => candidate.slug === "ollama/qwen-coder-3b");
expect(entry?.supported_reasoning_levels).toEqual([]);
expect(entry).not.toHaveProperty("default_reasoning_level");
});
59 changes: 19 additions & 40 deletions tests/catalog-vision-sidecar-modalities.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,7 @@ describe("vision-sidecar custom-model override (#349/#344)", () => {
// vision sidecar could run. The image augmentation must come from the REGISTRY-enriched clone,
// so we deliberately do NOT set noVisionModels on the persisted provider — opencode-go's
// registry entry classifies glm-5.2 as text-only, and enrichment must supply it.
const originalFetch = globalThis.fetch;
globalThis.fetch = (() => { throw new Error("fetch should not be called"); }) as typeof fetch;
try {
const models = await gatherRoutedModels({
const models = await gatherRoutedModels({
port: 10100,
defaultProvider: "opencode-go",
providers: {
Expand All @@ -74,20 +71,14 @@ describe("vision-sidecar custom-model override (#349/#344)", () => {
customModels: [
{ id: "cm-1", provider: "opencode-go", modelId: "glm-5.2", displayName: "GLM 5.2", addedAt: "2026-01-01T00:00:00.000Z" },
],
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "glm-5.2");
expect(custom).toBeDefined();
expect(custom?.inputModalities).toEqual(["text", "image"]);
} finally {
globalThis.fetch = originalFetch;
}
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "glm-5.2");
expect(custom).toBeDefined();
expect(custom?.inputModalities).toEqual(["text", "image"]);
});

test("a custom row NOT in noVisionModels keeps its declared modalities untouched", async () => {
const originalFetch = globalThis.fetch;
globalThis.fetch = (() => { throw new Error("fetch should not be called"); }) as typeof fetch;
try {
const models = await gatherRoutedModels({
const models = await gatherRoutedModels({
port: 10100,
defaultProvider: "opencode-go",
providers: {
Expand All @@ -103,22 +94,13 @@ describe("vision-sidecar custom-model override (#349/#344)", () => {
customModels: [
{ id: "cm-2", provider: "opencode-go", modelId: "kimi-text", inputModalities: ["text"], addedAt: "2026-01-01T00:00:00.000Z" },
],
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "kimi-text");
expect(custom?.inputModalities).toEqual(["text"]);
} finally {
globalThis.fetch = originalFetch;
}
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "kimi-text");
expect(custom?.inputModalities).toEqual(["text"]);
});

test("the image augmentation does NOT overwrite a custom row's explicit context/modalities/reasoning", async () => {
// The augmentation must be narrow: for a noVisionModels custom row we only ADD image; every
// other explicitly configured custom field (contextWindow, extra modalities) stays verbatim,
// and no registry reasoning metadata leaks onto the user override.
const originalFetch = globalThis.fetch;
globalThis.fetch = (() => { throw new Error("fetch should not be called"); }) as typeof fetch;
try {
const models = await gatherRoutedModels({
test("the image augmentation preserves explicit custom fields and canonical provider reasoning", async () => {
const models = await gatherRoutedModels({
port: 10100,
defaultProvider: "opencode-go",
providers: {
Expand All @@ -133,17 +115,14 @@ describe("vision-sidecar custom-model override (#349/#344)", () => {
customModels: [
{ id: "cm-3", provider: "opencode-go", modelId: "glm-5.2", contextWindow: 2_000_000, inputModalities: ["text", "video"], addedAt: "2026-01-01T00:00:00.000Z" },
],
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "glm-5.2");
// image is appended to the user's declared modalities (not replaced), context is preserved,
// and no registry reasoning fields were injected onto the custom override.
expect(custom?.inputModalities).toEqual(["text", "video", "image"]);
expect(custom?.contextWindow).toBe(2_000_000);
expect(custom?.reasoningEfforts).toBeUndefined();
expect(custom?.defaultReasoningEffort).toBeUndefined();
} finally {
globalThis.fetch = originalFetch;
}
});
const custom = models.find(m => m.provider === "opencode-go" && m.id === "glm-5.2");
// image is appended to the user's declared modalities (not replaced), custom context is
// preserved, and the registry-enriched provider supplies its canonical reasoning metadata.
expect(custom?.inputModalities).toEqual(["text", "video", "image"]);
expect(custom?.contextWindow).toBe(2_000_000);
expect(custom?.reasoningEfforts).toEqual(["low", "medium", "high", "xhigh", "max"]);
expect(custom?.defaultReasoningEffort).toBeUndefined();
});
});

Expand Down
Loading