Skip to content

Commit af9a438

Browse files
committed
fix(embeddings): restore clobbered billing test and harden the Ollama path
Second round of review findings: - Restore apps/sim/lib/knowledge/embeddings.test.ts, whose recordSearchEmbeddingUsage suite a whole-file write had destroyed; the new configuration tests move to a sibling file so the original's vi.spyOn wiring stays intact - Refuse to create a knowledge base when an Ollama model's width cannot be established, rather than pinning it at 1536 where every document would fail - Resolve an Ollama model through /api/show for that model alone instead of scanning the catalog, so one unrelated probe cannot delay or break an embedding, and a 404 stays a 400 while anything else is a 502 - Classify the raw KB_EMBEDDING_MODEL value, since the runtime does not trim it - Clear apiKey and openRouterApiKey explicitly when switching a block to Ollama - Offer the OpenAI-family embedding model as a choice, so switching families in the wizard cannot carry a Gemini or Ollama model forward - Correct the docs on Ollama dimension support, run charges, the OpenRouter knowledge-base fallback, and what Sim can determine about a local model's width
1 parent 97668cf commit af9a438

11 files changed

Lines changed: 369 additions & 161 deletions

File tree

‎apps/docs/content/docs/integrations/embeddings.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ Models differ in what they are good at and what they cost. `text-embedding-3-sma
1919

2020
Two things worth knowing before you build on it. Vectors are only comparable when they come from the same model at the same size, so changing either means re-embedding everything you intend to compare. And input longer than the model's limit is shortened to fit rather than rejected, with a warning in the run, so chunk long documents yourself when the tail matters.
2121

22-
Ollama is the exception to most of the above. It runs on your own deployment, so it needs no API key and costs nothing, and the model list is whatever you have pulled onto that server rather than a catalog Sim maintains — the block reads it live, drops the models that report a non-embedding capability, and shows each one's vector width next to its name where Ollama reports one. A server too old to report either will list its chat models too and label none of them, so check the model you pick. Ollama accepts neither a task type nor a size reduction, so the block does not offer those controls for it. Self-hosted deployments configure the server with `OLLAMA_URL`; on Sim Cloud there is no Ollama to reach, so the list comes back empty.
22+
Ollama is the exception to most of the above. It runs on your own deployment, so it needs no API key and adds no provider charge — Sim's own per-run charge still applies — and the model list is whatever you have pulled onto that server rather than a catalog Sim maintains. The block reads it live, drops the models that report a non-embedding capability, and shows each one's vector width next to its name where Ollama reports one. A server too old to report either will list its chat models too and label none of them, so check the model you pick. The block offers no task-type or dimension control for Ollama: task conditioning has no equivalent there, and while recent Ollama builds do accept a dimension override for Matryoshka models, older ones silently ignore it, so Sim uses each model's own width rather than one that may or may not take effect. Self-hosted deployments configure the server with `OLLAMA_URL`; on Sim Cloud there is no Ollama to reach, so the list comes back empty.
2323

2424
Sim's knowledge bases embed separately: a base fixes one model and one vector width when it is created, from a smaller set of models. This block is for embedding text yourself inside a workflow.
2525
{/* MANUAL-CONTENT-END */}

‎apps/docs/content/docs/platform/self-hosting/environment-variables.mdx‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -199,9 +199,11 @@ Both variables apply at creation time and are recorded on the knowledge base, so
199199
affects new knowledge bases only. Existing ones keep the model and width they were built with, and
200200
knowledge bases with different settings cannot be searched together.
201201

202-
Matching the width to the model is yours to get right. Sim only knows the widths a hosted provider
203-
documents, and nothing at all about a model on your Ollama, so a mismatch surfaces as a failed
204-
document rather than a rejected configuration. Common pairings:
202+
Matching the width to the model is yours to get right for a hosted provider: Sim knows only the
203+
widths each one documents, so a width that model cannot emit falls back to `1536` with a warning.
204+
For Ollama it is checked — leave `EMBEDDING_OUTPUT_DIMS` unset and Sim reads the model's width from
205+
your server, refusing to create the knowledge base if it cannot; set it explicitly and a mismatch
206+
surfaces as a failed document instead. Common pairings:
205207

206208
| `EMBEDDING_OUTPUT_DIMS` | Works with |
207209
|----------|-------------|

‎apps/docs/content/docs/platform/self-hosting/troubleshooting.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -250,7 +250,7 @@ The load balancer's backend timeout is closing them. On GKE, attach a `BackendCo
250250

251251
## Knowledge base upload fails
252252

253-
Embeddings need a provider — set `OPENAI_API_KEY`, configure Azure OpenAI, set `KB_EMBEDDING_MODEL=gemini-embedding-001` with a Gemini key, or set `KB_EMBEDDING_MODEL=ollama/<model>` with `OLLAMA_URL` to embed on your own Ollama. If one is configured, verify pgvector is installed on the database.
253+
Embeddings need a provider — set `OPENAI_API_KEY`, configure Azure OpenAI, set `KB_EMBEDDING_MODEL=gemini-embedding-001` with a Gemini key, or set `KB_EMBEDDING_MODEL=ollama/<model>` with `OLLAMA_URL` to embed on your own Ollama. On a self-hosted deployment `OPENROUTER_API_KEY` also serves the OpenAI-family models on its own, as a fallback behind any OpenAI or Azure credentials you have. If one is configured, verify pgvector is installed on the database.
254254

255255
A document that fails with `vector 0 has N unexpected dimensions` means `EMBEDDING_OUTPUT_DIMS` does not match what the model actually emits. The message names both widths. If the width the model returned is one of `384`, `768`, `1024`, `1536`, or `3072`, set the variable to it and recreate the knowledge base. If it is anything else, no column can store it — choose a model that emits one of those five instead, since setting an unstorable width silently falls back to `1536` and the next document fails the same way. Existing knowledge bases keep the width they were created with.
256256

‎apps/sim/blocks/blocks/embeddings.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -276,6 +276,14 @@ export const EmbeddingsBlock: BlockConfig<EmbeddingsResponse> = {
276276
return {
277277
input: params.input,
278278
model,
279+
/**
280+
* Explicitly cleared, not omitted: the executor merges this over the
281+
* saved inputs, so an omitted key leaves the previous provider's
282+
* credential in place and serializes it into a request that has no
283+
* use for one.
284+
*/
285+
apiKey: undefined,
286+
openRouterApiKey: undefined,
279287
taskType: undefined,
280288
dimensions: undefined,
281289
}

‎apps/sim/lib/embeddings/ollama-model-catalog.server.test.ts‎

Lines changed: 42 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import {
88
getOllamaEmbeddingModelMetadata,
99
OllamaEmbeddingModelNotFoundError,
1010
OllamaEmbeddingWidthUnknownError,
11+
OllamaUnreachableError,
1112
} from '@/lib/embeddings/ollama-model-catalog.server'
1213

1314
const fetchMock = vi.fn()
@@ -23,7 +24,8 @@ function serve(
2324
)
2425
}
2526
const { model } = JSON.parse(String(init?.body)) as { model: string }
26-
const detail = models[model]
27+
/** Ollama resolves a bare name to its `:latest` tag server-side. */
28+
const detail = models[model] ?? models[`${model}:latest`]
2729
if (!detail) return Promise.resolve(new Response('not found', { status: 404 }))
2830
return Promise.resolve(
2931
Response.json({
@@ -112,7 +114,8 @@ describe('Ollama embedding model catalog', () => {
112114
await expect(fetchOllamaEmbeddingModelCatalog()).resolves.toEqual([])
113115
})
114116

115-
it('resolves a bare name against the tag Ollama installed it under', async () => {
117+
/** Ollama resolves a bare name to its `:latest` tag itself, so Sim need not. */
118+
it('accepts a bare name and a routing-prefixed one alike', async () => {
116119
serve({
117120
'all-minilm:latest': {
118121
capabilities: ['embedding'],
@@ -121,11 +124,11 @@ describe('Ollama embedding model catalog', () => {
121124
})
122125

123126
await expect(getOllamaEmbeddingModelMetadata('all-minilm')).resolves.toEqual({
124-
id: 'all-minilm:latest',
127+
id: 'all-minilm',
125128
dimensions: 384,
126129
})
127130
await expect(getOllamaEmbeddingModelMetadata('ollama/all-minilm')).resolves.toEqual({
128-
id: 'all-minilm:latest',
131+
id: 'all-minilm',
129132
dimensions: 384,
130133
})
131134
})
@@ -138,6 +141,41 @@ describe('Ollama embedding model catalog', () => {
138141
)
139142
})
140143

144+
/**
145+
* Resolution asks about one model rather than listing the server, so an
146+
* unrelated model's slow or failing `/api/show` cannot delay or break it.
147+
*/
148+
it('asks only about the selected model, never the whole catalog', async () => {
149+
serve({
150+
'all-minilm:latest': {
151+
capabilities: ['embedding'],
152+
modelInfo: { 'bert.embedding_length': 384 },
153+
},
154+
})
155+
156+
await getOllamaEmbeddingModelMetadata('all-minilm')
157+
158+
const paths = fetchMock.mock.calls.map(([url]) => String(url))
159+
expect(paths.every((url) => url.endsWith('/api/show'))).toBe(true)
160+
expect(paths).toHaveLength(1)
161+
})
162+
163+
it('reports a server that stops answering as an outage, not a missing model', async () => {
164+
fetchMock.mockRejectedValue(new Error('ECONNREFUSED'))
165+
166+
await expect(getOllamaEmbeddingModelMetadata('all-minilm')).rejects.toBeInstanceOf(
167+
OllamaUnreachableError
168+
)
169+
})
170+
171+
it('rejects a chat model asked for by name', async () => {
172+
serve({ 'smollm2:135m': { capabilities: ['completion'] } })
173+
174+
await expect(getOllamaEmbeddingModelMetadata('smollm2:135m')).rejects.toBeInstanceOf(
175+
OllamaEmbeddingModelNotFoundError
176+
)
177+
})
178+
141179
/**
142180
* The width is what the client validates the response against, so a model
143181
* whose width cannot be read is refused rather than embedded unchecked.

‎apps/sim/lib/embeddings/ollama-model-catalog.server.ts‎

Lines changed: 34 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -154,8 +154,15 @@ async function loadOllamaEmbeddingModelCatalog(
154154
}
155155

156156
/**
157-
* Resolves one selected model against the server's live catalog, the way the
158-
* OpenRouter tool resolves a model against OpenRouter's.
157+
* Resolves one selected model, the way the OpenRouter tool resolves a model
158+
* against OpenRouter's catalog — but by asking `/api/show` for that model alone.
159+
*
160+
* Deliberately not a catalog scan: this runs on every Ollama embedding request,
161+
* and listing the server would make one unrelated model's slow or failing
162+
* `/api/show` delay or break an embedding that never involved it. Asking
163+
* directly also keeps the failures separable, which a scan cannot do — an
164+
* absent model comes back 404 and is the caller's to fix, while anything else
165+
* is the server's problem and must not be reported as a bad request.
159166
*
160167
* The width is required rather than optional because it is what the client
161168
* validates the response against. Without it a model that quietly returns a
@@ -167,14 +174,29 @@ export async function getOllamaEmbeddingModelMetadata(
167174
signal?: AbortSignal
168175
): Promise<Required<OllamaEmbeddingModel>> {
169176
const name = isOllamaEmbeddingModel(model) ? ollamaEmbeddingModelName(model) : model
170-
const { models, unreachable } = await loadOllamaEmbeddingModelCatalog(signal)
171-
/** An empty catalog because nothing answered is an outage, not a missing model. */
172-
if (unreachable !== undefined) throw new OllamaUnreachableError(unreachable)
173-
/** Ollama resolves a bare name to its `:latest` tag, so both spellings match. */
174-
const metadata = models.find(
175-
(candidate) => candidate.id === name || candidate.id === `${name}:latest`
176-
)
177-
if (!metadata) throw new OllamaEmbeddingModelNotFoundError(name)
178-
if (metadata.dimensions === undefined) throw new OllamaEmbeddingWidthUnknownError(name)
179-
return { id: metadata.id, dimensions: metadata.dimensions }
177+
if (!name) throw new OllamaEmbeddingModelNotFoundError(model)
178+
179+
let detail: { capabilities?: string[]; model_info?: Record<string, unknown> }
180+
try {
181+
detail = ollamaShowUpstreamResponseSchema.parse(
182+
await fetchOllamaJson(
183+
'/api/show',
184+
{ method: 'POST', body: JSON.stringify({ model: name }) },
185+
signal
186+
)
187+
)
188+
} catch (error) {
189+
signal?.throwIfAborted()
190+
const cause = getErrorMessage(error, 'Unknown error')
191+
/** Ollama answers 404 for a model it does not have; anything else is its problem. */
192+
if (cause.includes('404')) throw new OllamaEmbeddingModelNotFoundError(name)
193+
throw new OllamaUnreachableError(cause)
194+
}
195+
196+
if (detail.capabilities && !detail.capabilities.includes('embedding')) {
197+
throw new OllamaEmbeddingModelNotFoundError(name)
198+
}
199+
const dimensions = readEmbeddingLength(detail.model_info)
200+
if (dimensions === undefined) throw new OllamaEmbeddingWidthUnknownError(name)
201+
return { id: name, dimensions }
180202
}
Lines changed: 145 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,145 @@
1+
/**
2+
* @vitest-environment node
3+
*/
4+
import { beforeEach, describe, expect, it, vi } from 'vitest'
5+
6+
const { mockEnv, mockOllamaMetadata } = vi.hoisted(() => ({
7+
mockEnv: {} as { KB_EMBEDDING_MODEL?: string; EMBEDDING_OUTPUT_DIMS?: string },
8+
mockOllamaMetadata: vi.fn(),
9+
}))
10+
11+
/**
12+
* `envNumber` is the real implementation, not a stub: the whole point of the
13+
* cases below is that `createEnv` runs with `skipValidation`, so every value
14+
* arrives as the raw string from the environment however its schema is declared.
15+
*/
16+
vi.mock('@/lib/core/config/env', async (importOriginal) => ({
17+
...(await importOriginal<typeof import('@/lib/core/config/env')>()),
18+
env: mockEnv,
19+
}))
20+
21+
vi.mock('@/lib/embeddings/ollama-model-catalog.server', () => ({
22+
getOllamaEmbeddingModelMetadata: mockOllamaMetadata,
23+
}))
24+
25+
import { getConfiguredKbEmbedding } from '@/lib/knowledge/embeddings'
26+
27+
describe('getConfiguredKbEmbedding', () => {
28+
beforeEach(() => {
29+
vi.clearAllMocks()
30+
mockEnv.KB_EMBEDDING_MODEL = undefined
31+
mockEnv.EMBEDDING_OUTPUT_DIMS = undefined
32+
mockOllamaMetadata.mockRejectedValue(new Error('no server configured'))
33+
})
34+
35+
it('defaults to the model and width knowledge bases were always created at', async () => {
36+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
37+
model: 'text-embedding-3-small',
38+
dimensions: 1536,
39+
})
40+
})
41+
42+
it('stores at the configured width when the model can emit it', async () => {
43+
mockEnv.KB_EMBEDDING_MODEL = 'text-embedding-3-large'
44+
mockEnv.EMBEDDING_OUTPUT_DIMS = '3072'
45+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
46+
model: 'text-embedding-3-large',
47+
dimensions: 3072,
48+
})
49+
})
50+
51+
it('accepts any storable width from a model on the deployment’s own Ollama', async () => {
52+
mockEnv.KB_EMBEDDING_MODEL = 'ollama/nomic-embed-text'
53+
mockEnv.EMBEDDING_OUTPUT_DIMS = '768'
54+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
55+
model: 'ollama/nomic-embed-text',
56+
dimensions: 768,
57+
})
58+
})
59+
60+
/**
61+
* Sim can read a local model's width off the server it is installed on, so an
62+
* unstated width is resolved rather than defaulted — defaulting would create
63+
* every base at 1,536 and fail each document against a 768-wide model.
64+
*/
65+
it('reads an unstated Ollama width from the server the model is installed on', async () => {
66+
mockEnv.KB_EMBEDDING_MODEL = 'ollama/nomic-embed-text'
67+
mockOllamaMetadata.mockResolvedValue({ id: 'nomic-embed-text:latest', dimensions: 768 })
68+
69+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
70+
model: 'ollama/nomic-embed-text',
71+
dimensions: 768,
72+
})
73+
expect(mockOllamaMetadata).toHaveBeenCalledWith('ollama/nomic-embed-text')
74+
})
75+
76+
it('prefers a stated width over the server, and never asks when one is stated', async () => {
77+
mockEnv.KB_EMBEDDING_MODEL = 'ollama/nomic-embed-text'
78+
mockEnv.EMBEDDING_OUTPUT_DIMS = '1024'
79+
mockOllamaMetadata.mockResolvedValue({ id: 'nomic-embed-text:latest', dimensions: 768 })
80+
81+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
82+
model: 'ollama/nomic-embed-text',
83+
dimensions: 1024,
84+
})
85+
expect(mockOllamaMetadata).not.toHaveBeenCalled()
86+
})
87+
88+
/**
89+
* There is no width to fall back to: the adapter cannot ask Ollama for a
90+
* different one, so the platform default would pin the base at a width the
91+
* model does not emit and fail every document. Refusing is recoverable;
92+
* a base created at an impossible width is not.
93+
*/
94+
it('refuses to create a base when the server cannot be asked', async () => {
95+
mockEnv.KB_EMBEDDING_MODEL = 'ollama/nomic-embed-text'
96+
mockOllamaMetadata.mockRejectedValue(new Error('ECONNREFUSED'))
97+
98+
await expect(getConfiguredKbEmbedding()).rejects.toThrow('Set EMBEDDING_OUTPUT_DIMS')
99+
})
100+
101+
it('refuses to create a base when the model emits a width no column can store', async () => {
102+
mockEnv.KB_EMBEDDING_MODEL = 'ollama/odd-width'
103+
mockOllamaMetadata.mockResolvedValue({ id: 'odd-width:latest', dimensions: 1152 })
104+
105+
await expect(getConfiguredKbEmbedding()).rejects.toThrow(
106+
'emits 1152-dimensional vectors, which knowledge bases cannot store'
107+
)
108+
})
109+
110+
it('falls back when the width is not a number at all', async () => {
111+
mockEnv.KB_EMBEDDING_MODEL = 'text-embedding-3-large'
112+
mockEnv.EMBEDDING_OUTPUT_DIMS = 'wide'
113+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
114+
model: 'text-embedding-3-large',
115+
dimensions: 1536,
116+
})
117+
})
118+
119+
it('falls back when the width has no storage column, keeping the chosen model', async () => {
120+
mockEnv.KB_EMBEDDING_MODEL = 'text-embedding-3-large'
121+
mockEnv.EMBEDDING_OUTPUT_DIMS = '1000'
122+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
123+
model: 'text-embedding-3-large',
124+
dimensions: 1536,
125+
})
126+
})
127+
128+
it('falls back when the model cannot emit the configured width', async () => {
129+
mockEnv.KB_EMBEDDING_MODEL = 'gemini-embedding-001'
130+
mockEnv.EMBEDDING_OUTPUT_DIMS = '1024'
131+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
132+
model: 'gemini-embedding-001',
133+
dimensions: 1536,
134+
})
135+
})
136+
137+
it('falls back to the default model when the configured one cannot index a knowledge base', async () => {
138+
mockEnv.KB_EMBEDDING_MODEL = 'text-embedding-ada-002'
139+
mockEnv.EMBEDDING_OUTPUT_DIMS = '768'
140+
await expect(getConfiguredKbEmbedding()).resolves.toEqual({
141+
model: 'text-embedding-3-small',
142+
dimensions: 768,
143+
})
144+
})
145+
})

0 commit comments

Comments
 (0)