Skip to content

Commit 42d920a

Browse files
DavertMikclaude
andcommitted
feat(Decision): use OpenRouter Decisions API only, document custom decisions
Drop the decision provider option and @ai-sdk/typesafe-ai: all decision models, including TypeSafe Jev, are served by the OpenRouter Decisions API. Document how decisions use AI SDK experimental_decide and how to ask choice/score questions from a custom helper. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 827a459 commit 42d920a

9 files changed

Lines changed: 87 additions & 79 deletions

File tree

‎docs/agents.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,8 @@ I.decide([
186186

187187
The agent runs the statement on the live page like any other command and keeps it only if it passes. The test then checks it on every run.
188188

189+
To let the agent use decisions, enable the Decision helper and set `OPENROUTER_API_KEY`. See [Decision Assertions](/assertions#decision-assertions) for the config.
190+
189191
Decision models like [Jev](https://openrouter.ai/typesafe/jev-1.13) are built for this. They answer in a fraction of a second, cost a fraction of a cent per request, and return a probability instead of free text. That makes them fast and cheap enough to run on every CI build, and predictable enough to keep in a test.
190192

191193
## Skills bundle

‎docs/ai.md‎

Lines changed: 43 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,49 @@ I.decideVisually('sidebar is shown')
353353
354354
It uses a [decision model](https://openrouter.ai/models?output_modalities=decisions) like [Jev](https://openrouter.ai/typesafe/jev-1.13) instead of a chat model. A decision model reads the page and returns the probability that a statement is true. It is fast, costs a fraction of a cent per request, and gives a probability instead of free text, so a step passes or fails on a confidence threshold you set.
355355
356-
Configure it in `ai.decisionModel`. Decisions call the decisions API directly, so they don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
356+
Decisions are requested through the [OpenRouter Decisions API](https://openrouter.ai/models?output_modalities=decisions) with AI SDK [`experimental_decide`](https://ai-sdk.dev/docs/ai-sdk-core/decisions). Set `OPENROUTER_API_KEY` and configure the model in `ai.decisionModel`. Decisions don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
357+
358+
### Custom Decisions
359+
360+
`I.decide` asks yes/no questions. AI SDK decisions also support `choice` and `score` questions. Use [`experimental_decide`](https://ai-sdk.dev/docs/ai-sdk-core/decisions) in a [custom helper](/helpers) to ask them about the current page:
361+
362+
```js
363+
import Helper from '@codeceptjs/helper'
364+
import { experimental_decide as decide } from 'ai'
365+
import { createOpenRouter } from '@openrouter/ai-sdk-provider'
366+
367+
class PageDecisions extends Helper {
368+
async grabPageKind() {
369+
const { Playwright } = this.helpers
370+
const openrouter = createOpenRouter()
371+
372+
const { answers } = await decide({
373+
model: openrouter.evaluationModel('typesafe/jev-1.13'),
374+
state: {
375+
url: await Playwright.grabCurrentUrl(),
376+
aria: await Playwright.grabAriaSnapshot(),
377+
},
378+
questions: {
379+
page: {
380+
type: 'choice',
381+
instructions: 'What kind of page is shown?',
382+
criteria: { login: 'sign in form', dashboard: 'user dashboard', error: 'error message' },
383+
},
384+
},
385+
})
386+
387+
return answers.page.choice
388+
}
389+
}
390+
391+
export default PageDecisions
392+
```
393+
394+
```js
395+
const kind = await I.grabPageKind()
396+
```
397+
398+
`ai` and `@openrouter/ai-sdk-provider` are installed with CodeceptJS. A `score` question returns a fractional position on a list of levels, and a `boolean` question returns a probability. See [AI SDK decisions](https://ai-sdk.dev/docs/ai-sdk-core/decisions) for all question types.
357399
358400
## Advanced Configuration
359401

‎docs/assertions.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -426,15 +426,14 @@ helpers: {
426426

427427
| Option | Default | Description |
428428
|---|---|---|
429-
| `provider` | `openrouter` | `openrouter` reads `OPENROUTER_API_KEY`, `typesafe` reads `TYPESAFE_API_KEY` |
430-
| `apiKey` | | API key, overrides the environment variable |
429+
| `apiKey` | | OpenRouter API key, overrides `OPENROUTER_API_KEY` |
431430
| `model` | `typesafe/jev-1.13` | model for `I.decide` |
432-
| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually`, OpenRouter only |
431+
| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually` |
433432
| `confidence` | `0.7` | minimal probability for a statement to pass |
434433
| `timeout` | `15000` | request timeout in ms |
435434
| `maxLength` | `12000` | maximal length of ARIA snapshot or HTML sent to the model |
436435

437-
Decisions don't need the `--ai` flag, and `ai.model` is not required.
436+
Any model from the [OpenRouter decision models](https://openrouter.ai/models?output_modalities=decisions) list can be used. Decisions don't need the `--ai` flag, and `ai.model` is not required.
438437

439438
Then assert statements about the current page:
440439

‎docs/helpers/Decision.md‎

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,6 @@ Decision model is configured in the `ai.decisionModel` section of the config:
4141
```js
4242
ai: {
4343
decisionModel: {
44-
provider: 'openrouter',
4544
model: 'typesafe/jev-1.13',
4645
visualModel: 'cloudflare/clef',
4746
confidence: 0.7,
@@ -53,10 +52,12 @@ helpers: {
5352
}
5453
```
5554

56-
* `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`).
57-
* `apiKey` (optional) - API key, overrides the environment variable.
58-
* `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider.
59-
* `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only.
55+
Decisions are requested through the [OpenRouter Decisions API][3]
56+
with AI SDK [`experimental_decide`][4]. Set `OPENROUTER_API_KEY` to use it.
57+
58+
* `apiKey` (optional) - OpenRouter API key, overrides `OPENROUTER_API_KEY` environment variable.
59+
* `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`.
60+
* `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`.
6061
* `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass.
6162
* `timeout` (default: `15000`) - request timeout in ms.
6263
* `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
@@ -95,9 +96,9 @@ const probability = await I.decide('cart is empty');
9596

9697
#### Parameters
9798

98-
* `statements` **([string][4] | [Array][5]<[string][4]>)** statement or list of statements to verify.
99+
* `statements` **([string][5] | [Array][6]<[string][5]>)** statement or list of statements to verify.
99100

100-
Returns **[Promise][6]<([number][7] | [Array][5]<[number][7]> | [undefined][8])>** probability of each statement, `undefined` in `skip` mode.
101+
Returns **[Promise][7]<([number][8] | [Array][6]<[number][8]> | [undefined][9])>** probability of each statement, `undefined` in `skip` mode.
101102

102103
### decideVisually
103104

@@ -112,22 +113,24 @@ I.decideVisually(['logo is in the header', 'page uses dark theme']);
112113

113114
#### Parameters
114115

115-
* `statements` **([string][4] | [Array][5]<[string][4]>)** statement or list of statements to verify.
116+
* `statements` **([string][5] | [Array][6]<[string][5]>)** statement or list of statements to verify.
116117

117-
Returns **[Promise][6]<([number][7] | [Array][5]<[number][7]> | [undefined][8])>** probability of each statement, `undefined` in `skip` mode.
118+
Returns **[Promise][7]<([number][8] | [Array][6]<[number][8]> | [undefined][9])>** probability of each statement, `undefined` in `skip` mode.
118119

119120
[1]: https://openrouter.ai/typesafe/jev-1.13
120121

121122
[2]: https://openrouter.ai/cloudflare/clef
122123

123124
[3]: https://openrouter.ai/models?output_modalities=decisions
124125

125-
[4]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String
126+
[4]: https://ai-sdk.dev/docs/ai-sdk-core/decisions
127+
128+
[5]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String
126129

127-
[5]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array
130+
[6]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array
128131

129-
[6]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise
132+
[7]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Promise
130133

131-
[7]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number
134+
[8]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number
132135

133-
[8]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/undefined
136+
[9]: https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/undefined

‎lib/ai.js‎

Lines changed: 4 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,6 @@ import event from './event.js'
55
import { removeNonInteractiveElements, minifyHtml, splitByChunks } from './html.js'
66
import { generateText, experimental_decide as decide, APICallError } from 'ai'
77
import { createOpenRouter } from '@openrouter/ai-sdk-provider'
8-
import { createTypeSafeAi } from '@ai-sdk/typesafe-ai'
98
import { fileURLToPath } from 'url'
109
import path from 'path'
1110
import { fileExists, resolveImportModulePath } from './utils.js'
@@ -273,13 +272,7 @@ function parseCodeBlocks(response) {
273272
return modifiedSnippets.filter(snippet => !!snippet)
274273
}
275274

276-
const DECISION_PROVIDERS = {
277-
openrouter: { keyName: 'OPENROUTER_API_KEY' },
278-
typesafe: { keyName: 'TYPESAFE_API_KEY' },
279-
}
280-
281275
const defaultDecisionConfig = {
282-
provider: 'openrouter',
283276
model: 'typesafe/jev-1.13',
284277
visualModel: 'cloudflare/clef',
285278
confidence: 0.7,
@@ -293,23 +286,19 @@ class DecisionAI {
293286
constructor(config = {}) {
294287
this.config = { ...defaultDecisionConfig, ...config }
295288

296-
const { provider, confidence } = this.config
297-
this.provider = DECISION_PROVIDERS[provider]
298-
if (!this.provider) throw new Error(`Unknown decision provider "${provider}" in ai.decisionModel, use one of: ${Object.keys(DECISION_PROVIDERS).join(', ')}`)
289+
const { confidence } = this.config
299290
if (!(confidence > 0 && confidence < 1)) throw new Error(`ai.decisionModel.confidence must be between 0 and 1, got ${confidence}`)
300291

301292
this.fetchImpl = fetch
302293
}
303294

304295
checkModel() {
305-
if (this.config.apiKey || process.env[this.provider.keyName]) return
296+
if (this.config.apiKey || process.env.OPENROUTER_API_KEY) return
306297

307298
const noKeyErrorMessage = `
308299
No API key is set for decision model.
309300
310-
[!] Set ${this.provider.keyName} environment variable or apiKey in ai.decisionModel config.
311-
312-
Example (connect to OpenRouter, default):
301+
[!] Set OPENROUTER_API_KEY environment variable or apiKey in ai.decisionModel config.
313302
314303
export OPENROUTER_API_KEY=sk-or-...
315304
@@ -322,17 +311,6 @@ class DecisionAI {
322311
323312
Get a key at https://openrouter.ai/settings/keys
324313
325-
Example (connect to TypeSafe):
326-
327-
export TYPESAFE_API_KEY=...
328-
329-
ai: {
330-
decisionModel: {
331-
provider: 'typesafe',
332-
model: 'jev-latest',
333-
}
334-
}
335-
336314
See https://openrouter.ai/models?output_modalities=decisions for all decision models.
337315
`.trim()
338316

@@ -372,7 +350,7 @@ class DecisionAI {
372350

373351
decisionModel(model) {
374352
const settings = {
375-
apiKey: this.config.apiKey || process.env[this.provider.keyName],
353+
apiKey: this.config.apiKey || process.env.OPENROUTER_API_KEY,
376354
fetch: async (url, options) => {
377355
try {
378356
return await this.fetchImpl(url, options)
@@ -382,7 +360,6 @@ class DecisionAI {
382360
}
383361
},
384362
}
385-
if (this.config.provider === 'typesafe') return createTypeSafeAi(settings).decisionModel(model)
386363
return createOpenRouter(settings).evaluationModel(model)
387364
}
388365
}

‎lib/helper/Decision.js‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,6 @@ import { DecisionAI, DecisionConnectionError } from '../ai.js'
4343
* ```js
4444
* ai: {
4545
* decisionModel: {
46-
* provider: 'openrouter',
4746
* model: 'typesafe/jev-1.13',
4847
* visualModel: 'cloudflare/clef',
4948
* confidence: 0.7,
@@ -55,10 +54,12 @@ import { DecisionAI, DecisionConnectionError } from '../ai.js'
5554
* }
5655
* ```
5756
*
58-
* * `provider` (default: `openrouter`) - decision API to call: `openrouter` (reads `OPENROUTER_API_KEY`) or `typesafe` (reads `TYPESAFE_API_KEY`).
59-
* * `apiKey` (optional) - API key, overrides the environment variable.
60-
* * `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`. Use `jev-latest` with the `typesafe` provider.
61-
* * `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`. Available on OpenRouter only.
57+
* Decisions are requested through the [OpenRouter Decisions API](https://openrouter.ai/models?output_modalities=decisions)
58+
* with AI SDK [`experimental_decide`](https://ai-sdk.dev/docs/ai-sdk-core/decisions). Set `OPENROUTER_API_KEY` to use it.
59+
*
60+
* * `apiKey` (optional) - OpenRouter API key, overrides `OPENROUTER_API_KEY` environment variable.
61+
* * `model` (default: `typesafe/jev-1.13`) - decision model used by `decide`.
62+
* * `visualModel` (default: `cloudflare/clef`) - decision model with image input used by `decideVisually`.
6263
* * `confidence` (default: `0.7`) - minimal probability, between 0 and 1, for a statement to pass.
6364
* * `timeout` (default: `15000`) - request timeout in ms.
6465
* * `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.

‎package.json‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,6 @@
9393
"publish-beta": "./runok.cjs publish:next-beta-version"
9494
},
9595
"dependencies": {
96-
"@ai-sdk/typesafe-ai": "^3.0.15",
9796
"@codeceptjs/configure": "^4.0.0-beta.4",
9897
"@codeceptjs/helper": "2.0.4",
9998
"@cucumber/cucumber-expressions": "18",

‎test/unit/ai_test.js‎

Lines changed: 7 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -300,24 +300,20 @@ describe('AI module with mock provider', () => {
300300

301301
describe('DecisionAI', () => {
302302
it('guides user to set API key', async () => {
303-
const key = process.env.TYPESAFE_API_KEY
304-
delete process.env.TYPESAFE_API_KEY
303+
const key = process.env.OPENROUTER_API_KEY
304+
delete process.env.OPENROUTER_API_KEY
305305
try {
306-
const decisionAI = new DecisionAI({ provider: 'typesafe' })
307-
expect(() => decisionAI.checkModel()).to.throw(/TYPESAFE_API_KEY[\s\S]*openrouter\.ai\/settings\/keys/)
308-
const err = await decisionAI.decide('jev-latest', 'state', ['page is loaded']).catch(e => e)
306+
const decisionAI = new DecisionAI()
307+
expect(() => decisionAI.checkModel()).to.throw(/OPENROUTER_API_KEY[\s\S]*openrouter\.ai\/settings\/keys/)
308+
const err = await decisionAI.decide('typesafe/jev-1.13', 'state', ['page is loaded']).catch(e => e)
309309
expect(err.message).to.include('No API key is set for decision model')
310310
} finally {
311-
if (key) process.env.TYPESAFE_API_KEY = key
311+
if (key) process.env.OPENROUTER_API_KEY = key
312312
}
313313
})
314314

315315
it('accepts API key from config', () => {
316-
expect(() => new DecisionAI({ provider: 'typesafe', apiKey: 'secret' }).checkModel()).not.to.throw()
317-
})
318-
319-
it('rejects unknown provider', () => {
320-
expect(() => new DecisionAI({ provider: 'unknown' })).to.throw('Unknown decision provider')
316+
expect(() => new DecisionAI({ apiKey: 'secret' }).checkModel()).not.to.throw()
321317
})
322318

323319
it('asks all statements in one request', async () => {

‎test/unit/helper/Decision_test.js‎

Lines changed: 5 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -123,16 +123,6 @@ describe('Decision helper', () => {
123123
expect(calls[0].body.state.html).to.include('<h1>Checkout</h1>')
124124
})
125125

126-
it('uses typesafe endpoint', async () => {
127-
decision = createDecision({ apiKey: 'secret', provider: 'typesafe', model: 'jev-latest' })
128-
decision._actingHelper = () => browser
129-
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
130-
await decision.decide('page is loaded')
131-
132-
expect(calls[0].url).to.equal('https://api.typesafe.ai/v1/systemone')
133-
expect(calls[0].body.model).to.equal('jev-latest')
134-
})
135-
136126
it('reports http errors', async () => {
137127
decision.decisionAI.fetchImpl = fakeFetch({}, calls, 402)
138128
const err = await decision.decide('page is loaded').catch(e => e)
@@ -204,23 +194,22 @@ describe('Decision helper', () => {
204194
})
205195

206196
it('validates config', () => {
207-
expect(() => createDecision({ apiKey: 'secret', provider: 'unknown' })).to.throw('Unknown decision provider')
208197
expect(() => createDecision({ apiKey: 'secret', confidence: 1.5 })).to.throw('between 0 and 1')
209198
})
210199

211200
it('requires api key only when deciding', async () => {
212-
const key = process.env.TYPESAFE_API_KEY
213-
delete process.env.TYPESAFE_API_KEY
201+
const key = process.env.OPENROUTER_API_KEY
202+
delete process.env.OPENROUTER_API_KEY
214203
try {
215-
decision = createDecision({ provider: 'typesafe' })
204+
decision = createDecision({})
216205
decision._actingHelper = () => browser
217206
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
218207

219208
const err = await decision.decide('page is loaded').catch(e => e)
220-
expect(err.message).to.include('TYPESAFE_API_KEY')
209+
expect(err.message).to.include('OPENROUTER_API_KEY')
221210
expect(calls).to.be.empty
222211
} finally {
223-
if (key) process.env.TYPESAFE_API_KEY = key
212+
if (key) process.env.OPENROUTER_API_KEY = key
224213
}
225214
})
226215

0 commit comments

Comments
 (0)