Skip to content

Commit b5d2241

Browse files
DavertMikclaude
andcommitted
feat: configure decision model in ai.decisionModel
DecisionAI owns decision defaults and validation and reads them from ai.decisionModel (same key as explorbot). Decision helper has no own options and works without --ai flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 parent cc44fa5 commit b5d2241

5 files changed

Lines changed: 92 additions & 49 deletions

File tree

‎docs/ai.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -353,7 +353,7 @@ I.decideVisually('sidebar is shown')
353353
354354
It uses a [decision model](https://openrouter.ai/models?output_modalities=decisions) like [Jev](https://openrouter.ai/typesafe/jev-1.13) instead of a chat model. A decision model reads the page and returns the probability that a statement is true. It is fast, costs a fraction of a cent per request, and gives a probability instead of free text, so a step passes or fails on a confidence threshold you set.
355355
356-
The Decision helper calls the decisions API directly and does not use the `ai` config section or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
356+
Configure it in `ai.decisionModel`. Decisions call the decisions API directly, so they don't need `ai.model` or the `--ai` flag. See [Decision Assertions](/assertions#decision-assertions) for setup and usage.
357357
358358
## Advanced Configuration
359359

‎docs/assertions.md‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -407,18 +407,35 @@ A [decision model](https://openrouter.ai/models?output_modalities=decisions), li
407407
- **Cost-efficient.** A request costs a fraction of a cent. You can run decision assertions in every CI build.
408408
- **Reliable.** The answer is a probability, not free text. There is nothing to parse, and you choose how confident the model must be for the step to pass.
409409

410-
Enable the helper next to your browser helper and set `OPENROUTER_API_KEY`:
410+
Set `OPENROUTER_API_KEY`, configure the decision model in the `ai` section, and enable the helper next to your browser helper:
411411

412412
```js
413-
helpers: {
414-
Playwright: { url: 'http://localhost' },
415-
Decision: {
413+
ai: {
414+
decisionModel: {
416415
model: 'typesafe/jev-1.13',
417416
confidence: 0.7,
418417
},
418+
},
419+
helpers: {
420+
Playwright: { url: 'http://localhost' },
421+
Decision: {},
419422
}
420423
```
421424

425+
`ai.decisionModel` accepts:
426+
427+
| Option | Default | Description |
428+
|---|---|---|
429+
| `provider` | `openrouter` | `openrouter` reads `OPENROUTER_API_KEY`, `typesafe` reads `TYPESAFE_API_KEY` |
430+
| `apiKey` | | API key, overrides the environment variable |
431+
| `model` | `typesafe/jev-1.13` | model for `I.decide` |
432+
| `visualModel` | `cloudflare/clef` | model with image input for `I.decideVisually`, OpenRouter only |
433+
| `confidence` | `0.7` | minimal probability for a statement to pass |
434+
| `timeout` | `15000` | request timeout in ms |
435+
| `maxLength` | `12000` | maximal length of ARIA snapshot or HTML sent to the model |
436+
437+
Decisions don't need the `--ai` flag, and `ai.model` is not required.
438+
422439
Then assert statements about the current page:
423440

424441
```js

‎lib/ai.js‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ class AiAssistant {
5959
debug('Enabling AI assistant')
6060
this.isEnabled = true
6161

62-
const { html, prompts, ...aiConfig } = config
62+
const { html, prompts, decisionModel, ...aiConfig } = config
6363

6464
this.config = Object.assign(this.config, aiConfig)
6565
this.htmlConfig = Object.assign(defaultHtmlConfig, html)
@@ -276,37 +276,60 @@ const DECISION_ENDPOINTS = {
276276
typesafe: { url: 'https://api.typesafe.ai/v1/systemone', keyName: 'TYPESAFE_API_KEY' },
277277
}
278278

279+
const defaultDecisionConfig = {
280+
provider: 'openrouter',
281+
model: 'typesafe/jev-1.13',
282+
visualModel: 'cloudflare/clef',
283+
confidence: 0.7,
284+
timeout: 15000,
285+
maxLength: 12000,
286+
}
287+
279288
class DecisionConnectionError extends Error {}
280289

281290
class DecisionAI {
282-
constructor({ provider = 'openrouter', apiKey, timeout = 15000 } = {}) {
291+
constructor(config = {}) {
292+
this.config = { ...defaultDecisionConfig, ...config }
293+
294+
const { provider, confidence } = this.config
283295
this.endpoint = DECISION_ENDPOINTS[provider]
284-
if (!this.endpoint) throw new Error(`Unknown decision provider "${provider}", use one of: ${Object.keys(DECISION_ENDPOINTS).join(', ')}`)
285-
this.provider = provider
286-
this.apiKey = apiKey
287-
this.timeout = timeout
296+
if (!this.endpoint) throw new Error(`Unknown decision provider "${provider}" in ai.decisionModel, use one of: ${Object.keys(DECISION_ENDPOINTS).join(', ')}`)
297+
if (!(confidence > 0 && confidence < 1)) throw new Error(`ai.decisionModel.confidence must be between 0 and 1, got ${confidence}`)
298+
288299
this.fetchImpl = fetch
289300
}
290301

291302
checkModel() {
292-
if (this.apiKey || process.env[this.endpoint.keyName]) return
303+
if (this.config.apiKey || process.env[this.endpoint.keyName]) return
293304

294305
const noKeyErrorMessage = `
295306
No API key is set for decision model.
296307
297-
[!] Set ${this.endpoint.keyName} environment variable or pass apiKey in config.
308+
[!] Set ${this.endpoint.keyName} environment variable or apiKey in ai.decisionModel config.
298309
299310
Example (connect to OpenRouter, default):
300311
301312
export OPENROUTER_API_KEY=sk-or-...
302313
314+
ai: {
315+
decisionModel: {
316+
model: 'typesafe/jev-1.13',
317+
confidence: 0.7,
318+
}
319+
}
320+
303321
Get a key at https://openrouter.ai/settings/keys
304322
305323
Example (connect to TypeSafe):
306324
307325
export TYPESAFE_API_KEY=...
308326
309-
{ provider: 'typesafe', model: 'jev-latest' }
327+
ai: {
328+
decisionModel: {
329+
provider: 'typesafe',
330+
model: 'jev-latest',
331+
}
332+
}
310333
311334
See https://openrouter.ai/models?output_modalities=decisions for all decision models.
312335
`.trim()
@@ -317,12 +340,12 @@ class DecisionAI {
317340
async decide(model, state, statements) {
318341
this.checkModel()
319342

320-
const apiKey = this.apiKey || process.env[this.endpoint.keyName]
343+
const apiKey = this.config.apiKey || process.env[this.endpoint.keyName]
321344
const questions = Object.fromEntries(statements.map((statement, i) => [`q${i}`, { type: 'noul', instructions: statement }]))
322345
debug('Decision request', model, statements)
323346

324347
const controller = new AbortController()
325-
const timer = setTimeout(() => controller.abort(), this.timeout)
348+
const timer = setTimeout(() => controller.abort(), this.config.timeout)
326349

327350
let response
328351
try {
@@ -333,7 +356,7 @@ class DecisionAI {
333356
signal: controller.signal,
334357
})
335358
} catch (err) {
336-
if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.timeout}ms`)
359+
if (controller.signal.aborted) throw new DecisionConnectionError(`Decision model ${model} did not respond in ${this.config.timeout}ms`)
337360
throw new DecisionConnectionError(`Decision model ${model} request failed: ${err.message}`)
338361
} finally {
339362
clearTimeout(timer)

‎lib/helper/Decision.js‎

Lines changed: 13 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -3,22 +3,14 @@ import path from 'path'
33

44
import Helper from '@codeceptjs/helper'
55
import Container from '../container.js'
6+
import Config from '../config.js'
67
import store from '../store.js'
78
import AssertionFailedError from '../assert/error.js'
89
import { compactAriaSnapshot } from '../aria.js'
910
import { minifyHtml } from '../html.js'
1011
import { pickActingHelper } from '../utils/trace.js'
1112
import { DecisionAI, DecisionConnectionError } from '../ai.js'
1213

13-
const defaultConfig = {
14-
provider: 'openrouter',
15-
model: 'typesafe/jev-1.13',
16-
visualModel: 'cloudflare/clef',
17-
confidence: 0.7,
18-
timeout: 15000,
19-
maxLength: 12000,
20-
}
21-
2214
/**
2315
* Asserts statements about the current page with a decision model.
2416
*
@@ -45,15 +37,21 @@ const defaultConfig = {
4537
*
4638
* ## Configuration
4739
*
40+
* Decision model is configured in the `ai.decisionModel` section of the config.
41+
* The helper itself has no options:
42+
*
4843
* ```js
49-
* helpers: {
50-
* Playwright: { url: 'http://localhost', browser: 'chromium' },
51-
* Decision: {
44+
* ai: {
45+
* decisionModel: {
5246
* provider: 'openrouter',
5347
* model: 'typesafe/jev-1.13',
5448
* visualModel: 'cloudflare/clef',
5549
* confidence: 0.7,
5650
* },
51+
* },
52+
* helpers: {
53+
* Playwright: { url: 'http://localhost', browser: 'chromium' },
54+
* Decision: {},
5755
* }
5856
* ```
5957
*
@@ -65,26 +63,16 @@ const defaultConfig = {
6563
* * `timeout` (default: `15000`) - request timeout in ms.
6664
* * `maxLength` (default: `12000`) - maximal length of ARIA snapshot or HTML sent to the model.
6765
*
66+
* Decisions work without the `--ai` flag.
6867
* Browse all decision models at [OpenRouter](https://openrouter.ai/models?output_modalities=decisions).
6968
*
7069
* ## Methods
7170
*/
7271
class Decision extends Helper {
7372
constructor(config = {}) {
7473
super(config)
75-
this.options = { ...defaultConfig, ...config }
76-
77-
const { confidence } = this.options
78-
if (!(confidence > 0 && confidence < 1)) throw new Error(`Decision confidence must be between 0 and 1, got ${confidence}`)
79-
80-
this.decisionAI = new DecisionAI(this.options)
81-
}
82-
83-
static _config() {
84-
return [
85-
{ name: 'provider', message: 'Decision API provider (openrouter, typesafe)', default: defaultConfig.provider },
86-
{ name: 'model', message: 'Decision model', default: defaultConfig.model },
87-
]
74+
this.decisionAI = new DecisionAI(Config.get('ai', {}).decisionModel)
75+
this.options = this.decisionAI.config
8876
}
8977

9078
/**

‎test/unit/helper/Decision_test.js‎

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,12 @@ import os from 'os'
44
import path from 'path'
55
import Decision from '../../../lib/helper/Decision.js'
66
import store from '../../../lib/store.js'
7+
import Config from '../../../lib/config.js'
8+
9+
function createDecision(decisionModel) {
10+
Config.create({ ai: { decisionModel } })
11+
return new Decision({})
12+
}
713

814
function fakeFetch(answers, calls, status = 200) {
915
return async (url, options) => {
@@ -32,9 +38,11 @@ describe('Decision helper', () => {
3238
let decision
3339
let calls
3440

41+
afterEach(() => Config.reset())
42+
3543
beforeEach(() => {
3644
calls = []
37-
decision = new Decision({ apiKey: 'secret' })
45+
decision = createDecision({ apiKey: 'secret' })
3846
decision._actingHelper = () => browser
3947
})
4048

@@ -63,7 +71,7 @@ describe('Decision helper', () => {
6371
})
6472

6573
it('respects configured confidence', async () => {
66-
decision = new Decision({ apiKey: 'secret', confidence: 0.95 })
74+
decision = createDecision({ apiKey: 'secret', confidence: 0.95 })
6775
decision._actingHelper = () => browser
6876
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
6977

@@ -120,7 +128,7 @@ describe('Decision helper', () => {
120128
})
121129

122130
it('uses typesafe endpoint', async () => {
123-
decision = new Decision({ apiKey: 'secret', provider: 'typesafe', model: 'jev-latest' })
131+
decision = createDecision({ apiKey: 'secret', provider: 'typesafe', model: 'jev-latest' })
124132
decision._actingHelper = () => browser
125133
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
126134
await decision.decide('page is loaded')
@@ -142,7 +150,7 @@ describe('Decision helper', () => {
142150
})
143151

144152
it('times out hanging requests', async () => {
145-
decision = new Decision({ apiKey: 'secret', timeout: 50 })
153+
decision = createDecision({ apiKey: 'secret', timeout: 50 })
146154
decision._actingHelper = () => browser
147155
decision.decisionAI.fetchImpl = (url, { signal }) => new Promise((_, reject) => signal.addEventListener('abort', () => reject(new Error('aborted'))))
148156

@@ -179,16 +187,23 @@ describe('Decision helper', () => {
179187
expect(err.isTerminal).to.be.undefined
180188
})
181189

190+
it('reads config from ai.decisionModel', () => {
191+
decision = createDecision({ apiKey: 'secret', model: 'jev-latest', confidence: 0.9 })
192+
expect(decision.options.model).to.equal('jev-latest')
193+
expect(decision.options.confidence).to.equal(0.9)
194+
expect(decision.options.visualModel).to.equal('cloudflare/clef')
195+
})
196+
182197
it('validates config', () => {
183-
expect(() => new Decision({ apiKey: 'secret', provider: 'unknown' })).to.throw('Unknown decision provider')
184-
expect(() => new Decision({ apiKey: 'secret', confidence: 1.5 })).to.throw('between 0 and 1')
198+
expect(() => createDecision({ apiKey: 'secret', provider: 'unknown' })).to.throw('Unknown decision provider')
199+
expect(() => createDecision({ apiKey: 'secret', confidence: 1.5 })).to.throw('between 0 and 1')
185200
})
186201

187202
it('requires api key only when deciding', async () => {
188203
const key = process.env.TYPESAFE_API_KEY
189204
delete process.env.TYPESAFE_API_KEY
190205
try {
191-
decision = new Decision({ provider: 'typesafe' })
206+
decision = createDecision({ provider: 'typesafe' })
192207
decision._actingHelper = () => browser
193208
decision.decisionAI.fetchImpl = fakeFetch(noul(0.9), calls)
194209

0 commit comments

Comments
 (0)