Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,12 @@ jobs:
npm --prefix ../shared ci
npm --prefix ../shared/packages/sync run build
npm --prefix ../shared/packages/models run build
# uiohook-napi (the rails' pause-on-user-input hook, an optional dep) has no
# Linux prebuild for Electron's ABI, so install-app-deps compiles libuiohook
# from source - which needs the X11 dev headers. The app never ships Linux;
# these packages exist only so `npm ci` completes on this runner.
- name: X11 headers for uiohook-napi
run: sudo apt-get update && sudo apt-get install -y libx11-dev libxtst-dev libxt-dev libxinerama-dev libx11-xcb-dev libxkbcommon-dev libxkbcommon-x11-dev libxkbfile-dev libxrandr-dev
- run: npm ci
# Hard gates: types + the full test suite.
- name: Typecheck (core)
Expand Down
23 changes: 16 additions & 7 deletions docs/SAFETY_REVIEW.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,29 @@ and where that defense is tested - so a later change that weakens a defense
fails a test instead of shipping.

The governing principle: **the model only proposes; the pipeline guarantees.**
Every mutation is a durable Action that gates for approval, binds its payload
by hash, executes once, and verifies. Injection cannot manufacture an approved
action out of nothing - it can only try to steer a task the user already
approved. So the defenses below are about bounding that steering, and about
never letting the agent cross an identity or payment boundary on its own.
Every mutation is a durable Action that binds its payload by hash, executes
once, and verifies. The approval policy is risk-tiered (one rule, in
`gate-host.ts needsApproval`): **sends (message, email) and computer-use tasks
(the accessibility/vision rails) gate for human approval every time** - a send
is irreversible with no reliable read-back, and computer use takes over the
cursor. Undoable mutations (calendar, reminders) auto-run with an Undo chip;
reads run free; web_task runs unprompted because it acts inside Off Grid's own
watched browser pane - supervised by design, hands back at any sign-in or
payment. The pro "auto-approve" toggle covers computer-use tasks ONLY; sends
ask every time regardless. Injection cannot manufacture an approved action out
of nothing - it can only try to steer a task the user already approved. So the
defenses below are about bounding that steering, and about never letting the
agent cross an identity or payment boundary on its own.

## The threats and the defenses, per rail

### Semantic rail (calendar, reminders, mail, open)

- **Threat:** low. The arguments come from the user's chat turn, not from
scraped content. The model fills a typed tool schema.
- **Defense:** the payload-hash gate - what the user approves is byte-for-byte
what runs; an edit re-binds and re-gates. Sends are `none_fuzzy` and single-
- **Defense:** sends (message, email) gate for approval every time, and the
payload-hash binding means what the user approves is byte-for-byte what
runs; an edit re-binds and re-gates. Sends are `none_fuzzy` and single-
attempt, so a wrong verify can never double-send.
- **Tested:** `shared/packages/use` retry + machine tests (never-double-fire),
`use-runtime.integration.dbtest.ts` (real propose -> verify -> undo).
Expand Down
30 changes: 29 additions & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,8 @@
"unified": "^11.0.5"
},
"optionalDependencies": {
"@nut-tree-fork/nut-js": "^4.2.6"
"@nut-tree-fork/nut-js": "^4.2.6",
"uiohook-napi": "^1.5.5"
},
"devDependencies": {
"@electron-toolkit/eslint-config-prettier": "^3.0.0",
Expand Down
12 changes: 8 additions & 4 deletions src/main/__tests__/gate-host.integration.dbtest.ts
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,12 @@ function makeWorld() {
db.exec(`CREATE TABLE test_reminders (title TEXT NOT NULL)`)

const registry = new HandlerRegistry()
// The fixture is a SEND ('email'): under the current approval policy the gate
// covers sends + the computer-use rails, so the seam these tests exercise -
// park, approve, reject, edit-rebind - only fires for those types. (It was a
// 'reminder' when every mutation gated; reminders now auto-run with Undo.)
registry.register({
type: 'reminder',
type: 'email',
rail: 'semantic',
defaultRisk: 'mutate',
verification: 'read_back',
Expand Down Expand Up @@ -102,8 +106,8 @@ function requestAt(requests: Record<string, unknown>[], index: number): Record<s
}

const proposal = {
type: 'reminder',
intent: 'remind me to send the deck',
type: 'email',
intent: 'email the deck to Sam',
args: { title: 'Send the deck' },
risk: 'mutate'
}
Expand All @@ -126,7 +130,7 @@ describe('the engine gated through the real approval seam', () => {
expect(request).toMatchObject({
kind: 'native',
risk: 'mutate',
actionType: 'reminder',
actionType: 'email',
args: { title: 'Send the deck' }
})
resolveActionGate(String(request.actionId), { kind: 'approve' })
Expand Down
22 changes: 21 additions & 1 deletion src/main/__tests__/harness/fake-llama-server.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,10 @@ export interface FakeLlamaServer {
/** Clear any queued-but-unconsumed turns + the recorded requests — call between tests
* so a case that over-enqueues (e.g. the step-budget cap) can't leak into the next. */
reset(): void
/** The request bodies received, parsed — for asserting what the REAL llm actually sent. */
/** The request bodies received, parsed — for asserting what the REAL llm actually sent.
* Planner (PLAN_SCHEMA) calls are answered out-of-band and recorded separately. */
readonly requests: Array<Record<string, unknown>>
readonly plannerRequests: Array<Record<string, unknown>>
close(): Promise<void>
}

Expand Down Expand Up @@ -101,6 +103,7 @@ function sseFramesFor(turn: FakeTurn): string[] {
export async function startFakeLlamaServer(): Promise<FakeLlamaServer> {
const queue: FakeTurn[] = []
const requests: Array<Record<string, unknown>> = []
const plannerRequests: Array<Record<string, unknown>> = []

const server = http.createServer((req, res) => {
if (req.method === 'GET' && (req.url === '/health' || req.url === '/v1/models')) {
Expand All @@ -120,6 +123,21 @@ export async function startFakeLlamaServer(): Promise<FakeLlamaServer> {
} catch {
/* keep {} */
}
// The orchestrator's PLANNING pass (tools.ts shouldPlan -> planTask) fires
// before the reactive loop on action-shaped queries, grammar-constrained to
// PLAN_SCHEMA. Answer it with an EMPTY plan out-of-band - it never consumes
// a queued turn and never lands in `requests` - so every test keeps driving
// the reactive loop it scripts, exactly as before the orchestrator existed.
// (A test that wants to exercise planning itself can assert plannerRequests.)
const responseFormat = JSON.stringify(parsed.response_format ?? '')
if (responseFormat.includes('"steps"') && responseFormat.includes('"bindings"')) {
plannerRequests.push(parsed)
res.writeHead(200, { 'Content-Type': 'application/json' })
res.end(
JSON.stringify({ choices: [{ message: { content: '{"steps":[]}' } }] })
)
return
}
requests.push(parsed)
const turn = queue.shift() ?? { content: '' }
if (turn.errorStatus) {
Expand Down Expand Up @@ -189,12 +207,14 @@ export async function startFakeLlamaServer(): Promise<FakeLlamaServer> {
return {
port,
requests,
plannerRequests,
enqueue: (...turns: FakeTurn[]) => {
queue.push(...turns)
},
reset: () => {
queue.length = 0
requests.length = 0
plannerRequests.length = 0
},
close: () => new Promise<void>((r) => server.close(() => r()))
}
Expand Down
4 changes: 4 additions & 0 deletions src/main/accessibility/ax-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import { globalShortcut, systemPreferences } from 'electron'
import { binRoots, exe } from '../runtime-env'
import { llm } from '../llm'
import { loadActuation, type ActuationPort } from '../input/actuation'
import { startUserInputWatch } from '../input/user-input-watch'
import { parseAxElements, type AxElement, type AxSnapshot } from './ax-elements'
import { windowsAxBackend, type AxBackend } from './ax-win'
import { pickTargetApp } from './ax-target'
Expand Down Expand Up @@ -237,6 +238,8 @@ class AxRailHost {
// The kill switch: Esc halts for good. The overlay's Stop routes to the SAME
// guard through the controller session, so both paths end one run.
globalShortcut.register('Escape', () => guard.halt('stopped with Esc'))
// Pause on user input - same defense as the vision rail, same watchdog.
const stopInputWatch = startUserInputWatch((why) => guard.pauseForUser(why))
const releaseSession = registerVisionSession(guard)
// The AX rail is model-agnostic and needs no grounder, so there is no
// grounder notice here (unlike the vision rail).
Expand Down Expand Up @@ -288,6 +291,7 @@ class AxRailHost {
return { ok: false, summary, steps: [] }
} finally {
globalShortcut.unregister('Escape')
stopInputWatch()
releaseSession()
hideSupervisorWindow()
}
Expand Down
51 changes: 44 additions & 7 deletions src/main/actions/__tests__/gate-host.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -284,13 +284,19 @@ describe('parseGateDecision', () => {
})
})

describe('needsApproval (only computer use is gated)', () => {
it('gates the computer-use rails, runs in-app actions straight through', () => {
expect(needsApproval('accessibility')).toBe(true)
expect(needsApproval('vision')).toBe(true)
expect(needsApproval('browser')).toBe(false) // web_task runs in-app
expect(needsApproval('semantic')).toBe(false) // native actions
expect(needsApproval(undefined)).toBe(false)
describe('needsApproval (computer use AND sends are gated)', () => {
it('gates the computer-use rails and send actions; everything else runs through', () => {
expect(needsApproval({ rail: 'accessibility', type: 'computer' })).toBe(true)
expect(needsApproval({ rail: 'vision', type: 'computer' })).toBe(true)
// Sends are irreversible with no reliable read-back - always confirmed.
expect(needsApproval({ rail: 'semantic', type: 'message' })).toBe(true)
expect(needsApproval({ rail: 'semantic', type: 'email' })).toBe(true)
// web_task acts in Off Grid's own watched pane - supervised, not gated.
expect(needsApproval({ rail: 'browser', type: 'web' })).toBe(false)
// Undoable mutations and reads run through (calendar/reminders auto-run + Undo).
expect(needsApproval({ rail: 'semantic', type: 'calendar' })).toBe(false)
expect(needsApproval({ rail: 'semantic', type: 'lookup' })).toBe(false)
expect(needsApproval({ type: 'calendar' })).toBe(false)
})

it('gateHost auto-approves a browser (web_task) action even with a surface listening', async () => {
Expand All @@ -304,6 +310,37 @@ describe('needsApproval (only computer use is gated)', () => {
})
})

describe('send gating (mail_send / messages_send confirm every time)', () => {
it('parks an email send for approval when a surface is listening', async () => {
const seen: InlineGateRequest[] = []
const dispose = registerInlineGateSurface((request) => void seen.push(request))
const parked = gateHost({
action: record({ rail: 'semantic', type: 'email', intent: 'email the deck to Sam' })
})
expect(pendingActionGateCount()).toBe(1)
expect(seen[0]).toMatchObject({ actionType: 'email' })
resolveActionGate('act_1', { kind: 'approve' })
expect(await parked).toEqual({ kind: 'approve' })
dispose()
})

it('the auto toggle covers computer use only - a send still parks in auto mode', async () => {
const unregister = registerApprovalModeProvider(() => 'auto')
const dispose = registerInlineGateSurface(() => {})
try {
const parked = gateHost({
action: record({ rail: 'semantic', type: 'message', intent: 'text Sam' })
})
expect(pendingActionGateCount()).toBe(1) // parked despite auto mode
resolveActionGate('act_1', { kind: 'reject' })
expect(await parked).toMatchObject({ kind: 'reject' })
} finally {
dispose()
unregister()
}
})
})

describe('computerApprovalMode (the Sync-sharing auto/ask setting)', () => {
afterEach(() => {
// Ensure no provider leaks into other tests (default must be 'ask').
Expand Down
36 changes: 23 additions & 13 deletions src/main/actions/gate-host.ts
Original file line number Diff line number Diff line change
Expand Up @@ -199,26 +199,36 @@ export function computerApprovalMode(): ComputerApprovalMode {
return approvalModeProvider?.() ?? 'ask'
}

/** Only COMPUTER-USE tasks ask for approval. The accessibility / vision rails
* drive the real desktop - they take over the user's cursor and keyboard - so
* the user confirms before that happens. Every other action runs IN-APP without
* taking over the machine (the browser rail acts in Off Grid's own page; native
* actions call an API), so it runs without a prompt. */
export function needsApproval(rail: Rail | undefined): boolean {
/** The rails that take over the user's cursor and keyboard. */
function isComputerRail(rail: Rail | undefined): boolean {
return rail === 'accessibility' || rail === 'vision'
}

/** The action types that SEND on the user's behalf (iMessage, email). A send is
* irreversible and has no reliable read-back, so a wrong one cannot be undone
* or even verified - the user confirms before it leaves. */
const SEND_ACTION_TYPES: ReadonlySet<string> = new Set(['message', 'email'])

/** The approval policy, in one place: COMPUTER-USE tasks gate (the
* accessibility / vision rails take over the user's cursor and keyboard) and
* SENDS gate (irreversible, invisible until too late). Everything else runs
* without a prompt: reads are safe, undoable mutations (calendar, reminders)
* auto-run with the Undo chip, and web_task acts inside Off Grid's own watched
* browser pane - supervised by design, never touching the user's cursor. */
export function needsApproval(action: { rail?: Rail; type: string }): boolean {
return isComputerRail(action.rail) || SEND_ACTION_TYPES.has(action.type)
}

/** The GateCallback the engine host is constructed with. */
export async function gateHost({ action }: { action: ActionRecord }): Promise<GateDecision> {
// In-app actions run straight through; only computer use is gated. The env
// flag bypasses even that, for headless testing.
if (approvalBypassed() || !needsApproval(action.rail)) {
// The env flag bypasses the gate entirely, for headless testing.
if (approvalBypassed() || !needsApproval(action)) {
return { kind: 'approve' }
}
// The user's Sync-sharing policy: "Auto-approve" runs computer-use tasks with no
// prompt (they still journal, and the outcome shows in chat); "Ask every time"
// (the default) falls through to park for approval below.
if (computerApprovalMode() === 'auto') {
// The user's Sync-sharing policy: "Auto-approve" runs COMPUTER-USE tasks with
// no prompt (they still journal, and the outcome shows in chat). It never
// covers sends - those ask every time; the toggle's scope is computer use.
if (isComputerRail(action.rail) && computerApprovalMode() === 'auto') {
return { kind: 'approve' }
}
const queued = proposeActionApproval({
Expand Down
Loading
Loading