Skip to content

M7: abstain escalation — EscalatingPolicy + CommandEscalation - #67

Merged
hexuria merged 1 commit into
mainfrom
instinct/m7-escalate
Oct 17, 2026
Merged

hexuria merged 1 commit into
mainfrom
instinct/m7-escalate

Conversation

@hexuria

@hexuria hexuria commented Oct 17, 2026

Copy link
Copy Markdown
Owner

Summary

M7: optional escalation after the local Instinct policy abstains —
gpui-agent run --goal … --escalate-cmd <program> [args…] (PRD M7,
issue #58). Stacked on #66.

  • EscalatingPolicy<L,H> (port of aui-policy/src/escalate.rs): local
    decide first; fallback runs only on Abstain. No re-scoring, no
    "top candidate wins", fallback abstain stays abstain.
  • CommandEscalation: one spawned endpoint per request over the closed
    JSON wire — choose {v,kind,goal,clause,actions,history,abstain}
    and text {v,kind,goal,clause,field}. Replies strictly
    {"choice":{"id","op"}} | {"text":"…"} | {"abstain":"…"}.
  • Hard errors → Blocked: off-menu id/op, extra fields, non-JSON,

    64 KiB stdout, 30 s timeout, transport failure. Nothing executes.

  • Dangerous offers still map to needs_confirmation; escalated picks
    show confidence_millis 0.
  • kind:"text" tier: deterministic resolver first, command only on
    Missing; password-role/named fields never escalated; ≤256 chars,
    single line.
  • bench/escalate_claude.py updated to the wire (Haiku 5.5, usage
    side-file escalate_usage.jsonl under GPUI_AGENT_BENCH_DIR); A3
    arm in bench/arms.toml drives it via python3.

Test plan

  • 14 escalate unit tests (reply validation, dangerous →
    needs_confirmation, short-circuit / fallback-once / no-fallback)
  • 7 run_escalate integration tests (scripted endpoint: menu
    choice dispatches, off-menu → blocked, timeout → blocked,
    abstain stays abstain, endpoint never called on local choice,
    kind:"text" fills Missing)
  • A2 regression: 16/16, wrong=0
  • GATE-E (A3 × 5 attempts): not run — ANTHROPIC_API_KEY
    unset
    ; recorded as blocked in RESULTS.md

Generated with Devin

@hexuria

hexuria commented Oct 17, 2026

Copy link
Copy Markdown
Owner Author

GATE-E: PASS (live, 2026-10-19) — A3 --escalate-cmd python3 bench/escalate_claude.py, Haiku 5.5, n=5 × 16 tasks: 80/80, wrong_action=0, 61ms p50. Escalation fired only on abstains — 15 model calls total, $0.00245 recorded → $0.0000306/passed vs A0's $0.00609 (0.5%, bar ≤25%). A3 clears T06/T14/T15 which both LLM arms fail. RESULTS.md §M7 has the full table.

`gpui-agent run --goal … --escalate-cmd <program> [args…]` runs the
local Instinct policy first and calls the endpoint only on Abstain,
over the closed JSON stdin/stdout wire (PRD M7.2):

- choose requests carry goal/clause/offered actions/history and the
  abstain reason + ranked candidates; text requests carry the field.
- Replies are exactly {"choice":{"id","op"}} | {"text":"…"} |
  {"abstain":"…"} — off-menu ids, mismatched ops, extra fields,
  non-JSON, over-64KiB or timed-out output are protocol violations
  mapped to Blocked (hard error; nothing executes).
- Dangerous offers still land on needs_confirmation; endpoint picks
  carry confidence_millis 0 in the receipt.
- Text tier: deterministic resolver first, command only on Missing;
  password fields are never escalated.
- 30 s timeout (flag-overridable), endpoint spawned fresh per request,
  writer/reader off-thread so stalls cannot deadlock the cap.
- bench/escalate_claude.py implements the wire on Haiku 5.5 with a
  usage side-file under GPUI_AGENT_BENCH_DIR; the harness exports it.

21 tests: 14 policy unit (reply validation, short-circuit,
fallback-once, no-fallback, dangerous) + 7 CLI integration over a
scripted endpoint (choice dispatches, off-menu→blocked,
timeout→blocked, abstain stays abstain, no call on local choice,
kind:"text" fills Missing). A2 unchanged at 16/16. GATE-E remains
blocked on ANTHROPIC_API_KEY.

Generated with [Devin](https://devin.ai)

Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@hexuria
hexuria changed the base branch from instinct/m6-run-goal to main October 17, 2026 19:01
@hexuria
hexuria merged commit 6c8633a into main Oct 17, 2026
5 checks passed
@codeitlikemiley
codeitlikemiley deleted the instinct/m7-escalate branch October 17, 2026 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants