Context
The escalation judge's packaged prompt and the capability classifier's rubric read like they were calibrated on final answers from coding agents (verifiers, harnesses, test passes). When libsy runs on the live step path of a general agent runtime, most judged responses are intermediate tool steps: a couple of tool calls, little or no prose, success only visible in the tool results.
Ask
Knobs (or guidance) for calibrating the judges on tool-step responses, for example:
- a documented way to extend or re-anchor the escalation judge's rubric for intermediate steps without replacing the whole packaged prompt (today the prompt override is all-or-nothing);
- optionally letting the escalation judge see recent tool results (not just message text) the way the stage router's signal extractor does — failure text in tool results is the strongest escalate evidence we have in practice;
- published calibration notes: what verdict base rates the defaults were tuned for, so hosts can tell drift from miscalibration.
We run escalation mode per agent step in production behind a flag (fork: gumloop/gumloop-switchyard) and can share verdict distributions from our replay harness if that helps ground the defaults.
Context
The escalation judge's packaged prompt and the capability classifier's rubric read like they were calibrated on final answers from coding agents (verifiers, harnesses, test passes). When libsy runs on the live step path of a general agent runtime, most judged responses are intermediate tool steps: a couple of tool calls, little or no prose, success only visible in the tool results.
Ask
Knobs (or guidance) for calibrating the judges on tool-step responses, for example:
We run escalation mode per agent step in production behind a flag (fork: gumloop/gumloop-switchyard) and can share verdict distributions from our replay harness if that helps ground the defaults.