report.md

Agent-loop v0 — harness report

Total queries: 50. Generated by python -m eval.agent_loop_v0.harness on the noop SPIs.

Overall

• iterations: mean=2.82, p95=4
• plan-reuse rate (mean): 58.93%
• sticky-route rate (mean): 58.93%
• embedding cache hit rate (mean per query): 53.67%
• wall_ms: p50=1.4, p95=2.3
• stop reasons: budget_iter=3, budget_tokens=5, budget_wall=1, max_iterations=41

By bucket

budget_stressor (n=10)

• iterations: mean=2.10, p95=5
• plan-reuse rate: 36.33%
• sticky-route rate: 36.33%
• embedding cache hit rate: 75.00%
• wall_ms: p50=0.9, p95=2.6
• stop reasons: budget_iter=3, budget_tokens=5, budget_wall=1, max_iterations=1

decomposable (n=10)

• iterations: mean=3.00, p95=3
• plan-reuse rate: 66.67%
• sticky-route rate: 66.67%
• embedding cache hit rate: 43.33%
• wall_ms: p50=1.5, p95=1.6
• stop reasons: max_iterations=10

follow_up (n=10)

• iterations: mean=3.00, p95=3
• plan-reuse rate: 66.67%
• sticky-route rate: 66.67%
• embedding cache hit rate: 50.00%
• wall_ms: p50=1.3, p95=2.3
• stop reasons: max_iterations=10

refinement (n=10)

• iterations: mean=4.00, p95=4
• plan-reuse rate: 75.00%
• sticky-route rate: 75.00%
• embedding cache hit rate: 50.00%
• wall_ms: p50=1.8, p95=2.4
• stop reasons: max_iterations=10

shape_shifter (n=10)

• iterations: mean=2.00, p95=2
• plan-reuse rate: 50.00%
• sticky-route rate: 50.00%
• embedding cache hit rate: 50.00%
• wall_ms: p50=0.9, p95=1.1
• stop reasons: max_iterations=10