sdebench: does memory help a coding agent? (boltons-hosted benchmark) - #23
Draft
nicoloboschi wants to merge 180 commits into
Draft
sdebench: does memory help a coding agent? (boltons-hosted benchmark)#23nicoloboschi wants to merge 180 commits into
nicoloboschi wants to merge 180 commits into
Conversation
- synthetic repo built with engineered git history (build.py); a perf commit bundles an int() refill floor (regression) with a legit available() accessor - FAIL_TO_PASS regression repro + HIDDEN_TO_PASS held-out variants + PASS_TO_PASS suite - validated: suite green at HEAD, regression+hidden red, surgical fix -> all green, git revert of the regression commit conflicts (forces surgical fix) - README documents the benchmark design + grading + history A/B
- Dockerfile: deterministic python+pytest+git grading env - harness/run.py: build repo (full|squashed history) -> ship bug report + failing repro -> run opencode (gemini-3.5-flash) -> capture SOURCE diff (tests excluded) -> grade in Docker vs FAIL_TO_PASS+PASS_TO_PASS+HIDDEN from pristine copies - reports resolution/cost(tokens)/speed(wall,turns) - smoke test (full history): agent solved it — 591B fix, 10 passed, 20 turns, 176s
…y pass/fail - if grading fails, surface the NEW problem (failing tests' assertion output, NOT the fix) back to the agent and RESUME its session (opencode -c) to continue its work - metric = number of human-like interventions needed (capped at 5 = drift guard) - cost/turns/wall accumulate across all rounds; solved = passed within the cap - realistic: feedback is what CI/QA would report; agent must still generalize the fix
… regression - a refactor changed DEFAULT_TTL 300->600 and dropped the rationale comment; 300 now lives ONLY in git history (blame/log the constant). bundled with a legit clear(). - repro proves entries expire too late but CANNOT reveal the value; hidden tests pin it to EXACTLY 300. validated: TTL=450 passes repro but fails hidden (underdetermined). - with history: read 300 off blame -> 1-shot. without: binary-search via feedback rounds. this is the task designed to make the intervention-count A/B diverge.
…c discriminates - 300 is a conventional TTL the agent guesses without history (A/B showed 0 interventions both, though full used ~half the tokens). 287 (a 'measured p99' value) can't be guessed: validated TTL=300 now passes repro but FAILS hidden. only history states 287. - expectation: full reads 287 off git blame (0 interv); squashed must discover it via feedback rounds (>0 interv) -> intervention metric diverges.
…e full trace
- run_agent returns a token SPLIT {input, output, reasoning, cache_read, cache_write}
(verified cache_read populated, e.g. 101k cached of 186k input) — $ computed later per model
- captures the structured trajectory (tool steps + assistant text) per round
- main accumulates the split across all feedback rounds; writes result.json (metrics) +
trace.json (full multi-round conversation: bug report -> agent -> feedback -> agent ...)
- ui_export.py maps trace.json -> outputs/sdebench/<run>/agent/all.json (view='agent'); each task-run is a QueryResult whose trajectory is the FULL multi-round conversation (bug report -> agent -> feedback -> agent), with the token split + interventions in meta - ported the purpose-built agent-trace view (RunDetail.vue + style.css) from feat/swebench-cl and rebuilt ui/dist - harness now stores final_patch in trace.json (the UI 'answer') - verified: full & squashed runs render with bug report, clickable tool steps, patch diff
…kens in UI - compute_cost() per-class: $1.50/1M input, $0.15 cached, $9.00 output(+reasoning); cost_usd now live in result.json + ui_export (full ~$0.31, squashed ~$0.59 — the extra intervention ~doubles $) - per model-step in/out tokens stamped onto each trajectory step (tok_in/tok_out) - UI: run-level stat boxes (Total cost / Interventions / Tokens in-out), per-task pills (interventions, $cost, in/out tokens, turns, wall), per-tool-step in→out badge
…adding, compact tokens - each tool step is now one truncated line: [tool] [arg ↳ output-preview …] [in→out] [▸] (was wrapping one char per line via word-break:break-all on a squished flex column) - combined arg + out-preview into one ellipsised .traj-mid (min-width:0 so it truncates) - tighter L/R padding (1rem -> 0.55rem); tokens compact (9.4k→338, tabular, right-aligned) - feedback/say steps wrap normally (pre-wrap), expand still shows full input/output
…put) - verified the mapping is correct (not inverted): tok_in sums to input+cached (285k), tok_out to output+reasoning (4.7k) — input context is just naturally >> output - show ↑ for input/prompt and ↓ for output/generated so it's unambiguous (was a bare →)
…ount) empirically confirmed via raw step tokens: total = input + cache_read + output + reasoning, so 'input' is the NON-cached prompt and 'cache_read' the cached prompt (separate, not nested). => tok_in = input + cache_read and cost = input*$1.50 + cache_read*$0.15 + out*$9.00 are correct.
- harness records each round's submitted patch + grade outcome (passed + pytest) on the trace round — incl. the ones that FAILED eval and triggered the feedback - ui_export inserts a 'patch' step after each round's trajectory; UI renders it as a clickable ✓/✗ row (round + pytest summary) that expands to that round's diff - so the trajectory now shows: agent work -> ✗ submitted (2 failed) -> 🔁 feedback -> agent work -> ✓ submitted (9 passed)
… opencode plugin - new --history hindsight mode: builds the SQUASHED repo (agent can't git-blame), ingests the full git history (each commit's message+diff) into a Hindsight bank, and runs with the opencode Hindsight plugin active (recall mode) pointed at that bank - tests whether memory of history recovers the value of history (the buried 287 constant) when raw git access is gone — vs full (git) and squashed (nothing) - local Hindsight server (main-ish, gemini-3.1-flash-lite) on :8888; ingest verified to surface '287 measured p99' on recall
…ng MODE) - a refactor switched round_cents from banker's rounding (ROUND_HALF_EVEN) to half-up and dropped the 'to match the ledger' rationale; the rule now lives ONLY in git history - different mechanism than ttlcache (algorithmic choice, not a magic number) — tests generalization. validated: repro red at HEAD, banker's fix -> all green, a half-DOWN fix passes the repro but FAILS hidden (underdetermined), banker's only in history - bundled with a legit format_cents() so revert fails PASS_TO_PASS
…r non-guessable - removed task #1 ratelimiter (int() floor too obvious — full==squashed, 0 interv both) - ledger first A/B failed to discriminate: banker's rounding is the GUESSABLE convention for money, so squashed solved it without history. Changed the real rule to round-half-DOWN ('match legacy billing') which agents DON'T default to. Validated: a banker's fix passes the repro but fails hidden -> the natural guess is wrong -> only history reveals half-down. - README documents the non-guessability design rule
… in the UI - capture_git_history(): the task repo's engineered commits (sha/subject/body/diff), newest first - ui_export attaches it to every QueryResult (backfills existing runs by rebuilding the repo); also splits exports by task (ttlcache.json / ledger.json) so tasks don't overwrite each other - UI: a 'Repository history' panel listing the commits, each expandable to its full diff — so you can see the source documents the full/hindsight arms had and squashed didn't
… multi-task layout
- billing: 4 longer modules (money/discount/tax/invoice) + an 18-commit history full of
noise (docs, changelog, vague refactors) where TWO regressions and their guarantees are
buried: 'tidy money module' (rounding half-down->half-up) and 'refactor invoice pipeline'
(tax base discounted->pre-discount). 'find the relevant history' is now actually exercised.
- new dataset layout: datasets/<codebase>/{build.py, tasks/<task>/{task.json,*_test.py}} —
many tasks share ONE codebase + git history (easy to iterate). task.json gains 'codebase'.
- harness resolves build from codebase, test files from the task dir; stores codebase.
- ui_export splits per task_id (tasks on a codebase share the git history view).
- two tasks validated: billing-rounding-001 (critical, non-guessable half-down) and
billing-taxbase-001 (navigate noisy history). both: regression+hidden red -> fix -> 10 green.
…e input - ↑ is the CUMULATIVE prompt (system prompt + tool defs + all prior steps, mostly cached); added +Δ = the NEW context that step introduced (cumulative minus previous step), so a big read shows up as the jump on the FOLLOWING step - first step's ↑ (~9k) is the baseline: opencode system prompt + tool schemas + bug report, re-sent every call; +Δ on step 1 equals that baseline - clearer tooltip distinguishing cumulative vs delta vs generated
- group tool calls into TURNS: opencode issues several tools from ONE prompt, so tokens are per-turn. Each turn shows ONE ↑cumulative +Δnew ↓generated badge instead of repeating it on every batched tool row (the +0 rows that looked free / the row that looked like it 'cost' the whole previous batch). Feedback markers and submitted patches stay as standalone blocks. - reasoning is token-only: gemini-3.5-flash emits no thinking TEXT (event types are just tool_use/step_start/step_finish/text), only a reasoning token COUNT in step_finish. Harness now stamps per-turn reasoning tokens; UI shows '🧠 N' on the turn (new runs).
…ed box) the ↑ prompt is re-sent every turn (sum of per-turn ↑ = total input processed; e.g. 581k over 28 turns even though the last prompt is only 32k). most of it is CACHED (re-sent prefix -> ~60% cache hit) and billed ~10x cheaper, but that was invisible: - per-turn badge now shows ⚡<cached> alongside ↑<total prompt> - run sidebar adds an '⚡ Cached input' stat (k + % of ↑) - per-task pill shows ⚡<cached> within the in-tokens harness stamps per-turn tok_cache (new runs get the per-turn ⚡; run-level works on all runs)
- mem_index.py builds /tmp/sdebench/memindex/<codebase>.json (raw commits: subject/body/files/diff) - --history memtool: squashed repo (no git trail) + recall_intent tool over the full history's index, so the TOOL replaces git — the fair 'beat full git' comparison - load_env/run_agent gain mem_index (sets MEM_INDEX, enables the recall_intent plugin mode) - smoke (rounding): memtool 16 turns $0.30 48s vs full 25 turns $0.41 80s
…a engine) - 9 modules (tokens/nodes/parser/refs/sheet/functions/evaluator/errors/engine), longer files, ~22-commit noisy history. Keeps billing as the 'easy' codebase. - HARD regression FAR FROM SYMPTOM + history-dependent + underdetermined: a 'centralize argument evaluation' refactor made the evaluator short-circuit on any error arg, so COUNT/AVG/MIN/MAX over a range with an error cell return the error instead of aggregating the numbers (SUM is unaffected since it propagates anyway -> slips past existing tests). The bug is in evaluator.py, not in the COUNT code the symptom points at. Policy (functions decide; SUM propagates, aggregates skip) lives in functions.py + history; the precise 'don't short-circuit calls' invariant is only in history. Validated: HEAD 8 green / regression+hidden 3 fail; COUNT-only fix passes repro but fails AVG/MIN/MAX hidden; correct general fix -> 13 green. - also: base PROMPT now asks the agent to work efficiently (applied to ALL arms = fair)
…e prompt) Exp1: tests push (auto-inject top-2 TF-ranked commits into the bug report) vs pull (recall_intent tool the agent must invoke). Same retrieval, different delivery. squashed repo + injected context.
…per bound) Research ablation (not a deployable method): injects the exact regression commit's diff into the prompt. The ceiling of 'perfect pushed memory' — if oracle doesn't beat full, behavior (not knowledge/retrieval) is the wall. Added cause_subject to all 5 tasks.
…via SDE_VARIANT Exp2: test whether constraining exploration (uniform, fair prompt) cuts cost independent of memory. variant stored in result.json.
…ers stack push (inject) beats git on 4/5 tasks where pull (recall_intent tool) doesn't; behavioral constraint (minimal) helps exploration-bound but not knowledge-bound tasks; the two levers fix different bottlenecks and STACK (inject+minimal -30% to -41% vs git, never hurts).
…al ablations Offline result: neither top-k nor a bug+repro 'rich query' surfaces minicalc's symptom-distant cause commit — simple symptom-based retrieval can't find it (the bug is a wrong return value, no traceback; the cause shares no terms with the symptom). Bounds what push retrieval can do.
…ble by symptom query
… tool Tests whether keeping the pull tool available on top of pushed symptom-context lets the agent find symptom-distant causes (the inject->oracle gap) by querying after it understands the code.
The claude arm reflects harness-side (no plugin), but was sending the raw bug report as the query — the plugin now frames reflect with strict historian rendering rules (declarative past-tense facts, no imperatives, no cross-episode narratives), so the harness mirrors buildReflectQuery to keep the arm faithful to the product. SDE_TASK_FILTER (comma-separated dir-name substrings) selects non-alphabetical task subsets, e.g. just-added families.
…n attr An expired-OAuth claude run parsed as a normal 0-token turn and silently burned the whole intervention budget in 8 seconds; is_error results now raise with the agent's message. The final-save truncation used QueryResult's field name on Query objects (query_id -> id).
The vanilla-arm chat pointer in the prompt made a strong agent read the transcripts every run, collapsing the vanilla/memory gap by prompt design rather than agent ability — chats stay reachable (sessions / transcript files) but the prompt no longer mentions them, for every agent. Claude memory runs now record memory_injected_chars so a silently-failed reflect can't read as 'memory didn't help'.
… workaround dropped The claude memory arm now gets memory exactly as production does: the plugin's UserPromptSubmit hook (claude-hook.js) wired into the container's settings.json the same way the installer wires a real machine, config in ~/.hindsight/coding-agent.json, retention/seed/survey off for trial isolation. The old harness-side reflect + --append-system-prompt path (and its _reflect_query mirror) is deleted — it predated the calibrated <hindsight_memory> wrapper and historian reflect, which is what made hook-injected memory usable by claude in the first place. memory_diag now collected for claude too; verified live: reflect_ok from harness claude-code in-container, full policy injected, task solved.
…the coding benchmark The coding path no longer bypasses the framework: - SdebenchDataset declares isolation_unit='task' and load_documents exposes each task's real corpus (decision chats / decision commit + decoy conversations + host-history noise, user_id = task_id) — ingestion runs through the runner's standard per-unit pipeline with ingestion_time_ms reported. - HsCodingProvider is a real MemoryProvider (own module): ingest = the plugin's deepen engine over the built repo + chats, synced-polling included; prepare() handles fresh-trial bank resets; --skip-ingestion replaces SDE_HSCODING_REUSE_BANK (still honored). - CodingMode dispatches generically: none -> vanilla; hscoding -> agent-side plugin (bank passthrough only); ANY other registered provider -> retrieve(bug_report) -> memories injected via the provided arm (run.py --external-memory). - History task.jsons now self-describe their decision commit (dataset change), so generic providers ingest it without building repos. Verified end-to-end: bm25 (generic) ingested 241 docs and solved slalog with 0 corrections through the standard pipeline; hscoding via --skip-ingestion green; dataset-stats/splits/--query-id all work.
…ng runs
- coding runs are graded by pytest: judge_llm is no longer recorded for them
(and the run page says 'Graded by pytest' instead of showing a judge model)
- dataset table shows the agent's MODEL alongside the agent
- run detail now renders WHAT HAPPENED: the coding mode lifts the harness's
trace.json into the saved rows (flattened step trajectory with 🔁 feedback
separators, final patch as the answer, repo git history, and — on memory
arms — the injected reflect blocks from memory_diag), and coding summaries
declare view:'agent' so the existing multi-turn renderer picks them up
- claude runs now use --output-format stream-json --verbose so the trajectory
carries real tool steps (plain json returned only the final message)
- demo runs with full traces committed (trace-demo-{none,hs} on
boltons-budget-history-001)
…oc, Recall p50, Ctx tokens) on coding runs — they don't apply and showed garbage
--memory vanilla (baseline) and --memory hindsight-coding are the canonical
CLI names ('none'/'hscoding' stay as hidden legacy aliases so existing scripts
keep working; 'hindsight' proper was already taken by the QA provider). UI arm
badges and docs follow.
…ame fix (submodule bump)
…arness README The dataset repo's README now covers the whole story incl. how to run; the in-PR harness README duplicated it. Charts were internal-doc tooling, not benchmark infrastructure.
…rovider was silently ingesting ZERO decoys The provider forwards only Documents with messages to the deepen engine; decoy docs had content but no messages, so every bank ingested 1 chat instead of 141 and the retrieval-noise model silently vanished. Caught in pre-campaign verification (a 75s single-chat ingest where minutes were expected).
A failed build.py (e.g. SDEBENCH_BOLTONS_HOST unset — now required) was silently swallowed; deepen then ran against a missing repo, ingested chats, and polled for a gitlog that could never appear until the sync deadline.
gpt-5.1-codex-mini 404s on /v1/responses for plain API keys as of Aug 2026 (codex-branded models gated); the whole overnight codex block silently produced capped-and-unsolved garbage because _parse_codex swallowed turn.failed/error events — it now raises like the claude is_error guard. Pricing updated (0.75/4.50, cached 90% off).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sdebench: does memory help a coding agent?
A coding benchmark where each task is a bug-fix on a real codebase (boltons) whose correct solution hinges on a non-guessable, project-specific decision — the obvious fix passes the visible repro but fails a held-out hidden test. The decision lives in git history, a past developer chat, or a chat later amended (testing cross-chat consolidation). Memory-system-agnostic: any system that ingests git rationale + chats can be plugged in.
Dataset: the sde-bench submodule — 43 tasks (19 conversation / 4 conversation-amended / 20 history; 9 real-function / 34 planted; 7 decision categories), every task machine-validated to discriminate (HEAD fails repro+hidden; correct passes all; each plausible naive fix passes repro but fails hidden).
Agents:
opencode(gemini-3.5-flash),claude-code(sonnet-5),codex(gpt-5.1-codex-mini) — same task, same grading, per-agent memory delivery (plugin / system-prompt / codex hooks). Vanilla-arm fairness: past chats are reachable (seeded sessions or transcripts), never injected.Primary metric: corrections — failing grades feed the pytest tail back and resume (cap 5); 0 = solved first try. Grading in Docker from pristine copies (agent test-edits ignored).
Committed results (
outputs/sdebench/dz-*): the 33-task campaign — Claude Sonnet n=5, OpenCode/Gemini n=3, vanilla vs memory (hindsight-coding-agents). The 43-task set + codex arms are the next campaign.In this PR
sdebench/harness (harness/run.py), Dockerfiles (grading + 3 agent images)sde-benchdataset submodule pinned to itsmain(docs + HF mirror: https://huggingface.co/datasets/vectorize-io/sde-bench)codingmode (dataset/mode/provider/runner/server), results viewer support