release: v2.0.0 — evidence memory (includes unreleased 1.10.0) - #28
Merged
Merged
Conversation
A measured research spike on whether an agent can be given a computed set of necessary repository knowledge rather than similar data. 420 git-derived queries across eight repositories, paired bootstrap and permutation tests, leave-one-repository-out fitting, and a no-lookahead co-change model. Retrieval plane refuted: anchor-and-expand lost MRR -0.0073 (p<0.001). Recovered gold per completion slot (0.0310) is below the cost of evicting the cheapest baseline slot (0.0347). Appending contract slices instead beat a page-matched baseline by +0.0056 useful@budget for +6.4% tokens, i.e. lower useful-per-token. The LORO weight fit did not generalise (held-out -0.0024). Memory plane validated: span-keyed evidence survives 0.795/0.521/0.309 of reuse after 1/10/20 commits, where a query-keyed cache would serve 20.5%, 47.9% and 69.1% stale entries. 42.0% of evidence tokens repeat across the workload. Both mechanisms have close prior art; the contribution is the measurements. Generated data and private corpora stay git-ignored.
Phase B design, written before implementation: content-addressed evidence references validated against the working tree, explicit session-scoped withholding with unchanged page composition, a separate content-free memory.sqlite, and the sequential real-history benchmark that decides whether withholding is worth shipping.
Evidence validation reads files long after they were indexed. Any path the walker refuses (ignore rules, dependency and build directories, secret filenames, size, NUL sniff) must be refused identically there, so both now go through one PathGate. The walker keeps its exact order and semantics; a parity test checks that the gate admits precisely the files a walk indexes. PathGate.read also re-gates the resolved on-disk path, so a symlink or a differently-cased path on a case-insensitive filesystem cannot reach a file the walker would never have indexed.
An evidence reference is path:start-end@hash, where the hash covers the exact bytes of the span under the indexer's own line model (universal newlines, splitlines). Only line terminators are normalised. Validity is byte identity with the working tree: valid, relocated (identical bytes occur exactly once elsewhere in the same file), or changed, ambiguous, deleted, excluded and unreadable, all of which are invalid. References are untrusted input, so absolute paths, drive letters, .. segments and NUL are rejected before any filesystem access. memory.sqlite is separate from index.sqlite because rebuilds and clean delete the index file, and so that ledger writes never queue behind an update transaction. It stores hashes, paths, line numbers, token counts and timestamps only. Session tags are stored hashed. Migrations run in single transactions, a newer schema is refused without touching the file, a corrupt file is moved aside rather than deleted, lock contention degrades instead of blocking, and GC only removes rows.
…ssion search() gains an optional evidence hook that runs after ranking, budgeting and pagination are final, so memory cannot change which results are returned or which carry snippets. Without a session the packet is unchanged except for stale: true on a result whose index text no longer matches the working tree. With an explicit session tag, a snippet the session already received from byte-identical source is replaced by snippet: null, reused: true (a skeleton is only withheld when the same skeleton or the whole span was delivered), and evidence the session received earlier that has since changed is reported once under memory.invalidated. Budget accounting is untouched: filling withheld snippets back in reproduces the no-memory packet exactly, which the tests assert. CBX_MEMORY=0 or memory.enabled=false restores 1.10.0 output. Sessions are never inferred from a process, an environment variable or a time window, because none of those identify what an agent still holds in context.
- search/explain --session TAG and the MCP session parameter name one agent context for evidence reuse; malformed tags are rejected before any work. - verify [REF ...] [--session TAG] [--strict] and MCP verify_evidence re-check evidence against the working tree. Read-only and index-free; references are untrusted input. all_valid is false for an empty or unknown set. - memory gc and memory clear (with confirmation) are CLI-only maintenance, like clean: neither the skill wrappers nor MCP expose them. - stats, index_stats and healthcheck gain an additive memory block (counts, schema, size; no paths or content). doctor probes memory.sqlite read-only and reports corruption or a newer schema without repairing anything. - Markdown marks withheld snippets, stale results and invalidated evidence. MCP schema_version stays 1: every payload change is an added field. The three affected goldens change only by that block.
verify joins the wrapper whitelists; memory (gc/clear) stays excluded together with clean, init and watch, and the packaging test asserts it. The plugin wrappers in bin/ had drifted behind the skill template and were also missing architecture, diff-impact, path and describe, which the plugin skill tells agents to run.
Real git drives every case: unrelated commits, branch switch and return, detached HEAD, worktrees as separate scopes, dirty/staged/unstaged edits, renames, deletions, new files, rebase and CRLF checkouts. Excluded content (.env, secrets.json, node_modules, ignored paths, a file ignored after it was indexed) never reaches memory.sqlite and verifies as excluded. The signature test replays T0 learn a fact, T1 unrelated commit, T2 the fact changes, T3 a similar question: a query-keyed cache serves the stale fact, evidence memory reports the change and delivers the current text.
The benchmark replay flagged results as stale right after the index was
updated. Every sampled case was index text that is derived rather than copied:
config-key chunks ("config key: scripts.build = ...") stored against a "{"
line, and markdown section summaries ("### Heading: body"). No byte comparison
can match those, so they were false alarms.
A mismatch between index text and the working tree now means stale only when
the file's current sha256 differs from files.sha256, the fingerprint the
index was built from. When the index is current for the file, the excerpt is
derived text: it is neither flagged, recorded, nor withheld.
The fixture was built by the 1.10.0 release code (abb67df) over tests/fixtures/sample_repo, not synthesised by 2.0. The index schema stays at version 3, so an upgraded project keeps its index: search works, update reparses nothing and built_at survives, a 1.x config.json without a memory section loads with defaults, memory.sqlite attaches on the first session, and clean removes the index while memory survives.
Replays each corpus's own history in a shared clone (sources are never modified): for every git-derived query the tree is checked out at the parent of the query's commit, the index is updated incrementally, and one retrieval call is made. Consecutive tasks form sessions of K tasks, so the repository really evolves inside a session. All arms share one packet per task: A (reread whole files), A-mem (skip unchanged files), B (1.10.0 packet), S (withhold by locator without checking the source) and C (2.0 evidence memory). The oracle is independent of memory's hashing: a withheld snippet is stale unless the session was handed that exact text or the whole current span, and notices are scored against a list-comparison re-implementation of span presence. Every task also checks that restoring withheld snippets reproduces the B packet exactly.
os.walk lists symlinks lexically under the root, but _rel resolved them before relative_to(root), so any link pointing outside the repository (Bazel convenience symlinks being the common case) raised ValueError before a single file was indexed. Resolve, and skip entries that land outside the root, consistent with PathGate.read. Closes #27
…gent wrappers The skill and scaffolded AGENTS.md blocks now route every search and explain through one session tag per conversation, add a Verify step to the Find/Trace/Predict loop, and document reused and memory.invalidated payload fields in a new references/memory.md.
Conflicts resolved: keep verify in the plugin wrappers, keep both the ranking-diagnostics pool and the bounded read plan in the retrieval pipeline, thread max_read_lines through the evidence-aware search path, take the rewritten README/ARCHITECTURE/RETRIEVAL_PIPELINE from main and merge both unreleased changelog sections.
Version 2.0.0 across the package, plugin manifest, lock pin and skill stamps. Changelog section for 2.0.0 built from the evidence-memory commits, the unreleased 1.10.0 ranking work and the community-readiness items from #26. New docs/MEMORY.md; verify_evidence and session in docs/MCP.md; Verify added to the roadmap's shipped foundation; the logged evidence-memory benchmark run under tests/eval/results with its summary in docs/BENCHMARKS.md. Supported line in SECURITY/FAQ/README is 2.0.x.
denfry
marked this pull request as ready for review
September 14, 2026 08:40
tests/fixtures/sample_repo had no line-ending rule for web/app.ts, so a Windows clone with core.autocrlf=true checked it out with CRLF while Linux and macOS got LF. The fixture index had been built from the Windows bytes, so on CI update saw a different sha256 and reparsed the file (indexed == 1). -text on the fixture keeps checkout bytes equal to the repository bytes on every OS, and the index is rebuilt by the 1.10.0 release code (abb67df) over those bytes. Chunks, edges, symbols and FTS content are unchanged; only web/app.ts's recorded size and hash, internal row ids and the root-dependent config_hash (written, never read) differ.
No v1.10.0 tag or PyPI release exists, so the 1.10.0 compare link and the Unreleased link pointed at a missing tag. Point 1.10.0 at its release commit and Unreleased at v2.0.0, and say in the 2.0.0 section, which becomes the GitHub release body, that 2.0.0 includes the 1.10.0 work.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release line for 2.0.0: evidence memory.
Every delivered snippet is identified by the exact bytes of its span (
path:start-end@hash), can be re-checked withverify/ MCPverify_evidence, is withheld from a session that already holds it (--session TAG), and is reported undermemory.invalidatedwhen it changes. Content-freememory.sqlite;CBX_MEMORY=0restores 1.10.0 output byte for byte. MCPschema_versionstays 1.Also carries the unreleased 1.10.0 ranking work and the community-readiness merge (#26). Closes #27.
Benchmark: replaying this repository's history, zero stale snippets withheld at every session length (the locator-only alternative withholds 15–54), change-notice precision 1.000, snippet tokens −4…−11%. Logged run in
tests/eval/results/2026-09-14-evidence-memory.md.Release checklist: version sync, ruff, mypy, pytest (86% coverage),
doctor --strict,release_smoke.py, skill-copy sync, link and version gates all pass locally.