Comprehensive v0.23 benchmark: indexing at scale, Redis reads, reranked search

[phernandez]

Why

The single-metric LoCoMo retrieval comparison (run artifacts v0221-baseline-long-r1 vs
main-rerank-*-long-r1, discussed on #1234/#951) measures one dimension — sequential
single-client query latency — and misses the dimensions v0.23 was actually built for. On
that narrow metric v0.22.1 looks faster (122ms vs 256ms mean, rerank off); but v0.22.1
under real concurrent load deadlocks on the relation table until jobs terminally fail
(#1213, 16 Logfire fingerprints), silently accumulates duplicate observations (37%
redundancy in a day in the #1214 field report), and misranks the queries reranking now
fixes. The v0.23 performance story is composite: correct, scalable indexing + cached
reads + reranked search, together. This issue defines the benchmark that measures that
story, v0.22.1 vs v0.23, and produces the numbers for release notes.

Benchmark axes

1. Indexing at scale

• Large corpus (thousands of notes; scale up the existing generators or replicate the
  moby-dick corpus Nx) with N concurrent writers (the Generation-versioned observation persistence (companion to relation deadlock fix) #1214 field-report shape: 4-5
  concurrent MCP clients writing notes with overlapping entities), SQLite and Postgres.
• Measure per version: index throughput (notes/min), terminal failures/deadlocks
  (v0.22.1-Postgres expected to reproduce the Generation-versioned relation persistence: eliminate indexing deadlocks on the relation table #1213 cluster), duplicate
  observation/relation rows after the run (v0.22.1 expected to accumulate; v0.23 must be
  zero — the generation-fence guarantee), full reindex wall time.

2. Redis read cache

• v0.23 with and without the basic-memory[redis] extra (perf(api): add optional Redis read caching #1168/perf(api): add Redis-cached QUERY and MCP reads #1172, Upstream a BMQ3-style read cache: MCP tool latency is product-defining for multi-agent use #980): hot MCP
  read paths (read_note, search_notes, build_context) under repeated agent-style read
  load; p50/p95 per tool, cache hit rate.

3. Reranked search

• Quality + latency envelope, already partially measured (Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 results comment, Tune reranker document-length cap for long-document corpora #1234
  sweep): fold those numbers in — R@5 +5.0pp / MRR +9.7pp over v0.23-off; MRR 0.616→0.719
  (+17% relative) over the v0.22.1 baseline; ~90ms P50 overhead on short notes with the
  2000-char cap bounding long documents.

4. Composite "multi-agent day" scenario

• Concurrent writers + readers + searchers against one project for a sustained window,
  v0.22.1 vs v0.23 (same hardware, isolated homes, worktree + --bm-local-path workflow).
• The claim to validate: v0.23 sustains concurrent multi-agent load with zero deadlocks
  and zero data duplication at comparable-or-better throughput, while ranking results
  meaningfully better.

Also resolve while in here

• Contextualize the sequential-baseline delta (v0.22.1 122ms → v0.23 256ms mean, rerank
  off): how much is full-content FTS (fix(core): search complete SQLite note content #1071 — indexes all text instead of truncating at
  ~6KB), generation-fence candidate over-fetch, vs run noise. Decide whether any of it is
  reclaimable or all justified-and-accepted.

Tooling

basic-memory-benchmarks (bm-bench) + worktree/--bm-local-path revision comparison.
Needs: a concurrent-writer driver, a redis on/off toggle, indexing-throughput and
row-integrity metrics (duplicate-tuple counts post-run). The run-rerank-comparison.sh /
run-cap-sweep.sh scripts in that repo are the single-axis precedents.

Deliverable

A results write-up (issue comment + Basic Memory note) with the v0.22.1-vs-v0.23 table
per axis — the source of truth for release-notes/blog performance claims. Changelog perf
claims in #1244 intentionally wait on these numbers.

[claude]

Claude finished @phernandez's task in 45s —— View job

---

Triage

Type: Enhancement — this defines a benchmarking effort (not a code defect or feature request against the product itself), applied as enhancement per the triage mapping.

Priority: Medium-High — no user-facing breakage, but the changelog/release-notes performance claims for v0.23 (#1244) are explicitly blocked on these numbers landing, so it's on the critical path for the release.

Component: Spans indexing/sync (SQLite + Postgres), MCP tools (read_note, search_notes, build_context), the Redis read-cache extra, and reranked search — plus benchmark tooling in the separate basic-memory-benchmarks repo (bm-bench). Not a Cloud-product-behavior issue, so no cloud label applied.

Complexity: Complex — four benchmark axes (indexing at scale, Redis cache, reranked search, composite multi-agent scenario) plus new tooling (concurrent-writer driver, redis on/off toggle, duplicate-tuple integrity metrics) that doesn't exist yet, across two repos.

Duplicate check: No existing open issue duplicates this. Related but distinct: #951 (entity-aware ranking / LoCoMo results, referenced here as an input), #1243 (v0.23 release doc review), #1244 (changelog, downstream consumer of this issue's numbers), #1213/#1214 (the deadlock/duplication incidents this benchmark is meant to validate are fixed).

Labels applied: enhancement.