blocklog: look a block up by index, 51us down to 0.5us on a full log

[gerardrecinto]

Profiling a blocked execute_step with memory on put blocklog.Summary at about a quarter of the time. It scanned every kept entry (up to 10,000) under the log's mutex, on the first block of each step in a run, so the cost grew with the log and held up every other recorder.

The log now keeps, per (workflow, version, step, rule, missing state), the sequence numbers of its block entries in order. Summary reads only those. Trimming the oldest entry pops it from the front of its list, so nothing stale is left behind, and the result is the same aggregation in the same order as before.

Same machine, full 10,000-entry log: Summary 51,015 ns to 493 ns (0 allocs both ways). The whole memory-on blocked execute_step through the in-process MCP client went from about 159 us to 127 us per call; the rest of that is the MCP library's JSON handling, which this does not touch.

TestSummaryIndexAgreesWithAFullScan records 600 random blocks, runs and recoveries through trims and expiry with a small cap and a short TTL, and after every step checks all lookups against the old scan (kept in the test file as the reference, and as BenchmarkSummaryByScan for comparison). It also checks the index lists exactly the block entries still held. go test -race passes for blocklog, mcpserver, cmd/sop-mcp-server and verify.

Not changed: the barrier itself (43 ns allowed, 350 ns blocked, no allocations on the allowed path) is already tight, and Summaries and Stats still scan, since they are called on connect and on read_lessons, not per block.

Thanks, Gerard Recinto

[github-actions]

Gemini PR Review

Reviewed commit: 49b3e6c70d1077aac528df6d149a71cb0cdb9fb8
Verdict: PASS

• Correctness: In tools/blocklog/blocklog.go, the sumKey struct, Entry.sumKey() method, Log.base field, Log.bySum map, and the updated add, trim, and Summary methods introduce complex logic for an in-memory index. The correctness of the index's consistency with the entries slice, especially during trimming and sequence number (base) adjustments, is crucial. However, the new TestSummaryIndexAgreesWithAFullScan in tools/blocklog/blocklog_test.go provides comprehensive validation by comparing the results of the indexed Summary with a full scan (summaryByScan) across various operations, including trims and TTL expiry, and verifies the count of indexed entries. This extensive testing adequately covers potential correctness bugs and edge cases related to the indexing logic.