Collapse byte-identical turn hits at recall time

[ericwalisko]

Problem

The work laptop found a second duplicate source the schema-v2 skip-prefix sweep can't catch. That sweep only removes harness-injected boilerplate (Base directory for this skill:, the local-commands caveat). But a user's own scheduled-automation prompts — a nightly cron's system prompt captured identically every day — are real turns, so no static prefix list will ever match them. Observed: 3 of 8 MCP recall slots going to byte-identical copies of one string; one preamble repeated 11× across the store.

prune --glob would delete those sessions' real findings along with the preambles, so it's the wrong tool.

Fix

Widen the existing "collapse to one hit per turn" to "collapse turns whose quoted text is byte-identical", keeping the highest-ranked copy. Applied in both the single-query path (search_turns) and the multi-query path (after RRF fusion, since RRF keys on (kind, id) and wouldn't dedup distinct turn ids).

Properties:

• Recall-time view only — no turn is deleted. Sessions still read back whole; distinct findings inside those automation sessions still surface on their own text.
• Deterministic, model-free — byte-identical match, no similarity thresholds. Consistent with memware's no-model-in-the-loop design.
• Opt out with collapse_duplicates=False.

Tests

• New test_recall_collapses_byte_identical_turns_but_keeps_distinct_findings: three identical cron prompts collapse to one slot, a distinct finding sharing vocabulary still surfaces, opt-out returns the copies, and the store still holds all four turns.
• Two existing tests seeded byte-identical text across sessions purely as a convenience (recency ordering; session separation). Made their seeds distinct so they test what they intend without tripping the new collapse.

Full suite, ruff, ruff format, mypy all green locally.

🤖 Generated with Claude Code