Skip to content

fix: memory system bugs + end-to-end validation suite - #14

Merged
emanuilo merged 26 commits into
mainfrom
fix/memory-review
Oct 18, 2026
Merged

emanuilo merged 26 commits into
mainfrom
fix/memory-review

Conversation

@emanuilo

@emanuilo emanuilo commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

I reviewed the whole memory path (store → index → search → context → consolidation → tools), fixed the bugs listed below, and added an end-to-end suite that runs against the real OpenAI and Claude APIs in a throwaway vault.

 search(query)
-  fts MATCH <raw query>                # "What's…?" → syntax error → no keyword hits
+  fts MATCH "word" OR "word" …         # quoted, stopwords dropped
-  keyword = 1 - |rank| / max           # best BM25 match scored 0
+  keyword = |rank| / max               # best match scores 1
-  matrix cache keyed on COUNT(*)       # same-count re-index → stale vectors, dropped hits
+  matrix cache keyed on (COUNT, MAX(id))
-  merge → truncate → file_filter
+  merge + file_filter → truncate
 consolidate()
-  files = all unconsolidated (incl. today)
+  files = unconsolidated before today              # today's later saves were never consolidated
-  mark through files[-1] even if cut by 30k budget # rest of the files were skipped for good
+  mark through the last file actually sent
   result = backend.run_one_shot(...)
-  MEMORY.md = result                               # wiped permanent saves made meanwhile
+  .history/MEMORY-<time>.md = old; MEMORY.md = unfence(result) + appended-since
+  (manual edit meanwhile → abort, retry later)

MEMORY.md is redesigned as a profile plus an index. The old prompt asked for under 5,000 characters, had no date, and said "remove outdated entries". On real data it swapped saved credentials for placeholders ("see prior backup") and gave the user's birthday to their partner. Copying everything verbatim instead turned MEMORY.md into a 14 KB dump of the daily notes.

 MEMORY.md
-  every detail copied in (secrets, receipts, instructions)
+  ## About / People / Work & Projects / Health / Finances / Open threads   # summaries
+  ## Index
+  - Lion8 prod login and SSH commands → memory/2026-05-12.md, memory/2026-06-13.md
 consolidate()
+  input: today's date + list of other notes (todos.md, bekele.md, …)
+  rules: reference details, never copy secrets; never a placeholder; later entry and
+         "ISPRAVKA" corrections win; facts stay with the right person; drop expired
+         one-offs, links, assistant instructions
+  check_references(): drop refs to missing notes, log refs whose note shares no word
 system prompt
+  "→ file" lines are references: open the note with file_read for exact details
 memory_save
-  permanent=true → appended raw to MEMORY.md
+  today's notes only; MEMORY.md has one writer: consolidation   # daily log = complete record
 consolidation
-  after 7 unconsolidated days, Sonnet (chat model), MEMORY.md.bak
+  after each finished day (threshold 1, after the next reply), Opus 5.5 medium
+  (CLAUDE_CONSOLIDATION_MODEL / _EFFORT), dated copies in .history/ (last 60)
 budget per call: 30k → 80k chars

Other fixes:

Area Bug Fix
Index AGENTS.md indexed and returned as a "memory" index *.md minus AGENTS.md and hidden dirs
Index emptied file kept its chunks; failed embed lost them embed first, swap chunks in one transaction
Index entry chunks started with the previous entry's ## HH:MM heading --- ends a chunk with no carry-over
Index restored file with older mtime never re-indexed sync on mtime !=
Agent concurrent Slack chats shared chat_id / found images (reminder to wrong chat) turns serialized with a lock
Agent OpenAI outage failed every message (context search) and the bot start (sync) log and continue without memories
Tools memory_save reported an error after saving if indexing failed (agent retries → duplicates) reports saved, sync indexes later
Tools image_search returned other platforms' images (Telegram can't send a WhatsApp ref) scoped to the current platform
Tools file_write/file_read could hit memclaw.db, .env, sessions.json, .cursor/ hooks blocked
Tools .md files from file_write weren't searchable until a sync indexed on write
Stats image count only counted Telegram counts platform_images
Images a score threshold sent the ID card for "my passport", and two searches (Serbian + English) sent the passport twice image_search only lists candidates; the agent picks with image_send; each image sent once per reply
Images image_save reported an error after writing the note when embedding failed (retry → duplicate) image registered before the note is written
Tools file_write could replace MEMORY.md or a daily note refused (reading still allowed)
Consolidation a persistent failure (bad model id) retried after every reply retried an hour later
Consolidation default effort passed to any CLAUDE_CONSOLIDATION_MODEL, even one without effort default effort only on the default model

Code structure: _maybe_consolidate (108 → 60 lines) reads as its steps: _read_consolidated_through, _daily_text (files within the budget), _consolidation_message, _replace_memory (race check, history copy, write), _write_consolidated_through.

Validation:

tests/
├── conftest.py          # autouse fake embedder: unit tests never call OpenAI
└── e2e/                 # MEMCLAW_E2E=1 pytest tests/e2e  (real APIs, temp vault)
    ├── test_retrieval_e2e.py   # 11 tests: recall, keyword, freshness, scope, guardrails
    └── test_agent_e2e.py       # 9 tests: save→/new→recall, fact reaches MEMORY.md after the day,
                                #   restart resume, instructions, OpenAI outage, real consolidation,
                                #   prompt quality (refs not secrets, refs resolve, corrections, attribution),
                                #   agent follows a reference with search down, "my passport" → only the passport
.github/workflows/e2e.yml       # manual workflow_dispatch, needs OPENAI/ANTHROPIC secrets

Evidence

  • Before (E2E on the parent commit, real APIs): 12 of 17 failed

    keyword_channel_handles_natural_language   assert []            # "What's…?" → nothing
    keyword_best_match_scores_highest          0.0 == 0.2399        # best match scored lowest
    rewritten_file_is_searchable_with_warm_cache  penicillin not found
    agents_md_is_not_a_memory                  AGENTS.md returned
    emptied_file_drops_its_chunks              sunflower-42 still found
    chunks_do_not_carry_previous_entry_heading 2 == 1
    image_search_is_scoped_to_platform         {'telegram','whatsapp'}
    file_write_markdown_is_searchable          passport not found
    file_write_cannot_clobber_internal_files   "File written: …/memclaw.db"
    context_survives_embedding_failure         AuthenticationError 401
    turn_survives_openai_outage                AuthenticationError 401
    consolidation_with_real_llm                "a save made during consolidation was lost"
    

    After: 20 passed (claude-haiku-5-5) + 376 passed, 20 skipped unit tests, ruff clean.
    test_consolidation_prompt_quality failed once in ~10 runs on Haiku (production consolidation uses Opus).

  • Image retrieval (production log, "Daj mi sliku pasosa"):

    before  image_search ×2 → 3 images sent: passport, ID card printout, passport again
    after   image_search ×2 → image_send([26]) → 1 image: the passport
    
  • Consolidation prompt (real model):

    old prompt   FAILED x2   API key → "not copied here (sensitive)"; saved link kept
    new prompt   passed x3   secret referenced (→ memory/…md), not copied; all refs resolve
    
  • Production rebuild (77 daily files, Opus 5.5):

    old MEMORY.md verbatim rebuild profile + index
    size 6.2 KB 14.4 KB 7.8 KB
    placeholders instead of values 4 0 0
    secret values in the system prompt ~20 22 0
    secrets reachable via a reference — — 32 / 32
    references (dropped by check) — — 106 (0)

Merge Danger

Door: two-way

Revert restores the old behavior. Data written in the meantime stays compatible: same DB schema, same file formats. .history/ and credentials.md are just extra files.

Blast Radius: memory

  • The system-prompt template changed, so existing chat sessions start fresh once (memories are kept).
  • Turns are now serialized across chats. A slow turn in one Slack channel delays another. That's fine for a personal bot, but it is a throughput change.
  • Existing chunks keep the old chunking until their file changes. The first sync removes any AGENTS.md chunks.
  • memory_save no longer takes permanent, and memclaw save --permanent is gone. An old session that still passes the flag gets a normal daily save.
  • Stacked on feat: persistent Claude sessions, /new, /model, /effort #12 (feat/claude-persistent-sessions). Merge that first. feat: native Cursor conversations, /model and /effort for Cursor #13 conflicts with this PR only in docs/memory-architecture.md.

🤖 Generated with Claude Code

- Reuse one Claude CLI client per chat instead of spawning one per turn
- Resume conversations after restart (sessions.json), fresh session on failure
- Replace the rolling history window with native sessions; /new resets
- Static system prompt; time and memory context move into the user message
- Load no ~/.claude settings/plugins/MCP servers, no built-in tools, dontAsk
- Telegram /model and /effort, persisted to ~/.memclaw/.env
- Background consolidation after the reply
- Default to claude-sonnet-5-5 at medium effort; bump claude-agent-sdk
…n suite

Search: build a safe OR-ed FTS5 query (punctuation no longer kills keyword
search), fix inverted BM25 normalization, invalidate the embedding matrix
cache on any re-index, apply file_filter before truncation.

Index: skip AGENTS.md and hidden dirs, drop chunks of emptied files, embed
before swapping chunks, re-sync on any mtime change, one chunk per entry
without the previous entry's heading, count images from all platforms.

Consolidation: never mark today's file done, only mark files that fit the
budget, keep permanent memories saved while it runs, strip code fences,
back up MEMORY.md.

Agent/tools: serialize turns so concurrent chats don't share tool state,
survive OpenAI outages in context search and startup sync, report saves as
saved when only indexing failed, scope image_search to the platform, block
file tools from Memclaw's internal files, index .md files from file_write.

Tests: offline fake embedder for unit tests; tests/e2e runs against the
real OpenAI and Claude APIs in a temp vault (MEMCLAW_E2E=1).
…gged daily

The old prompt replaced saved credentials with placeholders ("see backup")
to meet a 5,000-char target, had no notion of today's date, and mixed up
facts between people. Rewritten with explicit rules for exact values,
corrections, attribution and what to leave out; told today's date; budget
raised to 80k chars. memory_save(permanent=true) now also appends to the
daily file so MEMORY.md can always be rebuilt from the daily log.
Consolidation now writes a holistic profile and an index of references
(→ memory/YYYY-MM-DD.md) instead of copying every detail; secrets and
document numbers stay in the notes. References are checked before saving
(missing notes dropped, unconfirmed ones logged), the consolidation input
lists the vault's other notes, and the system prompt tells the agent to
open a referenced note with file_read when it needs a detail.
Consolidation (the only one-shot call) gets its own model and effort,
CLAUDE_CONSOLIDATION_MODEL / CLAUDE_CONSOLIDATION_EFFORT, independent of
the chat model. Shown in memclaw status.
memory_save loses its permanent option (and memclaw save --permanent): the
agent only writes today's notes and consolidation is MEMORY.md's single
writer. consolidation_threshold defaults to 1, so it runs after the first
reply once a day with notes is over. Each consolidation keeps a dated copy
of the previous MEMORY.md in .history/ (last 60) instead of MEMORY.md.bak.
- file_write refuses MEMORY.md and the daily notes (read stays allowed):
  consolidation is MEMORY.md's only writer and the daily notes are the
  append-only record it is rebuilt from
- consolidation effort defaults only on the default model, as for chat
- a failed consolidation is retried an hour later, not after every reply
- image_save registers the image before writing its note, so a failed
  embedding doesn't leave a note behind that a retry duplicates
- file_write indexes under the path the index sync uses
image_search now returns the closest images with ids and descriptions and
sends nothing; the agent calls the new image_send with the ids that match
the request. A similarity threshold can't tell a passport from an ID card
when both mention the user's name, and two searches (Serbian + English)
sent the passport twice. image_send queues each image at most once per
reply.
- /new forgets the session before awaiting the backend reset, so a
  message sent right after it can't resume the old session
- /model refuses to switch when the model list can't be loaded, instead
  of saving an unchecked (possibly mistyped) id that breaks every turn
- the fallback cost estimate includes cache-write tokens
- long-id model buttons carry a hash of the id instead of a list
  position, so they can't pick a different model if the list changed
- drain the bridge's stderr in chunks: readline() raised on a line over
  64 KiB, ending the drain, after which a full pipe would block the bridge
- a timeout no longer counts as a dead bridge, so one slow run doesn't
  drop the shared bridge and abort other chats' runs
- relaunch a bridge that exited between calls instead of failing the next
  message (or /model) first
- shutdown iterates over a copy of the sessions, so a message arriving
  mid-shutdown can't skip closing the bridge and MCP server
The language-learning note sat at 90 days old, where temporal decay keeps
about 12% of its score, so a vague query put it right at the top-3 edge
and the test flipped with small embedding changes (e.g. weekday headers).
- one builder for the locked-down CLI options, shared by chat sessions and
  one-shots, so the lockdown can't drift between them
- turn cost computed in _turn_cost, out of run_turn's connection handling
- agent.handle's logging, pending-note acknowledgement and session saving
  pulled into named helpers
- clear the repo's ruff findings (unused imports, ambiguous names)
- model/effort state, model-list lookups, the strict model check and
  saving to ~/.memclaw/.env move into ModelSelectionMixin, used by both
  the Claude and Cursor backends. Cursor's /model now also refuses an id
  it can't check when the model list doesn't load.
- Cursor run_turn: _ensure_agent says whether the agent was already open,
  resumed or created (replacing two flags), and _send does the send
# Conflicts:
#	memclaw/agent.py
#	memclaw/backends/claude.py
#	memclaw/index.py
#	tests/conftest.py
Reading the consolidation marker, picking the daily files within the
budget, building the prompt and replacing MEMORY.md (with the race check
and history copy) become their own methods; meta.json is read in one
place. 108 lines → 60, behaviour unchanged.
The Cursor backend is written and checked against the 0.1.x SDK; CI
installed 1.0.38 (>=0.1.0 allowed it), which no longer exports
UnknownAgentError. Pin to <0.2 so a major SDK upgrade is a deliberate
change.
0.1.4 calls it UnknownAgentError, 0.1.9 AgentNotFoundError (both
CursorAgentError subclasses, which is what the backend catches). The
suite passes on both versions.
@emanuilo
emanuilo changed the base branch from feat/claude-persistent-sessions to main October 18, 2026 17:15
@emanuilo
emanuilo merged commit f16c356 into main Oct 18, 2026
4 checks passed
@emanuilo
emanuilo deleted the fix/memory-review branch October 18, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant