Skip to content

fix(memory): self-heal missing entry markers so hot-layer memory persists#1443

Closed
asdf8675309 wants to merge 1 commit into
danielmiessler:mainfrom
asdf8675309:fix/hot-layer-memory-markers
Closed

fix(memory): self-heal missing entry markers so hot-layer memory persists#1443
asdf8675309 wants to merge 1 commit into
danielmiessler:mainfrom
asdf8675309:fix/hot-layer-memory-markers

Conversation

@asdf8675309

Copy link
Copy Markdown

Problem

The two hot-layer memory templates — USER/PRINCIPAL/PRINCIPAL_MEMORY.md and USER/DIGITAL_ASSISTANT/DA_MEMORY.md — ship without the <!-- BEGIN ENTRIES --> / <!-- END ENTRIES --> marker pair that MemoryWriter.ts requires. On a stock install the "memory that compounds" loop silently never persists:

  • parseFile() hits its markers-missing branch and returns entries: [], so every reader (LoadMemory.hook, MemoryHealthCheck, Pulse) sees zero entries no matter how many the reviewer has written.
  • serializeFile() re-adds a lone END marker on each write, so the file accretes orphan END markers while staying unreadable — and nothing alarms (setEntries returns ok).

Reported in #1409 and #1426.

Fix

Minimal, no new abstractions:

  1. Templates — add the BEGIN/END marker pair to both shipped templates so fresh installs start well-formed.
  2. parseFile() self-heal — on a marker-less or corrupted file, recover well-formed entry lines, drop stray markers, preserve the intro text, and rebuild a single clean BEGIN/END scaffold. This also repairs files already corrupted in the field, not just fresh installs — a serialize-side BEGIN-guard alone would leave existing orphaned entries stranded outside the markers.

Verified (macOS, LifeOS v6.0.5)

  • Reproduced on a live v6.0.5 install: the shipped template had 0 BEGIN markers; after the reviewer ran, PRINCIPAL_MEMORY.md held real entries interleaved with several stray <!-- END ENTRIES --> markers and no BEGIN; LoadMemory injected (no entries yet) while read() returned 0.
  • Parser change, checked in isolation:
    • marker-less file → first serialize yields exactly one BEGIN/END pair
    • corrupted file (entries + multiple stray ENDs, no BEGIN) → parseFile recovers all N entries, serialize emits one marker pair, round-trips idempotently
  • After the fix, read() returns the correct entry count and LoadMemory injects the entries.

Manual verification steps

# Before: stock template has no markers
grep -c 'BEGIN ENTRIES' LifeOS/install/USER/PRINCIPAL/PRINCIPAL_MEMORY.md   # -> 0

# After this PR: marker pair present
grep -c 'BEGIN ENTRIES' LifeOS/install/USER/PRINCIPAL/PRINCIPAL_MEMORY.md   # -> 1

# Self-heal: on a marker-less file that already has entry lines,
# MemoryWriter read/parse now returns the entries instead of 0, and the
# next write rebuilds a single clean BEGIN/END pair (no stray markers).

Limitations

  • Verified on macOS only. The change is platform-agnostic — pure string/marker parsing, no OS-specific calls — but I have not run it on Linux.
  • No automated test added. MemoryWriter.ts's built-in smokeTest() writes to the installed ~/.claude/… path (via ALLOWED_FILES keyed to homedir()), not a repo fixture, so it can't run hermetically against the repo copy — I verified the parser logic in isolation instead. Happy to add a fixture-based test if you'd prefer one.

Fixes #1409, #1426

…ists

The two hot-layer memory templates (USER/PRINCIPAL/PRINCIPAL_MEMORY.md and
USER/DIGITAL_ASSISTANT/DA_MEMORY.md) ship without the
<!-- BEGIN ENTRIES --> / <!-- END ENTRIES --> marker pair that
MemoryWriter.ts requires. On a stock install this silently breaks the
"memory that compounds" loop:

- parseFile() hits its markers-missing branch and returns entries: [], so
  every reader (LoadMemory.hook, MemoryHealthCheck, Pulse) sees zero
  entries no matter how many the reviewer has written;
- serializeFile() re-adds a lone END marker on each write, so the file
  accretes orphan END markers while remaining unreadable, and nothing
  alarms (setEntries returns ok).

Two-layer fix:
1. Add the BEGIN/END marker pair to both shipped templates so fresh
   installs start well-formed.
2. Make parseFile() self-heal a marker-less or corrupted file: recover
   well-formed entry lines, drop stray markers, preserve the intro text,
   and rebuild a single clean BEGIN/END scaffold. This also repairs files
   already corrupted in the field, not just fresh installs.

Refs danielmiessler#1409, danielmiessler#1426

Co-authored-by: Claude <noreply@anthropic.com>
@xg-gh-25

xg-gh-25 commented Jul 7, 2026

Copy link
Copy Markdown

Problem confirmed: stock templates ship marker-less → parseFile() returns entries: [] → memory never persists, and serializeFile() orphans END markers on every write.

Production lesson we've extracted: marker-based parsing must self-heal when the template/install is incomplete. The failure mode here (silent loss of all entries + accumulating corruption) is the worst case — data is written but never read back, so the author sees success but the system has amnesia.

Why this fix is architecturally sound:

  1. Templates get markers — fresh installs start well-formed (obvious).
  2. parseFile() self-heals — on marker-less or corrupted files, recover well-formed entry lines, rebuild clean scaffold. This repairs existing broken files in the field, not just fresh installs. A serialize-side guard alone would strand existing orphaned entries forever.

The key insight: when a read/write loop depends on structural markers, the reader should be defensive (reconstruct valid structure from partial data), not just the writer (assume structure exists). Otherwise, installs that shipped broken stay broken.

Minimal-abstraction verification path:

# Before: 0 markers in shipped template
grep -c 'BEGIN ENTRIES' LifeOS/install/USER/PRINCIPAL/PRINCIPAL_MEMORY.md  # -> 0

# After: 1 marker pair present
grep -c 'BEGIN ENTRIES' LifeOS/install/USER/PRINCIPAL/PRINCIPAL_MEMORY.md  # -> 1

# Self-heal on corrupted file: reader recovers all entries, next write emits
# clean single BEGIN/END pair (no stray markers), round-trips idempotently.

Our similar pattern: we hit the same class of problem in session transcript markers — when a hook fails mid-write, the marker structure breaks, and subsequent reads fail silently. Our fix: reader always rebuilds valid structure from line-level data (entries/events) rather than trusting markers to be present. This PR does exactly that.

Tested in isolation (macOS, v6.0.5). No automated test added because smokeTest() writes to live ~/.claude/ paths (keyed to homedir()), not hermetic fixtures — but the parser change is platform-agnostic (pure string/marker logic).

Fixes #1409, #1426. Strongly recommend merge — silent memory loss is a trust-breaker for "memory that compounds."


Hit identical marker-corruption in session checkpoints. Discussion: T-MEM: Memory is the Moat + T-6SX: Six Self-X Properties

@danielmiessler

Copy link
Copy Markdown
Owner

Thanks a lot for this, @asdf8675309 — really appreciate you flagging it and taking the time to write the fix.

When I went back through the current source, this turns out to already be handled: hot-layer memory marker self-heal already shipped independently in both templates and MemoryWriter. The system has moved a fair bit since you opened this (the PAI → LifeOS rename and a few subsystem rewrites), so the gap you spotted has since been closed independently.

Going to close this one out on that basis — but genuinely grateful for the contribution. 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v6.0.0: hot-layer memory templates lack <!-- BEGIN ENTRIES --> — reads return 0 entries, writes stack orphan END markers (memory never persists)

3 participants