feat(memory): compile the memories room into a family chronicle - #89
Merged
Merged
Conversation
A Simpsons-themed Memories-room corpus (German + English) mirroring the chaotic message patterns of the real room: addressed voice memos with spoken dates, out-of-order sync bursts, mid-sentence fragment pairs, a two-voice kitchen-table dialogue, late caption replies, m.replace edits, and implicit-context texts. Audio is synthesized through the stack's local speech service; every item carries true_date/date_source ground truth so the diary compiler can be scored on date recovery, fragment joining, and context attachment. ingest.py replays the timeline into a test rig and refuses to target the production homeserver.
The step before diary-journal.md's timeline: resolve → transcribe → classify → join → date → compile, grounded in the structural probe of the real memories room. Key decisions: burst-aware date precedence with honest uncertainty, fragment joining by content not timing, dialogue labeling without v1 diarization, backfill-first idempotent compilation. Scored against the family-memories corpus ground truth.
Half the corpus was silence. The speech service answers 200 with a header-only WAV when a voice's Piper model is missing, so four of the eight memos rendered as empty files and nobody noticed. - Stop when the speech service returns a silent clip - Point both English speakers at a voice that actually resolves - Send each sync burst's members back-to-back, so the one memo whose date is meant to be unrecoverable lands inside its burst - Document what a replay cannot reproduce: it stamps everything with the day it runs, so live-timestamp dating is unscored and the burst window has to shrink
`stack memory diary` reads the whole memories room back and publishes it as month pages in the wiki, in the words it was recorded in. The room is messy on purpose: recordings sync days late, one memo arrives split across two files, a caption trails its photo. Matrix only ever stamps the time the server received an event, so anything that synced late would file under the wrong week forever. The compiler prefers a date spoken inside the recording, falls back to the timestamp for anything sent live, and refuses to guess for a recording that synced late without saying its own date -- those are marked on the page instead of given a confident wrong date. Nothing is summarised. The model is asked to read each message, never to rewrite one, so what reaches the page is what was said. Transcripts are cached by event id and shared with the bots, so a second run over a full room costs nothing.
Months now nest under their year, and each one opens by recalling what happened rather than counting what is on the page. - Root, year and month pages: a year names its months and who recorded them, a month holds the entries - Each month opens with a paragraph the local model writes from that month's entries. It may surface, never replace: the entries under it stay word for word, and the promise is made once on the front page instead of restated on every month - Only the sender is shown to the summariser, never the classifier's guess at who a memo was for, so one reading's mistake cannot become a person in the prose - Folder pages are `about.md` like every other entity here, because a folder URL renders no body in this wiki - Sort wiki pages named for a date by that date, so a year reads March, April, September rather than alphabetically
Six ways the compiler read a real room wrong, none of which the test corpus could reach. - Three memos recorded one after another were all filed as undateable. Arriving together proves nothing: a run is a sync burst only when some memo in it says aloud that it was made on a day its own timestamp disagrees with - A memo recorded at half past midnight landed on the previous day. Days are now read on the household's clock, not UTC - Videos and files posted to the room were dropped without a word - An entry that opened with a quote had its first line eaten, because every leading blockquote was treated as a reply's quoted original - A photo whose client repeats the filename showed the filename where a caption belongs - A recording that merely tailed off could swallow one made days later; halves must have arrived together to be joined
Reading one message alone could never see what it was part of, so the compiler guessed with timing heuristics and got it wrong in both directions. The reader now takes a slice of room history at once and returns the links it finds. - "The picture above is from the barbecue" now files under the picture above. It carries no Matrix relation, so nothing but reading the two together could ever have connected them - A memo is joined to the one before it because a reader saw the sentence run across the break, not because the two arrived close together. The room still checks the link: halves must be adjacent, same person, same kind - No more inventing a person out of a nickname. A conversation between two people who are both there is addressed to neither The entries stay the family's own words. The model reads and links; it never writes an entry.
New recordings now reach the wiki overnight. The curator runs the compiler on the same sweep it rebuilds the wiki with. Every run is still a full pass over the room rather than an append. It has to be: a reply to a memo from March can arrive in September, an edit can land on a year-old note, a remark can turn out to be about a photo from last spring. A compiler that walked forward from where it last stopped would file all three under today, orphaned from what they belong to. It stays cheap because what each piece cost is kept against the thing it describes, not against how far we got. A cold compile of the demo room takes 37s; a run with nothing new takes 1.3s and calls no model at all. A late reply to a March memo re-reads one message and rewrites one page. - Readings and month summaries cached; transcripts already were - A month is re-summarised when its content moves and not otherwise, so a settled month is never quietly reworded overnight - `--rebuild` reads everything again, for when the model improves - httpx instead of aiohttp, so the compiler runs in the curator too
A recording ended "the doctor says the cast comes off in four weeks". A note months later said it did. The reader tied the second to the first as a remark about it, so a memory of its own lost its date, its place in the month, and was credited as a reply nobody made. Related is not the same as subordinate. A remark now has to be about something still in view when it was written: a caption arrives while its photo is on screen, a follow-up weeks later does not. The reader is told the difference too, but the room is what checks it.
Replying to a memo is how the family adds to it or puts it right, and they do that whenever they happen to reread it. The age guard added alongside exists for links the reader inferred, never for one the family drew themselves, so a reply attaches however old its parent is and the memory keeps the date it happened on. Pinned because the two paths look alike in the code and only one of them should ever reach back.
A clip posted to the memories room read "video, 0:04" in a line of otherwise written English, because only photos and notes had a word for themselves and everything else fell through to the internal one. It also offered to "open" a video rather than watch it, and said nothing where a photo would have said it arrived without a caption. Every kind the compiler accepts now has a name, a length where it has one, and its own way of pointing back into the room.
A memo opening "Bart, today is July the fifth" transcribed as "Part", and without the name at all when the audio was quieter. The polish pass could not repair it: it is forbidden from changing words, and it should be, because the memories room holds what people said to their children. The only place to fix a name is before the audio is read. Whisper now decodes against the household's own vocabulary, taken from the people the wiki knows and the topics the ontology names. The same memo now comes back with the name in it. A reading is also kept against the words it was taken from rather than just the message. A better transcript used to leave the old reading in place, still addressed to nobody; now it re-reads itself.
Two open questions closed by building it, and the answers were not the ones the draft expected. The burst window stopped being a threshold to tune: timing alone marked an ordinary evening undateable, so a run now has to contradict itself before its timestamps are doubted. Join thresholds went away entirely. Two new ones noted: giving the polish pass the same household vocabulary whisper now decodes against, and an explicit way to re-transcribe when that vocabulary changes.
…does Two flags invented where a convention existed. A room is a positional everywhere else in this CLI: `stack messages read <room>`, `join <room>`, `send <room>`. So it is one here too, and `stack memory diary --room letters` becomes `stack memory diary letters`. Skipping a cache to redo work is `--force`, as in `stack photos import --force`. `--rebuild` appeared nowhere that ships.
The replay refused one hardcoded hostname, which put a real family's domain in a public repo and protected only that one household. Anyone else pointing it at their own server got no warning. The instance's own configuration is the honest check. Everything this sends is an invented Simpsons memory, so it runs when stack.toml says the household is the Simpsons and refuses otherwise, including when there is no stack.toml to read.
The transcription client is generic; its docstring described the parameter through one caller's data, down to a family member's name. Now it says what the parameter does: primes the decoder with terms the audio is likely to contain, biases towards them, has a small window that drops overflow, and is the only place a misheard word can be corrected because polish may not alter the word sequence.
Core config is read by every stacklet, and this comment explained a timezone setting through one of them: whose clock it is, and a voice memo recorded after midnight. It now says what the value is and the mistake it prevents.
A first compile of a populated room spends most of its runtime in whisper and printed nothing until it finished. Ctrl+C produced an asyncio traceback and discarded that run's readings. - Progress logging per phase: pagination reports a running event count; transcription reports the recording count and how many need decoding, then logs each one with its duration before the whisper call; reading logs each slice; summarising logs each month. Cached items are not logged. - Reading and summary caches are written incrementally instead of once at the end, so an interrupted run resumes from the last completed slice. - Dry runs write those caches too. They are cost caches keyed to immutable input, not output. - SIGINT exits 130 with a single line, in both the container entrypoint and the host dispatch wrapper. Progress goes to stderr; stdout still carries only the rendered pages.
A transcript cleanup call ran for 25 minutes and produced about 100k tokens from a 6-minute recording, then had to be killed by hand. The call had no token cap, so the model generated until the client timeout. That timeout is 900s with one retry, which allows 30 minutes on a host the endpoint shares with everything else. - complete() accepts max_tokens and timeout per call. - polish() caps output at twice the estimated input, minimum 256 tokens, and uses a 180s timeout. A 6-minute transcript now stops at roughly 2,500 tokens instead of 100k. A truncated reply fails the existing word check, so the caller gets the raw transcript. - The diary's reading and summary calls are capped the same way. - complete() logs a warning when a reply stops on length rather than on a stop token. - polish() logs the character offset where the words first diverge, with context from both sides. Previously it logged lengths only, which did not identify the input that caused the loop.
The previous cap was twice the estimated input. Measured over 15 real transcripts, polish changes the character count by between -1.5% and +1.0%, mean 0.994. The output is the same size as the input, so 10% headroom covers it and 100% does not buy anything. The loose part is the character-to-token estimate, not the ratio. It now assumes three characters per token rather than four. English BPE averages about four, but German compounds and accented characters tokenize denser, and a cap below a correct answer truncates every polish in that language and returns the raw transcript instead. polish() takes an optional max_tokens to override the derived cap. cleanup_budget() is public so a caller can read the default first. A 6-minute transcript is now capped at about 1,800 tokens.
…onfig Whisper hallucinates on speechless audio: repetition loops (measured 5-147x phrase repeats vs 1-2x in real speech) or CJK output from failed language detection. One such loop caused the 100k-token runaway generation on 2026-09-14. - Add Transcriber.looks_degenerate(): flags transcripts with >=3 repeats of a 6-gram or >30% CJK letters. - polish() skips degenerate input; never feeds a loop to the LLM. - Diary files degenerate recordings without text (entry + audio link remain; raw kept in the transcript record). - Whisper plist: pin --language from [core].language, add --max-context 0 and --suppress-nst. Only the combination fixed all three incident recordings. - on_start reconciles the plist by content, so existing installs pick up config changes on restart instead of requiring 'stack setup ai'.
Whisper computes segment and word confidence on every call; requesting plain json discards it. The diary needs it, voice commands do not. - Add Transcriber.transcribe_verbose(): verbose_json + word timestamps, returns text plus a compact quality record (segment metrics, words below 0.5 confidence, detected language). transcribe() is unchanged and stays on plain json — no extra cost for voice commands/notes. - Both paths share error translation via new _stt() helper. - Add Transcriber.quality_verdict(): flags transcripts where most segments fail OpenAI's reference thresholds (avg_logprob < -1.0, no_speech_prob > 0.6). Complements the n-gram gate: loops are high-confidence failures, mumble is low-confidence. - Diary stores the quality record in the transcript cache and gates on both signals. Gated recordings stay in the diary (entry + audio, no words) instead of being dropped by the no-speech skip. - Add --retranscribe: forces re-decoding after whisper config or vocabulary changes (TranscriptStore.run(force=True)). - Text messages remain verbatim; none of this touches them.
… lib STT post-processing was spread over the bot-runner and the diary CLI. Other voice consumers could not reuse it, and replacing a cleanup step required touching each consumer. - New lib/stack/ai/transcripts.py: TranscriptStore (moved from bot-runner voice.py, one-line re-export kept for the bots) plus a pass framework: TranscriptPass (name, version, apply), run_passes() with failure isolation, stale_passes() for staleness detection. - Records store an audit list per pass: name, version, model, outcome. Bumping a pass version makes affected records detectably stale, so a sweep re-runs one pass on cached raw text without re-running whisper. - gate_pass() and polish_pass() wrap the existing checks; the diary now composes them via run_passes(). Behavior unchanged. - Diary-specific logic (keep entry without words, dates, vocabulary sourcing) stays in the diary bot. Voice commands and text messages are untouched. - 16 new tests; 172 total pass.
…ings Long memos rendered as one text block; gated recordings rendered as a bare header with no explanation. - transcribe_verbose() stores a per-segment word_count. Polish keeps the word sequence, so cumulative counts map segments onto polished text without storing segment text. - New structure_pass(): inserts paragraph breaks where the pause between segments is >= 1.5s. Deterministic, no model call. Skips on word-count mismatch (polish hyphenation can shift counts). - Diary chain is now gate -> polish -> structure. - render: an empty voice entry states 'This recording could not be transcribed.' next to the audio link instead of showing nothing.
The 'never paraphrase, verbatim only' stance was introduced during implementation, not a product decision. The product goal is a chronicle/diary hybrid: right amount of detail, word-for-word quotes where words are shown as someone's own, audio links as the archival originals. - Module docstring, render_month docstring, and the front-page text now state: quoted words are unchanged from the recording; narrative renders as narrative, never as quotation; entries link to sources. - memories-pipeline.md documents the tiered page format (gist, narrative, verified quotes, audio link; full transcript stays in the transcript store).
…anscript Long entries rendered as full transcript dumps. The goal is a chronicle/diary hybrid: right amount of detail, exact quotes, audio preserved as the original. - Reading gains gist (one narrative sentence) and moments (passages the model copies from the text). Prompt and parsing extended; per-message read budget 120 -> 240 tokens. - verify_moments() accepts a claimed quote only when it matches one sentence or a consecutive run in the body (case/punctuation insensitive) and returns the body's own text. Invented quotes drop. - Entries >= 120 words with a gist render distilled: gist, quote callouts, full transcript in a folded block, audio link unchanged. Shorter entries stay verbatim. - Cached readings without a moments key are read again once (pre-distillation cache entries). - 7 new tests; 2683 pass.
Bounded preview for dry runs: pagination stops early, so a limited run also skips the full-history fetch.
stack down unloads the LaunchAgent and RunAtLoad only fires at login, so every down/up cycle left whisper dead until reboot (pre-existing; the cause of the Sept 12 outage). The reconcile now checks launchctl: unchanged config + loaded job = no-op; unloaded job = load without a rewrite; changed config = rewrite + reload.
The summary prompt is English and named no target language, so the model answered in English. The prompt now states the language from the core LANGUAGE env (de/en), with a same-as-entries fallback.
Summaries misattributed events because dialogue transcripts carry no speaker labels. Three measures, composed: - Mode-tiered evidence: a monologue line reads '<sender> spoke' (one speaker, structurally known); a conversation line states that who said which line is unknown. Prompt forbids attributing statements inside a conversation to named people. - Reported-speech register: summaries write what people recorded and told, not bare facts. An attribution error becomes a misreport of speech, not a false claim about a person. - Unclear-word gate: low-confidence words from the transcript quality record are listed as unclear per entry; the prompt forbids building statements or attributions on them.
Rendering strings were hardcoded English and the gist prompt named no target language, so German diaries carried English meta lines and English gists. - diary.py gains a string table (en/de) covering every reader-facing string: kind labels, date basis, callouts, links, front page, month and weekday names. configure_language() selects once at startup from the core LANGUAGE env; English stays the default and the test baseline. - Read prompt states the gist language explicitly, same fix as the summaries. Whisper vocabulary priming is localized too. - 2 German rendering tests; 73 pass.
Gists and summaries read like topic inventories ('X und Y berichten
über A, B, C und D').
- Gist prompt: sober-chronicle register (tone C) with anonymized
few-shot examples (de/en), verbs over nominalizations, at most two
moments, no topic lists. A conversation participant may be named
only when the words address them by name.
- Summary prompt: warm-chronicle register (tone A) with anonymized
few-shot examples (de/en), same list ban.
- Messages addressed to one person are letters: never distilled,
posted 1:1 with the addressee in the heading; the gist is null for
them by prompt and ignored by the renderer.
- 1 new test; 74 pass.
Title: Family Memories / Familienerinnerungen. The intro now says what the place is for and that every entry links to the original recording; the verbatim-mechanics sentence is gone.
A summary presented a misheard name as a family member. The name sat in one transcript at confidence 0.19, flagged but not fixed; telling the summariser a word is unreliable did not stop it from using it. - New correct_pass in the transcript chain (gate -> polish -> correct -> structure): low-confidence words may be mapped to a household name from the wiki person pages, closed set only. The model can pick a known name or stay silent; it cannot introduce words. Replacements are recorded in the pass metadata. - Summary evidence redacts remaining unclear words to [unclear]; a word the model never sees cannot become a person. - Summary prompt is primed with the household people list and caps the paragraph at four short sentences, one moment each, no event chaining. - 4 new tests; 92 pass.
A moment quote showcased a misheard name. Claimed moments that contain any low-confidence word from the transcript's quality record are dropped before rendering; the full transcript keeps them, marked by the audio link beside it.
'Eure Erinnerungen, festhalten in einer Chronik zum Nachlesen. ...' English matches: 'Your memories, kept in a chronicle to read back.'
Components, data flow, the ontology layer, verification and capping rules with their measured bases, and an incident-derived pitfalls table. Replaces scattered knowledge from the Sept 2026 work.
Prompt and pass-parameter edits did not reach cached artifacts; regeneration required a manual --force. - TranscriptPass carries a fingerprint (hash of its prompt or parameters); the pass trail stores it; stale_passes compares version and fingerprint. The correct pass includes the household names, so an ontology change re-evaluates old refusals. - The diary re-runs stale passes on cached records from raw text during any compile; whisper does not re-run. - ReadingStore and SummaryStore stamp each entry with the hash of the prompt that produced it; a mismatch is a cache miss. - A stack version hash was considered and rejected: every commit would invalidate everything. Fingerprints scope invalidation to the artifacts a change actually affects. - 5 new tests; 190 pass.
Inline images and an audio player per recording on diary pages.
- Decided: compile-time export of media to plain dated files under
{data_dir}/memory/media — originals, an m4a transcode for Safari
(Ogg/Opus), page-weight thumbnails. The archive stays readable
without famstack or Synapse.
- Open: serving via a static mount or a thin filesystem proxy with
logical URLs.
- Rejected: view-time serving from Synapse's authenticated media
API (1.160); it keeps the archive dependent on a running
homeserver.
arthware-dev
force-pushed
the
feature/memory-diary
branch
from
September 14, 2026 19:35
acff3d5 to
c4064de
Compare
arthware-dev
changed the title
The project rule excludes em dashes from user-facing prose, and the diary emitted them on every published page and in two stderr lines. Changed: the front-page intro sentence, the month list and the year list, and the two operator messages. Kept: the entry heading, `### Marge — for Bart`. It reads better than the alternatives and is the one place the dash carries meaning.
Three things a reader should not have to see. - Every entry carried how its date was derived: "dated from the spoken opening", "dated from when it was sent". That is our concern and it repeated under every entry on every page. The entry line is now the kind and the length. Where we could not derive a date, the warning callout still says so, in the place it matters. - The front page opened its second section with "Dates we could not recover" and the phrase "sync burst". A system caveat in our own words is the wrong thing to meet on the front door of a family's diary. Each affected entry already carries the warning where it is read. - The page title and its heading disagreed: "Family Diary" in the frontmatter, "Family Memories" as the heading. Both now read Family Diary, Familientagebuch in German. The frontmatter had kept the original title because the preamble is only applied when a page is first created. Drops the now-unused unrecovered_h and unrecovered_body strings, and the German test assertion on the removed date-basis line.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Turns the memories room into diary pages in the wiki:
stack memory diary,run nightly by the curator. Recordings become dated entries with verified
quotes and an audio link. The room stays the source of truth and the audio
the archival original.
How an entry is made
transcribe (whisper, quality captured) → cleanup passes (gate → polish →
correct → structure) → reading (dates, joins, gist, quotes) → compile →
render (de/en).
message without one is filed as not datable, and the page says so.
names are repaired from the household ontology (closed set) or kept off
the page.
messages to one person render whole, whatever their length.
produced them, so prompt edits regenerate exactly what they touched on
the next compile.
Also in here
config,
--max-context 0,--suppress-nst, output caps derived frommeasured ratios. The LaunchAgent now survives
stack down ai && stack up ai, which used to leave it dead until reboot.tools/family-memories: a Simpsons test corpus (de/en) with groundtruth for dates, fragments and quotes. The diary tests assert against
it instead of hand-written fixtures.
docs/design/brain/memory-pipeline-architecture.mdis the reviewer'smap: components, thresholds with their measured bases, pitfalls, and
the open items (media export with inline player, episode grouping,
diarization).
--dry-runpreviews without publishing,--limit Nbounds a preview,--retranscribere-decodes after a whisper config change.