diff --git a/.gitignore b/.gitignore index b1e76ed4..f031c933 100644 --- a/.gitignore +++ b/.gitignore @@ -68,3 +68,10 @@ impl/ !/AGENT.md # Local planning workspace (Nimbalyst) — not product source nimbalyst-local/ + +# Rendered test corpora — the spec is the source of truth, the audio +# and images it renders are disposable artifacts. +tools/family-memories/out/ + +# Nimbalyst editor scratch (screenshots, transcript images) +.nimbalyst/ diff --git a/docs/design/brain/memories-pipeline.md b/docs/design/brain/memories-pipeline.md new file mode 100644 index 00000000..0757ed49 --- /dev/null +++ b/docs/design/brain/memories-pipeline.md @@ -0,0 +1,107 @@ +# Memories Processing Pipeline + +Status: **draft** — companion to [diary-journal.md](diary-journal.md) +(which covers the timeline/diary/journal split and rendering). This doc +covers the step before: turning a messy, already-populated memories +room into clean timeline entries. Grounded in a structural probe of the +real room (Sept 2026) and scored by the `tools/family-memories` corpus. + +## What the room actually contains + +| Pattern | Consequence for the pipeline | +|---|---| +| Voice memos with spoken date openings ("Hallo X, heute ist der …") | the date is in the audio, nowhere else | +| Sync bursts: offline recordings uploaded together, out of order | `origin_server_ts` = sync time, days off; ordering within burst is meaningless | +| True fragments: one recording split mid-sentence | must be joined before summarizing | +| Kitchen-table dialogues (2+ speakers) | need dialogue-aware handling, not verbatim monologue treatment | +| Images + late caption via reply relation | caption belongs to the image | +| Texts referencing "the picture above" with no relation | implicit context, adjacency-based | +| Long texts with `m.replace` edits | only the final version counts | + +Matrix stores **no compose time** — verified against the spec and the +room's own events. Element strips filenames and metadata. A synced +memo without a spoken date has an unrecoverable date. + +## Pipeline + +``` +room history (paginated, oldest-first) + → 1 resolve edits collapsed, replies attached, bursts detected + → 2 transcribe Whisper; cached in TRANSCRIPT_DIR (never re-pay GPU) + → 3 classify per item, local LLM, structured output: + {mode, spoken_date?, fragment_boundary?, addressee?} + → 4 join fragment pairs merged (ends-mid-thought ⨯ continues) + → 5 date spoken > live timestamp > burst ⇒ UNCERTAIN + → 6 compile timeline entries (diary-journal.md takes over) +``` + +1. **Resolve** — pure Matrix mechanics, no AI: apply `m.replace`, attach + reply-captions to parents, group events <120s apart from the same + sender into candidate bursts. Keep event ids for idempotency. +2. **Transcribe** — existing Whisper path + `TRANSCRIPT_DIR` cache + (core already reserves it for exactly this backfill). +3. **Classify** — one structured-output call per transcript + (temperature 0): monologue/dialogue, spoken date if present, + starts/ends mid-thought, addressee. This worked cleanly in the probe. +4. **Join** — a burst is NOT a fragment chain (the probe's key trap: + three same-second uploads were three independent memos). Join only + when A ends mid-thought AND B continues it — an LLM judgment on the + pair, not a timing heuristic. +5. **Date** — the timestamp is usually right: most messages are sent + live, so `origin_server_ts` is the default. Spoken dates override + it when present. Only messages inside a detected sync burst get the + exception treatment — there the timestamp is days off, so without a + spoken date the entry is dated "week of " and marked + uncertain rather than confidently wrong. +6. **Compile** — entries carry `{date, date_confidence, kind, people, + transcript/summary, assets, source event ids}` into + `memory/timeline/`, per diary-journal.md. + +## Decisions + +- **Backfill-first.** The room is already populated; the compiler is a + rerunnable batch over full history, incremental later. Idempotency + via source event ids in each timeline entry. +- **Diarization stays out of v1** (per diary-journal.md), but step 3 + cheaply *labels* dialogues, so the renderer can mark them "captured + by X, conversation" and a later diarization pass knows exactly which + few recordings to touch. +- **Goal is topics, not verbatim accuracy.** Whisper large-v3-turbo was + rated clean on real German memos; good enough. No model change needed. +- **Pages are a chronicle/diary hybrid, not transcript dumps.** Detail + scales with the source: short memos stay verbatim, long recordings + get a gist, a short narrative, selected word-for-word quotes, and + the audio link. Quoted words are verified against the transcript; + narrative renders as narrative. The full transcript stays in the + transcript store; audio in Matrix is the archival original. +- **Privacy shape:** content flows machine-to-machine (Synapse → + Whisper → oMLX → vault); only structure and compiled entries surface. + +## Open + +- Date fix at the source: a small upload path that stamps + `dev.famstack.recorded_ts` into event content would eliminate the + uncertain class for future memos. Family habit of speaking the date + covers the past. +- Burst window (120s): settled differently than expected. Timing alone + turned out to be the wrong signal -- three memos recorded a minute + apart at the dinner table are not a sync burst, and calling them one + filed a normal evening as undateable. A run now has to contradict + itself (some memo says aloud it was made on a day its own timestamp + disagrees with) before its timestamps are distrusted, which leaves + the window doing nothing but grouping what arrived together. +- Join thresholds: gone. The model names which message finishes which, + having both in front of it; the room checks the link is adjacent, + same sender, same kind. + +- The polish pass could take the household vocabulary too. Whisper now + decodes against the family's names and topics, which is where a + misheard name has to be fixed -- polish may not change words and + should not. But the same vocabulary would help polish decide where + sentences break around a proper noun it now knows is a name. Its + contract does not move: clean sentences out of an imperfect + transcription, same tone, same words. +- Re-transcription when the vocabulary changes. A new family member + does not improve recordings already decoded, and re-running whisper + over years of audio to pick up one name is the wrong default. An + explicit `--retranscribe` would make it a choice. diff --git a/docs/design/brain/memory-pipeline-architecture.md b/docs/design/brain/memory-pipeline-architecture.md new file mode 100644 index 00000000..387563cc --- /dev/null +++ b/docs/design/brain/memory-pipeline-architecture.md @@ -0,0 +1,211 @@ +# Memory Pipeline Architecture + +Status: reference, September 2026. Describes the voice path from the +Matrix memories room to the published diary, the ontology layer, and +the mitigations this system needs because it runs on local models. + +## Context and constraints + +The pipeline runs on one Apple Silicon host (M1 Max, 64 GB). Models: +Qwen3.6-35B-A3B 4-bit on oMLX for text, whisper.cpp large-v3-turbo +for speech. Both are small compared to hosted models and show error +classes that hosted APIs rarely surface. The output is a family +archive: pages are read years later, by people who were in the +recordings. An error in a published page misstates someone's life. + +Consequences for the design: + +- Generated text is verified mechanically or constrained to closed + answer sets. Prompt instructions alone did not prevent any of the + failures listed under Pitfalls. +- Every model call has an output token cap derived from a measured + ratio. Uncapped local calls monopolize the single GPU. +- All expensive results are cached, keyed by content, so a full + recompile is affordable and deterministic. + +## Component overview + +``` +Matrix memories room (source of truth, audio = archival original) + │ + ▼ + transcription (whisper.cpp, configured flags, quality capture) + │ + ▼ + transcript store ~/famstack-data/core/transcripts/, 1 JSON/event + │ {raw, text, quality, passes[]} + ▼ + pass chain gate → polish → correct → structure + │ (lib/stack/ai/transcripts.py, versioned) + ▼ + reading facts per message: date, mode, addressee, + │ fragment links, gist, candidate quotes + ▼ + compile date precedence, fragment joins, reply/caption + │ attachment, entry assembly + ▼ + render language table, tiered entry formats + │ + ▼ + wiki (Quartz) pages in memory/brain, committed by curator +``` + +The room is append-only and the compiler is a fold over the full +history. There is no watermark: a reply or an edit can land on a +year-old entry, and only a full pass attaches it. Caches make the +fold cheap; they store cost, not position. + +## Transcription + +whisper.cpp flags, set in the LaunchAgent and reconciled on every +`stack up ai`: + +| Flag | Value | Reason | +|---|---|---| +| `--language` | from `[core].language` | auto-detection fails on silence and noise; one incident recording of infant sounds was transcribed as CJK text | +| `--max-context` | 0 | a hallucinated segment otherwise seeds the next segment | +| `--suppress-nst` | on | drops non-speech tokens | + +Each flag was tested against the three recordings that produced the +2026-09-14 incident. Each flag fixed loops the other two did not; +only the combination produced zero repetition on all three. + +Transcription requests use `verbose_json` with word timestamps. The +stored quality record keeps per-segment `avg_logprob`, +`no_speech_prob`, `temperature`, a word count, and every word below +0.5 confidence. whisper computes these values on every call; the +plain `json` format discards them. Voice commands and chat notes use +the plain format and skip all quality machinery — the cost is only +paid where an archive is built. + +## Transcript store and pass framework + +One record per Matrix event, atomic writes, single-flight production. +Record fields: + +- `raw` — whisper output, never modified by any pass +- `text` — current output of the pass chain +- `quality` — segment metrics and low-confidence words +- `passes[]` — `{name, version, outcome, model?, replacements?}` + +The pass list makes model upgrades incremental: increasing a pass +version marks affected records stale (`stale_passes`), and a sweep +re-runs that single pass on stored raw text without re-running +whisper. `--retranscribe` exists for the case where whisper itself +changed. + +| Pass | Function | Failure class covered | +|---|---|---| +| gate | empty the text when the transcript is hallucinated | repetition loops (measured 5–147 repeats of one 6-gram vs 1–2 in real speech); majority of segments failing whisper's own thresholds (avg_logprob < −1.0, no_speech_prob > 0.6) | +| polish | restore punctuation; word sequence verified unchanged | unreadable single-block output | +| correct | map low-confidence words to household names, closed set | misheard names (measured: p=0.19 on the one confirmed case) | +| structure | paragraph breaks at ≥1.5 s segment pauses, from word counts | wall-of-text rendering; no model call | + +The gate keeps two independent signals because the failure classes +are disjoint: a repetition loop is a high-confidence failure the +logprob check does not see, and mumble is a low-confidence failure +the repetition check does not see. + +## Ontology layer + +Sources: person pages in the wiki (canonical spelling plus synonyms) +and `ontology.toml` topics. Consumers: + +| Consumer | Use | Effect | +|---|---|---| +| whisper priming | names and topics as decoder prompt | fewer mishearings at the source | +| correct pass | closed replacement set | a repaired word is always a real name | +| summary prompt | list of family members | a name outside the list is treated as a mishearing and kept off the page | +| archivist (documents domain) | tags and correspondents | same principle, pre-existing | + +The pattern in all four: the ontology converts an open generation +problem into selection from a known set. Selection is the reliable +operation at this model size. + +## Reading, compilation, attribution + +The reading returns facts per message, JSON, temperature 0: spoken +date, mode (monologue/dialogue/note), addressee, fragment links, +gist, candidate quotes. Rules with rationale: + +- **Quotes are verified.** A candidate quote renders only if it + matches one sentence or a consecutive run in the transcript + (case/punctuation-insensitive). The page shows the transcript's own + text. Quotes containing any low-confidence word are dropped. +- **Date precedence:** spoken date > live timestamp > unrecoverable. + Matrix records only server receipt time; a synced message can be + days off. Sync-burst messages without a spoken date are filed under + the week they surfaced and labeled. +- **Attribution is limited to structural facts.** Sender and spoken + addressee are known. Line-level attribution inside a conversation + is unknown until diarization exists; summaries may name a + conversation's participants and nothing finer. +- **Fragment joins are content decisions.** Three uploads in one + second are usually three memos; a join requires one message to end + mid-sentence and the next to continue it. Timing alone misfiled + ordinary evenings as sync bursts before this rule. + +Summary input is preprocessed: words the pipeline knows are unclear +are replaced with `[unclear]` before the model sees them. This +replaced an instruction ("do not use unreliable words"), which the +model had ignored. + +## Rendering + +- Entries under 120 words render verbatim. +- Longer recordings render as gist, verified quotes, folded full + transcript, audio link. +- Messages addressed to one person render whole at any length. +- Recordings whose transcript the gate emptied render with a fixed + explanatory sentence and the audio link. +- All reader-facing strings, month and weekday names come from a + language table (`en`/`de`), selected once per run from config. + Prompts that produce family-facing text name the target language + explicitly; a model otherwise answers in the prompt's language. + +## Pitfalls (incident-derived) + +| Incident | Cause | Mitigation | +|---|---|---| +| 100k-token generation, 25 min GPU monopoly (2026-09-14) | polish is an echo task; a looping transcript has no natural end; no output cap; server default max_tokens was 128000 | gate before any echo task; caps at input size + 10% (measured output ratio 0.98–1.01); server default lowered | +| misheard name presented as a family member | whisper flagged the word (p=0.19) but the summary prompt's warning was ignored | correction pass (closed set), evidence redaction, ontology priming, quote filter | +| English output in a German diary | prompts are English; models answer in the prompt language | target language stated in every prompt; rendered strings from the language table | +| whisper dead after every `stack down ai` / `up ai` | stop hook unloads the LaunchAgent; nothing on the up path loaded it; `RunAtLoad` fires only at login | on_start reconciles the agent by content and loads it when absent | +| undatable memories | Matrix has no compose-time field; offline recordings carry sync time | spoken-date extraction; honest "unrecoverable" state; recording habit: say the date aloud | +| stale caches serving old wording | caches were keyed by content only; prompt changes did not change the keys | every cached artifact stores a fingerprint (hash) of the prompt or pass parameters that produced it; a prompt edit invalidates exactly the affected artifacts on the next compile | + +## Open items + +- Diarization for conversations (would upgrade attribution from + participant-level to line-level; mode labels already mark the + affected entries). +- Episode grouping: a vacation spanning many entries currently + renders as independent entries plus one month summary. +- Inline media on diary pages: images, and an audio player with a + play button per recording. Design: media export at compile time, + not a proxy. Matrix is capture transport and timeline anchor; the + archive is plain files, the same pattern the archivist uses for + documents (chat -> Paperless). + - The compiler writes each media original to + `{data_dir}/memory/media///.`, + idempotent by event id. Audio gets an `.m4a` transcode beside + the original (Safari does not play Ogg/Opus reliably); images + get a page-weight thumbnail. ffmpeg is a compile-time + dependency. + - Serving is an open choice: a static mount in the wiki + container, or a thin proxy that serves from the same filesystem + and gives logical URLs independent of the on-disk layout. Either + way the renderer emits stable URLs and + `