From a46f403705aaf60d44ae02823bc24f8d5f1c92a2 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 12:01:54 +0200 Subject: [PATCH 01/43] feat(tools): family-memories test corpus generator for the diary feature A Simpsons-themed Memories-room corpus (German + English) mirroring the chaotic message patterns of the real room: addressed voice memos with spoken dates, out-of-order sync bursts, mid-sentence fragment pairs, a two-voice kitchen-table dialogue, late caption replies, m.replace edits, and implicit-context texts. Audio is synthesized through the stack's local speech service; every item carries true_date/date_source ground truth so the diary compiler can be scored on date recovery, fragment joining, and context attachment. ingest.py replays the timeline into a test rig and refuses to target the production homeserver. --- tools/family-memories/README.md | 60 +++++++ tools/family-memories/generate.py | 169 ++++++++++++++++++++ tools/family-memories/ingest.py | 183 ++++++++++++++++++++++ tools/family-memories/spec.de.yaml | 242 +++++++++++++++++++++++++++++ tools/family-memories/spec.en.yaml | 213 +++++++++++++++++++++++++ 5 files changed, 867 insertions(+) create mode 100644 tools/family-memories/README.md create mode 100644 tools/family-memories/generate.py create mode 100644 tools/family-memories/ingest.py create mode 100644 tools/family-memories/spec.de.yaml create mode 100644 tools/family-memories/spec.en.yaml diff --git a/tools/family-memories/README.md b/tools/family-memories/README.md new file mode 100644 index 0000000..73f44c1 --- /dev/null +++ b/tools/family-memories/README.md @@ -0,0 +1,60 @@ +# family-memories + +A generator for a demo Memories room, set in the Simpsons world — the +companion to `family-docs`. It renders German voice memos (via the +stack's local Piper TTS), a kitchen-table dialogue, images, and diary +texts, then replays them into a **test rig's** Matrix room with the +same chaotic patterns a real Memories room accumulates. Built to +develop and test the diary compiler against known ground truth. + +The **spec is the source of truth** (`spec.yaml`, one ordered +timeline). Rendered audio/images in `out/` are disposable artifacts. + +## The patterns it reproduces + +Observed in a real Memories room (metadata-level analysis, March 2026 +onward) and deliberately kept messy: + +| Pattern | The trap it sets for the compiler | +|---|---| +| addressed-memo | spoken "Hallo , heute ist der " opening — date lives in the audio | +| batch-upload | 3 separate memos synced in one burst, out of order — upload timestamps are lies | +| fragment-pair | ONE recording split mid-sentence — looks identical to a batch upload | +| kitchen-table-dialogue | two speakers, needs diarization or dialogue-aware summarization | +| memo-no-date | live-sent, so the server timestamp happens to be right | +| memo-no-date-in-sync-burst | no spoken date AND a synced timestamp — date is unrecoverable; the compiler must say so | +| image-with-caption-reply | caption arrives ~90s later as a reply relation | +| image-no-context | nothing to anchor it but the timestamp | +| text-with-edit | `m.replace` — only the final version counts | +| text-reply-to-audio | commentary attached to a memo | +| implicit-context | text referencing "the picture above" with NO relation | + +Every item carries `true_date` + `date_source` ground truth so +pipeline tests can score date recovery, fragment joining, and context +attachment against `out/manifest.json`. + +## Usage + +```sh +# render everything into out/ (needs the ai stacklet's speech service) +python tools/family-memories/generate.py # both locales; --locale de/en + +# replay into a TEST RIG (never production — the script refuses merles.eu) +python tools/family-memories/ingest.py \ + --homeserver http://:42031 \ + --room '#memories:' \ + --login marge:PW --login homer:PW --locale de +``` + +`ingest.py` writes `out/ingest-log.json` (item id → event id) for +assertions. Bursts land back-to-back; replies, edits, and the MSC3245 +voice flag are sent exactly as real clients send them. + +## What the corpus encodes about dates + +Matrix stamps events with **server receipt time only** — there is no +compose-time field, and Element's offline queue discards it. Live-sent +messages have trustworthy timestamps; synced batches do not. The only +in-band recording date is the spoken opening. The compiler's date +logic (spoken date > live timestamp > burst-aware uncertainty) is +exactly what this corpus exercises. diff --git a/tools/family-memories/generate.py b/tools/family-memories/generate.py new file mode 100644 index 0000000..f8464b3 --- /dev/null +++ b/tools/family-memories/generate.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +"""generate.py — render the Memories test corpus from spec.yaml. + +Voice memos and dialogues are synthesized through the stack's local +speech service (OpenAI-compatible TTS, Piper under the hood), images +through Pillow. Everything lands in out/ next to a manifest.json that +carries the pattern annotations, durations, bursts, and relations the +ingester and the diary-pipeline tests key off. + + python tools/family-memories/generate.py # render all + python tools/family-memories/generate.py --list # show the set + python tools/family-memories/generate.py --only fragment + +Rendering only WRITES LOCAL FILES. Nothing here talks to Matrix — that +is ingest.py's job, and it targets a test rig, never production. +""" + +from __future__ import annotations + +import argparse +import io +import json +import struct +import sys +import urllib.request +import wave +from pathlib import Path + +import yaml + +HERE = Path(__file__).resolve().parent +OUT = HERE / "out" +TTS_URL = "http://localhost:42063/v1/audio/speech" +TURN_GAP_MS = 400 + + +def tts(text: str, voice: str) -> bytes: + body = json.dumps({ + "model": "tts-1", "voice": voice, + "response_format": "wav", "input": text.strip(), + }).encode() + req = urllib.request.Request( + TTS_URL, data=body, headers={"Content-Type": "application/json"}) + return urllib.request.urlopen(req, timeout=300).read() + + +def wav_params(blob: bytes): + """Read format + ALL frames. Streamed WAVs (like the TTS output) + carry bogus RIFF/nframes headers, so read to EOF and never trust + the declared frame count.""" + with wave.open(io.BytesIO(blob)) as w: + fmt = (w.getnchannels(), w.getsampwidth(), w.getframerate()) + frames = w.readframes(0x7FFFFFF) + return fmt, frames + + +def write_wav(fmt, frames: bytes) -> bytes: + """Re-emit with a clean, correct header.""" + nchannels, sampwidth, framerate = fmt + out = io.BytesIO() + with wave.open(out, "wb") as w: + w.setnchannels(nchannels) + w.setsampwidth(sampwidth) + w.setframerate(framerate) + w.writeframes(frames) + return out.getvalue() + + +def concat_wavs(blobs: list[bytes]) -> bytes: + """Join turns with a short silence gap. All clips must share the + same format (same speech backend => same rate; refuse otherwise).""" + first_fmt, _ = wav_params(blobs[0]) + nchannels, sampwidth, framerate = first_fmt + silence = b"\x00" * (int(framerate * TURN_GAP_MS / 1000) + * sampwidth * nchannels) + joined = b"" + for i, blob in enumerate(blobs): + fmt, frames = wav_params(blob) + if fmt != first_fmt: + sys.exit(f"voice sample formats differ ({fmt} vs {first_fmt}); " + "use voices from the same speech backend") + if i: + joined += silence + joined += frames + return write_wav(first_fmt, joined) + + +def wav_duration_ms(blob: bytes) -> int: + (nchannels, sampwidth, framerate), frames = wav_params(blob) + return int(len(frames) / (nchannels * sampwidth) / framerate * 1000) + + +def render_image(label: str, path: Path) -> None: + from PIL import Image, ImageDraw + img = Image.new("RGB", (1134, 1512), "#fdf6e3") + d = ImageDraw.Draw(img) + # crayon-ish scribble scene: house, sun, four blobs — enough for a + # VLM to describe, deliberately childlike + d.rectangle([300, 700, 830, 1200], outline="#8b4513", width=12) + d.polygon([(260, 700), (565, 450), (870, 700)], outline="#b22222", width=12) + d.ellipse([900, 120, 1060, 280], fill="#ffd700") + for i, color in enumerate(["#1565c0", "#2e7d32", "#ef6c00", "#8e24aa"]): + x = 180 + i * 220 + d.ellipse([x, 1250, x + 120, 1370], outline=color, width=10) + d.text((80, 60), label, fill="#555555") + img.save(path, "PNG") + + +def render_locale(locale: str, args) -> None: + spec = yaml.safe_load((HERE / f"spec.{locale}.yaml").read_text()) + voices = spec["voices"] + items = [i for i in spec["items"] + if not args.only or args.only in i["id"]] + + if args.list: + for it in items: + print(f" {locale}/{it['id']:<28} {it['kind']:<9} {it['pattern']}") + return + + out = OUT / locale + out.mkdir(parents=True, exist_ok=True) + manifest = [] + for it in items: + entry = {k: it[k] for k in + ("id", "pattern", "kind", "sender") if k in it} + for opt in ("burst", "fragment_of", "reply_to", "delay_after_prev"): + if opt in it: + entry[opt] = it[opt] + if it["kind"] in ("voice", "dialogue"): + if it["kind"] == "voice": + blob = write_wav(*wav_params(tts(it["text"], + voices[it["sender"]]))) + else: + blob = concat_wavs( + [tts(t["text"], voices[t["voice"]]) for t in it["turns"]]) + path = out / f"{it['id']}.wav" + path.write_bytes(blob) + entry["file"] = path.name + entry["duration_ms"] = wav_duration_ms(blob) + elif it["kind"] == "image": + path = out / f"{it['id']}.png" + render_image(it["image_label"], path) + entry["file"] = path.name + else: # text + entry["text"] = it["text"].strip() + if "edit_text" in it: + entry["edit_text"] = it["edit_text"].strip() + manifest.append(entry) + print(f"rendered {locale}/{it['id']} ({it['pattern']})") + + (out / "manifest.json").write_text( + json.dumps(manifest, indent=2, ensure_ascii=False)) + print(f"{len(manifest)} items -> {out}/manifest.json\n") + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--list", action="store_true") + ap.add_argument("--only", default=None, help="substring filter on item id") + ap.add_argument("--locale", choices=["de", "en"], default=None, + help="render one locale (default: all)") + args = ap.parse_args() + locales = [args.locale] if args.locale else ["de", "en"] + for locale in locales: + render_locale(locale, args) + + +if __name__ == "__main__": + main() diff --git a/tools/family-memories/ingest.py b/tools/family-memories/ingest.py new file mode 100644 index 0000000..f5d075c --- /dev/null +++ b/tools/family-memories/ingest.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""ingest.py — replay the rendered Memories corpus into a TEST RIG room. + +Reads out/manifest.json (produced by generate.py) and posts each item +into a Matrix room in timeline order, reproducing the real room's +chaos: sync bursts land back-to-back, captions arrive late as replies, +diary entries get edited, voice messages carry the MSC3245 voice flag. +Writes out/ingest-log.json mapping item id -> event id so pipeline +tests can assert against ground truth. + + python tools/family-memories/ingest.py \ + --homeserver http://testrig.local:42031 \ + --room '#memories:testrig.local' \ + --login marge:PASSWORD --login homer:PASSWORD + +SAFETY: this tool refuses to run against the production homeserver +(merles.eu). There is no default homeserver on purpose. --force-i-know +overrides the guard and should never be needed. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +import urllib.parse +import urllib.request +from pathlib import Path + +HERE = Path(__file__).resolve().parent +OUT = HERE / "out" +PRODUCTION_MARKERS = ("merles.eu",) + + +class Client: + def __init__(self, homeserver: str): + self.hs = homeserver.rstrip("/") + self.txn = 0 + + def call(self, path, token=None, method="GET", body=None, + raw_body=None, content_type="application/json"): + data = raw_body if raw_body is not None else ( + json.dumps(body).encode() if body is not None else None) + req = urllib.request.Request( + self.hs + path, data=data, method=method, + headers={"Content-Type": content_type, + **({"Authorization": f"Bearer {token}"} if token else {})}) + return json.load(urllib.request.urlopen(req, timeout=120)) + + def login(self, user, password): + r = self.call("/_matrix/client/v3/login", method="POST", body={ + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": user}, + "password": password}) + return r["access_token"], r["user_id"] + + def resolve_room(self, room, token): + if room.startswith("!"): + return room + r = self.call("/_matrix/client/v3/directory/room/" + + urllib.parse.quote(room), token) + return r["room_id"] + + def join(self, room_id, token): + self.call(f"/_matrix/client/v3/join/{urllib.parse.quote(room_id)}", + token, method="POST", body={}) + + def upload(self, blob, mimetype, token): + r = self.call("/_matrix/media/v3/upload?filename=upload", + token, method="POST", raw_body=blob, + content_type=mimetype) + return r["content_uri"] + + def send(self, room_id, content, token): + self.txn += 1 + r = self.call( + f"/_matrix/client/v3/rooms/{urllib.parse.quote(room_id)}" + f"/send/m.room.message/corpus{self.txn}-{int(time.time()*1000)}", + token, method="PUT", body=content) + return r["event_id"] + + +def voice_content(item, mxc): + dur = item["duration_ms"] + return { + "msgtype": "m.audio", "body": item["file"], "url": mxc, + "info": {"duration": dur, "mimetype": "audio/wav", + "size": (OUT / item["file"]).stat().st_size}, + "org.matrix.msc1767.audio": { + "duration": dur, + "waveform": [512 + (i * 37) % 400 for i in range(30)]}, + "org.matrix.msc3245.voice": {}, + } + + +def image_content(item, mxc): + return {"msgtype": "m.image", "body": item["file"], "url": mxc, + "info": {"mimetype": "image/png", "w": 1134, "h": 1512, + "size": (OUT / item["file"]).stat().st_size}} + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--homeserver", required=True) + ap.add_argument("--room", required=True, + help="room id (!..) or alias (#memories:server)") + ap.add_argument("--login", action="append", required=True, + metavar="USER:PASSWORD") + ap.add_argument("--locale", choices=["de", "en"], default="de") + ap.add_argument("--delay", type=float, default=2.0, + help="seconds between non-burst items") + ap.add_argument("--force-i-know", action="store_true") + args = ap.parse_args() + global OUT + OUT = OUT / args.locale + + if not args.force_i_know and any( + m in args.homeserver or m in args.room for m in PRODUCTION_MARKERS): + sys.exit("REFUSING: target looks like the production instance. " + "This corpus is for test rigs only.") + + manifest = json.loads((OUT / "manifest.json").read_text()) + c = Client(args.homeserver) + tokens, user_ids = {}, {} + for spec in args.login: + user, _, pw = spec.partition(":") + tokens[user], user_ids[user] = c.login(user, pw) + if not args.force_i_know and any( + uid.endswith(m) for uid in user_ids.values() + for m in PRODUCTION_MARKERS): + sys.exit("REFUSING: logged-in server is production (merles.eu).") + + missing = {i["sender"] for i in manifest} - set(tokens) + if missing: + sys.exit(f"no --login for sender(s): {', '.join(sorted(missing))}") + + room_id = c.resolve_room(args.room, next(iter(tokens.values()))) + for t in tokens.values(): + c.join(room_id, t) + + event_ids, prev_burst = {}, None + for item in manifest: + tok = tokens[item["sender"]] + burst = item.get("burst") + if "delay_after_prev" in item: + time.sleep(min(item["delay_after_prev"], 10)) + elif not (burst and burst == prev_burst): + time.sleep(args.delay) + prev_burst = burst + + if item["kind"] in ("voice", "dialogue"): + mxc = c.upload((OUT / item["file"]).read_bytes(), "audio/wav", tok) + content = voice_content(item, mxc) + elif item["kind"] == "image": + mxc = c.upload((OUT / item["file"]).read_bytes(), "image/png", tok) + content = image_content(item, mxc) + else: + content = {"msgtype": "m.text", "body": item["text"]} + if item.get("reply_to"): + content["m.relates_to"] = { + "m.in_reply_to": {"event_id": event_ids[item["reply_to"]]}} + eid = event_ids[item["id"]] = c.send(room_id, content, tok) + print(f"sent {item['id']} -> {eid}") + + if item.get("edit_text"): + time.sleep(args.delay) + eid2 = c.send(room_id, { + "msgtype": "m.text", "body": "* " + item["edit_text"], + "m.new_content": {"msgtype": "m.text", + "body": item["edit_text"]}, + "m.relates_to": {"rel_type": "m.replace", "event_id": eid}, + }, tok) + event_ids[item["id"] + "#edit"] = eid2 + print(f"sent {item['id']}#edit -> {eid2}") + + (OUT / "ingest-log.json").write_text(json.dumps( + {"room_id": room_id, "events": event_ids}, indent=2)) + print(f"\n{len(event_ids)} events -> {OUT}/ingest-log.json") + + +if __name__ == "__main__": + main() diff --git a/tools/family-memories/spec.de.yaml b/tools/family-memories/spec.de.yaml new file mode 100644 index 0000000..f525a2f --- /dev/null +++ b/tools/family-memories/spec.de.yaml @@ -0,0 +1,242 @@ +# spec.yaml — the Memories test corpus, Simpsons edition (German). +# +# One ordered timeline. Every item reproduces a message pattern observed +# in a real Memories room, including the chaotic ones. The `pattern` +# field names what each item is testing so pipeline assertions can key +# off the manifest instead of guessing. +# +# Voices map to the speech service's OpenAI-compatible voice names. +# +# Dates — the core difficulty this corpus encodes. Matrix stamps only +# the moment an event reaches the server. Live-sent messages therefore +# carry a usable timestamp; offline-recorded, later-synced messages do +# NOT (their timestamp is the sync moment, days off). Some memos speak +# their date, some don't. `true_date` is the ground truth for scoring: +# the pipeline should recover it from the spoken opening, or from a +# trustworthy live timestamp, or explicitly mark the date uncertain +# (no-date memo inside a sync burst — the unanswerable case). + +voices: + marge: alloy + homer: onyx + +items: + # -- 1: the canonical addressed memo: greeting + addressee + spoken date + - id: memo-bart-zeugnis + true_date: 2026-03-16 + date_source: spoken + pattern: addressed-memo + kind: voice + sender: marge + text: > + Hallo Bart, heute ist der sechzehnte März. Ich wollte dir sagen, + dass ich richtig stolz auf dich war heute, auch wenn dein Zeugnis + wieder ein Abenteuer war. Direktor Skinner hat angerufen, aber + diesmal ausnahmsweise mit einer guten Nachricht. Du hast der + neuen Schülerin geholfen, als die anderen gemein zu ihr waren. + Das vergesse ich dir nicht. + + # -- 2: batch upload — three SEPARATE memos, recorded on different days, + # uploaded in one go, and OUT OF ORDER (spoken dates shuffled). + - id: batch-lisa-saxophon + true_date: 2026-03-20 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hallo Lisa, heute ist der zwanzigste März. Dein Saxophonkonzert + gestern Abend war wunderschön. Sogar dein Vater hat nicht + geschnarcht, und das will etwas heißen. Frau Krabappel hat gesagt, + so ein Talent hatte Springfield seit Jahren nicht. + - id: batch-maggie-zahn + true_date: 2026-03-17 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hallo Maggie, heute ist der siebzehnte März. Dein erster Zahn ist + da! Du hast den ganzen Tag auf deinem Schnuller herumgekaut und + niemand hat gemerkt, warum. Papa hat es beim Abendessen entdeckt + und fast geweint. Er sagt natürlich, ihm war nur etwas im Auge. + - id: batch-bart-fahrrad + true_date: 2026-03-19 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hallo Bart, heute ist der neunzehnte März. Du bist heute ohne + Stützräder gefahren, einmal ganz um den Block. Beim zweiten Mal + bist du in Flanders' Hecke gelandet, aber du hast gelacht und + bist gleich wieder aufgestiegen. + + # -- 3: a TRUE fragment: one recording split mid-sentence into two + # messages. A ends mid-thought, B continues it. The trap's twin. + - id: fragment-urlaub-a + true_date: 2026-03-22 + date_source: spoken + pattern: fragment-pair + burst: sync-2 + fragment_of: urlaub + kind: voice + sender: homer + text: > + Hallo Kinder, heute ist der zweiundzwanzigste März. Ich sitze + gerade in der Garage und plane unseren Sommerurlaub. Eure Mutter + weiß noch nichts davon, also pssst. Ich dachte, wir fahren dieses + Jahr endlich mal ans Meer, und zwar + - id: fragment-urlaub-b + true_date: 2026-03-22 + date_source: inherited-from-fragment + pattern: fragment-pair + burst: sync-2 + fragment_of: urlaub + kind: voice + sender: homer + text: > + nicht in dieses Motel wie letztes Mal, sondern in ein richtiges + Hotel mit Pool. Ich habe schon drei Monate lang das Wechselgeld + aus dem Sofa gespart. Sagt Mama nichts, das ist unsere + Überraschung. + + # -- 4: kitchen-table dialogue, two speakers, no date opening + - id: dialog-kueche + true_date: 2026-03-28 + date_source: live-timestamp + pattern: kitchen-table-dialogue + kind: dialogue + sender: homer + turns: + - voice: marge + text: > + Homie, erinnerst du dich an unser erstes Date? Du hast + behauptet, du hättest das Restaurant ausgesucht. + - voice: homer + text: > + Natürlich habe ich das. Moe hat nur... geholfen. Und die + Kerzen waren meine Idee. + - voice: marge + text: > + Die Kerzen waren Geburtstagskerzen aus dem Kwik-E-Mart, auf + einem Donut. + - voice: homer + text: > + Es war ein romantischer Donut, Marge. Die Kinder sollen + wissen, dass ihr Vater Stil hat. + - voice: marge + text: > + Sie sollen wissen, dass ihre Eltern über denselben Donut + heute noch lachen können. Deshalb nehmen wir das hier auf. + + # -- 5: memo with NO date opening at all — date must come from nowhere + - id: memo-ohne-datum + true_date: 2026-03-25 + date_source: live-timestamp + pattern: memo-no-date + kind: voice + sender: homer + text: > + Bart, alter Kumpel. Nur ganz kurz: das mit dem Baumhaus heute, + das bleibt unter uns Männern. Deine Mutter muss nicht wissen, + dass die Leiter schon vorher kaputt war. Du warst sehr tapfer, + und der Arzt hat gesagt, der Gips kann in vier Wochen ab. + + # -- 6: image + caption arriving late as a reply + - id: bild-zeichnung + true_date: 2026-04-02 + date_source: live-timestamp + pattern: image-with-caption-reply + kind: image + sender: marge + image_label: "Kinderzeichnung" + - id: bild-zeichnung-caption + true_date: 2026-04-02 + date_source: live-timestamp + pattern: image-with-caption-reply + kind: text + sender: marge + reply_to: bild-zeichnung + delay_after_prev: 90 + text: > + Das hat Maggie heute gemalt — das sollen wir alle sein, vor dem + Haus. Homer ist der große gelbe Kreis. + + # -- 7: image WITHOUT any caption or context + - id: bild-kontextlos + true_date: 2026-04-03 + date_source: live-timestamp + pattern: image-no-context + kind: image + sender: homer + image_label: "Foto ohne Kontext" + + # -- 8: written diary entry, then edited (m.replace) + - id: text-tagebuch + true_date: 2026-04-04 + date_source: spoken + pattern: text-with-edit + kind: text + sender: marge + text: > + Heute war einer dieser Tage, die man festhalten möchte. Maggie + hat ihren ersten Schritt gemacht, mitten im Wohnzimmer, und + niemand hatte eine Kamera in der Hand. Homer ist vom Sofa + gesprungen und hat dabei die Chipsschale durchs Zimmer geworfen. + Bart hat behauptet, er hätte es gefilmt, aber es war nur ein + Video von seinem Skateboard. Lisa hat sofort ein Lied darüber + komponiert. Ich schreibe das hier auf, damit wir es nie + vergessen. + edit_text: > + Heute war einer dieser Tage, die man festhalten möchte. Maggie + hat ihren ersten Schritt gemacht, mitten im Wohnzimmer, und + niemand hatte eine Kamera in der Hand. Homer ist vom Sofa + gesprungen und hat dabei die Chipsschale durchs Zimmer geworfen. + Bart hat behauptet, er hätte es gefilmt, aber es war nur ein + Video von seinem Skateboard. Lisa hat sofort ein Lied darüber + komponiert — sie nennt es „Maggies erster Marsch". Ich schreibe + das hier auf, damit wir es nie vergessen. Nachtrag: Es war der + vierte April, kurz nach dem Mittagessen. + + # -- 9: short text reply commenting on an audio + - id: kommentar-zu-memo + true_date: 2026-04-05 + date_source: live-timestamp + pattern: text-reply-to-audio + kind: text + sender: homer + reply_to: memo-bart-zeugnis + text: > + Das hat er von mir. Das mit dem Helfen, nicht das mit dem Zeugnis. + + # -- 10: text referencing an image WITHOUT a reply relation (implicit + # context — the nastiest pattern for the compiler) + - id: impliziter-kontext + true_date: 2026-04-03 + date_source: live-timestamp + pattern: implicit-context + kind: text + sender: homer + text: > + Das Bild oben ist übrigens vom Grillfest. Der schwarze Fleck + links ist nicht der Grill, das ist das Steak. + + # -- 11: the unanswerable case — NO spoken date, inside a sync burst. + # The upload timestamp is days off; the pipeline must mark the + # date uncertain instead of trusting the server timestamp. + - id: memo-ohne-datum-im-sync + true_date: 2026-03-21 + date_source: unrecoverable + pattern: memo-no-date-in-sync-burst + burst: sync-2 + kind: voice + sender: homer + text: > + Lisa, Schatz, dein Vater hier. Ich habe deinen Aufsatz über + Gerechtigkeit gelesen, den du am Kühlschrank vergessen hast. + Ich habe nicht alles verstanden, aber ich war noch nie so stolz, + etwas nicht zu verstehen. Weiter so. diff --git a/tools/family-memories/spec.en.yaml b/tools/family-memories/spec.en.yaml new file mode 100644 index 0000000..ea01a00 --- /dev/null +++ b/tools/family-memories/spec.en.yaml @@ -0,0 +1,213 @@ +# spec.en.yaml — the Memories test corpus, Simpsons edition (English). +# Same items, patterns, bursts, and ground-truth dates as spec.de.yaml; +# only the language differs, so pipeline behavior can be compared +# across locales. See spec.de.yaml for the full pattern commentary. + +voices: + marge: alloy + homer: onyx + +items: + - id: memo-bart-zeugnis + true_date: 2026-03-16 + date_source: spoken + pattern: addressed-memo + kind: voice + sender: marge + text: > + Hi Bart, today is March sixteenth. I wanted to tell you how proud + I was of you today, even if your report card was another + adventure. Principal Skinner called, but for once with good news. + You stood up for the new girl when the others were mean to her. + I won't forget that. + + - id: batch-lisa-saxophon + true_date: 2026-03-20 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hi Lisa, today is March twentieth. Your saxophone concert last + night was beautiful. Even your father didn't snore, and that's + saying something. Mrs. Krabappel said Springfield hasn't seen + talent like yours in years. + + - id: batch-maggie-zahn + true_date: 2026-03-17 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hi Maggie, today is March seventeenth. Your first tooth is here! + You chewed on your pacifier all day and nobody knew why. Daddy + spotted it at dinner and almost cried. He says he just had + something in his eye, of course. + + - id: batch-bart-fahrrad + true_date: 2026-03-19 + date_source: spoken + pattern: batch-upload + burst: sync-1 + kind: voice + sender: marge + text: > + Hi Bart, today is March nineteenth. You rode without training + wheels today, once all the way around the block. The second time + you landed in Flanders' hedge, but you laughed and got right + back on. + + - id: fragment-urlaub-a + true_date: 2026-03-22 + date_source: spoken + pattern: fragment-pair + burst: sync-2 + fragment_of: urlaub + kind: voice + sender: homer + text: > + Hi kids, today is March twenty-second. I'm sitting in the garage + planning our summer vacation. Your mother doesn't know anything + yet, so shush. I was thinking this year we finally go to the + beach, and I mean + + - id: fragment-urlaub-b + true_date: 2026-03-22 + date_source: inherited-from-fragment + pattern: fragment-pair + burst: sync-2 + fragment_of: urlaub + kind: voice + sender: homer + text: > + not that motel like last time, but a real hotel with a pool. + I've been saving the couch change for three months. Don't tell + Mom, that's our surprise. + + - id: dialog-kueche + true_date: 2026-03-28 + date_source: live-timestamp + pattern: kitchen-table-dialogue + kind: dialogue + sender: homer + turns: + - voice: marge + text: > + Homie, do you remember our first date? You claimed you picked + the restaurant. + - voice: homer + text: > + Of course I did. Moe just... helped. And the candles were my + idea. + - voice: marge + text: > + The candles were birthday candles from the Kwik-E-Mart, on a + donut. + - voice: homer + text: > + It was a romantic donut, Marge. The kids should know their + father has style. + - voice: marge + text: > + They should know their parents can still laugh about the same + donut today. That's why we're recording this. + + - id: memo-ohne-datum + true_date: 2026-03-25 + date_source: live-timestamp + pattern: memo-no-date + kind: voice + sender: homer + text: > + Bart, old buddy. Just real quick: that thing with the treehouse + today stays between us men. Your mother doesn't need to know the + ladder was already broken. You were very brave, and the doctor + says the cast can come off in four weeks. + + - id: bild-zeichnung + true_date: 2026-04-02 + date_source: live-timestamp + pattern: image-with-caption-reply + kind: image + sender: marge + image_label: "Kid's drawing" + + - id: bild-zeichnung-caption + true_date: 2026-04-02 + date_source: live-timestamp + pattern: image-with-caption-reply + kind: text + sender: marge + reply_to: bild-zeichnung + delay_after_prev: 90 + text: > + Maggie drew this today — it's supposed to be all of us in front + of the house. Homer is the big yellow circle. + + - id: bild-kontextlos + true_date: 2026-04-03 + date_source: live-timestamp + pattern: image-no-context + kind: image + sender: homer + image_label: "Photo without context" + + - id: text-tagebuch + true_date: 2026-04-04 + date_source: spoken + pattern: text-with-edit + kind: text + sender: marge + text: > + Today was one of those days you want to hold on to. Maggie took + her first step, right in the middle of the living room, and + nobody had a camera in hand. Homer jumped off the couch and threw + the chip bowl across the room doing it. Bart claimed he filmed + it, but it was just a video of his skateboard. Lisa immediately + composed a song about it. I'm writing this down so we never + forget it. + edit_text: > + Today was one of those days you want to hold on to. Maggie took + her first step, right in the middle of the living room, and + nobody had a camera in hand. Homer jumped off the couch and threw + the chip bowl across the room doing it. Bart claimed he filmed + it, but it was just a video of his skateboard. Lisa immediately + composed a song about it — she calls it "Maggie's First March". + I'm writing this down so we never forget it. P.S.: It was April + fourth, just after lunch. + + - id: kommentar-zu-memo + true_date: 2026-04-05 + date_source: live-timestamp + pattern: text-reply-to-audio + kind: text + sender: homer + reply_to: memo-bart-zeugnis + text: > + He gets that from me. The helping part, not the report card part. + + - id: impliziter-kontext + true_date: 2026-04-03 + date_source: live-timestamp + pattern: implicit-context + kind: text + sender: homer + text: > + By the way, the picture above is from the barbecue. The black + spot on the left is not the grill, that's the steak. + + - id: memo-ohne-datum-im-sync + true_date: 2026-03-21 + date_source: unrecoverable + pattern: memo-no-date-in-sync-burst + burst: sync-2 + kind: voice + sender: homer + text: > + Lisa, sweetie, your dad here. I read your essay about justice, + the one you left on the fridge. I didn't understand all of it, + but I've never been so proud of not understanding something. + Keep it up. From 82048498d4c89bc3efbf4bc65cb7070fa32d105b Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 12:05:35 +0200 Subject: [PATCH 02/43] docs(brain): memories processing pipeline design MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The step before diary-journal.md's timeline: resolve → transcribe → classify → join → date → compile, grounded in the structural probe of the real memories room. Key decisions: burst-aware date precedence with honest uncertainty, fragment joining by content not timing, dialogue labeling without v1 diarization, backfill-first idempotent compilation. Scored against the family-memories corpus ground truth. --- docs/design/brain/memories-pipeline.md | 81 ++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 docs/design/brain/memories-pipeline.md diff --git a/docs/design/brain/memories-pipeline.md b/docs/design/brain/memories-pipeline.md new file mode 100644 index 0000000..e81d6cc --- /dev/null +++ b/docs/design/brain/memories-pipeline.md @@ -0,0 +1,81 @@ +# Memories Processing Pipeline + +Status: **draft** — companion to [diary-journal.md](diary-journal.md) +(which covers the timeline/diary/journal split and rendering). This doc +covers the step before: turning a messy, already-populated memories +room into clean timeline entries. Grounded in a structural probe of the +real room (Sept 2026) and scored by the `tools/family-memories` corpus. + +## What the room actually contains + +| Pattern | Consequence for the pipeline | +|---|---| +| Voice memos with spoken date openings ("Hallo X, heute ist der …") | the date is in the audio, nowhere else | +| Sync bursts: offline recordings uploaded together, out of order | `origin_server_ts` = sync time, days off; ordering within burst is meaningless | +| True fragments: one recording split mid-sentence | must be joined before summarizing | +| Kitchen-table dialogues (2+ speakers) | need dialogue-aware handling, not verbatim monologue treatment | +| Images + late caption via reply relation | caption belongs to the image | +| Texts referencing "the picture above" with no relation | implicit context, adjacency-based | +| Long texts with `m.replace` edits | only the final version counts | + +Matrix stores **no compose time** — verified against the spec and the +room's own events. Element strips filenames and metadata. A synced +memo without a spoken date has an unrecoverable date. + +## Pipeline + +``` +room history (paginated, oldest-first) + → 1 resolve edits collapsed, replies attached, bursts detected + → 2 transcribe Whisper; cached in TRANSCRIPT_DIR (never re-pay GPU) + → 3 classify per item, local LLM, structured output: + {mode, spoken_date?, fragment_boundary?, addressee?} + → 4 join fragment pairs merged (ends-mid-thought ⨯ continues) + → 5 date spoken > live timestamp > burst ⇒ UNCERTAIN + → 6 compile timeline entries (diary-journal.md takes over) +``` + +1. **Resolve** — pure Matrix mechanics, no AI: apply `m.replace`, attach + reply-captions to parents, group events <120s apart from the same + sender into candidate bursts. Keep event ids for idempotency. +2. **Transcribe** — existing Whisper path + `TRANSCRIPT_DIR` cache + (core already reserves it for exactly this backfill). +3. **Classify** — one structured-output call per transcript + (temperature 0): monologue/dialogue, spoken date if present, + starts/ends mid-thought, addressee. This worked cleanly in the probe. +4. **Join** — a burst is NOT a fragment chain (the probe's key trap: + three same-second uploads were three independent memos). Join only + when A ends mid-thought AND B continues it — an LLM judgment on the + pair, not a timing heuristic. +5. **Date** — the timestamp is usually right: most messages are sent + live, so `origin_server_ts` is the default. Spoken dates override + it when present. Only messages inside a detected sync burst get the + exception treatment — there the timestamp is days off, so without a + spoken date the entry is dated "week of " and marked + uncertain rather than confidently wrong. +6. **Compile** — entries carry `{date, date_confidence, kind, people, + transcript/summary, assets, source event ids}` into + `memory/timeline/`, per diary-journal.md. + +## Decisions + +- **Backfill-first.** The room is already populated; the compiler is a + rerunnable batch over full history, incremental later. Idempotency + via source event ids in each timeline entry. +- **Diarization stays out of v1** (per diary-journal.md), but step 3 + cheaply *labels* dialogues, so the renderer can mark them "captured + by X, conversation" and a later diarization pass knows exactly which + few recordings to touch. +- **Goal is topics, not verbatim accuracy.** Whisper large-v3-turbo was + rated clean on real German memos; good enough. No model change needed. +- **Privacy shape:** content flows machine-to-machine (Synapse → + Whisper → oMLX → vault); only structure and compiled entries surface. + +## Open + +- Date fix at the source: a small upload path that stamps + `dev.famstack.recorded_ts` into event content would eliminate the + uncertain class for future memos. Family habit of speaking the date + covers the past. +- Burst window (120s) and join thresholds: tune against the corpus + (`true_date` / `fragment_of` ground truth in the manifests). From 61fbffde0d03386614fc3b7c661e9e7a9f04f22a Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 13:42:26 +0200 Subject: [PATCH 03/43] fix(tools): render the memories corpus with real audio Half the corpus was silence. The speech service answers 200 with a header-only WAV when a voice's Piper model is missing, so four of the eight memos rendered as empty files and nobody noticed. - Stop when the speech service returns a silent clip - Point both English speakers at a voice that actually resolves - Send each sync burst's members back-to-back, so the one memo whose date is meant to be unrecoverable lands inside its burst - Document what a replay cannot reproduce: it stamps everything with the day it runs, so live-timestamp dating is unscored and the burst window has to shrink --- .gitignore | 7 +++++++ tools/family-memories/README.md | 22 ++++++++++++++++++++++ tools/family-memories/generate.py | 17 +++++++++++++++-- tools/family-memories/ingest.py | 27 ++++++++++++++++++++++++++- tools/family-memories/spec.en.yaml | 7 ++++++- 5 files changed, 76 insertions(+), 4 deletions(-) diff --git a/.gitignore b/.gitignore index b1e76ed..f031c93 100644 --- a/.gitignore +++ b/.gitignore @@ -68,3 +68,10 @@ impl/ !/AGENT.md # Local planning workspace (Nimbalyst) — not product source nimbalyst-local/ + +# Rendered test corpora — the spec is the source of truth, the audio +# and images it renders are disposable artifacts. +tools/family-memories/out/ + +# Nimbalyst editor scratch (screenshots, transcript images) +.nimbalyst/ diff --git a/tools/family-memories/README.md b/tools/family-memories/README.md index 73f44c1..6f83539 100644 --- a/tools/family-memories/README.md +++ b/tools/family-memories/README.md @@ -50,6 +50,28 @@ python tools/family-memories/ingest.py \ assertions. Bursts land back-to-back; replies, edits, and the MSC3245 voice flag are sent exactly as real clients send them. +## What a replay cannot reproduce + +Matrix stamps an event when the server receives it, and only an +application service may backdate one. So a replay lands the whole +corpus on the day it runs, and two things follow: + +- **Live-timestamp dating is not scored.** Items with + `date_source: live-timestamp` carry a `true_date` months before the + replay date, and no compiler could recover it. They verify that a + message *without* a spoken date falls back to its timestamp, not that + the timestamp is right. +- **The burst window has to shrink.** In a real room, live messages sit + hours or days apart and a sync burst lands within seconds, so 120s + separates them. The replay compresses the live gaps to `--delay` + (2s by default) while burst members still land ~0.1s apart. The ratio + survives; the absolute threshold does not. Compile this corpus with + `stack memory diary --burst-window 1`. + +Everything else scores against ground truth as authored: spoken-date +recovery, fragment joining, burst grouping, the unrecoverable case, +edits, and caption attachment. + ## What the corpus encodes about dates Matrix stamps events with **server receipt time only** — there is no diff --git a/tools/family-memories/generate.py b/tools/family-memories/generate.py index f8464b3..2deb0e4 100644 --- a/tools/family-memories/generate.py +++ b/tools/family-memories/generate.py @@ -20,7 +20,6 @@ import argparse import io import json -import struct import sys import urllib.request import wave @@ -35,13 +34,27 @@ def tts(text: str, voice: str) -> bytes: + """Synthesize `text`, or stop. + + The speech service answers 200 with a header-only WAV when the + Piper model behind a voice is missing, so silence arrives looking + like success. Checked here because a corpus of empty recordings + still ingests, still transcribes to nothing, and scores as a + pipeline result rather than a broken rig. + """ body = json.dumps({ "model": "tts-1", "voice": voice, "response_format": "wav", "input": text.strip(), }).encode() req = urllib.request.Request( TTS_URL, data=body, headers={"Content-Type": "application/json"}) - return urllib.request.urlopen(req, timeout=300).read() + blob = urllib.request.urlopen(req, timeout=300).read() + _, frames = wav_params(blob) + if not frames: + sys.exit(f"speech service returned silence for voice '{voice}'. " + "Its Piper model is most likely not downloaded: check " + "'docker logs stack-ai-speech'.") + return blob def wav_params(blob: bytes): diff --git a/tools/family-memories/ingest.py b/tools/family-memories/ingest.py index f5d075c..eda5909 100644 --- a/tools/family-memories/ingest.py +++ b/tools/family-memories/ingest.py @@ -100,6 +100,31 @@ def image_content(item, mxc): "size": (OUT / item["file"]).stat().st_size}} +def burst_ordered(manifest): + """Send order with each burst's members contiguous. + + A sync burst is a phone coming back online and flushing its queue, + so its members reach the server back-to-back regardless of when + they were recorded. The spec lists items in timeline order, which + interleaves burst members with live messages; replaying that order + literally spaces them out and erases the pattern they exist to + encode. Members are emitted where the burst's first one appears. + """ + groups = {} + for item in manifest: + if burst := item.get("burst"): + groups.setdefault(burst, []).append(item) + + out, sent = [], set() + for item in manifest: + if item["id"] in sent: + continue + for member in groups.get(item.get("burst"), [item]): + out.append(member) + sent.add(member["id"]) + return out + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--homeserver", required=True) @@ -140,7 +165,7 @@ def main(): c.join(room_id, t) event_ids, prev_burst = {}, None - for item in manifest: + for item in burst_ordered(manifest): tok = tokens[item["sender"]] burst = item.get("burst") if "delay_after_prev" in item: diff --git a/tools/family-memories/spec.en.yaml b/tools/family-memories/spec.en.yaml index ea01a00..7d4a5ac 100644 --- a/tools/family-memories/spec.en.yaml +++ b/tools/family-memories/spec.en.yaml @@ -3,9 +3,14 @@ # only the language differs, so pipeline behavior can be compared # across locales. See spec.de.yaml for the full pattern commentary. +# Both speakers share one voice on purpose. The stack's Piper map +# points every non-alloy name at a model it never downloads, so a +# second voice would render as silence. Nothing in the pipeline reads +# the audio for speaker identity (v1 has no diarization), so the +# corpus loses realism here and no ground truth. voices: marge: alloy - homer: onyx + homer: alloy items: - id: memo-bart-zeugnis From b50565e056351cfc18e971b114da9211198faf3f Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 13:42:42 +0200 Subject: [PATCH 04/43] feat(memory): turn the memories room into a family diary `stack memory diary` reads the whole memories room back and publishes it as month pages in the wiki, in the words it was recorded in. The room is messy on purpose: recordings sync days late, one memo arrives split across two files, a caption trails its photo. Matrix only ever stamps the time the server received an event, so anything that synced late would file under the wrong week forever. The compiler prefers a date spoken inside the recording, falls back to the timestamp for anything sent live, and refuses to guess for a recording that synced late without saying its own date -- those are marked on the page instead of given a confident wrong date. Nothing is summarised. The model is asked to read each message, never to rewrite one, so what reaches the page is what was said. Transcripts are cached by event id and shared with the bots, so a second run over a full room costs nothing. --- stacklets/memory/bot/cli/diary.py | 342 +++++++++++++++ stacklets/memory/bot/cli_entrypoint.py | 11 +- stacklets/memory/bot/diary.py | 572 +++++++++++++++++++++++++ stacklets/memory/cli/diary.py | 49 +++ tests/stacklets/test_memory_diary.py | 448 +++++++++++++++++++ 5 files changed, 1421 insertions(+), 1 deletion(-) create mode 100644 stacklets/memory/bot/cli/diary.py create mode 100644 stacklets/memory/bot/diary.py create mode 100644 stacklets/memory/cli/diary.py create mode 100644 tests/stacklets/test_memory_diary.py diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py new file mode 100644 index 0000000..6d10caf --- /dev/null +++ b/stacklets/memory/bot/cli/diary.py @@ -0,0 +1,342 @@ +"""stack memory diary — compile the memories room into the family diary. + +The memories room is already full. Somebody has been recording voice +memos to their kids for months, and nothing has ever read them back. +This command is the reader: it walks the room's whole history, decodes +every recording, works out when each one actually happened, and +publishes the result as diary pages in the family wiki. + +It is a batch, not a listener. The room is the source of truth and it +only grows, so re-running recompiles the same history into the same +pages rather than appending to something stateful. That makes the +command safe to run again after a bad model day, and it means the +expensive half is cached rather than repeated: transcripts live in +`TRANSCRIPT_DIR`, keyed by event id, shared with the bots. + + stack memory diary compile and publish + stack memory diary --dry-run print the pages, write nothing + stack memory diary --room memories a different room + stack memory diary --burst-window 1 see below + +WHY THE BURST WINDOW IS A KNOB + Messages that synced late carry arrival timestamps, not recording + times, and the tell is that they land in a tight run. In a real room + the gaps between live messages are hours, so 120s separates the two + cleanly. A *replayed* test corpus compresses those gaps to seconds + and needs a window well under a second. There is no single value + that fits both, so the caller picks. + +Runs inside `stack-core-bot-runner`: it has the whisper client, the LLM +client, the brain working copy, and the Matrix admin credentials. The +host-side wrapper is a thin docker-exec, like `stack memory wiki`. +""" + +from __future__ import annotations + +import asyncio +import json +import os +import sys +from pathlib import Path +from urllib.parse import quote + +import aiohttp + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # bot/ +sys.path.insert(0, "/app") # voice, stack.ai.client + +import diary # noqa: E402 +import voice # noqa: E402 +from stack.ai.client import LLMError, Transcriber # noqa: E402 + +from . import wiki # noqa: E402 + +HELP = "Compile the memories room into the family diary" + +# One page of room history per request. Synapse caps this well above +# 100; the number only trades round trips against response size. +_PAGE = 100 + + +def _err(msg: str) -> None: + print(msg, file=sys.stderr) + + +# ── Reading the room ────────────────────────────────────────────────── +# +# Through the Synapse admin API, which reads any room without the +# reader having to be a member. A compiler that had to be invited to the +# memories room would be one more thing to set up, and one more account +# with standing access to the most private room in the house. + + +async def _admin_token(session: aiohttp.ClientSession, homeserver: str) -> str: + user = os.environ.get("MATRIX_ADMIN_USER", "") + password = os.environ.get("MATRIX_ADMIN_PASSWORD", "") + if not user or not password: + raise RuntimeError("MATRIX_ADMIN_USER/PASSWORD not set in this container") + async with session.post(f"{homeserver}/_matrix/client/v3/login", json={ + "type": "m.login.password", + "identifier": {"type": "m.id.user", "user": user}, + "password": password, + }) as resp: + if resp.status != 200: + raise RuntimeError(f"admin login failed: HTTP {resp.status}") + return (await resp.json())["access_token"] + + +async def _resolve_room(session, homeserver, token, room: str) -> str: + if room.startswith("!"): + return room + alias = room if room.startswith("#") else \ + f"#{room}:{os.environ.get('MATRIX_SERVER_NAME', '')}" + async with session.get( + f"{homeserver}/_matrix/client/v3/directory/room/{quote(alias)}", + headers={"Authorization": f"Bearer {token}"}, + ) as resp: + if resp.status != 200: + raise RuntimeError(f"no such room: {alias}") + return (await resp.json())["room_id"] + + +async def _history(session, homeserver, token, room_id: str) -> list[dict]: + """Every message event in the room, newest page first. + + Paginates backwards until Synapse stops handing back a cursor. The + caller sorts; order here is only what the API gives us. + """ + events: list[dict] = [] + cursor = "" + headers = {"Authorization": f"Bearer {token}"} + while True: + url = (f"{homeserver}/_synapse/admin/v1/rooms/{quote(room_id)}" + f"/messages?dir=b&limit={_PAGE}") + if cursor: + url += f"&from={quote(cursor)}" + async with session.get(url, headers=headers) as resp: + if resp.status != 200: + raise RuntimeError(f"could not read room: HTTP {resp.status}") + payload = await resp.json() + chunk = payload.get("chunk") or [] + events.extend(chunk) + cursor = payload.get("end") or "" + if not chunk or not cursor: + return events + + +async def _download(session, homeserver, token, mxc: str) -> bytes | None: + server, _, media_id = mxc.replace("mxc://", "").partition("/") + url = (f"{homeserver}/_matrix/client/v1/media/download/" + f"{quote(server)}/{quote(media_id)}") + async with session.get(url, headers={"Authorization": f"Bearer {token}"}) as resp: + if resp.status != 200: + _err(f" media {mxc}: HTTP {resp.status}") + return None + return await resp.read() + + +# ── Decoding and reading ────────────────────────────────────────────── + + +async def _transcribe(message, *, session, homeserver, token, + transcriber, llm) -> str: + """The words of a recording, transcribed once and remembered. + + Shares `TRANSCRIPT_DIR` with the bots, so a memo the archivist + already heard costs nothing here, and a second compile costs nothing + at all. That is the whole reason a backfill over a full room is + affordable. + """ + async def produce() -> dict: + audio = await _download(session, homeserver, token, message.url or "") + if not audio: + raise LLMError(f"could not download {message.url}") + raw = await transcriber.transcribe(audio, filename=message.body or "voice.wav") + text = await Transcriber.polish(raw, llm) if raw.strip() else raw + return {"raw": raw, "text": text, "url": message.url, + "filename": message.body} + + try: + return (await voice.TRANSCRIPTS.run(message.event_id, produce))["text"] + except LLMError as e: + _err(f" could not transcribe {message.event_id}: {e}") + return "" + + +_READ_PROMPT = """\ +You are reading one message from a family's private memories room so it +can be filed in their diary. Do not rewrite it, summarise it, translate +it, or comment on it. Report only facts about the text as it stands. + +This message reached the server on {arrival}. + +Message from {sender}: +--- +{body} +--- + +Reply with a JSON object with exactly these keys: + +"mode": a recorded conversation carries no speaker labels, so judge by + the turns rather than by names. Answer "dialogue" when a statement in + the text is answered by another statement inside the same text: a + question and its reply, a claim and its contradiction, someone being + teased and teasing back. Answer "monologue" when one person speaks + throughout, however many people they mention or address. Answer "note" + if it reads as written rather than spoken. + +"spoken_date": the date the speaker states inside the message, as + YYYY-MM-DD. Use only a date the text actually names, such as "today is + March sixteenth". If it names a day and month but no year, choose the + most recent such date on or before {arrival}. If the text states no + date at all, use null. Never derive a date from the arrival date + alone. + +"starts_mid_thought": true if the text begins part-way through a + sentence or thought, as though the recording started late. + +"ends_mid_thought": true if the text stops part-way through a sentence + or thought, as though the recording was cut off. + +"addressee": who the message is spoken to, written exactly as the + message names them ("Bart", "kids", "Maggie"), or null if it is not + addressed to anyone in particular. Never name the speaker themselves: + in a conversation between two people who are both present, there is + no addressee, so use null. +""" + + +async def _read(message, llm) -> diary.Reading: + """Ask the model what this message says about itself. + + Temperature 0: the same recording must read the same way on every + compile, or a rerun would silently reshuffle the diary. A model that + fails or answers with nonsense yields an empty reading, which dates + the entry from its timestamp -- worse, but not wrong in a way that + hides anything. + + Dates, fragment boundaries and addressees come back reliably at this + model tier. `mode` does not: an unlabelled two-speaker transcript + reads as one person recounting a conversation, and a 35B model calls + it a monologue. The prompt is tuned to suppress the false positive + rather than chase the false negative, because "Conversation" printed + over a private memo to a child is a worse page than "Voice note" + printed over a dinner-table recording. Recovering the rest needs + diarization, which v1 does not have. + """ + prompt = _READ_PROMPT.format( + arrival=message.sent_on.isoformat(), + sender=message.sender, + body=message.body.strip(), + ) + try: + raw = await llm.complete("classifier", prompt, + json_mode=True, temperature=0) + data = json.loads(raw) + except (LLMError, json.JSONDecodeError, TypeError) as e: + _err(f" could not read {message.event_id}: {e}") + return diary.Reading() + + if not isinstance(data, dict): + return diary.Reading() + return diary.Reading( + mode=str(data.get("mode") or "monologue"), + spoken_date=data.get("spoken_date") or None, + starts_mid_thought=bool(data.get("starts_mid_thought")), + ends_mid_thought=bool(data.get("ends_mid_thought")), + addressee=(data.get("addressee") or None), + ) + + +# ── The command ─────────────────────────────────────────────────────── + + +def _opt(argv: list[str], flag: str, fallback: str) -> str: + for i, arg in enumerate(argv): + if arg == flag and i + 1 < len(argv): + return argv[i + 1] + return fallback + + +async def run(llm, argv: list[str]) -> int: + room_arg = _opt(argv, "--room", "memories") + dry_run = "--dry-run" in argv + try: + window = float(_opt(argv, "--burst-window", + str(diary.DEFAULT_BURST_WINDOW_S))) + except ValueError: + _err("--burst-window wants a number of seconds") + return 2 + + homeserver = os.environ.get("MATRIX_HOMESERVER", "").rstrip("/") + if not homeserver: + _err("MATRIX_HOMESERVER not set — is core up?") + return 1 + bucket = os.environ.get("SHARED_BUCKET", "family") + + try: + transcriber = Transcriber.from_env(namespace="memory-diary") + except LLMError as e: + _err(f"no transcription available: {e}") + return 1 + + async with aiohttp.ClientSession() as session: + try: + token = await _admin_token(session, homeserver) + room_id = await _resolve_room(session, homeserver, token, room_arg) + events = await _history(session, homeserver, token, room_id) + except (RuntimeError, aiohttp.ClientError) as e: + _err(str(e)) + return 1 + + messages = diary.resolve(events, burst_window_s=window) + if not messages: + _err(f"nothing in {room_arg} to compile") + return 0 + _err(f"{len(messages)} message(s) in {room_arg}") + + # Transcription first and on its own: every later step reads + # words, and a recording that cannot be decoded should drop out + # before the model is asked to interpret its filename. + decoded = [] + for msg in messages: + if msg.kind != "voice": + decoded.append(msg) + continue + text = await _transcribe( + msg, session=session, homeserver=homeserver, token=token, + transcriber=transcriber, llm=llm) + if not text.strip(): + _err(f" no speech in {msg.event_id}, skipped") + continue + decoded.append(msg.__class__(**{**msg.__dict__, "body": text})) + + readings = {} + for msg in decoded: + if msg.kind == "image" or not msg.body.strip(): + readings[msg.event_id] = diary.Reading(mode="note") + continue + readings[msg.event_id] = await _read(msg, llm) + + await transcriber.aclose() + + entries = diary.compile_entries(decoded, readings) + _err(f"{len(entries)} diary entr{'y' if len(entries) == 1 else 'ies'}") + + pages = diary.pages_for(entries, room_id=room_id) + if dry_run: + for path, body, _title in pages: + print(f"\n{'=' * 70}\n{bucket}/{path}\n{'=' * 70}\n{body}") + return 0 + + rc = 0 + for path, body, title in pages: + rc |= wiki._publish( + body, target_path=f"{bucket}/{path}", + default_preamble=f"---\ntitle: {wiki._yaml_str(title)}\n---", + ) + return rc + + +if __name__ == "__main__": # pragma: no cover - exercised via cli_entrypoint + raise SystemExit(asyncio.run(run(None, sys.argv[1:]))) diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index d73abe9..be75166 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -17,6 +17,14 @@ words. Exit 1 means no keywords, which the host treats as "search it literally" rather than as a failure. + diary [--room ] [--burst-window ] [--dry-run] + Compile the memories room into the family diary. Walks the + room's full history, transcribes every recording (cached in + TRANSCRIPT_DIR), recovers the date each one was made, and + publishes month pages under the shared bucket. Rerunnable: + the room is the source of truth, so a second run recompiles + rather than appends. See `cli/diary.py` for the date rules. + wiki [--home] [--member ]... [--topic ]... [--dry-run] Regenerate the family wiki's entry pages. Apply by default; `--dry-run` previews to stdout. Bare invocation regenerates @@ -37,10 +45,11 @@ from stack.ai.client import LLM, LLMUnavailableError -from cli import rewrite, wiki +from cli import diary, rewrite, wiki _HANDLERS = { + "diary": diary.run, "rewrite": rewrite.run, "wiki": wiki.run, } diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py new file mode 100644 index 0000000..c4a00a4 --- /dev/null +++ b/stacklets/memory/bot/diary.py @@ -0,0 +1,572 @@ +"""Turning a memories room into dated diary entries. + +The memories room is where a family talks to its future self: voice +memos to a child, a photo with a caption, a dinner-table conversation +someone hit record on. It is also a mess. Recordings arrive days after +they were made, one memo gets split across two files, a caption shows +up ninety seconds behind its picture, and the only statement of when +something happened is a sentence spoken inside the audio. + +This module is the part of the compiler that does not touch the world: +given the room's events and a reading of each transcript, it works out +what happened, when, and renders it. Every I/O concern -- paginating +Synapse, downloading audio, calling whisper, calling the model, writing +the wiki -- lives in `cli/diary.py`. Splitting it this way is what lets +the hard parts (which date wins, what is one recording and what is two) +be tested against the corpus in `tools/family-memories` without a rig. + +The diary never paraphrases. The model is asked to *read* a transcript, +never to rewrite one: what lands on the page is the words that were +said. That is a promise to the reader in 2040, and it is also why the +classification step returns a small record of facts rather than prose. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field, replace +from datetime import date, datetime, timezone + +# A sync burst is a phone coming back online and flushing its queue, so +# its members land seconds apart whatever their recording dates. 120s is +# the design's starting point for a real room, where live messages sit +# hours or days apart. Note that a *replayed* corpus compresses those +# day-scale gaps to seconds, so a replay needs a far smaller window -- +# the value is a parameter for exactly that reason, not for tuning. +DEFAULT_BURST_WINDOW_S = 120.0 + + +# ── What the room gives us ──────────────────────────────────────────── + + +@dataclass(frozen=True) +class Message: + """One room message, after Matrix mechanics and before meaning. + + `body` is the words: a transcript for voice, the caption for text, + the filename for a photo. `ts` is server receipt time in epoch + milliseconds, which is the *only* time Matrix records -- there is no + compose-time field, so for anything that synced late this is days + off. Recovering the real date is `date_for`'s problem. + """ + + event_id: str + sender: str + ts: int + kind: str # "voice" | "image" | "text" + body: str = "" + url: str | None = None + duration_ms: int | None = None + reply_to: str | None = None + burst: str | None = None + + @property + def sent_on(self) -> date: + return datetime.fromtimestamp(self.ts / 1000, timezone.utc).date() + + +@dataclass(frozen=True) +class Reading: + """What the classifier read out of one message. + + Facts about the text, not a rewrite of it. `spoken_date` is the date + said aloud ("today is March sixteenth") and is the only in-band + record of when a recording was made. The two mid-thought flags exist + because a burst and a split recording look identical from timing + alone: three files a second apart are three memos or one memo in + three pieces, and only the words can tell you which. + """ + + mode: str = "monologue" # "monologue" | "dialogue" | "note" + spoken_date: str | None = None + starts_mid_thought: bool = False + ends_mid_thought: bool = False + addressee: str | None = None + + +@dataclass +class Entry: + """One thing that happened, ready to render. + + `event_ids` carries every message that fed the entry, so a rerun can + recognise what it already compiled and a reader can trace a line on + the page back to the recording it came from. + """ + + on: date + confidence: str # "spoken" | "sent" | "uncertain" + basis: str + kind: str + sender: str + body: str + at: int = 0 # arrival time of the first message, for same-day ordering + event_ids: list[str] = field(default_factory=list) + addressee: str | None = None + duration_ms: int | None = None + mode: str = "monologue" + comments: list[tuple[str, str]] = field(default_factory=list) + + +# ── Step 1: resolve ─────────────────────────────────────────────────── +# +# Pure Matrix mechanics, no reading of meaning. Edits collapse onto the +# message they replace, reply fallbacks come off the front of bodies, +# and runs of messages that arrived together get a burst label. Nothing +# here needs a model, and getting it wrong corrupts every later step. + + +# Clients prepend the quoted original to a reply's body, fenced off by a +# blank line (the "rich reply fallback"). It is the same text twice; left +# in, every reply would enter the diary quoting its parent. +_FALLBACK_LINE = re.compile(r"^>.*$") + + +def strip_reply_fallback(body: str) -> str: + """Drop the quoted-original block a client prepends to a reply.""" + lines = body.splitlines() + i = 0 + while i < len(lines) and _FALLBACK_LINE.match(lines[i]): + i += 1 + if i == 0: + return body + while i < len(lines) and not lines[i].strip(): + i += 1 + return "\n".join(lines[i:]) + + +def _kind_of(msgtype: str) -> str | None: + return {"m.audio": "voice", "m.image": "image", "m.text": "text"}.get(msgtype) + + +def resolve(events, *, burst_window_s: float = DEFAULT_BURST_WINDOW_S): + """Room events to messages: edits applied, replies linked, bursts marked. + + `events` is the raw chunk from Synapse in any order; the result is + oldest-first. Redacted events arrive with an empty content dict and + are dropped -- the message is gone, and a diary that renders a + tombstone is worse than one that renders nothing. + """ + edits: dict[str, tuple[int, str]] = {} + plain: list[Message] = [] + + for ev in events: + if ev.get("type") != "m.room.message": + continue + content = ev.get("content") or {} + kind = _kind_of(content.get("msgtype", "")) + if kind is None: + continue + + relates = content.get("m.relates_to") or {} + ts = ev.get("origin_server_ts") or 0 + + # An edit is not a message. It is a correction to one, and only + # the final version counts. + if relates.get("rel_type") == "m.replace": + target = relates.get("event_id") + new_body = (content.get("m.new_content") or {}).get("body", "") + if target and new_body and ts >= edits.get(target, (0, ""))[0]: + edits[target] = (ts, new_body) + continue + + info = content.get("info") or {} + # For an upload, `body` is the filename and `filename` is absent; + # a client that attaches a caption puts the caption in `body` and + # moves the real name to `filename`. Without this the diary would + # print "IMG_4021.png" where the caption belongs. + body = strip_reply_fallback(content.get("body", "")) + if kind == "image" and not content.get("filename"): + body = "" + plain.append(Message( + event_id=ev.get("event_id", ""), + sender=(ev.get("sender") or "").split(":")[0].lstrip("@"), + ts=ts, + kind=kind, + body=body, + url=content.get("url"), + duration_ms=info.get("duration"), + reply_to=(relates.get("m.in_reply_to") or {}).get("event_id"), + )) + + plain.sort(key=lambda m: (m.ts, m.event_id)) + resolved = [ + replace(m, body=edits[m.event_id][1]) if m.event_id in edits else m + for m in plain + ] + return mark_bursts(resolved, window_s=burst_window_s) + + +def mark_bursts(messages, *, window_s: float = DEFAULT_BURST_WINDOW_S): + """Label runs of messages that reached the server together. + + A run is consecutive messages from one sender separated by less than + `window_s`. Only runs of two or more get a label: a single message + arriving shortly after another is just someone typing quickly, and + calling that a burst would throw away a timestamp that is fine. + + The label means "these timestamps are arrival times, not recording + times". It does not mean the messages belong together -- that is + `join_fragments`, and confusing the two is the trap this corpus was + built to set. + """ + out = list(messages) + run: list[int] = [] + counter = 0 + + def close(run_idx): + nonlocal counter + if len(run_idx) < 2: + return + counter += 1 + label = f"burst-{counter}" + for i in run_idx: + out[i] = replace(out[i], burst=label) + + for i, msg in enumerate(out): + if run: + prev = out[run[-1]] + same_run = (msg.sender == prev.sender + and (msg.ts - prev.ts) / 1000.0 < window_s) + if same_run: + run.append(i) + continue + close(run) + run = [i] + close(run) + return out + + +# ── Step 2: join ────────────────────────────────────────────────────── + + +def join_fragments(messages, readings): + """Group messages into the recordings they actually are. + + One memo split mid-sentence arrives as two files that look exactly + like two memos sent back to back. The only evidence that separates + them is the words: the first stops mid-thought and the second picks + it up. So a join needs both halves to agree, and a burst label alone + is never enough -- three same-second uploads are usually three + independent memos. + + Returns a list of groups, each a list of messages in order. + """ + groups: list[list[Message]] = [] + for msg in messages: + reading = readings.get(msg.event_id, Reading()) + if groups: + prev = groups[-1][-1] + prev_reading = readings.get(prev.event_id, Reading()) + continues = ( + msg.kind == "voice" and prev.kind == "voice" + and msg.sender == prev.sender + and prev_reading.ends_mid_thought + and reading.starts_mid_thought + ) + if continues: + groups[-1].append(msg) + continue + groups.append([msg]) + return groups + + +# ── Step 3: date ────────────────────────────────────────────────────── + + +def parse_spoken_date(value: str | None) -> date | None: + """An ISO date the classifier heard, or None if it heard nothing.""" + if not value: + return None + try: + return date.fromisoformat(value.strip()[:10]) + except ValueError: + return None + + +def date_for(msg: Message, reading: Reading) -> tuple[date, str, str]: + """When this happened, how sure we are, and why. + + Three sources, in order of trust: + + 1. A date spoken inside the recording. It is the only one that + describes when the thing happened rather than when a server heard + about it, so it wins outright. + 2. The server timestamp, when the message arrived on its own. Most + messages are sent live, so this is usually right. + 3. Nothing usable, when a message without a spoken date arrived + inside a sync burst. The timestamp is known to be days off, so + the entry is dated to the week it surfaced and says so. Being + visibly unsure is the point: a confidently wrong date in a family + diary is worse than an honest gap. + """ + spoken = parse_spoken_date(reading.spoken_date) + if spoken is not None: + return spoken, "spoken", "dated from the spoken opening" + if msg.burst: + return msg.sent_on, "uncertain", ( + "arrived in a sync burst with no spoken date, " + "so this is the week it surfaced, not when it happened" + ) + return msg.sent_on, "sent", "dated from when it was sent" + + +# ── Step 4: compile ─────────────────────────────────────────────────── + + +def compile_entries(messages, readings) -> list[Entry]: + """Messages and their readings to dated diary entries. + + Two kinds of message do not earn an entry of their own. A reply + belongs to what it replies to, and a caption that arrives behind its + photo is that photo's caption -- rendering either separately breaks + the pair apart and leaves a line of commentary floating with no + subject. + """ + groups = join_fragments(messages, readings) + entries: list[Entry] = [] + by_event: dict[str, Entry] = {} + pending: list[tuple[Message, list[Message]]] = [] + + for group in groups: + head = group[0] + reading = readings.get(head.event_id, Reading()) + if head.reply_to: + pending.append((head, group)) + continue + + on, confidence, basis = date_for(head, reading) + entry = Entry( + on=on, + confidence=confidence, + basis=basis, + kind=head.kind, + sender=head.sender, + body=_joined_body(group), + at=head.ts, + event_ids=[m.event_id for m in group], + addressee=reading.addressee, + duration_ms=_total_duration(group), + mode=reading.mode, + ) + entries.append(entry) + for m in group: + by_event[m.event_id] = entry + + # Replies resolve after every parent exists, so a reply to a message + # later in the room still finds its subject. + for msg, group in pending: + parent = by_event.get(msg.reply_to or "") + if parent is None: + reading = readings.get(msg.event_id, Reading()) + on, confidence, basis = date_for(msg, reading) + orphan = Entry( + on=on, confidence=confidence, basis=basis, kind=msg.kind, + sender=msg.sender, body=_joined_body(group), at=msg.ts, + event_ids=[m.event_id for m in group], + addressee=reading.addressee, mode=reading.mode, + ) + entries.append(orphan) + by_event[msg.event_id] = orphan + continue + parent.comments.append((msg.sender, msg.body)) + parent.event_ids.append(msg.event_id) + by_event[msg.event_id] = parent + + # Within a day, keep the order the room has. Sorting by event id + # instead would scatter a day's entries into hash order, which reads + # as randomness on the page. + entries.sort(key=lambda e: (e.on, e.at)) + return entries + + +def _joined_body(group) -> str: + """The words of a group, with a split recording read back as one.""" + parts = [m.body.strip() for m in group if m.body.strip()] + if len(parts) < 2: + return parts[0] if parts else "" + # The break falls mid-sentence, so the halves join with a space + # rather than a paragraph break -- it was one sentence when it was + # spoken and it reads as one now. + return " ".join(parts) + + +def _total_duration(group) -> int | None: + durations = [m.duration_ms for m in group if m.duration_ms] + return sum(durations) if durations else None + + +# ── Step 5: render ──────────────────────────────────────────────────── +# +# The diary is a reading surface, not a report. It quotes and it links; +# it never summarises. An LLM decided what date an entry carries and +# whether two files were one recording -- it never decided what the page +# says, because the words on the page are the family's own. + +DIARY_DIR = "diary" + + +def _duration(ms: int | None) -> str: + if not ms: + return "" + total = round(ms / 1000) + return f"{total // 60}:{total % 60:02d}" + + +def _kind_label(entry: Entry) -> str: + if entry.kind == "voice": + noun = "Conversation" if entry.mode == "dialogue" else "Voice note" + length = _duration(entry.duration_ms) + return f"{noun}, {length}" if length else noun + return {"image": "Photo", "text": "Written note"}.get(entry.kind, entry.kind) + + +def _permalink(room_id: str, event_id: str) -> str: + return f"https://matrix.to/#/{room_id}/{event_id}" + + +def _entry_block(entry: Entry, *, room_id: str) -> str: + """One entry: who, what it was, and then their words untouched.""" + who = entry.sender.title() + # An addressee is rendered as the message names them ("Bart", "kids"), + # not title-cased, so a group reads as a group. A message whose + # addressee resolves to its own sender is a misread, not a dedication. + to = (entry.addressee or "").strip() + if to and to.lower() != entry.sender.lower(): + heading = f"### {who} — for {to}" + else: + heading = f"### {who}" + + meta = [_kind_label(entry)] + if entry.confidence != "uncertain": + meta.append(entry.basis) + lines = [heading, f"*{' · '.join(m for m in meta if m)}*", ""] + + if entry.confidence == "uncertain": + lines += [ + "> [!warning] When this happened is not recoverable", + f"> {entry.basis.capitalize()}.", + "", + ] + + if entry.body.strip(): + lines += [entry.body.strip(), ""] + elif entry.kind == "image" and not entry.comments: + lines += ["No caption came with this one.", ""] + + for who_replied, text in entry.comments: + lines += [f"> [!quote] {who_replied.title()} replied", ] + lines += [f"> {line}" for line in text.strip().splitlines()] + lines.append("") + + if room_id and entry.event_ids: + label = {"voice": "Listen in the room", "image": "See it in the room"} + lines.append( + f"[{label.get(entry.kind, 'Open in the room')}]" + f"({_permalink(room_id, entry.event_ids[0])})" + ) + lines.append("") + + return "\n".join(lines).rstrip() + + +def month_key(on: date) -> str: + return on.strftime("%Y-%m") + + +def render_month(entries, *, room_id: str = "") -> str: + """A month of entries, grouped by the day they happened. + + Entries whose date could not be recovered are still shown on the day + they surfaced, under a heading that says as much. Hiding them would + lose the memory to protect the timeline, which is the wrong trade for + a diary. + """ + if not entries: + return "No entries yet." + + title = entries[0].on.strftime("%B %Y") + count = len(entries) + lines = [ + f"# {title}", + "", + f"{count} {'moment' if count == 1 else 'moments'} from the family's " + "memories room, in the words they were recorded in. Nothing on this " + "page has been summarised.", + "", + ] + + current: date | None = None + for entry in entries: + if entry.on != current: + current = entry.on + # The heading is the day. An entry that is unsure of its + # date says so in its own block -- putting "week of" in the + # heading would cast that doubt over every other entry + # filed the same day. + lines += [f"## {entry.on.strftime('%A, %-d %B')}", ""] + lines += [_entry_block(entry, room_id=room_id), ""] + + return "\n".join(lines).rstrip() + "\n" + + +def render_index(entries) -> str: + """The diary's front door: what it is, and a way into every month.""" + lines = [ + "# Family Diary", + "", + "Everything the family has put in the memories room: voice notes, " + "photos, conversations someone hit record on. Entries are quoted " + "exactly as they were said or written.", + "", + ] + if not entries: + lines += ["Nothing has been compiled yet.", ""] + return "\n".join(lines) + + by_month: dict[str, list[Entry]] = {} + for entry in entries: + by_month.setdefault(month_key(entry.on), []).append(entry) + + lines += ["## Months", ""] + for key in sorted(by_month, reverse=True): + month = by_month[key] + label = month[0].on.strftime("%B %Y") + n = len(month) + lines.append(f"- [{label}]({key}) — {n} {'entry' if n == 1 else 'entries'}") + lines.append("") + + unsure = [e for e in entries if e.confidence == "uncertain"] + if unsure: + n = len(unsure) + lines += [ + "## Dates we could not recover", + "", + f"{n} {'entry' if n == 1 else 'entries'} arrived in a sync burst " + "without a spoken date. They are filed under the week they " + "surfaced and marked on their page. Saying the date aloud at the " + "start of a recording is what prevents this.", + "", + ] + + return "\n".join(lines).rstrip() + "\n" + + +def pages_for(entries, *, room_id: str = "") -> list[tuple[str, str, str]]: + """Every page the diary publishes: (path, body, title). + + Paths are relative to the shared bucket, which the caller prefixes -- + the bucket is named in config (`family`, `office`, a surname) and + this module has no business knowing which. + """ + by_month: dict[str, list[Entry]] = {} + for entry in entries: + by_month.setdefault(month_key(entry.on), []).append(entry) + + out = [(f"{DIARY_DIR}/index.md", render_index(entries), "Family Diary")] + for key, month in sorted(by_month.items()): + out.append(( + f"{DIARY_DIR}/{key}.md", + render_month(month, room_id=room_id), + f"Diary: {month[0].on.strftime('%B %Y')}", + )) + return out diff --git a/stacklets/memory/cli/diary.py b/stacklets/memory/cli/diary.py new file mode 100644 index 0000000..f205af4 --- /dev/null +++ b/stacklets/memory/cli/diary.py @@ -0,0 +1,49 @@ +"""stack memory diary — compile the memories room into the family diary. + +The memories room is where a family records things for its future self: +voice notes to a child, a photo with a caption, a dinner conversation +someone hit record on. This reads the whole room back and publishes it +as diary pages in the wiki, in the words it was recorded in. + + stack memory diary compile and publish + stack memory diary --dry-run print the pages, write nothing + stack memory diary --room memories read a different room + stack memory diary --burst-window 1 tighten sync-burst detection + +WHAT IT RECOVERS + Matrix stamps an event with the time the server received it, never + the time it was recorded. A memo made on a walk and synced three + days later carries the sync date, which would file it under the + wrong week forever. So the compiler prefers a date spoken inside the + recording ("today is the sixteenth") over the timestamp, falls back + to the timestamp for anything sent live, and refuses to guess for a + recording that synced late without saying its own date -- those are + filed under the week they surfaced and marked as unrecovered on the + page. + + The habit that prevents the third case costs nothing: say the date + at the start of the recording. + +NOTHING IS SUMMARISED + The model is asked to read each message, never to rewrite one. What + lands on the page is what was said. Transcription and reading both + run on the local AI stacklet; the room's contents never leave the + box. + +Runs inside `stack-core-bot-runner` (it has the whisper client, the LLM +client, and the brain working copy); this is a thin docker-exec, the +same shape as `stack memory wiki`. +""" + +HELP = "Compile the memories room into the family diary" + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) +from _common import dispatch # noqa: E402 + + +def run(args, stacklet, config): + argv = sys.argv[3:] + return dispatch("diary", *argv) diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py new file mode 100644 index 0000000..187d373 --- /dev/null +++ b/tests/stacklets/test_memory_diary.py @@ -0,0 +1,448 @@ +"""`stack memory diary` — what the compiler makes of a messy room. + +The memories room is the hardest input in the stack: recordings arrive +days after they were made, one memo gets split across two files, a +caption trails its photo, and the only statement of when something +happened is a sentence spoken inside the audio. This file pins what the +compiler does about that. + +Ground truth comes from `tools/family-memories/spec.en.yaml`, which was +authored to describe a real room's patterns and carries `true_date` and +`date_source` for every item. Asserting against that file rather than +against hand-written fixtures is deliberate: a fixture written next to +the compiler would only prove the two agree with each other. + +The model's own accuracy is not tested here. `Reading`s are supplied +directly, so these tests pin what the compiler does *given* a reading -- +whether the model produces a good one is a question for the rig, where +a real transcript meets a real model. +""" + +from __future__ import annotations + +import sys +from dataclasses import replace +from datetime import date, datetime, timezone +from pathlib import Path + +import pytest +import yaml + +_REPO_ROOT = Path(__file__).resolve().parent.parent.parent +sys.path.insert(0, str(_REPO_ROOT / "stacklets" / "memory" / "bot")) +sys.path.insert(0, str(_REPO_ROOT / "tools" / "family-memories")) + +import diary # noqa: E402 +from ingest import burst_ordered # noqa: E402 + +SPEC = _REPO_ROOT / "tools" / "family-memories" / "spec.en.yaml" + +# The room the corpus replays into. Burst members land back-to-back; +# everything else is spaced by the replay's own delay. Both are far +# tighter than a real room, which is exactly why the compiler takes the +# burst window as a parameter instead of assuming one. +SYNC_GAP_MS = 80 +LIVE_GAP_MS = 2_100 +WINDOW_S = 1.0 +REPLAYED_ON = datetime(2026, 9, 12, 8, tzinfo=timezone.utc) +BASE_TS = int(REPLAYED_ON.timestamp() * 1000) + + +# ── Building a room ─────────────────────────────────────────────────── + + +def _spec_items() -> list[dict]: + return yaml.safe_load(SPEC.read_text())["items"] + + +def _room_from_spec(items: list[dict]) -> list[dict]: + """The Matrix events the corpus produces, without a homeserver. + + Mirrors `ingest.py`: burst members contiguous, an edit sent as a + separate `m.replace`, a caption as a reply. Timings follow the + replay, not the items' true dates -- recovering those is the + compiler's job, and handing them to it would test nothing. + """ + events: list[dict] = [] + ids: dict[str, str] = {} + ts = BASE_TS + prev_burst = None + + for item in burst_ordered(items): + burst = item.get("burst") + ts += SYNC_GAP_MS if (burst and burst == prev_burst) else LIVE_GAP_MS + prev_burst = burst + + event_id = ids[item["id"]] = f"${item['id']}" + kind = item["kind"] + if kind in ("voice", "dialogue"): + content = {"msgtype": "m.audio", "body": f"{item['id']}.wav", + "url": f"mxc://test/{item['id']}", + "info": {"duration": 9000}} + elif kind == "image": + content = {"msgtype": "m.image", "body": f"{item['id']}.png", + "url": f"mxc://test/{item['id']}"} + else: + content = {"msgtype": "m.text", "body": item["text"].strip()} + + if target := item.get("reply_to"): + content["m.relates_to"] = {"m.in_reply_to": {"event_id": ids[target]}} + + events.append({"type": "m.room.message", "event_id": event_id, + "sender": f"@{item['sender']}:test", + "origin_server_ts": ts, "content": content}) + + if edit := item.get("edit_text"): + ts += LIVE_GAP_MS + events.append({ + "type": "m.room.message", "event_id": f"{event_id}#edit", + "sender": f"@{item['sender']}:test", "origin_server_ts": ts, + "content": { + "msgtype": "m.text", "body": "* " + edit.strip(), + "m.new_content": {"msgtype": "m.text", "body": edit.strip()}, + "m.relates_to": {"rel_type": "m.replace", + "event_id": event_id}}, + }) + return events + + +def _readings_from_spec(items: list[dict]) -> dict[str, diary.Reading]: + """The reading a correct model would return for each item. + + `date_source: spoken` means the recording says its own date, so the + reading carries it; anything else leaves it null and the compiler + has to fall back. Fragment halves are marked where the spec says the + recording was cut. + """ + readings = {} + for item in items: + source = item["date_source"] + fragment = item.get("fragment_of") + first_half = fragment and item["id"].endswith("-a") + second_half = fragment and not item["id"].endswith("-a") + readings[f"${item['id']}"] = diary.Reading( + mode="note" if item["kind"] == "text" else "monologue", + spoken_date=(item["true_date"].isoformat() + if source == "spoken" else None), + starts_mid_thought=bool(second_half), + ends_mid_thought=bool(first_half), + ) + return readings + + +def _words(item: dict) -> str: + """What whisper would return: a memo's text, a dialogue's turns.""" + if turns := item.get("turns"): + return " ".join(t["text"].strip() for t in turns) + return item.get("text", "").strip() + + +def _transcribed(messages, items): + """Stand in for whisper: a recording's body becomes its words. + + The compiler never sees a filename where a transcript belongs -- + `cli/diary.py` substitutes the decoded text before compiling, so the + harness does too. Doing it here rather than in `_room_from_spec` + keeps `resolve` under test against the events a room really holds. + """ + spoken = {f"${i['id']}": _words(i) for i in items} + return [m if m.kind != "voice" or not spoken.get(m.event_id) + else replace(m, body=spoken[m.event_id]) + for m in messages] + + +def _compile(items=None): + items = items if items is not None else _spec_items() + messages = diary.resolve(_room_from_spec(items), burst_window_s=WINDOW_S) + messages = _transcribed(messages, items) + return diary.compile_entries(messages, _readings_from_spec(items)) + + +def _entry_for(entries, item_id: str) -> diary.Entry: + for entry in entries: + if f"${item_id}" in entry.event_ids: + return entry + raise AssertionError(f"no entry carries {item_id}") + + +def _msg(event_id="$a", sender="marge", ts=BASE_TS, kind="voice", **kw): + return diary.Message(event_id=event_id, sender=sender, ts=ts, + kind=kind, **kw) + + +# ── Matrix mechanics ────────────────────────────────────────────────── + + +class TestResolve: + """What the room means before anyone reads it.""" + + def test_an_edited_message_keeps_only_its_final_wording(self): + events = _room_from_spec(_spec_items()) + messages = diary.resolve(events, burst_window_s=WINDOW_S) + + edited = next(m for m in messages if m.event_id == "$text-tagebuch") + assert "Maggie's First March" in edited.body + assert not any(m.event_id.endswith("#edit") for m in messages), \ + "an edit is a correction to a message, not a message" + + def test_a_redacted_message_leaves_no_trace(self): + events = [{"type": "m.room.message", "event_id": "$gone", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {}}] + assert diary.resolve(events) == [] + + def test_a_reply_does_not_quote_its_parent_back(self): + body = "> <@marge:test> the original memo\n\nHe gets that from me." + events = [{"type": "m.room.message", "event_id": "$r", + "sender": "@homer:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.text", "body": body}}] + + assert diary.resolve(events)[0].body == "He gets that from me." + + def test_a_bare_upload_has_no_caption(self): + """An image's `body` is its filename until a caption displaces it.""" + events = [{"type": "m.room.message", "event_id": "$i", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.image", "body": "IMG_4021.png", + "url": "mxc://test/x"}}] + + assert diary.resolve(events)[0].body == "" + + def test_a_captioned_upload_keeps_the_caption(self): + events = [{"type": "m.room.message", "event_id": "$i", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.image", "body": "Maggie's drawing", + "filename": "IMG_4021.png", + "url": "mxc://test/x"}}] + + assert diary.resolve(events)[0].body == "Maggie's drawing" + + +class TestBursts: + """Which timestamps are arrival times rather than recording times.""" + + def test_a_run_of_uploads_from_one_sender_is_a_burst(self): + run = [_msg(event_id=f"$m{i}", ts=BASE_TS + i * SYNC_GAP_MS) + for i in range(3)] + + marked = diary.mark_bursts(run, window_s=WINDOW_S) + + assert len({m.burst for m in marked}) == 1 + assert all(m.burst for m in marked) + + def test_a_message_on_its_own_is_not_a_burst(self): + alone = [_msg(event_id="$a"), _msg(event_id="$b", ts=BASE_TS + 60_000)] + + assert [m.burst for m in diary.mark_bursts(alone, window_s=WINDOW_S)] \ + == [None, None] + + def test_a_different_sender_ends_the_run(self): + run = [_msg(event_id="$a", sender="marge"), + _msg(event_id="$b", sender="homer", ts=BASE_TS + SYNC_GAP_MS)] + + assert [m.burst for m in diary.mark_bursts(run, window_s=WINDOW_S)] \ + == [None, None] + + def test_the_window_decides_where_a_run_stops(self): + """The same room splits differently under a different window. + + This is the property that makes the window a parameter: a + replayed corpus and a real room disagree about what "together" + means by three orders of magnitude. + """ + run = [_msg(event_id="$a"), _msg(event_id="$b", ts=BASE_TS + 5_000)] + + assert all(m.burst for m in diary.mark_bursts(run, window_s=10)) + assert not any(m.burst for m in diary.mark_bursts(run, window_s=1)) + + +# ── The trap the corpus was built to set ────────────────────────────── + + +class TestJoiningFragments: + """One recording split in two looks exactly like two recordings.""" + + def test_halves_that_agree_become_one_recording(self): + entries = _compile() + + joined = _entry_for(entries, "fragment-urlaub-a") + assert "$fragment-urlaub-b" in joined.event_ids + assert joined.body.startswith("Hi kids, today is March twenty-second") + assert joined.body.endswith("that's our surprise.") + + def test_a_burst_of_separate_memos_is_not_joined(self): + """Three same-second uploads are usually three memos, not one. + + Timing alone cannot tell the two apart, so only the words may + join them. A compiler that joined on adjacency would merge a + month of the family's memos into one paragraph. + """ + items = [i for i in _spec_items() if i.get("burst") == "sync-1"] + entries = _compile(items) + + assert len(entries) == 3 + assert all(len(e.event_ids) == 1 for e in entries) + + +# ── Dating ──────────────────────────────────────────────────────────── + + +class TestDating: + """Which of the two clocks to believe, and when to admit neither.""" + + def test_a_spoken_date_beats_the_servers_timestamp(self): + sent_in_september = _msg(ts=BASE_TS, burst="burst-1") + reading = diary.Reading(spoken_date="2026-03-16") + + on, confidence, _basis = diary.date_for(sent_in_september, reading) + + assert on == date(2026, 3, 16) + assert confidence == "spoken" + + def test_a_message_sent_live_is_dated_from_its_timestamp(self): + on, confidence, _ = diary.date_for(_msg(), diary.Reading()) + + assert on == datetime.fromtimestamp( + BASE_TS / 1000, timezone.utc).date() + assert confidence == "sent" + + def test_a_silent_memo_in_a_burst_admits_it_does_not_know(self): + """The one case where the compiler must not produce a date. + + Its timestamp is known to be days off and it never said when it + was made, so any date here would be invented. A family diary + with a confidently wrong date is worse than one with a gap. + """ + on, confidence, basis = diary.date_for( + _msg(burst="burst-1"), diary.Reading()) + + assert confidence == "uncertain" + assert "sync burst" in basis + assert on == datetime.fromtimestamp( + BASE_TS / 1000, timezone.utc).date() + + +class TestAgainstCorpusGroundTruth: + """Scored against the dates the corpus says are true.""" + + def test_every_spoken_date_is_recovered(self): + items = _spec_items() + entries = _compile(items) + + spoken = [i for i in items if i["date_source"] == "spoken"] + assert spoken, "the corpus should carry spoken-date items" + for item in spoken: + entry = _entry_for(entries, item["id"]) + assert entry.on == item["true_date"], item["id"] + assert entry.confidence == "spoken", item["id"] + + def test_a_fragments_second_half_inherits_the_spoken_date(self): + items = _spec_items() + inherited = next(i for i in items + if i["date_source"] == "inherited-from-fragment") + + entry = _entry_for(_compile(items), inherited["id"]) + + assert entry.on == inherited["true_date"] + + def test_the_unrecoverable_memo_is_the_only_uncertain_entry(self): + items = _spec_items() + entries = _compile(items) + + unrecoverable = [i["id"] for i in items + if i["date_source"] == "unrecoverable"] + uncertain = [e for e in entries if e.confidence == "uncertain"] + + assert len(uncertain) == len(unrecoverable) + for item_id in unrecoverable: + assert _entry_for(entries, item_id).confidence == "uncertain" + + +# ── Composing entries ───────────────────────────────────────────────── + + +class TestEntries: + """What ends up being one thing on the page.""" + + def test_a_late_caption_belongs_to_its_photo(self): + entries = _compile() + + photo = _entry_for(entries, "bild-zeichnung") + assert photo.kind == "image" + assert photo.comments, "the caption should hang off the photo" + assert "Maggie drew this today" in photo.comments[0][1] + with pytest.raises(AssertionError): + # It is not an entry of its own. + assert _entry_for(entries, "bild-zeichnung-caption").kind == "text" + + def test_a_reply_to_a_memo_is_filed_under_that_memo(self): + entries = _compile() + + memo = _entry_for(entries, "memo-bart-zeugnis") + assert [who for who, _ in memo.comments] == ["homer"] + assert memo.on == date(2026, 3, 16), \ + "a reply months later must not move the memo's date" + + def test_entries_of_one_day_keep_the_order_the_room_had(self): + entries = _compile() + same_day = [e for e in entries if e.on == REPLAYED_ON.date()] + + assert same_day, "the replay files undated items on its own date" + assert [e.at for e in same_day] == sorted(e.at for e in same_day) + + +# ── The page ────────────────────────────────────────────────────────── + + +class TestRendering: + """The diary quotes; it never summarises.""" + + def test_an_uncertain_entry_says_so_where_it_is_read(self): + entries = _compile() + uncertain = next(e for e in entries if e.confidence == "uncertain") + + page = diary.render_month([uncertain]) + + assert "[!warning]" in page + assert "sync burst" in page + + def test_a_photo_without_a_caption_says_so_rather_than_naming_a_file(self): + page = diary.render_month([diary.Entry( + on=date(2026, 4, 3), confidence="sent", basis="dated from when it " + "was sent", kind="image", sender="homer", body="")]) + + assert "No caption came with this one." in page + assert ".png" not in page + + def test_a_speaker_is_not_addressed_to_themselves(self): + """A misread addressee must not become a dedication.""" + page = diary.render_month([diary.Entry( + on=date(2026, 3, 28), confidence="sent", basis="b", kind="voice", + sender="homer", body="words", addressee="Homer")]) + + assert "for Homer" not in page + assert "### Homer" in page + + def test_the_words_reach_the_page_unaltered(self): + spoken = ("Hi Bart, today is March 16th. I won't forget that.") + page = diary.render_month([diary.Entry( + on=date(2026, 3, 16), confidence="spoken", basis="b", kind="voice", + sender="marge", body=spoken, addressee="Bart")]) + + assert spoken in page + + def test_the_index_lists_a_page_for_every_month(self): + entries = _compile() + + index = diary.render_index(entries) + + for key in {diary.month_key(e.on) for e in entries}: + assert f"]({key})" in index + + def test_pages_are_named_for_the_months_they_cover(self): + pages = diary.pages_for(_compile()) + paths = [path for path, _body, _title in pages] + + assert "diary/index.md" in paths + assert "diary/2026-03.md" in paths + assert all(p.startswith("diary/") for p in paths) From ccc4b3a36afdb71645b4e8a1d6014611c0ea8362 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 14:40:54 +0200 Subject: [PATCH 05/43] feat(memory): give the diary a shape that outlives its first year Months now nest under their year, and each one opens by recalling what happened rather than counting what is on the page. - Root, year and month pages: a year names its months and who recorded them, a month holds the entries - Each month opens with a paragraph the local model writes from that month's entries. It may surface, never replace: the entries under it stay word for word, and the promise is made once on the front page instead of restated on every month - Only the sender is shown to the summariser, never the classifier's guess at who a memo was for, so one reading's mistake cannot become a person in the prose - Folder pages are `about.md` like every other entity here, because a folder URL renders no body in this wiki - Sort wiki pages named for a date by that date, so a year reads March, April, September rather than alphabetically --- stacklets/memory/bot/cli/diary.py | 99 +++++++++++++- stacklets/memory/bot/diary.py | 157 +++++++++++++++++------ stacklets/memory/quartz/quartz.layout.ts | 25 +++- tests/stacklets/test_memory_diary.py | 86 +++++++++++-- 4 files changed, 320 insertions(+), 47 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 6d10caf..85dc63a 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -248,6 +248,94 @@ async def _read(message, llm) -> diary.Reading: ) +# ── Recalling a month ───────────────────────────────────────────────── +# +# The one piece of writing on a diary page that is not the family's own. +# It opens a month and it is allowed to be warm, but it may not invent: +# the entries underneath are the record, and a summary that adds to them +# is a lie told about someone's childhood. + + +_SUMMARY_PROMPT = """\ +Write the opening paragraph of a family's diary page for {month}. Below +are that month's entries, quoted exactly as the family recorded them. + +Write two to four sentences recalling what happened that month, the way +someone in the family would remember it later. + +Rules: +- Use only what the entries say. Never add an event, a feeling, a place + or an outcome that is not in them. +- Keep every detail with the person the entry keeps it with. Do not move + something one child did onto another child. +- Keep the direction of what happened. If one person did something for, + to, or about another, do not swap them round. +- Name people as the entries name them. +- Report what the entries report, and no more. Do not frame the month as + an occasion, and do not describe an event the entries only mention in + passing as though the family gathered for it. +- An entry marked "date unknown" happened at no stated time. Do not give + it one. +- Plain, warm, specific. No marketing words. Never write "heartwarming", + "cherished", "precious", "journey", "chapter", or a closing sentence + about what the month meant. +- Return the paragraph and nothing else: no heading, no list, no + preamble, no quotation marks around it. + +Entries: +{evidence} +""" + + +def _evidence(entries) -> str: + """A month's entries as the summariser sees them. + + Each line names who recorded it and when, because attribution is the + thing the model gets wrong: without the date and the sender pinned to + the words, a summary quietly reassigns a first tooth to the wrong + child. + """ + out = [] + for entry in entries: + when = ("date unknown" if entry.confidence == "uncertain" + else entry.on.strftime("%-d %B")) + # Sender only. The addressee is the classifier's reading, and + # feeding one generation's guess into another compounds it: a + # nickname misread as a third person becomes a third person in + # the prose. Whoever a memo is spoken to is named in its words + # anyway, where the model can read it as evidence. + who = entry.sender.title() + body = entry.body.strip() or "(a photo, no caption)" + for sender, text in entry.comments: + body += f"\n {sender.title()} replied: {text.strip()}" + out.append(f"- [{when}] {who}: {body}") + return "\n".join(out) + + +async def _summarise(entries, llm) -> str: + """A paragraph recalling one month, or "" if the model cannot. + + Temperature 0, so a recompile of an unchanged month reads the same + way. A failure returns empty and the page falls back to its factual + opening -- a diary missing its introduction is fine, a diary whose + introduction changes wording every night is not. + """ + month = entries[0].on.strftime("%B %Y") + prompt = _SUMMARY_PROMPT.format(month=month, evidence=_evidence(entries)) + try: + text = await llm.complete("writer", prompt, temperature=0) + except LLMError as e: + _err(f" could not summarise {month}: {e}") + return "" + # A model that answers with a heading or a bulleted list has ignored + # the brief; the opening is prose or it is nothing. + cleaned = " ".join(text.strip().split()) + if cleaned.startswith(("#", "-", "*")): + _err(f" discarded a non-prose summary for {month}") + return "" + return cleaned + + # ── The command ─────────────────────────────────────────────────────── @@ -323,7 +411,16 @@ async def run(llm, argv: list[str]) -> int: entries = diary.compile_entries(decoded, readings) _err(f"{len(entries)} diary entr{'y' if len(entries) == 1 else 'ies'}") - pages = diary.pages_for(entries, room_id=room_id) + months: dict[str, list] = {} + for entry in entries: + key = f"{diary.year_key(entry.on)}-{diary.month_key(entry.on)}" + months.setdefault(key, []).append(entry) + + summaries = {} + for key, in_month in sorted(months.items()): + summaries[key] = await _summarise(in_month, llm) + + pages = diary.pages_for(entries, room_id=room_id, summaries=summaries) if dry_run: for path, body, _title in pages: print(f"\n{'=' * 70}\n{bucket}/{path}\n{'=' * 70}\n{body}") diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index c4a00a4..ec8df59 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -469,13 +469,61 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: return "\n".join(lines).rstrip() +def year_key(on: date) -> str: + return on.strftime("%Y") + + def month_key(on: date) -> str: - return on.strftime("%Y-%m") + """A month's own slug, which is its number. + + The year is already the folder, so the file is `03.md` rather than + `2026-03.md`: it reads as `/diary/2026/03`, and numbering sorts the + explorer chronologically where month names would sort April before + March. + """ + return on.strftime("%m") + + +def _by_year(entries) -> "dict[str, list[Entry]]": + out: dict[str, list[Entry]] = {} + for entry in entries: + out.setdefault(year_key(entry.on), []).append(entry) + return out -def render_month(entries, *, room_id: str = "") -> str: +def _by_month(entries) -> "dict[str, list[Entry]]": + out: dict[str, list[Entry]] = {} + for entry in entries: + out.setdefault(month_key(entry.on), []).append(entry) + return out + + +def _recorded_by(entries) -> list[str]: + """Who captured a year's entries, in order of first appearance. + + Senders only. A sender is a Matrix account and is therefore a fact; + an addressee is the model's reading of who was spoken to, and a + landing page is the wrong place for a guess -- it reads as a roster + of the household. Addressees stay on the entries themselves, where + a misread is visible next to the words that caused it. + """ + seen: list[str] = [] + for entry in entries: + name = (entry.sender or "").strip() + pretty = name.title() if name.islower() else name + if pretty and pretty not in seen: + seen.append(pretty) + return seen + + +def render_month(entries, *, room_id: str = "", summary: str = "") -> str: """A month of entries, grouped by the day they happened. + `summary` is an optional paragraph recalling the month, and the one + piece of writing here that is not the family's own. It opens the + page; everything under it is verbatim. That promise is made once, on + the diary's front page, rather than restated on every month. + Entries whose date could not be recovered are still shown on the day they surfaced, under a heading that says as much. Hiding them would lose the memory to protect the timeline, which is the wrong trade for @@ -484,16 +532,9 @@ def render_month(entries, *, room_id: str = "") -> str: if not entries: return "No entries yet." - title = entries[0].on.strftime("%B %Y") - count = len(entries) - lines = [ - f"# {title}", - "", - f"{count} {'moment' if count == 1 else 'moments'} from the family's " - "memories room, in the words they were recorded in. Nothing on this " - "page has been summarised.", - "", - ] + lines = [f"# {entries[0].on.strftime('%B %Y')}", ""] + if summary.strip(): + lines += [summary.strip(), ""] current: date | None = None for entry in entries: @@ -509,8 +550,35 @@ def render_month(entries, *, room_id: str = "") -> str: return "\n".join(lines).rstrip() + "\n" +def render_year(entries) -> str: + """A year's landing page: its months, and who is in them. + + Deterministic. Counting entries and naming the people who appear is + reading, not summarising, so nothing here needs a model and nothing + here can drift between runs. + """ + year = entries[0].on.strftime("%Y") + lines = [f"# {year}", ""] + + n = len(entries) + people = _recorded_by(entries) + opening = f"{n} {'entry' if n == 1 else 'entries'} this year" + if people: + opening += f", recorded by {_and_list(people)}" + lines += [opening + ".", "", "## Months", ""] + + for key, month in sorted(_by_month(entries).items()): + label = month[0].on.strftime("%B") + count = len(month) + lines.append( + f"- [{label}]({key}) — {count} {'entry' if count == 1 else 'entries'}") + lines.append("") + + return "\n".join(lines).rstrip() + "\n" + + def render_index(entries) -> str: - """The diary's front door: what it is, and a way into every month.""" + """The diary's front door: what it is, and a way into every year.""" lines = [ "# Family Diary", "", @@ -523,16 +591,14 @@ def render_index(entries) -> str: lines += ["Nothing has been compiled yet.", ""] return "\n".join(lines) - by_month: dict[str, list[Entry]] = {} - for entry in entries: - by_month.setdefault(month_key(entry.on), []).append(entry) - - lines += ["## Months", ""] - for key in sorted(by_month, reverse=True): - month = by_month[key] - label = month[0].on.strftime("%B %Y") - n = len(month) - lines.append(f"- [{label}]({key}) — {n} {'entry' if n == 1 else 'entries'}") + lines += ["## Years", ""] + for key, year in sorted(_by_year(entries).items(), reverse=True): + count = len(year) + months = len(_by_month(year)) + lines.append( + f"- [{key}]({key}/about) — {count} " + f"{'entry' if count == 1 else 'entries'} across {months} " + f"{'month' if months == 1 else 'months'}") lines.append("") unsure = [e for e in entries if e.confidence == "uncertain"] @@ -551,22 +617,41 @@ def render_index(entries) -> str: return "\n".join(lines).rstrip() + "\n" -def pages_for(entries, *, room_id: str = "") -> list[tuple[str, str, str]]: +def _and_list(names: list[str]) -> str: + if len(names) == 1: + return names[0] + return ", ".join(names[:-1]) + f" and {names[-1]}" + + +def pages_for(entries, *, room_id: str = "", + summaries: "dict[str, str] | None" = None, + ) -> list[tuple[str, str, str]]: """Every page the diary publishes: (path, body, title). - Paths are relative to the shared bucket, which the caller prefixes -- - the bucket is named in config (`family`, `office`, a surname) and - this module has no business knowing which. - """ - by_month: dict[str, list[Entry]] = {} - for entry in entries: - by_month.setdefault(month_key(entry.on), []).append(entry) + Three levels, because a diary outlives its first year: the root + names the years, a year names its months, and a month holds the + entries. Breadcrumbs come free from the path. - out = [(f"{DIARY_DIR}/index.md", render_index(entries), "Family Diary")] - for key, month in sorted(by_month.items()): + A folder's own page is `about.md`, not `index.md`, matching every + other entity in this wiki. That convention exists for a reason: + Quartz serves a folder URL through its folder-page layout, which + renders no body here, so an `index.md` would be a page whose + contents nobody can read. + + Paths are relative to the shared bucket, which the caller prefixes + -- the bucket is named in config (`family`, `office`, a surname) + and this module has no business knowing which. + """ + out = [(f"{DIARY_DIR}/about.md", render_index(entries), "Family Diary")] + for year, in_year in sorted(_by_year(entries).items()): out.append(( - f"{DIARY_DIR}/{key}.md", - render_month(month, room_id=room_id), - f"Diary: {month[0].on.strftime('%B %Y')}", + f"{DIARY_DIR}/{year}/about.md", render_year(in_year), year, )) + for month, in_month in sorted(_by_month(in_year).items()): + out.append(( + f"{DIARY_DIR}/{year}/{month}.md", + render_month(in_month, room_id=room_id, + summary=(summaries or {}).get(f"{year}-{month}", "")), + in_month[0].on.strftime("%B %Y"), + )) return out diff --git a/stacklets/memory/quartz/quartz.layout.ts b/stacklets/memory/quartz/quartz.layout.ts index bb1e0e7..447e6a5 100644 --- a/stacklets/memory/quartz/quartz.layout.ts +++ b/stacklets/memory/quartz/quartz.layout.ts @@ -8,6 +8,7 @@ */ import { PageLayout, SharedLayout } from "./quartz/cfg" +import { FileTrieNode } from "./quartz/util/fileTrie" import * as Component from "./quartz/components" // Our own components, imported directly rather than through the // `Component` namespace so we do not have to overlay upstream's @@ -19,6 +20,26 @@ import Welcome from "./quartz/components/Welcome" // `CODE_URL` is set in the container env from {code_url} — the // user-facing Forgejo URL. Empty falls back to a `#` placeholder so // the footer still renders even if env wiring drifts. +// Dated pages are named for their number, not their title: a diary's +// March lives at `2026/03`. Upstream's explorer sorts files by display +// title, which files April above March and makes a year read as +// nonsense. Compare numeric names as numbers and leave everything else +// on upstream's alphabetical order, so this only ever reorders folders +// whose pages are named for a date. +// +// Quartz serialises this function with `toString()` and re-evaluates it +// in the browser, so it must not reference anything outside itself. +const sortByDateThenTitle = (a: FileTrieNode, b: FileTrieNode): number => { + if (a.isFolder !== b.isFolder) return a.isFolder ? -1 : 1 + const an = a.slugSegment + const bn = b.slugSegment + if (/^\d+$/.test(an) && /^\d+$/.test(bn)) return Number(an) - Number(bn) + return a.displayName.localeCompare(b.displayName, undefined, { + numeric: true, + sensitivity: "base", + }) +} + const codeUrl = process.env.CODE_URL || "" const repoUrl = codeUrl ? `${codeUrl.replace(/\/$/, "")}/family/memory` : "#" @@ -64,7 +85,7 @@ export const defaultContentPageLayout: PageLayout = { { Component: Component.ReaderMode() }, ], }), - Component.Explorer(), + Component.Explorer({ sortFn: sortByDateThenTitle }), ], right: [ Component.Graph(), @@ -88,7 +109,7 @@ export const defaultListPageLayout: PageLayout = { Component.Flex({ components: [{ Component: Component.Search(), grow: true }], }), - Component.Explorer(), + Component.Explorer({ sortFn: sortByDateThenTitle }), ], right: [], } diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 187d373..54afc20 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -431,18 +431,88 @@ def test_the_words_reach_the_page_unaltered(self): assert spoken in page - def test_the_index_lists_a_page_for_every_month(self): + def test_a_month_opens_with_its_summary(self): + entries = _compile() + march = [e for e in entries if e.on.month == 3] + + page = diary.render_month(march, summary="A month of firsts.") + + assert page.startswith("# March 2026\n\nA month of firsts.") + + def test_a_month_without_a_summary_goes_straight_to_its_entries(self): + """The summary is the only writing here that is not the family's, + so its absence leaves the page shorter, never padded with + boilerplate the reader sees on every other month.""" + entries = _compile() + march = [e for e in entries if e.on.month == 3] + + page = diary.render_month(march) + + assert page.startswith("# March 2026\n\n## ") + assert "memories room" not in page + + def test_a_summary_reaches_the_month_it_describes(self): + pages = {path: body for path, body, _title in + diary.pages_for(_compile(), + summaries={"2026-03": "Only March."})} + + assert "Only March." in pages["diary/2026/03.md"] + assert "Only March." not in pages["diary/2026/04.md"] + + def test_the_index_lists_a_page_for_every_year(self): entries = _compile() index = diary.render_index(entries) - for key in {diary.month_key(e.on) for e in entries}: - assert f"]({key})" in index + for key in {diary.year_key(e.on) for e in entries}: + assert f"]({key}/about)" in index + + def test_a_year_lists_its_months_and_who_is_in_them(self): + entries = _compile() + in_2026 = [e for e in entries if e.on.year == 2026] - def test_pages_are_named_for_the_months_they_cover(self): - pages = diary.pages_for(_compile()) - paths = [path for path, _body, _title in pages] + page = diary.render_year(in_2026) - assert "diary/index.md" in paths - assert "diary/2026-03.md" in paths + assert "## Months" in page + assert "[March](03)" in page + assert "recorded by Marge and Homer" in page + + def test_a_year_credits_only_who_recorded(self): + """An addressee is the model's reading, not a fact about the + household, so it never reaches a landing page.""" + page = diary.render_year([diary.Entry( + on=date(2026, 3, 16), confidence="spoken", basis="b", kind="voice", + sender="marge", body="words", addressee="Bart")]) + + assert "recorded by Marge" in page + assert "Bart" not in page + + def test_a_diary_nests_month_inside_year(self): + """A diary outlives its first year, so months live under one. + + Flat `2026-03.md` files pile every month of every year into one + folder, which is the explorer sidebar the family actually reads. + """ + paths = [path for path, _body, _title in diary.pages_for(_compile())] + + assert "diary/about.md" in paths + assert "diary/2026/about.md" in paths + assert "diary/2026/03.md" in paths assert all(p.startswith("diary/") for p in paths) + + def test_a_folders_own_page_is_about_not_index(self): + """Quartz renders a folder URL through a layout with no body in + this wiki, so an `index.md` would be unreadable. Every other + entity here is `about.md` for the same reason.""" + paths = [path for path, _body, _title in diary.pages_for(_compile())] + + assert not any(p.endswith("index.md") for p in paths) + + def test_a_month_page_is_titled_with_its_year(self): + """The folder gives context in the sidebar; a link or a search + result does not, so the title carries the year itself.""" + titles = {path: title for path, _body, title in + diary.pages_for(_compile())} + + assert titles["diary/2026/03.md"] == "March 2026" + assert titles["diary/2026/about.md"] == "2026" From c1bff7caffc254c833a6e1c51ca8d3d755fa9cc9 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 14:54:15 +0200 Subject: [PATCH 06/43] fix(memory): stop the diary inventing doubt about ordinary evenings Six ways the compiler read a real room wrong, none of which the test corpus could reach. - Three memos recorded one after another were all filed as undateable. Arriving together proves nothing: a run is a sync burst only when some memo in it says aloud that it was made on a day its own timestamp disagrees with - A memo recorded at half past midnight landed on the previous day. Days are now read on the household's clock, not UTC - Videos and files posted to the room were dropped without a word - An entry that opened with a quote had its first line eaten, because every leading blockquote was treated as a reply's quoted original - A photo whose client repeats the filename showed the filename where a caption belongs - A recording that merely tailed off could swallow one made days later; halves must have arrived together to be joined --- stacklets/core/stacklet.toml | 6 ++ stacklets/memory/bot/cli/diary.py | 24 ++++- stacklets/memory/bot/diary.py | 133 +++++++++++++++++++++++---- tests/stacklets/test_memory_diary.py | 132 +++++++++++++++++++++++++- 4 files changed, 273 insertions(+), 22 deletions(-) diff --git a/stacklets/core/stacklet.toml b/stacklets/core/stacklet.toml index b8cb915..ddace63 100644 --- a/stacklets/core/stacklet.toml +++ b/stacklets/core/stacklet.toml @@ -110,6 +110,12 @@ IMMICH_API_KEY = "{photos__API_KEY}" # Tools server port — Open WebUI connects here for tool calling TOOLS_PORT = "42000" +# Household timezone — the clock the family keeps. Anything that turns a +# timestamp into a calendar day needs it: a voice memo recorded at half +# past midnight belongs to the day the family would say it happened on, +# not to whatever day it was in UTC. +TIMEZONE = "{timezone}" + # Household language — drives bot-side i18n, ontology rendering, and # any prompt that should be in the family's primary language. Distinct # from `[ai].language` (`{ai_language}`), which controls AI subsystem diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 85dc63a..460aa37 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -37,8 +37,10 @@ import json import os import sys +from datetime import timezone from pathlib import Path from urllib.parse import quote +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError import aiohttp @@ -339,6 +341,25 @@ async def _summarise(entries, llm) -> str: # ── The command ─────────────────────────────────────────────────────── +def _household_zone(): + """The clock the family keeps, for turning timestamps into days. + + Falls back to UTC with a warning rather than failing: a diary with + some entries an hour either side of midnight is worth more than no + diary, and the operator can see why in the output. + """ + name = os.environ.get("TIMEZONE", "").strip() + if not name: + _err("TIMEZONE not set, reading timestamps as UTC " + "— late-night entries may land on the wrong day") + return timezone.utc + try: + return ZoneInfo(name) + except (ZoneInfoNotFoundError, ValueError): + _err(f"unknown timezone {name!r}, reading timestamps as UTC") + return timezone.utc + + def _opt(argv: list[str], flag: str, fallback: str) -> str: for i, arg in enumerate(argv): if arg == flag and i + 1 < len(argv): @@ -356,6 +377,7 @@ async def run(llm, argv: list[str]) -> int: _err("--burst-window wants a number of seconds") return 2 + zone = _household_zone() homeserver = os.environ.get("MATRIX_HOMESERVER", "").rstrip("/") if not homeserver: _err("MATRIX_HOMESERVER not set — is core up?") @@ -377,7 +399,7 @@ async def run(llm, argv: list[str]) -> int: _err(str(e)) return 1 - messages = diary.resolve(events, burst_window_s=window) + messages = diary.resolve(events, burst_window_s=window, zone=zone) if not messages: _err(f"nothing in {room_arg} to compile") return 0 diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index ec8df59..155ca30 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -25,7 +25,7 @@ import re from dataclasses import dataclass, field, replace -from datetime import date, datetime, timezone +from datetime import date, datetime, timezone, tzinfo # A sync burst is a phone coming back online and flushing its queue, so # its members land seconds apart whatever their recording dates. 120s is @@ -53,16 +53,24 @@ class Message: event_id: str sender: str ts: int - kind: str # "voice" | "image" | "text" + kind: str # "voice" | "image" | "video" | "file" | "text" body: str = "" url: str | None = None duration_ms: int | None = None reply_to: str | None = None burst: str | None = None + zone: tzinfo = timezone.utc @property def sent_on(self) -> date: - return datetime.fromtimestamp(self.ts / 1000, timezone.utc).date() + """The calendar day the family would say this happened on. + + Read in the household's timezone, not UTC. A memo recorded at + half past midnight in Berlin is a UTC message from the previous + day, and filing it there puts it under the wrong heading in a + diary whose whole job is saying when things happened. + """ + return datetime.fromtimestamp(self.ts / 1000, self.zone).date() @dataclass(frozen=True) @@ -122,7 +130,13 @@ class Entry: def strip_reply_fallback(body: str) -> str: - """Drop the quoted-original block a client prepends to a reply.""" + """Drop the quoted-original block a client prepends to a reply. + + Only ever called for a message that really is a reply. A leading + blockquote is otherwise just a leading blockquote, and stripping it + unconditionally would silently eat the opening of any entry that + starts by quoting something. + """ lines = body.splitlines() i = 0 while i < len(lines) and _FALLBACK_LINE.match(lines[i]): @@ -134,11 +148,28 @@ def strip_reply_fallback(body: str) -> str: return "\n".join(lines[i:]) +# Uploads whose `body` is a filename rather than words. A caption, when +# a client sends one, displaces it (MSC2530) and the real name moves to +# `filename`. +_UPLOADS = ("image", "video", "file") + + def _kind_of(msgtype: str) -> str | None: - return {"m.audio": "voice", "m.image": "image", "m.text": "text"}.get(msgtype) + """The kind of entry a message makes, or None to ignore it. + + Video and arbitrary files count. A family posts a clip of a first + step to the memories room as readily as a photo, and a compiler that + recognised only the three types its test corpus happened to contain + would drop it without saying so. + """ + return { + "m.audio": "voice", "m.image": "image", "m.video": "video", + "m.file": "file", "m.text": "text", + }.get(msgtype) -def resolve(events, *, burst_window_s: float = DEFAULT_BURST_WINDOW_S): +def resolve(events, *, burst_window_s: float = DEFAULT_BURST_WINDOW_S, + zone: tzinfo = timezone.utc): """Room events to messages: edits applied, replies linked, bursts marked. `events` is the raw chunk from Synapse in any order; the result is @@ -170,13 +201,20 @@ def resolve(events, *, burst_window_s: float = DEFAULT_BURST_WINDOW_S): continue info = content.get("info") or {} - # For an upload, `body` is the filename and `filename` is absent; - # a client that attaches a caption puts the caption in `body` and - # moves the real name to `filename`. Without this the diary would - # print "IMG_4021.png" where the caption belongs. - body = strip_reply_fallback(content.get("body", "")) - if kind == "image" and not content.get("filename"): - body = "" + in_reply_to = (relates.get("m.in_reply_to") or {}).get("event_id") + + body = content.get("body", "") + if in_reply_to: + body = strip_reply_fallback(body) + if kind in _UPLOADS: + # `body` is the filename until a caption displaces it, at + # which point the name moves to `filename`. Clients that set + # `filename` to the same string are still sending a bare + # upload, so compare rather than test for presence -- else + # the diary prints "IMG_4021.png" where a caption belongs. + filename = content.get("filename") + body = body if (filename and filename != body) else "" + plain.append(Message( event_id=ev.get("event_id", ""), sender=(ev.get("sender") or "").split(":")[0].lstrip("@"), @@ -185,7 +223,8 @@ def resolve(events, *, burst_window_s: float = DEFAULT_BURST_WINDOW_S): body=body, url=content.get("url"), duration_ms=info.get("duration"), - reply_to=(relates.get("m.in_reply_to") or {}).get("event_id"), + reply_to=in_reply_to, + zone=zone, )) plain.sort(key=lambda m: (m.ts, m.event_id)) @@ -204,10 +243,12 @@ def mark_bursts(messages, *, window_s: float = DEFAULT_BURST_WINDOW_S): arriving shortly after another is just someone typing quickly, and calling that a burst would throw away a timestamp that is fine. - The label means "these timestamps are arrival times, not recording - times". It does not mean the messages belong together -- that is - `join_fragments`, and confusing the two is the trap this corpus was - built to set. + The label means only "these arrived together". Whether that makes + them a *sync* burst -- a queue being flushed, whose timestamps are + days off -- is `confirm_bursts`'s question, and it needs evidence + this function does not have. Nor does arriving together mean the + messages belong together: that is `join_fragments`, and confusing + the two is the trap this corpus was built to set. """ out = list(messages) run: list[int] = [] @@ -236,6 +277,49 @@ def close(run_idx): return out +# How far a spoken date must sit from its own timestamp before that +# timestamp is provably not the recording date. A day of slack absorbs +# the honest cases: a memo recorded before midnight that reaches the +# server after it, or a household clock a few hours off UTC. +CONTRADICTION_DAYS = 1 + + +def confirm_bursts(messages, readings): + """Keep the burst label only where the timestamps are provably lying. + + Arriving together is not evidence of anything by itself. A family + recording three memos at the dinner table sends them a minute apart, + and those timestamps are perfectly good; marking that run a sync + burst would file three entries as undateable when nothing was wrong + with any of them. That is the failure worth avoiding, because a + diary that cries uncertainty over ordinary evenings teaches the + family to ignore the warning on the one entry that earned it. + + So a run must contradict itself: some message in it says aloud that + it was made on a day its own timestamp disagrees with. That is proof + the queue was flushed rather than lived, and it makes the rest of + the run suspect too. + + A synced burst in which nobody spoke a date is therefore dated as + though it were live. Wrong, but undetectably so -- there is no + signal in the room to find, and inventing suspicion from timing + alone costs more than it recovers. + """ + lying = set() + for msg in messages: + if not msg.burst: + continue + spoken = parse_spoken_date( + readings.get(msg.event_id, Reading()).spoken_date) + if spoken is None: + continue + if abs((msg.sent_on - spoken).days) > CONTRADICTION_DAYS: + lying.add(msg.burst) + + return [m if m.burst in lying else replace(m, burst=None) + for m in messages] + + # ── Step 2: join ────────────────────────────────────────────────────── @@ -260,6 +344,11 @@ def join_fragments(messages, readings): continues = ( msg.kind == "voice" and prev.kind == "voice" and msg.sender == prev.sender + # Uploaded together: a recording that was cut in two + # arrives as two files back to back. Without this, a + # memo whose transcript merely tails off joins itself to + # whatever was recorded days later. + and msg.burst is not None and msg.burst == prev.burst and prev_reading.ends_mid_thought and reading.starts_mid_thought ) @@ -322,13 +411,18 @@ def compile_entries(messages, readings) -> list[Entry]: the pair apart and leaves a line of commentary floating with no subject. """ + # Join on candidate runs (a split recording arrives as two adjacent + # uploads), then decide which runs were really a queue being + # flushed. Dating reads the confirmed view; joining cannot, because + # confirmation strips the very label that pairs the halves. groups = join_fragments(messages, readings) + confirmed = {m.event_id: m for m in confirm_bursts(messages, readings)} entries: list[Entry] = [] by_event: dict[str, Entry] = {} pending: list[tuple[Message, list[Message]]] = [] for group in groups: - head = group[0] + head = confirmed.get(group[0].event_id, group[0]) reading = readings.get(head.event_id, Reading()) if head.reply_to: pending.append((head, group)) @@ -357,6 +451,7 @@ def compile_entries(messages, readings) -> list[Entry]: for msg, group in pending: parent = by_event.get(msg.reply_to or "") if parent is None: + msg = confirmed.get(msg.event_id, msg) reading = readings.get(msg.event_id, Reading()) on, confidence, basis = date_for(msg, reading) orphan = Entry( diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 54afc20..dfc0f4e 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -22,8 +22,9 @@ import sys from dataclasses import replace -from datetime import date, datetime, timezone +from datetime import date, datetime, timedelta, timezone from pathlib import Path +from zoneinfo import ZoneInfo import pytest import yaml @@ -195,10 +196,65 @@ def test_a_reply_does_not_quote_its_parent_back(self): body = "> <@marge:test> the original memo\n\nHe gets that from me." events = [{"type": "m.room.message", "event_id": "$r", "sender": "@homer:test", "origin_server_ts": BASE_TS, - "content": {"msgtype": "m.text", "body": body}}] + "content": {"msgtype": "m.text", "body": body, + "m.relates_to": { + "m.in_reply_to": {"event_id": "$p"}}}}] assert diary.resolve(events)[0].body == "He gets that from me." + def test_an_entry_may_open_with_a_quote(self): + """A leading blockquote is only a reply fallback on a reply. + + Stripping it from everything would eat the opening of any entry + that starts by quoting something, which in a diary is a natural + way to write. + """ + body = "> the only thing we have to fear\n\nLisa said this today." + events = [{"type": "m.room.message", "event_id": "$t", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.text", "body": body}}] + + assert diary.resolve(events)[0].body == body + + def test_a_video_is_not_dropped_on_the_floor(self): + """A family posts a clip as readily as a photo.""" + events = [{"type": "m.room.message", "event_id": "$v", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.video", "body": "clip.mp4", + "url": "mxc://test/v"}}] + + resolved = diary.resolve(events) + + assert [m.kind for m in resolved] == ["video"] + assert resolved[0].body == "", "a filename is not a caption" + + def test_a_client_that_repeats_the_filename_sends_no_caption(self): + """Some clients set `filename` and `body` to the same string.""" + events = [{"type": "m.room.message", "event_id": "$i", + "sender": "@marge:test", "origin_server_ts": BASE_TS, + "content": {"msgtype": "m.image", "body": "IMG_4021.png", + "filename": "IMG_4021.png", + "url": "mxc://test/x"}}] + + assert diary.resolve(events)[0].body == "" + + def test_a_late_night_memo_belongs_to_the_night_it_was_recorded(self): + """Day boundaries are the household's, not UTC's. + + Half past midnight in Berlin is still the previous day in UTC, + so reading the clock in UTC files the memo under a heading the + family would not recognise. + """ + berlin = ZoneInfo("Europe/Berlin") + recorded = datetime(2026, 3, 17, 0, 30, tzinfo=berlin) + events = [{"type": "m.room.message", "event_id": "$n", + "sender": "@marge:test", + "origin_server_ts": int(recorded.timestamp() * 1000), + "content": {"msgtype": "m.text", "body": "still awake"}}] + + assert diary.resolve(events, zone=berlin)[0].sent_on == date(2026, 3, 17) + assert diary.resolve(events)[0].sent_on == date(2026, 3, 16) + def test_a_bare_upload_has_no_caption(self): """An image's `body` is its filename until a caption displaces it.""" events = [{"type": "m.room.message", "event_id": "$i", @@ -256,6 +312,78 @@ def test_the_window_decides_where_a_run_stops(self): assert not any(m.burst for m in diary.mark_bursts(run, window_s=1)) +class TestBurstsInARealRoom: + """Pinned at the default window, against timings a replay cannot have. + + The corpus is replayed, so it compresses day-scale gaps to seconds + and has to be compiled with a tiny window. These cases use the + shipped default and the spacing a real room has, because the + question they answer -- does an ordinary evening get called + undateable -- is the one the corpus cannot ask. + """ + + def _run(self, gap_s, count=3, **reading_kw): + messages = diary.mark_bursts( + [_msg(event_id=f"$m{i}", ts=BASE_TS + i * gap_s * 1000, body="x") + for i in range(count)], + window_s=diary.DEFAULT_BURST_WINDOW_S, + ) + readings = {"$m0": diary.Reading(**reading_kw)} if reading_kw else {} + return diary.compile_entries(messages, readings) + + def test_memos_recorded_one_after_another_keep_their_timestamps(self): + """Three memos at the dinner table are not a sync burst. + + This is the common case in a real room, and calling it + undateable would put a warning on most of the diary. + """ + entries = self._run(gap_s=40) + + assert [e.confidence for e in entries] == ["sent", "sent", "sent"] + + def test_a_memo_that_contradicts_its_own_timestamp_condemns_its_run(self): + """One spoken date days off its arrival proves a flushed queue.""" + entries = self._run(gap_s=40, spoken_date="2026-03-16") + + assert [e.confidence for e in entries] \ + == ["spoken", "uncertain", "uncertain"] + + def test_a_memo_synced_just_after_midnight_is_not_a_contradiction(self): + """Recorded before midnight, received after, is an honest day of + drift rather than evidence the timestamps are lying.""" + spoken = datetime.fromtimestamp( + BASE_TS / 1000, timezone.utc).date() - timedelta(days=1) + entries = self._run(gap_s=40, spoken_date=spoken.isoformat()) + + assert [e.confidence for e in entries] == ["spoken", "sent", "sent"] + + def test_messages_far_apart_never_form_a_run(self): + entries = self._run(gap_s=3600, spoken_date="2026-03-16") + + assert [e.confidence for e in entries] == ["spoken", "sent", "sent"] + + +class TestJoiningInARealRoom: + def test_a_memo_that_tails_off_does_not_swallow_a_later_one(self): + """Whisper often clips the end of a recording, so "ends mid + thought" is common. Only halves that arrived together may join; + otherwise a Tuesday memo absorbs Thursday's.""" + days_apart = [ + _msg(event_id="$a", ts=BASE_TS, body="I was going to say"), + _msg(event_id="$b", ts=BASE_TS + 2 * 86_400_000, body="and then"), + ] + readings = { + "$a": diary.Reading(ends_mid_thought=True), + "$b": diary.Reading(starts_mid_thought=True), + } + messages = diary.mark_bursts( + days_apart, window_s=diary.DEFAULT_BURST_WINDOW_S) + + groups = diary.join_fragments(messages, readings) + + assert [len(g) for g in groups] == [1, 1] + + # ── The trap the corpus was built to set ────────────────────────────── From d03002dc5ada2ee6c10cdbd2052bc75142121c94 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 15:00:48 +0200 Subject: [PATCH 07/43] refactor(memory): read the room a page at a time, not a line at a time Reading one message alone could never see what it was part of, so the compiler guessed with timing heuristics and got it wrong in both directions. The reader now takes a slice of room history at once and returns the links it finds. - "The picture above is from the barbecue" now files under the picture above. It carries no Matrix relation, so nothing but reading the two together could ever have connected them - A memo is joined to the one before it because a reader saw the sentence run across the break, not because the two arrived close together. The room still checks the link: halves must be adjacent, same person, same kind - No more inventing a person out of a nickname. A conversation between two people who are both there is addressed to neither The entries stay the family's own words. The model reads and links; it never writes an entry. --- stacklets/memory/bot/cli/diary.py | 211 ++++++++++++++++++--------- stacklets/memory/bot/diary.py | 105 +++++++------ tests/stacklets/test_memory_diary.py | 96 ++++++++---- 3 files changed, 266 insertions(+), 146 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 460aa37..e2f9bee 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -37,7 +37,7 @@ import json import os import sys -from datetime import timezone +from datetime import datetime, timezone from pathlib import Path from urllib.parse import quote from zoneinfo import ZoneInfo, ZoneInfoNotFoundError @@ -165,19 +165,36 @@ async def produce() -> dict: return "" +# How many messages go to the model at once, and how many of the +# previous chunk to repeat. The overlap exists so a recording split +# across a chunk boundary is still seen whole by one call; a split is +# always two adjacent uploads, so a few messages of run-up is plenty. +_CHUNK = 40 +_OVERLAP = 4 + + _READ_PROMPT = """\ -You are reading one message from a family's private memories room so it -can be filed in their diary. Do not rewrite it, summarise it, translate -it, or comment on it. Report only facts about the text as it stands. +You are reading a family's private memories room so their diary can be +compiled. Report facts about these messages. Never rewrite one, never +summarise one, never translate one. + +The messages are in the order the server received them, which is not +always the order they were recorded: a phone that has been offline +uploads everything at once when it reconnects. -This message reached the server on {arrival}. +{messages} -Message from {sender}: ---- -{body} ---- +Reply with a JSON object {{"messages": [...]}} holding one object per +message above, in the same order, each with these keys: -Reply with a JSON object with exactly these keys: +"n": the message's number. + +"spoken_date": the date the speaker states inside the message, as + YYYY-MM-DD. Use only a date the text actually names, such as "today is + March sixteenth". If it names a day and month but no year, choose the + most recent such date on or before the day the message was received. + If the text states no date, use null. Never derive one from the + received date alone. "mode": a recorded conversation carries no speaker labels, so judge by the turns rather than by names. Answer "dialogue" when a statement in @@ -187,67 +204,118 @@ async def produce() -> dict: throughout, however many people they mention or address. Answer "note" if it reads as written rather than spoken. -"spoken_date": the date the speaker states inside the message, as - YYYY-MM-DD. Use only a date the text actually names, such as "today is - March sixteenth". If it names a day and month but no year, choose the - most recent such date on or before {arrival}. If the text states no - date at all, use null. Never derive a date from the arrival date - alone. - -"starts_mid_thought": true if the text begins part-way through a - sentence or thought, as though the recording started late. - -"ends_mid_thought": true if the text stops part-way through a sentence - or thought, as though the recording was cut off. - -"addressee": who the message is spoken to, written exactly as the - message names them ("Bart", "kids", "Maggie"), or null if it is not - addressed to anyone in particular. Never name the speaker themselves: - in a conversation between two people who are both present, there is - no addressee, so use null. +"addressee": who the message is spoken to, exactly as it names them + ("Bart", "kids"), or null if it is not addressed to anyone in + particular. In a dialogue both speakers are present, so use null. + +"continues": the number of the message directly before this one, when + the two are halves of a single recording that was cut in the middle + of a sentence: the earlier one stops mid-thought and this one picks + up the same sentence. Otherwise null. Messages that merely arrived + together are not halves of each other -- three uploads in the same + second are usually three separate memos, and joining them would fuse + three memories into one. + +"refers_to": the number of an earlier message this one is a remark + about rather than a memory of its own: a caption for a photo, or a + line like "the picture above is from the barbecue". Otherwise null. """ -async def _read(message, llm) -> diary.Reading: - """Ask the model what this message says about itself. - - Temperature 0: the same recording must read the same way on every - compile, or a rerun would silently reshuffle the diary. A model that - fails or answers with nonsense yields an empty reading, which dates - the entry from its timestamp -- worse, but not wrong in a way that - hides anything. +def _as_prompt(chunk) -> str: + """The messages as the reader sees them, numbered from one. - Dates, fragment boundaries and addressees come back reliably at this - model tier. `mode` does not: an unlabelled two-speaker transcript - reads as one person recounting a conversation, and a 35B model calls - it a monologue. The prompt is tuned to suppress the false positive - rather than chase the false negative, because "Conversation" printed - over a private memo to a child is a worse page than "Voice note" - printed over a dinner-table recording. Recovering the rest needs - diarization, which v1 does not have. + Numbered rather than keyed by event id because the links come back + as references and a model copying a 43-character Matrix id is a + transcription test, not a reading one. """ - prompt = _READ_PROMPT.format( - arrival=message.sent_on.isoformat(), - sender=message.sender, - body=message.body.strip(), - ) - try: - raw = await llm.complete("classifier", prompt, - json_mode=True, temperature=0) - data = json.loads(raw) - except (LLMError, json.JSONDecodeError, TypeError) as e: - _err(f" could not read {message.event_id}: {e}") - return diary.Reading() - - if not isinstance(data, dict): - return diary.Reading() - return diary.Reading( - mode=str(data.get("mode") or "monologue"), - spoken_date=data.get("spoken_date") or None, - starts_mid_thought=bool(data.get("starts_mid_thought")), - ends_mid_thought=bool(data.get("ends_mid_thought")), - addressee=(data.get("addressee") or None), - ) + lines = [] + for n, msg in enumerate(chunk, start=1): + when = datetime.fromtimestamp( + msg.ts / 1000, msg.zone).strftime("%Y-%m-%d %H:%M") + kind = {"voice": "voice recording", "image": "photo", + "video": "video", "file": "file"}.get(msg.kind, "text") + body = msg.body.strip() or "(no caption)" + lines.append(f"[{n}] {msg.sender}, received {when}, {kind}:\n{body}") + return "\n\n".join(lines) + + +def _chunks(messages): + """Slices of room history, in arrival order, with a little run-up. + + Arrival order rather than calendar day on purpose: the day a memo + belongs to is what this pass works out, so it cannot also be what + decides the batching. + """ + if len(messages) <= _CHUNK: + return [list(messages)] + out, start = [], 0 + while start < len(messages): + out.append(list(messages[start:start + _CHUNK])) + start += _CHUNK - _OVERLAP + return out + + +async def _read_room(messages, llm): + """Read the room: facts per message, and the links between them. + + One call per slice of history rather than one per message. Reading a + message alone cannot see that it finishes the sentence before it, or + that it is a remark about the photo above -- and a call with no + neighbours also mistakes a two-person conversation for one person + reminiscing. The batch is what makes those answerable. + + Returns `(readings, continues, refers_to)`. A slice the model fails + or garbles contributes nothing and the rest still compiles: the + entries are the family's words either way, and a missing reading + costs a date, not a memory. + """ + readings: dict[str, diary.Reading] = {} + continues: dict[str, str] = {} + refers_to: dict[str, str] = {} + + for chunk in _chunks(messages): + prompt = _READ_PROMPT.format(messages=_as_prompt(chunk)) + try: + raw = await llm.complete("classifier", prompt, + json_mode=True, temperature=0) + payload = json.loads(raw) + rows = payload.get("messages") if isinstance(payload, dict) else None + except (LLMError, json.JSONDecodeError, TypeError) as e: + _err(f" could not read {len(chunk)} message(s): {e}") + continue + if not isinstance(rows, list): + _err(f" unreadable answer for {len(chunk)} message(s)") + continue + + for row in rows: + if not isinstance(row, dict): + continue + here = _resolve_n(row.get("n"), chunk) + if here is None: + continue + readings.setdefault(here.event_id, diary.Reading( + mode=str(row.get("mode") or "monologue"), + spoken_date=row.get("spoken_date") or None, + addressee=(row.get("addressee") or None), + )) + # A later chunk sees a pair the earlier one straddled, so a + # link found anywhere wins over one found nowhere. + if (target := _resolve_n(row.get("continues"), chunk)) is not None: + continues[here.event_id] = target.event_id + if (target := _resolve_n(row.get("refers_to"), chunk)) is not None: + refers_to[here.event_id] = target.event_id + + return readings, continues, refers_to + + +def _resolve_n(value, chunk): + """The message a returned number points at, or None if it points off + the end. Models occasionally answer with a number that is not in + front of them; a link into thin air is dropped rather than guessed.""" + if not isinstance(value, int) or not 1 <= value <= len(chunk): + return None + return chunk[value - 1] # ── Recalling a month ───────────────────────────────────────────────── @@ -421,16 +489,13 @@ async def run(llm, argv: list[str]) -> int: continue decoded.append(msg.__class__(**{**msg.__dict__, "body": text})) - readings = {} - for msg in decoded: - if msg.kind == "image" or not msg.body.strip(): - readings[msg.event_id] = diary.Reading(mode="note") - continue - readings[msg.event_id] = await _read(msg, llm) + readings, continues, refers_to = await _read_room(decoded, llm) await transcriber.aclose() - entries = diary.compile_entries(decoded, readings) + entries = diary.compile_entries(decoded, readings, + continues=continues, + refers_to=refers_to) _err(f"{len(entries)} diary entr{'y' if len(entries) == 1 else 'ies'}") months: dict[str, list] = {} diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 155ca30..9647f40 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -75,20 +75,21 @@ def sent_on(self) -> date: @dataclass(frozen=True) class Reading: - """What the classifier read out of one message. + """What the model read out of one message. Facts about the text, not a rewrite of it. `spoken_date` is the date said aloud ("today is March sixteenth") and is the only in-band - record of when a recording was made. The two mid-thought flags exist - because a burst and a split recording look identical from timing - alone: three files a second apart are three memos or one memo in - three pieces, and only the words can tell you which. + record of when a recording was made. + + Relationships between messages are not here. They live in the + `continues` and `refers_to` maps, because reading one message can + never establish them: whether an upload finishes the one before it, + or whether a line of text is about the photo above it, is only + visible to something looking at both. """ mode: str = "monologue" # "monologue" | "dialogue" | "note" spoken_date: str | None = None - starts_mid_thought: bool = False - ends_mid_thought: bool = False addressee: str | None = None @@ -323,39 +324,44 @@ def confirm_bursts(messages, readings): # ── Step 2: join ────────────────────────────────────────────────────── -def join_fragments(messages, readings): +def join_fragments(messages, continues): """Group messages into the recordings they actually are. One memo split mid-sentence arrives as two files that look exactly - like two memos sent back to back. The only evidence that separates - them is the words: the first stops mid-thought and the second picks - it up. So a join needs both halves to agree, and a burst label alone - is never enough -- three same-second uploads are usually three - independent memos. + like two memos sent back to back, and timing cannot tell them apart: + three uploads in one second are usually three separate thoughts. + Only the words decide, and only to something reading both halves at + once -- which is why `continues` is handed in, mapping a message to + the one it finishes. + + The link is still checked against the room: a join must be with the + message immediately before, from the same person, of the same kind. + A recording cut in two arrives as two adjacent uploads, so a link + reaching further than that is a misreading, and merging on it would + fuse two unrelated memories into one entry. Returns a list of groups, each a list of messages in order. """ groups: list[list[Message]] = [] - for msg in messages: - reading = readings.get(msg.event_id, Reading()) - if groups: - prev = groups[-1][-1] - prev_reading = readings.get(prev.event_id, Reading()) - continues = ( - msg.kind == "voice" and prev.kind == "voice" - and msg.sender == prev.sender - # Uploaded together: a recording that was cut in two - # arrives as two files back to back. Without this, a - # memo whose transcript merely tails off joins itself to - # whatever was recorded days later. - and msg.burst is not None and msg.burst == prev.burst - and prev_reading.ends_mid_thought - and reading.starts_mid_thought - ) - if continues: - groups[-1].append(msg) - continue - groups.append([msg]) + holding: dict[str, list[Message]] = {} + + for i, msg in enumerate(messages): + target = continues.get(msg.event_id) + previous = messages[i - 1] if i else None + adjacent = ( + previous is not None + and target == previous.event_id + and msg.sender == previous.sender + and msg.kind == previous.kind + ) + group = holding.get(target) if adjacent else None + if group is None: + group = [msg] + groups.append(group) + else: + group.append(msg) + holding[msg.event_id] = group + return groups @@ -402,20 +408,26 @@ def date_for(msg: Message, reading: Reading) -> tuple[date, str, str]: # ── Step 4: compile ─────────────────────────────────────────────────── -def compile_entries(messages, readings) -> list[Entry]: +def compile_entries(messages, readings, *, + continues: "dict[str, str] | None" = None, + refers_to: "dict[str, str] | None" = None) -> list[Entry]: """Messages and their readings to dated diary entries. - Two kinds of message do not earn an entry of their own. A reply - belongs to what it replies to, and a caption that arrives behind its - photo is that photo's caption -- rendering either separately breaks - the pair apart and leaves a line of commentary floating with no - subject. + Some messages do not earn an entry of their own. A reply belongs to + what it replies to; so does a caption trailing its photo, and so + does a line like "the picture above is from the barbecue" -- which + carries no Matrix relation at all and is only recognisable to + something that read the two together. `refers_to` carries those. + Rendering any of them separately breaks the pair apart and leaves a + remark floating with no subject. """ - # Join on candidate runs (a split recording arrives as two adjacent - # uploads), then decide which runs were really a queue being - # flushed. Dating reads the confirmed view; joining cannot, because - # confirmation strips the very label that pairs the halves. - groups = join_fragments(messages, readings) + continues = continues or {} + about = refers_to or {} + + # Join first (a split recording is two adjacent uploads), then + # decide which runs were really a queue being flushed. Dating reads + # the confirmed view. + groups = join_fragments(messages, continues) confirmed = {m.event_id: m for m in confirm_bursts(messages, readings)} entries: list[Entry] = [] by_event: dict[str, Entry] = {} @@ -424,8 +436,9 @@ def compile_entries(messages, readings) -> list[Entry]: for group in groups: head = confirmed.get(group[0].event_id, group[0]) reading = readings.get(head.event_id, Reading()) - if head.reply_to: - pending.append((head, group)) + parent = head.reply_to or about.get(head.event_id) + if parent: + pending.append((replace(head, reply_to=parent), group)) continue on, confidence, basis = date_for(head, reading) diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index dfc0f4e..73c52fa 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -112,23 +112,43 @@ def _readings_from_spec(items: list[dict]) -> dict[str, diary.Reading]: `date_source: spoken` means the recording says its own date, so the reading carries it; anything else leaves it null and the compiler - has to fall back. Fragment halves are marked where the spec says the - recording was cut. + has to fall back. """ - readings = {} - for item in items: - source = item["date_source"] - fragment = item.get("fragment_of") - first_half = fragment and item["id"].endswith("-a") - second_half = fragment and not item["id"].endswith("-a") - readings[f"${item['id']}"] = diary.Reading( + return { + f"${item['id']}": diary.Reading( mode="note" if item["kind"] == "text" else "monologue", spoken_date=(item["true_date"].isoformat() - if source == "spoken" else None), - starts_mid_thought=bool(second_half), - ends_mid_thought=bool(first_half), + if item["date_source"] == "spoken" else None), ) - return readings + for item in items + } + + +def _links_from_spec(items: list[dict]): + """The links between messages a correct reader would find. + + These are the judgments that need two messages in view at once, so + the spec is where they come from: `fragment_of` names the halves of + a split recording, and an `implicit-context` item is a remark about + the last picture posted before it. + """ + continues: dict[str, str] = {} + refers_to: dict[str, str] = {} + halves: dict[str, str] = {} + last_image: str | None = None + + for item in items: + event_id = f"${item['id']}" + if fragment := item.get("fragment_of"): + if earlier := halves.get(fragment): + continues[event_id] = earlier + halves[fragment] = event_id + if item["pattern"] == "implicit-context" and last_image: + refers_to[event_id] = last_image + if item["kind"] == "image": + last_image = event_id + + return continues, refers_to def _words(item: dict) -> str: @@ -156,7 +176,9 @@ def _compile(items=None): items = items if items is not None else _spec_items() messages = diary.resolve(_room_from_spec(items), burst_window_s=WINDOW_S) messages = _transcribed(messages, items) - return diary.compile_entries(messages, _readings_from_spec(items)) + continues, refers_to = _links_from_spec(items) + return diary.compile_entries(messages, _readings_from_spec(items), + continues=continues, refers_to=refers_to) def _entry_for(entries, item_id: str) -> diary.Entry: @@ -364,22 +386,30 @@ def test_messages_far_apart_never_form_a_run(self): class TestJoiningInARealRoom: - def test_a_memo_that_tails_off_does_not_swallow_a_later_one(self): - """Whisper often clips the end of a recording, so "ends mid - thought" is common. Only halves that arrived together may join; - otherwise a Tuesday memo absorbs Thursday's.""" - days_apart = [ + def test_a_link_that_reaches_past_the_previous_message_is_refused(self): + """A split recording is two adjacent uploads, always. + + A model that links across an intervening message has misread, + and merging on it would fuse two unrelated memories into one + entry. The room is the check on the reading. + """ + messages = [ _msg(event_id="$a", ts=BASE_TS, body="I was going to say"), - _msg(event_id="$b", ts=BASE_TS + 2 * 86_400_000, body="and then"), + _msg(event_id="$b", ts=BASE_TS + 60_000, body="something else"), + _msg(event_id="$c", ts=BASE_TS + 120_000, body="and then"), + ] + + groups = diary.join_fragments(messages, {"$c": "$a"}) + + assert [len(g) for g in groups] == [1, 1, 1] + + def test_a_different_speaker_never_finishes_your_sentence(self): + messages = [ + _msg(event_id="$a", sender="marge", ts=BASE_TS, body="I meant"), + _msg(event_id="$b", sender="homer", ts=BASE_TS + 1000, body="to say"), ] - readings = { - "$a": diary.Reading(ends_mid_thought=True), - "$b": diary.Reading(starts_mid_thought=True), - } - messages = diary.mark_bursts( - days_apart, window_s=diary.DEFAULT_BURST_WINDOW_S) - groups = diary.join_fragments(messages, readings) + groups = diary.join_fragments(messages, {"$b": "$a"}) assert [len(g) for g in groups] == [1, 1] @@ -503,6 +533,18 @@ def test_a_late_caption_belongs_to_its_photo(self): # It is not an entry of its own. assert _entry_for(entries, "bild-zeichnung-caption").kind == "text" + def test_a_remark_about_a_photo_is_filed_under_that_photo(self): + """"The picture above is from the barbecue" carries no Matrix + relation at all. Nothing in the event says what it is about, so + the link can only come from reading the two together -- and once + it does, the remark stops being a memory of its own. + """ + entries = _compile() + + photo = _entry_for(entries, "bild-kontextlos") + assert "$impliziter-kontext" in photo.event_ids + assert "barbecue" in photo.comments[0][1] + def test_a_reply_to_a_memo_is_filed_under_that_memo(self): entries = _compile() From 4d487e064739777447f8211e196ee2700dd26bf7 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sat, 12 Sep 2026 15:12:58 +0200 Subject: [PATCH 08/43] feat(memory): compile the diary nightly, paying only for what is new New recordings now reach the wiki overnight. The curator runs the compiler on the same sweep it rebuilds the wiki with. Every run is still a full pass over the room rather than an append. It has to be: a reply to a memo from March can arrive in September, an edit can land on a year-old note, a remark can turn out to be about a photo from last spring. A compiler that walked forward from where it last stopped would file all three under today, orphaned from what they belong to. It stays cheap because what each piece cost is kept against the thing it describes, not against how far we got. A cold compile of the demo room takes 37s; a run with nothing new takes 1.3s and calls no model at all. A late reply to a March memo re-reads one message and rewrites one page. - Readings and month summaries cached; transcripts already were - A month is re-summarised when its content moves and not otherwise, so a settled month is never quietly reworded overnight - `--rebuild` reads everything again, for when the model improves - httpx instead of aiohttp, so the compiler runs in the curator too --- stacklets/memory/bot/cli/diary.py | 139 ++++++++++++++++++------- stacklets/memory/bot/cli_entrypoint.py | 16 ++- stacklets/memory/bot/curator.py | 35 +++++-- stacklets/memory/bot/diary.py | 21 ++++ stacklets/memory/bot/diary_store.py | 138 ++++++++++++++++++++++++ stacklets/memory/cli/diary.py | 14 +++ stacklets/memory/docker-compose.yml | 13 +++ stacklets/memory/stacklet.toml | 22 ++++ tests/stacklets/test_memory_diary.py | 95 +++++++++++++++++ 9 files changed, 443 insertions(+), 50 deletions(-) create mode 100644 stacklets/memory/bot/diary_store.py diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index e2f9bee..d7f6321 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -17,6 +17,7 @@ stack memory diary --dry-run print the pages, write nothing stack memory diary --room memories a different room stack memory diary --burst-window 1 see below + stack memory diary --rebuild re-read everything from scratch WHY THE BURST WINDOW IS A KNOB Messages that synced late carry arrival timestamps, not recording @@ -42,12 +43,20 @@ from urllib.parse import quote from zoneinfo import ZoneInfo, ZoneInfoNotFoundError -import aiohttp +# httpx rather than aiohttp: the OpenAI SDK already pulls it into every +# container that can talk to a model, so the compiler runs unchanged in +# the bot-runner and in the curator that schedules it nightly. +import httpx sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # bot/ -sys.path.insert(0, "/app") # voice, stack.ai.client +sys.path.insert(0, "/app") # stack.ai.client, and voice in the bot-runner +# The transcript store lives with the bot-runner's voice module. The +# bot-runner has it baked at /app; the curator, which schedules this +# nightly, only mounts /stacklets. Both see it here. +sys.path.append("/stacklets/core/bot-runner") import diary # noqa: E402 +import diary_store # noqa: E402 import voice # noqa: E402 from stack.ai.client import LLMError, Transcriber # noqa: E402 @@ -72,19 +81,19 @@ def _err(msg: str) -> None: # with standing access to the most private room in the house. -async def _admin_token(session: aiohttp.ClientSession, homeserver: str) -> str: +async def _admin_token(session: httpx.AsyncClient, homeserver: str) -> str: user = os.environ.get("MATRIX_ADMIN_USER", "") password = os.environ.get("MATRIX_ADMIN_PASSWORD", "") if not user or not password: raise RuntimeError("MATRIX_ADMIN_USER/PASSWORD not set in this container") - async with session.post(f"{homeserver}/_matrix/client/v3/login", json={ + resp = await session.post(f"{homeserver}/_matrix/client/v3/login", json={ "type": "m.login.password", "identifier": {"type": "m.id.user", "user": user}, "password": password, - }) as resp: - if resp.status != 200: - raise RuntimeError(f"admin login failed: HTTP {resp.status}") - return (await resp.json())["access_token"] + }) + if resp.status_code != 200: + raise RuntimeError(f"admin login failed: HTTP {resp.status_code}") + return resp.json()["access_token"] async def _resolve_room(session, homeserver, token, room: str) -> str: @@ -92,13 +101,13 @@ async def _resolve_room(session, homeserver, token, room: str) -> str: return room alias = room if room.startswith("#") else \ f"#{room}:{os.environ.get('MATRIX_SERVER_NAME', '')}" - async with session.get( + resp = await session.get( f"{homeserver}/_matrix/client/v3/directory/room/{quote(alias)}", headers={"Authorization": f"Bearer {token}"}, - ) as resp: - if resp.status != 200: - raise RuntimeError(f"no such room: {alias}") - return (await resp.json())["room_id"] + ) + if resp.status_code != 200: + raise RuntimeError(f"no such room: {alias}") + return resp.json()["room_id"] async def _history(session, homeserver, token, room_id: str) -> list[dict]: @@ -115,10 +124,10 @@ async def _history(session, homeserver, token, room_id: str) -> list[dict]: f"/messages?dir=b&limit={_PAGE}") if cursor: url += f"&from={quote(cursor)}" - async with session.get(url, headers=headers) as resp: - if resp.status != 200: - raise RuntimeError(f"could not read room: HTTP {resp.status}") - payload = await resp.json() + resp = await session.get(url, headers=headers) + if resp.status_code != 200: + raise RuntimeError(f"could not read room: HTTP {resp.status_code}") + payload = resp.json() chunk = payload.get("chunk") or [] events.extend(chunk) cursor = payload.get("end") or "" @@ -130,11 +139,11 @@ async def _download(session, homeserver, token, mxc: str) -> bytes | None: server, _, media_id = mxc.replace("mxc://", "").partition("/") url = (f"{homeserver}/_matrix/client/v1/media/download/" f"{quote(server)}/{quote(media_id)}") - async with session.get(url, headers={"Authorization": f"Bearer {token}"}) as resp: - if resp.status != 200: - _err(f" media {mxc}: HTTP {resp.status}") - return None - return await resp.read() + resp = await session.get(url, headers={"Authorization": f"Bearer {token}"}) + if resp.status_code != 200: + _err(f" media {mxc}: HTTP {resp.status_code}") + return None + return resp.content # ── Decoding and reading ────────────────────────────────────────────── @@ -256,7 +265,7 @@ def _chunks(messages): return out -async def _read_room(messages, llm): +async def _read_room(messages, llm, cache=None): """Read the room: facts per message, and the links between them. One call per slice of history rather than one per message. Reading a @@ -265,6 +274,13 @@ async def _read_room(messages, llm): neighbours also mistakes a two-person conversation for one person reminiscing. The batch is what makes those answerable. + What the model finds is remembered against the message it describes. + Both halves of that are permanent: a message never changes (an edit + is a new event pointing at the old one), and a link, once seen, is a + fact about two messages rather than about the run they arrived in. + So a slice read end to end is never sent again, and a nightly pass + costs only what is genuinely new. + Returns `(readings, continues, refers_to)`. A slice the model fails or garbles contributes nothing and the rest still compiles: the entries are the family's words either way, and a missing reading @@ -273,8 +289,34 @@ async def _read_room(messages, llm): readings: dict[str, diary.Reading] = {} continues: dict[str, str] = {} refers_to: dict[str, str] = {} + known: set[str] = set() + + def remember(event_id: str, row: dict) -> None: + readings[event_id] = diary.Reading( + mode=str(row.get("mode") or "monologue"), + spoken_date=row.get("spoken_date") or None, + addressee=row.get("addressee") or None, + ) + if target := row.get("continues"): + continues[event_id] = target + if target := row.get("refers_to"): + refers_to[event_id] = target + known.add(event_id) + + if cache is not None: + for msg in messages: + if (stored := cache.get(msg.event_id)) is not None: + remember(msg.event_id, stored) + if known: + _err(f" {len(known)} message(s) already read, " + f"{len(messages) - len(known)} new") for chunk in _chunks(messages): + # A slice whose every message is on file has nothing left to + # say: its links were recorded with the messages they join. + if all(m.event_id in known for m in chunk): + continue + prompt = _READ_PROMPT.format(messages=_as_prompt(chunk)) try: raw = await llm.complete("classifier", prompt, @@ -292,23 +334,28 @@ async def _read_room(messages, llm): if not isinstance(row, dict): continue here = _resolve_n(row.get("n"), chunk) - if here is None: + if here is None or here.event_id in known: continue - readings.setdefault(here.event_id, diary.Reading( - mode=str(row.get("mode") or "monologue"), - spoken_date=row.get("spoken_date") or None, - addressee=(row.get("addressee") or None), - )) - # A later chunk sees a pair the earlier one straddled, so a - # link found anywhere wins over one found nowhere. - if (target := _resolve_n(row.get("continues"), chunk)) is not None: - continues[here.event_id] = target.event_id - if (target := _resolve_n(row.get("refers_to"), chunk)) is not None: - refers_to[here.event_id] = target.event_id + found = { + "mode": str(row.get("mode") or "monologue"), + "spoken_date": row.get("spoken_date") or None, + "addressee": row.get("addressee") or None, + "continues": _link(row.get("continues"), chunk), + "refers_to": _link(row.get("refers_to"), chunk), + } + remember(here.event_id, found) + if cache is not None: + cache.put(here.event_id, found) return readings, continues, refers_to +def _link(value, chunk) -> str | None: + """The event id a returned number points at, or None.""" + target = _resolve_n(value, chunk) + return target.event_id if target is not None else None + + def _resolve_n(value, chunk): """The message a returned number points at, or None if it points off the end. Models occasionally answer with a number that is not in @@ -438,6 +485,7 @@ def _opt(argv: list[str], flag: str, fallback: str) -> str: async def run(llm, argv: list[str]) -> int: room_arg = _opt(argv, "--room", "memories") dry_run = "--dry-run" in argv + rebuild = "--rebuild" in argv try: window = float(_opt(argv, "--burst-window", str(diary.DEFAULT_BURST_WINDOW_S))) @@ -446,6 +494,7 @@ async def run(llm, argv: list[str]) -> int: return 2 zone = _household_zone() + readings_cache, summaries_cache = diary_store.open_stores() homeserver = os.environ.get("MATRIX_HOMESERVER", "").rstrip("/") if not homeserver: _err("MATRIX_HOMESERVER not set — is core up?") @@ -458,12 +507,14 @@ async def run(llm, argv: list[str]) -> int: _err(f"no transcription available: {e}") return 1 - async with aiohttp.ClientSession() as session: + # Recordings can be tens of megabytes; the default five seconds is + # for APIs, not for media. + async with httpx.AsyncClient(timeout=120.0, follow_redirects=True) as session: try: token = await _admin_token(session, homeserver) room_id = await _resolve_room(session, homeserver, token, room_arg) events = await _history(session, homeserver, token, room_id) - except (RuntimeError, aiohttp.ClientError) as e: + except (RuntimeError, httpx.HTTPError) as e: _err(str(e)) return 1 @@ -489,7 +540,8 @@ async def run(llm, argv: list[str]) -> int: continue decoded.append(msg.__class__(**{**msg.__dict__, "body": text})) - readings, continues, refers_to = await _read_room(decoded, llm) + readings, continues, refers_to = await _read_room( + decoded, llm, None if rebuild else readings_cache) await transcriber.aclose() @@ -505,7 +557,18 @@ async def run(llm, argv: list[str]) -> int: summaries = {} for key, in_month in sorted(months.items()): + digest = diary.month_digest(in_month) + kept = "" if rebuild else summaries_cache.get(key, digest) + if kept: + summaries[key] = kept + continue summaries[key] = await _summarise(in_month, llm) + if summaries[key]: + summaries_cache.put(key, digest, summaries[key]) + + if not dry_run: + readings_cache.save() + summaries_cache.save() pages = diary.pages_for(entries, room_id=room_id, summaries=summaries) if dry_run: diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index be75166..e9e4653 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -18,12 +18,18 @@ "search it literally" rather than as a failure. diary [--room ] [--burst-window ] [--dry-run] + [--rebuild] Compile the memories room into the family diary. Walks the - room's full history, transcribes every recording (cached in - TRANSCRIPT_DIR), recovers the date each one was made, and - publishes month pages under the shared bucket. Rerunnable: - the room is the source of truth, so a second run recompiles - rather than appends. See `cli/diary.py` for the date rules. + room's full history, transcribes every recording, recovers the + date each one was made, and publishes month pages under the + shared bucket. The curator runs it on the nightly sweep. + + Always a full pass, never an append: a reply or an edit + arriving tonight can belong to an entry from years back. It + stays cheap because transcripts, readings, and month summaries + are all kept against the thing they describe, so only what is + new costs anything. `--rebuild` ignores those and reads + everything again, for when the model has improved. wiki [--home] [--member ]... [--topic ]... [--dry-run] Regenerate the family wiki's entry pages. Apply by default; diff --git a/stacklets/memory/bot/curator.py b/stacklets/memory/bot/curator.py index 1378b64..088d0b6 100644 --- a/stacklets/memory/bot/curator.py +++ b/stacklets/memory/bot/curator.py @@ -764,23 +764,40 @@ def _reconcile(self) -> SyncResult: # ── Rebuild ────────────────────────────────────────────────────────────── +async def compile_diary() -> bool: + """One diary pass over the memories room, on the nightly sweep. + + The compiler re-reads the whole room every time, because a reply or + an edit arriving tonight can belong to an entry from years back, and + anything that walked forward from a watermark would never attach it. + Re-reading is affordable because what each message cost is on file: + the nightly pays for recordings and readings that are genuinely new + and reuses the rest. + """ + return await _run_command("diary", [], "compiling diary") + + async def rebuild(selection: list[str]) -> bool: """One wiki generation pass via the CLI entrypoint — the same code path `stack memory wiki` execs, in a subprocess so a wedged LLM call dies with the child instead of inside this loop.""" label = " ".join(selection) if selection else "(full sweep)" - logger.info("[curator] rebuilding wiki: {}", label) + return await _run_command("wiki", selection, f"rebuilding wiki: {label}") + + +async def _run_command(command: str, selection: list[str], what: str) -> bool: + logger.info("[curator] {}", what) # Starting the child and waiting on it are separate failure modes, and # only the second one has a child to kill. Keeping them in one block # left `proc.kill()` reachable on a path where `proc` was never bound. try: proc = await asyncio.create_subprocess_exec( - sys.executable, ENTRYPOINT, "wiki", *selection, + sys.executable, ENTRYPOINT, command, *selection, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.STDOUT, ) except Exception as e: - logger.warning("[curator] rebuild failed to start: {}", e) + logger.warning("[curator] {} failed to start: {}", command, e) return False try: @@ -789,19 +806,20 @@ async def rebuild(selection: list[str]) -> bool: ) except TimeoutError: proc.kill() - logger.warning("[curator] rebuild timed out after {}s", REBUILD_TIMEOUT_SECS) + logger.warning("[curator] {} timed out after {}s", + command, REBUILD_TIMEOUT_SECS) return False except Exception as e: - logger.warning("[curator] rebuild failed: {}", e) + logger.warning("[curator] {} failed: {}", command, e) return False output = out_bytes.decode(errors="replace").strip() if proc.returncode != 0: tail = "\n".join(output.splitlines()[-5:]) - logger.warning("[curator] wiki generation rc={}: {}", proc.returncode, tail) + logger.warning("[curator] {} rc={}: {}", command, proc.returncode, tail) return False published = sum(1 for ln in output.splitlines() if ln.startswith("published ")) - logger.info("[curator] wiki refreshed — {} page(s) published", published) + logger.info("[curator] {} done — {} page(s) published", command, published) return True @@ -963,6 +981,9 @@ async def mirror_reconcile() -> bool: _write(nightly_file, time.strftime("%Y-%m-%d", time.localtime())) if await mirror_reconcile(): mirror_sha = _write(mirror_file, head) + # Diary first: it writes pages into the same working tree, + # and the commit below should carry both nights' work. + await compile_diary() if await rebuild([]): # Generation wrote pages into brain's working tree; commit # and push them (one commit alongside the reconcile). diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 9647f40..b826848 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -23,6 +23,7 @@ from __future__ import annotations +import hashlib import re from dataclasses import dataclass, field, replace from datetime import date, datetime, timezone, tzinfo @@ -592,6 +593,26 @@ def month_key(on: date) -> str: return on.strftime("%m") +def month_digest(entries) -> str: + """A fingerprint of everything a month's summary was written from. + + The summary is cached against this, so it is recomputed exactly when + the month's content moves and not otherwise. It covers what the + summariser is shown -- who, when, and the words, including remarks + attached later -- so a memo surfacing months after the fact rewrites + the page it lands on, while an untouched month keeps its paragraph + word for word rather than being quietly reworded every night. + """ + parts = [] + for entry in sorted(entries, key=lambda e: (e.on, e.at)): + parts.append("\x1f".join([ + ",".join(entry.event_ids), entry.on.isoformat(), + entry.confidence, entry.sender, entry.body, + "|".join(f"{who}:{text}" for who, text in entry.comments), + ])) + return hashlib.sha256("\x1e".join(parts).encode("utf-8")).hexdigest() + + def _by_year(entries) -> "dict[str, list[Entry]]": out: dict[str, list[Entry]] = {} for entry in entries: diff --git a/stacklets/memory/bot/diary_store.py b/stacklets/memory/bot/diary_store.py new file mode 100644 index 0000000..e479003 --- /dev/null +++ b/stacklets/memory/bot/diary_store.py @@ -0,0 +1,138 @@ +"""What the diary compiler remembers between runs. + +The compiler is a fold over the whole room, and it stays one. That is +what makes it correct: a reply to a memo from March arrives in +September, an edit lands on a year-old note, a remark turns out to be +about a photo posted last spring. Anything that walked forward from a +watermark and appended would never attach those, and would be wrong in +a way nobody notices until they read the page. + +So the room is re-read every time and the pages are recomputed from +scratch. What is remembered is not *where we got to* but *what each +piece cost*: the model's reading of a message, and the paragraph +opening a month. Those are keyed by the thing they describe, so a +nightly run pays for what is genuinely new and nothing else, while a +recompile still produces the same diary it would have produced from an +empty cache. + +Single-writer, like the archivist's tag cache: one batch at a time, no +locking. Every failure mode -- missing file, corrupt JSON, unwritable +directory -- degrades to an empty cache, because paying for a reading +twice is cheaper than a compile that refuses to run. +""" + +from __future__ import annotations + +import json +import os +from pathlib import Path + +from loguru import logger + +DEFAULT_STATE_DIR = "/data/memory/diary" + + +def state_dir() -> Path: + return Path(os.environ.get("DIARY_STATE_DIR", DEFAULT_STATE_DIR)) + + +class JsonStore: + """A dict on disk, written whole. + + One file rather than a file per key: the diary compiles as a single + batch, so there is no concurrent writer to lose, and a room with ten + thousand messages is a couple of megabytes rather than ten thousand + inodes. + """ + + section = "items" + + def __init__(self, path: Path): + self.path = Path(path) + self._items: dict[str, dict] = {} + + def load(self) -> "JsonStore": + try: + data = json.loads(self.path.read_text(encoding="utf-8")) + except FileNotFoundError: + self._items = {} + return self + except (json.JSONDecodeError, OSError) as e: + logger.warning("[diary] unreadable cache at {} ({}), starting empty", + self.path, e) + self._items = {} + return self + raw = data.get(self.section) if isinstance(data, dict) else None + self._items = raw if isinstance(raw, dict) else {} + return self + + def save(self) -> None: + """Write the cache, or log and carry on. + + A cache that cannot be saved costs the next run some model + calls. Failing the compile over it would cost the family their + diary, which is the worse trade. + """ + try: + self.path.parent.mkdir(parents=True, exist_ok=True) + tmp = self.path.with_suffix(".tmp") + tmp.write_text(json.dumps({self.section: self._items}, + ensure_ascii=False, indent=2), + encoding="utf-8") + os.replace(tmp, self.path) + except OSError as e: + logger.warning("[diary] could not save cache {}: {}", self.path, e) + + def __len__(self) -> int: + return len(self._items) + + +class ReadingStore(JsonStore): + """The model's reading of each message, keyed by event id. + + A message never changes: an edit is a new event pointing at the old + one, so a reading is good forever and this cache never needs + invalidating. The links between messages are deliberately not kept + here -- they are a property of a slice of history rather than of one + message, and a later message can create one. + """ + + section = "readings" + + def get(self, event_id: str) -> dict | None: + found = self._items.get(event_id) + return found if isinstance(found, dict) else None + + def put(self, event_id: str, reading: dict) -> None: + self._items[event_id] = reading + + +class SummaryStore(JsonStore): + """The paragraph opening each month, keyed by month and by content. + + Keyed by a digest of the month's entries as well as its name, + because a month is not finished when it ends: a memo recorded in + March can surface in September and belongs on March's page. When + that happens the digest moves and the month is written again. When + nothing moved, the paragraph is reused word for word, which is also + what stops a nightly run quietly rewording the family's past. + """ + + section = "summaries" + + def get(self, month: str, digest: str) -> str: + found = self._items.get(month) + if not isinstance(found, dict) or found.get("digest") != digest: + return "" + text = found.get("text") + return text if isinstance(text, str) else "" + + def put(self, month: str, digest: str, text: str) -> None: + self._items[month] = {"digest": digest, "text": text} + + +def open_stores(directory: Path | None = None): + """Both caches, loaded. Missing files are simply empty ones.""" + root = Path(directory) if directory else state_dir() + return (ReadingStore(root / "readings.json").load(), + SummaryStore(root / "summaries.json").load()) diff --git a/stacklets/memory/cli/diary.py b/stacklets/memory/cli/diary.py index f205af4..c8b3e94 100644 --- a/stacklets/memory/cli/diary.py +++ b/stacklets/memory/cli/diary.py @@ -9,6 +9,20 @@ stack memory diary --dry-run print the pages, write nothing stack memory diary --room memories read a different room stack memory diary --burst-window 1 tighten sync-burst detection + stack memory diary --rebuild read it all again from scratch + +WHEN IT RUNS + The curator compiles the diary on its nightly sweep, so new + recordings reach the wiki overnight without anyone asking. Running + it by hand does the same thing sooner. + + Every run is a full pass over the room rather than an append. It has + to be: a reply to a memo from March can arrive in September, an edit + can land on a year-old note, and a remark can turn out to be about a + photo from last spring. A compiler that walked forward from where it + last stopped would never attach any of them. It stays cheap because + what each recording and each reading cost is kept against the + message it belongs to, so a nightly pass pays only for what is new. WHAT IT RECOVERS Matrix stamps an event with the time the server received it, never diff --git a/stacklets/memory/docker-compose.yml b/stacklets/memory/docker-compose.yml index c968d95..527595c 100644 --- a/stacklets/memory/docker-compose.yml +++ b/stacklets/memory/docker-compose.yml @@ -81,6 +81,15 @@ services: MEMORY_VAULT_DIR: /data/memory/vault BRAIN_REPO_DIR: /data/memory/brain CURATOR_STATE_DIR: /data/memory/curator + DIARY_STATE_DIR: /data/memory/diary + # The diary compiler reads the memories room and decodes its + # recordings, so the curator needs the homeserver, whisper, and + # the transcript cache the bots already fill. + MATRIX_HOMESERVER: ${MATRIX_HOMESERVER} + MATRIX_SERVER_NAME: ${MATRIX_SERVER_NAME} + WHISPER_URL: ${WHISPER_URL} + TRANSCRIPT_DIR: /data/core/transcripts + TIMEZONE: ${TIMEZONE:-UTC} # Forgejo, container-network address — memory's own CODE_URL in # .env is the host-side URL the install hooks use. CODE_URL: http://stack-code:3000 @@ -107,6 +116,10 @@ services: - ../../lib/stack:/app/stack:ro # Vault working copy (the curator pulls it) + state dir. - ${MEMORY_DATA_DIR}:/data/memory + # Transcripts, shared with the bot-runner. Writable: a diary + # backfill decodes recordings no bot has heard yet, and the next + # bot to meet one should not pay for it again. + - ${CORE_DATA_DIR}:/data/core restart: unless-stopped networks: diff --git a/stacklets/memory/stacklet.toml b/stacklets/memory/stacklet.toml index d7d2e1b..54f1819 100644 --- a/stacklets/memory/stacklet.toml +++ b/stacklets/memory/stacklet.toml @@ -76,6 +76,19 @@ BRAIN_REPO_DIR = "{data_dir}/memory/brain" # containers expect. MEMORY_DATA_DIR = "{data_dir}/memory" +# What the diary compiler remembers between runs: the model's reading of +# each message and the paragraph opening each month. Not a record of how +# far it got -- the compiler re-reads the whole room every time, because +# a reply or an edit can land on an entry from years back. These are a +# cache of what each piece cost, so a nightly pass pays only for what is +# new. +DIARY_STATE_DIR = "{data_dir}/memory/diary" + +# The shared transcript cache, written by the bots and read by the diary +# backfill. Same directory core gives the bot-runner: whisper costs +# minutes of GPU per recording and neither side should pay twice. +CORE_DATA_DIR = "{data_dir}/core" + # Curator runtime — the slim sidecar runs the same wiki generation the # CLI does, so it needs the LLM endpoint (docker-side), model config, # Forgejo admin credentials (CODE_URL is overridden to the container- @@ -85,6 +98,15 @@ OPENAI_KEY = "{ai_openai_key}" AI_DEFAULT_MODEL = "{ai_default_model}" AI_MODELS_JSON = "{ai_models_json}" LANGUAGE = "{language}" +# The memories room the diary compiler reads, and the whisper endpoint +# it decodes recordings with. Same values core gives the bot-runner: +# the curator runs the compiler on its nightly sweep, so it needs them +# too. The literal homeserver address resolves on the stack network +# whether or not messages was installed first. +MATRIX_HOMESERVER = "http://stack-messages-synapse:8008" +MATRIX_SERVER_NAME = "{messages_server_name}" +WHISPER_URL = "{ai_whisper_url_docker}/audio/transcriptions" + TIMEZONE = "{timezone}" MATRIX_ADMIN_USER = "{admin_username}" MATRIX_ADMIN_PASSWORD = "{admin_password}" diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 73c52fa..821e248 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -34,6 +34,7 @@ sys.path.insert(0, str(_REPO_ROOT / "tools" / "family-memories")) import diary # noqa: E402 +import diary_store # noqa: E402 from ingest import burst_ordered # noqa: E402 SPEC = _REPO_ROOT / "tools" / "family-memories" / "spec.en.yaml" @@ -686,3 +687,97 @@ def test_a_month_page_is_titled_with_its_year(self): assert titles["diary/2026/03.md"] == "March 2026" assert titles["diary/2026/about.md"] == "2026" + + +# ── What survives between runs ──────────────────────────────────────── + + +class TestRememberingBetweenRuns: + """The nightly must pay for what is new and nothing else. + + Deliberately not a watermark. The compiler re-reads the whole room + every run, because a reply, an edit, or a remark arriving tonight + can belong to an entry from years back, and anything walking forward + from a last-processed id would never attach it. What is kept is what + each message cost, keyed by the message. + """ + + def test_a_reading_survives_a_restart(self, tmp_path): + readings, _ = diary_store.open_stores(tmp_path) + readings.put("$a", {"mode": "monologue", "spoken_date": "2026-03-16", + "addressee": "Bart", "continues": None, + "refers_to": None}) + readings.save() + + reopened, _ = diary_store.open_stores(tmp_path) + + assert reopened.get("$a")["spoken_date"] == "2026-03-16" + + def test_a_link_is_remembered_with_the_message_that_carries_it(self, tmp_path): + """A slice read end to end is never sent again, so its links + have to come back with it or a joined recording would split.""" + readings, _ = diary_store.open_stores(tmp_path) + readings.put("$b", {"mode": "monologue", "continues": "$a", + "refers_to": None}) + readings.save() + + reopened, _ = diary_store.open_stores(tmp_path) + + assert reopened.get("$b")["continues"] == "$a" + + def test_a_first_run_finds_an_empty_cache(self, tmp_path): + readings, summaries = diary_store.open_stores(tmp_path / "nothing-here") + + assert len(readings) == 0 + assert readings.get("$a") is None + assert summaries.get("2026-03", "digest") == "" + + def test_a_corrupt_cache_is_not_a_broken_compile(self, tmp_path): + """Paying for a reading twice beats refusing to run.""" + (tmp_path / "readings.json").write_text("{not json at all") + + readings, _ = diary_store.open_stores(tmp_path) + + assert readings.get("$a") is None + + def test_an_unchanged_month_keeps_the_words_it_had(self, tmp_path): + """Not only a saving. Re-summarising a settled month every night + would reword the family's past while they slept. + """ + _, summaries = diary_store.open_stores(tmp_path) + summaries.put("2026-03", "abc", "A month of firsts.") + + assert summaries.get("2026-03", "abc") == "A month of firsts." + + def test_a_month_that_moved_is_written_again(self, tmp_path): + _, summaries = diary_store.open_stores(tmp_path) + summaries.put("2026-03", "abc", "A month of firsts.") + + assert summaries.get("2026-03", "xyz") == "" + + +class TestMonthDigest: + def test_the_same_month_fingerprints_the_same(self): + march = [e for e in _compile() if e.on.month == 3] + + assert diary.month_digest(march) == diary.month_digest(march) + + def test_a_reply_arriving_later_moves_the_month(self): + """A memo from March can collect a remark in September. The + March page has to be written again when it does. + """ + march = [e for e in _compile() if e.on.month == 3] + before = diary.month_digest(march) + march[0].comments.append(("homer", "He gets that from me.")) + + assert diary.month_digest(march) != before + + def test_a_memo_surfacing_late_moves_the_month_it_lands_in(self): + march = [e for e in _compile() if e.on.month == 3] + latecomer = diary.Entry( + on=date(2026, 3, 30), confidence="spoken", basis="b", + kind="voice", sender="marge", body="one more thing", + at=BASE_TS, event_ids=["$late"]) + + assert diary.month_digest(march + [latecomer]) \ + != diary.month_digest(march) From 677cbef6ad69bce8855feb72775211f2092a3290 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 09:38:28 +0200 Subject: [PATCH 09/43] fix(memory): stop a later memory becoming a footnote on an older one A recording ended "the doctor says the cast comes off in four weeks". A note months later said it did. The reader tied the second to the first as a remark about it, so a memory of its own lost its date, its place in the month, and was credited as a reply nobody made. Related is not the same as subordinate. A remark now has to be about something still in view when it was written: a caption arrives while its photo is on screen, a follow-up weeks later does not. The reader is told the difference too, but the room is what checks it. --- stacklets/memory/bot/cli/diary.py | 7 +++++- stacklets/memory/bot/diary.py | 34 +++++++++++++++++++++++++++- tests/stacklets/test_memory_diary.py | 31 +++++++++++++++++++++++++ 3 files changed, 70 insertions(+), 2 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index d7f6321..540e4a5 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -227,7 +227,12 @@ async def produce() -> dict: "refers_to": the number of an earlier message this one is a remark about rather than a memory of its own: a caption for a photo, or a - line like "the picture above is from the barbecue". Otherwise null. + line like "the picture above is from the barbecue". A message that + reports something that happened is a memory of its own even when it + follows up on an earlier one -- "his cast came off today" answers a + recording from weeks ago and is still its own memory, not a footnote + to it. Use this only when the message would make no sense on its own + page. Otherwise null. """ diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index b826848..269b6ea 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -409,6 +409,38 @@ def date_for(msg: Message, reading: Reading) -> tuple[date, str, str]: # ── Step 4: compile ─────────────────────────────────────────────────── +# How long a remark may trail the thing it is about. A caption is posted +# while its photo is still on screen; a follow-up months later is its own +# memory, however clearly it answers an older one. Time rather than +# position, so a busy evening and a quiet week are judged the same way. +REMARK_WINDOW_S = 3600.0 + + +def _remarks_only_on_what_is_still_in_view(messages, refers_to): + """Drop `refers_to` links that reach back beyond living memory. + + The model is asked which messages are remarks about an earlier one + rather than memories of their own, and it is right about captions. + It also, given a recording that ends "the cast comes off in four + weeks" and a note four weeks later saying it did, reads the second + as a remark on the first. They are related; that does not make the + later one a footnote. Filing it as one costs it its own date, its + own place in the month, and credits it as a reply nobody made. + + So the room checks the reading, as it does for joined recordings: a + remark belongs to something still in view when it was written. + """ + at = {m.event_id: m.ts for m in messages} + kept = {} + for child, parent in refers_to.items(): + if child not in at or parent not in at: + continue + gap = (at[child] - at[parent]) / 1000.0 + if 0 <= gap <= REMARK_WINDOW_S: + kept[child] = parent + return kept + + def compile_entries(messages, readings, *, continues: "dict[str, str] | None" = None, refers_to: "dict[str, str] | None" = None) -> list[Entry]: @@ -423,7 +455,7 @@ def compile_entries(messages, readings, *, remark floating with no subject. """ continues = continues or {} - about = refers_to or {} + about = _remarks_only_on_what_is_still_in_view(messages, refers_to or {}) # Join first (a split recording is two adjacent uploads), then # decide which runs were really a queue being flushed. Dating reads diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 821e248..6397252 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -546,6 +546,37 @@ def test_a_remark_about_a_photo_is_filed_under_that_photo(self): assert "$impliziter-kontext" in photo.event_ids assert "barbecue" in photo.comments[0][1] + def test_a_follow_up_months_later_keeps_its_own_page(self): + """Related is not the same as subordinate. + + A recording ends "the cast comes off in four weeks"; a note four + weeks later says it did. A reader sees a remark on the first. + Filing it as one costs that note its own date, its own place in + the month, and credits it as a reply nobody made. + """ + memo = _msg(event_id="$memo", sender="homer", ts=BASE_TS, + body="the doctor says the cast comes off in four weeks") + later = _msg(event_id="$later", sender="homer", kind="text", + ts=BASE_TS + 28 * 86_400_000, + body="Bart got his cast off today.") + + entries = diary.compile_entries( + [memo, later], {}, refers_to={"$later": "$memo"}) + + assert len(entries) == 2 + assert not any(e.comments for e in entries) + + def test_a_caption_minutes_behind_its_photo_still_belongs_to_it(self): + photo = _msg(event_id="$photo", kind="image", ts=BASE_TS, body="") + caption = _msg(event_id="$cap", kind="text", ts=BASE_TS + 90_000, + body="Maggie drew this today.") + + entries = diary.compile_entries( + [photo, caption], {}, refers_to={"$cap": "$photo"}) + + assert len(entries) == 1 + assert entries[0].comments[0][1] == "Maggie drew this today." + def test_a_reply_to_a_memo_is_filed_under_that_memo(self): entries = _compile() From 32eef02431cfe736aa3284e8980be311869ae2f2 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 09:53:31 +0200 Subject: [PATCH 10/43] test(memory): a reply may correct a memory of any age Replying to a memo is how the family adds to it or puts it right, and they do that whenever they happen to reread it. The age guard added alongside exists for links the reader inferred, never for one the family drew themselves, so a reply attaches however old its parent is and the memory keeps the date it happened on. Pinned because the two paths look alike in the code and only one of them should ever reach back. --- tests/stacklets/test_memory_diary.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 6397252..2dfd5f5 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -546,6 +546,30 @@ def test_a_remark_about_a_photo_is_filed_under_that_photo(self): assert "$impliziter-kontext" in photo.event_ids assert "barbecue" in photo.comments[0][1] + def test_a_reply_reaches_back_as_far_as_it_likes(self): + """Pointing at a memory is not the same as being read as one. + + Replying to a memo from March is how the family corrects or adds + to it, and they do that whenever they happen to reread it. The + age guard exists for links the model inferred; a reply carries + the family's own intent, so it attaches however old its parent + is -- and the memory keeps March's date, because that is when it + happened. + """ + memo = _msg(event_id="$memo", sender="marge", ts=BASE_TS, + body="Hi Bart, today is March 16th.") + correction = _msg( + event_id="$fix", sender="marge", kind="text", + ts=BASE_TS + 180 * 86_400_000, reply_to="$memo", + body="It was Principal Skinner who called, not Mrs Krabappel.") + + entries = diary.compile_entries( + [memo, correction], {"$memo": diary.Reading(spoken_date="2026-03-16")}) + + assert len(entries) == 1 + assert entries[0].on == date(2026, 3, 16) + assert "Principal Skinner" in entries[0].comments[0][1] + def test_a_follow_up_months_later_keeps_its_own_page(self): """Related is not the same as subordinate. From d77a13509cb69b830aa4f7e88741d6a75a71f5e2 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 10:19:51 +0200 Subject: [PATCH 11/43] fix(memory): give videos and files a name on the page A clip posted to the memories room read "video, 0:04" in a line of otherwise written English, because only photos and notes had a word for themselves and everything else fell through to the internal one. It also offered to "open" a video rather than watch it, and said nothing where a photo would have said it arrived without a caption. Every kind the compiler accepts now has a name, a length where it has one, and its own way of pointing back into the room. --- stacklets/memory/bot/diary.py | 23 +++++++++++++++++------ tests/stacklets/test_memory_diary.py | 12 +++++++++++- 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 269b6ea..52d5b39 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -553,12 +553,22 @@ def _duration(ms: int | None) -> str: return f"{total // 60}:{total % 60:02d}" +# What each kind of entry is called on the page. Every kind the +# compiler accepts needs a name here: falling through to the internal +# word prints "video" in a line of otherwise written English. +_KIND_NOUNS = { + "image": "Photo", "video": "Video", "file": "File", + "text": "Written note", +} + + def _kind_label(entry: Entry) -> str: if entry.kind == "voice": noun = "Conversation" if entry.mode == "dialogue" else "Voice note" - length = _duration(entry.duration_ms) - return f"{noun}, {length}" if length else noun - return {"image": "Photo", "text": "Written note"}.get(entry.kind, entry.kind) + else: + noun = _KIND_NOUNS.get(entry.kind, "Attachment") + length = _duration(entry.duration_ms) + return f"{noun}, {length}" if length else noun def _permalink(room_id: str, event_id: str) -> str: @@ -591,8 +601,8 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: if entry.body.strip(): lines += [entry.body.strip(), ""] - elif entry.kind == "image" and not entry.comments: - lines += ["No caption came with this one.", ""] + elif entry.kind in _UPLOADS and not entry.comments: + lines += ["Nothing was written alongside this one.", ""] for who_replied, text in entry.comments: lines += [f"> [!quote] {who_replied.title()} replied", ] @@ -600,7 +610,8 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: lines.append("") if room_id and entry.event_ids: - label = {"voice": "Listen in the room", "image": "See it in the room"} + label = {"voice": "Listen in the room", "image": "See it in the room", + "video": "Watch it in the room"} lines.append( f"[{label.get(entry.kind, 'Open in the room')}]" f"({_permalink(room_id, entry.event_ids[0])})" diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 2dfd5f5..701ead7 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -632,12 +632,22 @@ def test_an_uncertain_entry_says_so_where_it_is_read(self): assert "[!warning]" in page assert "sync burst" in page + def test_every_kind_of_upload_is_named_in_plain_words(self): + """A kind with no name falls through to the internal word and + prints "video" in a line of otherwise written English.""" + for kind, expected in (("image", "Photo"), ("video", "Video"), + ("file", "File"), ("text", "Written note")): + page = diary.render_month([diary.Entry( + on=date(2026, 9, 13), confidence="sent", basis="b", kind=kind, + sender="bart", body="")]) + assert expected in page, kind + def test_a_photo_without_a_caption_says_so_rather_than_naming_a_file(self): page = diary.render_month([diary.Entry( on=date(2026, 4, 3), confidence="sent", basis="dated from when it " "was sent", kind="image", sender="homer", body="")]) - assert "No caption came with this one." in page + assert "Nothing was written alongside this one." in page assert ".png" not in page def test_a_speaker_is_not_addressed_to_themselves(self): From 251ab47d5fc4308b83a0af3404e3dd1d4a777263 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 10:30:24 +0200 Subject: [PATCH 12/43] fix(memory): tell whisper who lives here before it decodes A memo opening "Bart, today is July the fifth" transcribed as "Part", and without the name at all when the audio was quieter. The polish pass could not repair it: it is forbidden from changing words, and it should be, because the memories room holds what people said to their children. The only place to fix a name is before the audio is read. Whisper now decodes against the household's own vocabulary, taken from the people the wiki knows and the topics the ontology names. The same memo now comes back with the name in it. A reading is also kept against the words it was taken from rather than just the message. A better transcript used to leave the old reading in place, still addressed to nobody; now it re-reads itself. --- lib/stack/ai/client.py | 11 ++++ stacklets/memory/bot/cli/diary.py | 85 ++++++++++++++++++++++++++-- stacklets/memory/bot/diary.py | 37 ++++++++++++ stacklets/memory/bot/diary_store.py | 34 +++++++---- tests/stacklets/test_memory_diary.py | 66 +++++++++++++++++++-- 5 files changed, 212 insertions(+), 21 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index 1ab1ee4..65920dd 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -424,6 +424,7 @@ def from_env(cls, *, namespace: str | None = None, async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", model: str | None = None, + vocabulary: str = "", cleanup_with: "LLM | None" = None) -> str: """Transcribe audio bytes to text, stripped of leading/trailing space. @@ -432,6 +433,15 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", the SDK for OpenAI-compat servers that route by model name; the native whisper-server ignores it. + ``vocabulary`` is a hint about words this household says: the + names of the people in it, the topics they keep. Whisper decodes + against it, so a family name it would otherwise hear as a common + word comes back right the first time. That matters more than it + sounds: a memo opening "Bart, today is..." transcribes as "Part" + or loses the name entirely, and the polish pass cannot repair it + without rewriting what was said, which it is forbidden to do. + Fixing the input is the only way to fix the words. + ``cleanup_with`` is an optional :class:`LLM` to polish the raw STT output with punctuation and sentence breaks. When provided, the result is the LLM-cleaned text; when omitted or the LLM call @@ -448,6 +458,7 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", model=model or _DEFAULT_WHISPER_MODEL, file=(filename, audio), response_format="json", + **({"prompt": vocabulary} if vocabulary.strip() else {}), ) except openai.APITimeoutError as e: raise LLMTimeoutError( diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 540e4a5..4360ec1 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -47,6 +47,7 @@ # container that can talk to a model, so the compiler runs unchanged in # the bot-runner and in the curator that schedules it nightly. import httpx +import yaml sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) # bot/ sys.path.insert(0, "/app") # stack.ai.client, and voice in the bot-runner @@ -149,8 +150,77 @@ async def _download(session, homeserver, token, mxc: str) -> bytes | None: # ── Decoding and reading ────────────────────────────────────────────── +# Whisper's decoder prompt is a small window (a couple of hundred +# tokens); past it the hint is truncated from the front, which would +# drop the names silently. Names first, topics only with room to spare. +_VOCAB_BUDGET = 600 + + +def _household_vocabulary() -> str: + """The names and subjects this family uses, for whisper to decode against. + + People come from the wiki's person pages, which already carry the + household's own spelling of each name and any variants it uses. The + ontology's topics follow, because a family's proper nouns are not + only its people -- a campsite, a school, a pet -- and those mishear + just as readily. + + Best-effort: a vault that has not been generated yet simply yields + nothing, and transcription proceeds exactly as it did before. + """ + people: list[str] = [] + brain = Path(os.environ.get("BRAIN_REPO_DIR", "")) + if brain.is_dir(): + for about in sorted(brain.glob("*/about.md")): + front = _frontmatter(about) + if front.get("type") != "person": + continue + for key in ("canonical", "title"): + if value := front.get(key): + people.append(str(value)) + break + synonyms = front.get("synonyms") + if isinstance(synonyms, list): + people.extend(str(x) for x in synonyms) + + topics: list[str] = [] + vault = Path(os.environ.get("MEMORY_VAULT_DIR", "")) + lang = os.environ.get("LANGUAGE", "en") + try: + from stack.ontology import Ontology + loaded = Ontology.load(vault / "ontology.toml") + for topic in loaded.topics.values(): + topics.append(topic.name(lang)) + topics.extend(topic.synonyms_for(lang)) + except Exception as e: # noqa: BLE001 - a hint is never worth failing over + _err(f" no ontology for the transcript hint: {e}") + + hint = diary.spoken_vocabulary(people, topics) + if len(hint) <= _VOCAB_BUDGET: + return hint + # Over budget: keep the people, who are what mishears most. + return diary.spoken_vocabulary(people)[:_VOCAB_BUDGET] + + +def _frontmatter(path: Path) -> dict: + try: + text = path.read_text(encoding="utf-8") + except OSError: + return {} + if not text.startswith("---\n"): + return {} + end = text.find("\n---", 4) + if end < 0: + return {} + try: + loaded = yaml.safe_load(text[4:end]) + except yaml.YAMLError: + return {} + return loaded if isinstance(loaded, dict) else {} + + async def _transcribe(message, *, session, homeserver, token, - transcriber, llm) -> str: + transcriber, llm, vocabulary: str = "") -> str: """The words of a recording, transcribed once and remembered. Shares `TRANSCRIPT_DIR` with the bots, so a memo the archivist @@ -162,7 +232,8 @@ async def produce() -> dict: audio = await _download(session, homeserver, token, message.url or "") if not audio: raise LLMError(f"could not download {message.url}") - raw = await transcriber.transcribe(audio, filename=message.body or "voice.wav") + raw = await transcriber.transcribe( + audio, filename=message.body or "voice.wav", vocabulary=vocabulary) text = await Transcriber.polish(raw, llm) if raw.strip() else raw return {"raw": raw, "text": text, "url": message.url, "filename": message.body} @@ -310,7 +381,7 @@ def remember(event_id: str, row: dict) -> None: if cache is not None: for msg in messages: - if (stored := cache.get(msg.event_id)) is not None: + if (stored := cache.get(msg.event_id, msg.body)) is not None: remember(msg.event_id, stored) if known: _err(f" {len(known)} message(s) already read, " @@ -350,7 +421,7 @@ def remember(event_id: str, row: dict) -> None: } remember(here.event_id, found) if cache is not None: - cache.put(here.event_id, found) + cache.put(here.event_id, found, here.body) return readings, continues, refers_to @@ -532,6 +603,10 @@ async def run(llm, argv: list[str]) -> int: # Transcription first and on its own: every later step reads # words, and a recording that cannot be decoded should drop out # before the model is asked to interpret its filename. + vocabulary = _household_vocabulary() + if vocabulary: + _err(f" decoding against: {vocabulary[:90]}...") + decoded = [] for msg in messages: if msg.kind != "voice": @@ -539,7 +614,7 @@ async def run(llm, argv: list[str]) -> int: continue text = await _transcribe( msg, session=session, homeserver=homeserver, token=token, - transcriber=transcriber, llm=llm) + transcriber=transcriber, llm=llm, vocabulary=vocabulary) if not text.strip(): _err(f" no speech in {msg.event_id}, skipped") continue diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 52d5b39..fdbef9e 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -117,6 +117,43 @@ class Entry: comments: list[tuple[str, str]] = field(default_factory=list) +# ── What this household says ────────────────────────────────────────── + + +def spoken_vocabulary(people, topics=()) -> str: + """A hint for whisper about the words this family uses. + + Whisper decodes against it, so names it would otherwise hear as + ordinary words come back right. This is the only place a name can be + fixed: the polish pass is forbidden from changing words, and it is + right to be -- the memories room holds what people said to their + children, and a model quietly editing that is not a transcript any + more. So the input is corrected instead of the output. + + Phrased as a sentence rather than a bare list because that is what + the parameter is for: whisper treats it as preceding speech, and a + list of nouns biases the decoder toward answering in lists. + """ + names = [n.strip() for n in people if n and n.strip()] + subjects = [t.strip() for t in topics if t and t.strip()] + parts = [] + if names: + parts.append("The people in this family are " + + _and_list(_unique(names)) + ".") + if subjects: + parts.append("They often talk about " + + _and_list(_unique(subjects)) + ".") + return " ".join(parts) + + +def _unique(values): + seen = [] + for v in values: + if v not in seen: + seen.append(v) + return seen + + # ── Step 1: resolve ─────────────────────────────────────────────────── # # Pure Matrix mechanics, no reading of meaning. Edits collapse onto the diff --git a/stacklets/memory/bot/diary_store.py b/stacklets/memory/bot/diary_store.py index e479003..1a71fc0 100644 --- a/stacklets/memory/bot/diary_store.py +++ b/stacklets/memory/bot/diary_store.py @@ -23,6 +23,7 @@ from __future__ import annotations +import hashlib import json import os from pathlib import Path @@ -88,23 +89,36 @@ def __len__(self) -> int: class ReadingStore(JsonStore): - """The model's reading of each message, keyed by event id. + """The model's reading of each message, keyed by the words it read. - A message never changes: an edit is a new event pointing at the old - one, so a reading is good forever and this cache never needs - invalidating. The links between messages are deliberately not kept - here -- they are a property of a slice of history rather than of one - message, and a later message can create one. + An edit arrives as a new event, so a message's identity never moves. + Its words can: hand whisper the household's names and a memo it once + heard as "Part" comes back as "Bart", and the reading taken from the + old wording is now wrong about who was being spoken to. So the words + are part of the key, and a better transcript re-reads itself. + + The links between messages are deliberately not kept here -- they + are a property of a slice of history rather than of one message, and + a later message can create one. """ section = "readings" - def get(self, event_id: str) -> dict | None: + def get(self, event_id: str, body: str = "") -> dict | None: found = self._items.get(event_id) - return found if isinstance(found, dict) else None + if not isinstance(found, dict): + return None + if found.get("said") != _said(body): + return None + return found + + def put(self, event_id: str, reading: dict, body: str = "") -> None: + self._items[event_id] = {**reading, "said": _said(body)} + - def put(self, event_id: str, reading: dict) -> None: - self._items[event_id] = reading +def _said(body: str) -> str: + """A fingerprint of the words a reading was taken from.""" + return hashlib.sha256(body.strip().encode("utf-8")).hexdigest()[:16] class SummaryStore(JsonStore): diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 701ead7..31a3938 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -771,30 +771,43 @@ def test_a_reading_survives_a_restart(self, tmp_path): readings, _ = diary_store.open_stores(tmp_path) readings.put("$a", {"mode": "monologue", "spoken_date": "2026-03-16", "addressee": "Bart", "continues": None, - "refers_to": None}) + "refers_to": None}, "today is March 16th") readings.save() reopened, _ = diary_store.open_stores(tmp_path) - assert reopened.get("$a")["spoken_date"] == "2026-03-16" + assert reopened.get("$a", "today is March 16th")["spoken_date"] \ + == "2026-03-16" + + def test_a_better_transcript_is_read_again(self, tmp_path): + """Give whisper the household's names and a memo it heard as + "Part" comes back as "Bart". The reading taken from the old + wording is now wrong about who was spoken to, so it has to go. + """ + readings, _ = diary_store.open_stores(tmp_path) + readings.put("$a", {"addressee": None}, + "Part, today is July the 5th") + + assert readings.get("$a", "Part, today is July the 5th") is not None + assert readings.get("$a", "Bart, today is July the 5th") is None def test_a_link_is_remembered_with_the_message_that_carries_it(self, tmp_path): """A slice read end to end is never sent again, so its links have to come back with it or a joined recording would split.""" readings, _ = diary_store.open_stores(tmp_path) readings.put("$b", {"mode": "monologue", "continues": "$a", - "refers_to": None}) + "refers_to": None}, "and then") readings.save() reopened, _ = diary_store.open_stores(tmp_path) - assert reopened.get("$b")["continues"] == "$a" + assert reopened.get("$b", "and then")["continues"] == "$a" def test_a_first_run_finds_an_empty_cache(self, tmp_path): readings, summaries = diary_store.open_stores(tmp_path / "nothing-here") assert len(readings) == 0 - assert readings.get("$a") is None + assert readings.get("$a", "x") is None assert summaries.get("2026-03", "digest") == "" def test_a_corrupt_cache_is_not_a_broken_compile(self, tmp_path): @@ -803,7 +816,7 @@ def test_a_corrupt_cache_is_not_a_broken_compile(self, tmp_path): readings, _ = diary_store.open_stores(tmp_path) - assert readings.get("$a") is None + assert readings.get("$a", "x") is None def test_an_unchanged_month_keeps_the_words_it_had(self, tmp_path): """Not only a saving. Re-summarising a settled month every night @@ -846,3 +859,44 @@ def test_a_memo_surfacing_late_moves_the_month_it_lands_in(self): assert diary.month_digest(march + [latecomer]) \ != diary.month_digest(march) + + +# ── Telling whisper who lives here ──────────────────────────────────── + + +class TestSpokenVocabulary: + """The only place a misheard name can be put right. + + The polish pass may not change words, and should not: the memories + room holds what people said to their children. So the names go in + before the audio is decoded rather than after. + """ + + def test_the_family_is_named(self): + hint = diary.spoken_vocabulary(["Bart", "Lisa", "Maggie"]) + + assert "Bart, Lisa and Maggie" in hint + + def test_topics_follow_the_people(self): + hint = diary.spoken_vocabulary(["Marge"], ["camping", "the PTA"]) + + assert hint.index("Marge") < hint.index("camping") + + def test_a_name_said_twice_is_written_once(self): + hint = diary.spoken_vocabulary(["Bart", "Bart", "Lisa"]) + + assert hint.count("Bart") == 1 + + def test_nothing_known_is_no_hint_at_all(self): + """An ungenerated vault must not send whisper an empty sentence + to decode against.""" + assert diary.spoken_vocabulary([], []) == "" + assert diary.spoken_vocabulary(["", " "]) == "" + + def test_it_reads_as_speech_not_as_a_word_list(self): + """Whisper treats the hint as preceding speech, so a bare list + biases the decoder toward answering in lists.""" + hint = diary.spoken_vocabulary(["Homer"], ["camping"]) + + assert hint.endswith(".") + assert "The people in this family are Homer." in hint From c709d4f207b09f5041b3f04505af880272931864 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 10:31:34 +0200 Subject: [PATCH 13/43] docs(brain): record what the memories pipeline settled and what is left Two open questions closed by building it, and the answers were not the ones the draft expected. The burst window stopped being a threshold to tune: timing alone marked an ordinary evening undateable, so a run now has to contradict itself before its timestamps are doubted. Join thresholds went away entirely. Two new ones noted: giving the polish pass the same household vocabulary whisper now decodes against, and an explicit way to re-transcribe when that vocabulary changes. --- docs/design/brain/memories-pipeline.md | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/design/brain/memories-pipeline.md b/docs/design/brain/memories-pipeline.md index e81d6cc..6567fac 100644 --- a/docs/design/brain/memories-pipeline.md +++ b/docs/design/brain/memories-pipeline.md @@ -77,5 +77,25 @@ room history (paginated, oldest-first) `dev.famstack.recorded_ts` into event content would eliminate the uncertain class for future memos. Family habit of speaking the date covers the past. -- Burst window (120s) and join thresholds: tune against the corpus - (`true_date` / `fragment_of` ground truth in the manifests). +- Burst window (120s): settled differently than expected. Timing alone + turned out to be the wrong signal -- three memos recorded a minute + apart at the dinner table are not a sync burst, and calling them one + filed a normal evening as undateable. A run now has to contradict + itself (some memo says aloud it was made on a day its own timestamp + disagrees with) before its timestamps are distrusted, which leaves + the window doing nothing but grouping what arrived together. +- Join thresholds: gone. The model names which message finishes which, + having both in front of it; the room checks the link is adjacent, + same sender, same kind. + +- The polish pass could take the household vocabulary too. Whisper now + decodes against the family's names and topics, which is where a + misheard name has to be fixed -- polish may not change words and + should not. But the same vocabulary would help polish decide where + sentences break around a proper noun it now knows is a name. Its + contract does not move: clean sentences out of an imperfect + transcription, same tone, same words. +- Re-transcription when the vocabulary changes. A new family member + does not improve recordings already decoded, and re-running whisper + over years of audio to pick up one name is the wrong default. An + explicit `--retranscribe` would make it a choice. From 32b55dd7ffaa9c6eaffa2273d35eed61d8b8573b Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 10:44:15 +0200 Subject: [PATCH 14/43] refactor(memory): name the diary's arguments the way the CLI already does Two flags invented where a convention existed. A room is a positional everywhere else in this CLI: `stack messages read `, `join `, `send `. So it is one here too, and `stack memory diary --room letters` becomes `stack memory diary letters`. Skipping a cache to redo work is `--force`, as in `stack photos import --force`. `--rebuild` appeared nowhere that ships. --- stacklets/memory/bot/cli/diary.py | 32 ++++++++++++++++++++++---- stacklets/memory/bot/cli_entrypoint.py | 5 ++-- stacklets/memory/cli/diary.py | 4 ++-- 3 files changed, 32 insertions(+), 9 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 4360ec1..d14b605 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -15,9 +15,9 @@ stack memory diary compile and publish stack memory diary --dry-run print the pages, write nothing - stack memory diary --room memories a different room + stack memory diary letters a different room stack memory diary --burst-window 1 see below - stack memory diary --rebuild re-read everything from scratch + stack memory diary --force re-read everything from scratch WHY THE BURST WINDOW IS A KNOB Messages that synced late carry arrival timestamps, not recording @@ -551,6 +551,11 @@ def _household_zone(): return timezone.utc +# Flags that consume the token after them, so the room can be picked out +# of the rest without mistaking a flag's value for it. +_TAKES_A_VALUE = ("--burst-window",) + + def _opt(argv: list[str], flag: str, fallback: str) -> str: for i, arg in enumerate(argv): if arg == flag and i + 1 < len(argv): @@ -558,10 +563,29 @@ def _opt(argv: list[str], flag: str, fallback: str) -> str: return fallback +def _positional(argv: list[str], fallback: str) -> str: + """The room, named the way every other command names one. + + `stack messages read `, `join `, `send `: a room + is a positional everywhere in this CLI, so it is one here too. + """ + skip = False + for arg in argv: + if skip: + skip = False + continue + if arg in _TAKES_A_VALUE: + skip = True + continue + if not arg.startswith("-"): + return arg + return fallback + + async def run(llm, argv: list[str]) -> int: - room_arg = _opt(argv, "--room", "memories") + room_arg = _positional(argv, "memories") dry_run = "--dry-run" in argv - rebuild = "--rebuild" in argv + rebuild = "--force" in argv try: window = float(_opt(argv, "--burst-window", str(diary.DEFAULT_BURST_WINDOW_S))) diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index e9e4653..1e38aec 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -17,8 +17,7 @@ words. Exit 1 means no keywords, which the host treats as "search it literally" rather than as a failure. - diary [--room ] [--burst-window ] [--dry-run] - [--rebuild] + diary [] [--burst-window ] [--dry-run] [--force] Compile the memories room into the family diary. Walks the room's full history, transcribes every recording, recovers the date each one was made, and publishes month pages under the @@ -28,7 +27,7 @@ arriving tonight can belong to an entry from years back. It stays cheap because transcripts, readings, and month summaries are all kept against the thing they describe, so only what is - new costs anything. `--rebuild` ignores those and reads + new costs anything. `--force` ignores those and reads everything again, for when the model has improved. wiki [--home] [--member ]... [--topic ]... [--dry-run] diff --git a/stacklets/memory/cli/diary.py b/stacklets/memory/cli/diary.py index c8b3e94..1710a8f 100644 --- a/stacklets/memory/cli/diary.py +++ b/stacklets/memory/cli/diary.py @@ -7,9 +7,9 @@ stack memory diary compile and publish stack memory diary --dry-run print the pages, write nothing - stack memory diary --room memories read a different room + stack memory diary letters read a different room stack memory diary --burst-window 1 tighten sync-burst detection - stack memory diary --rebuild read it all again from scratch + stack memory diary --force read it all again from scratch WHEN IT RUNS The curator compiles the diary on its nightly sweep, so new From 0ed3370fe11189a25fbae46a0769f91eda78d260 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 11:04:05 +0200 Subject: [PATCH 15/43] fix(tools): only replay the Simpsons corpus onto a Simpsons instance The replay refused one hardcoded hostname, which put a real family's domain in a public repo and protected only that one household. Anyone else pointing it at their own server got no warning. The instance's own configuration is the honest check. Everything this sends is an invented Simpsons memory, so it runs when stack.toml says the household is the Simpsons and refuses otherwise, including when there is no stack.toml to read. --- tools/family-memories/README.md | 2 +- tools/family-memories/ingest.py | 49 ++++++++++++++++++++++++--------- 2 files changed, 37 insertions(+), 14 deletions(-) diff --git a/tools/family-memories/README.md b/tools/family-memories/README.md index 6f83539..0e689c2 100644 --- a/tools/family-memories/README.md +++ b/tools/family-memories/README.md @@ -39,7 +39,7 @@ attachment against `out/manifest.json`. # render everything into out/ (needs the ai stacklet's speech service) python tools/family-memories/generate.py # both locales; --locale de/en -# replay into a TEST RIG (never production — the script refuses merles.eu) +# replay into a TEST RIG (only runs on a Simpsons instance) python tools/family-memories/ingest.py \ --homeserver http://:42031 \ --room '#memories:' \ diff --git a/tools/family-memories/ingest.py b/tools/family-memories/ingest.py index eda5909..ba54959 100644 --- a/tools/family-memories/ingest.py +++ b/tools/family-memories/ingest.py @@ -13,9 +13,11 @@ --room '#memories:testrig.local' \ --login marge:PASSWORD --login homer:PASSWORD -SAFETY: this tool refuses to run against the production homeserver -(merles.eu). There is no default homeserver on purpose. --force-i-know -overrides the guard and should never be needed. +SAFETY: this writes invented Simpsons memories into a room, so it only +runs on an instance configured as the Simpsons. Any other household is +somebody's real family, and this is the last place fabricated memories +belong. There is no default homeserver on purpose, and --force-i-know +overrides the check. """ from __future__ import annotations @@ -24,13 +26,35 @@ import json import sys import time +import tomllib import urllib.parse import urllib.request from pathlib import Path HERE = Path(__file__).resolve().parent OUT = HERE / "out" -PRODUCTION_MARKERS = ("merles.eu",) +REPO_ROOT = HERE.parents[1] + +# The household this corpus is about. Everything it sends is a Simpsons +# memory, so the instance being the Simpsons is the real precondition: +# on any other household these are fabricated memories in a room meant +# for the family's own. +DEMO_HOUSEHOLD = "simpson" + + +def configured_household(root: Path) -> str: + """The family name this checkout is installed for, or "" if unknown. + + Unknown fails the check. An instance with no `stack.toml` has not + been set up, and guessing in its favour is the wrong way to be + wrong about where invented memories get written. + """ + try: + with open(root / "stack.toml", "rb") as f: + core = tomllib.load(f).get("core") or {} + except (OSError, tomllib.TOMLDecodeError): + return "" + return str(core.get("stack_owner") or "") class Client: @@ -140,10 +164,14 @@ def main(): global OUT OUT = OUT / args.locale - if not args.force_i_know and any( - m in args.homeserver or m in args.room for m in PRODUCTION_MARKERS): - sys.exit("REFUSING: target looks like the production instance. " - "This corpus is for test rigs only.") + household = configured_household(REPO_ROOT) + if not args.force_i_know and \ + not household.lower().startswith(DEMO_HOUSEHOLD): + sys.exit( + f"REFUSING: this instance is set up for " + f"{household or 'no household'}, not the Simpsons. Everything " + "below is an invented Simpsons memory and does not belong in " + "another family's room.") manifest = json.loads((OUT / "manifest.json").read_text()) c = Client(args.homeserver) @@ -151,11 +179,6 @@ def main(): for spec in args.login: user, _, pw = spec.partition(":") tokens[user], user_ids[user] = c.login(user, pw) - if not args.force_i_know and any( - uid.endswith(m) for uid in user_ids.values() - for m in PRODUCTION_MARKERS): - sys.exit("REFUSING: logged-in server is production (merles.eu).") - missing = {i["sender"] for i in manifest} - set(tokens) if missing: sys.exit(f"no --login for sender(s): {', '.join(sorted(missing))}") From 603fb84db0f8e15b5401972931b396a087b0c604 Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 21:13:40 +0200 Subject: [PATCH 16/43] docs(ai): describe the decoder hint without a caller's domain in it The transcription client is generic; its docstring described the parameter through one caller's data, down to a family member's name. Now it says what the parameter does: primes the decoder with terms the audio is likely to contain, biases towards them, has a small window that drops overflow, and is the only place a misheard word can be corrected because polish may not alter the word sequence. --- lib/stack/ai/client.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index 65920dd..b1a0946 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -433,14 +433,14 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", the SDK for OpenAI-compat servers that route by model name; the native whisper-server ignores it. - ``vocabulary`` is a hint about words this household says: the - names of the people in it, the topics they keep. Whisper decodes - against it, so a family name it would otherwise hear as a common - word comes back right the first time. That matters more than it - sounds: a memo opening "Bart, today is..." transcribes as "Part" - or loses the name entirely, and the polish pass cannot repair it - without rewriting what was said, which it is forbidden to do. - Fixing the input is the only way to fix the words. + ``vocabulary`` primes the decoder with terms the audio is likely + to contain -- proper nouns, names, jargon. Whisper reads it as + speech preceding the clip and biases towards it, so a word it + would otherwise render as a commoner homophone comes back right. + The window is a couple of hundred tokens and overflow is dropped, + so put what matters first. Priming is the only place a misheard + word can be corrected: :meth:`polish` may not alter the word + sequence and verifies that it did not. ``cleanup_with`` is an optional :class:`LLM` to polish the raw STT output with punctuation and sentence breaks. When provided, the From fc464c024a4804070b1103a9d5ba62c09f6ba8cb Mon Sep 17 00:00:00 2001 From: Arthur Date: Sun, 13 Sep 2026 21:19:23 +0200 Subject: [PATCH 17/43] docs(core): describe TIMEZONE by what it does Core config is read by every stacklet, and this comment explained a timezone setting through one of them: whose clock it is, and a voice memo recorded after midnight. It now says what the value is and the mistake it prevents. --- stacklets/core/stacklet.toml | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/stacklets/core/stacklet.toml b/stacklets/core/stacklet.toml index ddace63..244d3fb 100644 --- a/stacklets/core/stacklet.toml +++ b/stacklets/core/stacklet.toml @@ -110,10 +110,9 @@ IMMICH_API_KEY = "{photos__API_KEY}" # Tools server port — Open WebUI connects here for tool calling TOOLS_PORT = "42000" -# Household timezone — the clock the family keeps. Anything that turns a -# timestamp into a calendar day needs it: a voice memo recorded at half -# past midnight belongs to the day the family would say it happened on, -# not to whatever day it was in UTC. +# Local timezone, as an IANA name. Anything that turns a timestamp into +# a calendar day needs it: read as UTC, something recorded shortly after +# midnight is filed under the previous day. TIMEZONE = "{timezone}" # Household language — drives bot-side i18n, ontology rendering, and From 7b44ceb4b1dc776848af207518bc5e37f57cf16a Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 09:48:18 +0200 Subject: [PATCH 18/43] fix(memory): report progress and handle SIGINT during long compiles A first compile of a populated room spends most of its runtime in whisper and printed nothing until it finished. Ctrl+C produced an asyncio traceback and discarded that run's readings. - Progress logging per phase: pagination reports a running event count; transcription reports the recording count and how many need decoding, then logs each one with its duration before the whisper call; reading logs each slice; summarising logs each month. Cached items are not logged. - Reading and summary caches are written incrementally instead of once at the end, so an interrupted run resumes from the last completed slice. - Dry runs write those caches too. They are cost caches keyed to immutable input, not output. - SIGINT exits 130 with a single line, in both the container entrypoint and the host dispatch wrapper. Progress goes to stderr; stdout still carries only the rendered pages. --- stacklets/memory/bot/cli/diary.py | 52 +++++++++++++++++++++++--- stacklets/memory/bot/cli_entrypoint.py | 8 +++- stacklets/memory/cli/_common.py | 4 ++ 3 files changed, 57 insertions(+), 7 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index d14b605..268f5bd 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -14,7 +14,7 @@ `TRANSCRIPT_DIR`, keyed by event id, shared with the bots. stack memory diary compile and publish - stack memory diary --dry-run print the pages, write nothing + stack memory diary --dry-run print the pages, publish nothing stack memory diary letters a different room stack memory diary --burst-window 1 see below stack memory diary --force re-read everything from scratch @@ -27,6 +27,10 @@ and needs a window well under a second. There is no single value that fits both, so the caller picks. +A dry run publishes no pages but still writes the transcript, reading +and summary caches. These are cost caches, not output: discarding them +would make a preview run as expensive as the compile that follows. + Runs inside `stack-core-bot-runner`: it has the whisper client, the LLM client, the brain working copy, and the Matrix admin credentials. The host-side wrapper is a thin docker-exec, like `stack memory wiki`. @@ -116,6 +120,11 @@ async def _history(session, homeserver, token, room_id: str) -> list[dict]: Paginates backwards until Synapse stops handing back a cursor. The caller sorts; order here is only what the API gives us. + + Counts out loud as it goes. A room with years in it takes many + round trips before anything else can start, and a command that + prints nothing for that long is indistinguishable from one that has + hung. """ events: list[dict] = [] cursor = "" @@ -131,6 +140,8 @@ async def _history(session, homeserver, token, room_id: str) -> list[dict]: payload = resp.json() chunk = payload.get("chunk") or [] events.extend(chunk) + if chunk: + _err(f" read {len(events)} events so far") cursor = payload.get("end") or "" if not chunk or not cursor: return events @@ -156,6 +167,14 @@ async def _download(session, homeserver, token, mxc: str) -> bytes | None: _VOCAB_BUDGET = 600 +def _length(ms: int | None) -> str: + """Duration as m:ss, for the progress line.""" + if not ms: + return "length unknown" + total = round(ms / 1000) + return f"{total // 60}:{total % 60:02d}" + + def _household_vocabulary() -> str: """The names and subjects this family uses, for whisper to decode against. @@ -387,11 +406,13 @@ def remember(event_id: str, row: dict) -> None: _err(f" {len(known)} message(s) already read, " f"{len(messages) - len(known)} new") - for chunk in _chunks(messages): + slices = _chunks(messages) + for n, chunk in enumerate(slices, start=1): # A slice whose every message is on file has nothing left to # say: its links were recorded with the messages they join. if all(m.event_id in known for m in chunk): continue + _err(f" reading slice {n} of {len(slices)}") prompt = _READ_PROMPT.format(messages=_as_prompt(chunk)) try: @@ -423,6 +444,11 @@ def remember(event_id: str, row: dict) -> None: if cache is not None: cache.put(here.event_id, found, here.body) + # Saved per slice rather than once at the end, so an + # interrupted run resumes from the last completed slice. + if cache is not None: + cache.save() + return readings, continues, refers_to @@ -631,11 +657,24 @@ async def run(llm, argv: list[str]) -> int: if vocabulary: _err(f" decoding against: {vocabulary[:90]}...") - decoded = [] + # Transcription dominates runtime, so each recording is logged + # before the whisper call rather than after. Cached transcripts + # return immediately and are not logged. + recordings = [m for m in messages if m.kind == "voice"] + pending = sum(1 for m in recordings + if voice.TRANSCRIPTS.read(m.event_id) is None) + if recordings: + _err(f" {len(recordings)} recording(s), {pending} to decode") + + decoded, heard = [], 0 for msg in messages: if msg.kind != "voice": decoded.append(msg) continue + heard += 1 + if voice.TRANSCRIPTS.read(msg.event_id) is None: + _err(f" [{heard}/{len(recordings)}] decoding {msg.sender}, " + f"{_length(msg.duration_ms)}") text = await _transcribe( msg, session=session, homeserver=homeserver, token=token, transcriber=transcriber, llm=llm, vocabulary=vocabulary) @@ -666,13 +705,14 @@ async def run(llm, argv: list[str]) -> int: if kept: summaries[key] = kept continue + _err(f" summarising {in_month[0].on.strftime('%B %Y')}") summaries[key] = await _summarise(in_month, llm) if summaries[key]: summaries_cache.put(key, digest, summaries[key]) + summaries_cache.save() - if not dry_run: - readings_cache.save() - summaries_cache.save() + readings_cache.save() + summaries_cache.save() pages = diary.pages_for(entries, room_id=room_id, summaries=summaries) if dry_run: diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index 1e38aec..460cd18 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -93,4 +93,10 @@ def _usage() -> None: if __name__ == "__main__": - sys.exit(asyncio.run(main(sys.argv[1:]))) + try: + sys.exit(asyncio.run(main(sys.argv[1:]))) + except KeyboardInterrupt: + # Exit 130 (128 + SIGINT) without the asyncio traceback. + # Completed transcripts and readings are already on disk. + _err("\ninterrupted; completed work is cached") + sys.exit(130) diff --git a/stacklets/memory/cli/_common.py b/stacklets/memory/cli/_common.py index 8eee49b..1e22748 100644 --- a/stacklets/memory/cli/_common.py +++ b/stacklets/memory/cli/_common.py @@ -53,6 +53,10 @@ def dispatch(command: str, *argv: str) -> dict: rc = subprocess.call(cmd) except FileNotFoundError: return {"error": "docker CLI not found on this host"} + except KeyboardInterrupt: + # The exec'd process received the same SIGINT and reported it. + # Exit 130 without adding a host-side traceback. + sys.exit(130) # Pass rc through to the shell without letting the harness print a # generic "command failed (exit N)" on top of the container's own From 9285692a4b50d0a5bf12448355fc0faef850cde2 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 11:32:35 +0200 Subject: [PATCH 19/43] fix(ai): cap generation so a repetition loop cannot occupy the endpoint A transcript cleanup call ran for 25 minutes and produced about 100k tokens from a 6-minute recording, then had to be killed by hand. The call had no token cap, so the model generated until the client timeout. That timeout is 900s with one retry, which allows 30 minutes on a host the endpoint shares with everything else. - complete() accepts max_tokens and timeout per call. - polish() caps output at twice the estimated input, minimum 256 tokens, and uses a 180s timeout. A 6-minute transcript now stops at roughly 2,500 tokens instead of 100k. A truncated reply fails the existing word check, so the caller gets the raw transcript. - The diary's reading and summary calls are capped the same way. - complete() logs a warning when a reply stops on length rather than on a stop token. - polish() logs the character offset where the words first diverge, with context from both sides. Previously it logged lengths only, which did not identify the input that caused the loop. --- lib/stack/ai/client.py | 72 +++++++++++++++++++++++++++++-- stacklets/memory/bot/cli/diary.py | 20 +++++++-- tests/framework/test_ai_client.py | 49 +++++++++++++++++++++ 3 files changed, 135 insertions(+), 6 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index b1a0946..1bd54db 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -221,7 +221,9 @@ async def complete(self, role: str, prompt: str, *, images: "list | None" = None, json_mode: bool = False, model_override: str | None = None, - temperature: float | None = None) -> str: + temperature: float | None = None, + max_tokens: int | None = None, + timeout: float | None = None) -> str: """Run a single chat completion and return the response text. ``role`` resolves to a concrete model via `resolve_model`. Pass @@ -232,6 +234,14 @@ async def complete(self, role: str, prompt: str, *, evidence to yield the same page); None keeps the server default. SDK errors are translated to the typed LLM errors above. + + ``max_tokens`` caps the generation. Set it whenever the size of + a valid answer is known in advance. Without a cap, a model that + enters a repetition loop generates until the client timeout, + which on a local endpoint means the host is busy for that whole + time. ``timeout`` overrides the client default for one call and + should accompany a cap that is much smaller than the default + budget allows for. """ model = model_override or resolve_model(self._full_role(role)) content = prompt if not images else _content_parts(prompt, images) @@ -241,6 +251,10 @@ async def complete(self, role: str, prompt: str, *, kwargs["response_format"] = {"type": "json_object"} if temperature is not None: kwargs["temperature"] = temperature + if max_tokens is not None: + kwargs["max_tokens"] = max_tokens + if timeout is not None: + kwargs["timeout"] = timeout try: resp = await self._client.chat.completions.create( @@ -262,7 +276,18 @@ async def complete(self, role: str, prompt: str, *, except openai.APIStatusError as e: raise LLMUnavailableError(f"HTTP {e.status_code}: {str(e)[:200]}") from e - return resp.choices[0].message.content or "" + # A capped call that stops on length rather than on a stop token + # did not finish its answer. Every caller treats a truncated + # reply as a failure of some kind, and without this line the only + # evidence is a short answer that looks deliberate. + choice = resp.choices[0] + if getattr(choice, "finish_reason", None) == "length": + logger.warning( + "[llm] {} hit the {}-token cap and was cut off; " + "the answer is incomplete", + model, kwargs.get("max_tokens", "default"), + ) + return choice.message.content or "" async def has_vision(self, *, role: str = "classifier", model_override: str | None = None) -> bool: @@ -360,6 +385,19 @@ def _content_parts(prompt: str, images: list) -> list[dict]: # these constraints reliably when the rules are imperative and the # input is the last thing in the prompt. _CLEANUP_ROLE = "transcript_cleanup" + +# Polish may only add punctuation, so its output is the input plus a few +# tokens. Two times the estimated input size is generous headroom and +# still stops a repetition loop early. Without a cap the call runs to +# the client timeout, which on a local endpoint occupies the host for +# that whole period. +_CLEANUP_TOKEN_RATIO = 2.0 +_CLEANUP_TOKEN_FLOOR = 256 + +# Generation is slower than prefill, so a cap of N tokens still takes +# time. This bounds one call to minutes rather than the client default, +# which is sized for reading long documents. +_CLEANUP_TIMEOUT_S = 180.0 _CLEANUP_PROMPT = """\ You are cleaning up a raw speech-to-text transcript that has no \ punctuation, capitalization, or sentence breaks. Restore them in the \ @@ -483,6 +521,21 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", return raw return await self.polish(raw, cleanup_with) + @staticmethod + def _divergence(expected: str, actual: str, window: int = 60) -> str: + """Where two word sequences first differ, with context. + + A length comparison says a polish went wrong; this says where. A + model that loops repeats a phrase from the point it lost track, + and that point is what identifies the input that triggered it. + """ + limit = min(len(expected), len(actual)) + i = 0 + while i < limit and expected[i] == actual[i]: + i += 1 + return (f"char {i}: expected ...{expected[i:i + window]!r}, " + f"got ...{actual[i:i + window]!r}") + @staticmethod def _comparable(text: str) -> str: """`text` reduced to what the polish is not allowed to change. @@ -522,7 +575,17 @@ async def polish(raw: str, llm: "LLM") -> str: still gets a usable transcript. We log a warning so the admin can see drift between raw and polished if they want to investigate model quality. + + The call is capped in output tokens and in time. A repetition + loop therefore ends in a truncated answer, which fails the word + check below and yields the raw transcript, instead of occupying + the endpoint until the client timeout. """ + # ~4 characters per token is the usual estimate for this family + # of models. The exact figure does not matter; the cap only has + # to be above any correct answer and far below a runaway one. + budget = max(_CLEANUP_TOKEN_FLOOR, + int(len(raw) / 4 * _CLEANUP_TOKEN_RATIO)) try: cleaned = await llm.complete( _CLEANUP_ROLE, _CLEANUP_PROMPT.format(raw=raw), @@ -531,6 +594,8 @@ async def polish(raw: str, llm: "LLM") -> str: # invites the rephrasing the word check below rejects, # which turns a paid model call into a raw transcript. temperature=0.0, + max_tokens=budget, + timeout=_CLEANUP_TIMEOUT_S, ) except LLMError as e: logger.warning("[transcriber] cleanup failed, returning raw: {}", e) @@ -551,8 +616,9 @@ async def polish(raw: str, llm: "LLM") -> str: if raw_cmp != clean_cmp: logger.warning( "[transcriber] polish changed the words, keeping raw " - "(raw={} chars, polished={} chars)", + "(raw={} chars, polished={} chars); diverges at {}", len(raw_cmp), len(clean_cmp), + Transcriber._divergence(raw_cmp, clean_cmp), ) return raw return cleaned diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 268f5bd..f86a62e 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -271,6 +271,16 @@ async def produce() -> dict: _CHUNK = 40 _OVERLAP = 4 +# One small JSON object per message in the slice, plus the enclosing +# structure. A model that loops instead of closing the array is capped +# here rather than at the client timeout. +_READ_TOKENS_PER_MESSAGE = 120 +_READ_TIMEOUT_S = 300.0 + +# Two to four sentences. +_SUMMARY_TOKENS = 600 +_SUMMARY_TIMEOUT_S = 180.0 + _READ_PROMPT = """\ You are reading a family's private memories room so their diary can be @@ -416,8 +426,10 @@ def remember(event_id: str, row: dict) -> None: prompt = _READ_PROMPT.format(messages=_as_prompt(chunk)) try: - raw = await llm.complete("classifier", prompt, - json_mode=True, temperature=0) + raw = await llm.complete( + "classifier", prompt, json_mode=True, temperature=0, + max_tokens=len(chunk) * _READ_TOKENS_PER_MESSAGE + 256, + timeout=_READ_TIMEOUT_S) payload = json.loads(raw) rows = payload.get("messages") if isinstance(payload, dict) else None except (LLMError, json.JSONDecodeError, TypeError) as e: @@ -542,7 +554,9 @@ async def _summarise(entries, llm) -> str: month = entries[0].on.strftime("%B %Y") prompt = _SUMMARY_PROMPT.format(month=month, evidence=_evidence(entries)) try: - text = await llm.complete("writer", prompt, temperature=0) + text = await llm.complete("writer", prompt, temperature=0, + max_tokens=_SUMMARY_TOKENS, + timeout=_SUMMARY_TIMEOUT_S) except LLMError as e: _err(f" could not summarise {month}: {e}") return "" diff --git a/tests/framework/test_ai_client.py b/tests/framework/test_ai_client.py index 42bcf24..6d59365 100644 --- a/tests/framework/test_ai_client.py +++ b/tests/framework/test_ai_client.py @@ -736,6 +736,55 @@ async def test_whitespace_and_line_breaks_are_accepted( await tr.aclose() +class TestPolishIsBounded: + """A polish may only add punctuation, so its output size is known in + advance. Without a cap, a model that repeats itself generates until + the client timeout, which on a local endpoint occupies the host for + that whole period.""" + + async def test_the_cap_scales_with_the_transcript(self, httpserver: HTTPServer): + long_transcript = "book the campsite " * 500 + httpserver.expect_request( + "/v1/audio/transcriptions", method="POST", + ).respond_with_json({"text": long_transcript}) + llm = _StubLLM(result=long_transcript) + tr = _make_transcriber(httpserver) + + await tr.transcribe(b"a", cleanup_with=llm) + + cap = llm.kwargs[0]["max_tokens"] + assert cap < len(long_transcript) + assert cap > len(long_transcript) / 4 + await tr.aclose() + + async def test_a_short_transcript_still_gets_room_to_work( + self, httpserver: HTTPServer): + httpserver.expect_request( + "/v1/audio/transcriptions", method="POST", + ).respond_with_json({"text": "hi"}) + llm = _StubLLM(result="Hi.") + tr = _make_transcriber(httpserver) + + await tr.transcribe(b"a", cleanup_with=llm) + + assert llm.kwargs[0]["max_tokens"] >= 256 + await tr.aclose() + + async def test_the_call_does_not_use_the_document_reading_budget( + self, httpserver: HTTPServer): + """The client default is sized for reading long documents.""" + httpserver.expect_request( + "/v1/audio/transcriptions", method="POST", + ).respond_with_json({"text": "book the campsite"}) + llm = _StubLLM(result="Book the campsite.") + tr = _make_transcriber(httpserver) + + await tr.transcribe(b"a", cleanup_with=llm) + + assert llm.kwargs[0]["timeout"] <= 300 + await tr.aclose() + + class TestPolishIsDeterministic: """Polishing is a transformation, not a generation: the same words in should give the same punctuation out. Sampling only invites the model From ff58659a79e4a0b80a8aa97038d35e62ebc92bca Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 11:37:44 +0200 Subject: [PATCH 20/43] fix(ai): cap the polish call at the transcript size plus 10% The previous cap was twice the estimated input. Measured over 15 real transcripts, polish changes the character count by between -1.5% and +1.0%, mean 0.994. The output is the same size as the input, so 10% headroom covers it and 100% does not buy anything. The loose part is the character-to-token estimate, not the ratio. It now assumes three characters per token rather than four. English BPE averages about four, but German compounds and accented characters tokenize denser, and a cap below a correct answer truncates every polish in that language and returns the raw transcript instead. polish() takes an optional max_tokens to override the derived cap. cleanup_budget() is public so a caller can read the default first. A 6-minute transcript is now capped at about 1,800 tokens. --- lib/stack/ai/client.py | 40 +++++++++++++++++++++---------- tests/framework/test_ai_client.py | 23 +++++++++++++++--- 2 files changed, 47 insertions(+), 16 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index 1bd54db..86a08d5 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -386,12 +386,17 @@ def _content_parts(prompt: str, images: list) -> list[dict]: # input is the last thing in the prompt. _CLEANUP_ROLE = "transcript_cleanup" -# Polish may only add punctuation, so its output is the input plus a few -# tokens. Two times the estimated input size is generous headroom and -# still stops a repetition loop early. Without a cap the call runs to -# the client timeout, which on a local endpoint occupies the host for -# that whole period. -_CLEANUP_TOKEN_RATIO = 2.0 +# Polish may only add punctuation, so its output is the same size as its +# input. Measured over real transcripts the character count lands +# between 0.98 and 1.01 of the raw text, so 10% headroom covers it. +# +# The estimate is the loose part, not the ratio. Three characters per +# token is deliberately pessimistic: English BPE averages about four, +# but German compounds and accented characters tokenize denser, and a +# cap set below a correct answer truncates every polish in that language +# and silently returns the raw transcript instead. +_CLEANUP_CHARS_PER_TOKEN = 3 +_CLEANUP_HEADROOM = 1.1 _CLEANUP_TOKEN_FLOOR = 256 # Generation is slower than prefill, so a cap of N tokens still takes @@ -521,6 +526,16 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", return raw return await self.polish(raw, cleanup_with) + @staticmethod + def cleanup_budget(raw: str) -> int: + """Token cap for polishing `raw`: its own size plus 10%. + + Public so a caller can see the default before deciding to + override it. + """ + estimated = len(raw) / _CLEANUP_CHARS_PER_TOKEN + return max(_CLEANUP_TOKEN_FLOOR, int(estimated * _CLEANUP_HEADROOM)) + @staticmethod def _divergence(expected: str, actual: str, window: int = 60) -> str: """Where two word sequences first differ, with context. @@ -557,7 +572,8 @@ def _comparable(text: str) -> str: return re.sub(r"[^a-z0-9]+", "", text.lower()) @staticmethod - async def polish(raw: str, llm: "LLM") -> str: + async def polish(raw: str, llm: "LLM", *, + max_tokens: int | None = None) -> str: """Restore punctuation and sentence breaks in a raw transcript. whisper.cpp emits one unbroken lowercase run of words. This pass @@ -579,13 +595,11 @@ async def polish(raw: str, llm: "LLM") -> str: The call is capped in output tokens and in time. A repetition loop therefore ends in a truncated answer, which fails the word check below and yields the raw transcript, instead of occupying - the endpoint until the client timeout. + the endpoint until the client timeout. ``max_tokens`` overrides + the derived cap for a caller that knows better; the derived one + is the input size plus 10%. """ - # ~4 characters per token is the usual estimate for this family - # of models. The exact figure does not matter; the cap only has - # to be above any correct answer and far below a runaway one. - budget = max(_CLEANUP_TOKEN_FLOOR, - int(len(raw) / 4 * _CLEANUP_TOKEN_RATIO)) + budget = max_tokens or Transcriber.cleanup_budget(raw) try: cleaned = await llm.complete( _CLEANUP_ROLE, _CLEANUP_PROMPT.format(raw=raw), diff --git a/tests/framework/test_ai_client.py b/tests/framework/test_ai_client.py index 6d59365..e23a3b9 100644 --- a/tests/framework/test_ai_client.py +++ b/tests/framework/test_ai_client.py @@ -752,11 +752,28 @@ async def test_the_cap_scales_with_the_transcript(self, httpserver: HTTPServer): await tr.transcribe(b"a", cleanup_with=llm) - cap = llm.kwargs[0]["max_tokens"] - assert cap < len(long_transcript) - assert cap > len(long_transcript) / 4 + # `transcribe` strips the transcript before polishing it. + assert llm.kwargs[0]["max_tokens"] == \ + Transcriber.cleanup_budget(long_transcript.strip()) await tr.aclose() + async def test_the_cap_allows_the_transcript_back_plus_a_margin(self): + """Polish restores punctuation and nothing else, so the answer is + the same size as the input. Measured growth is under 2%.""" + raw = "book the campsite " * 500 + + cap = Transcriber.cleanup_budget(raw) + + # Denser than any real tokenizer, so a correct answer always fits. + assert cap >= len(raw) / 3 + # Far below a run that repeats itself. + assert cap < len(raw) / 2 + + def test_a_caller_may_set_its_own_cap(self): + """`polish` takes an override for a caller that knows better.""" + import inspect + assert "max_tokens" in inspect.signature(Transcriber.polish).parameters + async def test_a_short_transcript_still_gets_room_to_work( self, httpserver: HTTPServer): httpserver.expect_request( From 03000093b6989bff040a61db6601c78b5ea89823 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 14:04:02 +0200 Subject: [PATCH 21/43] fix(ai): detect hallucinated whisper transcripts and harden whisper config Whisper hallucinates on speechless audio: repetition loops (measured 5-147x phrase repeats vs 1-2x in real speech) or CJK output from failed language detection. One such loop caused the 100k-token runaway generation on 2026-09-14. - Add Transcriber.looks_degenerate(): flags transcripts with >=3 repeats of a 6-gram or >30% CJK letters. - polish() skips degenerate input; never feeds a loop to the LLM. - Diary files degenerate recordings without text (entry + audio link remain; raw kept in the transcript record). - Whisper plist: pin --language from [core].language, add --max-context 0 and --suppress-nst. Only the combination fixed all three incident recordings. - on_start reconciles the plist by content, so existing installs pick up config changes on restart instead of requiring 'stack setup ai'. --- lib/stack/ai/client.py | 51 +++++++++++++++++++++++++++++++ stacklets/ai/hooks/on_install.py | 43 +++++++++++++++++++++++--- stacklets/ai/hooks/on_start.py | 39 +++++++++++++++++++++++ stacklets/memory/bot/cli/diary.py | 15 ++++++++- tests/framework/test_ai_client.py | 42 ++++++++++++++++++++++++- 5 files changed, 183 insertions(+), 7 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index 86a08d5..6c8575d 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -399,6 +399,13 @@ def _content_parts(prompt: str, images: list) -> list[dict]: _CLEANUP_HEADROOM = 1.1 _CLEANUP_TOKEN_FLOOR = 256 +# Hallucination signatures, thresholds measured on the recordings that +# triggered the September 2026 runaway (see looks_degenerate). Real +# loops repeated a phrase 5-147x; legitimately repetitive speech never +# exceeded 2x. Three keeps a child singing the same line twice. +_DEGENERATE_REPEATS = 3 +_DEGENERATE_CJK_FRACTION = 0.3 + # Generation is slower than prefill, so a cap of N tokens still takes # time. This bounds one call to minutes rather than the client default, # which is sized for reading long documents. @@ -536,6 +543,40 @@ def cleanup_budget(raw: str) -> int: estimated = len(raw) / _CLEANUP_CHARS_PER_TOKEN return max(_CLEANUP_TOKEN_FLOOR, int(estimated * _CLEANUP_HEADROOM)) + @staticmethod + def looks_degenerate(text: str) -> str | None: + """Why `text` looks hallucinated rather than heard, or None. + + Whisper fails loudly in two measured ways when a recording has + no usable speech. It loops: over the recordings behind the + September runaway, one six-word phrase covered the transcript 5 + to 147 times, while real speech — including a child repeating a + song line — stayed at 1 to 2. Or its language detection + free-falls and lands in CJK: famstack households write German or + English, so a transcript that is mostly CJK letters was never + heard, it was invented. + + Public because two different decisions hang on it: `polish` + refuses to feed such text to a model (a loop is an unbounded + echo task), and the diary refuses to publish it as words a + person said. + """ + words = text.split() + if len(words) >= 12: + counts: dict[tuple, int] = {} + for i in range(len(words) - 6): + gram = tuple(words[i:i + 6]) + counts[gram] = counts.get(gram, 0) + 1 + top = max(counts.values(), default=0) + if top >= _DEGENERATE_REPEATS: + return f"one phrase repeats {top}x" + letters = [c for c in text if c.isalpha()] + cjk = sum(1 for c in letters + if "぀" <= c <= "鿿" or "가" <= c <= "힯") + if letters and cjk / len(letters) > _DEGENERATE_CJK_FRACTION: + return "mostly CJK letters" + return None + @staticmethod def _divergence(expected: str, actual: str, window: int = 60) -> str: """Where two word sequences first differ, with context. @@ -599,6 +640,16 @@ async def polish(raw: str, llm: "LLM", *, the derived cap for a caller that knows better; the derived one is the input size plus 10%. """ + # A degenerate transcript never reaches the model. Polish is an + # echo task, and echoing a loop is how the September runaway + # happened — the token cap below bounds the damage, this removes + # the exposure. The caller gets the raw text back and decides + # what a transcript that was never really heard is worth. + if (reason := Transcriber.looks_degenerate(raw)) is not None: + logger.warning( + "[transcriber] transcript looks hallucinated ({}), " + "not polishing", reason) + return raw budget = max_tokens or Transcriber.cleanup_budget(raw) try: cleaned = await llm.complete( diff --git a/stacklets/ai/hooks/on_install.py b/stacklets/ai/hooks/on_install.py index ad63a66..5268ca9 100644 --- a/stacklets/ai/hooks/on_install.py +++ b/stacklets/ai/hooks/on_install.py @@ -235,22 +235,36 @@ def _install_whisper(ctx, data_dir: Path, state_dir: Path): def _setup_whisper_launchd(ctx, data_dir: Path, whisper_bin: Path, model_path: Path, state_dir: Path): - section("Whisper server", "launchd service") + """Write the wrapper + LaunchAgent and (re)load the service. + Idempotent by content: when what is on disk already matches, nothing + is written and the running service is not bounced. on_start calls + this on every `stack up ai`, which is how a flag change here reaches + existing installs on a plain restart instead of waiting for a + `stack setup ai` nobody thinks to run. + """ agents_dir = Path.home() / "Library" / "LaunchAgents" agents_dir.mkdir(parents=True, exist_ok=True) # Wrapper script — launchd doesn't inherit PATH, ffmpeg needs it wrapper = data_dir / "ai" / "famstack-whisper" - wrapper.write_text( + wrapper_text = ( f"#!/bin/bash\n" f'export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin"\n' f'exec "{whisper_bin}" "$@"\n' ) - wrapper.chmod(0o755) + + # Whisper hears what the household speaks: [core].language first, + # with the ai-stacklet language (primarily a TTS-voice knob) as the + # fallback. Pinning beats auto-detection, which free-falls on + # silence and noise — but pinning the wrong language would be worse + # than auto, so this must never read a voice preference as a + # transcription language when the household says otherwise. + language = ctx.stack._cfg("core", "language", + ctx.cfg("language", default="auto")) plist_path = agents_dir / f"{PLIST_LABEL}.plist" - plist_path.write_text( + plist_content = ( f'\n' f'\n' @@ -276,8 +290,18 @@ def _setup_whisper_launchd(ctx, data_dir: Path, whisper_bin: Path, model_path: P f' --inference-path\n' f' /v1/audio/transcriptions\n' f' --convert\n' + # The language comes from config — auto-detection free-falls on + # silence and noise (measured: baby sounds transcribed as CJK). + # --max-context 0 stops a hallucinated segment from seeding the + # next one, and --suppress-nst drops non-speech tokens; each of + # the three flags fixed loops the other two did not, and only + # the combination cleared every recording behind the September + # 2026 runaway generation. f' --language\n' - f' auto\n' + f' {language}\n' + f' --max-context\n' + f' 0\n' + f' --suppress-nst\n' f' --threads\n' f' 4\n' f' \n' @@ -289,6 +313,15 @@ def _setup_whisper_launchd(ctx, data_dir: Path, whisper_bin: Path, model_path: P f'\n' ) + if (plist_path.exists() and plist_path.read_text() == plist_content + and wrapper.exists() and wrapper.read_text() == wrapper_text): + return + + section("Whisper server", "launchd service") + wrapper.write_text(wrapper_text) + wrapper.chmod(0o755) + plist_path.write_text(plist_content) + ctx.step("Loading whisper-server into launchd...") try: ctx.shell(f'launchctl unload "{plist_path}"') diff --git a/stacklets/ai/hooks/on_start.py b/stacklets/ai/hooks/on_start.py index bf8aa1a..bf32121 100644 --- a/stacklets/ai/hooks/on_start.py +++ b/stacklets/ai/hooks/on_start.py @@ -60,3 +60,42 @@ def run(ctx): out(f" {TEAL}stack destroy ai && stack up ai{RESET}") nl() raise RuntimeError("Missing openai_url for external provider") + + _reconcile_whisper(ctx) + + +def _reconcile_whisper(ctx): + """Bring the whisper LaunchAgent in line with the current code. + + The plist is generated by on_install, which only runs once — so a + change to the whisper flags (language pinning, --max-context, + --suppress-nst) would otherwise never reach an existing install. + This re-derives the plist on every start and lets the content- + idempotent setup decide: identical on disk means no write and no + service bounce, changed means rewrite and reload. A plain + `stack restart ai` is thereby enough to deploy a config change. + """ + if os.environ.get("STACK_AI_NO_VOICE") == "1": + return + # Best-effort: reconciliation needs the real Stack for paths and + # config. A context without one (the start-guard tests, an exotic + # embedding) simply skips it — config validation above is this + # hook's contract, keeping the plist current is a courtesy. + if getattr(ctx, "stack", None) is None: + return + import importlib.util + from pathlib import Path + + hooks_dir = Path(__file__).resolve().parent + spec = importlib.util.spec_from_file_location( + "hook.ai_on_install", hooks_dir / "on_install.py") + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + data_dir = Path(ctx.stack.data) + whisper_bin = data_dir / "ai" / "whisper.cpp" / "build" / "bin" / "whisper-server" + model_path = data_dir / "ai" / "whisper-models" / mod.WHISPER_MODEL + if not whisper_bin.exists() or not model_path.exists(): + return # whisper never installed here (or opted out) — nothing to reconcile + state_dir = Path(__file__).resolve().parent.parent / ".state" + mod._setup_whisper_launchd(ctx, data_dir, whisper_bin, model_path, state_dir) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index f86a62e..359c952 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -253,7 +253,20 @@ async def produce() -> dict: raise LLMError(f"could not download {message.url}") raw = await transcriber.transcribe( audio, filename=message.body or "voice.wav", vocabulary=vocabulary) - text = await Transcriber.polish(raw, llm) if raw.strip() else raw + # A hallucinated transcript (whisper looping on noise, or its + # language detection landing in CJK on baby sounds) must not + # reach the page as words somebody said. The entry keeps its + # place and its audio; the words are simply not there. The raw + # text stays in the record so a rerun under a better whisper + # config can be compared against what this one heard. + if raw.strip() and (why := Transcriber.looks_degenerate(raw)): + _err(f" transcript of {message.event_id} looks hallucinated " + f"({why}); keeping the recording without words") + text = "" + elif raw.strip(): + text = await Transcriber.polish(raw, llm) + else: + text = raw return {"raw": raw, "text": text, "url": message.url, "filename": message.body} diff --git a/tests/framework/test_ai_client.py b/tests/framework/test_ai_client.py index e23a3b9..fd9b1e4 100644 --- a/tests/framework/test_ai_client.py +++ b/tests/framework/test_ai_client.py @@ -743,7 +743,9 @@ class TestPolishIsBounded: that whole period.""" async def test_the_cap_scales_with_the_transcript(self, httpserver: HTTPServer): - long_transcript = "book the campsite " * 500 + # Long but not repetitive — a looping transcript would (rightly) + # be refused by the degeneracy gate before any cap applies. + long_transcript = " ".join(f"wort{i}" for i in range(1500)) httpserver.expect_request( "/v1/audio/transcriptions", method="POST", ).respond_with_json({"text": long_transcript}) @@ -862,3 +864,41 @@ async def test_a_changed_word_is_still_rejected(self, httpserver: HTTPServer): "theres decorations in the loft" ) await tr.aclose() + + +class TestDegenerateTranscriptsNeverReachTheModel: + """Polish is an echo task: handed a transcript that loops, the model + loops with it. The gate refuses the input instead of trusting the + output cap to contain it — the cap bounds the damage, the gate + removes the exposure. Thresholds come from the September 2026 + incident: real hallucination loops repeated one phrase 5-147x, + legitimately repetitive speech (a child singing the same line) + stayed at 1-2x.""" + + def test_a_looping_transcript_is_flagged(self): + looping = "und dann sind wir los und dann sind wir " * 12 + assert Transcriber.looks_degenerate(looping) is not None + + def test_a_song_line_sung_twice_is_not(self): + song = ("die affen rasen durch den wald der eine macht den " + "andern kalt die affen rasen durch den wald wer hat " + "die kokosnuss geklaut") + assert Transcriber.looks_degenerate(song) is None + + def test_cjk_on_a_latin_household_is_flagged(self): + """Whisper's language detection free-falls on non-speech (baby + sounds, music) and lands in CJK. famstack households write + German or English, so a mostly-CJK transcript was invented.""" + assert Transcriber.looks_degenerate("嬰兒 咿呀 學語 的 聲音 嬰兒") is not None + + def test_ordinary_german_is_not(self): + assert Transcriber.looks_degenerate( + "hallo bart heute ist der sechzehnte märz ich wollte dir " + "sagen dass ich stolz auf dich war") is None + + async def test_polish_refuses_a_looping_transcript(self): + looping = "and then we went " * 40 + llm = _StubLLM(result="should never be asked") + + assert await Transcriber.polish(looping, llm) == looping + assert llm.kwargs == [] From 23b3ea1d675b5fc90129042103c9b00617e8ce25 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 15:06:26 +0200 Subject: [PATCH 22/43] feat(ai): capture whisper quality metrics for the diary pipeline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Whisper computes segment and word confidence on every call; requesting plain json discards it. The diary needs it, voice commands do not. - Add Transcriber.transcribe_verbose(): verbose_json + word timestamps, returns text plus a compact quality record (segment metrics, words below 0.5 confidence, detected language). transcribe() is unchanged and stays on plain json — no extra cost for voice commands/notes. - Both paths share error translation via new _stt() helper. - Add Transcriber.quality_verdict(): flags transcripts where most segments fail OpenAI's reference thresholds (avg_logprob < -1.0, no_speech_prob > 0.6). Complements the n-gram gate: loops are high-confidence failures, mumble is low-confidence. - Diary stores the quality record in the transcript cache and gates on both signals. Gated recordings stay in the diary (entry + audio, no words) instead of being dropped by the no-speech skip. - Add --retranscribe: forces re-decoding after whisper config or vocabulary changes (TranscriptStore.run(force=True)). - Text messages remain verbatim; none of this touches them. --- lib/stack/ai/client.py | 117 ++++++++++++++++++++++--- stacklets/core/bot-runner/voice.py | 8 +- stacklets/memory/bot/cli/diary.py | 57 ++++++++---- stacklets/memory/bot/cli_entrypoint.py | 2 +- tests/framework/test_ai_client.py | 84 ++++++++++++++++++ 5 files changed, 240 insertions(+), 28 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index 6c8575d..b33e2d5 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -406,6 +406,16 @@ def _content_parts(prompt: str, images: list) -> list[dict]: _DEGENERATE_REPEATS = 3 _DEGENERATE_CJK_FRACTION = 0.3 +# Whisper's own quality signals (OpenAI reference thresholds). Kept +# alongside the text-level checks above: loops are often HIGH-confidence +# failures the logprob gate cannot see, and mumble is a low-confidence +# failure the n-gram gate cannot see. Both gates stay. +_QUALITY_LOGPROB_FLOOR = -1.0 +_QUALITY_NO_SPEECH = 0.6 +_QUALITY_POOR_FRACTION = 0.5 +_LOW_WORD_CONFIDENCE = 0.5 +_LOW_WORDS_KEPT = 50 + # Generation is slower than prefill, so a cap of N tokens still takes # time. This bounds one call to minutes rather than the client default, # which is sized for reading long documents. @@ -503,11 +513,106 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", LLM errors :class:`LLM` uses so callers can ``except LLMError`` once for both surfaces. """ + resp = await self._stt(audio, filename=filename, model=model, + vocabulary=vocabulary, + response_format="json") + raw = (getattr(resp, "text", "") or "").strip() + if not raw or cleanup_with is None: + return raw + return await self.polish(raw, cleanup_with) + + async def transcribe_verbose(self, audio: bytes, *, + filename: str = "voice.ogg", + model: str | None = None, + vocabulary: str = "") -> dict: + """Transcribe and keep whisper's own confidence about the result. + + Returns ``{"text": str, "quality": dict}``. This is the diary + pipeline's path, deliberately separate from :meth:`transcribe`: + voice commands and chat voice notes want text fast and get the + plain-json call with no quality machinery; the diary compiles + an archive and wants whisper's own judgement kept. Whisper + computes per-segment decode quality and per-word probabilities + either way — plain ``json`` merely discards them. ``vocabulary`` + primes the decoder exactly as in :meth:`transcribe` and is just + as optional. ``quality`` holds the small part worth keeping: + + duration, detected_language, detected_language_probability + segments: [{start, end, avg_logprob, no_speech_prob, + temperature}] + low_words: [{word, probability, start}] — only words below + the confidence threshold, ranked worst first + + Voice recordings only by design: text messages never pass + through here and are stored verbatim with no processing. + """ + resp = await self._stt(audio, filename=filename, model=model, + vocabulary=vocabulary, + response_format="verbose_json", + timestamp_granularities=["word"]) + # Defensive extraction: the SDK parses verbose_json into a typed + # model, but whisper.cpp adds fields OpenAI's schema lacks + # (detected_language, per-segment words) and may omit others. + # model_dump keeps extras; every read below tolerates absence. + data = resp.model_dump() if hasattr(resp, "model_dump") else dict(resp) + segments = data.get("segments") or [] + words = data.get("words") or [ + w for s in segments for w in (s.get("words") or [])] + quality = { + "duration": data.get("duration"), + "detected_language": data.get("detected_language"), + "detected_language_probability": + data.get("detected_language_probability"), + "segments": [ + {k: s.get(k) for k in + ("start", "end", "avg_logprob", "no_speech_prob", + "temperature")} + for s in segments], + "low_words": sorted( + ({"word": (w.get("word") or "").strip(), + "probability": w.get("probability"), + "start": w.get("start")} + for w in words + if (w.get("probability") or 1.0) < _LOW_WORD_CONFIDENCE), + key=lambda w: w["probability"] or 0.0, + )[:_LOW_WORDS_KEPT], + } + return {"text": (data.get("text") or "").strip(), "quality": quality} + + @staticmethod + def quality_verdict(quality: dict) -> str | None: + """Why whisper's own metrics call this transcript poor, or None. + + The thresholds are OpenAI's reference values, not invented ones: + a segment with avg_logprob < -1.0 failed decoding, one with + no_speech_prob > 0.6 was probably not speech. A transcript + failing most of its segments is not text — the recording keeps + its place and its audio, the words do not reach the page. + """ + segments = quality.get("segments") or [] + if not segments: + return None + poor = sum(1 for s in segments + if (s.get("avg_logprob") or 0.0) < _QUALITY_LOGPROB_FLOOR + or (s.get("no_speech_prob") or 0.0) > _QUALITY_NO_SPEECH) + if poor / len(segments) > _QUALITY_POOR_FRACTION: + return (f"{poor} of {len(segments)} segments failed " + f"whisper's own quality checks") + return None + + async def _stt(self, audio: bytes, *, filename: str, + model: str | None, vocabulary: str, **params): + """One STT call with famstack's error translation. + + Both transcription paths (plain json and verbose_json) go + through here so a whisper failure means the same typed LLMError + to every caller. ``params`` carries the per-path request shape. + """ try: - resp = await self._client.audio.transcriptions.create( + return await self._client.audio.transcriptions.create( model=model or _DEFAULT_WHISPER_MODEL, file=(filename, audio), - response_format="json", + **params, **({"prompt": vocabulary} if vocabulary.strip() else {}), ) except openai.APITimeoutError as e: @@ -525,14 +630,6 @@ async def transcribe(self, audio: bytes, *, filename: str = "voice.ogg", except openai.APIStatusError as e: raise LLMUnavailableError(f"HTTP {e.status_code}: {str(e)[:200]}") from e - # The SDK returns a Transcription object whose `.text` mirrors - # whisper-server's `{"text": ...}` response. Strip incidental - # whitespace so callers don't have to. - raw = (getattr(resp, "text", "") or "").strip() - if not raw or cleanup_with is None: - return raw - return await self.polish(raw, cleanup_with) - @staticmethod def cleanup_budget(raw: str) -> int: """Token cap for polishing `raw`: its own size plus 10%. diff --git a/stacklets/core/bot-runner/voice.py b/stacklets/core/bot-runner/voice.py index 8a08531..88da388 100644 --- a/stacklets/core/bot-runner/voice.py +++ b/stacklets/core/bot-runner/voice.py @@ -160,6 +160,7 @@ def write(self, event_id: str, record: dict) -> None: async def run( self, event_id: str, produce: Callable[[], Awaitable[dict]], + *, force: bool = False, ) -> dict: """Return the record for `event_id`, producing it at most once. @@ -167,8 +168,13 @@ async def run( first one's result. Failures are not retained: whisper outages are transient and the drain retries, so caching an error would make a message permanently undecodable. + + ``force`` skips the stored record and re-produces it — the + `--retranscribe` path for when the whisper config or vocabulary + improved and old recordings deserve a second hearing. The new + record overwrites the old one; single-flight still applies. """ - if (stored := self.read(event_id)) is not None: + if not force and (stored := self.read(event_id)) is not None: return stored if (pending := self._inflight.get(event_id)) is not None: return await pending diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 359c952..b05cd15 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -18,6 +18,9 @@ stack memory diary letters a different room stack memory diary --burst-window 1 see below stack memory diary --force re-read everything from scratch + stack memory diary --retranscribe decode all recordings again + (after a whisper config or + vocabulary change) WHY THE BURST WINDOW IS A KNOB Messages that synced late carry arrival timestamps, not recording @@ -239,7 +242,8 @@ def _frontmatter(path: Path) -> dict: async def _transcribe(message, *, session, homeserver, token, - transcriber, llm, vocabulary: str = "") -> str: + transcriber, llm, vocabulary: str = "", + retranscribe: bool = False) -> str: """The words of a recording, transcribed once and remembered. Shares `TRANSCRIPT_DIR` with the bots, so a memo the archivist @@ -251,27 +255,35 @@ async def produce() -> dict: audio = await _download(session, homeserver, token, message.url or "") if not audio: raise LLMError(f"could not download {message.url}") - raw = await transcriber.transcribe( + verbose = await transcriber.transcribe_verbose( audio, filename=message.body or "voice.wav", vocabulary=vocabulary) - # A hallucinated transcript (whisper looping on noise, or its - # language detection landing in CJK on baby sounds) must not - # reach the page as words somebody said. The entry keeps its - # place and its audio; the words are simply not there. The raw - # text stays in the record so a rerun under a better whisper - # config can be compared against what this one heard. - if raw.strip() and (why := Transcriber.looks_degenerate(raw)): - _err(f" transcript of {message.event_id} looks hallucinated " - f"({why}); keeping the recording without words") + raw = verbose["text"] + # A transcript whisper itself rates as failed (most segments + # below its own decode/no-speech thresholds), or one that shows + # the hallucination signatures (a loop, CJK on a latin + # household), must not reach the page as words somebody said. + # The entry keeps its place and its audio; the words are simply + # not there. Raw text and quality metrics stay in the record — + # for comparison after a whisper config change, and for the + # planned confidence-restricted correction pass. + why = None + if raw.strip(): + why = (Transcriber.quality_verdict(verbose["quality"]) + or Transcriber.looks_degenerate(raw)) + if why: + _err(f" transcript of {message.event_id} unusable ({why}); " + f"keeping the recording without words") text = "" elif raw.strip(): text = await Transcriber.polish(raw, llm) else: text = raw - return {"raw": raw, "text": text, "url": message.url, - "filename": message.body} + return {"raw": raw, "text": text, "quality": verbose["quality"], + "url": message.url, "filename": message.body} try: - return (await voice.TRANSCRIPTS.run(message.event_id, produce))["text"] + return (await voice.TRANSCRIPTS.run( + message.event_id, produce, force=retranscribe))["text"] except LLMError as e: _err(f" could not transcribe {message.event_id}: {e}") return "" @@ -639,6 +651,10 @@ async def run(llm, argv: list[str]) -> int: room_arg = _positional(argv, "memories") dry_run = "--dry-run" in argv rebuild = "--force" in argv + # --force re-reads and re-summarises but keeps transcripts: whisper + # costs minutes of GPU per recording and its output only changes + # when its config or vocabulary does. --retranscribe is that case. + retranscribe = "--retranscribe" in argv try: window = float(_opt(argv, "--burst-window", str(diary.DEFAULT_BURST_WINDOW_S))) @@ -704,9 +720,18 @@ async def run(llm, argv: list[str]) -> int: f"{_length(msg.duration_ms)}") text = await _transcribe( msg, session=session, homeserver=homeserver, token=token, - transcriber=transcriber, llm=llm, vocabulary=vocabulary) + transcriber=transcriber, llm=llm, vocabulary=vocabulary, + retranscribe=retranscribe) if not text.strip(): - _err(f" no speech in {msg.event_id}, skipped") + record = voice.TRANSCRIPTS.read(msg.event_id) or {} + if (record.get("raw") or "").strip(): + # Whisper heard something but the gate judged it + # unusable (loop, CJK, failed segments). The memory + # is not dropped: the entry keeps its place and its + # audio, with no words attached. + decoded.append(msg.__class__(**{**msg.__dict__, "body": ""})) + else: + _err(f" no speech in {msg.event_id}, skipped") continue decoded.append(msg.__class__(**{**msg.__dict__, "body": text})) diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index 460cd18..5967204 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -17,7 +17,7 @@ words. Exit 1 means no keywords, which the host treats as "search it literally" rather than as a failure. - diary [] [--burst-window ] [--dry-run] [--force] + diary [] [--burst-window ] [--dry-run] [--force] [--retranscribe] Compile the memories room into the family diary. Walks the room's full history, transcribes every recording, recovers the date each one was made, and publishes month pages under the diff --git a/tests/framework/test_ai_client.py b/tests/framework/test_ai_client.py index fd9b1e4..47af819 100644 --- a/tests/framework/test_ai_client.py +++ b/tests/framework/test_ai_client.py @@ -902,3 +902,87 @@ async def test_polish_refuses_a_looping_transcript(self): assert await Transcriber.polish(looping, llm) == looping assert llm.kwargs == [] + + +class TestTranscribeVerbose: + """The diary path keeps whisper's own confidence; the plain path + stays cheap. Same endpoint, different request shape.""" + + _VERBOSE = { + "text": " hallo bart heute ist der sechzehnte ", + "duration": 4.2, + "detected_language": "german", + "detected_language_probability": 0.98, + "segments": [{ + "id": 0, "start": 0.0, "end": 4.2, "temperature": 0.0, + "avg_logprob": -0.2, "no_speech_prob": 0.01, + "text": "hallo bart", + "words": [ + {"word": " hallo", "probability": 0.99, "start": 0.0, "end": 0.4}, + {"word": " Panorana", "probability": 0.13, "start": 0.5, "end": 1.0}, + ], + }], + } + + async def test_quality_record_is_extracted(self, httpserver: HTTPServer): + httpserver.expect_request( + "/v1/audio/transcriptions", method="POST", + ).respond_with_json(self._VERBOSE) + tr = _make_transcriber(httpserver) + + record = await tr.transcribe_verbose(b"a") + + assert record["text"] == "hallo bart heute ist der sechzehnte" + q = record["quality"] + assert q["detected_language"] == "german" + assert q["segments"] == [{"start": 0.0, "end": 4.2, + "avg_logprob": -0.2, + "no_speech_prob": 0.01, + "temperature": 0.0}] + # Only the low-confidence word is kept, worst first. + assert q["low_words"] == [ + {"word": "Panorana", "probability": 0.13, "start": 0.5}] + await tr.aclose() + + async def test_plain_transcribe_stays_on_the_cheap_format( + self, httpserver: HTTPServer): + seen: dict = {} + + def handler(request): + seen["format"] = request.form.get("response_format") + from werkzeug.wrappers import Response + return Response(json.dumps({"text": "hi"}), + content_type="application/json") + + httpserver.expect_request( + "/v1/audio/transcriptions", method="POST", + ).respond_with_handler(handler) + tr = _make_transcriber(httpserver) + + assert await tr.transcribe(b"a") == "hi" + assert seen["format"] == "json" + await tr.aclose() + + +class TestQualityVerdict: + """OpenAI's own segment thresholds, applied across the transcript: + most segments failed means the text never really happened.""" + + @staticmethod + def _seg(logprob=-0.2, no_speech=0.01): + return {"avg_logprob": logprob, "no_speech_prob": no_speech} + + def test_mostly_failed_segments_is_poor(self): + q = {"segments": [self._seg(logprob=-1.4)] * 3 + [self._seg()]} + assert Transcriber.quality_verdict(q) is not None + + def test_silence_counts_as_failure(self): + q = {"segments": [self._seg(no_speech=0.9)] * 3 + [self._seg()]} + assert Transcriber.quality_verdict(q) is not None + + def test_one_bad_segment_in_a_good_recording_is_fine(self): + q = {"segments": [self._seg()] * 5 + [self._seg(logprob=-1.4)]} + assert Transcriber.quality_verdict(q) is None + + def test_no_segments_is_no_verdict(self): + assert Transcriber.quality_verdict({"segments": []}) is None From 19b2ccaa52d983bd0d87625ec1827715114aa8aa Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 15:16:15 +0200 Subject: [PATCH 23/43] refactor(ai): extract transcript store and cleanup passes into shared lib STT post-processing was spread over the bot-runner and the diary CLI. Other voice consumers could not reuse it, and replacing a cleanup step required touching each consumer. - New lib/stack/ai/transcripts.py: TranscriptStore (moved from bot-runner voice.py, one-line re-export kept for the bots) plus a pass framework: TranscriptPass (name, version, apply), run_passes() with failure isolation, stale_passes() for staleness detection. - Records store an audit list per pass: name, version, model, outcome. Bumping a pass version makes affected records detectably stale, so a sweep re-runs one pass on cached raw text without re-running whisper. - gate_pass() and polish_pass() wrap the existing checks; the diary now composes them via run_passes(). Behavior unchanged. - Diary-specific logic (keep entry without words, dates, vocabulary sourcing) stays in the diary bot. Voice commands and text messages are untouched. - 16 new tests; 172 total pass. --- lib/stack/ai/transcripts.py | 245 +++++++++++++++++++++++++ stacklets/core/bot-runner/voice.py | 106 +---------- stacklets/memory/bot/cli/diary.py | 39 ++-- tests/framework/test_ai_transcripts.py | 163 ++++++++++++++++ 4 files changed, 429 insertions(+), 124 deletions(-) create mode 100644 lib/stack/ai/transcripts.py create mode 100644 tests/framework/test_ai_transcripts.py diff --git a/lib/stack/ai/transcripts.py b/lib/stack/ai/transcripts.py new file mode 100644 index 0000000..4f5a061 --- /dev/null +++ b/lib/stack/ai/transcripts.py @@ -0,0 +1,245 @@ +"""Transcript storage and cleanup passes. + +Transcription is expensive and stable. It costs minutes of GPU per +recording, and its output changes only with the whisper configuration +or the vocabulary. The steps after transcription are cheap and depend +on the model: the hallucination gate, punctuation restoration, and +future correction passes. This module separates the two stages. +`TranscriptStore` persists transcription results. Passes process the +stored records and can run again. + +A pass is a named, versioned operation on a transcript record. Each +record lists the passes that ran, with their version and model: + + {"raw": ..., "text": ..., "quality": {...}, + "passes": [{"name": "gate", "version": 1, "outcome": "clean"}, + {"name": "polish", "version": 1, + "model": "...", "outcome": "applied"}]} + +To adopt a better model, increase the pass version. `stale_passes` +finds the records that an older version produced. A sweep then runs +one pass again on the cached raw text. Whisper does not run again. +No pass modifies `raw`. It holds the words as transcribed. + +Each consumer selects its own pass chain. Voice commands use +`Transcriber.transcribe` only. The diary uses gate and polish and +keeps the quality metrics. Text messages do not enter this module: +famstack stores them verbatim. +""" + +from __future__ import annotations + +import asyncio +import base64 +import json +import os +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Awaitable, Callable + +from loguru import logger + +from .client import LLMError, Transcriber +from .models import resolve_model + + +class TranscriptStore: + """Transcripts on disk, keyed by Matrix event id. + + Transcription costs minutes of GPU for a long recording, and three + callers arrive at the same message independently: every bot in a room + drains the same timeline, the drain is at-least-once so a failed + handler brings its event back, and a backfill walks history in a + separate process. The store is therefore durable and shared, with one + file per message written atomically. + + Each record holds the raw whisper output beside the processed text. + Processing is cheap and improves with better models; transcription is + neither, so keeping both allows a later re-run of any pass without + returning to the audio. + """ + + def __init__(self, path: str | Path | None = None): + self.path = Path( + path or os.environ.get("TRANSCRIPT_DIR", "/data/core/transcripts") + ) + self._inflight: dict[str, asyncio.Future] = {} + + # ── Durable half ───────────────────────────────────────────────── + + def _file(self, event_id: str) -> Path: + # Event ids carry `$`, `/` and `+`; base64url keeps one file per + # id without inventing a collision-prone slug. + name = base64.urlsafe_b64encode(event_id.encode()).decode().rstrip("=") + return self.path / f"{name}.json" + + def read(self, event_id: str) -> dict | None: + """The stored record for `event_id`, or None if we never ran it.""" + try: + return json.loads(self._file(event_id).read_text()) + except FileNotFoundError: + return None + except (OSError, ValueError) as e: + logger.warning("[voice] unreadable transcript for {}: {}", event_id, e) + return None + + def write(self, event_id: str, record: dict) -> None: + """Write a record, replacing any existing one atomically. + + A store that cannot be written is logged and ignored. The + transcript still reaches the handler; only the saving is lost, at + the cost of transcribing again later. + """ + target = self._file(event_id) + try: + self.path.mkdir(parents=True, exist_ok=True) + tmp = target.with_suffix(".tmp") + tmp.write_text(json.dumps(record, ensure_ascii=False, indent=2)) + os.replace(tmp, target) + except OSError as e: + logger.warning("[voice] could not store transcript {}: {}", event_id, e) + + # ── Single-flight half ─────────────────────────────────────────── + + async def run( + self, event_id: str, produce: Callable[[], Awaitable[dict]], + *, force: bool = False, + ) -> dict: + """Return the record for `event_id`, producing it at most once. + + A stored record short-circuits; concurrent callers await the + first one's result. Failures are not retained: whisper outages + are transient and the drain retries, so caching an error would + make a message permanently undecodable. + + ``force`` skips the stored record and re-produces it — the + `--retranscribe` path for when the whisper config or vocabulary + improved and old recordings deserve a second hearing. The new + record overwrites the old one; single-flight still applies. + """ + if not force and (stored := self.read(event_id)) is not None: + return stored + if (pending := self._inflight.get(event_id)) is not None: + return await pending + + loop = asyncio.get_running_loop() + future: asyncio.Future = loop.create_future() + self._inflight[event_id] = future + try: + record = await produce() + except BaseException as e: + self._inflight.pop(event_id, None) + if not future.done(): + future.set_exception(e) + # Retrieve it so an unawaited future does not warn. + future.exception() + raise + record.setdefault("at", time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())) + # Written and published before the in-flight slot is released, so + # a caller arriving in between finds the result rather than + # starting a second transcription. + self.write(event_id, record) + if not future.done(): + future.set_result(record) + self._inflight.pop(event_id, None) + return record + + +# ── Cleanup passes ──────────────────────────────────────────────────── + + +@dataclass(frozen=True) +class TranscriptPass: + """One named, versioned operation on a transcript record. + + ``apply`` receives the record and returns (text, outcome, extra). + ``text`` is the new text, or the old text if unchanged. ``outcome`` + is a short status for the pass list: "applied", "clean", + "blocked: ". ``extra`` holds metadata to keep, for example + the model name. The runner writes the pass list. A pass makes one + decision. + """ + + name: str + version: int + apply: Callable[[dict], Awaitable[tuple[str, str, dict]]] + + +async def run_passes(record: dict, passes: list[TranscriptPass]) -> dict: + """Run a pass chain on a record and update its pass list. + + A pass that raises an LLMError is logged and recorded as failed. + The chain continues with the unchanged text. A broken cleanup pass + does not remove the transcript. No pass modifies ``raw``. + """ + trail = list(record.get("passes") or []) + for p in passes: + try: + text, outcome, extra = await p.apply(record) + record["text"] = text + except LLMError as e: + outcome, extra = f"failed: {e}", {} + logger.warning("[transcripts] pass {} failed: {}", p.name, e) + trail = [e for e in trail if e.get("name") != p.name] + trail.append({"name": p.name, "version": p.version, + "outcome": outcome, **extra}) + record["passes"] = trail + return record + + +def stale_passes(record: dict, passes: list[TranscriptPass]) -> list[TranscriptPass]: + """Return the passes with a missing or outdated entry in the record. + + A sweep uses this to run only the passes that a version change + invalidated. The input is the cached raw text. Whisper does not + run again. + """ + ran = {e.get("name"): e.get("version") + for e in (record.get("passes") or [])} + return [p for p in passes if ran.get(p.name) != p.version] + + +def gate_pass() -> TranscriptPass: + """Block hallucinated transcripts before a model receives them. + + The gate uses two independent signals. Whisper's segment metrics + detect low-confidence failures: unclear speech and silence. The + text checks detect high-confidence failures: repetition loops and + CJK output for a latin-language household. A blocked record gets + empty text. The consumer keeps the entry and its audio link. + """ + async def apply(record: dict) -> tuple[str, str, dict]: + raw = (record.get("raw") or "").strip() + if not raw: + return "", "clean", {} + why = (Transcriber.quality_verdict(record.get("quality") or {}) + or Transcriber.looks_degenerate(raw)) + if why: + return "", f"blocked: {why}", {} + return record.get("text") or raw, "clean", {} + + return TranscriptPass(name="gate", version=1, apply=apply) + + +def polish_pass(llm) -> TranscriptPass: + """Restore punctuation. Do not change words (see Transcriber.polish). + + The pass skips records with empty text. The gate empties the text + on purpose; an empty input is not a failure. + """ + async def apply(record: dict) -> tuple[str, str, dict]: + text = (record.get("text") or "").strip() + if not text: + return record.get("text") or "", "skipped: no text", {} + polished = await Transcriber.polish(text, llm) + outcome = "applied" if polished != text else "unchanged" + # The model name is metadata. A missing configuration must not + # fail the pass. + try: + extra = {"model": resolve_model("transcript_cleanup")} + except ValueError: + extra = {} + return polished, outcome, extra + + return TranscriptPass(name="polish", version=1, apply=apply) diff --git a/stacklets/core/bot-runner/voice.py b/stacklets/core/bot-runner/voice.py index 88da388..19dc89a 100644 --- a/stacklets/core/bot-runner/voice.py +++ b/stacklets/core/bot-runner/voice.py @@ -100,107 +100,11 @@ def transcribed_source(source: dict, transcript: str) -> dict: return out -class TranscriptStore: - """Transcripts on disk, keyed by Matrix event id. - - Transcription costs minutes of GPU for a long recording, and three - callers arrive at the same message independently: every bot in a room - drains the same timeline, the drain is at-least-once so a failed - handler brings its event back, and a backfill walks history in a - separate process. The store is therefore durable and shared, with one - file per message written atomically. - - Each record holds the raw whisper output beside the polished text. - Polishing is cheap and improves with better models; transcription is - neither, so keeping both allows a later re-polish without returning - to the audio. - """ - - def __init__(self, path: str | Path | None = None): - self.path = Path( - path or os.environ.get("TRANSCRIPT_DIR", "/data/core/transcripts") - ) - self._inflight: dict[str, asyncio.Future] = {} - - # ── Durable half ───────────────────────────────────────────────── - - def _file(self, event_id: str) -> Path: - # Event ids carry `$`, `/` and `+`; base64url keeps one file per - # id without inventing a collision-prone slug. - name = base64.urlsafe_b64encode(event_id.encode()).decode().rstrip("=") - return self.path / f"{name}.json" - - def read(self, event_id: str) -> dict | None: - """The stored record for `event_id`, or None if we never ran it.""" - try: - return json.loads(self._file(event_id).read_text()) - except FileNotFoundError: - return None - except (OSError, ValueError) as e: - logger.warning("[voice] unreadable transcript for {}: {}", event_id, e) - return None - - def write(self, event_id: str, record: dict) -> None: - """Write a record, replacing any existing one atomically. - - A store that cannot be written is logged and ignored. The - transcript still reaches the handler; only the saving is lost, at - the cost of transcribing again later. - """ - target = self._file(event_id) - try: - self.path.mkdir(parents=True, exist_ok=True) - tmp = target.with_suffix(".tmp") - tmp.write_text(json.dumps(record, ensure_ascii=False, indent=2)) - os.replace(tmp, target) - except OSError as e: - logger.warning("[voice] could not store transcript {}: {}", event_id, e) - - # ── Single-flight half ─────────────────────────────────────────── - - async def run( - self, event_id: str, produce: Callable[[], Awaitable[dict]], - *, force: bool = False, - ) -> dict: - """Return the record for `event_id`, producing it at most once. - - A stored record short-circuits; concurrent callers await the - first one's result. Failures are not retained: whisper outages - are transient and the drain retries, so caching an error would - make a message permanently undecodable. - - ``force`` skips the stored record and re-produces it — the - `--retranscribe` path for when the whisper config or vocabulary - improved and old recordings deserve a second hearing. The new - record overwrites the old one; single-flight still applies. - """ - if not force and (stored := self.read(event_id)) is not None: - return stored - if (pending := self._inflight.get(event_id)) is not None: - return await pending - - loop = asyncio.get_running_loop() - future: asyncio.Future = loop.create_future() - self._inflight[event_id] = future - try: - record = await produce() - except BaseException as e: - self._inflight.pop(event_id, None) - if not future.done(): - future.set_exception(e) - # Retrieve it so an unawaited future does not warn. - future.exception() - raise - record.setdefault("at", time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())) - # Written and published before the in-flight slot is released, so - # a caller arriving in between finds the result rather than - # starting a second transcription. - self.write(event_id, record) - if not future.done(): - future.set_result(record) - self._inflight.pop(event_id, None) - return record - +# The store class lives in the shared AI library. All consumers use +# one implementation and one directory: the bots in this runner, the +# diary backfill, and future services. This module keeps the +# process-global instance that the bots import. +from stack.ai.transcripts import TranscriptStore # noqa: E402,F401 # Process-global: the point is to share across the bots in this runner, # and with whatever process backfills history. diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index b05cd15..af2fef3 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -67,6 +67,7 @@ import diary_store # noqa: E402 import voice # noqa: E402 from stack.ai.client import LLMError, Transcriber # noqa: E402 +from stack.ai import transcripts # noqa: E402 from . import wiki # noqa: E402 @@ -257,29 +258,21 @@ async def produce() -> dict: raise LLMError(f"could not download {message.url}") verbose = await transcriber.transcribe_verbose( audio, filename=message.body or "voice.wav", vocabulary=vocabulary) - raw = verbose["text"] - # A transcript whisper itself rates as failed (most segments - # below its own decode/no-speech thresholds), or one that shows - # the hallucination signatures (a loop, CJK on a latin - # household), must not reach the page as words somebody said. - # The entry keeps its place and its audio; the words are simply - # not there. Raw text and quality metrics stay in the record — - # for comparison after a whisper config change, and for the - # planned confidence-restricted correction pass. - why = None - if raw.strip(): - why = (Transcriber.quality_verdict(verbose["quality"]) - or Transcriber.looks_degenerate(raw)) - if why: - _err(f" transcript of {message.event_id} unusable ({why}); " - f"keeping the recording without words") - text = "" - elif raw.strip(): - text = await Transcriber.polish(raw, llm) - else: - text = raw - return {"raw": raw, "text": text, "quality": verbose["quality"], - "url": message.url, "filename": message.body} + record = {"raw": verbose["text"], "text": verbose["text"], + "quality": verbose["quality"], + "url": message.url, "filename": message.body} + # The diary's cleanup chain. The gate blocks hallucinated + # words; the entry keeps its place and its audio. Polish + # restores punctuation. The record lists each pass, so a + # better future model can run one pass again on the cached + # raw text. Whisper does not run again. + record = await transcripts.run_passes( + record, [transcripts.gate_pass(), transcripts.polish_pass(llm)]) + gate = next((p for p in record["passes"] if p["name"] == "gate"), {}) + if str(gate.get("outcome", "")).startswith("blocked"): + _err(f" transcript of {message.event_id} unusable " + f"({gate['outcome']}); keeping the recording without words") + return record try: return (await voice.TRANSCRIPTS.run( diff --git a/tests/framework/test_ai_transcripts.py b/tests/framework/test_ai_transcripts.py new file mode 100644 index 0000000..907f2d8 --- /dev/null +++ b/tests/framework/test_ai_transcripts.py @@ -0,0 +1,163 @@ +"""Unit tests for `stack.ai.transcripts`. + +The store persists whisper output. Passes process stored records and +write an audit list. These tests pin the store round-trip, the force +path, pass isolation, and staleness detection. +""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).parent.parent.parent / "lib")) + +from stack.ai.client import LLMUnavailableError # noqa: E402 +from stack.ai.transcripts import ( # noqa: E402 + TranscriptPass, + TranscriptStore, + gate_pass, + polish_pass, + run_passes, + stale_passes, +) + + +class _StubLLM: + """Returns a fixed string, or raises. Records calls.""" + + def __init__(self, result: str = "", error: Exception | None = None): + self.result, self.error = result, error + self.calls: list[dict] = [] + + async def complete(self, role, prompt, **kw): + self.calls.append({"role": role, **kw}) + if self.error: + raise self.error + return self.result + + +class TestStore: + async def test_round_trip_and_single_production(self, tmp_path): + store = TranscriptStore(tmp_path) + produced = 0 + + async def produce(): + nonlocal produced + produced += 1 + return {"raw": "hi", "text": "hi"} + + first = await store.run("$ev1", produce) + second = await store.run("$ev1", produce) + assert first["text"] == second["text"] == "hi" + assert produced == 1 + assert store.read("$ev1")["raw"] == "hi" + + async def test_force_reproduces_and_overwrites(self, tmp_path): + store = TranscriptStore(tmp_path) + results = iter([{"text": "old"}, {"text": "new"}]) + + async def produce(): + return next(results) + + await store.run("$ev1", produce) + forced = await store.run("$ev1", produce, force=True) + assert forced["text"] == "new" + assert store.read("$ev1")["text"] == "new" + + +class TestRunPasses: + async def test_writes_the_pass_list(self): + async def upper(record): + return record["text"].upper(), "applied", {"model": "m1"} + + record = await run_passes( + {"raw": "hi", "text": "hi"}, + [TranscriptPass("upper", 2, upper)]) + + assert record["text"] == "HI" + assert record["passes"] == [ + {"name": "upper", "version": 2, "outcome": "applied", + "model": "m1"}] + assert record["raw"] == "hi" + + async def test_a_failing_pass_keeps_the_text_and_the_chain(self): + async def broken(record): + raise LLMUnavailableError("down") + + async def upper(record): + return record["text"].upper(), "applied", {} + + record = await run_passes( + {"raw": "hi", "text": "hi"}, + [TranscriptPass("broken", 1, broken), + TranscriptPass("upper", 1, upper)]) + + assert record["text"] == "HI" + outcomes = {p["name"]: p["outcome"] for p in record["passes"]} + assert outcomes["broken"].startswith("failed") + assert outcomes["upper"] == "applied" + + async def test_a_rerun_replaces_the_old_entry(self): + async def noop(record): + return record["text"], "clean", {} + + record = {"raw": "hi", "text": "hi", + "passes": [{"name": "gate", "version": 1, + "outcome": "clean"}]} + record = await run_passes(record, [TranscriptPass("gate", 2, noop)]) + assert record["passes"] == [ + {"name": "gate", "version": 2, "outcome": "clean"}] + + +class TestStalePasses: + def test_version_change_marks_stale(self): + async def noop(record): + return record["text"], "clean", {} + + p1, p2 = (TranscriptPass("gate", 2, noop), + TranscriptPass("polish", 1, noop)) + record = {"passes": [{"name": "gate", "version": 1}, + {"name": "polish", "version": 1}]} + assert stale_passes(record, [p1, p2]) == [p1] + + def test_a_record_without_a_list_is_fully_stale(self): + async def noop(record): + return "", "clean", {} + + p = TranscriptPass("gate", 1, noop) + assert stale_passes({}, [p]) == [p] + + +class TestGatePass: + async def test_blocks_a_loop_and_empties_the_text(self): + looping = "and then we went " * 40 + text, outcome, _ = await gate_pass().apply( + {"raw": looping, "text": looping, "quality": {}}) + assert text == "" + assert outcome.startswith("blocked") + + async def test_clean_text_passes_through(self): + text, outcome, _ = await gate_pass().apply( + {"raw": "hallo bart", "text": "hallo bart", "quality": {}}) + assert (text, outcome) == ("hallo bart", "clean") + + +class TestPolishPass: + async def test_skips_empty_text_without_a_model_call(self): + llm = _StubLLM() + text, outcome, _ = await polish_pass(llm).apply( + {"raw": "garbage", "text": ""}) + assert (text, outcome) == ("", "skipped: no text") + assert llm.calls == [] + + async def test_applies_polish_and_records_the_model(self): + llm = _StubLLM(result="Hallo Bart.") + text, outcome, extra = await polish_pass(llm).apply( + {"raw": "hallo bart", "text": "hallo bart"}) + assert text == "Hallo Bart." + assert outcome == "applied" + # No model is configured in the test env; extra stays empty. + assert extra == {} From a77d77e20193c2ebc1da7ec87ebc6fe6d4713f16 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 15:47:00 +0200 Subject: [PATCH 24/43] feat(diary): paragraph structure from segments, note for gated recordings Long memos rendered as one text block; gated recordings rendered as a bare header with no explanation. - transcribe_verbose() stores a per-segment word_count. Polish keeps the word sequence, so cumulative counts map segments onto polished text without storing segment text. - New structure_pass(): inserts paragraph breaks where the pause between segments is >= 1.5s. Deterministic, no model call. Skips on word-count mismatch (polish hyphenation can shift counts). - Diary chain is now gate -> polish -> structure. - render: an empty voice entry states 'This recording could not be transcribed.' next to the audio link instead of showing nothing. --- lib/stack/ai/client.py | 12 +++++-- lib/stack/ai/transcripts.py | 47 ++++++++++++++++++++++++++ stacklets/memory/bot/cli/diary.py | 3 +- stacklets/memory/bot/diary.py | 5 +++ tests/framework/test_ai_client.py | 3 +- tests/framework/test_ai_transcripts.py | 41 ++++++++++++++++++++++ tests/stacklets/test_memory_diary.py | 9 +++++ 7 files changed, 115 insertions(+), 5 deletions(-) diff --git a/lib/stack/ai/client.py b/lib/stack/ai/client.py index b33e2d5..85c7146 100644 --- a/lib/stack/ai/client.py +++ b/lib/stack/ai/client.py @@ -564,9 +564,15 @@ async def transcribe_verbose(self, audio: bytes, *, "detected_language_probability": data.get("detected_language_probability"), "segments": [ - {k: s.get(k) for k in - ("start", "end", "avg_logprob", "no_speech_prob", - "temperature")} + {**{k: s.get(k) for k in + ("start", "end", "avg_logprob", "no_speech_prob", + "temperature")}, + # The word count maps this segment to its part of the + # text. Polish keeps the word sequence, so cumulative + # counts let a later pass insert paragraph breaks at + # segment boundaries without stored segment text. + "word_count": (len(s["words"]) if s.get("words") + else len((s.get("text") or "").split()))} for s in segments], "low_words": sorted( ({"word": (w.get("word") or "").strip(), diff --git a/lib/stack/ai/transcripts.py b/lib/stack/ai/transcripts.py index 4f5a061..7eab902 100644 --- a/lib/stack/ai/transcripts.py +++ b/lib/stack/ai/transcripts.py @@ -222,6 +222,53 @@ async def apply(record: dict) -> tuple[str, str, dict]: return TranscriptPass(name="gate", version=1, apply=apply) +def structure_pass(min_pause_s: float = 1.5, + min_words: int = 8) -> TranscriptPass: + """Insert paragraph breaks at long pauses between segments. + + Whisper's segment boundaries fall on pauses in the speech. A pause + of ``min_pause_s`` or more starts a new paragraph. Paragraphs + shorter than ``min_words`` merge into the next one. The split uses + the per-segment word counts from the quality record; no model runs. + + Polish can merge or split words (hyphenation), which shifts the + counts. When the counts do not match the text, the pass skips and + the text stays unchanged. Run this pass after polish. + """ + async def apply(record: dict) -> tuple[str, str, dict]: + text = (record.get("text") or "").strip() + segments = (record.get("quality") or {}).get("segments") or [] + counts = [s.get("word_count") for s in segments] + if not text or not segments or None in counts: + return record.get("text") or "", "skipped: no segment data", {} + words = text.split() + if sum(counts) != len(words): + return text, "skipped: word counts do not match", {} + + paragraphs: list[str] = [] + current: list[str] = [] + cursor = 0 + for i, segment in enumerate(segments): + current.extend(words[cursor:cursor + counts[i]]) + cursor += counts[i] + following = segments[i + 1] if i + 1 < len(segments) else None + pause = 0.0 + if (following and segment.get("end") is not None + and following.get("start") is not None): + pause = following["start"] - segment["end"] + if following is None or (pause >= min_pause_s + and len(current) >= min_words): + paragraphs.append(" ".join(current)) + current = [] + if current: + paragraphs.append(" ".join(current)) + structured = "\n\n".join(paragraphs) + outcome = "applied" if len(paragraphs) > 1 else "unchanged" + return structured, outcome, {} + + return TranscriptPass(name="structure", version=1, apply=apply) + + def polish_pass(llm) -> TranscriptPass: """Restore punctuation. Do not change words (see Transcriber.polish). diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index af2fef3..ef02e34 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -267,7 +267,8 @@ async def produce() -> dict: # better future model can run one pass again on the cached # raw text. Whisper does not run again. record = await transcripts.run_passes( - record, [transcripts.gate_pass(), transcripts.polish_pass(llm)]) + record, [transcripts.gate_pass(), transcripts.polish_pass(llm), + transcripts.structure_pass()]) gate = next((p for p in record["passes"] if p["name"] == "gate"), {}) if str(gate.get("outcome", "")).startswith("blocked"): _err(f" transcript of {message.event_id} unusable " diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index fdbef9e..8d6eb11 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -638,6 +638,11 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: if entry.body.strip(): lines += [entry.body.strip(), ""] + elif entry.kind == "voice" and not entry.comments: + # A gated recording: the transcript was unusable and the words + # stay off the page. The sentence tells the reader this is + # deliberate. The audio link below stays the way to hear it. + lines += ["This recording could not be transcribed.", ""] elif entry.kind in _UPLOADS and not entry.comments: lines += ["Nothing was written alongside this one.", ""] diff --git a/tests/framework/test_ai_client.py b/tests/framework/test_ai_client.py index 47af819..f866027 100644 --- a/tests/framework/test_ai_client.py +++ b/tests/framework/test_ai_client.py @@ -938,7 +938,8 @@ async def test_quality_record_is_extracted(self, httpserver: HTTPServer): assert q["segments"] == [{"start": 0.0, "end": 4.2, "avg_logprob": -0.2, "no_speech_prob": 0.01, - "temperature": 0.0}] + "temperature": 0.0, + "word_count": 2}] # Only the low-confidence word is kept, worst first. assert q["low_words"] == [ {"word": "Panorana", "probability": 0.13, "start": 0.5}] diff --git a/tests/framework/test_ai_transcripts.py b/tests/framework/test_ai_transcripts.py index 907f2d8..cac51f4 100644 --- a/tests/framework/test_ai_transcripts.py +++ b/tests/framework/test_ai_transcripts.py @@ -161,3 +161,44 @@ async def test_applies_polish_and_records_the_model(self): assert outcome == "applied" # No model is configured in the test env; extra stays empty. assert extra == {} + + +class TestStructurePass: + """Paragraph breaks come from segment pauses and word counts. No + model runs; a count mismatch skips the pass.""" + + @staticmethod + def _seg(start, end, n): + return {"start": start, "end": end, "word_count": n} + + async def test_breaks_at_a_long_pause(self): + from stack.ai.transcripts import structure_pass + record = { + "text": "one two three four five six seven eight " + "nine ten eleven twelve thirteen fourteen fifteen sixteen", + "quality": {"segments": [self._seg(0, 5, 8), + self._seg(7.5, 12, 8)]}, + } + text, outcome, _ = await structure_pass().apply(record) + assert outcome == "applied" + assert text.split("\n\n") == [ + "one two three four five six seven eight", + "nine ten eleven twelve thirteen fourteen fifteen sixteen"] + + async def test_a_short_pause_does_not_break(self): + from stack.ai.transcripts import structure_pass + record = { + "text": "a b c d e f g h i j k l m n o p", + "quality": {"segments": [self._seg(0, 5, 8), + self._seg(5.2, 9, 8)]}, + } + text, outcome, _ = await structure_pass().apply(record) + assert (outcome, "\n\n" in text) == ("unchanged", False) + + async def test_count_mismatch_skips(self): + from stack.ai.transcripts import structure_pass + record = {"text": "only three words", + "quality": {"segments": [self._seg(0, 5, 8)]}} + text, outcome, _ = await structure_pass().apply(record) + assert text == "only three words" + assert outcome.startswith("skipped") diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 31a3938..79b739c 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -650,6 +650,15 @@ def test_a_photo_without_a_caption_says_so_rather_than_naming_a_file(self): assert "Nothing was written alongside this one." in page assert ".png" not in page + def test_an_untranscribable_recording_says_so(self): + """A gated voice entry keeps its place; the page states why the + words are missing instead of showing nothing.""" + page = diary.render_month([diary.Entry( + on=date(2026, 9, 14), confidence="sent", basis="dated from when " + "it was sent", kind="voice", sender="homer", body="")]) + + assert "This recording could not be transcribed." in page + def test_a_speaker_is_not_addressed_to_themselves(self): """A misread addressee must not become a dedication.""" page = diary.render_month([diary.Entry( From 7d76d6b33284f7d4f22a09ae034e1c41af0305ed Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 16:06:10 +0200 Subject: [PATCH 25/43] docs(diary): replace the verbatim-only promise with the quote promise The 'never paraphrase, verbatim only' stance was introduced during implementation, not a product decision. The product goal is a chronicle/diary hybrid: right amount of detail, word-for-word quotes where words are shown as someone's own, audio links as the archival originals. - Module docstring, render_month docstring, and the front-page text now state: quoted words are unchanged from the recording; narrative renders as narrative, never as quotation; entries link to sources. - memories-pipeline.md documents the tiered page format (gist, narrative, verified quotes, audio link; full transcript stays in the transcript store). --- docs/design/brain/memories-pipeline.md | 6 ++++++ stacklets/memory/bot/diary.py | 25 +++++++++++++++---------- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/docs/design/brain/memories-pipeline.md b/docs/design/brain/memories-pipeline.md index 6567fac..0757ed4 100644 --- a/docs/design/brain/memories-pipeline.md +++ b/docs/design/brain/memories-pipeline.md @@ -68,6 +68,12 @@ room history (paginated, oldest-first) few recordings to touch. - **Goal is topics, not verbatim accuracy.** Whisper large-v3-turbo was rated clean on real German memos; good enough. No model change needed. +- **Pages are a chronicle/diary hybrid, not transcript dumps.** Detail + scales with the source: short memos stay verbatim, long recordings + get a gist, a short narrative, selected word-for-word quotes, and + the audio link. Quoted words are verified against the transcript; + narrative renders as narrative. The full transcript stays in the + transcript store; audio in Matrix is the archival original. - **Privacy shape:** content flows machine-to-machine (Synapse → Whisper → oMLX → vault); only structure and compiled entries surface. diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 8d6eb11..8279dbf 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -15,10 +15,13 @@ the hard parts (which date wins, what is one recording and what is two) be tested against the corpus in `tools/family-memories` without a rig. -The diary never paraphrases. The model is asked to *read* a transcript, -never to rewrite one: what lands on the page is the words that were -said. That is a promise to the reader in 2040, and it is also why the -classification step returns a small record of facts rather than prose. +The diary quotes, it does not invent. Words shown as someone's own — +a quote, a transcript — come from the recording unchanged. Narrative +text (summaries, chronicle paragraphs) is permitted and renders as +narrative, never as quotation. Every entry links to its source events, +and the audio stays the archival original. The classification step +returns a small record of facts rather than prose for the same reason: +generated text must never blend into quoted text. """ from __future__ import annotations @@ -733,10 +736,10 @@ def _recorded_by(entries) -> list[str]: def render_month(entries, *, room_id: str = "", summary: str = "") -> str: """A month of entries, grouped by the day they happened. - `summary` is an optional paragraph recalling the month, and the one - piece of writing here that is not the family's own. It opens the - page; everything under it is verbatim. That promise is made once, on - the diary's front page, rather than restated on every month. + `summary` is an optional paragraph recalling the month. It opens + the page as narrative. Material quoted below it is word-for-word + from the recordings. The full promise is stated once, on the + diary's front page, not on every month. Entries whose date could not be recovered are still shown on the day they surfaced, under a heading that says as much. Hiding them would @@ -797,8 +800,10 @@ def render_index(entries) -> str: "# Family Diary", "", "Everything the family has put in the memories room: voice notes, " - "photos, conversations someone hit record on. Entries are quoted " - "exactly as they were said or written.", + "photos, conversations someone hit record on. Words shown as " + "someone's own are word-for-word from the recording. The text " + "around them is the chronicle, and every entry links back to the " + "original in the room.", "", ] if not entries: From 855ee17900ef60385d2cfbf68ed0a7bb0fcc1c8d Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 16:20:25 +0200 Subject: [PATCH 26/43] feat(diary): distill long recordings into gist, quotes, and folded transcript Long entries rendered as full transcript dumps. The goal is a chronicle/diary hybrid: right amount of detail, exact quotes, audio preserved as the original. - Reading gains gist (one narrative sentence) and moments (passages the model copies from the text). Prompt and parsing extended; per-message read budget 120 -> 240 tokens. - verify_moments() accepts a claimed quote only when it matches one sentence or a consecutive run in the body (case/punctuation insensitive) and returns the body's own text. Invented quotes drop. - Entries >= 120 words with a gist render distilled: gist, quote callouts, full transcript in a folded block, audio link unchanged. Shorter entries stay verbatim. - Cached readings without a moments key are read again once (pre-distillation cache entries). - 7 new tests; 2683 pass. --- stacklets/memory/bot/cli/diary.py | 28 +++++++-- stacklets/memory/bot/diary.py | 89 +++++++++++++++++++++++++++- tests/stacklets/test_memory_diary.py | 62 +++++++++++++++++++ 3 files changed, 174 insertions(+), 5 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index ef02e34..3be894b 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -293,7 +293,9 @@ async def produce() -> dict: # One small JSON object per message in the slice, plus the enclosing # structure. A model that loops instead of closing the array is capped # here rather than at the client timeout. -_READ_TOKENS_PER_MESSAGE = 120 +# The budget covers the facts plus the distillation: a gist sentence +# and up to three copied passages for long messages. +_READ_TOKENS_PER_MESSAGE = 240 _READ_TIMEOUT_S = 300.0 # Two to four sentences. @@ -303,8 +305,8 @@ async def produce() -> dict: _READ_PROMPT = """\ You are reading a family's private memories room so their diary can be -compiled. Report facts about these messages. Never rewrite one, never -summarise one, never translate one. +compiled. Report facts about these messages. Never translate one. When +you quote, copy the words exactly. The messages are in the order the server received them, which is not always the order they were recorded: a phone that has been offline @@ -352,6 +354,15 @@ async def produce() -> dict: recording from weeks ago and is still its own memory, not a footnote to it. Use this only when the message would make no sense on its own page. Otherwise null. + +"gist": for a message longer than about 100 words: one sentence, in + the language of the message, saying what it is about and for whom. + Plain and specific, no marketing words. For shorter messages null. + +"moments": for a message longer than about 100 words: up to three + short passages copied word-for-word from the message, the lines most + worth keeping. Copy them exactly as written, complete sentences + only, no edits. Otherwise an empty list. """ @@ -420,6 +431,8 @@ def remember(event_id: str, row: dict) -> None: mode=str(row.get("mode") or "monologue"), spoken_date=row.get("spoken_date") or None, addressee=row.get("addressee") or None, + gist=row.get("gist") or None, + moments=tuple(row.get("moments") or ()), ) if target := row.get("continues"): continues[event_id] = target @@ -429,7 +442,10 @@ def remember(event_id: str, row: dict) -> None: if cache is not None: for msg in messages: - if (stored := cache.get(msg.event_id, msg.body)) is not None: + stored = cache.get(msg.event_id, msg.body) + # Readings from before distillation carry no "moments" key. + # Treat them as absent so the message is read again once. + if stored is not None and "moments" in stored: remember(msg.event_id, stored) if known: _err(f" {len(known)} message(s) already read, " @@ -470,6 +486,10 @@ def remember(event_id: str, row: dict) -> None: "addressee": row.get("addressee") or None, "continues": _link(row.get("continues"), chunk), "refers_to": _link(row.get("refers_to"), chunk), + "gist": (str(row.get("gist")).strip() + if row.get("gist") else None), + "moments": [str(m) for m in row.get("moments") or [] + if str(m).strip()][:5], } remember(here.event_id, found) if cache is not None: diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 8279dbf..653335e 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -95,6 +95,12 @@ class Reading: mode: str = "monologue" # "monologue" | "dialogue" | "note" spoken_date: str | None = None addressee: str | None = None + # Distillation, for long recordings. `gist` is one narrative + # sentence about the message. `moments` are passages the model + # copied from the text; verify_moments() checks each one against + # the body before it can render as a quote. + gist: str | None = None + moments: tuple = () @dataclass @@ -118,6 +124,11 @@ class Entry: duration_ms: int | None = None mode: str = "monologue" comments: list[tuple[str, str]] = field(default_factory=list) + # Distilled view for long recordings: one narrative sentence and + # verified word-for-word quotes. Empty for short entries; the + # renderer then shows the body in full. + gist: str = "" + moments: list[str] = field(default_factory=list) # ── What this household says ────────────────────────────────────────── @@ -527,6 +538,8 @@ def compile_entries(messages, readings, *, addressee=reading.addressee, duration_ms=_total_duration(group), mode=reading.mode, + gist=(reading.gist or "").strip(), + moments=verify_moments(_joined_body(group), reading.moments), ) entries.append(entry) for m in group: @@ -545,6 +558,8 @@ def compile_entries(messages, readings, *, sender=msg.sender, body=_joined_body(group), at=msg.ts, event_ids=[m.event_id for m in group], addressee=reading.addressee, mode=reading.mode, + gist=(reading.gist or "").strip(), + moments=verify_moments(_joined_body(group), reading.moments), ) entries.append(orphan) by_event[msg.event_id] = orphan @@ -560,6 +575,57 @@ def compile_entries(messages, readings, *, return entries +# A distilled entry shows at most this many quotes. +_MAX_MOMENTS = 3 + +# Entries below this length render in full; a gist would only repeat +# them. At or above it, the renderer prefers the distilled view. +DISTILL_MIN_WORDS = 120 + +_SENTENCE_END = re.compile(r"(?<=[.!?\u2026])\s+") + + +def _normalized(text: str) -> str: + """Text reduced to lowercase letters and digits, for comparison.""" + return re.sub(r"[^a-z0-9\u00c0-\u024f]+", "", text.lower()) + + +def verify_moments(body: str, claimed) -> list[str]: + """Return the claimed quotes that the body really contains. + + The model copies passages; this function checks them. A claimed + quote is accepted when it equals one sentence of the body, or a run + of consecutive sentences, compared without case and punctuation. + The returned text is the body's own text, not the model's copy, so + a quote on the page is an exact excerpt. Claims that match nothing + are dropped. + """ + sentences = [x.strip() for x in _SENTENCE_END.split(body) if x.strip()] + norms = [_normalized(x) for x in sentences] + kept: list[str] = [] + for claim in claimed or (): + want = _normalized(str(claim)) + if not want: + continue + found = None + for i in range(len(norms)): + joined = "" + for j in range(i, len(norms)): + joined += norms[j] + if joined == want: + found = " ".join(sentences[i:j + 1]) + break + if len(joined) > len(want): + break + if found: + break + if found and found not in kept: + kept.append(found) + if len(kept) >= _MAX_MOMENTS: + break + return kept + + def _joined_body(group) -> str: """The words of a group, with a split recording read back as one.""" parts = [m.body.strip() for m in group if m.body.strip()] @@ -611,6 +677,16 @@ def _kind_label(entry: Entry) -> str: return f"{noun}, {length}" if length else noun +def _distills(entry: Entry) -> bool: + """Whether the renderer shows the distilled view for this entry. + + Requires a gist and a long body. Without a gist the full text is + the only faithful rendering. Short entries are already the right + amount of detail. + """ + return bool(entry.gist) and len(entry.body.split()) >= DISTILL_MIN_WORDS + + def _permalink(room_id: str, event_id: str) -> str: return f"https://matrix.to/#/{room_id}/{event_id}" @@ -639,7 +715,18 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: "", ] - if entry.body.strip(): + if entry.body.strip() and _distills(entry): + # The distilled view for long recordings: one narrative line, + # verified quotes, and the full transcript in a folded block. + # verify_moments() guarantees each quote is an exact excerpt. + lines += [entry.gist, ""] + for moment in entry.moments: + lines += [f"> [!quote] {moment}", ""] + lines += ["> [!note]- Full transcript"] + lines += [f"> {line}" if line.strip() else ">" + for line in entry.body.strip().splitlines()] + lines += [""] + elif entry.body.strip(): lines += [entry.body.strip(), ""] elif entry.kind == "voice" and not entry.comments: # A gated recording: the transcript was unusable and the words diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 79b739c..c4b2e46 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -659,6 +659,68 @@ def test_an_untranscribable_recording_says_so(self): assert "This recording could not be transcribed." in page + def test_a_long_entry_renders_distilled_with_verified_quotes(self): + """Long recordings render as gist + quotes + folded transcript, + not as a wall of text.""" + body = " ".join(f"wort{i}." for i in range(130)) + \ + " Das ist der Satz der bleibt." + page = diary.render_month([diary.Entry( + on=date(2026, 9, 14), confidence="sent", basis="b", + kind="voice", sender="marge", body=body, + gist="Marge erzählt von einem langen Tag.", + moments=["Das ist der Satz der bleibt."])]) + + assert "Marge erzählt von einem langen Tag." in page + assert "> [!quote] Das ist der Satz der bleibt." in page + assert "> [!note]- Full transcript" in page + # The body appears only inside the folded block, quoted. + assert "\nwort0." not in page + + def test_a_short_entry_stays_verbatim_even_with_a_gist(self): + page = diary.render_month([diary.Entry( + on=date(2026, 9, 14), confidence="sent", basis="b", + kind="voice", sender="marge", body="Kurz und wichtig.", + gist="A gist that must not replace the words.")]) + + assert "Kurz und wichtig." in page + assert "A gist that must not replace the words." not in page + + +class TestVerifyMoments: + """A quote reaches the page only when the body really contains it. + The page shows the body's own text, not the model's copy.""" + + BODY = ("Hallo Bart, heute ist der sechzehnte März. Direktor " + "Skinner hat angerufen! Du hast der neuen Schülerin " + "geholfen. Das vergesse ich dir nicht.") + + def test_an_exact_sentence_is_kept(self): + assert diary.verify_moments( + self.BODY, ["Direktor Skinner hat angerufen!"]) == \ + ["Direktor Skinner hat angerufen!"] + + def test_case_and_punctuation_differences_still_match_the_body(self): + got = diary.verify_moments( + self.BODY, ["direktor skinner hat angerufen"]) + assert got == ["Direktor Skinner hat angerufen!"] + + def test_a_run_of_sentences_matches_as_one_quote(self): + got = diary.verify_moments( + self.BODY, + ["Du hast der neuen Schülerin geholfen. " + "Das vergesse ich dir nicht."]) + assert got == ["Du hast der neuen Schülerin geholfen. " + "Das vergesse ich dir nicht."] + + def test_an_invented_quote_is_dropped(self): + assert diary.verify_moments( + self.BODY, ["Bart hat die Schule angezündet."]) == [] + + def test_a_partial_sentence_is_dropped(self): + assert diary.verify_moments( + self.BODY, ["der neuen Schülerin"]) == [] + + def test_a_speaker_is_not_addressed_to_themselves(self): """A misread addressee must not become a dedication.""" page = diary.render_month([diary.Entry( From 3179e9b707a4a8f50522bfbcc98d67a7ec282ad6 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 16:23:41 +0200 Subject: [PATCH 27/43] feat(diary): --limit N processes only the newest N messages Bounded preview for dry runs: pagination stops early, so a limited run also skips the full-history fetch. --- stacklets/memory/bot/cli/diary.py | 23 ++++++++++++++++++----- stacklets/memory/bot/cli_entrypoint.py | 2 +- 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 3be894b..c555f44 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -21,6 +21,8 @@ stack memory diary --retranscribe decode all recordings again (after a whisper config or vocabulary change) + stack memory diary --limit 20 only the newest 20 messages + (bounded preview) WHY THE BURST WINDOW IS A KNOB Messages that synced late carry arrival timestamps, not recording @@ -119,11 +121,14 @@ async def _resolve_room(session, homeserver, token, room: str) -> str: return resp.json()["room_id"] -async def _history(session, homeserver, token, room_id: str) -> list[dict]: - """Every message event in the room, newest page first. +async def _history(session, homeserver, token, room_id: str, + limit: int = 0) -> list[dict]: + """Message events in the room, newest page first. - Paginates backwards until Synapse stops handing back a cursor. The - caller sorts; order here is only what the API gives us. + Paginates backwards until Synapse stops handing back a cursor. A + positive ``limit`` stops after that many events: the newest part + of the room, for a bounded preview run. The caller sorts; order + here is only what the API gives us. Counts out loud as it goes. A room with years in it takes many round trips before anything else can start, and a command that @@ -147,6 +152,8 @@ async def _history(session, homeserver, token, room_id: str) -> list[dict]: if chunk: _err(f" read {len(events)} events so far") cursor = payload.get("end") or "" + if limit and len(events) >= limit: + return events[:limit] if not chunk or not cursor: return events @@ -675,6 +682,11 @@ async def run(llm, argv: list[str]) -> int: except ValueError: _err("--burst-window wants a number of seconds") return 2 + try: + limit = int(_opt(argv, "--limit", "0")) + except ValueError: + _err("--limit wants a number of messages") + return 2 zone = _household_zone() readings_cache, summaries_cache = diary_store.open_stores() @@ -696,7 +708,8 @@ async def run(llm, argv: list[str]) -> int: try: token = await _admin_token(session, homeserver) room_id = await _resolve_room(session, homeserver, token, room_arg) - events = await _history(session, homeserver, token, room_id) + events = await _history(session, homeserver, token, room_id, + limit=limit) except (RuntimeError, httpx.HTTPError) as e: _err(str(e)) return 1 diff --git a/stacklets/memory/bot/cli_entrypoint.py b/stacklets/memory/bot/cli_entrypoint.py index 5967204..e76a4bc 100644 --- a/stacklets/memory/bot/cli_entrypoint.py +++ b/stacklets/memory/bot/cli_entrypoint.py @@ -17,7 +17,7 @@ words. Exit 1 means no keywords, which the host treats as "search it literally" rather than as a failure. - diary [] [--burst-window ] [--dry-run] [--force] [--retranscribe] + diary [] [--burst-window ] [--dry-run] [--force] [--retranscribe] [--limit ] Compile the memories room into the family diary. Walks the room's full history, transcribes every recording, recovers the date each one was made, and publishes month pages under the From 0f147234be7f746f5554075ae36638fba71aa568 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 16:47:47 +0200 Subject: [PATCH 28/43] fix(ai): load the whisper agent on every up, not only on config change stack down unloads the LaunchAgent and RunAtLoad only fires at login, so every down/up cycle left whisper dead until reboot (pre-existing; the cause of the Sept 12 outage). The reconcile now checks launchctl: unchanged config + loaded job = no-op; unloaded job = load without a rewrite; changed config = rewrite + reload. --- stacklets/ai/hooks/on_install.py | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/stacklets/ai/hooks/on_install.py b/stacklets/ai/hooks/on_install.py index 5268ca9..77e2ee9 100644 --- a/stacklets/ai/hooks/on_install.py +++ b/stacklets/ai/hooks/on_install.py @@ -313,14 +313,28 @@ def _setup_whisper_launchd(ctx, data_dir: Path, whisper_bin: Path, model_path: P f'\n' ) - if (plist_path.exists() and plist_path.read_text() == plist_content - and wrapper.exists() and wrapper.read_text() == wrapper_text): + unchanged = (plist_path.exists() + and plist_path.read_text() == plist_content + and wrapper.exists() + and wrapper.read_text() == wrapper_text) + try: + ctx.shell(f'launchctl list "{PLIST_LABEL}"') + loaded = True + except RuntimeError: + loaded = False + + # Unchanged and loaded: nothing to do, do not bounce the service. + # Unchanged but not loaded: `stack down` unloads the agent and + # RunAtLoad only fires at login, so every up must load it again. + # Changed: rewrite, then reload. + if unchanged and loaded: return - section("Whisper server", "launchd service") - wrapper.write_text(wrapper_text) - wrapper.chmod(0o755) - plist_path.write_text(plist_content) + if not unchanged: + section("Whisper server", "launchd service") + wrapper.write_text(wrapper_text) + wrapper.chmod(0o755) + plist_path.write_text(plist_content) ctx.step("Loading whisper-server into launchd...") try: From 061f7143fb6bceb32d69be2fd3c516ca23a74184 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 16:51:36 +0200 Subject: [PATCH 29/43] fix(diary): --limit takes a value; keep it out of the room positional --- stacklets/memory/bot/cli/diary.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index c555f44..632ffd4 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -639,7 +639,7 @@ def _household_zone(): # Flags that consume the token after them, so the room can be picked out # of the rest without mistaking a flag's value for it. -_TAKES_A_VALUE = ("--burst-window",) +_TAKES_A_VALUE = ("--burst-window", "--limit") def _opt(argv: list[str], flag: str, fallback: str) -> str: From d76d50af549c9a51cc8827772b5f5c83566da4ab Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 17:02:28 +0200 Subject: [PATCH 30/43] fix(diary): month summaries in the household language The summary prompt is English and named no target language, so the model answered in English. The prompt now states the language from the core LANGUAGE env (de/en), with a same-as-entries fallback. --- stacklets/memory/bot/cli/diary.py | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 632ffd4..ee32396 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -541,6 +541,7 @@ def _resolve_n(value, chunk): someone in the family would remember it later. Rules: +- Write in {language}. The diary belongs to a family that speaks it. - Use only what the entries say. Never add an event, a feeling, a place or an outcome that is not in them. - Keep every detail with the person the entry keeps it with. Do not move @@ -564,6 +565,17 @@ def _resolve_n(value, chunk): """ +# The prompt is English, so without a stated target language the model +# answers in English. The household language comes from the core env. +_LANGUAGE_NAMES = {"de": "German", "en": "English"} + + +def _household_language() -> str: + code = (os.environ.get("LANGUAGE") or "").strip().lower()[:2] + return _LANGUAGE_NAMES.get( + code, "the language the entries are written in") + + def _evidence(entries) -> str: """A month's entries as the summariser sees them. @@ -598,7 +610,7 @@ async def _summarise(entries, llm) -> str: introduction changes wording every night is not. """ month = entries[0].on.strftime("%B %Y") - prompt = _SUMMARY_PROMPT.format(month=month, evidence=_evidence(entries)) + prompt = _SUMMARY_PROMPT.format(language=_household_language(), month=month, evidence=_evidence(entries)) try: text = await llm.complete("writer", prompt, temperature=0, max_tokens=_SUMMARY_TOKENS, From 39e4e1eaa9287f2eedda174a1f60958442739494 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 17:17:02 +0200 Subject: [PATCH 31/43] feat(diary): safer person attribution in month summaries Summaries misattributed events because dialogue transcripts carry no speaker labels. Three measures, composed: - Mode-tiered evidence: a monologue line reads ' spoke' (one speaker, structurally known); a conversation line states that who said which line is unknown. Prompt forbids attributing statements inside a conversation to named people. - Reported-speech register: summaries write what people recorded and told, not bare facts. An attribution error becomes a misreport of speech, not a false claim about a person. - Unclear-word gate: low-confidence words from the transcript quality record are listed as unclear per entry; the prompt forbids building statements or attributions on them. --- stacklets/memory/bot/cli/diary.py | 42 ++++++++++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index ee32396..614d4e5 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -548,6 +548,15 @@ def _resolve_n(value, chunk): something one child did onto another child. - Keep the direction of what happened. If one person did something for, to, or about another, do not swap them round. +- Prefer reported speech: write what people recorded, told and + described ("Marge erzählt, dass ..."), not bare statements of fact. + These entries are people telling things, and the diary recalls the + telling. +- An entry marked as a conversation has no speaker labels in its text. + Name its participants and its topics. Never attribute a statement + inside it to a named person. +- A word listed as "unclear" was not heard clearly. Do not use it, and + do not attribute anything to a person through it. - Name people as the entries name them. - Report what the entries report, and no more. Do not frame the month as an occasion, and do not describe an event the entries only mention in @@ -576,6 +585,23 @@ def _household_language() -> str: code, "the language the entries are written in") +def _unclear_words(entry) -> list[str]: + """Words whisper did not hear clearly, for this entry's recordings. + + Read from the transcript records' quality data. The summariser is + told these words are unreliable, so it cannot hang an event or a + person on a misheard name. + """ + words: list[str] = [] + for eid in entry.event_ids: + record = voice.TRANSCRIPTS.read(eid) or {} + for w in (record.get("quality") or {}).get("low_words") or []: + token = (w.get("word") or "").strip() + if token and token not in words: + words.append(token) + return words[:5] + + def _evidence(entries) -> str: """A month's entries as the summariser sees them. @@ -594,10 +620,24 @@ def _evidence(entries) -> str: # the prose. Whoever a memo is spoken to is named in its words # anyway, where the model can read it as evidence. who = entry.sender.title() + # The mode separates safe attribution from unsafe: a monologue + # has one speaker (the sender); a conversation carries no + # speaker labels inside its text. + if entry.kind == "voice" and entry.mode == "dialogue": + role = (f"{who} recorded a conversation " + f"(who said which line is unknown)") + elif entry.kind == "voice": + role = f"{who} spoke" + else: + role = who body = entry.body.strip() or "(a photo, no caption)" for sender, text in entry.comments: body += f"\n {sender.title()} replied: {text.strip()}" - out.append(f"- [{when}] {who}: {body}") + line = f"- [{when}] {role}: {body}" + if unclear := _unclear_words(entry): + line += ("\n unclear words, not heard clearly: " + + ", ".join(unclear)) + out.append(line) return "\n".join(out) From 03be5c9322dbd56b03d7a3012a5a2871807e804c Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 17:28:07 +0200 Subject: [PATCH 32/43] feat(diary): render pages in the household language Rendering strings were hardcoded English and the gist prompt named no target language, so German diaries carried English meta lines and English gists. - diary.py gains a string table (en/de) covering every reader-facing string: kind labels, date basis, callouts, links, front page, month and weekday names. configure_language() selects once at startup from the core LANGUAGE env; English stays the default and the test baseline. - Read prompt states the gist language explicitly, same fix as the summaries. Whisper vocabulary priming is localized too. - 2 German rendering tests; 73 pass. --- stacklets/memory/bot/cli/diary.py | 11 +- stacklets/memory/bot/diary.py | 237 +++++++++++++++++++++------ tests/stacklets/test_memory_diary.py | 36 ++++ 3 files changed, 226 insertions(+), 58 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 614d4e5..1be0022 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -362,9 +362,9 @@ async def produce() -> dict: to it. Use this only when the message would make no sense on its own page. Otherwise null. -"gist": for a message longer than about 100 words: one sentence, in - the language of the message, saying what it is about and for whom. - Plain and specific, no marketing words. For shorter messages null. +"gist": for a message longer than about 100 words: one sentence in + {language}, saying what it is about and for whom. Plain and + specific, no marketing words. For shorter messages null. "moments": for a message longer than about 100 words: up to three short passages copied word-for-word from the message, the lines most @@ -466,7 +466,8 @@ def remember(event_id: str, row: dict) -> None: continue _err(f" reading slice {n} of {len(slices)}") - prompt = _READ_PROMPT.format(messages=_as_prompt(chunk)) + prompt = _READ_PROMPT.format( + language=_household_language(), messages=_as_prompt(chunk)) try: raw = await llm.complete( "classifier", prompt, json_mode=True, temperature=0, @@ -741,6 +742,8 @@ async def run(llm, argv: list[str]) -> int: return 2 zone = _household_zone() + # Pages render in the household language. Selected once per run. + diary.configure_language(os.environ.get("LANGUAGE", "")) readings_cache, summaries_cache = diary_store.open_stores() homeserver = os.environ.get("MATRIX_HOMESERVER", "").rstrip("/") if not homeserver: diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 653335e..a8759a7 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -40,6 +40,146 @@ DEFAULT_BURST_WINDOW_S = 120.0 +# ── Language ────────────────────────────────────────────────────────── +# +# Every string a family member reads on a page comes from this table. +# The compiler selects the household language once, at startup, via +# configure_language(). English is the default and the test baseline. + +_STRINGS = { + "en": { + "and": "and", + "months": ["January", "February", "March", "April", "May", + "June", "July", "August", "September", "October", + "November", "December"], + "days": ["Monday", "Tuesday", "Wednesday", "Thursday", + "Friday", "Saturday", "Sunday"], + "day_heading": "{day}, {dom} {month}", + "vocab_people": "The people in this family are {names}.", + "vocab_topics": "They often talk about {topics}.", + "basis_spoken": "dated from the spoken opening", + "basis_sent": "dated from when it was sent", + "basis_burst": ("arrived in a sync burst with no spoken date, " + "so this is the week it surfaced, not when it " + "happened"), + "kind_image": "Photo", "kind_video": "Video", + "kind_file": "File", "kind_text": "Written note", + "kind_attachment": "Attachment", + "kind_voice": "Voice note", "kind_dialogue": "Conversation", + "for": "for", + "when_unknown": "When this happened is not recoverable", + "untranscribable": "This recording could not be transcribed.", + "nothing_written": "Nothing was written alongside this one.", + "full_transcript": "Full transcript", + "replied": "{who} replied", + "link_voice": "Listen in the room", + "link_image": "See it in the room", + "link_video": "Watch it in the room", + "link_other": "Open in the room", + "no_entries": "No entries yet.", + "entry_one": "entry", "entry_many": "entries", + "month_one": "month", "month_many": "months", + "year_opening": "{count} this year", + "recorded_by": "recorded by", + "months_h": "Months", "years_h": "Years", + "diary_title": "Family Diary", + "index_intro": ( + "Everything the family has put in the memories room: voice " + "notes, photos, conversations someone hit record on. Words " + "shown as someone's own are word-for-word from the " + "recording. The text around them is the chronicle, and " + "every entry links back to the original in the room."), + "nothing_compiled": "Nothing has been compiled yet.", + "unrecovered_h": "Dates we could not recover", + "unrecovered_body": ( + "{count} arrived in a sync burst without a spoken date. " + "They are filed under the week they surfaced and marked on " + "their page. Saying the date aloud at the start of a " + "recording is what prevents this."), + "across": "across", + }, + "de": { + "and": "und", + "months": ["Januar", "Februar", "M\u00e4rz", "April", "Mai", + "Juni", "Juli", "August", "September", "Oktober", + "November", "Dezember"], + "days": ["Montag", "Dienstag", "Mittwoch", "Donnerstag", + "Freitag", "Samstag", "Sonntag"], + "day_heading": "{day}, {dom}. {month}", + "vocab_people": "Die Personen in dieser Familie sind {names}.", + "vocab_topics": "Sie sprechen oft \u00fcber {topics}.", + "basis_spoken": "datiert nach dem gesprochenen Datum", + "basis_sent": "datiert nach dem Sendezeitpunkt", + "basis_burst": ("kam in einem Sync-Schub ohne gesprochenes " + "Datum an; eingeordnet in der Woche des " + "Auftauchens, nicht des Geschehens"), + "kind_image": "Foto", "kind_video": "Video", + "kind_file": "Datei", "kind_text": "Notiz", + "kind_attachment": "Anhang", + "kind_voice": "Sprachnotiz", "kind_dialogue": "Gespr\u00e4ch", + "for": "f\u00fcr", + "when_unknown": ("Wann dies geschah, l\u00e4sst sich nicht " + "mehr feststellen"), + "untranscribable": ("Diese Aufnahme konnte nicht " + "transkribiert werden."), + "nothing_written": "Hierzu wurde nichts geschrieben.", + "full_transcript": "Vollst\u00e4ndiges Transkript", + "replied": "{who} antwortete", + "link_voice": "Im Chat anh\u00f6ren", + "link_image": "Im Chat ansehen", + "link_video": "Im Chat ansehen", + "link_other": "Im Chat \u00f6ffnen", + "no_entries": "Noch keine Eintr\u00e4ge.", + "entry_one": "Eintrag", "entry_many": "Eintr\u00e4ge", + "month_one": "Monat", "month_many": "Monaten", + "year_opening": "{count} in diesem Jahr", + "recorded_by": "aufgenommen von", + "months_h": "Monate", "years_h": "Jahre", + "diary_title": "Familientagebuch", + "index_intro": ( + "Alles, was die Familie im Erinnerungsraum festgehalten " + "hat: Sprachnotizen, Fotos, Gespr\u00e4che, die jemand " + "aufgenommen hat. W\u00f6rter, die als jemandes eigene " + "erscheinen, stammen Wort f\u00fcr Wort aus der Aufnahme. " + "Der Text darum herum ist die Chronik, und jeder Eintrag " + "verlinkt auf das Original im Chat."), + "nothing_compiled": "Noch nichts zusammengestellt.", + "unrecovered_h": "Nicht datierbare Eintr\u00e4ge", + "unrecovered_body": ( + "{count} kamen in einem Sync-Schub ohne gesprochenes Datum " + "an. Sie sind in der Woche ihres Auftauchens eingeordnet " + "und auf ihrer Seite markiert. Das Datum am Anfang einer " + "Aufnahme laut zu sagen verhindert das."), + "across": "in", + }, +} + +_L = _STRINGS["en"] + + +def configure_language(code: str) -> None: + """Select the render language. Unknown codes keep English.""" + global _L + _L = _STRINGS.get((code or "").strip().lower()[:2], _STRINGS["en"]) + + +def _month_name(on: date) -> str: + return _L["months"][on.month - 1] + + +def _month_year(on: date) -> str: + return f"{_month_name(on)} {on.year}" + + +def _day_heading(on: date) -> str: + return _L["day_heading"].format( + day=_L["days"][on.weekday()], dom=on.day, month=_month_name(on)) + + +def _counted(n: int, one: str, many: str) -> str: + return f"{n} {_L[one] if n == 1 else _L[many]}" + + # ── What the room gives us ──────────────────────────────────────────── @@ -152,11 +292,11 @@ def spoken_vocabulary(people, topics=()) -> str: subjects = [t.strip() for t in topics if t and t.strip()] parts = [] if names: - parts.append("The people in this family are " - + _and_list(_unique(names)) + ".") + parts.append(_L["vocab_people"].format( + names=_and_list(_unique(names)))) if subjects: - parts.append("They often talk about " - + _and_list(_unique(subjects)) + ".") + parts.append(_L["vocab_topics"].format( + topics=_and_list(_unique(subjects)))) return " ".join(parts) @@ -448,13 +588,10 @@ def date_for(msg: Message, reading: Reading) -> tuple[date, str, str]: """ spoken = parse_spoken_date(reading.spoken_date) if spoken is not None: - return spoken, "spoken", "dated from the spoken opening" + return spoken, "spoken", _L["basis_spoken"] if msg.burst: - return msg.sent_on, "uncertain", ( - "arrived in a sync burst with no spoken date, " - "so this is the week it surfaced, not when it happened" - ) - return msg.sent_on, "sent", "dated from when it was sent" + return msg.sent_on, "uncertain", _L["basis_burst"] + return msg.sent_on, "sent", _L["basis_sent"] # ── Step 4: compile ─────────────────────────────────────────────────── @@ -662,17 +799,12 @@ def _duration(ms: int | None) -> str: # What each kind of entry is called on the page. Every kind the # compiler accepts needs a name here: falling through to the internal # word prints "video" in a line of otherwise written English. -_KIND_NOUNS = { - "image": "Photo", "video": "Video", "file": "File", - "text": "Written note", -} - - def _kind_label(entry: Entry) -> str: if entry.kind == "voice": - noun = "Conversation" if entry.mode == "dialogue" else "Voice note" + noun = (_L["kind_dialogue"] if entry.mode == "dialogue" + else _L["kind_voice"]) else: - noun = _KIND_NOUNS.get(entry.kind, "Attachment") + noun = _L.get(f"kind_{entry.kind}", _L["kind_attachment"]) length = _duration(entry.duration_ms) return f"{noun}, {length}" if length else noun @@ -699,7 +831,7 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: # addressee resolves to its own sender is a misread, not a dedication. to = (entry.addressee or "").strip() if to and to.lower() != entry.sender.lower(): - heading = f"### {who} — for {to}" + heading = f"### {who} — {_L['for']} {to}" else: heading = f"### {who}" @@ -710,7 +842,7 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: if entry.confidence == "uncertain": lines += [ - "> [!warning] When this happened is not recoverable", + f"> [!warning] {_L['when_unknown']}", f"> {entry.basis.capitalize()}.", "", ] @@ -722,7 +854,7 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: lines += [entry.gist, ""] for moment in entry.moments: lines += [f"> [!quote] {moment}", ""] - lines += ["> [!note]- Full transcript"] + lines += [f"> [!note]- {_L['full_transcript']}"] lines += [f"> {line}" if line.strip() else ">" for line in entry.body.strip().splitlines()] lines += [""] @@ -732,20 +864,20 @@ def _entry_block(entry: Entry, *, room_id: str) -> str: # A gated recording: the transcript was unusable and the words # stay off the page. The sentence tells the reader this is # deliberate. The audio link below stays the way to hear it. - lines += ["This recording could not be transcribed.", ""] + lines += [_L["untranscribable"], ""] elif entry.kind in _UPLOADS and not entry.comments: - lines += ["Nothing was written alongside this one.", ""] + lines += [_L["nothing_written"], ""] for who_replied, text in entry.comments: - lines += [f"> [!quote] {who_replied.title()} replied", ] + lines += ['> [!quote] ' + _L['replied'].format(who=who_replied.title())] lines += [f"> {line}" for line in text.strip().splitlines()] lines.append("") if room_id and entry.event_ids: - label = {"voice": "Listen in the room", "image": "See it in the room", - "video": "Watch it in the room"} + label = {"voice": _L["link_voice"], "image": _L["link_image"], + "video": _L["link_video"]} lines.append( - f"[{label.get(entry.kind, 'Open in the room')}]" + f"[{label.get(entry.kind, _L['link_other'])}]" f"({_permalink(room_id, entry.event_ids[0])})" ) lines.append("") @@ -834,9 +966,9 @@ def render_month(entries, *, room_id: str = "", summary: str = "") -> str: a diary. """ if not entries: - return "No entries yet." + return _L["no_entries"] - lines = [f"# {entries[0].on.strftime('%B %Y')}", ""] + lines = [f"# {_month_year(entries[0].on)}", ""] if summary.strip(): lines += [summary.strip(), ""] @@ -848,7 +980,7 @@ def render_month(entries, *, room_id: str = "", summary: str = "") -> str: # date says so in its own block -- putting "week of" in the # heading would cast that doubt over every other entry # filed the same day. - lines += [f"## {entry.on.strftime('%A, %-d %B')}", ""] + lines += [f"## {_day_heading(entry.on)}", ""] lines += [_entry_block(entry, room_id=room_id), ""] return "\n".join(lines).rstrip() + "\n" @@ -866,16 +998,18 @@ def render_year(entries) -> str: n = len(entries) people = _recorded_by(entries) - opening = f"{n} {'entry' if n == 1 else 'entries'} this year" + opening = _L["year_opening"].format( + count=_counted(n, "entry_one", "entry_many")) if people: - opening += f", recorded by {_and_list(people)}" - lines += [opening + ".", "", "## Months", ""] + opening += f", {_L['recorded_by']} {_and_list(people)}" + lines += [opening + ".", "", f"## {_L['months_h']}", ""] for key, month in sorted(_by_month(entries).items()): - label = month[0].on.strftime("%B") + label = _month_name(month[0].on) count = len(month) lines.append( - f"- [{label}]({key}) — {count} {'entry' if count == 1 else 'entries'}") + f"- [{label}]({key}) — " + f"{_counted(count, 'entry_one', 'entry_many')}") lines.append("") return "\n".join(lines).rstrip() + "\n" @@ -884,39 +1018,34 @@ def render_year(entries) -> str: def render_index(entries) -> str: """The diary's front door: what it is, and a way into every year.""" lines = [ - "# Family Diary", + f"# {_L['diary_title']}", "", - "Everything the family has put in the memories room: voice notes, " - "photos, conversations someone hit record on. Words shown as " - "someone's own are word-for-word from the recording. The text " - "around them is the chronicle, and every entry links back to the " - "original in the room.", + _L["index_intro"], "", ] if not entries: - lines += ["Nothing has been compiled yet.", ""] + lines += [_L["nothing_compiled"], ""] return "\n".join(lines) - lines += ["## Years", ""] + lines += [f"## {_L['years_h']}", ""] for key, year in sorted(_by_year(entries).items(), reverse=True): count = len(year) months = len(_by_month(year)) lines.append( - f"- [{key}]({key}/about) — {count} " - f"{'entry' if count == 1 else 'entries'} across {months} " - f"{'month' if months == 1 else 'months'}") + f"- [{key}]({key}/about) — " + f"{_counted(count, 'entry_one', 'entry_many')} " + f"{_L['across']} " + f"{_counted(months, 'month_one', 'month_many')}") lines.append("") unsure = [e for e in entries if e.confidence == "uncertain"] if unsure: n = len(unsure) lines += [ - "## Dates we could not recover", + f"## {_L['unrecovered_h']}", "", - f"{n} {'entry' if n == 1 else 'entries'} arrived in a sync burst " - "without a spoken date. They are filed under the week they " - "surfaced and marked on their page. Saying the date aloud at the " - "start of a recording is what prevents this.", + _L["unrecovered_body"].format( + count=_counted(n, "entry_one", "entry_many")), "", ] @@ -926,7 +1055,7 @@ def render_index(entries) -> str: def _and_list(names: list[str]) -> str: if len(names) == 1: return names[0] - return ", ".join(names[:-1]) + f" and {names[-1]}" + return ", ".join(names[:-1]) + f" {_L['and']} {names[-1]}" def pages_for(entries, *, room_id: str = "", @@ -948,7 +1077,7 @@ def pages_for(entries, *, room_id: str = "", -- the bucket is named in config (`family`, `office`, a surname) and this module has no business knowing which. """ - out = [(f"{DIARY_DIR}/about.md", render_index(entries), "Family Diary")] + out = [(f"{DIARY_DIR}/about.md", render_index(entries), _L["diary_title"])] for year, in_year in sorted(_by_year(entries).items()): out.append(( f"{DIARY_DIR}/{year}/about.md", render_year(in_year), year, @@ -958,6 +1087,6 @@ def pages_for(entries, *, room_id: str = "", f"{DIARY_DIR}/{year}/{month}.md", render_month(in_month, room_id=room_id, summary=(summaries or {}).get(f"{year}-{month}", "")), - in_month[0].on.strftime("%B %Y"), + _month_year(in_month[0].on), )) return out diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index c4b2e46..67d3e47 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -971,3 +971,39 @@ def test_it_reads_as_speech_not_as_a_word_list(self): assert hint.endswith(".") assert "The people in this family are Homer." in hint + + +class TestGermanRendering: + """configure_language swaps every reader-facing string. English is + the module default; tests restore it.""" + + def test_a_german_page_has_no_english_strings(self): + diary.configure_language("de") + try: + page = diary.render_month([diary.Entry( + on=date(2026, 9, 14), confidence="sent", + basis=diary._L["basis_sent"], kind="voice", + sender="marge", body="", mode="dialogue", + event_ids=["$ev1"])], + room_id="!r:x") + finally: + diary.configure_language("en") + assert "# September 2026" in page + assert "Gespräch" in page + assert "Diese Aufnahme konnte nicht transkribiert werden." in page + assert "Im Chat anhören" in page + assert "datiert nach dem Sendezeitpunkt" in page + for english in ("Conversation", "Listen in the room", + "could not be transcribed"): + assert english not in page + + def test_german_month_names(self): + diary.configure_language("de") + try: + page = diary.render_month([diary.Entry( + on=date(2026, 3, 2), confidence="sent", basis="b", + kind="text", sender="lisa", body="Hallo.")]) + finally: + diary.configure_language("en") + assert "# März 2026" in page + assert "Montag, 2. März" in page From 5cebe67460a661c18ce196c7ee404a7bd28c0290 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 17:45:51 +0200 Subject: [PATCH 33/43] feat(diary): agreed tone for gists and summaries; letters post whole MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Gists and summaries read like topic inventories ('X und Y berichten über A, B, C und D'). - Gist prompt: sober-chronicle register (tone C) with anonymized few-shot examples (de/en), verbs over nominalizations, at most two moments, no topic lists. A conversation participant may be named only when the words address them by name. - Summary prompt: warm-chronicle register (tone A) with anonymized few-shot examples (de/en), same list ban. - Messages addressed to one person are letters: never distilled, posted 1:1 with the addressee in the heading; the gist is null for them by prompt and ignored by the renderer. - 1 new test; 74 pass. --- stacklets/memory/bot/cli/diary.py | 40 +++++++++++++++++++++++----- stacklets/memory/bot/diary.py | 8 ++++-- tests/stacklets/test_memory_diary.py | 18 +++++++++++++ 3 files changed, 57 insertions(+), 9 deletions(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 1be0022..2ea0e7f 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -362,9 +362,22 @@ async def produce() -> dict: to it. Use this only when the message would make no sense on its own page. Otherwise null. -"gist": for a message longer than about 100 words: one sentence in - {language}, saying what it is about and for whom. Plain and - specific, no marketing words. For shorter messages null. +"gist": for a message longer than about 100 words: one or two short + sentences in {language} saying what the message is about. Write + with verbs, as things that happened, never as a list of topics. + Name at most two moments and let the rest go; the full text sits + below the gist on the page. You may name a conversation participant + only when the words address them by name. No marketing words. Use + null for shorter messages, and for a message spoken to one person + as a personal message -- those are posted whole. + + Wanted register (invented examples, not this family): + "Anna erzählt mit einem der Kinder vom Tag am See und vom + Schuh, der im Wasser landete." + "Anna and one of the kids talk about the day at the lake and the + shoe that landed in the water." + Not wanted: "Anna und ein Kind berichten über einen Ausflug, + ein Picknick, einen verlorenen Schuh und das Wetter." "moments": for a message longer than about 100 words: up to three short passages copied word-for-word from the message, the lines most @@ -549,10 +562,23 @@ def _resolve_n(value, chunk): something one child did onto another child. - Keep the direction of what happened. If one person did something for, to, or about another, do not swap them round. -- Prefer reported speech: write what people recorded, told and - described ("Marge erzählt, dass ..."), not bare statements of fact. - These entries are people telling things, and the diary recalls the - telling. +- Write a warm chronicle in the third person: what the month had to + tell. Prefer reported speech -- what people recorded, told and + described -- over bare statements of fact. Use verbs, not + nominalizations. Never a list of topics: pick the moments that + matter and let the rest go. + + Wanted register (invented examples, not this family): + German: "Im September gab es viel zu erzählen: Anna hielt fest, + wie der erste Zahn endlich durchkam, und am Küchentisch wurde + der Sommer am See noch einmal lebendig -- samt dem Schuh, der im + Wasser blieb." + English: "September had a lot to tell: Anna recorded the first + tooth finally coming through, and around the kitchen table the + summer at the lake came back to life -- including the shoe that + stayed in the water." + Not wanted: "Im September berichteten Anna und Jonas über einen + Urlaub, einen ersten Zahn und ein Konzert." - An entry marked as a conversation has no speaker labels in its text. Name its participants and its topics. Never attribute a statement inside it to a named person. diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index a8759a7..b479a5b 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -814,9 +814,13 @@ def _distills(entry: Entry) -> bool: Requires a gist and a long body. Without a gist the full text is the only faithful rendering. Short entries are already the right - amount of detail. + amount of detail. A message addressed to one person never + distills, whatever its length: it is a personal message, and the + diary posts it whole, framed by the addressee in its heading. """ - return bool(entry.gist) and len(entry.body.split()) >= DISTILL_MIN_WORDS + return (bool(entry.gist) + and not (entry.addressee or "").strip() + and len(entry.body.split()) >= DISTILL_MIN_WORDS) def _permalink(room_id: str, event_id: str) -> str: diff --git a/tests/stacklets/test_memory_diary.py b/tests/stacklets/test_memory_diary.py index 67d3e47..26a277f 100644 --- a/tests/stacklets/test_memory_diary.py +++ b/tests/stacklets/test_memory_diary.py @@ -1007,3 +1007,21 @@ def test_german_month_names(self): diary.configure_language("en") assert "# März 2026" in page assert "Montag, 2. März" in page + + +class TestPersonalMessagesPostWhole: + """A message addressed to one person is a letter. It renders in + full whatever its length; distillation never touches it.""" + + def test_a_long_addressed_memo_is_not_distilled(self): + body = " ".join(f"wort{i}" for i in range(150)) + page = diary.render_month([diary.Entry( + on=date(2026, 9, 14), confidence="spoken", basis="b", + kind="voice", sender="marge", body=body, addressee="Bart", + gist="Should never render.", event_ids=["$ev1"])], + room_id="!r:x") + + assert "wort0" in page + assert "Should never render." not in page + assert "Full transcript" not in page + assert "— for Bart" in page From 55b948105ab95e3321fd839af8959fc23788251c Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 17:54:49 +0200 Subject: [PATCH 34/43] feat(diary): warmer front page, no word-for-word claim Title: Family Memories / Familienerinnerungen. The intro now says what the place is for and that every entry links to the original recording; the verbatim-mechanics sentence is gone. --- stacklets/memory/bot/diary.py | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index b479a5b..041a4c1 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -82,13 +82,12 @@ "year_opening": "{count} this year", "recorded_by": "recorded by", "months_h": "Months", "years_h": "Years", - "diary_title": "Family Diary", + "diary_title": "Family Memories", "index_intro": ( - "Everything the family has put in the memories room: voice " - "notes, photos, conversations someone hit record on. Words " - "shown as someone's own are word-for-word from the " - "recording. The text around them is the chronicle, and " - "every entry links back to the original in the room."), + "The things this family wanted to keep: voice notes, " + "photos, conversations someone hit record on. Every entry " + "links back to the original recording — there to be " + "listened to, today or in twenty years."), "nothing_compiled": "Nothing has been compiled yet.", "unrecovered_h": "Dates we could not recover", "unrecovered_body": ( @@ -135,14 +134,13 @@ "year_opening": "{count} in diesem Jahr", "recorded_by": "aufgenommen von", "months_h": "Monate", "years_h": "Jahre", - "diary_title": "Familientagebuch", + "diary_title": "Familienerinnerungen", "index_intro": ( - "Alles, was die Familie im Erinnerungsraum festgehalten " - "hat: Sprachnotizen, Fotos, Gespr\u00e4che, die jemand " - "aufgenommen hat. W\u00f6rter, die als jemandes eigene " - "erscheinen, stammen Wort f\u00fcr Wort aus der Aufnahme. " - "Der Text darum herum ist die Chronik, und jeder Eintrag " - "verlinkt auf das Original im Chat."), + "Was diese Familie festhalten wollte: Sprachnotizen, " + "Fotos, Gespr\u00e4che, die jemand aufgenommen hat. Jeder " + "Eintrag f\u00fchrt zur\u00fcck zur Originalaufnahme " + "\u2014 zum Nachh\u00f6ren, heute oder in zwanzig " + "Jahren."), "nothing_compiled": "Noch nichts zusammengestellt.", "unrecovered_h": "Nicht datierbare Eintr\u00e4ge", "unrecovered_body": ( From 74f2d5516a78c1da00c5ab35b1666c6a84b1a347 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 18:06:23 +0200 Subject: [PATCH 35/43] feat(ai): correct misheard names from the household ontology A summary presented a misheard name as a family member. The name sat in one transcript at confidence 0.19, flagged but not fixed; telling the summariser a word is unreliable did not stop it from using it. - New correct_pass in the transcript chain (gate -> polish -> correct -> structure): low-confidence words may be mapped to a household name from the wiki person pages, closed set only. The model can pick a known name or stay silent; it cannot introduce words. Replacements are recorded in the pass metadata. - Summary evidence redacts remaining unclear words to [unclear]; a word the model never sees cannot become a person. - Summary prompt is primed with the household people list and caps the paragraph at four short sentences, one moment each, no event chaining. - 4 new tests; 92 pass. --- lib/stack/ai/transcripts.py | 71 ++++++++++++++++++++++++-- stacklets/memory/bot/cli/diary.py | 66 ++++++++++++++++-------- tests/framework/test_ai_transcripts.py | 43 ++++++++++++++++ 3 files changed, 156 insertions(+), 24 deletions(-) diff --git a/lib/stack/ai/transcripts.py b/lib/stack/ai/transcripts.py index 7eab902..0267e20 100644 --- a/lib/stack/ai/transcripts.py +++ b/lib/stack/ai/transcripts.py @@ -22,9 +22,9 @@ No pass modifies `raw`. It holds the words as transcribed. Each consumer selects its own pass chain. Voice commands use -`Transcriber.transcribe` only. The diary uses gate and polish and -keeps the quality metrics. Text messages do not enter this module: -famstack stores them verbatim. +`Transcriber.transcribe` only. The diary uses gate, polish, correct, +and structure, and keeps the quality metrics. Text messages do not +enter this module: famstack stores them verbatim. """ from __future__ import annotations @@ -33,6 +33,7 @@ import base64 import json import os +import re import time from dataclasses import dataclass from pathlib import Path @@ -269,6 +270,70 @@ async def apply(record: dict) -> tuple[str, str, dict]: return TranscriptPass(name="structure", version=1, apply=apply) +_CORRECT_PROMPT = """\ +A speech transcript contains words the recognizer was not sure of. +The people in this household are: {names}. + +Unsure words: {words} + +Transcript: +{text} + +For each unsure word that is clearly a misheard household name in its +context, give the correct name. Only names from the list above are +allowed. When in doubt, omit the word. Reply with a JSON object that +maps the misheard word to the correct name, or {{}} when nothing is +clear. +""" + + +def correct_pass(llm, people: list[str]) -> TranscriptPass: + """Repair misheard household names, and only those. + + Candidates are the low-confidence words from the quality record; + a word whisper was sure of is never touched. Replacements come + from a closed set: the household's known names. The model can map + a candidate to a name or stay silent; it cannot introduce a word + of its own. Each replacement is recorded in the pass metadata. + + Measured basis: a misheard name in a real recording carried + probability 0.19 while clearly spoken words scored above 0.7. + """ + allowed = {name.strip() for name in people if name.strip()} + + async def apply(record: dict) -> tuple[str, str, dict]: + text = record.get("text") or "" + low = [w.get("word", "").strip() + for w in (record.get("quality") or {}).get("low_words") or []] + low = [w for w in low if w] + if not text.strip() or not low or not allowed: + return text, "skipped: no candidates", {} + raw = await llm.complete( + "transcript_cleanup", + _CORRECT_PROMPT.format(names=", ".join(sorted(allowed)), + words=", ".join(low), + text=text[:6000]), + json_mode=True, temperature=0, max_tokens=200, timeout=60.0) + try: + mapping = json.loads(raw or "{}") + except json.JSONDecodeError: + return text, "failed: unparseable answer", {} + applied: dict[str, str] = {} + for wrong, right in (mapping or {}).items(): + if (isinstance(wrong, str) and isinstance(right, str) + and wrong.strip() in low and right.strip() in allowed): + pattern = r"\b" + re.escape(wrong.strip()) + r"\b" + new_text = re.sub(pattern, right.strip(), text) + if new_text != text: + text = new_text + applied[wrong.strip()] = right.strip() + if not applied: + return text, "unchanged", {} + return text, "applied", {"replacements": applied} + + return TranscriptPass(name="correct", version=1, apply=apply) + + def polish_pass(llm) -> TranscriptPass: """Restore punctuation. Do not change words (see Transcriber.polish). diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 2ea0e7f..317b425 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -46,6 +46,7 @@ import asyncio import json import os +import re import sys from datetime import datetime, timezone from pathlib import Path @@ -186,17 +187,11 @@ def _length(ms: int | None) -> str: return f"{total // 60}:{total % 60:02d}" -def _household_vocabulary() -> str: - """The names and subjects this family uses, for whisper to decode against. +def _household_people() -> list[str]: + """The household's names, from the wiki's person pages. - People come from the wiki's person pages, which already carry the - household's own spelling of each name and any variants it uses. The - ontology's topics follow, because a family's proper nouns are not - only its people -- a campsite, a school, a pet -- and those mishear - just as readily. - - Best-effort: a vault that has not been generated yet simply yields - nothing, and transcription proceeds exactly as it did before. + The pages carry the family's own spelling of each name and its + variants. Best-effort: no vault means an empty list. """ people: list[str] = [] brain = Path(os.environ.get("BRAIN_REPO_DIR", "")) @@ -212,6 +207,20 @@ def _household_vocabulary() -> str: synonyms = front.get("synonyms") if isinstance(synonyms, list): people.extend(str(x) for x in synonyms) + return people + + +def _household_vocabulary() -> str: + """The names and subjects this family uses, for whisper to decode against. + + People come from the wiki's person pages. The ontology's topics + follow, because a family's proper nouns are not only its people -- + a campsite, a school, a pet -- and those mishear just as readily. + + Best-effort: a vault that has not been generated yet simply yields + nothing, and transcription proceeds exactly as it did before. + """ + people = _household_people() topics: list[str] = [] vault = Path(os.environ.get("MEMORY_VAULT_DIR", "")) @@ -251,6 +260,7 @@ def _frontmatter(path: Path) -> dict: async def _transcribe(message, *, session, homeserver, token, transcriber, llm, vocabulary: str = "", + people: list | None = None, retranscribe: bool = False) -> str: """The words of a recording, transcribed once and remembered. @@ -275,6 +285,7 @@ async def produce() -> dict: # raw text. Whisper does not run again. record = await transcripts.run_passes( record, [transcripts.gate_pass(), transcripts.polish_pass(llm), + transcripts.correct_pass(llm, people or []), transcripts.structure_pass()]) gate = next((p for p in record["passes"] if p["name"] == "gate"), {}) if str(gate.get("outcome", "")).startswith("blocked"): @@ -551,8 +562,16 @@ def _resolve_n(value, chunk): Write the opening paragraph of a family's diary page for {month}. Below are that month's entries, quoted exactly as the family recorded them. -Write two to four sentences recalling what happened that month, the way -someone in the family would remember it later. +The people in this family are: {people}. No other family members +exist. A name in the entries that is not in this list is a mishearing: +leave it out. + +Write two to four short sentences recalling that month, the way +someone in the family would remember it later. One moment per +sentence. Do not chain several events into one sentence with words +like "bevor", "während" or "und dann". The summary is not a complete +account: pick at most four moments and let the rest go -- the entries +below the paragraph carry everything else. Rules: - Write in {language}. The diary belongs to a family that speaks it. @@ -582,8 +601,8 @@ def _resolve_n(value, chunk): - An entry marked as a conversation has no speaker labels in its text. Name its participants and its topics. Never attribute a statement inside it to a named person. -- A word listed as "unclear" was not heard clearly. Do not use it, and - do not attribute anything to a person through it. +- [unclear] marks a word that was not heard clearly. Never guess what + it was, and never treat it as a person. - Name people as the entries name them. - Report what the entries report, and no more. Do not frame the month as an occasion, and do not describe an event the entries only mention in @@ -660,11 +679,12 @@ def _evidence(entries) -> str: body = entry.body.strip() or "(a photo, no caption)" for sender, text in entry.comments: body += f"\n {sender.title()} replied: {text.strip()}" - line = f"- [{when}] {role}: {body}" - if unclear := _unclear_words(entry): - line += ("\n unclear words, not heard clearly: " - + ", ".join(unclear)) - out.append(line) + # Unclear words are removed from what the summariser sees. + # Telling the model a word is unreliable does not stop it from + # using it; a word it never sees cannot become a person. + for token in _unclear_words(entry): + body = re.sub(rf"\\b{re.escape(token)}\\b", "[unclear]", body) + out.append(f"- [{when}] {role}: {body}") return "\n".join(out) @@ -677,7 +697,10 @@ async def _summarise(entries, llm) -> str: introduction changes wording every night is not. """ month = entries[0].on.strftime("%B %Y") - prompt = _SUMMARY_PROMPT.format(language=_household_language(), month=month, evidence=_evidence(entries)) + prompt = _SUMMARY_PROMPT.format( + language=_household_language(), month=month, + people=", ".join(_household_people()) or "unknown", + evidence=_evidence(entries)) try: text = await llm.complete("writer", prompt, temperature=0, max_tokens=_SUMMARY_TOKENS, @@ -805,6 +828,7 @@ async def run(llm, argv: list[str]) -> int: # words, and a recording that cannot be decoded should drop out # before the model is asked to interpret its filename. vocabulary = _household_vocabulary() + people = _household_people() if vocabulary: _err(f" decoding against: {vocabulary[:90]}...") @@ -829,7 +853,7 @@ async def run(llm, argv: list[str]) -> int: text = await _transcribe( msg, session=session, homeserver=homeserver, token=token, transcriber=transcriber, llm=llm, vocabulary=vocabulary, - retranscribe=retranscribe) + people=people, retranscribe=retranscribe) if not text.strip(): record = voice.TRANSCRIPTS.read(msg.event_id) or {} if (record.get("raw") or "").strip(): diff --git a/tests/framework/test_ai_transcripts.py b/tests/framework/test_ai_transcripts.py index cac51f4..739b69c 100644 --- a/tests/framework/test_ai_transcripts.py +++ b/tests/framework/test_ai_transcripts.py @@ -202,3 +202,46 @@ async def test_count_mismatch_skips(self): text, outcome, _ = await structure_pass().apply(record) assert text == "only three words" assert outcome.startswith("skipped") + + +class TestCorrectPass: + """Misheard household names are repaired from a closed set. A word + whisper was sure of is never touched; a name outside the household + never enters the text.""" + + RECORD = { + "text": "Anka geht heute in die Schule.", + "quality": {"low_words": [{"word": "Anka", "probability": 0.19}]}, + } + + async def test_replaces_a_low_confidence_name(self): + from stack.ai.transcripts import correct_pass + llm = _StubLLM(result='{"Anka": "Anna"}') + text, outcome, extra = await correct_pass( + llm, ["Anna", "Jonas"]).apply(dict(self.RECORD)) + assert text == "Anna geht heute in die Schule." + assert outcome == "applied" + assert extra == {"replacements": {"Anka": "Anna"}} + + async def test_a_name_outside_the_household_is_refused(self): + from stack.ai.transcripts import correct_pass + llm = _StubLLM(result='{"Anka": "Godzilla"}') + text, outcome, _ = await correct_pass( + llm, ["Anna"]).apply(dict(self.RECORD)) + assert text == self.RECORD["text"] + assert outcome == "unchanged" + + async def test_a_sure_word_is_never_touched(self): + from stack.ai.transcripts import correct_pass + llm = _StubLLM(result='{"Schule": "Anna"}') + text, outcome, _ = await correct_pass( + llm, ["Anna"]).apply(dict(self.RECORD)) + assert text == self.RECORD["text"] + + async def test_no_candidates_means_no_model_call(self): + from stack.ai.transcripts import correct_pass + llm = _StubLLM(result="{}") + record = {"text": "Alles klar.", "quality": {"low_words": []}} + text, outcome, _ = await correct_pass(llm, ["Anna"]).apply(record) + assert outcome.startswith("skipped") + assert llm.calls == [] From fc5fa02d7a0120ebe236629f49f28608373c74bd Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 19:14:56 +0200 Subject: [PATCH 36/43] fix(diary): quote callouts never contain words whisper flagged A moment quote showcased a misheard name. Claimed moments that contain any low-confidence word from the transcript's quality record are dropped before rendering; the full transcript keeps them, marked by the audio link beside it. --- stacklets/memory/bot/cli/diary.py | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 317b425..3291bb9 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -457,13 +457,28 @@ async def _read_room(messages, llm, cache=None): refers_to: dict[str, str] = {} known: set[str] = set() + def _sure_moments(event_id: str, moments) -> tuple: + """Drop claimed quotes that contain a word whisper was unsure + of. A quote is the most prominent text in an entry; it must + not showcase a word the recognizer flagged.""" + record = voice.TRANSCRIPTS.read(event_id) or {} + unsure = {(w.get("word") or "").strip().lower() + for w in (record.get("quality") or {}).get("low_words") or []} + unsure.discard("") + if not unsure: + return tuple(moments or ()) + return tuple( + m for m in moments or () + if not (set(re.findall(r"[\\w\\u00c0-\\u024f]+", str(m).lower())) + & unsure)) + def remember(event_id: str, row: dict) -> None: readings[event_id] = diary.Reading( mode=str(row.get("mode") or "monologue"), spoken_date=row.get("spoken_date") or None, addressee=row.get("addressee") or None, gist=row.get("gist") or None, - moments=tuple(row.get("moments") or ()), + moments=_sure_moments(event_id, row.get("moments")), ) if target := row.get("continues"): continues[event_id] = target From c2c6bc17aacaf8ef1e0b5fadd48e9bf55522df57 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 19:19:36 +0200 Subject: [PATCH 37/43] fix(diary): front-page intro addresses the family directly 'Eure Erinnerungen, festhalten in einer Chronik zum Nachlesen. ...' English matches: 'Your memories, kept in a chronicle to read back.' --- stacklets/memory/bot/diary.py | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/stacklets/memory/bot/diary.py b/stacklets/memory/bot/diary.py index 041a4c1..fbc2e3b 100644 --- a/stacklets/memory/bot/diary.py +++ b/stacklets/memory/bot/diary.py @@ -84,9 +84,9 @@ "months_h": "Months", "years_h": "Years", "diary_title": "Family Memories", "index_intro": ( - "The things this family wanted to keep: voice notes, " - "photos, conversations someone hit record on. Every entry " - "links back to the original recording — there to be " + "Your memories, kept in a chronicle to read back. Voice " + "notes, photos, conversations you recorded. Every entry " + "leads back to the original recording — there to be " "listened to, today or in twenty years."), "nothing_compiled": "Nothing has been compiled yet.", "unrecovered_h": "Dates we could not recover", @@ -136,11 +136,11 @@ "months_h": "Monate", "years_h": "Jahre", "diary_title": "Familienerinnerungen", "index_intro": ( - "Was diese Familie festhalten wollte: Sprachnotizen, " - "Fotos, Gespr\u00e4che, die jemand aufgenommen hat. Jeder " - "Eintrag f\u00fchrt zur\u00fcck zur Originalaufnahme " - "\u2014 zum Nachh\u00f6ren, heute oder in zwanzig " - "Jahren."), + "Eure Erinnerungen, festgehalten in einer Chronik zum " + "Nachlesen. Sprachnotizen, Fotos, Gespr\u00e4che, die ihr " + "aufgenommen habt. Jeder Eintrag f\u00fchrt zur\u00fcck " + "zur Originalaufnahme \u2014 zum Nachh\u00f6ren, heute " + "oder in zwanzig Jahren."), "nothing_compiled": "Noch nichts zusammengestellt.", "unrecovered_h": "Nicht datierbare Eintr\u00e4ge", "unrecovered_body": ( From b07fa596c0f3229706d6db12c1fd3fffe75155d9 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 21:06:16 +0200 Subject: [PATCH 38/43] docs(brain): memory pipeline architecture reference Components, data flow, the ontology layer, verification and capping rules with their measured bases, and an incident-derived pitfalls table. Replaces scattered knowledge from the Sept 2026 work. --- .../brain/memory-pipeline-architecture.md | 187 ++++++++++++++++++ 1 file changed, 187 insertions(+) create mode 100644 docs/design/brain/memory-pipeline-architecture.md diff --git a/docs/design/brain/memory-pipeline-architecture.md b/docs/design/brain/memory-pipeline-architecture.md new file mode 100644 index 0000000..da24f0e --- /dev/null +++ b/docs/design/brain/memory-pipeline-architecture.md @@ -0,0 +1,187 @@ +# Memory Pipeline Architecture + +Status: reference, September 2026. Describes the voice path from the +Matrix memories room to the published diary, the ontology layer, and +the mitigations this system needs because it runs on local models. + +## Context and constraints + +The pipeline runs on one Apple Silicon host (M1 Max, 64 GB). Models: +Qwen3.6-35B-A3B 4-bit on oMLX for text, whisper.cpp large-v3-turbo +for speech. Both are small compared to hosted models and show error +classes that hosted APIs rarely surface. The output is a family +archive: pages are read years later, by people who were in the +recordings. An error in a published page misstates someone's life. + +Consequences for the design: + +- Generated text is verified mechanically or constrained to closed + answer sets. Prompt instructions alone did not prevent any of the + failures listed under Pitfalls. +- Every model call has an output token cap derived from a measured + ratio. Uncapped local calls monopolize the single GPU. +- All expensive results are cached, keyed by content, so a full + recompile is affordable and deterministic. + +## Component overview + +``` +Matrix memories room (source of truth, audio = archival original) + │ + ▼ + transcription (whisper.cpp, configured flags, quality capture) + │ + ▼ + transcript store ~/famstack-data/core/transcripts/, 1 JSON/event + │ {raw, text, quality, passes[]} + ▼ + pass chain gate → polish → correct → structure + │ (lib/stack/ai/transcripts.py, versioned) + ▼ + reading facts per message: date, mode, addressee, + │ fragment links, gist, candidate quotes + ▼ + compile date precedence, fragment joins, reply/caption + │ attachment, entry assembly + ▼ + render language table, tiered entry formats + │ + ▼ + wiki (Quartz) pages in memory/brain, committed by curator +``` + +The room is append-only and the compiler is a fold over the full +history. There is no watermark: a reply or an edit can land on a +year-old entry, and only a full pass attaches it. Caches make the +fold cheap; they store cost, not position. + +## Transcription + +whisper.cpp flags, set in the LaunchAgent and reconciled on every +`stack up ai`: + +| Flag | Value | Reason | +|---|---|---| +| `--language` | from `[core].language` | auto-detection fails on silence and noise; one incident recording of infant sounds was transcribed as CJK text | +| `--max-context` | 0 | a hallucinated segment otherwise seeds the next segment | +| `--suppress-nst` | on | drops non-speech tokens | + +Each flag was tested against the three recordings that produced the +2026-09-14 incident. Each flag fixed loops the other two did not; +only the combination produced zero repetition on all three. + +Transcription requests use `verbose_json` with word timestamps. The +stored quality record keeps per-segment `avg_logprob`, +`no_speech_prob`, `temperature`, a word count, and every word below +0.5 confidence. whisper computes these values on every call; the +plain `json` format discards them. Voice commands and chat notes use +the plain format and skip all quality machinery — the cost is only +paid where an archive is built. + +## Transcript store and pass framework + +One record per Matrix event, atomic writes, single-flight production. +Record fields: + +- `raw` — whisper output, never modified by any pass +- `text` — current output of the pass chain +- `quality` — segment metrics and low-confidence words +- `passes[]` — `{name, version, outcome, model?, replacements?}` + +The pass list makes model upgrades incremental: increasing a pass +version marks affected records stale (`stale_passes`), and a sweep +re-runs that single pass on stored raw text without re-running +whisper. `--retranscribe` exists for the case where whisper itself +changed. + +| Pass | Function | Failure class covered | +|---|---|---| +| gate | empty the text when the transcript is hallucinated | repetition loops (measured 5–147 repeats of one 6-gram vs 1–2 in real speech); majority of segments failing whisper's own thresholds (avg_logprob < −1.0, no_speech_prob > 0.6) | +| polish | restore punctuation; word sequence verified unchanged | unreadable single-block output | +| correct | map low-confidence words to household names, closed set | misheard names (measured: p=0.19 on the one confirmed case) | +| structure | paragraph breaks at ≥1.5 s segment pauses, from word counts | wall-of-text rendering; no model call | + +The gate keeps two independent signals because the failure classes +are disjoint: a repetition loop is a high-confidence failure the +logprob check does not see, and mumble is a low-confidence failure +the repetition check does not see. + +## Ontology layer + +Sources: person pages in the wiki (canonical spelling plus synonyms) +and `ontology.toml` topics. Consumers: + +| Consumer | Use | Effect | +|---|---|---| +| whisper priming | names and topics as decoder prompt | fewer mishearings at the source | +| correct pass | closed replacement set | a repaired word is always a real name | +| summary prompt | list of family members | a name outside the list is treated as a mishearing and kept off the page | +| archivist (documents domain) | tags and correspondents | same principle, pre-existing | + +The pattern in all four: the ontology converts an open generation +problem into selection from a known set. Selection is the reliable +operation at this model size. + +## Reading, compilation, attribution + +The reading returns facts per message, JSON, temperature 0: spoken +date, mode (monologue/dialogue/note), addressee, fragment links, +gist, candidate quotes. Rules with rationale: + +- **Quotes are verified.** A candidate quote renders only if it + matches one sentence or a consecutive run in the transcript + (case/punctuation-insensitive). The page shows the transcript's own + text. Quotes containing any low-confidence word are dropped. +- **Date precedence:** spoken date > live timestamp > unrecoverable. + Matrix records only server receipt time; a synced message can be + days off. Sync-burst messages without a spoken date are filed under + the week they surfaced and labeled. +- **Attribution is limited to structural facts.** Sender and spoken + addressee are known. Line-level attribution inside a conversation + is unknown until diarization exists; summaries may name a + conversation's participants and nothing finer. +- **Fragment joins are content decisions.** Three uploads in one + second are usually three memos; a join requires one message to end + mid-sentence and the next to continue it. Timing alone misfiled + ordinary evenings as sync bursts before this rule. + +Summary input is preprocessed: words the pipeline knows are unclear +are replaced with `[unclear]` before the model sees them. This +replaced an instruction ("do not use unreliable words"), which the +model had ignored. + +## Rendering + +- Entries under 120 words render verbatim. +- Longer recordings render as gist, verified quotes, folded full + transcript, audio link. +- Messages addressed to one person render whole at any length. +- Recordings whose transcript the gate emptied render with a fixed + explanatory sentence and the audio link. +- All reader-facing strings, month and weekday names come from a + language table (`en`/`de`), selected once per run from config. + Prompts that produce family-facing text name the target language + explicitly; a model otherwise answers in the prompt's language. + +## Pitfalls (incident-derived) + +| Incident | Cause | Mitigation | +|---|---|---| +| 100k-token generation, 25 min GPU monopoly (2026-09-14) | polish is an echo task; a looping transcript has no natural end; no output cap; server default max_tokens was 128000 | gate before any echo task; caps at input size + 10% (measured output ratio 0.98–1.01); server default lowered | +| misheard name presented as a family member | whisper flagged the word (p=0.19) but the summary prompt's warning was ignored | correction pass (closed set), evidence redaction, ontology priming, quote filter | +| English output in a German diary | prompts are English; models answer in the prompt language | target language stated in every prompt; rendered strings from the language table | +| whisper dead after every `stack down ai` / `up ai` | stop hook unloads the LaunchAgent; nothing on the up path loaded it; `RunAtLoad` fires only at login | on_start reconciles the agent by content and loads it when absent | +| undatable memories | Matrix has no compose-time field; offline recordings carry sync time | spoken-date extraction; honest "unrecoverable" state; recording habit: say the date aloud | +| stale caches serving old wording | summary cache keyed by month digest; prompt changes do not change the digest | `--force` re-reads; pass versioning covers the transcript side; prompt-affected caches need a manual force after prompt changes | + +## Open items + +- Diarization for conversations (would upgrade attribution from + participant-level to line-level; mode labels already mark the + affected entries). +- Episode grouping: a vacation spanning many entries currently + renders as independent entries plus one month summary. +- Retranscription sweeps driven by `stale_passes` are manual; no + scheduled job exists. +- Prompt changes do not invalidate reading/summary caches + automatically (see Pitfalls, last row). From 4c5e755c43604dbf1ed5c52e3de19d7316dc0294 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 21:12:05 +0200 Subject: [PATCH 39/43] feat(ai): prompt fingerprints invalidate caches on prompt changes Prompt and pass-parameter edits did not reach cached artifacts; regeneration required a manual --force. - TranscriptPass carries a fingerprint (hash of its prompt or parameters); the pass trail stores it; stale_passes compares version and fingerprint. The correct pass includes the household names, so an ontology change re-evaluates old refusals. - The diary re-runs stale passes on cached records from raw text during any compile; whisper does not re-run. - ReadingStore and SummaryStore stamp each entry with the hash of the prompt that produced it; a mismatch is a cache miss. - A stack version hash was considered and rejected: every commit would invalidate everything. Fingerprints scope invalidation to the artifacts a change actually affects. - 5 new tests; 190 pass. --- .../brain/memory-pipeline-architecture.md | 10 ++--- lib/stack/ai/transcripts.py | 42 ++++++++++++++++--- stacklets/memory/bot/cli/diary.py | 38 ++++++++++++----- stacklets/memory/bot/diary_store.py | 32 +++++++++++--- tests/framework/test_ai_transcripts.py | 35 ++++++++++++++-- tests/stacklets/test_diary_store.py | 38 +++++++++++++++++ 6 files changed, 165 insertions(+), 30 deletions(-) create mode 100644 tests/stacklets/test_diary_store.py diff --git a/docs/design/brain/memory-pipeline-architecture.md b/docs/design/brain/memory-pipeline-architecture.md index da24f0e..995afbf 100644 --- a/docs/design/brain/memory-pipeline-architecture.md +++ b/docs/design/brain/memory-pipeline-architecture.md @@ -172,7 +172,7 @@ model had ignored. | English output in a German diary | prompts are English; models answer in the prompt language | target language stated in every prompt; rendered strings from the language table | | whisper dead after every `stack down ai` / `up ai` | stop hook unloads the LaunchAgent; nothing on the up path loaded it; `RunAtLoad` fires only at login | on_start reconciles the agent by content and loads it when absent | | undatable memories | Matrix has no compose-time field; offline recordings carry sync time | spoken-date extraction; honest "unrecoverable" state; recording habit: say the date aloud | -| stale caches serving old wording | summary cache keyed by month digest; prompt changes do not change the digest | `--force` re-reads; pass versioning covers the transcript side; prompt-affected caches need a manual force after prompt changes | +| stale caches serving old wording | caches were keyed by content only; prompt changes did not change the keys | every cached artifact stores a fingerprint (hash) of the prompt or pass parameters that produced it; a prompt edit invalidates exactly the affected artifacts on the next compile | ## Open items @@ -181,7 +181,7 @@ model had ignored. affected entries). - Episode grouping: a vacation spanning many entries currently renders as independent entries plus one month summary. -- Retranscription sweeps driven by `stale_passes` are manual; no - scheduled job exists. -- Prompt changes do not invalidate reading/summary caches - automatically (see Pitfalls, last row). +- Retranscription (`--retranscribe`) is manual; it is needed only + when whisper's configuration or vocabulary changes. Pass and prompt + changes regenerate automatically via fingerprints during any + compile, including the nightly one. diff --git a/lib/stack/ai/transcripts.py b/lib/stack/ai/transcripts.py index 0267e20..42c0023 100644 --- a/lib/stack/ai/transcripts.py +++ b/lib/stack/ai/transcripts.py @@ -165,6 +165,10 @@ class TranscriptPass: name: str version: int apply: Callable[[dict], Awaitable[tuple[str, str, dict]]] + # Hash of the prompt or parameters that shape this pass's output. + # A changed fingerprint marks stored results stale, so a prompt + # edit regenerates exactly the artifacts it produced. + fingerprint: str = "" async def run_passes(record: dict, passes: list[TranscriptPass]) -> dict: @@ -184,6 +188,7 @@ async def run_passes(record: dict, passes: list[TranscriptPass]) -> dict: logger.warning("[transcripts] pass {} failed: {}", p.name, e) trail = [e for e in trail if e.get("name") != p.name] trail.append({"name": p.name, "version": p.version, + "fingerprint": p.fingerprint, "outcome": outcome, **extra}) record["passes"] = trail return record @@ -196,9 +201,17 @@ def stale_passes(record: dict, passes: list[TranscriptPass]) -> list[TranscriptP invalidated. The input is the cached raw text. Whisper does not run again. """ - ran = {e.get("name"): e.get("version") + ran = {e.get("name"): (e.get("version"), e.get("fingerprint", "")) for e in (record.get("passes") or [])} - return [p for p in passes if ran.get(p.name) != p.version] + return [p for p in passes + if ran.get(p.name) != (p.version, p.fingerprint)] + + +def fingerprint(*parts) -> str: + """A short stable hash of the values that shape a pass's output.""" + import hashlib + joined = "\x1f".join(str(p) for p in parts) + return hashlib.sha256(joined.encode("utf-8")).hexdigest()[:12] def gate_pass() -> TranscriptPass: @@ -220,7 +233,13 @@ async def apply(record: dict) -> tuple[str, str, dict]: return "", f"blocked: {why}", {} return record.get("text") or raw, "clean", {} - return TranscriptPass(name="gate", version=1, apply=apply) + from . import client as _c + return TranscriptPass( + name="gate", version=1, apply=apply, + fingerprint=fingerprint( + _c._DEGENERATE_REPEATS, _c._DEGENERATE_CJK_FRACTION, + _c._QUALITY_LOGPROB_FLOOR, _c._QUALITY_NO_SPEECH, + _c._QUALITY_POOR_FRACTION)) def structure_pass(min_pause_s: float = 1.5, @@ -267,7 +286,9 @@ async def apply(record: dict) -> tuple[str, str, dict]: outcome = "applied" if len(paragraphs) > 1 else "unchanged" return structured, outcome, {} - return TranscriptPass(name="structure", version=1, apply=apply) + return TranscriptPass( + name="structure", version=1, apply=apply, + fingerprint=fingerprint(min_pause_s, min_words)) _CORRECT_PROMPT = """\ @@ -331,7 +352,11 @@ async def apply(record: dict) -> tuple[str, str, dict]: return text, "unchanged", {} return text, "applied", {"replacements": applied} - return TranscriptPass(name="correct", version=1, apply=apply) + # The people list is part of the fingerprint: a new household + # member makes old refusals worth another look. + return TranscriptPass( + name="correct", version=1, apply=apply, + fingerprint=fingerprint(_CORRECT_PROMPT, *sorted(allowed))) def polish_pass(llm) -> TranscriptPass: @@ -354,4 +379,9 @@ async def apply(record: dict) -> tuple[str, str, dict]: extra = {} return polished, outcome, extra - return TranscriptPass(name="polish", version=1, apply=apply) + from . import client as _c + return TranscriptPass( + name="polish", version=1, apply=apply, + fingerprint=fingerprint(_c._CLEANUP_PROMPT, + _c._CLEANUP_CHARS_PER_TOKEN, + _c._CLEANUP_HEADROOM)) diff --git a/stacklets/memory/bot/cli/diary.py b/stacklets/memory/bot/cli/diary.py index 3291bb9..c7510a1 100644 --- a/stacklets/memory/bot/cli/diary.py +++ b/stacklets/memory/bot/cli/diary.py @@ -258,6 +258,12 @@ def _frontmatter(path: Path) -> dict: return loaded if isinstance(loaded, dict) else {} +def _pass_chain(llm, people: list | None): + return [transcripts.gate_pass(), transcripts.polish_pass(llm), + transcripts.correct_pass(llm, people or []), + transcripts.structure_pass()] + + async def _transcribe(message, *, session, homeserver, token, transcriber, llm, vocabulary: str = "", people: list | None = None, @@ -269,6 +275,8 @@ async def _transcribe(message, *, session, homeserver, token, at all. That is the whole reason a backfill over a full room is affordable. """ + chain = _pass_chain(llm, people) + async def produce() -> dict: audio = await _download(session, homeserver, token, message.url or "") if not audio: @@ -283,10 +291,7 @@ async def produce() -> dict: # restores punctuation. The record lists each pass, so a # better future model can run one pass again on the cached # raw text. Whisper does not run again. - record = await transcripts.run_passes( - record, [transcripts.gate_pass(), transcripts.polish_pass(llm), - transcripts.correct_pass(llm, people or []), - transcripts.structure_pass()]) + record = await transcripts.run_passes(record, chain) gate = next((p for p in record["passes"] if p["name"] == "gate"), {}) if str(gate.get("outcome", "")).startswith("blocked"): _err(f" transcript of {message.event_id} unusable " @@ -294,8 +299,20 @@ async def produce() -> dict: return record try: - return (await voice.TRANSCRIPTS.run( - message.event_id, produce, force=retranscribe))["text"] + record = await voice.TRANSCRIPTS.run( + message.event_id, produce, force=retranscribe) + # A cached record skipped produce() and with it the passes. A + # changed pass fingerprint (prompt edit, new household member, + # version bump) re-runs the whole chain from raw text. Whisper + # does not run again. + if stale := transcripts.stale_passes(record, chain): + _err(f" passes changed for {message.event_id}: " + + ", ".join(p.name for p in stale)) + record = dict(record) + record["text"] = record.get("raw") or "" + record = await transcripts.run_passes(record, chain) + voice.TRANSCRIPTS.write(message.event_id, record) + return record["text"] except LLMError as e: _err(f" could not transcribe {message.event_id}: {e}") return "" @@ -488,10 +505,7 @@ def remember(event_id: str, row: dict) -> None: if cache is not None: for msg in messages: - stored = cache.get(msg.event_id, msg.body) - # Readings from before distillation carry no "moments" key. - # Treat them as absent so the message is read again once. - if stored is not None and "moments" in stored: + if (stored := cache.get(msg.event_id, msg.body)) is not None: remember(msg.event_id, stored) if known: _err(f" {len(known)} message(s) already read, " @@ -808,7 +822,9 @@ async def run(llm, argv: list[str]) -> int: zone = _household_zone() # Pages render in the household language. Selected once per run. diary.configure_language(os.environ.get("LANGUAGE", "")) - readings_cache, summaries_cache = diary_store.open_stores() + readings_cache, summaries_cache = diary_store.open_stores( + reading_fingerprint=transcripts.fingerprint(_READ_PROMPT), + summary_fingerprint=transcripts.fingerprint(_SUMMARY_PROMPT)) homeserver = os.environ.get("MATRIX_HOMESERVER", "").rstrip("/") if not homeserver: _err("MATRIX_HOMESERVER not set — is core up?") diff --git a/stacklets/memory/bot/diary_store.py b/stacklets/memory/bot/diary_store.py index 1a71fc0..c12c2f6 100644 --- a/stacklets/memory/bot/diary_store.py +++ b/stacklets/memory/bot/diary_store.py @@ -104,16 +104,26 @@ class ReadingStore(JsonStore): section = "readings" + def __init__(self, path, fingerprint: str = ""): + super().__init__(path) + # Hash of the reading prompt. A prompt edit changes it, every + # stored reading misses once, and the next compile re-reads + # the room under the new prompt. No manual invalidation. + self.fingerprint = fingerprint + def get(self, event_id: str, body: str = "") -> dict | None: found = self._items.get(event_id) if not isinstance(found, dict): return None if found.get("said") != _said(body): return None + if found.get("prompt", "") != self.fingerprint: + return None return found def put(self, event_id: str, reading: dict, body: str = "") -> None: - self._items[event_id] = {**reading, "said": _said(body)} + self._items[event_id] = {**reading, "said": _said(body), + "prompt": self.fingerprint} def _said(body: str) -> str: @@ -134,19 +144,31 @@ class SummaryStore(JsonStore): section = "summaries" + def __init__(self, path, fingerprint: str = ""): + super().__init__(path) + # Same mechanism as ReadingStore: the summary prompt's hash. + self.fingerprint = fingerprint + def get(self, month: str, digest: str) -> str: found = self._items.get(month) if not isinstance(found, dict) or found.get("digest") != digest: return "" + if found.get("prompt", "") != self.fingerprint: + return "" text = found.get("text") return text if isinstance(text, str) else "" def put(self, month: str, digest: str, text: str) -> None: - self._items[month] = {"digest": digest, "text": text} + self._items[month] = {"digest": digest, "text": text, + "prompt": self.fingerprint} -def open_stores(directory: Path | None = None): +def open_stores(directory: Path | None = None, *, + reading_fingerprint: str = "", + summary_fingerprint: str = ""): """Both caches, loaded. Missing files are simply empty ones.""" root = Path(directory) if directory else state_dir() - return (ReadingStore(root / "readings.json").load(), - SummaryStore(root / "summaries.json").load()) + return (ReadingStore(root / "readings.json", + reading_fingerprint).load(), + SummaryStore(root / "summaries.json", + summary_fingerprint).load()) diff --git a/tests/framework/test_ai_transcripts.py b/tests/framework/test_ai_transcripts.py index 739b69c..f82f9d8 100644 --- a/tests/framework/test_ai_transcripts.py +++ b/tests/framework/test_ai_transcripts.py @@ -79,8 +79,8 @@ async def upper(record): assert record["text"] == "HI" assert record["passes"] == [ - {"name": "upper", "version": 2, "outcome": "applied", - "model": "m1"}] + {"name": "upper", "version": 2, "fingerprint": "", + "outcome": "applied", "model": "m1"}] assert record["raw"] == "hi" async def test_a_failing_pass_keeps_the_text_and_the_chain(self): @@ -109,7 +109,8 @@ async def noop(record): "outcome": "clean"}]} record = await run_passes(record, [TranscriptPass("gate", 2, noop)]) assert record["passes"] == [ - {"name": "gate", "version": 2, "outcome": "clean"}] + {"name": "gate", "version": 2, "fingerprint": "", + "outcome": "clean"}] class TestStalePasses: @@ -245,3 +246,31 @@ async def test_no_candidates_means_no_model_call(self): text, outcome, _ = await correct_pass(llm, ["Anna"]).apply(record) assert outcome.startswith("skipped") assert llm.calls == [] + + +class TestFingerprints: + """A pass result is stale when the version or the fingerprint of + the pass changed. A prompt edit changes the fingerprint, so the + affected artifacts regenerate without manual invalidation.""" + + @staticmethod + def _pass(fp): + async def noop(record): + return record.get("text", ""), "clean", {} + return TranscriptPass("gate", 1, noop, fingerprint=fp) + + async def test_the_trail_records_the_fingerprint(self): + record = await run_passes({"raw": "hi", "text": "hi"}, + [self._pass("abc")]) + assert record["passes"][0]["fingerprint"] == "abc" + + def test_a_changed_fingerprint_is_stale(self): + record = {"passes": [{"name": "gate", "version": 1, + "fingerprint": "old"}]} + assert stale_passes(record, [self._pass("new")]) != [] + assert stale_passes(record, [self._pass("old")]) == [] + + def test_fingerprint_is_stable_and_input_sensitive(self): + from stack.ai.transcripts import fingerprint + assert fingerprint("a", 1) == fingerprint("a", 1) + assert fingerprint("a", 1) != fingerprint("a", 2) diff --git a/tests/stacklets/test_diary_store.py b/tests/stacklets/test_diary_store.py new file mode 100644 index 0000000..42a22b5 --- /dev/null +++ b/tests/stacklets/test_diary_store.py @@ -0,0 +1,38 @@ +"""Prompt fingerprints on the diary caches. A stored artifact is +valid only under the prompt that produced it.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent.parent.parent + / "stacklets" / "memory" / "bot")) + +import diary_store # noqa: E402 + + +class TestReadingStorePromptFingerprint: + def test_a_reading_from_another_prompt_misses(self, tmp_path): + store = diary_store.ReadingStore(tmp_path / "r.json", "prompt-v1") + store.put("$ev", {"mode": "monologue"}, "hello") + store.save() + + same = diary_store.ReadingStore(tmp_path / "r.json", "prompt-v1").load() + assert same.get("$ev", "hello") is not None + + changed = diary_store.ReadingStore(tmp_path / "r.json", "prompt-v2").load() + assert changed.get("$ev", "hello") is None + + +class TestSummaryStorePromptFingerprint: + def test_a_summary_from_another_prompt_misses(self, tmp_path): + store = diary_store.SummaryStore(tmp_path / "s.json", "prompt-v1") + store.put("2026-09", "digest", "Ein Monat.") + store.save() + + same = diary_store.SummaryStore(tmp_path / "s.json", "prompt-v1").load() + assert same.get("2026-09", "digest") == "Ein Monat." + + changed = diary_store.SummaryStore(tmp_path / "s.json", "prompt-v2").load() + assert changed.get("2026-09", "digest") == "" From c4064de782636f55a6c717885f30de5ee2e5c424 Mon Sep 17 00:00:00 2001 From: Arthur Date: Mon, 14 Sep 2026 21:35:22 +0200 Subject: [PATCH 40/43] docs(brain): design the wiki media follow-up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Inline images and an audio player per recording on diary pages. - Decided: compile-time export of media to plain dated files under {data_dir}/memory/media — originals, an m4a transcode for Safari (Ogg/Opus), page-weight thumbnails. The archive stays readable without famstack or Synapse. - Open: serving via a static mount or a thin filesystem proxy with logical URLs. - Rejected: view-time serving from Synapse's authenticated media API (1.160); it keeps the archive dependent on a running homeserver. --- .../brain/memory-pipeline-architecture.md | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/design/brain/memory-pipeline-architecture.md b/docs/design/brain/memory-pipeline-architecture.md index 995afbf..387563c 100644 --- a/docs/design/brain/memory-pipeline-architecture.md +++ b/docs/design/brain/memory-pipeline-architecture.md @@ -181,6 +181,30 @@ model had ignored. affected entries). - Episode grouping: a vacation spanning many entries currently renders as independent entries plus one month summary. +- Inline media on diary pages: images, and an audio player with a + play button per recording. Design: media export at compile time, + not a proxy. Matrix is capture transport and timeline anchor; the + archive is plain files, the same pattern the archivist uses for + documents (chat -> Paperless). + - The compiler writes each media original to + `{data_dir}/memory/media///.`, + idempotent by event id. Audio gets an `.m4a` transcode beside + the original (Safari does not play Ogg/Opus reliably); images + get a page-weight thumbnail. ffmpeg is a compile-time + dependency. + - Serving is an open choice: a static mount in the wiki + container, or a thin proxy that serves from the same filesystem + and gives logical URLs independent of the on-disk layout. Either + way the renderer emits stable URLs and + `