feat(learn): consolidation layer + RDF-inspired entities/relationships graph#84
Open
ashu17706 wants to merge 3 commits into
Open
feat(learn): consolidation layer + RDF-inspired entities/relationships graph#84ashu17706 wants to merge 3 commits into
ashu17706 wants to merge 3 commits into
Conversation
Adds `smriti consolidate` and `smriti learnings`, applying Progressive Summarization: cheap Stage-1 segmentation runs broadly over dense sessions into a new smriti_knowledge_units table, and expensive Stage-2 polish (existing segmentSession/generateDocument pipeline) only runs once a unit proves reuse via recall or scored high relevance at extraction time. - src/db.ts: smriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits) - src/search/recall.ts: track retrieval_count on every recall() path - src/learn/consolidate.ts: segment + promote phases, reusing the existing 3-stage segmentation pipeline from src/team/segment.ts and document.ts - src/index.ts, src/format.ts: CLI wiring for `consolidate` and `learnings` CLI-only, not wired into the daemon — consolidation runs two LLM stages, which the daemon's flush path deliberately avoids (see the enrichOnIngest comment in src/daemon/index.ts). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
Extends the Continuous Knowledge Consolidation Layer with canonical entities and typed subject-predicate-object relationship triples, inspired by RDF's data model (adapted pragmatically: no URIs/SPARQL, just typed (subject_type, subject_id) pairs in SQLite). - src/db.ts: smriti_entities + smriti_relationships tables; new `minEntityReach` promotion criterion in findPromotableUnits (a unit becomes promotable once its entity is independently mentioned by K other units, even at 0 retrievals); seeds a "team" agent (fixes a latent FK bug in syncTeamKnowledge's existing agent fallback) - src/learn/entities.ts: resolveEntity (exact-normalize canonicalization via slugify), insertRelationship/getRelationships (triple store), findRelatedCandidates, findEntity, getUnitsForEntity - src/learn/consolidate.ts: segment phase turns Stage-1 entities into "mentions" edges for free; promote phase adds one bounded LLM call to infer relatesTo/supersedes/contradicts edges against entity-sharing candidates, persisting what ollamaCheckConflicts previously only computed ephemerally - src/team/config.ts, share.ts, sync.ts: entities propagate team/org-wide through the same .smriti/config.json round-trip custom categories already use (exportEntities/mergeEntities mirror exportCustomCategories/mergeCategories); unit-to-unit relationship edges propagate via frontmatter directly, needing no canonicalization since unit ids are already portable UUIDs. Also fixes sync.ts treating "consolidated" pipeline docs as raw conversation transcripts (only "segmented" was previously recognized as single-message). - src/index.ts, src/format.ts: `smriti graph <entity>` command; `--min-entity-reach` flag on `smriti consolidate` Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
When two units sharing an entity are promoted in the same consolidate run, each independently asks the LLM "do I supersede/contradict the other" — found via a live demo that this can produce both directions asserted simultaneously (A supersedes B and B supersedes A), which is incoherent for a directional predicate. Skip inserting the reverse edge if the candidate already asserted it in the other direction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Started as the Continuous Knowledge Consolidation Layer and grew a second piece: an RDF-inspired entities/relationships graph on top of it, after discussing how RDF's triple model (subject-predicate-object, canonical resource identity, typed vocabularies) could strengthen the self-learning loop.
Part 1 — Continuous Knowledge Consolidation (Progressive Summarization): cheap Stage-1 extraction (
segmentSession) runs broadly over dense sessions into a newsmriti_knowledge_unitstable; expensive Stage-2 polish (generateDocument) only runs once a unit proves it's reused (recalled repeatedly) or scored high relevance at extraction time.src/db.ts—smriti_knowledge_unitstable + CRUD helpers (insertKnowledgeUnit,findUnsegmentedDenseSessions,findPromotableUnits,incrementRetrievalCount,promoteKnowledgeUnit,listKnowledgeUnits)src/search/recall.ts— tracksretrieval_counton everyrecall()exit pathsrc/learn/consolidate.ts— segment phase + promote phasesrc/index.ts,src/format.ts—smriti consolidateandsmriti learningsPart 2 — Entities & relationships graph (RDF-inspired, not literal RDF — no URIs/SPARQL, just typed
(subject_type, subject_id)pairs in SQLite): today'sentities: string[]field is write-only — never queried or matched, so "JWT" in one session and "JSON Web Token" in another look unrelated. This closes that gap and also persists whatollamaCheckConflictspreviously only computed ephemerally.src/db.ts—smriti_entities(canonical registry) +smriti_relationships(triple store) tables; newminEntityReachpromotion criterion — a unit becomes promotable once its entity is independently mentioned by K other units, even at 0 retrievalssrc/learn/entities.ts—resolveEntity(exact-normalize canonicalization viaslugify),insertRelationship/getRelationships(triple store),findRelatedCandidates,findEntity,getUnitsForEntitysrc/learn/consolidate.ts— segment phase turns Stage-1 entities intomentionsedges for free (no extra LLM calls); promote phase adds one bounded LLM call (only when entity-sharing candidates exist) to inferrelatesTo/supersedes/contradictsedgessrc/team/config.ts/share.ts/sync.tsroute entities through the exact same.smriti/config.jsongit round-trip custom categories already use (exportEntities/mergeEntitiesmirrorexportCustomCategories/mergeCategories). Unit-to-unit relationship edges propagate via frontmatter directly — no canonicalization needed there, since unit ids are already portable UUIDs once shared.src/index.ts,src/format.ts—smriti graph <entity>command,--min-entity-reachflag onconsolidateTwo pre-existing bugs found and fixed while building the propagation tests (both predate this PR, exposed because nothing previously tested a real share→sync round trip on disk):
syncTeamKnowledge's agent fallback used"team"as an agent id that was never seeded intosmriti_agents, so importing any shared file without an explicitagentfield (true for the entire pre-existing segmented pipeline) hit an FK violation. Fixed by seeding ateamagent.syncTeamKnowledgeonly recognizedpipeline === "segmented"as a single-message document;pipeline === "consolidated"docs were misparsed as raw transcripts and silently skipped on import.Both remain CLI-only, never wired into the daemon — same reasoning as
enrich: LLM work per-flush is unsafe (seesrc/daemon/index.ts'senrichOnIngestcomment).Test plan
test/learn-consolidate.test.ts— segment-phase dedup, promotion threshold, graceful degradation, retrieval trackingtest/learn-entities.test.ts— entity canonicalization (exact-normalize merge, no false-merge on real synonyms), relationship triple insert/dedup/pattern-lookup,mentionsedges from segment phase (zero extra LLM calls),minEntityReachpromotion, LLM-inferred relationship edges (+ best-effort failure handling),findRelatedCandidatestest/learn-entities-sync.test.ts— two independent in-memory DBs standing in for two teammates' machines, connected only through a shared tmp.smriti/directory: confirms an entity converges to the same canonical id via share→sync (not an independently re-slugified duplicate), and thatsupersedesedges between shared units survive the round-tripbun test --cwd ./test— 391 pass, 0 fail across 33 files, no regressionssmriti consolidate,smriti learnings,smriti graph <entity>run cleanly against a fresh DB