Skip to content

feat(learn): consolidation layer + RDF-inspired entities/relationships graph#84

Open
ashu17706 wants to merge 3 commits into
mainfrom
claude/refine-local-plan-ff68rg
Open

feat(learn): consolidation layer + RDF-inspired entities/relationships graph#84
ashu17706 wants to merge 3 commits into
mainfrom
claude/refine-local-plan-ff68rg

Conversation

@ashu17706

@ashu17706 ashu17706 commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Started as the Continuous Knowledge Consolidation Layer and grew a second piece: an RDF-inspired entities/relationships graph on top of it, after discussing how RDF's triple model (subject-predicate-object, canonical resource identity, typed vocabularies) could strengthen the self-learning loop.

Part 1 — Continuous Knowledge Consolidation (Progressive Summarization): cheap Stage-1 extraction (segmentSession) runs broadly over dense sessions into a new smriti_knowledge_units table; expensive Stage-2 polish (generateDocument) only runs once a unit proves it's reused (recalled repeatedly) or scored high relevance at extraction time.

  • src/db.tssmriti_knowledge_units table + CRUD helpers (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits, incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits)
  • src/search/recall.ts — tracks retrieval_count on every recall() exit path
  • src/learn/consolidate.ts — segment phase + promote phase
  • src/index.ts, src/format.tssmriti consolidate and smriti learnings

Part 2 — Entities & relationships graph (RDF-inspired, not literal RDF — no URIs/SPARQL, just typed (subject_type, subject_id) pairs in SQLite): today's entities: string[] field is write-only — never queried or matched, so "JWT" in one session and "JSON Web Token" in another look unrelated. This closes that gap and also persists what ollamaCheckConflicts previously only computed ephemerally.

  • src/db.tssmriti_entities (canonical registry) + smriti_relationships (triple store) tables; new minEntityReach promotion criterion — a unit becomes promotable once its entity is independently mentioned by K other units, even at 0 retrievals
  • src/learn/entities.tsresolveEntity (exact-normalize canonicalization via slugify), insertRelationship/getRelationships (triple store), findRelatedCandidates, findEntity, getUnitsForEntity
  • src/learn/consolidate.ts — segment phase turns Stage-1 entities into mentions edges for free (no extra LLM calls); promote phase adds one bounded LLM call (only when entity-sharing candidates exist) to infer relatesTo/supersedes/contradicts edges
  • Team/org propagation — an entity's canonical id is only meaningful if every teammate's machine agrees on it. src/team/config.ts/share.ts/sync.ts route entities through the exact same .smriti/config.json git round-trip custom categories already use (exportEntities/mergeEntities mirror exportCustomCategories/mergeCategories). Unit-to-unit relationship edges propagate via frontmatter directly — no canonicalization needed there, since unit ids are already portable UUIDs once shared.
  • src/index.ts, src/format.tssmriti graph <entity> command, --min-entity-reach flag on consolidate

Two pre-existing bugs found and fixed while building the propagation tests (both predate this PR, exposed because nothing previously tested a real share→sync round trip on disk):

  • syncTeamKnowledge's agent fallback used "team" as an agent id that was never seeded into smriti_agents, so importing any shared file without an explicit agent field (true for the entire pre-existing segmented pipeline) hit an FK violation. Fixed by seeding a team agent.
  • syncTeamKnowledge only recognized pipeline === "segmented" as a single-message document; pipeline === "consolidated" docs were misparsed as raw transcripts and silently skipped on import.

Both remain CLI-only, never wired into the daemon — same reasoning as enrich: LLM work per-flush is unsafe (see src/daemon/index.ts's enrichOnIngest comment).

Test plan

  • test/learn-consolidate.test.ts — segment-phase dedup, promotion threshold, graceful degradation, retrieval tracking
  • test/learn-entities.test.ts — entity canonicalization (exact-normalize merge, no false-merge on real synonyms), relationship triple insert/dedup/pattern-lookup, mentions edges from segment phase (zero extra LLM calls), minEntityReach promotion, LLM-inferred relationship edges (+ best-effort failure handling), findRelatedCandidates
  • test/learn-entities-sync.test.ts — two independent in-memory DBs standing in for two teammates' machines, connected only through a shared tmp .smriti/ directory: confirms an entity converges to the same canonical id via share→sync (not an independently re-slugified duplicate), and that supersedes edges between shared units survive the round-trip
  • bun test --cwd ./test — 391 pass, 0 fail across 33 files, no regressions
  • Manual smoke test: smriti consolidate, smriti learnings, smriti graph <entity> run cleanly against a fresh DB

claude added 2 commits July 25, 2026 09:35
Adds `smriti consolidate` and `smriti learnings`, applying Progressive
Summarization: cheap Stage-1 segmentation runs broadly over dense sessions
into a new smriti_knowledge_units table, and expensive Stage-2 polish
(existing segmentSession/generateDocument pipeline) only runs once a unit
proves reuse via recall or scored high relevance at extraction time.

- src/db.ts: smriti_knowledge_units table + CRUD helpers
  (insertKnowledgeUnit, findUnsegmentedDenseSessions, findPromotableUnits,
  incrementRetrievalCount, promoteKnowledgeUnit, listKnowledgeUnits)
- src/search/recall.ts: track retrieval_count on every recall() path
- src/learn/consolidate.ts: segment + promote phases, reusing the existing
  3-stage segmentation pipeline from src/team/segment.ts and document.ts
- src/index.ts, src/format.ts: CLI wiring for `consolidate` and `learnings`

CLI-only, not wired into the daemon — consolidation runs two LLM stages,
which the daemon's flush path deliberately avoids (see the enrichOnIngest
comment in src/daemon/index.ts).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
Extends the Continuous Knowledge Consolidation Layer with canonical
entities and typed subject-predicate-object relationship triples,
inspired by RDF's data model (adapted pragmatically: no URIs/SPARQL,
just typed (subject_type, subject_id) pairs in SQLite).

- src/db.ts: smriti_entities + smriti_relationships tables; new
  `minEntityReach` promotion criterion in findPromotableUnits (a unit
  becomes promotable once its entity is independently mentioned by K
  other units, even at 0 retrievals); seeds a "team" agent (fixes a
  latent FK bug in syncTeamKnowledge's existing agent fallback)
- src/learn/entities.ts: resolveEntity (exact-normalize canonicalization
  via slugify), insertRelationship/getRelationships (triple store),
  findRelatedCandidates, findEntity, getUnitsForEntity
- src/learn/consolidate.ts: segment phase turns Stage-1 entities into
  "mentions" edges for free; promote phase adds one bounded LLM call to
  infer relatesTo/supersedes/contradicts edges against entity-sharing
  candidates, persisting what ollamaCheckConflicts previously only
  computed ephemerally
- src/team/config.ts, share.ts, sync.ts: entities propagate team/org-wide
  through the same .smriti/config.json round-trip custom categories
  already use (exportEntities/mergeEntities mirror
  exportCustomCategories/mergeCategories); unit-to-unit relationship
  edges propagate via frontmatter directly, needing no canonicalization
  since unit ids are already portable UUIDs. Also fixes sync.ts treating
  "consolidated" pipeline docs as raw conversation transcripts (only
  "segmented" was previously recognized as single-message).
- src/index.ts, src/format.ts: `smriti graph <entity>` command;
  `--min-entity-reach` flag on `smriti consolidate`

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
@ashu17706 ashu17706 changed the title feat(learn): add continuous knowledge consolidation layer feat(learn): consolidation layer + RDF-inspired entities/relationships graph Jul 26, 2026
When two units sharing an entity are promoted in the same consolidate
run, each independently asks the LLM "do I supersede/contradict the
other" — found via a live demo that this can produce both directions
asserted simultaneously (A supersedes B and B supersedes A), which is
incoherent for a directional predicate. Skip inserting the reverse edge
if the candidate already asserted it in the other direction.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnYgLUDwWALu1bUAgrFQDG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants