feat(context): inject high-order memory layers

.env.example

@@ -167,6 +167,7 @@
 # AGENTMEMORY_INJECT_CONTEXT=true                # Inject recalled memories back into agent prompts (#143). Default off — hooks capture observations but do not modify conversation.
 # AGENTMEMORY_COMPRESS_FILE_ROOTS=/repo/docs,/Users/me/notes  # Extra narrow roots allowed for memory_compress_file. Defaults to the daemon cwd when safe; avoid home or /.
 # CONSOLIDATION_ENABLED=true                     # Run the 4-tier consolidation pipeline (memories → semantic → procedural). Default off — opt in once you've measured the LLM cost.
+# AGENTMEMORY_HIGH_ORDER_CONTEXT=false           # Opt out of semantic/procedural/crystal/insight context blocks and smart-search arrays. Unset follows CONSOLIDATION_ENABLED.
 # EVICTION_ENABLED=true                          # Run mem::evict on a timer. Default on; set false to disable scheduled eviction.
 # EVICTION_INTERVAL_MS=86400000                  # Scheduled mem::evict interval in ms. Default 24h.
 # CONSOLIDATION_DECAY_DAYS=30                    # Age (days) after which non-reinforced memories decay during consolidation

README.md

@@ -977,6 +977,24 @@ Inspired by how human brains process memory — not unlike sleep consolidation.
 
 Memories decay over time (Ebbinghaus curve). Frequently accessed memories strengthen. Stale memories auto-evict. Contradictions are detected and resolved.
 
+### Injected Context Blocks
+
+`mem::context` uses the existing token budget to inject bounded blocks. When high-order context is enabled, the injected candidates include:
+
+| Block | Source |
+|------|--------|
+| `Project Profile` | `KV.profiles` |
+| `Lessons Learned` | `KV.lessons` |
+| `Distilled Insights` | `KV.insights` |
+| `Architectural Facts` | `KV.semantic` |
+| `Procedural Memories` | `KV.procedural` |
+| `Crystals` | `KV.crystals` |
+| Recent session summaries and observations | `KV.summaries` and `KV.observations(...)` |
+
+Set `AGENTMEMORY_HIGH_ORDER_CONTEXT=false` to omit the four high-order blocks (`Distilled Insights`, `Architectural Facts`, `Procedural Memories`, and `Crystals`) and their `memory_smart_search` side-channel arrays. When unset, high-order context follows `CONSOLIDATION_ENABLED`; set `AGENTMEMORY_HIGH_ORDER_CONTEXT=true` to read already-stored high-order rows even when automatic consolidation is disabled.
+
+`memory_smart_search` returns high-order matches as optional `semantic`, `procedural`, `crystals`, and `insights` arrays beside normal observation results.
+
 ### What Gets Captured
 
 | Hook | Captures |
@@ -1778,6 +1796,7 @@ Create `~/.agentmemory/.env`:
                                    # PostToolUse regardless of this flag.
 # GRAPH_EXTRACTION_ENABLED=false
 # CONSOLIDATION_ENABLED=false   # on by default when an LLM provider is configured
+# AGENTMEMORY_HIGH_ORDER_CONTEXT=false  # opt out of semantic/procedural/crystal/insight context + smart-search arrays
 # EVICTION_ENABLED=true          # ON by default. Runs mem::evict on a timer.
 # EVICTION_INTERVAL_MS=86400000  # Default: 24h
 # LESSON_DECAY_ENABLED=true

docs/todos/2026-06-19-issue-295-context-memory-layers/plan.md

@@ -0,0 +1,171 @@
+# Issue 295 High-Order Context Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make existing semantic, procedural, crystal, and insight tiers visible through `mem::context` and `mem::smart-search` behind `AGENTMEMORY_HIGH_ORDER_CONTEXT`.
+
+**Architecture:** Use a read-path-only change over existing KV scopes. Add one config helper, gate all four high-order tier reads, render each tier as a distinct context block through the existing token-budget loop, and add optional smart-search side-channel arrays without new storage, routes, tools, auth, dependencies, or version changes.
+
+**Tech Stack:** TypeScript ESM, iii-sdk registered functions, StateKV, Vitest, README docs.
+
+---
+
+## Files
+
+- Modify `src/config.ts`: add `isHighOrderContextEnabled()`.
+- Modify `src/functions/context.ts`: gate existing insight/semantic reads, add procedural/crystal blocks, visible `[tier:id]` IDs, and conservative project/agent scoping.
+- Modify `src/functions/smart-search.ts`: add bounded read-time search for semantic/procedural/crystals, gate existing insights, and keep observation/lesson behavior unchanged.
+- Modify `src/functions/query.ts`: flatten the new smart-search side arrays so `mem::query` callers do not silently lose them.
+- Modify `test/consolidation-default.test.ts`: red/green config flag tests.
+- Modify `test/context-insights-semantic.test.ts`: red/green context high-order tier tests and opt-out coverage.
+- Modify `test/smart-search.test.ts`: red/green high-order smart-search tests, `includeInsights:false`, and scoping.
+- Modify `test/query-integration.test.ts`: red/green normalization for new smart-search arrays.
+- Modify `README.md` and `.env.example`: document injected blocks, search side channels, and flag behavior.
+
+## Acceptance Mapping
+
+| Acceptance | Planned proof |
+| --- | --- |
+| `mem::context` reads semantic/procedural/crystals/insights when enabled | Context tests seed all four KV scopes and assert distinct blocks with IDs |
+| Each tier is distinct with source IDs | Context tests assert headings and `[tier:id]` markers |
+| Token budget respected | Context test uses tiny budget and verifies whole high-order blocks are skipped by existing loop |
+| `memory_smart_search` indexes/searches four tiers | Smart-search tests assert `semantic`, `procedural`, `crystals`, and `insights` arrays |
+| `AGENTMEMORY_HIGH_ORDER_CONTEXT=false` opts out | Config, context, and smart-search disabled tests |
+| Docs updated | README/.env diff and stale-reference search |
+
+## Task 1: Config Gate
+
+**Files:** `test/consolidation-default.test.ts`, `src/config.ts`
+
+- [x] Add `AGENTMEMORY_HIGH_ORDER_CONTEXT` to the env cleanup list in `test/consolidation-default.test.ts`.
+- [x] Add failing tests for:
+  - unset high-order flag follows `CONSOLIDATION_ENABLED=true`;
+  - unset high-order flag is false with no provider/consolidation;
+  - `AGENTMEMORY_HIGH_ORDER_CONTEXT=false` overrides enabled consolidation;
+  - `AGENTMEMORY_HIGH_ORDER_CONTEXT=true` overrides disabled consolidation.
+- [x] Run:
+
+```bash
+corepack pnpm exec vitest run --exclude test/integration.test.ts test/consolidation-default.test.ts
+```
+
+Expected red: `isHighOrderContextEnabled` is not exported.
+
+- [x] Implement:
+
+```ts
+export function isHighOrderContextEnabled(): boolean {
+  const env = getMergedEnv();
+  const explicit = env["AGENTMEMORY_HIGH_ORDER_CONTEXT"];
+  if (explicit === "false" || explicit === "0") return false;
+  if (explicit === "true" || explicit === "1") return true;
+  return isConsolidationEnabled();
+}
+```
+
+## Task 2: Context Tests And Implementation
+
+**Files:** `test/context-insights-semantic.test.ts`, `src/functions/context.ts`
+
+- [x] Extend the config mock with `highOrderContext: true` and `isHighOrderContextEnabled`.
+- [x] Add `ProceduralMemory` and `Crystal` helpers.
+- [x] Add failing tests that:
+  - seed semantic, procedural, crystal, and insight rows and assert headings plus `[semantic:id]`, `[procedural:id]`, `[crystal:id]`, `[insight:id]`;
+  - set high-order disabled and assert no high-order headings or markers appear;
+  - use a tiny budget and assert high-order blocks are skipped whole;
+  - exclude other-project/unresolved procedural source sessions;
+  - exclude ambiguous crystals in isolated mode and include only crystals tied to current-agent sessions.
+- [x] Run:
+
+```bash
+corepack pnpm exec vitest run --exclude test/integration.test.ts test/context-insights-semantic.test.ts
+```
+
+Expected red: procedural/crystal blocks and visible IDs are missing; disabled mode still reads/renders existing semantic/insights.
+
+- [x] Implement in `src/functions/context.ts`:
+  - import `ProceduralMemory`, `Crystal`, and `isHighOrderContextEnabled`;
+  - when disabled, read none of `KV.insights`, `KV.semantic`, `KV.procedural`, `KV.crystals`;
+  - keep lessons outside the gate;
+  - derive semantic/procedural visibility from source sessions, allowing source-less rows only in shared mode;
+  - derive crystal visibility from `project` and/or `sessionId`, failing closed in isolated mode;
+  - render `## Distilled Insights`, `## Architectural Facts`, `## Procedural Memories`, and `## Crystals` with visible typed IDs;
+  - keep `ContextBlock.sourceIds` and the final block sort/budget loop unchanged.
+
+## Task 3: Smart Search Tests And Implementation
+
+**Files:** `test/smart-search.test.ts`, `src/functions/smart-search.ts`
+
+- [x] Extend the config mock with `highOrderContext: true` and `isHighOrderContextEnabled`.
+- [x] Add semantic/procedural/crystal test helpers.
+- [x] Add failing tests that:
+  - seed all high-order tiers and assert `semantic`, `procedural`, `crystals`, and existing `insights`;
+  - disable the high-order gate and assert no high-order arrays and no `mem::insight-search` call;
+  - keep `includeInsights:false` insight-only while semantic/procedural/crystals remain enabled;
+  - filter project/agent scope conservatively;
+  - tolerate one high-order `kv.list` failure without suppressing observation results.
+- [x] Run:
+
+```bash
+corepack pnpm exec vitest run --exclude test/integration.test.ts test/smart-search.test.ts
+```
+
+Expected red: semantic/procedural/crystal arrays are absent; disabled mode still calls insight search.
+
+- [x] Implement in `src/functions/smart-search.ts`:
+  - import high-order types and `isHighOrderContextEnabled`;
+  - add local compact result interfaces;
+  - add deterministic term-overlap scoring with strength/confidence/frequency/recency weighting;
+  - add read-only helpers for `KV.semantic`, `KV.procedural`, and `KV.crystals`;
+  - gate existing insight recall with the high-order flag and preserve `includeInsights:false`;
+  - return optional `semantic`, `procedural`, and `crystals` arrays beside existing `results`, `lessons`, and `insights`.
+
+## Task 4: Query Normalization
+
+**Files:** `test/query-integration.test.ts`, `src/functions/query.ts`
+
+- [x] Add a failing test for a `smart_search` response containing `results`, `lessons`, `insights`, `semantic`, `procedural`, and `crystals`.
+- [x] Assert normalized `_id` and `_kind` are `obs`, `lesson`, `insight`, `semantic`, `procedural`, and `crystal`.
+- [x] Implement:
+  - include `semantic`, `procedural`, and `crystals` in `smart_search` flatten keys;
+  - recognize `semanticId`, `proceduralId`, and `crystalId` in `_id`;
+  - map those IDs to `_kind`.
+
+## Task 5: Docs
+
+**Files:** `README.md`, `.env.example`
+
+- [x] Document the high-order context block list near the 4-tier consolidation section.
+- [x] Document `AGENTMEMORY_HIGH_ORDER_CONTEXT=false` as the opt-out and `true` as an opt-in for existing rows when consolidation is disabled.
+- [x] Distinguish it from `AGENTMEMORY_INJECT_CONTEXT`, which controls hook stdout injection.
+- [x] Add `.env.example` flag near behavior flags.
+- [x] Run:
+
+```bash
+rg -n "AGENTMEMORY_HIGH_ORDER_CONTEXT|Distilled Insights|Architectural Facts|Procedural Memories|Crystals" README.md .env.example src test
+```
+
+## Verification
+
+Run targeted checks first:
+
+```bash
+corepack pnpm exec vitest run --exclude test/integration.test.ts test/consolidation-default.test.ts test/context-insights-semantic.test.ts test/context-lessons.test.ts test/smart-search.test.ts test/query-integration.test.ts
+```
+
+Then run repo-native checks:
+
+```bash
+corepack pnpm run lint
+corepack pnpm run build
+corepack pnpm test
+```
+
+Security gates for this non-trivial TypeScript/docs/test change:
+
+```bash
+semgrep scan --config p/default --error --metrics=off .
+gitleaks protect --staged --redact
+```
+
+OSV is not required unless dependency, lockfile, vendored, container, or package-manager surfaces change.