diff --git a/README.md b/README.md index bf2ab751..6bc01821 100644 --- a/README.md +++ b/README.md @@ -1171,6 +1171,8 @@ Use `memory_lesson_save` for reusable workflow guidance: rules, pitfalls, and "p `memory_save`, `mem::remember`, and `POST /agentmemory/remember` accept optional `external_id` and `metadata` fields for memories imported from another system or evaluation dataset. `external_id` is a stable caller-owned source identifier; `metadata` must be a small public object. Secret-like metadata keys and values are redacted and oversized metadata is bounded before storage. `memory_recall` and `memory_smart_search` include `external_id` and `metadata` in compact and narrative results when present; full results include them on the returned observation object. +`memory_recall`, `memory_smart_search`, `POST /agentmemory/search`, and `POST /agentmemory/smart-search` accept optional `targetLayer: "all" | "memory" | "observation"` to restrict primary results to saved memories or session observations. Filtered `memory_smart_search` calls omit lesson, insight, semantic, procedural, and crystal side arrays so the response only contains the requested layer. + `memory_lesson_save` accepts `content` (required), plus optional `context`, `confidence`, `project`, and comma-separated `tags`. New lessons default to `confidence: 0.5` unless a value from `0.0` to `1.0` is provided; out-of-range values fall back to `0.5`. Saving the same lesson content again in the same `project` and `source` strengthens the existing lesson instead of creating a duplicate. Lesson confidence changes through reinforcement and decay: diff --git a/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/plan.md b/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/plan.md new file mode 100644 index 00000000..ed4581f2 --- /dev/null +++ b/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/plan.md @@ -0,0 +1,332 @@ +# Smart Search Layer Filter Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add an additive `targetLayer` selector to existing recall/search surfaces so agents can search only saved memories or only observations without changing default behavior. + +**Architecture:** Keep the selector as request-time retrieval filtering, not storage or index structure. Normalize accepted values at system boundaries and function entry points, then over-fetch whenever the selector is not `all` so post-resolution filtering can still fill the requested limit. Apply the selector after candidate rows are resolved to either `KV.memories` or `KV.observations`; smart-search filtered modes suppress non-primary side arrays. + +**Tech Stack:** TypeScript ESM, iii-sdk registered functions/triggers, MCP tool schema/handlers, standalone MCP shim, Vitest. + +--- + +## Files + +- Modify: `src/functions/search.ts` + - Add `targetLayer?: string` input, normalize to `all | memory | observation`, over-fetch when filtered, filter resolved candidates by source. +- Modify: `src/functions/smart-search.ts` + - Add `targetLayer?: string`, over-fetch when filtered, filter hybrid/expanded rows by source, suppress side arrays when `targetLayer` is not `all`. +- Modify: `src/mcp/tools-registry.ts` + - Add `targetLayer` schema property to `memory_recall` and `memory_smart_search`. +- Modify: `src/mcp/server.ts` + - Validate and forward `targetLayer` in both MCP handlers. +- Modify: `src/triggers/api.ts` + - Validate and whitelist `targetLayer` in `/agentmemory/search` and `/agentmemory/smart-search`. +- Modify: `src/mcp/standalone.ts` + - Validate selector, forward in proxy mode, apply in local fallback. +- Modify: `src/functions/query.ts` + - Include `targetLayer` in `search` and `smart_search` producer whitelists. +- Modify: `README.md` and generated/plugin references only if they document parameters, without changing tool or endpoint counts. +- Test: `test/search.test.ts` +- Test: `test/smart-search.test.ts` +- Test: `test/mcp-server-surface.test.ts` +- Test: `test/mcp-surface-default.test.ts` +- Test: `test/api-boundary-coverage.test.ts` +- Test: `test/mcp-standalone.test.ts` +- Test: `test/mcp-standalone-proxy.test.ts` +- Test: `test/query-integration.test.ts` or the nearest existing query test file. + +## Task 1: Search Function Target Layer + +**Files:** +- Modify: `test/search.test.ts` +- Modify: `src/functions/search.ts` + +- [ ] **Step 1: Write failing `mem::search` tests** + +Add tests that seed one session observation and one saved memory with matching text, then assert: + +```ts +const all = await sdk.trigger("mem::search", { query: "layer needle", limit: 10 }); +expect(all.results.map((r: any) => r.observation.id).sort()).toEqual(["mem_layer", "obs_layer"]); + +const memoryOnly = await sdk.trigger("mem::search", { + query: "layer needle", + targetLayer: "memory", + limit: 10, +}); +expect(memoryOnly.results.map((r: any) => r.observation.id)).toEqual(["mem_layer"]); + +const observationOnly = await sdk.trigger("mem::search", { + query: "layer needle", + targetLayer: "observation", + limit: 10, +}); +expect(observationOnly.results.map((r: any) => r.observation.id)).toEqual(["obs_layer"]); +``` + +Also add an invalid selector assertion: + +```ts +await expect( + sdk.trigger("mem::search", { query: "layer needle", targetLayer: "lesson" }), +).rejects.toThrow("targetLayer must be one of"); +``` + +Add an over-fetch regression where `limit: 1`, the first-ranked result is an observation, and a matching memory ranks below it: + +```ts +const memoryOnly = await sdk.trigger("mem::search", { + query: "ranked layer needle", + targetLayer: "memory", + limit: 1, +}); +expect(memoryOnly.results.map((r: any) => r.observation.id)).toEqual(["mem_ranked_layer"]); +``` + +- [ ] **Step 2: Run the focused red test** + +Run: `corepack pnpm exec vitest run test/search.test.ts -t "targetLayer"` + +Expected: FAIL because `targetLayer` is ignored, invalid values are not rejected, or selector filtering under-fills after the first mixed-layer page. + +- [ ] **Step 3: Implement minimal `mem::search` filtering** + +Add a local type and normalizer: + +```ts +type TargetLayer = "all" | "memory" | "observation"; + +function parseTargetLayer(value: unknown, context: string): TargetLayer { + if (value === undefined || value === null || value === "") return "all"; + if (typeof value !== "string") { + throw new Error(`${context}: targetLayer must be one of: all, memory, observation`); + } + const normalized = value.trim().toLowerCase(); + if (normalized === "all" || normalized === "memory" || normalized === "observation") { + return normalized; + } + throw new Error(`${context}: targetLayer must be one of: all, memory, observation`); +} +``` + +When resolving candidates, keep source metadata: + +```ts +return mem ? { observation: memoryToObservation(mem), targetLayer: "memory" as const } : null; +``` + +and for observation hits: + +```ts +return { observation: obs, targetLayer: "observation" as const }; +``` + +Filter before pushing enriched results: + +```ts +if (targetLayer !== "all" && resolved.targetLayer !== targetLayer) continue; +``` + +Make selector filtering part of the over-fetch decision: + +```ts +const layerFiltering = targetLayer !== "all"; +const filtering = !!(projectFilter || cwdFilter || filterAgentId || timeRange || layerFiltering); +``` + +- [ ] **Step 4: Run green search test** + +Run: `corepack pnpm exec vitest run test/search.test.ts -t "targetLayer"` + +Expected: PASS. + +## Task 2: Smart Search Target Layer + +**Files:** +- Modify: `test/smart-search.test.ts` +- Modify: `src/functions/smart-search.ts` + +- [ ] **Step 1: Write failing smart-search tests** + +Add tests that seed matching observation and memory rows and assert: + +```ts +const result = await sdk.trigger("mem::smart-search", { + query: "layer needle", + targetLayer: "memory", + limit: 10, +}); +expect(result.results.map((r: any) => r.obsId)).toEqual(["mem_layer"]); +expect(result.lessons).toBeUndefined(); +expect(result.insights).toBeUndefined(); +expect(result.semantic).toBeUndefined(); +expect(result.procedural).toBeUndefined(); +expect(result.crystals).toBeUndefined(); +``` + +Add observation-only and invalid selector tests. + +Add an over-fetch regression where `limit: 1`, the first hybrid hit is the wrong layer, and a memory hit below it is still returned for `targetLayer: "memory"`. + +Add expanded-mode tests: + +```ts +const all = await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], +}); +expect(all.results.map((r: any) => r.obsId)).toEqual(["mem_layer", "obs_layer"]); + +const memoryOnly = await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], + targetLayer: "memory", +}); +expect(memoryOnly.results.map((r: any) => r.obsId)).toEqual(["mem_layer"]); + +const observationOnly = await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], + targetLayer: "observation", +}); +expect(observationOnly.results.map((r: any) => r.obsId)).toEqual(["obs_layer"]); +``` + +- [ ] **Step 2: Run red smart-search tests** + +Run: `corepack pnpm exec vitest run test/smart-search.test.ts -t "targetLayer"` + +Expected: FAIL because smart-search ignores the selector, does not over-fetch for target-layer filtering, cannot expand memory IDs, or still emits side arrays in filtered modes. + +- [ ] **Step 3: Implement smart-search filtering** + +Normalize `targetLayer` at function start. Add source resolution using `KV.memories` lookups for hybrid hits where needed, mirroring `makeProjectMatcher` cache style. Make `targetLayer !== "all"` part of the smart-search over-fetch condition, filter by source before slicing to `limit`, and add a source-aware expanded resolver that falls back to `KV.memories`. + +Set include flags for filtered modes: + +```ts +const includeSideArrays = targetLayer === "all"; +const includeLessons = includeSideArrays && data.includeLessons !== false; +const includeInsights = includeSideArrays && highOrderEnabled && data.includeInsights !== false; +``` + +Only assign high-order arrays when `includeSideArrays` is true. + +- [ ] **Step 4: Run green smart-search tests** + +Run: `corepack pnpm exec vitest run test/smart-search.test.ts -t "targetLayer"` + +Expected: PASS. + +## Task 3: Public Adapters + +**Files:** +- Modify: `src/mcp/tools-registry.ts` +- Modify: `src/mcp/server.ts` +- Modify: `src/triggers/api.ts` +- Modify: `src/mcp/standalone.ts` +- Modify: `src/functions/query.ts` +- Modify: related adapter tests + +- [ ] **Step 1: Write failing adapter tests** + +Add assertions that: + +```ts +expect(tools.get("memory_recall")?.inputSchema.properties.targetLayer).toBeDefined(); +expect(tools.get("memory_smart_search")?.inputSchema.properties.targetLayer).toBeDefined(); +``` + +MCP handler payload assertions should include: + +```ts +args: { query: "auth", targetLayer: "memory" } +payload: { query: "auth", targetLayer: "memory" } +``` + +REST whitelist tests should assert unknown fields are dropped but `targetLayer` is forwarded. Standalone proxy tests should assert request bodies include `targetLayer`. + +Add invalid-value tests for MCP `memory_recall` and `memory_smart_search`, REST `/search` and `/smart-search`, standalone proxy/local fallback, and function-level calls. All should reject `targetLayer: "lesson"` before dispatch where a boundary handler exists. + +Add `memory_query` producer tests that assert both `search` and `smart_search` pipeline steps forward `targetLayer` unchanged. + +- [ ] **Step 2: Run red adapter tests** + +Run: + +```bash +corepack pnpm exec vitest run test/mcp-server-surface.test.ts test/mcp-surface-default.test.ts test/api-boundary-coverage.test.ts test/mcp-standalone.test.ts test/mcp-standalone-proxy.test.ts test/query-integration.test.ts +``` + +Expected: FAIL on missing `targetLayer` schema/forwarding/validation or missing query producer forwarding. + +- [ ] **Step 3: Implement adapter forwarding** + +Add the schema property description: + +```ts +targetLayer: { + type: "string", + description: "Optional result source filter: all, memory, or observation (default all). Use memory for saved memory_save entries.", +} +``` + +Validate at MCP and REST boundaries with the same accepted values and safe error text. Add `targetLayer` to standalone `Validated`, proxy request bodies, local fallback filtering, and `memory_query` producer `pick()` lists. Standalone local fallback stores only memories; `targetLayer: "observation"` returns the normal empty result shape rather than reading other storage. + +- [ ] **Step 4: Run green adapter tests** + +Run the same adapter test command. + +Expected: PASS. + +## Task 4: Docs, Cleanup, And Verification + +**Files:** +- Modify: `README.md` +- Modify: `docs/todos/2026-06-19-issue-328-smart-search-layer-filter/todo.md` + +- [ ] **Step 1: Document the selector** + +Update the MCP tool docs section so `memory_recall` and `memory_smart_search` mention `targetLayer`. Do not change MCP tool counts or REST endpoint counts. + +- [ ] **Step 2: Focused simplification pass** + +Inspect the active diff for duplicated selector parsing or unclear naming. Keep API contracts unchanged. + +- [ ] **Step 3: Run targeted verification** + +Run: + +```bash +corepack pnpm exec vitest run test/search.test.ts test/smart-search.test.ts test/mcp-server-surface.test.ts test/mcp-surface-default.test.ts test/api-boundary-coverage.test.ts test/mcp-standalone.test.ts test/mcp-standalone-proxy.test.ts test/query-integration.test.ts +``` + +Expected: PASS. + +- [ ] **Step 4: Run broader repo-native checks as feasible** + +Run: + +```bash +corepack pnpm run lint +corepack pnpm test +``` + +Expected: PASS, or record blocker and closest targeted evidence. + +- [ ] **Step 5: Security gates before any commit/PR readiness claim** + +Because this changes MCP/REST API contracts and retrieval filtering, run required local gates before commit/PR readiness: + +```bash +semgrep scan --config p/default --error --metrics=off . +gitleaks protect --staged --redact +``` + +OSV is not required unless dependency, lockfile, vendored, container, or third-party package surfaces change. + +## Review Notes + +Spec coverage: Covers issue #328, approved `targetLayer` contract, default compatibility, MCP/REST/standalone/query propagation, smart-search side arrays, docs, and verification. + +Placeholder scan: No TBD placeholders remain. + +Type consistency: Use canonical `targetLayer` and values `all`, `memory`, `observation` throughout. diff --git a/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/todo.md b/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/todo.md new file mode 100644 index 00000000..041b1d7f --- /dev/null +++ b/docs/todos/2026-06-19-issue-328-smart-search-layer-filter/todo.md @@ -0,0 +1,81 @@ +# Issue 328 Smart Search Layer Filter + +Scope: repository worktree `/Users/A1538552/.codex/worktrees/1e40/agentmemory` on branch `issue/328-smart-search-layer-filter`. + +Issue: GitHub #328, `memory_smart_search returns results from all layers flat — no way to search only memory_save entries (L2)`. + +## Sprint Contract + +Goal: add an additive `targetLayer` selector so existing recall/search surfaces can search only saved `memory_save` rows or only session observations while preserving today's default mixed search. + +Scope: +- `mem::search`, `mem::smart-search`, MCP `memory_recall`, MCP `memory_smart_search`, REST `/agentmemory/search`, REST `/agentmemory/smart-search`, standalone MCP shim/proxy, and `memory_query` producers. +- Target selector values: `all`, `memory`, and `observation`; default `all`. +- `memory` means rows persisted in `KV.memories` by `memory_save`/`mem::remember`. +- `observation` means rows persisted under per-session `KV.observations`. + +Non-goals: +- No new MCP tool. +- No new REST endpoint. +- No schema, persistence, auth, tenancy, routing, dependency, index format, or migration changes. +- No changes to `/agentmemory/memories?q=...` semantics except documentation references if needed. +- No remote writes without separate current-turn confirmation. + +Acceptance criteria: +- Existing calls without `targetLayer` preserve current behavior. +- `targetLayer: "memory"` returns only saved memories in `memory_recall`/`memory_smart_search` primary results. +- `targetLayer: "observation"` returns only session observations in `memory_recall`/`memory_smart_search` primary results. +- `targetLayer` filtering over-fetches before post-resolution filtering so a high-ranked opposite-layer page does not hide lower-ranked matching rows. +- Smart-search expanded mode resolves and filters both observation IDs and saved memory IDs. +- Invalid `targetLayer` values are rejected at MCP/REST boundaries and reported safely by function-level calls. +- REST and MCP adapters whitelist/forward `targetLayer`. +- Standalone MCP proxy/local fallback validates and forwards or applies `targetLayer`. +- `memory_query` forwards `targetLayer` for `search` and `smart_search` producers. +- Smart-search `targetLayer` values other than `all` omit side arrays (`lessons`, `insights`, `semantic`, `procedural`, `crystals`) so filtered mode returns only the requested primary row source. + +Intended verification: +- Red/green focused Vitest for search, smart-search, MCP server surface, REST boundary, standalone shim, and query producer forwarding. +- `corepack pnpm exec vitest run` on touched test files. +- `corepack pnpm run lint` and broader tests if feasible. +- Required security gates before commit/PR readiness because this changes MCP/REST API contracts and retrieval filtering. + +Known boundaries: +- Public API/tool input schema changes are approved by the user in the current turn. +- Push, PR creation, PR merge, and remote issue/PR writes are not approved in this current turn. + +Stop conditions: +- Any implementation path requires storage/schema/index-format migration, auth behavior changes, new endpoints/tools, dependency changes, or a product choice outside the approved `targetLayer` contract. +- Verification fails twice for the same unexplained reason. +- Required security tools are missing or report findings that cannot be fixed without user risk acceptance. + +## Feature / Verification Matrix + +| Change | Verification method | Status | Evidence | +| --- | --- | --- | --- | +| Function-level `mem::search` filters memory/observation rows | Red/green tests in `test/search.test.ts` or focused adjacent test | Complete | Initial focused suite failed before implementation; targeted suite later passed with `test/search.test.ts`. | +| Function-level `mem::smart-search` filters primary results and suppresses non-L2 arrays for memory mode | Red/green tests in `test/smart-search.test.ts` | Complete | Initial focused suite failed before implementation; targeted suite later passed with `test/smart-search.test.ts`. | +| MCP schemas and handlers expose/validate/forward `targetLayer` | `test/mcp-server-surface.test.ts`, `test/mcp-surface-default.test.ts`, `test/mcp-project-scope.test.ts` | Complete | Targeted suite passed after schema/handler changes. | +| REST `/search` and `/smart-search` whitelist/validate/forward `targetLayer` | `test/api-boundary-coverage.test.ts` | Complete | Targeted suite passed after REST whitelist changes. | +| Standalone MCP proxy/local fallback supports selector | `test/mcp-standalone.test.ts`, `test/mcp-standalone-proxy.test.ts` | Complete | Targeted suite passed; local fallback returns empty results for `targetLayer: "observation"`. | +| `memory_query` forwards selector | `test/query-integration.test.ts` or focused query test | Complete | Targeted suite passed with producer forwarding assertions. | +| Documentation reflects the new selector without count changes | README/doc search and diff review | Complete | `README.md` and `plugin/skills/agentmemory-mcp-tools/REFERENCE.md` updated; no tool/endpoint counts changed. | + +## Subagent Ledger + +| Workstream | Scope | Edits allowed | Expected output | Result | Residual risk | +| --- | --- | --- | --- | --- | --- | +| Pre-implementation plan review | `docs/todos/2026-06-19-issue-328-smart-search-layer-filter/plan.md`, approved issue scope | No | High/Medium findings only | Complete | Valid findings incorporated: targetLayer over-fetch, smart-search expandIds, side-array suppression for all filtered modes, standalone observation fallback semantics, invalid adapter tests, query forwarding tests. | +| Final implementation review | Task-owned diff after verification | No | ACCEPT or evidence-backed findings | Complete | Self-review found no scope drift: no new tools/endpoints/schema/dependencies; selector is additive and defaults to `all`. | + +## Progress Notes + +- 2026-06-19: User approved the checkpoint recommendation to add `targetLayer`. +- 2026-06-19: Existing arena synthesis recorded at `/tmp/arena-328/synthesis.md`. +- 2026-06-19: Branch is `issue/328-smart-search-layer-filter`; initial git status was clean. +- 2026-06-19: Pre-implementation reviewers found valid plan gaps. Plan updated before implementation. +- 2026-06-19: Wrote red tests across search, smart-search, MCP, REST, standalone shim, and `memory_query`; focused targeted suite failed as expected before implementation. +- 2026-06-19: Implemented shared `targetLayer` parsing, function-level filtering with over-fetch, smart-search memory-ID expansion, side-array suppression for filtered smart-search, adapter validation/forwarding, standalone fallback behavior, query producer forwarding, and docs. +- 2026-06-19: Verification passed: `corepack pnpm exec vitest run test/search.test.ts test/smart-search.test.ts test/mcp-server-surface.test.ts test/mcp-surface-default.test.ts test/api-boundary-coverage.test.ts test/mcp-standalone.test.ts test/mcp-standalone-proxy.test.ts test/query-integration.test.ts` (384 tests), `corepack pnpm run lint`, `corepack pnpm test` (211 files, 2916 tests), `corepack pnpm run skills:check`, and `corepack pnpm run build`. +- 2026-06-19: Security gates passed: `semgrep scan --config p/default --error --metrics=off .` (0 findings) and `gitleaks protect --staged --redact` (no leaks). OSV not run because no dependency, lockfile, container, vendored, or third-party package surface changed. +- 2026-06-19: Final self-review accepted the task-owned diff. Residual risk: `targetLayer` is a public additive parameter and should be called out in PR notes for downstream MCP/REST clients. +- 2026-06-19: GitHub push-prepare fetched `origin/main` at `24ff6779f5618d0f07039161be4f750252747f31` and merged it locally without conflicts. Post-merge verification passed: targeted Vitest suite (384 tests), `corepack pnpm run lint`, `corepack pnpm test` (211 files, 2919 tests), `corepack pnpm run skills:check`, `corepack pnpm run build`, and `semgrep scan --config p/default --error --metrics=off .` (0 findings). diff --git a/plugin/skills/agentmemory-mcp-tools/REFERENCE.md b/plugin/skills/agentmemory-mcp-tools/REFERENCE.md index c1c438b7..7b44eb99 100644 --- a/plugin/skills/agentmemory-mcp-tools/REFERENCE.md +++ b/plugin/skills/agentmemory-mcp-tools/REFERENCE.md @@ -43,7 +43,7 @@ agentmemory exposes 61 MCP tools. 8 are in the default core set (`--tools core` | `memory_patterns` | | `project`: string | Use to detect recurring patterns across sessions when reviewing a project for repeated bugs, recurring workflows, or common pitfalls worth formalizing as lessons. | | `memory_profile` | | `project`*: string, `refresh`: string | Use to get a project's top concepts and file patterns when starting work in an unfamiliar project or checking common terminology. | | `memory_query` | | `pipeline`*: array, `options`: object, `dry_run`: boolean | Use to run a server-side retrieval pipeline in one call. Pipeline examples: [{"op":"search","query":"auth","out":"obs"},{"op":"lesson_recall","query":"auth","out":"lessons"},{"op":"concat","in":["obs","lessons"]},{"op":"rank_by_relevance","target":"auth decisions"},{"op":"limit","count":5}]. Supported producers include search, smart_search, lesson_recall, graph_query, facet_query, insight_list, timeline, sessions, frontier, vision_search, and profile. Use dry_run first to validate cost and shape without executing producers. | -| `memory_recall` | yes | `query`*: string, `limit`: number, `format`: string, `token_budget`: number, `project`: string, `start_time`: string, `end_time`: string | Search past session observations for relevant context. Use when you need to recall what happened in previous sessions, find past decisions, or look up how a file was modified before. | +| `memory_recall` | yes | `query`*: string, `limit`: number, `format`: string, `token_budget`: number, `project`: string, `targetLayer`: string, `start_time`: string, `end_time`: string | Search past session observations for relevant context. Use when you need to recall what happened in previous sessions, find past decisions, or look up how a file was modified before. | | `memory_reflect` | yes | `project`: string, `maxClusters`: number | Use to synthesize higher-order insights from accumulated memories when looking for emergent patterns, cross-project themes, or new best practices. | | `memory_relations` | | `memoryId`*: string, `maxHops`: number, `minConfidence`: number | Use to explore how memories are connected when finding all items related to a concept or tracing a topic through the knowledge graph. | | `memory_routine_run` | | `routineId`*: string, `project`: string, `initiatedBy`: string | Use to start a predefined multi-step process such as a release checklist or deploy pipeline. Instantiates a frozen routine, creating actions for each step with proper dependencies. | @@ -61,7 +61,7 @@ agentmemory exposes 61 MCP tools. 8 are in the default core set (`--tools core` | `memory_slot_get` | | `label`*: string | Use to read a single slot by label when checking the current value of a slot like 'persona' or 'pending_items'. | | `memory_slot_list` | | none | Use to list all memory slots (pinned, project, and global) when checking what persistent context is available across sessions. | | `memory_slot_replace` | | `label`*: string, `content`*: string | Use to update a slot's entire content when a persistent context slot needs a fresh state. Fails if content exceeds sizeLimit. | -| `memory_smart_search` | yes | `query`*: string, `expandIds`: string, `limit`: number, `format`: string, `includeInsights`: boolean, `start_time`: string, `end_time`: string, `project`: string, `agentId`: string | Use for broad exploratory search when exact terms are uncertain or keyword search returns too little. Hybrid semantic+keyword search returns initial matches; expand with expandIds to get full details. | +| `memory_smart_search` | yes | `query`*: string, `expandIds`: string, `limit`: number, `format`: string, `includeInsights`: boolean, `targetLayer`: string, `start_time`: string, `end_time`: string, `project`: string, `agentId`: string | Use for broad exploratory search when exact terms are uncertain or keyword search returns too little. Hybrid semantic+keyword search returns initial matches; expand with expandIds to get full details. | | `memory_snapshot_create` | | `message`: string | Use to create a git-versioned checkpoint of current memory state before bulk deletes, consolidations, or imports. | | `memory_team_feed` | | `limit`: number, `userId`: string | Use to see what other agents have shared since you last checked. For multi-agent setups. | | `memory_team_share` | | `itemId`*: string, `itemType`*: string, `userId`: string | Use to broadcast a memory or observation to other agents on the team. For multi-agent setups. | diff --git a/src/functions/query.ts b/src/functions/query.ts index 12bcff88..27e83971 100644 --- a/src/functions/query.ts +++ b/src/functions/query.ts @@ -894,6 +894,7 @@ async function executeProducer(sdk: ISdk, kv: StateKV, step: QueryStep): Promise "format", "token_budget", "agentId", + "targetLayer", "start_time", "end_time", ]) }); @@ -907,6 +908,7 @@ async function executeProducer(sdk: ISdk, kv: StateKV, step: QueryStep): Promise "includeLessons", "includeInsights", "agentId", + "targetLayer", "sessionId", "source", "start_time", diff --git a/src/functions/search.ts b/src/functions/search.ts index e131e066..0e085f52 100644 --- a/src/functions/search.ts +++ b/src/functions/search.ts @@ -11,6 +11,7 @@ import { compactResultSessionAttribution } from "./session-attribution.js"; import { logger } from "../logger.js"; import { getAgentId, isAgentScopeIsolated } from "../config.js"; import { inTimeRange, parseTimeRange, TimeRangeError } from "../state/time-filter.js"; +import { parseTargetLayer, TARGET_LAYER_ERROR, type TargetLayer } from "../target-layer.js"; let index: SearchIndex | null = null let vectorIndex: VectorIndex | null = null @@ -334,6 +335,7 @@ export function registerSearchFunction(sdk: ISdk, kv: StateKV): void { agentId?: string start_time?: string end_time?: string + targetLayer?: string }) => { const idx = getSearchIndex() @@ -392,6 +394,10 @@ export function registerSearchFunction(sdk: ISdk, kv: StateKV): void { if (!['full', 'compact', 'narrative'].includes(format)) { throw new Error("mem::search: format must be one of 'full', 'compact', or 'narrative'") } + const targetLayer = parseTargetLayer(data.targetLayer) + if (targetLayer === null) { + throw new Error(`mem::search: ${TARGET_LAYER_ERROR}`) + } let tokenBudget: number | undefined if (data.token_budget !== undefined) { if (!Number.isInteger(data.token_budget) || data.token_budget < 1) { @@ -424,7 +430,7 @@ export function registerSearchFunction(sdk: ISdk, kv: StateKV): void { // doesn't carry it), so without the over-fetch isolated-mode // queries return underfilled pages when same-agent matches // rank lower than cross-agent ones in the hybrid score. - const filtering = !!(projectFilter || cwdFilter || filterAgentId || timeRange) + const filtering = !!(projectFilter || cwdFilter || filterAgentId || timeRange || targetLayer !== "all") const fetchLimit = filtering ? Math.max(effectiveLimit * 10, 100) : effectiveLimit const results = idx.search(query, fetchLimit) @@ -474,7 +480,7 @@ export function registerSearchFunction(sdk: ISdk, kv: StateKV): void { // rows, and capping early would underfill the result page. Use // fetchLimit as the upper bound in that case; the final // truncation lives at the end of the second pass. - const earlyCap = filterAgentId || timeRange ? fetchLimit : effectiveLimit + const earlyCap = filterAgentId || timeRange || targetLayer !== "all" ? fetchLimit : effectiveLimit const candidates: typeof results = [] for (const r of results) { if (candidates.length >= earlyCap) break @@ -517,17 +523,19 @@ export function registerSearchFunction(sdk: ISdk, kv: StateKV): void { const obs = await kv .get(KV.observations(r.sessionId), r.obsId) .catch(() => null) - if (obs) return obs + if (obs) return { observation: obs, targetLayer: "observation" as TargetLayer } const mem = await kv .get(KV.memories, r.obsId) .catch(() => null) - return mem ? memoryToObservation(mem) : null + return mem ? { observation: memoryToObservation(mem), targetLayer: "memory" as TargetLayer } : null }) ) const enriched: SearchResult[] = [] for (let i = 0; i < candidates.length; i++) { - const obs = obsResults[i] - if (!obs) continue + const resolved = obsResults[i] + if (!resolved) continue + if (targetLayer !== "all" && resolved.targetLayer !== targetLayer) continue + const obs = resolved.observation // #817: enforce agent-scope after the observation/memory is // loaded. The BM25 index doesn't carry agentId so the filter // happens post-lookup. Wildcard ("*") and no-isolation paths diff --git a/src/functions/smart-search.ts b/src/functions/smart-search.ts index 091c3fe8..0158c82d 100644 --- a/src/functions/smart-search.ts +++ b/src/functions/smart-search.ts @@ -15,7 +15,7 @@ import type { } from "../types.js"; import { KV } from "../state/schema.js"; import { StateKV } from "../state/kv.js"; -import { observationPassthroughFields } from "../state/memory-utils.js"; +import { memoryToObservation, observationPassthroughFields } from "../state/memory-utils.js"; import { withKeyedLock } from "../state/keyed-mutex.js"; import { recordAccessBatch } from "./access-tracker.js"; import { compactResultSessionAttribution } from "./session-attribution.js"; @@ -29,6 +29,7 @@ import { logger } from "../logger.js"; import { getCounters } from "../telemetry/setup.js"; import { inTimeRange, parseTimeRange, TimeRangeError } from "../state/time-filter.js"; import { filterInsightsBySourceMemoryScope } from "./insight-visibility.js"; +import { parseTargetLayer, TARGET_LAYER_ERROR, type TargetLayer } from "../target-layer.js"; // #771: smart-search followup-rate diagnostic. Stored per session as // the most recent search payload, used to detect whether the next @@ -377,6 +378,7 @@ export function registerSmartSearchFunction( source?: string; start_time?: string; end_time?: string; + targetLayer?: string; }) => { // Compute the agent filter once, up front. Both the expandIds @@ -419,6 +421,14 @@ export function registerSmartSearchFunction( error: "format must be one of: full, compact, narrative", }; } + const targetLayer = parseTargetLayer(data.targetLayer); + if (targetLayer === null) { + return { + mode: "compact", + results: [], + error: TARGET_LAYER_ERROR, + }; + } const parseRequestedTimeRange = (): ReturnType => parseTimeRange({ @@ -484,13 +494,12 @@ export function registerSmartSearchFunction( obsId: string; sessionId: string; observation: CompressedObservation; + targetLayer: TargetLayer; }> = []; const results = await Promise.all( items.map(({ obsId, sessionId }) => - findObservation(kv, obsId, sessionId).then((obs) => - obs ? { obsId, sessionId: obs.sessionId, observation: obs } : null, - ), + findSearchTarget(kv, obsId, sessionId), ), ); for (const r of results) { @@ -513,23 +522,26 @@ export function registerSmartSearchFunction( const timeScoped = timeRange ? projectScoped.filter((e) => inTimeRange(e.observation.timestamp, timeRange)) : projectScoped; + const layerScoped = targetLayer === "all" + ? timeScoped + : timeScoped.filter((e) => e.targetLayer === targetLayer); void recordAccessBatch( kv, - timeScoped.map((e) => e.observation.id), + layerScoped.map((e) => e.observation.id), ); const truncated = data.expandIds.length > raw.length; logger.info("Smart search expanded", { requested: data.expandIds.length, attempted: raw.length, - returned: timeScoped.length, - filteredOutOfScope: expanded.length - timeScoped.length, + returned: layerScoped.length, + filteredOutOfScope: expanded.length - layerScoped.length, truncated, }); return { mode: "expanded", - results: (await attachAttribution(timeScoped)).map((r) => ({ + results: (await attachAttribution(layerScoped)).map((r) => ({ ...r, ...observationPassthroughFields(r.observation), })), @@ -561,9 +573,10 @@ export function registerSmartSearchFunction( const insightLimit = Math.min(limit, 10); const highOrderLimit = Math.min(limit, 10); const highOrderEnabled = isHighOrderContextEnabled(); - const includeLessons = data.includeLessons !== false; + const includeSideArrays = targetLayer === "all"; + const includeLessons = includeSideArrays && data.includeLessons !== false; const includeInsights = - highOrderEnabled && data.includeInsights !== false; + includeSideArrays && highOrderEnabled && data.includeInsights !== false; // Over-fetch when filtering. Hybrid search can't filter on // agentId, project, or timestamp inside every retrieval stream, so we ask @@ -571,7 +584,7 @@ export function registerSmartSearchFunction( const namedConceptSearchLimit = namedConcept ? Math.min(limit * 3, 500) : limit; - const overFetchLimit = filterAgentId || filterProject || timeRange + const overFetchLimit = filterAgentId || filterProject || timeRange || targetLayer !== "all" ? Math.min(limit * 10, 500) : namedConceptSearchLimit; @@ -597,7 +610,7 @@ export function registerSmartSearchFunction( includeInsights ? recallInsights(sdk, kv, data.query, insightLimit, filterProject, filterAgentId) : Promise.resolve([]), - highOrderEnabled + includeSideArrays && highOrderEnabled ? searchSemanticMemories( kv, data.query, @@ -606,7 +619,7 @@ export function registerSmartSearchFunction( filterAgentId, ) : Promise.resolve([]), - highOrderEnabled + includeSideArrays && highOrderEnabled ? searchProceduralMemories( kv, data.query, @@ -615,7 +628,7 @@ export function registerSmartSearchFunction( filterAgentId, ) : Promise.resolve([]), - highOrderEnabled + includeSideArrays && highOrderEnabled ? searchCrystals( kv, data.query, @@ -642,8 +655,17 @@ export function registerSmartSearchFunction( const timeFilteredHybrid = timeRange ? scopedHybrid.filter((r) => inTimeRange(r.observation.timestamp, timeRange)) : scopedHybrid; + let layerFilteredHybrid = timeFilteredHybrid; + if (targetLayer !== "all") { + const layerMatches = await Promise.all( + timeFilteredHybrid.map(async (r) => + (await resolveHybridTargetLayer(kv, r.observation.id)) === targetLayer, + ), + ); + layerFilteredHybrid = timeFilteredHybrid.filter((_r, i) => layerMatches[i]); + } const filteredHybrid = boostHybridResults( - timeFilteredHybrid, + layerFilteredHybrid, namedConcept, ).slice(0, limit); @@ -756,7 +778,7 @@ export function registerSmartSearchFunction( } = { mode: format, results }; if (includeLessons) response.lessons = lessons; if (includeInsights) response.insights = insights; - if (highOrderEnabled) { + if (includeSideArrays && highOrderEnabled) { response.semantic = semantic; response.procedural = procedural; response.crystals = crystals; @@ -1093,3 +1115,41 @@ async function findObservation( } return null; } + +async function findSearchTarget( + kv: StateKV, + obsId: string, + sessionIdHint?: string, +): Promise<{ + obsId: string; + sessionId: string; + observation: CompressedObservation; + targetLayer: TargetLayer; +} | null> { + const obs = await findObservation(kv, obsId, sessionIdHint); + if (obs) { + return { + obsId, + sessionId: obs.sessionId, + observation: obs, + targetLayer: "observation", + }; + } + + const mem = await kv.get(KV.memories, obsId).catch(() => null); + if (!mem) return null; + return { + obsId: mem.id, + sessionId: mem.sessionIds?.[0] ?? "memory", + observation: memoryToObservation(mem), + targetLayer: "memory", + }; +} + +async function resolveHybridTargetLayer( + kv: StateKV, + obsId: string, +): Promise { + const mem = await kv.get(KV.memories, obsId).catch(() => null); + return mem ? "memory" : "observation"; +} diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 107afca2..d7bc6aef 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -19,6 +19,7 @@ import { parseTimeRange, TimeRangeError, } from "../state/time-filter.js"; +import { parseOptionalTargetLayer, TARGET_LAYER_ERROR } from "../target-layer.js"; type McpResponse = { status_code: number; @@ -316,6 +317,13 @@ export function registerMcpEndpoints( body: { error: "token_budget must be a positive integer" }, }; } + const targetLayer = parseOptionalTargetLayer(args.targetLayer); + if (targetLayer === null) { + return { + status_code: 400, + body: { error: TARGET_LAYER_ERROR }, + }; + } // #817: forward agentId so mem::search applies the same // isolation filter smart-search uses. Default behavior is // unchanged (no agentId → falls back to env AGENT_ID when @@ -341,6 +349,7 @@ export function registerMcpEndpoints( format, token_budget: tokenBudget, agentId: recallAgentId, + ...(targetLayer !== undefined && { targetLayer }), ...(project !== undefined && { project }), ...(timeRange ? { start_time: args.start_time, end_time: args.end_time } : {}), } }); @@ -595,6 +604,13 @@ export function registerMcpEndpoints( }, }; } + const targetLayer = parseOptionalTargetLayer(args.targetLayer); + if (targetLayer === null) { + return { + status_code: 400, + body: { error: TARGET_LAYER_ERROR }, + }; + } let timeRange: ReturnType; try { timeRange = parseToolTimeRange(args); @@ -615,6 +631,7 @@ export function registerMcpEndpoints( limit, ...(typeof args.format === "string" && args.format.trim().length > 0 && { format }), ...(typeof args.includeInsights === "boolean" && { includeInsights: args.includeInsights }), + ...(targetLayer !== undefined && { targetLayer }), ...(timeRange ? { start_time: args.start_time, end_time: args.end_time } : {}), ...(project !== undefined && { project }), ...(agentId !== undefined && { agentId }), diff --git a/src/mcp/standalone.ts b/src/mcp/standalone.ts index 8b124e60..0e8492c4 100644 --- a/src/mcp/standalone.ts +++ b/src/mcp/standalone.ts @@ -24,6 +24,7 @@ import { type ProxyHandle, } from "./rest-proxy.js"; import { isPlainObject, normalizeMetadataObject } from "../functions/session-metadata.js"; +import { parseOptionalTargetLayer, TARGET_LAYER_ERROR, type TargetLayer } from "../target-layer.js"; const IMPLEMENTED_TOOLS = new Set([ "memory_save", @@ -265,6 +266,7 @@ interface Validated { endTime?: string; includeInsights?: boolean; expandIds?: string[]; + targetLayer?: TargetLayer; memoryIds?: string[]; reason?: string; operation?: string; @@ -297,6 +299,7 @@ async function searchLocalMemories( kvInstance: InMemoryKV, v: Validated, ): Promise { + if (v.targetLayer === "observation") return []; const query = (v.query || "").toLowerCase(); const limit = v.limit ?? DEFAULT_LIMIT; const all = await kvInstance.list("mem:memories"); @@ -390,6 +393,9 @@ function validate(toolName: string, args: Record): Validated { if (Number.isFinite(n) && n > 0) v.tokenBudget = Math.floor(n); } } + const targetLayer = parseOptionalTargetLayer(args["targetLayer"]); + if (targetLayer === null) throw new Error(TARGET_LAYER_ERROR); + if (targetLayer !== undefined) v.targetLayer = targetLayer; const project = args["project"]; if (typeof project === "string" && project.trim()) { v.project = project.trim(); @@ -481,6 +487,7 @@ async function handleProxy( if (v.tokenBudget != null) body["token_budget"] = v.tokenBudget; if (v.project != null) body["project"] = v.project; if (v.agentId != null) body["agentId"] = v.agentId; + if (v.targetLayer !== undefined) body["targetLayer"] = v.targetLayer; if (v.timeRange) { if (v.startTime !== undefined) body["start_time"] = v.startTime; if (v.endTime !== undefined) body["end_time"] = v.endTime; @@ -497,6 +504,7 @@ async function handleProxy( if (v.tokenBudget != null) body["token_budget"] = v.tokenBudget; if (v.project != null) body["project"] = v.project; if (v.agentId != null) body["agentId"] = v.agentId; + if (v.targetLayer !== undefined) body["targetLayer"] = v.targetLayer; if (v.timeRange) { if (v.startTime !== undefined) body["start_time"] = v.startTime; if (v.endTime !== undefined) body["end_time"] = v.endTime; diff --git a/src/mcp/tools-registry.ts b/src/mcp/tools-registry.ts index 9ab18a04..14de9b12 100644 --- a/src/mcp/tools-registry.ts +++ b/src/mcp/tools-registry.ts @@ -40,6 +40,11 @@ export const CORE_TOOLS: McpToolDef[] = [ "Optional opaque canonical project identifier to restrict recall. Use the same value " + "that session hooks store as project; linked Git worktrees share one git: value.", }, + targetLayer: { + type: "string", + description: + "Optional result layer: all, memory, or observation (default all).", + }, start_time: { type: "string", description: "Optional inclusive ISO 8601 lower time bound for observation timestamps", @@ -190,6 +195,11 @@ export const CORE_TOOLS: McpToolDef[] = [ type: "boolean", description: "Set false to omit distilled insights from smart-search results", }, + targetLayer: { + type: "string", + description: + "Optional result layer: all, memory, or observation (default all).", + }, start_time: { type: "string", description: "Optional inclusive ISO 8601 lower time bound for observation timestamps", diff --git a/src/target-layer.ts b/src/target-layer.ts new file mode 100644 index 00000000..0d29d7a3 --- /dev/null +++ b/src/target-layer.ts @@ -0,0 +1,25 @@ +export const TARGET_LAYER_ERROR = + "targetLayer must be one of: all, memory, observation"; + +export type TargetLayer = "all" | "memory" | "observation"; + +export function parseTargetLayer(value: unknown): TargetLayer | null { + if (value === undefined) return "all"; + if (typeof value !== "string") return null; + const normalized = value.trim().toLowerCase(); + if ( + normalized === "all" || + normalized === "memory" || + normalized === "observation" + ) { + return normalized; + } + return null; +} + +export function parseOptionalTargetLayer( + value: unknown, +): TargetLayer | undefined | null { + if (value === undefined) return undefined; + return parseTargetLayer(value); +} diff --git a/src/triggers/api.ts b/src/triggers/api.ts index 32762edb..5138e191 100644 --- a/src/triggers/api.ts +++ b/src/triggers/api.ts @@ -33,6 +33,7 @@ import { getAgentId, isAgentScopeIsolated, } from "../config.js"; +import { parseOptionalTargetLayer, TARGET_LAYER_ERROR } from "../target-layer.js"; type Response = { status_code: number; @@ -528,6 +529,13 @@ export function registerApiTriggers( if (err instanceof TimeRangeError) return timeRangeResponse(err); throw err; } + const targetLayer = parseOptionalTargetLayer(body.targetLayer); + if (targetLayer === null) { + return { + status_code: 400, + body: { error: TARGET_LAYER_ERROR }, + }; + } // #817: propagate agentId so the upstream isolation filter // applies. Honors body.agentId (POST body), ?agentId=... query // param, or implicit fallback to the worker's AGENT_ID when @@ -547,6 +555,7 @@ export function registerApiTriggers( : undefined, token_budget: body.token_budget as number | undefined, agentId: bodyAgentId ?? queryAgentId, + ...(targetLayer !== undefined ? { targetLayer } : {}), ...(timeRange ? { start_time: body.start_time as string | undefined, end_time: body.end_time as string | undefined, @@ -1461,6 +1470,7 @@ export function registerApiTriggers( source?: string; start_time?: string; end_time?: string; + targetLayer?: string; }>, ): Promise => { const authErr = checkAuth(req, secret); @@ -1481,6 +1491,13 @@ export function registerApiTriggers( body: { error: "format must be one of: full, compact, narrative" }, }; } + const targetLayer = parseOptionalTargetLayer(req.body?.targetLayer); + if (targetLayer === null) { + return { + status_code: 400, + body: { error: TARGET_LAYER_ERROR }, + }; + } // #771: route the X-Agentmemory-Source header into the payload so // the followup-rate diagnostic can skip viewer-originated calls. // Body wins if both are set (advanced callers explicitly override). @@ -1511,6 +1528,7 @@ export function registerApiTriggers( : {}), includeLessons: req.body?.includeLessons, includeInsights: req.body?.includeInsights, + ...(targetLayer !== undefined ? { targetLayer } : {}), agentId: req.body?.agentId, sessionId: req.body?.sessionId, source: req.body?.source ?? sourceFromHeader, diff --git a/test/api-boundary-coverage.test.ts b/test/api-boundary-coverage.test.ts index eeff6766..82bed619 100644 --- a/test/api-boundary-coverage.test.ts +++ b/test/api-boundary-coverage.test.ts @@ -514,6 +514,10 @@ describe("REST API boundary coverage", () => { const search = sdk.getFunction("api::search")!; await expect(search(req({ body: { query: " " } }))).resolves.toMatchObject({ status_code: 400 }); await expect(search(req({ body: { query: "x", format: "wide" } }))).resolves.toMatchObject({ status_code: 400 }); + await expect(search(req({ body: { query: "x", targetLayer: "lesson" } }))).resolves.toMatchObject({ + status_code: 400, + body: { error: "targetLayer must be one of: all, memory, observation" }, + }); await search(req({ body: { query: " auth ", @@ -523,6 +527,7 @@ describe("REST API boundary coverage", () => { format: "COMPACT", token_budget: 50, agentId: "agent-body", + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", ignored: true, @@ -539,6 +544,7 @@ describe("REST API boundary coverage", () => { format: "compact", token_budget: 50, agentId: "agent-body", + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }, @@ -956,6 +962,7 @@ describe("REST API boundary coverage", () => { body: { query: "api", includeInsights: false, + targetLayer: "memory", ignored: "drop-me", }, })); @@ -965,10 +972,29 @@ describe("REST API boundary coverage", () => { expect(call?.payload).toMatchObject({ query: "api", includeInsights: false, + targetLayer: "memory", }); expect(call?.payload).not.toHaveProperty("ignored"); }); + it("rejects invalid REST smart-search targetLayer before dispatch", async () => { + const smartSearch = sdk.getFunction("api::smart-search")!; + + const before = sdk.triggerCalls.length; + const response = await smartSearch(req({ + body: { + query: "api", + targetLayer: "lesson", + }, + })); + + expect(response).toEqual({ + status_code: 400, + body: { error: "targetLayer must be one of: all, memory, observation" }, + }); + expect(sdk.triggerCalls).toHaveLength(before); + }); + it("rejects invalid smart-search format before dispatch", async () => { const smartSearch = sdk.getFunction("api::smart-search")!; diff --git a/test/mcp-project-scope.test.ts b/test/mcp-project-scope.test.ts index 8d9654ab..3e20cf59 100644 --- a/test/mcp-project-scope.test.ts +++ b/test/mcp-project-scope.test.ts @@ -78,6 +78,7 @@ describe("MCP project scoping", () => { limit: 5, format: "compact", project: "git:repo-main", + targetLayer: "memory", }, })) as { status_code: number }; @@ -87,6 +88,7 @@ describe("MCP project scoping", () => { limit: 5, format: "compact", project: "git:repo-main", + targetLayer: "memory", }); }); @@ -104,6 +106,7 @@ describe("MCP project scoping", () => { query: "worktree auth decision", limit: 5, project: "git:repo-main", + targetLayer: "observation", }, })) as { status_code: number }; @@ -112,6 +115,7 @@ describe("MCP project scoping", () => { query: "worktree auth decision", limit: 5, project: "git:repo-main", + targetLayer: "observation", }); }); }); diff --git a/test/mcp-server-surface.test.ts b/test/mcp-server-surface.test.ts index 8572f075..3161ff7f 100644 --- a/test/mcp-server-surface.test.ts +++ b/test/mcp-server-surface.test.ts @@ -343,11 +343,21 @@ describe("MCP tools/call validation boundaries", () => { { query: "x", token_budget: 0 }, "token_budget must be a positive integer", ], + [ + "memory_recall", + { query: "x", targetLayer: "lesson" }, + "targetLayer must be one of: all, memory, observation", + ], [ "memory_smart_search", { query: "x", format: "xml" }, "format must be one of: full, compact, narrative", ], + [ + "memory_smart_search", + { query: "x", targetLayer: "lesson" }, + "targetLayer must be one of: all, memory, observation", + ], ["memory_compress_file", {}, "filePath is required for memory_compress_file"], ["memory_save", {}, "content is required for memory_save"], ["memory_file_history", {}, "files is required for memory_file_history"], @@ -575,9 +585,9 @@ describe("MCP tools/call payload shaping", () => { }, { name: "memory_smart_search", - args: { query: "auth", expandIds: [" a ", 42, "b"], limit: 500, format: "narrative", includeInsights: false }, + args: { query: "auth", expandIds: [" a ", 42, "b"], limit: 500, format: "narrative", includeInsights: false, targetLayer: "memory" }, function_id: "mem::smart-search", - payload: { query: "auth", expandIds: ["a", "b"], limit: 100, format: "narrative", includeInsights: false }, + payload: { query: "auth", expandIds: ["a", "b"], limit: 100, format: "narrative", includeInsights: false, targetLayer: "memory" }, }, { name: "memory_query", @@ -806,6 +816,7 @@ describe("MCP tools/call payload shaping", () => { token_budget: 400, agentId: " codex ", project: "git:repo", + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", limit: 3, @@ -823,6 +834,7 @@ describe("MCP tools/call payload shaping", () => { token_budget: 400, agentId: "codex", project: "git:repo", + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }, diff --git a/test/mcp-standalone-proxy.test.ts b/test/mcp-standalone-proxy.test.ts index f1a9516d..0ea8bb51 100644 --- a/test/mcp-standalone-proxy.test.ts +++ b/test/mcp-standalone-proxy.test.ts @@ -118,6 +118,7 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { limit: 5, format: " Narrative ", includeInsights: false, + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }); @@ -130,6 +131,7 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { limit: 5, format: "narrative", includeInsights: false, + targetLayer: "memory", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }); @@ -152,6 +154,23 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { expect(calledSmartSearch).toBe(false); }); + it("rejects invalid memory_smart_search targetLayer before proxying", async () => { + let calledSmartSearch = false; + installFetch((url) => { + if (url.endsWith("/agentmemory/livez")) return new Response("ok", { status: 200 }); + if (url.endsWith("/agentmemory/smart-search")) { + calledSmartSearch = true; + return new Response(JSON.stringify({ mode: "compact", results: [] }), { status: 200 }); + } + return new Response("", { status: 404 }); + }); + + await expect( + handleToolCall("memory_smart_search", { query: "auth", targetLayer: "lesson" }), + ).rejects.toThrow("targetLayer must be one of"); + expect(calledSmartSearch).toBe(false); + }); + it("forwards expandIds to /agentmemory/smart-search as an array", async () => { let searchBody: Record | undefined; installFetch((url, init) => { @@ -202,6 +221,7 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { format: "full", token_budget: 800, project: "git:repo-main", + targetLayer: "observation", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }); @@ -217,6 +237,7 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { format: "full", token_budget: 800, project: "git:repo-main", + targetLayer: "observation", start_time: "2026-06-01T00:00:00Z", end_time: "2026-06-30T23:59:59Z", }); @@ -238,6 +259,23 @@ describe("@agentmemory/mcp standalone — server proxy (issue #159)", () => { expect(recallBody).not.toHaveProperty("token_budget"); }); + it("rejects invalid memory_recall targetLayer before proxying", async () => { + let calledSearch = false; + installFetch((url) => { + if (url.endsWith("/agentmemory/livez")) return new Response("ok", { status: 200 }); + if (url.endsWith("/agentmemory/search")) { + calledSearch = true; + return new Response(JSON.stringify({ mode: "full", facts: [] }), { status: 200 }); + } + return new Response("not found", { status: 404 }); + }); + + await expect( + handleToolCall("memory_recall", { query: "x", targetLayer: "lesson" }), + ).rejects.toThrow("targetLayer must be one of"); + expect(calledSearch).toBe(false); + }); + it("proxies memory_governance_delete to the DELETE REST endpoint", async () => { const calls: Array<{ url: string; method: string; body?: unknown }> = []; installFetch((url, init) => { diff --git a/test/mcp-standalone.test.ts b/test/mcp-standalone.test.ts index 6854ad40..6f66ec5f 100644 --- a/test/mcp-standalone.test.ts +++ b/test/mcp-standalone.test.ts @@ -430,6 +430,51 @@ describe("handleToolCall", () => { ); }); + it("local fallback filters memory_recall and memory_smart_search by targetLayer", async () => { + const kv = new InMemoryKV(); + await handleToolCall( + "memory_save", + { content: "Store the target layer checklist in memory" }, + kv, + ); + + const recallMemory = JSON.parse( + ( + await handleToolCall( + "memory_recall", + { query: "target layer checklist", targetLayer: "memory" }, + kv, + ) + ).content[0].text, + ); + expect(recallMemory.results).toHaveLength(1); + expect(recallMemory.results[0].observation.narrative).toBe( + "Store the target layer checklist in memory", + ); + + const recallObservation = JSON.parse( + ( + await handleToolCall( + "memory_recall", + { query: "target layer checklist", targetLayer: "observation" }, + kv, + ) + ).content[0].text, + ); + expect(recallObservation.results).toEqual([]); + + const smartObservation = JSON.parse( + ( + await handleToolCall( + "memory_smart_search", + { query: "target layer checklist", targetLayer: "observation" }, + kv, + ) + ).content[0].text, + ); + expect(smartObservation.results).toEqual([]); + }); + it("memory_recall preserves project filtering for local fallback formats", async () => { const kv = new InMemoryKV(); await handleToolCall( @@ -469,13 +514,19 @@ describe("handleToolCall", () => { await expect( handleToolCall("memory_recall", { query: "x", token_budget: 0 }, kv), ).rejects.toThrow("token_budget must be a positive integer"); + await expect( + handleToolCall("memory_recall", { query: "x", targetLayer: "lesson" }, kv), + ).rejects.toThrow("targetLayer must be one of"); }); - it("memory_smart_search rejects invalid local fallback format values", async () => { + it("memory_smart_search rejects invalid local fallback format and targetLayer values", async () => { const kv = new InMemoryKV(); await expect( handleToolCall("memory_smart_search", { query: "x", format: "verbose" }, kv), ).rejects.toThrow("format must be one of"); + await expect( + handleToolCall("memory_smart_search", { query: "x", targetLayer: "lesson" }, kv), + ).rejects.toThrow("targetLayer must be one of"); }); it("local fallback stores and filters memories by project", async () => { diff --git a/test/mcp-surface-default.test.ts b/test/mcp-surface-default.test.ts index e9c25eab..a1c06585 100644 --- a/test/mcp-surface-default.test.ts +++ b/test/mcp-surface-default.test.ts @@ -78,9 +78,11 @@ describe("MCP tool surface default", () => { const { getAllTools } = await freshToolsRegistry(); const tools = new Map(getAllTools().map((tool) => [tool.name, tool])); expect(tools.get("memory_recall")?.inputSchema.properties.project).toBeDefined(); + expect(tools.get("memory_recall")?.inputSchema.properties.targetLayer).toBeDefined(); expect(tools.get("memory_smart_search")?.inputSchema.properties.project).toBeDefined(); expect(tools.get("memory_smart_search")?.inputSchema.properties.format).toBeDefined(); expect(tools.get("memory_smart_search")?.inputSchema.properties.includeInsights).toBeDefined(); + expect(tools.get("memory_smart_search")?.inputSchema.properties.targetLayer).toBeDefined(); }); it("plugin .mcp.json provides default env interpolation so CC parse never fails (#510)", () => { diff --git a/test/query-integration.test.ts b/test/query-integration.test.ts index 7f518f49..dbd1f64b 100644 --- a/test/query-integration.test.ts +++ b/test/query-integration.test.ts @@ -677,6 +677,34 @@ describe("mem::query", () => { }); }); + it("forwards targetLayer through search producer payloads", async () => { + const h = createHarness(); + + await h.handler({ + pipeline: [{ op: "search", query: "auth", targetLayer: "memory" }], + }); + + expect(h.triggerCalls.find((c) => c.function_id === "mem::search")?.payload) + .toMatchObject({ + query: "auth", + targetLayer: "memory", + }); + }); + + it("forwards targetLayer through smart_search producer payloads", async () => { + const h = createHarness(); + + await h.handler({ + pipeline: [{ op: "smart_search", query: "auth", targetLayer: "observation" }], + }); + + expect(h.triggerCalls.find((c) => c.function_id === "mem::smart-search")?.payload) + .toMatchObject({ + query: "auth", + targetLayer: "observation", + }); + }); + it("sessions producer applies time filters and limits", async () => { const h = createHarness(); diff --git a/test/search.test.ts b/test/search.test.ts index dccdbb73..06db7d16 100644 --- a/test/search.test.ts +++ b/test/search.test.ts @@ -7,7 +7,7 @@ vi.mock("../src/logger.js", () => ({ import { registerSearchFunction, getSearchIndex, rebuildIndex, setVectorIndex, setEmbeddingProvider, getVectorIndex } from "../src/functions/search.js"; import { VectorIndex } from "../src/state/vector-index.js"; import { KV } from "../src/state/schema.js"; -import type { CompressedObservation, Session } from "../src/types.js"; +import type { CompressedObservation, Memory, Session } from "../src/types.js"; function mockKV() { const store = new Map>(); @@ -51,6 +51,24 @@ function mockSdk() { }; } +function makeMemory(id: string, content: string, overrides: Partial = {}): Memory { + return { + id, + createdAt: "2026-02-01T00:00:00Z", + updatedAt: "2026-02-01T00:00:00Z", + type: "fact", + title: content, + content, + concepts: ["layer", "needle"], + files: [], + sessionIds: [], + strength: 7, + version: 1, + isLatest: true, + ...overrides, + }; +} + describe("mem::search", () => { let sdk: ReturnType; let kv: ReturnType; @@ -399,6 +417,87 @@ describe("mem::search", () => { expect(hit?.title).toBe("Pineapple belongs on pizza"); }); + it("filters search results by targetLayer without changing the default mixed results", async () => { + await kv.set(KV.observations("ses_1"), "obs_layer", { + id: "obs_layer", + sessionId: "ses_1", + timestamp: "2026-02-02T00:00:00Z", + type: "decision", + title: "Layer needle observation", + facts: ["layer needle shared result"], + narrative: "Observation layer needle shared result.", + concepts: ["layer", "needle"], + files: [], + importance: 8, + } satisfies CompressedObservation); + await kv.set( + KV.memories, + "mem_layer", + makeMemory("mem_layer", "Layer needle saved memory"), + ); + await rebuildIndex(kv as never); + + const all = (await sdk.trigger("mem::search", { + query: "layer needle", + format: "compact", + limit: 10, + })) as { results: Array<{ obsId: string }> }; + const memoryOnly = (await sdk.trigger("mem::search", { + query: "layer needle", + targetLayer: "memory", + format: "compact", + limit: 10, + })) as { results: Array<{ obsId: string }> }; + const observationOnly = (await sdk.trigger("mem::search", { + query: "layer needle", + targetLayer: "observation", + format: "compact", + limit: 10, + })) as { results: Array<{ obsId: string }> }; + + expect(all.results.map((r) => r.obsId).sort()).toEqual(["mem_layer", "obs_layer"]); + expect(memoryOnly.results.map((r) => r.obsId)).toEqual(["mem_layer"]); + expect(observationOnly.results.map((r) => r.obsId)).toEqual(["obs_layer"]); + }); + + it("over-fetches when targetLayer filtering would otherwise underfill the page", async () => { + await kv.set(KV.observations("ses_1"), "obs_ranked_layer", { + id: "obs_ranked_layer", + sessionId: "ses_1", + timestamp: "2026-02-02T00:00:00Z", + type: "decision", + title: "Ranked layer needle observation", + facts: ["ranked layer needle exact exact exact"], + narrative: "Ranked layer needle exact exact exact observation.", + concepts: ["ranked", "layer", "needle"], + files: [], + importance: 10, + } satisfies CompressedObservation); + await kv.set( + KV.memories, + "mem_ranked_layer", + makeMemory("mem_ranked_layer", "Ranked layer needle saved memory", { + concepts: ["ranked", "layer", "needle"], + }), + ); + await rebuildIndex(kv as never); + + const memoryOnly = (await sdk.trigger("mem::search", { + query: "ranked layer needle", + targetLayer: "memory", + format: "compact", + limit: 1, + })) as { results: Array<{ obsId: string }> }; + + expect(memoryOnly.results.map((r) => r.obsId)).toEqual(["mem_ranked_layer"]); + }); + + it("rejects invalid targetLayer values", async () => { + await expect( + sdk.trigger("mem::search", { query: "auth", targetLayer: "lesson" }), + ).rejects.toThrow("targetLayer must be one of"); + }); + it("returns external id and metadata for saved memories in compact and narrative formats", async () => { await kv.set(KV.memories, "mem_external", { id: "mem_external", diff --git a/test/smart-search.test.ts b/test/smart-search.test.ts index 13195254..84f2c51a 100644 --- a/test/smart-search.test.ts +++ b/test/smart-search.test.ts @@ -25,6 +25,7 @@ import type { HybridSearchResult, CompactSearchResult, ProceduralMemory, + Memory, SemanticMemory, Session, } from "../src/types.js"; @@ -86,6 +87,24 @@ function makeObs( }; } +function makeMemory(id: string, content: string, overrides: Partial = {}): Memory { + return { + id, + createdAt: "2026-02-01T00:00:00Z", + updatedAt: "2026-02-01T00:00:00Z", + type: "fact", + title: content, + content, + concepts: ["layer", "needle"], + files: [], + sessionIds: [], + strength: 7, + version: 1, + isLatest: true, + ...overrides, + }; +} + function makeSemantic( overrides: Partial = {}, ): SemanticMemory { @@ -785,6 +804,137 @@ describe("Smart Search Function", () => { expect(result.error).toBe("format must be one of: full, compact, narrative"); }); + it("filters compact smart-search results by targetLayer and suppresses side arrays in filtered modes", async () => { + const memoryObs = makeObs({ + id: "mem_layer", + sessionId: "memory", + title: "Layer needle saved memory", + narrative: "Layer needle saved memory", + }); + await kv.set( + KV.memories, + "mem_layer", + makeMemory("mem_layer", "Layer needle saved memory"), + ); + sdk.registerFunction("mem::lesson-recall", async () => ({ + success: true, + lessons: [ + { + id: "lsn_layer", + content: "Layer needle lesson", + context: "workflow", + confidence: 0.8, + reinforcements: 0, + source: "manual", + sourceIds: [], + tags: [], + createdAt: "2026-04-01T00:00:00Z", + updatedAt: "2026-04-01T00:00:00Z", + decayRate: 0, + score: 0.9, + }, + ], + })); + searchResults = [ + { + observation: makeObs({ id: "obs_layer", title: "Layer needle observation" }), + bm25Score: 0.9, + vectorScore: 0, + combinedScore: 0.9, + sessionId: "ses_1", + }, + { + observation: memoryObs, + bm25Score: 0.8, + vectorScore: 0, + combinedScore: 0.8, + sessionId: "memory", + }, + ]; + + const all = (await sdk.trigger("mem::smart-search", { + query: "layer needle", + includeInsights: false, + })) as { results: CompactSearchResult[]; lessons?: unknown[] }; + const memoryOnly = (await sdk.trigger("mem::smart-search", { + query: "layer needle", + targetLayer: "memory", + includeInsights: false, + })) as { + results: CompactSearchResult[]; + lessons?: unknown; + insights?: unknown; + semantic?: unknown; + procedural?: unknown; + crystals?: unknown; + }; + const observationOnly = (await sdk.trigger("mem::smart-search", { + query: "layer needle", + targetLayer: "observation", + includeInsights: false, + })) as { results: CompactSearchResult[]; lessons?: unknown }; + + expect(all.results.map((r) => r.obsId)).toEqual(["obs_layer", "mem_layer"]); + expect(all.lessons).toBeDefined(); + expect(memoryOnly.results.map((r) => r.obsId)).toEqual(["mem_layer"]); + expect(memoryOnly.lessons).toBeUndefined(); + expect(memoryOnly.insights).toBeUndefined(); + expect(memoryOnly.semantic).toBeUndefined(); + expect(memoryOnly.procedural).toBeUndefined(); + expect(memoryOnly.crystals).toBeUndefined(); + expect(observationOnly.results.map((r) => r.obsId)).toEqual(["obs_layer"]); + expect(observationOnly.lessons).toBeUndefined(); + }); + + it("over-fetches smart-search candidates when targetLayer filtering would otherwise underfill the page", async () => { + const observationHit: HybridSearchResult = { + observation: makeObs({ id: "obs_ranked_layer", title: "Ranked layer needle observation" }), + bm25Score: 1, + vectorScore: 0, + combinedScore: 1, + sessionId: "ses_1", + }; + const memoryHit: HybridSearchResult = { + observation: makeObs({ + id: "mem_ranked_layer", + sessionId: "memory", + title: "Ranked layer needle saved memory", + narrative: "Ranked layer needle saved memory", + }), + bm25Score: 0.8, + vectorScore: 0, + combinedScore: 0.8, + sessionId: "memory", + }; + await kv.set( + KV.memories, + "mem_ranked_layer", + makeMemory("mem_ranked_layer", "Ranked layer needle saved memory"), + ); + registerSmartSearchFunction(sdk as never, kv as never, async (_query, requestedLimit) => + requestedLimit === 1 ? [observationHit] : [observationHit, memoryHit], + ); + + const result = (await sdk.trigger("mem::smart-search", { + query: "ranked layer needle", + targetLayer: "memory", + limit: 1, + })) as { results: CompactSearchResult[] }; + + expect(result.results.map((r) => r.obsId)).toEqual(["mem_ranked_layer"]); + }); + + it("rejects invalid targetLayer values", async () => { + const result = (await sdk.trigger("mem::smart-search", { + query: "auth", + targetLayer: "lesson", + })) as { mode: string; error?: string; results: unknown[] }; + + expect(result.mode).toBe("compact"); + expect(result.results).toEqual([]); + expect(result.error).toBe("targetLayer must be one of: all, memory, observation"); + }); + it("expand mode returns full observations for given IDs", async () => { const result = (await sdk.trigger("mem::smart-search", { expandIds: ["obs_1"], @@ -795,6 +945,36 @@ describe("Smart Search Function", () => { expect(result.results[0].observation.title).toBe("Auth handler"); }); + it("expanded mode resolves memory IDs and filters by targetLayer", async () => { + await kv.set( + KV.memories, + "mem_layer", + makeMemory("mem_layer", "Layer needle saved memory"), + ); + await kv.set( + KV.observations("ses_1"), + "obs_layer", + makeObs({ id: "obs_layer", sessionId: "ses_1", title: "Layer needle observation" }), + ); + + const all = (await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], + })) as { mode: string; results: Array<{ obsId: string }> }; + const memoryOnly = (await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], + targetLayer: "memory", + })) as { mode: string; results: Array<{ obsId: string }> }; + const observationOnly = (await sdk.trigger("mem::smart-search", { + expandIds: ["mem_layer", "obs_layer"], + targetLayer: "observation", + })) as { mode: string; results: Array<{ obsId: string }> }; + + expect(all.mode).toBe("expanded"); + expect(all.results.map((r) => r.obsId)).toEqual(["mem_layer", "obs_layer"]); + expect(memoryOnly.results.map((r) => r.obsId)).toEqual(["mem_layer"]); + expect(observationOnly.results.map((r) => r.obsId)).toEqual(["obs_layer"]); + }); + it("filters expanded observations by time range", async () => { const oldObs = makeObs({ id: "obs_old",