Add target layer filters to memory search

README.md

@@ -1171,6 +1171,8 @@ Use `memory_lesson_save` for reusable workflow guidance: rules, pitfalls, and "p
 
 `memory_save`, `mem::remember`, and `POST /agentmemory/remember` accept optional `external_id` and `metadata` fields for memories imported from another system or evaluation dataset. `external_id` is a stable caller-owned source identifier; `metadata` must be a small public object. Secret-like metadata keys and values are redacted and oversized metadata is bounded before storage. `memory_recall` and `memory_smart_search` include `external_id` and `metadata` in compact and narrative results when present; full results include them on the returned observation object.
 
+`memory_recall`, `memory_smart_search`, `POST /agentmemory/search`, and `POST /agentmemory/smart-search` accept optional `targetLayer: "all" | "memory" | "observation"` to restrict primary results to saved memories or session observations. Filtered `memory_smart_search` calls omit lesson, insight, semantic, procedural, and crystal side arrays so the response only contains the requested layer.
+
 `memory_lesson_save` accepts `content` (required), plus optional `context`, `confidence`, `project`, and comma-separated `tags`. New lessons default to `confidence: 0.5` unless a value from `0.0` to `1.0` is provided; out-of-range values fall back to `0.5`. Saving the same lesson content again in the same `project` and `source` strengthens the existing lesson instead of creating a duplicate.
 
 Lesson confidence changes through reinforcement and decay:

docs/todos/2026-06-19-issue-328-smart-search-layer-filter/plan.md

@@ -0,0 +1,332 @@
+# Smart Search Layer Filter Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Add an additive `targetLayer` selector to existing recall/search surfaces so agents can search only saved memories or only observations without changing default behavior.
+
+**Architecture:** Keep the selector as request-time retrieval filtering, not storage or index structure. Normalize accepted values at system boundaries and function entry points, then over-fetch whenever the selector is not `all` so post-resolution filtering can still fill the requested limit. Apply the selector after candidate rows are resolved to either `KV.memories` or `KV.observations`; smart-search filtered modes suppress non-primary side arrays.
+
+**Tech Stack:** TypeScript ESM, iii-sdk registered functions/triggers, MCP tool schema/handlers, standalone MCP shim, Vitest.
+
+---
+
+## Files
+
+- Modify: `src/functions/search.ts`
+  - Add `targetLayer?: string` input, normalize to `all | memory | observation`, over-fetch when filtered, filter resolved candidates by source.
+- Modify: `src/functions/smart-search.ts`
+  - Add `targetLayer?: string`, over-fetch when filtered, filter hybrid/expanded rows by source, suppress side arrays when `targetLayer` is not `all`.
+- Modify: `src/mcp/tools-registry.ts`
+  - Add `targetLayer` schema property to `memory_recall` and `memory_smart_search`.
+- Modify: `src/mcp/server.ts`
+  - Validate and forward `targetLayer` in both MCP handlers.
+- Modify: `src/triggers/api.ts`
+  - Validate and whitelist `targetLayer` in `/agentmemory/search` and `/agentmemory/smart-search`.
+- Modify: `src/mcp/standalone.ts`
+  - Validate selector, forward in proxy mode, apply in local fallback.
+- Modify: `src/functions/query.ts`
+  - Include `targetLayer` in `search` and `smart_search` producer whitelists.
+- Modify: `README.md` and generated/plugin references only if they document parameters, without changing tool or endpoint counts.
+- Test: `test/search.test.ts`
+- Test: `test/smart-search.test.ts`
+- Test: `test/mcp-server-surface.test.ts`
+- Test: `test/mcp-surface-default.test.ts`
+- Test: `test/api-boundary-coverage.test.ts`
+- Test: `test/mcp-standalone.test.ts`
+- Test: `test/mcp-standalone-proxy.test.ts`
+- Test: `test/query-integration.test.ts` or the nearest existing query test file.
+
+## Task 1: Search Function Target Layer
+
+**Files:**
+- Modify: `test/search.test.ts`
+- Modify: `src/functions/search.ts`
+
+- [ ] **Step 1: Write failing `mem::search` tests**
+
+Add tests that seed one session observation and one saved memory with matching text, then assert:
+
+```ts
+const all = await sdk.trigger("mem::search", { query: "layer needle", limit: 10 });
+expect(all.results.map((r: any) => r.observation.id).sort()).toEqual(["mem_layer", "obs_layer"]);
+
+const memoryOnly = await sdk.trigger("mem::search", {
+  query: "layer needle",
+  targetLayer: "memory",
+  limit: 10,
+});
+expect(memoryOnly.results.map((r: any) => r.observation.id)).toEqual(["mem_layer"]);
+
+const observationOnly = await sdk.trigger("mem::search", {
+  query: "layer needle",
+  targetLayer: "observation",
+  limit: 10,
+});
+expect(observationOnly.results.map((r: any) => r.observation.id)).toEqual(["obs_layer"]);
+```
+
+Also add an invalid selector assertion:
+
+```ts
+await expect(
+  sdk.trigger("mem::search", { query: "layer needle", targetLayer: "lesson" }),
+).rejects.toThrow("targetLayer must be one of");
+```
+
+Add an over-fetch regression where `limit: 1`, the first-ranked result is an observation, and a matching memory ranks below it:
+
+```ts
+const memoryOnly = await sdk.trigger("mem::search", {
+  query: "ranked layer needle",
+  targetLayer: "memory",
+  limit: 1,
+});
+expect(memoryOnly.results.map((r: any) => r.observation.id)).toEqual(["mem_ranked_layer"]);
+```
+
+- [ ] **Step 2: Run the focused red test**
+
+Run: `corepack pnpm exec vitest run test/search.test.ts -t "targetLayer"`
+
+Expected: FAIL because `targetLayer` is ignored, invalid values are not rejected, or selector filtering under-fills after the first mixed-layer page.
+
+- [ ] **Step 3: Implement minimal `mem::search` filtering**
+
+Add a local type and normalizer:
+
+```ts
+type TargetLayer = "all" | "memory" | "observation";
+
+function parseTargetLayer(value: unknown, context: string): TargetLayer {
+  if (value === undefined || value === null || value === "") return "all";
+  if (typeof value !== "string") {
+    throw new Error(`${context}: targetLayer must be one of: all, memory, observation`);
+  }
+  const normalized = value.trim().toLowerCase();
+  if (normalized === "all" || normalized === "memory" || normalized === "observation") {
+    return normalized;
+  }
+  throw new Error(`${context}: targetLayer must be one of: all, memory, observation`);
+}
+```
+
+When resolving candidates, keep source metadata:
+
+```ts
+return mem ? { observation: memoryToObservation(mem), targetLayer: "memory" as const } : null;
+```
+
+and for observation hits:
+
+```ts
+return { observation: obs, targetLayer: "observation" as const };
+```
+
+Filter before pushing enriched results:
+
+```ts
+if (targetLayer !== "all" && resolved.targetLayer !== targetLayer) continue;
+```
+
+Make selector filtering part of the over-fetch decision:
+
+```ts
+const layerFiltering = targetLayer !== "all";
+const filtering = !!(projectFilter || cwdFilter || filterAgentId || timeRange || layerFiltering);
+```
+
+- [ ] **Step 4: Run green search test**
+
+Run: `corepack pnpm exec vitest run test/search.test.ts -t "targetLayer"`
+
+Expected: PASS.
+
+## Task 2: Smart Search Target Layer
+
+**Files:**
+- Modify: `test/smart-search.test.ts`
+- Modify: `src/functions/smart-search.ts`
+
+- [ ] **Step 1: Write failing smart-search tests**
+
+Add tests that seed matching observation and memory rows and assert:
+
+```ts
+const result = await sdk.trigger("mem::smart-search", {
+  query: "layer needle",
+  targetLayer: "memory",
+  limit: 10,
+});
+expect(result.results.map((r: any) => r.obsId)).toEqual(["mem_layer"]);
+expect(result.lessons).toBeUndefined();
+expect(result.insights).toBeUndefined();
+expect(result.semantic).toBeUndefined();
+expect(result.procedural).toBeUndefined();
+expect(result.crystals).toBeUndefined();
+```
+
+Add observation-only and invalid selector tests.
+
+Add an over-fetch regression where `limit: 1`, the first hybrid hit is the wrong layer, and a memory hit below it is still returned for `targetLayer: "memory"`.
+
+Add expanded-mode tests:
+
+```ts
+const all = await sdk.trigger("mem::smart-search", {
+  expandIds: ["mem_layer", "obs_layer"],
+});
+expect(all.results.map((r: any) => r.obsId)).toEqual(["mem_layer", "obs_layer"]);
+
+const memoryOnly = await sdk.trigger("mem::smart-search", {
+  expandIds: ["mem_layer", "obs_layer"],
+  targetLayer: "memory",
+});
+expect(memoryOnly.results.map((r: any) => r.obsId)).toEqual(["mem_layer"]);
+
+const observationOnly = await sdk.trigger("mem::smart-search", {
+  expandIds: ["mem_layer", "obs_layer"],
+  targetLayer: "observation",
+});
+expect(observationOnly.results.map((r: any) => r.obsId)).toEqual(["obs_layer"]);
+```
+
+- [ ] **Step 2: Run red smart-search tests**
+
+Run: `corepack pnpm exec vitest run test/smart-search.test.ts -t "targetLayer"`
+
+Expected: FAIL because smart-search ignores the selector, does not over-fetch for target-layer filtering, cannot expand memory IDs, or still emits side arrays in filtered modes.
+
+- [ ] **Step 3: Implement smart-search filtering**
+
+Normalize `targetLayer` at function start. Add source resolution using `KV.memories` lookups for hybrid hits where needed, mirroring `makeProjectMatcher` cache style. Make `targetLayer !== "all"` part of the smart-search over-fetch condition, filter by source before slicing to `limit`, and add a source-aware expanded resolver that falls back to `KV.memories`.
+
+Set include flags for filtered modes:
+
+```ts
+const includeSideArrays = targetLayer === "all";
+const includeLessons = includeSideArrays && data.includeLessons !== false;
+const includeInsights = includeSideArrays && highOrderEnabled && data.includeInsights !== false;
+```
+
+Only assign high-order arrays when `includeSideArrays` is true.
+
+- [ ] **Step 4: Run green smart-search tests**
+
+Run: `corepack pnpm exec vitest run test/smart-search.test.ts -t "targetLayer"`
+
+Expected: PASS.
+
+## Task 3: Public Adapters
+
+**Files:**
+- Modify: `src/mcp/tools-registry.ts`
+- Modify: `src/mcp/server.ts`
+- Modify: `src/triggers/api.ts`
+- Modify: `src/mcp/standalone.ts`
+- Modify: `src/functions/query.ts`
+- Modify: related adapter tests
+
+- [ ] **Step 1: Write failing adapter tests**
+
+Add assertions that:
+
+```ts
+expect(tools.get("memory_recall")?.inputSchema.properties.targetLayer).toBeDefined();
+expect(tools.get("memory_smart_search")?.inputSchema.properties.targetLayer).toBeDefined();
+```
+
+MCP handler payload assertions should include:
+
+```ts
+args: { query: "auth", targetLayer: "memory" }
+payload: { query: "auth", targetLayer: "memory" }
+```
+
+REST whitelist tests should assert unknown fields are dropped but `targetLayer` is forwarded. Standalone proxy tests should assert request bodies include `targetLayer`.
+
+Add invalid-value tests for MCP `memory_recall` and `memory_smart_search`, REST `/search` and `/smart-search`, standalone proxy/local fallback, and function-level calls. All should reject `targetLayer: "lesson"` before dispatch where a boundary handler exists.
+
+Add `memory_query` producer tests that assert both `search` and `smart_search` pipeline steps forward `targetLayer` unchanged.
+
+- [ ] **Step 2: Run red adapter tests**
+
+Run:
+
+```bash
+corepack pnpm exec vitest run test/mcp-server-surface.test.ts test/mcp-surface-default.test.ts test/api-boundary-coverage.test.ts test/mcp-standalone.test.ts test/mcp-standalone-proxy.test.ts test/query-integration.test.ts
+```
+
+Expected: FAIL on missing `targetLayer` schema/forwarding/validation or missing query producer forwarding.
+
+- [ ] **Step 3: Implement adapter forwarding**
+
+Add the schema property description:
+
+```ts
+targetLayer: {
+  type: "string",
+  description: "Optional result source filter: all, memory, or observation (default all). Use memory for saved memory_save entries.",
+}
+```
+
+Validate at MCP and REST boundaries with the same accepted values and safe error text. Add `targetLayer` to standalone `Validated`, proxy request bodies, local fallback filtering, and `memory_query` producer `pick()` lists. Standalone local fallback stores only memories; `targetLayer: "observation"` returns the normal empty result shape rather than reading other storage.
+
+- [ ] **Step 4: Run green adapter tests**
+
+Run the same adapter test command.
+
+Expected: PASS.
+
+## Task 4: Docs, Cleanup, And Verification
+
+**Files:**
+- Modify: `README.md`
+- Modify: `docs/todos/2026-06-19-issue-328-smart-search-layer-filter/todo.md`
+
+- [ ] **Step 1: Document the selector**
+
+Update the MCP tool docs section so `memory_recall` and `memory_smart_search` mention `targetLayer`. Do not change MCP tool counts or REST endpoint counts.
+
+- [ ] **Step 2: Focused simplification pass**
+
+Inspect the active diff for duplicated selector parsing or unclear naming. Keep API contracts unchanged.
+
+- [ ] **Step 3: Run targeted verification**
+
+Run:
+
+```bash
+corepack pnpm exec vitest run test/search.test.ts test/smart-search.test.ts test/mcp-server-surface.test.ts test/mcp-surface-default.test.ts test/api-boundary-coverage.test.ts test/mcp-standalone.test.ts test/mcp-standalone-proxy.test.ts test/query-integration.test.ts
+```
+
+Expected: PASS.
+
+- [ ] **Step 4: Run broader repo-native checks as feasible**
+
+Run:
+
+```bash
+corepack pnpm run lint
+corepack pnpm test
+```
+
+Expected: PASS, or record blocker and closest targeted evidence.
+
+- [ ] **Step 5: Security gates before any commit/PR readiness claim**
+
+Because this changes MCP/REST API contracts and retrieval filtering, run required local gates before commit/PR readiness:
+
+```bash
+semgrep scan --config p/default --error --metrics=off .
+gitleaks protect --staged --redact
+```
+
+OSV is not required unless dependency, lockfile, vendored, container, or third-party package surfaces change.
+
+## Review Notes
+
+Spec coverage: Covers issue #328, approved `targetLayer` contract, default compatibility, MCP/REST/standalone/query propagation, smart-search side arrays, docs, and verification.
+
+Placeholder scan: No TBD placeholders remain.
+
+Type consistency: Use canonical `targetLayer` and values `all`, `memory`, `observation` throughout.