feat: mind/memory MVP + trajectory archive (PR6 of OpenAI Agents SDK migration)

CLAUDE.md

@@ -29,42 +29,63 @@ quantmind/
 Key principle: QuantMind does NOT rebuild Agent runtime, lifecycle hooks, tracing,
 multi-agent handoff, or tool framework. Those come from `openai-agents`.
 
-## Current Repository State (after PR #70 / #73 / #74 / #75 / PR5)
+## Current Repository State (after PR #70 / #73 / #74 / #75 / #76 / PR6)
 
 | Module | Status | Notes |
 |--------|--------|-------|
 | `quantmind/knowledge/` | landed (PR3) | data standard with three shapes: `FlattenKnowledge` (`News` / `Earnings` / `PaperKnowledgeCard`), `TreeKnowledge` (`Paper`), `GraphKnowledge` (placeholder); shared base = `BaseKnowledge` with typed `SourceRef` / `ExtractionRef` provenance + `embedding_text()` contract |
 | `quantmind/configs/` | landed (PR3) | `BaseFlowCfg` / `BaseInput` + per-flow cfg + discriminated-union input types |
 | `quantmind/preprocess/` | landed (PR4) | `fetch/` (`fetch_arxiv` / `fetch_url` / `resolve_doi` / `read_local_file` returning `Fetched` / `RawPaper` / `CrossrefMetadata` frozen dataclasses) + `format/` (`pdf_to_markdown` via PyMuPDF, `html_to_markdown` via trafilatura) + `clean.py` + `time.py`; leaf module — only depends on `quantmind.utils` |
-| `quantmind/flows/` | landed (PR5) | apex layer: `paper_flow` (`PaperInput` → `Paper` via SDK Agent), `batch_run` + `BatchResult` (bounded-concurrency fan-out, `memory=` rejected by design), `_runner.run_with_observability` + `_compose_hooks` + `_archive_run_artifacts` (PR6 stub); only depends on configs/knowledge/preprocess/utils + `agents` SDK |
+| `quantmind/flows/` | landed (PR5, refined PR6) | apex layer: `paper_flow` (`PaperInput` → `Paper` via SDK Agent, now wired with `memory.mcp_servers()` + `memory.tools()`), `batch_run` + `BatchResult` (bounded-concurrency fan-out, `memory=` rejected by design), `_runner.run_with_observability` (PR6: `try/finally` invokes `MemoryRunHooks.persist`); depends on configs/knowledge/preprocess/utils + `mind` + `agents` SDK |
 | `quantmind/magic.py` | landed (PR5) | `resolve_magic_input(natural_language, *, target_flow, ...) -> (input, cfg)` plus `preview_resolve` debug helper; introspects flow signatures and runs a lightweight resolver Agent with `output_type=ResolvedFlowConfig[InputT, CfgT]` |
+| `quantmind/mind/memory/` | landed (PR6) | `Memory` Protocol (granular: `tools()` / `mcp_servers()` / `run_hooks()` / `reset()`) + `FilesystemMemory` MVP (MCP filesystem server via `npx`) + `MemoryRunHooks` accumulator + `RunRecord` trajectory archive under `<memory_dir>/runs/` (atomic write + `runs.jsonl` index); sixth import-linter contract pins `mind` as bounded |
 | `quantmind/utils/logger.py` | permanent | only general-purpose utility |
 
 PR5 removed the transitional packages (`quantmind/{flow,llm,config,models}/`
 and their tests under `tests/{config,models}/`); PR4 had already removed
 `quantmind/parsers/`, `quantmind/sources/`, and `quantmind/utils/tmp.py`.
-The codebase has now converged to the five permanent module roots
-(`flows/`, `configs/`, `knowledge/`, `preprocess/`, `mind/`) plus
-`magic.py` and `utils/`.
+PR6 added `quantmind/mind/memory/` and tightened the `paper_flow` /
+`_runner` signatures to consume the `Memory` Protocol directly.
+
+The codebase now has six permanent module roots (`flows/`, `configs/`,
+`knowledge/`, `preprocess/`, `mind/`) plus `magic.py` and `utils/`.
 
 `basedpyright` runs in standard mode across the whole `quantmind/`
-package — there are no per-module exclusions left. Five `import-linter`
+package — there are no per-module exclusions left. Six `import-linter`
 contracts pin the dependency graph: `utils` and `knowledge` are leaves,
 `configs` only depends on `knowledge`, `preprocess` only depends on
-`utils`, and `flows + magic` is the apex (cannot import the deleted
-transitional packages, which are listed in the contract as a tripwire
-against accidental re-introduction).
+`utils`, `flows + magic` is the apex, and `mind` is a bounded subsystem
+(cannot import `flows` / `magic` / the deleted transitional packages).
 
 ## Development Commands
 
 ### Environment
 
+One-shot bootstrap (recommended):
+
+```bash
+bash scripts/setup.sh
+```
+
+This creates `.venv`, installs Python deps (`uv pip install -e ".[dev]"`),
+and audits external (non-Python) deps via `scripts/check_system_deps.py`.
+Optional deps that are missing are reported informationally; required
+deps that are missing fail the script.
+
+Manual equivalent:
+
 ```bash
 uv venv
 source .venv/bin/activate
 uv pip install -e ".[dev]"
+.venv/bin/python scripts/check_system_deps.py   # optional audit
 ```
 
+**Adding a new external dependency** (e.g., `tiktoken` is pip; node /
+`sqlite-vec` / etc. are external): append one row to
+`scripts/check_system_deps.py::_DEPS`. README install flow does not
+change — `scripts/setup.sh` picks it up automatically.
+
 ### Verify (canonical local check)
 
 `scripts/verify.sh` is the single source of truth for "is this branch
@@ -150,7 +171,7 @@ issue instead.
 - ❌ Introduce class-based `BaseFlow` / plugin registry / hook discovery
 - ❌ Wrap `from agents import ...` in a QuantMind-side facade — use the SDK directly
 - ❌ Mix `batch_run` and `memory` (mutually exclusive in MVP; `batch_run` rejects
-  `memory=` at the signature layer — design doc §4.3.5)
+  `memory=` at the signature layer)
 - ❌ Use `Dict[str, Any]` in init functions; use Pydantic models
 - ❌ Add hard deps on observability platforms (Langfuse / Logfire / etc.); document
   integration via `add_trace_processor()` in user-facing cookbook only
@@ -173,7 +194,7 @@ issue instead.
 | #73 (merged) | Golden Harness — `scripts/verify.sh` with ruff + basedpyright + import-linter + pytest --cov, plus matching CI |
 | #74 (merged) | `knowledge/` data standard (Flatten / Tree / Graph shapes) + `configs/` skeleton; `openai-agents>=0.14` introduced for `BaseFlowCfg.model_settings` |
 | #75 (merged) | `preprocess/` (fetch + format two layers); deletes `parsers/` + `sources/` + `utils/tmp.py`; coverage floor 60→65; 4th import-linter contract |
-| PR5 (this PR) | `flows/` (`paper_flow` + `batch_run` + `BatchResult` + `_runner`) + `magic.py`; deletes `quantmind/{flow,llm,config,models}/`; coverage floor 65→75; 5th import-linter contract pins `flows + magic` as apex |
-| PR6 | `mind/memory/filesystem` MVP + trajectory archive (fills `_archive_run_artifacts` stub) |
+| #76 (merged) | `flows/` (`paper_flow` + `batch_run` + `BatchResult` + `_runner`) + `magic.py`; deletes `quantmind/{flow,llm,config,models}/`; coverage floor 65→75; 5th import-linter contract pins `flows + magic` as apex |
+| PR6 (this PR) | `mind/memory/filesystem` MVP + trajectory archive: `Memory` Protocol + `FilesystemMemory` (MCP filesystem server) + `MemoryRunHooks` + `RunRecord`; tightens `paper_flow.memory` + `_runner` to consume `Memory \| None`; replaces the PR5 `_archive_run_artifacts` stub with `try/finally` invoking `MemoryRunHooks.persist`; 6th import-linter contract pins `mind` boundaries |
 | PR7 | `mind/store/` + SQLite + `sqlite-vec` MVP; introduces `preprocess/chunk.py` with `tiktoken` |
 | PR8+ | Second flow (news/earnings) / observability cookbook / longer-term modules |

README.md

@@ -146,10 +146,28 @@ We use [uv](https://github.com/astral-sh/uv) for fast and reliable Python packag
    .venv\Scripts\activate
    ```
 
-4. **Install dependencies:**
+4. **One-shot setup (recommended):**
 
    ```bash
-   uv pip install -e .
+   bash scripts/setup.sh
+   ```
+
+   This creates `.venv`, installs Python deps (editable, with `[dev]`
+   extras), and audits non-Python dependencies (Node.js / `npx`,
+   etc.). Missing **required** deps fail the script with a clear hint;
+   missing **optional** deps are reported informationally so you know
+   what is unavailable (e.g., `FilesystemMemory` needs Node.js).
+
+   Adding a new external dependency in a future PR means appending one
+   row to `scripts/check_system_deps.py` — the install flow above
+   stays unchanged.
+
+   **Manual fallback** (equivalent to `setup.sh`):
+
+   ```bash
+   uv venv
+   uv pip install -e ".[dev]"
+   .venv/bin/python scripts/check_system_deps.py   # optional audit
    ```
 
 ### 📚 Usage Examples
@@ -224,21 +242,55 @@ async def main() -> None:
 asyncio.run(main())
 ```
 
+#### Persistent memory across a serial loop
+
+```python
+import asyncio
+
+from quantmind.configs.paper import ArxivIdentifier
+from quantmind.flows import paper_flow
+from quantmind.mind.memory import FilesystemMemory
+
+
+async def main() -> None:
+    mem = FilesystemMemory("./.qm-memory")
+    arxiv_ids = ["1706.03762", "1810.04805", "2005.11401"]
+    for paper_id in arxiv_ids:
+        paper = await paper_flow(
+            ArxivIdentifier(id=paper_id),
+            memory=mem,
+        )
+        print(paper.title)
+    # Trajectory records are now under ./.qm-memory/runs/.
+
+
+asyncio.run(main())
+```
+
+`FilesystemMemory` requires Node.js + `npx` on PATH (the SDK launches
+`@modelcontextprotocol/server-filesystem` over stdio). The Agent sees
+`<memory_dir>/workspace/` (`notes/` and `items/`); system trajectory
+records stay outside that MCP root under `<memory_dir>/runs/<run_id>.json`
+and `<memory_dir>/runs.jsonl`. `FilesystemMemory` is for serial loops
+only — `batch_run` rejects `memory=` at the signature layer.
+
 > **Note**: QuantMind is mid-migration to OpenAI Agents SDK
-> (see [#71](https://github.com/LLMQuant/quant-mind/issues/71)). PR5 lands the
-> apex layer (`flows/` + `magic.py`); the remaining work is the `mind/`
-> memory + store layer scheduled for PR6 and PR7.
+> (see [#71](https://github.com/LLMQuant/quant-mind/issues/71)). PR6 lands
+> `mind/memory/` (Memory Protocol + `FilesystemMemory` MVP + trajectory
+> archive); the remaining work is the `mind/store/` knowledge layer
+> scheduled for PR7+.
 
 ---
 
 ### 🗺️ Roadmap
 
 - [x] Better `flow` design for user-friendly usage
 - [x] First production level example (Quant Paper Agent)
-- [ ] Migrate Agent layer to OpenAI Agents SDK
-- [ ] Standardize knowledge format with `knowledge/` (Pydantic-based)
+- [x] Migrate Agent layer to OpenAI Agents SDK
+- [x] Standardize knowledge format with `knowledge/` (Pydantic-based)
+- [x] Cross-step working memory (`mind/memory`) for serial document processing
 - [ ] Additional content sources (financial news, blogs, reports)
-- [ ] Cross-step working memory (`mind/memory`) for batch document processing
+- [ ] `mind/store/` — durable knowledge store with hybrid retrieval (PR7+)
 
 ---
 
@@ -252,18 +304,10 @@ QuantMind is designed with a larger vision: to become a comprehensive intelligen
 The foundation we're building today—starting with papers—will expand to encompass the entire financial information ecosystem.
 
 > [!NOTE]
-> **Future Conceptual Example (PR6 brings `FilesystemMemory`):**
->
-> ```python
-> from quantmind.configs.paper import ArxivIdentifier
-> from quantmind.flows import paper_flow
-> from quantmind.knowledge import Paper
-> from quantmind.mind.memory import FilesystemMemory  # PR6
->
-> memory = FilesystemMemory("./mem/factor-research/")
-> for arxiv_id in arxiv_ids:
->     paper: Paper = await paper_flow(ArxivIdentifier(id=arxiv_id), memory=memory)
-> ```
+> **`FilesystemMemory` landed in PR6.** See the runbook example above
+> (*Persistent memory across a serial loop*) for the canonical usage.
+> Future cognitive layers (`mind/store`, `mind/summarize_run`) build on
+> this foundation — they share the same `<memory_dir>/` layout.
 
 This future state represents our commitment to moving beyond simple data aggregation and toward genuine machine intelligence in the financial domain.
 

examples/memory/01_basic.py

@@ -0,0 +1,63 @@
+"""01 — The shortest memory run.
+
+Builds a ``FilesystemMemory``, runs ``paper_flow`` once with it, then
+prints what the trajectory archive looks like on disk afterwards.
+
+What to look for:
+
+- ``./.qm-memory/workspace/`` is created with ``notes/`` ``items/`` and
+  a seeded ``README.md`` (the agent's own usage guide for the dir).
+- One ``./.qm-memory/runs/<run_id>.json`` file lands per
+  ``paper_flow`` call.
+- ``./.qm-memory/runs.jsonl`` gets one new line per call (a
+  denormalised index of the per-run files — handy for `jq` / `pandas`).
+
+Prerequisites: OPENAI_API_KEY in env, Node.js + npx on PATH.
+
+Run:
+    python examples/memory/01_basic.py
+"""
+
+import asyncio
+from pathlib import Path
+
+from quantmind.configs.paper import RawText
+from quantmind.flows import paper_flow
+from quantmind.mind.memory import FilesystemMemory
+
+
+async def main() -> None:
+    memory_dir = Path("./.qm-memory")
+    mem = FilesystemMemory(memory_dir)
+
+    # A tiny paper-shaped input keeps cost low; swap in
+    # ``ArxivIdentifier(id="...")`` to see a real extraction end to end.
+    paper = await paper_flow(
+        RawText(
+            text=(
+                "# Toy paper\n\n"
+                "Title: Cross-sectional momentum, simplified\n"
+                "Author: Demo\n\n"
+                "Body: We illustrate momentum on a 3-stock universe."
+            )
+        ),
+        memory=mem,
+    )
+    print(f"Extracted paper: {paper.title!r}")
+
+    # Show what landed under .qm-memory/.
+    print(f"\nMemory layout under {memory_dir}:")
+    for child in sorted(memory_dir.rglob("*")):
+        if child.is_file():
+            print(
+                f"  {child.relative_to(memory_dir)}  ({child.stat().st_size}B)"
+            )
+
+    runs_jsonl = memory_dir / "runs.jsonl"
+    if runs_jsonl.exists():
+        print(f"\nLast line of {runs_jsonl}:")
+        print(runs_jsonl.read_text().splitlines()[-1][:200] + " ...")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

examples/memory/02_serial_loop.py

@@ -0,0 +1,75 @@
+"""02 — Serial loop sharing one FilesystemMemory.
+
+The point of cross-step memory is letting run N see what runs 1..N-1
+left behind. ``FilesystemMemory`` exposes ``notes/`` and ``items/`` to
+the Agent through an MCP filesystem server, so the Agent can list /
+read / write files there during its own turn.
+
+What to look for:
+
+- ``./.qm-memory/runs/`` accumulates one trajectory record per loop
+  iteration (3 here).
+- ``./.qm-memory/workspace/notes/`` may grow if the Agent decides to
+  write notes while extracting (it sees the seeded ``README.md`` and is
+  encouraged to do so).
+- ``./.qm-memory/runs.jsonl`` has 3 appended lines after this script.
+
+This is the "memory-accumulating workflow" pattern. ``batch_run``
+rejects ``memory=`` at the signature level — for memory accumulation
+you write the loop yourself, exactly like below.
+
+Prerequisites: OPENAI_API_KEY, Node.js + npx, network.
+
+Run:
+    python examples/memory/02_serial_loop.py
+"""
+
+import asyncio
+from pathlib import Path
+
+from quantmind.configs.paper import RawText
+from quantmind.flows import paper_flow
+from quantmind.mind.memory import FilesystemMemory
+
+_FAKE_PAPERS = [
+    (
+        "# Momentum returns 1\n"
+        "Title: Cross-sectional momentum on US equities\n"
+        "Body: Long winners short losers monthly."
+    ),
+    (
+        "# Momentum returns 2\n"
+        "Title: Time-series momentum on commodities\n"
+        "Body: 12-month look-back, monthly rebalance."
+    ),
+    (
+        "# Momentum returns 3\n"
+        "Title: Combining XS and TS momentum\n"
+        "Body: 50/50 equal weight composite."
+    ),
+]
+
+
+async def main() -> None:
+    mem = FilesystemMemory(Path("./.qm-memory"))
+
+    for idx, body in enumerate(_FAKE_PAPERS, start=1):
+        print(f"\n=== run {idx}/3 ===")
+        paper = await paper_flow(RawText(text=body), memory=mem)
+        print(f"  -> {paper.title!r}")
+
+    # Quick post-run summary.
+    runs_dir = mem.memory_dir / "runs"
+    print(
+        f"\nTrajectory files: {len(list(runs_dir.glob('*.json')))} in {runs_dir}"
+    )
+
+    notes_dir = mem.workspace / "notes"
+    notes = list(notes_dir.glob("*"))
+    print(f"Notes the agent left: {len(notes)} in {notes_dir}")
+    for n in notes[:5]:
+        print(f"  - {n.name}")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())