From 87e5f5e5f3976f32abcd7b594e7243785dbcf329 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 18:04:56 -0400 Subject: [PATCH 01/11] Draft single-score citation ranking and bounded shadow retrieval Track surfaced discoveries with deterministic identities and an atomic local SQLite ledger shared across worktrees. Add bounded symbol-aware retrieval, stable pagination, per-entry expansion and retry-safe counting while preserving canonical knowledge formats and hook fail-open behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 10 + README.md | 8 + RESPONSIBLE_AI.md | 8 + agent-context.md | 17 +- claude.md | 15 + .../scripts/shadow-frog-pre-tool.sh | 2 + skills/shadow-frog-viewer/SKILL.md | 80 +- skills/shadow-frog-viewer/_citations.py | 163 ++++ skills/shadow-frog-viewer/shadow-viewer.py | 881 ++++++++++-------- skills/shadow-frog/SKILL.md | 35 +- tests/hooks/test_pre_tool_sh.py | 18 + tests/skills/shadow_frog_viewer/conftest.py | 9 + .../shadow_frog_viewer/test_citations.py | 166 ++++ .../shadow_frog_viewer/test_retrieval.py | 431 +++++++++ .../shadow_frog_viewer/test_shadow_viewer.py | 11 +- 15 files changed, 1428 insertions(+), 426 deletions(-) create mode 100644 skills/shadow-frog-viewer/_citations.py create mode 100644 tests/skills/shadow_frog_viewer/conftest.py create mode 100644 tests/skills/shadow_frog_viewer/test_citations.py create mode 100644 tests/skills/shadow_frog_viewer/test_retrieval.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 875a339..f10b22f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,17 @@ shadow knowledge bases for any codebase. ## Unreleased +### Added +- **Single-score knowledge retrieval (draft)** — derived discovery fingerprints, + zero-default local citation scores, and concurrent-safe SQLite bookkeeping + shared across Git worktrees. Bounded search/symbol views, stable pagination, + and individual expansion avoid loading entire large shadow sections. + Citation scores measure emitted content, not correctness or proven usefulness. + ### Changed +- Retrieval views now show IDs/scores and paginate by default. Preferences + remain separately accessible, trust and relevance precede popularity, and + hooks remain fail-open while surfacing citation warnings. - **More concise documentation** — consolidated README onboarding and workflow guidance, with advanced operations linked to the skill references. Condensed repeated guidance and examples in the core, Dream, Init, Meditate, Update, diff --git a/README.md b/README.md index 8b6b37f..9c73b38 100644 --- a/README.md +++ b/README.md @@ -110,6 +110,7 @@ For example, use Viewer to find relevant knowledge or audit its structure: ``` /shadow-frog-viewer --search "auth" +/shadow-frog-viewer --symbol src/auth.py::login /shadow-frog-viewer --top src/auth.py /shadow-frog-viewer --check-invariants ``` @@ -117,6 +118,13 @@ For example, use Viewer to find relevant knowledge or audit its structure: The [Viewer reference](skills/shadow-frog-viewer/SKILL.md) also covers summaries, recent discoveries, label filters, preferences, and interactive dream-lineage HTML. +Knowledge retrieval is bounded and pageable, with IDs for expanding individual +claims. A single local `citation_score` counts helper exposures, not proven +usefulness; relevance and trust outrank popularity. Scores are updated safely +across local Git worktrees without editing shadow Markdown or requiring a vector +index. New claims start at zero. See the Viewer reference for retries, local +storage, and the limits of this signal. + --- ## Choose Dream or Nap diff --git a/RESPONSIBLE_AI.md b/RESPONSIBLE_AI.md index 94ea957..5a198ad 100644 --- a/RESPONSIBLE_AI.md +++ b/RESPONSIBLE_AI.md @@ -98,6 +98,14 @@ At a high level, we found that ShadowFrog performed strongly on knowledge retrie ## Limitations +Citation scores are local counts of discovery content emitted by the viewer, +not proof that an agent used it, that it improved an outcome, or that it is true. +Raw file reads and failed ledger updates are not counted. Scores can be biased +by prior ranking and repeated exposure; relevance, provenance, and verification +remain more important. Bounded results may omit relevant knowledge, so agents +must follow pagination and inspect preferences rather than treating a shortlist +as exhaustive. Claim rewrites and renames may create fresh zero-score identities. + ShadowFrog was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios. ShadowFrog was designed and tested using the English language. Performance in other languages may vary and should be assessed by someone who is both an expert in the expected outputs and a native speaker of that language. diff --git a/agent-context.md b/agent-context.md index 11db20a..98af356 100644 --- a/agent-context.md +++ b/agent-context.md @@ -2,17 +2,22 @@ This project uses a `.shadow/` knowledge base with verified discoveries about non-obvious code behavior. **You MUST consult the shadow before making any code change.** -1. **Check the shadow first** — before editing any file, read its shadow: - ``` - cat .shadow/.md - ``` -2. **Check preferences** — `cat .shadow/_prefs.md` for project conventions -3. **Check cross-cutting** — `cat .shadow/_cross/*.md` for multi-file patterns +1. **Check the shadow first** — use `/shadow-frog-viewer --symbol ::` + or `--top ` before editing; expand returned IDs with `--get` and page + large results rather than loading an entire shadow into context. +2. **Check preferences** — use `--prefs` and follow all pages for project conventions. +3. **Check cross-cutting** — use `--search` for related multi-file knowledge. 4. **Act on what you find** — apply what you learn from the shadow to your work. 5. **After making changes** — run `/shadow-frog-update` to capture learnings The shadow contains discoveries from code analysis and user conversations. Always consult it before making assumptions about code behavior. +The viewer records one local `citation_score` per emitted discovery, atomically +across worktrees; never edit counters in Markdown. It is an exposure count, not +proof of correctness or usefulness. Trust and relevance outrank popularity. +Raw file reads are a fallback and are not counted. A ranked shortlist is not +exhaustive: search the specific claim before adding duplicate knowledge. + ### Key directories - `.shadow/.md` — per-file shadows with symbol-level discoveries diff --git a/claude.md b/claude.md index 2d623e6..babdf92 100644 --- a/claude.md +++ b/claude.md @@ -32,6 +32,7 @@ ShadowFrog/ shadow-frog-viewer/ Browse and query the shadow knowledge base SKILL.md Query instructions + shell fallbacks shadow-viewer.py Python helper script + _citations.py Atomic local citation ledger and retrieval cursors dream-lineage.py Dream lineage visualization hook-templates/ shadow-frog-hooks.json Copilot CLI hook config (sessionStart, preToolUse) @@ -116,6 +117,20 @@ Preference (`_prefs.md` — project-wide, no file/symbol anchor): - Do-based: write and run a short test/script to confirm or refute. - `source: user` and `source: interaction` → always `verified`. +### Citation-Aware Retrieval + +- Keep the discovery grammar unchanged: viewer fingerprints and `citation_score` + are derived/local metadata, not additional Markdown fields. +- Scores start at zero and increment only for content emitted by a retrieval + view, once per entry/event. They measure exposure, not verified usefulness. +- Use the common-Git SQLite ledger for multiprocess/worktree updates; do not + rewrite shadow files on reads. Telemetry failures must warn without hiding + knowledge. Summary/audit/parser-only operations do not increment scores. +- Rank relevance and trust ahead of scores; preserve room for zero-score entries. + Page large results, expand by ID, and never use only the shortlist for dedup. +- Fingerprints bind kind, anchor, normalized claim and refs. Metadata-only edits + retain identity; rewritten/merged claims and renamed anchors may reset scores. + ### Dedup - Before writing, read existing discoveries at the target symbol. - Same claim → update existing. Extends existing → merge. Contradicts → keep both, mark weaker `refuted`. diff --git a/hook-templates/scripts/shadow-frog-pre-tool.sh b/hook-templates/scripts/shadow-frog-pre-tool.sh index 52adb33..10dcbf7 100755 --- a/hook-templates/scripts/shadow-frog-pre-tool.sh +++ b/hook-templates/scripts/shadow-frog-pre-tool.sh @@ -188,6 +188,8 @@ if viewer: ) if r.returncode == 0: sys.stdout.write(r.stdout.strip()) + if r.stderr.strip(): + sys.stdout.write("\n[ShadowFrog] Viewer reported a warning; rerun it directly for details. Citation updates may be unavailable.") except Exception: pass PYEOF diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 3c0e68e..13ce939 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -1,8 +1,9 @@ --- name: shadow-frog-viewer description: >- - Browse and query the shadow knowledge base: overview, search for files - or symbols or text, view preferences, or see recent discoveries. + Browse and query the shadow knowledge base with bounded, citation-ranked + retrieval: search files, symbols, or text, expand individual discoveries, + page through preferences, or see recent discoveries. Invoke when the user wants to see what's in the shadow, get an overview, or find specific knowledge. scripts: @@ -31,11 +32,13 @@ python3 .claude/skills/shadow-frog-viewer/shadow-viewer.py [options] | Command | What it shows | |---------|--------------| | `--summary` | Overview: counts, source/status/label breakdown, per-file table, cross-cutting titles (default) | -| `--search QUERY` | Universal search — matches file names, symbol names, and discovery text. Includes cross-cutting and preferences | -| `--prefs` | Project-wide preferences | -| `--recent [N]` | N most recent discoveries with full content (default: 10) | -| `--labels LABEL` | Discoveries filtered by label (e.g., `bug`, `security`, `bug,performance`) | -| `--top FILE` | Top actionable discoveries for FILE — concise output (default: 3 entries, ~600 chars) suitable for the preToolUse hook. Includes both per-file shadow entries and any `_cross/` discoveries that reference FILE. Verified discoveries rank first. | +| `--search QUERY` | Bounded search across paths, symbols, text, cross-cutting entries, and preferences | +| `--symbol FILE::SYMBOL` | Bounded discoveries at an exact symbol, plus matching cross-cutting refs; use `File-Level` for a file-level section | +| `--get ID` | Expand one current discovery returned by a content view | +| `--prefs` | Project-wide preferences; follow all pages before treating them as complete | +| `--recent [N]` | Most recent discovery previews by file mtime (default page size: 10) | +| `--labels LABEL` | Bounded discoveries matching labels (e.g., `bug`, `security`, `bug,performance`) | +| `--top FILE` | Hook-sized actionable previews (default: up to 3 entries, 600 characters). Includes per-file and cross-cutting findings; trust/status precedes citation score. | | `--check-invariants` | Audit structural integrity — bidirectional cross-references, label/source/category enum compliance, heading format, no-orphan-back-pointer. Exits 0 if clean, 1 with one violation per line. Run after dream reconciliation or before commit. | No arguments defaults to `--summary`. @@ -45,9 +48,15 @@ No arguments defaults to `--summary`. | Flag | Effect | |------|--------| | `--shadow-dir DIR` | Override .shadow/ location (default: auto-detect from CWD) | +| `--limit N` | Positive page size for search, symbol, labels, or preferences (default: 10) | +| `--max-chars N` | Hard output cap, including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output). Use `--top-max-chars` with `--top`. | +| `--cursor TOKEN` | Continue the same view and filters using the returned ordering snapshot | +| `--text-offset N` | Continue a long `--get` result at the returned character offset | +| `--event-id ID` | Optional retry ID: each discovery counts at most once per ID (1-128 letters/digits or `. _ : -`). Default: a new event per invocation. | +| `--no-record` | Do not increase citation scores; existing scores still rank results, and pagination may store a local snapshot | | `--top-labels LABELS` | Comma-separated label filter for `--top` (default: `bug,security`). Empty string disables label filtering. | | `--top-limit N` | Max discoveries to show in `--top` (default: 3) | -| `--top-max-chars N` | Hard cap on `--top` total output length (default: 600). Use 0 for no cap. | +| `--top-max-chars N` | Hard cap on `--top` total output length (default: 600; minimum 256). Use 0 for no cap. | ### Examples @@ -55,6 +64,13 @@ No arguments defaults to `--summary`. # Search file names, symbols, discoveries, cross-cutting entries, and preferences python3 shadow-viewer.py --search "token expiry" +# Inspect one symbol without loading its entire shadow +python3 shadow-viewer.py --symbol src/auth.py::UserAuth.validate --limit 5 + +# Expand a returned id, or continue the same search with its returned cursor +python3 shadow-viewer.py --get DISCOVERY_ID +python3 shadow-viewer.py --search "token expiry" --cursor CURSOR_TOKEN + # Security and performance issues python3 shadow-viewer.py --labels security,performance @@ -62,6 +78,54 @@ python3 shadow-viewer.py --labels security,performance python3 shadow-viewer.py --top src/auth.py --top-labels bug,security,performance --top-limit 5 ``` +### Citation Score and Retrieval Contract + +Replace `DISCOVERY_ID` and `CURSOR_TOKEN` with the exact values returned by the helper. + +Content views (`search`, `symbol`, `get`, `prefs`, `labels`, `recent`, `top`) +show an `id` and one `citation_score`. Every discovery starts at zero by +default, regardless of which workflow wrote it. The helper increments only +entries whose content it emits, once per retrieval event. Scanning/matching, +summary statistics, invariant audits, and raw file reads do not count. +Displayed scores are the values **before** the current read. A citation here +measures helper exposure, not proven usefulness, correctness, or LLM influence. +Agents must not manually edit counters or add them to discovery metadata. + +Exact search matches and source trust/status rank ahead of citation history; +recent views also prioritize mtime. Within a tied tier, higher scores rank +first, reserving room for a zero-score entry within tied tiers when the page +and character budget can fit multiple entries. +Popularity never overrides a refuted status or authorizes dropping a constraint. +Use targeted searches and additional pages for deduplication rather than +assuming the popular shortlist is exhaustive. + +The `d_...` ID is a content fingerprint of kind, file/symbol anchor, normalized +claim text, and related refs. Metadata-only status/source/label changes keep it; +rewording, renaming, or merging claims/refs can create a new zero-score identity. +Scores are not fuzzily transferred or summed during Meditate. Removed identities +can remain in the local ledger but cannot be expanded unless their claim exists. + +Scores live in SQLite under the repository's **common Git directory** at +`shadowfrog/citations.sqlite3`, shared by its local worktrees. Different shadow +roots in the same repo have separate scopes. Outside Git, the cache lives under +`$XDG_STATE_HOME/shadowfrog/citations` (Windows: `$LOCALAPPDATA`), falling back to +`~/.local/state/shadowfrog/citations`. It contains identities/counters/events, +not discovery bodies. It is local metadata, not a tracked or multi-machine DB; +Markdown and its format remain authoritative and unchanged. + +Transactions prevent lost increments from concurrent agents. Reuse `--event-id` +when retrying one retrieval; do not reuse it for unrelated visits. A failed +stdout emission is not recorded. Ledger failures warn on stderr, return the +knowledge, and show unknown scores as `?` when scores cannot be read. +Do not report those failed increments as successful. + +Pagination snapshots freeze ordering despite score changes and expire after +24 hours. Keep the original view/filters with `--cursor`; a changed knowledge +snapshot requires restarting the query. For a large claim, `--get` prints the +next `--text-offset`. Character limits bound helper output, not token counts; +the helper can still scan the underlying Markdown locally. If the ledger is +unavailable, narrow the query until pagination can be restored. + ## Dream Lineage Visualization The companion script `dream-lineage.py` generates an interactive HTML diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog-viewer/_citations.py new file mode 100644 index 0000000..f31132a --- /dev/null +++ b/skills/shadow-frog-viewer/_citations.py @@ -0,0 +1,163 @@ +"""Local citation scores; Markdown remains the authoritative knowledge store.""" + +from contextlib import contextmanager +from dataclasses import dataclass +import hashlib +import json +import os +from pathlib import Path +import re +import sqlite3 +import subprocess +import time +import uuid + + +PAGE_TTL = 24 * 60 * 60 + + +def discovery_id(kind, anchor, text, refs=()): + """Content identity excludes status, provenance, labels, and usage metadata.""" + value = [kind, anchor, " ".join(text.split()), sorted(set(refs))] + digest = hashlib.sha256( + json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + ).hexdigest() + return "d_" + digest[:32] + + +def validate_event_id(value): + if not isinstance(value, str) or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", value): + raise ValueError("event ID must be 1-128 letters, digits, or . _ : -") + return value + + +@dataclass(frozen=True) +class CitationStore: + """One short-lived connection per operation; SQLite coordinates processes.""" + + path: Path + scope: str + timeout: float = 0.1 + + def __post_init__(self): + object.__setattr__(self, "path", Path(self.path)) + if not isinstance(self.scope, str) or not self.scope: + raise ValueError("citation scope must be nonempty") + if self.timeout <= 0: + raise ValueError("citation timeout must be positive") + + @classmethod + def for_shadow(cls, shadow_dir): + shadow = Path(shadow_dir).resolve() + env = os.environ.copy() + for name in ("GIT_DIR", "GIT_WORK_TREE", "GIT_COMMON_DIR", "GIT_INDEX_FILE"): + env.pop(name, None) + env["LC_ALL"] = "C" + result = subprocess.run( + ["git", "-C", str(shadow), "rev-parse", "--show-toplevel", "--git-common-dir"], + capture_output=True, text=True, encoding="utf-8", timeout=0.2, env=env, + ) + if result.returncode == 0: + root_text, common_text = result.stdout.rstrip("\n").split("\n") + root = Path(root_text).resolve() + common = Path(common_text) + if not common.is_absolute(): + common = shadow / common + return cls( + common.resolve() / "shadowfrog/citations.sqlite3", + shadow.relative_to(root).as_posix(), + ) + if "not a git repository" not in result.stderr.lower(): + raise ValueError(f"Cannot resolve Git citation storage: {result.stderr.strip()}") + # Standalone shadows use untracked local state, not a sidecar in .shadow/. + base = os.environ.get("LOCALAPPDATA" if os.name == "nt" else "XDG_STATE_HOME") + state = Path(base) if base else Path.home() / ".local/state" + identity = hashlib.sha256(os.fsencode(shadow)).hexdigest() + return cls(state / "shadowfrog/citations" / f"{identity}.sqlite3", str(shadow)) + + @contextmanager + def _connection(self): + self.path.parent.mkdir(parents=True, exist_ok=True) + db = sqlite3.connect(self.path, timeout=self.timeout) + try: + version = db.execute("PRAGMA user_version").fetchone()[0] + if version not in (0, 1): + raise ValueError(f"Unsupported citation database version {version}; use a compatible helper") + if version == 0: + with db: + db.execute("BEGIN IMMEDIATE") + db.execute( + "CREATE TABLE IF NOT EXISTS scores (scope TEXT, id TEXT, " + "citation_score INTEGER NOT NULL CHECK(citation_score >= 0), PRIMARY KEY(scope, id))" + ) + db.execute( + "CREATE TABLE IF NOT EXISTS events (scope TEXT, event TEXT, id TEXT, " + "PRIMARY KEY(scope, event, id))" + ) + db.execute( + "CREATE TABLE IF NOT EXISTS pages (token TEXT PRIMARY KEY, scope TEXT, " + "request TEXT, catalog TEXT, ids TEXT, created REAL)" + ) + db.execute("PRAGMA user_version = 1") + yield db + finally: + db.close() + + def scores(self, ids): + if not self.path.exists() or not ids: + return {} + result = {} + ids = list(dict.fromkeys(ids)) + with self._connection() as db: + for offset in range(0, len(ids), 400): + chunk = ids[offset:offset + 400] + placeholders = ",".join("?" for _ in chunk) + result.update(db.execute( + f"SELECT id, citation_score FROM scores WHERE scope=? AND id IN ({placeholders})", + [self.scope, *chunk], + )) + return result + + def record(self, ids, event): + """Increment each emitted identity once per event, including across retries.""" + validate_event_id(event) + ids = list(dict.fromkeys(ids)) + if not ids: + return + with self._connection() as db, db: + db.execute("BEGIN IMMEDIATE") + for identity in ids: + inserted = db.execute( + "INSERT OR IGNORE INTO events(scope, event, id) VALUES (?, ?, ?)", + (self.scope, event, identity), + ) + if inserted.rowcount: + db.execute( + "INSERT INTO scores(scope, id, citation_score) VALUES (?, ?, 1) " + "ON CONFLICT(scope, id) DO UPDATE SET citation_score=citation_score+1", + (self.scope, identity), + ) + + def save_page(self, request, catalog, ids): + token = uuid.uuid4().hex + with self._connection() as db, db: + db.execute("DELETE FROM pages WHERE created < ?", (time.time() - PAGE_TTL,)) + db.execute( + "INSERT INTO pages VALUES (?, ?, ?, ?, ?, ?)", + (token, self.scope, request, catalog, json.dumps(ids), time.time()), + ) + return token + + def load_page(self, token, request, catalog): + with self._connection() as db: + row = db.execute( + "SELECT request, catalog, ids, created FROM pages WHERE token=? AND scope=?", + (token, self.scope), + ).fetchone() + if not row or row[3] < time.time() - PAGE_TTL: + raise ValueError("Retrieval cursor expired or unavailable; rerun the query without --cursor") + if row[0] != request: + raise ValueError("Cursor belongs to a different query; reuse its original view and filters") + if row[1] != catalog: + raise ValueError("Shadow knowledge changed; rerun the query without --cursor") + return json.loads(row[2]) diff --git a/skills/shadow-frog-viewer/shadow-viewer.py b/skills/shadow-frog-viewer/shadow-viewer.py index 3516b36..f4508d1 100755 --- a/skills/shadow-frog-viewer/shadow-viewer.py +++ b/skills/shadow-frog-viewer/shadow-viewer.py @@ -11,10 +11,17 @@ --recent [N] N most recent discoveries with content (default: 10) --labels LABEL Show discoveries by label (bug, security, etc.) --top FILE Top actionable discoveries for FILE (hook-sized) + --symbol FILE::SYMBOL Bounded knowledge for a specific symbol + --get ID Expand one discovery by its retrieval identity --check-invariants Report structural violations (exit 1 if any) Options: --shadow-dir DIR Path to .shadow/ directory (default: auto-detect) + --limit N Maximum results (default: 10) + --max-chars N Output budget including metadata (default: 4000) + --cursor TOKEN Continue a previous result snapshot + --no-record Do not increment local citation scores + --event-id ID Reuse an ID for retry-idempotent bookkeeping Exit codes: 0 Success (possibly with warnings on stderr) @@ -22,16 +29,30 @@ """ import argparse +from dataclasses import dataclass +import hashlib import json import os import re import sys +import sqlite3 +import subprocess import traceback +import uuid from collections import defaultdict from datetime import datetime from pathlib import Path +sys.path.insert(0, str(Path(__file__).resolve().parent)) +try: + from _citations import CitationStore, discovery_id, validate_event_id +except ImportError as exc: + raise SystemExit("[shadow-viewer error] Missing citation helper/dependency; reinstall the complete Viewer skill") from exc +finally: + sys.path.pop(0) + + _DISCOVERY_META_RE = re.compile( r"_\((\w+),\s*source:\s*(\w+)" r"(?:,\s*labels:\s*\[([^\]]*)\])?" @@ -425,13 +446,11 @@ def collect_all_discoveries(shadow_dir): failed_files.append( (str(sf), parsed["parse_errors"]) ) + modified = _mtime(sf) for d in parsed["discoveries"]: d.setdefault("file", parsed["source_file"]) d["shadow_path"] = str(sf.relative_to(shadow_dir)) - try: - d["shadow_mtime"] = os.path.getmtime(sf) - except OSError: - d["shadow_mtime"] = 0.0 + d["shadow_mtime"] = modified all_disc.append(d) except Exception as e: msg = f"Failed to parse {sf}: {type(e).__name__}: {e}" @@ -447,6 +466,327 @@ def collect_all_discoveries(shadow_dir): # --- View Functions --- +@dataclass(frozen=True) +class RetrievalOptions: + limit: int = 10 + max_chars: int = 4000 + cursor: str | None = None + event_id: str | None = None + record: bool = True + text_offset: int = 0 + + def __post_init__(self): + if type(self.limit) is not int or self.limit < 1: + raise ValueError("Result limit must be positive") + if type(self.max_chars) is not int or self.max_chars < 0 or (self.max_chars and self.max_chars < 256): + raise ValueError("Output budget must be at least 256 characters, or 0 for no cap") + if type(self.text_offset) is not int or self.text_offset < 0: + raise ValueError("Text offset must be nonnegative") + if self.event_id is not None: + validate_event_id(self.event_id) + + +def _knowledge_entry(kind, file, symbol, text, data, refs=(), mtime=0): + anchor = f"{file}::{symbol}" if symbol else file + return { + "id": discovery_id(kind, anchor, text, refs), + "kind": kind, "file": file, "symbol": symbol, "anchor": anchor, + "text": text, "refs": list(refs), "mtime": mtime, + "status": data.get("status", "verified" if kind == "preference" else "?"), + "source": data.get("source", "?"), "labels": data.get("labels", []), + "title": data.get("title", ""), "category": data.get("category", "?"), + "dream_report": data.get("dream_report", ""), + } + + +def _mtime(path): + try: + return path.stat().st_mtime + except OSError as exc: + warn(f"Modification time unavailable for {path}: {exc}; treating it as undated.") + return 0 + + +def _preference_entries(shadow_dir): + prefs = parse_prefs(shadow_dir) + modified = _mtime(shadow_dir / "_prefs.md") if prefs else 0 + return _unique_entries([ + _knowledge_entry( + "preference", "_prefs.md", "", pref.get("text", ""), pref, + mtime=modified, + ) + for pref in prefs + ]) + + +def _unique_entries(entries): + # Duplicate claims at the same location have one identity and one score. + unique = {} + for entry in entries: + if not entry["text"].strip(): + warn(f"Empty discovery at {entry['anchor']}; repair its text before retrieval.") + continue + prior = unique.get(entry["id"]) + if prior is None or _trust(entry) < _trust(prior): + unique[entry["id"]] = entry + return list(unique.values()) + + +def _knowledge_entries(shadow_dir, source_file=None): + """Collect identities without recording a citation for parsing or matching.""" + entries = [] + if source_file is None: + discoveries = collect_all_discoveries(shadow_dir) + else: + if ( + not source_file or ":" in source_file or "\\" in source_file + or any(part in ("", ".", "..") for part in source_file.split("/")) + ): + raise ValueError("Use a repository-relative source path with forward slashes") + path = shadow_dir / (source_file + ".md") + if not path.resolve().is_relative_to(shadow_dir.resolve()): + raise ValueError("Requested shadow resolves outside --shadow-dir") + parsed = parse_shadow_file(path) if path.is_file() else {"discoveries": []} + modified = _mtime(path) if parsed["discoveries"] else 0 + discoveries = [ + {**disc, "shadow_path": source_file + ".md", "shadow_mtime": modified} + for disc in parsed["discoveries"] + ] + for disc in discoveries: + file = Path(disc["shadow_path"]).as_posix()[:-3] + entries.append(_knowledge_entry( + "discovery", file, disc.get("symbol", "file-level"), disc.get("text", ""), + disc, disc.get("also_involves", []), disc.get("shadow_mtime", 0), + )) + for cross in parse_cross_cutting(shadow_dir): + refs = cross.get("refs", []) + if source_file is not None and not any(ref.split("::", 1)[0] == source_file for ref in refs): + continue + relative = "_cross/" + cross["file"] + entries.append(_knowledge_entry( + "cross-cutting", relative, "", cross.get("discovery", cross.get("title", "")), + cross, refs, _mtime(shadow_dir / relative), + )) + if source_file is None: + entries.extend(_preference_entries(shadow_dir)) + return _unique_entries(entries) + + +def _trust(entry): + if entry["status"] == "refuted": + return 5 + if entry["source"] == "user": + return 0 + if entry["source"] == "interaction": + return 1 + return {"verified": 2, "uncertain": 3}.get(entry["status"], 4) + + +def _rank_entries(entries, scores, limit, recent=False): + groups = defaultdict(list) + for entry in entries: + priority = ((-entry["mtime"],) if recent else ()) + ( + entry.get("relevance", 0), _trust(entry), + ) + groups[priority].append(entry) + result = [] + for priority in sorted(groups): + group = groups[priority] + seen = sorted( + (entry for entry in group if scores.get(entry["id"], 0)), + key=lambda entry: -scores[entry["id"]], + ) + unseen = [entry for entry in group if not scores.get(entry["id"], 0)] + # Reserve one slot per group/page-sized block for a zero-score entry. + width = max(2, limit) + while seen and unseen: + result.extend(seen[:width - 1]) + del seen[:width - 1] + result.append(unseen.pop(0)) + result.extend(seen) + result.extend(unseen) + return result + + +def _citation_store(shadow_dir): + try: + return CitationStore.for_shadow(shadow_dir) + except (OSError, ValueError, subprocess.SubprocessError) as exc: + warn(f"Citation tracking unavailable: {exc}. Knowledge is still returned; repair local Git/state access.") + return None + + +def _clip(text, limit): + return text if len(text) <= limit else text[:max(0, limit - 3)] + "..." + + +def _entry_preview(entry, score, style, group_count): + text = _clip(entry["text"].replace("\n", " "), 180) + anchor = _clip(entry["anchor"], 160) + identity = f"id={entry['id']} citation_score={score}" + metadata = f"({entry['status']}, source: {entry['source']})" + labels = ",".join(entry["labels"]) or "-" + if style == "top": + return f"- [{labels}] `{anchor}` ({entry['status']}) {identity}: {text}" + if style == "recent": + stamp = datetime.fromtimestamp(entry["mtime"]).strftime("%Y-%m-%d %H:%M") + heading = f" [{stamp}] ({entry['kind']})" + elif entry["kind"] == "cross-cutting": + heading = f"Cross-cutting: {_clip(entry['title'], 120)}\n Category: {entry['category']}" + elif entry["kind"] == "preference": + heading = f"Preferences [{entry['source']}]" + else: + heading = f"{_clip(entry['file'], 160)} ({group_count} matches)" + body = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n {text}" + if entry["refs"]: + label = "Refs" if entry["kind"] == "cross-cutting" else "Also involves" + body += f"\n {label}: {_clip(', '.join(entry['refs']), 160)}" + if style == "labels" and len(entry["labels"]) > 1: + body += f"\n Also labeled: {labels}" + return body + + +def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): + store = _citation_store(shadow_dir) + scores = {} + if store: + try: + scores = store.scores([entry["id"] for entry in entries]) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Cannot read citation scores: {exc}. Scores are unknown; repair the local ledger and retry.") + store = None + ranked = _rank_entries(entries, scores, options.limit, recent=style == "recent") + catalog = "" if style == "top" else hashlib.sha256(json.dumps( + sorted(entries, key=lambda entry: entry["id"]), sort_keys=True, ensure_ascii=False, + ).encode("utf-8")).hexdigest() + token = None + offset = 0 + if options.cursor: + match = re.fullmatch(r"([0-9a-f]{32}):(\d+)", options.cursor) + if not match: + raise ValueError("Invalid cursor; copy the --cursor value from the previous response") + if store is None: + raise ValueError("Cannot resume cursor without the local ledger; repair it or restart the query") + token, offset = match.group(1), int(match.group(2)) + ids = store.load_page(token, request, catalog) + by_id = {entry["id"]: entry for entry in entries} + ranked = [by_id[identity] for identity in ids] + if offset >= len(ranked): + raise ValueError("Cursor is past the available results; restart the query") + + counts = defaultdict(int) + for entry in entries: + counts[entry["file"]] += 1 + # Include the terminal newline and continuation instructions in the budget. + cap = options.max_chars + prefix = _clip(header, min(300, cap // 5)) if cap else header + reserve = 90 if style != "top" else 6 + if cap and not options.cursor: + width = options.limit + while width > 1: + used = len(prefix) + reserve + 3 + fits = 0 + for entry in ranked[:width]: + score = scores.get(entry["id"], 0) if store else "?" + used += len(_entry_preview(entry, score, style, counts[entry["file"]])) + 1 + if used > cap: + break + fits += 1 + if fits >= width: + break + width = max(1, fits) + ranked = _rank_entries(entries, scores, width, recent=style == "recent") + output = prefix + shown = [] + for entry in ranked[offset:offset + options.limit]: + score = scores.get(entry["id"], 0) if store else "?" + block = _entry_preview(entry, score, style, counts[entry["file"]]) + available = cap - len(output) - reserve - 3 if cap else len(block) + if cap and len(block) > available: + if shown: + break + identity = ( + f"({entry['status']}, source: {entry['source']}) " + f"id={entry['id']} citation_score={score}: " + ) + if available <= len(identity) + 4: + raise ValueError("Output budget cannot fit an entry; increase --max-chars") + block = identity + _clip(entry["text"], available - len(identity)) + output += "\n" + block + shown.append(entry["id"]) + next_offset = offset + len(shown) + if next_offset < len(ranked): + if style == "top": + output += "\n(...)" + else: + if token is None and store is not None: + try: + token = store.save_page(request, catalog, [entry["id"] for entry in ranked]) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Cannot save pagination: {exc}. Repair the local ledger or narrow the query.") + if token: + output += f"\nMore: --cursor {token}:{next_offset} (same view)" + else: + output += "\nMore omitted: narrow the query (local ledger unavailable)." + if style != "top": + output += "\nExpand a claim: --get ID" + output = prefix.replace("{shown}", str(len(shown))) + output[len(prefix):] + if cap and len(output) + 1 > cap: + raise ValueError("Output budget cannot fit retrieval metadata; increase --max-chars") + print(output, flush=True) + if store and shown and options.record: + try: + store.record(shown, options.event_id or uuid.uuid4().hex) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Citation scores were not updated: {exc}. Reuse --event-id when retrying.") + + +def view_get(shadow_dir, identity, *, options=None): + """Expand a current discovery, chunking long text without changing its ID.""" + options = options or RetrievalOptions() + if not re.fullmatch(r"d_[0-9a-f]{32}", identity): + raise ValueError("--get requires the complete id=d_... value from a retrieval result") + entry = next((entry for entry in _knowledge_entries(shadow_dir) if entry["id"] == identity), None) + if entry is None: + raise ValueError("Discovery ID is absent or its claim changed; search again for its current ID") + store = _citation_store(shadow_dir) + score = "?" + if store: + try: + score = store.scores([identity]).get(identity, 0) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Cannot read citation score: {exc}. Repair the local ledger and retry.") + store = None + body = entry["anchor"] + "\n\n" + entry["text"] + if entry["labels"]: + body += "\nLabels: " + ", ".join(entry["labels"]) + if entry["kind"] == "cross-cutting": + body += "\nCategory: " + entry["category"] + "\nTitle: " + entry["title"] + if entry["refs"]: + body += "\nRefs: " + ", ".join(entry["refs"]) + if entry["dream_report"]: + body += "\nDream report: " + entry["dream_report"] + if options.text_offset >= len(body) and options.text_offset: + raise ValueError("--text-offset is past the end of this discovery") + header = ( + f"id={identity} citation_score={score}\n" + f"({entry['status']}, source: {entry['source']})\n" + ) + start = options.text_offset + available = options.max_chars - len(header) - 100 if options.max_chars else len(body) + if available < 1: + raise ValueError("Output budget cannot fit discovery content; increase --max-chars") + content = body[start:start + available] + output = header + content + if start + len(content) < len(body): + output += f"\nContinue: --get {identity} --text-offset {start + len(content)}" + print(output, flush=True) + if store and options.record: + try: + store.record([identity], options.event_id or uuid.uuid4().hex) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Citation score was not updated: {exc}. Reuse --event-id when retrying.") + def view_summary(shadow_dir): """Overview + detailed statistics. @@ -579,250 +919,87 @@ def view_summary(shadow_dir): warn(f"Failed to render state info: {e}") -def view_search(shadow_dir, query): - """Universal search: matches file names, symbol names, and discovery text. - - Also searches cross-cutting discoveries and preferences. - Results are grouped by match location for readability. - Each search domain (per-file, cross-cutting, prefs) is independent — - if one fails, the others still return results. - """ +def view_search(shadow_dir, query, *, options=None): + """Bounded search with stable continuation over matching knowledge identities.""" + options = options or RetrievalOptions() query_lower = query.lower() - disc_matches = [] - cross_matches = [] - pref_matches = [] - section_errors = [] - - # Search per-file shadows - try: - for sf in get_all_shadow_files(shadow_dir): - try: - parsed = parse_shadow_file(sf) - source_file = parsed["source_file"] or str( - sf.relative_to(shadow_dir) - ) - file_name_hit = query_lower in source_file.lower() - - for d in parsed["discoveries"]: - sym = d.get("symbol", "") - text = d.get("text", "") - sym_hit = query_lower in sym.lower() - text_hit = query_lower in text.lower() - also_hit = any( - query_lower in ref.lower() - for ref in d.get("also_involves", []) - ) - - if file_name_hit or sym_hit or text_hit or also_hit: - disc_matches.append({ - "file": source_file, - "symbol": sym, - "text": text, - "status": d.get("status", "?"), - "source": d.get("source", "?"), - "also_involves": d.get("also_involves", []), - "match": ( - "file" if file_name_hit else - "symbol" if sym_hit else - "also_involves" if also_hit else "text" - ), - }) - except Exception as e: - warn(f"Search: error processing {sf}: {e}") - except Exception as e: - msg = f"Search: failed to search per-file shadows: {e}" - warn(msg) - section_errors.append(msg) - - # Search cross-cutting discoveries - try: - for e in parse_cross_cutting(shadow_dir): - title = e.get("title", "") - disc_text = e.get("discovery", "") - refs = e.get("refs", []) - if (query_lower in title.lower() - or query_lower in disc_text.lower() - or any(query_lower in r.lower() for r in refs)): - cross_matches.append(e) - except Exception as e: - msg = f"Search: failed to search cross-cutting: {e}" - warn(msg) - section_errors.append(msg) - - # Search preferences - try: - for p in parse_prefs(shadow_dir): - if query_lower in p.get("text", "").lower(): - pref_matches.append(p) - except Exception as e: - msg = f"Search: failed to search preferences: {e}" - warn(msg) - section_errors.append(msg) - - total = len(disc_matches) + len(cross_matches) + len(pref_matches) - if total == 0: - print(f"No results for '{query}'.") - if section_errors: - print(f"Note: {len(section_errors)} search section(s) had errors " - f"— results may be incomplete. Check stderr for details.") + if not query.strip(): + raise ValueError("Search query must be nonempty") + matches = [] + for entry in _knowledge_entries(shadow_dir): + fields = [entry["anchor"], entry["file"], entry["symbol"], entry["text"], + entry["title"], *entry["refs"]] + if any(query_lower in field.lower() for field in fields): + entry["relevance"] = 0 if query_lower in [field.lower() for field in fields[:3]] else 1 + matches.append(entry) + if not matches and not options.cursor: + print(_clip(f"No results for '{query}'.", options.max_chars - 1) if options.max_chars + else f"No results for '{query}'.") return - - print(f"Search: '{query}' ({total} results)") - print("=" * 50) - - # Per-file discoveries, grouped by file - if disc_matches: - try: - by_file = defaultdict(list) - for d in disc_matches: - by_file[d["file"]].append(d) - - for file, discs in sorted(by_file.items()): - print(f"\n{file} ({len(discs)} matches)") - print("-" * (len(file) + 15)) - for d in discs: - sym = d["symbol"] - print(f" {file}::{sym}") - print(f" {d['text'][:120]}") - print(f" ({d['status']}, source: {d['source']})") - if d.get("also_involves"): - print( - f" Also involves: " - f"{', '.join(d['also_involves'])}" - ) - except Exception as e: - warn(f"Search: failed to render per-file results: {e}") - - # Cross-cutting - if cross_matches: - try: - print(f"\nCross-cutting ({len(cross_matches)} matches)") - print("-" * 30) - for e in cross_matches: - title = e.get("title", e.get("slug", "?")) - cat = e.get("category", "?") - status = e.get("status", "?") - source = e.get("source", "?") - print(f"\n {title}") - print(f" Category: {cat} | {status}, source: {source}") - print(f" Refs: {', '.join(e.get('refs', [])[:5])}") - if e.get("discovery"): - print(f" {e['discovery'][:120]}") - except Exception as e: - warn(f"Search: failed to render cross-cutting results: {e}") - - # Preferences - if pref_matches: - try: - print(f"\nPreferences ({len(pref_matches)} matches)") - print("-" * 30) - for p in pref_matches: - print(f" [{p.get('source', '?')}] {p['text'][:120]}") - except Exception as e: - warn(f"Search: failed to render preference results: {e}") + _emit_knowledge( + shadow_dir, matches, f"Search: '{query}' ({len(matches)} results)", options, + request=json.dumps(["search", query_lower]), + ) -def view_prefs(shadow_dir): - """Show all preferences.""" - try: - prefs = parse_prefs(shadow_dir) - except Exception as e: - error(f"Failed to parse preferences: {e}") +def view_symbol(shadow_dir, anchor, *, options=None): + options = options or RetrievalOptions() + file, separator, symbol = anchor.partition("::") + if not separator or not symbol: + raise ValueError("--symbol requires file::symbol (use File-Level for a file-level section)") + symbol = "file-level" if symbol == "File-Level" else symbol + canonical = f"{file}::{symbol}" + matches = [ + entry for entry in _knowledge_entries(shadow_dir, file) + if entry["anchor"] == canonical or canonical in entry["refs"] + ] + if not matches and not options.cursor: + print(_clip(f"No knowledge for '{anchor}'.", options.max_chars - 1) + if options.max_chars else f"No knowledge for '{anchor}'.") return + _emit_knowledge( + shadow_dir, matches, f"Knowledge for {anchor} ({len(matches)} results)", options, + request=json.dumps(["symbol", canonical]), + ) + - if not prefs: +def view_prefs(shadow_dir, *, options=None): + """Page preferences without letting popular code discoveries hide directives.""" + options = options or RetrievalOptions() + prefs = _preference_entries(shadow_dir) + if not prefs and not options.cursor: print("No preferences recorded yet.") return + _emit_knowledge(shadow_dir, prefs, f"Project Preferences ({len(prefs)} total)", options, request="prefs") - print(f"Project Preferences ({len(prefs)} total)") - print("=" * 40) - for p in prefs: - try: - source = p.get("source", "?") - print(f"\n [{source}] {p['text']}") - except Exception as e: - warn(f"Failed to render preference: {e}") - -def view_labels(shadow_dir, label_filter): +def view_labels(shadow_dir, label_filter, *, options=None): """Show discoveries filtered by label(s). label_filter can be a single label or comma-separated list. """ - try: - filters = [l.strip().lower() for l in label_filter.split(",")] - except Exception as e: - error(f"Invalid label filter '{label_filter}': {e}") - return - - try: - all_disc = collect_all_discoveries(shadow_dir) - except Exception as e: - error(f"Failed to collect discoveries for label filtering: {e}") + options = options or RetrievalOptions() + filters = [label.strip().lower() for label in label_filter.split(",") if label.strip()] + if not filters: + raise ValueError("Supply at least one label with --labels") + matching = [ + entry for entry in _knowledge_entries(shadow_dir) + if set(filters) & {label.lower() for label in entry["labels"]} + ] + + if not matching and not options.cursor: + message = f"No discoveries with label(s): {', '.join(filters)}" + print(_clip(message, options.max_chars - 1) if options.max_chars else message) return - # Also include cross-cutting discoveries with labels - try: - for entry in parse_cross_cutting(shadow_dir): - if entry.get("labels"): - all_disc.append({ - "file": f"_cross/{entry.get('file', '?')}", - "symbol": entry.get("title", entry.get("slug", "?")), - "text": entry.get("discovery", entry.get("title", "")), - "status": entry.get("status", "?"), - "source": entry.get("source", "?"), - "labels": entry["labels"], - }) - except Exception as e: - warn(f"Failed to include cross-cutting in label search: {e}") - - matching = [] - for d in all_disc: - try: - disc_labels = [l.lower() for l in d.get("labels", [])] - if any(f in disc_labels for f in filters): - matching.append(d) - except Exception as e: - warn(f"Failed to check labels on discovery in " - f"{d.get('file', '?')}::{d.get('symbol', '?')}: {e}") - - if not matching: - print(f"No discoveries with label(s): {', '.join(filters)}") - return - - print(f"Discoveries with label(s): {', '.join(filters)} " - f"({len(matching)} results)") - print("=" * 50) - - by_label = defaultdict(list) - for d in matching: - for lbl in d.get("labels", []): - if lbl.lower() in filters: - by_label[lbl.lower()].append(d) - - for lbl in filters: - discs = by_label.get(lbl, []) - if not discs: - continue - print(f"\n[{lbl}] ({len(discs)} discoveries)") - print("-" * 30) - for d in discs: - try: - sym = d.get("symbol", "?") - src_file = d.get("file", "?") - print(f" {src_file}::{sym}") - print(f" {d['text'][:120]}") - print(f" ({d.get('status', '?')}, " - f"source: {d.get('source', '?')})") - all_labels = d.get("labels", []) - other = [l for l in all_labels if l.lower() != lbl] - if other: - print(f" Also labeled: {', '.join(other)}") - except Exception as e: - warn(f"Failed to render labeled discovery: {e}") + _emit_knowledge( + shadow_dir, matching, + f"Discoveries with label(s): {', '.join(filters)} ({len(matching)} results)", + options, style="labels", request=json.dumps(["labels", sorted(set(filters))]), + ) -def view_recent(shadow_dir, count=10): +def view_recent(shadow_dir, count=10, *, options=None): """Show the N most recent discoveries (by shadow file mtime). Collects all discoveries across all shadow files, cross-cutting entries, @@ -831,108 +1008,19 @@ def view_recent(shadow_dir, count=10): Each data source is independent — if cross-cutting fails, per-file discoveries still appear. """ - all_items = [] - - # Per-file discoveries - try: - for sf in get_all_shadow_files(shadow_dir): - try: - mtime = os.path.getmtime(sf) - parsed = parse_shadow_file(sf) - source_file = parsed["source_file"] or str( - sf.relative_to(shadow_dir) - ) - for d in parsed["discoveries"]: - all_items.append({ - "type": "discovery", - "file": source_file, - "symbol": d.get("symbol", "?"), - "text": d.get("text", ""), - "status": d.get("status", "?"), - "source": d.get("source", "?"), - "mtime": mtime, - }) - except Exception as e: - warn(f"Recent: failed to process {sf}: {e}") - except Exception as e: - warn(f"Recent: failed to list shadow files: {e}") - - # Cross-cutting discoveries - try: - cross_entries = parse_cross_cutting(shadow_dir) - cross_by_file = defaultdict(list) - for e in cross_entries: - cross_by_file[e["file"]].append(e) - - cross_dir = shadow_dir / "_cross" - if cross_dir.exists(): - for cf in cross_dir.glob("*.md"): - try: - mtime = os.path.getmtime(cf) - for e in cross_by_file.get(cf.name, []): - all_items.append({ - "type": "cross-cutting", - "file": f"_cross/{cf.name}", - "symbol": e.get("title", e.get("slug", "?")), - "text": e.get("discovery", e.get("title", "")), - "status": e.get("status", "?"), - "source": e.get("source", "?"), - "mtime": mtime, - }) - except Exception as e: - warn(f"Recent: failed to process cross-cutting {cf}: {e}") - except Exception as e: - warn(f"Recent: failed to process cross-cutting discoveries: {e}") - - # Preferences - try: - prefs_path = shadow_dir / "_prefs.md" - if prefs_path.exists(): - mtime = os.path.getmtime(prefs_path) - for p in parse_prefs(shadow_dir): - all_items.append({ - "type": "preference", - "file": "_prefs.md", - "symbol": "-", - "text": p.get("text", ""), - "status": "-", - "source": p.get("source", "?"), - "mtime": mtime, - }) - except Exception as e: - warn(f"Recent: failed to process preferences: {e}") - - all_items.sort(key=lambda x: x.get("mtime", 0), reverse=True) - - if not all_items: + options = options or RetrievalOptions(limit=count) + all_items = _knowledge_entries(shadow_dir) + if not all_items and not options.cursor: print("No discoveries found.") return - shown = all_items[:count] - print(f"Most Recent Discoveries (top {count})") - print("=" * 50) - for item in shown: - try: - ts = datetime.fromtimestamp( - item.get("mtime", 0) - ).strftime("%Y-%m-%d %H:%M") - kind = item.get("type", "?") - sym = item.get("symbol", "?") - - print(f"\n [{ts}] ({kind})") - if kind == "preference": - print(f" {item.get('text', '')[:120]}") - print(f" source: {item.get('source', '?')}") - else: - print(f" {item.get('file', '?')}::{sym}") - print(f" {item.get('text', '')[:120]}") - print(f" ({item.get('status', '?')}, " - f"source: {item.get('source', '?')})") - except Exception as e: - warn(f"Recent: failed to render item: {e}") + _emit_knowledge( + shadow_dir, all_items, f"Most Recent Discoveries (top {count})", + options, style="recent", request="recent", + ) -def view_top(shadow_dir, file_path, labels_filter, limit, max_chars): +def view_top(shadow_dir, file_path, labels_filter, limit, max_chars, *, options=None): """Show the top N actionable discoveries for a single source file. Designed for the preToolUse hook: concise output suitable for @@ -940,82 +1028,29 @@ def view_top(shadow_dir, file_path, labels_filter, limit, max_chars): file. Pulls from both the per-file shadow and any _cross/ entries whose refs touch this file. - Ranking: verified > uncertain > refuted; within a tier, source - order is preserved. Output is hard-capped at max_chars (the trailing - "(...)" marker still fits). + Trust/status precedes citation score. Output, including the final newline, + is hard-capped; only entries actually emitted are counted. """ norm = file_path.strip() if norm.startswith("./"): norm = norm[2:] - shadow_path = shadow_dir / f"{norm}.md" label_set = {l.strip().lower() for l in labels_filter.split(",") if l.strip()} - - candidates = [] - - if shadow_path.is_file(): - try: - parsed = parse_shadow_file(shadow_path) - for d in parsed.get("discoveries", []): - disc_labels = {l.lower() for l in d.get("labels", [])} - if label_set and not (disc_labels & label_set): - continue - candidates.append({ - "kind": "file", - "anchor": d.get("symbol") or "file-level", - "text": d.get("text", "").strip(), - "status": d.get("status", "?"), - "labels": sorted(disc_labels), - }) - except Exception as e: - warn(f"--top: failed parsing {shadow_path}: {e}") - - try: - for entry in parse_cross_cutting(shadow_dir): - refs = entry.get("refs", []) or [] - if not any(r.split("::", 1)[0].strip() == norm for r in refs): - continue - cross_labels = {l.lower() for l in entry.get("labels", [])} - if label_set and not (cross_labels & label_set): - continue - candidates.append({ - "kind": "cross", - "anchor": f"_cross/{entry.get('file', entry.get('slug', '?'))}", - "text": (entry.get("discovery") or entry.get("title") or "").strip(), - "status": entry.get("status", "?"), - "labels": sorted(cross_labels), - }) - except Exception as e: - warn(f"--top: failed scanning _cross/: {e}") + options = options or RetrievalOptions(limit=limit, max_chars=max_chars) + candidates = [ + entry for entry in _knowledge_entries(shadow_dir, norm) + if not label_set or label_set & {label.lower() for label in entry["labels"]} + ] if not candidates: labels_disp = ",".join(sorted(label_set)) if label_set else "any" - print( - f"No actionable discoveries ({labels_disp}) for {norm}." - ) + message = f"No actionable discoveries ({labels_disp}) for {norm}." + print(_clip(message, max_chars - 1) if max_chars else message) return - - tier = {"verified": 0, "uncertain": 1, "refuted": 2} - candidates.sort(key=lambda d: tier.get(d.get("status", "?"), 3)) - - shown = candidates[:limit] - header = ( - f"Top {len(shown)} of {len(candidates)} actionable discoveries " - f"for {norm}:" + _emit_knowledge( + shadow_dir, candidates, + f"Top {{shown}} of {len(candidates)} actionable discoveries for {norm}:", + options, style="top", request=json.dumps(["top", norm, sorted(label_set)]), ) - lines = [header] - for d in shown: - labels = ",".join(d["labels"]) if d["labels"] else "—" - anchor = d["anchor"] - text = d["text"].replace("\n", " ").strip() - lines.append( - f"- [{labels}] `{anchor}` ({d['status']}): {text}" - ) - - out = "\n".join(lines) - if max_chars and len(out) > max_chars: - truncated = out[: max_chars - 6].rstrip() - out = truncated + "\n(...)" - print(out) def view_check_invariants(shadow_dir): @@ -1269,6 +1304,8 @@ def main(): "--search", metavar="QUERY", help="Universal search: files, symbols, and discovery text", ) + views.add_argument("--symbol", metavar="FILE::SYMBOL", help="Knowledge for an exact symbol and its cross-cutting refs") + views.add_argument("--get", metavar="ID", help="Expand a discovery returned by the viewer") views.add_argument( "--prefs", action="store_true", help="Show project-wide preferences", @@ -1327,8 +1364,44 @@ def main(): "(default: 600). Use 0 for no cap." ), ) + parser.add_argument("--limit", type=int, help="Results per page for search/symbol/labels/prefs (default: 10)") + parser.add_argument("--max-chars", type=int, help="Retrieval output budget (default: 4000; 0 disables cap)") + parser.add_argument("--cursor", help="Continue the same view/filters with a frozen result ordering") + parser.add_argument("--event-id", help="Retry token; an entry counts once per token (default: fresh event)") + parser.add_argument("--no-record", action="store_true", help="Do not increment citation scores") + parser.add_argument("--text-offset", type=int, help="Continue long --get output at this character offset") args = parser.parse_args() + retrieval = ( + args.top is not None or args.search is not None or args.symbol is not None + or args.get is not None or args.prefs or args.labels is not None or args.recent is not None + ) + if not retrieval and ( + args.limit is not None or args.max_chars is not None or args.cursor + or args.event_id is not None or args.no_record or args.text_offset is not None + ): + parser.error("Retrieval options require --search, --symbol, --get, --top, --prefs, --labels, or --recent") + if args.text_offset is not None and args.get is None: + parser.error("--text-offset requires --get") + if args.cursor and (args.get is not None or args.top is not None): + parser.error("--cursor is for paged search/symbol/labels/prefs/recent; use --text-offset with --get") + if args.limit is not None and (args.top is not None or args.recent is not None or args.get is not None): + parser.error("Use --top-limit or --recent N instead of --limit; --get returns one discovery") + if args.max_chars is not None and args.top is not None: + parser.error("Use --top-max-chars with --top") + try: + options = RetrievalOptions( + limit=args.top_limit if args.top is not None else ( + args.recent if args.recent is not None else (args.limit if args.limit is not None else 10) + ), + max_chars=args.top_max_chars if args.top is not None else ( + args.max_chars if args.max_chars is not None else 4000 + ), + cursor=args.cursor, event_id=args.event_id, record=not args.no_record, + text_offset=args.text_offset or 0, + ) + except ValueError as exc: + parser.error(str(exc)) # Find shadow dir if args.shadow_dir: @@ -1355,22 +1428,27 @@ def main(): # Dispatch if args.check_invariants: sys.exit(view_check_invariants(shadow_dir)) - if args.top: + if args.top is not None: view_top( shadow_dir, args.top, args.top_labels, args.top_limit, args.top_max_chars, + options=options, ) - elif args.search: - view_search(shadow_dir, args.search) + elif args.search is not None: + view_search(shadow_dir, args.search, options=options) + elif args.symbol is not None: + view_symbol(shadow_dir, args.symbol, options=options) + elif args.get is not None: + view_get(shadow_dir, args.get, options=options) elif args.prefs: - view_prefs(shadow_dir) - elif args.labels: - view_labels(shadow_dir, args.labels) + view_prefs(shadow_dir, options=options) + elif args.labels is not None: + view_labels(shadow_dir, args.labels, options=options) elif args.recent is not None: - view_recent(shadow_dir, args.recent) + view_recent(shadow_dir, args.recent, options=options) else: view_summary(shadow_dir) @@ -1379,6 +1457,9 @@ def main(): except KeyboardInterrupt: error("Interrupted by user.") sys.exit(130) + except (ValueError, sqlite3.Error) as exc: + error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") + sys.exit(1) except Exception as e: error( f"Unexpected error: {type(e).__name__}: {e}\n" diff --git a/skills/shadow-frog/SKILL.md b/skills/shadow-frog/SKILL.md index 9074571..c531d64 100644 --- a/skills/shadow-frog/SKILL.md +++ b/skills/shadow-frog/SKILL.md @@ -23,16 +23,17 @@ to that code location. **Every time you work on code in a repo with `.shadow/`:** -1. **Read `_prefs.md` first** — it contains project-wide conventions, - user preferences, and things to avoid. +1. **Read preferences first** with the viewer's `--prefs`, following all pages. + `_prefs.md` contains project-wide conventions, user preferences, and things to avoid. 2. **Read relevant `_cross/` discoveries** — list `_cross/` and read entries whose titles relate to the current area, including cross-file contracts and interactions. 3. **Check `_dreams/_index.md`** and read relevant experiment reports, especially when investigating bugs or unfamiliar code. They may contain findings not yet distilled into per-file shadows. -4. **Before editing a file**, read its shadow (`.shadow/.md`) and - relevant `_cross/` entries, then apply the discoveries. +4. **Before editing a file**, query its shadow (`.shadow/.md`) with + `--symbol file::symbol` or `--top FILE`, and search relevant `_cross/` + knowledge. Expand matching entries and follow pages as needed, then apply them. `_index.md` counts may be stale; inspect the actual shadows and `_cross/`. 5. **When the user explains something about code** (gotcha, design intent, warning, history): write a `source: user` discovery to the shadow @@ -42,6 +43,25 @@ to that code location. specific file): write it to `_prefs.md` immediately. 7. **After code changes**: run `/shadow-frog-update` +## Bounded Knowledge Retrieval + +Prefer the `/shadow-frog-viewer` helper over loading an entire large shadow. +It returns short previews, discovery IDs, and one `citation_score`; `--get ID` +expands an entry, and returned cursors continue a stable result ordering. +Citation scores count content emitted by the helper, not proven use in reasoning. +Do not manually increment scores or put them into Markdown: the helper records +visits atomically in local state shared across worktrees. New discoveries from +Dream, Update, conversation capture, or manual writes implicitly start at zero. + +Use relevance and trust first, then citation history; a popular claim is not +automatically correct. Never omit relevant user constraints because of a low +score. A shortlist is not a complete symbol history. Follow all preference pages +and use targeted searches/pagination when completeness matters. +If retrieval warns that telemetry failed, knowledge remains usable but the +count may be missing. Reuse `--event-id` for retries; `--no-record` is available +for inspection that should not affect ranking. Raw reads remain possible but +are not counted. See the Viewer skill for identity and pagination semantics. + ## Directory Layout ``` @@ -171,7 +191,7 @@ When to write to `_prefs.md` vs per-file shadow vs `_cross/`: ## Discovery Format -Per-file discoveries (no IDs — anchored by their `file::symbol` heading): +Per-file discoveries (no stored IDs — anchored by their `file::symbol` heading): ``` - _(, source: )_ @@ -336,8 +356,9 @@ types, or static properties. Before writing any discovery, follow this procedure: -1. **Read before write**: Read all existing discoveries under the target - `file::symbol`. If an existing discovery makes the same behavioral +1. **Read before write**: Query the target `file::symbol` and search for the + specific claim, expanding candidates and following pages when needed. + Do not infer absence from a citation-ranked shortlist. If an existing discovery makes the same behavioral claim (even if worded differently) → update the existing one. If the new one extends an existing one → merge into a single richer entry. If they conflict → investigate the code, keep the correct one, mark diff --git a/tests/hooks/test_pre_tool_sh.py b/tests/hooks/test_pre_tool_sh.py index 0508621..aee37c6 100644 --- a/tests/hooks/test_pre_tool_sh.py +++ b/tests/hooks/test_pre_tool_sh.py @@ -154,6 +154,24 @@ def test_emits_both_output_shapes(self, coupon_demo): assert hso["hookEventName"] == "PreToolUse" assert hso["additionalContext"] == data["additionalContext"] + def test_citation_failure_is_visible_but_never_denies_edit(self, coupon_demo, tmp_path): + _fix_state_json_for_test(coupon_demo) + ledger = coupon_demo / ".git/shadowfrog/citations.sqlite3" + ledger.parent.mkdir() + ledger.write_bytes(b"invalid sqlite") + dedup = tmp_path / "dedup" + dedup.mkdir() + result = run_hook( + {"tool_name": "edit", "tool_input": {"file_path": "cart.py"}}, + cwd=coupon_demo, env_extra={"SHADOWFROG_TMP_DIR": str(dedup)}, + ) + assert result.returncode == 0 + context = json.loads(result.stdout)["additionalContext"] + assert "Actionable discoveries" in context + assert "citation_score=?" in context + assert "Viewer reported a warning" in context + assert ledger.read_bytes() == b"invalid sqlite" + @pytest.mark.slow @pytest.mark.integration diff --git a/tests/skills/shadow_frog_viewer/conftest.py b/tests/skills/shadow_frog_viewer/conftest.py new file mode 100644 index 0000000..71b710b --- /dev/null +++ b/tests/skills/shadow_frog_viewer/conftest.py @@ -0,0 +1,9 @@ +"""Keep standalone-viewer citation state inside each test's temporary directory.""" + +import pytest + + +@pytest.fixture(autouse=True) +def local_viewer_state(tmp_path, monkeypatch): + monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "viewer-state")) + monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "viewer-state")) diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog_viewer/test_citations.py new file mode 100644 index 0000000..bc21e9c --- /dev/null +++ b/tests/skills/shadow_frog_viewer/test_citations.py @@ -0,0 +1,166 @@ +"""Real SQLite and Git tests for local, multiprocess citation bookkeeping.""" + +from pathlib import Path +import sqlite3 +import subprocess +import sys + +import pytest + +from tests.conftest import _load_script + + +HELPER = Path(__file__).resolve().parents[3] / "skills/shadow-frog-viewer/_citations.py" + + +@pytest.fixture +def citations(): + return _load_script(HELPER) + + +def test_identity_is_stable_without_mutable_metadata(citations): + identity = citations.discovery_id("file", "src/a.py::run", "A claim\nwith spacing.") + assert identity == citations.discovery_id("file", "src/a.py::run", "A claim with spacing.") + assert identity != citations.discovery_id("file", "src/a.py::run", "A different claim.") + assert identity != citations.discovery_id("file", "src/a.py::other", "A claim with spacing.") + assert identity != citations.discovery_id("preference", "src/a.py::run", "A claim with spacing.") + assert citations.discovery_id("cross", "_cross/a.md", "Claim", ["b::f", "a::g"]) == ( + citations.discovery_id("cross", "_cross/a.md", "Claim", ["a::g", "b::f"]) + ) + + +def test_zero_default_atomic_increment_and_event_retry(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + first = citations.discovery_id("file", "a::run", "One") + second = citations.discovery_id("file", "a::run", "Two") + assert store.scores([first, second]) == {} + store.record([first, first], "event-a") + store.record([first], "event-a") + store.record([first, second], "event-b") + assert store.scores([first, second]) == {first: 2, second: 1} + other = citations.CitationStore(store.path, "examples/demo/.shadow") + assert other.scores([first]) == {} + + +def test_processes_do_not_lose_increments(citations, tmp_path): + path = tmp_path / "citations.sqlite3" + identity = citations.discovery_id("file", "a::run", "A concurrent claim") + code = """ +import sys +sys.path.insert(0, sys.argv[1]) +from _citations import CitationStore +from pathlib import Path +store = CitationStore(Path(sys.argv[2]), ".shadow", timeout=5) +for index in range(30): + store.record([sys.argv[3]], f"{sys.argv[4]}-{index}") +""" + processes = [ + subprocess.Popen( + [sys.executable, "-c", code, str(HELPER.parent), str(path), identity, str(worker)], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", + ) + for worker in range(6) + ] + for process in processes: + stdout, stderr = process.communicate(timeout=40) + assert process.returncode == 0, stdout + stderr + assert citations.CitationStore(path, ".shadow").scores([identity])[identity] == 180 + + +def test_worktrees_share_scores_but_not_nested_shadows(citations, coupon_demo, tmp_path): + source = coupon_demo / ".shadow" + checkout = tmp_path / "linked" + subprocess.run( + ["git", "-C", str(coupon_demo), "worktree", "add", "--detach", str(checkout), "HEAD"], + check=True, capture_output=True, + ) + try: + first = citations.CitationStore.for_shadow(source) + second = citations.CitationStore.for_shadow(checkout / ".shadow") + assert first == second + identity = citations.discovery_id("file", "cart.py::calculate_total", "A claim") + first.record([identity], "one") + assert second.scores([identity]) == {identity: 1} + nested = coupon_demo / "example/.shadow" + nested.mkdir(parents=True) + third = citations.CitationStore.for_shadow(nested) + assert third.path == first.path and third.scope != first.scope + assert third.scores([identity]) == {} + status = subprocess.check_output( + ["git", "-C", str(coupon_demo), "status", "--porcelain"], text=True, + ) + assert status == "" + finally: + subprocess.run( + ["git", "-C", str(coupon_demo), "worktree", "remove", str(checkout)], + check=True, capture_output=True, + ) + + +def test_standalone_cache_stays_outside_shadow(citations, tmp_path, monkeypatch): + monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "local")) + monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "state")) + shadow = tmp_path / "repo/.shadow" + shadow.mkdir(parents=True) + store = citations.CitationStore.for_shadow(shadow) + identity = citations.discovery_id("file", "a::f", "Claim") + store.record([identity], "event") + assert not store.path.is_relative_to(shadow) + assert list(shadow.iterdir()) == [] + assert store.scores([identity]) == {identity: 1} + + +def test_locked_ledger_fails_within_bounded_wait(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow", timeout=0.02) + identity = citations.discovery_id("file", "a::f", "Claim") + store.record([identity], "first") + connection = sqlite3.connect(store.path) + try: + connection.execute("BEGIN EXCLUSIVE") + with pytest.raises(sqlite3.OperationalError, match="locked"): + store.record([identity], "second") + finally: + connection.rollback() + connection.close() + assert store.scores([identity])[identity] == 1 + + +def test_cursor_freezes_ids_not_scores(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + ids = [citations.discovery_id("file", "a::f", text) for text in ("A", "B", "C")] + cursor = store.save_page("request", "catalog", ids) + store.record([ids[-1]], "later") + assert store.load_page(cursor, "request", "catalog") == ids + with pytest.raises(ValueError, match="changed"): + store.load_page(cursor, "request", "different catalog") + with pytest.raises(ValueError, match="query"): + store.load_page(cursor, "different request", "catalog") + with pytest.raises(ValueError, match="expired|unavailable"): + store.load_page("0" * 32, "request", "catalog") + + +def test_expired_cursor_gives_restart_guidance(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + cursor = store.save_page("q", "c", ["d_" + "a" * 32]) + with sqlite3.connect(store.path) as db: + db.execute("UPDATE pages SET created = 0") + with pytest.raises(ValueError, match="expired.*rerun"): + store.load_page(cursor, "q", "c") + + +def test_schema_does_not_reset_newer_database(citations, tmp_path): + path = tmp_path / "citations.sqlite3" + with sqlite3.connect(path) as db: + db.execute("PRAGMA user_version = 99") + with pytest.raises(ValueError, match="version"): + citations.CitationStore(path, ".shadow").record(["d_" + "a" * 32], "event") + with sqlite3.connect(path) as db: + assert db.execute("PRAGMA user_version").fetchone()[0] == 99 + + +@pytest.mark.parametrize("event", ["", "x" * 129, "bad\nvalue"]) +def test_invalid_event_id_fails_early(citations, tmp_path, event): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + with pytest.raises(ValueError, match="event"): + store.record(["d_" + "a" * 32], event) + assert not store.path.exists() diff --git a/tests/skills/shadow_frog_viewer/test_retrieval.py b/tests/skills/shadow_frog_viewer/test_retrieval.py new file mode 100644 index 0000000..d648538 --- /dev/null +++ b/tests/skills/shadow_frog_viewer/test_retrieval.py @@ -0,0 +1,431 @@ +"""Citation ranking, bounded context, expansion, and stable pagination.""" + +from dataclasses import replace +import re +import sqlite3 +import subprocess +import sys +import time + +import pytest + + +def make_shadow(tmp_path, count=4, text_size=20): + shadow = tmp_path / ".shadow" + shadow.mkdir(exist_ok=True) + path = shadow / "source.py.md" + path.write_text( + "# Shadow: source.py\n\n## `run`\n\n" + "\n".join( + f"- Claim {index:05}: " + ("details " * text_size) + + "\n _(verified, source: exploration, labels: [bug])_\n" + for index in range(count) + ) + "\n## Cross-References\n", + encoding="utf-8", + ) + return shadow + + +def ids(text): + return re.findall(r"\bid=(d_[0-9a-f]{32})", text) + + +def cursor(text): + match = re.search(r"--cursor ([0-9a-f]{32}:\d+)", text) + return match.group(1) if match else None + + +def test_only_emitted_entries_count_and_no_markdown_changes(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 12) + before = {path: path.read_bytes() for path in shadow.rglob("*") if path.is_file()} + entries = shadow_viewer._knowledge_entries(shadow) + store = shadow_viewer.CitationStore.for_shadow(shadow) + assert store.scores([entry["id"] for entry in entries]) == {} + shadow_viewer.view_search( + shadow, "Claim", options=shadow_viewer.RetrievalOptions(limit=2, event_id="first"), + ) + output = capsys.readouterr().out + emitted = ids(output) + assert len(emitted) == 2 and "citation_score=0" in output + assert store.scores([entry["id"] for entry in entries]) == dict.fromkeys(emitted, 1) + assert all(path.read_bytes() == content for path, content in before.items()) + assert sorted(path.name for path in shadow.iterdir()) == ["source.py.md"] + + +def test_retries_and_no_record_do_not_double_count(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + options = shadow_viewer.RetrievalOptions(event_id="retry") + for _ in range(2): + shadow_viewer.view_search(shadow, "Claim", options=options) + output = capsys.readouterr().out + identity = ids(output)[0] + store = shadow_viewer.CitationStore.for_shadow(shadow) + assert store.scores([identity]) == {identity: 1} + shadow_viewer.view_get(shadow, identity, options=replace(options, record=False)) + capsys.readouterr() + assert store.scores([identity]) == {identity: 1} + + +def test_ten_thousand_claims_have_bounded_output(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 10000) + options = shadow_viewer.RetrievalOptions(limit=10, max_chars=1800) + started = time.monotonic() + shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + output = capsys.readouterr().out + assert len(output) <= 1800 + assert "10000 results" in output and len(ids(output)) <= 10 + assert ids(output) and cursor(output) + assert time.monotonic() - started < 10 + store = shadow_viewer.CitationStore.for_shadow(shadow) + all_ids = [entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)] + assert len(all_ids) == len(set(all_ids)) == 10000 + assert store.scores(all_ids) == dict.fromkeys(ids(output), 1) + + +def test_cursor_is_stable_despite_other_readers_updating_scores(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 23, text_size=2) + options = shadow_viewer.RetrievalOptions(limit=3, max_chars=1400) + entries = shadow_viewer._knowledge_entries(shadow) + expected = {entry["id"] for entry in entries} + found = [] + next_cursor = None + while True: + shadow_viewer.view_symbol( + shadow, "source.py::run", options=replace(options, cursor=next_cursor), + ) + output = capsys.readouterr().out + assert len(output) <= 1400 + found.extend(ids(output)) + next_cursor = cursor(output) + if next_cursor is None: + break + shadow_viewer.CitationStore.for_shadow(shadow).record( + [entries[-1]["id"]], f"concurrent-{len(found)}", + ) + assert len(found) == len(set(found)) == 23 + assert set(found) == expected + + +def test_changed_knowledge_invalidates_cursor(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path) + options = shadow_viewer.RetrievalOptions(limit=1) + shadow_viewer.view_search(shadow, "Claim", options=options) + previous = cursor(capsys.readouterr().out) + path = shadow / "source.py.md" + path.write_text(path.read_text(encoding="utf-8").replace("details", "different"), encoding="utf-8") + with pytest.raises(ValueError, match="changed"): + shadow_viewer.view_search(shadow, "Claim", options=replace(options, cursor=previous)) + + +def test_removed_results_invalidate_cursor_instead_of_claiming_empty_success(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path) + options = shadow_viewer.RetrievalOptions(limit=1) + shadow_viewer.view_search(shadow, "Claim", options=options) + previous = cursor(capsys.readouterr().out) + (shadow / "source.py.md").unlink() + with pytest.raises(ValueError, match="changed"): + shadow_viewer.view_search(shadow, "Claim", options=replace(options, cursor=previous)) + + +def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_viewer, tmp_path): + entries = [ + {"id": "user", "source": "user", "status": "verified"}, + {"id": "popular", "source": "exploration", "status": "verified"}, + {"id": "popular2", "source": "exploration", "status": "verified"}, + {"id": "popular3", "source": "exploration", "status": "verified"}, + {"id": "new", "source": "exploration", "status": "verified"}, + {"id": "refuted", "source": "user", "status": "refuted"}, + ] + scores = {"popular": 100, "popular2": 90, "popular3": 80, "refuted": 10000} + ranked = shadow_viewer._rank_entries(entries, scores, 3) + assert ranked[0]["id"] == "user" and ranked[-1]["id"] == "refuted" + assert [entry["id"] for entry in ranked][1:4] == ["popular", "popular2", "new"] + + +def test_zero_score_opportunity_accounts_for_character_budget(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 7) + entries = shadow_viewer._knowledge_entries(shadow) + store = shadow_viewer.CitationStore.for_shadow(shadow) + store.record([entry["id"] for entry in entries[:-1]], "prior-read") + shadow_viewer.view_search( + shadow, "Claim", options=shadow_viewer.RetrievalOptions(limit=10, max_chars=1000), + ) + output = capsys.readouterr().out + assert len(output) <= 1000 + assert len(ids(output)) >= 2 + assert entries[-1]["id"] in ids(output) + + +def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_viewer, tmp_path): + shadow = make_shadow(tmp_path, 1) + original = shadow_viewer._knowledge_entries(shadow)[0]["id"] + path = shadow / "source.py.md" + path.write_text( + path.read_text(encoding="utf-8").replace( + "verified, source: exploration, labels: [bug]", "uncertain, source: interaction" + ), + encoding="utf-8", + ) + assert shadow_viewer._knowledge_entries(shadow)[0]["id"] == original + path.write_text(path.read_text(encoding="utf-8").replace("Claim", "Different claim"), encoding="utf-8") + assert shadow_viewer._knowledge_entries(shadow)[0]["id"] != original + + +def test_duplicate_preferences_share_one_identity_without_losing_trust(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + (shadow / "_prefs.md").write_text( + "# Preferences\n\n- Preserve the contract.\n _(source: interaction)_\n" + "\n- Preserve the contract.\n _(source: user)_\n", + encoding="utf-8", + ) + shadow_viewer.view_prefs(shadow) + output = capsys.readouterr().out + assert "Project Preferences (1 total)" in output and "[user]" in output + assert len(ids(output)) == 1 + assert shadow_viewer.CitationStore.for_shadow(shadow).scores(ids(output)) == dict.fromkeys(ids(output), 1) + + +def test_get_chunks_long_claim_without_exceeding_budget(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1, text_size=500) + entry = shadow_viewer._knowledge_entries(shadow)[0] + identity = entry["id"] + offset = 0 + pieces = [] + while True: + shadow_viewer.view_get( + shadow, identity, + options=shadow_viewer.RetrievalOptions(max_chars=500, text_offset=offset, event_id="read-one"), + ) + output = capsys.readouterr().out + assert len(output) <= 500 + match = re.search(r"--text-offset (\d+)", output) + pieces.append(output.removesuffix("\n").split("\n", 2)[2].split("\nContinue:", 1)[0]) + if match is None: + break + offset = int(match.group(1)) + assert "".join(pieces) == entry["anchor"] + "\n\n" + entry["text"] + "\nLabels: bug" + assert shadow_viewer.CitationStore.for_shadow(shadow).scores([identity]) == {identity: 1} + + +def test_summary_and_invariant_checks_do_not_count(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path) + store = shadow_viewer.CitationStore.for_shadow(shadow) + shadow_viewer.view_summary(shadow) + shadow_viewer.view_check_invariants(shadow) + capsys.readouterr() + assert not store.path.exists() + + +def test_all_creation_sources_start_at_zero_without_schema_changes( + shadow_viewer, dream_reconcile, tmp_path, capsys, +): + shadow = tmp_path / ".shadow" + discoveries = [ + {"anchor": f"source.py::run_{source}", "text": f"Behavior from {source}.", + "source": source, "status": "verified"} + for source in ("exploration", "user", "interaction") + ] + dream_reconcile.merge_discoveries( + str(tmp_path), + [("dream/test/20260923-000000Z-test", "20260923-000000Z-test", + {"discoveries": discoveries})], + ) + entries = shadow_viewer._knowledge_entries(shadow) + assert len(entries) == 3 + assert shadow_viewer.CitationStore.for_shadow(shadow).scores([entry["id"] for entry in entries]) == {} + shadow_viewer.view_search(shadow, "Behavior", options=shadow_viewer.RetrievalOptions(record=False)) + output = capsys.readouterr().out + assert output.count("citation_score=0") == 3 + assert "citation_score" not in (shadow / "source.py.md").read_text(encoding="utf-8") + + +def test_top_hard_budget_counts_only_visible_claims(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 10, text_size=150) + shadow_viewer.view_top(shadow, "source.py", "bug", 3, 600) + output = capsys.readouterr().out + assert len(output) <= 600 + emitted = ids(output) + assert emitted and len(emitted) <= 3 + assert f"Top {len(emitted)} of 10" in output + store = shadow_viewer.CitationStore.for_shadow(shadow) + assert store.scores([entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)]) == dict.fromkeys(emitted, 1) + + +def test_smallest_budget_still_emits_identifiable_content(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 10) + shadow_viewer.view_search(shadow, "Claim", options=shadow_viewer.RetrievalOptions(max_chars=256)) + output = capsys.readouterr().out + assert len(output) <= 256 and len(ids(output)) == 1 + assert "verified" in output and cursor(output) + + +def test_broken_ledger_returns_knowledge_with_warning(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + store = shadow_viewer.CitationStore.for_shadow(shadow) + store.path.parent.mkdir(parents=True) + store.path.write_bytes(b"not sqlite") + shadow_viewer.view_search(shadow, "Claim") + captured = capsys.readouterr() + assert "Claim 00000" in captured.out and "citation_score=?" in captured.out + assert "warning" in captured.err and "citation" in captured.err + assert store.path.read_bytes() == b"not sqlite" + + +@pytest.mark.parametrize("view", ["search", "symbol", "get", "prefs", "labels", "recent", "top"]) +def test_all_content_views_share_identity_and_score(shadow_viewer, tmp_path, capsys, view): + shadow = make_shadow(tmp_path, 1, text_size=2) + (shadow / "_prefs.md").write_text("# Preferences\n\n- Keep the contract.\n _(source: user)_\n", encoding="utf-8") + options = shadow_viewer.RetrievalOptions(event_id="same-visit") + entry = next(item for item in shadow_viewer._knowledge_entries(shadow) if item["kind"] != "preference") + if view == "search": + shadow_viewer.view_search(shadow, "Claim", options=options) + elif view == "symbol": + shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + elif view == "get": + shadow_viewer.view_get(shadow, entry["id"], options=options) + elif view == "prefs": + shadow_viewer.view_prefs(shadow, options=options) + elif view == "labels": + shadow_viewer.view_labels(shadow, "bug", options=options) + elif view == "recent": + shadow_viewer.view_recent(shadow, options=options) + else: + shadow_viewer.view_top(shadow, "source.py", "", 3, 600, options=options) + output = capsys.readouterr().out + emitted = ids(output) + store = shadow_viewer.CitationStore.for_shadow(shadow) + assert emitted and store.scores(emitted) == dict.fromkeys(emitted, 1) + if view != "prefs": + assert entry["id"] in emitted + + +def test_write_failure_warns_without_hiding_knowledge(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + store = shadow_viewer.CitationStore.for_shadow(shadow) + identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] + store.record([identity], "initial") + connection = sqlite3.connect(store.path) + try: + connection.execute("BEGIN IMMEDIATE") + shadow_viewer.view_search(shadow, "Claim") + captured = capsys.readouterr() + assert identity in captured.out + assert "scores were not updated" in captured.err + finally: + connection.rollback() + connection.close() + assert store.scores([identity])[identity] == 1 + + +def test_failed_stdout_does_not_increment_score(shadow_viewer, tmp_path, monkeypatch): + shadow = make_shadow(tmp_path, 1) + store = shadow_viewer.CitationStore.for_shadow(shadow) + identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] + + class ClosedOutput: + def write(self, text): + raise BrokenPipeError("reader closed the pipe") + + def flush(self): + pass + + monkeypatch.setattr(sys, "stdout", ClosedOutput()) + with pytest.raises(BrokenPipeError): + shadow_viewer.view_search(shadow, "Claim") + assert store.scores([identity]) == {} + + +def test_symbol_context_includes_related_cross_but_not_other_symbols(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + with (shadow / "source.py.md").open("a", encoding="utf-8") as stream: + stream.write("\n## `unrelated`\n\n- Unrelated claim.\n _(verified, source: user)_\n") + cross = shadow / "_cross/related.md" + cross.parent.mkdir() + cross.write_text( + "# Related behavior\n\n**Category**: contract\n**Refs**:\n" + "- `source.py::run`\n- `other.py::call`\n\n" + "**Discovery**: Cross-file constraint.\n\n_(verified, source: user)_\n", + encoding="utf-8", + ) + shadow_viewer.view_symbol(shadow, "source.py::run") + output = capsys.readouterr().out + assert "Cross-file constraint." in output + assert "Claim 00000" in output and "Unrelated claim." not in output + + +def test_nested_paths_have_one_identity_across_scoped_and_global_views(shadow_viewer, tmp_path): + shadow = tmp_path / ".shadow" + source = shadow / "src/caf\u00e9 tools.py.md" + source.parent.mkdir(parents=True) + source.write_text( + "# Shadow: src/caf\u00e9 tools.py\n\n## `run`\n\n" + "- Nested claim.\n _(verified, source: exploration)_\n", + encoding="utf-8", + ) + global_entry = shadow_viewer._knowledge_entries(shadow)[0] + scoped_entry = shadow_viewer._knowledge_entries(shadow, "src/caf\u00e9 tools.py")[0] + assert global_entry["anchor"] == scoped_entry["anchor"] == "src/caf\u00e9 tools.py::run" + assert global_entry["id"] == scoped_entry["id"] + + +def test_expansion_preserves_long_anchors(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1) + path = shadow / "source.py.md" + symbol = "method_" + "name" * 50 + path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{symbol}`"), encoding="utf-8") + entry = shadow_viewer._knowledge_entries(shadow)[0] + shadow_viewer.view_get(shadow, entry["id"]) + assert "source.py::" + symbol in capsys.readouterr().out + + +def test_cursor_cli_continuation_and_get_are_wired(repo_root, tmp_path): + shadow = make_shadow(tmp_path) + command = [ + sys.executable, str(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py"), + "--shadow-dir", str(shadow), "--symbol", "source.py::run", + "--limit", "1", "--max-chars", "600", + ] + first = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", check=True) + second = subprocess.run( + [*command, "--cursor", cursor(first.stdout)], + capture_output=True, text=True, encoding="utf-8", check=True, + ) + assert ids(first.stdout) != ids(second.stdout) + expanded = subprocess.run( + command[:4] + ["--get", ids(first.stdout)[0], "--max-chars", "600", "--no-record"], + capture_output=True, text=True, encoding="utf-8", check=True, + ) + assert ids(expanded.stdout) == ids(first.stdout) + assert "citation_score=1" in expanded.stdout + + +@pytest.mark.slow +def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_path): + import shutil + + for agent in (".github", ".claude"): + installed = tmp_path / agent / "skills/shadow-frog-viewer" + shutil.copytree(repo_root / "skills/shadow-frog-viewer", installed) + command = [ + sys.executable, str(installed / "shadow-viewer.py"), + "--shadow-dir", str(coupon_demo / ".shadow"), "--search", "coupon", + "--limit", "2", "--max-chars", "1000", "--event-id", agent, + ] + result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", check=True) + assert len(result.stdout) <= 1000 and len(ids(result.stdout)) == 2 + assert "citation_score=" in result.stdout + + +@pytest.mark.parametrize("args", [ + ["--summary", "--limit", "2"], + ["--search", "claim", "--limit", "0"], + ["--search", "claim", "--max-chars", "20"], + ["--search", "claim", "--text-offset", "1"], + ["--top", "source.py", "--max-chars", "600"], + ["--get", "d_" + "a" * 32, "--cursor", "bad"], +]) +def test_invalid_cli_combinations_are_explicit(repo_root, tmp_path, args): + result = subprocess.run( + [sys.executable, str(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py"), *args], + cwd=tmp_path, capture_output=True, text=True, encoding="utf-8", + ) + assert result.returncode == 2 and "error:" in result.stderr diff --git a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py index 6e3bdf1..2f13fc4 100644 --- a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py +++ b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py @@ -1,9 +1,8 @@ r"""Tests for `skills/shadow-frog-viewer/shadow-viewer.py`. -Philosophy: USE REAL FILES (per `minimal-mocking-tests`). The viewer is a -pure-read tool — every test either constructs a small shadow tree on -disk and calls a viewer function, or exercises the CLI via subprocess -against the `coupon_demo` fixture. +Philosophy: USE REAL FILES (per `minimal-mocking-tests`). Retrieval does not +edit shadow Markdown; local citation bookkeeping is isolated by the fixtures. +Tests construct shadow trees or exercise the CLI against `coupon_demo`. Test categories: * In-process function tests (no `@pytest.mark.slow`): exercise @@ -1390,7 +1389,9 @@ def test_cross_cutting_labels_included( self, shadow_viewer, coupon_demo, capsys ): sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, "bug") + shadow_viewer.view_labels( + sd, "bug", options=shadow_viewer.RetrievalOptions(limit=20, max_chars=8000), + ) out = capsys.readouterr().out # Cross-cutting entries are prefixed with `_cross/` in the # file column. From d2143cc97d4038d494adfad079da79c6e6cbd9d1 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 18:06:01 -0400 Subject: [PATCH 02/11] Keep search text out of the citation pagination ledger Bind cursor requests by digest rather than storing raw query text in local usage metadata. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- skills/shadow-frog-viewer/_citations.py | 2 ++ tests/skills/shadow_frog_viewer/test_citations.py | 11 +++++++++++ 2 files changed, 13 insertions(+) diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog-viewer/_citations.py index f31132a..7a53465 100644 --- a/skills/shadow-frog-viewer/_citations.py +++ b/skills/shadow-frog-viewer/_citations.py @@ -140,6 +140,7 @@ def record(self, ids, event): def save_page(self, request, catalog, ids): token = uuid.uuid4().hex + request = hashlib.sha256(request.encode("utf-8")).hexdigest() with self._connection() as db, db: db.execute("DELETE FROM pages WHERE created < ?", (time.time() - PAGE_TTL,)) db.execute( @@ -149,6 +150,7 @@ def save_page(self, request, catalog, ids): return token def load_page(self, token, request, catalog): + request = hashlib.sha256(request.encode("utf-8")).hexdigest() with self._connection() as db: row = db.execute( "SELECT request, catalog, ids, created FROM pages WHERE token=? AND scope=?", diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog_viewer/test_citations.py index bc21e9c..0dd99c8 100644 --- a/tests/skills/shadow_frog_viewer/test_citations.py +++ b/tests/skills/shadow_frog_viewer/test_citations.py @@ -139,6 +139,17 @@ def test_cursor_freezes_ids_not_scores(citations, tmp_path): store.load_page("0" * 32, "request", "catalog") +def test_pagination_stores_query_hash_not_search_text(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + query = "distinctive search phrase about the code" + cursor = store.save_page(query, "catalog", ["d_" + "a" * 32]) + with sqlite3.connect(store.path) as db: + stored = db.execute("SELECT request FROM pages").fetchone()[0] + assert stored != query and len(stored) == 64 + assert store.load_page(cursor, query, "catalog") == ["d_" + "a" * 32] + assert query.encode() not in store.path.read_bytes() + + def test_expired_cursor_gives_restart_guidance(citations, tmp_path): store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") cursor = store.save_page("q", "c", ["d_" + "a" * 32]) From 9eb548aee950d54778fb2c8f2b7c342e11fa2ab6 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 18:55:48 -0400 Subject: [PATCH 03/11] Handle concurrent citation writes within bounded lock budgets Reuse fully synchronized rollback journals and retry only SQLite lock contention after rolling back each whole write attempt. Keep hook timeouts and the six-process exact-count workload unchanged; cover blocked commits and non-lock failures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 2 + skills/shadow-frog-viewer/SKILL.md | 5 +- skills/shadow-frog-viewer/_citations.py | 39 +++++++-- .../shadow_frog_viewer/test_citations.py | 82 ++++++++++++++++++- 4 files changed, 119 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f10b22f..08e3c63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,8 @@ shadow knowledge bases for any codebase. Citation scores measure emitted content, not correctness or proven usefulness. ### Changed +- Citation writes retry temporary SQLite lock contention within their existing + wait budget and reuse the rollback journal without weakening synchronization. - Retrieval views now show IDs/scores and paginate by default. Preferences remain separately accessible, trust and relevance precede popularity, and hooks remain fail-open while surfacing citation warnings. diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 13ce939..193fbe9 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -113,7 +113,10 @@ roots in the same repo have separate scopes. Outside Git, the cache lives under not discovery bodies. It is local metadata, not a tracked or multi-machine DB; Markdown and its format remain authoritative and unchanged. -Transactions prevent lost increments from concurrent agents. Reuse `--event-id` +Transactions prevent lost increments from concurrent agents. Lock contention +retries the whole transaction within the original short wait budget; each failed +attempt rolls back. The ledger reuses its rollback journal with full disk +synchronization rather than repeatedly creating/deleting it. Reuse `--event-id` when retrying one retrieval; do not reuse it for unrelated visits. A failed stdout emission is not recorded. Ledger failures warn on stderr, return the knowledge, and show unknown scores as `?` when scores cannot be read. diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog-viewer/_citations.py index 7a53465..a92f710 100644 --- a/skills/shadow-frog-viewer/_citations.py +++ b/skills/shadow-frog-viewer/_citations.py @@ -6,6 +6,7 @@ import json import os from pathlib import Path +import random import re import sqlite3 import subprocess @@ -76,13 +77,19 @@ def for_shadow(cls, shadow_dir): return cls(state / "shadowfrog/citations" / f"{identity}.sqlite3", str(shadow)) @contextmanager - def _connection(self): + def _connection(self, *, timeout=None): self.path.parent.mkdir(parents=True, exist_ok=True) - db = sqlite3.connect(self.path, timeout=self.timeout) + db = sqlite3.connect(self.path, timeout=self.timeout if timeout is None else timeout) try: version = db.execute("PRAGMA user_version").fetchone()[0] if version not in (0, 1): raise ValueError(f"Unsupported citation database version {version}; use a compatible helper") + # Reuse the rollback journal instead of creating/deleting it on every + # tiny update, while retaining FULL synchronization and atomic commits. + mode = db.execute("PRAGMA journal_mode=PERSIST").fetchone()[0] + if mode != "persist": + raise ValueError(f"Cannot enable persistent citation journaling (got {mode})") + db.execute("PRAGMA synchronous=FULL") if version == 0: with db: db.execute("BEGIN IMMEDIATE") @@ -103,6 +110,27 @@ def _connection(self): finally: db.close() + def _write(self, action): + """Retry only lock contention, rolling back each attempt within one budget.""" + deadline = time.monotonic() + self.timeout + while True: + try: + wait = min(0.01, max(0, deadline - time.monotonic())) + with self._connection(timeout=wait) as db, db: + db.execute("BEGIN IMMEDIATE") + return action(db) + except sqlite3.OperationalError as exc: + code = getattr(exc, "sqlite_errorcode", None) + locked = ( + (code is not None and code & 0xff in (5, 6)) # SQLITE_BUSY / SQLITE_LOCKED + or (code is None and str(exc) in ("database is locked", "database table is locked")) + ) + remaining = deadline - time.monotonic() + if not locked or remaining <= 0: + raise + # Avoid letting repeated writers monopolize SQLite's polling slots. + time.sleep(min(remaining, random.uniform(0.001, 0.01))) + def scores(self, ids): if not self.path.exists() or not ids: return {} @@ -124,8 +152,7 @@ def record(self, ids, event): ids = list(dict.fromkeys(ids)) if not ids: return - with self._connection() as db, db: - db.execute("BEGIN IMMEDIATE") + def update(db): for identity in ids: inserted = db.execute( "INSERT OR IGNORE INTO events(scope, event, id) VALUES (?, ?, ?)", @@ -137,16 +164,18 @@ def record(self, ids, event): "ON CONFLICT(scope, id) DO UPDATE SET citation_score=citation_score+1", (self.scope, identity), ) + self._write(update) def save_page(self, request, catalog, ids): token = uuid.uuid4().hex request = hashlib.sha256(request.encode("utf-8")).hexdigest() - with self._connection() as db, db: + def publish(db): db.execute("DELETE FROM pages WHERE created < ?", (time.time() - PAGE_TTL,)) db.execute( "INSERT INTO pages VALUES (?, ?, ?, ?, ?, ?)", (token, self.scope, request, catalog, json.dumps(ids), time.time()), ) + self._write(publish) return token def load_page(self, token, request, catalog): diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog_viewer/test_citations.py index 0dd99c8..d30b7a5 100644 --- a/tests/skills/shadow_frog_viewer/test_citations.py +++ b/tests/skills/shadow_frog_viewer/test_citations.py @@ -4,6 +4,7 @@ import sqlite3 import subprocess import sys +import time import pytest @@ -61,12 +62,85 @@ def test_processes_do_not_lose_increments(citations, tmp_path): ) for worker in range(6) ] - for process in processes: - stdout, stderr = process.communicate(timeout=40) - assert process.returncode == 0, stdout + stderr + outcomes = [] + try: + for process in processes: + stdout, stderr = process.communicate(timeout=40) + outcomes.append((process.returncode, stdout, stderr)) + finally: + for process in processes: + if process.poll() is None: + process.terminate() + process.communicate(timeout=10) + assert all(code == 0 for code, _, _ in outcomes), outcomes assert citations.CitationStore(path, ".shadow").scores([identity])[identity] == 180 +def test_writes_reuse_journal_without_disabling_synchronization(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + identity = citations.discovery_id("file", "a::f", "Claim") + store.record([identity], "first") + with store._connection() as db: + assert db.execute("PRAGMA journal_mode").fetchone()[0] == "persist" + assert db.execute("PRAGMA synchronous").fetchone()[0] >= 2 + journal = tmp_path / "citations.sqlite3-journal" + assert journal.is_file() + assert journal.read_bytes()[:28] == b"\0" * 28 + store.record([identity], "second") + assert store.scores([identity])[identity] == 2 + + +def test_busy_commit_retries_whole_transaction_without_duplicate_updates(citations, tmp_path): + path = tmp_path / "citations.sqlite3" + store = citations.CitationStore(path, ".shadow") + identity = citations.discovery_id("file", "a::f", "Claim") + store.record([identity], "initial") + reader = sqlite3.connect(path) + reader.execute("BEGIN") + reader.execute("SELECT * FROM scores").fetchall() + code = """ +import sys +from pathlib import Path +sys.path.insert(0, sys.argv[1]) +from _citations import CitationStore +store = CitationStore(Path(sys.argv[2]), ".shadow", timeout=2) +def update(db): + db.execute("UPDATE scores SET citation_score = citation_score + 1") + print("attempt", flush=True) +store._write(update) +""" + process = subprocess.Popen( + [sys.executable, "-c", code, str(HELPER.parent), str(path)], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", + ) + try: + # Reader permits BEGIN IMMEDIATE and the update, but prevents COMMIT. + assert process.stdout.readline().strip() == "attempt" + assert process.stdout.readline().strip() == "attempt" + reader.rollback() + stdout, stderr = process.communicate(timeout=15) + assert process.returncode == 0, stdout + stderr + finally: + reader.close() + if process.poll() is None: + process.terminate() + process.communicate(timeout=10) + assert store.scores([identity])[identity] == 2 + + +def test_non_lock_errors_are_not_retried(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + attempts = [] + + def invalid_sql(db): + attempts.append(True) + db.execute("INSERT INTO nonexistent_table VALUES (1)") + + with pytest.raises(sqlite3.OperationalError, match="no such table"): + store._write(invalid_sql) + assert len(attempts) == 1 + + def test_worktrees_share_scores_but_not_nested_shadows(citations, coupon_demo, tmp_path): source = coupon_demo / ".shadow" checkout = tmp_path / "linked" @@ -117,8 +191,10 @@ def test_locked_ledger_fails_within_bounded_wait(citations, tmp_path): connection = sqlite3.connect(store.path) try: connection.execute("BEGIN EXCLUSIVE") + start = time.monotonic() with pytest.raises(sqlite3.OperationalError, match="locked"): store.record([identity], "second") + assert time.monotonic() - start < 0.5 finally: connection.rollback() connection.close() From 9ed972a27e86b43fd14d75794861ff32dbee150d Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 20:30:00 -0400 Subject: [PATCH 04/11] Refine citation identity, retrieval, and local accounting Keep canonical file/symbol identities, literal text, and merged labels intact. Share hook budgets across previews, bind long reads to their content revision and event, and bound local retry/pagination metadata while preserving fail-open accounting. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/tests.yml | 15 + CHANGELOG.md | 4 + README.md | 4 +- RESPONSIBLE_AI.md | 4 + agent-context.md | 9 +- claude.md | 8 +- skills/shadow-frog-viewer/SKILL.md | 62 ++-- skills/shadow-frog-viewer/_citations.py | 171 ++++++--- skills/shadow-frog-viewer/shadow-viewer.py | 329 ++++++++++++------ skills/shadow-frog/SKILL.md | 8 +- tests/hooks/test_pre_tool_sh.py | 2 +- .../shadow_frog_viewer/test_citations.py | 130 ++++++- .../shadow_frog_viewer/test_retrieval.py | 301 +++++++++++++++- 13 files changed, 855 insertions(+), 192 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 6d16cb1..07f318f 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -8,6 +8,21 @@ permissions: contents: read jobs: + viewer-compatibility: + name: Viewer (Python 3.9) + runs-on: ubuntu-latest + steps: + - name: Checkout + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - name: Set up Python + uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 + with: + python-version: '3.9' + - name: Exercise the standalone viewer + run: | + python skills/shadow-frog-viewer/shadow-viewer.py --help + python skills/shadow-frog-viewer/shadow-viewer.py --shadow-dir examples/coupon-demo/.shadow --search coupon --limit 2 --no-record + pytest: name: pytest (${{ matrix.os }}, Python 3.12) runs-on: ${{ matrix.os }} diff --git a/CHANGELOG.md b/CHANGELOG.md index 08e3c63..5828c5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,10 @@ shadow knowledge bases for any codebase. ### Changed - Citation writes retry temporary SQLite lock contention within their existing wait budget and reuse the rollback journal without weakening synchronization. +- Retrieval keeps file/symbol identities consistent, preserves literal whitespace + and duplicate labels, and binds expansion continuations to one unchanged logical + read. Compact hooks share their budget across several previews, and local retry + receipts, pagination snapshots, and journals have explicit retention limits. - Retrieval views now show IDs/scores and paginate by default. Preferences remain separately accessible, trust and relevance precede popularity, and hooks remain fail-open while surfacing citation warnings. diff --git a/README.md b/README.md index 9c73b38..36cfbcc 100644 --- a/README.md +++ b/README.md @@ -123,7 +123,9 @@ claims. A single local `citation_score` counts helper exposures, not proven usefulness; relevance and trust outrank popularity. Scores are updated safely across local Git worktrees without editing shadow Markdown or requiring a vector index. New claims start at zero. See the Viewer reference for retries, local -storage, and the limits of this signal. +storage, and the limits of this signal. Long-entry continuation counts as one +logical read and rejects changed content; local retry and pagination metadata +have retention limits. Compact hook hints are not a substitute for a file review. --- diff --git a/RESPONSIBLE_AI.md b/RESPONSIBLE_AI.md index 5a198ad..b161a69 100644 --- a/RESPONSIBLE_AI.md +++ b/RESPONSIBLE_AI.md @@ -105,6 +105,10 @@ by prior ranking and repeated exposure; relevance, provenance, and verification remain more important. Bounded results may omit relevant knowledge, so agents must follow pagination and inspect preferences rather than treating a shortlist as exhaustive. Claim rewrites and renames may create fresh zero-score identities. +Continuation chunks share one logical-read event. Short contention budgets can +leave explicitly warned, unrecorded visits; atomicity does not guarantee complete +accounting. Retry receipts and pagination snapshots expire or reach capacity, +while score storage grows with distinct knowledge identities. ShadowFrog was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios. diff --git a/agent-context.md b/agent-context.md index 98af356..ee980c6 100644 --- a/agent-context.md +++ b/agent-context.md @@ -2,9 +2,9 @@ This project uses a `.shadow/` knowledge base with verified discoveries about non-obvious code behavior. **You MUST consult the shadow before making any code change.** -1. **Check the shadow first** — use `/shadow-frog-viewer --symbol ::` - or `--top ` before editing; expand returned IDs with `--get` and page - large results rather than loading an entire shadow into context. +1. **Check the shadow first** — use `/shadow-frog-viewer --search ` before + editing and `--symbol ::` for focused follow-up. Expand IDs with + `--get` and page large results. Hook-sized `--top` hints are not exhaustive. 2. **Check preferences** — use `--prefs` and follow all pages for project conventions. 3. **Check cross-cutting** — use `--search` for related multi-file knowledge. 4. **Act on what you find** — apply what you learn from the shadow to your work. @@ -13,7 +13,8 @@ This project uses a `.shadow/` knowledge base with verified discoveries about no The shadow contains discoveries from code analysis and user conversations. Always consult it before making assumptions about code behavior. The viewer records one local `citation_score` per emitted discovery, atomically -across worktrees; never edit counters in Markdown. It is an exposure count, not +across worktrees; continuation chunks share one logical read. Never edit counters +in Markdown. It is an exposure count, not proof of correctness or usefulness. Trust and relevance outrank popularity. Raw file reads are a fallback and are not counted. A ranked shortlist is not exhaustive: search the specific claim before adding duplicate knowledge. diff --git a/claude.md b/claude.md index babdf92..5870599 100644 --- a/claude.md +++ b/claude.md @@ -122,14 +122,16 @@ Preference (`_prefs.md` — project-wide, no file/symbol anchor): - Keep the discovery grammar unchanged: viewer fingerprints and `citation_score` are derived/local metadata, not additional Markdown fields. - Scores start at zero and increment only for content emitted by a retrieval - view, once per entry/event. They measure exposure, not verified usefulness. + view; expansion chunks share one revision-bound logical read. They measure + exposure, not verified usefulness. - Use the common-Git SQLite ledger for multiprocess/worktree updates; do not rewrite shadow files on reads. Telemetry failures must warn without hiding knowledge. Summary/audit/parser-only operations do not increment scores. - Rank relevance and trust ahead of scores; preserve room for zero-score entries. Page large results, expand by ID, and never use only the shortlist for dedup. -- Fingerprints bind kind, anchor, normalized claim and refs. Metadata-only edits - retain identity; rewritten/merged claims and renamed anchors may reset scores. +- Fingerprints bind kind, canonical anchor, whitespace-preserving parsed claim + and refs. Metadata-only edits retain identity; rewritten/merged claims and + renamed anchors may reset scores. Deduplication must retain all labels. ### Dedup - Before writing, read existing discoveries at the target symbol. diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 193fbe9..7936fe4 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -17,7 +17,7 @@ Query and browse `.shadow/` content. Prerequisite: `.shadow/` exists. ## Primary: Python Helper Script -The companion script `shadow-viewer.py` lives in the same directory as +The companion script `shadow-viewer.py` supports Python 3.9+ and lives beside this SKILL.md file. To find and run it: ```bash @@ -34,11 +34,11 @@ python3 .claude/skills/shadow-frog-viewer/shadow-viewer.py [options] | `--summary` | Overview: counts, source/status/label breakdown, per-file table, cross-cutting titles (default) | | `--search QUERY` | Bounded search across paths, symbols, text, cross-cutting entries, and preferences | | `--symbol FILE::SYMBOL` | Bounded discoveries at an exact symbol, plus matching cross-cutting refs; use `File-Level` for a file-level section | -| `--get ID` | Expand one current discovery returned by a content view | +| `--get ID` | Expand one current discovery; long expansions return a revision-bound continuation | | `--prefs` | Project-wide preferences; follow all pages before treating them as complete | | `--recent [N]` | Most recent discovery previews by file mtime (default page size: 10) | | `--labels LABEL` | Bounded discoveries matching labels (e.g., `bug`, `security`, `bug,performance`) | -| `--top FILE` | Hook-sized actionable previews (default: up to 3 entries, 600 characters). Includes per-file and cross-cutting findings; trust/status precedes citation score. | +| `--top FILE` | Hook-sized actionable previews (default: up to 3 entries, 600 characters). Includes per-file and cross-cutting findings; short symbol/ID lines omit numeric scores to leave room for content. Not an exhaustive file view. | | `--check-invariants` | Audit structural integrity — bidirectional cross-references, label/source/category enum compliance, heading format, no-orphan-back-pointer. Exits 0 if clean, 1 with one violation per line. Run after dream reconciliation or before commit. | No arguments defaults to `--summary`. @@ -51,8 +51,8 @@ No arguments defaults to `--summary`. | `--limit N` | Positive page size for search, symbol, labels, or preferences (default: 10) | | `--max-chars N` | Hard output cap, including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output). Use `--top-max-chars` with `--top`. | | `--cursor TOKEN` | Continue the same view and filters using the returned ordering snapshot | -| `--text-offset N` | Continue a long `--get` result at the returned character offset | -| `--event-id ID` | Optional retry ID: each discovery counts at most once per ID (1-128 letters/digits or `. _ : -`). Default: a new event per invocation. | +| `--text-cursor TOKEN` | Continue the same `--get` body and logical read; copy the returned token rather than fabricating an offset | +| `--event-id ID` | Optional retry ID: each discovery counts at most once per ID within 24 hours (1-128 letters/digits or `. _ : -`) | | `--no-record` | Do not increase citation scores; existing scores still rank results, and pagination may store a local snapshot | | `--top-labels LABELS` | Comma-separated label filter for `--top` (default: `bug,security`). Empty string disables label filtering. | | `--top-limit N` | Max discoveries to show in `--top` (default: 3) | @@ -83,9 +83,11 @@ python3 shadow-viewer.py --top src/auth.py --top-labels bug,security,performance Replace `DISCOVERY_ID` and `CURSOR_TOKEN` with the exact values returned by the helper. Content views (`search`, `symbol`, `get`, `prefs`, `labels`, `recent`, `top`) -show an `id` and one `citation_score`. Every discovery starts at zero by +return an `id`; all except compact `top` also show one `citation_score`. +Every discovery starts at zero by default, regardless of which workflow wrote it. The helper increments only -entries whose content it emits, once per retrieval event. Scanning/matching, +entries whose content it emits. Expanding a long claim counts as one logical +read across its continuation chunks. Scanning/matching, summary statistics, invariant audits, and raw file reads do not count. Displayed scores are the values **before** the current read. A citation here measures helper exposure, not proven usefulness, correctness, or LLM influence. @@ -99,8 +101,10 @@ Popularity never overrides a refuted status or authorizes dropping a constraint. Use targeted searches and additional pages for deduplication rather than assuming the popular shortlist is exhaustive. -The `d_...` ID is a content fingerprint of kind, file/symbol anchor, normalized -claim text, and related refs. Metadata-only status/source/label changes keep it; +The `d_...` ID binds kind, canonical file/symbol anchor, parsed claim text, and +related refs. Internal whitespace is preserved, including code literals. +Filesystem aliases resolve to the same on-disk name; container-heading prefixes +are removed from symbol anchors. Metadata-only status/source/label changes keep it; rewording, renaming, or merging claims/refs can create a new zero-score identity. Scores are not fuzzily transferred or summed during Meditate. Removed identities can remain in the local ledger but cannot be expanded unless their claim exists. @@ -113,21 +117,33 @@ roots in the same repo have separate scopes. Outside Git, the cache lives under not discovery bodies. It is local metadata, not a tracked or multi-machine DB; Markdown and its format remain authoritative and unchanged. -Transactions prevent lost increments from concurrent agents. Lock contention -retries the whole transaction within the original short wait budget; each failed -attempt rolls back. The ledger reuses its rollback journal with full disk -synchronization rather than repeatedly creating/deleting it. Reuse `--event-id` -when retrying one retrieval; do not reuse it for unrelated visits. A failed -stdout emission is not recorded. Ledger failures warn on stderr, return the -knowledge, and show unknown scores as `?` when scores cannot be read. -Do not report those failed increments as successful. - -Pagination snapshots freeze ordering despite score changes and expire after -24 hours. Keep the original view/filters with `--cursor`; a changed knowledge -snapshot requires restarting the query. For a large claim, `--get` prints the -next `--text-offset`. Character limits bound helper output, not token counts; +Successful transactions cannot overwrite concurrent increments. Accounting is +best-effort: a busy ledger can exhaust the 100 ms wait budget. Unknown scores +display as `?`, but counting is still attempted after output; failed increments +warn explicitly that the visit was not recorded. Retry a busy database later, +not by deleting it. Failed stdout emission is not recorded. + +Ordinary reads need no retained event receipt. Caller-supplied retry IDs and +generated long-read IDs retain receipts for 24 hours, up to 100,000 receipts. +New explicit receipts beyond capacity fail visibly rather than weakening retry +deduplication. Never reuse a retry ID for an unrelated visit. Expired receipts +are pruned in bounded batches. The fully synchronized rollback journal is capped +at 1 MiB; SQLite can retain reusable free pages in the database. + +Result snapshots freeze ordering despite score changes. Identical snapshots +reuse compressed storage; at most 32 snapshots / 8 MiB are retained locally. +They expire after 24 hours or capacity eviction. Keep the original view/filters +with `--cursor`; changed matching content/trust/labels require restarting. +Timestamp-only or unrelated-symbol edits do not invalidate non-recent views. + +For a long claim, copy the returned `--text-cursor` continuation. It binds the +exact expanded body and metadata, preserves `--no-record`, and reuses one read +event. Changed content or an expired token requires restarting `--get`. Character +limits bound helper output, not token counts; the helper can still scan the underlying Markdown locally. If the ledger is -unavailable, narrow the query until pagination can be restored. +unavailable, repair local-state access before relying on exhaustive pagination. +An incompatible prerelease cache must be moved aside to reset local scores; +never alter shadow content to repair telemetry. ## Dream Lineage Visualization diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog-viewer/_citations.py index a92f710..976e0cd 100644 --- a/skills/shadow-frog-viewer/_citations.py +++ b/skills/shadow-frog-viewer/_citations.py @@ -11,27 +11,67 @@ import sqlite3 import subprocess import time -import uuid +import unicodedata +import zlib PAGE_TTL = 24 * 60 * 60 +RECEIPT_TTL = PAGE_TTL +MAX_RECEIPTS = 100_000 +MAX_PAGES = 32 +MAX_PAGE_BYTES = 8 * 1024 * 1024 +JOURNAL_BYTES = 1024 * 1024 +SCHEMA_VERSION = 2 def discovery_id(kind, anchor, text, refs=()): """Content identity excludes status, provenance, labels, and usage metadata.""" - value = [kind, anchor, " ".join(text.split()), sorted(set(refs))] + value = [kind, anchor, text.strip(), sorted(set(refs))] digest = hashlib.sha256( json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8") ).hexdigest() return "d_" + digest[:32] +def canonical_path(path): + """Use actual directory-entry spelling without folding distinct filesystem names.""" + resolved = Path(path).resolve() + current = Path(resolved.anchor) + for part in resolved.parts[1:]: + requested = current / part + if requested.exists(): + key = unicodedata.normalize("NFC", part).casefold() + matches = [] + for child in current.iterdir(): + if child.name == part: + matches = [child] + break + if unicodedata.normalize("NFC", child.name).casefold() == key and child.samefile(requested): + matches.append(child) + if len(matches) != 1: + raise ValueError(f"Cannot identify a unique filesystem path for {requested}") + current = matches[0] + else: + current = requested + return current + + def validate_event_id(value): if not isinstance(value, str) or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", value): raise ValueError("event ID must be 1-128 letters, digits, or . _ : -") return value +def is_busy_error(exc): + if not isinstance(exc, sqlite3.OperationalError): + return False + code = getattr(exc, "sqlite_errorcode", None) + return ( + (code is not None and code & 0xff in (5, 6)) # SQLITE_BUSY / SQLITE_LOCKED + or (code is None and str(exc) in ("database is locked", "database table is locked")) + ) + + @dataclass(frozen=True) class CitationStore: """One short-lived connection per operation; SQLite coordinates processes.""" @@ -49,7 +89,7 @@ def __post_init__(self): @classmethod def for_shadow(cls, shadow_dir): - shadow = Path(shadow_dir).resolve() + shadow = canonical_path(shadow_dir) env = os.environ.copy() for name in ("GIT_DIR", "GIT_WORK_TREE", "GIT_COMMON_DIR", "GIT_INDEX_FILE"): env.pop(name, None) @@ -60,12 +100,12 @@ def for_shadow(cls, shadow_dir): ) if result.returncode == 0: root_text, common_text = result.stdout.rstrip("\n").split("\n") - root = Path(root_text).resolve() + root = canonical_path(root_text) common = Path(common_text) if not common.is_absolute(): common = shadow / common return cls( - common.resolve() / "shadowfrog/citations.sqlite3", + canonical_path(common) / "shadowfrog/citations.sqlite3", shadow.relative_to(root).as_posix(), ) if "not a git repository" not in result.stderr.lower(): @@ -82,14 +122,18 @@ def _connection(self, *, timeout=None): db = sqlite3.connect(self.path, timeout=self.timeout if timeout is None else timeout) try: version = db.execute("PRAGMA user_version").fetchone()[0] - if version not in (0, 1): - raise ValueError(f"Unsupported citation database version {version}; use a compatible helper") + if version not in (0, SCHEMA_VERSION): + raise ValueError( + f"Unsupported citation database version {version} at {self.path}; " + "use a matching helper or move this local cache aside to reset scores" + ) # Reuse the rollback journal instead of creating/deleting it on every # tiny update, while retaining FULL synchronization and atomic commits. mode = db.execute("PRAGMA journal_mode=PERSIST").fetchone()[0] if mode != "persist": raise ValueError(f"Cannot enable persistent citation journaling (got {mode})") db.execute("PRAGMA synchronous=FULL") + db.execute(f"PRAGMA journal_size_limit={JOURNAL_BYTES}") if version == 0: with db: db.execute("BEGIN IMMEDIATE") @@ -99,34 +143,33 @@ def _connection(self, *, timeout=None): ) db.execute( "CREATE TABLE IF NOT EXISTS events (scope TEXT, event TEXT, id TEXT, " - "PRIMARY KEY(scope, event, id))" + "created REAL NOT NULL, PRIMARY KEY(scope, event, id))" ) + db.execute("CREATE INDEX IF NOT EXISTS event_expiry ON events(created)") db.execute( "CREATE TABLE IF NOT EXISTS pages (token TEXT PRIMARY KEY, scope TEXT, " - "request TEXT, catalog TEXT, ids TEXT, created REAL)" + "request TEXT, catalog TEXT, ids BLOB, created REAL)" ) - db.execute("PRAGMA user_version = 1") + db.execute(f"PRAGMA user_version={SCHEMA_VERSION}") yield db finally: db.close() - def _write(self, action): - """Retry only lock contention, rolling back each attempt within one budget.""" + def _operation(self, action, *, write=False): + """Retry only lock contention; each write attempt is one atomic transaction.""" deadline = time.monotonic() + self.timeout while True: try: wait = min(0.01, max(0, deadline - time.monotonic())) - with self._connection(timeout=wait) as db, db: - db.execute("BEGIN IMMEDIATE") + with self._connection(timeout=wait) as db: + if write: + with db: + db.execute("BEGIN IMMEDIATE") + return action(db) return action(db) except sqlite3.OperationalError as exc: - code = getattr(exc, "sqlite_errorcode", None) - locked = ( - (code is not None and code & 0xff in (5, 6)) # SQLITE_BUSY / SQLITE_LOCKED - or (code is None and str(exc) in ("database is locked", "database table is locked")) - ) remaining = deadline - time.monotonic() - if not locked or remaining <= 0: + if not is_busy_error(exc) or remaining <= 0: raise # Avoid letting repeated writers monopolize SQLite's polling slots. time.sleep(min(remaining, random.uniform(0.001, 0.01))) @@ -134,9 +177,10 @@ def _write(self, action): def scores(self, ids): if not self.path.exists() or not ids: return {} - result = {} ids = list(dict.fromkeys(ids)) - with self._connection() as db: + + def read(db): + result = {} for offset in range(0, len(ids), 400): chunk = ids[offset:offset + 400] placeholders = ",".join("?" for _ in chunk) @@ -144,51 +188,94 @@ def scores(self, ids): f"SELECT id, citation_score FROM scores WHERE scope=? AND id IN ({placeholders})", [self.scope, *chunk], )) - return result + return result + return self._operation(read) - def record(self, ids, event): - """Increment each emitted identity once per event, including across retries.""" - validate_event_id(event) + def record(self, ids, event=None): + """Count anonymous reads directly; retain explicit retry receipts for one day.""" + if event is not None: + validate_event_id(event) ids = list(dict.fromkeys(ids)) if not ids: return + def update(db): + now = time.time() + cutoff = now - RECEIPT_TTL + db.execute( + "DELETE FROM events WHERE rowid IN " + "(SELECT rowid FROM events WHERE created < ? LIMIT 1000)", (cutoff,), + ) + receipt_count = db.execute("SELECT COUNT(*) FROM events").fetchone()[0] if event else 0 for identity in ids: - inserted = db.execute( - "INSERT OR IGNORE INTO events(scope, event, id) VALUES (?, ?, ?)", - (self.scope, event, identity), - ) - if inserted.rowcount: + if event is not None: + prior = db.execute( + "SELECT created FROM events WHERE scope=? AND event=? AND id=?", + (self.scope, event, identity), + ).fetchone() + if prior and prior[0] >= cutoff: + continue + if prior is None: + if receipt_count >= MAX_RECEIPTS: + raise ValueError("Citation retry receipt capacity reached; wait for expiry or use --no-record") + receipt_count += 1 db.execute( - "INSERT INTO scores(scope, id, citation_score) VALUES (?, ?, 1) " - "ON CONFLICT(scope, id) DO UPDATE SET citation_score=citation_score+1", - (self.scope, identity), + "INSERT INTO events VALUES (?, ?, ?, ?) " + "ON CONFLICT(scope, event, id) DO UPDATE SET created=excluded.created", + (self.scope, event, identity, now), ) - self._write(update) + db.execute( + "INSERT INTO scores(scope, id, citation_score) VALUES (?, ?, 1) " + "ON CONFLICT(scope, id) DO UPDATE SET citation_score=citation_score+1", + (self.scope, identity), + ) + self._operation(update, write=True) def save_page(self, request, catalog, ids): - token = uuid.uuid4().hex request = hashlib.sha256(request.encode("utf-8")).hexdigest() + payload = zlib.compress(json.dumps(ids, separators=(",", ":")).encode("utf-8")) + if len(payload) > MAX_PAGE_BYTES: + raise ValueError("Result snapshot exceeds local pagination capacity; narrow the query") + key = json.dumps([self.scope, request, catalog]).encode("utf-8") + payload + token = hashlib.sha256(key).hexdigest()[:32] + def publish(db): - db.execute("DELETE FROM pages WHERE created < ?", (time.time() - PAGE_TTL,)) + now = time.time() + db.execute("DELETE FROM pages WHERE created < ?", (now - PAGE_TTL,)) + db.execute("DELETE FROM pages WHERE token=?", (token,)) + rows = db.execute("SELECT token, length(ids) FROM pages ORDER BY created DESC, token").fetchall() + used = len(payload) + for index, (old_token, size) in enumerate(rows, 1): + used += size + if index >= MAX_PAGES or used > MAX_PAGE_BYTES: + db.execute("DELETE FROM pages WHERE token=?", (old_token,)) db.execute( "INSERT INTO pages VALUES (?, ?, ?, ?, ?, ?)", - (token, self.scope, request, catalog, json.dumps(ids), time.time()), + (token, self.scope, request, catalog, payload, now), ) - self._write(publish) + self._operation(publish, write=True) return token def load_page(self, token, request, catalog): request = hashlib.sha256(request.encode("utf-8")).hexdigest() - with self._connection() as db: - row = db.execute( + if not self.path.exists(): + raise ValueError("Retrieval cursor expired or unavailable; rerun the query without --cursor") + row = self._operation( + lambda db: db.execute( "SELECT request, catalog, ids, created FROM pages WHERE token=? AND scope=?", (token, self.scope), ).fetchone() + ) if not row or row[3] < time.time() - PAGE_TTL: raise ValueError("Retrieval cursor expired or unavailable; rerun the query without --cursor") if row[0] != request: raise ValueError("Cursor belongs to a different query; reuse its original view and filters") if row[1] != catalog: raise ValueError("Shadow knowledge changed; rerun the query without --cursor") - return json.loads(row[2]) + try: + ids = json.loads(zlib.decompress(row[2]).decode("utf-8")) + except (zlib.error, UnicodeError, ValueError, TypeError) as exc: + raise ValueError("Invalid local pagination snapshot; rerun the query without --cursor") from exc + if not isinstance(ids, list) or not all(isinstance(identity, str) for identity in ids): + raise ValueError("Invalid local pagination snapshot; rerun the query without --cursor") + return ids diff --git a/skills/shadow-frog-viewer/shadow-viewer.py b/skills/shadow-frog-viewer/shadow-viewer.py index f4508d1..7379131 100755 --- a/skills/shadow-frog-viewer/shadow-viewer.py +++ b/skills/shadow-frog-viewer/shadow-viewer.py @@ -20,6 +20,7 @@ --limit N Maximum results (default: 10) --max-chars N Output budget including metadata (default: 4000) --cursor TOKEN Continue a previous result snapshot + --text-cursor TOKEN Continue an unchanged discovery as one logical read --no-record Do not increment local citation scores --event-id ID Reuse an ID for retry-idempotent bookkeeping @@ -28,6 +29,8 @@ 1 Fatal error (shadow dir not found, no results possible) """ +from __future__ import annotations + import argparse from dataclasses import dataclass import hashlib @@ -37,6 +40,7 @@ import sys import sqlite3 import subprocess +import time import traceback import uuid from collections import defaultdict @@ -45,12 +49,18 @@ sys.path.insert(0, str(Path(__file__).resolve().parent)) +_bytecode = sys.dont_write_bytecode +sys.dont_write_bytecode = True try: - from _citations import CitationStore, discovery_id, validate_event_id + from _citations import ( + CitationStore, RECEIPT_TTL, canonical_path, discovery_id, is_busy_error, + validate_event_id, + ) except ImportError as exc: raise SystemExit("[shadow-viewer error] Missing citation helper/dependency; reinstall the complete Viewer skill") from exc finally: sys.path.pop(0) + sys.dont_write_bytecode = _bytecode _DISCOVERY_META_RE = re.compile( @@ -473,32 +483,55 @@ class RetrievalOptions: cursor: str | None = None event_id: str | None = None record: bool = True - text_offset: int = 0 + text_cursor: str | None = None def __post_init__(self): if type(self.limit) is not int or self.limit < 1: raise ValueError("Result limit must be positive") if type(self.max_chars) is not int or self.max_chars < 0 or (self.max_chars and self.max_chars < 256): raise ValueError("Output budget must be at least 256 characters, or 0 for no cap") - if type(self.text_offset) is not int or self.text_offset < 0: - raise ValueError("Text offset must be nonnegative") + for name in ("cursor", "text_cursor"): + value = getattr(self, name) + if value is not None and (not isinstance(value, str) or not value): + raise ValueError(f"{name} must be a nonempty returned cursor") if self.event_id is not None: validate_event_id(self.event_id) def _knowledge_entry(kind, file, symbol, text, data, refs=(), mtime=0): + symbol = _canonical_symbol(symbol) anchor = f"{file}::{symbol}" if symbol else file return { "id": discovery_id(kind, anchor, text, refs), "kind": kind, "file": file, "symbol": symbol, "anchor": anchor, "text": text, "refs": list(refs), "mtime": mtime, "status": data.get("status", "verified" if kind == "preference" else "?"), - "source": data.get("source", "?"), "labels": data.get("labels", []), + "source": data.get("source", "?"), "labels": sorted(set(data.get("labels", []))), "title": data.get("title", ""), "category": data.get("category", "?"), "dream_report": data.get("dream_report", ""), } +def _canonical_symbol(symbol): + # Container headings use the same prefixes as shadow-init.py::Symbol.heading_text. + if symbol in ("File-Level", "file-level"): + return "file-level" + return re.sub(r"^(?:class|interface|enum|trait|struct|protocol|module) ", "", symbol, count=1) + + +def _canonical_source_file(shadow_dir, source_file): + if ( + not source_file or any(char in source_file for char in (":", "\\", "\0", "\n", "\r")) + or any(part in ("", ".", "..") for part in source_file.split("/")) + ): + raise ValueError("Use a repository-relative source path with forward slashes") + root = canonical_path(shadow_dir) + target = canonical_path(root / (source_file + ".md")) + if not target.is_relative_to(root): + raise ValueError("Requested shadow resolves outside --shadow-dir") + return target.relative_to(root).as_posix()[:-3] + + def _mtime(path): try: return path.stat().st_mtime @@ -527,25 +560,43 @@ def _unique_entries(entries): warn(f"Empty discovery at {entry['anchor']}; repair its text before retrieval.") continue prior = unique.get(entry["id"]) - if prior is None or _trust(entry) < _trust(prior): + if prior is None: unique[entry["id"]] = entry + else: + strongest = entry if _trust(entry) < _trust(prior) else prior + unique[entry["id"]] = { + **strongest, "labels": sorted(set(entry["labels"]) | set(prior["labels"])), + } return list(unique.values()) def _knowledge_entries(shadow_dir, source_file=None): """Collect identities without recording a citation for parsing or matching.""" entries = [] + files = {} + + def canonical_file(file): + if file not in files: + files[file] = _canonical_source_file(shadow_dir, file) + return files[file] + + def canonical_refs(refs): + normalized = [] + for ref in refs: + file, separator, symbol = ref.partition("::") + if separator: + try: + ref = f"{canonical_file(file)}::{_canonical_symbol(symbol)}" + except (OSError, ValueError, RuntimeError) as exc: + warn(f"Cannot resolve reference {ref!r}: {exc}; inspect and repair that reference.") + normalized.append(ref) + return sorted(set(normalized)) + if source_file is None: discoveries = collect_all_discoveries(shadow_dir) else: - if ( - not source_file or ":" in source_file or "\\" in source_file - or any(part in ("", ".", "..") for part in source_file.split("/")) - ): - raise ValueError("Use a repository-relative source path with forward slashes") + source_file = canonical_file(source_file) path = shadow_dir / (source_file + ".md") - if not path.resolve().is_relative_to(shadow_dir.resolve()): - raise ValueError("Requested shadow resolves outside --shadow-dir") parsed = parse_shadow_file(path) if path.is_file() else {"discoveries": []} modified = _mtime(path) if parsed["discoveries"] else 0 discoveries = [ @@ -553,13 +604,13 @@ def _knowledge_entries(shadow_dir, source_file=None): for disc in parsed["discoveries"] ] for disc in discoveries: - file = Path(disc["shadow_path"]).as_posix()[:-3] + file = canonical_file(Path(disc["shadow_path"]).as_posix()[:-3]) entries.append(_knowledge_entry( "discovery", file, disc.get("symbol", "file-level"), disc.get("text", ""), - disc, disc.get("also_involves", []), disc.get("shadow_mtime", 0), + disc, canonical_refs(disc.get("also_involves", [])), disc.get("shadow_mtime", 0), )) for cross in parse_cross_cutting(shadow_dir): - refs = cross.get("refs", []) + refs = canonical_refs(cross.get("refs", [])) if source_file is not None and not any(ref.split("::", 1)[0] == source_file for ref in refs): continue relative = "_cross/" + cross["file"] @@ -597,11 +648,12 @@ def _rank_entries(entries, scores, limit, recent=False): key=lambda entry: -scores[entry["id"]], ) unseen = [entry for entry in group if not scores.get(entry["id"], 0)] - # Reserve one slot per group/page-sized block for a zero-score entry. - width = max(2, limit) + width = max(1, limit) while seen and unseen: - result.extend(seen[:width - 1]) - del seen[:width - 1] + remaining = width - len(result) % width + take = min(len(seen), remaining - 1) + result.extend(seen[:take]) + del seen[:take] result.append(unseen.pop(0)) result.extend(seen) result.extend(unseen) @@ -611,23 +663,27 @@ def _rank_entries(entries, scores, limit, recent=False): def _citation_store(shadow_dir): try: return CitationStore.for_shadow(shadow_dir) - except (OSError, ValueError, subprocess.SubprocessError) as exc: - warn(f"Citation tracking unavailable: {exc}. Knowledge is still returned; repair local Git/state access.") + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as exc: + warn(f"Citation tracking unavailable: {exc}. This visit will not be recorded; check local Git/state access.") return None def _clip(text, limit): - return text if len(text) <= limit else text[:max(0, limit - 3)] + "..." + if len(text) <= limit: + return text + return text[:limit] if limit < 3 else text[:limit - 3] + "..." -def _entry_preview(entry, score, style, group_count): - text = _clip(entry["text"].replace("\n", " "), 180) +def _preview_parts(entry, score, style, group_count): + text = entry["text"].replace("\n", " ") anchor = _clip(entry["anchor"], 160) identity = f"id={entry['id']} citation_score={score}" metadata = f"({entry['status']}, source: {entry['source']})" labels = ",".join(entry["labels"]) or "-" if style == "top": - return f"- [{labels}] `{anchor}` ({entry['status']}) {identity}: {text}" + anchor = entry["symbol"] if entry["kind"] == "discovery" else entry["file"] + prefix = f"- [{_clip(labels, 40)}] `{_clip(anchor, 48)}` ({entry['status']}) id={entry['id']}: " + return prefix, text if style == "recent": stamp = datetime.fromtimestamp(entry["mtime"]).strftime("%Y-%m-%d %H:%M") heading = f" [{stamp}] ({entry['kind']})" @@ -637,28 +693,51 @@ def _entry_preview(entry, score, style, group_count): heading = f"Preferences [{entry['source']}]" else: heading = f"{_clip(entry['file'], 160)} ({group_count} matches)" - body = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n {text}" + prefix = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n" if entry["refs"]: label = "Refs" if entry["kind"] == "cross-cutting" else "Also involves" - body += f"\n {label}: {_clip(', '.join(entry['refs']), 160)}" + prefix += f" {label}: {_clip(', '.join(entry['refs']), 160)}\n" if style == "labels" and len(entry["labels"]) > 1: - body += f"\n Also labeled: {labels}" - return body + prefix += f" Also labeled: {labels}\n" + return prefix + " ", text -def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): - store = _citation_store(shadow_dir) - scores = {} - if store: +def _read_scores(store, identities): + if store is not None: + try: + return store.scores(identities), True + except (OSError, ValueError, sqlite3.Error) as exc: + if is_busy_error(exc): + warn("Citation ledger busy; scores are unknown. Counting will still be attempted after output if enabled.") + else: + warn(f"Cannot read citation scores: {exc}. Scores are unknown; check the local cache.") + return {}, False + + +def _record_visible(store, identities, options, event=None): + if store is not None and identities and options.record: try: - scores = store.scores([entry["id"] for entry in entries]) + store.record(identities, options.event_id if event is None else event) except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Cannot read citation scores: {exc}. Scores are unknown; repair the local ledger and retry.") - store = None + if is_busy_error(exc): + warn("Citation ledger busy; this visit was not recorded (scores were not updated). Retry later with the same --event-id if supplied.") + else: + warn(f"Citation scores were not updated: {exc}. This visit was not recorded; repair local state and retry.") + + +def _catalog_digest(entries, recent): + values = [ + {key: value for key, value in entry.items() if recent or key != "mtime"} + for entry in sorted(entries, key=lambda entry: entry["id"]) + ] + return hashlib.sha256(json.dumps(values, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest() + + +def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): + store = _citation_store(shadow_dir) + scores, scores_known = _read_scores(store, [entry["id"] for entry in entries]) ranked = _rank_entries(entries, scores, options.limit, recent=style == "recent") - catalog = "" if style == "top" else hashlib.sha256(json.dumps( - sorted(entries, key=lambda entry: entry["id"]), sort_keys=True, ensure_ascii=False, - ).encode("utf-8")).hexdigest() + catalog = "" if style == "top" else _catalog_digest(entries, recent=style == "recent") token = None offset = 0 if options.cursor: @@ -670,7 +749,10 @@ def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", req token, offset = match.group(1), int(match.group(2)) ids = store.load_page(token, request, catalog) by_id = {entry["id"]: entry for entry in entries} - ranked = [by_id[identity] for identity in ids] + try: + ranked = [by_id[identity] for identity in ids] + except KeyError as exc: + raise ValueError("Invalid local pagination snapshot; restart the query") from exc if offset >= len(ranked): raise ValueError("Cursor is past the available results; restart the query") @@ -681,39 +763,54 @@ def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", req cap = options.max_chars prefix = _clip(header, min(300, cap // 5)) if cap else header reserve = 90 if style != "top" else 6 - if cap and not options.cursor: - width = options.limit - while width > 1: + width = min(options.limit, len(ranked) - offset) + while width: + selected = ranked[offset:offset + width] + parts = [ + _preview_parts(entry, scores.get(entry["id"], 0) if scores_known else "?", style, counts[entry["file"]]) + for entry in selected + ] + if cap: used = len(prefix) + reserve + 3 fits = 0 - for entry in ranked[:width]: - score = scores.get(entry["id"], 0) if store else "?" - used += len(_entry_preview(entry, score, style, counts[entry["file"]])) + 1 + for entry_prefix, text in parts: + used += len(entry_prefix) + min(12, len(text)) + 1 if used > cap: break fits += 1 - if fits >= width: - break - width = max(1, fits) - ranked = _rank_entries(entries, scores, width, recent=style == "recent") - output = prefix - shown = [] - for entry in ranked[offset:offset + options.limit]: - score = scores.get(entry["id"], 0) if store else "?" - block = _entry_preview(entry, score, style, counts[entry["file"]]) - available = cap - len(output) - reserve - 3 if cap else len(block) - if cap and len(block) > available: - if shown: - break - identity = ( - f"({entry['status']}, source: {entry['source']}) " - f"id={entry['id']} citation_score={score}: " - ) - if available <= len(identity) + 4: - raise ValueError("Output budget cannot fit an entry; increase --max-chars") - block = identity + _clip(entry["text"], available - len(identity)) - output += "\n" + block - shown.append(entry["id"]) + if fits < width: + if width > 1: + width = max(1, fits) + if not options.cursor: + ranked = _rank_entries(entries, scores, width, recent=style == "recent") + continue + entry = selected[0] + score = scores.get(entry["id"], 0) if scores_known else "?" + minimal = f"({entry['status']}, source: {entry['source']}) id={entry['id']} citation_score={score}: " + parts = [(minimal, entry["text"])] + break + body_budget = cap - len(prefix) - reserve - 1 - width - sum(len(part[0]) for part in parts) if cap else 180 * width + if width and body_budget < width: + raise ValueError("Output budget cannot fit discovery content; increase the character budget") + snippets = [0] * width + # Distribute spare room across entries rather than dropping a whole warning. + pending = list(range(width)) + while pending and body_budget: + share = max(1, body_budget // len(pending)) + next_pending = [] + for index in pending: + remaining = min(180, len(parts[index][1])) - snippets[index] + take = min(remaining, share, body_budget) + snippets[index] += take + body_budget -= take + if snippets[index] < min(180, len(parts[index][1])): + next_pending.append(index) + pending = next_pending + output = prefix + "".join( + "\n" + entry_prefix + _clip(text, snippets[index]) + for index, (entry_prefix, text) in enumerate(parts) + ) + shown = [entry["id"] for entry in selected] next_offset = offset + len(shown) if next_offset < len(ranked): if style == "top": @@ -734,11 +831,7 @@ def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", req if cap and len(output) + 1 > cap: raise ValueError("Output budget cannot fit retrieval metadata; increase --max-chars") print(output, flush=True) - if store and shown and options.record: - try: - store.record(shown, options.event_id or uuid.uuid4().hex) - except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Citation scores were not updated: {exc}. Reuse --event-id when retrying.") + _record_visible(store, shown, options) def view_get(shadow_dir, identity, *, options=None): @@ -750,13 +843,8 @@ def view_get(shadow_dir, identity, *, options=None): if entry is None: raise ValueError("Discovery ID is absent or its claim changed; search again for its current ID") store = _citation_store(shadow_dir) - score = "?" - if store: - try: - score = store.scores([identity]).get(identity, 0) - except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Cannot read citation score: {exc}. Repair the local ledger and retry.") - store = None + scores, scores_known = _read_scores(store, [identity]) + score = scores.get(identity, 0) if scores_known else "?" body = entry["anchor"] + "\n\n" + entry["text"] if entry["labels"]: body += "\nLabels: " + ", ".join(entry["labels"]) @@ -766,26 +854,55 @@ def view_get(shadow_dir, identity, *, options=None): body += "\nRefs: " + ", ".join(entry["refs"]) if entry["dream_report"]: body += "\nDream report: " + entry["dream_report"] - if options.text_offset >= len(body) and options.text_offset: - raise ValueError("--text-offset is past the end of this discovery") + revision = hashlib.sha256(json.dumps( + [entry["status"], entry["source"], body], ensure_ascii=False, + ).encode("utf-8")).hexdigest()[:32] + start = 0 + event = options.event_id + expires = int(time.time()) + RECEIPT_TTL + record = options.record + if options.text_cursor: + match = re.fullmatch( + r"([0-9a-f]{32})~(\d{1,12})~([01])~([A-Za-z0-9._:-]{1,128})~(\d+)", + options.text_cursor, + ) + if not match: + raise ValueError("Invalid text cursor; copy the --text-cursor value from the previous expansion") + prior_revision, expiry, recording, prior_event, offset = match.groups() + if prior_revision != revision: + raise ValueError("Discovery body or metadata changed; restart --get without --text-cursor") + if int(expiry) <= time.time(): + raise ValueError("Text cursor expired; restart --get without --text-cursor") + if event is not None and event != prior_event: + raise ValueError("Text cursor has a different --event-id; reuse its original event") + start, expires, event = int(offset), int(expiry), prior_event + record = record and recording == "1" + if start >= len(body): + raise ValueError("Text cursor is past the end of this discovery; restart --get") header = ( f"id={identity} citation_score={score}\n" f"({entry['status']}, source: {entry['source']})\n" ) - start = options.text_offset - available = options.max_chars - len(header) - 100 if options.max_chars else len(body) - if available < 1: - raise ValueError("Output budget cannot fit discovery content; increase --max-chars") - content = body[start:start + available] - output = header + content - if start + len(content) < len(body): - output += f"\nContinue: --get {identity} --text-offset {start + len(content)}" + tail = "" + available = len(body) - start + if options.max_chars and len(header) + available + 1 > options.max_chars: + event = event or uuid.uuid4().hex + + def continuation(offset): + token = f"{revision}~{expires}~{int(record)}~{event}~{offset}" + value = f"\nContinue: --get {identity} --text-cursor {token}" + if options.max_chars != 4000: + value += f" --max-chars {options.max_chars}" + return value + + available = options.max_chars - len(header) - len(continuation(len(body))) - 1 + if available < 1: + raise ValueError("Output budget cannot fit continuation metadata; increase --max-chars") + tail = continuation(start + available) + output = header + body[start:start + available] + tail print(output, flush=True) - if store and options.record: - try: - store.record([identity], options.event_id or uuid.uuid4().hex) - except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Citation score was not updated: {exc}. Reuse --event-id when retrying.") + if record: + _record_visible(store, [identity], options, event) def view_summary(shadow_dir): @@ -947,7 +1064,8 @@ def view_symbol(shadow_dir, anchor, *, options=None): file, separator, symbol = anchor.partition("::") if not separator or not symbol: raise ValueError("--symbol requires file::symbol (use File-Level for a file-level section)") - symbol = "file-level" if symbol == "File-Level" else symbol + file = _canonical_source_file(shadow_dir, file) + symbol = _canonical_symbol(symbol) canonical = f"{file}::{symbol}" matches = [ entry for entry in _knowledge_entries(shadow_dir, file) @@ -1369,7 +1487,7 @@ def main(): parser.add_argument("--cursor", help="Continue the same view/filters with a frozen result ordering") parser.add_argument("--event-id", help="Retry token; an entry counts once per token (default: fresh event)") parser.add_argument("--no-record", action="store_true", help="Do not increment citation scores") - parser.add_argument("--text-offset", type=int, help="Continue long --get output at this character offset") + parser.add_argument("--text-cursor", help="Continue the same logical read of an unchanged --get body") args = parser.parse_args() retrieval = ( @@ -1378,13 +1496,13 @@ def main(): ) if not retrieval and ( args.limit is not None or args.max_chars is not None or args.cursor - or args.event_id is not None or args.no_record or args.text_offset is not None + or args.event_id is not None or args.no_record or args.text_cursor is not None ): parser.error("Retrieval options require --search, --symbol, --get, --top, --prefs, --labels, or --recent") - if args.text_offset is not None and args.get is None: - parser.error("--text-offset requires --get") + if args.text_cursor is not None and args.get is None: + parser.error("--text-cursor requires --get") if args.cursor and (args.get is not None or args.top is not None): - parser.error("--cursor is for paged search/symbol/labels/prefs/recent; use --text-offset with --get") + parser.error("--cursor is for paged search/symbol/labels/prefs/recent; use --text-cursor with --get") if args.limit is not None and (args.top is not None or args.recent is not None or args.get is not None): parser.error("Use --top-limit or --recent N instead of --limit; --get returns one discovery") if args.max_chars is not None and args.top is not None: @@ -1398,7 +1516,7 @@ def main(): args.max_chars if args.max_chars is not None else 4000 ), cursor=args.cursor, event_id=args.event_id, record=not args.no_record, - text_offset=args.text_offset or 0, + text_cursor=args.text_cursor, ) except ValueError as exc: parser.error(str(exc)) @@ -1458,7 +1576,10 @@ def main(): error("Interrupted by user.") sys.exit(130) except (ValueError, sqlite3.Error) as exc: - error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") + if is_busy_error(exc): + error("Local citation ledger is busy; retry the same command later. Do not reset a busy database.") + else: + error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") sys.exit(1) except Exception as e: error( diff --git a/skills/shadow-frog/SKILL.md b/skills/shadow-frog/SKILL.md index c531d64..96f5dc4 100644 --- a/skills/shadow-frog/SKILL.md +++ b/skills/shadow-frog/SKILL.md @@ -32,8 +32,9 @@ to that code location. especially when investigating bugs or unfamiliar code. They may contain findings not yet distilled into per-file shadows. 4. **Before editing a file**, query its shadow (`.shadow/.md`) with - `--symbol file::symbol` or `--top FILE`, and search relevant `_cross/` - knowledge. Expand matching entries and follow pages as needed, then apply them. + `--search FILE`; use `--symbol file::symbol` for focused follow-up, including + file-level and related `_cross/` knowledge. Expand entries and follow pages + as needed. `--top` supplies compact hints, not a complete file review. `_index.md` counts may be stale; inspect the actual shadows and `_cross/`. 5. **When the user explains something about code** (gotcha, design intent, warning, history): write a `source: user` discovery to the shadow @@ -47,7 +48,8 @@ to that code location. Prefer the `/shadow-frog-viewer` helper over loading an entire large shadow. It returns short previews, discovery IDs, and one `citation_score`; `--get ID` -expands an entry, and returned cursors continue a stable result ordering. +expands an entry, and returned cursors continue a stable result ordering or +revision-bound logical read. Compact `--top` output omits the numeric score. Citation scores count content emitted by the helper, not proven use in reasoning. Do not manually increment scores or put them into Markdown: the helper records visits atomically in local state shared across worktrees. New discoveries from diff --git a/tests/hooks/test_pre_tool_sh.py b/tests/hooks/test_pre_tool_sh.py index aee37c6..f029087 100644 --- a/tests/hooks/test_pre_tool_sh.py +++ b/tests/hooks/test_pre_tool_sh.py @@ -168,7 +168,7 @@ def test_citation_failure_is_visible_but_never_denies_edit(self, coupon_demo, tm assert result.returncode == 0 context = json.loads(result.stdout)["additionalContext"] assert "Actionable discoveries" in context - assert "citation_score=?" in context + assert "id=d_" in context assert "Viewer reported a warning" in context assert ledger.read_bytes() == b"invalid sqlite" diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog_viewer/test_citations.py index d30b7a5..4da6df7 100644 --- a/tests/skills/shadow_frog_viewer/test_citations.py +++ b/tests/skills/shadow_frog_viewer/test_citations.py @@ -1,6 +1,8 @@ """Real SQLite and Git tests for local, multiprocess citation bookkeeping.""" from pathlib import Path +import hashlib +import json import sqlite3 import subprocess import sys @@ -20,7 +22,7 @@ def citations(): def test_identity_is_stable_without_mutable_metadata(citations): - identity = citations.discovery_id("file", "src/a.py::run", "A claim\nwith spacing.") + identity = citations.discovery_id("file", "src/a.py::run", " A claim with spacing. ") assert identity == citations.discovery_id("file", "src/a.py::run", "A claim with spacing.") assert identity != citations.discovery_id("file", "src/a.py::run", "A different claim.") assert identity != citations.discovery_id("file", "src/a.py::other", "A claim with spacing.") @@ -30,6 +32,13 @@ def test_identity_is_stable_without_mutable_metadata(citations): ) +@pytest.mark.parametrize("text", ['Key `a b` is accepted.', 'Key "a b" is accepted.', "Indented:\n value"]) +def test_identity_preserves_literal_whitespace(citations, text): + assert citations.discovery_id("file", "a::f", text) != citations.discovery_id( + "file", "a::f", " ".join(text.split()), + ) + + def test_zero_default_atomic_increment_and_event_retry(citations, tmp_path): store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") first = citations.discovery_id("file", "a::run", "One") @@ -107,7 +116,7 @@ def test_busy_commit_retries_whole_transaction_without_duplicate_updates(citatio def update(db): db.execute("UPDATE scores SET citation_score = citation_score + 1") print("attempt", flush=True) -store._write(update) +store._operation(update, write=True) """ process = subprocess.Popen( [sys.executable, "-c", code, str(HELPER.parent), str(path)], @@ -137,7 +146,7 @@ def invalid_sql(db): db.execute("INSERT INTO nonexistent_table VALUES (1)") with pytest.raises(sqlite3.OperationalError, match="no such table"): - store._write(invalid_sql) + store._operation(invalid_sql, write=True) assert len(attempts) == 1 @@ -251,3 +260,118 @@ def test_invalid_event_id_fails_early(citations, tmp_path, event): with pytest.raises(ValueError, match="event"): store.record(["d_" + "a" * 32], event) assert not store.path.exists() + + +def test_anonymous_reads_do_not_leave_retry_receipts(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + for _ in range(30): + store.record(["a", "b"]) + assert store.scores(["a", "b"]) == {"a": 30, "b": 30} + with store._connection() as db: + assert db.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 + + +def test_explicit_receipts_expire_without_deleting_scores(citations, tmp_path): + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + store.record(["a"], "event") + with store._connection() as db, db: + db.execute("UPDATE events SET created=0") + store.record(["b"]) + assert store.scores(["a", "b"]) == {"a": 1, "b": 1} + with store._connection() as db: + assert db.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 + store.record(["a"], "event") + assert store.scores(["a"]) == {"a": 2} + + +def test_receipt_capacity_rejects_new_events_atomically(citations, tmp_path, monkeypatch): + monkeypatch.setattr(citations, "MAX_RECEIPTS", 2) + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + with pytest.raises(ValueError, match="capacity"): + store.record(["a", "b", "c"], "too-many") + assert store.scores(["a", "b", "c"]) == {} + store.record(["a", "b"], "fits") + store.record(["a", "b"], "fits") + assert store.scores(["a", "b"]) == {"a": 1, "b": 1} + store.record(["c"]) + assert store.scores(["c"]) == {"c": 1} + + +def test_identical_snapshots_reuse_storage_and_cache_size_is_bounded(citations, tmp_path, monkeypatch): + monkeypatch.setattr(citations, "MAX_PAGES", 3) + monkeypatch.setattr(citations, "MAX_PAGE_BYTES", 4096) + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + ids = [hashlib.sha256(str(index).encode()).hexdigest() for index in range(50)] + token = store.save_page("same", "catalog", ids) + assert store.save_page("same", "catalog", ids) == token + with store._connection() as db: + assert db.execute("SELECT COUNT(*) FROM pages").fetchone()[0] == 1 + assert db.execute("SELECT length(ids) FROM pages").fetchone()[0] < len(json.dumps(ids)) + for index in range(8): + latest = store.save_page(f"query-{index}", "catalog", ids) + with store._connection() as db: + count, size = db.execute("SELECT COUNT(*), SUM(length(ids)) FROM pages").fetchone() + assert count <= 3 and size <= 4096 + assert store.load_page(latest, "query-7", "catalog") == ids + with pytest.raises(ValueError, match="expired|unavailable"): + store.load_page(token, "same", "catalog") + + +def test_journal_size_remains_capped_after_large_snapshot_cleanup(citations, tmp_path, monkeypatch): + monkeypatch.setattr(citations, "JOURNAL_BYTES", 4096) + store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") + ids = [hashlib.sha256(str(index).encode()).hexdigest() for index in range(1000)] + store.save_page("large", "catalog", ids) + with store._connection() as db, db: + db.execute("UPDATE pages SET created=0") + store.save_page("small", "catalog", ["one"]) + assert (tmp_path / "citations.sqlite3-journal").stat().st_size <= 4096 + + +def test_production_budget_preserves_all_successful_writes(citations, tmp_path): + path = tmp_path / "citations.sqlite3" + code = """ +import json, sqlite3, sys +from pathlib import Path +sys.path.insert(0, sys.argv[1]) +from _citations import CitationStore, is_busy_error +store = CitationStore(Path(sys.argv[2]), ".shadow") +assert store.timeout == 0.1 +ok = busy = 0 +for index in range(30): + try: + store.record(["shared"]) + except sqlite3.OperationalError as exc: + if not is_busy_error(exc): + raise + busy += 1 + else: + ok += 1 +print(json.dumps({"ok": ok, "busy": busy})) +""" + workers = [ + subprocess.Popen( + [sys.executable, "-c", code, str(HELPER.parent), str(path)], + stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", + ) + for _ in range(6) + ] + outcomes = [] + try: + for worker in workers: + stdout, stderr = worker.communicate(timeout=40) + assert worker.returncode == 0, stderr + outcomes.append(json.loads(stdout)) + finally: + for worker in workers: + if worker.poll() is None: + worker.terminate() + worker.communicate(timeout=10) + successful = sum(item["ok"] for item in outcomes) + assert successful > 0 + assert successful + sum(item["busy"] for item in outcomes) == 180 + assert store_score(citations, path) == successful + + +def store_score(citations, path): + return citations.CitationStore(path, ".shadow").scores(["shared"]).get("shared", 0) diff --git a/tests/skills/shadow_frog_viewer/test_retrieval.py b/tests/skills/shadow_frog_viewer/test_retrieval.py index d648538..d1bbd22 100644 --- a/tests/skills/shadow_frog_viewer/test_retrieval.py +++ b/tests/skills/shadow_frog_viewer/test_retrieval.py @@ -1,11 +1,15 @@ """Citation ranking, bounded context, expansion, and stable pagination.""" from dataclasses import replace +import io +import os import re +import shutil import sqlite3 import subprocess import sys import time +import unicodedata import pytest @@ -34,6 +38,11 @@ def cursor(text): return match.group(1) if match else None +def text_cursor(text): + match = re.search(r"--text-cursor (\S+)", text) + return match.group(1) if match else None + + def test_only_emitted_entries_count_and_no_markdown_changes(shadow_viewer, tmp_path, capsys): shadow = make_shadow(tmp_path, 12) before = {path: path.read_bytes() for path in shadow.rglob("*") if path.is_file()} @@ -138,7 +147,7 @@ def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_viewer, tm scores = {"popular": 100, "popular2": 90, "popular3": 80, "refuted": 10000} ranked = shadow_viewer._rank_entries(entries, scores, 3) assert ranked[0]["id"] == "user" and ranked[-1]["id"] == "refuted" - assert [entry["id"] for entry in ranked][1:4] == ["popular", "popular2", "new"] + assert [entry["id"] for entry in ranked][:3] == ["user", "popular", "new"] def test_zero_score_opportunity_accounts_for_character_budget(shadow_viewer, tmp_path, capsys): @@ -155,6 +164,37 @@ def test_zero_score_opportunity_accounts_for_character_budget(shadow_viewer, tmp assert entries[-1]["id"] in ids(output) +@pytest.mark.parametrize("view", ["top", "symbol"]) +def test_zero_score_slot_accounts_for_higher_trust_rows(shadow_viewer, tmp_path, capsys, view): + shadow = make_shadow(tmp_path, 5, text_size=2) + path = shadow / "source.py.md" + path.write_text( + path.read_text(encoding="utf-8").replace("source: exploration", "source: user", 1), + encoding="utf-8", + ) + entries = shadow_viewer._knowledge_entries(shadow) + store = shadow_viewer.CitationStore.for_shadow(shadow) + store.record([entry["id"] for entry in entries[1:4]], "prior") + options = shadow_viewer.RetrievalOptions(limit=3, max_chars=600) + if view == "top": + shadow_viewer.view_top(shadow, "source.py", "bug", 3, 600, options=options) + else: + shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + output = capsys.readouterr().out + assert len(output) <= 600 + assert ids(output)[0] == entries[0]["id"] + assert entries[-1]["id"] in ids(output) + + +@pytest.mark.parametrize("file", ["cart.py", "inventory.py", "test_cart.py"]) +def test_default_hook_budget_returns_multiple_warnings(shadow_viewer, coupon_demo, capsys, file): + shadow_viewer.view_top(coupon_demo / ".shadow", file, "bug,security", 3, 600) + output = capsys.readouterr().out + assert len(output) <= 600 + assert len(ids(output)) == 3 + assert "citation_score=" not in output + + def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_viewer, tmp_path): shadow = make_shadow(tmp_path, 1) original = shadow_viewer._knowledge_entries(shadow)[0]["id"] @@ -184,28 +224,222 @@ def test_duplicate_preferences_share_one_identity_without_losing_trust(shadow_vi assert shadow_viewer.CitationStore.for_shadow(shadow).scores(ids(output)) == dict.fromkeys(ids(output), 1) +@pytest.mark.parametrize("kind", ["class", "interface", "enum", "trait", "struct", "protocol", "module"]) +def test_container_symbols_use_canonical_anchors(shadow_viewer, shadow_init, tmp_path, capsys, kind): + shadow = make_shadow(tmp_path, 1) + heading = shadow_init.Symbol("Container", kind).heading_text + path = shadow / "source.py.md" + path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{heading}`"), encoding="utf-8") + shadow_viewer.view_symbol(shadow, "source.py::Container") + result = capsys.readouterr().out + assert "Claim 00000" in result and "source.py::Container" in result + shadow_viewer.view_get(shadow, ids(result)[0]) + assert "source.py::Container" in capsys.readouterr().out + + +@pytest.mark.parametrize("query", ["src/auth.py", unicodedata.normalize("NFD", "Src/caf\u00e9.py")]) +def test_filesystem_alias_ids_expand_and_include_cross_refs(shadow_viewer, tmp_path, capsys, query): + actual = "Src/Auth.py" if query == "src/auth.py" else "Src/caf\u00e9.py" + shadow = tmp_path / ".shadow" + path = shadow / f"{actual}.md" + path.parent.mkdir(parents=True) + path.write_text( + f"# Shadow: {actual}\n\n## `run`\n\n- Keep the audit trail.\n" + " _(verified, source: exploration, labels: [bug])_\n", + encoding="utf-8", + ) + if not (shadow / (query + ".md")).is_file(): + pytest.skip("Filesystem distinguishes these case/Unicode spellings") + cross = shadow / "_cross/audit.md" + cross.parent.mkdir() + cross.write_text( + f"# Audit\n\n**Category**: contract\n**Refs**:\n- `{actual}::run`\n\n" + "**Discovery**: Cross-file audit contract.\n\n_(verified, source: exploration)_\n", + encoding="utf-8", + ) + global_entries = shadow_viewer._knowledge_entries(shadow) + shadow_viewer.view_symbol(shadow, f"{query}::run") + result = capsys.readouterr().out + assert "Cross-file audit contract." in result and "Keep the audit trail." in result + assert set(ids(result)) == {entry["id"] for entry in global_entries} + for identity in ids(result): + shadow_viewer.view_get(shadow, identity, options=shadow_viewer.RetrievalOptions(record=False)) + assert identity in capsys.readouterr().out + + +def test_distinct_case_sensitive_files_are_not_folded(shadow_viewer, tmp_path): + shadow = tmp_path / ".shadow" + shadow.mkdir() + upper, lower = shadow / "A.py.md", shadow / "a.py.md" + upper.write_text("# Shadow: A.py\n\n## `run`\n\n- Claim.\n", encoding="utf-8") + if lower.exists(): + pytest.skip("Filesystem does not support distinct case-only names") + lower.write_text("# Shadow: a.py\n\n## `run`\n\n- Claim.\n", encoding="utf-8") + assert len({entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)}) == 2 + + +def test_literal_whitespace_remains_separately_searchable(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 0) + path = shadow / "source.py.md" + path.write_text( + "# Shadow: source.py\n\n## `run`\n\n" + "- Key `a b` is accepted.\n _(verified, source: exploration)_\n\n" + "- Key `a b` is accepted.\n _(verified, source: exploration)_\n", + encoding="utf-8", + ) + entries = shadow_viewer._knowledge_entries(shadow) + assert len(entries) == 2 and entries[0]["id"] != entries[1]["id"] + shadow_viewer.view_search(shadow, "a b") + result = capsys.readouterr().out + assert ids(result) == [entries[1]["id"]] + shadow_viewer.view_get(shadow, entries[1]["id"]) + assert "`a b`" in capsys.readouterr().out + + +def test_duplicate_claims_union_labels_and_preserve_stronger_source(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 0) + (shadow / "source.py.md").write_text( + "# Shadow: source.py\n\n## `run`\n\n" + "- Shared claim.\n _(verified, source: user, labels: [bug])_\n\n" + "- Shared claim.\n _(verified, source: exploration, labels: [security])_\n", + encoding="utf-8", + ) + entries = shadow_viewer._knowledge_entries(shadow) + assert len(entries) == 1 + assert entries[0]["labels"] == ["bug", "security"] and entries[0]["source"] == "user" + shadow_viewer.view_labels(shadow, "security") + assert ids(capsys.readouterr().out) == [entries[0]["id"]] + + +def test_installed_import_does_not_create_bytecode(repo_root, tmp_path): + installed = tmp_path / ".github/skills/shadow-frog-viewer" + shutil.copytree( + repo_root / "skills/shadow-frog-viewer", installed, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + env = os.environ.copy() + env.pop("PYTHONDONTWRITEBYTECODE", None) + env.pop("PYTHONPYCACHEPREFIX", None) + result = subprocess.run( + [sys.executable, str(installed / "shadow-viewer.py"), "--help"], + env=env, capture_output=True, text=True, encoding="utf-8", + ) + assert result.returncode == 0, result.stderr + assert not list(installed.rglob("*.pyc")) + + +def test_missing_home_does_not_hide_standalone_knowledge(shadow_viewer, tmp_path, monkeypatch, capsys): + from pathlib import Path + + shadow = make_shadow(tmp_path, 1) + monkeypatch.delenv("XDG_STATE_HOME", raising=False) + monkeypatch.delenv("LOCALAPPDATA", raising=False) + + def missing_home(): + raise RuntimeError("Could not determine home directory") + + monkeypatch.setattr(Path, "home", missing_home) + shadow_viewer.view_search(shadow, "Claim") + result = capsys.readouterr() + assert "Claim 00000" in result.out + assert "will not be recorded" in result.err and "home" in result.err + + def test_get_chunks_long_claim_without_exceeding_budget(shadow_viewer, tmp_path, capsys): shadow = make_shadow(tmp_path, 1, text_size=500) entry = shadow_viewer._knowledge_entries(shadow)[0] identity = entry["id"] - offset = 0 + continuation = None pieces = [] while True: shadow_viewer.view_get( shadow, identity, - options=shadow_viewer.RetrievalOptions(max_chars=500, text_offset=offset, event_id="read-one"), + options=shadow_viewer.RetrievalOptions(max_chars=500, text_cursor=continuation), ) output = capsys.readouterr().out assert len(output) <= 500 - match = re.search(r"--text-offset (\d+)", output) + continuation = text_cursor(output) pieces.append(output.removesuffix("\n").split("\n", 2)[2].split("\nContinue:", 1)[0]) - if match is None: + if continuation is None: break - offset = int(match.group(1)) assert "".join(pieces) == entry["anchor"] + "\n\n" + entry["text"] + "\nLabels: bug" assert shadow_viewer.CitationStore.for_shadow(shadow).scores([identity]) == {identity: 1} +@pytest.mark.parametrize("change", ["labels", "source", "status", "text"]) +def test_expansion_rejects_changed_body_or_metadata(shadow_viewer, tmp_path, capsys, change): + shadow = make_shadow(tmp_path, 1, text_size=200) + entry = shadow_viewer._knowledge_entries(shadow)[0] + shadow_viewer.view_get(shadow, entry["id"], options=shadow_viewer.RetrievalOptions(max_chars=500)) + continuation = text_cursor(capsys.readouterr().out) + path = shadow / "source.py.md" + replacements = { + "labels": ("labels: [bug]", "labels: [bug, security]"), + "source": ("source: exploration", "source: user"), + "status": ("verified,", "refuted,"), + "text": ("details ", "new details "), + } + path.write_text(path.read_text(encoding="utf-8").replace(*replacements[change]), encoding="utf-8") + with pytest.raises(ValueError, match="changed"): + shadow_viewer.view_get( + shadow, entry["id"], options=shadow_viewer.RetrievalOptions(text_cursor=continuation), + ) + + +def test_no_record_continuation_stays_unrecorded(shadow_viewer, tmp_path, capsys): + shadow = make_shadow(tmp_path, 1, text_size=200) + entry = shadow_viewer._knowledge_entries(shadow)[0] + shadow_viewer.view_get( + shadow, entry["id"], options=shadow_viewer.RetrievalOptions(max_chars=500, record=False), + ) + continuation = text_cursor(capsys.readouterr().out) + shadow_viewer.view_get( + shadow, entry["id"], options=shadow_viewer.RetrievalOptions(text_cursor=continuation), + ) + capsys.readouterr() + assert shadow_viewer.CitationStore.for_shadow(shadow).scores([entry["id"]]) == {} + + +@pytest.mark.parametrize("change", ["mtime", "other-symbol"]) +def test_nonrecent_pages_survive_irrelevant_file_changes(shadow_viewer, tmp_path, capsys, change): + shadow = make_shadow(tmp_path) + options = shadow_viewer.RetrievalOptions(limit=1) + shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + first = capsys.readouterr().out + path = shadow / "source.py.md" + if change == "mtime": + timestamp = path.stat().st_mtime + 10 + os.utime(path, (timestamp, timestamp)) + else: + with path.open("a", encoding="utf-8") as stream: + stream.write("\n## `other`\n\n- Other knowledge.\n _(verified, source: exploration)_\n") + shadow_viewer.view_symbol(shadow, "source.py::run", options=replace(options, cursor=cursor(first))) + assert not set(ids(first)) & set(ids(capsys.readouterr().out)) + + +@pytest.mark.parametrize("change", ["source", "labels", "status", "recent-mtime"]) +def test_pages_invalidate_on_relevant_metadata_changes(shadow_viewer, tmp_path, capsys, change): + shadow = make_shadow(tmp_path) + options = shadow_viewer.RetrievalOptions(limit=1) + view = shadow_viewer.view_recent if change == "recent-mtime" else shadow_viewer.view_symbol + args = [shadow] if change == "recent-mtime" else [shadow, "source.py::run"] + view(*args, options=options) + continuation = cursor(capsys.readouterr().out) + path = shadow / "source.py.md" + if change == "recent-mtime": + timestamp = path.stat().st_mtime + 10 + os.utime(path, (timestamp, timestamp)) + else: + replacements = { + "source": ("source: exploration", "source: user"), + "labels": ("labels: [bug]", "labels: [security]"), + "status": ("verified,", "refuted,"), + } + path.write_text(path.read_text(encoding="utf-8").replace(*replacements[change]), encoding="utf-8") + with pytest.raises(ValueError, match="changed"): + view(*args, options=replace(options, cursor=continuation)) + + def test_summary_and_invariant_checks_do_not_count(shadow_viewer, tmp_path, capsys): shadow = make_shadow(tmp_path) store = shadow_viewer.CitationStore.for_shadow(shadow) @@ -309,13 +543,39 @@ def test_write_failure_warns_without_hiding_knowledge(shadow_viewer, tmp_path, c shadow_viewer.view_search(shadow, "Claim") captured = capsys.readouterr() assert identity in captured.out - assert "scores were not updated" in captured.err + assert "this visit was not recorded" in captured.err finally: connection.rollback() connection.close() assert store.scores([identity])[identity] == 1 +def test_counting_can_recover_after_a_failed_score_read(shadow_viewer, tmp_path, monkeypatch, capsys): + shadow = make_shadow(tmp_path, 1) + identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] + store = shadow_viewer.CitationStore.for_shadow(shadow) + store.record([identity]) + lock = sqlite3.connect(store.path) + lock.execute("BEGIN EXCLUSIVE") + + class UnlockOnOutput(io.StringIO): + def flush(self): + lock.rollback() + super().flush() + + output = UnlockOnOutput() + try: + with monkeypatch.context() as context: + context.setattr(sys, "stdout", output) + shadow_viewer.view_search(shadow, "Claim") + finally: + lock.close() + warnings = capsys.readouterr().err + assert "ledger busy" in warnings and "was not recorded" not in warnings + assert "citation_score=?" in output.getvalue() + assert store.scores([identity])[identity] == 2 + + def test_failed_stdout_does_not_increment_score(shadow_viewer, tmp_path, monkeypatch): shadow = make_shadow(tmp_path, 1) store = shadow_viewer.CitationStore.for_shadow(shadow) @@ -398,6 +658,31 @@ def test_cursor_cli_continuation_and_get_are_wired(repo_root, tmp_path): assert "citation_score=1" in expanded.stdout +def test_cli_text_continuation_is_revision_bound_and_counts_one_read(repo_root, tmp_path): + shadow = make_shadow(tmp_path, 1, text_size=300) + script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" + prefix = [sys.executable, str(script), "--shadow-dir", str(shadow)] + found = subprocess.run( + [*prefix, "--symbol", "source.py::run", "--no-record"], + capture_output=True, text=True, encoding="utf-8", check=True, + ) + identity = ids(found.stdout)[0] + command = ["--get", identity, "--max-chars", "500"] + last = None + for _ in range(40): + result = subprocess.run( + [*prefix, *command], capture_output=True, text=True, encoding="utf-8", check=True, + ) + assert len(result.stdout) <= 500 + last = result.stdout + if "\nContinue: " not in result.stdout: + break + command = result.stdout.split("\nContinue: ", 1)[1].strip().split() + else: + pytest.fail("Text continuation did not finish") + assert "citation_score=1" in last + + @pytest.mark.slow def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_path): import shutil @@ -419,7 +704,7 @@ def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_ ["--summary", "--limit", "2"], ["--search", "claim", "--limit", "0"], ["--search", "claim", "--max-chars", "20"], - ["--search", "claim", "--text-offset", "1"], + ["--search", "claim", "--text-cursor", "bad"], ["--top", "source.py", "--max-chars", "600"], ["--get", "d_" + "a" * 32, "--cursor", "bad"], ]) From f579a6129728bb508c8270fd62425e63dd51f584 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 20:49:55 -0400 Subject: [PATCH 05/11] Use local WAL storage for concurrent citation access Keep full commit synchronization while allowing readers and score writers to progress independently. Retain bounded retries and exact-count stress coverage, configure automatic checkpoints, and give the expanded cross-platform regression job time to finish. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/tests.yml | 2 +- CHANGELOG.md | 3 +- skills/shadow-frog-viewer/SKILL.md | 13 +++++--- skills/shadow-frog-viewer/_citations.py | 14 ++++++--- .../shadow_frog_viewer/test_citations.py | 31 +++++++++---------- .../shadow_frog_viewer/test_retrieval.py | 2 ++ 6 files changed, 37 insertions(+), 28 deletions(-) diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 07f318f..e807614 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -26,7 +26,7 @@ jobs: pytest: name: pytest (${{ matrix.os }}, Python 3.12) runs-on: ${{ matrix.os }} - timeout-minutes: 10 + timeout-minutes: 15 strategy: matrix: os: [ubuntu-latest, windows-latest] diff --git a/CHANGELOG.md b/CHANGELOG.md index 4494f2b..dde17b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,8 @@ shadow knowledge bases for any codebase. ### Changed - Citation writes retry temporary SQLite lock contention within their existing - wait budget and reuse the rollback journal without weakening synchronization. + wait budget and use fully synchronized local WAL storage so readers do not + block concurrent score updates. - Retrieval keeps file/symbol identities consistent, preserves literal whitespace and duplicate labels, and binds expansion continuations to one unchanged logical read. Compact hooks share their budget across several previews, and local retry diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 7936fe4..1a38463 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -109,13 +109,16 @@ rewording, renaming, or merging claims/refs can create a new zero-score identity Scores are not fuzzily transferred or summed during Meditate. Removed identities can remain in the local ledger but cannot be expanded unless their claim exists. -Scores live in SQLite under the repository's **common Git directory** at +Scores live in a local-filesystem SQLite database under the repository's +**common Git directory** at `shadowfrog/citations.sqlite3`, shared by its local worktrees. Different shadow roots in the same repo have separate scopes. Outside Git, the cache lives under `$XDG_STATE_HOME/shadowfrog/citations` (Windows: `$LOCALAPPDATA`), falling back to `~/.local/state/shadowfrog/citations`. It contains identities/counters/events, not discovery bodies. It is local metadata, not a tracked or multi-machine DB; -Markdown and its format remain authoritative and unchanged. +Markdown and its format remain authoritative and unchanged. Keep this database +on a local filesystem, not a network share: its WAL journal coordinates readers +and writers on one host. Successful transactions cannot overwrite concurrent increments. Accounting is best-effort: a busy ledger can exhaust the 100 ms wait budget. Unknown scores @@ -127,8 +130,10 @@ Ordinary reads need no retained event receipt. Caller-supplied retry IDs and generated long-read IDs retain receipts for 24 hours, up to 100,000 receipts. New explicit receipts beyond capacity fail visibly rather than weakening retry deduplication. Never reuse a retry ID for an unrelated visit. Expired receipts -are pruned in bounded batches. The fully synchronized rollback journal is capped -at 1 MiB; SQLite can retain reusable free pages in the database. +are pruned in bounded batches. Commits use full synchronization. WAL checkpoints +run automatically after 256 pages, with a 1 MiB retained-journal limit after +reset; an active transaction can temporarily keep a larger journal. SQLite can +retain reusable free pages in the database. Result snapshots freeze ordering despite score changes. Identical snapshots reuse compressed storage; at most 32 snapshots / 8 MiB are retained locally. diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog-viewer/_citations.py index 976e0cd..c6068f9 100644 --- a/skills/shadow-frog-viewer/_citations.py +++ b/skills/shadow-frog-viewer/_citations.py @@ -21,6 +21,7 @@ MAX_PAGES = 32 MAX_PAGE_BYTES = 8 * 1024 * 1024 JOURNAL_BYTES = 1024 * 1024 +CHECKPOINT_PAGES = 256 SCHEMA_VERSION = 2 @@ -127,13 +128,16 @@ def _connection(self, *, timeout=None): f"Unsupported citation database version {version} at {self.path}; " "use a matching helper or move this local cache aside to reset scores" ) - # Reuse the rollback journal instead of creating/deleting it on every - # tiny update, while retaining FULL synchronization and atomic commits. - mode = db.execute("PRAGMA journal_mode=PERSIST").fetchone()[0] - if mode != "persist": - raise ValueError(f"Cannot enable persistent citation journaling (got {mode})") + # Local worktrees share WAL so readers do not contend with each + # small score update. FULL still synchronizes successful commits. + mode = db.execute("PRAGMA journal_mode").fetchone()[0] + if mode != "wal": + mode = db.execute("PRAGMA journal_mode=WAL").fetchone()[0] + if mode != "wal": + raise ValueError(f"Cannot enable local WAL citation storage (got {mode})") db.execute("PRAGMA synchronous=FULL") db.execute(f"PRAGMA journal_size_limit={JOURNAL_BYTES}") + db.execute(f"PRAGMA wal_autocheckpoint={CHECKPOINT_PAGES}") if version == 0: with db: db.execute("BEGIN IMMEDIATE") diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog_viewer/test_citations.py index 4da6df7..3e7ff28 100644 --- a/tests/skills/shadow_frog_viewer/test_citations.py +++ b/tests/skills/shadow_frog_viewer/test_citations.py @@ -85,21 +85,20 @@ def test_processes_do_not_lose_increments(citations, tmp_path): assert citations.CitationStore(path, ".shadow").scores([identity])[identity] == 180 -def test_writes_reuse_journal_without_disabling_synchronization(citations, tmp_path): +def test_wal_writes_keep_full_synchronization_and_checkpoint_limits(citations, tmp_path): store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") identity = citations.discovery_id("file", "a::f", "Claim") store.record([identity], "first") with store._connection() as db: - assert db.execute("PRAGMA journal_mode").fetchone()[0] == "persist" + assert db.execute("PRAGMA journal_mode").fetchone()[0] == "wal" assert db.execute("PRAGMA synchronous").fetchone()[0] >= 2 - journal = tmp_path / "citations.sqlite3-journal" - assert journal.is_file() - assert journal.read_bytes()[:28] == b"\0" * 28 + assert db.execute("PRAGMA journal_size_limit").fetchone()[0] == citations.JOURNAL_BYTES + assert db.execute("PRAGMA wal_autocheckpoint").fetchone()[0] == citations.CHECKPOINT_PAGES store.record([identity], "second") assert store.scores([identity])[identity] == 2 -def test_busy_commit_retries_whole_transaction_without_duplicate_updates(citations, tmp_path): +def test_active_reader_does_not_block_committing_citations(citations, tmp_path): path = tmp_path / "citations.sqlite3" store = citations.CitationStore(path, ".shadow") identity = citations.discovery_id("file", "a::f", "Claim") @@ -112,23 +111,20 @@ def test_busy_commit_retries_whole_transaction_without_duplicate_updates(citatio from pathlib import Path sys.path.insert(0, sys.argv[1]) from _citations import CitationStore -store = CitationStore(Path(sys.argv[2]), ".shadow", timeout=2) -def update(db): - db.execute("UPDATE scores SET citation_score = citation_score + 1") - print("attempt", flush=True) -store._operation(update, write=True) +store = CitationStore(Path(sys.argv[2]), ".shadow") +store.record([sys.argv[3]], "concurrent") +store.record([sys.argv[3]], "concurrent") """ process = subprocess.Popen( - [sys.executable, "-c", code, str(HELPER.parent), str(path)], + [sys.executable, "-c", code, str(HELPER.parent), str(path), identity], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", ) try: - # Reader permits BEGIN IMMEDIATE and the update, but prevents COMMIT. - assert process.stdout.readline().strip() == "attempt" - assert process.stdout.readline().strip() == "attempt" - reader.rollback() + # A held read snapshot no longer prevents the writer's commit. stdout, stderr = process.communicate(timeout=15) assert process.returncode == 0, stdout + stderr + assert reader.execute("SELECT citation_score FROM scores").fetchone()[0] == 1 + reader.rollback() finally: reader.close() if process.poll() is None: @@ -325,7 +321,8 @@ def test_journal_size_remains_capped_after_large_snapshot_cleanup(citations, tmp with store._connection() as db, db: db.execute("UPDATE pages SET created=0") store.save_page("small", "catalog", ["one"]) - assert (tmp_path / "citations.sqlite3-journal").stat().st_size <= 4096 + journal = tmp_path / "citations.sqlite3-wal" + assert not journal.exists() or journal.stat().st_size <= 4096 def test_production_budget_preserves_all_successful_writes(citations, tmp_path): diff --git a/tests/skills/shadow_frog_viewer/test_retrieval.py b/tests/skills/shadow_frog_viewer/test_retrieval.py index d1bbd22..aa63537 100644 --- a/tests/skills/shadow_frog_viewer/test_retrieval.py +++ b/tests/skills/shadow_frog_viewer/test_retrieval.py @@ -556,6 +556,8 @@ def test_counting_can_recover_after_a_failed_score_read(shadow_viewer, tmp_path, store = shadow_viewer.CitationStore.for_shadow(shadow) store.record([identity]) lock = sqlite3.connect(store.path) + # A pre-existing rollback cache can still be locked during WAL activation. + lock.execute("PRAGMA journal_mode=DELETE") lock.execute("BEGIN EXCLUSIVE") class UnlockOnOutput(io.StringIO): From 641f48f6bc7ede4471029e134089766e2d128369 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 22:10:03 -0400 Subject: [PATCH 06/11] Keep file-first navigation and separate optional agent retrieval Move shared knowledge/citation helpers into the core skill, expose bounded reads by known file or symbol, and keep the Viewer user-facing. Restore direct-navigation guidance, route hooks through the core reader, and include the optional reference in pinned tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/tests.yml | 4 +- CHANGELOG.md | 11 +- README.md | 21 +- RESPONSIBLE_AI.md | 7 +- agent-context.md | 26 +- claude.md | 19 +- .../scripts/shadow-frog-pre-tool.sh | 34 +- skills/shadow-frog-dream/dream-tools.py | 4 +- skills/shadow-frog-viewer/SKILL.md | 110 +- skills/shadow-frog-viewer/shadow-viewer.py | 1587 +----------- skills/shadow-frog/SKILL.md | 83 +- .../_citations.py | 2 +- skills/shadow-frog/_knowledge.py | 1613 ++++++++++++ skills/shadow-frog/retrieval.md | 108 + skills/shadow-frog/shadow-read.py | 34 + tests/conftest.py | 10 + tests/hooks/test_pre_tool_sh.py | 2 +- tests/skills/shadow_frog/conftest.py | 9 + .../test_citations.py | 2 +- tests/skills/shadow_frog/test_knowledge.py | 2186 ++++++++++++++++ .../test_retrieval.py | 300 +-- tests/skills/shadow_frog/test_shadow_read.py | 171 ++ .../shadow_frog_dream/test_dream_tools.py | 2 + tests/skills/shadow_frog_nap/test_nap.py | 6 +- .../shadow_frog_viewer/test_shadow_viewer.py | 2215 +---------------- tests/test_smoke.py | 3 + 26 files changed, 4493 insertions(+), 4076 deletions(-) mode change 100755 => 100644 skills/shadow-frog-viewer/shadow-viewer.py rename skills/{shadow-frog-viewer => shadow-frog}/_citations.py (99%) create mode 100644 skills/shadow-frog/_knowledge.py create mode 100644 skills/shadow-frog/retrieval.md create mode 100644 skills/shadow-frog/shadow-read.py create mode 100644 tests/skills/shadow_frog/conftest.py rename tests/skills/{shadow_frog_viewer => shadow_frog}/test_citations.py (99%) create mode 100644 tests/skills/shadow_frog/test_knowledge.py rename tests/skills/{shadow_frog_viewer => shadow_frog}/test_retrieval.py (67%) create mode 100644 tests/skills/shadow_frog/test_shadow_read.py diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index e807614..6d379b6 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -9,7 +9,7 @@ permissions: jobs: viewer-compatibility: - name: Viewer (Python 3.9) + name: Shadow helpers (Python 3.9) runs-on: ubuntu-latest steps: - name: Checkout @@ -22,6 +22,8 @@ jobs: run: | python skills/shadow-frog-viewer/shadow-viewer.py --help python skills/shadow-frog-viewer/shadow-viewer.py --shadow-dir examples/coupon-demo/.shadow --search coupon --limit 2 --no-record + python skills/shadow-frog/shadow-read.py --help + python skills/shadow-frog/shadow-read.py cart.py --shadow-dir examples/coupon-demo/.shadow --limit 2 --no-record pytest: name: pytest (${{ matrix.os }}, Python 3.12) diff --git a/CHANGELOG.md b/CHANGELOG.md index dde17b2..004b58a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,8 +12,8 @@ shadow knowledge bases for any codebase. ### Added - **Single-score knowledge retrieval (draft)** — derived discovery fingerprints, zero-default local citation scores, and concurrent-safe SQLite bookkeeping - shared across Git worktrees. Bounded search/symbol views, stable pagination, - and individual expansion avoid loading entire large shadow sections. + shared across Git worktrees. Optional core file/symbol retrieval, stable + pagination, and individual expansion avoid loading entire large sections. Citation scores measure emitted content, not correctness or proven usefulness. ### Fixed @@ -30,9 +30,10 @@ shadow knowledge bases for any codebase. and duplicate labels, and binds expansion continuations to one unchanged logical read. Compact hooks share their budget across several previews, and local retry receipts, pagination snapshots, and journals have explicit retention limits. -- Retrieval views now show IDs/scores and paginate by default. Preferences - remain separately accessible, trust and relevance precede popularity, and - hooks remain fail-open while surfacing citation warnings. +- Direct file/symbol navigation remains the agent default. Optional agent + retrieval lives in `shadow-frog`; the Viewer serves user browsing and + visualization. Both share parsing and citation metadata. Hooks use the core + reader and remain fail-open while surfacing citation warnings. - **More concise documentation** — consolidated README onboarding and workflow guidance, with advanced operations linked to the skill references. Condensed repeated guidance and examples in the core, Dream, Init, Meditate, Update, diff --git a/README.md b/README.md index 36cfbcc..d0bedfd 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ helper commands, and format definitions. | [`/shadow-frog-dream`](skills/shadow-frog-dream/SKILL.md) | Run autonomous experiments while you're away | | [`/shadow-frog-nap`](skills/shadow-frog-nap/SKILL.md) | Generate reviewed feature-task briefs without implementing them | | [`/shadow-frog-meditate`](skills/shadow-frog-meditate/SKILL.md) | Merge duplicates and resolve conflicting discoveries | -| [`/shadow-frog-viewer`](skills/shadow-frog-viewer/SKILL.md) | Browse, search, inspect lineage, and audit structural integrity | +| [`/shadow-frog-viewer`](skills/shadow-frog-viewer/SKILL.md) | User-facing CLI browsing, lineage visualization, and structural audits | As you work, the agent captures your code context as `source: user` and collaborative findings as `source: interaction`. After commits, the pre-tool @@ -118,14 +118,17 @@ For example, use Viewer to find relevant knowledge or audit its structure: The [Viewer reference](skills/shadow-frog-viewer/SKILL.md) also covers summaries, recent discoveries, label filters, preferences, and interactive dream-lineage HTML. -Knowledge retrieval is bounded and pageable, with IDs for expanding individual -claims. A single local `citation_score` counts helper exposures, not proven -usefulness; relevance and trust outrank popularity. Scores are updated safely -across local Git worktrees without editing shadow Markdown or requiring a vector -index. New claims start at zero. See the Viewer reference for retries, local -storage, and the limits of this signal. Long-entry continuation counts as one -logical read and rejects changed content; local retry and pagination metadata -have retention limits. Compact hook hints are not a substitute for a file review. +Agents normally navigate directly from source files/symbols to their mirrored +Markdown shadows. The **Viewer is for users**; it is not required for agent +lookup. When a section is too large, agents can use the optional core +[`shadow-read.py` helper](skills/shadow-frog/retrieval.md) to read a known file or +symbol within a context budget, or to search and page matching knowledge. + +A single local `citation_score` counts helper exposures, not proven usefulness; +relevance and trust outrank popularity. The core reader and user Viewer share +safe local bookkeeping across worktrees without changing Markdown or requiring +a vector index. Native reads remain normal and uncounted. New claims start at +zero, and long-entry continuation counts as one logical read. --- diff --git a/RESPONSIBLE_AI.md b/RESPONSIBLE_AI.md index 0bb3ca5..a7f11ed 100644 --- a/RESPONSIBLE_AI.md +++ b/RESPONSIBLE_AI.md @@ -104,9 +104,12 @@ At a high level, we found that ShadowFrog performed strongly on knowledge retrie ## Limitations -Citation scores are local counts of discovery content emitted by the viewer, +Citation scores are local counts of discovery content emitted by the optional +core retrieval helper or the user-facing Viewer, not proof that an agent used it, that it improved an outcome, or that it is true. -Raw file reads and failed ledger updates are not counted. Scores can be biased +Direct file/symbol reads are the primary agent workflow and do not require these +helpers or their database. Native reads and failed ledger updates are not counted; +instrumentation is therefore partial by design. Scores can be biased by prior ranking and repeated exposure; relevance, provenance, and verification remain more important. Bounded results may omit relevant knowledge, so agents must follow pagination and inspect preferences rather than treating a shortlist diff --git a/agent-context.md b/agent-context.md index ee980c6..05caaf9 100644 --- a/agent-context.md +++ b/agent-context.md @@ -2,22 +2,26 @@ This project uses a `.shadow/` knowledge base with verified discoveries about non-obvious code behavior. **You MUST consult the shadow before making any code change.** -1. **Check the shadow first** — use `/shadow-frog-viewer --search ` before - editing and `--symbol ::` for focused follow-up. Expand IDs with - `--get` and page large results. Hook-sized `--top` hints are not exhaustive. -2. **Check preferences** — use `--prefs` and follow all pages for project conventions. -3. **Check cross-cutting** — use `--search` for related multi-file knowledge. +1. **Read preferences first** — `.shadow/_prefs.md` contains project conventions. +2. **Navigate directly** — before editing ``, read `.shadow/.md` + and its relevant symbol sections using native file reads/searches. +3. **Follow cross-cutting links** — inspect the relevant `.shadow/_cross/` entries. 4. **Act on what you find** — apply what you learn from the shadow to your work. 5. **After making changes** — run `/shadow-frog-update` to capture learnings The shadow contains discoveries from code analysis and user conversations. Always consult it before making assumptions about code behavior. -The viewer records one local `citation_score` per emitted discovery, atomically -across worktrees; continuation chunks share one logical read. Never edit counters -in Markdown. It is an exposure count, not -proof of correctness or usefulness. Trust and relevance outrank popularity. -Raw file reads are a fallback and are not counted. A ranked shortlist is not -exhaustive: search the specific claim before adding duplicate knowledge. +File/symbol paths are sufficient; no viewer or database is required to read the +knowledge. For large sections, the optional `shadow-read.py` in the core +`shadow-frog` skill can retrieve a known file/symbol or search and page results. +Use the user-facing `/shadow-frog-viewer` when the **user** asks to browse or visualize knowledge, +not as the mandatory read path for code work. + +Helper reads update one local `citation_score` atomically across worktrees; +continuation chunks share one logical read. Native reads are normal and uncounted. +Never edit counters in Markdown or reread only to increase them. Scores measure +exposure, not correctness or usefulness. A shortlist is not exhaustive: inspect +the specific claim before adding duplicate knowledge. ### Key directories diff --git a/claude.md b/claude.md index 5870599..55b8fe7 100644 --- a/claude.md +++ b/claude.md @@ -11,6 +11,10 @@ ShadowFrog/ skills/ shadow-frog/SKILL.md Main entrypoint (docs, reference system, search) shadow-frog/_coherence.py Shared structural parent-connection validation + shadow-frog/shadow-read.py Optional bounded agent retrieval by file/symbol + shadow-frog/_knowledge.py Shared parsing, retrieval, and user-facing views + shadow-frog/_citations.py Atomic local citation ledger and retrieval cursors + shadow-frog/retrieval.md On-demand helper and telemetry reference shadow-frog-init/ First-time setup (create .shadow/) SKILL.md Init instructions + fallback steps shadow-init.py Python helper script @@ -29,10 +33,9 @@ ShadowFrog/ SKILL.md Bounded ideation, evidence, and task export instructions nap.py Portable record validator, parent context, and exporter shadow-frog-meditate/SKILL.md Dedup, merge, and resolve conflicting discoveries - shadow-frog-viewer/ Browse and query the shadow knowledge base + shadow-frog-viewer/ User-facing knowledge inspection and visualization SKILL.md Query instructions + shell fallbacks - shadow-viewer.py Python helper script - _citations.py Atomic local citation ledger and retrieval cursors + shadow-viewer.py User CLI using the core shared knowledge implementation dream-lineage.py Dream lineage visualization hook-templates/ shadow-frog-hooks.json Copilot CLI hook config (sessionStart, preToolUse) @@ -119,14 +122,22 @@ Preference (`_prefs.md` — project-wide, no file/symbol anchor): ### Citation-Aware Retrieval +- Direct `.shadow/.md` and symbol navigation is primary for agents. + Optional agent helpers live under `shadow-frog/`; the Viewer serves user + browsing/visualization requests, not a required code-work retrieval gateway. +- Share parsing, identities and counters through core `_knowledge.py` and + `_citations.py`. Hooks use the core reader. Do not duplicate backend logic or + add compatibility re-exports in the user Viewer. - Keep the discovery grammar unchanged: viewer fingerprints and `citation_score` are derived/local metadata, not additional Markdown fields. - Scores start at zero and increment only for content emitted by a retrieval view; expansion chunks share one revision-bound logical read. They measure exposure, not verified usefulness. -- Use the common-Git SQLite ledger for multiprocess/worktree updates; do not +- Use the common-Git SQLite ledger for instrumented multiprocess/worktree updates; do not rewrite shadow files on reads. Telemetry failures must warn without hiding knowledge. Summary/audit/parser-only operations do not increment scores. +- Native reads remain normal and uncounted. Never make citation accounting a + prerequisite for accessing Markdown, or require an extra read merely to count. - Rank relevance and trust ahead of scores; preserve room for zero-score entries. Page large results, expand by ID, and never use only the shortlist for dedup. - Fingerprints bind kind, canonical anchor, whitespace-preserving parsed claim diff --git a/hook-templates/scripts/shadow-frog-pre-tool.sh b/hook-templates/scripts/shadow-frog-pre-tool.sh index 10dcbf7..061f18d 100755 --- a/hook-templates/scripts/shadow-frog-pre-tool.sh +++ b/hook-templates/scripts/shadow-frog-pre-tool.sh @@ -6,7 +6,7 @@ # When a mutation tool (edit/create/str_replace/write) targets a file with # a shadow that has actionable discoveries (bug/security labels), the # top entries are inlined into additionalContext via -# shadow-viewer.py --top. Per-session dedup ensures the same file's +# shadow-frog/shadow-read.py --top. Per-session dedup ensures the same file's # content is injected at most once per Copilot CLI process. # This hook is ADVISORY — it only injects shadow context, it is NOT a security @@ -19,7 +19,7 @@ # 3. trap on TERM/HUP/INT — converts runner-initiated signal kills to 0. # (bash 3.2+ on macOS and bash 5+ on Linux verified: EXIT alone is NOT # enough — SIGTERM still produces exit 143/-15 without a TERM trap.) -# 4. Every external call (git, python3, shadow-viewer.py) MUST be wrapped in +# 4. Every external call (git, python3, shadow-read.py) MUST be wrapped in # a bounded subprocess timeout. If the foreground child hangs, bash will # queue the signal until the child returns, so the trap can't save us # unless boundedness holds. CI enforces this via .github/workflows/shellcheck.yml. @@ -133,13 +133,13 @@ if [ "$IS_MUTATION" = "1" ]; then if [ -n "$0" ]; then SCRIPT_DIR="$(cd "$(dirname "$0")" 2>/dev/null && pwd -P)" || SCRIPT_DIR="" fi - # Resolve viewer + run it inside ONE Python block. Every - # external call (git rev-parse, viewer subprocess) is + # Resolve reader + run it inside ONE Python block. Every + # external call (git rev-parse, reader subprocess) is # bounded with subprocess.run(timeout=...) so a hung git - # or hung viewer can't blow the hook's 5s budget — even + # or hung reader can't blow the hook's 5s budget — even # the trap pyramid can't help if bash is blocked waiting # on an unbounded foreground child (signals are queued). - # Total bounded work here is ~1.5s (rev-parse 0.5s + viewer 1.0s). + # Total bounded work here is ~1.5s (rev-parse 0.5s + reader 1.0s). TOP_OUTPUT=$(SF_SCRIPT_DIR="$SCRIPT_DIR" SF_REL_PATH="$REL_PATH" python3 - <<'PYEOF' 2>/dev/null || true import os, subprocess, sys @@ -155,7 +155,7 @@ def _git(args, timeout): pass return '' -# Locate shadow-viewer.py. Order: +# Locate the core shadow-read.py helper. Order: # 1. Script-relative — works for source repo dev AND project installs # (hooks at .github/hooks/scripts/ co-located with .github/skills/). # 2-3. Parent repo's .github/ or .claude/ skills — useful when the hook @@ -164,21 +164,21 @@ repo_root = _git(['rev-parse', '--show-toplevel'], 0.5) candidates = [] if script_dir: - candidates.append(os.path.join(script_dir, '..', '..', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) + candidates.append(os.path.join(script_dir, '..', '..', 'skills', 'shadow-frog', 'shadow-read.py')) if repo_root: - candidates.append(os.path.join(repo_root, '.github', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) - candidates.append(os.path.join(repo_root, '.claude', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) + candidates.append(os.path.join(repo_root, '.github', 'skills', 'shadow-frog', 'shadow-read.py')) + candidates.append(os.path.join(repo_root, '.claude', 'skills', 'shadow-frog', 'shadow-read.py')) -viewer = '' +reader = '' for candidate in candidates: if os.path.isfile(candidate): - viewer = candidate + reader = candidate break -if viewer: +if reader: try: r = subprocess.run( - ['python3', viewer, + ['python3', reader, '--shadow-dir', '.shadow', '--top', rel_path, '--top-labels', 'bug,security', @@ -189,12 +189,12 @@ if viewer: if r.returncode == 0: sys.stdout.write(r.stdout.strip()) if r.stderr.strip(): - sys.stdout.write("\n[ShadowFrog] Viewer reported a warning; rerun it directly for details. Citation updates may be unavailable.") + sys.stdout.write("\n[ShadowFrog] Reader reported a warning; rerun it directly for details. Citation updates may be unavailable.") except Exception: pass PYEOF ) - # Only inline when the viewer produced an actionable + # Only inline when the reader produced an actionable # response (non-empty and not the "no discoveries" sentinel). if [ -n "$TOP_OUTPUT" ] && [[ "$TOP_OUTPUT" != "No actionable"* ]]; then MSG="[ShadowFrog] Actionable discoveries for ${REL_PATH} (verify against source before acting): @@ -210,7 +210,7 @@ fi # Staleness warning (appended when shadow is behind HEAD). # All git work is bounded with per-call subprocess timeouts so a huge/locked # repo can't blow past the hook's 5s budget. Timeouts sum to 2.0s here, -# matched with the viewer's ~1.5s above + bash overhead = ~4s worst case, +# matched with the reader's ~1.5s above + bash overhead = ~4s worst case, # leaving 1s headroom under timeoutSec=5. Any failure/timeout -> no warning. CHANGED=$(python3 - <<'PYEOF' 2>/dev/null || echo "" import json, subprocess diff --git a/skills/shadow-frog-dream/dream-tools.py b/skills/shadow-frog-dream/dream-tools.py index c29c41d..9abaa82 100644 --- a/skills/shadow-frog-dream/dream-tools.py +++ b/skills/shadow-frog-dream/dream-tools.py @@ -78,13 +78,13 @@ def _source_files() -> dict[str, Path]: sources = {} for directory in (DREAM_DIR, CORE_DIR): for path in directory.iterdir(): - if path.name != "SKILL.md" and path.suffix not in (".py", ".sh"): + if path.suffix not in (".py", ".sh", ".md"): continue if path.is_symlink() or not path.is_file(): raise ValueError(f"Tooling assets must be regular files: {path}") sources[f"{directory.name}/{path.name}"] = path required = { - "shadow-frog/SKILL.md", "shadow-frog/_coherence.py", + "shadow-frog/SKILL.md", "shadow-frog/retrieval.md", "shadow-frog/_coherence.py", "shadow-frog-dream/SKILL.md", "shadow-frog-dream/dream-tools.py", "shadow-frog-dream/_worktree_safety.py", *(f"shadow-frog-dream/{name}" for name in TOOLS.values()), diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 1a38463..392fb59 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -1,11 +1,12 @@ --- name: shadow-frog-viewer description: >- - Browse and query the shadow knowledge base with bounded, citation-ranked - retrieval: search files, symbols, or text, expand individual discoveries, - page through preferences, or see recent discoveries. - Invoke when the user wants to see what's in the shadow, get an - overview, or find specific knowledge. + Help users browse and visualize their collected shadow knowledge in the + terminal or as an interactive dream-lineage report. Show an overview, + search results, preferences, recent discoveries, and structural audits. + Invoke when the user asks to inspect the shadow. For agents' own code work, + direct file/symbol navigation is primary; optional retrieval helpers live + in the core shadow-frog skill. scripts: - shadow-viewer.py - dream-lineage.py @@ -13,7 +14,11 @@ scripts: # ShadowFrog Viewer -Query and browse `.shadow/` content. Prerequisite: `.shadow/` exists. +**User-facing inspection and visualization** of `.shadow/`. Prerequisite: +`.shadow/` exists. An agent can run these views on the user's behalf, but this +skill is not the agent's required knowledge interface. Agent work starts with +the mirrored file/symbol locations; the core `shadow-read.py` is optional for +large sections or targeted searches. ## Primary: Python Helper Script @@ -33,12 +38,13 @@ python3 .claude/skills/shadow-frog-viewer/shadow-viewer.py [options] |---------|--------------| | `--summary` | Overview: counts, source/status/label breakdown, per-file table, cross-cutting titles (default) | | `--search QUERY` | Bounded search across paths, symbols, text, cross-cutting entries, and preferences | +| `--file FILE` | Browse one known source file's shadow, including file-level and cross-cutting knowledge | | `--symbol FILE::SYMBOL` | Bounded discoveries at an exact symbol, plus matching cross-cutting refs; use `File-Level` for a file-level section | | `--get ID` | Expand one current discovery; long expansions return a revision-bound continuation | | `--prefs` | Project-wide preferences; follow all pages before treating them as complete | | `--recent [N]` | Most recent discovery previews by file mtime (default page size: 10) | | `--labels LABEL` | Bounded discoveries matching labels (e.g., `bug`, `security`, `bug,performance`) | -| `--top FILE` | Hook-sized actionable previews (default: up to 3 entries, 600 characters). Includes per-file and cross-cutting findings; short symbol/ID lines omit numeric scores to leave room for content. Not an exhaustive file view. | +| `--top FILE` | Compact actionable previews (default: up to 3 entries, 600 characters). Not an exhaustive file view. Agent hooks use the separate core reader. | | `--check-invariants` | Audit structural integrity — bidirectional cross-references, label/source/category enum compliance, heading format, no-orphan-back-pointer. Exits 0 if clean, 1 with one violation per line. Run after dream reconciliation or before commit. | No arguments defaults to `--summary`. @@ -48,7 +54,7 @@ No arguments defaults to `--summary`. | Flag | Effect | |------|--------| | `--shadow-dir DIR` | Override .shadow/ location (default: auto-detect from CWD) | -| `--limit N` | Positive page size for search, symbol, labels, or preferences (default: 10) | +| `--limit N` | Positive page size for file, search, symbol, labels, or preferences (default: 10) | | `--max-chars N` | Hard output cap, including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output). Use `--top-max-chars` with `--top`. | | `--cursor TOKEN` | Continue the same view and filters using the returned ordering snapshot | | `--text-cursor TOKEN` | Continue the same `--get` body and logical read; copy the returned token rather than fabricating an offset | @@ -78,77 +84,19 @@ python3 shadow-viewer.py --labels security,performance python3 shadow-viewer.py --top src/auth.py --top-labels bug,security,performance --top-limit 5 ``` -### Citation Score and Retrieval Contract - -Replace `DISCOVERY_ID` and `CURSOR_TOKEN` with the exact values returned by the helper. - -Content views (`search`, `symbol`, `get`, `prefs`, `labels`, `recent`, `top`) -return an `id`; all except compact `top` also show one `citation_score`. -Every discovery starts at zero by -default, regardless of which workflow wrote it. The helper increments only -entries whose content it emits. Expanding a long claim counts as one logical -read across its continuation chunks. Scanning/matching, -summary statistics, invariant audits, and raw file reads do not count. -Displayed scores are the values **before** the current read. A citation here -measures helper exposure, not proven usefulness, correctness, or LLM influence. -Agents must not manually edit counters or add them to discovery metadata. - -Exact search matches and source trust/status rank ahead of citation history; -recent views also prioritize mtime. Within a tied tier, higher scores rank -first, reserving room for a zero-score entry within tied tiers when the page -and character budget can fit multiple entries. -Popularity never overrides a refuted status or authorizes dropping a constraint. -Use targeted searches and additional pages for deduplication rather than -assuming the popular shortlist is exhaustive. - -The `d_...` ID binds kind, canonical file/symbol anchor, parsed claim text, and -related refs. Internal whitespace is preserved, including code literals. -Filesystem aliases resolve to the same on-disk name; container-heading prefixes -are removed from symbol anchors. Metadata-only status/source/label changes keep it; -rewording, renaming, or merging claims/refs can create a new zero-score identity. -Scores are not fuzzily transferred or summed during Meditate. Removed identities -can remain in the local ledger but cannot be expanded unless their claim exists. - -Scores live in a local-filesystem SQLite database under the repository's -**common Git directory** at -`shadowfrog/citations.sqlite3`, shared by its local worktrees. Different shadow -roots in the same repo have separate scopes. Outside Git, the cache lives under -`$XDG_STATE_HOME/shadowfrog/citations` (Windows: `$LOCALAPPDATA`), falling back to -`~/.local/state/shadowfrog/citations`. It contains identities/counters/events, -not discovery bodies. It is local metadata, not a tracked or multi-machine DB; -Markdown and its format remain authoritative and unchanged. Keep this database -on a local filesystem, not a network share: its WAL journal coordinates readers -and writers on one host. - -Successful transactions cannot overwrite concurrent increments. Accounting is -best-effort: a busy ledger can exhaust the 100 ms wait budget. Unknown scores -display as `?`, but counting is still attempted after output; failed increments -warn explicitly that the visit was not recorded. Retry a busy database later, -not by deleting it. Failed stdout emission is not recorded. - -Ordinary reads need no retained event receipt. Caller-supplied retry IDs and -generated long-read IDs retain receipts for 24 hours, up to 100,000 receipts. -New explicit receipts beyond capacity fail visibly rather than weakening retry -deduplication. Never reuse a retry ID for an unrelated visit. Expired receipts -are pruned in bounded batches. Commits use full synchronization. WAL checkpoints -run automatically after 256 pages, with a 1 MiB retained-journal limit after -reset; an active transaction can temporarily keep a larger journal. SQLite can -retain reusable free pages in the database. - -Result snapshots freeze ordering despite score changes. Identical snapshots -reuse compressed storage; at most 32 snapshots / 8 MiB are retained locally. -They expire after 24 hours or capacity eviction. Keep the original view/filters -with `--cursor`; changed matching content/trust/labels require restarting. -Timestamp-only or unrelated-symbol edits do not invalidate non-recent views. - -For a long claim, copy the returned `--text-cursor` continuation. It binds the -exact expanded body and metadata, preserves `--no-record`, and reuses one read -event. Changed content or an expired token requires restarting `--get`. Character -limits bound helper output, not token counts; -the helper can still scan the underlying Markdown locally. If the ledger is -unavailable, repair local-state access before relying on exhaustive pagination. -An incompatible prerelease cache must be moved aside to reset local scores; -never alter shadow content to repair telemetry. +### Scores and Continuation + +Replace `DISCOVERY_ID` and `CURSOR_TOKEN` with exact returned values. Content +views display the same IDs and local `citation_score` as the optional core +reader, recording only the entries they show. Scores measure helper exposure, +not correctness or proven use; direct file reads remain normal and uncounted. +Knowledge is still plain Markdown at its file/symbol location, not in the ledger. + +Use returned cursors to continue, `--get` to inspect a claim, and `--no-record` +when browsing should not affect scores. Forward any stderr diagnostics to the +user; optional telemetry failures must not hide knowledge. See the shared +[retrieval reference](../shadow-frog/retrieval.md) for exact identity, budget, +retry, and local storage contracts. ## Dream Lineage Visualization @@ -212,7 +160,7 @@ find .shadow -name '*.md' -not -path '*/_meta/*' -printf '%T@ %p\n' | sort -rn | ## Responding to the User -- Preserve `--top` output as-is; it is intentionally compact and pre-formatted - for the preToolUse hook. +- Preserve the meaning of returned trust/status and citation information; + scores are not confidence estimates. Expand or page results on user request. - If the shadow is empty or has no discoveries, suggest running `/shadow-frog-dream` to populate it diff --git a/skills/shadow-frog-viewer/shadow-viewer.py b/skills/shadow-frog-viewer/shadow-viewer.py old mode 100755 new mode 100644 index 7379131..7c828ed --- a/skills/shadow-frog-viewer/shadow-viewer.py +++ b/skills/shadow-frog-viewer/shadow-viewer.py @@ -1,1598 +1,31 @@ #!/usr/bin/env python3 -"""shadow-viewer: Query and browse a .shadow/ knowledge base. +"""Browse and visualize collected shadow knowledge for users in the terminal. -Usage: - shadow-viewer.py [options] - -Views: - --summary Overview + detailed statistics (default) - --search QUERY Universal search across files, symbols, and text - --prefs Show project-wide preferences - --recent [N] N most recent discoveries with content (default: 10) - --labels LABEL Show discoveries by label (bug, security, etc.) - --top FILE Top actionable discoveries for FILE (hook-sized) - --symbol FILE::SYMBOL Bounded knowledge for a specific symbol - --get ID Expand one discovery by its retrieval identity - --check-invariants Report structural violations (exit 1 if any) - -Options: - --shadow-dir DIR Path to .shadow/ directory (default: auto-detect) - --limit N Maximum results (default: 10) - --max-chars N Output budget including metadata (default: 4000) - --cursor TOKEN Continue a previous result snapshot - --text-cursor TOKEN Continue an unchanged discovery as one logical read - --no-record Do not increment local citation scores - --event-id ID Reuse an ID for retry-idempotent bookkeeping - -Exit codes: - 0 Success (possibly with warnings on stderr) - 1 Fatal error (shadow dir not found, no results possible) +Run without arguments for an overview, or use --search, --prefs, --recent, +--labels and --check-invariants. The separate dream-lineage.py renders HTML. +Agent workflows use direct file/symbol navigation, with optional bounded +retrieval through the core shadow-frog/shadow-read.py helper. """ -from __future__ import annotations - -import argparse -from dataclasses import dataclass -import hashlib -import json -import os -import re import sys -import sqlite3 -import subprocess -import time -import traceback -import uuid -from collections import defaultdict -from datetime import datetime from pathlib import Path -sys.path.insert(0, str(Path(__file__).resolve().parent)) _bytecode = sys.dont_write_bytecode sys.dont_write_bytecode = True +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "shadow-frog")) try: - from _citations import ( - CitationStore, RECEIPT_TTL, canonical_path, discovery_id, is_busy_error, - validate_event_id, - ) + import _knowledge except ImportError as exc: - raise SystemExit("[shadow-viewer error] Missing citation helper/dependency; reinstall the complete Viewer skill") from exc + raise SystemExit("ERROR: Missing core knowledge helper; reinstall the full ShadowFrog skill set") from exc finally: sys.path.pop(0) sys.dont_write_bytecode = _bytecode -_DISCOVERY_META_RE = re.compile( - r"_\((\w+),\s*source:\s*(\w+)" - r"(?:,\s*labels:\s*\[([^\]]*)\])?" - r"\)_" -) - - -def warn(msg): - """Print a warning to stderr. Agents read these to adjust strategy.""" - print(f"[shadow-viewer warning] {msg}", file=sys.stderr) - - -def error(msg): - """Print an error to stderr.""" - print(f"[shadow-viewer error] {msg}", file=sys.stderr) - - -def find_shadow_dir(start="."): - """Walk up from start to find .shadow/ directory.""" - try: - p = Path(start).resolve() - while p != p.parent: - candidate = p / ".shadow" - if candidate.is_dir(): - return candidate - p = p.parent - except (OSError, PermissionError) as e: - error(f"Failed to search for .shadow/ directory from '{start}': {e}") - return None - - -def parse_discovery(line, continuation_lines=None): - """Parse a discovery bullet and its metadata line(s).""" - try: - text = line[2:].strip() if line.startswith("- ") else line.strip() - except (TypeError, AttributeError) as e: - warn(f"parse_discovery: bad line input ({type(line).__name__}): {e}") - return {"text": str(line) if line else ""} - - meta = {} - full_text = text - - if continuation_lines: - for cl in continuation_lines: - try: - stripped = cl.strip() - # _(status, source: type, labels: [l1, l2])_ or _(status, source: type)_ - m = _DISCOVERY_META_RE.match(stripped) - if m: - meta["status"] = m.group(1) - meta["source"] = m.group(2) - if m.group(3): - meta["labels"] = [ - l.strip() - for l in m.group(3).split(",") - if l.strip() - ] - else: - # _(source: type)_ (preferences format) - m2 = re.match(r"_\(source:\s*(\w+)\)_", stripped) - if m2: - meta["source"] = m2.group(1) - elif stripped.startswith("Also involves:"): - refs = re.findall(r"`([^`]+)`", stripped) - meta["also_involves"] = refs - elif stripped.startswith("Dream report:"): - m_dr = re.search(r"`([^`]+)`", stripped) - if m_dr: - meta["dream_report"] = m_dr.group(1) - else: - full_text += " " + stripped - except Exception as e: - warn(f"parse_discovery: failed parsing continuation line " - f"'{cl[:80]}': {e}") - - return {"text": full_text, **meta} - - -def parse_shadow_file(filepath): - """Parse a per-file shadow into structured data. - - Returns a result dict even on partial failure — whatever was parsed - before the error is preserved. Warnings go to stderr. - """ - result = { - "path": str(filepath), - "source_file": None, - "language": None, - "lines": None, - "last_modified": None, - "symbols": [], - "discoveries": [], - "cross_references": [], - "parse_errors": [], - } - - try: - content = filepath.read_text(encoding="utf-8") - except UnicodeDecodeError as e: - msg = (f"Cannot read {filepath}: encoding error at byte " - f"{e.start}: {e.reason}. File may not be UTF-8.") - warn(msg) - result["parse_errors"].append(msg) - return result - except OSError as e: - msg = f"Cannot read {filepath}: {e}" - warn(msg) - result["parse_errors"].append(msg) - return result - - lines = content.split("\n") - current_symbol = None - i = 0 - - while i < len(lines): - line = lines[i] - - try: - # Header: # Shadow: src/auth.py - if line.startswith("# Shadow: "): - result["source_file"] = line[len("# Shadow: "):].strip() - - # Metadata: **Language**: Python | **Lines**: 142 | ... - elif line.startswith("**Language**"): - parts = line.split("|") - for part in parts: - part = part.strip() - if part.startswith("**Language**"): - m = re.search(r"\*\*:\s*(.+)", part) - if m: - result["language"] = m.group(1).strip() - elif "Lines" in part: - m = re.search(r"(\d+)", part) - if m: - try: - result["lines"] = int(m.group(1)) - except ValueError: - pass - elif "Last modified" in part: - m = re.search(r"\*\*:\s*(.+)", part) - if m: - result["last_modified"] = m.group(1).strip() - - # Symbol heading: ## `symbol_name` or ### `Class.method` - elif re.match(r"^#{2,3}\s", line): - sym_match = re.match(r"^(#{2,3})\s+`(.+?)`", line) - if sym_match: - name = sym_match.group(2) - current_symbol = name - result["symbols"].append(name) - elif "Cross-References" in line: - current_symbol = "__cross_refs__" - elif "File-Level" in line: - current_symbol = "__file_level__" - else: - current_symbol = None - - # Discovery bullet (skip cross-reference links) - elif ( - line.strip().startswith("- ") - and current_symbol - and current_symbol != "__cross_refs__" - ): - # Collect continuation lines - continuation = [] - j = i + 1 - while j < len(lines): - next_line = lines[j] - if ( - next_line.strip() == "" - or next_line.strip().startswith("- ") - or re.match(r"^#{1,3}\s", next_line) - ): - break - continuation.append(next_line) - j += 1 - - disc = parse_discovery(line.strip(), continuation) - disc["symbol"] = ( - "file-level" if current_symbol == "__file_level__" - else current_symbol - ) - disc["file"] = result["source_file"] - result["discoveries"].append(disc) - i = j - continue - - # Cross-reference link - elif ( - current_symbol == "__cross_refs__" - and line.strip().startswith("- ") - ): - link_match = re.search(r"\[(.+?)\]", line) - if link_match: - result["cross_references"].append(link_match.group(1)) - - except Exception as e: - msg = (f"Error parsing {filepath} at line {i + 1}: " - f"{type(e).__name__}: {e}") - warn(msg) - result["parse_errors"].append(msg) - - i += 1 - - return result - - -def parse_prefs(shadow_dir): - """Parse _prefs.md into a list of preferences.""" - prefs_path = shadow_dir / "_prefs.md" - if not prefs_path.exists(): - return [] - - try: - content = prefs_path.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - warn(f"Cannot read preferences file {prefs_path}: {e}") - return [] - - prefs = [] - lines = content.split("\n") - i = 0 - while i < len(lines): - line = lines[i] - try: - if line.strip().startswith("- ") and not line.strip().startswith( - "- [" - ): - continuation = [] - j = i + 1 - while j < len(lines): - next_line = lines[j] - if next_line.strip() == "" or next_line.strip().startswith( - "- " - ): - break - continuation.append(next_line) - j += 1 - - pref = parse_discovery(line.strip(), continuation) - pref["type"] = "preference" - prefs.append(pref) - i = j - continue - except Exception as e: - warn(f"Error parsing preference at line {i + 1} in " - f"{prefs_path}: {e}") - i += 1 - - return prefs - - -def parse_cross_cutting(shadow_dir): - """Parse all _cross/*.md files.""" - cross_dir = shadow_dir / "_cross" - if not cross_dir.exists(): - return [] - - entries = [] - try: - md_files = sorted(cross_dir.glob("*.md")) - except OSError as e: - warn(f"Cannot list cross-cutting directory {cross_dir}: {e}") - return [] - - for f in md_files: - try: - content = f.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - warn(f"Cannot read cross-cutting file {f}: {e}") - continue - - entry = {"slug": f.stem, "file": str(f.name)} - - try: - # Title - m = re.search(r"^# (.+)", content, re.MULTILINE) - if m: - entry["title"] = m.group(1).strip() - - # Category - m = re.search(r"\*\*Category\*\*:\s*(.+)", content) - if m: - entry["category"] = m.group(1).strip() - - # Refs — only within the **Refs**: section, not backticked - # bullets elsewhere in the file (e.g., inside the Discovery body). - refs = [] - refs_block = re.search( - r"\*\*Refs\*\*:\s*\n(.*?)(?=\n[ \t]*\n|\n\*\*|\Z)", - content, - re.DOTALL, - ) - if refs_block: - refs = re.findall(r"-\s*`([^`]+)`", refs_block.group(1)) - entry["refs"] = refs - - # Discovery text - m = re.search( - r"\*\*Discovery\*\*:\s*(.+?)(?=\n\n|\n_\(|\Z)", - content, - re.DOTALL, - ) - if m: - entry["discovery"] = m.group(1).strip() - - # Status/source (with optional labels) - m = _DISCOVERY_META_RE.search(content) - if m: - entry["status"] = m.group(1) - entry["source"] = m.group(2) - if m.group(3): - entry["labels"] = [ - l.strip() - for l in m.group(3).split(",") - if l.strip() - ] - else: - # Fallback to simpler pattern - m2 = re.search( - r"_\((\w+),\s*source:\s*(\w+)\)_", content - ) - if m2: - entry["status"] = m2.group(1) - entry["source"] = m2.group(2) - except Exception as e: - warn(f"Error parsing cross-cutting file {f.name}: " - f"{type(e).__name__}: {e}") - entry.setdefault("title", f.stem) - - entries.append(entry) - - return entries - - -def load_state(shadow_dir): - """Load _meta/state.json.""" - state_path = shadow_dir / "_meta" / "state.json" - if not state_path.exists(): - return {} - try: - content = state_path.read_text(encoding="utf-8") - state = json.loads(content) - if not isinstance(state, dict): - warn(f"state.json is not a JSON object (got {type(state).__name__})") - return {} - return state - except json.JSONDecodeError as e: - warn(f"Invalid JSON in {state_path}: {e}") - return {} - except OSError as e: - warn(f"Cannot read {state_path}: {e}") - return {} - - -def get_all_shadow_files(shadow_dir): - """Get all per-file shadow .md files (excluding special files).""" - special = {"_index.md", "_prefs.md"} - special_dirs = {"_cross", "_meta", "_dreams"} - - results = [] - try: - for f in shadow_dir.rglob("*.md"): - try: - rel = f.relative_to(shadow_dir) - parts = rel.parts - if parts[0] in special_dirs: - continue - if str(rel) in special: - continue - results.append(f) - except (ValueError, IndexError) as e: - warn(f"Skipping file {f}: {e}") - except OSError as e: - warn(f"Error walking shadow directory {shadow_dir}: {e}") - - return sorted(results) - - -def collect_all_discoveries(shadow_dir): - """Parse all shadow files and collect every discovery. - - Continues past individual file failures — reports errors and moves on. - """ - all_disc = [] - failed_files = [] - for sf in get_all_shadow_files(shadow_dir): - try: - parsed = parse_shadow_file(sf) - if parsed.get("parse_errors"): - failed_files.append( - (str(sf), parsed["parse_errors"]) - ) - modified = _mtime(sf) - for d in parsed["discoveries"]: - d.setdefault("file", parsed["source_file"]) - d["shadow_path"] = str(sf.relative_to(shadow_dir)) - d["shadow_mtime"] = modified - all_disc.append(d) - except Exception as e: - msg = f"Failed to parse {sf}: {type(e).__name__}: {e}" - warn(msg) - failed_files.append((str(sf), [msg])) - - if failed_files: - warn(f"{len(failed_files)} file(s) had parse errors " - f"(discoveries from other files still collected)") - - return all_disc - - -# --- View Functions --- - -@dataclass(frozen=True) -class RetrievalOptions: - limit: int = 10 - max_chars: int = 4000 - cursor: str | None = None - event_id: str | None = None - record: bool = True - text_cursor: str | None = None - - def __post_init__(self): - if type(self.limit) is not int or self.limit < 1: - raise ValueError("Result limit must be positive") - if type(self.max_chars) is not int or self.max_chars < 0 or (self.max_chars and self.max_chars < 256): - raise ValueError("Output budget must be at least 256 characters, or 0 for no cap") - for name in ("cursor", "text_cursor"): - value = getattr(self, name) - if value is not None and (not isinstance(value, str) or not value): - raise ValueError(f"{name} must be a nonempty returned cursor") - if self.event_id is not None: - validate_event_id(self.event_id) - - -def _knowledge_entry(kind, file, symbol, text, data, refs=(), mtime=0): - symbol = _canonical_symbol(symbol) - anchor = f"{file}::{symbol}" if symbol else file - return { - "id": discovery_id(kind, anchor, text, refs), - "kind": kind, "file": file, "symbol": symbol, "anchor": anchor, - "text": text, "refs": list(refs), "mtime": mtime, - "status": data.get("status", "verified" if kind == "preference" else "?"), - "source": data.get("source", "?"), "labels": sorted(set(data.get("labels", []))), - "title": data.get("title", ""), "category": data.get("category", "?"), - "dream_report": data.get("dream_report", ""), - } - - -def _canonical_symbol(symbol): - # Container headings use the same prefixes as shadow-init.py::Symbol.heading_text. - if symbol in ("File-Level", "file-level"): - return "file-level" - return re.sub(r"^(?:class|interface|enum|trait|struct|protocol|module) ", "", symbol, count=1) - - -def _canonical_source_file(shadow_dir, source_file): - if ( - not source_file or any(char in source_file for char in (":", "\\", "\0", "\n", "\r")) - or any(part in ("", ".", "..") for part in source_file.split("/")) - ): - raise ValueError("Use a repository-relative source path with forward slashes") - root = canonical_path(shadow_dir) - target = canonical_path(root / (source_file + ".md")) - if not target.is_relative_to(root): - raise ValueError("Requested shadow resolves outside --shadow-dir") - return target.relative_to(root).as_posix()[:-3] - - -def _mtime(path): - try: - return path.stat().st_mtime - except OSError as exc: - warn(f"Modification time unavailable for {path}: {exc}; treating it as undated.") - return 0 - - -def _preference_entries(shadow_dir): - prefs = parse_prefs(shadow_dir) - modified = _mtime(shadow_dir / "_prefs.md") if prefs else 0 - return _unique_entries([ - _knowledge_entry( - "preference", "_prefs.md", "", pref.get("text", ""), pref, - mtime=modified, - ) - for pref in prefs - ]) - - -def _unique_entries(entries): - # Duplicate claims at the same location have one identity and one score. - unique = {} - for entry in entries: - if not entry["text"].strip(): - warn(f"Empty discovery at {entry['anchor']}; repair its text before retrieval.") - continue - prior = unique.get(entry["id"]) - if prior is None: - unique[entry["id"]] = entry - else: - strongest = entry if _trust(entry) < _trust(prior) else prior - unique[entry["id"]] = { - **strongest, "labels": sorted(set(entry["labels"]) | set(prior["labels"])), - } - return list(unique.values()) - - -def _knowledge_entries(shadow_dir, source_file=None): - """Collect identities without recording a citation for parsing or matching.""" - entries = [] - files = {} - - def canonical_file(file): - if file not in files: - files[file] = _canonical_source_file(shadow_dir, file) - return files[file] - - def canonical_refs(refs): - normalized = [] - for ref in refs: - file, separator, symbol = ref.partition("::") - if separator: - try: - ref = f"{canonical_file(file)}::{_canonical_symbol(symbol)}" - except (OSError, ValueError, RuntimeError) as exc: - warn(f"Cannot resolve reference {ref!r}: {exc}; inspect and repair that reference.") - normalized.append(ref) - return sorted(set(normalized)) - - if source_file is None: - discoveries = collect_all_discoveries(shadow_dir) - else: - source_file = canonical_file(source_file) - path = shadow_dir / (source_file + ".md") - parsed = parse_shadow_file(path) if path.is_file() else {"discoveries": []} - modified = _mtime(path) if parsed["discoveries"] else 0 - discoveries = [ - {**disc, "shadow_path": source_file + ".md", "shadow_mtime": modified} - for disc in parsed["discoveries"] - ] - for disc in discoveries: - file = canonical_file(Path(disc["shadow_path"]).as_posix()[:-3]) - entries.append(_knowledge_entry( - "discovery", file, disc.get("symbol", "file-level"), disc.get("text", ""), - disc, canonical_refs(disc.get("also_involves", [])), disc.get("shadow_mtime", 0), - )) - for cross in parse_cross_cutting(shadow_dir): - refs = canonical_refs(cross.get("refs", [])) - if source_file is not None and not any(ref.split("::", 1)[0] == source_file for ref in refs): - continue - relative = "_cross/" + cross["file"] - entries.append(_knowledge_entry( - "cross-cutting", relative, "", cross.get("discovery", cross.get("title", "")), - cross, refs, _mtime(shadow_dir / relative), - )) - if source_file is None: - entries.extend(_preference_entries(shadow_dir)) - return _unique_entries(entries) - - -def _trust(entry): - if entry["status"] == "refuted": - return 5 - if entry["source"] == "user": - return 0 - if entry["source"] == "interaction": - return 1 - return {"verified": 2, "uncertain": 3}.get(entry["status"], 4) - - -def _rank_entries(entries, scores, limit, recent=False): - groups = defaultdict(list) - for entry in entries: - priority = ((-entry["mtime"],) if recent else ()) + ( - entry.get("relevance", 0), _trust(entry), - ) - groups[priority].append(entry) - result = [] - for priority in sorted(groups): - group = groups[priority] - seen = sorted( - (entry for entry in group if scores.get(entry["id"], 0)), - key=lambda entry: -scores[entry["id"]], - ) - unseen = [entry for entry in group if not scores.get(entry["id"], 0)] - width = max(1, limit) - while seen and unseen: - remaining = width - len(result) % width - take = min(len(seen), remaining - 1) - result.extend(seen[:take]) - del seen[:take] - result.append(unseen.pop(0)) - result.extend(seen) - result.extend(unseen) - return result - - -def _citation_store(shadow_dir): - try: - return CitationStore.for_shadow(shadow_dir) - except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as exc: - warn(f"Citation tracking unavailable: {exc}. This visit will not be recorded; check local Git/state access.") - return None - - -def _clip(text, limit): - if len(text) <= limit: - return text - return text[:limit] if limit < 3 else text[:limit - 3] + "..." - - -def _preview_parts(entry, score, style, group_count): - text = entry["text"].replace("\n", " ") - anchor = _clip(entry["anchor"], 160) - identity = f"id={entry['id']} citation_score={score}" - metadata = f"({entry['status']}, source: {entry['source']})" - labels = ",".join(entry["labels"]) or "-" - if style == "top": - anchor = entry["symbol"] if entry["kind"] == "discovery" else entry["file"] - prefix = f"- [{_clip(labels, 40)}] `{_clip(anchor, 48)}` ({entry['status']}) id={entry['id']}: " - return prefix, text - if style == "recent": - stamp = datetime.fromtimestamp(entry["mtime"]).strftime("%Y-%m-%d %H:%M") - heading = f" [{stamp}] ({entry['kind']})" - elif entry["kind"] == "cross-cutting": - heading = f"Cross-cutting: {_clip(entry['title'], 120)}\n Category: {entry['category']}" - elif entry["kind"] == "preference": - heading = f"Preferences [{entry['source']}]" - else: - heading = f"{_clip(entry['file'], 160)} ({group_count} matches)" - prefix = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n" - if entry["refs"]: - label = "Refs" if entry["kind"] == "cross-cutting" else "Also involves" - prefix += f" {label}: {_clip(', '.join(entry['refs']), 160)}\n" - if style == "labels" and len(entry["labels"]) > 1: - prefix += f" Also labeled: {labels}\n" - return prefix + " ", text - - -def _read_scores(store, identities): - if store is not None: - try: - return store.scores(identities), True - except (OSError, ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - warn("Citation ledger busy; scores are unknown. Counting will still be attempted after output if enabled.") - else: - warn(f"Cannot read citation scores: {exc}. Scores are unknown; check the local cache.") - return {}, False - - -def _record_visible(store, identities, options, event=None): - if store is not None and identities and options.record: - try: - store.record(identities, options.event_id if event is None else event) - except (OSError, ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - warn("Citation ledger busy; this visit was not recorded (scores were not updated). Retry later with the same --event-id if supplied.") - else: - warn(f"Citation scores were not updated: {exc}. This visit was not recorded; repair local state and retry.") - - -def _catalog_digest(entries, recent): - values = [ - {key: value for key, value in entry.items() if recent or key != "mtime"} - for entry in sorted(entries, key=lambda entry: entry["id"]) - ] - return hashlib.sha256(json.dumps(values, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest() - - -def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): - store = _citation_store(shadow_dir) - scores, scores_known = _read_scores(store, [entry["id"] for entry in entries]) - ranked = _rank_entries(entries, scores, options.limit, recent=style == "recent") - catalog = "" if style == "top" else _catalog_digest(entries, recent=style == "recent") - token = None - offset = 0 - if options.cursor: - match = re.fullmatch(r"([0-9a-f]{32}):(\d+)", options.cursor) - if not match: - raise ValueError("Invalid cursor; copy the --cursor value from the previous response") - if store is None: - raise ValueError("Cannot resume cursor without the local ledger; repair it or restart the query") - token, offset = match.group(1), int(match.group(2)) - ids = store.load_page(token, request, catalog) - by_id = {entry["id"]: entry for entry in entries} - try: - ranked = [by_id[identity] for identity in ids] - except KeyError as exc: - raise ValueError("Invalid local pagination snapshot; restart the query") from exc - if offset >= len(ranked): - raise ValueError("Cursor is past the available results; restart the query") - - counts = defaultdict(int) - for entry in entries: - counts[entry["file"]] += 1 - # Include the terminal newline and continuation instructions in the budget. - cap = options.max_chars - prefix = _clip(header, min(300, cap // 5)) if cap else header - reserve = 90 if style != "top" else 6 - width = min(options.limit, len(ranked) - offset) - while width: - selected = ranked[offset:offset + width] - parts = [ - _preview_parts(entry, scores.get(entry["id"], 0) if scores_known else "?", style, counts[entry["file"]]) - for entry in selected - ] - if cap: - used = len(prefix) + reserve + 3 - fits = 0 - for entry_prefix, text in parts: - used += len(entry_prefix) + min(12, len(text)) + 1 - if used > cap: - break - fits += 1 - if fits < width: - if width > 1: - width = max(1, fits) - if not options.cursor: - ranked = _rank_entries(entries, scores, width, recent=style == "recent") - continue - entry = selected[0] - score = scores.get(entry["id"], 0) if scores_known else "?" - minimal = f"({entry['status']}, source: {entry['source']}) id={entry['id']} citation_score={score}: " - parts = [(minimal, entry["text"])] - break - body_budget = cap - len(prefix) - reserve - 1 - width - sum(len(part[0]) for part in parts) if cap else 180 * width - if width and body_budget < width: - raise ValueError("Output budget cannot fit discovery content; increase the character budget") - snippets = [0] * width - # Distribute spare room across entries rather than dropping a whole warning. - pending = list(range(width)) - while pending and body_budget: - share = max(1, body_budget // len(pending)) - next_pending = [] - for index in pending: - remaining = min(180, len(parts[index][1])) - snippets[index] - take = min(remaining, share, body_budget) - snippets[index] += take - body_budget -= take - if snippets[index] < min(180, len(parts[index][1])): - next_pending.append(index) - pending = next_pending - output = prefix + "".join( - "\n" + entry_prefix + _clip(text, snippets[index]) - for index, (entry_prefix, text) in enumerate(parts) - ) - shown = [entry["id"] for entry in selected] - next_offset = offset + len(shown) - if next_offset < len(ranked): - if style == "top": - output += "\n(...)" - else: - if token is None and store is not None: - try: - token = store.save_page(request, catalog, [entry["id"] for entry in ranked]) - except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Cannot save pagination: {exc}. Repair the local ledger or narrow the query.") - if token: - output += f"\nMore: --cursor {token}:{next_offset} (same view)" - else: - output += "\nMore omitted: narrow the query (local ledger unavailable)." - if style != "top": - output += "\nExpand a claim: --get ID" - output = prefix.replace("{shown}", str(len(shown))) + output[len(prefix):] - if cap and len(output) + 1 > cap: - raise ValueError("Output budget cannot fit retrieval metadata; increase --max-chars") - print(output, flush=True) - _record_visible(store, shown, options) - - -def view_get(shadow_dir, identity, *, options=None): - """Expand a current discovery, chunking long text without changing its ID.""" - options = options or RetrievalOptions() - if not re.fullmatch(r"d_[0-9a-f]{32}", identity): - raise ValueError("--get requires the complete id=d_... value from a retrieval result") - entry = next((entry for entry in _knowledge_entries(shadow_dir) if entry["id"] == identity), None) - if entry is None: - raise ValueError("Discovery ID is absent or its claim changed; search again for its current ID") - store = _citation_store(shadow_dir) - scores, scores_known = _read_scores(store, [identity]) - score = scores.get(identity, 0) if scores_known else "?" - body = entry["anchor"] + "\n\n" + entry["text"] - if entry["labels"]: - body += "\nLabels: " + ", ".join(entry["labels"]) - if entry["kind"] == "cross-cutting": - body += "\nCategory: " + entry["category"] + "\nTitle: " + entry["title"] - if entry["refs"]: - body += "\nRefs: " + ", ".join(entry["refs"]) - if entry["dream_report"]: - body += "\nDream report: " + entry["dream_report"] - revision = hashlib.sha256(json.dumps( - [entry["status"], entry["source"], body], ensure_ascii=False, - ).encode("utf-8")).hexdigest()[:32] - start = 0 - event = options.event_id - expires = int(time.time()) + RECEIPT_TTL - record = options.record - if options.text_cursor: - match = re.fullmatch( - r"([0-9a-f]{32})~(\d{1,12})~([01])~([A-Za-z0-9._:-]{1,128})~(\d+)", - options.text_cursor, - ) - if not match: - raise ValueError("Invalid text cursor; copy the --text-cursor value from the previous expansion") - prior_revision, expiry, recording, prior_event, offset = match.groups() - if prior_revision != revision: - raise ValueError("Discovery body or metadata changed; restart --get without --text-cursor") - if int(expiry) <= time.time(): - raise ValueError("Text cursor expired; restart --get without --text-cursor") - if event is not None and event != prior_event: - raise ValueError("Text cursor has a different --event-id; reuse its original event") - start, expires, event = int(offset), int(expiry), prior_event - record = record and recording == "1" - if start >= len(body): - raise ValueError("Text cursor is past the end of this discovery; restart --get") - header = ( - f"id={identity} citation_score={score}\n" - f"({entry['status']}, source: {entry['source']})\n" - ) - tail = "" - available = len(body) - start - if options.max_chars and len(header) + available + 1 > options.max_chars: - event = event or uuid.uuid4().hex - - def continuation(offset): - token = f"{revision}~{expires}~{int(record)}~{event}~{offset}" - value = f"\nContinue: --get {identity} --text-cursor {token}" - if options.max_chars != 4000: - value += f" --max-chars {options.max_chars}" - return value - - available = options.max_chars - len(header) - len(continuation(len(body))) - 1 - if available < 1: - raise ValueError("Output budget cannot fit continuation metadata; increase --max-chars") - tail = continuation(start + available) - output = header + body[start:start + available] + tail - print(output, flush=True) - if record: - _record_visible(store, [identity], options, event) - - -def view_summary(shadow_dir): - """Overview + detailed statistics. - - Each section is independently wrapped — if label stats fail, you - still get counts and the per-file table. - """ - # Load data (each can fail independently) - state = {} - shadow_files = [] - prefs = [] - cross = [] - - try: - state = load_state(shadow_dir) - except Exception as e: - warn(f"Failed to load state.json: {e}") - - try: - shadow_files = get_all_shadow_files(shadow_dir) - except Exception as e: - warn(f"Failed to list shadow files: {e}") - - try: - prefs = parse_prefs(shadow_dir) - except Exception as e: - warn(f"Failed to parse preferences: {e}") - - try: - cross = parse_cross_cutting(shadow_dir) - except Exception as e: - warn(f"Failed to parse cross-cutting discoveries: {e}") - - # Parse each file once for both stats and discoveries - file_stats = [] - all_disc = [] - total_symbols = 0 - for sf in shadow_files: - try: - parsed = parse_shadow_file(sf) - src = parsed["source_file"] or str(sf.relative_to(shadow_dir)) - n_sym = len(parsed["symbols"]) - n_disc = len(parsed["discoveries"]) - total_symbols += n_sym - file_stats.append((src, n_sym, n_disc)) - for d in parsed["discoveries"]: - d.setdefault("file", parsed["source_file"]) - d["shadow_path"] = str(sf.relative_to(shadow_dir)) - all_disc.append(d) - except Exception as e: - warn(f"Failed to process {sf}: {e}") - file_stats.sort(key=lambda x: x[2], reverse=True) - - # Header counts (always shown) - print("Shadow Knowledge Base Summary") - print("=" * 50) - print(f" Files shadowed: {len(shadow_files)}") - print(f" Symbols tracked: {total_symbols}") - print(f" Discoveries: {len(all_disc)}") - print(f" Preferences: {len(prefs)}") - print(f" Cross-cutting: {len(cross)}") - - # Source breakdown - try: - source_counts = defaultdict(int) - status_counts = defaultdict(int) - for d in all_disc: - source_counts[d.get("source", "unknown")] += 1 - status_counts[d.get("status", "unknown")] += 1 - - if source_counts: - print("\nBy source:") - for src, cnt in sorted(source_counts.items(), key=lambda x: -x[1]): - pct = cnt / len(all_disc) * 100 if all_disc else 0 - bar = "#" * int(pct / 2) - print(f" {src:15s} {cnt:4d} ({pct:5.1f}%) {bar}") - - if status_counts: - print("\nBy status:") - for st, cnt in sorted(status_counts.items(), key=lambda x: -x[1]): - pct = cnt / len(all_disc) * 100 if all_disc else 0 - bar = "#" * int(pct / 2) - print(f" {st:15s} {cnt:4d} ({pct:5.1f}%) {bar}") - except Exception as e: - warn(f"Failed to compute source/status breakdown: {e}") - - # Label breakdown - try: - label_counts = defaultdict(int) - for d in all_disc: - for lbl in d.get("labels", []): - label_counts[lbl] += 1 - if label_counts: - print("\nBy label:") - for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]): - print(f" {lbl:15s} {cnt:4d}") - except Exception as e: - warn(f"Failed to compute label breakdown: {e}") - - # Per-file table - try: - if file_stats: - print(f"\n{'File':<40s} {'Symbols':>8s} {'Disc.':>6s}") - print(f"{'-'*40} {'-'*8} {'-'*6}") - for src, n_sym, n_disc in file_stats[:20]: - print(f"{src:<40s} {n_sym:>8d} {n_disc:>6d}") - if len(file_stats) > 20: - print(f"... and {len(file_stats) - 20} more files") - except Exception as e: - warn(f"Failed to render per-file table: {e}") - - # Cross-cutting titles - try: - if cross: - print(f"\nCross-cutting discoveries:") - for e in cross: - title = e.get("title", e.get("slug", "?")) - cat = e.get("category", "?") - print(f" [{cat}] {title}") - except Exception as e: - warn(f"Failed to render cross-cutting list: {e}") - - # State info - try: - if state: - print(f"\nLast update: {state.get('last_update_at', '?')} " - f"({state.get('last_update_type', '?')})") - print(f"Last commit: {state.get('last_commit', '?')}") - except Exception as e: - warn(f"Failed to render state info: {e}") - - -def view_search(shadow_dir, query, *, options=None): - """Bounded search with stable continuation over matching knowledge identities.""" - options = options or RetrievalOptions() - query_lower = query.lower() - if not query.strip(): - raise ValueError("Search query must be nonempty") - matches = [] - for entry in _knowledge_entries(shadow_dir): - fields = [entry["anchor"], entry["file"], entry["symbol"], entry["text"], - entry["title"], *entry["refs"]] - if any(query_lower in field.lower() for field in fields): - entry["relevance"] = 0 if query_lower in [field.lower() for field in fields[:3]] else 1 - matches.append(entry) - if not matches and not options.cursor: - print(_clip(f"No results for '{query}'.", options.max_chars - 1) if options.max_chars - else f"No results for '{query}'.") - return - _emit_knowledge( - shadow_dir, matches, f"Search: '{query}' ({len(matches)} results)", options, - request=json.dumps(["search", query_lower]), - ) - - -def view_symbol(shadow_dir, anchor, *, options=None): - options = options or RetrievalOptions() - file, separator, symbol = anchor.partition("::") - if not separator or not symbol: - raise ValueError("--symbol requires file::symbol (use File-Level for a file-level section)") - file = _canonical_source_file(shadow_dir, file) - symbol = _canonical_symbol(symbol) - canonical = f"{file}::{symbol}" - matches = [ - entry for entry in _knowledge_entries(shadow_dir, file) - if entry["anchor"] == canonical or canonical in entry["refs"] - ] - if not matches and not options.cursor: - print(_clip(f"No knowledge for '{anchor}'.", options.max_chars - 1) - if options.max_chars else f"No knowledge for '{anchor}'.") - return - _emit_knowledge( - shadow_dir, matches, f"Knowledge for {anchor} ({len(matches)} results)", options, - request=json.dumps(["symbol", canonical]), - ) - - -def view_prefs(shadow_dir, *, options=None): - """Page preferences without letting popular code discoveries hide directives.""" - options = options or RetrievalOptions() - prefs = _preference_entries(shadow_dir) - if not prefs and not options.cursor: - print("No preferences recorded yet.") - return - _emit_knowledge(shadow_dir, prefs, f"Project Preferences ({len(prefs)} total)", options, request="prefs") - - -def view_labels(shadow_dir, label_filter, *, options=None): - """Show discoveries filtered by label(s). - - label_filter can be a single label or comma-separated list. - """ - options = options or RetrievalOptions() - filters = [label.strip().lower() for label in label_filter.split(",") if label.strip()] - if not filters: - raise ValueError("Supply at least one label with --labels") - matching = [ - entry for entry in _knowledge_entries(shadow_dir) - if set(filters) & {label.lower() for label in entry["labels"]} - ] - - if not matching and not options.cursor: - message = f"No discoveries with label(s): {', '.join(filters)}" - print(_clip(message, options.max_chars - 1) if options.max_chars else message) - return - - _emit_knowledge( - shadow_dir, matching, - f"Discoveries with label(s): {', '.join(filters)} ({len(matching)} results)", - options, style="labels", request=json.dumps(["labels", sorted(set(filters))]), - ) - - -def view_recent(shadow_dir, count=10, *, options=None): - """Show the N most recent discoveries (by shadow file mtime). - - Collects all discoveries across all shadow files, cross-cutting entries, - and preferences, sorts by the source file's modification time (most recent - first), and shows the actual discovery content. - Each data source is independent — if cross-cutting fails, per-file - discoveries still appear. - """ - options = options or RetrievalOptions(limit=count) - all_items = _knowledge_entries(shadow_dir) - if not all_items and not options.cursor: - print("No discoveries found.") - return - - _emit_knowledge( - shadow_dir, all_items, f"Most Recent Discoveries (top {count})", - options, style="recent", request="recent", - ) - - -def view_top(shadow_dir, file_path, labels_filter, limit, max_chars, *, options=None): - """Show the top N actionable discoveries for a single source file. - - Designed for the preToolUse hook: concise output suitable for - inlining into additionalContext when the agent is about to mutate a - file. Pulls from both the per-file shadow and any _cross/ entries - whose refs touch this file. - - Trust/status precedes citation score. Output, including the final newline, - is hard-capped; only entries actually emitted are counted. - """ - norm = file_path.strip() - if norm.startswith("./"): - norm = norm[2:] - label_set = {l.strip().lower() for l in labels_filter.split(",") if l.strip()} - options = options or RetrievalOptions(limit=limit, max_chars=max_chars) - candidates = [ - entry for entry in _knowledge_entries(shadow_dir, norm) - if not label_set or label_set & {label.lower() for label in entry["labels"]} - ] - - if not candidates: - labels_disp = ",".join(sorted(label_set)) if label_set else "any" - message = f"No actionable discoveries ({labels_disp}) for {norm}." - print(_clip(message, max_chars - 1) if max_chars else message) - return - _emit_knowledge( - shadow_dir, candidates, - f"Top {{shown}} of {len(candidates)} actionable discoveries for {norm}:", - options, style="top", request=json.dumps(["top", norm, sorted(label_set)]), - ) - - -def view_check_invariants(shadow_dir): - """Walk the shadow knowledge base and report invariant violations. - - Statically-checkable invariants from shadow-frog/SKILL.md: - #3 (partial) Per-file 'Also involves:' uses file::symbol notation - #4 Cross-ref back-pointers match: _cross/.md refs <-> - per-file ## Cross-References - #5 Every ## Cross-References entry has a matching _cross/*.md - - Plus syntactic guards that catch the most common drift: - - Symbol headings use the required backtick form - - Discovery metadata uses valid status enum - - Discovery metadata uses valid source enum - - Discovery labels are from the allowed set - - _cross/ Category field uses a known value - - Invariants #1, #2, #7 are NOT checked (would require source parsing - and semantic match); #6 is filesystem-enforced. - - Exit 0 = clean, 1 = at least one violation. Violations print one per - line in `path:line: kind: message` form so grep/editors can navigate. - """ - VALID_STATUS = {"verified", "uncertain", "refuted"} - VALID_SOURCE = {"exploration", "user", "interaction"} - VALID_LABELS = {"bug", "performance", "security", - "feature-gap", "tech-debt"} - VALID_CATEGORIES = { - "pattern", "behavior", "edge-case", "contract", - "performance", "intent", "warning", "history", "convention", - } - - violations = [] - def v(path, line, kind, msg): - violations.append(f"{path}:{line}: {kind}: {msg}") - - # Pass 1: walk per-file shadows -> collect cross-reference entries - # they declare and validate their internal format. - per_file_xref_targets = {} # rel_shadow_path -> set(slug declared) - cross_dir = shadow_dir / "_cross" - cross_slugs_on_disk = set() - if cross_dir.is_dir(): - try: - cross_slugs_on_disk = {f.stem for f in cross_dir.glob("*.md")} - except OSError as e: - warn(f"Cannot list {cross_dir}: {e}") - - md_heading_re = re.compile(r"^(#{2,3})\s+(.*)$") - backtick_heading_re = re.compile(r"^(#{2,3})\s+`[^`]+`\s*$") - also_involves_re = re.compile(r"^\s*Also involves:\s*(.+)$", re.I) - file_sym_re = re.compile(r"`([^`]+::[^`]+)`") - - for shadow_path in get_all_shadow_files(shadow_dir): - try: - rel = shadow_path.relative_to(shadow_dir) - except ValueError: - continue - try: - text = shadow_path.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - v(rel, 0, "unreadable", str(e)) - continue - - in_cross_refs = False - declared = set() - for ln, raw in enumerate(text.split("\n"), 1): - line = raw.rstrip() - - heading = md_heading_re.match(line) - if heading: - title = heading.group(2).strip() - if title.lower().startswith("cross-references"): - in_cross_refs = True - continue - in_cross_refs = False - # Skip special headings ("File-Level Notes", "Notes", etc.) - if ( - title.lower().startswith("file-level") - or title.lower() in {"notes", "metadata"} - ): - continue - # Symbol heading must use backtick form - if not backtick_heading_re.match(line): - v(rel, ln, "heading", - f"symbol heading must be `## `name`` or " - f"`### `Class.name``; got: {line[:80]}") - continue - - if in_cross_refs and line.strip().startswith("- "): - # Format: - [slug](.shadow/_cross/slug.md) — title - slug_match = re.search( - r"_cross/([^)\s]+?)\.md", line - ) - if slug_match: - declared.add(slug_match.group(1)) - else: - # Looser fallback: bare slug in brackets - alt = re.search(r"\[([^\]]+)\]", line) - if alt: - declared.add(alt.group(1).strip()) - - # Discovery metadata line - md = _DISCOVERY_META_RE.search(line) - if md: - status, source = md.group(1), md.group(2) - labels_raw = md.group(3) or "" - if status not in VALID_STATUS: - v(rel, ln, "enum", - f"status '{status}' not in {sorted(VALID_STATUS)}") - if source not in VALID_SOURCE: - v(rel, ln, "enum", - f"source '{source}' not in {sorted(VALID_SOURCE)}") - for lbl in (l.strip() for l in labels_raw.split(",") if l.strip()): - if lbl not in VALID_LABELS: - v(rel, ln, "enum", - f"label '{lbl}' not in {sorted(VALID_LABELS)}") - - # `Also involves:` must list file::symbol anchors in backticks - ai = also_involves_re.match(line) - if ai: - rest = ai.group(1) - anchors = file_sym_re.findall(rest) - if not anchors: - v(rel, ln, "anchor", - "Also involves: needs `file::symbol` " - "backtick anchors") - # Light sanity: every anchor has both file and symbol - for a in anchors: - if "::" not in a or not a.split("::", 1)[1].strip(): - v(rel, ln, "anchor", - f"anchor '{a}' missing symbol after ::") - - per_file_xref_targets[str(rel)] = declared - - # Invariant #5: every declared cross slug must exist on disk - for slug in declared: - if slug not in cross_slugs_on_disk: - v(rel, 0, "cross-ref", - f"references _cross/{slug}.md but file does not exist") - - # Pass 2: walk _cross/*.md -> validate refs format + back-pointer. - # Build the reverse map: cross_slug -> set(file::symbol it points at). - cross_back = {} # slug -> set(file paths it should be linked from) - if cross_dir.is_dir(): - for cf in sorted(cross_dir.glob("*.md")): - slug = cf.stem - try: - text = cf.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - v(cf.relative_to(shadow_dir), 0, "unreadable", str(e)) - continue - - rel_cf = cf.relative_to(shadow_dir) - - # Category enum check - cat_m = re.search(r"\*\*Category\*\*:\s*(.+)", text) - if cat_m: - cat = cat_m.group(1).strip().lower() - if cat not in VALID_CATEGORIES: - v(rel_cf, 0, "enum", - f"Category '{cat}' not in {sorted(VALID_CATEGORIES)}") - else: - v(rel_cf, 0, "schema", - "missing **Category**: field") - - # Discovery metadata - md = _DISCOVERY_META_RE.search(text) - if md: - status, source = md.group(1), md.group(2) - if status not in VALID_STATUS: - v(rel_cf, 0, "enum", - f"status '{status}' not in {sorted(VALID_STATUS)}") - if source not in VALID_SOURCE: - v(rel_cf, 0, "enum", - f"source '{source}' not in {sorted(VALID_SOURCE)}") - else: - v(rel_cf, 0, "schema", - "missing trailing _(status, source: ...)_ metadata") - - # Refs must be `file::symbol` anchors - refs_block = re.search( - r"\*\*Refs\*\*:\s*\n((?:\s*-\s+`[^`]+`\s*\n?)+)", - text, - ) - if not refs_block: - v(rel_cf, 0, "schema", - "missing **Refs**: block (one per line, " - "`- `file::symbol``)") - else: - anchors = file_sym_re.findall(refs_block.group(1)) - if not anchors: - v(rel_cf, 0, "anchor", - "Refs block has no `file::symbol` entries") - for a in anchors: - if "::" not in a or not a.split("::", 1)[1].strip(): - v(rel_cf, 0, "anchor", - f"ref '{a}' missing symbol after ::") - else: - # Convert file part to shadow path: - # src/foo.py -> src/foo.py.md (relative to shadow_dir) - file_part = a.split("::", 1)[0].strip() - shadow_rel = f"{file_part}.md" - cross_back.setdefault(slug, set()).add(shadow_rel) - - # Invariant #4 back-pointer: every file referenced by a cross slug - # must declare that slug in its ## Cross-References. - for slug, expected_files in cross_back.items(): - for shadow_rel in expected_files: - declared = per_file_xref_targets.get(shadow_rel) - if declared is None: - v(f"_cross/{slug}.md", 0, "cross-ref", - f"refs {shadow_rel} but no such shadow file exists") - elif slug not in declared: - v(f"_cross/{slug}.md", 0, "cross-ref", - f"refs {shadow_rel} but that shadow's ## " - f"Cross-References does not link back to " - f"_cross/{slug}.md") - - # Output - if not violations: - print(f"✓ Invariants OK ({len(per_file_xref_targets)} per-file " - f"shadows, {len(cross_slugs_on_disk)} cross-cutting " - f"discoveries)") - return 0 - - for line in violations: - print(line) - print(f"\n{len(violations)} invariant violation(s) found.", - file=sys.stderr) - return 1 - - def main(): - for _stream in (sys.stdout, sys.stderr): - if hasattr(_stream, "reconfigure"): - _stream.reconfigure(encoding="utf-8") - try: - parser = argparse.ArgumentParser( - description="Query and browse a .shadow/ knowledge base.", - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - - # Views (mutually exclusive) - views = parser.add_mutually_exclusive_group() - views.add_argument( - "--summary", action="store_true", - help="Overview + detailed statistics (default)", - ) - views.add_argument( - "--search", metavar="QUERY", - help="Universal search: files, symbols, and discovery text", - ) - views.add_argument("--symbol", metavar="FILE::SYMBOL", help="Knowledge for an exact symbol and its cross-cutting refs") - views.add_argument("--get", metavar="ID", help="Expand a discovery returned by the viewer") - views.add_argument( - "--prefs", action="store_true", - help="Show project-wide preferences", - ) - views.add_argument( - "--recent", nargs="?", const=10, type=int, metavar="N", - help="N most recent discoveries with content (default: 10)", - ) - views.add_argument( - "--labels", metavar="LABEL", - help=( - "Show discoveries by label " - "(e.g., bug, security, bug,performance)" - ), - ) - views.add_argument( - "--top", metavar="FILE", - help=( - "Top actionable discoveries for FILE (a source path " - "like src/auth.py). Concise output for the preToolUse " - "hook: filters to actionable labels (default: " - "bug,security), includes both per-file and _cross/ " - "entries that reference FILE, ranks verified first." - ), - ) - views.add_argument( - "--check-invariants", action="store_true", - help=( - "Walk the shadow and report structural violations: " - "missing back-pointers, dangling _cross/ refs, invalid " - "enums, bad heading format. Exit 1 if any are found." - ), - ) - - # Options - parser.add_argument( - "--shadow-dir", default=None, - help="Path to .shadow/ directory (default: auto-detect)", - ) - parser.add_argument( - "--top-labels", default="bug,security", metavar="LABELS", - help=( - "Comma-separated labels to include in --top " - "(default: bug,security). Pass empty string to include " - "all labeled discoveries." - ), - ) - parser.add_argument( - "--top-limit", type=int, default=3, metavar="N", - help="Max discoveries to show in --top (default: 3)", - ) - parser.add_argument( - "--top-max-chars", type=int, default=600, metavar="N", - help=( - "Hard cap on --top total output length " - "(default: 600). Use 0 for no cap." - ), - ) - parser.add_argument("--limit", type=int, help="Results per page for search/symbol/labels/prefs (default: 10)") - parser.add_argument("--max-chars", type=int, help="Retrieval output budget (default: 4000; 0 disables cap)") - parser.add_argument("--cursor", help="Continue the same view/filters with a frozen result ordering") - parser.add_argument("--event-id", help="Retry token; an entry counts once per token (default: fresh event)") - parser.add_argument("--no-record", action="store_true", help="Do not increment citation scores") - parser.add_argument("--text-cursor", help="Continue the same logical read of an unchanged --get body") - - args = parser.parse_args() - retrieval = ( - args.top is not None or args.search is not None or args.symbol is not None - or args.get is not None or args.prefs or args.labels is not None or args.recent is not None - ) - if not retrieval and ( - args.limit is not None or args.max_chars is not None or args.cursor - or args.event_id is not None or args.no_record or args.text_cursor is not None - ): - parser.error("Retrieval options require --search, --symbol, --get, --top, --prefs, --labels, or --recent") - if args.text_cursor is not None and args.get is None: - parser.error("--text-cursor requires --get") - if args.cursor and (args.get is not None or args.top is not None): - parser.error("--cursor is for paged search/symbol/labels/prefs/recent; use --text-cursor with --get") - if args.limit is not None and (args.top is not None or args.recent is not None or args.get is not None): - parser.error("Use --top-limit or --recent N instead of --limit; --get returns one discovery") - if args.max_chars is not None and args.top is not None: - parser.error("Use --top-max-chars with --top") - try: - options = RetrievalOptions( - limit=args.top_limit if args.top is not None else ( - args.recent if args.recent is not None else (args.limit if args.limit is not None else 10) - ), - max_chars=args.top_max_chars if args.top is not None else ( - args.max_chars if args.max_chars is not None else 4000 - ), - cursor=args.cursor, event_id=args.event_id, record=not args.no_record, - text_cursor=args.text_cursor, - ) - except ValueError as exc: - parser.error(str(exc)) - - # Find shadow dir - if args.shadow_dir: - shadow_dir = Path(args.shadow_dir) - else: - shadow_dir = find_shadow_dir() - - if not shadow_dir or not shadow_dir.is_dir(): - cwd = os.getcwd() - error( - f"No .shadow/ directory found. " - f"Searched from: {cwd}\n" - f"[shadow-viewer error] " - f"Run /shadow-frog-init first to create the shadow, " - f"or pass --shadow-dir /path/to/.shadow/ explicitly." - ) - if args.shadow_dir: - error( - f"Provided --shadow-dir '{args.shadow_dir}' does not " - f"exist or is not a directory." - ) - sys.exit(1) - - # Dispatch - if args.check_invariants: - sys.exit(view_check_invariants(shadow_dir)) - if args.top is not None: - view_top( - shadow_dir, - args.top, - args.top_labels, - args.top_limit, - args.top_max_chars, - options=options, - ) - elif args.search is not None: - view_search(shadow_dir, args.search, options=options) - elif args.symbol is not None: - view_symbol(shadow_dir, args.symbol, options=options) - elif args.get is not None: - view_get(shadow_dir, args.get, options=options) - elif args.prefs: - view_prefs(shadow_dir, options=options) - elif args.labels is not None: - view_labels(shadow_dir, args.labels, options=options) - elif args.recent is not None: - view_recent(shadow_dir, args.recent, options=options) - else: - view_summary(shadow_dir) - - except SystemExit: - raise - except KeyboardInterrupt: - error("Interrupted by user.") - sys.exit(130) - except (ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - error("Local citation ledger is busy; retry the same command later. Do not reset a busy database.") - else: - error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") - sys.exit(1) - except Exception as e: - error( - f"Unexpected error: {type(e).__name__}: {e}\n" - f"[shadow-viewer error] Full traceback:\n" - f"{traceback.format_exc()}" - f"This is likely a bug in shadow-viewer.py. " - f"The shadow data may be in an unexpected format. " - f"Try running with --shadow-dir to confirm the path, " - f"or inspect the .shadow/ files manually." - ) - sys.exit(1) + return _knowledge.main() if __name__ == "__main__": - main() + sys.exit(main()) diff --git a/skills/shadow-frog/SKILL.md b/skills/shadow-frog/SKILL.md index 96f5dc4..e8cb8b8 100644 --- a/skills/shadow-frog/SKILL.md +++ b/skills/shadow-frog/SKILL.md @@ -10,7 +10,9 @@ description: >- to create it, shadow-frog-update to refresh it, shadow-frog-dream for autonomous experiments, shadow-frog-nap for lightweight feature-task ideation, shadow-frog-meditate for shadow hygiene, - or shadow-frog-viewer to browse it. + or shadow-frog-viewer for user-facing browsing and visualization. +scripts: + - shadow-read.py --- # ShadowFrog @@ -23,18 +25,18 @@ to that code location. **Every time you work on code in a repo with `.shadow/`:** -1. **Read preferences first** with the viewer's `--prefs`, following all pages. - `_prefs.md` contains project-wide conventions, user preferences, and things to avoid. +1. **Read `.shadow/_prefs.md` first** — it contains project-wide conventions, + user preferences, and things to avoid. 2. **Read relevant `_cross/` discoveries** — list `_cross/` and read entries whose titles relate to the current area, including cross-file contracts and interactions. 3. **Check `_dreams/_index.md`** and read relevant experiment reports, especially when investigating bugs or unfamiliar code. They may contain findings not yet distilled into per-file shadows. -4. **Before editing a file**, query its shadow (`.shadow/.md`) with - `--search FILE`; use `--symbol file::symbol` for focused follow-up, including - file-level and related `_cross/` knowledge. Expand entries and follow pages - as needed. `--top` supplies compact hints, not a complete file review. +4. **Before editing a file**, navigate directly to `.shadow/.md` and its + symbol headings, then follow relevant `_cross/` back-pointers. Use native + file reads/searches; for unusually large sections, the optional core helper + below can return bounded selections. A shortlist is not a complete file review. `_index.md` counts may be stale; inspect the actual shadows and `_cross/`. 5. **When the user explains something about code** (gotcha, design intent, warning, history): write a `source: user` discovery to the shadow @@ -44,25 +46,37 @@ to that code location. specific file): write it to `_prefs.md` immediately. 7. **After code changes**: run `/shadow-frog-update` -## Bounded Knowledge Retrieval - -Prefer the `/shadow-frog-viewer` helper over loading an entire large shadow. -It returns short previews, discovery IDs, and one `citation_score`; `--get ID` -expands an entry, and returned cursors continue a stable result ordering or -revision-bound logical read. Compact `--top` output omits the numeric score. -Citation scores count content emitted by the helper, not proven use in reasoning. -Do not manually increment scores or put them into Markdown: the helper records -visits atomically in local state shared across worktrees. New discoveries from -Dream, Update, conversation capture, or manual writes implicitly start at zero. - -Use relevance and trust first, then citation history; a popular claim is not -automatically correct. Never omit relevant user constraints because of a low -score. A shortlist is not a complete symbol history. Follow all preference pages -and use targeted searches/pagination when completeness matters. -If retrieval warns that telemetry failed, knowledge remains usable but the -count may be missing. Reuse `--event-id` for retries; `--no-record` is available -for inspection that should not affect ranking. Raw reads remain possible but -are not counted. See the Viewer skill for identity and pagination semantics. +## Optional Agent Retrieval + +**File/symbol navigation is primary.** The source path already locates its +shadow; no viewer, retrieval service, or citation database is required to read it. +`/shadow-frog-viewer` is the user-facing browsing/visualization skill, not the +agent's required knowledge interface. + +For large files/symbol sections or a targeted search, use `shadow-read.py` +beside this skill. Known paths are read directly, not rediscovered by global +search. Examples below use the Copilot install; Claude Code uses `.claude/skills/`. + +```text +python .github/skills/shadow-frog/shadow-read.py src/auth.py +python .github/skills/shadow-frog/shadow-read.py src/auth.py::UserAuth.validate --limit 5 +python .github/skills/shadow-frog/shadow-read.py --search "token expiry" --max-chars 1800 +``` + +The optional helper can rank/paginate large sections and expand returned IDs. +Keep file/symbol anchors as the navigation address; IDs only identify individual +entries within this helper. Full-file reads include file-level discoveries and +cross-cutting refs; `--top` is only a compact hint. + +Native file reads remain normal and uncounted. Helper-emitted entries increment +one local `citation_score` atomically; do not edit counters in Markdown or run a +second retrieval just to inflate them. Scores measure exposure, not correctness +or proven usefulness. Never omit user constraints because they are unpopular. +Use targeted searches and follow pages when completeness matters. + +Read [the helper reference](retrieval.md) when using its budgets, continuation, +or citation options. If optional telemetry fails, keep using the knowledge and +heed the diagnostic; do not substitute score availability for reference integrity. ## Directory Layout @@ -295,14 +309,14 @@ Links 4 and 5 are bidirectional: if `_cross/db-connection-lifecycle.md` referenc 7. No duplicate discoveries (same behavioral claim at same symbol) To audit a shadow for structural drift (invariant 3 format, invariants 4–5, -plus enum and heading-format guards), locate the viewer script and run it: +plus enum and heading-format guards), the optional core helper also supports: ```bash -VIEWER="" -for DIR in .github/skills/shadow-frog-viewer .claude/skills/shadow-frog-viewer; do - [ -f "$DIR/shadow-viewer.py" ] && VIEWER="$DIR/shadow-viewer.py" && break +READER="" +for DIR in .github/skills/shadow-frog .claude/skills/shadow-frog; do + [ -f "$DIR/shadow-read.py" ] && READER="$DIR/shadow-read.py" && break done -python3 "$VIEWER" --check-invariants +python3 "$READER" --check-invariants ``` Exits 0 if clean, 1 with one violation per line otherwise. Invariant 3 is @@ -358,8 +372,9 @@ types, or static properties. Before writing any discovery, follow this procedure: -1. **Read before write**: Query the target `file::symbol` and search for the - specific claim, expanding candidates and following pages when needed. +1. **Read before write**: Inspect the target `file::symbol` in its shadow and + search for the specific claim. For a large section, use the optional bounded + helper, expanding candidates and following pages when needed. Do not infer absence from a citation-ranked shortlist. If an existing discovery makes the same behavioral claim (even if worded differently) → update the existing one. If the new one extends an existing one → merge into a single richer entry. @@ -435,4 +450,4 @@ semantic truth. Approval is planning confidence, not execution proof. - `/shadow-frog-dream` — autonomous exploration and experimentation while user is AFK - `/shadow-frog-nap` — implementation-free, source-grounded feature-task ideation within a work budget - `/shadow-frog-meditate` — deduplicate, merge, and resolve conflicting discoveries -- `/shadow-frog-viewer` — browse and query the shadow (overview, search, preferences, recent) +- `/shadow-frog-viewer` — user-facing CLI browsing and lineage visualization diff --git a/skills/shadow-frog-viewer/_citations.py b/skills/shadow-frog/_citations.py similarity index 99% rename from skills/shadow-frog-viewer/_citations.py rename to skills/shadow-frog/_citations.py index c6068f9..faa3e76 100644 --- a/skills/shadow-frog-viewer/_citations.py +++ b/skills/shadow-frog/_citations.py @@ -1,4 +1,4 @@ -"""Local citation scores; Markdown remains the authoritative knowledge store.""" +"""Shared local citation scores; Markdown remains the authoritative knowledge store.""" from contextlib import contextmanager from dataclasses import dataclass diff --git a/skills/shadow-frog/_knowledge.py b/skills/shadow-frog/_knowledge.py new file mode 100644 index 0000000..40cb5c8 --- /dev/null +++ b/skills/shadow-frog/_knowledge.py @@ -0,0 +1,1613 @@ +#!/usr/bin/env python3 +"""Shared Markdown parsing, bounded retrieval, and human-facing shadow views. + +The core shadow-read.py helper and user-facing shadow-viewer.py use this +implementation without changing source-to-shadow paths or discovery syntax. +Parsing and structural inspection alone never increment citation scores. +""" + +from __future__ import annotations + +import argparse +from dataclasses import dataclass +import hashlib +import json +import os +import re +import sys +import sqlite3 +import subprocess +import time +import traceback +import uuid +from collections import defaultdict +from datetime import datetime +from pathlib import Path + + +sys.path.insert(0, str(Path(__file__).resolve().parent)) +_bytecode = sys.dont_write_bytecode +sys.dont_write_bytecode = True +try: + from _citations import ( + CitationStore, RECEIPT_TTL, canonical_path, discovery_id, is_busy_error, + validate_event_id, + ) +except ImportError as exc: + raise SystemExit("[shadow error] Missing citation helper/dependency; reinstall the complete core skill") from exc +finally: + sys.path.pop(0) + sys.dont_write_bytecode = _bytecode + + +_DISCOVERY_META_RE = re.compile( + r"_\((\w+),\s*source:\s*(\w+)" + r"(?:,\s*labels:\s*\[([^\]]*)\])?" + r"\)_" +) + + +def warn(msg): + """Print a warning to stderr. Agents read these to adjust strategy.""" + print(f"[shadow warning] {msg}", file=sys.stderr) + + +def error(msg): + """Print an error to stderr.""" + print(f"[shadow error] {msg}", file=sys.stderr) + + +def find_shadow_dir(start="."): + """Walk up from start to find .shadow/ directory.""" + try: + p = Path(start).resolve() + while p != p.parent: + candidate = p / ".shadow" + if candidate.is_dir(): + return candidate + p = p.parent + except (OSError, PermissionError) as e: + error(f"Failed to search for .shadow/ directory from '{start}': {e}") + return None + + +def parse_discovery(line, continuation_lines=None): + """Parse a discovery bullet and its metadata line(s).""" + try: + text = line[2:].strip() if line.startswith("- ") else line.strip() + except (TypeError, AttributeError) as e: + warn(f"parse_discovery: bad line input ({type(line).__name__}): {e}") + return {"text": str(line) if line else ""} + + meta = {} + full_text = text + + if continuation_lines: + for cl in continuation_lines: + try: + stripped = cl.strip() + # _(status, source: type, labels: [l1, l2])_ or _(status, source: type)_ + m = _DISCOVERY_META_RE.match(stripped) + if m: + meta["status"] = m.group(1) + meta["source"] = m.group(2) + if m.group(3): + meta["labels"] = [ + l.strip() + for l in m.group(3).split(",") + if l.strip() + ] + else: + # _(source: type)_ (preferences format) + m2 = re.match(r"_\(source:\s*(\w+)\)_", stripped) + if m2: + meta["source"] = m2.group(1) + elif stripped.startswith("Also involves:"): + refs = re.findall(r"`([^`]+)`", stripped) + meta["also_involves"] = refs + elif stripped.startswith("Dream report:"): + m_dr = re.search(r"`([^`]+)`", stripped) + if m_dr: + meta["dream_report"] = m_dr.group(1) + else: + full_text += " " + stripped + except Exception as e: + warn(f"parse_discovery: failed parsing continuation line " + f"'{cl[:80]}': {e}") + + return {"text": full_text, **meta} + + +def parse_shadow_file(filepath): + """Parse a per-file shadow into structured data. + + Returns a result dict even on partial failure — whatever was parsed + before the error is preserved. Warnings go to stderr. + """ + result = { + "path": str(filepath), + "source_file": None, + "language": None, + "lines": None, + "last_modified": None, + "symbols": [], + "discoveries": [], + "cross_references": [], + "parse_errors": [], + } + + try: + content = filepath.read_text(encoding="utf-8") + except UnicodeDecodeError as e: + msg = (f"Cannot read {filepath}: encoding error at byte " + f"{e.start}: {e.reason}. File may not be UTF-8.") + warn(msg) + result["parse_errors"].append(msg) + return result + except OSError as e: + msg = f"Cannot read {filepath}: {e}" + warn(msg) + result["parse_errors"].append(msg) + return result + + lines = content.split("\n") + current_symbol = None + i = 0 + + while i < len(lines): + line = lines[i] + + try: + # Header: # Shadow: src/auth.py + if line.startswith("# Shadow: "): + result["source_file"] = line[len("# Shadow: "):].strip() + + # Metadata: **Language**: Python | **Lines**: 142 | ... + elif line.startswith("**Language**"): + parts = line.split("|") + for part in parts: + part = part.strip() + if part.startswith("**Language**"): + m = re.search(r"\*\*:\s*(.+)", part) + if m: + result["language"] = m.group(1).strip() + elif "Lines" in part: + m = re.search(r"(\d+)", part) + if m: + try: + result["lines"] = int(m.group(1)) + except ValueError: + pass + elif "Last modified" in part: + m = re.search(r"\*\*:\s*(.+)", part) + if m: + result["last_modified"] = m.group(1).strip() + + # Symbol heading: ## `symbol_name` or ### `Class.method` + elif re.match(r"^#{2,3}\s", line): + sym_match = re.match(r"^(#{2,3})\s+`(.+?)`", line) + if sym_match: + name = sym_match.group(2) + current_symbol = name + result["symbols"].append(name) + elif "Cross-References" in line: + current_symbol = "__cross_refs__" + elif "File-Level" in line: + current_symbol = "__file_level__" + else: + current_symbol = None + + # Discovery bullet (skip cross-reference links) + elif ( + line.strip().startswith("- ") + and current_symbol + and current_symbol != "__cross_refs__" + ): + # Collect continuation lines + continuation = [] + j = i + 1 + while j < len(lines): + next_line = lines[j] + if ( + next_line.strip() == "" + or next_line.strip().startswith("- ") + or re.match(r"^#{1,3}\s", next_line) + ): + break + continuation.append(next_line) + j += 1 + + disc = parse_discovery(line.strip(), continuation) + disc["symbol"] = ( + "file-level" if current_symbol == "__file_level__" + else current_symbol + ) + disc["file"] = result["source_file"] + result["discoveries"].append(disc) + i = j + continue + + # Cross-reference link + elif ( + current_symbol == "__cross_refs__" + and line.strip().startswith("- ") + ): + link_match = re.search(r"\[(.+?)\]", line) + if link_match: + result["cross_references"].append(link_match.group(1)) + + except Exception as e: + msg = (f"Error parsing {filepath} at line {i + 1}: " + f"{type(e).__name__}: {e}") + warn(msg) + result["parse_errors"].append(msg) + + i += 1 + + return result + + +def parse_prefs(shadow_dir): + """Parse _prefs.md into a list of preferences.""" + prefs_path = shadow_dir / "_prefs.md" + if not prefs_path.exists(): + return [] + + try: + content = prefs_path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + warn(f"Cannot read preferences file {prefs_path}: {e}") + return [] + + prefs = [] + lines = content.split("\n") + i = 0 + while i < len(lines): + line = lines[i] + try: + if line.strip().startswith("- ") and not line.strip().startswith( + "- [" + ): + continuation = [] + j = i + 1 + while j < len(lines): + next_line = lines[j] + if next_line.strip() == "" or next_line.strip().startswith( + "- " + ): + break + continuation.append(next_line) + j += 1 + + pref = parse_discovery(line.strip(), continuation) + pref["type"] = "preference" + prefs.append(pref) + i = j + continue + except Exception as e: + warn(f"Error parsing preference at line {i + 1} in " + f"{prefs_path}: {e}") + i += 1 + + return prefs + + +def parse_cross_cutting(shadow_dir): + """Parse all _cross/*.md files.""" + cross_dir = shadow_dir / "_cross" + if not cross_dir.exists(): + return [] + + entries = [] + try: + md_files = sorted(cross_dir.glob("*.md")) + except OSError as e: + warn(f"Cannot list cross-cutting directory {cross_dir}: {e}") + return [] + + for f in md_files: + try: + content = f.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + warn(f"Cannot read cross-cutting file {f}: {e}") + continue + + entry = {"slug": f.stem, "file": str(f.name)} + + try: + # Title + m = re.search(r"^# (.+)", content, re.MULTILINE) + if m: + entry["title"] = m.group(1).strip() + + # Category + m = re.search(r"\*\*Category\*\*:\s*(.+)", content) + if m: + entry["category"] = m.group(1).strip() + + # Refs — only within the **Refs**: section, not backticked + # bullets elsewhere in the file (e.g., inside the Discovery body). + refs = [] + refs_block = re.search( + r"\*\*Refs\*\*:\s*\n(.*?)(?=\n[ \t]*\n|\n\*\*|\Z)", + content, + re.DOTALL, + ) + if refs_block: + refs = re.findall(r"-\s*`([^`]+)`", refs_block.group(1)) + entry["refs"] = refs + + # Discovery text + m = re.search( + r"\*\*Discovery\*\*:\s*(.+?)(?=\n\n|\n_\(|\Z)", + content, + re.DOTALL, + ) + if m: + entry["discovery"] = m.group(1).strip() + + # Status/source (with optional labels) + m = _DISCOVERY_META_RE.search(content) + if m: + entry["status"] = m.group(1) + entry["source"] = m.group(2) + if m.group(3): + entry["labels"] = [ + l.strip() + for l in m.group(3).split(",") + if l.strip() + ] + else: + # Fallback to simpler pattern + m2 = re.search( + r"_\((\w+),\s*source:\s*(\w+)\)_", content + ) + if m2: + entry["status"] = m2.group(1) + entry["source"] = m2.group(2) + except Exception as e: + warn(f"Error parsing cross-cutting file {f.name}: " + f"{type(e).__name__}: {e}") + entry.setdefault("title", f.stem) + + entries.append(entry) + + return entries + + +def load_state(shadow_dir): + """Load _meta/state.json.""" + state_path = shadow_dir / "_meta" / "state.json" + if not state_path.exists(): + return {} + try: + content = state_path.read_text(encoding="utf-8") + state = json.loads(content) + if not isinstance(state, dict): + warn(f"state.json is not a JSON object (got {type(state).__name__})") + return {} + return state + except json.JSONDecodeError as e: + warn(f"Invalid JSON in {state_path}: {e}") + return {} + except OSError as e: + warn(f"Cannot read {state_path}: {e}") + return {} + + +def get_all_shadow_files(shadow_dir): + """Get all per-file shadow .md files (excluding special files).""" + special = {"_index.md", "_prefs.md"} + special_dirs = {"_cross", "_meta", "_dreams"} + + results = [] + try: + for f in shadow_dir.rglob("*.md"): + try: + rel = f.relative_to(shadow_dir) + parts = rel.parts + if parts[0] in special_dirs: + continue + if str(rel) in special: + continue + results.append(f) + except (ValueError, IndexError) as e: + warn(f"Skipping file {f}: {e}") + except OSError as e: + warn(f"Error walking shadow directory {shadow_dir}: {e}") + + return sorted(results) + + +def collect_all_discoveries(shadow_dir): + """Parse all shadow files and collect every discovery. + + Continues past individual file failures — reports errors and moves on. + """ + all_disc = [] + failed_files = [] + for sf in get_all_shadow_files(shadow_dir): + try: + parsed = parse_shadow_file(sf) + if parsed.get("parse_errors"): + failed_files.append( + (str(sf), parsed["parse_errors"]) + ) + modified = _mtime(sf) + for d in parsed["discoveries"]: + d.setdefault("file", parsed["source_file"]) + d["shadow_path"] = str(sf.relative_to(shadow_dir)) + d["shadow_mtime"] = modified + all_disc.append(d) + except Exception as e: + msg = f"Failed to parse {sf}: {type(e).__name__}: {e}" + warn(msg) + failed_files.append((str(sf), [msg])) + + if failed_files: + warn(f"{len(failed_files)} file(s) had parse errors " + f"(discoveries from other files still collected)") + + return all_disc + + +# --- View Functions --- + +@dataclass(frozen=True) +class RetrievalOptions: + limit: int = 10 + max_chars: int = 4000 + cursor: str | None = None + event_id: str | None = None + record: bool = True + text_cursor: str | None = None + + def __post_init__(self): + if type(self.limit) is not int or self.limit < 1: + raise ValueError("Result limit must be positive") + if type(self.max_chars) is not int or self.max_chars < 0 or (self.max_chars and self.max_chars < 256): + raise ValueError("Output budget must be at least 256 characters, or 0 for no cap") + for name in ("cursor", "text_cursor"): + value = getattr(self, name) + if value is not None and (not isinstance(value, str) or not value): + raise ValueError(f"{name} must be a nonempty returned cursor") + if self.event_id is not None: + validate_event_id(self.event_id) + + +def _knowledge_entry(kind, file, symbol, text, data, refs=(), mtime=0): + symbol = _canonical_symbol(symbol) + anchor = f"{file}::{symbol}" if symbol else file + return { + "id": discovery_id(kind, anchor, text, refs), + "kind": kind, "file": file, "symbol": symbol, "anchor": anchor, + "text": text, "refs": list(refs), "mtime": mtime, + "status": data.get("status", "verified" if kind == "preference" else "?"), + "source": data.get("source", "?"), "labels": sorted(set(data.get("labels", []))), + "title": data.get("title", ""), "category": data.get("category", "?"), + "dream_report": data.get("dream_report", ""), + } + + +def _canonical_symbol(symbol): + # Container headings use the same prefixes as shadow-init.py::Symbol.heading_text. + if symbol in ("File-Level", "file-level"): + return "file-level" + return re.sub(r"^(?:class|interface|enum|trait|struct|protocol|module) ", "", symbol, count=1) + + +def _canonical_source_file(shadow_dir, source_file): + if ( + not source_file or any(char in source_file for char in (":", "\\", "\0", "\n", "\r")) + or any(part in ("", ".", "..") for part in source_file.split("/")) + ): + raise ValueError("Use a repository-relative source path with forward slashes") + root = canonical_path(shadow_dir) + target = canonical_path(root / (source_file + ".md")) + if not target.is_relative_to(root): + raise ValueError("Requested shadow resolves outside --shadow-dir") + return target.relative_to(root).as_posix()[:-3] + + +def _mtime(path): + try: + return path.stat().st_mtime + except OSError as exc: + warn(f"Modification time unavailable for {path}: {exc}; treating it as undated.") + return 0 + + +def _preference_entries(shadow_dir): + prefs = parse_prefs(shadow_dir) + modified = _mtime(shadow_dir / "_prefs.md") if prefs else 0 + return _unique_entries([ + _knowledge_entry( + "preference", "_prefs.md", "", pref.get("text", ""), pref, + mtime=modified, + ) + for pref in prefs + ]) + + +def _unique_entries(entries): + # Duplicate claims at the same location have one identity and one score. + unique = {} + for entry in entries: + if not entry["text"].strip(): + warn(f"Empty discovery at {entry['anchor']}; repair its text before retrieval.") + continue + prior = unique.get(entry["id"]) + if prior is None: + unique[entry["id"]] = entry + else: + strongest = entry if _trust(entry) < _trust(prior) else prior + unique[entry["id"]] = { + **strongest, "labels": sorted(set(entry["labels"]) | set(prior["labels"])), + } + return list(unique.values()) + + +def _knowledge_entries(shadow_dir, source_file=None): + """Collect identities without recording a citation for parsing or matching.""" + entries = [] + files = {} + + def canonical_file(file): + if file not in files: + files[file] = _canonical_source_file(shadow_dir, file) + return files[file] + + def canonical_refs(refs): + normalized = [] + for ref in refs: + file, separator, symbol = ref.partition("::") + if separator: + try: + ref = f"{canonical_file(file)}::{_canonical_symbol(symbol)}" + except (OSError, ValueError, RuntimeError) as exc: + warn(f"Cannot resolve reference {ref!r}: {exc}; inspect and repair that reference.") + normalized.append(ref) + return sorted(set(normalized)) + + if source_file is None: + discoveries = collect_all_discoveries(shadow_dir) + else: + source_file = canonical_file(source_file) + path = shadow_dir / (source_file + ".md") + parsed = parse_shadow_file(path) if path.is_file() else {"discoveries": []} + modified = _mtime(path) if parsed["discoveries"] else 0 + discoveries = [ + {**disc, "shadow_path": source_file + ".md", "shadow_mtime": modified} + for disc in parsed["discoveries"] + ] + for disc in discoveries: + file = canonical_file(Path(disc["shadow_path"]).as_posix()[:-3]) + entries.append(_knowledge_entry( + "discovery", file, disc.get("symbol", "file-level"), disc.get("text", ""), + disc, canonical_refs(disc.get("also_involves", [])), disc.get("shadow_mtime", 0), + )) + for cross in parse_cross_cutting(shadow_dir): + refs = canonical_refs(cross.get("refs", [])) + if source_file is not None and not any(ref.split("::", 1)[0] == source_file for ref in refs): + continue + relative = "_cross/" + cross["file"] + entries.append(_knowledge_entry( + "cross-cutting", relative, "", cross.get("discovery", cross.get("title", "")), + cross, refs, _mtime(shadow_dir / relative), + )) + if source_file is None: + entries.extend(_preference_entries(shadow_dir)) + return _unique_entries(entries) + + +def _trust(entry): + if entry["status"] == "refuted": + return 5 + if entry["source"] == "user": + return 0 + if entry["source"] == "interaction": + return 1 + return {"verified": 2, "uncertain": 3}.get(entry["status"], 4) + + +def _rank_entries(entries, scores, limit, recent=False): + groups = defaultdict(list) + for entry in entries: + priority = ((-entry["mtime"],) if recent else ()) + ( + entry.get("relevance", 0), _trust(entry), + ) + groups[priority].append(entry) + result = [] + for priority in sorted(groups): + group = groups[priority] + seen = sorted( + (entry for entry in group if scores.get(entry["id"], 0)), + key=lambda entry: -scores[entry["id"]], + ) + unseen = [entry for entry in group if not scores.get(entry["id"], 0)] + width = max(1, limit) + while seen and unseen: + remaining = width - len(result) % width + take = min(len(seen), remaining - 1) + result.extend(seen[:take]) + del seen[:take] + result.append(unseen.pop(0)) + result.extend(seen) + result.extend(unseen) + return result + + +def _citation_store(shadow_dir): + try: + return CitationStore.for_shadow(shadow_dir) + except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as exc: + warn(f"Citation tracking unavailable: {exc}. This visit will not be recorded; check local Git/state access.") + return None + + +def _clip(text, limit): + if len(text) <= limit: + return text + return text[:limit] if limit < 3 else text[:limit - 3] + "..." + + +def _preview_parts(entry, score, style, group_count): + text = entry["text"].replace("\n", " ") + anchor = _clip(entry["anchor"], 160) + identity = f"id={entry['id']} citation_score={score}" + metadata = f"({entry['status']}, source: {entry['source']})" + labels = ",".join(entry["labels"]) or "-" + if style == "top": + anchor = entry["symbol"] if entry["kind"] == "discovery" else entry["file"] + prefix = f"- [{_clip(labels, 40)}] `{_clip(anchor, 48)}` ({entry['status']}) id={entry['id']}: " + return prefix, text + if style == "recent": + stamp = datetime.fromtimestamp(entry["mtime"]).strftime("%Y-%m-%d %H:%M") + heading = f" [{stamp}] ({entry['kind']})" + elif entry["kind"] == "cross-cutting": + heading = f"Cross-cutting: {_clip(entry['title'], 120)}\n Category: {entry['category']}" + elif entry["kind"] == "preference": + heading = f"Preferences [{entry['source']}]" + else: + heading = f"{_clip(entry['file'], 160)} ({group_count} matches)" + prefix = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n" + if entry["refs"]: + label = "Refs" if entry["kind"] == "cross-cutting" else "Also involves" + prefix += f" {label}: {_clip(', '.join(entry['refs']), 160)}\n" + if style == "labels" and len(entry["labels"]) > 1: + prefix += f" Also labeled: {labels}\n" + return prefix + " ", text + + +def _read_scores(store, identities): + if store is not None: + try: + return store.scores(identities), True + except (OSError, ValueError, sqlite3.Error) as exc: + if is_busy_error(exc): + warn("Citation ledger busy; scores are unknown. Counting will still be attempted after output if enabled.") + else: + warn(f"Cannot read citation scores: {exc}. Scores are unknown; check the local cache.") + return {}, False + + +def _record_visible(store, identities, options, event=None): + if store is not None and identities and options.record: + try: + store.record(identities, options.event_id if event is None else event) + except (OSError, ValueError, sqlite3.Error) as exc: + if is_busy_error(exc): + warn("Citation ledger busy; this visit was not recorded (scores were not updated). Retry later with the same --event-id if supplied.") + else: + warn(f"Citation scores were not updated: {exc}. This visit was not recorded; repair local state and retry.") + + +def _catalog_digest(entries, recent): + values = [ + {key: value for key, value in entry.items() if recent or key != "mtime"} + for entry in sorted(entries, key=lambda entry: entry["id"]) + ] + return hashlib.sha256(json.dumps(values, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest() + + +def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): + store = _citation_store(shadow_dir) + scores, scores_known = _read_scores(store, [entry["id"] for entry in entries]) + ranked = _rank_entries(entries, scores, options.limit, recent=style == "recent") + catalog = "" if style == "top" else _catalog_digest(entries, recent=style == "recent") + token = None + offset = 0 + if options.cursor: + match = re.fullmatch(r"([0-9a-f]{32}):(\d+)", options.cursor) + if not match: + raise ValueError("Invalid cursor; copy the --cursor value from the previous response") + if store is None: + raise ValueError("Cannot resume cursor without the local ledger; repair it or restart the query") + token, offset = match.group(1), int(match.group(2)) + ids = store.load_page(token, request, catalog) + by_id = {entry["id"]: entry for entry in entries} + try: + ranked = [by_id[identity] for identity in ids] + except KeyError as exc: + raise ValueError("Invalid local pagination snapshot; restart the query") from exc + if offset >= len(ranked): + raise ValueError("Cursor is past the available results; restart the query") + + counts = defaultdict(int) + for entry in entries: + counts[entry["file"]] += 1 + # Include the terminal newline and continuation instructions in the budget. + cap = options.max_chars + prefix = _clip(header, min(300, cap // 5)) if cap else header + reserve = 90 if style != "top" else 6 + width = min(options.limit, len(ranked) - offset) + while width: + selected = ranked[offset:offset + width] + parts = [ + _preview_parts(entry, scores.get(entry["id"], 0) if scores_known else "?", style, counts[entry["file"]]) + for entry in selected + ] + if cap: + used = len(prefix) + reserve + 3 + fits = 0 + for entry_prefix, text in parts: + used += len(entry_prefix) + min(12, len(text)) + 1 + if used > cap: + break + fits += 1 + if fits < width: + if width > 1: + width = max(1, fits) + if not options.cursor: + ranked = _rank_entries(entries, scores, width, recent=style == "recent") + continue + entry = selected[0] + score = scores.get(entry["id"], 0) if scores_known else "?" + minimal = f"({entry['status']}, source: {entry['source']}) id={entry['id']} citation_score={score}: " + parts = [(minimal, entry["text"])] + break + body_budget = cap - len(prefix) - reserve - 1 - width - sum(len(part[0]) for part in parts) if cap else 180 * width + if width and body_budget < width: + raise ValueError("Output budget cannot fit discovery content; increase the character budget") + snippets = [0] * width + # Distribute spare room across entries rather than dropping a whole warning. + pending = list(range(width)) + while pending and body_budget: + share = max(1, body_budget // len(pending)) + next_pending = [] + for index in pending: + remaining = min(180, len(parts[index][1])) - snippets[index] + take = min(remaining, share, body_budget) + snippets[index] += take + body_budget -= take + if snippets[index] < min(180, len(parts[index][1])): + next_pending.append(index) + pending = next_pending + output = prefix + "".join( + "\n" + entry_prefix + _clip(text, snippets[index]) + for index, (entry_prefix, text) in enumerate(parts) + ) + shown = [entry["id"] for entry in selected] + next_offset = offset + len(shown) + if next_offset < len(ranked): + if style == "top": + output += "\n(...)" + else: + if token is None and store is not None: + try: + token = store.save_page(request, catalog, [entry["id"] for entry in ranked]) + except (OSError, ValueError, sqlite3.Error) as exc: + warn(f"Cannot save pagination: {exc}. Repair the local ledger or narrow the query.") + if token: + output += f"\nMore: --cursor {token}:{next_offset} (same view)" + else: + output += "\nMore omitted: narrow the query (local ledger unavailable)." + if style != "top": + output += "\nExpand a claim: --get ID" + output = prefix.replace("{shown}", str(len(shown))) + output[len(prefix):] + if cap and len(output) + 1 > cap: + raise ValueError("Output budget cannot fit retrieval metadata; increase --max-chars") + print(output, flush=True) + _record_visible(store, shown, options) + + +def view_get(shadow_dir, identity, *, options=None): + """Expand a current discovery, chunking long text without changing its ID.""" + options = options or RetrievalOptions() + if not re.fullmatch(r"d_[0-9a-f]{32}", identity): + raise ValueError("--get requires the complete id=d_... value from a retrieval result") + entry = next((entry for entry in _knowledge_entries(shadow_dir) if entry["id"] == identity), None) + if entry is None: + raise ValueError("Discovery ID is absent or its claim changed; search again for its current ID") + store = _citation_store(shadow_dir) + scores, scores_known = _read_scores(store, [identity]) + score = scores.get(identity, 0) if scores_known else "?" + body = entry["anchor"] + "\n\n" + entry["text"] + if entry["labels"]: + body += "\nLabels: " + ", ".join(entry["labels"]) + if entry["kind"] == "cross-cutting": + body += "\nCategory: " + entry["category"] + "\nTitle: " + entry["title"] + if entry["refs"]: + body += "\nRefs: " + ", ".join(entry["refs"]) + if entry["dream_report"]: + body += "\nDream report: " + entry["dream_report"] + revision = hashlib.sha256(json.dumps( + [entry["status"], entry["source"], body], ensure_ascii=False, + ).encode("utf-8")).hexdigest()[:32] + start = 0 + event = options.event_id + expires = int(time.time()) + RECEIPT_TTL + record = options.record + if options.text_cursor: + match = re.fullmatch( + r"([0-9a-f]{32})~(\d{1,12})~([01])~([A-Za-z0-9._:-]{1,128})~(\d+)", + options.text_cursor, + ) + if not match: + raise ValueError("Invalid text cursor; copy the --text-cursor value from the previous expansion") + prior_revision, expiry, recording, prior_event, offset = match.groups() + if prior_revision != revision: + raise ValueError("Discovery body or metadata changed; restart --get without --text-cursor") + if int(expiry) <= time.time(): + raise ValueError("Text cursor expired; restart --get without --text-cursor") + if event is not None and event != prior_event: + raise ValueError("Text cursor has a different --event-id; reuse its original event") + start, expires, event = int(offset), int(expiry), prior_event + record = record and recording == "1" + if start >= len(body): + raise ValueError("Text cursor is past the end of this discovery; restart --get") + header = ( + f"id={identity} citation_score={score}\n" + f"({entry['status']}, source: {entry['source']})\n" + ) + tail = "" + available = len(body) - start + if options.max_chars and len(header) + available + 1 > options.max_chars: + event = event or uuid.uuid4().hex + + def continuation(offset): + token = f"{revision}~{expires}~{int(record)}~{event}~{offset}" + value = f"\nContinue: --get {identity} --text-cursor {token}" + if options.max_chars != 4000: + value += f" --max-chars {options.max_chars}" + return value + + available = options.max_chars - len(header) - len(continuation(len(body))) - 1 + if available < 1: + raise ValueError("Output budget cannot fit continuation metadata; increase --max-chars") + tail = continuation(start + available) + output = header + body[start:start + available] + tail + print(output, flush=True) + if record: + _record_visible(store, [identity], options, event) + + +def view_summary(shadow_dir): + """Overview + detailed statistics. + + Each section is independently wrapped — if label stats fail, you + still get counts and the per-file table. + """ + # Load data (each can fail independently) + state = {} + shadow_files = [] + prefs = [] + cross = [] + + try: + state = load_state(shadow_dir) + except Exception as e: + warn(f"Failed to load state.json: {e}") + + try: + shadow_files = get_all_shadow_files(shadow_dir) + except Exception as e: + warn(f"Failed to list shadow files: {e}") + + try: + prefs = parse_prefs(shadow_dir) + except Exception as e: + warn(f"Failed to parse preferences: {e}") + + try: + cross = parse_cross_cutting(shadow_dir) + except Exception as e: + warn(f"Failed to parse cross-cutting discoveries: {e}") + + # Parse each file once for both stats and discoveries + file_stats = [] + all_disc = [] + total_symbols = 0 + for sf in shadow_files: + try: + parsed = parse_shadow_file(sf) + src = parsed["source_file"] or str(sf.relative_to(shadow_dir)) + n_sym = len(parsed["symbols"]) + n_disc = len(parsed["discoveries"]) + total_symbols += n_sym + file_stats.append((src, n_sym, n_disc)) + for d in parsed["discoveries"]: + d.setdefault("file", parsed["source_file"]) + d["shadow_path"] = str(sf.relative_to(shadow_dir)) + all_disc.append(d) + except Exception as e: + warn(f"Failed to process {sf}: {e}") + file_stats.sort(key=lambda x: x[2], reverse=True) + + # Header counts (always shown) + print("Shadow Knowledge Base Summary") + print("=" * 50) + print(f" Files shadowed: {len(shadow_files)}") + print(f" Symbols tracked: {total_symbols}") + print(f" Discoveries: {len(all_disc)}") + print(f" Preferences: {len(prefs)}") + print(f" Cross-cutting: {len(cross)}") + + # Source breakdown + try: + source_counts = defaultdict(int) + status_counts = defaultdict(int) + for d in all_disc: + source_counts[d.get("source", "unknown")] += 1 + status_counts[d.get("status", "unknown")] += 1 + + if source_counts: + print("\nBy source:") + for src, cnt in sorted(source_counts.items(), key=lambda x: -x[1]): + pct = cnt / len(all_disc) * 100 if all_disc else 0 + bar = "#" * int(pct / 2) + print(f" {src:15s} {cnt:4d} ({pct:5.1f}%) {bar}") + + if status_counts: + print("\nBy status:") + for st, cnt in sorted(status_counts.items(), key=lambda x: -x[1]): + pct = cnt / len(all_disc) * 100 if all_disc else 0 + bar = "#" * int(pct / 2) + print(f" {st:15s} {cnt:4d} ({pct:5.1f}%) {bar}") + except Exception as e: + warn(f"Failed to compute source/status breakdown: {e}") + + # Label breakdown + try: + label_counts = defaultdict(int) + for d in all_disc: + for lbl in d.get("labels", []): + label_counts[lbl] += 1 + if label_counts: + print("\nBy label:") + for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]): + print(f" {lbl:15s} {cnt:4d}") + except Exception as e: + warn(f"Failed to compute label breakdown: {e}") + + # Per-file table + try: + if file_stats: + print(f"\n{'File':<40s} {'Symbols':>8s} {'Disc.':>6s}") + print(f"{'-'*40} {'-'*8} {'-'*6}") + for src, n_sym, n_disc in file_stats[:20]: + print(f"{src:<40s} {n_sym:>8d} {n_disc:>6d}") + if len(file_stats) > 20: + print(f"... and {len(file_stats) - 20} more files") + except Exception as e: + warn(f"Failed to render per-file table: {e}") + + # Cross-cutting titles + try: + if cross: + print(f"\nCross-cutting discoveries:") + for e in cross: + title = e.get("title", e.get("slug", "?")) + cat = e.get("category", "?") + print(f" [{cat}] {title}") + except Exception as e: + warn(f"Failed to render cross-cutting list: {e}") + + # State info + try: + if state: + print(f"\nLast update: {state.get('last_update_at', '?')} " + f"({state.get('last_update_type', '?')})") + print(f"Last commit: {state.get('last_commit', '?')}") + except Exception as e: + warn(f"Failed to render state info: {e}") + + +def view_search(shadow_dir, query, *, options=None): + """Bounded search with stable continuation over matching knowledge identities.""" + options = options or RetrievalOptions() + query_lower = query.lower() + if not query.strip(): + raise ValueError("Search query must be nonempty") + matches = [] + for entry in _knowledge_entries(shadow_dir): + fields = [entry["anchor"], entry["file"], entry["symbol"], entry["text"], + entry["title"], *entry["refs"]] + if any(query_lower in field.lower() for field in fields): + entry["relevance"] = 0 if query_lower in [field.lower() for field in fields[:3]] else 1 + matches.append(entry) + if not matches and not options.cursor: + print(_clip(f"No results for '{query}'.", options.max_chars - 1) if options.max_chars + else f"No results for '{query}'.") + return + _emit_knowledge( + shadow_dir, matches, f"Search: '{query}' ({len(matches)} results)", options, + request=json.dumps(["search", query_lower]), + ) + + +def view_symbol(shadow_dir, anchor, *, options=None): + options = options or RetrievalOptions() + file, separator, symbol = anchor.partition("::") + if not separator or not symbol: + raise ValueError("--symbol requires file::symbol (use File-Level for a file-level section)") + file = _canonical_source_file(shadow_dir, file) + symbol = _canonical_symbol(symbol) + canonical = f"{file}::{symbol}" + matches = [ + entry for entry in _knowledge_entries(shadow_dir, file) + if entry["anchor"] == canonical or canonical in entry["refs"] + ] + if not matches and not options.cursor: + print(_clip(f"No knowledge for '{anchor}'.", options.max_chars - 1) + if options.max_chars else f"No knowledge for '{anchor}'.") + return + _emit_knowledge( + shadow_dir, matches, f"Knowledge for {anchor} ({len(matches)} results)", options, + request=json.dumps(["symbol", canonical]), + ) + + +def view_file(shadow_dir, source_file, *, options=None): + """Read one known source file's shadow, including file-level and cross refs.""" + options = options or RetrievalOptions() + source_file = _canonical_source_file(shadow_dir, source_file) + entries = _knowledge_entries(shadow_dir, source_file) + if not entries and not options.cursor: + message = f"No knowledge for '{source_file}'." + print(_clip(message, options.max_chars - 1) if options.max_chars else message) + return + _emit_knowledge( + shadow_dir, entries, f"Knowledge for {source_file} ({len(entries)} results)", + options, request=json.dumps(["file", source_file]), + ) + + +def view_prefs(shadow_dir, *, options=None): + """Page preferences without letting popular code discoveries hide directives.""" + options = options or RetrievalOptions() + prefs = _preference_entries(shadow_dir) + if not prefs and not options.cursor: + print("No preferences recorded yet.") + return + _emit_knowledge(shadow_dir, prefs, f"Project Preferences ({len(prefs)} total)", options, request="prefs") + + +def view_labels(shadow_dir, label_filter, *, options=None): + """Show discoveries filtered by label(s). + + label_filter can be a single label or comma-separated list. + """ + options = options or RetrievalOptions() + filters = [label.strip().lower() for label in label_filter.split(",") if label.strip()] + if not filters: + raise ValueError("Supply at least one label with --labels") + matching = [ + entry for entry in _knowledge_entries(shadow_dir) + if set(filters) & {label.lower() for label in entry["labels"]} + ] + + if not matching and not options.cursor: + message = f"No discoveries with label(s): {', '.join(filters)}" + print(_clip(message, options.max_chars - 1) if options.max_chars else message) + return + + _emit_knowledge( + shadow_dir, matching, + f"Discoveries with label(s): {', '.join(filters)} ({len(matching)} results)", + options, style="labels", request=json.dumps(["labels", sorted(set(filters))]), + ) + + +def view_recent(shadow_dir, count=10, *, options=None): + """Show the N most recent discoveries (by shadow file mtime). + + Collects all discoveries across all shadow files, cross-cutting entries, + and preferences, sorts by the source file's modification time (most recent + first), and shows the actual discovery content. + Each data source is independent — if cross-cutting fails, per-file + discoveries still appear. + """ + options = options or RetrievalOptions(limit=count) + all_items = _knowledge_entries(shadow_dir) + if not all_items and not options.cursor: + print("No discoveries found.") + return + + _emit_knowledge( + shadow_dir, all_items, f"Most Recent Discoveries (top {count})", + options, style="recent", request="recent", + ) + + +def view_top(shadow_dir, file_path, labels_filter, limit, max_chars, *, options=None): + """Show the top N actionable discoveries for a single source file. + + Designed for the preToolUse hook: concise output suitable for + inlining into additionalContext when the agent is about to mutate a + file. Pulls from both the per-file shadow and any _cross/ entries + whose refs touch this file. + + Trust/status precedes citation score. Output, including the final newline, + is hard-capped; only entries actually emitted are counted. + """ + norm = file_path.strip() + if norm.startswith("./"): + norm = norm[2:] + label_set = {l.strip().lower() for l in labels_filter.split(",") if l.strip()} + options = options or RetrievalOptions(limit=limit, max_chars=max_chars) + candidates = [ + entry for entry in _knowledge_entries(shadow_dir, norm) + if not label_set or label_set & {label.lower() for label in entry["labels"]} + ] + + if not candidates: + labels_disp = ",".join(sorted(label_set)) if label_set else "any" + message = f"No actionable discoveries ({labels_disp}) for {norm}." + print(_clip(message, max_chars - 1) if max_chars else message) + return + _emit_knowledge( + shadow_dir, candidates, + f"Top {{shown}} of {len(candidates)} actionable discoveries for {norm}:", + options, style="top", request=json.dumps(["top", norm, sorted(label_set)]), + ) + + +def view_check_invariants(shadow_dir): + """Walk the shadow knowledge base and report invariant violations. + + Statically-checkable invariants from shadow-frog/SKILL.md: + #3 (partial) Per-file 'Also involves:' uses file::symbol notation + #4 Cross-ref back-pointers match: _cross/.md refs <-> + per-file ## Cross-References + #5 Every ## Cross-References entry has a matching _cross/*.md + + Plus syntactic guards that catch the most common drift: + - Symbol headings use the required backtick form + - Discovery metadata uses valid status enum + - Discovery metadata uses valid source enum + - Discovery labels are from the allowed set + - _cross/ Category field uses a known value + + Invariants #1, #2, #7 are NOT checked (would require source parsing + and semantic match); #6 is filesystem-enforced. + + Exit 0 = clean, 1 = at least one violation. Violations print one per + line in `path:line: kind: message` form so grep/editors can navigate. + """ + VALID_STATUS = {"verified", "uncertain", "refuted"} + VALID_SOURCE = {"exploration", "user", "interaction"} + VALID_LABELS = {"bug", "performance", "security", + "feature-gap", "tech-debt"} + VALID_CATEGORIES = { + "pattern", "behavior", "edge-case", "contract", + "performance", "intent", "warning", "history", "convention", + } + + violations = [] + def v(path, line, kind, msg): + violations.append(f"{path}:{line}: {kind}: {msg}") + + # Pass 1: walk per-file shadows -> collect cross-reference entries + # they declare and validate their internal format. + per_file_xref_targets = {} # rel_shadow_path -> set(slug declared) + cross_dir = shadow_dir / "_cross" + cross_slugs_on_disk = set() + if cross_dir.is_dir(): + try: + cross_slugs_on_disk = {f.stem for f in cross_dir.glob("*.md")} + except OSError as e: + warn(f"Cannot list {cross_dir}: {e}") + + md_heading_re = re.compile(r"^(#{2,3})\s+(.*)$") + backtick_heading_re = re.compile(r"^(#{2,3})\s+`[^`]+`\s*$") + also_involves_re = re.compile(r"^\s*Also involves:\s*(.+)$", re.I) + file_sym_re = re.compile(r"`([^`]+::[^`]+)`") + + for shadow_path in get_all_shadow_files(shadow_dir): + try: + rel = shadow_path.relative_to(shadow_dir) + except ValueError: + continue + try: + text = shadow_path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + v(rel, 0, "unreadable", str(e)) + continue + + in_cross_refs = False + declared = set() + for ln, raw in enumerate(text.split("\n"), 1): + line = raw.rstrip() + + heading = md_heading_re.match(line) + if heading: + title = heading.group(2).strip() + if title.lower().startswith("cross-references"): + in_cross_refs = True + continue + in_cross_refs = False + # Skip special headings ("File-Level Notes", "Notes", etc.) + if ( + title.lower().startswith("file-level") + or title.lower() in {"notes", "metadata"} + ): + continue + # Symbol heading must use backtick form + if not backtick_heading_re.match(line): + v(rel, ln, "heading", + f"symbol heading must be `## `name`` or " + f"`### `Class.name``; got: {line[:80]}") + continue + + if in_cross_refs and line.strip().startswith("- "): + # Format: - [slug](.shadow/_cross/slug.md) — title + slug_match = re.search( + r"_cross/([^)\s]+?)\.md", line + ) + if slug_match: + declared.add(slug_match.group(1)) + else: + # Looser fallback: bare slug in brackets + alt = re.search(r"\[([^\]]+)\]", line) + if alt: + declared.add(alt.group(1).strip()) + + # Discovery metadata line + md = _DISCOVERY_META_RE.search(line) + if md: + status, source = md.group(1), md.group(2) + labels_raw = md.group(3) or "" + if status not in VALID_STATUS: + v(rel, ln, "enum", + f"status '{status}' not in {sorted(VALID_STATUS)}") + if source not in VALID_SOURCE: + v(rel, ln, "enum", + f"source '{source}' not in {sorted(VALID_SOURCE)}") + for lbl in (l.strip() for l in labels_raw.split(",") if l.strip()): + if lbl not in VALID_LABELS: + v(rel, ln, "enum", + f"label '{lbl}' not in {sorted(VALID_LABELS)}") + + # `Also involves:` must list file::symbol anchors in backticks + ai = also_involves_re.match(line) + if ai: + rest = ai.group(1) + anchors = file_sym_re.findall(rest) + if not anchors: + v(rel, ln, "anchor", + "Also involves: needs `file::symbol` " + "backtick anchors") + # Light sanity: every anchor has both file and symbol + for a in anchors: + if "::" not in a or not a.split("::", 1)[1].strip(): + v(rel, ln, "anchor", + f"anchor '{a}' missing symbol after ::") + + per_file_xref_targets[str(rel)] = declared + + # Invariant #5: every declared cross slug must exist on disk + for slug in declared: + if slug not in cross_slugs_on_disk: + v(rel, 0, "cross-ref", + f"references _cross/{slug}.md but file does not exist") + + # Pass 2: walk _cross/*.md -> validate refs format + back-pointer. + # Build the reverse map: cross_slug -> set(file::symbol it points at). + cross_back = {} # slug -> set(file paths it should be linked from) + if cross_dir.is_dir(): + for cf in sorted(cross_dir.glob("*.md")): + slug = cf.stem + try: + text = cf.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + v(cf.relative_to(shadow_dir), 0, "unreadable", str(e)) + continue + + rel_cf = cf.relative_to(shadow_dir) + + # Category enum check + cat_m = re.search(r"\*\*Category\*\*:\s*(.+)", text) + if cat_m: + cat = cat_m.group(1).strip().lower() + if cat not in VALID_CATEGORIES: + v(rel_cf, 0, "enum", + f"Category '{cat}' not in {sorted(VALID_CATEGORIES)}") + else: + v(rel_cf, 0, "schema", + "missing **Category**: field") + + # Discovery metadata + md = _DISCOVERY_META_RE.search(text) + if md: + status, source = md.group(1), md.group(2) + if status not in VALID_STATUS: + v(rel_cf, 0, "enum", + f"status '{status}' not in {sorted(VALID_STATUS)}") + if source not in VALID_SOURCE: + v(rel_cf, 0, "enum", + f"source '{source}' not in {sorted(VALID_SOURCE)}") + else: + v(rel_cf, 0, "schema", + "missing trailing _(status, source: ...)_ metadata") + + # Refs must be `file::symbol` anchors + refs_block = re.search( + r"\*\*Refs\*\*:\s*\n((?:\s*-\s+`[^`]+`\s*\n?)+)", + text, + ) + if not refs_block: + v(rel_cf, 0, "schema", + "missing **Refs**: block (one per line, " + "`- `file::symbol``)") + else: + anchors = file_sym_re.findall(refs_block.group(1)) + if not anchors: + v(rel_cf, 0, "anchor", + "Refs block has no `file::symbol` entries") + for a in anchors: + if "::" not in a or not a.split("::", 1)[1].strip(): + v(rel_cf, 0, "anchor", + f"ref '{a}' missing symbol after ::") + else: + # Convert file part to shadow path: + # src/foo.py -> src/foo.py.md (relative to shadow_dir) + file_part = a.split("::", 1)[0].strip() + shadow_rel = f"{file_part}.md" + cross_back.setdefault(slug, set()).add(shadow_rel) + + # Invariant #4 back-pointer: every file referenced by a cross slug + # must declare that slug in its ## Cross-References. + for slug, expected_files in cross_back.items(): + for shadow_rel in expected_files: + declared = per_file_xref_targets.get(shadow_rel) + if declared is None: + v(f"_cross/{slug}.md", 0, "cross-ref", + f"refs {shadow_rel} but no such shadow file exists") + elif slug not in declared: + v(f"_cross/{slug}.md", 0, "cross-ref", + f"refs {shadow_rel} but that shadow's ## " + f"Cross-References does not link back to " + f"_cross/{slug}.md") + + # Output + if not violations: + print(f"✓ Invariants OK ({len(per_file_xref_targets)} per-file " + f"shadows, {len(cross_slugs_on_disk)} cross-cutting " + f"discoveries)") + return 0 + + for line in violations: + print(line) + print(f"\n{len(violations)} invariant violation(s) found.", + file=sys.stderr) + return 1 + + +def main(*, agent=False): + for _stream in (sys.stdout, sys.stderr): + if hasattr(_stream, "reconfigure"): + _stream.reconfigure(encoding="utf-8") + try: + parser = argparse.ArgumentParser( + description=( + "Optional bounded agent retrieval. Navigate directly to shadow files first; " + "use this helper for large sections or targeted search." + if agent else + "Browse and visualize a .shadow/ knowledge base for users." + ), + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + if agent: + parser.add_argument( + "target", nargs="?", metavar="FILE[::SYMBOL]", + help="Known source file or file::symbol; its shadow path is determined directly", + ) + + # Views (mutually exclusive) + views = parser.add_mutually_exclusive_group() + views.add_argument( + "--summary", action="store_true", + help="Overview + detailed statistics (default)", + ) + views.add_argument( + "--search", metavar="QUERY", + help="Universal search: files, symbols, and discovery text", + ) + views.add_argument("--file", metavar="FILE", help="Read a known file's shadow without a repository-wide search") + views.add_argument("--symbol", metavar="FILE::SYMBOL", help="Knowledge for an exact symbol and its cross-cutting refs") + views.add_argument("--get", metavar="ID", help="Expand a discovery returned by a bounded read") + views.add_argument( + "--prefs", action="store_true", + help="Show project-wide preferences", + ) + views.add_argument( + "--recent", nargs="?", const=10, type=int, metavar="N", + help="N most recent discoveries with content (default: 10)", + ) + views.add_argument( + "--labels", metavar="LABEL", + help=( + "Show discoveries by label " + "(e.g., bug, security, bug,performance)" + ), + ) + views.add_argument( + "--top", metavar="FILE", + help=( + "Top actionable discoveries for FILE (a source path " + "like src/auth.py). Concise output for the preToolUse " + "hook: filters to actionable labels (default: " + "bug,security), includes both per-file and _cross/ " + "entries that reference FILE, ranks verified first." + ), + ) + views.add_argument( + "--check-invariants", action="store_true", + help=( + "Walk the shadow and report structural violations: " + "missing back-pointers, dangling _cross/ refs, invalid " + "enums, bad heading format. Exit 1 if any are found." + ), + ) + + # Options + parser.add_argument( + "--shadow-dir", default=None, + help="Path to .shadow/ directory (default: auto-detect)", + ) + parser.add_argument( + "--top-labels", default="bug,security", metavar="LABELS", + help=( + "Comma-separated labels to include in --top " + "(default: bug,security). Pass empty string to include " + "all labeled discoveries." + ), + ) + parser.add_argument( + "--top-limit", type=int, default=3, metavar="N", + help="Max discoveries to show in --top (default: 3)", + ) + parser.add_argument( + "--top-max-chars", type=int, default=600, metavar="N", + help=( + "Hard cap on --top total output length " + "(default: 600). Use 0 for no cap." + ), + ) + parser.add_argument("--limit", type=int, help="Results per page for file/search/symbol/labels/prefs (default: 10)") + parser.add_argument("--max-chars", type=int, help="Retrieval output budget (default: 4000; 0 disables cap)") + parser.add_argument("--cursor", help="Continue the same view/filters with a frozen result ordering") + parser.add_argument("--event-id", help="Retry token; an entry counts once per token (default: fresh event)") + parser.add_argument("--no-record", action="store_true", help="Do not increment citation scores") + parser.add_argument("--text-cursor", help="Continue the same logical read of an unchanged --get body") + + args = parser.parse_args() + selected_view = ( + args.summary or args.check_invariants or args.top is not None + or args.search is not None or args.file is not None or args.symbol is not None + or args.get is not None or args.prefs or args.labels is not None or args.recent is not None + ) + if agent and args.target is not None: + if selected_view: + parser.error("Use a positional target or an explicit view, not both") + if "::" in args.target: + args.symbol = args.target + else: + args.file = args.target + elif agent and not selected_view: + parser.error("Provide a known FILE[::SYMBOL] or an explicit retrieval operation such as --search") + retrieval = ( + args.top is not None or args.search is not None or args.file is not None or args.symbol is not None + or args.get is not None or args.prefs or args.labels is not None or args.recent is not None + ) + if not retrieval and ( + args.limit is not None or args.max_chars is not None or args.cursor + or args.event_id is not None or args.no_record or args.text_cursor is not None + ): + parser.error("Retrieval options require --search, --file, --symbol, --get, --top, --prefs, --labels, or --recent") + if args.text_cursor is not None and args.get is None: + parser.error("--text-cursor requires --get") + if args.cursor and (args.get is not None or args.top is not None): + parser.error("--cursor is for paged file/search/symbol/labels/prefs/recent; use --text-cursor with --get") + if args.limit is not None and (args.top is not None or args.recent is not None or args.get is not None): + parser.error("Use --top-limit or --recent N instead of --limit; --get returns one discovery") + if args.max_chars is not None and args.top is not None: + parser.error("Use --top-max-chars with --top") + try: + options = RetrievalOptions( + limit=args.top_limit if args.top is not None else ( + args.recent if args.recent is not None else (args.limit if args.limit is not None else 10) + ), + max_chars=args.top_max_chars if args.top is not None else ( + args.max_chars if args.max_chars is not None else 4000 + ), + cursor=args.cursor, event_id=args.event_id, record=not args.no_record, + text_cursor=args.text_cursor, + ) + except ValueError as exc: + parser.error(str(exc)) + + # Find shadow dir + if args.shadow_dir: + shadow_dir = Path(args.shadow_dir) + else: + shadow_dir = find_shadow_dir() + + if not shadow_dir or not shadow_dir.is_dir(): + cwd = os.getcwd() + error( + f"No .shadow/ directory found. " + f"Searched from: {cwd}\n" + f"[shadow error] " + f"Run /shadow-frog-init first to create the shadow, " + f"or pass --shadow-dir /path/to/.shadow/ explicitly." + ) + if args.shadow_dir: + error( + f"Provided --shadow-dir '{args.shadow_dir}' does not " + f"exist or is not a directory." + ) + sys.exit(1) + + # Dispatch + if args.check_invariants: + sys.exit(view_check_invariants(shadow_dir)) + if args.top is not None: + view_top( + shadow_dir, + args.top, + args.top_labels, + args.top_limit, + args.top_max_chars, + options=options, + ) + elif args.search is not None: + view_search(shadow_dir, args.search, options=options) + elif args.file is not None: + view_file(shadow_dir, args.file, options=options) + elif args.symbol is not None: + view_symbol(shadow_dir, args.symbol, options=options) + elif args.get is not None: + view_get(shadow_dir, args.get, options=options) + elif args.prefs: + view_prefs(shadow_dir, options=options) + elif args.labels is not None: + view_labels(shadow_dir, args.labels, options=options) + elif args.recent is not None: + view_recent(shadow_dir, args.recent, options=options) + else: + view_summary(shadow_dir) + + except SystemExit: + raise + except KeyboardInterrupt: + error("Interrupted by user.") + sys.exit(130) + except (ValueError, sqlite3.Error) as exc: + if is_busy_error(exc): + error("Local citation ledger is busy; retry the same command later. Do not reset a busy database.") + else: + error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") + sys.exit(1) + except Exception as e: + error( + f"Unexpected error: {type(e).__name__}: {e}\n" + f"[shadow error] Full traceback:\n" + f"{traceback.format_exc()}" + f"This is likely a bug in the shadow knowledge helper. " + f"The shadow data may be in an unexpected format. " + f"Try running with --shadow-dir to confirm the path, " + f"or inspect the .shadow/ files manually." + ) + sys.exit(1) diff --git a/skills/shadow-frog/retrieval.md b/skills/shadow-frog/retrieval.md new file mode 100644 index 0000000..9864248 --- /dev/null +++ b/skills/shadow-frog/retrieval.md @@ -0,0 +1,108 @@ +# Optional Agent Retrieval Reference + +Direct `.shadow/.md` reads and symbol navigation remain the default. +Use the core `shadow-read.py` when a known section is too large for context or +when a targeted search is useful. It supports Python 3.9+ and does not require +the user-facing Viewer. Neither tool changes the canonical Markdown format. + +```text +python .github/skills/shadow-frog/shadow-read.py src/auth.py +python .github/skills/shadow-frog/shadow-read.py src/auth.py::UserAuth.validate --limit 5 +python .github/skills/shadow-frog/shadow-read.py --search "token expiry" +python .github/skills/shadow-frog/shadow-read.py --get DISCOVERY_ID +``` + +Use `.claude/skills/` for Claude Code. Replace `DISCOVERY_ID` and cursor values +with values returned by the helper. A known file or `file::symbol` can be passed +positionally; the equivalent explicit selectors are `--file` and `--symbol`. +Do not combine a positional target with another view. + +## Views and Options + +| View | Behavior | +|------|----------| +| `--file FILE` | Bounded per-file knowledge, including file-level sections and related cross-cutting entries; does not search unrelated per-file shadows | +| `--symbol FILE::SYMBOL` | Exact symbol and cross-cutting refs; `File-Level` selects its file-level section | +| `--search QUERY` | Search paths, symbols, claim text, preferences, and cross-cutting entries | +| `--get ID` | Expand one current entry; long content returns a revision-bound continuation | +| `--prefs` | Preferences; follow every page when relying on the full set of directives | +| `--labels LABELS` | Comma-separated actionable label filter | +| `--recent [N]` | Most recent previews by source-shadow mtime (default: 10 per page) | +| `--top FILE` | Compact actionable hints (default labels: `bug,security`, up to 3 entries / 600 characters); not exhaustive or pageable | +| `--check-invariants` | Structural audit; no citation recording | +| `--summary` | Statistics/overview; no citation recording | + +| Option | Contract | +|--------|----------| +| `--shadow-dir DIR` | Override the shadow root; otherwise locate it from the working directory | +| `--limit N` | Positive page size for file/search/symbol/labels/preferences (default: 10) | +| `--max-chars N` | Output cap including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output) | +| `--cursor TOKEN` | Continue the same view and filters with its frozen result ordering | +| `--text-cursor TOKEN` | Continue the same unchanged `--get` body as one logical read | +| `--event-id ID` | Retry ID: each entry counts once per ID within 24 hours; 1-128 letters/digits or `. _ : -` | +| `--no-record` | Do not increment scores; existing scores still rank results, and pagination can store a local snapshot | +| `--top-labels LABELS` | Labels for `--top`; an empty string removes its label filter | +| `--top-limit N` | Positive result limit for `--top` (default: 3) | +| `--top-max-chars N` | Cap for `--top` instead of `--max-chars` (default: 600; minimum 256, or 0 for no cap) | + +For a truncated result set, repeat the same view with the returned `--cursor`. +For a long expanded claim, copy its `--get ... --text-cursor ...` continuation. +Character limits constrain output, not token counts or local parsing work. +Errors use stderr and exit 1 (invalid arguments: 2). Optional telemetry warnings +do not hide knowledge; raw reads remain available independently. + +## Citation and Identity Semantics + +There is one `citation_score`, initially zero regardless of the creation workflow. +The user Viewer and the optional core helper share identities and the same ledger. +Only emitted claim content increments scores. Parsing, matching, summaries, +structural audits, and native file reads do not. Do not manually add score fields +to Markdown or treat partial instrumentation as a complete access history. + +Displayed scores precede the current read. Long expansion chunks share one +logical-read event. Compact `--top` lines omit numeric scores to preserve room for +content. A citation measures exposure, not correctness or influence on reasoning. +Relevance and source trust/status rank ahead of scores; tied tiers reserve room +for zero-score entries when page and character budgets allow multiple results. +Popularity never overrides refutation or authorizes dropping a user constraint. + +The `d_...` fingerprint binds kind, canonical file/symbol anchor, parsed claim +text, and related refs. Internal whitespace is preserved, including code literals. +Filesystem aliases share on-disk identity and container prefixes are removed +from symbol anchors. Metadata-only source/status/label changes retain identity; +rewriting, moving, or merging claims/refs can produce a new zero-score identity. +Duplicate labels are unioned. No fuzzy score transfer is performed by Meditate. +An ID is an optional handle, not a replacement for the file/symbol address. + +## Local State and Concurrency + +SQLite lives at `shadowfrog/citations.sqlite3` under the repository's **common Git +directory**, shared by local worktrees. Different shadow roots have separate +scopes. Standalone shadows use `$XDG_STATE_HOME/shadowfrog/citations` +(Windows: `$LOCALAPPDATA`), falling back to `~/.local/state/shadowfrog/citations`. +The cache stores IDs, scores, retry receipts, hashed requests and ordered ID +snapshots, not discovery bodies. It is local metadata, not a Git-synchronized DB. +WAL requires a local filesystem, not a network share. + +Successful transactions are atomic. Busy reads/writes retry within a 100 ms +budget; accounting is best-effort, and a timed-out increment warns that the visit +was not recorded. Unknown scores display as `?`, but recording can still recover +after output. Retry a busy database later rather than deleting it. + +Ordinary reads retain no event receipt. Explicit retry and generated logical-read +receipts expire after 24 hours and are capped at 100,000; excess new receipts fail +visibly without weakening deduplication. Expired receipts are pruned in bounded +batches. Do not reuse one event ID for unrelated visits. + +Identical page snapshots reuse compressed storage, with at most 32 snapshots / +8 MiB retained. They expire after 24 hours or capacity eviction. Score changes +do not change a cursor's order. Relevant matching content/metadata changes require +restarting; mtime-only or unrelated-symbol edits do not invalidate non-recent views. +Text cursors bind the exact expanded body and metadata, preserve `--no-record`, +and expire after 24 hours; changed or expired content requires restarting `--get`. + +Commits use full synchronization. WAL checkpoints run after 256 pages with a +1 MiB retained-journal limit after reset; active transactions can keep a larger +journal temporarily. SQLite can retain reusable free pages. Score rows grow with +distinct identities. An incompatible prerelease cache emits reset guidance; move +that cache aside, not the shadow, if deliberately resetting local telemetry. diff --git a/skills/shadow-frog/shadow-read.py b/skills/shadow-frog/shadow-read.py new file mode 100644 index 0000000..9f6dd02 --- /dev/null +++ b/skills/shadow-frog/shadow-read.py @@ -0,0 +1,34 @@ +#!/usr/bin/env python3 +"""Optional bounded knowledge retrieval for agents that already navigate files. + +Examples: + python shadow-read.py src/auth.py + python shadow-read.py src/auth.py::UserAuth.validate --limit 5 + python shadow-read.py --search "token expiry" + +Direct .shadow/.md reads remain the primary navigation method. +This helper selects and pages large sections; it does not replace their paths. +""" + +import sys +from pathlib import Path + + +_bytecode = sys.dont_write_bytecode +sys.dont_write_bytecode = True +sys.path.insert(0, str(Path(__file__).resolve().parent)) +try: + import _knowledge +except ImportError as exc: + raise SystemExit("ERROR: Missing core knowledge helper; reinstall the full ShadowFrog skill set") from exc +finally: + sys.path.pop(0) + sys.dont_write_bytecode = _bytecode + + +def main(): + return _knowledge.main(agent=True) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/conftest.py b/tests/conftest.py index 25cd182..25eff26 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -53,6 +53,16 @@ def shadow_viewer(repo_root): return _load_script(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py") +@pytest.fixture(scope="session") +def shadow_knowledge(repo_root): + return _load_script(repo_root / "skills/shadow-frog/_knowledge.py") + + +@pytest.fixture(scope="session") +def shadow_reader(repo_root): + return _load_script(repo_root / "skills/shadow-frog/shadow-read.py") + + @pytest.fixture(scope="session") def dream_reconcile(repo_root): return _load_script(repo_root / "skills/shadow-frog-dream/dream-reconcile.py") diff --git a/tests/hooks/test_pre_tool_sh.py b/tests/hooks/test_pre_tool_sh.py index f029087..d9e86fc 100644 --- a/tests/hooks/test_pre_tool_sh.py +++ b/tests/hooks/test_pre_tool_sh.py @@ -169,7 +169,7 @@ def test_citation_failure_is_visible_but_never_denies_edit(self, coupon_demo, tm context = json.loads(result.stdout)["additionalContext"] assert "Actionable discoveries" in context assert "id=d_" in context - assert "Viewer reported a warning" in context + assert "Reader reported a warning" in context assert ledger.read_bytes() == b"invalid sqlite" diff --git a/tests/skills/shadow_frog/conftest.py b/tests/skills/shadow_frog/conftest.py new file mode 100644 index 0000000..fe04522 --- /dev/null +++ b/tests/skills/shadow_frog/conftest.py @@ -0,0 +1,9 @@ +"""Keep optional core-helper telemetry in each test's private state directory.""" + +import pytest + + +@pytest.fixture(autouse=True) +def local_citation_state(tmp_path, monkeypatch): + monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "citation-state")) + monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "citation-state")) diff --git a/tests/skills/shadow_frog_viewer/test_citations.py b/tests/skills/shadow_frog/test_citations.py similarity index 99% rename from tests/skills/shadow_frog_viewer/test_citations.py rename to tests/skills/shadow_frog/test_citations.py index 3e7ff28..0596144 100644 --- a/tests/skills/shadow_frog_viewer/test_citations.py +++ b/tests/skills/shadow_frog/test_citations.py @@ -13,7 +13,7 @@ from tests.conftest import _load_script -HELPER = Path(__file__).resolve().parents[3] / "skills/shadow-frog-viewer/_citations.py" +HELPER = Path(__file__).resolve().parents[3] / "skills/shadow-frog/_citations.py" @pytest.fixture diff --git a/tests/skills/shadow_frog/test_knowledge.py b/tests/skills/shadow_frog/test_knowledge.py new file mode 100644 index 0000000..19b6c2e --- /dev/null +++ b/tests/skills/shadow_frog/test_knowledge.py @@ -0,0 +1,2186 @@ +r"""Tests for shared `skills/shadow-frog/_knowledge.py` and the user viewer CLI. + +Philosophy: USE REAL FILES (per `minimal-mocking-tests`). Retrieval does not +edit shadow Markdown; local citation bookkeeping is isolated by the fixtures. +Tests construct shadow trees or exercise the CLI against `coupon_demo`. + +Test categories: + * In-process function tests (no `@pytest.mark.slow`): exercise + parsing helpers directly via the `shadow_knowledge` fixture. + * CLI integration tests (`@pytest.mark.slow @pytest.mark.integration`): + invoke shadow-viewer.py as a subprocess against `coupon_demo`. + +B3 regression: a discovery whose continuation lines include +``Dream report: `_dreams//` `` must extract the slug path into +`meta["dream_report"]` and must NOT include "Dream report" or the slug +in the discovery body text. +""" +import json +import os +import re +import subprocess +import sys +import textwrap +from datetime import datetime + +import pytest + + +# --- Helpers --------------------------------------------------------------- + + +def _write_shadow(shadow_dir, rel_path, content): + """Write `content` to /, creating parents.""" + p = shadow_dir / rel_path + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(textwrap.dedent(content), encoding="utf-8") + return p + + +def _make_shadow_root(tmp_path): + """Create an empty .shadow/ dir under tmp_path and return it.""" + sd = tmp_path / ".shadow" + sd.mkdir() + return sd + + +def _run_viewer(repo_root, cwd, *args): + """Run shadow-viewer.py as a subprocess from `cwd`.""" + script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" + return subprocess.run( + [sys.executable, str(script), *args], + cwd=str(cwd), + capture_output=True, + text=True, + encoding="utf-8", + check=False, + ) + + +# --- parse_discovery ------------------------------------------------------- + + +def test_parse_discovery_standard(shadow_knowledge): + """Basic verified/exploration discovery, no labels.""" + line = "- Caches None for invalid codes" + cont = [" _(verified, source: exploration)_"] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["text"] == "Caches None for invalid codes" + assert d["status"] == "verified" + assert d["source"] == "exploration" + assert "labels" not in d + assert "dream_report" not in d + + +def test_parse_discovery_with_labels(shadow_knowledge): + """Labels are parsed into a list, trimmed, lowercase comma-split.""" + line = "- Foo" + cont = [" _(verified, source: user, labels: [bug, security])_"] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["text"] == "Foo" + assert d["status"] == "verified" + assert d["source"] == "user" + assert d["labels"] == ["bug", "security"] + + +@pytest.mark.parametrize("status", ["verified", "uncertain", "refuted"]) +def test_parse_discovery_status_variants(shadow_knowledge, status): + line = "- Some discovery" + cont = [f" _({status}, source: exploration)_"] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["status"] == status + assert d["source"] == "exploration" + + +@pytest.mark.parametrize("source", ["exploration", "user", "interaction"]) +def test_parse_discovery_source_variants(shadow_knowledge, source): + line = "- Some discovery" + cont = [f" _(verified, source: {source})_"] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["source"] == source + + +def test_parse_discovery_also_involves(shadow_knowledge): + """`Also involves:` populates a list of file::symbol anchors.""" + line = "- A multi-symbol discovery" + cont = [ + " _(verified, source: exploration)_", + " Also involves: `inventory.py::validate_coupon`, `cart.py::COUPON_CACHE`", + ] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["text"] == "A multi-symbol discovery" + assert d["also_involves"] == [ + "inventory.py::validate_coupon", + "cart.py::COUPON_CACHE", + ] + # also_involves line must not leak into the body text + assert "Also involves" not in d["text"] + + +def test_parse_discovery_b3_dream_report_regression(shadow_knowledge): + """B3 regression: Dream report goes into meta['dream_report'] and is + excluded from the body text.""" + line = "- Case-variant lookups create duplicate cache entries" + cont = [ + " _(verified, source: exploration, labels: [bug, performance])_", + " Dream report: `_dreams/20260420-140000Z-cache-poison-sequence/`", + ] + d = shadow_knowledge.parse_discovery(line, cont) + # Body text is preserved, with no Dream report leakage + assert d["text"] == "Case-variant lookups create duplicate cache entries" + assert "Dream report" not in d["text"] + assert "_dreams/" not in d["text"] + # meta["dream_report"] captures the backtick payload (slug folder path) + assert d["dream_report"] == ( + "_dreams/20260420-140000Z-cache-poison-sequence/" + ) + + +def test_parse_discovery_b3_dream_report_with_also_involves(shadow_knowledge): + """Dream report + Also involves on the same discovery — both extracted, + neither leaks into body text.""" + line = "- Discovery with both extras" + cont = [ + " _(verified, source: exploration, labels: [security])_", + " Dream report: `_dreams/20260420-142000Z-adversarial-inputs/`", + " Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE`", + ] + d = shadow_knowledge.parse_discovery(line, cont) + assert d["text"] == "Discovery with both extras" + assert "Dream report" not in d["text"] + assert "Also involves" not in d["text"] + assert d["dream_report"] == ( + "_dreams/20260420-142000Z-adversarial-inputs/" + ) + assert d["also_involves"] == [ + "cart.py::get_coupon", + "cart.py::COUPON_CACHE", + ] + + +def test_parse_discovery_multiline_body(shadow_knowledge): + """Lines that are neither metadata nor structured extras are appended to + the body text.""" + line = "- Lead sentence." + cont = [ + " continuation prose", + " _(verified, source: exploration)_", + ] + d = shadow_knowledge.parse_discovery(line, cont) + assert "Lead sentence." in d["text"] + assert "continuation prose" in d["text"] + assert d["status"] == "verified" + + +def test_parse_discovery_no_metadata(shadow_knowledge): + """Bullet with no metadata blob still returns a dict with text but no + status/source keys.""" + d = shadow_knowledge.parse_discovery("- bare bullet", []) + assert d["text"] == "bare bullet" + assert "status" not in d + assert "source" not in d + + +def test_parse_discovery_preferences_source_only(shadow_knowledge): + """Preferences use `_(source: user)_` (no status). Extracts source.""" + d = shadow_knowledge.parse_discovery( + "- Prefer X over Y", [" _(source: user)_"] + ) + assert d["text"] == "Prefer X over Y" + assert d["source"] == "user" + assert "status" not in d + + +def test_parse_discovery_none_input_does_not_crash(shadow_knowledge): + """Passing a non-string line shouldn't raise — should return a dict.""" + d = shadow_knowledge.parse_discovery(None, None) + assert isinstance(d, dict) + assert "text" in d + + +# --- parse_shadow_file ----------------------------------------------------- + + +def test_parse_shadow_file_placeholder(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "foo.py.md", """\ + # Shadow: foo.py + + **Language**: Python | **Lines**: 10 + + _No discoveries yet._ + """) + res = shadow_knowledge.parse_shadow_file(f) + assert res["source_file"] == "foo.py" + assert res["language"] == "Python" + assert res["lines"] == 10 + assert res["symbols"] == [] + assert res["discoveries"] == [] + assert res["cross_references"] == [] + assert res["parse_errors"] == [] + + +def test_parse_shadow_file_one_symbol_one_discovery(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "auth.py.md", """\ + # Shadow: auth.py + + **Language**: Python | **Lines**: 42 + + ## `authenticate` + + - Returns None on expired tokens, silently. + _(verified, source: exploration, labels: [security])_ + """) + res = shadow_knowledge.parse_shadow_file(f) + assert res["source_file"] == "auth.py" + assert res["symbols"] == ["authenticate"] + assert len(res["discoveries"]) == 1 + d = res["discoveries"][0] + assert d["symbol"] == "authenticate" + assert d["file"] == "auth.py" + assert d["status"] == "verified" + assert d["source"] == "exploration" + assert d["labels"] == ["security"] + assert "Returns None" in d["text"] + + +def test_parse_shadow_file_cross_references_backpointers( + shadow_knowledge, tmp_path +): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "bar.py.md", """\ + # Shadow: bar.py + + ## `func` + + - A discovery. + _(verified, source: exploration)_ + + ## Cross-References + + - [my-cross-cutting](_cross/my-cross-cutting.md) + (involves `bar.py::func`) + - [another-one](_cross/another-one.md) + """) + res = shadow_knowledge.parse_shadow_file(f) + assert res["symbols"] == ["func"] + # Cross-references back-pointer link labels are collected + assert "my-cross-cutting" in res["cross_references"] + assert "another-one" in res["cross_references"] + # Cross-ref bullets are NOT mistaken for discoveries + assert len(res["discoveries"]) == 1 + + +def test_parse_shadow_file_file_level_and_cross_refs(shadow_knowledge, tmp_path): + """`## File-Level` discoveries are tagged with symbol='file-level' and + `## Cross-References` bullets are not treated as discoveries.""" + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "mix.py.md", """\ + # Shadow: mix.py + + ## File-Level + + - A module-wide observation. + _(verified, source: exploration)_ + + ## `helper` + + - A symbol discovery. + _(verified, source: user)_ + + ## Cross-References + + - [shared](_cross/shared.md) + """) + res = shadow_knowledge.parse_shadow_file(f) + discs = res["discoveries"] + assert len(discs) == 2 + by_sym = {d["symbol"]: d for d in discs} + assert "file-level" in by_sym + assert "helper" in by_sym + assert by_sym["file-level"]["source"] == "exploration" + assert by_sym["helper"]["source"] == "user" + assert res["cross_references"] == ["shared"] + + +def test_parse_shadow_file_malformed_does_not_crash(shadow_knowledge, tmp_path): + """Bizarre / structurally broken content shouldn't raise.""" + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "junk.py.md", """\ + # Shadow: junk.py + **Language**: notnumeric | **Lines**: notanumber + + ## not a backtick heading + - orphan bullet with no metadata + ## `realsym` + - real disc + _(verified, source: exploration)_ + """) + res = shadow_knowledge.parse_shadow_file(f) + # Parse succeeds despite weirdness + assert res["source_file"] == "junk.py" + # The bad "Lines" cell stays None (int parse skipped) + assert res["lines"] is None + # `realsym` is captured; the non-backtick heading is not a symbol + assert "realsym" in res["symbols"] + assert "not a backtick heading" not in res["symbols"] + + +def test_parse_shadow_file_missing_file_returns_error( + shadow_knowledge, tmp_path +): + """Reading a non-existent path records a parse_error, doesn't raise.""" + res = shadow_knowledge.parse_shadow_file(tmp_path / "ghost.md") + assert res["parse_errors"] + assert res["symbols"] == [] + assert res["discoveries"] == [] + + +# --- parse_cross_cutting --------------------------------------------------- + + +def test_parse_cross_cutting_empty_dir(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_cross").mkdir() + assert shadow_knowledge.parse_cross_cutting(sd) == [] + + +def test_parse_cross_cutting_no_dir(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + # _cross/ not created + assert shadow_knowledge.parse_cross_cutting(sd) == [] + + +def test_parse_cross_cutting_one_entry(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/example-pattern.md", """\ + # Example pattern + + **Category**: pattern + **Refs**: + - `cart.py::calculate_total` + - `inventory.py::validate_coupon` + + **Discovery**: A multi-file pattern observed across the codebase. + + _(verified, source: exploration, labels: [bug])_ + """) + entries = shadow_knowledge.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["slug"] == "example-pattern" + assert e["title"] == "Example pattern" + assert e["category"] == "pattern" + assert "cart.py::calculate_total" in e["refs"] + assert "inventory.py::validate_coupon" in e["refs"] + assert "multi-file pattern" in e["discovery"] + assert e["status"] == "verified" + assert e["source"] == "exploration" + assert e["labels"] == ["bug"] + + +def test_parse_cross_cutting_no_labels(shadow_knowledge, tmp_path): + """A cross-cutting entry without labels is still parsed, with no + `labels` key in the entry.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-label.md", """\ + # Plain entry + + **Category**: behavior + **Refs**: + - `foo.py::bar` + + **Discovery**: Something happens. + + _(uncertain, source: exploration)_ + """) + entries = shadow_knowledge.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["status"] == "uncertain" + assert e["source"] == "exploration" + assert "labels" not in e + + +def test_parse_cross_cutting_minor_format_variation(shadow_knowledge, tmp_path): + """File missing a `**Category**:` field doesn't crash; entry is still + emitted with whatever fields could be parsed.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/sparse.md", """\ + # Sparse entry + + Some prose with no structured fields. + + _(refuted, source: user)_ + """) + entries = shadow_knowledge.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["slug"] == "sparse" + assert e["title"] == "Sparse entry" + assert e["status"] == "refuted" + assert e["source"] == "user" + + +# --- parse_prefs ----------------------------------------------------------- + + +def test_parse_prefs_missing_file(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + assert shadow_knowledge.parse_prefs(sd) == [] + + +def test_parse_prefs_zero(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + _No preferences recorded yet._ + """) + assert shadow_knowledge.parse_prefs(sd) == [] + + +def test_parse_prefs_one(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Always use type hints on public APIs. + _(source: user)_ + """) + prefs = shadow_knowledge.parse_prefs(sd) + assert len(prefs) == 1 + assert prefs[0]["text"] == "Always use type hints on public APIs." + assert prefs[0]["source"] == "user" + assert prefs[0]["type"] == "preference" + + +def test_parse_prefs_three(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Use kebab-case for slugs. + _(source: user)_ + - Never commit secrets to source control. + _(source: interaction)_ + - Prefer fail-fast for required dependencies. + _(source: user)_ + """) + prefs = shadow_knowledge.parse_prefs(sd) + assert len(prefs) == 3 + texts = [p["text"] for p in prefs] + assert any("kebab-case" in t for t in texts) + assert any("secrets" in t for t in texts) + assert any("fail-fast" in t for t in texts) + sources = [p["source"] for p in prefs] + assert "user" in sources + assert "interaction" in sources + + +# --- load_state ------------------------------------------------------------ + + +def test_load_state_missing(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + assert shadow_knowledge.load_state(sd) == {} + + +def test_load_state_valid_json(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + payload = { + "version": 1, + "total_files": 5, + "total_discoveries": 42, + "last_update_type": "auto", + } + (sd / "_meta" / "state.json").write_text( + json.dumps(payload), encoding="utf-8" + ) + state = shadow_knowledge.load_state(sd) + assert state == payload + + +def test_load_state_malformed_json(shadow_knowledge, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text( + "{not valid json", encoding="utf-8" + ) + state = shadow_knowledge.load_state(sd) + assert state == {} + + +def test_load_state_non_object_json(shadow_knowledge, tmp_path): + """JSON parses but is not a dict — returns empty dict sentinel.""" + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text("[1, 2, 3]", encoding="utf-8") + assert shadow_knowledge.load_state(sd) == {} + + +# --- get_all_shadow_files -------------------------------------------------- + + +def test_get_all_shadow_files_excludes_special(shadow_knowledge, tmp_path): + """Per-file shadows are returned; _cross/, _dreams/, _meta/, _index.md, + _prefs.md, state.json are all excluded.""" + sd = _make_shadow_root(tmp_path) + # Files that SHOULD be returned + _write_shadow(sd, "a.py.md", "# Shadow: a.py\n") + _write_shadow(sd, "src/b.py.md", "# Shadow: src/b.py\n") + _write_shadow(sd, "deep/nested/c.py.md", "# Shadow: deep/nested/c.py\n") + # Files that should be EXCLUDED + _write_shadow(sd, "_index.md", "# Shadow Index\n") + _write_shadow(sd, "_prefs.md", "# Preferences\n") + _write_shadow(sd, "_cross/some.md", "# Some cross\n") + _write_shadow(sd, "_dreams/dream-1/report.md", "# A dream\n") + _write_shadow(sd, "_meta/state.json", "{}") + # Junk files that are not .md and shouldn't show up anyway + (sd / "notes.json").write_text("{}", encoding="utf-8") + + files = shadow_knowledge.get_all_shadow_files(sd) + rels = sorted(f.relative_to(sd).as_posix() for f in files) + assert rels == ["a.py.md", "deep/nested/c.py.md", "src/b.py.md"] + + +def test_get_all_shadow_files_against_coupon_demo(shadow_knowledge, coupon_demo): + sd = coupon_demo / ".shadow" + files = shadow_knowledge.get_all_shadow_files(sd) + rels = sorted(f.relative_to(sd).as_posix() for f in files) + assert rels == ["cart.py.md", "inventory.py.md", "test_cart.py.md"] + # Make sure none of the special files leaked through + for r in rels: + assert not r.startswith(("_cross/", "_dreams/", "_meta/")) + assert r not in ("_index.md", "_prefs.md") + + +# --- collect_all_discoveries (against fixture) ----------------------------- + + +def test_collect_all_discoveries_counts(shadow_knowledge, coupon_demo): + """Coupon demo has 33 per-file discoveries (matches state.json).""" + sd = coupon_demo / ".shadow" + all_disc = shadow_knowledge.collect_all_discoveries(sd) + # state.json claims 33 total discoveries + assert len(all_disc) == 33 + + # File breakdown: cart=14, inventory=10, test_cart=9 + by_file = {} + for d in all_disc: + by_file.setdefault(d.get("file"), 0) + by_file[d.get("file")] += 1 + assert by_file == {"cart.py": 14, "inventory.py": 10, "test_cart.py": 9} + + +def test_collect_all_discoveries_shadow_path_and_mtime( + shadow_knowledge, coupon_demo +): + sd = coupon_demo / ".shadow" + all_disc = shadow_knowledge.collect_all_discoveries(sd) + for d in all_disc: + assert "shadow_path" in d + assert d["shadow_path"].endswith(".md") + assert "shadow_mtime" in d + assert isinstance(d["shadow_mtime"], float) + + +def test_collect_all_discoveries_b3_no_dream_report_in_text( + shadow_knowledge, coupon_demo +): + """B3 regression on real fixture: no discovery body should contain + 'Dream report' or the literal `_dreams/` slug path.""" + sd = coupon_demo / ".shadow" + all_disc = shadow_knowledge.collect_all_discoveries(sd) + leaks = [ + d for d in all_disc + if "Dream report" in d.get("text", "") + or "_dreams/" in d.get("text", "") + ] + assert leaks == [], ( + f"Dream report leaked into {len(leaks)} discovery body/bodies: " + f"{[d['text'][:80] for d in leaks]}" + ) + + # And at least some discoveries actually have dream_report metadata + # (the fixture has several Dream report continuation lines) + with_dream = [d for d in all_disc if d.get("dream_report")] + assert len(with_dream) >= 3, ( + "Expected coupon-demo fixture to have multiple Dream-report-tagged " + f"discoveries; found {len(with_dream)}" + ) + for d in with_dream: + assert d["dream_report"].startswith("_dreams/") + + +# --- CLI: --summary -------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_summary(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--summary") + assert r.returncode == 0, r.stderr + out = r.stdout + # Header is "Files shadowed:", "Symbols tracked:", "Discoveries:" per + # current --summary output. Task wording used "Total files:"/"Symbols:" + # /"Discoveries:" loosely — match the actual labels. + assert "Files shadowed:" in out + assert "Symbols tracked:" in out + assert "Discoveries:" in out + # Cross-cutting block exists + assert "Cross-cutting" in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_default_view_is_summary(repo_root, coupon_demo): + """Running with no flags should produce the summary view.""" + r = _run_viewer(repo_root, coupon_demo) + assert r.returncode == 0, r.stderr + assert "Shadow Knowledge Base Summary" in r.stdout + + +# --- CLI: --search --------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_search_matches_text(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--search", "coupon") + assert r.returncode == 0, r.stderr + # Header echoes the query and results were found + assert "'coupon'" in r.stdout + # Some matched line should contain the query (case-insensitive) + assert "coupon" in r.stdout.lower() + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_search_no_results(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--search", "zzznotpresentzzz") + assert r.returncode == 0, r.stderr + assert "No results for 'zzznotpresentzzz'" in r.stdout + + +# --- CLI: --top ------------------------------------------------------------ + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_cart_py_header_and_no_dream_report_leak( + repo_root, coupon_demo +): + """B3 regression at the CLI layer: --top output for a file whose shadow + contains `Dream report:` continuation lines must NOT inline that text + in any discovery body.""" + r = _run_viewer(repo_root, coupon_demo, "--top", "cart.py") + assert r.returncode == 0, r.stderr + out = r.stdout + # Header form: "Top N of M actionable discoveries for cart.py:" + assert "actionable discoveries for cart.py:" in out + # Match the documented prefix exactly + assert out.splitlines()[0].startswith("Top ") + assert "for cart.py:" in out.splitlines()[0] + + # B3: no literal "Dream report:" or raw `_dreams/...` slug paths in any + # of the discovery bullets + assert "Dream report:" not in out + assert "_dreams/" not in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_label_filter_bug(repo_root, coupon_demo): + r = _run_viewer( + repo_root, coupon_demo, "--top", "cart.py", "--top-labels", "bug" + ) + assert r.returncode == 0, r.stderr + out = r.stdout + assert "for cart.py:" in out + # Every bulleted result line should mention 'bug' in its label bracket + bullet_lines = [ + l for l in out.splitlines() if l.startswith("- [") + ] + assert bullet_lines, f"No bullets in --top output:\n{out}" + for line in bullet_lines: + bracket = line.split("]", 1)[0] + assert "bug" in bracket, ( + f"Expected 'bug' label in bracket of: {line}" + ) + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_unknown_file(repo_root, coupon_demo): + """A file with no shadow + no cross refs reports nothing actionable.""" + r = _run_viewer(repo_root, coupon_demo, "--top", "does/not/exist.py") + assert r.returncode == 0, r.stderr + assert "No actionable discoveries" in r.stdout + + +# --- CLI: --labels (repo-wide) --------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_labels_bug(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--labels", "bug") + assert r.returncode == 0, r.stderr + out = r.stdout + assert "label(s): bug" in out + # There are multiple bug-labeled discoveries in the fixture + assert "[bug]" in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_labels_unknown(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--labels", "nonexistent") + assert r.returncode == 0, r.stderr + assert "No discoveries with label(s): nonexistent" in r.stdout + + +# --- CLI: --recent --------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_recent_caps_at_n(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--recent", "5") + assert r.returncode == 0, r.stderr + out = r.stdout + assert "Most Recent Discoveries (top 5)" in out + # Each recent entry has a timestamp prefix " [YYYY-MM-DD HH:MM]" + entries = [l for l in out.splitlines() if l.strip().startswith("[20")] + assert len(entries) <= 5 + # Coupon-demo has > 5 total items so we expect exactly 5 + assert len(entries) == 5 + + +# --- CLI: --prefs ---------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_prefs_empty(repo_root, coupon_demo): + """Coupon-demo ships with no preferences recorded.""" + r = _run_viewer(repo_root, coupon_demo, "--prefs") + assert r.returncode == 0, r.stderr + assert "No preferences recorded yet." in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_prefs_populated(repo_root, coupon_demo): + """After populating _prefs.md, --prefs lists them.""" + prefs_path = coupon_demo / ".shadow" / "_prefs.md" + prefs_path.write_text( + textwrap.dedent("""\ + # Preferences + + - Use snake_case for Python identifiers. + _(source: user)_ + - Avoid mutable default arguments. + _(source: interaction)_ + """), + encoding="utf-8", + ) + r = _run_viewer(repo_root, coupon_demo, "--prefs") + assert r.returncode == 0, r.stderr + assert "Project Preferences (2 total)" in r.stdout + assert "snake_case" in r.stdout + assert "mutable default" in r.stdout + + +# --- CLI: --check-invariants ----------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_clean(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode == 0, ( + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + assert "✓ Invariants OK" in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_detects_missing_cross_file( + repo_root, coupon_demo +): + """Deleting a _cross/ file leaves dangling back-pointers in per-file + shadows. Invariant #5 must flag this as a violation.""" + cross_path = ( + coupon_demo / ".shadow" / "_cross" + / "coupon-case-normalization-mismatch.md" + ) + assert cross_path.is_file() + cross_path.unlink() + + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode != 0, ( + f"Expected nonzero exit when _cross/ file missing; got 0.\n" + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + # Violations report the dangling slug + assert "coupon-case-normalization-mismatch" in r.stdout + assert "cross-ref" in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_detects_bad_heading(repo_root, coupon_demo): + """Renaming a symbol heading to a non-backtick form is a heading-format + violation.""" + cart_path = coupon_demo / ".shadow" / "cart.py.md" + text = cart_path.read_text(encoding="utf-8") + # `## `COUPON_CACHE`` -> `## COUPON_CACHE` (drop backticks) + mutated = text.replace("## `COUPON_CACHE`", "## COUPON_CACHE", 1) + assert mutated != text, "Substitution did not match" + cart_path.write_text(mutated, encoding="utf-8") + + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode != 0, ( + f"Expected nonzero exit for bad heading; got 0.\n" + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + assert "heading" in r.stdout + + +# --- CLI: missing shadow dir ---------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_missing_shadow_dir_fails(repo_root, tmp_path): + """Running with no shadow and a bogus --shadow-dir exits 1.""" + r = _run_viewer( + repo_root, tmp_path, + "--shadow-dir", str(tmp_path / "nope"), "--summary", + ) + assert r.returncode == 1 + assert "No .shadow/ directory found" in r.stderr + + +# =========================================================================== +# RENDER FUNCTIONS: in-process tests +# +# The CLI integration tests above invoke shadow-viewer.py as a subprocess — +# that exercises the dispatcher but doesn't contribute to coverage of the +# loaded module. The tests below call view_* and main() directly via the +# `shadow_knowledge` fixture so coverage actually accumulates. +# =========================================================================== + + +# --- shared helpers -------------------------------------------------------- + + +def _call_main(shadow_knowledge, argv): + """Invoke `shadow_knowledge.main()` in-process with the given argv. + + Returns the integer exit code. We mutate `sys.argv` directly (no + monkeypatch / no mocking) and always restore it in a finally. + """ + saved_argv = sys.argv + sys.argv = ["shadow-viewer.py", *argv] + try: + shadow_knowledge.main() + return 0 + except SystemExit as e: + code = e.code + if code is None: + return 0 + if isinstance(code, int): + return code + return 1 + finally: + sys.argv = saved_argv + + +def _make_minimal_shadow(tmp_path, extras=None): + """Build a minimal but valid .shadow/ tree in tmp_path. + + Returns the .shadow/ Path. `extras` is a dict of {relpath: content} + appended on top of the baseline. + """ + sd = tmp_path / ".shadow" + sd.mkdir() + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text( + json.dumps({ + "version": 1, + "last_update_at": "2026-04-20T16:30:00Z", + "last_update_type": "manual", + "last_commit": "deadbeef" * 5, + }), + encoding="utf-8", + ) + _write_shadow(sd, "foo.py.md", """\ + # Shadow: foo.py + + **Language**: Python | **Lines**: 10 + + ## `bar` + + - A neat bug. + _(verified, source: exploration, labels: [bug])_ + """) + for rel, content in (extras or {}).items(): + _write_shadow(sd, rel, content) + return sd + + +# =========================================================================== +# view_summary +# =========================================================================== + + +class TestViewSummary: + """In-process tests for `view_summary(shadow_dir)`.""" + + def test_basic_header_on_coupon_demo( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Shadow Knowledge Base Summary" in out + assert "=" * 50 in out + assert "Files shadowed:" in out + assert "Symbols tracked:" in out + assert "Discoveries:" in out + assert "Preferences:" in out + assert "Cross-cutting:" in out + + def test_counts_reflect_fixture( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + # 3 source files, 33 discoveries, 3 cross-cutting (per fixture) + assert "Files shadowed: 3" in out + assert "Discoveries: 33" in out + assert "Cross-cutting: 3" in out + + def test_by_source_section_renders( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "By source:" in out + # All discoveries in the fixture are source: exploration + assert "exploration" in out + + def test_by_status_section_renders( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "By status:" in out + assert "verified" in out + + def test_by_label_section_renders( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "By label:" in out + assert "bug" in out + assert "security" in out + + def test_per_file_table_lists_files( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + # Per-file table header + assert "File" in out + assert "Symbols" in out + assert "Disc." in out + # All three fixture files appear in the table + assert "cart.py" in out + assert "inventory.py" in out + assert "test_cart.py" in out + + def test_cross_cutting_titles_section( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Cross-cutting discoveries:" in out + assert "Coupon case normalization mismatch" in out + assert "[edge-case]" in out + + def test_state_info_section( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Last update:" in out + assert "Last commit:" in out + # The fixture state.json says last_update_type: dream + assert "(dream)" in out + + def test_empty_shadow_dir_renders_zeros( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Files shadowed: 0" in out + assert "Discoveries: 0" in out + # With zero discoveries there's no source/status breakdown + assert "By source:" not in out + assert "By status:" not in out + assert "By label:" not in out + # And no state info (no state.json) + assert "Last update:" not in out + + def test_corrupted_state_json_does_not_break_summary( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # Clobber state.json with junk + (sd / "_meta" / "state.json").write_text( + "{ not json", encoding="utf-8" + ) + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + # Header and counts still rendered + assert "Shadow Knowledge Base Summary" in out + assert "Files shadowed: 1" in out + # State section silently dropped + assert "Last update:" not in out + + def test_unreadable_shadow_file_does_not_break_summary( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # Create a file with invalid UTF-8 — parse_shadow_file records a + # parse_error but returns a result. view_summary should still + # render the rest of the report. + bad = sd / "bad.py.md" + bad.write_bytes(b"\xff\xfe\x00garbage\x00") + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Shadow Knowledge Base Summary" in out + assert "Files shadowed: 2" in out + + def test_more_than_20_files_truncated( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + for i in range(25): + _write_shadow( + sd, f"file{i:02d}.py.md", + f"# Shadow: file{i:02d}.py\n\n## `sym{i}`\n\n" + f"- D\n _(verified, source: exploration)_\n", + ) + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Files shadowed: 25" in out + assert "... and 5 more files" in out + + def test_no_cross_cutting_dir_omits_section( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # no _cross/ dir + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Cross-cutting: 0" in out + # The titled list is suppressed when empty + assert "Cross-cutting discoveries:" not in out + + def test_summary_no_prefs(self, shadow_knowledge, tmp_path, capsys): + sd = _make_minimal_shadow(tmp_path) + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Preferences: 0" in out + + def test_summary_with_prefs(self, shadow_knowledge, tmp_path, capsys): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Pref one. + _(source: user)_ + - Pref two. + _(source: interaction)_ + """) + shadow_knowledge.view_summary(sd) + out = capsys.readouterr().out + assert "Preferences: 2" in out + + +# =========================================================================== +# view_search +# =========================================================================== + + +class TestViewSearch: + """In-process tests for `view_search(shadow_dir, query)`.""" + + def test_finds_text_match(self, shadow_knowledge, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "tax") + out = capsys.readouterr().out + assert "Search: 'tax'" in out + assert "results" in out + # The "8% tax" / "Tax rate" discoveries on cart.py match + assert "cart.py" in out + + def test_finds_symbol_match(self, shadow_knowledge, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "COUPON_CACHE") + out = capsys.readouterr().out + assert "COUPON_CACHE" in out + assert "cart.py" in out + + def test_finds_file_name_match( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Searching for the literal file name pulls every discovery in + # that file (file_name_hit branch). + shadow_knowledge.view_search(sd, "inventory.py") + out = capsys.readouterr().out + assert "inventory.py" in out + assert "matches" in out + + def test_case_insensitive(self, shadow_knowledge, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "COUPON") + upper = capsys.readouterr().out + shadow_knowledge.view_search(sd, "coupon") + lower = capsys.readouterr().out + # Same number of result lines either way + assert ("results" in upper) and ("results" in lower) + # And both contain at least one match + assert "::" in upper + assert "::" in lower + + def test_no_matches_prints_no_results( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "zzzdoesnotexistzzz") + out = capsys.readouterr().out + assert "No results for 'zzzdoesnotexistzzz'." in out + + def test_finds_cross_cutting_title( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Title of _cross/coupon-case-normalization-mismatch.md is + # "Coupon case normalization mismatch" + shadow_knowledge.view_search(sd, "normalization mismatch") + out = capsys.readouterr().out + assert "Cross-cutting" in out + assert "Coupon case normalization mismatch" in out + assert "Category:" in out + assert "edge-case" in out + + def test_finds_cross_cutting_by_ref( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Search for a ref that appears in a _cross file + shadow_knowledge.view_search(sd, "apply_bulk_discount") + out = capsys.readouterr().out + assert "Cross-cutting" in out + assert "Mutation through discount pipeline" in out + + def test_finds_also_involves_match( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + Also involves: `b.py::weird_symbol_zzz` + """) + shadow_knowledge.view_search(sd, "weird_symbol_zzz") + out = capsys.readouterr().out + # The match flag is "also_involves" and a line shows that ref + assert "weird_symbol_zzz" in out + assert "Also involves:" in out + + def test_finds_preference_match( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Prefer rusty pelicans for everything. + _(source: user)_ + """) + shadow_knowledge.view_search(sd, "pelican") + out = capsys.readouterr().out + assert "Preferences" in out + assert "pelican" in out.lower() + assert "[user]" in out + + def test_groups_per_file_results_by_file( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "coupon") + out = capsys.readouterr().out + # Per-file groups present a header like "cart.py (N matches)" + assert re.search(r"cart\.py \(\d+ matches\)", out) + + def test_results_show_status_and_source( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "tax") + out = capsys.readouterr().out + assert "(verified, source: exploration)" in out + + def test_total_count_in_header( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_search(sd, "coupon") + out = capsys.readouterr().out + m = re.search(r"\((\d+) results\)", out) + assert m is not None + assert int(m.group(1)) > 0 + + def test_empty_shadow_returns_no_results( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_knowledge.view_search(sd, "anything") + out = capsys.readouterr().out + assert "No results for 'anything'." in out + + +# =========================================================================== +# view_prefs +# =========================================================================== + + +class TestViewPrefs: + """In-process tests for `view_prefs(shadow_dir)`.""" + + def test_missing_prefs_file(self, shadow_knowledge, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + shadow_knowledge.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + def test_empty_prefs_placeholder( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + _No preferences recorded yet._ + """) + shadow_knowledge.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + def test_populated_prefs_lists_count_and_sources( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Use snake_case. + _(source: user)_ + - Avoid mutable default args. + _(source: interaction)_ + - Prefer fail-fast for required deps. + _(source: user)_ + """) + shadow_knowledge.view_prefs(sd) + out = capsys.readouterr().out + assert "Project Preferences (3 total)" in out + assert "[user]" in out + assert "[interaction]" in out + assert "snake_case" in out + assert "fail-fast" in out + assert "mutable default" in out + + def test_against_coupon_demo_is_empty( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + +# =========================================================================== +# view_labels +# =========================================================================== + + +class TestViewLabels: + """In-process tests for `view_labels(shadow_dir, label_filter)`.""" + + @pytest.mark.parametrize("label", [ + "bug", "security", "performance", "feature-gap", "tech-debt", + ]) + def test_each_label_returns_results_on_fixture( + self, shadow_knowledge, coupon_demo, capsys, label + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels(sd, label) + out = capsys.readouterr().out + assert f"label(s): {label}" in out + assert f"[{label}]" in out + # Result header always has "(N results)" + m = re.search(r"\((\d+) results\)", out) + assert m and int(m.group(1)) >= 1 + + def test_unknown_label_no_results( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels(sd, "zznotalabelzz") + out = capsys.readouterr().out + assert "No discoveries with label(s): zznotalabelzz" in out + + def test_multiple_labels_comma_split( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels(sd, "bug,security") + out = capsys.readouterr().out + assert "label(s): bug, security" in out + # Both grouped section headers present + assert "[bug]" in out + assert "[security]" in out + + def test_label_filter_is_lowercased( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels(sd, "BUG") + out = capsys.readouterr().out + # Filter is lowercased before matching + assert "label(s): bug" in out + assert "[bug]" in out + + def test_cross_cutting_labels_included( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels( + sd, "bug", options=shadow_knowledge.RetrievalOptions(limit=20, max_chars=8000), + ) + out = capsys.readouterr().out + # Cross-cutting entries are prefixed with `_cross/` in the + # file column. + assert "_cross/" in out + + def test_also_labeled_displayed( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # One cart.py discovery has labels [bug, performance] + shadow_knowledge.view_labels(sd, "bug") + out = capsys.readouterr().out + assert "Also labeled:" in out + + def test_empty_shadow_no_results( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_knowledge.view_labels(sd, "bug") + out = capsys.readouterr().out + assert "No discoveries with label(s): bug" in out + + def test_each_result_row_has_file_and_symbol( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_labels(sd, "feature-gap") + out = capsys.readouterr().out + # `::` separator for file::symbol form + assert "::" in out + assert "(verified, source: exploration)" in out + + +# =========================================================================== +# view_recent +# =========================================================================== + + +class TestViewRecent: + """In-process tests for `view_recent(shadow_dir, count)`.""" + + def test_default_count_caps_at_10( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_recent(sd, 10) + out = capsys.readouterr().out + assert "Most Recent Discoveries (top 10)" in out + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + # Fixture has 33 per-file + 3 cross + 0 prefs > 10 + assert len(entries) == 10 + + def test_custom_count(self, shadow_knowledge, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_recent(sd, 3) + out = capsys.readouterr().out + assert "Most Recent Discoveries (top 3)" in out + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + assert len(entries) == 3 + + def test_count_larger_than_available( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + shadow_knowledge.view_recent(sd, 50) + out = capsys.readouterr().out + # Only 1 discovery exists in the minimal shadow + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + assert len(entries) == 1 + + def test_empty_shadow_prints_nothing( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_knowledge.view_recent(sd, 10) + out = capsys.readouterr().out + assert "No discoveries found." in out + + def test_recently_modified_file_appears_first( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Bump test_cart.py's mtime to "now" so its discoveries should + # rank first. + target = sd / "test_cart.py.md" + now = datetime.now().timestamp() + os.utime(target, (now + 10, now + 10)) + shadow_knowledge.view_recent(sd, 3) + out = capsys.readouterr().out + # First entry block should reference test_cart.py + first_entry_idx = out.find("[20") + first_block = out[first_entry_idx:first_entry_idx + 400] + assert "test_cart.py" in first_block + + def test_includes_cross_cutting_type( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Bump a cross file mtime so it shows up in the top + cf = sd / "_cross" / "coupon-case-normalization-mismatch.md" + now = datetime.now().timestamp() + os.utime(cf, (now + 100, now + 100)) + shadow_knowledge.view_recent(sd, 5) + out = capsys.readouterr().out + assert "(cross-cutting)" in out + + def test_includes_preferences_when_present( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - A pref we care about. + _(source: user)_ + """) + shadow_knowledge.view_recent(sd, 10) + out = capsys.readouterr().out + assert "(preference)" in out + assert "A pref we care about" in out + # Preference-typed rows show `source:` not `(verified, ...)` + assert "source: user" in out + + def test_no_cross_dir_does_not_crash( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # No _cross/ created + shadow_knowledge.view_recent(sd, 10) + out = capsys.readouterr().out + assert "Most Recent Discoveries" in out + + def test_entries_include_timestamp_and_kind( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_knowledge.view_recent(sd, 2) + out = capsys.readouterr().out + # Entry header line format: " [YYYY-MM-DD HH:MM] (kind)" + assert re.search( + r"\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}\] \((discovery|cross-cutting|preference)\)", + out, + ) + + +# =========================================================================== +# view_check_invariants +# =========================================================================== + + +class TestViewCheckInvariants: + """In-process tests for `view_check_invariants(shadow_dir)`. + + Each test builds a deliberately broken `.shadow/` tree in tmp_path + and asserts that the right violation kind is reported. + """ + + def test_clean_coupon_demo_returns_zero( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = shadow_knowledge.view_check_invariants(coupon_demo / ".shadow") + out = capsys.readouterr().out + assert rc == 0 + assert "✓ Invariants OK" in out + + def test_empty_shadow_returns_zero( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0 + assert "Invariants OK" in out + + def test_missing_cross_file_is_violation( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + (sd / "_cross" / "coupon-case-normalization-mismatch.md").unlink() + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "cross-ref" in out + assert "coupon-case-normalization-mismatch" in out + + def test_invalid_status_enum(self, shadow_knowledge, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad status. + _(maybeverified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "maybeverified" in out + + def test_invalid_source_enum(self, shadow_knowledge, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad source. + _(verified, source: psychic)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "psychic" in out + + def test_invalid_label(self, shadow_knowledge, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad label. + _(verified, source: exploration, labels: [unicorn])_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "unicorn" in out + + def test_heading_without_backticks( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_in_backticks + + - Hi. + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "heading" in out + + def test_file_level_heading_does_not_violate( + self, shadow_knowledge, tmp_path, capsys + ): + """`## File-Level` is allowed without backticks.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## File-Level + + - A file-level discovery. + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0, out + assert "Invariants OK" in out + + def test_also_involves_without_backticks( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - With bad anchors. + _(verified, source: exploration)_ + Also involves: b.py::bar, c.py::baz + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "anchor" in out + + def test_also_involves_missing_symbol_after_colons( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Empty sym. + _(verified, source: exploration)_ + Also involves: `b.py::` + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + # The empty-symbol form fails the strict regex (which requires + # non-empty after ::), so the file_sym_re finds zero anchors and + # we hit the "needs backtick anchors" branch instead. + assert "anchor" in out + assert "needs `file::symbol`" in out + + def test_cross_missing_category( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + """) + _write_shadow(sd, "_cross/no-cat.md", """\ + # No category here + + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "Category" in out + + def test_cross_invalid_category( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/bad-cat.md", """\ + # Bad category here + + **Category**: bogus + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "bogus" in out + + def test_cross_missing_metadata_line( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-meta.md", """\ + # No meta here + + **Category**: pattern + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "missing trailing" in out + + def test_cross_missing_refs_block( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-refs.md", """\ + # No refs here + + **Category**: pattern + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "Refs" in out + + def test_cross_ref_missing_symbol( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + # Note: file_sym_re demands "::" in backticks. We use a backtick + # ref with no symbol after `::`. + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - hi. + _(verified, source: exploration)_ + + ## Cross-References + + - [bad-anchor](_cross/bad-anchor.md) + """) + _write_shadow(sd, "_cross/bad-anchor.md", """\ + # Bad anchor + + **Category**: pattern + **Refs**: + - `a.py::` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "anchor" in out + + def test_back_pointer_missing( + self, shadow_knowledge, tmp_path, capsys + ): + """_cross/x.md references a.py::foo but a.py.md has no + Cross-References section pointing back to x.md.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + """) + _write_shadow(sd, "_cross/orphan.md", """\ + # Orphan + + **Category**: pattern + **Refs**: + - `a.py::foo` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "cross-ref" in out + assert "does not link back to" in out + + def test_back_pointer_references_nonexistent_file( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/ghost.md", """\ + # Ghost + + **Category**: pattern + **Refs**: + - `does/not/exist.py::foo` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "no such shadow file exists" in out + + def test_violation_line_format( + self, shadow_knowledge, tmp_path, capsys + ): + """Output is grep-friendly: `path:line: kind: message`.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_backticked + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + # At least one line of form path:line: kind: msg + assert re.search(r"a\.py\.md:\d+: heading: ", out) + + def test_multiple_violations_reported( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_backticked + + ## `foo` + + - bad. + _(maybeverified, source: psychic, labels: [unicorn])_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + err = capsys.readouterr().err + assert rc == 1 + # heading + 3 enum violations = at least 4 lines + violation_lines = [ + l for l in out.splitlines() + if re.match(r"^[^:]+:\d+: \w+: ", l) + ] + assert len(violation_lines) >= 3 + + def test_count_summary_emitted_on_stderr( + self, shadow_knowledge, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", "## not_backticked\n") + rc = shadow_knowledge.view_check_invariants(sd) + captured = capsys.readouterr() + assert rc == 1 + # Final count line goes to stderr + assert "invariant violation(s) found" in captured.err + + def test_special_headings_allowed( + self, shadow_knowledge, tmp_path, capsys + ): + """`## Notes`, `## Metadata`, `## File-Level Notes` are allowed + without backticks.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## Notes + + Some prose. + + ## Metadata + + Some metadata. + + ## File-Level Notes + + More prose. + + ## `real_sym` + + - A discovery. + _(verified, source: exploration)_ + """) + rc = shadow_knowledge.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0, out + + +# =========================================================================== +# main() — in-process via sys.argv (covers the dispatcher) +# =========================================================================== + + +class TestMainInProcess: + """Drive `main()` directly so coverage of the dispatch arms is captured. + + All paths use `--shadow-dir ` to avoid relying + on cwd. Where we need a true CLI smoke (e.g., to verify `--help` + output), use subprocess. + """ + + def _shadow(self, coupon_demo): + return str(coupon_demo / ".shadow") + + def test_summary_dispatch( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, ["--shadow-dir", self._shadow(coupon_demo), + "--summary"] + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_no_args_default_is_summary( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, ["--shadow-dir", self._shadow(coupon_demo)] + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_search_dispatch( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--search", "coupon"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Search: 'coupon'" in out + + def test_prefs_dispatch( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), "--prefs"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "No preferences recorded yet." in out + + def test_labels_dispatch_bug( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--labels", "bug"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "label(s): bug" in out + + def test_recent_dispatch_with_n( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--recent", "4"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Most Recent Discoveries (top 4)" in out + + def test_recent_dispatch_no_n( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), "--recent"], + ) + out = capsys.readouterr().out + assert rc == 0 + # Default count is 10 + assert "Most Recent Discoveries (top 10)" in out + + def test_top_dispatch( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--top", "cart.py"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "for cart.py:" in out + + def test_top_dispatch_with_labels_and_limits( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--top", "cart.py", + "--top-labels", "bug", + "--top-limit", "2", + "--top-max-chars", "0"], + ) + out = capsys.readouterr().out + assert rc == 0 + bullets = [l for l in out.splitlines() if l.startswith("- [")] + assert len(bullets) <= 2 + + def test_check_invariants_clean_exits_zero( + self, shadow_knowledge, coupon_demo, capsys + ): + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", self._shadow(coupon_demo), + "--check-invariants"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Invariants OK" in out + + def test_check_invariants_dirty_exits_one( + self, shadow_knowledge, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Inject a heading violation + cart = sd / "cart.py.md" + cart.write_text( + cart.read_text(encoding="utf-8").replace( + "## `COUPON_CACHE`", "## COUPON_CACHE", 1 + ), + encoding="utf-8", + ) + rc = _call_main( + shadow_knowledge, + ["--shadow-dir", str(sd), "--check-invariants"], + ) + out = capsys.readouterr().out + assert rc == 1 + assert "heading" in out + + def test_explicit_shadow_dir_missing( + self, shadow_knowledge, tmp_path, capsys + ): + bogus = tmp_path / "does-not-exist" + rc = _call_main( + shadow_knowledge, ["--shadow-dir", str(bogus), "--summary"] + ) + err = capsys.readouterr().err + assert rc == 1 + assert "No .shadow/ directory found" in err + + def test_auto_detect_via_chdir( + self, shadow_knowledge, coupon_demo, monkeypatch, capsys + ): + """No --shadow-dir: cwd is walked up to find .shadow/.""" + monkeypatch.chdir(coupon_demo) + rc = _call_main(shadow_knowledge, ["--summary"]) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_auto_detect_no_shadow_in_cwd( + self, shadow_knowledge, tmp_path, monkeypatch, capsys + ): + monkeypatch.chdir(tmp_path) + rc = _call_main(shadow_knowledge, ["--summary"]) + err = capsys.readouterr().err + assert rc == 1 + assert "No .shadow/ directory found" in err + + +# --- main(): subprocess smoke (covers true argv parsing, help, errors) ---- + + +class TestMainSubprocess: + """End-to-end CLI smoke. Subprocess output is the contract here — + these don't add coverage but they catch dispatcher / argparse regressions + the in-process tests can't (e.g. --help, mutually exclusive errors).""" + + @pytest.mark.slow + @pytest.mark.integration + def test_help_exits_zero(self, repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--help") + assert r.returncode == 0 + # argparse prints usage and the description + assert "usage:" in r.stdout.lower() + assert "--summary" in r.stdout + assert "--search" in r.stdout + assert "--check-invariants" in r.stdout + + @pytest.mark.slow + @pytest.mark.integration + def test_unknown_flag_exits_nonzero(self, repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--no-such-flag") + assert r.returncode != 0 + assert "unrecognized" in r.stderr or "unrecognized" in r.stdout + + @pytest.mark.slow + @pytest.mark.integration + def test_mutually_exclusive_flags(self, repo_root, coupon_demo): + """--summary and --prefs are in the same exclusive group.""" + r = _run_viewer( + repo_root, coupon_demo, "--summary", "--prefs" + ) + assert r.returncode != 0 + # argparse error mentions "not allowed with" + assert "not allowed with" in r.stderr diff --git a/tests/skills/shadow_frog_viewer/test_retrieval.py b/tests/skills/shadow_frog/test_retrieval.py similarity index 67% rename from tests/skills/shadow_frog_viewer/test_retrieval.py rename to tests/skills/shadow_frog/test_retrieval.py index aa63537..a89447c 100644 --- a/tests/skills/shadow_frog_viewer/test_retrieval.py +++ b/tests/skills/shadow_frog/test_retrieval.py @@ -1,4 +1,4 @@ -"""Citation ranking, bounded context, expansion, and stable pagination.""" +"""Core agent retrieval: citation ranking, bounded context, and stable pagination.""" from dataclasses import replace import io @@ -43,14 +43,14 @@ def text_cursor(text): return match.group(1) if match else None -def test_only_emitted_entries_count_and_no_markdown_changes(shadow_viewer, tmp_path, capsys): +def test_only_emitted_entries_count_and_no_markdown_changes(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 12) before = {path: path.read_bytes() for path in shadow.rglob("*") if path.is_file()} - entries = shadow_viewer._knowledge_entries(shadow) - store = shadow_viewer.CitationStore.for_shadow(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) assert store.scores([entry["id"] for entry in entries]) == {} - shadow_viewer.view_search( - shadow, "Claim", options=shadow_viewer.RetrievalOptions(limit=2, event_id="first"), + shadow_knowledge.view_search( + shadow, "Claim", options=shadow_knowledge.RetrievalOptions(limit=2, event_id="first"), ) output = capsys.readouterr().out emitted = ids(output) @@ -60,45 +60,45 @@ def test_only_emitted_entries_count_and_no_markdown_changes(shadow_viewer, tmp_p assert sorted(path.name for path in shadow.iterdir()) == ["source.py.md"] -def test_retries_and_no_record_do_not_double_count(shadow_viewer, tmp_path, capsys): +def test_retries_and_no_record_do_not_double_count(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) - options = shadow_viewer.RetrievalOptions(event_id="retry") + options = shadow_knowledge.RetrievalOptions(event_id="retry") for _ in range(2): - shadow_viewer.view_search(shadow, "Claim", options=options) + shadow_knowledge.view_search(shadow, "Claim", options=options) output = capsys.readouterr().out identity = ids(output)[0] - store = shadow_viewer.CitationStore.for_shadow(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) assert store.scores([identity]) == {identity: 1} - shadow_viewer.view_get(shadow, identity, options=replace(options, record=False)) + shadow_knowledge.view_get(shadow, identity, options=replace(options, record=False)) capsys.readouterr() assert store.scores([identity]) == {identity: 1} -def test_ten_thousand_claims_have_bounded_output(shadow_viewer, tmp_path, capsys): +def test_ten_thousand_claims_have_bounded_output(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 10000) - options = shadow_viewer.RetrievalOptions(limit=10, max_chars=1800) + options = shadow_knowledge.RetrievalOptions(limit=10, max_chars=1800) started = time.monotonic() - shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) output = capsys.readouterr().out assert len(output) <= 1800 assert "10000 results" in output and len(ids(output)) <= 10 assert ids(output) and cursor(output) assert time.monotonic() - started < 10 - store = shadow_viewer.CitationStore.for_shadow(shadow) - all_ids = [entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)] + store = shadow_knowledge.CitationStore.for_shadow(shadow) + all_ids = [entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)] assert len(all_ids) == len(set(all_ids)) == 10000 assert store.scores(all_ids) == dict.fromkeys(ids(output), 1) -def test_cursor_is_stable_despite_other_readers_updating_scores(shadow_viewer, tmp_path, capsys): +def test_cursor_is_stable_despite_other_readers_updating_scores(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 23, text_size=2) - options = shadow_viewer.RetrievalOptions(limit=3, max_chars=1400) - entries = shadow_viewer._knowledge_entries(shadow) + options = shadow_knowledge.RetrievalOptions(limit=3, max_chars=1400) + entries = shadow_knowledge._knowledge_entries(shadow) expected = {entry["id"] for entry in entries} found = [] next_cursor = None while True: - shadow_viewer.view_symbol( + shadow_knowledge.view_symbol( shadow, "source.py::run", options=replace(options, cursor=next_cursor), ) output = capsys.readouterr().out @@ -107,35 +107,35 @@ def test_cursor_is_stable_despite_other_readers_updating_scores(shadow_viewer, t next_cursor = cursor(output) if next_cursor is None: break - shadow_viewer.CitationStore.for_shadow(shadow).record( + shadow_knowledge.CitationStore.for_shadow(shadow).record( [entries[-1]["id"]], f"concurrent-{len(found)}", ) assert len(found) == len(set(found)) == 23 assert set(found) == expected -def test_changed_knowledge_invalidates_cursor(shadow_viewer, tmp_path, capsys): +def test_changed_knowledge_invalidates_cursor(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path) - options = shadow_viewer.RetrievalOptions(limit=1) - shadow_viewer.view_search(shadow, "Claim", options=options) + options = shadow_knowledge.RetrievalOptions(limit=1) + shadow_knowledge.view_search(shadow, "Claim", options=options) previous = cursor(capsys.readouterr().out) path = shadow / "source.py.md" path.write_text(path.read_text(encoding="utf-8").replace("details", "different"), encoding="utf-8") with pytest.raises(ValueError, match="changed"): - shadow_viewer.view_search(shadow, "Claim", options=replace(options, cursor=previous)) + shadow_knowledge.view_search(shadow, "Claim", options=replace(options, cursor=previous)) -def test_removed_results_invalidate_cursor_instead_of_claiming_empty_success(shadow_viewer, tmp_path, capsys): +def test_removed_results_invalidate_cursor_instead_of_claiming_empty_success(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path) - options = shadow_viewer.RetrievalOptions(limit=1) - shadow_viewer.view_search(shadow, "Claim", options=options) + options = shadow_knowledge.RetrievalOptions(limit=1) + shadow_knowledge.view_search(shadow, "Claim", options=options) previous = cursor(capsys.readouterr().out) (shadow / "source.py.md").unlink() with pytest.raises(ValueError, match="changed"): - shadow_viewer.view_search(shadow, "Claim", options=replace(options, cursor=previous)) + shadow_knowledge.view_search(shadow, "Claim", options=replace(options, cursor=previous)) -def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_viewer, tmp_path): +def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_knowledge, tmp_path): entries = [ {"id": "user", "source": "user", "status": "verified"}, {"id": "popular", "source": "exploration", "status": "verified"}, @@ -145,18 +145,18 @@ def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_viewer, tm {"id": "refuted", "source": "user", "status": "refuted"}, ] scores = {"popular": 100, "popular2": 90, "popular3": 80, "refuted": 10000} - ranked = shadow_viewer._rank_entries(entries, scores, 3) + ranked = shadow_knowledge._rank_entries(entries, scores, 3) assert ranked[0]["id"] == "user" and ranked[-1]["id"] == "refuted" assert [entry["id"] for entry in ranked][:3] == ["user", "popular", "new"] -def test_zero_score_opportunity_accounts_for_character_budget(shadow_viewer, tmp_path, capsys): +def test_zero_score_opportunity_accounts_for_character_budget(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 7) - entries = shadow_viewer._knowledge_entries(shadow) - store = shadow_viewer.CitationStore.for_shadow(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) store.record([entry["id"] for entry in entries[:-1]], "prior-read") - shadow_viewer.view_search( - shadow, "Claim", options=shadow_viewer.RetrievalOptions(limit=10, max_chars=1000), + shadow_knowledge.view_search( + shadow, "Claim", options=shadow_knowledge.RetrievalOptions(limit=10, max_chars=1000), ) output = capsys.readouterr().out assert len(output) <= 1000 @@ -165,21 +165,21 @@ def test_zero_score_opportunity_accounts_for_character_budget(shadow_viewer, tmp @pytest.mark.parametrize("view", ["top", "symbol"]) -def test_zero_score_slot_accounts_for_higher_trust_rows(shadow_viewer, tmp_path, capsys, view): +def test_zero_score_slot_accounts_for_higher_trust_rows(shadow_knowledge, tmp_path, capsys, view): shadow = make_shadow(tmp_path, 5, text_size=2) path = shadow / "source.py.md" path.write_text( path.read_text(encoding="utf-8").replace("source: exploration", "source: user", 1), encoding="utf-8", ) - entries = shadow_viewer._knowledge_entries(shadow) - store = shadow_viewer.CitationStore.for_shadow(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) store.record([entry["id"] for entry in entries[1:4]], "prior") - options = shadow_viewer.RetrievalOptions(limit=3, max_chars=600) + options = shadow_knowledge.RetrievalOptions(limit=3, max_chars=600) if view == "top": - shadow_viewer.view_top(shadow, "source.py", "bug", 3, 600, options=options) + shadow_knowledge.view_top(shadow, "source.py", "bug", 3, 600, options=options) else: - shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) output = capsys.readouterr().out assert len(output) <= 600 assert ids(output)[0] == entries[0]["id"] @@ -187,17 +187,17 @@ def test_zero_score_slot_accounts_for_higher_trust_rows(shadow_viewer, tmp_path, @pytest.mark.parametrize("file", ["cart.py", "inventory.py", "test_cart.py"]) -def test_default_hook_budget_returns_multiple_warnings(shadow_viewer, coupon_demo, capsys, file): - shadow_viewer.view_top(coupon_demo / ".shadow", file, "bug,security", 3, 600) +def test_default_hook_budget_returns_multiple_warnings(shadow_knowledge, coupon_demo, capsys, file): + shadow_knowledge.view_top(coupon_demo / ".shadow", file, "bug,security", 3, 600) output = capsys.readouterr().out assert len(output) <= 600 assert len(ids(output)) == 3 assert "citation_score=" not in output -def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_viewer, tmp_path): +def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_knowledge, tmp_path): shadow = make_shadow(tmp_path, 1) - original = shadow_viewer._knowledge_entries(shadow)[0]["id"] + original = shadow_knowledge._knowledge_entries(shadow)[0]["id"] path = shadow / "source.py.md" path.write_text( path.read_text(encoding="utf-8").replace( @@ -205,40 +205,40 @@ def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_viewer, ), encoding="utf-8", ) - assert shadow_viewer._knowledge_entries(shadow)[0]["id"] == original + assert shadow_knowledge._knowledge_entries(shadow)[0]["id"] == original path.write_text(path.read_text(encoding="utf-8").replace("Claim", "Different claim"), encoding="utf-8") - assert shadow_viewer._knowledge_entries(shadow)[0]["id"] != original + assert shadow_knowledge._knowledge_entries(shadow)[0]["id"] != original -def test_duplicate_preferences_share_one_identity_without_losing_trust(shadow_viewer, tmp_path, capsys): +def test_duplicate_preferences_share_one_identity_without_losing_trust(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) (shadow / "_prefs.md").write_text( "# Preferences\n\n- Preserve the contract.\n _(source: interaction)_\n" "\n- Preserve the contract.\n _(source: user)_\n", encoding="utf-8", ) - shadow_viewer.view_prefs(shadow) + shadow_knowledge.view_prefs(shadow) output = capsys.readouterr().out assert "Project Preferences (1 total)" in output and "[user]" in output assert len(ids(output)) == 1 - assert shadow_viewer.CitationStore.for_shadow(shadow).scores(ids(output)) == dict.fromkeys(ids(output), 1) + assert shadow_knowledge.CitationStore.for_shadow(shadow).scores(ids(output)) == dict.fromkeys(ids(output), 1) @pytest.mark.parametrize("kind", ["class", "interface", "enum", "trait", "struct", "protocol", "module"]) -def test_container_symbols_use_canonical_anchors(shadow_viewer, shadow_init, tmp_path, capsys, kind): +def test_container_symbols_use_canonical_anchors(shadow_knowledge, shadow_init, tmp_path, capsys, kind): shadow = make_shadow(tmp_path, 1) heading = shadow_init.Symbol("Container", kind).heading_text path = shadow / "source.py.md" path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{heading}`"), encoding="utf-8") - shadow_viewer.view_symbol(shadow, "source.py::Container") + shadow_knowledge.view_symbol(shadow, "source.py::Container") result = capsys.readouterr().out assert "Claim 00000" in result and "source.py::Container" in result - shadow_viewer.view_get(shadow, ids(result)[0]) + shadow_knowledge.view_get(shadow, ids(result)[0]) assert "source.py::Container" in capsys.readouterr().out @pytest.mark.parametrize("query", ["src/auth.py", unicodedata.normalize("NFD", "Src/caf\u00e9.py")]) -def test_filesystem_alias_ids_expand_and_include_cross_refs(shadow_viewer, tmp_path, capsys, query): +def test_filesystem_alias_ids_expand_and_include_cross_refs(shadow_knowledge, tmp_path, capsys, query): actual = "Src/Auth.py" if query == "src/auth.py" else "Src/caf\u00e9.py" shadow = tmp_path / ".shadow" path = shadow / f"{actual}.md" @@ -257,17 +257,17 @@ def test_filesystem_alias_ids_expand_and_include_cross_refs(shadow_viewer, tmp_p "**Discovery**: Cross-file audit contract.\n\n_(verified, source: exploration)_\n", encoding="utf-8", ) - global_entries = shadow_viewer._knowledge_entries(shadow) - shadow_viewer.view_symbol(shadow, f"{query}::run") + global_entries = shadow_knowledge._knowledge_entries(shadow) + shadow_knowledge.view_symbol(shadow, f"{query}::run") result = capsys.readouterr().out assert "Cross-file audit contract." in result and "Keep the audit trail." in result assert set(ids(result)) == {entry["id"] for entry in global_entries} for identity in ids(result): - shadow_viewer.view_get(shadow, identity, options=shadow_viewer.RetrievalOptions(record=False)) + shadow_knowledge.view_get(shadow, identity, options=shadow_knowledge.RetrievalOptions(record=False)) assert identity in capsys.readouterr().out -def test_distinct_case_sensitive_files_are_not_folded(shadow_viewer, tmp_path): +def test_distinct_case_sensitive_files_are_not_folded(shadow_knowledge, tmp_path): shadow = tmp_path / ".shadow" shadow.mkdir() upper, lower = shadow / "A.py.md", shadow / "a.py.md" @@ -275,10 +275,10 @@ def test_distinct_case_sensitive_files_are_not_folded(shadow_viewer, tmp_path): if lower.exists(): pytest.skip("Filesystem does not support distinct case-only names") lower.write_text("# Shadow: a.py\n\n## `run`\n\n- Claim.\n", encoding="utf-8") - assert len({entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)}) == 2 + assert len({entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)}) == 2 -def test_literal_whitespace_remains_separately_searchable(shadow_viewer, tmp_path, capsys): +def test_literal_whitespace_remains_separately_searchable(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 0) path = shadow / "source.py.md" path.write_text( @@ -287,16 +287,16 @@ def test_literal_whitespace_remains_separately_searchable(shadow_viewer, tmp_pat "- Key `a b` is accepted.\n _(verified, source: exploration)_\n", encoding="utf-8", ) - entries = shadow_viewer._knowledge_entries(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) assert len(entries) == 2 and entries[0]["id"] != entries[1]["id"] - shadow_viewer.view_search(shadow, "a b") + shadow_knowledge.view_search(shadow, "a b") result = capsys.readouterr().out assert ids(result) == [entries[1]["id"]] - shadow_viewer.view_get(shadow, entries[1]["id"]) + shadow_knowledge.view_get(shadow, entries[1]["id"]) assert "`a b`" in capsys.readouterr().out -def test_duplicate_claims_union_labels_and_preserve_stronger_source(shadow_viewer, tmp_path, capsys): +def test_duplicate_claims_union_labels_and_preserve_stronger_source(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 0) (shadow / "source.py.md").write_text( "# Shadow: source.py\n\n## `run`\n\n" @@ -304,31 +304,31 @@ def test_duplicate_claims_union_labels_and_preserve_stronger_source(shadow_viewe "- Shared claim.\n _(verified, source: exploration, labels: [security])_\n", encoding="utf-8", ) - entries = shadow_viewer._knowledge_entries(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) assert len(entries) == 1 assert entries[0]["labels"] == ["bug", "security"] and entries[0]["source"] == "user" - shadow_viewer.view_labels(shadow, "security") + shadow_knowledge.view_labels(shadow, "security") assert ids(capsys.readouterr().out) == [entries[0]["id"]] def test_installed_import_does_not_create_bytecode(repo_root, tmp_path): - installed = tmp_path / ".github/skills/shadow-frog-viewer" + installed = tmp_path / ".github/skills/shadow-frog" shutil.copytree( - repo_root / "skills/shadow-frog-viewer", installed, + repo_root / "skills/shadow-frog", installed, ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), ) env = os.environ.copy() env.pop("PYTHONDONTWRITEBYTECODE", None) env.pop("PYTHONPYCACHEPREFIX", None) result = subprocess.run( - [sys.executable, str(installed / "shadow-viewer.py"), "--help"], + [sys.executable, str(installed / "shadow-read.py"), "--help"], env=env, capture_output=True, text=True, encoding="utf-8", ) assert result.returncode == 0, result.stderr assert not list(installed.rglob("*.pyc")) -def test_missing_home_does_not_hide_standalone_knowledge(shadow_viewer, tmp_path, monkeypatch, capsys): +def test_missing_home_does_not_hide_standalone_knowledge(shadow_knowledge, tmp_path, monkeypatch, capsys): from pathlib import Path shadow = make_shadow(tmp_path, 1) @@ -339,22 +339,22 @@ def missing_home(): raise RuntimeError("Could not determine home directory") monkeypatch.setattr(Path, "home", missing_home) - shadow_viewer.view_search(shadow, "Claim") + shadow_knowledge.view_search(shadow, "Claim") result = capsys.readouterr() assert "Claim 00000" in result.out assert "will not be recorded" in result.err and "home" in result.err -def test_get_chunks_long_claim_without_exceeding_budget(shadow_viewer, tmp_path, capsys): +def test_get_chunks_long_claim_without_exceeding_budget(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1, text_size=500) - entry = shadow_viewer._knowledge_entries(shadow)[0] + entry = shadow_knowledge._knowledge_entries(shadow)[0] identity = entry["id"] continuation = None pieces = [] while True: - shadow_viewer.view_get( + shadow_knowledge.view_get( shadow, identity, - options=shadow_viewer.RetrievalOptions(max_chars=500, text_cursor=continuation), + options=shadow_knowledge.RetrievalOptions(max_chars=500, text_cursor=continuation), ) output = capsys.readouterr().out assert len(output) <= 500 @@ -363,14 +363,14 @@ def test_get_chunks_long_claim_without_exceeding_budget(shadow_viewer, tmp_path, if continuation is None: break assert "".join(pieces) == entry["anchor"] + "\n\n" + entry["text"] + "\nLabels: bug" - assert shadow_viewer.CitationStore.for_shadow(shadow).scores([identity]) == {identity: 1} + assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([identity]) == {identity: 1} @pytest.mark.parametrize("change", ["labels", "source", "status", "text"]) -def test_expansion_rejects_changed_body_or_metadata(shadow_viewer, tmp_path, capsys, change): +def test_expansion_rejects_changed_body_or_metadata(shadow_knowledge, tmp_path, capsys, change): shadow = make_shadow(tmp_path, 1, text_size=200) - entry = shadow_viewer._knowledge_entries(shadow)[0] - shadow_viewer.view_get(shadow, entry["id"], options=shadow_viewer.RetrievalOptions(max_chars=500)) + entry = shadow_knowledge._knowledge_entries(shadow)[0] + shadow_knowledge.view_get(shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(max_chars=500)) continuation = text_cursor(capsys.readouterr().out) path = shadow / "source.py.md" replacements = { @@ -381,30 +381,30 @@ def test_expansion_rejects_changed_body_or_metadata(shadow_viewer, tmp_path, cap } path.write_text(path.read_text(encoding="utf-8").replace(*replacements[change]), encoding="utf-8") with pytest.raises(ValueError, match="changed"): - shadow_viewer.view_get( - shadow, entry["id"], options=shadow_viewer.RetrievalOptions(text_cursor=continuation), + shadow_knowledge.view_get( + shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(text_cursor=continuation), ) -def test_no_record_continuation_stays_unrecorded(shadow_viewer, tmp_path, capsys): +def test_no_record_continuation_stays_unrecorded(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1, text_size=200) - entry = shadow_viewer._knowledge_entries(shadow)[0] - shadow_viewer.view_get( - shadow, entry["id"], options=shadow_viewer.RetrievalOptions(max_chars=500, record=False), + entry = shadow_knowledge._knowledge_entries(shadow)[0] + shadow_knowledge.view_get( + shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(max_chars=500, record=False), ) continuation = text_cursor(capsys.readouterr().out) - shadow_viewer.view_get( - shadow, entry["id"], options=shadow_viewer.RetrievalOptions(text_cursor=continuation), + shadow_knowledge.view_get( + shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(text_cursor=continuation), ) capsys.readouterr() - assert shadow_viewer.CitationStore.for_shadow(shadow).scores([entry["id"]]) == {} + assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([entry["id"]]) == {} @pytest.mark.parametrize("change", ["mtime", "other-symbol"]) -def test_nonrecent_pages_survive_irrelevant_file_changes(shadow_viewer, tmp_path, capsys, change): +def test_nonrecent_pages_survive_irrelevant_file_changes(shadow_knowledge, tmp_path, capsys, change): shadow = make_shadow(tmp_path) - options = shadow_viewer.RetrievalOptions(limit=1) - shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + options = shadow_knowledge.RetrievalOptions(limit=1) + shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) first = capsys.readouterr().out path = shadow / "source.py.md" if change == "mtime": @@ -413,15 +413,15 @@ def test_nonrecent_pages_survive_irrelevant_file_changes(shadow_viewer, tmp_path else: with path.open("a", encoding="utf-8") as stream: stream.write("\n## `other`\n\n- Other knowledge.\n _(verified, source: exploration)_\n") - shadow_viewer.view_symbol(shadow, "source.py::run", options=replace(options, cursor=cursor(first))) + shadow_knowledge.view_symbol(shadow, "source.py::run", options=replace(options, cursor=cursor(first))) assert not set(ids(first)) & set(ids(capsys.readouterr().out)) @pytest.mark.parametrize("change", ["source", "labels", "status", "recent-mtime"]) -def test_pages_invalidate_on_relevant_metadata_changes(shadow_viewer, tmp_path, capsys, change): +def test_pages_invalidate_on_relevant_metadata_changes(shadow_knowledge, tmp_path, capsys, change): shadow = make_shadow(tmp_path) - options = shadow_viewer.RetrievalOptions(limit=1) - view = shadow_viewer.view_recent if change == "recent-mtime" else shadow_viewer.view_symbol + options = shadow_knowledge.RetrievalOptions(limit=1) + view = shadow_knowledge.view_recent if change == "recent-mtime" else shadow_knowledge.view_symbol args = [shadow] if change == "recent-mtime" else [shadow, "source.py::run"] view(*args, options=options) continuation = cursor(capsys.readouterr().out) @@ -440,17 +440,17 @@ def test_pages_invalidate_on_relevant_metadata_changes(shadow_viewer, tmp_path, view(*args, options=replace(options, cursor=continuation)) -def test_summary_and_invariant_checks_do_not_count(shadow_viewer, tmp_path, capsys): +def test_summary_and_invariant_checks_do_not_count(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path) - store = shadow_viewer.CitationStore.for_shadow(shadow) - shadow_viewer.view_summary(shadow) - shadow_viewer.view_check_invariants(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) + shadow_knowledge.view_summary(shadow) + shadow_knowledge.view_check_invariants(shadow) capsys.readouterr() assert not store.path.exists() def test_all_creation_sources_start_at_zero_without_schema_changes( - shadow_viewer, dream_reconcile, tmp_path, capsys, + shadow_knowledge, dream_reconcile, tmp_path, capsys, ): shadow = tmp_path / ".shadow" discoveries = [ @@ -463,41 +463,41 @@ def test_all_creation_sources_start_at_zero_without_schema_changes( [("dream/test/20260923-000000Z-test", "20260923-000000Z-test", {"discoveries": discoveries})], ) - entries = shadow_viewer._knowledge_entries(shadow) + entries = shadow_knowledge._knowledge_entries(shadow) assert len(entries) == 3 - assert shadow_viewer.CitationStore.for_shadow(shadow).scores([entry["id"] for entry in entries]) == {} - shadow_viewer.view_search(shadow, "Behavior", options=shadow_viewer.RetrievalOptions(record=False)) + assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([entry["id"] for entry in entries]) == {} + shadow_knowledge.view_search(shadow, "Behavior", options=shadow_knowledge.RetrievalOptions(record=False)) output = capsys.readouterr().out assert output.count("citation_score=0") == 3 assert "citation_score" not in (shadow / "source.py.md").read_text(encoding="utf-8") -def test_top_hard_budget_counts_only_visible_claims(shadow_viewer, tmp_path, capsys): +def test_top_hard_budget_counts_only_visible_claims(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 10, text_size=150) - shadow_viewer.view_top(shadow, "source.py", "bug", 3, 600) + shadow_knowledge.view_top(shadow, "source.py", "bug", 3, 600) output = capsys.readouterr().out assert len(output) <= 600 emitted = ids(output) assert emitted and len(emitted) <= 3 assert f"Top {len(emitted)} of 10" in output - store = shadow_viewer.CitationStore.for_shadow(shadow) - assert store.scores([entry["id"] for entry in shadow_viewer._knowledge_entries(shadow)]) == dict.fromkeys(emitted, 1) + store = shadow_knowledge.CitationStore.for_shadow(shadow) + assert store.scores([entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)]) == dict.fromkeys(emitted, 1) -def test_smallest_budget_still_emits_identifiable_content(shadow_viewer, tmp_path, capsys): +def test_smallest_budget_still_emits_identifiable_content(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 10) - shadow_viewer.view_search(shadow, "Claim", options=shadow_viewer.RetrievalOptions(max_chars=256)) + shadow_knowledge.view_search(shadow, "Claim", options=shadow_knowledge.RetrievalOptions(max_chars=256)) output = capsys.readouterr().out assert len(output) <= 256 and len(ids(output)) == 1 assert "verified" in output and cursor(output) -def test_broken_ledger_returns_knowledge_with_warning(shadow_viewer, tmp_path, capsys): +def test_broken_ledger_returns_knowledge_with_warning(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) - store = shadow_viewer.CitationStore.for_shadow(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) store.path.parent.mkdir(parents=True) store.path.write_bytes(b"not sqlite") - shadow_viewer.view_search(shadow, "Claim") + shadow_knowledge.view_search(shadow, "Claim") captured = capsys.readouterr() assert "Claim 00000" in captured.out and "citation_score=?" in captured.out assert "warning" in captured.err and "citation" in captured.err @@ -505,42 +505,42 @@ def test_broken_ledger_returns_knowledge_with_warning(shadow_viewer, tmp_path, c @pytest.mark.parametrize("view", ["search", "symbol", "get", "prefs", "labels", "recent", "top"]) -def test_all_content_views_share_identity_and_score(shadow_viewer, tmp_path, capsys, view): +def test_all_content_views_share_identity_and_score(shadow_knowledge, tmp_path, capsys, view): shadow = make_shadow(tmp_path, 1, text_size=2) (shadow / "_prefs.md").write_text("# Preferences\n\n- Keep the contract.\n _(source: user)_\n", encoding="utf-8") - options = shadow_viewer.RetrievalOptions(event_id="same-visit") - entry = next(item for item in shadow_viewer._knowledge_entries(shadow) if item["kind"] != "preference") + options = shadow_knowledge.RetrievalOptions(event_id="same-visit") + entry = next(item for item in shadow_knowledge._knowledge_entries(shadow) if item["kind"] != "preference") if view == "search": - shadow_viewer.view_search(shadow, "Claim", options=options) + shadow_knowledge.view_search(shadow, "Claim", options=options) elif view == "symbol": - shadow_viewer.view_symbol(shadow, "source.py::run", options=options) + shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) elif view == "get": - shadow_viewer.view_get(shadow, entry["id"], options=options) + shadow_knowledge.view_get(shadow, entry["id"], options=options) elif view == "prefs": - shadow_viewer.view_prefs(shadow, options=options) + shadow_knowledge.view_prefs(shadow, options=options) elif view == "labels": - shadow_viewer.view_labels(shadow, "bug", options=options) + shadow_knowledge.view_labels(shadow, "bug", options=options) elif view == "recent": - shadow_viewer.view_recent(shadow, options=options) + shadow_knowledge.view_recent(shadow, options=options) else: - shadow_viewer.view_top(shadow, "source.py", "", 3, 600, options=options) + shadow_knowledge.view_top(shadow, "source.py", "", 3, 600, options=options) output = capsys.readouterr().out emitted = ids(output) - store = shadow_viewer.CitationStore.for_shadow(shadow) + store = shadow_knowledge.CitationStore.for_shadow(shadow) assert emitted and store.scores(emitted) == dict.fromkeys(emitted, 1) if view != "prefs": assert entry["id"] in emitted -def test_write_failure_warns_without_hiding_knowledge(shadow_viewer, tmp_path, capsys): +def test_write_failure_warns_without_hiding_knowledge(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) - store = shadow_viewer.CitationStore.for_shadow(shadow) - identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] + store = shadow_knowledge.CitationStore.for_shadow(shadow) + identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] store.record([identity], "initial") connection = sqlite3.connect(store.path) try: connection.execute("BEGIN IMMEDIATE") - shadow_viewer.view_search(shadow, "Claim") + shadow_knowledge.view_search(shadow, "Claim") captured = capsys.readouterr() assert identity in captured.out assert "this visit was not recorded" in captured.err @@ -550,10 +550,10 @@ def test_write_failure_warns_without_hiding_knowledge(shadow_viewer, tmp_path, c assert store.scores([identity])[identity] == 1 -def test_counting_can_recover_after_a_failed_score_read(shadow_viewer, tmp_path, monkeypatch, capsys): +def test_counting_can_recover_after_a_failed_score_read(shadow_knowledge, tmp_path, monkeypatch, capsys): shadow = make_shadow(tmp_path, 1) - identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] - store = shadow_viewer.CitationStore.for_shadow(shadow) + identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] + store = shadow_knowledge.CitationStore.for_shadow(shadow) store.record([identity]) lock = sqlite3.connect(store.path) # A pre-existing rollback cache can still be locked during WAL activation. @@ -569,7 +569,7 @@ def flush(self): try: with monkeypatch.context() as context: context.setattr(sys, "stdout", output) - shadow_viewer.view_search(shadow, "Claim") + shadow_knowledge.view_search(shadow, "Claim") finally: lock.close() warnings = capsys.readouterr().err @@ -578,10 +578,10 @@ def flush(self): assert store.scores([identity])[identity] == 2 -def test_failed_stdout_does_not_increment_score(shadow_viewer, tmp_path, monkeypatch): +def test_failed_stdout_does_not_increment_score(shadow_knowledge, tmp_path, monkeypatch): shadow = make_shadow(tmp_path, 1) - store = shadow_viewer.CitationStore.for_shadow(shadow) - identity = shadow_viewer._knowledge_entries(shadow)[0]["id"] + store = shadow_knowledge.CitationStore.for_shadow(shadow) + identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] class ClosedOutput: def write(self, text): @@ -592,11 +592,11 @@ def flush(self): monkeypatch.setattr(sys, "stdout", ClosedOutput()) with pytest.raises(BrokenPipeError): - shadow_viewer.view_search(shadow, "Claim") + shadow_knowledge.view_search(shadow, "Claim") assert store.scores([identity]) == {} -def test_symbol_context_includes_related_cross_but_not_other_symbols(shadow_viewer, tmp_path, capsys): +def test_symbol_context_includes_related_cross_but_not_other_symbols(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) with (shadow / "source.py.md").open("a", encoding="utf-8") as stream: stream.write("\n## `unrelated`\n\n- Unrelated claim.\n _(verified, source: user)_\n") @@ -608,13 +608,13 @@ def test_symbol_context_includes_related_cross_but_not_other_symbols(shadow_view "**Discovery**: Cross-file constraint.\n\n_(verified, source: user)_\n", encoding="utf-8", ) - shadow_viewer.view_symbol(shadow, "source.py::run") + shadow_knowledge.view_symbol(shadow, "source.py::run") output = capsys.readouterr().out assert "Cross-file constraint." in output assert "Claim 00000" in output and "Unrelated claim." not in output -def test_nested_paths_have_one_identity_across_scoped_and_global_views(shadow_viewer, tmp_path): +def test_nested_paths_have_one_identity_across_scoped_and_global_views(shadow_knowledge, tmp_path): shadow = tmp_path / ".shadow" source = shadow / "src/caf\u00e9 tools.py.md" source.parent.mkdir(parents=True) @@ -623,26 +623,26 @@ def test_nested_paths_have_one_identity_across_scoped_and_global_views(shadow_vi "- Nested claim.\n _(verified, source: exploration)_\n", encoding="utf-8", ) - global_entry = shadow_viewer._knowledge_entries(shadow)[0] - scoped_entry = shadow_viewer._knowledge_entries(shadow, "src/caf\u00e9 tools.py")[0] + global_entry = shadow_knowledge._knowledge_entries(shadow)[0] + scoped_entry = shadow_knowledge._knowledge_entries(shadow, "src/caf\u00e9 tools.py")[0] assert global_entry["anchor"] == scoped_entry["anchor"] == "src/caf\u00e9 tools.py::run" assert global_entry["id"] == scoped_entry["id"] -def test_expansion_preserves_long_anchors(shadow_viewer, tmp_path, capsys): +def test_expansion_preserves_long_anchors(shadow_knowledge, tmp_path, capsys): shadow = make_shadow(tmp_path, 1) path = shadow / "source.py.md" symbol = "method_" + "name" * 50 path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{symbol}`"), encoding="utf-8") - entry = shadow_viewer._knowledge_entries(shadow)[0] - shadow_viewer.view_get(shadow, entry["id"]) + entry = shadow_knowledge._knowledge_entries(shadow)[0] + shadow_knowledge.view_get(shadow, entry["id"]) assert "source.py::" + symbol in capsys.readouterr().out def test_cursor_cli_continuation_and_get_are_wired(repo_root, tmp_path): shadow = make_shadow(tmp_path) command = [ - sys.executable, str(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py"), + sys.executable, str(repo_root / "skills/shadow-frog/shadow-read.py"), "--shadow-dir", str(shadow), "--symbol", "source.py::run", "--limit", "1", "--max-chars", "600", ] @@ -662,7 +662,7 @@ def test_cursor_cli_continuation_and_get_are_wired(repo_root, tmp_path): def test_cli_text_continuation_is_revision_bound_and_counts_one_read(repo_root, tmp_path): shadow = make_shadow(tmp_path, 1, text_size=300) - script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" + script = repo_root / "skills/shadow-frog/shadow-read.py" prefix = [sys.executable, str(script), "--shadow-dir", str(shadow)] found = subprocess.run( [*prefix, "--symbol", "source.py::run", "--no-record"], @@ -690,10 +690,10 @@ def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_ import shutil for agent in (".github", ".claude"): - installed = tmp_path / agent / "skills/shadow-frog-viewer" - shutil.copytree(repo_root / "skills/shadow-frog-viewer", installed) + installed = tmp_path / agent / "skills/shadow-frog" + shutil.copytree(repo_root / "skills/shadow-frog", installed) command = [ - sys.executable, str(installed / "shadow-viewer.py"), + sys.executable, str(installed / "shadow-read.py"), "--shadow-dir", str(coupon_demo / ".shadow"), "--search", "coupon", "--limit", "2", "--max-chars", "1000", "--event-id", agent, ] @@ -712,7 +712,7 @@ def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_ ]) def test_invalid_cli_combinations_are_explicit(repo_root, tmp_path, args): result = subprocess.run( - [sys.executable, str(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py"), *args], + [sys.executable, str(repo_root / "skills/shadow-frog/shadow-read.py"), *args], cwd=tmp_path, capture_output=True, text=True, encoding="utf-8", ) assert result.returncode == 2 and "error:" in result.stderr diff --git a/tests/skills/shadow_frog/test_shadow_read.py b/tests/skills/shadow_frog/test_shadow_read.py new file mode 100644 index 0000000..f5caef4 --- /dev/null +++ b/tests/skills/shadow_frog/test_shadow_read.py @@ -0,0 +1,171 @@ +"""Optional agent retrieval targets known files without depending on the Viewer.""" + +import json +import os +from pathlib import Path +import re +import shutil +import subprocess +import sys + +import pytest + + +def sample_shadow(tmp_path): + shadow = tmp_path / ".shadow" + shadow.mkdir() + (shadow / "source.py.md").write_text( + "# Shadow: source.py\n\n## File-Level\n\n" + "- Importing starts no workers.\n _(verified, source: user)_\n\n" + "## `class Worker`\n\n" + "- Construction opens no files.\n _(verified, source: exploration)_\n\n" + "### `Worker.run`\n\n" + "- Empty input returns immediately.\n _(verified, source: exploration)_\n\n" + "## Cross-References\n\n- [Worker contract](_cross/worker-contract.md)\n", + encoding="utf-8", + ) + cross = shadow / "_cross" + cross.mkdir() + (cross / "worker-contract.md").write_text( + "# Worker contract\n\n**Category**: contract\n**Refs**:\n" + "- `source.py::Worker.run`\n- `other.py::dispatch`\n\n" + "**Discovery**: Dispatch must retain worker ownership.\n\n" + "_(verified, source: exploration)_\n", + encoding="utf-8", + ) + return shadow + + +def invoke(script, shadow, *args): + return subprocess.run( + [sys.executable, str(script), "--shadow-dir", str(shadow), *args], + cwd=shadow.parent, capture_output=True, text=True, encoding="utf-8", timeout=15, + ) + + +def identities(text): + return re.findall(r"\bid=(d_[0-9a-f]{32})", text) + + +@pytest.mark.parametrize("layout", [".github", ".claude"]) +def test_core_only_install_supports_file_and_symbol_reads(repo_root, tmp_path, layout): + shadow = sample_shadow(tmp_path) + installed = tmp_path / layout / "skills/shadow-frog" + shutil.copytree( + repo_root / "skills/shadow-frog", installed, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + script = installed / "shadow-read.py" + assert not (installed.parent / "shadow-frog-viewer").exists() + file_view = invoke(script, shadow, "source.py") + assert file_view.returncode == 0, file_view.stderr + assert "Importing starts no workers." in file_view.stdout + assert "Construction opens no files." in file_view.stdout + assert "Empty input returns immediately." in file_view.stdout + assert "Dispatch must retain worker ownership." in file_view.stdout + symbol_view = invoke(script, shadow, "source.py::Worker.run") + assert symbol_view.returncode == 0, symbol_view.stderr + assert "Empty input returns immediately." in symbol_view.stdout + assert "Dispatch must retain worker ownership." in symbol_view.stdout + assert "Construction opens no files." not in symbol_view.stdout + assert "Importing starts no workers." not in symbol_view.stdout + assert set(identities(symbol_view.stdout)) <= set(identities(file_view.stdout)) + expanded = invoke(script, shadow, "--get", identities(symbol_view.stdout)[0]) + assert expanded.returncode == 0, expanded.stderr + + +def test_file_read_does_not_parse_unrelated_per_file_shadows(repo_root, tmp_path): + shadow = sample_shadow(tmp_path) + (shadow / "unrelated.py.md").write_bytes(b"\xff") + result = invoke(repo_root / "skills/shadow-frog/shadow-read.py", shadow, "--file", "source.py") + assert result.returncode == 0 + assert "Empty input" in result.stdout + assert result.stderr == "" + assert "unrelated" not in result.stdout + + +def test_file_and_symbol_positional_targets_conflict_with_other_views(repo_root, tmp_path): + shadow = sample_shadow(tmp_path) + result = invoke( + repo_root / "skills/shadow-frog/shadow-read.py", shadow, + "source.py", "--search", "workers", + ) + assert result.returncode == 2 + assert "positional target or an explicit view" in result.stderr + + +def test_reader_requires_an_explicit_target_or_operation(repo_root, tmp_path): + shadow = sample_shadow(tmp_path) + result = invoke(repo_root / "skills/shadow-frog/shadow-read.py", shadow) + assert result.returncode == 2 + assert "known FILE[::SYMBOL]" in result.stderr + + +def test_file_first_role_instructions_and_bundled_reference(repo_root): + core = repo_root / "skills/shadow-frog" + instructions = (core / "SKILL.md").read_text(encoding="utf-8") + context = (repo_root / "agent-context.md").read_text(encoding="utf-8") + user_skill = (repo_root / "skills/shadow-frog-viewer/SKILL.md").read_text(encoding="utf-8") + assert "**File/symbol navigation is primary.**" in instructions + assert "shadow-read.py" in instructions and "(retrieval.md)" in instructions + assert "Native file reads remain normal and uncounted." in instructions + assert "**Navigate directly**" in context + assert "user-facing" in context and "mandatory" in context + assert "**User-facing inspection and visualization**" in user_skill + assert (core / "retrieval.md").is_file() + + +def test_reader_without_modules_reports_error_but_raw_knowledge_stays_readable(repo_root, tmp_path): + shadow = sample_shadow(tmp_path) + script = tmp_path / "incomplete/shadow-read.py" + script.parent.mkdir() + shutil.copyfile(repo_root / "skills/shadow-frog/shadow-read.py", script) + before = (shadow / "source.py.md").read_bytes() + result = invoke(script, shadow, "source.py") + assert result.returncode != 0 and "reinstall" in result.stderr + assert (shadow / "source.py.md").read_bytes() == before + + +def test_known_file_read_is_bounded_and_pageable(repo_root, tmp_path): + shadow = sample_shadow(tmp_path) + script = repo_root / "skills/shadow-frog/shadow-read.py" + first = invoke(script, shadow, "source.py", "--limit", "1", "--max-chars", "600") + assert first.returncode == 0 and len(first.stdout) <= 600 + continuation = re.search(r"--cursor ([0-9a-f]{32}:\d+)", first.stdout).group(1) + second = invoke( + script, shadow, "source.py", "--limit", "1", "--max-chars", "600", + "--cursor", continuation, + ) + assert second.returncode == 0 and len(second.stdout) <= 600 + assert not set(identities(first.stdout)) & set(identities(second.stdout)) + + +@pytest.mark.parametrize("layout", [".github", ".claude"]) +def test_agent_hook_uses_core_reader_without_user_viewer(repo_root, coupon_demo, tmp_path, layout): + from tests._shell import BASH, HAVE_BASH, shell_path + + if not HAVE_BASH: + pytest.skip("No POSIX shell available for the existing hook") + installed = coupon_demo / layout / "skills/shadow-frog" + shutil.copytree( + repo_root / "skills/shadow-frog", installed, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + hook = coupon_demo / layout / "hooks/scripts/shadow-frog-pre-tool.sh" + hook.parent.mkdir(parents=True) + shutil.copyfile(repo_root / "hook-templates/scripts/shadow-frog-pre-tool.sh", hook) + assert not (installed.parent / "shadow-frog-viewer").exists() + env = os.environ.copy() + env.update( + PATH=shell_path(), HOME=str(coupon_demo), + GIT_CONFIG_GLOBAL=os.devnull, GIT_CONFIG_SYSTEM=os.devnull, + SHADOWFROG_TMP_DIR=str(tmp_path / "hook-state"), + ) + result = subprocess.run( + [BASH, str(hook)], + input=json.dumps({"tool_name": "edit", "tool_input": {"file_path": "cart.py"}}), + cwd=coupon_demo, env=env, capture_output=True, text=True, encoding="utf-8", timeout=10, + ) + assert result.returncode == 0, result.stderr + context = json.loads(result.stdout)["additionalContext"] + assert "Actionable discoveries" in context and "id=d_" in context diff --git a/tests/skills/shadow_frog_dream/test_dream_tools.py b/tests/skills/shadow_frog_dream/test_dream_tools.py index d071567..cd07d2c 100644 --- a/tests/skills/shadow_frog_dream/test_dream_tools.py +++ b/tests/skills/shadow_frog_dream/test_dream_tools.py @@ -49,6 +49,8 @@ def test_pin_creates_complete_external_snapshot(tmp_git_repo, tmp_path): assert Path(packet["manifest"]).is_absolute() assert Path(packet["skill_dir"]) == output / "shadow-frog-dream" assert (output / "shadow-frog/_coherence.py").is_file() + assert (output / "shadow-frog/retrieval.md").is_file() + assert (output / "shadow-frog/shadow-read.py").is_file() assert all(Path(path).is_file() for path in packet["instructions"]) metadata = json.loads(Path(packet["manifest"]).read_text(encoding="utf-8")) assert metadata["mode"] == "coherent" diff --git a/tests/skills/shadow_frog_nap/test_nap.py b/tests/skills/shadow_frog_nap/test_nap.py index 7e4d766..e92a9cb 100644 --- a/tests/skills/shadow_frog_nap/test_nap.py +++ b/tests/skills/shadow_frog_nap/test_nap.py @@ -416,12 +416,12 @@ def test_cli_export_is_utf8_and_does_not_overwrite_record(nap, record, tmp_path, assert json.loads((tmp_path / "nap run.json").read_text(encoding="utf-8")) == record -def test_exports_do_not_pollute_discovery_views(record, tmp_path, coupon_demo, shadow_viewer): - before = shadow_viewer.get_all_shadow_files(coupon_demo / ".shadow") +def test_exports_do_not_pollute_discovery_views(record, tmp_path, coupon_demo, shadow_knowledge): + before = shadow_knowledge.get_all_shadow_files(coupon_demo / ".shadow") output = coupon_demo / ".shadow" / "_meta" / "naps" / "tasks.md" result = run_cli(record, tmp_path, coupon_demo, "--export", str(output)) assert result.returncode == 0, result.stderr - assert shadow_viewer.get_all_shadow_files(coupon_demo / ".shadow") == before + assert shadow_knowledge.get_all_shadow_files(coupon_demo / ".shadow") == before result = run_cli( record, tmp_path, coupon_demo, "--export", str(coupon_demo / ".shadow" / "nap-proposals.md"), diff --git a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py index 2f13fc4..116469a 100644 --- a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py +++ b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py @@ -1,2186 +1,47 @@ -r"""Tests for `skills/shadow-frog-viewer/shadow-viewer.py`. +"""User-facing Viewer CLI shares knowledge identities with the optional agent helper.""" -Philosophy: USE REAL FILES (per `minimal-mocking-tests`). Retrieval does not -edit shadow Markdown; local citation bookkeeping is isolated by the fixtures. -Tests construct shadow trees or exercise the CLI against `coupon_demo`. - -Test categories: - * In-process function tests (no `@pytest.mark.slow`): exercise - parsing helpers directly via the `shadow_viewer` fixture. - * CLI integration tests (`@pytest.mark.slow @pytest.mark.integration`): - invoke shadow-viewer.py as a subprocess against `coupon_demo`. - -B3 regression: a discovery whose continuation lines include -``Dream report: `_dreams//` `` must extract the slug path into -`meta["dream_report"]` and must NOT include "Dream report" or the slug -in the discovery body text. -""" -import json import os import re +import shutil import subprocess import sys -import textwrap -from datetime import datetime import pytest -# --- Helpers --------------------------------------------------------------- - - -def _write_shadow(shadow_dir, rel_path, content): - """Write `content` to /, creating parents.""" - p = shadow_dir / rel_path - p.parent.mkdir(parents=True, exist_ok=True) - p.write_text(textwrap.dedent(content), encoding="utf-8") - return p - - -def _make_shadow_root(tmp_path): - """Create an empty .shadow/ dir under tmp_path and return it.""" - sd = tmp_path / ".shadow" - sd.mkdir() - return sd - - -def _run_viewer(repo_root, cwd, *args): - """Run shadow-viewer.py as a subprocess from `cwd`.""" - script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" - return subprocess.run( - [sys.executable, str(script), *args], - cwd=str(cwd), - capture_output=True, - text=True, - encoding="utf-8", - check=False, - ) - - -# --- parse_discovery ------------------------------------------------------- - - -def test_parse_discovery_standard(shadow_viewer): - """Basic verified/exploration discovery, no labels.""" - line = "- Caches None for invalid codes" - cont = [" _(verified, source: exploration)_"] - d = shadow_viewer.parse_discovery(line, cont) - assert d["text"] == "Caches None for invalid codes" - assert d["status"] == "verified" - assert d["source"] == "exploration" - assert "labels" not in d - assert "dream_report" not in d - - -def test_parse_discovery_with_labels(shadow_viewer): - """Labels are parsed into a list, trimmed, lowercase comma-split.""" - line = "- Foo" - cont = [" _(verified, source: user, labels: [bug, security])_"] - d = shadow_viewer.parse_discovery(line, cont) - assert d["text"] == "Foo" - assert d["status"] == "verified" - assert d["source"] == "user" - assert d["labels"] == ["bug", "security"] - - -@pytest.mark.parametrize("status", ["verified", "uncertain", "refuted"]) -def test_parse_discovery_status_variants(shadow_viewer, status): - line = "- Some discovery" - cont = [f" _({status}, source: exploration)_"] - d = shadow_viewer.parse_discovery(line, cont) - assert d["status"] == status - assert d["source"] == "exploration" - - -@pytest.mark.parametrize("source", ["exploration", "user", "interaction"]) -def test_parse_discovery_source_variants(shadow_viewer, source): - line = "- Some discovery" - cont = [f" _(verified, source: {source})_"] - d = shadow_viewer.parse_discovery(line, cont) - assert d["source"] == source - - -def test_parse_discovery_also_involves(shadow_viewer): - """`Also involves:` populates a list of file::symbol anchors.""" - line = "- A multi-symbol discovery" - cont = [ - " _(verified, source: exploration)_", - " Also involves: `inventory.py::validate_coupon`, `cart.py::COUPON_CACHE`", - ] - d = shadow_viewer.parse_discovery(line, cont) - assert d["text"] == "A multi-symbol discovery" - assert d["also_involves"] == [ - "inventory.py::validate_coupon", - "cart.py::COUPON_CACHE", - ] - # also_involves line must not leak into the body text - assert "Also involves" not in d["text"] - - -def test_parse_discovery_b3_dream_report_regression(shadow_viewer): - """B3 regression: Dream report goes into meta['dream_report'] and is - excluded from the body text.""" - line = "- Case-variant lookups create duplicate cache entries" - cont = [ - " _(verified, source: exploration, labels: [bug, performance])_", - " Dream report: `_dreams/20260420-140000Z-cache-poison-sequence/`", - ] - d = shadow_viewer.parse_discovery(line, cont) - # Body text is preserved, with no Dream report leakage - assert d["text"] == "Case-variant lookups create duplicate cache entries" - assert "Dream report" not in d["text"] - assert "_dreams/" not in d["text"] - # meta["dream_report"] captures the backtick payload (slug folder path) - assert d["dream_report"] == ( - "_dreams/20260420-140000Z-cache-poison-sequence/" - ) - - -def test_parse_discovery_b3_dream_report_with_also_involves(shadow_viewer): - """Dream report + Also involves on the same discovery — both extracted, - neither leaks into body text.""" - line = "- Discovery with both extras" - cont = [ - " _(verified, source: exploration, labels: [security])_", - " Dream report: `_dreams/20260420-142000Z-adversarial-inputs/`", - " Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE`", - ] - d = shadow_viewer.parse_discovery(line, cont) - assert d["text"] == "Discovery with both extras" - assert "Dream report" not in d["text"] - assert "Also involves" not in d["text"] - assert d["dream_report"] == ( - "_dreams/20260420-142000Z-adversarial-inputs/" - ) - assert d["also_involves"] == [ - "cart.py::get_coupon", - "cart.py::COUPON_CACHE", - ] - - -def test_parse_discovery_multiline_body(shadow_viewer): - """Lines that are neither metadata nor structured extras are appended to - the body text.""" - line = "- Lead sentence." - cont = [ - " continuation prose", - " _(verified, source: exploration)_", - ] - d = shadow_viewer.parse_discovery(line, cont) - assert "Lead sentence." in d["text"] - assert "continuation prose" in d["text"] - assert d["status"] == "verified" - - -def test_parse_discovery_no_metadata(shadow_viewer): - """Bullet with no metadata blob still returns a dict with text but no - status/source keys.""" - d = shadow_viewer.parse_discovery("- bare bullet", []) - assert d["text"] == "bare bullet" - assert "status" not in d - assert "source" not in d - - -def test_parse_discovery_preferences_source_only(shadow_viewer): - """Preferences use `_(source: user)_` (no status). Extracts source.""" - d = shadow_viewer.parse_discovery( - "- Prefer X over Y", [" _(source: user)_"] - ) - assert d["text"] == "Prefer X over Y" - assert d["source"] == "user" - assert "status" not in d - - -def test_parse_discovery_none_input_does_not_crash(shadow_viewer): - """Passing a non-string line shouldn't raise — should return a dict.""" - d = shadow_viewer.parse_discovery(None, None) - assert isinstance(d, dict) - assert "text" in d - - -# --- parse_shadow_file ----------------------------------------------------- - - -def test_parse_shadow_file_placeholder(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "foo.py.md", """\ - # Shadow: foo.py - - **Language**: Python | **Lines**: 10 - - _No discoveries yet._ - """) - res = shadow_viewer.parse_shadow_file(f) - assert res["source_file"] == "foo.py" - assert res["language"] == "Python" - assert res["lines"] == 10 - assert res["symbols"] == [] - assert res["discoveries"] == [] - assert res["cross_references"] == [] - assert res["parse_errors"] == [] - - -def test_parse_shadow_file_one_symbol_one_discovery(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "auth.py.md", """\ - # Shadow: auth.py - - **Language**: Python | **Lines**: 42 - - ## `authenticate` - - - Returns None on expired tokens, silently. - _(verified, source: exploration, labels: [security])_ - """) - res = shadow_viewer.parse_shadow_file(f) - assert res["source_file"] == "auth.py" - assert res["symbols"] == ["authenticate"] - assert len(res["discoveries"]) == 1 - d = res["discoveries"][0] - assert d["symbol"] == "authenticate" - assert d["file"] == "auth.py" - assert d["status"] == "verified" - assert d["source"] == "exploration" - assert d["labels"] == ["security"] - assert "Returns None" in d["text"] - - -def test_parse_shadow_file_cross_references_backpointers( - shadow_viewer, tmp_path -): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "bar.py.md", """\ - # Shadow: bar.py - - ## `func` - - - A discovery. - _(verified, source: exploration)_ - - ## Cross-References - - - [my-cross-cutting](_cross/my-cross-cutting.md) - (involves `bar.py::func`) - - [another-one](_cross/another-one.md) - """) - res = shadow_viewer.parse_shadow_file(f) - assert res["symbols"] == ["func"] - # Cross-references back-pointer link labels are collected - assert "my-cross-cutting" in res["cross_references"] - assert "another-one" in res["cross_references"] - # Cross-ref bullets are NOT mistaken for discoveries - assert len(res["discoveries"]) == 1 - - -def test_parse_shadow_file_file_level_and_cross_refs(shadow_viewer, tmp_path): - """`## File-Level` discoveries are tagged with symbol='file-level' and - `## Cross-References` bullets are not treated as discoveries.""" - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "mix.py.md", """\ - # Shadow: mix.py - - ## File-Level - - - A module-wide observation. - _(verified, source: exploration)_ - - ## `helper` - - - A symbol discovery. - _(verified, source: user)_ - - ## Cross-References - - - [shared](_cross/shared.md) - """) - res = shadow_viewer.parse_shadow_file(f) - discs = res["discoveries"] - assert len(discs) == 2 - by_sym = {d["symbol"]: d for d in discs} - assert "file-level" in by_sym - assert "helper" in by_sym - assert by_sym["file-level"]["source"] == "exploration" - assert by_sym["helper"]["source"] == "user" - assert res["cross_references"] == ["shared"] - - -def test_parse_shadow_file_malformed_does_not_crash(shadow_viewer, tmp_path): - """Bizarre / structurally broken content shouldn't raise.""" - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "junk.py.md", """\ - # Shadow: junk.py - **Language**: notnumeric | **Lines**: notanumber - - ## not a backtick heading - - orphan bullet with no metadata - ## `realsym` - - real disc - _(verified, source: exploration)_ - """) - res = shadow_viewer.parse_shadow_file(f) - # Parse succeeds despite weirdness - assert res["source_file"] == "junk.py" - # The bad "Lines" cell stays None (int parse skipped) - assert res["lines"] is None - # `realsym` is captured; the non-backtick heading is not a symbol - assert "realsym" in res["symbols"] - assert "not a backtick heading" not in res["symbols"] - - -def test_parse_shadow_file_missing_file_returns_error( - shadow_viewer, tmp_path -): - """Reading a non-existent path records a parse_error, doesn't raise.""" - res = shadow_viewer.parse_shadow_file(tmp_path / "ghost.md") - assert res["parse_errors"] - assert res["symbols"] == [] - assert res["discoveries"] == [] - - -# --- parse_cross_cutting --------------------------------------------------- - - -def test_parse_cross_cutting_empty_dir(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_cross").mkdir() - assert shadow_viewer.parse_cross_cutting(sd) == [] - - -def test_parse_cross_cutting_no_dir(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - # _cross/ not created - assert shadow_viewer.parse_cross_cutting(sd) == [] - - -def test_parse_cross_cutting_one_entry(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/example-pattern.md", """\ - # Example pattern - - **Category**: pattern - **Refs**: - - `cart.py::calculate_total` - - `inventory.py::validate_coupon` - - **Discovery**: A multi-file pattern observed across the codebase. - - _(verified, source: exploration, labels: [bug])_ - """) - entries = shadow_viewer.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["slug"] == "example-pattern" - assert e["title"] == "Example pattern" - assert e["category"] == "pattern" - assert "cart.py::calculate_total" in e["refs"] - assert "inventory.py::validate_coupon" in e["refs"] - assert "multi-file pattern" in e["discovery"] - assert e["status"] == "verified" - assert e["source"] == "exploration" - assert e["labels"] == ["bug"] - - -def test_parse_cross_cutting_no_labels(shadow_viewer, tmp_path): - """A cross-cutting entry without labels is still parsed, with no - `labels` key in the entry.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-label.md", """\ - # Plain entry - - **Category**: behavior - **Refs**: - - `foo.py::bar` - - **Discovery**: Something happens. - - _(uncertain, source: exploration)_ - """) - entries = shadow_viewer.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["status"] == "uncertain" - assert e["source"] == "exploration" - assert "labels" not in e - - -def test_parse_cross_cutting_minor_format_variation(shadow_viewer, tmp_path): - """File missing a `**Category**:` field doesn't crash; entry is still - emitted with whatever fields could be parsed.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/sparse.md", """\ - # Sparse entry - - Some prose with no structured fields. - - _(refuted, source: user)_ - """) - entries = shadow_viewer.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["slug"] == "sparse" - assert e["title"] == "Sparse entry" - assert e["status"] == "refuted" - assert e["source"] == "user" - - -# --- parse_prefs ----------------------------------------------------------- - - -def test_parse_prefs_missing_file(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - assert shadow_viewer.parse_prefs(sd) == [] - - -def test_parse_prefs_zero(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - _No preferences recorded yet._ - """) - assert shadow_viewer.parse_prefs(sd) == [] - - -def test_parse_prefs_one(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Always use type hints on public APIs. - _(source: user)_ - """) - prefs = shadow_viewer.parse_prefs(sd) - assert len(prefs) == 1 - assert prefs[0]["text"] == "Always use type hints on public APIs." - assert prefs[0]["source"] == "user" - assert prefs[0]["type"] == "preference" - - -def test_parse_prefs_three(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Use kebab-case for slugs. - _(source: user)_ - - Never commit secrets to source control. - _(source: interaction)_ - - Prefer fail-fast for required dependencies. - _(source: user)_ - """) - prefs = shadow_viewer.parse_prefs(sd) - assert len(prefs) == 3 - texts = [p["text"] for p in prefs] - assert any("kebab-case" in t for t in texts) - assert any("secrets" in t for t in texts) - assert any("fail-fast" in t for t in texts) - sources = [p["source"] for p in prefs] - assert "user" in sources - assert "interaction" in sources - - -# --- load_state ------------------------------------------------------------ - - -def test_load_state_missing(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - assert shadow_viewer.load_state(sd) == {} - - -def test_load_state_valid_json(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - payload = { - "version": 1, - "total_files": 5, - "total_discoveries": 42, - "last_update_type": "auto", - } - (sd / "_meta" / "state.json").write_text( - json.dumps(payload), encoding="utf-8" - ) - state = shadow_viewer.load_state(sd) - assert state == payload - - -def test_load_state_malformed_json(shadow_viewer, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text( - "{not valid json", encoding="utf-8" - ) - state = shadow_viewer.load_state(sd) - assert state == {} - - -def test_load_state_non_object_json(shadow_viewer, tmp_path): - """JSON parses but is not a dict — returns empty dict sentinel.""" - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text("[1, 2, 3]", encoding="utf-8") - assert shadow_viewer.load_state(sd) == {} - - -# --- get_all_shadow_files -------------------------------------------------- - - -def test_get_all_shadow_files_excludes_special(shadow_viewer, tmp_path): - """Per-file shadows are returned; _cross/, _dreams/, _meta/, _index.md, - _prefs.md, state.json are all excluded.""" - sd = _make_shadow_root(tmp_path) - # Files that SHOULD be returned - _write_shadow(sd, "a.py.md", "# Shadow: a.py\n") - _write_shadow(sd, "src/b.py.md", "# Shadow: src/b.py\n") - _write_shadow(sd, "deep/nested/c.py.md", "# Shadow: deep/nested/c.py\n") - # Files that should be EXCLUDED - _write_shadow(sd, "_index.md", "# Shadow Index\n") - _write_shadow(sd, "_prefs.md", "# Preferences\n") - _write_shadow(sd, "_cross/some.md", "# Some cross\n") - _write_shadow(sd, "_dreams/dream-1/report.md", "# A dream\n") - _write_shadow(sd, "_meta/state.json", "{}") - # Junk files that are not .md and shouldn't show up anyway - (sd / "notes.json").write_text("{}", encoding="utf-8") - - files = shadow_viewer.get_all_shadow_files(sd) - rels = sorted(f.relative_to(sd).as_posix() for f in files) - assert rels == ["a.py.md", "deep/nested/c.py.md", "src/b.py.md"] - - -def test_get_all_shadow_files_against_coupon_demo(shadow_viewer, coupon_demo): - sd = coupon_demo / ".shadow" - files = shadow_viewer.get_all_shadow_files(sd) - rels = sorted(f.relative_to(sd).as_posix() for f in files) - assert rels == ["cart.py.md", "inventory.py.md", "test_cart.py.md"] - # Make sure none of the special files leaked through - for r in rels: - assert not r.startswith(("_cross/", "_dreams/", "_meta/")) - assert r not in ("_index.md", "_prefs.md") - - -# --- collect_all_discoveries (against fixture) ----------------------------- - - -def test_collect_all_discoveries_counts(shadow_viewer, coupon_demo): - """Coupon demo has 33 per-file discoveries (matches state.json).""" - sd = coupon_demo / ".shadow" - all_disc = shadow_viewer.collect_all_discoveries(sd) - # state.json claims 33 total discoveries - assert len(all_disc) == 33 - - # File breakdown: cart=14, inventory=10, test_cart=9 - by_file = {} - for d in all_disc: - by_file.setdefault(d.get("file"), 0) - by_file[d.get("file")] += 1 - assert by_file == {"cart.py": 14, "inventory.py": 10, "test_cart.py": 9} - - -def test_collect_all_discoveries_shadow_path_and_mtime( - shadow_viewer, coupon_demo -): - sd = coupon_demo / ".shadow" - all_disc = shadow_viewer.collect_all_discoveries(sd) - for d in all_disc: - assert "shadow_path" in d - assert d["shadow_path"].endswith(".md") - assert "shadow_mtime" in d - assert isinstance(d["shadow_mtime"], float) - - -def test_collect_all_discoveries_b3_no_dream_report_in_text( - shadow_viewer, coupon_demo -): - """B3 regression on real fixture: no discovery body should contain - 'Dream report' or the literal `_dreams/` slug path.""" - sd = coupon_demo / ".shadow" - all_disc = shadow_viewer.collect_all_discoveries(sd) - leaks = [ - d for d in all_disc - if "Dream report" in d.get("text", "") - or "_dreams/" in d.get("text", "") - ] - assert leaks == [], ( - f"Dream report leaked into {len(leaks)} discovery body/bodies: " - f"{[d['text'][:80] for d in leaks]}" - ) - - # And at least some discoveries actually have dream_report metadata - # (the fixture has several Dream report continuation lines) - with_dream = [d for d in all_disc if d.get("dream_report")] - assert len(with_dream) >= 3, ( - "Expected coupon-demo fixture to have multiple Dream-report-tagged " - f"discoveries; found {len(with_dream)}" - ) - for d in with_dream: - assert d["dream_report"].startswith("_dreams/") - - -# --- CLI: --summary -------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_summary(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--summary") - assert r.returncode == 0, r.stderr - out = r.stdout - # Header is "Files shadowed:", "Symbols tracked:", "Discoveries:" per - # current --summary output. Task wording used "Total files:"/"Symbols:" - # /"Discoveries:" loosely — match the actual labels. - assert "Files shadowed:" in out - assert "Symbols tracked:" in out - assert "Discoveries:" in out - # Cross-cutting block exists - assert "Cross-cutting" in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_default_view_is_summary(repo_root, coupon_demo): - """Running with no flags should produce the summary view.""" - r = _run_viewer(repo_root, coupon_demo) - assert r.returncode == 0, r.stderr - assert "Shadow Knowledge Base Summary" in r.stdout - - -# --- CLI: --search --------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_search_matches_text(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--search", "coupon") - assert r.returncode == 0, r.stderr - # Header echoes the query and results were found - assert "'coupon'" in r.stdout - # Some matched line should contain the query (case-insensitive) - assert "coupon" in r.stdout.lower() - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_search_no_results(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--search", "zzznotpresentzzz") - assert r.returncode == 0, r.stderr - assert "No results for 'zzznotpresentzzz'" in r.stdout - - -# --- CLI: --top ------------------------------------------------------------ - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_cart_py_header_and_no_dream_report_leak( - repo_root, coupon_demo -): - """B3 regression at the CLI layer: --top output for a file whose shadow - contains `Dream report:` continuation lines must NOT inline that text - in any discovery body.""" - r = _run_viewer(repo_root, coupon_demo, "--top", "cart.py") - assert r.returncode == 0, r.stderr - out = r.stdout - # Header form: "Top N of M actionable discoveries for cart.py:" - assert "actionable discoveries for cart.py:" in out - # Match the documented prefix exactly - assert out.splitlines()[0].startswith("Top ") - assert "for cart.py:" in out.splitlines()[0] - - # B3: no literal "Dream report:" or raw `_dreams/...` slug paths in any - # of the discovery bullets - assert "Dream report:" not in out - assert "_dreams/" not in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_label_filter_bug(repo_root, coupon_demo): - r = _run_viewer( - repo_root, coupon_demo, "--top", "cart.py", "--top-labels", "bug" - ) - assert r.returncode == 0, r.stderr - out = r.stdout - assert "for cart.py:" in out - # Every bulleted result line should mention 'bug' in its label bracket - bullet_lines = [ - l for l in out.splitlines() if l.startswith("- [") - ] - assert bullet_lines, f"No bullets in --top output:\n{out}" - for line in bullet_lines: - bracket = line.split("]", 1)[0] - assert "bug" in bracket, ( - f"Expected 'bug' label in bracket of: {line}" - ) - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_unknown_file(repo_root, coupon_demo): - """A file with no shadow + no cross refs reports nothing actionable.""" - r = _run_viewer(repo_root, coupon_demo, "--top", "does/not/exist.py") - assert r.returncode == 0, r.stderr - assert "No actionable discoveries" in r.stdout - - -# --- CLI: --labels (repo-wide) --------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_labels_bug(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--labels", "bug") - assert r.returncode == 0, r.stderr - out = r.stdout - assert "label(s): bug" in out - # There are multiple bug-labeled discoveries in the fixture - assert "[bug]" in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_labels_unknown(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--labels", "nonexistent") - assert r.returncode == 0, r.stderr - assert "No discoveries with label(s): nonexistent" in r.stdout - - -# --- CLI: --recent --------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_recent_caps_at_n(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--recent", "5") - assert r.returncode == 0, r.stderr - out = r.stdout - assert "Most Recent Discoveries (top 5)" in out - # Each recent entry has a timestamp prefix " [YYYY-MM-DD HH:MM]" - entries = [l for l in out.splitlines() if l.strip().startswith("[20")] - assert len(entries) <= 5 - # Coupon-demo has > 5 total items so we expect exactly 5 - assert len(entries) == 5 - - -# --- CLI: --prefs ---------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_prefs_empty(repo_root, coupon_demo): - """Coupon-demo ships with no preferences recorded.""" - r = _run_viewer(repo_root, coupon_demo, "--prefs") - assert r.returncode == 0, r.stderr - assert "No preferences recorded yet." in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_prefs_populated(repo_root, coupon_demo): - """After populating _prefs.md, --prefs lists them.""" - prefs_path = coupon_demo / ".shadow" / "_prefs.md" - prefs_path.write_text( - textwrap.dedent("""\ - # Preferences - - - Use snake_case for Python identifiers. - _(source: user)_ - - Avoid mutable default arguments. - _(source: interaction)_ - """), - encoding="utf-8", - ) - r = _run_viewer(repo_root, coupon_demo, "--prefs") - assert r.returncode == 0, r.stderr - assert "Project Preferences (2 total)" in r.stdout - assert "snake_case" in r.stdout - assert "mutable default" in r.stdout - - -# --- CLI: --check-invariants ----------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_clean(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode == 0, ( - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - assert "✓ Invariants OK" in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_detects_missing_cross_file( - repo_root, coupon_demo -): - """Deleting a _cross/ file leaves dangling back-pointers in per-file - shadows. Invariant #5 must flag this as a violation.""" - cross_path = ( - coupon_demo / ".shadow" / "_cross" - / "coupon-case-normalization-mismatch.md" - ) - assert cross_path.is_file() - cross_path.unlink() - - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode != 0, ( - f"Expected nonzero exit when _cross/ file missing; got 0.\n" - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - # Violations report the dangling slug - assert "coupon-case-normalization-mismatch" in r.stdout - assert "cross-ref" in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_detects_bad_heading(repo_root, coupon_demo): - """Renaming a symbol heading to a non-backtick form is a heading-format - violation.""" - cart_path = coupon_demo / ".shadow" / "cart.py.md" - text = cart_path.read_text(encoding="utf-8") - # `## `COUPON_CACHE`` -> `## COUPON_CACHE` (drop backticks) - mutated = text.replace("## `COUPON_CACHE`", "## COUPON_CACHE", 1) - assert mutated != text, "Substitution did not match" - cart_path.write_text(mutated, encoding="utf-8") - - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode != 0, ( - f"Expected nonzero exit for bad heading; got 0.\n" - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - assert "heading" in r.stdout - - -# --- CLI: missing shadow dir ---------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_missing_shadow_dir_fails(repo_root, tmp_path): - """Running with no shadow and a bogus --shadow-dir exits 1.""" - r = _run_viewer( - repo_root, tmp_path, - "--shadow-dir", str(tmp_path / "nope"), "--summary", - ) - assert r.returncode == 1 - assert "No .shadow/ directory found" in r.stderr - - -# =========================================================================== -# RENDER FUNCTIONS: in-process tests -# -# The CLI integration tests above invoke shadow-viewer.py as a subprocess — -# that exercises the dispatcher but doesn't contribute to coverage of the -# loaded module. The tests below call view_* and main() directly via the -# `shadow_viewer` fixture so coverage actually accumulates. -# =========================================================================== - - -# --- shared helpers -------------------------------------------------------- - - -def _call_main(shadow_viewer, argv): - """Invoke `shadow_viewer.main()` in-process with the given argv. - - Returns the integer exit code. We mutate `sys.argv` directly (no - monkeypatch / no mocking) and always restore it in a finally. - """ - saved_argv = sys.argv - sys.argv = ["shadow-viewer.py", *argv] - try: - shadow_viewer.main() - return 0 - except SystemExit as e: - code = e.code - if code is None: - return 0 - if isinstance(code, int): - return code - return 1 - finally: - sys.argv = saved_argv - - -def _make_minimal_shadow(tmp_path, extras=None): - """Build a minimal but valid .shadow/ tree in tmp_path. - - Returns the .shadow/ Path. `extras` is a dict of {relpath: content} - appended on top of the baseline. - """ - sd = tmp_path / ".shadow" - sd.mkdir() - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text( - json.dumps({ - "version": 1, - "last_update_at": "2026-04-20T16:30:00Z", - "last_update_type": "manual", - "last_commit": "deadbeef" * 5, - }), - encoding="utf-8", - ) - _write_shadow(sd, "foo.py.md", """\ - # Shadow: foo.py - - **Language**: Python | **Lines**: 10 - - ## `bar` - - - A neat bug. - _(verified, source: exploration, labels: [bug])_ - """) - for rel, content in (extras or {}).items(): - _write_shadow(sd, rel, content) - return sd - - -# =========================================================================== -# view_summary -# =========================================================================== - - -class TestViewSummary: - """In-process tests for `view_summary(shadow_dir)`.""" - - def test_basic_header_on_coupon_demo( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Shadow Knowledge Base Summary" in out - assert "=" * 50 in out - assert "Files shadowed:" in out - assert "Symbols tracked:" in out - assert "Discoveries:" in out - assert "Preferences:" in out - assert "Cross-cutting:" in out - - def test_counts_reflect_fixture( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - # 3 source files, 33 discoveries, 3 cross-cutting (per fixture) - assert "Files shadowed: 3" in out - assert "Discoveries: 33" in out - assert "Cross-cutting: 3" in out - - def test_by_source_section_renders( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "By source:" in out - # All discoveries in the fixture are source: exploration - assert "exploration" in out - - def test_by_status_section_renders( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "By status:" in out - assert "verified" in out - - def test_by_label_section_renders( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "By label:" in out - assert "bug" in out - assert "security" in out - - def test_per_file_table_lists_files( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - # Per-file table header - assert "File" in out - assert "Symbols" in out - assert "Disc." in out - # All three fixture files appear in the table - assert "cart.py" in out - assert "inventory.py" in out - assert "test_cart.py" in out - - def test_cross_cutting_titles_section( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Cross-cutting discoveries:" in out - assert "Coupon case normalization mismatch" in out - assert "[edge-case]" in out - - def test_state_info_section( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Last update:" in out - assert "Last commit:" in out - # The fixture state.json says last_update_type: dream - assert "(dream)" in out - - def test_empty_shadow_dir_renders_zeros( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Files shadowed: 0" in out - assert "Discoveries: 0" in out - # With zero discoveries there's no source/status breakdown - assert "By source:" not in out - assert "By status:" not in out - assert "By label:" not in out - # And no state info (no state.json) - assert "Last update:" not in out - - def test_corrupted_state_json_does_not_break_summary( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # Clobber state.json with junk - (sd / "_meta" / "state.json").write_text( - "{ not json", encoding="utf-8" - ) - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - # Header and counts still rendered - assert "Shadow Knowledge Base Summary" in out - assert "Files shadowed: 1" in out - # State section silently dropped - assert "Last update:" not in out - - def test_unreadable_shadow_file_does_not_break_summary( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # Create a file with invalid UTF-8 — parse_shadow_file records a - # parse_error but returns a result. view_summary should still - # render the rest of the report. - bad = sd / "bad.py.md" - bad.write_bytes(b"\xff\xfe\x00garbage\x00") - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Shadow Knowledge Base Summary" in out - assert "Files shadowed: 2" in out - - def test_more_than_20_files_truncated( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - for i in range(25): - _write_shadow( - sd, f"file{i:02d}.py.md", - f"# Shadow: file{i:02d}.py\n\n## `sym{i}`\n\n" - f"- D\n _(verified, source: exploration)_\n", - ) - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Files shadowed: 25" in out - assert "... and 5 more files" in out - - def test_no_cross_cutting_dir_omits_section( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # no _cross/ dir - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Cross-cutting: 0" in out - # The titled list is suppressed when empty - assert "Cross-cutting discoveries:" not in out - - def test_summary_no_prefs(self, shadow_viewer, tmp_path, capsys): - sd = _make_minimal_shadow(tmp_path) - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Preferences: 0" in out - - def test_summary_with_prefs(self, shadow_viewer, tmp_path, capsys): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Pref one. - _(source: user)_ - - Pref two. - _(source: interaction)_ - """) - shadow_viewer.view_summary(sd) - out = capsys.readouterr().out - assert "Preferences: 2" in out - - -# =========================================================================== -# view_search -# =========================================================================== - - -class TestViewSearch: - """In-process tests for `view_search(shadow_dir, query)`.""" - - def test_finds_text_match(self, shadow_viewer, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "tax") - out = capsys.readouterr().out - assert "Search: 'tax'" in out - assert "results" in out - # The "8% tax" / "Tax rate" discoveries on cart.py match - assert "cart.py" in out - - def test_finds_symbol_match(self, shadow_viewer, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "COUPON_CACHE") - out = capsys.readouterr().out - assert "COUPON_CACHE" in out - assert "cart.py" in out - - def test_finds_file_name_match( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Searching for the literal file name pulls every discovery in - # that file (file_name_hit branch). - shadow_viewer.view_search(sd, "inventory.py") - out = capsys.readouterr().out - assert "inventory.py" in out - assert "matches" in out - - def test_case_insensitive(self, shadow_viewer, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "COUPON") - upper = capsys.readouterr().out - shadow_viewer.view_search(sd, "coupon") - lower = capsys.readouterr().out - # Same number of result lines either way - assert ("results" in upper) and ("results" in lower) - # And both contain at least one match - assert "::" in upper - assert "::" in lower - - def test_no_matches_prints_no_results( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "zzzdoesnotexistzzz") - out = capsys.readouterr().out - assert "No results for 'zzzdoesnotexistzzz'." in out - - def test_finds_cross_cutting_title( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Title of _cross/coupon-case-normalization-mismatch.md is - # "Coupon case normalization mismatch" - shadow_viewer.view_search(sd, "normalization mismatch") - out = capsys.readouterr().out - assert "Cross-cutting" in out - assert "Coupon case normalization mismatch" in out - assert "Category:" in out - assert "edge-case" in out - - def test_finds_cross_cutting_by_ref( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Search for a ref that appears in a _cross file - shadow_viewer.view_search(sd, "apply_bulk_discount") - out = capsys.readouterr().out - assert "Cross-cutting" in out - assert "Mutation through discount pipeline" in out - - def test_finds_also_involves_match( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - Also involves: `b.py::weird_symbol_zzz` - """) - shadow_viewer.view_search(sd, "weird_symbol_zzz") - out = capsys.readouterr().out - # The match flag is "also_involves" and a line shows that ref - assert "weird_symbol_zzz" in out - assert "Also involves:" in out - - def test_finds_preference_match( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Prefer rusty pelicans for everything. - _(source: user)_ - """) - shadow_viewer.view_search(sd, "pelican") - out = capsys.readouterr().out - assert "Preferences" in out - assert "pelican" in out.lower() - assert "[user]" in out - - def test_groups_per_file_results_by_file( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "coupon") - out = capsys.readouterr().out - # Per-file groups present a header like "cart.py (N matches)" - assert re.search(r"cart\.py \(\d+ matches\)", out) - - def test_results_show_status_and_source( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "tax") - out = capsys.readouterr().out - assert "(verified, source: exploration)" in out - - def test_total_count_in_header( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_search(sd, "coupon") - out = capsys.readouterr().out - m = re.search(r"\((\d+) results\)", out) - assert m is not None - assert int(m.group(1)) > 0 - - def test_empty_shadow_returns_no_results( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_viewer.view_search(sd, "anything") - out = capsys.readouterr().out - assert "No results for 'anything'." in out - - -# =========================================================================== -# view_prefs -# =========================================================================== - - -class TestViewPrefs: - """In-process tests for `view_prefs(shadow_dir)`.""" - - def test_missing_prefs_file(self, shadow_viewer, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - shadow_viewer.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - def test_empty_prefs_placeholder( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - _No preferences recorded yet._ - """) - shadow_viewer.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - def test_populated_prefs_lists_count_and_sources( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Use snake_case. - _(source: user)_ - - Avoid mutable default args. - _(source: interaction)_ - - Prefer fail-fast for required deps. - _(source: user)_ - """) - shadow_viewer.view_prefs(sd) - out = capsys.readouterr().out - assert "Project Preferences (3 total)" in out - assert "[user]" in out - assert "[interaction]" in out - assert "snake_case" in out - assert "fail-fast" in out - assert "mutable default" in out - - def test_against_coupon_demo_is_empty( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - -# =========================================================================== -# view_labels -# =========================================================================== - - -class TestViewLabels: - """In-process tests for `view_labels(shadow_dir, label_filter)`.""" - - @pytest.mark.parametrize("label", [ - "bug", "security", "performance", "feature-gap", "tech-debt", - ]) - def test_each_label_returns_results_on_fixture( - self, shadow_viewer, coupon_demo, capsys, label - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, label) - out = capsys.readouterr().out - assert f"label(s): {label}" in out - assert f"[{label}]" in out - # Result header always has "(N results)" - m = re.search(r"\((\d+) results\)", out) - assert m and int(m.group(1)) >= 1 - - def test_unknown_label_no_results( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, "zznotalabelzz") - out = capsys.readouterr().out - assert "No discoveries with label(s): zznotalabelzz" in out - - def test_multiple_labels_comma_split( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, "bug,security") - out = capsys.readouterr().out - assert "label(s): bug, security" in out - # Both grouped section headers present - assert "[bug]" in out - assert "[security]" in out - - def test_label_filter_is_lowercased( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, "BUG") - out = capsys.readouterr().out - # Filter is lowercased before matching - assert "label(s): bug" in out - assert "[bug]" in out - - def test_cross_cutting_labels_included( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels( - sd, "bug", options=shadow_viewer.RetrievalOptions(limit=20, max_chars=8000), - ) - out = capsys.readouterr().out - # Cross-cutting entries are prefixed with `_cross/` in the - # file column. - assert "_cross/" in out - - def test_also_labeled_displayed( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # One cart.py discovery has labels [bug, performance] - shadow_viewer.view_labels(sd, "bug") - out = capsys.readouterr().out - assert "Also labeled:" in out - - def test_empty_shadow_no_results( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_viewer.view_labels(sd, "bug") - out = capsys.readouterr().out - assert "No discoveries with label(s): bug" in out - - def test_each_result_row_has_file_and_symbol( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_labels(sd, "feature-gap") - out = capsys.readouterr().out - # `::` separator for file::symbol form - assert "::" in out - assert "(verified, source: exploration)" in out - - -# =========================================================================== -# view_recent -# =========================================================================== - - -class TestViewRecent: - """In-process tests for `view_recent(shadow_dir, count)`.""" - - def test_default_count_caps_at_10( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_recent(sd, 10) - out = capsys.readouterr().out - assert "Most Recent Discoveries (top 10)" in out - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - # Fixture has 33 per-file + 3 cross + 0 prefs > 10 - assert len(entries) == 10 - - def test_custom_count(self, shadow_viewer, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_viewer.view_recent(sd, 3) - out = capsys.readouterr().out - assert "Most Recent Discoveries (top 3)" in out - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - assert len(entries) == 3 - - def test_count_larger_than_available( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - shadow_viewer.view_recent(sd, 50) - out = capsys.readouterr().out - # Only 1 discovery exists in the minimal shadow - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - assert len(entries) == 1 - - def test_empty_shadow_prints_nothing( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_viewer.view_recent(sd, 10) - out = capsys.readouterr().out - assert "No discoveries found." in out - - def test_recently_modified_file_appears_first( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Bump test_cart.py's mtime to "now" so its discoveries should - # rank first. - target = sd / "test_cart.py.md" - now = datetime.now().timestamp() - os.utime(target, (now + 10, now + 10)) - shadow_viewer.view_recent(sd, 3) - out = capsys.readouterr().out - # First entry block should reference test_cart.py - first_entry_idx = out.find("[20") - first_block = out[first_entry_idx:first_entry_idx + 400] - assert "test_cart.py" in first_block - - def test_includes_cross_cutting_type( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Bump a cross file mtime so it shows up in the top - cf = sd / "_cross" / "coupon-case-normalization-mismatch.md" - now = datetime.now().timestamp() - os.utime(cf, (now + 100, now + 100)) - shadow_viewer.view_recent(sd, 5) - out = capsys.readouterr().out - assert "(cross-cutting)" in out - - def test_includes_preferences_when_present( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - A pref we care about. - _(source: user)_ - """) - shadow_viewer.view_recent(sd, 10) - out = capsys.readouterr().out - assert "(preference)" in out - assert "A pref we care about" in out - # Preference-typed rows show `source:` not `(verified, ...)` - assert "source: user" in out - - def test_no_cross_dir_does_not_crash( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # No _cross/ created - shadow_viewer.view_recent(sd, 10) - out = capsys.readouterr().out - assert "Most Recent Discoveries" in out - - def test_entries_include_timestamp_and_kind( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_viewer.view_recent(sd, 2) - out = capsys.readouterr().out - # Entry header line format: " [YYYY-MM-DD HH:MM] (kind)" - assert re.search( - r"\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}\] \((discovery|cross-cutting|preference)\)", - out, - ) - - -# =========================================================================== -# view_check_invariants -# =========================================================================== - - -class TestViewCheckInvariants: - """In-process tests for `view_check_invariants(shadow_dir)`. - - Each test builds a deliberately broken `.shadow/` tree in tmp_path - and asserts that the right violation kind is reported. - """ - - def test_clean_coupon_demo_returns_zero( - self, shadow_viewer, coupon_demo, capsys - ): - rc = shadow_viewer.view_check_invariants(coupon_demo / ".shadow") - out = capsys.readouterr().out - assert rc == 0 - assert "✓ Invariants OK" in out - - def test_empty_shadow_returns_zero( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0 - assert "Invariants OK" in out - - def test_missing_cross_file_is_violation( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - (sd / "_cross" / "coupon-case-normalization-mismatch.md").unlink() - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "cross-ref" in out - assert "coupon-case-normalization-mismatch" in out - - def test_invalid_status_enum(self, shadow_viewer, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad status. - _(maybeverified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "maybeverified" in out - - def test_invalid_source_enum(self, shadow_viewer, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad source. - _(verified, source: psychic)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "psychic" in out - - def test_invalid_label(self, shadow_viewer, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad label. - _(verified, source: exploration, labels: [unicorn])_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "unicorn" in out - - def test_heading_without_backticks( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_in_backticks - - - Hi. - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "heading" in out - - def test_file_level_heading_does_not_violate( - self, shadow_viewer, tmp_path, capsys - ): - """`## File-Level` is allowed without backticks.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## File-Level - - - A file-level discovery. - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0, out - assert "Invariants OK" in out - - def test_also_involves_without_backticks( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - With bad anchors. - _(verified, source: exploration)_ - Also involves: b.py::bar, c.py::baz - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "anchor" in out - - def test_also_involves_missing_symbol_after_colons( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Empty sym. - _(verified, source: exploration)_ - Also involves: `b.py::` - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - # The empty-symbol form fails the strict regex (which requires - # non-empty after ::), so the file_sym_re finds zero anchors and - # we hit the "needs backtick anchors" branch instead. - assert "anchor" in out - assert "needs `file::symbol`" in out - - def test_cross_missing_category( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - """) - _write_shadow(sd, "_cross/no-cat.md", """\ - # No category here - - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "Category" in out - - def test_cross_invalid_category( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/bad-cat.md", """\ - # Bad category here - - **Category**: bogus - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "bogus" in out - - def test_cross_missing_metadata_line( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-meta.md", """\ - # No meta here - - **Category**: pattern - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "missing trailing" in out - - def test_cross_missing_refs_block( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-refs.md", """\ - # No refs here - - **Category**: pattern - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "Refs" in out - - def test_cross_ref_missing_symbol( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - # Note: file_sym_re demands "::" in backticks. We use a backtick - # ref with no symbol after `::`. - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - hi. - _(verified, source: exploration)_ - - ## Cross-References - - - [bad-anchor](_cross/bad-anchor.md) - """) - _write_shadow(sd, "_cross/bad-anchor.md", """\ - # Bad anchor - - **Category**: pattern - **Refs**: - - `a.py::` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "anchor" in out - - def test_back_pointer_missing( - self, shadow_viewer, tmp_path, capsys - ): - """_cross/x.md references a.py::foo but a.py.md has no - Cross-References section pointing back to x.md.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - """) - _write_shadow(sd, "_cross/orphan.md", """\ - # Orphan - - **Category**: pattern - **Refs**: - - `a.py::foo` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "cross-ref" in out - assert "does not link back to" in out - - def test_back_pointer_references_nonexistent_file( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/ghost.md", """\ - # Ghost - - **Category**: pattern - **Refs**: - - `does/not/exist.py::foo` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "no such shadow file exists" in out - - def test_violation_line_format( - self, shadow_viewer, tmp_path, capsys - ): - """Output is grep-friendly: `path:line: kind: message`.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_backticked - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - # At least one line of form path:line: kind: msg - assert re.search(r"a\.py\.md:\d+: heading: ", out) - - def test_multiple_violations_reported( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_backticked - - ## `foo` - - - bad. - _(maybeverified, source: psychic, labels: [unicorn])_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - err = capsys.readouterr().err - assert rc == 1 - # heading + 3 enum violations = at least 4 lines - violation_lines = [ - l for l in out.splitlines() - if re.match(r"^[^:]+:\d+: \w+: ", l) - ] - assert len(violation_lines) >= 3 - - def test_count_summary_emitted_on_stderr( - self, shadow_viewer, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", "## not_backticked\n") - rc = shadow_viewer.view_check_invariants(sd) - captured = capsys.readouterr() - assert rc == 1 - # Final count line goes to stderr - assert "invariant violation(s) found" in captured.err - - def test_special_headings_allowed( - self, shadow_viewer, tmp_path, capsys - ): - """`## Notes`, `## Metadata`, `## File-Level Notes` are allowed - without backticks.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## Notes - - Some prose. - - ## Metadata - - Some metadata. - - ## File-Level Notes - - More prose. - - ## `real_sym` - - - A discovery. - _(verified, source: exploration)_ - """) - rc = shadow_viewer.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0, out - - -# =========================================================================== -# main() — in-process via sys.argv (covers the dispatcher) -# =========================================================================== - - -class TestMainInProcess: - """Drive `main()` directly so coverage of the dispatch arms is captured. - - All paths use `--shadow-dir ` to avoid relying - on cwd. Where we need a true CLI smoke (e.g., to verify `--help` - output), use subprocess. - """ - - def _shadow(self, coupon_demo): - return str(coupon_demo / ".shadow") - - def test_summary_dispatch( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, ["--shadow-dir", self._shadow(coupon_demo), - "--summary"] - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_no_args_default_is_summary( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, ["--shadow-dir", self._shadow(coupon_demo)] - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_search_dispatch( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--search", "coupon"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Search: 'coupon'" in out - - def test_prefs_dispatch( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), "--prefs"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "No preferences recorded yet." in out - - def test_labels_dispatch_bug( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--labels", "bug"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "label(s): bug" in out - - def test_recent_dispatch_with_n( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--recent", "4"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Most Recent Discoveries (top 4)" in out - - def test_recent_dispatch_no_n( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), "--recent"], - ) - out = capsys.readouterr().out - assert rc == 0 - # Default count is 10 - assert "Most Recent Discoveries (top 10)" in out - - def test_top_dispatch( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--top", "cart.py"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "for cart.py:" in out - - def test_top_dispatch_with_labels_and_limits( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--top", "cart.py", - "--top-labels", "bug", - "--top-limit", "2", - "--top-max-chars", "0"], - ) - out = capsys.readouterr().out - assert rc == 0 - bullets = [l for l in out.splitlines() if l.startswith("- [")] - assert len(bullets) <= 2 - - def test_check_invariants_clean_exits_zero( - self, shadow_viewer, coupon_demo, capsys - ): - rc = _call_main( - shadow_viewer, - ["--shadow-dir", self._shadow(coupon_demo), - "--check-invariants"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Invariants OK" in out - - def test_check_invariants_dirty_exits_one( - self, shadow_viewer, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Inject a heading violation - cart = sd / "cart.py.md" - cart.write_text( - cart.read_text(encoding="utf-8").replace( - "## `COUPON_CACHE`", "## COUPON_CACHE", 1 - ), - encoding="utf-8", - ) - rc = _call_main( - shadow_viewer, - ["--shadow-dir", str(sd), "--check-invariants"], - ) - out = capsys.readouterr().out - assert rc == 1 - assert "heading" in out - - def test_explicit_shadow_dir_missing( - self, shadow_viewer, tmp_path, capsys - ): - bogus = tmp_path / "does-not-exist" - rc = _call_main( - shadow_viewer, ["--shadow-dir", str(bogus), "--summary"] - ) - err = capsys.readouterr().err - assert rc == 1 - assert "No .shadow/ directory found" in err - - def test_auto_detect_via_chdir( - self, shadow_viewer, coupon_demo, monkeypatch, capsys - ): - """No --shadow-dir: cwd is walked up to find .shadow/.""" - monkeypatch.chdir(coupon_demo) - rc = _call_main(shadow_viewer, ["--summary"]) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_auto_detect_no_shadow_in_cwd( - self, shadow_viewer, tmp_path, monkeypatch, capsys - ): - monkeypatch.chdir(tmp_path) - rc = _call_main(shadow_viewer, ["--summary"]) - err = capsys.readouterr().err - assert rc == 1 - assert "No .shadow/ directory found" in err - - -# --- main(): subprocess smoke (covers true argv parsing, help, errors) ---- - - -class TestMainSubprocess: - """End-to-end CLI smoke. Subprocess output is the contract here — - these don't add coverage but they catch dispatcher / argparse regressions - the in-process tests can't (e.g. --help, mutually exclusive errors).""" - - @pytest.mark.slow - @pytest.mark.integration - def test_help_exits_zero(self, repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--help") - assert r.returncode == 0 - # argparse prints usage and the description - assert "usage:" in r.stdout.lower() - assert "--summary" in r.stdout - assert "--search" in r.stdout - assert "--check-invariants" in r.stdout - - @pytest.mark.slow - @pytest.mark.integration - def test_unknown_flag_exits_nonzero(self, repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--no-such-flag") - assert r.returncode != 0 - assert "unrecognized" in r.stderr or "unrecognized" in r.stdout - - @pytest.mark.slow - @pytest.mark.integration - def test_mutually_exclusive_flags(self, repo_root, coupon_demo): - """--summary and --prefs are in the same exclusive group.""" - r = _run_viewer( - repo_root, coupon_demo, "--summary", "--prefs" - ) - assert r.returncode != 0 - # argparse error mentions "not allowed with" - assert "not allowed with" in r.stderr +@pytest.mark.parametrize("layout", [".github", ".claude"]) +def test_user_and_agent_interfaces_share_citation_identity(repo_root, tmp_path, layout): + shadow = tmp_path / ".shadow" + shadow.mkdir() + content = "# Shadow: source.py\n\n## `run`\n\n- Shared knowledge.\n _(verified, source: exploration)_\n" + (shadow / "source.py.md").write_text(content, encoding="utf-8") + for skill in ("shadow-frog", "shadow-frog-viewer"): + shutil.copytree( + repo_root / "skills" / skill, tmp_path / layout / "skills" / skill, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + viewer = tmp_path / layout / "skills/shadow-frog-viewer/shadow-viewer.py" + reader = tmp_path / layout / "skills/shadow-frog/shadow-read.py" + env = os.environ.copy() + env.pop("PYTHONDONTWRITEBYTECODE", None) + env.pop("PYTHONPYCACHEPREFIX", None) + + def call(script, *args): + return subprocess.run( + [sys.executable, str(script), "--shadow-dir", str(shadow), *args], + cwd=tmp_path, env=env, capture_output=True, text=True, encoding="utf-8", check=True, + ).stdout + + overview = call(viewer) + assert "Shadow Knowledge Base Summary" in overview + user_result = call(viewer, "--search", "Shared") + assert "citation_score=0" in user_result + agent_result = call(reader, "source.py::run") + assert "citation_score=1" in agent_result + assert re.findall(r"id=(d_[0-9a-f]{32})", user_result) == re.findall( + r"id=(d_[0-9a-f]{32})", agent_result, + ) + assert (shadow / "source.py.md").read_text(encoding="utf-8") == content + assert not list((tmp_path / layout / "skills").rglob("*.pyc")) + assert "for users" in call(viewer, "--help") + assert "Optional bounded agent retrieval" in call(reader, "--help") diff --git a/tests/test_smoke.py b/tests/test_smoke.py index 3d57fc4..7a10cf2 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -14,10 +14,13 @@ def test_repo_root_resolves(repo_root): def test_all_script_fixtures_load( shadow_init, shadow_viewer, dream_reconcile, dream_validate, dream_coverage, dream_lineage, meditate_repair, nap, coherence, dream_tools, + shadow_knowledge, shadow_reader, ): for mod, expected_attr in [ (shadow_init, "main"), (shadow_viewer, "main"), + (shadow_reader, "main"), + (shadow_knowledge, "view_file"), (dream_reconcile, "main"), (dream_validate, "main"), (dream_tools, "pin_tooling"), From ee85f142597d5489b25d97ae9490af692ade1837 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 22:10:27 -0400 Subject: [PATCH 07/11] Preserve executable knowledge entry points Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- skills/shadow-frog-viewer/shadow-viewer.py | 0 skills/shadow-frog/shadow-read.py | 0 2 files changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 skills/shadow-frog-viewer/shadow-viewer.py mode change 100644 => 100755 skills/shadow-frog/shadow-read.py diff --git a/skills/shadow-frog-viewer/shadow-viewer.py b/skills/shadow-frog-viewer/shadow-viewer.py old mode 100644 new mode 100755 diff --git a/skills/shadow-frog/shadow-read.py b/skills/shadow-frog/shadow-read.py old mode 100644 new mode 100755 From aa8bb26860697e55c8a3226a40bd504e07ce1b01 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 23:05:40 -0400 Subject: [PATCH 08/11] Keep Responsible AI documentation unchanged in citation PR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- RESPONSIBLE_AI.md | 15 --------------- 1 file changed, 15 deletions(-) diff --git a/RESPONSIBLE_AI.md b/RESPONSIBLE_AI.md index a7f11ed..f5ad374 100644 --- a/RESPONSIBLE_AI.md +++ b/RESPONSIBLE_AI.md @@ -104,21 +104,6 @@ At a high level, we found that ShadowFrog performed strongly on knowledge retrie ## Limitations -Citation scores are local counts of discovery content emitted by the optional -core retrieval helper or the user-facing Viewer, -not proof that an agent used it, that it improved an outcome, or that it is true. -Direct file/symbol reads are the primary agent workflow and do not require these -helpers or their database. Native reads and failed ledger updates are not counted; -instrumentation is therefore partial by design. Scores can be biased -by prior ranking and repeated exposure; relevance, provenance, and verification -remain more important. Bounded results may omit relevant knowledge, so agents -must follow pagination and inspect preferences rather than treating a shortlist -as exhaustive. Claim rewrites and renames may create fresh zero-score identities. -Continuation chunks share one logical-read event. Short contention budgets can -leave explicitly warned, unrecorded visits; atomicity does not guarantee complete -accounting. Retry receipts and pagination snapshots expire or reach capacity, -while score storage grows with distinct knowledge identities. - ShadowFrog was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios. ShadowFrog was designed and tested using the English language. Performance in other languages may vary and should be assessed by someone who is both an expert in the expected outputs and a native speaker of that language. From 1fb71425b1b250814e3316c8ce31d85639329241 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Wed, 23 Sep 2026 23:54:48 -0400 Subject: [PATCH 09/11] Track explicit knowledge revisits directly in Markdown Use visible citation_score metadata and an exact-claim increment helper with per-file locking and atomic publication. Preserve native navigation and user-facing views, retain counters during knowledge merges, and remove database-backed retrieval and pagination machinery. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- .github/workflows/tests.yml | 14 +- CHANGELOG.md | 21 +- README.md | 27 +- agent-context.md | 30 +- claude.md | 53 +- .../coupon-case-normalization-mismatch.md | 2 +- .../global-coupon-cache-side-effects.md | 2 +- .../mutation-through-discount-pipeline.md | 2 +- examples/coupon-demo/.shadow/cart.py.md | 28 +- examples/coupon-demo/.shadow/inventory.py.md | 20 +- examples/coupon-demo/.shadow/test_cart.py.md | 18 +- .../scripts/shadow-frog-pre-tool.sh | 34 +- skills/shadow-frog-dream/SKILL.md | 19 +- skills/shadow-frog-dream/dream-reconcile.py | 54 +- skills/shadow-frog-dream/dream-tools.py | 4 +- skills/shadow-frog-dream/dream-validate.py | 12 + skills/shadow-frog-init/SKILL.md | 2 + skills/shadow-frog-meditate/SKILL.md | 6 +- skills/shadow-frog-update/SKILL.md | 8 +- skills/shadow-frog-viewer/SKILL.md | 73 +- skills/shadow-frog-viewer/shadow-viewer.py | 1436 ++++++++++- skills/shadow-frog/SKILL.md | 120 +- skills/shadow-frog/_citations.py | 422 ++-- skills/shadow-frog/_knowledge.py | 1613 ------------ skills/shadow-frog/retrieval.md | 108 - skills/shadow-frog/shadow-cite.py | 51 + skills/shadow-frog/shadow-read.py | 34 - tests/conftest.py | 15 +- tests/hooks/test_pre_tool_sh.py | 18 - tests/skills/shadow_frog/conftest.py | 9 - tests/skills/shadow_frog/test_citations.py | 546 ++-- tests/skills/shadow_frog/test_knowledge.py | 2186 ---------------- tests/skills/shadow_frog/test_retrieval.py | 718 ------ tests/skills/shadow_frog/test_shadow_read.py | 171 -- .../shadow_frog_dream/test_citation_scores.py | 87 + .../shadow_frog_dream/test_dream_reconcile.py | 6 +- .../shadow_frog_dream/test_dream_tools.py | 3 +- tests/skills/shadow_frog_nap/test_nap.py | 6 +- tests/skills/shadow_frog_viewer/conftest.py | 9 - .../test_citation_metadata.py | 119 + .../shadow_frog_viewer/test_shadow_viewer.py | 2214 ++++++++++++++++- tests/test_smoke.py | 6 +- 42 files changed, 4522 insertions(+), 5804 deletions(-) delete mode 100644 skills/shadow-frog/_knowledge.py delete mode 100644 skills/shadow-frog/retrieval.md create mode 100755 skills/shadow-frog/shadow-cite.py delete mode 100755 skills/shadow-frog/shadow-read.py delete mode 100644 tests/skills/shadow_frog/conftest.py delete mode 100644 tests/skills/shadow_frog/test_knowledge.py delete mode 100644 tests/skills/shadow_frog/test_retrieval.py delete mode 100644 tests/skills/shadow_frog/test_shadow_read.py create mode 100644 tests/skills/shadow_frog_dream/test_citation_scores.py delete mode 100644 tests/skills/shadow_frog_viewer/conftest.py create mode 100644 tests/skills/shadow_frog_viewer/test_citation_metadata.py diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 6d379b6..0d089c3 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -8,8 +8,8 @@ permissions: contents: read jobs: - viewer-compatibility: - name: Shadow helpers (Python 3.9) + citation-compatibility: + name: Citation helpers (Python 3.9) runs-on: ubuntu-latest steps: - name: Checkout @@ -18,17 +18,15 @@ jobs: uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0 with: python-version: '3.9' - - name: Exercise the standalone viewer + - name: Check citation and viewer entry points run: | - python skills/shadow-frog-viewer/shadow-viewer.py --help - python skills/shadow-frog-viewer/shadow-viewer.py --shadow-dir examples/coupon-demo/.shadow --search coupon --limit 2 --no-record - python skills/shadow-frog/shadow-read.py --help - python skills/shadow-frog/shadow-read.py cart.py --shadow-dir examples/coupon-demo/.shadow --limit 2 --no-record + python skills/shadow-frog/shadow-cite.py --help + python skills/shadow-frog-viewer/shadow-viewer.py --shadow-dir examples/coupon-demo/.shadow --check-invariants pytest: name: pytest (${{ matrix.os }}, Python 3.12) runs-on: ${{ matrix.os }} - timeout-minutes: 15 + timeout-minutes: 10 strategy: matrix: os: [ubuntu-latest, windows-latest] diff --git a/CHANGELOG.md b/CHANGELOG.md index 004b58a..04a7c2a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,11 +10,11 @@ shadow knowledge bases for any codebase. ## Unreleased ### Added -- **Single-score knowledge retrieval (draft)** — derived discovery fingerprints, - zero-default local citation scores, and concurrent-safe SQLite bookkeeping - shared across Git worktrees. Optional core file/symbol retrieval, stable - pagination, and individual expansion avoid loading entire large sections. - Citation scores measure emitted content, not correctness or proven usefulness. +- **Visible knowledge citations** — `citation_score` lives alongside discovery + metadata in Markdown. Agents record consulted entries once per task after + normal file/symbol reads. A small locked increment helper preserves other + content; Viewer and reconciliation understand the same score field without + a database or additional retrieval workflow. ### Fixed - **Reconciliation path containment** — validate untrusted manifest destinations @@ -23,17 +23,6 @@ shadow knowledge bases for any codebase. than modifying files outside the shadow tree. ### Changed -- Citation writes retry temporary SQLite lock contention within their existing - wait budget and use fully synchronized local WAL storage so readers do not - block concurrent score updates. -- Retrieval keeps file/symbol identities consistent, preserves literal whitespace - and duplicate labels, and binds expansion continuations to one unchanged logical - read. Compact hooks share their budget across several previews, and local retry - receipts, pagination snapshots, and journals have explicit retention limits. -- Direct file/symbol navigation remains the agent default. Optional agent - retrieval lives in `shadow-frog`; the Viewer serves user browsing and - visualization. Both share parsing and citation metadata. Hooks use the core - reader and remain fail-open while surfacing citation warnings. - **More concise documentation** — consolidated README onboarding and workflow guidance, with advanced operations linked to the skill references. Condensed repeated guidance and examples in the core, Dream, Init, Meditate, Update, diff --git a/README.md b/README.md index d0bedfd..960bd13 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ helper commands, and format definitions. | [`/shadow-frog-dream`](skills/shadow-frog-dream/SKILL.md) | Run autonomous experiments while you're away | | [`/shadow-frog-nap`](skills/shadow-frog-nap/SKILL.md) | Generate reviewed feature-task briefs without implementing them | | [`/shadow-frog-meditate`](skills/shadow-frog-meditate/SKILL.md) | Merge duplicates and resolve conflicting discoveries | -| [`/shadow-frog-viewer`](skills/shadow-frog-viewer/SKILL.md) | User-facing CLI browsing, lineage visualization, and structural audits | +| [`/shadow-frog-viewer`](skills/shadow-frog-viewer/SKILL.md) | Browse, search, inspect lineage, and audit structural integrity | As you work, the agent captures your code context as `source: user` and collaborative findings as `source: interaction`. After commits, the pre-tool @@ -110,7 +110,6 @@ For example, use Viewer to find relevant knowledge or audit its structure: ``` /shadow-frog-viewer --search "auth" -/shadow-frog-viewer --symbol src/auth.py::login /shadow-frog-viewer --top src/auth.py /shadow-frog-viewer --check-invariants ``` @@ -118,17 +117,12 @@ For example, use Viewer to find relevant knowledge or audit its structure: The [Viewer reference](skills/shadow-frog-viewer/SKILL.md) also covers summaries, recent discoveries, label filters, preferences, and interactive dream-lineage HTML. -Agents normally navigate directly from source files/symbols to their mirrored -Markdown shadows. The **Viewer is for users**; it is not required for agent -lookup. When a section is too large, agents can use the optional core -[`shadow-read.py` helper](skills/shadow-frog/retrieval.md) to read a known file or -symbol within a context budget, or to search and page matching knowledge. - -A single local `citation_score` counts helper exposures, not proven usefulness; -relevance and trust outrank popularity. The core reader and user Viewer share -safe local bookkeeping across worktrees without changing Markdown or requiring -a vector index. Native reads remain normal and uncounted. New claims start at -zero, and long-entry continuation counts as one logical read. +Agents still read shadow files and symbols directly. Each entry carries a visible +`citation_score`, initially 0, as an approximate hint of how often agents revisit +it. After consulting an entry, the agent records it once per task with the small +core `shadow-cite.py` helper; its only job is to update the selected Markdown +counter safely. No database or special retrieval interface is required. Scores +never replace relevance, trust, or verification. --- @@ -231,21 +225,21 @@ For example: ```markdown - authenticate_user() silently returns None on expired tokens instead of raising. 3 of 7 callers don't check the return value. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ ``` **User knowledge**: ```markdown - The retry logic here took 3 iterations to get right -- it handles a subtle race condition during rolling deployments. Do not simplify. - _(verified, source: user)_ + _(verified, source: user, citation_score: 0)_ ``` **Collaborative work**: ```markdown - While debugging issue #42, discovered that process_batch() silently drops items exceeding 1MB -- logged at DEBUG level only. - _(verified, source: interaction)_ + _(verified, source: interaction, citation_score: 0)_ ``` | Property | Values | Meaning | @@ -253,6 +247,7 @@ For example: | **Status** | `verified` / `uncertain` / `refuted` | Has the claim been confirmed? | | **Source** | `exploration` / `user` / `interaction` | Where did this knowledge come from? | | **Labels** | `bug`, `performance`, `security`, `feature-gap`, `tech-debt` | Optional; marks actionable discoveries | +| **Citation score** | Nonnegative integer; missing means `0` | Approximate agent-reported revisits, stored as `citation_score` in the entry | ### Trust Hierarchy diff --git a/agent-context.md b/agent-context.md index 05caaf9..888d8b8 100644 --- a/agent-context.md +++ b/agent-context.md @@ -2,26 +2,26 @@ This project uses a `.shadow/` knowledge base with verified discoveries about non-obvious code behavior. **You MUST consult the shadow before making any code change.** -1. **Read preferences first** — `.shadow/_prefs.md` contains project conventions. -2. **Navigate directly** — before editing ``, read `.shadow/.md` - and its relevant symbol sections using native file reads/searches. -3. **Follow cross-cutting links** — inspect the relevant `.shadow/_cross/` entries. +1. **Check the shadow first** — before editing any file, read its shadow: + ``` + cat .shadow/.md + ``` +2. **Check preferences** — `cat .shadow/_prefs.md` for project conventions +3. **Check cross-cutting** — `cat .shadow/_cross/*.md` for multi-file patterns 4. **Act on what you find** — apply what you learn from the shadow to your work. 5. **After making changes** — run `/shadow-frog-update` to capture learnings The shadow contains discoveries from code analysis and user conversations. Always consult it before making assumptions about code behavior. -File/symbol paths are sufficient; no viewer or database is required to read the -knowledge. For large sections, the optional `shadow-read.py` in the core -`shadow-frog` skill can retrieve a known file/symbol or search and page results. -Use the user-facing `/shadow-frog-viewer` when the **user** asks to browse or visualize knowledge, -not as the mandatory read path for code work. - -Helper reads update one local `citation_score` atomically across worktrees; -continuation chunks share one logical read. Native reads are normal and uncounted. -Never edit counters in Markdown or reread only to increase them. Scores measure -exposure, not correctness or usefulness. A shortlist is not exhaustive: inspect -the specific claim before adding duplicate knowledge. +Each entry's Markdown metadata includes `citation_score` (initially 0; omitted +also means 0). After deliberately consulting an entry, increment it once per +task using `shadow-cite.py` in the core `shadow-frog` skill, supplying the file, +symbol and exact claim text already read. Batch repeated `--text` arguments +for one file/section. Coordinate subagent updates through one writer. +Do not count every entry in an opened file or recount repeated reads. +The score is approximate revisit frequency, not confidence; relevance and trust +take precedence. `/shadow-frog-viewer` is for user-facing inspection, not a +required read path. There is no citation database or retrieval protocol. ### Key directories diff --git a/claude.md b/claude.md index 55b8fe7..d6c911a 100644 --- a/claude.md +++ b/claude.md @@ -11,10 +11,8 @@ ShadowFrog/ skills/ shadow-frog/SKILL.md Main entrypoint (docs, reference system, search) shadow-frog/_coherence.py Shared structural parent-connection validation - shadow-frog/shadow-read.py Optional bounded agent retrieval by file/symbol - shadow-frog/_knowledge.py Shared parsing, retrieval, and user-facing views - shadow-frog/_citations.py Atomic local citation ledger and retrieval cursors - shadow-frog/retrieval.md On-demand helper and telemetry reference + shadow-frog/_citations.py Visible score metadata and safe per-file increments + shadow-frog/shadow-cite.py Record exact consulted claims after native reads shadow-frog-init/ First-time setup (create .shadow/) SKILL.md Init instructions + fallback steps shadow-init.py Python helper script @@ -33,9 +31,9 @@ ShadowFrog/ SKILL.md Bounded ideation, evidence, and task export instructions nap.py Portable record validator, parent context, and exporter shadow-frog-meditate/SKILL.md Dedup, merge, and resolve conflicting discoveries - shadow-frog-viewer/ User-facing knowledge inspection and visualization + shadow-frog-viewer/ Browse and query the shadow knowledge base SKILL.md Query instructions + shell fallbacks - shadow-viewer.py User CLI using the core shared knowledge implementation + shadow-viewer.py Python helper script dream-lineage.py Dream lineage visualization hook-templates/ shadow-frog-hooks.json Copilot CLI hook config (sessionStart, preToolUse) @@ -88,7 +86,7 @@ Canonical formal spec: `/shadow-frog`. The shapes below are the minimum an agent Per-file discovery (anchored by `file::symbol` heading; labels and `Also involves:` are optional): ``` - - _(, source: [, labels: [bug, security]])_ + _(, source: [, labels: [bug, security]], citation_score: 0)_ Also involves: `file::symbol`, `file::symbol` ``` @@ -102,48 +100,37 @@ Cross-cutting (`_cross/.md`, slug = kebab-case from title, e.g. "DB connec **Discovery**: -_(, source: )_ +_(, source: , citation_score: 0)_ ``` Preference (`_prefs.md` — project-wide, no file/symbol anchor): ``` - - _(source: )_ + _(source: , citation_score: 0)_ ``` - Labels (lowercase, comma-separated): `bug`, `performance`, `security`, `feature-gap`, `tech-debt`. Only for actionable discoveries. - `Also involves:` always uses `file::symbol`, never bare file paths. - `Dream report: _dreams//` is optional — only for experiment-derived discoveries. +### Citation Scores + +- Keep one visible nonnegative integer `citation_score` in Markdown metadata, + after optional labels. New entries start at 0; omitted scores also mean 0. +- Agents read files/symbols directly and explicitly cite consulted entries once + per task. Use core `shadow-cite.py` for serialized exact-claim increments; no + database, opaque IDs, or required retrieval service. +- Score updates must not change discovery prose, provenance, references, or counts. + Coordinate ordinary edits with citation writes; only helper calls share its lock. +- Keep scores when rewording/moving the same claim, and use max rather than sum + when combining duplicates or inherited branch state. Scores are approximate, + not a global audited count or a trust/confidence value. + ### Verification - Observe-based: read source at `file::symbol`, trace logic, confirm claim. - Do-based: write and run a short test/script to confirm or refute. - `source: user` and `source: interaction` → always `verified`. -### Citation-Aware Retrieval - -- Direct `.shadow/.md` and symbol navigation is primary for agents. - Optional agent helpers live under `shadow-frog/`; the Viewer serves user - browsing/visualization requests, not a required code-work retrieval gateway. -- Share parsing, identities and counters through core `_knowledge.py` and - `_citations.py`. Hooks use the core reader. Do not duplicate backend logic or - add compatibility re-exports in the user Viewer. -- Keep the discovery grammar unchanged: viewer fingerprints and `citation_score` - are derived/local metadata, not additional Markdown fields. -- Scores start at zero and increment only for content emitted by a retrieval - view; expansion chunks share one revision-bound logical read. They measure - exposure, not verified usefulness. -- Use the common-Git SQLite ledger for instrumented multiprocess/worktree updates; do not - rewrite shadow files on reads. Telemetry failures must warn without hiding - knowledge. Summary/audit/parser-only operations do not increment scores. -- Native reads remain normal and uncounted. Never make citation accounting a - prerequisite for accessing Markdown, or require an extra read merely to count. -- Rank relevance and trust ahead of scores; preserve room for zero-score entries. - Page large results, expand by ID, and never use only the shortlist for dedup. -- Fingerprints bind kind, canonical anchor, whitespace-preserving parsed claim - and refs. Metadata-only edits retain identity; rewritten/merged claims and - renamed anchors may reset scores. Deduplication must retain all labels. - ### Dedup - Before writing, read existing discoveries at the target symbol. - Same claim → update existing. Extends existing → merge. Contradicts → keep both, mark weaker `refuted`. diff --git a/examples/coupon-demo/.shadow/_cross/coupon-case-normalization-mismatch.md b/examples/coupon-demo/.shadow/_cross/coupon-case-normalization-mismatch.md index 4538e31..1b0c04f 100644 --- a/examples/coupon-demo/.shadow/_cross/coupon-case-normalization-mismatch.md +++ b/examples/coupon-demo/.shadow/_cross/coupon-case-normalization-mismatch.md @@ -8,4 +8,4 @@ **Discovery**: validate_coupon normalizes coupon codes to uppercase via code.upper() before lookup, but calculate_total passes coupon_code directly to get_coupon without normalization. A user who validates "save20" (returns True) and then passes "save20" to calculate_total gets no discount — the coupon silently fails because load_coupon's keys are uppercase. This creates a validate-then-use inconsistency where validated codes don't work. -_(verified, source: exploration, labels: [bug])_ +_(verified, source: exploration, labels: [bug], citation_score: 0)_ diff --git a/examples/coupon-demo/.shadow/_cross/global-coupon-cache-side-effects.md b/examples/coupon-demo/.shadow/_cross/global-coupon-cache-side-effects.md index 3ee95a3..a939b78 100644 --- a/examples/coupon-demo/.shadow/_cross/global-coupon-cache-side-effects.md +++ b/examples/coupon-demo/.shadow/_cross/global-coupon-cache-side-effects.md @@ -9,4 +9,4 @@ **Discovery**: COUPON_CACHE is a module-level global dict shared by all importers. Any call to get_coupon (directly or via validate_coupon) permanently populates the cache, including caching None for invalid codes. In tests, cache entries from one test persist into the next — there is no reset mechanism. validate_coupon caches under the uppercased key, while calculate_total would cache under the original-case key, so a single logical coupon code can produce two separate cache entries ("SAVE20" and "save20") with different values. -_(verified, source: exploration, labels: [bug])_ +_(verified, source: exploration, labels: [bug], citation_score: 0)_ diff --git a/examples/coupon-demo/.shadow/_cross/mutation-through-discount-pipeline.md b/examples/coupon-demo/.shadow/_cross/mutation-through-discount-pipeline.md index 3dc557d..428f7b5 100644 --- a/examples/coupon-demo/.shadow/_cross/mutation-through-discount-pipeline.md +++ b/examples/coupon-demo/.shadow/_cross/mutation-through-discount-pipeline.md @@ -8,4 +8,4 @@ **Discovery**: apply_bulk_discount mutates item dicts in-place (modifying "price" keys) and returns the same list object. When piped into calculate_total, the mutation is invisible — calculate_total sees already-reduced prices. But any code holding a reference to the original items list now sees the discounted prices permanently. Calling apply_bulk_discount multiple times compounds discounts (0.90^N multiplier). The test_bulk_then_coupon test avoids this by creating fresh items, but real usage with shared item references would silently corrupt prices. -_(verified, source: exploration, labels: [bug])_ +_(verified, source: exploration, labels: [bug], citation_score: 0)_ diff --git a/examples/coupon-demo/.shadow/cart.py.md b/examples/coupon-demo/.shadow/cart.py.md index 63a62af..2238aec 100644 --- a/examples/coupon-demo/.shadow/cart.py.md +++ b/examples/coupon-demo/.shadow/cart.py.md @@ -5,53 +5,53 @@ ## File-Level - COUPON_CACHE is a module-level mutable global dict shared across all importers — any module that imports from cart.py shares the same cache instance, causing cross-module state pollution. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Also involves: `inventory.py::validate_coupon` ## `COUPON_CACHE` - Never cleared or evicted — grows monotonically for the lifetime of the process. In a long-running server, every unique coupon code ever queried remains cached forever. - _(verified, source: exploration, labels: [performance])_ + _(verified, source: exploration, labels: [performance], citation_score: 0)_ - Caches None for invalid codes. Once an invalid code is looked up, the None result is permanently cached, preventing any future lookup even if the underlying data changes. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ - Case-variant lookups create duplicate cache entries for the same logical coupon. validate_coupon("save20") caches "SAVE20" → valid, then calculate_total("save20") caches "save20" → None. Cache grows 2× faster with mixed-case usage. - _(verified, source: exploration, labels: [bug, performance])_ + _(verified, source: exploration, labels: [bug, performance], citation_score: 0)_ Dream report: `_dreams/20260420-140000Z-cache-poison-sequence/` ## `load_coupon` - Case-sensitive lookup against uppercase keys ("SAVE20", "HALF"). Passing lowercase (e.g., "save20") returns None even though the coupon conceptually exists. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Returns None for unknown codes (via dict.get default), not an exception. Callers must handle None. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ ## `get_coupon` - The `if code not in COUPON_CACHE` guard means each code is loaded exactly once per process. But because None is a valid cached value, invalid codes are also "loaded once" and permanently considered invalid. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `cart.py::COUPON_CACHE`, `cart.py::load_coupon` - Accepts any hashable type as key (True, 42, lists-as-errors). Non-string keys permanently cache None entries that can never resolve to valid coupons — silent cache pollution. - _(verified, source: exploration, labels: [security])_ + _(verified, source: exploration, labels: [security], citation_score: 0)_ Dream report: `_dreams/20260420-142000Z-adversarial-inputs/` Also involves: `cart.py::COUPON_CACHE` ## `calculate_total` - Does NOT normalize coupon_code to uppercase before lookup. Lowercase codes silently produce no discount (coupon returns None from cache or load_coupon). This contradicts validate_coupon which does normalize. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `inventory.py::validate_coupon` - `if coupon_code:` is falsy for empty string "", None, 0, and False — all skip coupon lookup silently. No distinction between "no coupon" and "invalid coupon". - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Accepts negative prices and quantities without validation. Negative subtotals still have 8% tax applied, producing negative totals (e.g., price=-10, qty=1 → total=-10.80). - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ - Tax rate (0.08 = 8%) is hardcoded with no configuration mechanism. Changing tax requires editing source code. - _(verified, source: exploration, labels: [tech-debt])_ + _(verified, source: exploration, labels: [tech-debt], citation_score: 0)_ - Coupon min_total check uses the actual subtotal computed from items at call time. Since apply_bulk_discount mutates prices in-place before calculate_total runs, the min_total check sees post-bulk-discount prices. If bulk discount drops subtotal below min_total, the coupon is correctly rejected. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Dream report: `_dreams/20260420-141000Z-bulk-min-total-interaction/` Also involves: `inventory.py::apply_bulk_discount` - Non-string coupon_code values (True, 42, etc.) pass the `if coupon_code:` truthiness check, reach get_coupon, cache None under the non-string key, and silently produce no discount. No type validation exists. - _(verified, source: exploration, labels: [security])_ + _(verified, source: exploration, labels: [security], citation_score: 0)_ Dream report: `_dreams/20260420-142000Z-adversarial-inputs/` Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE` diff --git a/examples/coupon-demo/.shadow/inventory.py.md b/examples/coupon-demo/.shadow/inventory.py.md index 2dea944..c6afe7a 100644 --- a/examples/coupon-demo/.shadow/inventory.py.md +++ b/examples/coupon-demo/.shadow/inventory.py.md @@ -5,36 +5,36 @@ ## File-Level - Imports get_coupon from cart, creating a dependency on cart's COUPON_CACHE. Any call to validate_coupon pollutes the shared cache as a side effect. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE` ## `validate_coupon` - Normalizes code to uppercase via `code.upper()` before lookup, but calculate_total does NOT — so a code that validates successfully may still produce no discount when passed directly to calculate_total. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `cart.py::calculate_total`, `cart.py::load_coupon` - Side effect: populates COUPON_CACHE with the uppercased code. Calling validate_coupon("save20") caches under key "SAVE20", but a later calculate_total("save20") looks up lowercase "save20" — a cache miss that then caches None under "save20". - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `cart.py::COUPON_CACHE`, `cart.py::get_coupon` - Will crash with AttributeError if code is None (None has no .upper() method). No guard against non-string input. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ - Also crashes on any non-string type: int, list, dict all raise AttributeError on .upper(). Needs `isinstance(code, str)` guard. - _(verified, source: exploration, labels: [security])_ + _(verified, source: exploration, labels: [security], citation_score: 0)_ Dream report: `_dreams/20260420-142000Z-adversarial-inputs/` ## `apply_bulk_discount` - Mutates items in-place — modifies the original dict objects' "price" keys. The caller's list is permanently altered. Returns the same list object (not a copy). - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Crashes with KeyError if items lack 'qty' key, and TypeError if 'qty' is a string. No input validation. - _(verified, source: exploration, labels: [security])_ + _(verified, source: exploration, labels: [security], citation_score: 0)_ Dream report: `_dreams/20260420-142000Z-adversarial-inputs/` - Calling apply_bulk_discount twice on the same items compounds the discount: first call gives 0.90×, second gives 0.81×, third gives 0.729×. No idempotency guard. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ - Only applies discount to items with qty >= 5. Items with qty 4 or below are untouched, even if the total quantity across all items exceeds 5. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Uses round(price * 0.90, 2) which can produce floating-point artifacts on certain prices. For example, 33.33 * 0.90 = 29.997 → rounds to 30.0, not 29.997. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ ## Cross-References diff --git a/examples/coupon-demo/.shadow/test_cart.py.md b/examples/coupon-demo/.shadow/test_cart.py.md index ffcdbe4..1aff2ca 100644 --- a/examples/coupon-demo/.shadow/test_cart.py.md +++ b/examples/coupon-demo/.shadow/test_cart.py.md @@ -5,35 +5,35 @@ ## File-Level - Uses a manual `if __name__ == "__main__"` test runner, not pytest or unittest. Tests are plain functions with assert statements and no setup/teardown. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - No test isolation: COUPON_CACHE is a shared global. test_coupon populates the cache with "SAVE20", and test_bulk_then_coupon populates it with "HALF". If test order changes or tests are rerun in the same process, cached values persist from earlier tests. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `cart.py::COUPON_CACHE` - No negative-path tests: no test for invalid coupon codes, negative prices, empty items, or the case-sensitivity mismatch between validate_coupon and calculate_total. - _(verified, source: exploration, labels: [feature-gap])_ + _(verified, source: exploration, labels: [feature-gap], citation_score: 0)_ ## `test_basic_total` - Verifies: 25.00 × 3 = 75.00 subtotal + 8% tax = 81.00. Tests the simplest happy path with no coupon. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ ## `test_coupon` - Verifies SAVE20 on 75.00 subtotal: 75.00 − 20% = 60.00 + 4.80 tax = 64.80. The 75.00 subtotal exceeds SAVE20's min_total of 50. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Only tests with uppercase coupon code "SAVE20". Does not test lowercase, revealing nothing about the case-normalization bug. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Also involves: `cart.py::calculate_total` ## `test_bulk_then_coupon` - Tests the full pipeline: apply_bulk_discount mutates items (25.00 → 22.50), then calculate_total applies HALF coupon on 112.50 subtotal → 56.25 + 4.50 tax = 60.75. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ - Creates fresh items list, avoiding the mutation-persistence issue. But if this test's items object were reused in a subsequent test, prices would already be 22.50, not 25.00. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Also involves: `inventory.py::apply_bulk_discount`, `cart.py::calculate_total` - Imports apply_bulk_discount inside the function body (lazy import), unlike the module-level import of calculate_total. This is inconsistent but functionally irrelevant. - _(verified, source: exploration, labels: [tech-debt])_ + _(verified, source: exploration, labels: [tech-debt], citation_score: 0)_ ## Cross-References diff --git a/hook-templates/scripts/shadow-frog-pre-tool.sh b/hook-templates/scripts/shadow-frog-pre-tool.sh index 061f18d..52adb33 100755 --- a/hook-templates/scripts/shadow-frog-pre-tool.sh +++ b/hook-templates/scripts/shadow-frog-pre-tool.sh @@ -6,7 +6,7 @@ # When a mutation tool (edit/create/str_replace/write) targets a file with # a shadow that has actionable discoveries (bug/security labels), the # top entries are inlined into additionalContext via -# shadow-frog/shadow-read.py --top. Per-session dedup ensures the same file's +# shadow-viewer.py --top. Per-session dedup ensures the same file's # content is injected at most once per Copilot CLI process. # This hook is ADVISORY — it only injects shadow context, it is NOT a security @@ -19,7 +19,7 @@ # 3. trap on TERM/HUP/INT — converts runner-initiated signal kills to 0. # (bash 3.2+ on macOS and bash 5+ on Linux verified: EXIT alone is NOT # enough — SIGTERM still produces exit 143/-15 without a TERM trap.) -# 4. Every external call (git, python3, shadow-read.py) MUST be wrapped in +# 4. Every external call (git, python3, shadow-viewer.py) MUST be wrapped in # a bounded subprocess timeout. If the foreground child hangs, bash will # queue the signal until the child returns, so the trap can't save us # unless boundedness holds. CI enforces this via .github/workflows/shellcheck.yml. @@ -133,13 +133,13 @@ if [ "$IS_MUTATION" = "1" ]; then if [ -n "$0" ]; then SCRIPT_DIR="$(cd "$(dirname "$0")" 2>/dev/null && pwd -P)" || SCRIPT_DIR="" fi - # Resolve reader + run it inside ONE Python block. Every - # external call (git rev-parse, reader subprocess) is + # Resolve viewer + run it inside ONE Python block. Every + # external call (git rev-parse, viewer subprocess) is # bounded with subprocess.run(timeout=...) so a hung git - # or hung reader can't blow the hook's 5s budget — even + # or hung viewer can't blow the hook's 5s budget — even # the trap pyramid can't help if bash is blocked waiting # on an unbounded foreground child (signals are queued). - # Total bounded work here is ~1.5s (rev-parse 0.5s + reader 1.0s). + # Total bounded work here is ~1.5s (rev-parse 0.5s + viewer 1.0s). TOP_OUTPUT=$(SF_SCRIPT_DIR="$SCRIPT_DIR" SF_REL_PATH="$REL_PATH" python3 - <<'PYEOF' 2>/dev/null || true import os, subprocess, sys @@ -155,7 +155,7 @@ def _git(args, timeout): pass return '' -# Locate the core shadow-read.py helper. Order: +# Locate shadow-viewer.py. Order: # 1. Script-relative — works for source repo dev AND project installs # (hooks at .github/hooks/scripts/ co-located with .github/skills/). # 2-3. Parent repo's .github/ or .claude/ skills — useful when the hook @@ -164,21 +164,21 @@ repo_root = _git(['rev-parse', '--show-toplevel'], 0.5) candidates = [] if script_dir: - candidates.append(os.path.join(script_dir, '..', '..', 'skills', 'shadow-frog', 'shadow-read.py')) + candidates.append(os.path.join(script_dir, '..', '..', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) if repo_root: - candidates.append(os.path.join(repo_root, '.github', 'skills', 'shadow-frog', 'shadow-read.py')) - candidates.append(os.path.join(repo_root, '.claude', 'skills', 'shadow-frog', 'shadow-read.py')) + candidates.append(os.path.join(repo_root, '.github', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) + candidates.append(os.path.join(repo_root, '.claude', 'skills', 'shadow-frog-viewer', 'shadow-viewer.py')) -reader = '' +viewer = '' for candidate in candidates: if os.path.isfile(candidate): - reader = candidate + viewer = candidate break -if reader: +if viewer: try: r = subprocess.run( - ['python3', reader, + ['python3', viewer, '--shadow-dir', '.shadow', '--top', rel_path, '--top-labels', 'bug,security', @@ -188,13 +188,11 @@ if reader: ) if r.returncode == 0: sys.stdout.write(r.stdout.strip()) - if r.stderr.strip(): - sys.stdout.write("\n[ShadowFrog] Reader reported a warning; rerun it directly for details. Citation updates may be unavailable.") except Exception: pass PYEOF ) - # Only inline when the reader produced an actionable + # Only inline when the viewer produced an actionable # response (non-empty and not the "no discoveries" sentinel). if [ -n "$TOP_OUTPUT" ] && [[ "$TOP_OUTPUT" != "No actionable"* ]]; then MSG="[ShadowFrog] Actionable discoveries for ${REL_PATH} (verify against source before acting): @@ -210,7 +208,7 @@ fi # Staleness warning (appended when shadow is behind HEAD). # All git work is bounded with per-call subprocess timeouts so a huge/locked # repo can't blow past the hook's 5s budget. Timeouts sum to 2.0s here, -# matched with the reader's ~1.5s above + bash overhead = ~4s worst case, +# matched with the viewer's ~1.5s above + bash overhead = ~4s worst case, # leaving 1s headroom under timeoutSec=5. Any failure/timeout -> no warning. CHANGED=$(python3 - <<'PYEOF' 2>/dev/null || echo "" import json, subprocess diff --git a/skills/shadow-frog-dream/SKILL.md b/skills/shadow-frog-dream/SKILL.md index c643813..ce020a4 100644 --- a/skills/shadow-frog-dream/SKILL.md +++ b/skills/shadow-frog-dream/SKILL.md @@ -612,6 +612,15 @@ Follow the dedup and writing rules in `/shadow-frog`. Dream discoveries are typically `source: exploration`. Mark `verified` when confirmed by running code; `uncertain` if not fully testable. +New discoveries start with visible `citation_score: 0`. Keep that field in +the corresponding manifest entry as well. Existing knowledge that informed +the task can be cited once through the core increment helper or reported to +the coordinator for serialized updates. Do not count merely enumerated entries. +Scores are approximate within the relevant checkout; reconciliation preserves +the larger score when the same claim is supplied again, rather than summing +inherited counts. Counter-only branch edits are not imported unless represented +in a matching manifest discovery; they do not justify unsupported `op` values. + #### How to Append Find the `##`/`###` heading for the symbol, then: @@ -642,7 +651,7 @@ Example: ``` - /api/upload accepts paths from request body without normalization, allowing `../` traversal into /etc/. - _(verified, source: exploration, labels: [bug, security])_ + _(verified, source: exploration, labels: [bug, security], citation_score: 0)_ Dream report: `_dreams/20260518-161200Z-upload-traversal/` ``` @@ -682,7 +691,7 @@ Per-file discoveries should reference the dream report: ``` - Retrying with exponential backoff recovers from 99% of transient errors, but must exclude 4xx or it retries bad requests for 30s. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ Dream report: `_dreams/20250612-143012Z-retry-logic/` ``` @@ -707,6 +716,7 @@ After shadow writes, create `.shadow/_dreams/$DREAM_ID/manifest.json`: "status": "verified", "source": "exploration", "labels": ["bug"], + "citation_score": 0, "also_involves": ["src/parsers/utils.py::unescape"], "dream_report": "_dreams//" } @@ -731,6 +741,9 @@ reject any discovery whose `op` is not `add`. To revise or contradict an existing discovery, run a meditate session against main's `.shadow/` instead of trying to do it from a dream branch. +`citation_score` on per-file or cross-cutting manifest entries is a nonnegative +integer, defaulting to 0. It is a reuse hint, never a verification or trust signal. + **Hard gate — discoveries must be mirrored into per-file shadows.** The reconciler merges `manifest.json` entries into main directly (so discoveries are not lost at merge time), but the branch's per-file shadows must ALSO be @@ -904,7 +917,7 @@ rich area and the second half keeps digging there instead of spreading. 2. Each agent gets its own branch (inherently isolated) 3. Each agent writes its own manifest in its `$DREAM_ID/` directory 4. Do NOT write to main or shared files (`_index.md`, `state.json`) -5. Do NOT update metadata — reconciled post-dream by orchestrator +5. Do NOT update shared indexes or `state.json` — reconciled post-dream by orchestrator 6. Broad mode: fetch once and use the initial snapshot. Coherent mode: after a parent is pushed, the orchestrator refreshes that parent's ref/commit and branch map before launching its children. Siblings share the refreshed diff --git a/skills/shadow-frog-dream/dream-reconcile.py b/skills/shadow-frog-dream/dream-reconcile.py index 62253d3..869cefb 100755 --- a/skills/shadow-frog-dream/dream-reconcile.py +++ b/skills/shadow-frog-dream/dream-reconcile.py @@ -43,6 +43,17 @@ from datetime import datetime, timezone from pathlib import Path, PureWindowsPath +_bytecode = sys.dont_write_bytecode +sys.dont_write_bytecode = True +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "shadow-frog")) +try: + from _citations import CitationError, metadata_score, set_metadata_score, validate_score +except ImportError as exc: + raise SystemExit("ERROR: Missing core citation metadata parser; reinstall the full skill set") from exc +finally: + sys.path.pop(0) + sys.dont_write_bytecode = _bytecode + # Shared safety gate for `rm -rf `. Lives next to this script so # bash callers (dream-cleanup.sh, dream-gc.sh) and this module share ONE # source of truth for the "is this path safe to remove?" rules. Imported @@ -262,12 +273,14 @@ def _validate_manifest_paths(repo_root, dream_id, manifest): for filename in ('report.md', 'manifest.json', 'patch.diff'): _shadow_output_path(repo_root, f'_dreams/{dream_id}/{filename}', f"{label} {filename}") for index, disc in _manifest_entries(manifest, 'discoveries', dream_id): + validate_score(disc.get('citation_score', 0), f"{label} discoveries[{index}].citation_score") field = f"{label} discoveries[{index}].anchor" file_part = _anchor_file_part(disc.get('anchor', ''), field) if file_part is not None: _shadow_file_path(repo_root, file_part, field) for index, cross in _manifest_entries(manifest, 'cross_cutting', dream_id): field = f"{label} cross_cutting[{index}]" + validate_score(cross.get('citation_score', 0), f"{field}.citation_score") slug = cross.get('slug', '') if not slug: continue @@ -565,10 +578,12 @@ def _merge_meta(existing, new_status, new_source, new_labels): return status, source, labels, changed -def _format_meta_line(status, source, labels): +def _format_meta_line(status, source, labels, citation_score=0): + validate_score(citation_score) parts = [status, f'source: {source}'] if labels: parts.append(f"labels: [{', '.join(labels)}]") + parts.append(f"citation_score: {citation_score}") return f' _({", ".join(parts)})_\n' @@ -582,13 +597,11 @@ def merge_discovery_into_file(shadow_path, anchor_symbol, discovery, dream_id, * status = discovery.get('status', 'verified') source = discovery.get('source', 'exploration') labels = discovery.get('labels', []) + citation_score = validate_score(discovery.get('citation_score', 0)) also_involves = discovery.get('also_involves', []) # Build the discovery line - meta_parts = [status, f'source: {source}'] - if labels: - meta_parts.append(f"labels: [{', '.join(labels)}]") - meta_line = f' _({", ".join(meta_parts)})_' + meta_line = _format_meta_line(status, source, labels, citation_score).rstrip("\n") lines_to_add = [f'- {text}\n', f'{meta_line}\n'] if also_involves: @@ -639,9 +652,12 @@ def merge_discovery_into_file(shadow_path, anchor_symbol, discovery, dream_id, * return False merged = _merge_meta(existing_meta, status, source, labels) m_status, m_source, m_labels, changed = merged + previous_score = metadata_score(lines[meta_idx]) + merged_score = max(previous_score, citation_score) + changed = changed or merged_score != previous_score if not changed: return False - lines[meta_idx] = _format_meta_line(m_status, m_source, m_labels) + lines[meta_idx] = _format_meta_line(m_status, m_source, m_labels, merged_score) with open(shadow_path, 'w', encoding="utf-8") as f: f.writelines(lines) return True @@ -752,7 +768,7 @@ def add_cross_reference_backpointer(repo_root, file_part, slug, title, dream_id) return True -def _merge_refs_into_cross_file(cross_path, new_refs, *, repo_root): +def _merge_refs_into_cross_file(cross_path, new_refs, *, repo_root, citation_score=0): """Union new refs into an existing _cross/.md **Refs**: section. When two dreams use the same cross-cutting slug, the later one must not @@ -762,6 +778,7 @@ def _merge_refs_into_cross_file(cross_path, new_refs, *, repo_root): """ cross_path = _checked_shadow_destination(repo_root, cross_path, "cross-cutting destination") _validate_refs(repo_root, new_refs, "cross-cutting refs") + validate_score(citation_score) try: with open(cross_path, encoding="utf-8") as f: content = f.read() @@ -785,7 +802,15 @@ def _merge_refs_into_cross_file(cross_path, new_refs, *, repo_root): else: break to_add = [r for r in new_refs if r and r not in existing] - if not to_add: + score_changed = False + for index, line in enumerate(lines): + if line.strip().startswith("_(") and "source:" in line: + previous_score = metadata_score(line) + if citation_score > previous_score: + lines[index] = set_metadata_score(line, citation_score) + score_changed = True + break + if not to_add and not score_changed: return False lines[block_end:block_end] = [f'- `{r}`' for r in to_add] with open(cross_path, 'w', encoding="utf-8") as f: @@ -855,8 +880,10 @@ def merge_discoveries(repo_root, manifests, dry_run=False): f"**Category**: {cross.get('category', 'behavior')}\n" f"**Refs**:\n{refs_str}\n\n" f"**Discovery**: {cross.get('text', '')}\n\n" - f"_({cross.get('status', 'verified')}, " - f"source: {cross.get('source', 'exploration')})_\n" + + _format_meta_line( + cross.get('status', 'verified'), cross.get('source', 'exploration'), + cross.get('labels', []), cross.get('citation_score', 0), + ).lstrip() ) with open(cross_path, 'w', encoding="utf-8") as f: f.write(content) @@ -865,7 +892,10 @@ def merge_discoveries(repo_root, manifests, dry_run=False): # Cross file already exists (e.g. a prior dream used the same # slug). Union our refs into its **Refs**: block so it stays # consistent with the back-pointers added below. - if _merge_refs_into_cross_file(cross_path, refs, repo_root=repo_root): + if _merge_refs_into_cross_file( + cross_path, refs, repo_root=repo_root, + citation_score=cross.get('citation_score', 0), + ): merged_count += 1 else: skipped_count += 1 @@ -2080,6 +2110,6 @@ def main(): if __name__ == '__main__': try: main() - except (CoherentLineageError, UnsafeShadowPath) as exc: + except (CoherentLineageError, UnsafeShadowPath, CitationError) as exc: print(f"ERROR: {exc}", file=sys.stderr) sys.exit(1) diff --git a/skills/shadow-frog-dream/dream-tools.py b/skills/shadow-frog-dream/dream-tools.py index 9abaa82..8e1da95 100644 --- a/skills/shadow-frog-dream/dream-tools.py +++ b/skills/shadow-frog-dream/dream-tools.py @@ -78,13 +78,13 @@ def _source_files() -> dict[str, Path]: sources = {} for directory in (DREAM_DIR, CORE_DIR): for path in directory.iterdir(): - if path.suffix not in (".py", ".sh", ".md"): + if path.name != "SKILL.md" and path.suffix not in (".py", ".sh"): continue if path.is_symlink() or not path.is_file(): raise ValueError(f"Tooling assets must be regular files: {path}") sources[f"{directory.name}/{path.name}"] = path required = { - "shadow-frog/SKILL.md", "shadow-frog/retrieval.md", "shadow-frog/_coherence.py", + "shadow-frog/SKILL.md", "shadow-frog/_coherence.py", "shadow-frog/_citations.py", "shadow-frog-dream/SKILL.md", "shadow-frog-dream/dream-tools.py", "shadow-frog-dream/_worktree_safety.py", *(f"shadow-frog-dream/{name}" for name in TOOLS.values()), diff --git a/skills/shadow-frog-dream/dream-validate.py b/skills/shadow-frog-dream/dream-validate.py index 1b71823..09520b3 100644 --- a/skills/shadow-frog-dream/dream-validate.py +++ b/skills/shadow-frog-dream/dream-validate.py @@ -38,6 +38,7 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "shadow-frog")) try: from _coherence import MODES, validate_connection + from _citations import CitationError, validate_score except ImportError as exc: raise SystemExit("ERROR: Missing shared shadow-frog/_coherence.py; reinstall the full skill set") from exc finally: @@ -219,6 +220,10 @@ def main(): f'{type(disc).__name__}.' ) continue + try: + validate_score(disc.get('citation_score', 0), f"discoveries[{i}].citation_score") + except CitationError as exc: + errors.append(str(exc)) op = (disc.get('op') or 'add').lower() if op != 'add': errors.append( @@ -227,6 +232,13 @@ def main(): f'"add"), or split this into a meditate session.' ) + for i, cross in enumerate(manifest.get('cross_cutting', []) or []): + if isinstance(cross, dict): + try: + validate_score(cross.get('citation_score', 0), f"cross_cutting[{i}].citation_score") + except CitationError as exc: + errors.append(str(exc)) + # 10. Discoveries must be mirrored into per-file shadows on the dream # branch. The reconciler reads manifest entries directly when merging # into main, so the discoveries themselves are NOT lost — but the diff --git a/skills/shadow-frog-init/SKILL.md b/skills/shadow-frog-init/SKILL.md index df52a93..e8a5561 100644 --- a/skills/shadow-frog-init/SKILL.md +++ b/skills/shadow-frog-init/SKILL.md @@ -147,6 +147,8 @@ _No preferences recorded yet._ This file stores project-wide user preferences and conventions that are not tied to any specific file or symbol. It is populated by `/shadow-frog-update` when the user shares general directives. +New preference/discovery entries use `citation_score: 0`; empty placeholders +are not knowledge entries and do not have scores. ### 6. Create `_meta/state.json` diff --git a/skills/shadow-frog-meditate/SKILL.md b/skills/shadow-frog-meditate/SKILL.md index 9a65ceb..6ecf24d 100644 --- a/skills/shadow-frog-meditate/SKILL.md +++ b/skills/shadow-frog-meditate/SKILL.md @@ -55,7 +55,7 @@ orchestrator can auto-apply resolutions. This is critical for automation — prose recommendations require manual interpretation. ``` -{"action": "merge", "file": "src/auth.py.md", "symbol": "authenticate_user", "keep": "- silently returns None on expired tokens...", "remove": "- returns None when token expires...", "merged": "- authenticate_user() silently returns None on expired tokens instead of raising. 3 of 7 callers don't check.\n _(verified, source: exploration)_", "reason": "duplicate: same claim, different wording"} +{"action": "merge", "file": "src/auth.py.md", "symbol": "authenticate_user", "keep": "- silently returns None on expired tokens...", "remove": "- returns None when token expires...", "merged": "- authenticate_user() silently returns None on expired tokens instead of raising. 3 of 7 callers don't check.\n _(verified, source: exploration, citation_score: 0)_", "reason": "duplicate: same claim, different wording"} {"action": "merge", "file": "src/db.py.md", "symbol": "connect", "keep": "- connection pool exhaustion...", "remove": "- pool runs out...", "merged": "...", "reason": "near-duplicate: first extends second"} {"action": "conflict", "file": "src/auth.py.md", "symbol": "validate_token", "entry_a": "- raises ValueError...", "entry_b": "- returns False...", "resolution": "verified_a", "reason": "code inspection: line 42 raises ValueError"} {"action": "conflict", "file": "src/cache.py.md", "symbol": "invalidate", "entry_a": "...", "entry_b": "...", "resolution": "escalate", "reason": "both claims have evidence, needs user input"} @@ -97,6 +97,7 @@ Combine into a single discovery: - Keep the **stronger** trust: `source: user` > `source: interaction` > `source: exploration` - Keep the **stronger** status: `verified` > `uncertain` > `refuted` - Merge `Also involves:` refs (union of both) +- Preserve the larger `citation_score`; do not sum counts that may share history. - Delete the weaker entry ### Near-Duplicates → Absorb @@ -105,6 +106,7 @@ The broader discovery absorbs the narrower one: - Expand the broader entry to include any extra detail from the narrower - Delete the narrower entry - Preserve the stronger trust/status between the two +- Preserve the larger citation score for the merged knowledge. ### Conflicts → Investigate @@ -279,6 +281,8 @@ malformed discovery is worse than a duplicate; it breaks the viewer parser and downstream agents. Meditate-specific rules: +- Keep visible citation scores when rewording or moving the same knowledge; + a new behavioral claim starts at 0. Counts never override trust or correctness. - When merging, take the **union** of `labels: [...]` from both entries. - When merging, preserve every `Also involves: file::symbol` from both entries (union, not intersection). diff --git a/skills/shadow-frog-update/SKILL.md b/skills/shadow-frog-update/SKILL.md index 05190e0..941e954 100644 --- a/skills/shadow-frog-update/SKILL.md +++ b/skills/shadow-frog-update/SKILL.md @@ -72,13 +72,13 @@ and conventions. Representative examples: Write as: ```markdown - - _(verified, source: user)_ + _(verified, source: user, citation_score: 0)_ ``` For knowledge emerging from collaborative work (debugging, refactoring, test failures): ```markdown - - _(verified, source: interaction)_ + _(verified, source: interaction, citation_score: 0)_ ``` Anchor to the specific `file::symbol`. `source: user` and `source: interaction` @@ -187,6 +187,10 @@ See `/shadow-frog` § Discovery Format for the verbatim per-file, cross-cutting, and preference formats. Rules to keep in mind during update sessions: +- Start new claims/preferences with `citation_score: 0`; preserve the score + when updating the same knowledge. Citation-only edits do not add discoveries. +- Record deliberately consulted existing entries once per task using the core + `shadow-cite.py` helper, not a retrieval requirement or a second hidden store. - Be behavioral: "silently returns None on expired tokens" not "handles token expiration" - `source: user` and `source: interaction` → always `verified`, use user's own words - `source: exploration` → mark `uncertain` unless verified by code reading or tests diff --git a/skills/shadow-frog-viewer/SKILL.md b/skills/shadow-frog-viewer/SKILL.md index 392fb59..b5f9d85 100644 --- a/skills/shadow-frog-viewer/SKILL.md +++ b/skills/shadow-frog-viewer/SKILL.md @@ -1,12 +1,10 @@ --- name: shadow-frog-viewer description: >- - Help users browse and visualize their collected shadow knowledge in the - terminal or as an interactive dream-lineage report. Show an overview, - search results, preferences, recent discoveries, and structural audits. - Invoke when the user asks to inspect the shadow. For agents' own code work, - direct file/symbol navigation is primary; optional retrieval helpers live - in the core shadow-frog skill. + Help users browse and visualize the shadow knowledge base: overview, search for files + or symbols or text, view preferences, or see recent discoveries. + Invoke when the user wants to see what's in the shadow, get an + overview, or find specific knowledge. scripts: - shadow-viewer.py - dream-lineage.py @@ -14,15 +12,13 @@ scripts: # ShadowFrog Viewer -**User-facing inspection and visualization** of `.shadow/`. Prerequisite: -`.shadow/` exists. An agent can run these views on the user's behalf, but this -skill is not the agent's required knowledge interface. Agent work starts with -the mirrored file/symbol locations; the core `shadow-read.py` is optional for -large sections or targeted searches. +User-facing inspection of `.shadow/` content. Prerequisite: `.shadow/` exists. +Agents navigate the Markdown files/symbols directly; this viewer is not a +required retrieval interface. ## Primary: Python Helper Script -The companion script `shadow-viewer.py` supports Python 3.9+ and lives beside +The companion script `shadow-viewer.py` lives in the same directory as this SKILL.md file. To find and run it: ```bash @@ -37,32 +33,30 @@ python3 .claude/skills/shadow-frog-viewer/shadow-viewer.py [options] | Command | What it shows | |---------|--------------| | `--summary` | Overview: counts, source/status/label breakdown, per-file table, cross-cutting titles (default) | -| `--search QUERY` | Bounded search across paths, symbols, text, cross-cutting entries, and preferences | -| `--file FILE` | Browse one known source file's shadow, including file-level and cross-cutting knowledge | -| `--symbol FILE::SYMBOL` | Bounded discoveries at an exact symbol, plus matching cross-cutting refs; use `File-Level` for a file-level section | -| `--get ID` | Expand one current discovery; long expansions return a revision-bound continuation | -| `--prefs` | Project-wide preferences; follow all pages before treating them as complete | -| `--recent [N]` | Most recent discovery previews by file mtime (default page size: 10) | -| `--labels LABEL` | Bounded discoveries matching labels (e.g., `bug`, `security`, `bug,performance`) | -| `--top FILE` | Compact actionable previews (default: up to 3 entries, 600 characters). Not an exhaustive file view. Agent hooks use the separate core reader. | +| `--search QUERY` | Universal search — matches file names, symbol names, and discovery text. Includes cross-cutting and preferences | +| `--prefs` | Project-wide preferences | +| `--recent [N]` | N most recent discoveries with full content (default: 10) | +| `--labels LABEL` | Discoveries filtered by label (e.g., `bug`, `security`, `bug,performance`) | +| `--top FILE` | Top actionable discoveries for FILE — concise output (default: 3 entries, ~600 chars) suitable for the preToolUse hook. Includes both per-file shadow entries and any `_cross/` discoveries that reference FILE. Verified discoveries rank first. | | `--check-invariants` | Audit structural integrity — bidirectional cross-references, label/source/category enum compliance, heading format, no-orphan-back-pointer. Exits 0 if clean, 1 with one violation per line. Run after dream reconciliation or before commit. | No arguments defaults to `--summary`. +Discovery views display the visible Markdown `citation_score` (missing means 0). +Search/label/top ordering uses it only after source trust and verification status; +refuted claims remain last. Reading, searching, and automatic previews do not +increment counts. The agent explicitly records deliberately consulted entries +with the core `shadow-cite.py` helper once per task. `--recent` is based on shadow +file modification time, which includes citation updates, not discovery creation time. + ### Options | Flag | Effect | |------|--------| | `--shadow-dir DIR` | Override .shadow/ location (default: auto-detect from CWD) | -| `--limit N` | Positive page size for file, search, symbol, labels, or preferences (default: 10) | -| `--max-chars N` | Hard output cap, including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output). Use `--top-max-chars` with `--top`. | -| `--cursor TOKEN` | Continue the same view and filters using the returned ordering snapshot | -| `--text-cursor TOKEN` | Continue the same `--get` body and logical read; copy the returned token rather than fabricating an offset | -| `--event-id ID` | Optional retry ID: each discovery counts at most once per ID within 24 hours (1-128 letters/digits or `. _ : -`) | -| `--no-record` | Do not increase citation scores; existing scores still rank results, and pagination may store a local snapshot | | `--top-labels LABELS` | Comma-separated label filter for `--top` (default: `bug,security`). Empty string disables label filtering. | | `--top-limit N` | Max discoveries to show in `--top` (default: 3) | -| `--top-max-chars N` | Hard cap on `--top` total output length (default: 600; minimum 256). Use 0 for no cap. | +| `--top-max-chars N` | Hard cap on `--top` total output length (default: 600). Use 0 for no cap. | ### Examples @@ -70,13 +64,6 @@ No arguments defaults to `--summary`. # Search file names, symbols, discoveries, cross-cutting entries, and preferences python3 shadow-viewer.py --search "token expiry" -# Inspect one symbol without loading its entire shadow -python3 shadow-viewer.py --symbol src/auth.py::UserAuth.validate --limit 5 - -# Expand a returned id, or continue the same search with its returned cursor -python3 shadow-viewer.py --get DISCOVERY_ID -python3 shadow-viewer.py --search "token expiry" --cursor CURSOR_TOKEN - # Security and performance issues python3 shadow-viewer.py --labels security,performance @@ -84,20 +71,6 @@ python3 shadow-viewer.py --labels security,performance python3 shadow-viewer.py --top src/auth.py --top-labels bug,security,performance --top-limit 5 ``` -### Scores and Continuation - -Replace `DISCOVERY_ID` and `CURSOR_TOKEN` with exact returned values. Content -views display the same IDs and local `citation_score` as the optional core -reader, recording only the entries they show. Scores measure helper exposure, -not correctness or proven use; direct file reads remain normal and uncounted. -Knowledge is still plain Markdown at its file/symbol location, not in the ledger. - -Use returned cursors to continue, `--get` to inspect a claim, and `--no-record` -when browsing should not affect scores. Forward any stderr diagnostics to the -user; optional telemetry failures must not hide knowledge. See the shared -[retrieval reference](../shadow-frog/retrieval.md) for exact identity, budget, -retry, and local storage contracts. - ## Dream Lineage Visualization The companion script `dream-lineage.py` generates an interactive HTML @@ -160,7 +133,7 @@ find .shadow -name '*.md' -not -path '*/_meta/*' -printf '%T@ %p\n' | sort -rn | ## Responding to the User -- Preserve the meaning of returned trust/status and citation information; - scores are not confidence estimates. Expand or page results on user request. +- Preserve `--top` output as-is; it is intentionally compact and pre-formatted + for the preToolUse hook. - If the shadow is empty or has no discoveries, suggest running `/shadow-frog-dream` to populate it diff --git a/skills/shadow-frog-viewer/shadow-viewer.py b/skills/shadow-frog-viewer/shadow-viewer.py index 7c828ed..e379cdf 100755 --- a/skills/shadow-frog-viewer/shadow-viewer.py +++ b/skills/shadow-frog-viewer/shadow-viewer.py @@ -1,13 +1,34 @@ #!/usr/bin/env python3 -"""Browse and visualize collected shadow knowledge for users in the terminal. +"""shadow-viewer: Query and browse a .shadow/ knowledge base. -Run without arguments for an overview, or use --search, --prefs, --recent, ---labels and --check-invariants. The separate dream-lineage.py renders HTML. -Agent workflows use direct file/symbol navigation, with optional bounded -retrieval through the core shadow-frog/shadow-read.py helper. +Usage: + shadow-viewer.py [options] + +Views: + --summary Overview + detailed statistics (default) + --search QUERY Universal search across files, symbols, and text + --prefs Show project-wide preferences + --recent [N] N most recent discoveries with content (default: 10) + --labels LABEL Show discoveries by label (bug, security, etc.) + --top FILE Top actionable discoveries for FILE (hook-sized) + --check-invariants Report structural violations (exit 1 if any) + +Options: + --shadow-dir DIR Path to .shadow/ directory (default: auto-detect) + +Exit codes: + 0 Success (possibly with warnings on stderr) + 1 Fatal error (shadow dir not found, no results possible) """ +import argparse +import json +import os +import re import sys +import traceback +from collections import defaultdict +from datetime import datetime from pathlib import Path @@ -15,17 +36,1414 @@ sys.dont_write_bytecode = True sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "shadow-frog")) try: - import _knowledge + from _citations import DISCOVERY_META_RE as _DISCOVERY_META_RE, PREFERENCE_META_RE, CitationError, metadata_score except ImportError as exc: - raise SystemExit("ERROR: Missing core knowledge helper; reinstall the full ShadowFrog skill set") from exc + raise SystemExit("ERROR: Missing core citation metadata parser; reinstall the full skill set") from exc finally: sys.path.pop(0) sys.dont_write_bytecode = _bytecode +def warn(msg): + """Print a warning to stderr. Agents read these to adjust strategy.""" + print(f"[shadow-viewer warning] {msg}", file=sys.stderr) + + +def error(msg): + """Print an error to stderr.""" + print(f"[shadow-viewer error] {msg}", file=sys.stderr) + + +def find_shadow_dir(start="."): + """Walk up from start to find .shadow/ directory.""" + try: + p = Path(start).resolve() + while p != p.parent: + candidate = p / ".shadow" + if candidate.is_dir(): + return candidate + p = p.parent + except (OSError, PermissionError) as e: + error(f"Failed to search for .shadow/ directory from '{start}': {e}") + return None + + +def parse_discovery(line, continuation_lines=None): + """Parse a discovery bullet and its metadata line(s).""" + try: + text = line[2:].strip() if line.startswith("- ") else line.strip() + except (TypeError, AttributeError) as e: + warn(f"parse_discovery: bad line input ({type(line).__name__}): {e}") + return {"text": str(line) if line else ""} + + meta = {"citation_score": 0} + full_text = text + + if continuation_lines: + for cl in continuation_lines: + try: + stripped = cl.strip() + # _(status, source: type, labels: [l1, l2])_ or _(status, source: type)_ + m = _DISCOVERY_META_RE.match(stripped) + if m: + meta["status"] = m.group(1) + meta["source"] = m.group(2) + meta["citation_score"] = int(m.group(4) or 0) + if m.group(3): + meta["labels"] = [ + l.strip() + for l in m.group(3).split(",") + if l.strip() + ] + else: + # _(source: type)_ (preferences format) + m2 = PREFERENCE_META_RE.fullmatch(stripped) + if m2: + meta["source"] = m2.group(1) + meta["citation_score"] = int(m2.group(2) or 0) + elif stripped.startswith("_(") and "citation_score:" in stripped: + warn("Invalid citation_score metadata; repair it before recording citations") + elif stripped.startswith("Also involves:"): + refs = re.findall(r"`([^`]+)`", stripped) + meta["also_involves"] = refs + elif stripped.startswith("Dream report:"): + m_dr = re.search(r"`([^`]+)`", stripped) + if m_dr: + meta["dream_report"] = m_dr.group(1) + else: + full_text += " " + stripped + except Exception as e: + warn(f"parse_discovery: failed parsing continuation line " + f"'{cl[:80]}': {e}") + + return {"text": full_text, **meta} + + +def parse_shadow_file(filepath): + """Parse a per-file shadow into structured data. + + Returns a result dict even on partial failure — whatever was parsed + before the error is preserved. Warnings go to stderr. + """ + result = { + "path": str(filepath), + "source_file": None, + "language": None, + "lines": None, + "last_modified": None, + "symbols": [], + "discoveries": [], + "cross_references": [], + "parse_errors": [], + } + + try: + content = filepath.read_text(encoding="utf-8") + except UnicodeDecodeError as e: + msg = (f"Cannot read {filepath}: encoding error at byte " + f"{e.start}: {e.reason}. File may not be UTF-8.") + warn(msg) + result["parse_errors"].append(msg) + return result + except OSError as e: + msg = f"Cannot read {filepath}: {e}" + warn(msg) + result["parse_errors"].append(msg) + return result + + lines = content.split("\n") + current_symbol = None + i = 0 + + while i < len(lines): + line = lines[i] + + try: + # Header: # Shadow: src/auth.py + if line.startswith("# Shadow: "): + result["source_file"] = line[len("# Shadow: "):].strip() + + # Metadata: **Language**: Python | **Lines**: 142 | ... + elif line.startswith("**Language**"): + parts = line.split("|") + for part in parts: + part = part.strip() + if part.startswith("**Language**"): + m = re.search(r"\*\*:\s*(.+)", part) + if m: + result["language"] = m.group(1).strip() + elif "Lines" in part: + m = re.search(r"(\d+)", part) + if m: + try: + result["lines"] = int(m.group(1)) + except ValueError: + pass + elif "Last modified" in part: + m = re.search(r"\*\*:\s*(.+)", part) + if m: + result["last_modified"] = m.group(1).strip() + + # Symbol heading: ## `symbol_name` or ### `Class.method` + elif re.match(r"^#{2,3}\s", line): + sym_match = re.match(r"^(#{2,3})\s+`(.+?)`", line) + if sym_match: + name = sym_match.group(2) + current_symbol = name + result["symbols"].append(name) + elif "Cross-References" in line: + current_symbol = "__cross_refs__" + elif "File-Level" in line: + current_symbol = "__file_level__" + else: + current_symbol = None + + # Discovery bullet (skip cross-reference links) + elif ( + line.strip().startswith("- ") + and current_symbol + and current_symbol != "__cross_refs__" + ): + # Collect continuation lines + continuation = [] + j = i + 1 + while j < len(lines): + next_line = lines[j] + if ( + next_line.strip() == "" + or next_line.strip().startswith("- ") + or re.match(r"^#{1,3}\s", next_line) + ): + break + continuation.append(next_line) + j += 1 + + disc = parse_discovery(line.strip(), continuation) + disc["symbol"] = ( + "file-level" if current_symbol == "__file_level__" + else current_symbol + ) + disc["file"] = result["source_file"] + result["discoveries"].append(disc) + i = j + continue + + # Cross-reference link + elif ( + current_symbol == "__cross_refs__" + and line.strip().startswith("- ") + ): + link_match = re.search(r"\[(.+?)\]", line) + if link_match: + result["cross_references"].append(link_match.group(1)) + + except Exception as e: + msg = (f"Error parsing {filepath} at line {i + 1}: " + f"{type(e).__name__}: {e}") + warn(msg) + result["parse_errors"].append(msg) + + i += 1 + + return result + + +def parse_prefs(shadow_dir): + """Parse _prefs.md into a list of preferences.""" + prefs_path = shadow_dir / "_prefs.md" + if not prefs_path.exists(): + return [] + + try: + content = prefs_path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + warn(f"Cannot read preferences file {prefs_path}: {e}") + return [] + + prefs = [] + lines = content.split("\n") + i = 0 + while i < len(lines): + line = lines[i] + try: + if line.strip().startswith("- ") and not line.strip().startswith( + "- [" + ): + continuation = [] + j = i + 1 + while j < len(lines): + next_line = lines[j] + if next_line.strip() == "" or next_line.strip().startswith( + "- " + ): + break + continuation.append(next_line) + j += 1 + + pref = parse_discovery(line.strip(), continuation) + pref["type"] = "preference" + prefs.append(pref) + i = j + continue + except Exception as e: + warn(f"Error parsing preference at line {i + 1} in " + f"{prefs_path}: {e}") + i += 1 + + return prefs + + +def parse_cross_cutting(shadow_dir): + """Parse all _cross/*.md files.""" + cross_dir = shadow_dir / "_cross" + if not cross_dir.exists(): + return [] + + entries = [] + try: + md_files = sorted(cross_dir.glob("*.md")) + except OSError as e: + warn(f"Cannot list cross-cutting directory {cross_dir}: {e}") + return [] + + for f in md_files: + try: + content = f.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + warn(f"Cannot read cross-cutting file {f}: {e}") + continue + + entry = {"slug": f.stem, "file": str(f.name), "citation_score": 0} + + try: + # Title + m = re.search(r"^# (.+)", content, re.MULTILINE) + if m: + entry["title"] = m.group(1).strip() + + # Category + m = re.search(r"\*\*Category\*\*:\s*(.+)", content) + if m: + entry["category"] = m.group(1).strip() + + # Refs — only within the **Refs**: section, not backticked + # bullets elsewhere in the file (e.g., inside the Discovery body). + refs = [] + refs_block = re.search( + r"\*\*Refs\*\*:\s*\n(.*?)(?=\n[ \t]*\n|\n\*\*|\Z)", + content, + re.DOTALL, + ) + if refs_block: + refs = re.findall(r"-\s*`([^`]+)`", refs_block.group(1)) + entry["refs"] = refs + + # Discovery text + m = re.search( + r"\*\*Discovery\*\*:\s*(.+?)(?=\n\n|\n_\(|\Z)", + content, + re.DOTALL, + ) + if m: + entry["discovery"] = m.group(1).strip() + + # Status/source (with optional labels) + m = _DISCOVERY_META_RE.search(content) + if m: + entry["status"] = m.group(1) + entry["source"] = m.group(2) + entry["citation_score"] = int(m.group(4) or 0) + if m.group(3): + entry["labels"] = [ + l.strip() + for l in m.group(3).split(",") + if l.strip() + ] + else: + # Fallback to simpler pattern + m2 = re.search( + r"_\((\w+),\s*source:\s*(\w+)\)_", content + ) + if m2: + entry["status"] = m2.group(1) + entry["source"] = m2.group(2) + except Exception as e: + warn(f"Error parsing cross-cutting file {f.name}: " + f"{type(e).__name__}: {e}") + entry.setdefault("title", f.stem) + + entries.append(entry) + + return entries + + +def load_state(shadow_dir): + """Load _meta/state.json.""" + state_path = shadow_dir / "_meta" / "state.json" + if not state_path.exists(): + return {} + try: + content = state_path.read_text(encoding="utf-8") + state = json.loads(content) + if not isinstance(state, dict): + warn(f"state.json is not a JSON object (got {type(state).__name__})") + return {} + return state + except json.JSONDecodeError as e: + warn(f"Invalid JSON in {state_path}: {e}") + return {} + except OSError as e: + warn(f"Cannot read {state_path}: {e}") + return {} + + +def get_all_shadow_files(shadow_dir): + """Get all per-file shadow .md files (excluding special files).""" + special = {"_index.md", "_prefs.md"} + special_dirs = {"_cross", "_meta", "_dreams"} + + results = [] + try: + for f in shadow_dir.rglob("*.md"): + try: + rel = f.relative_to(shadow_dir) + parts = rel.parts + if parts[0] in special_dirs: + continue + if str(rel) in special: + continue + results.append(f) + except (ValueError, IndexError) as e: + warn(f"Skipping file {f}: {e}") + except OSError as e: + warn(f"Error walking shadow directory {shadow_dir}: {e}") + + return sorted(results) + + +def collect_all_discoveries(shadow_dir): + """Parse all shadow files and collect every discovery. + + Continues past individual file failures — reports errors and moves on. + """ + all_disc = [] + failed_files = [] + for sf in get_all_shadow_files(shadow_dir): + try: + parsed = parse_shadow_file(sf) + if parsed.get("parse_errors"): + failed_files.append( + (str(sf), parsed["parse_errors"]) + ) + for d in parsed["discoveries"]: + d.setdefault("file", parsed["source_file"]) + d["shadow_path"] = str(sf.relative_to(shadow_dir)) + try: + d["shadow_mtime"] = os.path.getmtime(sf) + except OSError: + d["shadow_mtime"] = 0.0 + all_disc.append(d) + except Exception as e: + msg = f"Failed to parse {sf}: {type(e).__name__}: {e}" + warn(msg) + failed_files.append((str(sf), [msg])) + + if failed_files: + warn(f"{len(failed_files)} file(s) had parse errors " + f"(discoveries from other files still collected)") + + return all_disc + + +# --- View Functions --- + +def _citation_rank(item): + if item.get("status") == "refuted": + trust = 5 + elif item.get("source") == "user": + trust = 0 + elif item.get("source") == "interaction": + trust = 1 + else: + trust = {"verified": 2, "uncertain": 3}.get(item.get("status"), 4) + return trust, -item.get("citation_score", 0) + + +def view_summary(shadow_dir): + """Overview + detailed statistics. + + Each section is independently wrapped — if label stats fail, you + still get counts and the per-file table. + """ + # Load data (each can fail independently) + state = {} + shadow_files = [] + prefs = [] + cross = [] + + try: + state = load_state(shadow_dir) + except Exception as e: + warn(f"Failed to load state.json: {e}") + + try: + shadow_files = get_all_shadow_files(shadow_dir) + except Exception as e: + warn(f"Failed to list shadow files: {e}") + + try: + prefs = parse_prefs(shadow_dir) + except Exception as e: + warn(f"Failed to parse preferences: {e}") + + try: + cross = parse_cross_cutting(shadow_dir) + except Exception as e: + warn(f"Failed to parse cross-cutting discoveries: {e}") + + # Parse each file once for both stats and discoveries + file_stats = [] + all_disc = [] + total_symbols = 0 + for sf in shadow_files: + try: + parsed = parse_shadow_file(sf) + src = parsed["source_file"] or str(sf.relative_to(shadow_dir)) + n_sym = len(parsed["symbols"]) + n_disc = len(parsed["discoveries"]) + total_symbols += n_sym + file_stats.append((src, n_sym, n_disc)) + for d in parsed["discoveries"]: + d.setdefault("file", parsed["source_file"]) + d["shadow_path"] = str(sf.relative_to(shadow_dir)) + all_disc.append(d) + except Exception as e: + warn(f"Failed to process {sf}: {e}") + file_stats.sort(key=lambda x: x[2], reverse=True) + + # Header counts (always shown) + print("Shadow Knowledge Base Summary") + print("=" * 50) + print(f" Files shadowed: {len(shadow_files)}") + print(f" Symbols tracked: {total_symbols}") + print(f" Discoveries: {len(all_disc)}") + print(f" Preferences: {len(prefs)}") + print(f" Cross-cutting: {len(cross)}") + + # Source breakdown + try: + source_counts = defaultdict(int) + status_counts = defaultdict(int) + for d in all_disc: + source_counts[d.get("source", "unknown")] += 1 + status_counts[d.get("status", "unknown")] += 1 + + if source_counts: + print("\nBy source:") + for src, cnt in sorted(source_counts.items(), key=lambda x: -x[1]): + pct = cnt / len(all_disc) * 100 if all_disc else 0 + bar = "#" * int(pct / 2) + print(f" {src:15s} {cnt:4d} ({pct:5.1f}%) {bar}") + + if status_counts: + print("\nBy status:") + for st, cnt in sorted(status_counts.items(), key=lambda x: -x[1]): + pct = cnt / len(all_disc) * 100 if all_disc else 0 + bar = "#" * int(pct / 2) + print(f" {st:15s} {cnt:4d} ({pct:5.1f}%) {bar}") + except Exception as e: + warn(f"Failed to compute source/status breakdown: {e}") + + # Label breakdown + try: + label_counts = defaultdict(int) + for d in all_disc: + for lbl in d.get("labels", []): + label_counts[lbl] += 1 + if label_counts: + print("\nBy label:") + for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]): + print(f" {lbl:15s} {cnt:4d}") + except Exception as e: + warn(f"Failed to compute label breakdown: {e}") + + # Per-file table + try: + if file_stats: + print(f"\n{'File':<40s} {'Symbols':>8s} {'Disc.':>6s}") + print(f"{'-'*40} {'-'*8} {'-'*6}") + for src, n_sym, n_disc in file_stats[:20]: + print(f"{src:<40s} {n_sym:>8d} {n_disc:>6d}") + if len(file_stats) > 20: + print(f"... and {len(file_stats) - 20} more files") + except Exception as e: + warn(f"Failed to render per-file table: {e}") + + # Cross-cutting titles + try: + if cross: + print(f"\nCross-cutting discoveries:") + for e in cross: + title = e.get("title", e.get("slug", "?")) + cat = e.get("category", "?") + print(f" [{cat}] {title}") + except Exception as e: + warn(f"Failed to render cross-cutting list: {e}") + + # State info + try: + if state: + print(f"\nLast update: {state.get('last_update_at', '?')} " + f"({state.get('last_update_type', '?')})") + print(f"Last commit: {state.get('last_commit', '?')}") + except Exception as e: + warn(f"Failed to render state info: {e}") + + +def view_search(shadow_dir, query): + """Universal search: matches file names, symbol names, and discovery text. + + Also searches cross-cutting discoveries and preferences. + Results are grouped by match location for readability. + Each search domain (per-file, cross-cutting, prefs) is independent — + if one fails, the others still return results. + """ + query_lower = query.lower() + disc_matches = [] + cross_matches = [] + pref_matches = [] + section_errors = [] + + # Search per-file shadows + try: + for sf in get_all_shadow_files(shadow_dir): + try: + parsed = parse_shadow_file(sf) + source_file = parsed["source_file"] or str( + sf.relative_to(shadow_dir) + ) + file_name_hit = query_lower in source_file.lower() + + for d in parsed["discoveries"]: + sym = d.get("symbol", "") + text = d.get("text", "") + sym_hit = query_lower in sym.lower() + text_hit = query_lower in text.lower() + also_hit = any( + query_lower in ref.lower() + for ref in d.get("also_involves", []) + ) + + if file_name_hit or sym_hit or text_hit or also_hit: + disc_matches.append({ + "file": source_file, + "symbol": sym, + "text": text, + "status": d.get("status", "?"), + "source": d.get("source", "?"), + "citation_score": d.get("citation_score", 0), + "also_involves": d.get("also_involves", []), + "match": ( + "file" if file_name_hit else + "symbol" if sym_hit else + "also_involves" if also_hit else "text" + ), + }) + except Exception as e: + warn(f"Search: error processing {sf}: {e}") + except Exception as e: + msg = f"Search: failed to search per-file shadows: {e}" + warn(msg) + section_errors.append(msg) + + # Search cross-cutting discoveries + try: + for e in parse_cross_cutting(shadow_dir): + title = e.get("title", "") + disc_text = e.get("discovery", "") + refs = e.get("refs", []) + if (query_lower in title.lower() + or query_lower in disc_text.lower() + or any(query_lower in r.lower() for r in refs)): + cross_matches.append(e) + except Exception as e: + msg = f"Search: failed to search cross-cutting: {e}" + warn(msg) + section_errors.append(msg) + + # Search preferences + try: + for p in parse_prefs(shadow_dir): + if query_lower in p.get("text", "").lower(): + pref_matches.append(p) + except Exception as e: + msg = f"Search: failed to search preferences: {e}" + warn(msg) + section_errors.append(msg) + + total = len(disc_matches) + len(cross_matches) + len(pref_matches) + if total == 0: + print(f"No results for '{query}'.") + if section_errors: + print(f"Note: {len(section_errors)} search section(s) had errors " + f"— results may be incomplete. Check stderr for details.") + return + + print(f"Search: '{query}' ({total} results)") + print("=" * 50) + + # Per-file discoveries, grouped by file + if disc_matches: + try: + by_file = defaultdict(list) + for d in disc_matches: + by_file[d["file"]].append(d) + + for file, discs in sorted(by_file.items()): + print(f"\n{file} ({len(discs)} matches)") + print("-" * (len(file) + 15)) + for d in sorted(discs, key=_citation_rank): + sym = d["symbol"] + print(f" {file}::{sym}") + print(f" {d['text'][:120]}") + print(f" ({d['status']}, source: {d['source']})") + print(f" citation_score: {d['citation_score']}") + if d.get("also_involves"): + print( + f" Also involves: " + f"{', '.join(d['also_involves'])}" + ) + except Exception as e: + warn(f"Search: failed to render per-file results: {e}") + + # Cross-cutting + if cross_matches: + try: + print(f"\nCross-cutting ({len(cross_matches)} matches)") + print("-" * 30) + for e in sorted(cross_matches, key=_citation_rank): + title = e.get("title", e.get("slug", "?")) + cat = e.get("category", "?") + status = e.get("status", "?") + source = e.get("source", "?") + print(f"\n {title}") + print(f" Category: {cat} | {status}, source: {source}") + print(f" citation_score: {e.get('citation_score', 0)}") + print(f" Refs: {', '.join(e.get('refs', [])[:5])}") + if e.get("discovery"): + print(f" {e['discovery'][:120]}") + except Exception as e: + warn(f"Search: failed to render cross-cutting results: {e}") + + # Preferences + if pref_matches: + try: + print(f"\nPreferences ({len(pref_matches)} matches)") + print("-" * 30) + for p in sorted(pref_matches, key=_citation_rank): + print(f" [{p.get('source', '?')}] {p['text'][:120]}") + print(f" citation_score: {p.get('citation_score', 0)}") + except Exception as e: + warn(f"Search: failed to render preference results: {e}") + + +def view_prefs(shadow_dir): + """Show all preferences.""" + try: + prefs = parse_prefs(shadow_dir) + except Exception as e: + error(f"Failed to parse preferences: {e}") + return + + if not prefs: + print("No preferences recorded yet.") + return + + print(f"Project Preferences ({len(prefs)} total)") + print("=" * 40) + for p in sorted(prefs, key=_citation_rank): + try: + source = p.get("source", "?") + print(f"\n [{source}] {p['text']}") + print(f" citation_score: {p.get('citation_score', 0)}") + except Exception as e: + warn(f"Failed to render preference: {e}") + + +def view_labels(shadow_dir, label_filter): + """Show discoveries filtered by label(s). + + label_filter can be a single label or comma-separated list. + """ + try: + filters = [l.strip().lower() for l in label_filter.split(",")] + except Exception as e: + error(f"Invalid label filter '{label_filter}': {e}") + return + + try: + all_disc = collect_all_discoveries(shadow_dir) + except Exception as e: + error(f"Failed to collect discoveries for label filtering: {e}") + return + + # Also include cross-cutting discoveries with labels + try: + for entry in parse_cross_cutting(shadow_dir): + if entry.get("labels"): + all_disc.append({ + "file": f"_cross/{entry.get('file', '?')}", + "symbol": entry.get("title", entry.get("slug", "?")), + "text": entry.get("discovery", entry.get("title", "")), + "status": entry.get("status", "?"), + "source": entry.get("source", "?"), + "labels": entry["labels"], + "citation_score": entry.get("citation_score", 0), + }) + except Exception as e: + warn(f"Failed to include cross-cutting in label search: {e}") + + matching = [] + for d in all_disc: + try: + disc_labels = [l.lower() for l in d.get("labels", [])] + if any(f in disc_labels for f in filters): + matching.append(d) + except Exception as e: + warn(f"Failed to check labels on discovery in " + f"{d.get('file', '?')}::{d.get('symbol', '?')}: {e}") + + if not matching: + print(f"No discoveries with label(s): {', '.join(filters)}") + return + + print(f"Discoveries with label(s): {', '.join(filters)} " + f"({len(matching)} results)") + print("=" * 50) + + by_label = defaultdict(list) + for d in matching: + for lbl in d.get("labels", []): + if lbl.lower() in filters: + by_label[lbl.lower()].append(d) + + for lbl in filters: + discs = by_label.get(lbl, []) + if not discs: + continue + print(f"\n[{lbl}] ({len(discs)} discoveries)") + print("-" * 30) + for d in sorted(discs, key=_citation_rank): + try: + sym = d.get("symbol", "?") + src_file = d.get("file", "?") + print(f" {src_file}::{sym}") + print(f" {d['text'][:120]}") + print(f" ({d.get('status', '?')}, " + f"source: {d.get('source', '?')})") + print(f" citation_score: {d.get('citation_score', 0)}") + all_labels = d.get("labels", []) + other = [l for l in all_labels if l.lower() != lbl] + if other: + print(f" Also labeled: {', '.join(other)}") + except Exception as e: + warn(f"Failed to render labeled discovery: {e}") + + +def view_recent(shadow_dir, count=10): + """Show the N most recent discoveries (by shadow file mtime). + + Collects all discoveries across all shadow files, cross-cutting entries, + and preferences, sorts by the source file's modification time (most recent + first), and shows the actual discovery content. + Each data source is independent — if cross-cutting fails, per-file + discoveries still appear. + """ + all_items = [] + + # Per-file discoveries + try: + for sf in get_all_shadow_files(shadow_dir): + try: + mtime = os.path.getmtime(sf) + parsed = parse_shadow_file(sf) + source_file = parsed["source_file"] or str( + sf.relative_to(shadow_dir) + ) + for d in parsed["discoveries"]: + all_items.append({ + "type": "discovery", + "file": source_file, + "symbol": d.get("symbol", "?"), + "text": d.get("text", ""), + "status": d.get("status", "?"), + "source": d.get("source", "?"), + "citation_score": d.get("citation_score", 0), + "mtime": mtime, + }) + except Exception as e: + warn(f"Recent: failed to process {sf}: {e}") + except Exception as e: + warn(f"Recent: failed to list shadow files: {e}") + + # Cross-cutting discoveries + try: + cross_entries = parse_cross_cutting(shadow_dir) + cross_by_file = defaultdict(list) + for e in cross_entries: + cross_by_file[e["file"]].append(e) + + cross_dir = shadow_dir / "_cross" + if cross_dir.exists(): + for cf in cross_dir.glob("*.md"): + try: + mtime = os.path.getmtime(cf) + for e in cross_by_file.get(cf.name, []): + all_items.append({ + "type": "cross-cutting", + "file": f"_cross/{cf.name}", + "symbol": e.get("title", e.get("slug", "?")), + "text": e.get("discovery", e.get("title", "")), + "status": e.get("status", "?"), + "source": e.get("source", "?"), + "citation_score": e.get("citation_score", 0), + "mtime": mtime, + }) + except Exception as e: + warn(f"Recent: failed to process cross-cutting {cf}: {e}") + except Exception as e: + warn(f"Recent: failed to process cross-cutting discoveries: {e}") + + # Preferences + try: + prefs_path = shadow_dir / "_prefs.md" + if prefs_path.exists(): + mtime = os.path.getmtime(prefs_path) + for p in parse_prefs(shadow_dir): + all_items.append({ + "type": "preference", + "file": "_prefs.md", + "symbol": "-", + "text": p.get("text", ""), + "status": "-", + "source": p.get("source", "?"), + "citation_score": p.get("citation_score", 0), + "mtime": mtime, + }) + except Exception as e: + warn(f"Recent: failed to process preferences: {e}") + + all_items.sort(key=lambda x: x.get("mtime", 0), reverse=True) + + if not all_items: + print("No discoveries found.") + return + + shown = all_items[:count] + print(f"Most Recent Discoveries (top {count})") + print("=" * 50) + for item in shown: + try: + ts = datetime.fromtimestamp( + item.get("mtime", 0) + ).strftime("%Y-%m-%d %H:%M") + kind = item.get("type", "?") + sym = item.get("symbol", "?") + + print(f"\n [{ts}] ({kind})") + if kind == "preference": + print(f" {item.get('text', '')[:120]}") + print(f" source: {item.get('source', '?')}") + else: + print(f" {item.get('file', '?')}::{sym}") + print(f" {item.get('text', '')[:120]}") + print(f" ({item.get('status', '?')}, " + f"source: {item.get('source', '?')})") + print(f" citation_score: {item.get('citation_score', 0)}") + except Exception as e: + warn(f"Recent: failed to render item: {e}") + + +def view_top(shadow_dir, file_path, labels_filter, limit, max_chars): + """Show the top N actionable discoveries for a single source file. + + Designed for the preToolUse hook: concise output suitable for + inlining into additionalContext when the agent is about to mutate a + file. Pulls from both the per-file shadow and any _cross/ entries + whose refs touch this file. + + Source trust/status precedes the visible citation score; equal scores + preserve document order. Reading does not increment citations. + Output is hard-capped at max_chars (the trailing "(...)" marker still fits). + """ + norm = file_path.strip() + if norm.startswith("./"): + norm = norm[2:] + shadow_path = shadow_dir / f"{norm}.md" + label_set = {l.strip().lower() for l in labels_filter.split(",") if l.strip()} + + candidates = [] + + if shadow_path.is_file(): + try: + parsed = parse_shadow_file(shadow_path) + for d in parsed.get("discoveries", []): + disc_labels = {l.lower() for l in d.get("labels", [])} + if label_set and not (disc_labels & label_set): + continue + candidates.append({ + "kind": "file", + "anchor": d.get("symbol") or "file-level", + "text": d.get("text", "").strip(), + "status": d.get("status", "?"), + "source": d.get("source", "?"), + "citation_score": d.get("citation_score", 0), + "labels": sorted(disc_labels), + }) + except Exception as e: + warn(f"--top: failed parsing {shadow_path}: {e}") + + try: + for entry in parse_cross_cutting(shadow_dir): + refs = entry.get("refs", []) or [] + if not any(r.split("::", 1)[0].strip() == norm for r in refs): + continue + cross_labels = {l.lower() for l in entry.get("labels", [])} + if label_set and not (cross_labels & label_set): + continue + candidates.append({ + "kind": "cross", + "anchor": f"_cross/{entry.get('file', entry.get('slug', '?'))}", + "text": (entry.get("discovery") or entry.get("title") or "").strip(), + "status": entry.get("status", "?"), + "source": entry.get("source", "?"), + "citation_score": entry.get("citation_score", 0), + "labels": sorted(cross_labels), + }) + except Exception as e: + warn(f"--top: failed scanning _cross/: {e}") + + if not candidates: + labels_disp = ",".join(sorted(label_set)) if label_set else "any" + print( + f"No actionable discoveries ({labels_disp}) for {norm}." + ) + return + + candidates.sort(key=_citation_rank) + + shown = candidates[:limit] + header = ( + f"Top {len(shown)} of {len(candidates)} actionable discoveries " + f"for {norm}:" + ) + lines = [header] + for d in shown: + labels = ",".join(d["labels"]) if d["labels"] else "—" + anchor = d["anchor"] + text = d["text"].replace("\n", " ").strip() + lines.append( + f"- [{labels}] `{anchor}` ({d['status']}, citation_score: {d['citation_score']}): {text}" + ) + + out = "\n".join(lines) + if max_chars and len(out) > max_chars: + truncated = out[: max_chars - 6].rstrip() + out = truncated + "\n(...)" + print(out) + + +def view_check_invariants(shadow_dir): + """Walk the shadow knowledge base and report invariant violations. + + Statically-checkable invariants from shadow-frog/SKILL.md: + #3 (partial) Per-file 'Also involves:' uses file::symbol notation + #4 Cross-ref back-pointers match: _cross/.md refs <-> + per-file ## Cross-References + #5 Every ## Cross-References entry has a matching _cross/*.md + + Plus syntactic guards that catch the most common drift: + - Symbol headings use the required backtick form + - Discovery metadata uses valid status enum + - Discovery metadata uses valid source enum + - Discovery labels are from the allowed set + - _cross/ Category field uses a known value + + Invariants #1, #2, #7 are NOT checked (would require source parsing + and semantic match); #6 is filesystem-enforced. + + Exit 0 = clean, 1 = at least one violation. Violations print one per + line in `path:line: kind: message` form so grep/editors can navigate. + """ + VALID_STATUS = {"verified", "uncertain", "refuted"} + VALID_SOURCE = {"exploration", "user", "interaction"} + VALID_LABELS = {"bug", "performance", "security", + "feature-gap", "tech-debt"} + VALID_CATEGORIES = { + "pattern", "behavior", "edge-case", "contract", + "performance", "intent", "warning", "history", "convention", + } + + violations = [] + def v(path, line, kind, msg): + violations.append(f"{path}:{line}: {kind}: {msg}") + + # Pass 1: walk per-file shadows -> collect cross-reference entries + # they declare and validate their internal format. + per_file_xref_targets = {} # rel_shadow_path -> set(slug declared) + cross_dir = shadow_dir / "_cross" + cross_slugs_on_disk = set() + if cross_dir.is_dir(): + try: + cross_slugs_on_disk = {f.stem for f in cross_dir.glob("*.md")} + except OSError as e: + warn(f"Cannot list {cross_dir}: {e}") + + md_heading_re = re.compile(r"^(#{2,3})\s+(.*)$") + backtick_heading_re = re.compile(r"^(#{2,3})\s+`[^`]+`\s*$") + also_involves_re = re.compile(r"^\s*Also involves:\s*(.+)$", re.I) + file_sym_re = re.compile(r"`([^`]+::[^`]+)`") + + def check_citation(path, line_number, line): + if line.strip().startswith("_(") and "citation_score:" in line: + try: + metadata_score(line) + except CitationError as exc: + v(path, line_number, "citation_score", str(exc)) + + for shadow_path in get_all_shadow_files(shadow_dir): + try: + rel = shadow_path.relative_to(shadow_dir) + except ValueError: + continue + try: + text = shadow_path.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + v(rel, 0, "unreadable", str(e)) + continue + + in_cross_refs = False + declared = set() + for ln, raw in enumerate(text.split("\n"), 1): + line = raw.rstrip() + check_citation(rel, ln, line) + + heading = md_heading_re.match(line) + if heading: + title = heading.group(2).strip() + if title.lower().startswith("cross-references"): + in_cross_refs = True + continue + in_cross_refs = False + # Skip special headings ("File-Level Notes", "Notes", etc.) + if ( + title.lower().startswith("file-level") + or title.lower() in {"notes", "metadata"} + ): + continue + # Symbol heading must use backtick form + if not backtick_heading_re.match(line): + v(rel, ln, "heading", + f"symbol heading must be `## `name`` or " + f"`### `Class.name``; got: {line[:80]}") + continue + + if in_cross_refs and line.strip().startswith("- "): + # Format: - [slug](.shadow/_cross/slug.md) — title + slug_match = re.search( + r"_cross/([^)\s]+?)\.md", line + ) + if slug_match: + declared.add(slug_match.group(1)) + else: + # Looser fallback: bare slug in brackets + alt = re.search(r"\[([^\]]+)\]", line) + if alt: + declared.add(alt.group(1).strip()) + + # Discovery metadata line + md = _DISCOVERY_META_RE.search(line) + if md: + status, source = md.group(1), md.group(2) + labels_raw = md.group(3) or "" + if status not in VALID_STATUS: + v(rel, ln, "enum", + f"status '{status}' not in {sorted(VALID_STATUS)}") + if source not in VALID_SOURCE: + v(rel, ln, "enum", + f"source '{source}' not in {sorted(VALID_SOURCE)}") + for lbl in (l.strip() for l in labels_raw.split(",") if l.strip()): + if lbl not in VALID_LABELS: + v(rel, ln, "enum", + f"label '{lbl}' not in {sorted(VALID_LABELS)}") + + # `Also involves:` must list file::symbol anchors in backticks + ai = also_involves_re.match(line) + if ai: + rest = ai.group(1) + anchors = file_sym_re.findall(rest) + if not anchors: + v(rel, ln, "anchor", + "Also involves: needs `file::symbol` " + "backtick anchors") + # Light sanity: every anchor has both file and symbol + for a in anchors: + if "::" not in a or not a.split("::", 1)[1].strip(): + v(rel, ln, "anchor", + f"anchor '{a}' missing symbol after ::") + + per_file_xref_targets[str(rel)] = declared + + # Invariant #5: every declared cross slug must exist on disk + for slug in declared: + if slug not in cross_slugs_on_disk: + v(rel, 0, "cross-ref", + f"references _cross/{slug}.md but file does not exist") + + # Pass 2: walk _cross/*.md -> validate refs format + back-pointer. + # Build the reverse map: cross_slug -> set(file::symbol it points at). + cross_back = {} # slug -> set(file paths it should be linked from) + if cross_dir.is_dir(): + for cf in sorted(cross_dir.glob("*.md")): + slug = cf.stem + try: + text = cf.read_text(encoding="utf-8") + except (OSError, UnicodeDecodeError) as e: + v(cf.relative_to(shadow_dir), 0, "unreadable", str(e)) + continue + + rel_cf = cf.relative_to(shadow_dir) + for line_number, line in enumerate(text.splitlines(), 1): + check_citation(rel_cf, line_number, line) + + # Category enum check + cat_m = re.search(r"\*\*Category\*\*:\s*(.+)", text) + if cat_m: + cat = cat_m.group(1).strip().lower() + if cat not in VALID_CATEGORIES: + v(rel_cf, 0, "enum", + f"Category '{cat}' not in {sorted(VALID_CATEGORIES)}") + else: + v(rel_cf, 0, "schema", + "missing **Category**: field") + + # Discovery metadata + md = _DISCOVERY_META_RE.search(text) + if md: + status, source = md.group(1), md.group(2) + if status not in VALID_STATUS: + v(rel_cf, 0, "enum", + f"status '{status}' not in {sorted(VALID_STATUS)}") + if source not in VALID_SOURCE: + v(rel_cf, 0, "enum", + f"source '{source}' not in {sorted(VALID_SOURCE)}") + else: + v(rel_cf, 0, "schema", + "missing trailing _(status, source: ...)_ metadata") + + # Refs must be `file::symbol` anchors + refs_block = re.search( + r"\*\*Refs\*\*:\s*\n((?:\s*-\s+`[^`]+`\s*\n?)+)", + text, + ) + if not refs_block: + v(rel_cf, 0, "schema", + "missing **Refs**: block (one per line, " + "`- `file::symbol``)") + else: + anchors = file_sym_re.findall(refs_block.group(1)) + if not anchors: + v(rel_cf, 0, "anchor", + "Refs block has no `file::symbol` entries") + for a in anchors: + if "::" not in a or not a.split("::", 1)[1].strip(): + v(rel_cf, 0, "anchor", + f"ref '{a}' missing symbol after ::") + else: + # Convert file part to shadow path: + # src/foo.py -> src/foo.py.md (relative to shadow_dir) + file_part = a.split("::", 1)[0].strip() + shadow_rel = f"{file_part}.md" + cross_back.setdefault(slug, set()).add(shadow_rel) + + # Invariant #4 back-pointer: every file referenced by a cross slug + # must declare that slug in its ## Cross-References. + for slug, expected_files in cross_back.items(): + for shadow_rel in expected_files: + declared = per_file_xref_targets.get(shadow_rel) + if declared is None: + v(f"_cross/{slug}.md", 0, "cross-ref", + f"refs {shadow_rel} but no such shadow file exists") + elif slug not in declared: + v(f"_cross/{slug}.md", 0, "cross-ref", + f"refs {shadow_rel} but that shadow's ## " + f"Cross-References does not link back to " + f"_cross/{slug}.md") + + prefs_path = shadow_dir / "_prefs.md" + if prefs_path.is_file(): + try: + for line_number, line in enumerate(prefs_path.read_text(encoding="utf-8").splitlines(), 1): + check_citation("_prefs.md", line_number, line) + except (OSError, UnicodeError) as exc: + v("_prefs.md", 0, "unreadable", str(exc)) + + # Output + if not violations: + print(f"✓ Invariants OK ({len(per_file_xref_targets)} per-file " + f"shadows, {len(cross_slugs_on_disk)} cross-cutting " + f"discoveries)") + return 0 + + for line in violations: + print(line) + print(f"\n{len(violations)} invariant violation(s) found.", + file=sys.stderr) + return 1 + + def main(): - return _knowledge.main() + for _stream in (sys.stdout, sys.stderr): + if hasattr(_stream, "reconfigure"): + _stream.reconfigure(encoding="utf-8") + try: + parser = argparse.ArgumentParser( + description="Query and browse a .shadow/ knowledge base.", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + + # Views (mutually exclusive) + views = parser.add_mutually_exclusive_group() + views.add_argument( + "--summary", action="store_true", + help="Overview + detailed statistics (default)", + ) + views.add_argument( + "--search", metavar="QUERY", + help="Universal search: files, symbols, and discovery text", + ) + views.add_argument( + "--prefs", action="store_true", + help="Show project-wide preferences", + ) + views.add_argument( + "--recent", nargs="?", const=10, type=int, metavar="N", + help="N most recent discoveries with content (default: 10)", + ) + views.add_argument( + "--labels", metavar="LABEL", + help=( + "Show discoveries by label " + "(e.g., bug, security, bug,performance)" + ), + ) + views.add_argument( + "--top", metavar="FILE", + help=( + "Top actionable discoveries for FILE (a source path " + "like src/auth.py). Concise output for the preToolUse " + "hook: filters to actionable labels (default: " + "bug,security), includes both per-file and _cross/ " + "entries that reference FILE, ranks verified first." + ), + ) + views.add_argument( + "--check-invariants", action="store_true", + help=( + "Walk the shadow and report structural violations: " + "missing back-pointers, dangling _cross/ refs, invalid " + "enums, bad heading format. Exit 1 if any are found." + ), + ) + + # Options + parser.add_argument( + "--shadow-dir", default=None, + help="Path to .shadow/ directory (default: auto-detect)", + ) + parser.add_argument( + "--top-labels", default="bug,security", metavar="LABELS", + help=( + "Comma-separated labels to include in --top " + "(default: bug,security). Pass empty string to include " + "all labeled discoveries." + ), + ) + parser.add_argument( + "--top-limit", type=int, default=3, metavar="N", + help="Max discoveries to show in --top (default: 3)", + ) + parser.add_argument( + "--top-max-chars", type=int, default=600, metavar="N", + help=( + "Hard cap on --top total output length " + "(default: 600). Use 0 for no cap." + ), + ) + + args = parser.parse_args() + + # Find shadow dir + if args.shadow_dir: + shadow_dir = Path(args.shadow_dir) + else: + shadow_dir = find_shadow_dir() + + if not shadow_dir or not shadow_dir.is_dir(): + cwd = os.getcwd() + error( + f"No .shadow/ directory found. " + f"Searched from: {cwd}\n" + f"[shadow-viewer error] " + f"Run /shadow-frog-init first to create the shadow, " + f"or pass --shadow-dir /path/to/.shadow/ explicitly." + ) + if args.shadow_dir: + error( + f"Provided --shadow-dir '{args.shadow_dir}' does not " + f"exist or is not a directory." + ) + sys.exit(1) + + # Dispatch + if args.check_invariants: + sys.exit(view_check_invariants(shadow_dir)) + if args.top: + view_top( + shadow_dir, + args.top, + args.top_labels, + args.top_limit, + args.top_max_chars, + ) + elif args.search: + view_search(shadow_dir, args.search) + elif args.prefs: + view_prefs(shadow_dir) + elif args.labels: + view_labels(shadow_dir, args.labels) + elif args.recent is not None: + view_recent(shadow_dir, args.recent) + else: + view_summary(shadow_dir) + + except SystemExit: + raise + except KeyboardInterrupt: + error("Interrupted by user.") + sys.exit(130) + except Exception as e: + error( + f"Unexpected error: {type(e).__name__}: {e}\n" + f"[shadow-viewer error] Full traceback:\n" + f"{traceback.format_exc()}" + f"This is likely a bug in shadow-viewer.py. " + f"The shadow data may be in an unexpected format. " + f"Try running with --shadow-dir to confirm the path, " + f"or inspect the .shadow/ files manually." + ) + sys.exit(1) if __name__ == "__main__": - sys.exit(main()) + main() diff --git a/skills/shadow-frog/SKILL.md b/skills/shadow-frog/SKILL.md index e8cb8b8..3a5238e 100644 --- a/skills/shadow-frog/SKILL.md +++ b/skills/shadow-frog/SKILL.md @@ -12,7 +12,7 @@ description: >- ideation, shadow-frog-meditate for shadow hygiene, or shadow-frog-viewer for user-facing browsing and visualization. scripts: - - shadow-read.py + - shadow-cite.py --- # ShadowFrog @@ -25,7 +25,7 @@ to that code location. **Every time you work on code in a repo with `.shadow/`:** -1. **Read `.shadow/_prefs.md` first** — it contains project-wide conventions, +1. **Read `_prefs.md` first** — it contains project-wide conventions, user preferences, and things to avoid. 2. **Read relevant `_cross/` discoveries** — list `_cross/` and read entries whose titles relate to the current area, including cross-file contracts @@ -33,10 +33,8 @@ to that code location. 3. **Check `_dreams/_index.md`** and read relevant experiment reports, especially when investigating bugs or unfamiliar code. They may contain findings not yet distilled into per-file shadows. -4. **Before editing a file**, navigate directly to `.shadow/.md` and its - symbol headings, then follow relevant `_cross/` back-pointers. Use native - file reads/searches; for unusually large sections, the optional core helper - below can return bounded selections. A shortlist is not a complete file review. +4. **Before editing a file**, read its shadow (`.shadow/.md`) and + relevant `_cross/` entries, then apply the discoveries. `_index.md` counts may be stale; inspect the actual shadows and `_cross/`. 5. **When the user explains something about code** (gotcha, design intent, warning, history): write a `source: user` discovery to the shadow @@ -46,37 +44,50 @@ to that code location. specific file): write it to `_prefs.md` immediately. 7. **After code changes**: run `/shadow-frog-update` -## Optional Agent Retrieval +## Record Revisited Knowledge -**File/symbol navigation is primary.** The source path already locates its -shadow; no viewer, retrieval service, or citation database is required to read it. -`/shadow-frog-viewer` is the user-facing browsing/visualization skill, not the -agent's required knowledge interface. +Continue navigating directly to shadow files and symbol headings. Each +discovery or preference has one visible `citation_score`: a nonnegative integer, +initially 0. Missing scores also mean 0. It is approximate, agent-reported +revisit frequency, not confidence or proof of usefulness. -For large files/symbol sections or a targeted search, use `shadow-read.py` -beside this skill. Known paths are read directly, not rediscovered by global -search. Examples below use the Copilot install; Claude Code uses `.claude/skills/`. +After deliberately consulting an entry, record one citation for it **per task**. +Do not count every entry merely because its file was opened, repeat the count +on rereads, or count automatic previews that you did not use. Batch updates when +practical; the agent/coordinator tracks which entries it already counted. + +Use the small increment helper to avoid competing score edits. It does not +retrieve knowledge, create a database, or require opaque IDs: ```text -python .github/skills/shadow-frog/shadow-read.py src/auth.py -python .github/skills/shadow-frog/shadow-read.py src/auth.py::UserAuth.validate --limit 5 -python .github/skills/shadow-frog/shadow-read.py --search "token expiry" --max-chars 1800 +python .github/skills/shadow-frog/shadow-cite.py .shadow/src/auth.py.md --symbol UserAuth.validate --text "Rejects expired tokens." +python .github/skills/shadow-frog/shadow-cite.py .shadow/_prefs.md --text "Keep public APIs stable." ``` -The optional helper can rank/paginate large sections and expand returned IDs. -Keep file/symbol anchors as the navigation address; IDs only identify individual -entries within this helper. Full-file reads include file-level discoveries and -cross-cutting refs; `--top` is only a compact hint. - -Native file reads remain normal and uncounted. Helper-emitted entries increment -one local `citation_score` atomically; do not edit counters in Markdown or run a -second retrieval just to inflate them. Scores measure exposure, not correctness -or proven usefulness. Never omit user constraints because they are unpopular. -Use targeted searches and follow pages when completeness matters. - -Read [the helper reference](retrieval.md) when using its budgets, continuation, -or citation options. If optional telemetry fails, keep using the knowledge and -heed the diagnostic; do not substitute score availability for reference integrity. +For Claude Code, use `.claude/skills/`. Copy the exact claim text already read; +omit the bullet marker and metadata. Line wrapping is joined, but spaces inside +literals remain significant. +Use `--symbol File-Level` for file-wide entries; omit `--symbol` for preferences +and `_cross/` files. Repeat `--text` to update several entries in one section +atomically. `--shadow-dir` identifies an explicit nonstandard shadow root. +Unknown/ambiguous claims and invalid scores fail with corrective feedback. + +The helper locks only its target file and publishes the score changes atomically. +Coordinate citation writes with ordinary knowledge edits; unrelated editors do +not participate in this lock. Subagents should report consulted entries to their +coordinator rather than race full-file rewrites. If a lock survives interruption, +confirm its writer stopped before removing that specific `.citation.lock` file. + +Scores stay with the Markdown and travel through Git. They are not exact global +counts across branches/clones: when merging the same knowledge, keep the larger +score rather than summing inherited counts. Keep scores when rewording/moving +the same claim; a genuinely new claim starts at 0. Citation-only edits do not +create discoveries or change discovery totals. + +Treat higher scores as a secondary hint after relevance and trust. Never omit +a relevant user constraint or new discovery because its score is low. Native +searches and targeted reads remain the normal tools; `/shadow-frog-viewer` +serves user browsing and does not automatically record citations. ## Directory Layout @@ -117,7 +128,7 @@ The symbol name is the stable anchor. ## File-Level - This module has no __all__ — all top-level names are public. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ ## `class UserAuth` @@ -125,12 +136,12 @@ The symbol name is the stable anchor. - Catches ALL exceptions and returns False — swallows connection errors, making network failures look like invalid tokens. - _(verified, source: exploration)_ + _(verified, source: exploration, citation_score: 0)_ ## `authenticate_user` - Silently returns None on expired tokens. Callers must check. - _(verified, source: exploration, labels: [bug])_ + _(verified, source: exploration, labels: [bug], citation_score: 0)_ Also involves: `src/middleware.py::require_auth` ## Cross-References @@ -170,7 +181,7 @@ Examples: **Discovery**: All database access goes through a connection pool that silently reconnects on failure. First request after DB restart is slow (~2s). -_(verified, source: exploration)_ +_(verified, source: exploration, citation_score: 0)_ ``` ## Preferences File (`_prefs.md`) @@ -182,19 +193,19 @@ specific file or symbol. These guide all agent work across the codebase. # Preferences - No backward compatibility — only keep the latest code, no shims or aliases. - _(source: user)_ + _(source: user, citation_score: 0)_ - Use snake_case for all Python function and variable names. - _(source: user)_ + _(source: user, citation_score: 0)_ - Prefer small, focused PRs over large sweeping changes. - _(source: interaction)_ + _(source: interaction, citation_score: 0)_ ``` Format: ``` - - _(source: )_ + _(source: , citation_score: 0)_ ``` Preferences are always trusted (same rank as `source: user`). They don't @@ -207,24 +218,24 @@ When to write to `_prefs.md` vs per-file shadow vs `_cross/`: ## Discovery Format -Per-file discoveries (no stored IDs — anchored by their `file::symbol` heading): +Per-file discoveries (no IDs — anchored by their `file::symbol` heading): ``` - - _(, source: )_ + _(, source: , citation_score: 0)_ Also involves: `file::symbol`, `file::symbol` ``` With labels (optional — only when the discovery is actionable): ``` - - _(, source: , labels: [bug, security])_ + _(, source: , labels: [bug, security], citation_score: 0)_ Also involves: `file::symbol` ``` With dream report link (optional — only for experiment-derived discoveries): ``` - - _(, source: )_ + _(, source: , citation_score: 0)_ Dream report: `_dreams//` ``` @@ -238,7 +249,7 @@ Cross-cutting discoveries (one per `_cross/.md` file): **Discovery**: -_(, source: )_ +_(, source: , citation_score: 0)_ ``` Slug naming: use descriptive kebab-case derived from the title. @@ -263,7 +274,7 @@ Omit labels entirely for pure observational knowledge. Labels go in the metadata line: ``` -_(verified, source: exploration, labels: [bug])_ +_(verified, source: exploration, labels: [bug], citation_score: 0)_ ``` Cross-cutting discoveries can also have labels — add them to the metadata line. @@ -275,6 +286,7 @@ Cross-cutting discoveries can also have labels — add them to the metadata line - `source: user` — human stated it in conversation - `source: interaction` — emerged from collaborative work (debugging, refactoring) - `labels: [...]` — optional, actionable labels (see table above) +- `citation_score: N` — nonnegative integer after optional labels; new entries use 0 and omitted values mean 0 - `Also involves:` — `file::symbol` refs to other code locations (required if discovery touches other files) - `Dream report:` — optional, `_dreams//` link for experiment-derived discoveries - `Category` (cross-cutting only): pattern, behavior, edge-case, contract, performance, intent, warning, history, convention @@ -309,14 +321,14 @@ Links 4 and 5 are bidirectional: if `_cross/db-connection-lifecycle.md` referenc 7. No duplicate discoveries (same behavioral claim at same symbol) To audit a shadow for structural drift (invariant 3 format, invariants 4–5, -plus enum and heading-format guards), the optional core helper also supports: +plus enum and heading-format guards), locate the viewer script and run it: ```bash -READER="" -for DIR in .github/skills/shadow-frog .claude/skills/shadow-frog; do - [ -f "$DIR/shadow-read.py" ] && READER="$DIR/shadow-read.py" && break +VIEWER="" +for DIR in .github/skills/shadow-frog-viewer .claude/skills/shadow-frog-viewer; do + [ -f "$DIR/shadow-viewer.py" ] && VIEWER="$DIR/shadow-viewer.py" && break done -python3 "$READER" --check-invariants +python3 "$VIEWER" --check-invariants ``` Exits 0 if clean, 1 with one violation per line otherwise. Invariant 3 is @@ -372,10 +384,8 @@ types, or static properties. Before writing any discovery, follow this procedure: -1. **Read before write**: Inspect the target `file::symbol` in its shadow and - search for the specific claim. For a large section, use the optional bounded - helper, expanding candidates and following pages when needed. - Do not infer absence from a citation-ranked shortlist. If an existing discovery makes the same behavioral +1. **Read before write**: Read all existing discoveries under the target + `file::symbol`. If an existing discovery makes the same behavioral claim (even if worded differently) → update the existing one. If the new one extends an existing one → merge into a single richer entry. If they conflict → investigate the code, keep the correct one, mark @@ -450,4 +460,4 @@ semantic truth. Approval is planning confidence, not execution proof. - `/shadow-frog-dream` — autonomous exploration and experimentation while user is AFK - `/shadow-frog-nap` — implementation-free, source-grounded feature-task ideation within a work budget - `/shadow-frog-meditate` — deduplicate, merge, and resolve conflicting discoveries -- `/shadow-frog-viewer` — user-facing CLI browsing and lineage visualization +- `/shadow-frog-viewer` — browse and query the shadow (overview, search, preferences, recent) diff --git a/skills/shadow-frog/_citations.py b/skills/shadow-frog/_citations.py index faa3e76..a293901 100644 --- a/skills/shadow-frog/_citations.py +++ b/skills/shadow-frog/_citations.py @@ -1,285 +1,179 @@ -"""Shared local citation scores; Markdown remains the authoritative knowledge store.""" +"""Visible Markdown citation metadata and serialized, exact-entry increments.""" from contextlib import contextmanager -from dataclasses import dataclass -import hashlib -import json import os from pathlib import Path -import random import re -import sqlite3 -import subprocess +import stat +import tempfile import time -import unicodedata -import zlib -PAGE_TTL = 24 * 60 * 60 -RECEIPT_TTL = PAGE_TTL -MAX_RECEIPTS = 100_000 -MAX_PAGES = 32 -MAX_PAGE_BYTES = 8 * 1024 * 1024 -JOURNAL_BYTES = 1024 * 1024 -CHECKPOINT_PAGES = 256 -SCHEMA_VERSION = 2 +DISCOVERY_META_RE = re.compile( + r"_\((\w+),\s*source:\s*(\w+)" + r"(?:,\s*labels:\s*\[([^\]]*)\])?" + r"(?:,\s*citation_score:\s*([0-9]+))?\)_" +) +PREFERENCE_META_RE = re.compile( + r"_\(source:\s*(\w+)(?:,\s*citation_score:\s*([0-9]+))?\)_" +) -def discovery_id(kind, anchor, text, refs=()): - """Content identity excludes status, provenance, labels, and usage metadata.""" - value = [kind, anchor, text.strip(), sorted(set(refs))] - digest = hashlib.sha256( - json.dumps(value, ensure_ascii=False, separators=(",", ":")).encode("utf-8") - ).hexdigest() - return "d_" + digest[:32] +class CitationError(ValueError): + """An entry cannot be safely identified or its score cannot be updated.""" -def canonical_path(path): - """Use actual directory-entry spelling without folding distinct filesystem names.""" - resolved = Path(path).resolve() - current = Path(resolved.anchor) - for part in resolved.parts[1:]: - requested = current / part - if requested.exists(): - key = unicodedata.normalize("NFC", part).casefold() - matches = [] - for child in current.iterdir(): - if child.name == part: - matches = [child] - break - if unicodedata.normalize("NFC", child.name).casefold() == key and child.samefile(requested): - matches.append(child) - if len(matches) != 1: - raise ValueError(f"Cannot identify a unique filesystem path for {requested}") - current = matches[0] - else: - current = requested - return current - - -def validate_event_id(value): - if not isinstance(value, str) or not re.fullmatch(r"[A-Za-z0-9._:-]{1,128}", value): - raise ValueError("event ID must be 1-128 letters, digits, or . _ : -") +def validate_score(value, field="citation_score"): + if type(value) is not int or value < 0: + raise CitationError(f"{field} must be a nonnegative integer") return value -def is_busy_error(exc): - if not isinstance(exc, sqlite3.OperationalError): - return False - code = getattr(exc, "sqlite_errorcode", None) - return ( - (code is not None and code & 0xff in (5, 6)) # SQLITE_BUSY / SQLITE_LOCKED - or (code is None and str(exc) in ("database is locked", "database table is locked")) +def metadata_score(line): + stripped = line.strip() + discovery = DISCOVERY_META_RE.fullmatch(stripped) + preference = PREFERENCE_META_RE.fullmatch(stripped) + if discovery: + return int(discovery.group(4) or 0) + if preference: + return int(preference.group(2) or 0) + raise CitationError("Malformed metadata: use a nonnegative integer citation_score after optional labels") + + +def set_metadata_score(line, score): + """Change only the score field, preserving other text and line endings.""" + validate_score(score) + metadata_score(line) + if "citation_score:" in line: + return re.sub(r"(citation_score:\s*)[0-9]+", lambda match: match.group(1) + str(score), line, count=1) + closing = line.rfind(")_") + return line[:closing] + f", citation_score: {score}" + line[closing:] + + +def _symbol(value): + if value is None: + return None + value = value.strip().strip("`") + return re.sub(r"^(?:class|interface|enum|trait|struct|protocol|module) ", "", value) + + +def _claim(value): + # Join visual line wrapping, but do not collapse whitespace inside literals. + return re.sub(r"[ \t]*\r?\n[ \t]*", " ", value.strip()) + + +def _entries(lines, kind): + section = None + claim = None + for index, line in enumerate(lines): + stripped = line.strip() + heading = re.fullmatch(r"#{2,3}\s+(?:`(.+)`|(File-Level|Cross-References))", stripped) + if heading: + section = _symbol(heading.group(1) or heading.group(2)) + claim = None + if kind == "cross" and stripped.startswith("**Discovery**:"): + claim = [stripped.partition(":")[2].strip()] + elif kind != "cross" and stripped.startswith("- "): + claim = [stripped[2:]] if kind == "preference" or section not in (None, "Cross-References") else None + elif claim is not None: + if stripped.startswith("_("): + metadata_score(line) + yield section if kind == "file" else None, _claim("\n".join(claim)), index + claim = None + elif stripped.startswith(("Also involves:", "Dream report:", "#")): + claim = None + elif stripped: + claim.append(stripped) + + +def _target(path, shadow_dir): + literal = Path(path).absolute() + if shadow_dir is None: + root = next((parent for parent in literal.parents if parent.name == ".shadow"), None) + if root is None: + raise CitationError("Provide a file under .shadow/ or an explicit --shadow-dir") + else: + root = Path(shadow_dir).absolute() + resolved_root = root.resolve() + if resolved_root != root.parent.resolve() / root.name: + raise CitationError("The shadow root is a filesystem alias; use a real shadow directory before recording citations") + resolved = literal.resolve() + if not resolved.is_relative_to(resolved_root) or not resolved.is_file(): + raise CitationError("Citation target must be an existing Markdown file inside the shadow directory") + relative = resolved.relative_to(resolved_root) + if relative.suffix != ".md" or relative.parts[0] in ("_meta", "_dreams", "_index.md"): + raise CitationError("Cite discovery or preference entries, not indexes or dream reports") + kind = "preference" if relative.as_posix() == "_prefs.md" else ( + "cross" if len(relative.parts) == 2 and relative.parts[0] == "_cross" else "file" ) + return resolved, kind -@dataclass(frozen=True) -class CitationStore: - """One short-lived connection per operation; SQLite coordinates processes.""" - - path: Path - scope: str - timeout: float = 0.1 - - def __post_init__(self): - object.__setattr__(self, "path", Path(self.path)) - if not isinstance(self.scope, str) or not self.scope: - raise ValueError("citation scope must be nonempty") - if self.timeout <= 0: - raise ValueError("citation timeout must be positive") - - @classmethod - def for_shadow(cls, shadow_dir): - shadow = canonical_path(shadow_dir) - env = os.environ.copy() - for name in ("GIT_DIR", "GIT_WORK_TREE", "GIT_COMMON_DIR", "GIT_INDEX_FILE"): - env.pop(name, None) - env["LC_ALL"] = "C" - result = subprocess.run( - ["git", "-C", str(shadow), "rev-parse", "--show-toplevel", "--git-common-dir"], - capture_output=True, text=True, encoding="utf-8", timeout=0.2, env=env, - ) - if result.returncode == 0: - root_text, common_text = result.stdout.rstrip("\n").split("\n") - root = canonical_path(root_text) - common = Path(common_text) - if not common.is_absolute(): - common = shadow / common - return cls( - canonical_path(common) / "shadowfrog/citations.sqlite3", - shadow.relative_to(root).as_posix(), - ) - if "not a git repository" not in result.stderr.lower(): - raise ValueError(f"Cannot resolve Git citation storage: {result.stderr.strip()}") - # Standalone shadows use untracked local state, not a sidecar in .shadow/. - base = os.environ.get("LOCALAPPDATA" if os.name == "nt" else "XDG_STATE_HOME") - state = Path(base) if base else Path.home() / ".local/state" - identity = hashlib.sha256(os.fsencode(shadow)).hexdigest() - return cls(state / "shadowfrog/citations" / f"{identity}.sqlite3", str(shadow)) - - @contextmanager - def _connection(self, *, timeout=None): - self.path.parent.mkdir(parents=True, exist_ok=True) - db = sqlite3.connect(self.path, timeout=self.timeout if timeout is None else timeout) +@contextmanager +def _locked(path, timeout): + lock = path.with_name(path.name + ".citation.lock") + deadline = time.monotonic() + timeout + while True: try: - version = db.execute("PRAGMA user_version").fetchone()[0] - if version not in (0, SCHEMA_VERSION): - raise ValueError( - f"Unsupported citation database version {version} at {self.path}; " - "use a matching helper or move this local cache aside to reset scores" - ) - # Local worktrees share WAL so readers do not contend with each - # small score update. FULL still synchronizes successful commits. - mode = db.execute("PRAGMA journal_mode").fetchone()[0] - if mode != "wal": - mode = db.execute("PRAGMA journal_mode=WAL").fetchone()[0] - if mode != "wal": - raise ValueError(f"Cannot enable local WAL citation storage (got {mode})") - db.execute("PRAGMA synchronous=FULL") - db.execute(f"PRAGMA journal_size_limit={JOURNAL_BYTES}") - db.execute(f"PRAGMA wal_autocheckpoint={CHECKPOINT_PAGES}") - if version == 0: - with db: - db.execute("BEGIN IMMEDIATE") - db.execute( - "CREATE TABLE IF NOT EXISTS scores (scope TEXT, id TEXT, " - "citation_score INTEGER NOT NULL CHECK(citation_score >= 0), PRIMARY KEY(scope, id))" - ) - db.execute( - "CREATE TABLE IF NOT EXISTS events (scope TEXT, event TEXT, id TEXT, " - "created REAL NOT NULL, PRIMARY KEY(scope, event, id))" - ) - db.execute("CREATE INDEX IF NOT EXISTS event_expiry ON events(created)") - db.execute( - "CREATE TABLE IF NOT EXISTS pages (token TEXT PRIMARY KEY, scope TEXT, " - "request TEXT, catalog TEXT, ids BLOB, created REAL)" - ) - db.execute(f"PRAGMA user_version={SCHEMA_VERSION}") - yield db - finally: - db.close() - - def _operation(self, action, *, write=False): - """Retry only lock contention; each write attempt is one atomic transaction.""" - deadline = time.monotonic() + self.timeout - while True: - try: - wait = min(0.01, max(0, deadline - time.monotonic())) - with self._connection(timeout=wait) as db: - if write: - with db: - db.execute("BEGIN IMMEDIATE") - return action(db) - return action(db) - except sqlite3.OperationalError as exc: - remaining = deadline - time.monotonic() - if not is_busy_error(exc) or remaining <= 0: - raise - # Avoid letting repeated writers monopolize SQLite's polling slots. - time.sleep(min(remaining, random.uniform(0.001, 0.01))) - - def scores(self, ids): - if not self.path.exists() or not ids: - return {} - ids = list(dict.fromkeys(ids)) - - def read(db): - result = {} - for offset in range(0, len(ids), 400): - chunk = ids[offset:offset + 400] - placeholders = ",".join("?" for _ in chunk) - result.update(db.execute( - f"SELECT id, citation_score FROM scores WHERE scope=? AND id IN ({placeholders})", - [self.scope, *chunk], - )) - return result - return self._operation(read) - - def record(self, ids, event=None): - """Count anonymous reads directly; retain explicit retry receipts for one day.""" - if event is not None: - validate_event_id(event) - ids = list(dict.fromkeys(ids)) - if not ids: - return - - def update(db): - now = time.time() - cutoff = now - RECEIPT_TTL - db.execute( - "DELETE FROM events WHERE rowid IN " - "(SELECT rowid FROM events WHERE created < ? LIMIT 1000)", (cutoff,), - ) - receipt_count = db.execute("SELECT COUNT(*) FROM events").fetchone()[0] if event else 0 - for identity in ids: - if event is not None: - prior = db.execute( - "SELECT created FROM events WHERE scope=? AND event=? AND id=?", - (self.scope, event, identity), - ).fetchone() - if prior and prior[0] >= cutoff: - continue - if prior is None: - if receipt_count >= MAX_RECEIPTS: - raise ValueError("Citation retry receipt capacity reached; wait for expiry or use --no-record") - receipt_count += 1 - db.execute( - "INSERT INTO events VALUES (?, ?, ?, ?) " - "ON CONFLICT(scope, event, id) DO UPDATE SET created=excluded.created", - (self.scope, event, identity, now), - ) - db.execute( - "INSERT INTO scores(scope, id, citation_score) VALUES (?, ?, 1) " - "ON CONFLICT(scope, id) DO UPDATE SET citation_score=citation_score+1", - (self.scope, identity), + descriptor = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600) + break + except FileExistsError: + if time.monotonic() >= deadline: + raise CitationError( + f"Citation writer busy: {lock}. Retry later; after an interruption, " + "confirm its writer stopped before removing only that lock." + ) from None + time.sleep(min(0.02, max(0, deadline - time.monotonic()))) + try: + with os.fdopen(descriptor, "w", encoding="utf-8") as stream: + stream.write(f"pid={os.getpid()}\n") + yield + finally: + lock.unlink() + + +def record_citations(path, texts, *, symbol=None, shadow_dir=None, timeout=2.0): + """Increment each selected claim once; no retrieval, database, or hidden IDs.""" + path, kind = _target(path, shadow_dir) + if kind == "file" and not symbol: + raise CitationError("Per-file citations require --symbol (use File-Level for file-wide knowledge)") + if kind != "file" and symbol is not None: + raise CitationError("Preferences and cross-cutting discoveries do not use --symbol") + if not texts or any(not isinstance(text, str) or not text.strip() for text in texts): + raise CitationError("Supply --text with the exact discovery text already consulted") + targets = list(dict.fromkeys(_claim(text) for text in texts)) + selected_symbol = _symbol(symbol) + with _locked(path, timeout): + original = path.read_bytes() + lines = original.decode("utf-8").splitlines(keepends=True) + entries = list(_entries(lines, kind)) + updates = [] + for text in targets: + matches = [ + index for entry_symbol, entry_text, index in entries + if entry_symbol == selected_symbol and entry_text == text + ] + if len(matches) != 1: + raise CitationError( + f"{path}: expected one matching entry for {symbol or kind} / {text!r}, " + f"found {len(matches)}. Re-read that entry; correct the text or resolve duplicates." ) - self._operation(update, write=True) - - def save_page(self, request, catalog, ids): - request = hashlib.sha256(request.encode("utf-8")).hexdigest() - payload = zlib.compress(json.dumps(ids, separators=(",", ":")).encode("utf-8")) - if len(payload) > MAX_PAGE_BYTES: - raise ValueError("Result snapshot exceeds local pagination capacity; narrow the query") - key = json.dumps([self.scope, request, catalog]).encode("utf-8") + payload - token = hashlib.sha256(key).hexdigest()[:32] - - def publish(db): - now = time.time() - db.execute("DELETE FROM pages WHERE created < ?", (now - PAGE_TTL,)) - db.execute("DELETE FROM pages WHERE token=?", (token,)) - rows = db.execute("SELECT token, length(ids) FROM pages ORDER BY created DESC, token").fetchall() - used = len(payload) - for index, (old_token, size) in enumerate(rows, 1): - used += size - if index >= MAX_PAGES or used > MAX_PAGE_BYTES: - db.execute("DELETE FROM pages WHERE token=?", (old_token,)) - db.execute( - "INSERT INTO pages VALUES (?, ?, ?, ?, ?, ?)", - (token, self.scope, request, catalog, payload, now), - ) - self._operation(publish, write=True) - return token - - def load_page(self, token, request, catalog): - request = hashlib.sha256(request.encode("utf-8")).hexdigest() - if not self.path.exists(): - raise ValueError("Retrieval cursor expired or unavailable; rerun the query without --cursor") - row = self._operation( - lambda db: db.execute( - "SELECT request, catalog, ids, created FROM pages WHERE token=? AND scope=?", - (token, self.scope), - ).fetchone() - ) - if not row or row[3] < time.time() - PAGE_TTL: - raise ValueError("Retrieval cursor expired or unavailable; rerun the query without --cursor") - if row[0] != request: - raise ValueError("Cursor belongs to a different query; reuse its original view and filters") - if row[1] != catalog: - raise ValueError("Shadow knowledge changed; rerun the query without --cursor") + index = matches[0] + before = metadata_score(lines[index]) + lines[index] = set_metadata_score(lines[index], before + 1) + updates.append({"text": text, "before": before, "after": before + 1}) + temporary = None try: - ids = json.loads(zlib.decompress(row[2]).decode("utf-8")) - except (zlib.error, UnicodeError, ValueError, TypeError) as exc: - raise ValueError("Invalid local pagination snapshot; rerun the query without --cursor") from exc - if not isinstance(ids, list) or not all(isinstance(identity, str) for identity in ids): - raise ValueError("Invalid local pagination snapshot; rerun the query without --cursor") - return ids + with tempfile.NamedTemporaryFile(dir=path.parent, prefix=path.name + ".cite-", delete=False) as stream: + temporary = Path(stream.name) + stream.write("".join(lines).encode("utf-8")) + stream.flush() + os.fsync(stream.fileno()) + os.chmod(temporary, stat.S_IMODE(path.stat().st_mode)) + if path.read_bytes() != original: + raise CitationError("Shadow changed during citation update; coordinate writers and retry") + os.replace(temporary, path) + finally: + if temporary is not None and temporary.exists(): + temporary.unlink() + return updates diff --git a/skills/shadow-frog/_knowledge.py b/skills/shadow-frog/_knowledge.py deleted file mode 100644 index 40cb5c8..0000000 --- a/skills/shadow-frog/_knowledge.py +++ /dev/null @@ -1,1613 +0,0 @@ -#!/usr/bin/env python3 -"""Shared Markdown parsing, bounded retrieval, and human-facing shadow views. - -The core shadow-read.py helper and user-facing shadow-viewer.py use this -implementation without changing source-to-shadow paths or discovery syntax. -Parsing and structural inspection alone never increment citation scores. -""" - -from __future__ import annotations - -import argparse -from dataclasses import dataclass -import hashlib -import json -import os -import re -import sys -import sqlite3 -import subprocess -import time -import traceback -import uuid -from collections import defaultdict -from datetime import datetime -from pathlib import Path - - -sys.path.insert(0, str(Path(__file__).resolve().parent)) -_bytecode = sys.dont_write_bytecode -sys.dont_write_bytecode = True -try: - from _citations import ( - CitationStore, RECEIPT_TTL, canonical_path, discovery_id, is_busy_error, - validate_event_id, - ) -except ImportError as exc: - raise SystemExit("[shadow error] Missing citation helper/dependency; reinstall the complete core skill") from exc -finally: - sys.path.pop(0) - sys.dont_write_bytecode = _bytecode - - -_DISCOVERY_META_RE = re.compile( - r"_\((\w+),\s*source:\s*(\w+)" - r"(?:,\s*labels:\s*\[([^\]]*)\])?" - r"\)_" -) - - -def warn(msg): - """Print a warning to stderr. Agents read these to adjust strategy.""" - print(f"[shadow warning] {msg}", file=sys.stderr) - - -def error(msg): - """Print an error to stderr.""" - print(f"[shadow error] {msg}", file=sys.stderr) - - -def find_shadow_dir(start="."): - """Walk up from start to find .shadow/ directory.""" - try: - p = Path(start).resolve() - while p != p.parent: - candidate = p / ".shadow" - if candidate.is_dir(): - return candidate - p = p.parent - except (OSError, PermissionError) as e: - error(f"Failed to search for .shadow/ directory from '{start}': {e}") - return None - - -def parse_discovery(line, continuation_lines=None): - """Parse a discovery bullet and its metadata line(s).""" - try: - text = line[2:].strip() if line.startswith("- ") else line.strip() - except (TypeError, AttributeError) as e: - warn(f"parse_discovery: bad line input ({type(line).__name__}): {e}") - return {"text": str(line) if line else ""} - - meta = {} - full_text = text - - if continuation_lines: - for cl in continuation_lines: - try: - stripped = cl.strip() - # _(status, source: type, labels: [l1, l2])_ or _(status, source: type)_ - m = _DISCOVERY_META_RE.match(stripped) - if m: - meta["status"] = m.group(1) - meta["source"] = m.group(2) - if m.group(3): - meta["labels"] = [ - l.strip() - for l in m.group(3).split(",") - if l.strip() - ] - else: - # _(source: type)_ (preferences format) - m2 = re.match(r"_\(source:\s*(\w+)\)_", stripped) - if m2: - meta["source"] = m2.group(1) - elif stripped.startswith("Also involves:"): - refs = re.findall(r"`([^`]+)`", stripped) - meta["also_involves"] = refs - elif stripped.startswith("Dream report:"): - m_dr = re.search(r"`([^`]+)`", stripped) - if m_dr: - meta["dream_report"] = m_dr.group(1) - else: - full_text += " " + stripped - except Exception as e: - warn(f"parse_discovery: failed parsing continuation line " - f"'{cl[:80]}': {e}") - - return {"text": full_text, **meta} - - -def parse_shadow_file(filepath): - """Parse a per-file shadow into structured data. - - Returns a result dict even on partial failure — whatever was parsed - before the error is preserved. Warnings go to stderr. - """ - result = { - "path": str(filepath), - "source_file": None, - "language": None, - "lines": None, - "last_modified": None, - "symbols": [], - "discoveries": [], - "cross_references": [], - "parse_errors": [], - } - - try: - content = filepath.read_text(encoding="utf-8") - except UnicodeDecodeError as e: - msg = (f"Cannot read {filepath}: encoding error at byte " - f"{e.start}: {e.reason}. File may not be UTF-8.") - warn(msg) - result["parse_errors"].append(msg) - return result - except OSError as e: - msg = f"Cannot read {filepath}: {e}" - warn(msg) - result["parse_errors"].append(msg) - return result - - lines = content.split("\n") - current_symbol = None - i = 0 - - while i < len(lines): - line = lines[i] - - try: - # Header: # Shadow: src/auth.py - if line.startswith("# Shadow: "): - result["source_file"] = line[len("# Shadow: "):].strip() - - # Metadata: **Language**: Python | **Lines**: 142 | ... - elif line.startswith("**Language**"): - parts = line.split("|") - for part in parts: - part = part.strip() - if part.startswith("**Language**"): - m = re.search(r"\*\*:\s*(.+)", part) - if m: - result["language"] = m.group(1).strip() - elif "Lines" in part: - m = re.search(r"(\d+)", part) - if m: - try: - result["lines"] = int(m.group(1)) - except ValueError: - pass - elif "Last modified" in part: - m = re.search(r"\*\*:\s*(.+)", part) - if m: - result["last_modified"] = m.group(1).strip() - - # Symbol heading: ## `symbol_name` or ### `Class.method` - elif re.match(r"^#{2,3}\s", line): - sym_match = re.match(r"^(#{2,3})\s+`(.+?)`", line) - if sym_match: - name = sym_match.group(2) - current_symbol = name - result["symbols"].append(name) - elif "Cross-References" in line: - current_symbol = "__cross_refs__" - elif "File-Level" in line: - current_symbol = "__file_level__" - else: - current_symbol = None - - # Discovery bullet (skip cross-reference links) - elif ( - line.strip().startswith("- ") - and current_symbol - and current_symbol != "__cross_refs__" - ): - # Collect continuation lines - continuation = [] - j = i + 1 - while j < len(lines): - next_line = lines[j] - if ( - next_line.strip() == "" - or next_line.strip().startswith("- ") - or re.match(r"^#{1,3}\s", next_line) - ): - break - continuation.append(next_line) - j += 1 - - disc = parse_discovery(line.strip(), continuation) - disc["symbol"] = ( - "file-level" if current_symbol == "__file_level__" - else current_symbol - ) - disc["file"] = result["source_file"] - result["discoveries"].append(disc) - i = j - continue - - # Cross-reference link - elif ( - current_symbol == "__cross_refs__" - and line.strip().startswith("- ") - ): - link_match = re.search(r"\[(.+?)\]", line) - if link_match: - result["cross_references"].append(link_match.group(1)) - - except Exception as e: - msg = (f"Error parsing {filepath} at line {i + 1}: " - f"{type(e).__name__}: {e}") - warn(msg) - result["parse_errors"].append(msg) - - i += 1 - - return result - - -def parse_prefs(shadow_dir): - """Parse _prefs.md into a list of preferences.""" - prefs_path = shadow_dir / "_prefs.md" - if not prefs_path.exists(): - return [] - - try: - content = prefs_path.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - warn(f"Cannot read preferences file {prefs_path}: {e}") - return [] - - prefs = [] - lines = content.split("\n") - i = 0 - while i < len(lines): - line = lines[i] - try: - if line.strip().startswith("- ") and not line.strip().startswith( - "- [" - ): - continuation = [] - j = i + 1 - while j < len(lines): - next_line = lines[j] - if next_line.strip() == "" or next_line.strip().startswith( - "- " - ): - break - continuation.append(next_line) - j += 1 - - pref = parse_discovery(line.strip(), continuation) - pref["type"] = "preference" - prefs.append(pref) - i = j - continue - except Exception as e: - warn(f"Error parsing preference at line {i + 1} in " - f"{prefs_path}: {e}") - i += 1 - - return prefs - - -def parse_cross_cutting(shadow_dir): - """Parse all _cross/*.md files.""" - cross_dir = shadow_dir / "_cross" - if not cross_dir.exists(): - return [] - - entries = [] - try: - md_files = sorted(cross_dir.glob("*.md")) - except OSError as e: - warn(f"Cannot list cross-cutting directory {cross_dir}: {e}") - return [] - - for f in md_files: - try: - content = f.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - warn(f"Cannot read cross-cutting file {f}: {e}") - continue - - entry = {"slug": f.stem, "file": str(f.name)} - - try: - # Title - m = re.search(r"^# (.+)", content, re.MULTILINE) - if m: - entry["title"] = m.group(1).strip() - - # Category - m = re.search(r"\*\*Category\*\*:\s*(.+)", content) - if m: - entry["category"] = m.group(1).strip() - - # Refs — only within the **Refs**: section, not backticked - # bullets elsewhere in the file (e.g., inside the Discovery body). - refs = [] - refs_block = re.search( - r"\*\*Refs\*\*:\s*\n(.*?)(?=\n[ \t]*\n|\n\*\*|\Z)", - content, - re.DOTALL, - ) - if refs_block: - refs = re.findall(r"-\s*`([^`]+)`", refs_block.group(1)) - entry["refs"] = refs - - # Discovery text - m = re.search( - r"\*\*Discovery\*\*:\s*(.+?)(?=\n\n|\n_\(|\Z)", - content, - re.DOTALL, - ) - if m: - entry["discovery"] = m.group(1).strip() - - # Status/source (with optional labels) - m = _DISCOVERY_META_RE.search(content) - if m: - entry["status"] = m.group(1) - entry["source"] = m.group(2) - if m.group(3): - entry["labels"] = [ - l.strip() - for l in m.group(3).split(",") - if l.strip() - ] - else: - # Fallback to simpler pattern - m2 = re.search( - r"_\((\w+),\s*source:\s*(\w+)\)_", content - ) - if m2: - entry["status"] = m2.group(1) - entry["source"] = m2.group(2) - except Exception as e: - warn(f"Error parsing cross-cutting file {f.name}: " - f"{type(e).__name__}: {e}") - entry.setdefault("title", f.stem) - - entries.append(entry) - - return entries - - -def load_state(shadow_dir): - """Load _meta/state.json.""" - state_path = shadow_dir / "_meta" / "state.json" - if not state_path.exists(): - return {} - try: - content = state_path.read_text(encoding="utf-8") - state = json.loads(content) - if not isinstance(state, dict): - warn(f"state.json is not a JSON object (got {type(state).__name__})") - return {} - return state - except json.JSONDecodeError as e: - warn(f"Invalid JSON in {state_path}: {e}") - return {} - except OSError as e: - warn(f"Cannot read {state_path}: {e}") - return {} - - -def get_all_shadow_files(shadow_dir): - """Get all per-file shadow .md files (excluding special files).""" - special = {"_index.md", "_prefs.md"} - special_dirs = {"_cross", "_meta", "_dreams"} - - results = [] - try: - for f in shadow_dir.rglob("*.md"): - try: - rel = f.relative_to(shadow_dir) - parts = rel.parts - if parts[0] in special_dirs: - continue - if str(rel) in special: - continue - results.append(f) - except (ValueError, IndexError) as e: - warn(f"Skipping file {f}: {e}") - except OSError as e: - warn(f"Error walking shadow directory {shadow_dir}: {e}") - - return sorted(results) - - -def collect_all_discoveries(shadow_dir): - """Parse all shadow files and collect every discovery. - - Continues past individual file failures — reports errors and moves on. - """ - all_disc = [] - failed_files = [] - for sf in get_all_shadow_files(shadow_dir): - try: - parsed = parse_shadow_file(sf) - if parsed.get("parse_errors"): - failed_files.append( - (str(sf), parsed["parse_errors"]) - ) - modified = _mtime(sf) - for d in parsed["discoveries"]: - d.setdefault("file", parsed["source_file"]) - d["shadow_path"] = str(sf.relative_to(shadow_dir)) - d["shadow_mtime"] = modified - all_disc.append(d) - except Exception as e: - msg = f"Failed to parse {sf}: {type(e).__name__}: {e}" - warn(msg) - failed_files.append((str(sf), [msg])) - - if failed_files: - warn(f"{len(failed_files)} file(s) had parse errors " - f"(discoveries from other files still collected)") - - return all_disc - - -# --- View Functions --- - -@dataclass(frozen=True) -class RetrievalOptions: - limit: int = 10 - max_chars: int = 4000 - cursor: str | None = None - event_id: str | None = None - record: bool = True - text_cursor: str | None = None - - def __post_init__(self): - if type(self.limit) is not int or self.limit < 1: - raise ValueError("Result limit must be positive") - if type(self.max_chars) is not int or self.max_chars < 0 or (self.max_chars and self.max_chars < 256): - raise ValueError("Output budget must be at least 256 characters, or 0 for no cap") - for name in ("cursor", "text_cursor"): - value = getattr(self, name) - if value is not None and (not isinstance(value, str) or not value): - raise ValueError(f"{name} must be a nonempty returned cursor") - if self.event_id is not None: - validate_event_id(self.event_id) - - -def _knowledge_entry(kind, file, symbol, text, data, refs=(), mtime=0): - symbol = _canonical_symbol(symbol) - anchor = f"{file}::{symbol}" if symbol else file - return { - "id": discovery_id(kind, anchor, text, refs), - "kind": kind, "file": file, "symbol": symbol, "anchor": anchor, - "text": text, "refs": list(refs), "mtime": mtime, - "status": data.get("status", "verified" if kind == "preference" else "?"), - "source": data.get("source", "?"), "labels": sorted(set(data.get("labels", []))), - "title": data.get("title", ""), "category": data.get("category", "?"), - "dream_report": data.get("dream_report", ""), - } - - -def _canonical_symbol(symbol): - # Container headings use the same prefixes as shadow-init.py::Symbol.heading_text. - if symbol in ("File-Level", "file-level"): - return "file-level" - return re.sub(r"^(?:class|interface|enum|trait|struct|protocol|module) ", "", symbol, count=1) - - -def _canonical_source_file(shadow_dir, source_file): - if ( - not source_file or any(char in source_file for char in (":", "\\", "\0", "\n", "\r")) - or any(part in ("", ".", "..") for part in source_file.split("/")) - ): - raise ValueError("Use a repository-relative source path with forward slashes") - root = canonical_path(shadow_dir) - target = canonical_path(root / (source_file + ".md")) - if not target.is_relative_to(root): - raise ValueError("Requested shadow resolves outside --shadow-dir") - return target.relative_to(root).as_posix()[:-3] - - -def _mtime(path): - try: - return path.stat().st_mtime - except OSError as exc: - warn(f"Modification time unavailable for {path}: {exc}; treating it as undated.") - return 0 - - -def _preference_entries(shadow_dir): - prefs = parse_prefs(shadow_dir) - modified = _mtime(shadow_dir / "_prefs.md") if prefs else 0 - return _unique_entries([ - _knowledge_entry( - "preference", "_prefs.md", "", pref.get("text", ""), pref, - mtime=modified, - ) - for pref in prefs - ]) - - -def _unique_entries(entries): - # Duplicate claims at the same location have one identity and one score. - unique = {} - for entry in entries: - if not entry["text"].strip(): - warn(f"Empty discovery at {entry['anchor']}; repair its text before retrieval.") - continue - prior = unique.get(entry["id"]) - if prior is None: - unique[entry["id"]] = entry - else: - strongest = entry if _trust(entry) < _trust(prior) else prior - unique[entry["id"]] = { - **strongest, "labels": sorted(set(entry["labels"]) | set(prior["labels"])), - } - return list(unique.values()) - - -def _knowledge_entries(shadow_dir, source_file=None): - """Collect identities without recording a citation for parsing or matching.""" - entries = [] - files = {} - - def canonical_file(file): - if file not in files: - files[file] = _canonical_source_file(shadow_dir, file) - return files[file] - - def canonical_refs(refs): - normalized = [] - for ref in refs: - file, separator, symbol = ref.partition("::") - if separator: - try: - ref = f"{canonical_file(file)}::{_canonical_symbol(symbol)}" - except (OSError, ValueError, RuntimeError) as exc: - warn(f"Cannot resolve reference {ref!r}: {exc}; inspect and repair that reference.") - normalized.append(ref) - return sorted(set(normalized)) - - if source_file is None: - discoveries = collect_all_discoveries(shadow_dir) - else: - source_file = canonical_file(source_file) - path = shadow_dir / (source_file + ".md") - parsed = parse_shadow_file(path) if path.is_file() else {"discoveries": []} - modified = _mtime(path) if parsed["discoveries"] else 0 - discoveries = [ - {**disc, "shadow_path": source_file + ".md", "shadow_mtime": modified} - for disc in parsed["discoveries"] - ] - for disc in discoveries: - file = canonical_file(Path(disc["shadow_path"]).as_posix()[:-3]) - entries.append(_knowledge_entry( - "discovery", file, disc.get("symbol", "file-level"), disc.get("text", ""), - disc, canonical_refs(disc.get("also_involves", [])), disc.get("shadow_mtime", 0), - )) - for cross in parse_cross_cutting(shadow_dir): - refs = canonical_refs(cross.get("refs", [])) - if source_file is not None and not any(ref.split("::", 1)[0] == source_file for ref in refs): - continue - relative = "_cross/" + cross["file"] - entries.append(_knowledge_entry( - "cross-cutting", relative, "", cross.get("discovery", cross.get("title", "")), - cross, refs, _mtime(shadow_dir / relative), - )) - if source_file is None: - entries.extend(_preference_entries(shadow_dir)) - return _unique_entries(entries) - - -def _trust(entry): - if entry["status"] == "refuted": - return 5 - if entry["source"] == "user": - return 0 - if entry["source"] == "interaction": - return 1 - return {"verified": 2, "uncertain": 3}.get(entry["status"], 4) - - -def _rank_entries(entries, scores, limit, recent=False): - groups = defaultdict(list) - for entry in entries: - priority = ((-entry["mtime"],) if recent else ()) + ( - entry.get("relevance", 0), _trust(entry), - ) - groups[priority].append(entry) - result = [] - for priority in sorted(groups): - group = groups[priority] - seen = sorted( - (entry for entry in group if scores.get(entry["id"], 0)), - key=lambda entry: -scores[entry["id"]], - ) - unseen = [entry for entry in group if not scores.get(entry["id"], 0)] - width = max(1, limit) - while seen and unseen: - remaining = width - len(result) % width - take = min(len(seen), remaining - 1) - result.extend(seen[:take]) - del seen[:take] - result.append(unseen.pop(0)) - result.extend(seen) - result.extend(unseen) - return result - - -def _citation_store(shadow_dir): - try: - return CitationStore.for_shadow(shadow_dir) - except (OSError, ValueError, RuntimeError, subprocess.SubprocessError) as exc: - warn(f"Citation tracking unavailable: {exc}. This visit will not be recorded; check local Git/state access.") - return None - - -def _clip(text, limit): - if len(text) <= limit: - return text - return text[:limit] if limit < 3 else text[:limit - 3] + "..." - - -def _preview_parts(entry, score, style, group_count): - text = entry["text"].replace("\n", " ") - anchor = _clip(entry["anchor"], 160) - identity = f"id={entry['id']} citation_score={score}" - metadata = f"({entry['status']}, source: {entry['source']})" - labels = ",".join(entry["labels"]) or "-" - if style == "top": - anchor = entry["symbol"] if entry["kind"] == "discovery" else entry["file"] - prefix = f"- [{_clip(labels, 40)}] `{_clip(anchor, 48)}` ({entry['status']}) id={entry['id']}: " - return prefix, text - if style == "recent": - stamp = datetime.fromtimestamp(entry["mtime"]).strftime("%Y-%m-%d %H:%M") - heading = f" [{stamp}] ({entry['kind']})" - elif entry["kind"] == "cross-cutting": - heading = f"Cross-cutting: {_clip(entry['title'], 120)}\n Category: {entry['category']}" - elif entry["kind"] == "preference": - heading = f"Preferences [{entry['source']}]" - else: - heading = f"{_clip(entry['file'], 160)} ({group_count} matches)" - prefix = f"{heading}\n {anchor}\n {metadata} [{labels}]\n {identity}\n" - if entry["refs"]: - label = "Refs" if entry["kind"] == "cross-cutting" else "Also involves" - prefix += f" {label}: {_clip(', '.join(entry['refs']), 160)}\n" - if style == "labels" and len(entry["labels"]) > 1: - prefix += f" Also labeled: {labels}\n" - return prefix + " ", text - - -def _read_scores(store, identities): - if store is not None: - try: - return store.scores(identities), True - except (OSError, ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - warn("Citation ledger busy; scores are unknown. Counting will still be attempted after output if enabled.") - else: - warn(f"Cannot read citation scores: {exc}. Scores are unknown; check the local cache.") - return {}, False - - -def _record_visible(store, identities, options, event=None): - if store is not None and identities and options.record: - try: - store.record(identities, options.event_id if event is None else event) - except (OSError, ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - warn("Citation ledger busy; this visit was not recorded (scores were not updated). Retry later with the same --event-id if supplied.") - else: - warn(f"Citation scores were not updated: {exc}. This visit was not recorded; repair local state and retry.") - - -def _catalog_digest(entries, recent): - values = [ - {key: value for key, value in entry.items() if recent or key != "mtime"} - for entry in sorted(entries, key=lambda entry: entry["id"]) - ] - return hashlib.sha256(json.dumps(values, sort_keys=True, ensure_ascii=False).encode("utf-8")).hexdigest() - - -def _emit_knowledge(shadow_dir, entries, header, options, *, style="search", request=""): - store = _citation_store(shadow_dir) - scores, scores_known = _read_scores(store, [entry["id"] for entry in entries]) - ranked = _rank_entries(entries, scores, options.limit, recent=style == "recent") - catalog = "" if style == "top" else _catalog_digest(entries, recent=style == "recent") - token = None - offset = 0 - if options.cursor: - match = re.fullmatch(r"([0-9a-f]{32}):(\d+)", options.cursor) - if not match: - raise ValueError("Invalid cursor; copy the --cursor value from the previous response") - if store is None: - raise ValueError("Cannot resume cursor without the local ledger; repair it or restart the query") - token, offset = match.group(1), int(match.group(2)) - ids = store.load_page(token, request, catalog) - by_id = {entry["id"]: entry for entry in entries} - try: - ranked = [by_id[identity] for identity in ids] - except KeyError as exc: - raise ValueError("Invalid local pagination snapshot; restart the query") from exc - if offset >= len(ranked): - raise ValueError("Cursor is past the available results; restart the query") - - counts = defaultdict(int) - for entry in entries: - counts[entry["file"]] += 1 - # Include the terminal newline and continuation instructions in the budget. - cap = options.max_chars - prefix = _clip(header, min(300, cap // 5)) if cap else header - reserve = 90 if style != "top" else 6 - width = min(options.limit, len(ranked) - offset) - while width: - selected = ranked[offset:offset + width] - parts = [ - _preview_parts(entry, scores.get(entry["id"], 0) if scores_known else "?", style, counts[entry["file"]]) - for entry in selected - ] - if cap: - used = len(prefix) + reserve + 3 - fits = 0 - for entry_prefix, text in parts: - used += len(entry_prefix) + min(12, len(text)) + 1 - if used > cap: - break - fits += 1 - if fits < width: - if width > 1: - width = max(1, fits) - if not options.cursor: - ranked = _rank_entries(entries, scores, width, recent=style == "recent") - continue - entry = selected[0] - score = scores.get(entry["id"], 0) if scores_known else "?" - minimal = f"({entry['status']}, source: {entry['source']}) id={entry['id']} citation_score={score}: " - parts = [(minimal, entry["text"])] - break - body_budget = cap - len(prefix) - reserve - 1 - width - sum(len(part[0]) for part in parts) if cap else 180 * width - if width and body_budget < width: - raise ValueError("Output budget cannot fit discovery content; increase the character budget") - snippets = [0] * width - # Distribute spare room across entries rather than dropping a whole warning. - pending = list(range(width)) - while pending and body_budget: - share = max(1, body_budget // len(pending)) - next_pending = [] - for index in pending: - remaining = min(180, len(parts[index][1])) - snippets[index] - take = min(remaining, share, body_budget) - snippets[index] += take - body_budget -= take - if snippets[index] < min(180, len(parts[index][1])): - next_pending.append(index) - pending = next_pending - output = prefix + "".join( - "\n" + entry_prefix + _clip(text, snippets[index]) - for index, (entry_prefix, text) in enumerate(parts) - ) - shown = [entry["id"] for entry in selected] - next_offset = offset + len(shown) - if next_offset < len(ranked): - if style == "top": - output += "\n(...)" - else: - if token is None and store is not None: - try: - token = store.save_page(request, catalog, [entry["id"] for entry in ranked]) - except (OSError, ValueError, sqlite3.Error) as exc: - warn(f"Cannot save pagination: {exc}. Repair the local ledger or narrow the query.") - if token: - output += f"\nMore: --cursor {token}:{next_offset} (same view)" - else: - output += "\nMore omitted: narrow the query (local ledger unavailable)." - if style != "top": - output += "\nExpand a claim: --get ID" - output = prefix.replace("{shown}", str(len(shown))) + output[len(prefix):] - if cap and len(output) + 1 > cap: - raise ValueError("Output budget cannot fit retrieval metadata; increase --max-chars") - print(output, flush=True) - _record_visible(store, shown, options) - - -def view_get(shadow_dir, identity, *, options=None): - """Expand a current discovery, chunking long text without changing its ID.""" - options = options or RetrievalOptions() - if not re.fullmatch(r"d_[0-9a-f]{32}", identity): - raise ValueError("--get requires the complete id=d_... value from a retrieval result") - entry = next((entry for entry in _knowledge_entries(shadow_dir) if entry["id"] == identity), None) - if entry is None: - raise ValueError("Discovery ID is absent or its claim changed; search again for its current ID") - store = _citation_store(shadow_dir) - scores, scores_known = _read_scores(store, [identity]) - score = scores.get(identity, 0) if scores_known else "?" - body = entry["anchor"] + "\n\n" + entry["text"] - if entry["labels"]: - body += "\nLabels: " + ", ".join(entry["labels"]) - if entry["kind"] == "cross-cutting": - body += "\nCategory: " + entry["category"] + "\nTitle: " + entry["title"] - if entry["refs"]: - body += "\nRefs: " + ", ".join(entry["refs"]) - if entry["dream_report"]: - body += "\nDream report: " + entry["dream_report"] - revision = hashlib.sha256(json.dumps( - [entry["status"], entry["source"], body], ensure_ascii=False, - ).encode("utf-8")).hexdigest()[:32] - start = 0 - event = options.event_id - expires = int(time.time()) + RECEIPT_TTL - record = options.record - if options.text_cursor: - match = re.fullmatch( - r"([0-9a-f]{32})~(\d{1,12})~([01])~([A-Za-z0-9._:-]{1,128})~(\d+)", - options.text_cursor, - ) - if not match: - raise ValueError("Invalid text cursor; copy the --text-cursor value from the previous expansion") - prior_revision, expiry, recording, prior_event, offset = match.groups() - if prior_revision != revision: - raise ValueError("Discovery body or metadata changed; restart --get without --text-cursor") - if int(expiry) <= time.time(): - raise ValueError("Text cursor expired; restart --get without --text-cursor") - if event is not None and event != prior_event: - raise ValueError("Text cursor has a different --event-id; reuse its original event") - start, expires, event = int(offset), int(expiry), prior_event - record = record and recording == "1" - if start >= len(body): - raise ValueError("Text cursor is past the end of this discovery; restart --get") - header = ( - f"id={identity} citation_score={score}\n" - f"({entry['status']}, source: {entry['source']})\n" - ) - tail = "" - available = len(body) - start - if options.max_chars and len(header) + available + 1 > options.max_chars: - event = event or uuid.uuid4().hex - - def continuation(offset): - token = f"{revision}~{expires}~{int(record)}~{event}~{offset}" - value = f"\nContinue: --get {identity} --text-cursor {token}" - if options.max_chars != 4000: - value += f" --max-chars {options.max_chars}" - return value - - available = options.max_chars - len(header) - len(continuation(len(body))) - 1 - if available < 1: - raise ValueError("Output budget cannot fit continuation metadata; increase --max-chars") - tail = continuation(start + available) - output = header + body[start:start + available] + tail - print(output, flush=True) - if record: - _record_visible(store, [identity], options, event) - - -def view_summary(shadow_dir): - """Overview + detailed statistics. - - Each section is independently wrapped — if label stats fail, you - still get counts and the per-file table. - """ - # Load data (each can fail independently) - state = {} - shadow_files = [] - prefs = [] - cross = [] - - try: - state = load_state(shadow_dir) - except Exception as e: - warn(f"Failed to load state.json: {e}") - - try: - shadow_files = get_all_shadow_files(shadow_dir) - except Exception as e: - warn(f"Failed to list shadow files: {e}") - - try: - prefs = parse_prefs(shadow_dir) - except Exception as e: - warn(f"Failed to parse preferences: {e}") - - try: - cross = parse_cross_cutting(shadow_dir) - except Exception as e: - warn(f"Failed to parse cross-cutting discoveries: {e}") - - # Parse each file once for both stats and discoveries - file_stats = [] - all_disc = [] - total_symbols = 0 - for sf in shadow_files: - try: - parsed = parse_shadow_file(sf) - src = parsed["source_file"] or str(sf.relative_to(shadow_dir)) - n_sym = len(parsed["symbols"]) - n_disc = len(parsed["discoveries"]) - total_symbols += n_sym - file_stats.append((src, n_sym, n_disc)) - for d in parsed["discoveries"]: - d.setdefault("file", parsed["source_file"]) - d["shadow_path"] = str(sf.relative_to(shadow_dir)) - all_disc.append(d) - except Exception as e: - warn(f"Failed to process {sf}: {e}") - file_stats.sort(key=lambda x: x[2], reverse=True) - - # Header counts (always shown) - print("Shadow Knowledge Base Summary") - print("=" * 50) - print(f" Files shadowed: {len(shadow_files)}") - print(f" Symbols tracked: {total_symbols}") - print(f" Discoveries: {len(all_disc)}") - print(f" Preferences: {len(prefs)}") - print(f" Cross-cutting: {len(cross)}") - - # Source breakdown - try: - source_counts = defaultdict(int) - status_counts = defaultdict(int) - for d in all_disc: - source_counts[d.get("source", "unknown")] += 1 - status_counts[d.get("status", "unknown")] += 1 - - if source_counts: - print("\nBy source:") - for src, cnt in sorted(source_counts.items(), key=lambda x: -x[1]): - pct = cnt / len(all_disc) * 100 if all_disc else 0 - bar = "#" * int(pct / 2) - print(f" {src:15s} {cnt:4d} ({pct:5.1f}%) {bar}") - - if status_counts: - print("\nBy status:") - for st, cnt in sorted(status_counts.items(), key=lambda x: -x[1]): - pct = cnt / len(all_disc) * 100 if all_disc else 0 - bar = "#" * int(pct / 2) - print(f" {st:15s} {cnt:4d} ({pct:5.1f}%) {bar}") - except Exception as e: - warn(f"Failed to compute source/status breakdown: {e}") - - # Label breakdown - try: - label_counts = defaultdict(int) - for d in all_disc: - for lbl in d.get("labels", []): - label_counts[lbl] += 1 - if label_counts: - print("\nBy label:") - for lbl, cnt in sorted(label_counts.items(), key=lambda x: -x[1]): - print(f" {lbl:15s} {cnt:4d}") - except Exception as e: - warn(f"Failed to compute label breakdown: {e}") - - # Per-file table - try: - if file_stats: - print(f"\n{'File':<40s} {'Symbols':>8s} {'Disc.':>6s}") - print(f"{'-'*40} {'-'*8} {'-'*6}") - for src, n_sym, n_disc in file_stats[:20]: - print(f"{src:<40s} {n_sym:>8d} {n_disc:>6d}") - if len(file_stats) > 20: - print(f"... and {len(file_stats) - 20} more files") - except Exception as e: - warn(f"Failed to render per-file table: {e}") - - # Cross-cutting titles - try: - if cross: - print(f"\nCross-cutting discoveries:") - for e in cross: - title = e.get("title", e.get("slug", "?")) - cat = e.get("category", "?") - print(f" [{cat}] {title}") - except Exception as e: - warn(f"Failed to render cross-cutting list: {e}") - - # State info - try: - if state: - print(f"\nLast update: {state.get('last_update_at', '?')} " - f"({state.get('last_update_type', '?')})") - print(f"Last commit: {state.get('last_commit', '?')}") - except Exception as e: - warn(f"Failed to render state info: {e}") - - -def view_search(shadow_dir, query, *, options=None): - """Bounded search with stable continuation over matching knowledge identities.""" - options = options or RetrievalOptions() - query_lower = query.lower() - if not query.strip(): - raise ValueError("Search query must be nonempty") - matches = [] - for entry in _knowledge_entries(shadow_dir): - fields = [entry["anchor"], entry["file"], entry["symbol"], entry["text"], - entry["title"], *entry["refs"]] - if any(query_lower in field.lower() for field in fields): - entry["relevance"] = 0 if query_lower in [field.lower() for field in fields[:3]] else 1 - matches.append(entry) - if not matches and not options.cursor: - print(_clip(f"No results for '{query}'.", options.max_chars - 1) if options.max_chars - else f"No results for '{query}'.") - return - _emit_knowledge( - shadow_dir, matches, f"Search: '{query}' ({len(matches)} results)", options, - request=json.dumps(["search", query_lower]), - ) - - -def view_symbol(shadow_dir, anchor, *, options=None): - options = options or RetrievalOptions() - file, separator, symbol = anchor.partition("::") - if not separator or not symbol: - raise ValueError("--symbol requires file::symbol (use File-Level for a file-level section)") - file = _canonical_source_file(shadow_dir, file) - symbol = _canonical_symbol(symbol) - canonical = f"{file}::{symbol}" - matches = [ - entry for entry in _knowledge_entries(shadow_dir, file) - if entry["anchor"] == canonical or canonical in entry["refs"] - ] - if not matches and not options.cursor: - print(_clip(f"No knowledge for '{anchor}'.", options.max_chars - 1) - if options.max_chars else f"No knowledge for '{anchor}'.") - return - _emit_knowledge( - shadow_dir, matches, f"Knowledge for {anchor} ({len(matches)} results)", options, - request=json.dumps(["symbol", canonical]), - ) - - -def view_file(shadow_dir, source_file, *, options=None): - """Read one known source file's shadow, including file-level and cross refs.""" - options = options or RetrievalOptions() - source_file = _canonical_source_file(shadow_dir, source_file) - entries = _knowledge_entries(shadow_dir, source_file) - if not entries and not options.cursor: - message = f"No knowledge for '{source_file}'." - print(_clip(message, options.max_chars - 1) if options.max_chars else message) - return - _emit_knowledge( - shadow_dir, entries, f"Knowledge for {source_file} ({len(entries)} results)", - options, request=json.dumps(["file", source_file]), - ) - - -def view_prefs(shadow_dir, *, options=None): - """Page preferences without letting popular code discoveries hide directives.""" - options = options or RetrievalOptions() - prefs = _preference_entries(shadow_dir) - if not prefs and not options.cursor: - print("No preferences recorded yet.") - return - _emit_knowledge(shadow_dir, prefs, f"Project Preferences ({len(prefs)} total)", options, request="prefs") - - -def view_labels(shadow_dir, label_filter, *, options=None): - """Show discoveries filtered by label(s). - - label_filter can be a single label or comma-separated list. - """ - options = options or RetrievalOptions() - filters = [label.strip().lower() for label in label_filter.split(",") if label.strip()] - if not filters: - raise ValueError("Supply at least one label with --labels") - matching = [ - entry for entry in _knowledge_entries(shadow_dir) - if set(filters) & {label.lower() for label in entry["labels"]} - ] - - if not matching and not options.cursor: - message = f"No discoveries with label(s): {', '.join(filters)}" - print(_clip(message, options.max_chars - 1) if options.max_chars else message) - return - - _emit_knowledge( - shadow_dir, matching, - f"Discoveries with label(s): {', '.join(filters)} ({len(matching)} results)", - options, style="labels", request=json.dumps(["labels", sorted(set(filters))]), - ) - - -def view_recent(shadow_dir, count=10, *, options=None): - """Show the N most recent discoveries (by shadow file mtime). - - Collects all discoveries across all shadow files, cross-cutting entries, - and preferences, sorts by the source file's modification time (most recent - first), and shows the actual discovery content. - Each data source is independent — if cross-cutting fails, per-file - discoveries still appear. - """ - options = options or RetrievalOptions(limit=count) - all_items = _knowledge_entries(shadow_dir) - if not all_items and not options.cursor: - print("No discoveries found.") - return - - _emit_knowledge( - shadow_dir, all_items, f"Most Recent Discoveries (top {count})", - options, style="recent", request="recent", - ) - - -def view_top(shadow_dir, file_path, labels_filter, limit, max_chars, *, options=None): - """Show the top N actionable discoveries for a single source file. - - Designed for the preToolUse hook: concise output suitable for - inlining into additionalContext when the agent is about to mutate a - file. Pulls from both the per-file shadow and any _cross/ entries - whose refs touch this file. - - Trust/status precedes citation score. Output, including the final newline, - is hard-capped; only entries actually emitted are counted. - """ - norm = file_path.strip() - if norm.startswith("./"): - norm = norm[2:] - label_set = {l.strip().lower() for l in labels_filter.split(",") if l.strip()} - options = options or RetrievalOptions(limit=limit, max_chars=max_chars) - candidates = [ - entry for entry in _knowledge_entries(shadow_dir, norm) - if not label_set or label_set & {label.lower() for label in entry["labels"]} - ] - - if not candidates: - labels_disp = ",".join(sorted(label_set)) if label_set else "any" - message = f"No actionable discoveries ({labels_disp}) for {norm}." - print(_clip(message, max_chars - 1) if max_chars else message) - return - _emit_knowledge( - shadow_dir, candidates, - f"Top {{shown}} of {len(candidates)} actionable discoveries for {norm}:", - options, style="top", request=json.dumps(["top", norm, sorted(label_set)]), - ) - - -def view_check_invariants(shadow_dir): - """Walk the shadow knowledge base and report invariant violations. - - Statically-checkable invariants from shadow-frog/SKILL.md: - #3 (partial) Per-file 'Also involves:' uses file::symbol notation - #4 Cross-ref back-pointers match: _cross/.md refs <-> - per-file ## Cross-References - #5 Every ## Cross-References entry has a matching _cross/*.md - - Plus syntactic guards that catch the most common drift: - - Symbol headings use the required backtick form - - Discovery metadata uses valid status enum - - Discovery metadata uses valid source enum - - Discovery labels are from the allowed set - - _cross/ Category field uses a known value - - Invariants #1, #2, #7 are NOT checked (would require source parsing - and semantic match); #6 is filesystem-enforced. - - Exit 0 = clean, 1 = at least one violation. Violations print one per - line in `path:line: kind: message` form so grep/editors can navigate. - """ - VALID_STATUS = {"verified", "uncertain", "refuted"} - VALID_SOURCE = {"exploration", "user", "interaction"} - VALID_LABELS = {"bug", "performance", "security", - "feature-gap", "tech-debt"} - VALID_CATEGORIES = { - "pattern", "behavior", "edge-case", "contract", - "performance", "intent", "warning", "history", "convention", - } - - violations = [] - def v(path, line, kind, msg): - violations.append(f"{path}:{line}: {kind}: {msg}") - - # Pass 1: walk per-file shadows -> collect cross-reference entries - # they declare and validate their internal format. - per_file_xref_targets = {} # rel_shadow_path -> set(slug declared) - cross_dir = shadow_dir / "_cross" - cross_slugs_on_disk = set() - if cross_dir.is_dir(): - try: - cross_slugs_on_disk = {f.stem for f in cross_dir.glob("*.md")} - except OSError as e: - warn(f"Cannot list {cross_dir}: {e}") - - md_heading_re = re.compile(r"^(#{2,3})\s+(.*)$") - backtick_heading_re = re.compile(r"^(#{2,3})\s+`[^`]+`\s*$") - also_involves_re = re.compile(r"^\s*Also involves:\s*(.+)$", re.I) - file_sym_re = re.compile(r"`([^`]+::[^`]+)`") - - for shadow_path in get_all_shadow_files(shadow_dir): - try: - rel = shadow_path.relative_to(shadow_dir) - except ValueError: - continue - try: - text = shadow_path.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - v(rel, 0, "unreadable", str(e)) - continue - - in_cross_refs = False - declared = set() - for ln, raw in enumerate(text.split("\n"), 1): - line = raw.rstrip() - - heading = md_heading_re.match(line) - if heading: - title = heading.group(2).strip() - if title.lower().startswith("cross-references"): - in_cross_refs = True - continue - in_cross_refs = False - # Skip special headings ("File-Level Notes", "Notes", etc.) - if ( - title.lower().startswith("file-level") - or title.lower() in {"notes", "metadata"} - ): - continue - # Symbol heading must use backtick form - if not backtick_heading_re.match(line): - v(rel, ln, "heading", - f"symbol heading must be `## `name`` or " - f"`### `Class.name``; got: {line[:80]}") - continue - - if in_cross_refs and line.strip().startswith("- "): - # Format: - [slug](.shadow/_cross/slug.md) — title - slug_match = re.search( - r"_cross/([^)\s]+?)\.md", line - ) - if slug_match: - declared.add(slug_match.group(1)) - else: - # Looser fallback: bare slug in brackets - alt = re.search(r"\[([^\]]+)\]", line) - if alt: - declared.add(alt.group(1).strip()) - - # Discovery metadata line - md = _DISCOVERY_META_RE.search(line) - if md: - status, source = md.group(1), md.group(2) - labels_raw = md.group(3) or "" - if status not in VALID_STATUS: - v(rel, ln, "enum", - f"status '{status}' not in {sorted(VALID_STATUS)}") - if source not in VALID_SOURCE: - v(rel, ln, "enum", - f"source '{source}' not in {sorted(VALID_SOURCE)}") - for lbl in (l.strip() for l in labels_raw.split(",") if l.strip()): - if lbl not in VALID_LABELS: - v(rel, ln, "enum", - f"label '{lbl}' not in {sorted(VALID_LABELS)}") - - # `Also involves:` must list file::symbol anchors in backticks - ai = also_involves_re.match(line) - if ai: - rest = ai.group(1) - anchors = file_sym_re.findall(rest) - if not anchors: - v(rel, ln, "anchor", - "Also involves: needs `file::symbol` " - "backtick anchors") - # Light sanity: every anchor has both file and symbol - for a in anchors: - if "::" not in a or not a.split("::", 1)[1].strip(): - v(rel, ln, "anchor", - f"anchor '{a}' missing symbol after ::") - - per_file_xref_targets[str(rel)] = declared - - # Invariant #5: every declared cross slug must exist on disk - for slug in declared: - if slug not in cross_slugs_on_disk: - v(rel, 0, "cross-ref", - f"references _cross/{slug}.md but file does not exist") - - # Pass 2: walk _cross/*.md -> validate refs format + back-pointer. - # Build the reverse map: cross_slug -> set(file::symbol it points at). - cross_back = {} # slug -> set(file paths it should be linked from) - if cross_dir.is_dir(): - for cf in sorted(cross_dir.glob("*.md")): - slug = cf.stem - try: - text = cf.read_text(encoding="utf-8") - except (OSError, UnicodeDecodeError) as e: - v(cf.relative_to(shadow_dir), 0, "unreadable", str(e)) - continue - - rel_cf = cf.relative_to(shadow_dir) - - # Category enum check - cat_m = re.search(r"\*\*Category\*\*:\s*(.+)", text) - if cat_m: - cat = cat_m.group(1).strip().lower() - if cat not in VALID_CATEGORIES: - v(rel_cf, 0, "enum", - f"Category '{cat}' not in {sorted(VALID_CATEGORIES)}") - else: - v(rel_cf, 0, "schema", - "missing **Category**: field") - - # Discovery metadata - md = _DISCOVERY_META_RE.search(text) - if md: - status, source = md.group(1), md.group(2) - if status not in VALID_STATUS: - v(rel_cf, 0, "enum", - f"status '{status}' not in {sorted(VALID_STATUS)}") - if source not in VALID_SOURCE: - v(rel_cf, 0, "enum", - f"source '{source}' not in {sorted(VALID_SOURCE)}") - else: - v(rel_cf, 0, "schema", - "missing trailing _(status, source: ...)_ metadata") - - # Refs must be `file::symbol` anchors - refs_block = re.search( - r"\*\*Refs\*\*:\s*\n((?:\s*-\s+`[^`]+`\s*\n?)+)", - text, - ) - if not refs_block: - v(rel_cf, 0, "schema", - "missing **Refs**: block (one per line, " - "`- `file::symbol``)") - else: - anchors = file_sym_re.findall(refs_block.group(1)) - if not anchors: - v(rel_cf, 0, "anchor", - "Refs block has no `file::symbol` entries") - for a in anchors: - if "::" not in a or not a.split("::", 1)[1].strip(): - v(rel_cf, 0, "anchor", - f"ref '{a}' missing symbol after ::") - else: - # Convert file part to shadow path: - # src/foo.py -> src/foo.py.md (relative to shadow_dir) - file_part = a.split("::", 1)[0].strip() - shadow_rel = f"{file_part}.md" - cross_back.setdefault(slug, set()).add(shadow_rel) - - # Invariant #4 back-pointer: every file referenced by a cross slug - # must declare that slug in its ## Cross-References. - for slug, expected_files in cross_back.items(): - for shadow_rel in expected_files: - declared = per_file_xref_targets.get(shadow_rel) - if declared is None: - v(f"_cross/{slug}.md", 0, "cross-ref", - f"refs {shadow_rel} but no such shadow file exists") - elif slug not in declared: - v(f"_cross/{slug}.md", 0, "cross-ref", - f"refs {shadow_rel} but that shadow's ## " - f"Cross-References does not link back to " - f"_cross/{slug}.md") - - # Output - if not violations: - print(f"✓ Invariants OK ({len(per_file_xref_targets)} per-file " - f"shadows, {len(cross_slugs_on_disk)} cross-cutting " - f"discoveries)") - return 0 - - for line in violations: - print(line) - print(f"\n{len(violations)} invariant violation(s) found.", - file=sys.stderr) - return 1 - - -def main(*, agent=False): - for _stream in (sys.stdout, sys.stderr): - if hasattr(_stream, "reconfigure"): - _stream.reconfigure(encoding="utf-8") - try: - parser = argparse.ArgumentParser( - description=( - "Optional bounded agent retrieval. Navigate directly to shadow files first; " - "use this helper for large sections or targeted search." - if agent else - "Browse and visualize a .shadow/ knowledge base for users." - ), - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - if agent: - parser.add_argument( - "target", nargs="?", metavar="FILE[::SYMBOL]", - help="Known source file or file::symbol; its shadow path is determined directly", - ) - - # Views (mutually exclusive) - views = parser.add_mutually_exclusive_group() - views.add_argument( - "--summary", action="store_true", - help="Overview + detailed statistics (default)", - ) - views.add_argument( - "--search", metavar="QUERY", - help="Universal search: files, symbols, and discovery text", - ) - views.add_argument("--file", metavar="FILE", help="Read a known file's shadow without a repository-wide search") - views.add_argument("--symbol", metavar="FILE::SYMBOL", help="Knowledge for an exact symbol and its cross-cutting refs") - views.add_argument("--get", metavar="ID", help="Expand a discovery returned by a bounded read") - views.add_argument( - "--prefs", action="store_true", - help="Show project-wide preferences", - ) - views.add_argument( - "--recent", nargs="?", const=10, type=int, metavar="N", - help="N most recent discoveries with content (default: 10)", - ) - views.add_argument( - "--labels", metavar="LABEL", - help=( - "Show discoveries by label " - "(e.g., bug, security, bug,performance)" - ), - ) - views.add_argument( - "--top", metavar="FILE", - help=( - "Top actionable discoveries for FILE (a source path " - "like src/auth.py). Concise output for the preToolUse " - "hook: filters to actionable labels (default: " - "bug,security), includes both per-file and _cross/ " - "entries that reference FILE, ranks verified first." - ), - ) - views.add_argument( - "--check-invariants", action="store_true", - help=( - "Walk the shadow and report structural violations: " - "missing back-pointers, dangling _cross/ refs, invalid " - "enums, bad heading format. Exit 1 if any are found." - ), - ) - - # Options - parser.add_argument( - "--shadow-dir", default=None, - help="Path to .shadow/ directory (default: auto-detect)", - ) - parser.add_argument( - "--top-labels", default="bug,security", metavar="LABELS", - help=( - "Comma-separated labels to include in --top " - "(default: bug,security). Pass empty string to include " - "all labeled discoveries." - ), - ) - parser.add_argument( - "--top-limit", type=int, default=3, metavar="N", - help="Max discoveries to show in --top (default: 3)", - ) - parser.add_argument( - "--top-max-chars", type=int, default=600, metavar="N", - help=( - "Hard cap on --top total output length " - "(default: 600). Use 0 for no cap." - ), - ) - parser.add_argument("--limit", type=int, help="Results per page for file/search/symbol/labels/prefs (default: 10)") - parser.add_argument("--max-chars", type=int, help="Retrieval output budget (default: 4000; 0 disables cap)") - parser.add_argument("--cursor", help="Continue the same view/filters with a frozen result ordering") - parser.add_argument("--event-id", help="Retry token; an entry counts once per token (default: fresh event)") - parser.add_argument("--no-record", action="store_true", help="Do not increment citation scores") - parser.add_argument("--text-cursor", help="Continue the same logical read of an unchanged --get body") - - args = parser.parse_args() - selected_view = ( - args.summary or args.check_invariants or args.top is not None - or args.search is not None or args.file is not None or args.symbol is not None - or args.get is not None or args.prefs or args.labels is not None or args.recent is not None - ) - if agent and args.target is not None: - if selected_view: - parser.error("Use a positional target or an explicit view, not both") - if "::" in args.target: - args.symbol = args.target - else: - args.file = args.target - elif agent and not selected_view: - parser.error("Provide a known FILE[::SYMBOL] or an explicit retrieval operation such as --search") - retrieval = ( - args.top is not None or args.search is not None or args.file is not None or args.symbol is not None - or args.get is not None or args.prefs or args.labels is not None or args.recent is not None - ) - if not retrieval and ( - args.limit is not None or args.max_chars is not None or args.cursor - or args.event_id is not None or args.no_record or args.text_cursor is not None - ): - parser.error("Retrieval options require --search, --file, --symbol, --get, --top, --prefs, --labels, or --recent") - if args.text_cursor is not None and args.get is None: - parser.error("--text-cursor requires --get") - if args.cursor and (args.get is not None or args.top is not None): - parser.error("--cursor is for paged file/search/symbol/labels/prefs/recent; use --text-cursor with --get") - if args.limit is not None and (args.top is not None or args.recent is not None or args.get is not None): - parser.error("Use --top-limit or --recent N instead of --limit; --get returns one discovery") - if args.max_chars is not None and args.top is not None: - parser.error("Use --top-max-chars with --top") - try: - options = RetrievalOptions( - limit=args.top_limit if args.top is not None else ( - args.recent if args.recent is not None else (args.limit if args.limit is not None else 10) - ), - max_chars=args.top_max_chars if args.top is not None else ( - args.max_chars if args.max_chars is not None else 4000 - ), - cursor=args.cursor, event_id=args.event_id, record=not args.no_record, - text_cursor=args.text_cursor, - ) - except ValueError as exc: - parser.error(str(exc)) - - # Find shadow dir - if args.shadow_dir: - shadow_dir = Path(args.shadow_dir) - else: - shadow_dir = find_shadow_dir() - - if not shadow_dir or not shadow_dir.is_dir(): - cwd = os.getcwd() - error( - f"No .shadow/ directory found. " - f"Searched from: {cwd}\n" - f"[shadow error] " - f"Run /shadow-frog-init first to create the shadow, " - f"or pass --shadow-dir /path/to/.shadow/ explicitly." - ) - if args.shadow_dir: - error( - f"Provided --shadow-dir '{args.shadow_dir}' does not " - f"exist or is not a directory." - ) - sys.exit(1) - - # Dispatch - if args.check_invariants: - sys.exit(view_check_invariants(shadow_dir)) - if args.top is not None: - view_top( - shadow_dir, - args.top, - args.top_labels, - args.top_limit, - args.top_max_chars, - options=options, - ) - elif args.search is not None: - view_search(shadow_dir, args.search, options=options) - elif args.file is not None: - view_file(shadow_dir, args.file, options=options) - elif args.symbol is not None: - view_symbol(shadow_dir, args.symbol, options=options) - elif args.get is not None: - view_get(shadow_dir, args.get, options=options) - elif args.prefs: - view_prefs(shadow_dir, options=options) - elif args.labels is not None: - view_labels(shadow_dir, args.labels, options=options) - elif args.recent is not None: - view_recent(shadow_dir, args.recent, options=options) - else: - view_summary(shadow_dir) - - except SystemExit: - raise - except KeyboardInterrupt: - error("Interrupted by user.") - sys.exit(130) - except (ValueError, sqlite3.Error) as exc: - if is_busy_error(exc): - error("Local citation ledger is busy; retry the same command later. Do not reset a busy database.") - else: - error(f"{exc}. Correct the request or repair the local citation ledger, then retry.") - sys.exit(1) - except Exception as e: - error( - f"Unexpected error: {type(e).__name__}: {e}\n" - f"[shadow error] Full traceback:\n" - f"{traceback.format_exc()}" - f"This is likely a bug in the shadow knowledge helper. " - f"The shadow data may be in an unexpected format. " - f"Try running with --shadow-dir to confirm the path, " - f"or inspect the .shadow/ files manually." - ) - sys.exit(1) diff --git a/skills/shadow-frog/retrieval.md b/skills/shadow-frog/retrieval.md deleted file mode 100644 index 9864248..0000000 --- a/skills/shadow-frog/retrieval.md +++ /dev/null @@ -1,108 +0,0 @@ -# Optional Agent Retrieval Reference - -Direct `.shadow/.md` reads and symbol navigation remain the default. -Use the core `shadow-read.py` when a known section is too large for context or -when a targeted search is useful. It supports Python 3.9+ and does not require -the user-facing Viewer. Neither tool changes the canonical Markdown format. - -```text -python .github/skills/shadow-frog/shadow-read.py src/auth.py -python .github/skills/shadow-frog/shadow-read.py src/auth.py::UserAuth.validate --limit 5 -python .github/skills/shadow-frog/shadow-read.py --search "token expiry" -python .github/skills/shadow-frog/shadow-read.py --get DISCOVERY_ID -``` - -Use `.claude/skills/` for Claude Code. Replace `DISCOVERY_ID` and cursor values -with values returned by the helper. A known file or `file::symbol` can be passed -positionally; the equivalent explicit selectors are `--file` and `--symbol`. -Do not combine a positional target with another view. - -## Views and Options - -| View | Behavior | -|------|----------| -| `--file FILE` | Bounded per-file knowledge, including file-level sections and related cross-cutting entries; does not search unrelated per-file shadows | -| `--symbol FILE::SYMBOL` | Exact symbol and cross-cutting refs; `File-Level` selects its file-level section | -| `--search QUERY` | Search paths, symbols, claim text, preferences, and cross-cutting entries | -| `--get ID` | Expand one current entry; long content returns a revision-bound continuation | -| `--prefs` | Preferences; follow every page when relying on the full set of directives | -| `--labels LABELS` | Comma-separated actionable label filter | -| `--recent [N]` | Most recent previews by source-shadow mtime (default: 10 per page) | -| `--top FILE` | Compact actionable hints (default labels: `bug,security`, up to 3 entries / 600 characters); not exhaustive or pageable | -| `--check-invariants` | Structural audit; no citation recording | -| `--summary` | Statistics/overview; no citation recording | - -| Option | Contract | -|--------|----------| -| `--shadow-dir DIR` | Override the shadow root; otherwise locate it from the working directory | -| `--limit N` | Positive page size for file/search/symbol/labels/preferences (default: 10) | -| `--max-chars N` | Output cap including metadata/newline (default: 4000; minimum 256, or 0 for explicit uncapped output) | -| `--cursor TOKEN` | Continue the same view and filters with its frozen result ordering | -| `--text-cursor TOKEN` | Continue the same unchanged `--get` body as one logical read | -| `--event-id ID` | Retry ID: each entry counts once per ID within 24 hours; 1-128 letters/digits or `. _ : -` | -| `--no-record` | Do not increment scores; existing scores still rank results, and pagination can store a local snapshot | -| `--top-labels LABELS` | Labels for `--top`; an empty string removes its label filter | -| `--top-limit N` | Positive result limit for `--top` (default: 3) | -| `--top-max-chars N` | Cap for `--top` instead of `--max-chars` (default: 600; minimum 256, or 0 for no cap) | - -For a truncated result set, repeat the same view with the returned `--cursor`. -For a long expanded claim, copy its `--get ... --text-cursor ...` continuation. -Character limits constrain output, not token counts or local parsing work. -Errors use stderr and exit 1 (invalid arguments: 2). Optional telemetry warnings -do not hide knowledge; raw reads remain available independently. - -## Citation and Identity Semantics - -There is one `citation_score`, initially zero regardless of the creation workflow. -The user Viewer and the optional core helper share identities and the same ledger. -Only emitted claim content increments scores. Parsing, matching, summaries, -structural audits, and native file reads do not. Do not manually add score fields -to Markdown or treat partial instrumentation as a complete access history. - -Displayed scores precede the current read. Long expansion chunks share one -logical-read event. Compact `--top` lines omit numeric scores to preserve room for -content. A citation measures exposure, not correctness or influence on reasoning. -Relevance and source trust/status rank ahead of scores; tied tiers reserve room -for zero-score entries when page and character budgets allow multiple results. -Popularity never overrides refutation or authorizes dropping a user constraint. - -The `d_...` fingerprint binds kind, canonical file/symbol anchor, parsed claim -text, and related refs. Internal whitespace is preserved, including code literals. -Filesystem aliases share on-disk identity and container prefixes are removed -from symbol anchors. Metadata-only source/status/label changes retain identity; -rewriting, moving, or merging claims/refs can produce a new zero-score identity. -Duplicate labels are unioned. No fuzzy score transfer is performed by Meditate. -An ID is an optional handle, not a replacement for the file/symbol address. - -## Local State and Concurrency - -SQLite lives at `shadowfrog/citations.sqlite3` under the repository's **common Git -directory**, shared by local worktrees. Different shadow roots have separate -scopes. Standalone shadows use `$XDG_STATE_HOME/shadowfrog/citations` -(Windows: `$LOCALAPPDATA`), falling back to `~/.local/state/shadowfrog/citations`. -The cache stores IDs, scores, retry receipts, hashed requests and ordered ID -snapshots, not discovery bodies. It is local metadata, not a Git-synchronized DB. -WAL requires a local filesystem, not a network share. - -Successful transactions are atomic. Busy reads/writes retry within a 100 ms -budget; accounting is best-effort, and a timed-out increment warns that the visit -was not recorded. Unknown scores display as `?`, but recording can still recover -after output. Retry a busy database later rather than deleting it. - -Ordinary reads retain no event receipt. Explicit retry and generated logical-read -receipts expire after 24 hours and are capped at 100,000; excess new receipts fail -visibly without weakening deduplication. Expired receipts are pruned in bounded -batches. Do not reuse one event ID for unrelated visits. - -Identical page snapshots reuse compressed storage, with at most 32 snapshots / -8 MiB retained. They expire after 24 hours or capacity eviction. Score changes -do not change a cursor's order. Relevant matching content/metadata changes require -restarting; mtime-only or unrelated-symbol edits do not invalidate non-recent views. -Text cursors bind the exact expanded body and metadata, preserve `--no-record`, -and expire after 24 hours; changed or expired content requires restarting `--get`. - -Commits use full synchronization. WAL checkpoints run after 256 pages with a -1 MiB retained-journal limit after reset; active transactions can keep a larger -journal temporarily. SQLite can retain reusable free pages. Score rows grow with -distinct identities. An incompatible prerelease cache emits reset guidance; move -that cache aside, not the shadow, if deliberately resetting local telemetry. diff --git a/skills/shadow-frog/shadow-cite.py b/skills/shadow-frog/shadow-cite.py new file mode 100755 index 0000000..114c970 --- /dev/null +++ b/skills/shadow-frog/shadow-cite.py @@ -0,0 +1,51 @@ +#!/usr/bin/env python3 +"""Record visits to existing Markdown discoveries after reading them normally. + +Example: + python shadow-cite.py .shadow/src/auth.py.md --symbol login --text "Rejects expired tokens." + +Repeat --text to cite several entries in the same file/section atomically. +Cite each deliberately consulted entry once per task. Repeated invocations +increment again; task-level deduplication belongs to the agent/coordinator. +""" + +import argparse +from pathlib import Path +import sys + + +_bytecode = sys.dont_write_bytecode +sys.dont_write_bytecode = True +sys.path.insert(0, str(Path(__file__).resolve().parent)) +try: + from _citations import record_citations +except ImportError as exc: + raise SystemExit("ERROR: Missing core citation helper; reinstall the full skill set") from exc +finally: + sys.path.pop(0) + sys.dont_write_bytecode = _bytecode + + +def main(): + for stream in (sys.stdout, sys.stderr): + if hasattr(stream, "reconfigure"): + stream.reconfigure(encoding="utf-8") + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("file", type=Path, help="The .shadow Markdown file already read") + parser.add_argument("--symbol", help="Exact source symbol, or File-Level (per-file shadows only)") + parser.add_argument("--text", action="append", required=True, help="Exact consulted claim; repeat for several") + parser.add_argument("--shadow-dir", type=Path, help="Explicit root for a nonstandard shadow location") + args = parser.parse_args() + try: + updates = record_citations(args.file, args.text, symbol=args.symbol, shadow_dir=args.shadow_dir) + except (ValueError, OSError, UnicodeError, RuntimeError) as exc: + print(f"ERROR: {exc}", file=sys.stderr) + return 1 + print(f"Recorded {len(updates)} citation(s) in {args.file}:") + for update in updates: + print(f" {update['before']} -> {update['after']}: {update['text']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/skills/shadow-frog/shadow-read.py b/skills/shadow-frog/shadow-read.py deleted file mode 100755 index 9f6dd02..0000000 --- a/skills/shadow-frog/shadow-read.py +++ /dev/null @@ -1,34 +0,0 @@ -#!/usr/bin/env python3 -"""Optional bounded knowledge retrieval for agents that already navigate files. - -Examples: - python shadow-read.py src/auth.py - python shadow-read.py src/auth.py::UserAuth.validate --limit 5 - python shadow-read.py --search "token expiry" - -Direct .shadow/.md reads remain the primary navigation method. -This helper selects and pages large sections; it does not replace their paths. -""" - -import sys -from pathlib import Path - - -_bytecode = sys.dont_write_bytecode -sys.dont_write_bytecode = True -sys.path.insert(0, str(Path(__file__).resolve().parent)) -try: - import _knowledge -except ImportError as exc: - raise SystemExit("ERROR: Missing core knowledge helper; reinstall the full ShadowFrog skill set") from exc -finally: - sys.path.pop(0) - sys.dont_write_bytecode = _bytecode - - -def main(): - return _knowledge.main(agent=True) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/tests/conftest.py b/tests/conftest.py index 25eff26..d165c07 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -53,16 +53,6 @@ def shadow_viewer(repo_root): return _load_script(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py") -@pytest.fixture(scope="session") -def shadow_knowledge(repo_root): - return _load_script(repo_root / "skills/shadow-frog/_knowledge.py") - - -@pytest.fixture(scope="session") -def shadow_reader(repo_root): - return _load_script(repo_root / "skills/shadow-frog/shadow-read.py") - - @pytest.fixture(scope="session") def dream_reconcile(repo_root): return _load_script(repo_root / "skills/shadow-frog-dream/dream-reconcile.py") @@ -98,6 +88,11 @@ def coherence(repo_root): return _load_script(repo_root / "skills/shadow-frog/_coherence.py") +@pytest.fixture(scope="session") +def citations(repo_root): + return _load_script(repo_root / "skills/shadow-frog/_citations.py") + + @pytest.fixture(scope="session") def nap(repo_root): return _load_script(repo_root / "skills/shadow-frog-nap/nap.py") diff --git a/tests/hooks/test_pre_tool_sh.py b/tests/hooks/test_pre_tool_sh.py index d9e86fc..0508621 100644 --- a/tests/hooks/test_pre_tool_sh.py +++ b/tests/hooks/test_pre_tool_sh.py @@ -154,24 +154,6 @@ def test_emits_both_output_shapes(self, coupon_demo): assert hso["hookEventName"] == "PreToolUse" assert hso["additionalContext"] == data["additionalContext"] - def test_citation_failure_is_visible_but_never_denies_edit(self, coupon_demo, tmp_path): - _fix_state_json_for_test(coupon_demo) - ledger = coupon_demo / ".git/shadowfrog/citations.sqlite3" - ledger.parent.mkdir() - ledger.write_bytes(b"invalid sqlite") - dedup = tmp_path / "dedup" - dedup.mkdir() - result = run_hook( - {"tool_name": "edit", "tool_input": {"file_path": "cart.py"}}, - cwd=coupon_demo, env_extra={"SHADOWFROG_TMP_DIR": str(dedup)}, - ) - assert result.returncode == 0 - context = json.loads(result.stdout)["additionalContext"] - assert "Actionable discoveries" in context - assert "id=d_" in context - assert "Reader reported a warning" in context - assert ledger.read_bytes() == b"invalid sqlite" - @pytest.mark.slow @pytest.mark.integration diff --git a/tests/skills/shadow_frog/conftest.py b/tests/skills/shadow_frog/conftest.py deleted file mode 100644 index fe04522..0000000 --- a/tests/skills/shadow_frog/conftest.py +++ /dev/null @@ -1,9 +0,0 @@ -"""Keep optional core-helper telemetry in each test's private state directory.""" - -import pytest - - -@pytest.fixture(autouse=True) -def local_citation_state(tmp_path, monkeypatch): - monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "citation-state")) - monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "citation-state")) diff --git a/tests/skills/shadow_frog/test_citations.py b/tests/skills/shadow_frog/test_citations.py index 0596144..28ddad3 100644 --- a/tests/skills/shadow_frog/test_citations.py +++ b/tests/skills/shadow_frog/test_citations.py @@ -1,374 +1,244 @@ -"""Real SQLite and Git tests for local, multiprocess citation bookkeeping.""" +"""Visible citation updates use real Markdown and per-file writer coordination.""" from pathlib import Path -import hashlib -import json -import sqlite3 import subprocess import sys -import time import pytest from tests.conftest import _load_script -HELPER = Path(__file__).resolve().parents[3] / "skills/shadow-frog/_citations.py" +CORE = Path(__file__).resolve().parents[3] / "skills/shadow-frog" @pytest.fixture def citations(): - return _load_script(HELPER) - - -def test_identity_is_stable_without_mutable_metadata(citations): - identity = citations.discovery_id("file", "src/a.py::run", " A claim with spacing. ") - assert identity == citations.discovery_id("file", "src/a.py::run", "A claim with spacing.") - assert identity != citations.discovery_id("file", "src/a.py::run", "A different claim.") - assert identity != citations.discovery_id("file", "src/a.py::other", "A claim with spacing.") - assert identity != citations.discovery_id("preference", "src/a.py::run", "A claim with spacing.") - assert citations.discovery_id("cross", "_cross/a.md", "Claim", ["b::f", "a::g"]) == ( - citations.discovery_id("cross", "_cross/a.md", "Claim", ["a::g", "b::f"]) + return _load_script(CORE / "_citations.py") + + +def per_file(tmp_path, *, score="", newline="\n"): + path = tmp_path / ".shadow/src/auth.py.md" + path.parent.mkdir(parents=True) + text = ( + "# Shadow: src/auth.py\n\n## File-Level\n\n" + "- Importing opens no sockets.\n _(verified, source: exploration)_\n\n" + "## `class Auth`\n\n" + "- Construction leaves credentials untouched.\n _(verified, source: user)_\n\n" + "### `Auth.login`\n\n" + "- Key `a b` is distinct.\n _(verified, source: exploration, labels: [bug]" + score + ")_\n\n" + "- Rejects expired tokens.\n _(verified, source: interaction, citation_score: 4)_\n\n" + "## Cross-References\n\n- [auth](../_cross/auth.md)\n" ) - - -@pytest.mark.parametrize("text", ['Key `a b` is accepted.', 'Key "a b" is accepted.', "Indented:\n value"]) -def test_identity_preserves_literal_whitespace(citations, text): - assert citations.discovery_id("file", "a::f", text) != citations.discovery_id( - "file", "a::f", " ".join(text.split()), + path.write_bytes(text.replace("\n", newline).encode("utf-8")) + return path + + +def test_direct_read_then_increment_only_the_selected_visible_score(citations, tmp_path): + path = per_file(tmp_path) + before = path.read_text(encoding="utf-8") + assert "Key `a b`" in before + updates = citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + after = path.read_text(encoding="utf-8") + assert after == before.replace("labels: [bug])_", "labels: [bug], citation_score: 1)_") + assert updates[0]["before"] == 0 and updates[0]["after"] == 1 + assert sorted(p.name for p in path.parent.iterdir()) == ["auth.py.md"] + + +def test_batch_is_atomic_and_repeated_text_counts_once(citations, tmp_path): + path = per_file(tmp_path) + citations.record_citations( + path, ["Key `a b` is distinct.", "Rejects expired tokens.", "Rejects expired tokens."], + symbol="Auth.login", + ) + content = path.read_text(encoding="utf-8") + assert "citation_score: 1" in content and "citation_score: 5" in content + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="found 0"): + citations.record_citations(path, ["Rejects expired tokens.", "Does not exist."], symbol="Auth.login") + assert path.read_bytes() == before + assert not list(path.parent.glob("*.citation.lock")) + + +def test_duplicate_claim_is_ambiguous_not_a_bulk_increment(citations, tmp_path): + path = per_file(tmp_path) + text = path.read_text(encoding="utf-8") + claim = "- Rejects expired tokens.\n _(verified, source: interaction, citation_score: 4)_\n" + path.write_text(text.replace(claim, claim + "\n" + claim), encoding="utf-8") + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="found 2"): + citations.record_citations(path, ["Rejects expired tokens."], symbol="Auth.login") + assert path.read_bytes() == before + + +@pytest.mark.parametrize("symbol,text", [ + ("File-Level", "Importing opens no sockets."), + ("Auth", "Construction leaves credentials untouched."), +]) +def test_file_level_and_container_symbols(citations, tmp_path, symbol, text): + path = per_file(tmp_path) + updates = citations.record_citations(path, [text], symbol=symbol) + assert updates == [{"text": text, "before": 0, "after": 1}] + + +def test_literal_whitespace_is_not_normalized_away(citations, tmp_path): + path = per_file(tmp_path) + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="found 0"): + citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + assert path.read_bytes() == before + + +def test_crlf_bom_and_existing_metadata_are_preserved(citations, tmp_path): + path = per_file(tmp_path, score=", citation_score: 7", newline="\r\n") + before = b"\xef\xbb\xbf" + path.read_bytes() + path.write_bytes(before) + citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + assert path.read_bytes() == before.replace(b"citation_score: 7", b"citation_score: 8") + + +def test_wrapped_claim_text_can_be_reported_without_copying_layout(citations, tmp_path): + path = per_file(tmp_path) + path.write_text( + path.read_text(encoding="utf-8").replace("Rejects expired tokens.", "Rejects expired\n tokens."), + encoding="utf-8", ) + citations.record_citations(path, ["Rejects expired tokens."], symbol="Auth.login") + assert "citation_score: 5" in path.read_text(encoding="utf-8") -def test_zero_default_atomic_increment_and_event_retry(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - first = citations.discovery_id("file", "a::run", "One") - second = citations.discovery_id("file", "a::run", "Two") - assert store.scores([first, second]) == {} - store.record([first, first], "event-a") - store.record([first], "event-a") - store.record([first, second], "event-b") - assert store.scores([first, second]) == {first: 2, second: 1} - other = citations.CitationStore(store.path, "examples/demo/.shadow") - assert other.scores([first]) == {} +@pytest.mark.parametrize("kind", ["preference", "cross"]) +def test_preferences_and_cross_cutting_have_visible_scores(citations, tmp_path, kind): + path = tmp_path / ".shadow" / ("_prefs.md" if kind == "preference" else "_cross/shared.md") + path.parent.mkdir(parents=True) + if kind == "preference": + path.write_text("# Preferences\n\n- Keep the contract.\n _(source: user)_\n", encoding="utf-8") + else: + path.write_text( + "# Shared\n\n**Category**: contract\n**Refs**:\n- `src/a.py::run`\n\n" + "**Discovery**: Keep the contract.\n\n_(verified, source: exploration)_\n", + encoding="utf-8", + ) + citations.record_citations(path, ["Keep the contract."]) + assert "citation_score: 1" in path.read_text(encoding="utf-8") -def test_processes_do_not_lose_increments(citations, tmp_path): - path = tmp_path / "citations.sqlite3" - identity = citations.discovery_id("file", "a::run", "A concurrent claim") - code = """ -import sys -sys.path.insert(0, sys.argv[1]) -from _citations import CitationStore -from pathlib import Path -store = CitationStore(Path(sys.argv[2]), ".shadow", timeout=5) -for index in range(30): - store.record([sys.argv[3]], f"{sys.argv[4]}-{index}") -""" - processes = [ - subprocess.Popen( - [sys.executable, "-c", code, str(HELPER.parent), str(path), identity, str(worker)], - stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", - ) - for worker in range(6) +def test_concurrent_processes_preserve_all_increments(citations, tmp_path): + path = per_file(tmp_path) + command = [ + sys.executable, str(CORE / "shadow-cite.py"), str(path), + "--symbol", "Auth.login", "--text", "Key `a b` is distinct.", ] - outcomes = [] + processes = [subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE) for _ in range(8)] try: for process in processes: - stdout, stderr = process.communicate(timeout=40) - outcomes.append((process.returncode, stdout, stderr)) + stdout, stderr = process.communicate(timeout=15) + assert process.returncode == 0, stderr.decode("utf-8") finally: for process in processes: if process.poll() is None: process.terminate() process.communicate(timeout=10) - assert all(code == 0 for code, _, _ in outcomes), outcomes - assert citations.CitationStore(path, ".shadow").scores([identity])[identity] == 180 - - -def test_wal_writes_keep_full_synchronization_and_checkpoint_limits(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - identity = citations.discovery_id("file", "a::f", "Claim") - store.record([identity], "first") - with store._connection() as db: - assert db.execute("PRAGMA journal_mode").fetchone()[0] == "wal" - assert db.execute("PRAGMA synchronous").fetchone()[0] >= 2 - assert db.execute("PRAGMA journal_size_limit").fetchone()[0] == citations.JOURNAL_BYTES - assert db.execute("PRAGMA wal_autocheckpoint").fetchone()[0] == citations.CHECKPOINT_PAGES - store.record([identity], "second") - assert store.scores([identity])[identity] == 2 - - -def test_active_reader_does_not_block_committing_citations(citations, tmp_path): - path = tmp_path / "citations.sqlite3" - store = citations.CitationStore(path, ".shadow") - identity = citations.discovery_id("file", "a::f", "Claim") - store.record([identity], "initial") - reader = sqlite3.connect(path) - reader.execute("BEGIN") - reader.execute("SELECT * FROM scores").fetchall() - code = """ -import sys -from pathlib import Path -sys.path.insert(0, sys.argv[1]) -from _citations import CitationStore -store = CitationStore(Path(sys.argv[2]), ".shadow") -store.record([sys.argv[3]], "concurrent") -store.record([sys.argv[3]], "concurrent") -""" - process = subprocess.Popen( - [sys.executable, "-c", code, str(HELPER.parent), str(path), identity], - stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", - ) - try: - # A held read snapshot no longer prevents the writer's commit. - stdout, stderr = process.communicate(timeout=15) - assert process.returncode == 0, stdout + stderr - assert reader.execute("SELECT citation_score FROM scores").fetchone()[0] == 1 - reader.rollback() - finally: - reader.close() - if process.poll() is None: - process.terminate() - process.communicate(timeout=10) - assert store.scores([identity])[identity] == 2 + content = path.read_text(encoding="utf-8") + assert "labels: [bug], citation_score: 8" in content + assert "source: interaction, citation_score: 4" in content + assert sorted(p.name for p in path.parent.iterdir()) == ["auth.py.md"] -def test_non_lock_errors_are_not_retried(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - attempts = [] +def test_existing_lock_fails_visibly_without_removing_it(citations, tmp_path): + path = per_file(tmp_path) + lock = path.with_name(path.name + ".citation.lock") + lock.write_text("other writer", encoding="utf-8") + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="writer busy"): + citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login", timeout=0.02) + assert path.read_bytes() == before and lock.read_text() == "other writer" - def invalid_sql(db): - attempts.append(True) - db.execute("INSERT INTO nonexistent_table VALUES (1)") - with pytest.raises(sqlite3.OperationalError, match="no such table"): - store._operation(invalid_sql, write=True) - assert len(attempts) == 1 +def test_failed_publication_preserves_knowledge_and_cleans_temporary_files(citations, tmp_path, monkeypatch): + path = per_file(tmp_path) + before = path.read_bytes() + def denied(*args): + raise PermissionError("simulated sharing violation") -def test_worktrees_share_scores_but_not_nested_shadows(citations, coupon_demo, tmp_path): - source = coupon_demo / ".shadow" - checkout = tmp_path / "linked" - subprocess.run( - ["git", "-C", str(coupon_demo), "worktree", "add", "--detach", str(checkout), "HEAD"], - check=True, capture_output=True, - ) - try: - first = citations.CitationStore.for_shadow(source) - second = citations.CitationStore.for_shadow(checkout / ".shadow") - assert first == second - identity = citations.discovery_id("file", "cart.py::calculate_total", "A claim") - first.record([identity], "one") - assert second.scores([identity]) == {identity: 1} - nested = coupon_demo / "example/.shadow" - nested.mkdir(parents=True) - third = citations.CitationStore.for_shadow(nested) - assert third.path == first.path and third.scope != first.scope - assert third.scores([identity]) == {} - status = subprocess.check_output( - ["git", "-C", str(coupon_demo), "status", "--porcelain"], text=True, - ) - assert status == "" - finally: - subprocess.run( - ["git", "-C", str(coupon_demo), "worktree", "remove", str(checkout)], - check=True, capture_output=True, - ) + monkeypatch.setattr(citations.os, "replace", denied) + with pytest.raises(PermissionError, match="sharing violation"): + citations.record_citations(path, ["Rejects expired tokens."], symbol="Auth.login") + assert path.read_bytes() == before + assert sorted(p.name for p in path.parent.iterdir()) == ["auth.py.md"] -def test_standalone_cache_stays_outside_shadow(citations, tmp_path, monkeypatch): - monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "local")) - monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "state")) - shadow = tmp_path / "repo/.shadow" - shadow.mkdir(parents=True) - store = citations.CitationStore.for_shadow(shadow) - identity = citations.discovery_id("file", "a::f", "Claim") - store.record([identity], "event") - assert not store.path.is_relative_to(shadow) - assert list(shadow.iterdir()) == [] - assert store.scores([identity]) == {identity: 1} - - -def test_locked_ledger_fails_within_bounded_wait(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow", timeout=0.02) - identity = citations.discovery_id("file", "a::f", "Claim") - store.record([identity], "first") - connection = sqlite3.connect(store.path) - try: - connection.execute("BEGIN EXCLUSIVE") - start = time.monotonic() - with pytest.raises(sqlite3.OperationalError, match="locked"): - store.record([identity], "second") - assert time.monotonic() - start < 0.5 - finally: - connection.rollback() - connection.close() - assert store.scores([identity])[identity] == 1 - - -def test_cursor_freezes_ids_not_scores(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - ids = [citations.discovery_id("file", "a::f", text) for text in ("A", "B", "C")] - cursor = store.save_page("request", "catalog", ids) - store.record([ids[-1]], "later") - assert store.load_page(cursor, "request", "catalog") == ids - with pytest.raises(ValueError, match="changed"): - store.load_page(cursor, "request", "different catalog") - with pytest.raises(ValueError, match="query"): - store.load_page(cursor, "different request", "catalog") - with pytest.raises(ValueError, match="expired|unavailable"): - store.load_page("0" * 32, "request", "catalog") - - -def test_pagination_stores_query_hash_not_search_text(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - query = "distinctive search phrase about the code" - cursor = store.save_page(query, "catalog", ["d_" + "a" * 32]) - with sqlite3.connect(store.path) as db: - stored = db.execute("SELECT request FROM pages").fetchone()[0] - assert stored != query and len(stored) == 64 - assert store.load_page(cursor, query, "catalog") == ["d_" + "a" * 32] - assert query.encode() not in store.path.read_bytes() - - -def test_expired_cursor_gives_restart_guidance(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - cursor = store.save_page("q", "c", ["d_" + "a" * 32]) - with sqlite3.connect(store.path) as db: - db.execute("UPDATE pages SET created = 0") - with pytest.raises(ValueError, match="expired.*rerun"): - store.load_page(cursor, "q", "c") - - -def test_schema_does_not_reset_newer_database(citations, tmp_path): - path = tmp_path / "citations.sqlite3" - with sqlite3.connect(path) as db: - db.execute("PRAGMA user_version = 99") - with pytest.raises(ValueError, match="version"): - citations.CitationStore(path, ".shadow").record(["d_" + "a" * 32], "event") - with sqlite3.connect(path) as db: - assert db.execute("PRAGMA user_version").fetchone()[0] == 99 - - -@pytest.mark.parametrize("event", ["", "x" * 129, "bad\nvalue"]) -def test_invalid_event_id_fails_early(citations, tmp_path, event): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - with pytest.raises(ValueError, match="event"): - store.record(["d_" + "a" * 32], event) - assert not store.path.exists() - - -def test_anonymous_reads_do_not_leave_retry_receipts(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - for _ in range(30): - store.record(["a", "b"]) - assert store.scores(["a", "b"]) == {"a": 30, "b": 30} - with store._connection() as db: - assert db.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 - - -def test_explicit_receipts_expire_without_deleting_scores(citations, tmp_path): - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - store.record(["a"], "event") - with store._connection() as db, db: - db.execute("UPDATE events SET created=0") - store.record(["b"]) - assert store.scores(["a", "b"]) == {"a": 1, "b": 1} - with store._connection() as db: - assert db.execute("SELECT COUNT(*) FROM events").fetchone()[0] == 0 - store.record(["a"], "event") - assert store.scores(["a"]) == {"a": 2} - - -def test_receipt_capacity_rejects_new_events_atomically(citations, tmp_path, monkeypatch): - monkeypatch.setattr(citations, "MAX_RECEIPTS", 2) - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - with pytest.raises(ValueError, match="capacity"): - store.record(["a", "b", "c"], "too-many") - assert store.scores(["a", "b", "c"]) == {} - store.record(["a", "b"], "fits") - store.record(["a", "b"], "fits") - assert store.scores(["a", "b"]) == {"a": 1, "b": 1} - store.record(["c"]) - assert store.scores(["c"]) == {"c": 1} - - -def test_identical_snapshots_reuse_storage_and_cache_size_is_bounded(citations, tmp_path, monkeypatch): - monkeypatch.setattr(citations, "MAX_PAGES", 3) - monkeypatch.setattr(citations, "MAX_PAGE_BYTES", 4096) - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - ids = [hashlib.sha256(str(index).encode()).hexdigest() for index in range(50)] - token = store.save_page("same", "catalog", ids) - assert store.save_page("same", "catalog", ids) == token - with store._connection() as db: - assert db.execute("SELECT COUNT(*) FROM pages").fetchone()[0] == 1 - assert db.execute("SELECT length(ids) FROM pages").fetchone()[0] < len(json.dumps(ids)) - for index in range(8): - latest = store.save_page(f"query-{index}", "catalog", ids) - with store._connection() as db: - count, size = db.execute("SELECT COUNT(*), SUM(length(ids)) FROM pages").fetchone() - assert count <= 3 and size <= 4096 - assert store.load_page(latest, "query-7", "catalog") == ids - with pytest.raises(ValueError, match="expired|unavailable"): - store.load_page(token, "same", "catalog") - - -def test_journal_size_remains_capped_after_large_snapshot_cleanup(citations, tmp_path, monkeypatch): - monkeypatch.setattr(citations, "JOURNAL_BYTES", 4096) - store = citations.CitationStore(tmp_path / "citations.sqlite3", ".shadow") - ids = [hashlib.sha256(str(index).encode()).hexdigest() for index in range(1000)] - store.save_page("large", "catalog", ids) - with store._connection() as db, db: - db.execute("UPDATE pages SET created=0") - store.save_page("small", "catalog", ["one"]) - journal = tmp_path / "citations.sqlite3-wal" - assert not journal.exists() or journal.stat().st_size <= 4096 - - -def test_production_budget_preserves_all_successful_writes(citations, tmp_path): - path = tmp_path / "citations.sqlite3" - code = """ -import json, sqlite3, sys -from pathlib import Path -sys.path.insert(0, sys.argv[1]) -from _citations import CitationStore, is_busy_error -store = CitationStore(Path(sys.argv[2]), ".shadow") -assert store.timeout == 0.1 -ok = busy = 0 -for index in range(30): - try: - store.record(["shared"]) - except sqlite3.OperationalError as exc: - if not is_busy_error(exc): - raise - busy += 1 - else: - ok += 1 -print(json.dumps({"ok": ok, "busy": busy})) -""" - workers = [ - subprocess.Popen( - [sys.executable, "-c", code, str(HELPER.parent), str(path)], - stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, encoding="utf-8", - ) - for _ in range(6) - ] - outcomes = [] - try: - for worker in workers: - stdout, stderr = worker.communicate(timeout=40) - assert worker.returncode == 0, stderr - outcomes.append(json.loads(stdout)) - finally: - for worker in workers: - if worker.poll() is None: - worker.terminate() - worker.communicate(timeout=10) - successful = sum(item["ok"] for item in outcomes) - assert successful > 0 - assert successful + sum(item["busy"] for item in outcomes) == 180 - assert store_score(citations, path) == successful - - -def store_score(citations, path): - return citations.CitationStore(path, ".shadow").scores(["shared"]).get("shared", 0) +def test_detected_ordinary_editor_race_is_not_overwritten(citations, tmp_path, monkeypatch): + path = per_file(tmp_path) + before = path.read_bytes() + actual_fsync = citations.os.fsync + + def another_writer(fd): + path.write_bytes(before + b"\nAdditional knowledge from another writer.\n") + actual_fsync(fd) + + monkeypatch.setattr(citations.os, "fsync", another_writer) + with pytest.raises(citations.CitationError, match="changed during"): + citations.record_citations(path, ["Rejects expired tokens."], symbol="Auth.login") + assert path.read_bytes() == before + b"\nAdditional knowledge from another writer.\n" + assert sorted(p.name for p in path.parent.iterdir()) == ["auth.py.md"] + + +@pytest.mark.parametrize("score", [-1, 1.2, True, None, "3"]) +def test_json_score_requires_a_nonnegative_integer(citations, score): + with pytest.raises(citations.CitationError, match="nonnegative integer"): + citations.validate_score(score) + + +@pytest.mark.parametrize("raw", ["-1", "1.2", "true", "NaN", ""]) +def test_invalid_existing_score_is_not_reset(citations, tmp_path, raw): + path = per_file(tmp_path, score=f", citation_score: {raw}") + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="Malformed metadata"): + citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + assert path.read_bytes() == before + + +def test_cli_error_names_the_entry_and_preserves_file(tmp_path): + path = per_file(tmp_path) + before = path.read_bytes() + result = subprocess.run( + [sys.executable, str(CORE / "shadow-cite.py"), str(path), "--symbol", "Auth.login", "--text", "Wrong claim."], + capture_output=True, text=True, encoding="utf-8", + ) + assert result.returncode == 1 and "ERROR:" in result.stderr and "Re-read" in result.stderr + assert result.stdout == "" and path.read_bytes() == before + + +@pytest.mark.parametrize("relative", ["_index.md", "_dreams/dream/report.md", "_meta/notes.md"]) +def test_indexes_and_experiment_reports_are_not_citation_targets(citations, tmp_path, relative): + path = tmp_path / ".shadow" / relative + path.parent.mkdir(parents=True) + path.write_text("# Metadata\n", encoding="utf-8") + with pytest.raises(citations.CitationError, match="not indexes or dream reports"): + citations.record_citations(path, ["Metadata"], symbol="File-Level") + + +def test_outside_target_symlink_is_refused(citations, tmp_path, make_symlink): + path = per_file(tmp_path) + outside = tmp_path / "outside.md" + outside.write_bytes(path.read_bytes()) + path.unlink() + make_symlink(path, outside) + before = outside.read_bytes() + with pytest.raises(citations.CitationError, match="inside"): + citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + assert outside.read_bytes() == before + + +def test_aliased_shadow_root_is_refused(citations, tmp_path, make_symlink): + real_root = tmp_path / "real" + real_root.mkdir() + path = real_root / "file.md" + path.write_text("# Shadow: file\n\n## `run`\n\n- Claim.\n _(verified, source: exploration)_\n", encoding="utf-8") + make_symlink(tmp_path / ".shadow", real_root, target_is_directory=True) + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="filesystem alias"): + citations.record_citations(tmp_path / ".shadow/file.md", ["Claim."], symbol="run") + assert path.read_bytes() == before diff --git a/tests/skills/shadow_frog/test_knowledge.py b/tests/skills/shadow_frog/test_knowledge.py deleted file mode 100644 index 19b6c2e..0000000 --- a/tests/skills/shadow_frog/test_knowledge.py +++ /dev/null @@ -1,2186 +0,0 @@ -r"""Tests for shared `skills/shadow-frog/_knowledge.py` and the user viewer CLI. - -Philosophy: USE REAL FILES (per `minimal-mocking-tests`). Retrieval does not -edit shadow Markdown; local citation bookkeeping is isolated by the fixtures. -Tests construct shadow trees or exercise the CLI against `coupon_demo`. - -Test categories: - * In-process function tests (no `@pytest.mark.slow`): exercise - parsing helpers directly via the `shadow_knowledge` fixture. - * CLI integration tests (`@pytest.mark.slow @pytest.mark.integration`): - invoke shadow-viewer.py as a subprocess against `coupon_demo`. - -B3 regression: a discovery whose continuation lines include -``Dream report: `_dreams//` `` must extract the slug path into -`meta["dream_report"]` and must NOT include "Dream report" or the slug -in the discovery body text. -""" -import json -import os -import re -import subprocess -import sys -import textwrap -from datetime import datetime - -import pytest - - -# --- Helpers --------------------------------------------------------------- - - -def _write_shadow(shadow_dir, rel_path, content): - """Write `content` to /, creating parents.""" - p = shadow_dir / rel_path - p.parent.mkdir(parents=True, exist_ok=True) - p.write_text(textwrap.dedent(content), encoding="utf-8") - return p - - -def _make_shadow_root(tmp_path): - """Create an empty .shadow/ dir under tmp_path and return it.""" - sd = tmp_path / ".shadow" - sd.mkdir() - return sd - - -def _run_viewer(repo_root, cwd, *args): - """Run shadow-viewer.py as a subprocess from `cwd`.""" - script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" - return subprocess.run( - [sys.executable, str(script), *args], - cwd=str(cwd), - capture_output=True, - text=True, - encoding="utf-8", - check=False, - ) - - -# --- parse_discovery ------------------------------------------------------- - - -def test_parse_discovery_standard(shadow_knowledge): - """Basic verified/exploration discovery, no labels.""" - line = "- Caches None for invalid codes" - cont = [" _(verified, source: exploration)_"] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["text"] == "Caches None for invalid codes" - assert d["status"] == "verified" - assert d["source"] == "exploration" - assert "labels" not in d - assert "dream_report" not in d - - -def test_parse_discovery_with_labels(shadow_knowledge): - """Labels are parsed into a list, trimmed, lowercase comma-split.""" - line = "- Foo" - cont = [" _(verified, source: user, labels: [bug, security])_"] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["text"] == "Foo" - assert d["status"] == "verified" - assert d["source"] == "user" - assert d["labels"] == ["bug", "security"] - - -@pytest.mark.parametrize("status", ["verified", "uncertain", "refuted"]) -def test_parse_discovery_status_variants(shadow_knowledge, status): - line = "- Some discovery" - cont = [f" _({status}, source: exploration)_"] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["status"] == status - assert d["source"] == "exploration" - - -@pytest.mark.parametrize("source", ["exploration", "user", "interaction"]) -def test_parse_discovery_source_variants(shadow_knowledge, source): - line = "- Some discovery" - cont = [f" _(verified, source: {source})_"] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["source"] == source - - -def test_parse_discovery_also_involves(shadow_knowledge): - """`Also involves:` populates a list of file::symbol anchors.""" - line = "- A multi-symbol discovery" - cont = [ - " _(verified, source: exploration)_", - " Also involves: `inventory.py::validate_coupon`, `cart.py::COUPON_CACHE`", - ] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["text"] == "A multi-symbol discovery" - assert d["also_involves"] == [ - "inventory.py::validate_coupon", - "cart.py::COUPON_CACHE", - ] - # also_involves line must not leak into the body text - assert "Also involves" not in d["text"] - - -def test_parse_discovery_b3_dream_report_regression(shadow_knowledge): - """B3 regression: Dream report goes into meta['dream_report'] and is - excluded from the body text.""" - line = "- Case-variant lookups create duplicate cache entries" - cont = [ - " _(verified, source: exploration, labels: [bug, performance])_", - " Dream report: `_dreams/20260420-140000Z-cache-poison-sequence/`", - ] - d = shadow_knowledge.parse_discovery(line, cont) - # Body text is preserved, with no Dream report leakage - assert d["text"] == "Case-variant lookups create duplicate cache entries" - assert "Dream report" not in d["text"] - assert "_dreams/" not in d["text"] - # meta["dream_report"] captures the backtick payload (slug folder path) - assert d["dream_report"] == ( - "_dreams/20260420-140000Z-cache-poison-sequence/" - ) - - -def test_parse_discovery_b3_dream_report_with_also_involves(shadow_knowledge): - """Dream report + Also involves on the same discovery — both extracted, - neither leaks into body text.""" - line = "- Discovery with both extras" - cont = [ - " _(verified, source: exploration, labels: [security])_", - " Dream report: `_dreams/20260420-142000Z-adversarial-inputs/`", - " Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE`", - ] - d = shadow_knowledge.parse_discovery(line, cont) - assert d["text"] == "Discovery with both extras" - assert "Dream report" not in d["text"] - assert "Also involves" not in d["text"] - assert d["dream_report"] == ( - "_dreams/20260420-142000Z-adversarial-inputs/" - ) - assert d["also_involves"] == [ - "cart.py::get_coupon", - "cart.py::COUPON_CACHE", - ] - - -def test_parse_discovery_multiline_body(shadow_knowledge): - """Lines that are neither metadata nor structured extras are appended to - the body text.""" - line = "- Lead sentence." - cont = [ - " continuation prose", - " _(verified, source: exploration)_", - ] - d = shadow_knowledge.parse_discovery(line, cont) - assert "Lead sentence." in d["text"] - assert "continuation prose" in d["text"] - assert d["status"] == "verified" - - -def test_parse_discovery_no_metadata(shadow_knowledge): - """Bullet with no metadata blob still returns a dict with text but no - status/source keys.""" - d = shadow_knowledge.parse_discovery("- bare bullet", []) - assert d["text"] == "bare bullet" - assert "status" not in d - assert "source" not in d - - -def test_parse_discovery_preferences_source_only(shadow_knowledge): - """Preferences use `_(source: user)_` (no status). Extracts source.""" - d = shadow_knowledge.parse_discovery( - "- Prefer X over Y", [" _(source: user)_"] - ) - assert d["text"] == "Prefer X over Y" - assert d["source"] == "user" - assert "status" not in d - - -def test_parse_discovery_none_input_does_not_crash(shadow_knowledge): - """Passing a non-string line shouldn't raise — should return a dict.""" - d = shadow_knowledge.parse_discovery(None, None) - assert isinstance(d, dict) - assert "text" in d - - -# --- parse_shadow_file ----------------------------------------------------- - - -def test_parse_shadow_file_placeholder(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "foo.py.md", """\ - # Shadow: foo.py - - **Language**: Python | **Lines**: 10 - - _No discoveries yet._ - """) - res = shadow_knowledge.parse_shadow_file(f) - assert res["source_file"] == "foo.py" - assert res["language"] == "Python" - assert res["lines"] == 10 - assert res["symbols"] == [] - assert res["discoveries"] == [] - assert res["cross_references"] == [] - assert res["parse_errors"] == [] - - -def test_parse_shadow_file_one_symbol_one_discovery(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "auth.py.md", """\ - # Shadow: auth.py - - **Language**: Python | **Lines**: 42 - - ## `authenticate` - - - Returns None on expired tokens, silently. - _(verified, source: exploration, labels: [security])_ - """) - res = shadow_knowledge.parse_shadow_file(f) - assert res["source_file"] == "auth.py" - assert res["symbols"] == ["authenticate"] - assert len(res["discoveries"]) == 1 - d = res["discoveries"][0] - assert d["symbol"] == "authenticate" - assert d["file"] == "auth.py" - assert d["status"] == "verified" - assert d["source"] == "exploration" - assert d["labels"] == ["security"] - assert "Returns None" in d["text"] - - -def test_parse_shadow_file_cross_references_backpointers( - shadow_knowledge, tmp_path -): - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "bar.py.md", """\ - # Shadow: bar.py - - ## `func` - - - A discovery. - _(verified, source: exploration)_ - - ## Cross-References - - - [my-cross-cutting](_cross/my-cross-cutting.md) - (involves `bar.py::func`) - - [another-one](_cross/another-one.md) - """) - res = shadow_knowledge.parse_shadow_file(f) - assert res["symbols"] == ["func"] - # Cross-references back-pointer link labels are collected - assert "my-cross-cutting" in res["cross_references"] - assert "another-one" in res["cross_references"] - # Cross-ref bullets are NOT mistaken for discoveries - assert len(res["discoveries"]) == 1 - - -def test_parse_shadow_file_file_level_and_cross_refs(shadow_knowledge, tmp_path): - """`## File-Level` discoveries are tagged with symbol='file-level' and - `## Cross-References` bullets are not treated as discoveries.""" - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "mix.py.md", """\ - # Shadow: mix.py - - ## File-Level - - - A module-wide observation. - _(verified, source: exploration)_ - - ## `helper` - - - A symbol discovery. - _(verified, source: user)_ - - ## Cross-References - - - [shared](_cross/shared.md) - """) - res = shadow_knowledge.parse_shadow_file(f) - discs = res["discoveries"] - assert len(discs) == 2 - by_sym = {d["symbol"]: d for d in discs} - assert "file-level" in by_sym - assert "helper" in by_sym - assert by_sym["file-level"]["source"] == "exploration" - assert by_sym["helper"]["source"] == "user" - assert res["cross_references"] == ["shared"] - - -def test_parse_shadow_file_malformed_does_not_crash(shadow_knowledge, tmp_path): - """Bizarre / structurally broken content shouldn't raise.""" - sd = _make_shadow_root(tmp_path) - f = _write_shadow(sd, "junk.py.md", """\ - # Shadow: junk.py - **Language**: notnumeric | **Lines**: notanumber - - ## not a backtick heading - - orphan bullet with no metadata - ## `realsym` - - real disc - _(verified, source: exploration)_ - """) - res = shadow_knowledge.parse_shadow_file(f) - # Parse succeeds despite weirdness - assert res["source_file"] == "junk.py" - # The bad "Lines" cell stays None (int parse skipped) - assert res["lines"] is None - # `realsym` is captured; the non-backtick heading is not a symbol - assert "realsym" in res["symbols"] - assert "not a backtick heading" not in res["symbols"] - - -def test_parse_shadow_file_missing_file_returns_error( - shadow_knowledge, tmp_path -): - """Reading a non-existent path records a parse_error, doesn't raise.""" - res = shadow_knowledge.parse_shadow_file(tmp_path / "ghost.md") - assert res["parse_errors"] - assert res["symbols"] == [] - assert res["discoveries"] == [] - - -# --- parse_cross_cutting --------------------------------------------------- - - -def test_parse_cross_cutting_empty_dir(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_cross").mkdir() - assert shadow_knowledge.parse_cross_cutting(sd) == [] - - -def test_parse_cross_cutting_no_dir(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - # _cross/ not created - assert shadow_knowledge.parse_cross_cutting(sd) == [] - - -def test_parse_cross_cutting_one_entry(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/example-pattern.md", """\ - # Example pattern - - **Category**: pattern - **Refs**: - - `cart.py::calculate_total` - - `inventory.py::validate_coupon` - - **Discovery**: A multi-file pattern observed across the codebase. - - _(verified, source: exploration, labels: [bug])_ - """) - entries = shadow_knowledge.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["slug"] == "example-pattern" - assert e["title"] == "Example pattern" - assert e["category"] == "pattern" - assert "cart.py::calculate_total" in e["refs"] - assert "inventory.py::validate_coupon" in e["refs"] - assert "multi-file pattern" in e["discovery"] - assert e["status"] == "verified" - assert e["source"] == "exploration" - assert e["labels"] == ["bug"] - - -def test_parse_cross_cutting_no_labels(shadow_knowledge, tmp_path): - """A cross-cutting entry without labels is still parsed, with no - `labels` key in the entry.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-label.md", """\ - # Plain entry - - **Category**: behavior - **Refs**: - - `foo.py::bar` - - **Discovery**: Something happens. - - _(uncertain, source: exploration)_ - """) - entries = shadow_knowledge.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["status"] == "uncertain" - assert e["source"] == "exploration" - assert "labels" not in e - - -def test_parse_cross_cutting_minor_format_variation(shadow_knowledge, tmp_path): - """File missing a `**Category**:` field doesn't crash; entry is still - emitted with whatever fields could be parsed.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/sparse.md", """\ - # Sparse entry - - Some prose with no structured fields. - - _(refuted, source: user)_ - """) - entries = shadow_knowledge.parse_cross_cutting(sd) - assert len(entries) == 1 - e = entries[0] - assert e["slug"] == "sparse" - assert e["title"] == "Sparse entry" - assert e["status"] == "refuted" - assert e["source"] == "user" - - -# --- parse_prefs ----------------------------------------------------------- - - -def test_parse_prefs_missing_file(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - assert shadow_knowledge.parse_prefs(sd) == [] - - -def test_parse_prefs_zero(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - _No preferences recorded yet._ - """) - assert shadow_knowledge.parse_prefs(sd) == [] - - -def test_parse_prefs_one(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Always use type hints on public APIs. - _(source: user)_ - """) - prefs = shadow_knowledge.parse_prefs(sd) - assert len(prefs) == 1 - assert prefs[0]["text"] == "Always use type hints on public APIs." - assert prefs[0]["source"] == "user" - assert prefs[0]["type"] == "preference" - - -def test_parse_prefs_three(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Use kebab-case for slugs. - _(source: user)_ - - Never commit secrets to source control. - _(source: interaction)_ - - Prefer fail-fast for required dependencies. - _(source: user)_ - """) - prefs = shadow_knowledge.parse_prefs(sd) - assert len(prefs) == 3 - texts = [p["text"] for p in prefs] - assert any("kebab-case" in t for t in texts) - assert any("secrets" in t for t in texts) - assert any("fail-fast" in t for t in texts) - sources = [p["source"] for p in prefs] - assert "user" in sources - assert "interaction" in sources - - -# --- load_state ------------------------------------------------------------ - - -def test_load_state_missing(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - assert shadow_knowledge.load_state(sd) == {} - - -def test_load_state_valid_json(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - payload = { - "version": 1, - "total_files": 5, - "total_discoveries": 42, - "last_update_type": "auto", - } - (sd / "_meta" / "state.json").write_text( - json.dumps(payload), encoding="utf-8" - ) - state = shadow_knowledge.load_state(sd) - assert state == payload - - -def test_load_state_malformed_json(shadow_knowledge, tmp_path): - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text( - "{not valid json", encoding="utf-8" - ) - state = shadow_knowledge.load_state(sd) - assert state == {} - - -def test_load_state_non_object_json(shadow_knowledge, tmp_path): - """JSON parses but is not a dict — returns empty dict sentinel.""" - sd = _make_shadow_root(tmp_path) - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text("[1, 2, 3]", encoding="utf-8") - assert shadow_knowledge.load_state(sd) == {} - - -# --- get_all_shadow_files -------------------------------------------------- - - -def test_get_all_shadow_files_excludes_special(shadow_knowledge, tmp_path): - """Per-file shadows are returned; _cross/, _dreams/, _meta/, _index.md, - _prefs.md, state.json are all excluded.""" - sd = _make_shadow_root(tmp_path) - # Files that SHOULD be returned - _write_shadow(sd, "a.py.md", "# Shadow: a.py\n") - _write_shadow(sd, "src/b.py.md", "# Shadow: src/b.py\n") - _write_shadow(sd, "deep/nested/c.py.md", "# Shadow: deep/nested/c.py\n") - # Files that should be EXCLUDED - _write_shadow(sd, "_index.md", "# Shadow Index\n") - _write_shadow(sd, "_prefs.md", "# Preferences\n") - _write_shadow(sd, "_cross/some.md", "# Some cross\n") - _write_shadow(sd, "_dreams/dream-1/report.md", "# A dream\n") - _write_shadow(sd, "_meta/state.json", "{}") - # Junk files that are not .md and shouldn't show up anyway - (sd / "notes.json").write_text("{}", encoding="utf-8") - - files = shadow_knowledge.get_all_shadow_files(sd) - rels = sorted(f.relative_to(sd).as_posix() for f in files) - assert rels == ["a.py.md", "deep/nested/c.py.md", "src/b.py.md"] - - -def test_get_all_shadow_files_against_coupon_demo(shadow_knowledge, coupon_demo): - sd = coupon_demo / ".shadow" - files = shadow_knowledge.get_all_shadow_files(sd) - rels = sorted(f.relative_to(sd).as_posix() for f in files) - assert rels == ["cart.py.md", "inventory.py.md", "test_cart.py.md"] - # Make sure none of the special files leaked through - for r in rels: - assert not r.startswith(("_cross/", "_dreams/", "_meta/")) - assert r not in ("_index.md", "_prefs.md") - - -# --- collect_all_discoveries (against fixture) ----------------------------- - - -def test_collect_all_discoveries_counts(shadow_knowledge, coupon_demo): - """Coupon demo has 33 per-file discoveries (matches state.json).""" - sd = coupon_demo / ".shadow" - all_disc = shadow_knowledge.collect_all_discoveries(sd) - # state.json claims 33 total discoveries - assert len(all_disc) == 33 - - # File breakdown: cart=14, inventory=10, test_cart=9 - by_file = {} - for d in all_disc: - by_file.setdefault(d.get("file"), 0) - by_file[d.get("file")] += 1 - assert by_file == {"cart.py": 14, "inventory.py": 10, "test_cart.py": 9} - - -def test_collect_all_discoveries_shadow_path_and_mtime( - shadow_knowledge, coupon_demo -): - sd = coupon_demo / ".shadow" - all_disc = shadow_knowledge.collect_all_discoveries(sd) - for d in all_disc: - assert "shadow_path" in d - assert d["shadow_path"].endswith(".md") - assert "shadow_mtime" in d - assert isinstance(d["shadow_mtime"], float) - - -def test_collect_all_discoveries_b3_no_dream_report_in_text( - shadow_knowledge, coupon_demo -): - """B3 regression on real fixture: no discovery body should contain - 'Dream report' or the literal `_dreams/` slug path.""" - sd = coupon_demo / ".shadow" - all_disc = shadow_knowledge.collect_all_discoveries(sd) - leaks = [ - d for d in all_disc - if "Dream report" in d.get("text", "") - or "_dreams/" in d.get("text", "") - ] - assert leaks == [], ( - f"Dream report leaked into {len(leaks)} discovery body/bodies: " - f"{[d['text'][:80] for d in leaks]}" - ) - - # And at least some discoveries actually have dream_report metadata - # (the fixture has several Dream report continuation lines) - with_dream = [d for d in all_disc if d.get("dream_report")] - assert len(with_dream) >= 3, ( - "Expected coupon-demo fixture to have multiple Dream-report-tagged " - f"discoveries; found {len(with_dream)}" - ) - for d in with_dream: - assert d["dream_report"].startswith("_dreams/") - - -# --- CLI: --summary -------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_summary(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--summary") - assert r.returncode == 0, r.stderr - out = r.stdout - # Header is "Files shadowed:", "Symbols tracked:", "Discoveries:" per - # current --summary output. Task wording used "Total files:"/"Symbols:" - # /"Discoveries:" loosely — match the actual labels. - assert "Files shadowed:" in out - assert "Symbols tracked:" in out - assert "Discoveries:" in out - # Cross-cutting block exists - assert "Cross-cutting" in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_default_view_is_summary(repo_root, coupon_demo): - """Running with no flags should produce the summary view.""" - r = _run_viewer(repo_root, coupon_demo) - assert r.returncode == 0, r.stderr - assert "Shadow Knowledge Base Summary" in r.stdout - - -# --- CLI: --search --------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_search_matches_text(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--search", "coupon") - assert r.returncode == 0, r.stderr - # Header echoes the query and results were found - assert "'coupon'" in r.stdout - # Some matched line should contain the query (case-insensitive) - assert "coupon" in r.stdout.lower() - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_search_no_results(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--search", "zzznotpresentzzz") - assert r.returncode == 0, r.stderr - assert "No results for 'zzznotpresentzzz'" in r.stdout - - -# --- CLI: --top ------------------------------------------------------------ - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_cart_py_header_and_no_dream_report_leak( - repo_root, coupon_demo -): - """B3 regression at the CLI layer: --top output for a file whose shadow - contains `Dream report:` continuation lines must NOT inline that text - in any discovery body.""" - r = _run_viewer(repo_root, coupon_demo, "--top", "cart.py") - assert r.returncode == 0, r.stderr - out = r.stdout - # Header form: "Top N of M actionable discoveries for cart.py:" - assert "actionable discoveries for cart.py:" in out - # Match the documented prefix exactly - assert out.splitlines()[0].startswith("Top ") - assert "for cart.py:" in out.splitlines()[0] - - # B3: no literal "Dream report:" or raw `_dreams/...` slug paths in any - # of the discovery bullets - assert "Dream report:" not in out - assert "_dreams/" not in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_label_filter_bug(repo_root, coupon_demo): - r = _run_viewer( - repo_root, coupon_demo, "--top", "cart.py", "--top-labels", "bug" - ) - assert r.returncode == 0, r.stderr - out = r.stdout - assert "for cart.py:" in out - # Every bulleted result line should mention 'bug' in its label bracket - bullet_lines = [ - l for l in out.splitlines() if l.startswith("- [") - ] - assert bullet_lines, f"No bullets in --top output:\n{out}" - for line in bullet_lines: - bracket = line.split("]", 1)[0] - assert "bug" in bracket, ( - f"Expected 'bug' label in bracket of: {line}" - ) - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_top_unknown_file(repo_root, coupon_demo): - """A file with no shadow + no cross refs reports nothing actionable.""" - r = _run_viewer(repo_root, coupon_demo, "--top", "does/not/exist.py") - assert r.returncode == 0, r.stderr - assert "No actionable discoveries" in r.stdout - - -# --- CLI: --labels (repo-wide) --------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_labels_bug(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--labels", "bug") - assert r.returncode == 0, r.stderr - out = r.stdout - assert "label(s): bug" in out - # There are multiple bug-labeled discoveries in the fixture - assert "[bug]" in out - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_labels_unknown(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--labels", "nonexistent") - assert r.returncode == 0, r.stderr - assert "No discoveries with label(s): nonexistent" in r.stdout - - -# --- CLI: --recent --------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_recent_caps_at_n(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--recent", "5") - assert r.returncode == 0, r.stderr - out = r.stdout - assert "Most Recent Discoveries (top 5)" in out - # Each recent entry has a timestamp prefix " [YYYY-MM-DD HH:MM]" - entries = [l for l in out.splitlines() if l.strip().startswith("[20")] - assert len(entries) <= 5 - # Coupon-demo has > 5 total items so we expect exactly 5 - assert len(entries) == 5 - - -# --- CLI: --prefs ---------------------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_prefs_empty(repo_root, coupon_demo): - """Coupon-demo ships with no preferences recorded.""" - r = _run_viewer(repo_root, coupon_demo, "--prefs") - assert r.returncode == 0, r.stderr - assert "No preferences recorded yet." in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_prefs_populated(repo_root, coupon_demo): - """After populating _prefs.md, --prefs lists them.""" - prefs_path = coupon_demo / ".shadow" / "_prefs.md" - prefs_path.write_text( - textwrap.dedent("""\ - # Preferences - - - Use snake_case for Python identifiers. - _(source: user)_ - - Avoid mutable default arguments. - _(source: interaction)_ - """), - encoding="utf-8", - ) - r = _run_viewer(repo_root, coupon_demo, "--prefs") - assert r.returncode == 0, r.stderr - assert "Project Preferences (2 total)" in r.stdout - assert "snake_case" in r.stdout - assert "mutable default" in r.stdout - - -# --- CLI: --check-invariants ----------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_clean(repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode == 0, ( - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - assert "✓ Invariants OK" in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_detects_missing_cross_file( - repo_root, coupon_demo -): - """Deleting a _cross/ file leaves dangling back-pointers in per-file - shadows. Invariant #5 must flag this as a violation.""" - cross_path = ( - coupon_demo / ".shadow" / "_cross" - / "coupon-case-normalization-mismatch.md" - ) - assert cross_path.is_file() - cross_path.unlink() - - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode != 0, ( - f"Expected nonzero exit when _cross/ file missing; got 0.\n" - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - # Violations report the dangling slug - assert "coupon-case-normalization-mismatch" in r.stdout - assert "cross-ref" in r.stdout - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_check_invariants_detects_bad_heading(repo_root, coupon_demo): - """Renaming a symbol heading to a non-backtick form is a heading-format - violation.""" - cart_path = coupon_demo / ".shadow" / "cart.py.md" - text = cart_path.read_text(encoding="utf-8") - # `## `COUPON_CACHE`` -> `## COUPON_CACHE` (drop backticks) - mutated = text.replace("## `COUPON_CACHE`", "## COUPON_CACHE", 1) - assert mutated != text, "Substitution did not match" - cart_path.write_text(mutated, encoding="utf-8") - - r = _run_viewer(repo_root, coupon_demo, "--check-invariants") - assert r.returncode != 0, ( - f"Expected nonzero exit for bad heading; got 0.\n" - f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" - ) - assert "heading" in r.stdout - - -# --- CLI: missing shadow dir ---------------------------------------------- - - -@pytest.mark.slow -@pytest.mark.integration -def test_cli_missing_shadow_dir_fails(repo_root, tmp_path): - """Running with no shadow and a bogus --shadow-dir exits 1.""" - r = _run_viewer( - repo_root, tmp_path, - "--shadow-dir", str(tmp_path / "nope"), "--summary", - ) - assert r.returncode == 1 - assert "No .shadow/ directory found" in r.stderr - - -# =========================================================================== -# RENDER FUNCTIONS: in-process tests -# -# The CLI integration tests above invoke shadow-viewer.py as a subprocess — -# that exercises the dispatcher but doesn't contribute to coverage of the -# loaded module. The tests below call view_* and main() directly via the -# `shadow_knowledge` fixture so coverage actually accumulates. -# =========================================================================== - - -# --- shared helpers -------------------------------------------------------- - - -def _call_main(shadow_knowledge, argv): - """Invoke `shadow_knowledge.main()` in-process with the given argv. - - Returns the integer exit code. We mutate `sys.argv` directly (no - monkeypatch / no mocking) and always restore it in a finally. - """ - saved_argv = sys.argv - sys.argv = ["shadow-viewer.py", *argv] - try: - shadow_knowledge.main() - return 0 - except SystemExit as e: - code = e.code - if code is None: - return 0 - if isinstance(code, int): - return code - return 1 - finally: - sys.argv = saved_argv - - -def _make_minimal_shadow(tmp_path, extras=None): - """Build a minimal but valid .shadow/ tree in tmp_path. - - Returns the .shadow/ Path. `extras` is a dict of {relpath: content} - appended on top of the baseline. - """ - sd = tmp_path / ".shadow" - sd.mkdir() - (sd / "_meta").mkdir() - (sd / "_meta" / "state.json").write_text( - json.dumps({ - "version": 1, - "last_update_at": "2026-04-20T16:30:00Z", - "last_update_type": "manual", - "last_commit": "deadbeef" * 5, - }), - encoding="utf-8", - ) - _write_shadow(sd, "foo.py.md", """\ - # Shadow: foo.py - - **Language**: Python | **Lines**: 10 - - ## `bar` - - - A neat bug. - _(verified, source: exploration, labels: [bug])_ - """) - for rel, content in (extras or {}).items(): - _write_shadow(sd, rel, content) - return sd - - -# =========================================================================== -# view_summary -# =========================================================================== - - -class TestViewSummary: - """In-process tests for `view_summary(shadow_dir)`.""" - - def test_basic_header_on_coupon_demo( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Shadow Knowledge Base Summary" in out - assert "=" * 50 in out - assert "Files shadowed:" in out - assert "Symbols tracked:" in out - assert "Discoveries:" in out - assert "Preferences:" in out - assert "Cross-cutting:" in out - - def test_counts_reflect_fixture( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - # 3 source files, 33 discoveries, 3 cross-cutting (per fixture) - assert "Files shadowed: 3" in out - assert "Discoveries: 33" in out - assert "Cross-cutting: 3" in out - - def test_by_source_section_renders( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "By source:" in out - # All discoveries in the fixture are source: exploration - assert "exploration" in out - - def test_by_status_section_renders( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "By status:" in out - assert "verified" in out - - def test_by_label_section_renders( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "By label:" in out - assert "bug" in out - assert "security" in out - - def test_per_file_table_lists_files( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - # Per-file table header - assert "File" in out - assert "Symbols" in out - assert "Disc." in out - # All three fixture files appear in the table - assert "cart.py" in out - assert "inventory.py" in out - assert "test_cart.py" in out - - def test_cross_cutting_titles_section( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Cross-cutting discoveries:" in out - assert "Coupon case normalization mismatch" in out - assert "[edge-case]" in out - - def test_state_info_section( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Last update:" in out - assert "Last commit:" in out - # The fixture state.json says last_update_type: dream - assert "(dream)" in out - - def test_empty_shadow_dir_renders_zeros( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Files shadowed: 0" in out - assert "Discoveries: 0" in out - # With zero discoveries there's no source/status breakdown - assert "By source:" not in out - assert "By status:" not in out - assert "By label:" not in out - # And no state info (no state.json) - assert "Last update:" not in out - - def test_corrupted_state_json_does_not_break_summary( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # Clobber state.json with junk - (sd / "_meta" / "state.json").write_text( - "{ not json", encoding="utf-8" - ) - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - # Header and counts still rendered - assert "Shadow Knowledge Base Summary" in out - assert "Files shadowed: 1" in out - # State section silently dropped - assert "Last update:" not in out - - def test_unreadable_shadow_file_does_not_break_summary( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # Create a file with invalid UTF-8 — parse_shadow_file records a - # parse_error but returns a result. view_summary should still - # render the rest of the report. - bad = sd / "bad.py.md" - bad.write_bytes(b"\xff\xfe\x00garbage\x00") - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Shadow Knowledge Base Summary" in out - assert "Files shadowed: 2" in out - - def test_more_than_20_files_truncated( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - for i in range(25): - _write_shadow( - sd, f"file{i:02d}.py.md", - f"# Shadow: file{i:02d}.py\n\n## `sym{i}`\n\n" - f"- D\n _(verified, source: exploration)_\n", - ) - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Files shadowed: 25" in out - assert "... and 5 more files" in out - - def test_no_cross_cutting_dir_omits_section( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # no _cross/ dir - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Cross-cutting: 0" in out - # The titled list is suppressed when empty - assert "Cross-cutting discoveries:" not in out - - def test_summary_no_prefs(self, shadow_knowledge, tmp_path, capsys): - sd = _make_minimal_shadow(tmp_path) - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Preferences: 0" in out - - def test_summary_with_prefs(self, shadow_knowledge, tmp_path, capsys): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Pref one. - _(source: user)_ - - Pref two. - _(source: interaction)_ - """) - shadow_knowledge.view_summary(sd) - out = capsys.readouterr().out - assert "Preferences: 2" in out - - -# =========================================================================== -# view_search -# =========================================================================== - - -class TestViewSearch: - """In-process tests for `view_search(shadow_dir, query)`.""" - - def test_finds_text_match(self, shadow_knowledge, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "tax") - out = capsys.readouterr().out - assert "Search: 'tax'" in out - assert "results" in out - # The "8% tax" / "Tax rate" discoveries on cart.py match - assert "cart.py" in out - - def test_finds_symbol_match(self, shadow_knowledge, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "COUPON_CACHE") - out = capsys.readouterr().out - assert "COUPON_CACHE" in out - assert "cart.py" in out - - def test_finds_file_name_match( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Searching for the literal file name pulls every discovery in - # that file (file_name_hit branch). - shadow_knowledge.view_search(sd, "inventory.py") - out = capsys.readouterr().out - assert "inventory.py" in out - assert "matches" in out - - def test_case_insensitive(self, shadow_knowledge, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "COUPON") - upper = capsys.readouterr().out - shadow_knowledge.view_search(sd, "coupon") - lower = capsys.readouterr().out - # Same number of result lines either way - assert ("results" in upper) and ("results" in lower) - # And both contain at least one match - assert "::" in upper - assert "::" in lower - - def test_no_matches_prints_no_results( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "zzzdoesnotexistzzz") - out = capsys.readouterr().out - assert "No results for 'zzzdoesnotexistzzz'." in out - - def test_finds_cross_cutting_title( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Title of _cross/coupon-case-normalization-mismatch.md is - # "Coupon case normalization mismatch" - shadow_knowledge.view_search(sd, "normalization mismatch") - out = capsys.readouterr().out - assert "Cross-cutting" in out - assert "Coupon case normalization mismatch" in out - assert "Category:" in out - assert "edge-case" in out - - def test_finds_cross_cutting_by_ref( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Search for a ref that appears in a _cross file - shadow_knowledge.view_search(sd, "apply_bulk_discount") - out = capsys.readouterr().out - assert "Cross-cutting" in out - assert "Mutation through discount pipeline" in out - - def test_finds_also_involves_match( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - Also involves: `b.py::weird_symbol_zzz` - """) - shadow_knowledge.view_search(sd, "weird_symbol_zzz") - out = capsys.readouterr().out - # The match flag is "also_involves" and a line shows that ref - assert "weird_symbol_zzz" in out - assert "Also involves:" in out - - def test_finds_preference_match( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Prefer rusty pelicans for everything. - _(source: user)_ - """) - shadow_knowledge.view_search(sd, "pelican") - out = capsys.readouterr().out - assert "Preferences" in out - assert "pelican" in out.lower() - assert "[user]" in out - - def test_groups_per_file_results_by_file( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "coupon") - out = capsys.readouterr().out - # Per-file groups present a header like "cart.py (N matches)" - assert re.search(r"cart\.py \(\d+ matches\)", out) - - def test_results_show_status_and_source( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "tax") - out = capsys.readouterr().out - assert "(verified, source: exploration)" in out - - def test_total_count_in_header( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_search(sd, "coupon") - out = capsys.readouterr().out - m = re.search(r"\((\d+) results\)", out) - assert m is not None - assert int(m.group(1)) > 0 - - def test_empty_shadow_returns_no_results( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_knowledge.view_search(sd, "anything") - out = capsys.readouterr().out - assert "No results for 'anything'." in out - - -# =========================================================================== -# view_prefs -# =========================================================================== - - -class TestViewPrefs: - """In-process tests for `view_prefs(shadow_dir)`.""" - - def test_missing_prefs_file(self, shadow_knowledge, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - shadow_knowledge.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - def test_empty_prefs_placeholder( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - _No preferences recorded yet._ - """) - shadow_knowledge.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - def test_populated_prefs_lists_count_and_sources( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - Use snake_case. - _(source: user)_ - - Avoid mutable default args. - _(source: interaction)_ - - Prefer fail-fast for required deps. - _(source: user)_ - """) - shadow_knowledge.view_prefs(sd) - out = capsys.readouterr().out - assert "Project Preferences (3 total)" in out - assert "[user]" in out - assert "[interaction]" in out - assert "snake_case" in out - assert "fail-fast" in out - assert "mutable default" in out - - def test_against_coupon_demo_is_empty( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_prefs(sd) - out = capsys.readouterr().out - assert "No preferences recorded yet." in out - - -# =========================================================================== -# view_labels -# =========================================================================== - - -class TestViewLabels: - """In-process tests for `view_labels(shadow_dir, label_filter)`.""" - - @pytest.mark.parametrize("label", [ - "bug", "security", "performance", "feature-gap", "tech-debt", - ]) - def test_each_label_returns_results_on_fixture( - self, shadow_knowledge, coupon_demo, capsys, label - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels(sd, label) - out = capsys.readouterr().out - assert f"label(s): {label}" in out - assert f"[{label}]" in out - # Result header always has "(N results)" - m = re.search(r"\((\d+) results\)", out) - assert m and int(m.group(1)) >= 1 - - def test_unknown_label_no_results( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels(sd, "zznotalabelzz") - out = capsys.readouterr().out - assert "No discoveries with label(s): zznotalabelzz" in out - - def test_multiple_labels_comma_split( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels(sd, "bug,security") - out = capsys.readouterr().out - assert "label(s): bug, security" in out - # Both grouped section headers present - assert "[bug]" in out - assert "[security]" in out - - def test_label_filter_is_lowercased( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels(sd, "BUG") - out = capsys.readouterr().out - # Filter is lowercased before matching - assert "label(s): bug" in out - assert "[bug]" in out - - def test_cross_cutting_labels_included( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels( - sd, "bug", options=shadow_knowledge.RetrievalOptions(limit=20, max_chars=8000), - ) - out = capsys.readouterr().out - # Cross-cutting entries are prefixed with `_cross/` in the - # file column. - assert "_cross/" in out - - def test_also_labeled_displayed( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # One cart.py discovery has labels [bug, performance] - shadow_knowledge.view_labels(sd, "bug") - out = capsys.readouterr().out - assert "Also labeled:" in out - - def test_empty_shadow_no_results( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_knowledge.view_labels(sd, "bug") - out = capsys.readouterr().out - assert "No discoveries with label(s): bug" in out - - def test_each_result_row_has_file_and_symbol( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_labels(sd, "feature-gap") - out = capsys.readouterr().out - # `::` separator for file::symbol form - assert "::" in out - assert "(verified, source: exploration)" in out - - -# =========================================================================== -# view_recent -# =========================================================================== - - -class TestViewRecent: - """In-process tests for `view_recent(shadow_dir, count)`.""" - - def test_default_count_caps_at_10( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_recent(sd, 10) - out = capsys.readouterr().out - assert "Most Recent Discoveries (top 10)" in out - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - # Fixture has 33 per-file + 3 cross + 0 prefs > 10 - assert len(entries) == 10 - - def test_custom_count(self, shadow_knowledge, coupon_demo, capsys): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_recent(sd, 3) - out = capsys.readouterr().out - assert "Most Recent Discoveries (top 3)" in out - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - assert len(entries) == 3 - - def test_count_larger_than_available( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - shadow_knowledge.view_recent(sd, 50) - out = capsys.readouterr().out - # Only 1 discovery exists in the minimal shadow - entries = [ - l for l in out.splitlines() if l.strip().startswith("[20") - ] - assert len(entries) == 1 - - def test_empty_shadow_prints_nothing( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - shadow_knowledge.view_recent(sd, 10) - out = capsys.readouterr().out - assert "No discoveries found." in out - - def test_recently_modified_file_appears_first( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Bump test_cart.py's mtime to "now" so its discoveries should - # rank first. - target = sd / "test_cart.py.md" - now = datetime.now().timestamp() - os.utime(target, (now + 10, now + 10)) - shadow_knowledge.view_recent(sd, 3) - out = capsys.readouterr().out - # First entry block should reference test_cart.py - first_entry_idx = out.find("[20") - first_block = out[first_entry_idx:first_entry_idx + 400] - assert "test_cart.py" in first_block - - def test_includes_cross_cutting_type( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Bump a cross file mtime so it shows up in the top - cf = sd / "_cross" / "coupon-case-normalization-mismatch.md" - now = datetime.now().timestamp() - os.utime(cf, (now + 100, now + 100)) - shadow_knowledge.view_recent(sd, 5) - out = capsys.readouterr().out - assert "(cross-cutting)" in out - - def test_includes_preferences_when_present( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - _write_shadow(sd, "_prefs.md", """\ - # Preferences - - - A pref we care about. - _(source: user)_ - """) - shadow_knowledge.view_recent(sd, 10) - out = capsys.readouterr().out - assert "(preference)" in out - assert "A pref we care about" in out - # Preference-typed rows show `source:` not `(verified, ...)` - assert "source: user" in out - - def test_no_cross_dir_does_not_crash( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_minimal_shadow(tmp_path) - # No _cross/ created - shadow_knowledge.view_recent(sd, 10) - out = capsys.readouterr().out - assert "Most Recent Discoveries" in out - - def test_entries_include_timestamp_and_kind( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - shadow_knowledge.view_recent(sd, 2) - out = capsys.readouterr().out - # Entry header line format: " [YYYY-MM-DD HH:MM] (kind)" - assert re.search( - r"\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}\] \((discovery|cross-cutting|preference)\)", - out, - ) - - -# =========================================================================== -# view_check_invariants -# =========================================================================== - - -class TestViewCheckInvariants: - """In-process tests for `view_check_invariants(shadow_dir)`. - - Each test builds a deliberately broken `.shadow/` tree in tmp_path - and asserts that the right violation kind is reported. - """ - - def test_clean_coupon_demo_returns_zero( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = shadow_knowledge.view_check_invariants(coupon_demo / ".shadow") - out = capsys.readouterr().out - assert rc == 0 - assert "✓ Invariants OK" in out - - def test_empty_shadow_returns_zero( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0 - assert "Invariants OK" in out - - def test_missing_cross_file_is_violation( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - (sd / "_cross" / "coupon-case-normalization-mismatch.md").unlink() - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "cross-ref" in out - assert "coupon-case-normalization-mismatch" in out - - def test_invalid_status_enum(self, shadow_knowledge, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad status. - _(maybeverified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "maybeverified" in out - - def test_invalid_source_enum(self, shadow_knowledge, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad source. - _(verified, source: psychic)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "psychic" in out - - def test_invalid_label(self, shadow_knowledge, tmp_path, capsys): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Bad label. - _(verified, source: exploration, labels: [unicorn])_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "unicorn" in out - - def test_heading_without_backticks( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_in_backticks - - - Hi. - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "heading" in out - - def test_file_level_heading_does_not_violate( - self, shadow_knowledge, tmp_path, capsys - ): - """`## File-Level` is allowed without backticks.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## File-Level - - - A file-level discovery. - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0, out - assert "Invariants OK" in out - - def test_also_involves_without_backticks( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - With bad anchors. - _(verified, source: exploration)_ - Also involves: b.py::bar, c.py::baz - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "anchor" in out - - def test_also_involves_missing_symbol_after_colons( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - Empty sym. - _(verified, source: exploration)_ - Also involves: `b.py::` - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - # The empty-symbol form fails the strict regex (which requires - # non-empty after ::), so the file_sym_re finds zero anchors and - # we hit the "needs backtick anchors" branch instead. - assert "anchor" in out - assert "needs `file::symbol`" in out - - def test_cross_missing_category( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - """) - _write_shadow(sd, "_cross/no-cat.md", """\ - # No category here - - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "Category" in out - - def test_cross_invalid_category( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/bad-cat.md", """\ - # Bad category here - - **Category**: bogus - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "enum" in out - assert "bogus" in out - - def test_cross_missing_metadata_line( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-meta.md", """\ - # No meta here - - **Category**: pattern - **Refs**: - - `a.py::foo` - - **Discovery**: Something. - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "missing trailing" in out - - def test_cross_missing_refs_block( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/no-refs.md", """\ - # No refs here - - **Category**: pattern - - **Discovery**: Something. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "schema" in out - assert "Refs" in out - - def test_cross_ref_missing_symbol( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - # Note: file_sym_re demands "::" in backticks. We use a backtick - # ref with no symbol after `::`. - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - hi. - _(verified, source: exploration)_ - - ## Cross-References - - - [bad-anchor](_cross/bad-anchor.md) - """) - _write_shadow(sd, "_cross/bad-anchor.md", """\ - # Bad anchor - - **Category**: pattern - **Refs**: - - `a.py::` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "anchor" in out - - def test_back_pointer_missing( - self, shadow_knowledge, tmp_path, capsys - ): - """_cross/x.md references a.py::foo but a.py.md has no - Cross-References section pointing back to x.md.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## `foo` - - - A discovery. - _(verified, source: exploration)_ - """) - _write_shadow(sd, "_cross/orphan.md", """\ - # Orphan - - **Category**: pattern - **Refs**: - - `a.py::foo` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "cross-ref" in out - assert "does not link back to" in out - - def test_back_pointer_references_nonexistent_file( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "_cross/ghost.md", """\ - # Ghost - - **Category**: pattern - **Refs**: - - `does/not/exist.py::foo` - - **Discovery**: stuff. - - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - assert "no such shadow file exists" in out - - def test_violation_line_format( - self, shadow_knowledge, tmp_path, capsys - ): - """Output is grep-friendly: `path:line: kind: message`.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_backticked - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 1 - # At least one line of form path:line: kind: msg - assert re.search(r"a\.py\.md:\d+: heading: ", out) - - def test_multiple_violations_reported( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## not_backticked - - ## `foo` - - - bad. - _(maybeverified, source: psychic, labels: [unicorn])_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - err = capsys.readouterr().err - assert rc == 1 - # heading + 3 enum violations = at least 4 lines - violation_lines = [ - l for l in out.splitlines() - if re.match(r"^[^:]+:\d+: \w+: ", l) - ] - assert len(violation_lines) >= 3 - - def test_count_summary_emitted_on_stderr( - self, shadow_knowledge, tmp_path, capsys - ): - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", "## not_backticked\n") - rc = shadow_knowledge.view_check_invariants(sd) - captured = capsys.readouterr() - assert rc == 1 - # Final count line goes to stderr - assert "invariant violation(s) found" in captured.err - - def test_special_headings_allowed( - self, shadow_knowledge, tmp_path, capsys - ): - """`## Notes`, `## Metadata`, `## File-Level Notes` are allowed - without backticks.""" - sd = _make_shadow_root(tmp_path) - _write_shadow(sd, "a.py.md", """\ - # Shadow: a.py - - ## Notes - - Some prose. - - ## Metadata - - Some metadata. - - ## File-Level Notes - - More prose. - - ## `real_sym` - - - A discovery. - _(verified, source: exploration)_ - """) - rc = shadow_knowledge.view_check_invariants(sd) - out = capsys.readouterr().out - assert rc == 0, out - - -# =========================================================================== -# main() — in-process via sys.argv (covers the dispatcher) -# =========================================================================== - - -class TestMainInProcess: - """Drive `main()` directly so coverage of the dispatch arms is captured. - - All paths use `--shadow-dir ` to avoid relying - on cwd. Where we need a true CLI smoke (e.g., to verify `--help` - output), use subprocess. - """ - - def _shadow(self, coupon_demo): - return str(coupon_demo / ".shadow") - - def test_summary_dispatch( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, ["--shadow-dir", self._shadow(coupon_demo), - "--summary"] - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_no_args_default_is_summary( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, ["--shadow-dir", self._shadow(coupon_demo)] - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_search_dispatch( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--search", "coupon"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Search: 'coupon'" in out - - def test_prefs_dispatch( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), "--prefs"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "No preferences recorded yet." in out - - def test_labels_dispatch_bug( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--labels", "bug"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "label(s): bug" in out - - def test_recent_dispatch_with_n( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--recent", "4"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Most Recent Discoveries (top 4)" in out - - def test_recent_dispatch_no_n( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), "--recent"], - ) - out = capsys.readouterr().out - assert rc == 0 - # Default count is 10 - assert "Most Recent Discoveries (top 10)" in out - - def test_top_dispatch( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--top", "cart.py"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "for cart.py:" in out - - def test_top_dispatch_with_labels_and_limits( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--top", "cart.py", - "--top-labels", "bug", - "--top-limit", "2", - "--top-max-chars", "0"], - ) - out = capsys.readouterr().out - assert rc == 0 - bullets = [l for l in out.splitlines() if l.startswith("- [")] - assert len(bullets) <= 2 - - def test_check_invariants_clean_exits_zero( - self, shadow_knowledge, coupon_demo, capsys - ): - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", self._shadow(coupon_demo), - "--check-invariants"], - ) - out = capsys.readouterr().out - assert rc == 0 - assert "Invariants OK" in out - - def test_check_invariants_dirty_exits_one( - self, shadow_knowledge, coupon_demo, capsys - ): - sd = coupon_demo / ".shadow" - # Inject a heading violation - cart = sd / "cart.py.md" - cart.write_text( - cart.read_text(encoding="utf-8").replace( - "## `COUPON_CACHE`", "## COUPON_CACHE", 1 - ), - encoding="utf-8", - ) - rc = _call_main( - shadow_knowledge, - ["--shadow-dir", str(sd), "--check-invariants"], - ) - out = capsys.readouterr().out - assert rc == 1 - assert "heading" in out - - def test_explicit_shadow_dir_missing( - self, shadow_knowledge, tmp_path, capsys - ): - bogus = tmp_path / "does-not-exist" - rc = _call_main( - shadow_knowledge, ["--shadow-dir", str(bogus), "--summary"] - ) - err = capsys.readouterr().err - assert rc == 1 - assert "No .shadow/ directory found" in err - - def test_auto_detect_via_chdir( - self, shadow_knowledge, coupon_demo, monkeypatch, capsys - ): - """No --shadow-dir: cwd is walked up to find .shadow/.""" - monkeypatch.chdir(coupon_demo) - rc = _call_main(shadow_knowledge, ["--summary"]) - out = capsys.readouterr().out - assert rc == 0 - assert "Shadow Knowledge Base Summary" in out - - def test_auto_detect_no_shadow_in_cwd( - self, shadow_knowledge, tmp_path, monkeypatch, capsys - ): - monkeypatch.chdir(tmp_path) - rc = _call_main(shadow_knowledge, ["--summary"]) - err = capsys.readouterr().err - assert rc == 1 - assert "No .shadow/ directory found" in err - - -# --- main(): subprocess smoke (covers true argv parsing, help, errors) ---- - - -class TestMainSubprocess: - """End-to-end CLI smoke. Subprocess output is the contract here — - these don't add coverage but they catch dispatcher / argparse regressions - the in-process tests can't (e.g. --help, mutually exclusive errors).""" - - @pytest.mark.slow - @pytest.mark.integration - def test_help_exits_zero(self, repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--help") - assert r.returncode == 0 - # argparse prints usage and the description - assert "usage:" in r.stdout.lower() - assert "--summary" in r.stdout - assert "--search" in r.stdout - assert "--check-invariants" in r.stdout - - @pytest.mark.slow - @pytest.mark.integration - def test_unknown_flag_exits_nonzero(self, repo_root, coupon_demo): - r = _run_viewer(repo_root, coupon_demo, "--no-such-flag") - assert r.returncode != 0 - assert "unrecognized" in r.stderr or "unrecognized" in r.stdout - - @pytest.mark.slow - @pytest.mark.integration - def test_mutually_exclusive_flags(self, repo_root, coupon_demo): - """--summary and --prefs are in the same exclusive group.""" - r = _run_viewer( - repo_root, coupon_demo, "--summary", "--prefs" - ) - assert r.returncode != 0 - # argparse error mentions "not allowed with" - assert "not allowed with" in r.stderr diff --git a/tests/skills/shadow_frog/test_retrieval.py b/tests/skills/shadow_frog/test_retrieval.py deleted file mode 100644 index a89447c..0000000 --- a/tests/skills/shadow_frog/test_retrieval.py +++ /dev/null @@ -1,718 +0,0 @@ -"""Core agent retrieval: citation ranking, bounded context, and stable pagination.""" - -from dataclasses import replace -import io -import os -import re -import shutil -import sqlite3 -import subprocess -import sys -import time -import unicodedata - -import pytest - - -def make_shadow(tmp_path, count=4, text_size=20): - shadow = tmp_path / ".shadow" - shadow.mkdir(exist_ok=True) - path = shadow / "source.py.md" - path.write_text( - "# Shadow: source.py\n\n## `run`\n\n" + "\n".join( - f"- Claim {index:05}: " + ("details " * text_size) + - "\n _(verified, source: exploration, labels: [bug])_\n" - for index in range(count) - ) + "\n## Cross-References\n", - encoding="utf-8", - ) - return shadow - - -def ids(text): - return re.findall(r"\bid=(d_[0-9a-f]{32})", text) - - -def cursor(text): - match = re.search(r"--cursor ([0-9a-f]{32}:\d+)", text) - return match.group(1) if match else None - - -def text_cursor(text): - match = re.search(r"--text-cursor (\S+)", text) - return match.group(1) if match else None - - -def test_only_emitted_entries_count_and_no_markdown_changes(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 12) - before = {path: path.read_bytes() for path in shadow.rglob("*") if path.is_file()} - entries = shadow_knowledge._knowledge_entries(shadow) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - assert store.scores([entry["id"] for entry in entries]) == {} - shadow_knowledge.view_search( - shadow, "Claim", options=shadow_knowledge.RetrievalOptions(limit=2, event_id="first"), - ) - output = capsys.readouterr().out - emitted = ids(output) - assert len(emitted) == 2 and "citation_score=0" in output - assert store.scores([entry["id"] for entry in entries]) == dict.fromkeys(emitted, 1) - assert all(path.read_bytes() == content for path, content in before.items()) - assert sorted(path.name for path in shadow.iterdir()) == ["source.py.md"] - - -def test_retries_and_no_record_do_not_double_count(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - options = shadow_knowledge.RetrievalOptions(event_id="retry") - for _ in range(2): - shadow_knowledge.view_search(shadow, "Claim", options=options) - output = capsys.readouterr().out - identity = ids(output)[0] - store = shadow_knowledge.CitationStore.for_shadow(shadow) - assert store.scores([identity]) == {identity: 1} - shadow_knowledge.view_get(shadow, identity, options=replace(options, record=False)) - capsys.readouterr() - assert store.scores([identity]) == {identity: 1} - - -def test_ten_thousand_claims_have_bounded_output(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 10000) - options = shadow_knowledge.RetrievalOptions(limit=10, max_chars=1800) - started = time.monotonic() - shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) - output = capsys.readouterr().out - assert len(output) <= 1800 - assert "10000 results" in output and len(ids(output)) <= 10 - assert ids(output) and cursor(output) - assert time.monotonic() - started < 10 - store = shadow_knowledge.CitationStore.for_shadow(shadow) - all_ids = [entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)] - assert len(all_ids) == len(set(all_ids)) == 10000 - assert store.scores(all_ids) == dict.fromkeys(ids(output), 1) - - -def test_cursor_is_stable_despite_other_readers_updating_scores(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 23, text_size=2) - options = shadow_knowledge.RetrievalOptions(limit=3, max_chars=1400) - entries = shadow_knowledge._knowledge_entries(shadow) - expected = {entry["id"] for entry in entries} - found = [] - next_cursor = None - while True: - shadow_knowledge.view_symbol( - shadow, "source.py::run", options=replace(options, cursor=next_cursor), - ) - output = capsys.readouterr().out - assert len(output) <= 1400 - found.extend(ids(output)) - next_cursor = cursor(output) - if next_cursor is None: - break - shadow_knowledge.CitationStore.for_shadow(shadow).record( - [entries[-1]["id"]], f"concurrent-{len(found)}", - ) - assert len(found) == len(set(found)) == 23 - assert set(found) == expected - - -def test_changed_knowledge_invalidates_cursor(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path) - options = shadow_knowledge.RetrievalOptions(limit=1) - shadow_knowledge.view_search(shadow, "Claim", options=options) - previous = cursor(capsys.readouterr().out) - path = shadow / "source.py.md" - path.write_text(path.read_text(encoding="utf-8").replace("details", "different"), encoding="utf-8") - with pytest.raises(ValueError, match="changed"): - shadow_knowledge.view_search(shadow, "Claim", options=replace(options, cursor=previous)) - - -def test_removed_results_invalidate_cursor_instead_of_claiming_empty_success(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path) - options = shadow_knowledge.RetrievalOptions(limit=1) - shadow_knowledge.view_search(shadow, "Claim", options=options) - previous = cursor(capsys.readouterr().out) - (shadow / "source.py.md").unlink() - with pytest.raises(ValueError, match="changed"): - shadow_knowledge.view_search(shadow, "Claim", options=replace(options, cursor=previous)) - - -def test_popularity_never_overrides_trust_and_allows_new_claim(shadow_knowledge, tmp_path): - entries = [ - {"id": "user", "source": "user", "status": "verified"}, - {"id": "popular", "source": "exploration", "status": "verified"}, - {"id": "popular2", "source": "exploration", "status": "verified"}, - {"id": "popular3", "source": "exploration", "status": "verified"}, - {"id": "new", "source": "exploration", "status": "verified"}, - {"id": "refuted", "source": "user", "status": "refuted"}, - ] - scores = {"popular": 100, "popular2": 90, "popular3": 80, "refuted": 10000} - ranked = shadow_knowledge._rank_entries(entries, scores, 3) - assert ranked[0]["id"] == "user" and ranked[-1]["id"] == "refuted" - assert [entry["id"] for entry in ranked][:3] == ["user", "popular", "new"] - - -def test_zero_score_opportunity_accounts_for_character_budget(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 7) - entries = shadow_knowledge._knowledge_entries(shadow) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - store.record([entry["id"] for entry in entries[:-1]], "prior-read") - shadow_knowledge.view_search( - shadow, "Claim", options=shadow_knowledge.RetrievalOptions(limit=10, max_chars=1000), - ) - output = capsys.readouterr().out - assert len(output) <= 1000 - assert len(ids(output)) >= 2 - assert entries[-1]["id"] in ids(output) - - -@pytest.mark.parametrize("view", ["top", "symbol"]) -def test_zero_score_slot_accounts_for_higher_trust_rows(shadow_knowledge, tmp_path, capsys, view): - shadow = make_shadow(tmp_path, 5, text_size=2) - path = shadow / "source.py.md" - path.write_text( - path.read_text(encoding="utf-8").replace("source: exploration", "source: user", 1), - encoding="utf-8", - ) - entries = shadow_knowledge._knowledge_entries(shadow) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - store.record([entry["id"] for entry in entries[1:4]], "prior") - options = shadow_knowledge.RetrievalOptions(limit=3, max_chars=600) - if view == "top": - shadow_knowledge.view_top(shadow, "source.py", "bug", 3, 600, options=options) - else: - shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) - output = capsys.readouterr().out - assert len(output) <= 600 - assert ids(output)[0] == entries[0]["id"] - assert entries[-1]["id"] in ids(output) - - -@pytest.mark.parametrize("file", ["cart.py", "inventory.py", "test_cart.py"]) -def test_default_hook_budget_returns_multiple_warnings(shadow_knowledge, coupon_demo, capsys, file): - shadow_knowledge.view_top(coupon_demo / ".shadow", file, "bug,security", 3, 600) - output = capsys.readouterr().out - assert len(output) <= 600 - assert len(ids(output)) == 3 - assert "citation_score=" not in output - - -def test_metadata_changes_preserve_identity_but_not_claim_changes(shadow_knowledge, tmp_path): - shadow = make_shadow(tmp_path, 1) - original = shadow_knowledge._knowledge_entries(shadow)[0]["id"] - path = shadow / "source.py.md" - path.write_text( - path.read_text(encoding="utf-8").replace( - "verified, source: exploration, labels: [bug]", "uncertain, source: interaction" - ), - encoding="utf-8", - ) - assert shadow_knowledge._knowledge_entries(shadow)[0]["id"] == original - path.write_text(path.read_text(encoding="utf-8").replace("Claim", "Different claim"), encoding="utf-8") - assert shadow_knowledge._knowledge_entries(shadow)[0]["id"] != original - - -def test_duplicate_preferences_share_one_identity_without_losing_trust(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - (shadow / "_prefs.md").write_text( - "# Preferences\n\n- Preserve the contract.\n _(source: interaction)_\n" - "\n- Preserve the contract.\n _(source: user)_\n", - encoding="utf-8", - ) - shadow_knowledge.view_prefs(shadow) - output = capsys.readouterr().out - assert "Project Preferences (1 total)" in output and "[user]" in output - assert len(ids(output)) == 1 - assert shadow_knowledge.CitationStore.for_shadow(shadow).scores(ids(output)) == dict.fromkeys(ids(output), 1) - - -@pytest.mark.parametrize("kind", ["class", "interface", "enum", "trait", "struct", "protocol", "module"]) -def test_container_symbols_use_canonical_anchors(shadow_knowledge, shadow_init, tmp_path, capsys, kind): - shadow = make_shadow(tmp_path, 1) - heading = shadow_init.Symbol("Container", kind).heading_text - path = shadow / "source.py.md" - path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{heading}`"), encoding="utf-8") - shadow_knowledge.view_symbol(shadow, "source.py::Container") - result = capsys.readouterr().out - assert "Claim 00000" in result and "source.py::Container" in result - shadow_knowledge.view_get(shadow, ids(result)[0]) - assert "source.py::Container" in capsys.readouterr().out - - -@pytest.mark.parametrize("query", ["src/auth.py", unicodedata.normalize("NFD", "Src/caf\u00e9.py")]) -def test_filesystem_alias_ids_expand_and_include_cross_refs(shadow_knowledge, tmp_path, capsys, query): - actual = "Src/Auth.py" if query == "src/auth.py" else "Src/caf\u00e9.py" - shadow = tmp_path / ".shadow" - path = shadow / f"{actual}.md" - path.parent.mkdir(parents=True) - path.write_text( - f"# Shadow: {actual}\n\n## `run`\n\n- Keep the audit trail.\n" - " _(verified, source: exploration, labels: [bug])_\n", - encoding="utf-8", - ) - if not (shadow / (query + ".md")).is_file(): - pytest.skip("Filesystem distinguishes these case/Unicode spellings") - cross = shadow / "_cross/audit.md" - cross.parent.mkdir() - cross.write_text( - f"# Audit\n\n**Category**: contract\n**Refs**:\n- `{actual}::run`\n\n" - "**Discovery**: Cross-file audit contract.\n\n_(verified, source: exploration)_\n", - encoding="utf-8", - ) - global_entries = shadow_knowledge._knowledge_entries(shadow) - shadow_knowledge.view_symbol(shadow, f"{query}::run") - result = capsys.readouterr().out - assert "Cross-file audit contract." in result and "Keep the audit trail." in result - assert set(ids(result)) == {entry["id"] for entry in global_entries} - for identity in ids(result): - shadow_knowledge.view_get(shadow, identity, options=shadow_knowledge.RetrievalOptions(record=False)) - assert identity in capsys.readouterr().out - - -def test_distinct_case_sensitive_files_are_not_folded(shadow_knowledge, tmp_path): - shadow = tmp_path / ".shadow" - shadow.mkdir() - upper, lower = shadow / "A.py.md", shadow / "a.py.md" - upper.write_text("# Shadow: A.py\n\n## `run`\n\n- Claim.\n", encoding="utf-8") - if lower.exists(): - pytest.skip("Filesystem does not support distinct case-only names") - lower.write_text("# Shadow: a.py\n\n## `run`\n\n- Claim.\n", encoding="utf-8") - assert len({entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)}) == 2 - - -def test_literal_whitespace_remains_separately_searchable(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 0) - path = shadow / "source.py.md" - path.write_text( - "# Shadow: source.py\n\n## `run`\n\n" - "- Key `a b` is accepted.\n _(verified, source: exploration)_\n\n" - "- Key `a b` is accepted.\n _(verified, source: exploration)_\n", - encoding="utf-8", - ) - entries = shadow_knowledge._knowledge_entries(shadow) - assert len(entries) == 2 and entries[0]["id"] != entries[1]["id"] - shadow_knowledge.view_search(shadow, "a b") - result = capsys.readouterr().out - assert ids(result) == [entries[1]["id"]] - shadow_knowledge.view_get(shadow, entries[1]["id"]) - assert "`a b`" in capsys.readouterr().out - - -def test_duplicate_claims_union_labels_and_preserve_stronger_source(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 0) - (shadow / "source.py.md").write_text( - "# Shadow: source.py\n\n## `run`\n\n" - "- Shared claim.\n _(verified, source: user, labels: [bug])_\n\n" - "- Shared claim.\n _(verified, source: exploration, labels: [security])_\n", - encoding="utf-8", - ) - entries = shadow_knowledge._knowledge_entries(shadow) - assert len(entries) == 1 - assert entries[0]["labels"] == ["bug", "security"] and entries[0]["source"] == "user" - shadow_knowledge.view_labels(shadow, "security") - assert ids(capsys.readouterr().out) == [entries[0]["id"]] - - -def test_installed_import_does_not_create_bytecode(repo_root, tmp_path): - installed = tmp_path / ".github/skills/shadow-frog" - shutil.copytree( - repo_root / "skills/shadow-frog", installed, - ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), - ) - env = os.environ.copy() - env.pop("PYTHONDONTWRITEBYTECODE", None) - env.pop("PYTHONPYCACHEPREFIX", None) - result = subprocess.run( - [sys.executable, str(installed / "shadow-read.py"), "--help"], - env=env, capture_output=True, text=True, encoding="utf-8", - ) - assert result.returncode == 0, result.stderr - assert not list(installed.rglob("*.pyc")) - - -def test_missing_home_does_not_hide_standalone_knowledge(shadow_knowledge, tmp_path, monkeypatch, capsys): - from pathlib import Path - - shadow = make_shadow(tmp_path, 1) - monkeypatch.delenv("XDG_STATE_HOME", raising=False) - monkeypatch.delenv("LOCALAPPDATA", raising=False) - - def missing_home(): - raise RuntimeError("Could not determine home directory") - - monkeypatch.setattr(Path, "home", missing_home) - shadow_knowledge.view_search(shadow, "Claim") - result = capsys.readouterr() - assert "Claim 00000" in result.out - assert "will not be recorded" in result.err and "home" in result.err - - -def test_get_chunks_long_claim_without_exceeding_budget(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1, text_size=500) - entry = shadow_knowledge._knowledge_entries(shadow)[0] - identity = entry["id"] - continuation = None - pieces = [] - while True: - shadow_knowledge.view_get( - shadow, identity, - options=shadow_knowledge.RetrievalOptions(max_chars=500, text_cursor=continuation), - ) - output = capsys.readouterr().out - assert len(output) <= 500 - continuation = text_cursor(output) - pieces.append(output.removesuffix("\n").split("\n", 2)[2].split("\nContinue:", 1)[0]) - if continuation is None: - break - assert "".join(pieces) == entry["anchor"] + "\n\n" + entry["text"] + "\nLabels: bug" - assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([identity]) == {identity: 1} - - -@pytest.mark.parametrize("change", ["labels", "source", "status", "text"]) -def test_expansion_rejects_changed_body_or_metadata(shadow_knowledge, tmp_path, capsys, change): - shadow = make_shadow(tmp_path, 1, text_size=200) - entry = shadow_knowledge._knowledge_entries(shadow)[0] - shadow_knowledge.view_get(shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(max_chars=500)) - continuation = text_cursor(capsys.readouterr().out) - path = shadow / "source.py.md" - replacements = { - "labels": ("labels: [bug]", "labels: [bug, security]"), - "source": ("source: exploration", "source: user"), - "status": ("verified,", "refuted,"), - "text": ("details ", "new details "), - } - path.write_text(path.read_text(encoding="utf-8").replace(*replacements[change]), encoding="utf-8") - with pytest.raises(ValueError, match="changed"): - shadow_knowledge.view_get( - shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(text_cursor=continuation), - ) - - -def test_no_record_continuation_stays_unrecorded(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1, text_size=200) - entry = shadow_knowledge._knowledge_entries(shadow)[0] - shadow_knowledge.view_get( - shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(max_chars=500, record=False), - ) - continuation = text_cursor(capsys.readouterr().out) - shadow_knowledge.view_get( - shadow, entry["id"], options=shadow_knowledge.RetrievalOptions(text_cursor=continuation), - ) - capsys.readouterr() - assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([entry["id"]]) == {} - - -@pytest.mark.parametrize("change", ["mtime", "other-symbol"]) -def test_nonrecent_pages_survive_irrelevant_file_changes(shadow_knowledge, tmp_path, capsys, change): - shadow = make_shadow(tmp_path) - options = shadow_knowledge.RetrievalOptions(limit=1) - shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) - first = capsys.readouterr().out - path = shadow / "source.py.md" - if change == "mtime": - timestamp = path.stat().st_mtime + 10 - os.utime(path, (timestamp, timestamp)) - else: - with path.open("a", encoding="utf-8") as stream: - stream.write("\n## `other`\n\n- Other knowledge.\n _(verified, source: exploration)_\n") - shadow_knowledge.view_symbol(shadow, "source.py::run", options=replace(options, cursor=cursor(first))) - assert not set(ids(first)) & set(ids(capsys.readouterr().out)) - - -@pytest.mark.parametrize("change", ["source", "labels", "status", "recent-mtime"]) -def test_pages_invalidate_on_relevant_metadata_changes(shadow_knowledge, tmp_path, capsys, change): - shadow = make_shadow(tmp_path) - options = shadow_knowledge.RetrievalOptions(limit=1) - view = shadow_knowledge.view_recent if change == "recent-mtime" else shadow_knowledge.view_symbol - args = [shadow] if change == "recent-mtime" else [shadow, "source.py::run"] - view(*args, options=options) - continuation = cursor(capsys.readouterr().out) - path = shadow / "source.py.md" - if change == "recent-mtime": - timestamp = path.stat().st_mtime + 10 - os.utime(path, (timestamp, timestamp)) - else: - replacements = { - "source": ("source: exploration", "source: user"), - "labels": ("labels: [bug]", "labels: [security]"), - "status": ("verified,", "refuted,"), - } - path.write_text(path.read_text(encoding="utf-8").replace(*replacements[change]), encoding="utf-8") - with pytest.raises(ValueError, match="changed"): - view(*args, options=replace(options, cursor=continuation)) - - -def test_summary_and_invariant_checks_do_not_count(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - shadow_knowledge.view_summary(shadow) - shadow_knowledge.view_check_invariants(shadow) - capsys.readouterr() - assert not store.path.exists() - - -def test_all_creation_sources_start_at_zero_without_schema_changes( - shadow_knowledge, dream_reconcile, tmp_path, capsys, -): - shadow = tmp_path / ".shadow" - discoveries = [ - {"anchor": f"source.py::run_{source}", "text": f"Behavior from {source}.", - "source": source, "status": "verified"} - for source in ("exploration", "user", "interaction") - ] - dream_reconcile.merge_discoveries( - str(tmp_path), - [("dream/test/20260923-000000Z-test", "20260923-000000Z-test", - {"discoveries": discoveries})], - ) - entries = shadow_knowledge._knowledge_entries(shadow) - assert len(entries) == 3 - assert shadow_knowledge.CitationStore.for_shadow(shadow).scores([entry["id"] for entry in entries]) == {} - shadow_knowledge.view_search(shadow, "Behavior", options=shadow_knowledge.RetrievalOptions(record=False)) - output = capsys.readouterr().out - assert output.count("citation_score=0") == 3 - assert "citation_score" not in (shadow / "source.py.md").read_text(encoding="utf-8") - - -def test_top_hard_budget_counts_only_visible_claims(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 10, text_size=150) - shadow_knowledge.view_top(shadow, "source.py", "bug", 3, 600) - output = capsys.readouterr().out - assert len(output) <= 600 - emitted = ids(output) - assert emitted and len(emitted) <= 3 - assert f"Top {len(emitted)} of 10" in output - store = shadow_knowledge.CitationStore.for_shadow(shadow) - assert store.scores([entry["id"] for entry in shadow_knowledge._knowledge_entries(shadow)]) == dict.fromkeys(emitted, 1) - - -def test_smallest_budget_still_emits_identifiable_content(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 10) - shadow_knowledge.view_search(shadow, "Claim", options=shadow_knowledge.RetrievalOptions(max_chars=256)) - output = capsys.readouterr().out - assert len(output) <= 256 and len(ids(output)) == 1 - assert "verified" in output and cursor(output) - - -def test_broken_ledger_returns_knowledge_with_warning(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - store.path.parent.mkdir(parents=True) - store.path.write_bytes(b"not sqlite") - shadow_knowledge.view_search(shadow, "Claim") - captured = capsys.readouterr() - assert "Claim 00000" in captured.out and "citation_score=?" in captured.out - assert "warning" in captured.err and "citation" in captured.err - assert store.path.read_bytes() == b"not sqlite" - - -@pytest.mark.parametrize("view", ["search", "symbol", "get", "prefs", "labels", "recent", "top"]) -def test_all_content_views_share_identity_and_score(shadow_knowledge, tmp_path, capsys, view): - shadow = make_shadow(tmp_path, 1, text_size=2) - (shadow / "_prefs.md").write_text("# Preferences\n\n- Keep the contract.\n _(source: user)_\n", encoding="utf-8") - options = shadow_knowledge.RetrievalOptions(event_id="same-visit") - entry = next(item for item in shadow_knowledge._knowledge_entries(shadow) if item["kind"] != "preference") - if view == "search": - shadow_knowledge.view_search(shadow, "Claim", options=options) - elif view == "symbol": - shadow_knowledge.view_symbol(shadow, "source.py::run", options=options) - elif view == "get": - shadow_knowledge.view_get(shadow, entry["id"], options=options) - elif view == "prefs": - shadow_knowledge.view_prefs(shadow, options=options) - elif view == "labels": - shadow_knowledge.view_labels(shadow, "bug", options=options) - elif view == "recent": - shadow_knowledge.view_recent(shadow, options=options) - else: - shadow_knowledge.view_top(shadow, "source.py", "", 3, 600, options=options) - output = capsys.readouterr().out - emitted = ids(output) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - assert emitted and store.scores(emitted) == dict.fromkeys(emitted, 1) - if view != "prefs": - assert entry["id"] in emitted - - -def test_write_failure_warns_without_hiding_knowledge(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] - store.record([identity], "initial") - connection = sqlite3.connect(store.path) - try: - connection.execute("BEGIN IMMEDIATE") - shadow_knowledge.view_search(shadow, "Claim") - captured = capsys.readouterr() - assert identity in captured.out - assert "this visit was not recorded" in captured.err - finally: - connection.rollback() - connection.close() - assert store.scores([identity])[identity] == 1 - - -def test_counting_can_recover_after_a_failed_score_read(shadow_knowledge, tmp_path, monkeypatch, capsys): - shadow = make_shadow(tmp_path, 1) - identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] - store = shadow_knowledge.CitationStore.for_shadow(shadow) - store.record([identity]) - lock = sqlite3.connect(store.path) - # A pre-existing rollback cache can still be locked during WAL activation. - lock.execute("PRAGMA journal_mode=DELETE") - lock.execute("BEGIN EXCLUSIVE") - - class UnlockOnOutput(io.StringIO): - def flush(self): - lock.rollback() - super().flush() - - output = UnlockOnOutput() - try: - with monkeypatch.context() as context: - context.setattr(sys, "stdout", output) - shadow_knowledge.view_search(shadow, "Claim") - finally: - lock.close() - warnings = capsys.readouterr().err - assert "ledger busy" in warnings and "was not recorded" not in warnings - assert "citation_score=?" in output.getvalue() - assert store.scores([identity])[identity] == 2 - - -def test_failed_stdout_does_not_increment_score(shadow_knowledge, tmp_path, monkeypatch): - shadow = make_shadow(tmp_path, 1) - store = shadow_knowledge.CitationStore.for_shadow(shadow) - identity = shadow_knowledge._knowledge_entries(shadow)[0]["id"] - - class ClosedOutput: - def write(self, text): - raise BrokenPipeError("reader closed the pipe") - - def flush(self): - pass - - monkeypatch.setattr(sys, "stdout", ClosedOutput()) - with pytest.raises(BrokenPipeError): - shadow_knowledge.view_search(shadow, "Claim") - assert store.scores([identity]) == {} - - -def test_symbol_context_includes_related_cross_but_not_other_symbols(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - with (shadow / "source.py.md").open("a", encoding="utf-8") as stream: - stream.write("\n## `unrelated`\n\n- Unrelated claim.\n _(verified, source: user)_\n") - cross = shadow / "_cross/related.md" - cross.parent.mkdir() - cross.write_text( - "# Related behavior\n\n**Category**: contract\n**Refs**:\n" - "- `source.py::run`\n- `other.py::call`\n\n" - "**Discovery**: Cross-file constraint.\n\n_(verified, source: user)_\n", - encoding="utf-8", - ) - shadow_knowledge.view_symbol(shadow, "source.py::run") - output = capsys.readouterr().out - assert "Cross-file constraint." in output - assert "Claim 00000" in output and "Unrelated claim." not in output - - -def test_nested_paths_have_one_identity_across_scoped_and_global_views(shadow_knowledge, tmp_path): - shadow = tmp_path / ".shadow" - source = shadow / "src/caf\u00e9 tools.py.md" - source.parent.mkdir(parents=True) - source.write_text( - "# Shadow: src/caf\u00e9 tools.py\n\n## `run`\n\n" - "- Nested claim.\n _(verified, source: exploration)_\n", - encoding="utf-8", - ) - global_entry = shadow_knowledge._knowledge_entries(shadow)[0] - scoped_entry = shadow_knowledge._knowledge_entries(shadow, "src/caf\u00e9 tools.py")[0] - assert global_entry["anchor"] == scoped_entry["anchor"] == "src/caf\u00e9 tools.py::run" - assert global_entry["id"] == scoped_entry["id"] - - -def test_expansion_preserves_long_anchors(shadow_knowledge, tmp_path, capsys): - shadow = make_shadow(tmp_path, 1) - path = shadow / "source.py.md" - symbol = "method_" + "name" * 50 - path.write_text(path.read_text(encoding="utf-8").replace("`run`", f"`{symbol}`"), encoding="utf-8") - entry = shadow_knowledge._knowledge_entries(shadow)[0] - shadow_knowledge.view_get(shadow, entry["id"]) - assert "source.py::" + symbol in capsys.readouterr().out - - -def test_cursor_cli_continuation_and_get_are_wired(repo_root, tmp_path): - shadow = make_shadow(tmp_path) - command = [ - sys.executable, str(repo_root / "skills/shadow-frog/shadow-read.py"), - "--shadow-dir", str(shadow), "--symbol", "source.py::run", - "--limit", "1", "--max-chars", "600", - ] - first = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", check=True) - second = subprocess.run( - [*command, "--cursor", cursor(first.stdout)], - capture_output=True, text=True, encoding="utf-8", check=True, - ) - assert ids(first.stdout) != ids(second.stdout) - expanded = subprocess.run( - command[:4] + ["--get", ids(first.stdout)[0], "--max-chars", "600", "--no-record"], - capture_output=True, text=True, encoding="utf-8", check=True, - ) - assert ids(expanded.stdout) == ids(first.stdout) - assert "citation_score=1" in expanded.stdout - - -def test_cli_text_continuation_is_revision_bound_and_counts_one_read(repo_root, tmp_path): - shadow = make_shadow(tmp_path, 1, text_size=300) - script = repo_root / "skills/shadow-frog/shadow-read.py" - prefix = [sys.executable, str(script), "--shadow-dir", str(shadow)] - found = subprocess.run( - [*prefix, "--symbol", "source.py::run", "--no-record"], - capture_output=True, text=True, encoding="utf-8", check=True, - ) - identity = ids(found.stdout)[0] - command = ["--get", identity, "--max-chars", "500"] - last = None - for _ in range(40): - result = subprocess.run( - [*prefix, *command], capture_output=True, text=True, encoding="utf-8", check=True, - ) - assert len(result.stdout) <= 500 - last = result.stdout - if "\nContinue: " not in result.stdout: - break - command = result.stdout.split("\nContinue: ", 1)[1].strip().split() - else: - pytest.fail("Text continuation did not finish") - assert "citation_score=1" in last - - -@pytest.mark.slow -def test_installed_layout_and_cli_options_are_wired(repo_root, coupon_demo, tmp_path): - import shutil - - for agent in (".github", ".claude"): - installed = tmp_path / agent / "skills/shadow-frog" - shutil.copytree(repo_root / "skills/shadow-frog", installed) - command = [ - sys.executable, str(installed / "shadow-read.py"), - "--shadow-dir", str(coupon_demo / ".shadow"), "--search", "coupon", - "--limit", "2", "--max-chars", "1000", "--event-id", agent, - ] - result = subprocess.run(command, capture_output=True, text=True, encoding="utf-8", check=True) - assert len(result.stdout) <= 1000 and len(ids(result.stdout)) == 2 - assert "citation_score=" in result.stdout - - -@pytest.mark.parametrize("args", [ - ["--summary", "--limit", "2"], - ["--search", "claim", "--limit", "0"], - ["--search", "claim", "--max-chars", "20"], - ["--search", "claim", "--text-cursor", "bad"], - ["--top", "source.py", "--max-chars", "600"], - ["--get", "d_" + "a" * 32, "--cursor", "bad"], -]) -def test_invalid_cli_combinations_are_explicit(repo_root, tmp_path, args): - result = subprocess.run( - [sys.executable, str(repo_root / "skills/shadow-frog/shadow-read.py"), *args], - cwd=tmp_path, capture_output=True, text=True, encoding="utf-8", - ) - assert result.returncode == 2 and "error:" in result.stderr diff --git a/tests/skills/shadow_frog/test_shadow_read.py b/tests/skills/shadow_frog/test_shadow_read.py deleted file mode 100644 index f5caef4..0000000 --- a/tests/skills/shadow_frog/test_shadow_read.py +++ /dev/null @@ -1,171 +0,0 @@ -"""Optional agent retrieval targets known files without depending on the Viewer.""" - -import json -import os -from pathlib import Path -import re -import shutil -import subprocess -import sys - -import pytest - - -def sample_shadow(tmp_path): - shadow = tmp_path / ".shadow" - shadow.mkdir() - (shadow / "source.py.md").write_text( - "# Shadow: source.py\n\n## File-Level\n\n" - "- Importing starts no workers.\n _(verified, source: user)_\n\n" - "## `class Worker`\n\n" - "- Construction opens no files.\n _(verified, source: exploration)_\n\n" - "### `Worker.run`\n\n" - "- Empty input returns immediately.\n _(verified, source: exploration)_\n\n" - "## Cross-References\n\n- [Worker contract](_cross/worker-contract.md)\n", - encoding="utf-8", - ) - cross = shadow / "_cross" - cross.mkdir() - (cross / "worker-contract.md").write_text( - "# Worker contract\n\n**Category**: contract\n**Refs**:\n" - "- `source.py::Worker.run`\n- `other.py::dispatch`\n\n" - "**Discovery**: Dispatch must retain worker ownership.\n\n" - "_(verified, source: exploration)_\n", - encoding="utf-8", - ) - return shadow - - -def invoke(script, shadow, *args): - return subprocess.run( - [sys.executable, str(script), "--shadow-dir", str(shadow), *args], - cwd=shadow.parent, capture_output=True, text=True, encoding="utf-8", timeout=15, - ) - - -def identities(text): - return re.findall(r"\bid=(d_[0-9a-f]{32})", text) - - -@pytest.mark.parametrize("layout", [".github", ".claude"]) -def test_core_only_install_supports_file_and_symbol_reads(repo_root, tmp_path, layout): - shadow = sample_shadow(tmp_path) - installed = tmp_path / layout / "skills/shadow-frog" - shutil.copytree( - repo_root / "skills/shadow-frog", installed, - ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), - ) - script = installed / "shadow-read.py" - assert not (installed.parent / "shadow-frog-viewer").exists() - file_view = invoke(script, shadow, "source.py") - assert file_view.returncode == 0, file_view.stderr - assert "Importing starts no workers." in file_view.stdout - assert "Construction opens no files." in file_view.stdout - assert "Empty input returns immediately." in file_view.stdout - assert "Dispatch must retain worker ownership." in file_view.stdout - symbol_view = invoke(script, shadow, "source.py::Worker.run") - assert symbol_view.returncode == 0, symbol_view.stderr - assert "Empty input returns immediately." in symbol_view.stdout - assert "Dispatch must retain worker ownership." in symbol_view.stdout - assert "Construction opens no files." not in symbol_view.stdout - assert "Importing starts no workers." not in symbol_view.stdout - assert set(identities(symbol_view.stdout)) <= set(identities(file_view.stdout)) - expanded = invoke(script, shadow, "--get", identities(symbol_view.stdout)[0]) - assert expanded.returncode == 0, expanded.stderr - - -def test_file_read_does_not_parse_unrelated_per_file_shadows(repo_root, tmp_path): - shadow = sample_shadow(tmp_path) - (shadow / "unrelated.py.md").write_bytes(b"\xff") - result = invoke(repo_root / "skills/shadow-frog/shadow-read.py", shadow, "--file", "source.py") - assert result.returncode == 0 - assert "Empty input" in result.stdout - assert result.stderr == "" - assert "unrelated" not in result.stdout - - -def test_file_and_symbol_positional_targets_conflict_with_other_views(repo_root, tmp_path): - shadow = sample_shadow(tmp_path) - result = invoke( - repo_root / "skills/shadow-frog/shadow-read.py", shadow, - "source.py", "--search", "workers", - ) - assert result.returncode == 2 - assert "positional target or an explicit view" in result.stderr - - -def test_reader_requires_an_explicit_target_or_operation(repo_root, tmp_path): - shadow = sample_shadow(tmp_path) - result = invoke(repo_root / "skills/shadow-frog/shadow-read.py", shadow) - assert result.returncode == 2 - assert "known FILE[::SYMBOL]" in result.stderr - - -def test_file_first_role_instructions_and_bundled_reference(repo_root): - core = repo_root / "skills/shadow-frog" - instructions = (core / "SKILL.md").read_text(encoding="utf-8") - context = (repo_root / "agent-context.md").read_text(encoding="utf-8") - user_skill = (repo_root / "skills/shadow-frog-viewer/SKILL.md").read_text(encoding="utf-8") - assert "**File/symbol navigation is primary.**" in instructions - assert "shadow-read.py" in instructions and "(retrieval.md)" in instructions - assert "Native file reads remain normal and uncounted." in instructions - assert "**Navigate directly**" in context - assert "user-facing" in context and "mandatory" in context - assert "**User-facing inspection and visualization**" in user_skill - assert (core / "retrieval.md").is_file() - - -def test_reader_without_modules_reports_error_but_raw_knowledge_stays_readable(repo_root, tmp_path): - shadow = sample_shadow(tmp_path) - script = tmp_path / "incomplete/shadow-read.py" - script.parent.mkdir() - shutil.copyfile(repo_root / "skills/shadow-frog/shadow-read.py", script) - before = (shadow / "source.py.md").read_bytes() - result = invoke(script, shadow, "source.py") - assert result.returncode != 0 and "reinstall" in result.stderr - assert (shadow / "source.py.md").read_bytes() == before - - -def test_known_file_read_is_bounded_and_pageable(repo_root, tmp_path): - shadow = sample_shadow(tmp_path) - script = repo_root / "skills/shadow-frog/shadow-read.py" - first = invoke(script, shadow, "source.py", "--limit", "1", "--max-chars", "600") - assert first.returncode == 0 and len(first.stdout) <= 600 - continuation = re.search(r"--cursor ([0-9a-f]{32}:\d+)", first.stdout).group(1) - second = invoke( - script, shadow, "source.py", "--limit", "1", "--max-chars", "600", - "--cursor", continuation, - ) - assert second.returncode == 0 and len(second.stdout) <= 600 - assert not set(identities(first.stdout)) & set(identities(second.stdout)) - - -@pytest.mark.parametrize("layout", [".github", ".claude"]) -def test_agent_hook_uses_core_reader_without_user_viewer(repo_root, coupon_demo, tmp_path, layout): - from tests._shell import BASH, HAVE_BASH, shell_path - - if not HAVE_BASH: - pytest.skip("No POSIX shell available for the existing hook") - installed = coupon_demo / layout / "skills/shadow-frog" - shutil.copytree( - repo_root / "skills/shadow-frog", installed, - ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), - ) - hook = coupon_demo / layout / "hooks/scripts/shadow-frog-pre-tool.sh" - hook.parent.mkdir(parents=True) - shutil.copyfile(repo_root / "hook-templates/scripts/shadow-frog-pre-tool.sh", hook) - assert not (installed.parent / "shadow-frog-viewer").exists() - env = os.environ.copy() - env.update( - PATH=shell_path(), HOME=str(coupon_demo), - GIT_CONFIG_GLOBAL=os.devnull, GIT_CONFIG_SYSTEM=os.devnull, - SHADOWFROG_TMP_DIR=str(tmp_path / "hook-state"), - ) - result = subprocess.run( - [BASH, str(hook)], - input=json.dumps({"tool_name": "edit", "tool_input": {"file_path": "cart.py"}}), - cwd=coupon_demo, env=env, capture_output=True, text=True, encoding="utf-8", timeout=10, - ) - assert result.returncode == 0, result.stderr - context = json.loads(result.stdout)["additionalContext"] - assert "Actionable discoveries" in context and "id=d_" in context diff --git a/tests/skills/shadow_frog_dream/test_citation_scores.py b/tests/skills/shadow_frog_dream/test_citation_scores.py new file mode 100644 index 0000000..a875a8f --- /dev/null +++ b/tests/skills/shadow_frog_dream/test_citation_scores.py @@ -0,0 +1,87 @@ +"""Dream artifacts preserve visible citation hints without summing shared ancestry.""" + +import pytest + + +def test_merge_keeps_larger_score_while_upgrading_metadata(dream_reconcile, tmp_path): + path = tmp_path / ".shadow/source.py.md" + path.parent.mkdir() + path.write_text( + "# Shadow: source.py\n\n## `run`\n\n- Claim.\n" + " _(uncertain, source: exploration, citation_score: 7)_\n\n## Cross-References\n", + encoding="utf-8", + ) + discovery = {"text": "Claim.", "status": "verified", "source": "user", "citation_score": 3} + assert dream_reconcile.merge_discovery_into_file( + str(path), "run", discovery, "dream", repo_root=str(tmp_path), + ) + assert "source: user, citation_score: 7" in path.read_text(encoding="utf-8") + discovery["citation_score"] = 9 + assert dream_reconcile.merge_discovery_into_file( + str(path), "run", discovery, "dream", repo_root=str(tmp_path), + ) + assert "citation_score: 9" in path.read_text(encoding="utf-8") + assert not dream_reconcile.merge_discovery_into_file( + str(path), "run", discovery, "dream", repo_root=str(tmp_path), + ) + + +def test_existing_cross_refs_and_scores_merge_independently(dream_reconcile, tmp_path): + path = tmp_path / ".shadow/_cross/contract.md" + path.parent.mkdir(parents=True) + path.write_text( + "# Contract\n\n**Refs**:\n- `a.py::run`\n\n**Discovery**: Shared claim.\n\n" + "_(verified, source: exploration, citation_score: 4)_\n", + encoding="utf-8", + ) + assert dream_reconcile._merge_refs_into_cross_file( + str(path), ["b.py::call"], repo_root=str(tmp_path), citation_score=2, + ) + assert "citation_score: 4" in path.read_text(encoding="utf-8") + assert dream_reconcile._merge_refs_into_cross_file( + str(path), ["a.py::run"], repo_root=str(tmp_path), citation_score=7, + ) + assert "citation_score: 7" in path.read_text(encoding="utf-8") + assert "`b.py::call`" in path.read_text(encoding="utf-8") + + +@pytest.mark.parametrize("score", [-1, 0.5, True, None, "4"]) +@pytest.mark.parametrize("kind", ["discoveries", "cross_cutting"]) +def test_invalid_manifest_score_fails_before_any_publication(dream_reconcile, tmp_path, score, kind): + invalid = {"anchor": "a.py::run", "text": "Claim.", "slug": "contract", "refs": [], "citation_score": score} + manifest = {"discoveries": [{"anchor": "valid.py::run", "text": "Valid claim."}]} + if kind == "discoveries": + manifest["discoveries"].append(invalid) + else: + manifest[kind] = [invalid] + with pytest.raises(dream_reconcile.CitationError, match="nonnegative integer"): + dream_reconcile.merge_discoveries(str(tmp_path), [("dream/p/id", "id", manifest)]) + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.slow +@pytest.mark.parametrize("score,expected", [(0, 0), (7, 0), (-1, 1), (False, 1), (0.5, 1), ("3", 1)]) +@pytest.mark.parametrize("kind", ["discoveries", "cross_cutting"]) +def test_validator_checks_manifest_citation_fields(tmp_git_repo, score, expected, kind): + from tests.skills.shadow_frog_dream.test_dream_validate import ( + _commit_base, _default_manifest, _default_report, _run_validate, _write_dream, + ) + + base = _commit_base(tmp_git_repo) + dream_id = "20260924-010000Z-citations" + entry = { + "anchor": "a.py::run", "slug": "contract", "refs": ["a.py::run"], + "text": "The function leaves its input unchanged.", "citation_score": score, + } + manifest = _default_manifest(dream_id, **{kind: [entry]}) + _write_dream(tmp_git_repo, dream_id, manifest=manifest, report=_default_report(dream_id, base)) + (tmp_git_repo / ".shadow/a.py.md").write_text( + "# Shadow: a.py\n\n## `run`\n\n- Leaves its input unchanged.\n" + " _(verified, source: exploration, citation_score: 0)_\n", + encoding="utf-8", + ) + result = _run_validate(dream_id, tmp_git_repo) + assert result.returncode == expected, result.stdout + result.stderr + if expected: + assert f"{kind}[0].citation_score" in result.stdout + assert "nonnegative integer" in result.stdout diff --git a/tests/skills/shadow_frog_dream/test_dream_reconcile.py b/tests/skills/shadow_frog_dream/test_dream_reconcile.py index 49b15d6..32ec085 100644 --- a/tests/skills/shadow_frog_dream/test_dream_reconcile.py +++ b/tests/skills/shadow_frog_dream/test_dream_reconcile.py @@ -375,7 +375,7 @@ def test_merge_discovery_creates_new_shadow(dream_reconcile, tmp_path): body = shadow.read_text(encoding="utf-8") assert "## `do_thing`" in body assert "- Returns None on empty input." in body - assert "_(verified, source: exploration)_" in body + assert "_(verified, source: exploration, citation_score: 0)_" in body assert "Dream report: `_dreams/20260101-000000Z-x/`" in body assert "## Cross-References" in body @@ -502,7 +502,7 @@ def test_merge_discovery_upgrades_uncertain_to_verified(dream_reconcile, tmp_pat ) assert written is True body = shadow.read_text(encoding="utf-8") - assert "_(verified, source: exploration)_" in body + assert "_(verified, source: exploration, citation_score: 0)_" in body def test_merge_discovery_never_downgrades_verified(dream_reconcile, tmp_path): @@ -1426,7 +1426,7 @@ def test_merge_discoveries_creates_cross_cutting_file_with_back_pointers( assert "**Category**: behavior" in body assert "`src/auth.py::login`" in body assert "`lib/session.py::Session`" in body - assert "_(verified, source: exploration)_" in body + assert "_(verified, source: exploration, citation_score: 0)_" in body # Each referenced per-file shadow must have a back-pointer. auth = (repo / ".shadow" / "src" / "auth.py.md").read_text(encoding="utf-8") diff --git a/tests/skills/shadow_frog_dream/test_dream_tools.py b/tests/skills/shadow_frog_dream/test_dream_tools.py index cd07d2c..8c7e855 100644 --- a/tests/skills/shadow_frog_dream/test_dream_tools.py +++ b/tests/skills/shadow_frog_dream/test_dream_tools.py @@ -49,8 +49,7 @@ def test_pin_creates_complete_external_snapshot(tmp_git_repo, tmp_path): assert Path(packet["manifest"]).is_absolute() assert Path(packet["skill_dir"]) == output / "shadow-frog-dream" assert (output / "shadow-frog/_coherence.py").is_file() - assert (output / "shadow-frog/retrieval.md").is_file() - assert (output / "shadow-frog/shadow-read.py").is_file() + assert (output / "shadow-frog/_citations.py").is_file() assert all(Path(path).is_file() for path in packet["instructions"]) metadata = json.loads(Path(packet["manifest"]).read_text(encoding="utf-8")) assert metadata["mode"] == "coherent" diff --git a/tests/skills/shadow_frog_nap/test_nap.py b/tests/skills/shadow_frog_nap/test_nap.py index e92a9cb..7e4d766 100644 --- a/tests/skills/shadow_frog_nap/test_nap.py +++ b/tests/skills/shadow_frog_nap/test_nap.py @@ -416,12 +416,12 @@ def test_cli_export_is_utf8_and_does_not_overwrite_record(nap, record, tmp_path, assert json.loads((tmp_path / "nap run.json").read_text(encoding="utf-8")) == record -def test_exports_do_not_pollute_discovery_views(record, tmp_path, coupon_demo, shadow_knowledge): - before = shadow_knowledge.get_all_shadow_files(coupon_demo / ".shadow") +def test_exports_do_not_pollute_discovery_views(record, tmp_path, coupon_demo, shadow_viewer): + before = shadow_viewer.get_all_shadow_files(coupon_demo / ".shadow") output = coupon_demo / ".shadow" / "_meta" / "naps" / "tasks.md" result = run_cli(record, tmp_path, coupon_demo, "--export", str(output)) assert result.returncode == 0, result.stderr - assert shadow_knowledge.get_all_shadow_files(coupon_demo / ".shadow") == before + assert shadow_viewer.get_all_shadow_files(coupon_demo / ".shadow") == before result = run_cli( record, tmp_path, coupon_demo, "--export", str(coupon_demo / ".shadow" / "nap-proposals.md"), diff --git a/tests/skills/shadow_frog_viewer/conftest.py b/tests/skills/shadow_frog_viewer/conftest.py deleted file mode 100644 index 71b710b..0000000 --- a/tests/skills/shadow_frog_viewer/conftest.py +++ /dev/null @@ -1,9 +0,0 @@ -"""Keep standalone-viewer citation state inside each test's temporary directory.""" - -import pytest - - -@pytest.fixture(autouse=True) -def local_viewer_state(tmp_path, monkeypatch): - monkeypatch.setenv("XDG_STATE_HOME", str(tmp_path / "viewer-state")) - monkeypatch.setenv("LOCALAPPDATA", str(tmp_path / "viewer-state")) diff --git a/tests/skills/shadow_frog_viewer/test_citation_metadata.py b/tests/skills/shadow_frog_viewer/test_citation_metadata.py new file mode 100644 index 0000000..a332216 --- /dev/null +++ b/tests/skills/shadow_frog_viewer/test_citation_metadata.py @@ -0,0 +1,119 @@ +"""Viewer reads visible scores without recording implicit views or creating state.""" + +import os +import shutil +import subprocess +import sys + +import pytest + + +def test_metadata_score_is_not_discovery_text(shadow_viewer): + discovery = shadow_viewer.parse_discovery( + "- Rejects empty input.", + [" _(verified, source: user, labels: [bug], citation_score: 7)_"], + ) + assert discovery["text"] == "Rejects empty input." + assert discovery["citation_score"] == 7 + assert discovery["status"] == "verified" and discovery["labels"] == ["bug"] + assert shadow_viewer.parse_discovery("- Older claim.", [" _(verified, source: exploration)_"])["citation_score"] == 0 + assert shadow_viewer.parse_discovery("- Preference.", [" _(source: user, citation_score: 3)_"])["citation_score"] == 3 + + +def test_reading_and_ranking_scores_never_rewrites_shadow(shadow_viewer, tmp_path, capsys): + shadow = tmp_path / ".shadow" + shadow.mkdir() + path = shadow / "source.py.md" + path.write_text( + "# Shadow: source.py\n\n## `run`\n\n" + "- Rare finding.\n _(verified, source: exploration, labels: [bug], citation_score: 1)_\n\n" + "- Popular finding.\n _(verified, source: exploration, labels: [bug], citation_score: 9)_\n\n" + "- User constraint.\n _(verified, source: user, labels: [bug], citation_score: 0)_\n\n" + "## Cross-References\n", + encoding="utf-8", + ) + before = path.read_bytes() + shadow_viewer.view_top(shadow, "source.py", "bug", 3, 0) + output = capsys.readouterr().out + assert output.index("User constraint") < output.index("Popular finding") < output.index("Rare finding") + assert "citation_score: 9" in output + shadow_viewer.view_search(shadow, "finding") + output = capsys.readouterr().out + assert output.index("Popular finding") < output.index("Rare finding") + assert path.read_bytes() == before + assert sorted(p.name for p in shadow.iterdir()) == ["source.py.md"] + + +@pytest.mark.parametrize("location", ["source.py.md", "_prefs.md", "_cross/contract.md"]) +@pytest.mark.parametrize("score", ["-1", "1.5", "true"]) +def test_structural_audit_reports_invalid_visible_score(shadow_viewer, tmp_path, capsys, location, score): + shadow = tmp_path / ".shadow" + path = shadow / location + path.parent.mkdir(parents=True) + content = "# Shadow: source.py\n\n## `run`\n\n- Claim.\n" + metadata = f" _(verified, source: exploration, citation_score: {score})_\n" + if location == "_prefs.md": + content = "# Preferences\n\n- Claim.\n" + metadata = f" _(source: user, citation_score: {score})_\n" + elif location.startswith("_cross"): + content = "# Contract\n\n**Category**: contract\n**Refs**:\n- `source.py::run`\n\n**Discovery**: Claim.\n\n" + path.write_text(content + metadata, encoding="utf-8") + assert shadow_viewer.view_check_invariants(shadow) == 1 + assert "citation_score" in capsys.readouterr().out + + +@pytest.mark.slow +def test_direct_read_then_cite_then_user_viewer(repo_root, tmp_path): + shadow = tmp_path / ".shadow" + shadow.mkdir() + path = shadow / "source.py.md" + path.write_text( + "# Shadow: source.py\n\n## `run`\n\n- Rejects empty input.\n" + " _(verified, source: exploration, citation_score: 0)_\n", + encoding="utf-8", + ) + assert "citation_score: 0" in path.read_text(encoding="utf-8") + result = subprocess.run( + [sys.executable, str(repo_root / "skills/shadow-frog/shadow-cite.py"), str(path), + "--symbol", "run", "--text", "Rejects empty input."], + capture_output=True, text=True, encoding="utf-8", check=True, + ) + assert "0 -> 1" in result.stdout + before_view = path.read_bytes() + result = subprocess.run( + [sys.executable, str(repo_root / "skills/shadow-frog-viewer/shadow-viewer.py"), + "--shadow-dir", str(shadow), "--search", "empty"], + capture_output=True, text=True, encoding="utf-8", check=True, + ) + assert "citation_score: 1" in result.stdout and path.read_bytes() == before_view + + +@pytest.mark.parametrize("layout", [".github", ".claude"]) +def test_installed_counter_helper_has_no_hidden_store_or_bytecode(repo_root, tmp_path, layout): + installed = tmp_path / layout / "skills" + for name in ("shadow-frog", "shadow-frog-viewer"): + shutil.copytree( + repo_root / "skills" / name, installed / name, + ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), + ) + shadow = tmp_path / ".shadow" + shadow.mkdir() + target = shadow / "sample.py.md" + target.write_text( + "# Shadow: sample.py\n\n## `run`\n\n- Keep the input unchanged.\n" + " _(verified, source: user, citation_score: 0)_\n", + encoding="utf-8", + ) + env = os.environ.copy() + env.pop("PYTHONDONTWRITEBYTECODE", None) + env.pop("PYTHONPYCACHEPREFIX", None) + result = subprocess.run( + [sys.executable, str(installed / "shadow-frog/shadow-cite.py"), str(target), + "--symbol", "run", "--text", "Keep the input unchanged."], + env=env, capture_output=True, text=True, encoding="utf-8", + ) + assert result.returncode == 0, result.stderr + assert "citation_score: 1" in target.read_text(encoding="utf-8") + assert not list(tmp_path.rglob("*.sqlite3")) + assert not list(installed.rglob("*.pyc")) + assert not list(shadow.rglob("*.citation.lock")) diff --git a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py index 116469a..6e3bdf1 100644 --- a/tests/skills/shadow_frog_viewer/test_shadow_viewer.py +++ b/tests/skills/shadow_frog_viewer/test_shadow_viewer.py @@ -1,47 +1,2185 @@ -"""User-facing Viewer CLI shares knowledge identities with the optional agent helper.""" +r"""Tests for `skills/shadow-frog-viewer/shadow-viewer.py`. +Philosophy: USE REAL FILES (per `minimal-mocking-tests`). The viewer is a +pure-read tool — every test either constructs a small shadow tree on +disk and calls a viewer function, or exercises the CLI via subprocess +against the `coupon_demo` fixture. + +Test categories: + * In-process function tests (no `@pytest.mark.slow`): exercise + parsing helpers directly via the `shadow_viewer` fixture. + * CLI integration tests (`@pytest.mark.slow @pytest.mark.integration`): + invoke shadow-viewer.py as a subprocess against `coupon_demo`. + +B3 regression: a discovery whose continuation lines include +``Dream report: `_dreams//` `` must extract the slug path into +`meta["dream_report"]` and must NOT include "Dream report" or the slug +in the discovery body text. +""" +import json import os import re -import shutil import subprocess import sys +import textwrap +from datetime import datetime import pytest -@pytest.mark.parametrize("layout", [".github", ".claude"]) -def test_user_and_agent_interfaces_share_citation_identity(repo_root, tmp_path, layout): - shadow = tmp_path / ".shadow" - shadow.mkdir() - content = "# Shadow: source.py\n\n## `run`\n\n- Shared knowledge.\n _(verified, source: exploration)_\n" - (shadow / "source.py.md").write_text(content, encoding="utf-8") - for skill in ("shadow-frog", "shadow-frog-viewer"): - shutil.copytree( - repo_root / "skills" / skill, tmp_path / layout / "skills" / skill, - ignore=shutil.ignore_patterns("__pycache__", "*.pyc"), - ) - viewer = tmp_path / layout / "skills/shadow-frog-viewer/shadow-viewer.py" - reader = tmp_path / layout / "skills/shadow-frog/shadow-read.py" - env = os.environ.copy() - env.pop("PYTHONDONTWRITEBYTECODE", None) - env.pop("PYTHONPYCACHEPREFIX", None) - - def call(script, *args): - return subprocess.run( - [sys.executable, str(script), "--shadow-dir", str(shadow), *args], - cwd=tmp_path, env=env, capture_output=True, text=True, encoding="utf-8", check=True, - ).stdout - - overview = call(viewer) - assert "Shadow Knowledge Base Summary" in overview - user_result = call(viewer, "--search", "Shared") - assert "citation_score=0" in user_result - agent_result = call(reader, "source.py::run") - assert "citation_score=1" in agent_result - assert re.findall(r"id=(d_[0-9a-f]{32})", user_result) == re.findall( - r"id=(d_[0-9a-f]{32})", agent_result, - ) - assert (shadow / "source.py.md").read_text(encoding="utf-8") == content - assert not list((tmp_path / layout / "skills").rglob("*.pyc")) - assert "for users" in call(viewer, "--help") - assert "Optional bounded agent retrieval" in call(reader, "--help") +# --- Helpers --------------------------------------------------------------- + + +def _write_shadow(shadow_dir, rel_path, content): + """Write `content` to /, creating parents.""" + p = shadow_dir / rel_path + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(textwrap.dedent(content), encoding="utf-8") + return p + + +def _make_shadow_root(tmp_path): + """Create an empty .shadow/ dir under tmp_path and return it.""" + sd = tmp_path / ".shadow" + sd.mkdir() + return sd + + +def _run_viewer(repo_root, cwd, *args): + """Run shadow-viewer.py as a subprocess from `cwd`.""" + script = repo_root / "skills/shadow-frog-viewer/shadow-viewer.py" + return subprocess.run( + [sys.executable, str(script), *args], + cwd=str(cwd), + capture_output=True, + text=True, + encoding="utf-8", + check=False, + ) + + +# --- parse_discovery ------------------------------------------------------- + + +def test_parse_discovery_standard(shadow_viewer): + """Basic verified/exploration discovery, no labels.""" + line = "- Caches None for invalid codes" + cont = [" _(verified, source: exploration)_"] + d = shadow_viewer.parse_discovery(line, cont) + assert d["text"] == "Caches None for invalid codes" + assert d["status"] == "verified" + assert d["source"] == "exploration" + assert "labels" not in d + assert "dream_report" not in d + + +def test_parse_discovery_with_labels(shadow_viewer): + """Labels are parsed into a list, trimmed, lowercase comma-split.""" + line = "- Foo" + cont = [" _(verified, source: user, labels: [bug, security])_"] + d = shadow_viewer.parse_discovery(line, cont) + assert d["text"] == "Foo" + assert d["status"] == "verified" + assert d["source"] == "user" + assert d["labels"] == ["bug", "security"] + + +@pytest.mark.parametrize("status", ["verified", "uncertain", "refuted"]) +def test_parse_discovery_status_variants(shadow_viewer, status): + line = "- Some discovery" + cont = [f" _({status}, source: exploration)_"] + d = shadow_viewer.parse_discovery(line, cont) + assert d["status"] == status + assert d["source"] == "exploration" + + +@pytest.mark.parametrize("source", ["exploration", "user", "interaction"]) +def test_parse_discovery_source_variants(shadow_viewer, source): + line = "- Some discovery" + cont = [f" _(verified, source: {source})_"] + d = shadow_viewer.parse_discovery(line, cont) + assert d["source"] == source + + +def test_parse_discovery_also_involves(shadow_viewer): + """`Also involves:` populates a list of file::symbol anchors.""" + line = "- A multi-symbol discovery" + cont = [ + " _(verified, source: exploration)_", + " Also involves: `inventory.py::validate_coupon`, `cart.py::COUPON_CACHE`", + ] + d = shadow_viewer.parse_discovery(line, cont) + assert d["text"] == "A multi-symbol discovery" + assert d["also_involves"] == [ + "inventory.py::validate_coupon", + "cart.py::COUPON_CACHE", + ] + # also_involves line must not leak into the body text + assert "Also involves" not in d["text"] + + +def test_parse_discovery_b3_dream_report_regression(shadow_viewer): + """B3 regression: Dream report goes into meta['dream_report'] and is + excluded from the body text.""" + line = "- Case-variant lookups create duplicate cache entries" + cont = [ + " _(verified, source: exploration, labels: [bug, performance])_", + " Dream report: `_dreams/20260420-140000Z-cache-poison-sequence/`", + ] + d = shadow_viewer.parse_discovery(line, cont) + # Body text is preserved, with no Dream report leakage + assert d["text"] == "Case-variant lookups create duplicate cache entries" + assert "Dream report" not in d["text"] + assert "_dreams/" not in d["text"] + # meta["dream_report"] captures the backtick payload (slug folder path) + assert d["dream_report"] == ( + "_dreams/20260420-140000Z-cache-poison-sequence/" + ) + + +def test_parse_discovery_b3_dream_report_with_also_involves(shadow_viewer): + """Dream report + Also involves on the same discovery — both extracted, + neither leaks into body text.""" + line = "- Discovery with both extras" + cont = [ + " _(verified, source: exploration, labels: [security])_", + " Dream report: `_dreams/20260420-142000Z-adversarial-inputs/`", + " Also involves: `cart.py::get_coupon`, `cart.py::COUPON_CACHE`", + ] + d = shadow_viewer.parse_discovery(line, cont) + assert d["text"] == "Discovery with both extras" + assert "Dream report" not in d["text"] + assert "Also involves" not in d["text"] + assert d["dream_report"] == ( + "_dreams/20260420-142000Z-adversarial-inputs/" + ) + assert d["also_involves"] == [ + "cart.py::get_coupon", + "cart.py::COUPON_CACHE", + ] + + +def test_parse_discovery_multiline_body(shadow_viewer): + """Lines that are neither metadata nor structured extras are appended to + the body text.""" + line = "- Lead sentence." + cont = [ + " continuation prose", + " _(verified, source: exploration)_", + ] + d = shadow_viewer.parse_discovery(line, cont) + assert "Lead sentence." in d["text"] + assert "continuation prose" in d["text"] + assert d["status"] == "verified" + + +def test_parse_discovery_no_metadata(shadow_viewer): + """Bullet with no metadata blob still returns a dict with text but no + status/source keys.""" + d = shadow_viewer.parse_discovery("- bare bullet", []) + assert d["text"] == "bare bullet" + assert "status" not in d + assert "source" not in d + + +def test_parse_discovery_preferences_source_only(shadow_viewer): + """Preferences use `_(source: user)_` (no status). Extracts source.""" + d = shadow_viewer.parse_discovery( + "- Prefer X over Y", [" _(source: user)_"] + ) + assert d["text"] == "Prefer X over Y" + assert d["source"] == "user" + assert "status" not in d + + +def test_parse_discovery_none_input_does_not_crash(shadow_viewer): + """Passing a non-string line shouldn't raise — should return a dict.""" + d = shadow_viewer.parse_discovery(None, None) + assert isinstance(d, dict) + assert "text" in d + + +# --- parse_shadow_file ----------------------------------------------------- + + +def test_parse_shadow_file_placeholder(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "foo.py.md", """\ + # Shadow: foo.py + + **Language**: Python | **Lines**: 10 + + _No discoveries yet._ + """) + res = shadow_viewer.parse_shadow_file(f) + assert res["source_file"] == "foo.py" + assert res["language"] == "Python" + assert res["lines"] == 10 + assert res["symbols"] == [] + assert res["discoveries"] == [] + assert res["cross_references"] == [] + assert res["parse_errors"] == [] + + +def test_parse_shadow_file_one_symbol_one_discovery(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "auth.py.md", """\ + # Shadow: auth.py + + **Language**: Python | **Lines**: 42 + + ## `authenticate` + + - Returns None on expired tokens, silently. + _(verified, source: exploration, labels: [security])_ + """) + res = shadow_viewer.parse_shadow_file(f) + assert res["source_file"] == "auth.py" + assert res["symbols"] == ["authenticate"] + assert len(res["discoveries"]) == 1 + d = res["discoveries"][0] + assert d["symbol"] == "authenticate" + assert d["file"] == "auth.py" + assert d["status"] == "verified" + assert d["source"] == "exploration" + assert d["labels"] == ["security"] + assert "Returns None" in d["text"] + + +def test_parse_shadow_file_cross_references_backpointers( + shadow_viewer, tmp_path +): + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "bar.py.md", """\ + # Shadow: bar.py + + ## `func` + + - A discovery. + _(verified, source: exploration)_ + + ## Cross-References + + - [my-cross-cutting](_cross/my-cross-cutting.md) + (involves `bar.py::func`) + - [another-one](_cross/another-one.md) + """) + res = shadow_viewer.parse_shadow_file(f) + assert res["symbols"] == ["func"] + # Cross-references back-pointer link labels are collected + assert "my-cross-cutting" in res["cross_references"] + assert "another-one" in res["cross_references"] + # Cross-ref bullets are NOT mistaken for discoveries + assert len(res["discoveries"]) == 1 + + +def test_parse_shadow_file_file_level_and_cross_refs(shadow_viewer, tmp_path): + """`## File-Level` discoveries are tagged with symbol='file-level' and + `## Cross-References` bullets are not treated as discoveries.""" + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "mix.py.md", """\ + # Shadow: mix.py + + ## File-Level + + - A module-wide observation. + _(verified, source: exploration)_ + + ## `helper` + + - A symbol discovery. + _(verified, source: user)_ + + ## Cross-References + + - [shared](_cross/shared.md) + """) + res = shadow_viewer.parse_shadow_file(f) + discs = res["discoveries"] + assert len(discs) == 2 + by_sym = {d["symbol"]: d for d in discs} + assert "file-level" in by_sym + assert "helper" in by_sym + assert by_sym["file-level"]["source"] == "exploration" + assert by_sym["helper"]["source"] == "user" + assert res["cross_references"] == ["shared"] + + +def test_parse_shadow_file_malformed_does_not_crash(shadow_viewer, tmp_path): + """Bizarre / structurally broken content shouldn't raise.""" + sd = _make_shadow_root(tmp_path) + f = _write_shadow(sd, "junk.py.md", """\ + # Shadow: junk.py + **Language**: notnumeric | **Lines**: notanumber + + ## not a backtick heading + - orphan bullet with no metadata + ## `realsym` + - real disc + _(verified, source: exploration)_ + """) + res = shadow_viewer.parse_shadow_file(f) + # Parse succeeds despite weirdness + assert res["source_file"] == "junk.py" + # The bad "Lines" cell stays None (int parse skipped) + assert res["lines"] is None + # `realsym` is captured; the non-backtick heading is not a symbol + assert "realsym" in res["symbols"] + assert "not a backtick heading" not in res["symbols"] + + +def test_parse_shadow_file_missing_file_returns_error( + shadow_viewer, tmp_path +): + """Reading a non-existent path records a parse_error, doesn't raise.""" + res = shadow_viewer.parse_shadow_file(tmp_path / "ghost.md") + assert res["parse_errors"] + assert res["symbols"] == [] + assert res["discoveries"] == [] + + +# --- parse_cross_cutting --------------------------------------------------- + + +def test_parse_cross_cutting_empty_dir(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_cross").mkdir() + assert shadow_viewer.parse_cross_cutting(sd) == [] + + +def test_parse_cross_cutting_no_dir(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + # _cross/ not created + assert shadow_viewer.parse_cross_cutting(sd) == [] + + +def test_parse_cross_cutting_one_entry(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/example-pattern.md", """\ + # Example pattern + + **Category**: pattern + **Refs**: + - `cart.py::calculate_total` + - `inventory.py::validate_coupon` + + **Discovery**: A multi-file pattern observed across the codebase. + + _(verified, source: exploration, labels: [bug])_ + """) + entries = shadow_viewer.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["slug"] == "example-pattern" + assert e["title"] == "Example pattern" + assert e["category"] == "pattern" + assert "cart.py::calculate_total" in e["refs"] + assert "inventory.py::validate_coupon" in e["refs"] + assert "multi-file pattern" in e["discovery"] + assert e["status"] == "verified" + assert e["source"] == "exploration" + assert e["labels"] == ["bug"] + + +def test_parse_cross_cutting_no_labels(shadow_viewer, tmp_path): + """A cross-cutting entry without labels is still parsed, with no + `labels` key in the entry.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-label.md", """\ + # Plain entry + + **Category**: behavior + **Refs**: + - `foo.py::bar` + + **Discovery**: Something happens. + + _(uncertain, source: exploration)_ + """) + entries = shadow_viewer.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["status"] == "uncertain" + assert e["source"] == "exploration" + assert "labels" not in e + + +def test_parse_cross_cutting_minor_format_variation(shadow_viewer, tmp_path): + """File missing a `**Category**:` field doesn't crash; entry is still + emitted with whatever fields could be parsed.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/sparse.md", """\ + # Sparse entry + + Some prose with no structured fields. + + _(refuted, source: user)_ + """) + entries = shadow_viewer.parse_cross_cutting(sd) + assert len(entries) == 1 + e = entries[0] + assert e["slug"] == "sparse" + assert e["title"] == "Sparse entry" + assert e["status"] == "refuted" + assert e["source"] == "user" + + +# --- parse_prefs ----------------------------------------------------------- + + +def test_parse_prefs_missing_file(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + assert shadow_viewer.parse_prefs(sd) == [] + + +def test_parse_prefs_zero(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + _No preferences recorded yet._ + """) + assert shadow_viewer.parse_prefs(sd) == [] + + +def test_parse_prefs_one(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Always use type hints on public APIs. + _(source: user)_ + """) + prefs = shadow_viewer.parse_prefs(sd) + assert len(prefs) == 1 + assert prefs[0]["text"] == "Always use type hints on public APIs." + assert prefs[0]["source"] == "user" + assert prefs[0]["type"] == "preference" + + +def test_parse_prefs_three(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Use kebab-case for slugs. + _(source: user)_ + - Never commit secrets to source control. + _(source: interaction)_ + - Prefer fail-fast for required dependencies. + _(source: user)_ + """) + prefs = shadow_viewer.parse_prefs(sd) + assert len(prefs) == 3 + texts = [p["text"] for p in prefs] + assert any("kebab-case" in t for t in texts) + assert any("secrets" in t for t in texts) + assert any("fail-fast" in t for t in texts) + sources = [p["source"] for p in prefs] + assert "user" in sources + assert "interaction" in sources + + +# --- load_state ------------------------------------------------------------ + + +def test_load_state_missing(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + assert shadow_viewer.load_state(sd) == {} + + +def test_load_state_valid_json(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + payload = { + "version": 1, + "total_files": 5, + "total_discoveries": 42, + "last_update_type": "auto", + } + (sd / "_meta" / "state.json").write_text( + json.dumps(payload), encoding="utf-8" + ) + state = shadow_viewer.load_state(sd) + assert state == payload + + +def test_load_state_malformed_json(shadow_viewer, tmp_path): + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text( + "{not valid json", encoding="utf-8" + ) + state = shadow_viewer.load_state(sd) + assert state == {} + + +def test_load_state_non_object_json(shadow_viewer, tmp_path): + """JSON parses but is not a dict — returns empty dict sentinel.""" + sd = _make_shadow_root(tmp_path) + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text("[1, 2, 3]", encoding="utf-8") + assert shadow_viewer.load_state(sd) == {} + + +# --- get_all_shadow_files -------------------------------------------------- + + +def test_get_all_shadow_files_excludes_special(shadow_viewer, tmp_path): + """Per-file shadows are returned; _cross/, _dreams/, _meta/, _index.md, + _prefs.md, state.json are all excluded.""" + sd = _make_shadow_root(tmp_path) + # Files that SHOULD be returned + _write_shadow(sd, "a.py.md", "# Shadow: a.py\n") + _write_shadow(sd, "src/b.py.md", "# Shadow: src/b.py\n") + _write_shadow(sd, "deep/nested/c.py.md", "# Shadow: deep/nested/c.py\n") + # Files that should be EXCLUDED + _write_shadow(sd, "_index.md", "# Shadow Index\n") + _write_shadow(sd, "_prefs.md", "# Preferences\n") + _write_shadow(sd, "_cross/some.md", "# Some cross\n") + _write_shadow(sd, "_dreams/dream-1/report.md", "# A dream\n") + _write_shadow(sd, "_meta/state.json", "{}") + # Junk files that are not .md and shouldn't show up anyway + (sd / "notes.json").write_text("{}", encoding="utf-8") + + files = shadow_viewer.get_all_shadow_files(sd) + rels = sorted(f.relative_to(sd).as_posix() for f in files) + assert rels == ["a.py.md", "deep/nested/c.py.md", "src/b.py.md"] + + +def test_get_all_shadow_files_against_coupon_demo(shadow_viewer, coupon_demo): + sd = coupon_demo / ".shadow" + files = shadow_viewer.get_all_shadow_files(sd) + rels = sorted(f.relative_to(sd).as_posix() for f in files) + assert rels == ["cart.py.md", "inventory.py.md", "test_cart.py.md"] + # Make sure none of the special files leaked through + for r in rels: + assert not r.startswith(("_cross/", "_dreams/", "_meta/")) + assert r not in ("_index.md", "_prefs.md") + + +# --- collect_all_discoveries (against fixture) ----------------------------- + + +def test_collect_all_discoveries_counts(shadow_viewer, coupon_demo): + """Coupon demo has 33 per-file discoveries (matches state.json).""" + sd = coupon_demo / ".shadow" + all_disc = shadow_viewer.collect_all_discoveries(sd) + # state.json claims 33 total discoveries + assert len(all_disc) == 33 + + # File breakdown: cart=14, inventory=10, test_cart=9 + by_file = {} + for d in all_disc: + by_file.setdefault(d.get("file"), 0) + by_file[d.get("file")] += 1 + assert by_file == {"cart.py": 14, "inventory.py": 10, "test_cart.py": 9} + + +def test_collect_all_discoveries_shadow_path_and_mtime( + shadow_viewer, coupon_demo +): + sd = coupon_demo / ".shadow" + all_disc = shadow_viewer.collect_all_discoveries(sd) + for d in all_disc: + assert "shadow_path" in d + assert d["shadow_path"].endswith(".md") + assert "shadow_mtime" in d + assert isinstance(d["shadow_mtime"], float) + + +def test_collect_all_discoveries_b3_no_dream_report_in_text( + shadow_viewer, coupon_demo +): + """B3 regression on real fixture: no discovery body should contain + 'Dream report' or the literal `_dreams/` slug path.""" + sd = coupon_demo / ".shadow" + all_disc = shadow_viewer.collect_all_discoveries(sd) + leaks = [ + d for d in all_disc + if "Dream report" in d.get("text", "") + or "_dreams/" in d.get("text", "") + ] + assert leaks == [], ( + f"Dream report leaked into {len(leaks)} discovery body/bodies: " + f"{[d['text'][:80] for d in leaks]}" + ) + + # And at least some discoveries actually have dream_report metadata + # (the fixture has several Dream report continuation lines) + with_dream = [d for d in all_disc if d.get("dream_report")] + assert len(with_dream) >= 3, ( + "Expected coupon-demo fixture to have multiple Dream-report-tagged " + f"discoveries; found {len(with_dream)}" + ) + for d in with_dream: + assert d["dream_report"].startswith("_dreams/") + + +# --- CLI: --summary -------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_summary(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--summary") + assert r.returncode == 0, r.stderr + out = r.stdout + # Header is "Files shadowed:", "Symbols tracked:", "Discoveries:" per + # current --summary output. Task wording used "Total files:"/"Symbols:" + # /"Discoveries:" loosely — match the actual labels. + assert "Files shadowed:" in out + assert "Symbols tracked:" in out + assert "Discoveries:" in out + # Cross-cutting block exists + assert "Cross-cutting" in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_default_view_is_summary(repo_root, coupon_demo): + """Running with no flags should produce the summary view.""" + r = _run_viewer(repo_root, coupon_demo) + assert r.returncode == 0, r.stderr + assert "Shadow Knowledge Base Summary" in r.stdout + + +# --- CLI: --search --------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_search_matches_text(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--search", "coupon") + assert r.returncode == 0, r.stderr + # Header echoes the query and results were found + assert "'coupon'" in r.stdout + # Some matched line should contain the query (case-insensitive) + assert "coupon" in r.stdout.lower() + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_search_no_results(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--search", "zzznotpresentzzz") + assert r.returncode == 0, r.stderr + assert "No results for 'zzznotpresentzzz'" in r.stdout + + +# --- CLI: --top ------------------------------------------------------------ + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_cart_py_header_and_no_dream_report_leak( + repo_root, coupon_demo +): + """B3 regression at the CLI layer: --top output for a file whose shadow + contains `Dream report:` continuation lines must NOT inline that text + in any discovery body.""" + r = _run_viewer(repo_root, coupon_demo, "--top", "cart.py") + assert r.returncode == 0, r.stderr + out = r.stdout + # Header form: "Top N of M actionable discoveries for cart.py:" + assert "actionable discoveries for cart.py:" in out + # Match the documented prefix exactly + assert out.splitlines()[0].startswith("Top ") + assert "for cart.py:" in out.splitlines()[0] + + # B3: no literal "Dream report:" or raw `_dreams/...` slug paths in any + # of the discovery bullets + assert "Dream report:" not in out + assert "_dreams/" not in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_label_filter_bug(repo_root, coupon_demo): + r = _run_viewer( + repo_root, coupon_demo, "--top", "cart.py", "--top-labels", "bug" + ) + assert r.returncode == 0, r.stderr + out = r.stdout + assert "for cart.py:" in out + # Every bulleted result line should mention 'bug' in its label bracket + bullet_lines = [ + l for l in out.splitlines() if l.startswith("- [") + ] + assert bullet_lines, f"No bullets in --top output:\n{out}" + for line in bullet_lines: + bracket = line.split("]", 1)[0] + assert "bug" in bracket, ( + f"Expected 'bug' label in bracket of: {line}" + ) + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_top_unknown_file(repo_root, coupon_demo): + """A file with no shadow + no cross refs reports nothing actionable.""" + r = _run_viewer(repo_root, coupon_demo, "--top", "does/not/exist.py") + assert r.returncode == 0, r.stderr + assert "No actionable discoveries" in r.stdout + + +# --- CLI: --labels (repo-wide) --------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_labels_bug(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--labels", "bug") + assert r.returncode == 0, r.stderr + out = r.stdout + assert "label(s): bug" in out + # There are multiple bug-labeled discoveries in the fixture + assert "[bug]" in out + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_labels_unknown(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--labels", "nonexistent") + assert r.returncode == 0, r.stderr + assert "No discoveries with label(s): nonexistent" in r.stdout + + +# --- CLI: --recent --------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_recent_caps_at_n(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--recent", "5") + assert r.returncode == 0, r.stderr + out = r.stdout + assert "Most Recent Discoveries (top 5)" in out + # Each recent entry has a timestamp prefix " [YYYY-MM-DD HH:MM]" + entries = [l for l in out.splitlines() if l.strip().startswith("[20")] + assert len(entries) <= 5 + # Coupon-demo has > 5 total items so we expect exactly 5 + assert len(entries) == 5 + + +# --- CLI: --prefs ---------------------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_prefs_empty(repo_root, coupon_demo): + """Coupon-demo ships with no preferences recorded.""" + r = _run_viewer(repo_root, coupon_demo, "--prefs") + assert r.returncode == 0, r.stderr + assert "No preferences recorded yet." in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_prefs_populated(repo_root, coupon_demo): + """After populating _prefs.md, --prefs lists them.""" + prefs_path = coupon_demo / ".shadow" / "_prefs.md" + prefs_path.write_text( + textwrap.dedent("""\ + # Preferences + + - Use snake_case for Python identifiers. + _(source: user)_ + - Avoid mutable default arguments. + _(source: interaction)_ + """), + encoding="utf-8", + ) + r = _run_viewer(repo_root, coupon_demo, "--prefs") + assert r.returncode == 0, r.stderr + assert "Project Preferences (2 total)" in r.stdout + assert "snake_case" in r.stdout + assert "mutable default" in r.stdout + + +# --- CLI: --check-invariants ----------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_clean(repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode == 0, ( + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + assert "✓ Invariants OK" in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_detects_missing_cross_file( + repo_root, coupon_demo +): + """Deleting a _cross/ file leaves dangling back-pointers in per-file + shadows. Invariant #5 must flag this as a violation.""" + cross_path = ( + coupon_demo / ".shadow" / "_cross" + / "coupon-case-normalization-mismatch.md" + ) + assert cross_path.is_file() + cross_path.unlink() + + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode != 0, ( + f"Expected nonzero exit when _cross/ file missing; got 0.\n" + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + # Violations report the dangling slug + assert "coupon-case-normalization-mismatch" in r.stdout + assert "cross-ref" in r.stdout + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_check_invariants_detects_bad_heading(repo_root, coupon_demo): + """Renaming a symbol heading to a non-backtick form is a heading-format + violation.""" + cart_path = coupon_demo / ".shadow" / "cart.py.md" + text = cart_path.read_text(encoding="utf-8") + # `## `COUPON_CACHE`` -> `## COUPON_CACHE` (drop backticks) + mutated = text.replace("## `COUPON_CACHE`", "## COUPON_CACHE", 1) + assert mutated != text, "Substitution did not match" + cart_path.write_text(mutated, encoding="utf-8") + + r = _run_viewer(repo_root, coupon_demo, "--check-invariants") + assert r.returncode != 0, ( + f"Expected nonzero exit for bad heading; got 0.\n" + f"stdout:\n{r.stdout}\nstderr:\n{r.stderr}" + ) + assert "heading" in r.stdout + + +# --- CLI: missing shadow dir ---------------------------------------------- + + +@pytest.mark.slow +@pytest.mark.integration +def test_cli_missing_shadow_dir_fails(repo_root, tmp_path): + """Running with no shadow and a bogus --shadow-dir exits 1.""" + r = _run_viewer( + repo_root, tmp_path, + "--shadow-dir", str(tmp_path / "nope"), "--summary", + ) + assert r.returncode == 1 + assert "No .shadow/ directory found" in r.stderr + + +# =========================================================================== +# RENDER FUNCTIONS: in-process tests +# +# The CLI integration tests above invoke shadow-viewer.py as a subprocess — +# that exercises the dispatcher but doesn't contribute to coverage of the +# loaded module. The tests below call view_* and main() directly via the +# `shadow_viewer` fixture so coverage actually accumulates. +# =========================================================================== + + +# --- shared helpers -------------------------------------------------------- + + +def _call_main(shadow_viewer, argv): + """Invoke `shadow_viewer.main()` in-process with the given argv. + + Returns the integer exit code. We mutate `sys.argv` directly (no + monkeypatch / no mocking) and always restore it in a finally. + """ + saved_argv = sys.argv + sys.argv = ["shadow-viewer.py", *argv] + try: + shadow_viewer.main() + return 0 + except SystemExit as e: + code = e.code + if code is None: + return 0 + if isinstance(code, int): + return code + return 1 + finally: + sys.argv = saved_argv + + +def _make_minimal_shadow(tmp_path, extras=None): + """Build a minimal but valid .shadow/ tree in tmp_path. + + Returns the .shadow/ Path. `extras` is a dict of {relpath: content} + appended on top of the baseline. + """ + sd = tmp_path / ".shadow" + sd.mkdir() + (sd / "_meta").mkdir() + (sd / "_meta" / "state.json").write_text( + json.dumps({ + "version": 1, + "last_update_at": "2026-04-20T16:30:00Z", + "last_update_type": "manual", + "last_commit": "deadbeef" * 5, + }), + encoding="utf-8", + ) + _write_shadow(sd, "foo.py.md", """\ + # Shadow: foo.py + + **Language**: Python | **Lines**: 10 + + ## `bar` + + - A neat bug. + _(verified, source: exploration, labels: [bug])_ + """) + for rel, content in (extras or {}).items(): + _write_shadow(sd, rel, content) + return sd + + +# =========================================================================== +# view_summary +# =========================================================================== + + +class TestViewSummary: + """In-process tests for `view_summary(shadow_dir)`.""" + + def test_basic_header_on_coupon_demo( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Shadow Knowledge Base Summary" in out + assert "=" * 50 in out + assert "Files shadowed:" in out + assert "Symbols tracked:" in out + assert "Discoveries:" in out + assert "Preferences:" in out + assert "Cross-cutting:" in out + + def test_counts_reflect_fixture( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + # 3 source files, 33 discoveries, 3 cross-cutting (per fixture) + assert "Files shadowed: 3" in out + assert "Discoveries: 33" in out + assert "Cross-cutting: 3" in out + + def test_by_source_section_renders( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "By source:" in out + # All discoveries in the fixture are source: exploration + assert "exploration" in out + + def test_by_status_section_renders( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "By status:" in out + assert "verified" in out + + def test_by_label_section_renders( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "By label:" in out + assert "bug" in out + assert "security" in out + + def test_per_file_table_lists_files( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + # Per-file table header + assert "File" in out + assert "Symbols" in out + assert "Disc." in out + # All three fixture files appear in the table + assert "cart.py" in out + assert "inventory.py" in out + assert "test_cart.py" in out + + def test_cross_cutting_titles_section( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Cross-cutting discoveries:" in out + assert "Coupon case normalization mismatch" in out + assert "[edge-case]" in out + + def test_state_info_section( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Last update:" in out + assert "Last commit:" in out + # The fixture state.json says last_update_type: dream + assert "(dream)" in out + + def test_empty_shadow_dir_renders_zeros( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Files shadowed: 0" in out + assert "Discoveries: 0" in out + # With zero discoveries there's no source/status breakdown + assert "By source:" not in out + assert "By status:" not in out + assert "By label:" not in out + # And no state info (no state.json) + assert "Last update:" not in out + + def test_corrupted_state_json_does_not_break_summary( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # Clobber state.json with junk + (sd / "_meta" / "state.json").write_text( + "{ not json", encoding="utf-8" + ) + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + # Header and counts still rendered + assert "Shadow Knowledge Base Summary" in out + assert "Files shadowed: 1" in out + # State section silently dropped + assert "Last update:" not in out + + def test_unreadable_shadow_file_does_not_break_summary( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # Create a file with invalid UTF-8 — parse_shadow_file records a + # parse_error but returns a result. view_summary should still + # render the rest of the report. + bad = sd / "bad.py.md" + bad.write_bytes(b"\xff\xfe\x00garbage\x00") + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Shadow Knowledge Base Summary" in out + assert "Files shadowed: 2" in out + + def test_more_than_20_files_truncated( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + for i in range(25): + _write_shadow( + sd, f"file{i:02d}.py.md", + f"# Shadow: file{i:02d}.py\n\n## `sym{i}`\n\n" + f"- D\n _(verified, source: exploration)_\n", + ) + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Files shadowed: 25" in out + assert "... and 5 more files" in out + + def test_no_cross_cutting_dir_omits_section( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # no _cross/ dir + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Cross-cutting: 0" in out + # The titled list is suppressed when empty + assert "Cross-cutting discoveries:" not in out + + def test_summary_no_prefs(self, shadow_viewer, tmp_path, capsys): + sd = _make_minimal_shadow(tmp_path) + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Preferences: 0" in out + + def test_summary_with_prefs(self, shadow_viewer, tmp_path, capsys): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Pref one. + _(source: user)_ + - Pref two. + _(source: interaction)_ + """) + shadow_viewer.view_summary(sd) + out = capsys.readouterr().out + assert "Preferences: 2" in out + + +# =========================================================================== +# view_search +# =========================================================================== + + +class TestViewSearch: + """In-process tests for `view_search(shadow_dir, query)`.""" + + def test_finds_text_match(self, shadow_viewer, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "tax") + out = capsys.readouterr().out + assert "Search: 'tax'" in out + assert "results" in out + # The "8% tax" / "Tax rate" discoveries on cart.py match + assert "cart.py" in out + + def test_finds_symbol_match(self, shadow_viewer, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "COUPON_CACHE") + out = capsys.readouterr().out + assert "COUPON_CACHE" in out + assert "cart.py" in out + + def test_finds_file_name_match( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Searching for the literal file name pulls every discovery in + # that file (file_name_hit branch). + shadow_viewer.view_search(sd, "inventory.py") + out = capsys.readouterr().out + assert "inventory.py" in out + assert "matches" in out + + def test_case_insensitive(self, shadow_viewer, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "COUPON") + upper = capsys.readouterr().out + shadow_viewer.view_search(sd, "coupon") + lower = capsys.readouterr().out + # Same number of result lines either way + assert ("results" in upper) and ("results" in lower) + # And both contain at least one match + assert "::" in upper + assert "::" in lower + + def test_no_matches_prints_no_results( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "zzzdoesnotexistzzz") + out = capsys.readouterr().out + assert "No results for 'zzzdoesnotexistzzz'." in out + + def test_finds_cross_cutting_title( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Title of _cross/coupon-case-normalization-mismatch.md is + # "Coupon case normalization mismatch" + shadow_viewer.view_search(sd, "normalization mismatch") + out = capsys.readouterr().out + assert "Cross-cutting" in out + assert "Coupon case normalization mismatch" in out + assert "Category:" in out + assert "edge-case" in out + + def test_finds_cross_cutting_by_ref( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Search for a ref that appears in a _cross file + shadow_viewer.view_search(sd, "apply_bulk_discount") + out = capsys.readouterr().out + assert "Cross-cutting" in out + assert "Mutation through discount pipeline" in out + + def test_finds_also_involves_match( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + Also involves: `b.py::weird_symbol_zzz` + """) + shadow_viewer.view_search(sd, "weird_symbol_zzz") + out = capsys.readouterr().out + # The match flag is "also_involves" and a line shows that ref + assert "weird_symbol_zzz" in out + assert "Also involves:" in out + + def test_finds_preference_match( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Prefer rusty pelicans for everything. + _(source: user)_ + """) + shadow_viewer.view_search(sd, "pelican") + out = capsys.readouterr().out + assert "Preferences" in out + assert "pelican" in out.lower() + assert "[user]" in out + + def test_groups_per_file_results_by_file( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "coupon") + out = capsys.readouterr().out + # Per-file groups present a header like "cart.py (N matches)" + assert re.search(r"cart\.py \(\d+ matches\)", out) + + def test_results_show_status_and_source( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "tax") + out = capsys.readouterr().out + assert "(verified, source: exploration)" in out + + def test_total_count_in_header( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_search(sd, "coupon") + out = capsys.readouterr().out + m = re.search(r"\((\d+) results\)", out) + assert m is not None + assert int(m.group(1)) > 0 + + def test_empty_shadow_returns_no_results( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_viewer.view_search(sd, "anything") + out = capsys.readouterr().out + assert "No results for 'anything'." in out + + +# =========================================================================== +# view_prefs +# =========================================================================== + + +class TestViewPrefs: + """In-process tests for `view_prefs(shadow_dir)`.""" + + def test_missing_prefs_file(self, shadow_viewer, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + shadow_viewer.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + def test_empty_prefs_placeholder( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + _No preferences recorded yet._ + """) + shadow_viewer.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + def test_populated_prefs_lists_count_and_sources( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - Use snake_case. + _(source: user)_ + - Avoid mutable default args. + _(source: interaction)_ + - Prefer fail-fast for required deps. + _(source: user)_ + """) + shadow_viewer.view_prefs(sd) + out = capsys.readouterr().out + assert "Project Preferences (3 total)" in out + assert "[user]" in out + assert "[interaction]" in out + assert "snake_case" in out + assert "fail-fast" in out + assert "mutable default" in out + + def test_against_coupon_demo_is_empty( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_prefs(sd) + out = capsys.readouterr().out + assert "No preferences recorded yet." in out + + +# =========================================================================== +# view_labels +# =========================================================================== + + +class TestViewLabels: + """In-process tests for `view_labels(shadow_dir, label_filter)`.""" + + @pytest.mark.parametrize("label", [ + "bug", "security", "performance", "feature-gap", "tech-debt", + ]) + def test_each_label_returns_results_on_fixture( + self, shadow_viewer, coupon_demo, capsys, label + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, label) + out = capsys.readouterr().out + assert f"label(s): {label}" in out + assert f"[{label}]" in out + # Result header always has "(N results)" + m = re.search(r"\((\d+) results\)", out) + assert m and int(m.group(1)) >= 1 + + def test_unknown_label_no_results( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, "zznotalabelzz") + out = capsys.readouterr().out + assert "No discoveries with label(s): zznotalabelzz" in out + + def test_multiple_labels_comma_split( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, "bug,security") + out = capsys.readouterr().out + assert "label(s): bug, security" in out + # Both grouped section headers present + assert "[bug]" in out + assert "[security]" in out + + def test_label_filter_is_lowercased( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, "BUG") + out = capsys.readouterr().out + # Filter is lowercased before matching + assert "label(s): bug" in out + assert "[bug]" in out + + def test_cross_cutting_labels_included( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, "bug") + out = capsys.readouterr().out + # Cross-cutting entries are prefixed with `_cross/` in the + # file column. + assert "_cross/" in out + + def test_also_labeled_displayed( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # One cart.py discovery has labels [bug, performance] + shadow_viewer.view_labels(sd, "bug") + out = capsys.readouterr().out + assert "Also labeled:" in out + + def test_empty_shadow_no_results( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_viewer.view_labels(sd, "bug") + out = capsys.readouterr().out + assert "No discoveries with label(s): bug" in out + + def test_each_result_row_has_file_and_symbol( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_labels(sd, "feature-gap") + out = capsys.readouterr().out + # `::` separator for file::symbol form + assert "::" in out + assert "(verified, source: exploration)" in out + + +# =========================================================================== +# view_recent +# =========================================================================== + + +class TestViewRecent: + """In-process tests for `view_recent(shadow_dir, count)`.""" + + def test_default_count_caps_at_10( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_recent(sd, 10) + out = capsys.readouterr().out + assert "Most Recent Discoveries (top 10)" in out + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + # Fixture has 33 per-file + 3 cross + 0 prefs > 10 + assert len(entries) == 10 + + def test_custom_count(self, shadow_viewer, coupon_demo, capsys): + sd = coupon_demo / ".shadow" + shadow_viewer.view_recent(sd, 3) + out = capsys.readouterr().out + assert "Most Recent Discoveries (top 3)" in out + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + assert len(entries) == 3 + + def test_count_larger_than_available( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + shadow_viewer.view_recent(sd, 50) + out = capsys.readouterr().out + # Only 1 discovery exists in the minimal shadow + entries = [ + l for l in out.splitlines() if l.strip().startswith("[20") + ] + assert len(entries) == 1 + + def test_empty_shadow_prints_nothing( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + shadow_viewer.view_recent(sd, 10) + out = capsys.readouterr().out + assert "No discoveries found." in out + + def test_recently_modified_file_appears_first( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Bump test_cart.py's mtime to "now" so its discoveries should + # rank first. + target = sd / "test_cart.py.md" + now = datetime.now().timestamp() + os.utime(target, (now + 10, now + 10)) + shadow_viewer.view_recent(sd, 3) + out = capsys.readouterr().out + # First entry block should reference test_cart.py + first_entry_idx = out.find("[20") + first_block = out[first_entry_idx:first_entry_idx + 400] + assert "test_cart.py" in first_block + + def test_includes_cross_cutting_type( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Bump a cross file mtime so it shows up in the top + cf = sd / "_cross" / "coupon-case-normalization-mismatch.md" + now = datetime.now().timestamp() + os.utime(cf, (now + 100, now + 100)) + shadow_viewer.view_recent(sd, 5) + out = capsys.readouterr().out + assert "(cross-cutting)" in out + + def test_includes_preferences_when_present( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + _write_shadow(sd, "_prefs.md", """\ + # Preferences + + - A pref we care about. + _(source: user)_ + """) + shadow_viewer.view_recent(sd, 10) + out = capsys.readouterr().out + assert "(preference)" in out + assert "A pref we care about" in out + # Preference-typed rows show `source:` not `(verified, ...)` + assert "source: user" in out + + def test_no_cross_dir_does_not_crash( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_minimal_shadow(tmp_path) + # No _cross/ created + shadow_viewer.view_recent(sd, 10) + out = capsys.readouterr().out + assert "Most Recent Discoveries" in out + + def test_entries_include_timestamp_and_kind( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + shadow_viewer.view_recent(sd, 2) + out = capsys.readouterr().out + # Entry header line format: " [YYYY-MM-DD HH:MM] (kind)" + assert re.search( + r"\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}\] \((discovery|cross-cutting|preference)\)", + out, + ) + + +# =========================================================================== +# view_check_invariants +# =========================================================================== + + +class TestViewCheckInvariants: + """In-process tests for `view_check_invariants(shadow_dir)`. + + Each test builds a deliberately broken `.shadow/` tree in tmp_path + and asserts that the right violation kind is reported. + """ + + def test_clean_coupon_demo_returns_zero( + self, shadow_viewer, coupon_demo, capsys + ): + rc = shadow_viewer.view_check_invariants(coupon_demo / ".shadow") + out = capsys.readouterr().out + assert rc == 0 + assert "✓ Invariants OK" in out + + def test_empty_shadow_returns_zero( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0 + assert "Invariants OK" in out + + def test_missing_cross_file_is_violation( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + (sd / "_cross" / "coupon-case-normalization-mismatch.md").unlink() + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "cross-ref" in out + assert "coupon-case-normalization-mismatch" in out + + def test_invalid_status_enum(self, shadow_viewer, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad status. + _(maybeverified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "maybeverified" in out + + def test_invalid_source_enum(self, shadow_viewer, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad source. + _(verified, source: psychic)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "psychic" in out + + def test_invalid_label(self, shadow_viewer, tmp_path, capsys): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Bad label. + _(verified, source: exploration, labels: [unicorn])_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "unicorn" in out + + def test_heading_without_backticks( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_in_backticks + + - Hi. + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "heading" in out + + def test_file_level_heading_does_not_violate( + self, shadow_viewer, tmp_path, capsys + ): + """`## File-Level` is allowed without backticks.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## File-Level + + - A file-level discovery. + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0, out + assert "Invariants OK" in out + + def test_also_involves_without_backticks( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - With bad anchors. + _(verified, source: exploration)_ + Also involves: b.py::bar, c.py::baz + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "anchor" in out + + def test_also_involves_missing_symbol_after_colons( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - Empty sym. + _(verified, source: exploration)_ + Also involves: `b.py::` + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + # The empty-symbol form fails the strict regex (which requires + # non-empty after ::), so the file_sym_re finds zero anchors and + # we hit the "needs backtick anchors" branch instead. + assert "anchor" in out + assert "needs `file::symbol`" in out + + def test_cross_missing_category( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + """) + _write_shadow(sd, "_cross/no-cat.md", """\ + # No category here + + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "Category" in out + + def test_cross_invalid_category( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/bad-cat.md", """\ + # Bad category here + + **Category**: bogus + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "enum" in out + assert "bogus" in out + + def test_cross_missing_metadata_line( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-meta.md", """\ + # No meta here + + **Category**: pattern + **Refs**: + - `a.py::foo` + + **Discovery**: Something. + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "missing trailing" in out + + def test_cross_missing_refs_block( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/no-refs.md", """\ + # No refs here + + **Category**: pattern + + **Discovery**: Something. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "schema" in out + assert "Refs" in out + + def test_cross_ref_missing_symbol( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + # Note: file_sym_re demands "::" in backticks. We use a backtick + # ref with no symbol after `::`. + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - hi. + _(verified, source: exploration)_ + + ## Cross-References + + - [bad-anchor](_cross/bad-anchor.md) + """) + _write_shadow(sd, "_cross/bad-anchor.md", """\ + # Bad anchor + + **Category**: pattern + **Refs**: + - `a.py::` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "anchor" in out + + def test_back_pointer_missing( + self, shadow_viewer, tmp_path, capsys + ): + """_cross/x.md references a.py::foo but a.py.md has no + Cross-References section pointing back to x.md.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## `foo` + + - A discovery. + _(verified, source: exploration)_ + """) + _write_shadow(sd, "_cross/orphan.md", """\ + # Orphan + + **Category**: pattern + **Refs**: + - `a.py::foo` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "cross-ref" in out + assert "does not link back to" in out + + def test_back_pointer_references_nonexistent_file( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "_cross/ghost.md", """\ + # Ghost + + **Category**: pattern + **Refs**: + - `does/not/exist.py::foo` + + **Discovery**: stuff. + + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + assert "no such shadow file exists" in out + + def test_violation_line_format( + self, shadow_viewer, tmp_path, capsys + ): + """Output is grep-friendly: `path:line: kind: message`.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_backticked + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 1 + # At least one line of form path:line: kind: msg + assert re.search(r"a\.py\.md:\d+: heading: ", out) + + def test_multiple_violations_reported( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## not_backticked + + ## `foo` + + - bad. + _(maybeverified, source: psychic, labels: [unicorn])_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + err = capsys.readouterr().err + assert rc == 1 + # heading + 3 enum violations = at least 4 lines + violation_lines = [ + l for l in out.splitlines() + if re.match(r"^[^:]+:\d+: \w+: ", l) + ] + assert len(violation_lines) >= 3 + + def test_count_summary_emitted_on_stderr( + self, shadow_viewer, tmp_path, capsys + ): + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", "## not_backticked\n") + rc = shadow_viewer.view_check_invariants(sd) + captured = capsys.readouterr() + assert rc == 1 + # Final count line goes to stderr + assert "invariant violation(s) found" in captured.err + + def test_special_headings_allowed( + self, shadow_viewer, tmp_path, capsys + ): + """`## Notes`, `## Metadata`, `## File-Level Notes` are allowed + without backticks.""" + sd = _make_shadow_root(tmp_path) + _write_shadow(sd, "a.py.md", """\ + # Shadow: a.py + + ## Notes + + Some prose. + + ## Metadata + + Some metadata. + + ## File-Level Notes + + More prose. + + ## `real_sym` + + - A discovery. + _(verified, source: exploration)_ + """) + rc = shadow_viewer.view_check_invariants(sd) + out = capsys.readouterr().out + assert rc == 0, out + + +# =========================================================================== +# main() — in-process via sys.argv (covers the dispatcher) +# =========================================================================== + + +class TestMainInProcess: + """Drive `main()` directly so coverage of the dispatch arms is captured. + + All paths use `--shadow-dir ` to avoid relying + on cwd. Where we need a true CLI smoke (e.g., to verify `--help` + output), use subprocess. + """ + + def _shadow(self, coupon_demo): + return str(coupon_demo / ".shadow") + + def test_summary_dispatch( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, ["--shadow-dir", self._shadow(coupon_demo), + "--summary"] + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_no_args_default_is_summary( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, ["--shadow-dir", self._shadow(coupon_demo)] + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_search_dispatch( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--search", "coupon"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Search: 'coupon'" in out + + def test_prefs_dispatch( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), "--prefs"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "No preferences recorded yet." in out + + def test_labels_dispatch_bug( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--labels", "bug"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "label(s): bug" in out + + def test_recent_dispatch_with_n( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--recent", "4"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Most Recent Discoveries (top 4)" in out + + def test_recent_dispatch_no_n( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), "--recent"], + ) + out = capsys.readouterr().out + assert rc == 0 + # Default count is 10 + assert "Most Recent Discoveries (top 10)" in out + + def test_top_dispatch( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--top", "cart.py"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "for cart.py:" in out + + def test_top_dispatch_with_labels_and_limits( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--top", "cart.py", + "--top-labels", "bug", + "--top-limit", "2", + "--top-max-chars", "0"], + ) + out = capsys.readouterr().out + assert rc == 0 + bullets = [l for l in out.splitlines() if l.startswith("- [")] + assert len(bullets) <= 2 + + def test_check_invariants_clean_exits_zero( + self, shadow_viewer, coupon_demo, capsys + ): + rc = _call_main( + shadow_viewer, + ["--shadow-dir", self._shadow(coupon_demo), + "--check-invariants"], + ) + out = capsys.readouterr().out + assert rc == 0 + assert "Invariants OK" in out + + def test_check_invariants_dirty_exits_one( + self, shadow_viewer, coupon_demo, capsys + ): + sd = coupon_demo / ".shadow" + # Inject a heading violation + cart = sd / "cart.py.md" + cart.write_text( + cart.read_text(encoding="utf-8").replace( + "## `COUPON_CACHE`", "## COUPON_CACHE", 1 + ), + encoding="utf-8", + ) + rc = _call_main( + shadow_viewer, + ["--shadow-dir", str(sd), "--check-invariants"], + ) + out = capsys.readouterr().out + assert rc == 1 + assert "heading" in out + + def test_explicit_shadow_dir_missing( + self, shadow_viewer, tmp_path, capsys + ): + bogus = tmp_path / "does-not-exist" + rc = _call_main( + shadow_viewer, ["--shadow-dir", str(bogus), "--summary"] + ) + err = capsys.readouterr().err + assert rc == 1 + assert "No .shadow/ directory found" in err + + def test_auto_detect_via_chdir( + self, shadow_viewer, coupon_demo, monkeypatch, capsys + ): + """No --shadow-dir: cwd is walked up to find .shadow/.""" + monkeypatch.chdir(coupon_demo) + rc = _call_main(shadow_viewer, ["--summary"]) + out = capsys.readouterr().out + assert rc == 0 + assert "Shadow Knowledge Base Summary" in out + + def test_auto_detect_no_shadow_in_cwd( + self, shadow_viewer, tmp_path, monkeypatch, capsys + ): + monkeypatch.chdir(tmp_path) + rc = _call_main(shadow_viewer, ["--summary"]) + err = capsys.readouterr().err + assert rc == 1 + assert "No .shadow/ directory found" in err + + +# --- main(): subprocess smoke (covers true argv parsing, help, errors) ---- + + +class TestMainSubprocess: + """End-to-end CLI smoke. Subprocess output is the contract here — + these don't add coverage but they catch dispatcher / argparse regressions + the in-process tests can't (e.g. --help, mutually exclusive errors).""" + + @pytest.mark.slow + @pytest.mark.integration + def test_help_exits_zero(self, repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--help") + assert r.returncode == 0 + # argparse prints usage and the description + assert "usage:" in r.stdout.lower() + assert "--summary" in r.stdout + assert "--search" in r.stdout + assert "--check-invariants" in r.stdout + + @pytest.mark.slow + @pytest.mark.integration + def test_unknown_flag_exits_nonzero(self, repo_root, coupon_demo): + r = _run_viewer(repo_root, coupon_demo, "--no-such-flag") + assert r.returncode != 0 + assert "unrecognized" in r.stderr or "unrecognized" in r.stdout + + @pytest.mark.slow + @pytest.mark.integration + def test_mutually_exclusive_flags(self, repo_root, coupon_demo): + """--summary and --prefs are in the same exclusive group.""" + r = _run_viewer( + repo_root, coupon_demo, "--summary", "--prefs" + ) + assert r.returncode != 0 + # argparse error mentions "not allowed with" + assert "not allowed with" in r.stderr diff --git a/tests/test_smoke.py b/tests/test_smoke.py index 7a10cf2..b88b3be 100644 --- a/tests/test_smoke.py +++ b/tests/test_smoke.py @@ -13,14 +13,11 @@ def test_repo_root_resolves(repo_root): def test_all_script_fixtures_load( shadow_init, shadow_viewer, dream_reconcile, dream_validate, - dream_coverage, dream_lineage, meditate_repair, nap, coherence, dream_tools, - shadow_knowledge, shadow_reader, + dream_coverage, dream_lineage, meditate_repair, nap, coherence, dream_tools, citations, ): for mod, expected_attr in [ (shadow_init, "main"), (shadow_viewer, "main"), - (shadow_reader, "main"), - (shadow_knowledge, "view_file"), (dream_reconcile, "main"), (dream_validate, "main"), (dream_tools, "pin_tooling"), @@ -29,6 +26,7 @@ def test_all_script_fixtures_load( (meditate_repair, "main"), (nap, "main"), (coherence, "validate_connection"), + (citations, "record_citations"), ]: assert hasattr(mod, expected_attr), \ f"{mod.__name__} missing expected attribute {expected_attr!r}" From 10b76dc2e1024274d7e80c386feaf7138f530f7c Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Thu, 24 Sep 2026 00:15:51 -0400 Subject: [PATCH 10/11] Respect discovery boundaries when updating citation scores Exclude fenced examples and unrelated headings from exact-entry matching, merge cross-cutting scores at the actual discovery metadata, and clarify that existing shadows require no score reset. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- skills/shadow-frog-dream/dream-reconcile.py | 20 ++++++---- skills/shadow-frog/SKILL.md | 5 +++ skills/shadow-frog/_citations.py | 38 +++++++++++++++--- tests/skills/shadow_frog/test_citations.py | 39 ++++++++++++++++++- .../shadow_frog_dream/test_citation_scores.py | 18 +++++++++ 5 files changed, 106 insertions(+), 14 deletions(-) diff --git a/skills/shadow-frog-dream/dream-reconcile.py b/skills/shadow-frog-dream/dream-reconcile.py index 869cefb..3348df0 100755 --- a/skills/shadow-frog-dream/dream-reconcile.py +++ b/skills/shadow-frog-dream/dream-reconcile.py @@ -47,7 +47,10 @@ sys.dont_write_bytecode = True sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "shadow-frog")) try: - from _citations import CitationError, metadata_score, set_metadata_score, validate_score + from _citations import ( + CitationError, cross_metadata_line, metadata_score, set_metadata_score, + validate_score, + ) except ImportError as exc: raise SystemExit("ERROR: Missing core citation metadata parser; reinstall the full skill set") from exc finally: @@ -803,13 +806,14 @@ def _merge_refs_into_cross_file(cross_path, new_refs, *, repo_root, citation_sco break to_add = [r for r in new_refs if r and r not in existing] score_changed = False - for index, line in enumerate(lines): - if line.strip().startswith("_(") and "source:" in line: - previous_score = metadata_score(line) - if citation_score > previous_score: - lines[index] = set_metadata_score(line, citation_score) - score_changed = True - break + metadata_index = cross_metadata_line(lines) + if metadata_index is not None: + previous_score = metadata_score(lines[metadata_index]) + if citation_score > previous_score: + lines[metadata_index] = set_metadata_score(lines[metadata_index], citation_score) + score_changed = True + elif citation_score: + raise CitationError(f"{cross_path}: restore discovery metadata before merging citation scores") if not to_add and not score_changed: return False lines[block_end:block_end] = [f'- `{r}`' for r in to_add] diff --git a/skills/shadow-frog/SKILL.md b/skills/shadow-frog/SKILL.md index 3a5238e..1817ec6 100644 --- a/skills/shadow-frog/SKILL.md +++ b/skills/shadow-frog/SKILL.md @@ -51,6 +51,10 @@ discovery or preference has one visible `citation_score`: a nonnegative integer, initially 0. Missing scores also mean 0. It is approximate, agent-reported revisit frequency, not confidence or proof of usefulness. +When updating an existing shadow, preserve all knowledge and existing scores: +missing `citation_score` fields mean `0` and are added on the first recorded +citation, so no reinitialization is required. + After deliberately consulting an entry, record one citation for it **per task**. Do not count every entry merely because its file was opened, repeat the count on rereads, or count automatic previews that you did not use. Batch updates when @@ -71,6 +75,7 @@ Use `--symbol File-Level` for file-wide entries; omit `--symbol` for preferences and `_cross/` files. Repeat `--text` to update several entries in one section atomically. `--shadow-dir` identifies an explicit nonstandard shadow root. Unknown/ambiguous claims and invalid scores fail with corrective feedback. +Fenced examples and unrelated headings are not citation targets. The helper locks only its target file and publishes the score changes atomically. Coordinate citation writes with ordinary knowledge edits; unrelated editors do diff --git a/skills/shadow-frog/_citations.py b/skills/shadow-frog/_citations.py index a293901..795b808 100644 --- a/skills/shadow-frog/_citations.py +++ b/skills/shadow-frog/_citations.py @@ -65,19 +65,36 @@ def _claim(value): def _entries(lines, kind): section = None claim = None + fence = None for index, line in enumerate(lines): stripped = line.strip() - heading = re.fullmatch(r"#{2,3}\s+(?:`(.+)`|(File-Level|Cross-References))", stripped) - if heading: - section = _symbol(heading.group(1) or heading.group(2)) + if fence is not None: + if claim is not None: + claim.append(stripped) + if re.fullmatch(re.escape(fence[0]) + "{" + str(len(fence)) + ",}", stripped): + fence = None + continue + opening = re.match(r"`{3,}|~{3,}", stripped) + if opening: + fence = opening.group(0) + if claim is not None: + claim.append(stripped) + continue + if re.match(r"#{1,6}\s", stripped): + heading = re.fullmatch(r"#{2,3}\s+(?:`(.+)`|(File-Level|Cross-References))", stripped) + section = _symbol(heading.group(1) or heading.group(2)) if heading else None claim = None + continue if kind == "cross" and stripped.startswith("**Discovery**:"): claim = [stripped.partition(":")[2].strip()] elif kind != "cross" and stripped.startswith("- "): claim = [stripped[2:]] if kind == "preference" or section not in (None, "Cross-References") else None elif claim is not None: if stripped.startswith("_("): - metadata_score(line) + try: + metadata_score(line) + except CitationError as exc: + raise CitationError(f"Line {index + 1}: {exc}") from exc yield section if kind == "file" else None, _claim("\n".join(claim)), index claim = None elif stripped.startswith(("Also involves:", "Dream report:", "#")): @@ -86,6 +103,14 @@ def _entries(lines, kind): claim.append(stripped) +def cross_metadata_line(lines): + """Locate the single actual cross-cutting discovery's metadata, excluding examples.""" + entries = list(_entries(lines, "cross")) + if len(entries) > 1: + raise CitationError("Cross-cutting file contains multiple discoveries; resolve ambiguity before updating scores") + return entries[0][2] if entries else None + + def _target(path, shadow_dir): literal = Path(path).absolute() if shadow_dir is None: @@ -146,7 +171,10 @@ def record_citations(path, texts, *, symbol=None, shadow_dir=None, timeout=2.0): with _locked(path, timeout): original = path.read_bytes() lines = original.decode("utf-8").splitlines(keepends=True) - entries = list(_entries(lines, kind)) + try: + entries = list(_entries(lines, kind)) + except CitationError as exc: + raise CitationError(f"{path}: {exc}") from exc updates = [] for text in targets: matches = [ diff --git a/tests/skills/shadow_frog/test_citations.py b/tests/skills/shadow_frog/test_citations.py index 28ddad3..bad62ab 100644 --- a/tests/skills/shadow_frog/test_citations.py +++ b/tests/skills/shadow_frog/test_citations.py @@ -89,6 +89,42 @@ def test_literal_whitespace_is_not_normalized_away(citations, tmp_path): assert path.read_bytes() == before +@pytest.mark.parametrize("boundary", ["notes-heading", "fenced-example"]) +def test_non_discovery_sections_cannot_be_cited_as_previous_symbol(citations, tmp_path, boundary): + path = per_file(tmp_path) + sample = "- Example-only claim.\n _(verified, source: exploration, citation_score: 2)_\n" + if boundary == "notes-heading": + sample = "## Notes\n\n" + sample + else: + sample = "```markdown\n" + sample + "```\n" + path.write_text( + path.read_text(encoding="utf-8").replace("## Cross-References", sample + "\n## Cross-References"), + encoding="utf-8", + ) + before = path.read_bytes() + with pytest.raises(citations.CitationError, match="found 0"): + citations.record_citations(path, ["Example-only claim."], symbol="Auth.login") + assert path.read_bytes() == before + + +def test_discovery_after_fenced_example_keeps_its_actual_symbol(citations, tmp_path): + path = per_file(tmp_path) + fenced = ( + "```markdown\n## `WrongSymbol`\n\n- Example-only claim.\n" + " _(verified, source: exploration, citation_score: 2)_\n```\n\n" + "- Actual later claim.\n _(verified, source: user, citation_score: 0)_\n\n" + ) + path.write_text( + path.read_text(encoding="utf-8").replace("## Cross-References", fenced + "## Cross-References"), + encoding="utf-8", + ) + before = path.read_text(encoding="utf-8") + citations.record_citations(path, ["Actual later claim."], symbol="Auth.login") + assert path.read_text(encoding="utf-8") == before.replace( + "source: user, citation_score: 0", "source: user, citation_score: 1", + ) + + def test_crlf_bom_and_existing_metadata_are_preserved(citations, tmp_path): path = per_file(tmp_path, score=", citation_score: 7", newline="\r\n") before = b"\xef\xbb\xbf" + path.read_bytes() @@ -195,8 +231,9 @@ def test_json_score_requires_a_nonnegative_integer(citations, score): def test_invalid_existing_score_is_not_reset(citations, tmp_path, raw): path = per_file(tmp_path, score=f", citation_score: {raw}") before = path.read_bytes() - with pytest.raises(citations.CitationError, match="Malformed metadata"): + with pytest.raises(citations.CitationError, match="Line .*Malformed metadata") as exc: citations.record_citations(path, ["Key `a b` is distinct."], symbol="Auth.login") + assert str(path.resolve()) in str(exc.value) assert path.read_bytes() == before diff --git a/tests/skills/shadow_frog_dream/test_citation_scores.py b/tests/skills/shadow_frog_dream/test_citation_scores.py index a875a8f..ceee34d 100644 --- a/tests/skills/shadow_frog_dream/test_citation_scores.py +++ b/tests/skills/shadow_frog_dream/test_citation_scores.py @@ -45,6 +45,24 @@ def test_existing_cross_refs_and_scores_merge_independently(dream_reconcile, tmp assert "`b.py::call`" in path.read_text(encoding="utf-8") +def test_cross_merge_does_not_edit_metadata_inside_examples(dream_reconcile, tmp_path): + path = tmp_path / ".shadow/_cross/contract.md" + path.parent.mkdir(parents=True) + original = ( + "# Contract\n\n**Refs**:\n- `a.py::run`\n\n" + "```markdown\n_(verified, source: exploration, citation_score: 2)_\n```\n\n" + "**Discovery**: Shared claim.\n\n" + "_(verified, source: exploration, citation_score: 4)_\n" + ) + path.write_text(original, encoding="utf-8") + assert dream_reconcile._merge_refs_into_cross_file( + str(path), ["a.py::run"], repo_root=str(tmp_path), citation_score=7, + ) + assert path.read_text(encoding="utf-8") == original.replace( + "citation_score: 4", "citation_score: 7", + ) + + @pytest.mark.parametrize("score", [-1, 0.5, True, None, "4"]) @pytest.mark.parametrize("kind", ["discoveries", "cross_cutting"]) def test_invalid_manifest_score_fails_before_any_publication(dream_reconcile, tmp_path, score, kind): From cb107033b6c581fccb0db858b76a1ab71b738125 Mon Sep 17 00:00:00 2001 From: "Xingdi (Eric) Yuan" <4028684+xingdi-eric-yuan@users.noreply.github.com> Date: Mon, 28 Sep 2026 09:31:09 -0400 Subject: [PATCH 11/11] Date changelog entry for knowledge citations Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- CHANGELOG.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 04a7c2a..7e2910c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ shadow knowledge bases for any codebase. --- -## Unreleased +## 2026-09-28 ### Added - **Visible knowledge citations** — `citation_score` lives alongside discovery