diff --git a/.agents/skills/kb-cite-audit/SKILL.md b/.agents/skills/kb-cite-audit/SKILL.md new file mode 100644 index 0000000..7024cbe --- /dev/null +++ b/.agents/skills/kb-cite-audit/SKILL.md @@ -0,0 +1,129 @@ +--- +name: kb-cite-audit +description: 知识库引用语义审计工作流——确定性枚举（论断, 引用）审计对，逐条以 fresh-context 判定被引原文是否真的支撑论断，判定入验证台账，未通过的落 CAUTION 标注交人裁决。当用户提到"审计引用"、"核对引用"、"引用是否正确"、"cite audit"、"验证来源"、"检查论断与原文一致性"时使用此 skill。 +--- + +# 知识库引用语义审计工作流（kb-cite-audit） + +你现在是知识库的 **引用审计员**。任务：验证 wiki 论断的块级引用在**语义上**成立——被引的那个块是否真的支撑该论断。结构 lint（broken-refs / bare-claims / coarse-citations）只保证「锚点存在、有引用、粒度够细」；本流程补上「引的内容对不对」。 + +> **Workspace 前提（必读）**：数据层按主题隔离在 `workspaces//` 下。本文所有 `wiki/`、`raw/`、`log.md` 路径均相对当前 workspace；`k.py` 命令加 `--workspace `；`Read` / `Edit` / `git add` 用带 workspace 的全路径。 + +> **raw 不在场即降级（必读）**：本仓的 demo 库不分发 `raw/`（版权），协作者浅 clone 同理。`extract-claims` 会把这类引用标为 `raw-not-distributed`——**直接跳过、勿烧 token 强行判定**；语义审计只在 raw 在场的环境（dev 仓 / 用户自己的库）实际执行。 + +**核心原则**： + +- **KB 出数据、agent 出判断**：`extract-claims` / `cite-audit-log` 是确定性工具；「支撑与否」的判定由你（外部 agent）做，这是全流程唯一需要智能的一步。 +- **fresh-context 判定**：只依据 `claim_text` + 现场取回的被引原文下判断。**禁止**凭「我 ingest 时读过这篇」的记忆判定——上下文污染正是错引的成因，不能再用它做裁判。 +- **不默默修正**：发现错引不改论断本身，落 CAUTION 标注 + `citation-suspect` 标签，让人在工作台裁决（与 kb-lint 的冲突处理铁律一致）。 +- **台账是 memoization，不是真相**：`.cache/citation_audit.jsonl` 只记「谁在何时核验过什么」，删了重审即重建；「这条引用有问题」这一知识状态**只以 markdown 标注为准**（`list-suspect-citations` 扫 markdown，不读台账）。 + +--- + +## 第 1 步：枚举待审对 + +```bash +# 增量模式（周检 / 例行）：只审从未审过 + 内容已漂移的对，配额抽样 +python scripts/k.py --workspace extract-claims --unaudited-only --sample 20 --seed --json + +# 全量首审（一次性存量清偿）：去掉 --sample；量大时分多轮 +python scripts/k.py --workspace extract-claims --unaudited-only --json +``` + +- `--seed` 用当周周号（如 `2026-W27`）：同周重跑取样一致（可复现），跨周覆盖累积。 +- 「已审」判定是双 hash pin：claim 内容变 → pair_id 变；被引块内容变（含 ^h- 节**正文**重写而标题不变）→ `target_content_hash` 漂移——任一侧变化都自动回到未审，无需人为盯。 + +## 第 2 步：分诊 + +按 `target_status` 分流，只对 `ok` 的对做语义判定： + +| target_status | 处置 | +|---|---| +| `ok` | 进入第 3 步判定 | +| `raw-not-distributed` | `cite-audit-log --verdict UNVERIFIABLE` 批量入账（不烧 token；k.py 校验：只有目标确实不可得才接受 UNVERIFIABLE） | +| `file-missing` / `anchor-missing` | 归 broken-refs 流程修复，本流程跳过（不双报） | + +## 第 3 步：逐条 fresh-context 判定 + +对每对（建议 20-30 条一批，批间不携带前批内容）： + +1. 取回被引原文（**必须现场取，不用记忆**）： + ```bash + python scripts/k.py --workspace read-block # ^p-/^t-/^c-/^f- + python scripts/k.py --workspace read-section # ^h- + ``` +2. 判定三问： + - ① **关键事实在场**：论断中的数字 / 日期 / 主体是否出现在被引原文？ + - ② **直接支撑**：论断是否被原文直接支撑（不需要额外推理 / 拼接其他来源）？ + - ③ **语义 drift**：有无过度概括（"多数"写成"所有"）、加了原文没有的限定词 / 程度词、因果错置（correlation 写成 causation）？ +3. 映射 verdict：三问全过 → `SUPPORTED`；部分支撑 / 措辞偏移 → `PARTIAL`；被引块与论断无关或关键事实不在 → `UNSUPPORTED`；论断与被引证据**相反** → `CONTRADICTED`。 +4. **盲填复核（含数字的对必做）**：`extract-claims --unaudited-only --cloze` 输出挖空论断（数字→⟦N1⟧）；核验时**只看「挖空论断 + 被引原文」**填空（不看期望值），`python scripts/k.py cloze-check --batch ` 机器判分（块级 union：一块多引用时数字由块内任一引用的原文填出即可）。判分未过按 UNSUPPORTED 处置——盲填从原理上消灭判定式审核的附和偏差，并抓「数字巧合在场但归属错误」。 +5. **跨模型二审（推荐）**：`python tools/cite-audit/audit.py --workspace [--unaudited-only|--all --sample N --seed ]`——外部客户端（tools/ 例外区）调 DeepSeek 自动跑双通道并入台账；`--all --sample` 模式可对已 SUPPORTED 记录换模型交叉复查（防橡皮图章与同源盲区）。问答 / 导出草稿可用 `--draft /tmp/answer.md`，判定会通过草稿专属 pair 受控入账。需 `DEEPSEEK_API_KEY`；无 key 时以 fresh-context 子 agent 反驳式抽查替代。 + + 审计器的结果协议是 fail-closed：exit `0` = 所有实际判定对通过，exit `1` = 完成判定且发现语义未通过，exit `2` = 网络 / 协议 / 核验包 / quote / skipped 等导致审计未完成。`2` 不是「没发现问题」，禁止当作通过；截断 claim/evidence/cloze 和多来源盲填冲突都进 incomplete，不冒充 UNSUPPORTED 或语义查全命中。 + +**防误伤细则**（判定前先过一遍）： + +- **多来源合成论断**（`multi_source: true`，多见于 analysis / comparison 页）：一块多引用时，引用只对其**紧邻的前方分句**负责；单个被引块只支撑论断的一部分是**正常形态** → `PARTIAL` 且**不落标注**。只有「被引块与归属分句无关或矛盾」才 `UNSUPPORTED`。 +- **合法转述白名单**（按 `SUPPORTED` 处理，note 记明）：约数舍入（约 40% ↔ 39.2%）、跨语言日期 / 数字格式转写（August 29 ↔ 2025-08-29）、已标 `[KB 推算: ^锚]` 且现场验算成立的派生算术（差值 / 倍数 / 单位换算——验算不成立则 `UNSUPPORTED`）。 +- **^h- 大节引用**：支撑句埋在整节多处（分布式支撑）属正常 → `PARTIAL` 不落标注，建议 note 记「宜降 ^p- 级锚」。 + +## 第 4 步：判定入台账 + +```bash +# 单条（agent 记 SUPPORTED 必须附 --evidence：被引块原文的一段字面子串，k.py 会校验—— +# 证明确实取回过原文，杜绝橡皮图章） +python scripts/k.py --workspace cite-audit-log --pair --verdict SUPPORTED \ + --evidence "<从 read-block 返回内容里复制的一段>" --note "三问全过" + +# 批量：JSONL 文件每行 {"pair_id": "...", "verdict": "...", "note": "...", "evidence": "..."} +python scripts/k.py --workspace cite-audit-log --batch /tmp/verdicts.jsonl +``` + +k.py 的确定性校验（被拒说明流程有问题，不要绕）：pair 过期（内容已变）拒绝；目标可解析时记 UNVERIFIABLE 拒绝；SUPPORTED 无 / 假 evidence 拒绝。 + +## 第 5 步：UNSUPPORTED 落 CAUTION 标注 + +对 `UNSUPPORTED` / `CONTRADICTED`（以及**实质事实错**的 PARTIAL），`Edit` 在论断块**正下方**追加审计标注（格式固定，`list-suspect-citations` 靠它扫描）： + +```markdown +> [!CAUTION] 引用审计未通过 — YYYY-MM-DD +> **论断**：<论断句摘录>（块 ^p-4-34d5b1） +> **被引块**：[[#^]] +> **审计判定**：UNSUPPORTED — <一句差异说明，如"被引表格中数字为 65.9 非 66.9"> +> **建议**：<改引 [[#^<正确锚>]] / 修正论断数字 / 删除论断> +> **状态**：⏳ 待人类判别 +``` + +同时该页 frontmatter `tags` 追加 `citation-suspect`。**不改论断原文**——修复由人裁决（或人授权后走正规修复 + 删标注，claim 变化会自动触发重审）。 + +## 第 6 步：收尾对账 + 汇报 + commit + +```bash +python scripts/k.py --workspace list-suspect-citations --check-ledger +``` + +- `ledger-unsupported-without-marker` 必须为空——有 = CAUTION 标注被删而论断未改（「删标注蒸发」），恢复标注。 +- log.md 追加 `lint` 类条目：`- 引用审计： 对（S/P/U = x/y/z），UNSUPPORTED 已标注 z 条`。 +- git commit（只含 `wiki/**` 标注改动 + log.md；**.cache 台账不入库**）：`git commit -m "lint: 引用审计 对"`。 + +--- + +## 完成检查清单 + +- [ ] `extract-claims --unaudited-only` 本批返回的 `ok` 对已全部判定并入账 +- [ ] 核验包的 claim / evidence / cloze 未截断，审计器没有 incomplete / skipped / ledger error +- [ ] `raw-not-distributed` 的对已批量记 UNVERIFIABLE（没有烧 token 强判） +- [ ] 每条 SUPPORTED 都带真实 `--evidence`（k.py 校验通过） +- [ ] UNSUPPORTED / CONTRADICTED 已落 CAUTION 标注 + `citation-suspect` 标签，**没有默默修正论断** +- [ ] `list-suspect-citations --check-ledger` 对账一致 +- [ ] log.md 记账、git commit 完成（.cache 不入库） + +## 反例（绝对不要做） + +- ❌ 凭「我 ingest 时读过」的记忆判定，不现场 `read-block` 取回原文——上下文污染不能当裁判 +- ❌ 对可解析的目标记 UNVERIFIABLE 跳过劳动（k.py 会拒绝；被拒就老实取原文） +- ❌ 把多来源合成论断的正常 PARTIAL 当错引落标注（误伤会让人对审计失去信任） +- ❌ 发现错引直接改论断数字（默默修正）——必须落标注走人裁 +- ❌ 删 CAUTION 标注但不修论断（对账会报 ledger-unsupported-without-marker） +- ❌ 把台账 jsonl 提交进 git（它是派生层缓存，删了重审即重建） diff --git a/.agents/skills/kb-edit-source/SKILL.md b/.agents/skills/kb-edit-source/SKILL.md new file mode 100644 index 0000000..c671887 --- /dev/null +++ b/.agents/skills/kb-edit-source/SKILL.md @@ -0,0 +1,261 @@ +--- +name: kb-edit-source +description: 知识库来源编辑工作流——当底层来源（raw 的原文）内容发生变化时，安全地改源头、重转换、把受影响的 wiki 引用与论断同步更新，原子提交。当用户提到"改 raw"、"修改来源"、"原文改了要同步 wiki"、"来源内容更新了"、"edit source"、"改了原始笔记"、"更新引用的原文"时使用此 skill。 +--- + +# 知识库来源编辑工作流（kb-edit-source） + +你现在是知识库的 **来源同步专家**。当一份**原文来源的内容变了**（数字改了、段落重写、章节增删），你要保证引用它的 wiki 页面不悄悄过期。严格遵守以下流程。 + +> **Workspace 前提（必读）**：数据层按主题隔离在 `workspaces//` 下。本文中所有 `wiki/`、`raw/`、`exports/`、`log.md` 路径均**相对于当前 workspace**，实际位于 `workspaces//`（如 `workspaces/smb-ecommerce/wiki/...`）。 +> - 默认 workspace 为 `smb-ecommerce`，不显式指定时即用它（向后兼容）。 +> - `k.py` / `convert.py` 用 `--workspace ` 指定 workspace（参数紧跟脚本名后）。 +> - **跨独立项目（KB_ROOT 外置）时**：数据不在引擎目录，而在 `KB_ROOT` 指向的数据根。此时**每条 `k.py` / `convert.py` / `git` 命令都必须携带 `KB_ROOT`，且 git 必须用 `git -C "$KB_ROOT"` 打到数据仓**（不是引擎 cwd）。`KB_ROOT` 含空格 / 中文时**务必加引号**。 +> - `Read` / `Edit` 与 wiki 路径参数用**带 workspace 的全路径**（如 `workspaces//wiki/sources/.md`）。 + +**核心原则（不可违反）**： + +- **agent 绝不写 `raw/`**。`Edit(raw/**)` / `Write(raw/**)` / `Edit(**/raw/**)` 等被 `.claude/settings.json` 硬 deny（绝对优先、无 ask 回退）。**raw 内容的唯一编辑发生在 raw 之外的「真相源原件」上**（如 Obsidian vault 里的 `.md`）；**raw 的唯一写入者是 `convert.py`（只加锚点）**；把更新后的原件放进 raw 用 `cp`（= ingest「把原始文件放入 raw」的入口，是机械镜像、不是手改内容）。 +- **markdown + Git 是唯一真相源**；锚点是内容的确定性函数——改内容 → 锚点变 → `list-broken-refs` 精确暴露失效引用。你的工作就是把这些失效引用逐条修回，并把**事实真变了**的论断按新原文改写。 +- **删除即标记**：被删/被反驳的论断，改引新锚 / 挂 `[需要来源]` + `#to-be-updated` / 走冲突标注——**绝不静默删引用洗白**。 +- **一次来源同步 = 一个 git commit**（只含 wiki + log.md；raw 是否入库见第 8 步按 `git check-ignore` 决定）。 + +> **本 skill 与 partial re-ingest 的分工**：partial re-ingest 是「**来源没变**、把扫读章节升级为深读」；本 skill 是「**来源内容真的变了**」。别混。 + +--- + +## 第 0 步：先判来源类型（硬分支，决定本 skill 适不适用） + +| 来源类型 | 判据 | 本 skill 怎么走 | +|---|---|---| +| **原生 markdown 来源** | raw 里那个 `.md` 就是内容本体（如 Obsidian 笔记的副本），无同名二进制原件 | ✅ 适用：内容在**真相源原件**（Obsidian `.md`）上改，见下 | +| **二进制派生来源** | raw 里有同名 `.pdf`/`.docx`/`.html`，`.md` 是 `convert.py` 的派生物 | ⚠️ 内容编辑**不适用 agent**：需人改二进制原件再 convert；agent 只做「重转换 + wiki 迁移」（第 4-8 步） | +| **human-only / locked 来源** | 对应 source_summary 或原件带 `#human-only` / `locked: true` | ❌ 拒绝驱动编辑，转交人类 | + +判二进制：`ls workspaces//raw// | grep ` 看有无同名 pdf/docx。**下文以「原生 markdown 来源、真相源在外部（如 Obsidian）」为主线**；二进制来源跳过第 2-3 步（人改原件），其余相同。 + +--- + +## 第 1 步：编辑前——用 backlinks 圈定全部受影响面 + +**改任何东西之前**，先落一份「谁引用了这个来源」的清单——因为改动后锚点会变，事后光靠 broken-refs 可能定位不全扇出。 + +```bash +# 列出所有引用该 raw 的 wiki 页 + 具体锚点（KB_ROOT 外置时加前缀） +python scripts/k.py --workspace backlinks raw/articles/.md +``` + +把命中的 wiki 页、每处 `#^anchor`、上下文抄下来（这是第 5-6 步要逐条修的清单）。 + +- **backlinks 为空**（该来源还没被任何 wiki 引用）：跳过第 5-6 步的引用迁移，但**仍要做第 3-4 步重转换**保持派生层新鲜；若本次无 wiki 变更则无需 KB commit（同步只落原件 + raw 副本），log.md 仍记一条 `update` 备案。 + +**留证（DELETE/重写高危时必做）**：对将被删或大改的被引块，先 `read-block` 取回**旧原文快照**贴进 log.md 或对应 source_summary 的历史 NOTE——KB 仓不留 raw 历史（见第 8 步），旧原文一旦删掉在 KB 内不可恢复。 + +```bash +python scripts/k.py --workspace read-block raw/articles/.md ^p-12-7d8e9a +``` + +--- + +## 第 2 步：在「真相源原件」上改内容（不碰 raw） + +单向同步，方向写死：**改 Obsidian（或其它外部）原件 → 覆盖 raw 副本 → convert**。**永远不要反向把 KB 锚点写回原件**。 + +``` +# 用 Edit 改真相源原件（它在 raw/** deny glob 之外，Edit 允许）： +# $SRC_ROOT/.md +``` + +- 内容编辑一律用 `Edit` 改这个**外部原件**。它不在任何 `raw/**` deny 内，可以改。 +- 三种编辑心里有数（决定第 5 步怎么修引用）：**MODIFY**（改某块正文/数字）/ **ADD**（插入新段/新章节）/ **DELETE**（删段/删章节）。ADD、DELETE 会让**下游所有块的 seq 位移**（见第 5 步）。 + +--- + +## 第 3 步：把更新后的原件镜像进 raw（cp，不是 Edit） + +raw 内容的唯一合法来源是「被 `cp` 进来的真相源原件」。agent **绝不**用 `Edit`/`Write`/`>` 重定向手改 raw；`cp` 是机械镜像（等同 ingest 前置「把原始文件放入 raw」）。 + +```bash +# SRC_ROOT = 真相源原件所在目录（在 raw/** deny 之外的可编辑位置，如你的 Obsidian +# vault 的笔记文件夹）。KB_ROOT = 数据根（含 workspaces/）。二者按你的实际布局设。 +SRC_ROOT="<你的真相源原件目录>" # 例：某个 Obsidian vault 的笔记文件夹 +# 干净原件（无 KB 锚点）整文件覆盖 raw 副本 +cp "$SRC_ROOT/.md" \ + "$KB_ROOT/workspaces//raw/articles/.md" + +# 一致性硬校验：两份必须逐字节相同（不同说明没覆盖成功 / 覆盖错文件） +diff "$SRC_ROOT/.md" \ + "$KB_ROOT/workspaces//raw/articles/.md" && echo "SYNCED" + +# 防「原件里混入了 KB 锚点」导致 convert 静默跳过：grep 到就得处理（见第 4 步硬规则） +grep -nE '\^[hpctf]-[0-9]' "$KB_ROOT/workspaces//raw/articles/.md" && echo "⚠️ 原件含 KB 锚点，需 --force 或先剥锚" || echo "无残留锚点，OK" +``` + +> `cp`/`diff`/`grep` 不在 deny 列表内（deny 的是 `Edit`/`Write`/`rm`/`mv` 对 raw）。若不想让 agent 碰 raw 副本，也可让**人**做这一步 cp，agent 从第 4 步接手——二选一，别让 agent 用 Edit/Write 写 raw。 + +--- + +## 第 4 步：重转换——只重锚被改的那个文件 + +刚 `cp` 进来的原件没有 KB 锚点，所以 `convert.py --workspace` 会重新处理它。对其它文件，`should_convert()` 现在校验 outline schema、全文 SHA-256、章节 SHA-256、canonical anchors 与结构；只有内容地址化契约完整且新鲜才跳过，mtime 不能掩盖同长度改写或坏 outline。 + +```bash +python scripts/convert.py --workspace +``` + +> **硬规则**：仍推荐用无锚干净原件覆盖再 convert；未改块会确定性复现原 anchor，真改块才换锚。不要直接手改已锚 raw（权限上也禁止），也不要裸跑全量 `--force`；新 validator 会拒绝坏/旧 outline，但它不是授权手改 raw 的理由。 + +重转换后必须重建派生证据索引；raw SHA 已变化时旧索引会 fail-closed： + +```bash +python scripts/k.py --workspace rebuild-evidence-index +python scripts/k.py --workspace evidence-index-coverage +``` + +--- + +## 第 5 步：查失效引用 + 三诊断逐条修 + +```bash +python scripts/k.py --workspace list-broken-refs +``` + +它会列出所有指向该 raw、锚点已对不上的 wiki 引用。**注意 raw 目标无 hash6 容错**——任何使块 seq 位移的增删（ADD/DELETE）都会让**下游每一条 raw 引用**被报失效，哪怕内容一字未改。别慌，用**三诊断**逐条判：对每条 broken raw ref，拿它的**旧锚**去 `read-block`： + +```bash +python scripts/k.py --workspace read-block raw/articles/.md ^p-3-7cb619 +``` + +| 诊断 | read-block 结果 | 含义 | 处置 | +|---|---|---|---| +| **(a) 纯 seq 位移** | 成功回收（带 `recovered_from`），preview 与 wiki 论断**一致** | 块只是被上游增删挤动了位置，内容没变（hash6 不变） | **只机械改锚串里的 seq**（保留 hash6），无需重新语义回验 | +| **(b) MODIFY** | 成功回收，但 preview 内容**已变** | 被引块正文真的改了（hash6 变） | 打开新原文，**按原文改写 wiki 论断**（数字按原文改；结论若反转 → 挂 `[!WARNING]` 冲突标注、不覆盖），改锚到新块，重新引用回验 | +| **(c) DELETE** | **回收失败**（hash6 已不存在） | 被引块被整段删了 | 该论断在新原文别处仍有支撑 → 改引新锚；无支撑 → 挂 `[需要来源]` + 页面加 `#to-be-updated`；来源撤稿致论断失真 → 走冲突/降级。**绝不静默删引用** | + +**确定性批量重映射**（应对 seq 级联）：改一处正文没事，但**增删一个块会让下游一片引用失效**。用 `blocks` 建 hash6→当前锚 的映射，机械改 seq： + +```bash +python scripts/k.py --workspace blocks raw/articles/.md --json +# 对每条 broken raw ref：解析其 hash6（⚠️ 锚点可能带碰撞后缀 ^p-2-7cb619-2，别假设结尾就是 6 位 hex） +# → 在 blocks 输出里按 hash6 找当前锚 → 保留 hash6、只把 wiki 引用里的 seq 改成新值 +``` + +用 `find-anchor` 按**新内容**反查新锚（用于 MODIFY 的新块定位）： + +```bash +python scripts/k.py --workspace find-anchor raw/articles/.md "<改后的原文片段>" +``` + +**扇出**：backlinks 清单里的**每一处**都要修；同一个被改数字若散落多个 wiki 页，**所有出现点**都按新原文一致更新。修到 `list-broken-refs` **归零**才算完。 + +--- + +## 第 6 步：连带更新（③ 档登记表 / 摘要 / frontmatter） + +- **③ 档「章节深度登记」表**（无 lint 兜底）：若被编辑的 source_summary 含此表，且编辑增删/改名了 H 标题，表内 `^h-...` 锚点与原标题会集体陈旧、`search-raw` 的 deepen 联动失明。用 `outline` / `blocks` 拉新 H 锚全表，逐行重写 **Anchor 列**与**原标题列**（原标题改了要同步、不许意译）。 +- **agent_summary 摘要**：对内容实变的章节，用 `annotate-section` 重写其摘要（否则摘要层描述旧内容而无告警）： + ```bash + python scripts/k.py --workspace annotate-section raw/articles/.md h-2-3-abc123 "本节现在论证..." + ``` +- **frontmatter**：更新受影响 wiki 页的 `last_modified`（今天）+ `last_modified_by: LLM`；若编辑删掉了某 concept 论断的唯一来源支撑，按需调 `source_count` / `sources` 或补 stub 标记。 +- **ADD 了全新实质章节**：这不是「修引用」，是「新材料进来了」——对新段走 kb-ingest 式**深读 + annotate-section**，按需**新建 / 更新** wiki 论断（区别于 partial re-ingest 的 ⊙→✓ 升级）。 + +--- + +## 第 7 步：提交前质量闸门（用 `--paths` 显式驱动，别用 `--changed`） + +> **为什么不用 `--changed`**：KB_ROOT 外置 + 中文/空格路径 + 独立数据仓时，`--changed` 依赖 `git -C /workspaces/` 判定改动文件；一旦 wiki 未跟踪 / 被 gitignore，改动集为空 → 枚举 **0 对**却报「通过」——**核心闸门假绿**。改用 `--paths <本次改过的 wiki 页...>`，并断言**枚举对数 > 0**（你改了 N 页，就该有 ≥N 对）。 + +```bash +P="wiki/sources/a.md wiki/concepts/b.md ..." # 本次改过的所有 wiki 页 + +python scripts/k.py --workspace list-broken-refs # 必须归零 +python scripts/k.py --workspace list-cite-mismatches # 闸门项（mismatch/exempt-missing-basis）须为空 +python scripts/k.py --workspace list-bare-claims # 空 +python scripts/k.py --workspace list-coarse-citations # 空 +python scripts/k.py --workspace list-source-issues # 空 +# 引用回验：显式 --paths + 断言 pairs>0，再对新增/改写的（论断,引用）对做 fresh-context 判定 +python scripts/k.py --workspace extract-claims --paths $P --with-evidence --json +``` + +验收：`list-broken-refs` = 0；`list-cite-mismatches` 闸门项 = 0；bare/coarse/source-issues 全空；`extract-claims` 返回的对数 > 0（不是 0！）且新增/改写对的语义回验 UNSUPPORTED/CONTRADICTED = 0（判定入 `cite-audit-log --mode ingest`）。**wiki 页无手写块锚点导致 extract-claims 枚举为 0 时**：先 `convert.py --dir workspaces//wiki` 给 wiki 加锚点、再删生成的 `*.outline.json`（wiki 不留 outline.json），然后重跑。 + +--- + +## 第 8 步：原子提交（打到数据仓，raw 是否入库看 check-ignore） + +```bash +# git 必须用 -C 打到「数据仓」（KB_ROOT），不是引擎 cwd +git -C "$KB_ROOT" add workspaces//wiki workspaces//log.md + +# raw 是否入库：由该仓 .gitignore 决定，别猜——用 check-ignore 验 +git -C "$KB_ROOT" check-ignore workspaces//raw/articles/.md \ + && echo "raw 被 ignore：commit 只含 wiki+log（原文可复现性依赖外部原件仓）" \ + || git -C "$KB_ROOT" add "workspaces//raw/articles/.md" # 未 ignore（本库 raw 入库）→ 一并提交 + +git -C "$KB_ROOT" -c user.name="" -c user.email="" \ + commit -m "edit-source: <来源标题> 内容更新，同步 wiki 引用" +``` + +- **raw 被 gitignore 时**（发布版默认）：commit 只含 `wiki/** + log.md`；被引原文的可复现性**依赖外部真相源仓**（如 Obsidian vault 自身受版本控制）——log.md 里记下对应原件路径（+ 若原件仓有版本，记其 commit sha），把两仓这次变更人工挂钩。诚实边界：**raw 内容变更无法从 KB 仓回滚**。 +- **raw 未 ignore 时**（如自建私人库把 raw 纳入版控）：raw 副本一并提交，KB 仓自包含、可回滚。 +- 绝不 `git add -A` / `git add .`。 + +--- + +## 第 9 步：追加 log.md + +`Edit` `log.md`，文件**头部**插入： + +```markdown +## [YYYY-MM-DD] update | <来源标题> 内容更新 +- 来源：`raw/articles/.md`（真相源原件：``） +- 编辑类型：MODIFY / ADD / DELETE（一句话说改了什么） +- 修引用：<改了哪几页的哪几处 anchor>；纯 seq 位移 N 处、实质改写 M 处、DELETE 处置 K 处 +- 连带：<③ 档登记表重对齐 / annotate-section 刷摘要 / frontmatter> +- 闸门：list-broken-refs 归零；cite-mismatch 闸门项 0；extract-claims pairs= +- 溯源挂钩：外部原件仓 sha=<...>（若 raw 未入 KB 仓） +``` + +--- + +## 基线漂移自检（首次在某库用本 skill 前跑一次） + +信「干净覆盖复现锚点」之前，先证它：拿一个**未编辑**的原件，`cp` 覆盖 raw 副本 → `convert` → `list-broken-refs`。**必须零新增失效引用**。若非零，说明外部原件与 KB 里当初 ingest 的版本已漂移（例如原件被塞了 Obsidian 专属语法：原生 `^blockid`、`%%注释%%`、`![[嵌入]]`、dataview 块），需先归一化/剥掉这些再走本流程。 + +--- + +## 完成检查清单 + +- [ ] 第 0 步判明来源类型（原生 md / 二进制派生 / human-only-locked），走对分支 +- [ ] 编辑前 `backlinks` 落全部受影响 wiki 页清单；高危块 `read-block` 留旧原文快照 +- [ ] 内容只在**外部真相源原件**上用 `Edit` 改；**没有对任何 `raw/**` 文件 Edit/Write** +- [ ] `cp` 干净原件覆盖 raw 副本；`diff` 校验逐字节一致；`grep` 确认无残留 KB 锚点 +- [ ] `convert.py --workspace`（覆盖后无锚，免 --force）；未裸跑全量 `--force` +- [ ] `rebuild-evidence-index` 后 natural/content/structural coverage=100%、freshness=true；异常空章节已确认 +- [ ] `list-broken-refs` 逐条三诊断修完，**归零**；扇出全部引用页都修 +- [ ] MODIFY 的数字/结论按新原文改写（结论反转挂冲突标注，不覆盖）；DELETE 未静默删引用 +- [ ] ③ 档「章节深度登记」表 Anchor/原标题重对齐；内容实变章节 `annotate-section` 刷摘要 +- [ ] 受影响 wiki 页 frontmatter（last_modified + last_modified_by + 按需 source_count/sources） +- [ ] 闸门用 `--paths` 显式跑、断言 `extract-claims` pairs>0：broken-refs / cite-mismatch 闸门项 / bare / coarse / source-issues 全过 +- [ ] `git -C "$KB_ROOT"` 提交（不是引擎 cwd）；raw 按 `check-ignore` 决定是否入库；无 `git add -A` +- [ ] log.md 追加 `update` 条目 + 溯源挂钩 + +## 错误恢复 + +- 中途出错**不要 partial commit**。`git -C "$KB_ROOT" status` 查看修改。 +- 撤回单个 wiki 文件：`git -C "$KB_ROOT" checkout `。 +- raw 副本改错了：从外部真相源原件重新 `cp` 覆盖再 convert（原件是源，raw 只是镜像）。 +- 整体撤回未 commit 的 wiki 改动：与用户确认后 `git -C "$KB_ROOT" stash`。 + +## 反例（绝对不要做） + +- ❌ 用 `Edit` / `Write` / `>` 重定向手改 `raw/**` 下任何文件（含原生 md）——被 deny，且违反「raw 只读、唯一写入者是 convert.py」。内容改在**外部真相源原件**上。 +- ❌ 把 KB 锚点（`^p-...`）反向写回 Obsidian 原件——污染真相源，下次覆盖会级联乱套。同步严格单向。 +- ❌ 把 mtime/“文件里已有锚点”当作 outline 新鲜度证明——现行 convert 必须通过 schema + 全文/章节 hash + 结构 validator；推荐仍是无锚干净原件覆盖 +- ❌ 裸跑 `convert --dir raw/... --force` 全量重锚——顺带重锚别的陈旧文件，冒出无关 broken-refs、信号不可归因。 +- ❌ 用 `--changed` 驱动 `extract-claims` / `check-provenance` 闸门——KB_ROOT 外置/未跟踪时会枚举 0 对却报通过（假绿）。用 `--paths` + 断言 pairs>0。 +- ❌ `git` 在引擎 cwd 里跑 commit——会打到引擎仓、漏掉数据仓的 wiki 改动。必须 `git -C "$KB_ROOT"`。 +- ❌ 对 DELETE 掉的被引块「删引用、挂 [需要来源]」当无事发生——要么改引新锚、要么显式 `[需要来源]` + `#to-be-updated` + 留旧原文快照。 +- ❌ 只修 `list-broken-refs` 报的那几条，忘了同一被改数字在别的 wiki 页的其它出现点（扇出漏改）。 +- ❌ 把「来源没变、扫读升深读」的 partial re-ingest 和「来源内容真变了」的本 skill 混为一谈。 diff --git a/.agents/skills/kb-export/SKILL.md b/.agents/skills/kb-export/SKILL.md index 7134341..ed7ce28 100644 --- a/.agents/skills/kb-export/SKILL.md +++ b/.agents/skills/kb-export/SKILL.md @@ -83,7 +83,7 @@ type: source_summary created_date: 2026-04-28 last_modified: 2026-04-28 last_modified_by: LLM -status: reviewed +status: draft # LLM 写入一律 draft；reviewed 仅人类审阅后设（并把 last_modified_by 改 Human） confidence: high source_count: 1 sources: diff --git a/.agents/skills/kb-ingest/SKILL.md b/.agents/skills/kb-ingest/SKILL.md index d1c8a8e..8124448 100644 --- a/.agents/skills/kb-ingest/SKILL.md +++ b/.agents/skills/kb-ingest/SKILL.md @@ -40,6 +40,27 @@ python scripts/convert.py --workspace 如果只想处理某 workspace 的某个子目录，用 `--dir` 显式指定（给出时覆盖 `--workspace`）：`python scripts/convert.py --dir workspaces//raw/papers`。 +### 第 1.5 步：强制建立“全细节证据地图”（不得抽样） + +转换后立即从**全部** `raw/**/*.md` 重建派生证据索引，并核对分母： + +```bash +python scripts/k.py --workspace rebuild-evidence-index +python scripts/k.py --workspace evidence-index-coverage +``` + +两个机械闸门都必须满足： + +- `natural_units.coverage == 1.0`：paragraph、每条 list item、每条 table data row、blockquote、code、figure 全部进入索引； +- `content_sections.coverage == 1.0` 且 `structural_sections.coverage == 1.0`；`unexpected_empty_sections` 逐项确认是故意占位，否则视为转换丢正文并修复； +- `manifest.ok == true`：构建时冻结的完整单位/章节 inventory 指纹、每文档声明分母、物化表与 FTS 对账；保留结构空白的 exact text hash 也必须一致，代码缩进等变化不得被空白折叠掩盖；两张 FTS 的规范 DDL/列序/tokenizer 必须匹配，内部 `quick_check` 也必须为 `ok`，不能只看 shadow content 行； +- `corpus_freshness.ok == true`：当前 raw 文件集合/全文 SHA-256，以及真正参与路由的已验证 `agent_summary` 指纹都与索引一致；`annotate-section` 后必须重建； +- 任一分母为 0、raw 新增/删除/同长度改写、FTS 通道缺行或解析错误都不是“100%”，必须重建或修复后再继续。 + +这是**可发现性层**，与 AI 阅读深度分开：第 ③ 档可以只深读少量章节，但未深读章节的原文自然单元也必须 100% 可检索。机械索引不等于语义理解，不能把 coverage=100% 写成“摘要没有遗漏”或“问答一定正确”。 + +**分母边界**：这里的 100% 是“当前转换后 markdown 的 parser inventory 全部物化”，不是“原 PDF/DOCX 每页、每表、每脚注都转换成功”。高风险来源还要做格式专用 conversion receipt/页表计数/视觉抽检；在该层未认证前，对外只能声明“转换文本内无机械漏索引”。 + ## 第 2 步：看大纲，AI 自动决定阅读策略 ```bash @@ -53,8 +74,8 @@ python scripts/k.py outline raw/papers/.md | 档 | 字数（中文等价） | 策略 | 综合保真度 | |---|---|---|---| | ① 短文 | < 30K | 一次 Read 全文 | 高 | -| ② 中长文 | 30K – 150K（论文 / 报告 / 长文） | 按 H1 切块、每块 ≤ 30K，分段 Read，每段读完调 annotate-section | 高（多步但不漏信息） | -| ③ 整本书规模 | > 150K（专著 / 法规全文 / 长篇手册） | TOC 扫全 + AI 决定深读章节；**全部章节**登记到 source_summary 章节登记表 | 中（透明声明深度差异，保留 partial re-ingest 升级路径） | +| ② 中长文 | 30K – 150K（论文 / 报告 / 长文） | 按标题树选 ≤ 30K 的完整节 Read，超长 H1/H2 继续下钻 H3/H4，每节读完调 annotate-section | 综合层有损；全部原文细节另由机械索引保留 | +| ③ 整本书规模 | > 150K（专著 / 法规全文 / 长篇手册） | TOC 扫全 + AI 决定深读章节；**全部章节**登记到 source_summary 章节登记表 | 综合层显式分级；全部原文细节仍须 100% 可检索 | > 英文文档按 `字符数 × 0.5` 估算中文等价（英文 1 token ≈ 4 char，中文 1 token ≈ 1.5-2 char）。 > 单次 Read 严格控制在 30K 中文字符内——避免 LLM "lost in the middle" 衰减。 @@ -68,7 +89,7 @@ python scripts/k.py outline raw/papers/.md ### 第 ② 档：中长文分段读 -按 H1 章节顺序切块（必要时合并相邻短章节凑近 30K），每块独立 Read： +按标题树顺序选择不超过 30K 的完整节；H1/H2 本身超限时继续下钻 H3/H4，无子标题的超长节则用 `search-evidence → read-evidence-unit → read-block` 按自然块阅读。若**单个**自然块仍超 30K，搜索只返围绕命中词的有界摘录，精确读取 fail-closed；优先修复来源结构/转换分块，`--max-chars 0` 只能作为人工明示的无限制逃生口，不得由 agent 自动绕过： ```bash python scripts/k.py read-section raw/papers/.md ``` @@ -76,7 +97,7 @@ python scripts/k.py read-section raw/papers/.md ```bash python scripts/k.py annotate-section raw/papers/.md h-2-3-abc123 "本节论证..." ``` -最终综合判断（第 3 步）基于**全部章节摘要**，不丢信息。 +最终综合判断（第 3 步）基于已读章节和章节摘要；摘要是有损的，不得声称其“不丢信息”。问答需要摘要未收录的细节时，必须回到全量证据索引发现并核对原文。 ### 第 ③ 档：长篇文档结构化深度选读 @@ -88,13 +109,19 @@ python scripts/k.py annotate-section raw/papers/.md h-2-3-abc123 "本节 3. 对深读章节走第 ② 档流程（read-section + annotate-section） 4. **关键**：source_summary 的「## 章节深度登记」H2 节按 anchor 列出**全部章节**（详见第 5 步模板），扫读 / 跳过的章节**保留 partial re-ingest 升级路径**（详见后文「增量深化」节） -### 精确取段（任何档都可用） +**摘要路由新鲜度回收**：`annotate-section` 会改变参与结构路由的 validated summary 指纹，因此第 1.5 步建的索引会按设计变 stale。完成本次全部 annotation 后，必须**再跑一次** `rebuild-evidence-index` + `evidence-index-coverage`，确认 `outline_summary_changed=[]` 且全部闸门仍绿；不得带旧摘要路由进入后续问答。 + +### 精确取段与 quote-first 写作纪律（任何档都适用） -如果分析中发现需要精确取出某段（比如某个关键数据），调： +取出某段精确原文： ```bash python scripts/k.py read-block raw/papers/.md p-12-7d8e9a ``` +**quote-first 硬规则**：写任何含**数字 / 日期 / 金额 / 百分比 / 精确引文**的论断之前，必须先用 `read-block`（^p-/^t-）或 `read-section`（^h-）打开目标块，**从屏幕上的返回原文抄写**，anchor 也从返回内容行尾复制——**禁止凭「刚才通读时的记忆」写，禁止从 source_summary 二手转抄而不核对**。记忆漂移正是数字抄错与张冠李戴的根源；`list-cite-mismatches` 会把违规兜出来。 + +零成本主路径：②③ 档分段阅读时**「读完一节 → 立即写该节相关论断」**——此刻原文就在上下文里，与 annotate-section 回填并列为「读完即写」双动作。只有离上下文写作（第 5/6 步补数据、更新概念页）才需要回头 read-block。 + ## 第 3 步：基于 wiki 现状做综合判断 读完原文后，**不询问用户**——AI 自行综合"本文相对已有知识库提供了什么"。这一步是后续写作（5-7 步）的信息基础，**不能跳过**。 @@ -212,7 +239,7 @@ tags: |---|---|---|---| | ^h-2-1-... | 摘要 | ✓ 深读 | 已含完整 anchor 引用 | | ^h-2-2-... | 引言 | ✓ 深读 | | -| ^h-2-3-... | 方法 | ⊙ 扫读 | 仅基于 outline preview 概览，本次综合不深入 | +| ^h-2-3-... | 3 Method | ⊙ 扫读 | 仅 preview 概览；关键实体:对比学习 / hard negative / in-batch 采样（从 preview 提取） | | ^h-2-4-... | 实验 | ✓ 深读 | 含数据表 ^t-... | | ^h-2-5-... | 附录 A | × 跳过 | 元信息（参考文献清单） | @@ -220,13 +247,18 @@ tags: - ✓ **深读**：完整 read 该章节，提取了 anchor 级引用，可直接被 wiki 论断引用 - ⊙ **扫读**：仅基于 outline preview / 章节标题做概览判断，未读全文；**保留升级路径**——后续可触发 partial re-ingest 升级到深读 - × **跳过**：与 wiki 主题无关或为元信息（附录 / 致谢 / 索引），不计入价值评估，但仍登记可见，避免"消失" + +**登记表书写硬规范**（`search-raw` 的 deepen_hint 联动与 partial re-ingest 触发都依赖它）： +- **首列必须写真实 `^h-` anchor**（从 `outline` 输出复制），**原标题列必须抄原文标题**、不许意译——意译名（如把 "2 Approach" 写成 "KAG Framework"）会让内容级检索命中后无法联动回登记表，deepen 触发器直接失明 +- **⊙ 扫读行的备注必须点名本节关键实体 / 指标名**（从 outline preview 免费提取，如"关键实体:LLMFriSPG / Mutual Indexing / Logical Form Solver"）——这是扫读章节留给未来检索与人工浏览的唯一索引密度，一行备注换一整章的可发现性 ``` **引用规范**： - 优先用 anchor 形式（`#^h-...` / `#^p-...`）而非 heading 文本 - 整章/整节论证 → `^h-{level}-{seq}-{hash}` -- 关键数据/精确论断 → `^p-{seq}-{hash}` -- 不知道 anchor 时调 `python scripts/k.py find-anchor raw/papers/.md "<原文片段>"` 反查 +- 关键数据/精确论断 → `^p-{seq}-{hash}`（数字论断**必须**锚到 ^p-/^t- 级，不要用 ^h- 大节当支撑） +- 不知道 anchor 时调 `python scripts/k.py find-anchor raw/papers/.md "<原文片段>"` 反查——**返回的 preview 必须与论断核对一致才可采用**；不一致就换 snippet 重查或 read-block 确认，不要拿相邻段落的 anchor 凑数 +- 数字为跨块计算 / 单位换算所得、原文无该字面时，标 `[KB 推算: ^依据锚]`（必须带依据锚，裸 `[KB 推算]` 会被 lint 报 exempt-missing-basis） 校验： @@ -356,6 +388,50 @@ python scripts/k.py --workspace graph - `graph` 能正常输出节点 / 边统计，本次新建 / 更新的页面**出现在节点里且有边相连**；出现意外**孤立节点**说明漏了互链——回第 6 步给它补 `[[...]]`（相关概念 / 实体用标准关系类型 `SUPPORTS` / `EXTENDS` / `PART_OF` 等）。 - web 端 `/graph` 可直接渲染本图谱（节点按 type 染色、边按 link_type 染色），无需额外构建步骤。 +## 第 9.5 步：引用回验（writer/verifier 分离） + +写作者不能自证——刚写完的引用必须经**确定性核对 + 没有写作上下文的核验者**双闸复核，才允许 commit。 + +### 9.5a 确定性核对（零 token） + +```bash +python scripts/k.py --workspace list-cite-mismatches +``` + +验收（accuracy-first）：`mismatch` / `exempt-missing-basis` / `canonical-anchor-mismatch` / `canonical-target-mismatch` / `imprecise-anchor` / `unverifiable` **全部必须为空**。Ingest 是 raw 在场的权威写入环境，「无法核验」不能在这里当信息项放过；hash recovery、`./raw/...` 等非 canonical 写法必须先改成返回的精确 path/anchor。`exempted` 只允许于已标 `[KB 推算: ^依据锚]` 的紧邻**单个值**，依据锚还必须在同一核对单元实际被引。 + +```bash +python scripts/k.py --workspace check-provenance --changed +``` + +验收：**须全空**——每条新引用必须有「取回过被引块当前内容版本」的检索凭证（`read-block` / `read-section` / `blocks` / `extract-claims --with-evidence` 都会自动登记；quote-first 走对了凭证自然齐）。缺凭证 = writer 没真正打开原文（或读的是旧版本），回去 read-block 取回再引用。 + +### 9.5b 语义回验（fresh-context 子 agent） + +1. 枚举本次新增 / 修改的（论断, 引用）对： + ```bash + python scripts/k.py --workspace extract-claims --changed --json + ``` + `summary.broken > 0` 先修引用再回验。核验时必须用 `--with-evidence --max-evidence-chars 1000000000`（或等价的全量取回），claim / evidence / cloze 任一截断都是 incomplete，不得对未见内容写 `SUPPORTED`。事实型 NOTE/TIP/IMPORTANT callout 和 table data row 同样进入枚举；只有知识冲突 / 引用审计的协议外壳排除。 +2. 用 **Task 工具起 fresh-context 子 agent**（**绝不共享写作上下文**——writer 的记忆偏差正是要防的东西；也不得由 writer 自己兼任），每批 ≤ 20 对。子 agent prompt 固定为： + - 身份：「你是**对抗性**引用核验员，立场是尽力反驳，没读过原文全文、不了解写作过程。只依据给你的论断文本与被引块原文判断，禁止用自身领域知识补全证据。」（反驳式立场比中性判定显著降低附和偏差） + - 材料：每对给 `claim_text` + `read-block` / `read-section` 取回的被引块原文（可用 `extract-claims --with-evidence` 组装）。 + - 判定三问：① 关键事实（数字 / 日期 / 主体）是否出现在被引原文？② 论断是否被原文**直接支撑**（不需要额外推理）？③ 有无语义 drift（过度概括 / 加了原文没有的限定词 / 因果错置）？ + - 输出：每对 `{pair_id, verdict: SUPPORTED|PARTIAL|UNSUPPORTED|CONTRADICTED, reason 一句, fix_suggestion}`。 +3. 处置表： + +| verdict | 处置 | +|---|---| +| SUPPORTED | 通过；`cite-audit-log --mode ingest` 入账（附 `--evidence "<被引块原文子串>"`，k.py 会字面校验） | +| PARTIAL | 收窄措辞 / 补 anchor 后复验。**合法转述白名单**（判 SUPPORTED 并在 note 记明）：约数舍入（约 40% ↔ 39.2%）、跨语言日期 / 数字格式转写、已标 `[KB 推算: ^锚]` 的派生算术 | +| UNSUPPORTED | `read-block` 打开原文**按原文改写**，或 `find-anchor` 换正确块，或删掉该论断——**禁止删引用降级 `[需要来源]` 了事**（那是把错误论断洗进 wiki 的通道） | +| CONTRADICTED | 论断与自己引的证据相反 → 按原文改写；若实为「新证据 vs 既有 wiki 论断」的知识冲突 → 按 CLAUDE.md 冲突标注格式写 `[!WARNING]` 块，不覆盖 | + +4. 复验：修完重跑 `extract-claims --changed`，只对上轮非 SUPPORTED 的对再起一次小 Task。**最多 2 轮**——仍不收敛的对保留 PARTIAL、打 `#to-be-updated`、log 记账、交周检跟进；不许反复重掷骰子刷 verdict。 +5. **盲填复核（含数字的对必做）**：`extract-claims --changed --cloze` 输出挖空论断（数字→⟦N1⟧）；给核验 agent **只看「挖空论断 + 被引原文」**填空（绝不给期望值——从原理上消灭附和偏差），填回值经 `python scripts/k.py cloze-check --batch ` 机器判分（数值容差、块级 union 语义）。判分未过按 UNSUPPORTED 处置。 +6. **跨模型二审（发布质量 ingest 必做）**：`python tools/cite-audit/audit.py --workspace --changed`——外部客户端调 DeepSeek 跑「盲填 + 反驳」双通道并自动入台账（独立执行器，不依赖会话内 agent；需 `DEEPSEEK_API_KEY` 环境变量）。换模型降低同源相关盲区。exit `0` 才是全部通过，`1` = 真实语义未通过，`2` = 网络 / 截断 / quote / skipped / 台账等导致核验未完成；后两者都不得 commit。无 API 时必须换一个真正 fresh-context 核验者完成同等双通道，不得由 writer 自审替代。 +7. 验收（与提交前质量闸门同构、可对账）：本次 verifiable 对 **UNSUPPORTED = 0、CONTRADICTED = 0**，全部判定已入台账，log.md 条目含 citation-verify 对账行（见第 10 步）。事后任何人可用 `extract-claims --commit ` 复枚举对账。另有 pre-commit hook 机械兜底：staged wiki 文件带 cite 闸门项时提交直接被拒。 + ## 第 10 步：追加 log.md `Edit` `log.md`，在文件**头部**（最近的 `---` 后）插入： @@ -367,6 +443,7 @@ python scripts/k.py --workspace graph - 更新：`wiki/concepts/.md`、`wiki/concepts/.md` - 标记待更新：<5-10 个文件> - MOC：`wiki/indexes/_index.md` +- citation-verify: pairs= verifiable= SUPPORTED= rewritten= downgraded= unverifiable= - 摘要：<一两句核心收获> ``` @@ -428,11 +505,15 @@ git commit -m "ingest: <来源标题简短>" ## 完成检查清单 - [ ] `convert.py` 已对原始文件生成 `.md` + `.outline.json` +- [ ] **最后一次 `annotate-section` 之后**已重跑 `rebuild-evidence-index`；`evidence-index-coverage` 的自然单元、内容/结构章节覆盖率均为 100%，异常空章节已逐项确认/修复，且 `corpus_freshness.ok=true`、`outline_summary_changed=[]` - [ ] 中长文档（≥30K 字符，即第 ②/③ 档）通过 `outline` → `read-section` 路线分段读取，不是 Read 全文 - [ ] 关键章节已 `annotate-section` 回填精排摘要（②③ 档必经；① 档建议性、非硬性） - [ ] 摘要页 frontmatter 完整且 `validate-frontmatter` 通过 - [ ] 所有实质性论断都有 `[[raw/...#^h-...]]` 或 `[[raw/...#^p-...]]` **块级** anchor 引用，没有"裸论断"、没有"整页引用 `[[raw/X]]` 支撑论断"、没有用 heading 文本作引用——`python scripts/k.py list-bare-claims` / `list-coarse-citations` / `list-source-issues` 三者均须为空 - [ ] `python scripts/k.py list-broken-refs` 没有新增失效引用 +- [ ] `python scripts/k.py list-cite-mismatches` 的 mismatch / exempt-missing-basis / canonical-* / imprecise / unverifiable 全空；推算豁免只绑定紧邻单值 +- [ ] `python scripts/k.py check-provenance --changed` 全空（每条新引用都有当前版本的检索凭证） +- [ ] 第 9.5 步引用回验完成：全量核验包无截断 / incomplete / skipped，`extract-claims --changed` 枚举的 verifiable 对 UNSUPPORTED=0 / CONTRADICTED=0，判定已 `cite-audit-log --mode ingest` 入账，log.md 含 citation-verify 对账行 - [ ] 核心节点（2-3 个）已立即更新 - [ ] 次要节点已标记 `#to-be-updated` - [ ] MOC 索引已更新 @@ -462,7 +543,12 @@ git commit -m "ingest: <来源标题简短>" - ❌ 用 `git add -A` 或 `git add .`（可能误提交无关文件） - ❌ 跳过第 3 步直接进 5-7 步（缺了 wiki 上下文，写出来的摘要页"不知道周围有什么"，会重复造轮子或漏掉冲突） - ❌ 第 3 / 8 步去问用户「这次核心价值是什么」「该建哪个 MOC」（这两步明确改为 agent 自决；用户的修订路径是 web 端审计与冲突工作台，不是 ingest 时实时打断） -- ❌ 单次 Read 超过 30K 中文字符（"lost in the middle" 衰减；中长文必须按 H1 切块到 ≤30K 再分段读） +- ❌ 单次 Read 超过 30K 中文字符（"lost in the middle" 衰减；中长文必须沿标题树下钻到 ≤30K 的完整节，或按自然块读） +- ❌ 只给深读章节建索引、或拿“章节摘要覆盖率”冒充原文细节覆盖率——机械证据地图必须先覆盖全部 raw 自然单元；AI 摘要只是附加路由信号 - ❌ 第 ③ 档长文档把扫读章节当成「已读」用——query 时若命中扫读章节关键词，应**先看 source_summary 章节登记表确认深度**，必要时触发 partial re-ingest，不能直接拿 outline preview 当真知识 - ❌ 把扫读 / 跳过的章节从 source_summary 章节登记表中省略——所有章节必须可见，省略 = 失去升级路径 - ❌ ingest 后忘记跑 `k.py list-broken-refs` 检查新引用是否解析成功 +- ❌ 凭「刚才通读的记忆」写数字 / 日期 / 引文论断而不打开被引块核对——quote-first：写前必须 `read-block` / `read-section` 从返回原文抄写；`list-cite-mismatches` 会报 mismatch +- ❌ 第 9.5 步让 verifier 继承写作上下文、或 writer 自己兼任 verifier——核验者必须 fresh context，否则回验只是自我合理化 +- ❌ 对 UNSUPPORTED 的论断「删引用、挂 [需要来源]」过闸——那是把错误论断洗进 wiki；只能按原文改写、换正确锚点或删掉论断本身 +- ❌ 为过数字核对滥标 `[KB 推算]`（抄错说成推算）——豁免必须带依据锚，且 kb-lint 周检会抽查豁免块、豁免占比异常会被追问 diff --git a/.agents/skills/kb-lint/SKILL.md b/.agents/skills/kb-lint/SKILL.md index 1fd616d..e930cd2 100644 --- a/.agents/skills/kb-lint/SKILL.md +++ b/.agents/skills/kb-lint/SKILL.md @@ -38,9 +38,12 @@ python scripts/k.py health --json - 被引用但缺章节摘要（`unsummarized_sections_count`，被 wiki 章节引用但 outline.json 中 `agent_summary` 为 null 的章节） - 裸论断（`bare_claims_count`，含数字但无引用支撑的段落） - 索引 page_count drift（`index_count_mismatches_count`，type=index 页声明的 page_count 与 scope 实际匹配数不等） -- **source_count 一致性问题（`source_issues_count`，六类：count-mismatch / missing-source / analysis-undersourced / source-summary-mismatch / broken-source-link / declared-but-uncited — 详见 AGENTS.md "source_count 字段约定" 与本文第 6d 步）** -- **关系类型问题（`relation_issues_count`，`[[X|RELATION]]` 中非标准关系类型词——拼写错误 / 未在白名单，详见 AGENTS.md "关系类型语法"）** -- **Web i18n 违规（`i18n_violations_count`，web/ 下硬编码中文 UI 字符串，违反 AGENTS.md "Web 管理台国际化方案"）** +- **引用核对（`cite_mismatches_count`：数字 / 引文与被引块不符 + [KB 推算] 无依据锚——闸门项；`cite_imprecise_count`：锚点挂偏 / 引文未逐字命中——观察项；`cite_unverifiable_count`：被引 raw 不在场——信息项；`cite_exempted_count`：[KB 推算] 豁免块数——环比异常增长要抽查）** +- **引用语义审计（`suspect_citations_count`：待人处理的 CAUTION 审计标注，扫自 markdown；`citation_pairs_count`：审计对总数；`unaudited_citations_count`：未审对数——全指标中唯一依赖 .cache 台账的，新 clone / 删缓存后回升到 verifiable 总数属预期）** +- **source_count 一致性问题（`source_issues_count`，六类：count-mismatch / missing-source / analysis-undersourced / source-summary-mismatch / broken-source-link / declared-but-uncited — 详见 CLAUDE.md "source_count 字段约定" 与本文第 6d 步）** +- **关系类型问题（`relation_issues_count`，`[[X|RELATION]]` 中非标准关系类型词——拼写错误 / 未在白名单，详见 CLAUDE.md "关系类型语法"）** +- **Web i18n 违规（`i18n_violations_count`，web/ 下硬编码中文 UI 字符串，违反 CLAUDE.md "Web 管理台国际化方案"）** +- **长文档证据索引（`evidence_index`：status 必须 complete，natural/content/structural coverage=100%，`corpus_freshness.ok=true`；空结构章节逐项确认。missing/stale/incomplete 时先重建，不能在旧索引上做“知识库没有”判断）** ## 第 2 步：处理 `#to-be-updated` 积压 @@ -103,23 +106,38 @@ python scripts/k.py list-bare-claims --json **对每条裸论断**： - 如果能立刻定位 raw 来源 → `Edit` 给段落补 `[[raw/#^]]`（可用 `python scripts/k.py find-anchor raw/.md "<片段>"` 反查 anchor） -- 如果暂时找不到来源 → `Edit` 段落末加 `[需要来源]` 占位（AGENTS.md 推荐的诚实标注） +- 如果暂时找不到来源 → `Edit` 段落末加 `[需要来源]` 占位（CLAUDE.md 推荐的诚实标注） - 如果数字是上下文性提及（如"2017 年"指 Transformer 提出年）而非论断 → 也加 `[需要来源]` 让占位显式化 > **目的**：把"无声的裸论断"逐步转为"有声的占位"，等真正的 raw 入库后能 grep `[需要来源]` 一次性补全。 -### 5b. fact-check 抽查 +### 5b. 引用语义审计（配额制，取代旧「随机抽 3-5 页」） -随机选 3-5 个 `wiki/sources/.md`，做事实抽查： +**先确定性分诊**（零 token）： -1. `Read` 摘要页 -2. 找一条具体数据/论断（如"准确率 95.3%"） -3. `Read` 它引用的 `[[raw/...#^anchor]]` -4. 验证引用对应的原文是否真的支持这个数字 +```bash +python scripts/k.py list-cite-mismatches --json +``` + +对每条 `mismatch` 四选一处置：① 改数字（按被引原文）② `find-anchor` 换正确锚点 ③ 确属「新旧证据打架」→ 建 `[!WARNING]` 知识更新冲突块交人裁 ④ 确属跨块推算 → 标 `[KB 推算: ^依据锚]`。`imprecise-anchor` 批量修锚点。`cite_exempted_count` 环比异常增长时抽查豁免块真伪（防「抄错说成推算」）。 + +**再做语义审计周配额**（协议细节与 CAUTION 标注格式见 CLAUDE.md「引用语义审计规范」）： + +```bash +python scripts/k.py extract-claims --unaudited-only --sample 20 --seed --json +``` -如果发现不一致： -- 手写 `> [!WARNING] 知识更新冲突` 块（详见 AGENTS.md "冲突处理规范"），不要默默修正 -- 在周报中列出 +- `--seed` 用当周周号：同周重跑取样一致（可复现），跨周覆盖可累积——`unaudited_citations_count` 应逐周下降 +- 对每对：`read-block` / `read-section` 取回被引原文 → **fresh-context 判定**（禁止凭「ingest 时读过」的记忆）三问：关键事实在场？直接支撑？语义 drift？→ `cite-audit-log` 入账（SUPPORTED 必须附 `--evidence "<被引块原文子串>"`） +- **UNSUPPORTED / 实质事实错的 PARTIAL**：在论断块正下方追加 CAUTION 审计标注 + 页面 tags 加 `citation-suspect`——**不默默修正**，让人在工作台看到；修复建议写进标注块 +- **多来源合成论断防误伤**（`multi_source: true`，多见于 analysis / comparison 页）：单块只支撑论断的一部分是正常形态 → PARTIAL 且**不落标注**；只有「被引块与归属分句无关或矛盾」才 UNSUPPORTED。约数舍入 / 跨语言转写 / 已标 `[KB 推算: ^锚]` 的派生算术按 SUPPORTED 处理 +- 收尾对账（堵「删标注蒸发」）： + ```bash + python scripts/k.py list-suspect-citations --check-ledger + ``` + `ledger-unsupported-without-marker` 必须为空（有 = 标注被删而论断未改，恢复标注或走正规修复） +- **agent_summary 抽查**（章节精排摘要的蒸馏忠实度——它被 ingest 综合判断 / 升级判定 / query 深度判断三处消费但无自动核验）：随机抽 3 个被 wiki 引用的章节，`read-section` 对照原文核摘要是否失真；失真则 `annotate-section` 重写并在周报记一笔 +- **SUPPORTED 交叉复查**（防橡皮图章与同源盲区）：每周对已 SUPPORTED 台账记录换模型抽查——`python tools/cite-audit/audit.py --workspace --all --sample 10 --seed `（DeepSeek 盲填+反驳双通道，同 seed 可复现；需 `DEEPSEEK_API_KEY`，无 key 时以 fresh-context 子 agent 反驳式抽查替代）。复查翻案的对按 UNSUPPORTED 流程落 CAUTION ## 第 5c 步：partial re-ingest 升级候选检测 @@ -171,10 +189,10 @@ python scripts/k.py search "<某高频被提及的概念>" --json python scripts/k.py list-i18n-violations --json ``` -扫描 `web/` 下 `.tsx` 文件，找硬编码的中文 UI 字符串（`` / `placeholder="中文"` / `aria-label="中文"` 等）。AGENTS.md "Web 管理台国际化方案" 明令所有 UI 字符串走 `t()` / `useT()`，硬编码会让英文用户看不懂。 +扫描 `web/` 下 `.tsx` 文件，找硬编码的中文 UI 字符串（`` / `placeholder="中文"` / `aria-label="中文"` 等）。CLAUDE.md "Web 管理台国际化方案" 明令所有 UI 字符串走 `t()` / `useT()`，硬编码会让英文用户看不懂。 **对每条违规**： -1. 在 `web/lib/i18n.ts` 的 `TRANSLATIONS.zh` 与 `.en` 同时加 key + 翻译（i18n.ts 的 TS 类型会强约束两边对称） +1. 在 `web/lib/i18n.ts` 的 `TRANSLATIONS.zh` 与 `.en` 同时加 key + 翻译（注意：TS 类型只从 zh 侧派生 key、**不强约束 en 侧**，en 缺 key 会静默回退——同步靠约定 + `test_i18n_sync.py` CI 守护） 2. 把 `.tsx` 中的硬编码字符串改为： - Server component：`{t("key", locale)}`（locale 从 `getServerLocale()` 拿） - Client component：`{t("key")}`（t 来自 `useT()`） @@ -271,7 +289,7 @@ python scripts/k.py list-unsummarized --json python scripts/k.py list-relation-issues --json ``` -扫描 `[[X|RELATION]]` 三段链接中**看起来像关系类型、但不在白名单**的词（对应 health 的 `relation_issues_count`；典型是拼写错误 `SUPORTS` 或用了非标准词 `IMPLEMENTS`）。白名单 7 个：`SUPPORTS` / `REFUTES` / `EXTENDS` / `IS_A` / `PART_OF` / `ALTERNATIVE_TO` / `CITES`（详见 AGENTS.md "关系类型语法"）。 +扫描 `[[X|RELATION]]` 三段链接中**看起来像关系类型、但不在白名单**的词（对应 health 的 `relation_issues_count`；典型是拼写错误 `SUPORTS` 或用了非标准词 `IMPLEMENTS`）。白名单 7 个：`SUPPORTS` / `REFUTES` / `EXTENDS` / `IS_A` / `PART_OF` / `ALTERNATIVE_TO` / `CITES`（详见 CLAUDE.md "关系类型语法"）。 **对每条**： 1. `Read` 该页，看上下文判断作者本意是哪种标准关系 @@ -322,8 +340,10 @@ tags: ## 本周处理 - 处理 `#to-be-updated`： 条 → <列出页面> - 修复孤儿： 个 → <列出页面与处理方式> -- fact-check 通过： 条 -- fact-check 发现不一致： 条 → <已标注冲突的页面> +- cite-mismatch 修复： 条（改数字 / 换锚 / 建冲突块 / 标推算各 ） +- 引用审计： 对（SUPPORTED/PARTIAL/UNSUPPORTED = //），累计未审 对 +- 统计保证：`python scripts/k.py audit-confidence` → 95% 置信未通过率上界

%（覆盖率 ；上界解释见命令输出的诚实边界说明） +- 引用审计未通过已标注： 条 → <已标 CAUTION 的页面> ## 待人类判别 <对所有未决冲突列表，每条带链接和摘要> @@ -349,7 +369,8 @@ tags: ## [2026-04-28] lint | 周度健康检查 WXX - 处理积压： 条 - 修复孤儿： 个 -- fact-check：/ 通过 +- cite-mismatch 修复： 条 +- 引用审计： 对（S/P/U = //），UNSUPPORTED 已标注 条 - 待人类判别冲突： 条 - 产出：`wiki/analyses/周报-YYYY-WXX.md` ``` @@ -366,6 +387,7 @@ git commit -m "lint: 周度健康检查 WXX" ## 完成检查清单 - [ ] 跑了 health 拿到全景 +- [ ] evidence_index 为 complete/fresh，三类机械覆盖均为 100%，异常空章节已确认/修复 - [ ] 处理了 to-be-updated 积压（或在周报中说明为什么没处理） - [ ] 处理了孤儿页面（4 种处理方式之一） - [ ] 复核了所有未决冲突 diff --git a/.claude/skills/kb-query/SKILL.md b/.claude/skills/kb-query/SKILL.md index a3bb118..6bdbcad 100644 --- a/.claude/skills/kb-query/SKILL.md +++ b/.claude/skills/kb-query/SKILL.md @@ -47,7 +47,7 @@ description: 知识库查询工作流——像研究员一样按 root_index → 这些都是后续产品化开发的工作。**当前 Claude Code 用户**：默认 quick，第 1-8 步走完即可，第 7a/b/c 步当作设计文档参考。 -> **重要：第 4.6 步「细节下钻判据」是所有模式共有的核心行为，quick 也执行**——它不属于产品化预留。即 Claude Code 默认的 quick 模式**会在真正需要原文细节时主动 `read-block` / `read-section` 回查原文**，只是不像 audit 那样对每条引用全量核验。模式差异仅在第 7a/b/c 的追加动作，不在"要不要按需下钻原文"这件事上。 +> **重要：第 4.6 步「细节下钻判据」与第 4.7 步「细节发现」是所有模式共有的核心行为，quick 也执行**——它不属于产品化预留。即 Claude Code 默认的 quick 模式**会在真正需要原文细节时主动 `read-block` / `read-section` 回查原文**，只是不像 audit 那样对每条引用全量核验。模式差异仅在第 7a/b/c 的追加动作，不在"要不要按需下钻原文"这件事上。 --- @@ -152,6 +152,75 @@ python scripts/k.py read-section raw/papers/.md # 取完整 H2/ > **与 audit 模式（第 7a 步）的关系**：本步是日常的"按需精确"——只对真正需要原文级精确的论断下钻；audit 是"全量强化版"——对答案里**每条** anchor 都回查。本步在所有模式（含 quick）默认执行，audit 在此之上把核验扩展到全部引用，两者不冲突。 +## 第 4.7 步：细节发现（原文级检索，wiki 蒸馏层没有时） + +第 4.6 步管「核验」——wiki 已有论断，回原文核对；本步管「发现」——**问题要的细节 +wiki 综合层根本没有**（长文档 ingest 后 90%+ 的内容只在 raw）。此时不要直接回答 +"知识库未涵盖"，**按问题类型选路由**（两路可交替、可都走）： + +| 问题特征 | 首选路径 | +|---|---| +| 有明确术语 / 专名 / 数字（"SCaNN 的查询延迟是多少"） | **自然单元证据索引路**（快而准） | +| 概念性 / 改述性 / 跨文档（"有没有讲过用 RL 替代监督微调这类思想的内容"） | **浏览路**（你阅读目录做语义判断——无 embedding 架构下语义召回的正解） | + +### 自然单元证据索引路（search-evidence） + +0. **先验新鲜度闸门**： + ```bash + python scripts/k.py evidence-index-coverage + ``` + 只有自然单元与章节覆盖为 100%、`corpus_freshness.ok=true` 才能检索。索引 missing/stale/incomplete 时先 `rebuild-evidence-index`；不得在旧索引上把“没搜到”解释为知识库没有。 +1. **先保留原问题，再做显式分面/关键词扩展**：先用用户原问题跑一遍；再把问题拆成必答 facets，逐 facet 补中英同义词、术语/俗称、缩写/全称。扩展只用于召回，不得把第一轮候选中的未知答案或 Gold 信息偷塞进查询： + ```bash + python scripts/k.py search-evidence "<用户原问题>" --limit 20 + python scripts/k.py search-evidence "" --limit 20 \ + --expand "<英文术语/全称>" --expand "<中文别名/缩写>" + ``` + multi-hop 的后续查询只能使用**第一跳实际读到**的新实体，不能预填中间答案。 +2. **按完整证据集而非单一 top-1 收集**：合并各 facet 的 top-20，确认每个必答 facet 至少有一份候选；表格行/list item 虽共享父块 anchor，仍按返回的 `unit_id/content_hash/subordinal` 区分，不能把同锚的错误行当成正确行。先 `read-evidence-unit ` 读取被选中的精确行/条目，再读取其 `canonical_ref` 父块获得上下文。 + - `search-evidence` 对超长自然单元只返回最多 30K、围绕命中词的原文摘录，并标 `text_truncated=true`。`read-evidence-unit` 默认同样在 30K 处 fail-closed；不得由 agent 自动用 `--max-chars 0` 绕过。此时应下钻来源结构/换更细自然块，或将“显式无限制阅读”作为人工审查逃生口。 +3. **精读与选择裁决**：search-evidence 是机械候选生成，命中片段 ≠ 语义支撑——对拟选候选必须先 `read-evidence-unit` 核对精确自然单元，再 `read-block` 打开 canonical parent block，核对主体、指标、值、单位、条件、否定和时间范围。只有核对通过的 unit handle 才能成为 selected evidence；都不对就换措辞、迭代检索或转浏览路。 + + **粒度边界必须披露**：当前 Markdown `[[raw/...#^t/p-...]]` 仍引用整个父 table/list block，精确 row/item 身份存在 `unit_id` 派生索引中。最终写入 wiki 前应尽量把论断拆到单行职责，并保留所选 unit handle 的审计记录；不能声称现有 Markdown 已实现 row-level block anchor，也不能用父表中另一行的值支撑当前行。 +4. **兼容回退**：旧索引不可用且已明确披露降级时，才用 `search-raw` 做纯扫描；它不是与新索引静默混用的第二套排名真相。 + +### 浏览路（corpus-map → outline → read-section） + +像人类研究员翻图书馆目录，三跳定位任何章节，每跳都是你在做语义判断： + +1. **全库地图**（一次调用获得全库视野；~98 篇约 12K token，可 --file 聚焦）： + ```bash + python scripts/k.py corpus-map # 全库 + python scripts/k.py corpus-map --file <路径子串> --sections # 聚焦候选，逐节显示摘要 + ``` + 每篇给出：标题 / 字符数 / 档位 / 顶层章节树（标题 + 摘要或首段预览）/ 深度登记 + 状态（✓⊙×）/ 关联 source_summary（「未 ingest」= 从未蒸馏过的盲区文档，优先怀疑）。 +2. **选篇钻取**：读地图选 2-4 篇候选 → `outline ` 看完整章节树 + preview → + 选章节 → `read-section`（单节 ≤30K；超长节继续下钻完整子节，无子标题时按自然单元定位并读完整父块，不做任意 chunk）。 +3. **精读裁决**：读完由你判断内容是否回答问题；不是 → 回地图换候选。 + +### 两路共享的收尾（缺一不可） + +- **懒回填摘要**：经浏览路 `read-section` 读过的章节，若其 agent_summary 为空， + 顺手 `annotate-section` 回填一句——读都读了，零边际成本；全库摘要密度（目前 + 普遍很低）靠真实使用增长，与懒深化同构，地图会因此越用越准。**但 annotation 会使证据索引的摘要路由指纹过期**：本次所有懒回填完成后，作答/归档前统一重跑 `rebuild-evidence-index` + `evidence-index-coverage`；若还要继续 `search-evidence`，则必须先重建。 +- **deepen 触发**：读到的内容所在章节在登记表为 ⊙ 扫读且被用于作答 → 按第 4.5 步 + 触发 partial re-ingest（search-raw 命中自带 deepen_hint 标注；浏览路看 corpus-map + 的 [⊙] 标注即知）。 +- **作答必须引用读过并通过主体/条件核对的原文块锚点**（`[[raw/...#^p-...]]`）；table/list 候选还必须先用 `read-evidence-unit` 锁定具体行/条目——search-evidence / corpus-map / outline 的 + 标题、摘要、preview 都只是路标，**不许当内容作答**（摘要可能失真，且无块级锚点 + 不可溯源）。 +- 照常过第 7.5 步 check-draft 闸门。 + +**raw 不在场时（release demo / 浅 clone）**：`search-evidence` / `search-raw` / `corpus-map` 会明确提示 +raw/ 未分发——此时原文级检索不可用，基于 wiki 蒸馏层作答并在答案中说明「原文 +不在场，细节无法核验」，不要假装检索过原文（与引用核对的 unverifiable 降级同一 +诚实原则）。 + +**何时放弃**：自然单元索引路（原问题 + 每个 facet 至少 2 组措辞）与浏览路（corpus-map 候选筛过）都落空，才可以 +回答"知识库未涵盖"，并说明尝试过的检索词与翻过的候选文档（诚实透明，也方便用户 +提供正确术语后重试）。 + ## 第 5 步：顺藤摸瓜（backlinks / outlinks） 如果第 4 步发现某页很关键，查它的关系网络： @@ -210,6 +279,28 @@ python scripts/k.py list-conflicts --json # 看有没有相关冲突 - 不要编造未在 wiki 中出现的内容 - 如果 wiki 没说，明确说"知识库中未涵盖这个主题" +## 第 7.5 步：答案落笔前的严格引用闸门（实质性 KB 回答必做） + +只要答案包含从 KB 提取的**实质性事实论断**（不再只限于数字 / 引文），发出前都必须经过完整证据取回、fresh-context 语义审计和 `strict` 总闸门。仅纯导航回应或「知识库未覆盖」的拒答可跳过。 + +```bash +# 1) 把最终答案草稿写到临时文件（如 /tmp/answer-draft.md），先跑零 LLM 确定性层 +python scripts/k.py --workspace check-draft /tmp/answer-draft.md + +# 2) 外部跨模型二审：完整 evidence + 盲填 + 对抗反驳 + 受控入账 +# 需 DEEPSEEK_API_KEY；工具/网络/核验包任一失败都是非 0，不得当作通过 +python tools/cite-audit/audit.py --workspace --draft /tmp/answer-draft.md + +# 3) 最终 fail-closed 总闸门 +python scripts/k.py --workspace check-draft /tmp/answer-draft.md --strict +``` + +`--strict` 必须全绿：它会阻断 broken / raw 不可得 / 非 canonical 锚点 / imprecise / pending `[需要来源]` / bare 或 coarse / 无当前版本 provenance / 未审或非 `SUPPORTED` verdict / 截断核验包。任一 checker 异常或超时同样视为失败，禁止用「删引用」或改成整页链接洗白。 + +如果当前环境无跨模型 API，则必须由**不共享写作上下文**的核验 agent 逐对判定，再用 `cite-audit-log --draft /tmp/answer-draft.md` 受控入账；无法完成这一步时，只能对相应事实拒答 / 明确标为未核验，不得交付「已验证」引用。 + +> 这是第 4.6 步「按需下钻」的机器化收口：quote-first 减少写错，`strict` 证明当前草稿的每条保留引用都已现场取回、语义审计并与现行内容 hash 绑定。 + ## 第 7a 步：audit 模式扩展 — 引用核实通道 > **产品化预留**——Claude Code 默认不执行此步。下方流程是为后续产品化（CLI / API / Web UI 显式调用）准备的设计规范，未来 agent 在 mode=audit 时按此执行。 @@ -411,9 +502,12 @@ git commit -m "query: <分析主题>" - [ ] 读了完整页面，没有读 chunk - [ ] 检查了相关 source_summary 的「章节深度登记」表；命中扫读章节已触发 partial re-ingest 升级 - [ ] 命中"需要原文级精确"的论断（精确引文/数字/日期/条款，或来源为第③档长文）已按第 4.6 步 `read-block`/`read-section` 核验原文 +- [ ] 原文细节问题已先确认 `evidence-index-coverage` 为 100% 且 fresh；按 required facets 跑 `search-evidence` top-20，拟选引用逐块核验，未把 candidate non-empty 冒充回答正确 - [ ] 至少做了一次 backlinks 或 outlinks 检查（除非问题极简单） - [ ] 做了覆盖度自检 - [ ] 答案的每个论断都有引用 +- [ ] 实质性 KB 回答已完成草稿跨模型审计，并跑 `check-draft --strict` 全绿（第 7.5 步） +- [ ] 细节不在 wiki 时走过第 4.7 步双路由（关键词路 ≥2 组措辞 / 浏览路 corpus-map 筛过候选），没有轻易回答"未涵盖"；读到 ⊙ 扫读章节已触发 partial re-ingest；浏览路读过的无摘要章节已顺手 annotate-section - [ ] （如果有价值）归档到 analyses/ 并 commit **产品化预留模式**（Claude Code 不执行，留作未来 CLI / API / Web UI 显式调用时的执行规范）： @@ -430,3 +524,5 @@ git commit -m "query: <分析主题>" - ❌ 切片读取（只读某段、想象其他段的内容） - ❌ 把综合分析丢弃在对话历史里（应归档到 analyses/） - ❌ 跳过第 4.5 步——拿 ⊙ 扫读章节的 outline preview 当真知识用（必须先触发 partial re-ingest 升级到 ✓ 深读再综合） +- ❌ wiki 综合层查不到就直接回答"知识库未涵盖"——细节大概率在 raw 里，必须先走第 4.7 步双路由（关键词路 + corpus-map 浏览路）；也不要只试一组关键词就放弃 +- ❌ 拿 corpus-map / outline 的标题、摘要、preview 当内容作答——它们只是路标，作答必须 read 原文块并引用块级锚点 diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 65fd558..0c811d9 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -14,6 +14,7 @@ What changed, and why? ## Checks - [ ] `python -m pytest scripts/tests` +- [ ] `python evals/run_stage2.py` (public/synthetic gates; not hidden certification) - [ ] `python scripts/k.py health --json` - [ ] `cd web && npm run lint` - [ ] `cd web && npm run build` diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e01f580..e7b4a5f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -34,6 +34,12 @@ jobs: - name: Run Python tests run: python -m pytest scripts/tests + - name: Run Stage-2 public accuracy gates + # Public/synthetic protocol gates only; the result explicitly remains + # hidden_certification=false. Run on every supported Python version so + # fixture/scorer drift cannot merge behind a green unit-test summary. + run: python evals/run_stage2.py + - name: Run knowledge base health check (default workspace must be non-empty) # A fresh clone with no wiki pages (total_pages == 0) is a regression: # it usually means the default workspace data was lost or gitignored away. diff --git a/.gitignore b/.gitignore index 2daa6c6..ef8aa7b 100644 --- a/.gitignore +++ b/.gitignore @@ -53,6 +53,12 @@ workspaces/*/wiki/**/*.outline.json # 本地烟测临时目录 .tmp/ +# 外部隐藏评测材料必须与公开 smoke/Gold 隔离，避免误提交导致评测泄漏。 +# 真正的发布 bundle 应放仓库外；以下目录只作为最后一道本地防误提交保护。 +/evals/holdout/private/ +/evals/holdout/results-private/ +/evals/answer_citation/private/ + # Claude Code 源码副本（仅供架构借鉴参考，不属于知识库本身） claude_code源码/ diff --git a/AGENTS.md b/AGENTS.md index 8ed756f..224d461 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -20,12 +20,12 @@ ## 核心设计原则（不可违反） -1. **知识库不调用 LLM**。所有 LLM 推理由外部 agent 完成；知识库本身只暴露 MCP 工具与 REST API，不内嵌任何 agent runtime、LLM SDK 或对话能力。 +1. **知识库不调用 LLM**。所有 LLM 推理由外部 agent 完成；知识库本身只暴露 CLI 与 REST API，不内嵌任何 agent runtime、LLM SDK 或对话能力。 - **本条原则的范围**：`scripts/`、`web/`（KB 核心）严禁内嵌 LLM SDK。 - **例外**：`tools/debug-console/` 是独立子项目，**作为 KB 的外部客户端存在**，可以引入 LLM SDK。它只通过 HTTP 调主 `web/` 的 REST API（`/api/agent-tool` 等），不直接读 markdown / `.cache/`。删掉 `tools/` 整个目录不影响 KB 任何功能。 -2. **markdown + Git 是唯一真相源**。SQLite 索引（`.cache/index.db`）是派生层，可随时从 markdown 全量重建。删 `.cache/` 系统仍能跑。 -3. **完整页面优先**。所有读取工具返回**完整页面或完整 H2/H3 段**，绝不返回 chunk。 -4. **严禁 embedding 召回**。embedding 模型 / 向量存储 / 文档切片不出现在系统的任何"找相关内容"逻辑中。检索靠 BM25 全文 + 元数据过滤 + agent 阅读完整页面。 +2. **markdown + Git 是唯一真相源**。SQLite 证据索引（`.cache/retrieval_index.db`）是派生层，可随时从 markdown 全量重建。删 `.cache/` 只会失去检索加速/路由凭证，原文与 wiki 仍在。机械 100% 覆盖的分母是**当前已转换 markdown 的 parser inventory**；它不自动证明 PDF/DOCX→markdown 转换保真，原始载体仍需绑定 source SHA +（实际后端版本 + 实现文件 SHA）converter fingerprint 的 conversion receipt 与视觉抽检。 +3. **自然单元检索，完整块阅读**。检索坐标可以是 paragraph、单条 list item、单条 table row、blockquote、code、figure 等自然单元；它们不是固定长度 chunk。作语义判断或引用前必须读取其完整 canonical parent block 或完整 H2/H3 段。 +4. **严禁 embedding 召回**。embedding 模型 / 向量存储 / 固定长度文档切片不出现在系统的任何“找相关内容”逻辑中。检索靠 FTS5/BM25 + trigram/exact + 元数据/结构导航 + agent 阅读完整原文块。 5. **写权限硬约束**：写 `raw/**`、`my_thoughts/**`、含 `#human-only` 标签或 `locked: true` frontmatter 的文件 → 工具直接拒绝（PermissionError），不是 ask、不是 warn、是 deny。 6. **删除即标记**：所有"删除"操作只能改 `status: deprecated`，绝不真删文件；历史信息有内在价值。 @@ -37,8 +37,8 @@ | 层级 | 入口 | 用途 | |---|---|---| -| **工作流** | Skill：`kb-ingest` / `kb-query` / `kb-lint` / `kb-conflict-resolve` / `kb-export` | 端到端的摄入 / 查询 / 周检 / 冲突 / 导出 | -| **原子操作** | `python scripts/k.py ` | outline / search / read-section / read-block / backlinks / outlinks / annotate-section / list-source-issues / list-broken-refs / list-to-update / list-orphans / list-conflicts / health 等 | +| **工作流** | Skill：`kb-ingest` / `kb-query` / `kb-lint` / `kb-cite-audit` / `kb-conflict-resolve` / `kb-export` / `kb-edit-source` | 端到端的摄入 / 查询 / 周检 / 引用审计 / 冲突 / 导出 / 来源编辑 | +| **原子操作** | `python scripts/k.py ` | rebuild-evidence-index / evidence-index-coverage / search-evidence / read-evidence-unit / outline / search / read-section / read-block / backlinks / outlinks / annotate-section / list-source-issues / list-broken-refs / list-to-update / list-orphans / list-conflicts / list-cite-mismatches / extract-claims / cite-audit-log / list-suspect-citations / search-raw / corpus-map / health 等 | | **直接读写** | `Read` / `Edit` / `Write` | `wiki/**` markdown 文件 | 下文「四大操作流程」中出现的 `read_page` / `archive_analysis` / `mark_conflict` / `append_log` / `git_commit` 等是**抽象动作名（接口契约）**，描述 agent 应做什么，**不是可调用的 MCP tool**。落地实现 = skill 文档 + `k.py` 子命令 + `web/lib/operations.ts` 的 server action。 @@ -77,44 +77,6 @@ groundmap/ # 引擎根（通用代码 + 规范） --- -## 双仓库同步约定（dev ↔ release） - -本仓库（`AI知识库/`）与 `groundmap-release/` 是**两个独立 Git 仓库**承担不同角色： - -| 仓库 | 角色 | 数据 | 分支 | -|---|---|---|---| -| `AI知识库/`（本仓） | **开发版** | 含实际 wiki / raw / exports 数据 | `rag-evolution-ip-standard` 等 | -| `groundmap-release/` | **发布版** | 引擎 + 3 个精选示例 demo 库（仅 `wiki/` 随仓；`raw/`、`my_thoughts/` 不分发）+ 已审的发布准备改动 | `main` | - -**哪些修改必须双仓同步**（任一改完都需在另一仓做对应改动，否则下次 sync 漂移）： - -1. **`scripts/k.py`、`scripts/convert.py`、`scripts/section_parser.py`、其他通用引擎代码**： - 同步整个文件；release 的 workspace fallback 逻辑（k.py 第 2644-2671 行）保留不动。 -2. **`scripts/tests/`**（含 `TestMirrorSync` 守护）：**完全镜像**——dev 改了测试，release 必须改相同处。 -3. **`.claude/skills/kb-*/SKILL.md` 与 `.agents/skills/kb-*/SKILL.md`**：SKILL.md 内容**逐字相同**（`.claude ↔ .agents` 由 `TestMirrorSync.test_skills_mirror` 守），dev 与 release 之间靠人肉 / rsync 同步。 -4. **`web/`（Next.js 管理台）**：dev 与 release 同步主要 UI 改动；release 可能含更多发布准备（i18n key 整理、未发布特性等），合并时以 release 为基线。 -5. **`docs/`**（用户文档）：dev 写新内容 → 同步到 release；release 的发布准备改动（demo 视频、新手教程）一般不回 dev。 - -**哪些修改不要镜像到 release**： - -- dev 里**实际在用的**工作数据（`workspaces//raw/**`、私人 `my_thoughts/**`、`exports/**`）——不镜像到 release。release 自带的是另一套**精选 demo 库**：仅 `wiki/` 随仓分发，`raw/`、`my_thoughts/` 不分发。 -- dev 专属的实验性 lint / 临时脚本。 - -**不变量清单**（任一变动都视作"破坏不变量"、必须同时同步）： - -- `RELATION_TYPES` 白名单 7 类（k.py ↔ web/lib/markdown.ts） -- `WIKILINK_RE` 正则（k.py ↔ web/lib/markdown.ts） -- `TestMirrorSync` 的归一化规则（CLAUDE.md ↔ AGENTS.md ↔ `.claude/skills ↔ .agents/skills`） -- 默认 workspace 解析（dev 默认 `smb-ecommerce`；release 不设固定默认——未指定时 CLI 自动选用存在的第一个 workspace，**故意不同**，反映发布清理意图） - -**`TestMirrorSync` 守护的镜像范围**（仅在单仓内）： - -- `CLAUDE.md` ↔ `AGENTS.md`（按归一化：Claude Code/Codex、CLAUDE.md/AGENTS.md、`.claude/skills/.agents/skills` 三组替换后必须一致） -- `.claude/skills/*/SKILL.md` ↔ `.agents/skills/*/SKILL.md`（同归一化） -- **dev 与 release 之间没有跨仓镜像测试**——必须靠"修改后手动 sync + 跑两侧 `pytest scripts/tests/`"保证 - ---- - ## 权限规则 | 路径 / 标记 | 权限 | @@ -291,9 +253,9 @@ tags: [] - **不依赖任何特定渲染器**——知识库的解析独立实现，与 Obsidian、Foam、Logseq 等的兼容是顺便的 - 链接目标可以是：完整页面 `[[wiki/concepts/transformer]]`、特定段 `[[wiki/concepts/transformer#注意力机制]]`、特定块 `[[raw/papers/smith2026#^p-12-7d8e9a]]` -无法提供精确来源时，必须显式标注 `[需要来源]`，**不得省略**或猜测。 +无法提供精确来源时，必须显式标注 `[需要来源]`，**不得省略**或猜测。数字为**跨块计算 / 单位换算**所得、被引原文无该字面时，标 `[KB 推算: ^依据锚]`（详见「引用语义审计规范」）。 -**粒度硬约束（lint 守门）**：整篇引用 `[[raw/X]]`（无 `#^`）**仅限**「来源绑定 / 纯背景介绍」；任何含数字 / 指标 / 结论的**实质论断必须**用块级 anchor `[[raw/X#^...]]`——整页引用去支撑论断属"引用粒度不足"。由 `python scripts/k.py list-coarse-citations` 扫出（接入 `health` 的 `coarse_citations_count`），与 `list-bare-claims`（有数字但无任何引用）互补：前者治"引得太粗"，后者治"没引"。块级 anchor 才能精确溯源、并在 web 端渲染为论文式 `[n]` 上标。 +**粒度硬约束（lint 守门）**：整篇引用 `[[raw/X]]`（无 `#^`）**仅限**「来源绑定 / 纯背景介绍」；任何含数字 / 指标 / 结论的**实质论断必须**用块级 anchor `[[raw/X#^...]]`——整页引用去支撑论断属"引用粒度不足"。由 `python scripts/k.py list-coarse-citations` 扫出（接入 `health` 的 `coarse_citations_count`），与 `list-bare-claims`（有数字但无任何引用）互补：前者治"引得太粗"，后者治"没引"。块级 anchor 才能精确溯源、被 `list-cite-mismatches` 语义核对、并在 web 端渲染为论文式 `[n]` 上标。 ### 关系类型语法（v0.4b 图谱） @@ -341,6 +303,54 @@ tags: [] --- +## 引用语义审计规范 + +结构 lint（broken-refs / bare-claims / coarse-citations）只保证引用「结构正确」；「被引块是否真的支撑论断」由三层机制保证——**KB 只出确定性数据，语义判定由外部 agent 完成，终审在人**： + +1. **确定性核对**：`python scripts/k.py list-cite-mismatches`——论断中的数字 / 逐字引文必须出现在被引块原文中（数值按论断声明精度做舍入容差匹配；约数标记放宽）。`mismatch` / `exempt-missing-basis` / `canonical-anchor-mismatch` / `canonical-target-mismatch` 是**闸门项**（接入 `health` 的 `cite_mismatches_count`）；accuracy-first ingest 还要求 `imprecise-anchor` 与 `unverifiable` 为空。引用依赖 hash recovery、`./raw/...` 等非 canonical handle 时必须先改写再复验；缺失文件只使该引用 `unverifiable`，同块其他可用引用仍继续核对，且 anchor-missing 不得遮蔽另一可用引用的 mismatch。`[KB 推算: ^依据锚]` 只豁免紧邻的单个值，依据锚必须在同一核对单元实际被引。**语义边界：数字共现 ≠ 语义支撑**，本层归零不构成来源正确性证明，曲解 / 过度概括靠下面两层。 +2. **写入时回验**（writer/verifier 分离）：kb-ingest 第 9.5 步——`k.py extract-claims --changed` 确定性枚举本次（论断, 引用）对，**fresh-context 子 agent**（不共享写作上下文）逐条判 `SUPPORTED / PARTIAL / UNSUPPORTED / CONTRADICTED`；UNSUPPORTED / CONTRADICTED 清零才可 commit，且**禁止**用「删引用、挂 `[需要来源]`」洗白；log.md 记 citation-verify 对账行，事后可 `extract-claims --commit ` 复枚举对账。 +3. **存量审计**：`kb-cite-audit` skill（kb-lint 第 5b 步按周配额调用：`extract-claims --unaudited-only --sample 20 --seed `，同 seed 取样可复现、跨周覆盖累积）——只审「从未审过 + 内容漂移」的对；^h- 目标对**整节正文**算内容 hash，确定性补上「锚点只 hash 标题、正文重写不被察觉」的盲区。 + +**`[KB 推算]` 标记**（与 `[需要来源]` 平级的内联标记）：数字为跨块计算 / 单位换算所得、被引原文无该字面时必标，且**必须带依据锚**（如 `[KB 推算: ^t-33-0c8446]`）；裸 `[KB 推算]` 本身是闸门项（exempt-missing-basis）。豁免只关数字核对，不改「块必须有引用」的语义；豁免块数进 `health`（`cite_exempted_count`），kb-lint 周检抽查真伪。 + +**CAUTION 审计标注**（审计未通过的唯一 markdown 落点；与「知识更新冲突」WARNING 分工明确——错引是「论断 vs 其引用」，不是新旧证据冲突，不走冲突决议四选项）： + +```markdown +> [!CAUTION] 引用审计未通过 — YYYY-MM-DD +> **论断**：…（块 ^p-4-34d5b1） +> **被引块**：[[raw/papers/X#^t-77-9e8336]] +> **审计判定**：UNSUPPORTED — 被引表格中数字为 65.9 非 66.9 +> **建议**：改引 [[raw/papers/X#^t-79-…]] 或修正论断数字 +> **状态**：⏳ 待人类判别 +``` + +`python scripts/k.py list-suspect-citations` 从 markdown 扫此标注（待办清单不依赖 .cache，删台账不丢待办；接入 `health` 的 `suspect_citations_count`）。人类处理 = 修论断 / 换锚后删标注（内容变化自动触发重审），或确认误报时把标注改写为一行 `> [!NOTE] 引用审计误报（YYYY-MM-DD 由人类复核通过）`（删除即标记，误报裁决留在真相源）。 + +**验证台账**（`workspaces//.cache/citation_audit.jsonl`，gitignored）：纯派生层——只记「谁在何时核验过什么」（agent 劳动的 memoization），删了唯一后果是全部回到未审、重审即重建。写入走 `k.py cite-audit-log`（`--mode audit|ingest|manual`、`--by agent|human`）；受控校验：pair 过期拒绝、目标可解析时记 UNVERIFIABLE 拒绝、**agent 记 SUPPORTED 必须附 `--evidence`（被引块现行原文字面子串，k.py 校验）**——强制取回原文，堵橡皮图章。台账 ↔ markdown 一致性由 `list-suspect-citations --check-ledger` 对账（堵「删标注蒸发」）。**知识状态（标注、论断修正、误报裁决）落 markdown 进 git；台账绝不入库。** + +**盲填复核（blind cloze，二次 LLM 审核的推荐形态）**：`extract-claims --cloze` 把论断里的可核对数字挖成 ⟦N1⟧ 占位；核验者**只看「挖空论断 + 被引原文」填空、全程看不到期望值**（从原理上消灭判定式审核的附和偏差，并抓「数字巧合在场但归属错误」）；填回值经 `k.py cloze-check` 按数值容差**机器判分**（块级 union：一块多引用时数字由块内任一引用的原文填出即可）。判分未过按 UNSUPPORTED 处置。 + +**查询侧核对（check-draft）**：查询答案 / 导出稿等「不在 wiki 里的文本」先跑 `k.py check-draft <草稿文件>` 的确定性层，再用 `tools/cite-audit --draft` 对完整核验包做盲填 + 反驳并入账，最后跑 `k.py check-draft <草稿文件> --strict`。Strict 对 broken / non-canonical / raw 不可得 / imprecise / pending / bare / coarse / 未映射定性事实 / 无当前 provenance / 非 `SUPPORTED` / 核验包截断全部 fail-closed；实质性 KB 答案必跑（kb-query 第 7.5 步）。 + +**跨模型二审（tools/cite-audit）**：`python tools/cite-audit/audit.py --workspace [--changed|--unaudited-only|--all --sample N --seed |--draft <草稿>]`——外部客户端（与 debug-console 同属原则 1 的 tools/ 例外区，只经 k.py CLI 取数 / 回写）调 DeepSeek 跑「完整 evidence 盲填 + 反驳」双通道并自动入台账。结果协议：exit 0 = 全部判定通过，1 = 完成且有语义未通过，2 = incomplete / skipped / 网络 / 协议 / 台账故障；1/2 均不得当成绿灯。API key 走环境变量 `DEEPSEEK_API_KEY`，不硬编码。 + +**原子论断分解（核对单元）**：list 块按**条目**拆分核对与枚举——条目是自然的原子论断边界，每条目对自己的引用负责（条目无引用回退整块引用并集），抓「条目 A 的数字只在条目 B 的引用目标里」这类块级 union 盲区；审计对的 `claim_text` 也随之聚焦，核验者不必自行推断归属。 + +**检索凭证链（check-provenance）**：`read-block` / `read-section` / `blocks` / `extract-claims --with-evidence` 自动向 `.cache/retrieval_log.jsonl` 登记「何时取回过哪个块 / 节的哪个内容版本」（派生层：删了 = 凭证重置，重新取回即可）；`k.py check-provenance --changed` 校验每条新引用都有**匹配当前内容 hash** 的取回凭证——没真读过（或读的是旧版本）就造不出凭证，把 quote-first 从行为规范升级为可机器校验的溯源链（ingest 提交前闸门之一）。 + +**审计统计保证（audit-confidence）**：`k.py audit-confidence` 把「审过多少、发现多少」换算成验收抽样口径的陈述——已审引用未通过率的 Clopper-Pearson 置信上界（纯数学，零 LLM）。诚实边界随结果携带：上界对已审子集精确成立，推广到全库要求样本有代表性（seed 抽样 / 全覆盖）；且以验证器判定为准，验证器自身查全率由 evals/cite-check 度量。 + +**对抗评测集（evals/cite-check）**：给防线本身立尺子——已知错引类型的毒化案例 + 干净对照，`python evals/cite-check/run_eval.py` 度量确定性层查全 / 误报（pytest `test_cite_eval.py` 钉死为回归闸门），`--semantic` 加测语义层（DeepSeek 双通道）。新增错引类型时先加 case：哪层抓不住，哪里就是下一道要补的防线。 + +**长文 / 回答 / 原始载体分层评测（evals Stage-2）**：`python evals/run_stage2.py` 统一运行 external-holdout 协议 public smoke、canonical atomic claim→citation / abstention、PDF/DOCX/HTML 源格式保真三条闸门。三者必须分别报告 recall/coverage/precision/abstention/forbidden selection 与明确分母，不能合成一个平均分；公开结果终态固定 `hidden_certification: false`，不得冒充独立隐藏认证。修改 `scripts/convert.py` / `scripts/pdf_layout.py` / 自然单元检索或评测 scorer 时，必须运行对应定向 pytest 和 Stage-2；协议错误、缺依赖、零分母、Gold/hash 漂移均 fail-closed。 + +**PDF accuracy-first 转换**：PDF 强制走几何 layout-aware 路径，恢复 ruled/保守 borderless table、跨栏标题和 column-major 阅读顺序；不得在失败后静默回退到内容流顺序。扫描页、内容级光栅/矢量图、任一非已知 bullet 的无法解析 CID 字形、损坏/加密等无法可靠表示的来源必须明确失败并先走 OCR/图表专用流程；双栏/键值几何无高置信区分信号时也必须 fail-closed。独立图片与音频扩展名仅用于让批处理识别并明确拒绝：ExifTool 元数据不能代替视觉内容，现有音频路径又依赖 ambient ffmpeg/ffprobe 与无不可变模型 revision 的 Google SpeechRecognition，因此在专用 OCR/视觉或版本锁定转写流程产生可验 receipt 前必须 fail-closed。每份可转换的非 Markdown 来源先复制到只读快照后转换，派生 outline 用 canonical `source_path` + source SHA + converter fingerprint + receipt 自校验哈希锁定原载体与转换语义；同目标/大小写碰撞、派生 Markdown 二次处理、孤儿原载体、symlink、转换/写入期间原文变动、receipt 缺失/过期均 fail-closed。证据索引 schema v5 同时冻结这组 source binding，并在 rebuild、coverage、search、read 时重验；原 PDF/DOCX 被改、删、移动或转换实现/依赖漂移后，旧派生内容不得继续召回。只读位与事后 SHA 用于发现**受信任本地转换器**的意外写入和源文件竞态，不是恶意转换器的安全沙箱；具有同等本机权限的代码能改权限并在校验前恢复字节，所以只允许项目锁定/经审阅的 converter，不受信任的插件必须放入 OS 级沙箱。DOCX 的空表头只在“生成 header 全空 + 第一数据行每格均全粗体”的高置信形态下提升，禁止一般性猜表头。 + +**pre-commit 机械闸门**：staged 改动含 `workspaces//wiki/**.md` 时，hook 把 Git index 物化为隔离临时树，挂载本地 raw 并复制审计 / 检索台账快照后，对 staged 页强制：cite mismatch/imprecise/unverifiable/canonical 问题为 0，broken/bare/coarse/unmapped/source issue 为 0，每个 pair `target_status=ok` + `audited=true` + `last_verdict=SUPPORTED`，`check-provenance --paths` findings 为 0。Python / 临时树 / checker / 退出码 / JSON / schema 任一异常均 fail-closed；范围只限 staged 页，人类可 `--no-verify` 显式绕过并以 `human:` 说明。 + +--- + + ## 四大操作流程 > 所有操作均通过外部 agent 调用知识库的 CLI / REST 工具完成。知识库本身不驱动这些流程，也不内嵌 LLM 调用。 @@ -351,21 +361,22 @@ tags: [] 1. 用户将原始文件放入 `raw/` 对应子目录（agent 不得修改原始文件） 2. agent 调 `python scripts/convert.py`：把原始格式转为 markdown，自动加锚点（`^h-`/`^p-`/`^t-`/`^c-`/`^f-`），并生成 `.outline.json` -3. agent 调 `python scripts/k.py outline ` 看大纲，**按字符数三档自决阅读策略**： +3. agent 调 `rebuild-evidence-index` 后跑 `evidence-index-coverage`：全部 raw 转换后 markdown 的自然单元、内容/结构章节机械覆盖必须为 100%，完整 inventory 指纹与 exact text hash 对账、异常空章节逐项确认或修复，且 raw 全文 SHA 与已验证 outline 摘要指纹均新鲜；这是“已转换文本中的所有细节可发现”的结构闸门，不等于原始载体转换 100% 保真、AI 已理解或摘要无遗漏 +4. agent 调 `python scripts/k.py outline ` 看大纲，**按字符数三档自决阅读策略**： - **① 短文** `< 30000`（约 3 万中文字）：`Read` 全文 - - **② 中长文** `30000 – 150000`（论文 / 报告级）：按 H1 切块、每块 ≤ 3 万分段 `read-section` - - **③ 整本书规模** `> 150000`：TOC 扫全 + AI 自决深读章节，**全部章节登记**到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过）。⊙ 扫读章节保留 partial re-ingest 升级路径 + - **② 中长文** `30000 – 150000`（论文 / 报告级）：按标题树选 ≤ 3 万字的完整节 `read-section`，超长 H1/H2 继续下钻；无子标题的超长节按自然块阅读 + - **③ 整本书规模** `> 150000`：TOC 扫全 + AI 自决深读章节，**全部章节登记**到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过；首列写真实 `^h-` anchor、原标题不许意译、⊙ 行备注点名关键实体——`search-raw` 的 deepen 联动依赖此规范）。⊙ 扫读章节保留 partial re-ingest 升级路径 - 单次 Read 严格 ≤ 3 万中文字符，避免 LLM "lost in the middle" 衰减 -4. **每读完一个 H2/H3 章节，立即调 `k.py annotate-section "<一两句摘要>"` 回填精排摘要** —— **②③ 档（分段阅读）的必经步骤**；① 档短文一次 Read 全文，不强制分段回填（建议至少给主要 H2 回填一句摘要，非硬性） -5. agent **基于 wiki 现状做综合判断**：调 `k.py search` 反查相关 wiki 页，轻量阅读，自决三件事——**核心价值**（新东西在哪）/ **关联**（哪些 wiki 页有重叠）/ **冲突**（哪些论断打架） -6. agent 基于第 5 步综合**决定写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated -7. agent 在 `wiki/sources/` 创建摘要页（标准 frontmatter + 块级引用 + 「## AI 综合判断」H2 节固化第 5 步结论 + 第 ③ 档的「## 章节深度登记」表） -8. agent 立即更新最核心的 2-3 个节点页面 -9. agent 给其余受影响页面打 `#to-be-updated` 标签 -10. agent **自动决定 MOC 归属**：用 source_summary 的 tags 反查现有 MOC，命中则在「近期更新」节追加；无命中则用 `_templates/index_template.md` 自动新建 MOC + 在 root_index 加入口 -11. agent **图谱接入与校验**：写互链时凡关系属标准类型（支持/反驳/延伸/属于/组成/替代/引用）即用 `[[目标|SUPPORTS]]` 等标准关系类型（白名单见「关系类型语法（v0.4b 图谱）」节）让图谱可按边染色；收尾跑 `python scripts/k.py list-relation-issues`（须为空）与 `python scripts/k.py graph`（确认本次新页面已作为节点接入、边正常、无意外孤立节点——孤立即回第 8 步补 `[[...]]` 互链）。图谱是**派生层**（从 wiki 双链实时计算、无持久文件，对齐「markdown 是唯一真相源」），本步只校验、不产出需提交的文件 -12. agent 追加 log.md 条目 -13. **提交前质量闸门**（须全过，不过则补齐再提交）：`python scripts/k.py list-bare-claims` / `list-coarse-citations` / `list-source-issues` / `list-broken-refs` / `list-relation-issues` 须全空——裸论断补块级引用、整页引用升块级、缺 source 补全、失效引用修掉、非法关系词改正。通过后 agent `git commit -m "ingest: <来源标题>"` 原子提交（只含 `wiki/**` 改动 + `log.md`；`raw/` 及其派生 .md / .outline.json 默认被 `.gitignore` 排除、留在本地，不入库——版权与隐私原因） +5. **每读完一个 H2/H3 章节，立即调 `k.py annotate-section "<一两句摘要>"` 回填精排摘要** —— **②③ 档（分段阅读）的必经步骤**；① 档短文一次 Read 全文，不强制分段回填（建议至少给主要 H2 回填一句摘要，非硬性）。最后一次回填完成后必须再次运行 `rebuild-evidence-index` + `evidence-index-coverage`，确认摘要路由指纹新鲜且覆盖闸门仍为 100% +6. agent **基于 wiki 现状做综合判断**：调 `k.py search` 反查相关 wiki 页，轻量阅读，自决三件事——**核心价值**（新东西在哪）/ **关联**（哪些 wiki 页有重叠）/ **冲突**（哪些论断打架） +7. agent 基于第 6 步综合**决定写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated +8. agent 在 `wiki/sources/` 创建摘要页（标准 frontmatter + 块级引用 + 「## AI 综合判断」H2 节固化第 6 步结论 + 第 ③ 档的「## 章节深度登记」表） +9. agent 立即更新最核心的 2-3 个节点页面 +10. agent 给其余受影响页面打 `#to-be-updated` 标签 +11. agent **自动决定 MOC 归属**：用 source_summary 的 tags 反查现有 MOC，命中则在「近期更新」节追加；无命中则用 `_templates/index_template.md` 自动新建 MOC + 在 root_index 加入口 +12. agent **图谱接入与校验**：写互链时凡关系属标准类型（支持/反驳/延伸/属于/组成/替代/引用）即用 `[[目标|SUPPORTS]]` 等标准关系类型（白名单见「关系类型语法（v0.4b 图谱）」节）让图谱可按边染色；收尾跑 `python scripts/k.py list-relation-issues`（须为空）与 `python scripts/k.py graph`（确认本次新页面已作为节点接入、边正常、无意外孤立节点——孤立即回第 9 步补 `[[...]]` 互链）。图谱是**派生层**（从 wiki 双链实时计算、无持久文件，对齐「markdown 是唯一真相源」），本步只校验、不产出需提交的文件 +13. agent 追加 log.md 条目 +14. **提交前质量闸门**（须全过，不过则补齐再提交）：`list-bare-claims` / `list-coarse-citations` / `list-unmapped-claims` / `list-source-issues` / `list-broken-refs` / `list-relation-issues` 须全空，`list-cite-mismatches` 的 mismatch / exempt-missing-basis / canonical-* / imprecise / unverifiable 须全空。另须完成**引用回验**（writer/verifier 分离）：`extract-claims --changed --with-evidence --max-evidence-chars 1000000000` 枚举完整（论断, 引用）对，fresh-context 子 agent 与跨模型审计完成盲填 + 反驳，incomplete / skipped / UNSUPPORTED / CONTRADICTED 清零，每对经 `cite-audit-log --mode ingest` 入账且有当前 provenance，log.md 含 citation-verify 对账行（详见 `.agents/skills/kb-ingest/SKILL.md` 第 9.5 步）。通过后 agent `git commit -m "ingest: <来源标题>"` 原子提交（只含 `wiki/**` 改动 + `log.md`；`raw/` 及其派生 .md / .outline.json 默认被 `.gitignore` 排除、留在本地，不入库） > **partial re-ingest（增量深化）**：第 ③ 档扫读 / 跳过的章节保留升级路径，由 kb-query / kb-lint / 用户 web 端三种方式触发深化。AI 自动重读该章节 → 更新现有 source_summary（不新建）→ 章节登记表 ⊙ → ✓ → log.md 记 `partial-ingest` 类型 → git commit。 @@ -401,14 +412,14 @@ skill 设计了 4 个深度模式，但**Claude Code 中默认且唯一行为是 3 个高级模式的执行规范详见 `.claude/skills/kb-query/SKILL.md` 第 7a/b/c 步。它们是**设计契约**——为产品化时的 agent 提供明确的"该模式下做什么"指令，不在 Claude Code 默认行为内。 -> **细节下钻判据（所有模式共有，quick 也执行——与上面 3 个产品化预留模式不同）**：回答前若某论断需要**原文级精确**（精确引文 / 数字 / 日期 / 条款，或来源为第 ③ 档长文档），agent 必须先 `read-block` / `read-section` 打开对应 anchor 核验原文再下结论——保证"真正需要原文细节时必然回查"、不靠 agent 自觉；A 类常见问题（定义 / 概览，detail 已蒸馏进 wiki）不命中、不额外费 token。audit 是它的"全量强化版"（对答案里**每条**引用都回查）。详见 `.claude/skills/kb-query/SKILL.md` 第 4.6 步。 +> **细节下钻判据（所有模式共有，quick 也执行——与上面 3 个产品化预留模式不同）**：回答前若某论断需要**原文级精确**（精确引文 / 数字 / 日期 / 条款，或来源为第 ③ 档长文档），agent 必须先 `read-block` / `read-section` 打开对应 anchor 核验原文再下结论。细节不在 wiki 蒸馏层时，先确认 `evidence-index-coverage` 为 100% 且 fresh，再把问题拆成 required facets，用 `search-evidence` 对原问题与显式中英别名做 top-20 自然单元召回；multi-hop 后续查询只能使用第一跳实际读到的新实体。拟选候选先用 `read-evidence-unit ` 锁定精确行/条目，再 `read-block` 核对父块上下文与主体/指标/值/单位/条件/否定。当前 Markdown citation 对 table/list 仍是父块 anchor，row/item 精确身份在派生 unit handle 中，不能把父表另一行当成支撑。概念性/改述性问题同时保留浏览路（`corpus-map → outline → read-section`）；旧 `search-raw` 只作明确披露的扫描回退。详见 `.claude/skills/kb-query/SKILL.md` 第 4.6-4.7 步。 ### Lint 操作流程 1. agent 调 `list_to_update` → 逐个处理 `#to-be-updated` 积压 2. agent 调 `list_orphans` → 处理孤儿页面（无入链） 3. agent 调 `list_conflicts` → 复核所有冲突标注 -4. agent 抽查 wiki 论断与 raw 来源的一致性（fact-check） +4. agent 做引用语义审计（`kb-cite-audit` 周配额模式；先 `list-cite-mismatches` 确定性分诊，再 `extract-claims --unaudited-only --sample 20 --seed ` 逐条 fresh-context 判定入台账，UNSUPPORTED 落 CAUTION 标注——详见「引用语义审计规范」） 5. agent 检查缺少独立页面的重要概念 6. agent 调 `archive_analysis` 生成 `wiki/analyses/周报-YYYY-WXX.md` 7. agent 调 `append_log` @@ -530,7 +541,7 @@ export function MyButton() { | 层 | 内容 | 改它时 | |---|---|---| | **真相源** | `wiki/**.md`（agent 维护的知识）、`raw/` 原始文件 | 纯 markdown，不依赖任何代码即可读懂 / diff；删光代码内容仍在 | -| **派生层** | `.cache/index.db`、`raw/**.outline.json` | 可随时从真相源**全量重建**，删了不丢信息 | +| **派生层** | `.cache/retrieval_index.db`、未来的 `.cache/index.db`、`raw/**.outline.json` | 可随时从真相源**全量重建**，删了不丢原文/知识（但会失去检索物化与审计凭证） | | **代码层** | `scripts/`、`web/` | 改它**不改数据**——除非它正是负责生成"数据契约"的那部分（见 B） | 核心保障：内容是人可读的 markdown，不像向量 RAG 那样把知识锁进"换模型就得全量重算"的黑盒。 @@ -575,15 +586,17 @@ export function MyButton() { v0.1（CLAUDE.md + Skill + CLI + Git hook）+ v0.2（Web 管理台 + i18n）已构成完整可用的知识库。以下是预留的演进方向，**触发条件出现前不要预先建设**——避免空写未用的代码与抽象。 -### v0.3 — `k.py` 加 SQLite + FTS5 索引层 +### v0.3 — SQLite + FTS5 索引层（raw 自然单元已实现；wiki 页面索引按需） + +raw 长文档的自然单元证据索引已作为准确性基础设施落地：`.cache/retrieval_index.db`、`rebuild-evidence-index`、`evidence-index-coverage`、`search-evidence`。它对 paragraph/list item/table row/blockquote/code/figure 做 FTS5 unicode61 + trigram + exact + RRF，全文 SHA 变化时查询 fail-closed；不调用 LLM、不使用 embedding。 -**触发条件**：`wiki/` 页面数 > ~1000 且 `k.py search` / `list-orphans` / `list-conflicts` 等命令延迟可感知（>2 秒）。 +尚未实现的是通用 `wiki/` 页面/关系增量索引。其触发条件仍为：`wiki/` 页面数 > ~1000 且 `k.py search` / `list-orphans` / `list-conflicts` 等命令延迟可感知（>2 秒）。 **实施提示**： -- 索引存 `.cache/index.db`（已加入 `.gitignore`，纯派生数据） +- wiki 页面索引将另存 `.cache/index.db`（纯派生数据，不与已落地的 raw 证据索引混淆） - 表设计：`pages`（路径 / 元数据 / content_hash）、`pages_fts`（FTS5 + jieba 中文预分词）、`links`（双向链接邻接表）、`tags` - `watchdog` 监听 `wiki/` 文件变化做增量更新；hash 比对决定是否重新解析 -- `k.py` 检测到 `.cache/index.db` 则走索引，否则 fallback 到现在的纯 Python 全文件扫描——**保持向后兼容** +- `k.py` 检测到 `.cache/index.db` 则让 wiki 页面查询走索引，否则 fallback 到现在的纯 Python 全文件扫描；raw 原文细节统一走已落地的 `search-evidence` - 删 `.cache/` 后启动时自动从 `wiki/` 全量重建。**markdown 仍是唯一真相源** ### ~~v0.4 — 冲突工作台高级处理 + 类型化关系图谱~~（均已实现） diff --git a/CLAUDE.md b/CLAUDE.md index 8ed756f..a250953 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -20,12 +20,12 @@ ## 核心设计原则（不可违反） -1. **知识库不调用 LLM**。所有 LLM 推理由外部 agent 完成；知识库本身只暴露 MCP 工具与 REST API，不内嵌任何 agent runtime、LLM SDK 或对话能力。 +1. **知识库不调用 LLM**。所有 LLM 推理由外部 agent 完成；知识库本身只暴露 CLI 与 REST API，不内嵌任何 agent runtime、LLM SDK 或对话能力。 - **本条原则的范围**：`scripts/`、`web/`（KB 核心）严禁内嵌 LLM SDK。 - **例外**：`tools/debug-console/` 是独立子项目，**作为 KB 的外部客户端存在**，可以引入 LLM SDK。它只通过 HTTP 调主 `web/` 的 REST API（`/api/agent-tool` 等），不直接读 markdown / `.cache/`。删掉 `tools/` 整个目录不影响 KB 任何功能。 -2. **markdown + Git 是唯一真相源**。SQLite 索引（`.cache/index.db`）是派生层，可随时从 markdown 全量重建。删 `.cache/` 系统仍能跑。 -3. **完整页面优先**。所有读取工具返回**完整页面或完整 H2/H3 段**，绝不返回 chunk。 -4. **严禁 embedding 召回**。embedding 模型 / 向量存储 / 文档切片不出现在系统的任何"找相关内容"逻辑中。检索靠 BM25 全文 + 元数据过滤 + agent 阅读完整页面。 +2. **markdown + Git 是唯一真相源**。SQLite 证据索引（`.cache/retrieval_index.db`）是派生层，可随时从 markdown 全量重建。删 `.cache/` 只会失去检索加速/路由凭证，原文与 wiki 仍在。机械 100% 覆盖的分母是**当前已转换 markdown 的 parser inventory**；它不自动证明 PDF/DOCX→markdown 转换保真，原始载体仍需绑定 source SHA +（实际后端版本 + 实现文件 SHA）converter fingerprint 的 conversion receipt 与视觉抽检。 +3. **自然单元检索，完整块阅读**。检索坐标可以是 paragraph、单条 list item、单条 table row、blockquote、code、figure 等自然单元；它们不是固定长度 chunk。作语义判断或引用前必须读取其完整 canonical parent block 或完整 H2/H3 段。 +4. **严禁 embedding 召回**。embedding 模型 / 向量存储 / 固定长度文档切片不出现在系统的任何“找相关内容”逻辑中。检索靠 FTS5/BM25 + trigram/exact + 元数据/结构导航 + agent 阅读完整原文块。 5. **写权限硬约束**：写 `raw/**`、`my_thoughts/**`、含 `#human-only` 标签或 `locked: true` frontmatter 的文件 → 工具直接拒绝（PermissionError），不是 ask、不是 warn、是 deny。 6. **删除即标记**：所有"删除"操作只能改 `status: deprecated`，绝不真删文件；历史信息有内在价值。 @@ -37,8 +37,8 @@ | 层级 | 入口 | 用途 | |---|---|---| -| **工作流** | Skill：`kb-ingest` / `kb-query` / `kb-lint` / `kb-conflict-resolve` / `kb-export` | 端到端的摄入 / 查询 / 周检 / 冲突 / 导出 | -| **原子操作** | `python scripts/k.py ` | outline / search / read-section / read-block / backlinks / outlinks / annotate-section / list-source-issues / list-broken-refs / list-to-update / list-orphans / list-conflicts / health 等 | +| **工作流** | Skill：`kb-ingest` / `kb-query` / `kb-lint` / `kb-cite-audit` / `kb-conflict-resolve` / `kb-export` / `kb-edit-source` | 端到端的摄入 / 查询 / 周检 / 引用审计 / 冲突 / 导出 / 来源编辑 | +| **原子操作** | `python scripts/k.py ` | rebuild-evidence-index / evidence-index-coverage / search-evidence / read-evidence-unit / outline / search / read-section / read-block / backlinks / outlinks / annotate-section / list-source-issues / list-broken-refs / list-to-update / list-orphans / list-conflicts / list-cite-mismatches / extract-claims / cite-audit-log / list-suspect-citations / search-raw / corpus-map / health 等 | | **直接读写** | `Read` / `Edit` / `Write` | `wiki/**` markdown 文件 | 下文「四大操作流程」中出现的 `read_page` / `archive_analysis` / `mark_conflict` / `append_log` / `git_commit` 等是**抽象动作名（接口契约）**，描述 agent 应做什么，**不是可调用的 MCP tool**。落地实现 = skill 文档 + `k.py` 子命令 + `web/lib/operations.ts` 的 server action。 @@ -77,44 +77,6 @@ groundmap/ # 引擎根（通用代码 + 规范） --- -## 双仓库同步约定（dev ↔ release） - -本仓库（`AI知识库/`）与 `groundmap-release/` 是**两个独立 Git 仓库**承担不同角色： - -| 仓库 | 角色 | 数据 | 分支 | -|---|---|---|---| -| `AI知识库/`（本仓） | **开发版** | 含实际 wiki / raw / exports 数据 | `rag-evolution-ip-standard` 等 | -| `groundmap-release/` | **发布版** | 引擎 + 3 个精选示例 demo 库（仅 `wiki/` 随仓；`raw/`、`my_thoughts/` 不分发）+ 已审的发布准备改动 | `main` | - -**哪些修改必须双仓同步**（任一改完都需在另一仓做对应改动，否则下次 sync 漂移）： - -1. **`scripts/k.py`、`scripts/convert.py`、`scripts/section_parser.py`、其他通用引擎代码**： - 同步整个文件；release 的 workspace fallback 逻辑（k.py 第 2644-2671 行）保留不动。 -2. **`scripts/tests/`**（含 `TestMirrorSync` 守护）：**完全镜像**——dev 改了测试，release 必须改相同处。 -3. **`.claude/skills/kb-*/SKILL.md` 与 `.agents/skills/kb-*/SKILL.md`**：SKILL.md 内容**逐字相同**（`.claude ↔ .agents` 由 `TestMirrorSync.test_skills_mirror` 守），dev 与 release 之间靠人肉 / rsync 同步。 -4. **`web/`（Next.js 管理台）**：dev 与 release 同步主要 UI 改动；release 可能含更多发布准备（i18n key 整理、未发布特性等），合并时以 release 为基线。 -5. **`docs/`**（用户文档）：dev 写新内容 → 同步到 release；release 的发布准备改动（demo 视频、新手教程）一般不回 dev。 - -**哪些修改不要镜像到 release**： - -- dev 里**实际在用的**工作数据（`workspaces//raw/**`、私人 `my_thoughts/**`、`exports/**`）——不镜像到 release。release 自带的是另一套**精选 demo 库**：仅 `wiki/` 随仓分发，`raw/`、`my_thoughts/` 不分发。 -- dev 专属的实验性 lint / 临时脚本。 - -**不变量清单**（任一变动都视作"破坏不变量"、必须同时同步）： - -- `RELATION_TYPES` 白名单 7 类（k.py ↔ web/lib/markdown.ts） -- `WIKILINK_RE` 正则（k.py ↔ web/lib/markdown.ts） -- `TestMirrorSync` 的归一化规则（CLAUDE.md ↔ AGENTS.md ↔ `.claude/skills ↔ .agents/skills`） -- 默认 workspace 解析（dev 默认 `smb-ecommerce`；release 不设固定默认——未指定时 CLI 自动选用存在的第一个 workspace，**故意不同**，反映发布清理意图） - -**`TestMirrorSync` 守护的镜像范围**（仅在单仓内）： - -- `CLAUDE.md` ↔ `AGENTS.md`（按归一化：Claude Code/Codex、CLAUDE.md/AGENTS.md、`.claude/skills/.agents/skills` 三组替换后必须一致） -- `.claude/skills/*/SKILL.md` ↔ `.agents/skills/*/SKILL.md`（同归一化） -- **dev 与 release 之间没有跨仓镜像测试**——必须靠"修改后手动 sync + 跑两侧 `pytest scripts/tests/`"保证 - ---- - ## 权限规则 | 路径 / 标记 | 权限 | @@ -291,9 +253,9 @@ tags: [] - **不依赖任何特定渲染器**——知识库的解析独立实现，与 Obsidian、Foam、Logseq 等的兼容是顺便的 - 链接目标可以是：完整页面 `[[wiki/concepts/transformer]]`、特定段 `[[wiki/concepts/transformer#注意力机制]]`、特定块 `[[raw/papers/smith2026#^p-12-7d8e9a]]` -无法提供精确来源时，必须显式标注 `[需要来源]`，**不得省略**或猜测。 +无法提供精确来源时，必须显式标注 `[需要来源]`，**不得省略**或猜测。数字为**跨块计算 / 单位换算**所得、被引原文无该字面时，标 `[KB 推算: ^依据锚]`（详见「引用语义审计规范」）。 -**粒度硬约束（lint 守门）**：整篇引用 `[[raw/X]]`（无 `#^`）**仅限**「来源绑定 / 纯背景介绍」；任何含数字 / 指标 / 结论的**实质论断必须**用块级 anchor `[[raw/X#^...]]`——整页引用去支撑论断属"引用粒度不足"。由 `python scripts/k.py list-coarse-citations` 扫出（接入 `health` 的 `coarse_citations_count`），与 `list-bare-claims`（有数字但无任何引用）互补：前者治"引得太粗"，后者治"没引"。块级 anchor 才能精确溯源、并在 web 端渲染为论文式 `[n]` 上标。 +**粒度硬约束（lint 守门）**：整篇引用 `[[raw/X]]`（无 `#^`）**仅限**「来源绑定 / 纯背景介绍」；任何含数字 / 指标 / 结论的**实质论断必须**用块级 anchor `[[raw/X#^...]]`——整页引用去支撑论断属"引用粒度不足"。由 `python scripts/k.py list-coarse-citations` 扫出（接入 `health` 的 `coarse_citations_count`），与 `list-bare-claims`（有数字但无任何引用）互补：前者治"引得太粗"，后者治"没引"。块级 anchor 才能精确溯源、被 `list-cite-mismatches` 语义核对、并在 web 端渲染为论文式 `[n]` 上标。 ### 关系类型语法（v0.4b 图谱） @@ -341,6 +303,54 @@ tags: [] --- +## 引用语义审计规范 + +结构 lint（broken-refs / bare-claims / coarse-citations）只保证引用「结构正确」；「被引块是否真的支撑论断」由三层机制保证——**KB 只出确定性数据，语义判定由外部 agent 完成，终审在人**： + +1. **确定性核对**：`python scripts/k.py list-cite-mismatches`——论断中的数字 / 逐字引文必须出现在被引块原文中（数值按论断声明精度做舍入容差匹配；约数标记放宽）。`mismatch` / `exempt-missing-basis` / `canonical-anchor-mismatch` / `canonical-target-mismatch` 是**闸门项**（接入 `health` 的 `cite_mismatches_count`）；accuracy-first ingest 还要求 `imprecise-anchor` 与 `unverifiable` 为空。引用依赖 hash recovery、`./raw/...` 等非 canonical handle 时必须先改写再复验；缺失文件只使该引用 `unverifiable`，同块其他可用引用仍继续核对，且 anchor-missing 不得遮蔽另一可用引用的 mismatch。`[KB 推算: ^依据锚]` 只豁免紧邻的单个值，依据锚必须在同一核对单元实际被引。**语义边界：数字共现 ≠ 语义支撑**，本层归零不构成来源正确性证明，曲解 / 过度概括靠下面两层。 +2. **写入时回验**（writer/verifier 分离）：kb-ingest 第 9.5 步——`k.py extract-claims --changed` 确定性枚举本次（论断, 引用）对，**fresh-context 子 agent**（不共享写作上下文）逐条判 `SUPPORTED / PARTIAL / UNSUPPORTED / CONTRADICTED`；UNSUPPORTED / CONTRADICTED 清零才可 commit，且**禁止**用「删引用、挂 `[需要来源]`」洗白；log.md 记 citation-verify 对账行，事后可 `extract-claims --commit ` 复枚举对账。 +3. **存量审计**：`kb-cite-audit` skill（kb-lint 第 5b 步按周配额调用：`extract-claims --unaudited-only --sample 20 --seed `，同 seed 取样可复现、跨周覆盖累积）——只审「从未审过 + 内容漂移」的对；^h- 目标对**整节正文**算内容 hash，确定性补上「锚点只 hash 标题、正文重写不被察觉」的盲区。 + +**`[KB 推算]` 标记**（与 `[需要来源]` 平级的内联标记）：数字为跨块计算 / 单位换算所得、被引原文无该字面时必标，且**必须带依据锚**（如 `[KB 推算: ^t-33-0c8446]`）；裸 `[KB 推算]` 本身是闸门项（exempt-missing-basis）。豁免只关数字核对，不改「块必须有引用」的语义；豁免块数进 `health`（`cite_exempted_count`），kb-lint 周检抽查真伪。 + +**CAUTION 审计标注**（审计未通过的唯一 markdown 落点；与「知识更新冲突」WARNING 分工明确——错引是「论断 vs 其引用」，不是新旧证据冲突，不走冲突决议四选项）： + +```markdown +> [!CAUTION] 引用审计未通过 — YYYY-MM-DD +> **论断**：…（块 ^p-4-34d5b1） +> **被引块**：[[raw/papers/X#^t-77-9e8336]] +> **审计判定**：UNSUPPORTED — 被引表格中数字为 65.9 非 66.9 +> **建议**：改引 [[raw/papers/X#^t-79-…]] 或修正论断数字 +> **状态**：⏳ 待人类判别 +``` + +`python scripts/k.py list-suspect-citations` 从 markdown 扫此标注（待办清单不依赖 .cache，删台账不丢待办；接入 `health` 的 `suspect_citations_count`）。人类处理 = 修论断 / 换锚后删标注（内容变化自动触发重审），或确认误报时把标注改写为一行 `> [!NOTE] 引用审计误报（YYYY-MM-DD 由人类复核通过）`（删除即标记，误报裁决留在真相源）。 + +**验证台账**（`workspaces//.cache/citation_audit.jsonl`，gitignored）：纯派生层——只记「谁在何时核验过什么」（agent 劳动的 memoization），删了唯一后果是全部回到未审、重审即重建。写入走 `k.py cite-audit-log`（`--mode audit|ingest|manual`、`--by agent|human`）；受控校验：pair 过期拒绝、目标可解析时记 UNVERIFIABLE 拒绝、**agent 记 SUPPORTED 必须附 `--evidence`（被引块现行原文字面子串，k.py 校验）**——强制取回原文，堵橡皮图章。台账 ↔ markdown 一致性由 `list-suspect-citations --check-ledger` 对账（堵「删标注蒸发」）。**知识状态（标注、论断修正、误报裁决）落 markdown 进 git；台账绝不入库。** + +**盲填复核（blind cloze，二次 LLM 审核的推荐形态）**：`extract-claims --cloze` 把论断里的可核对数字挖成 ⟦N1⟧ 占位；核验者**只看「挖空论断 + 被引原文」填空、全程看不到期望值**（从原理上消灭判定式审核的附和偏差，并抓「数字巧合在场但归属错误」）；填回值经 `k.py cloze-check` 按数值容差**机器判分**（块级 union：一块多引用时数字由块内任一引用的原文填出即可）。判分未过按 UNSUPPORTED 处置。 + +**查询侧核对（check-draft）**：查询答案 / 导出稿等「不在 wiki 里的文本」先跑 `k.py check-draft <草稿文件>` 的确定性层，再用 `tools/cite-audit --draft` 对完整核验包做盲填 + 反驳并入账，最后跑 `k.py check-draft <草稿文件> --strict`。Strict 对 broken / non-canonical / raw 不可得 / imprecise / pending / bare / coarse / 未映射定性事实 / 无当前 provenance / 非 `SUPPORTED` / 核验包截断全部 fail-closed；实质性 KB 答案必跑（kb-query 第 7.5 步）。 + +**跨模型二审（tools/cite-audit）**：`python tools/cite-audit/audit.py --workspace [--changed|--unaudited-only|--all --sample N --seed |--draft <草稿>]`——外部客户端（与 debug-console 同属原则 1 的 tools/ 例外区，只经 k.py CLI 取数 / 回写）调 DeepSeek 跑「完整 evidence 盲填 + 反驳」双通道并自动入台账。结果协议：exit 0 = 全部判定通过，1 = 完成且有语义未通过，2 = incomplete / skipped / 网络 / 协议 / 台账故障；1/2 均不得当成绿灯。API key 走环境变量 `DEEPSEEK_API_KEY`，不硬编码。 + +**原子论断分解（核对单元）**：list 块按**条目**拆分核对与枚举——条目是自然的原子论断边界，每条目对自己的引用负责（条目无引用回退整块引用并集），抓「条目 A 的数字只在条目 B 的引用目标里」这类块级 union 盲区；审计对的 `claim_text` 也随之聚焦，核验者不必自行推断归属。 + +**检索凭证链（check-provenance）**：`read-block` / `read-section` / `blocks` / `extract-claims --with-evidence` 自动向 `.cache/retrieval_log.jsonl` 登记「何时取回过哪个块 / 节的哪个内容版本」（派生层：删了 = 凭证重置，重新取回即可）；`k.py check-provenance --changed` 校验每条新引用都有**匹配当前内容 hash** 的取回凭证——没真读过（或读的是旧版本）就造不出凭证，把 quote-first 从行为规范升级为可机器校验的溯源链（ingest 提交前闸门之一）。 + +**审计统计保证（audit-confidence）**：`k.py audit-confidence` 把「审过多少、发现多少」换算成验收抽样口径的陈述——已审引用未通过率的 Clopper-Pearson 置信上界（纯数学，零 LLM）。诚实边界随结果携带：上界对已审子集精确成立，推广到全库要求样本有代表性（seed 抽样 / 全覆盖）；且以验证器判定为准，验证器自身查全率由 evals/cite-check 度量。 + +**对抗评测集（evals/cite-check）**：给防线本身立尺子——已知错引类型的毒化案例 + 干净对照，`python evals/cite-check/run_eval.py` 度量确定性层查全 / 误报（pytest `test_cite_eval.py` 钉死为回归闸门），`--semantic` 加测语义层（DeepSeek 双通道）。新增错引类型时先加 case：哪层抓不住，哪里就是下一道要补的防线。 + +**长文 / 回答 / 原始载体分层评测（evals Stage-2）**：`python evals/run_stage2.py` 统一运行 external-holdout 协议 public smoke、canonical atomic claim→citation / abstention、PDF/DOCX/HTML 源格式保真三条闸门。三者必须分别报告 recall/coverage/precision/abstention/forbidden selection 与明确分母，不能合成一个平均分；公开结果终态固定 `hidden_certification: false`，不得冒充独立隐藏认证。修改 `scripts/convert.py` / `scripts/pdf_layout.py` / 自然单元检索或评测 scorer 时，必须运行对应定向 pytest 和 Stage-2；协议错误、缺依赖、零分母、Gold/hash 漂移均 fail-closed。 + +**PDF accuracy-first 转换**：PDF 强制走几何 layout-aware 路径，恢复 ruled/保守 borderless table、跨栏标题和 column-major 阅读顺序；不得在失败后静默回退到内容流顺序。扫描页、内容级光栅/矢量图、任一非已知 bullet 的无法解析 CID 字形、损坏/加密等无法可靠表示的来源必须明确失败并先走 OCR/图表专用流程；双栏/键值几何无高置信区分信号时也必须 fail-closed。独立图片与音频扩展名仅用于让批处理识别并明确拒绝：ExifTool 元数据不能代替视觉内容，现有音频路径又依赖 ambient ffmpeg/ffprobe 与无不可变模型 revision 的 Google SpeechRecognition，因此在专用 OCR/视觉或版本锁定转写流程产生可验 receipt 前必须 fail-closed。每份可转换的非 Markdown 来源先复制到只读快照后转换，派生 outline 用 canonical `source_path` + source SHA + converter fingerprint + receipt 自校验哈希锁定原载体与转换语义；同目标/大小写碰撞、派生 Markdown 二次处理、孤儿原载体、symlink、转换/写入期间原文变动、receipt 缺失/过期均 fail-closed。证据索引 schema v5 同时冻结这组 source binding，并在 rebuild、coverage、search、read 时重验；原 PDF/DOCX 被改、删、移动或转换实现/依赖漂移后，旧派生内容不得继续召回。只读位与事后 SHA 用于发现**受信任本地转换器**的意外写入和源文件竞态，不是恶意转换器的安全沙箱；具有同等本机权限的代码能改权限并在校验前恢复字节，所以只允许项目锁定/经审阅的 converter，不受信任的插件必须放入 OS 级沙箱。DOCX 的空表头只在“生成 header 全空 + 第一数据行每格均全粗体”的高置信形态下提升，禁止一般性猜表头。 + +**pre-commit 机械闸门**：staged 改动含 `workspaces//wiki/**.md` 时，hook 把 Git index 物化为隔离临时树，挂载本地 raw 并复制审计 / 检索台账快照后，对 staged 页强制：cite mismatch/imprecise/unverifiable/canonical 问题为 0，broken/bare/coarse/unmapped/source issue 为 0，每个 pair `target_status=ok` + `audited=true` + `last_verdict=SUPPORTED`，`check-provenance --paths` findings 为 0。Python / 临时树 / checker / 退出码 / JSON / schema 任一异常均 fail-closed；范围只限 staged 页，人类可 `--no-verify` 显式绕过并以 `human:` 说明。 + +--- + + ## 四大操作流程 > 所有操作均通过外部 agent 调用知识库的 CLI / REST 工具完成。知识库本身不驱动这些流程，也不内嵌 LLM 调用。 @@ -351,21 +361,22 @@ tags: [] 1. 用户将原始文件放入 `raw/` 对应子目录（agent 不得修改原始文件） 2. agent 调 `python scripts/convert.py`：把原始格式转为 markdown，自动加锚点（`^h-`/`^p-`/`^t-`/`^c-`/`^f-`），并生成 `.outline.json` -3. agent 调 `python scripts/k.py outline ` 看大纲，**按字符数三档自决阅读策略**： +3. agent 调 `rebuild-evidence-index` 后跑 `evidence-index-coverage`：全部 raw 转换后 markdown 的自然单元、内容/结构章节机械覆盖必须为 100%，完整 inventory 指纹与 exact text hash 对账、异常空章节逐项确认或修复，且 raw 全文 SHA 与已验证 outline 摘要指纹均新鲜；这是“已转换文本中的所有细节可发现”的结构闸门，不等于原始载体转换 100% 保真、AI 已理解或摘要无遗漏 +4. agent 调 `python scripts/k.py outline ` 看大纲，**按字符数三档自决阅读策略**： - **① 短文** `< 30000`（约 3 万中文字）：`Read` 全文 - - **② 中长文** `30000 – 150000`（论文 / 报告级）：按 H1 切块、每块 ≤ 3 万分段 `read-section` - - **③ 整本书规模** `> 150000`：TOC 扫全 + AI 自决深读章节，**全部章节登记**到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过）。⊙ 扫读章节保留 partial re-ingest 升级路径 + - **② 中长文** `30000 – 150000`（论文 / 报告级）：按标题树选 ≤ 3 万字的完整节 `read-section`，超长 H1/H2 继续下钻；无子标题的超长节按自然块阅读 + - **③ 整本书规模** `> 150000`：TOC 扫全 + AI 自决深读章节，**全部章节登记**到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过；首列写真实 `^h-` anchor、原标题不许意译、⊙ 行备注点名关键实体——`search-raw` 的 deepen 联动依赖此规范）。⊙ 扫读章节保留 partial re-ingest 升级路径 - 单次 Read 严格 ≤ 3 万中文字符，避免 LLM "lost in the middle" 衰减 -4. **每读完一个 H2/H3 章节，立即调 `k.py annotate-section "<一两句摘要>"` 回填精排摘要** —— **②③ 档（分段阅读）的必经步骤**；① 档短文一次 Read 全文，不强制分段回填（建议至少给主要 H2 回填一句摘要，非硬性） -5. agent **基于 wiki 现状做综合判断**：调 `k.py search` 反查相关 wiki 页，轻量阅读，自决三件事——**核心价值**（新东西在哪）/ **关联**（哪些 wiki 页有重叠）/ **冲突**（哪些论断打架） -6. agent 基于第 5 步综合**决定写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated -7. agent 在 `wiki/sources/` 创建摘要页（标准 frontmatter + 块级引用 + 「## AI 综合判断」H2 节固化第 5 步结论 + 第 ③ 档的「## 章节深度登记」表） -8. agent 立即更新最核心的 2-3 个节点页面 -9. agent 给其余受影响页面打 `#to-be-updated` 标签 -10. agent **自动决定 MOC 归属**：用 source_summary 的 tags 反查现有 MOC，命中则在「近期更新」节追加；无命中则用 `_templates/index_template.md` 自动新建 MOC + 在 root_index 加入口 -11. agent **图谱接入与校验**：写互链时凡关系属标准类型（支持/反驳/延伸/属于/组成/替代/引用）即用 `[[目标|SUPPORTS]]` 等标准关系类型（白名单见「关系类型语法（v0.4b 图谱）」节）让图谱可按边染色；收尾跑 `python scripts/k.py list-relation-issues`（须为空）与 `python scripts/k.py graph`（确认本次新页面已作为节点接入、边正常、无意外孤立节点——孤立即回第 8 步补 `[[...]]` 互链）。图谱是**派生层**（从 wiki 双链实时计算、无持久文件，对齐「markdown 是唯一真相源」），本步只校验、不产出需提交的文件 -12. agent 追加 log.md 条目 -13. **提交前质量闸门**（须全过，不过则补齐再提交）：`python scripts/k.py list-bare-claims` / `list-coarse-citations` / `list-source-issues` / `list-broken-refs` / `list-relation-issues` 须全空——裸论断补块级引用、整页引用升块级、缺 source 补全、失效引用修掉、非法关系词改正。通过后 agent `git commit -m "ingest: <来源标题>"` 原子提交（只含 `wiki/**` 改动 + `log.md`；`raw/` 及其派生 .md / .outline.json 默认被 `.gitignore` 排除、留在本地，不入库——版权与隐私原因） +5. **每读完一个 H2/H3 章节，立即调 `k.py annotate-section "<一两句摘要>"` 回填精排摘要** —— **②③ 档（分段阅读）的必经步骤**；① 档短文一次 Read 全文，不强制分段回填（建议至少给主要 H2 回填一句摘要，非硬性）。最后一次回填完成后必须再次运行 `rebuild-evidence-index` + `evidence-index-coverage`，确认摘要路由指纹新鲜且覆盖闸门仍为 100% +6. agent **基于 wiki 现状做综合判断**：调 `k.py search` 反查相关 wiki 页，轻量阅读，自决三件事——**核心价值**（新东西在哪）/ **关联**（哪些 wiki 页有重叠）/ **冲突**（哪些论断打架） +7. agent 基于第 6 步综合**决定写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated +8. agent 在 `wiki/sources/` 创建摘要页（标准 frontmatter + 块级引用 + 「## AI 综合判断」H2 节固化第 6 步结论 + 第 ③ 档的「## 章节深度登记」表） +9. agent 立即更新最核心的 2-3 个节点页面 +10. agent 给其余受影响页面打 `#to-be-updated` 标签 +11. agent **自动决定 MOC 归属**：用 source_summary 的 tags 反查现有 MOC，命中则在「近期更新」节追加；无命中则用 `_templates/index_template.md` 自动新建 MOC + 在 root_index 加入口 +12. agent **图谱接入与校验**：写互链时凡关系属标准类型（支持/反驳/延伸/属于/组成/替代/引用）即用 `[[目标|SUPPORTS]]` 等标准关系类型（白名单见「关系类型语法（v0.4b 图谱）」节）让图谱可按边染色；收尾跑 `python scripts/k.py list-relation-issues`（须为空）与 `python scripts/k.py graph`（确认本次新页面已作为节点接入、边正常、无意外孤立节点——孤立即回第 9 步补 `[[...]]` 互链）。图谱是**派生层**（从 wiki 双链实时计算、无持久文件，对齐「markdown 是唯一真相源」），本步只校验、不产出需提交的文件 +13. agent 追加 log.md 条目 +14. **提交前质量闸门**（须全过，不过则补齐再提交）：`list-bare-claims` / `list-coarse-citations` / `list-unmapped-claims` / `list-source-issues` / `list-broken-refs` / `list-relation-issues` 须全空，`list-cite-mismatches` 的 mismatch / exempt-missing-basis / canonical-* / imprecise / unverifiable 须全空。另须完成**引用回验**（writer/verifier 分离）：`extract-claims --changed --with-evidence --max-evidence-chars 1000000000` 枚举完整（论断, 引用）对，fresh-context 子 agent 与跨模型审计完成盲填 + 反驳，incomplete / skipped / UNSUPPORTED / CONTRADICTED 清零，每对经 `cite-audit-log --mode ingest` 入账且有当前 provenance，log.md 含 citation-verify 对账行（详见 `.claude/skills/kb-ingest/SKILL.md` 第 9.5 步）。通过后 agent `git commit -m "ingest: <来源标题>"` 原子提交（只含 `wiki/**` 改动 + `log.md`；`raw/` 及其派生 .md / .outline.json 默认被 `.gitignore` 排除、留在本地，不入库） > **partial re-ingest（增量深化）**：第 ③ 档扫读 / 跳过的章节保留升级路径，由 kb-query / kb-lint / 用户 web 端三种方式触发深化。AI 自动重读该章节 → 更新现有 source_summary（不新建）→ 章节登记表 ⊙ → ✓ → log.md 记 `partial-ingest` 类型 → git commit。 @@ -401,14 +412,14 @@ skill 设计了 4 个深度模式，但**Claude Code 中默认且唯一行为是 3 个高级模式的执行规范详见 `.claude/skills/kb-query/SKILL.md` 第 7a/b/c 步。它们是**设计契约**——为产品化时的 agent 提供明确的"该模式下做什么"指令，不在 Claude Code 默认行为内。 -> **细节下钻判据（所有模式共有，quick 也执行——与上面 3 个产品化预留模式不同）**：回答前若某论断需要**原文级精确**（精确引文 / 数字 / 日期 / 条款，或来源为第 ③ 档长文档），agent 必须先 `read-block` / `read-section` 打开对应 anchor 核验原文再下结论——保证"真正需要原文细节时必然回查"、不靠 agent 自觉；A 类常见问题（定义 / 概览，detail 已蒸馏进 wiki）不命中、不额外费 token。audit 是它的"全量强化版"（对答案里**每条**引用都回查）。详见 `.claude/skills/kb-query/SKILL.md` 第 4.6 步。 +> **细节下钻判据（所有模式共有，quick 也执行——与上面 3 个产品化预留模式不同）**：回答前若某论断需要**原文级精确**（精确引文 / 数字 / 日期 / 条款，或来源为第 ③ 档长文档），agent 必须先 `read-block` / `read-section` 打开对应 anchor 核验原文再下结论。细节不在 wiki 蒸馏层时，先确认 `evidence-index-coverage` 为 100% 且 fresh，再把问题拆成 required facets，用 `search-evidence` 对原问题与显式中英别名做 top-20 自然单元召回；multi-hop 后续查询只能使用第一跳实际读到的新实体。拟选候选先用 `read-evidence-unit ` 锁定精确行/条目，再 `read-block` 核对父块上下文与主体/指标/值/单位/条件/否定。当前 Markdown citation 对 table/list 仍是父块 anchor，row/item 精确身份在派生 unit handle 中，不能把父表另一行当成支撑。概念性/改述性问题同时保留浏览路（`corpus-map → outline → read-section`）；旧 `search-raw` 只作明确披露的扫描回退。详见 `.claude/skills/kb-query/SKILL.md` 第 4.6-4.7 步。 ### Lint 操作流程 1. agent 调 `list_to_update` → 逐个处理 `#to-be-updated` 积压 2. agent 调 `list_orphans` → 处理孤儿页面（无入链） 3. agent 调 `list_conflicts` → 复核所有冲突标注 -4. agent 抽查 wiki 论断与 raw 来源的一致性（fact-check） +4. agent 做引用语义审计（`kb-cite-audit` 周配额模式；先 `list-cite-mismatches` 确定性分诊，再 `extract-claims --unaudited-only --sample 20 --seed ` 逐条 fresh-context 判定入台账，UNSUPPORTED 落 CAUTION 标注——详见「引用语义审计规范」） 5. agent 检查缺少独立页面的重要概念 6. agent 调 `archive_analysis` 生成 `wiki/analyses/周报-YYYY-WXX.md` 7. agent 调 `append_log` @@ -530,7 +541,7 @@ export function MyButton() { | 层 | 内容 | 改它时 | |---|---|---| | **真相源** | `wiki/**.md`（agent 维护的知识）、`raw/` 原始文件 | 纯 markdown，不依赖任何代码即可读懂 / diff；删光代码内容仍在 | -| **派生层** | `.cache/index.db`、`raw/**.outline.json` | 可随时从真相源**全量重建**，删了不丢信息 | +| **派生层** | `.cache/retrieval_index.db`、未来的 `.cache/index.db`、`raw/**.outline.json` | 可随时从真相源**全量重建**，删了不丢原文/知识（但会失去检索物化与审计凭证） | | **代码层** | `scripts/`、`web/` | 改它**不改数据**——除非它正是负责生成"数据契约"的那部分（见 B） | 核心保障：内容是人可读的 markdown，不像向量 RAG 那样把知识锁进"换模型就得全量重算"的黑盒。 @@ -575,15 +586,17 @@ export function MyButton() { v0.1（CLAUDE.md + Skill + CLI + Git hook）+ v0.2（Web 管理台 + i18n）已构成完整可用的知识库。以下是预留的演进方向，**触发条件出现前不要预先建设**——避免空写未用的代码与抽象。 -### v0.3 — `k.py` 加 SQLite + FTS5 索引层 +### v0.3 — SQLite + FTS5 索引层（raw 自然单元已实现；wiki 页面索引按需） + +raw 长文档的自然单元证据索引已作为准确性基础设施落地：`.cache/retrieval_index.db`、`rebuild-evidence-index`、`evidence-index-coverage`、`search-evidence`。它对 paragraph/list item/table row/blockquote/code/figure 做 FTS5 unicode61 + trigram + exact + RRF，全文 SHA 变化时查询 fail-closed；不调用 LLM、不使用 embedding。 -**触发条件**：`wiki/` 页面数 > ~1000 且 `k.py search` / `list-orphans` / `list-conflicts` 等命令延迟可感知（>2 秒）。 +尚未实现的是通用 `wiki/` 页面/关系增量索引。其触发条件仍为：`wiki/` 页面数 > ~1000 且 `k.py search` / `list-orphans` / `list-conflicts` 等命令延迟可感知（>2 秒）。 **实施提示**： -- 索引存 `.cache/index.db`（已加入 `.gitignore`，纯派生数据） +- wiki 页面索引将另存 `.cache/index.db`（纯派生数据，不与已落地的 raw 证据索引混淆） - 表设计：`pages`（路径 / 元数据 / content_hash）、`pages_fts`（FTS5 + jieba 中文预分词）、`links`（双向链接邻接表）、`tags` - `watchdog` 监听 `wiki/` 文件变化做增量更新；hash 比对决定是否重新解析 -- `k.py` 检测到 `.cache/index.db` 则走索引，否则 fallback 到现在的纯 Python 全文件扫描——**保持向后兼容** +- `k.py` 检测到 `.cache/index.db` 则让 wiki 页面查询走索引，否则 fallback 到现在的纯 Python 全文件扫描；raw 原文细节统一走已落地的 `search-evidence` - 删 `.cache/` 后启动时自动从 `wiki/` 全量重建。**markdown 仍是唯一真相源** ### ~~v0.4 — 冲突工作台高级处理 + 类型化关系图谱~~（均已实现） diff --git "a/GroundMap-\350\256\276\350\256\241\346\226\207\346\241\243.md" "b/GroundMap-\350\256\276\350\256\241\346\226\207\346\241\243.md" index f7d43a5..2ed4d51 100644 --- "a/GroundMap-\350\256\276\350\256\241\346\226\207\346\241\243.md" +++ "b/GroundMap-\350\256\276\350\256\241\346\226\207\346\241\243.md" @@ -89,7 +89,7 @@ Wiki 是一个持久的、复利增长的产出物。交叉引用已经建好。 - **Raw 层**：你策源的文档集合。**不可变** —— agent 只读不改。这是事实来源（Source of Truth）。 - **Wiki 层**：agent 生成的 Markdown 文件目录。摘要、实体页、概念页、对比、综述。Agent 完全拥有这一层（人类可干预、覆盖）。 - **Schema 层**：CLAUDE.md，告诉外部 agent 如何摄入、查询、维护 Wiki。 -- **工具层**：`scripts/k.py` CLI + `.claude/skills`（/`.agents/skills`）工作流 + `web/`（Next.js）的 REST route handler 与 server action（其中可选挂 SQLite/FTS5 派生索引，v0.3 触发条件未到时不建）。**这一层不调 LLM**——它只暴露能力，由外部 agent 调度。原计划的 MCP Server 已废弃（见 §10.5），文中所有 `read_*` / `mark_conflict` 等是抽象动作名而非可调用 tool。 +- **工具层**：`scripts/k.py` CLI + `.claude/skills`（/`.agents/skills`）工作流 + `web/`（Next.js）的 REST route handler 与 server action。raw 长文档已有可删重建的 SQLite/FTS5 自然单元证据索引；wiki 页面/关系仍按需纯扫描。**这一层不调 LLM**——它只暴露能力，由外部 agent 调度。原计划的 MCP Server 已废弃（见 §10.5），文中所有 `read_*` / `mark_conflict` 等是抽象动作名而非可调用 tool。 ### 2.2 推荐目录结构 @@ -98,7 +98,7 @@ Wiki 是一个持久的、复利增长的产出物。交叉引用已经建好。 实际结构（引擎代码通用 + 数据按 workspace 隔离）： ``` -AI知识库/ # 引擎根（通用代码 + 规范） +groundmap/ # 引擎根（通用代码 + 规范） ├── CLAUDE.md # Schema：行为规范（唯一真相源） ├── GroundMap-设计文档.md # 本文档 ├── scripts/ # 自动化脚本（k.py、convert.py）——通用 @@ -178,19 +178,20 @@ KB_ROOT=~/work/项目B/kb-data python ~/tools/groundmap/scripts/k.py --workspace 用户将新来源放入 `raw/` 后，告知 agent 处理。**全流程 agent 自决，不询问用户**。概念骨架： -1. agent 用 `Bash: python scripts/convert.py` 把原始格式（pdf/docx/...）转为带锚点的 markdown + outline.json -2. agent 用 `Read` / `python scripts/k.py outline / read-section` 阅读来源——按字符数三档自决：① 短文 < 3 万 一次读完；② 中长文 3-15 万按 H1 切块每块 ≤ 3 万分段读；③ 整本书 > 15 万 TOC 扫全 + AI 选读 + 全部章节登记到「章节深度登记」表（⊙ 扫读 / × 跳过的章节保留 partial re-ingest 升级路径） -3. agent 用 `python scripts/k.py search` 反查相关 wiki 页轻读，**AI 自决**综合判断：核心价值 / 关联 / 冲突——基于 wiki 现状而非孤立总结原文 -4. agent 基于第 3 步综合自决**写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated -5. agent 用 `Write` 在 `wiki/sources/` 创建摘要页（必含标准 frontmatter + 块级 anchor 引用 + 「## AI 综合判断」节 + 第 ③ 档「## 章节深度登记」表） -6. agent 用 `Edit` 更新最核心的 2-3 个节点页面 -7. agent 用 `Edit` 给其余受影响页面追加 `#to-be-updated` 标签（懒更新） -8. agent **自动判断 MOC 归属**：tags 匹配现有 MOC 则在「近期更新」节追加；无命中则自动新建 MOC + root_index 加入口 -9. agent 用 `Edit` 追加 `log.md` + `Bash: git commit -m "ingest: ..."` +1. agent 用 `Bash: python scripts/convert.py` 把原始格式（pdf/docx/...）转为带锚点的 markdown + outline.json。PDF 强制走 layout-aware 几何路径恢复表格、跨栏与阅读顺序；扫描页、内容级光栅/矢量图、CID/损坏等不可可靠转换情形明确失败，双栏/键值几何含糊时也拒绝猜测，整批返回非零；独立图片/音频扩展名可被识别但必须在 ExifTool 元数据或无版本 Google SpeechRecognition/ambient ffmpeg 路径运行前 fail-closed，改走专用可验 OCR/视觉/转写流程；可转换的非 Markdown 来源在只读快照上转换，outline 的 schema v2 receipt 用 canonical source path + source SHA +（实际后端版本 + 实现文件 SHA）converter fingerprint + 自校验哈希绑定原载体，转换前统一拒绝同目标碰撞、派生 Markdown 二次处理、孤儿来源与 symlink。只读位 + 事后 SHA 只是受信任本地 converter 的意外写入/竞态防护，不是恶意 converter 沙箱；未审阅代码须放入 OS 级隔离环境 +2. agent 运行 `rebuild-evidence-index` + `evidence-index-coverage`：转换后 markdown 的 paragraph/list item/table row/blockquote/code/figure 与内容/结构章节必须 100% 物化，构建时冻结的完整 inventory 指纹和 exact text hash 必须一致，且全文 SHA、已验证 outline 摘要指纹与 schema v5 原载体 source binding 均新鲜；原 PDF/DOCX 被改、删、移动或转换实现/依赖漂移时，coverage 必须变 stale，search/read 必须拒绝旧索引；异常空章节逐项确认。该分母不自动证明任意 PDF/DOCX→markdown 转换保真；公开格式闸门另由 `evals/conversion_fidelity/` 承担 +3. agent 用 `Read` / `python scripts/k.py outline / read-section` 阅读来源——按字符数三档自决：① 短文 < 3 万 一次读完；② 中长文 3-15 万按标题树选 ≤ 3 万字的完整节，超长节继续下钻或按自然块读；③ 整本书 > 15 万 TOC 扫全 + AI 选读 + 全部章节登记到「章节深度登记」表（⊙ 扫读 / × 跳过的章节保留 partial re-ingest 升级路径） +4. agent 用 `python scripts/k.py search` 反查相关 wiki 页轻读，**AI 自决**综合判断：核心价值 / 关联 / 冲突——基于 wiki 现状而非孤立总结原文 +5. agent 基于第 4 步综合自决**写作策略**：新建摘要页 + 更新哪几个核心 + 标哪几个 #to-be-updated +6. agent 用 `Write` 在 `wiki/sources/` 创建摘要页（必含标准 frontmatter + 块级 anchor 引用 + 「## AI 综合判断」节 + 第 ③ 档「## 章节深度登记」表） +7. agent 用 `Edit` 更新最核心的 2-3 个节点页面 +8. agent 用 `Edit` 给其余受影响页面追加 `#to-be-updated` 标签（懒更新） +9. agent **自动判断 MOC 归属**：tags 匹配现有 MOC 则在「近期更新」节追加；无命中则自动新建 MOC + root_index 加入口 +10. agent 用 `Edit` 追加 `log.md` + `Bash: git commit -m "ingest: ..."` > **懒更新机制**：一次 Ingest 可能影响 10-15 个页面，但不必全部当场更新。agent 只立即处理最核心的节点，其余标记 `#to-be-updated`，在闲时或 Lint 阶段批量处理。这降低了单次操作的复杂度和出错概率。 > -> **AI 自决 + 可纠错**：3、4、8 步都是 AI 自决而非询问用户——一次 ingest 不打断对话流。所有 AI 判断落到 source_summary 或 MOC 的具体节，可在 web 端审计 / 修改 / 用冲突工作台覆盖。mis-classification 由 lint 流程检测后人工纠正。 +> **AI 自决 + 可纠错**：4、5、9 步都是 AI 自决而非询问用户——一次 ingest 不打断对话流。所有 AI 判断落到 source_summary 或 MOC 的具体节，可在 web 端审计 / 修改 / 用冲突工作台覆盖。mis-classification 由 lint 流程检测后人工纠正。 > > **partial re-ingest（增量深化）**：第 ③ 档扫读章节由 kb-query / kb-lint / 用户 web 端三种方式之一触发深化升级——一次 ingest 不是终点，知识可按需深化。 @@ -279,7 +280,7 @@ python scripts/k.py list-pages --type=entity --modified-by=LLM --json \ → "所有由 LLM 生成但人类尚未 Review 的实体页面，按修改时间降序"。 -> **不依赖 Dataview 等插件**——查询能力由 `scripts/k.py` 内置（v0.3 起会加 SQLite/FTS5 索引层，命令接口不变）。 +> **不依赖 Dataview 等插件**——查询能力由 `scripts/k.py` 内置。raw 长文档细节已用可重建的 SQLite/FTS5 自然单元索引；wiki 元数据查询目前仍扫 Markdown，将来可在命令接口不变的前提下切换索引层。 --- @@ -412,9 +413,17 @@ page_count: 47 可用命令快速查看：`grep "^## \[" log.md | tail -10` -### 7.2 SQLite 索引（派生层 — v0.3 演进项，**触发条件未到不预先建设**） +### 7.2 SQLite 索引（派生层 — raw 证据索引已落地） -> 当前 `scripts/k.py` 走纯 Python 全文件扫描，1k 页内秒级返回。当 wiki 页面数 > ~1000 且 `k.py search` / `list-orphans` 等命令延迟 > 2 秒时启用本节方案。 +`.cache/retrieval_index.db` 已作为长文档准确性基础设施落地。`rebuild-evidence-index` 对全部转换后 raw markdown 建自然单元 inventory，`evidence-index-coverage` 对自然单元的归一化/精确文本哈希、构建时冻结的完整单位与章节 inventory 指纹、完整路由元数据、内容/结构章节、两个 FTS5 的规范 DDL/列序/tokenizer、内容行与倒排内部 `quick_check`、raw 全文 SHA 与已验证 outline 摘要指纹 fail-closed 核对；`search-evidence` 用 unicode61 BM25、trigram、exact 与低权重结构路由经 RRF 融合，`read-evidence-unit` 保留 table row/list item 的精确 unit handle。Markdown table/list 引用仍是父块 anchor，这是必须披露的粒度边界。 + +### 7.3 分层准确性评测（公开闸门） + +`python evals/run_stage2.py` 统一运行三条互不替代的防线：外部 holdout bundle 协议（自然单元/章节完整性、CES@20、facet top-1、forbidden selector 与机械切片）；answer→canonical atomic claim→exact citation / abstention；PDF/DOCX/HTML 源坐标保真（事实、限定、真实表格、标题 inventory、独立文本、列表、顺序和脚注关系）。runner 严格校验应有指标集合、分子/分母/value、阈值方向、schema 与结构化终态，协议/依赖错误不能冒充阈值通过。 + +该统一入口只运行 CC0 public smoke 与合成格式 fixture，终态固定携带 `hidden_certification: false`。独立 hidden bundle 必须由未参与调优的一方外部托管并预先承诺完整文件 SHA、分母与阈值；KB runner 只能验证机械契约，不能自证独立作者、未见数据或开放世界 100%。 + +wiki 页面/关系索引仍未建设：当前 `search` / `list-orphans` 等走纯 Python 扫描，1k 页内秒级返回。当 wiki 页面数 > ~1000 且延迟 > 2 秒时再启用下列 `.cache/index.db` 方案。 `.cache/index.db`，纯派生数据，加入 `.gitignore`： @@ -426,7 +435,7 @@ tags (page_id, tag) audit_log (id, tool_name, input_summary, timestamp) ``` -`watchdog` 监听 `wiki/` 与 `raw/` 文件变化，hash 对比增量更新。删 `.cache/` 后启动时全量重建。`k.py` 检测到 `.cache/index.db` 走索引，否则 fallback 到现在的纯 Python 扫描——**保持向后兼容**。 +未来可用 `watchdog` 监听 `wiki/` 文件变化，hash 对比增量更新。删 `.cache/` 后从 markdown 全量重建；markdown + Git 始终是真相源。 --- @@ -483,7 +492,7 @@ workspaces/*/my_thoughts/ workspaces/*/.cache/ workspaces/*/exports/ -# wiki 大纲缓存（派生层：k.py 按 doc_chars 判新鲜、过期即现场重建） +# wiki 大纲缓存（派生层：k.py 按 schema + 全文/章节 SHA-256 + 结构校验新鲜度） workspaces/*/wiki/**/*.outline.json # 派生索引（可重建） @@ -513,7 +522,7 @@ web/node_modules/ > **注意**：`raw/` 中的原始资料及其派生物（`.md` / `.outline.json`）**默认不纳入 Git**——版权与隐私原因（原始 PDF/文章往往不可再分发，路径名本身可能泄露私人语境）。入库的是 `wiki/` 提炼物：agent 自己的表述 + 指向 raw 的块级引用坐标。代价是 fresh clone 后 `[[raw/...]]` 引用会显示为失效（预期行为，README 已声明）；raw 资料靠本地备份策略保管，与 CLAUDE.md「权限规则」/ ingest 第 12 步同口径。 > -> **关于 `wiki/**/*.outline.json`**（显式取舍）：wiki 页的大纲缓存也不入库。它是派生层——结构部分（章节树 / 字符偏移）由 `k.py` 按 `doc_chars` 判新鲜、过期即从当前 markdown 现场重建，无需持久化。唯一不可从 markdown 重建的是 `agent_summary`（annotate-section 回填的精排摘要），故 fresh clone 的 wiki 大纲面板不展示这些摘要——这是为「派生层不入库、可重建」的整洁不变量付的小代价。真正承载摘要的是 raw 页的 outline（ingest 阅读时回填），而 raw 派生物本就不入库；wiki 侧摘要属次要、本地便利数据。若未来需要让摘要随仓库分发，应把它写进 markdown 真相源（如 frontmatter 或正文 H2 节），而非持久化派生的 outline.json。 +> **关于 `wiki/**/*.outline.json`**（显式取舍）：wiki 页的大纲缓存也不入库。它是派生层——结构部分由 `k.py` 按 outline schema + Markdown 全文 SHA-256 + 每节 SHA-256 + canonical anchors + 唯一锚/区间/父子结构 validator 判新鲜；任一失效即从当前 markdown 现场重建。唯一不可从 markdown 重建的是 `agent_summary`，故 fresh clone 的 wiki 大纲面板不展示这些摘要。真正承载摘要的是 raw 页的 outline（ingest 阅读时回填），而 raw 派生物本就不入库；若未来需要让摘要随仓库分发，应写进 markdown 真相源，而非持久化派生 outline。 ### 9.4 回滚操作 @@ -531,7 +540,7 @@ Web 管理台 `/history` 可视化界面为规划项（尚未实现）；当前 ## 十、技术架构 -> **当前状态（v0.4 已就绪）**：v0.1（Schema + Skill + CLI + Git hook）+ v0.2（Web 管理台 + i18n + 引用基础设施 + 章节大纲）+ v0.4a（冲突工作台四条解决路径）+ v0.4b（类型化关系图谱 + `/graph` 可视化）已构成完整可用的知识库。v0.3 的 SQLite/FTS5 索引层按触发条件（>1000 页）保留未建。 +> **当前状态**：v0.1（Schema + Skill + CLI + Git hook）+ v0.2（Web 管理台 + i18n + 引用基础设施 + 章节大纲）+ v0.3 raw 自然单元 SQLite/FTS5 证据索引、layout-aware PDF 转换与公开分层准确性闸门 + v0.4a（冲突工作台）+ v0.4b（类型化关系图谱）已可用。v0.3 的通用 wiki 页面/关系索引仍按 >1000 页触发条件保留未建。 > > **本章是技术总览**。组件级细节（具体命令、permissions 配置、hook 行为、Web 路由清单）见 `CLAUDE.md` 与 `web/README.md`。两者 drift 时**以 CLAUDE.md 为准**。 @@ -556,12 +565,13 @@ Web 管理台 `/history` 可视化界面为规划项（尚未实现）；当前 只做"内置 Read/Grep/Glob 做不到或绕弯路"的事，全部支持 `--json`。命令分三组： - **查询**：`search` / `list-pages` / `backlinks` / `outlinks` +- **长文档证据**：`rebuild-evidence-index` / `evidence-index-coverage` / `search-evidence` / `read-evidence-unit` - **健康度**：`health` / `list-orphans` / `list-conflicts` / `list-to-update` / `list-broken-refs` / `list-unsummarized` / `list-bare-claims` / `validate-frontmatter` - **章节级引用基础设施**（v0.3）：`outline` / `read-section` / `read-block` / `find-anchor` / `annotate-section` > 完整列表与参数见 `python scripts/k.py --help` 与各子命令的 `--help`。命令会持续增加，本文档不维护一一对应——避免 drift。 -实现：纯 Python 扫 markdown + 解析 frontmatter，**不上数据库**。1k 页内秒级返回；将来真慢了再加 SQLite/FTS5（见 §7.2 / v0.3 演进路线）。 +实现：wiki 查询仍纯 Python 扫 markdown；raw 长文档证据走可删重建的 SQLite/FTS5 派生层。两者均不调用 LLM、不使用 embedding。 ### 10.3 写权限保护（双层硬约束） @@ -613,7 +623,7 @@ Web 管理台 `/history` 可视化界面为规划项（尚未实现）；当前 |---|---|---|---| | v0.1 | ✅ 已完成 | — | Schema (CLAUDE.md) + 5 个 Skill + `scripts/k.py` CLI + Git pre-commit hook | | v0.2 | ✅ 已完成 | 想要"非 chat 形式"管理 | Web 管理台（Next.js + shadcn + i18n 双语） | -| v0.3 | 部分完成 | 引用基础设施 / 性能 | ✅ 章节大纲 + 块锚点 + `list-broken-refs` 已落地；⏳ SQLite + FTS5 索引层等触发条件（>1000 页） | +| v0.3 | 部分完成 | 引用基础设施 / 检索准确性 / 性能 | ✅ 章节大纲 + 块锚点 + raw 自然单元 SQLite/FTS5 + coverage/freshness/read-unit + layout-aware PDF + Stage-2 public gates 已落地；⏳ 独立 hidden bundle 与通用 wiki 页面/关系索引等仍按条件建设 | | v0.4a | ✅ 已完成（2026-05） | 复杂冲突频繁 | Web 工作台四条冲突解决路径全部落地：`adopt_new` / `merge` / `keep_old` / `keep_watching`（见 §6.3 与 `web/lib/operations.ts`） | | v0.4b | ✅ 已完成（2026-05） | wiki > 500 页 + 真在用类型化关系 | `[[X|REFUTES]]` 类型化关系图谱（7 个标准关系白名单 + `k.py list-relation-issues` lint + `k.py graph` JSON）+ `/graph` 可视化路由（`web/components/GraphView.tsx`，React Flow） | | ~~v0.5~~ | ❌ 已废弃 | — | ~~MCP server~~ — 用户决定不需要。如未来想给 Cursor / Claude Desktop / 自建应用用，可把 `scripts/k.py` 与 `web/lib/operations.ts` 包成 MCP tool，沉没成本为零 | diff --git a/Makefile b/Makefile index db6b884..7926f63 100644 --- a/Makefile +++ b/Makefile @@ -9,7 +9,7 @@ NPM ?= npm NOPROXY := localhost,127.0.0.1,::1 PROXY_BYPASS := no_proxy="$(NOPROXY)" NO_PROXY="$(NOPROXY)" -.PHONY: setup install-python install-web hooks test test-python lint-web build-web typecheck-web health web dev clean +.PHONY: setup install-python install-web hooks test test-python eval-stage2 lint-web build-web typecheck-web health web dev clean setup: install-python install-web hooks @@ -22,11 +22,15 @@ install-web: hooks: bash scripts/install_hooks.sh -test: test-python lint-web build-web health +test: test-python eval-stage2 lint-web build-web health test-python: $(PYTHON) -m pytest scripts/tests +# Public protocol/synthetic fidelity gates only; never reports hidden certification. +eval-stage2: + $(PYTHON) evals/run_stage2.py + lint-web: cd web && $(NPM) run lint diff --git a/README.md b/README.md index 249952b..ddc2021 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,7 @@ That gives you an AI-ready wiki that is easier to audit, diff, review, and maint - **No embeddings by default**: search uses BM25-style text search, metadata, backlinks, outlinks, and full-page reading. - **Stable anchors**: converted raw documents get block anchors such as `^h-*`, `^p-*`, and `^t-*` so claims can cite exact source blocks. +- **Accuracy-first PDF conversion**: tables, headings, and multi-column reading order are reconstructed from page geometry; content raster/vector figures, unresolved glyphs, OCR-only, or corrupt cases fail explicitly. Receipts bind canonical source path + source SHA + converter fingerprint, and the evidence index revalidates that binding before coverage, search, or read. - **Markdown is truth**: SQLite/cache layers are optional derived indexes and can be rebuilt. - **Agent outside, KB inside**: the repository exposes scripts, templates, and Web UI. LLM reasoning happens in external agents. - **Git-native governance**: all meaningful changes can be reviewed, reverted, audited, and discussed as normal commits. @@ -107,10 +108,13 @@ cd web && npm install && cd .. bash scripts/install_hooks.sh python -m pytest scripts/tests +make eval-stage2 # public protocol + synthetic format gates; not hidden certification python scripts/k.py health --json cd web && npm run lint && npm run build ``` +> Windows note: the pre-commit hook is a bash script and only runs under Git Bash / WSL; native cmd/PowerShell will not execute it (everything else works). + > ⚠️ **Stop your dev server before running `npm run build`.** The Web console (`npm run dev`) and `next build` share the same `web/.next/` directory. Running a production build while a dev server is live can leave the dev server serving 404s. To validate types only without building, run `cd web && npx tsc --noEmit` instead. (CI runs the full build in a clean environment, which is fine.) ### Working behind a proxy (Clash / VPN / etc.) @@ -178,6 +182,7 @@ npm run dev ├── AGENTS.md # Codex mirror of CLAUDE.md (kept byte-aligned) ├── GroundMap-设计文档.md # System design document ├── scripts/ # CLI (k.py), conversion (convert.py), parsing, tests, hooks +├── evals/ # citation/retrieval/conversion public evaluation gates ├── web/ # Next.js reading/editing console (+ REST/server actions) ├── .claude/skills/ # Claude Code workflow skills (kb-ingest / query / lint / export / conflict-resolve) ├── .agents/skills/ # Codex mirror of the skills above @@ -213,6 +218,9 @@ Those boundaries are deliberate. The open-source core focuses on the durable kno ## Documentation - 🎓 **[Step-by-step beginner tutorial (中文, with screenshots)](docs/新手教程-手把手搭建知识库.md)** — zero-to-running walkthrough with a full worked example; the best place to start (also available as a [standalone HTML page](docs/新手教程-手把手搭建知识库.html) with a sidebar TOC for offline reading) +- 🛡️ **[Citation integrity: seven layers of defense for provenance correctness](docs/citation-integrity.md)** — deterministic checks / blind-cloze verification / cross-model review / retrieval provenance / adversarial eval suite +- 🧭 **[Citation Accuracy V2 design proposal](docs/citation-accuracy-v2-design.md)** — evidence-first selective QA / lexical multi-route recall / claim-evidence sets / fail-closed certification and statistical release gates +- 🧪 **[Stage-2 evaluation protocol](docs/evaluation-stage2-design.md)** — external holdout schema / answer→citation scoring / PDF-DOCX-HTML fidelity gates and honest certification boundaries - [Quickstart](docs/quickstart.md) - [Why No Embeddings](docs/why-no-embeddings.md) - [Demo Plan](docs/demo.md) diff --git a/README.zh-CN.md b/README.zh-CN.md index 9c5ff59..8b4f231 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -24,6 +24,7 @@ GroundMap 的出发点不同： - **默认不用 embedding**：检索依赖 BM25 风格全文搜索、元数据、反链、出链和完整页面阅读。 - **稳定锚点**：转换后的原始文档带有 `^h-*`、`^p-*`、`^t-*` 等块级锚点，论断可以精确引用到来源块。 +- **准确性优先的 PDF 转换**：按页面几何恢复表格、标题和双栏阅读顺序；遇到内容级光栅/矢量图、无法解析字形、需 OCR 或损坏文件时明确失败。receipt 绑定 canonical source path + source SHA + converter fingerprint，证据索引在 coverage、search、read 前持续复验。 - **Markdown 是真相源**：SQLite / 缓存层都是可选的派生索引，可随时重建。 - **agent 在外，知识库在内**：仓库只提供脚本、模板和 Web 管理台；LLM 推理发生在外部 agent。 - **Git 原生治理**：所有有意义的改动都是普通 commit，可评审、可回滚、可审计、可讨论。 @@ -107,10 +108,13 @@ cd web && npm install && cd .. bash scripts/install_hooks.sh python -m pytest scripts/tests +make eval-stage2 # 公开协议 + 合成格式闸门，不是隐藏认证 python scripts/k.py health --json cd web && npm run lint && npm run build ``` +> Windows 用户注意：pre-commit hook 为 bash 脚本，需在 Git Bash / WSL 环境下生效；原生 cmd/PowerShell 不会执行该 hook（其余功能不受影响）。 + > ⚠️ **跑 `npm run build` 前请先停掉本地 dev server。** Web 管理台（`npm run dev`）与 `next build` 共用同一个 `web/.next/` 目录；dev server 在跑时执行生产构建会让运行中的 dev server 全部返回 404。仅验证类型请改用 `cd web && npx tsc --noEmit`。（CI 在干净环境里跑完整 build，不受此影响。） ### 开了代理（Clash / VPN 等）也能正常访问 @@ -178,6 +182,7 @@ npm run dev ├── AGENTS.md # CLAUDE.md 的 Codex 镜像（逐字对齐） ├── GroundMap-设计文档.md # 系统设计文档 ├── scripts/ # CLI（k.py）、转换（convert.py）、解析、测试、Git hooks +├── evals/ # 引用 / 检索 / 格式转换的公开评测闸门 ├── web/ # Next.js 阅读/编辑管理台（+ REST / server actions） ├── .claude/skills/ # Claude Code 工作流技能（kb-ingest / query / lint / export / conflict-resolve） ├── .agents/skills/ # 上述技能的 Codex 镜像 @@ -213,6 +218,9 @@ GroundMap 刻意不包含： ## 文档 - 🎓 **[新手图文教程：手把手搭建并使用知识库](docs/新手教程-手把手搭建知识库.md)** —— 零基础，带真实截图和完整案例，强烈建议从这里开始（也有 [HTML 版](docs/新手教程-手把手搭建知识库.html)，带侧栏目录、适合本地离线阅读） +- 🛡️ **[引用保障体系：七层防线确保每条论断有正确出处](docs/citation-integrity.md)** —— 确定性核对 / 盲填复核 / 跨模型二审 / 检索凭证 / 对抗评测集 +- 🧭 **[引用准确性 V2 设计](docs/citation-accuracy-v2-design.md)** —— evidence-first 选择性问答 / 长文自然单元召回 / 严格认证边界 +- 🧪 **[第二阶段评测协议](docs/evaluation-stage2-design.md)** —— 外部留出集 schema / 回答→引用计分 / PDF-DOCX-HTML 保真闸门 - [Quickstart](docs/quickstart.md) - [为什么不用 embedding](docs/why-no-embeddings.md) - [Demo 方案](docs/demo.md) diff --git a/docs/citation-accuracy-v2-design.md b/docs/citation-accuracy-v2-design.md new file mode 100644 index 0000000..dc75f1a --- /dev/null +++ b/docs/citation-accuracy-v2-design.md @@ -0,0 +1,787 @@ +# 引用准确性 V2：Evidence-First Selective QA 设计 + +> 状态：P0 严格闸门、P1 长文档词法召回与公开格式保真闸门已落地；完整 evidence-set 语义认证、独立隐藏发布集与 P2–P3 仍未完成 +> +> 日期：2026-07-11 +> +> 目标：在不引入 embedding、不把 LLM 放进知识库核心、仍以 Markdown + Git 为真相源的前提下，显著提高知识构建与问答的相关证据召回、引用完整性和引用语义准确性。 + +### 实施进度（2026-07-11） + +已落地的基础： + +- `check-draft --strict` 及草稿 `cite-audit-log --draft`；broken、非 canonical 锚、raw 不可得、imprecise、pending/bare/coarse、无 provenance、非 `SUPPORTED`、截断包均 fail-closed； +- claim 不再静默截断，pair 身份绑定所属 heading 等去歧义上下文；hash 容错恢复 / Web fallback 不得进入 strict 成功集； +- `[KB 推算]` 豁免收窄到紧邻的单个值，依据锚必须实际被引，且推算值不再被错送到原文字面盲填； +- `tools/cite-audit --draft` 使用完整 evidence、盲填 + 反驳双通道与结构化终态；incomplete / skipped / 协议故障不再冒充语义命中； +- pre-commit 在 staged blob 临时树上执行并对 checker 故障 fail-closed； +- Debug Console 在验证完成前去链接化，仅承认实际取回的 canonical `path#anchor`，拒绝粗引用、fallback、hash recovery、无判官和未引用实质论断的假绿。 +- `.outline.json` schema v2 使用全文 `doc_sha256` + 每节 `section_sha256`，对 canonical heading anchor、唯一性、字符范围、顺序、父子包含与完整结构 fail-closed 校验；同长度改写也会强制重转，旧摘要只在章节 hash 不变时迁移。 +- `.cache/retrieval_index.db` 枚举 paragraph / list item / table row / blockquote / code / figure 自然单元，同时登记内容/结构章节；`evidence-index-coverage` 对归一化与保留结构空白的 exact text hash、构建时冻结的完整单位/章节 inventory 指纹、完整路由元数据、分类计数、两个 FTS5 的规范 DDL/列序/tokenizer、内容行与倒排内部 `quick_check`、raw 全文 SHA 与已验证 outline 摘要指纹做 100% 机械对账。 +- `search-evidence` 使用 unicode61 BM25、trigram、exact 和低权重结构路由的 RRF 融合，文件集或全文 hash 变动后 fail-closed；`read-evidence-unit` 保留 row/item 的 `unit_id + subordinal + content_hash`，再通过 `read-block` 审查 canonical 父块。 +- `kb-ingest` 将索引重建 + 覆盖率闸门放在转换后；`kb-query` 将原问题 + 显式 facets/aliases 的 top-20 自然单元召回与目录浏览路组合，禁止从中间候选泄漏未知答案到查询。 +- PDF 转换改为基于 `pdfplumber` 几何坐标的 layout-aware 路径：表格优先、跨栏分带、左右栏 column-major、标题恢复、页眉页码去重；失败不静默回退，扫描页、内容级大图、CID 映射异常与损坏文件明确拒绝。DOCX 的空表头 + 全粗体首行会在高置信条件下恢复为真实表头。 +- 第二阶段公开闸门已落地：外部 holdout bundle/schema 与唯一问题/机械切片/方向化阈值；绑定 canonical text + value + evidence set 的 atomic claim→citation E2E；PDF/DOCX/HTML 源坐标保真 Gold、独立文本/列表/脚注关系与 14 类 mutation。统一入口 `python evals/run_stage2.py` 会重算子指标与阈值方向，终态明确不是 hidden certification。 + +仍未落地：生产链路的完整 claim-slot / evidence-set schema 与最小充分集、递归 wiki→raw leaf 认证、任意真实文档的持久 source locator/视觉对齐、真正的 row/item Markdown 子锚、选择性回答的 risk–coverage 校准与独立隐藏发布集。因此当前实现不等于「开放世界 100% 引用正确」。 + +### 长文档固定开发集结果 + +`python evals/longdoc/run_eval.py --json` 在冻结的 CC0 合成开发集上输出： + +| 指标 | 结果 | +|---|---:| +| 两个位置变体的自然单元机械覆盖 | 1488 / 1488 | +| 内容长章节登记 | 72 / 72 | +| 50 个毒化/位置移动回归 case | 50 / 50 | +| 仅用原始问题的 Complete Evidence Set Recall@20 | 50 / 50 | +| 60 个 base facet 跨位置变体 selected-evidence retrieval proxy | 60 / 60 | +| answerability / top-1 Gold-set retrieval proxy | 50 / 50；50 / 50 | + +评测中 Gold 不在被测 workspace，主 Recall@20 不允许使用 Gold facet expansion，row/item 按 `path + anchor + kind + subordinal + content_hash + unit_id` 精确计分；独立 oracle 还逐单位核对所属章节/标题路径/锚链/摘要/精确文本哈希，逐章节核对 `path + anchor + title + content_hash + route`，并强制证据在位置变体中真正移位。但这仍是**检索代理指标**：它不测量最终答案文本是否覆盖每个 required facet，不测量 claim→citation 的语义蕴含，也不测量 PDF/DOCX→Markdown 保真度。 + +上述三个缺口现在分别由 [第二阶段评测](evaluation-stage2-design.md) 的 public holdout、answer→citation 和 conversion-fidelity 协议覆盖；它们当前仍是公开/合成闸门。独立外部 hidden bundle 和真实生产模型 predictions 未提供时，不能把满分升级为统计认证。 + +## 1. 决策摘要 + +GroundMap 已有的块锚点、数字核对、盲填、fresh-context 审计和检索凭证是很好的基础，但它们主要回答： + +> “已经选中的这条引用，有没有明显错？” + +要把准确性继续推近 100%，还必须回答另外三个问题： + +1. **相关证据是否被完整召回？** +2. **证据集合是否足以支撑整条原子论断，而不只是出现了相同数字？** +3. **证据不足时，系统是否能稳定拒答，而不是带着引用硬答？** + +本设计建议把系统升级为 **Evidence-First Selective QA（证据先行的选择性问答）**： + +```text +问题分面 → 多路宽召回 → 证据充分性判定 → 先选引用 → 再写原子论断 + → 独立反证 → 严格闸门 → 输出已认证论断 / 明确拒答 +``` + +核心目标不应表述为“所有问题都无条件 100% 正确”，而应表述为： + +> **对系统实际输出的“已认证事实论断”，在定义明确的隐藏评测集上达到观测 100% 引用正确与完整；无法认证的内容不输出为事实。与此同时，单独报告回答覆盖率、检索召回率和统计置信上界。** + +有限测试无法证明开放世界中的绝对 100%。真正可实现的路径是： + +- 结构性错误 100% 清零； +- 有限隐藏集观测 100%； +- 已回答样本的错误率上界可量化； +- 证据不足或验证器分歧时拒答； +- 高风险场景允许显式进入穷举模式或人审。 + +## 2. 不变约束 + +本方案保持现有架构底线： + +- **知识库核心零 LLM**：`scripts/`、`web/` 只做确定性索引、取数、校验、哈希、统计和状态管理；查询扩写、语义 rerank、蕴含判定继续由外部 agent 或 `tools/` 客户端执行。 +- **禁止 embedding 与向量库**：召回使用自然结构单元上的 FTS5/BM25、字符 n-gram、实体别名、图谱邻居、查询扩写与浏览式检索。 +- **不做任意文档切片**：索引单位只使用已经存在的 heading、paragraph、list item、table row、figure caption 等自然单元；搜索命中后仍读取完整块、完整 H2/H3 节或完整页面。 +- **Markdown + Git 是唯一真相源**：引用、论断、冲突与人工裁决留在 Markdown；检索运行包、验证台账、索引与源坐标表放 `.cache/` 或 raw 派生层，可删除重建。 +- **人是最终闸门**：来源冲突、验证器分歧和高风险结论仍由人裁决。 + +## 3. 立项时基线与为什么还不能声称接近 100% + +> **历史快照**：本节保留 2026-07-11 本轮实施**之前**的只读审计，用于说明设计动机，不是当前实现清单。其中 strict draft 闸门、完整 evidence 包、staged-blob pre-commit、outline SHA 新鲜度与 raw 自然单元 FTS5/RRF 索引已按文首「实施进度」修复。仍未解决的是原始载体保真、开放世界语义蕴含/完整性认证与隐藏发布集。 + +### 3.1 已有强项 + +当前七层引用防线已经覆盖了很多传统 RAG 没有解决的问题： + +- 内容哈希块锚点与 H 节正文 hash； +- 数字、单位、量级、舍入和逐字引文核对； +- quote-first 检索凭证； +- 数字盲填； +- fresh-context 对抗回验； +- 双侧内容漂移后自动重审； +- CAUTION 标注、pre-commit 和人审。 + +相关设计见 [citation-integrity.md](citation-integrity.md)。这些机制应保留并纳入 V2，而不是推倒重来。 + +### 3.2 实施前实测（历史快照） + +2026-07-11 的只读审计结果： + +| 项目 | 结果 | 能证明什么 | +|---|---:|---| +| Python 完整测试 | 301 passed | 实现符合当前测试样例 | +| 确定性引用 eval | 20/20 | 现有 20 个固定 case 的预期命中 | +| `smb-ecommerce` | broken 392；coarse 16；unverifiable 115 | release demo 缺 raw，无法做完整语义核验 | +| `rag-evolution` | broken 136；coarse 61；unverifiable 75 | 同上 | +| audit confidence | audited = 0；上界 = 100% | 目前没有可推广的存量语义准确率证据 | + +CI 当前只断言 `total_pages > 0`，不会因为 broken、coarse、unverifiable 或 unaudited 很高而失败。20-case eval 还把 4 个“确定性层应 pass、语义层应拦”的毒化 case 统计进“干净”分母。因此，测试全绿与引用接近 100% 之间没有直接等号。 + +### 3.3 实施前主要失效面（历史快照） + +| 阶段 | 当前缺口 | 后果 | +|---|---|---| +| 原始格式 → Markdown | 没有 PDF 页码/bbox、DOCX 段落、表格行列等源坐标，也没有转换保真黄金集 | 可能“精准引用了错误转换结果” | +| 相关段落召回 | `search` / `search-raw` 是空格分词与子串计数，不是真正 BM25；中文、同义改述、中英跨语弱 | 相关证据可能根本没进入候选集 | +| 论断完整性 | `bare-claims` 主要只看数字；无引用的定性事实、因果、建议、比较可绕过 | “X 导致 Y”即使无引用也可能假绿 | +| 证据原子性 | paragraph 与整表仍是大核对单元；多引用数字采用并集 | 主体互换、分句错配、表格错行可能通过 | +| 语义审计 | claim 默认截 500 字符，evidence 默认截 1500 字符 | 未被 verifier 看见的尾部内容也可能被整对认证 | +| 问答闸门 | `check-draft` 不阻断 broken、unverifiable、imprecise、无语义审计和无 provenance | “命令通过”不等于引用真实支撑 | +| 提交守门 | pre-commit 检查工作树而非 staged blob，且工具故障时可能 fail-open | 坏的 staged 内容可能被好工作树掩盖 | +| 统计保证 | `audit-confidence` 以验证器 verdict 为真值 | 只能描述验证器判定，不能独立估计验证器漏检 | + +更具体的实现位置包括： + +- 简单子串检索：`scripts/k.py` 的 `search_pages` / `search_raw`； +- 只枚举已有块级引用：`extract_claims`； +- 任一缺失引用使整块 `unverifiable`：`list_cite_mismatches`； +- 查询侧窄闸门：`check_draft` 与其 CLI exit-code 分支； +- 转换只接收 `MarkItDown` 的 Markdown 文本：`scripts/convert.py`； +- CI 仅检查知识库非空：`.github/workflows/ci.yml`。 + +## 4. 准确性的正确分解 + +不要把“引用准确率”压成一个平均分。至少要分别测六层： + +1. **Source Fidelity**：派生 Markdown 是否忠实于原始 PDF / DOCX / HTML / 表格 / 图片。 +2. **Retrieval Recall**：黄金证据块是否进入候选集。 +3. **Evidence Sufficiency**：候选证据集合是否覆盖论断的全部必要子事实。 +4. **Citation Precision**：每一条被选引用是否真的必要且支持相邻原子论断。 +5. **Citation Completeness**：所有实质性原子论断是否都有充分引用。 +6. **Answer Correctness / Selective Risk**：答案是否正确；拒答时机是否合理。 + +这些指标不能互相替代： + +- 引用 precision 高，不代表没有漏引； +- citation completeness 高，不代表相关证据没有漏召回； +- 数字共现，不代表主体、条件、方向、因果关系正确； +- 来源忠实，不代表来源本身真实；来源冲突仍走冲突工作流。 + +主要发布指标建议使用 **Complete Fully Grounded Answer Rate**（下文记为 `fully_grounded_coverage`）：一份答案只有在“所有 required facets 均被覆盖、所有重要原子论断都正确、都有充分引用、锚点可解析、无错引、无未决冲突”时才计 1，分母是全部可回答黄金问题。它同时惩罚错答、漏答和靠输出一个安全小事实刷高 precision，比平均每条引用分数更符合用户实际信任边界。 + +## 5. V2 总体架构 + +```mermaid +flowchart TD + A["原始来源"] --> B["转换 + 源坐标 + 保真检查"] + B --> C["自然结构单元索引"] + Q["用户问题"] --> D["问题分面 / 原子子问题"] + D --> E["多路词法宽召回"] + C --> E + E --> F["RRF 融合 + 来源多样化"] + F --> G["读取完整块 / H2-H3 节"] + G --> H["最小充分证据集 + 反证搜索"] + H --> I{"上下文充分且无分歧?"} + I -- "否" --> J["补检索 / 穷举 / 人审 / 拒答"] + I -- "是" --> K["Reference → Atomic Claim"] + K --> L["确定性检查 + 独立语义复核"] + L --> M{"严格闸门通过?"} + M -- "否" --> J + M -- "是" --> N["已认证回答 + 证据清单"] +``` + +### 5.1 核心与外部 agent 的职责边界 + +| 层 | KB 核心（确定性） | 外部 agent / tools（语义） | 人 | +|---|---|---|---| +| 转换 | hash、parser 版本、源坐标、结构与数字保存检查 | 低置信页视觉复核 | 关键源确认 | +| 查询规划 | 校验 query-plan schema、执行检索 | 分解子问题、生成同义/双语扩写 | 可指定范围/模式 | +| 候选召回 | FTS5/BM25、n-gram、图谱、RRF、去重 | semantic rerank、相关/反证判定 | 高风险时审候选 | +| 证据选择 | coverage slots、hash、最小集计算 | sufficiency / entailment / contradiction | 分歧仲裁 | +| 回答 | 校验 claim-evidence 包和门禁 | 先引用后生成论断 | 最终 review | +| 统计 | 混淆矩阵、置信区间、risk-coverage | 不负责给自己打分 | 独立黄金标注 | + +原子分解与定性事实识别的边界必须明确：外部 writer 产出精确文本 span → decontextualized atomic claim 的 claim map；另一个不共享写作上下文的外部 decomposer 检查 span 覆盖、代词消解、条件和否定是否丢失。KB 核心只校验 schema、span/hash 完整性、attestation 版本和“最终渲染文本不得含未映射 factual span”。严格回答由已验证 packet 确定性渲染，不允许模型在最后润色时加入自由事实文本。 + +表中的 semantic rerank 只能由外部 agent 对完整自然单元执行，核心不得保存或使用向量。MiniCheck 等可选模型只能作为 `tools/` 外部客户端，经受控 CLI/REST 取证与回写 verdict；不得被移入 `scripts/`、`web/`，也不得绕过接口直接把 Markdown/`.cache` 当私有数据库读取。 + +## 6. Layer 0：先保证“被引用的原文”没有在转换时失真 + +### 6.1 源版本与转换 manifest + +每份 raw 来源的派生层应记录： + +```json +{ + "source_sha256": "...", + "source_type": "pdf", + "converter": {"name": "markitdown", "version": "..."}, + "parser_version": "...", + "markdown_sha256": "...", + "generated_at": "..." +} +``` + +`.outline.json` 的新鲜度应使用全文 hash + parser version，而不是只比较字符数。相同字符数的内容改写不能复用旧 outline 或旧摘要。 + +元数据分层必须固定，避免形成隐性第二真相源： + +- 公开、稳定、可审阅的 bibliographic metadata（language、published/updated date、source kind）若要参与长期查询，应通过一次 schema 迁移进入 source_summary frontmatter； +- extractor/parser/version、source hash、origin locator、OCR/fidelity 属于 raw 本地派生 manifest； +- authority 排名、query relevance、临时风险分只属于 retrieval certificate，不得写成永久事实； +- 人工确认的来源可信等级若需要持久化，必须以 Markdown 审阅字段和变更历史表达，不能只留在 `.cache`。 + +canonical source/block/evidence ID 应改用带 normalization-version 的较长 SHA-256 派生值；现有 6 位 MD5 锚仅作为人读短坐标保留。升级时生成旧锚 → 新 ID 的迁移表，任何 hash 容错恢复只能用于定位修复，不能直接视为 strict 引用有效。 + +### 6.2 原始载体坐标 + +为每个自然块生成本地派生的 `origin_locator`： + +- PDF：页码 + bbox； +- DOCX：paragraph/table/cell 路径； +- PPTX：slide + shape； +- XLSX：sheet + range； +- HTML：DOM path； +- 音频：时间戳； +- 图片 OCR：bbox + OCR confidence。 + +Markdown 仍使用现有 `[[raw/...#^anchor]]` 语法；源坐标只作为可重建的本地验证信息，不进入公开 wiki 的敏感数据层。 + +现有 `MarkItDown` 接口只返回最终 Markdown，无法凭空补齐上述坐标。因此这不是“增加几个字段”，而是 P1 的转换管线重构：按格式接入 extractor、文本对齐和 OCR 坐标层。暂时无法生成可靠坐标的格式必须明确标 `fidelity=unknown`，不得伪造 locator；旧资料迁移时先保留现有锚，再按风险分批回填坐标。 + +### 6.3 转换保真闸门 + +建立 `evals/conversion_fidelity/` 黄金集，至少测试： + +- 数字、正负号、单位、日期与否定词保存率； +- PDF 阅读顺序； +- table cell、行列和表头关联； +- figure-caption 与脚注关联； +- OCR CER/WER 与低置信区域； +- 重转后的 anchor 迁移与旧摘要失效。 + +严格模式下，`fidelity=low|unknown` 的块不能支撑精确数字、引文或高风险结论，除非人工复核。 + +## 7. Layer 1：Coverage-aware Ingest + +### 7.1 从“写完再找引用”改为证据卡先行 + +每读完一个完整节，外部 agent 先形成候选证据卡，再写 wiki： + +```json +{ + "target": "raw/papers/example.md", + "anchor": "p-12-7d8e9a", + "content_hash": "...", + "quote_span": "原文字面子串", + "entities": ["..."], + "polarity": "positive|negative|mixed", + "time_scope": "...", + "claim_types": ["numeric", "comparison"] +} +``` + +证据卡是 `.cache` 派生对象；Markdown 中的论断与引用仍是真相源。`read-block --json` 应直接返回 canonical evidence handle，避免 agent 手抄路径、锚点和 hash。 + +### 7.2 原子论断书写契约 + +严格模式要求引用紧跟其支持的最小子句。解析单位扩展为： + +- sentence / clause； +- list item； +- table row / cell relation； +- figure caption； +- code result； +- blockquote 中的独立陈述。 + +一个复合句若由多份来源分别支撑，必须拆成多个原子子句并各自挂引用。不能再依赖整段所有数字与所有引用的并集。 + +这一步不能由 KB 核心里的正则假装完成。严格模式的 claim map 由外部 agent/tools 生成，第二个独立外部分解器做 coverage review；核心只验证： + +- 每个 claim 的原文 span 与 draft hash 匹配； +- 所有输出 factual spans 均被映射且不得重叠冲突； +- decontextualized claim 保留实体、时间、条件、否定、范围与模态字段； +- 每个 claim 都绑定 evidence packet 和外部 verdict attestation； +- 最终 renderer 只组合已验证 claim，不接受未映射的生成式补写。 + +运行时的双分解器只能降低漏抽风险；真正的 claim extraction coverage 与 facet-plan recall 仍必须以独立人工黄金标签为分母，不能由系统自己生成的 claims/facets 自证完整。 + +### 7.3 不同论断类型的证据契约 + +| 论断类型 | 最低证据要求 | +|---|---| +| 直接事实 | 至少一个直接陈述该事实的可解析块 | +| 数字 / 日期 / 金额 / 引文 | 数值、单位、方向、主体、条件均匹配；优先 paragraph/table-row 级 | +| 比较 | 每个比较对象与指标各有证据；比较运算本身可重算 | +| 因果 / 建议 | 来源明确陈述因果或建议；只有相关性时必须标为 `[KB 推断]` | +| 多源综合 | 每个子事实分别有证据，且综合步骤显式标记 | +| 否定 / “知识库没有” | 必须有范围明确的穷举检索凭证，不能由 top-k 未命中推出 | +| “全部 / 唯一 / 最佳” | 必须证明比较范围完整；否则收窄措辞 | +| 时效性事实 | 来源版本、日期和适用时间窗均可核验 | + +### 7.4 source summary 的直接证据要求 + +source_summary 的“核心论点 / 数据要点 / 方法”中的实质性陈述应直接指向 raw 自然块。整页 raw 链接只用于来源绑定与背景，不得支撑实质论断。概念页和分析页引用 wiki 摘要时，严格模式还要递归验证 wiki → raw 的证据链。 + +## 8. Layer 2：无 embedding 的高召回检索 + +### 8.1 真正的自然单元 FTS5/BM25 + +把 SQLite FTS5 从“页面多了才做的性能优化”提升为“准确性基础设施”。索引仍是可删可重建的派生层。 + +建议字段： + +- page title； +- heading path； +- block body； +- agent_summary； +- entity / alias / acronym； +- tags / page type / status； +- source date / language； +- anchor 与 content hash。 + +英文采用词项 BM25；中文增加确定性字符 bigram/trigram 通道，避免无空格整句被当作一个词。deprecated / archive / stub / low-confidence 默认进入单独通道并降权，不能在高召回模式里静默硬过滤；只有用户显式限定“当前有效内容”时才可排除。`exhaustive` 必须纳入时间范围内的全部状态，以便发现历史版本和反证。 + +索引行可以是 paragraph、list item、table row 和 H 节等自然单元；检索结果只作为坐标，语义判断前仍按协议读完整块或完整 H2/H3 节。 + +### 8.2 问题分面与 query plan + +外部 agent 先把问题拆成必答分面，再提交结构化计划： + +```json +{ + "query": "...", + "mode": "verified", + "facets": [ + { + "id": "F1", + "question": "...", + "required": true, + "phrases": ["..."], + "entities": ["..."], + "aliases": ["..."], + "bilingual_terms": ["..."], + "time_scope": "..." + } + ] +} +``` + +问题分面不是装饰。开放式问答经常漏掉核心子问题；只有先显式列出分面，才可能分别测 retrieval recall 和答案覆盖度。 + +### 8.3 多通道候选并集 + +每个 facet 的必跑通道是：原始查询的精确实体/短语/数字、block BM25、section BM25 与结构导航。其余通道必须先在本库 retrieval 黄金集上做消融校准，再按 query 类型启用： + +1. 精确短语、数字、标识符、实体与别名； +2. block BM25； +3. section BM25； +4. wiki root/MOC/graph/backlinks/outlinks 邻居； +5. 外部 agent 生成的双语同义查询； +6. 可选：Query2doc 式伪文档扩写后再走 BM25； +7. 可选：通过可信来源与相关性门控后的 pseudo relevance feedback； +8. 低覆盖时走 `corpus-map → outline → read-section` 浏览路。 + +各通道排名使用 Reciprocal Rank Fusion（RRF）合并，并在展示/rerank 阶段增加来源多样化，防止 top-k 被单篇长文或常见词占满。来源配额不能在高召回候选生成阶段硬裁剪：某些问题的全部必要证据可能合法地来自同一权威文档。 + +生成式 query expansion 只能作为召回分支：原始问题分支必须保留并具有最高基础权重；伪文档与扩写词不得作为事实写回 Markdown。多次采样扩写时分别检索、最后融合，不能把所有生成文本无差别拼成一个长查询。 + +PRF 也只能是独立低权重通道：扩展前后必须保持实体、时间、否定、比较方向和范围约束，检测到漂移立即丢弃。图谱与 backlinks/outlinks 通道只接受 canonical full path；短名解析出多个候选时必须返回 ambiguity，不能沿 basename 猜测或把 `wiki/concepts/foo` 与 `wiki/sources/foo` 串页。 + +### 8.4 迭代检索与反证检索 + +对 multi-hop 问题，首轮证据中的新实体可触发下一轮检索。对每个拟输出结论，还要强制生成反证查询： + +- 相同主体 + 否定词； +- 比较方向反转； +- 旧版 / 新版 / 不同时间窗； +- `REFUTES` / `ALTERNATIVE_TO` 图谱边； +- 同数值但不同主体或不同指标。 + +### 8.5 检索凭证升级为 retrieval certificate + +当前 provenance 只证明“读过某块”。V2 应记录完整检索运行包： + +```json +{ + "run_id": "...", + "corpus_merkle_root": "...", + "query_plan_hash": "...", + "channels": [{"name": "bm25-block", "ranking": ["..."]}], + "selected": ["..."], + "rejected": [{"anchor": "...", "reason": "irrelevant"}], + "facet_coverage": {"F1": "sufficient"}, + "stop_reason": "all_required_facets_sufficient" +} +``` + +证书不能证明世界上没有漏召回，但它使查询是否按约定完成、是否覆盖所有必答分面、为何停止变得可审计。语料快照用文档/自然块 content hash 构造 Merkle root，既支持增量更新，也能准确说明本次查询覆盖了哪些版本。 + +### 8.6 两种显式高准确模式 + +不做关键词自动判别，模式只能由 CLI/API/UI 显式选择： + +- `verified`：多通道宽召回 + evidence sufficiency + 双重验证 + 可拒答；适合大多数高准确查询。 +- `exhaustive`：对限定的闭集语料逐节扫描，特别用于“没有 / 全部 / 唯一”、高风险或小语料场景；成本高，但最接近闭集召回完整性证明。certificate 必须记录包含/排除规则、语料 Merkle root、自然单元总数、实际扫描数、解析失败文档和停止原因；任一应扫描单元失败或遗漏，结论即为 `UNVERIFIABLE`。 + +## 9. Layer 3：最小充分证据集，而不是“相关段落列表” + +### 9.1 Sufficiency 与 entailment 必须分开 + +“这段是否蕴含某个子事实”与“这些段落是否足以回答整个问题”是两种不同判断。V2 的 evidence packet 应包含： + +- 论断的原子事实 slots； +- 每个 slot 的支持证据与 `edge_verdict`； +- 反证或冲突证据； +- 来源直接性、时效、权威层级与独立性； +- 尚未覆盖的 slot； +- conversion fidelity 状态。 + +两级判定不能混淆： + +- `edge_verdict`：`SUPPORTS_SLOT | CORROBORATES_SLOT | IRRELEVANT | CONTRADICTS`，说明某条引用对被分配 slot 的实际职责； +- `set_verdict`：全部 required slots 是否被整个 evidence set 覆盖，且没有未处理反证。 + +只有每条保留引用的 `edge_verdict` 属于 `SUPPORTS_SLOT | CORROBORATES_SLOT`、每个 required slot 至少有一条 `SUPPORTS_SLOT`，且整个集合 `set_verdict=VERIFIED`，claim 才能通过。不能要求单份来源独自蕴含完整的多源综合 claim，也不能用“集合整体似乎相关”掩盖某条错引。 + +### 9.2 最小充分集选择 + +候选证据集的选择目标不是“越多越安全”，而是： + +1. 覆盖全部必要 slots； +2. 优先直接、一手、精确自然块； +3. 保留实质冲突； +4. 删除对任何原子子事实都不必要的引用； +5. 多源综合时，确保每个来源的职责清楚。 + +这可以被建模为小规模 set cover，再由外部 verifier 检查语义。无关引用同样是错误，因为它会增加人工审查负担并制造虚假可信感。 + +证据必要性可做 leave-one-out 检查：移除某条引用后仍能完整支持全部 slots，则该引用是冗余候选；移除后某个 slot 失去支持，则它属于最小充分集。要区分“无关冗余”和“独立交叉验证冗余”：前者是 citation precision 错误，后者在高风险事实中可以保留，但必须标明它承担的是独立复核职责。证据选择还要比较是否存在更直接、更权威、更新的可用来源，不能只问“当前引用勉强能否 entail”。 + +packet 分开记录 `minimal_support_set` 与有数量上限的 `corroborating_sources`。Leave-one-out 要迭代重算，不能同时把多个互为替代的来源各判“可移除”后一次性全部删除。 + +### 9.3 判定状态 + +严格模式只允许以下状态： + +- `VERIFIED`：最小充分证据集存在，所有验证器通过； +- `INSUFFICIENT`：证据相关但不完整； +- `CONFLICTED`：存在尚未裁决的实质冲突； +- `UNVERIFIABLE`：源文件、锚点、转换或版本不可验证； +- `REJECTED`：证据不支持或反驳论断。 + +只有 `VERIFIED` 论断能以事实语气进入严格回答。其他状态必须补检索、收窄措辞、显式报告不确定性、进入人审或拒答。 + +## 10. Layer 4：Reference → Claim，而不是 Claim → Citation + +答案生成改成循环： + +1. 选择一个“锚点/哈希有效、已读取、与 facet 相关”的候选 evidence packet（可含一个或多个 handle）；此时尚不能称为语义已验证； +2. 只基于 packet 生成一个原子 claim，并分解 required slots； +3. 对每条 evidence→slot 边和整个 evidence set 做语义验证； +4. 只有 `edge_verdict` / `set_verdict` 通过后，才把引用与 claim 标为 `VERIFIED`； +5. 将引用紧跟在最小支持子句后并做确定性复核； +6. 再进入下一条 claim。 + +这与当前 quote-first 一致，但把纪律提升为可检查的数据流。ReClaim 在其评测设置中表明，交替 reference/claim、约束模型只复制给定引用可以改善 citation quality；这不是对所有 reference-first 方法的普遍证明，论文也观察到答案准确度下降且效果依赖 reference passage 的信息密度。因此 V2 必须同时守 answer correctness 与 coverage，不能只优化引用分数。 + +对推算或综合，必须记录操作： + +```text +结果 = operand_1 - operand_2 +依据 = [anchor_1, anchor_2] +标记 = [KB 推算: ...] +``` + +现有 `[KB 推算]` 豁免要收窄到紧邻的单个原子值，不能因为一个合法 marker 而豁免同块所有数字。 + +## 11. Layer 5：统一严格验证器 + +### 11.1 `check-draft --strict` + +新增严格模式，并让所有 `verified` / `exhaustive` 查询答案必跑。以下任一出现都应失败： + +- broken target / anchor； +- 引用依赖 `recovered_from`、basename 猜测或 Web `_fallback` 才能解析；strict 要求草稿 target/anchor 与 canonical handle 完全一致，先重写再复验； +- raw target 不可得；所选 release profile 要求原始格式保真时，origin locator / fidelity 不可得或为 unknown； +- imprecise anchor / quote mismatch； +- coarse citation； +- 未引用的数字或定性事实； +- pending `[需要来源]`； +- 无 retrieval certificate / 当前版本 provenance； +- 任一保留引用的 `edge_verdict` 不属于 `SUPPORTS_SLOT | CORROBORATES_SLOT`，或某个 required slot 缺少直接支持； +- evidence set 的 `set_verdict` 非 `VERIFIED`； +- 未决冲突； +- conversion fidelity 低； +- verifier 异常或超时。 + +这必须 **fail-closed**。工具错误不能解释为“没有 finding”。 + +NOTE/TIP/IMPORTANT 等含事实陈述的 callout 同样进入 claim map；只排除纯协议性的冲突/审计标注外壳，不能因为内容写进 callout 就绕过严格验证。旧 ledger 中的 `PARTIAL` 一律视为未认证，迁移时按新的 slot/edge/set schema 重审，不得继续进入成功分母。 + +### 11.2 原子化与 decontextualization + +严格校验包应包含完整原子论断，不再静默截到 500 字符。长段先按句子/分句/list item/table row 拆分；代词和省略主语需要 decontextualize，避免 verifier 不知道“它”指谁。 + +原子分解器本身也会犯错，因此必须在黄金集上单独测： + +- atomicity； +- completeness； +- entity preservation； +- condition / negation preservation。 + +### 11.3 三通道验证 + +每个 claim-evidence set 使用： + +1. **确定性通道**：数字、日期、单位、符号、方向、引文、主体槽位、hash、源坐标； +2. **盲证通道**：只对可客观恢复并能机器判分的数字、实体、日期、单位和符号做 blind cloze；一般定性事实不能把“另一个 LLM 猜填”冒充盲填真值； +3. **反驳通道**：fresh-context verifier 主动寻找主体互换、条件删除、过度概括、因果错置和反例。 + +一般语义审核要隐藏 writer 的期望 verdict，让 fresh-context verifier 做对抗性反驳；两个 verifier 的联合放行策略作为一个固定系统在黄金集上整体校准。 + +高准确模式下至少使用两个独立 verifier 配置；“独立”至少意味着不同模型家族/供应商，或不同验证范式（如盲填确定性判分 vs 对抗蕴含），且不共享写作上下文。同一模型只换 prompt 不得宣称独立。模型、prompt hash、采样参数、输入覆盖范围、run ID、输出和失败状态全部进入 attestation。任意分歧不得多数投票硬过，而应进入 `INSUFFICIENT` 或人审。 + +MiniCheck 一类 grounding fact-checker 可以作为 `tools/` 下的可选外部 verifier，但不应被放进 KB 核心，也不能单独成为最终真值。 + +NLI 或单一 LLM judge 只能是代理指标。它通常只判断“给定 passage 是否蕴含 claim”，看不到候选池里是否有更直接来源、是否漏掉关键证据、来源是否过期或是否存在冲突；因此 V2 的最终判定必须同时使用 query、完整候选池、父节上下文、来源元数据与反证结果。 + +### 11.4 不截断证据后为全文背书 + +H 节过长时可以提供“命中窗口 + 完整节分段”，但验证包必须证明所有片段都被覆盖。若 verifier 只看到前 1500 字符，就只能给这 1500 字符范围内的结论，不能给整节 hash 的 pair 记 `SUPPORTED`。 + +### 11.5 递归证据链 + +答案引用 wiki 综合页时，严格验证应沿 wiki 引用递归追到 raw 叶子： + +```text +answer claim → wiki claim → raw evidence +``` + +每条边都要 `SUPPORTED`，且 leaf source 可解析。事实型答案优先直接引用 raw；真正的综合判断可同时保留 wiki 页面和必要 raw 叶子。 + +### 11.6 `.cache`、本地认证与 CI 的边界 + +验证台账、evidence packet 与 retrieval certificate 仍是可删重建的派生层，不能因为它们存在就变成第二真相源。可执行模型固定为： + +1. **本地权威环境**：raw 在场，外部 verifier 可用；严格查询与 wiki 写入在这里现场重建 content-addressed attestation。删 `.cache` 后必须重审，不能沿用“已认证”状态。 +2. **公开 fresh clone / CI**：只对许可清晰的 raw eval fixture 跑完整链路，并对公开 demo 做可执行的结构检查；demo raw 未分发时明确 `UNVERIFIABLE`，CI 不得声称其实际 wiki 语义已认证。 +3. **发布证明**：如需随 release 分发认证结果，可把由本地权威环境生成的 content-addressed、可签名 attestation 作为 release artifact；它是可验证的派生证明，不是 Markdown 知识真相。任一 claim/source/prompt/verifier hash 变化即失效。 + +因此，运行时操作闸门“当前 claim-evidence set 必须 VERIFIED”与统计声明必须分开：前者是系统内部放行条件，后者只能来自独立隐藏黄金集或人工抽审，不能由 `.cache` verdict 自证。 + +## 12. Layer 6：选择性回答与风险—覆盖率 + +要把 precision 推近 100%，必须允许 coverage 下降。系统应同时报告： + +- `certification_pass_rate`：运行时有多少候选回答通过内部 strict 门禁；它是操作指标，不是准确率真值； +- `human_gold_fully_grounded_precision`：已回答样本中，经独立黄金标签判定完整、正确且 fully grounded 的比例； +- `answerable_coverage`：可回答黄金问题中，多少给出了完整答案； +- `required_facet_recall`：required facets 被答案覆盖的比例； +- `fully_grounded_coverage`：完整且经黄金判定 fully grounded 的答案数 / 全部可回答黄金问题； +- `false_abstention_rate`：可回答问题被拒答的比例； +- `false_answer_on_unanswerable`：不可回答问题却给出事实答案的比例； +- `abstention_accuracy`：该拒答时是否拒答； +- `retrieval_recall`：黄金证据召回率； +- `citation_precision` / `citation_completeness`； +- `risk@coverage` 曲线。 + +任何准确率目标都必须同时预注册 coverage 与 required-facet 下限，并按语言、表格、多跳、时效问题等关键切片分别报告。否则系统可以通过全拒答，或只回答最安全的一个次要分面，得到虚假的“100%”。 + +少答任一 required facet 的结果标为 `PARTIAL_ABSTENTION`，不能计作 complete answer；混合回答要逐 facet 标明 `VERIFIED` 或 `ABSTAINED`。回答一个安全但非 required 的背景事实不算 answered。 + +不使用 LLM 自报的“95% 置信度”作为保证。置信等级应来自校准评测与证据状态： + +- 证据充分性； +- verifier 一致性； +- 来源与转换可验证性； +- 检索模式和覆盖度； +- 隐藏集上的经验错误率上界。 + +可以在有代表性的独立校准集上使用 conformal 方法，但要区分目标：TRAQ 类方法主要给“预测集合包含正确回答”的 coverage 保证，不等于单个自由文本答案正确；Conformal Risk Control 若用于回答/拒答，必须先定义嵌套阈值策略、单调有界损失，并用人工金标计算损失。二者都依赖样本可交换与分布稳定，提供的是总体边际保证，不自动覆盖语言/表格/多跳切片，也不是单条回答绝对正确的证明。 + +### 12.1 检索与证据投毒威胁模型 + +高 BM25 词频、重复文档、伪造权威元数据和 raw 文本中的提示注入都可能把错误证据推到前排。严格模式还应增加： + +- raw 内容始终按不可信数据处理，不能执行其中的指令； +- 同源重复与近重复去重，单来源设置候选上限； +- source identity、source hash、版本、来源类型与人工可信等级参与证据选择； +- 生成式扩写不得改变问题中的实体、时间、否定和范围约束； +- verifier prompt 明确隔离“证据文本”和“系统指令”； +- 对高排名结论强制寻找独立来源或相反证据； +- eval 加入 BM25 关键词堆叠、重复注入和提示注入案例。 + +## 13. 评测与统计认证 + +### 13.1 四套黄金集 + +1. `evals/conversion_fidelity/`：原格式 → Markdown 保真； +2. `evals/retrieval/`：query → 允许的最小充分黄金 evidence sets； +3. `evals/cite-check/`：atomic claim ↔ evidence entailment； +4. `evals/e2e/`：问题 → 已认证答案 / 正确拒答。 + +每条端到端样本至少标注： + +- 可回答 / 不可回答； +- required facets； +- 原子黄金事实； +- 一个或多个允许的最小充分证据集合； +- 允许的替代 anchor； +- 支持 / 部分支持 / 不支持 / 矛盾； +- 推算操作数与公式； +- 来源版本、时效和冲突状态。 + +还要分别标注人工黄金 material claims、required facets 与 evidence slots；`facet-plan recall`、claim extraction coverage、citation completeness 都以人工黄金为分母，不能以系统自己抽出的 facets/claims/slots 为分母。 + +至少两名独立标注者 + 仲裁；标注者看不到系统预测。开发集与隐藏发布集按 document/query group 切分，近重复文档不得跨集。标注质量同时报告 raw agreement、逐标签分歧率、仲裁率和 rare critical label 的表现；κ 或 Krippendorff α 按标签结构预先选择，不能用一个模糊阈值替代最终仲裁。 + +检索主指标定义为： + +```text +Complete Evidence Set Recall@k(q) = 1 +iff 存在一个允许的最小充分集合 S，且 S 完整包含在该 query 的 top-k 候选内 +``` + +另报告 slot-level recall、strict-anchor recall、alternative-anchor-compatible recall；固定 k 和 candidate budget，以 query 做 macro average，不能要求召回所有互为替代的 anchor，也不能把同一 query 的多个块当成独立样本。 + +### 13.2 毒化与困难干净集 + +毒化案例必须覆盖： + +- 数字、单位、量级、正负号和方向； +- 主体/客体/分母/指标角色互换； +- 否定、条件、范围、地区、时间和版本删除； +- 相关性偷换因果； +- “部分”改“全部”； +- sibling / cross-section / cross-document 错锚； +- table row/column、figure、footnote 错配； +- 缺失引用遮蔽其他可核验错引； +- `[需要来源]` / `[KB 推算]` 绕过； +- 支撑文本位于截断点之后； +- 同数值不同实体的干扰段； +- OCR 粘连、列错序和丢否定词； +- prompt injection 文本。 + +困难干净集要等量覆盖合法舍入、跨语言转述、格式差异、多源综合和合法推算，控制误报。 + +毒化变体按 base query/document 成簇；统计单元是独立 base case，而不是把同一基础样本的几十个最小扰动当几十个独立样本。对抗平衡集与生产分布集分别报告，不能混成一个“真实线上错误率”。 + +conversion eval 也要预注册每种主要格式的样本量和阈值：关键数字、正负号、单位、日期、否定词、表头—单元格关系与阅读顺序的黄金集要求观测 100% 保存；其他 OCR/布局指标再报告区间。fidelity 不能由 converter 自报，必须来自黄金映射或独立视觉复核。 + +### 13.3 发布闸门 + +对 changed files / query drafts： + +- broken、bare、coarse、mismatch、imprecise、unverifiable、pending-source、source issues、suspect/ledger inconsistency、provenance findings 均为 0； +- 所有原子事实 claim 都有 claim→evidence 映射； +- 每条 changed evidence edge 的现行 verdict 属于 `SUPPORTS_SLOT | CORROBORATES_SLOT`，每个 required slot 有直接支持，每个 changed claim-evidence set 为 `VERIFIED`；旧 `PARTIAL` 不得当作认证成功； +- checker 异常 fail-closed； +- pre-commit 在 staged blob 的临时只读树上执行； +- CI 用可公开 raw eval fixture 断言端到端阈值；raw 未分发的 demo 只做结构检查并明确不可语义认证，不再把“页面非空”冒充质量证明。 + +以上是**运行时操作闸门**，不是准确率证据。统计认证只能来自独立隐藏金标或独立人工抽审；内部 `VERIFIED` 状态不能用来评价 verifier 自己。 + +隐藏评测建议： + +- 已知确定性毒化：观测 recall 100%，困难干净 FP 0； +- semantic verifier sensitivity 与 specificity 的单侧 95% 置信下界均 ≥99%； +- Complete Evidence Set Recall@k 与 `human_gold_fully_grounded_precision` 的单侧 simultaneous 95% 置信下界均 ≥99%； +- `answerable_coverage`、`required_facet_recall` 和 `fully_grounded_coverage` 必须达到预注册下限；没有 coverage 门槛的 100% precision 不得作为发布结论； +- 不可回答集的 `false_answer_on_unanswerable` 必须有独立上界； +- 主体互换、结论反转、矛盾引用等关键类别不允许漏检； +- verifier 固定版本，冻结“多次运行如何聚合”的决策策略；同一 case 重复运行不得增加有效样本量，最终按聚合后的 answer/abstain 行为认证； +- 模型、prompt、converter、parser 或检索排序变化后重新认证。 + +coverage 下限必须在查看隐藏集结果前由 release profile 冻结，并同时规定整体与语言、表格、多跳、时效等关键切片下限；分母为 0 时指标是 undefined，不是 100%。自动系统与 human-assisted 系统分开计分，后者还要报告人工介入率和预算。 + +零错误样本量的含义： + +| 观测结果 | 单侧 95% 错误率上界（约） | +|---:|---:| +| 0 / 299 | 1.0% | +| 0 / 500 | 0.60% | +| 0 / 2995 | 0.10% | + +因此，“20/20 通过”只说明这 20 个 case 没出错，不能支持 1% 级风险声明。 + +上表还要求每个分母是独立、代表性的 Bernoulli 评测单元。299 是**每个指标自身**的有效分母：sensitivity 需要独立错误正例，specificity 需要独立干净负例，answer 指标需要独立 query/answer；299 个总样本不能同时证明每种语言、表格、多跳和时间敏感切片都低于 1%。每个需要单独声明的总体都要有足够样本，或采用预注册的多重比较校正。若四个主指标要求 family-wise simultaneous 95% 置信，可用 Holm/Bonferroni 等预注册方法；零错误、1% 门槛时每项所需样本会高于 299（简单 Bonferroni 四项约 437）。小分片“固定 case 零漏检”可以作为工程硬闸门，但不能自动升级成 99% 统计声明。 + +多条 claim 同属一份答案时存在集群相关，优先使用 query/答案级 all-or-nothing 指标；若必须做 claim 级区间，应使用 cluster-aware bootstrap 或分层模型，不能把相关 claim 当成独立样本虚增 n。长期重复发布会间接污染隐藏集，应由独立维护者保管并周期轮换。 + +### 13.4 独立线上抽审 + +不能用 verifier 自己的台账 verdict 评价 verifier。从已经判 `SUPPORTED` 的论断中分层随机抽样并独立人工双审，只能估计 accepted set 的残余错误率 / false-accept risk（或 precision/NPV 口径），不能估 sensitivity / specificity。完整混淆矩阵必须在同时覆盖 accepted 与 rejected、且对目标流量具有代表性的独立金标样本上建立；若采用分层/病例对照抽样，部署侧 PPV/NPV 与总体风险还要按生产基率重新加权。 + +## 14. 实施优先级 + +### P0：先消灭假绿，建立可认证答案 + +1. 修正 eval 的毒化/干净分类，扩大 fail-open、截断、缺 raw、复合句和定性论断案例。 +2. 先实现最小可用 V2 schema：source SHA/parser version、外部 claim map、decontextualized atomic claim、slots、canonical evidence handle、`edge_verdict`、`set_verdict` 与 content-addressed attestation。没有这层，`strict` 只能复用当前假绿。 +3. 新增 `check-draft --strict`，只消费上述 packet；把 broken、canonical mismatch、unverifiable、imprecise、pending、semantic、set sufficiency、raw leaf 和 provenance 全部纳入闸门。 +4. 去掉 claim/evidence 静默截断；多来源盲填值分歧必须报 conflict；至少两种真正独立的外部验证范式进入 attestation。 +5. `[KB 推算]` 改为原子值级豁免；缺失引用不得遮蔽其他引用的独立核对；旧 PARTIAL 一律重审。 +6. audit 失败与 CAUTION / pair_id 原子写入；health 统计现行失败 verdict。 +7. pre-commit 改为检查 staged blob，所有检查器故障 fail-closed。 +8. Debug Console 接入同一 strict 后验；同时修 H-anchor 不验、任一数字命中即放行、定性 judge 条数上限与 fail-open、已读来源登记不一致。接入前禁用“已认证”标签，错引不得降为整页引用。 +9. 提交一套许可清晰的 raw eval fixture，让 CI 能跑完整原文级链路；公开 demo 缺 raw 时只报告不可认证。 + +P0 先消除当前链路的假绿，但在 P1 的格式专用 source locator 与 conversion gold gate 完成前，不得把 P0-alpha 宣称为“已验证原始载体保真”的最终认证模式。 + +### P1：提高证据召回与原始来源保真 + +1. 先修 canonical full-path 解析、basename 串页与 ambiguity，再把图谱/backlinks 纳入召回。 +2. ✅ 已落地自然单元 SQLite FTS5/BM25 + 中文字符 trigram。 +3. ◐ 已落地多通道候选与 RRF；完整 query-plan schema、来源多样化校准与 retrieval certificate 待完成。 +4. ◐ 已建 `evals/longdoc/` 固定开发集、`evals/holdout/` 外部 bundle 协议与 public smoke；独立 hidden bundle 仍待外部提供。 +5. ◐ 已落地 layout-aware PDF 转换与 `evals/conversion_fidelity/` 的 PDF/DOCX/HTML 合成源坐标闸门、source/semantic SHA receipt；任意真实文档的 origin locator、OCR/图表/公式与视觉对齐仍待完成。 +6. ◐ list item / table row 已有精确派生 evidence handle；Markdown 子锚、cell 与 figure caption 细分待完成。 +7. ✅ outline 新鲜度已改为 schema + 全文/章节 SHA-256 + 完整结构校验。 + +### P2：Evidence-first 生成与语义充分性 + +1. 在 P0 最小 packet 上实现自动化的最小充分集选择与反证检索。 +2. 实现 Reference → Atomic Claim 写作协议与确定性 renderer，禁止最终润色新增事实。 +3. 扩展 verifier 插件、分歧升级与人审队列，但保持 P0 的独立性底线。 +4. 把递归 wiki → raw 与 claim DAG 可视化、解释化。 +5. CLI、Web、Debug Console 共享同一严格核验服务和状态语义。 +6. 通过 CLI/API/UI 显式提供 `verified` / `exhaustive` 模式。 + +### P3：统计认证与运行治理 + +1. 建隐藏发布集和独立人工抽审流程。 +2. health 页面展示 risk-coverage、检索 recall、fully grounded rate、verifier 混淆矩阵和统计上界。 +3. 保存 model/prompt/verifier 版本与可复现审计包。 +4. 模型、prompt、parser 或检索配置变化时自动触发重新认证。 + +## 15. 不建议的捷径 + +- **只增加 top-k**：召回可能上升，但噪音和错配也上升；没有 sufficiency 与 rerank 仍不可控。 +- **只换更强 LLM judge**：同一 judge 既分解 claim 又判 evidence 会产生相关错误；没有黄金集无法知道漏检率。 +- **只增加 embedding**：违反项目约束，也不能给出召回完整性或引用蕴含保证。 +- **把 whole-page 引用当安全降级**：只是隐藏了错锚，不会让错误论断变正确。 +- **让模型自报置信度**：未经校准的概率不是统计保证。 +- **用 verifier verdict 评价 verifier**：这是自证循环。 +- **要求所有问题都必须回答**：与“尽量接近 100% precision”直接冲突。 + +## 16. 研究依据 + +- [ALCE](https://arxiv.org/abs/2305.14627) 将 citation recall 与 citation precision 分开，并使用 NLI 检查 statement 是否被引用段落支撑；这直接支持“正确性与完整性分开度量”。 +- [Query2doc](https://aclanthology.org/2023.emnlp-main.585/) 用 LLM 生成伪文档扩写查询，再交给 BM25，并在其 ad-hoc IR 测试集上改善检索效果；这支持把它作为可校准词法分支，但不构成每类查询的召回保证。 +- [MuGI](https://aclanthology.org/2024.findings-emnlp.103/) 系统研究 LLM query expansion，支持多次扩写、保留原查询权重、平衡原查询与伪文档并结合伪相关反馈。 +- [Reciprocal Rank Fusion](https://cormack.uwaterloo.ca/cormacksigir09-rrf.pdf) 提供了简单、稳健的多排名融合方法，适合合并精确词、BM25、图谱与浏览通道。 +- [RAGChecker](https://arxiv.org/abs/2408.08067) 强调 claim-level 诊断，并把 retrieval 与 generation 的错误分开衡量。 +- [MiniCheck](https://aclanthology.org/2024.emnlp-main.499/) 说明 grounding fact-checking 可以由外部小模型高效执行，但它仍应作为 verifier 之一而非真值本身。 +- [Sufficient Context](https://arxiv.org/abs/2411.06037) 说明“上下文是否足够”与“论断是否被某段蕴含”不同，并用 sufficient-context 信号做选择性拒答。 +- [Do RAG Systems Cover What Matters?](https://aclanthology.org/2025.naacl-long.301/) 说明开放式问题需要先分解 core/background/follow-up 子问题，并单独评估覆盖度。 +- [Ground Every Sentence / ReClaim](https://aclanthology.org/2025.findings-naacl.55/) 采用 reference 与 claim 交替生成，支持本设计的 Reference → Atomic Claim 路线。 +- [A Closer Look at Claim Decomposition](https://aclanthology.org/2024.starsem-1.13/) 说明 claim decomposition 会显著影响事实性评估，因此原子分解器本身也必须进入黄金评测。 +- [Claimify](https://aclanthology.org/2025.acl-long.348/) 进一步指出事实抽取要同时处理覆盖度、去上下文化和歧义，无法可靠消歧时不应强行制造原子论断。 +- [CiteEval](https://aclanthology.org/2025.acl-long.1574/) 指出只用 NLI 做 citation evaluation 是不完整的代理，需要考虑问题、候选证据池、来源质量、冗余与遗漏证据。 +- [Verify with Caution](https://aclanthology.org/2025.findings-acl.1175/) 展示多种事实核验指标之间的不一致和对改写/远距离证据的偏差，支持“所有 verifier 必须在本库黄金集上重新校准”。 +- [TRAQ](https://aclanthology.org/2024.naacl-long.210/) 提供预测集合 coverage 思路；[Conformal Risk Control](https://research.google/pubs/conformal-risk-control/) 在满足嵌套策略、单调有界损失和校准分布假设时控制总体风险。两者都不是逐答案真值证明，CRC 才可能经明确定义的损失用于回答/拒答阈值。 +- [RAGuard](https://proceedings.neurips.cc/paper_files/paper/2025/hash/ed25c00ff6900989116d3ba5d607d33d-Abstract-Datasets_and_Benchmarks_Track.html) 与 [PoisonedRAG](https://www.usenix.org/conference/usenixsecurity25/presentation/zou-poisonedrag) 说明误导性检索与语料投毒会系统性破坏 RAG，因而投毒、冲突和伪权威来源必须进入对抗 CI。 + +## 17. 最终建议 + +GroundMap 不需要改成传统 embedding RAG。更合理的升级是： + +> **保留完整页面/章节导航与 Markdown 块锚点，把检索升级为自然结构单元上的多路词法高召回，把写作升级为证据先行，把验证升级为原子论断级充分性与反证，把产品目标升级为“可认证回答 + 可校准拒答”。** + +实施顺序必须是： + +1. 先修评测，并建立最小 claim/evidence packet、独立验证与 fail-closed strict 闸门； +2. 再提升词法召回和原始格式保真； +3. 再把 evidence-first、反证与人审流程产品化； +4. 最后用隐藏集和独立人审给出统计声明。 + +这条路线不能诚实地承诺开放世界绝对 100%，但可以做到：结构不变量 100% 执行、有限评测观测 100%、已回答内容的残余风险有上界、每条证据链可复核，并把不能证明的内容挡在答案之外。 diff --git a/docs/citation-integrity.md b/docs/citation-integrity.md new file mode 100644 index 0000000..30e82cd --- /dev/null +++ b/docs/citation-integrity.md @@ -0,0 +1,80 @@ +# 引用保障体系：如何让 wiki 和问答引用尽量正确 + +> 本文记录当前已实现的防线。2026-07 已落地 [引用准确性 V2 设计提案](citation-accuracy-v2-design.md) 的 P0 假绿清除、长文自然单元召回，以及公开 answer→citation / 外部 holdout 协议 / PDF-DOCX-HTML 格式保真闸门。生产链路的完整 claim-slot/set 充分性、递归 raw-leaf 认证、任意真实载体视觉对齐与独立隐藏金标统计认证**仍未全部实现**。 + +GroundMap 的核心承诺是**可溯源**：wiki 里所有实质论断都带块级引用锚点（`[[raw/X#^p-12-7d8e9a]]`）。但「有引用」不等于「引对了」——agent 可能引一个真实存在、内容却不符的锚点，数字抄错、张冠李戴、过度概括。本文说明 GroundMap 如何系统性地保证**引用的出处正确、内容正确**。 + +设计遵守三条架构底线：知识库核心零 LLM（语义判定全部由外部 agent 完成，KB 只出确定性数据）；markdown 是唯一真相源（所有验证状态里，「知识状态的改变」落 markdown 进 git，可重建的审计流水放 `.cache/` 派生层）；人是最终闸门（`status: reviewed` 只能人给）。 + +## 八层防线 + +| 层 | 机制 | 抓什么 | 成本 | +|---|---|---|---| +| 1 写入纪律 | **quote-first**：写数字/引文前必须 `read-block` 打开原文抄写 | 记忆漂移（错引的根源） | 零 | +| 2 确定性核对 | `k.py list-cite-mismatches`：论断中的数字/逐字引文必须出现在被引块原文（数值按声明精度舍入容差、单位敏感、list 块按条目分解） | 数字抄错、量级/单位错位、锚点挂错节、条目级张冠李戴 | 零 LLM，秒级 | +| 3 检索凭证 | `k.py check-provenance`：每条新引用必须有「取回过被引块**当前内容版本**」的检索凭证（read-block 等命令自动登记） | 没真读过原文的引用、读旧版引新版 | 零 LLM | +| 4 盲填复核 | `extract-claims --cloze` 把数字挖成 ⟦N1⟧，核验 LLM **只看挖空论断+被引原文**填空，`cloze-check` 机器判分 | 判定式审核的附和偏差；数字巧合在场但归属错误 | 每对一次小调用 | +| 5 对抗回验 | fresh-context 核验员以「尽力反驳」立场判定语义支撑（kb-ingest 第 9.5 步）；可用 `tools/cite-audit` 换 DeepSeek 跨模型二审 | 曲解、主体张冠李戴、过度概括、因果错置 | 每对 1-3k token | +| 6 增量审计 | 验证台账 + 双向内容 hash：claim 或被引块内容一变，自动回到待审（`kb-cite-audit` 周配额清偿） | 审过之后内容漂移（含 raw 重转换） | 增量 | +| 7 问答严格闸门 | `check-draft --strict`：broken、非 canonical 锚、raw 不可得、imprecise、pending/bare/coarse、无 provenance、非 `SUPPORTED`、截断核验包任一项均拒绝；`tools/cite-audit --draft` 可完成问答草稿取证与入账 | 答案端把「没检出」误当「已证明」 | 按引用对调用 | +| 8 机械闸门 + 人审 | pre-commit 在 Git index 的 staged blob 临时树上运行，checker / Python / JSON 异常全部 fail-closed；审计未通过落 `[!CAUTION]` 标注交人裁决 | 工作树与待提交内容不一致；工具故障伪造绿灯；语义层假阴的终审 | — | + +外加两件配套： + +- **对抗评测集**（`evals/cite-check/`）：20 个 case 明确分为确定性毒化 8、语义毒化 4、干净 8（总毒化 12），不再把「确定性层抓不住的语义毒化」误算成干净。runner 使用结构化终态 + 退出码；无 key、截断、无效 quote、跳过或协议异常不得靠 stdout 图标假绿。2026-07-11 本次实测：确定性毒化 8/8、干净误报 0/8；`deepseek-v4-flash` 语义毒化 4/4（进入语义层的全部毒化 6/6）、干净误报 0/8。这只是固定小集的本次观测，不是 100% 统计证明。 +- **第二阶段分层评测**（`evals/run_stage2.py`）：holdout 协议把 Gold/分母与 workspace 隔离，拒绝重复问题和伪切片；answer→citation 把 canonical claim text、value、最小证据集绑定成不可交叉拼接的 accepted variant，同时核对拒答；conversion-fidelity 把事实/限定/真实表格/章节/顺序/独立文本/列表/脚注绑定到源格式坐标并做 mutation。2026-07-12 public/synthetic gates 全绿，但终态固定为 `hidden_certification=false`，不代表生产模型或开放世界 100%。 +- **统计保证**（`k.py audit-confidence`）：把「审过多少、发现多少」换算成验收抽样口径的 Clopper-Pearson 置信上界——「95% 置信下已审引用未通过率 ≤ x%」，诚实边界随结果输出。 + +## 日常怎么用 + +**摄入时**（kb-ingest 流程自带，无需额外操作）：提交前闸门要求 `list-cite-mismatches` 的 mismatch 为空、`check-provenance --changed` 全空、fresh-context 回验 UNSUPPORTED/CONTRADICTED 清零，log.md 留 citation-verify 对账行。就算 agent 跳过流程，pre-commit hook 也会拒绝 staged 文件里带错引的提交。 + +**查询时**（kb-query 流程自带）：实质性 KB 回答必须走三步；任一步非 0 都不得交付「已验证」引用。 + +```bash +python scripts/k.py --workspace check-draft /tmp/answer.md +python tools/cite-audit/audit.py --workspace --draft /tmp/answer.md +python scripts/k.py --workspace check-draft /tmp/answer.md --strict +``` + +Debug Console 同样采用 fail-closed 后验：只有本轮实际取回过的 canonical `path#anchor` 才算已读；错路径 fallback、hash 恢复、粗粒度、无判官或未通过语义核对的引用都去链接化，不会「降级成整页链接」继续暗示支撑。 + +**周期维护**： + +```bash +# 增量语义审计（同 seed 取样可复现，跨周覆盖累积） +python scripts/k.py extract-claims --unaudited-only --sample 20 --seed 2026-W27 --json + +# 跨模型二审（需 DEEPSEEK_API_KEY；独立执行器，可挂定时任务） +python tools/cite-audit/audit.py --workspace --unaudited-only + +# 台账 ↔ markdown 对账（堵「删标注蒸发」）+ 统计保证 +python scripts/k.py list-suspect-citations --check-ledger +python scripts/k.py audit-confidence +``` + +**人的角色**：审计未通过的论断带着 `[!CAUTION] 引用审计未通过` 标注等你裁决——修论断/换锚（内容变化自动触发重审），或确认误报（标注改写为一行 NOTE 留痕）。`status: reviewed` 永远只由人设置。 + +## 防作弊设计 + +验证体系自身也是攻击面，这些口子是特意堵死的： + +- **橡皮图章**：agent 记 SUPPORTED 必须附被引块现行原文的字面子串（`--evidence`，k.py 校验）——不能证明「比对过」，但强制「至少取回过」。 +- **附和偏差**：盲填通道的核验者从头到尾看不到期望数字，附和无从谈起。 +- **删引用洗白**：对 UNSUPPORTED 论断「删引用挂 [需要来源]」过闸被流程明令禁止。 +- **滥标推算**：`[KB 推算]` 豁免必须带依据锚（`[KB 推算: ^t-33-0c8446]`），裸标记本身是闸门项；豁免块数进 health 供周检抽查。 +- **删标注蒸发**：台账判定未通过、标注却被删而论断未改——`--check-ledger` 对账当场报出。 +- **同源盲区**：写作与回验若同模型有相关性失误，`tools/cite-audit` 用另一家模型（DeepSeek）交叉复查，且作为独立执行器不依赖会话内 agent 的自觉。 +- **错锚扩大洗白**：严格模式拒绝 hash 恢复、basename 猜测或 Web fallback 后的「成功内容」；必须先把引用改写为返回的 canonical path/anchor 再复验。 +- **上下文漂移**：pair 身份绑定所属 heading 等去歧义上下文；只改「Alpha → Beta」而保留「它」的论断时，旧 `SUPPORTED` 不得复用。 + +## 诚实边界 + +- 数字类论断接近机器级保证；**定性论断的语义蕴含没有决定性算法**，任何验证器（包括人）都有非零错误率——所以上限的形状是「确定性可判的子集持续扩大 + 剩余部分错误率有统计上界 + 全程留痕可审计」，而不是 100%。 +- `strict passed` 只表示当前 parser / 召回 / verifier 组合的所有已实现闸门均通过，**不是验证器对自己的 100% 准确性证明**。只有独立隐藏金标 / 人工双审才能估计残余假接受风险。 +- 公开 synthetic conversion gate 已覆盖三份 PDF/DOCX/HTML fixture，但生产文档的 OCR、复杂公式、旋转文字、跨页/合并单元格表格、图表语义和持久 origin locator 仍未全覆盖；代表性 hidden 评测也尚未由独立方提供，因此不宣称开放世界 100%。 +- 语义核对只在 raw 在场的环境执行；本仓 demo 库不分发 raw（版权），相关引用显式标 `unverifiable`，**不假装验过**。 +- 验证的是「论断忠实于被引来源」；来源本身错了属于知识冲突的领域（见冲突处理规范）。 +- 台账与检索凭证都在 `.cache/`（派生层）：删了不丢任何知识，只是全部回到待审/待取回，重跑即重建。 + +> 各机制的完整行为规范见 `CLAUDE.md` 的「引用语义审计规范」节与 `.claude/skills/` 下 kb-ingest / kb-cite-audit / kb-query / kb-lint 各工作流。 diff --git a/docs/evaluation-stage2-design.md b/docs/evaluation-stage2-design.md new file mode 100644 index 0000000..644d019 --- /dev/null +++ b/docs/evaluation-stage2-design.md @@ -0,0 +1,103 @@ +# 引用准确性第二阶段评测设计 + +> 状态：协议、公开 smoke 与格式保真闸门已实现；独立外部 hidden bundle 尚未提供 +> 日期：2026-07-12 + +## 1. 目标与诚实边界 + +第一阶段固定长文开发集证明了转换后 Markdown 的自然单元可以完整建图，并在已知合成分布上达到 100% 检索代理指标。第二阶段补三条正交防线： + +1. **外部留出集**：Gold bundle 与被测 workspace、检索器实现和 public smoke 数据分离； +2. **端到端回答—引用**：不只看 top-k，而是检查系统实际选择回答还是拒答、输出了哪些原子论断、每条论断引用了什么； +3. **源格式转换保真**：验证 PDF、DOCX、HTML 中的关键事实、限定词、独立文本、列表、表格/脚注关系和阅读顺序是否在进入 Markdown 前已经丢失。 + +这些评测仍不能证明开放世界绝对 100%。当前 public holdout smoke 明确复用公开 longdoc 合成分布，只证明外部 bundle 协议与回归防线可运行；真正的发布认证需要未参与开发调优的一方提供独立 bundle，并在查看结果前冻结 bundle digest、阈值和切片分母。 + +## 2. 统一防作弊协议 + +- Gold 不得写入被测 workspace，也不得进入检索字段、query expansion、章节摘要或答案 prompt。 +- runner 必须从独立 manifest 取得分母；API 自报的 case 数、unit 数或 coverage 不能替代 Gold。 +- bundle 必须携带 schema version、内容 SHA-256、case/slice 配额和阈值 profile；运行配置另携唯一 run id。任一漂移或分母为 0 均 fail-closed。 +- case 的规范化问题必须唯一；hidden profile 至少 200 个 unique questions。`poison`、`unanswerable`、`multi_hop`、`multilingual` 切片由证据结构与脚本特征机械判定，不能靠任意标签膨胀分母。 +- public smoke bundle 与 external holdout bundle 必须在报告里分开标识；随机 seed 只能证明位置/排列变化，不能冒充独立分布。 +- 所有精度指标必须同时报告 coverage、abstention 和 required-facet completeness，禁止用全拒答换取高 precision。 +- 统一 runner 必须精确核对每个子评测的应有指标集合、schema/status/scope、分子/分母/value 等式、阈值方向与重算 verdict；只提交 `dummy: {passed: true}` 或降低阈值必须判协议错误。 +- runner 自身必须有 mutation tests：删 case、改分母、泄漏 Gold、伪造 citation、删否定/单位/表头、重排证据时至少一个闸门必失败。 + +## 3. 预注册指标 + +### 3.1 外部留出集发布 profile + +外部 bundle 建议至少 200 个 base cases，并包含 technical、policy、bilingual、table、3–5 hop、conflict/temporal、unanswerable 和 same-value-wrong-subject 切片。查看结果前冻结以下门槛： + +| 指标 | 门槛 | +|---|---:| +| Complete Evidence Set Recall@20 | ≥ 98% | +| selected evidence precision | ≥ 99% | +| answerable-case fully-grounded evidence coverage | ≥ 98% | +| forbidden evidence selection rate | 0% | +| unanswerable poison rejection@20 | 100% | +| 每个关键切片最小 case 数 | ≥ 20 | + +公开 smoke 集可要求观测 100%，但不得把它写成隐藏集认证。 + +### 3.2 端到端回答—引用 profile + +答案协议以原子 claim 为核对单位。每条 claim 至少包含 `facet_id`、规范化 value、canonical claim text 和一个或多个精确 evidence handles；系统也可返回 abstention。public/hidden Gold 为每个 facet 注册一个或多个不可拆分的 `{canonical text, value, evidence set}` 变体，一条 claim 必须整体命中同一变体。这同时防止展示正文写错主体/否定词，以及从不同等价答案交叉拼接文本、值与证据而假绿。 + +| 指标 | Public smoke | External holdout | +|---|---:|---:| +| claim→citation precision | 100% | ≥ 99% | +| citation completeness | 100% | ≥ 99% | +| required-facet coverage | 100% | ≥ 95% | +| answer coverage | 预注册且 > 0 | ≥ 90% | +| correct abstention | 100% | ≥ 98% | +| unsupported claim rate | 0% | ≤ 1% | +| fully-grounded answer rate | 100% | ≥ 95% | + +free-form LLM 答案若无法确定性映射为注册的 canonical atomic claims，应判协议失败或进入独立盲审，不能由 writer 自评通过。公开 reference adapter 的满分只证明协议，不代表生产模型能力。 + +### 3.3 转换保真 profile + +每个源格式 fixture 必须由生成规则直接登记 Gold locator；Gold 不从转换后 Markdown 反推。关键事实包括数字、正负号、单位、日期、否定/例外、表头—数据行关系、脚注限定和阅读顺序。 + +| 指标 | 门槛 | +|---|---:| +| critical fact preservation | 100% | +| qualifier/negation preservation | 100% | +| unit/sign/date preservation | 100% | +| table header→row relation preservation | 100% | +| footnote marker→note relation preservation | 100% | +| registered section preservation | 100% | +| exact heading inventory / hierarchy | 100% | +| preregistered reading-order constraints | 100% | + +每条事实/限定/独立文本/列表/表格/脚注/顺序 Gold 必须绑定目标章节与自然块坐标，且全文只能唯一出现；表格必须在同一个真实 pipe table 内逐行精确对应，列表项必须同时保留文本、顺序与有序/无序类型。fixture 自身也必须通过 PDF 双栏坐标、DOCX XML 表格几何、HTML DOM 结构检查和逐页渲染目视 QA；文本抽取成功不能证明表格、两栏、脚注或分页布局真的被构造出来。 + +## 4. 当前公开闸门观测 + +统一入口： + +```bash +python evals/run_stage2.py +``` + +2026-07-12 的本地公开观测： + +| 层 | 结果 | +|---|---:| +| public holdout：自然单元 / 章节 | 744/744；36/36 | +| public holdout：回归 / CES@20 / facet top-1 | 51/51；50/50；60/60 | +| public holdout：最终 forbidden 选择 / unanswerable poison | 0/60；1/1 | +| answer→citation：精确引用对 / required facets | 8/8；7/7 | +| answer→citation：answerable / 正确拒答 / fully grounded | 4/4；2/2；4/4 | +| conversion：事实 / 限定 / 表格 / 顺序 / 独立文本 / 列表 / 脚注 | 6/6；6/6；3/3；4/4；3/3；2/2；3/3 | +| conversion：精确标题 inventory / hierarchy / mutation | 3/3；3/3；14/14 | + +统一终态是 `public-stage2-gates-passed`，`hidden_certification=false`。其中 answer public smoke 使用 Gold-blind 的确定性 reference adapter；未传入真实系统 predictions 时不测生产模型。PDF/DOCX/HTML 是三份合成格式 fixture；不外推到 OCR、复杂公式、旋转文本、跨页/合并单元格表格或开放世界文档。 + +## 5. 报告规则 + +每次评测至少输出 bundle/fixture digest、runner/schema version、case/facet/claim 分母、整体与适用的切片指标，以及明确的 `evaluation_scope` / `not_measured`。公共单项 runner 仅在显式使用 `--details` 时输出逐 case 诊断；hidden 模式禁止 `--details`，只输出聚合结果，也不回显失败 case id 或其他 per-case Gold 标签。公共单项状态写 `protocol-smoke-passed`，统一入口状态写 `public-stage2-gates-passed`，两者都不得写成 `holdout-certified`。runner 只能报告机械阈值是否通过，不能自行证明数据确由独立作者保管且在运行前未见。 + +外部 bundle 不进 Git。公开仓只保存 schema、runner、CC0 smoke fixture 和 mutation tests；真实隐藏结果可保存签名摘要与聚合指标，但不能回写会泄漏 Gold 的 case 文本或答案。 diff --git a/docs/quickstart.md b/docs/quickstart.md index 7c09f5b..38e4487 100644 --- a/docs/quickstart.md +++ b/docs/quickstart.md @@ -107,6 +107,7 @@ Read AGENTS.md first. Use KB_ROOT=~/work/my-kb-data and workspace my-research. I placed source documents under raw/papers/. Please ingest them into the knowledge base: - convert the raw files, +- rebuild the natural-unit evidence index and require 100% coverage/freshness, - create source summaries with block-level citations, - update the relevant concept/entity/index pages, - run list-bare-claims, list-coarse-citations, list-source-issues, list-broken-refs, and list-relation-issues, @@ -166,10 +167,22 @@ cp /path/to/my_article.html workspaces/my-research/raw/articles/ python scripts/convert.py --dir workspaces/my-research/raw/articles --ext .html # k.py path arguments are workspace-relative (resolved under the active workspace) -python scripts/k.py outline raw/articles/my_article.md +python scripts/k.py --workspace my-research rebuild-evidence-index +python scripts/k.py --workspace my-research evidence-index-coverage --json +python scripts/k.py --workspace my-research outline raw/articles/my_article.md ``` -Converted markdown and `.outline.json` files are derived artifacts. Do not hand-edit them; regenerate from the original source file. +Converted markdown, `.outline.json`, and `.cache/retrieval_index.db` are derived artifacts. Do not hand-edit them; regenerate from the original source file. The coverage command must report 100% natural-unit and section coverage with a fresh corpus before an ingest or detail-sensitive query relies on the index. + +To locate a detail that was not promoted into a wiki summary: + +```bash +python scripts/k.py --workspace my-research search-evidence "" --limit 20 +python scripts/k.py --workspace my-research read-evidence-unit +python scripts/k.py --workspace my-research read-block raw/articles/my_article.md +``` + +Search selects an exact natural unit (including one list item or table row); final Markdown citations currently point to that unit's canonical parent block. Always inspect both the selected unit and its parent block before citing it. The 100% coverage denominator is the current converted-Markdown parser inventory, not proof that a PDF/DOCX conversion preserved every source-layout element. ## Work With an External Agent diff --git "a/docs/raw-to-wiki-\346\265\201\347\250\213.md" "b/docs/raw-to-wiki-\346\265\201\347\250\213.md" index 239bc2a..e556589 100644 --- "a/docs/raw-to-wiki-\346\265\201\347\250\213.md" +++ "b/docs/raw-to-wiki-\346\265\201\347\250\213.md" @@ -9,20 +9,22 @@ ## 0. 一句话概括 -> **raw/ 是"原矿"，wiki/ 是"提炼物"。中间靠两件事衔接：`scripts/convert.py` 把原矿格式标准化（markdown + 锚点 + 大纲），agent 用 `scripts/k.py` 按章节阅读并把提炼后的论点写到 wiki/ 页面，每条实质性论断都用块级锚点（`^h-...` / `^p-...`）回溯到原文。** +> **raw/ 是"原矿"，wiki/ 是"提炼物"。中间靠三件事衔接：`scripts/convert.py` 标准化原始格式，自然单元证据索引把转换后文本的每个段落/条目/表格行变成可发现坐标，外部 agent 再阅读、综合并写入 wiki。实质性论断用块级锚点回溯到原文；摘要没写进去的细节仍可从原文索引找回。** -整条流水线分 5 个阶段： +整条流水线分 6 个阶段： ``` ① 放置原始文件 ← 人类 ↓ ② convert.py：转 md + 加锚点 + 生成 outline ← 脚本 ↓ -③ agent 阅读：outline → read-section → annotate-section（每节都调） ← 外部 LLM agent +③ 证据地图：rebuild-evidence-index → coverage 100% ← 脚本 ↓ -④ 写 wiki/sources/ 摘要页 + 含块级引用 ← 外部 LLM agent +④ agent 阅读：outline → read-section → annotate-section ← 外部 LLM agent ↓ -⑤ 更新核心节点 + 索引 + log + git commit ← 外部 LLM agent +⑤ 写 wiki/sources/ 摘要页 + 含块级引用 ← 外部 LLM agent + ↓ +⑥ 更新核心节点 + MOC + log + git commit ← 外部 LLM agent ``` > ⚠️ 注意第 0 条原则：**知识库本身不调用 LLM**。所有"阅读 / 综合 / 写 wiki"动作由外部 agent 完成。本仓库只提供 `convert.py` 和 `k.py` 两个工具，外加一个 web 管理台。 @@ -35,7 +37,7 @@ > **路径约定**：数据按主题隔离在 `workspaces//` 下，本文所有 `raw/` / `wiki/` 均指**当前 workspace 内**的目录（默认 `my-research`，即 `workspaces/my-research/raw/...`）。`scripts/k.py` 的路径参数是 **workspace 相对**的（写 `raw/papers/foo.md` 即可，不要写全 `workspaces/.../raw/...`）；`scripts/convert.py` 不传 `--dir` 时按 `--workspace` 自动扫 `workspaces//raw/`；`--dir` 是显式扫描目录，按运行命令时所在目录解析（通常在仓库根运行，即写**仓库相对**路径，如 `--dir workspaces/my-research/raw/articles`），且必须位于数据根内。切换 workspace 用 `--workspace ` / `KB_WORKSPACE=`。 -把任意支持的格式拖到当前 workspace 的 `raw/` 对应子目录： +把任意已识别的格式拖到当前 workspace 的 `raw/` 对应子目录： ``` workspaces//raw/ @@ -44,7 +46,7 @@ workspaces//raw/ └── assets/ # 配图、表格、独立资源 ``` -支持格式（来自 `scripts/convert.py` 的 `SUPPORTED_EXTENSIONS` 常量）： +可识别格式（来自 `scripts/convert.py` 的 `SUPPORTED_EXTENSIONS` 常量）： ``` .md .pdf .docx .pptx .xlsx .xls @@ -55,6 +57,8 @@ workspaces//raw/ .msg ``` +> `SUPPORTED_EXTENSIONS` 表示“扫描时不忽略并给出确定结果”，不等于每个格式都允许当前的确定性转换。独立图片（`.jpg/.jpeg/.png/.gif/.bmp/.tiff/.tif/.webp`）和音频（`.mp3/.wav`）会被识别，但 accuracy-first 模式必定以非零状态拒绝：ExifTool 只能提供元数据、不能保留视觉事实；现有音频路径依赖 ambient ffmpeg/ffprobe 与无不可变模型 revision 的 Google SpeechRecognition。这些来源必须先用专用 OCR/视觉或版本锁定转写流程产生可核验的 Markdown + receipt，不得把元数据或不可复现转写冒充完整摄入。 + ### 硬约束 | 路径 | 权限 | @@ -85,7 +89,7 @@ python scripts/convert.py --dir workspaces/my-research/wiki/ # 显式指定目 ``` 原始文件 (smith2024.pdf) - ↓ markitdown.MarkItDown().convert() + ↓ PDF: pdfplumber 几何布局恢复；其他可转换格式: MarkItDown markdown 纯文本 ↓ postprocess.process(markdown, doc_path) 含锚点的 markdown + outline 数据 @@ -93,6 +97,16 @@ markdown 纯文本 smith2024.md + smith2024.outline.json ``` +PDF 不再依赖内容流的偶然顺序：转换器会先识别表格，再按坐标重建视觉行，区分 +跨栏标题与左右栏，并以“左栏完整读完 → 右栏”的顺序输出；重复页眉/页脚和确认的页码 +会被保守去重，字体层级用于恢复 Markdown heading。扫描页、任一非已知 bullet 的无法映射 CID 字形、 +损坏文件，以及含内容级光栅/矢量图却没有可核验图像/OCR/图表产物的页面会明确失败，整批命令 +返回非零退出码，不会静默退回到可能交错或漏图的文本抽取。DOCX 转换还会修复 +“空表头 + 第一行全粗体”这一高置信结构，使表头—数据行关系进入后续自然单元索引。 +非 Markdown 来源先复制到系统临时目录的只读快照，转换器不直接读正在变动的 live raw，返回后还会重算快照 SHA。派生 outline 的 schema v2 receipt 同时记录 canonical `source_path`、source SHA-256、原格式、converter fingerprint 与 receipt 自校验哈希；fingerprint 包含 Python implementation/version、实际解析后端版本及 `conversion_receipt.py` / `convert.py` / `postprocess.py` / `section_parser.py` / `pdf_layout.py`（PDF）实现 SHA。原文字节、位置、后端环境或实现一旦变化，旧产物自动失效；同目标碰撞、派生 Markdown 二次处理、孤儿原载体与 symlink 在批处理写入前统一拒绝。证据索引 schema v5 也冻结并持续重验该 source binding，所以原载体变化或消失后 coverage 会变为 stale，search/read 会拒绝旧索引。转换/写入期间原文变动则本次明确失败，不依赖 mtime 猜新旧。这是受信任本地 converter 的一致性防护，不是恶意代码沙箱：同权限代码可改权限、读写后恢复原字节来绕过事后 hash，所以未审阅插件必须先放入 OS 级沙箱，不得直接交给本流程。 + +这些规则优先避免“看似成功但内容已错序/丢失”。对齐的窄左栏既可能是键值表也可能是真双栏；若缺少冒号/粗体 label/value-like 或双侧完整句等高置信信号，转换器会拒绝猜测并要求视觉复核。需要 OCR、复杂公式、旋转文字、跨页表格或含糊合并几何的来源，应先走专用预处理并做视觉复核；可验证的多级合并表头会展平为 `组 / 子组 / 叶列`。 + ### 锚点编号（`postprocess.py` 的 `add_anchors()`） 每个段落、heading、表格、代码块、图片块的**行末**追加一个空格 + 锚点： @@ -111,11 +125,13 @@ smith2024.md + smith2024.outline.json ### 幂等性 -`convert.py` 的 `should_convert()` 决定是否重新处理： -- 派生 `.md` 或 `.outline.json` 不存在 → 处理 -- 非 `.md`：原文件 mtime > 派生 `.md` mtime → 重转 -- `.md`：检查文中是否已有 ≥3 处锚点尾巴（`postprocess.py` 的 `has_anchors()`） -- 内容未变时输出与原文 byte-equal——不会污染 git 工作树 +`convert.py` 的 `should_convert()` 以 fail-closed 方式决定是否重新处理： + +- 派生 `.md` 或 `.outline.json` 不存在 → 重转； +- outline schema、`doc_sha256`、每节 `section_sha256`、canonical heading anchor、字符范围或嵌套结构任一不一致 → 重转； +- 原文件 mtime 仍是非 Markdown 来源的附加变更信号，但不再是 outline 新鲜度证明； +- 即使改写后字符数相同，全文 SHA-256 也会使旧 outline 失效； +- 只有章节内容 hash 仍相同时，旧 `agent_summary` 才能迁移，避免把旧摘要粘到已改写的章节。 ### 同时生成的 `outline.json` @@ -123,9 +139,11 @@ smith2024.md + smith2024.outline.json ```json { + "outline_schema_version": 2, "doc_path": "raw/papers/smith2024.md", "doc_chars": 87532, "doc_paragraphs": 234, + "doc_sha256": "<转换后 markdown 全文 SHA-256>", "generated_at": "2026-05-03", "sections": [ { @@ -135,6 +153,7 @@ smith2024.md + smith2024.outline.json "line": 1, "char_start": 0, "char_end": 234, + "section_sha256": "<该节归一化内容 SHA-256>", "preview": "Vaswani et al. ...", "agent_summary": null, "children": [ @@ -152,13 +171,31 @@ smith2024.md + smith2024.outline.json > `agent_summary` 字段一开始是 null，由 agent 阅读后用 `k.py annotate-section` 回填精排摘要——后续查询时直接看摘要就能判断这一节是否相关，不必每次都全文 read。 +## 3. 阶段 ③：建立“全细节证据地图” + +```bash +python scripts/k.py --workspace my-research rebuild-evidence-index +python scripts/k.py --workspace my-research evidence-index-coverage --json +``` + +> schema v5 是可重建的派生索引契约；从旧版本升级后必须重新运行 `rebuild-evidence-index`，旧 schema 会被明确拒绝而不会静默沿用。 + +`rebuild-evidence-index` 对每份转换后 raw Markdown 枚举 paragraph、单条 list item、单行 table row、blockquote、code 和 figure，同时登记所有结构章节与有内容章节。覆盖报告必须同时满足： + +- 自然单元 inventory 与 SQLite 物化的精确多重集、exact text hash、完整单位/章节 manifest 指纹一致； +- unicode61 与 trigram 两个 FTS5 物化通道均无缺失； +- 内容章节和结构章节覆盖率都是 100%； +- raw 文件集/全文 SHA-256、已验证 outline `agent_summary` 指纹，以及非 Markdown 原载体的 canonical path / source SHA / converter fingerprint / receipt hash 当前新鲜。原载体或转换后正文变动、原载体缺失、转换实现漂移，或 `annotate-section` 改摘要后，`search-evidence` / `read-evidence-unit` 都会拒绝旧索引。 + +**边界**：这个 100% 的分母是“当前转换后 Markdown 的 parser inventory”，它证明已转换文本的细节都已建坐标，不能单独证明任意原始载体绝无遗漏。仓库另提供 `python evals/conversion_fidelity/run_eval.py`，用合成 PDF/DOCX/HTML 的源坐标 Gold 检查数值、正负号、单位、否定/限定、真实表格关系、章节、独立文本、列表、脚注和阅读顺序；它仍只是公开合成保真闸门，不是开放世界证明。扫描 PDF、复杂公式、旋转文字、跨页/合并单元格表格和图片 OCR 仍需专用处理或人工抽检。若有人把原载体与其 outline receipt **同时删除**、只留下派生 `.md` 后再全量重建，仅凭剩余文件无法证明它曾是派生物；这属于真相源被成套移除的信任边界，需靠 Git/备份或未来独立 manifest 恢复，而不能伪称机器可辨。 + --- -## 3. 阶段 ③：agent 阅读 ——大纲驱动 + 章节级精读 +## 4. 阶段 ④：agent 阅读 ——大纲驱动 + 章节级精读 外部 agent（Claude Code、Cursor 等）**不允许**直接 `Read` 整个 PDF（数十万字符会撑爆 context），而是按以下顺序操作： -### 3.1 看大纲 +### 4.1 看大纲 ```bash python scripts/k.py outline raw/papers/smith2024.md @@ -185,15 +222,15 @@ python scripts/k.py outline raw/papers/smith2024.md ## Conclusion [^h-2-5-...] (line 312, 980 字符) ``` -### 3.2 决策分支：三档自决（CLAUDE.md "Ingest 操作流程" 第 3 步） +### 4.2 决策分支：三档自决（CLAUDE.md "Ingest 操作流程" 第 4 步） **agent 自决，不询问用户**。按字符数三档处理： | 档 | 字数（中文等价） | 策略 | 综合保真度 | |---|---|---|---| | ① **短文** | < 30K（约 3 万中文字） | `Read` 全文，一次性读完 | 高 | -| ② **中长文** | 30K – 150K（论文 / 报告级） | 按 H1 切块、每块 ≤ 3 万分段 `read-section`，每段读完 `annotate-section` | 高（多步但不漏） | -| ③ **整本书规模** | > 150K（专著 / 法规全文） | TOC 扫全 + AI 自决深读章节；**全部章节**登记到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过） | 中（透明声明深度差异） | +| ② **中长文** | 30K – 150K（论文 / 报告级） | 按标题树以 ≤ 3 万字的完整节为单位 `read-section`，每节读完 `annotate-section` | 综合有损，但未写入摘要的细节可由证据索引找回 | +| ③ **整本书规模** | > 150K（专著 / 法规全文） | TOC 扫全 + AI 自决深读章节；**全部章节**登记到 source_summary 的「## 章节深度登记」表（含状态：✓ 深读 / ⊙ 扫读 / × 跳过） | 摘要明示深度差异，全量自然单元仍可检索 | > **30K 上限的依据**：单次 Read 超过 30K 中文字符会触发 LLM "lost in the middle" 衰减——综合质量下降。所以即便 Claude Opus 是 1M context，也不在单次塞太多。 > @@ -201,7 +238,7 @@ python scripts/k.py outline raw/papers/smith2024.md > > **第 ③ 档的 ⊙ 扫读章节**保留 partial re-ingest 升级路径——后续 query 命中关键词、lint 探测高频被引、或用户在 web 端主动触发，都可让 AI 重读该章节升级到 ✓ 深读。详见 `.claude/skills/kb-ingest/SKILL.md` 「增量深化」节。 -### 3.3 读单节 +### 4.3 读单节 ```bash python scripts/k.py read-section raw/papers/smith2024.md h-2-3-0fdf24 @@ -209,9 +246,9 @@ python scripts/k.py read-section raw/papers/smith2024.md h-2-3-0fdf24 python scripts/k.py read-section raw/papers/smith2024.md "Model Architecture" ``` -返回该 H2 段从 `char_start` 到下一同级 heading 之前的全部原文，**不会切片**——保持完整段落、表格、公式不被截断。 +默认返回该节的完整原文，不生成任意 chunk。若整节超过 30K 字符，命令会 fail-closed；应下钻 H3/H4 子节，或先用 `search-evidence` 定位自然单元、再用 `read-evidence-unit` + `read-block` 逐块阅读。`--max-chars 0` 只是显式无限制逃生口，agent ingest 不应用它绕过 30K 阅读上限。 -### 3.4 读单块（精确到段） +### 4.4 读单块（精确到段） 写 wiki 时若要引用某段具体数据： @@ -220,7 +257,7 @@ python scripts/k.py read-block raw/papers/smith2024.md p-87-cd4741 # 输出仅该段原文（去掉行末锚点尾巴） ``` -### 3.5 反查锚点 +### 4.5 反查锚点 如果手上只有"某段文字"想找它的 anchor： @@ -230,7 +267,7 @@ python scripts/k.py find-anchor raw/papers/smith2024.md "BLEU 28.4" # ...we achieve a new state-of-the-art BLEU score of 28.4... ``` -### 3.6 回填精排摘要（**ingest 必经**，每个精读章节都要调） +### 4.6 回填精排摘要（**ingest 必经**，每个精读章节都要调） > **关键规则**：每读完一个 H2 / H3 章节，**立即**对它调一次 `annotate-section`。LLM 反正都把这节读了一遍，写一两句概括的边际成本接近零，但能给下次任何 agent / 查询提供章节级二级索引。 > @@ -260,7 +297,7 @@ python scripts/k.py annotate-section raw/papers/smith2024.md h-2-3-0fdf24 \ --- -## 4. 阶段 ④：在 `wiki/sources/` 创建摘要页 +## 5. 阶段 ⑤：在 `wiki/sources/` 创建摘要页 > **写前：AI 综合判断**——在打开模板填内容前，agent 已先用 `python scripts/k.py search` 反查 wiki 现状做了三件事综合：核心价值（新东西在哪）/ 关联（哪些 wiki 页有重叠）/ 冲突（哪些论断打架）。这些结论作为 source_summary 的「## AI 综合判断」H2 节固化下来。详见 `.claude/skills/kb-ingest/SKILL.md` 第 3 步。 @@ -300,17 +337,17 @@ python scripts/k.py annotate-section raw/papers/smith2024.md h-2-3-0fdf24 \ --- -## 5. 阶段 ⑤：更新核心节点 + 索引联动 + 收尾 +## 6. 阶段 ⑥：更新核心节点 + 索引联动 + 收尾 > 本节示例中的 DPO / Vaswani 等页面名仅作**格式示意**；落在本仓库的真实完整例子见 §6。 -### 5.1 立即更新最核心的 2-3 个节点 +### 6.1 立即更新最核心的 2-3 个节点 举例：刚 ingest 了一篇关于 DPO 的论文，则： - **必须立即更新**：`wiki/concepts/dpo.md`、`wiki/concepts/rlhf.md`（DPO 是 RLHF 的对比项） - **必须立即更新**：`wiki/indexes/ai_index.md` 的"近期更新"和"关键来源"小节 -### 5.2 给次要受影响页面打 `#to-be-updated` +### 6.2 给次要受影响页面打 `#to-be-updated` ```markdown --- @@ -320,7 +357,7 @@ python scripts/k.py annotate-section raw/papers/smith2024.md h-2-3-0fdf24 \ > 这是**懒标记**——不要求当下就改完所有相关页，但必须留下线索，供下一次 lint 流程消化。 -### 5.3 自动判断 MOC 归属 +### 6.3 自动判断 MOC 归属 agent 用 source_summary 的 `tags` 字段反查现有 MOC（`python scripts/k.py list-pages --type=index --json`）： @@ -329,7 +366,7 @@ agent 用 source_summary 的 `tags` 字段反查现有 MOC（`python scripts/k.p **agent 自决，不询问用户**。归属错了由 lint 流程检测后人工纠正。 -### 5.4 检查冲突 +### 6.4 检查冲突 发现新论文与既有 wiki 论断矛盾时，**禁止覆盖**，写冲突标注： @@ -341,7 +378,7 @@ agent 用 source_summary 的 `tags` 字段反查现有 MOC（`python scripts/k.p > **状态**：⏳ 待人类判别 ``` -### 5.5 追加 `log.md` +### 6.5 追加 `log.md` ```markdown ## [2026-05-03] ingest | Vaswani et al. 2017 - Attention Is All You Need @@ -352,7 +389,7 @@ agent 用 source_summary 的 `tags` 字段反查现有 MOC（`python scripts/k.p - 给 wiki/concepts/in_context_learning.md 打 #to-be-updated ``` -### 5.6 原子 commit +### 6.6 原子 commit ```bash # git 在仓库根运行，路径写全 workspaces//... @@ -370,11 +407,11 @@ git commit -m "ingest: Vaswani et al. 2017 - Attention Is All You Need" --- -## 6. 完整真实例子：Sheng Lu《Shein Lost Market Share in the U.S. ...》 +## 7. 完整真实例子：Sheng Lu《Shein Lost Market Share in the U.S. ...》 下面用一个**真实落在本仓库默认 workspace（`workspaces/my-research/`）**的例子走一遍：Sheng Lu（特拉华大学）2026-02 的博客文章《Shein Lost Market Share in the U.S. Apparel Retail Market in 2025 Amid Trade Tensions》。除 `raw/` 本身（被 `.gitignore` 排除、只存在于本地）外，最终产物都能在仓库里看到。 -### 6.1 起点：原始文件入库 +### 7.1 起点：原始文件入库 ```bash # 把下载好的网页 HTML 放进当前 workspace 的 raw/articles/ @@ -382,7 +419,7 @@ cp ~/Downloads/shein_us_market_share_2025.html \ workspaces/my-research/raw/articles/shein_us_market_share_2025.html ``` -### 6.2 跑 convert +### 7.2 跑 convert ```bash python scripts/convert.py # 默认 workspace = my-research @@ -405,7 +442,7 @@ python scripts/convert.py # 默认 workspace = my-research - `raw/articles/shein_us_market_share_2025.md`（含锚点的纯 markdown） - `raw/articles/shein_us_market_share_2025.outline.json`（结构化大纲） -### 6.3 看大纲决定怎么读 +### 7.3 看大纲决定怎么读 ```bash python scripts/k.py outline raw/articles/shein_us_market_share_2025.md @@ -426,11 +463,11 @@ python scripts/k.py outline raw/articles/shein_us_market_share_2025.md ... ``` -字符数 92466 落在第②档（30000 – 150000，见 §3.2 三档），所以**不能直接 Read 全文**，要按章节读。大纲进一步显示：这 92K 里约 64K 是博客读者评论（`h-2-2-464cca`），核心论点集中在文档头部——深读正文、跳过噪声节。 +字符数 92466 落在第②档（30000 – 150000，见 §4.2 三档），所以**不能直接 Read 全文**，要按章节读。大纲进一步显示：这 92K 里约 64K 是博客读者评论（`h-2-2-464cca`），核心论点集中在文档头部——深读正文、跳过噪声节。 -### 6.4 精读正文 + 回填摘要 +### 7.4 精读正文 + 回填摘要 -本文结构特殊：核心论点集中在 H1 开头的几个段落（`p-10` ~ `p-13`），后面挂着 64K 的评论区。所以直接用 `read-block` 精读关键数据段（结构干净的论文 / 报告则按 §3.3 用 `read-section` 整段读）： +本文结构特殊：核心论点集中在 H1 开头的几个段落（`p-10` ~ `p-13`），后面挂着 64K 的评论区。所以直接用 `read-block` 精读关键数据段（结构干净的论文 / 报告则按 §4.3 用 `read-section` 整段读）： ```bash # 核心数据段：美国服装份额 1.8% → 1.7%、销售额 -4.5% @@ -440,7 +477,7 @@ python scripts/k.py read-block raw/articles/shein_us_market_share_2025.md p-10-4 python scripts/k.py read-block raw/articles/shein_us_market_share_2025.md p-12-6c45f0 ``` -**读完（或决定跳过）一个章节，立即回填摘要**（§3.6 的必经步骤——"未深读"也要登记。下面这条就是真实仓库里这节 annotation 的来历）： +**读完（或决定跳过）一个章节，立即回填摘要**（§4.6 的必经步骤——"未深读"也要登记。下面这条就是真实仓库里这节 annotation 的来历）： ```bash python scripts/k.py annotate-section raw/articles/shein_us_market_share_2025.md h-2-2-464cca \ @@ -449,7 +486,7 @@ python scripts/k.py annotate-section raw/articles/shein_us_market_share_2025.md 回填后任何 agent 再跑 `k.py outline`，该节就从 `(预览)` 截断的头部变成 `(LLM)` 标的精排摘要——§6.3 的输出正是回填后的状态。 -### 6.5 检查重复 +### 7.5 检查重复 ```bash python scripts/k.py search "shein" @@ -457,7 +494,7 @@ python scripts/k.py search "shein" wiki 中已存在 `wiki/entities/shein.md` 实体页与另一篇 SHEIN 来源 `wiki/sources/analyzify_shein_stats_2025.md`——所以**不要**新建实体页，而是把新发现合并进 `shein.md`；两篇来源数字对不上的地方进「AI 综合判断 → 冲突」节（见 6.6）。 -### 6.6 写 `wiki/sources/shein_us_market_share_2025.md` +### 7.6 写 `wiki/sources/shein_us_market_share_2025.md` 最终产物（真实仓库文件，节选）： @@ -508,7 +545,7 @@ tags: - 横向出链 `[[wiki/entities/shein]]` `[[wiki/concepts/de_minimis_exemption]]`——让节点页与来源页双向连通 - 自身的段落锚点 `^h-1-1-...` `^p-1-...` 由 `convert.py --dir workspaces/my-research/wiki/` 自动加上 -### 6.7 更新核心节点 +### 7.7 更新核心节点 本次 ingest 实际更新了三个节点页 + 一个 MOC，都可在仓库里查证： @@ -538,7 +575,7 @@ tags: - 2026-05-20 — 全量首次建立（[[wiki/sources/modern_retail_tiktok_smb_66]] / [[wiki/sources/emarketer_tiktok_social_commerce]] / [[wiki/sources/shein_us_market_share_2025]] 等） ``` -### 6.8 给次要受影响页面打 `#to-be-updated` +### 7.8 给次要受影响页面打 `#to-be-updated` 例如 `wiki/entities/temu.md`（Temu 同受 De Minimis 冲击但缺专门来源）末尾的真实标记： @@ -548,7 +585,7 @@ tags: 后续 lint 流程（`python scripts/k.py list-to-update`）会列出所有 `#to-be-updated` 积压。 -### 6.9 写 log + commit +### 7.9 写 log + commit 本例实际是作为 demo 数据集批量 ingest 的一部分入库的，对应 `workspaces/my-research/log.md` 的「[2026-05-20] ingest | 跨境电商 demo 数据集」条目。若单独 ingest 一篇，log 条目与 commit 形如： @@ -570,7 +607,7 @@ git add workspaces/my-research/log.md git commit -m "ingest: Shein 美国市场份额 2025（Sheng Lu / Euromonitor）" ``` -### 6.10 验收：从 wiki 反查到 raw +### 7.10 验收：从 wiki 反查到 raw 完成 ingest 后任意时刻，下面这些动作都应该顺畅（均为本仓库真实可跑的命令）： @@ -596,11 +633,14 @@ python scripts/k.py health --- -## 7. 异常路径速查 +## 8. 异常路径速查 | 现象 | 可能原因 | 处理 | |---|---|---| -| `convert.py` 报"转换结果为空" | markitdown 不识别（如扫描版 PDF） | 手动转 OCR；或换 `pdfplumber` / `pymupdf` 预处理 | +| `convert.py` 报需 OCR / 视觉图像或矢量图提取 | 扫描页、内容级光栅/矢量图或无可靠文本映射 | 先做 OCR / 图表抽取并逐页核验，再重新转换；不要绕过后把失败当成功 | +| `two-column/key-value layout is ambiguous` | 对齐窄左栏同时符合双栏与键值表几何，缺少可验证语义信号 | 视觉复核原页；在可控副本中加明确冒号/表格线/栏结构后重转，不得强制忽略 | +| `source changed ... conversion` | 转换期间同步工具或人正在替换 raw 原文 | 等原文写入稳定后重试；本次不会把旧文本与新 receipt 混在一起 | +| `convert.py` 报 PDF layout extraction 失败 | PDF 损坏、加密、复杂旋转/布局超出确定性解析范围 | 修复或解密副本，或用专用转换器产出可核验 Markdown；保留原始文件只读 | | outline.json 与 md 不同步 | 手改了 md 但没重跑 convert | `python scripts/convert.py --force`（整个 workspace 的 raw/）或 `--force --dir workspaces/my-research/raw/papers` 只重转一处 | | `k.py list-broken-refs` 报失效 | 原文内容微调导致 hash 变 | 手动改 wiki 引用为新 anchor，或在原文恢复变更 | | 摘要页里有 `[需要来源]` | 写时找不到精确出处 | lint 流程逐条补 anchor，或把论断降级为 `confidence: low` | @@ -609,20 +649,20 @@ python scripts/k.py health --- -## 8. 关键设计决策（为什么是这样） +## 9. 关键设计决策（为什么是这样） | 决策 | 替代方案 | 为什么不选 | |---|---|---| | 锚点用 `^h-/^p-/^t-/^c-/^f-` 写在行末 | 用 heading 文本做引用 | heading 文本有大小写 / 空白差异；agent 写错概率高 | | `hash6` 内容微调时自动失效 | 永久稳定 ID（如 UUID） | 内容变了引用不该还指向旧位置；失效暴露才是对的 | | 知识库**不**调用 LLM | 内嵌 embedding + 向量搜索 | 违反原则 4；切片召回会破坏完整段语义；embedding 模型选型成本高 | -| `read-section` / `read-block` 返回**完整段** | 返回 chunk | 切片会断裂表格、公式、列表；完整段才有上下文 | +| `search-evidence` 选自然单元，`read-evidence-unit` + `read-block` 读完整父块 | 返回固定长度 chunk | 精确 row/item 身份用于选择，完整父块用于核对上下文；两者缺一会导致跨行错引 | | `raw/` 连同派生 `.md` / `.outline.json` 进 `.gitignore`，不入库 | 派生产物提交 git | 原始资料常含版权 / 隐私内容，开源仓库只发布 wiki 提炼物；派生物可随时由 `convert.py` 重建。代价是换台机器后 `[[raw/...]]` 引用无法跳转——这是有意取舍：raw 留在本地，公开的是 wiki | | `删除 = status: deprecated` | 真删文件 | 历史信息有内在价值；真删后 backlinks 断裂 | --- -## 9. 想得更深一点 +## 10. 想得更深一点 - **为什么不直接把 PDF 整本喂给 LLM？** 三个理由：① 大文档 context 成本高；② 一次性读完往往**只记住开头和结尾**（lost-in-the-middle）；③ 没有锚点就没法回溯——下个月想验证某个结论时找不到原文出处。 - **为什么强制每条论断都要 anchor？** 知识库的目的不是"存储看过什么"，而是"任意时刻都能回到原文重新验证"。anchor 是这个能力的物理基础。 diff --git "a/docs/\346\226\260\346\211\213\346\225\231\347\250\213-\346\211\213\346\212\212\346\211\213\346\220\255\345\273\272\347\237\245\350\257\206\345\272\223.md" "b/docs/\346\226\260\346\211\213\346\225\231\347\250\213-\346\211\213\346\212\212\346\211\213\346\220\255\345\273\272\347\237\245\350\257\206\345\272\223.md" index 61f36cd..39a7071 100644 --- "a/docs/\346\226\260\346\211\213\346\225\231\347\250\213-\346\211\213\346\212\212\346\211\213\346\220\255\345\273\272\347\237\245\350\257\206\345\272\223.md" +++ "b/docs/\346\226\260\346\211\213\346\225\231\347\250\213-\346\211\213\346\212\212\346\211\213\346\220\255\345\273\272\347\237\245\350\257\206\345\272\223.md" @@ -232,6 +232,15 @@ python scripts/convert.py --dir workspaces/smb-ecommerce/raw/articles --ext .htm 1. 把 `.html`（或 PDF/Word）转成纯文本 `.md`； 2. **自动给每一段、每个标题、每张表加上锚点**（`^p-…`/`^h-…`/`^t-…`），并生成一份 `.outline.json` 大纲。 +转换后立即建立“全细节证据地图”： + +```bash +python scripts/k.py --workspace smb-ecommerce rebuild-evidence-index +python scripts/k.py --workspace smb-ecommerce evidence-index-coverage +``` + +第二条必须显示自然单元、内容章节、结构章节都是 100% 且 corpus fresh。这保证转换后 Markdown 中未被摘要的段落、列表条目和表格行仍可检索；它不等于 PDF/Word 转换保真认证。 + 看看大纲（agent 后面就靠它知道"这份资料有哪些段、各段的书签是什么"）： ```bash @@ -330,9 +339,9 @@ agent 会钻取你刚建好的页面，给出**带引用清单**的回答（查 ### 这一章的小结：raw → wiki 全流程 ``` -你放 raw 文件 ─▶ convert.py 转 md + 加锚点 ─▶ AI agent 读原文 - │ - ┌───────────────────────────┘ +你放 raw 文件 ─▶ convert.py 转 md + 加锚点 ─▶ 全细节证据地图 ─▶ AI agent 读原文 + │ + ┌───────────────────────────────────────┘ ▼ 写来源摘要页（带块级引用 + AI 综合判断） │ @@ -470,8 +479,9 @@ python scripts/k.py new-workspace ai-research 下一步： 1. 把原始资料（HTML/PDF/Word/Markdown）放进 workspaces/ai-research/raw/articles/ 或 raw/papers/ 2. python scripts/convert.py --workspace ai-research # 转 markdown + 自动加锚点 - 3. 让 AI agent 执行摄入（如在 Claude Code 里说「把 raw/... 摄入到 ai-research 知识库」或用 /kb-ingest） - 4. python scripts/k.py --workspace ai-research health # 体检 + 3. python scripts/k.py --workspace ai-research rebuild-evidence-index # 建全细节证据地图 + 4. 让 AI agent 执行摄入（如在 Claude Code 里说「把 raw/... 摄入到 ai-research 知识库」或用 /kb-ingest） + 5. python scripts/k.py --workspace ai-research health # 体检 ``` 这条命令生成了和示例库完全相同的目录结构：`wiki/`（含一个空的根索引）、`raw/articles/`、`raw/papers/`、`exports/`、`my_thoughts/`、`log.md`。 @@ -482,6 +492,8 @@ python scripts/k.py new-workspace ai-research ```bash python scripts/convert.py --workspace ai-research +python scripts/k.py --workspace ai-research rebuild-evidence-index +python scripts/k.py --workspace ai-research evidence-index-coverage ``` ### 7.3 让 agent 摄入 diff --git "a/docs/\346\236\204\345\273\272\350\207\252\345\267\261\347\232\204\346\234\254\345\234\260\347\237\245\350\257\206\345\272\223.md" "b/docs/\346\236\204\345\273\272\350\207\252\345\267\261\347\232\204\346\234\254\345\234\260\347\237\245\350\257\206\345\272\223.md" new file mode 100644 index 0000000..a2c368b --- /dev/null +++ "b/docs/\346\236\204\345\273\272\350\207\252\345\267\261\347\232\204\346\234\254\345\234\260\347\237\245\350\257\206\345\272\223.md" @@ -0,0 +1,85 @@ +# 构建自己的本地知识库（引擎在此、数据在别处） + +> 一句话：**引擎（本项目）只装一份、保持纯代码；你的知识库数据存在别处的独立文件夹里**，靠环境变量 `KB_ROOT` 把引擎指过去。源文档和构建结果放在**同一个文件夹**里，自包含、可整体备份/迁移。 + +## 为什么数据不放在引擎项目里 + +- 本项目要**开源**：私人文档、原始资料不能进这个仓（`workspaces/*/raw/`、`my_thoughts/` 已被 `.gitignore` 排除，放进来要么泄露、要么不进版本控制）。 +- 引擎保持**纯代码**才能升级、pin 版本、跟上游同步——数据混进来每次拉更新都要处理冲突。 +- 一份引擎可以服务**多个**主题各异的知识库，各自一个路径，互不干扰。 + +## 推荐布局：一个文件夹 = 一个完整知识库 + +``` +~/任意位置/我的知识库/ ← 这一层就是 KB_ROOT（放在引擎项目之外） +└── workspaces/ + └── main/ ← 单库项目用 main 即可（也可按主题起名） + ├── raw/ ← 【源文档】放这里：raw/papers/ 或 raw/articles/ + │ ├── papers/ + │ └── articles/ + ├── wiki/ ← 【构建结果】摄入综合出的知识，与 raw 同一级 + ├── exports/ ← 导出物 + ├── my_thoughts/ ← 人类专属区（agent 只读） + ├── .cache/ ← SQLite 索引（可重建，gitignored） + └── log.md ← 操作日志 +``` + +- **源文档（`raw/`）和构建结果（`wiki/`）在同一级、同一个文件夹内** —— 整个知识库自包含在 `~/任意位置/我的知识库/` 这一个目录下，拷走即带走全部。 +- `KB_ROOT` 指向**含 `workspaces/` 的那一层**（`~/任意位置/我的知识库` ✅，`.../workspaces/main` ❌）。 + +## 两级定位 + +| 变量 | 选什么 | 例子 | +|---|---|---| +| `KB_ROOT` | 哪个知识库（数据根） | `~/任意位置/我的知识库` | +| `--workspace` / `KB_WORKSPACE` | 该数据根里的哪个库 | `main` | + +## 一次性初始化 + +```bash +cd <引擎项目路径> # 即本项目 groundmap-release +KB_ROOT=~/任意位置/我的知识库 python scripts/k.py new-workspace main +``` + +这会在 `~/任意位置/我的知识库/workspaces/main/` 下建好上面的完整骨架（含空根索引 + log.md）。 + +## 日常使用（每条命令挂 `KB_ROOT`） + +```bash +# 1. 把要摄入的 pdf/docx/html/md 放进 workspaces/main/raw/papers/（或 raw/articles/） + +# 2. 转换 + 自动加锚点，生成 .outline.json +KB_ROOT=~/任意位置/我的知识库 python scripts/convert.py --workspace main + +# 3. 建立转换文本的全细节证据地图，并确认覆盖率/新鲜度全绿 +KB_ROOT=~/任意位置/我的知识库 python scripts/k.py --workspace main rebuild-evidence-index +KB_ROOT=~/任意位置/我的知识库 python scripts/k.py --workspace main evidence-index-coverage + +# 4. 摄入：在 Claude Code 里走 /kb-ingest（agent 阅读 + 综合 + 写 wiki + 原子提交） + +# 5. 查询：/kb-query，或 +KB_ROOT=~/任意位置/我的知识库 python scripts/k.py --workspace main search "关键词" + +# 6. Web 管理台 +cd web && KB_ROOT=~/任意位置/我的知识库 KB_WORKSPACE=main npm run dev + +# 健康检查 +KB_ROOT=~/任意位置/我的知识库 python scripts/k.py --workspace main health +``` + +> 嫌每次敲 `KB_ROOT` 麻烦，可在数据文件夹里放一个 `.envrc`/别名，或临时 `export KB_ROOT=~/任意位置/我的知识库`（有多个库时按需切换）。 + +## 在 Claude Code 里的极简流程（推荐） + +下次要用知识库时： + +1. 启动 Claude Code，进入**本引擎项目**（`groundmap-release`）。 +2. 直接告诉它：**「摄入 `<你的知识库文件夹>` 里的新文档」**（或先把文档丢进该库的 `raw/papers/` 再说一句「摄入」）。 +3. agent 会自动：确认/新建 workspace → `convert.py` 转换 → 重建并核对自然单元证据索引 → 走 `/kb-ingest` 综合入 wiki → 原子提交。 + +agent 已把「co-located KB」约定记进项目记忆，因此只要你给出那个文件夹路径，它就知道 `KB_ROOT=<该文件夹>`、workspace 用 `main`、结果写回同一文件夹的 `wiki/`。 + +## 版本控制建议 + +- 你的知识库文件夹是**独立的 Git 仓库**（和引擎仓分开）：在 `~/任意位置/我的知识库/` 里 `git init`，把 `wiki/`、`log.md` 纳入版本控制；`raw/`（版权/隐私）、`.cache/`（可重建）、`my_thoughts/`（私人）按需 `.gitignore`。 +- 引擎侧只 pin 版本，不动你的数据；契约类升级（锚点/frontmatter schema 等，见 `CLAUDE.md`「演进与兼容性」）才需对数据走迁移四步。 diff --git a/evals/README.md b/evals/README.md new file mode 100644 index 0000000..11e9bd0 --- /dev/null +++ b/evals/README.md @@ -0,0 +1,47 @@ +# GroundMap 评测入口 + +评测按层拆开报告，避免把“检索到了”“引用选对了”“最终答案完整”压成一个容易 +假绿的平均分。所有公开 runner 均使用结构化终态和 `0 / 1 / 2` 退出码：通过、 +阈值未通过、协议/依赖/完整性错误。 + +## 第二阶段统一公开闸门 + +```bash +python evals/run_stage2.py +python evals/run_stage2.py --json --details +``` + +统一命令依次运行： + +1. `holdout/`：外部 bundle 协议的 CC0 public smoke； +2. `answer_citation/`：answer/abstain → atomic claim → exact citation； +3. `conversion_fidelity/`：真实 PDF/DOCX/HTML → `convert.py` → Markdown。 + +统一 runner 不信任子评测自报的 `passed`：它会精确核对应有指标集合、 +schema/status/scope、分子/分母/value 等式、阈值方向与重算 verdict。缺少指标、 +伪造 `dummy: {passed: true}` 或子进程无结构化终态都返回协议错误。 + +即使三项全绿，终态也只能是 `public-stage2-gates-passed`。公开 Gold 与合成格式 +fixture 用于协议回归，不能替代独立隐藏发布集，也不能证明开放世界 100%。 + +## 单项入口 + +```bash +python evals/longdoc/run_eval.py --json +python evals/answer_citation/run_eval.py --json --details +python evals/conversion_fidelity/run_eval.py --json --details + +python evals/holdout/generate_public_smoke.py --output /tmp/holdout.json +# 复制上一条命令输出 JSON 里的 bundle_sha256 +python evals/holdout/run_eval.py \ + --bundle /tmp/holdout.json \ + --bundle-sha256 <上一步完整文件摘要> \ + --run-id public-smoke-v1 --seed 20260712 --json +``` + +真正的 external holdout bundle 必须由未参与实现与调优的一方生成和保管，按 +`holdout/bundle.schema.json` 提供；查看结果前在独立渠道冻结完整文件 SHA-256、 +threshold profile、case/slice 分母和运行配置。hidden bundle 不进入 Git。 + +各目录 README 记录指标分子/分母、Gold 隔离方式、mutation guards 和诚实边界； +总体预注册设计见 `docs/evaluation-stage2-design.md`。 diff --git a/evals/answer_citation/LICENSE b/evals/answer_citation/LICENSE new file mode 100644 index 0000000..1fadf6e --- /dev/null +++ b/evals/answer_citation/LICENSE @@ -0,0 +1,104 @@ +CC0 1.0 Universal + +Statement of Purpose + +The laws of most jurisdictions throughout the world automatically confer +exclusive Copyright and Related Rights (defined below) upon the creator and +subsequent owner(s) (each and all, an "owner") of an original work of authorship +and/or a database (each, a "Work"). + +Certain owners wish to permanently relinquish those rights to a Work for the +purpose of contributing to a commons of creative, cultural and scientific works +("Commons") that the public can reliably and without fear of later claims of +infringement build upon, modify, incorporate in other works, reuse and +redistribute as freely as possible in any form whatsoever and for any purposes, +including without limitation commercial purposes. These owners may contribute +to the Commons to promote the ideal of a free culture and the further production +of creative, cultural and scientific works, or to gain reputation or greater +distribution for their Work in part through the use and efforts of others. + +For these and/or other purposes and motivations, and without any expectation of +additional consideration or compensation, the person associating CC0 with a +Work (the "Affirmer"), to the extent that he or she is an owner of Copyright and +Related Rights in the Work, voluntarily elects to apply CC0 to the Work and +publicly distribute the Work under its terms, with knowledge of his or her +Copyright and Related Rights in the Work and the meaning and intended legal +effect of CC0 on those rights. + +1. Copyright and Related Rights. A Work made available under CC0 may be protected +by copyright and related or neighboring rights ("Copyright and Related Rights"). +Copyright and Related Rights include, but are not limited to, the following: + + i. the right to reproduce, adapt, distribute, perform, display, communicate, + and translate a Work; + ii. moral rights retained by the original author(s) and/or performer(s); +iii. publicity and privacy rights pertaining to a person's image or likeness + depicted in a Work; + iv. rights protecting against unfair competition in regards to a Work, subject + to the limitations in paragraph 4(a), below; + v. rights protecting the extraction, dissemination, use and reuse of data in + a Work; + vi. database rights (such as those arising under Directive 96/9/EC of the + European Parliament and of the Council of 11 March 1996 on the legal + protection of databases, and under any national implementation thereof, + including any amended or successor version of such directive); and +vii. other similar, equivalent or corresponding rights throughout the world + based on applicable law or treaty, and any national implementations thereof. + +2. Waiver. To the greatest extent permitted by, but not in contravention of, +applicable law, Affirmer hereby overtly, fully, permanently, irrevocably and +unconditionally waives, abandons, and surrenders all of Affirmer's Copyright and +Related Rights and associated claims and causes of action, whether now known or +unknown (including existing as well as future claims and causes of action), in +the Work (i) in all territories worldwide, (ii) for the maximum duration +provided by applicable law or treaty (including future time extensions), (iii) +in any current or future medium and for any number of copies, and (iv) for any +purpose whatsoever, including without limitation commercial, advertising or +promotional purposes (the "Waiver"). Affirmer makes the Waiver for the benefit +of each member of the public at large and to the detriment of Affirmer's heirs +and successors, fully intending that such Waiver shall not be subject to +revocation, rescission, cancellation, termination, or any other legal or +equitable action to disrupt the quiet enjoyment of the Work by the public as +contemplated by Affirmer's express Statement of Purpose. + +3. Public License Fallback. Should any part of the Waiver for any reason be +judged legally invalid or ineffective under applicable law, then the Waiver +shall be preserved to the maximum extent permitted taking into account +Affirmer's express Statement of Purpose. In addition, to the extent the Waiver +is so judged Affirmer hereby grants to each affected person a royalty-free, +non transferable, non sublicensable, non exclusive, irrevocable and +unconditional license to exercise Affirmer's Copyright and Related Rights in +the Work (i) in all territories worldwide, (ii) for the maximum duration +provided by applicable law or treaty (including future time extensions), (iii) +in any current or future medium and for any number of copies, and (iv) for any +purpose whatsoever, including without limitation commercial, advertising or +promotional purposes (the "License"). The License shall be deemed effective as +of the date CC0 was applied by Affirmer to the Work. Should any part of the +License for any reason be judged legally invalid or ineffective under applicable +law, such partial invalidity or ineffectiveness shall not invalidate the +remainder of the License, and in such case Affirmer hereby affirms that he or she +will not (i) exercise any of his or her remaining Copyright and Related Rights in +the Work or (ii) assert any associated claims and causes of action with respect +to the Work, in either case contrary to Affirmer's express Statement of Purpose. + +4. Limitations and Disclaimers. + + a. No trademark or patent rights held by Affirmer are waived, abandoned, + surrendered, licensed or otherwise affected by this document. + b. Affirmer offers the Work as-is and makes no representations or warranties of + any kind concerning the Work, express, implied, statutory or otherwise, + including without limitation warranties of title, merchantability, fitness + for a particular purpose, non infringement, or the absence of latent or + other defects, accuracy, or the present or absence of errors, whether or not + discoverable, all to the greatest extent permissible under applicable law. + c. Affirmer disclaims responsibility for clearing rights of other persons that + may apply to the Work or any use thereof, including without limitation any + person's Copyright and Related Rights in the Work. Further, Affirmer + disclaims responsibility for obtaining any necessary consents, permissions + or other rights required for any use of the Work. + d. Affirmer understands and acknowledges that Creative Commons is not a party + to this document and has no duty or obligation with respect to this CC0 or + use of the Work. + +For the complete legal code and additional information, visit: +https://creativecommons.org/publicdomain/zero/1.0/legalcode diff --git a/evals/answer_citation/README.md b/evals/answer_citation/README.md new file mode 100644 index 0000000..c2dc554 --- /dev/null +++ b/evals/answer_citation/README.md @@ -0,0 +1,163 @@ +# Answer→Citation E2E 协议评测 + +这组 CC0 样例补上 LongDoc Evidence Atlas 没有覆盖的一层：实际回答中的 +**原子论断 → 精确引用**是否正确，以及回答是否覆盖了必答分面。它不调用 +LLM，不使用 embedding，也不从 Gold 回填被测输出。 + +```bash +python evals/answer_citation/run_eval.py +python evals/answer_citation/run_eval.py --json --details +``` + +## 被测 adapter 协议 + +Python adapter 形式为 `module:function`： + +```python +def answer(request: dict) -> dict: + # request 只有 question 和未标注的检索证据 + assert set(request) == {"question", "evidence"} + ... +``` + +`request` 结构： + +```json +{ + "question": "...", + "evidence": [ + {"citation": "raw/x.md#^p-3-abc123", "text": "完整原文块…"} + ] +} +``` + +回答结构： + +```json +{ + "decision": "answer", + "answer": "第一条原子论断。\n第二条原子论断。", + "claims": [ + { + "claim_id": "c1", + "text": "第一条原子论断。", + "facet_id": "launch_mass", + "value": "18.4 kg", + "citations": ["raw/x.md#^p-3-abc123"] + }, + { + "claim_id": "c2", + "text": "第二条原子论断。", + "facet_id": "battery_endurance", + "value": "11.5 hours", + "citations": ["raw/x.md#^t-4-def456"] + } + ] +} +``` + +`answer` 必须等于所有 `claims[].text` 按顺序用换行连接的结果。这是一个 +normalization contract：真实系统可以有自己的展示文本，但接入评测时必须把所有 +实质性论断完整枚举为原子 claim，不能在未计分的 prose 中隐藏断言。 + +每个 Gold facet 注册一个或多个 `accepted_variants`。每个变体是一个 +不可拆分的 `{variant_id, claim_text, value, evidence_set}` 四元组： +`claims[].text` 、`claims[].value` 与完整引用集必须同时命中**同一变体**。 +从变体 A 取文本、变体 B 取值或证据的交叉拼接不被接受，即使每个字段 +单独看都出现在 Gold 中。`claim_text` 仍必须完整相等，不做子串、关键词 +或语义相似匹配。如果要评估开放式同义改写，必须另外设置不看期望答案的 +独立盲审；本确定性 scorer 不把自由改写冒充为已验证的内容正确。 + +Gold schema v3 由 v2 的三个并行列表 `accepted_claim_texts` / +`accepted_values` / `minimal_evidence_sets` 迁移为上述显式变体四元组。 +runner 只接受 v3，v2 Gold 会以协议错误退出 2，不会隐式做笛卡尔积迁移。 + +拒答结构为： + +```json +{"decision": "abstain", "answer": "", "claims": []} +``` + +引用必须是本轮检索证据中实际出现的 canonical block ref。无引用 claim 可以 +被提交，但会被计为 unsupported，而不是协议异常。 + +## 接入真实系统输出 + +可以直接指定 Python adapter： + +```bash +python evals/answer_citation/run_eval.py \ + --adapter your_package.answer_adapter:answer --json --details +``` + +也可以先在独立进程/服务中运行真实系统，再提交 JSONL： + +```json +{"case_id":"ac-smoke-001","response":{"decision":"answer","answer":"...","claims":[...]}} +``` + +```bash +python evals/answer_citation/run_eval.py \ + --predictions /path/to/predictions.jsonl --json --details +``` + +JSONL 必须对全部 case 每个提交一行；缺失、重复、多余 case 都 fail-closed。 + +## Gold 隔离 + +- `public_cases.py` 只有 question 和 retrieval evidence，没有 answerability、答案、 + required facets、minimal evidence sets 或 forbidden labels。 +- `gold.py` 独立定义每题的 answerable/unanswerable、required facets、 + 绑定 canonical claim text + value + evidence set 的 accepted variants，以及 + forbidden evidence。 +- runner 先收集完所有 adapter 输出，然后才把输出与 Gold 计分；调用请求 + 只含 `{question,evidence}`。 +- 公开 fixture 和 Gold 各自有冻结 SHA-256，case 分母或内容静默漂移会 + 退出 2。Gold canary 若出现在 adapter 请求或输出中也会退出 2。 +- public question 按 NFKC + casefold，再把标点/符号/空白/不可见 format control 统一为分隔后必须唯一；结果在 + `fixture.unique_questions` 显式报告这一分母。 + +canary 只防意外管道泄漏，不是恶意代码的安全沙箱。由于这是开源公开集， +任何能读仓库的恶意 adapter 都可以主动 import Gold；真正的能力认证必须在隔离 +环境中使用未公开的 hidden set，并在解密 Gold 前冻结 predictions。 + +## 指标 + +| 指标 | 分子 / 分母 | +|---|---| +| `claim_citation_precision` | claim 命中同一 canonical 变体，且引用属于该变体 evidence set 的精确引用对 / 全部被选引用对 | +| `citation_completeness` | claim 的文本、值与完整引用集命中同一 accepted variant 的 required facet / 全部 required facets | +| `required_facet_coverage` | 至少有一条文本与值命中同一 accepted variant 的 required facet / 全部 required facets | +| `answer_coverage` | 被回答的 answerable cases / 全部 answerable cases | +| `correct_abstention_rate` | 正确拒答的 unanswerable cases / 全部 unanswerable cases | +| `unsupported_claim_rate` | 文本非 canonical、值错或未满足 minimal evidence set 的 claims / 全部 claims | +| `fully_grounded_answer_rate` | required facets 恰好全覆盖、每条 claim 完整支撑且无错引的 answerable cases / 全部 answerable cases | + +precision 与 coverage 在同一份结果里必须同时达标；全部拒答会导致 claim/ +citation 分母为零，runner 直接退出 2，绝不把 undefined 报成 100%。 +每个终态都有固定 `status`：全部闸门通过为 +`protocol-smoke-passed`，完成评分但未达阈值为 `threshold-failed`，协议、 +依赖、零分母或外部边界异常为 `protocol-error`。`SystemExit(0)` 也不能绕过 +这个契约变成 shell 成功。 + +## Mutation 守护与边界 + +`scripts/tests/test_answer_citation_eval.py` 会故意注入： + +- 漏 required facet； +- 从不同 accepted variants 交叉拼接 claim text / value / evidence set； +- 同数值但错主体的引用； +- value/引用不变，但 claim 文本换成错主体； +- value/引用不变，但 claim 文本加入否定反转； +- value/引用不变，但 claim 文本追加未支撑事实； +- 无引用 claim； +- 正确引用后追加错引（过引）； +- answerable 题错误拒答； +- unanswerable 题强行作答； +- Gold canary 泄漏； +- 零分母、非 canonical/未检索引用、隐藏未枚举的 answer prose。 +- adapter / evaluator 用 `SystemExit(0)` 在无结构化结果时提前终止。 + +`reference_adapter.py` 只是一个确定性规则 adapter，用来证明协议与计分器 +能端到端运行。其 100% 只是 **protocol smoke**，不是模型能力、真实检索、 +PDF/DOCX 转换保真或开放世界精度证明。 diff --git a/evals/answer_citation/__init__.py b/evals/answer_citation/__init__.py new file mode 100644 index 0000000..a30b5c9 --- /dev/null +++ b/evals/answer_citation/__init__.py @@ -0,0 +1 @@ +"""GroundMap answer→citation end-to-end protocol evaluation.""" diff --git a/evals/answer_citation/gold.py b/evals/answer_citation/gold.py new file mode 100644 index 0000000..df48dee --- /dev/null +++ b/evals/answer_citation/gold.py @@ -0,0 +1,207 @@ +"""Independent answer/citation Gold for the public protocol-smoke questions. + +Do not import this module from answer adapters. ``run_eval.py`` loads it only +after collecting adapter outputs and never includes these fields in a request. +""" +from __future__ import annotations + +import hashlib +import json +from typing import Any + +GOLD_SCHEMA_VERSION = 3 +GOLD_CANARY = "ANSWER_CITATION_GOLD_ONLY_6f39dca2" + +GOLD_CASES: tuple[dict[str, Any], ...] = ( + { + "case_id": "ac-smoke-001", + "answerable": True, + "required_facets": [ + { + "facet_id": "launch_mass", + "accepted_variants": [ + { + "variant_id": "manual-paragraph-kg", + "claim_text": "Atlas Skiff's launch mass is 18.4 kg.", + "value": "18.4 kg", + "evidence_set": [ + "raw/manuals/atlas-skiff.md#^p-10-a1b2c3" + ], + }, + { + "variant_id": "spec-table-kilograms", + "claim_text": ( + "Atlas Skiff's specified launch mass is " + "18.4 kilograms." + ), + "value": "18.4 kilograms", + "evidence_set": [ + "raw/specifications/atlas-skiff.md#^t-3-f0e1d2" + ], + }, + ], + }, + { + "facet_id": "battery_endurance", + "accepted_variants": [ + { + "variant_id": "manual-table-hours", + "claim_text": ( + "Atlas Skiff's battery endurance is 11.5 hours." + ), + "value": "11.5 hours", + "evidence_set": [ + "raw/manuals/atlas-skiff.md#^t-11-b2c3d4" + ], + } + ], + }, + ], + "forbidden_evidence": [ + "raw/manuals/boreal-skiff.md#^p-4-c3d4e5", + "raw/manuals/atlas-cargo.md#^p-9-d4e5f6", + "raw/manuals/atlas-skiff.md#^p-12-e5f6a7", + ], + }, + { + "case_id": "ac-smoke-002", + "answerable": True, + "required_facets": [ + { + "facet_id": "retention_period", + "accepted_variants": [ + { + "variant_id": "policy-paragraph-days", + "claim_text": ( + "Cedar Records' ordinary retention period is 30 days." + ), + "value": "30 days", + "evidence_set": [ + "raw/policies/cedar-records.md#^p-7-a2b3c4" + ], + } + ], + }, + { + "facet_id": "legal_hold_effect", + "accepted_variants": [ + { + "variant_id": "policy-paragraph-release-condition", + "claim_text": ( + "During a Cedar Records legal hold, deletion pauses " + "until the hold is released." + ), + "value": "deletion pauses until the hold is released", + "evidence_set": [ + "raw/policies/cedar-records.md#^p-8-b3c4d5" + ], + } + ], + }, + ], + "forbidden_evidence": [ + "raw/policies/hemlock-records.md#^p-3-c4d5e6", + "raw/policies/cedar-records.md#^p-10-d5e6f7", + ], + }, + { + "case_id": "ac-smoke-003", + "answerable": True, + "required_facets": [ + { + "facet_id": "inspection_interval", + "accepted_variants": [ + { + "variant_id": "assignment-plus-depot-policy", + "claim_text": ( + "Nimbus Probe NP-7's assigned depot requires " + "inspection every 42 days." + ), + "value": "42 days", + "evidence_set": [ + "raw/assets/nimbus-probes.md#^t-14-a3b4c5", + "raw/depots/d-17.md#^p-5-b4c5d6", + ], + } + ], + }, + ], + "forbidden_evidence": [ + "raw/depots/d-71.md#^p-5-c5d6e7", + "raw/assets/cirrus-probes.md#^t-2-d6e7f8", + ], + }, + { + "case_id": "ac-smoke-004", + "answerable": True, + "required_facets": [ + { + "facet_id": "alarm_threshold", + "accepted_variants": [ + { + "variant_id": "manual-table-celsius", + "claim_text": "星港-4号冷却回路的告警阈值为 73.2 摄氏度。", + "value": "73.2 摄氏度", + "evidence_set": [ + "raw/manuals/starport-4-zh.md#^t-21-a4b5c6" + ], + } + ], + }, + { + "facet_id": "reset_window", + "accepted_variants": [ + { + "variant_id": "manual-table-minutes", + "claim_text": "星港-4号冷却回路的复位窗口为 9 分钟。", + "value": "9 分钟", + "evidence_set": [ + "raw/manuals/starport-4-zh.md#^t-22-b5c6d7" + ], + } + ], + }, + ], + "forbidden_evidence": [ + "raw/manuals/starport-5-zh.md#^t-21-c6d7e8", + "raw/manuals/starport-4-zh.md#^p-23-d7e8f9", + ], + }, + { + "case_id": "ac-smoke-005", + "answerable": False, + "required_facets": [], + "forbidden_evidence": [ + "raw/manuals/atlas-skiff.md#^p-10-a1b2c3", + "raw/manuals/atlas-skiff.md#^t-11-b2c3d4", + "raw/budgets/boreal-skiff.md#^t-6-e8f9a0", + ], + }, + { + "case_id": "ac-smoke-006", + "answerable": False, + "required_facets": [], + "forbidden_evidence": [ + "raw/policies/cedar-records.md#^p-7-a2b3c4", + "raw/policies/cedar-records.md#^p-10-d5e6f7", + "raw/policies/hemlock-records.md#^t-9-f9a0b1", + ], + }, +) + + +def gold_sha256() -> str: + payload = { + "schema_version": GOLD_SCHEMA_VERSION, + "cases": GOLD_CASES, + "canary": GOLD_CANARY, + } + encoded = json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +FROZEN_GOLD_SHA256 = ( + "7c159d95ef83f20c68c78fc301e2f0e29131cf51cf5a8a7d85186c55712cbb23" +) diff --git a/evals/answer_citation/public_cases.py b/evals/answer_citation/public_cases.py new file mode 100644 index 0000000..26514be --- /dev/null +++ b/evals/answer_citation/public_cases.py @@ -0,0 +1,206 @@ +"""CC0 public protocol-smoke questions and retrieval evidence. + +This module intentionally contains no answerability labels, accepted answers, +minimal evidence sets, or forbidden-evidence labels. An answer adapter receives +only a deep copy of ``question`` and ``evidence`` from each case. +""" +from __future__ import annotations + +import copy +import hashlib +import json +from typing import Any + +PUBLIC_CASE_SCHEMA_VERSION = 1 +LICENSE = "CC0-1.0" + + +PUBLIC_CASES: tuple[dict[str, Any], ...] = ( + { + "case_id": "ac-smoke-001", + "category": "same-value-wrong-subject", + "question": ( + "For Atlas Skiff, what are the launch mass and battery endurance?" + ), + "evidence": [ + { + "citation": "raw/manuals/atlas-skiff.md#^p-10-a1b2c3", + "text": "Atlas Skiff's launch mass is 18.4 kg.", + }, + { + "citation": "raw/manuals/atlas-skiff.md#^t-11-b2c3d4", + "text": "Atlas Skiff's battery endurance is 11.5 hours.", + }, + { + "citation": "raw/manuals/boreal-skiff.md#^p-4-c3d4e5", + "text": "Boreal Skiff's launch mass is 18.4 kg.", + }, + { + "citation": "raw/manuals/atlas-cargo.md#^p-9-d4e5f6", + "text": "Atlas Cargo's payload mass is 18.4 kg.", + }, + { + "citation": "raw/manuals/atlas-skiff.md#^p-12-e5f6a7", + "text": "Atlas Skiff accepts a 28-volt service supply.", + }, + { + "citation": "raw/specifications/atlas-skiff.md#^t-3-f0e1d2", + "text": ( + "Atlas Skiff's specified launch mass is 18.4 kilograms." + ), + }, + ], + }, + { + "case_id": "ac-smoke-002", + "category": "policy-exception", + "question": ( + "Under the Cedar Records policy, what is the ordinary retention " + "period and what happens during a legal hold?" + ), + "evidence": [ + { + "citation": "raw/policies/cedar-records.md#^p-7-a2b3c4", + "text": ( + "The Cedar Records policy ordinarily retains audit bundles " + "for 30 days." + ), + }, + { + "citation": "raw/policies/cedar-records.md#^p-8-b3c4d5", + "text": ( + "When a Cedar Records legal hold is active, deletion pauses " + "until the hold is released." + ), + }, + { + "citation": "raw/policies/hemlock-records.md#^p-3-c4d5e6", + "text": ( + "The Hemlock Records policy ordinarily retains audit bundles " + "for 30 days." + ), + }, + { + "citation": "raw/policies/cedar-records.md#^p-10-d5e6f7", + "text": "Cedar Records encrypts export archives at rest.", + }, + ], + }, + { + "case_id": "ac-smoke-003", + "category": "multi-hop", + "question": ( + "For Nimbus Probe NP-7, what inspection interval applies at its " + "assigned depot?" + ), + "evidence": [ + { + "citation": "raw/assets/nimbus-probes.md#^t-14-a3b4c5", + "text": "Nimbus Probe NP-7 is assigned to depot D-17.", + }, + { + "citation": "raw/depots/d-17.md#^p-5-b4c5d6", + "text": "Depot D-17 requires probe inspection every 42 days.", + }, + { + "citation": "raw/depots/d-71.md#^p-5-c5d6e7", + "text": "Depot D-71 requires probe inspection every 42 days.", + }, + { + "citation": "raw/assets/cirrus-probes.md#^t-2-d6e7f8", + "text": "Cirrus Probe CP-2 is assigned to depot D-17.", + }, + ], + }, + { + "case_id": "ac-smoke-004", + "category": "multilingual-table", + "question": "星港-4号冷却回路的告警阈值和复位窗口分别是多少？", + "evidence": [ + { + "citation": "raw/manuals/starport-4-zh.md#^t-21-a4b5c6", + "text": "星港-4号冷却回路的告警阈值为 73.2 摄氏度。", + }, + { + "citation": "raw/manuals/starport-4-zh.md#^t-22-b5c6d7", + "text": "星港-4号冷却回路的复位窗口为 9 分钟。", + }, + { + "citation": "raw/manuals/starport-5-zh.md#^t-21-c6d7e8", + "text": "星港-5号冷却回路的告警阈值为 73.2 摄氏度。", + }, + { + "citation": "raw/manuals/starport-4-zh.md#^p-23-d7e8f9", + "text": "星港-4号冷却回路的检查员代码为 LQ-8。", + }, + ], + }, + { + "case_id": "ac-smoke-005", + "category": "near-neighbor-cost", + "question": "What is Atlas Skiff's annual maintenance cost?", + "evidence": [ + { + "citation": "raw/manuals/atlas-skiff.md#^p-10-a1b2c3", + "text": "Atlas Skiff's launch mass is 18.4 kg.", + }, + { + "citation": "raw/manuals/atlas-skiff.md#^t-11-b2c3d4", + "text": "Atlas Skiff's battery endurance is 11.5 hours.", + }, + { + "citation": "raw/budgets/boreal-skiff.md#^t-6-e8f9a0", + "text": "Boreal Skiff's annual maintenance cost is 4,200 credits.", + }, + ], + }, + { + "case_id": "ac-smoke-006", + "category": "policy-neighbor-fee", + "question": "What export fee does the Cedar Records policy impose?", + "evidence": [ + { + "citation": "raw/policies/cedar-records.md#^p-7-a2b3c4", + "text": ( + "The Cedar Records policy ordinarily retains audit bundles " + "for 30 days." + ), + }, + { + "citation": "raw/policies/cedar-records.md#^p-10-d5e6f7", + "text": "Cedar Records encrypts export archives at rest.", + }, + { + "citation": "raw/policies/hemlock-records.md#^t-9-f9a0b1", + "text": "The Hemlock Records policy imposes a 7-credit export fee.", + }, + ], + }, +) + + +def _payload() -> dict[str, Any]: + return { + "schema_version": PUBLIC_CASE_SCHEMA_VERSION, + "license": LICENSE, + "cases": PUBLIC_CASES, + } + + +def fixture_sha256() -> str: + encoded = json.dumps( + _payload(), ensure_ascii=False, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +# Frozen after the fixture was reviewed. The runner compares this value before +# invoking an adapter, so a silent denominator/content change is a protocol error. +FROZEN_PUBLIC_FIXTURE_SHA256 = ( + "abc73f49aac8346b5a42c0c0d07a99659a01da2b5bdef41cc896b24b50a345cc" +) + + +def get_public_cases() -> list[dict[str, Any]]: + """Return a deep copy so an adapter cannot mutate the runner's fixture.""" + return copy.deepcopy(list(PUBLIC_CASES)) diff --git a/evals/answer_citation/reference_adapter.py b/evals/answer_citation/reference_adapter.py new file mode 100644 index 0000000..0d4998e --- /dev/null +++ b/evals/answer_citation/reference_adapter.py @@ -0,0 +1,163 @@ +"""Deterministic, Gold-blind adapter for *protocol smoke testing only*. + +The adapter is deliberately small and rule based. It consumes exactly two +request fields (``question`` and retrieval ``evidence``), imports no fixture or +Gold module, and demonstrates the structured response contract. Its score must +never be reported as model/system capability. +""" +from __future__ import annotations + +import re +from typing import Any + +ADAPTER_SCOPE = "protocol-smoke-only-not-model-capability" + + +def _claim(facet_id: str, value: str, text: str, citations: list[str]) -> dict: + return { + "claim_id": "", # assigned after all extraction rules run + "text": text, + "facet_id": facet_id, + "value": value, + "citations": citations, + } + + +def _matching(evidence: list[dict], *needles: str) -> dict | None: + lowered = tuple(needle.casefold() for needle in needles) + for row in evidence: + text = row["text"] + haystack = text.casefold() + if all(needle in haystack for needle in lowered): + return row + return None + + +def answer(request: dict[str, Any]) -> dict[str, Any]: + """Return one normalized answer/abstention response. + + The strict key check is intentional: adding ``case_id``, Gold facets, + answerability, accepted values, or evidence labels to the request breaks the + reference adapter and exposes a leakage regression immediately. + """ + if not isinstance(request, dict) or set(request) != {"question", "evidence"}: + raise ValueError("adapter request must contain only question and evidence") + question = request["question"] + evidence = request["evidence"] + if not isinstance(question, str) or not isinstance(evidence, list): + raise ValueError("invalid adapter request types") + if any( + not isinstance(row, dict) + or set(row) != {"citation", "text"} + or not isinstance(row["citation"], str) + or not isinstance(row["text"], str) + for row in evidence + ): + raise ValueError("invalid retrieval evidence") + + q = question.casefold() + claims: list[dict] = [] + + if "atlas skiff" in q and "launch mass" in q: + row = _matching(evidence, "Atlas Skiff", "launch mass") + if row: + match = re.search(r"launch mass is ([0-9.]+ kg)", row["text"], re.I) + if match: + value = match.group(1) + claims.append(_claim( + "launch_mass", value, + f"Atlas Skiff's launch mass is {value}.", + [row["citation"]], + )) + + if "atlas skiff" in q and "battery endurance" in q: + row = _matching(evidence, "Atlas Skiff", "battery endurance") + if row: + match = re.search( + r"battery endurance is ([0-9.]+ hours)", row["text"], re.I + ) + if match: + value = match.group(1) + claims.append(_claim( + "battery_endurance", value, + f"Atlas Skiff's battery endurance is {value}.", + [row["citation"]], + )) + + if "cedar records" in q and "retention" in q: + row = _matching(evidence, "Cedar Records", "retains", "days") + if row: + match = re.search(r"for ([0-9]+ days)", row["text"], re.I) + if match: + value = match.group(1) + claims.append(_claim( + "retention_period", value, + f"Cedar Records' ordinary retention period is {value}.", + [row["citation"]], + )) + + if "cedar records" in q and "legal hold" in q: + row = _matching(evidence, "Cedar Records", "legal hold", "deletion pauses") + if row: + value = "deletion pauses until the hold is released" + claims.append(_claim( + "legal_hold_effect", value, + "During a Cedar Records legal hold, deletion pauses until the " + "hold is released.", + [row["citation"]], + )) + + if "nimbus probe np-7" in q and "inspection interval" in q: + assignment = _matching(evidence, "Nimbus Probe NP-7", "assigned to depot") + if assignment: + depot_match = re.search(r"assigned to depot ([A-Z]-[0-9]+)", assignment["text"]) + if depot_match: + depot = depot_match.group(1) + interval = _matching(evidence, f"Depot {depot}", "inspection every") + if interval: + value_match = re.search( + r"inspection every ([0-9]+ days)", interval["text"], re.I + ) + if value_match: + value = value_match.group(1) + claims.append(_claim( + "inspection_interval", value, + f"Nimbus Probe NP-7's assigned depot requires " + f"inspection every {value}.", + [assignment["citation"], interval["citation"]], + )) + + if "星港-4号冷却回路" in question and "告警阈值" in question: + row = _matching(evidence, "星港-4号冷却回路", "告警阈值") + if row: + match = re.search(r"告警阈值为\s*([0-9.]+\s*摄氏度)", row["text"]) + if match: + value = " ".join(match.group(1).split()) + claims.append(_claim( + "alarm_threshold", value, + f"星港-4号冷却回路的告警阈值为 {value}。", + [row["citation"]], + )) + + if "星港-4号冷却回路" in question and "复位窗口" in question: + row = _matching(evidence, "星港-4号冷却回路", "复位窗口") + if row: + match = re.search(r"复位窗口为\s*([0-9.]+\s*分钟)", row["text"]) + if match: + value = " ".join(match.group(1).split()) + claims.append(_claim( + "reset_window", value, + f"星港-4号冷却回路的复位窗口为 {value}。", + [row["citation"]], + )) + + if not claims: + return {"decision": "abstain", "answer": "", "claims": []} + + for ordinal, claim in enumerate(claims, start=1): + claim["claim_id"] = f"c{ordinal}" + return { + "decision": "answer", + "answer": "\n".join(claim["text"] for claim in claims), + "claims": claims, + } diff --git a/evals/answer_citation/run_eval.py b/evals/answer_citation/run_eval.py new file mode 100644 index 0000000..d8a956c --- /dev/null +++ b/evals/answer_citation/run_eval.py @@ -0,0 +1,717 @@ +#!/usr/bin/env python3 +"""Fail-closed answer→citation end-to-end protocol evaluator. + +Exit codes: 0=all registered smoke thresholds pass; 1=evaluation completed but +at least one threshold failed; 2=fixture, Gold, adapter, denominator, or protocol +error. A reference-adapter pass is only a protocol smoke result. +""" +from __future__ import annotations + +import argparse +import copy +import importlib +import json +import math +import re +import sys +import unicodedata +from pathlib import Path +from typing import Any, Callable + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +if str(ENGINE_ROOT) not in sys.path: + sys.path.insert(0, str(ENGINE_ROOT)) + +from evals.answer_citation import gold, public_cases # noqa: E402 + +RESULT_PREFIX = "ANSWER_CITATION_EVAL_RESULT " +PROTOCOL_VERSION = 3 +EXPECTED_PUBLIC_CASE_SCHEMA_VERSION = 1 +EXPECTED_GOLD_SCHEMA_VERSION = 3 +SCOPE = "public-cc0-protocol-smoke-not-model-capability" + +THRESHOLDS: dict[str, dict[str, float]] = { + "claim_citation_precision": {"min": 1.0}, + "citation_completeness": {"min": 1.0}, + "required_facet_coverage": {"min": 1.0}, + "answer_coverage": {"min": 1.0}, + "correct_abstention_rate": {"min": 1.0}, + "unsupported_claim_rate": {"max": 0.0}, + "fully_grounded_answer_rate": {"min": 1.0}, +} + +CANONICAL_REF_RE = re.compile( + r"^(?:raw|wiki)/[^#\[\]\r\n]+\.md#\^(?:" + r"h-[1-6]-[0-9]+-[0-9a-f]{6}|[ptcf]-[0-9]+-[0-9a-f]{6})$" +) + + +class EvaluationProtocolError(RuntimeError): + """The run cannot support a trustworthy metric and must fail closed.""" + + +def _norm(value: str) -> str: + return " ".join(unicodedata.normalize("NFKC", value).split()).casefold() + + +def _normalize_question(value: str) -> str: + """Collapse cosmetic punctuation/symbol/control edits for case uniqueness.""" + folded = unicodedata.normalize("NFKC", value).casefold() + lexical = "".join( + character if unicodedata.category(character)[:1] in {"L", "N"} else " " + for character in folded + ) + return " ".join(lexical.split()) + + +def _metric(numerator: int, denominator: int, *, scope: str) -> dict[str, Any]: + if isinstance(numerator, bool) or isinstance(denominator, bool): + raise EvaluationProtocolError("metric counts must not be bool") + if not isinstance(numerator, int) or not isinstance(denominator, int): + raise EvaluationProtocolError("metric counts must be integers") + if denominator <= 0: + raise EvaluationProtocolError( + f"undefined metric denominator for {scope}; refusing to report 100%" + ) + if numerator < 0 or numerator > denominator: + raise EvaluationProtocolError( + f"invalid metric count for {scope}: {numerator}/{denominator}" + ) + value = numerator / denominator + if not math.isfinite(value): + raise EvaluationProtocolError(f"non-finite metric for {scope}") + return { + "numerator": numerator, + "denominator": denominator, + "value": value, + "scope": scope, + } + + +def _contains_canary(value: Any) -> bool: + if isinstance(value, str): + return gold.GOLD_CANARY in value + if isinstance(value, dict): + return any(_contains_canary(key) or _contains_canary(item) + for key, item in value.items()) + if isinstance(value, (list, tuple)): + return any(_contains_canary(item) for item in value) + return False + + +def _validate_fixture() -> tuple[list[dict], dict[str, dict]]: + if public_cases.PUBLIC_CASE_SCHEMA_VERSION != EXPECTED_PUBLIC_CASE_SCHEMA_VERSION: + raise EvaluationProtocolError("unsupported public fixture schema") + if gold.GOLD_SCHEMA_VERSION != EXPECTED_GOLD_SCHEMA_VERSION: + raise EvaluationProtocolError("unsupported independent Gold schema") + if public_cases.fixture_sha256() != public_cases.FROZEN_PUBLIC_FIXTURE_SHA256: + raise EvaluationProtocolError("public fixture digest drift") + if gold.gold_sha256() != gold.FROZEN_GOLD_SHA256: + raise EvaluationProtocolError("independent Gold digest drift") + + cases = public_cases.get_public_cases() + if not isinstance(cases, list) or not cases: + raise EvaluationProtocolError("public cases must be a non-empty list") + public_by_id: dict[str, dict] = {} + normalized_questions: set[str] = set() + for case in cases: + if not isinstance(case, dict) or set(case) != { + "case_id", "category", "question", "evidence" + }: + raise EvaluationProtocolError("invalid public case schema") + case_id = case["case_id"] + if not isinstance(case_id, str) or not case_id or case_id in public_by_id: + raise EvaluationProtocolError("missing or duplicate public case_id") + if not isinstance(case["category"], str) or not case["category"].strip(): + raise EvaluationProtocolError(f"{case_id}: invalid category") + if not isinstance(case["question"], str) or not case["question"].strip(): + raise EvaluationProtocolError(f"{case_id}: invalid question") + normalized_question = _normalize_question(case["question"]) + if not normalized_question: + raise EvaluationProtocolError( + f"{case_id}: question has no lexical content after normalization" + ) + if normalized_question in normalized_questions: + raise EvaluationProtocolError( + f"{case_id}: duplicate normalized public question" + ) + normalized_questions.add(normalized_question) + evidence = case["evidence"] + if not isinstance(evidence, list) or not evidence: + raise EvaluationProtocolError(f"{case_id}: evidence must be non-empty") + refs: set[str] = set() + for row in evidence: + if not isinstance(row, dict) or set(row) != {"citation", "text"}: + raise EvaluationProtocolError(f"{case_id}: invalid evidence schema") + ref, text = row["citation"], row["text"] + if not isinstance(ref, str) or not CANONICAL_REF_RE.fullmatch(ref): + raise EvaluationProtocolError(f"{case_id}: non-canonical evidence ref") + if ref in refs: + raise EvaluationProtocolError(f"{case_id}: duplicate evidence ref") + if not isinstance(text, str) or not text.strip(): + raise EvaluationProtocolError(f"{case_id}: empty evidence text") + refs.add(ref) + public_by_id[case_id] = case + + gold_by_id: dict[str, dict] = {} + if not isinstance(gold.GOLD_CASES, tuple) or not gold.GOLD_CASES: + raise EvaluationProtocolError("Gold must be a non-empty frozen tuple") + answerable_count = 0 + unanswerable_count = 0 + required_facet_count = 0 + for row in gold.GOLD_CASES: + if not isinstance(row, dict) or set(row) != { + "case_id", "answerable", "required_facets", "forbidden_evidence" + }: + raise EvaluationProtocolError("invalid Gold case schema") + case_id = row["case_id"] + if not isinstance(case_id, str) or not case_id or case_id in gold_by_id: + raise EvaluationProtocolError("missing or duplicate Gold case_id") + if not isinstance(row["answerable"], bool): + raise EvaluationProtocolError(f"{case_id}: answerable must be bool") + facets = row["required_facets"] + forbidden = row["forbidden_evidence"] + if not isinstance(facets, list) or not isinstance(forbidden, list): + raise EvaluationProtocolError(f"{case_id}: invalid Gold lists") + if row["answerable"]: + answerable_count += 1 + if not facets: + raise EvaluationProtocolError(f"{case_id}: answerable without facets") + else: + unanswerable_count += 1 + if facets: + raise EvaluationProtocolError(f"{case_id}: unanswerable has facets") + facet_ids: set[str] = set() + allowed: set[str] = set() + for facet in facets: + if not isinstance(facet, dict) or set(facet) != { + "facet_id", "accepted_variants" + }: + raise EvaluationProtocolError(f"{case_id}: invalid facet schema") + facet_id = facet["facet_id"] + variants = facet["accepted_variants"] + if (not isinstance(facet_id, str) or not facet_id + or facet_id in facet_ids): + raise EvaluationProtocolError(f"{case_id}: duplicate/invalid facet_id") + facet_ids.add(facet_id) + if not isinstance(variants, list) or not variants: + raise EvaluationProtocolError( + f"{case_id}/{facet_id}: no accepted variant" + ) + variant_ids: set[str] = set() + frozen_variants: set[tuple[str, str, tuple[str, ...]]] = set() + for variant in variants: + if not isinstance(variant, dict) or set(variant) != { + "variant_id", "claim_text", "value", "evidence_set" + }: + raise EvaluationProtocolError( + f"{case_id}/{facet_id}: invalid accepted variant schema" + ) + variant_id = variant["variant_id"] + claim_text = variant["claim_text"] + value = variant["value"] + evidence_set = variant["evidence_set"] + if (not isinstance(variant_id, str) or not variant_id.strip() + or variant_id in variant_ids): + raise EvaluationProtocolError( + f"{case_id}/{facet_id}: duplicate/invalid variant_id" + ) + variant_ids.add(variant_id) + if not isinstance(claim_text, str) or not claim_text.strip(): + raise EvaluationProtocolError( + f"{case_id}/{facet_id}/{variant_id}: invalid claim_text" + ) + if not isinstance(value, str) or not value.strip(): + raise EvaluationProtocolError( + f"{case_id}/{facet_id}/{variant_id}: invalid value" + ) + if (not isinstance(evidence_set, list) or not evidence_set + or any(not isinstance(ref, str) for ref in evidence_set) + or len(set(evidence_set)) != len(evidence_set)): + raise EvaluationProtocolError( + f"{case_id}/{facet_id}/{variant_id}: invalid evidence set" + ) + identity = ( + claim_text, _norm(value), tuple(sorted(evidence_set)) + ) + if identity in frozen_variants: + raise EvaluationProtocolError( + f"{case_id}/{facet_id}: duplicate accepted variant" + ) + frozen_variants.add(identity) + allowed.update(evidence_set) + required_facet_count += 1 + if (any(not isinstance(ref, str) for ref in forbidden) + or len(set(forbidden)) != len(forbidden)): + raise EvaluationProtocolError(f"{case_id}: invalid forbidden evidence") + if allowed & set(forbidden): + raise EvaluationProtocolError(f"{case_id}: evidence both allowed/forbidden") + gold_by_id[case_id] = row + + if set(public_by_id) != set(gold_by_id): + raise EvaluationProtocolError("public/Gold case_id denominators differ") + for case_id, case in public_by_id.items(): + evidence_refs = {row["citation"] for row in case["evidence"]} + row = gold_by_id[case_id] + allowed_refs = { + ref + for facet in row["required_facets"] + for variant in facet["accepted_variants"] + for ref in variant["evidence_set"] + } + classified = allowed_refs | set(row["forbidden_evidence"]) + if classified != evidence_refs: + raise EvaluationProtocolError( + f"{case_id}: Gold must classify every retrieved evidence ref exactly once" + ) + if answerable_count <= 0 or unanswerable_count <= 0 or required_facet_count <= 0: + raise EvaluationProtocolError("answerability/facet metric denominator is empty") + if _contains_canary(cases): + raise EvaluationProtocolError("Gold canary leaked into public adapter input") + return cases, gold_by_id + + +def _request_for_case(case: dict) -> dict[str, Any]: + request = { + "question": copy.deepcopy(case["question"]), + "evidence": copy.deepcopy(case["evidence"]), + } + if set(request) != {"question", "evidence"} or _contains_canary(request): + raise EvaluationProtocolError("adapter request contains non-public fields") + return request + + +def _validate_response(case: dict, response: Any) -> dict[str, Any]: + case_id = case["case_id"] + if _contains_canary(response): + raise EvaluationProtocolError(f"{case_id}: Gold canary leaked into output") + if not isinstance(response, dict) or set(response) != { + "decision", "answer", "claims" + }: + raise EvaluationProtocolError(f"{case_id}: invalid response schema") + decision = response["decision"] + answer = response["answer"] + claims = response["claims"] + if decision not in {"answer", "abstain"}: + raise EvaluationProtocolError(f"{case_id}: invalid decision") + if not isinstance(answer, str) or not isinstance(claims, list): + raise EvaluationProtocolError(f"{case_id}: invalid answer/claims types") + if decision == "abstain": + if answer != "" or claims != []: + raise EvaluationProtocolError( + f"{case_id}: abstention must have empty answer and claims" + ) + return copy.deepcopy(response) + if not claims: + raise EvaluationProtocolError(f"{case_id}: answer decision has no claims") + + retrieved_refs = {row["citation"] for row in case["evidence"]} + claim_ids: set[str] = set() + for claim in claims: + if not isinstance(claim, dict) or set(claim) != { + "claim_id", "text", "facet_id", "value", "citations" + }: + raise EvaluationProtocolError(f"{case_id}: invalid atomic claim schema") + claim_id = claim["claim_id"] + if (not isinstance(claim_id, str) or not claim_id.strip() + or claim_id in claim_ids): + raise EvaluationProtocolError(f"{case_id}: duplicate/invalid claim_id") + claim_ids.add(claim_id) + for key in ("text", "facet_id", "value"): + if not isinstance(claim[key], str) or not claim[key].strip(): + raise EvaluationProtocolError( + f"{case_id}/{claim_id}: invalid {key}" + ) + citations = claim["citations"] + if not isinstance(citations, list): + raise EvaluationProtocolError(f"{case_id}/{claim_id}: citations not list") + if (any(not isinstance(ref, str) for ref in citations) + or len(set(citations)) != len(citations)): + raise EvaluationProtocolError( + f"{case_id}/{claim_id}: duplicate/invalid exact citations" + ) + for ref in citations: + if not CANONICAL_REF_RE.fullmatch(ref): + raise EvaluationProtocolError( + f"{case_id}/{claim_id}: non-canonical citation" + ) + if ref not in retrieved_refs: + raise EvaluationProtocolError( + f"{case_id}/{claim_id}: citation was not in retrieval evidence" + ) + rendered = "\n".join(claim["text"] for claim in claims) + if answer != rendered: + raise EvaluationProtocolError( + f"{case_id}: answer must be the exact newline join of atomic claim text" + ) + return copy.deepcopy(response) + + +def collect_responses(adapter: Callable[[dict[str, Any]], Any], + cases: list[dict]) -> dict[str, dict]: + if not callable(adapter): + raise EvaluationProtocolError("adapter must be callable") + responses: dict[str, dict] = {} + for case in cases: + request = _request_for_case(case) + pristine = copy.deepcopy(request) + try: + raw = adapter(request) + except KeyboardInterrupt: + raise + except BaseException as exc: # fail closed, including SystemExit(0) + raise EvaluationProtocolError( + f"{case['case_id']}: adapter failed: {type(exc).__name__}: {exc}" + ) from exc + if request != pristine: + raise EvaluationProtocolError( + f"{case['case_id']}: adapter mutated its public request" + ) + responses[case["case_id"]] = _validate_response(case, raw) + return responses + + +def load_predictions(path: Path, cases: list[dict]) -> dict[str, dict]: + """Load JSONL emitted by a real system: {case_id, response}.""" + try: + lines = path.read_text(encoding="utf-8").splitlines() + except OSError as exc: + raise EvaluationProtocolError(f"cannot read predictions: {exc}") from exc + responses: dict[str, dict] = {} + case_by_id = {case["case_id"]: case for case in cases} + for line_number, line in enumerate(lines, start=1): + if not line.strip(): + raise EvaluationProtocolError( + f"predictions line {line_number} is empty" + ) + try: + row = json.loads(line) + except json.JSONDecodeError as exc: + raise EvaluationProtocolError( + f"predictions line {line_number} is invalid JSON" + ) from exc + if not isinstance(row, dict) or set(row) != {"case_id", "response"}: + raise EvaluationProtocolError( + f"predictions line {line_number} has invalid envelope" + ) + case_id = row["case_id"] + if case_id not in case_by_id or case_id in responses: + raise EvaluationProtocolError( + f"predictions line {line_number} has unknown/duplicate case_id" + ) + responses[case_id] = _validate_response(case_by_id[case_id], row["response"]) + if set(responses) != set(case_by_id): + missing = sorted(set(case_by_id) - set(responses)) + raise EvaluationProtocolError(f"predictions missing cases: {missing}") + return responses + + +def score_responses(cases: list[dict], gold_by_id: dict[str, dict], + responses: dict[str, dict], *, details: bool = False) -> dict: + case_ids = {case["case_id"] for case in cases} + if set(responses) != case_ids: + raise EvaluationProtocolError("response denominator differs from fixture") + + counts = { + "answerable_cases": 0, + "unanswerable_cases": 0, + "answered_answerable_cases": 0, + "correct_abstentions": 0, + "required_facets": 0, + "covered_required_facets": 0, + "citation_complete_facets": 0, + "claims": 0, + "unsupported_claims": 0, + "citation_pairs": 0, + "supported_citation_pairs": 0, + "fully_grounded_answers": 0, + } + case_details: list[dict] = [] + + for case in cases: + case_id = case["case_id"] + row = gold_by_id[case_id] + response = responses[case_id] + answerable = row["answerable"] + if answerable: + counts["answerable_cases"] += 1 + if response["decision"] == "answer": + counts["answered_answerable_cases"] += 1 + else: + counts["unanswerable_cases"] += 1 + if response["decision"] == "abstain": + counts["correct_abstentions"] += 1 + + facet_by_id = { + facet["facet_id"]: facet for facet in row["required_facets"] + } + forbidden = set(row["forbidden_evidence"]) + evaluated_claims: list[dict] = [] + claims_by_facet: dict[str, list[dict]] = {} + for claim in response["claims"]: + counts["claims"] += 1 + counts["citation_pairs"] += len(claim["citations"]) + facet = facet_by_id.get(claim["facet_id"]) + variants = facet["accepted_variants"] if facet else [] + value_correct = bool( + variants + and _norm(claim["value"]) + in {_norm(variant["value"]) for variant in variants} + ) + text_correct = bool( + variants + and claim["text"] + in {variant["claim_text"] for variant in variants} + ) + matching_variants = [ + variant for variant in variants + if claim["text"] == variant["claim_text"] + and _norm(claim["value"]) == _norm(variant["value"]) + ] + content_correct = bool(matching_variants) + allowed = { + ref + for variant in matching_variants + for ref in variant["evidence_set"] + } + selected = set(claim["citations"]) + complete_variants = [ + variant for variant in matching_variants + if set(variant["evidence_set"]) <= selected + ] + evidence_complete = bool( + complete_variants + ) + supported_pairs = sum( + 1 for ref in claim["citations"] if content_correct and ref in allowed + ) + counts["supported_citation_pairs"] += supported_pairs + supported = bool(content_correct and evidence_complete) + strict_supported = bool( + supported + and selected + and selected <= allowed + and not (selected & forbidden) + ) + if not supported: + counts["unsupported_claims"] += 1 + evaluated = { + "claim_id": claim["claim_id"], + "facet_id": claim["facet_id"], + "value_correct": value_correct, + "text_correct": text_correct, + "content_correct": content_correct, + "matched_variant_ids": [ + variant["variant_id"] for variant in matching_variants + ], + "complete_variant_ids": [ + variant["variant_id"] for variant in complete_variants + ], + "evidence_complete": evidence_complete, + "supported": supported, + "strict_supported": strict_supported, + "supported_citations": supported_pairs, + "selected_citations": len(claim["citations"]), + "selected_forbidden": sorted(selected & forbidden), + } + evaluated_claims.append(evaluated) + claims_by_facet.setdefault(claim["facet_id"], []).append(evaluated) + + facet_results: list[dict] = [] + for facet_id in facet_by_id: + counts["required_facets"] += 1 + candidates = claims_by_facet.get(facet_id, []) + covered = any(item["content_correct"] for item in candidates) + complete = any( + item["content_correct"] and item["evidence_complete"] + for item in candidates + ) + if covered: + counts["covered_required_facets"] += 1 + if complete: + counts["citation_complete_facets"] += 1 + facet_results.append({ + "facet_id": facet_id, + "canonical_claim_covered": covered, + "citation_complete": complete, + }) + + exact_facet_multiset = ( + len(response["claims"]) == len(facet_by_id) + and {claim["facet_id"] for claim in response["claims"]} + == set(facet_by_id) + ) + fully_grounded = bool( + answerable + and response["decision"] == "answer" + and exact_facet_multiset + and evaluated_claims + and all(item["strict_supported"] for item in evaluated_claims) + and all(item["citation_complete"] for item in facet_results) + ) + if fully_grounded: + counts["fully_grounded_answers"] += 1 + if details: + case_details.append({ + "case_id": case_id, + "answerable": answerable, + "decision": response["decision"], + "facets": facet_results, + "claims": evaluated_claims, + "fully_grounded": fully_grounded, + }) + + metrics = { + "claim_citation_precision": _metric( + counts["supported_citation_pairs"], counts["citation_pairs"], + scope="supported exact claim→citation pairs / all selected exact pairs", + ), + "citation_completeness": _metric( + counts["citation_complete_facets"], counts["required_facets"], + scope=("required facets with a claim matching one canonical variant " + "and containing that same variant's full evidence set / " + "all required facets"), + ), + "required_facet_coverage": _metric( + counts["covered_required_facets"], counts["required_facets"], + scope=("required facets stated with canonical text and value from " + "the same accepted variant / all required facets"), + ), + "answer_coverage": _metric( + counts["answered_answerable_cases"], counts["answerable_cases"], + scope="answer decisions on answerable cases / answerable cases", + ), + "correct_abstention_rate": _metric( + counts["correct_abstentions"], counts["unanswerable_cases"], + scope="abstentions on unanswerable cases / unanswerable cases", + ), + "unsupported_claim_rate": _metric( + counts["unsupported_claims"], counts["claims"], + scope=("claims not matching one canonical text+value+evidence-set " + "variant / all claims"), + ), + "fully_grounded_answer_rate": _metric( + counts["fully_grounded_answers"], counts["answerable_cases"], + scope=("answerable cases with every required facet exactly covered and " + "every citation allowed / answerable cases"), + ), + } + threshold_results: dict[str, dict] = {} + passed = True + for name, spec in THRESHOLDS.items(): + value = metrics[name]["value"] + metric_passed = True + if "min" in spec: + metric_passed = metric_passed and value >= spec["min"] + if "max" in spec: + metric_passed = metric_passed and value <= spec["max"] + threshold_results[name] = {**spec, "passed": metric_passed} + passed = passed and metric_passed + + result = { + "schema_version": PROTOCOL_VERSION, + "scope": SCOPE, + "interpretation": ( + "A pass validates the structured protocol and scoring mutations only; " + "it is not a model, retrieval, conversion, or open-world accuracy claim." + ), + "fixture": { + "license": public_cases.LICENSE, + "public_fixture_sha256": public_cases.FROZEN_PUBLIC_FIXTURE_SHA256, + "gold_sha256": gold.FROZEN_GOLD_SHA256, + "cases": len(cases), + "unique_questions": len({ + _normalize_question(case["question"]) for case in cases + }), + "answerable_cases": counts["answerable_cases"], + "unanswerable_cases": counts["unanswerable_cases"], + "required_facets": counts["required_facets"], + }, + "metrics": metrics, + "thresholds": threshold_results, + "passed": passed, + "status": "protocol-smoke-passed" if passed else "threshold-failed", + } + if details: + result["details"] = case_details + return result + + +def evaluate(adapter: Callable[[dict[str, Any]], Any], *, details: bool = False) -> dict: + cases, gold_by_id = _validate_fixture() + responses = collect_responses(adapter, cases) + # Gold is deliberately consulted only after all adapter calls have completed. + return score_responses(cases, gold_by_id, responses, details=details) + + +def evaluate_predictions(path: Path, *, details: bool = False) -> dict: + cases, gold_by_id = _validate_fixture() + responses = load_predictions(path, cases) + return score_responses(cases, gold_by_id, responses, details=details) + + +def _load_adapter(spec: str) -> Callable[[dict[str, Any]], Any]: + if ":" not in spec: + raise EvaluationProtocolError("adapter must use module:function syntax") + module_name, function_name = spec.rsplit(":", 1) + if not module_name or not function_name: + raise EvaluationProtocolError("adapter must use module:function syntax") + try: + module = importlib.import_module(module_name) + adapter = getattr(module, function_name) + except (ImportError, AttributeError) as exc: + raise EvaluationProtocolError(f"cannot load adapter {spec}: {exc}") from exc + if not callable(adapter): + raise EvaluationProtocolError(f"adapter {spec} is not callable") + return adapter + + +def _parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description=__doc__) + source = parser.add_mutually_exclusive_group() + source.add_argument( + "--adapter", + default="evals.answer_citation.reference_adapter:answer", + help="Python module:function receiving only {question,evidence}", + ) + source.add_argument( + "--predictions", type=Path, + help="JSONL from a real system; each row is {case_id,response}", + ) + parser.add_argument("--json", action="store_true") + parser.add_argument("--details", action="store_true") + return parser + + +def main(argv: list[str] | None = None) -> int: + args = _parser().parse_args(argv) + try: + if args.predictions is not None: + result = evaluate_predictions(args.predictions, details=args.details) + else: + result = evaluate(_load_adapter(args.adapter), details=args.details) + except KeyboardInterrupt: + raise + except BaseException as exc: + if isinstance(exc, EvaluationProtocolError): + error = str(exc) + else: + error = ( + "unexpected evaluator termination: " + f"{type(exc).__name__}: {exc}" + ) + payload = {"schema_version": PROTOCOL_VERSION, "error": error, + "passed": False, "protocol_error": True, + "status": "protocol-error"} + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + if args.json: + print(json.dumps(result, ensure_ascii=False, sort_keys=True, indent=2)) + else: + print(RESULT_PREFIX + json.dumps(result, ensure_ascii=False, sort_keys=True)) + return 0 if result["passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/cite-check/corpus.py b/evals/cite-check/corpus.py new file mode 100644 index 0000000..430834c --- /dev/null +++ b/evals/cite-check/corpus.py @@ -0,0 +1,154 @@ +"""引用核对管线的毒化评测语料（adversarial eval corpus）。 + +给「防线」本身立一把尺子：已知错引类型的种子案例 + 干净对照案例，度量各层的 +查全率（recall）与误报率（FP）。确定性层的期望由 pytest 钉死（回归闸门）； +语义层（盲填 / 反驳）的查全率由 run_eval.py --semantic 度量（报告，不断言—— +LLM 判定天然有方差，指标用于跟踪趋势）。 + +新增错引类型时：加一条 CASE + 期望，跑 run_eval.py 看哪层能抓——抓不住的就是 +下一个要补的防线。 +""" +from __future__ import annotations + +from pathlib import Path + +RAW_REL = "raw/articles/eval-corpus.md" + +RAW_DOC = """# 检索系统 Alpha/Beta 评测（评测语料） ^h-1-1-e00001 + +## 性能 ^h-2-1-e00002 + +系统 Alpha 的准确率为 87.5%，Beta 为 62.3%。 ^p-1-e0a001 + +Alpha 的 P95 延迟为 120ms，索引占用 16GB。 ^p-2-e0a002 + +## 失败分析 ^h-2-2-e00003 + +Beta 的错误中有 41% 来自多跳查询，而多跳查询只占全部流量的 9%。 ^p-3-e0a003 + +作者结论："Multi-hop failures dominate Beta's error profile." ^p-4-e0a004 + +## 建议 ^h-2-3-e00004 + +预算低于 100ms 时建议用 Beta，并对多跳查询做路由兜底。 ^p-5-e0a005 +""" + +R = "raw/articles/eval-corpus" + +# 每条 case：id / error_type / body（wiki 页正文块）/ l1（期望的确定性层判定： +# mismatch / imprecise-anchor / exempt-missing-basis / exempted / pass）/ +# semantic_expect_fail（语义层——盲填或反驳——应否拦截；仅 --semantic 模式度量） +CASES = [ + # ── 毒化：确定性层应拦 ───────────────────────────────────────── + dict(id="num-drift", error_type="数字漂移", + body=f"Alpha 的准确率为 97.5%[[{R}#^p-1-e0a001]]。", + l1="mismatch", semantic_expect_fail=True), + dict(id="unit-drift", error_type="计量单位数字漂移", + body=f"Alpha 的 P95 延迟为 125ms[[{R}#^p-2-e0a002]]。", + l1="mismatch", semantic_expect_fail=True), + dict(id="magnitude-swap", error_type="量级错位", + body=f"Alpha 的索引占用 16TB[[{R}#^p-2-e0a002]]。", + l1="mismatch", semantic_expect_fail=True), + dict(id="cross-section-anchor", error_type="跨节张冠李戴（数字不在被引节）", + body=f"Beta 的错误中有 41% 来自多跳查询[[{R}#^p-1-e0a001]]。", + l1="mismatch", semantic_expect_fail=True), + dict(id="sibling-anchor", error_type="锚点挂偏（数字在同节邻块）", + body=f"Beta 的错误中有 41% 来自多跳查询[[{R}#^p-4-e0a004]]。", + l1="imprecise-anchor", semantic_expect_fail=True), + dict(id="bare-exempt", error_type="[KB 推算] 无依据锚（洗白标记）", + body=f"Alpha 比 Beta 高 25.2 个百分点 [KB 推算][[{R}#^p-1-e0a001]]。", + l1="exempt-missing-basis", semantic_expect_fail=False), + dict(id="quote-fabrication", error_type="引文伪造（同文字系统未逐字命中）", + body=f"作者说\"Beta failures are mostly single-hop noise\"[[{R}#^p-4-e0a004]]。", + l1="imprecise-anchor", semantic_expect_fail=True), + dict(id="list-item-misattribution", error_type="条目级张冠李戴（块级 union 盲区）", + body=(f"- Beta 错误中 41% 来自多跳查询[[{R}#^p-3-e0a003]]\n" + f"- Alpha 的多跳错误同样占 41%[[{R}#^p-2-e0a002]]"), + l1="mismatch", semantic_expect_fail=True), + # ── 毒化：确定性层已知盲区，语义层应拦 ───────────────────────── + dict(id="subject-swap", error_type="主体张冠李戴（数字全对）", + body=f"Alpha 的错误中有 41% 来自多跳查询，该类只占流量 9%[[{R}#^p-3-e0a003]]。", + l1="pass", semantic_expect_fail=True), + dict(id="recommendation-flip", error_type="建议主体反转（数字在场）", + body=f"预算低于 100ms 时建议用 Alpha[[{R}#^p-5-e0a005]]。", + l1="pass", semantic_expect_fail=True), + dict(id="overgeneralize", error_type="过度概括（无数字）", + body=f"Beta 的所有错误都来自多跳查询[[{R}#^p-3-e0a003]]。", + l1="pass", semantic_expect_fail=True), + dict(id="role-swap", error_type="比例语义错置（错误占比说成流量占比）", + body=f"多跳查询占全部流量的比例高达 41%[[{R}#^p-3-e0a003]]。", + l1="pass", semantic_expect_fail=True), + # ── 干净对照：任何层都不得误报 ───────────────────────────────── + dict(id="clean-exact", error_type="干净：逐字数字", + body=f"Alpha 的准确率为 87.5%[[{R}#^p-1-e0a001]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-rounding", error_type="干净：约数舍入", + body=f"Alpha 的准确率约 88%[[{R}#^p-1-e0a001]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-format", error_type="干净：加粗/全角/格式差异", + body=f"Alpha 准确率 **87.5％**[[{R}#^p-1-e0a001]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-multi-cite", error_type="干净：一句多引用并列", + body=f"Alpha 准确率 87.5%、P95 延迟 120ms[[{R}#^p-1-e0a001]][[{R}#^p-2-e0a002]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-exempt", error_type="干净：[KB 推算] 带依据锚（合法派生算术）", + body=f"Alpha 比 Beta 高 25.2 个百分点 [KB 推算: ^p-1-e0a001][[{R}#^p-1-e0a001]]。", + l1="exempted", semantic_expect_fail=False), + dict(id="clean-quote", error_type="干净：逐字引文", + body=f"作者结论：\"Multi-hop failures dominate Beta's error profile.\"[[{R}#^p-4-e0a004]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-h-section", error_type="干净：^h- 节级引用（节内命中）", + body=f"性能对比见评测报告，Alpha 87.5%[[{R}#^h-2-1-e00002]]。", + l1="pass", semantic_expect_fail=False), + dict(id="clean-pending", error_type="干净：数字显式挂 [需要来源]", + body=f"对照组约 55%[需要来源]，Alpha 为 87.5%[[{R}#^p-1-e0a001]]。", + l1="pass", semantic_expect_fail=False), +] + +# 这是「案例本身是否被毒化」的分类，不是「确定性层当前是否能抓住」。 +# subject-swap 等 4 例虽然 l1=pass，仍是必须由语义层拦下的真毒化； +# 不能因为确定性层的已知盲区就把它们统计成「干净」。 +DETERMINISTIC_POISON_L1 = {"mismatch", "imprecise-anchor", "exempt-missing-basis"} + + +def case_kind(case: dict) -> str: + """返回 deterministic_poison / semantic_poison / clean（三者互斥）。""" + if case["l1"] in DETERMINISTIC_POISON_L1: + return "deterministic_poison" + if case["semantic_expect_fail"]: + return "semantic_poison" + return "clean" + +FM = """--- +title: "eval case: {cid}" +type: concept +created_date: 2026-07-02 +last_modified: 2026-07-02 +last_modified_by: LLM +status: draft +confidence: medium +source_count: 1 +sources: + - "[[raw/articles/eval-corpus]]" +tags: [] +--- + +# eval case: {cid} + +{body} +""" + + +def materialize(root: Path) -> None: + """把评测语料落成一个 KB_ROOT 布局（workspaces/main/...）。""" + ws = root / "workspaces" / "main" + (ws / "raw" / "articles").mkdir(parents=True, exist_ok=True) + (ws / "wiki" / "concepts").mkdir(parents=True, exist_ok=True) + (ws / "raw" / "articles" / "eval-corpus.md").write_text(RAW_DOC, encoding="utf-8") + for c in CASES: + (ws / "wiki" / "concepts" / f"{c['id']}.md").write_text( + FM.format(cid=c["id"], body=c["body"]), encoding="utf-8") + + +def case_page(cid: str) -> str: + return f"wiki/concepts/{cid}.md" diff --git a/evals/cite-check/run_eval.py b/evals/cite-check/run_eval.py new file mode 100644 index 0000000..c8d46bf --- /dev/null +++ b/evals/cite-check/run_eval.py @@ -0,0 +1,220 @@ +#!/usr/bin/env python +"""引用核对管线的对抗评测 runner——给防线量查全率/误报率。 + +用法（在引擎根运行）： + python evals/cite-check/run_eval.py # 只测确定性层（零 LLM，秒级） + python evals/cite-check/run_eval.py --semantic # 加测语义层（DeepSeek 双通道， + # 需 DEEPSEEK_API_KEY） + python evals/cite-check/run_eval.py --semantic --model deepseek-v4-pro + +输出：逐 case 判定表 + 分层指标（确定性层 recall / FP；语义层 recall / FP）。 +确定性层的期望另有 pytest 钉死（scripts/tests/test_cite_eval.py）——本 runner +用于看全貌与跟踪语义层趋势。 +""" +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +import tempfile +from pathlib import Path + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +sys.path.insert(0, str(EVAL_DIR)) +import corpus # noqa: E402 + +GATE = {"mismatch", "exempt-missing-basis"} +AUDIT_RESULT_PREFIX = "CITE_AUDIT_RESULT " + + +class SemanticExecutionError(RuntimeError): + """语义审计子进程没有产出可信的完成结果。""" + + +def parse_audit_result(result: subprocess.CompletedProcess[str]) -> bool: + """校验 audit.py 的机器可读结果与退出码，返回「语义是否未通过」。 + + 只有真正产生 UNSUPPORTED/CONTRADICTED verdict 的 ``semantic_failed`` + 才能计入毒化 recall。截断、无效 quote、跳过不可核对目标、0 实审 + 或执行异常都是评测未完成，统一拒绝，不得冒充「拦截成功」。 + """ + marker_lines = [ + line[len(AUDIT_RESULT_PREFIX):] + for line in result.stdout.splitlines() + if line.startswith(AUDIT_RESULT_PREFIX) + ] + if len(marker_lines) != 1: + raise SemanticExecutionError( + f"审计子进程结果标记数异常（{len(marker_lines)}）" + ) + try: + payload = json.loads(marker_lines[0]) + except (TypeError, json.JSONDecodeError) as exc: + raise SemanticExecutionError("审计子进程结果标记不是合法 JSON") from exc + if payload.get("ledger_errors", 0): + raise SemanticExecutionError("审计子进程报告了台账错误") + if payload.get("ledger_written", 0): + raise SemanticExecutionError("dry-run 却报告写入了台账") + try: + returned = int(payload["returned"]) + judged = int(payload["judged"]) + semantic_failed = int(payload["semantic_failed"]) + incomplete = int(payload["incomplete"]) + skipped = int(payload["skipped"]) + except (KeyError, TypeError, ValueError) as exc: + raise SemanticExecutionError("审计子进程缺失分类计数") from exc + counts = (returned, judged, semantic_failed, incomplete, skipped) + if any(value < 0 for value in counts) or semantic_failed > judged: + raise SemanticExecutionError( + f"审计计数非法：returned={returned}, judged={judged}, " + f"semantic_failed={semantic_failed}, incomplete={incomplete}, skipped={skipped}" + ) + if judged + incomplete + skipped != returned: + raise SemanticExecutionError( + f"审计分类未对账：{judged}+{incomplete}+{skipped}!={returned}" + ) + if payload.get("status") != "completed" or payload.get("dry_run") is not True: + raise SemanticExecutionError(f"审计子进程未完成 dry-run：{payload.get('status')!r}") + if incomplete: + raise SemanticExecutionError(f"审计有 {incomplete} 对核验未完成") + if skipped: + raise SemanticExecutionError(f"审计跳过了 {skipped} 对不可核对目标") + if judged == 0: + raise SemanticExecutionError("审计未产生任何语义 verdict（judged=0）") + if result.returncode not in (0, 1): + detail = (result.stderr or payload.get("error") or "").strip()[:300] + raise SemanticExecutionError( + f"审计子进程执行失败（exit={result.returncode}）{': ' + detail if detail else ''}" + ) + expected_rc = 1 if semantic_failed else 0 + if result.returncode != expected_rc: + raise SemanticExecutionError( + f"审计退出码 {result.returncode} 与 semantic_failed={semantic_failed} 矛盾" + ) + return semantic_failed > 0 + + +def run_k(kb_root: Path, args_list: list[str]) -> str: + env = dict(os.environ, KB_ROOT=str(kb_root)) + r = subprocess.run( + [sys.executable, str(ENGINE_ROOT / "scripts" / "k.py"), "--workspace", "main"] + args_list, + capture_output=True, text=True, env=env) + if r.returncode not in (0, 1): + raise RuntimeError( + f"k.py {' '.join(args_list[:2])} 失败（exit={r.returncode}）: {r.stderr.strip()[:300]}" + ) + return r.stdout + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--semantic", action="store_true", help="加测语义层（DeepSeek 双通道）") + ap.add_argument("--model", default="deepseek-v4-flash") + args = ap.parse_args() + + tmp = Path(tempfile.mkdtemp(prefix="cite-eval-")) + corpus.materialize(tmp) + + # ── 确定性层 ── + findings = json.loads(run_k(tmp, ["list-cite-mismatches", "--json"])) + by_page: dict[str, list] = {} + for f in findings: + by_page.setdefault(f["path"], []).append(f) + + det_ok, det_bad = 0, [] + print(f"{'case':28} {'错引类型':26} 期望L1 实际L1 判定") + for c in corpus.CASES: + page = corpus.case_page(c["id"]) + got = by_page.get(page, []) + # 实际判定：取最严重的一档（gate > imprecise > exempted > pass） + issues = [g["issue"] for g in got] + actual = ("mismatch" if "mismatch" in issues + else "exempt-missing-basis" if "exempt-missing-basis" in issues + else "imprecise-anchor" if "imprecise-anchor" in issues + else "exempted" if "exempted" in issues + else "pass") + # 判定口径：pass/exempted 要求无闸门/观察项；毒化要求期望档位出现在 findings 里 + # （允许并发多档——如裸 [KB 推算] 同时报 exempt-missing-basis 与 mismatch） + if c["l1"] in ("pass", "exempted"): + ok = actual == c["l1"] + else: + ok = c["l1"] in issues + det_ok += ok + if not ok: + det_bad.append((c["id"], c["l1"], actual)) + print(f"{c['id']:28} {c['error_type']:26} {c['l1']:13} {actual:13} {'✓' if ok else '✗'}") + + deterministic_poison = [c for c in corpus.CASES + if corpus.case_kind(c) == "deterministic_poison"] + semantic_poison = [c for c in corpus.CASES + if corpus.case_kind(c) == "semantic_poison"] + clean = [c for c in corpus.CASES if corpus.case_kind(c) == "clean"] + poisoned_l1 = deterministic_poison + bad_ids = {row[0] for row in det_bad} + l1_recall = sum(1 for c in poisoned_l1 if c["id"] not in bad_ids) + clean_ids = {c["id"] for c in clean} + fp = [b for b in det_bad if b[0] in clean_ids] + print(f"\n确定性层：期望命中 {det_ok}/{len(corpus.CASES)} | " + f"毒化查全 {l1_recall}/{len(poisoned_l1)} | 干净误报 {len(fp)}/{len(clean)}") + + sem_summary = "" + semantic_bad: list[tuple[str, bool, bool]] = [] + semantic_exec_error: str | None = None + if args.semantic: + # ── 语义层（DeepSeek 双通道，逐 case dry-run） ── + sem_rows = [] + for c in corpus.CASES: + if c["l1"] in GATE: + continue # 已被确定性闸门拦下的不进语义层（流程上到不了） + r = subprocess.run( + [sys.executable, str(ENGINE_ROOT / "tools" / "cite-audit" / "audit.py"), + "--workspace", "main", "--kb-root", str(tmp), + "--paths", corpus.case_page(c["id"]), + "--model", args.model, "--dry-run"], + capture_output=True, text=True, env=dict(os.environ, KB_ROOT=str(tmp))) + try: + failed = parse_audit_result(r) + except SemanticExecutionError as exc: + semantic_exec_error = f"{c['id']}: {exc}" + print(f"[semantic] {c['id']:28} 执行失败 ✗ — {exc}") + break + expect = c["semantic_expect_fail"] + sem_rows.append((c["id"], expect, failed)) + if failed != expect: + semantic_bad.append((c["id"], expect, failed)) + mark = "✓" if failed == expect else "✗" + print(f"[semantic] {c['id']:28} 期望{'拦' if expect else '过'} " + f"实际{'拦' if failed else '过'} {mark}") + if semantic_exec_error is None: + should_fail = [r for r in sem_rows if r[1]] + should_pass = [r for r in sem_rows if not r[1]] + rec = sum(1 for r in should_fail if r[2]) + sfp = sum(1 for r in should_pass if r[2]) + sem_poison_ids = {c["id"] for c in semantic_poison} + semantic_only = [r for r in sem_rows if r[0] in sem_poison_ids] + semantic_only_rec = sum(1 for r in semantic_only if r[2]) + sem_summary = ( + f"语义层（{args.model}）：语义毒化查全 " + f"{semantic_only_rec}/{len(semantic_only)} | " + f"进入语义层的全部毒化查全 {rec}/{len(should_fail)} | " + f"干净误报 {sfp}/{len(should_pass)}" + ) + print(f"\n{sem_summary}") + + print(f"\n评测语料: {len(corpus.CASES)} cases（确定性毒化 " + f"{len(deterministic_poison)} / 语义毒化 {len(semantic_poison)} / " + f"毒化合计 {len(deterministic_poison) + len(semantic_poison)} / " + f"干净 {len(clean)}）| 临时库: {tmp}") + if semantic_exec_error is not None: + print(f"语义层执行失败：{semantic_exec_error}", file=sys.stderr) + return 2 + if det_ok != len(corpus.CASES) or semantic_bad: + return 1 + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/evals/conversion_fidelity/LICENSE b/evals/conversion_fidelity/LICENSE new file mode 100644 index 0000000..2d6e4b9 --- /dev/null +++ b/evals/conversion_fidelity/LICENSE @@ -0,0 +1,7 @@ +CC0 1.0 Universal + +To the extent possible under law, the authors have waived all copyright and +related or neighboring rights to the synthetic fixtures and Gold definitions +in this directory. This work is published from the GroundMap project. + +See https://creativecommons.org/publicdomain/zero/1.0/ for the legal text. diff --git a/evals/conversion_fidelity/README.md b/evals/conversion_fidelity/README.md new file mode 100644 index 0000000..ce1d083 --- /dev/null +++ b/evals/conversion_fidelity/README.md @@ -0,0 +1,186 @@ +# Conversion Fidelity Protocol Smoke + +This evaluation checks whether GroundMap's **real** source conversion path +preserves critical evidence before ingest begins: + +```text +original PDF / DOCX / HTML + -> layout-aware PDF extraction / MarkItDown DOCX+HTML + -> scripts/convert.py + -> postprocess anchors + outline + -> converted Markdown scored against source-authored Gold +``` + +The fixture facts, organisations, product names, identifiers, and dates are +original synthetic material released under [CC0-1.0](LICENSE). Source binaries +are generated at runtime and are never committed. + +## Run the protocol smoke + +From the repository root: + +```bash +python evals/conversion_fidelity/run_eval.py +python evals/conversion_fidelity/run_eval.py --json +python evals/conversion_fidelity/run_eval.py --json --details +``` + +The default run uses a temporary `KB_ROOT`, invokes +`python scripts/convert.py --dir ... --force --ext .pdf,.docx,.html`, validates +every generated Markdown/outline pair with the production outline validator, +and removes the temporary directory afterward. + +To keep the generated sources and derivatives for inspection: + +```bash +python evals/conversion_fidelity/run_eval.py --details --keep-temp +``` + +The final `Artifacts:` line points to a system temporary directory with this +layout: + +```text +/kb-data/workspaces/conversion-fidelity/raw/ + calibration_protocol.pdf + calibration_protocol.md + calibration_protocol.outline.json + compact_reference.docx + compact_reference.md + compact_reference.outline.json + operational_notice.html + operational_notice.md + operational_notice.outline.json +``` + +Generate only the three original sources into an explicit temporary directory: + +```bash +OUT="$(mktemp -d)/conversion-sources" +python evals/conversion_fidelity/fixtures.py --output-dir "$OUT" +``` + +`--force` is required if one of the stable fixture names already exists. + +## Runtime dependencies + +The runner requires all of the following and exits with code `2` if any is +missing; it never converts a missing format into a skipped/green denominator: + +- `markitdown[all]` for the production conversion path; +- `reportlab` for PDF source generation; +- `python-docx` for DOCX source generation and source-native inspection; +- `pdfplumber` for PDF geometry and semantic-receipt inspection; +- `lxml` for source HTML DOM inspection. + +Poppler and LibreOffice are not +needed to calculate the metrics, but are recommended for visual fixture QA. + +## Authored fixtures and locators + +`gold.py` contains frozen, pre-registered Gold v4. It does not read converter +output. Every fact, qualifier, table, reading-order, list, and standalone-text row binds to an executable +`target_section`; facts/qualifiers/tables also bind to an exact block kind and +one-based ordinal. Every row retains an authored-source locator: + +- PDF page, visual section, table row, footnote, and left/right frame; +- DOCX outline heading, paragraph, table row, list, and footnote marker; +- HTML CSS selector, `/` coordinate, DOM list order, and note ID. + +Each PDF, DOCX, and HTML footnote also has an explicit marker-to-note relation: +both endpoints bind to a heading section plus paragraph ordinal, both patterns +must be unique, and the marker block must precede the note block. The HTML +relation additionally requires `[1](#note-1)` in the converted marker block. + +The three sources cover: + +- ASCII negative signs, `±`, decimals, units, ISO dates, and explicit negation; +- visual and semantic heading ladders plus named section counts; +- real numbered/bulleted Word lists; +- table header-to-row relationships with same-format distractor rows; +- qualifying footnotes and exception scope; +- PDF page transition and a deliberately difficult two-column page; +- DOCX paragraph order and HTML DOM order. + +The DOCX uses resolved `compact_reference_guide` tokens: Letter page geometry, +1-inch margins, explicit Normal/Title/Subtitle/Heading styles, real numbering +definitions, and a fixed 9360-DXA table whose `tblW`, `tblInd`, `tblGrid`, and +every `tcW` agree. + +The complete per-document heading inventories are also frozen. Their title, +level, order, and cardinality must match exactly; missing, duplicate, extra, or +re-levelled headings fail. + +## Receipts + +Every run records the observed SHA-256 of each source binary. Those hashes are +receipts, not claims that PDF or OOXML container bytes are reproducible across +generation times or runtimes. A separate frozen semantic fixture digest is +computed from the original formats with `pdfplumber`, `python-docx`, and +`lxml`; it never reads converter Markdown or Gold. + +## Metrics and gates + +All metric denominators must be non-zero. Every acceptance threshold is 100%: + +| Metric | Meaning | +|---|---| +| `critical_fact_preservation` | A fact matches exactly once in its declared section/block and exactly once in the document. | +| `qualifier_preservation` | A negation or scope qualifier matches exactly once at its declared coordinate. | +| `table_alignment` | Exact header and exact rows coexist in one real pipe table at the declared section/table ordinal. | +| `reading_order` | Every token is unique inside its declared section and remains in source order. | +| `standalone_text_preservation` | Document preamble title/subtitle and a standalone table caption remain exact, unique, and at their declared blocks. | +| `list_structure_preservation` | Every list item keeps its exact text, source order, and ordered/unordered kind in the declared section. | +| `section_count_preservation` | The exact ordered heading-title inventory survives, with no missing/extra/duplicate headings. | +| `heading_hierarchy_preservation` | The exact ordered `(title, level)` inventory survives. | +| `footnote_relation_preservation` | Marker and note are uniquely preserved at their declared blocks, ordered correctly, with the HTML link target intact. | +| `mutation_sensitivity` | Fourteen deliberate corruptions are each rejected by their locator-level oracle. | + +The fourteen mutation controls delete a negative sign, unit, negation, exact +table header; swap reading-order sentinels; move a fact to the wrong section; +flatten a table to prose; cross values between table rows; and insert an extra +heading. Three more delete a footnote marker, corrupt the HTML note target, and +move a note to the wrong block. One deletes a list item, and one deletes the +document subtitle. Additional pytest guards prove duplicate facts, +duplicate order tokens, and wrong heading levels fail. A mutation counts only +when its specific Gold row passed before mutation and failed afterward. + +Exit codes are fail-closed: + +- `0`: status is `protocol-smoke-passed`; all thresholds and mutation controls pass; +- `1`: the evaluation completed but at least one 100% gate failed; +- `2`: dependency, fixture, Gold, conversion, outline, denominator, or protocol + error, so no quality conclusion is valid. + +## Result semantics + +The result always carries a non-empty `not_measured` list. In particular, a +green synthetic protocol smoke does not measure open-world conversion accuracy, +OCR accuracy, visual fidelity beyond the explicit fixture geometry checks, or +downstream answer/citation correctness. A threshold failure is reported as +`protocol-smoke-failed`; a dependency or contract failure is `protocol-error`. + +## Visual QA + +The fixtures are evaluation inputs, but their layouts should still be checked +when the generator changes. Keep a run and render the originals: + +```bash +ARTIFACTS=/tmp/path-printed-by-the-runner +RAW="$ARTIFACTS/kb-data/workspaces/conversion-fidelity/raw" + +pdftoppm -png "$RAW/calibration_protocol.pdf" /tmp/calibration-protocol +python /path/to/documents-skill/render_docx.py \ + "$RAW/compact_reference.docx" --output_dir /tmp/compact-reference-render +``` + +Inspect every page image at 100% zoom. Confirm that the PDF columns do not +overlap, its table and footnote are legible, and the DOCX title/subtitle, +heading ladder, lists, table widths, wrapping, and page breaks are clean. + +## Scope boundary + +A green result proves 100% preservation only for this frozen, synthetic fixture +set and these executable coordinates. It does not imply open-world +PDF/DOCX/HTML fidelity, OCR quality, arbitrary tagged/untagged PDF reading +order, or citation semantic correctness. Add a pre-registered case before +fixing each newly observed conversion failure. diff --git a/evals/conversion_fidelity/__init__.py b/evals/conversion_fidelity/__init__.py new file mode 100644 index 0000000..73012d6 --- /dev/null +++ b/evals/conversion_fidelity/__init__.py @@ -0,0 +1 @@ +"""GroundMap source-format conversion fidelity evaluation.""" diff --git a/evals/conversion_fidelity/fixtures.py b/evals/conversion_fidelity/fixtures.py new file mode 100644 index 0000000..080ebb1 --- /dev/null +++ b/evals/conversion_fidelity/fixtures.py @@ -0,0 +1,683 @@ +#!/usr/bin/env python3 +"""Generate original PDF, DOCX, and HTML conversion-fidelity fixtures. + +All documents are synthetic and deterministic. Binary files are generated in +the caller's temporary directory; no fixture binary belongs in the repository. +""" +from __future__ import annotations + +import argparse +import hashlib +import importlib +import json +import re +import unicodedata +from pathlib import Path +from typing import Any + + +FROZEN_SEMANTIC_FIXTURE_SHA256 = ( + "82136d6cb2fcb6cadfa7480db28c4031a23386cc0a26f95bd64227e69961db4f" +) + + +class FixtureDependencyError(RuntimeError): + """A required public fixture-generation dependency is unavailable.""" + + +def require_dependencies() -> dict[str, Any]: + """Import public runtime dependencies or fail explicitly. + + The evaluation deliberately does not turn a missing format generator into + a skipped format, because that would silently reduce metric denominators. + """ + + modules: dict[str, Any] = {} + missing: list[str] = [] + for import_name, package_name in ( + ("reportlab", "reportlab"), + ("docx", "python-docx"), + ("pdfplumber", "pdfplumber"), + ("lxml", "lxml"), + ): + try: + modules[import_name] = importlib.import_module(import_name) + except ImportError: + missing.append(package_name) + if missing: + raise FixtureDependencyError( + "missing required public runtime dependencies: " + ", ".join(missing) + ) + return modules + + +def _set_docx_cell_width(cell: Any, width_dxa: int) -> None: + from docx.oxml import OxmlElement + from docx.oxml.ns import qn + + tc_pr = cell._tc.get_or_add_tcPr() + tc_w = tc_pr.find(qn("w:tcW")) + if tc_w is None: + tc_w = OxmlElement("w:tcW") + tc_pr.append(tc_w) + tc_w.set(qn("w:w"), str(width_dxa)) + tc_w.set(qn("w:type"), "dxa") + + +def _set_docx_table_geometry(table: Any, widths_dxa: tuple[int, ...]) -> None: + """Apply fixed DXA geometry (tblW/tblInd/tblGrid/tcW).""" + from docx.oxml import OxmlElement + from docx.oxml.ns import qn + + if sum(widths_dxa) != 9360: + raise ValueError("DOCX fixture table widths must total 9360 DXA") + table.autofit = False + tbl_pr = table._tbl.tblPr + tbl_w = tbl_pr.find(qn("w:tblW")) + if tbl_w is None: + tbl_w = OxmlElement("w:tblW") + tbl_pr.append(tbl_w) + tbl_w.set(qn("w:w"), "9360") + tbl_w.set(qn("w:type"), "dxa") + tbl_ind = tbl_pr.find(qn("w:tblInd")) + if tbl_ind is None: + tbl_ind = OxmlElement("w:tblInd") + tbl_pr.append(tbl_ind) + tbl_ind.set(qn("w:w"), "120") + tbl_ind.set(qn("w:type"), "dxa") + + grid = table._tbl.tblGrid + for child in list(grid): + grid.remove(child) + for width in widths_dxa: + col = OxmlElement("w:gridCol") + col.set(qn("w:w"), str(width)) + grid.append(col) + for row in table.rows: + for cell, width in zip(row.cells, widths_dxa, strict=True): + _set_docx_cell_width(cell, width) + + +def _configure_docx_list_numbering(document: Any) -> None: + """Bind List Bullet/Number to explicit compact-reference numbering XML.""" + from docx.oxml import OxmlElement + from docx.oxml.ns import qn + + numbering = document.part.numbering_part.element + abstract_ids = [ + int(node.get(qn("w:abstractNumId"))) + for node in numbering.findall(qn("w:abstractNum")) + ] + num_ids = [ + int(node.get(qn("w:numId"))) + for node in numbering.findall(qn("w:num")) + ] + next_abstract = max(abstract_ids, default=-1) + 1 + next_num = max(num_ids, default=0) + 1 + + for offset, (style_name, num_format, marker) in enumerate(( + ("List Bullet", "bullet", "•"), + ("List Number", "decimal", "%1."), + )): + abstract_id = next_abstract + offset + num_id = next_num + offset + abstract = OxmlElement("w:abstractNum") + abstract.set(qn("w:abstractNumId"), str(abstract_id)) + multi = OxmlElement("w:multiLevelType") + multi.set(qn("w:val"), "singleLevel") + abstract.append(multi) + level = OxmlElement("w:lvl") + level.set(qn("w:ilvl"), "0") + for tag, value in (("w:start", "1"), ("w:numFmt", num_format), + ("w:lvlText", marker), ("w:lvlJc", "left")): + node = OxmlElement(tag) + node.set(qn("w:val"), value) + level.append(node) + p_pr = OxmlElement("w:pPr") + tabs = OxmlElement("w:tabs") + tab = OxmlElement("w:tab") + tab.set(qn("w:val"), "num") + tab.set(qn("w:pos"), "540") + tabs.append(tab) + p_pr.append(tabs) + ind = OxmlElement("w:ind") + ind.set(qn("w:left"), "540") + ind.set(qn("w:hanging"), "270") + p_pr.append(ind) + spacing = OxmlElement("w:spacing") + spacing.set(qn("w:after"), "80") + spacing.set(qn("w:line"), "300") + spacing.set(qn("w:lineRule"), "auto") + p_pr.append(spacing) + level.append(p_pr) + abstract.append(level) + numbering.insert(len(numbering.findall(qn("w:abstractNum"))), abstract) + + concrete = OxmlElement("w:num") + concrete.set(qn("w:numId"), str(num_id)) + abstract_ref = OxmlElement("w:abstractNumId") + abstract_ref.set(qn("w:val"), str(abstract_id)) + concrete.append(abstract_ref) + numbering.append(concrete) + + style_p_pr = document.styles[style_name]._element.get_or_add_pPr() + old_num_pr = style_p_pr.find(qn("w:numPr")) + if old_num_pr is not None: + style_p_pr.remove(old_num_pr) + num_pr = OxmlElement("w:numPr") + ilvl = OxmlElement("w:ilvl") + ilvl.set(qn("w:val"), "0") + num_id_node = OxmlElement("w:numId") + num_id_node.set(qn("w:val"), str(num_id)) + num_pr.extend((ilvl, num_id_node)) + style_p_pr.append(num_pr) + + +def _configure_docx_styles(document: Any) -> None: + """Resolve the compact_reference_guide tokens into real Word styles.""" + from docx.enum.style import WD_STYLE_TYPE + from docx.oxml.ns import qn + from docx.shared import Inches, Pt, RGBColor + + section = document.sections[0] + section.page_width = Inches(8.5) + section.page_height = Inches(11) + section.top_margin = Inches(1) + section.right_margin = Inches(1) + section.bottom_margin = Inches(1) + section.left_margin = Inches(1) + section.header_distance = Inches(0.492) + section.footer_distance = Inches(0.492) + + styles = document.styles + normal = styles["Normal"] + normal.font.name = "Calibri" + normal.font.size = Pt(11) + normal.paragraph_format.space_before = Pt(0) + normal.paragraph_format.space_after = Pt(6) + normal.paragraph_format.line_spacing = 1.25 + + # Named compact technical masthead override: restrained and borderless. + title = styles["Title"] + title.font.name = "Calibri" + title.font.size = Pt(22) + title.font.bold = True + title.font.color.rgb = RGBColor.from_string("0B2545") + title.paragraph_format.space_before = Pt(0) + title.paragraph_format.space_after = Pt(8) + title.paragraph_format.line_spacing = 1.0 + # Some Word templates put a decorative rule on the built-in Title style. + # The fixture deliberately uses a restrained, borderless memo masthead so + # the generated source is stable across Word/LibreOffice renderers. + title_p_pr = title._element.get_or_add_pPr() + title_border = title_p_pr.find(qn("w:pBdr")) + if title_border is not None: + title_p_pr.remove(title_border) + subtitle = styles["Subtitle"] + subtitle.font.name = "Calibri" + subtitle.font.size = Pt(10.5) + subtitle.font.italic = False + subtitle.font.color.rgb = RGBColor.from_string("475467") + subtitle.paragraph_format.space_before = Pt(0) + subtitle.paragraph_format.space_after = Pt(12) + subtitle.paragraph_format.line_spacing = 1.0 + + heading_tokens = { + "Heading 1": (16, "2E74B5", 18, 10), + "Heading 2": (13, "2E74B5", 14, 7), + "Heading 3": (12, "1F4D78", 10, 5), + } + for name, (size, color, before, after) in heading_tokens.items(): + style = styles[name] + style.font.name = "Calibri" + style.font.size = Pt(size) + style.font.bold = True + style.font.color.rgb = RGBColor.from_string(color) + style.paragraph_format.space_before = Pt(before) + style.paragraph_format.space_after = Pt(after) + + # Built-in list styles carry actual numbering definitions. Resolve the + # compact preset's paragraph tokens instead of relying on Word defaults. + for name in ("List Bullet", "List Number"): + style = styles[name] + if style.type != WD_STYLE_TYPE.PARAGRAPH: + raise RuntimeError(f"{name} is not a paragraph style") + style.font.name = "Calibri" + style.font.size = Pt(11) + style.paragraph_format.left_indent = Inches(0.375) + style.paragraph_format.first_line_indent = Inches(-0.188) + style.paragraph_format.space_after = Pt(4) + style.paragraph_format.line_spacing = 1.25 + _configure_docx_list_numbering(document) + + +def build_pdf(path: Path) -> None: + """Build a two-page, two-column calibration protocol PDF.""" + from reportlab.lib import colors + from reportlab.lib.enums import TA_CENTER + from reportlab.lib.pagesizes import letter + from reportlab.lib.styles import ParagraphStyle, getSampleStyleSheet + from reportlab.lib.units import inch + from reportlab.platypus import ( + BaseDocTemplate, + Frame, + FrameBreak, + PageBreak, + PageTemplate, + Paragraph, + Spacer, + Table, + TableStyle, + ) + + path.parent.mkdir(parents=True, exist_ok=True) + styles = getSampleStyleSheet() + styles.add(ParagraphStyle( + "FixtureTitle", parent=styles["Title"], fontName="Helvetica-Bold", + fontSize=18, leading=22, alignment=TA_CENTER, spaceAfter=12, + )) + styles.add(ParagraphStyle( + "FixtureH1", parent=styles["Heading1"], fontName="Helvetica-Bold", + fontSize=15, leading=18, spaceBefore=8, spaceAfter=7, + )) + styles.add(ParagraphStyle( + "FixtureH2", parent=styles["Heading2"], fontName="Helvetica-Bold", + fontSize=12, leading=15, spaceBefore=7, spaceAfter=5, + )) + styles.add(ParagraphStyle( + "FixtureBody", parent=styles["BodyText"], fontName="Helvetica", + fontSize=10.5, leading=14, spaceAfter=6, + )) + styles.add(ParagraphStyle( + "FixtureFootnote", parent=styles["BodyText"], fontName="Helvetica", + fontSize=8.5, leading=11, textColor=colors.HexColor("#404040"), + )) + + page_width, page_height = letter + margin = 0.72 * inch + gap = 0.28 * inch + column_width = (page_width - (2 * margin) - gap) / 2 + full_frame = Frame( + margin, margin, page_width - 2 * margin, page_height - 2 * margin, + id="full", leftPadding=0, rightPadding=0, topPadding=0, bottomPadding=0, + ) + left_frame = Frame( + margin, margin, column_width, page_height - 2 * margin, + id="left", leftPadding=0, rightPadding=5, topPadding=0, bottomPadding=0, + ) + right_frame = Frame( + margin + column_width + gap, margin, column_width, + page_height - 2 * margin, id="right", leftPadding=5, rightPadding=0, + topPadding=0, bottomPadding=0, + ) + document = BaseDocTemplate( + str(path), pagesize=letter, leftMargin=margin, rightMargin=margin, + topMargin=margin, bottomMargin=margin, + title="Cold Storage Calibration Protocol", + author="GroundMap synthetic evaluation", + ) + document.addPageTemplates([ + PageTemplate(id="full-page", frames=[full_frame]), + PageTemplate(id="two-column", frames=[left_frame, right_frame]), + ]) + + story: list[Any] = [ + Paragraph("Cold Storage Calibration Protocol", styles["FixtureTitle"]), + Paragraph("1. Measurement controls", styles["FixtureH1"]), + Paragraph( + "The minimum chamber offset is -7.25 °C during maintenance mode.", + styles["FixtureBody"], + ), + Paragraph( + "Production use is not permitted while that offset is active.1", + styles["FixtureBody"], + ), + Paragraph( + "The probe tolerance is ±0.08 mm, valid from 2031-04-17.", + styles["FixtureBody"], + ), + Paragraph("2. Acceptance register", styles["FixtureH1"]), + Table( + [ + ["Unit", "Pressure", "Decision"], + ["Atlas-7", "2.45 bar", "ACCEPT"], + ["Boreal-9", "3.10 bar", "REJECT"], + ], + colWidths=[2.2 * inch, 1.7 * inch, 1.55 * inch], + repeatRows=1, + style=TableStyle([ + ("BACKGROUND", (0, 0), (-1, 0), colors.HexColor("#E8EEF5")), + ("FONTNAME", (0, 0), (-1, 0), "Helvetica-Bold"), + ("GRID", (0, 0), (-1, -1), 0.5, colors.HexColor("#667085")), + ("FONTNAME", (0, 1), (-1, -1), "Helvetica"), + ("FONTSIZE", (0, 0), (-1, -1), 9.5), + ("LEADING", (0, 0), (-1, -1), 12), + ("VALIGN", (0, 0), (-1, -1), "MIDDLE"), + ("TOPPADDING", (0, 0), (-1, -1), 5), + ("BOTTOMPADDING", (0, 0), (-1, -1), 5), + ]), + ), + Spacer(1, 10), + Paragraph( + "1 The maintenance exception applies only when service ticket MX-204 is open.", + styles["FixtureFootnote"], + ), + PageBreak(), + ] + + # The second page uses two explicit frames. Content-stream order is left + # column then right column, giving a deterministic reading-order oracle. + document.handle_nextPageTemplate("two-column") + story.extend([ + Paragraph("3. Two-column operating sequence", styles["FixtureH1"]), + Paragraph("LEFT-START: isolate the chamber supply.", styles["FixtureBody"]), + Paragraph("LEFT-MIDDLE: record the baseline pressure.", styles["FixtureBody"]), + Paragraph("LEFT-END: close the isolation checklist.", styles["FixtureBody"]), + FrameBreak(), + Paragraph("RIGHT-START: restore the sensor bridge.", styles["FixtureBody"]), + Paragraph("RIGHT-MIDDLE: verify the alarm state.", styles["FixtureBody"]), + Paragraph("RIGHT-END: release the maintenance hold.", styles["FixtureBody"]), + ]) + document.build(story) + + +def build_docx(path: Path) -> None: + """Build a compact-reference DOCX with real styles, lists, and table.""" + from docx import Document + from docx.enum.table import WD_ALIGN_VERTICAL + from docx.shared import Pt + + path.parent.mkdir(parents=True, exist_ok=True) + document = Document() + _configure_docx_styles(document) + document.core_properties.title = "Compact Reactor Reference Guide" + document.core_properties.author = "GroundMap synthetic evaluation" + + document.add_heading("Compact Reactor Reference Guide", level=0) + document.add_paragraph("Controlled limits and release checks", style="Subtitle") + document.add_heading("Operating envelope", level=1) + document.add_heading("Verified limits", level=2) + document.add_paragraph( + "The inlet pressure correction is -0.45 kPa at 18.6 °C." + ) + document.add_paragraph( + "This correction is not valid during purge mode." + ) + document.add_paragraph( + "The alignment tolerance is ±0.14 mm, effective 2032-06-30." + ) + + document.add_heading("Operator checklist", level=2) + document.add_paragraph( + "Confirm that valve C-17 is locked.", style="List Number" + ) + document.add_paragraph( + "Record the inlet temperature before adjustment.", style="List Number" + ) + document.add_paragraph( + "Do not bypass the purge interlock.", style="List Bullet" + ) + + document.add_heading("Calibration matrix", level=1) + table = document.add_table(rows=1, cols=4) + table.style = "Table Grid" + headers = ("Model", "Limit", "Unit", "Condition") + for cell, value in zip(table.rows[0].cells, headers, strict=True): + cell.text = value + cell.vertical_alignment = WD_ALIGN_VERTICAL.CENTER + for run in cell.paragraphs[0].runs: + run.bold = True + rows = ( + ("Cobalt-X", "-0.45", "kPa", "purge mode excluded"), + ("Dune-Y", "0.62", "kPa", "standby only"), + ) + for values in rows: + cells = table.add_row().cells + for cell, value in zip(cells, values, strict=True): + cell.text = value + cell.vertical_alignment = WD_ALIGN_VERTICAL.CENTER + for paragraph in cell.paragraphs: + paragraph.paragraph_format.space_after = Pt(0) + _set_docx_table_geometry(table, (2160, 1800, 1800, 3600)) + + document.add_heading("Scope notes", level=2) + paragraph = document.add_paragraph() + paragraph.add_run("The Cobalt-X row carries qualifier ") + marker = paragraph.add_run("1") + marker.font.superscript = True + paragraph.add_run(".") + note = document.add_paragraph() + note_run = note.add_run("1 Applies only to reactor batch RX-31; it does not apply to RX-32.") + note_run.font.size = Pt(9) + + document.add_heading("Release order", level=1) + for sentence in ( + "DOCX-ORDER-1: lock the manifold.", + "DOCX-ORDER-2: verify the correction.", + "DOCX-ORDER-3: archive the signed record.", + ): + document.add_paragraph(sentence) + document.save(path) + + +def build_html(path: Path) -> None: + """Build a semantic HTML notice with headings, table, list, and footnote.""" + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text( + """ + +Sealed Transfer Operational Notice + +

+

Sealed Transfer Operational Notice

+
+

Measurement limits

+

The stability change is -1.75 % after the 4.2 h conditioning cycle.

+

This value is not valid after 2032-09-30.1

+

The sensor tolerance remains ±0.05 mg/L.

+
+
+

Required actions

+
    +
  1. HTML-ORDER-1: seal the transfer coupling.
    2. +
  2. HTML-ORDER-2: capture the reference sample.
    3. +
  3. HTML-ORDER-3: release the batch record.
    4. +
+
+
+

Validation register

+ + + + + + + +
Transfer limits
DeviceConcentrationCondition
Helios-Q18.6 mg/Lsealed transfer only
Iris-R21.4 mg/Lopen rinse only
+
+
+

Notes

+

1 The expiry restriction applies even when a supervisor approves an override.

+
+
+ + +""", + encoding="utf-8", + ) + + +def materialize(output_dir: Path) -> dict[str, Path]: + """Generate every source fixture and return stable document IDs.""" + require_dependencies() + output_dir.mkdir(parents=True, exist_ok=True) + paths = { + "pdf_protocol": output_dir / "calibration_protocol.pdf", + "docx_reference": output_dir / "compact_reference.docx", + "html_notice": output_dir / "operational_notice.html", + } + build_pdf(paths["pdf_protocol"]) + build_docx(paths["docx_reference"]) + build_html(paths["html_notice"]) + return paths + + +def _semantic_text(value: str | None) -> str: + if not value: + return "" + value = unicodedata.normalize("NFC", value).replace("\u00a0", " ") + return re.sub(r"\s+", " ", value).strip() + + +def _pdf_semantic_payload(path: Path) -> dict[str, Any]: + """Read authored PDF text/geometry without using the converter.""" + import pdfplumber + + pages: list[dict[str, Any]] = [] + with pdfplumber.open(path) as document: + for page_number, page in enumerate(document.pages, start=1): + words = page.extract_words( + x_tolerance=2, + y_tolerance=3, + keep_blank_chars=False, + use_text_flow=False, + ) + semantic_words = [ + { + "text": _semantic_text(word["text"]), + "x0": round(float(word["x0"]), 2), + "top": round(float(word["top"]), 2), + "x1": round(float(word["x1"]), 2), + "bottom": round(float(word["bottom"]), 2), + } + for word in sorted( + words, + key=lambda word: ( + 0 if float(word["x0"]) < float(page.width) / 2 else 1, + round(float(word["top"]), 2), + float(word["x0"]), + ), + ) + ] + tables = [ + [[_semantic_text(cell) for cell in row] for row in table] + for table in page.extract_tables() + ] + pages.append({ + "page_number": page_number, + "width": round(float(page.width), 2), + "height": round(float(page.height), 2), + "words": semantic_words, + "tables": tables, + }) + return {"format": "pdf", "pages": pages} + + +def _docx_semantic_payload(path: Path) -> dict[str, Any]: + """Read Word paragraphs/styles and cell relations from OOXML semantics.""" + from docx import Document + + document = Document(path) + return { + "format": "docx", + "paragraphs": [ + {"style": paragraph.style.name, "text": _semantic_text(paragraph.text)} + for paragraph in document.paragraphs + if _semantic_text(paragraph.text) + ], + "tables": [ + [[_semantic_text(cell.text) for cell in row.cells] for row in table.rows] + for table in document.tables + ], + } + + +def _html_semantic_payload(path: Path) -> dict[str, Any]: + """Read semantic DOM nodes directly with lxml, independent of Markdown.""" + from lxml import html + + root = html.parse(str(path)).getroot() + nodes: list[dict[str, Any]] = [] + for element in root.xpath("//main//*"): + nodes.append({ + "tag": str(element.tag).lower(), + "id": element.get("id") or "", + "href": element.get("href") or "", + "direct_text": _semantic_text(element.text), + "tail": _semantic_text(element.tail), + }) + return {"format": "html", "nodes": nodes} + + +def semantic_fixture_payload(source_paths: dict[str, Path]) -> dict[str, Any]: + """Return a source-native semantic manifest, never converter Markdown.""" + expected = {"pdf_protocol", "docx_reference", "html_notice"} + if set(source_paths) != expected: + raise ValueError(f"semantic fixture paths differ: {sorted(source_paths)}") + return { + "schema_version": 1, + "documents": { + "pdf_protocol": _pdf_semantic_payload(source_paths["pdf_protocol"]), + "docx_reference": _docx_semantic_payload(source_paths["docx_reference"]), + "html_notice": _html_semantic_payload(source_paths["html_notice"]), + }, + } + + +def _json_digest(payload: Any) -> str: + canonical = json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def semantic_document_digests(payload: dict[str, Any]) -> dict[str, str]: + documents = payload.get("documents") + if not isinstance(documents, dict) or not documents: + raise ValueError("semantic fixture payload has no documents") + return {doc_id: _json_digest(document) for doc_id, document in documents.items()} + + +def semantic_fixture_digest(source_paths: dict[str, Path]) -> str: + return _json_digest(semantic_fixture_payload(source_paths)) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + description="Generate the original conversion-fidelity source fixtures." + ) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument( + "--force", action="store_true", + help="replace fixture names if they already exist in the output directory", + ) + args = parser.parse_args(argv) + expected_names = { + "calibration_protocol.pdf", "compact_reference.docx", + "operational_notice.html", + } + conflicts = [ + path for path in (args.output_dir / name for name in expected_names) + if path.exists() + ] + if conflicts and not args.force: + print( + "refusing to replace existing fixture files without --force: " + + ", ".join(str(path) for path in conflicts) + ) + return 2 + try: + generated = materialize(args.output_dir) + except (FixtureDependencyError, OSError, RuntimeError, ValueError) as exc: + print(f"fixture generation failed: {exc}") + return 2 + payload = { + doc_id: {"path": str(path), "bytes": path.stat().st_size} + for doc_id, path in generated.items() + } + print(json.dumps(payload, ensure_ascii=False, sort_keys=True, indent=2)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/conversion_fidelity/gold.py b/evals/conversion_fidelity/gold.py new file mode 100644 index 0000000..0e9fcde --- /dev/null +++ b/evals/conversion_fidelity/gold.py @@ -0,0 +1,491 @@ +"""Pre-registered conversion-fidelity Gold, independent of converter output. + +Gold v4 turns every semantic assertion into an executable source coordinate: +an exact Markdown heading section plus, where relevant, a canonical block kind +and ordinal. The coordinates are authored from the synthetic source fixtures; +nothing here is sampled from MarkItDown output. +""" +from __future__ import annotations + +import hashlib +import json +from typing import Any + + +GOLD_SCHEMA_VERSION = 4 +FROZEN_GOLD_SHA256 = "2289da6634ff0925f3b13d3c7bc4e3c320c4ee7e8c94a8b39ebe974567f72315" +EXPECTED_DOCUMENTS = { + "pdf_protocol": {"source_name": "calibration_protocol.pdf", "format": "pdf"}, + "docx_reference": {"source_name": "compact_reference.docx", "format": "docx"}, + "html_notice": {"source_name": "operational_notice.html", "format": "html"}, +} + + +def _section(title: str, level: int) -> dict[str, Any]: + return {"title": title, "level": level} + + +def _paragraph(ordinal: int) -> dict[str, Any]: + return {"kind": "paragraph", "ordinal": ordinal} + + +# The complete Markdown heading inventory is exact: same title, level, order, +# and cardinality. Any missing, extra, duplicate, or re-levelled heading fails. +HEADING_INVENTORIES = { + "pdf_protocol": ( + {"title": "Cold Storage Calibration Protocol", "level": 1}, + {"title": "1. Measurement controls", "level": 2}, + {"title": "2. Acceptance register", "level": 2}, + {"title": "3. Two-column operating sequence", "level": 2}, + ), + "docx_reference": ( + {"title": "Operating envelope", "level": 1}, + {"title": "Verified limits", "level": 2}, + {"title": "Operator checklist", "level": 2}, + {"title": "Calibration matrix", "level": 1}, + {"title": "Scope notes", "level": 2}, + {"title": "Release order", "level": 1}, + ), + "html_notice": ( + {"title": "Sealed Transfer Operational Notice", "level": 1}, + {"title": "Measurement limits", "level": 2}, + {"title": "Required actions", "level": 2}, + {"title": "Validation register", "level": 2}, + {"title": "Notes", "level": 2}, + ), +} + + +# Regexes are exact about signs, values, units, dates, and negation. Each row +# must match exactly once in its declared block and exactly once in the entire +# document, preventing both wrong-section relocation and duplicate laundering. +CRITICAL_FACTS = ( + { + "id": "pdf-negative-temperature", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 > section 1 > paragraph 1", + "target_section": _section("1. Measurement controls", 2), + "target_block": _paragraph(1), + "pattern": r"minimum chamber offset is -7\.25\s*°\s*C during maintenance mode", + }, + { + "id": "pdf-plus-minus-tolerance", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 > section 1 > paragraph 3", + "target_section": _section("1. Measurement controls", 2), + "target_block": _paragraph(3), + "pattern": r"probe tolerance is ±0\.08\s*mm,? valid from 2031-04-17", + }, + { + "id": "docx-negative-pressure", + "doc_id": "docx_reference", + "source_locator": "DOCX Operating envelope/Verified limits > paragraph 1", + "target_section": _section("Verified limits", 2), + "target_block": _paragraph(1), + "pattern": r"inlet pressure correction is -0\.45\s*kPa at 18\.6\s*°\s*C", + }, + { + "id": "docx-plus-minus-tolerance", + "doc_id": "docx_reference", + "source_locator": "DOCX Operating envelope/Verified limits > paragraph 3", + "target_section": _section("Verified limits", 2), + "target_block": _paragraph(3), + "pattern": r"alignment tolerance is ±0\.14\s*mm,? effective 2032-06-30", + }, + { + "id": "html-negative-change", + "doc_id": "html_notice", + "source_locator": "HTML css=#limits > p:nth-of-type(1)", + "target_section": _section("Measurement limits", 2), + "target_block": _paragraph(1), + "pattern": r"stability change is -1\.75\s*% after the 4\.2\s*h conditioning cycle", + }, + { + "id": "html-plus-minus-tolerance", + "doc_id": "html_notice", + "source_locator": "HTML css=#limits > p:nth-of-type(3)", + "target_section": _section("Measurement limits", 2), + "target_block": _paragraph(3), + "pattern": r"sensor tolerance remains ±0\.05\s*mg/L", + }, +) + + +QUALIFIERS = ( + { + "id": "pdf-production-negation", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 > section 1 > paragraph 2", + "target_section": _section("1. Measurement controls", 2), + "target_block": _paragraph(2), + "pattern": r"Production use is not permitted while that offset is active", + }, + { + "id": "pdf-footnote-scope", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 > footnote 1", + "target_section": _section("2. Acceptance register", 2), + "target_block": _paragraph(1), + "pattern": r"maintenance exception applies only when service ticket MX-204 is open", + }, + { + "id": "docx-purge-negation", + "doc_id": "docx_reference", + "source_locator": "DOCX Operating envelope/Verified limits > paragraph 2", + "target_section": _section("Verified limits", 2), + "target_block": _paragraph(2), + "pattern": r"correction is not valid during purge mode", + }, + { + "id": "docx-footnote-batch-scope", + "doc_id": "docx_reference", + "source_locator": "DOCX Scope notes > footnote marker 1", + "target_section": _section("Scope notes", 2), + "target_block": _paragraph(2), + "pattern": r"Applies only to reactor batch RX-31;? it does not apply to RX-32", + }, + { + "id": "html-expiry-negation", + "doc_id": "html_notice", + "source_locator": "HTML css=#limits > p:nth-of-type(2)", + "target_section": _section("Measurement limits", 2), + "target_block": _paragraph(2), + "pattern": r"value is not valid after 2032-09-30", + }, + { + "id": "html-footnote-override", + "doc_id": "html_notice", + "source_locator": "HTML css=#note-1", + "target_section": _section("Notes", 2), + "target_block": _paragraph(1), + "pattern": r"expiry restriction applies even when a supervisor approves an override", + }, +) + + +TABLE_RELATIONS = ( + { + "id": "pdf-acceptance-register", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 > section 2 > table rows 1-2", + "target_section": _section("2. Acceptance register", 2), + "target_block": {"kind": "table", "ordinal": 1}, + "headers": ("Unit", "Pressure", "Decision"), + "rows": ( + ("Atlas-7", "2.45 bar", "ACCEPT"), + ("Boreal-9", "3.10 bar", "REJECT"), + ), + }, + { + "id": "docx-calibration-matrix", + "doc_id": "docx_reference", + "source_locator": "DOCX Calibration matrix > table 1 rows 1-2", + "target_section": _section("Calibration matrix", 1), + "target_block": {"kind": "table", "ordinal": 1}, + "headers": ("Model", "Limit", "Unit", "Condition"), + "rows": ( + ("Cobalt-X", "-0.45", "kPa", "purge mode excluded"), + ("Dune-Y", "0.62", "kPa", "standby only"), + ), + }, + { + "id": "html-validation-register", + "doc_id": "html_notice", + "source_locator": "HTML css=#register table > thead/tbody", + "target_section": _section("Validation register", 2), + "target_block": {"kind": "table", "ordinal": 1}, + "headers": ("Device", "Concentration", "Condition"), + "rows": ( + ("Helios-Q", "18.6 mg/L", "sealed transfer only"), + ("Iris-R", "21.4 mg/L", "open rinse only"), + ), + }, +) + + +READING_ORDERS = ( + { + "id": "pdf-page-transition-order", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 section 1 -> page 1 section 2 -> page 2 section 3", + "target_section": _section("Cold Storage Calibration Protocol", 1), + "tokens": ( + "1. Measurement controls", + "2. Acceptance register", + "3. Two-column operating sequence", + ), + }, + { + "id": "pdf-two-column-order", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 2 > left frame, then right frame", + "target_section": _section("3. Two-column operating sequence", 2), + "tokens": ( + "LEFT-START: isolate the chamber supply.", + "LEFT-MIDDLE: record the baseline pressure.", + "LEFT-END: close the isolation checklist.", + "RIGHT-START: restore the sensor bridge.", + "RIGHT-MIDDLE: verify the alarm state.", + "RIGHT-END: release the maintenance hold.", + ), + }, + { + "id": "docx-release-order", + "doc_id": "docx_reference", + "source_locator": "DOCX Release order > paragraphs 1-3", + "target_section": _section("Release order", 1), + "tokens": ( + "DOCX-ORDER-1: lock the manifold.", + "DOCX-ORDER-2: verify the correction.", + "DOCX-ORDER-3: archive the signed record.", + ), + }, + { + "id": "html-dom-order", + "doc_id": "html_notice", + "source_locator": "HTML css=#actions > ol > li:nth-child(1..3)", + "target_section": _section("Required actions", 2), + "tokens": ( + "HTML-ORDER-1: seal the transfer coupling.", + "HTML-ORDER-2: capture the reference sample.", + "HTML-ORDER-3: release the batch record.", + ), + }, +) + + +STANDALONE_TEXT_RELATIONS = ( + { + "id": "docx-document-title", + "doc_id": "docx_reference", + "source_locator": "DOCX body > Title paragraph 1", + "target_scope": "preamble", + "target_block": _paragraph(1), + "text": "Compact Reactor Reference Guide", + }, + { + "id": "docx-document-subtitle", + "doc_id": "docx_reference", + "source_locator": "DOCX body > Subtitle paragraph 2", + "target_scope": "preamble", + "target_block": _paragraph(2), + "text": "Controlled limits and release checks", + }, + { + "id": "html-validation-caption", + "doc_id": "html_notice", + "source_locator": "HTML css=#register table > caption", + "target_section": _section("Validation register", 2), + "target_block": _paragraph(1), + "text": "Transfer limits", + }, +) + + +LIST_RELATIONS = ( + { + "id": "docx-operator-checklist", + "doc_id": "docx_reference", + "source_locator": "DOCX Operating envelope/Operator checklist > list items 1-3", + "target_section": _section("Operator checklist", 2), + "items": ( + {"kind": "ordered", "text": "Confirm that valve C-17 is locked."}, + { + "kind": "ordered", + "text": "Record the inlet temperature before adjustment.", + }, + {"kind": "unordered", "text": "Do not bypass the purge interlock."}, + ), + }, + { + "id": "html-required-actions", + "doc_id": "html_notice", + "source_locator": "HTML css=#actions > ol > li:nth-child(1..3)", + "target_section": _section("Required actions", 2), + "items": ( + { + "kind": "ordered", + "text": "HTML-ORDER-1: seal the transfer coupling.", + }, + { + "kind": "ordered", + "text": "HTML-ORDER-2: capture the reference sample.", + }, + { + "kind": "ordered", + "text": "HTML-ORDER-3: release the batch record.", + }, + ), + }, +) + + +FOOTNOTE_RELATIONS = ( + { + "id": "pdf-maintenance-footnote-relation", + "doc_id": "pdf_protocol", + "source_locator": "PDF page 1 paragraph marker 1 -> page 1 footnote 1", + "marker_section": _section("1. Measurement controls", 2), + "marker_block": _paragraph(2), + "marker_pattern": r"offset is active\.\s*1", + "note_section": _section("2. Acceptance register", 2), + "note_block": _paragraph(1), + "note_pattern": r"1\s+The maintenance exception applies only when service ticket MX-204 is open", + }, + { + "id": "docx-batch-footnote-relation", + "doc_id": "docx_reference", + "source_locator": "DOCX Scope notes marker 1 -> following note 1", + "marker_section": _section("Scope notes", 2), + "marker_block": _paragraph(1), + "marker_pattern": r"Cobalt-X row carries qualifier\s+1\.", + "note_section": _section("Scope notes", 2), + "note_block": _paragraph(2), + "note_pattern": r"1\s+Applies only to reactor batch RX-31;? it does not apply to RX-32", + }, + { + "id": "html-expiry-footnote-relation", + "doc_id": "html_notice", + "source_locator": "HTML #limits marker link -> #note-1", + "marker_section": _section("Measurement limits", 2), + "marker_block": _paragraph(2), + "marker_pattern": r"not valid after 2032-09-30\.\s*\[1\]\(#note-1\)", + "required_link_pattern": r"\[1\]\(#note-1\)", + "note_section": _section("Notes", 2), + "note_block": _paragraph(1), + "note_pattern": r"1\s+The expiry restriction applies even when a supervisor approves an override", + }, +) + + +def gold_payload() -> dict[str, Any]: + """Return the immutable public Gold contract in JSON-safe form.""" + return { + "schema_version": GOLD_SCHEMA_VERSION, + "documents": EXPECTED_DOCUMENTS, + "heading_inventories": HEADING_INVENTORIES, + "critical_facts": list(CRITICAL_FACTS), + "qualifiers": list(QUALIFIERS), + "table_relations": list(TABLE_RELATIONS), + "reading_orders": list(READING_ORDERS), + "standalone_text_relations": list(STANDALONE_TEXT_RELATIONS), + "list_relations": list(LIST_RELATIONS), + "footnote_relations": list(FOOTNOTE_RELATIONS), + } + + +def gold_digest() -> str: + payload = json.dumps( + gold_payload(), ensure_ascii=False, sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def _section_key(section: dict[str, Any]) -> tuple[str, int]: + return section.get("title"), section.get("level") + + +def validate_gold() -> None: + """Fail closed on empty denominators, coordinates, IDs, or target drift.""" + if set(HEADING_INVENTORIES) != set(EXPECTED_DOCUMENTS): + raise ValueError("heading inventories must cover every fixture document") + section_keys: dict[str, set[tuple[str, int]]] = {} + for doc_id, inventory in HEADING_INVENTORIES.items(): + if not inventory: + raise ValueError(f"empty heading inventory: {doc_id}") + keys = [_section_key(row) for row in inventory] + if any(not title or not isinstance(level, int) or not 1 <= level <= 6 + for title, level in keys): + raise ValueError(f"invalid heading inventory: {doc_id}") + if len(keys) != len(set(keys)): + raise ValueError(f"duplicate heading coordinate in inventory: {doc_id}") + section_keys[doc_id] = set(keys) + + groups = ( + CRITICAL_FACTS, + QUALIFIERS, + TABLE_RELATIONS, + READING_ORDERS, + STANDALONE_TEXT_RELATIONS, + LIST_RELATIONS, + FOOTNOTE_RELATIONS, + ) + ids: list[str] = [] + for group in groups: + if not group: + raise ValueError("Gold metric denominator must be non-zero") + for row in group: + doc_id = row.get("doc_id") + if doc_id not in EXPECTED_DOCUMENTS: + raise ValueError(f"Gold row has dangling document: {row!r}") + if not isinstance(row.get("source_locator"), str) or not row["source_locator"].strip(): + raise ValueError(f"Gold row lacks an authored-source locator: {row!r}") + if not isinstance(row.get("id"), str) or not row["id"]: + raise ValueError(f"Gold row lacks an ID: {row!r}") + if group is FOOTNOTE_RELATIONS: + for section_name, block_name in ( + ("marker_section", "marker_block"), + ("note_section", "note_block"), + ): + section = row.get(section_name) + block = row.get(block_name) + if (not isinstance(section, dict) + or _section_key(section) not in section_keys[doc_id]): + raise ValueError( + f"Gold footnote has non-executable {section_name}: {row!r}" + ) + if (not isinstance(block, dict) + or block.get("kind") != "paragraph" + or not isinstance(block.get("ordinal"), int) + or block["ordinal"] <= 0): + raise ValueError( + f"Gold footnote has non-executable {block_name}: {row!r}" + ) + if not row.get("marker_pattern") or not row.get("note_pattern"): + raise ValueError(f"Gold footnote lacks marker/note patterns: {row!r}") + ids.append(row["id"]) + continue + + target_section = row.get("target_section") + if group is STANDALONE_TEXT_RELATIONS and row.get("target_scope") == "preamble": + if target_section is not None: + raise ValueError(f"Gold preamble row cannot declare a section: {row!r}") + elif (not isinstance(target_section, dict) + or _section_key(target_section) not in section_keys[doc_id]): + raise ValueError(f"Gold row has non-executable target section: {row!r}") + if group is STANDALONE_TEXT_RELATIONS: + block = row.get("target_block") + if (not isinstance(block, dict) + or block.get("kind") != "paragraph" + or not isinstance(block.get("ordinal"), int) + or block["ordinal"] <= 0 + or not isinstance(row.get("text"), str) + or not row["text"].strip()): + raise ValueError(f"Gold standalone text row is invalid: {row!r}") + elif group is LIST_RELATIONS: + items = row.get("items") + if (not isinstance(items, (tuple, list)) or not items + or any( + not isinstance(item, dict) + or item.get("kind") not in {"ordered", "unordered"} + or not isinstance(item.get("text"), str) + or not item["text"].strip() + for item in items + )): + raise ValueError(f"Gold list has invalid items: {row!r}") + elif group is not READING_ORDERS: + block = row.get("target_block") + if (not isinstance(block, dict) + or block.get("kind") not in {"paragraph", "table"} + or not isinstance(block.get("ordinal"), int) + or block["ordinal"] <= 0): + raise ValueError(f"Gold row has non-executable target block: {row!r}") + ids.append(row["id"]) + if len(ids) != len(set(ids)): + raise ValueError("Gold IDs must be globally unique") + if set(EXPECTED_DOCUMENTS) != {row["doc_id"] for group in groups for row in group}: + raise ValueError("every fixture document must contribute Gold rows") + if gold_digest() != FROZEN_GOLD_SHA256: + raise ValueError( + "Gold payload changed without an explicit review and frozen digest update" + ) diff --git a/evals/conversion_fidelity/run_eval.py b/evals/conversion_fidelity/run_eval.py new file mode 100644 index 0000000..665d5b9 --- /dev/null +++ b/evals/conversion_fidelity/run_eval.py @@ -0,0 +1,1112 @@ +#!/usr/bin/env python3 +"""End-to-end PDF/DOCX/HTML conversion-fidelity protocol smoke test. + +Exit codes: + 0: every pre-registered threshold and mutation control passed; + 1: the smoke run completed but at least one threshold failed; + 2: dependency, fixture, Gold, conversion, denominator, or protocol error. +""" +from __future__ import annotations + +import argparse +import copy +import hashlib +import importlib.metadata +import importlib.util +import json +import os +import re +import subprocess +import sys +import tempfile +import unicodedata +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +SCRIPTS_DIR = ENGINE_ROOT / "scripts" +for candidate in (ENGINE_ROOT, EVAL_DIR, SCRIPTS_DIR): + if str(candidate) not in sys.path: + sys.path.insert(0, str(candidate)) + +from evals.conversion_fidelity import fixtures, gold # noqa: E402 +from postprocess import validate_outline # noqa: E402 + + +RESULT_PREFIX = "CONVERSION_FIDELITY_RESULT " +THRESHOLDS = { + "critical_fact_preservation": 1.0, + "qualifier_preservation": 1.0, + "table_alignment": 1.0, + "reading_order": 1.0, + "standalone_text_preservation": 1.0, + "list_structure_preservation": 1.0, + "section_count_preservation": 1.0, + "heading_hierarchy_preservation": 1.0, + "footnote_relation_preservation": 1.0, + "mutation_sensitivity": 1.0, +} +NOT_MEASURED = ( + "open-world conversion accuracy outside the three synthetic fixtures", + "OCR accuracy for scanned or image-only documents", + "visual fidelity beyond the separately asserted fixture geometry", + "semantic correctness of downstream answers or citations", +) + + +class EvaluationProtocolError(RuntimeError): + """The evaluation cannot produce a trustworthy score.""" + + +@dataclass(frozen=True) +class Heading: + title: str + level: int + line_index: int + end_line_index: int + + +@dataclass(frozen=True) +class Block: + kind: str + text: str + start_line_index: int + end_line_index: int + + +@dataclass(frozen=True) +class MarkdownDocument: + raw: str + lines: tuple[str, ...] + headings: tuple[Heading, ...] + + +_HEADING_RE = re.compile(r"^ {0,3}(#{1,6})[ \t]+(.+?)[ \t]*$") +_ANCHOR_RE = re.compile( + r"\s+\^(?:h-\d+-\d+|[ptcf]-\d+)-[0-9a-f]{6}\b", re.IGNORECASE +) +_TABLE_SEPARATOR_CELL_RE = re.compile(r"^:?-{3,}:?$") +_LIST_ITEM_RE = re.compile( + r"^\s*(?P\d+[.)]|[-+*])\s+(?P.+?)\s*$" +) + + +def _metric(numerator: int, denominator: int) -> dict[str, Any]: + if isinstance(numerator, bool) or isinstance(denominator, bool): + raise EvaluationProtocolError("metric numerator/denominator cannot be bool") + if not isinstance(numerator, int) or not isinstance(denominator, int): + raise EvaluationProtocolError("metric numerator/denominator must be integers") + if denominator <= 0: + raise EvaluationProtocolError("metric denominator must be greater than zero") + if numerator < 0 or numerator > denominator: + raise EvaluationProtocolError(f"invalid metric counts: {numerator}/{denominator}") + return {"numerator": numerator, "denominator": denominator, + "value": numerator / denominator} + + +def _strip_anchor(text: str) -> str: + return _ANCHOR_RE.sub("", text).strip() + + +def _normalize(text: str) -> str: + text = unicodedata.normalize("NFC", text).replace("\u00a0", " ") + text = _ANCHOR_RE.sub("", text) + return re.sub(r"\s+", " ", text).strip() + + +def _parse_document(raw: str) -> MarkdownDocument: + """Parse ATX headings outside fenced code and derive hierarchical spans.""" + lines = tuple(raw.splitlines()) + heading_rows: list[tuple[str, int, int]] = [] + fence: str | None = None + for index, line in enumerate(lines): + fence_match = re.match(r"^\s*(`{3,}|~{3,})", line) + if fence_match: + marker = fence_match.group(1) + if fence is None: + fence = marker[0] + elif marker[0] == fence: + fence = None + continue + if fence is not None: + continue + match = _HEADING_RE.match(line) + if not match: + continue + title = _strip_anchor(match.group(2)) + title = re.sub(r"\s+#+\s*$", "", title).strip() + heading_rows.append((title, len(match.group(1)), index)) + + headings: list[Heading] = [] + for offset, (title, level, line_index) in enumerate(heading_rows): + end = len(lines) + for _, candidate_level, candidate_line in heading_rows[offset + 1:]: + if candidate_level <= level: + end = candidate_line + break + headings.append(Heading(title, level, line_index, end)) + return MarkdownDocument(raw=raw, lines=lines, headings=tuple(headings)) + + +def _resolve_section( + document: MarkdownDocument, coordinate: dict[str, Any] +) -> tuple[Heading | None, int]: + matches = [ + heading for heading in document.headings + if heading.title == coordinate["title"] and heading.level == coordinate["level"] + ] + return (matches[0] if len(matches) == 1 else None), len(matches) + + +def _section_text(document: MarkdownDocument, heading: Heading) -> str: + # Include descendant heading lines so a parent-section order contract can + # verify page/section transitions, but exclude the target heading itself. + return "\n".join(document.lines[heading.line_index + 1:heading.end_line_index]) + + +def _looks_like_pipe_table(lines: list[str]) -> bool: + if len(lines) < 2 or not all("|" in line for line in lines): + return False + separator = _split_pipe_row(lines[1]) + return bool(separator) and all(_TABLE_SEPARATOR_CELL_RE.fullmatch(cell) for cell in separator) + + +def _parse_blocks( + lines: tuple[str, ...] | list[str], *, base_line_index: int = 0 +) -> tuple[Block, ...]: + """Return blank-line-delimited canonical blocks, excluding headings.""" + blocks: list[Block] = [] + group: list[str] = [] + group_start = 0 + + def flush(end_index: int) -> None: + nonlocal group + if not group: + return + if len(group) == 1 and _HEADING_RE.match(group[0]): + group = [] + return + kind = "table" if _looks_like_pipe_table(group) else "paragraph" + blocks.append(Block( + kind=kind, + text="\n".join(group), + start_line_index=base_line_index + group_start, + end_line_index=base_line_index + end_index, + )) + group = [] + + for offset, line in enumerate(lines): + if not line.strip(): + flush(offset) + continue + if not group: + group_start = offset + # Converted fixtures separate headings with blanks. Still flush here + # so an adversarial extra heading cannot be absorbed into prose. + if _HEADING_RE.match(line): + flush(offset) + group_start = offset + group = [line] + flush(offset + 1) + continue + group.append(line) + flush(len(lines)) + return tuple(blocks) + + +def _target_block( + document: MarkdownDocument, heading: Heading, coordinate: dict[str, Any] +) -> Block | None: + blocks = _parse_blocks( + document.lines[heading.line_index + 1:heading.end_line_index], + base_line_index=heading.line_index + 1, + ) + matching = [block for block in blocks if block.kind == coordinate["kind"]] + ordinal = coordinate["ordinal"] + return matching[ordinal - 1] if len(matching) >= ordinal else None + + +def _pattern_count(pattern: str, text: str, row_id: str) -> int: + try: + return len(re.findall(pattern, _normalize(text), flags=re.IGNORECASE)) + except re.error as exc: + raise EvaluationProtocolError(f"invalid Gold regex {row_id}: {exc}") from exc + + +def _scoped_regex_result( + document: MarkdownDocument, row: dict[str, Any] +) -> dict[str, Any]: + heading, section_match_count = _resolve_section(document, row["target_section"]) + block = ( + _target_block(document, heading, row["target_block"]) + if heading is not None else None + ) + block_matches = _pattern_count(row["pattern"], block.text, row["id"]) if block else 0 + document_matches = _pattern_count(row["pattern"], document.raw, row["id"]) + passed = ( + section_match_count == 1 + and block is not None + and block_matches == 1 + and document_matches == 1 + ) + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "target_section": row["target_section"], + "target_block": row["target_block"], + "section_match_count": section_match_count, + "block_resolved": block is not None, + "target_block_matches": block_matches, + "document_matches": document_matches, + "passed": passed, + } + + +def _split_pipe_row(line: str) -> tuple[str, ...]: + """Split a Markdown pipe row while respecting backslash-escaped pipes.""" + line = _strip_anchor(line.strip()) + if line.startswith("|"): + line = line[1:] + if line.endswith("|"): + line = line[:-1] + cells: list[str] = [] + buffer: list[str] = [] + escaped = False + for char in line: + if escaped: + buffer.append(char) + escaped = False + elif char == "\\": + escaped = True + buffer.append(char) + elif char == "|": + cells.append(_canonical_cell("".join(buffer))) + buffer = [] + else: + buffer.append(char) + if escaped: + buffer.append("\\") + cells.append(_canonical_cell("".join(buffer))) + return tuple(cells) + + +def _canonical_cell(cell: str) -> str: + cell = unicodedata.normalize("NFC", cell).replace("\\|", "|").strip() + previous = None + while previous != cell: + previous = cell + for marker in ("**", "__", "`"): + if cell.startswith(marker) and cell.endswith(marker) and len(cell) >= 2 * len(marker): + cell = cell[len(marker):-len(marker)].strip() + return re.sub(r"\s+", " ", cell) + + +def _pipe_table_rows(block: Block) -> tuple[tuple[str, ...], ...] | None: + lines = [line for line in block.text.splitlines() if line.strip()] + if not _looks_like_pipe_table(lines): + return None + parsed = [_split_pipe_row(line) for line in lines] + widths = {len(row) for row in parsed} + if len(widths) != 1: + return None + # Line 2 is the Markdown delimiter. The first row must itself be the exact + # semantic header; a blank presentation header followed by header-like data + # is not equivalent and must fail the relation gate. + return (parsed[0], *parsed[2:]) + + +def _table_result(document: MarkdownDocument, row: dict[str, Any]) -> dict[str, Any]: + heading, section_match_count = _resolve_section(document, row["target_section"]) + block = ( + _target_block(document, heading, row["target_block"]) + if heading is not None else None + ) + expected = (tuple(row["headers"]), *(tuple(cells) for cells in row["rows"])) + observed = _pipe_table_rows(block) if block is not None else None + + document_tables = [ + table + for candidate in _parse_blocks(document.lines) + if candidate.kind == "table" + for table in [_pipe_table_rows(candidate)] + if table is not None + ] + exact_document_matches = sum(1 for table in document_tables if table == expected) + passed = ( + section_match_count == 1 + and block is not None + and observed == expected + and exact_document_matches == 1 + ) + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "target_section": row["target_section"], + "target_block": row["target_block"], + "section_match_count": section_match_count, + "block_resolved": block is not None, + "expected_rows": expected, + "observed_rows": observed, + "exact_document_matches": exact_document_matches, + "passed": passed, + } + + +def _reading_order_result( + document: MarkdownDocument, row: dict[str, Any] +) -> dict[str, Any]: + heading, section_match_count = _resolve_section(document, row["target_section"]) + section_text = _section_text(document, heading) if heading is not None else "" + section_folded = _normalize(section_text).casefold() + document_folded = _normalize(document.raw).casefold() + tokens = tuple(row["tokens"]) + section_counts = [section_folded.count(token.casefold()) for token in tokens] + document_counts = [document_folded.count(token.casefold()) for token in tokens] + positions = [section_folded.find(token.casefold()) for token in tokens] + unique = all(count == 1 for count in section_counts) and all( + count == 1 for count in document_counts + ) + ordered = all(left < right for left, right in zip(positions, positions[1:])) + passed = section_match_count == 1 and unique and ordered + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "target_section": row["target_section"], + "section_match_count": section_match_count, + "section_token_counts": section_counts, + "document_token_counts": document_counts, + "positions": positions, + "passed": passed, + } + + +def _standalone_text_result( + document: MarkdownDocument, row: dict[str, Any] +) -> dict[str, Any]: + if row.get("target_scope") == "preamble": + first_heading_line = ( + document.headings[0].line_index if document.headings else len(document.lines) + ) + blocks = _parse_blocks(document.lines[:first_heading_line]) + scope_match_count = 1 + else: + heading, scope_match_count = _resolve_section(document, row["target_section"]) + blocks = ( + _parse_blocks( + document.lines[heading.line_index + 1:heading.end_line_index], + base_line_index=heading.line_index + 1, + ) + if heading is not None else () + ) + paragraphs = [block for block in blocks if block.kind == "paragraph"] + ordinal = row["target_block"]["ordinal"] + block = paragraphs[ordinal - 1] if len(paragraphs) >= ordinal else None + expected = _normalize(row["text"]) + observed = _normalize(block.text) if block is not None else None + document_matches = _normalize(document.raw).count(expected) + passed = ( + scope_match_count == 1 + and block is not None + and observed == expected + and document_matches == 1 + ) + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "target_scope": row.get("target_scope"), + "target_section": row.get("target_section"), + "target_block": row["target_block"], + "scope_match_count": scope_match_count, + "block_resolved": block is not None, + "expected_text": expected, + "observed_text": observed, + "document_matches": document_matches, + "passed": passed, + } + + +def _list_items(text: str) -> tuple[dict[str, str], ...]: + """Extract top-level Markdown list semantics without treating footnotes as lists.""" + items: list[dict[str, str]] = [] + for line in text.splitlines(): + match = _LIST_ITEM_RE.match(line) + if not match: + continue + marker = match.group("marker") + item_text = _normalize(_strip_anchor(match.group("text"))) + items.append({ + "kind": "ordered" if marker[0].isdigit() else "unordered", + "text": item_text, + }) + return tuple(items) + + +def _list_relation_result( + document: MarkdownDocument, row: dict[str, Any] +) -> dict[str, Any]: + heading, section_match_count = _resolve_section(document, row["target_section"]) + section_text = _section_text(document, heading) if heading is not None else "" + observed = _list_items(section_text) + expected = tuple( + {"kind": item["kind"], "text": _normalize(item["text"])} + for item in row["items"] + ) + normalized_document = _normalize(document.raw) + document_item_counts = [ + normalized_document.count(item["text"]) for item in expected + ] + passed = ( + section_match_count == 1 + and observed == expected + and all(count == 1 for count in document_item_counts) + ) + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "target_section": row["target_section"], + "section_match_count": section_match_count, + "expected_items": expected, + "observed_items": observed, + "document_item_counts": document_item_counts, + "passed": passed, + } + + +def _footnote_relation_result( + document: MarkdownDocument, row: dict[str, Any] +) -> dict[str, Any]: + marker_heading, marker_section_matches = _resolve_section( + document, row["marker_section"] + ) + note_heading, note_section_matches = _resolve_section( + document, row["note_section"] + ) + marker_block = ( + _target_block(document, marker_heading, row["marker_block"]) + if marker_heading is not None else None + ) + note_block = ( + _target_block(document, note_heading, row["note_block"]) + if note_heading is not None else None + ) + marker_block_matches = ( + _pattern_count(row["marker_pattern"], marker_block.text, row["id"]) + if marker_block is not None else 0 + ) + marker_document_matches = _pattern_count( + row["marker_pattern"], document.raw, row["id"] + ) + note_block_matches = ( + _pattern_count(row["note_pattern"], note_block.text, row["id"]) + if note_block is not None else 0 + ) + note_document_matches = _pattern_count( + row["note_pattern"], document.raw, row["id"] + ) + marker_before_note = ( + marker_block is not None + and note_block is not None + and marker_block.end_line_index <= note_block.start_line_index + ) + + link_pattern = row.get("required_link_pattern") + link_block_matches = ( + _pattern_count(link_pattern, marker_block.text, row["id"]) + if link_pattern and marker_block is not None else None + ) + link_document_matches = ( + _pattern_count(link_pattern, document.raw, row["id"]) + if link_pattern else None + ) + link_passed = ( + link_pattern is None + or (link_block_matches == 1 and link_document_matches == 1) + ) + passed = ( + marker_section_matches == 1 + and note_section_matches == 1 + and marker_block is not None + and note_block is not None + and marker_block_matches == 1 + and marker_document_matches == 1 + and note_block_matches == 1 + and note_document_matches == 1 + and marker_before_note + and link_passed + ) + return { + "id": row["id"], + "doc_id": row["doc_id"], + "source_locator": row["source_locator"], + "marker_section": row["marker_section"], + "marker_block": row["marker_block"], + "note_section": row["note_section"], + "note_block": row["note_block"], + "marker_section_match_count": marker_section_matches, + "note_section_match_count": note_section_matches, + "marker_block_matches": marker_block_matches, + "marker_document_matches": marker_document_matches, + "note_block_matches": note_block_matches, + "note_document_matches": note_document_matches, + "marker_before_note": marker_before_note, + "required_link_pattern": link_pattern, + "link_block_matches": link_block_matches, + "link_document_matches": link_document_matches, + "passed": passed, + } + + +def _heading_results( + documents: dict[str, MarkdownDocument] +) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + count_rows: list[dict[str, Any]] = [] + hierarchy_rows: list[dict[str, Any]] = [] + for doc_id, expected_rows in gold.HEADING_INVENTORIES.items(): + observed = tuple( + {"title": heading.title, "level": heading.level} + for heading in documents[doc_id].headings + ) + expected = tuple(dict(row) for row in expected_rows) + expected_titles = tuple(row["title"] for row in expected) + observed_titles = tuple(row["title"] for row in observed) + count_rows.append({ + "id": f"{doc_id}-exact-heading-inventory", + "doc_id": doc_id, + "expected_titles": expected_titles, + "observed_titles": observed_titles, + "passed": observed_titles == expected_titles, + }) + hierarchy_rows.append({ + "id": f"{doc_id}-exact-heading-levels", + "doc_id": doc_id, + "expected_inventory": expected, + "observed_inventory": observed, + "passed": observed == expected, + }) + return count_rows, hierarchy_rows + + +def evaluate_outputs(outputs: dict[str, str], *, include_details: bool = True) -> dict[str, Any]: + """Score converted Markdown against the section-scoped Gold v4 contract.""" + gold.validate_gold() + if set(outputs) != set(gold.EXPECTED_DOCUMENTS): + raise EvaluationProtocolError( + f"output document IDs differ from Gold: {sorted(outputs)}" + ) + for doc_id, text in outputs.items(): + if not isinstance(text, str) or not text.strip(): + raise EvaluationProtocolError(f"empty/non-text converted output: {doc_id}") + + documents = {doc_id: _parse_document(text) for doc_id, text in outputs.items()} + details: dict[str, list[dict[str, Any]]] = {} + details["critical_fact_preservation"] = [ + _scoped_regex_result(documents[row["doc_id"]], row) for row in gold.CRITICAL_FACTS + ] + details["qualifier_preservation"] = [ + _scoped_regex_result(documents[row["doc_id"]], row) for row in gold.QUALIFIERS + ] + details["table_alignment"] = [ + _table_result(documents[row["doc_id"]], row) for row in gold.TABLE_RELATIONS + ] + details["reading_order"] = [ + _reading_order_result(documents[row["doc_id"]], row) + for row in gold.READING_ORDERS + ] + details["standalone_text_preservation"] = [ + _standalone_text_result(documents[row["doc_id"]], row) + for row in gold.STANDALONE_TEXT_RELATIONS + ] + details["list_structure_preservation"] = [ + _list_relation_result(documents[row["doc_id"]], row) + for row in gold.LIST_RELATIONS + ] + details["footnote_relation_preservation"] = [ + _footnote_relation_result(documents[row["doc_id"]], row) + for row in gold.FOOTNOTE_RELATIONS + ] + section_rows, hierarchy_rows = _heading_results(documents) + details["section_count_preservation"] = section_rows + details["heading_hierarchy_preservation"] = hierarchy_rows + + metrics: dict[str, dict[str, Any]] = {} + for name, rows in details.items(): + numerator = sum(1 for row in rows if row["passed"]) + metric = _metric(numerator, len(rows)) + metric["threshold"] = THRESHOLDS[name] + metric["passed"] = metric["value"] >= THRESHOLDS[name] + metrics[name] = metric + result: dict[str, Any] = {"metrics": metrics} + if include_details: + result["details"] = details + return result + + +def _mutate_once(outputs: dict[str, str], doc_id: str, needle: str, replacement: str) -> dict[str, str]: + if outputs[doc_id].count(needle) != 1: + raise EvaluationProtocolError( + f"mutation control needs exactly one needle: {doc_id}: {needle!r}" + ) + mutated = copy.deepcopy(outputs) + mutated[doc_id] = mutated[doc_id].replace(needle, replacement, 1) + return mutated + + +def _swap_once(outputs: dict[str, str], doc_id: str, first: str, second: str) -> dict[str, str]: + if outputs[doc_id].count(first) != 1 or outputs[doc_id].count(second) != 1: + raise EvaluationProtocolError( + f"mutation control swap needs unique tokens: {doc_id}" + ) + placeholder = "__GROUND_MAP_MUTATION_SWAP_SENTINEL__" + if placeholder in outputs[doc_id]: + raise EvaluationProtocolError("mutation swap sentinel unexpectedly present") + mutated = copy.deepcopy(outputs) + text = mutated[doc_id].replace(first, placeholder, 1) + text = text.replace(second, first, 1).replace(placeholder, second, 1) + mutated[doc_id] = text + return mutated + + +def _move_line_to_section_once( + outputs: dict[str, str], doc_id: str, needle: str, destination_title: str +) -> dict[str, str]: + lines = outputs[doc_id].splitlines() + source_indices = [index for index, line in enumerate(lines) if needle in line] + heading_indices = [ + index for index, line in enumerate(lines) + if (match := _HEADING_RE.match(line)) + and _strip_anchor(match.group(2)) == destination_title + ] + if len(source_indices) != 1 or len(heading_indices) != 1: + raise EvaluationProtocolError("move mutation requires unique source and destination") + moved_line = lines.pop(source_indices[0]) + # Resolve again because deleting an earlier line changes the destination index. + destination_index = next( + index for index, line in enumerate(lines) + if (match := _HEADING_RE.match(line)) + and _strip_anchor(match.group(2)) == destination_title + ) + lines.insert(destination_index + 1, moved_line) + mutated = copy.deepcopy(outputs) + mutated[doc_id] = "\n".join(lines) + ("\n" if outputs[doc_id].endswith("\n") else "") + return mutated + + +def _flatten_pipe_table_once( + outputs: dict[str, str], doc_id: str, header_token: str +) -> dict[str, str]: + lines = outputs[doc_id].splitlines() + candidates: list[tuple[int, int]] = [] + index = 0 + while index < len(lines): + if "|" not in lines[index]: + index += 1 + continue + end = index + while end < len(lines) and lines[end].strip() and "|" in lines[end]: + end += 1 + if any(header_token in line for line in lines[index:end]): + candidates.append((index, end)) + index = max(end, index + 1) + if len(candidates) != 1: + raise EvaluationProtocolError("flatten mutation requires one real pipe table") + start, end = candidates[0] + tokens = [ + cell for line in lines[start:end] for cell in _split_pipe_row(line) + if cell and not _TABLE_SEPARATOR_CELL_RE.fullmatch(cell) + ] + lines[start:end] = ["Flattened table tokens: " + "; ".join(tokens)] + mutated = copy.deepcopy(outputs) + mutated[doc_id] = "\n".join(lines) + ("\n" if outputs[doc_id].endswith("\n") else "") + return mutated + + +def _insert_extra_heading_once( + outputs: dict[str, str], doc_id: str, heading: str +) -> dict[str, str]: + if heading in outputs[doc_id]: + raise EvaluationProtocolError("extra-heading mutation must start absent") + mutated = copy.deepcopy(outputs) + mutated[doc_id] = outputs[doc_id].rstrip() + f"\n\n## {heading}\n\nSynthetic extra section.\n" + return mutated + + +def _delete_line_once( + outputs: dict[str, str], doc_id: str, needle: str +) -> dict[str, str]: + lines = outputs[doc_id].splitlines() + matches = [index for index, line in enumerate(lines) if needle in line] + if len(matches) != 1: + raise EvaluationProtocolError( + f"line-delete mutation needs exactly one matching line: {doc_id}: {needle!r}" + ) + del lines[matches[0]] + mutated = copy.deepcopy(outputs) + mutated[doc_id] = "\n".join(lines) + ("\n" if outputs[doc_id].endswith("\n") else "") + return mutated + + +def _detail_passed(scored: dict[str, Any], metric: str, row_id: str) -> bool: + matches = [ + row for row in scored.get("details", {}).get(metric, []) + if row.get("id") == row_id + ] + if len(matches) != 1: + raise EvaluationProtocolError( + f"mutation control expected one detail row: {metric}/{row_id}" + ) + return bool(matches[0].get("passed")) + + +def run_mutation_controls(outputs: dict[str, str]) -> dict[str, Any]: + """Prove the scorer rejects fourteen independent corruption classes.""" + controls: list[tuple[str, str, str, dict[str, str]]] = [] + + def add( + control_id: str, mutated: dict[str, str], metric: str, row_id: str + ) -> None: + controls.append((control_id, metric, row_id, mutated)) + + add( + "delete-negative-sign", + _mutate_once(outputs, "docx_reference", "-0.45 kPa", "0.45 kPa"), + "critical_fact_preservation", "docx-negative-pressure", + ) + add( + "delete-unit", + _mutate_once(outputs, "docx_reference", "±0.14 mm", "±0.14"), + "critical_fact_preservation", "docx-plus-minus-tolerance", + ) + add( + "delete-negation", + _mutate_once(outputs, "html_notice", "not valid after", "valid after"), + "qualifier_preservation", "html-expiry-negation", + ) + add( + "delete-table-header", + _mutate_once(outputs, "html_notice", "Concentration", "Concentration lost"), + "table_alignment", "html-validation-register", + ) + add( + "reorder-reading-sequence", + _swap_once(outputs, "html_notice", "HTML-ORDER-1", "HTML-ORDER-2"), + "reading_order", "html-dom-order", + ) + add( + "move-fact-to-wrong-section", + _move_line_to_section_once( + outputs, "docx_reference", "inlet pressure correction", "Scope notes" + ), + "critical_fact_preservation", "docx-negative-pressure", + ) + add( + "flatten-table-to-prose", + _flatten_pipe_table_once(outputs, "html_notice", "Concentration"), + "table_alignment", "html-validation-register", + ) + add( + "cross-table-row-values", + _swap_once(outputs, "pdf_protocol", "2.45 bar", "3.10 bar"), + "table_alignment", "pdf-acceptance-register", + ) + add( + "insert-extra-heading", + _insert_extra_heading_once(outputs, "html_notice", "Unexpected appendix"), + "section_count_preservation", "html_notice-exact-heading-inventory", + ) + add( + "delete-footnote-marker", + _mutate_once(outputs, "pdf_protocol", "offset is active. 1", "offset is active."), + "footnote_relation_preservation", "pdf-maintenance-footnote-relation", + ) + add( + "corrupt-footnote-link", + _mutate_once(outputs, "html_notice", "(#note-1)", "(#note-9)"), + "footnote_relation_preservation", "html-expiry-footnote-relation", + ) + add( + "move-footnote-to-wrong-block", + _move_line_to_section_once( + outputs, + "docx_reference", + "Applies only to reactor batch RX-31", + "Release order", + ), + "footnote_relation_preservation", "docx-batch-footnote-relation", + ) + add( + "delete-list-item", + _delete_line_once( + outputs, "docx_reference", "Do not bypass the purge interlock." + ), + "list_structure_preservation", "docx-operator-checklist", + ) + add( + "delete-document-subtitle", + _delete_line_once( + outputs, "docx_reference", "Controlled limits and release checks" + ), + "standalone_text_preservation", "docx-document-subtitle", + ) + + rows: list[dict[str, Any]] = [] + baseline = evaluate_outputs(outputs, include_details=True) + for control_id, expected_metric, expected_row, mutated in controls: + scored = evaluate_outputs(mutated, include_details=True) + baseline_passed = _detail_passed(baseline, expected_metric, expected_row) + mutated_passed = _detail_passed(scored, expected_metric, expected_row) + rows.append({ + "id": control_id, + "expected_failed_metric": expected_metric, + "expected_failed_row": expected_row, + "baseline_row_passed": baseline_passed, + "mutated_row_passed": mutated_passed, + "passed": baseline_passed and not mutated_passed, + }) + metric = _metric(sum(1 for row in rows if row["passed"]), len(rows)) + metric["threshold"] = THRESHOLDS["mutation_sensitivity"] + metric["passed"] = metric["value"] >= THRESHOLDS["mutation_sensitivity"] + return {"metric": metric, "details": rows} + + +def _require_runtime_dependencies() -> dict[str, str]: + fixtures.require_dependencies() + missing_modules = [ + name for name in ("markitdown", "pdfplumber", "lxml") + if importlib.util.find_spec(name) is None + ] + if missing_modules: + raise EvaluationProtocolError( + "missing required conversion/evaluation dependencies: " + + ", ".join(missing_modules) + ) + versions: dict[str, str] = {} + for package in ("markitdown", "reportlab", "python-docx", "pdfplumber", "lxml"): + try: + versions[package] = importlib.metadata.version(package) + except importlib.metadata.PackageNotFoundError as exc: + raise EvaluationProtocolError( + f"required runtime package has no distribution metadata: {package}" + ) from exc + return versions + + +def _convert_real_pipeline(kb_root: Path, workspace_root: Path) -> dict[str, Any]: + raw_dir = workspace_root / "raw" + environment = os.environ.copy() + environment["KB_ROOT"] = str(kb_root) + command = [ + sys.executable, str(SCRIPTS_DIR / "convert.py"), + "--dir", str(raw_dir), "--force", "--ext", ".pdf,.docx,.html", + ] + completed = subprocess.run( + command, cwd=ENGINE_ROOT, env=environment, text=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=False, + ) + if completed.returncode != 0: + raise EvaluationProtocolError( + "real convert pipeline failed " + f"(exit {completed.returncode}): {completed.stderr or completed.stdout}" + ) + return { + "command": command, + "returncode": completed.returncode, + "stdout": completed.stdout, + "stderr": completed.stderr, + } + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _load_and_validate_outputs( + source_paths: dict[str, Path], workspace_root: Path +) -> tuple[dict[str, str], list[dict[str, Any]]]: + outputs: dict[str, str] = {} + receipts: list[dict[str, Any]] = [] + semantic_payload = fixtures.semantic_fixture_payload(source_paths) + semantic_by_doc = fixtures.semantic_document_digests(semantic_payload) + for doc_id, definition in gold.EXPECTED_DOCUMENTS.items(): + source = source_paths.get(doc_id) + if source is None or source.name != definition["source_name"] or not source.is_file(): + raise EvaluationProtocolError(f"fixture source missing or renamed: {doc_id}") + if source.stat().st_size <= 0: + raise EvaluationProtocolError(f"fixture source is empty: {source}") + markdown_path = source.with_suffix(".md") + outline_path = source.with_suffix(".outline.json") + if not markdown_path.is_file() or not outline_path.is_file(): + raise EvaluationProtocolError( + f"convert pipeline did not create both derivatives for {doc_id}" + ) + markdown = markdown_path.read_text(encoding="utf-8") + try: + outline = json.loads(outline_path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise EvaluationProtocolError(f"invalid outline for {doc_id}: {exc}") from exc + expected_doc_path = f"raw/{markdown_path.name}" + errors = validate_outline(outline, markdown, expected_doc_path=expected_doc_path) + if errors: + raise EvaluationProtocolError( + f"postprocess outline contract failed for {doc_id}: {errors}" + ) + outputs[doc_id] = markdown + receipts.append({ + "doc_id": doc_id, + "format": definition["format"], + "source_name": source.name, + "source_bytes": source.stat().st_size, + # Binary hashes are observed receipts, not frozen reproducibility + # claims: PDF/OOXML containers may carry variable metadata. + "source_sha256": _sha256_file(source), + "semantic_sha256": semantic_by_doc[doc_id], + "markdown_name": markdown_path.name, + "markdown_chars": len(markdown), + "outline_sections": _count_outline_sections(outline.get("sections", [])), + }) + extra_sources = { + path.name for path in (workspace_root / "raw").iterdir() + if path.suffix.lower() in {".pdf", ".docx", ".html"} + } - {row["source_name"] for row in gold.EXPECTED_DOCUMENTS.values()} + if extra_sources: + raise EvaluationProtocolError(f"unexpected fixture sources: {sorted(extra_sources)}") + return outputs, receipts + + +def _count_outline_sections(sections: list[dict[str, Any]]) -> int: + total = 0 + for section in sections: + total += 1 + _count_outline_sections(section.get("children", [])) + return total + + +def run_evaluation(work_root: Path, *, include_details: bool = False) -> dict[str, Any]: + versions = _require_runtime_dependencies() + gold.validate_gold() + kb_root = work_root / "kb-data" + workspace_root = kb_root / "workspaces" / "conversion-fidelity" + raw_dir = workspace_root / "raw" + raw_dir.mkdir(parents=True, exist_ok=False) + source_paths = fixtures.materialize(raw_dir) + fixture_semantic_sha256 = fixtures.semantic_fixture_digest(source_paths) + if fixture_semantic_sha256 != fixtures.FROZEN_SEMANTIC_FIXTURE_SHA256: + raise EvaluationProtocolError( + "source-native semantic fixture digest changed without explicit review" + ) + pipeline = _convert_real_pipeline(kb_root, workspace_root) + outputs, receipts = _load_and_validate_outputs(source_paths, workspace_root) + scored = evaluate_outputs(outputs, include_details=include_details) + mutation = run_mutation_controls(outputs) + scored["metrics"]["mutation_sensitivity"] = mutation["metric"] + if include_details: + scored.setdefault("details", {})["mutation_sensitivity"] = mutation["details"] + + all_passed = all(row["passed"] for row in scored["metrics"].values()) + result = { + "status": "protocol-smoke-passed" if all_passed else "protocol-smoke-failed", + "overall_passed": all_passed, + "gold_schema_version": gold.GOLD_SCHEMA_VERSION, + "gold_sha256": gold.gold_digest(), + "fixture_semantic_sha256": fixture_semantic_sha256, + "documents": len(receipts), + "runtime_versions": versions, + "receipts": receipts, + "metrics": scored["metrics"], + "pipeline": { + "returncode": pipeline["returncode"], + "stdout": pipeline["stdout"] if include_details else None, + "stderr": pipeline["stderr"] if include_details else None, + }, + "scope": ( + "protocol smoke over three synthetic source formats through the real " + "conversion and GroundMap postprocess pipeline" + ), + "not_measured": list(NOT_MEASURED), + } + if include_details: + result["details"] = scored["details"] + result["artifact_root"] = str(work_root) + return result + + +def _print_human(result: dict[str, Any]) -> None: + print("Conversion Fidelity Protocol Smoke") + print(f"Gold: schema={result['gold_schema_version']} sha256={result['gold_sha256']}") + print(f"Fixture semantic sha256: {result['fixture_semantic_sha256']}") + print(f"Documents: {result['documents']}") + for name, metric in result["metrics"].items(): + print( + f" {name}: {metric['numerator']}/{metric['denominator']} " + f"({metric['value']:.1%}), threshold={metric['threshold']:.1%}, " + f"{'PASS' if metric['passed'] else 'FAIL'}" + ) + print(f"Overall: {'PASS' if result['overall_passed'] else 'FAIL'}") + print("Not measured:") + for item in result["not_measured"]: + print(f" - {item}") + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--json", action="store_true", help="emit one JSON result line") + parser.add_argument("--details", action="store_true", help="include per-locator details") + parser.add_argument( + "--keep-temp", action="store_true", + help="keep generated source/Markdown artifacts under the printed /tmp path", + ) + args = parser.parse_args(argv) + + cleanup: tempfile.TemporaryDirectory[str] | None = None + if args.keep_temp: + work_root = Path(tempfile.mkdtemp(prefix="groundmap-conversion-fidelity-")) + else: + cleanup = tempfile.TemporaryDirectory(prefix="groundmap-conversion-fidelity-") + work_root = Path(cleanup.name) + try: + result = run_evaluation(work_root, include_details=args.details) + except ( + fixtures.FixtureDependencyError, + EvaluationProtocolError, + ValueError, + OSError, + RuntimeError, + SystemExit, + ) as exc: + result = { + "status": "protocol-error", + "overall_passed": False, + "error": str(exc), + "not_measured": list(NOT_MEASURED), + } + if args.json: + print(RESULT_PREFIX + json.dumps(result, ensure_ascii=False, sort_keys=True)) + else: + print(f"PROTOCOL ERROR: {exc}", file=sys.stderr) + return 2 + finally: + if cleanup is not None: + cleanup.cleanup() + + if args.keep_temp: + result["artifact_root"] = str(work_root) + if args.json: + print(RESULT_PREFIX + json.dumps(result, ensure_ascii=False, sort_keys=True)) + else: + _print_human(result) + if args.keep_temp: + print(f"Artifacts: {work_root}") + return 0 if result["overall_passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/holdout/LICENSE b/evals/holdout/LICENSE new file mode 100644 index 0000000..aceda8b --- /dev/null +++ b/evals/holdout/LICENSE @@ -0,0 +1,7 @@ +CC0 1.0 Universal + +To the extent possible under law, the authors have waived all copyright and +related or neighboring rights to the public smoke fixture generated by this +directory. This work is published from the GroundMap project. + +https://creativecommons.org/publicdomain/zero/1.0/ diff --git a/evals/holdout/README.md b/evals/holdout/README.md new file mode 100644 index 0000000..beaf576 --- /dev/null +++ b/evals/holdout/README.md @@ -0,0 +1,162 @@ +# External Holdout Protocol + +This directory adds a fail-closed **external** holdout protocol above the public +`evals/longdoc` development set. It uses the same model-free retrieval API: + +```python +rebuild_index(workspace_root, db_path=None) +coverage_report(db_path) +search_evidence(db_path, query, limit=20, expansions=[]) +``` + +Neither the runner nor the public generator calls an LLM, an embedding model, a +network service, or a vector store. + +## Hidden bundle workflow + +A hidden evaluator prepares a v1 bundle following +[`bundle.schema.json`](bundle.schema.json), freezes the **complete file bytes** +and publishes/escrows their SHA-256 before seeing a system run. The bundle file +stays outside the tested workspace: + +```bash +python evals/holdout/run_eval.py \ + --bundle /secure/evaluator/round-07.json \ + --bundle-sha256 '' \ + --run-id round-07-system-a \ + --seed 735019 +``` + +The SHA argument is mandatory. `payload_sha256` inside JSON detects accidental +in-memory mutation but is not a trust anchor: an attacker able to rewrite a +bundle can also recompute that field. Certification therefore requires the +out-of-band complete-file digest, a pre-run timestamp, stable bundle version, +predeclared thresholds, and frozen denominators. + +The runner writes only bundled Markdown and optional outline files to a fresh +workspace, then asks the index to build its own derived database. Gold cases, +facets, minimal evidence sets, forbidden IDs, and exact oracle inventories remain +in runner memory. A Gold canary and forbidden Gold-like filenames are scanned +before and after indexing/search. Main CES retrieval gets exactly the registered +question with `expansions=[]`; facet queries run separately only for the clearly +labelled selected-citation retrieval proxy and never expand the main query. + +The oracle identifies every natural unit by +`path + anchor + kind + subordinal + content_hash + unit_id` and independently +freezes all content-section identities. The runner reads SQLite in read-only +mode and compares every identity/routing/text field; API totals cannot replace +this exact reconciliation. A search hit not byte-for-byte consistent with that +materialized inventory is a protocol error, not a miss. + +Required slices are: + +- `poison`: **if and only if** `forbidden_evidence_ids` is non-empty; +- `unanswerable`: **if and only if** `answerable: false` (v1 additionally + requires its poison set so rejection@20 has a real target); +- `multi_hop`: only for an answerable case with at least two facets and a + facet-supported minimal complete evidence set spanning at least two distinct + `raw/*.md` paths; +- `multilingual`: only when the question plus at least two allowed evidence + records mechanically contain both Latin and CJK scripts, with both scripts + present in the allowed evidence collection; at least one record must be + Latin-only and another must contain CJK, so one mixed identifier is + insufficient. + +The runtime recomputes this exact eligibility; slice labels are not evaluator +assertions. Assigning all four labels to every case therefore fails closed. +All required slices must have a non-zero preregistered denominator. V1 also +freezes answerable cases, unanswerable cases, facets, documents, natural units, +content sections, natural-unit types, normalized-unique questions, and +per-slice case counts. Questions are compared after Unicode NFKC, +case-folding, invisible-format removal, and non-letter/digit separator +normalization; duplicates are rejected rather than counted as independent +cases. Deleting or duplicating a case, changing a +denominator, padding a multi-hop set with unrelated evidence, truncating an +inventory, leaking Gold into the workspace, or returning a forged hit fails +closed (exit `2`). + +For `certification_kind: hidden`, the runtime additionally requires at least +200 normalized-unique base questions and at least 20 mechanically eligible +cases in each required critical slice. These checks prevent simple denominator +inflation but still do not prove that the author or distribution is independent. +Public smoke only requires non-zero slice coverage because it is a protocol +fixture, not a statistical certification set. + +## Rotation is not independence + +`--run-id` and `--seed` deterministically rotate document/case execution order +and produce a unique run fingerprint. The result always says +`independent_distribution: false`. Changing the seed, shuffling positions, or +re-running the same bundle does **not** create a new independent sample. A real +new holdout round needs a separately authored distribution/bundle, a new +`distribution_id`, and a new out-of-band digest committed before evaluation. +Even a schema-valid hidden bundle is certification evidence only when those +external authorship and preregistration conditions are independently +documented; the runner cannot infer them from JSON. + +Hidden CLI output is aggregate-only. `--details` is deliberately rejected for +`certification_kind: hidden`, and the default JSON result removes both per-case +records and failed case IDs. This prevents the reporting interface from +turning a hidden Gold bundle into a reusable development set. + +## Public CC0 smoke bundle + +The repository includes a deterministic generator, not a committed Gold file: + +```bash +tmp="$(mktemp -d)" +meta="$(python evals/holdout/generate_public_smoke.py --output "$tmp/smoke.json")" +sha="$(python -c 'import json,sys; print(json.loads(sys.argv[1])["bundle_sha256"])' "$meta")" +python evals/holdout/run_eval.py \ + --bundle "$tmp/smoke.json" --bundle-sha256 "$sha" \ + --run-id public-smoke-01 --seed 1 +``` + +Every document, entity, number, policy, and case in this generated fixture is +fictional project-authored material released under CC0-1.0. It derives from the +public long-document development fixture and deliberately exposes its Gold, so +its output is always marked `certification_kind: public-smoke` and +`is_hidden_certification: false`. Passing it proves protocol plumbing and +regression behavior only; it is **not** hidden certification and cannot support +an out-of-distribution claim. + +## Metrics and honest boundary + +The bundle preregisters the existing hard gates plus an unanswerable poison gate: + +- natural-unit index coverage = 100%; +- content-section registration coverage = 100%; +- fixed regression pass = 100%; +- Complete Evidence Set Recall@20 >= 98%; +- selected citation precision proxy >= 99%; +- forbidden selected rate = 0% (maximum threshold; lower is better); +- answer coverage >= 98%; +- fully grounded coverage >= 98%; +- unanswerable poison rejection@20 = 100%. + +The stage-two design recommends CES/answerable evidence coverage floors of 95% +for a real external release profile. Protocol v1 intentionally keeps the +stricter first-stage objectives here (98% for both CES and coverage); a bundle +may preregister an even stricter value but never weaken these runtime minima. + +Selected citation precision remains a Gold-facet top-1 **retrieval proxy** and is +reported together with answer/fully-grounded coverage so refusal cannot inflate +precision. `forbidden_selected_rate` counts preregistered facet opportunities +whose final top-1 selector chose a forbidden exact unit and is gated at zero. +By contrast, `diagnostics.answerable_forbidden_candidate_hit_at_20` reports a +registered hard negative anywhere in the answerable top-20 candidate list; that +is expected stress evidence and does **not** fail a case by itself. Only final +selection of the poison fails the selector gate. For unanswerable cases, +`unanswerable_poison_rejection_at_20` still checks whether registered false +evidence entered retrieval; it does not prove semantic answer abstention. This +runner does not measure final answer correctness, claim-to-citation entailment, +required-facet text coverage, or PDF/DOCX/HTML conversion fidelity. Those need +the separate E2E and conversion-fidelity suites. + +A passing public run reports `status: protocol-smoke-passed`; it is never a +certification. A passing `certification_kind: hidden` bundle reports the more +literal `status: hidden-holdout-thresholds-passed`, +`runner_verified_independence: false`, and an explicit conditional statement. +Only the evaluator's separately documented independent authorship, unseen-data +custody, and preregistered complete-file digest can promote that mechanical +result into certification evidence. diff --git a/evals/holdout/__init__.py b/evals/holdout/__init__.py new file mode 100644 index 0000000..20b540e --- /dev/null +++ b/evals/holdout/__init__.py @@ -0,0 +1 @@ +"""GroundMap external holdout evaluation protocol.""" diff --git a/evals/holdout/bundle.schema.json b/evals/holdout/bundle.schema.json new file mode 100644 index 0000000..bde6bfc --- /dev/null +++ b/evals/holdout/bundle.schema.json @@ -0,0 +1,209 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://groundmap.dev/schemas/holdout-bundle-v1.json", + "title": "GroundMap external holdout bundle v1", + "type": "object", + "additionalProperties": false, + "required": [ + "schema_version", + "bundle_id", + "bundle_version", + "distribution_id", + "license", + "certification_kind", + "payload_sha256", + "preregistration", + "documents", + "oracle", + "cases" + ], + "properties": { + "schema_version": {"const": "groundmap.holdout.bundle.v1"}, + "bundle_id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9._-]{2,127}$"}, + "bundle_version": {"type": "string", "pattern": "^[1-9][0-9]*\\.[0-9]+\\.[0-9]+$"}, + "distribution_id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9._-]{2,127}$"}, + "license": {"const": "CC0-1.0"}, + "certification_kind": {"enum": ["hidden", "public-smoke"]}, + "payload_sha256": {"$ref": "#/$defs/sha256"}, + "preregistration": { + "type": "object", + "additionalProperties": false, + "required": [ + "registered_at", "top_k", "thresholds", "denominators", + "required_slices", "rotation_policy" + ], + "properties": { + "registered_at": {"type": "string", "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"}, + "top_k": {"const": 20}, + "thresholds": { + "type": "object", + "additionalProperties": false, + "required": [ + "natural_unit_index_coverage", + "content_section_registration_coverage", + "fixed_regression_pass_rate", + "complete_evidence_set_recall_at_20", + "selected_citation_precision", + "forbidden_selected_rate", + "answer_coverage", + "fully_grounded_coverage", + "unanswerable_poison_rejection_at_20" + ], + "properties": { + "natural_unit_index_coverage": {"type": "number", "minimum": 1, "maximum": 1}, + "content_section_registration_coverage": {"type": "number", "minimum": 1, "maximum": 1}, + "fixed_regression_pass_rate": {"type": "number", "minimum": 1, "maximum": 1}, + "complete_evidence_set_recall_at_20": {"type": "number", "minimum": 0.98, "maximum": 1}, + "selected_citation_precision": {"type": "number", "minimum": 0.99, "maximum": 1}, + "forbidden_selected_rate": {"const": 0}, + "answer_coverage": {"type": "number", "minimum": 0.98, "maximum": 1}, + "fully_grounded_coverage": {"type": "number", "minimum": 0.98, "maximum": 1}, + "unanswerable_poison_rejection_at_20": {"type": "number", "minimum": 1, "maximum": 1} + } + }, + "denominators": { + "type": "object", + "additionalProperties": false, + "required": [ + "documents", "natural_units", "content_sections", "cases", + "unique_questions", "answerable_cases", "unanswerable_cases", "facets", + "natural_unit_types", "slice_cases" + ], + "properties": { + "documents": {"type": "integer", "minimum": 1}, + "natural_units": {"type": "integer", "minimum": 1}, + "content_sections": {"type": "integer", "minimum": 1}, + "cases": {"type": "integer", "minimum": 1}, + "unique_questions": {"type": "integer", "minimum": 1}, + "answerable_cases": {"type": "integer", "minimum": 1}, + "unanswerable_cases": {"type": "integer", "minimum": 1}, + "facets": {"type": "integer", "minimum": 1}, + "natural_unit_types": {"type": "object", "minProperties": 1, "additionalProperties": {"type": "integer", "minimum": 1}}, + "slice_cases": { + "type": "object", + "additionalProperties": false, + "required": ["poison", "unanswerable", "multi_hop", "multilingual"], + "properties": { + "poison": {"type": "integer", "minimum": 1}, + "unanswerable": {"type": "integer", "minimum": 1}, + "multi_hop": {"type": "integer", "minimum": 1}, + "multilingual": {"type": "integer", "minimum": 1} + } + } + } + }, + "required_slices": {"const": ["poison", "unanswerable", "multi_hop", "multilingual"]}, + "rotation_policy": {"const": "deterministic-order-only-v1"} + } + }, + "documents": { + "type": "array", + "minItems": 1, + "items": { + "type": "object", + "additionalProperties": false, + "required": ["path", "markdown", "markdown_sha256"], + "properties": { + "path": {"type": "string", "pattern": "^raw/.+\\.md$"}, + "markdown": {"type": "string", "minLength": 1}, + "markdown_sha256": {"$ref": "#/$defs/sha256"}, + "outline": {"type": "string", "minLength": 1}, + "outline_sha256": {"$ref": "#/$defs/sha256"} + } + } + }, + "oracle": { + "type": "object", + "additionalProperties": false, + "required": ["gold_canary", "expected_inventory", "expected_section_inventory", "evidence"], + "properties": { + "gold_canary": {"type": "string", "pattern": "^HOLDOUT-GOLD-[A-Z0-9]{32,}$"}, + "expected_inventory": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/unit"}}, + "expected_section_inventory": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/section"}}, + "evidence": {"type": "object", "minProperties": 1, "additionalProperties": {"$ref": "#/$defs/evidence"}} + } + }, + "cases": { + "type": "array", + "minItems": 1, + "items": {"$ref": "#/$defs/case"} + } + }, + "allOf": [ + { + "if": {"properties": {"certification_kind": {"const": "hidden"}}}, + "then": { + "properties": { + "preregistration": { + "properties": { + "denominators": { + "properties": { + "cases": {"minimum": 200}, + "unique_questions": {"minimum": 200}, + "slice_cases": { + "properties": { + "poison": {"minimum": 20}, + "unanswerable": {"minimum": 20}, + "multi_hop": {"minimum": 20}, + "multilingual": {"minimum": 20} + } + } + } + } + } + } + } + } + } + ], + "$defs": { + "sha256": {"type": "string", "pattern": "^[0-9a-f]{64}$"}, + "unit": { + "type": "object", + "required": [ + "unit_id", "path", "anchor", "kind", "ordinal", "subordinal", + "owning_section_anchor", "heading_path", "heading_anchors", + "section_summary", "content_hash", "exact_text_hash", "text" + ] + }, + "section": { + "type": "object", + "required": [ + "section_id", "path", "anchor", "level", "ordinal", "title", + "heading_path", "heading_anchors", "content_hash", "agent_summary", "is_content" + ] + }, + "evidence": { + "type": "object", + "required": ["unit_id", "path", "anchor", "kind", "subordinal", "content_hash", "text"] + }, + "facet": { + "type": "object", + "additionalProperties": false, + "required": ["facet_id", "query", "acceptable_evidence_ids"], + "properties": { + "facet_id": {"type": "string", "minLength": 1}, + "query": {"type": "string", "minLength": 1}, + "acceptable_evidence_ids": {"type": "array", "minItems": 1, "items": {"type": "string", "minLength": 1}, "uniqueItems": true} + } + }, + "case": { + "type": "object", + "additionalProperties": false, + "required": [ + "case_id", "question", "category", "answerable", "required_facets", + "minimal_evidence_sets", "forbidden_evidence_ids", "slices" + ], + "properties": { + "case_id": {"type": "string", "minLength": 1}, + "question": {"type": "string", "minLength": 1}, + "category": {"type": "string", "minLength": 1}, + "answerable": {"type": "boolean"}, + "required_facets": {"type": "array", "items": {"$ref": "#/$defs/facet"}}, + "minimal_evidence_sets": {"type": "array", "items": {"type": "array", "minItems": 1, "items": {"type": "string", "minLength": 1}, "uniqueItems": true}}, + "forbidden_evidence_ids": {"type": "array", "items": {"type": "string", "minLength": 1}, "uniqueItems": true}, + "slices": {"type": "array", "items": {"enum": ["poison", "unanswerable", "multi_hop", "multilingual"]}, "uniqueItems": true} + } + } + } +} diff --git a/evals/holdout/generate_public_smoke.py b/evals/holdout/generate_public_smoke.py new file mode 100644 index 0000000..f05257d --- /dev/null +++ b/evals/holdout/generate_public_smoke.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Generate the deterministic CC0 public smoke bundle (never a hidden cert).""" +from __future__ import annotations + +import argparse +import hashlib +import json +import shutil +import sys +import tempfile +from collections import Counter +from pathlib import Path + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +if str(ENGINE_ROOT) not in sys.path: + sys.path.insert(0, str(ENGINE_ROOT)) + +from evals.holdout import protocol # noqa: E402 +from evals.longdoc import corpus # noqa: E402 + + +def _case_slices(case: dict) -> list[str]: + slices = ["poison"] + if case["category"] == "bilingual": + slices.append("multilingual") + if case["category"] == "multi_hop": + slices.append("multi_hop") + return slices + + +def build_public_smoke_bundle() -> dict: + """Build original fictional CC0 data derived from the public longdoc fixture. + + The resulting Gold is intentionally public. It exercises the external + protocol but cannot certify generalization or an unseen distribution. + """ + temporary = Path(tempfile.mkdtemp(prefix="groundmap-holdout-smoke-source-")) + try: + manifest = corpus.materialize(temporary, "baseline") + workspace = Path(manifest["workspace_root"]) + documents = [] + for report in sorted(manifest["documents"], key=lambda row: row["path"]): + markdown_path = workspace / report["path"] + outline_path = markdown_path.with_suffix(".outline.json") + markdown = markdown_path.read_text(encoding="utf-8") + outline_data = json.loads(outline_path.read_text(encoding="utf-8")) + # postprocess records today's date for normal ingest provenance. + # A public fixture generator must instead be reproducible across + # calendar days, so freeze this non-semantic receipt field. + outline_data["generated_at"] = "2026-07-12" + outline = json.dumps( + outline_data, ensure_ascii=False, indent=2, sort_keys=False, + ) + "\n" + documents.append({ + "path": report["path"], + "markdown": markdown, + "markdown_sha256": hashlib.sha256(markdown.encode("utf-8")).hexdigest(), + "outline": outline, + "outline_sha256": hashlib.sha256(outline.encode("utf-8")).hexdigest(), + }) + + cases = [] + for source in manifest["cases"]: + cases.append({ + "case_id": source["case_id"], + "question": source["question"], + "category": source["category"], + "answerable": True, + "required_facets": source["required_facets"], + "minimal_evidence_sets": source["minimal_evidence_sets"], + "forbidden_evidence_ids": source["forbidden_evidence_ids"], + "slices": _case_slices(source), + }) + # Deliberately uses tokens absent from every document. It verifies the + # protocol's unanswerable/poison accounting, not hard open-world abstention. + cases.append({ + "case_id": "unanswerable-smoke-01", + "question": "QZX991 lunar-kelp resonance quarantine duration?", + "category": "unanswerable", + "answerable": False, + "required_facets": [], + "minimal_evidence_sets": [], + "forbidden_evidence_ids": ["MH-X-010"], + "slices": ["poison", "unanswerable"], + }) + + slice_counts = Counter( + slice_name for case in cases for slice_name in case["slices"] + ) + type_counts = Counter(row["kind"] for row in manifest["expected_inventory"]) + answerable_count = sum(case["answerable"] for case in cases) + facet_count = sum(len(case["required_facets"]) for case in cases) + bundle = { + "schema_version": protocol.SCHEMA_VERSION, + "bundle_id": "groundmap-public-holdout-smoke", + "bundle_version": "1.0.0", + "distribution_id": "groundmap-public-longdoc-derived-v1", + "license": "CC0-1.0", + "certification_kind": "public-smoke", + "payload_sha256": "0" * 64, + "preregistration": { + "registered_at": "2026-07-12T00:00:00Z", + "top_k": protocol.TOP_K, + "thresholds": dict(protocol.PROTOCOL_DEFAULT_THRESHOLDS), + "denominators": { + "documents": len(documents), + "natural_units": len(manifest["expected_inventory"]), + "content_sections": len(manifest["expected_section_inventory"]), + "cases": len(cases), + "unique_questions": len(cases), + "answerable_cases": answerable_count, + "unanswerable_cases": len(cases) - answerable_count, + "facets": facet_count, + "natural_unit_types": dict(sorted(type_counts.items())), + "slice_cases": { + name: slice_counts[name] for name in protocol.REQUIRED_SLICES + }, + }, + "required_slices": list(protocol.REQUIRED_SLICES), + "rotation_policy": "deterministic-order-only-v1", + }, + "documents": documents, + "oracle": { + "gold_canary": "HOLDOUT-GOLD-7C48B09A61E24F3582D7422F9A94C13D", + "expected_inventory": manifest["expected_inventory"], + "expected_section_inventory": manifest["expected_section_inventory"], + "evidence": manifest["evidence"], + }, + "cases": cases, + } + bundle["payload_sha256"] = protocol.payload_sha256(bundle) + protocol.validate_bundle(bundle) + return bundle + finally: + shutil.rmtree(temporary, ignore_errors=True) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output", required=True, type=Path) + args = parser.parse_args(argv) + bundle = build_public_smoke_bundle() + digest = protocol.write_bundle(bundle, args.output) + print(json.dumps({ + "output": str(args.output), + "bundle_sha256": digest, + "bundle_id": bundle["bundle_id"], + "certification_kind": "public-smoke", + "hidden_certification": False, + "warning": "Public Gold is visible; this is protocol smoke only, not a hidden certification.", + }, ensure_ascii=False, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/holdout/protocol.py b/evals/holdout/protocol.py new file mode 100644 index 0000000..b3fc018 --- /dev/null +++ b/evals/holdout/protocol.py @@ -0,0 +1,602 @@ +#!/usr/bin/env python3 +"""Fail-closed schema and helpers for external GroundMap holdout bundles. + +The external file SHA-256 is the trust anchor. ``payload_sha256`` additionally +detects accidental in-memory mutation, but intentionally cannot replace an +out-of-band commitment to the complete file bytes. +""" +from __future__ import annotations + +import hashlib +import json +import math +import re +import unicodedata +from collections import Counter +from pathlib import Path, PurePosixPath +from typing import Any + +SCHEMA_VERSION = "groundmap.holdout.bundle.v1" +TOP_K = 20 +REQUIRED_SLICES = ("poison", "unanswerable", "multi_hop", "multilingual") +METRIC_NAMES = ( + "natural_unit_index_coverage", + "content_section_registration_coverage", + "fixed_regression_pass_rate", + "complete_evidence_set_recall_at_20", + "selected_citation_precision", + "forbidden_selected_rate", + "answer_coverage", + "fully_grounded_coverage", + "unanswerable_poison_rejection_at_20", +) + +# A bundle may preregister stricter values, never weaker ones. The file hash +# commits the exact chosen values before a run starts. +PROTOCOL_MIN_THRESHOLDS = { + "natural_unit_index_coverage": 1.0, + "content_section_registration_coverage": 1.0, + "fixed_regression_pass_rate": 1.0, + "complete_evidence_set_recall_at_20": 0.98, + "selected_citation_precision": 0.99, + "answer_coverage": 0.98, + "fully_grounded_coverage": 0.98, + "unanswerable_poison_rejection_at_20": 1.0, +} +# Unlike the quality rates above, lower is better for this safety rate. A +# selected forbidden unit is never tolerable, so v1 fixes the maximum at zero. +PROTOCOL_MAX_THRESHOLDS = { + "forbidden_selected_rate": 0.0, +} +PROTOCOL_DEFAULT_THRESHOLDS = { + **PROTOCOL_MIN_THRESHOLDS, + **PROTOCOL_MAX_THRESHOLDS, +} +METRIC_DIRECTIONS = { + name: ("maximum" if name in PROTOCOL_MAX_THRESHOLDS else "minimum") + for name in METRIC_NAMES +} + + +class HoldoutProtocolError(RuntimeError): + """The evaluation is incomplete and must not be reported as pass/fail.""" + + +def canonical_json(value: Any) -> bytes: + return json.dumps( + value, ensure_ascii=False, sort_keys=True, separators=(",", ":"), + ).encode("utf-8") + + +def payload_sha256(bundle: dict) -> str: + payload = dict(bundle) + payload.pop("payload_sha256", None) + return hashlib.sha256(canonical_json(payload)).hexdigest() + + +def file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def write_bundle(bundle: dict, path: Path) -> str: + """Validate and deterministically write a bundle; return complete-file SHA.""" + materialized = dict(bundle) + materialized["payload_sha256"] = payload_sha256(materialized) + validate_bundle(materialized) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_bytes(canonical_json(materialized) + b"\n") + return file_sha256(path) + + +def load_bundle(path: Path, expected_file_sha256: str) -> dict: + """Load only after matching an out-of-band, preregistered file digest.""" + if not re.fullmatch(r"[0-9a-f]{64}", expected_file_sha256 or ""): + raise HoldoutProtocolError("--bundle-sha256 must be 64 lowercase hex characters") + try: + raw = path.read_bytes() + except OSError as exc: + raise HoldoutProtocolError(f"bundle is not readable: {exc}") from exc + # Hash and parse the same immutable byte snapshot; a second path read would + # introduce a check/use race between the external commitment and JSON input. + actual = hashlib.sha256(raw).hexdigest() + if actual != expected_file_sha256: + raise HoldoutProtocolError( + f"external bundle SHA-256 mismatch: {actual} != {expected_file_sha256}" + ) + try: + bundle = json.loads(raw.decode("utf-8")) + except (UnicodeError, json.JSONDecodeError) as exc: + raise HoldoutProtocolError(f"bundle is not readable canonical JSON: {exc}") from exc + validate_bundle(bundle) + return bundle + + +def normalize_unit_text(text: str) -> str: + return " ".join(unicodedata.normalize("NFC", text).split()) + + +def normalize_question(text: str) -> str: + """Canonical comparison form used to prevent duplicate case inflation.""" + folded = unicodedata.normalize("NFKC", text).casefold() + # Treat punctuation, symbols, whitespace, and invisible format controls as + # separators. Cosmetic edits such as '?'/full-width forms/zero-width marks + # therefore cannot manufacture independent denominators. + lexical = "".join( + character if unicodedata.category(character)[:1] in {"L", "N"} else " " + for character in folded + ) + return " ".join(lexical.split()) + + +def _script_flags(text: str) -> tuple[bool, bool]: + """Return mechanically detectable (Latin, CJK) script presence. + + Script presence is intentionally a conservative eligibility check, not a + language-identification claim. The multilingual slice needs both flags in + the question/allowed-evidence package and at least two allowed evidence + records so a single mixed-script identifier cannot manufacture the slice. + """ + latin = cjk = False + for character in unicodedata.normalize("NFC", text): + name = unicodedata.name(character, "") + latin = latin or name.startswith("LATIN ") + cjk = cjk or ( + "CJK UNIFIED IDEOGRAPH" in name + or "CJK COMPATIBILITY IDEOGRAPH" in name + or name.startswith("HIRAGANA ") + or name.startswith("KATAKANA ") + or name.startswith("HANGUL ") + ) + if latin and cjk: + break + return latin, cjk + + +def content_hash(text: str) -> str: + return hashlib.sha256(normalize_unit_text(text).encode("utf-8")).hexdigest() + + +def exact_text_hash(text: str) -> str: + return hashlib.sha256(unicodedata.normalize("NFC", text).encode("utf-8")).hexdigest() + + +def handle_key(row: dict) -> tuple: + return ( + row["path"], row["anchor"].lstrip("^"), row["kind"], + row["subordinal"], row["content_hash"], row["unit_id"], + ) + + +def _require_dict(value: Any, label: str) -> dict: + if not isinstance(value, dict): + raise HoldoutProtocolError(f"{label} must be an object") + return value + + +def _require_list(value: Any, label: str, *, nonempty: bool = False) -> list: + if not isinstance(value, list) or (nonempty and not value): + qualifier = "a non-empty list" if nonempty else "a list" + raise HoldoutProtocolError(f"{label} must be {qualifier}") + return value + + +def _positive_int(value: Any, label: str, *, allow_zero: bool = False) -> int: + minimum = 0 if allow_zero else 1 + if isinstance(value, bool) or not isinstance(value, int) or value < minimum: + raise HoldoutProtocolError(f"{label} must be an integer >= {minimum}") + return value + + +def _safe_raw_path(value: Any, label: str) -> str: + if not isinstance(value, str) or not value: + raise HoldoutProtocolError(f"{label} must be a non-empty string") + path = PurePosixPath(value) + if path.is_absolute() or ".." in path.parts or not path.parts or path.parts[0] != "raw": + raise HoldoutProtocolError(f"{label} must stay below raw/") + if path.suffix != ".md": + raise HoldoutProtocolError(f"{label} must name converted markdown") + return value + + +def _unique_strings(values: Any, label: str, *, nonempty: bool = False) -> list[str]: + rows = _require_list(values, label, nonempty=nonempty) + if any(not isinstance(row, str) or not row for row in rows): + raise HoldoutProtocolError(f"{label} must contain non-empty strings") + if len(rows) != len(set(rows)): + raise HoldoutProtocolError(f"{label} contains duplicates") + return rows + + +def _validate_documents(bundle: dict) -> tuple[set[str], dict[str, str]]: + documents = _require_list(bundle.get("documents"), "documents", nonempty=True) + paths: set[str] = set() + markdown_by_path: dict[str, str] = {} + for index, document in enumerate(documents): + row = _require_dict(document, f"documents[{index}]") + required = {"path", "markdown", "markdown_sha256"} + if set(row) - {"path", "markdown", "markdown_sha256", "outline", "outline_sha256"}: + raise HoldoutProtocolError(f"documents[{index}] has unknown fields") + if not required <= set(row): + raise HoldoutProtocolError(f"documents[{index}] is missing required fields") + path = _safe_raw_path(row["path"], f"documents[{index}].path") + if path in paths: + raise HoldoutProtocolError(f"duplicate document path: {path}") + paths.add(path) + markdown = row["markdown"] + if not isinstance(markdown, str) or not markdown: + raise HoldoutProtocolError(f"{path}: markdown must be non-empty") + expected = hashlib.sha256(markdown.encode("utf-8")).hexdigest() + if row["markdown_sha256"] != expected: + raise HoldoutProtocolError(f"{path}: markdown SHA-256 mismatch") + outline = row.get("outline") + outline_digest = row.get("outline_sha256") + if (outline is None) != (outline_digest is None): + raise HoldoutProtocolError(f"{path}: outline and outline_sha256 must appear together") + if outline is not None: + if not isinstance(outline, str) or not outline: + raise HoldoutProtocolError(f"{path}: outline must be non-empty JSON text") + if hashlib.sha256(outline.encode("utf-8")).hexdigest() != outline_digest: + raise HoldoutProtocolError(f"{path}: outline SHA-256 mismatch") + try: + parsed_outline = json.loads(outline) + except json.JSONDecodeError as exc: + raise HoldoutProtocolError(f"{path}: outline is invalid JSON") from exc + if not isinstance(parsed_outline, dict): + raise HoldoutProtocolError(f"{path}: outline root must be an object") + markdown_by_path[path] = markdown + return paths, markdown_by_path + + +UNIT_FIELDS = { + "unit_id", "path", "anchor", "kind", "ordinal", "subordinal", + "owning_section_anchor", "heading_path", "heading_anchors", + "section_summary", "content_hash", "exact_text_hash", "text", +} +SECTION_FIELDS = { + "section_id", "path", "anchor", "level", "ordinal", "title", + "heading_path", "heading_anchors", "content_hash", "agent_summary", + "is_content", +} +EVIDENCE_FIELDS = { + "unit_id", "path", "anchor", "kind", "subordinal", "content_hash", "text", +} + + +def _validate_oracle(bundle: dict, document_paths: set[str]) -> tuple[dict, dict, Counter]: + oracle = _require_dict(bundle.get("oracle"), "oracle") + if set(oracle) != { + "gold_canary", "expected_inventory", "expected_section_inventory", "evidence" + }: + raise HoldoutProtocolError("oracle field set does not match schema v1") + canary = oracle["gold_canary"] + if not isinstance(canary, str) or not re.fullmatch(r"HOLDOUT-GOLD-[A-Z0-9]{32,}", canary): + raise HoldoutProtocolError("oracle.gold_canary has invalid format") + if any(canary in document["markdown"] or canary in (document.get("outline") or "") + for document in bundle["documents"]): + raise HoldoutProtocolError("Gold canary leaked into a tested document") + + inventory_rows = _require_list( + oracle["expected_inventory"], "oracle.expected_inventory", nonempty=True, + ) + inventory: dict[str, dict] = {} + for index, row in enumerate(inventory_rows): + row = _require_dict(row, f"expected_inventory[{index}]") + if not UNIT_FIELDS <= set(row): + raise HoldoutProtocolError(f"expected_inventory[{index}] has incomplete identity") + unit_id = row["unit_id"] + if not isinstance(unit_id, str) or not unit_id or unit_id in inventory: + raise HoldoutProtocolError("expected_inventory unit_id is empty or duplicated") + if row["path"] not in document_paths: + raise HoldoutProtocolError(f"{unit_id}: unit path is not a bundled document") + if not isinstance(row["anchor"], str) or not row["anchor"]: + raise HoldoutProtocolError(f"{unit_id}: anchor is invalid") + if (isinstance(row["subordinal"], bool) or + not isinstance(row["subordinal"], int) or row["subordinal"] <= 0): + raise HoldoutProtocolError(f"{unit_id}: subordinal is invalid") + if not isinstance(row["text"], str) or not row["text"]: + raise HoldoutProtocolError(f"{unit_id}: text is invalid") + if content_hash(row["text"]) != row["content_hash"]: + raise HoldoutProtocolError(f"{unit_id}: text/content_hash mismatch") + if exact_text_hash(row["text"]) != row["exact_text_hash"]: + raise HoldoutProtocolError(f"{unit_id}: text/exact_text_hash mismatch") + if not isinstance(row["heading_path"], list) or not isinstance( + row["heading_anchors"], list): + raise HoldoutProtocolError(f"{unit_id}: route fields are invalid") + inventory[unit_id] = row + + sections = _require_list( + oracle["expected_section_inventory"], + "oracle.expected_section_inventory", nonempty=True, + ) + section_by_id: dict[str, dict] = {} + for index, row in enumerate(sections): + row = _require_dict(row, f"expected_section_inventory[{index}]") + if not SECTION_FIELDS <= set(row): + raise HoldoutProtocolError(f"expected_section_inventory[{index}] is incomplete") + section_id = row["section_id"] + if not isinstance(section_id, str) or not section_id or section_id in section_by_id: + raise HoldoutProtocolError("section_id is empty or duplicated") + if row["path"] not in document_paths or row["is_content"] not in (1, True): + raise HoldoutProtocolError(f"{section_id}: content section identity is invalid") + section_by_id[section_id] = row + + evidence = _require_dict(oracle["evidence"], "oracle.evidence") + if not evidence: + raise HoldoutProtocolError("oracle.evidence must not be empty") + for logical_id, row in evidence.items(): + if not isinstance(logical_id, str) or not logical_id: + raise HoldoutProtocolError("evidence logical ID is invalid") + row = _require_dict(row, f"oracle.evidence[{logical_id}]") + if not EVIDENCE_FIELDS <= set(row): + raise HoldoutProtocolError(f"{logical_id}: evidence handle is incomplete") + source = inventory.get(row["unit_id"]) + if source is None: + raise HoldoutProtocolError(f"{logical_id}: evidence unit is outside inventory") + for field in EVIDENCE_FIELDS: + if row[field] != source[field]: + raise HoldoutProtocolError(f"{logical_id}: evidence {field} disagrees with inventory") + canonical = f"{row['path']}#^{row['anchor'].lstrip('^')}" + if row.get("canonical_ref", canonical) != canonical: + raise HoldoutProtocolError(f"{logical_id}: evidence canonical_ref is non-canonical") + return inventory, section_by_id, Counter(row["kind"] for row in inventory.values()) + + +def _validate_cases(bundle: dict, evidence: dict) -> tuple[dict[str, dict], Counter, int, int, int]: + cases = _require_list(bundle.get("cases"), "cases", nonempty=True) + case_by_id: dict[str, dict] = {} + slices: Counter = Counter() + answerable = unanswerable = facets_total = 0 + facet_global_ids: set[str] = set() + normalized_questions: dict[str, str] = {} + for index, case in enumerate(cases): + case = _require_dict(case, f"cases[{index}]") + required = { + "case_id", "question", "category", "answerable", "required_facets", + "minimal_evidence_sets", "forbidden_evidence_ids", "slices", + } + if set(case) != required: + raise HoldoutProtocolError(f"cases[{index}] field set does not match schema v1") + case_id = case["case_id"] + if not isinstance(case_id, str) or not case_id or case_id in case_by_id: + raise HoldoutProtocolError("case_id is empty or duplicated") + if not isinstance(case["question"], str) or not case["question"].strip(): + raise HoldoutProtocolError(f"{case_id}: question is empty") + normalized = normalize_question(case["question"]) + duplicate_of = normalized_questions.get(normalized) + if duplicate_of is not None: + raise HoldoutProtocolError( + f"{case_id}: normalized question duplicates {duplicate_of}; " + "case-count inflation is forbidden" + ) + normalized_questions[normalized] = case_id + if not isinstance(case["category"], str) or not case["category"]: + raise HoldoutProtocolError(f"{case_id}: category is invalid") + case_slices = _unique_strings(case["slices"], f"{case_id}.slices") + unknown = set(case_slices) - set(REQUIRED_SLICES) + if unknown: + raise HoldoutProtocolError(f"{case_id}: unregistered slices {sorted(unknown)}") + forbidden = _unique_strings( + case["forbidden_evidence_ids"], f"{case_id}.forbidden_evidence_ids", + ) + if set(forbidden) - set(evidence): + raise HoldoutProtocolError(f"{case_id}: forbidden evidence is dangling") + if not isinstance(case["answerable"], bool): + raise HoldoutProtocolError(f"{case_id}: answerable must be bool") + facets = _require_list(case["required_facets"], f"{case_id}.required_facets") + evidence_sets = _require_list( + case["minimal_evidence_sets"], f"{case_id}.minimal_evidence_sets", + ) + parsed_evidence_sets: list[set[str]] = [] + if case["answerable"]: + answerable += 1 + if not facets or not evidence_sets: + raise HoldoutProtocolError(f"{case_id}: answerable case needs facets and evidence sets") + for set_index, evidence_set in enumerate(evidence_sets): + ids = _unique_strings( + evidence_set, f"{case_id}.minimal_evidence_sets[{set_index}]", nonempty=True, + ) + if set(ids) - set(evidence): + raise HoldoutProtocolError(f"{case_id}: minimal evidence set is dangling") + if set(ids) & set(forbidden): + raise HoldoutProtocolError(f"{case_id}: Gold and forbidden evidence overlap") + parsed_evidence_sets.append(set(ids)) + else: + unanswerable += 1 + if facets or evidence_sets: + raise HoldoutProtocolError( + f"{case_id}: unanswerable case must have no facets/evidence sets" + ) + if not forbidden: + raise HoldoutProtocolError( + f"{case_id}: unanswerable poison gate requires forbidden evidence" + ) + facet_acceptable: list[set[str]] = [] + for facet_index, facet in enumerate(facets): + facet = _require_dict(facet, f"{case_id}.required_facets[{facet_index}]") + if set(facet) != {"facet_id", "query", "acceptable_evidence_ids"}: + raise HoldoutProtocolError(f"{case_id}: facet field set is invalid") + facet_id = facet["facet_id"] + global_id = f"{case_id}\0{facet_id}" + if not isinstance(facet_id, str) or not facet_id or global_id in facet_global_ids: + raise HoldoutProtocolError(f"{case_id}: facet_id is empty or duplicated") + facet_global_ids.add(global_id) + if not isinstance(facet["query"], str) or not facet["query"].strip(): + raise HoldoutProtocolError(f"{case_id}/{facet_id}: facet query is empty") + acceptable = _unique_strings( + facet["acceptable_evidence_ids"], + f"{case_id}/{facet_id}.acceptable_evidence_ids", nonempty=True, + ) + if set(acceptable) - set(evidence) or set(acceptable) & set(forbidden): + raise HoldoutProtocolError(f"{case_id}/{facet_id}: acceptable evidence is invalid") + facet_acceptable.append(set(acceptable)) + facets_total += 1 + + # Every declared "minimal complete" set must actually be explainable by + # the facets: it covers every facet and contains no unrelated padding. + # This prevents adding a cross-document decoy solely to qualify as + # multi-hop while leaving the Gold answer unchanged. + for set_index, evidence_set in enumerate(parsed_evidence_sets): + if any(not (evidence_set & acceptable) for acceptable in facet_acceptable): + raise HoldoutProtocolError( + f"{case_id}.minimal_evidence_sets[{set_index}] does not cover every facet" + ) + allowed_union = set().union(*facet_acceptable) + if not evidence_set <= allowed_union: + raise HoldoutProtocolError( + f"{case_id}.minimal_evidence_sets[{set_index}] contains evidence " + "not supported by any facet" + ) + + multi_hop_eligible = bool( + case["answerable"] and len(facet_acceptable) >= 2 and any( + len({evidence[logical_id]["path"] for logical_id in evidence_set}) >= 2 + for evidence_set in parsed_evidence_sets + ) + ) + allowed_ids = set().union(*facet_acceptable) if facet_acceptable else set() + allowed_texts = [evidence[logical_id]["text"] for logical_id in sorted(allowed_ids)] + evidence_profiles = [_script_flags(text) for text in allowed_texts] + evidence_latin, evidence_cjk = _script_flags("\n".join(allowed_texts)) + package_latin, package_cjk = _script_flags( + case["question"] + "\n" + "\n".join(allowed_texts) + ) + multilingual_eligible = bool( + case["answerable"] + and len(allowed_ids) >= 2 + and evidence_latin and evidence_cjk + and any(latin and not cjk for latin, cjk in evidence_profiles) + and any(cjk for _latin, cjk in evidence_profiles) + and package_latin and package_cjk + ) + eligible_slices = set() + if forbidden: + eligible_slices.add("poison") + if not case["answerable"]: + eligible_slices.add("unanswerable") + if multi_hop_eligible: + eligible_slices.add("multi_hop") + if multilingual_eligible: + eligible_slices.add("multilingual") + if set(case_slices) != eligible_slices: + raise HoldoutProtocolError( + f"{case_id}: slice labels do not match mechanical eligibility: " + f"declared={sorted(case_slices)!r}, eligible={sorted(eligible_slices)!r}" + ) + slices.update(case_slices) + case_by_id[case_id] = case + if answerable <= 0 or unanswerable <= 0 or facets_total <= 0: + raise HoldoutProtocolError("bundle needs answerable, unanswerable, and facet denominators") + if set(slices) != set(REQUIRED_SLICES) or any(slices[name] <= 0 for name in REQUIRED_SLICES): + raise HoldoutProtocolError("all required slices must have a non-zero denominator") + return case_by_id, slices, answerable, unanswerable, facets_total + + +def _validate_preregistration( + bundle: dict, *, documents: int, inventory: int, sections: int, + types: Counter, cases: int, answerable: int, unanswerable: int, + facets: int, slices: Counter, +) -> None: + prereg = _require_dict(bundle.get("preregistration"), "preregistration") + if set(prereg) != { + "registered_at", "top_k", "thresholds", "denominators", + "required_slices", "rotation_policy", + }: + raise HoldoutProtocolError("preregistration field set does not match schema v1") + if not isinstance(prereg["registered_at"], str) or not re.fullmatch( + r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z", prereg["registered_at"]): + raise HoldoutProtocolError("preregistration.registered_at must be fixed UTC seconds") + if prereg["top_k"] != TOP_K: + raise HoldoutProtocolError(f"top_k must be preregistered as {TOP_K}") + if prereg["rotation_policy"] != "deterministic-order-only-v1": + raise HoldoutProtocolError("unknown rotation policy") + if prereg["required_slices"] != list(REQUIRED_SLICES): + raise HoldoutProtocolError("required_slices order/set drifted") + thresholds = _require_dict(prereg["thresholds"], "preregistration.thresholds") + if set(thresholds) != set(METRIC_NAMES): + raise HoldoutProtocolError("preregistered threshold metric set drifted") + for name, minimum in PROTOCOL_MIN_THRESHOLDS.items(): + value = thresholds[name] + if (isinstance(value, bool) or not isinstance(value, (int, float)) or + not math.isfinite(float(value)) or not minimum <= float(value) <= 1.0): + raise HoldoutProtocolError( + f"threshold {name} must be finite and >= protocol minimum {minimum}" + ) + for name, maximum in PROTOCOL_MAX_THRESHOLDS.items(): + value = thresholds[name] + if (isinstance(value, bool) or not isinstance(value, (int, float)) or + not math.isfinite(float(value)) or not 0.0 <= float(value) <= maximum): + raise HoldoutProtocolError( + f"threshold {name} must be finite and <= protocol maximum {maximum}" + ) + + denominators = _require_dict(prereg["denominators"], "preregistration.denominators") + expected = { + "documents": documents, + "natural_units": inventory, + "content_sections": sections, + "cases": cases, + "unique_questions": cases, + "answerable_cases": answerable, + "unanswerable_cases": unanswerable, + "facets": facets, + "natural_unit_types": dict(sorted(types.items())), + "slice_cases": {name: slices[name] for name in REQUIRED_SLICES}, + } + if denominators != expected: + raise HoldoutProtocolError( + f"preregistered denominators do not match exact oracle: {denominators!r} != {expected!r}" + ) + if bundle["certification_kind"] == "hidden": + if cases < 200: + raise HoldoutProtocolError( + "hidden certification requires at least 200 preregistered base cases" + ) + undersized = {name: slices[name] for name in REQUIRED_SLICES + if slices[name] < 20} + if undersized: + raise HoldoutProtocolError( + f"hidden certification requires >=20 cases in every critical slice: {undersized}" + ) + + +def validate_bundle(bundle: Any) -> None: + bundle = _require_dict(bundle, "bundle") + required_top = { + "schema_version", "bundle_id", "bundle_version", "distribution_id", + "license", "certification_kind", "payload_sha256", "preregistration", + "documents", "oracle", "cases", + } + if set(bundle) != required_top: + raise HoldoutProtocolError("top-level bundle field set does not match schema v1") + if bundle["schema_version"] != SCHEMA_VERSION: + raise HoldoutProtocolError(f"unsupported bundle schema: {bundle['schema_version']!r}") + for field in ("bundle_id", "distribution_id"): + if not isinstance(bundle[field], str) or not re.fullmatch(r"[a-z0-9][a-z0-9._-]{2,127}", bundle[field]): + raise HoldoutProtocolError(f"{field} has invalid format") + if not isinstance(bundle["bundle_version"], str) or not re.fullmatch( + r"[1-9]\d*\.\d+\.\d+", bundle["bundle_version"]): + raise HoldoutProtocolError("bundle_version must be stable semver without prerelease") + if bundle["license"] != "CC0-1.0": + raise HoldoutProtocolError("v1 accepts only redistributable CC0 bundles") + if bundle["certification_kind"] not in ("hidden", "public-smoke"): + raise HoldoutProtocolError("certification_kind must be hidden or public-smoke") + internal = bundle["payload_sha256"] + if not isinstance(internal, str) or not re.fullmatch(r"[0-9a-f]{64}", internal): + raise HoldoutProtocolError("payload_sha256 is invalid") + if payload_sha256(bundle) != internal: + raise HoldoutProtocolError("bundle payload_sha256 mismatch") + + document_paths, _ = _validate_documents(bundle) + inventory, sections, types = _validate_oracle(bundle, document_paths) + evidence = bundle["oracle"]["evidence"] + case_by_id, slices, answerable, unanswerable, facets = _validate_cases(bundle, evidence) + _validate_preregistration( + bundle, documents=len(document_paths), inventory=len(inventory), + sections=len(sections), types=types, cases=len(case_by_id), + answerable=answerable, unanswerable=unanswerable, facets=facets, + slices=slices, + ) diff --git a/evals/holdout/run_eval.py b/evals/holdout/run_eval.py new file mode 100644 index 0000000..9349719 --- /dev/null +++ b/evals/holdout/run_eval.py @@ -0,0 +1,709 @@ +#!/usr/bin/env python3 +"""Run a preregistered external holdout bundle against the evidence index. + +Exit codes: 0=all preregistered thresholds passed; 1=completed but below at +least one threshold; 2=protocol/input/API/integrity error (fail-closed). +""" +from __future__ import annotations + +import argparse +import hashlib +import importlib +import json +import math +import random +import re +import shutil +import sqlite3 +import sys +import tempfile +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +SCRIPTS_DIR = ENGINE_ROOT / "scripts" +for candidate in (str(ENGINE_ROOT), str(SCRIPTS_DIR)): + if candidate not in sys.path: + sys.path.insert(0, candidate) + +from evals.holdout.protocol import ( # noqa: E402 + HoldoutProtocolError, + METRIC_DIRECTIONS, + METRIC_NAMES, + REQUIRED_SLICES, + TOP_K, + content_hash, + exact_text_hash, + handle_key, + load_bundle, +) + +RESULT_PREFIX = "HOLDOUT_EVAL_RESULT " +RUNNER_VERSION = "1.0.0" + + +def _metric(numerator: int, denominator: int, *, scope: str | None = None) -> dict: + if (isinstance(numerator, bool) or isinstance(denominator, bool) or + not isinstance(numerator, int) or not isinstance(denominator, int)): + raise HoldoutProtocolError("metric numerator/denominator must be integers") + if denominator <= 0 or numerator < 0 or numerator > denominator: + raise HoldoutProtocolError(f"invalid metric count: {numerator}/{denominator}") + row = {"numerator": numerator, "denominator": denominator, + "value": numerator / denominator} + if scope is not None: + row["scope"] = scope + return row + + +def _nonnegative_int(value: Any, label: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise HoldoutProtocolError(f"{label} must be a non-negative integer") + return value + + +def _assert_no_gold_leak(workspace: Path, canary: str) -> None: + forbidden_name_parts = ("holdout-gold", "eval-gold", "gold-manifest", "gold-oracle") + needle = canary.encode("utf-8") + for path in workspace.rglob("*"): + if not path.is_file(): + continue + if any(part in path.name.casefold() for part in forbidden_name_parts): + raise HoldoutProtocolError(f"Gold-like file appeared in tested workspace: {path.name}") + try: + with path.open("rb") as handle: + tail = b"" + while True: + chunk = handle.read(1024 * 1024) + if not chunk: + break + haystack = tail + chunk + if needle in haystack: + raise HoldoutProtocolError( + f"Gold canary leaked into tested workspace: {path.name}" + ) + tail = haystack[-max(0, len(needle) - 1):] + except HoldoutProtocolError: + raise + except OSError as exc: + raise HoldoutProtocolError(f"cannot inspect workspace for Gold leak: {exc}") from exc + + +def _materialize_documents(bundle: dict, workspace: Path, rng: random.Random) -> None: + if workspace.exists(): + raise HoldoutProtocolError("run workspace must not pre-exist") + workspace.mkdir(parents=True) + documents = list(bundle["documents"]) + rng.shuffle(documents) + expected_files: set[Path] = set() + for document in documents: + target = workspace / document["path"] + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(document["markdown"], encoding="utf-8") + expected_files.add(target.resolve()) + if "outline" in document: + outline = target.with_suffix(".outline.json") + outline.write_text(document["outline"], encoding="utf-8") + expected_files.add(outline.resolve()) + actual_files = {path.resolve() for path in workspace.rglob("*") if path.is_file()} + if actual_files != expected_files: + raise HoldoutProtocolError("materializer wrote files outside the preregistered document set") + _assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + + +def _validate_coverage(report: Any, bundle: dict) -> dict: + if not isinstance(report, dict) or report.get("errors") not in (None, []): + raise HoldoutProtocolError(f"coverage_report is invalid: {report!r}") + denominators = bundle["preregistration"]["denominators"] + documents = report.get("documents") + if not isinstance(documents, list) or len(documents) != denominators["documents"]: + raise HoldoutProtocolError("coverage document denominator disagrees with bundle") + expected_paths = {row["path"] for row in bundle["documents"]} + actual_paths = {row.get("path") for row in documents if isinstance(row, dict)} + if actual_paths != expected_paths: + raise HoldoutProtocolError("coverage document path set disagrees with bundle") + + result = {} + for report_name, denominator_name, done_name in ( + ("natural_units", "natural_units", "indexed"), + ("content_sections", "content_sections", "registered"), + ): + row = report.get(report_name) + if not isinstance(row, dict): + raise HoldoutProtocolError(f"coverage_report lacks {report_name}") + expected = _nonnegative_int(row.get("expected"), f"{report_name}.expected") + done = _nonnegative_int(row.get(done_name), f"{report_name}.{done_name}") + missing = _nonnegative_int(row.get("missing"), f"{report_name}.missing") + missing_items = row.get("missing_items") + gold_denominator = denominators[denominator_name] + if expected <= 0 or expected != gold_denominator: + raise HoldoutProtocolError( + f"{report_name} denominator disagrees with preregistered oracle" + ) + if done > expected or missing != expected - done: + raise HoldoutProtocolError(f"{report_name} counts do not reconcile") + if not isinstance(missing_items, list) or len(missing_items) != missing: + raise HoldoutProtocolError(f"{report_name}.missing_items does not reconcile") + reported = row.get("coverage") + computed = done / expected + if (isinstance(reported, bool) or not isinstance(reported, (int, float)) or + not math.isfinite(float(reported)) or + not math.isclose(float(reported), computed, rel_tol=0.0, abs_tol=1e-12)): + raise HoldoutProtocolError(f"{report_name}.coverage is forged or malformed") + result[report_name] = { + "expected": expected, done_name: done, "missing": missing, + "missing_items": missing_items, "coverage": computed, + } + return result + + +def _validate_exact_inventory(db_path: Path, bundle: dict) -> dict[str, dict]: + try: + connection = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) + connection.row_factory = sqlite3.Row + units = list(connection.execute( + """SELECT unit_id, path, anchor, kind, ordinal, subordinal, + owning_section_anchor, heading_path_json, + heading_anchors_json, section_summary, content_hash, + exact_text_hash, text + FROM units ORDER BY path, ordinal, subordinal""" + )) + sections = list(connection.execute( + """SELECT section_id, path, anchor, level, ordinal, title, + heading_path_json, heading_anchors_json, content_hash, + agent_summary, is_content + FROM sections WHERE is_content=1 ORDER BY path, ordinal""" + )) + connection.close() + except sqlite3.Error as exc: + raise HoldoutProtocolError(f"cannot read exact retrieval inventory: {exc}") from exc + + actual: dict[str, dict] = {} + for stored in units: + row = dict(stored) + try: + row["heading_path"] = json.loads(row.pop("heading_path_json")) + row["heading_anchors"] = json.loads(row.pop("heading_anchors_json")) + except (TypeError, json.JSONDecodeError) as exc: + raise HoldoutProtocolError(f"{row.get('unit_id')}: invalid route JSON") from exc + unit_id = str(row["unit_id"]) + if unit_id in actual: + raise HoldoutProtocolError("retrieval DB has duplicate unit_id") + actual[unit_id] = row + expected = {row["unit_id"]: row for row in bundle["oracle"]["expected_inventory"]} + if set(actual) != set(expected): + raise HoldoutProtocolError("retrieval DB exact natural-unit set disagrees with oracle") + fields = ( + "path", "anchor", "kind", "ordinal", "subordinal", + "owning_section_anchor", "heading_path", "heading_anchors", + "section_summary", "content_hash", "exact_text_hash", "text", + ) + for unit_id, gold in expected.items(): + got = actual[unit_id] + mismatch = [field for field in fields if got.get(field) != gold.get(field)] + if mismatch: + raise HoldoutProtocolError( + f"{unit_id}: exact unit oracle mismatch in {','.join(mismatch)}" + ) + if content_hash(got["text"]) != got["content_hash"] or ( + exact_text_hash(got["text"]) != got["exact_text_hash"]): + raise HoldoutProtocolError(f"{unit_id}: DB text/hash mismatch") + + actual_sections: dict[str, dict] = {} + for stored in sections: + row = dict(stored) + try: + row["heading_path"] = json.loads(row.pop("heading_path_json")) + row["heading_anchors"] = json.loads(row.pop("heading_anchors_json")) + except (TypeError, json.JSONDecodeError) as exc: + raise HoldoutProtocolError(f"{row.get('section_id')}: invalid section route JSON") from exc + section_id = str(row["section_id"]) + if section_id in actual_sections: + raise HoldoutProtocolError("retrieval DB has duplicate content section_id") + actual_sections[section_id] = row + expected_sections = { + row["section_id"]: row + for row in bundle["oracle"]["expected_section_inventory"] + } + if set(actual_sections) != set(expected_sections): + raise HoldoutProtocolError("retrieval DB exact content-section set disagrees with oracle") + section_fields = ( + "path", "anchor", "level", "ordinal", "title", "heading_path", + "heading_anchors", "content_hash", "agent_summary", "is_content", + ) + for section_id, gold in expected_sections.items(): + mismatch = [field for field in section_fields + if actual_sections[section_id].get(field) != gold.get(field)] + if mismatch: + raise HoldoutProtocolError( + f"{section_id}: exact section oracle mismatch in {','.join(mismatch)}" + ) + actual_types = Counter(row["kind"] for row in actual.values()) + expected_types = bundle["preregistration"]["denominators"]["natural_unit_types"] + if dict(sorted(actual_types.items())) != expected_types: + raise HoldoutProtocolError("retrieval DB natural-unit type denominators drifted") + return actual + + +def _validate_search_result(payload: Any, exact_inventory: dict[str, dict], limit: int) -> list[dict]: + if not isinstance(payload, dict) or not isinstance(payload.get("hits"), list): + raise HoldoutProtocolError("search_evidence must return an object with hits list") + hits = payload["hits"] + if len(hits) > limit: + raise HoldoutProtocolError("search_evidence returned more than the requested limit") + result = [] + seen: set[str] = set() + fields = ( + "path", "anchor", "canonical_ref", "unit_id", "kind", "subordinal", + "content_hash", "text", "score", + ) + for rank, hit in enumerate(hits, 1): + if not isinstance(hit, dict) or any(field not in hit for field in fields): + raise HoldoutProtocolError(f"search hit #{rank} has incomplete identity") + unit_id = hit["unit_id"] + if not isinstance(unit_id, str) or not unit_id or unit_id in seen: + raise HoldoutProtocolError(f"search hit #{rank} has duplicate/invalid unit_id") + seen.add(unit_id) + stored = exact_inventory.get(unit_id) + if stored is None: + raise HoldoutProtocolError(f"search hit #{rank} was forged outside exact inventory") + for field in ("path", "anchor", "kind", "subordinal", "content_hash", "text"): + if hit[field] != stored[field]: + raise HoldoutProtocolError( + f"search hit #{rank} forged {field} relative to exact inventory" + ) + expected_ref = f"{hit['path']}#^{hit['anchor'].lstrip('^')}" + if hit["canonical_ref"] != expected_ref: + raise HoldoutProtocolError(f"search hit #{rank} has non-canonical reference") + if content_hash(hit["text"]) != hit["content_hash"]: + raise HoldoutProtocolError(f"search hit #{rank} has forged text/hash") + score = hit["score"] + if (isinstance(score, bool) or not isinstance(score, (int, float)) or + not math.isfinite(float(score))): + raise HoldoutProtocolError(f"search hit #{rank} has invalid score") + result.append(hit) + return result + + +def _search(api: Any, db_path: Path, query: str, inventory: dict[str, dict]) -> list[dict]: + if not isinstance(query, str) or not query.strip(): + raise HoldoutProtocolError("query must not be empty") + try: + # Never inject Gold facets or intermediate answers into the main query. + payload = api.search_evidence(db_path, query, limit=TOP_K, expansions=[]) + except Exception as exc: + raise HoldoutProtocolError(f"search_evidence failed: {exc}") from exc + return _validate_search_result(payload, inventory, TOP_K) + + +def _handles_for_ids(bundle: dict, logical_ids: list[str]) -> set[tuple]: + evidence = bundle["oracle"]["evidence"] + return {handle_key(evidence[logical_id]) for logical_id in logical_ids} + + +def _evaluate_cases( + api: Any, db_path: Path, bundle: dict, inventory: dict[str, dict], rng: random.Random, +) -> list[dict]: + cases = list(bundle["cases"]) + rng.shuffle(cases) + handle_to_ids: dict[tuple, set[str]] = defaultdict(set) + for logical_id, evidence in bundle["oracle"]["evidence"].items(): + handle_to_ids[handle_key(evidence)].add(logical_id) + results = [] + for case in cases: + # This is the only main-query call for the case. Facet queries below are + # separately labelled retrieval proxies and never expand this call. + top_hits = _search(api, db_path, case["question"], inventory) + top_handles = {handle_key(hit) for hit in top_hits} + forbidden = _handles_for_ids(bundle, case["forbidden_evidence_ids"]) + forbidden_top20 = top_handles & forbidden + if not case["answerable"]: + results.append({ + "case_id": case["case_id"], "category": case["category"], + "answerable": False, "slices": case["slices"], + "unanswerable_rejected": not forbidden_top20, + "regression_pass": not forbidden_top20, + "ces_recalled": None, "answered": None, "fully_grounded": None, + "facets": [], "top20_refs": [row["canonical_ref"] for row in top_hits], + "forbidden_selected_ids": [], + "forbidden_top20_ids": sorted( + logical_id for handle in forbidden_top20 + for logical_id in handle_to_ids.get(handle, set()) + ), + }) + continue + + allowed_sets = [ + _handles_for_ids(bundle, logical_ids) + for logical_ids in case["minimal_evidence_sets"] + ] + ces_recalled = any(evidence_set <= top_handles for evidence_set in allowed_sets) + selected: list[tuple] = [] + facet_results = [] + for facet in case["required_facets"]: + facet_hits = _search(api, db_path, facet["query"], inventory) + selected_hit = facet_hits[0] if facet_hits else None + selected_handle = handle_key(selected_hit) if selected_hit else None + acceptable = _handles_for_ids(bundle, facet["acceptable_evidence_ids"]) + correct = selected_handle in acceptable if selected_handle else False + selected_forbidden = selected_handle in forbidden if selected_handle else False + if selected_handle: + selected.append(selected_handle) + facet_results.append({ + "facet_id": facet["facet_id"], "selected": selected_handle is not None, + "correct": correct, "forbidden_selected": selected_forbidden, + "selected_ref": selected_hit["canonical_ref"] if selected_hit else None, + "selected_unit_id": selected_hit["unit_id"] if selected_hit else None, + }) + selected_set = set(selected) + answered = len(selected) == len(case["required_facets"]) + fully_grounded = ( + answered and all(row["correct"] for row in facet_results) + and any(evidence_set <= selected_set for evidence_set in allowed_sets) + ) + forbidden_selected = selected_set & forbidden + results.append({ + "case_id": case["case_id"], "category": case["category"], + "answerable": True, "slices": case["slices"], + "ces_recalled": ces_recalled, "answered": answered, + "fully_grounded": fully_grounded, + "unanswerable_rejected": None, + "regression_pass": ces_recalled and fully_grounded and not forbidden_selected, + "facets": facet_results, + "top20_refs": [row["canonical_ref"] for row in top_hits], + "forbidden_selected_ids": sorted( + logical_id for handle in forbidden_selected + for logical_id in handle_to_ids.get(handle, set()) + ), + "forbidden_top20_ids": sorted( + logical_id for handle in forbidden_top20 + for logical_id in handle_to_ids.get(handle, set()) + ), + }) + return sorted(results, key=lambda row: row["case_id"]) + + +def _compute_metrics(bundle: dict, coverage: dict, cases: list[dict]) -> tuple[dict, dict]: + answerable = [row for row in cases if row["answerable"]] + unanswerable = [row for row in cases if not row["answerable"]] + facets = [facet for row in answerable for facet in row["facets"]] + metrics = { + "natural_unit_index_coverage": _metric( + coverage["natural_units"]["indexed"], coverage["natural_units"]["expected"]), + "content_section_registration_coverage": _metric( + coverage["content_sections"]["registered"], + coverage["content_sections"]["expected"]), + "fixed_regression_pass_rate": _metric( + sum(row["regression_pass"] for row in cases), len(cases)), + "complete_evidence_set_recall_at_20": _metric( + sum(row["ces_recalled"] for row in answerable), len(answerable)), + "selected_citation_precision": _metric( + sum(facet["correct"] for facet in facets), len(facets), + scope="Gold-facet top-1 exact natural-unit retrieval proxy; not final answer citations"), + "forbidden_selected_rate": _metric( + sum(facet["forbidden_selected"] for facet in facets), len(facets), + scope=("forbidden exact units chosen by the final facet top-1 selector " + "per preregistered facet; candidate-list poison is reported separately")), + "answer_coverage": _metric( + sum(row["answered"] for row in answerable), len(answerable), + scope="all preregistered facets returned a candidate; no answer text generated"), + "fully_grounded_coverage": _metric( + sum(row["fully_grounded"] for row in answerable), len(answerable), + scope="selected facet handles form a complete Gold evidence set"), + "unanswerable_poison_rejection_at_20": _metric( + sum(row["unanswerable_rejected"] for row in unanswerable), len(unanswerable), + scope="retrieval poison safety, not semantic answer abstention"), + } + denominators = bundle["preregistration"]["denominators"] + expected_denominators = { + "natural_unit_index_coverage": denominators["natural_units"], + "content_section_registration_coverage": denominators["content_sections"], + "fixed_regression_pass_rate": denominators["cases"], + "complete_evidence_set_recall_at_20": denominators["answerable_cases"], + "selected_citation_precision": denominators["facets"], + "forbidden_selected_rate": denominators["facets"], + "answer_coverage": denominators["answerable_cases"], + "fully_grounded_coverage": denominators["answerable_cases"], + "unanswerable_poison_rejection_at_20": denominators["unanswerable_cases"], + } + for name, denominator in expected_denominators.items(): + if metrics[name]["denominator"] != denominator: + raise HoldoutProtocolError(f"metric denominator drifted after execution: {name}") + + slices = {} + for slice_name in REQUIRED_SLICES: + rows = [row for row in cases if slice_name in row["slices"]] + if len(rows) != denominators["slice_cases"][slice_name]: + raise HoldoutProtocolError(f"slice denominator drifted after execution: {slice_name}") + answerable_rows = [row for row in rows if row["answerable"]] + unanswerable_rows = [row for row in rows if not row["answerable"]] + report: dict[str, Any] = {"cases": len(rows)} + if answerable_rows: + slice_facets = [facet for row in answerable_rows for facet in row["facets"]] + report["complete_evidence_set_recall_at_20"] = _metric( + sum(row["ces_recalled"] for row in answerable_rows), len(answerable_rows)) + report["fully_grounded_coverage"] = _metric( + sum(row["fully_grounded"] for row in answerable_rows), len(answerable_rows)) + report["forbidden_selected_rate"] = _metric( + sum(facet["forbidden_selected"] for facet in slice_facets), + len(slice_facets), + scope="final facet selector only; top-20 candidate poison is not a failure", + ) + if unanswerable_rows: + report["unanswerable_poison_rejection_at_20"] = _metric( + sum(row["unanswerable_rejected"] for row in unanswerable_rows), + len(unanswerable_rows), + ) + slices[slice_name] = report + return metrics, slices + + +def _candidate_diagnostics(cases: list[dict]) -> dict: + """Non-gating diagnostics kept separate from final selector safety.""" + answerable = [row for row in cases if row["answerable"]] + return { + "answerable_forbidden_candidate_hit_at_20": _metric( + sum(bool(row["forbidden_top20_ids"]) for row in answerable), + len(answerable), + scope=("registered hard-negative appeared anywhere in the top-20 candidate list; " + "diagnostic only and not a failure unless the facet selector chooses it"), + ), + } + + +def _rotation(bundle_sha256: str, bundle: dict, run_id: str, seed: int) -> tuple[dict, random.Random]: + if not isinstance(run_id, str) or not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9._-]{2,127}", run_id): + raise HoldoutProtocolError("run_id has invalid format") + if isinstance(seed, bool) or not isinstance(seed, int) or not 0 <= seed < 2**63: + raise HoldoutProtocolError("seed must be an integer in [0, 2^63)") + material = f"{bundle_sha256}\0{bundle['distribution_id']}\0{run_id}\0{seed}" + fingerprint = hashlib.sha256(material.encode("utf-8")).hexdigest() + return ({ + "run_id": run_id, + "seed": seed, + "run_fingerprint": fingerprint, + "distribution_id": bundle["distribution_id"], + "policy": "deterministic-order-only-v1", + "independent_distribution": False, + "warning": ( + "A new run-id/seed only rotates deterministic materialization/case order; " + "it is not a new independent corpus distribution or hidden certification." + ), + }, random.Random(int(fingerprint[:16], 16))) + + +def evaluate( + api: Any, bundle_path: Path, expected_bundle_sha256: str, work_dir: Path, + *, run_id: str, seed: int, +) -> dict: + for name in ("rebuild_index", "coverage_report", "search_evidence"): + if not callable(getattr(api, name, None)): + raise HoldoutProtocolError(f"retrieval API lacks callable {name}") + bundle = load_bundle(bundle_path, expected_bundle_sha256) + rotation, rng = _rotation(expected_bundle_sha256, bundle, run_id, seed) + run_root = work_dir / f"holdout-{rotation['run_fingerprint'][:16]}" + workspace = run_root / "workspace" + if run_root.exists(): + raise HoldoutProtocolError("run-id/seed already has state in work-dir; use a fresh run") + try: + run_root.mkdir(parents=True) + if bundle_path.resolve().is_relative_to(workspace.resolve()): + raise HoldoutProtocolError("external bundle must not live inside tested workspace") + _materialize_documents(bundle, workspace, rng) + db_path = workspace / ".cache" / "holdout-eval.db" + db_path.parent.mkdir(parents=True) + try: + rebuild = api.rebuild_index(workspace, db_path=db_path) + except Exception as exc: + raise HoldoutProtocolError(f"rebuild_index failed: {exc}") from exc + if not isinstance(rebuild, dict) or rebuild.get("errors") not in (None, []): + raise HoldoutProtocolError(f"rebuild_index returned errors: {rebuild!r}") + if not db_path.is_file(): + raise HoldoutProtocolError("rebuild_index did not create requested database") + _assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + try: + coverage_raw = api.coverage_report(db_path) + except Exception as exc: + raise HoldoutProtocolError(f"coverage_report failed: {exc}") from exc + coverage = _validate_coverage(coverage_raw, bundle) + inventory = _validate_exact_inventory(db_path, bundle) + case_results = _evaluate_cases(api, db_path, bundle, inventory, rng) + _assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + metrics, slices = _compute_metrics(bundle, coverage, case_results) + diagnostics = _candidate_diagnostics(case_results) + thresholds = {} + registered = bundle["preregistration"]["thresholds"] + for name in METRIC_NAMES: + value = metrics[name]["value"] + thresholds[name] = { + "threshold": registered[name], "value": value, + "direction": METRIC_DIRECTIONS[name], + "passed": ( + value <= registered[name] + if METRIC_DIRECTIONS[name] == "maximum" + else value >= registered[name] + ), + } + passed = all(row["passed"] for row in thresholds.values()) + status = ( + "protocol-smoke-passed" + if passed and bundle["certification_kind"] == "public-smoke" + else "hidden-holdout-thresholds-passed" + if passed + else "failed" + ) + return { + "status": status, + "passed": passed, + "runner_version": RUNNER_VERSION, + "schema_version": bundle["schema_version"], + "bundle_id": bundle["bundle_id"], + "bundle_version": bundle["bundle_version"], + "bundle_sha256": expected_bundle_sha256, + "certification_kind": bundle["certification_kind"], + "is_hidden_certification": bundle["certification_kind"] == "hidden", + "runner_verified_independence": False, + "certification_statement": ( + "Public Gold is visible: protocol smoke only, never hidden certification." + if bundle["certification_kind"] == "public-smoke" + else + "Conditional mechanical threshold result only: independent authorship, " + "unseen-data custody, and pre-run commitment require external attestation." + ), + "evaluation_scope": "retrieval-only exact evidence selection", + "not_measured": [ + "final answer factual correctness", + "claim-to-citation semantic entailment", + "semantic abstention for unanswerable questions", + "source-format conversion fidelity", + ], + "rotation": rotation, + "top_k": TOP_K, + "preregistered_denominators": bundle["preregistration"]["denominators"], + "metrics": metrics, + "diagnostics": diagnostics, + "thresholds": thresholds, + "slices": slices, + "failed_case_ids": [row["case_id"] for row in case_results + if not row["regression_pass"]], + "case_results": case_results, + } + except Exception: + # Leave explicit --work-dir evidence for debugging. The caller owns it. + raise + + +def _load_api() -> Any: + try: + return importlib.import_module("retrieval_index") + except Exception as exc: + raise HoldoutProtocolError(f"cannot import scripts/retrieval_index.py: {exc}") from exc + + +def _public_result(result: dict, details: bool) -> dict: + if result.get("certification_kind") == "hidden": + if details: + raise HoldoutProtocolError( + "--details is disabled for hidden certification because per-case " + "labels and evidence selections are Gold-sensitive" + ) + # Case IDs can themselves disclose the hidden construction. Only + # aggregate, preregistered statistics cross the CLI boundary. + return {key: value for key, value in result.items() + if key not in {"case_results", "failed_case_ids"}} + return result if details else {key: value for key, value in result.items() + if key != "case_results"} + + +def _print_human(result: dict) -> None: + label = ( + "HIDDEN-BUNDLE THRESHOLD RUN (external independence not runner-verified)" + if result["certification_kind"] == "hidden" + else "PUBLIC SMOKE (not certification)" + ) + print(f"GroundMap external holdout: {label}") + print(f"bundle={result['bundle_id']}@{result['bundle_version']} sha256={result['bundle_sha256']}") + print(result["rotation"]["warning"]) + print(result["certification_statement"]) + for name in METRIC_NAMES: + metric = result["metrics"][name] + threshold = result["thresholds"][name] + mark = "PASS" if threshold["passed"] else "FAIL" + comparator = "<=" if threshold["direction"] == "maximum" else ">=" + print( + f"{name:48} {metric['value']:.2%} " + f"({metric['numerator']}/{metric['denominator']}) " + f"{comparator} {threshold['threshold']:.2%} {mark}" + ) + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--bundle", required=True, type=Path, + help="external bundle JSON; never place it in tested workspace") + parser.add_argument("--bundle-sha256", required=True, + help="out-of-band preregistered complete-file SHA-256") + parser.add_argument("--run-id", required=True, + help="unique rotating execution ID (does not create independent data)") + parser.add_argument("--seed", required=True, type=int, + help="execution-order seed (does not create independent data)") + parser.add_argument("--work-dir", type=Path, + help="retain run artifacts; default temporary directory is removed") + parser.add_argument("--json", action="store_true") + parser.add_argument("--details", action="store_true") + args = parser.parse_args(argv) + owned_tmp = args.work_dir is None + work_dir = args.work_dir or Path(tempfile.mkdtemp(prefix="groundmap-holdout-")) + try: + work_dir.mkdir(parents=True, exist_ok=True) + result = evaluate( + _load_api(), args.bundle, args.bundle_sha256, work_dir, + run_id=args.run_id, seed=args.seed, + ) + public = _public_result(result, args.details) + if args.json: + print(json.dumps(public, ensure_ascii=False, sort_keys=True)) + else: + _print_human(result) + print(RESULT_PREFIX + json.dumps( + {"status": result["status"], "bundle_sha256": result["bundle_sha256"], + "metrics": result["metrics"], "thresholds": result["thresholds"], + "rotation": result["rotation"]}, + ensure_ascii=False, sort_keys=True, + )) + return 0 if result["passed"] else 1 + except SystemExit as exc: + payload = { + "status": "error", + "error": f"external component attempted SystemExit({exc.code!r})", + } + if args.json: + print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) + else: + print(f"holdout incomplete (fail-closed): {payload['error']}", file=sys.stderr) + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + except HoldoutProtocolError as exc: + payload = {"status": "error", "error": str(exc)} + if args.json: + print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) + else: + print(f"holdout incomplete (fail-closed): {exc}", file=sys.stderr) + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + except Exception as exc: + payload = {"status": "error", "error": f"unexpected {type(exc).__name__}: {exc}"} + if args.json: + print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) + else: + print(f"holdout incomplete (fail-closed): {payload['error']}", file=sys.stderr) + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + finally: + if owned_tmp: + shutil.rmtree(work_dir, ignore_errors=True) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/longdoc/LICENSE b/evals/longdoc/LICENSE new file mode 100644 index 0000000..9ff2710 --- /dev/null +++ b/evals/longdoc/LICENSE @@ -0,0 +1,9 @@ +SPDX-License-Identifier: CC0-1.0 + +To the extent possible under law, the GroundMap contributors waive all +copyright and related or neighboring rights to the synthetic evaluation +corpus, Gold labels, and generated fixture text in this directory under +CC0 1.0 Universal: https://creativecommons.org/publicdomain/zero/1.0/ + +All organizations, products, policies, identifiers, and measurements in the +fixture are fictional and were written specifically for software testing. diff --git a/evals/longdoc/README.md b/evals/longdoc/README.md new file mode 100644 index 0000000..782b84f --- /dev/null +++ b/evals/longdoc/README.md @@ -0,0 +1,113 @@ +# LongDoc Evidence Atlas 开发评测集 + +本目录验证长文档 ingest 的两个“不能丢”约束，以及问答前证据检索的质量： + +```bash +python evals/longdoc/run_eval.py +python evals/longdoc/run_eval.py --json +python evals/longdoc/run_eval.py --json --details # 含逐 case/top-20，输出较大 +``` + +语料由 `corpus.py` 程序生成，所有机构、设备、政策、指标与编号均为虚构原创， +按 [CC0-1.0](LICENSE) 发布。它不会下载第三方材料，也不会调用 LLM。每个生成 +文档约 29–37 万字符，超过长文档档位。每篇只设 6 个通用 H2，每节容纳 16 个 +source slots 且最短也超过 30K 字符；事实没有带查询词的专用 heading，而是真正位于 +长章节内部的首部、中部或尾部。语料共覆盖： + +- 技术手册； +- 政策与条款（列表、引用块、条件与例外）； +- 中英双语与同事实的替代证据； +- 表格数据行； +- 跨文档 multi-hop； +- 同数值错主体毒化干扰； +- 事实位于首部、中部、尾部，并在 `position_moved` 变体中重排。 + +## Gold 与防泄漏 + +`build_fixture()` 中的静态事实表是人工预注册 Gold。每条 case 都记录： + +- `required_facets`：回答必须覆盖的分面，以及该分面的独立检索问题； +- `minimal_evidence_sets`：一个或多个允许的最小充分证据集合； +- `forbidden_evidence_ids`：同数值错主体等不可选证据； +- category/tags：技术、条款、双语、表格、多跳等切片。 + +`materialize()` 只把这些固定事实排进长文档，并通过确定性转换器取得位置变换后 +的 canonical anchor。Gold manifest 只保留在 runner 进程内，**不会写入被测 +workspace**；索引 API 只能看到 raw 和自己的派生 DB。fixture 定义有冻结 SHA-256， +case ID、类别配额、50-case/60-facet 分母和 Gold payload 任一漂移都会 fail-closed。 + +Complete Evidence Set Recall@20 的主检索只提交原始 question，`expansions=[]`； +Gold facet 不会注入主查询，因此 multi-hop 的中间实体不能从 Gold 泄漏给检索器。 +每个 facet 另做一次独立 top-1 检索，只用于 evidence-selection proxy；runner 不会 +根据 Gold 从 top-20 中挑一个看起来正确的结果。 + +自然单元分母来自 corpus 生成规则的独立 inventory，而不是 Markdown parser 回读 +计数。runner 会直接只读 SQLite，对 744 个 unit 的 `unit_id/path/anchor/kind/ordinal/ +subordinal/owning_section/heading_path/heading_anchors/section_summary/content_hash/ +exact_text_hash/text` 做精确集合核对，并逐类型冻结 paragraph/list item/table row/ +blockquote/code/figure 的分母。36 个内容 H2 也按生成器独立登记的 `section_id/path/ +anchor/title/content_hash/route` 逐节核对，不只比较章节数量。引用计分使用包含 +`path+anchor+kind+subordinal+content_hash+unit_id` 的精确 handle；共享父 anchor 的 +不同 table row/list item 不会被折叠。 + +## 指标定义 + +- `natural_unit_index_coverage`：索引完成的自然单元 / fixture 独立登记的全部自然 + 单元。自然单元为 paragraph、单条 list item、单条 table data row、blockquote、 + code、figure；heading 不进此分母。 +- `content_section_registration_coverage`：已登记的内容章节 / fixture 全部内容章节。 + 单一 H1 是文档标题，不计内容章节；本 fixture 每篇固定 6 个通用长 H2，故每变体 + 36 节、两个位置变体合计 72 节。 +- `fixed_regression_pass_rate`：以 50 个 base case 为分母；同一 case 必须在 baseline + 与 position-moved 两变体都召回完整证据、所有 facet top-1 正确、且未选择毒化证据。 +- `complete_evidence_set_recall_at_20`：以 base query 为 macro 单位；存在至少一个 + 允许的最小充分集合完全包含在 top-20，且两个位置变体都成立，才记 1。 +- `selected_citation_precision`：**检索代理指标**。60 个唯一 base facet 各算一次； + 其 top-1 精确自然单元 handle 必须在两个位置变体都属于允许 Gold 才记正确。 + 保留该键以报告用户要求的指标，但它不是最终答案的 citation precision。 +- `answer_coverage`：**answerability 代理指标**。所有 required facets 都返回 candidate + 的 base query 比例；未生成答案文本，不能解释成实际回答覆盖率。 +- `fully_grounded_coverage`：**检索代理指标**。facet top-1 handles 在两个位置变体 + 都共同组成一个最小充分 Gold 集；不包含 answer entailment 判断。 + +位置变体按同一个 base case 聚类做 all-or-nothing 计分，不能把变体伪装成独立样本 +扩充分母。机械覆盖是运行状态而非统计样本：每变体 744 个自然单元，两个变体的 +报告计数为 1488；它不表示 1488 个独立事实样本。 + +## 预注册开发阈值 + +| 指标 | 阈值 | +|---|---:| +| 自然单元机械索引覆盖率 | 100% | +| 内容章节登记覆盖率 | 100% | +| 固定毒化/位置移动回归通过率 | 100% | +| Complete Evidence Set Recall@20 | ≥98% | +| selected citation precision | ≥99% | +| answer coverage | ≥98% | +| fully grounded coverage | ≥98% | + +coverage 与 precision 同时设门槛，避免“全部拒答”或只回答安全分面换得虚假的 +100% precision。 + +> 范围边界：本 runner 测量机械索引和 evidence selection，不生成最终答案，也不 +> 测量答案事实正确性、required-facet 文本覆盖或 claim→citation 语义蕴含。最终回答 +> 的引用精度必须由独立 E2E Gold/语义审计另行认证。 + +## Fail-closed 协议 + +runner 直接调用 `scripts/retrieval_index.py`： + +```python +rebuild_index(workspace_root, db_path=None) -> dict +coverage_report(db_path) -> dict +search_evidence(db_path, query, limit=20, expansions=None) -> dict +``` + +以下任一情况退出码为 `2`，不得当作检出或通过：Gold 悬空/重复/截短、冻结 digest +或 case/category 分母漂移、workspace 出现 Gold、独立 inventory 与 DB 不一致、任一 +自然单元类型缺失、自然单元路由或内容章节身份被同步伪造、FTS 内容影子行与倒排索引不一致、API 缺字段、DB 未生成、coverage 自报值与计数不一致、API 分母 +与 fixture 分母不同、missing 未对账、重复 `unit_id`、非 canonical ref、伪造 +text/hash、同父锚行身份不明、NaN score、超过 top-k、解析失败。 + +退出码 `0` 表示本固定开发集全部达标；`1` 表示评测完成但阈值未达标。固定开发集 +全绿只能作为工程回归证据，不能证明开放世界 100% 或替代隐藏集/人工双审的统计认证。 diff --git a/evals/longdoc/__init__.py b/evals/longdoc/__init__.py new file mode 100644 index 0000000..c9edb0f --- /dev/null +++ b/evals/longdoc/__init__.py @@ -0,0 +1 @@ +"""GroundMap 长文档检索评测套件。""" diff --git a/evals/longdoc/corpus.py b/evals/longdoc/corpus.py new file mode 100644 index 0000000..d287438 --- /dev/null +++ b/evals/longdoc/corpus.py @@ -0,0 +1,825 @@ +#!/usr/bin/env python3 +"""程序生成、人工预注册 Gold 的长文档检索评测语料。 + +语料文本是本项目原创的虚构材料（见同目录 LICENSE，CC0-1.0），不抓取或 +改写第三方文档。Gold 事实、required facets 和允许的最小充分 evidence sets +来自下方静态 fixture 表；materialize 只负责排版、位置变换和计算转换后的 +canonical anchor，绝不读取检索结果生成 Gold。 +""" +from __future__ import annotations + +import json +import hashlib +import re +import sys +import unicodedata +from collections import Counter +from dataclasses import asdict +from dataclasses import dataclass +from pathlib import Path +from typing import Iterable + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +SCRIPTS_DIR = ENGINE_ROOT / "scripts" +if str(SCRIPTS_DIR) not in sys.path: + sys.path.insert(0, str(SCRIPTS_DIR)) + +import postprocess # noqa: E402 +from section_parser import split_blocks, strip_frontmatter # noqa: E402 + +VARIANTS = ("baseline", "position_moved") +SECTION_SLOTS = 96 +GOLD_SCHEMA_VERSION = 2 +EXPECTED_CASE_COUNT = 50 +EXPECTED_EVIDENCE_UNIT_COUNT = 120 +EXPECTED_DOCUMENT_COUNT = 6 +SLOTS_PER_LONG_SECTION = 16 +EXPECTED_CONTENT_SECTION_COUNT = 36 +EXPECTED_NATURAL_UNIT_COUNT = 744 +EXPECTED_CATEGORY_COUNTS = { + "technical_manual": 10, + "policy_terms": 10, + "bilingual": 10, + "table": 10, + "multi_hop": 10, +} +EXPECTED_NATURAL_UNIT_TYPES = { + "paragraph": 536, + "list_item": 84, + "table_row": 68, + "blockquote": 25, + "code": 18, + "figure": 13, +} +# Gold/fact definitions change only by an explicit review that updates this digest. +# Fixture layout/filler counts have their own frozen constants above. +FROZEN_FIXTURE_DEFINITION_SHA256 = "9bbac13248639c449084b7c905bd33505cfc679d43565b018ede647d7f5f900a" + + +@dataclass(frozen=True) +class EvidenceUnit: + logical_id: str + document: str + title: str + markdown: str + kind: str + is_gold: bool + + +@dataclass(frozen=True) +class Facet: + facet_id: str + query: str + acceptable_evidence_ids: tuple[str, ...] + + +@dataclass(frozen=True) +class GoldCase: + case_id: str + question: str + category: str + required_facets: tuple[Facet, ...] + minimal_evidence_sets: tuple[tuple[str, ...], ...] + forbidden_evidence_ids: tuple[str, ...] + tags: tuple[str, ...] + + +def _normalize_unit_text(text: str) -> str: + """自然单元 hash 的公开契约；独立于被测索引/Markdown parser。""" + return re.sub(r"\s+", " ", unicodedata.normalize("NFC", text)).strip() + + +def _unit_content_hash(text: str) -> str: + return hashlib.sha256(_normalize_unit_text(text).encode("utf-8")).hexdigest() + + +def _exact_text_hash(text: str) -> str: + return hashlib.sha256(unicodedata.normalize("NFC", text).encode("utf-8")).hexdigest() + + +def _anchor_normalize(text: str) -> str: + """Independent copy of the public canonical-anchor normalization contract.""" + + value = re.sub(r"`+|\*+|_+|~+|#+|>+", "", text) + value = re.sub(r"!\[([^\]]*)\]\([^\)]*\)", r"\1", value) + value = re.sub(r"\[([^\]]+)\]\([^\)]*\)", r"\1", value) + value = re.sub(r"[\[\]\(\)\|]", "", value) + return re.sub(r"\s+", " ", value).strip().lower() + + +def _canonical_anchor(prefix: str, seq: int, text: str, *, level: int | None = None) -> str: + digest = hashlib.md5(_anchor_normalize(text).encode("utf-8")).hexdigest()[:6] + if level is None: + return f"{prefix}-{seq}-{digest}" + return f"{prefix}-{level}-{seq}-{digest}" + + +def _section_id(path: str, title: str, occurrence: int) -> str: + title_hash = hashlib.sha256( + _normalize_unit_text(title).casefold().encode("utf-8") + ).hexdigest() + material = f"{path}\0section-v1\0heading\0{title_hash}\0{occurrence}" + return hashlib.sha256(material.encode("utf-8")).hexdigest()[:32] + + +def _unit_id(path: str, kind: str, content_hash: str, occurrence: int) -> str: + material = f"{path}\0unit-v1\0{kind}\0{content_hash}\0{occurrence}" + return hashlib.sha256(material.encode("utf-8")).hexdigest()[:32] + + +def _paragraph(code: str, sentence: str) -> str: + return f"**{code}.** {sentence}" + + +def _list_item(code: str, sentence: str) -> str: + return f"- **{code}.** {sentence}" + + +def _table(code: str, subject: str, field: str, value: str, condition: str) -> str: + return ( + "| Clause | Subject | Field | Value | Condition |\n" + "|---|---|---|---:|---|\n" + f"| {code} | {subject} | {field} | {value} | {condition} |" + ) + + +def _tech_specs() -> list[dict]: + rows = [ + ("Aster thermal controller", "maximum continuous inlet temperature", "43 °C", "Quiet-2 fan profile", "Boreal relay"), + ("Beacon edge node", "cold-start snapshot retention", "18 minutes", "battery-isolated restart", "Cinder edge node"), + ("Cobalt dosing pump", "service-coupler torque", "7.4 N·m", "dry maintenance mode", "Dune dosing pump"), + ("Delta telemetry cache", "reserved recovery capacity", "640 MiB", "firmware 4.2", "Elm telemetry cache"), + ("Ember optical sensor", "permitted calibration drift", "±0.12 mm", "reference plate R7", "Flint optical sensor"), + ("Fjord field gateway", "automatic handshake retries", "3 attempts", "satellite fallback disabled", "Grove field gateway"), + ("Garnet backup pack", "minimum monitoring endurance", "28 hours", "load profile L3", "Hearth backup pack"), + ("Harbor exhaust module", "declared acoustic ceiling", "61 dBA", "one metre free-field test", "Islet exhaust module"), + ("Indigo isolation valve", "opening pressure", "2.7 bar", "medium class M2", "Juniper isolation valve"), + ("Juniper control bus", "maximum failover latency", "125 ms", "dual-coordinator mode", "Kestrel control bus"), + ] + out = [] + for idx, (subject, field, value, condition, distractor) in enumerate(rows, 1): + code = f"TM-{idx:03d}" + out.append({ + "case_id": f"tech-{idx:02d}", "code": code, "subject": subject, + "field": field, "value": value, "condition": condition, + "distractor": distractor, + "question": f"What is the {field} for the {subject} under {condition}?", + }) + return out + + +def _policy_specs() -> list[dict]: + rows = [ + ("Northbank household plan", "cancellation request window", "14 calendar days", "after electronic activation", "Southbank household plan"), + ("Redwood merchant account", "approved refund transmission deadline", "9 business days", "after approval is recorded", "Maple merchant account"), + ("Silverlake archive service", "verified deletion completion", "72 hours", "after identity verification", "Copperlake archive service"), + ("Tern district permit", "administrative appeal period", "21 calendar days", "after the written decision", "Heron district permit"), + ("Umber audit ledger", "mandatory retention period", "6 years", "from fiscal-year close", "Ochre audit ledger"), + ("Violet incident policy", "regulator notification deadline", "36 hours", "after a confirmed material breach", "Lilac incident policy"), + ("Willow youth account", "minimum self-consent age", "16 years", "unless local law requires older", "Aspen youth account"), + ("Xenon supplier programme", "remediation evidence period", "45 days", "after a major audit finding", "Argon supplier programme"), + ("Yarrow trial subscription", "cooling-off period", "11 calendar days", "after the trial converts to paid", "Saffron trial subscription"), + ("Zephyr pricing notice", "advance-notice period", "30 calendar days", "before a recurring fee increase", "Mistral pricing notice"), + ] + out = [] + for idx, (subject, field, value, condition, distractor) in enumerate(rows, 1): + out.append({ + "case_id": f"policy-{idx:02d}", "code": f"POL-{idx:03d}", + "subject": subject, "field": field, "value": value, + "condition": condition, "distractor": distractor, + "question": f"Under the terms, what is the {field} for the {subject} {condition}?", + }) + return out + + +def _bilingual_specs() -> list[dict]: + rows = [ + ("Jade assembly line", "紧急停机后的复位等待时间", "reset wait after an emergency stop", "17 分钟", "17 minutes", "Pearl assembly line"), + ("Kite sealed enclosure", "允许的最高相对湿度", "maximum permitted relative humidity", "68%", "68%", "Lark sealed enclosure"), + ("Lotus inspection cell", "双人复核抽样数量", "dual-review sample size", "24 件", "24 items", "Moss inspection cell"), + ("Moonrise sterilizer", "高温循环后的冷却时间", "cool-down after a high-temperature cycle", "32 分钟", "32 minutes", "Sunset sterilizer"), + ("Nacre storage room", "夜间最低照度", "minimum night-time illuminance", "85 勒克斯", "85 lux", "Opal storage room"), + ("Orchid packaging line", "标签偏移容差", "label-offset tolerance", "1.6 毫米", "1.6 mm", "Poppy packaging line"), + ("Pine test bench", "连续运行后的休止周期", "rest interval after continuous operation", "12 分钟", "12 minutes", "Cedar test bench"), + ("Quartz clean zone", "人员进入上限", "maximum simultaneous occupancy", "7 人", "7 people", "Granite clean zone"), + ("Reed mixing vessel", "校验液最低体积", "minimum verification-fluid volume", "4.5 升", "4.5 litres", "Rush mixing vessel"), + ("Spruce transfer arm", "安全制动距离", "safe braking distance", "38 厘米", "38 centimetres", "Fir transfer arm"), + ] + out = [] + for idx, (subject, zh_field, en_field, zh_value, en_value, distractor) in enumerate(rows, 1): + question = (f"{subject} 的{zh_field}是多少？" if idx % 2 else + f"What is the {en_field} for the {subject}?") + out.append({ + "case_id": f"bilingual-{idx:02d}", "code": f"BI-{idx:03d}", + "subject": subject, "zh_field": zh_field, "en_field": en_field, + "zh_value": zh_value, "en_value": en_value, + "distractor": distractor, "question": question, + }) + return out + + +def _table_specs() -> list[dict]: + rows = [ + ("Quartz-A classifier", "verified precision", "87.5%", "validation set V4", "Quartz-B classifier"), + ("Rill-B detector", "verified recall", "62.3%", "night subset N2", "Rill-C detector"), + ("Sable-C router", "p95 routing latency", "48 ms", "profile P7", "Sable-D router"), + ("Topaz-D filter", "retained-signal ratio", "93.1%", "noise band B3", "Topaz-E filter"), + ("Umber-E sampler", "median sample yield", "1,240 units", "batch family F8", "Umber-F sampler"), + ("Vela-F monitor", "false-alarm rate", "0.8%", "scenario S5", "Vela-G monitor"), + ("Wren-G scheduler", "jobs completed per hour", "360 jobs", "queue mode Q2", "Wren-H scheduler"), + ("Xylem-H compressor", "specific energy use", "2.4 kWh", "one-tonne reference run", "Xylem-I compressor"), + ("Yucca-I reader", "successful scan rate", "99.2%", "matte-label cohort", "Yucca-J reader"), + ("Zinnia-J balancer", "maximum skew", "4.6%", "eight-shard topology", "Zinnia-K balancer"), + ] + out = [] + for idx, (subject, field, value, condition, distractor) in enumerate(rows, 1): + out.append({ + "case_id": f"table-{idx:02d}", "code": f"TAB-{idx:03d}", + "subject": subject, "field": field, "value": value, + "condition": condition, "distractor": distractor, + "question": f"In the calibration register, what is the {field} of {subject} for {condition}?", + }) + return out + + +def _multihop_specs() -> list[dict]: + rows = [ + ("Lantern", "LX-41", "Cedar", "post-restart observation", "6 minutes", "Birch"), + ("Mariner", "MR-22", "Falcon", "isolation hold", "13 minutes", "Hawk"), + ("Nimbus", "NB-73", "Glacier", "sensor stabilization", "27 minutes", "Tundra"), + ("Osprey", "OS-14", "Harbor", "network quiet period", "8 minutes", "Quay"), + ("Prairie", "PR-65", "Ivory", "operator watch", "19 minutes", "Ebony"), + ("Quill", "QL-09", "Juniper", "pressure confirmation", "11 minutes", "Laurel"), + ("Raven", "RV-38", "Keystone", "redundancy check", "23 minutes", "Archway"), + ("Solstice", "SO-57", "Lagoon", "coolant circulation", "16 minutes", "Estuary"), + ("Tempest", "TP-86", "Meadow", "load recovery", "31 minutes", "Pasture"), + ("Umbra", "UM-20", "Northstar", "clock resynchronization", "9 minutes", "Southstar"), + ] + out = [] + for idx, (project, alert, family, action, value, distractor) in enumerate(rows, 1): + out.append({ + "case_id": f"multihop-{idx:02d}", "code": f"MH-{idx:03d}", + "project": project, "alert": alert, "family": family, + "action": action, "value": value, "distractor": distractor, + "question": (f"For Project {project} alert {alert}, which runbook family applies " + f"and how long is the required {action}?") + }) + return out + + +def build_fixture() -> tuple[list[EvidenceUnit], list[GoldCase]]: + """返回固定事实表和 Gold；不接受系统预测输入。""" + units: list[EvidenceUnit] = [] + cases: list[GoldCase] = [] + + for row in _tech_specs(): + correct = row["code"] + poison = f"TM-X-{correct[-3:]}" + units.extend([ + EvidenceUnit(correct, "raw/manuals/orchid_manual.md", + f"{correct} {row['subject']} operating limit", + _paragraph(correct, f"The {row['subject']} has a {row['field']} of {row['value']} under {row['condition']}."), + "paragraph", True), + EvidenceUnit(poison, "raw/manuals/orchid_manual.md", + f"Comparison note for {row['subject']} and {row['distractor']}", + _paragraph(poison, f"The {row['distractor']}, not the {row['subject']}, has a {row['field']} of {row['value']} under {row['condition']}; this value must not be applied to the {row['subject']}."), + "paragraph", False), + ]) + facet = Facet("limit", row["question"], (correct,)) + cases.append(GoldCase(row["case_id"], row["question"], "technical_manual", + (facet,), ((correct,),), (poison,), + ("same-value-wrong-subject", "technical", "position-move"))) + + for idx, row in enumerate(_policy_specs(), 1): + correct = row["code"] + poison = f"POL-X-{correct[-3:]}" + correct_md = (_list_item(correct, f"For the {row['subject']}, the {row['field']} is {row['value']} {row['condition']}.") + if idx % 2 else + f"> **{correct}.** For the {row['subject']}, the {row['field']} is {row['value']} {row['condition']}." ) + units.extend([ + EvidenceUnit(correct, "raw/policies/northbank_terms.md", + f"{correct} {row['subject']} clause", correct_md, + "list" if idx % 2 else "blockquote", True), + EvidenceUnit(poison, "raw/policies/northbank_terms.md", + f"Non-applicable comparison: {row['distractor']}", + _list_item(poison, f"The {row['distractor']} has a {row['field']} of {row['value']} {row['condition']}; the clause does not govern the {row['subject']}."), + "list", False), + ]) + facet = Facet("term", row["question"], (correct,)) + cases.append(GoldCase(row["case_id"], row["question"], "policy_terms", + (facet,), ((correct,),), (poison,), + ("policy-condition", "same-value-wrong-subject", "position-move"))) + + for row in _bilingual_specs(): + base = row["code"] + zh = f"{base}-ZH" + en = f"{base}-EN" + poison = f"BI-X-{base[-3:]}" + units.extend([ + EvidenceUnit(zh, "raw/guides/bilingual_safety.md", + f"{zh} {row['subject']} 中文安全条款", + _paragraph(zh, f"{row['subject']} 的{row['zh_field']}是 {row['zh_value']}。该数值仅适用于 {row['subject']}。"), + "paragraph", True), + EvidenceUnit(en, "raw/guides/bilingual_safety.md", + f"{en} English safety clause for {row['subject']}", + _paragraph(en, f"For the {row['subject']}, the {row['en_field']} is {row['en_value']}. This value applies only to the {row['subject']}."), + "paragraph", True), + EvidenceUnit(poison, "raw/guides/bilingual_safety.md", + f"双语对照排除项 {row['distractor']} / {row['subject']}", + _paragraph(poison, f"{row['distractor']} 的{row['zh_field']}是 {row['zh_value']}；the same {row['en_value']} value belongs to the {row['distractor']}, not the {row['subject']}."), + "paragraph", False), + ]) + facet = Facet("bilingual_fact", row["question"], (zh, en)) + cases.append(GoldCase(row["case_id"], row["question"], "bilingual", + (facet,), ((zh,), (en,)), (poison,), + ("cross-language", "alternative-evidence", "same-value-wrong-subject", "position-move"))) + + for row in _table_specs(): + correct = row["code"] + poison = f"TAB-X-{correct[-3:]}" + units.extend([ + EvidenceUnit(correct, "raw/catalogs/calibration_register.md", + f"{correct} result for {row['subject']}", + _table(correct, row["subject"], row["field"], row["value"], row["condition"]), + "table_row", True), + EvidenceUnit(poison, "raw/catalogs/calibration_register.md", + f"Comparison row for {row['subject']} and {row['distractor']}", + _table(poison, row["distractor"], row["field"], row["value"], + f"{row['condition']}; not applicable to {row['subject']}"), + "table_row", False), + ]) + facet = Facet("table_metric", row["question"], (correct,)) + cases.append(GoldCase(row["case_id"], row["question"], "table", + (facet,), ((correct,),), (poison,), + ("table-row", "same-value-wrong-subject", "position-move"))) + + for row in _multihop_specs(): + route = f"{row['code']}-ROUTE" + rule = f"{row['code']}-RULE" + poison = f"MH-X-{row['code'][-3:]}" + route_query = (f"Which runbook family is assigned to Project {row['project']} " + f"alert {row['alert']}?") + rule_query = f"How long is the {row['action']} for runbook family {row['family']}?" + units.extend([ + EvidenceUnit(route, "raw/operations/route_registry.md", + f"{route} Project {row['project']} route", + _paragraph(route, f"Project {row['project']} alert {row['alert']} maps to runbook family {row['family']}."), + "paragraph", True), + EvidenceUnit(rule, "raw/operations/runbook_rules.md", + f"{rule} family {row['family']} rule", + _paragraph(rule, f"Runbook family {row['family']} requires a {row['action']} of {row['value']} before closure."), + "paragraph", True), + EvidenceUnit(poison, "raw/operations/runbook_rules.md", + f"Distractor family {row['distractor']} near {row['family']}", + _paragraph(poison, f"Runbook family {row['distractor']}, not family {row['family']}, requires a {row['action']} of {row['value']} before closure."), + "paragraph", False), + ]) + facets = ( + Facet("route", route_query, (route,)), + Facet("rule", rule_query, (rule,)), + ) + cases.append(GoldCase(row["case_id"], row["question"], "multi_hop", + facets, ((route, rule),), (poison,), + ("cross-document", "multi-hop", "same-value-wrong-subject", "position-move"))) + + validate_fixture(units, cases) + return units, cases + + +def validate_fixture(units: Iterable[EvidenceUnit], cases: Iterable[GoldCase]) -> None: + """Gold 任一重复、悬空或空分母都拒绝。""" + units = list(units) + cases = list(cases) + unit_ids = [u.logical_id for u in units] + case_ids = [c.case_id for c in cases] + if not units or not cases: + raise ValueError("longdoc fixture 不得为空") + if len(units) != EXPECTED_EVIDENCE_UNIT_COUNT: + raise ValueError( + f"evidence fixture 分母漂移：{len(units)} != {EXPECTED_EVIDENCE_UNIT_COUNT}" + ) + if len(cases) != EXPECTED_CASE_COUNT: + raise ValueError(f"Gold case 分母漂移：{len(cases)} != {EXPECTED_CASE_COUNT}") + if len(unit_ids) != len(set(unit_ids)): + raise ValueError("evidence logical_id 重复") + if len(case_ids) != len(set(case_ids)): + raise ValueError("Gold case_id 重复") + category_counts = Counter(case.category for case in cases) + if dict(category_counts) != EXPECTED_CATEGORY_COUNTS: + raise ValueError( + f"Gold category 配额漂移：{dict(category_counts)} != {EXPECTED_CATEGORY_COUNTS}" + ) + questions = [case.question for case in cases] + if len(questions) != len(set(questions)): + raise ValueError("Gold question 重复") + known = set(unit_ids) + gold = {u.logical_id for u in units if u.is_gold} + poison = known - gold + for case in cases: + if not case.required_facets or not case.minimal_evidence_sets: + raise ValueError(f"{case.case_id}: required facets/evidence sets 为空") + if len({f.facet_id for f in case.required_facets}) != len(case.required_facets): + raise ValueError(f"{case.case_id}: facet_id 重复") + for facet in case.required_facets: + if not facet.query.strip() or not facet.acceptable_evidence_ids: + raise ValueError(f"{case.case_id}/{facet.facet_id}: facet Gold 不完整") + if not set(facet.acceptable_evidence_ids) <= gold: + raise ValueError(f"{case.case_id}/{facet.facet_id}: facet 引用了非 Gold evidence") + for evidence_set in case.minimal_evidence_sets: + if not evidence_set or not set(evidence_set) <= gold: + raise ValueError(f"{case.case_id}: 最小充分集合为空或悬空") + for facet in case.required_facets: + if not set(evidence_set) & set(facet.acceptable_evidence_ids): + raise ValueError( + f"{case.case_id}: 最小充分集合未覆盖 facet {facet.facet_id}" + ) + # 集合中每个元素都必须是至少一个 facet 的唯一覆盖，否则并非最小。 + for evidence_id in evidence_set: + reduced = set(evidence_set) - {evidence_id} + if all(reduced & set(f.acceptable_evidence_ids) + for f in case.required_facets): + raise ValueError( + f"{case.case_id}: evidence set 含可移除的非最小元素 {evidence_id}" + ) + if not set(case.forbidden_evidence_ids) <= poison: + raise ValueError(f"{case.case_id}: forbidden evidence 不是毒化单元") + + +def _filler_render_and_units(document_key: str, slot: int) -> tuple[str, list[dict]]: + """从生成规则同时渲染 filler 和独立自然单元 oracle，不调用 parser。""" + stem = ( + f"Archive note {document_key}-{slot:03d} describes a fictional verification walk-through. " + "The reviewer checks document identity, records the local sequence, compares the visible label, " + "and closes the worksheet only after a second person confirms the transcription. " + "This narrative intentionally contains no product limit, policy deadline, benchmark result, " + "runbook assignment, or operational answer used by the evaluation questions. " + "It exists to reproduce the length, vocabulary drift, and middle-position pressure of a real manual. " + ) + prose = " ".join(stem for _ in range(8)) + units = [{"kind": "paragraph", "text": prose.strip(), "logical_id": None, + "parent_text": prose.strip(), "block_key": f"{slot}:prose", + "subordinal": 1}] + extras = "" + if slot % 17 == 0: + items = [ + "- record the worksheet identity", + "- compare the visible label", + "- countersign the archival note", + ] + extras = "\n\n" + "\n".join(items) + parent_text = "\n".join(items) + units.extend( + {"kind": "list_item", "text": text, "logical_id": None, + "parent_text": parent_text, "block_key": f"{slot}:list", + "subordinal": index} + for index, text in enumerate(items, 1) + ) + elif slot % 19 == 0: + rows = ["| worksheet identity | recorded |", "| archival label | compared |"] + extras = "\n\n| Check | State |\n|---|---|\n" + "\n".join(rows) + parent_text = "| Check | State |\n|---|---|\n" + "\n".join(rows) + units.extend( + {"kind": "table_row", "text": text, "logical_id": None, + "parent_text": parent_text, "block_key": f"{slot}:table", + "subordinal": index} + for index, text in enumerate(rows, 1) + ) + elif slot % 23 == 0: + extras = "\n\n> Archival reminder: this fictional note carries no normative measurement." + units.append({"kind": "blockquote", "text": extras.strip(), + "logical_id": None, "parent_text": extras.strip(), + "block_key": f"{slot}:quote", "subordinal": 1}) + elif slot % 29 == 0: + extras = "\n\n```text\nverify_document_identity\nrecord_local_sequence\nclose_archival_note\n```" + units.append({"kind": "code", "text": extras.strip(), + "logical_id": None, "parent_text": extras.strip(), + "block_key": f"{slot}:code", "subordinal": 1}) + elif slot % 31 == 0: + extras = f"\n\n![Fictional archival flow {slot}](assets/archival-flow-{slot}.png)" + units.append({"kind": "figure", "text": extras.strip(), + "logical_id": None, "parent_text": extras.strip(), + "block_key": f"{slot}:figure", "subordinal": 1}) + return prose + extras, units + + +def _special_unit_spec(unit: EvidenceUnit, slot: int) -> dict: + if unit.kind == "list": + kind, text = "list_item", unit.markdown + elif unit.kind == "table_row": + kind, text = "table_row", unit.markdown.splitlines()[-1] + elif unit.kind in {"paragraph", "blockquote", "code", "figure"}: + kind, text = unit.kind, unit.markdown + else: + raise ValueError(f"未知 fixture unit kind: {unit.kind}") + return {"kind": kind, "text": text, "logical_id": unit.logical_id, + "parent_text": unit.markdown, + "block_key": f"{slot}:special", "subordinal": 1} + + +def _expected_inventory(path: str, document_title: str, specs: list[dict]) -> list[dict]: + """按生成顺序建立独立 inventory；不消费 split_blocks/索引输出。""" + occurrences: dict[tuple[str, str], int] = {} + rows = [] + block_sequence = 0 + previous_block_key: str | None = None + parent_anchor = "" + h1_anchor = _canonical_anchor("h", 1, document_title, level=1) + for ordinal, spec in enumerate(specs, 1): + if spec["block_key"] != previous_block_key: + block_sequence += 1 + previous_block_key = spec["block_key"] + prefix = { + "table_row": "t", + "code": "c", + "figure": "f", + }.get(spec["kind"], "p") + parent_anchor = _canonical_anchor( + prefix, block_sequence, spec["parent_text"] + ) + content_hash = _unit_content_hash(spec["text"]) + key = (spec["kind"], content_hash) + occurrence = occurrences.get(key, 0) + 1 + occurrences[key] = occurrence + section_number = int(spec["section_index"]) + 1 + section_title = f"Evidence ledger segment {section_number:02d}" + section_anchor = _canonical_anchor("h", section_number, section_title, level=2) + rows.append({ + "unit_id": _unit_id(path, spec["kind"], content_hash, occurrence), + "path": path, + "anchor": parent_anchor, + "kind": spec["kind"], + "ordinal": ordinal, + "subordinal": spec["subordinal"], + "owning_section_anchor": section_anchor, + "heading_path": [document_title, section_title], + "heading_anchors": [h1_anchor, section_anchor], + "section_summary": None, + "content_hash": content_hash, + "exact_text_hash": _exact_text_hash(spec["text"]), + "text": spec["text"], + "logical_id": spec["logical_id"], + }) + return rows + + +def _expected_section_inventory( + path: str, document_title: str, section_markdown: list[str] +) -> list[dict]: + """Build exact H2 identities from generator-owned text, not index rows.""" + + h1_anchor = _canonical_anchor("h", 1, document_title, level=1) + occurrences: dict[str, int] = {} + rows: list[dict] = [] + for index, markdown in enumerate(section_markdown, 1): + title = f"Evidence ledger segment {index:02d}" + title_key = _normalize_unit_text(title).casefold() + occurrence = occurrences.get(title_key, 0) + 1 + occurrences[title_key] = occurrence + anchor = _canonical_anchor("h", index, title, level=2) + rows.append({ + "section_id": _section_id(path, title, occurrence), + "path": path, + "anchor": anchor, + "level": 2, + # The document H1 is ordinal 1 in the materialized heading table. + "ordinal": index + 1, + "title": title, + "heading_path": [document_title, title], + "heading_anchors": [h1_anchor, anchor], + "content_hash": _unit_content_hash(markdown), + "agent_summary": None, + "is_content": 1, + }) + return rows + + +def _position_map(specials: list[EvidenceUnit], variant: str) -> dict[int, EvidenceUnit]: + if variant not in VARIANTS: + raise ValueError(f"未知位置变体: {variant}") + if len(specials) >= SECTION_SLOTS: + raise ValueError("特殊章节数量超过位置槽") + # 均匀铺到首/中/尾；metamorphic 变体反转事实顺序并整体错位一个槽。 + base_positions = [round(i * (SECTION_SLOTS - 1) / max(1, len(specials) - 1)) + for i in range(len(specials))] + if len(set(base_positions)) != len(base_positions): + raise ValueError("位置映射发生碰撞") + ordered = specials if variant == "baseline" else list(reversed(specials)) + return dict(zip(base_positions, ordered)) + + +def _flatten_sections(sections: list[dict]) -> list[dict]: + out: list[dict] = [] + for section in sections: + out.append(section) + out.extend(_flatten_sections(section.get("children", []))) + return out + + +def _anchor_map(markdown_with_anchors: str, units: list[EvidenceUnit], path: str) -> dict[str, dict]: + _fm, body = strip_frontmatter(markdown_with_anchors) + blocks = split_blocks(body) + result: dict[str, dict] = {} + anchor_re = re.compile(r"\^([hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?)\s*$") + for unit in units: + matches = [] + for block in blocks: + if block.kind == "heading" or unit.logical_id not in block.text: + continue + match = anchor_re.search(block.text) + if match: + matches.append((block, match.group(1))) + if len(matches) != 1: + raise ValueError(f"{path}: {unit.logical_id} 应唯一映射到一个自然块，实际 {len(matches)}") + block, anchor = matches[0] + result[unit.logical_id] = { + "path": path, + "anchor": anchor, + "canonical_ref": f"{path}#^{anchor}", + "block_kind": block.kind, + "is_gold": unit.is_gold, + } + return result + + +def _case_payload(case: GoldCase) -> dict: + return { + "case_id": case.case_id, + "question": case.question, + "category": case.category, + "required_facets": [ + {"facet_id": facet.facet_id, "query": facet.query, + "acceptable_evidence_ids": list(facet.acceptable_evidence_ids)} + for facet in case.required_facets + ], + "minimal_evidence_sets": [list(s) for s in case.minimal_evidence_sets], + "forbidden_evidence_ids": list(case.forbidden_evidence_ids), + "tags": list(case.tags), + } + + +def materialize(root: Path, variant: str) -> dict: + """在 ``root`` 下生成一个可直接 rebuild 的 workspace，并返回独立 Gold manifest。""" + units, cases = build_fixture() + workspace = root / "workspaces" / "main" + (workspace / "raw").mkdir(parents=True, exist_ok=True) + by_doc: dict[str, list[EvidenceUnit]] = {} + for unit in units: + by_doc.setdefault(unit.document, []).append(unit) + + evidence: dict[str, dict] = {} + expected_inventory: list[dict] = [] + expected_section_inventory: list[dict] = [] + document_reports: list[dict] = [] + expected_natural_units = 0 + expected_content_sections = 0 + total_type_counts: Counter[str] = Counter() + for doc_path, specials in sorted(by_doc.items()): + placements = _position_map(specials, variant) + title = Path(doc_path).stem.replace("_", " ").title() + section_blocks: list[list[str]] = [ + [] for _ in range(SECTION_SLOTS // SLOTS_PER_LONG_SECTION) + ] + generated_specs: list[dict] = [] + for slot in range(SECTION_SLOTS): + if slot in placements: + unit = placements[slot] + # special 不拥有带查询词的专用 heading；事实真正埋在通用长 H2 内。 + block_markdown = unit.markdown + slot_specs = [_special_unit_spec(unit, slot)] + else: + filler, slot_specs = _filler_render_and_units( + Path(doc_path).stem, slot + 1 + ) + block_markdown = filler + for spec in slot_specs: + spec["section_index"] = slot // SLOTS_PER_LONG_SECTION + generated_specs.extend(slot_specs) + section_blocks[slot // SLOTS_PER_LONG_SECTION].append(block_markdown) + sections = [ + f"## Evidence ledger segment {index + 1:02d}\n\n" + "\n\n".join(blocks) + for index, blocks in enumerate(section_blocks) + ] + plain = f"# {title}\n\n" + "\n\n".join(sections) + "\n" + anchored, outline = postprocess.process(plain, doc_path) + target = workspace / doc_path + target.parent.mkdir(parents=True, exist_ok=True) + target.write_text(anchored, encoding="utf-8") + target.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False, indent=2), encoding="utf-8" + ) + doc_inventory = _expected_inventory(doc_path, title, generated_specs) + doc_section_inventory = _expected_section_inventory(doc_path, title, sections) + natural_count = len(doc_inventory) + type_counts = Counter(row["kind"] for row in doc_inventory) + # 单一 H1 是文档标题；每篇固定 6 个通用长 H2，每节含 16 个 source slots。 + content_sections = [s for s in _flatten_sections(outline["sections"]) + if int(s.get("level", 0)) in (2, 3)] + expected_sections_per_doc = SECTION_SLOTS // SLOTS_PER_LONG_SECTION + if len(content_sections) != expected_sections_per_doc: + raise ValueError(f"{doc_path}: 内容章节分母异常 {len(content_sections)}") + min_section_chars = min( + int(section["char_end"]) - int(section["char_start"]) + for section in content_sections + ) + if min_section_chars <= 30_000: + raise ValueError( + f"{doc_path}: 长章节不足 30K 字符（最短 {min_section_chars}）" + ) + expected_natural_units += natural_count + expected_content_sections += len(content_sections) + total_type_counts.update(type_counts) + expected_inventory.extend(doc_inventory) + expected_section_inventory.extend(doc_section_inventory) + doc_evidence = _anchor_map(anchored, specials, doc_path) + inventory_by_logical = { + row["logical_id"]: row for row in doc_inventory if row["logical_id"] + } + if set(inventory_by_logical) != {unit.logical_id for unit in specials}: + raise ValueError(f"{doc_path}: 生成规则未覆盖全部 logical evidence") + slot_by_id = {unit.logical_id: slot for slot, unit in placements.items()} + for logical_id, row in doc_evidence.items(): + expected_unit = inventory_by_logical[logical_id] + row.update({ + "unit_id": expected_unit["unit_id"], + "kind": expected_unit["kind"], + "subordinal": expected_unit["subordinal"], + "content_hash": expected_unit["content_hash"], + "text": expected_unit["text"], + }) + slot = slot_by_id[logical_id] + row["section_slot"] = slot + row["position_bucket"] = ( + "first" if slot < SECTION_SLOTS / 3 else + "middle" if slot < 2 * SECTION_SLOTS / 3 else "tail" + ) + evidence.update(doc_evidence) + document_reports.append({ + "path": doc_path, + "chars": len(anchored), + "natural_units": natural_count, + "natural_unit_types": dict(sorted(type_counts.items())), + "content_sections": len(content_sections), + "min_content_section_chars": min_section_chars, + }) + + if set(evidence) != {u.logical_id for u in units}: + raise ValueError("manifest evidence 映射未覆盖全部 fixture 单元") + if expected_natural_units != EXPECTED_NATURAL_UNIT_COUNT: + raise ValueError( + f"独立自然单元分母漂移：{expected_natural_units} != {EXPECTED_NATURAL_UNIT_COUNT}" + ) + if expected_content_sections != EXPECTED_CONTENT_SECTION_COUNT: + raise ValueError( + f"独立章节分母漂移：{expected_content_sections} != {EXPECTED_CONTENT_SECTION_COUNT}" + ) + if len(expected_section_inventory) != EXPECTED_CONTENT_SECTION_COUNT: + raise ValueError( + "独立章节身份 inventory 分母漂移：" + f"{len(expected_section_inventory)} != {EXPECTED_CONTENT_SECTION_COUNT}" + ) + if dict(total_type_counts) != EXPECTED_NATURAL_UNIT_TYPES: + raise ValueError( + f"独立自然单元类型分母漂移：{dict(total_type_counts)} " + f"!= {EXPECTED_NATURAL_UNIT_TYPES}" + ) + definition_sha = _fixture_definition_sha256(units, cases) + if definition_sha != FROZEN_FIXTURE_DEFINITION_SHA256: + raise ValueError( + "fixture/Gold 定义 digest 漂移；必须经显式审查后更新冻结值" + ) + manifest = { + "schema_version": GOLD_SCHEMA_VERSION, + "license": "CC0-1.0", + "variant": variant, + "workspace_root": str(workspace), + "fixture_definition_sha256": definition_sha, + "expected": { + "documents": len(by_doc), + "natural_units": expected_natural_units, + "content_sections": expected_content_sections, + "natural_unit_types": dict(sorted(total_type_counts.items())), + }, + "documents": document_reports, + "evidence": evidence, + "expected_inventory": expected_inventory, + "expected_section_inventory": expected_section_inventory, + "cases": [_case_payload(case) for case in cases], + } + return manifest + + +def _fixture_definition_sha256(units: list[EvidenceUnit], cases: list[GoldCase]) -> str: + payload = { + "units": [asdict(unit) for unit in units], + "cases": [asdict(case) for case in cases], + } + encoded = json.dumps( + payload, ensure_ascii=False, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + return hashlib.sha256(encoded).hexdigest() + + +UNITS, CASES = build_fixture() +EXPECTED_CASE_PAYLOADS = [_case_payload(case) for case in CASES] +EXPECTED_CASE_IDS = tuple(case.case_id for case in CASES) diff --git a/evals/longdoc/run_eval.py b/evals/longdoc/run_eval.py new file mode 100644 index 0000000..0fa24cc --- /dev/null +++ b/evals/longdoc/run_eval.py @@ -0,0 +1,805 @@ +#!/usr/bin/env python3 +"""长文档机械覆盖与检索质量评测 runner。 + +退出码：0=全部预注册阈值通过；1=评测完成但至少一个阈值未通过； +2=fixture、Gold、索引、搜索、分母或协议错误（fail-closed）。 +""" +from __future__ import annotations + +import argparse +import hashlib +import importlib +import json +import math +import re +import shutil +import sqlite3 +import sys +import tempfile +import unicodedata +from collections import defaultdict +from collections import Counter +from pathlib import Path +from typing import Any + +EVAL_DIR = Path(__file__).resolve().parent +ENGINE_ROOT = EVAL_DIR.parent.parent +SCRIPTS_DIR = ENGINE_ROOT / "scripts" +for candidate in (str(ENGINE_ROOT), str(EVAL_DIR), str(SCRIPTS_DIR)): + if candidate not in sys.path: + sys.path.insert(0, candidate) + +from evals.longdoc import corpus # noqa: E402 + +RESULT_PREFIX = "LONGDOC_EVAL_RESULT " +TOP_K = 20 + +# 开发集阈值在运行前固定。coverage 同时设门槛，避免用拒答换 citation precision。 +THRESHOLDS = { + "natural_unit_index_coverage": 1.0, + "content_section_registration_coverage": 1.0, + "fixed_regression_pass_rate": 1.0, + "complete_evidence_set_recall_at_20": 0.98, + "selected_citation_precision": 0.99, + "answer_coverage": 0.98, + "fully_grounded_coverage": 0.98, +} + + +class EvaluationProtocolError(RuntimeError): + """评测输入或被测 API 不足以支持可信指标。""" + + +def _metric(numerator: int, denominator: int) -> dict: + if isinstance(numerator, bool) or isinstance(denominator, bool): + raise EvaluationProtocolError("指标分子/分母不得为 bool") + if not isinstance(numerator, int) or not isinstance(denominator, int): + raise EvaluationProtocolError("指标分子/分母必须为整数") + if denominator <= 0: + raise EvaluationProtocolError("指标分母必须大于 0，不能把 undefined 当 100%") + if numerator < 0 or numerator > denominator: + raise EvaluationProtocolError(f"指标计数非法：{numerator}/{denominator}") + return {"numerator": numerator, "denominator": denominator, + "value": numerator / denominator} + + +def _scoped_metric(numerator: int, denominator: int, scope: str) -> dict: + row = _metric(numerator, denominator) + row["scope"] = scope + return row + + +def _as_nonnegative_int(value: Any, label: str) -> int: + if isinstance(value, bool) or not isinstance(value, int) or value < 0: + raise EvaluationProtocolError(f"{label} 必须是非负整数") + return value + + +def _normalize_unit_text(text: str) -> str: + return " ".join(unicodedata.normalize("NFC", text).split()) + + +def _content_hash(text: str) -> str: + return hashlib.sha256(_normalize_unit_text(text).encode("utf-8")).hexdigest() + + +def _exact_text_hash(text: str) -> str: + return hashlib.sha256(unicodedata.normalize("NFC", text).encode("utf-8")).hexdigest() + + +def _handle_key(row: dict) -> tuple: + """自然单元精确 handle；父块 canonical_ref 不能替代 row/item 身份。""" + return ( + row["path"], row["anchor"].lstrip("^"), row["kind"], + row["subordinal"], row["content_hash"], row["unit_id"], + ) + + +def _validate_manifest(manifest: Any, variant: str) -> None: + if not isinstance(manifest, dict): + raise EvaluationProtocolError(f"{variant}: materialize 未返回 manifest object") + if manifest.get("schema_version") != corpus.GOLD_SCHEMA_VERSION: + raise EvaluationProtocolError(f"{variant}: Gold schema_version 漂移") + if manifest.get("variant") != variant: + raise EvaluationProtocolError(f"{variant}: manifest variant 不一致") + if manifest.get("fixture_definition_sha256") != corpus.FROZEN_FIXTURE_DEFINITION_SHA256: + raise EvaluationProtocolError(f"{variant}: fixture definition digest 漂移") + if manifest.get("cases") != corpus.EXPECTED_CASE_PAYLOADS: + raise EvaluationProtocolError( + f"{variant}: Gold case 定义/ID/类别/分母与冻结的 50 cases 不一致" + ) + expected = manifest.get("expected") + required_expected = { + "documents": corpus.EXPECTED_DOCUMENT_COUNT, + "natural_units": corpus.EXPECTED_NATURAL_UNIT_COUNT, + "content_sections": corpus.EXPECTED_CONTENT_SECTION_COUNT, + "natural_unit_types": dict(sorted(corpus.EXPECTED_NATURAL_UNIT_TYPES.items())), + } + if expected != required_expected: + raise EvaluationProtocolError( + f"{variant}: 独立 fixture 分母漂移：{expected!r} != {required_expected!r}" + ) + inventory = manifest.get("expected_inventory") + if not isinstance(inventory, list) or len(inventory) != corpus.EXPECTED_NATURAL_UNIT_COUNT: + raise EvaluationProtocolError(f"{variant}: 独立自然单元 inventory 分母异常") + ids = [row.get("unit_id") for row in inventory if isinstance(row, dict)] + if len(ids) != len(inventory) or len(set(ids)) != len(ids): + raise EvaluationProtocolError(f"{variant}: 独立 inventory unit_id 缺失或重复") + types = Counter(row.get("kind") for row in inventory) + if dict(types) != corpus.EXPECTED_NATURAL_UNIT_TYPES: + raise EvaluationProtocolError(f"{variant}: 独立 inventory 类型分母异常") + required_unit_fields = { + "unit_id", "path", "anchor", "kind", "ordinal", "subordinal", + "owning_section_anchor", "heading_path", "heading_anchors", + "section_summary", "content_hash", "exact_text_hash", "text", + } + for row in inventory: + if not isinstance(row, dict) or not required_unit_fields <= set(row): + raise EvaluationProtocolError(f"{variant}: 独立自然单元身份字段不完整") + if not isinstance(row["heading_path"], list) or not isinstance( + row["heading_anchors"], list + ): + raise EvaluationProtocolError(f"{variant}: 独立自然单元路由字段非法") + + section_inventory = manifest.get("expected_section_inventory") + if ( + not isinstance(section_inventory, list) + or len(section_inventory) != corpus.EXPECTED_CONTENT_SECTION_COUNT + ): + raise EvaluationProtocolError(f"{variant}: 独立章节身份 inventory 分母异常") + section_ids = [ + row.get("section_id") for row in section_inventory if isinstance(row, dict) + ] + if len(section_ids) != len(section_inventory) or len(set(section_ids)) != len( + section_ids + ): + raise EvaluationProtocolError(f"{variant}: 独立章节 section_id 缺失或重复") + required_section_fields = { + "section_id", "path", "anchor", "level", "ordinal", "title", + "heading_path", "heading_anchors", "content_hash", "agent_summary", + "is_content", + } + if any( + not isinstance(row, dict) or not required_section_fields <= set(row) + for row in section_inventory + ): + raise EvaluationProtocolError(f"{variant}: 独立章节身份字段不完整") + evidence = manifest.get("evidence") + expected_evidence_ids = {unit.logical_id for unit in corpus.UNITS} + if not isinstance(evidence, dict) or set(evidence) != expected_evidence_ids: + raise EvaluationProtocolError(f"{variant}: evidence Gold ID 集漂移") + inventory_by_logical = { + row["logical_id"]: row for row in inventory if row.get("logical_id") + } + gold_flags = {unit.logical_id: unit.is_gold for unit in corpus.UNITS} + for logical_id, row in evidence.items(): + source = inventory_by_logical.get(logical_id) + if source is None: + raise EvaluationProtocolError(f"{variant}: {logical_id} 不在独立 inventory") + for key in ( + "unit_id", "path", "anchor", "kind", "subordinal", + "content_hash", "text", + ): + if row.get(key) != source.get(key): + raise EvaluationProtocolError( + f"{variant}: {logical_id} evidence handle 的 {key} 漂移" + ) + anchor = row.get("anchor") + if (not isinstance(anchor, str) or not anchor or + row.get("canonical_ref") != f"{row['path']}#^{anchor.lstrip('^')}"): + raise EvaluationProtocolError(f"{variant}: {logical_id} canonical anchor 非法") + if row.get("is_gold") is not gold_flags[logical_id]: + raise EvaluationProtocolError(f"{variant}: {logical_id} Gold/poison 标签漂移") + slot = row.get("section_slot") + if isinstance(slot, bool) or not isinstance(slot, int) or not 0 <= slot < corpus.SECTION_SLOTS: + raise EvaluationProtocolError(f"{variant}: {logical_id} section_slot 非法") + + +def _validate_coverage(report: Any, manifest: dict) -> dict: + if not isinstance(report, dict): + raise EvaluationProtocolError("coverage_report 必须返回 object") + errors = report.get("errors") + if errors not in (None, []): + raise EvaluationProtocolError(f"coverage_report 报告错误：{errors!r}") + documents = report.get("documents") + if not isinstance(documents, list) or len(documents) != manifest["expected"]["documents"]: + raise EvaluationProtocolError("coverage_report.documents 与 fixture 文档分母不一致") + expected_paths = {row["path"] for row in manifest["documents"]} + actual_paths = {row.get("path") for row in documents if isinstance(row, dict)} + if actual_paths != expected_paths: + raise EvaluationProtocolError("coverage_report.documents 路径集与 fixture 不一致") + + out = {} + specs = ( + ("natural_units", "natural_units", "indexed"), + ("content_sections", "content_sections", "registered"), + ) + for report_key, manifest_key, done_key in specs: + row = report.get(report_key) + if not isinstance(row, dict): + raise EvaluationProtocolError(f"coverage_report 缺少 {report_key}") + expected = _as_nonnegative_int(row.get("expected"), f"{report_key}.expected") + done = _as_nonnegative_int(row.get(done_key), f"{report_key}.{done_key}") + expected_gold = _as_nonnegative_int( + manifest["expected"][manifest_key], f"manifest.expected.{manifest_key}" + ) + if expected <= 0 or expected != expected_gold: + raise EvaluationProtocolError( + f"{report_key} 分母未与独立 fixture Gold 对账：API={expected}, Gold={expected_gold}" + ) + if done > expected: + raise EvaluationProtocolError(f"{report_key} 完成数超过分母") + missing = _as_nonnegative_int(row.get("missing"), f"{report_key}.missing") + missing_items = row.get("missing_items") + if (not isinstance(missing_items, list) or + missing != expected - done or len(missing_items) != missing): + raise EvaluationProtocolError( + f"{report_key}.missing 未与 expected-{done_key} 对账" + ) + coverage = row.get("coverage") + if isinstance(coverage, bool) or not isinstance(coverage, (int, float)): + raise EvaluationProtocolError(f"{report_key}.coverage 不是数值") + computed = done / expected + if not math.isfinite(float(coverage)) or not math.isclose( + float(coverage), computed, rel_tol=0.0, abs_tol=1e-12): + raise EvaluationProtocolError( + f"{report_key}.coverage 自报值 {coverage!r} 与计数 {computed} 不一致" + ) + out[report_key] = {"expected": expected, done_key: done, + "missing": missing, "missing_items": missing_items, + "coverage": computed} + return out + + +def _validate_exact_inventory(db_path: Path, manifest: dict) -> dict[str, dict]: + """用生成规则 oracle 与物化 DB 做逐自然单元对账，不信 coverage 自报总数。""" + try: + connection = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True) + connection.row_factory = sqlite3.Row + rows = list(connection.execute( + """SELECT unit_id, path, anchor, kind, ordinal, subordinal, + owning_section_anchor, heading_path_json, + heading_anchors_json, section_summary, content_hash, + exact_text_hash, text + FROM units ORDER BY path, ordinal""" + )) + section_rows = list(connection.execute( + """SELECT section_id, path, anchor, level, ordinal, title, + heading_path_json, heading_anchors_json, content_hash, + agent_summary, is_content + FROM sections WHERE is_content=1 + ORDER BY path, ordinal""" + )) + connection.close() + except sqlite3.Error as exc: + raise EvaluationProtocolError(f"无法独立读取索引 inventory：{exc}") from exc + + actual: dict[str, dict] = {} + for row in rows: + materialized = dict(row) + try: + materialized["heading_path"] = json.loads( + materialized.pop("heading_path_json") + ) + materialized["heading_anchors"] = json.loads( + materialized.pop("heading_anchors_json") + ) + except (TypeError, ValueError, json.JSONDecodeError) as exc: + raise EvaluationProtocolError( + f"自然单元 {row['unit_id']} 的路由 JSON 非法" + ) from exc + actual[str(row["unit_id"])] = materialized + if len(actual) != len(rows): + raise EvaluationProtocolError("物化 units 表有重复 unit_id") + expected = {row["unit_id"]: row for row in manifest["expected_inventory"]} + if set(actual) != set(expected): + missing = sorted(set(expected) - set(actual))[:5] + extra = sorted(set(actual) - set(expected))[:5] + raise EvaluationProtocolError( + f"自然单元精确 inventory 不一致：missing={missing}, extra={extra}" + ) + compare_fields = ( + "path", "anchor", "kind", "ordinal", "subordinal", + "owning_section_anchor", "heading_path", "heading_anchors", + "section_summary", "content_hash", "exact_text_hash", "text", + ) + for unit_id, gold in expected.items(): + got = actual[unit_id] + mismatched = [key for key in compare_fields if got.get(key) != gold.get(key)] + if mismatched: + raise EvaluationProtocolError( + f"自然单元 {unit_id} 与生成规则不一致：{','.join(mismatched)}" + ) + if _content_hash(got["text"]) != got["content_hash"]: + raise EvaluationProtocolError(f"自然单元 {unit_id} 的 text/hash 不一致") + if _exact_text_hash(got["text"]) != got["exact_text_hash"]: + raise EvaluationProtocolError( + f"自然单元 {unit_id} 的 exact text/hash 不一致" + ) + actual_types = Counter(row["kind"] for row in actual.values()) + if dict(actual_types) != corpus.EXPECTED_NATURAL_UNIT_TYPES: + raise EvaluationProtocolError( + f"物化自然单元类型覆盖异常：{dict(actual_types)}" + ) + + actual_sections: dict[str, dict] = {} + for row in section_rows: + section = dict(row) + try: + section["heading_path"] = json.loads(section.pop("heading_path_json")) + section["heading_anchors"] = json.loads( + section.pop("heading_anchors_json") + ) + except (TypeError, ValueError, json.JSONDecodeError) as exc: + raise EvaluationProtocolError( + f"内容章节 {row['section_id']} 的路由 JSON 非法" + ) from exc + actual_sections[str(row["section_id"])] = section + if len(actual_sections) != len(section_rows): + raise EvaluationProtocolError("物化内容章节有重复 section_id") + expected_sections = { + row["section_id"]: row for row in manifest["expected_section_inventory"] + } + if set(actual_sections) != set(expected_sections): + missing = sorted(set(expected_sections) - set(actual_sections))[:5] + extra = sorted(set(actual_sections) - set(expected_sections))[:5] + raise EvaluationProtocolError( + f"内容章节精确 inventory 不一致：missing={missing}, extra={extra}" + ) + section_fields = ( + "path", "anchor", "level", "ordinal", "title", "heading_path", + "heading_anchors", "content_hash", "agent_summary", "is_content", + ) + for section_id, gold in expected_sections.items(): + got = actual_sections[section_id] + mismatched = [ + key for key in section_fields if got.get(key) != gold.get(key) + ] + if mismatched: + raise EvaluationProtocolError( + f"内容章节 {section_id} 与生成规则不一致：{','.join(mismatched)}" + ) + return actual + + +def _validate_search_result( + payload: Any, + *, + limit: int, + exact_inventory: dict[str, dict] | None = None, +) -> list[dict]: + if not isinstance(payload, dict): + raise EvaluationProtocolError("search_evidence 必须返回 object") + hits = payload.get("hits") + if not isinstance(hits, list): + raise EvaluationProtocolError("search_evidence.hits 必须是 list") + if len(hits) > limit: + raise EvaluationProtocolError(f"search_evidence 返回 {len(hits)} 条，超过 limit={limit}") + out = [] + seen_units: set[str] = set() + for rank, hit in enumerate(hits, 1): + if not isinstance(hit, dict): + raise EvaluationProtocolError(f"hit #{rank} 不是 object") + required = ( + "path", "anchor", "canonical_ref", "unit_id", "kind", "subordinal", + "content_hash", "text", "score", + ) + missing = [key for key in required if key not in hit] + if missing: + raise EvaluationProtocolError(f"hit #{rank} 缺字段：{', '.join(missing)}") + if not all(isinstance(hit[key], str) and hit[key].strip() + for key in ("path", "anchor", "canonical_ref", "unit_id", "kind", "text")): + raise EvaluationProtocolError(f"hit #{rank} 的字符串字段为空或类型错误") + if hit["unit_id"] in seen_units: + raise EvaluationProtocolError(f"search_evidence 返回重复 unit_id: {hit['unit_id']}") + seen_units.add(hit["unit_id"]) + anchor = hit["anchor"].lstrip("^") + expected_ref = f"{hit['path']}#^{anchor}" + if hit["canonical_ref"] != expected_ref: + raise EvaluationProtocolError( + f"hit #{rank} canonical_ref 非 canonical：{hit['canonical_ref']!r} != {expected_ref!r}" + ) + score = hit["score"] + if isinstance(score, bool) or not isinstance(score, (int, float)) or not math.isfinite(float(score)): + raise EvaluationProtocolError(f"hit #{rank}.score 不是有限数值") + if (isinstance(hit["subordinal"], bool) or + not isinstance(hit["subordinal"], int) or hit["subordinal"] <= 0): + raise EvaluationProtocolError(f"hit #{rank}.subordinal 非法") + if (not isinstance(hit["content_hash"], str) or + not re.fullmatch(r"[0-9a-f]{64}", hit["content_hash"])): + raise EvaluationProtocolError(f"hit #{rank}.content_hash 非法") + if _content_hash(hit["text"]) != hit["content_hash"]: + raise EvaluationProtocolError(f"hit #{rank} 的 text/content_hash 不一致") + if exact_inventory is not None: + stored = exact_inventory.get(hit["unit_id"]) + if stored is None: + raise EvaluationProtocolError(f"hit #{rank} unit_id 不在物化 inventory") + for key in ("path", "anchor", "kind", "subordinal", "content_hash", "text"): + if hit[key] != stored[key]: + raise EvaluationProtocolError( + f"hit #{rank} 的 {key} 与物化 inventory 不一致" + ) + out.append(hit) + return out + + +def _search( + api: Any, + db_path: Path, + query: str, + *, + exact_inventory: dict[str, dict], + limit: int = TOP_K, +) -> list[dict]: + if not isinstance(query, str) or not query.strip(): + raise EvaluationProtocolError("检索 query 不得为空") + try: + # 主 CES 与 facet proxy 都只消费各自原始 query；Gold facets 绝不作为 + # 主 query expansions 注入，尤其不能泄漏 multi-hop 中间答案。 + payload = api.search_evidence(db_path, query, limit=limit, expansions=[]) + except Exception as exc: + raise EvaluationProtocolError(f"search_evidence 执行失败：{exc}") from exc + return _validate_search_result( + payload, limit=limit, exact_inventory=exact_inventory + ) + + +def _handles_for_ids(manifest: dict, evidence_ids: list[str] | tuple[str, ...]) -> set[tuple]: + handles = set() + mapping = manifest.get("evidence") + if not isinstance(mapping, dict): + raise EvaluationProtocolError("manifest.evidence 缺失") + for evidence_id in evidence_ids: + row = mapping.get(evidence_id) + required = ("path", "anchor", "kind", "subordinal", "content_hash", "unit_id", "text") + if not isinstance(row, dict) or any(key not in row for key in required): + raise EvaluationProtocolError(f"Gold evidence {evidence_id!r} 没有精确自然单元 handle") + if _content_hash(row["text"]) != row["content_hash"]: + raise EvaluationProtocolError(f"Gold evidence {evidence_id!r} text/hash 不一致") + handles.add(_handle_key(row)) + return handles + + +def _logical_ids_for_handle(manifest: dict) -> dict[tuple, set[str]]: + out: dict[tuple, set[str]] = defaultdict(set) + for logical_id, row in manifest["evidence"].items(): + out[_handle_key(row)].add(logical_id) + return out + + +def _evaluate_variant(api: Any, variant_root: Path, variant: str) -> dict: + manifest = corpus.materialize(variant_root, variant) + _validate_manifest(manifest, variant) + workspace_root = Path(manifest["workspace_root"]) + if (workspace_root / ".eval-gold.json").exists(): + raise EvaluationProtocolError(f"{variant}: Gold 不得写入被测 workspace") + db_path = workspace_root / ".cache" / "longdoc-eval.db" + db_path.parent.mkdir(parents=True, exist_ok=True) + try: + rebuild = api.rebuild_index(workspace_root, db_path=db_path) + except Exception as exc: + raise EvaluationProtocolError(f"{variant}: rebuild_index 失败：{exc}") from exc + if not isinstance(rebuild, dict): + raise EvaluationProtocolError(f"{variant}: rebuild_index 必须返回 object") + if rebuild.get("errors") not in (None, []): + raise EvaluationProtocolError(f"{variant}: rebuild_index 报告错误：{rebuild['errors']!r}") + if (workspace_root / ".eval-gold.json").exists(): + raise EvaluationProtocolError(f"{variant}: 被测 API 在 workspace 创建了 Gold 文件") + if not db_path.is_file(): + raise EvaluationProtocolError(f"{variant}: rebuild_index 未生成 db_path") + try: + raw_coverage = api.coverage_report(db_path) + except Exception as exc: + raise EvaluationProtocolError(f"{variant}: coverage_report 失败：{exc}") from exc + coverage = _validate_coverage(raw_coverage, manifest) + exact_inventory = _validate_exact_inventory(db_path, manifest) + + handle_to_ids = _logical_ids_for_handle(manifest) + case_rows = [] + selected_total = 0 + selected_correct = 0 + for case in manifest["cases"]: + facets = case.get("required_facets") + sets = case.get("minimal_evidence_sets") + if not isinstance(facets, list) or not facets or not isinstance(sets, list) or not sets: + raise EvaluationProtocolError(f"{case.get('case_id')}: Gold facets/sets 非法") + top_hits = _search( + api, db_path, case["question"], exact_inventory=exact_inventory, + limit=TOP_K, + ) + top_handles = {_handle_key(hit) for hit in top_hits} + allowed_sets = [_handles_for_ids(manifest, evidence_set) for evidence_set in sets] + ces_recalled = any(evidence_set <= top_handles for evidence_set in allowed_sets) + + selected_handles: list[tuple] = [] + facet_rows = [] + for facet in facets: + facet_hits = _search( + api, db_path, facet["query"], exact_inventory=exact_inventory, + limit=TOP_K, + ) + selected_hit = facet_hits[0] if facet_hits else None + selected = _handle_key(selected_hit) if selected_hit else None + acceptable = _handles_for_ids(manifest, facet["acceptable_evidence_ids"]) + correct = selected in acceptable if selected is not None else False + if selected is not None: + selected_total += 1 + selected_correct += int(correct) + selected_handles.append(selected) + facet_rows.append({ + "facet_id": facet["facet_id"], + "selected_ref": selected_hit["canonical_ref"] if selected_hit else None, + "selected_unit_id": selected_hit["unit_id"] if selected_hit else None, + "selected_logical_ids": sorted(handle_to_ids.get(selected, set())) if selected else [], + "correct": correct, + }) + + answered = len(selected_handles) == len(facets) + selected_set = set(selected_handles) + all_facets_correct = all(row["correct"] for row in facet_rows) + sufficient = any(evidence_set <= selected_set for evidence_set in allowed_sets) + fully_grounded = answered and all_facets_correct and sufficient + forbidden = _handles_for_ids(manifest, case.get("forbidden_evidence_ids", [])) + forbidden_selected = sorted( + handle[-1] for handle in selected_set & forbidden + ) + regression_pass = ces_recalled and fully_grounded and not forbidden_selected + case_rows.append({ + "case_id": case["case_id"], "category": case["category"], + "ces_recalled": ces_recalled, "answered": answered, + "fully_grounded": fully_grounded, "regression_pass": regression_pass, + "forbidden_selected": forbidden_selected, "facets": facet_rows, + "top20_refs": [hit["canonical_ref"] for hit in top_hits], + }) + + if not case_rows or selected_total <= 0: + raise EvaluationProtocolError(f"{variant}: query/citation 分母为空") + return { + "variant": variant, + "manifest_summary": { + "documents": manifest["expected"]["documents"], + "natural_units": manifest["expected"]["natural_units"], + "content_sections": manifest["expected"]["content_sections"], + }, + "coverage": coverage, + "evidence_layout": { + logical_id: { + "section_slot": row["section_slot"], + "canonical_ref": row["canonical_ref"], + } + for logical_id, row in manifest["evidence"].items() + }, + "cases": case_rows, + "selected_total": selected_total, + "selected_correct": selected_correct, + } + + +def _cluster_metrics(variant_results: list[dict]) -> tuple[dict, dict]: + if len(variant_results) != len(corpus.VARIANTS): + raise EvaluationProtocolError("位置变体结果不完整") + by_variant = {row["variant"]: row for row in variant_results} + if set(by_variant) != set(corpus.VARIANTS): + raise EvaluationProtocolError("位置变体名称重复或缺失") + baseline_layout = by_variant["baseline"]["evidence_layout"] + moved_layout = by_variant["position_moved"]["evidence_layout"] + expected_evidence_ids = {unit.logical_id for unit in corpus.UNITS} + if set(baseline_layout) != expected_evidence_ids or set(moved_layout) != expected_evidence_ids: + raise EvaluationProtocolError("位置变体 evidence layout ID 集不完整") + unchanged = [ + logical_id for logical_id in sorted(expected_evidence_ids) + if (baseline_layout[logical_id]["section_slot"] == + moved_layout[logical_id]["section_slot"] or + baseline_layout[logical_id]["canonical_ref"] == + moved_layout[logical_id]["canonical_ref"]) + ] + if unchanged: + raise EvaluationProtocolError( + f"position_moved 未移动全部 evidence slot/ref：{unchanged[:5]}" + ) + + # 覆盖率按真实单位计数；两位置变体各自都必须 100%。 + natural_num = natural_den = section_num = section_den = 0 + for result in variant_results: + natural = result["coverage"]["natural_units"] + sections = result["coverage"]["content_sections"] + natural_num += natural["indexed"] + natural_den += natural["expected"] + section_num += sections["registered"] + section_den += sections["expected"] + + case_maps = {row["variant"]: {c["case_id"]: c for c in row["cases"]} + for row in variant_results} + case_ids = set(next(iter(case_maps.values()))) + if not case_ids or any(set(mapping) != case_ids for mapping in case_maps.values()): + raise EvaluationProtocolError("两个位置变体的 Gold case 集不一致") + + clustered = [] + clustered_facets = [] + expected_cases = {row["case_id"]: row for row in corpus.EXPECTED_CASE_PAYLOADS} + for case_id in sorted(case_ids): + rows = [case_maps[variant][case_id] for variant in corpus.VARIANTS] + category = rows[0]["category"] + if any(row["category"] != category for row in rows): + raise EvaluationProtocolError(f"{case_id}: 变体 category 不一致") + clustered.append({ + "case_id": case_id, "category": category, + "ces_recalled": all(row["ces_recalled"] for row in rows), + "answered": all(row["answered"] for row in rows), + "fully_grounded": all(row["fully_grounded"] for row in rows), + "regression_pass": all(row["regression_pass"] for row in rows), + }) + expected_facet_ids = { + facet["facet_id"] for facet in expected_cases[case_id]["required_facets"] + } + facet_maps = [ + {facet["facet_id"]: facet for facet in row["facets"]} for row in rows + ] + if any(set(mapping) != expected_facet_ids for mapping in facet_maps): + raise EvaluationProtocolError(f"{case_id}: facet 结果集与冻结 Gold 不一致") + for facet_id in sorted(expected_facet_ids): + variants = [mapping[facet_id] for mapping in facet_maps] + clustered_facets.append({ + "case_id": case_id, + "facet_id": facet_id, + # 同一个 base facet 的位置变体是一个评测单元；任一位置失败即失败。 + "selected": all(row["selected_unit_id"] is not None for row in variants), + "correct": all(row["correct"] for row in variants), + }) + + total_cases = len(clustered) + total_facets = len(clustered_facets) + metrics = { + "natural_unit_index_coverage": _metric(natural_num, natural_den), + "content_section_registration_coverage": _metric(section_num, section_den), + "fixed_regression_pass_rate": _metric( + sum(c["regression_pass"] for c in clustered), total_cases), + "complete_evidence_set_recall_at_20": _metric( + sum(c["ces_recalled"] for c in clustered), total_cases), + "selected_citation_precision": _scoped_metric( + sum(f["correct"] for f in clustered_facets), total_facets, + "retrieval proxy: Gold-facet top-1 exact natural-unit handle; not final answer citations", + ), + "answer_coverage": _scoped_metric( + sum(c["answered"] for c in clustered), total_cases, + "answerability proxy: every Gold facet returned a candidate; no answer text generated", + ), + "fully_grounded_coverage": _scoped_metric( + sum(c["fully_grounded"] for c in clustered), total_cases, + "retrieval proxy: facet top-1 handles form an allowed Gold set; no answer entailment measured", + ), + } + + slices: dict[str, dict] = {} + categories = sorted({c["category"] for c in clustered}) + for category in categories: + rows = [c for c in clustered if c["category"] == category] + if not rows: + raise EvaluationProtocolError(f"slice {category} 分母为空") + slices[category] = { + "cases": len(rows), + "complete_evidence_set_recall_at_20": _metric( + sum(c["ces_recalled"] for c in rows), len(rows)), + "answer_coverage": _metric(sum(c["answered"] for c in rows), len(rows)), + "fully_grounded_coverage": _metric( + sum(c["fully_grounded"] for c in rows), len(rows)), + } + return metrics, {"cases": clustered, "facets": clustered_facets, "slices": slices} + + +def evaluate(api: Any, work_dir: Path) -> dict: + for name in ("rebuild_index", "coverage_report", "search_evidence"): + if not callable(getattr(api, name, None)): + raise EvaluationProtocolError(f"retrieval_index 缺少 callable {name}") + variant_results = [ + _evaluate_variant(api, work_dir / variant, variant) + for variant in corpus.VARIANTS + ] + metrics, clustered = _cluster_metrics(variant_results) + threshold_results = {} + for name, threshold in THRESHOLDS.items(): + value = metrics[name]["value"] + threshold_results[name] = { + "threshold": threshold, "value": value, "passed": value >= threshold, + } + passed = all(row["passed"] for row in threshold_results.values()) + return { + "status": "passed" if passed else "failed", + "evaluation_scope": "retrieval-only evidence selection; not end-to-end answer correctness", + "not_measured": [ + "final answer factual correctness", + "final answer required-facet completeness", + "claim-to-citation semantic entailment", + ], + "top_k": TOP_K, + "variants": list(corpus.VARIANTS), + "gold_cases": len(clustered["cases"]), + "gold_facets": len(clustered["facets"]), + "metrics": metrics, + "thresholds": threshold_results, + "slices": clustered["slices"], + "variant_results": variant_results, + } + + +def _load_api() -> Any: + try: + return importlib.import_module("retrieval_index") + except Exception as exc: + raise EvaluationProtocolError(f"无法导入 scripts/retrieval_index.py：{exc}") from exc + + +def _print_human(result: dict) -> None: + print("LongDoc Evidence Atlas 开发集") + print(f"Gold: {result['gold_cases']} base cases × {len(result['variants'])} 位置变体 | top-k={result['top_k']}") + for name, row in result["metrics"].items(): + threshold = result["thresholds"][name] + mark = "PASS" if threshold["passed"] else "FAIL" + print(f"{name:45} {row['value']:.2%} ({row['numerator']}/{row['denominator']}) " + f">= {threshold['threshold']:.2%} {mark}") + print("\n关键切片（CES / answer coverage / fully grounded）") + for category, row in result["slices"].items(): + print( + f"{category:20} n={row['cases']:2d} " + f"{row['complete_evidence_set_recall_at_20']['value']:.2%} / " + f"{row['answer_coverage']['value']:.2%} / " + f"{row['fully_grounded_coverage']['value']:.2%}" + ) + + +def _public_result(result: dict, *, details: bool) -> dict: + """默认机器输出保持紧凑；显式 --details 才携带每条 top-20 调试包。""" + if details: + return result + return {key: value for key, value in result.items() if key != "variant_results"} + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--json", action="store_true", help="只输出机器可读结果") + parser.add_argument("--details", action="store_true", + help="JSON 中包含逐 case/top-20 调试明细（输出较大）") + parser.add_argument("--work-dir", type=Path, + help="保留/复用评测临时目录（默认运行后删除）") + args = parser.parse_args(argv) + owned_tmp = args.work_dir is None + work_dir = args.work_dir or Path(tempfile.mkdtemp(prefix="groundmap-longdoc-eval-")) + try: + work_dir.mkdir(parents=True, exist_ok=True) + result = evaluate(_load_api(), work_dir) + if args.json: + print(json.dumps(_public_result(result, details=args.details), + ensure_ascii=False, sort_keys=True)) + else: + _print_human(result) + print(RESULT_PREFIX + json.dumps({ + "status": result["status"], "metrics": result["metrics"], + "thresholds": result["thresholds"], + }, ensure_ascii=False, sort_keys=True)) + return 0 if result["status"] == "passed" else 1 + except EvaluationProtocolError as exc: + payload = {"status": "error", "error": str(exc)} + if args.json: + print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) + else: + print(f"评测未完成（fail-closed）：{exc}", file=sys.stderr) + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + except Exception as exc: # parser/storage/library bugs are incomplete, never a failed/pass case + payload = { + "status": "error", + "error": f"unexpected {type(exc).__name__}: {exc}", + } + if args.json: + print(json.dumps(payload, ensure_ascii=False, sort_keys=True)) + else: + print(f"评测未完成（fail-closed）：{payload['error']}", file=sys.stderr) + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + return 2 + finally: + if owned_tmp: + shutil.rmtree(work_dir, ignore_errors=True) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/evals/run_stage2.py b/evals/run_stage2.py new file mode 100644 index 0000000..c7e99f6 --- /dev/null +++ b/evals/run_stage2.py @@ -0,0 +1,608 @@ +#!/usr/bin/env python3 +"""Run GroundMap's public Stage-2 citation-accuracy gates. + +This command deliberately runs only public protocol smoke data and synthetic +source-format fixtures. It can never emit a hidden-certification status. + +Exit codes: 0=all public gates passed; 1=a quality threshold failed; +2=a runner, dependency, integrity, or protocol error occurred. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import subprocess +import sys +import tempfile +from pathlib import Path +from typing import Any + + +ROOT = Path(__file__).resolve().parent.parent +RESULT_PREFIX = "STAGE2_EVAL_RESULT " +KNOWN_PREFIXES = ( + "ANSWER_CITATION_EVAL_RESULT ", + "HOLDOUT_EVAL_RESULT ", + "CONVERSION_FIDELITY_RESULT ", +) + +ANSWER_SCHEMA_VERSION = 3 +ANSWER_PUBLIC_FIXTURE_SHA256 = ( + "abc73f49aac8346b5a42c0c0d07a99659a01da2b5bdef41cc896b24b50a345cc" +) +ANSWER_GOLD_SHA256 = ( + "7c159d95ef83f20c68c78fc301e2f0e29131cf51cf5a8a7d85186c55712cbb23" +) +ANSWER_METRICS = { + "answer_coverage": ("minimum", 1.0), + "citation_completeness": ("minimum", 1.0), + "claim_citation_precision": ("minimum", 1.0), + "correct_abstention_rate": ("minimum", 1.0), + "fully_grounded_answer_rate": ("minimum", 1.0), + "required_facet_coverage": ("minimum", 1.0), + "unsupported_claim_rate": ("maximum", 0.0), +} +ANSWER_DENOMINATORS = { + "answer_coverage": 4, + "citation_completeness": 7, + "claim_citation_precision": 8, + "correct_abstention_rate": 2, + "fully_grounded_answer_rate": 4, + "required_facet_coverage": 7, + "unsupported_claim_rate": 7, +} +HOLDOUT_SCHEMA_VERSION = "groundmap.holdout.bundle.v1" +HOLDOUT_PUBLIC_BUNDLE_SHA256 = ( + "1b1f4c3d621371b15cbb41bb5e264c5c7c47a981652379eda17650d1aa9f5b17" +) +HOLDOUT_METRICS = { + "answer_coverage": ("minimum", 0.98), + "complete_evidence_set_recall_at_20": ("minimum", 0.98), + "content_section_registration_coverage": ("minimum", 1.0), + "fixed_regression_pass_rate": ("minimum", 1.0), + "forbidden_selected_rate": ("maximum", 0.0), + "fully_grounded_coverage": ("minimum", 0.98), + "natural_unit_index_coverage": ("minimum", 1.0), + "selected_citation_precision": ("minimum", 0.99), + "unanswerable_poison_rejection_at_20": ("minimum", 1.0), +} +HOLDOUT_DENOMINATORS = { + "answer_coverage": 50, + "complete_evidence_set_recall_at_20": 50, + "content_section_registration_coverage": 36, + "fixed_regression_pass_rate": 51, + "forbidden_selected_rate": 60, + "fully_grounded_coverage": 50, + "natural_unit_index_coverage": 744, + "selected_citation_precision": 60, + "unanswerable_poison_rejection_at_20": 1, +} +HOLDOUT_PREREGISTERED_DENOMINATORS = { + "documents": 6, + "natural_units": 744, + "content_sections": 36, + "cases": 51, + "unique_questions": 51, + "answerable_cases": 50, + "unanswerable_cases": 1, + "facets": 60, + "natural_unit_types": { + "blockquote": 25, + "code": 18, + "figure": 13, + "list_item": 84, + "paragraph": 536, + "table_row": 68, + }, + "slice_cases": { + "multi_hop": 10, + "multilingual": 10, + "poison": 51, + "unanswerable": 1, + }, +} +CONVERSION_GOLD_SCHEMA_VERSION = 4 +CONVERSION_GOLD_SHA256 = ( + "2289da6634ff0925f3b13d3c7bc4e3c320c4ee7e8c94a8b39ebe974567f72315" +) +CONVERSION_FIXTURE_SEMANTIC_SHA256 = ( + "82136d6cb2fcb6cadfa7480db28c4031a23386cc0a26f95bd64227e69961db4f" +) +CONVERSION_METRICS = { + name: ("minimum", 1.0) + for name in ( + "critical_fact_preservation", + "footnote_relation_preservation", + "heading_hierarchy_preservation", + "list_structure_preservation", + "mutation_sensitivity", + "qualifier_preservation", + "reading_order", + "section_count_preservation", + "standalone_text_preservation", + "table_alignment", + ) +} +CONVERSION_DENOMINATORS = { + "critical_fact_preservation": 6, + "footnote_relation_preservation": 3, + "heading_hierarchy_preservation": 3, + "list_structure_preservation": 2, + "mutation_sensitivity": 14, + "qualifier_preservation": 6, + "reading_order": 4, + "section_count_preservation": 3, + "standalone_text_preservation": 3, + "table_alignment": 3, +} + + +class Stage2ProtocolError(RuntimeError): + """A child runner did not return a trustworthy structured terminal state.""" + + +def _is_finite_number(value: Any) -> bool: + return ( + not isinstance(value, bool) + and isinstance(value, (int, float)) + and math.isfinite(float(value)) + ) + + +def _metric_values( + payload: dict[str, Any], + expected: dict[str, tuple[str, float]], + expected_denominators: dict[str, int], + errors: list[str], +) -> dict[str, float]: + metrics = payload.get("metrics") + if not isinstance(metrics, dict) or set(metrics) != set(expected): + observed = sorted(metrics) if isinstance(metrics, dict) else type(metrics).__name__ + errors.append(f"metric keys differ: {observed!r}") + return {} + values: dict[str, float] = {} + for name in expected: + row = metrics[name] + if not isinstance(row, dict): + errors.append(f"{name}: metric row is not an object") + continue + numerator = row.get("numerator") + denominator = row.get("denominator") + value = row.get("value") + if ( + isinstance(numerator, bool) + or not isinstance(numerator, int) + or isinstance(denominator, bool) + or not isinstance(denominator, int) + or denominator != expected_denominators[name] + or numerator < 0 + or numerator > denominator + or not _is_finite_number(value) + ): + errors.append( + f"{name}: invalid numerator/denominator/value; " + f"expected denominator {expected_denominators[name]}" + ) + continue + expected_value = numerator / denominator + if not math.isclose(float(value), expected_value, rel_tol=0.0, abs_tol=1e-12): + errors.append( + f"{name}: value {value!r} does not equal {numerator}/{denominator}" + ) + continue + values[name] = float(value) + return values + + +def _comparison_passed(value: float, direction: str, threshold: float) -> bool: + return value <= threshold if direction == "maximum" else value >= threshold + + +def _answer_terminal_contract(payload: dict[str, Any]) -> tuple[bool, list[str]]: + errors: list[str] = [] + if payload.get("schema_version") != ANSWER_SCHEMA_VERSION: + errors.append("answer schema_version differs") + if payload.get("scope") != "public-cc0-protocol-smoke-not-model-capability": + errors.append("answer scope differs") + fixture = payload.get("fixture") + expected_fixture = { + "cases": 6, + "unique_questions": 6, + "answerable_cases": 4, + "unanswerable_cases": 2, + "required_facets": 7, + "public_fixture_sha256": ANSWER_PUBLIC_FIXTURE_SHA256, + "gold_sha256": ANSWER_GOLD_SHA256, + } + if not isinstance(fixture, dict) or any( + fixture.get(key) != value for key, value in expected_fixture.items() + ): + errors.append("answer fixture digest/denominators differ") + values = _metric_values( + payload, ANSWER_METRICS, ANSWER_DENOMINATORS, errors + ) + thresholds = payload.get("thresholds") + if not isinstance(thresholds, dict) or set(thresholds) != set(ANSWER_METRICS): + observed = sorted(thresholds) if isinstance(thresholds, dict) else type(thresholds).__name__ + errors.append(f"answer threshold keys differ: {observed!r}") + thresholds = {} + recomputed: dict[str, bool] = {} + for name, (direction, expected_threshold) in ANSWER_METRICS.items(): + row = thresholds.get(name) + key = "max" if direction == "maximum" else "min" + if not isinstance(row, dict) or set(row) != {key, "passed"}: + errors.append(f"{name}: malformed answer threshold row") + continue + threshold = row.get(key) + if not _is_finite_number(threshold) or float(threshold) != expected_threshold: + errors.append(f"{name}: answer threshold changed") + continue + if name not in values: + continue + expected_pass = _comparison_passed(values[name], direction, expected_threshold) + if row.get("passed") is not expected_pass: + errors.append(f"{name}: answer threshold verdict disagrees with value") + continue + recomputed[name] = expected_pass + all_passed = len(recomputed) == len(ANSWER_METRICS) and all(recomputed.values()) + if payload.get("passed") is not all_passed: + errors.append("answer top-level passed disagrees with thresholds") + expected_status = "protocol-smoke-passed" if all_passed else "threshold-failed" + if payload.get("status") != expected_status: + errors.append("answer status disagrees with thresholds") + return all_passed and not errors, errors + + +def _holdout_terminal_contract(payload: dict[str, Any]) -> tuple[bool, list[str]]: + errors: list[str] = [] + if payload.get("schema_version") != HOLDOUT_SCHEMA_VERSION: + errors.append("holdout schema_version differs") + if payload.get("certification_kind") != "public-smoke": + errors.append("holdout certification_kind is not public-smoke") + if payload.get("is_hidden_certification") is not False: + errors.append("holdout hidden-certification flag is not false") + if payload.get("runner_verified_independence") is not False: + errors.append("holdout independence flag is not false") + if payload.get("runner_version") != "1.0.0": + errors.append("holdout runner_version differs") + if payload.get("bundle_id") != "groundmap-public-holdout-smoke": + errors.append("holdout bundle_id differs") + if payload.get("bundle_version") != "1.0.0": + errors.append("holdout bundle_version differs") + if payload.get("top_k") != 20: + errors.append("holdout top_k differs") + if payload.get("evaluation_scope") != "retrieval-only exact evidence selection": + errors.append("holdout evaluation_scope differs") + if payload.get("certification_statement") != ( + "Public Gold is visible: protocol smoke only, never hidden certification." + ): + errors.append("holdout certification_statement differs") + if payload.get("preregistered_denominators") != HOLDOUT_PREREGISTERED_DENOMINATORS: + errors.append("holdout preregistered denominators differ") + values = _metric_values( + payload, HOLDOUT_METRICS, HOLDOUT_DENOMINATORS, errors + ) + thresholds = payload.get("thresholds") + if not isinstance(thresholds, dict) or set(thresholds) != set(HOLDOUT_METRICS): + observed = sorted(thresholds) if isinstance(thresholds, dict) else type(thresholds).__name__ + errors.append(f"holdout threshold keys differ: {observed!r}") + thresholds = {} + recomputed: dict[str, bool] = {} + for name, (direction, expected_threshold) in HOLDOUT_METRICS.items(): + row = thresholds.get(name) + if not isinstance(row, dict): + errors.append(f"{name}: holdout threshold row is not an object") + continue + if row.get("direction") != direction: + errors.append(f"{name}: holdout threshold direction changed") + continue + threshold = row.get("threshold") + reported_value = row.get("value") + if ( + not _is_finite_number(threshold) + or float(threshold) != expected_threshold + or not _is_finite_number(reported_value) + or name not in values + or not math.isclose( + float(reported_value), values.get(name, math.nan), + rel_tol=0.0, abs_tol=1e-12, + ) + ): + errors.append(f"{name}: holdout threshold/value changed") + continue + expected_pass = _comparison_passed(values[name], direction, expected_threshold) + if row.get("passed") is not expected_pass: + errors.append(f"{name}: holdout threshold verdict disagrees with value") + continue + recomputed[name] = expected_pass + all_passed = len(recomputed) == len(HOLDOUT_METRICS) and all(recomputed.values()) + failed_case_ids = payload.get("failed_case_ids") + fixed_regression_value = values.get("fixed_regression_pass_rate") + if not isinstance(failed_case_ids, list) or any( + not isinstance(case_id, str) or not case_id for case_id in failed_case_ids + ): + errors.append("holdout failed_case_ids is malformed") + elif fixed_regression_value == 1.0 and failed_case_ids: + errors.append("holdout failed_case_ids contradicts fixed regression pass") + elif fixed_regression_value is not None and fixed_regression_value < 1.0 and not failed_case_ids: + errors.append("holdout failed_case_ids omits failed regressions") + if payload.get("passed") is not all_passed: + errors.append("holdout top-level passed disagrees with thresholds") + expected_status = "protocol-smoke-passed" if all_passed else "failed" + if payload.get("status") != expected_status: + errors.append("holdout status disagrees with thresholds") + return all_passed and not errors, errors + + +def _conversion_terminal_contract(payload: dict[str, Any]) -> tuple[bool, list[str]]: + errors: list[str] = [] + if payload.get("gold_schema_version") != CONVERSION_GOLD_SCHEMA_VERSION: + errors.append("conversion Gold schema_version differs") + if payload.get("gold_sha256") != CONVERSION_GOLD_SHA256: + errors.append("conversion Gold digest differs") + if payload.get("fixture_semantic_sha256") != CONVERSION_FIXTURE_SEMANTIC_SHA256: + errors.append("conversion fixture semantic digest differs") + if payload.get("documents") != 3: + errors.append("conversion document denominator differs") + values = _metric_values( + payload, CONVERSION_METRICS, CONVERSION_DENOMINATORS, errors + ) + metrics = payload.get("metrics") if isinstance(payload.get("metrics"), dict) else {} + recomputed: dict[str, bool] = {} + for name, (direction, expected_threshold) in CONVERSION_METRICS.items(): + row = metrics.get(name) + if not isinstance(row, dict) or name not in values: + continue + threshold = row.get("threshold") + if not _is_finite_number(threshold) or float(threshold) != expected_threshold: + errors.append(f"{name}: conversion threshold changed") + continue + expected_pass = _comparison_passed(values[name], direction, expected_threshold) + if row.get("passed") is not expected_pass: + errors.append(f"{name}: conversion verdict disagrees with value") + continue + recomputed[name] = expected_pass + all_passed = len(recomputed) == len(CONVERSION_METRICS) and all(recomputed.values()) + if payload.get("overall_passed") is not all_passed: + errors.append("conversion overall_passed disagrees with metrics") + expected_status = "protocol-smoke-passed" if all_passed else "protocol-smoke-failed" + if payload.get("status") != expected_status: + errors.append("conversion status disagrees with metrics") + return all_passed and not errors, errors + + +CHILD_CONTRACTS = { + "holdout_public_smoke": _holdout_terminal_contract, + "answer_citation_public_smoke": _answer_terminal_contract, + "conversion_fidelity_synthetic": _conversion_terminal_contract, +} + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _parse_terminal_json(stdout: str, label: str) -> dict[str, Any]: + whole = stdout.strip() + for prefix in KNOWN_PREFIXES: + if whole.startswith(prefix): + whole = whole[len(prefix):] + break + if whole: + try: + value = json.loads(whole) + except json.JSONDecodeError: + pass + else: + if isinstance(value, dict): + return value + for raw_line in reversed(stdout.splitlines()): + line = raw_line.strip() + if not line: + continue + for prefix in KNOWN_PREFIXES: + if line.startswith(prefix): + line = line[len(prefix):] + break + try: + value = json.loads(line) + except json.JSONDecodeError: + continue + if isinstance(value, dict): + return value + raise Stage2ProtocolError(f"{label} did not emit a structured JSON terminal state") + + +def _run(label: str, command: list[str]) -> dict[str, Any]: + completed = subprocess.run( + command, + cwd=ROOT, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + check=False, + ) + try: + payload = _parse_terminal_json(completed.stdout, label) + except Stage2ProtocolError as exc: + detail = (completed.stderr or completed.stdout).strip()[-1200:] + raise Stage2ProtocolError(f"{exc}; exit={completed.returncode}; {detail}") from exc + if completed.returncode not in (0, 1, 2): + raise Stage2ProtocolError( + f"{label} returned unsupported exit code {completed.returncode}" + ) + return { + "exit_code": completed.returncode, + "payload": payload, + "stderr": completed.stderr.strip(), + } + + +def run_stage2(*, run_id: str, seed: int, details: bool = False) -> dict[str, Any]: + if not run_id.strip(): + raise Stage2ProtocolError("run_id must be non-empty") + if isinstance(seed, bool) or not isinstance(seed, int): + raise Stage2ProtocolError("seed must be an integer") + + with tempfile.TemporaryDirectory(prefix="groundmap-stage2-") as tmp: + bundle_path = Path(tmp) / "public-holdout-smoke.json" + generated = _run( + "holdout fixture generator", + [ + sys.executable, + str(ROOT / "evals" / "holdout" / "generate_public_smoke.py"), + "--output", + str(bundle_path), + ], + ) + if generated["exit_code"] != 0 or not bundle_path.is_file(): + raise Stage2ProtocolError("public holdout fixture generation failed") + bundle_digest = _sha256(bundle_path) + if bundle_digest != HOLDOUT_PUBLIC_BUNDLE_SHA256: + raise Stage2ProtocolError( + "public holdout bundle drifted without Stage-2 contract review" + ) + + holdout_command = [ + sys.executable, + str(ROOT / "evals" / "holdout" / "run_eval.py"), + "--bundle", + str(bundle_path), + "--bundle-sha256", + bundle_digest, + "--run-id", + run_id, + "--seed", + str(seed), + "--json", + ] + answer_command = [ + sys.executable, + str(ROOT / "evals" / "answer_citation" / "run_eval.py"), + "--json", + ] + conversion_command = [ + sys.executable, + str(ROOT / "evals" / "conversion_fidelity" / "run_eval.py"), + "--json", + ] + if details: + holdout_command.append("--details") + answer_command.append("--details") + conversion_command.append("--details") + + runs = { + "holdout_public_smoke": _run("holdout public smoke", holdout_command), + "answer_citation_public_smoke": _run( + "answer/citation public smoke", answer_command + ), + "conversion_fidelity_synthetic": _run( + "conversion fidelity", conversion_command + ), + } + + terminal_contract_errors: list[str] = [] + for name, row in runs.items(): + if row["exit_code"] == 2: + continue + child_passed, contract_errors = CHILD_CONTRACTS[name](row["payload"]) + if ( + name == "holdout_public_smoke" + and row["payload"].get("bundle_sha256") != bundle_digest + ): + contract_errors.append("holdout echoed bundle digest differs") + terminal_contract_errors.extend( + f"{name}: {error}" for error in contract_errors + ) + if (row["exit_code"] == 0) != child_passed: + terminal_contract_errors.append( + f"{name}: exit {row['exit_code']} disagrees with recomputed pass state" + ) + protocol_error = ( + any(row["exit_code"] == 2 for row in runs.values()) + or bool(terminal_contract_errors) + ) + threshold_failed = any(row["exit_code"] == 1 for row in runs.values()) + passed = not protocol_error and not threshold_failed + if protocol_error: + status = "protocol-error" + elif threshold_failed: + status = "threshold-failed" + else: + status = "public-stage2-gates-passed" + + return { + "status": status, + "passed": passed, + "evaluation_scope": ( + "public CC0 protocol smoke plus synthetic PDF/DOCX/HTML conversion " + "fixtures; not an external hidden certification" + ), + "hidden_certification": False, + "not_measured": [ + "independent hidden-distribution accuracy", + "open-world 100% correctness", + "production-model answer quality unless external predictions are supplied", + ], + "rotation": { + "run_id": run_id, + "seed": seed, + "independent_distribution": False, + }, + "runs": { + name: { + "exit_code": row["exit_code"], + "result": row["payload"], + **({"stderr": row["stderr"]} if details and row["stderr"] else {}), + } + for name, row in runs.items() + }, + "terminal_contract_errors": terminal_contract_errors, + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--run-id", default="public-stage2-smoke-v1") + parser.add_argument("--seed", type=int, default=20260712) + parser.add_argument("--details", action="store_true") + parser.add_argument("--json", action="store_true") + args = parser.parse_args(argv) + try: + result = run_stage2(run_id=args.run_id, seed=args.seed, details=args.details) + except (OSError, Stage2ProtocolError) as exc: + result = { + "status": "protocol-error", + "passed": False, + "hidden_certification": False, + "error": str(exc), + } + if args.json: + print(RESULT_PREFIX + json.dumps(result, ensure_ascii=False, sort_keys=True)) + else: + print(f"Stage-2 protocol error: {exc}", file=sys.stderr) + return 2 + + if args.json: + print(RESULT_PREFIX + json.dumps(result, ensure_ascii=False, sort_keys=True)) + else: + print("GroundMap citation-accuracy Stage-2 public gates") + for name, row in result["runs"].items(): + child = row["result"] + child_status = child.get("status", child.get("passed", "unknown")) + print(f" {name}: exit={row['exit_code']} status={child_status}") + print(f"Overall: {result['status']}") + print("Scope: public/synthetic gates only; this is not hidden certification.") + if result["status"] == "protocol-error": + return 2 + return 0 if result["passed"] else 1 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/requirements-dev.txt b/requirements-dev.txt index 5b5a5b3..a10622b 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -1,3 +1,5 @@ -r requirements.txt pytest markdown # 仅用于 scripts/build_tutorial_html.py 把新手教程渲染成 HTML +python-docx # conversion_fidelity：生成可复现的 DOCX 源 fixture +reportlab # conversion_fidelity：生成可复现的 PDF 源 fixture diff --git a/scripts/conversion_receipt.py b/scripts/conversion_receipt.py new file mode 100644 index 0000000..fb106d2 --- /dev/null +++ b/scripts/conversion_receipt.py @@ -0,0 +1,709 @@ +"""Pure conversion-receipt contract shared by conversion and retrieval. + +This module deliberately has no converter-runtime imports. In particular it +does not import :mod:`convert`, MarkItDown, or the PDF extractor. It can +therefore be used by evidence-index health checks even when optional conversion +dependencies are unavailable. + +The fingerprint implementation is the sole schema-v2 writer/verifier contract +used by both ``convert.py`` and the evidence index. Its implementation bytes +are themselves fingerprinted, so changing receipt semantics invalidates older +derivatives instead of silently widening acceptance. +""" + +from __future__ import annotations + +import ast +from dataclasses import dataclass +import hashlib +import importlib.metadata +import json +import os +import platform +from pathlib import Path, PurePosixPath +import re +import stat +import unicodedata +from typing import Any, Iterable + + +CONVERSION_RECEIPT_SCHEMA_VERSION = 2 +CONVERSION_PIPELINE_VERSION = 7 + +STANDALONE_IMAGE_EXTENSIONS = frozenset({ + ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".tif", ".webp", +}) +AUDIO_EXTENSIONS = frozenset({".mp3", ".wav"}) +CONVERSION_SOURCE_EXTENSIONS = frozenset({ + ".pdf", ".docx", ".pptx", ".xlsx", ".xls", + ".html", ".htm", ".csv", ".json", ".xml", ".epub", ".msg", + *STANDALONE_IMAGE_EXTENSIONS, + *AUDIO_EXTENSIONS, +}) +SUCCESSFUL_CONVERSION_SOURCE_EXTENSIONS = frozenset( + CONVERSION_SOURCE_EXTENSIONS - STANDALONE_IMAGE_EXTENSIONS - AUDIO_EXTENSIONS +) + +_MARKITDOWN_BASE_DISTRIBUTIONS = ( + "beautifulsoup4", + "charset-normalizer", + "defusedxml", + "magika", + "markdownify", + "requests", +) +_SUFFIX_RUNTIME_DISTRIBUTIONS: dict[str, tuple[str, ...]] = { + ".docx": ("lxml", "mammoth", "cobble"), + ".pptx": ("python-pptx", "lxml", "Pillow", "XlsxWriter"), + ".xlsx": ("pandas", "openpyxl", "numpy", "et_xmlfile"), + ".xls": ("pandas", "xlrd", "numpy"), + ".msg": ("olefile",), +} +_PDF_RUNTIME_DISTRIBUTIONS = ( + "pdfplumber", + "pdfminer.six", + "Pillow", + "pypdfium2", +) +_RECEIPT_FIELDS = ( + "schema_version", + "source_path", + "source_extension", + "source_sha256", + "converter_fingerprint", +) +_HEX64_RE = re.compile(r"[0-9a-f]{64}") +_EXTENSION_RE = re.compile(r"\.[a-z0-9]+") + + +class ConversionReceiptError(RuntimeError): + """A machine-readable failure in conversion provenance.""" + + def __init__( + self, + code: str, + message: str, + details: dict[str, Any] | None = None, + ) -> None: + super().__init__(message) + self.code = code + self.message = message + self.details = details or {} + + def to_dict(self) -> dict[str, Any]: + return { + "code": self.code, + "message": self.message, + "details": self.details, + } + + +@dataclass(frozen=True) +class ConversionBinding: + """Validated source binding for one indexed Markdown document.""" + + document_origin: str + original_source_path: str | None = None + original_source_sha256: str | None = None + converter_fingerprint: str | None = None + conversion_receipt_hash: str | None = None + + def manifest_tuple(self) -> tuple[str | None, ...]: + return ( + self.document_origin, + self.original_source_path, + self.original_source_sha256, + self.converter_fingerprint, + self.conversion_receipt_hash, + ) + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _package_version(distribution: str) -> str: + try: + return importlib.metadata.version(distribution) + except importlib.metadata.PackageNotFoundError as exc: + raise ConversionReceiptError( + "converter-fingerprint-unavailable", + "A required converter distribution has no version metadata", + {"distribution": distribution}, + ) from exc + + +def _pdf_layout_converter_version(scripts_dir: Path) -> int: + """Read the literal PDF version without importing the extractor.""" + + source_path = scripts_dir / "pdf_layout.py" + try: + tree = ast.parse(source_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, SyntaxError) as exc: + raise ConversionReceiptError( + "converter-fingerprint-unavailable", + "The PDF converter version could not be read safely", + {"file": source_path.name, "exception": type(exc).__name__}, + ) from exc + for node in tree.body: + if not isinstance(node, (ast.Assign, ast.AnnAssign)): + continue + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + if not any( + isinstance(target, ast.Name) + and target.id == "PDF_LAYOUT_CONVERTER_VERSION" + for target in targets + ): + continue + value = node.value + if isinstance(value, ast.Constant) and type(value.value) is int: + return value.value + break + raise ConversionReceiptError( + "converter-fingerprint-unavailable", + "PDF_LAYOUT_CONVERTER_VERSION must be a literal integer", + {"file": source_path.name}, + ) + + +def _implementation_hashes(suffix: str, scripts_dir: Path) -> dict[str, str]: + names = [ + "conversion_receipt.py", + "convert.py", + "postprocess.py", + "section_parser.py", + ] + if suffix == ".pdf": + names.append("pdf_layout.py") + try: + return {name: _sha256_file(scripts_dir / name) for name in names} + except OSError as exc: + raise ConversionReceiptError( + "converter-fingerprint-unavailable", + "A converter implementation file could not be hashed", + {"exception": type(exc).__name__}, + ) from exc + + +def current_converter_fingerprint( + source_extension: str, + *, + scripts_dir: Path | None = None, +) -> str: + """Return the current schema-v2 converter fingerprint. + + The descriptor intentionally matches ``convert.py`` byte-for-byte at the + canonical JSON layer while avoiding imports of its optional runtimes. + """ + + suffix = source_extension.lower() + root = (scripts_dir or Path(__file__).resolve().parent).resolve() + python_runtime = { + "implementation": platform.python_implementation(), + "version": platform.python_version(), + } + if suffix in STANDALONE_IMAGE_EXTENSIONS or suffix in AUDIO_EXTENSIONS: + descriptor: dict[str, Any] = { + "backend": "accuracy-first-unsupported-media", + "policy_version": 1, + "source_extension": suffix, + "python_runtime": python_runtime, + "pipeline_version": CONVERSION_PIPELINE_VERSION, + } + elif suffix == ".pdf": + descriptor = { + "backend": "groundmap-pdf-layout", + "backend_version": _pdf_layout_converter_version(root), + "runtime_versions": { + name: _package_version(name) + for name in _PDF_RUNTIME_DISTRIBUTIONS + }, + "python_runtime": python_runtime, + "implementation_sha256": _implementation_hashes(suffix, root), + "pipeline_version": CONVERSION_PIPELINE_VERSION, + } + else: + descriptor = { + "backend": "markitdown", + "backend_version": _package_version("markitdown"), + "runtime_versions": { + name: _package_version(name) + for name in dict.fromkeys([ + *_MARKITDOWN_BASE_DISTRIBUTIONS, + *_SUFFIX_RUNTIME_DISTRIBUTIONS.get(suffix, ()), + ]) + }, + "python_runtime": python_runtime, + "implementation_sha256": _implementation_hashes(suffix, root), + "docx_header_repair_version": 2 if suffix == ".docx" else None, + "pipeline_version": CONVERSION_PIPELINE_VERSION, + } + canonical = json.dumps( + descriptor, ensure_ascii=True, sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def canonical_receipt_digest(receipt: dict[str, Any]) -> str: + """Hash the five schema-v2 payload fields, excluding ``receipt_sha256``.""" + + payload = {key: receipt[key] for key in _RECEIPT_FIELDS} + canonical = json.dumps( + payload, ensure_ascii=True, sort_keys=True, separators=(",", ":") + ) + return hashlib.sha256(canonical.encode("utf-8")).hexdigest() + + +def validate_receipt_integrity( + receipt: object, + *, + require_raw: bool, +) -> dict[str, Any]: + """Validate the closed schema-v2 receipt envelope without touching I/O.""" + + if not isinstance(receipt, dict): + raise ConversionReceiptError( + "conversion-receipt-invalid-type", + "conversion_receipt must be an object", + {"actual_type": type(receipt).__name__}, + ) + required_types = { + "schema_version": int, + "source_path": str, + "source_extension": str, + "source_sha256": str, + "converter_fingerprint": str, + "receipt_sha256": str, + } + if set(receipt) != set(required_types): + raise ConversionReceiptError( + "conversion-receipt-schema-fields-mismatch", + "The conversion receipt fields do not match schema v2", + { + "missing": sorted(set(required_types) - set(receipt)), + "unexpected": sorted(set(receipt) - set(required_types)), + }, + ) + for field, expected_type in required_types.items(): + value = receipt.get(field) + if type(value) is not expected_type: + raise ConversionReceiptError( + "conversion-receipt-invalid-type", + f"{field} has the wrong type", + { + "field": field, + "expected_type": expected_type.__name__, + "actual_type": type(value).__name__, + }, + ) + if receipt["schema_version"] != CONVERSION_RECEIPT_SCHEMA_VERSION: + raise ConversionReceiptError( + "conversion-receipt-schema-mismatch", + "The conversion receipt schema is unsupported", + { + "expected": CONVERSION_RECEIPT_SCHEMA_VERSION, + "actual": receipt["schema_version"], + }, + ) + source_relative = _canonical_relative_source_path( + receipt["source_path"], require_raw=require_raw + ) + source_extension = receipt["source_extension"] + if ( + not _EXTENSION_RE.fullmatch(source_extension) + or source_extension != source_extension.lower() + or source_relative.suffix.lower() != source_extension + or source_extension == ".md" + ): + raise ConversionReceiptError( + "conversion-source-extension-mismatch", + "The declared source extension is invalid or inconsistent", + { + "source_extension": source_extension, + "source_path": source_relative.as_posix(), + }, + ) + if source_extension not in SUCCESSFUL_CONVERSION_SOURCE_EXTENSIONS: + raise ConversionReceiptError( + "conversion-source-extension-unsupported", + "The source extension cannot produce an accuracy-first receipt", + {"source_extension": source_extension}, + ) + for field in ("source_sha256", "converter_fingerprint", "receipt_sha256"): + if _HEX64_RE.fullmatch(receipt[field]) is None: + raise ConversionReceiptError( + "conversion-receipt-invalid-hash", + f"{field} must be a lowercase SHA-256 hex digest", + {"field": field}, + ) + expected_receipt_hash = canonical_receipt_digest(receipt) + if receipt["receipt_sha256"] != expected_receipt_hash: + raise ConversionReceiptError( + "conversion-receipt-digest-mismatch", + "The conversion receipt payload has been modified", + { + "expected": expected_receipt_hash, + "actual": receipt["receipt_sha256"], + }, + ) + return receipt + + +def _canonical_relative_source_path( + value: object, + *, + require_raw: bool = True, +) -> PurePosixPath: + if type(value) is not str: + raise ConversionReceiptError( + "conversion-receipt-invalid-type", + "source_path must be a string", + {"field": "source_path", "actual_type": type(value).__name__}, + ) + raw = value + parts = raw.split("/") + parsed = PurePosixPath(raw) + if ( + not raw + or "\\" in raw + or "\x00" in raw + or raw.startswith("/") + or raw.endswith("/") + or any(part in {"", ".", ".."} for part in parts) + or unicodedata.normalize("NFC", raw) != raw + or parsed.is_absolute() + or not parsed.parts + or (require_raw and parsed.parts[0] != "raw") + ): + raise ConversionReceiptError( + "conversion-source-path-noncanonical", + ( + "source_path must be a canonical POSIX path below raw/" + if require_raw + else "source_path must be a canonical relative POSIX path" + ), + {"source_path": raw, "require_raw": require_raw}, + ) + return parsed + + +def _reject_symlink_components(candidate: Path, raw_root: Path) -> None: + current = raw_root + relative = candidate.relative_to(raw_root) + for part in relative.parts: + current = current / part + try: + if current.is_symlink(): + raise ConversionReceiptError( + "conversion-source-symlink-refused", + "Conversion source symlinks are refused", + {"component": part}, + ) + except OSError as exc: + raise ConversionReceiptError( + "conversion-source-unreadable", + "A conversion source path component could not be inspected", + {"component": part, "exception": type(exc).__name__}, + ) from exc + + +def _stable_regular_file_sha256(path: Path) -> str: + """Hash one regular file and reject replacement/mutation during the read.""" + + flags = os.O_RDONLY + if hasattr(os, "O_NOFOLLOW"): + flags |= os.O_NOFOLLOW + try: + descriptor = os.open(path, flags) + except FileNotFoundError as exc: + raise ConversionReceiptError( + "conversion-source-missing", + "The source declared by the conversion receipt is missing", + ) from exc + except OSError as exc: + raise ConversionReceiptError( + "conversion-source-unreadable", + "The source declared by the conversion receipt cannot be opened", + {"exception": type(exc).__name__}, + ) from exc + try: + before = os.fstat(descriptor) + if not stat.S_ISREG(before.st_mode): + raise ConversionReceiptError( + "conversion-source-not-regular-file", + "The conversion source must be a regular file", + ) + digest = hashlib.sha256() + while True: + chunk = os.read(descriptor, 1024 * 1024) + if not chunk: + break + digest.update(chunk) + after = os.fstat(descriptor) + finally: + os.close(descriptor) + signature = lambda row: ( + row.st_dev, + row.st_ino, + row.st_size, + row.st_mtime_ns, + row.st_ctime_ns, + ) + if signature(before) != signature(after): + raise ConversionReceiptError( + "conversion-source-changed-during-validation", + "The conversion source changed while it was being hashed", + ) + try: + current = path.stat(follow_symlinks=False) + except FileNotFoundError as exc: + raise ConversionReceiptError( + "conversion-source-missing", + "The conversion source disappeared during validation", + ) from exc + except OSError as exc: + raise ConversionReceiptError( + "conversion-source-unreadable", + "The conversion source could not be rechecked after hashing", + {"exception": type(exc).__name__}, + ) from exc + if signature(current) != signature(after): + raise ConversionReceiptError( + "conversion-source-changed-during-validation", + "The source path was replaced while its receipt was being validated", + ) + return digest.hexdigest() + + +def discover_sibling_sources(md_path: Path) -> list[Path]: + """Find non-Markdown sources whose deterministic target is ``md_path``.""" + + try: + entries: Iterable[Path] = md_path.parent.iterdir() + return sorted( + ( + entry + for entry in entries + if entry.suffix.lower() in CONVERSION_SOURCE_EXTENSIONS + and entry.with_suffix(".md").name.casefold() + == md_path.name.casefold() + ), + key=lambda value: value.name.casefold(), + ) + except OSError as exc: + raise ConversionReceiptError( + "conversion-source-discovery-failed", + "Sibling conversion sources could not be enumerated", + {"exception": type(exc).__name__}, + ) from exc + + +def validate_conversion_binding( + md_path: Path, + outline: object, + workspace_root: Path, + *, + markdown_sha256: str, + expected_outline_schema_version: int, +) -> ConversionBinding: + """Validate native/converted ownership and return manifest-ready fields.""" + + workspace = workspace_root.resolve() + raw_root = (workspace / "raw").resolve() + md_resolved = md_path.resolve(strict=True) + try: + md_relative = md_resolved.relative_to(workspace).as_posix() + md_resolved.relative_to(raw_root) + except ValueError as exc: + raise ConversionReceiptError( + "conversion-derived-path-escape", + "The derived Markdown must remain below the workspace raw/ root", + ) from exc + + siblings = discover_sibling_sources(md_resolved) + if len(siblings) > 1: + raise ConversionReceiptError( + "conversion-source-collision", + "Multiple non-Markdown sources map to the same derived Markdown", + {"sources": [item.name for item in siblings]}, + ) + + receipt = outline.get("conversion_receipt") if isinstance(outline, dict) else None + if receipt is None: + if siblings: + raise ConversionReceiptError( + "conversion-receipt-missing", + "A non-Markdown sibling owns this derivative but no receipt is present", + {"source": siblings[0].name}, + ) + return ConversionBinding(document_origin="native_markdown") + receipt = validate_receipt_integrity(receipt, require_raw=True) + source_relative = PurePosixPath(receipt["source_path"]) + source_extension = receipt["source_extension"] + + expected_md = source_relative.with_suffix(".md").as_posix() + if expected_md != md_relative: + raise ConversionReceiptError( + "conversion-derived-path-mismatch", + "source_path does not map to the current derived Markdown", + {"expected": expected_md, "actual": md_relative}, + ) + if not isinstance(outline, dict): # guarded by receipt extraction, for typing + raise AssertionError("receipt-bearing outline must be an object") + if ( + type(outline.get("outline_schema_version")) is not int + or outline.get("outline_schema_version") != expected_outline_schema_version + or outline.get("doc_path") != md_relative + or outline.get("doc_sha256") != markdown_sha256 + or not isinstance(outline.get("sections"), list) + ): + raise ConversionReceiptError( + "conversion-outline-binding-mismatch", + "The receipt-bearing outline does not bind the current Markdown", + { + "doc_path": outline.get("doc_path"), + "expected_doc_path": md_relative, + "doc_sha256": outline.get("doc_sha256"), + "expected_doc_sha256": markdown_sha256, + }, + ) + + source_candidate = workspace.joinpath(*source_relative.parts) + try: + source_candidate.relative_to(raw_root) + except ValueError as exc: + raise ConversionReceiptError( + "conversion-source-path-escape", + "The declared source path escapes raw/", + ) from exc + _reject_symlink_components(source_candidate, raw_root) + try: + source_resolved = source_candidate.resolve(strict=True) + source_resolved.relative_to(raw_root) + except FileNotFoundError as exc: + raise ConversionReceiptError( + "conversion-source-missing", + "The source declared by the conversion receipt is missing", + {"source_path": source_relative.as_posix()}, + ) from exc + except ValueError as exc: + raise ConversionReceiptError( + "conversion-source-path-escape", + "The resolved conversion source escapes raw/", + {"source_path": source_relative.as_posix()}, + ) from exc + except OSError as exc: + raise ConversionReceiptError( + "conversion-source-unreadable", + "The conversion source could not be resolved", + {"exception": type(exc).__name__}, + ) from exc + if siblings and source_resolved != siblings[0].resolve(strict=True): + raise ConversionReceiptError( + "conversion-source-ownership-mismatch", + "The receipt names a different owner than the deterministic sibling source", + { + "declared": source_relative.as_posix(), + "sibling": siblings[0].name, + }, + ) + + actual_source_sha256 = _stable_regular_file_sha256(source_resolved) + if actual_source_sha256 != receipt["source_sha256"]: + raise ConversionReceiptError( + "conversion-source-sha256-mismatch", + "The original source bytes no longer match the conversion receipt", + { + "source_path": source_relative.as_posix(), + "expected": receipt["source_sha256"], + "actual": actual_source_sha256, + }, + ) + expected_fingerprint = current_converter_fingerprint(source_extension) + if receipt["converter_fingerprint"] != expected_fingerprint: + raise ConversionReceiptError( + "conversion-fingerprint-mismatch", + "The conversion implementation or runtime differs from the receipt", + { + "expected": expected_fingerprint, + "actual": receipt["converter_fingerprint"], + }, + ) + return ConversionBinding( + document_origin="converted", + original_source_path=source_relative.as_posix(), + original_source_sha256=receipt["source_sha256"], + converter_fingerprint=receipt["converter_fingerprint"], + conversion_receipt_hash=receipt["receipt_sha256"], + ) + + +def build_conversion_receipt( + source: Path, + workspace_root: Path, + *, + source_sha256: str | None = None, + require_raw: bool = False, +) -> dict[str, Any]: + """Build the shared schema-v2 receipt used by conversion and retrieval.""" + + workspace = workspace_root.resolve() + if source.is_symlink(): + raise ConversionReceiptError( + "conversion-source-symlink-refused", + "Conversion source symlinks are refused", + ) + resolved = source.resolve(strict=True) + try: + relative = resolved.relative_to(workspace) + except ValueError as exc: + raise ConversionReceiptError( + "conversion-source-path-escape", + "The conversion source must be inside the workspace", + ) from exc + source_path = unicodedata.normalize("NFC", relative.as_posix()) + _canonical_relative_source_path(source_path, require_raw=require_raw) + if resolved.suffix.lower() not in SUCCESSFUL_CONVERSION_SOURCE_EXTENSIONS: + raise ConversionReceiptError( + "conversion-source-extension-unsupported", + "The source extension cannot produce an accuracy-first receipt", + {"source_extension": resolved.suffix.lower()}, + ) + digest = source_sha256 or _stable_regular_file_sha256(resolved) + if type(digest) is not str or _HEX64_RE.fullmatch(digest) is None: + raise ConversionReceiptError( + "conversion-receipt-invalid-hash", + "source_sha256 must be a lowercase SHA-256 hex digest", + {"field": "source_sha256"}, + ) + receipt: dict[str, Any] = { + "schema_version": CONVERSION_RECEIPT_SCHEMA_VERSION, + "source_path": source_path, + "source_extension": resolved.suffix.lower(), + "source_sha256": digest, + "converter_fingerprint": current_converter_fingerprint(resolved.suffix.lower()), + } + receipt["receipt_sha256"] = canonical_receipt_digest(receipt) + return receipt + + +__all__ = [ + "AUDIO_EXTENSIONS", + "CONVERSION_PIPELINE_VERSION", + "CONVERSION_RECEIPT_SCHEMA_VERSION", + "CONVERSION_SOURCE_EXTENSIONS", + "SUCCESSFUL_CONVERSION_SOURCE_EXTENSIONS", + "ConversionBinding", + "ConversionReceiptError", + "STANDALONE_IMAGE_EXTENSIONS", + "build_conversion_receipt", + "canonical_receipt_digest", + "current_converter_fingerprint", + "discover_sibling_sources", + "validate_conversion_binding", + "validate_receipt_integrity", +] diff --git a/scripts/convert.py b/scripts/convert.py index 798ccdf..14e59b3 100644 --- a/scripts/convert.py +++ b/scripts/convert.py @@ -16,11 +16,15 @@ """ import argparse +from collections import defaultdict +import hashlib import json import os +import re import sys import tempfile -from pathlib import Path +import unicodedata +from pathlib import Path, PurePosixPath def _atomic_write_text(target: Path, content: str, encoding: str = "utf-8") -> None: @@ -80,7 +84,17 @@ def _atomic_write_text(target: Path, content: str, encoding: str = "utf-8") -> N # 让 postprocess 模块可被 import（与 convert.py 同目录） sys.path.insert(0, str(Path(__file__).resolve().parent)) -from postprocess import has_anchors, process as postprocess_text +from conversion_receipt import ( + AUDIO_EXTENSIONS, + STANDALONE_IMAGE_EXTENSIONS, + ConversionReceiptError, + build_conversion_receipt, + current_converter_fingerprint, + validate_receipt_integrity, +) +from postprocess import process as postprocess_text, validate_outline +from pdf_layout import extract_pdf_to_markdown + SUPPORTED_EXTENSIONS = { # 已是 markdown：仅做 postprocess（加锚点 + 生成 outline） @@ -92,14 +106,313 @@ def _atomic_write_text(target: Path, content: str, encoding: str = "utf-8") -> N # 电子书 ".epub", # 图片 - ".jpg", ".jpeg", ".png", ".gif", ".bmp", ".tiff", ".tif", ".webp", + *STANDALONE_IMAGE_EXTENSIONS, # 音频 - ".mp3", ".wav", + *AUDIO_EXTENSIONS, # 邮件 ".msg", } +_TABLE_DELIMITER_RE = re.compile( + r"^\s*\|(?:\s*:?-{3,}:?\s*\|)+\s*$" +) +_BOLD_RUN_RE = re.compile( + r"(?:\*\*((?:(?!\*\*).)+)\*\*|__((?:(?!__).)+)__)", + re.DOTALL, +) + + +def _converter_fingerprint(source: Path) -> str: + """Compatibility wrapper around the pure shared receipt contract.""" + return current_converter_fingerprint(source.suffix.lower()) + + +def _accuracy_first_backend_rejection(suffix: str) -> str | None: + """Explain media formats that cannot currently earn a stable receipt. + + MarkItDown's standalone image path emits ExifTool metadata, not the visual + content required for evidence-grade ingest. Its audio path uses ambient + ffmpeg/ffprobe for compressed media and SpeechRecognition's unversioned + Google service, so there is no immutable model/backend revision to bind. + Keep the extensions discoverable, but fail before invoking those backends. + """ + if suffix in STANDALONE_IMAGE_EXTENSIONS: + return ( + "accuracy-first conversion refuses standalone images: ExifTool " + "metadata does not preserve visual content; use a dedicated " + "OCR/visual extraction workflow with a verifiable receipt" + ) + if suffix in AUDIO_EXTENSIONS: + return ( + "accuracy-first conversion refuses audio: the current MarkItDown " + "path depends on ambient ffmpeg/ffprobe and an unversioned Google " + "SpeechRecognition backend; use a version-pinned transcription " + "workflow with a verifiable model receipt" + ) + return None + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _conversion_receipt_for_sha( + source: Path, source_sha256: str +) -> dict[str, object]: + base = (_BASE_ROOT or get_project_root()).resolve() + resolved = source.resolve() + try: + resolved.relative_to(base) + except ValueError: + # Preserve the library-call contract outside a configured workspace: + # such a source receives a basename-relative receipt. + base = resolved.parent + return build_conversion_receipt( + source, + base, + source_sha256=source_sha256, + require_raw=False, + ) + + +def _conversion_receipt(source: Path) -> dict[str, object]: + """Build the source/content/converter receipt stored in outline.json.""" + return _conversion_receipt_for_sha(source, _sha256_file(source)) + + +def _source_stat_signature(source: Path) -> tuple[int, int, int, int, int]: + stat = source.stat() + return stat.st_dev, stat.st_ino, stat.st_size, stat.st_mtime_ns, stat.st_ctime_ns + + +def _materialize_conversion_snapshot( + source: Path, directory: Path +) -> tuple[Path, dict[str, object]]: + """Copy a read-only consistency snapshot and bind its exact bytes. + + Converters receive the snapshot, never the live raw path. A stat check plus + a second full live-source hash closes the window where the source changes + while it is being copied. The caller checks the same receipt again before + and after writing derivatives. + """ + before = _source_stat_signature(source) + snapshot = directory / source.name + digest = hashlib.sha256() + with source.open("rb") as source_handle, snapshot.open("xb") as snapshot_handle: + for chunk in iter(lambda: source_handle.read(1024 * 1024), b""): + digest.update(chunk) + snapshot_handle.write(chunk) + after = _source_stat_signature(source) + if before != after: + raise RuntimeError( + f"source changed while conversion snapshot was created: {source.name}" + ) + receipt = _conversion_receipt_for_sha(source, digest.hexdigest()) + if _conversion_receipt(source) != receipt: + raise RuntimeError( + f"source changed while conversion snapshot was verified: {source.name}" + ) + # Best-effort guard for trusted local converters: read-only mode blocks + # accidental writes and the post-conversion SHA catches ordinary mutation. + # This is not an adversarial sandbox; same-user code can change permissions, + # mutate/read, then restore the original bytes before the check. + snapshot.chmod(0o444) + return snapshot, receipt + + +def _assert_source_matches_receipt( + source: Path, expected: dict[str, object], phase: str +) -> None: + try: + current = _conversion_receipt(source) + except Exception as exc: + raise RuntimeError( + f"source became unreadable {phase}: {source.name}" + ) from exc + if current != expected: + raise RuntimeError(f"source changed {phase}: {source.name}") + + +def _convert_non_markdown_source( + md: MarkItDown, source: Path +) -> tuple[str, str, dict[str, object]]: + """Convert a trusted snapshot and return its bound receipt. + + The snapshot/hash checks protect consistency against source races and + accidental converter writes. They do not isolate hostile converter code; + only reviewed, locally trusted converter implementations may run here. + """ + rejection = _accuracy_first_backend_rejection(source.suffix.lower()) + if rejection is not None: + raise RuntimeError(rejection) + with tempfile.TemporaryDirectory(prefix="groundmap-convert-snapshot-") as tmp: + snapshot, receipt = _materialize_conversion_snapshot(source, Path(tmp)) + if source.suffix.lower() == ".pdf": + pdf_result = extract_pdf_to_markdown(snapshot) + markdown = pdf_result.markdown + diagnostics = pdf_result.diagnostics + conversion_detail = ( + f", PDF layout-aware: {diagnostics.page_count} 页/" + f"{len(diagnostics.two_column_pages)} 个分栏页/" + f"{diagnostics.table_count} 表格/" + f"{len(diagnostics.image_pages)} 个含图页需视觉核对" + ) + else: + result = md.convert(str(snapshot)) + markdown = result.markdown if result.markdown else "" + if source.suffix.lower() == ".docx" and markdown: + markdown = _promote_docx_bold_table_headers(markdown) + conversion_detail = "" + try: + snapshot_sha256 = _sha256_file(snapshot) + except Exception as exc: + raise RuntimeError( + f"conversion snapshot became unreadable: {source.name}" + ) from exc + if snapshot_sha256 != receipt["source_sha256"]: + raise RuntimeError( + f"conversion snapshot changed during conversion: {source.name}" + ) + _assert_source_matches_receipt(source, receipt, "during conversion") + return markdown, conversion_detail, receipt + + +def validate_conversion_receipt( + outline: object, source: Path +) -> list[dict[str, object]]: + """Validate the non-Markdown conversion receipt; empty means current. + + The Markdown/outline validator proves only that two derived artifacts agree + with each other. This receipt additionally binds them to the current source + bytes and converter semantics. Missing legacy receipts fail closed and + trigger a one-time re-conversion. + """ + if source.suffix.lower() == ".md": + return [] + rejection = _accuracy_first_backend_rejection(source.suffix.lower()) + if rejection is not None: + return [{ + "code": "conversion-backend-unsupported", + "reason": rejection, + }] + if not isinstance(outline, dict): + return [{"code": "outline-not-object"}] + receipt = outline.get("conversion_receipt") + if not isinstance(receipt, dict): + return [{"code": "conversion-receipt-missing"}] + try: + validate_receipt_integrity(receipt, require_raw=False) + except ConversionReceiptError as exc: + return [{ + "code": "conversion-receipt-invalid", + "issue": exc.to_dict(), + }] + try: + expected = _conversion_receipt(source) + except Exception as exc: + return [{ + "code": "conversion-receipt-source-unreadable", + "exception": type(exc).__name__, + }] + issues: list[dict[str, object]] = [] + for field, expected_value in expected.items(): + actual = receipt.get(field) + if type(actual) is not type(expected_value) or actual != expected_value: + issues.append({ + "code": f"conversion-receipt-{field.replace('_', '-')}-mismatch", + "field": field, + "expected": expected_value, + "actual": actual, + }) + return issues + + +def _pipe_cells(line: str) -> list[str]: + """Split a simple Markdown table row while respecting escaped pipes.""" + value = line.strip() + if not value.startswith("|") or not value.endswith("|"): + return [] + value = value[1:-1] + cells = re.split(r"(? str: + return cell.replace("\\", "\\\\").replace("|", r"\|") + + +def _fully_bold_cell_text(cell: str) -> str | None: + """Return visible text only when every non-space character is bold. + + A single greedy regex incorrectly accepted cells such as + ``**Model** legacy **ID**`` because the outer markers happened to span the + whole string. Parse one or more adjacent Markdown bold runs instead; any + plain text between runs makes the cell ineligible for header promotion. + """ + value = cell.strip() + if not value: + return None + position = 0 + fragments: list[str] = [] + while position < len(value): + match = _BOLD_RUN_RE.match(value, position) + if match is None: + return None + fragment = (match.group(1) or match.group(2) or "").strip() + if not fragment: + return None + fragments.append(fragment) + position = match.end() + while position < len(value) and value[position].isspace(): + position += 1 + return " ".join(fragments) + + +def _promote_docx_bold_table_headers(markdown: str) -> str: + """Repair Mammoth's blank-header DOCX table representation. + + Word tables do not expose a native HTML ``thead`` flag. Mammoth therefore + sometimes emits an empty Markdown header followed by the visually bold + source header as the first body row. That destroys header→row semantics + downstream. Promote only this narrow, high-confidence shape: every + generated header cell is empty and every first body cell is wholly bold. + """ + lines = markdown.splitlines() + index = 0 + while index + 2 < len(lines): + header = _pipe_cells(lines[index]) + delimiter = _pipe_cells(lines[index + 1]) + first_body = _pipe_cells(lines[index + 2]) + if ( + header + and _TABLE_DELIMITER_RE.fullmatch(lines[index + 1]) + and len(header) == len(delimiter) == len(first_body) + and all(not cell for cell in header) + ): + promoted: list[str] = [] + for cell in first_body: + bold_text = _fully_bold_cell_text(cell) + if bold_text is None: + promoted = [] + break + promoted.append(bold_text) + if promoted and all(promoted): + lines[index] = ( + "| " + " | ".join(_escape_pipe_cell(cell) for cell in promoted) + " |" + ) + del lines[index + 2] + index += 2 + continue + index += 1 + trailing_newline = "\n" if markdown.endswith("\n") else "" + return "\n".join(lines) + trailing_newline + + def get_project_root() -> Path: """获取项目根目录（scripts/ 的父目录）""" return Path(__file__).resolve().parent.parent @@ -107,29 +420,49 @@ def get_project_root() -> Path: def should_convert(source: Path, force: bool) -> bool: """判断文件是否需要（重新）处理。 - - .md：检查是否已加锚点 && outline.json 是否存在 - - 其他：检查 .md 是否存在、mtime、outline.json 是否存在 + - 所有格式：派生 Markdown 与 outline 必须通过 schema + 全文/章节 + SHA-256 + 结构校验；任一失效即重处理。 + - 非 .md：outline 内的 conversion receipt 必须同时匹配原文件 + SHA-256 与转换器指纹；旧产物缺 receipt 时自动重转。 + + mtime 不是新鲜度证明，也不参与终判；否则仅 touch 了原文件、 + 但转换结果字节未变时，会因派生 .md 不重写而陷入永久重转。 """ + is_md = source.suffix.lower() == ".md" + if source.is_symlink(): + return True + if is_md: + try: + if _load_derivative_declaration(source) is not None: + # Owned derivatives are never standalone Markdown sources. + return True + except RuntimeError: + return True if force: return True - is_md = source.suffix.lower() == ".md" target_md = source if is_md else source.with_suffix(".md") target_outline = source.with_suffix(".outline.json") if not target_md.exists(): return True if not target_outline.exists(): return True - if not is_md: - # 非 .md：原文件 mtime 新于派生 .md → 重转 - if source.stat().st_mtime > target_md.stat().st_mtime: + + # 先验完整的内容地址化契约，不让 mtime 快路掩盖旧/坏 outline。 + try: + target_text = target_md.read_text(encoding="utf-8") + outline = json.loads(target_outline.read_text(encoding="utf-8")) + if validate_outline( + outline, + target_text, + expected_doc_path=_project_relative_posix(target_md), + ): return True - else: - # .md：检查是否已加锚点 - try: - if not has_anchors(source.read_text(encoding="utf-8")): - return True - except Exception: + if validate_conversion_receipt(outline, source): return True + except Exception: + # 读取、JSON 或 validator 任何故障都 fail-closed 为需要重处理。 + return True + return False @@ -144,8 +477,9 @@ def collect_files(scan_dir: Path, extensions: set[str] | None) -> list[Path]: continue if file_path.name == ".gitkeep": continue - # 跳过 convert.py 自己的派生产物（避免被再次处理） - if file_path.stem.lower().endswith(".outline"): + # 只跳过确切的 outline 派生后缀； + # ``project.outline.pdf`` 仍是合法原始文档。 + if file_path.name.lower().endswith((".outline.json", ".outline.md")): continue files.append(file_path) return files @@ -163,8 +497,201 @@ def _project_relative_posix(path: Path) -> str: try: rel = path.resolve().relative_to(base) except ValueError: - return path.name - return str(rel).replace("\\", "/") + return unicodedata.normalize("NFC", path.name) + return unicodedata.normalize("NFC", str(rel).replace("\\", "/")) + + +def _target_key(path: Path) -> str: + """Cross-platform collision key for a would-be derivative path.""" + return os.path.normcase(str(path.resolve(strict=False))).casefold() + + +def _derivative_targets(source: Path) -> tuple[Path, Path]: + is_md = source.suffix.lower() == ".md" + target_md = source if is_md else source.with_suffix(".md") + return target_md, source.with_suffix(".outline.json") + + +def _receipt_integrity_valid(receipt: object) -> bool: + try: + validate_receipt_integrity(receipt, require_raw=False) + except ConversionReceiptError: + return False + return True + + +def _resolve_receipt_source(target_md: Path, source_path: str) -> Path: + relative = PurePosixPath(source_path) + base = (_BASE_ROOT or get_project_root()).resolve() + candidate = base.joinpath(*relative.parts) + try: + target_md.resolve(strict=False).relative_to(base) + except ValueError: + # Unit tests and external callers may operate outside KB_ROOT. In that + # case a basename-only receipt is relative to its derivative directory. + if len(relative.parts) == 1: + candidate = target_md.parent / relative.name + return candidate + + +def _load_derivative_declaration(target_md: Path) -> tuple[Path, dict[str, object]] | None: + """Return the non-Markdown source declared as owner of ``target_md``. + + A conversion_receipt that exists but cannot satisfy the current ownership + schema is an error, not permission to reinterpret the Markdown as original. + """ + target_outline = target_md.with_suffix(".outline.json") + if not target_outline.exists(): + return None + if target_outline.is_symlink(): + raise RuntimeError( + f"derivative outline symlinks are refused: {target_outline.name}" + ) + try: + outline = json.loads(target_outline.read_text(encoding="utf-8")) + except Exception as exc: + raise RuntimeError( + f"cannot verify derivative ownership from {target_outline.name}" + ) from exc + if not isinstance(outline, dict) or "conversion_receipt" not in outline: + return None + receipt = outline.get("conversion_receipt") + if not _receipt_integrity_valid(receipt): + raise RuntimeError( + f"conversion receipt cannot prove derivative ownership: {target_outline.name}" + ) + assert isinstance(receipt, dict) + source_path = str(receipt["source_path"]) + declared_source = _resolve_receipt_source(target_md, source_path) + if declared_source.suffix.lower() == ".md": + raise RuntimeError( + f"conversion receipt illegally declares Markdown as converter source: " + f"{target_outline.name}" + ) + expected_md, expected_outline = _derivative_targets(declared_source) + if ( + _target_key(expected_md) != _target_key(target_md) + or _target_key(expected_outline) != _target_key(target_outline) + or receipt["source_extension"] != declared_source.suffix.lower() + ): + raise RuntimeError( + f"conversion receipt target/source mapping is inconsistent: " + f"{target_outline.name}" + ) + return declared_source, receipt + + +def _casefold_existing_paths(target: Path) -> list[Path]: + if not target.parent.is_dir(): + return [] + return [ + candidate + for candidate in target.parent.iterdir() + if candidate.name.casefold() == target.name.casefold() + ] + + +def _assert_safe_source(source: Path) -> None: + if source.is_symlink(): + raise RuntimeError( + f"conversion source symlinks are refused: {source.name}" + ) + if not source.is_file(): + raise RuntimeError(f"conversion source is not a regular file: {source}") + + +def _assert_derivative_targets_owned_by(source: Path) -> None: + target_md, target_outline = _derivative_targets(source) + md_matches = _casefold_existing_paths(target_md) + outline_matches = _casefold_existing_paths(target_outline) + for target in [*md_matches, *outline_matches]: + if target.is_symlink(): + raise RuntimeError( + f"derivative target symlinks are refused: {target.name}" + ) + case_variant = [ + target + for target in [*md_matches, *outline_matches] + if target != target_md and target != target_outline + ] + if case_variant: + raise RuntimeError( + f"case-insensitive derivative target collision for {source.name}: " + + ", ".join(path.name for path in case_variant) + ) + md_exists = target_md.exists() + outline_exists = target_outline.exists() + if not md_exists and not outline_exists: + return + if not md_exists or not outline_exists: + raise RuntimeError( + f"existing derivative pair is incomplete; refusing overwrite for {source.name}" + ) + declaration = _load_derivative_declaration(target_md) + if declaration is None: + raise RuntimeError( + f"existing {target_md.name} has no ownership receipt; refusing overwrite" + ) + declared_source, receipt = declaration + if ( + receipt["source_path"] != _project_relative_posix(source) + or declared_source.resolve(strict=False) != source.resolve(strict=False) + ): + raise RuntimeError( + f"existing derivative belongs to {receipt['source_path']}, not {source.name}" + ) + + +def preflight_conversion_sources(files: list[Path]) -> list[Path]: + """Validate an entire batch before any conversion, write, or backend init.""" + errors: list[str] = [] + logical_sources: list[Path] = [] + for source in files: + try: + _assert_safe_source(source) + if source.suffix.lower() == ".md": + declaration = _load_derivative_declaration(source) + if declaration is not None: + declared_source, _receipt = declaration + if declared_source.is_symlink(): + raise RuntimeError( + f"declared converter source is a symlink: {declared_source}" + ) + if not declared_source.is_file(): + raise RuntimeError( + f"orphan derivative {source.name}: declared source " + f"{declared_source} is missing" + ) + # This Markdown is an owned derivative, not a second source. + continue + logical_sources.append(source) + except RuntimeError as exc: + errors.append(str(exc)) + + target_groups: dict[tuple[str, str], list[Path]] = defaultdict(list) + for source in logical_sources: + target_md, target_outline = _derivative_targets(source) + target_groups[(_target_key(target_md), _target_key(target_outline))].append(source) + rejection = _accuracy_first_backend_rejection(source.suffix.lower()) + if rejection is not None: + errors.append(f"{source.name}: {rejection}") + if source.suffix.lower() != ".md": + try: + _assert_derivative_targets_owned_by(source) + except RuntimeError as exc: + errors.append(str(exc)) + for sources in target_groups.values(): + unique = {source.resolve(strict=False) for source in sources} + if len(unique) >= 2: + errors.append( + "multiple sources claim the same Markdown/outline target: " + + ", ".join(sorted(source.name for source in sources)) + ) + if errors: + raise RuntimeError( + "conversion batch preflight failed:\n- " + "\n- ".join(sorted(set(errors))) + ) + return logical_sources def convert_file(md: MarkItDown, source: Path) -> tuple[bool, str]: @@ -172,21 +699,34 @@ def convert_file(md: MarkItDown, source: Path) -> tuple[bool, str]: 转换/处理单个文件。返回 (成功与否, 消息)。 Pipeline: - 1. 非 .md：markitdown 转为 markdown 文本；.md：直接读原文 + 1. PDF：按几何布局恢复表格/分栏阅读顺序；其他非 .md： + markitdown 转为 markdown 文本；.md：直接读原文 2. postprocess.process 加锚点 + 生成 outline 数据 3. 写 .md（仅当内容变化）+ 写 .outline.json """ + _assert_safe_source(source) is_md = source.suffix.lower() == ".md" + source_receipt: dict[str, object] | None = None if is_md: + if _load_derivative_declaration(source) is not None: + raise RuntimeError( + f"owned derivative Markdown cannot be converted as a source: {source.name}" + ) try: markdown = source.read_text(encoding="utf-8") except Exception as e: return False, f"读取失败: {e}" target_md = source + conversion_detail = "" else: - result = md.convert(str(source)) - markdown = result.markdown if result.markdown else "" + _assert_derivative_targets_owned_by(source) + # Convert a read-only consistency snapshot, never the live raw path. This closes + # the TOCTOU gap where Markdown could come from v1 while a post-hoc + # receipt accidentally binds v2. + markdown, conversion_detail, source_receipt = ( + _convert_non_markdown_source(md, source) + ) if not markdown.strip(): return False, "转换结果为空" target_md = source.with_suffix(".md") @@ -194,17 +734,36 @@ def convert_file(md: MarkItDown, source: Path) -> tuple[bool, str]: target_outline = source.with_suffix(".outline.json") doc_path = _project_relative_posix(target_md) - # 读旧 outline（如果存在），让 process 保留 agent_summary + # 读旧 outline 作为摘要迁移候选。全文 hash 变化是正常的 + # re-convert 触发条件，不能因此丢掉其他未变章节的摘要； + # postprocess 会逐节要求 section_sha256 与新正文一致才恢复。 previous_outline = None if target_outline.exists(): try: - previous_outline = json.loads(target_outline.read_text(encoding="utf-8")) + candidate_outline = json.loads( + target_outline.read_text(encoding="utf-8") + ) + if ( + isinstance(candidate_outline, dict) + and isinstance(candidate_outline.get("sections"), list) + ): + previous_outline = candidate_outline except Exception: previous_outline = None text_with_anchors, outline_data = postprocess_text( markdown, doc_path, previous_outline=previous_outline ) + if source_receipt is not None: + _assert_source_matches_receipt(source, source_receipt, "before derivative write") + # Re-check ownership after the potentially long converter call. A new + # native Markdown/outline created during conversion must never be + # overwritten by the stale preflight decision. + _assert_derivative_targets_owned_by(source) + # Bind the derived pair to both the original bytes and the converter + # semantics. Without this field, an old generic/row-interleaved PDF + # derivative could look internally valid and be skipped forever. + outline_data["conversion_receipt"] = source_receipt # 仅当内容变化时写 .md（保护 git 工作树） md_changed = ( @@ -218,12 +777,18 @@ def convert_file(md: MarkItDown, source: Path) -> tuple[bool, str]: target_outline, json.dumps(outline_data, ensure_ascii=False, indent=2), ) + if source_receipt is not None: + # A concurrent edit after the pre-write check leaves a deliberately + # stale receipt, so should_convert will retry; this call still fails and + # cannot report the batch as successful. + _assert_source_matches_receipt(source, source_receipt, "during derivative write") sec_count = sum(_count_sections(s) for s in outline_data["sections"]) md_msg = "新增" if md_changed else "未变" return True, ( f"-> {target_md.name} ({md_msg}, {len(text_with_anchors)} 字符, " - f"{sec_count} 章节, {outline_data['doc_paragraphs']} 段)" + f"{sec_count} 章节, {outline_data['doc_paragraphs']} 段" + f"{conversion_detail})" ) @@ -243,9 +808,35 @@ def parse_extensions(ext_str: str) -> set[str]: return exts +def _doc_path_base_for_explicit_dir(scan_dir: Path, data_root: Path) -> Path: + """Choose the canonical ``outline.doc_path`` base for ``--dir``. + + ``k.py --workspace `` addresses files relative to + ``workspaces//`` (for example ``raw/papers/a.md``). An explicit + ``--dir`` inside that workspace must use the same base; otherwise a valid + outline is rejected as stale solely because it says + ``workspaces//raw/...``. Explicit directories elsewhere under + ``KB_ROOT`` retain the historical data-root-relative semantics. + """ + scan_dir = scan_dir.resolve() + data_root = data_root.resolve() + workspaces_root = (data_root / "workspaces").resolve() + try: + relative = scan_dir.relative_to(workspaces_root) + except ValueError: + return data_root + if not relative.parts: + return data_root + workspace_root = workspaces_root / relative.parts[0] + return workspace_root if workspace_root.is_dir() else data_root + + def main(): parser = argparse.ArgumentParser( - description="将 raw/ 目录中的文档批量转换为 Markdown（基于 markitdown）" + description=( + "将 raw/ 目录中的文档批量转换为 Markdown" + "（PDF 按布局抽取，其他格式基于 markitdown）" + ) ) parser.add_argument( "--dir", @@ -287,9 +878,11 @@ def main(): _kb_root_env = os.environ.get("KB_ROOT") data_root = Path(_kb_root_env).expanduser().resolve() if _kb_root_env else get_project_root() if args.dir: - # 显式 --dir：doc_path 以数据根为基准 + # 显式 --dir：落在 workspaces// 内时以该 workspace 为 + # doc_path 基准（与 k.py 的 raw/... 口径一致）；其他合法 + # 目录保持以数据根为基准的旧语义。 scan_dir = Path(args.dir).resolve() - _BASE_ROOT = data_root + _BASE_ROOT = _doc_path_base_for_explicit_dir(scan_dir, data_root) else: # 解析 workspace（对齐 k.py：workspaces/，挡 ../ 穿越与不存在的名字） workspaces_root = (data_root / "workspaces").resolve() @@ -332,6 +925,18 @@ def main(): print(f"未找到待转换的文件（目录: {scan_dir}）") return + # Whole-batch atomic preflight: reject media backends, symlinks, orphan + # derivatives, and target collisions before should_convert, writes, or even + # constructing MarkItDown (whose plugins/backends may have side effects). + try: + files = preflight_conversion_sources(files) + except RuntimeError as exc: + print(f"错误: {exc}") + raise SystemExit(1) from exc + if not files: + print("未找到独立源文件（扫描结果仅含已绑定派生物）") + return + # 筛选需要转换的文件 to_convert = [] skipped_uptodate = 0 @@ -401,6 +1006,10 @@ def main(): print(' python scripts/k.py annotate-section "<一两句概括>"') print(" ↑ ②③ 档（分段阅读）的必经步骤；① 档短文不强制（详见 CLAUDE.md Ingest 操作流程 / docs/raw-to-wiki-流程.md §3.6）") + # 批处理的任一转换失败/空输出都不得用 exit 0 伪装整批成功。 + if fail_count or empty_count: + raise SystemExit(1) + if __name__ == "__main__": main() diff --git a/scripts/hooks/pre-commit b/scripts/hooks/pre-commit index 8ccd9d9..e55f75d 100644 --- a/scripts/hooks/pre-commit +++ b/scripts/hooks/pre-commit @@ -27,11 +27,456 @@ set -e # 行首引号使 ^ 锚定正则失配——中文知识库的常见文件名会直接穿透保护。 forbidden=$( git -c core.quotepath=false diff --cached --name-only \ - | grep -E '^(raw|my_thoughts)/|^workspaces/[^/]+/(raw|my_thoughts)/' \ + | grep -E '^(raw|my_thoughts)/|(^|/)workspaces/[^/]+/(raw|my_thoughts)/' \ | grep -vE '(^|/)\.gitkeep$' \ || true ) +# ── 引用核对机械闸门 ────────────────────────────────────────── +# staged 改动含 /workspaces//wiki/**.md 时，把 Git index 安全物化 +# 到临时树，再从该树执行 k.py。这保证核对的是「将要 commit 的 blob」， +# 而不是可能已被二次改写的工作树。未入 Git 的 raw 转换产物以只读链接挂载回 +# 临时 workspace，否则 release 布局的 .gitkeep 会把本地可核验原文错降级为 unverifiable。 +# +# 闸门范围只限 staged 页：引用错配/挂偏/不可核验/非 canonical、broken/bare/ +# coarse/source 结构问题、非 SUPPORTED 语义审计对、缺当前内容版本检索凭证均拒绝。 +# 存量债务不拦无关提交。Python、临时树、staged k.py、checker 运行或 JSON +# schema 任一失败都 fail-closed；不再用 [] 把工具故障伪装成绿灯。 +staged_wiki=$( + git -c core.quotepath=false diff --cached --name-only \ + | grep -E '(^|/)workspaces/[^/]+/wiki/.+\.md$' || true +) +if [ -n "$staged_wiki" ]; then + repo_root="$(git rev-parse --show-toplevel)" + + # 只在确有 wiki staged 变更时需要 Python；普通代码/文档提交不增加环境依赖。 + if [ -n "${PYTHON:-}" ]; then + python_bin="$PYTHON" + elif command -v python >/dev/null 2>&1; then + python_bin="$(command -v python)" + elif command -v python3 >/dev/null 2>&1; then + python_bin="$(command -v python3)" + else + echo "" + echo "❌ 拒绝提交：存在 staged wiki 变更，但找不到 Python，引用校验无法执行（fail-closed）。" + echo "" + exit 1 + fi + if ! command -v "$python_bin" >/dev/null 2>&1; then + echo "" + echo "❌ 拒绝提交：PYTHON=$python_bin 不可执行，引用校验无法运行（fail-closed）。" + echo "" + exit 1 + fi + + citation_tmp="$(mktemp -d "${TMPDIR:-/tmp}/groundmap-precommit.XXXXXX")" || { + echo "❌ 拒绝提交：无法创建引用校验临时树（fail-closed）。" + exit 1 + } + cleanup_citation_tmp() { + rm -rf -- "$citation_tmp" + } + trap cleanup_citation_tmp EXIT + + citation_index="$citation_tmp/index" + staged_paths_file="$citation_tmp/staged-paths" + mkdir -p "$citation_index" + git -c core.quotepath=false diff --cached --name-only -z > "$staged_paths_file" + if ! git checkout-index --all --force --prefix="$citation_index/"; then + echo "" + echo "❌ 拒绝提交：无法物化 Git index，引用校验未执行（fail-closed）。" + echo "" + exit 1 + fi + + # 驱动器对 NUL 分隔的 staged 路径分组，因此不会把中文/空格文件名拼进 + # Python 源码。LIST_CHECKERS 是后续增加 list 型 strict lint 的单一接入点， + # 语义审计/provenance 则由下方两个有类型的 result-object adapter 负责。 + "$python_bin" - "$citation_index" "$repo_root" "$staged_paths_file" <<'PY' +from __future__ import annotations + +import json +import os +from pathlib import Path, PurePosixPath +import re +import shutil +import subprocess +import sys +from typing import NoReturn + + +INDEX_ROOT = Path(sys.argv[1]) +REPO_ROOT = Path(sys.argv[2]) +STAGED_PATHS_FILE = Path(sys.argv[3]) +WIKI_PATH_RE = re.compile( + r"^(?:(?P.+)/)?workspaces/(?P[^/]+)/" + r"wiki/(?P.+\.md)$" +) + +# list 型 checker 共享 staged-file 过滤。path_field / issue_field 显式适配 +# 各 CLI 的真实 schema；gate_issues=None 表示该 checker 的所有 finding 都拦截。 +LIST_CHECKERS = ( + { + "name": "list-cite-mismatches", + "args": ("list-cite-mismatches", "--json"), + "path_field": "path", + "issue_field": "issue", + "issue_label": None, + "gate_issues": frozenset(( + "mismatch", "exempt-missing-basis", "imprecise-anchor", "unverifiable", + "canonical-anchor-mismatch", "canonical-target-mismatch", + )), + "required": (("line", int),), + }, + { + "name": "list-broken-refs", + "args": ("list-broken-refs", "--json"), + "path_field": "from_path", + "issue_field": None, + "issue_label": "broken-ref", + "gate_issues": None, + "required": (("line", int), ("reason", str)), + }, + { + "name": "list-bare-claims", + "args": ("list-bare-claims", "--json"), + "path_field": "path", + "issue_field": None, + "issue_label": "bare-claim", + "gate_issues": None, + "required": (("line", int), ("matched", list)), + }, + { + "name": "list-coarse-citations", + "args": ("list-coarse-citations", "--json"), + "path_field": "path", + "issue_field": None, + "issue_label": "coarse-citation", + "gate_issues": None, + "required": (("line", int), ("matched", list)), + }, + { + "name": "list-unmapped-claims", + "args": ("list-unmapped-claims", "--json"), + "path_field": "path", + "issue_field": "issue", + "issue_label": None, + "gate_issues": None, + "required": ( + ("page", str), ("line", int), ("claim_text", str), + ("kind", str), ("detail", str), + ), + }, + { + "name": "list-source-issues", + "args": ("list-source-issues", "--json"), + "path_field": "path", + "issue_field": "issue_type", + "issue_label": None, + "gate_issues": None, + "required": (), + }, +) + + +def fail(message: str) -> NoReturn: + print("", file=sys.stderr) + print( + "❌ 拒绝提交：引用校验失败（fail-closed）——" + message, + file=sys.stderr, + ) + print("", file=sys.stderr) + raise SystemExit(2) + + +def safe_parts(path: str) -> tuple[str, ...]: + p = PurePosixPath(path) + if p.is_absolute() or not p.parts or any(part in ("", ".", "..") for part in p.parts): + raise ValueError(f"不安全的相对路径: {path!r}") + return p.parts + + +def replace_with_raw_link(destination: Path, source: Path) -> None: + """临时树仅挂载工作区 raw；不写 source，清理时也只删 symlink。""" + if not source.is_dir(): + return + if destination.is_symlink() or destination.is_file(): + destination.unlink() + elif destination.exists(): + shutil.rmtree(destination) + destination.parent.mkdir(parents=True, exist_ok=True) + destination.symlink_to(source, target_is_directory=True) + + +def materialize_cache_snapshot(destination: Path, source: Path) -> None: + """把 strict 门禁所需的两份台账拷入临时 workspace。 + + 不直接 symlink 整个 .cache：即使 staged k.py 意外写缓存，也只会改临时快照， + 不会修改用户的 citation_audit / retrieval_log 真实台账。 + """ + if destination.is_symlink() or destination.is_file(): + destination.unlink() + elif destination.exists(): + shutil.rmtree(destination) + destination.mkdir(parents=True, exist_ok=True) + if not source.is_dir(): + return + for name in ("citation_audit.jsonl", "retrieval_log.jsonl"): + src = source / name + if src.is_file(): + # read_bytes/write_bytes 对 symlink 来源也只拷内容，不把链接带入临时树。 + (destination / name).write_bytes(src.read_bytes()) + + +def run_json( + k_script: Path, + workspace: str, + args: tuple[str, ...] | list[str], + env: dict[str, str], + *, + allowed_returncodes: frozenset[int] = frozenset((0,)), +) -> tuple[object, int]: + name = args[0] + proc = subprocess.run( + [sys.executable, str(k_script), "--workspace", workspace, *args], + cwd=str(INDEX_ROOT), + env=env, + capture_output=True, + text=True, + ) + if proc.returncode not in allowed_returncodes: + detail = (proc.stderr or proc.stdout).strip().replace("\n", " ")[:500] + fail(f"{name} 退出码 {proc.returncode}" + (f"：{detail}" if detail else "")) + try: + return json.loads(proc.stdout), proc.returncode + except json.JSONDecodeError as exc: + fail(f"{name} 输出非法 JSON：{exc}") + + +try: + raw_names = STAGED_PATHS_FILE.read_bytes().split(b"\0") + staged_paths = {name.decode("utf-8") for name in raw_names if name} + + # key=(data-root prefix, workspace), value=该 workspace 本次真正存在于 index 的 md。 + jobs: dict[tuple[str, str], set[str]] = {} + for rel in staged_paths: + match = WIKI_PATH_RE.fullmatch(rel) + if match is None: + continue + safe_parts(rel) + # staged 删除没有可核对的页面 blob，不把工作树残留误当提交内容。 + if not (INDEX_ROOT.joinpath(*safe_parts(rel))).is_file(): + continue + prefix = match.group("prefix") or "" + jobs.setdefault((prefix, match.group("workspace")), set()).add(rel) + + if not jobs: + raise SystemExit(0) + + k_script = INDEX_ROOT / "scripts" / "k.py" + if not k_script.is_file(): + fail("staged tree 中缺少 scripts/k.py") + + all_gate_findings: list[dict] = [] + for (prefix, workspace), staged_for_job in sorted(jobs.items()): + prefix_parts = safe_parts(prefix) if prefix else () + temp_data_root = INDEX_ROOT.joinpath(*prefix_parts) + actual_data_root = REPO_ROOT.joinpath(*prefix_parts) + temp_workspace = temp_data_root / "workspaces" / workspace + if not temp_workspace.is_dir(): + fail(f"staged workspace 物化不完整: {prefix or '.'}/workspaces/{workspace}") + + # raw/**/*.md 按规范不入 index；在临时树链接本地只读事实层， + # 但 wiki 及被引 wiki source 仍完全来自 staged index。 + replace_with_raw_link( + temp_workspace / "raw", + actual_data_root / "workspaces" / workspace / "raw", + ) + materialize_cache_snapshot( + temp_workspace / ".cache", + actual_data_root / "workspaces" / workspace / ".cache", + ) + + env = os.environ.copy() + env["KB_ROOT"] = str(temp_data_root) + staged_local_paths: set[str] = set() + for staged_path in staged_for_job: + match = WIKI_PATH_RE.fullmatch(staged_path) + if match is None: + fail(f"内部 staged 路径分组失配: {staged_path}") + staged_local_paths.add(f"wiki/{match.group('wiki_path')}") + + # 1) 确定性引用 + 结构 lint。所有命令扫完整 workspace，但只把 + # finding 的来源页在 staged_local_paths 内的项作为本次提交闸门。 + for checker in LIST_CHECKERS: + items, _returncode = run_json( + k_script, workspace, checker["args"], env + ) + if not isinstance(items, list): + fail(f"{checker['name']} JSON 顶层必须是 list") + + for item in items: + if not isinstance(item, dict): + fail(f"{checker['name']} finding 必须是 object") + path_field = checker["path_field"] + item_path = item.get(path_field) + if not isinstance(item_path, str): + fail(f"{checker['name']} finding 缺少字符串 {path_field}") + item_path = PurePosixPath(*safe_parts(item_path)).as_posix() + for field, expected_type in checker["required"]: + if not isinstance(item.get(field), expected_type): + fail( + f"{checker['name']} finding 字段 {field} 必须是 " + f"{expected_type.__name__}" + ) + issue_field = checker["issue_field"] + if issue_field is None: + issue = checker["issue_label"] + else: + issue = item.get(issue_field) + if not isinstance(issue, str): + fail(f"{checker['name']} finding 缺少字符串 {issue_field}") + gate_issues = checker["gate_issues"] + should_gate = gate_issues is None or issue in gate_issues + if should_gate and item_path in staged_local_paths: + detail = item.get("reason") or item.get("detail") or "" + all_gate_findings.append({ + "workspace": workspace, + "checker": checker["name"], + "path": item_path, + "line": item.get("line", "?"), + "issue": issue, + "detail": str(detail), + }) + + sorted_paths = sorted(staged_local_paths) + + # 2) 语义审计闸门：每个 staged claim→evidence pair 必须能精确解析， + # 且当前 claim/target hash 对应的台账 verdict 必须是 SUPPORTED。 + claim_data, _returncode = run_json( + k_script, + workspace, + ("extract-claims", "--json", "--paths", *sorted_paths), + env, + ) + if not isinstance(claim_data, dict): + fail("extract-claims JSON 顶层必须是 object") + pairs = claim_data.get("pairs") + summary = claim_data.get("summary") + if not isinstance(pairs, list) or not isinstance(summary, dict): + fail("extract-claims JSON 缺少 pairs list / summary object") + if not isinstance(summary.get("returned"), int) or summary["returned"] != len(pairs): + fail("extract-claims summary.returned 与 pairs 长度不一致") + for pair in pairs: + if not isinstance(pair, dict): + fail("extract-claims pair 必须是 object") + required_pair_fields = ( + ("pair_id", str), ("page", str), ("line", int), ("target", str), + ("anchor", str), ("target_status", str), ("audited", bool), + ) + for field, expected_type in required_pair_fields: + if not isinstance(pair.get(field), expected_type): + fail(f"extract-claims pair 字段 {field} 必须是 {expected_type.__name__}") + verdict = pair.get("last_verdict") + if verdict is not None and not isinstance(verdict, str): + fail("extract-claims pair.last_verdict 必须是 string/null") + page_path = PurePosixPath(*safe_parts(pair["page"])).as_posix() + if page_path not in staged_local_paths: + fail(f"extract-claims --paths 返回了非 staged 页: {page_path}") + + if pair["target_status"] != "ok": + all_gate_findings.append({ + "workspace": workspace, + "checker": "extract-claims", + "path": page_path, + "line": pair["line"], + "issue": f"target-status:{pair['target_status']}", + "detail": f"[[{pair['target']}#^{pair['anchor']}]] 不可精确核验", + }) + elif pair["audited"] is not True: + all_gate_findings.append({ + "workspace": workspace, + "checker": "extract-claims", + "path": page_path, + "line": pair["line"], + "issue": "semantic-unaudited", + "detail": f"pair {pair['pair_id']} unaudited", + }) + elif verdict != "SUPPORTED": + all_gate_findings.append({ + "workspace": workspace, + "checker": "extract-claims", + "path": page_path, + "line": pair["line"], + "issue": "semantic-verdict", + "detail": f"pair {pair['pair_id']} verdict={verdict!r}（必须为 SUPPORTED）", + }) + + # 3) quote-first 凭证闸门。check-provenance 有 finding 时按 CLI 契约退出 1； + # 因此允许 0/1 后仍强制解析 JSON，其他退出码一律是 checker 故障。 + provenance, provenance_rc = run_json( + k_script, + workspace, + ("check-provenance", "--json", "--paths", *sorted_paths), + env, + allowed_returncodes=frozenset((0, 1)), + ) + if not isinstance(provenance, dict): + fail("check-provenance JSON 顶层必须是 object") + checked = provenance.get("checked") + provenance_findings = provenance.get("findings") + if not isinstance(checked, int) or checked < 0 or not isinstance(provenance_findings, list): + fail("check-provenance JSON 缺少非负 checked int / findings list") + if provenance_rc == 1 and not provenance_findings: + fail("check-provenance 退出 1 但 findings 为空") + for finding in provenance_findings: + if not isinstance(finding, dict): + fail("check-provenance finding 必须是 object") + provenance_required = ( + ("page", str), ("line", int), ("target", str), + ("anchor", str), ("issue", str), + ) + for field, expected_type in provenance_required: + if not isinstance(finding.get(field), expected_type): + fail( + f"check-provenance finding 字段 {field} 必须是 " + f"{expected_type.__name__}" + ) + page_path = PurePosixPath(*safe_parts(finding["page"])).as_posix() + if page_path not in staged_local_paths: + fail(f"check-provenance --paths 返回了非 staged 页: {page_path}") + all_gate_findings.append({ + "workspace": workspace, + "checker": "check-provenance", + "path": page_path, + "line": finding["line"], + "issue": finding["issue"], + "detail": f"[[{finding['target']}#^{finding['anchor']}]]", + }) + + if all_gate_findings: + print("") + print("❌ 拒绝提交：staged blob 存在引用核对闸门项：") + print("") + for item in all_gate_findings: + suffix = f" — {item['detail']}" if item["detail"] else "" + print( + f" [{item['workspace']}] {item['path']}:{item['line']} " + f"{item['issue']} ({item['checker']}){suffix}" + ) + print("") + print("修复：按 finding 运行对应 k.py checker，补精确锚点/来源/审计/检索凭证后重新 stage。") + print("公开 clone 缺 raw/.cache 时会按 accuracy-first 拒绝 wiki commit，请在可核验环境完成。") + print("人类如确需带闸门项提交：git commit --no-verify（并以 'human:' 开头写明缘由）") + print("") + raise SystemExit(1) +except SystemExit: + raise +except Exception as exc: + fail(f"{type(exc).__name__}: {exc}") +PY +fi + if [ -z "$forbidden" ]; then exit 0 fi diff --git a/scripts/k.py b/scripts/k.py index be8063b..e968457 100644 --- a/scripts/k.py +++ b/scripts/k.py @@ -13,11 +13,14 @@ """ import argparse +import hashlib import json import os import re +import subprocess import sys import tempfile +import unicodedata from collections import namedtuple from dataclasses import asdict, dataclass from datetime import datetime @@ -97,8 +100,29 @@ def _atomic_write_text(target: Path, content: str, encoding: str = "utf-8") -> N # 让 section_parser / postprocess 可被 import（与 k.py 同目录） sys.path.insert(0, str(Path(__file__).resolve().parent)) +from functools import lru_cache + from section_parser import ANCHOR_TAIL_RE, split_blocks -from postprocess import build_outline_data + + +@lru_cache(maxsize=4096) +def _split_blocks_cached(text: str): + """wiki 扫描类 lint 的共享块切分缓存——health 一次跑全家族时同一页只切一次。 + 仅限只读消费（bare/coarse/cite/extract 不改 Block）；parse_blocks_with_anchors + 会回填 .anchor，走自己的直调路径，不共享本缓存。""" + return split_blocks(text) +from postprocess import ( + _normalize_for_hash, + build_outline_data, + validate_outline, +) +from retrieval_index import ( + RetrievalIndexError, + coverage_report as evidence_index_coverage, + read_evidence_unit, + rebuild_index as rebuild_evidence_index, + search_evidence, +) # ============================================================ @@ -418,6 +442,449 @@ def search_pages(query, pages, limit=20): return results[:limit] +# ============================================================ +# raw 原文块级全文检索（search-raw） +# ============================================================ +# 设计定位：这是 KB 提供给 agent 的**宽召回机械原语**——正文/标题/摘要的关键词 +# 加权计数，确定性、零 LLM、零 embedding（原则 1/4）。语义能力由调用方 agent +# 提供，用在三个位置：①检索前把问题扩写成多组措辞逐组搜（同义联想）；②搜空时 +# 读 outline 结构化导航（像人翻目录）；③检索后对 top 命中逐块精读裁决哪个真正 +# 回答问题（LLM 当 reranker）。协议见 kb-query 第 4.7 步。 +# +# 存在意义：wiki 是蒸馏层，长文档 ingest 后 90%+ 的细节只在 raw——此前 `search` +# 只覆盖 wiki，未蒸馏的细节没有任何检索面，partial re-ingest 的触发也只能靠 +# 章节标题命中。本命令补上「内容级」检索面：命中直接返回可 read-block 的块锚点， +# 且标注命中块所属章节是否在 source_summary 登记表中为 ⊙ 扫读（deepen_hint， +# 提示按 kb-query 4.5 触发 partial re-ingest 把该细节蒸馏进 wiki）。 + +_SKIM_MARK = "⊙" + +_SEARCH_MD_LINK_RE = re.compile(r"\[([^\]\n]*)\]\([^)\n]*\)") + + +def _search_visible_link_text(text: str) -> str: + """搜索视图只隐藏链接坐标，保留用户真正看到的 label/alias。 + + 引用核对仍使用 `_mask_link_noise` 整段等长掩码，防止 URL/锚点数字 + 冒充论断；搜索不能复用那个视图，否则 `[HippoRAG](...)` 中唯一的 + 实体名会被删掉。返回值保持原长度，让 snippet 偏移仍可对回原文。 + """ + def keep_wikilink(m: re.Match) -> str: + display = (m.group(3) or m.group(1) or "").strip() + return (display + (" " * len(m.group(0))))[:len(m.group(0))] + + def keep_md_label(m: re.Match) -> str: + display = (m.group(1) or "").strip() + return (display + (" " * len(m.group(0))))[:len(m.group(0))] + + text = WIKILINK_RE.sub(keep_wikilink, text) + text = _SEARCH_MD_LINK_RE.sub(keep_md_label, text) + return _URL_RE.sub(_blank_span, text) + + +def _is_link_catalog(text: str) -> bool: + """识别几乎只由多个链接组成的 TOC/目录块。 + + 这使搜索能保留正文里的可见 label，同时避免一个目录块因列出所有 + 章节名而刷屏。 + """ + link_count = len(WIKILINK_RE.findall(text)) + len(_SEARCH_MD_LINK_RE.findall(text)) + if link_count < 2: + return False + rest = WIKILINK_RE.sub(" ", text) + rest = _SEARCH_MD_LINK_RE.sub(" ", rest) + rest = re.sub(r"[\W\d_]+", "", rest, flags=re.UNICODE) + return len(rest) < 20 + + +def _norm_title_for_match(t: str) -> str: + """标题匹配归一化：压空白、小写、剥前导章节编号——原文标题带编号 + （"2.2 Mutual Indexing"）而登记表常只写名字（"Mutual Indexing"）。""" + t = re.sub(r"^\s*\d+(?:\.\d+)*\.?\s+", "", (t or "").strip()) + return re.sub(r"\s+", " ", t).lower() + + +def _registry_key(target: str) -> str: + """登记表绑定 key：去 .md 的**完整相对路径**（raw/papers/foo）——不能用文件 + stem：跨目录同名（raw/papers/foo 与 raw/notes/foo）会串档，通用章节标题 + （Introduction 等）叠加后假阳 deepen 概率现实可感。""" + t = normalize_link_target(target) + return t[:-3] if t.endswith(".md") else t + + +_DEPTH_MARKS = ("✓", "⊙", "×") + + +def _source_page_raw_key(page) -> str | None: + """source_summary 页 → 其绑定的 raw 文档 key。 + + 绑定优先级：frontmatter `sources` 里第一条 raw/ 链接（source_summary schema + 规定恰好 1 条、受 list-source-issues 守护）> 正文第一个 [[raw/...]] 链接 + （后者顺序脆弱——对照引用挪到「原始文件」行之前就挂错论文，仅作回退）。 + """ + for src in getattr(page, "sources", None) or []: + for link in parse_wikilinks(str(src)): + if RAW_REF_PREFIX_RE.match(link.target): + return _registry_key(link.target) + for link in parse_wikilinks(mask_code_spans(page.raw_content)): + if RAW_REF_PREFIX_RE.match(link.target): + return _registry_key(link.target) + return None + + +def _load_depth_registry(pages) -> dict[str, list[dict]]: + """「章节深度登记」表全量解析 → {raw_key: [{"anchors", "titles", "status"}]}。 + + status 取行内**第一个**以 ✓/⊙/× 开头的单元格的首字符——状态列恒在备注列 + 之前，升级行「✓ 深读 | 备注: 由 ⊙ 升级…」会正确判为 ✓（备注里的 ⊙ 不算）。 + 兼容两种行格式：模板式首列 ^h- anchor、demo 式首列章节标题。 + 归档 / deprecated 页跳过（过期登记不该继续产生任何提示）。 + """ + out: dict[str, list[dict]] = {} + for page in pages: + if "章节深度登记" not in page.raw_content: + continue + if _is_exempt(page): + continue + raw_key = _source_page_raw_key(page) + if not raw_key: + continue + rows = out.setdefault(raw_key, []) + for line in page.raw_content.split("\n"): + if not line.lstrip().startswith("|"): + continue + cells = [c.strip().strip("*` ") for c in line.strip().strip("|").split("|")] + # 优先级状态判定：✓ > × > ⊙（"any ✓ wins"——多 mark 共存行 + # [✓ 主导]，避免反向 bug「⊙ 在前 ✓ 在备注」被 next() 误取为 ⊙） + mark = None + if any(c[:1] == "✓" for c in cells): + mark = "✓" + elif any(c[:1] == "×" for c in cells): + mark = "×" + elif any(c[:1] == _SKIM_MARK for c in cells): + mark = _SKIM_MARK + if mark is None: + continue + row = {"anchors": set(), "titles": set(), "status": mark, + "is_title_only": True} + m = ANCHOR_RE_INLINE.search(cells[0]) + if m: + row["anchors"].add(m.group(1)) + row["is_title_only"] = False + # title 收集：cells[0]（anchor 或标题）、cells[1]（标题列）—— + # 排除状态格（首字符 ∈ _DEPTH_MARKS），防止 "⊙ 扫读" 字面串污染匹配 + for col_idx in (0, 1): + if col_idx >= len(cells): + continue + c = cells[col_idx] + if c[:1] in _DEPTH_MARKS: + continue + c2 = _norm_title_for_match(re.sub(r"\^[\w-]+", "", c)) + if c2 and set(c2) - {"-", " "}: + row["titles"].add(c2) + rows.append(row) + return out + + +def _load_skim_registry(pages) -> dict[str, dict]: + """_load_depth_registry 的 ⊙ 扫读视图 → {raw_key: {"titles", "anchors"}} + （search-raw 的 deepen_hint 用）。包含 title_only 标记供 _row_status 唯一性 + 闸门使用（详见 corpus_map 注释）。""" + out: dict[str, dict] = {} + for raw_key, rows in _load_depth_registry(pages).items(): + entry = out.setdefault(raw_key, {"titles": set(), "anchors": set(), + "title_only_titles": set()}) + for row in rows: + if row["status"] != _SKIM_MARK: + continue + entry["anchors"] |= row["anchors"] + entry["titles"] |= row["titles"] + if row.get("is_title_only"): + entry["title_only_titles"] |= row["titles"] + return out + + +def corpus_map(pages, file_filter=None, depth=2) -> dict: + """raw 层的「root_index」：全库文档地图——LLM 浏览式检索的第一跳。 + + 无 embedding 架构下语义召回的正解是「大模型阅读结构化索引做判断」： + corpus-map（全库视野，~几千 token）→ outline（单篇章节树）→ read-section + （≤30K 单节），三跳定位任何章节，每跳都由 agent 做语义判断。此前 raw 层 + 没有目录视图——想浏览 98 篇文档得跑 98 次 outline，没人会做，检索被迫 + 全部退化到关键词路。协议见 kb-query 第 4.7 步（浏览路）。 + + 每篇输出：标题 / 字符数 / 档位（①<30K ②30-150K ③>150K）/ 顶层章节 + （标题 + agent_summary 或 preview 兜底 + 深度登记状态 ✓⊙×）/ 摘要覆盖率 / + 关联 source_summary（无 = 未 ingest）。纯读 .outline.json，零 LLM。 + """ + registry = _load_depth_registry(pages) + src_map: dict[str, str] = {} + for page in pages: + if getattr(page, "type", None) != "source_summary" or _is_exempt(page): + continue + key = _source_page_raw_key(page) + if key and key not in src_map: + src_map[key] = page.path + depth = max(1, int(depth)) + docs = [] + agg_secs = agg_sum = agg_chars = 0 + _broken = [] + if RAW_DIR.exists(): + for md in sorted(RAW_DIR.rglob("*.md")): + if md.name.startswith("."): + continue + rel = _to_rel_posix(md) + if file_filter and file_filter not in rel: + continue + try: + outline = load_or_build_outline(md) + except Exception as e: + _broken.append((rel, str(e))) + continue + chars = outline.get("doc_chars") or 0 + tier = "③" if chars > 150000 else ("②" if chars >= 30000 else "①") + raw_key = _registry_key(rel) + rows = registry.get(raw_key) or [] + + def _row_status(sec) -> str | None: + anc = sec.get("anchor") + ttl = _norm_title_for_match(sec.get("title") or "") + for row in rows: + if anc and anc in row["anchors"]: + return row["status"] + if ttl and ttl in row["titles"]: + # title-only 行要求该 title 在本文档内归一化计数 = 1—— + # 否则多个同名 H2 都会被错误打上登记状态 + if not row["is_title_only"] or title_counts.get(ttl, 0) == 1: + return row["status"] + return None + + title = None + listed = [] + n_secs = n_sum = 0 + # 前置 pass：全文档章节标题归一化计数——_row_status 的 title-only + # 唯一性闸门依赖它，必须在 _walk（内含 _row_status 调用）前填满，否则 + # 每个 title-only 登记行都恒判 None（title_counts.get 恒为 0） + title_counts: dict[str, int] = {} + h1_count = 0 + + def _count_titles(secs): + nonlocal h1_count + for sec in secs: + t = _norm_title_for_match(sec.get("title") or "") + if t: + title_counts[t] = title_counts.get(t, 0) + 1 + if int(sec.get("level", 1)) == 1: + h1_count += 1 + _count_titles(sec.get("children", [])) + + _count_titles(outline.get("sections", [])) + + def _walk(secs, parents=None): + nonlocal title, n_secs, n_sum + parents = parents or [] + for sec in secs: + n_secs += 1 + if sec.get("agent_summary"): + n_sum += 1 + lvl = sec.get("level", 1) + sec_title = sec.get("title") + if lvl == 1 and title is None: + title = sec_title + # 唯一 H1 通常是文档标题，可不在章节地图里重复。 + # 多 H1 则通常是“每章一个 H1”的书籍/合并文档，必须全部列出。 + should_list = ( + (lvl == 1 and h1_count > 1) + or (2 <= lvl <= depth) + ) + if should_list: + listed.append({ + "anchor": sec.get("anchor"), + "level": lvl, + "title": sec_title, + "parent_anchor": parents[-1][0] if parents else None, + "heading_path": [p[1] for p in parents] + [sec_title], + "chars": max(0, (sec.get("char_end") or 0) - (sec.get("char_start") or 0)), + "summary": sec.get("agent_summary") + or ((sec.get("preview") or "")[:80] or None), + "has_agent_summary": bool(sec.get("agent_summary")), + "depth_status": _row_status(sec), + }) + _walk( + sec.get("children", []), + parents + [(sec.get("anchor"), sec_title)], + ) + + _walk(outline.get("sections", [])) + agg_secs += n_secs + agg_sum += n_sum + agg_chars += chars + docs.append({ + "file": rel, + "chars": chars, + "tier": tier, + "title": title or Path(rel).stem, + "summary_coverage": f"{n_sum}/{n_secs}", + "source_summary": src_map.get(raw_key), + "sections": listed, + }) + if _broken: + print(f"⚠️ {len(_broken)} 个 raw 文档 outline 读取失败（已跳过）：", file=sys.stderr) + for rel, err in _broken[:5]: + print(f" {rel}: {err}", file=sys.stderr) + return { + "docs": docs, + "summary": { + "total_docs": len(docs), + "total_chars": agg_chars, + "summary_coverage": f"{agg_sum}/{agg_secs}", + "not_ingested": sum(1 for d in docs if not d["source_summary"]), + }, + } + + +def search_raw(query, pages=None, limit=10, file_filter=None, include_wiki=False): + """块级全文检索 raw 原文（可选含 wiki 页），返回可直接 read-block 的命中块。 + + 评分（宽召回，语义裁决交给 agent）：正文子串计数（每词封顶 5 防单块刷分）×1、 + 所属章节标题命中 ×3、章节 agent_summary 命中 ×2、整句短语出现 +5、 + 多词全覆盖 +2。要求至少一词命中**正文**（纯标题命中会让整节所有块刷屏）。 + """ + terms = [t for t in query.lower().split() if t] + if not terms: + return [] + registry = _load_skim_registry(pages) if pages is not None else {} + roots = [] + if RAW_DIR.exists(): + roots.append(RAW_DIR) + if include_wiki and WIKI_DIR.exists(): + roots.append(WIKI_DIR) + results = [] + for root in roots: + for md in sorted(root.rglob("*.md")): + if md.name.startswith(".") or "_templates" in md.parts: + continue + rel = _to_rel_posix(md) + if file_filter and file_filter not in rel: + continue + try: + low = md.read_text(encoding="utf-8").lower() + except Exception: + continue + if not any(t in low for t in terms): + continue # 文件级预过滤：一个词都不含的文件不必切块 + summaries: dict[str, str] = {} + try: + outline = load_or_build_outline(md) + + def _walk(secs): + for sec in secs: + if sec.get("anchor"): + summaries[sec["anchor"]] = sec.get("agent_summary") or "" + _walk(sec.get("children", [])) + + _walk(outline.get("sections", [])) + except Exception: + pass + try: + blocks = parse_blocks_with_anchors(md) + except Exception: + continue + # 行号偏移：块行号是剥 frontmatter 后的 body 相对行，报给用户前 + # 加回 frontmatter 行数（与 list_all_blocks 对 char 偏移的处理同口径） + try: + from section_parser import strip_frontmatter as _sf + _fm, _ = _sf(md.read_text(encoding="utf-8")) + fm_lines = _fm.count("\n") + except Exception: + fm_lines = 0 + # heading 栈：既给块定位最近所属节（标题加权用），也保留祖先链 + # （deepen_hint 用——登记表通常只登记顶层章节，命中块常在子节里） + heading_stack: list[tuple[int, str, str]] = [] # (level, title, anchor) + for blk in blocks: + if blk.kind == "heading": + lvl = blk.level or 1 + while heading_stack and heading_stack[-1][0] >= lvl: + heading_stack.pop() + # heading_stack 的 title 字段存归一化形式（剥前导编号 + 压空白 + + # 小写），与 _load_depth_registry 收集的 reg["titles"] 格式一致—— + # search_raw 后续比对可直用 `ttl in reg["titles"]`，不用每次重算 + nt = _norm_title_for_match(blk.title or "") + heading_stack.append((lvl, nt, blk.anchor or "")) + continue + if blk.kind not in ("paragraph", "list", "blockquote", "table", "code"): + continue + cur_title = heading_stack[-1][1] if heading_stack else "" + cur_anchor = heading_stack[-1][2] if heading_stack else None + # 掩码链接后再计数：arXiv 转换的 TOC 目录块 / 行内引用链接的 + # 链接文本会把所有标题词都"含"一遍，不掩码会让目录块刷到榜首 + bt = ANCHOR_TAIL_RE.sub("", blk.text) + if _is_link_catalog(bt): + continue + btl = _search_visible_link_text(bt).lower() + tl = cur_title.lower() + sl = (summaries.get(cur_anchor or "", "") or "").lower() + score = 0.0 + matched = [] + for t in terms: + c = btl.count(t) + if c: + score += min(c, 5) + matched.append(t) + if t in tl: + score += 3 + if sl and t in sl: + score += 2 + if not matched: + continue + if len(terms) > 1: + # 短语匹配双侧压空白——query 里的多余空白 / 原文换行不该让 +5 静默失效 + if " ".join(terms) in re.sub(r"\s+", " ", btl): + score += 5 + if len(set(matched)) == len(set(terms)): + score += 2 + pos = btl.find(matched[0]) + start = max(0, pos - 60) + snippet = re.sub(r"\s+", " ", bt[start:pos + 100]).strip() # 掩码保长，偏移可直用原文 + if start > 0: + snippet = "…" + snippet + reg = registry.get(_registry_key(rel)) + deepen = False + if reg: + title_only = reg.get("title_only_titles") or set() + # title-only 登记要求该 title 在本文档内归一化计数 = 1——直接 + # 统计 heading_stack 里同名 title 出现次数（与 corpus_map + # _row_status 的 title_counts 闭包同口径，无需为 search_raw + # 单跑一遍 outline） + # stack_titles 与 reg["titles"] 都做归一化（剥前导编号）才能正确比较 + stack_titles = [_norm_title_for_match(tt) for _lvl, tt, _a in heading_stack] + for _lvl, ttl, anc in heading_stack: + if anc and anc in reg["anchors"]: + deepen = True + break + if ttl and ttl in reg["titles"]: + # title-only 文档内同名 >1 时不生效（避免对未登记的 + # 同名 H2 误打 deepen）；anchor 行不受此限 + if not (ttl in title_only and stack_titles.count(ttl) > 1): + deepen = True + break + results.append({ + "file": rel, + "anchor": blk.anchor, + "kind": blk.kind, + "line": blk.line_start + fm_lines, + "score": round(score, 1), + "section_title": cur_title, + "section_anchor": cur_anchor, + "agent_summary": summaries.get(cur_anchor or "") or None, + "matched_terms": sorted(set(matched)), + "snippet": snippet, + "deepen_hint": deepen, + }) + results.sort(key=lambda r: (-r["score"], r["file"], r["line"])) + return results[:limit] + + # ============================================================ # 列表查询 # ============================================================ @@ -918,6 +1385,12 @@ def health_report(pages, backlinks): unsummarized = list_unsummarized_sections(pages) bare_claims = list_bare_claims(pages) coarse_citations = list_coarse_citations(pages) + cite_findings = list_cite_mismatches(pages) + cite_by_issue: dict[str, int] = {} + for f in cite_findings: + cite_by_issue[f["issue"]] = cite_by_issue.get(f["issue"], 0) + 1 + suspect_citations = list_suspect_citations(pages) + claims_summary = extract_claims(pages)["summary"] index_mismatches = list_index_count_mismatches(pages) source_issues = list_source_count_issues(pages) status_issues = list_status_issues(pages) @@ -925,6 +1398,36 @@ def health_report(pages, backlinks): relation_balance_issues = list_relation_balance(pages) implicit_relations = list_implicit_relations(pages) i18n_violations = list_i18n_violations() + index_path = _evidence_index_path() + if not index_path.is_file(): + evidence_index = { + "ok": False, + "coverage_status": "missing", + "db_path": str(index_path), + } + else: + try: + raw_index_report = evidence_index_coverage(index_path) + evidence_index = { + "ok": raw_index_report.get("ok", False), + "coverage_status": raw_index_report.get("coverage_status", "unknown"), + "db_path": str(index_path), + "natural_units": raw_index_report.get("natural_units", {}), + "structural_sections": raw_index_report.get("structural_sections", {}), + "content_sections": raw_index_report.get("content_sections", {}), + "manifest": raw_index_report.get("manifest", {}), + "unexpected_empty_sections_count": raw_index_report.get( + "unexpected_empty_sections_count", 0 + ), + "corpus_freshness": raw_index_report.get("corpus_freshness", {}), + } + except RetrievalIndexError as exc: + evidence_index = { + "ok": False, + "coverage_status": "error", + "db_path": str(index_path), + "error": exc.to_dict()["error"], + } today = datetime.now().date() stale = [] @@ -961,6 +1464,23 @@ def health_report(pages, backlinks): "unsummarized_sections_count": len(unsummarized), "bare_claims_count": len(bare_claims), "coarse_citations_count": len(coarse_citations), + # cite-check：内容 mismatch + 无依据豁免 + 非 canonical 引用 = 闸门项； + # imprecise = 观察项； + # unverifiable / exempted = 信息项（release 无 raw 时前者即全部候选块） + "cite_mismatches_count": ( + cite_by_issue.get("mismatch", 0) + + cite_by_issue.get("exempt-missing-basis", 0) + + cite_by_issue.get("canonical-anchor-mismatch", 0) + + cite_by_issue.get("canonical-target-mismatch", 0) + ), + "cite_imprecise_count": cite_by_issue.get("imprecise-anchor", 0), + "cite_unverifiable_count": cite_by_issue.get("unverifiable", 0), + "cite_exempted_count": cite_by_issue.get("exempted", 0), + "suspect_citations_count": len(suspect_citations), + "citation_pairs_count": claims_summary["pairs_total"], + # 注意：这是全指标中唯一依赖 .cache 台账的计数——新 clone / 删缓存后 + # 会回升到 verifiable 总数，属预期（台账是可重建的 memoization） + "unaudited_citations_count": claims_summary["unaudited_verifiable"], "index_count_mismatches_count": len(index_mismatches), "source_issues_count": len(source_issues), "status_issues_count": len(status_issues), @@ -968,6 +1488,7 @@ def health_report(pages, backlinks): "relation_balance_issues_count": len(relation_balance_issues), "implicit_relations_count": len(implicit_relations), "i18n_violations_count": len(i18n_violations), + "evidence_index": evidence_index, "last_check": today.isoformat(), } @@ -1380,11 +1901,18 @@ def validate_frontmatter(file_path): # 字母-数字-字母这种缩写形式中的字母前缀。 NUMERIC_CLAIM_PATTERNS = [ re.compile(r"(?\s*\[!(?:WARNING|NOTE|TIP|IMPORTANT|CAUTION)\]", re.IGNORECASE) +PROTOCOL_CALLOUT_RE = re.compile( + r"^\s*>\s*\[!(?:WARNING|CAUTION)\]\s*(?:知识更新冲突|引用审计未通过)", + re.IGNORECASE | re.MULTILINE, +) + + +def _is_protocol_callout(text: str) -> bool: + """只有冲突/引用审计外壳免于 claim 核对；事实型 callout 不豁免。""" + return bool(PROTOCOL_CALLOUT_RE.search(text)) # 这些类型的页面跳过"裸论断"扫描（导航 / lint 报告 / 模板） BARE_CLAIMS_SKIP_TYPES = {"index"} @@ -1504,9 +2047,9 @@ def resolve_doc_path(arg: str) -> Path: def load_or_build_outline(md_path: Path) -> dict: """优先读 .outline.json；缺失、损坏或**已过期**时现场基于当前 .md 内容重建（不写盘）。 - 过期判定：outline 的 doc_chars 与 md 实际字符数不符。wiki 页被直接编辑后 - outline.json 不会自动再生，旧 char_start/char_end 切当前文本必然错位—— - 此时丢弃盘上缓存、按当前内容重建。 + 过期判定：outline schema 版本 + 完整 Markdown SHA-256 必须与 + 当前文件一致。doc_chars 仅作快速诊断字段，不再作新鲜度证明； + 同长度改写也必须使旧章节边界和摘要失效。 注意：重建**不**合并旧缓存的 agent_summary——heading 锚点的 hash 只对标题 文本计算，整页换主题而标题未变（「目录」「快速导航」等通用标题）时锚点不变， @@ -1521,12 +2064,19 @@ def load_or_build_outline(md_path: Path) -> dict: # 损坏的缓存按「缺失」处理：md 还在就能现场重建，不让一个坏文件卡死读路径 disk_outline = None if not md_path.exists(): - if disk_outline is not None: - return disk_outline raise FileNotFoundError(str(md_path)) text = md_path.read_text(encoding="utf-8") - if disk_outline is not None and disk_outline.get("doc_chars") == len(text): - return disk_outline + if disk_outline is not None: + try: + issues = validate_outline( + disk_outline, + text, + expected_doc_path=_to_rel_posix(md_path), + ) + except Exception: + issues = [{"code": "outline-validator-error"}] + if not issues: + return disk_outline return build_outline_data(text, _to_rel_posix(md_path)) @@ -1547,42 +2097,81 @@ def parse_blocks_with_anchors(md_path: Path) -> list: def find_section_in_outline(sections: list[dict], anchor_or_title: str) -> dict | None: - """在嵌套 sections 里递归查找：先按 anchor 精确匹配，再按 title 精确匹配。""" + """在嵌套 sections 里递归查找：先按 anchor 精确匹配，再按 title 精确匹配。 + + 标题只是人读别名，不是唯一坐标。同名章节必须 fail-closed 并返回 + 候选 anchor，禁止深度优先猜第一个。 + """ target = anchor_or_title.lstrip("^") target_title_norm = re.sub(r"\s+", " ", anchor_or_title.strip().lower()) - # 第一遍：按 anchor - def by_anchor(secs): + anchor_matches: list[dict] = [] + title_matches: list[dict] = [] + + def collect(secs): for s in secs: if s.get("anchor") == target: - return s - r = by_anchor(s.get("children", [])) - if r: - return r - return None - hit = by_anchor(sections) - if hit: - return hit - # 第二遍：按 title - def by_title(secs): - for s in secs: + anchor_matches.append(s) t = re.sub(r"\s+", " ", (s.get("title") or "").strip().lower()) if t == target_title_norm: - return s - r = by_title(s.get("children", [])) - if r: - return r - return None - return by_title(sections) + title_matches.append(s) + collect(s.get("children", [])) + + collect(sections) + if len(anchor_matches) == 1: + return anchor_matches[0] + if len(anchor_matches) > 1: + raise LookupError(f"anchor 不唯一: {anchor_or_title}") + if len(title_matches) == 1: + return title_matches[0] + if len(title_matches) > 1: + candidates = ", ".join( + f"^{s.get('anchor')} ({s.get('title')})" for s in title_matches + ) + raise LookupError( + f"标题不唯一: {anchor_or_title!r}；请改用 anchor: {candidates}" + ) + return None -def read_section(md_path: Path, anchor_or_title: str) -> dict: +def read_section( + md_path: Path, + anchor_or_title: str, + max_chars: int = 30000, +) -> dict: """根据 anchor 或 title 取出整段 H 段（到下一同级 heading 之前）。""" + if isinstance(max_chars, bool) or not isinstance(max_chars, int) or max_chars < 0: + raise ValueError("max_chars 必须是非负整数（0 表示显式不限制）") outline = load_or_build_outline(md_path) sec = find_section_in_outline(outline["sections"], anchor_or_title) if not sec: raise LookupError(f"未找到 anchor 或标题: {anchor_or_title}") text = md_path.read_text(encoding="utf-8") body = text[sec["char_start"]:sec["char_end"]] + if max_chars > 0 and len(body) > max_chars: + children = [ + { + "anchor": child.get("anchor"), + "title": child.get("title"), + "chars": max( + 0, + int(child.get("char_end", 0)) + - int(child.get("char_start", 0)), + ), + } + for child in sec.get("children", []) + ] + child_hint = ( + "; 可下钻子节: " + + ", ".join( + f"^{c['anchor']}({c['chars']} chars)" for c in children[:12] + ) + if children + else "; 该叶子节无子 heading，请用 blocks 列目录后逐个 read-block" + ) + raise LookupError( + f"章节过长: ^{sec.get('anchor')} 有 {len(body)} 字符，" + f"超过单次上限 {max_chars}{child_hint}" + ) return { "path": _to_rel_posix(md_path), "anchor": sec["anchor"], @@ -1608,7 +2197,7 @@ def _anchor_hash6(anchor: str) -> str | None: return m.group(1) if m else None -def read_block(md_path: Path, anchor: str) -> dict: +def read_block(md_path: Path, anchor: str, max_chars: int = 30000) -> dict: """根据 ^p-/^t-/^c-/^f- anchor 取出单个 block 原文。 精确锚点未命中时，做一次**哈希容错回收**：按 anchor 末段的 hash6（内容指纹）在全文块里找， @@ -1616,11 +2205,24 @@ def read_block(md_path: Path, anchor: str) -> dict: 但 type/seq 写错"的引用（典型：表格被解析为 ^p- 却被引为 ^t-；或把 ^p-7-x 误写成 ^t-7-x）， 而内容真不在本页的孤儿锚点（hash6 零命中）仍按原样报"未找到"、由上层降级处理。 """ + if isinstance(max_chars, bool) or not isinstance(max_chars, int) or max_chars < 0: + raise ValueError("max_chars 必须是非负整数（0 表示显式不限制）") + + def checked_text(raw: str, resolved_anchor: str) -> str: + text = ANCHOR_TAIL_RE.sub("", raw).rstrip() + if max_chars > 0 and len(text) > max_chars: + raise LookupError( + f"自然块过长: ^{resolved_anchor} 有 {len(text)} 字符，" + f"超过单次上限 {max_chars}；请用 search-evidence/read-evidence-unit " + "先锁定所需自然单元，或显式 --max-chars 0 由调用方承担上下文风险" + ) + return text + target = anchor.lstrip("^") blocks = parse_blocks_with_anchors(md_path) for blk in blocks: if blk.anchor == target: - text = ANCHOR_TAIL_RE.sub("", blk.text).rstrip() + text = checked_text(blk.text, target) return { "path": _to_rel_posix(md_path), "anchor": target, @@ -1635,7 +2237,7 @@ def read_block(md_path: Path, anchor: str) -> dict: matches = [b for b in blocks if b.anchor and _anchor_hash6(b.anchor) == th] if len(matches) == 1: blk = matches[0] - text = ANCHOR_TAIL_RE.sub("", blk.text).rstrip() + text = checked_text(blk.text, blk.anchor) return { "path": _to_rel_posix(md_path), "anchor": blk.anchor, @@ -2016,11 +2618,16 @@ def list_bare_claims(pages) -> list[dict]: def list_coarse_citations(pages) -> list[dict]: """扫 wiki paragraph / list / blockquote 块：含具体数字论断、且**只挂整页 - `[[raw/X]]` 引用、未精确到块级 `[[raw/X#^anchor]]`** → 列入「引用粒度不足」。 + 引用（`[[raw/X]]` 或 `[[wiki/sources/X]]`）、未精确到块级 `#^anchor`** → + 列入「引用粒度不足」。 + + 与 list_bare_claims 互补：bare = 有数字但无任何引用；coarse = 有数字 + 有 + 引用但只到整页。块级 anchor 才能精确溯源、被 cite-check 核对、且渲染论文式 + [n] 上标。任一块级 `#^` 引用（raw 或 wiki 目标）即视为合规、不报； + 无任何整页引用的归 bare-claims、不在此报。 - 与 list_bare_claims 互补：bare = 有数字但无任何 raw 引用；coarse = 有数字 + 有 - raw 引用但只到整页。块级 anchor 才能精确溯源、且渲染论文式 [n] 上标。 - 任一块级 `#^` 引用即视为合规、不报；无任何整页引用的归 bare-claims、不在此报。 + 整页 [[wiki/sources/X]] 同样算 coarse——它能过 bare-claims（REFERENCE_SUPPORT_RE + 认它是支撑），若不在此查，就是数字论断绕开块级 anchor 的最廉价逃逸路径。 跳过同 list_bare_claims（deprecated / 归档 / index / lint·stub 标签 / callout / 表格行 / 代码内字面）。 @@ -2035,7 +2642,7 @@ def list_coarse_citations(pages) -> list[dict]: continue if any(t in {"to-be-updated", "stub"} for t in page.tags): continue - for blk in split_blocks(page.raw_content): + for blk in _split_blocks_cached(page.raw_content): if blk.kind not in ("paragraph", "list", "blockquote"): continue block_text = ANCHOR_TAIL_RE.sub("", blk.text) @@ -2049,13 +2656,15 @@ def list_coarse_citations(pages) -> list[dict]: hits.extend(m.group().strip() for m in pat.finditer(scan_text)) if not hits: continue - # 已含块级引用 → 合规；无整页引用 → 属 bare-claims 范畴，不在此报 - if RAW_BLOCK_CITE_RE.search(scan_text): + # 已含任一块级引用（raw 或 wiki 目标）→ 合规；无整页引用 → 属 bare-claims 范畴，不在此报 + if BLOCK_CITE_ANY_RE.search(scan_text): continue - if not RAW_PAGE_CITE_RE.search(scan_text): + coarse_refs = list(dict.fromkeys( + RAW_PAGE_CITE_RE.findall(scan_text) + WIKI_SOURCES_PAGE_CITE_RE.findall(scan_text) + )) + if not coarse_refs: continue hits = list(dict.fromkeys(hits)) - coarse_refs = list(dict.fromkeys(RAW_PAGE_CITE_RE.findall(scan_text))) preview = re.sub(r"\s+", " ", block_text).strip() if len(preview) > 200: preview = preview[:200] + "…" @@ -2071,121 +2680,1836 @@ def list_coarse_citations(pages) -> list[dict]: return out -def list_status_issues(pages) -> list[dict]: - """扫 `status: reviewed` 但 `last_modified_by != Human` 的页面： - reviewed（"已审阅"）语义 = 人类审阅过；LLM 自己写入的页面不该自称已审， - 应保持 `draft`，由人类审阅后才改成 `reviewed` + `last_modified_by: Human`。 - 跳过 deprecated / 归档区（_is_exempt）。 - """ - out = [] - for page in pages: - if _is_exempt(page): - continue - if page.status == "reviewed" and page.last_modified_by != "Human": - out.append({ - "path": page.path, - "title": page.title, - "type": page.type, - "status": page.status, - "last_modified_by": page.last_modified_by, - }) - return out +# ============================================================ +# 引用语义核对（cite-check） +# ============================================================ +# 现有结构 lint（bare-claims / coarse-citations / broken-refs）只保证引用 +# 「结构正确」：锚点存在、有引用、粒度够细。本节建立「被引块内容 vs 论断内容」 +# 的确定性比对与审计基础设施——零 LLM（原则 1）： +# +# - list_cite_mismatches 第 1 层确定性 lint：论断中的数字 / 引文必须出现在 +# 被引块原文中（数值舍入容差匹配），不在场 = 高置信错引 +# - extract_claims 确定性枚举全库（论断块, 块级引用）审计对，供外部 +# agent（kb-cite-audit / kb-ingest 第 9.5 步回验）做 +# 语义判定——KB 只出数据，判定在外部 +# - cite_audit_log 验证台账 .cache/citation_audit.jsonl 的受控写入口： +# 台账是「agent 劳动的 memoization」（删了重审即重建， +# 纯派生层）；「引用有问题」这一知识状态则落 markdown +# （CAUTION 审计标注块），不依赖 .cache +# - list_suspect_citations 从 markdown 扫「引用审计未通过」CAUTION 标注块—— +# 待人处理错引的唯一真相源 +# +# 语义边界（勿产生虚假安全感）：数字共现 ≠ 语义支撑。cite-mismatch 归零只说明 +# 「没有确定性可判的错引」，曲解 / 过度概括仍需 agent 审计（extract-claims + +# fresh-context 判定）与人审兜底。 + +# [KB 推算] 豁免标记：数字为跨块计算 / 单位换算所得、被引原文无该字面时显式声明。 +# 必须带依据锚（如 `[KB 推算: ^t-33-0c8446]`）才生效——裸 [KB 推算] 本身是 finding， +# 防止「抄错数字后贴标记洗白」。豁免只关掉本 lint 的数字核对，不改 bare-claims / +# coarse-citations 语义（块仍必须有引用）。 +CITE_EXEMPT_ANY_RE = re.compile(r"\[KB 推算[^\]]*\]") +CITE_EXEMPT_VALID_RE = re.compile(r"\[KB 推算[:：][^\]]*\^[hpcft]-[^\]]+\]") + +# ^h- 章节目标的「草垛上限」：超长章节（如整章 Results）里几乎任何数字都能巧合 +# 命中——超过此字符数的 ^h- 节即便命中也只降级 imprecise-anchor，不作通过证据。 +H_SECTION_HAYSTACK_LIMIT = 8000 + +# 论断侧「可核对数字」提取（比 NUMERIC_CLAIM_PATTERNS 更宽也更严）： +# 宽——补裸小数 / 小写 k / 中文倍数 / 万亿量级；严——默认排除两大误报源： +# 纯年份（叙述性元数据居多）与 <4 位无小数裸整数（遍地巧合）。 +_CITE_NUM_PATTERNS = [ + ("percent", re.compile(r"(?|<|≥|≤)\s*$") +_APPROX_AFTER_RE = re.compile(r"^\s*(?:左右|上下|以上|以下|\+)") + +# 逐字引文核对：成对引号包裹、归一化后 ≥12 字符的引文才进核对（短引号多为术语强调）。 +# 引文按省略号切成片段（每段 ≥8 字符）分别匹配——容忍「摘引中省略中段」的合法写法。 +_QUOTE_SPAN_RES = [ + re.compile(r"“([^”\n]{12,}?)”"), + re.compile(r"「([^」\n]{12,}?)」"), + re.compile(r"\"([^\"\n]{12,}?)\""), +] +_ELLIPSIS_SPLIT_RE = re.compile(r"…+|\.{3,}|\[…\]|\[\.\.\.\]") -def fmt_status_issues(items: list[dict]): - if not items: - print("✅ 没有发现 status 矛盾（reviewed 均由人类 Human 设置）") - return - print(f"⚠️ 发现 {len(items)} 处 status 矛盾（标 reviewed 但 last_modified_by 非 Human）：\n") - print(" （reviewed=「已审阅」应由人类审阅后设置；LLM 写入的页面应为 draft）\n") - for it in items: - print(f" {it['path']} ({it['title']}, {it['type']}) — status={it['status']}, by={it['last_modified_by']}") - print("\n 修复: 改回 status: draft（或人类审阅后把 last_modified_by 改为 Human）") - print() +_MD_LINK_RE = re.compile(r"\[[^\]\n]*\]\([^)\n]*\)") +_URL_RE = re.compile(r"https?://[^\s)\]]+") -def fmt_relation_balance(items: list[dict]): - if not items: - print("✅ 关系词频次均衡（无单一关系词占比 > 30%）") - return - print(f"⚠️ 发现 {len(items)} 个关系词占比过高（> 30% 阈值）：\n") - print(f" （标准关系白名单：{sorted(RELATION_TYPES)}）\n") - for it in items: - pct = it["ratio"] * 100 - print(f" {it['relation']}: {it['count']}/{it['total_relations']} = {pct:.0f}% (阈值 {it['threshold']:.0%})") - print(f" 建议: {it['suggestion']}") - print() +def _blank_span(m: re.Match) -> str: + return "".join("\n" if ch == "\n" else " " for ch in m.group(0)) -def fmt_implicit_relations(items: list[dict]): - if not items: - print("✅ 没有发现隐含关系（plain wikilink 配判断/立场动词的段落都已加 RELATION）") - return - print(f"⚠️ 发现 {len(items)} 处隐含关系（plain wikilink + 判断动词，应改 [[?|RELATION]]）：\n") - print(" （判断词: " + "、".join(sorted(_IMPLICIT_RELATION_VERBS_ZH)) + "）\n") - for it in items: - print(f" {it['path']}:{it['line']} ({it['title']}, {it['type']})") - print(f" 判断词: {', '.join(it['matched_verbs'])}") - print(f" plain wikilink: {' '.join(it['plain_wikilinks'])}") - print(f" 片段: {it['preview']}") - print() +def _mask_wikilinks(text: str) -> str: + """把 [[...]] 双链整个 span 替换为等长空白（保行号 / 偏移）。 + 数字 / 引文提取前必须先做——链接目标文件名（gpt-3.5-turbo）、锚点串 + （^p-38-54f895）、别名里的数字都不是论断数字。 + """ + return WIKILINK_RE.sub(_blank_span, text) -# ========== i18n 硬编码扫描（list-i18n-violations） ========== -# 扫 web/ 下的 .tsx 文件，找硬编码的中文 UI 字符串。 -# CLAUDE.md "Web 管理台国际化方案" 明文禁止： -# "不允许在组件里写硬编码的中文 / 英文 UI 字符串 -# （除非是 markdown 内容本身的渲染）" -# 但之前没有自动 lint，禁令形同虚设。本扫描器作为守门员。 -# 中文字符范围（含汉字） -CN_CHAR = r"一-鿿" -# JSX 文本节点：>...内容...< （单行内） -JSX_TEXT_CN_RE = re.compile(rf">([^<>{{}}\n]*[{CN_CHAR}][^<>{{}}\n]*)<") -# UI 相关 JSX 属性的字符串值含中文（白名单几个最常见的 UI 属性） -JSX_ATTR_CN_RE = re.compile( - rf'\b(aria-label|placeholder|title|alt|label)\s*=\s*' - rf'(["\'])([^"\'\n]*[{CN_CHAR}][^"\'\n]*)\2' +def _mask_link_noise(text: str) -> str: + """掩码所有链接形态：[[wikilink]]、[text](url) markdown 链接、裸 URL。 + + 链接目标 / 标识符里的数字（arXiv 编号 2309.15217、URL 路径、文件名版本号） + 不是论断数字——双侧（论断 / 被引原文）提取数字前都要过这层。 + """ + text = _mask_wikilinks(text) + text = _MD_LINK_RE.sub(_blank_span, text) + text = _URL_RE.sub(_blank_span, text) + return text + + +# arXiv HTML→markdown 转换的已知瑕疵：数学模式（MathML/LaTeX）里的数字被 +# accessibility 文本重复渲染 2-4 遍原样拼接（如 "78.4278.4278.42"、"101010k"、 +# "4.84.84.84.8B"），偶尔夹杂 LaTeX 残留（"\mathbf{...}"、"\,"细空格、零宽字符）。 +# 归一化时清理这层噪音，否则目标侧数字提取会把这些数字判定为"不存在"（假阴） +# 或提取出拼接后的乱码数值（更糟）。 +_ZERO_WIDTH_RE = re.compile(r"[​‌‍]") +_LATEX_SPACING_RE = re.compile(r"\\[,;:!]|\\quad|\\qquad") +_LATEX_WRAP_RE = re.compile(r"\\(?:mathbf|mathrm|mathit|text|textrm|boldsymbol)\{([^{}]*)\}") +_MAG_SUFFIX_ALT = r"[kKMB万亿]|(?i:million|billion|thousand)\b" +# 行尾防护：不允许紧跟"更多数字"或"小数点+数字"（真正的小数延续，如 +# "44.43" 里 "44" 后的 ".4"）；但**允许**紧跟"小数点+非数字"（句末句号， +# 如 "8.388.38." 三连折叠后跟句号收尾）——两者都是"后面有个点"，但语义 +# 相反，必须分开判断，否则会把大量以句号收尾的重复瑕疵误判为"疑似延续" +# 而拒绝折叠。 +_NUM_TAIL_GUARD = r"(?![\d]|\.\d)" +# 小数repeat：后缀可选（"78.4278.4278.42"→"78.42"；"4.84.84.84.8B"→"4.8B"）。 +# 边界锚定在整个重复片段之外（而非每次重复之间）——纯数字/点组成的连续串里 +# 单个重复单元前后天然被其他重复占满，无法在中间找到"非数字非点"的分隔， +# 这正是判定"这是重复而非巧合"所需要的信号；同时保证不误伤如 "44.43" 这类 +# 内部恰好有重复数字子串的正常小数（边界检查会让错误的切分方式匹配失败）。 +# group1 用惰性量词（\d+?）——四连重复（"300300300300"）若用贪婪量词， +# 回溯会先试到"300300"（合法但非最小的重复单元，2 个而非 4 个原子块）就满足 +# \1+ 而停手，只对半折叠成"300300M"；惰性量词从最短开始试，保证找到的是 +# 最小原子重复单元，一次性完整折叠。 +_DECIMAL_REPEAT_RE = re.compile( + r"(? set[float]: + """扫描纯整数重复瑕疵，枚举**全部**可能的原子重复单元对应的候选值。 -def list_i18n_violations(web_dir: Path | None = None) -> list[dict]: - """扫 web/ 下 .tsx，找硬编码中文（JSX text + UI 属性）。 + 与小数repeat不同（小数点是天然锚点，"78.4278.4278.42" 唯一对应 "78.42"）， + 纯数字重复存在结构性歧义："666666B" 无法从字符串本身判断原子单元是 + "6"（重复 6 次）还是 "66"（重复 3 次，真实值）还是 "666"（重复 2 次）—— + 正则回溯无论贪婪或惰性都只能猜一个，猜错就制造新的假阴（如 "66B" 被误 + 折叠成 "6B" 而消失）。改为枚举全部能整除、且实际重复 ≥2 次的候选单元长度， + 每个候选值都计入 target_values 集合——符合既定方向"目标侧宁可多算不可 + 漏算"（宽提取只造成假阴风险的降低，不产生新的假阳：即便某个候选值恰好 + 撞上一个真实错误的论断数字，也只是让确定性层错过这一条，不会把正确的 + 论断误判为错误）。 + """ + values: set[float] = set() + for m in _INT_REPEAT_SCAN_RE.finditer(text): + digits, suffix = m.group(1), m.group(2) + n = len(digits) + mult = _mag_mult_lookup(suffix.strip()) if suffix else None + for unit_len in range(1, n // 2 + 1): + if n % unit_len or digits[:unit_len] * (n // unit_len) != digits: + continue + try: + base = float(digits[:unit_len]) + except ValueError: + continue + values.add(base) + if mult is not None: + values.add(base * mult) + return values + + +def _normalize_for_cite_match(text: str) -> str: + """数字 / 引文比对用的归一化：NFKC 折叠全半角（％→%、全角数字→半角）、 + 去数字千分位逗号、剥 markdown 强调符、清理 arXiv 转换的数字重复瑕疵 + （小数点锚定、无歧义的部分——纯整数重复的歧义候选见 _int_repeat_candidate_values， + 不在这里做文本改写）。不压空白——保留偏移语义交调用方处理。""" + text = unicodedata.normalize("NFKC", text) + text = _ZERO_WIDTH_RE.sub("", text) + text = _LATEX_SPACING_RE.sub("", text) + for _ in range(2): # 嵌套 wrapper 罕见，两轮足够覆盖已知语料 + text = _LATEX_WRAP_RE.sub(r"\1", text) + text = re.sub(r"(?<=\d),(?=\d{3})", "", text) + text = re.sub(r"[*_`~]", "", text) + text = _DECIMAL_REPEAT_RE.sub(r"\1\2", text) + return text - 跳过： - - node_modules / .next / web/app/api/（API 响应不是 UI） - - i18n.ts / i18n-client.tsx（基建自身） - - // 单行 + /* */ 多行注释（用 mask_js_comments 剥除） - JSX 文本扫描前先 mask_jsx_expressions 剥除 {…} 表达式——否则被 `{n}` / - `{t("x")}` 打断的混排硬编码中文（如 `共 {n} 条`）会漏报。 +def _extract_claim_numbers(norm_text: str, dedupe: bool = True) -> list[dict]: + """从归一化论断文本提取可核对数字 token。 + + 返回 [{raw, value, tol, kind}]：value 为绝对值（量级 / 金额已乘倍率）， + tol 为匹配容差 = 0.5 × 10^(-小数位) × 倍率（即「按论断声明精度的舍入相等」： + 95 匹配 94.7、95.3 匹配 95.34），带约数标记时 ×3。 + 纯年份（1900-2100 的裸整数）不提取。 """ - if web_dir is None: - web_dir = ENGINE_ROOT / "web" - out = [] - if not web_dir.exists(): - return out + toks: list[dict] = [] + consumed: list[tuple[int, int]] = [] - for tsx in sorted(web_dir.rglob("*.tsx")): - if tsx.name in I18N_SKIP_FILE_NAMES: - continue - if any(part in I18N_SKIP_DIR_PARTS for part in tsx.parts): + def _overlaps(span: tuple[int, int]) -> bool: + return any(not (span[1] <= s or span[0] >= e) for s, e in consumed) + + for kind, pat in _CITE_NUM_PATTERNS: + for m in pat.finditer(norm_text): + if _overlaps(m.span()): + continue + num_str = m.group(1).replace(",", "") + try: + base = float(num_str) + except ValueError: + continue + mult = 1.0 + unit = None + if kind in ("magnitude", "money") and m.lastindex and m.lastindex >= 2 and m.group(2): + mult = _MAG_MULT.get(m.group(2), 1.0) + if kind == "unit": + um = re.search(r"(ms|毫秒|GB|TB|MB|QPS|qps)\s*$", m.group(0).strip()) + unit = um.group(1).upper().replace("毫秒", "MS") if um else None + if kind == "bigint" and 1900 <= base <= 2100: + continue # 年份形态：默认不核对（叙述性元数据居多） + if kind == "decimal" and re.fullmatch(r"(?:19|20)\d{2}\.\d{4,5}", num_str): + continue # arXiv 编号形态（YYMM.NNNNN）：标识符不是论断数字 + decimals = len(num_str.split(".")[1]) if "." in num_str else 0 + approx = bool( + _APPROX_BEFORE_RE.search(norm_text[: m.start()]) + or _APPROX_AFTER_RE.match(norm_text[m.end():]) + ) + tol = 0.5 * (10 ** -decimals) * mult * (3 if approx else 1) + consumed.append(m.span()) + toks.append({ + "raw": re.sub(r"\s+", " ", m.group(0)).strip(), + "value": base * mult, + "tol": tol, + "kind": kind, + "unit": unit, + "pos": m.start(), + }) + ordered = sorted(toks, key=lambda t: t["pos"]) + if not dedupe: + # cite mismatch / value-level exemption 需要保留每次出现的位置，防止一个 + # marker 通过“同值去重”顺带豁免该值在段内的其他未标记实例。 + return ordered + # 去重（同值同容差只留一个），按首现位置排序——cloze 占位符编号依据 + seen: set[tuple] = set() + out = [] + for t in ordered: + key = (t["value"], t["tol"]) + if key in seen: + continue + seen.add(key) + out.append(t) + return out + + +_TARGET_NUM_RE = re.compile(r"(\d+(?:\.\d+)?)\s*(" + _MAG_SUFFIX_ALT + r")?") + + +def _mag_mult_lookup(suffix: str | None) -> float: + """量级后缀→倍率，兼容大小写（"Billion"/"billion" 等）而不破坏既有单字符 + 大小写敏感语义（"M"=百万缩写，不因忽略大小写误把独立字母"m"当量级）。""" + if not suffix: + return 1.0 + return _MAG_MULT.get(suffix) or _MAG_MULT.get(suffix.lower(), 1.0) + + +def _extract_target_values(norm_text: str) -> set[float]: + """目标侧提取**所有**数字 token 的数值集合（含量级倍率展开 + 原始值双录）。 + + 比论断侧宽得多：目标是「论断数字是否在场」的存在性判断，宽提取只会造成 + 假阴（漏报 mismatch），符合「宁可漏报」方向。量级后缀除 k/K/M/B/万/亿 + 外也认英文拼写词（million/billion/thousand）——原文常用词形式而非缩写。 + """ + values: set[float] = set() + for m in _TARGET_NUM_RE.finditer(norm_text): + try: + base = float(m.group(1)) + except ValueError: + continue + values.add(base) + if m.group(2): + values.add(base * _mag_mult_lookup(m.group(2))) + values |= _int_repeat_candidate_values(norm_text) + return values + + +def _extract_unit_values(norm_texts: list[str], unit: str) -> set[float]: + """目标侧按**指定单位**提取数值——16GB ≠ 16TB：带单位的 token 必须与相同 + 单位相邻的数值匹配，防「数值相同、单位不同」的假阴。""" + unit_pat = {"MS": r"(?:ms|毫秒)", "GB": "GB", "TB": "TB", "MB": "MB", + "QPS": r"(?:QPS|qps)"}.get(unit, re.escape(unit)) + pat = re.compile(r"(? bool: + tol = tok["tol"] + 1e-9 + v = tok["value"] + return any(abs(tv - v) <= tol for tv in target_values) + + +def _drop_pending_source_numbers(claim_norm: str, toks: list[dict]) -> list[dict]: + """剔除 30 字符窗口内紧跟 [需要来源] 的数字 token——作者显式声明「此数不归属 + 本块引用」，既不核对（lint）也不挖空（cloze）也不计入 numeric_tokens。""" + markers = [m.start() for m in re.finditer(r"\[需要来源\]", claim_norm)] + if not markers or not toks: + return toks + def _pending(tok: dict) -> bool: + tok_end = int(tok.get("pos", 0)) + len(tok.get("raw", "")) + return any(0 <= start - tok_end <= 30 for start in markers) + return [t for t in toks if not _pending(t)] + + +def _calculation_basis_anchor(marker_text: str) -> str | None: + match = re.search(r"\^([hpcft]-[A-Za-z0-9-]+)", marker_text) + return match.group(1) if match else None + + +def _partition_calculated_numbers(claim_norm: str, toks: list[dict], + max_gap: int = 30, + allowed_basis_anchors: set[str] | None = None, + ) -> tuple[list[dict], list[dict]]: + """把带依据的 ``[KB 推算: ^锚]`` 收窄到其前方紧邻的**一个**数字 token。 + + 旧实现只要核对单元里出现一个合法标记，就把整段所有数字都从确定性核对中 + 删除；一个合法推算值因而可以顺带洗白同段其他抄错数字。这里按位置把每个 + marker 只绑定到它前方最近、间距不超过 ``max_gap`` 字符的一个 token。没有 + 邻近数字的 marker 不产生豁免；裸 ``[KB 推算]`` 仍由调用方作为闸门 finding。 + + 返回 ``(仍需核对, 已豁免)``，保持 token 原出现顺序。 + """ + if not toks: + return toks, [] + exempted_ids: set[int] = set() + for marker in CITE_EXEMPT_VALID_RE.finditer(claim_norm): + basis = _calculation_basis_anchor(marker.group(0)) + if allowed_basis_anchors is not None and basis not in allowed_basis_anchors: + continue + candidates: list[tuple[int, dict]] = [] + for idx, tok in enumerate(toks): + if idx in exempted_ids: + continue + tok_end = int(tok.get("pos", 0)) + len(tok.get("raw", "")) + gap = marker.start() - tok_end + if 0 <= gap <= max_gap: + candidates.append((idx, tok)) + if candidates: + # 只豁免 marker 前最近的值；一个 marker 永不扩散到更早的数字。 + idx, tok = max(candidates, key=lambda item: item[1].get("pos", 0)) + tok["calculation_basis_anchor"] = basis + exempted_ids.add(idx) + checked = [tok for idx, tok in enumerate(toks) if idx not in exempted_ids] + exempted = [tok for idx, tok in enumerate(toks) if idx in exempted_ids] + return checked, exempted + + +def _build_cloze(claim_norm: str, toks: list[dict]) -> dict: + """把论断里的可核对数字挖成占位符 ⟦N1⟧…——盲填复核（blind cloze）用。 + + 核验者只拿到挖空论断 + 被引原文去填空，全程看不到期望数字——从原理上 + 消灭「判定式审核」的附和偏差；填回的值由 cloze-check 按数值容差机器判分。 + 同一 token 的所有字面出现都会被挖掉（防泄漏）；占位符按首现位置编号。 + """ + ordered = sorted(toks, key=lambda t: t.get("pos", 0)) + labels = {id(t): f"N{i + 1}" for i, t in enumerate(ordered)} + text = claim_norm + for t in sorted(ordered, key=lambda t: -len(t["raw"])): + body = re.escape(t["raw"]).replace(r"\ ", r"\s*").replace(" ", r"\s*") + text = re.sub(r"(? float: + letters = [ch for ch in s if ch.isalpha()] + if not letters: + return 0.0 + cjk = sum(1 for ch in letters if "一" <= ch <= "鿿") + return cjk / len(letters) + + +def _norm_quote(s: str) -> str: + s = _normalize_for_cite_match(s) + return re.sub(r"\s+", " ", s).strip().casefold() + + +def _quote_matches(quote: str, haystacks: list[str]) -> bool | None: + """引文是否逐字（归一化后）出现在任一目标文本中。 + + 返回 True=命中；False=未命中（quote-mismatch）；None=不可核对（跨语言转写 / + 引文含链接 / 片段过短）——翻译引文本质不可逐字核对，交 agent 语义审计。 + """ + if "[[" in quote: + return None + q_cjk = _cjk_ratio(quote) > 0.5 + frags = [f for f in _ELLIPSIS_SPLIT_RE.split(quote) if len(_norm_quote(f)) >= 8] + if not frags: + return None + same_script = [h for h in haystacks if h and (_cjk_ratio(h) > 0.5) == q_cjk] + if not same_script: + return None + normed = [_norm_quote(h) for h in same_script] + for f in frags: + nf = _norm_quote(f) + if not any(nf in nh for nh in normed): + return False + return True + + +def _block_clean_text(blk) -> str: + return ANCHOR_TAIL_RE.sub("", blk.text).rstrip() + + +def _section_text_from_blocks(blocks: list, h_idx: int) -> str: + """从块列表拼出 h_idx 号 heading 所辖的整节文本（到下一同级或更高 heading 前）。 + + 与 outline char 切片相比，这条路径与 parse_blocks_with_anchors 共用同一坐标系 + （frontmatter 之后的 body），且逐块剥锚点尾——锚点串里的 seq 数字不会污染 + 目标侧数字提取。 + """ + lvl = blocks[h_idx].level or 1 + parts = [] + for j in range(h_idx, len(blocks)): + if j > h_idx and blocks[j].kind == "heading" and (blocks[j].level or 1) <= lvl: + break + parts.append(_block_clean_text(blocks[j])) + return "\n".join(parts) + + +def _load_doc_blocks(target_norm: str, cache: dict): + """per-run 缓存：目标文档 → 块列表（含回填锚点）。不存在 / 越界 → None。""" + if target_norm in cache: + return cache[target_norm] + doc_path = _safe_join_under_root(target_norm) + blocks = None + if doc_path is not None and doc_path.exists(): + try: + blocks = parse_blocks_with_anchors(doc_path) + except Exception: + blocks = None + cache[target_norm] = blocks + return blocks + + +def _find_block_resolution(blocks: list, anchor_id: str) -> tuple[int | None, str | None]: + """返回 ``(block_index, recovered_from)``。 + + 普通读取沿用 read_block 的 hash6 唯一回收，但把该事实显式带到上层；strict + 不能把“找到了相似锚”冒充 canonical 精确命中。 + """ + for i, b in enumerate(blocks): + if b.anchor == anchor_id: + return i, None + h = _anchor_hash6(anchor_id) + if h: + idxs = [i for i, b in enumerate(blocks) if b.anchor and _anchor_hash6(b.anchor) == h] + if len(idxs) == 1: + return idxs[0], anchor_id + return None, None + + +def _find_block_by_anchor(blocks: list, anchor_id: str) -> int | None: + """与 read_block 同口径定位；需要区分回收时用 _find_block_resolution。""" + return _find_block_resolution(blocks, anchor_id)[0] + + +def _resolve_cited_text(target_norm: str, anchor_id: str, cache: dict) -> dict: + """把一条块级引用解析为可比对文本。 + + 返回 {status, kind, primary, fallback, text_for_hash}： + - status: ok / file-missing / anchor-missing + - primary: 命中即「通过」的文本——^p-/^t-/^c-/^f- 为单块；^h- 为整节 + （超过 H_SECTION_HAYSTACK_LIMIT 的大节 primary 置 None，命中只算 imprecise） + - fallback: 命中降级 imprecise-anchor 的文本——块目标的 owning section / + 超限的 ^h- 大节 + - text_for_hash: 台账 target_content_hash 的取数（块文本 / 整节文本）—— + ^h- 对整节正文算 hash，确定性补上「锚点只 hash 标题、正文重写不被察觉」 + 的已知残留 + """ + doc_path = _safe_join_under_root(target_norm) + canonical_target = _to_rel_posix(doc_path) if doc_path is not None else None + blocks = _load_doc_blocks(target_norm, cache) + if blocks is None: + return {"status": "file-missing", "canonical_target": canonical_target} + idx, recovered_from = _find_block_resolution(blocks, anchor_id) + if idx is None: + return {"status": "anchor-missing", "canonical_target": canonical_target} + blk = blocks[idx] + resolution_meta = { + "canonical_target": canonical_target, + "canonical_anchor": blk.anchor, + "recovered_from": recovered_from, + } + if blk.kind == "heading": + sect = _section_text_from_blocks(blocks, idx) + text = _mask_link_noise(mask_code_spans(sect)) + if len(text) > H_SECTION_HAYSTACK_LIMIT: + return {"status": "ok", "kind": "heading-large", "primary": None, + "fallback": text, "text_for_hash": sect, **resolution_meta} + return {"status": "ok", "kind": "heading", "primary": text, + "fallback": None, "text_for_hash": sect, **resolution_meta} + clean = _block_clean_text(blk) + primary = _mask_link_noise(mask_code_spans(clean)) + h = None + for j in range(idx - 1, -1, -1): + if blocks[j].kind == "heading": + h = j + break + fallback = None + if h is not None: + fallback = _mask_link_noise(mask_code_spans(_section_text_from_blocks(blocks, h))) + return {"status": "ok", "kind": blk.kind, "primary": primary, + "fallback": fallback, "text_for_hash": clean, **resolution_meta} + + +def _block_level_citations(scan_text: str) -> list[tuple[str, str]]: + """从（已掩码代码的）块文本提取全部块级引用 (target_norm, anchor_id)，按出现序去重。""" + out: list[tuple[str, str]] = [] + seen: set[tuple[str, str]] = set() + for link in parse_wikilinks(scan_text): + if not link.anchor or not link.anchor.startswith("^"): + continue + target_norm = normalize_link_target(link.target) + prefix_probe = re.sub(r"^(?:\./)+", "", target_norm) + if not (RAW_REF_PREFIX_RE.match(prefix_probe) or WIKI_REF_PREFIX_RE.match(prefix_probe)): + continue + key = (target_norm, link.anchor.lstrip("^")) + if key in seen: + continue + seen.add(key) + out.append(key) + return out + + +_LIST_ITEM_START_RE = re.compile(r"^\s*(?:[-*+]|\d+\.)\s+") +# ANCHOR_TAIL_RE 只在整段文本的**末尾**（$ 无 MULTILINE）剥离锚点——对多条目 +# list 块通常够用（block 只有一个末尾锚点）。但部分 wiki 页每个条目自带独立 +# 行内锚点（历史遗留 / 手工整理内容），此时非末位条目的锚点尾巴不会被剥离； +# 按条目原子分解后，这些锚点串里的哈希数字会被误当成论断数字提取。用 +# MULTILINE 变体逐行剥离，作为 _block_units 拆分前的兜底清理。 +_ANCHOR_TAIL_MULTILINE_RE = re.compile(ANCHOR_TAIL_RE.pattern, re.MULTILINE) + + +def _strip_all_anchor_tails(text: str) -> str: + return _ANCHOR_TAIL_MULTILINE_RE.sub("", text) + + +def _block_units(blk, block_text: str) -> list[tuple[str, int]]: + """块 → 核对单元 [(unit_text, line)]——原子论断分解的确定性部分。 + + list 块按条目拆（条目是自然的原子论断边界；continuation 行并入所属条目）， + 每条目对**自己的**引用负责——抓「条目 A 的数字只在条目 B 的引用目标里」这类 + 块内张冠李戴；条目无引用时回退整块引用并集。其余块整块一个单元。 + """ + block_text = _strip_all_anchor_tails(block_text) + if blk.kind == "table": + units = [] + for i, line in enumerate(block_text.split("\n")): + stripped = line.strip() + if not stripped or re.fullmatch(r"\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*)+\|?", stripped): + continue + units.append((line, blk.line_start + i)) + return units or [(block_text, blk.line_start)] + if blk.kind != "list": + return [(block_text, blk.line_start)] + units: list[tuple[str, int]] = [] + cur: list[str] = [] + cur_line = blk.line_start + for i, line in enumerate(block_text.split("\n")): + if _LIST_ITEM_START_RE.match(line): + if cur: + units.append(("\n".join(cur), cur_line)) + cur = [line] + cur_line = blk.line_start + i + else: + if not cur: + cur_line = blk.line_start + i + cur.append(line) + if cur: + units.append(("\n".join(cur), cur_line)) + return units or [(block_text, blk.line_start)] + + +def _owning_heading_contexts(blocks: list) -> dict[int, list[str]]: + """每个块对应的 H1→当前最深 heading 标题路径。""" + stack: list[tuple[int, str]] = [] + out: dict[int, list[str]] = {} + for idx, blk in enumerate(blocks): + if blk.kind == "heading": + level = blk.level or 1 + while stack and stack[-1][0] >= level: + stack.pop() + title = re.sub(r"\s+", " ", (blk.title or _block_clean_text(blk)).strip()) + stack.append((level, title)) + out[idx] = [title for _level, title in stack] + return out + + +def _dedupe_number_tokens(toks: list[dict]) -> list[dict]: + seen: set[tuple] = set() + out = [] + for tok in toks: + key = (tok.get("value"), tok.get("tol"), tok.get("unit")) + if key in seen: + continue + seen.add(key) + out.append(tok) + return out + + +def list_cite_mismatches(pages) -> list[dict]: + """第 1 层确定性核对：对每个「含可核对数字 / 逐字引文 + 块级引用」的 wiki 块， + 核对数字（数值舍入容差）与引文（归一化子串）是否出现在被引块原文中。 + + finding 的 issue 分四档： + - mismatch 数字在被引块及其所在节都找不到 → 高置信错引（闸门项） + - exempt-missing-basis [KB 推算] 无依据锚（闸门项——防洗白） + - canonical-*-mismatch 引用依赖 hash recovery / 非 canonical 完整目标路径 + - imprecise-anchor 数字不在被引块、但在其所在节（锚点挂偏）；或仅在超限 + ^h- 大节命中；或逐字引文未命中（quote 未命中可能是 + 转述，观察项） + - unverifiable 某条被引文件不可得（raw 未分发 / 未 convert）→ 该引用 + 单独不可核验；同块其他可用引用仍核对。存在不可得目标时 + 不把“可用目标均未命中”升级 mismatch（避免缺失证据假阳） + - exempted [KB 推算: ^锚] 生效跳过数字核对（信息项，趋势可监控） + + 跳过链与 bare-claims 一致（deprecated / 归档 / index / lint·stub 标签 / + 协议型冲突/审计 callout / 代码内字面）。事实型 callout 与 table data row 纳入。 + 全部 anchor-missing 的块不双报（broken-refs 辖区）。 + """ + out = [] + doc_cache: dict[str, object] = {} + for page in pages: + if _is_exempt(page): + continue + if page.type in BARE_CLAIMS_SKIP_TYPES: + continue + if any(t in BARE_CLAIMS_SKIP_TAGS for t in page.tags): + continue + if any(t in {"to-be-updated", "stub"} for t in page.tags): + continue + for blk in _split_blocks_cached(page.raw_content): + if blk.kind not in ("paragraph", "list", "blockquote", "table"): + continue + block_text = ANCHOR_TAIL_RE.sub("", blk.text) + if _is_protocol_callout(block_text): + continue + block_citations = _block_level_citations(mask_code_spans(block_text)) + if not block_citations: + continue + for unit_text, unit_line in _block_units(blk, block_text): + scan_text = mask_code_spans(unit_text) + unit_citations = _block_level_citations(scan_text) + # 表格每一数据行对自己的引用负责；禁止某一行借用另一行的引用。 + citations = unit_citations or ([] if blk.kind == "table" else block_citations) + if not citations: + continue + + preview = re.sub(r"\s+", " ", unit_text).strip() + if len(preview) > 200: + preview = preview[:200] + "…" + base = {"path": page.path, "title": page.title, "type": page.type, + "line": unit_line, "preview": preview} + + # [KB 推算]：带依据锚才生效；裸标记本身是 finding + any_spans = [m.span() for m in CITE_EXEMPT_ANY_RE.finditer(scan_text)] + cited_anchors = {anchor for _target, anchor in citations} + valid_spans = { + m.span() for m in CITE_EXEMPT_VALID_RE.finditer(scan_text) + if _calculation_basis_anchor(m.group(0)) in cited_anchors + } + bare_exempts = [s for s in any_spans if s not in valid_spans] + if bare_exempts: + out.append({**base, "issue": "exempt-missing-basis", "numbers": [], "quotes": [], + "citations": [{"target": t, "anchor": a} for t, a in citations], + "suggestion": "[KB 推算] 必须带依据锚（如 [KB 推算: ^t-33-0c8446]）——" + "且该依据锚必须在同一核对单元实际引用；否则视为洗白标记"}) + + claim_norm = _normalize_for_cite_match(_mask_link_noise(scan_text)) + num_toks = _extract_claim_numbers(claim_norm, dedupe=False) + # 数字后紧跟 [需要来源] = 作者显式声明「此数不归属本块引用」—— + # 不算错引（cite-check 治的是虚假归因），归 [需要来源] 积压治理 + num_toks = _drop_pending_source_numbers(claim_norm, num_toks) + quotes: list[str] = [] + for qre in _QUOTE_SPAN_RES: + quotes.extend(m.group(1) for m in qre.finditer(scan_text)) + # 合法 [KB 推算: ^锚] 只豁免其前方紧邻的一个值，不能扩散到整段。 + # claim_norm 与 token 的 pos 共用归一化后的坐标系；scan_text 上的 + # valid_spans 仅用于上面的「裸 marker」语法判别。 + num_toks, exempted_toks = _partition_calculated_numbers( + claim_norm, num_toks, allowed_basis_anchors=cited_anchors) + if exempted_toks: + out.append({**base, "issue": "exempted", + "numbers": list(dict.fromkeys(t["raw"] for t in exempted_toks)), + "quotes": [], + "citations": [{"target": t, "anchor": a} for t, a in citations], + "suggestion": ""}) + + resolved = [] + for target_norm, anchor_id in citations: + r = _resolve_cited_text(target_norm, anchor_id, doc_cache) + r["target"] = target_norm + r["anchor"] = anchor_id + resolved.append(r) + for r in resolved: + if r.get("recovered_from"): + out.append({ + **base, "issue": "canonical-anchor-mismatch", + "requested_anchor": r["anchor"], + "canonical_anchor": r.get("canonical_anchor"), + "citations": [{"target": r["target"], "anchor": r["anchor"], + "status": r["status"]}], + "suggestion": "引用依赖 hash recovery；请把草稿锚点改为目标块的精确 canonical anchor", + }) + canonical_target = r.get("canonical_target") + if canonical_target and canonical_target != r["target"]: + out.append({ + **base, "issue": "canonical-target-mismatch", + "requested_target": r["target"], + "canonical_target": canonical_target, + "citations": [{"target": r["target"], "anchor": r["anchor"], + "status": r["status"]}], + "suggestion": "引用目标必须使用 workspace 根相对的 canonical 完整路径", + }) + if not num_toks and not quotes: + continue + cits_brief = [{"target": r["target"], "anchor": r["anchor"], "status": r["status"]} + for r in resolved] + missing_files = [r for r in resolved if r["status"] == "file-missing"] + if missing_files: + # 缺失目标单独不可核验，不能让它提前终止并遮蔽其他可用目标的 + # 正向命中 / 挂偏检查。若可用目标均未命中，仍不升级 mismatch： + # 数字可能由缺失目标支撑；strict 会因 unverifiable fail-closed。 + for missing in missing_files: + out.append({**base, "issue": "unverifiable", + "numbers": list(dict.fromkeys(t["raw"] for t in num_toks)), + "quotes": [], + "citations": [{ + "target": missing["target"], "anchor": missing["anchor"], + "status": missing["status"], + }], + "suggestion": "被引文件不可得（raw 未分发 / 未 convert）——" + "核对只在 raw 在场的环境执行"}) + usable = [r for r in resolved if r["status"] == "ok"] + if not usable: + continue # 全部 anchor-missing：broken-refs 辖区，不双报 + + primaries = [r["primary"] for r in usable if r["primary"]] + fallbacks = [r["fallback"] for r in usable if r["fallback"]] + prim_norm = [_normalize_for_cite_match(t) for t in primaries] + fb_norm = [_normalize_for_cite_match(t) for t in fallbacks] + prim_vals: set[float] = set() + for t in prim_norm: + prim_vals |= _extract_target_values(t) + fb_vals: set[float] = set() + for t in fb_norm: + fb_vals |= _extract_target_values(t) + + mismatched, imprecise = [], [] + for tok in num_toks: + if tok.get("unit"): + # 带计量单位的数字：单位敏感匹配（16GB ≠ 16TB） + pv = _extract_unit_values(prim_norm, tok["unit"]) + fv = _extract_unit_values(fb_norm, tok["unit"]) + else: + pv, fv = prim_vals, fb_vals + if _tok_matches(tok, pv): + continue + if _tok_matches(tok, fv): + imprecise.append(tok["raw"]) + else: + mismatched.append(tok["raw"]) + mismatched = list(dict.fromkeys(mismatched)) + imprecise = list(dict.fromkeys(imprecise)) + quote_missing = [] + for q in quotes: + if _quote_matches(q, primaries + fallbacks) is False: + quote_missing.append(q[:60]) + + if mismatched and not missing_files: + first = mismatched[0] + tgt = usable[0]["target"] + out.append({**base, "issue": "mismatch", + "numbers": mismatched, "quotes": quote_missing, + "citations": cits_brief, + "suggestion": f"数字不在被引块及其所在节：用 " + f"`python scripts/k.py find-anchor {tgt} \"{first}\"` " + f"反查正确块改锚，或修正论断数字；若为跨块推算，" + f"标 [KB 推算: ^依据锚]"}) + elif imprecise or quote_missing: + out.append({**base, "issue": "imprecise-anchor", + "numbers": imprecise, "quotes": quote_missing, + "citations": cits_brief, + "suggestion": "数字在被引块所在节的其他块（锚点挂偏）/ 仅在超长 ^h- 节命中" + " / 引文未逐字命中——建议把锚点降到承载该数字的 ^p-/^t- 块"}) + return out + + +# ---------- 审计对枚举（extract-claims）与验证台账 ---------- + +CITE_VERDICTS = {"SUPPORTED", "PARTIAL", "UNSUPPORTED", "CONTRADICTED", "UNVERIFIABLE"} +# 审计标注块起始行（与冲突块 CONFLICT_START_RE 平行的独立标记；CAUTION 已在 +# CALLOUT_RE 白名单，bare-claims / coarse-citations / cite-mismatches 自动跳过它） +SUSPECT_START_RE = re.compile(r"^>\s*\[!CAUTION\]\s*引用审计未通过") + + +def _md5_12(s: str) -> str: + return hashlib.md5(s.encode("utf-8")).hexdigest()[:12] + + +def _ledger_path() -> Path: + return PROJECT_ROOT / ".cache" / "citation_audit.jsonl" + + +def _load_ledger() -> dict[str, dict]: + """读台账（append-only jsonl），按 pair_id 取最新一条。坏行跳过。""" + path = _ledger_path() + if not path.exists(): + return {} + out: dict[str, dict] = {} + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except Exception: + continue + pid = rec.get("pair_id") + if pid: + out[pid] = rec + return out + + +def _append_ledger(record: dict) -> None: + path = _ledger_path() + path.parent.mkdir(parents=True, exist_ok=True) + with open(path, "a", encoding="utf-8") as f: + f.write(json.dumps(record, ensure_ascii=False) + "\n") + + +def _norm_for_evidence(s: str) -> str: + """evidence 比对用归一化：剥内联锚点串（read-block / read-section 返回的原文 + 带行尾 ^p-… 锚点，目标文本则已剥锚）、NFKC / 千分位 / 强调符、压空白、casefold。""" + s = ANCHOR_RE_INLINE.sub("", s) + s = _normalize_for_cite_match(s) + return re.sub(r"\s+", " ", s).strip().casefold() + + +def _raw_tree_has_files() -> bool: + """workspace 的 raw/ 树是否有实际文件（区分「整库未分发」与「单个文件缺失」）。""" + if not RAW_DIR.exists(): + return False + for f in RAW_DIR.rglob("*"): + if f.is_file() and f.name != ".gitkeep" and not f.name.startswith("."): + return True + return False + + +def _git_changed_wiki_paths(commit: str | None = None) -> list[str]: + """经 git 判定本 workspace wiki/ 下改动的 .md（相对 workspace 根）。 + + - 默认：工作区 vs HEAD 的改动 + 未跟踪新文件（ingest 提交前的回验场景） + - commit 指定时：该 commit 相对其父的改动（事后对账场景——注意证据取自 + **现行**工作树，内容若已漂移由 claim_hash 变化暴露） + 不在 git 仓库 / git 不可用时抛 ValueError，提示改用 --paths。 + """ + base = ["git", "-C", str(PROJECT_ROOT)] + try: + if commit: + r = subprocess.run( + base + ["diff", "--name-only", "--relative", f"{commit}^", commit, "--", "wiki"], + capture_output=True, text=True, check=True) + files = r.stdout.split() + else: + r1 = subprocess.run( + base + ["diff", "HEAD", "--name-only", "--relative", "--", "wiki"], + capture_output=True, text=True, check=True) + r2 = subprocess.run( + base + ["ls-files", "--others", "--exclude-standard", "--", "wiki"], + capture_output=True, text=True, check=True) + files = r1.stdout.split() + r2.stdout.split() + except (subprocess.CalledProcessError, FileNotFoundError) as e: + detail = "" + if isinstance(e, subprocess.CalledProcessError) and e.stderr: + detail = f"（{e.stderr.strip().splitlines()[0]}）" + raise ValueError(f"无法经 git 判定改动文件{detail}——不在 git 仓库内或 commit 无效；" + f"请改用 --paths 显式指定") + return sorted({f for f in files if f.endswith(".md")}) + + +def extract_claims(pages, paths=None, unaudited_only=False, sample=None, seed="", + with_evidence=False, max_evidence_chars=1500, cloze=False) -> dict: + """确定性枚举全库（论断块, 块级引用）审计对——语义判定的取数层，零 LLM。 + + 每个含块级引用的 paragraph / list / blockquote / **table** 块 × 其每条引用 + = 一条 pair。表格块刻意纳入（bare-claims 跳表格是压数字噪音，与「枚举所有 + 审计对」的语义无关）；NOTE/TIP/IMPORTANT 等事实 callout 纳入，只排除冲突块 / + 审计标注这两类协议外壳，防自审循环。 + + pair_id = md5(page|target|anchor|claim_hash)[:16]——claim 内容或 owning heading + 上下文一变 id 即变 + （自动回到未审），target 侧漂移由 target_content_hash 与台账记录比对暴露 + （^h- 目标对**整节正文**算 hash，确定性补上「锚点只 hash 标题」的盲区）。 + + 「已审」判定 = 台账存在该 pair_id 的记录 且 记录的 target_content_hash 与 + 现行重算值相等。台账是纯派生层（agent 劳动的 memoization）：删 .cache 唯一 + 后果是全部回到未审、重审即重建。 + """ + pathset = None + if paths: + pathset = set() + for p in paths: + p = p.replace("\\", "/").lstrip("./") + if not p.endswith(".md"): + p += ".md" + pathset.add(p) + raw_distributed = _raw_tree_has_files() + doc_cache: dict[str, object] = {} + pairs: list[dict] = [] + for page in pages: + if _is_exempt(page): + continue + if page.type in BARE_CLAIMS_SKIP_TYPES: + continue + if any(t in BARE_CLAIMS_SKIP_TAGS for t in page.tags): + continue + if pathset is not None and page.path not in pathset: + continue + blocks = _split_blocks_cached(page.raw_content) + heading_contexts = _owning_heading_contexts(blocks) + for block_idx, blk in enumerate(blocks): + if blk.kind not in ("paragraph", "list", "blockquote", "table"): + continue + block_text = ANCHOR_TAIL_RE.sub("", blk.text) + if _is_protocol_callout(block_text): + continue + block_citations = _block_level_citations(mask_code_spans(block_text)) + if not block_citations: + continue + # 原子论断分解：list 块按条目产 pair（claim_text 更聚焦，核验者不必 + # 在长块里自行推断归属）；条目无引用时回退整块引用并集 + for unit_text, unit_line in _block_units(blk, block_text): + scan_text = mask_code_spans(unit_text) + unit_citations = _block_level_citations(scan_text) + citations = unit_citations or ([] if blk.kind == "table" else block_citations) + if not citations: + continue + claim_body = re.sub(r"\s+", " ", unit_text).strip() + claim_context = heading_contexts.get(block_idx, []) + context_prefix = ( + f"[所属章节: {' > '.join(claim_context)}] " if claim_context else "" + ) + claim_text = context_prefix + claim_body + claim_hash_basis = "\n".join([*(f"HEADING:{h}" for h in claim_context), unit_text]) + claim_hash = _md5_12(_normalize_for_hash(claim_hash_basis)) + claim_norm = _normalize_for_cite_match(_mask_link_noise(scan_text)) + occurrence_toks = _drop_pending_source_numbers( + claim_norm, _extract_claim_numbers(claim_norm, dedupe=False)) + cited_anchors = {anchor for _target, anchor in citations} + checked_toks, calculated_toks = _partition_calculated_numbers( + claim_norm, occurrence_toks, allowed_basis_anchors=cited_anchors) + num_toks = _dedupe_number_tokens(checked_toks) + numeric_tokens = [t["raw"] for t in num_toks] + calculated_tokens = [ + {"raw": tok["raw"], "basis_anchor": tok.get("calculation_basis_anchor")} + for tok in calculated_toks + ] + block_cloze = _build_cloze(claim_norm, num_toks) if (cloze and num_toks) else None + multi_source = len({t for t, _ in citations}) > 1 + for target_norm, anchor_id in citations: + r = _resolve_cited_text(target_norm, anchor_id, doc_cache) + status = r["status"] + if (status == "file-missing" and RAW_REF_PREFIX_RE.match(target_norm) + and not raw_distributed): + status = "raw-not-distributed" + pair = { + "pair_id": hashlib.md5( + f"{page.path}|{target_norm}|{anchor_id}|{claim_hash}".encode("utf-8") + ).hexdigest()[:16], + "page": page.path, + "page_type": page.type, + "line": unit_line, + "claim_text": claim_text, + "claim_body": claim_body, + "claim_context": claim_context, + # 核验包必须拿到完整论断；旧实现静默截到 500 字符后仍让 + # verdict 为整段背书，会把截断点后的事实排除在核验视野外。 + # 字段保留显式状态，供 strict 对旧/外部截断包 fail-closed。 + "claim_text_truncated": False, + "claim_hash": claim_hash, + "numeric_tokens": numeric_tokens, + "calculated_tokens": calculated_tokens, + "multi_source": multi_source, + "target": target_norm, + "anchor": anchor_id, + "canonical_target": r.get("canonical_target"), + "canonical_anchor": r.get("canonical_anchor"), + "recovered_from": r.get("recovered_from"), + "target_kind": r.get("kind"), + "target_status": status, + "target_content_hash": ( + _md5_12(_normalize_for_hash(r["text_for_hash"])) if status == "ok" else None + ), + } + if with_evidence and status == "ok": + ev = r["text_for_hash"] + pair["evidence"] = ev[:max_evidence_chars] + pair["evidence_truncated"] = len(ev) > max_evidence_chars + pair["evidence_length"] = len(ev) + if block_cloze is not None: + pair["cloze"] = block_cloze + pairs.append(pair) + # join 台账：内容双侧未漂移的最新记录才算「已审」 + ledger = _load_ledger() + for p in pairs: + rec = ledger.get(p["pair_id"]) + valid = bool(rec) and rec.get("target_content_hash") == p["target_content_hash"] + p["audited"] = valid + p["last_verdict"] = rec.get("verdict") if valid and rec else None + total = len(pairs) + verifiable = [p for p in pairs if p["target_status"] == "ok"] + summary_all = { + "pairs_total": total, + "verifiable": len(verifiable), + "unverifiable_raw_missing": sum(1 for p in pairs if p["target_status"] == "raw-not-distributed"), + "broken": sum(1 for p in pairs if p["target_status"] in ("file-missing", "anchor-missing")), + "audited": sum(1 for p in verifiable if p["audited"]), + "unaudited_verifiable": sum(1 for p in verifiable if not p["audited"]), + } + if unaudited_only: + pairs = [p for p in pairs if p["target_status"] == "ok" and not p["audited"]] + if sample is not None and len(pairs) > sample: + # 可复现抽样：按 md5(seed|pair_id) 排序取前 N——同 seed（如 YYYY-WW 周号） + # 重跑取样一致，覆盖可累积 + pairs = sorted(pairs, key=lambda p: hashlib.md5( + f"{seed}|{p['pair_id']}".encode("utf-8")).hexdigest())[:sample] + pairs.sort(key=lambda p: (p["page"], p["line"])) + return {"pairs": pairs, "summary": {**summary_all, "returned": len(pairs)}} + + +def cite_audit_log_batch(entries: list[dict], by="agent", mode="audit", + draft_path: Path | None = None) -> dict: + """台账受控写入口（KB 是记录员不是裁判——verdict 由外部 agent / 人产出）。 + + 每条 entry: {pair_id, verdict, note?, evidence?}。写前用**现行内容**重算校验： + - pair_id 必须在现行枚举中（claim / 引用已变 → pair 过期，拒绝入账） + - UNVERIFIABLE 仅当目标确实不可得（target_status != ok）才接受；反之 + 目标可解析时必须给出实质判定——确定性堵死「记 UNVERIFIABLE 跳过劳动」 + - agent 记 SUPPORTED 必须附 evidence（被引块现行原文的一段字面子串， + ≥8 字符）——不能证明「比对过」，但强制「至少取回过原文」，把橡皮图章 + 成本从零抬到必须逐条 read-block + 返回 {written: [...], errors: [{pair_id, error}]}。 + """ + # 查询草稿的 pair_id 含 页面坐标，不能用 wiki 全库重算。 + # 显式 draft_path 让外部 fresh-context verifier 能经同一个受控入口把 + # 问答语义判定入账；默认仍保持原有 wiki ingest/audit 行为。 + pages = [_draft_page(draft_path)] if draft_path is not None else load_all_wiki_pages() + data = extract_claims(pages, with_evidence=True, max_evidence_chars=10 ** 9) + index = {p["pair_id"]: p for p in data["pairs"]} + written, errors = [], [] + now = datetime.now().strftime("%Y-%m-%d %H:%M") + for e in entries: + pid = str(e.get("pair_id", "")).strip() + verdict = str(e.get("verdict", "")).strip().upper() + note = str(e.get("note", "") or "") + evidence = str(e.get("evidence", "") or "") + try: + if verdict not in CITE_VERDICTS: + raise ValueError(f"非法 verdict {verdict!r}（可选：{'/'.join(sorted(CITE_VERDICTS))}）") + pair = index.get(pid) + if pair is None: + raise LookupError("pair 不存在或已过期（claim / 引用内容已变）——请重跑 extract-claims 取现行 pair_id") + status = pair["target_status"] + if verdict == "UNVERIFIABLE": + if status == "ok": + raise ValueError("目标可解析（target_status=ok）时不得记 UNVERIFIABLE——请实际核对后给出判定") + else: + if status != "ok": + raise ValueError(f"目标不可得（{status}）——只能记 UNVERIFIABLE 或先修复引用") + if by == "agent" and verdict == "SUPPORTED": + ev_norm = _norm_for_evidence(evidence) + if len(ev_norm) < 8: + raise ValueError("agent 记 SUPPORTED 必须附 --evidence（被引块原文字面子串，≥8 字符）——证明已取回原文") + hay = _norm_for_evidence(pair["evidence"]) + if ev_norm not in hay: + raise ValueError("--evidence 与被引块现行原文不匹配——请 read-block 重新取回原文再入账") + record = { + "pair_id": pid, + "page": pair["page"], + "line": pair["line"], + "claim_hash": pair["claim_hash"], + "target": pair["target"], + "anchor": pair["anchor"], + "target_content_hash": pair["target_content_hash"], + "verdict": verdict, + "by": by, + "mode": mode, + "note": note[:500], + "evidence_head": evidence[:120], + "checked_at": now, + } + _append_ledger(record) + written.append(record) + except (ValueError, LookupError) as err: + errors.append({"pair_id": pid, "error": str(err)}) + return {"written": written, "errors": errors} + + +def cloze_check_batch(entries: list[dict], draft_path: Path | None = None) -> list[dict]: + """盲填复核的机器判分（确定性、零 LLM）。 + + entries: [{pair_id, fills: {"N1": "72.4%", ...}}]——fills 是盲核验者只看 + 「挖空论断 + 被引原文」填回的值。判分：对每个占位符，从 fill 字符串解析数值 + （量级倍率展开），与该 pair 现行重算的期望 token 按声明精度舍入容差比对。 + 填不出 / 非数值 / 数值不符 → 该空 fail；全部空通过才算 pair 通过。 + + draft_path 给出时对草稿文件（而非 wiki）枚举 pair——查询侧 check-draft 配套。 + """ + if draft_path is not None: + pages = [_draft_page(draft_path)] + else: + pages = load_all_wiki_pages() + data = extract_claims(pages, cloze=True) + index = {pr["pair_id"]: pr for pr in data["pairs"]} + out = [] + for e in entries: + pid = str(e.get("pair_id", "")).strip() + fills = e.get("fills") or {} + pair = index.get(pid) + if pair is None: + out.append({"pair_id": pid, "passed": False, + "error": "pair 不存在或已过期——请重跑 extract-claims --cloze"}) + continue + blanks = (pair.get("cloze") or {}).get("blanks") or [] + if not blanks: + out.append({"pair_id": pid, "passed": False, "error": "该 pair 无可核对数字（无空可填）"}) + continue + detail = [] + for b in blanks: + expected = _extract_claim_numbers(_normalize_for_cite_match(b["raw"])) + fill = str(fills.get(b["ph"], "")).strip() + fill_norm = _normalize_for_cite_match(fill) if fill else "" + if expected and expected[0].get("unit"): + vals = _extract_unit_values([fill_norm], expected[0]["unit"]) + else: + vals = _extract_target_values(fill_norm) if fill_norm else set() + ok = bool(expected) and bool(vals) and _tok_matches(expected[0], vals) + detail.append({"ph": b["ph"], "expected": b["raw"], "fill": fill, "ok": ok}) + out.append({"pair_id": pid, "page": pair["page"], "line": pair["line"], + "passed": all(d["ok"] for d in detail), "blanks": detail}) + return out + + +def _draft_page(draft_path: Path): + """把任意 markdown 草稿包装成可进 cite 核对 / 审计对枚举的页面对象。 + + 查询答案、导出前的报告草稿等「不在 wiki 里的文本」由此获得与 wiki 页同一套 + 引用核对机器（check-draft）。frontmatter 有则剥掉。 + """ + from types import SimpleNamespace + from section_parser import strip_frontmatter + text = draft_path.read_text(encoding="utf-8") + _fm, body = strip_frontmatter(text) + return SimpleNamespace( + path=f"", + title=draft_path.stem, + type="analysis", + status="draft", + confidence="medium", + tags=[], + sources=[], + source_count=0, + raw_content=body, + ) + + +def _list_pending_source_markers(pages) -> list[dict]: + """列出正文中的 ``[需要来源]``；代码字面量不算真实待办。 + + ``list_bare_claims`` 有意把该标记视为“作者已诚实声明缺来源”，因此默认 lint + 不报裸论断；strict 输出闸门必须把这笔债务重新显式纳管，不能以占位符通过。 + """ + out: list[dict] = [] + marker_re = re.compile(r"\[需要来源\]") + for page in pages: + scan_text = mask_code_spans(page.raw_content) + for match in marker_re.finditer(scan_text): + line = scan_text[:match.start()].count("\n") + 1 + raw_line = page.raw_content.splitlines()[line - 1] if page.raw_content.splitlines() else "" + out.append({ + "path": page.path, + "line": line, + "issue": "pending-source", + "preview": raw_line.strip()[:200], + }) + return out + + +STRICT_DISCLOSURE_RE = re.compile(r"\[(?:知识库未覆盖|Agent 推断|Agent 综合)\]") +STRICT_PENDING_RE = re.compile(r"\[需要来源\]") + + +def _strip_callout_prefix(text: str) -> str: + lines = [] + for line in text.splitlines(): + clean = re.sub(r"^\s*>\s?", "", line) + if re.match(r"^\[!(?:WARNING|NOTE|TIP|IMPORTANT|CAUTION)\]", clean, re.I): + continue + lines.append(clean) + return "\n".join(lines).strip() + + +def _is_strict_structural_text(text: str) -> bool: + """strict 中允许的纯结构文字；其余自然语言默认按事实 fail-closed。""" + s = text.strip() + s = _LIST_ITEM_START_RE.sub("", s) + s = re.sub(r"^\s*>\s?", "", s) + if not s: + return True + # 纯导航链接 / 标点没有事实语义。 + without_links = WIKILINK_RE.sub("", s) + without_links = _MD_LINK_RE.sub("", without_links) + without_links = re.sub(r"[*_`~#|\s:：,，。；;、()（）\-]+", "", without_links) + if not without_links: + return True + if re.search(r"[?？]\s*$", s): + return True # 问题本身不是答案事实 + plain = re.sub(r"[*_`~]", "", s).strip() + if len(plain) <= 80 and plain.endswith((":", "：")): + return True + if re.fullmatch( + r"(?:回答|结论|摘要|说明|依据|来源|参考|要点|步骤|相关页面|延伸阅读|" + r"知识库依据|注意事项|局限|限制|建议)\s*[:：]?", plain): + return True + return False + + +def _strict_fact_units(page) -> list[dict]: + """把草稿正文拆成 strict 覆盖单元（句/list item/table data row/callout body）。""" + units: list[dict] = [] + for blk in _split_blocks_cached(page.raw_content): + if blk.kind not in ("paragraph", "list", "blockquote", "table"): + continue + block_text = _strip_all_anchor_tails(blk.text) + if _is_protocol_callout(block_text): + continue + raw_units = _block_units(blk, block_text) + if blk.kind == "table": + # 第一条非 separator 行是字段名，只做结构；其余每行独立负责引用。 + raw_units = raw_units[1:] + for raw_text, line in raw_units: + cleaned = _strip_callout_prefix(raw_text) if blk.kind == "blockquote" else raw_text.strip() + if not cleaned: + continue + parts = [cleaned] if blk.kind == "table" else [ + p.strip() for p in re.split(r"(?<=[。！？!?；;])\s*", cleaned) if p.strip() + ] + # 行尾契约 marker 常位于句号之后；把纯 marker 片段并回它所标注的前句。 + merged: list[str] = [] + marker_only = re.compile( + r"^(?:\[(?:知识库未覆盖|Agent 推断|Agent 综合|需要来源)\]\s*)+$") + citation_only = re.compile( + r"^(?:\s*\[\[[^\]]+\]\]\s*)+[.,，。；;:：!?！？]*$") + for part in parts: + # GroundMap 既有页常用「论断。[[证据]]」。句号切分后 + # citation 会独立成片；必须并回前句，否则正确引用会被误报 bare。 + if merged and (marker_only.fullmatch(part) or citation_only.fullmatch(part)): + merged[-1] += part + else: + merged.append(part) + parts = merged + for part in parts: + units.append({"text": part.strip(), "line": line, "kind": blk.kind}) + return units + + +def _list_strict_unmapped_facts(page) -> list[dict]: + """列出未映射到块级证据、也未按允许契约显式披露的事实单元。""" + findings = [] + for unit in _strict_fact_units(page): + text = unit["text"] + scan = mask_code_spans(text) + if STRICT_PENDING_RE.search(scan): + continue # 由 pending-source 独立阻断，避免双报 + if STRICT_DISCLOSURE_RE.search(scan): + continue + if _block_level_citations(scan): + continue + if _is_strict_structural_text(scan): + continue + findings.append({ + "path": page.path, + "page": page.path, + "line": unit["line"], + "issue": "unmapped-factual-claim", + "claim_text": re.sub(r"\s+", " ", text).strip(), + "kind": unit["kind"], + "detail": "strict 要求每个事实句/list item/table data row 有块级引用，" + "或显式标为 [知识库未覆盖]/[Agent 推断]/[Agent 综合]", + }) + return findings + + +def list_unmapped_claims(pages) -> list[dict]: + """列出 wiki 中没有块级证据或显式披露标记的事实单元。 + + 这是 check-draft strict 的同口径库内 scanner，供 staged-tree pre-commit + 对定性句、list item、table data row 与事实型 callout 做 coverage 闸门。 + """ + out: list[dict] = [] + for page in pages: + if _is_exempt(page) or page.type in BARE_CLAIMS_SKIP_TYPES: + continue + if any(t in BARE_CLAIMS_SKIP_TAGS for t in page.tags): + continue + if any(t in {"to-be-updated", "stub"} for t in page.tags): + continue + out.extend(_list_strict_unmapped_facts(page)) + return out + + +def check_draft(draft_path: Path, with_evidence=False, max_evidence_chars=1500, + cloze=False, strict=False) -> dict: + """对草稿文件跑与 wiki 同一套引用质量核对（查询侧闸门）。 + + 返回： + - cite_findings: list_cite_mismatches 口径（mismatch = 草稿数字与被引块不符） + - bare_claims / coarse_citations: 草稿里没引用 / 引用过粗的数字论断 + - pairs: 审计对（供语义回验 / 盲填复核，--with-evidence / --cloze 组装核验包） + 草稿引用的目标仍解析到当前 workspace 的 raw/ 与 wiki/ 下。 + + ``strict=False`` 保持旧门禁：mismatch / 无依据推算 + bare / coarse。 + ``strict=True`` 在此基础上 fail-closed：broken / raw 不可得 / imprecise / + pending-source / 无当前版本 provenance / 未经 SUPPORTED 语义审计 / 截断核验包 + 任一出现都失败。语义审计可用 ``cite-audit-log --draft`` 经受控入口入账。 + """ + fake = _draft_page(draft_path) + findings = list_cite_mismatches([fake]) + data = extract_claims([fake], with_evidence=with_evidence, + max_evidence_chars=max_evidence_chars, cloze=cloze) + base_gate_issues = {"mismatch", "exempt-missing-basis"} + # canonical 错误在 default 也必须阻断；strict 中它们已进 + # strict_findings，这里不重复计数。 + if not strict: + base_gate_issues |= {"canonical-anchor-mismatch", "canonical-target-mismatch"} + base_gate = [f for f in findings if f["issue"] in base_gate_issues] + bare = list_bare_claims([fake]) + coarse = list_coarse_citations([fake]) + pending = _list_pending_source_markers([fake]) + provenance_findings: list[dict] = [] + semantic_findings: list[dict] = [] + truncated_packets: list[dict] = [] + unmapped_facts: list[dict] = [] + strict_findings: list[dict] = [] + + if strict: + # imprecise 是可用目标上的确定性挂偏 / 引文未命中，strict 不允许观察项 + # 留在成功答案里。unverifiable 由下面每个 pair 的 target_status 统一纳管， + # 避免同一缺失目标既算 finding 又算 pair 的重复门禁。 + strict_findings.extend( + f for f in findings + if f["issue"] in { + "imprecise-anchor", "canonical-anchor-mismatch", "canonical-target-mismatch" + } + ) + + for pair in data["pairs"]: + status = pair["target_status"] + base = { + "page": pair["page"], "line": pair["line"], + "pair_id": pair["pair_id"], "target": pair["target"], + "anchor": pair["anchor"], + } + if status == "raw-not-distributed": + strict_findings.append({ + **base, "issue": "unverifiable-target", "target_status": status, + "detail": "raw 原文未分发，无法证明被引内容真实支撑论断", + }) + continue + if status in ("file-missing", "anchor-missing"): + strict_findings.append({ + **base, "issue": "broken-target", "target_status": status, + "detail": "引用目标文件或锚点不可解析", + }) + continue + + # 只有 target_status=ok 的 pair 才可能拥有有效语义 verdict。 + if not pair.get("audited"): + semantic_findings.append({ + **base, "issue": "semantic-unaudited", + "detail": "当前 claim/evidence 内容版本尚无语义审计判定", + }) + elif pair.get("last_verdict") != "SUPPORTED": + semantic_findings.append({ + **base, "issue": "semantic-verdict", + "verdict": pair.get("last_verdict"), + "detail": "strict 只接受现行内容版本的 SUPPORTED；PARTIAL 等旧状态不得认证", + }) + + truncated_fields = [ + name for name in ("claim_text_truncated", "evidence_truncated", "cloze_truncated") + if pair.get(name) + ] + if truncated_fields: + truncated_packets.append({ + **base, "issue": "audit-packet-truncated", + "fields": truncated_fields, + "detail": "核验器未看到完整 claim/evidence，不能为未见文本背书", + }) + + provenance_findings = check_provenance([fake])["findings"] + unmapped_facts = _list_strict_unmapped_facts(fake) + strict_findings.extend(pending) + strict_findings.extend(provenance_findings) + strict_findings.extend(semantic_findings) + strict_findings.extend(truncated_packets) + strict_findings.extend(unmapped_facts) + + strict_gate_count = (len(strict_findings) + len(bare) + len(coarse)) if strict else 0 + gate_count = len(base_gate) + strict_gate_count + passed = not (base_gate or bare or coarse or (strict and strict_findings)) + return { + "draft": str(draft_path), + "strict": strict, + "passed": passed, + "cite_findings": findings, + "bare_claims": bare, + "coarse_citations": coarse, + "pending_sources": pending, + "provenance_findings": provenance_findings, + "semantic_findings": semantic_findings, + "truncated_packets": truncated_packets, + "unmapped_facts": unmapped_facts, + "strict_findings": strict_findings, + "pairs": data["pairs"], + "summary": { + **data["summary"], + "base_gate_findings": len(base_gate), + "strict_gate_findings": strict_gate_count, + "gate_findings": gate_count, + }, + } + + +def list_suspect_citations(pages) -> list[dict]: + """从 markdown 扫「> [!CAUTION] 引用审计未通过」标注块——待人处理错引的 + 唯一真相源（与 find_conflicts 同机制；suspect 清单不依赖 .cache，删台账不丢待办）。""" + out = [] + for p in pages: + scan_lines = mask_code_spans(p.raw_content).split("\n") + raw_lines = p.raw_content.split("\n") + i, n = 0, len(scan_lines) + while i < n: + if not SUSPECT_START_RE.match(scan_lines[i]): + i += 1 + continue + start = i + j = i + 1 + while j < n: + s = scan_lines[j] + if s.startswith(">") or s.strip() == "": + j += 1 + continue + break + block_lines = raw_lines[start:j] + while block_lines and block_lines[-1].strip() == "": + block_lines.pop() + block = "\n".join(block_lines).strip() + anchors = [m.group(0) for m in WIKILINK_RE.finditer(block) if m.group(2)] + out.append({ + "path": p.path, + "title": p.title, + "line": start + 1, + "cited": anchors, + "block": block, + }) + i = max(j, start + 1) + return out + + +def check_citation_ledger_consistency(pages) -> list[dict]: + """台账 ↔ markdown 对账（确定性，堵「删标注蒸发」通道）： + + - ledger-unsupported-without-marker：台账最新判定为 UNSUPPORTED/CONTRADICTED、 + 且 pair 双侧内容都未变（仍是现行 pair），但页面上找不到含该锚点的 CAUTION + 标注块——标注被删而论断未改，错引从两处同时消失 + - marker-without-ledger：markdown 有 CAUTION 标注但台账无对应记录（信息级： + 可能是手写标注，不算错误但列出供核） + """ + issues = [] + suspects = list_suspect_citations(pages) + ledger = _load_ledger() + if not ledger and not suspects: + return issues + data = extract_claims(pages) + current = {p["pair_id"]: p for p in data["pairs"]} + # 页面 → 该页 CAUTION 块里出现过的锚点串集合 + page_marked_anchors: dict[str, set[str]] = {} + for s in suspects: + anchors = {m.group(2).lstrip("^") for m in WIKILINK_RE.finditer(s["block"]) + if m.group(2) and m.group(2).startswith("^")} + page_marked_anchors.setdefault(s["path"], set()).update(anchors) + for pid, rec in ledger.items(): + if rec.get("verdict") not in ("UNSUPPORTED", "CONTRADICTED"): + continue + pair = current.get(pid) + if pair is None or pair["target_content_hash"] != rec.get("target_content_hash"): + continue # 内容已变：论断被修过，标注移除是合法的（新 pair 回到未审） + if pair["anchor"] not in page_marked_anchors.get(pair["page"], set()): + issues.append({ + "issue_type": "ledger-unsupported-without-marker", + "path": pair["page"], + "line": pair["line"], + "anchor": pair["anchor"], + "target": pair["target"], + "verdict": rec.get("verdict"), + "detail": "台账判定未通过且论断未改，但页面上无对应 CAUTION 标注——标注疑被删除", + }) + marked_pairs = {(p, a) for p, anchors in page_marked_anchors.items() for a in anchors} + ledger_pairs = {(r.get("page"), r.get("anchor")) for r in ledger.values() + if r.get("verdict") in ("UNSUPPORTED", "CONTRADICTED")} + for pg, anc in sorted(marked_pairs - ledger_pairs): + issues.append({ + "issue_type": "marker-without-ledger", + "path": pg, + "anchor": anc, + "detail": "页面有 CAUTION 审计标注但台账无对应记录（可能为手写标注，列出供核）", + }) + return issues + + +def _retrieval_log_path() -> Path: + return PROJECT_ROOT / ".cache" / "retrieval_log.jsonl" + + +def _log_retrieval(events: list[dict]) -> None: + """检索凭证台账：记录「何时取回过哪个块 / 节的哪个内容版本」。 + + read-block / read-section / blocks / extract-claims --with-evidence 自动登记。 + 纯派生层（.cache，删了 = 凭证重置，重新取回即可）；check-provenance 用它校验 + 「每条新引用都对应一次真实的原文取回」——没读过就造不出凭证，把 quote-first + 从行为规范升级为可机器校验的溯源链。 + """ + path = _retrieval_log_path() + path.parent.mkdir(parents=True, exist_ok=True) + ts = datetime.now().strftime("%Y-%m-%d %H:%M:%S") + with open(path, "a", encoding="utf-8") as f: + for e in events: + f.write(json.dumps({"ts": ts, **e}, ensure_ascii=False) + "\n") + + +def _load_retrieval_index() -> dict[tuple, set]: + """(target, anchor) → 取回过的内容 hash 集合。""" + path = _retrieval_log_path() + out: dict[tuple, set] = {} + if not path.exists(): + return out + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except Exception: + continue + key = (rec.get("target"), rec.get("anchor")) + if key[0] and key[1] and rec.get("hash"): + out.setdefault(key, set()).add(rec["hash"]) + return out + + +def _log_retrieval_for(target_rel: str, anchor_id: str) -> None: + """按 checker 同口径（_resolve_cited_text 的 text_for_hash）登记一条凭证。""" + r = _resolve_cited_text(target_rel, anchor_id, {}) + if r.get("status") == "ok": + _log_retrieval([{"target": target_rel, "anchor": anchor_id, + "hash": _md5_12(_normalize_for_hash(r["text_for_hash"]))}]) + + +def check_provenance(pages, paths=None) -> dict: + """检索凭证校验：每条可核对引用必须有「取回过被引块当前内容版本」的凭证。 + + 通过条件（任一）： + - 块级凭证：(target, anchor) 事件的 hash == 现行 target_content_hash + - 节级凭证：包含该块的任一祖先 heading 节被取回过（read-section），且事件 + hash == 该节现行 hash——②③ 档分段阅读天然产生节级凭证 + - 全文凭证：(target, "*")（blocks 命令）hash == 现行全文 hash——①档短文 + 用 `k.py blocks` 取全文即产生 + 内容版本必须匹配：读的是旧版、引的是新版 → 凭证失效（防「读过一次终身有效」）。 + 凭证台账是 .cache 派生层：删了 = 全部待重取，重新 read-block 即可。 + """ + data = extract_claims(pages, paths=paths) + index = _load_retrieval_index() + doc_cache: dict[str, object] = {} + findings = [] + checked = 0 + for pr in data["pairs"]: + if pr["target_status"] != "ok": + continue + checked += 1 + target = pr["target"] + # 块级凭证 + if pr["target_content_hash"] in index.get((target, pr["anchor"]), set()): + continue + blocks = _load_doc_blocks(target, doc_cache) + ok = False + if blocks is not None: + # 全文凭证 + full_hash = _md5_12(_normalize_for_hash( + "\n".join(_block_clean_text(b) for b in blocks))) + if full_hash in index.get((target, "*"), set()): + ok = True + else: + # 祖先 heading 节级凭证 + idx = _find_block_by_anchor(blocks, pr["anchor"]) + if idx is not None: + level_cap = 7 + for j in range(idx, -1, -1): + b = blocks[j] + if b.kind == "heading" and (b.level or 1) < level_cap and b.anchor: + level_cap = b.level or 1 + sect_hash = _md5_12(_normalize_for_hash( + _section_text_from_blocks(blocks, j))) + if sect_hash in index.get((target, b.anchor), set()): + ok = True + break + if not ok: + findings.append({ + "page": pr["page"], "line": pr["line"], + "target": target, "anchor": pr["anchor"], + "issue": "no-retrieval-evidence", + "detail": "无「取回过被引块当前内容版本」的检索凭证——writer 未真正打开原文" + "（或读的是旧版本）。用 read-block / read-section 取回后再引用", + }) + return {"checked": checked, "findings": findings} + + +def _binom_cdf(k: int, n: int, prob: float) -> float: + """P(X ≤ k)，X ~ Binomial(n, prob)。n 小（审计样本量级），直接精确求和。""" + from math import comb + return sum(comb(n, i) * prob ** i * (1 - prob) ** (n - i) for i in range(k + 1)) + + +def _clopper_pearson_upper(k: int, n: int, confidence: float = 0.95) -> float: + """二项比例的 Clopper-Pearson 精确置信上界（二分求解，零依赖）。""" + if n == 0: + return 1.0 + if k >= n: + return 1.0 + lo, hi = k / n, 1.0 + for _ in range(60): + mid = (lo + hi) / 2 + if _binom_cdf(k, n, mid) > 1 - confidence: + lo = mid + else: + hi = mid + return hi + + +def audit_confidence(pages, confidence: float = 0.95) -> dict: + """把「审过多少、发现多少问题」换算成可陈述的统计保证（验收抽样口径）。 + + 对现行可核对引用对：n = 已审（台账最新判定且内容双侧未漂移），k = 其中判定 + 未通过（UNSUPPORTED / CONTRADICTED，即已标注待人裁的）。给出未通过率的 + Clopper-Pearson 置信上界。 + + 诚实边界（结果里显式携带）： + - 上界对「已审子集」精确成立；推广到未审部分要求审计样本有代表性 + （seed 随机抽样 / 全覆盖时成立，只审新增时对存量不成立） + - 上界以「验证器判定」为准——验证器自身的查全率由 evals/cite-check + 毒化评测集度量，两者相乘才是对真实错误率的界 + """ + data = extract_claims(pages) + verifiable = [p for p in data["pairs"] if p["target_status"] == "ok"] + audited = [p for p in verifiable if p["audited"]] + failures = [p for p in audited + if p["last_verdict"] in ("UNSUPPORTED", "CONTRADICTED")] + n, k, total = len(audited), len(failures), len(verifiable) + upper = _clopper_pearson_upper(k, n, confidence) + return { + "verifiable_pairs": total, + "audited": n, + "coverage": round(n / total, 4) if total else None, + "failures": k, + "observed_failure_rate": round(k / n, 4) if n else None, + "confidence": confidence, + "failure_rate_upper_bound": round(upper, 4), + "note": ( + f"在 {int(confidence * 100)}% 置信下，已审子集的引用未通过率 ≤ " + f"{upper * 100:.2f}%（n={n}, k={k}）。推广到全库要求审计样本有代表性" + f"（覆盖率 {n}/{total}；seed 随机抽样或全覆盖时成立）；" + f"上界以验证器判定为准，验证器自身查全率见 evals/cite-check 评测。" + ), + } + + +def compact_ledgers() -> dict: + """压缩 .cache 下两个 append-only 流水（纯派生层，重写无风险）： + citation_audit.jsonl 每 pair 只留最新一条；retrieval_log.jsonl 按 + (target, anchor, hash) 去重留最新。长期运行防无界增长。""" + result = {} + lp = _ledger_path() + if lp.exists(): + before = sum(1 for ln in lp.read_text(encoding="utf-8").splitlines() if ln.strip()) + latest = _load_ledger() # 后写覆盖 → 即最新 + lp.write_text("".join(json.dumps(r, ensure_ascii=False) + "\n" + for r in latest.values()), encoding="utf-8") + result["citation_audit"] = {"before": before, "after": len(latest)} + rp = _retrieval_log_path() + if rp.exists(): + lines = [ln for ln in rp.read_text(encoding="utf-8").splitlines() if ln.strip()] + latest2: dict[tuple, str] = {} + for ln in lines: + try: + rec = json.loads(ln) + latest2[(rec.get("target"), rec.get("anchor"), rec.get("hash"))] = ln + except Exception: + continue + rp.write_text("".join(ln + "\n" for ln in latest2.values()), encoding="utf-8") + result["retrieval_log"] = {"before": len(lines), "after": len(latest2)} + return result + + +def list_status_issues(pages) -> list[dict]: + """扫 `status: reviewed` 但 `last_modified_by != Human` 的页面： + reviewed（"已审阅"）语义 = 人类审阅过；LLM 自己写入的页面不该自称已审， + 应保持 `draft`，由人类审阅后才改成 `reviewed` + `last_modified_by: Human`。 + 跳过 deprecated / 归档区（_is_exempt）。 + """ + out = [] + for page in pages: + if _is_exempt(page): + continue + if page.status == "reviewed" and page.last_modified_by != "Human": + out.append({ + "path": page.path, + "title": page.title, + "type": page.type, + "status": page.status, + "last_modified_by": page.last_modified_by, + }) + return out + + +def fmt_status_issues(items: list[dict]): + if not items: + print("✅ 没有发现 status 矛盾（reviewed 均由人类 Human 设置）") + return + print(f"⚠️ 发现 {len(items)} 处 status 矛盾（标 reviewed 但 last_modified_by 非 Human）：\n") + print(" （reviewed=「已审阅」应由人类审阅后设置；LLM 写入的页面应为 draft）\n") + for it in items: + print(f" {it['path']} ({it['title']}, {it['type']}) — status={it['status']}, by={it['last_modified_by']}") + print("\n 修复: 改回 status: draft（或人类审阅后把 last_modified_by 改为 Human）") + print() + + +def fmt_relation_balance(items: list[dict]): + if not items: + print("✅ 关系词频次均衡（无单一关系词占比 > 30%）") + return + print(f"⚠️ 发现 {len(items)} 个关系词占比过高（> 30% 阈值）：\n") + print(f" （标准关系白名单：{sorted(RELATION_TYPES)}）\n") + for it in items: + pct = it["ratio"] * 100 + print(f" {it['relation']}: {it['count']}/{it['total_relations']} = {pct:.0f}% (阈值 {it['threshold']:.0%})") + print(f" 建议: {it['suggestion']}") + print() + + +def fmt_implicit_relations(items: list[dict]): + if not items: + print("✅ 没有发现隐含关系（plain wikilink 配判断/立场动词的段落都已加 RELATION）") + return + print(f"⚠️ 发现 {len(items)} 处隐含关系（plain wikilink + 判断动词，应改 [[?|RELATION]]）：\n") + print(" （判断词: " + "、".join(sorted(_IMPLICIT_RELATION_VERBS_ZH)) + "）\n") + for it in items: + print(f" {it['path']}:{it['line']} ({it['title']}, {it['type']})") + print(f" 判断词: {', '.join(it['matched_verbs'])}") + print(f" plain wikilink: {' '.join(it['plain_wikilinks'])}") + print(f" 片段: {it['preview']}") + print() + + +# ========== i18n 硬编码扫描（list-i18n-violations） ========== +# 扫 web/ 下的 .tsx 文件，找硬编码的中文 UI 字符串。 +# CLAUDE.md "Web 管理台国际化方案" 明文禁止： +# "不允许在组件里写硬编码的中文 / 英文 UI 字符串 +# （除非是 markdown 内容本身的渲染）" +# 但之前没有自动 lint，禁令形同虚设。本扫描器作为守门员。 + +# 中文字符范围（含汉字） +CN_CHAR = r"一-鿿" +# JSX 文本节点：>...内容...< （单行内） +JSX_TEXT_CN_RE = re.compile(rf">([^<>{{}}\n]*[{CN_CHAR}][^<>{{}}\n]*)<") +# UI 相关 JSX 属性的字符串值含中文（白名单几个最常见的 UI 属性） +JSX_ATTR_CN_RE = re.compile( + rf'\b(aria-label|placeholder|title|alt|label)\s*=\s*' + rf'(["\'])([^"\'\n]*[{CN_CHAR}][^"\'\n]*)\2' +) +# ARIA 属性硬编码英文（不限于中文）：aria-label / aria-roledescription / aria-description +# 这两个 ARIA 属性值是给 AT 读出来的"显示文案"，必须 i18n 化。 +# 故意不扫 placeholder/title/alt 的英文——会有大量误报（如 type="text" 与代码标识符） +JSX_ATTR_ENGLISH_ARIA_RE = re.compile( + r'\b(aria-label|aria-roledescription|aria-description)\s*=\s*' + r'(["\'])([A-Za-z][A-Za-z0-9 _\-]*[A-Za-z])\2' +) + +# 跳过的文件（自身就是 i18n 基建——翻译表 / Provider；其他组件包括 LocaleSwitcher +# 都必须通过 t() / useT() 调用翻译表，不允许硬编码字面量） +I18N_SKIP_FILE_NAMES = { + "i18n.ts", + "i18n-client.tsx", +} +# 跳过的目录（API 响应不是 UI 显示层；node_modules / .next 是构建产物） +I18N_SKIP_DIR_PARTS = {"node_modules", ".next", "api"} + + +def list_i18n_violations(web_dir: Path | None = None) -> list[dict]: + """扫 web/ 下 .tsx，找硬编码中文（JSX text + UI 属性）。 + + 跳过： + - node_modules / .next / web/app/api/（API 响应不是 UI） + - i18n.ts / i18n-client.tsx（基建自身） + - // 单行 + /* */ 多行注释（用 mask_js_comments 剥除） + + JSX 文本扫描前先 mask_jsx_expressions 剥除 {…} 表达式——否则被 `{n}` / + `{t("x")}` 打断的混排硬编码中文（如 `共 {n} 条`）会漏报。 + """ + if web_dir is None: + web_dir = ENGINE_ROOT / "web" + out = [] + if not web_dir.exists(): + return out + + for tsx in sorted(web_dir.rglob("*.tsx")): + if tsx.name in I18N_SKIP_FILE_NAMES: + continue + if any(part in I18N_SKIP_DIR_PARTS for part in tsx.parts): continue try: text = tsx.read_text(encoding="utf-8") @@ -2309,6 +4633,75 @@ def fmt_search_results(results): print() +def fmt_corpus_map(data: dict, with_sections=False): + if not _raw_tree_has_files(): + print("（本库 raw/ 未分发或为空——corpus-map 不可用）") + print("这是 release demo / 浅 clone 的预期状态：浏览式检索改走 wiki 层" + "（root_index → MOC → 页面），并说明原文不在场。") + return + docs = data["docs"] + if not docs: + print("（无匹配文档——检查 --file 过滤）") + return + sm = data["summary"] + print(f"库级地图：{sm['total_docs']} 篇 raw 文档，共 {sm['total_chars'] / 1000:.0f}K 字符 | " + f"章节摘要覆盖 {sm['summary_coverage']} | 未 ingest {sm['not_ingested']} 篇") + print("（这是浏览式检索的第一跳：选 2-4 篇候选 → `outline ` 看完整章节树 → " + "`read-section`；标题/摘要只是路标，作答必须读原文块并带锚点）\n") + for d in docs: + src = f"→ {d['source_summary']}" if d["source_summary"] else "→ （未 ingest）" + print(f"── {d['file']} {d['tier']} {d['chars'] / 1000:.1f}K 摘要 {d['summary_coverage']} {src}") + print(f" {d['title']}") + if not d["sections"]: + continue + if with_sections: + for sec in d["sections"]: + mark = f"[{sec['depth_status']}]" if sec["depth_status"] else "" + size = f" {sec['chars'] / 1000:.0f}K" if sec["chars"] >= 1000 else "" + summ = f" — {sec['summary'][:70]}" if sec.get("summary") else "" + print(f" {mark}{sec['title']}{size} (^{sec['anchor']}){summ}") + else: + parts = [] + for sec in d["sections"]: + mark = f"[{sec['depth_status']}]" if sec["depth_status"] else "" + size = f" {sec['chars'] / 1000:.0f}K" if sec["chars"] >= 1000 else "" + parts.append(f"{mark}{sec['title']}{size}") + print(f" {' · '.join(parts)}") + print() + + +def fmt_search_raw(results: list[dict], query: str): + if not results: + if not _raw_tree_has_files(): + print(f"（本库 raw/ 未分发或为空——原文级检索不可用：{query!r}）") + print("这是 release demo / 浅 clone 的预期状态：基于 wiki 蒸馏层作答，") + print("并在答案中说明「原文不在场，细节无法核验」；不要假装检索过原文。") + return + print(f"（raw 原文无命中：{query!r}）") + print("下一步建议：① 换 2-3 组不同措辞的关键词重试（中英同义词/术语/俗称）；") + print("② 用 `k.py outline ` 读章节树+摘要，按理解力定位后 read-section；") + print("③ 若确认知识库未覆盖，如实告知而不是编造。") + return + print(f"raw 原文命中 {len(results)} 块（宽召回——请逐块 read-block 精读裁决，勿只看片段就下结论）：\n") + for r in results: + deepen = " ⊙→可触发 partial re-ingest 深化" if r.get("deepen_hint") else "" + anchor = f"^{r['anchor']}" if r.get("anchor") else f"(无锚，行 {r['line']})" + print(f" [{r['score']}] {r['file']} {anchor} ({r['kind']}, §{r.get('section_title') or '—'}){deepen}") + if r.get("agent_summary"): + print(f" 节摘要: {r['agent_summary'][:80]}") + print(f" {r['snippet'][:160]}") + # 后续命令必须可直接执行：无块锚时按 节锚 > 节标题 > outline 退化，不吐占位符 + if r.get("anchor"): + print(f" → python scripts/k.py read-block {r['file']} {r['anchor']}") + elif r.get("section_anchor"): + print(f" → python scripts/k.py read-section {r['file']} {r['section_anchor']}") + elif r.get("section_title"): + print(f" → python scripts/k.py read-section {r['file']} \"{r['section_title']}\"") + else: + print(f" → python scripts/k.py outline {r['file']} （无锚块，按行 {r['line']} 直接 Read 定位）") + print() + + def fmt_page_list(pages): if not pages: print("（空）") @@ -2412,10 +4805,22 @@ def status_line(label, n): status_line("被引用但缺章节摘要", report.get("unsummarized_sections_count", 0)) status_line("裸论断（含数字但无引用支撑）", report.get("bare_claims_count", 0)) status_line("论断仅整页引用（应升块级 anchor）", report.get("coarse_citations_count", 0)) + status_line("引用数字与被引块不符（cite-mismatch）", report.get("cite_mismatches_count", 0)) + status_line("引用审计未通过标注待处理（suspect）", report.get("suspect_citations_count", 0)) + # 观察 / 信息项（不计闸门）： + print(f" · 锚点挂偏 / 引文未逐字命中（imprecise） {report.get('cite_imprecise_count', 0)}") + print(f" · 被引文件不可得（unverifiable，raw 未分发时为全部候选） {report.get('cite_unverifiable_count', 0)}") + print(f" · [KB 推算] 豁免块 {report.get('cite_exempted_count', 0)}") + print(f" · 引用审计对总数 / 其中未审 {report.get('citation_pairs_count', 0)}" + f" / {report.get('unaudited_citations_count', 0)}（未审数依赖 .cache 台账，删缓存后回升属预期）") status_line("索引 page_count 与 scope 不一致", report.get("index_count_mismatches_count", 0)) status_line("source_count 与 sources 数组不一致 / 论断页缺 source", report.get("source_issues_count", 0)) status_line("status=reviewed 但非人类审阅（last_modified_by≠Human）", report.get("status_issues_count", 0)) status_line("Web 硬编码中文（违反 i18n）", report.get("i18n_violations_count", 0)) + evidence_index = report.get("evidence_index", {}) + evidence_status = evidence_index.get("coverage_status", "missing") + evidence_icon = "✅" if evidence_index.get("ok") else "⚠️ " + print(f" {evidence_icon} 长文档证据索引 {evidence_status}") print() print(f"检查时间: {report['last_check']}") @@ -2699,6 +5104,151 @@ def fmt_coarse_citations(items: list[dict]): print() +_CITE_ISSUE_LABELS = { + "mismatch": "❌ 数字/引文与被引块不符（高置信错引——闸门项）", + "exempt-missing-basis": "❌ [KB 推算] 无依据锚（闸门项）", + "canonical-anchor-mismatch": "❌ 引用锚依赖 hash recovery（canonical 闸门项）", + "canonical-target-mismatch": "❌ 引用目标不是 canonical 完整路径（canonical 闸门项）", + "imprecise-anchor": "⚠️ 锚点挂偏 / 仅超长节命中 / 引文未逐字命中（观察项）", + "unverifiable": "ℹ️ 被引文件不可得（信息项——核对只在 raw 在场环境执行）", + "exempted": "ℹ️ [KB 推算] 生效豁免（信息项）", +} + + +def fmt_cite_mismatches(items: list[dict]): + if not items: + print("✅ 引用核对无发现（数字 / 引文均在被引块原文中，或无可核对项）") + return + by_issue: dict[str, list] = {} + for it in items: + by_issue.setdefault(it["issue"], []).append(it) + gate_issues = { + "mismatch", "exempt-missing-basis", + "canonical-anchor-mismatch", "canonical-target-mismatch", + } + gate = sum(len(by_issue.get(issue, [])) for issue in gate_issues) + print(f"引用核对发现 {len(items)} 条（其中闸门项 {gate} 条）：\n") + for issue in ( + "mismatch", "exempt-missing-basis", "canonical-anchor-mismatch", + "canonical-target-mismatch", "imprecise-anchor", "unverifiable", "exempted"): + group = by_issue.get(issue) + if not group: + continue + print(f"{_CITE_ISSUE_LABELS[issue]} × {len(group)}") + for it in group: + print(f" {it['path']}:{it['line']} ({it['title']}, {it['type']})") + if it.get("numbers"): + print(f" 数字: {', '.join(it['numbers'])}") + if it.get("quotes"): + print(f" 引文: {' / '.join(it['quotes'])}") + cits = ", ".join(f"[[{c['target']}#^{c['anchor']}]]" for c in it.get("citations", [])) + if cits: + print(f" 引用: {cits}") + print(f" 片段: {it['preview']}") + if it.get("suggestion"): + print(f" 修复: {it['suggestion']}") + print() + print("提示：数字共现 ≠ 语义支撑——mismatch 归零只排除确定性可判的错引，" + "曲解 / 过度概括仍需 extract-claims 回验与人审。") + + +def fmt_extract_claims(data: dict): + s = data["summary"] + print(f"审计对枚举：共 {s['pairs_total']} 对 | 可核对 {s['verifiable']}" + f"（已审 {s['audited']} / 未审 {s['unaudited_verifiable']}）" + f" | raw 未分发 {s['unverifiable_raw_missing']} | 失效 {s['broken']}" + f" | 本次返回 {s['returned']}") + for p in data["pairs"]: + mark = {"ok": "·", "raw-not-distributed": "≋", "file-missing": "✗", "anchor-missing": "✗"}.get( + p["target_status"], "?") + audited = f" [{p['last_verdict']}]" if p.get("last_verdict") else "" + print(f" {mark} {p['pair_id']} {p['page']}:{p['line']} → [[{p['target']}#^{p['anchor']}]]" + f" ({p['target_status']}){audited}") + print(f" {p['claim_text'][:120]}") + if data["pairs"]: + print("\n判定入账：python scripts/k.py cite-audit-log --pair --verdict " + "[--evidence \"<被引块原文子串>\"]（agent 记 SUPPORTED 必须附 evidence）") + + +def fmt_cite_audit_log(result: dict): + for r in result["written"]: + print(f"✅ 已入账 {r['pair_id']} {r['verdict']} {r['page']}:{r['line']} → " + f"[[{r['target']}#^{r['anchor']}]] （by={r['by']}, mode={r['mode']}）") + for e in result["errors"]: + print(f"❌ {e['pair_id'] or '<无 pair_id>'}: {e['error']}") + if result["errors"]: + sys.exit(1) + + +def fmt_suspect_citations(items: list[dict], consistency: list[dict] | None = None): + if not items: + print("✅ 没有待处理的「引用审计未通过」标注") + else: + print(f"⚠️ {len(items)} 处引用审计未通过标注待人处理：\n") + for it in items: + print(f" {it['path']}:{it['line']} ({it['title']})") + for c in it.get("cited", []): + print(f" 被引: {c}") + first_line = it["block"].split("\n")[0] + print(f" {first_line}") + print() + if consistency is not None: + if not consistency: + print("✅ 台账 ↔ markdown 对账一致") + else: + print(f"\n⚠️ 台账 ↔ markdown 对账发现 {len(consistency)} 处不一致：") + for c in consistency: + loc = f"{c['path']}:{c.get('line', '?')}" if c.get("line") else c["path"] + print(f" [{c['issue_type']}] {loc} ^{c.get('anchor', '')}") + print(f" {c['detail']}") + + +def fmt_cloze_check(results: list[dict]): + for r in results: + if r.get("error"): + print(f"❌ {r['pair_id']}: {r['error']}") + continue + icon = "✅" if r["passed"] else "❌" + print(f"{icon} {r['pair_id']} {r['page']}:{r['line']}") + for b in r["blanks"]: + mark = "✓" if b["ok"] else "✗" + print(f" {mark} {b['ph']}: 期望 {b['expected']} | 盲填 {b['fill'] or '<空>'}") + + +def fmt_check_draft(result: dict): + s = result["summary"] + gate = s["gate_findings"] + bare = len(result["bare_claims"]) + coarse = len(result["coarse_citations"]) + mode = "strict" if result.get("strict") else "default" + print(f"草稿核对 ({mode}): {result['draft']}") + print(f" 审计对 {s['pairs_total']}（可核对 {s['verifiable']} / raw 缺失 {s['unverifiable_raw_missing']} / 失效 {s['broken']}）") + icon = "✅" if result.get("passed", not (gate or bare or coarse)) else "❌" + if result.get("strict"): + print(f" {icon} 严格闸门 findings {gate}（含 bare/coarse） | " + f"无引用数字论断 {bare} | 仅整页引用 {coarse}") + else: + print(f" {icon} 闸门: mismatch 类 {gate} | 无引用数字论断 {bare} | 仅整页引用 {coarse}") + for f in result["cite_findings"]: + if f["issue"] in ("mismatch", "exempt-missing-basis"): + print(f" ❌ [{f['issue']}] 行{f['line']} 数字: {', '.join(f.get('numbers', []))} — {f.get('suggestion', '')}") + elif (not result.get("strict") + and f["issue"] in ("canonical-anchor-mismatch", "canonical-target-mismatch")): + print(f" ❌ [{f['issue']}] 行{f['line']} — {f.get('suggestion', '')}") + for it in result["bare_claims"]: + print(f" ❌ [无引用] 行{it['line']} 数字: {', '.join(it['matched'])}") + for it in result["coarse_citations"]: + print(f" ❌ [仅整页引用] 行{it['line']} 数字: {', '.join(it['matched'])}") + if result.get("strict"): + for finding in result.get("strict_findings", []): + issue = finding.get("issue", "strict-finding") + loc = finding.get("line", "?") + target = finding.get("target") + suffix = f" → [[{target}#^{finding.get('anchor')}]]" if target else "" + detail = finding.get("detail") or finding.get("suggestion") or "" + print(f" ❌ [{issue}] 行{loc}{suffix} {detail}".rstrip()) + + # ============================================================ # new-workspace 脚手架 # ============================================================ @@ -2798,11 +5348,113 @@ def fmt_new_workspace(result: dict) -> None: print("下一步：") print(f" 1. 把原始资料（HTML/PDF/Word/Markdown）放进 workspaces/{name}/raw/articles/ 或 raw/papers/") print(f" 2. python scripts/convert.py --workspace {name} # 转 markdown + 自动加锚点") - print(f" 3. 让 AI agent 执行摄入（如在 Claude Code 里说「把 raw/... 摄入到 {name} 知识库」或用 /kb-ingest）") - print(f" 4. python scripts/k.py --workspace {name} health # 体检") + print(f" 3. python scripts/k.py --workspace {name} rebuild-evidence-index # 建全细节证据地图") + print(f" 4. 让 AI agent 执行摄入（如在 Claude Code 里说「把 raw/... 摄入到 {name} 知识库」或用 /kb-ingest）") + print(f" 5. python scripts/k.py --workspace {name} health # 体检") print(f" Web 端：cd web && KB_WORKSPACE={name} npm run dev # 或在顶栏 workspace 切换器里切换") +def _evidence_index_path() -> Path: + """Return the disposable evidence-index path for the active workspace.""" + + return PROJECT_ROOT / ".cache" / "retrieval_index.db" + + +def fmt_evidence_index(report: dict, *, rebuilt: bool = False) -> None: + status = report.get("coverage_status", "unknown") + mark = "✅" if report.get("ok") else "❌" + action = "重建完成" if rebuilt else "覆盖检查" + print(f"{mark} 证据索引{action}：{status}") + natural = report.get("natural_units", {}) + structural = report.get("structural_sections", {}) + sections = report.get("content_sections", {}) + print( + f" 自然单元: {natural.get('indexed', 0)}/{natural.get('expected', 0)} " + f"({natural.get('coverage_pct') if natural.get('coverage_pct') is not None else 'undefined'}%)" + ) + print( + f" 结构章节: {structural.get('registered', 0)}/{structural.get('expected', 0)} " + f"({structural.get('coverage_pct') if structural.get('coverage_pct') is not None else 'undefined'}%)" + ) + print( + f" 内容章节: {sections.get('registered', 0)}/{sections.get('expected', 0)} " + f"({sections.get('coverage_pct') if sections.get('coverage_pct') is not None else 'undefined'}%)" + ) + manifest = report.get("manifest", {}) + if manifest and not manifest.get("ok"): + issues = manifest.get("issues", []) + print(f" 完整清单一致性: 失败（{len(issues)} 项）") + for issue in issues[:3]: + print( + f" - {issue.get('path', '?')}: {issue.get('field', '?')} " + f"[{issue.get('issue', 'mismatch')}]" + ) + freshness = report.get("corpus_freshness", {}) + if freshness and not freshness.get("ok"): + print( + " raw 变化: " + f"新增 {len(freshness.get('added', []))} / " + f"删除 {len(freshness.get('removed', []))} / " + f"改写 {len(freshness.get('changed', []))}" + ) + empty_count = int(report.get("unexpected_empty_sections_count", 0) or 0) + if empty_count: + print(f" ⚠️ 发现 {empty_count} 个无自然单元的空结构章节，请确认是否为故意占位") + for warning in report.get("warnings", []): + print(f" ⚠️ {warning}") + + +def fmt_evidence_search(result: dict) -> None: + hits = result.get("hits", []) + if not hits: + print("（没有证据候选）") + return + for hit in hits: + heading = " / ".join(hit.get("heading_path", [])) + print( + f"{hit['rank']:>2}. [[{hit['canonical_ref']}]] " + f"[{hit['kind']}#{hit['subordinal']}] score={hit['score']:.6f}" + ) + print( + f" unit_id={hit['unit_id']} selection=natural_unit " + "markdown_citation=parent_block" + ) + if heading: + print(f" {heading}") + if hit.get("text_truncated"): + print( + f" ⚠️ 超长自然单元：显示命中附近 " + f"{hit.get('text_excerpt_start', 0)}:{hit.get('text_excerpt_end', 0)} / " + f"{hit.get('text_chars', '?')} 字符" + ) + preview = re.sub(r"\s+", " ", hit.get("text", "")).strip() + print(f" {preview[:260]}{'…' if len(preview) > 260 else ''}") + + +def fmt_evidence_unit(unit: dict) -> None: + print(f"unit_id: {unit['unit_id']}") + print(f"selection_scope: {unit['selection_scope']}") + print(f"markdown_citation_scope: {unit['citation_scope']}") + print(f"parent_ref: [[{unit['canonical_ref']}]]") + print(f"kind/subordinal: {unit['kind']} / {unit['subordinal']}") + print(f"content_hash: {unit['content_hash']}") + if unit.get("exact_text_hash"): + print(f"exact_text_hash: {unit['exact_text_hash']}") + print(f"text_chars: {unit.get('text_chars', len(unit.get('text', '')))}") + print() + print(unit["text"]) + + +def fail_evidence_index(exc: RetrievalIndexError, *, as_json: bool) -> None: + if as_json: + output_json(exc.to_dict()) + else: + print(f"错误: {exc.message}", file=sys.stderr) + if exc.details: + print(json.dumps(exc.details, ensure_ascii=False, indent=2), file=sys.stderr) + sys.exit(2) + + # ============================================================ # CLI # ============================================================ @@ -2832,6 +5484,46 @@ def _main_impl(): sub = parser.add_subparsers(dest="cmd", required=True) p_search = sub.add_parser("search", help="跨 wiki 关键词搜索", parents=[common]) + sub.add_parser( + "rebuild-evidence-index", + help="从 raw/**/*.md 原子重建自然单元 SQLite/FTS5 证据索引", + parents=[common], + ) + sub.add_parser( + "evidence-index-coverage", + help="核对自然单元/内容章节机械覆盖率及 raw 内容新鲜度", + parents=[common], + ) + p_se = sub.add_parser( + "search-evidence", + help="检索 raw 自然证据单元（unicode61 BM25 + trigram + exact + RRF）", + parents=[common], + ) + p_se.add_argument("query", help="原始问题或一个证据分面查询") + p_se.add_argument("--limit", type=int, default=20, help="返回候选数（1-1000，默认 20）") + p_se.add_argument( + "--expand", action="append", default=[], + help="显式别名/译名/分面扩展；可重复传入，不会隐式调用模型", + ) + p_reu = sub.add_parser( + "read-evidence-unit", + help="按 unit_id 读取精确 list item/table row/paragraph 等自然单元", + parents=[common], + ) + p_reu.add_argument("unit_id", help="search-evidence 返回的 32 位 unit_id") + p_reu.add_argument( + "--max-chars", type=int, default=30_000, + help="安全读取上限（默认 30000；0=显式无限制人工逃生口）", + ) + p_sr = sub.add_parser("search-raw", help="块级全文检索 raw 原文（宽召回机械原语，命中返回可 read-block 的锚点；语义裁决交给 agent 精读——协议见 kb-query 第 4.7 步）", parents=[common]) + p_sr.add_argument("query", help="关键词（空格分隔多词；建议换 2-3 组措辞多试）") + p_sr.add_argument("--limit", type=int, default=10) + p_sr.add_argument("--file", dest="file_filter", help="限定文件（路径子串，如 raw/papers/2024-09-op-rag）") + p_sr.add_argument("--include-wiki", action="store_true", help="同时块级检索 wiki 页面") + p_cm = sub.add_parser("corpus-map", help="raw 层全库文档地图（标题/章节树/摘要覆盖/深度登记/ingest 状态）——LLM 浏览式检索的第一跳，协议见 kb-query 第 4.7 步浏览路", parents=[common]) + p_cm.add_argument("--file", dest="file_filter", help="限定文件（路径子串）") + p_cm.add_argument("--depth", type=int, default=2, help="章节列出深度（默认 2 = 顶层 H2；3 含 H3）") + p_cm.add_argument("--sections", action="store_true", help="每章节独立一行并显示摘要/首段预览（默认标题内联一行）") p_search.add_argument("query") p_search.add_argument("--limit", type=int, default=20) @@ -2863,10 +5555,20 @@ def _main_impl(): p_rs = sub.add_parser("read-section", help="按 anchor 或标题读取整个 H 段", parents=[common]) p_rs.add_argument("path", help="md 文件路径") p_rs.add_argument("anchor_or_title", help="anchor（如 h-2-3-abc123，可加 ^ 前缀）或标题文本") + p_rs.add_argument( + "--max-chars", + type=int, + default=30000, + help="单次返回字符上限（默认 30000；0 = 显式不限制）", + ) p_rb = sub.add_parser("read-block", help="按 anchor 读取单个块（^p-/^t-/^c-/^f-）", parents=[common]) p_rb.add_argument("path", help="md 文件路径") p_rb.add_argument("anchor", help="块锚点（如 p-12-7d8e9a，可加 ^ 前缀）") + p_rb.add_argument( + "--max-chars", type=int, default=30000, + help="单次返回字符上限（默认 30000；0 = 显式不限制）", + ) p_blocks = sub.add_parser("blocks", help="列出文档所有 block（含 anchor / 类型 / 所属章节 / preview）", parents=[common]) p_blocks.add_argument("path", help="md 文件路径（相对项目根）") @@ -2884,7 +5586,59 @@ def _main_impl(): sub.add_parser("list-broken-refs", help="扫描 wiki/ 中失效的 [[raw/...#^anchor]] 引用", parents=[common]) sub.add_parser("list-unsummarized", help="扫描被 wiki 章节引用但 outline.json 中 agent_summary 为 null 的章节", parents=[common]) sub.add_parser("list-bare-claims", help="扫描含数字 / 百分比 / NLP 指标但无 [[raw/...]] 或 [需要来源] 支撑的段落", parents=[common]) - sub.add_parser("list-coarse-citations", help="扫描含数字论断但只挂整页 [[raw/X]]、未到块级 [[raw/X#^anchor]] 的段落（引用粒度不足）", parents=[common]) + sub.add_parser("list-coarse-citations", help="扫描含数字论断但只挂整页 [[raw/X]] / [[wiki/sources/X]]、未到块级 #^anchor 的段落（引用粒度不足）", parents=[common]) + sub.add_parser("list-unmapped-claims", help="strict coverage：扫描无块级证据/显式披露的定性句、list item、table data row 与事实型 callout", parents=[common]) + sub.add_parser("list-cite-mismatches", help="确定性引用核对：论断中的数字 / 逐字引文是否出现在被引块原文中（不在场 = 高置信错引）", parents=[common]) + + p_ec = sub.add_parser("extract-claims", help="确定性枚举（论断块, 块级引用）审计对——语义回验 / 批量审计的取数层", parents=[common]) + p_ec.add_argument("--paths", nargs="+", help="只枚举这些 wiki 页（相对 workspace 根，如 wiki/sources/x.md）") + p_ec.add_argument("--changed", action="store_true", help="只枚举 git 判定的本次改动页（工作区 vs HEAD + 未跟踪）") + p_ec.add_argument("--commit", help="只枚举指定 commit 改动的页（事后对账；证据取自现行工作树）") + p_ec.add_argument("--unaudited-only", action="store_true", help="只返回未审 / 内容已漂移的可核对 pair（join 台账）") + p_ec.add_argument("--sample", type=int, help="可复现抽样条数（与 --seed 搭配，同 seed 重跑取样一致）") + p_ec.add_argument("--seed", default="", help="抽样种子（建议用周号如 2026-W27，覆盖可累积）") + p_ec.add_argument("--with-evidence", action="store_true", help="内联被引块原文（供组装 verifier 核验包）") + p_ec.add_argument("--max-evidence-chars", type=int, default=1500, help="evidence 截断上限（默认 1500）") + p_ec.add_argument("--cloze", action="store_true", help="附带盲填挖空（数字→⟦N1⟧），供盲填复核；判分用 cloze-check") + + p_cc = sub.add_parser("cloze-check", help="盲填复核判分：核验者只看挖空论断+被引原文填数，本命令按数值容差机器判分", parents=[common]) + p_cc.add_argument("--pair", help="pair_id（extract-claims --cloze 输出）") + p_cc.add_argument("--fills", help='单条填空 JSON，如 \'{"N1": "72.4%%"}\'') + p_cc.add_argument("--batch", help="批量：JSONL 文件，每行 {pair_id, fills}") + p_cc.add_argument("--draft", help="对草稿文件（而非 wiki）枚举 pair 判分（配合 check-draft）") + + sub.add_parser("compact-ledgers", help="压缩 .cache 台账与检索凭证流水（每 pair/凭证只留最新，防无界增长）", parents=[common]) + + p_ac = sub.add_parser("audit-confidence", help="审计覆盖的统计保证：已审引用未通过率的 Clopper-Pearson 置信上界（验收抽样口径）", parents=[common]) + p_ac.add_argument("--confidence", type=float, default=0.95) + + p_cp = sub.add_parser("check-provenance", help="检索凭证校验：每条可核对引用须有「取回过被引块当前内容版本」的凭证（read-block/read-section/blocks 自动登记）", parents=[common]) + p_cp.add_argument("--paths", nargs="+", help="只校验这些 wiki 页") + p_cp.add_argument("--changed", action="store_true", help="只校验 git 判定的本次改动页") + p_cp.add_argument("--commit", help="校验指定 commit 改动的页（事后对账）") + + p_cd = sub.add_parser("check-draft", help="对草稿文件（查询答案 / 导出稿）跑与 wiki 同一套引用核对：数字核对 + 裸论断 + 审计对枚举", parents=[common]) + p_cd.add_argument("draft_path", help="草稿文件路径（任意可读 markdown）") + p_cd.add_argument("--with-evidence", action="store_true") + p_cd.add_argument("--cloze", action="store_true") + p_cd.add_argument("--max-evidence-chars", type=int, default=1500) + p_cd.add_argument( + "--strict", action="store_true", + help="严格 fail-closed：broken/unverifiable/imprecise/pending/provenance/语义未审/截断均阻断", + ) + + p_cal = sub.add_parser("cite-audit-log", help="把外部 agent / 人的引用审计判定写入验证台账（.cache/citation_audit.jsonl，纯派生层）", parents=[common]) + p_cal.add_argument("--pair", help="pair_id（extract-claims 输出）") + p_cal.add_argument("--verdict", help="SUPPORTED / PARTIAL / UNSUPPORTED / CONTRADICTED / UNVERIFIABLE") + p_cal.add_argument("--note", default="", help="判定理由一句（UNSUPPORTED 时写明差异）") + p_cal.add_argument("--evidence", default="", help="被引块现行原文的字面子串（agent 记 SUPPORTED 必填，≥8 字符）") + p_cal.add_argument("--by", choices=["agent", "human"], default="agent") + p_cal.add_argument("--mode", choices=["audit", "ingest", "manual"], default="audit") + p_cal.add_argument("--batch", help="批量入账：JSONL 文件，每行 {pair_id, verdict, note?, evidence?}") + p_cal.add_argument("--draft", help="对查询草稿的 pair 入账（须与 extract/check-draft 时为同一文件）") + + p_lsc = sub.add_parser("list-suspect-citations", help="扫 markdown 中「引用审计未通过」CAUTION 标注块（待人处理错引清单）", parents=[common]) + p_lsc.add_argument("--check-ledger", action="store_true", help="附带台账 ↔ markdown 对账（堵「删标注蒸发」通道）") sub.add_parser("list-index-mismatches", help="扫描 type=index 页：page_count 字段与 scope 实际匹配数不一致的", parents=[common]) sub.add_parser("list-source-issues", help="扫描 source_count 与 sources 数组不一致 / 论断页缺 source 不标 #to-be-updated 等问题", parents=[common]) sub.add_parser("list-status-issues", help="扫描 status=reviewed 但 last_modified_by≠Human 的矛盾页（LLM 写入页自称已审阅）", parents=[common]) @@ -2985,13 +5739,77 @@ def _main_impl(): pages = load_all_wiki_pages() - if args.cmd == "search": + if args.cmd == "rebuild-evidence-index": + try: + report = rebuild_evidence_index(PROJECT_ROOT, _evidence_index_path()) + except RetrievalIndexError as exc: + fail_evidence_index(exc, as_json=args.json) + if args.json: + output_json(report) + else: + fmt_evidence_index(report, rebuilt=True) + if not report.get("ok", False): + sys.exit(1) + + elif args.cmd == "evidence-index-coverage": + try: + report = evidence_index_coverage(_evidence_index_path()) + except RetrievalIndexError as exc: + fail_evidence_index(exc, as_json=args.json) + if args.json: + output_json(report) + else: + fmt_evidence_index(report) + if not report.get("ok", False): + sys.exit(1) + + elif args.cmd == "search-evidence": + try: + results = search_evidence( + _evidence_index_path(), args.query, limit=args.limit, + expansions=args.expand, + ) + except RetrievalIndexError as exc: + fail_evidence_index(exc, as_json=args.json) + if args.json: + output_json(results) + else: + fmt_evidence_search(results) + + elif args.cmd == "read-evidence-unit": + try: + unit = read_evidence_unit( + _evidence_index_path(), args.unit_id, max_chars=args.max_chars + ) + except RetrievalIndexError as exc: + fail_evidence_index(exc, as_json=args.json) + if args.json: + output_json(unit) + else: + fmt_evidence_unit(unit) + + elif args.cmd == "search": results = search_pages(args.query, pages, args.limit) if args.json: output_json(results) else: fmt_search_results(results) + elif args.cmd == "search-raw": + results = search_raw(args.query, pages=pages, limit=args.limit, + file_filter=args.file_filter, include_wiki=args.include_wiki) + if args.json: + output_json(results) + else: + fmt_search_raw(results, args.query) + + elif args.cmd == "corpus-map": + data = corpus_map(pages, file_filter=args.file_filter, depth=args.depth) + if args.json: + output_json(data) + else: + fmt_corpus_map(data, with_sections=args.sections) + elif args.cmd == "list-pages": filters = { "type": args.filter_type, @@ -3080,10 +5898,12 @@ def _main_impl(): elif args.cmd == "read-section": md_path = resolve_doc_path(args.path) try: - sec = read_section(md_path, args.anchor_or_title) + sec = read_section(md_path, args.anchor_or_title, max_chars=args.max_chars) except (FileNotFoundError, LookupError) as e: print(f"错误: {e}", file=sys.stderr) sys.exit(2) + if sec.get("anchor"): + _log_retrieval_for(_to_rel_posix(md_path), sec["anchor"]) # 节级检索凭证 if args.json: output_json(sec) else: @@ -3092,10 +5912,11 @@ def _main_impl(): elif args.cmd == "read-block": md_path = resolve_doc_path(args.path) try: - blk = read_block(md_path, args.anchor) + blk = read_block(md_path, args.anchor, max_chars=args.max_chars) except (FileNotFoundError, LookupError) as e: print(f"错误: {e}", file=sys.stderr) sys.exit(2) + _log_retrieval_for(_to_rel_posix(md_path), blk["anchor"]) # 块级检索凭证 if args.json: output_json(blk) else: @@ -3108,6 +5929,15 @@ def _main_impl(): except FileNotFoundError as e: print(f"错误: 文件不存在: {e}", file=sys.stderr) sys.exit(2) + try: + _blks = parse_blocks_with_anchors(md_path) + _log_retrieval([{ # 全文检索凭证（blocks 返回所有块原文） + "target": _to_rel_posix(md_path), "anchor": "*", + "hash": _md5_12(_normalize_for_hash( + "\n".join(_block_clean_text(b) for b in _blks))), + }]) + except Exception: + pass if args.json: output_json(data) else: @@ -3164,6 +5994,188 @@ def _main_impl(): else: fmt_coarse_citations(items) + elif args.cmd == "list-unmapped-claims": + items = list_unmapped_claims(pages) + if args.json: + output_json(items) + elif not items: + print("✅ 没有未映射的事实论断") + else: + print(f"❌ {len(items)} 条事实论断缺少块级证据/显式披露：") + for item in items: + print(f" {item['path']}:{item['line']} {item['claim_text']}") + + elif args.cmd == "list-cite-mismatches": + items = list_cite_mismatches(pages) + if args.json: + output_json(items) + else: + fmt_cite_mismatches(items) + + elif args.cmd == "extract-claims": + paths = args.paths + if args.changed or args.commit: + changed = _git_changed_wiki_paths(args.commit) + paths = sorted(set(changed) | set(paths or [])) + if not paths: + # 无改动：返回空集而非全库（闸门语义——本次没有需要回验的对） + paths = ["__no_changed_files__"] + data = extract_claims( + pages, + paths=paths, + unaudited_only=args.unaudited_only, + sample=args.sample, + seed=args.seed, + with_evidence=args.with_evidence, + max_evidence_chars=args.max_evidence_chars, + cloze=args.cloze, + ) + if args.with_evidence: + # 核验包内含被引原文 → 等同取回，登记检索凭证（仅未截断的完整原文） + _log_retrieval([ + {"target": pr["target"], "anchor": pr["anchor"], + "hash": pr["target_content_hash"]} + for pr in data["pairs"] + if pr["target_status"] == "ok" and not pr.get("evidence_truncated") + ]) + if args.json: + output_json(data) + else: + fmt_extract_claims(data) + + elif args.cmd == "cite-audit-log": + if args.batch: + batch_path = Path(args.batch) + if not batch_path.exists(): + print(f"错误: batch 文件不存在: {args.batch}", file=sys.stderr) + sys.exit(2) + entries = [] + for ln in batch_path.read_text(encoding="utf-8").splitlines(): + ln = ln.strip() + if not ln: + continue + try: + entries.append(json.loads(ln)) + except Exception: + print(f"错误: batch 文件含非法 JSON 行: {ln[:80]}", file=sys.stderr) + sys.exit(2) + else: + if not args.pair or not args.verdict: + print("错误: 需要 --pair 与 --verdict（或 --batch ）", file=sys.stderr) + sys.exit(2) + entries = [{"pair_id": args.pair, "verdict": args.verdict, + "note": args.note, "evidence": args.evidence}] + draft_path = Path(args.draft).expanduser() if args.draft else None + if draft_path is not None and not draft_path.exists(): + print(f"错误: 草稿文件不存在: {args.draft}", file=sys.stderr) + sys.exit(2) + result = cite_audit_log_batch( + entries, by=args.by, mode=args.mode, draft_path=draft_path) + if args.json: + output_json(result) + if result["errors"]: + sys.exit(1) + else: + fmt_cite_audit_log(result) + + elif args.cmd == "list-suspect-citations": + items = list_suspect_citations(pages) + consistency = check_citation_ledger_consistency(pages) if args.check_ledger else None + if args.json: + output_json({"suspects": items, "ledger_consistency": consistency} + if consistency is not None else items) + else: + fmt_suspect_citations(items, consistency) + + elif args.cmd == "cloze-check": + if args.batch: + bp = Path(args.batch) + if not bp.exists(): + print(f"错误: batch 文件不存在: {args.batch}", file=sys.stderr) + sys.exit(2) + entries = [json.loads(ln) for ln in bp.read_text(encoding="utf-8").splitlines() if ln.strip()] + else: + if not args.pair or not args.fills: + print("错误: 需要 --pair 与 --fills（或 --batch ）", file=sys.stderr) + sys.exit(2) + entries = [{"pair_id": args.pair, "fills": json.loads(args.fills)}] + draft = Path(args.draft).expanduser() if args.draft else None + if draft is not None and not draft.exists(): + print(f"错误: 草稿文件不存在: {args.draft}", file=sys.stderr) + sys.exit(2) + results = cloze_check_batch(entries, draft_path=draft) + if args.json: + output_json(results) + else: + fmt_cloze_check(results) + if any(not r.get("passed") for r in results): + sys.exit(1) + + elif args.cmd == "compact-ledgers": + result = compact_ledgers() + if args.json: + output_json(result) + else: + if not result: + print("（无台账文件，无需压缩）") + for name, st in result.items(): + print(f"✅ {name}: {st['before']} → {st['after']} 条") + + elif args.cmd == "audit-confidence": + result = audit_confidence(pages, confidence=args.confidence) + if args.json: + output_json(result) + else: + print(f"引用审计统计保证（{int(result['confidence']*100)}% 置信）") + print(f" 可核对引用对: {result['verifiable_pairs']} | 已审: {result['audited']}" + f"（覆盖率 {result['coverage']}）| 未通过: {result['failures']}") + print(f" 未通过率上界: {result['failure_rate_upper_bound']*100:.2f}%") + print(f" {result['note']}") + + elif args.cmd == "check-provenance": + paths = args.paths + if args.changed or args.commit: + changed = _git_changed_wiki_paths(args.commit) + paths = sorted(set(changed) | set(paths or [])) or ["__no_changed_files__"] + result = check_provenance(pages, paths=paths) + if args.json: + output_json(result) + else: + if not result["findings"]: + print(f"✅ 检索凭证校验通过（{result['checked']} 条可核对引用均有当前版本取回记录）") + else: + print(f"❌ {len(result['findings'])} / {result['checked']} 条引用缺检索凭证：") + for f in result["findings"]: + print(f" {f['page']}:{f['line']} → [[{f['target']}#^{f['anchor']}]]") + print(f" {f['detail']}") + if result["findings"]: + sys.exit(1) + + elif args.cmd == "check-draft": + dp = Path(args.draft_path).expanduser() + if not dp.exists(): + print(f"错误: 草稿文件不存在: {args.draft_path}", file=sys.stderr) + sys.exit(2) + result = check_draft(dp, with_evidence=args.with_evidence, + max_evidence_chars=args.max_evidence_chars, cloze=args.cloze, + strict=args.strict) + if args.with_evidence: + # 与 extract-claims --with-evidence 同口径：核验包确实取回了 + # 当前内容版本时登记 provenance。仅完整 evidence 可入账， + # 截断包不能作为「真读过整个被引块/节」的凭证。 + _log_retrieval([ + {"target": pair["target"], "anchor": pair["anchor"], + "hash": pair["target_content_hash"]} + for pair in result["pairs"] + if pair["target_status"] == "ok" and not pair.get("evidence_truncated") + ]) + if args.json: + output_json(result) + else: + fmt_check_draft(result) + if not result["passed"]: + sys.exit(1) + elif args.cmd == "list-index-mismatches": items = list_index_count_mismatches(pages) if args.json: diff --git a/scripts/pdf_layout.py b/scripts/pdf_layout.py new file mode 100644 index 0000000..f2eb7c0 --- /dev/null +++ b/scripts/pdf_layout.py @@ -0,0 +1,1879 @@ +"""Deterministic, layout-aware PDF to Markdown extraction. + +The generic PDF converter used by :mod:`markitdown` is deliberately small, but +PDF text streams do not promise reading order. In particular, a two-column +page is commonly returned row-by-row (left fragment, right fragment, left +fragment, ...). That order destroys both retrieval phrases and citation +meaning. + +This module uses the geometry already exposed by ``pdfplumber`` (a dependency +of ``markitdown[all]``) and applies a conservative reading-order algorithm: + +* tables are detected first and emitted as Markdown tables; +* text is reconstructed into visual line fragments; +* pages with independently flowing columns are read column-major; +* full-width elements split a multi-column page into vertical bands; +* repeated running headers/footers and page-number furniture are de-duplicated; +* visual font information is used only to infer Markdown headings. + +There is intentionally no fallback to the generic PDF path. An encrypted, +image-only, content-image-bearing, corrupt, or otherwise unextractable PDF raises +``PDFLayoutExtractionError`` so callers cannot mistake a lossy/empty result for +a faithful conversion. Small decorative images are reported in diagnostics; +content-sized images require a separate visual/OCR figure workflow. OCR is +outside this deterministic module's scope. +""" + +from __future__ import annotations + +from collections import Counter, defaultdict +from dataclasses import dataclass, field +import math +from pathlib import Path +import re +from statistics import median +from typing import Any, Sequence + + +# Bump this whenever the semantics of the layout extractor change. convert.py +# binds the value into each non-Markdown source's conversion receipt, so a PDF +# produced by an older/generic converter is never accepted merely because its +# derived Markdown and outline agree with each other. +PDF_LAYOUT_CONVERTER_VERSION = 5 + + +class PDFLayoutExtractionError(RuntimeError): + """Raised when a PDF cannot be converted with an auditable text layout.""" + + +@dataclass(frozen=True) +class PDFExtractionDiagnostics: + page_count: int + text_pages: int + two_column_pages: tuple[int, ...] + image_pages: tuple[int, ...] + table_count: int + suppressed_running_elements: int + + +@dataclass(frozen=True) +class PDFExtractionResult: + markdown: str + diagnostics: PDFExtractionDiagnostics + + +@dataclass +class _Line: + page_number: int + x0: float + x1: float + top: float + bottom: float + text: str + font_size: float + bold_ratio: float + row_id: int + heading_level: int | None = None + suppressed: bool = False + + +@dataclass +class _Table: + page_number: int + x0: float + x1: float + top: float + bottom: float + rows: list[list[str]] + caption: str | None = None + cell_bboxes: tuple[tuple[float, float, float, float], ...] = () + header_band_bbox: tuple[float, float, float, float] | None = None + + +@dataclass +class _Page: + number: int + width: float + height: float + lines: list[_Line] = field(default_factory=list) + tables: list[_Table] = field(default_factory=list) + has_images: bool = False + material_image_count: int = 0 + material_vector_count: int = 0 + column_split: float | None = None + key_value_rows: set[int] = field(default_factory=set) + + +_EXPLICIT_PAGE_NUMBER_RE = re.compile( + r"^(?:page\s*\d+(?:\s*(?:/|of)\s*\d+)?|\d+\s*(?:/|of)\s*\d+)$", + re.IGNORECASE, +) +_BARE_PAGE_NUMBER_RE = re.compile(r"^\d{1,3}$") +_RUNNING_DIGITS_RE = re.compile(r"\d+") +_CID_RE = re.compile(r"\(cid:\d+\)", re.IGNORECASE) +_SPACE_RE = re.compile(r"\s+") +# pdfminer/pdfplumber expose the WinAnsi bullet used by ReportLab/Acrobat as +# ``(cid:127)`` when the font lacks a Unicode map. Only accept that placeholder +# at visual line start; elsewhere it remains an unresolved-glyph error. +_LEADING_BULLET_RE = re.compile(r"^(?:[•◦▪‣⁃●○■□]|\(cid:127\))\s*") +_BULLET_ONLY_RE = re.compile(r"^(?:[•◦▪‣⁃●○■□]|\(cid:127\))$") +_REPLACEMENT_CHAR = "\ufffd" +_VALUE_LIKE_RE = re.compile( + r"^\s*(?:" + r"[-+]?(?:\d+(?:\.\d+)?|\.\d+)\s*(?:%|bar|kpa|mpa|kg|ms|s|hz|v|a)?" + r"|yes|no|enabled|disabled|accept(?:ed)?|reject(?:ed)?" + r")\s*$", + re.IGNORECASE, +) + + +def _normalise_space(value: object) -> str: + return _SPACE_RE.sub(" ", str(value or "").replace("\x00", " ")).strip() + + +def _escape_table_cell(value: object) -> str: + text = _normalise_space(value) + return text.replace("\\", "\\\\").replace("|", "\\|") + + +def _normalise_list_marker(text: str) -> str: + """Translate only a visual line-leading bullet into Markdown list syntax.""" + return _LEADING_BULLET_RE.sub("- ", text, count=1) + + +def _material_image_count( + images: Sequence[dict[str, Any]], page_width: float, page_height: float +) -> int: + """Count raster regions large enough to carry substantive page content. + + Tiny logos/icons are allowed and surfaced through ``image_pages``. A + content-sized image cannot be represented faithfully by positioned-text + extraction, so accuracy-first conversion must stop instead of silently + dropping it. The aggregate guard also catches tiled scans. + """ + page_area = max(page_width * page_height, 1.0) + material = 0 + aggregate_area = 0.0 + for image in images: + try: + width = abs(float(image.get("x1", 0)) - float(image.get("x0", 0))) + if not width: + width = abs(float(image.get("width", 0))) + if "top" in image and "bottom" in image: + height = abs(float(image["bottom"]) - float(image["top"])) + else: + height = abs(float(image.get("y1", 0)) - float(image.get("y0", 0))) + if not height: + height = abs(float(image.get("height", 0))) + except (TypeError, ValueError): + # Unknown image geometry is not safe to ignore. + material += 1 + continue + area = max(width, 0.0) * max(height, 0.0) + aggregate_area += min(area, page_area) + width_ratio = width / max(page_width, 1.0) + height_ratio = height / max(page_height, 1.0) + area_ratio = area / page_area + if ( + area_ratio >= 0.025 + and width_ratio >= 0.12 + and height_ratio >= 0.08 + ): + material += 1 + if material == 0 and aggregate_area / page_area >= 0.08: + material = 1 + return material + + +def _object_bbox(shape: dict[str, Any]) -> tuple[float, float, float, float] | None: + """Return a normalised pdfplumber bbox, or ``None`` for unsafe geometry.""" + try: + x0 = float(shape["x0"]) + x1 = float(shape["x1"]) + top = float(shape["top"]) + bottom = float(shape["bottom"]) + except (KeyError, TypeError, ValueError): + return None + if not all(math.isfinite(value) for value in (x0, x1, top, bottom)): + return None + return min(x0, x1), min(top, bottom), max(x0, x1), max(top, bottom) + + +def _bbox_inside( + inner: tuple[float, float, float, float], + outer: tuple[float, float, float, float], + *, + tolerance: float = 1.5, +) -> bool: + return ( + inner[0] >= outer[0] - tolerance + and inner[1] >= outer[1] - tolerance + and inner[2] <= outer[2] + tolerance + and inner[3] <= outer[3] + tolerance + ) + + +def _bbox_near( + first: tuple[float, float, float, float], + second: tuple[float, float, float, float], + padding: float, +) -> bool: + return not ( + first[2] + padding < second[0] + or second[2] + padding < first[0] + or first[3] + padding < second[1] + or second[3] + padding < first[1] + ) + + +def _is_page_border( + bbox: tuple[float, float, float, float], page_width: float, page_height: float +) -> bool: + """Ignore a single near-page-size frame used as page furniture.""" + width = bbox[2] - bbox[0] + height = bbox[3] - bbox[1] + return ( + width >= page_width * 0.90 + and height >= page_height * 0.90 + and bbox[0] <= page_width * 0.06 + and bbox[1] <= page_height * 0.06 + ) + + +def _is_table_vector_furniture( + bbox: tuple[float, float, float, float], + kind: str, + shape: dict[str, Any], + table: _Table, +) -> bool: + """Return true only for verified, non-semantic table grid furniture.""" + table_bbox = (table.x0, table.top, table.x1, table.bottom) + if not _bbox_inside(bbox, table_bbox): + return False + width = bbox[2] - bbox[0] + height = bbox[3] - bbox[1] + + def same(first: float, second: float, tolerance: float = 1.25) -> bool: + return abs(first - second) <= tolerance + + def same_bbox( + first: tuple[float, float, float, float], + second: tuple[float, float, float, float], + ) -> bool: + return all( + abs(actual - expected) <= 0.35 + for actual, expected in zip(first, second) + ) + + def interval_covered( + start: float, end: float, intervals: list[tuple[float, float]] + ) -> bool: + cursor = start + for left, right in sorted(intervals): + if right < cursor - 1.25: + continue + if left > cursor + 1.25: + return False + cursor = max(cursor, right) + if cursor >= end - 1.25: + return True + return False + + if kind == "lines": + if height <= 1.0: + y = (bbox[1] + bbox[3]) / 2 + edge_intervals = [ + (cell[0], cell[2]) + for cell in table.cell_bboxes + if same(y, cell[1]) or same(y, cell[3]) + ] + return interval_covered(bbox[0], bbox[2], edge_intervals) + if width <= 1.0: + x = (bbox[0] + bbox[2]) / 2 + edge_intervals = [ + (cell[1], cell[3]) + for cell in table.cell_bboxes + if same(x, cell[0]) or same(x, cell[2]) + ] + return interval_covered(bbox[1], bbox[3], edge_intervals) + return False + if kind == "rects": + filled = shape.get("fill") is True + stroke_only = shape.get("stroke") is True and not filled + # A filled band is safe only when it exactly covers the verified header + # row whose textual labels are already represented in Markdown. + if ( + filled + and table.header_band_bbox is not None + and same_bbox(bbox, table.header_band_bbox) + ): + return True + if not stroke_only: + return False + # Some producers draw every ruled cell as a stroke-only rectangle. + if any(same_bbox(bbox, cell) for cell in table.cell_bboxes): + return True + if same_bbox(bbox, table_bbox): + return True + return False + + +def _recover_vector_bullets( + shapes: list[ + tuple[tuple[float, float, float, float], str, dict[str, Any]] + ], + lines: Sequence[_Line], + page_width: float, +) -> tuple[ + list[tuple[tuple[float, float, float, float], str, dict[str, Any]]], + bool, +]: + """Recover repeated circle bullets only when geometry proves a list. + + A scatter point is not exempt merely because it is small. A candidate + circle/square must sit immediately left of one unique text line and either + satisfy the strict single-marker geometry or belong to a repeated stable + cluster. The matched line receives a real list marker before removal. + """ + candidates: list[tuple[int, _Line, float, float, float, bool]] = [] + unresolved_bullet_like = False + for index, (bbox, kind, shape) in enumerate(shapes): + if kind not in {"curves", "rects"}: + continue + width = bbox[2] - bbox[0] + height = bbox[3] - bbox[1] + if ( + min(width, height) < 2.0 + or max(width, height) > 12.0 + or min(width, height) / max(width, height) < 0.78 + ): + continue + center_y = (bbox[1] + bbox[3]) / 2 + possible: list[tuple[float, _Line, float]] = [] + for line in lines: + if _LEADING_BULLET_RE.match(line.text): + continue + gap = line.x0 - bbox[2] + vertical_delta = abs(center_y - (line.top + line.bottom) / 2) + if ( + 0.5 <= gap <= max(22.0, line.font_size * 2.0) + and vertical_delta <= max(3.0, line.font_size * 0.38) + ): + possible.append((gap + vertical_delta, line, gap)) + if not possible: + continue + possible.sort(key=lambda item: item[0]) + if len(possible) > 1 and abs(possible[0][0] - possible[1][0]) <= 1.0: + unresolved_bullet_like = True + continue + _score, line, gap = possible[0] + strict = ( + max(width, height) <= 10.0 + and min(width, height) / max(width, height) >= 0.82 + and gap <= max(16.0, line.font_size * 1.6) + and abs(center_y - (line.top + line.bottom) / 2) + <= max(2.5, line.font_size * 0.28) + ) + candidates.append( + (index, line, gap, (bbox[0] + bbox[2]) / 2, width, strict) + ) + + stable: list[tuple[int, _Line]] = [] + for candidate in candidates: + _index, line, gap, center_x, diameter, strict = candidate + peers = [ + other + for other in candidates + if abs(other[1].x0 - line.x0) <= 2.0 + and abs(other[2] - gap) <= 2.0 + and abs(other[3] - center_x) <= 2.0 + and abs(other[4] - diameter) <= 1.5 + ] + if strict or len(peers) >= 3: + stable.append((_index, line)) + else: + unresolved_bullet_like = True + + stable_indices = {index for index, _line in stable} + matched_lines: set[int] = set() + for _index, line in stable: + if id(line) in matched_lines: + continue + line.text = "• " + line.text + matched_lines.add(id(line)) + return ( + [shape for index, shape in enumerate(shapes) if index not in stable_indices], + unresolved_bullet_like, + ) + + +def _material_vector_count( + page: Any, + tables: Sequence[_Table], + lines: Sequence[_Line], + page_width: float, + page_height: float, +) -> int: + """Count vector regions that may carry content unavailable as text. + + PDF charts and diagrams are frequently drawn only with rectangles, curves, + and line segments; dropping them would create a falsely complete Markdown + derivative. Table rules are already represented by extracted table cells, + while isolated horizontal/vertical separators and tiny decorative marks do + not carry standalone facts. The detector therefore removes table-contained + geometry, ignores isolated degenerate strokes, and fails only on a material + two-dimensional shape or a cohesive multi-shape region. + """ + shapes: list[ + tuple[tuple[float, float, float, float], str, dict[str, Any]] + ] = [] + unknown_geometry = 0 + for kind in ("rects", "curves", "lines"): + for shape in list(getattr(page, kind, []) or []): + bbox = _object_bbox(shape) + if bbox is None: + # A vector object exists but cannot be bounded; accuracy-first + # conversion cannot prove that it was merely decoration. + unknown_geometry += 1 + continue + containing_tables = [ + table + for table in tables + if _bbox_inside(bbox, (table.x0, table.top, table.x1, table.bottom)) + ] + if containing_tables: + if any( + _is_table_vector_furniture(bbox, kind, shape, table) + for table in containing_tables + ): + continue + # Once geometry is inside an extracted table cell, whole-page + # area thresholds are inappropriate: a small bar, fill, panel, + # or orthogonal trace can encode the cell's entire fact. + return 1 + if _is_page_border(bbox, page_width, page_height): + continue + shapes.append((bbox, kind, shape)) + + if unknown_geometry: + return unknown_geometry + if not shapes: + return 0 + + shapes, unresolved_bullet_like = _recover_vector_bullets( + shapes, lines, page_width + ) + if unresolved_bullet_like: + return 1 + if not shapes: + return 0 + + page_area = max(page_width * page_height, 1.0) + for bbox, kind, _shape in shapes: + width = bbox[2] - bbox[0] + height = bbox[3] - bbox[1] + area_ratio = width * height / page_area + # One substantial rectangle/curve is already enough to be a plot, + # diagram, or vector illustration. Filled bars also enter here. + if ( + kind in {"rects", "curves"} + and area_ratio >= 0.025 + and width >= page_width * 0.12 + and height >= page_height * 0.08 + ): + return 1 + # A single large diagonal line can itself be the complete trend or + # decision boundary. Horizontal/vertical separators have zero 2-D + # extent and remain allowed; short diagonal flourishes stay below the + # material width/height gates. + if ( + kind == "lines" + and area_ratio >= 0.025 + and width >= page_width * 0.20 + and height >= page_height * 0.12 + ): + return 1 + + # A timeline is often one long axis plus short perpendicular ticks. Its + # union box can be visually shallow, so a generic area threshold misses it. + line_bboxes = [bbox for bbox, kind, _shape in shapes if kind == "lines"] + for axis in line_bboxes: + axis_width = axis[2] - axis[0] + axis_height = axis[3] - axis[1] + if axis_width >= page_width * 0.35 and axis_height <= 1.0: + axis_y = (axis[1] + axis[3]) / 2 + ticks = [ + tick + for tick in line_bboxes + if (tick[2] - tick[0]) <= 1.0 + and (tick[3] - tick[1]) >= page_height * 0.012 + and axis[0] - 1.5 <= (tick[0] + tick[2]) / 2 <= axis[2] + 1.5 + and tick[1] - 1.5 <= axis_y <= tick[3] + 1.5 + ] + if len(ticks) >= 3: + return 1 + if axis_height >= page_height * 0.35 and axis_width <= 1.0: + axis_x = (axis[0] + axis[2]) / 2 + ticks = [ + tick + for tick in line_bboxes + if (tick[3] - tick[1]) <= 1.0 + and (tick[2] - tick[0]) >= page_width * 0.012 + and axis[1] - 1.5 <= (tick[1] + tick[3]) / 2 <= axis[3] + 1.5 + and tick[0] - 1.5 <= axis_x <= tick[2] + 1.5 + ] + if len(ticks) >= 3: + return 1 + + # Content may be distributed across separated panels or trend segments. + # Sum only individually non-trivial two-dimensional extents so a handful of + # tiny icons cannot trip the aggregate gate. + separated_material = [ + bbox + for bbox, _kind, _shape in shapes + if (bbox[2] - bbox[0]) >= page_width * 0.07 + and (bbox[3] - bbox[1]) >= page_height * 0.035 + ] + if ( + len(separated_material) >= 4 + and sum( + (bbox[2] - bbox[0]) * (bbox[3] - bbox[1]) + for bbox in separated_material + ) / page_area >= 0.02 + ): + return 1 + + # Scatter plots and node clouds may contain dozens of individually tiny, + # disconnected circles. Neither per-shape area nor connected components + # catches that topology. Require many non-degenerate rect/curve marks, + # material two-dimensional spread, and occupancy across the spread; a + # single logo/icon or a compact decorative flourish stays below the gate. + small_marks = [ + bbox + for bbox, kind, _shape in shapes + if kind in {"rects", "curves"} + and (bbox[2] - bbox[0]) >= 1.5 + and (bbox[3] - bbox[1]) >= 1.5 + and (bbox[2] - bbox[0]) <= page_width * 0.08 + and (bbox[3] - bbox[1]) <= page_height * 0.08 + ] + if len(small_marks) >= 12: + union = ( + min(bbox[0] for bbox in small_marks), + min(bbox[1] for bbox in small_marks), + max(bbox[2] for bbox in small_marks), + max(bbox[3] for bbox in small_marks), + ) + spread_width = union[2] - union[0] + spread_height = union[3] - union[1] + occupied: set[tuple[int, int]] = set() + if spread_width > 0 and spread_height > 0: + for bbox in small_marks: + center_x = (bbox[0] + bbox[2]) / 2 + center_y = (bbox[1] + bbox[3]) / 2 + x_bin = min(3, int(4 * (center_x - union[0]) / spread_width)) + y_bin = min(2, int(3 * (center_y - union[1]) / spread_height)) + occupied.add((x_bin, y_bin)) + if ( + spread_width >= page_width * 0.25 + and spread_height >= page_height * 0.12 + and spread_width * spread_height / page_area >= 0.04 + and len(occupied) >= 6 + ): + return 1 + + # Connected components catch charts built from several individually small + # bars/arrows, including horizontal/vertical ``page.lines``. A lone page + # separator never reaches the three-object threshold, and separated section + # rules do not join the same component. Three objects plus material 2-D + # extent still keeps small corner flourishes below the area gate. + remaining = set(range(len(shapes))) + padding = max(8.0, min(page_width, page_height) * 0.015) + components: list[list[int]] = [] + while remaining: + seed = remaining.pop() + component = [seed] + frontier = [seed] + while frontier: + current = frontier.pop() + neighbours = { + index + for index in remaining + if _bbox_near(shapes[current][0], shapes[index][0], padding) + } + remaining.difference_update(neighbours) + frontier.extend(neighbours) + component.extend(neighbours) + components.append(component) + + material = 0 + for component in components: + if len(component) < 3: + continue + bboxes = [shapes[index][0] for index in component] + union = ( + min(bbox[0] for bbox in bboxes), + min(bbox[1] for bbox in bboxes), + max(bbox[2] for bbox in bboxes), + max(bbox[3] for bbox in bboxes), + ) + width = union[2] - union[0] + height = union[3] - union[1] + if _is_page_border(union, page_width, page_height): + continue + if ( + width * height / page_area >= 0.018 + and width >= page_width * 0.15 + and height >= page_height * 0.07 + ): + material += 1 + return material + + +def _bbox_contains_word(bbox: tuple[float, float, float, float], word: dict[str, Any]) -> bool: + x = (float(word["x0"]) + float(word["x1"])) / 2 + y = (float(word["top"]) + float(word["bottom"])) / 2 + return bbox[0] - 0.5 <= x <= bbox[2] + 0.5 and bbox[1] - 0.5 <= y <= bbox[3] + 0.5 + + +def _extract_tables(page: Any, page_number: int) -> list[_Table]: + """Extract ruled/recognisable tables before reconstructing prose lines.""" + try: + found = page.find_tables() + except Exception as exc: # pdfplumber errors vary by PDF internals + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table geometry extraction failed: {exc}" + ) from exc + + tables: list[_Table] = [] + for candidate in found: + try: + raw_rows = candidate.extract() + except Exception as exc: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: detected table could not be extracted: {exc}" + ) from exc + normalised_rows = [ + [_normalise_space(cell) for cell in (row or [])] + for row in (raw_rows or []) + ] + indexed_rows = [ + (index, row) for index, row in enumerate(normalised_rows) if any(row) + ] + rows = [row for _index, row in indexed_rows] + row_source_indices = [index for index, _row in indexed_rows] + if not rows: + # A false-positive rectangle must not swallow text. It is ignored + # and its words remain available to the normal line path. + continue + width = max(len(row) for row in rows) + if width < 2: + # A one-cell rectangle is normally a callout, not a table. + continue + rows = [row + [""] * (width - len(row)) for row in rows] + x0, top, x1, bottom = (float(value) for value in candidate.bbox) + + def row_cells(source_index: int) -> list[Any]: + try: + cells = list(candidate.rows[source_index].cells) + except (AttributeError, IndexError, TypeError) as exc: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table header cell geometry is unavailable" + ) from exc + if len(cells) != width: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table header geometry width is ambiguous" + ) + return cells + + def numeric_bbox(cell: Any) -> tuple[float, float, float, float]: + try: + values = tuple(float(value) for value in cell) + except (TypeError, ValueError) as exc: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table header cell bbox is invalid" + ) from exc + if len(values) != 4: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table header cell bbox is invalid" + ) + return values + + def cell_bold_ratio(cell: Any) -> float: + bbox = numeric_bbox(cell) + total = 0 + bold = 0 + for char in list(getattr(page, "chars", []) or []): + text = str(char.get("text") or "") + if not text.strip(): + continue + try: + center_x = (float(char["x0"]) + float(char["x1"])) / 2 + center_y = (float(char["top"]) + float(char["bottom"])) / 2 + except (KeyError, TypeError, ValueError): + continue + if ( + bbox[0] - 0.5 <= center_x <= bbox[2] + 0.5 + and bbox[1] - 0.5 <= center_y <= bbox[3] + 0.5 + ): + weight = max(len(text), 1) + total += weight + if "bold" in str(char.get("fontname") or "").lower(): + bold += weight + return bold / total if total else 0.0 + + caption: str | None = None + # A common ruled-table shape is a first row merged across every column + # for a caption/title, followed by the real column header. Treating the + # caption as the Markdown header silently demotes the true header to + # data. Verify the merge in the candidate's cell geometry before + # detaching it; an ambiguous one-nonempty-cell first row fails closed. + if len(rows) >= 3: + first_nonempty = [cell for cell in rows[0] if cell] + if len(first_nonempty) == 1: + first_source_index = row_source_indices[0] + first_cells = row_cells(first_source_index) + candidate_cells = [cell for cell in first_cells if cell is not None] + if not candidate_cells: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table caption/header merge is empty" + ) + merged_bbox = numeric_bbox(candidate_cells[0]) + merged_is_full_width = ( + len(candidate_cells) == 1 + and abs(merged_bbox[0] - x0) <= 1.5 + and abs(merged_bbox[2] - x1) <= 1.5 + ) + if not merged_is_full_width: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: table has an ambiguous one-cell " + "first row; cannot safely choose the real header" + ) + caption = first_nonempty[0] + rows = rows[1:] + row_source_indices = row_source_indices[1:] + + header_source_indices = row_source_indices[:1] + + # Flatten one or more verified merged group-header rows into the leaf + # header. Markdown has only one header row, so each leaf receives its + # complete group path (for example "Identity / Unit"). Sparse rows + # whose merge geometry does not prove the span fail closed. + if rows and sum(bool(cell) for cell in rows[0]) < width: + leaf_index = next( + ( + index + for index, row in enumerate(rows) + if sum(bool(cell) for cell in row) == width + ), + None, + ) + if leaf_index is None or leaf_index == 0 or leaf_index >= len(rows) - 1: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: multi-level table header has no " + "unambiguous leaf header and body row" + ) + leaf_cells = row_cells(row_source_indices[leaf_index]) + if any(cell is None for cell in leaf_cells): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: leaf header geometry is merged or incomplete" + ) + if any(cell_bold_ratio(cell) < 0.55 for cell in leaf_cells): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: candidate leaf header lacks a " + "high-confidence visual header style" + ) + header_source_indices = row_source_indices[:leaf_index + 1] + + expanded_groups: list[list[str]] = [] + for group_index in range(leaf_index): + labels = rows[group_index] + cells = row_cells(row_source_indices[group_index]) + nonempty_indices = [ + index for index, label in enumerate(labels) if label + ] + if not nonempty_indices or nonempty_indices[0] != 0: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: merged group header has an " + "uncovered leading column" + ) + expanded = [""] * width + for position, start in enumerate(nonempty_indices): + end = ( + nonempty_indices[position + 1] + if position + 1 < len(nonempty_indices) + else width + ) + if cells[start] is None or any( + cells[index] is not None for index in range(start + 1, end) + ): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: sparse group header is not " + "backed by merged-cell geometry" + ) + group_bbox = numeric_bbox(cells[start]) + first_leaf_bbox = numeric_bbox(leaf_cells[start]) + last_leaf_bbox = numeric_bbox(leaf_cells[end - 1]) + if ( + abs(group_bbox[0] - first_leaf_bbox[0]) > 1.5 + or abs(group_bbox[2] - last_leaf_bbox[2]) > 1.5 + ): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: group header span does not " + "align with leaf columns" + ) + for column in range(start, end): + expanded[column] = labels[start] + if any(not label for label in expanded): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: group header leaves a column unlabeled" + ) + expanded_groups.append(expanded) + + leaf = rows[leaf_index] + flattened_header: list[str] = [] + for column in range(width): + path: list[str] = [] + for label in [ + *(group[column] for group in expanded_groups), + leaf[column], + ]: + if not path or path[-1].casefold() != label.casefold(): + path.append(label) + flattened_header.append(" / ".join(path)) + rows = [flattened_header, *rows[leaf_index + 1:]] + # Do not let a sparse chart grid masquerade as a ruled table and thereby + # exempt all of its vector geometry from the material-figure detector. + # A representable table needs a multi-cell header and at least one + # multi-cell data row. Otherwise leave its words/vector rules on the + # ordinary path, which will either preserve the text or fail explicitly. + if ( + len(rows) < 2 + or sum(bool(cell) for cell in rows[0]) < 2 + or not any(sum(bool(cell) for cell in row) >= 2 for row in rows[1:]) + ): + continue + try: + cell_bboxes = tuple({ + numeric_bbox(cell) + for candidate_row in candidate.rows + for cell in candidate_row.cells + if cell is not None + }) + header_cells = [ + numeric_bbox(cell) + for source_index in header_source_indices + for cell in candidate.rows[source_index].cells + if cell is not None + ] + except (AttributeError, TypeError) as exc: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: extracted table cell geometry is unavailable" + ) from exc + header_band_bbox = ( + ( + min(cell[0] for cell in header_cells), + min(cell[1] for cell in header_cells), + max(cell[2] for cell in header_cells), + max(cell[3] for cell in header_cells), + ) + if header_cells + else None + ) + tables.append(_Table( + page_number=page_number, + x0=x0, + x1=x1, + top=top, + bottom=bottom, + rows=rows, + caption=caption, + cell_bboxes=cell_bboxes, + header_band_bbox=header_band_bbox, + )) + return tables + + +def _cluster_rows(words: list[dict[str, Any]], page_width: float, page_number: int) -> list[_Line]: + """Turn positioned words into visual line fragments. + + Words on the same baseline are first clustered into a row, then split at a + large horizontal gap. The latter is what prevents simultaneous left/right + column lines from being concatenated. + """ + if not words: + return [] + + ordered = sorted(words, key=lambda item: (float(item["top"]), float(item["x0"]))) + rows: list[list[dict[str, Any]]] = [] + for word in ordered: + word_top = float(word["top"]) + word_bottom = float(word["bottom"]) + best: list[dict[str, Any]] | None = None + best_distance = math.inf + # Only the last few rows can overlap this word after top-order sorting. + for row in reversed(rows[-4:]): + row_top = median(float(item["top"]) for item in row) + row_bottom = median(float(item["bottom"]) for item in row) + vertical_overlap = min(word_bottom, row_bottom) - max(word_top, row_top) + tolerance = max(2.2, min(float(word.get("size", 10)), 12.0) * 0.24) + distance = abs(word_top - row_top) + if vertical_overlap > 0 or distance <= tolerance: + if distance < best_distance: + best = row + best_distance = distance + if best is None: + rows.append([word]) + else: + best.append(word) + + fragments: list[_Line] = [] + fragment_gap = max(22.0, page_width * 0.038) + for row_id, row in enumerate(rows): + sorted_row = sorted(row, key=lambda item: (float(item["x0"]), float(item["x1"]))) + groups: list[list[dict[str, Any]]] = [] + for word in sorted_row: + if not groups: + groups.append([word]) + continue + previous = groups[-1][-1] + gap = float(word["x0"]) - float(previous["x1"]) + if gap > fragment_gap: + groups.append([word]) + else: + groups[-1].append(word) + + merged_groups: list[list[dict[str, Any]]] = [] + group_index = 0 + while group_index < len(groups): + group = groups[group_index] + group_text = _normalise_space( + " ".join(str(item.get("text") or "") for item in group) + ) + if _BULLET_ONLY_RE.fullmatch(group_text): + if group_index + 1 >= len(groups): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: detached bullet has no text on row {row_id}" + ) + following = groups[group_index + 1] + gap = min(float(item["x0"]) for item in following) - max( + float(item["x1"]) for item in group + ) + if gap > max(72.0, page_width * 0.12): + raise PDFLayoutExtractionError( + f"PDF page {page_number}: detached bullet cannot be " + f"safely attached on row {row_id}" + ) + merged_groups.append( + sorted([*group, *following], key=lambda item: float(item["x0"])) + ) + group_index += 2 + continue + merged_groups.append(group) + group_index += 1 + groups = merged_groups + + for group in groups: + text = _normalise_space(" ".join(str(item["text"]) for item in group)) + if not text: + continue + sizes = [float(item.get("size") or 0.0) for item in group] + weights = [max(len(str(item.get("text") or "")), 1) for item in group] + weight_total = sum(weights) + size = sum(value * weight for value, weight in zip(sizes, weights)) / weight_total + bold_weight = sum( + weight + for item, weight in zip(group, weights) + if "bold" in str(item.get("fontname") or "").lower() + ) + fragments.append(_Line( + page_number=page_number, + x0=min(float(item["x0"]) for item in group), + x1=max(float(item["x1"]) for item in group), + top=min(float(item["top"]) for item in group), + bottom=max(float(item["bottom"]) for item in group), + text=text, + font_size=size, + bold_ratio=bold_weight / weight_total, + row_id=row_id, + )) + return fragments + + +def _extract_implicit_tables( + lines: list[_Line], page_width: float, page_number: int +) -> tuple[list[_Line], list[_Table]]: + """Recognise conservative borderless tables from aligned visual rows. + + ``pdfplumber.find_tables`` is strongest for ruled tables. A borderless + register still has a useful deterministic signal: at least three nearby + rows with the same three-or-more aligned column starts. Two-column groups + are accepted only when the first row is visibly a header, which avoids + turning ordinary key/value prose into a table. + """ + by_row: dict[int, list[_Line]] = defaultdict(list) + for line in lines: + by_row[line.row_id].append(line) + candidates = [ + sorted(row, key=lambda line: line.x0) + for _row_id, row in sorted( + by_row.items(), key=lambda item: min(line.top for line in item[1]) + ) + if 2 <= len(row) <= 8 + ] + if not candidates: + return lines, [] + + groups: list[list[list[_Line]]] = [] + x_tolerance = max(10.0, page_width * 0.018) + for row in candidates: + if not groups: + groups.append([row]) + continue + previous = groups[-1][-1] + vertical_gap = min(line.top for line in row) - max(line.bottom for line in previous) + aligned = ( + len(row) == len(previous) + and all(abs(a.x0 - b.x0) <= x_tolerance for a, b in zip(row, previous)) + and vertical_gap <= max(30.0, median(line.font_size for line in row) * 2.8) + ) + if aligned: + groups[-1].append(row) + else: + groups.append([row]) + + consumed: set[int] = set() + tables: list[_Table] = [] + for group in groups: + if len(group) < 3: + continue + column_count = len(group[0]) + header_is_distinct = ( + sum(line.bold_ratio for line in group[0]) / column_count >= 0.55 + and sum(line.bold_ratio for row in group[1:] for line in row) + / ((len(group) - 1) * column_count) < 0.4 + ) + body_signal_ratio = sum( + any(_VALUE_LIKE_RE.fullmatch(line.text) for line in row) + for row in group[1:] + ) / (len(group) - 1) + high_confidence_table = ( + header_is_distinct and body_signal_ratio >= 0.6 + ) + prose_sentence_ratio = sum( + bool(re.search(r"[.!?。！？]\s*$", line.text)) + for row in group + for line in row + ) / (len(group) * column_count) + if column_count == 2: + # Without a visually distinct first row, defer to the dedicated + # two-column/key-value discriminator below. When the first row is + # bold, require value-like body evidence before calling it a table; + # sentence-like prose remains a two-column candidate, otherwise the + # layout is genuinely ambiguous and must fail closed. + if not header_is_distinct: + continue + if not high_confidence_table: + if prose_sentence_ratio >= 0.6: + continue + raise PDFLayoutExtractionError( + f"PDF page {page_number}: aligned two-column borderless " + "layout is ambiguous between a table and independent prose" + ) + elif not high_confidence_table: + if prose_sentence_ratio >= 0.6: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: aligned {column_count}-column " + "independent prose is unsupported; visual review is required" + ) + raise PDFLayoutExtractionError( + f"PDF page {page_number}: aligned {column_count}-column " + "borderless layout lacks a high-confidence table header/body signal" + ) + # An independent prose column usually consumes most of its half-page; + # a register cell is shorter. Require this extra evidence for two + # columns even when its header is bold. + if column_count == 2: + median_widths = [ + median(row[index].x1 - row[index].x0 for row in group) + for index in range(column_count) + ] + if any(width > page_width * 0.28 for width in median_widths): + continue + flattened = [line for row in group for line in row] + tables.append(_Table( + page_number=page_number, + x0=min(line.x0 for line in flattened), + x1=max(line.x1 for line in flattened), + top=min(line.top for line in flattened), + bottom=max(line.bottom for line in flattened), + rows=[[line.text for line in row] for row in group], + )) + consumed.update(id(line) for line in flattened) + return [line for line in lines if id(line) not in consumed], tables + + +def _prepare_key_value_rows(page: _Page) -> None: + """Bind high-confidence labels to first values and tight visual wraps. + + This pass runs before column classification. A wrapped value creates extra + right-side row ids, which would otherwise make a form look like an + independently flowing right column. Plans are validated without mutation; + only a unique, contiguous label/value group is then applied. + """ + active = [line for line in page.lines if not line.suppressed] + if len(active) < 2: + return + by_row: dict[int, list[_Line]] = defaultdict(list) + for line in active: + by_row[line.row_id].append(line) + anchors: list[tuple[_Line, _Line]] = [] + minimum_gap = max(20.0, page.width * 0.03) + for row in by_row.values(): + ordered = sorted(row, key=lambda line: line.x0) + if len(ordered) != 2: + continue + label, value = ordered + if value.x0 - label.x1 >= minimum_gap: + anchors.append((label, value)) + anchors.sort(key=lambda pair: (pair[0].top, pair[0].x0)) + if not anchors: + return + + x_tolerance = max(10.0, page.width * 0.022) + groups: list[list[tuple[_Line, _Line]]] = [] + for anchor in anchors: + if not groups: + groups.append([anchor]) + continue + previous_label, previous_value = groups[-1][-1] + label, value = anchor + if ( + abs(label.x0 - previous_label.x0) <= x_tolerance + and abs(value.x0 - previous_value.x0) <= x_tolerance + and label.top - previous_label.top <= max(90.0, label.font_size * 8.0) + ): + groups[-1].append(anchor) + else: + groups.append([anchor]) + + qualified: list[list[tuple[_Line, _Line]]] = [] + for group in groups: + labels = [label for label, _value in group] + values = [value for _label, value in group] + colon_ratio = sum( + label.text.rstrip().endswith((":", "：")) for label in labels + ) / len(labels) + narrow_short_bold = ( + len(group) >= 3 + and median(label.x1 - label.x0 for label in labels) <= page.width * 0.23 + and median(len(label.text.rstrip(":：").split()) for label in labels) <= 3 + and sum(label.bold_ratio >= 0.65 for label in labels) / len(labels) >= 0.8 + and sum(value.bold_ratio >= 0.65 for value in values) / len(values) < 0.4 + ) + explicit_labels = colon_ratio >= 0.8 + if explicit_labels or narrow_short_bold: + qualified.append(group) + if not qualified: + return + if len(qualified) != 1: + raise PDFLayoutExtractionError( + f"PDF page {page.number}: multiple key/value interpretations remain plausible" + ) + + group = qualified[0] + anchor_line_ids = { + id(line) for pair in group for line in pair + } + continuations_by_row: dict[int, list[_Line]] = defaultdict(list) + consumed_continuations: set[int] = set() + for index, (label, first_value) in enumerate(group): + next_top = ( + group[index + 1][0].top + if index + 1 < len(group) + else first_value.bottom + max(42.0, first_value.font_size * 4.2) + ) + candidates = sorted( + ( + line + for line in active + if id(line) not in anchor_line_ids + and id(line) not in consumed_continuations + and line.top >= first_value.bottom + and line.top < next_top + ), + key=lambda line: (line.top, line.x0), + ) + last = first_value + planned: list[_Line] = [] + for candidate in candidates: + near_value_column = ( + first_value.x0 - x_tolerance + <= candidate.x0 + <= first_value.x0 + max(24.0, page.width * 0.06) + ) + if not near_value_column: + # Material in the form band that is neither another anchor nor + # a uniquely aligned continuation makes the relation ambiguous. + if candidate.x1 <= first_value.x0 + page.width * 0.5: + raise PDFLayoutExtractionError( + f"PDF page {page.number}: key/value continuation has " + "ambiguous horizontal ownership" + ) + continue + gap = candidate.top - last.bottom + font_ratio = candidate.font_size / max(last.font_size, 0.1) + last_is_complete = bool( + re.search(r"[.!?。！？]\s*$", last.text) + ) + if ( + last_is_complete + or gap < -1.0 + or gap > max(5.0, last.font_size * 0.75) + or not (0.85 <= font_ratio <= 1.15) + ): + raise PDFLayoutExtractionError( + f"PDF page {page.number}: key/value continuation cannot be " + "uniquely attached; visual review is required" + ) + planned.append(candidate) + consumed_continuations.add(id(candidate)) + last = candidate + continuations_by_row[label.row_id] = planned + + # Apply only after every continuation has a unique owner. + for label, first_value in group: + for continuation in continuations_by_row[label.row_id]: + separator = "" if first_value.text.endswith("-") else " " + first_value.text += separator + continuation.text + first_value.bottom = max(first_value.bottom, continuation.bottom) + first_value.x1 = max(first_value.x1, continuation.x1) + page.key_value_rows.add(label.row_id) + if consumed_continuations: + page.lines = [ + line for line in page.lines if id(line) not in consumed_continuations + ] + + +def _column_candidate(page: _Page) -> float | None: + """Return a gutter x-coordinate only for independently flowing columns. + + Form-like/key-value rows can also contain a large horizontal gap. They are + intentionally kept row-major when nearly every left and right fragment is + paired on the same baseline. + """ + lines = [line for line in page.lines if not line.suppressed] + if len(lines) < 4: + return None + + candidates = {page.width / 2} + by_row: dict[int, list[_Line]] = defaultdict(list) + for line in lines: + by_row[line.row_id].append(line) + minimum_gap = max(24.0, page.width * 0.045) + for row in by_row.values(): + row = sorted(row, key=lambda line: line.x0) + for left, right in zip(row, row[1:]): + if right.x0 - left.x1 >= minimum_gap: + midpoint = (left.x1 + right.x0) / 2 + if page.width * 0.28 <= midpoint <= page.width * 0.72: + candidates.add(midpoint) + + best: tuple[float, float] | None = None + ambiguous_paired_layout = False + row_major_layout = False + key_value_rows: set[int] = set() + for split in candidates: + edge_tolerance = max(2.0, page.width * 0.004) + left = [line for line in lines if line.x1 <= split + edge_tolerance] + right = [line for line in lines if line.x0 >= split - edge_tolerance] + crossing = [line for line in lines if line not in left and line not in right] + if len(left) < 2 or len(right) < 2: + continue + + # Both flows should occupy overlapping vertical territory. + left_range = (min(line.top for line in left), max(line.bottom for line in left)) + right_range = (min(line.top for line in right), max(line.bottom for line in right)) + if min(left_range[1], right_range[1]) <= max(left_range[0], right_range[0]): + continue + + left_rows = {line.row_id for line in left} + right_rows = {line.row_id for line in right} + paired_rows = left_rows & right_rows + left_pair_ratio = len(paired_rows) / max(len(left_rows), 1) + right_pair_ratio = len(paired_rows) / max(len(right_rows), 1) + # A ledger/form is row-correlated; a true column layout has at least + # one side flowing independently. + if left_pair_ratio >= 0.8 and right_pair_ratio >= 0.8: + median_left_width = median(line.x1 - line.x0 for line in left) + narrow_left = median_left_width <= page.width * 0.23 + colon_ratio = sum( + line.text.rstrip().endswith((':', '：')) for line in left + ) / len(left) + label_bold_ratio = sum(line.bold_ratio >= 0.65 for line in left) / len(left) + right_bold_ratio = sum(line.bold_ratio >= 0.65 for line in right) / len(right) + median_left_tokens = median(len(line.text.split()) for line in left) + value_like_ratio = sum( + bool(_VALUE_LIKE_RE.fullmatch(line.text)) + for line in right + ) / len(right) + sentence_left_ratio = sum( + bool(re.search(r"[.!?。！？]\s*$", line.text)) for line in left + ) / len(left) + sentence_right_ratio = sum( + bool(re.search(r"[.!?。！？]\s*$", line.text)) for line in right + ) / len(right) + high_confidence_key_value = ( + colon_ratio >= 0.6 + or ( + narrow_left + and median_left_tokens <= 4 + and value_like_ratio >= 0.6 + ) + or ( + narrow_left + and median_left_tokens <= 3 + and label_bold_ratio >= 0.8 + and right_bold_ratio < 0.4 + ) + ) + high_confidence_columns = ( + sentence_left_ratio >= 0.6 and sentence_right_ratio >= 0.6 + ) + if high_confidence_key_value: + row_major_layout = True + key_value_rows.update(paired_rows) + continue + if not high_confidence_columns: + ambiguous_paired_layout = True + continue + + # Column starts should be reasonably aligned. This filters scattered + # right-aligned labels inside otherwise single-column prose. + left_starts = [line.x0 for line in left] + right_starts = [line.x0 for line in right] + if (max(right_starts) - min(right_starts)) > page.width * 0.18: + continue + if (max(left_starts) - min(left_starts)) > page.width * 0.25: + continue + + score = ( + 2.0 * min(len(left), len(right)) + + 2.5 * len(paired_rows) + - 2.0 * len(crossing) + - abs(split - page.width / 2) / page.width + ) + if best is None or score > best[0]: + best = (score, split) + if best is not None and (ambiguous_paired_layout or row_major_layout): + raise PDFLayoutExtractionError( + f"PDF page {page.number}: competing reading-order interpretations " + "remain plausible; use explicit table/column structure or visual review" + ) + if best is not None: + return best[1] + if row_major_layout: + page.key_value_rows.update(key_value_rows) + return None + if ambiguous_paired_layout: + raise PDFLayoutExtractionError( + f"PDF page {page.number}: aligned narrow two-column/key-value layout " + "is ambiguous; add explicit labels/table structure or use visual review" + ) + return None + + +def _running_key(line: _Line, page: _Page) -> tuple[str, str] | None: + zone: str | None = None + if line.top <= page.height * 0.075: + zone = "top" + elif line.bottom >= page.height * 0.925: + zone = "bottom" + if zone is None: + return None + text = _normalise_space(line.text).casefold() + if not text: + return None + # Only normalise standalone page counters. Normalising every digit would + # incorrectly de-duplicate real headings such as "Experiment 1" and + # "Experiment 2" when they happen to sit near the top of adjacent pages. + if _EXPLICIT_PAGE_NUMBER_RE.fullmatch(text) or _BARE_PAGE_NUMBER_RE.fullmatch(text): + text = _RUNNING_DIGITS_RE.sub("#", text) + return zone, text + + +def _suppress_running_furniture(pages: list[_Page]) -> int: + occurrences: dict[tuple[str, str], list[tuple[_Page, _Line]]] = defaultdict(list) + suppressed = 0 + for page in pages: + for line in page.lines: + key = _running_key(line, page) + if key is not None: + occurrences[key].append((page, line)) + for key, values in occurrences.items(): + page_numbers = {page.number for page, _line in values} + original_values = [ + _normalise_space(line.text).casefold() for _page, line in values + ] + is_explicit_page_number = all( + _EXPLICIT_PAGE_NUMBER_RE.fullmatch(value) for value in original_values + ) + bare_values: list[int] = [] + for value in original_values: + if not _BARE_PAGE_NUMBER_RE.fullmatch(value): + bare_values = [] + break + bare_values.append(int(value)) + # A bare number is removed only when it exactly matches the physical + # page sequence. Constant-offset or merely distinct footer values may + # be years, scores, or measurements and are preserved. + is_bare_page_series = bool( + len(page_numbers) >= 2 + and bare_values + and all( + value == page.number + for (page, _line), value in zip(values, bare_values) + ) + ) + is_page_number_series = is_explicit_page_number or is_bare_page_series + if bare_values and not is_bare_page_series: + continue + if len(page_numbers) < 2 and not is_explicit_page_number: + continue + # Preserve the first occurrence: it may be the document's actual title + # or a meaningful disclaimer, while later copies are running furniture. + ordered = sorted(values, key=lambda item: item[0].number) + for _page, line in (ordered if is_page_number_series else ordered[1:]): + if not line.suppressed: + line.suppressed = True + suppressed += 1 + return suppressed + + +def _assign_heading_levels(pages: list[_Page]) -> None: + active = [line for page in pages for line in page.lines if not line.suppressed] + heading_candidates = [ + line for line in active if not _LEADING_BULLET_RE.match(line.text) + ] + if not heading_candidates: + return + size_weights: Counter[float] = Counter() + for line in heading_candidates: + size_weights[round(line.font_size, 1)] += max(len(line.text), 1) + body_size = size_weights.most_common(1)[0][0] + + page_for_line = { + id(line): page + for page in pages + for line in page.lines + } + + def question_heading_context(line: _Line) -> bool: + page = page_for_line[id(line)] + x_tolerance = max(8.0, page.width * 0.025) + aligned = sorted( + ( + candidate + for candidate in page.lines + if not candidate.suppressed + and candidate is not line + and abs(candidate.x0 - line.x0) <= x_tolerance + ), + key=lambda candidate: (candidate.top, candidate.x0), + ) + following = [candidate for candidate in aligned if candidate.top >= line.bottom] + if not following: + return False + successor = following[0] + after_gap = successor.top - line.bottom + successor_is_body = ( + body_size * 0.85 <= successor.font_size <= body_size * 1.15 + and successor.bold_ratio < 0.65 + and not _LEADING_BULLET_RE.match(successor.text) + ) + if not successor_is_body or not (0 <= after_gap <= body_size * 2.5): + return False + preceding = [candidate for candidate in aligned if candidate.bottom <= line.top] + if not preceding: + return True + before_gap = line.top - preceding[-1].bottom + return before_gap >= max(body_size * 0.8, after_gap * 1.2) + + def modest_heading_text(line: _Line) -> bool: + normalised = _normalise_space(line.text) + words = normalised.split() + declarative_terminal = bool(re.search(r"[.!。！]\s*$", normalised)) + question_terminal = bool(re.search(r"[?？]\s*$", normalised)) + # A modest size bump plus bold is also a common emphasis style for a + # complete factual sentence. Full stops/exclamation marks therefore + # reject the modest path regardless of sentence length. A compact + # question may be a real heading, but only with heading-to-body spacing. + compact = bool( + normalised + and len(normalised) <= 160 + and len(words) <= 16 + ) + if not compact or declarative_terminal: + return False + if question_terminal: + return question_heading_context(line) + return True + + def is_heading_candidate(line: _Line) -> bool: + return bool( + line.font_size >= body_size * 1.50 + or ( + line.font_size >= body_size * 1.28 + and modest_heading_text(line) + ) + or ( + line.bold_ratio >= 0.65 + and line.font_size >= body_size * 1.12 + and modest_heading_text(line) + ) + ) + + heading_sizes = sorted({ + round(line.font_size, 1) + for line in heading_candidates + if is_heading_candidate(line) + and len(line.text) <= 240 + }, reverse=True) + levels = {size: min(index + 1, 6) for index, size in enumerate(heading_sizes)} + for line in heading_candidates: + rounded = round(line.font_size, 1) + if rounded in levels and is_heading_candidate(line): + line.heading_level = levels[rounded] + + +def _table_markdown(table: _Table) -> str: + width = max(len(row) for row in table.rows) + rows = [row + [""] * (width - len(row)) for row in table.rows] + header = rows[0] + body = rows[1:] + output = [ + "| " + " | ".join(_escape_table_cell(cell) for cell in header) + " |", + "| " + " | ".join("---" for _ in range(width)) + " |", + ] + output.extend( + "| " + " | ".join(_escape_table_cell(cell) for cell in row) + " |" + for row in body + ) + rendered = "\n".join(output) + if table.caption: + return f"{_normalise_space(table.caption)}\n\n{rendered}" + return rendered + + +def _ordered_page_elements(page: _Page) -> list[_Line | _Table]: + lines = [line for line in page.lines if not line.suppressed] + elements: list[_Line | _Table] = [*lines, *page.tables] + split = page.column_split + if split is None: + return sorted(elements, key=lambda item: (item.top, item.x0, item.bottom)) + + tolerance = max(2.0, page.width * 0.004) + left: list[_Line | _Table] = [] + right: list[_Line | _Table] = [] + spanning: list[_Line | _Table] = [] + for item in elements: + if item.x1 <= split + tolerance: + left.append(item) + elif item.x0 >= split - tolerance: + right.append(item) + else: + spanning.append(item) + + # A spanning element is a safe boundary: read both columns above it, emit + # it, then continue with both columns below it. + ordered: list[_Line | _Table] = [] + lower = -math.inf + for boundary in sorted(spanning, key=lambda item: (item.top, item.x0)): + band_left = [item for item in left if lower <= item.top < boundary.top] + band_right = [item for item in right if lower <= item.top < boundary.top] + ordered.extend(sorted(band_left, key=lambda item: (item.top, item.x0))) + ordered.extend(sorted(band_right, key=lambda item: (item.top, item.x0))) + ordered.append(boundary) + # Use the boundary's start, not its bottom: side-column material beside + # a tall spanning object must be emitted after the boundary rather than + # being dropped from both adjacent bands. + lower = max(lower, boundary.top) + ordered.extend(sorted( + [item for item in left if item.top >= lower], + key=lambda item: (item.top, item.x0), + )) + ordered.extend(sorted( + [item for item in right if item.top >= lower], + key=lambda item: (item.top, item.x0), + )) + return ordered + + +def _render_page(page: _Page) -> str: + elements = _ordered_page_elements(page) + blocks: list[tuple[str, _Line | _Table]] = [] + key_value_lines: dict[int, list[_Line]] = defaultdict(list) + for line in page.lines: + if not line.suppressed and line.row_id in page.key_value_rows: + key_value_lines[line.row_id].append(line) + rendered_key_value_rows: set[int] = set() + for element in elements: + if isinstance(element, _Table): + blocks.append((_table_markdown(element), element)) + continue + if element.row_id in page.key_value_rows: + if element.row_id in rendered_key_value_rows: + continue + pair = sorted(key_value_lines[element.row_id], key=lambda line: line.x0) + if len(pair) != 2: + raise PDFLayoutExtractionError( + f"PDF page {page.number}: key/value row {element.row_id} " + "does not contain exactly one label and one value" + ) + label, value = pair + label_text = _normalise_space(label.text).rstrip(":：").rstrip() + if not label_text or not _normalise_space(value.text): + raise PDFLayoutExtractionError( + f"PDF page {page.number}: key/value row {element.row_id} is empty" + ) + label.heading_level = None + rendered_key_value_rows.add(element.row_id) + blocks.append((f"{label_text}: {_normalise_space(value.text)}", label)) + continue + prefix = "#" * element.heading_level + " " if element.heading_level else "" + line_text = ( + element.text + if element.heading_level is not None + else _normalise_list_marker(element.text) + ) + rendered = prefix + line_text + if blocks and isinstance(blocks[-1][1], _Line): + previous = blocks[-1][1] + previous_text, _ = blocks[-1] + same_heading = ( + element.heading_level is not None + and element.heading_level == previous.heading_level + ) + close_x = abs(element.x0 - previous.x0) <= max(8.0, page.width * 0.025) + vertical_gap = element.top - previous.bottom + same_column = not ( + page.column_split is not None + and ((previous.x1 <= page.column_split < element.x0) + or (element.x1 <= page.column_split < previous.x0)) + ) + if same_heading and close_x and same_column and vertical_gap <= max(6.0, element.font_size * 0.6): + blocks[-1] = (previous_text + " " + element.text, previous) + previous.bottom = element.bottom + previous.x1 = max(previous.x1, element.x1) + continue + # Join visual wraps into one Markdown paragraph. A tight leading + # gap is required so distinct source paragraphs remain distinct. + if ( + element.heading_level is None + and previous.heading_level is None + and close_x + and same_column + and vertical_gap <= max(4.0, element.font_size * 0.55) + ): + separator = "" if previous_text.endswith("-") else " " + blocks[-1] = (previous_text + separator + line_text, previous) + previous.bottom = element.bottom + previous.x1 = max(previous.x1, element.x1) + continue + blocks.append((rendered, element)) + return "\n\n".join(text for text, _element in blocks if text.strip()) + + +def extract_pdf_to_markdown(source: str | Path) -> PDFExtractionResult: + """Extract *source* using geometry-aware reading order. + + Raises :class:`PDFLayoutExtractionError` instead of silently calling a + lower-fidelity extractor. This is important because a successful-looking + Markdown file with interleaved columns is worse than an explicit ingest + failure. + """ + path = Path(source) + if not path.is_file(): + raise PDFLayoutExtractionError(f"PDF source does not exist: {path}") + try: + import pdfplumber + except ImportError as exc: # pragma: no cover - exercised by install state + raise PDFLayoutExtractionError( + "layout-aware PDF extraction requires pdfplumber; install markitdown[all]" + ) from exc + + try: + document = pdfplumber.open(str(path)) + except Exception as exc: + raise PDFLayoutExtractionError(f"cannot open PDF {path.name}: {exc}") from exc + + pages: list[_Page] = [] + table_count = 0 + try: + if not document.pages: + raise PDFLayoutExtractionError(f"PDF has no pages: {path.name}") + for page_number, raw_page in enumerate(document.pages, 1): + try: + # Duplicate paint operations are common in generated PDFs and + # otherwise appear as duplicated words. + page = raw_page.dedupe_chars(tolerance=1, extra_attrs=("fontname", "size")) + tables = _extract_tables(page, page_number) + table_bboxes = [ + (table.x0, table.top, table.x1, table.bottom) for table in tables + ] + words = page.extract_words( + keep_blank_chars=False, + use_text_flow=False, + extra_attrs=["fontname", "size"], + x_tolerance=2.5, + y_tolerance=2.5, + ) + except PDFLayoutExtractionError: + raise + except Exception as exc: + raise PDFLayoutExtractionError( + f"PDF page {page_number}: positioned text extraction failed: {exc}" + ) from exc + words = [ + word for word in words + if not any(_bbox_contains_word(bbox, word) for bbox in table_bboxes) + ] + lines = _cluster_rows(words, float(page.width), page_number) + lines, implicit_tables = _extract_implicit_tables( + lines, float(page.width), page_number + ) + tables.extend(implicit_tables) + table_count += len(tables) + raw_images = list(getattr(page, "images", [])) + pages.append(_Page( + number=page_number, + width=float(page.width), + height=float(page.height), + lines=lines, + tables=tables, + has_images=bool(raw_images), + material_image_count=_material_image_count( + raw_images, float(page.width), float(page.height) + ), + material_vector_count=_material_vector_count( + page, + tables, + lines, + float(page.width), + float(page.height), + ), + )) + finally: + document.close() + + if not any(page.lines or page.tables for page in pages): + raise PDFLayoutExtractionError( + f"PDF contains no extractable text or tables: {path.name}; OCR is required" + ) + image_only = [ + page.number for page in pages + if page.has_images and not page.lines and not page.tables + ] + if image_only: + raise PDFLayoutExtractionError( + "PDF has image-only page(s) with no extractable text: " + f"{image_only}; OCR is required before ingest" + ) + material_image_pages = [ + page.number for page in pages if page.material_image_count > 0 + ] + if material_image_pages: + raise PDFLayoutExtractionError( + "PDF has content-sized raster image(s) on page(s) " + f"{material_image_pages}; visual/OCR figure extraction is required " + "before accuracy-first ingest" + ) + + material_vector_pages = [ + page.number for page in pages if page.material_vector_count > 0 + ] + if material_vector_pages: + raise PDFLayoutExtractionError( + "PDF has content-sized vector figure(s) on page(s) " + f"{material_vector_pages}; visual/vector figure extraction is required " + "before accuracy-first ingest" + ) + + # The only safe CID recovery is pdfminer's well-known line-leading 127 + # bullet placeholder. One unresolved glyph elsewhere can negate a number, + # unit, or identifier, so fail on the first occurrence rather than applying + # a density threshold. + unresolved_cids: list[str] = [] + replacement_locations: list[str] = [] + for page in pages: + for line in page.lines: + without_known_bullet = _LEADING_BULLET_RE.sub("", line.text, count=1) + unresolved_cids.extend(_CID_RE.findall(without_known_bullet)) + if _REPLACEMENT_CHAR in line.text: + replacement_locations.append(f"page {page.number} text") + for table in page.tables: + for row in table.rows: + for cell in row: + unresolved_cids.extend(_CID_RE.findall(cell)) + if _REPLACEMENT_CHAR in cell: + replacement_locations.append(f"page {page.number} table") + if table.caption: + unresolved_cids.extend(_CID_RE.findall(table.caption)) + if _REPLACEMENT_CHAR in table.caption: + replacement_locations.append(f"page {page.number} caption") + if unresolved_cids: + raise PDFLayoutExtractionError( + "PDF text mapping is unreliable " + f"({len(unresolved_cids)} unresolved CID glyph(s)): {path.name}" + ) + if replacement_locations: + raise PDFLayoutExtractionError( + "PDF text mapping is unreliable " + f"({len(replacement_locations)} Unicode replacement glyph occurrence(s)): " + f"{path.name}" + ) + + suppressed = _suppress_running_furniture(pages) + for page in pages: + _prepare_key_value_rows(page) + _assign_heading_levels(pages) + two_column_pages: list[int] = [] + for page in pages: + page.column_split = _column_candidate(page) + if page.column_split is not None: + two_column_pages.append(page.number) + + rendered_pages = [_render_page(page) for page in pages] + markdown = "\n\n".join(text for text in rendered_pages if text.strip()).strip() + if not markdown: + raise PDFLayoutExtractionError( + f"PDF extraction produced empty Markdown after layout processing: {path.name}" + ) + if _CID_RE.search(markdown): + raise PDFLayoutExtractionError( + f"PDF rendered Markdown still contains an unresolved CID glyph: {path.name}" + ) + if _REPLACEMENT_CHAR in markdown: + raise PDFLayoutExtractionError( + f"PDF rendered Markdown still contains a Unicode replacement glyph: {path.name}" + ) + return PDFExtractionResult( + markdown=markdown + "\n", + diagnostics=PDFExtractionDiagnostics( + page_count=len(pages), + text_pages=sum(1 for page in pages if page.lines or page.tables), + two_column_pages=tuple(two_column_pages), + image_pages=tuple(page.number for page in pages if page.has_images), + table_count=table_count, + suppressed_running_elements=suppressed, + ), + ) + + +__all__ = [ + "PDF_LAYOUT_CONVERTER_VERSION", + "PDFExtractionDiagnostics", + "PDFExtractionResult", + "PDFLayoutExtractionError", + "extract_pdf_to_markdown", +] diff --git a/scripts/postprocess.py b/scripts/postprocess.py index ad0840f..3a105ab 100644 --- a/scripts/postprocess.py +++ b/scripts/postprocess.py @@ -19,6 +19,7 @@ import hashlib import re from datetime import date +from typing import Any from section_parser import ( Block, @@ -39,6 +40,12 @@ "figure": "f", } +# outline.json 的结构/新鲜度契约版本。任何会改变章节边界、 +# content hash 或摘要恢复语义的修改都必须递增它。 +OUTLINE_SCHEMA_VERSION = 2 + +_SHA256_RE = re.compile(r"^[0-9a-f]{64}$") + ANCHOR_DETECT_RE = re.compile( r"\s\^[hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?(?=\s|$)", re.MULTILINE, @@ -78,6 +85,19 @@ def _md5_6(s: str) -> str: return hashlib.md5(s.encode("utf-8")).hexdigest()[:6] +def _section_sha256(text_with_anchors: str, start: int, end: int) -> str: + """Return the content hash stored on one outline section. + + Heading anchors intentionally hash only their title. This separate hash + binds ``agent_summary`` and cached section ranges to the complete current + section body (including descendants), after removing derived anchors. + """ + normalized = _normalize_for_hash( + strip_anchors(text_with_anchors[start:end]) + ) + return hashlib.sha256(normalized.encode("utf-8")).hexdigest() + + def has_anchors(text: str) -> bool: """启发式：文本中有 ≥3 处形如 ` ^h-2-1-abcdef` / ` ^p-12-7d8e9a` 的尾巴 → 视为已加锚。""" matches = ANCHOR_DETECT_RE.findall(text) @@ -140,27 +160,34 @@ def make_unique(base: str) -> str: def _norm_title(t: str) -> str: """归一化标题文本：压空白 + 转小写，用于跨 re-convert 的 title 级匹配。""" - return re.sub(r"\s+", " ", (t or "")).strip().lower() + return re.sub(r"\s+", " ", t if isinstance(t, str) else "").strip().lower() def _collect_summaries(sections: list[dict]) -> list[dict]: """递归收集旧 outline 中所有非空 agent_summary。 - 返回 list[{anchor, title, summary}]——供 _restore_summaries 做 + 返回 list[{anchor, title, section_sha256, summary}]——供 _restore_summaries 做 anchor → title 多级匹配（pipe-2）。""" out: list[dict] = [] def walk(secs): + if not isinstance(secs, list): + return for s in secs: + if not isinstance(s, dict): + continue anc = s.get("anchor") summary = s.get("agent_summary") - if anc and summary: + if isinstance(anc, str) and isinstance(summary, str) and summary: out.append({ "anchor": anc, "title": _norm_title(s.get("title", "")), + "section_sha256": s.get("section_sha256"), "summary": summary, }) - walk(s.get("children", [])) + children = s.get("children", []) + if isinstance(children, list): + walk(children) walk(sections) return out @@ -184,11 +211,21 @@ def _restore_summaries(sections: list[dict], old_summaries: list[dict]) -> int: consumed: set[int] = set() - def take(bucket: dict[str, list[dict]], key: str | None) -> str | None: + def take( + bucket: dict[str, list[dict]], + key: str | None, + section_sha256: str | None, + ) -> str | None: if not key: return None for e in bucket.get(key, []): - if id(e) not in consumed: + # 摘要只能跟随它真正读过的章节内容。旧 outline 没有 + # section_sha256、或标题未变但正文已改，均不得恢复旧摘要。 + if ( + id(e) not in consumed + and section_sha256 + and e.get("section_sha256") == section_sha256 + ): consumed.add(id(e)) return e["summary"] return None @@ -199,9 +236,14 @@ def walk(secs): nonlocal restored for s in secs: if not s.get("agent_summary"): - summ = take(by_anchor, s.get("anchor")) + section_sha256 = s.get("section_sha256") + summ = take(by_anchor, s.get("anchor"), section_sha256) if summ is None: - summ = take(by_title, _norm_title(s.get("title", ""))) + summ = take( + by_title, + _norm_title(s.get("title", "")), + section_sha256, + ) if summ: s["agent_summary"] = summ restored += 1 @@ -238,8 +280,8 @@ def build_outline_data( （load_or_build_outline）**不传** previous_outline——wiki 页可被 agent/人直接 频繁编辑、整页换主题的风险高，过期即整体丢弃不合并。两条路径口径不同是**按对象 特性的有意取舍**，非疏忽。 - 已知残留：heading 锚点 hash 仅基于标题文本（见 :121），故"同标题换正文"在 - re-convert 路径仍可能复活旧摘要；因 raw 不可变 + 重转罕见，该残留风险低、接受。 + 摘要恢复额外绑定 section_sha256：heading 同名但正文变化时， + 旧摘要会精确失效；仅结构位置/锚点序号改变但章节内容未变时才恢复。 注：早期版本曾要求传入 add_anchors 返回的 blocks 列表，但实测 blocks 的 char_start/char_end 是基于"加锚前 body"的偏移，加锚后会漂移；保留 @@ -264,20 +306,407 @@ def build_outline_data( outline = build_outline(parsed_blocks, total_chars) sections_dicts = [section_to_dict(s) for s in outline["sections"]] + # heading 短锚只 hash 标题，不能表示章节正文是否变化。为每个 + # section 另存完整正文 hash（剥派生锚点+归一化），供摘要精确失效。 + def attach_section_hashes(sections: list[dict]) -> None: + for section in sections: + start = int(section.get("char_start", 0)) + end = int(section.get("char_end", start)) + section["section_sha256"] = _section_sha256( + text_with_anchors, start, end + ) + attach_section_hashes(section.get("children", [])) + + attach_section_hashes(sections_dicts) + if previous_outline: old_summaries = _collect_summaries(previous_outline.get("sections", [])) if old_summaries: _restore_summaries(sections_dicts, old_summaries) return { + "outline_schema_version": OUTLINE_SCHEMA_VERSION, "doc_path": doc_path, "doc_chars": total_chars, + "doc_sha256": hashlib.sha256(text_with_anchors.encode("utf-8")).hexdigest(), "doc_paragraphs": outline["paragraphs_count"], "generated_at": date.today().isoformat(), "sections": sections_dicts, } +def validate_outline( + outline: object, + text_with_anchors: str, + *, + expected_doc_path: str | None = None, +) -> list[dict[str, Any]]: + """Validate an ``outline.json`` against its current converted Markdown. + + The function is deliberately side-effect free and returns *all* detected + issues. An empty list is the only valid result. Callers must fail closed + on any issue or unexpected exception; a file's mtime is never evidence + that this content-addressed contract is fresh. + + Besides schema/document/section hashes, validation covers unique anchors, + ordered non-overlapping siblings, child containment, bounds, and exact + agreement with the section tree rebuilt from the current Markdown. The + last comparison catches structurally well-formed but incomplete outlines. + """ + issues: list[dict[str, Any]] = [] + + def add(code: str, path: str, **details: Any) -> None: + issues.append({"code": code, "path": path, **details}) + + if not isinstance(text_with_anchors, str): + add( + "markdown-not-text", + "$markdown", + actual_type=type(text_with_anchors).__name__, + ) + return issues + if not isinstance(outline, dict): + add("outline-not-object", "$", actual_type=type(outline).__name__) + return issues + + if outline.get("outline_schema_version") != OUTLINE_SCHEMA_VERSION: + add( + "outline-schema-version-mismatch", + "$.outline_schema_version", + expected=OUTLINE_SCHEMA_VERSION, + actual=outline.get("outline_schema_version"), + ) + + if expected_doc_path is not None and outline.get("doc_path") != expected_doc_path: + add( + "doc-path-mismatch", + "$.doc_path", + expected=expected_doc_path, + actual=outline.get("doc_path"), + ) + + doc_chars = outline.get("doc_chars") + if ( + isinstance(doc_chars, bool) + or not isinstance(doc_chars, int) + or doc_chars != len(text_with_anchors) + ): + add( + "doc-chars-mismatch", + "$.doc_chars", + expected=len(text_with_anchors), + actual=doc_chars, + ) + + actual_doc_sha256 = hashlib.sha256( + text_with_anchors.encode("utf-8") + ).hexdigest() + stored_doc_sha256 = outline.get("doc_sha256") + if not isinstance(stored_doc_sha256, str) or not _SHA256_RE.fullmatch( + stored_doc_sha256 + ): + add( + "doc-sha256-invalid", + "$.doc_sha256", + actual=stored_doc_sha256, + ) + elif stored_doc_sha256 != actual_doc_sha256: + add( + "doc-sha256-mismatch", + "$.doc_sha256", + expected=actual_doc_sha256, + actual=stored_doc_sha256, + ) + + # Canonical anchors are part of the converted-Markdown/outline contract. + # Re-anchor in memory and demand byte-for-byte stability; this catches a + # missing/stale non-heading anchor that outline section checks cannot see. + try: + canonical_markdown, _ = add_anchors(text_with_anchors) + except Exception as exc: # pragma: no cover - defensive fail-closed path + add( + "markdown-anchor-validation-error", + "$markdown", + exception=type(exc).__name__, + ) + else: + if canonical_markdown != text_with_anchors: + add("markdown-anchors-not-canonical", "$markdown") + + sections = outline.get("sections") + if not isinstance(sections, list): + add( + "sections-not-list", + "$.sections", + actual_type=type(sections).__name__, + ) + return issues + + seen_anchors: dict[str, str] = {} + actual_structure: dict[str, dict[str, Any]] = {} + reused_nodes: set[int] = set() + + def valid_int(value: object) -> bool: + return isinstance(value, int) and not isinstance(value, bool) + + def walk( + siblings: list[object], + *, + parent: dict[str, Any] | None, + parent_anchor: str | None, + json_path: str, + depth: int, + ) -> None: + if depth > 64: + add("section-depth-exceeded", json_path, maximum=64) + return + previous_end: int | None = None + previous_start: int | None = None + for index, raw_section in enumerate(siblings): + path = f"{json_path}[{index}]" + if not isinstance(raw_section, dict): + add( + "section-not-object", + path, + actual_type=type(raw_section).__name__, + ) + continue + node_id = id(raw_section) + if node_id in reused_nodes: + add("section-node-reused", path) + continue + reused_nodes.add(node_id) + + anchor = raw_section.get("anchor") + if not isinstance(anchor, str) or not re.fullmatch( + r"h-[1-6]-\d+-[0-9a-f]{6}(?:-(?:\d+|x))?", anchor + ): + add("section-anchor-invalid", f"{path}.anchor", actual=anchor) + anchor_key: str | None = None + else: + anchor_key = anchor + if anchor in seen_anchors: + add( + "section-anchor-duplicate", + f"{path}.anchor", + anchor=anchor, + first_path=seen_anchors[anchor], + ) + else: + seen_anchors[anchor] = f"{path}.anchor" + + level = raw_section.get("level") + seq = raw_section.get("seq") + line = raw_section.get("line") + if not valid_int(level) or not 1 <= level <= 6: + add("section-level-invalid", f"{path}.level", actual=level) + if not valid_int(seq) or seq < 1: + add("section-seq-invalid", f"{path}.seq", actual=seq) + if not valid_int(line) or line < 1: + add("section-line-invalid", f"{path}.line", actual=line) + if not isinstance(raw_section.get("title"), str): + add("section-title-invalid", f"{path}.title") + if not isinstance(raw_section.get("preview"), str): + add("section-preview-invalid", f"{path}.preview") + summary = raw_section.get("agent_summary") + if summary is not None and not isinstance(summary, str): + add("section-summary-invalid", f"{path}.agent_summary") + + start = raw_section.get("char_start") + end = raw_section.get("char_end") + range_valid = ( + valid_int(start) + and valid_int(end) + and 0 <= start < end <= len(text_with_anchors) + ) + if not range_valid: + add( + "section-range-out-of-bounds", + path, + char_start=start, + char_end=end, + doc_chars=len(text_with_anchors), + ) + else: + assert isinstance(start, int) and isinstance(end, int) + if previous_start is not None and start < previous_start: + add( + "section-order-invalid", + path, + previous_start=previous_start, + char_start=start, + ) + if previous_end is not None and start < previous_end: + add( + "section-sibling-overlap", + path, + previous_end=previous_end, + char_start=start, + ) + previous_start, previous_end = start, end + + if parent is not None: + parent_start = parent.get("char_start") + parent_end = parent.get("char_end") + if ( + not valid_int(parent_start) + or not valid_int(parent_end) + or not (parent_start < start and end <= parent_end) + ): + add( + "section-child-containment", + path, + parent_start=parent_start, + parent_end=parent_end, + child_start=start, + child_end=end, + ) + parent_level = parent.get("level") + if ( + valid_int(parent_level) + and valid_int(level) + and level <= parent_level + ): + add( + "section-child-level-invalid", + f"{path}.level", + parent_level=parent_level, + child_level=level, + ) + + stored_section_sha256 = raw_section.get("section_sha256") + if not isinstance(stored_section_sha256, str) or not _SHA256_RE.fullmatch( + stored_section_sha256 + ): + add( + "section-sha256-invalid", + f"{path}.section_sha256", + actual=stored_section_sha256, + ) + else: + current_section_sha256 = _section_sha256( + text_with_anchors, start, end + ) + if stored_section_sha256 != current_section_sha256: + add( + "section-sha256-mismatch", + f"{path}.section_sha256", + anchor=anchor, + expected=current_section_sha256, + actual=stored_section_sha256, + ) + + if anchor_key is not None: + actual_structure.setdefault(anchor_key, { + "parent_anchor": parent_anchor, + "sibling_index": index, + "level": level, + "seq": seq, + "title": raw_section.get("title"), + "line": line, + "char_start": start, + "char_end": end, + }) + + children = raw_section.get("children") + if not isinstance(children, list): + add( + "section-children-not-list", + f"{path}.children", + actual_type=type(children).__name__, + ) + else: + walk( + children, + parent=raw_section, + parent_anchor=anchor_key, + json_path=f"{path}.children", + depth=depth + 1, + ) + + try: + walk( + sections, + parent=None, + parent_anchor=None, + json_path="$.sections", + depth=0, + ) + except RecursionError: # pragma: no cover - malicious in-memory object + add("section-depth-exceeded", "$.sections", maximum=64) + + # Rebuild a clean structural oracle from the same current Markdown. The + # explicit range/hash checks above explain local corruption; this exact + # comparison additionally catches omitted/extra/reparented sections. + try: + expected = build_outline_data( + text_with_anchors, + expected_doc_path or str(outline.get("doc_path") or ""), + ) + except Exception as exc: # pragma: no cover - defensive fail-closed path + add( + "outline-rebuild-validation-error", + "$", + exception=type(exc).__name__, + ) + return issues + + expected_structure: dict[str, dict[str, Any]] = {} + + def collect_expected( + expected_sections: list[dict], + parent_anchor: str | None = None, + ) -> None: + for index, section in enumerate(expected_sections): + anchor = section.get("anchor") + if isinstance(anchor, str): + expected_structure[anchor] = { + "parent_anchor": parent_anchor, + "sibling_index": index, + "level": section.get("level"), + "seq": section.get("seq"), + "title": section.get("title"), + "line": section.get("line"), + "char_start": section.get("char_start"), + "char_end": section.get("char_end"), + } + collect_expected(section.get("children", []), anchor) + + collect_expected(expected["sections"]) + missing = sorted(set(expected_structure) - set(actual_structure)) + extra = sorted(set(actual_structure) - set(expected_structure)) + if missing or extra: + add( + "section-set-mismatch", + "$.sections", + missing_anchors=missing, + extra_anchors=extra, + ) + for anchor in sorted(set(expected_structure) & set(actual_structure)): + if actual_structure[anchor] != expected_structure[anchor]: + add( + "section-structure-mismatch", + seen_anchors.get(anchor, "$.sections"), + anchor=anchor, + expected=expected_structure[anchor], + actual=actual_structure[anchor], + ) + + doc_paragraphs = outline.get("doc_paragraphs") + if ( + isinstance(doc_paragraphs, bool) + or not isinstance(doc_paragraphs, int) + or doc_paragraphs != expected["doc_paragraphs"] + ): + add( + "doc-paragraphs-mismatch", + "$.doc_paragraphs", + expected=expected["doc_paragraphs"], + actual=doc_paragraphs, + ) + return issues + + def process(text: str, doc_path: str, previous_outline: dict | None = None) -> tuple[str, dict]: """ 一站式：text → (含锚点文本, outline 字典) diff --git a/scripts/retrieval_index.py b/scripts/retrieval_index.py new file mode 100644 index 0000000..0fd3b5c --- /dev/null +++ b/scripts/retrieval_index.py @@ -0,0 +1,2968 @@ +"""Deterministic long-document evidence index. + +This module is deliberately model-free: converted ``raw/**/*.md`` files are +the source, SQLite is a disposable derivative, and retrieval uses only FTS5 +BM25, the FTS5 trigram tokenizer, exact substring matching, and reciprocal +rank fusion (RRF). + +Public API +---------- +``rebuild_index(workspace_root, db_path=None)`` + Atomically rebuild the index for one workspace. +``coverage_report(db_path)`` + Compare the independently stored inventory with the materialized index. +``search_evidence(db_path, query, limit=20, expansions=None)`` + Run unicode61/trigram/exact channels and fuse their rankings with RRF. + +Natural evidence units are paragraphs, individual list items, table *data* +rows, blockquotes, fenced code blocks, and figures. Headings are registered +separately as sections. A child unit keeps the canonical anchor of its raw +parent block because list items and table rows do not have invented Markdown +anchors of their own. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import sqlite3 +import tempfile +import unicodedata +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Iterable, Sequence + +from conversion_receipt import ( + ConversionBinding, + ConversionReceiptError, + validate_conversion_binding, +) +from postprocess import OUTLINE_SCHEMA_VERSION, _normalize_for_hash, strip_anchors +from section_parser import FIGURE_RE, LIST_RE, split_blocks, strip_frontmatter + + +SCHEMA_VERSION = 5 +DEFAULT_DB_NAME = "retrieval_index.db" +RRF_K = 60 +ROUTING_RRF_WEIGHT = 0.01 +DEFAULT_EVIDENCE_TEXT_LIMIT = 30_000 +_FTS_DDL = { + "unit_fts_unicode": """CREATE VIRTUAL TABLE unit_fts_unicode USING fts5( + unit_id UNINDEXED, text, heading_path, section_summary, path, + tokenize='unicode61 remove_diacritics 2' + )""", + "unit_fts_trigram": """CREATE VIRTUAL TABLE unit_fts_trigram USING fts5( + unit_id UNINDEXED, text, heading_path, section_summary, path, + tokenize='trigram' + )""", +} +NATURAL_KINDS = frozenset( + {"paragraph", "list_item", "table_row", "blockquote", "code", "figure"} +) + +_BLOCK_ANCHOR_RE = re.compile( + r"(?:[ \t]+)\^([hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?)\s*$" +) +_CANONICAL_ANCHOR_RE = re.compile( + r"^[hpcft]-\d+(?:-\d+)?-([a-f0-9]{6})(?:-\d+)?$" +) +_ALL_ANCHOR_TAILS_RE = re.compile( + r"[ \t]+\^[hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?(?=\s|$)", + re.MULTILINE, +) +_TABLE_SEPARATOR_CELL_RE = re.compile(r"^:?-{3,}:?$") +_WORD_RE = re.compile(r"[\w]+", re.UNICODE) +_FTS_STOPWORDS = frozenset({ + "a", "an", "the", "is", "are", "was", "were", "be", "been", "being", + "what", "which", "who", "how", "when", "where", "why", "does", "do", "did", + "for", "of", "to", "in", "on", "at", "by", "with", "from", "under", "after", + "before", "and", "or", "this", "that", "these", "those", +}) +_QUERY_EXCLUSION_RE = re.compile( + r"(?:\bnot\b|\bdoes\s+not\b|\bmust\s+not\b|\bnot\s+applicable\b|" + r"\bnon[- ]applicable\b|\bexclude(?:d|s|ing)?\b|" + r"\bgovern(?:s|ed|ing|ance)?\b|\bapplicab(?:le|ility)\b|" + r"\bappl(?:y|ies|ied|ying)\b|" + r"不适用|不得|并非|不是|排除|除外)", + re.IGNORECASE, +) +_BINARY_QUERY_RE = re.compile( + r"^\s*(?:does|do|did|is|are|was|were|can|could|will|would|should|may|must|" + r"has|have|had)\b|\bwhether\b|是否|能否|可否|有没有|会不会", + re.IGNORECASE, +) +_QUANTITATIVE_QUERY_RE = re.compile( + r"\bhow\s+(?:much|many|long)\b|" + r"\b(?:maximum|minimum|limit|ceiling|capacity|duration|period|deadline|" + r"window|age|temperature|humidity|tolerance|distance|latency|precision|" + r"recall|rate|ratio|yield|volume|torque|pressure|retries|retention|" + r"endurance|drift|occupancy|energy|throughput|count|quantity|value)\b|" + r"多少|多长|数量|数值|时长|期限|温度|湿度|年龄|容量|" + r"体积|距离|速率|比例|精度|延迟|上限|下限|阈值|" + r"等待时间|休止周期|最低照度", + re.IGNORECASE, +) +_VALUE_PROPERTY_QUERY_RE = re.compile( + r"\b(?:what|which)\s+(?:is|are|was|were)\s+(?:the\s+)?" + r"(?P[\w][\w /-]{0,80}?)\s+(?:of|for)\b", + re.IGNORECASE, +) +_POSSESSIVE_PROPERTY_QUERY_RE = re.compile( + r"\b(?:what|which)\s+(?:is|are|was|were)\s+(?:the\s+)?" + r"[^?]{1,100}?(?:['’]s)\s+" + r"(?P[\w-]+(?:\s+[\w-]+){0,5}?)" + r"(?=\s+(?:of|for|over|under|in|at|on|with)\b|[?.!]|$)", + re.IGNORECASE, +) +_TELEGRAPHIC_VALUE_QUERY_RE = re.compile( + r"\b(?:what|which)\s+(?:is|are|was|were)\s+(?:the\s+)?" + r"(?P[^?!.]{2,120})[?!.]?\s*$", + re.IGNORECASE, +) +_RELATIONAL_QUERY_RE = re.compile( + r"\b(?:role|status|relationship|relation|governance|applicability|" + r"responsibility|ownership|support|involvement)\b|" + r"角色|状态|关系|关联|适用性|归属|职责|参与度", + re.IGNORECASE, +) +_PROPERTY_VALUE_AFTER_RE = re.compile( + r"^\s*(?:(?:is|are|was|were|equals?|of|at)\s+|[|:=]\s*){0,3}" + r"(?:(?:approximately|approx\.?|about|around|roughly|nearly|" + r"at\s+least|at\s+most|no\s+more\s+than|no\s+less\s+than|" + r"more\s+than|less\s+than|约|大约|近似)\s+)?" + r"(?:[<>≤≥~≈]\s*)?(?:[+\-−±]\s*)?" + r"(?P[$€£¥]\s*)?" + r"(?:[+\-−±]\s*)?" + r"(?P\d{1,3}(?:[,_]\d{3})*(?:\.\d+)?|\d+(?:\.\d+)?)" + r"(?P\s*(?:%|[A-Za-zµμ°][A-Za-z0-9µμ°/%._-]*))?", + re.IGNORECASE, +) +_EXCLUDED_OBJECT_RE = re.compile( + r"(?:,\s*not\s+(?:the\s+)?(?P[^,.;:\n]{1,120})(?=[,.;:\n]|$)|" + r"[;；]\s*(?:the\s+clause\s+)?does\s+not\s+govern\s+(?:the\s+)?(?P[^,.;:\n]{1,120})|" + r"[;；]\s*not\s+applicable\s+to\s+(?:the\s+)?(?P[^,.;:\n]{1,120})|" + r"[;；]\s*(?:this\s+value\s+)?must\s+not\s+be\s+applied\s+to\s+(?:the\s+)?(?P[^,.;:\n]{1,120})|" + r"[;；]\s*不适用于(?P[^，。；：\n]{1,120}))", + re.IGNORECASE, +) +# A full stop is non-boundary only when it is literally between two digits +# (for example 87.5 or firmware 4.2). Sentence-final ``17.`` must still split. +_ATOMIC_CLAUSE_BOUNDARY_RE = re.compile( + r"(? None: + super().__init__(message) + self.code = code + self.message = message + self.details = details or {} + + def to_dict(self) -> dict[str, Any]: + return { + "ok": False, + "error": { + "code": self.code, + "message": self.message, + "details": self.details, + }, + } + + +@dataclass(frozen=True) +class _Unit: + unit_id: str + path: str + anchor: str + kind: str + parent_kind: str + ordinal: int + subordinal: int + owning_section_anchor: str | None + heading_path: tuple[str, ...] + heading_anchors: tuple[str, ...] + section_summary: str | None + line_start: int + line_end: int + char_start: int + char_end: int + text: str + normalized_text: str + content_hash: str + exact_text_hash: str + + +@dataclass(frozen=True) +class _Section: + section_id: str + path: str + anchor: str + level: int + ordinal: int + title: str + heading_path: tuple[str, ...] + heading_anchors: tuple[str, ...] + line_start: int + char_start: int + char_end: int + content_hash: str + agent_summary: str | None + is_structural: bool + is_content: bool + direct_unit_count: int + descendant_unit_count: int + + +def _sha256(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _normalize_text(text: str) -> str: + """Full-content normalization used for stable hashes (never truncated).""" + + return re.sub(r"\s+", " ", unicodedata.normalize("NFC", text)).strip() + + +def _exact_text_hash(text: str) -> str: + """Hash the full unit without folding structure-significant whitespace.""" + + return _sha256(unicodedata.normalize("NFC", text)) + + +def _normalize_schema_sql(sql: str) -> str: + return re.sub(r"\s+", " ", sql).strip().rstrip(";").casefold() + + +def _searchable_text(text: str) -> str: + """Keep evidence readable while exposing Markdown labels to FTS.""" + + value = _normalize_text(text) + value = re.sub(r"!\[([^\]]*)\]\(([^)]*)\)", r"\1 \2", value) + value = re.sub(r"\[([^\]]+)\]\(([^)]*)\)", r"\1 \2", value) + value = re.sub(r"^\s*(?:[-*+]|\d+\.)\s+", "", value) + value = re.sub(r"^\s*>\s?", "", value) + return value + + +def _as_path(value: str | os.PathLike[str], *, label: str) -> Path: + try: + return Path(value).expanduser().resolve() + except (OSError, TypeError, ValueError) as exc: + raise RetrievalIndexError( + "invalid-path", + f"Invalid {label}", + {"label": label, "value": str(value), "exception": type(exc).__name__}, + ) from exc + + +def _is_relative_to(path: Path, parent: Path) -> bool: + try: + path.relative_to(parent) + return True + except ValueError: + return False + + +def _default_db_path(workspace_root: Path) -> Path: + return workspace_root / ".cache" / DEFAULT_DB_NAME + + +def _resolve_paths( + workspace_root: str | os.PathLike[str], + db_path: str | os.PathLike[str] | None, +) -> tuple[Path, Path, Path]: + workspace = _as_path(workspace_root, label="workspace_root") + if not workspace.is_dir(): + raise RetrievalIndexError( + "workspace-not-found", + "Workspace root does not exist or is not a directory", + {"workspace_root": str(workspace)}, + ) + raw_root = workspace / "raw" + if not raw_root.is_dir(): + raise RetrievalIndexError( + "raw-root-not-found", + "Workspace has no raw directory", + {"workspace_root": str(workspace), "raw_root": str(raw_root)}, + ) + target = _as_path(db_path, label="db_path") if db_path is not None else _default_db_path(workspace) + # The schema makes raw immutable. A caller-provided derivative path may + # never smuggle an SQLite write into raw/. + if _is_relative_to(target, raw_root.resolve()): + raise RetrievalIndexError( + "write-protected-path", + "The retrieval index may not be written under raw/", + {"db_path": str(target), "raw_root": str(raw_root)}, + ) + return workspace, raw_root.resolve(), target + + +def _strip_parent_anchor(text: str) -> tuple[str, str | None]: + match = _BLOCK_ANCHOR_RE.search(text) + if not match: + return text.rstrip(), None + return text[: match.start()].rstrip(), match.group(1) + + +def _anchor_kind(anchor: str) -> str: + return anchor.split("-", 1)[0] + + +def _validate_anchor( + path: str, + block_kind: str, + anchor: str | None, + line: int, + hash_source: str, + expected_anchor: str, +) -> str: + if not anchor: + raise RetrievalIndexError( + "missing-canonical-anchor", + "A converted raw block has no canonical anchor", + {"path": path, "kind": block_kind, "line": line}, + ) + expected = { + "heading": "h", + "paragraph": "p", + "list": "p", + "blockquote": "p", + "table": "t", + "code": "c", + # An anchored figure is parsed as paragraph by the legacy parser; + # prefix f is authoritative and handled by the caller. + "figure": "f", + }.get(block_kind) + actual = _anchor_kind(anchor) + if expected and actual != expected and not (block_kind == "paragraph" and actual == "f"): + raise RetrievalIndexError( + "canonical-anchor-kind-mismatch", + "Block kind and canonical anchor prefix disagree", + { + "path": path, + "kind": block_kind, + "anchor": anchor, + "line": line, + "expected_prefix": expected, + }, + ) + canonical_match = _CANONICAL_ANCHOR_RE.fullmatch(anchor) + expected_hash = hashlib.md5( + _normalize_for_hash(hash_source).encode("utf-8") + ).hexdigest()[:6] + if canonical_match is None or canonical_match.group(1) != expected_hash: + raise RetrievalIndexError( + "stale-canonical-anchor", + "Canonical anchor content hash does not match the current raw block", + { + "path": path, + "kind": block_kind, + "anchor": anchor, + "line": line, + "expected_hash": expected_hash, + "actual_hash": canonical_match.group(1) if canonical_match else None, + }, + ) + if anchor != expected_anchor: + raise RetrievalIndexError( + "canonical-anchor-coordinate-mismatch", + "Canonical anchor level/sequence does not match the current document order", + { + "path": path, + "kind": block_kind, + "anchor": anchor, + "expected_anchor": expected_anchor, + "line": line, + }, + ) + return anchor + + +def _line_slices(text: str) -> list[tuple[str, int, int, int]]: + """Return ``(line_without_newline, start, end, zero_based_line)``.""" + + result: list[tuple[str, int, int, int]] = [] + cursor = 0 + for number, raw_line in enumerate(text.splitlines(keepends=True)): + line = raw_line.rstrip("\r\n") + result.append((line, cursor, cursor + len(line), number)) + cursor += len(raw_line) + if not result and text: + result.append((text, 0, len(text), 0)) + return result + + +def _trim_range(text: str, start: int, end: int) -> tuple[str, int, int]: + while start < end and text[start].isspace(): + start += 1 + while end > start and text[end - 1].isspace(): + end -= 1 + return text[start:end], start, end + + +def _split_list_items(text: str) -> list[tuple[str, int, int, int, int]]: + lines = _line_slices(text) + starts = [index for index, (line, _s, _e, _n) in enumerate(lines) if LIST_RE.match(line)] + if not starts: + clean, start, end = _trim_range(text, 0, len(text)) + return [(clean, start, end, 0, max(0, len(lines) - 1))] if clean else [] + items: list[tuple[str, int, int, int, int]] = [] + for position, line_index in enumerate(starts): + next_line_index = starts[position + 1] if position + 1 < len(starts) else len(lines) + start = lines[line_index][1] + end = lines[next_line_index - 1][2] + clean, start, end = _trim_range(text, start, end) + if clean: + items.append((clean, start, end, line_index, next_line_index - 1)) + return items + + +def _is_table_separator(line: str) -> bool: + cells = [cell.strip() for cell in line.strip().strip("|").split("|")] + return bool(cells) and all(_TABLE_SEPARATOR_CELL_RE.fullmatch(cell) for cell in cells) + + +def _split_table_rows(text: str) -> list[tuple[str, int, int, int, int]]: + lines = _line_slices(text) + if not lines: + return [] + separator_indexes = [i for i, row in enumerate(lines) if _is_table_separator(row[0])] + if separator_indexes: + # For valid Markdown the first separator follows the header. Only + # rows after it are evidence data rows; later separator-like rows are + # formatting, not facts. + indexes = range(separator_indexes[0] + 1, len(lines)) + else: + # Malformed-but-parsed tables have no reliable header boundary. Do + # not silently lose them: index every non-empty row as data. + indexes = range(len(lines)) + rows: list[tuple[str, int, int, int, int]] = [] + for index in indexes: + line, start, end, _ = lines[index] + if not line.strip() or _is_table_separator(line): + continue + clean, start, end = _trim_range(text, start, end) + if clean: + rows.append((clean, start, end, index, index)) + return rows + + +def _load_outline_summaries( + md_path: Path, + text: str, + display_path: str, +) -> tuple[dict[str, str], list[dict[str, Any]], dict[str, Any] | None]: + outline_path = md_path.with_suffix(".outline.json") + if not outline_path.exists(): + return {}, [], None + try: + data = json.loads(outline_path.read_text(encoding="utf-8")) + except (OSError, UnicodeError, json.JSONDecodeError) as exc: + return {}, [{ + "code": "outline-unreadable", + "path": display_path, + "outline_path": str(Path(display_path).with_suffix(".outline.json")), + "exception": type(exc).__name__, + }], None + if not isinstance(data, dict) or not isinstance(data.get("sections", []), list): + return {}, [{"code": "outline-invalid-schema", "path": display_path}], ( + data if isinstance(data, dict) else None + ) + actual_sha256 = _sha256(text) + if ( + data.get("outline_schema_version") != OUTLINE_SCHEMA_VERSION + or data.get("doc_sha256") != actual_sha256 + ): + return {}, [{ + "code": "outline-stale", + "path": display_path, + "outline_schema_version": data.get("outline_schema_version"), + "expected_outline_schema_version": OUTLINE_SCHEMA_VERSION, + "outline_doc_chars": data.get("doc_chars"), + "actual_doc_chars": len(text), + "outline_doc_sha256": data.get("doc_sha256"), + "actual_doc_sha256": actual_sha256, + }], data + summaries: dict[str, str] = {} + warnings: list[dict[str, Any]] = [] + flattened: list[dict[str, Any]] = [] + + def walk(sections: Iterable[Any]) -> None: + for section in sections: + if not isinstance(section, dict): + warnings.append({ + "code": "outline-section-invalid-schema", + "path": display_path, + }) + continue + flattened.append(section) + children = section.get("children", []) + if isinstance(children, list): + walk(children) + else: + warnings.append({ + "code": "outline-section-invalid-children", + "path": display_path, + "anchor": section.get("anchor"), + }) + + walk(data["sections"]) + anchor_counts: dict[str, int] = {} + for section in flattened: + anchor = section.get("anchor") + if isinstance(anchor, str) and anchor.strip(): + normalized_anchor = anchor.lstrip("^") + anchor_counts[normalized_anchor] = anchor_counts.get(normalized_anchor, 0) + 1 + + for section in flattened: + summary = section.get("agent_summary") + if not isinstance(summary, str) or not summary.strip(): + continue + anchor_raw = section.get("anchor") + anchor = anchor_raw.lstrip("^") if isinstance(anchor_raw, str) else "" + start = section.get("char_start") + end = section.get("char_end") + declared_hash = section.get("section_sha256") + reasons: list[str] = [] + if not anchor or anchor_counts.get(anchor) != 1: + reasons.append("anchor-missing-or-duplicate") + if ( + isinstance(start, bool) + or isinstance(end, bool) + or not isinstance(start, int) + or not isinstance(end, int) + or start < 0 + or end <= start + or end > len(text) + ): + reasons.append("invalid-section-range") + section_text = "" + else: + section_text = text[start:end] + actual_hash = _sha256(_normalize_for_hash(strip_anchors(section_text))) + if not isinstance(declared_hash, str) or declared_hash != actual_hash: + reasons.append("section-hash-mismatch") + section_blocks = split_blocks(section_text) + first = section_blocks[0] if section_blocks else None + first_anchor = _strip_parent_anchor(first.text)[1] if first else None + if first is None or first.kind != "heading" or first_anchor != anchor: + reasons.append("section-anchor-range-mismatch") + if reasons: + warnings.append({ + "code": "outline-summary-unverifiable", + "path": display_path, + "anchor": anchor or None, + "reasons": reasons, + }) + continue + summaries[anchor] = summary.strip() + return summaries, warnings, data + + +def _outline_summary_fingerprint(summaries: dict[str, str]) -> str: + """Hash only validated outline summaries that participate in routing.""" + + payload = json.dumps( + sorted((anchor, summary) for anchor, summary in summaries.items()), + ensure_ascii=False, + separators=(",", ":"), + ) + return _sha256(payload) + + +def _inventory_fingerprint(records: Iterable[list[Any]]) -> str: + """Freeze a canonical, order-independent expected-ledger snapshot.""" + + encoded = sorted( + json.dumps(record, ensure_ascii=False, separators=(",", ":")).encode("utf-8") + for record in records + ) + digest = hashlib.sha256() + for record in encoded: + digest.update(len(record).to_bytes(8, "big")) + digest.update(record) + return digest.hexdigest() + + +def _unit_inventory_record(unit: _Unit) -> list[Any]: + return [ + "unit", + unit.unit_id, + unit.path, + unit.anchor, + unit.kind, + unit.parent_kind, + unit.ordinal, + unit.subordinal, + unit.owning_section_anchor, + json.dumps(unit.heading_path, ensure_ascii=False), + json.dumps(unit.heading_anchors, ensure_ascii=False), + unit.section_summary, + unit.line_start, + unit.line_end, + unit.char_start, + unit.char_end, + unit.normalized_text, + unit.content_hash, + unit.exact_text_hash, + ] + + +def _section_inventory_records(section: _Section) -> list[list[Any]]: + records: list[list[Any]] = [] + if section.is_structural: + records.append([ + "structural-section", + section.section_id, + section.path, + section.anchor, + section.level, + section.title, + section.content_hash, + ]) + if section.is_content: + records.append([ + "content-section", + section.section_id, + section.path, + section.anchor, + section.title, + section.content_hash, + ]) + return records + + +def _document_inventory_fingerprint( + units: Iterable[_Unit], sections: Iterable[_Section] +) -> str: + records = [_unit_inventory_record(unit) for unit in units] + for section in sections: + records.extend(_section_inventory_records(section)) + return _inventory_fingerprint(records) + + +def _stable_ids( + path: str, + kind_and_hash: Iterable[tuple[str, str]], + *, + namespace: str, +) -> list[str]: + occurrences: dict[tuple[str, str], int] = {} + ids: list[str] = [] + for kind, content_hash in kind_and_hash: + key = (kind, content_hash) + occurrence = occurrences.get(key, 0) + 1 + occurrences[key] = occurrence + ids.append(_sha256(f"{path}\0{namespace}\0{kind}\0{content_hash}\0{occurrence}")[:32]) + return ids + + +def _inventory_document( + md_path: Path, + workspace_root: Path, +) -> tuple[ + str, + str, + str, + int, + ConversionBinding, + list[_Unit], + list[_Section], + list[dict[str, Any]], +]: + try: + text = md_path.read_text(encoding="utf-8") + except (OSError, UnicodeError) as exc: + raise RetrievalIndexError( + "document-unreadable", + "Raw Markdown document could not be read as UTF-8", + {"path": md_path.as_posix(), "exception": type(exc).__name__}, + ) from exc + + path = md_path.relative_to(workspace_root).as_posix() + source_hash = _sha256(text) + frontmatter, body = strip_frontmatter(text) + offset = len(frontmatter) + line_offset = frontmatter.count("\n") + blocks = split_blocks(body) + summaries, warnings, outline = _load_outline_summaries(md_path, text, path) + outline_summary_hash = _outline_summary_fingerprint(summaries) + try: + conversion_binding = validate_conversion_binding( + md_path, + outline, + workspace_root, + markdown_sha256=source_hash, + expected_outline_schema_version=OUTLINE_SCHEMA_VERSION, + ) + except ConversionReceiptError as exc: + raise RetrievalIndexError( + "conversion-binding-invalid", + "A raw Markdown derivative has invalid conversion provenance", + {"path": path, "issue": exc.to_dict()}, + ) from exc + + heading_stack: list[dict[str, Any]] = [] + heading_records: list[dict[str, Any]] = [] + unit_drafts: list[dict[str, Any]] = [] + unit_ordinal = 0 + seen_block_anchors: set[str] = set() + heading_seq_by_level: dict[int, int] = {} + evidence_block_seq = 0 + + for block_ordinal, block in enumerate(blocks, start=1): + if block.kind == "hr": + continue + clean_text, detected_anchor = _strip_parent_anchor(block.text) + hash_source = (block.title or "") if block.kind == "heading" else clean_text + expected_hash = hashlib.md5( + _normalize_for_hash(hash_source).encode("utf-8") + ).hexdigest()[:6] + if block.kind == "heading": + level = int(block.level or 1) + heading_seq_by_level[level] = heading_seq_by_level.get(level, 0) + 1 + expected_anchor = f"h-{level}-{heading_seq_by_level[level]}-{expected_hash}" + else: + evidence_block_seq += 1 + expected_prefix = { + "paragraph": "p", + "list": "p", + "blockquote": "p", + "table": "t", + "code": "c", + "figure": "f", + }.get(block.kind, "p") + if ( + block.kind == "paragraph" + and detected_anchor + and _anchor_kind(detected_anchor) == "f" + ): + if FIGURE_RE.fullmatch(clean_text.strip()) is None: + raise RetrievalIndexError( + "canonical-anchor-kind-mismatch", + "A figure anchor may only label a Markdown image block", + { + "path": path, + "kind": block.kind, + "anchor": detected_anchor, + "line": block.line_start + line_offset, + "expected_prefix": "p", + }, + ) + expected_prefix = "f" + expected_anchor = f"{expected_prefix}-{evidence_block_seq}-{expected_hash}" + anchor = _validate_anchor( + path, + block.kind, + detected_anchor, + block.line_start + line_offset, + hash_source, + expected_anchor, + ) + if anchor in seen_block_anchors: + raise RetrievalIndexError( + "duplicate-canonical-anchor", + "Two different raw blocks share one canonical anchor", + {"path": path, "anchor": anchor, "line": block.line_start + line_offset}, + ) + seen_block_anchors.add(anchor) + block_start = offset + block.char_start + block_end = block_start + len(clean_text) + + if block.kind == "heading": + level = int(block.level or 1) + while heading_stack and int(heading_stack[-1]["level"]) >= level: + heading_stack.pop() + title = block.title or re.sub(r"^#{1,6}\s+", "", clean_text).strip() + record = { + "anchor": anchor, + "level": level, + "title": title, + "line_start": block.line_start + line_offset, + "char_start": block_start, + "heading_path": tuple([entry["title"] for entry in heading_stack] + [title]), + "heading_anchors": tuple([entry["anchor"] for entry in heading_stack] + [anchor]), + "ordinal": len(heading_records) + 1, + "agent_summary": summaries.get(anchor), + } + heading_records.append(record) + heading_stack.append(record) + continue + + actual_kind = "figure" if _anchor_kind(anchor) == "f" else block.kind + heading_path = tuple(entry["title"] for entry in heading_stack) + heading_anchors = tuple(entry["anchor"] for entry in heading_stack) + owning_anchor = heading_anchors[-1] if heading_anchors else None + section_summary = summaries.get(owning_anchor or "") + + subunits: list[tuple[str, str, int, int, int, int]] = [] + if actual_kind == "list": + subunits = [ + ("list_item", value, start, end, line_start, line_end) + for value, start, end, line_start, line_end in _split_list_items(clean_text) + ] + elif actual_kind == "table": + subunits = [ + ("table_row", value, start, end, line_start, line_end) + for value, start, end, line_start, line_end in _split_table_rows(clean_text) + ] + elif actual_kind in {"paragraph", "blockquote", "code", "figure"}: + value, start, end = _trim_range(clean_text, 0, len(clean_text)) + if value: + subunits = [(actual_kind, value, start, end, 0, block.line_end - block.line_start)] + else: + raise RetrievalIndexError( + "unsupported-natural-block", + "A non-empty Markdown block has no indexing policy", + {"path": path, "kind": actual_kind, "line": block.line_start}, + ) + + if not subunits and clean_text.strip() and actual_kind != "table": + raise RetrievalIndexError( + "natural-unit-enumeration-empty", + "A non-empty evidence block produced no natural units", + {"path": path, "kind": actual_kind, "anchor": anchor}, + ) + + for subordinal, (kind, value, rel_start, rel_end, rel_line_start, rel_line_end) in enumerate( + subunits, start=1 + ): + unit_ordinal += 1 + normalized = _normalize_text(value) + unit_drafts.append({ + "path": path, + "anchor": anchor, + "kind": kind, + "parent_kind": actual_kind, + "ordinal": unit_ordinal, + "subordinal": subordinal, + "owning_section_anchor": owning_anchor, + "heading_path": heading_path, + "heading_anchors": heading_anchors, + "section_summary": section_summary, + "line_start": block.line_start + line_offset + rel_line_start, + "line_end": block.line_start + line_offset + rel_line_end, + "char_start": block_start + rel_start, + "char_end": block_start + rel_end, + "text": value, + "normalized_text": _searchable_text(value), + "content_hash": _sha256(normalized), + "exact_text_hash": _exact_text_hash(value), + "block_ordinal": block_ordinal, + "block_end": block_end, + }) + + unit_ids = _stable_ids( + path, + ((draft["kind"], draft["content_hash"]) for draft in unit_drafts), + namespace="unit-v1", + ) + units = [ + _Unit( + unit_id=unit_id, + **{key: value for key, value in draft.items() if key not in {"block_ordinal", "block_end"}}, + ) + for unit_id, draft in zip(unit_ids, unit_drafts) + ] + + section_title_hashes = [ + ("heading", _sha256(_normalize_text(record["title"]).casefold())) + for record in heading_records + ] + section_ids = _stable_ids(path, section_title_hashes, namespace="section-v1") + sections: list[_Section] = [] + h1_count = sum(1 for record in heading_records if int(record["level"]) == 1) + for index, (record, section_id) in enumerate(zip(heading_records, section_ids)): + end = len(text) + for later in heading_records[index + 1 :]: + if int(later["level"]) <= int(record["level"]): + end = int(later["char_start"]) + break + descendants = [u for u in units if int(record["char_start"]) < u.char_start < end] + direct_count = sum(1 for unit in descendants if unit.owning_section_anchor == record["anchor"]) + section_text = _ALL_ANCHOR_TAILS_RE.sub( + "", text[int(record["char_start"]):end] + ) + # A lone H1 is the document title, not a content chapter. Multiple + # H1s represent book-like top-level chapters and are registered as + # content sections. H2-H6 keep their normal section semantics. + is_structural = int(record["level"]) != 1 or h1_count > 1 + is_content = bool(descendants) and is_structural + sections.append(_Section( + section_id=section_id, + path=path, + anchor=record["anchor"], + level=int(record["level"]), + ordinal=int(record["ordinal"]), + title=record["title"], + heading_path=record["heading_path"], + heading_anchors=record["heading_anchors"], + line_start=int(record["line_start"]), + char_start=int(record["char_start"]), + char_end=end, + content_hash=_sha256(_normalize_text(section_text)), + agent_summary=record["agent_summary"], + is_structural=is_structural, + is_content=is_content, + direct_unit_count=direct_count, + descendant_unit_count=len(descendants), + )) + + return ( + path, + source_hash, + outline_summary_hash, + len(text), + conversion_binding, + units, + sections, + warnings, + ) + + +def _create_schema(connection: sqlite3.Connection) -> None: + connection.executescript( + """ + PRAGMA foreign_keys = ON; + CREATE TABLE meta ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL + ); + CREATE TABLE documents ( + path TEXT PRIMARY KEY, + source_hash TEXT NOT NULL, + outline_summary_hash TEXT NOT NULL, + document_origin TEXT NOT NULL, + original_source_path TEXT, + original_source_sha256 TEXT, + converter_fingerprint TEXT, + conversion_receipt_hash TEXT, + inventory_hash TEXT NOT NULL, + char_count INTEGER NOT NULL, + expected_units INTEGER NOT NULL, + indexed_units INTEGER NOT NULL, + expected_structural_sections INTEGER NOT NULL, + registered_structural_sections INTEGER NOT NULL, + expected_content_sections INTEGER NOT NULL, + registered_content_sections INTEGER NOT NULL + ); + CREATE TABLE expected_units ( + unit_id TEXT PRIMARY KEY, + path TEXT NOT NULL, + anchor TEXT NOT NULL, + kind TEXT NOT NULL, + parent_kind TEXT NOT NULL, + ordinal INTEGER NOT NULL, + subordinal INTEGER NOT NULL, + owning_section_anchor TEXT, + heading_path_json TEXT NOT NULL, + heading_anchors_json TEXT NOT NULL, + section_summary TEXT, + line_start INTEGER NOT NULL, + line_end INTEGER NOT NULL, + char_start INTEGER NOT NULL, + char_end INTEGER NOT NULL, + normalized_text TEXT NOT NULL, + content_hash TEXT NOT NULL, + exact_text_hash TEXT NOT NULL + ); + CREATE TABLE units ( + unit_id TEXT PRIMARY KEY, + path TEXT NOT NULL, + anchor TEXT NOT NULL, + kind TEXT NOT NULL, + parent_kind TEXT NOT NULL, + ordinal INTEGER NOT NULL, + subordinal INTEGER NOT NULL, + owning_section_anchor TEXT, + heading_path_json TEXT NOT NULL, + heading_anchors_json TEXT NOT NULL, + section_summary TEXT, + line_start INTEGER NOT NULL, + line_end INTEGER NOT NULL, + char_start INTEGER NOT NULL, + char_end INTEGER NOT NULL, + text TEXT NOT NULL, + normalized_text TEXT NOT NULL, + content_hash TEXT NOT NULL, + exact_text_hash TEXT NOT NULL + ); + CREATE INDEX units_path_ordinal ON units(path, ordinal); + CREATE INDEX units_anchor ON units(path, anchor); + CREATE INDEX units_section ON units(path, owning_section_anchor); + CREATE TABLE expected_structural_sections ( + section_id TEXT PRIMARY KEY, + path TEXT NOT NULL, + anchor TEXT NOT NULL, + level INTEGER NOT NULL, + title TEXT NOT NULL, + content_hash TEXT NOT NULL + ); + CREATE TABLE expected_sections ( + section_id TEXT PRIMARY KEY, + path TEXT NOT NULL, + anchor TEXT NOT NULL, + title TEXT NOT NULL, + content_hash TEXT NOT NULL + ); + CREATE TABLE sections ( + section_id TEXT PRIMARY KEY, + path TEXT NOT NULL, + anchor TEXT NOT NULL, + level INTEGER NOT NULL, + ordinal INTEGER NOT NULL, + title TEXT NOT NULL, + heading_path_json TEXT NOT NULL, + heading_anchors_json TEXT NOT NULL, + line_start INTEGER NOT NULL, + char_start INTEGER NOT NULL, + char_end INTEGER NOT NULL, + content_hash TEXT NOT NULL, + agent_summary TEXT, + is_structural INTEGER NOT NULL, + is_content INTEGER NOT NULL, + direct_unit_count INTEGER NOT NULL, + descendant_unit_count INTEGER NOT NULL + ); + CREATE UNIQUE INDEX sections_path_anchor ON sections(path, anchor); + """ + ) + try: + for ddl in _FTS_DDL.values(): + connection.execute(ddl) + except sqlite3.Error as exc: + raise RetrievalIndexError( + "fts5-unavailable", + "SQLite must provide FTS5 unicode61 and trigram tokenizers", + {"sqlite_version": sqlite3.sqlite_version, "sqlite_error": str(exc)}, + ) from exc + + +def _insert_inventory( + connection: sqlite3.Connection, + path: str, + source_hash: str, + outline_summary_hash: str, + char_count: int, + conversion_binding: ConversionBinding, + units: list[_Unit], + sections: list[_Section], +) -> None: + structural_sections = [section for section in sections if section.is_structural] + content_sections = [section for section in sections if section.is_content] + inventory_hash = _document_inventory_fingerprint(units, sections) + connection.execute( + "INSERT INTO documents VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + path, + source_hash, + outline_summary_hash, + conversion_binding.document_origin, + conversion_binding.original_source_path, + conversion_binding.original_source_sha256, + conversion_binding.converter_fingerprint, + conversion_binding.conversion_receipt_hash, + inventory_hash, + char_count, + len(units), + len(units), + len(structural_sections), + len(structural_sections), + len(content_sections), + len(content_sections), + ), + ) + for unit in units: + connection.execute( + "INSERT INTO expected_units VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + unit.unit_id, + unit.path, + unit.anchor, + unit.kind, + unit.parent_kind, + unit.ordinal, + unit.subordinal, + unit.owning_section_anchor, + json.dumps(unit.heading_path, ensure_ascii=False), + json.dumps(unit.heading_anchors, ensure_ascii=False), + unit.section_summary, + unit.line_start, + unit.line_end, + unit.char_start, + unit.char_end, + unit.normalized_text, + unit.content_hash, + unit.exact_text_hash, + ), + ) + connection.execute( + "INSERT INTO units VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + unit.unit_id, + unit.path, + unit.anchor, + unit.kind, + unit.parent_kind, + unit.ordinal, + unit.subordinal, + unit.owning_section_anchor, + json.dumps(unit.heading_path, ensure_ascii=False), + json.dumps(unit.heading_anchors, ensure_ascii=False), + unit.section_summary, + unit.line_start, + unit.line_end, + unit.char_start, + unit.char_end, + unit.text, + unit.normalized_text, + unit.content_hash, + unit.exact_text_hash, + ), + ) + fts_values = ( + unit.unit_id, + unit.normalized_text, + " / ".join(unit.heading_path), + unit.section_summary or "", + unit.path, + ) + connection.execute("INSERT INTO unit_fts_unicode VALUES (?, ?, ?, ?, ?)", fts_values) + connection.execute("INSERT INTO unit_fts_trigram VALUES (?, ?, ?, ?, ?)", fts_values) + + for section in sections: + if section.is_structural: + connection.execute( + "INSERT INTO expected_structural_sections VALUES (?, ?, ?, ?, ?, ?)", + ( + section.section_id, + section.path, + section.anchor, + section.level, + section.title, + section.content_hash, + ), + ) + if section.is_content: + connection.execute( + "INSERT INTO expected_sections VALUES (?, ?, ?, ?, ?)", + ( + section.section_id, + section.path, + section.anchor, + section.title, + section.content_hash, + ), + ) + connection.execute( + "INSERT INTO sections VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + section.section_id, + section.path, + section.anchor, + section.level, + section.ordinal, + section.title, + json.dumps(section.heading_path, ensure_ascii=False), + json.dumps(section.heading_anchors, ensure_ascii=False), + section.line_start, + section.char_start, + section.char_end, + section.content_hash, + section.agent_summary, + int(section.is_structural), + int(section.is_content), + section.direct_unit_count, + section.descendant_unit_count, + ), + ) + + +def _validate_schema(connection: sqlite3.Connection, db_path: Path) -> None: + try: + row = connection.execute("SELECT value FROM meta WHERE key='schema_version'").fetchone() + except sqlite3.Error as exc: + raise RetrievalIndexError( + "invalid-index-schema", + "SQLite file is not a GroundMap retrieval index", + {"db_path": str(db_path), "sqlite_error": str(exc)}, + ) from exc + if row is None or str(row[0]) != str(SCHEMA_VERSION): + raise RetrievalIndexError( + "unsupported-index-version", + "Retrieval index schema version is missing or unsupported", + { + "db_path": str(db_path), + "expected": SCHEMA_VERSION, + "actual": row[0] if row else None, + }, + ) + expected_columns = [ + "unit_id", "text", "heading_path", "section_summary", "path" + ] + for table, expected_ddl in _FTS_DDL.items(): + schema_row = connection.execute( + "SELECT sql FROM sqlite_master WHERE type='table' AND name=?", + (table,), + ).fetchone() + actual_ddl = str(schema_row[0]) if schema_row and schema_row[0] else "" + columns = [ + str(column[1]) + for column in connection.execute(f"PRAGMA table_info('{table}')") + ] + if ( + _normalize_schema_sql(actual_ddl) + != _normalize_schema_sql(expected_ddl) + or columns != expected_columns + ): + raise RetrievalIndexError( + "invalid-index-schema", + "FTS table schema or tokenizer does not match the retrieval contract", + { + "db_path": str(db_path), + "table": table, + "expected_columns": expected_columns, + "actual_columns": columns, + }, + ) + + +def _corpus_freshness( + connection: sqlite3.Connection, + db_path: Path, +) -> dict[str, Any]: + """Compare the indexed document manifest with the current raw corpus. + + Coverage inside SQLite is insufficient on its own: an internally complete + index may still be stale after a raw file is added, removed, rewritten, or + after a validated outline summary used for routing is changed. Both + fingerprints are content-based (not mtime-based), so same-length edits and + ``annotate-section`` updates cannot leave silent stale routes. + """ + + row = connection.execute( + "SELECT value FROM meta WHERE key='workspace_root'" + ).fetchone() + if row is None or not str(row[0]).strip(): + return { + "ok": False, + "workspace_root": None, + "added": [], + "removed": [], + "changed": [], + "binding_changed": [], + "conversion_source_issues": [], + "errors": [{"code": "workspace-root-missing", "db_path": str(db_path)}], + } + + workspace = _as_path(str(row[0]), label="indexed_workspace_root") + raw_root = workspace / "raw" + errors: list[dict[str, Any]] = [] + current: dict[str, dict[str, Any]] = {} + conversion_source_issues: list[dict[str, Any]] = [] + if not workspace.is_dir() or not raw_root.is_dir(): + errors.append({ + "code": "indexed-workspace-unavailable", + "workspace_root": str(workspace), + "raw_root": str(raw_root), + }) + else: + try: + candidates = sorted(raw_root.rglob("*.md"), key=lambda value: value.as_posix()) + except OSError as exc: + candidates = [] + errors.append({ + "code": "raw-discovery-failed", + "raw_root": str(raw_root), + "exception": type(exc).__name__, + }) + for candidate in candidates: + try: + resolved = candidate.resolve(strict=True) + if not _is_relative_to(resolved, raw_root.resolve()): + errors.append({ + "code": "document-path-escape", + "path": candidate.as_posix(), + "resolved_path": resolved.as_posix(), + }) + continue + text = resolved.read_text(encoding="utf-8") + relative_path = resolved.relative_to(workspace).as_posix() + summaries, _warnings, outline = _load_outline_summaries( + resolved, text, relative_path + ) + markdown_hash = _sha256(text) + try: + binding = validate_conversion_binding( + resolved, + outline, + workspace, + markdown_sha256=markdown_hash, + expected_outline_schema_version=OUTLINE_SCHEMA_VERSION, + ) + binding_values = { + "document_origin": binding.document_origin, + "original_source_path": binding.original_source_path, + "original_source_sha256": binding.original_source_sha256, + "converter_fingerprint": binding.converter_fingerprint, + "conversion_receipt_hash": binding.conversion_receipt_hash, + } + except ConversionReceiptError as exc: + issue = { + "code": "conversion-binding-invalid", + "path": relative_path, + "issue": exc.to_dict(), + } + errors.append(issue) + conversion_source_issues.append(issue) + binding_values = { + "document_origin": "invalid", + "original_source_path": None, + "original_source_sha256": None, + "converter_fingerprint": None, + "conversion_receipt_hash": None, + } + current[relative_path] = { + "source_hash": markdown_hash, + "outline_summary_hash": _outline_summary_fingerprint(summaries), + **binding_values, + } + except (OSError, UnicodeError, ValueError) as exc: + errors.append({ + "code": "document-freshness-read-failed", + "path": candidate.as_posix(), + "exception": type(exc).__name__, + }) + + stored = { + str(document["path"]): { + "source_hash": str(document["source_hash"]), + "outline_summary_hash": str(document["outline_summary_hash"]), + "document_origin": document["document_origin"], + "original_source_path": document["original_source_path"], + "original_source_sha256": document["original_source_sha256"], + "converter_fingerprint": document["converter_fingerprint"], + "conversion_receipt_hash": document["conversion_receipt_hash"], + } + for document in connection.execute( + """SELECT path, source_hash, outline_summary_hash, document_origin, + original_source_path, original_source_sha256, + converter_fingerprint, conversion_receipt_hash + FROM documents""" + ) + } + added = sorted(set(current) - set(stored)) + removed = sorted(set(stored) - set(current)) + markdown_changed = sorted( + path for path in set(current) & set(stored) + if current[path]["source_hash"] != stored[path]["source_hash"] + ) + outline_summary_changed = sorted( + path for path in set(current) & set(stored) + if current[path]["outline_summary_hash"] + != stored[path]["outline_summary_hash"] + ) + binding_fields = ( + "document_origin", + "original_source_path", + "original_source_sha256", + "converter_fingerprint", + "conversion_receipt_hash", + ) + binding_changed = sorted( + path for path in set(current) & set(stored) + if any(current[path][field] != stored[path][field] for field in binding_fields) + ) + changed = sorted( + set(markdown_changed) | set(outline_summary_changed) | set(binding_changed) + ) + return { + "ok": not errors and not added and not removed and not changed, + "workspace_root": str(workspace), + "documents_current": len(current), + "documents_indexed": len(stored), + "added": added, + "removed": removed, + "changed": changed, + "markdown_changed": markdown_changed, + "outline_summary_changed": outline_summary_changed, + "binding_changed": binding_changed, + "conversion_source_issues": conversion_source_issues, + "errors": errors, + } + + +def _exact_multiset_fingerprint(rows: Iterable[sqlite3.Row]) -> str: + """Position-invariant fingerprint of every ``(path, kind, content_hash)``. + + Sorting preserves multiplicity while making document/block ordering + irrelevant. Length prefixes avoid delimiter ambiguity and make the + algorithm straightforward for an independent Gold implementation. + """ + + records = sorted( + json.dumps( + [str(row["path"]), str(row["kind"]), str(row["content_hash"])], + ensure_ascii=False, + separators=(",", ":"), + ).encode("utf-8") + for row in rows + ) + digest = hashlib.sha256() + for record in records: + digest.update(len(record).to_bytes(8, "big")) + digest.update(record) + return digest.hexdigest() + + +def _kind_counts(rows: Iterable[sqlite3.Row]) -> dict[str, int]: + counts: dict[str, int] = {} + for row in rows: + kind = str(row["kind"]) + counts[kind] = counts.get(kind, 0) + 1 + return { + kind: counts.get(kind, 0) + for kind in sorted(set(NATURAL_KINDS) | set(counts)) + } + + +def _coverage_from_connection(connection: sqlite3.Connection, db_path: Path) -> dict[str, Any]: + expected_multiset_rows = list(connection.execute( + "SELECT path, kind, content_hash FROM expected_units" + )) + indexed_multiset_rows = list(connection.execute( + "SELECT path, kind, content_hash FROM units" + )) + expected_fingerprint = _exact_multiset_fingerprint(expected_multiset_rows) + indexed_fingerprint = _exact_multiset_fingerprint(indexed_multiset_rows) + expected_kind_counts = _kind_counts(expected_multiset_rows) + indexed_kind_counts = _kind_counts(indexed_multiset_rows) + multiset_matches = ( + expected_fingerprint == indexed_fingerprint + and expected_kind_counts == indexed_kind_counts + and len(expected_multiset_rows) == len(indexed_multiset_rows) + ) + + # ``expected_*`` tables are an independent row-level ledger, while the + # document manifest freezes their per-document denominators. Comparing + # both prevents a synchronized deletion from quietly redefining 1/2 as + # 1/1 and reporting a false 100% coverage result. + documents = [ + dict(row) + for row in connection.execute("SELECT * FROM documents ORDER BY path") + ] + + def count_by_path(table: str, where: str = "") -> dict[str, int]: + clause = f" WHERE {where}" if where else "" + return { + str(row["path"]): int(row["row_count"]) + for row in connection.execute( + f"SELECT path, COUNT(*) AS row_count FROM {table}{clause} GROUP BY path" + ) + } + + manifest_counts = { + "expected_units": count_by_path("expected_units"), + "indexed_units": count_by_path("units"), + "expected_structural_sections": count_by_path( + "expected_structural_sections" + ), + "registered_structural_sections": count_by_path( + "sections", "is_structural=1" + ), + "expected_content_sections": count_by_path("expected_sections"), + "registered_content_sections": count_by_path( + "sections", "is_content=1" + ), + } + + ledger_records: dict[str, list[list[Any]]] = {} + + def add_ledger_record(path: Any, record: list[Any]) -> None: + ledger_records.setdefault(str(path), []).append(record) + + for row in connection.execute( + """SELECT unit_id, path, anchor, kind, parent_kind, ordinal, + subordinal, owning_section_anchor, heading_path_json, + heading_anchors_json, section_summary, line_start, line_end, + char_start, char_end, normalized_text, content_hash, + exact_text_hash + FROM expected_units""" + ): + add_ledger_record(row["path"], [ + "unit", + row["unit_id"], + row["path"], + row["anchor"], + row["kind"], + row["parent_kind"], + row["ordinal"], + row["subordinal"], + row["owning_section_anchor"], + row["heading_path_json"], + row["heading_anchors_json"], + row["section_summary"], + row["line_start"], + row["line_end"], + row["char_start"], + row["char_end"], + row["normalized_text"], + row["content_hash"], + row["exact_text_hash"], + ]) + for row in connection.execute( + """SELECT section_id, path, anchor, level, title, content_hash + FROM expected_structural_sections""" + ): + add_ledger_record(row["path"], [ + "structural-section", + row["section_id"], + row["path"], + row["anchor"], + row["level"], + row["title"], + row["content_hash"], + ]) + for row in connection.execute( + "SELECT section_id, path, anchor, title, content_hash FROM expected_sections" + ): + add_ledger_record(row["path"], [ + "content-section", + row["section_id"], + row["path"], + row["anchor"], + row["title"], + row["content_hash"], + ]) + ledger_hashes = { + path: _inventory_fingerprint(records) + for path, records in ledger_records.items() + } + empty_inventory_hash = _inventory_fingerprint([]) + manifest_issues: list[dict[str, Any]] = [] + document_paths = {str(document["path"]) for document in documents} + for document in documents: + path = str(document["path"]) + observed_inventory_hash = ledger_hashes.get(path, empty_inventory_hash) + if str(document["inventory_hash"]) != observed_inventory_hash: + manifest_issues.append({ + "path": path, + "field": "inventory_hash", + "declared": str(document["inventory_hash"]), + "observed": observed_inventory_hash, + "issue": "manifest-fingerprint-mismatch", + }) + for field, observed_by_path in manifest_counts.items(): + declared = int(document[field]) + observed = observed_by_path.get(path, 0) + if declared != observed: + manifest_issues.append({ + "path": path, + "field": field, + "declared": declared, + "observed": observed, + "issue": "manifest-count-mismatch", + }) + for field, observed_by_path in manifest_counts.items(): + for path, observed in sorted(observed_by_path.items()): + if path not in document_paths: + manifest_issues.append({ + "path": path, + "field": field, + "declared": None, + "observed": observed, + "issue": "manifest-document-missing", + }) + + declared_expected_units = sum(int(row["expected_units"]) for row in documents) + declared_expected_structural_sections = sum( + int(row["expected_structural_sections"]) for row in documents + ) + declared_expected_content_sections = sum( + int(row["expected_content_sections"]) for row in documents + ) + + # FTS5 ``UNINDEXED`` metadata columns have no lookup index. Joining each + # expected unit to both virtual tables makes a full integrity check + # quadratic on long documents. Scan each FTS materialization once and + # reconcile it in memory by unit_id instead. + def fts_rows_by_unit(table: str) -> dict[str, list[sqlite3.Row]]: + rows_by_unit: dict[str, list[sqlite3.Row]] = {} + for fts_row in connection.execute( + f"SELECT unit_id, text, heading_path, section_summary, path FROM {table}" + ): + rows_by_unit.setdefault(str(fts_row["unit_id"]), []).append(fts_row) + return rows_by_unit + + unicode_rows_by_unit = fts_rows_by_unit("unit_fts_unicode") + trigram_rows_by_unit = fts_rows_by_unit("unit_fts_trigram") + fts_integrity: dict[str, dict[str, Any]] = {} + for table in ("unit_fts_unicode", "unit_fts_trigram"): + try: + findings = [ + str(row[0]) + for row in connection.execute(f"PRAGMA quick_check('{table}')") + ] + fts_integrity[table] = { + "ok": findings == ["ok"], + "findings": findings, + } + except sqlite3.Error as exc: + fts_integrity[table] = { + "ok": False, + "findings": [f"sqlite-error:{type(exc).__name__}"], + } + unit_rows = list(connection.execute( + """SELECT e.unit_id, e.path, e.anchor, e.kind, + e.parent_kind AS expected_parent_kind, + e.ordinal AS expected_ordinal, + e.subordinal AS expected_subordinal, + e.owning_section_anchor AS expected_owning_section_anchor, + e.heading_path_json AS expected_heading_path_json, + e.heading_anchors_json AS expected_heading_anchors_json, + e.section_summary AS expected_section_summary, + e.line_start, e.line_end AS expected_line_end, + e.char_start, e.char_end AS expected_char_end, + e.normalized_text AS expected_normalized_text, + e.content_hash, e.exact_text_hash, + u.unit_id AS materialized_id, u.path AS unit_path, + u.anchor AS unit_anchor, u.kind AS unit_kind, + u.parent_kind AS unit_parent_kind, + u.ordinal AS unit_ordinal, + u.subordinal AS unit_subordinal, + u.owning_section_anchor AS unit_owning_section_anchor, + u.line_start AS unit_line_start, + u.line_end AS unit_line_end, + u.char_start AS unit_char_start, + u.char_end AS unit_char_end, + u.content_hash AS unit_content_hash, + u.exact_text_hash AS unit_exact_text_hash, u.text AS unit_text, + u.normalized_text, + u.heading_path_json, u.heading_anchors_json, u.section_summary + FROM expected_units e + LEFT JOIN units u ON u.unit_id=e.unit_id + ORDER BY e.path, e.line_start, e.unit_id""" + )) + expected_unit_ids = {str(row["unit_id"]) for row in unit_rows} + unexpected_fts_rows = [ + {"channel": channel, "unit_id": unit_id, "row_count": len(rows)} + for channel, rows_by_unit in ( + ("unicode61", unicode_rows_by_unit), + ("trigram", trigram_rows_by_unit), + ) + for unit_id, rows in rows_by_unit.items() + if unit_id not in expected_unit_ids + ] + missing_units: list[dict[str, Any]] = [] + materialized_units = 0 + unicode_units = 0 + trigram_units = 0 + indexed_by_path: dict[str, int] = {} + for row in unit_rows: + missing_components: list[str] = [] + if row["materialized_id"] is not None and row["unit_text"] is not None: + unit_text = str(row["unit_text"]) + recomputed_content_hash = _sha256(_normalize_text(unit_text)) + recomputed_exact_text_hash = _exact_text_hash(unit_text) + recomputed_searchable_text = _searchable_text(unit_text) + else: + recomputed_content_hash = None + recomputed_exact_text_hash = None + recomputed_searchable_text = None + unit_valid = row["materialized_id"] is not None and ( + row["unit_path"] == row["path"] + and row["unit_anchor"] == row["anchor"] + and row["unit_kind"] == row["kind"] + and row["unit_parent_kind"] == row["expected_parent_kind"] + and row["unit_ordinal"] == row["expected_ordinal"] + and row["unit_subordinal"] == row["expected_subordinal"] + and row["unit_owning_section_anchor"] + == row["expected_owning_section_anchor"] + and row["heading_path_json"] == row["expected_heading_path_json"] + and row["heading_anchors_json"] == row["expected_heading_anchors_json"] + and row["section_summary"] == row["expected_section_summary"] + and row["unit_line_start"] == row["line_start"] + and row["unit_line_end"] == row["expected_line_end"] + and row["unit_char_start"] == row["char_start"] + and row["unit_char_end"] == row["expected_char_end"] + and row["unit_content_hash"] == row["content_hash"] + and recomputed_content_hash == row["content_hash"] + and row["unit_exact_text_hash"] == row["exact_text_hash"] + and recomputed_exact_text_hash == row["exact_text_hash"] + and row["normalized_text"] == row["expected_normalized_text"] + and row["normalized_text"] == recomputed_searchable_text + ) + if not unit_valid: + missing_components.append( + "unit" if row["materialized_id"] is None else "unit-metadata-or-content" + ) + else: + materialized_units += 1 + heading_path = " / ".join(json.loads(row["heading_path_json"])) + section_summary = row["section_summary"] or "" + unicode_rows = unicode_rows_by_unit.get(str(row["unit_id"]), []) + trigram_rows = trigram_rows_by_unit.get(str(row["unit_id"]), []) + unicode_valid = len(unicode_rows) == 1 and ( + unicode_rows[0]["text"] == row["normalized_text"] + and unicode_rows[0]["heading_path"] == heading_path + and unicode_rows[0]["section_summary"] == section_summary + and unicode_rows[0]["path"] == row["path"] + ) + trigram_valid = len(trigram_rows) == 1 and ( + trigram_rows[0]["text"] == row["normalized_text"] + and trigram_rows[0]["heading_path"] == heading_path + and trigram_rows[0]["section_summary"] == section_summary + and trigram_rows[0]["path"] == row["path"] + ) + if unicode_valid: + unicode_units += 1 + else: + if len(unicode_rows) == 0: + missing_components.append("unicode61") + elif len(unicode_rows) > 1: + missing_components.append("unicode61-duplicate") + else: + missing_components.append("unicode61-stale") + if trigram_valid: + trigram_units += 1 + else: + if len(trigram_rows) == 0: + missing_components.append("trigram") + elif len(trigram_rows) > 1: + missing_components.append("trigram-duplicate") + else: + missing_components.append("trigram-stale") + if missing_components: + missing_units.append({ + "unit_id": row["unit_id"], + "path": row["path"], + "anchor": row["anchor"], + "kind": row["kind"], + "line_start": row["line_start"], + "char_start": row["char_start"], + "content_hash": row["content_hash"], + "exact_text_hash": row["exact_text_hash"], + "missing_components": missing_components, + }) + else: + indexed_by_path[row["path"]] = indexed_by_path.get(row["path"], 0) + 1 + + inventory_expected_units = len(unit_rows) + valid_indexed_units = inventory_expected_units - len(missing_units) + expected_units = max(declared_expected_units, inventory_expected_units) + indexed_units = min(valid_indexed_units, expected_units) + + structural_rows = list(connection.execute( + """SELECT e.section_id, e.path, e.anchor, e.level, e.title, e.content_hash, + s.section_id AS registered_id, s.path AS section_path, + s.anchor AS section_anchor, s.level AS section_level, + s.title AS section_title, s.content_hash AS section_content_hash, + s.is_structural + FROM expected_structural_sections e + LEFT JOIN sections s USING(section_id) + ORDER BY e.path, e.anchor""" + )) + missing_structural_sections: list[dict[str, Any]] = [] + registered_structural_by_path: dict[str, int] = {} + for row in structural_rows: + valid = row["registered_id"] is not None and ( + row["section_path"] == row["path"] + and row["section_anchor"] == row["anchor"] + and row["section_level"] == row["level"] + and row["section_title"] == row["title"] + and row["section_content_hash"] == row["content_hash"] + and row["is_structural"] == 1 + ) + if valid: + registered_structural_by_path[row["path"]] = ( + registered_structural_by_path.get(row["path"], 0) + 1 + ) + else: + missing_structural_sections.append({ + "section_id": row["section_id"], + "path": row["path"], + "anchor": row["anchor"], + "level": row["level"], + "title": row["title"], + "content_hash": row["content_hash"], + "issue": ( + "registration-missing" + if row["registered_id"] is None + else "registration-stale" + ), + }) + inventory_expected_structural_sections = len(structural_rows) + valid_registered_structural_sections = ( + inventory_expected_structural_sections - len(missing_structural_sections) + ) + expected_structural_sections = max( + declared_expected_structural_sections, + inventory_expected_structural_sections, + ) + registered_structural_sections = min( + valid_registered_structural_sections, + expected_structural_sections, + ) + unexpected_structural_sections = [ + dict(row) + for row in connection.execute( + """SELECT s.section_id, s.path, s.anchor, s.level, s.title, + s.content_hash + FROM sections s + LEFT JOIN expected_structural_sections e USING(section_id) + WHERE s.is_structural=1 AND e.section_id IS NULL + ORDER BY s.path, s.ordinal""" + ) + ] + unexpected_empty_sections = [ + dict(row) + for row in connection.execute( + """SELECT section_id, path, anchor, level, title, content_hash, + direct_unit_count, descendant_unit_count + FROM sections + WHERE is_structural=1 AND descendant_unit_count=0 + ORDER BY path, ordinal""" + ) + ] + + section_rows = list(connection.execute( + """SELECT e.section_id, e.path, e.anchor, e.title, e.content_hash, + s.section_id AS registered_id, s.path AS section_path, + s.anchor AS section_anchor, s.title AS section_title, + s.content_hash AS section_content_hash, s.is_content + FROM expected_sections e LEFT JOIN sections s USING(section_id) + ORDER BY e.path, e.anchor""" + )) + missing_sections: list[dict[str, Any]] = [] + registered_by_path: dict[str, int] = {} + for row in section_rows: + valid = row["registered_id"] is not None and ( + row["section_path"] == row["path"] + and row["section_anchor"] == row["anchor"] + and row["section_title"] == row["title"] + and row["section_content_hash"] == row["content_hash"] + and row["is_content"] == 1 + ) + if valid: + registered_by_path[row["path"]] = registered_by_path.get(row["path"], 0) + 1 + else: + missing_sections.append({ + "section_id": row["section_id"], + "path": row["path"], + "anchor": row["anchor"], + "title": row["title"], + "content_hash": row["content_hash"], + "issue": "registration-missing" if row["registered_id"] is None else "registration-stale", + }) + inventory_expected_sections = len(section_rows) + valid_registered_sections = inventory_expected_sections - len(missing_sections) + expected_sections = max( + declared_expected_content_sections, + inventory_expected_sections, + ) + registered_sections = min(valid_registered_sections, expected_sections) + + def metric(expected: int, actual: int, missing: list[dict[str, Any]], label: str) -> dict[str, Any]: + coverage = (actual / expected) if expected else None + return { + "expected": expected, + label: actual, + "missing": expected - actual, + "missing_items": missing, + "coverage": coverage, + "coverage_pct": (coverage * 100.0) if coverage is not None else None, + } + + for document in documents: + document["declared_indexed_units"] = document["indexed_units"] + document["declared_registered_structural_sections"] = document[ + "registered_structural_sections" + ] + document["declared_registered_content_sections"] = document[ + "registered_content_sections" + ] + document["observed_expected_units"] = manifest_counts["expected_units"].get( + document["path"], 0 + ) + document["observed_indexed_units"] = manifest_counts["indexed_units"].get( + document["path"], 0 + ) + document["observed_expected_structural_sections"] = manifest_counts[ + "expected_structural_sections" + ].get(document["path"], 0) + document["observed_registered_structural_sections"] = manifest_counts[ + "registered_structural_sections" + ].get(document["path"], 0) + document["observed_expected_content_sections"] = manifest_counts[ + "expected_content_sections" + ].get(document["path"], 0) + document["observed_registered_content_sections"] = manifest_counts[ + "registered_content_sections" + ].get(document["path"], 0) + document["observed_inventory_hash"] = ledger_hashes.get( + document["path"], empty_inventory_hash + ) + document["indexed_units"] = indexed_by_path.get(document["path"], 0) + document["registered_structural_sections"] = registered_structural_by_path.get( + document["path"], 0 + ) + document["registered_content_sections"] = registered_by_path.get(document["path"], 0) + empty_corpus = expected_units == 0 and expected_structural_sections == 0 + corpus_freshness = _corpus_freshness(connection, db_path) + structurally_complete = ( + not missing_units + and not unexpected_fts_rows + and multiset_matches + and not missing_structural_sections + and not unexpected_structural_sections + and not missing_sections + and not manifest_issues + and all(row.get("ok") for row in fts_integrity.values()) + ) + if not corpus_freshness["ok"]: + coverage_status = "stale-corpus" + elif empty_corpus: + coverage_status = "empty-corpus" + elif structurally_complete: + coverage_status = "complete" + else: + coverage_status = "incomplete" + return { + # With no denominator there is no evidence that either 100% coverage + # target was achieved. Report fail-closed instead of vacuous success. + "ok": not empty_corpus and structurally_complete and corpus_freshness["ok"], + "schema_version": SCHEMA_VERSION, + "db_path": str(db_path), + "empty_corpus": empty_corpus, + "coverage_status": coverage_status, + "corpus_freshness": corpus_freshness, + "documents": documents, + "documents_count": len(documents), + "manifest": { + "ok": not manifest_issues, + "issues_count": len(manifest_issues), + "issues": manifest_issues, + }, + "fts_integrity": fts_integrity, + "natural_units": { + **metric(expected_units, indexed_units, missing_units, "indexed"), + "materialized": materialized_units, + "unicode61_indexed": unicode_units, + "trigram_indexed": trigram_units, + "unexpected_fts_rows_count": len(unexpected_fts_rows), + "unexpected_fts_rows": unexpected_fts_rows, + "expected_kind_counts": expected_kind_counts, + "indexed_kind_counts": indexed_kind_counts, + "exact_multiset_fingerprint": { + "algorithm": "sha256-lenprefixed-json-v1", + "fields": ["path", "kind", "content_hash"], + "expected": expected_fingerprint, + "indexed": indexed_fingerprint, + "match": multiset_matches, + }, + }, + "structural_sections": { + **metric( + expected_structural_sections, + registered_structural_sections, + missing_structural_sections, + "registered", + ), + "unexpected": len(unexpected_structural_sections), + "unexpected_items": unexpected_structural_sections, + }, + "content_sections": metric( + expected_sections, registered_sections, missing_sections, "registered" + ), + "unexpected_empty_sections_count": len(unexpected_empty_sections), + "unexpected_empty_sections": unexpected_empty_sections, + } + + +def rebuild_index( + workspace_root: str | os.PathLike[str], + db_path: str | os.PathLike[str] | None = None, +) -> dict[str, Any]: + """Atomically rebuild a workspace's disposable retrieval index. + + Any unreadable/malformed document fails the whole rebuild with a + :class:`RetrievalIndexError`; the previous database remains untouched. + """ + + workspace, raw_root, target = _resolve_paths(workspace_root, db_path) + try: + candidates = sorted(raw_root.rglob("*.md"), key=lambda value: value.as_posix()) + except OSError as exc: + raise RetrievalIndexError( + "raw-discovery-failed", + "Could not enumerate raw Markdown documents", + {"raw_root": str(raw_root), "exception": type(exc).__name__}, + ) from exc + + errors: list[dict[str, Any]] = [] + inventories: list[ + tuple[ + str, + str, + str, + int, + ConversionBinding, + list[_Unit], + list[_Section], + ] + ] = [] + warnings: list[dict[str, Any]] = [] + for candidate in candidates: + try: + resolved = candidate.resolve(strict=True) + except OSError as exc: + errors.append({ + "code": "document-resolution-failed", + "path": candidate.as_posix(), + "exception": type(exc).__name__, + }) + continue + if not _is_relative_to(resolved, raw_root): + errors.append({ + "code": "document-path-escape", + "path": candidate.as_posix(), + "resolved_path": resolved.as_posix(), + }) + continue + try: + ( + path, + source_hash, + outline_summary_hash, + char_count, + conversion_binding, + units, + sections, + doc_warnings, + ) = _inventory_document(resolved, workspace) + inventories.append( + ( + path, + source_hash, + outline_summary_hash, + char_count, + conversion_binding, + units, + sections, + ) + ) + warnings.extend(doc_warnings) + except RetrievalIndexError as exc: + errors.append({ + "path": candidate.relative_to(workspace).as_posix(), + "code": exc.code, + "message": exc.message, + "details": exc.details, + }) + except Exception as exc: # fail closed; never silently skip a document + errors.append({ + "path": candidate.relative_to(workspace).as_posix(), + "code": "unexpected-document-error", + "message": str(exc), + "exception": type(exc).__name__, + }) + if errors: + raise RetrievalIndexError( + "rebuild-document-errors", + f"Retrieval index rebuild rejected {len(errors)} document error(s)", + {"errors": errors, "documents_discovered": len(candidates)}, + ) + + try: + target.parent.mkdir(parents=True, exist_ok=True) + fd, temporary_name = tempfile.mkstemp( + dir=str(target.parent), prefix=f".{target.name}.", suffix=".tmp" + ) + os.close(fd) + except OSError as exc: + raise RetrievalIndexError( + "rebuild-storage-failed", + "Could not create a temporary retrieval index", + { + "db_path": str(target), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + temporary = Path(temporary_name) + connection: sqlite3.Connection | None = None + try: + connection = sqlite3.connect(str(temporary)) + connection.row_factory = sqlite3.Row + connection.execute("PRAGMA journal_mode=DELETE") + connection.execute("PRAGMA synchronous=FULL") + _create_schema(connection) + connection.execute("INSERT INTO meta VALUES ('schema_version', ?)", (str(SCHEMA_VERSION),)) + connection.execute("INSERT INTO meta VALUES ('workspace_root', ?)", (str(workspace),)) + connection.execute("INSERT INTO meta VALUES ('documents_discovered', ?)", (str(len(candidates)),)) + for ( + path, + source_hash, + outline_summary_hash, + char_count, + conversion_binding, + units, + sections, + ) in inventories: + _insert_inventory( + connection, + path, + source_hash, + outline_summary_hash, + char_count, + conversion_binding, + units, + sections, + ) + connection.commit() + report = _coverage_from_connection(connection, target) + if report["coverage_status"] not in {"complete", "empty-corpus"}: + raise RetrievalIndexError( + "rebuild-coverage-incomplete", + "The materialized index is incomplete or the raw corpus changed during rebuild", + { + "coverage_status": report["coverage_status"], + "natural_units": report["natural_units"], + "structural_sections": report["structural_sections"], + "content_sections": report["content_sections"], + "corpus_freshness": report["corpus_freshness"], + }, + ) + report.update({ + "rebuilt": True, + "warnings": warnings, + "errors": [], + "channels": ["unicode61_bm25", "trigram_bm25", "exact_substring", "rrf"], + }) + connection.close() + connection = None + os.replace(temporary, target) + return report + except RetrievalIndexError: + if connection is not None: + connection.close() + temporary.unlink(missing_ok=True) + raise + except (OSError, sqlite3.Error) as exc: + if connection is not None: + connection.close() + temporary.unlink(missing_ok=True) + raise RetrievalIndexError( + "rebuild-storage-failed", + "Could not materialize the SQLite retrieval index", + { + "db_path": str(target), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + except Exception as exc: + if connection is not None: + connection.close() + temporary.unlink(missing_ok=True) + raise RetrievalIndexError( + "unexpected-rebuild-error", + "Unexpected failure while rebuilding the retrieval index", + { + "db_path": str(target), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + + +def coverage_report(db_path: str | os.PathLike[str]) -> dict[str, Any]: + """Return exact inventory-vs-index coverage, including every missing ID.""" + + target = _as_path(db_path, label="db_path") + if not target.is_file(): + raise RetrievalIndexError( + "index-not-found", "Retrieval index does not exist", {"db_path": str(target)} + ) + connection: sqlite3.Connection | None = None + try: + connection = sqlite3.connect(str(target)) + connection.row_factory = sqlite3.Row + _validate_schema(connection, target) + report = _coverage_from_connection(connection, target) + return report + except RetrievalIndexError: + raise + except sqlite3.Error as exc: + raise RetrievalIndexError( + "coverage-read-failed", + "Could not read retrieval coverage", + {"db_path": str(target), "sqlite_error": str(exc)}, + ) from exc + except Exception as exc: + raise RetrievalIndexError( + "coverage-read-failed", + "Retrieval coverage data is malformed", + { + "db_path": str(target), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + finally: + if connection is not None: + connection.close() + + +def _require_complete_current_index( + connection: sqlite3.Connection, + db_path: Path, + *, + operation: str, +) -> dict[str, Any]: + """Fail closed unless raw freshness and every derived component are current.""" + + try: + report = _coverage_from_connection(connection, db_path) + except RetrievalIndexError: + raise + except Exception as exc: + raise RetrievalIndexError( + "index-integrity-error", + f"The evidence index could not be verified before {operation}", + { + "db_path": str(db_path), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + + freshness = report["corpus_freshness"] + if not freshness["ok"]: + raise RetrievalIndexError( + "index-stale", + f"The raw corpus changed after the evidence index was built; rebuild it before {operation}", + freshness, + ) + if not report["ok"]: + natural = report.get("natural_units", {}) + structural = report.get("structural_sections", {}) + content = report.get("content_sections", {}) + raise RetrievalIndexError( + "index-integrity-error", + f"The evidence index is internally incomplete or inconsistent; rebuild it before {operation}", + { + "db_path": str(db_path), + "coverage_status": report.get("coverage_status"), + "natural_units": { + "expected": natural.get("expected"), + "indexed": natural.get("indexed"), + "missing": natural.get("missing"), + "missing_items": list(natural.get("missing_items", []))[:20], + "unexpected_fts_rows_count": natural.get( + "unexpected_fts_rows_count" + ), + "unexpected_fts_rows": list( + natural.get("unexpected_fts_rows", []) + )[:20], + }, + "structural_sections": { + "expected": structural.get("expected"), + "registered": structural.get("registered"), + "missing": structural.get("missing"), + "unexpected": structural.get("unexpected"), + }, + "content_sections": { + "expected": content.get("expected"), + "registered": content.get("registered"), + "missing": content.get("missing"), + }, + }, + ) + return freshness + + +def _fts_quote(value: str) -> str: + return '"' + value.replace('"', '""') + '"' + + +def _unicode_query(value: str) -> str | None: + terms = _meaningful_query_runs(value) + if not terms: + return None + return " OR ".join(_fts_quote(term) for term in terms) + + +def _meaningful_query_runs(value: str) -> list[str]: + runs = [term for term in _WORD_RE.findall(value) if term] + filtered = [ + term for term in runs + if re.search(r"[\u3400-\u9fff]", term) + or term.casefold() not in _FTS_STOPWORDS + ] + # A stopword-only query is unusual but still valid; do not turn it into an + # empty protocol error. The fallback preserves deterministic behaviour. + return filtered or runs + + +def _trigram_query(value: str) -> str | None: + # OR-ing deterministic character trigrams preserves the exact substring + # channel while also tolerating modest lexical rewrites/order changes. + # BM25 naturally rewards candidates matching several query trigrams. + grams = _trigram_terms(value) + if not grams: + return None + return " OR ".join(_fts_quote(gram) for gram in grams) + + +def _trigram_terms(value: str) -> list[str]: + grams: list[str] = [] + for run in _meaningful_query_runs(value): + if len(run) < 3: + continue + for index in range(len(run) - 2): + gram = run[index : index + 3] + if gram not in grams: + grams.append(gram) + return grams + + +def _fetch_with_routing_fanout( + connection: sqlite3.Connection, + sql: str, + params: Sequence[Any], + candidate_limit: int, + *, + per_section: int = 2, +) -> tuple[list[sqlite3.Row], int, int]: + """Collapse replicated section routing in one SQL window pass. + + Heading/path/summary text is repeated on every natural unit. A Python + ``LIMIT/OFFSET`` loop therefore rescanned and resorted a 100k-unit section + many times merely to retain two representatives. SQLite window + partitioning ranks every match once, then keeps at most ``per_section`` + representatives without sacrificing later matching sections. + """ + + query = f""" + WITH matched AS ( + {sql} + ), ranked AS ( + SELECT matched.*, + ROW_NUMBER() OVER ( + PARTITION BY path, + COALESCE(owning_section_anchor, '') + ORDER BY native_score, path, ordinal + ) AS section_rank + FROM matched + ) + SELECT ranked.*, + (SELECT COUNT(*) FROM ranked) AS matched_total, + (SELECT COUNT(*) FROM ranked all_ranked + WHERE all_ranked.section_rank > ?) AS collapsed_total + FROM ranked + WHERE section_rank <= ? + ORDER BY native_score, path, ordinal + LIMIT ? + """ + rows = list(connection.execute( + query, (*params, per_section, per_section, candidate_limit) + )) + matched_total = int(rows[0]["matched_total"]) if rows else 0 + collapsed_total = int(rows[0]["collapsed_total"]) if rows else 0 + return rows, collapsed_total, matched_total + + +def _bounded_text_excerpt( + text: str, + limit: int | None, + needles: Sequence[str] = (), +) -> tuple[str, bool, int, int]: + """Return a raw, query-centred excerpt without inventing chunk identity.""" + + if limit is None or limit == 0 or len(text) <= limit: + return text, False, 0, len(text) + folded = text.casefold() + centre = 0 + for needle in needles: + if not needle: + continue + position = folded.find(needle.casefold()) + if position >= 0: + centre = position + break + start = max(0, centre - limit // 2) + end = min(len(text), start + limit) + start = max(0, end - limit) + return text[start:end], True, start, end + + +def _row_to_hit( + row: sqlite3.Row, + *, + text_limit: int | None = None, + preview_needles: Sequence[str] = (), +) -> dict[str, Any]: + heading_path = json.loads(row["heading_path_json"]) + heading_anchors = json.loads(row["heading_anchors_json"]) + full_text = str(row["text"]) + shown_text, truncated, excerpt_start, excerpt_end = _bounded_text_excerpt( + full_text, text_limit, preview_needles + ) + evidence_handle = { + "unit_id": row["unit_id"], + "path": row["path"], + "parent_anchor": row["anchor"], + "kind": row["kind"], + "subordinal": row["subordinal"], + "content_hash": row["content_hash"], + } + return { + "unit_id": row["unit_id"], + "path": row["path"], + "anchor": row["anchor"], + "canonical_ref": f"{row['path']}#^{row['anchor']}", + # Markdown block references still point at the canonical parent block + # (a whole list/table). Selection identity is the natural-unit handle + # below; callers must not collapse sibling rows/items by canonical_ref. + "citation_scope": "parent_block", + "selection_scope": "natural_unit", + "evidence_handle": evidence_handle, + "kind": row["kind"], + "parent_kind": row["parent_kind"], + "ordinal": row["ordinal"], + "subordinal": row["subordinal"], + "owning_section_anchor": row["owning_section_anchor"], + "heading_path": heading_path, + "heading_anchors": heading_anchors, + "line_start": row["line_start"], + "line_end": row["line_end"], + "char_start": row["char_start"], + "char_end": row["char_end"], + "content_hash": row["content_hash"], + "text": shown_text, + "text_chars": len(full_text), + "text_truncated": truncated, + "text_excerpt_start": excerpt_start, + "text_excerpt_end": excerpt_end, + } + + +def read_evidence_unit( + db_path: str | os.PathLike[str], + unit_id: str, + max_chars: int = DEFAULT_EVIDENCE_TEXT_LIMIT, +) -> dict[str, Any]: + """Read one exact natural unit by its content-addressed selection ID.""" + + target = _as_path(db_path, label="db_path") + if not target.is_file(): + raise RetrievalIndexError( + "index-not-found", "Retrieval index does not exist", {"db_path": str(target)} + ) + if not isinstance(unit_id, str) or not re.fullmatch(r"[0-9a-f]{32}", unit_id): + raise RetrievalIndexError( + "invalid-unit-id", "unit_id must be a 32-character lowercase hex string" + ) + if not isinstance(max_chars, int) or isinstance(max_chars, bool) or max_chars < 0: + raise RetrievalIndexError( + "invalid-max-chars", + "max_chars must be a non-negative integer; 0 explicitly disables the limit", + {"max_chars": max_chars}, + ) + connection: sqlite3.Connection | None = None + try: + connection = sqlite3.connect(str(target)) + connection.row_factory = sqlite3.Row + _validate_schema(connection, target) + _require_complete_current_index(connection, target, operation="reading") + row = connection.execute( + "SELECT * FROM units WHERE unit_id=?", (unit_id,) + ).fetchone() + if row is None: + raise RetrievalIndexError( + "unit-not-found", + "Natural evidence unit does not exist in the current index", + {"unit_id": unit_id, "db_path": str(target)}, + ) + text_chars = len(str(row["text"])) + if max_chars and text_chars > max_chars: + raise RetrievalIndexError( + "evidence-unit-too-large", + "The natural evidence unit exceeds the safe read limit", + { + "unit_id": unit_id, + "path": row["path"], + "anchor": row["anchor"], + "kind": row["kind"], + "canonical_ref": f"{row['path']}#^{row['anchor']}", + "chars": text_chars, + "max_chars": max_chars, + "explicit_unlimited": "read-evidence-unit --max-chars 0", + }, + ) + result = _row_to_hit(row) + result.update({ + "ok": True, + "schema_version": SCHEMA_VERSION, + "db_path": str(target), + "max_chars": max_chars, + }) + return result + except RetrievalIndexError: + raise + except sqlite3.Error as exc: + raise RetrievalIndexError( + "unit-read-failed", + "Could not read the natural evidence unit", + {"db_path": str(target), "sqlite_error": str(exc)}, + ) from exc + finally: + if connection is not None: + connection.close() + + +def _has_explicit_property_value_shape(query: str, evidence_text: str) -> bool: + """Recognize open-set ``property of subject`` numeric requests safely. + + A broad ``What is ...`` test is unsafe because a qualitative fact can + contain an incidental year. Here the query must expose a property slot, + and the candidate must place a numeric value directly after that same + property (``mass of 17``, ``precision | 87.5%``). This keeps the guard + open to unseen property names without treating arbitrary numbers as the + requested value. + """ + + if _RELATIONAL_QUERY_RE.search(query): + return False + properties: list[str] = [] + for pattern in (_VALUE_PROPERTY_QUERY_RE, _POSSESSIVE_PROPERTY_QUERY_RE): + match = pattern.search(query) + if match is not None: + property_name = _normalize_text(match.group("property")).casefold() + properties.append(property_name) + property_words = re.findall(r"[\w-]+", property_name, re.UNICODE) + if len(property_words) > 1: + # Modifiers such as "current", "declared" or "maximum" + # are often omitted in the evidence wording. The terminal + # property noun still has to be directly adjacent to a value. + properties.append(property_words[-1].casefold()) + telegraphic = _TELEGRAPHIC_VALUE_QUERY_RE.search(query) + if telegraphic is not None: + words = re.findall(r"[\w-]+", telegraphic.group("phrase"), re.UNICODE) + if words: + # The terminal noun is a useful open-set property candidate for + # terse forms such as "What is the Aster unit mass?". It still + # has to sit directly beside a numeric value in the evidence. + properties.append(words[-1].casefold()) + properties = list(dict.fromkeys(value for value in properties if value)) + if not properties: + return False + evidence = unicodedata.normalize("NFC", evidence_text).casefold() + for property_name in properties: + for occurrence in re.finditer( + rf"(? str: + """Return the sentence/semicolon clause that owns one exclusion match.""" + + left = 0 + right = len(text) + for boundary in _ATOMIC_CLAUSE_BOUNDARY_RE.finditer(text): + if boundary.group(0) == ".": + prefix = text[max(0, boundary.start() - 16):boundary.end()] + suffix = text[boundary.end():boundary.end() + 32] + if ( + re.search(r"\bapprox\.$", prefix, re.IGNORECASE) + and re.match( + r"\s*(?:[<>≤≥~≈]\s*)?(?:[+\-−±]\s*)?" + r"(?:[$€£¥]\s*)?(?:[+\-−±]\s*)?\d", + suffix, + ) + ): + # ``approx. 43`` is one value expression, not two sentences. + continue + if boundary.end() <= start: + left = boundary.end() + continue + if boundary.start() >= end: + right = boundary.start() + break + return text[left:right] + + +def _has_non_calendar_numeric(text: str) -> bool: + for match in re.finditer(r"(? tuple[float, list[str]]: + """Downrank explicit exclusions for a positive information request. + + This is not semantic entailment. It is a conservative lexical guard + against a common high-risk retrieval failure: a value for another subject + is followed by an explicit applicability/governance exclusion. Generic + contrasts such as ``Policy A, not Policy B`` are never penalized: they may + themselves be the correct answer to an open-ended role/status question. + Negative queries keep all candidates unpenalized. + """ + + if _QUERY_EXCLUSION_RE.search(query) or _BINARY_QUERY_RE.search(query): + return 1.0, [] + query_normalized = _normalize_text(query).casefold() + for match in _EXCLUDED_OBJECT_RE.finditer(evidence_text): + # A bare contrast can be the answer to a role/status/relationship + # question. Use it as a wrong-value guard only when the query itself + # clearly asks for a quantitative property. + if match.groupdict().get("not_object"): + local_clause = _atomic_clause_containing( + evidence_text, match.start(), match.end() + ) + quantitative_request = bool( + _QUANTITATIVE_QUERY_RE.search(query) + and _has_non_calendar_numeric(local_clause) + ) + generic_value_request = _has_explicit_property_value_shape( + query, local_clause, + ) + if not quantitative_request and not generic_value_request: + continue + excluded_object = next( + (value for value in match.groupdict().values() if value is not None), "" + ) + excluded_normalized = _normalize_text(excluded_object).casefold() + excluded_normalized = re.sub(r"^(?:the|a|an)\s+", "", excluded_normalized) + excluded_normalized = excluded_normalized.strip(" |*_`[]()") + # Require the excluded entity phrase itself in the query. Token + # overlap (for example the generic word "policy") would incorrectly + # penalize useful contrast evidence such as "Policy A, not Policy B". + if len(excluded_normalized) >= 3 and excluded_normalized in query_normalized: + return 0.2, ["queried-subject-explicitly-excluded"] + return 1.0, [] + + +def _fetch_candidate_rows( + connection: sqlite3.Connection, + unit_ids: Iterable[str], +) -> dict[str, sqlite3.Row]: + identifiers = list(unit_ids) + rows: dict[str, sqlite3.Row] = {} + # Stay below SQLite's common 999-variable build limit. + for start in range(0, len(identifiers), 500): + chunk = identifiers[start : start + 500] + placeholders = ",".join("?" for _ in chunk) + if not placeholders: + continue + for row in connection.execute( + f"SELECT * FROM units WHERE unit_id IN ({placeholders})", chunk + ): + rows[row["unit_id"]] = row + return rows + + +def search_evidence( + db_path: str | os.PathLike[str], + query: str, + limit: int = 20, + expansions: Sequence[str] | None = None, +) -> dict[str, Any]: + """Search natural evidence units through three channels and RRF. + + ``expansions`` are explicit query variants (aliases, translations, or + decomposed facets). No model or implicit semantic expansion is invoked. + """ + + target = _as_path(db_path, label="db_path") + if not target.is_file(): + raise RetrievalIndexError( + "index-not-found", "Retrieval index does not exist", {"db_path": str(target)} + ) + if not isinstance(query, str) or not query.strip(): + raise RetrievalIndexError("invalid-query", "query must be a non-empty string") + if not isinstance(limit, int) or isinstance(limit, bool) or not 1 <= limit <= 1000: + raise RetrievalIndexError( + "invalid-limit", "limit must be an integer from 1 to 1000", {"limit": limit} + ) + if expansions is not None and (isinstance(expansions, (str, bytes)) or not isinstance(expansions, Sequence)): + raise RetrievalIndexError("invalid-expansions", "expansions must be a sequence of strings") + + variants: list[str] = [] + for value in [query, *(expansions or [])]: + if not isinstance(value, str) or not value.strip(): + raise RetrievalIndexError( + "invalid-expansions", "Every query expansion must be a non-empty string" + ) + normalized = _normalize_text(value) + if normalized.casefold() not in {item.casefold() for item in variants}: + variants.append(normalized) + if len(variants) > 64: + raise RetrievalIndexError( + "too-many-expansions", "At most 63 expansions are accepted", {"count": len(variants) - 1} + ) + + try: + connection = sqlite3.connect(str(target)) + except sqlite3.Error as exc: + raise RetrievalIndexError( + "search-open-failed", + "Could not open the SQLite retrieval index", + {"db_path": str(target), "sqlite_error": str(exc)}, + ) from exc + connection.row_factory = sqlite3.Row + try: + _validate_schema(connection, target) + freshness = _require_complete_current_index( + connection, target, operation="search" + ) + candidate_limit = max(100, limit * 5) + scores: dict[str, float] = {} + channel_ranks: dict[str, dict[str, int]] = {} + channel_native_scores: dict[str, dict[str, float | None]] = {} + channels: dict[str, dict[str, Any]] = {} + + def merge_channel( + name: str, + rows: list[sqlite3.Row], + native_score_key: str | None, + *, + weight: float = 1.0, + ) -> None: + channels[name] = { + "status": "ok", "returned": len(rows), "rrf_weight": weight, + } + for rank, row in enumerate(rows, start=1): + unit_id = row["unit_id"] + scores[unit_id] = ( + scores.get(unit_id, 0.0) + weight / (RRF_K + rank) + ) + channel_ranks.setdefault(unit_id, {})[name] = rank + native = float(row[native_score_key]) if native_score_key and row[native_score_key] is not None else None + channel_native_scores.setdefault(unit_id, {})[name] = native + + for variant_index, variant in enumerate(variants): + unicode_match = _unicode_query(variant) + if unicode_match: + content_name = f"unicode61_text:q{variant_index}" + content_rows = list(connection.execute( + """SELECT u.*, bm25(unit_fts_unicode, 0.0, 1.0, 5.0, 3.0, 0.5) AS native_score + FROM unit_fts_unicode f JOIN units u ON u.unit_id=f.unit_id + WHERE unit_fts_unicode MATCH ? + ORDER BY native_score, u.path, u.ordinal LIMIT ?""", + (f"text : ({unicode_match})", candidate_limit), + )) + merge_channel(content_name, content_rows, "native_score") + + routing_name = f"unicode61_routing:q{variant_index}" + routing_query = ( + f"heading_path : ({unicode_match}) OR " + f"section_summary : ({unicode_match}) OR path : ({unicode_match})" + ) + routing_rows, collapsed, scanned = _fetch_with_routing_fanout( + connection, + """SELECT u.*, bm25(unit_fts_unicode, 0.0, 1.0, 5.0, 3.0, 0.5) AS native_score + FROM unit_fts_unicode f JOIN units u ON u.unit_id=f.unit_id + WHERE unit_fts_unicode MATCH ?""", + (routing_query,), candidate_limit, + ) + merge_channel( + routing_name, routing_rows, "native_score", weight=ROUTING_RRF_WEIGHT + ) + channels[routing_name]["routing_collapsed"] = collapsed + channels[routing_name]["scanned"] = scanned + channels[routing_name]["routing_strategy"] = "window-partition" + channels[routing_name]["query_pages"] = 1 + else: + channels[f"unicode61_text:q{variant_index}"] = { + "status": "skipped", "reason": "no-tokenizable-term", "returned": 0, + } + channels[f"unicode61_routing:q{variant_index}"] = { + "status": "skipped", "reason": "no-tokenizable-term", "returned": 0, + } + + trigram_match = _trigram_query(variant) + if trigram_match: + content_name = f"trigram_text:q{variant_index}" + content_rows = list(connection.execute( + """SELECT u.*, bm25(unit_fts_trigram, 0.0, 1.0, 5.0, 3.0, 0.5) AS native_score + FROM unit_fts_trigram f JOIN units u ON u.unit_id=f.unit_id + WHERE unit_fts_trigram MATCH ? + ORDER BY native_score, u.path, u.ordinal LIMIT ?""", + (f"text : ({trigram_match})", candidate_limit), + )) + merge_channel(content_name, content_rows, "native_score") + + routing_name = f"trigram_routing:q{variant_index}" + routing_query = ( + f"heading_path : ({trigram_match}) OR " + f"section_summary : ({trigram_match}) OR path : ({trigram_match})" + ) + routing_rows, collapsed, scanned = _fetch_with_routing_fanout( + connection, + """SELECT u.*, bm25(unit_fts_trigram, 0.0, 1.0, 5.0, 3.0, 0.5) AS native_score + FROM unit_fts_trigram f JOIN units u ON u.unit_id=f.unit_id + WHERE unit_fts_trigram MATCH ?""", + (routing_query,), candidate_limit, + ) + merge_channel( + routing_name, routing_rows, "native_score", weight=ROUTING_RRF_WEIGHT + ) + channels[routing_name]["routing_collapsed"] = collapsed + channels[routing_name]["scanned"] = scanned + channels[routing_name]["routing_strategy"] = "window-partition" + channels[routing_name]["query_pages"] = 1 + else: + channels[f"trigram_text:q{variant_index}"] = { + "status": "skipped", "reason": "query-shorter-than-three-characters", "returned": 0, + } + channels[f"trigram_routing:q{variant_index}"] = { + "status": "skipped", "reason": "query-shorter-than-three-characters", "returned": 0, + } + + exact_text_name = f"exact_text:q{variant_index}" + exact_text_rows = list(connection.execute( + """SELECT u.*, 0 AS native_score FROM units u + WHERE instr(lower(u.normalized_text), lower(?)) > 0 + ORDER BY u.path, u.ordinal LIMIT ?""", + (variant, candidate_limit), + )) + merge_channel(exact_text_name, exact_text_rows, "native_score") + + exact_routing_name = f"exact_routing:q{variant_index}" + routing_rows, collapsed, scanned = _fetch_with_routing_fanout( + connection, + """SELECT u.*, + CASE + WHEN instr(lower(u.heading_path_json), lower(?)) > 0 THEN 0 + WHEN instr(lower(u.section_summary), lower(?)) > 0 THEN 1 + ELSE 2 + END AS native_score + FROM units u + WHERE instr(lower(u.normalized_text), lower(?)) = 0 + AND (instr(lower(u.heading_path_json), lower(?)) > 0 + OR instr(lower(COALESCE(u.section_summary, '')), lower(?)) > 0 + OR instr(lower(u.path), lower(?)) > 0) + """, + (variant, variant, variant, variant, variant, variant), + candidate_limit, + ) + merge_channel( + exact_routing_name, routing_rows, "native_score", weight=ROUTING_RRF_WEIGHT + ) + channels[exact_routing_name]["routing_collapsed"] = collapsed + channels[exact_routing_name]["scanned"] = scanned + channels[exact_routing_name]["routing_strategy"] = "window-partition" + channels[exact_routing_name]["query_pages"] = 1 + + candidate_rows = _fetch_candidate_rows(connection, scores) + absent = sorted(set(scores) - set(candidate_rows)) + if absent: + raise RetrievalIndexError( + "index-integrity-error", + "One or more fused candidates are absent from the units table", + {"unit_ids": absent[:20], "missing_count": len(absent)}, + ) + rerank: dict[str, tuple[float, float, list[str]]] = {} + for unit_id, raw_score in scores.items(): + factor, reasons = _polarity_factor(query, candidate_rows[unit_id]["text"]) + rerank[unit_id] = (raw_score * factor, factor, reasons) + ordered_ids = sorted( + scores, + key=lambda unit_id: (-rerank[unit_id][0], -scores[unit_id], unit_id), + )[:limit] + hits: list[dict[str, Any]] = [] + for rank, unit_id in enumerate(ordered_ids, start=1): + row = candidate_rows[unit_id] + preview_needles = [ + *variants, + *(term for variant in variants for term in _meaningful_query_runs(variant)), + ] + hit = _row_to_hit( + row, + text_limit=DEFAULT_EVIDENCE_TEXT_LIMIT, + preview_needles=preview_needles, + ) + hit.update({ + "rank": rank, + "score": rerank[unit_id][0], + "rrf_score": scores[unit_id], + "polarity_factor": rerank[unit_id][1], + "rerank_reasons": rerank[unit_id][2], + "channel_ranks": channel_ranks[unit_id], + "channel_scores": channel_native_scores[unit_id], + }) + hits.append(hit) + + return { + "ok": True, + "schema_version": SCHEMA_VERSION, + "db_path": str(target), + "corpus_freshness": freshness, + "query": _normalize_text(query), + "expansions": variants[1:], + "limit": limit, + "rrf_k": RRF_K, + "channels": channels, + "candidate_count": len(scores), + "returned": len(hits), + "hits": hits, + } + except RetrievalIndexError: + raise + except sqlite3.Error as exc: + raise RetrievalIndexError( + "search-failed", + "SQLite evidence search failed", + {"db_path": str(target), "sqlite_error": str(exc)}, + ) from exc + except Exception as exc: + raise RetrievalIndexError( + "search-failed", + "Retrieval index data is malformed", + { + "db_path": str(target), + "exception": type(exc).__name__, + "message": str(exc), + }, + ) from exc + finally: + connection.close() + + +__all__ = [ + "RetrievalIndexError", + "SCHEMA_VERSION", + "coverage_report", + "read_evidence_unit", + "rebuild_index", + "search_evidence", +] diff --git a/scripts/section_parser.py b/scripts/section_parser.py index 3ec73d4..ae97a5d 100644 --- a/scripts/section_parser.py +++ b/scripts/section_parser.py @@ -17,15 +17,16 @@ HEADING_RE = re.compile(r"^(#{1,6})\s+(.+?)\s*$") HR_RE = re.compile(r"^[-*_]{3,}\s*$") -FENCE_RE = re.compile(r"^(```|~~~)") -LIST_RE = re.compile(r"^\s*(?:[-*+]|\d+\.)\s+") -TABLE_LINE_RE = re.compile(r"^\s*\|") +FENCE_RE = re.compile(r"^ {0,3}(`{3,}|~{3,})") +LIST_RE = re.compile(r"^\s*(?:[-*+]|\d+[.)])\s+") FIGURE_RE = re.compile(r"^!\[.*?\]\(.*?\)\s*$") FRONTMATTER_RE = re.compile(r"\A---\r?\n.*?\r?\n---\r?\n", re.DOTALL) # 锚点字符类：必须与 postprocess.py 的 ANCHOR_DETECT_RE / ANCHOR_STRIP_RE / KIND_LETTER 同步 # （都只生成 h/p/c/t/f）。 # 修改时同步两处，并跑 scripts/tests/ 下的锚点相关测试。 -ANCHOR_TAIL_RE = re.compile(r"\s+\^[hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?\s*$") +ANCHOR_ID_PATTERN = r"\^[hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?" +ANCHOR_TAIL_RE = re.compile(rf"\s+{ANCHOR_ID_PATTERN}\s*$") +STANDALONE_ANCHOR_RE = re.compile(rf"^\s*{ANCHOR_ID_PATTERN}\s*$") @dataclass @@ -76,10 +77,16 @@ def _classify_lines(lines: list[str]) -> list[str]: """逐行分类：blank / heading / hr / fence / text / fence-mid。""" out = [] in_fence = False - fence_marker = None + fence_marker: tuple[str, int] | None = None for line in lines: if in_fence: - if fence_marker and line.startswith(fence_marker): + marker_char, marker_length = fence_marker or ("", 0) + closing = re.fullmatch( + rf" {{0,3}}{re.escape(marker_char)}{{{marker_length},}}" + rf"[ \t]*(?:{ANCHOR_ID_PATTERN})?[ \t]*\r?", + line, + ) if fence_marker else None + if closing: out.append("fence-end") in_fence = False fence_marker = None @@ -90,7 +97,8 @@ def _classify_lines(lines: list[str]) -> list[str]: if m: out.append("fence-start") in_fence = True - fence_marker = m.group(1) + marker = m.group(1) + fence_marker = (marker[0], len(marker)) continue stripped = line.strip() if not stripped: @@ -104,6 +112,121 @@ def _classify_lines(lines: list[str]) -> list[str]: return out +def _looks_like_gfm_table(lines: list[str]) -> bool: + """Recognize a GFM table even when rows omit leading/trailing pipes.""" + + non_empty = [line.strip() for line in lines if line.strip()] + if non_empty and STANDALONE_ANCHOR_RE.fullmatch(non_empty[-1]): + non_empty = non_empty[:-1] + if len(non_empty) < 2: + return False + header_cells = _table_cells(non_empty[0]) + separator_cells = _table_cells(non_empty[1]) + if ( + _table_delimiter_count(non_empty[0]) < 1 + or _table_delimiter_count(non_empty[1]) < 1 + or not separator_cells + or len(header_cells) != len(separator_cells) + or not all(re.fullmatch(r":?-{3,}:?", cell) for cell in separator_cells) + ): + return False + return all( + _table_delimiter_count(line) >= 1 + for line in non_empty[2:] + ) + + +def _table_cells(line: str) -> list[str]: + """Split a GFM pipe row, ignoring escaped and code-span pipes.""" + + cells, _delimiter_count = _split_table_row(line) + return cells + + +def _table_delimiter_count(line: str) -> int: + """Count real GFM delimiters, excluding escaped/code-span pipes.""" + + _cells, delimiter_count = _split_table_row(line) + return delimiter_count + + +def _split_table_row(line: str) -> tuple[list[str], int]: + """Return parsed cells and the count of real pipe delimiters.""" + + clean = ANCHOR_TAIL_RE.sub("", line).strip() + cells: list[str] = [] + buffer: list[str] = [] + code_run = 0 + delimiter_count = 0 + i = 0 + while i < len(clean): + char = clean[i] + if char == "\\" and i + 1 < len(clean): + buffer.extend((char, clean[i + 1])) + i += 2 + continue + if char == "`": + j = i + 1 + while j < len(clean) and clean[j] == "`": + j += 1 + run = j - i + marker = "`" * run + if code_run == run: + code_run = 0 + elif code_run == 0 and clean.find(marker, j) != -1: + code_run = run + buffer.append(marker) + i = j + continue + if char == "|" and code_run == 0: + delimiter_count += 1 + cells.append("".join(buffer).strip()) + buffer = [] + else: + buffer.append(char) + i += 1 + cells.append("".join(buffer).strip()) + if clean.startswith("|") and cells and not cells[0]: + cells.pop(0) + if clean.endswith("|") and cells and not cells[-1]: + cells.pop() + return cells, delimiter_count + + +def _gfm_table_end(lines: list[str], start: int, limit: int) -> int | None: + """Return the exclusive end of a valid contiguous GFM table.""" + + if start + 1 >= limit: + return None + header = lines[start].strip() + separator = lines[start + 1].strip() + header_cells = _table_cells(header) + separator_cells = _table_cells(separator) + if ( + _table_delimiter_count(header) < 1 + or _table_delimiter_count(separator) < 1 + or not separator_cells + or len(header_cells) != len(separator_cells) + or not all(re.fullmatch(r":?-{3,}:?", cell) for cell in separator_cells) + ): + return None + end = start + 2 + while end < limit: + row = lines[end].strip() + if STANDALONE_ANCHOR_RE.fullmatch(row): + end += 1 + break + # Lists and blockquotes start new Markdown blocks even if their text + # contains a pipe; swallowing them as table rows would corrupt the + # natural-unit inventory used by evidence coverage and retrieval. + if LIST_RE.match(lines[end]) or row.startswith(">"): + break + if _table_delimiter_count(row) < 1: + break + end += 1 + return end + + def split_blocks(text: str) -> list[Block]: """ 把 markdown 切分为块列表。 @@ -183,12 +306,15 @@ def line_char_end(idx: int) -> int: j = i + 1 while j < n and classes[j] == "text": j += 1 + table_end = _gfm_table_end(lines, i, j) + if table_end is not None: + j = table_end block_lines = lines[i:j] first = block_lines[0] first_stripped = first.strip() non_empty = [l for l in block_lines if l.strip()] - if non_empty and all(TABLE_LINE_RE.match(l) for l in non_empty): + if non_empty and _looks_like_gfm_table(block_lines): kind = "table" elif LIST_RE.match(first): kind = "list" @@ -235,7 +361,7 @@ def _clean_preview(text: str) -> str: # 双链 s = re.sub(r"\[\[([^\]|#]+)(?:#[^\]|]+)?(?:\|([^\]]+))?\]\]", r"\1", s) # 列表 / 引用 / heading 前缀 - s = re.sub(r"^\s*(?:[-*+]|\d+\.)\s+", "", s, flags=re.MULTILINE) + s = re.sub(r"^\s*(?:[-*+]|\d+[.)])\s+", "", s, flags=re.MULTILINE) s = re.sub(r"^\s*>\s?", "", s, flags=re.MULTILINE) s = re.sub(r"^\s*#{1,6}\s+", "", s, flags=re.MULTILINE) # 压缩空白 diff --git a/scripts/tests/test_answer_citation_eval.py b/scripts/tests/test_answer_citation_eval.py new file mode 100644 index 0000000..d3dd941 --- /dev/null +++ b/scripts/tests/test_answer_citation_eval.py @@ -0,0 +1,472 @@ +"""Answer→citation E2E protocol and mutation guards.""" +from __future__ import annotations + +import ast +import copy +import json +from pathlib import Path + +import pytest + +from evals.answer_citation import gold, public_cases, reference_adapter, run_eval + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent + + +def _baseline(): + cases, gold_by_id = run_eval._validate_fixture() + responses = run_eval.collect_responses(reference_adapter.answer, cases) + return cases, gold_by_id, responses + + +def _render(response: dict) -> None: + response["answer"] = "\n".join(claim["text"] for claim in response["claims"]) + + +class TestFixtureAndAdapterBoundary: + def test_public_and_gold_are_frozen_independent_and_nonempty(self): + cases, gold_by_id = run_eval._validate_fixture() + assert public_cases.fixture_sha256() == public_cases.FROZEN_PUBLIC_FIXTURE_SHA256 + assert gold.gold_sha256() == gold.FROZEN_GOLD_SHA256 + assert len(cases) == 6 + assert sum(row["answerable"] for row in gold_by_id.values()) == 4 + assert sum(not row["answerable"] for row in gold_by_id.values()) == 2 + assert sum(len(row["required_facets"]) for row in gold_by_id.values()) == 7 + assert gold.GOLD_SCHEMA_VERSION == 3 + for row in gold_by_id.values(): + for facet in row["required_facets"]: + assert set(facet) == {"facet_id", "accepted_variants"} + for variant in facet["accepted_variants"]: + assert set(variant) == { + "variant_id", "claim_text", "value", "evidence_set" + } + public_json = json.dumps(cases, ensure_ascii=False, sort_keys=True) + assert gold.GOLD_CANARY not in public_json + assert "accepted_values" not in public_json + assert "accepted_claim_texts" not in public_json + assert "minimal_evidence_sets" not in public_json + assert "accepted_variants" not in public_json + assert "forbidden_evidence" not in public_json + assert "answerable" not in public_json + + def test_reference_adapter_source_has_no_gold_or_fixture_import(self): + source_path = PROJECT_ROOT / "evals" / "answer_citation" / "reference_adapter.py" + tree = ast.parse(source_path.read_text(encoding="utf-8")) + imports: list[str] = [] + for node in ast.walk(tree): + if isinstance(node, ast.Import): + imports.extend(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + imports.append(node.module or "") + assert all("gold" not in name for name in imports) + assert all("public_cases" not in name for name in imports) + assert reference_adapter.ADAPTER_SCOPE == ( + "protocol-smoke-only-not-model-capability" + ) + + def test_adapter_receives_only_question_and_retrieval_evidence(self): + seen = [] + + def spy(request): + seen.append(copy.deepcopy(request)) + return reference_adapter.answer(request) + + result = run_eval.evaluate(spy) + assert result["passed"] is True + assert len(seen) == 6 + assert all(set(request) == {"question", "evidence"} for request in seen) + serialized = json.dumps(seen, ensure_ascii=False) + assert "case_id" not in serialized + assert "required_facets" not in serialized + assert "minimal_evidence_sets" not in serialized + assert "accepted_variants" not in serialized + assert gold.GOLD_CANARY not in serialized + + def test_reference_result_is_explicit_protocol_smoke_not_capability(self): + result = run_eval.evaluate(reference_adapter.answer, details=True) + assert result["passed"] is True + assert result["status"] == "protocol-smoke-passed" + assert result["scope"] == "public-cc0-protocol-smoke-not-model-capability" + assert "not a model" in result["interpretation"] + assert result["fixture"] == { + "license": "CC0-1.0", + "public_fixture_sha256": public_cases.FROZEN_PUBLIC_FIXTURE_SHA256, + "gold_sha256": gold.FROZEN_GOLD_SHA256, + "cases": 6, + "unique_questions": 6, + "answerable_cases": 4, + "unanswerable_cases": 2, + "required_facets": 7, + } + assert all( + row["value"] == 1.0 + for name, row in result["metrics"].items() + if name != "unsupported_claim_rate" + ) + assert result["metrics"]["unsupported_claim_rate"]["value"] == 0.0 + + def test_real_system_predictions_jsonl_uses_same_protocol(self, tmp_path): + cases, gold_by_id, responses = _baseline() + path = tmp_path / "predictions.jsonl" + path.write_text( + "\n".join(json.dumps({"case_id": case_id, "response": response}) + for case_id, response in responses.items()) + "\n", + encoding="utf-8", + ) + loaded = run_eval.load_predictions(path, cases) + result = run_eval.score_responses(cases, gold_by_id, loaded) + assert result["passed"] is True + + def test_all_abstain_does_not_turn_precision_or_error_rate_into_100(self): + cases, gold_by_id, _ = _baseline() + all_abstain = { + case["case_id"]: {"decision": "abstain", "answer": "", "claims": []} + for case in cases + } + with pytest.raises(run_eval.EvaluationProtocolError, match="denominator"): + run_eval.score_responses(cases, gold_by_id, all_abstain) + + def test_nfkc_casefold_whitespace_duplicate_question_fails_closed( + self, monkeypatch + ): + cases = public_cases.get_public_cases() + original = cases[0]["question"] + fullwidth = "".join( + chr(ord(char) + 0xFEE0) if "!" <= char <= "~" else char + for char in original + ) + cases[1]["question"] = " \n" + fullwidth.replace(" ", " ") + "\t" + monkeypatch.setattr( + public_cases, "get_public_cases", lambda: copy.deepcopy(cases) + ) + monkeypatch.setattr( + public_cases, + "fixture_sha256", + lambda: public_cases.FROZEN_PUBLIC_FIXTURE_SHA256, + ) + + with pytest.raises( + run_eval.EvaluationProtocolError, + match="duplicate normalized public question", + ): + run_eval._validate_fixture() + + def test_punctuation_symbols_and_format_controls_cannot_fake_unique_question( + self, monkeypatch + ): + cases = public_cases.get_public_cases() + original = cases[0]["question"] + cosmetic = original.replace("?", "") + " !!!\u200b" + cases[1]["question"] = cosmetic + monkeypatch.setattr( + public_cases, "get_public_cases", lambda: copy.deepcopy(cases) + ) + monkeypatch.setattr( + public_cases, + "fixture_sha256", + lambda: public_cases.FROZEN_PUBLIC_FIXTURE_SHA256, + ) + + with pytest.raises( + run_eval.EvaluationProtocolError, + match="duplicate normalized public question", + ): + run_eval._validate_fixture() + + def test_question_without_lexical_content_fails_closed(self, monkeypatch): + cases = public_cases.get_public_cases() + cases[0]["question"] = "?!★\u200b" + monkeypatch.setattr( + public_cases, "get_public_cases", lambda: copy.deepcopy(cases) + ) + monkeypatch.setattr( + public_cases, + "fixture_sha256", + lambda: public_cases.FROZEN_PUBLIC_FIXTURE_SHA256, + ) + with pytest.raises( + run_eval.EvaluationProtocolError, + match="no lexical content", + ): + run_eval._validate_fixture() + + def test_v2_gold_schema_is_rejected_instead_of_implicitly_migrated( + self, monkeypatch + ): + monkeypatch.setattr(gold, "GOLD_SCHEMA_VERSION", 2) + with pytest.raises( + run_eval.EvaluationProtocolError, + match="unsupported independent Gold schema", + ): + run_eval._validate_fixture() + + +class TestScoringMutations: + def test_missing_facet_lowers_coverage_and_completeness(self): + cases, gold_by_id, responses = _baseline() + response = responses["ac-smoke-001"] + response["claims"] = [response["claims"][0]] + _render(response) + result = run_eval.score_responses(cases, gold_by_id, responses) + assert result["passed"] is False + assert result["status"] == "threshold-failed" + assert result["metrics"]["required_facet_coverage"]["numerator"] == 6 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["answer_coverage"]["value"] == 1.0 + assert result["metrics"]["fully_grounded_answer_rate"]["numerator"] == 3 + + def test_cross_spliced_text_value_and_evidence_variants_are_unsupported(self): + cases, gold_by_id, responses = _baseline() + victim_response = responses["ac-smoke-001"] + victim = victim_response["claims"][0] + # Each field occurs in Gold, but never together in one accepted variant. + victim["value"] = "18.4 kilograms" + victim["citations"] = [ + "raw/specifications/atlas-skiff.md#^t-3-f0e1d2" + ] + _render(victim_response) + + result = run_eval.score_responses( + cases, gold_by_id, responses, details=True + ) + + assert result["status"] == "threshold-failed" + assert result["metrics"]["required_facet_coverage"]["numerator"] == 6 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + evaluated = result["details"][0]["claims"][0] + assert evaluated["text_correct"] is True + assert evaluated["value_correct"] is True + assert evaluated["content_correct"] is False + assert evaluated["matched_variant_ids"] == [] + assert evaluated["evidence_complete"] is False + assert evaluated["supported"] is False + + def test_evidence_from_another_content_variant_is_not_interchangeable(self): + cases, gold_by_id, responses = _baseline() + victim = responses["ac-smoke-001"]["claims"][0] + victim["citations"] = [ + "raw/specifications/atlas-skiff.md#^t-3-f0e1d2" + ] + + result = run_eval.score_responses( + cases, gold_by_id, responses, details=True + ) + + assert result["status"] == "threshold-failed" + assert result["metrics"]["required_facet_coverage"]["value"] == 1.0 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + evaluated = result["details"][0]["claims"][0] + assert evaluated["matched_variant_ids"] == ["manual-paragraph-kg"] + assert evaluated["complete_variant_ids"] == [] + assert evaluated["supported_citations"] == 0 + assert evaluated["supported"] is False + + def test_same_value_wrong_subject_citation_is_not_support(self): + cases, gold_by_id, responses = _baseline() + responses["ac-smoke-001"]["claims"][0]["citations"] = [ + "raw/manuals/boreal-skiff.md#^p-4-c3d4e5" + ] + result = run_eval.score_responses(cases, gold_by_id, responses, details=True) + assert result["passed"] is False + assert result["metrics"]["claim_citation_precision"]["numerator"] == 7 + assert result["metrics"]["claim_citation_precision"]["denominator"] == 8 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + victim = result["details"][0]["claims"][0] + assert victim["value_correct"] is True + assert victim["supported"] is False + assert victim["selected_forbidden"] + + def test_wrong_subject_text_with_right_value_and_citation_is_not_support(self): + cases, gold_by_id, responses = _baseline() + victim_response = responses["ac-smoke-001"] + victim_response["claims"][0]["text"] = ( + "Boreal Skiff's launch mass is 18.4 kg." + ) + _render(victim_response) + + result = run_eval.score_responses( + cases, gold_by_id, responses, details=True + ) + + assert result["passed"] is False + assert result["metrics"]["claim_citation_precision"]["numerator"] == 7 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["required_facet_coverage"]["numerator"] == 6 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + assert result["metrics"]["fully_grounded_answer_rate"]["numerator"] == 3 + victim = result["details"][0]["claims"][0] + assert victim["value_correct"] is True + assert victim["text_correct"] is False + assert victim["content_correct"] is False + assert victim["supported"] is False + + def test_negated_claim_text_with_right_value_and_citation_is_not_support(self): + cases, gold_by_id, responses = _baseline() + victim_response = responses["ac-smoke-001"] + victim_response["claims"][1]["text"] = ( + "Atlas Skiff's battery endurance is not 11.5 hours." + ) + _render(victim_response) + + result = run_eval.score_responses( + cases, gold_by_id, responses, details=True + ) + + assert result["passed"] is False + assert result["metrics"]["claim_citation_precision"]["numerator"] == 7 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["required_facet_coverage"]["numerator"] == 6 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + victim = result["details"][0]["claims"][1] + assert victim["value_correct"] is True + assert victim["text_correct"] is False + assert victim["evidence_complete"] is False + + def test_appended_unsupported_fact_with_right_value_and_citation_is_not_support(self): + cases, gold_by_id, responses = _baseline() + victim_response = responses["ac-smoke-001"] + victim_response["claims"][0]["text"] = ( + "Atlas Skiff's launch mass is 18.4 kg, and its service supply is " + "28 volts." + ) + _render(victim_response) + + result = run_eval.score_responses( + cases, gold_by_id, responses, details=True + ) + + assert result["passed"] is False + assert result["metrics"]["claim_citation_precision"]["numerator"] == 7 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["required_facet_coverage"]["numerator"] == 6 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + victim = result["details"][0]["claims"][0] + assert victim["value_correct"] is True + assert victim["text_correct"] is False + assert victim["supported_citations"] == 0 + + def test_uncited_claim_is_unsupported_and_incomplete(self): + cases, gold_by_id, responses = _baseline() + responses["ac-smoke-001"]["claims"][0]["citations"] = [] + result = run_eval.score_responses(cases, gold_by_id, responses) + assert result["passed"] is False + assert result["metrics"]["required_facet_coverage"]["value"] == 1.0 + assert result["metrics"]["citation_completeness"]["numerator"] == 6 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + + def test_overcitation_lowers_precision_without_hiding_coverage(self): + cases, gold_by_id, responses = _baseline() + responses["ac-smoke-001"]["claims"][0]["citations"].append( + "raw/manuals/boreal-skiff.md#^p-4-c3d4e5" + ) + result = run_eval.score_responses(cases, gold_by_id, responses) + assert result["passed"] is False + assert result["metrics"]["claim_citation_precision"]["numerator"] == 8 + assert result["metrics"]["claim_citation_precision"]["denominator"] == 9 + assert result["metrics"]["required_facet_coverage"]["value"] == 1.0 + assert result["metrics"]["citation_completeness"]["value"] == 1.0 + assert result["metrics"]["unsupported_claim_rate"]["value"] == 0.0 + assert result["metrics"]["fully_grounded_answer_rate"]["numerator"] == 3 + + def test_wrong_abstention_lowers_answer_and_facet_coverage(self): + cases, gold_by_id, responses = _baseline() + responses["ac-smoke-001"] = { + "decision": "abstain", "answer": "", "claims": [] + } + result = run_eval.score_responses(cases, gold_by_id, responses) + assert result["passed"] is False + assert result["metrics"]["answer_coverage"]["numerator"] == 3 + assert result["metrics"]["answer_coverage"]["denominator"] == 4 + assert result["metrics"]["required_facet_coverage"]["numerator"] == 5 + assert result["metrics"]["claim_citation_precision"]["value"] == 1.0 + + def test_answering_unanswerable_case_lowers_abstention_and_support(self): + cases, gold_by_id, responses = _baseline() + claim = { + "claim_id": "c1", + "text": "Cedar Records imposes a 7-credit export fee.", + "facet_id": "export_fee", + "value": "7 credits", + "citations": ["raw/policies/hemlock-records.md#^t-9-f9a0b1"], + } + responses["ac-smoke-006"] = { + "decision": "answer", "answer": claim["text"], "claims": [claim] + } + result = run_eval.score_responses(cases, gold_by_id, responses) + assert result["passed"] is False + assert result["metrics"]["correct_abstention_rate"]["numerator"] == 1 + assert result["metrics"]["correct_abstention_rate"]["denominator"] == 2 + assert result["metrics"]["unsupported_claim_rate"]["numerator"] == 1 + assert result["metrics"]["claim_citation_precision"]["denominator"] == 9 + + def test_gold_canary_leak_fails_before_scoring(self): + cases, _, responses = _baseline() + victim = responses["ac-smoke-001"] + victim["claims"][0]["text"] += " " + gold.GOLD_CANARY + _render(victim) + case = next(case for case in cases if case["case_id"] == "ac-smoke-001") + with pytest.raises(run_eval.EvaluationProtocolError, match="Gold canary"): + run_eval._validate_response(case, victim) + + +class TestProtocolFailClosed: + def test_adapter_system_exit_zero_becomes_protocol_error(self): + def exits_without_result(_request): + raise SystemExit(0) + + with pytest.raises( + run_eval.EvaluationProtocolError, match="SystemExit: 0" + ): + run_eval.evaluate(exits_without_result) + + def test_main_system_exit_zero_returns_structured_protocol_error( + self, monkeypatch, capsys + ): + def exits_without_result(*_args, **_kwargs): + raise SystemExit(0) + + monkeypatch.setattr(run_eval, "evaluate", exits_without_result) + exit_code = run_eval.main(["--json"]) + output = capsys.readouterr().out.strip() + + assert exit_code == 2 + assert output.startswith(run_eval.RESULT_PREFIX) + payload = json.loads(output.removeprefix(run_eval.RESULT_PREFIX)) + assert payload["status"] == "protocol-error" + assert payload["passed"] is False + assert payload["protocol_error"] is True + assert "SystemExit: 0" in payload["error"] + + def test_malformed_or_unretrieved_citation_is_protocol_error(self): + cases, _, responses = _baseline() + case = cases[0] + victim = responses[case["case_id"]] + victim["claims"][0]["citations"] = ["raw/a.md#^p-1-abcdef"] + with pytest.raises(run_eval.EvaluationProtocolError, match="not in retrieval"): + run_eval._validate_response(case, victim) + victim["claims"][0]["citations"] = ["./raw/a.md#^p-1-abcdef"] + with pytest.raises(run_eval.EvaluationProtocolError, match="non-canonical"): + run_eval._validate_response(case, victim) + + def test_answer_text_cannot_hide_unenumerated_prose(self): + cases, _, responses = _baseline() + case = cases[0] + victim = responses[case["case_id"]] + victim["answer"] += "\nAn extra factual assertion." + with pytest.raises(run_eval.EvaluationProtocolError, match="exact newline join"): + run_eval._validate_response(case, victim) + + def test_missing_prediction_case_and_empty_metric_fail_closed(self, tmp_path): + cases, _, responses = _baseline() + responses.pop("ac-smoke-006") + path = tmp_path / "short.jsonl" + path.write_text( + "\n".join(json.dumps({"case_id": key, "response": value}) + for key, value in responses.items()), + encoding="utf-8", + ) + with pytest.raises(run_eval.EvaluationProtocolError, match="missing cases"): + run_eval.load_predictions(path, cases) + with pytest.raises(run_eval.EvaluationProtocolError, match="denominator"): + run_eval._metric(0, 0, scope="test") diff --git a/scripts/tests/test_cite_eval.py b/scripts/tests/test_cite_eval.py new file mode 100644 index 0000000..2f4a21c --- /dev/null +++ b/scripts/tests/test_cite_eval.py @@ -0,0 +1,343 @@ +"""对抗评测集的确定性子集守护：evals/cite-check 的 20 个 case +（12 毒化 + 8 干净）在各自的期望层必须正确分类。 + +语义层（盲填/反驳）指标由 run_eval.py --semantic 度量（LLM 有方差，不进 CI 断言）。 +""" +from __future__ import annotations + +import importlib.util +import json +import subprocess +import sys +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent + + +def _load_module(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +CORPUS = _load_module("cite_eval_corpus_test", PROJECT_ROOT / "evals" / "cite-check" / "corpus.py") +RUN_EVAL = _load_module("cite_eval_runner_test", PROJECT_ROOT / "evals" / "cite-check" / "run_eval.py") +AUDIT = _load_module("cite_audit_client_test", PROJECT_ROOT / "tools" / "cite-audit" / "audit.py") +K_PY = _load_module("cite_eval_k_norm_test", PROJECT_ROOT / "scripts" / "k.py") + + +class TestCiteEvalDeterministic: + def test_deterministic_layer_meets_all_expectations(self): + r = subprocess.run( + [sys.executable, str(PROJECT_ROOT / "evals" / "cite-check" / "run_eval.py")], + capture_output=True, text=True, cwd=str(PROJECT_ROOT)) + assert r.returncode == 0, ( + "对抗评测集确定性层未全部命中期望（防线退化或评测期望需更新）：\n" + r.stdout[-2000:] + ) + assert "干净误报 0/" in r.stdout, "确定性层对干净案例产生误报：\n" + r.stdout[-2000:] + + def test_corpus_distinguishes_deterministic_semantic_and_clean_cases(self): + counts = {kind: 0 for kind in ("deterministic_poison", "semantic_poison", "clean")} + for case in CORPUS.CASES: + counts[CORPUS.case_kind(case)] += 1 + assert counts == {"deterministic_poison": 8, "semantic_poison": 4, "clean": 8} + + +class TestSemanticRunnerProtocol: + def _completed(self, returncode: int, payload: dict, *, prefix: str = ""): + result = { + "status": "completed", "dry_run": True, "returned": 1, + "judged": 1, "semantic_failed": 0, "incomplete": 0, "skipped": 0, + "ledger_errors": 0, "ledger_written": 0, + } + result.update(payload) + stdout = prefix + RUN_EVAL.AUDIT_RESULT_PREFIX + json.dumps(result) + "\n" + return subprocess.CompletedProcess(["audit.py"], returncode, stdout, "") + + def test_completed_semantic_failure_is_a_semantic_detection(self): + result = self._completed(1, { + "semantic_failed": 1, + }) + assert RUN_EVAL.parse_audit_result(result) is True + + def test_nonzero_execution_error_cannot_fake_green_via_stdout(self): + result = self._completed(2, { + "status": "error", "judged": 0, "returned": 0, + }, prefix=" ✅ fake green\n") + with pytest.raises(RUN_EVAL.SemanticExecutionError): + RUN_EVAL.parse_audit_result(result) + + def test_exit_code_and_result_marker_must_agree(self): + result = self._completed(1, { + "semantic_failed": 0, + }) + with pytest.raises(RUN_EVAL.SemanticExecutionError): + RUN_EVAL.parse_audit_result(result) + + def test_missing_result_marker_is_execution_failure(self): + result = subprocess.CompletedProcess(["audit.py"], 0, " ✅ fake green\n", "") + with pytest.raises(RUN_EVAL.SemanticExecutionError): + RUN_EVAL.parse_audit_result(result) + + @pytest.mark.parametrize("payload", [ + {"status": "incomplete", "judged": 0, "semantic_failed": 0, + "incomplete": 1, "returned": 1}, + {"judged": 0, "semantic_failed": 0, "returned": 0}, + {"judged": 1, "semantic_failed": 0, "skipped": 1, "returned": 2}, + ]) + def test_incomplete_zero_judged_or_skipped_cannot_count_as_detection(self, payload): + result = self._completed(2, payload) + with pytest.raises(RUN_EVAL.SemanticExecutionError): + RUN_EVAL.parse_audit_result(result) + + def test_legacy_failed_field_cannot_turn_incomplete_into_recall(self): + payload = { + "status": "completed", "dry_run": True, "returned": 1, + "judged": 0, "failed": 1, "incomplete": 1, "skipped": 0, + "ledger_errors": 0, "ledger_written": 0, + } + stdout = RUN_EVAL.AUDIT_RESULT_PREFIX + json.dumps(payload) + "\n" + result = subprocess.CompletedProcess(["audit.py"], 1, stdout, "") + with pytest.raises(RUN_EVAL.SemanticExecutionError): + RUN_EVAL.parse_audit_result(result) + + def _patch_single_clean_case(self, monkeypatch): + case = { + "id": "clean", "error_type": "clean", "l1": "pass", + "semantic_expect_fail": False, + } + monkeypatch.setattr(RUN_EVAL.corpus, "CASES", [case]) + monkeypatch.setattr(RUN_EVAL.corpus, "materialize", lambda _tmp: None) + monkeypatch.setattr(RUN_EVAL.corpus, "case_page", lambda _cid: "wiki/clean.md") + monkeypatch.setattr(RUN_EVAL.corpus, "case_kind", lambda _case: "clean") + monkeypatch.setattr(RUN_EVAL, "run_k", lambda _tmp, _args: "[]") + monkeypatch.setattr(sys, "argv", ["run_eval.py", "--semantic"]) + + def test_runner_exit_code_reflects_semantic_expectation_failure(self, monkeypatch): + self._patch_single_clean_case(monkeypatch) + child = self._completed(1, { + "semantic_failed": 1, + }) + monkeypatch.setattr(RUN_EVAL.subprocess, "run", lambda *_args, **_kwargs: child) + assert RUN_EVAL.main() == 1 + + def test_runner_exit_code_reflects_semantic_execution_failure(self, monkeypatch): + self._patch_single_clean_case(monkeypatch) + child = self._completed(2, { + "status": "error", "judged": 0, "returned": 0, + }) + monkeypatch.setattr(RUN_EVAL.subprocess, "run", lambda *_args, **_kwargs: child) + assert RUN_EVAL.main() == 2 + + +class TestAuditFailClosedHelpers: + def test_result_counts_keep_semantic_incomplete_and_skipped_disjoint(self): + counts = AUDIT.result_counts( + 4, + [{"verdict": "SUPPORTED"}, {"verdict": "UNSUPPORTED"}], + [("p2", "x:2", "semantic", "semantic"), + ("p3", "x:3", "quote missing", "incomplete")], + skipped=1, + ) + assert counts == { + "returned": 4, "judged": 2, "semantic_failed": 1, + "incomplete": 1, "skipped": 1, + } + + def test_conflicting_multi_source_cloze_values_do_not_first_win(self): + merged, conflicts = AUDIT.merge_cloze_fills([ + {"N1": "41%", "N2": "unknown"}, + {"N1": "9%", "N2": "120 ms"}, + ]) + assert "N1" not in merged + assert conflicts == {"N1": ["41%", "9%"]} + assert merged["N2"] == "120 ms" + + def test_equivalent_multi_source_values_are_not_conflicts(self): + merged, conflicts = AUDIT.merge_cloze_fills([ + {"N1": "87.5％", "N2": "1,000"}, + {"N1": " 87.50% ", "N2": "1000"}, + ]) + assert conflicts == {} + assert merged == {"N1": "87.5％", "N2": "1,000"} + + def test_genuinely_different_numeric_values_still_conflict(self): + merged, conflicts = AUDIT.merge_cloze_fills([ + {"N1": "87.5%", "N2": "1,000"}, + {"N1": "87.6%", "N2": "1001"}, + ]) + assert merged == {} + assert conflicts == {"N1": ["87.5%", "87.6%"], "N2": ["1,000", "1001"]} + + def test_truncated_packet_is_not_certifiable(self): + assert AUDIT.packet_issue({ + "claim_text": "短论断", "evidence": "完整原文", "evidence_truncated": True, + }) + assert AUDIT.packet_issue({ + "claim_text": "x" * 500 + "…", "evidence": "完整原文", + "evidence_truncated": False, + }) + assert AUDIT.packet_issue({ + "claim_text": "x" * 500 + "…", "claim_text_truncated": False, + "evidence": "完整原文", "evidence_truncated": False, + }) is None + assert AUDIT.packet_issue({ + "claim_text": "显式截断包", "claim_text_truncated": True, + "evidence": "完整原文", "evidence_truncated": False, + }) + assert AUDIT.packet_issue({ + "claim_text": "短论断", "evidence": "少一字", "evidence_length": 5, + "evidence_truncated": False, + }) + + def test_supported_quote_must_be_a_real_evidence_substring_even_in_dry_run(self): + evidence = "系统 Alpha 的准确率为 87.5%，Beta 为 62.3%。" + assert AUDIT.valid_evidence_quote("Alpha 的准确率为 87.5%", evidence) + assert not AUDIT.valid_evidence_quote("Alpha 的准确率为 97.5%", evidence) + assert not AUDIT.valid_evidence_quote("Alpha", evidence) + + @pytest.mark.parametrize("text", [ + "Alpha 1,000 **items** ^p-2-abc123", + "Beta\u200b \\mathbf{87.5}\\, % ^h-2-3-def456", + "repeat 78.4278.4278.42B and `code`", + "保留，普通句读逗号；只删 1,000 的千分位", + ]) + def test_quote_normalization_exactly_matches_ledger(self, text): + assert AUDIT._norm_evidence(text) == K_PY._norm_for_evidence(text) + + def test_prose_comma_cannot_pass_dry_run_then_fail_ledger(self): + evidence = "Alpha 准确，Beta 更快，两者都稳定。" + assert AUDIT.valid_evidence_quote("Alpha 准确，Beta 更快", evidence) + assert not AUDIT.valid_evidence_quote("Alpha 准确 Beta 更快", evidence) + + +class TestAuditLedgerFailures: + def test_multi_source_cloze_conflict_is_incomplete_not_semantic_failure( + self, monkeypatch, capsys): + common = { + "page": "wiki/x.md", "line": 8, + "claim_text": "比例为 41% [[raw/a#^p-1-a]][[raw/b#^p-1-b]]。", + "target_status": "ok", "evidence_truncated": False, + "claim_text_truncated": False, + "cloze": {"text": "比例为 ⟦N1⟧。", "blanks": [{"ph": "N1", "raw": "41%"}]}, + } + pairs = [ + {**common, "pair_id": "pair-a", "target": "raw/a", "anchor": "p-1-a", + "evidence": "A 报告比例为 41%。"}, + {**common, "pair_id": "pair-b", "target": "raw/b", "anchor": "p-1-b", + "evidence": "B 报告比例为 9%。"}, + ] + + def fake_run_k(args, _env): + assert "extract-claims" in args + return json.dumps({"pairs": pairs, "summary": {"returned": 2}}) + + fills = iter([{"N1": "41%"}, {"N1": "9%"}]) + monkeypatch.setattr(AUDIT, "run_k", fake_run_k) + monkeypatch.setattr(AUDIT, "call_llm", lambda *_args, **_kwargs: next(fills)) + monkeypatch.setenv("DEEPSEEK_API_KEY", "test-only") + monkeypatch.setattr(sys, "argv", ["audit.py", "--workspace", "main", "--dry-run"]) + + assert AUDIT.main() == 2 + output = capsys.readouterr().out + marker = next(line for line in output.splitlines() + if line.startswith(AUDIT.RESULT_PREFIX)) + payload = json.loads(marker[len(AUDIT.RESULT_PREFIX):]) + assert payload["semantic_failed"] == 0 + assert payload["judged"] == 0 + assert payload["incomplete"] == 2 + + def test_draft_mode_uses_check_draft_and_draft_scoped_ledger( + self, monkeypatch, tmp_path): + draft = tmp_path / "answer.md" + draft.write_text("论断 [[raw/x#^p-1-abc123]]\n", encoding="utf-8") + pair = { + "pair_id": "draft-pair", "page": "", "line": 1, + "claim_text": "Alpha 的性能在测试中表现较好。", + "target": "raw/x.md", "anchor": "p-1-abc123", + "target_status": "ok", "evidence": "Alpha 的性能在测试中表现较好。", + "evidence_length": 18, "evidence_truncated": False, + "claim_text_truncated": False, + } + calls = [] + + def fake_run_k(args, _env): + calls.append(args) + if "check-draft" in args: + return json.dumps({"pairs": [pair], "summary": {"returned": 1}}) + if "cite-audit-log" in args: + return json.dumps({"written": [{"pair_id": "draft-pair"}], "errors": []}) + raise AssertionError(args) + + monkeypatch.setattr(AUDIT, "run_k", fake_run_k) + monkeypatch.setattr(AUDIT, "call_llm", lambda *_args, **_kwargs: { + "refuted": False, "reason": "", "quote": "Alpha 的性能在测试中表现较好", + }) + monkeypatch.setenv("DEEPSEEK_API_KEY", "test-only") + monkeypatch.setattr(sys, "argv", [ + "audit.py", "--workspace", "main", "--draft", str(draft), + ]) + + assert AUDIT.main() == 0 + assert "check-draft" in calls[0] + assert "extract-claims" not in calls[0] + ledger_call = next(args for args in calls if "cite-audit-log" in args) + assert ledger_call[ledger_call.index("--draft") + 1] == str(draft) + + def test_truncated_pair_never_calls_llm_or_records_supported(self, monkeypatch, capsys): + pair = { + "pair_id": "pair-truncated", "page": "wiki/x.md", "line": 8, + "claim_text": "Alpha 的性能较好。", "target": "raw/x", "anchor": "p-1-a", + "target_status": "ok", "evidence": "被截断的原文", "evidence_truncated": True, + } + + def fake_run_k(args, _env): + assert "extract-claims" in args + assert "--max-evidence-chars" in args + return json.dumps({"pairs": [pair], "summary": {"returned": 1}}) + + monkeypatch.setattr(AUDIT, "run_k", fake_run_k) + monkeypatch.setattr(AUDIT, "call_llm", lambda *_args, **_kwargs: pytest.fail("truncated packet called LLM")) + monkeypatch.setenv("DEEPSEEK_API_KEY", "test-only") + monkeypatch.setattr(sys, "argv", ["audit.py", "--workspace", "main", "--dry-run"]) + assert AUDIT.main() == 2 + output = capsys.readouterr().out + assert "evidence 被截断" in output + assert "未写台账" in output + marker = next(line for line in output.splitlines() + if line.startswith(AUDIT.RESULT_PREFIX)) + payload = json.loads(marker[len(AUDIT.RESULT_PREFIX):]) + assert payload["judged"] == 0 + assert payload["semantic_failed"] == 0 + assert payload["incomplete"] == 1 + + def test_ledger_write_errors_make_audit_nonzero(self, monkeypatch, capsys): + pair = { + "pair_id": "pair-1", "page": "wiki/x.md", "line": 12, + "claim_text": "Alpha 的性能较好。", "target": "raw/x", "anchor": "p-1-a", + "target_status": "ok", "evidence": "Alpha 的性能在测试中表现较好。", + "evidence_truncated": False, + } + + def fake_run_k(args, _env): + if "extract-claims" in args: + return json.dumps({"pairs": [pair], "summary": {"returned": 1}}) + if "cite-audit-log" in args: + return json.dumps({"written": [], "errors": [ + {"pair_id": "pair-1", "error": "evidence mismatch"}, + ]}) + raise AssertionError(args) + + monkeypatch.setattr(AUDIT, "run_k", fake_run_k) + monkeypatch.setattr(AUDIT, "call_llm", lambda *_args, **_kwargs: { + "refuted": False, "reason": "", "quote": "Alpha 的性能在测试中表现较好", + }) + monkeypatch.setenv("DEEPSEEK_API_KEY", "test-only") + monkeypatch.setattr(sys, "argv", ["audit.py", "--workspace", "main"]) + assert AUDIT.main() != 0 + assert "拒绝 1 条" in capsys.readouterr().out diff --git a/scripts/tests/test_conversion_fidelity.py b/scripts/tests/test_conversion_fidelity.py new file mode 100644 index 0000000..e5d9986 --- /dev/null +++ b/scripts/tests/test_conversion_fidelity.py @@ -0,0 +1,586 @@ +"""PDF/DOCX/HTML conversion-fidelity protocol and mutation guards.""" +from __future__ import annotations + +import copy +import hashlib +import json +import subprocess +from pathlib import Path + +import pdfplumber +import pytest +from docx import Document +from docx.oxml.ns import qn +from lxml import html + +from evals.conversion_fidelity import fixtures, gold, run_eval + + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent + + +@pytest.fixture(scope="module") +def converted_fixture(tmp_path_factory): + work_root = tmp_path_factory.mktemp("conversion-fidelity") + kb_root = work_root / "kb-data" + workspace = kb_root / "workspaces" / "conversion-fidelity" + raw = workspace / "raw" + raw.mkdir(parents=True) + run_eval._require_runtime_dependencies() + sources = fixtures.materialize(raw) + pipeline = run_eval._convert_real_pipeline(kb_root, workspace) + outputs, receipts = run_eval._load_and_validate_outputs(sources, workspace) + return { + "work_root": work_root, + "workspace": workspace, + "sources": sources, + "outputs": outputs, + "receipts": receipts, + "pipeline": pipeline, + } + + +def _detail(result, metric, row_id): + rows = [row for row in result["details"][metric] if row["id"] == row_id] + assert len(rows) == 1 + return rows[0] + + +def _file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def test_gold_v4_is_frozen_and_every_assertion_has_executable_coordinates(): + gold.validate_gold() + assert gold.GOLD_SCHEMA_VERSION == 4 + assert gold.gold_digest() == gold.FROZEN_GOLD_SHA256 + groups = ( + gold.CRITICAL_FACTS, + gold.QUALIFIERS, + gold.TABLE_RELATIONS, + gold.READING_ORDERS, + gold.STANDALONE_TEXT_RELATIONS, + gold.LIST_RELATIONS, + gold.FOOTNOTE_RELATIONS, + ) + assert all(groups) + inventory = { + doc_id: {(row["title"], row["level"]) for row in rows} + for doc_id, rows in gold.HEADING_INVENTORIES.items() + } + for group in groups: + for row in group: + assert row["source_locator"].strip() + if group is gold.FOOTNOTE_RELATIONS: + for section_name, block_name in ( + ("marker_section", "marker_block"), + ("note_section", "note_block"), + ): + target = row[section_name] + assert (target["title"], target["level"]) in inventory[row["doc_id"]] + assert row[block_name]["kind"] == "paragraph" + assert row[block_name]["ordinal"] > 0 + continue + if group is gold.STANDALONE_TEXT_RELATIONS: + if row.get("target_scope") == "preamble": + assert "target_section" not in row + else: + target = row["target_section"] + assert (target["title"], target["level"]) in inventory[row["doc_id"]] + assert row["target_block"]["kind"] == "paragraph" + assert row["target_block"]["ordinal"] > 0 + assert row["text"].strip() + continue + target = row["target_section"] + assert (target["title"], target["level"]) in inventory[row["doc_id"]] + if group is gold.LIST_RELATIONS: + assert row["items"] + assert all( + item["kind"] in {"ordered", "unordered"} + and item["text"].strip() + for item in row["items"] + ) + elif group is not gold.READING_ORDERS: + assert row["target_block"]["kind"] in {"paragraph", "table"} + assert row["target_block"]["ordinal"] > 0 + assert {row["doc_id"] for group in groups for row in group} == { + "pdf_protocol", "docx_reference", "html_notice" + } + + +def test_fixture_generation_is_temp_only_and_records_binary_and_semantic_receipts( + converted_fixture, +): + sources = converted_fixture["sources"] + assert {path.suffix for path in sources.values()} == {".pdf", ".docx", ".html"} + assert all(path.is_file() and path.stat().st_size > 0 for path in sources.values()) + tracked = subprocess.run( + ["git", "ls-files", "evals/conversion_fidelity"], cwd=PROJECT_ROOT, + text=True, stdout=subprocess.PIPE, check=True, + ).stdout.splitlines() + assert not any(Path(path).suffix.lower() in {".pdf", ".docx", ".html"} + for path in tracked) + assert all(str(path).startswith(str(converted_fixture["work_root"])) + for path in sources.values()) + + payload = fixtures.semantic_fixture_payload(sources) + document_digests = fixtures.semantic_document_digests(payload) + assert fixtures.semantic_fixture_digest(sources) == ( + fixtures.FROZEN_SEMANTIC_FIXTURE_SHA256 + ) + receipts = {row["doc_id"]: row for row in converted_fixture["receipts"]} + for doc_id, source in sources.items(): + assert receipts[doc_id]["source_sha256"] == _file_sha256(source) + assert receipts[doc_id]["semantic_sha256"] == document_digests[doc_id] + assert len(receipts[doc_id]["source_sha256"]) == 64 + assert len(receipts[doc_id]["semantic_sha256"]) == 64 + # Binary hashes are receipts only. The frozen, source-native semantic + # digest is the reproducibility contract; container bytes are not fixed. + + +def test_pdf_fixture_has_real_two_column_geometry(converted_fixture): + path = converted_fixture["sources"]["pdf_protocol"] + with pdfplumber.open(path) as document: + assert len(document.pages) == 2 + source_text = "\n".join(page.extract_text() or "" for page in document.pages) + assert "-7.25 °C" in source_text + assert "±0.08 mm" in source_text + page = document.pages[1] + words = page.extract_words(use_text_flow=False) + by_token = { + token: [word for word in words if word["text"].startswith(token)] + for token in ( + "LEFT-START", "LEFT-MIDDLE", "LEFT-END", + "RIGHT-START", "RIGHT-MIDDLE", "RIGHT-END", + ) + } + assert all(len(matches) == 1 for matches in by_token.values()) + left = [by_token[token][0] for token in ("LEFT-START", "LEFT-MIDDLE", "LEFT-END")] + right = [ + by_token[token][0] + for token in ("RIGHT-START", "RIGHT-MIDDLE", "RIGHT-END") + ] + assert max(word["x1"] for word in left) < min(word["x0"] for word in right) + assert all(word["x1"] < page.width / 2 for word in left) + assert all(word["x0"] > page.width / 2 for word in right) + assert [word["top"] for word in left] == sorted(word["top"] for word in left) + assert [word["top"] for word in right] == sorted(word["top"] for word in right) + + +def test_html_fixture_has_real_semantic_dom(converted_fixture): + document = html.parse(str(converted_fixture["sources"]["html_notice"])) + root = document.getroot() + sections = root.xpath("//main/section") + assert [section.get("id") for section in sections] == [ + "limits", "actions", "register", "notes" + ] + assert [" ".join(node.text_content().split()) for node in root.xpath("//main//h2")] == [ + "Measurement limits", "Required actions", "Validation register", "Notes" + ] + assert [" ".join(node.text_content().split()) for node in root.xpath("//*[@id='actions']/ol/li")] == [ + "HTML-ORDER-1: seal the transfer coupling.", + "HTML-ORDER-2: capture the reference sample.", + "HTML-ORDER-3: release the batch record.", + ] + assert [" ".join(node.text_content().split()) for node in root.xpath("//*[@id='register']//th")] == [ + "Device", "Concentration", "Condition" + ] + assert [ + [" ".join(cell.text_content().split()) for cell in row.xpath("./td")] + for row in root.xpath("//*[@id='register']//tbody/tr") + ] == [ + ["Helios-Q", "18.6 mg/L", "sealed transfer only"], + ["Iris-R", "21.4 mg/L", "open rinse only"], + ] + assert root.xpath("count(//*[@id='note-1'])") == 1.0 + assert root.xpath("string(//*[@id='limits']//a/@href)") == "#note-1" + + +def test_docx_compact_tokens_real_lists_and_fixed_table_geometry(converted_fixture): + document = Document(converted_fixture["sources"]["docx_reference"]) + section = document.sections[0] + assert section.page_width == 7772400 # 8.5 in in EMU + assert section.page_height == 10058400 # 11 in in EMU + assert all(margin == 914400 for margin in ( + section.top_margin, section.right_margin, + section.bottom_margin, section.left_margin, + )) + assert document.styles["Normal"].font.name == "Calibri" + assert document.styles["Normal"].font.size.pt == 11 + assert document.styles["Normal"].paragraph_format.line_spacing == 1.25 + assert document.styles["Title"].font.name == "Calibri" + assert document.styles["Title"].font.size.pt == 22 + assert document.styles["Title"].paragraph_format.space_after.pt == 8 + assert document.styles["Title"]._element.pPr.find(qn("w:pBdr")) is None + assert document.styles["Subtitle"].font.name == "Calibri" + assert document.styles["Subtitle"].font.size.pt == 10.5 + assert document.styles["Subtitle"].paragraph_format.space_after.pt == 12 + + heading_expectations = { + "Heading 1": (16, 18, 10), + "Heading 2": (13, 14, 7), + "Heading 3": (12, 10, 5), + } + for name, (size, before, after) in heading_expectations.items(): + style = document.styles[name] + assert style.font.name == "Calibri" + assert style.font.size.pt == size + assert style.paragraph_format.space_before.pt == before + assert style.paragraph_format.space_after.pt == after + + list_paragraphs = [ + paragraph for paragraph in document.paragraphs + if paragraph.style.name in {"List Number", "List Bullet"} + ] + assert [paragraph.style.name for paragraph in list_paragraphs] == [ + "List Number", "List Number", "List Bullet" + ] + for style_name in ("List Number", "List Bullet"): + num_pr = document.styles[style_name]._element.pPr.find(qn("w:numPr")) + assert num_pr is not None + num_id = num_pr.find(qn("w:numId")).get(qn("w:val")) + concrete = next( + node for node in document.part.numbering_part.element.findall(qn("w:num")) + if node.get(qn("w:numId")) == num_id + ) + abstract_id = concrete.find(qn("w:abstractNumId")).get(qn("w:val")) + abstract = next( + node for node in document.part.numbering_part.element.findall(qn("w:abstractNum")) + if node.get(qn("w:abstractNumId")) == abstract_id + ) + level_p_pr = abstract.find(qn("w:lvl")).find(qn("w:pPr")) + indent = level_p_pr.find(qn("w:ind")) + spacing = level_p_pr.find(qn("w:spacing")) + assert indent.get(qn("w:left")) == "540" + assert indent.get(qn("w:hanging")) == "270" + assert spacing.get(qn("w:after")) == "80" + assert spacing.get(qn("w:line")) == "300" + + assert len(document.tables) == 1 + table = document.tables[0] + table_pr = table._tbl.tblPr + assert table_pr.find(qn("w:tblW")).get(qn("w:w")) == "9360" + assert table_pr.find(qn("w:tblW")).get(qn("w:type")) == "dxa" + assert table_pr.find(qn("w:tblInd")).get(qn("w:w")) == "120" + expected_widths = [2160, 1800, 1800, 3600] + assert [int(col.get(qn("w:w"))) for col in table._tbl.tblGrid] == expected_widths + for row in table.rows: + assert [ + int(cell._tc.tcPr.find(qn("w:tcW")).get(qn("w:w"))) + for cell in row.cells + ] == expected_widths + + +def test_real_pipeline_has_every_baseline_metric_hard_green(converted_fixture): + assert converted_fixture["pipeline"]["returncode"] == 0 + assert len(converted_fixture["receipts"]) == 3 + assert {row["format"] for row in converted_fixture["receipts"]} == { + "pdf", "docx", "html" + } + result = run_eval.evaluate_outputs(converted_fixture["outputs"], include_details=True) + assert set(result["metrics"]) == { + "critical_fact_preservation", + "qualifier_preservation", + "table_alignment", + "reading_order", + "standalone_text_preservation", + "list_structure_preservation", + "section_count_preservation", + "heading_hierarchy_preservation", + "footnote_relation_preservation", + } + for metric in result["metrics"].values(): + assert metric["value"] == 1.0 + assert metric["passed"] is True + + +@pytest.mark.parametrize( + "control_id,metric,row_id,mutator", + [ + ( + "delete-negative-sign", "critical_fact_preservation", + "docx-negative-pressure", + lambda outputs: run_eval._mutate_once( + outputs, "docx_reference", "-0.45 kPa", "0.45 kPa" + ), + ), + ( + "delete-unit", "critical_fact_preservation", + "docx-plus-minus-tolerance", + lambda outputs: run_eval._mutate_once( + outputs, "docx_reference", "±0.14 mm", "±0.14" + ), + ), + ( + "delete-negation", "qualifier_preservation", "html-expiry-negation", + lambda outputs: run_eval._mutate_once( + outputs, "html_notice", "not valid after", "valid after" + ), + ), + ( + "delete-table-header", "table_alignment", "html-validation-register", + lambda outputs: run_eval._mutate_once( + outputs, "html_notice", "Concentration", "Concentration lost" + ), + ), + ( + "reorder-reading-sequence", "reading_order", "html-dom-order", + lambda outputs: run_eval._swap_once( + outputs, "html_notice", "HTML-ORDER-1", "HTML-ORDER-2" + ), + ), + ( + "move-fact-to-wrong-section", "critical_fact_preservation", + "docx-negative-pressure", + lambda outputs: run_eval._move_line_to_section_once( + outputs, "docx_reference", "inlet pressure correction", "Scope notes" + ), + ), + ( + "flatten-table-to-prose", "table_alignment", "html-validation-register", + lambda outputs: run_eval._flatten_pipe_table_once( + outputs, "html_notice", "Concentration" + ), + ), + ( + "cross-table-row-values", "table_alignment", "pdf-acceptance-register", + lambda outputs: run_eval._swap_once( + outputs, "pdf_protocol", "2.45 bar", "3.10 bar" + ), + ), + ( + "insert-extra-heading", "section_count_preservation", + "html_notice-exact-heading-inventory", + lambda outputs: run_eval._insert_extra_heading_once( + outputs, "html_notice", "Unexpected appendix" + ), + ), + ( + "delete-footnote-marker", "footnote_relation_preservation", + "pdf-maintenance-footnote-relation", + lambda outputs: run_eval._mutate_once( + outputs, "pdf_protocol", "offset is active. 1", "offset is active." + ), + ), + ( + "corrupt-footnote-link", "footnote_relation_preservation", + "html-expiry-footnote-relation", + lambda outputs: run_eval._mutate_once( + outputs, "html_notice", "(#note-1)", "(#note-9)" + ), + ), + ( + "move-footnote-to-wrong-block", "footnote_relation_preservation", + "docx-batch-footnote-relation", + lambda outputs: run_eval._move_line_to_section_once( + outputs, + "docx_reference", + "Applies only to reactor batch RX-31", + "Release order", + ), + ), + ( + "delete-list-item", "list_structure_preservation", + "docx-operator-checklist", + lambda outputs: run_eval._delete_line_once( + outputs, "docx_reference", "Do not bypass the purge interlock." + ), + ), + ( + "delete-document-subtitle", "standalone_text_preservation", + "docx-document-subtitle", + lambda outputs: run_eval._delete_line_once( + outputs, "docx_reference", "Controlled limits and release checks" + ), + ), + ], +) +def test_mutation_is_rejected( + converted_fixture, control_id, metric, row_id, mutator, +): + baseline = run_eval.evaluate_outputs( + converted_fixture["outputs"], include_details=True + ) + assert _detail(baseline, metric, row_id)["passed"] is True + mutated = mutator(copy.deepcopy(converted_fixture["outputs"])) + result = run_eval.evaluate_outputs(mutated, include_details=True) + assert _detail(result, metric, row_id)["passed"] is False, control_id + + +def test_fact_duplicate_is_rejected_even_when_target_block_still_matches(converted_fixture): + sentence = "The sensor tolerance remains ±0.05 mg/L." + mutated = run_eval._mutate_once( + converted_fixture["outputs"], "html_notice", sentence, + sentence + "\n\n" + sentence, + ) + result = run_eval.evaluate_outputs(mutated, include_details=True) + row = _detail(result, "critical_fact_preservation", "html-plus-minus-tolerance") + assert row["document_matches"] == 2 + assert row["passed"] is False + + +def test_all_footnote_relations_are_unique_ordered_and_html_linked(converted_fixture): + result = run_eval.evaluate_outputs( + converted_fixture["outputs"], include_details=True + ) + rows = result["details"]["footnote_relation_preservation"] + assert len(rows) == 3 + assert all(row["marker_document_matches"] == 1 for row in rows) + assert all(row["note_document_matches"] == 1 for row in rows) + assert all(row["marker_before_note"] and row["passed"] for row in rows) + html_row = _detail( + result, + "footnote_relation_preservation", + "html-expiry-footnote-relation", + ) + assert html_row["required_link_pattern"] == r"\[1\]\(#note-1\)" + assert html_row["link_block_matches"] == 1 + assert html_row["link_document_matches"] == 1 + + +def test_reading_order_token_must_be_unique_inside_target_section(converted_fixture): + sentence = "HTML-ORDER-3: release the batch record." + mutated = run_eval._mutate_once( + converted_fixture["outputs"], "html_notice", sentence, + f"{sentence} {sentence}", + ) + result = run_eval.evaluate_outputs(mutated, include_details=True) + row = _detail(result, "reading_order", "html-dom-order") + assert row["section_token_counts"] == [1, 1, 2] + assert row["passed"] is False + + +def test_list_items_require_exact_text_order_and_ordered_unordered_kind(converted_fixture): + baseline = run_eval.evaluate_outputs( + converted_fixture["outputs"], include_details=True + ) + docx = _detail( + baseline, "list_structure_preservation", "docx-operator-checklist" + ) + assert [item["kind"] for item in docx["observed_items"]] == [ + "ordered", "ordered", "unordered" + ] + assert docx["observed_items"] == docx["expected_items"] + + wrong_kind = run_eval._mutate_once( + converted_fixture["outputs"], + "docx_reference", + "* Do not bypass the purge interlock.", + "3. Do not bypass the purge interlock.", + ) + result = run_eval.evaluate_outputs(wrong_kind, include_details=True) + assert _detail( + result, "list_structure_preservation", "docx-operator-checklist" + )["passed"] is False + + +def test_preamble_and_caption_text_are_not_unscored_content(converted_fixture): + result = run_eval.evaluate_outputs( + converted_fixture["outputs"], include_details=True + ) + rows = result["details"]["standalone_text_preservation"] + assert len(rows) == 3 + assert all(row["document_matches"] == 1 and row["passed"] for row in rows) + + deleted = run_eval._delete_line_once( + converted_fixture["outputs"], "docx_reference", + "Compact Reactor Reference Guide", + ) + result = run_eval.evaluate_outputs(deleted, include_details=True) + assert _detail( + result, "standalone_text_preservation", "docx-document-title" + )["passed"] is False + + +def test_wrong_heading_level_and_extra_heading_fail_exact_inventory(converted_fixture): + wrong_level = run_eval._mutate_once( + converted_fixture["outputs"], "html_notice", "## Measurement limits", + "### Measurement limits", + ) + result = run_eval.evaluate_outputs(wrong_level, include_details=True) + assert _detail( + result, "heading_hierarchy_preservation", "html_notice-exact-heading-levels" + )["passed"] is False + + extra = run_eval._insert_extra_heading_once( + converted_fixture["outputs"], "html_notice", "Unexpected appendix" + ) + result = run_eval.evaluate_outputs(extra, include_details=True) + assert _detail( + result, "section_count_preservation", "html_notice-exact-heading-inventory" + )["passed"] is False + assert _detail( + result, "heading_hierarchy_preservation", "html_notice-exact-heading-levels" + )["passed"] is False + + +def test_builtin_mutation_controls_all_have_real_sensitivity(converted_fixture): + result = run_eval.run_mutation_controls(converted_fixture["outputs"]) + assert result["metric"] == { + "numerator": 14, "denominator": 14, "value": 1.0, + "threshold": 1.0, "passed": True, + } + assert all(row["baseline_row_passed"] and not row["mutated_row_passed"] + for row in result["details"]) + + +def test_runner_reports_protocol_smoke_scope_and_not_measured(tmp_path): + result = run_eval.run_evaluation(tmp_path / "runner", include_details=False) + assert result["status"] == "protocol-smoke-passed" + assert result["overall_passed"] is True + assert result["fixture_semantic_sha256"] == fixtures.FROZEN_SEMANTIC_FIXTURE_SHA256 + assert result["not_measured"] == list(run_eval.NOT_MEASURED) + assert result["not_measured"] + assert all(metric["value"] == 1.0 for metric in result["metrics"].values()) + + +def test_missing_markitdown_is_protocol_error_not_skip(monkeypatch): + original = run_eval.importlib.util.find_spec + + def fake_find_spec(name, *args, **kwargs): + return None if name == "markitdown" else original(name, *args, **kwargs) + + monkeypatch.setattr(run_eval.importlib.util, "find_spec", fake_find_spec) + with pytest.raises(run_eval.EvaluationProtocolError, match="markitdown"): + run_eval._require_runtime_dependencies() + + +def test_zero_denominator_fails_closed(): + with pytest.raises(run_eval.EvaluationProtocolError, match="denominator"): + run_eval._metric(0, 0) + + +def test_fixture_runtime_error_is_structured_protocol_error( + monkeypatch, capsys, +): + def fail_materialize(_output_dir): + raise RuntimeError("fixture geometry could not be verified") + + monkeypatch.setattr(run_eval.fixtures, "materialize", fail_materialize) + assert run_eval.main(["--json"]) == 2 + line = capsys.readouterr().out.strip() + assert line.startswith(run_eval.RESULT_PREFIX) + payload = json.loads(line.removeprefix(run_eval.RESULT_PREFIX)) + assert payload["status"] == "protocol-error" + assert payload["overall_passed"] is False + assert "fixture geometry" in payload["error"] + + +def test_fixture_system_exit_zero_is_structured_protocol_error( + monkeypatch, capsys, +): + def exit_materialize(_output_dir): + raise SystemExit(0) + + monkeypatch.setattr(run_eval.fixtures, "materialize", exit_materialize) + assert run_eval.main(["--json"]) == 2 + line = capsys.readouterr().out.strip() + assert line.startswith(run_eval.RESULT_PREFIX) + payload = json.loads(line.removeprefix(run_eval.RESULT_PREFIX)) + assert payload["status"] == "protocol-error" + assert payload["overall_passed"] is False + assert payload["error"] == "0" diff --git a/scripts/tests/test_convert.py b/scripts/tests/test_convert.py new file mode 100644 index 0000000..0fe4c11 --- /dev/null +++ b/scripts/tests/test_convert.py @@ -0,0 +1,924 @@ +"""convert.py 增量判定与 outline 迁移安全性测试。""" +from __future__ import annotations + +import copy +import json +import os +import sys +from pathlib import Path + +import pytest + +import convert +import conversion_receipt +from postprocess import process + + +DOCUMENT = """# Manual + +Opening evidence. + +## Details + +The verified value is 17. +""" + + +def _write_pair( + source: Path, + *, + doc_path: str | None = None, + with_conversion_receipt: bool = True, +): + text, outline = process(DOCUMENT, doc_path or source.name) + if source.suffix.lower() != ".md" and with_conversion_receipt: + outline["conversion_receipt"] = convert._conversion_receipt(source) + target_md = source if source.suffix == ".md" else source.with_suffix(".md") + target_outline = source.with_suffix(".outline.json") + target_md.write_text(text, encoding="utf-8") + target_outline.write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + return text, outline, target_md, target_outline + + +def _rewrite_outline(path: Path, outline: dict) -> None: + path.write_text(json.dumps(outline, ensure_ascii=False), encoding="utf-8") + + +def test_valid_markdown_and_outline_are_skipped(tmp_path: Path): + source = tmp_path / "manual.md" + _write_pair(source) + assert convert.should_convert(source, force=False) is False + + +def test_small_valid_document_does_not_depend_on_three_anchor_heuristic(tmp_path: Path): + source = tmp_path / "small.md" + text, outline = process("# Small\n\nOne paragraph.\n", source.name) + source.write_text(text, encoding="utf-8") + source.with_suffix(".outline.json").write_text( + json.dumps(outline), encoding="utf-8" + ) + # Only two anchors exist, but the complete canonical/hash contract is valid. + assert convert.should_convert(source, force=False) is False + + +@pytest.mark.parametrize( + "mutate", + [ + lambda o: o.pop("outline_schema_version"), + lambda o: o.__setitem__("outline_schema_version", 1), + lambda o: o.pop("doc_sha256"), + lambda o: o.__setitem__("doc_sha256", "0" * 64), + lambda o: o["sections"][0].pop("section_sha256"), + lambda o: o["sections"][0].__setitem__("section_sha256", "0" * 64), + ], +) +def test_schema_or_hash_failure_forces_conversion(tmp_path: Path, mutate): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + outline = copy.deepcopy(original) + mutate(outline) + _rewrite_outline(target_outline, outline) + assert convert.should_convert(source, force=False) is True + + +def test_same_length_markdown_rewrite_forces_conversion(tmp_path: Path): + source = tmp_path / "manual.md" + text, _outline, _target_md, _target_outline = _write_pair(source) + edited = text.replace("value is 17", "value is 18") + assert len(edited) == len(text) + source.write_text(edited, encoding="utf-8") + assert convert.should_convert(source, force=False) is True + + +@pytest.mark.parametrize( + "mutate", + [ + lambda o: o.__setitem__("outline_schema_version", 1), + lambda o: o.__setitem__("doc_sha256", "0" * 64), + lambda o: o["sections"][0].__setitem__("section_sha256", "0" * 64), + ], +) +def test_old_source_mtime_cannot_hide_invalid_outline(tmp_path: Path, mutate): + source = tmp_path / "manual.pdf" + source.write_bytes(b"fictional-pdf-fixture") + _text, original, target_md, target_outline = _write_pair( + source, doc_path="manual.md" + ) + + # A valid pair is skipped when the original is older than its derivative. + os.utime(source, (1, 1)) + os.utime(target_md, (2, 2)) + assert convert.should_convert(source, force=False) is False + + invalid = copy.deepcopy(original) + mutate(invalid) + _rewrite_outline(target_outline, invalid) + os.utime(target_outline, (2, 2)) + assert source.stat().st_mtime < target_md.stat().st_mtime + assert convert.should_convert(source, force=False) is True + + +def test_legacy_pdf_without_conversion_receipt_is_reconverted_even_when_older( + tmp_path: Path, +): + source = tmp_path / "legacy.pdf" + source.write_bytes(b"legacy-generic-converter-source") + _text, _outline, target_md, _target_outline = _write_pair( + source, + doc_path="legacy.md", + with_conversion_receipt=False, + ) + os.utime(source, (1, 1)) + os.utime(target_md, (2, 2)) + assert convert.should_convert(source, force=False) is True + + +def test_pdf_receipt_binds_converter_fingerprint_and_source_bytes(tmp_path: Path): + source = tmp_path / "current.pdf" + source.write_bytes(b"current-source-v1") + _text, original, target_md, target_outline = _write_pair( + source, doc_path="current.md" + ) + os.utime(source, (1, 1)) + os.utime(target_md, (2, 2)) + assert convert.validate_conversion_receipt(original, source) == [] + assert convert.should_convert(source, force=False) is False + + # A timestamp-only touch must not create a permanent reconversion loop; + # the source hash receipt is the authoritative freshness proof. + os.utime(source, (3, 3)) + assert source.stat().st_mtime > target_md.stat().st_mtime + assert convert.should_convert(source, force=False) is False + + stale_converter = copy.deepcopy(original) + stale_converter["conversion_receipt"]["converter_fingerprint"] = "0" * 64 + _rewrite_outline(target_outline, stale_converter) + assert convert.should_convert(source, force=False) is True + + wrong_schema_type = copy.deepcopy(original) + wrong_schema_type["conversion_receipt"]["schema_version"] = True + _rewrite_outline(target_outline, wrong_schema_type) + assert convert.should_convert(source, force=False) is True + + unexpected_field = copy.deepcopy(original) + unexpected_field["conversion_receipt"]["unregistered"] = "must fail closed" + _rewrite_outline(target_outline, unexpected_field) + assert convert.should_convert(source, force=False) is True + + _rewrite_outline(target_outline, original) + source.write_bytes(b"current-source-v2") + os.utime(source, (1, 1)) + assert source.stat().st_mtime < target_md.stat().st_mtime + assert convert.should_convert(source, force=False) is True + + +@pytest.mark.parametrize( + "suffix,dependency", + [ + (".pdf", "pdfminer.six"), + (".docx", "mammoth"), + (".html", "markdownify"), + (".xlsx", "openpyxl"), + ], +) +def test_backend_dependency_version_change_invalidates_receipt( + tmp_path: Path, monkeypatch, suffix: str, dependency: str, +): + source = tmp_path / f"backend{suffix}" + source.write_bytes(b"stable-source") + _write_pair(source, doc_path=f"backend{suffix}.md") + original = conversion_receipt._package_version + + def changed_version(name: str) -> str: + return "changed-for-test" if name == dependency else original(name) + + monkeypatch.setattr(conversion_receipt, "_package_version", changed_version) + assert convert.should_convert(source, force=False) is True + + +def test_missing_required_backend_metadata_fails_closed(monkeypatch): + original = conversion_receipt.importlib.metadata.version + + def missing_mammoth(name: str) -> str: + if name == "mammoth": + raise conversion_receipt.importlib.metadata.PackageNotFoundError(name) + return original(name) + + monkeypatch.setattr(conversion_receipt.importlib.metadata, "version", missing_mammoth) + with pytest.raises(conversion_receipt.ConversionReceiptError) as caught: + conversion_receipt.current_converter_fingerprint(".docx") + assert caught.value.code == "converter-fingerprint-unavailable" + assert caught.value.details["distribution"] == "mammoth" + + +def test_backend_versions_are_scoped_to_the_source_format(monkeypatch): + before_docx = conversion_receipt.current_converter_fingerprint(".docx") + before_xlsx = conversion_receipt.current_converter_fingerprint(".xlsx") + original = conversion_receipt._package_version + + def changed_openpyxl(name: str) -> str: + return "changed-for-test" if name == "openpyxl" else original(name) + + monkeypatch.setattr(conversion_receipt, "_package_version", changed_openpyxl) + assert conversion_receipt.current_converter_fingerprint(".docx") == before_docx + assert conversion_receipt.current_converter_fingerprint(".xlsx") != before_xlsx + + +def test_converter_implementation_hash_change_invalidates_receipt( + tmp_path: Path, monkeypatch, +): + source = tmp_path / "implementation.docx" + source.write_bytes(b"stable-source") + _write_pair(source, doc_path="implementation.md") + hashes = conversion_receipt._implementation_hashes( + ".docx", Path(convert.__file__).resolve().parent + ) + assert set(hashes) == { + "conversion_receipt.py", "convert.py", "postprocess.py", "section_parser.py", + } + assert all(len(value) == 64 for value in hashes.values()) + + monkeypatch.setattr( + conversion_receipt, + "_implementation_hashes", + lambda _suffix, _scripts_dir: { + "conversion_receipt.py": "0" * 64, + "convert.py": "1" * 64, + "postprocess.py": "2" * 64, + "section_parser.py": "3" * 64, + }, + ) + assert convert.should_convert(source, force=False) is True + + +def test_python_runtime_change_invalidates_conversion_receipt( + tmp_path: Path, monkeypatch, +): + source = tmp_path / "runtime.docx" + source.write_bytes(b"stable-source") + _write_pair(source, doc_path="runtime.md") + + monkeypatch.setattr(conversion_receipt.platform, "python_version", lambda: "0.0.0") + assert convert.should_convert(source, force=False) is True + + +@pytest.mark.parametrize( + "suffix", + sorted(convert.STANDALONE_IMAGE_EXTENSIONS | convert.AUDIO_EXTENSIONS), +) +def test_unverifiable_standalone_media_fails_before_markitdown_and_never_stays_fresh( + tmp_path: Path, suffix: str, +): + source = tmp_path / f"media{suffix}" + source.write_bytes(b"synthetic-media-source") + + class UnexpectedConverter: + called = False + + def convert(self, _path: str): + self.called = True + raise AssertionError("accuracy-first rejection must precede MarkItDown") + + converter = UnexpectedConverter() + expected = "standalone images" if suffix in convert.STANDALONE_IMAGE_EXTENSIONS else "audio" + with pytest.raises(RuntimeError, match=expected): + convert._convert_non_markdown_source(converter, source) + assert converter.called is False + assert not source.with_suffix(".md").exists() + assert not source.with_suffix(".outline.json").exists() + + with pytest.raises(conversion_receipt.ConversionReceiptError) as caught: + convert._conversion_receipt(source) + assert caught.value.code == "conversion-source-extension-unsupported" + issues = convert.validate_conversion_receipt({"conversion_receipt": {}}, source) + assert [issue["code"] for issue in issues] == ["conversion-backend-unsupported"] + + +@pytest.mark.parametrize( + "suffix", sorted(conversion_receipt.SUCCESSFUL_CONVERSION_SOURCE_EXTENSIONS) +) +def test_convert_receipt_matches_shared_contract_for_every_supported_suffix( + tmp_path: Path, suffix: str, +): + source = tmp_path / f"source{suffix}" + source.write_bytes(b"shared-contract-source") + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path.resolve() + try: + actual = convert._conversion_receipt(source) + finally: + convert._BASE_ROOT = old_base + expected = conversion_receipt.build_conversion_receipt(source, tmp_path) + assert actual == expected + + +def test_receipt_binds_canonical_source_path_and_self_hash(tmp_path: Path): + source = tmp_path / "manual.docx" + source.write_bytes(b"source") + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + receipt = convert._conversion_receipt(source) + finally: + convert._BASE_ROOT = old_base + assert receipt["schema_version"] == 2 + assert receipt["source_path"] == "manual.docx" + assert receipt["source_extension"] == ".docx" + assert convert._receipt_integrity_valid(receipt) is True + receipt["source_path"] = "other.docx" + assert convert._receipt_integrity_valid(receipt) is False + + +@pytest.mark.parametrize( + ("source_path", "extension"), + [ + ("/absolute.docx", ".docx"), + ("raw\\manual.docx", ".docx"), + ("raw//manual.docx", ".docx"), + ("raw/./manual.docx", ".docx"), + ("raw/../manual.docx", ".docx"), + ("raw/manual.docx/", ".docx"), + ("raw/cafe\u0301.docx", ".docx"), + ("raw/manual.DOCX", ".DOCX"), + ("raw/manual.docx", ".pdf"), + ], +) +def test_receipt_rejects_noncanonical_source_coordinates( + tmp_path: Path, source_path: str, extension: str, +): + source = tmp_path / "manual.docx" + source.write_bytes(b"source") + receipt = convert._conversion_receipt(source) + receipt["source_path"] = source_path + receipt["source_extension"] = extension + payload = { + key: receipt[key] + for key in ( + "schema_version", + "source_path", + "source_extension", + "source_sha256", + "converter_fingerprint", + ) + } + import hashlib + + receipt["receipt_sha256"] = hashlib.sha256( + json.dumps( + payload, ensure_ascii=True, sort_keys=True, separators=(",", ":") + ).encode("utf-8") + ).hexdigest() + assert convert._receipt_integrity_valid(receipt) is False + + +def test_direct_conversion_never_overwrites_native_markdown_collision(tmp_path: Path): + source = tmp_path / "same.docx" + source.write_bytes(b"docx-source") + native = tmp_path / "same.md" + native.write_text("# ORIGINAL MARKDOWN\n", encoding="utf-8") + + class Converter: + called = False + + def convert(self, _path): + self.called = True + raise AssertionError("target ownership must be checked first") + + converter = Converter() + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + with pytest.raises(RuntimeError, match="incomplete|ownership receipt"): + convert.convert_file(converter, source) + finally: + convert._BASE_ROOT = old_base + assert converter.called is False + assert native.read_text(encoding="utf-8") == "# ORIGINAL MARKDOWN\n" + assert not source.with_suffix(".outline.json").exists() + + +def test_existing_owned_derivative_can_be_reconverted(tmp_path: Path): + source = tmp_path / "owned.docx" + source.write_bytes(b"source-v1") + + class ResultOne: + markdown = "# Owned\n\nDerivative version one.\n" + + class ResultTwo: + markdown = "# Owned\n\nDerivative version two.\n" + + class Converter: + def __init__(self, result): + self.result = result + + def convert(self, _path): + return self.result + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + assert convert.convert_file(Converter(ResultOne()), source)[0] is True + source.write_bytes(b"source-v2") + assert convert.convert_file(Converter(ResultTwo()), source)[0] is True + outline = json.loads( + source.with_suffix(".outline.json").read_text(encoding="utf-8") + ) + assert convert.validate_conversion_receipt(outline, source) == [] + finally: + convert._BASE_ROOT = old_base + assert "Derivative version two." in source.with_suffix(".md").read_text( + encoding="utf-8" + ) + + +def test_batch_media_rejection_happens_before_backend_init_or_partial_writes( + tmp_path: Path, monkeypatch, +): + raw = tmp_path / "raw" + raw.mkdir() + (raw / "blocked.png").write_bytes(b"image") + native = raw / "native.md" + native.write_text("# Native\n", encoding="utf-8") + constructor_calls = 0 + + class ForbiddenMarkItDown: + def __init__(self): + nonlocal constructor_calls + constructor_calls += 1 + raise AssertionError("preflight must precede backend construction") + + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr(convert, "MarkItDown", ForbiddenMarkItDown) + monkeypatch.setattr( + sys, "argv", ["convert.py", "--dir", str(raw), "--force"] + ) + with pytest.raises(SystemExit) as error: + convert.main() + assert error.value.code == 1 + assert constructor_calls == 0 + assert native.read_text(encoding="utf-8") == "# Native\n" + assert not native.with_suffix(".outline.json").exists() + + +@pytest.mark.parametrize( + "names", + [ + ("same.md", "same.docx"), + ("SAME.MD", "same.docx"), + ("same.docx", "same.pdf"), + ], +) +def test_batch_target_collisions_fail_atomically_before_backend_init( + tmp_path: Path, monkeypatch, names: tuple[str, str], +): + raw = tmp_path / "raw" + raw.mkdir() + originals: dict[Path, bytes] = {} + for name in names: + path = raw / name + content = b"# ORIGINAL MARKDOWN\n" if name.lower().endswith(".md") else b"source" + path.write_bytes(content) + originals[path] = content + constructor_calls = 0 + + class ForbiddenMarkItDown: + def __init__(self): + nonlocal constructor_calls + constructor_calls += 1 + raise AssertionError("collision preflight must run first") + + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr(convert, "MarkItDown", ForbiddenMarkItDown) + monkeypatch.setattr( + sys, "argv", ["convert.py", "--dir", str(raw), "--force"] + ) + with pytest.raises(SystemExit) as error: + convert.main() + assert error.value.code == 1 + assert constructor_calls == 0 + assert all(path.read_bytes() == content for path, content in originals.items()) + assert not (raw / "same.outline.json").exists() + + +def test_force_batch_twice_filters_owned_markdown_derivative(tmp_path: Path, monkeypatch): + raw = tmp_path / "raw" + raw.mkdir() + source = raw / "manual.docx" + source.write_bytes(b"source") + convert_calls = 0 + + class Result: + markdown = "# Manual\n\nOwned derivative.\n" + + class FakeMarkItDown: + def convert(self, _path): + nonlocal convert_calls + convert_calls += 1 + return Result() + + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr(convert, "MarkItDown", FakeMarkItDown) + monkeypatch.setattr( + sys, "argv", ["convert.py", "--dir", str(raw), "--force"] + ) + convert.main() + convert.main() + assert convert_calls == 2 + outline = json.loads( + source.with_suffix(".outline.json").read_text(encoding="utf-8") + ) + assert outline["conversion_receipt"]["source_path"] == "raw/manual.docx" + assert convert.validate_conversion_receipt(outline, source) == [] + + +def test_orphan_derivative_is_not_reinterpreted_as_native_markdown( + tmp_path: Path, monkeypatch, +): + raw = tmp_path / "raw" + raw.mkdir() + source = raw / "carrier.docx" + source.write_bytes(b"source") + constructor_calls = 0 + + class Result: + markdown = "# Carrier\n\nOwned derivative.\n" + + class FakeMarkItDown: + def __init__(self): + nonlocal constructor_calls + constructor_calls += 1 + + def convert(self, _path): + return Result() + + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr(convert, "MarkItDown", FakeMarkItDown) + monkeypatch.setattr( + sys, "argv", ["convert.py", "--dir", str(raw), "--force"] + ) + convert.main() + derivative = source.with_suffix(".md") + source.unlink() + assert convert.should_convert(derivative, force=False) is True + with pytest.raises(SystemExit) as error: + convert.main() + assert error.value.code == 1 + assert constructor_calls == 1 + assert derivative.exists() + + +def test_direct_conversion_rejects_owned_derivative_markdown(tmp_path: Path): + source = tmp_path / "owner.docx" + source.write_bytes(b"source") + + class Result: + markdown = "# Owner\n\nDerivative.\n" + + class Converter: + def convert(self, _path): + return Result() + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + convert.convert_file(Converter(), source) + with pytest.raises(RuntimeError, match="owned derivative Markdown"): + convert.convert_file(object(), source.with_suffix(".md")) + finally: + convert._BASE_ROOT = old_base + + +def test_source_symlink_is_rejected_without_replacing_link_or_external_file( + tmp_path: Path, monkeypatch, +): + raw = tmp_path / "raw" + raw.mkdir() + external = tmp_path / "external.md" + external.write_text("# External original\n", encoding="utf-8") + link = raw / "note.md" + link.symlink_to(external) + constructor_calls = 0 + + class ForbiddenMarkItDown: + def __init__(self): + nonlocal constructor_calls + constructor_calls += 1 + + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr(convert, "MarkItDown", ForbiddenMarkItDown) + monkeypatch.setattr( + sys, "argv", ["convert.py", "--dir", str(raw), "--force"] + ) + with pytest.raises(SystemExit) as error: + convert.main() + assert error.value.code == 1 + assert constructor_calls == 0 + assert link.is_symlink() + assert external.read_text(encoding="utf-8") == "# External original\n" + + +def test_outline_in_stem_is_not_mistaken_for_outline_artifact(tmp_path: Path): + source = tmp_path / "project.outline.pdf" + source.write_bytes(b"source") + assert source in convert.collect_files(tmp_path, None) + + +def test_structural_corruption_and_invalid_json_force_conversion(tmp_path: Path): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + + duplicate = copy.deepcopy(original) + duplicate["sections"][0]["children"][0]["anchor"] = duplicate["sections"][0]["anchor"] + _rewrite_outline(target_outline, duplicate) + assert convert.should_convert(source, force=False) is True + + target_outline.write_text("{not-json", encoding="utf-8") + assert convert.should_convert(source, force=False) is True + + +def test_structurally_corrupt_outline_can_be_repaired_without_summary_crash(tmp_path: Path): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + corrupt = copy.deepcopy(original) + corrupt["sections"][0]["agent_summary"] = "candidate summary" + corrupt["sections"][0]["children"] = "not-a-list" + _rewrite_outline(target_outline, corrupt) + + assert convert.should_convert(source, force=False) is True + ok, _message = convert.convert_file(object(), source) + assert ok is True + rebuilt = json.loads(target_outline.read_text(encoding="utf-8")) + assert isinstance(rebuilt["sections"][0]["children"], list) + + +def test_summary_with_wrong_section_hash_is_not_migrated(tmp_path: Path): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + invalid = copy.deepcopy(original) + invalid["sections"][0]["agent_summary"] = "untrusted stale summary" + invalid["sections"][0]["section_sha256"] = "0" * 64 + _rewrite_outline(target_outline, invalid) + + ok, _message = convert.convert_file(object(), source) + assert ok is True + rebuilt = json.loads(target_outline.read_text(encoding="utf-8")) + assert rebuilt["sections"][0]["agent_summary"] is None + + +def test_valid_previous_outline_summary_is_preserved(tmp_path: Path): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + valid = copy.deepcopy(original) + valid["sections"][0]["agent_summary"] = "verified summary" + _rewrite_outline(target_outline, valid) + + ok, _message = convert.convert_file(object(), source) + assert ok is True + rebuilt = json.loads(target_outline.read_text(encoding="utf-8")) + assert rebuilt["sections"][0]["agent_summary"] == "verified summary" + + +def test_doc_hash_migration_preserves_only_section_hash_bound_summary(tmp_path: Path): + source = tmp_path / "manual.md" + _text, original, _target_md, target_outline = _write_pair(source) + candidate = copy.deepcopy(original) + candidate["sections"][0]["agent_summary"] = "section-bound summary" + candidate["doc_sha256"] = "0" * 64 + _rewrite_outline(target_outline, candidate) + + assert convert.should_convert(source, force=False) is True + ok, _message = convert.convert_file(object(), source) + assert ok is True + rebuilt = json.loads(target_outline.read_text(encoding="utf-8")) + assert rebuilt["sections"][0]["agent_summary"] == "section-bound summary" + + +def test_docx_blank_table_header_promotes_wholly_bold_first_row(tmp_path: Path): + source = tmp_path / "matrix.docx" + source.write_bytes(b"synthetic-source-placeholder") + + class Result: + markdown = """# Matrix + +| | | | +| --- | --- | --- | +| **Model** | **Limit** | **Condition** | +| Cobalt-X | -0.45 kPa | purge excluded | +""" + + class Converter: + def convert(self, _path): + return Result() + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + ok, _message = convert.convert_file(Converter(), source) + finally: + convert._BASE_ROOT = old_base + assert ok is True + rendered = source.with_suffix(".md").read_text(encoding="utf-8") + assert "| Model | Limit | Condition |" in rendered + assert "| **Model** | **Limit** | **Condition** |" not in rendered + assert "| Cobalt-X | -0.45 kPa | purge excluded |" in rendered + + +def test_docx_table_without_bold_header_is_not_guessed(): + markdown = """| | | +| --- | --- | +| value-a | value-b | +""" + assert convert._promote_docx_bold_table_headers(markdown) == markdown + + +def test_docx_partially_bold_cells_are_not_promoted_as_wholly_bold_headers(): + markdown = """| | | +| --- | --- | +| **Model** legacy **ID** | **Limit** | +| Cobalt-X | -0.45 kPa | +""" + assert convert._promote_docx_bold_table_headers(markdown) == markdown + + +def test_docx_multiple_adjacent_bold_runs_are_still_wholly_bold(): + markdown = """| | | +| --- | --- | +| **Model** **ID** | __Limit__ | +| Cobalt-X | -0.45 kPa | +""" + promoted = convert._promote_docx_bold_table_headers(markdown) + assert "| Model ID | Limit |" in promoted + assert "| **Model** **ID** | __Limit__ |" not in promoted + + +def test_non_markdown_converter_reads_read_only_snapshot_and_binds_live_source( + tmp_path: Path, +): + source = tmp_path / "stable.docx" + source.write_bytes(b"stable-source-v1") + observed: dict[str, Path] = {} + + class Result: + markdown = "# Stable\n\nSnapshot-bound content.\n" + + class Converter: + def convert(self, path): + snapshot = Path(path) + observed["path"] = snapshot + assert snapshot != source + assert snapshot.name == source.name + assert snapshot.read_bytes() == b"stable-source-v1" + assert snapshot.stat().st_mode & 0o222 == 0 + with pytest.raises(PermissionError): + snapshot.write_bytes(b"accidental-mutation") + return Result() + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + ok, _message = convert.convert_file(Converter(), source) + finally: + convert._BASE_ROOT = old_base + assert ok is True + assert observed["path"] != source + outline = json.loads( + source.with_suffix(".outline.json").read_text(encoding="utf-8") + ) + assert convert.validate_conversion_receipt(outline, source) == [] + + +def test_source_change_during_snapshot_conversion_fails_before_derivative_write( + tmp_path: Path, +): + source = tmp_path / "changing.docx" + source.write_bytes(b"source-version-one") + + class Result: + markdown = "# Unsafe\n\nMust not be written.\n" + + class MutatingConverter: + def convert(self, path): + snapshot = Path(path) + assert snapshot.read_bytes() == b"source-version-one" + source.write_bytes(b"source-version-two") + return Result() + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + with pytest.raises(RuntimeError, match="source changed during conversion"): + convert.convert_file(MutatingConverter(), source) + finally: + convert._BASE_ROOT = old_base + assert not source.with_suffix(".md").exists() + assert not source.with_suffix(".outline.json").exists() + + +def test_converter_snapshot_mutation_is_detected_by_post_conversion_hash( + tmp_path: Path, +): + source = tmp_path / "snapshot-mutation.docx" + source.write_bytes(b"immutable-source") + + class Result: + markdown = "# Unsafe\n\nMutated snapshot output.\n" + + class SnapshotMutatingConverter: + def convert(self, path): + snapshot = Path(path) + snapshot.chmod(0o644) + snapshot.write_bytes(b"mutated-snapshot") + return Result() + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + with pytest.raises(RuntimeError, match="snapshot changed during conversion"): + convert.convert_file(SnapshotMutatingConverter(), source) + finally: + convert._BASE_ROOT = old_base + assert source.read_bytes() == b"immutable-source" + assert not source.with_suffix(".md").exists() + assert not source.with_suffix(".outline.json").exists() + + +def test_batch_conversion_returns_nonzero_when_any_pdf_fails( + tmp_path: Path, monkeypatch, +): + raw = tmp_path / "raw" + raw.mkdir() + (raw / "broken.pdf").write_bytes(b"not-a-pdf") + monkeypatch.setenv("KB_ROOT", str(tmp_path)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr( + sys, + "argv", + ["convert.py", "--dir", str(raw), "--force", "--ext", ".pdf"], + ) + with pytest.raises(SystemExit) as error: + convert.main() + assert error.value.code == 1 + + +def test_explicit_dir_base_is_workspace_aware_and_preserves_other_semantics(tmp_path: Path): + data_root = tmp_path / "kb-data" + workspace = data_root / "workspaces" / "research" + inside = workspace / "raw" / "papers" + outside = data_root / "imports" / "papers" + workspaces_root = data_root / "workspaces" + inside.mkdir(parents=True) + outside.mkdir(parents=True) + + assert convert._doc_path_base_for_explicit_dir(inside, data_root) == workspace + assert convert._doc_path_base_for_explicit_dir(outside, data_root) == data_root + assert convert._doc_path_base_for_explicit_dir(workspaces_root, data_root) == data_root + + +def test_explicit_workspace_dir_converts_to_k_canonical_outline_and_annotation( + tmp_path: Path, + monkeypatch, +): + import k + + data_root = tmp_path / "kb-data" + workspace = data_root / "workspaces" / "research" + scan_dir = workspace / "raw" / "papers" + scan_dir.mkdir(parents=True) + source = scan_dir / "manual.md" + source.write_text(DOCUMENT, encoding="utf-8") + + monkeypatch.setenv("KB_ROOT", str(data_root)) + monkeypatch.setattr(convert, "_BASE_ROOT", None) + monkeypatch.setattr( + sys, + "argv", + ["convert.py", "--dir", str(scan_dir)], + ) + convert.main() + + outline_path = source.with_suffix(".outline.json") + generated = json.loads(outline_path.read_text(encoding="utf-8")) + assert generated["doc_path"] == "raw/papers/manual.md" + assert convert.should_convert(source, force=False) is False + + # Prove k.py accepts the on-disk outline instead of rebuilding it and + # silently dropping its only non-reconstructible field, agent_summary. + generated["sections"][0]["agent_summary"] = "persisted sentinel" + _rewrite_outline(outline_path, generated) + monkeypatch.setattr(k, "PROJECT_ROOT", workspace) + monkeypatch.setattr(k, "WIKI_DIR", workspace / "wiki") + monkeypatch.setattr(k, "RAW_DIR", workspace / "raw") + + loaded = k.load_or_build_outline(source) + assert loaded["sections"][0]["agent_summary"] == "persisted sentinel" + anchor = loaded["sections"][0]["anchor"] + k.annotate_section(source, anchor, "updated through k") + reloaded = k.load_or_build_outline(source) + assert reloaded["sections"][0]["agent_summary"] == "updated through k" diff --git a/scripts/tests/test_holdout_eval.py b/scripts/tests/test_holdout_eval.py new file mode 100644 index 0000000..51367c0 --- /dev/null +++ b/scripts/tests/test_holdout_eval.py @@ -0,0 +1,442 @@ +"""Fail-closed external holdout protocol and public smoke regression tests.""" +from __future__ import annotations + +import copy +import hashlib +import json +import sys +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +SCRIPTS = PROJECT_ROOT / "scripts" +if str(SCRIPTS) not in sys.path: + sys.path.insert(0, str(SCRIPTS)) + +from evals.holdout import protocol # noqa: E402 +from evals.holdout import run_eval # noqa: E402 +from evals.holdout.generate_public_smoke import build_public_smoke_bundle # noqa: E402 +import retrieval_index # noqa: E402 + + +class RecordingApi: + def __init__(self): + self.search_calls: list[tuple[str, int, list[str]]] = [] + + def rebuild_index(self, workspace_root: Path, db_path: Path | None = None): + return retrieval_index.rebuild_index(workspace_root, db_path=db_path) + + def coverage_report(self, db_path: Path): + return retrieval_index.coverage_report(db_path) + + def search_evidence(self, db_path: Path, query: str, limit: int = 20, + expansions=None): + self.search_calls.append((query, limit, list(expansions or []))) + return retrieval_index.search_evidence( + db_path, query, limit=limit, expansions=expansions, + ) + + +@pytest.fixture(scope="module") +def smoke_file(tmp_path_factory): + root = tmp_path_factory.mktemp("external-holdout-bundle") + path = root / "public-smoke.json" + bundle = build_public_smoke_bundle() + digest = protocol.write_bundle(bundle, path) + return path, digest, bundle + + +@pytest.fixture(scope="module") +def smoke_run(smoke_file, tmp_path_factory): + path, digest, bundle = smoke_file + api = RecordingApi() + work = tmp_path_factory.mktemp("external-holdout-run") + result = run_eval.evaluate( + api, path, digest, work, run_id="pytest-public-smoke", seed=74021, + ) + return result, api, work, bundle + + +class TestExternalBundleSchema: + def test_public_bundle_is_deterministic_cc0_and_explicitly_not_hidden(self, tmp_path): + first = build_public_smoke_bundle() + second = build_public_smoke_bundle() + assert protocol.canonical_json(first) == protocol.canonical_json(second) + assert first["license"] == "CC0-1.0" + assert first["certification_kind"] == "public-smoke" + assert first["schema_version"] == protocol.SCHEMA_VERSION + assert set(first["preregistration"]["required_slices"]) == { + "poison", "unanswerable", "multi_hop", "multilingual", + } + assert all(first["preregistration"]["denominators"]["slice_cases"][name] > 0 + for name in protocol.REQUIRED_SLICES) + protocol.validate_bundle(first) + + first_path = tmp_path / "a.json" + second_path = tmp_path / "b.json" + assert protocol.write_bundle(first, first_path) == protocol.write_bundle( + second, second_path, + ) + assert first_path.read_bytes() == second_path.read_bytes() + + def test_machine_schema_is_valid_json_and_matches_runtime_version(self): + schema = json.loads( + (PROJECT_ROOT / "evals" / "holdout" / "bundle.schema.json").read_text() + ) + assert schema["properties"]["schema_version"]["const"] == protocol.SCHEMA_VERSION + assert schema["properties"]["preregistration"]["properties"]["top_k"]["const"] == 20 + + def test_complete_file_sha_is_mandatory_trust_anchor(self, smoke_file, tmp_path): + path, digest, _bundle = smoke_file + changed = tmp_path / "whitespace-mutated.json" + changed.write_bytes(path.read_bytes() + b" \n") + with pytest.raises(protocol.HoldoutProtocolError, match="external bundle SHA-256 mismatch"): + protocol.load_bundle(changed, digest) + with pytest.raises(protocol.HoldoutProtocolError, match="must be 64 lowercase"): + protocol.load_bundle(path, "not-a-commitment") + + def test_deleted_case_cannot_be_resigned_under_old_commitment(self, smoke_file, tmp_path): + _path, committed_sha, original = smoke_file + mutant = copy.deepcopy(original) + victim = next(case for case in mutant["cases"] if case["case_id"] == "tech-01") + mutant["cases"].remove(victim) + den = mutant["preregistration"]["denominators"] + den["cases"] -= 1 + den["unique_questions"] -= 1 + den["answerable_cases"] -= 1 + den["facets"] -= len(victim["required_facets"]) + for slice_name in victim["slices"]: + den["slice_cases"][slice_name] -= 1 + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + protocol.validate_bundle(mutant) # internally coherent attacker rewrite + mutant_path = tmp_path / "deleted-case.json" + protocol.write_bundle(mutant, mutant_path) + assert protocol.file_sha256(mutant_path) != committed_sha + with pytest.raises(protocol.HoldoutProtocolError, match="external bundle SHA-256 mismatch"): + protocol.load_bundle(mutant_path, committed_sha) + + def test_changed_denominator_fails_even_with_recomputed_internal_digest(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + mutant["preregistration"]["denominators"]["natural_units"] -= 1 + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="denominators do not match"): + protocol.validate_bundle(mutant) + + def test_missing_denominator_and_missing_slice_fail_closed(self, smoke_file): + _path, _digest, original = smoke_file + missing_denominator = copy.deepcopy(original) + del missing_denominator["preregistration"]["denominators"]["facets"] + missing_denominator["payload_sha256"] = protocol.payload_sha256( + missing_denominator + ) + with pytest.raises(protocol.HoldoutProtocolError, match="denominators do not match"): + protocol.validate_bundle(missing_denominator) + + missing_slice = copy.deepcopy(original) + for case in missing_slice["cases"]: + case["slices"] = [name for name in case["slices"] if name != "multilingual"] + missing_slice["preregistration"]["denominators"]["slice_cases"]["multilingual"] = 0 + missing_slice["payload_sha256"] = protocol.payload_sha256(missing_slice) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(missing_slice) + + def test_public_smoke_cannot_be_relabelled_as_hidden_certification(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + mutant["certification_kind"] = "hidden" + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="at least 200"): + protocol.validate_bundle(mutant) + + def test_dangling_or_nonexact_evidence_oracle_fails_closed(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + logical_id, evidence = next(iter(mutant["oracle"]["evidence"].items())) + evidence["subordinal"] += 1 + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="disagrees with inventory"): + protocol.validate_bundle(mutant) + + def test_normalized_questions_are_unique_and_200_duplicates_cannot_certify( + self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + templates = [ + next(case for case in original["cases"] if case["case_id"] == "tech-01"), + next(case for case in original["cases"] if case["case_id"] == "tech-02"), + ] + cases = [] + for index in range(200): + case = copy.deepcopy(templates[index % 2]) + case["case_id"] = f"inflated-{index:03d}" + # NFKC/case/whitespace normalization must not create a loophole. + if index >= 2: + case["question"] = " " + case["question"].swapcase() + " " + cases.append(case) + mutant["cases"] = cases + mutant["certification_kind"] = "hidden" + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + assert len(mutant["cases"]) == 200 + assert len({protocol.normalize_question(case["question"]) + for case in mutant["cases"]}) == 2 + assert protocol.normalize_question("Ａlpha\u200b question?!") == ( + protocol.normalize_question("alpha question") + ) + with pytest.raises(protocol.HoldoutProtocolError, match="normalized question duplicates"): + protocol.validate_bundle(mutant) + + def test_slice_labels_must_equal_mechanical_eligibility(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + for case in mutant["cases"]: + case["slices"] = list(protocol.REQUIRED_SLICES) + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels.*eligibility"): + protocol.validate_bundle(mutant) + + def test_poison_slice_is_exactly_equivalent_to_nonempty_forbidden_ids(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + target = next(case for case in mutant["cases"] if case["case_id"] == "tech-01") + target["forbidden_evidence_ids"] = [] + target["slices"].remove("poison") + mutant["preregistration"]["denominators"]["slice_cases"]["poison"] -= 1 + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + protocol.validate_bundle(mutant) + + mismatched = copy.deepcopy(mutant) + mismatched_target = next(case for case in mismatched["cases"] + if case["case_id"] == "tech-01") + mismatched_target["slices"].append("poison") + mismatched["payload_sha256"] = protocol.payload_sha256(mismatched) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(mismatched) + + def test_fake_multihop_and_multilingual_labels_fail_closed(self, smoke_file): + _path, _digest, original = smoke_file + simple = next(case for case in original["cases"] if case["case_id"] == "tech-01") + for fake_slice in ("multi_hop", "multilingual", "unanswerable"): + mutant = copy.deepcopy(original) + target = next(case for case in mutant["cases"] + if case["case_id"] == simple["case_id"]) + target["slices"].append(fake_slice) + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(mutant) + + def test_multihop_requires_two_facets_and_cross_document_complete_set(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + target = next(case for case in mutant["cases"] if case["case_id"] == "tech-01") + foreign_id = "MH-001-RULE" + target["required_facets"][0]["acceptable_evidence_ids"].append(foreign_id) + target["minimal_evidence_sets"] = [["TM-001", foreign_id]] + target["slices"].append("multi_hop") + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(mutant) + + def test_multilingual_requires_two_allowed_records_with_latin_and_cjk(self, smoke_file): + _path, _digest, original = smoke_file + mutant = copy.deepcopy(original) + target = next(case for case in mutant["cases"] if case["case_id"] == "bilingual-01") + target["required_facets"][0]["acceptable_evidence_ids"] = ["BI-001-EN"] + target["minimal_evidence_sets"] = [["BI-001-EN"]] + mutant["payload_sha256"] = protocol.payload_sha256(mutant) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(mutant) + + two_latin = copy.deepcopy(original) + target = next(case for case in two_latin["cases"] + if case["case_id"] == "bilingual-01") + target["required_facets"][0]["acceptable_evidence_ids"] = [ + "BI-001-EN", "TM-001", + ] + target["minimal_evidence_sets"] = [["BI-001-EN", "TM-001"]] + two_latin["payload_sha256"] = protocol.payload_sha256(two_latin) + with pytest.raises(protocol.HoldoutProtocolError, match="slice labels"): + protocol.validate_bundle(two_latin) + + +class TestHoldoutExecution: + def test_public_smoke_passes_exact_oracles_and_reports_honest_scope(self, smoke_run): + result, api, _work, bundle = smoke_run + assert result["status"] == "protocol-smoke-passed" + assert result["passed"] is True + assert result["certification_kind"] == "public-smoke" + assert result["is_hidden_certification"] is False + assert result["runner_verified_independence"] is False + assert "never hidden certification" in result["certification_statement"] + assert result["rotation"]["independent_distribution"] is False + assert result["metrics"]["natural_unit_index_coverage"]["value"] == 1.0 + assert result["metrics"]["content_section_registration_coverage"]["value"] == 1.0 + assert result["metrics"]["complete_evidence_set_recall_at_20"]["value"] >= 0.98 + assert result["metrics"]["selected_citation_precision"]["value"] >= 0.99 + assert result["metrics"]["forbidden_selected_rate"]["value"] == 0.0 + assert result["thresholds"]["forbidden_selected_rate"] == { + "threshold": 0.0, "value": 0.0, "direction": "maximum", "passed": True, + } + assert result["metrics"]["answer_coverage"]["denominator"] == ( + bundle["preregistration"]["denominators"]["answerable_cases"] + ) + assert result["metrics"]["unanswerable_poison_rejection_at_20"]["value"] == 1.0 + assert all(limit == 20 and expansions == [] + for _query, limit, expansions in api.search_calls) + # Every registered main question is submitted verbatim. Facet Gold is + # never concatenated into it or passed as expansion text. + seen_queries = [query for query, _limit, _expansions in api.search_calls] + for case in bundle["cases"]: + assert case["question"] in seen_queries + + def test_candidate_poison_is_diagnostic_but_final_forbidden_selection_is_gated( + self, smoke_run): + result, _api, _work, bundle = smoke_run + cases = copy.deepcopy(result["case_results"]) + target = next(row for row in cases if row["answerable"]) + target["forbidden_top20_ids"] = ["hard-negative-in-candidate-list"] + target["regression_pass"] = True + metrics, _slices = run_eval._compute_metrics( + bundle, + { + "natural_units": {"indexed": bundle["preregistration"]["denominators"]["natural_units"], + "expected": bundle["preregistration"]["denominators"]["natural_units"]}, + "content_sections": {"registered": bundle["preregistration"]["denominators"]["content_sections"], + "expected": bundle["preregistration"]["denominators"]["content_sections"]}, + }, + cases, + ) + assert metrics["fixed_regression_pass_rate"]["value"] == 1.0 + assert metrics["forbidden_selected_rate"]["value"] == 0.0 + diagnostic = run_eval._candidate_diagnostics(cases) + assert diagnostic["answerable_forbidden_candidate_hit_at_20"]["numerator"] >= 1 + + poisoned = copy.deepcopy(cases) + selected_case = next(row for row in poisoned if row["answerable"]) + selected_case["facets"][0]["forbidden_selected"] = True + selected_case["facets"][0]["correct"] = False + selected_case["fully_grounded"] = False + selected_case["regression_pass"] = False + poisoned_metrics, _ = run_eval._compute_metrics( + bundle, + { + "natural_units": {"indexed": bundle["preregistration"]["denominators"]["natural_units"], + "expected": bundle["preregistration"]["denominators"]["natural_units"]}, + "content_sections": {"registered": bundle["preregistration"]["denominators"]["content_sections"], + "expected": bundle["preregistration"]["denominators"]["content_sections"]}, + }, + poisoned, + ) + assert poisoned_metrics["forbidden_selected_rate"]["numerator"] == 1 + assert poisoned_metrics["forbidden_selected_rate"]["value"] > 0.0 + + def test_hidden_details_are_blocked_and_default_output_removes_case_labels(self, smoke_run): + result, _api, _work, _bundle = smoke_run + hidden = copy.deepcopy(result) + hidden["certification_kind"] = "hidden" + hidden["is_hidden_certification"] = True + hidden["failed_case_ids"] = ["secret-case-label"] + with pytest.raises(protocol.HoldoutProtocolError, match="--details is disabled"): + run_eval._public_result(hidden, details=True) + safe = run_eval._public_result(hidden, details=False) + assert "case_results" not in safe + assert "failed_case_ids" not in safe + assert "secret-case-label" not in json.dumps(safe) + + def test_hidden_details_cli_fails_closed_without_echoing_gold( + self, smoke_run, monkeypatch, tmp_path, capsys): + result, _api, _work, _bundle = smoke_run + hidden = copy.deepcopy(result) + hidden["certification_kind"] = "hidden" + hidden["is_hidden_certification"] = True + hidden["case_results"][0]["case_id"] = "DO-NOT-ECHO-GOLD-LABEL" + hidden["failed_case_ids"] = ["DO-NOT-ECHO-GOLD-LABEL"] + monkeypatch.setattr(run_eval, "_load_api", lambda: object()) + monkeypatch.setattr(run_eval, "evaluate", lambda *_args, **_kwargs: hidden) + rc = run_eval.main([ + "--bundle", str(tmp_path / "hidden.json"), + "--bundle-sha256", "0" * 64, + "--run-id", "hidden-details-test", + "--seed", "1", + "--work-dir", str(tmp_path / "run"), + "--json", "--details", + ]) + captured = capsys.readouterr() + assert rc == 2 + assert "--details is disabled" in captured.out + assert "DO-NOT-ECHO-GOLD-LABEL" not in captured.out + captured.err + + def test_external_system_exit_zero_is_structured_protocol_error( + self, monkeypatch, tmp_path, capsys): + def exit_zero(): + raise SystemExit(0) + + monkeypatch.setattr(run_eval, "_load_api", exit_zero) + rc = run_eval.main([ + "--bundle", str(tmp_path / "unused.json"), + "--bundle-sha256", "0" * 64, + "--run-id", "system-exit-guard", + "--seed", "1", + "--json", + ]) + assert rc == 2 + payload = json.loads(capsys.readouterr().out) + assert payload["status"] == "error" + assert "SystemExit(0)" in payload["error"] + + def test_gold_never_appears_in_tested_workspace(self, smoke_run): + result, _api, work, bundle = smoke_run + workspace = work / f"holdout-{result['rotation']['run_fingerprint'][:16]}" / "workspace" + assert workspace.is_dir() + assert not any("gold" in path.name.casefold() for path in workspace.rglob("*")) + run_eval._assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + + def test_gold_leak_mutations_fail_by_filename_and_canary(self, tmp_path, smoke_file): + _path, _digest, bundle = smoke_file + workspace = tmp_path / "workspace" + workspace.mkdir() + leaked = workspace / ".holdout-gold.json" + leaked.write_text(json.dumps(bundle["oracle"]), encoding="utf-8") + with pytest.raises(protocol.HoldoutProtocolError, match="Gold-like file"): + run_eval._assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + leaked.unlink() + disguised = workspace / "derived.db" + disguised.write_text(bundle["oracle"]["gold_canary"], encoding="utf-8") + with pytest.raises(protocol.HoldoutProtocolError, match="Gold canary leaked"): + run_eval._assert_no_gold_leak(workspace, bundle["oracle"]["gold_canary"]) + + def test_forged_search_hit_cannot_claim_oracle_success(self, smoke_file): + _path, _digest, bundle = smoke_file + source = bundle["oracle"]["expected_inventory"][0] + forged = { + "path": source["path"], "anchor": source["anchor"], + "canonical_ref": f"{source['path']}#^{source['anchor'].lstrip('^')}", + "unit_id": "forged-unit-not-in-db", "kind": source["kind"], + "subordinal": source["subordinal"], "content_hash": source["content_hash"], + "text": source["text"], "score": 999.0, + } + inventory = {row["unit_id"]: row + for row in bundle["oracle"]["expected_inventory"]} + with pytest.raises(protocol.HoldoutProtocolError, match="forged outside exact inventory"): + run_eval._validate_search_result({"hits": [forged]}, inventory, 20) + + def test_run_rotation_is_reproducible_but_never_claimed_independent(self, smoke_file): + _path, digest, bundle = smoke_file + first, _ = run_eval._rotation(digest, bundle, "rotation-a", 7) + replay, _ = run_eval._rotation(digest, bundle, "rotation-a", 7) + moved, _ = run_eval._rotation(digest, bundle, "rotation-b", 8) + assert first == replay + assert first["run_fingerprint"] != moved["run_fingerprint"] + assert first["distribution_id"] == moved["distribution_id"] + assert first["independent_distribution"] is False + assert "not a new independent" in first["warning"] + + def test_core_protocol_has_no_llm_embedding_or_network_client_imports(self): + code = "\n".join( + (PROJECT_ROOT / "evals" / "holdout" / name).read_text(encoding="utf-8") + for name in ("protocol.py", "run_eval.py", "generate_public_smoke.py") + ).casefold() + for forbidden in ("import openai", "import anthropic", "import requests", + "import httpx", "sentence_transformers", "faiss"): + assert forbidden not in code diff --git a/scripts/tests/test_k_citecheck.py b/scripts/tests/test_k_citecheck.py new file mode 100644 index 0000000..1d3f2d3 --- /dev/null +++ b/scripts/tests/test_k_citecheck.py @@ -0,0 +1,1051 @@ +"""引用语义核对（cite-check）测试：list_cite_mismatches / extract_claims / +cite_audit_log_batch / list_suspect_citations / 台账对账。 + +在临时知识库（fake_kb fixture）上跑——目标文件用手写行尾锚点 +（parse_blocks_with_anchors 只认块行尾的真实锚点标记）。 +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +import k +from conftest import write_md, standard_fm + + +def write_raw(fake_kb: Path, rel: str, body: str) -> None: + p = fake_kb / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body, encoding="utf-8") + + +def cite_items(issue=None): + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + if issue is None: + return items + return [i for i in items if i["issue"] == issue] + + +# ============================================================ +# list_cite_mismatches — 数字核对 +# ============================================================ + +class TestCiteMismatchNumbers: + def test_number_present_in_cited_block_passes(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "准确率达到 95.3%，超越基线。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + assert cite_items("imprecise-anchor") == [] + + def test_number_absent_reports_mismatch(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "本文提出一种新方法。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + items = cite_items("mismatch") + assert len(items) == 1 + assert "95.3%" in items[0]["numbers"] + + def test_number_in_owning_section_reports_imprecise(self, fake_kb): + write_raw( + fake_kb, "raw/papers/foo.md", + "## 结果 ^h-2-1-aaaa11\n\n方法介绍段。 ^p-1-abc123\n\n准确率 95.3%。 ^p-2-def456\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + items = cite_items("imprecise-anchor") + assert len(items) == 1 + assert "95.3%" in items[0]["numbers"] + + def test_rounding_tolerance(self, fake_kb): + # claim 95 匹配 94.7（按 claim 声明精度舍入相等）；claim 95.3 匹配 95.34 + write_raw(fake_kb, "raw/papers/foo.md", "结果 94.7%，另一项 95.34%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n约 95% 的样本通过 [[raw/papers/foo#^p-1-abc123]]，精确值 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_approx_marker_widens_tolerance(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "实测比例为 39.2%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n约 40% 的请求失败 [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_normalization_bold_fullwidth_thousands(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "提升 13.1%，样本 1000 条。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n提升(**+13.1％**)，样本 1,000 条 [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_magnitude_mismatch_1m_vs_1b(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "上下文可达 1B tokens。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n上下文可达 1M tokens [[raw/papers/foo#^p-1-abc123]]。\n", + ) + items = cite_items("mismatch") + assert len(items) == 1 + + def test_wikilink_and_mdlink_numbers_not_claims(self, fake_kb): + # 链接文件名 gpt-3.5 / arXiv 编号 / 锚点串里的数字不是论断数字 + write_raw(fake_kb, "raw/papers/foo.md", "介绍模型对比。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n对比 [[raw/papers/gpt-3.5-report|GPT-3.5 报告]] 与" + "[2309.15217](https://arxiv.org/abs/2309.15217)" + " [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_section_number_not_claim(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "介绍方法。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n论文 §3.1 显式说明该机制 [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_number_marked_pending_source_not_mismatch(self, fake_kb): + # 数字后紧跟 [需要来源] = 显式声明不归属本块引用——不算错引，归积压治理 + write_raw(fake_kb, "raw/papers/foo.md", "CRAG 上仅 40%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nCRAG 上仅 40%，远低于简单 benchmark 的 80%+[需要来源] " + "[[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + def test_year_not_checked(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "方法介绍。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n2023 年提出的方法 [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + + +class TestCiteMismatchTargets: + def test_raw_missing_degrades_to_unverifiable(self, fake_kb): + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/nonexist#^p-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + assert len(cite_items("unverifiable")) == 1 + + def test_mixed_targets_missing_is_unverifiable_without_false_mismatch(self, fake_kb): + # 可用目标未命中但另有缺失目标时不能断言 mismatch；strict 会因缺失失败。 + write_md( + fake_kb / "wiki" / "sources" / "s.md", standard_fm(type="source_summary"), + "# S\n\n摘要段，无数字。 ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[wiki/sources/s#^p-1-abc123]][[raw/papers/nonexist#^p-2-def456]]。\n", + ) + assert cite_items("mismatch") == [] + assert len(cite_items("unverifiable")) == 1 + + def test_missing_target_does_not_mask_imprecise_usable_target(self, fake_kb): + """缺失目标单独不可核验；仍须检查可用目标，不能把整块提前 continue。""" + write_raw( + fake_kb, "raw/papers/foo.md", + "## 结果 ^h-2-1-aaaa11\n\n方法介绍段。 ^p-1-abc123\n\n准确率 95.3%。 ^p-2-def456\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]" + "[[raw/papers/nonexist#^p-2-dead00]]。\n", + ) + assert cite_items("mismatch") == [] + unavailable = cite_items("unverifiable") + assert len(unavailable) == 1 + assert unavailable[0]["citations"] == [ + {"target": "raw/papers/nonexist.md", "anchor": "p-2-dead00", "status": "file-missing"} + ] + imprecise = cite_items("imprecise-anchor") + assert len(imprecise) == 1 + assert imprecise[0]["numbers"] == ["95.3%"] + + def test_missing_target_does_not_mask_quote_mismatch_on_usable_target(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "The method has limited gains. ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + '# X\n\n作者声称"completely solves the hallucination problem" ' + "[[raw/papers/foo#^p-1-abc123]][[raw/papers/nonexist#^p-2-dead00]]。\n", + ) + assert len(cite_items("unverifiable")) == 1 + imprecise = cite_items("imprecise-anchor") + assert len(imprecise) == 1 + assert imprecise[0]["quotes"] == ["completely solves the hallucination problem"] + + def test_anchor_missing_left_to_broken_refs(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "方法介绍。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-9-zzzz99]]。\n", + ) + assert cite_items() == [] # 不双报，broken-refs 辖区 + + def test_anchor_missing_does_not_mask_mismatch_on_usable_target(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "该段没有目标数字。 ^p-1-abc123\n") + write_raw(fake_kb, "raw/papers/bar.md", "另一个真实段。 ^p-1-def456\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]" + "[[raw/papers/bar#^p-9-not999]]。\n", + ) + mismatch = cite_items("mismatch") + assert len(mismatch) == 1 + assert mismatch[0]["numbers"] == ["95.3%"] + + def test_hash_recovered_anchor_is_visible_as_canonical_issue(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "方法确实有效。 ^p-1-abcdef\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n方法确实有效 [[raw/papers/foo#^t-9-abcdef]]。\n", + ) + items = cite_items("canonical-anchor-mismatch") + assert len(items) == 1 + assert items[0]["requested_anchor"] == "t-9-abcdef" + assert items[0]["canonical_anchor"] == "p-1-abcdef" + + def test_wiki_target_block_checked(self, fake_kb): + # 两跳链第一跳：wiki → wiki/sources 的块级引用同样核对 + write_md( + fake_kb / "wiki" / "sources" / "s.md", standard_fm(type="source_summary"), + "# S\n\n摘要：整体结论为方法可行。 ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nLC 49.70 vs RAG 37.33 [[wiki/sources/s#^p-1-abc123]]。\n", + ) + items = cite_items("mismatch") + assert len(items) == 1 + assert set(items[0]["numbers"]) == {"49.70", "37.33"} + + def test_h_section_hit_passes(self, fake_kb): + write_raw( + fake_kb, "raw/papers/foo.md", + "## 结果 ^h-2-1-aaaa11\n\n准确率 95.3%。 ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^h-2-1-aaaa11]]。\n", + ) + assert cite_items("mismatch") == [] + assert cite_items("imprecise-anchor") == [] + + def test_h_section_oversized_hit_is_imprecise(self, fake_kb): + filler = "。".join(["这是一段占位叙述文字" for _ in range(900)]) # > 8000 字符 + write_raw( + fake_kb, "raw/papers/foo.md", + f"## 结果 ^h-2-1-aaaa11\n\n{filler}，准确率 95.3%。 ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^h-2-1-aaaa11]]。\n", + ) + assert cite_items("mismatch") == [] + items = cite_items("imprecise-anchor") + assert len(items) == 1 + + +class TestCiteExemptAndQuotes: + def test_valid_exempt_skips_number_check(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "A 得分 70.3，B 得分 2.3。 ^t-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n提升约 30 倍 [KB 推算: ^t-1-abc123] [[raw/papers/foo#^t-1-abc123]]。\n", + ) + assert cite_items("mismatch") == [] + assert len(cite_items("exempted")) == 1 + + def test_valid_exempt_only_applies_to_adjacent_number(self, fake_kb): + """一个合法推算标记不能把同段不相邻的普通数字一起洗白。""" + write_raw(fake_kb, "raw/papers/foo.md", "A 得分 70.3，B 得分 2.3。 ^t-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3%，提升约 30 倍 [KB 推算: ^t-1-abc123] " + "[[raw/papers/foo#^t-1-abc123]]。\n", + ) + exempted = cite_items("exempted") + assert len(exempted) == 1 + assert exempted[0]["numbers"] == ["30 倍"] + mismatch = cite_items("mismatch") + assert len(mismatch) == 1 + assert mismatch[0]["numbers"] == ["95.3%"] + + def test_exempt_does_not_spread_to_later_same_value(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "A 得分 70.3，B 得分 2.3。 ^t-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n推算为 30 倍 [KB 推算: ^t-1-abc123]，另一个未经推算的指标也是 30 倍 " + "[[raw/papers/foo#^t-1-abc123]]。\n", + ) + assert cite_items("exempted")[0]["numbers"] == ["30 倍"] + assert cite_items("mismatch")[0]["numbers"] == ["30 倍"] + + def test_exempt_basis_anchor_must_actually_be_cited(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "A 得分 70.3，B 得分 2.3。 ^t-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n提升约 30 倍 [KB 推算: ^t-9-notcited] " + "[[raw/papers/foo#^t-1-abc123]]。\n", + ) + assert cite_items("exempted") == [] + assert len(cite_items("exempt-missing-basis")) == 1 + assert cite_items("mismatch")[0]["numbers"] == ["30 倍"] + + def test_bare_exempt_is_gate_finding(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "A 得分 70.3。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n提升 30 倍 [KB 推算] [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert len(cite_items("exempt-missing-basis")) == 1 + + def test_quote_verbatim_hit_passes(self, fake_kb): + write_raw( + fake_kb, "raw/papers/foo.md", + "The retrieval quality is the primary bottleneck of RAG systems. ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n作者指出\"the primary bottleneck of RAG systems\" " + "[[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items() == [] + + def test_quote_missing_is_imprecise(self, fake_kb): + write_raw( + fake_kb, "raw/papers/foo.md", + "The retrieval quality matters a lot in practice. ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n作者声称\"completely solves the hallucination problem\" " + "[[raw/papers/foo#^p-1-abc123]]。\n", + ) + items = cite_items("imprecise-anchor") + assert len(items) == 1 + assert items[0]["quotes"] + + def test_cross_language_quote_skipped(self, fake_kb): + # 中文引号引英文目标的转写（或反之）：不可逐字核对，跳过 + write_raw( + fake_kb, "raw/papers/foo.md", + "Retrieval quality is the bottleneck. ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n作者认为“检索质量是整个系统的根本瓶颈所在” [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items() == [] + + +class TestCiteMismatchSkips: + def test_callout_and_table_and_code_skipped(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "方法介绍。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n" + "> [!WARNING] 知识更新冲突 — 2026-01-01\n" + "> 旧观点 95.3% [[raw/papers/foo#^p-1-abc123]]\n\n" + "| 指标 | 值 [[raw/papers/foo#^p-1-abc123]] |\n|---|---|\n| acc | 95.3% |\n\n" + "`95.3% [[raw/papers/foo#^p-1-abc123]]`\n", + ) + assert cite_items("mismatch") == [] + + def test_deprecated_page_skipped(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "方法介绍。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(status="deprecated"), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + assert cite_items() == [] + + +# ============================================================ +# extract_claims + 台账 +# ============================================================ + +def _setup_pair(fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "准确率达到 95.3%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + + +class TestExtractClaims: + def test_enumerates_pairs_including_tables(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "准确率 95.3%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/foo#^p-1-abc123]]。\n\n" + "| 模型 | 分数 |\n|---|---|\n| A | 95.3 [[raw/papers/foo#^p-1-abc123]] |\n\n" + "> [!NOTE] 说明\n> 引用 [[raw/papers/foo#^p-1-abc123]] 也须进枚举\n", + ) + pages = k.load_all_wiki_pages() + data = k.extract_claims(pages) + # paragraph + table row + factual NOTE；只有冲突/审计协议 callout 排除。 + assert data["summary"]["pairs_total"] == 3 + assert all(p["target_status"] == "ok" for p in data["pairs"]) + + def test_pair_id_changes_when_claim_changes(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid1 = k.extract_claims(pages)["pairs"][0]["pair_id"] + page_file = fake_kb / "wiki" / "concepts" / "c.md" + page_file.write_text( + page_file.read_text(encoding="utf-8").replace("95.3%", "96.3%"), encoding="utf-8") + pages = k.load_all_wiki_pages() + pid2 = k.extract_claims(pages)["pairs"][0]["pair_id"] + assert pid1 != pid2 + + def test_h_target_hash_tracks_section_body(self, fake_kb): + # ^h- 目标对整节正文算 hash——标题不变、正文重写也会被察觉 + write_raw( + fake_kb, "raw/papers/foo.md", + "## 结果 ^h-2-1-aaaa11\n\n准确率 95.3%。 ^p-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n见结果节 [[raw/papers/foo#^h-2-1-aaaa11]]。\n", + ) + pages = k.load_all_wiki_pages() + h1 = k.extract_claims(pages)["pairs"][0]["target_content_hash"] + raw_file = fake_kb / "raw" / "papers" / "foo.md" + raw_file.write_text( + "## 结果 ^h-2-1-aaaa11\n\n结论被整段重写但标题没变。 ^p-1-abc123\n", + encoding="utf-8") + h2 = k.extract_claims(pages)["pairs"][0]["target_content_hash"] + assert h1 != h2 + + def test_raw_not_distributed_vs_file_missing(self, fake_kb): + # raw/ 树只有 .gitkeep → raw-not-distributed；有其他文件时单个缺失 → file-missing + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[raw/papers/nonexist#^p-1-abc123]]。\n", + ) + (fake_kb / "raw" / "papers" / ".gitkeep").write_text("", encoding="utf-8") + pages = k.load_all_wiki_pages() + assert k.extract_claims(pages)["pairs"][0]["target_status"] == "raw-not-distributed" + write_raw(fake_kb, "raw/papers/other.md", "存在的文件。 ^p-1-abc123\n") + assert k.extract_claims(pages)["pairs"][0]["target_status"] == "file-missing" + + def test_paths_filter_and_sample_reproducible(self, fake_kb): + _setup_pair(fake_kb) + write_md( + fake_kb / "wiki" / "concepts" / "d.md", standard_fm(), + "# Y\n\n召回率 88.1% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + only_c = k.extract_claims(pages, paths=["wiki/concepts/c.md"]) + assert {p["page"] for p in only_c["pairs"]} == {"wiki/concepts/c.md"} + s1 = k.extract_claims(pages, sample=1, seed="2026-W27")["pairs"] + s2 = k.extract_claims(pages, sample=1, seed="2026-W27")["pairs"] + assert [p["pair_id"] for p in s1] == [p["pair_id"] for p in s2] + + def test_claim_text_is_not_silently_truncated(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "完整证据。 ^p-1-abc123\n") + long_claim = "这是需要完整交给核验器的论断。" * 45 + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + f"# X\n\n{long_claim} [[raw/papers/foo#^p-1-abc123]]。\n", + ) + pair = k.extract_claims(k.load_all_wiki_pages())["pairs"][0] + assert len(pair["claim_text"]) > 500 + assert long_claim in pair["claim_text"] + assert pair["claim_text_truncated"] is False + + def test_heading_context_changes_pair_id_and_invalidates_audit(self, fake_kb, tmp_path): + write_raw(fake_kb, "raw/papers/foo.md", "该策略适用于目标实体。 ^p-1-abc123\n") + draft = tmp_path / "draft.md" + draft.write_text( + "## Alpha\n\n该策略适用于目标实体 [[raw/papers/foo#^p-1-abc123]]。\n", + encoding="utf-8", + ) + first = k.extract_claims([k._draft_page(draft)])["pairs"][0] + assert "Alpha" in first["claim_text"] + result = k.cite_audit_log_batch( + [{"pair_id": first["pair_id"], "verdict": "SUPPORTED", + "evidence": "该策略适用于目标实体"}], draft_path=draft) + assert result["errors"] == [] + assert k.extract_claims([k._draft_page(draft)])["pairs"][0]["audited"] is True + + draft.write_text( + "## Beta\n\n该策略适用于目标实体 [[raw/papers/foo#^p-1-abc123]]。\n", + encoding="utf-8", + ) + second = k.extract_claims([k._draft_page(draft)])["pairs"][0] + assert second["pair_id"] != first["pair_id"] + assert second["claim_hash"] != first["claim_hash"] + assert "Beta" in second["claim_text"] and "Alpha" not in second["claim_text"] + assert second["audited"] is False + + def test_calculated_value_is_metadata_not_cloze_blank(self, fake_kb): + write_raw( + fake_kb, "raw/papers/foo.md", + "准确率 95.3%，A 得分 70.3，B 得分 2.3。 ^t-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3%，提升约 30 倍 [KB 推算: ^t-1-abc123] " + "[[raw/papers/foo#^t-1-abc123]]。\n", + ) + pair = k.extract_claims(k.load_all_wiki_pages(), cloze=True)["pairs"][0] + assert pair["numeric_tokens"] == ["95.3%"] + assert pair["calculated_tokens"] == [ + {"raw": "30 倍", "basis_anchor": "t-1-abc123"} + ] + assert [b["raw"] for b in pair["cloze"]["blanks"]] == ["95.3%"] + assert "30 倍" in pair["cloze"]["text"] + + +class TestCiteAuditLedger: + def test_unverifiable_rejected_for_ok_target(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages)["pairs"][0]["pair_id"] + result = k.cite_audit_log_batch([{"pair_id": pid, "verdict": "UNVERIFIABLE"}]) + assert result["written"] == [] + assert "不得记 UNVERIFIABLE" in result["errors"][0]["error"] + + def test_supported_requires_real_evidence(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages)["pairs"][0]["pair_id"] + r1 = k.cite_audit_log_batch([{"pair_id": pid, "verdict": "SUPPORTED"}]) + assert "必须附 --evidence" in r1["errors"][0]["error"] + r2 = k.cite_audit_log_batch( + [{"pair_id": pid, "verdict": "SUPPORTED", "evidence": "这段假证据不在目标块里"}]) + assert "不匹配" in r2["errors"][0]["error"] + # 真原文片段（含行尾锚点的 read-block 形态也接受） + r3 = k.cite_audit_log_batch( + [{"pair_id": pid, "verdict": "SUPPORTED", "evidence": "准确率达到 95.3%。 ^p-1-abc123"}]) + assert r3["errors"] == [] + assert len(r3["written"]) == 1 + + def test_ledger_join_and_invalidation(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages)["pairs"][0]["pair_id"] + k.cite_audit_log_batch( + [{"pair_id": pid, "verdict": "SUPPORTED", "evidence": "准确率达到 95.3%"}]) + data = k.extract_claims(pages) + assert data["pairs"][0]["audited"] is True + assert data["summary"]["unaudited_verifiable"] == 0 + # 被引块内容变 → 台账失效（claim 未变，pair_id 不变，target hash 漂移） + raw_file = fake_kb / "raw" / "papers" / "foo.md" + raw_file.write_text("准确率其实是 75.3%。 ^p-1-abc123\n", encoding="utf-8") + data = k.extract_claims(pages) + assert data["pairs"][0]["audited"] is False + # 删台账 → 全部回未审（派生层可重建性） + k._ledger_path().unlink() + raw_file.write_text("准确率达到 95.3%。 ^p-1-abc123\n", encoding="utf-8") + data = k.extract_claims(pages) + assert data["pairs"][0]["audited"] is False + + def test_expired_pair_rejected(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages)["pairs"][0]["pair_id"] + page_file = fake_kb / "wiki" / "concepts" / "c.md" + page_file.write_text( + page_file.read_text(encoding="utf-8").replace("95.3%", "96.3%"), encoding="utf-8") + result = k.cite_audit_log_batch( + [{"pair_id": pid, "verdict": "SUPPORTED", "evidence": "准确率达到"}]) + assert "已过期" in result["errors"][0]["error"] or "不存在" in result["errors"][0]["error"] + + +# ============================================================ +# CAUTION 审计标注 + 台账对账 +# ============================================================ + +CAUTION_BLOCK = ( + "> [!CAUTION] 引用审计未通过 — 2026-07-02\n" + "> **论断**：准确率 95.3%（块 ^p-9-ffffff）\n" + "> **被引块**：[[raw/papers/foo#^p-1-abc123]]\n" + "> **审计判定**：UNSUPPORTED — 被引块无该数字\n" + "> **状态**：⏳ 待人类判别\n" +) + + +class TestSuspectCitations: + def test_caution_block_scanned_and_exempt_from_bare_claims(self, fake_kb): + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + f"# X\n\n正文段落。\n\n{CAUTION_BLOCK}\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_suspect_citations(pages) + assert len(items) == 1 + assert "[[raw/papers/foo#^p-1-abc123]]" in items[0]["cited"] + # CAUTION 在 CALLOUT_RE 白名单：bare-claims / cite-mismatches 不误报标注自身 + assert k.list_bare_claims(pages) == [] + assert k.list_cite_mismatches(pages) == [] + + def test_ledger_unsupported_without_marker_detected(self, fake_kb): + _setup_pair(fake_kb) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages)["pairs"][0]["pair_id"] + k.cite_audit_log_batch([{"pair_id": pid, "verdict": "UNSUPPORTED", + "note": "数字不符"}]) + issues = k.check_citation_ledger_consistency(pages) + assert any(i["issue_type"] == "ledger-unsupported-without-marker" for i in issues) + # 页面补上 CAUTION 标注（含同一锚点）后对账归零 + page_file = fake_kb / "wiki" / "concepts" / "c.md" + page_file.write_text( + page_file.read_text(encoding="utf-8") + "\n" + CAUTION_BLOCK, encoding="utf-8") + pages = k.load_all_wiki_pages() + issues = k.check_citation_ledger_consistency(pages) + assert not any(i["issue_type"] == "ledger-unsupported-without-marker" for i in issues) + + +# ============================================================ +# coarse-citations 的 wiki/sources 整页引用纳管（逃逸缝隙回归） +# ============================================================ + +class TestCoarseWikiSourcesEscape: + def test_page_level_wiki_sources_cite_is_coarse(self, fake_kb): + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3%，见 [[wiki/sources/foo]]。\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_coarse_citations(pages) + assert len(items) == 1 + assert "[[wiki/sources/foo]]" in items[0]["coarse_refs"] + + def test_block_level_wiki_cite_is_compliant(self, fake_kb): + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 95.3% [[wiki/sources/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + assert k.list_coarse_citations(pages) == [] + + +# ============================================================ +# 盲填复核（cloze）与草稿核对（check-draft） +# ============================================================ + +class TestClozeCheck: + def test_cloze_masks_all_occurrences_positional_order(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "A 得 72.4%，B 得 61.8%，延迟 42ms。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nA 72.4% 优于 B 61.8%，代价是 42ms 延迟；再强调一次 72.4% " + "[[raw/papers/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + pair = k.extract_claims(pages, cloze=True)["pairs"][0] + cz = pair["cloze"] + # 占位符按首现位置编号；同值全部挖掉（防泄漏） + assert [b["ph"] for b in cz["blanks"]] == ["N1", "N2", "N3"] + assert "72.4" not in cz["text"] and "61.8" not in cz["text"] and "42" not in cz["text"] + assert cz["text"].count("⟦N1⟧") == 2 # 72.4% 出现两次全被挖 + + def test_cloze_check_pass_fail_unknown(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "准确率 72.4%。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + pid = k.extract_claims(pages, cloze=True)["pairs"][0]["pair_id"] + ok = k.cloze_check_batch([{"pair_id": pid, "fills": {"N1": "72.4%"}}])[0] + assert ok["passed"] is True + # 格式差异（72.4 无百分号）也按数值判分通过 + ok2 = k.cloze_check_batch([{"pair_id": pid, "fills": {"N1": "约 72.4"}}])[0] + assert ok2["passed"] is True + bad = k.cloze_check_batch([{"pair_id": pid, "fills": {"N1": "95.7%"}}])[0] + assert bad["passed"] is False + unknown = k.cloze_check_batch([{"pair_id": pid, "fills": {"N1": "原文未给出"}}])[0] + assert unknown["passed"] is False + stale = k.cloze_check_batch([{"pair_id": "deadbeef00000000", "fills": {}}])[0] + assert stale["passed"] is False and "不存在" in stale["error"] + + def test_unit_numbers_extracted(self, fake_kb): + # 数字+计量单位（42ms）纳入核对：写错必被抓 + write_raw(fake_kb, "raw/papers/foo.md", "P95 延迟 42ms。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nP95 延迟 43ms [[raw/papers/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + assert len([i for i in items if i["issue"] == "mismatch"]) == 1 + + +class TestCheckDraft: + def _write_raw(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", "准确率 72.4%，延迟 42ms。 ^p-1-abc123\n") + + def test_clean_draft_passes(self, fake_kb, tmp_path): + self._write_raw(fake_kb) + draft = tmp_path / "draft.md" + draft.write_text("准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", encoding="utf-8") + r = k.check_draft(draft) + assert r["summary"]["gate_findings"] == 0 + assert r["bare_claims"] == [] and r["coarse_citations"] == [] + assert len(r["pairs"]) == 1 and r["pairs"][0]["target_status"] == "ok" + + def test_bad_draft_caught(self, fake_kb, tmp_path): + self._write_raw(fake_kb) + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 95.7% [[raw/papers/foo#^p-1-abc123]]。\n\n延迟 42ms，这句没引用。\n", + encoding="utf-8", + ) + r = k.check_draft(draft) + assert r["summary"]["gate_findings"] == 1 # 错引数字 + assert len(r["bare_claims"]) == 1 # 无引用数字论断 + + def test_strict_fails_closed_on_unverifiable_while_default_is_compatible( + self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 72.4% [[raw/papers/missing#^p-1-abc123]]。\n", encoding="utf-8") + default = k.check_draft(draft) + assert default["summary"]["gate_findings"] == 0 + strict = k.check_draft(draft, strict=True) + assert strict["summary"]["gate_findings"] > 0 + assert any(f["issue"] == "unverifiable-target" for f in strict["strict_findings"]) + + def test_strict_fails_on_pending_source(self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text("准确率 72.4% [需要来源]。\n", encoding="utf-8") + default = k.check_draft(draft) + assert default["summary"]["gate_findings"] == 0 + assert default["bare_claims"] == [] + strict = k.check_draft(draft, strict=True) + assert len(strict["pending_sources"]) == 1 + assert any(f["issue"] == "pending-source" for f in strict["strict_findings"]) + + def test_strict_fails_on_uncited_qualitative_fact(self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text("该方法会显著降低幻觉，并适用于生产环境。\n", encoding="utf-8") + result = k.check_draft(draft, strict=True) + assert result["passed"] is False + assert any(f["issue"] == "unmapped-factual-claim" for f in result["strict_findings"]) + + def test_library_unmapped_scanner_matches_strict_coverage(self, fake_kb): + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n该方法会显著降低幻觉。\n", + ) + findings = k.list_unmapped_claims(k.load_all_wiki_pages()) + assert len(findings) == 1 + assert findings[0]["path"] == "wiki/concepts/c.md" + assert findings[0]["issue"] == "unmapped-factual-claim" + + def test_strict_keeps_citation_after_sentence_punctuation_with_claim( + self, fake_kb, tmp_path): + self._write_raw(fake_kb) + draft = tmp_path / "draft.md" + draft.write_text( + "准确率为 72.4%。[[raw/papers/foo#^p-1-abc123]]\n", + encoding="utf-8", + ) + result = k.check_draft(draft, strict=True) + assert not any(f["issue"] == "unmapped-factual-claim" + for f in result["strict_findings"]) + + @pytest.mark.parametrize("marker", ["[知识库未覆盖]", "[Agent 推断]", "[Agent 综合]"]) + def test_strict_allows_explicit_non_kb_contract_markers( + self, fake_kb, tmp_path, marker): + draft = tmp_path / "draft.md" + draft.write_text(f"## 分析\n\n{marker} 该判断不作为知识库事实。\n", encoding="utf-8") + result = k.check_draft(draft, strict=True) + assert result["passed"] is True + assert not any(f["issue"] == "unmapped-factual-claim" + for f in result["strict_findings"]) + + def test_strict_qualitative_needs_source_remains_pending(self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text("该方法适合所有地区。[需要来源]\n", encoding="utf-8") + result = k.check_draft(draft, strict=True) + assert any(f["issue"] == "pending-source" for f in result["strict_findings"]) + assert not any(f["issue"] == "unmapped-factual-claim" + for f in result["strict_findings"]) + + def test_strict_allows_pure_structure_and_navigation(self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text( + "# 回答\n\n## 相关页面\n\n参考：\n\n- [[wiki/concepts/foo]]\n", + encoding="utf-8", + ) + result = k.check_draft(draft, strict=True) + assert result["passed"] is True + + @pytest.mark.parametrize("kind", ["NOTE", "TIP", "IMPORTANT"]) + def test_strict_checks_factual_callouts(self, fake_kb, tmp_path, kind): + draft = tmp_path / "draft.md" + draft.write_text( + f"> [!{kind}] 提醒\n> 该方法适用于所有地区。\n", encoding="utf-8") + result = k.check_draft(draft, strict=True) + assert any(f["issue"] == "unmapped-factual-claim" for f in result["strict_findings"]) + + def test_strict_checks_each_table_data_row(self, fake_kb, tmp_path): + draft = tmp_path / "draft.md" + draft.write_text( + "| 方法 | 结论 |\n|---|---|\n| A | 更可靠 |\n", encoding="utf-8") + result = k.check_draft(draft, strict=True) + unmapped = [f for f in result["strict_findings"] + if f["issue"] == "unmapped-factual-claim"] + assert len(unmapped) == 1 + assert "A" in unmapped[0]["claim_text"] + + def test_strict_rejects_hash_recovered_and_noncanonical_target(self, fake_kb, tmp_path): + write_raw(fake_kb, "raw/papers/foo.md", "方法确实有效。 ^p-1-abcdef\n") + recovered = tmp_path / "recovered.md" + recovered.write_text( + "方法确实有效 [[raw/papers/foo#^t-9-abcdef]]。\n", encoding="utf-8") + recovered_result = k.check_draft(recovered, strict=True) + assert any(f["issue"] == "canonical-anchor-mismatch" + for f in recovered_result["strict_findings"]) + + noncanonical = tmp_path / "noncanonical.md" + noncanonical.write_text( + "方法确实有效 [[./raw/papers/foo#^p-1-abcdef]]。\n", encoding="utf-8") + target_result = k.check_draft(noncanonical, strict=True) + assert any(f["issue"] == "canonical-target-mismatch" + for f in target_result["strict_findings"]) + + @pytest.mark.parametrize("body, field", [ + ("准确率 72.4%，没有引用。\n", "bare_claims"), + ("准确率 72.4% [[raw/papers/foo]]。\n", "coarse_citations"), + ]) + def test_strict_summary_and_passed_include_bare_and_coarse( + self, fake_kb, tmp_path, body, field): + draft = tmp_path / "draft.md" + draft.write_text(body, encoding="utf-8") + result = k.check_draft(draft, strict=True) + assert result[field] + assert result["passed"] is False + assert result["summary"]["gate_findings"] > 0 + + def test_strict_requires_supported_semantic_audit_and_provenance( + self, fake_kb, tmp_path): + self._write_raw(fake_kb) + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", encoding="utf-8") + + first = k.check_draft(draft, strict=True) + assert any(f["issue"] == "semantic-unaudited" for f in first["strict_findings"]) + assert any(f["issue"] == "no-retrieval-evidence" for f in first["strict_findings"]) + pair = first["pairs"][0] + + audited = k.cite_audit_log_batch( + [{"pair_id": pair["pair_id"], "verdict": "SUPPORTED", + "evidence": "准确率 72.4%，延迟 42ms"}], + draft_path=draft, + ) + assert audited["errors"] == [] + after_audit = k.check_draft(draft, strict=True) + assert not any(f["issue"] == "semantic-unaudited" for f in after_audit["strict_findings"]) + assert any(f["issue"] == "no-retrieval-evidence" for f in after_audit["strict_findings"]) + + k._log_retrieval([{ + "target": pair["target"], "anchor": pair["anchor"], + "hash": pair["target_content_hash"], + }]) + passed = k.check_draft(draft, strict=True) + assert passed["summary"]["gate_findings"] == 0 + assert passed["strict_findings"] == [] + + def test_strict_rejects_non_supported_audit_verdict(self, fake_kb, tmp_path): + self._write_raw(fake_kb) + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", encoding="utf-8") + pair = k.check_draft(draft)["pairs"][0] + result = k.cite_audit_log_batch( + [{"pair_id": pair["pair_id"], "verdict": "PARTIAL"}], draft_path=draft) + assert result["errors"] == [] + k._log_retrieval([{ + "target": pair["target"], "anchor": pair["anchor"], + "hash": pair["target_content_hash"], + }]) + strict = k.check_draft(draft, strict=True) + assert any(f["issue"] == "semantic-verdict" and f["verdict"] == "PARTIAL" + for f in strict["strict_findings"]) + + def test_strict_rejects_imprecise_anchor_even_when_semantics_audited( + self, fake_kb, tmp_path): + write_raw( + fake_kb, "raw/papers/foo.md", + "## 结果 ^h-2-1-aaaa11\n\n方法介绍。 ^p-1-abc123\n\n准确率 72.4%。 ^p-2-def456\n", + ) + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", encoding="utf-8") + strict = k.check_draft(draft, strict=True) + assert any(f["issue"] == "imprecise-anchor" for f in strict["strict_findings"]) + + def test_strict_rejects_truncated_evidence_packet(self, fake_kb, tmp_path): + long_evidence = "准确率 72.4%。" + ("完整证据上下文。" * 80) + write_raw(fake_kb, "raw/papers/foo.md", f"{long_evidence} ^p-1-abc123\n") + draft = tmp_path / "draft.md" + draft.write_text( + "准确率 72.4% [[raw/papers/foo#^p-1-abc123]]。\n", encoding="utf-8") + strict = k.check_draft( + draft, strict=True, with_evidence=True, max_evidence_chars=80) + assert strict["pairs"][0]["evidence_truncated"] is True + assert any(f["issue"] == "audit-packet-truncated" for f in strict["strict_findings"]) + + +# ============================================================ +# arXiv 转换瑕疵归一化（数字三连重复 + LaTeX 残留 + 英文量级词） +# ============================================================ + +class TestArxivConversionArtifactNormalization: + """真实场景（op-rag / retro / gtr 等论文）驱动的回归用例：数学模式数字被 + accessibility 文本重复 2-4 遍拼接、LaTeX wrapper 残留、英文拼写量级词。 + """ + + @pytest.mark.parametrize("raw,expect_substr", [ + ("78.4278.4278.42", "78.42"), # op-rag 表格：小数三连 + ("4.84.84.84.8B", "4.8B"), # e5 论文：小数+后缀四连 + ("101010k", "10k"), # splade：整数三连+后缀仅出现一次 + ("88.6588.65\\mathbf{88.65}", "88.65"), # LaTeX \mathbf{} 残留 + 三连 + ("6.76.76.7B", "6.7B"), # in-context-ralm + ("666666B", "66B"), # in-context-ralm（同段第二个数） + ]) + def test_collapses_known_artifacts(self, raw, expect_substr): + got = k._normalize_for_cite_match(raw) + assert expect_substr in got + + @pytest.mark.parametrize("raw", [ + "44.43", # 内部恰好有重复数字子串的正常小数——op-rag 真实数据 + "1111", # 巧合自重复的裸整数，无量级后缀 → 不应折叠 + "4477", + "144", + "88.65", # 单次出现，不应被误判为"重复" + ]) + def test_does_not_collapse_legit_numbers(self, raw): + assert k._normalize_for_cite_match(raw) == raw + + def test_latex_thin_space_and_zero_width_stripped(self): + # retro.md 原句："10​ms10ms10\,\textrm{ms}"（含零宽空格 + LaTeX 细空格命令） + got = k._normalize_for_cite_match("10​ms10ms10\\,\\textrm{ms}") + assert got == "10ms10ms10ms" # 三连未必需要在此步折叠，交给 unit 正则边界匹配最后一次出现 + + def test_full_pipeline_op_rag_table_case(self, fake_kb): + # 端到端：完整还原 op-rag 表格瑕疵场景，确认 list_cite_mismatches 不再假阳 + write_raw( + fake_kb, "raw/papers/foo.md", + "| Method | Acc |\n|---|---|\n| GPT-4O | 78.4278.4278.42 |\n ^t-1-abc123\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nGPT-4O 在该任务上取得 78.42[[raw/papers/foo#^t-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + assert [i for i in items if i["issue"] == "mismatch"] == [] + + def test_full_pipeline_int_repeat_with_suffix(self, fake_kb): + # splade 场景：整数三连 + 后缀仅出现一次 + write_raw(fake_kb, "raw/papers/foo.md", "在 101010k 篇文档上评测。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n在 1 万篇文档上评测[[raw/papers/foo#^p-1-abc123]]。\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + assert [i for i in items if i["issue"] == "mismatch"] == [] + + +class TestEnglishMagnitudeWords: + def test_billion_million_thousand_recognized_as_target_values(self): + assert 5e9 in k._extract_target_values( + k._normalize_for_cite_match("up to 5 billion parameters")) + assert 2e9 in k._extract_target_values( + k._normalize_for_cite_match("2 billion question-answer pairs")) + assert 170e6 in k._extract_target_values( + k._normalize_for_cite_match("valued at 170 million dollars")) + assert 3e3 in k._extract_target_values( + k._normalize_for_cite_match("roughly 3 thousand samples")) + + def test_case_insensitive_word_form_but_not_bare_letter_m(self): + # "Billion"（句首大写）也要识别；但裸字母 "m"（如做单位"米"缩写场景） + # 不应被当成 million——量级字符类 [kKMB...] 本身就是大小写敏感的， + # 只有多字符单词形式走不区分大小写 + assert 5e9 in k._extract_target_values( + k._normalize_for_cite_match("5 Billion-parameter model")) + vals = k._extract_target_values(k._normalize_for_cite_match("跑道长 30m")) + assert 30e6 not in vals # 不误判为 30 million + + def test_full_pipeline_chinese_magnitude_matches_english_words(self, fake_kb): + # gtr.md 真实场景：原文英文 "5 billion" / "2 billion"，wiki 写中文"50亿"/"20亿" + write_raw( + fake_kb, "raw/papers/foo.md", + "encoders of up to 5 billion parameters, embedding dim 768. ^p-1-abc123\n" + "This results in 2 billion question-answer pairs. ^p-2-def456\n", + ) + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n模型规模可达 50亿 参数[[raw/papers/foo#^p-1-abc123]]，" + "训练用 20亿 问答对[[raw/papers/foo#^p-2-def456]]。\n", + ) + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + assert [i for i in items if i["issue"] == "mismatch"] == [] + + +class TestMidBlockAnchorTailLeak: + """真实场景（ip 库）：wiki list 块内每个条目自带独立行内锚点（非标准的 + "整块仅末尾一个锚点"形态），原子分解按条目拆分后，非末位条目的锚点尾巴 + 未被剥离，其哈希数字被误当成论断数字——list_cite_mismatches / extract_claims + 都要能正确剥离每一行自己的锚点尾巴，不能只剥整块末尾那一个。 + """ + + def _write(self, fake_kb): + write_raw(fake_kb, "raw/notes/foo.md", "参考内容占位。 ^p-1-abc123\n") + write_md( + fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\n" + "- 条目一说准确率 91.0%[[raw/notes/foo#^p-1-abc123]] ^p-8-a10012\n" + "- 条目二说准确率 92.0%[[raw/notes/foo#^p-1-abc123]] ^p-9-a10013\n" + "- 条目三说准确率 93.0%[[raw/notes/foo#^p-1-abc123]] ^p-10-a10014\n", + ) + + def test_non_last_item_anchor_tail_not_leaked_into_numbers(self, fake_kb): + self._write(fake_kb) + raw_file = fake_kb / "raw" / "notes" / "foo.md" + raw_file.write_text( + "准确率 91.0%，92.0%，93.0% 均已验证。 ^p-1-abc123\n", encoding="utf-8") + pages = k.load_all_wiki_pages() + items = k.list_cite_mismatches(pages) + mismatches = [i for i in items if i["issue"] == "mismatch"] + # 10012/10013/10014（锚点哈希里的数字）不应作为"论断数字"出现 + for m in mismatches: + assert not any(n in ("10012", "10013", "10014") for n in m["numbers"]) + assert mismatches == [] + + def test_extract_claims_claim_text_excludes_mid_block_anchor(self, fake_kb): + self._write(fake_kb) + pages = k.load_all_wiki_pages() + pairs = k.extract_claims(pages)["pairs"] + assert all("a10012" not in p["claim_text"] and "a10013" not in p["claim_text"] + for p in pairs) diff --git a/scripts/tests/test_k_corpus_map.py b/scripts/tests/test_k_corpus_map.py new file mode 100644 index 0000000..f45c1ff --- /dev/null +++ b/scripts/tests/test_k_corpus_map.py @@ -0,0 +1,287 @@ +"""corpus-map（raw 层全库文档地图）测试。 + +设计要点回归：档位分级、章节列出深度、agent_summary/preview 兜底、摘要覆盖率、 +深度登记状态映射（anchor/标题两种行格式 + 升级行 ✓ 优先）、source_summary 关联 +（frontmatter sources 绑定）、未 ingest 标注、--file 过滤、空 raw 诚实降级。 +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +import k +from conftest import write_md, standard_fm + + +def write_raw(fake_kb: Path, rel: str, body: str) -> None: + p = fake_kb / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body, encoding="utf-8") + + +RAW_DOC = """# 示例论文标题 ^h-1-1-aaaa01 + +## 1 Introduction ^h-2-1-aaaa02 + +引言正文。 ^p-1-bbbb01 + +## 2 Approach ^h-2-2-aaaa03 + +### 2.1 子方法 ^h-3-1-aaaa04 + +方法正文。 ^p-2-bbbb02 + +## 3 Experiments ^h-2-3-aaaa05 + +实验正文。 ^p-3-bbbb03 +""" + + +def _write_source_page(fake_kb, registry_rows=""): + write_md( + fake_kb / "wiki" / "sources" / "foo_s.md", + standard_fm(type="source_summary", source_count=1, sources=['[[raw/papers/foo]]']), + "# Foo 摘要\n\n> **原始文件**: [[raw/papers/foo]]\n" + + ("\n## 章节深度登记\n\n| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + + registry_rows if registry_rows else ""), + ) + + +class TestCorpusMapBasics: + def test_doc_entry_fields(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + data = k.corpus_map(k.load_all_wiki_pages(), depth=3) + assert data["summary"]["total_docs"] == 1 + d = data["docs"][0] + assert d["file"] == "raw/papers/foo.md" + assert d["title"] == "示例论文标题" + assert d["tier"] == "①" + # 范围 2<=lvl<=depth：H1 永不列，depth=3 含 H3 子节 + assert [s["title"] for s in d["sections"]] == [ + "1 Introduction", "2 Approach", "2.1 子方法", "3 Experiments"] + + def test_depth_3_includes_h3(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + data = k.corpus_map(k.load_all_wiki_pages(), depth=3) + titles = [s["title"] for s in data["docs"][0]["sections"]] + assert "2.1 子方法" in titles + + def test_preview_fallback_and_summary_coverage(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + data = k.corpus_map(k.load_all_wiki_pages()) + d = data["docs"][0] + intro = next(s for s in d["sections"] if "Introduction" in s["title"]) + assert intro["summary"] == "引言正文。" # preview 兜底 + assert intro["has_agent_summary"] is False + assert d["summary_coverage"] == "0/5" # h1 + 3×h2 + 1×h3（覆盖率数全部章节，不受 depth 限制） + + def test_agent_summary_preferred_over_preview(self, fake_kb): + from postprocess import process + + raw, outline = process( + "# 示例论文标题\n\n## 1 Introduction\n\n引言正文。\n", + "raw/papers/foo.md", + ) + write_raw(fake_kb, "raw/papers/foo.md", raw) + def set_sum(secs): + for s in secs: + if s.get("title") == "1 Introduction": + s["agent_summary"] = "精排摘要在此" + set_sum(s.get("children", [])) + set_sum(outline["sections"]) + (fake_kb / "raw" / "papers" / "foo.outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8") + data = k.corpus_map(k.load_all_wiki_pages()) + intro = next(s for s in data["docs"][0]["sections"] if "Introduction" in s["title"]) + assert intro["summary"] == "精排摘要在此" + assert intro["has_agent_summary"] is True + + def test_file_filter_and_empty(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_raw(fake_kb, "raw/notes/bar.md", "# 笔记 ^h-1-1-cccc01\n\n内容。 ^p-1-dddd01\n") + data = k.corpus_map(k.load_all_wiki_pages(), file_filter="notes") + assert [d["file"] for d in data["docs"]] == ["raw/notes/bar.md"] + assert k.corpus_map([])["docs"] == [] or True # 空 pages 不炸 + + def test_empty_raw_dir(self, fake_kb): + data = k.corpus_map(k.load_all_wiki_pages()) + assert data["docs"] == [] and data["summary"]["total_docs"] == 0 + + +class TestCorpusMapLinkage: + def test_source_summary_linked_and_not_ingested(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_raw(fake_kb, "raw/papers/orphan.md", + "# 孤儿 ^h-1-1-eeee01\n\n没人 ingest 我。 ^p-1-ffff01\n") + _write_source_page(fake_kb) + data = k.corpus_map(k.load_all_wiki_pages()) + by_file = {d["file"]: d for d in data["docs"]} + assert by_file["raw/papers/foo.md"]["source_summary"] == "wiki/sources/foo_s.md" + assert by_file["raw/papers/orphan.md"]["source_summary"] is None + assert data["summary"]["not_ingested"] == 1 + + def test_depth_status_by_anchor_and_title(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + _write_source_page( + fake_kb, + "| ^h-2-1-aaaa02 | 1 Introduction | ✓ 深读 | x |\n" + "| Approach | ⊙ 扫读 | 标题行格式（无 anchor 列，剥编号匹配） |\n" + "| ^h-2-3-aaaa05 | 3 Experiments | × 跳过 | x |\n", + ) + data = k.corpus_map(k.load_all_wiki_pages()) + st = {s["title"]: s["depth_status"] for s in data["docs"][0]["sections"]} + assert st["1 Introduction"] == "✓" + assert st["2 Approach"] == "⊙" + assert st["3 Experiments"] == "×" + + def test_upgraded_row_reads_as_deep(self, fake_kb): + # 升级行「✓ 深读 | 由 ⊙ 升级」→ 状态列优先，显示 ✓ 不是 ⊙ + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + _write_source_page( + fake_kb, + "| ^h-2-2-aaaa03 | 2 Approach | ✓ 深读 | 2026-07-03 由 ⊙ 升级(query 触发) |\n", + ) + data = k.corpus_map(k.load_all_wiki_pages()) + st = {s["title"]: s["depth_status"] for s in data["docs"][0]["sections"]} + assert st["2 Approach"] == "✓" + + +class TestCorpusMapEdges: + """自检覆盖的边界（fmt/档位/标题回退）""" + + def test_empty_doc(self, fake_kb): + (fake_kb / "raw" / "papers" / "empty.md").write_text("", encoding="utf-8") + d = k.corpus_map(k.load_all_wiki_pages())["docs"][0] + assert d["tier"] == "①" and d["chars"] == 0 and d["title"] == "empty" + + def test_boundary_150k_strict_gt(self, fake_kb): + # ② 档上限是 30K-150K（严格 >150K 才进 ③），与 CLAUDE.md「③档 >150K」+ SKILL + # 「30K-150K 中长文」定义同口径 + for name, n in [("a.md", 150000), ("b.md", 150001)]: + (fake_kb / "raw" / "papers" / name).write_text("a" * n, encoding="utf-8") + tiers = {d["file"].rsplit("/", 1)[-1]: d["tier"] + for d in k.corpus_map(k.load_all_wiki_pages())["docs"]} + assert tiers["a.md"] == "②" and tiers["b.md"] == "③" + + def test_title_fallback_when_no_h1(self, fake_kb): + # 无 H1 文档（直接 H2 开头）：title 兜底为 stem，sections 仍正常列出 + (fake_kb / "raw" / "papers" / "no_h1.md").write_text( + "## 直接 H2 ^h-2-1-aaaa02\n\n内容 ^p-1-bbbb01\n", encoding="utf-8") + d = k.corpus_map(k.load_all_wiki_pages())["docs"][0] + assert d["title"] == "no_h1" + assert [s["title"] for s in d["sections"]] == ["直接 H2"] + + def test_multi_h1_takes_first(self, fake_kb): + # 多个 H1：title 取第一个，同时把所有 H1 作为书籍根章列出。 + (fake_kb / "raw" / "papers" / "multi.md").write_text( + "# 第一 ^h-1-1-aaaa01\n\n内容 ^p-1-bbbb01\n\n" + "# 第二 ^h-1-2-aaaa01\n\n## 第二节 ^h-2-1-aaaa02\n\n内容2 ^p-2-bbbb02\n", + encoding="utf-8") + d = k.corpus_map(k.load_all_wiki_pages())["docs"][0] + assert d["title"] == "第一" + # 多 H1 通常是书籍章名，全部必须进入地图，不得被当作单一文档标题隐去。 + titles = [s["title"] for s in d["sections"]] + assert "第一" in titles and "第二" in titles and "第二节" in titles + second_section = next(s for s in d["sections"] if s["title"] == "第二节") + assert second_section["heading_path"] == ["第二", "第二节"] + assert second_section["parent_anchor"] is not None + + +class TestReviewFixes: + """对抗审查确认项的回归钉子（review-corpus-map-v2 workflow, 2026-07-03）。""" + + def test_priority_status_any_check_wins(self, fake_kb): + # should-fix #1：状态判定优先级 ✓ > × > ⊙，避免反向「⊙ 在前 ✓ 在备注」误取 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + _write_source_page( + fake_kb, + "| ^h-2-2-aaaa03 | 2 Approach | ⊙ 扫读 | ✓ 待定 |\n", # ⊙ 在前 ✓ 在后 + ) + data = k.corpus_map(k.load_all_wiki_pages()) + st = {s["title"]: s["depth_status"] for s in data["docs"][0]["sections"]} + assert st["2 Approach"] == "✓" + + def test_title_only_unique_in_doc(self, fake_kb): + # should-fix #2：title-only 行要求该 title 在本文档内归一化计数 = 1 + # doc：两个同名 H2（1 Intro / 2 Intro，中间空行分隔），登记表用 title-only + # 行标 Intro 为 ⊙ → 应闸门拒绝（同名出现 2 次），两个 H2 都该 None + raw = "# P ^h-1-1-aaaa01\n\n## 1 Introduction ^h-2-1-aaaa02\n\nintro1 ^p-1-bbbb01\n\n\n" + raw += "\n## 2 Introduction ^h-2-2-aaaa03\n\nintro2 ^p-2-bbbb02\n" + write_raw(fake_kb, "raw/papers/foo.md", raw) + _write_source_page( + fake_kb, + "| Introduction | ⊙ 扫读 | 标题行（无 anchor） |\n", + ) + data = k.corpus_map(k.load_all_wiki_pages(), depth=3) + # corpus_map 的 section_title 是 raw 形式（与 search_raw 的 heading_stack + # 归一化版不同，corpus_map 自己重走 _walk 取 sec.title），验证时用 raw + # 名比对 + raw_titles = [s["title"] for s in data["docs"][0]["sections"]] + # 文档有两个同名 H2 "1 Introduction" / "2 Introduction"；title-only 行 + # "Introduction" 在 doc 内出现 >1 次 → 闸门应拒绝匹配，两个 section 都该 None + intro_titles = [t for t in raw_titles if t.endswith("Introduction")] + assert len(intro_titles) >= 2 # 至少两个 H2 同名（确认重名场景复现） + statuses = {s["title"]: s["depth_status"] for s in data["docs"][0]["sections"]} + for t in intro_titles: + assert statuses[t] is None, f"{t} 不该被 title-only 行匹配（重名闸门）" + + def test_cells_status_not_collected_as_title(self, fake_kb): + # should-fix #1b：cells[1]="⊙ 扫读" 状态格不应进 titles + # 验证：带 anchor 的登记行中 cells[0] 是 anchor（不进 titles），cells[1] 是 + # 原标题；状态格 cells[2]="⊙ 扫读" 被排除——避免污染 _row_status 匹配 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + _write_source_page( + fake_kb, + "| ^h-2-2-aaaa03 | 2 Approach | ⊙ 扫读 | 核心方法 |\n", + ) + pages = k.load_all_wiki_pages() + depth = k._load_depth_registry(pages) + titles = depth["raw/papers/foo"][0]["titles"] + assert "⊙ 扫读" not in titles + assert "⊙" not in titles + assert "核心方法" not in titles + assert "approach" in titles # 2 Approach 剥编号后 + + def test_search_raw_heading_stack_normalized(self, fake_kb): + # heading_stack 的 title 字段存归一化形式（剥前导编号 + 小写），与 + # reg["titles"]={"approach"} 直接字典匹配——无需各自重算 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + _write_source_page( + fake_kb, + "| Approach | ⊙ 扫读 | 核心方法 |\n", # title-only 唯一行 + ) + # "方法" 一词必命中 2.1 子方法块（被 search_raw 扫到）； + # 命中块所在 heading_stack 栈顶 = "2 approach"（归一化），与 reg 唯一匹配 + hits = k.search_raw("方法", pages=k.load_all_wiki_pages()) + assert hits, f"应至少有 1 条命中：{hits}" + assert hits[0]["deepen_hint"] is True + + +class TestCorpusMapFmtEdges: + """nit 修复的回归钉子。""" + + def test_empty_raw_honest_message(self, fake_kb): + # nit：空 raw 走 raw 缺失分支而非「无匹配文档」 + k.PROJECT_ROOT = fake_kb; k.WIKI_DIR = fake_kb / "wiki"; k.RAW_DIR = fake_kb / "raw" + # raw/papers 目录不创建 → RAW_DIR.is_dir() False → 走降级分支 + import io, contextlib + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + k.fmt_corpus_map(k.corpus_map([])) + assert "raw/ 未分发" in buf.getvalue() + + def test_broken_outline_skips(self, fake_kb, capsys): + # nit：outline 读取失败被 skip（之前是静默；本测试只验行为——stderr 报错 + # 留给运行时肉眼确认；断言 docs 不含此文件即可，不绑定具体报错方式） + write_raw(fake_kb, "raw/papers/broken.md", "# X ^h-1-1-aaaa01\n\n## Bad\n") + (fake_kb / "raw" / "papers" / "broken.outline.json").write_text( + "{ this is not valid json", encoding="utf-8") + k.PROJECT_ROOT = fake_kb; k.WIKI_DIR = fake_kb / "wiki"; k.RAW_DIR = fake_kb / "raw" + # 无论 outline 解析是否报错（不同实现版本可能回退重算），broken.md 至少 + # 不会让 corpus_map 整体挂掉 + data = k.corpus_map(k.load_all_wiki_pages()) + # 库内至少还有一个 raw 不挂，但 broken 可能或不在 docs 里 + # 本测试只确认 broken 不引发 unhandled exception + assert isinstance(data, dict) and "docs" in data diff --git a/scripts/tests/test_k_retrieval_cli.py b/scripts/tests/test_k_retrieval_cli.py new file mode 100644 index 0000000..b3dcb6b --- /dev/null +++ b/scripts/tests/test_k_retrieval_cli.py @@ -0,0 +1,149 @@ +"""CLI contract tests for the long-document evidence index.""" + +from __future__ import annotations + +import json +import os +import subprocess +import sys +from pathlib import Path + +from conversion_receipt import build_conversion_receipt +from postprocess import process + + +ENGINE_ROOT = Path(__file__).resolve().parents[2] +K_PY = ENGINE_ROOT / "scripts" / "k.py" + + +def _workspace(tmp_path: Path) -> tuple[Path, Path]: + data_root = tmp_path / "kb-data" + workspace = data_root / "workspaces" / "main" + (workspace / "raw" / "manuals").mkdir(parents=True) + (workspace / "wiki").mkdir() + plain = ( + "# Service manual\n\n" + "## Restart timing\n\n" + "The Aster controller requires a 17 minute observation period.\n" + ) + anchored, _outline = process(plain, "raw/manuals/service.md") + (workspace / "raw" / "manuals" / "service.md").write_text( + anchored, encoding="utf-8" + ) + return data_root, workspace + + +def _run(data_root: Path, *arguments: str) -> subprocess.CompletedProcess[str]: + env = os.environ.copy() + env["KB_ROOT"] = str(data_root) + return subprocess.run( + [sys.executable, str(K_PY), "--workspace", "main", *arguments, "--json"], + cwd=ENGINE_ROOT, + env=env, + text=True, + capture_output=True, + check=False, + ) + + +def _add_converted_source(workspace: Path) -> Path: + source = workspace / "raw" / "manuals" / "carrier.pdf" + source.write_bytes(b"carrier-container-v1") + markdown = source.with_suffix(".md") + anchored, outline = process( + "# Carrier manual\n\n## Calibration\n\nCarrier evidence token is 29.\n", + "raw/manuals/carrier.md", + ) + markdown.write_text(anchored, encoding="utf-8") + outline["conversion_receipt"] = build_conversion_receipt( + source, workspace, require_raw=True + ) + source.with_suffix(".outline.json").write_text( + json.dumps(outline), encoding="utf-8" + ) + return source + + +def test_rebuild_coverage_and_search_evidence_cli(tmp_path: Path) -> None: + data_root, _workspace_root = _workspace(tmp_path) + + rebuilt = _run(data_root, "rebuild-evidence-index") + assert rebuilt.returncode == 0, rebuilt.stderr + rebuild_payload = json.loads(rebuilt.stdout) + assert rebuild_payload["ok"] is True + assert rebuild_payload["natural_units"]["coverage"] == 1.0 + assert rebuild_payload["content_sections"]["coverage"] == 1.0 + + covered = _run(data_root, "evidence-index-coverage") + assert covered.returncode == 0, covered.stderr + assert json.loads(covered.stdout)["coverage_status"] == "complete" + + searched = _run(data_root, "search-evidence", "Aster observation period", "--limit", "20") + assert searched.returncode == 0, searched.stderr + payload = json.loads(searched.stdout) + assert payload["returned"] >= 1 + assert payload["hits"][0]["canonical_ref"].startswith( + "raw/manuals/service.md#^p-" + ) + unit = _run( + data_root, "read-evidence-unit", payload["hits"][0]["unit_id"] + ) + assert unit.returncode == 0, unit.stderr + unit_payload = json.loads(unit.stdout) + assert unit_payload["selection_scope"] == "natural_unit" + assert unit_payload["citation_scope"] == "parent_block" + assert "17 minute observation" in unit_payload["text"] + + +def test_cli_fails_closed_when_raw_changed_after_rebuild(tmp_path: Path) -> None: + data_root, workspace = _workspace(tmp_path) + assert _run(data_root, "rebuild-evidence-index").returncode == 0 + + source = workspace / "raw" / "manuals" / "service.md" + original = source.read_text(encoding="utf-8") + source.write_text(original.replace("17 minute", "18 minute"), encoding="utf-8") + + covered = _run(data_root, "evidence-index-coverage") + assert covered.returncode == 1 + coverage_payload = json.loads(covered.stdout) + assert coverage_payload["coverage_status"] == "stale-corpus" + assert coverage_payload["corpus_freshness"]["changed"] == [ + "raw/manuals/service.md" + ] + + searched = _run(data_root, "search-evidence", "Aster observation period") + assert searched.returncode == 2 + search_payload = json.loads(searched.stdout) + assert search_payload["error"]["code"] == "index-stale" + + +def test_cli_reports_missing_index_as_protocol_error(tmp_path: Path) -> None: + data_root, _workspace_root = _workspace(tmp_path) + result = _run(data_root, "evidence-index-coverage") + assert result.returncode == 2 + payload = json.loads(result.stdout) + assert payload["error"]["code"] == "index-not-found" + + +def test_cli_and_health_reject_stale_original_conversion_source(tmp_path: Path) -> None: + data_root, workspace = _workspace(tmp_path) + source = _add_converted_source(workspace) + assert _run(data_root, "rebuild-evidence-index").returncode == 0 + + source.write_bytes(b"carrier-container-v2") + covered = _run(data_root, "evidence-index-coverage") + assert covered.returncode == 1 + coverage = json.loads(covered.stdout) + assert coverage["coverage_status"] == "stale-corpus" + issues = coverage["corpus_freshness"]["conversion_source_issues"] + assert issues[0]["issue"]["code"] == "conversion-source-sha256-mismatch" + + searched = _run(data_root, "search-evidence", "Carrier evidence token") + assert searched.returncode == 2 + assert json.loads(searched.stdout)["error"]["code"] == "index-stale" + + health = _run(data_root, "health") + assert health.returncode == 0 # health remains a dashboard, not a strict gate + health_payload = json.loads(health.stdout) + assert health_payload["evidence_index"]["ok"] is False + assert health_payload["evidence_index"]["coverage_status"] == "stale-corpus" diff --git a/scripts/tests/test_k_search_raw.py b/scripts/tests/test_k_search_raw.py new file mode 100644 index 0000000..6921e65 --- /dev/null +++ b/scripts/tests/test_k_search_raw.py @@ -0,0 +1,262 @@ +"""search-raw（raw 原文块级全文检索）测试。 + +设计要点回归：宽召回机械原语（正文×1/标题×3/摘要×2/短语+5/全覆盖+2）、 +链接掩码防 TOC 刷分、纯标题命中不返回（防整节刷屏）、登记表 deepen_hint +祖先链联动（兼容 anchor 列与标题列两种登记格式、标题剥前导编号）。 +""" +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +import k +from conftest import write_md, standard_fm + + +def write_raw(fake_kb: Path, rel: str, body: str) -> None: + p = fake_kb / rel + p.parent.mkdir(parents=True, exist_ok=True) + p.write_text(body, encoding="utf-8") + + +RAW_DOC = """# 示例论文 ^h-1-1-aaaa01 + +## 1 Introduction ^h-2-1-aaaa02 + +本文提出新的检索方法。 ^p-1-bbbb01 + +## 2 Approach ^h-2-2-aaaa03 + +### 2.2 Mutual Indexing ^h-3-1-aaaa04 + +互索引机制把图结构与文本块 mutual indexing 双向关联，查询延迟 12ms。 ^p-2-bbbb02 + +## 3 Experiments ^h-2-3-aaaa05 + +在 HotpotQA 上 EM 提升 4.7 个点。 ^p-3-bbbb03 +""" + + +class TestSearchRawBasics: + def test_hit_returns_anchor_and_section(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + hits = k.search_raw("mutual indexing", pages=[]) + assert hits and hits[0]["file"] == "raw/papers/foo.md" + assert hits[0]["anchor"] == "p-2-bbbb02" + assert "mutual indexing" in hits[0]["section_title"] # heading_stack 存归一化形式以与 reg 一致 + + def test_no_hit_returns_empty(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + assert k.search_raw("量子纠缠", pages=[]) == [] + + def test_empty_raw_dir(self, fake_kb): + assert k.search_raw("anything", pages=[]) == [] + + def test_file_filter(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_raw(fake_kb, "raw/papers/bar.md", + "# 另一篇 ^h-1-1-cccc01\n\nmutual indexing 也出现在这里。 ^p-1-dddd01\n") + hits = k.search_raw("mutual indexing", pages=[], file_filter="bar") + assert hits and all("bar" in h["file"] for h in hits) + + def test_include_wiki_blocks(self, fake_kb): + write_md(fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), + "# X\n\nwiki 里也谈 mutual indexing 机制。\n") + assert k.search_raw("mutual indexing", pages=[]) == [] + hits = k.search_raw("mutual indexing", pages=[], include_wiki=True) + assert hits and hits[0]["file"].startswith("wiki/") + + +class TestSearchRawScoring: + def test_section_title_match_ranks_higher(self, fake_kb): + # 两块正文词频相同；一块所属章节标题也命中 → 排前 + write_raw(fake_kb, "raw/papers/foo.md", + "## indexing 方法 ^h-2-1-aaaa02\n\n本节讨论 indexing。 ^p-1-bbbb01\n\n" + "## 其他 ^h-2-2-aaaa03\n\n这里也提 indexing。 ^p-2-bbbb02\n") + hits = k.search_raw("indexing", pages=[]) + assert hits[0]["anchor"] == "p-1-bbbb01" + assert hits[0]["score"] > hits[1]["score"] + + def test_agent_summary_weight(self, fake_kb): + from postprocess import process + + raw, outline = process( + "## 甲节\n\n甲节正文提到延迟。\n\n" + "## 乙节\n\n乙节正文提到延迟。\n", + "raw/papers/foo.md", + ) + write_raw(fake_kb, "raw/papers/foo.md", raw) + outline["sections"][1]["agent_summary"] = "本节量化了延迟数据" + (fake_kb / "raw" / "papers" / "foo.outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8") + hits = k.search_raw("延迟", pages=[]) + expected = k.find_anchor( + fake_kb / "raw" / "papers" / "foo.md", "乙节正文提到延迟" + )[0]["anchor"] + assert hits[0]["anchor"] == expected # 摘要加权胜出 + + def test_multiword_coverage_and_phrase_bonus(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", + "# T ^h-1-1-aaaa01\n\n只有 alpha 一词。 ^p-1-bbbb01\n\n" + "alpha beta 两词都有且相邻。 ^p-2-bbbb02\n") + hits = k.search_raw("alpha beta", pages=[]) + assert hits[0]["anchor"] == "p-2-bbbb02" + + def test_title_only_match_not_returned(self, fake_kb): + # 纯标题命中（正文一词不含）不返回——防整节所有块刷屏 + write_raw(fake_kb, "raw/papers/foo.md", + "## unique_term 章 ^h-2-1-aaaa02\n\n正文完全无关。 ^p-1-bbbb01\n") + assert k.search_raw("unique_term", pages=[]) == [] + + def test_toc_link_text_masked(self, fake_kb): + # TOC 目录块（全是 markdown 链接）不得靠链接文本刷分 + write_raw(fake_kb, "raw/papers/foo.md", + "# T ^h-1-1-aaaa01\n\n" + "1. [2.2 Mutual Indexing](https://x/#S2.SS2)\n" + "2. [3 Experiments](https://x/#S3) ^p-1-bbbb01\n\n" + "正文真正讨论 mutual indexing 机制。 ^p-2-bbbb02\n") + hits = k.search_raw("mutual indexing", pages=[]) + assert hits and hits[0]["anchor"] == "p-2-bbbb02" + assert all(h["anchor"] != "p-1-bbbb01" for h in hits) + + def test_inline_link_label_remains_searchable(self, fake_kb): + # 搜索视图应只掩 URL，不得把正文里唯一的可见实体名一起删掉。 + write_raw( + fake_kb, + "raw/papers/foo.md", + "# T ^h-1-1-aaaa01\n\n" + "本节使用 [HippoRAG](https://example.test/hipporag) 处理检索。 ^p-1-bbbb01\n", + ) + hits = k.search_raw("HippoRAG", pages=[]) + assert hits and hits[0]["anchor"] == "p-1-bbbb01" + + +class TestSearchRawDeepenHint: + def _pages_with_registry(self, fake_kb, first_col): + write_md( + fake_kb / "wiki" / "sources" / "foo_summary.md", + standard_fm(type="source_summary", source_count=1, + sources=['[[raw/papers/foo]]']), + "# Foo 摘要\n\n> **原始文件**: [[raw/papers/foo]]\n\n" + "## 章节深度登记\n\n" + "| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + f"| {first_col} | 2 Approach | ⊙ 扫读 | 关键实体:mutual indexing |\n", + ) + return k.load_all_wiki_pages() + + def test_anchor_column_registry(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + pages = self._pages_with_registry(fake_kb, "^h-2-2-aaaa03") + hits = k.search_raw("mutual indexing", pages=pages) + assert hits[0]["deepen_hint"] is True # 命中子节 §2.2，祖先 2 Approach 在登记表 + + def test_title_column_registry_with_number_stripping(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + # demo 式登记：首列直接写标题（无编号），原文标题带编号"2 Approach" + write_md( + fake_kb / "wiki" / "sources" / "foo_summary.md", + standard_fm(type="source_summary", source_count=1, + sources=['[[raw/papers/foo]]']), + "# Foo 摘要\n\n> **原始文件**: [[raw/papers/foo]]\n\n" + "## 章节深度登记\n\n" + "| Section | 状态 | 理由 |\n|---|---|---|\n" + "| Approach | ⊙ 扫读 | 核心方法 |\n", + ) + pages = k.load_all_wiki_pages() + hits = k.search_raw("mutual indexing", pages=pages) + assert hits[0]["deepen_hint"] is True + + def test_deep_read_section_no_hint(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + pages = self._pages_with_registry(fake_kb, "^h-2-2-aaaa03") + hits = k.search_raw("HotpotQA", pages=pages) # 命中 3 Experiments（未登记 ⊙） + assert hits and hits[0]["deepen_hint"] is False + + +class TestReviewFixes: + """对抗审查确认项的回归钉子（review-search-raw workflow, 2026-07-03）。""" + + def test_same_stem_different_dirs_no_crosstalk(self, fake_kb): + # registry key 必须是完整相对路径：raw/papers/foo 与 raw/notes/foo 不串档 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_raw(fake_kb, "raw/notes/foo.md", RAW_DOC.replace("mutual indexing", "别的内容")) + write_md( + fake_kb / "wiki" / "sources" / "papers_foo.md", + standard_fm(type="source_summary", source_count=1, sources=['[[raw/papers/foo]]']), + "# 摘要\n\n> **原始文件**: [[raw/papers/foo]]\n\n## 章节深度登记\n\n" + "| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + "| ^h-2-2-aaaa03 | 2 Approach | ⊙ 扫读 | x |\n", + ) + pages = k.load_all_wiki_pages() + # 命中 raw/notes/foo（未登记）的同名章节 → 不得沾 papers 的登记 + hits = k.search_raw("别的内容", pages=pages) + assert hits and all(h["deepen_hint"] is False for h in hits) + # papers 自己照常联动 + hits2 = k.search_raw("mutual indexing", pages=pages) + assert hits2 and hits2[0]["deepen_hint"] is True + + def test_upgraded_row_with_skim_char_in_note_not_registered(self, fake_kb): + # partial re-ingest 升级后的 ✓ 行，备注含「由 ⊙ 升级」不得重新注册为扫读 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_md( + fake_kb / "wiki" / "sources" / "foo_s.md", + standard_fm(type="source_summary", source_count=1, sources=['[[raw/papers/foo]]']), + "# 摘要\n\n> **原始文件**: [[raw/papers/foo]]\n\n## 章节深度登记\n\n" + "| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + "| ^h-2-2-aaaa03 | 2 Approach | ✓ 深读 | 2026-07-03 由 ⊙ 升级(query 触发) |\n", + ) + pages = k.load_all_wiki_pages() + hits = k.search_raw("mutual indexing", pages=pages) + assert hits and hits[0]["deepen_hint"] is False + + def test_frontmatter_sources_binding_beats_body_order(self, fake_kb): + # 对照引用挪到「原始文件」行之前时，绑定仍以 frontmatter sources 为准 + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_raw(fake_kb, "raw/papers/other.md", + "## 2 Approach ^h-2-1-eeee01\n\n完全无关但同名章节 something。 ^p-1-ffff01\n") + write_md( + fake_kb / "wiki" / "sources" / "foo_s.md", + standard_fm(type="source_summary", source_count=1, sources=['[[raw/papers/foo]]']), + "# 摘要\n\n对照见 [[raw/papers/other#^p-1-ffff01]]\n\n" + "> **原始文件**: [[raw/papers/foo]]\n\n## 章节深度登记\n\n" + "| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + "| ^h-2-2-aaaa03 | 2 Approach | ⊙ 扫读 | x |\n", + ) + pages = k.load_all_wiki_pages() + # other.md 的同名章节不得被误标（若按正文首链接绑定就会挂到 other 名下） + hits = k.search_raw("something", pages=pages) + assert hits and hits[0]["deepen_hint"] is False + hits2 = k.search_raw("mutual indexing", pages=pages) + assert hits2 and hits2[0]["deepen_hint"] is True + + def test_archived_registry_ignored(self, fake_kb): + write_raw(fake_kb, "raw/papers/foo.md", RAW_DOC) + write_md( + fake_kb / "wiki" / "_archive_v0" / "foo_s.md", + standard_fm(type="source_summary", source_count=1, sources=['[[raw/papers/foo]]']), + "# 旧摘要\n\n> **原始文件**: [[raw/papers/foo]]\n\n## 章节深度登记\n\n" + "| Anchor | 原标题 | 状态 | 备注 |\n|---|---|---|---|\n" + "| ^h-2-2-aaaa03 | 2 Approach | ⊙ 扫读 | x |\n", + ) + pages = k.load_all_wiki_pages() + hits = k.search_raw("mutual indexing", pages=pages) + assert hits and hits[0]["deepen_hint"] is False + + def test_line_number_includes_frontmatter_offset(self, fake_kb): + # wiki 页带 frontmatter：报告的行号应是文件绝对行，不是 body 相对行 + body = "# X\n\n独特词 uniqward 在这里。\n" + write_md(fake_kb / "wiki" / "concepts" / "c.md", standard_fm(), body) + full = (fake_kb / "wiki" / "concepts" / "c.md").read_text(encoding="utf-8") + expect_line = next(i + 1 for i, ln in enumerate(full.split("\n")) if "uniqward" in ln) + hits = k.search_raw("uniqward", pages=[], include_wiki=True) + assert hits and hits[0]["line"] == expect_line + + def test_phrase_bonus_survives_whitespace(self, fake_kb): + # query 多余空白 / 原文换行不让短语 +5 失效 + write_raw(fake_kb, "raw/papers/foo.md", + "# T ^h-1-1-aaaa01\n\nalpha\nbeta 相邻但跨行。 ^p-1-bbbb01\n\n" + "只有 alpha 和别处的 beta 分离。 ^p-2-bbbb02\n") + hits = k.search_raw("alpha beta", pages=[]) + assert hits[0]["anchor"] == "p-1-bbbb01" diff --git a/scripts/tests/test_longdoc_eval.py b/scripts/tests/test_longdoc_eval.py new file mode 100644 index 0000000..bf11e3d --- /dev/null +++ b/scripts/tests/test_longdoc_eval.py @@ -0,0 +1,342 @@ +"""长文档 ingest / retrieval 开发集守护。""" +from __future__ import annotations + +import importlib.util +import json +import sqlite3 +import sys +import types +from pathlib import Path + +import pytest + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent + + +def _load(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + assert spec and spec.loader + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +CORPUS = _load("longdoc_corpus_test", PROJECT_ROOT / "evals" / "longdoc" / "corpus.py") +RUNNER = _load("longdoc_runner_test", PROJECT_ROOT / "evals" / "longdoc" / "run_eval.py") +RETRIEVAL = _load( + "retrieval_index_longdoc_test", PROJECT_ROOT / "scripts" / "retrieval_index.py" +) + + +class DelegatingApi: + """记录协议调用并委托真实索引；被测 workspace 内不允许存在 Gold。""" + + def __init__(self): + self.expansions_seen: list[list[str]] = [] + + def rebuild_index(self, workspace_root: Path, db_path: Path | None = None): + assert db_path is not None + assert not (Path(workspace_root) / ".eval-gold.json").exists() + return RETRIEVAL.rebuild_index(workspace_root, db_path=db_path) + + def coverage_report(self, db_path: Path): + return RETRIEVAL.coverage_report(db_path) + + def search_evidence(self, db_path: Path, query: str, limit: int = 20, + expansions=None): + self.expansions_seen.append(list(expansions or [])) + return RETRIEVAL.search_evidence( + db_path, query, limit=limit, expansions=expansions + ) + + +class TestLongDocCorpus: + def test_gold_has_required_slices_and_is_not_derived_from_predictions(self): + units, cases = CORPUS.build_fixture() + assert len(cases) == 50 + assert len(units) == 120 + assert {case.category for case in cases} == { + "technical_manual", "policy_terms", "bilingual", "table", "multi_hop" + } + assert all(case.required_facets and case.minimal_evidence_sets for case in cases) + assert all(case.forbidden_evidence_ids for case in cases) + assert sum(case.category == "multi_hop" for case in cases) == 10 + assert all(len(case.required_facets) == 2 + for case in cases if case.category == "multi_hop") + assert all(len(case.minimal_evidence_sets) == 2 + for case in cases if case.category == "bilingual") + assert all("same-value-wrong-subject" in case.tags for case in cases) + assert CORPUS._fixture_definition_sha256(units, cases) == ( + CORPUS.FROZEN_FIXTURE_DEFINITION_SHA256 + ) + + def test_materialized_documents_are_long_and_positions_really_move(self, tmp_path): + baseline = CORPUS.materialize(tmp_path / "a", "baseline") + moved = CORPUS.materialize(tmp_path / "b", "position_moved") + assert baseline["license"] == "CC0-1.0" + assert baseline["expected"] == moved["expected"] + assert baseline["expected"]["documents"] == 6 + assert baseline["expected"]["natural_units"] == CORPUS.EXPECTED_NATURAL_UNIT_COUNT + assert baseline["expected"]["content_sections"] == CORPUS.EXPECTED_CONTENT_SECTION_COUNT + assert baseline["expected"]["natural_unit_types"] == dict( + sorted(CORPUS.EXPECTED_NATURAL_UNIT_TYPES.items()) + ) + assert len(baseline["expected_inventory"]) == CORPUS.EXPECTED_NATURAL_UNIT_COUNT + assert len(baseline["expected_section_inventory"]) == ( + CORPUS.EXPECTED_CONTENT_SECTION_COUNT + ) + assert all( + row["anchor"] and row["owning_section_anchor"] + and len(row["heading_path"]) == 2 + and len(row["heading_anchors"]) == 2 + for row in baseline["expected_inventory"] + ) + assert not (Path(baseline["workspace_root"]) / ".eval-gold.json").exists() + assert not (Path(moved["workspace_root"]) / ".eval-gold.json").exists() + assert all(row["chars"] > 150_000 for row in baseline["documents"]) + assert all(row["content_sections"] == 6 for row in baseline["documents"]) + assert all(row["min_content_section_chars"] > 30_000 + for row in baseline["documents"]) + assert {row["position_bucket"] for row in baseline["evidence"].values()} == { + "first", "middle", "tail" + } + assert {row["position_bucket"] for row in moved["evidence"].values()} == { + "first", "middle", "tail" + } + # 相同事实的内容 hash 保持，seq/位置变化使 canonical anchor 改变。 + changed = sum( + baseline["evidence"][key]["canonical_ref"] != + moved["evidence"][key]["canonical_ref"] + for key in baseline["evidence"] + ) + assert changed == len(baseline["evidence"]) + assert all( + baseline["evidence"][key]["section_slot"] != + moved["evidence"][key]["section_slot"] + for key in baseline["evidence"] + ) + + def test_unknown_variant_and_empty_denominator_fail_closed(self, tmp_path): + with pytest.raises(ValueError): + CORPUS.materialize(tmp_path, "unregistered") + with pytest.raises(RUNNER.EvaluationProtocolError): + RUNNER._metric(0, 0) + + +class TestLongDocRunnerProtocol: + @staticmethod + def _hit(*, unit_id="u1", text="evidence", anchor="p-1-abc123", + kind="paragraph", subordinal=1, score=1.0): + return { + "path": "raw/a.md", "anchor": anchor, + "canonical_ref": f"raw/a.md#^{anchor}", "unit_id": unit_id, + "kind": kind, "subordinal": subordinal, + "content_hash": RUNNER._content_hash(text), + "text": text, "score": score, + } + + def test_case_id_category_and_denominator_are_frozen(self, tmp_path, monkeypatch): + original = RUNNER.corpus.materialize + + def truncated(root, variant): + manifest = original(root, variant) + manifest["cases"] = manifest["cases"][:1] + return manifest + + monkeypatch.setattr(RUNNER.corpus, "materialize", truncated) + with pytest.raises(RUNNER.EvaluationProtocolError, match="冻结的 50 cases"): + RUNNER.evaluate(DelegatingApi(), tmp_path) + + def test_runner_import_isolated_from_other_top_level_corpus_module(self, monkeypatch): + fake = types.ModuleType("corpus") + fake.VARIANTS = ("wrong-module",) + monkeypatch.setitem(sys.modules, "corpus", fake) + fresh = _load( + "longdoc_runner_import_collision_test", + PROJECT_ROOT / "evals" / "longdoc" / "run_eval.py", + ) + assert fresh.corpus.__name__ == "evals.longdoc.corpus" + assert fresh.corpus.VARIANTS == ("baseline", "position_moved") + + def test_api_coverage_denominator_cannot_replace_fixture_gold(self, tmp_path): + class WrongDenominator(DelegatingApi): + def coverage_report(self, db_path): + result = super().coverage_report(db_path) + result["natural_units"]["expected"] -= 1 + result["natural_units"]["indexed"] -= 1 + return result + + with pytest.raises(RUNNER.EvaluationProtocolError, match="独立 fixture Gold"): + RUNNER.evaluate(WrongDenominator(), tmp_path) + + def test_missing_list_must_reconcile_with_counts(self, tmp_path): + class HiddenMissing(DelegatingApi): + def coverage_report(self, db_path): + result = super().coverage_report(db_path) + result["content_sections"]["registered"] -= 1 + result["content_sections"]["coverage"] = ( + result["content_sections"]["registered"] / + result["content_sections"]["expected"] + ) + return result + + with pytest.raises(RUNNER.EvaluationProtocolError, match="missing"): + RUNNER.evaluate(HiddenMissing(), tmp_path) + + def test_independent_inventory_catches_drop_even_if_api_fakes_100_percent(self, tmp_path): + class DroppedUnit(DelegatingApi): + def rebuild_index(self, workspace_root, db_path=None): + result = super().rebuild_index(workspace_root, db_path=db_path) + connection = sqlite3.connect(db_path) + connection.execute( + "DELETE FROM units WHERE unit_id=(SELECT unit_id FROM units ORDER BY unit_id LIMIT 1)" + ) + connection.commit() + connection.close() + return result + + def coverage_report(self, db_path): + result = super().coverage_report(db_path) + row = result["natural_units"] + row.update({"indexed": row["expected"], "missing": 0, + "missing_items": [], "coverage": 1.0}) + return result + + with pytest.raises(RUNNER.EvaluationProtocolError, match="精确 inventory"): + RUNNER.evaluate(DroppedUnit(), tmp_path) + + def test_independent_oracle_catches_synchronized_route_and_section_poison( + self, tmp_path, + ): + manifest = CORPUS.materialize(tmp_path / "fixture", "baseline") + workspace = Path(manifest["workspace_root"]) + db_path = workspace / ".cache" / "retrieval_index.db" + RETRIEVAL.rebuild_index(workspace, db_path=db_path) + + unit_gold = manifest["expected_inventory"][0] + victim = unit_gold["unit_id"] + poison_path = json.dumps(["FORGED ROUTING METADATA"]) + poison_anchors = json.dumps(["h-2-99-ffffff"]) + connection = sqlite3.connect(db_path) + connection.execute( + """UPDATE expected_units + SET heading_path_json=?, heading_anchors_json=? + WHERE unit_id=?""", + (poison_path, poison_anchors, victim), + ) + connection.execute( + """UPDATE units + SET heading_path_json=?, heading_anchors_json=? + WHERE unit_id=?""", + (poison_path, poison_anchors, victim), + ) + for table in ("unit_fts_unicode", "unit_fts_trigram"): + connection.execute( + f"UPDATE {table} SET heading_path=? WHERE unit_id=?", + ("FORGED ROUTING METADATA", victim), + ) + connection.commit() + connection.close() + + with pytest.raises(RUNNER.EvaluationProtocolError, match="生成规则不一致"): + RUNNER._validate_exact_inventory(db_path, manifest) + + connection = sqlite3.connect(db_path) + connection.execute( + """UPDATE units + SET heading_path_json=?, heading_anchors_json=? + WHERE unit_id=?""", + ( + json.dumps(unit_gold["heading_path"]), + json.dumps(unit_gold["heading_anchors"]), + victim, + ), + ) + section = manifest["expected_section_inventory"][0] + section_id = section["section_id"] + forged_hash = "f" * 64 + connection.execute( + """UPDATE sections SET anchor='h-2-99-ffffff', + title='FORGED SECTION', content_hash=? + WHERE section_id=?""", + (forged_hash, section_id), + ) + connection.execute( + """UPDATE expected_structural_sections + SET anchor='h-2-99-ffffff', title='FORGED SECTION', content_hash=? + WHERE section_id=?""", + (forged_hash, section_id), + ) + connection.execute( + """UPDATE expected_sections + SET anchor='h-2-99-ffffff', title='FORGED SECTION', content_hash=? + WHERE section_id=?""", + (forged_hash, section_id), + ) + connection.commit() + connection.close() + + with pytest.raises(RUNNER.EvaluationProtocolError, match="内容章节"): + RUNNER._validate_exact_inventory(db_path, manifest) + + def test_noncanonical_or_duplicate_search_hits_fail_closed(self): + base = self._hit() + bad = dict(base, canonical_ref="./raw/a.md#^p-1-abc123") + with pytest.raises(RUNNER.EvaluationProtocolError, match="非 canonical"): + RUNNER._validate_search_result({"hits": [bad]}, limit=20) + with pytest.raises(RUNNER.EvaluationProtocolError, match="重复 unit_id"): + RUNNER._validate_search_result({"hits": [base, dict(base)]}, limit=20) + + def test_forged_text_or_hash_cannot_pass(self): + base = self._hit() + bad_text = dict(base, text="forged evidence") + with pytest.raises(RUNNER.EvaluationProtocolError, match="text/content_hash"): + RUNNER._validate_search_result({"hits": [bad_text]}, limit=20) + + forged = self._hit(text="forged evidence") + inventory = {base["unit_id"]: dict(base)} + with pytest.raises(RUNNER.EvaluationProtocolError, match="物化 inventory"): + RUNNER._validate_search_result( + {"hits": [forged]}, limit=20, exact_inventory=inventory + ) + + def test_same_parent_anchor_different_rows_keep_distinct_handles(self): + first = self._hit(unit_id="row-1", text="| A | 87.5% |", + anchor="t-1-abc123", kind="table_row", subordinal=1) + second = self._hit(unit_id="row-2", text="| B | 87.5% |", + anchor="t-1-abc123", kind="table_row", subordinal=2) + inventory = {row["unit_id"]: dict(row) for row in (first, second)} + hits = RUNNER._validate_search_result( + {"hits": [first, second]}, limit=20, exact_inventory=inventory + ) + assert hits[0]["canonical_ref"] == hits[1]["canonical_ref"] + assert RUNNER._handle_key(hits[0]) != RUNNER._handle_key(hits[1]) + + def test_nan_score_and_excess_topk_fail_closed(self): + base = self._hit(score=float("nan")) + with pytest.raises(RUNNER.EvaluationProtocolError, match="有限数值"): + RUNNER._validate_search_result({"hits": [base]}, limit=20) + with pytest.raises(RUNNER.EvaluationProtocolError, match="超过"): + RUNNER._validate_search_result({"hits": [{}] * 21}, limit=20) + + +def test_real_retrieval_index_meets_longdoc_development_thresholds(tmp_path): + """真实索引端到端守护；主 CES 不得注入 Gold facet expansions。""" + api = DelegatingApi() + result = RUNNER.evaluate(api, tmp_path) + assert api.expansions_seen and all(expansions == [] for expansions in api.expansions_seen) + assert result["status"] == "passed", json.dumps( + {name: row for name, row in result["thresholds"].items() if not row["passed"]}, + ensure_ascii=False, indent=2, + ) + assert result["metrics"]["natural_unit_index_coverage"]["value"] == 1.0 + assert result["metrics"]["content_section_registration_coverage"]["value"] == 1.0 + assert result["metrics"]["fixed_regression_pass_rate"]["value"] == 1.0 + assert result["metrics"]["complete_evidence_set_recall_at_20"]["value"] >= 0.98 + assert result["metrics"]["selected_citation_precision"]["value"] >= 0.99 + assert result["metrics"]["selected_citation_precision"]["denominator"] == 60 + # precision 必须与 coverage 一起交付，不能只断言前者。 + assert result["metrics"]["answer_coverage"]["value"] >= 0.98 + assert result["metrics"]["fully_grounded_coverage"]["value"] >= 0.98 + assert result["evaluation_scope"].startswith("retrieval-only") diff --git a/scripts/tests/test_outline_validator.py b/scripts/tests/test_outline_validator.py new file mode 100644 index 0000000..18a9c61 --- /dev/null +++ b/scripts/tests/test_outline_validator.py @@ -0,0 +1,129 @@ +"""outline.json 内容地址化与结构校验回归测试。""" +from __future__ import annotations + +import copy + +import pytest + +from postprocess import OUTLINE_SCHEMA_VERSION, process, validate_outline + + +DOCUMENT = """# Book + +Introduction. + +## One + +First section. + +### Detail + +Nested detail. + +## Two + +Second section. +""" + + +def _pair(): + return process(DOCUMENT, "raw/books/book.md") + + +def _codes(outline, text): + return { + issue["code"] + for issue in validate_outline( + outline, + text, + expected_doc_path="raw/books/book.md", + ) + } + + +def test_generated_outline_is_valid(): + text, outline = _pair() + assert outline["outline_schema_version"] == OUTLINE_SCHEMA_VERSION + assert validate_outline( + outline, text, expected_doc_path="raw/books/book.md" + ) == [] + + +@pytest.mark.parametrize( + ("mutate", "expected_code"), + [ + (lambda o: o.pop("outline_schema_version"), "outline-schema-version-mismatch"), + (lambda o: o.__setitem__("outline_schema_version", 1), "outline-schema-version-mismatch"), + (lambda o: o.pop("doc_sha256"), "doc-sha256-invalid"), + (lambda o: o.__setitem__("doc_sha256", "0" * 64), "doc-sha256-mismatch"), + (lambda o: o.__setitem__("doc_chars", o["doc_chars"] - 1), "doc-chars-mismatch"), + (lambda o: o.__setitem__("doc_path", "raw/books/other.md"), "doc-path-mismatch"), + ], +) +def test_document_contract_failures_are_reported(mutate, expected_code): + text, original = _pair() + outline = copy.deepcopy(original) + mutate(outline) + assert expected_code in _codes(outline, text) + + +def test_missing_and_wrong_section_hash_are_reported(): + text, original = _pair() + + missing = copy.deepcopy(original) + missing["sections"][0]["children"][0].pop("section_sha256") + assert "section-sha256-invalid" in _codes(missing, text) + + wrong = copy.deepcopy(original) + wrong["sections"][0]["children"][0]["section_sha256"] = "0" * 64 + assert "section-sha256-mismatch" in _codes(wrong, text) + + +def test_duplicate_anchor_is_reported(): + text, outline = _pair() + outline = copy.deepcopy(outline) + children = outline["sections"][0]["children"] + children[1]["anchor"] = children[0]["anchor"] + assert "section-anchor-duplicate" in _codes(outline, text) + + +def test_out_of_bounds_range_is_reported(): + text, outline = _pair() + outline = copy.deepcopy(outline) + outline["sections"][0]["char_end"] = len(text) + 1 + assert "section-range-out-of-bounds" in _codes(outline, text) + + +def test_overlapping_siblings_are_reported(): + text, outline = _pair() + outline = copy.deepcopy(outline) + children = outline["sections"][0]["children"] + children[0]["char_end"] = children[1]["char_end"] + assert "section-sibling-overlap" in _codes(outline, text) + + +def test_child_outside_parent_is_reported(): + text, outline = _pair() + outline = copy.deepcopy(outline) + parent = outline["sections"][0]["children"][0] + child = parent["children"][0] + child["char_start"] = parent["char_start"] + assert "section-child-containment" in _codes(outline, text) + + +def test_missing_section_is_not_a_vacuously_valid_tree(): + text, outline = _pair() + outline = copy.deepcopy(outline) + outline["sections"][0]["children"].pop() + assert "section-set-mismatch" in _codes(outline, text) + + +def test_same_length_body_rewrite_invalidates_document_and_section_hashes(): + text, outline = _pair() + edited = text.replace("First section.", "Other section.") + assert len(edited) == len(text) + codes = _codes(outline, edited) + assert "doc-chars-mismatch" not in codes + assert "doc-sha256-mismatch" in codes + assert "section-sha256-mismatch" in codes + assert "markdown-anchors-not-canonical" in codes diff --git a/scripts/tests/test_pdf_layout_conversion.py b/scripts/tests/test_pdf_layout_conversion.py new file mode 100644 index 0000000..5135631 --- /dev/null +++ b/scripts/tests/test_pdf_layout_conversion.py @@ -0,0 +1,1274 @@ +"""Layout-aware PDF extraction regression tests. + +Fixtures are generated from first-party strings at test time and are not +committed as binary artifacts. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +import convert +import pdf_layout +from pdf_layout import PDFLayoutExtractionError, extract_pdf_to_markdown + + +reportlab = pytest.importorskip("reportlab") + + +def _make_two_column_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica-Bold", 16) + c.drawString(48, 740, "Layout Calibration") + c.setFont("Helvetica", 10) + for index, text in enumerate(("LEFT-START alpha", "LEFT-MIDDLE beta", "LEFT-END gamma")): + c.drawString(48, 700 - index * 24, text) + for index, text in enumerate(("RIGHT-START delta", "RIGHT-MIDDLE epsilon", "RIGHT-END zeta")): + c.drawString(326, 706 - index * 24, text) + c.save() + + +def _make_aligned_two_column_pdf(path: Path) -> None: + """Both independent columns share baselines; this is not a key/value form.""" + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 10) + left = ( + "LEFT-A independent operational sentence.", + "LEFT-B second operational sentence.", + "LEFT-C third operational sentence.", + ) + right = ( + "RIGHT-A separate verification sentence.", + "RIGHT-B another verification sentence.", + "RIGHT-C final verification sentence.", + ) + for index, (left_text, right_text) in enumerate(zip(left, right)): + y = 700 - index * 20 + c.drawString(48, y, left_text) + c.drawString(326, y, right_text) + c.save() + + +def _make_wide_key_value_pdf(path: Path) -> None: + """Short labels plus wide descriptions must remain paired row-by-row.""" + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Owner", "Platform reliability group approves the release request"), + ("Retention", "Operational evidence remains available for seven years"), + ("Escalation", "Manual review begins when either threshold is exceeded"), + ) + for index, (label, description) in enumerate(rows): + y = 700 - index * 22 + c.setFont("Helvetica-Bold", 10) + c.drawString(48, y, label) + c.setFont("Helvetica", 10) + c.drawString(326, y, description) + c.save() + + +def _make_single_explicit_key_value_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica-Bold", 10) + c.drawString(48, 700, "Owner:") + c.setFont("Helvetica", 10) + c.drawString(180, 700, "Platform reliability group") + c.save() + + +def _make_wrapped_key_value_pdf( + path: Path, *, explicit_colons: bool, ambiguous_wrap: bool = False +) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Owner", "Platform reliability group approves", "the release request after review"), + ("Retention", "Operational evidence remains", "available for seven years"), + ("Escalation", "Manual review begins", "when either threshold is exceeded"), + ) + for index, (label, first, continuation) in enumerate(rows): + y = 700 - index * 48 + c.setFont("Helvetica-Bold", 10) + c.drawString(48, y, label + (":" if explicit_colons else "")) + c.setFont("Helvetica", 10) + c.drawString(260, y, first) + continuation_y = y - (30 if ambiguous_wrap else 13) + c.drawString(272, continuation_y, continuation) + c.save() + + +def _make_ambiguous_aligned_narrow_layout_pdf(path: Path) -> None: + """Geometry alone cannot distinguish these rows from narrow columns.""" + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 10) + rows = ( + ("Primary owner", "Platform reliability group approves the release request"), + ("Retention term", "Operational evidence remains available for seven years"), + ("Review route", "Manual review begins when either threshold is exceeded"), + ) + for index, (left, right) in enumerate(rows): + y = 700 - index * 22 + c.drawString(48, y, left) + c.drawString(326, y, right) + c.save() + + +def _make_narrow_sentence_columns_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 10) + left = ("Alpha note ends.", "Beta note follows.", "Gamma note closes.") + right = ( + "Right verification sentence ends.", + "Another independent sentence follows.", + "Final independent sentence closes.", + ) + for index, (left_text, right_text) in enumerate(zip(left, right)): + y = 700 - index * 22 + c.drawString(48, y, left_text) + c.drawString(326, y, right_text) + c.save() + + +def _make_long_ambiguous_labels_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 10) + rows = ( + ("Primary operational responsibility", "Platform group approves release"), + ("Evidence preservation requirement", "Records remain available long term"), + ("Exceptional escalation procedure", "Manual review begins after threshold"), + ) + for index, (left, right) in enumerate(rows): + y = 700 - index * 22 + c.drawString(48, y, left) + c.drawString(326, y, right) + c.save() + + +def _make_two_column_bold_first_row_prose_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Left lead ends.", "Right lead ends."), + ("Left detail follows.", "Right detail follows."), + ("Left close finishes.", "Right close finishes."), + ) + for index, (left, right) in enumerate(rows): + c.setFont("Helvetica-Bold" if index == 0 else "Helvetica", 10) + y = 700 - index * 22 + c.drawString(48, y, left) + c.drawString(326, y, right) + c.save() + + +def _make_three_column_independent_prose_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Alpha lead ends.", "Beta lead ends.", "Gamma lead ends."), + ("Alpha detail ends.", "Beta detail ends.", "Gamma detail ends."), + ("Alpha close ends.", "Beta close ends.", "Gamma close ends."), + ) + for row_index, row in enumerate(rows): + c.setFont("Helvetica-Bold" if row_index == 0 else "Helvetica", 9) + y = 700 - row_index * 22 + for x, text in zip((48, 230, 420), row): + c.drawString(x, y, text) + c.save() + + +def _make_table_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.lib import colors + from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph + from reportlab.lib.styles import getSampleStyleSheet + + doc = SimpleDocTemplate(str(path), pagesize=letter) + table = Table([ + ["Unit", "Pressure", "Decision"], + ["Atlas-7", "2.45 bar", "ACCEPT"], + ["Boreal-9", "3.10 bar", "REJECT"], + ]) + table.setStyle(TableStyle([ + ("GRID", (0, 0), (-1, -1), 0.75, colors.black), + ("FONTNAME", (0, 0), (-1, 0), "Helvetica-Bold"), + ])) + doc.build([Paragraph("Acceptance register", getSampleStyleSheet()["Heading1"]), table]) + + +def _make_cell_rectangle_table_pdf(path: Path) -> None: + """Draw every ruled cell as its own rectangle instead of shared grid lines.""" + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Unit", "Pressure", "Decision"), + ("Atlas-7", "2.45 bar", "ACCEPT"), + ("Boreal-9", "3.10 bar", "REJECT"), + ) + x_starts = (48.0, 228.0, 388.0) + widths = (180.0, 160.0, 160.0) + row_height = 28.0 + table_top = 700.0 + for row_index, row in enumerate(rows): + bottom = table_top - (row_index + 1) * row_height + for x, width, text in zip(x_starts, widths, row): + c.rect(x, bottom, width, row_height, stroke=1, fill=0) + c.setFont("Helvetica-Bold" if row_index == 0 else "Helvetica", 10) + c.drawString(x + 6, bottom + 9, text) + c.save() + + +def _make_cell_table_with_inner_vector(path: Path, kind: str) -> None: + """Draw a ruled table whose lower-right cell contains vector semantics.""" + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + x_starts = (48.0, 298.0) + widths = (250.0, 250.0) + row_heights = (44.0, 44.0, 180.0) + rows = ( + ("Unit", "Decision"), + ("Atlas-7", "ACCEPT"), + ("Boreal-9", "Trend"), + ) + table_top = 700.0 + current_top = table_top + cell_boxes: list[list[tuple[float, float, float, float]]] = [] + for row_index, (row, height) in enumerate(zip(rows, row_heights)): + bottom = current_top - height + row_boxes = [] + for x, width, text in zip(x_starts, widths, row): + c.rect(x, bottom, width, height, stroke=1, fill=0) + c.setFont("Helvetica-Bold" if row_index == 0 else "Helvetica", 10) + c.drawString(x + 6, current_top - 17, text) + row_boxes.append((x, bottom, x + width, current_top)) + cell_boxes.append(row_boxes) + current_top = bottom + + x0, bottom, x1, top = cell_boxes[-1][-1] + if kind == "step": + # Axes plus a seven-segment orthogonal step trace. + c.line(x0 + 24, bottom + 28, x0 + 24, top - 48) + c.line(x0 + 24, bottom + 28, x1 - 20, bottom + 28) + points = ( + (x0 + 36, bottom + 46), + (x0 + 78, bottom + 46), + (x0 + 78, bottom + 78), + (x0 + 125, bottom + 78), + (x0 + 125, bottom + 112), + (x0 + 172, bottom + 112), + (x0 + 172, bottom + 142), + (x1 - 24, bottom + 142), + ) + for first, second in zip(points, points[1:]): + c.line(first[0], first[1], second[0], second[1]) + elif kind == "bars": + for index, height in enumerate((35, 65, 95, 135)): + c.rect(x0 + 28 + index * 48, bottom + 24, 24, height, stroke=0, fill=1) + elif kind == "near-panel": + c.rect(x0 + 1, bottom + 1, (x1 - x0) - 2, (top - bottom) - 2, stroke=1, fill=0) + elif kind == "semantic-fill": + # A fill exactly covering the middle-right data cell encodes a decision + # that has no text representation. + fill_x0, fill_bottom, fill_x1, fill_top = cell_boxes[1][-1] + c.setFillColorRGB(0.1, 0.7, 0.2) + c.rect( + fill_x0, + fill_bottom, + fill_x1 - fill_x0, + fill_top - fill_bottom, + stroke=0, + fill=1, + ) + else: # pragma: no cover - helper contract + raise ValueError(kind) + c.save() + + +def _make_borderless_table_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + rows = ( + ("Unit", "Pressure", "Decision"), + ("Cedar-4", "1.85 bar", "HOLD"), + ("Delta-8", "2.20 bar", "RELEASE"), + ) + for row_index, row in enumerate(rows): + c.setFont("Helvetica-Bold" if row_index == 0 else "Helvetica", 10) + y = 700 - row_index * 22 + for x, cell in zip((48, 230, 420), row): + c.drawString(x, y, cell) + c.save() + + +def _make_running_furniture_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + for page_number in range(1, 4): + c.setFont("Helvetica", 9) + c.drawString(48, 770, "REPEATED TECHNICAL HEADER") + c.drawCentredString(306, 22, f"Page {page_number} of 3") + c.setFont("Helvetica", 11) + c.drawString(48, 700, f"BODY-{page_number} retained evidence") + c.showPage() + c.save() + + +def _make_numbered_top_content_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + for page_number in range(1, 3): + c.setFont("Helvetica-Bold", 12) + c.drawString(48, 770, f"Experiment {page_number} threshold") + c.setFont("Helvetica", 10) + c.drawString(48, 700, f"EXPERIMENT-BODY-{page_number}") + c.showPage() + c.save() + + +def _make_single_page_year_footer_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 700, "Publication record remains available.") + c.setFont("Helvetica", 9) + c.drawCentredString(306, 22, "2026") + c.save() + + +def _make_bare_page_series_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + for page_number in (1, 2): + c.setFont("Helvetica", 11) + c.drawString(48, 700, f"SERIES-BODY-{page_number}") + c.setFont("Helvetica", 9) + c.drawCentredString(306, 22, str(page_number)) + c.showPage() + c.save() + + +def _make_non_page_numeric_footers_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + for page_number, metric in ((1, "95"), (2, "97")): + c.setFont("Helvetica", 11) + c.drawString(48, 700, f"METRIC-BODY-{page_number}") + c.setFont("Helvetica", 9) + c.drawCentredString(306, 22, metric) + c.showPage() + c.save() + + +def _make_image_only_pdf(path: Path) -> None: + from io import BytesIO + from reportlab.lib.pagesizes import letter + from reportlab.lib.utils import ImageReader + from reportlab.pdfgen import canvas + from PIL import Image + + image = Image.new("RGB", (32, 32), "white") + buffer = BytesIO() + image.save(buffer, format="PNG") + buffer.seek(0) + c = canvas.Canvas(str(path), pagesize=letter) + c.drawImage(ImageReader(buffer), 48, 700, width=32, height=32) + c.save() + + +def _make_mixed_content_image_pdf(path: Path) -> None: + from io import BytesIO + from reportlab.lib.pagesizes import letter + from reportlab.lib.utils import ImageReader + from reportlab.pdfgen import canvas + from PIL import Image, ImageDraw + + image = Image.new("RGB", (480, 300), "white") + draw = ImageDraw.Draw(image) + draw.rectangle((20, 20, 460, 280), outline="black", width=4) + draw.text((45, 120), "CHART-ONLY-FACT: 83.4", fill="black") + buffer = BytesIO() + image.save(buffer, format="PNG") + buffer.seek(0) + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, "Figure 1 contains the calibration result.") + c.drawImage(ImageReader(buffer), 48, 430, width=360, height=225) + c.save() + + +def _make_vector_figure_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, "Figure 2 contains vector-only calibration facts.") + # A substantive vector panel plus bars. No figure labels are available as + # extractable text, so accepting only the caption would be lossy. + c.roundRect(80, 430, 360, 230, 8, stroke=1, fill=0) + for index, height in enumerate((45, 80, 120, 155)): + c.rect(125 + index * 65, 455, 34, height, stroke=1, fill=1) + c.save() + + +def _make_line_only_vector_plot_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, "Figure 3 is a line-only vector plot.") + c.line(95, 440, 95, 650) + c.line(95, 440, 445, 440) + points = ((95, 465), (170, 510), (245, 485), (325, 590), (420, 625)) + for first, second in zip(points, points[1:]): + c.line(first[0], first[1], second[0], second[1]) + c.save() + + +def _make_single_diagonal_trend_pdf(path: Path, *, material: bool) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, "A diagonal vector mark follows.") + if material: + c.line(90, 440, 450, 640) + else: + c.line(520, 700, 540, 715) + c.save() + + +def _make_separated_vector_aggregate_pdf(path: Path, kind: str) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, f"Separated vector fixture: {kind}.") + if kind == "panels": + for row in range(2): + for column in range(3): + c.rect(60 + column * 170, 470 + row * 100, 70, 50, stroke=1, fill=0) + elif kind == "timeline": + c.line(80, 520, 520, 520) + for x in (100, 190, 280, 370, 460): + c.line(x, 508, x, 532) + elif kind == "diagonals": + for index in range(4): + x = 60 + index * 130 + y = 470 + (index % 2) * 100 + c.line(x, y, x + 70, y + 50) + else: # pragma: no cover - test helper contract + raise ValueError(kind) + c.save() + + +def _make_separator_and_small_decoration_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 710, "SECTION-A retained") + c.line(48, 690, 560, 690) + c.rect(540, 735, 12, 12, stroke=1, fill=0) + c.drawString(48, 660, "SECTION-B retained") + c.save() + + +def _make_scatter_cloud_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 730, "A vector-only scatter cloud follows.") + for row in range(5): + for column in range(8): + x = 90 + column * 55 + (row % 2) * 9 + y = 455 + row * 38 + c.circle(x, y, 4.5, stroke=1, fill=0) + c.save() + + +def _make_vector_bullet_list_pdf( + path: Path, *, columns: int, count: int | None = None, marker: str = "circle" +) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + for column in range(columns): + bullet_x = 48 + column * 278 + text_x = bullet_x + 13 + item_count = count if count is not None else (12 if columns == 1 else 6) + prefix = "ITEM" if columns == 1 else ("LEFT" if column == 0 else "RIGHT") + for index in range(item_count): + y = 710 - index * 25 + if marker == "circle": + c.circle(bullet_x, y + 3, 3, stroke=1, fill=0) + elif marker == "square": + c.rect(bullet_x - 3, y, 6, 6, stroke=1, fill=1) + else: # pragma: no cover - helper contract + raise ValueError(marker) + c.setFont("Helvetica", 10) + c.drawString(text_x, y, f"{prefix}-{index + 1} is retained as a list item.") + c.save() + + +def _make_heading_discriminator_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica-Bold", 12) + c.drawString(48, 720, "Operating limits") + c.setFont("Helvetica", 10) + c.drawString(48, 690, "Ordinary body text establishes the dominant body size here.") + c.setFont("Helvetica-Bold", 12) + c.drawString(48, 650, "What does this mean?") + c.setFont("Helvetica", 10) + c.drawString(48, 620, "A body paragraph follows the compact question heading.") + c.setFont("Helvetica-Bold", 12) + c.drawString(48, 580, "System halted.") + c.setFont("Helvetica", 10) + c.drawString(48, 550, "More ordinary body text follows the emphasized factual claim.") + c.setFont("Helvetica-Bold", 16) + c.drawString(48, 505, "Strong visual title.") + c.setFont("Helvetica", 10) + c.drawString(48, 470, "Body text follows the strongly oversized title.") + c.save() + + +def _make_caption_table_pdf(path: Path) -> None: + from reportlab.lib import colors + from reportlab.lib.pagesizes import letter + from reportlab.platypus import SimpleDocTemplate, Table, TableStyle + + table = Table([ + ["Acceptance register", "", ""], + ["Unit", "Pressure", "Decision"], + ["Atlas-7", "2.45 bar", "ACCEPT"], + ["Boreal-9", "3.10 bar", "REJECT"], + ]) + table.setStyle(TableStyle([ + ("SPAN", (0, 0), (-1, 0)), + ("GRID", (0, 0), (-1, -1), 0.75, colors.black), + ("FONTNAME", (0, 1), (-1, 1), "Helvetica-Bold"), + ])) + SimpleDocTemplate(str(path), pagesize=letter).build([table]) + + +def _make_ambiguous_unmerged_first_table_row_pdf(path: Path) -> None: + from reportlab.lib import colors + from reportlab.lib.pagesizes import letter + from reportlab.platypus import SimpleDocTemplate, Table, TableStyle + + table = Table([ + ["Possible caption", "", ""], + ["Unit", "Pressure", "Decision"], + ["Atlas-7", "2.45 bar", "ACCEPT"], + ]) + table.setStyle(TableStyle([ + ("GRID", (0, 0), (-1, -1), 0.75, colors.black), + ("FONTNAME", (0, 1), (-1, 1), "Helvetica-Bold"), + ])) + SimpleDocTemplate(str(path), pagesize=letter).build([table]) + + +def _make_multilevel_header_table_pdf(path: Path, *, verified_spans: bool) -> None: + from reportlab.lib import colors + from reportlab.lib.pagesizes import letter + from reportlab.platypus import SimpleDocTemplate, Table, TableStyle + + table = Table([ + ["Acceptance matrix", "", "", ""], + ["Identity", "", "Outcome", ""], + ["Asset", "", "Limits", ""], + ["Unit", "Revision", "Pressure", "Decision"], + ["Atlas-7", "R2", "2.45 bar", "ACCEPT"], + ]) + commands = [ + ("SPAN", (0, 0), (-1, 0)), + ("GRID", (0, 0), (-1, -1), 0.75, colors.black), + ("FONTNAME", (0, 1), (-1, 3), "Helvetica-Bold"), + ] + if verified_spans: + commands.extend([ + ("SPAN", (0, 1), (1, 1)), + ("SPAN", (2, 1), (3, 1)), + ("SPAN", (0, 2), (1, 2)), + ("SPAN", (2, 2), (3, 2)), + ]) + table.setStyle(TableStyle(commands)) + SimpleDocTemplate(str(path), pagesize=letter).build([table]) + + +def _make_missing_leaf_header_table_pdf(path: Path) -> None: + from reportlab.lib import colors + from reportlab.lib.pagesizes import letter + from reportlab.platypus import SimpleDocTemplate, Table, TableStyle + + table = Table([ + ["Acceptance matrix", "", "", ""], + ["Identity", "", "Outcome", ""], + ["Atlas-7", "R2", "2.45 bar", "ACCEPT"], + ["Boreal-9", "R3", "3.10 bar", "REJECT"], + ]) + table.setStyle(TableStyle([ + ("SPAN", (0, 0), (-1, 0)), + ("SPAN", (0, 1), (1, 1)), + ("SPAN", (2, 1), (3, 1)), + ("GRID", (0, 0), (-1, -1), 0.75, colors.black), + ("FONTNAME", (0, 1), (-1, 1), "Helvetica-Bold"), + ])) + SimpleDocTemplate(str(path), pagesize=letter).build([table]) + + +def _make_bullet_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 11) + c.drawString(48, 700, "• Preserve the negative sign.") + c.drawString(48, 680, "• Preserve the limiting condition.") + c.save() + + +def _make_large_bold_bullet_pdf(path: Path) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica-Bold", 18) + c.drawString(48, 700, "• Critical bullet remains a list item.") + c.setFont("Helvetica", 10) + c.drawString(48, 660, "Ordinary body text establishes the body font size.") + c.save() + + +def _make_detached_bullet_pdf(path: Path, *, safely_attachable: bool) -> None: + from reportlab.lib.pagesizes import letter + from reportlab.pdfgen import canvas + + c = canvas.Canvas(str(path), pagesize=letter) + c.setFont("Helvetica", 18) + c.drawString(48, 700, "•") + c.setFont("Helvetica-Bold", 18) + c.drawString( + 100 if safely_attachable else 300, + 700, + "Detached bold bullet text.", + ) + c.setFont("Helvetica", 10) + c.drawString(48, 660, "Ordinary body text.") + c.save() + + +def test_two_column_page_is_column_major_not_interleaved(tmp_path: Path): + source = tmp_path / "columns.pdf" + _make_two_column_pdf(source) + result = extract_pdf_to_markdown(source) + + text = result.markdown + positions = [text.index(token) for token in ( + "LEFT-START", "LEFT-MIDDLE", "LEFT-END", + "RIGHT-START", "RIGHT-MIDDLE", "RIGHT-END", + )] + assert positions == sorted(positions) + assert result.diagnostics.two_column_pages == (1,) + + +def test_baseline_aligned_independent_columns_are_still_column_major(tmp_path: Path): + source = tmp_path / "aligned-columns.pdf" + _make_aligned_two_column_pdf(source) + result = extract_pdf_to_markdown(source) + + tokens = ("LEFT-A", "LEFT-B", "LEFT-C", "RIGHT-A", "RIGHT-B", "RIGHT-C") + assert [result.markdown.index(token) for token in tokens] == sorted( + result.markdown.index(token) for token in tokens + ) + assert result.diagnostics.two_column_pages == (1,) + + +def test_short_labels_with_wide_descriptions_remain_row_major_without_colons( + tmp_path: Path, +): + source = tmp_path / "key-values.pdf" + _make_wide_key_value_pdf(source) + result = extract_pdf_to_markdown(source) + + tokens = ( + "Owner", "Platform reliability", + "Retention", "Operational evidence", + "Escalation", "Manual review", + ) + assert [result.markdown.index(token) for token in tokens] == sorted( + result.markdown.index(token) for token in tokens + ) + assert "Owner: Platform reliability group approves the release request" in result.markdown + assert "Retention: Operational evidence remains available for seven years" in result.markdown + assert "Escalation: Manual review begins when either threshold is exceeded" in result.markdown + assert "Owner\n\nPlatform reliability" not in result.markdown + assert result.diagnostics.two_column_pages == () + + +def test_single_explicit_key_value_row_is_rendered_atomically(tmp_path: Path): + source = tmp_path / "single-key-value.pdf" + _make_single_explicit_key_value_pdf(source) + result = extract_pdf_to_markdown(source) + assert "Owner: Platform reliability group" in result.markdown + assert "Owner:\n\nPlatform" not in result.markdown + assert result.diagnostics.two_column_pages == () + + +@pytest.mark.parametrize("explicit_colons", [False, True]) +def test_wrapped_key_values_preserve_each_label_value_relation( + tmp_path: Path, explicit_colons: bool, +): + source = tmp_path / f"wrapped-kv-{explicit_colons}.pdf" + _make_wrapped_key_value_pdf(source, explicit_colons=explicit_colons) + result = extract_pdf_to_markdown(source) + + expected = ( + "Owner: Platform reliability group approves the release request after review", + "Retention: Operational evidence remains available for seven years", + "Escalation: Manual review begins when either threshold is exceeded", + ) + assert all(value in result.markdown for value in expected) + assert [result.markdown.index(value) for value in expected] == sorted( + result.markdown.index(value) for value in expected + ) + assert result.diagnostics.two_column_pages == () + + +def test_ambiguous_far_key_value_wrap_fails_instead_of_becoming_columns( + tmp_path: Path, +): + source = tmp_path / "ambiguous-wrapped-kv.pdf" + _make_wrapped_key_value_pdf( + source, explicit_colons=False, ambiguous_wrap=True + ) + with pytest.raises(PDFLayoutExtractionError, match="reading-order|key/value"): + extract_pdf_to_markdown(source) + + +def test_narrow_aligned_sentence_columns_remain_column_major(tmp_path: Path): + source = tmp_path / "narrow-sentence-columns.pdf" + _make_narrow_sentence_columns_pdf(source) + result = extract_pdf_to_markdown(source) + tokens = ( + "Alpha note", "Beta note", "Gamma note", + "Right verification", "Another independent", "Final independent", + ) + assert [result.markdown.index(token) for token in tokens] == sorted( + result.markdown.index(token) for token in tokens + ) + assert result.diagnostics.two_column_pages == (1,) + + +def test_ambiguous_aligned_narrow_layout_fails_instead_of_guessing_order( + tmp_path: Path, +): + source = tmp_path / "ambiguous-aligned.pdf" + _make_ambiguous_aligned_narrow_layout_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="two-column/key-value layout"): + extract_pdf_to_markdown(source) + + +def test_long_aligned_labels_without_high_confidence_signal_fail_closed( + tmp_path: Path, +): + source = tmp_path / "long-ambiguous-labels.pdf" + _make_long_ambiguous_labels_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="two-column/key-value layout"): + extract_pdf_to_markdown(source) + + +def test_bold_first_row_two_column_prose_is_not_promoted_to_implicit_table( + tmp_path: Path, +): + source = tmp_path / "bold-first-row-prose.pdf" + _make_two_column_bold_first_row_prose_pdf(source) + result = extract_pdf_to_markdown(source) + assert result.diagnostics.table_count == 0 + assert result.diagnostics.two_column_pages == (1,) + assert result.markdown.index("Left close") < result.markdown.index("Right lead") + + +def test_three_column_independent_prose_fails_instead_of_becoming_table( + tmp_path: Path, +): + source = tmp_path / "three-column-prose.pdf" + _make_three_column_independent_prose_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="3-column independent prose"): + extract_pdf_to_markdown(source) + + +def test_table_is_preserved_as_one_markdown_table(tmp_path: Path): + source = tmp_path / "table.pdf" + _make_table_pdf(source) + result = extract_pdf_to_markdown(source) + + assert result.diagnostics.table_count == 1 + assert "| Unit | Pressure | Decision |" in result.markdown + assert "| Atlas-7 | 2.45 bar | ACCEPT |" in result.markdown + assert "| Boreal-9 | 3.10 bar | REJECT |" in result.markdown + + +def test_individually_drawn_cell_rectangles_are_table_furniture(tmp_path: Path): + source = tmp_path / "cell-rectangles.pdf" + _make_cell_rectangle_table_pdf(source) + result = extract_pdf_to_markdown(source) + + assert result.diagnostics.table_count == 1 + assert "| Unit | Pressure | Decision |" in result.markdown + assert "| Atlas-7 | 2.45 bar | ACCEPT |" in result.markdown + assert "| Boreal-9 | 3.10 bar | REJECT |" in result.markdown + + +@pytest.mark.parametrize( + "kind", ["step", "bars", "near-panel", "semantic-fill"] +) +def test_cell_local_vector_semantics_are_never_exempted_as_table_grid( + tmp_path: Path, kind: str, +): + source = tmp_path / f"table-{kind}.pdf" + _make_cell_table_with_inner_vector(source, kind) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(source) + + +def test_merged_caption_row_is_not_used_as_the_markdown_table_header(tmp_path: Path): + source = tmp_path / "caption-table.pdf" + _make_caption_table_pdf(source) + result = extract_pdf_to_markdown(source) + + assert "Acceptance register\n\n| Unit | Pressure | Decision |" in result.markdown + assert "| Acceptance register | | |\n| ---" not in result.markdown + assert "| Atlas-7 | 2.45 bar | ACCEPT |" in result.markdown + + +def test_ambiguous_unmerged_caption_shape_fails_instead_of_guessing_header( + tmp_path: Path, +): + source = tmp_path / "ambiguous-caption-table.pdf" + _make_ambiguous_unmerged_first_table_row_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="ambiguous one-cell first row"): + extract_pdf_to_markdown(source) + + +def test_verified_multilevel_merged_headers_flatten_into_leaf_paths(tmp_path: Path): + source = tmp_path / "multilevel-table.pdf" + _make_multilevel_header_table_pdf(source, verified_spans=True) + result = extract_pdf_to_markdown(source) + assert "Acceptance matrix\n\n| Identity / Asset / Unit" in result.markdown + assert "Identity / Asset / Revision" in result.markdown + assert "Outcome / Limits / Pressure" in result.markdown + assert "Outcome / Limits / Decision" in result.markdown + assert "| Atlas-7 | R2 | 2.45 bar | ACCEPT |" in result.markdown + + +def test_unverified_sparse_multilevel_header_fails_closed(tmp_path: Path): + source = tmp_path / "ambiguous-multilevel-table.pdf" + _make_multilevel_header_table_pdf(source, verified_spans=False) + with pytest.raises(PDFLayoutExtractionError, match="sparse group header"): + extract_pdf_to_markdown(source) + + +def test_multilevel_table_without_visual_leaf_header_fails_closed(tmp_path: Path): + source = tmp_path / "missing-leaf-header.pdf" + _make_missing_leaf_header_table_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="leaf header lacks"): + extract_pdf_to_markdown(source) + + +def test_borderless_aligned_register_is_preserved_as_table(tmp_path: Path): + source = tmp_path / "borderless-table.pdf" + _make_borderless_table_pdf(source) + result = extract_pdf_to_markdown(source) + + assert result.diagnostics.table_count == 1 + assert "| Unit | Pressure | Decision |" in result.markdown + assert "| Cedar-4 | 1.85 bar | HOLD |" in result.markdown + assert "| Delta-8 | 2.20 bar | RELEASE |" in result.markdown + + +def test_running_header_is_deduplicated_and_page_numbers_removed(tmp_path: Path): + source = tmp_path / "running.pdf" + _make_running_furniture_pdf(source) + result = extract_pdf_to_markdown(source) + + assert result.markdown.count("REPEATED TECHNICAL HEADER") == 1 + assert "Page 1 of 3" not in result.markdown + assert "Page 2 of 3" not in result.markdown + assert "Page 3 of 3" not in result.markdown + assert result.markdown.index("BODY-1") < result.markdown.index("BODY-2") < result.markdown.index("BODY-3") + + +def test_numbered_content_near_page_top_is_not_mistaken_for_running_header(tmp_path: Path): + source = tmp_path / "numbered-content.pdf" + _make_numbered_top_content_pdf(source) + result = extract_pdf_to_markdown(source) + + assert "Experiment 1 threshold" in result.markdown + assert "Experiment 2 threshold" in result.markdown + + +def test_single_page_year_at_footer_is_not_deleted_as_a_page_number(tmp_path: Path): + source = tmp_path / "year-footer.pdf" + _make_single_page_year_footer_pdf(source) + result = extract_pdf_to_markdown(source) + assert "2026" in result.markdown + + +def test_bare_monotonic_multi_page_counters_are_still_removed(tmp_path: Path): + source = tmp_path / "bare-pages.pdf" + _make_bare_page_series_pdf(source) + result = extract_pdf_to_markdown(source) + assert "SERIES-BODY-1" in result.markdown + assert "SERIES-BODY-2" in result.markdown + assert "\n\n1\n" not in f"\n{result.markdown}" + assert "\n\n2\n" not in f"\n{result.markdown}" + + +def test_non_page_numeric_footers_are_preserved_when_not_physical_page_series( + tmp_path: Path, +): + source = tmp_path / "numeric-footers.pdf" + _make_non_page_numeric_footers_pdf(source) + result = extract_pdf_to_markdown(source) + assert "95" in result.markdown + assert "97" in result.markdown + + +def test_image_only_page_fails_explicitly_instead_of_silent_fallback(tmp_path: Path): + source = tmp_path / "scan.pdf" + _make_image_only_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="OCR is required"): + extract_pdf_to_markdown(source) + + +def test_mixed_page_with_content_sized_image_fails_instead_of_dropping_figure( + tmp_path: Path, +): + source = tmp_path / "mixed-figure.pdf" + _make_mixed_content_image_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="visual/OCR figure"): + extract_pdf_to_markdown(source) + + +def test_content_sized_vector_figure_fails_instead_of_silent_loss(tmp_path: Path): + source = tmp_path / "vector-figure.pdf" + _make_vector_figure_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(source) + + +def test_material_page_lines_alone_are_not_silently_dropped(tmp_path: Path): + source = tmp_path / "line-vector-plot.pdf" + _make_line_only_vector_plot_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(source) + + +def test_single_material_diagonal_line_fails_but_short_diagonal_decoration_passes( + tmp_path: Path, +): + material = tmp_path / "single-trend.pdf" + decoration = tmp_path / "short-diagonal.pdf" + _make_single_diagonal_trend_pdf(material, material=True) + _make_single_diagonal_trend_pdf(decoration, material=False) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(material) + result = extract_pdf_to_markdown(decoration) + assert "A diagonal vector mark follows." in result.markdown + + +@pytest.mark.parametrize("kind", ["panels", "timeline", "diagonals"]) +def test_separated_material_vector_aggregates_fail_closed(tmp_path: Path, kind: str): + source = tmp_path / f"{kind}.pdf" + _make_separated_vector_aggregate_pdf(source, kind) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(source) + + +def test_page_separator_and_small_vector_decoration_are_not_false_positives( + tmp_path: Path, +): + source = tmp_path / "separator.pdf" + _make_separator_and_small_decoration_pdf(source) + result = extract_pdf_to_markdown(source) + assert "SECTION-A retained" in result.markdown + assert "SECTION-B retained" in result.markdown + + +def test_distributed_small_vector_scatter_cloud_fails_closed(tmp_path: Path): + source = tmp_path / "scatter-cloud.pdf" + _make_scatter_cloud_pdf(source) + with pytest.raises(PDFLayoutExtractionError, match="vector figure"): + extract_pdf_to_markdown(source) + + +@pytest.mark.parametrize("columns", [1, 2]) +def test_repeated_vector_circles_adjacent_to_text_restore_markdown_lists( + tmp_path: Path, columns: int, +): + source = tmp_path / f"vector-bullets-{columns}.pdf" + _make_vector_bullet_list_pdf(source, columns=columns) + result = extract_pdf_to_markdown(source) + expected_count = 12 + assert result.markdown.count("- ") == expected_count + if columns == 1: + tokens = [f"ITEM-{index}" for index in range(1, 13)] + else: + tokens = [ + *(f"LEFT-{index}" for index in range(1, 7)), + *(f"RIGHT-{index}" for index in range(1, 7)), + ] + assert result.diagnostics.two_column_pages == (1,) + assert [result.markdown.index(token) for token in tokens] == sorted( + result.markdown.index(token) for token in tokens + ) + + +@pytest.mark.parametrize( + ("count", "marker"), [(1, "circle"), (2, "circle"), (1, "square"), (5, "square")] +) +def test_small_high_confidence_vector_bullets_are_not_silently_lost( + tmp_path: Path, count: int, marker: str, +): + source = tmp_path / f"vector-{marker}-{count}.pdf" + _make_vector_bullet_list_pdf( + source, columns=1, count=count, marker=marker + ) + result = extract_pdf_to_markdown(source) + assert result.markdown.count("- ") == count + assert all(f"ITEM-{index}" in result.markdown for index in range(1, count + 1)) + + +def test_modestly_larger_bold_sentence_is_not_promoted_to_heading(tmp_path: Path): + source = tmp_path / "heading-discriminator.pdf" + _make_heading_discriminator_pdf(source) + result = extract_pdf_to_markdown(source) + + assert "# Operating limits" in result.markdown + assert "# What does this mean?" in result.markdown + assert "# System halted." not in result.markdown + assert "System halted." in result.markdown + assert "# Strong visual title." in result.markdown + + +def test_modest_chinese_question_heading_and_fact_sentence_are_distinguished(): + body = pdf_layout._Line(1, 48, 500, 70, 82, "普通正文用于确立正文字号。", 10, 0.0, 0) + question = pdf_layout._Line(1, 48, 220, 110, 124, "这意味着什么？", 12, 1.0, 1) + question_body = pdf_layout._Line(1, 48, 500, 142, 154, "问句标题之后紧跟正文。", 10, 0.0, 2) + fact = pdf_layout._Line(1, 48, 220, 190, 204, "系统已停止。", 12, 1.0, 3) + fact_body = pdf_layout._Line(1, 48, 500, 222, 234, "事实陈述后面也有普通正文。", 10, 0.0, 4) + page = pdf_layout._Page( + number=1, + width=612, + height=792, + lines=[body, question, question_body, fact, fact_body], + ) + pdf_layout._assign_heading_levels([page]) + assert question.heading_level == 1 + assert fact.heading_level is None + + +def test_visual_line_bullets_become_markdown_list_items(tmp_path: Path): + source = tmp_path / "bullets.pdf" + _make_bullet_pdf(source) + result = extract_pdf_to_markdown(source) + assert "- Preserve the negative sign." in result.markdown + assert "- Preserve the limiting condition." in result.markdown + + +def test_large_bold_cid127_bullet_cannot_be_promoted_to_heading(tmp_path: Path): + source = tmp_path / "large-bold-bullet.pdf" + _make_large_bold_bullet_pdf(source) + result = extract_pdf_to_markdown(source) + assert "- Critical bullet remains a list item." in result.markdown + assert "# (cid:127)" not in result.markdown + assert "(cid:" not in result.markdown + + +def test_separately_drawn_cid127_bullet_and_bold_text_merge_by_visual_row( + tmp_path: Path, +): + source = tmp_path / "detached-bullet.pdf" + _make_detached_bullet_pdf(source, safely_attachable=True) + result = extract_pdf_to_markdown(source) + assert "- Detached bold bullet text." in result.markdown + assert "# Detached bold bullet text." not in result.markdown + assert "(cid:" not in result.markdown + + +def test_far_detached_bullet_fails_instead_of_cross_column_attachment(tmp_path: Path): + source = tmp_path / "far-detached-bullet.pdf" + _make_detached_bullet_pdf(source, safely_attachable=False) + with pytest.raises(PDFLayoutExtractionError, match="detached bullet cannot"): + extract_pdf_to_markdown(source) + + +class _SyntheticCIDPage: + width = 612 + height = 792 + images: list[dict] = [] + rects: list[dict] = [] + curves: list[dict] = [] + lines: list[dict] = [] + + def __init__(self, text: str): + self.text = text + + def dedupe_chars(self, **_kwargs): + return self + + def find_tables(self): + return [] + + def extract_words(self, **_kwargs): + return [{ + "text": self.text, + "x0": 48, + "x1": 180, + "top": 80, + "bottom": 92, + "fontname": "Helvetica", + "size": 11, + }] + + +class _SyntheticCIDDocument: + def __init__(self, text: str): + self.pages = [_SyntheticCIDPage(text)] + + def close(self): + return None + + +def test_single_unresolved_cid_outside_known_bullet_fails_closed( + tmp_path: Path, monkeypatch, +): + import pdfplumber + + source = tmp_path / "unresolved-cid.pdf" + source.write_bytes(b"synthetic") + monkeypatch.setattr( + pdfplumber, "open", lambda _path: _SyntheticCIDDocument("Value (cid:42) lost") + ) + with pytest.raises(PDFLayoutExtractionError, match="1 unresolved CID"): + extract_pdf_to_markdown(source) + + +def test_line_leading_cid127_is_the_only_accepted_cid_recovery( + tmp_path: Path, monkeypatch, +): + import pdfplumber + + source = tmp_path / "known-bullet.pdf" + source.write_bytes(b"synthetic") + monkeypatch.setattr( + pdfplumber, + "open", + lambda _path: _SyntheticCIDDocument("(cid:127) Retained bullet evidence"), + ) + result = extract_pdf_to_markdown(source) + assert "- Retained bullet evidence" in result.markdown + + +def test_unicode_replacement_character_is_an_unknown_glyph_and_fails_closed( + tmp_path: Path, monkeypatch, +): + import pdfplumber + + source = tmp_path / "replacement-glyph.pdf" + source.write_bytes(b"synthetic") + monkeypatch.setattr( + pdfplumber, + "open", + lambda _path: _SyntheticCIDDocument("Value \ufffd cannot be trusted"), + ) + with pytest.raises(PDFLayoutExtractionError, match="replacement glyph"): + extract_pdf_to_markdown(source) + + +def test_corrupt_pdf_fails_explicitly(tmp_path: Path): + source = tmp_path / "corrupt.pdf" + source.write_bytes(b"not-a-pdf") + with pytest.raises(PDFLayoutExtractionError, match="cannot open PDF"): + extract_pdf_to_markdown(source) + + +def test_convert_file_uses_pdf_layout_path_and_keeps_anchor_contract(tmp_path: Path): + source = tmp_path / "columns.pdf" + _make_two_column_pdf(source) + + class ExplodingGenericConverter: + def convert(self, _source: str): + raise AssertionError("generic converter must not handle PDFs") + + old_base = convert._BASE_ROOT + convert._BASE_ROOT = tmp_path + try: + ok, message = convert.convert_file(ExplodingGenericConverter(), source) + finally: + convert._BASE_ROOT = old_base + assert ok is True + assert "PDF layout-aware" in message + markdown = source.with_suffix(".md").read_text(encoding="utf-8") + assert "^p-" in markdown or "^h-" in markdown + assert markdown.index("LEFT-END") < markdown.index("RIGHT-START") + outline_path = source.with_suffix(".outline.json") + assert outline_path.is_file() + outline = json.loads(outline_path.read_text(encoding="utf-8")) + assert convert.validate_conversion_receipt(outline, source) == [] + assert convert.should_convert(source, force=False) is False diff --git a/scripts/tests/test_postprocess.py b/scripts/tests/test_postprocess.py index 4284fcc..e2696f0 100644 --- a/scripts/tests/test_postprocess.py +++ b/scripts/tests/test_postprocess.py @@ -131,6 +131,8 @@ def test_returns_tuple(self): assert isinstance(outline, dict) assert "sections" in outline assert "doc_chars" in outline + assert "doc_sha256" in outline + assert "outline_schema_version" in outline assert "doc_paragraphs" in outline def test_outline_section_count(self): @@ -153,14 +155,13 @@ def test_outline_preserves_existing_summaries(self): assert outline2["sections"][0]["agent_summary"] == "manually filled" def test_outline_summary_dropped_when_content_changed(self): - """anchor 变了 → 旧摘要不应被错误回填到新 anchor 上。""" + """标题/heading anchor 未变但章节正文变了，旧摘要也必须失效。""" text1, outline1 = process("# Title\n\nold.\n", "t.md") outline1["sections"][0]["agent_summary"] = "old summary" - # heading 没变 → heading anchor 不变 → 摘要保留 + # heading 没变 → heading anchor 不变，但 section_sha256 必须变 text2, _ = process("# Title\n\nNEW.\n", "t.md") outline2 = build_outline_data(text2, "t.md", previous_outline=outline1) - # heading 段摘要应保留（标题没改） - assert outline2["sections"][0]["agent_summary"] == "old summary" + assert outline2["sections"][0]["agent_summary"] is None def test_summary_restored_by_title_on_seq_drift(self): """pipe-2：在前面插入新标题导致 heading seq 漂移、anchor 变，但 title 不变 diff --git a/scripts/tests/test_precommit_hook.py b/scripts/tests/test_precommit_hook.py new file mode 100644 index 0000000..124a8a2 --- /dev/null +++ b/scripts/tests/test_precommit_hook.py @@ -0,0 +1,313 @@ +"""pre-commit 引用门禁的 staged-tree / fail-closed 端到端回归。""" + +from __future__ import annotations + +import os +import shutil +import subprocess +from pathlib import Path + +import pytest + + +PROJECT_ROOT = Path(__file__).resolve().parent.parent.parent +HOOK_SOURCE = PROJECT_ROOT / "scripts" / "hooks" / "pre-commit" + + +CHECKER = r'''#!/usr/bin/env python3 +import json +import os +from pathlib import Path +import sys + +ws = sys.argv[sys.argv.index("--workspace") + 1] +cmd = sys.argv[sys.argv.index("--workspace") + 2] +root = Path(os.environ["KB_ROOT"]) +page = root / "workspaces" / ws / "wiki" / "page.md" +text = page.read_text(encoding="utf-8") if page.exists() else "" + +if "TOOL_ERROR" in text: + print("simulated checker failure", file=sys.stderr) + raise SystemExit(23) +if "BAD_JSON" in text: + print("{not-json") + raise SystemExit(0) +if "BAD_SCHEMA" in text: + print(json.dumps({"unexpected": []})) + raise SystemExit(0) + +if cmd == "list-cite-mismatches": + marker_to_issue = { + "MISMATCH": "mismatch", + "BARE_EXEMPT": "exempt-missing-basis", + "IMPRECISE": "imprecise-anchor", + "UNVERIFIABLE": "unverifiable", + "CANONICAL_ANCHOR": "canonical-anchor-mismatch", + "CANONICAL_TARGET": "canonical-target-mismatch", + } + issues = [] + for marker, issue in marker_to_issue.items(): + if marker in text: + issues.append({ + "issue": issue, "path": "wiki/page.md", "line": 9, + "numbers": ["99"], + }) + print(json.dumps(issues)) +elif cmd == "list-broken-refs": + out = ([{"from_path": "wiki/page.md", "line": 12, "reason": "anchor 不存在"}] + if "BROKEN_REF" in text else []) + print(json.dumps(out)) +elif cmd == "list-bare-claims": + out = ([{"path": "wiki/page.md", "line": 13, "matched": ["99%"]}] + if "BARE_CLAIM" in text else []) + print(json.dumps(out)) +elif cmd == "list-coarse-citations": + out = ([{"path": "wiki/page.md", "line": 14, "matched": ["99%"]}] + if "COARSE_CITE" in text else []) + print(json.dumps(out)) +elif cmd == "list-unmapped-claims": + out = ([{ + "path": "wiki/page.md", "page": "wiki/page.md", "line": 14, + "issue": "unmapped-factual-claim", "claim_text": "an unsupported fact", + "kind": "paragraph", "detail": "missing block evidence", + }] if "UNMAPPED_CLAIM" in text else []) + print(json.dumps(out)) +elif cmd == "list-source-issues": + out = ([{"path": "wiki/page.md", "issue_type": "declared-but-uncited"}] + if "SOURCE_ISSUE" in text else []) + print(json.dumps(out)) +elif cmd == "extract-claims": + pair = None + if "TARGET_NOT_OK" in text: + pair = {"target_status": "file-missing", "audited": False, "last_verdict": None} + elif "UNAUDITED_PAIR" in text: + pair = {"target_status": "ok", "audited": False, "last_verdict": None} + elif "BAD_VERDICT" in text: + pair = {"target_status": "ok", "audited": True, "last_verdict": "PARTIAL"} + elif "SUPPORTED_PAIR" in text or "CACHE_REQUIRED" in text: + audit = root / "workspaces" / ws / ".cache" / "citation_audit.jsonl" + cache_ok = "CACHE_REQUIRED" not in text or ( + audit.exists() and audit.read_text(encoding="utf-8").strip() == "audit-ok" + ) + pair = { + "target_status": "ok", "audited": cache_ok, + "last_verdict": "SUPPORTED" if cache_ok else None, + } + pairs = [] + if pair is not None: + pairs.append({ + "pair_id": "pair-1", "page": "wiki/page.md", "line": 15, + "target": "raw/source.md", "anchor": "p-1-abcdef", **pair, + }) + print(json.dumps({ + "pairs": pairs, + "summary": {"pairs_total": len(pairs), "returned": len(pairs)}, + })) +elif cmd == "check-provenance": + retrieval = root / "workspaces" / ws / ".cache" / "retrieval_log.jsonl" + cache_ok = "CACHE_REQUIRED" not in text or ( + retrieval.exists() and retrieval.read_text(encoding="utf-8").strip() == "retrieval-ok" + ) + missing = "NO_PROVENANCE" in text or not cache_ok + findings = ([{ + "page": "wiki/page.md", "line": 16, "target": "raw/source.md", + "anchor": "p-1-abcdef", "issue": "no-retrieval-evidence", + }] if missing else []) + print(json.dumps({"checked": 1 if ("SUPPORTED_PAIR" in text or "CACHE_REQUIRED" in text) else 0, + "findings": findings})) + raise SystemExit(1 if findings else 0) +else: + print("unknown command", file=sys.stderr) + raise SystemExit(24) +''' + + +def _run(repo: Path, *args: str, check: bool = True, env=None): + return subprocess.run( + list(args), cwd=repo, check=check, capture_output=True, text=True, env=env + ) + + +@pytest.fixture +def repo(tmp_path: Path) -> Path: + root = tmp_path / "repo" + root.mkdir() + _run(root, "git", "init", "-q") + _run(root, "git", "config", "user.email", "t@example.com") + _run(root, "git", "config", "user.name", "t") + + checker = root / "scripts" / "k.py" + checker.parent.mkdir(parents=True) + checker.write_text(CHECKER, encoding="utf-8") + readme = root / "README.md" + readme.write_text("fixture\n", encoding="utf-8") + _run(root, "git", "add", "scripts/k.py", "README.md") + _run(root, "git", "commit", "-q", "-m", "fixture") + + hook = root / ".git" / "hooks" / "pre-commit" + shutil.copy2(HOOK_SOURCE, hook) + hook.chmod(0o755) + return root + + +def _stage_page( + repo: Path, + staged: str, + worktree: str | None = None, + *, + prefix="", + workspace="alpha", +) -> Path: + page = repo / prefix / "workspaces" / workspace / "wiki" / "page.md" + page.parent.mkdir(parents=True, exist_ok=True) + page.write_text(staged, encoding="utf-8") + _run(repo, "git", "add", "--", str(page.relative_to(repo))) + if worktree is not None: + page.write_text(worktree, encoding="utf-8") + return page + + +def _hook(repo: Path, *, env=None): + merged = os.environ.copy() + if env: + merged.update(env) + return _run(repo, str(repo / ".git" / "hooks" / "pre-commit"), check=False, env=merged) + + +def test_rejects_bad_staged_blob_even_when_worktree_is_fixed(repo: Path): + _stage_page(repo, "MISMATCH staged\n", "clean worktree\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert "wiki/page.md:9" in result.stdout + result.stderr + + +def test_does_not_validate_unstaged_bad_rewrite(repo: Path): + _stage_page(repo, "clean staged\n", "MISMATCH only in worktree\n") + result = _hook(repo) + assert result.returncode == 0, result.stdout + result.stderr + + +@pytest.mark.parametrize( + "marker", + [ + "MISMATCH", "BARE_EXEMPT", "IMPRECISE", "UNVERIFIABLE", + "CANONICAL_ANCHOR", "CANONICAL_TARGET", + ], +) +def test_blocks_current_deterministic_gate_issues(repo: Path, marker: str): + _stage_page(repo, marker + "\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + + +@pytest.mark.parametrize("marker", ["TOOL_ERROR", "BAD_JSON", "BAD_SCHEMA"]) +def test_checker_or_json_failure_is_fail_closed(repo: Path, marker: str): + _stage_page(repo, marker + "\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert "校验" in result.stdout + result.stderr + + +def test_missing_staged_k_py_is_fail_closed(repo: Path): + _stage_page(repo, "clean\n") + _run(repo, "git", "rm", "-q", "scripts/k.py") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert "k.py" in result.stdout + result.stderr + + +def test_missing_python_is_fail_closed(repo: Path): + _stage_page(repo, "clean\n") + result = _hook(repo, env={"PYTHON": str(repo / "missing-python")}) + assert result.returncode != 0, result.stdout + result.stderr + assert "PYTHON=" in result.stdout + result.stderr + + +def test_temp_tree_creation_failure_is_fail_closed(repo: Path): + _stage_page(repo, "clean\n") + not_a_directory = repo / "not-a-directory" + not_a_directory.write_text("x", encoding="utf-8") + result = _hook(repo, env={"TMPDIR": str(not_a_directory)}) + assert result.returncode != 0, result.stdout + result.stderr + assert "临时树" in result.stdout + result.stderr + + +def test_no_wiki_change_does_not_require_python_or_k_py(repo: Path): + (repo / "notes.txt").write_text("ordinary change\n", encoding="utf-8") + _run(repo, "git", "add", "notes.txt") + _run(repo, "git", "rm", "-q", "scripts/k.py") + # k.py 的 staged 删除也不是 wiki 变更；应保持快速放行。 + result = _hook(repo, env={"PATH": "/usr/bin:/bin"}) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_kb_root_nested_data_layout_uses_staged_tree(repo: Path): + data_root = repo / "kb-data" + _stage_page(repo, "MISMATCH nested staged\n", "clean nested worktree\n", prefix="kb-data") + result = _hook(repo, env={"KB_ROOT": str(data_root)}) + assert result.returncode != 0, result.stdout + result.stderr + + +def test_multiple_workspaces_are_checked_independently(repo: Path): + _stage_page(repo, "clean alpha\n", workspace="alpha") + _stage_page(repo, "MISMATCH beta\n", workspace="beta") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert "[beta]" in result.stdout + result.stderr + + +@pytest.mark.parametrize( + ("marker", "expected"), + [ + ("BROKEN_REF", "broken-ref"), + ("BARE_CLAIM", "bare-claim"), + ("COARSE_CITE", "coarse-citation"), + ("UNMAPPED_CLAIM", "unmapped-factual-claim"), + ("SOURCE_ISSUE", "declared-but-uncited"), + ], +) +def test_blocks_structural_findings_on_staged_files(repo: Path, marker: str, expected: str): + _stage_page(repo, marker + "\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert expected in result.stdout + result.stderr + + +@pytest.mark.parametrize( + ("marker", "expected"), + [ + ("TARGET_NOT_OK", "file-missing"), + ("UNAUDITED_PAIR", "unaudited"), + ("BAD_VERDICT", "PARTIAL"), + ], +) +def test_extract_claims_requires_ok_current_supported_pair( + repo: Path, marker: str, expected: str +): + _stage_page(repo, marker + "\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert expected in result.stdout + result.stderr + + +def test_check_provenance_findings_are_a_gate(repo: Path): + _stage_page(repo, "SUPPORTED_PAIR NO_PROVENANCE\n") + result = _hook(repo) + assert result.returncode != 0, result.stdout + result.stderr + assert "no-retrieval-evidence" in result.stdout + result.stderr + + +def test_actual_cache_is_snapshotted_for_audit_and_provenance(repo: Path): + _stage_page(repo, "CACHE_REQUIRED\n") + cache = repo / "workspaces" / "alpha" / ".cache" + cache.mkdir(parents=True) + (cache / "citation_audit.jsonl").write_text("audit-ok\n", encoding="utf-8") + (cache / "retrieval_log.jsonl").write_text("retrieval-ok\n", encoding="utf-8") + result = _hook(repo) + assert result.returncode == 0, result.stdout + result.stderr + + +def test_supported_pair_with_provenance_passes(repo: Path): + _stage_page(repo, "SUPPORTED_PAIR\n") + result = _hook(repo) + assert result.returncode == 0, result.stdout + result.stderr diff --git a/scripts/tests/test_release_guards.py b/scripts/tests/test_release_guards.py index eb1a522..8e9deea 100644 --- a/scripts/tests/test_release_guards.py +++ b/scripts/tests/test_release_guards.py @@ -16,7 +16,7 @@ 3. TestSettingsDenyGuard — .claude/settings.json 的 deny 必须含 workspaces 变体的 Write/Edit 拒绝模式（CLAUDE.md 核心原则 5 的 agent 层落地）。 -4. TestOutlineFreshness — k.py load_or_build_outline 必须按 doc_chars 判 +4. TestOutlineFreshness — k.py load_or_build_outline 必须按 schema + 全文 SHA-256 判 outline.json 新鲜度：过期（md 被编辑后未再生）→ 丢弃缓存现场重建； 新鲜 → 用盘上缓存；重建时 anchor 仍稳定的 agent_summary 保留。 历史缺陷：盘上存在即直接采用，过期偏移切当前文本返回错位内容。 @@ -66,6 +66,8 @@ class TestPreCommitHookGuard: "workspaces/smb-ecommerce/raw/x.pdf", "workspaces/smb-ecommerce/raw/x.md", "workspaces/any-name/my_thoughts/note.md", + "kb-data/workspaces/main/raw/private.pdf", # KB_ROOT 位于 repo 子目录 + "kb-data/workspaces/main/my_thoughts/private.md", ] ALLOWED = [ "wiki/concepts/a.md", @@ -127,6 +129,23 @@ def _run_hook_e2e(self, tmp_path, rel_path: str) -> int: subprocess.run(["git", "init", "-q"], cwd=repo, check=True) subprocess.run(["git", "config", "user.email", "t@example.com"], cwd=repo, check=True) subprocess.run(["git", "config", "user.name", "t"], cwd=repo, check=True) + # wiki 变更现在是 fail-closed：hook 会从 **staged tree** 执行 k.py， + # 因此临时仓库需要一个同样进入 index 的最小成功 checker。 + checker = repo / "scripts" / "k.py" + checker.parent.mkdir(parents=True, exist_ok=True) + checker.write_text( + "import json, sys\n" + "cmd = sys.argv[sys.argv.index('--workspace') + 2]\n" + "if cmd == 'extract-claims':\n" + " out = {'pairs': [], 'summary': {'pairs_total': 0, 'returned': 0}}\n" + "elif cmd == 'check-provenance':\n" + " out = {'checked': 0, 'findings': []}\n" + "else:\n" + " out = []\n" + "print(json.dumps(out))\n", + encoding="utf-8", + ) + subprocess.run(["git", "add", "--", "scripts/k.py"], cwd=repo, check=True) target = repo / rel_path target.parent.mkdir(parents=True, exist_ok=True) target.write_text("x", encoding="utf-8") @@ -267,19 +286,32 @@ def _write_pair(self, tmp_path, md_text: str, outline: dict | None): ) return md - def test_fresh_outline_used_as_is(self, tmp_path): + def test_fresh_outline_used_as_is(self, tmp_path, monkeypatch): import k + from postprocess import process + + monkeypatch.setattr(k, "PROJECT_ROOT", tmp_path) + + text, sentinel = process("# 主标题\n\n正文。\n", "page.md") + sentinel["sections"][0]["agent_summary"] = "盘上缓存哨兵" + md = self._write_pair(tmp_path, text, sentinel) + out = k.load_or_build_outline(md) + assert out["sections"][0]["agent_summary"] == "盘上缓存哨兵", "新鲜 outline 应直接采用盘上缓存" + + def test_same_length_rewrite_invalidates_outline(self, tmp_path): + """doc_chars 相同不是新鲜度证明；同长度改写必须按 SHA-256 重建。""" + import k + from postprocess import build_outline_data + + old = build_outline_data(self.MD, "page.md") + old["sections"][0]["title"] = "不应沿用的旧标题" + edited = self.MD.replace("第一节内容", "第一节改写") + assert len(edited) == len(self.MD) + md = self._write_pair(tmp_path, edited, old) - sentinel = { - "doc_path": "page.md", - "doc_chars": len(self.MD), # 与 md 一致 → 新鲜 - "sections": [{"title": "盘上缓存哨兵", "anchor": "h-1-1-aaaaaa", - "level": 1, "line": 1, "char_start": 0, - "char_end": 5, "children": []}], - } - md = self._write_pair(tmp_path, self.MD, sentinel) out = k.load_or_build_outline(md) - assert out["sections"][0]["title"] == "盘上缓存哨兵", "新鲜 outline 应直接采用盘上缓存" + assert out["doc_sha256"] != old["doc_sha256"] + assert out["sections"][0]["title"] != "不应沿用的旧标题" def test_stale_outline_rebuilt(self, tmp_path): import k @@ -347,7 +379,13 @@ def test_corrupt_outline_rebuilds_not_raises(self, tmp_path): def test_annotate_goes_through_freshness_guard(self, tmp_path): """annotate_section 写路径经新鲜度守卫：盘上 outline 过期时按当前 md 的新锚点工作。""" import k - from postprocess import build_outline_data + from postprocess import process + + current_md, current_outline = process( + "# 主标题\n\n## 第一节\n\n第一节内容。\n\n## 第二节\n\n第二节内容。\n", + "page.md", + ) + first_anchor = current_outline["sections"][0]["children"][0]["anchor"] # 盘上放一份过期 outline（旧锚点、doc_chars 不符） stale = { @@ -357,9 +395,9 @@ def test_annotate_goes_through_freshness_guard(self, tmp_path): "level": 2, "line": 1, "char_start": 0, "char_end": 5, "children": []}], } - md = self._write_pair(tmp_path, self.MD, stale) + md = self._write_pair(tmp_path, current_md, stale) # 用当前 md 的真实锚点 annotate——若不经守卫会 LookupError（旧 outline 无此锚点） - res = k.annotate_section(md, "^h-2-1-bbbbbb", "第一节摘要") + res = k.annotate_section(md, f"^{first_anchor}", "第一节摘要") assert res["agent_summary"] == "第一节摘要" # 写回后盘上 outline 变新鲜，且摘要可被读到 out = k.load_or_build_outline(md) @@ -367,7 +405,7 @@ def test_annotate_goes_through_freshness_guard(self, tmp_path): def collect(secs): for s in secs: - if s.get("anchor") == "h-2-1-bbbbbb": + if s.get("anchor") == first_anchor: found.append(s.get("agent_summary")) collect(s.get("children", [])) @@ -380,3 +418,97 @@ def test_missing_outline_builds_from_md(self, tmp_path): md = self._write_pair(tmp_path, self.MD, None) out = k.load_or_build_outline(md) assert any("主标题" in s["title"] for s in out["sections"]) + + +class TestReadSectionSafety: + def test_duplicate_title_is_ambiguous(self, tmp_path): + import k + from postprocess import process + + text, outline = process( + "# Book\n\n## Repeat\n\nfirst\n\n## Repeat\n\nsecond\n", + "book.md", + ) + md = tmp_path / "book.md" + md.write_text(text, encoding="utf-8") + md.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + + with pytest.raises(LookupError, match="标题不唯一"): + k.read_section(md, "Repeat") + + def test_overlong_leaf_requires_natural_block_drilldown(self, tmp_path): + import k + from postprocess import process + + text, outline = process("## Leaf\n\n" + ("x" * 30050) + "\n", "long.md") + md = tmp_path / "long.md" + md.write_text(text, encoding="utf-8") + md.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + + with pytest.raises(LookupError, match="章节过长"): + k.read_section(md, outline["sections"][0]["anchor"]) + with pytest.raises(ValueError, match="非负整数"): + k.read_section(md, outline["sections"][0]["anchor"], max_chars=-1) + block = next( + item for item in k.parse_blocks_with_anchors(md) + if item.kind == "paragraph" + ) + with pytest.raises(LookupError, match="自然块过长"): + k.read_block(md, block.anchor) + full_block = k.read_block(md, block.anchor, max_chars=0) + assert len(full_block["content"]) > 30000 + result = k.read_section(md, outline["sections"][0]["anchor"], max_chars=0) + assert len(result["content"]) > 30000 + + +class TestCiteCheckDemoGuard: + """demo 库引用核对守门：随仓分发的示例 wiki 不得携带确定性可判的错引。 + + - 闸门项（mismatch / exempt-missing-basis）必须为 0——这是 list-cite-mismatches + 对外承诺的「高置信错引」，demo 若带着它发布等于示范坏实践； + - release 环境 raw/ 不分发 → raw 引用全部落 unverifiable 信息桶（不是错误）， + 本测试同时钉住「raw 缺失环境零误报」的降级承诺。 + """ + + GATE_ISSUES = {"mismatch", "exempt-missing-basis"} + + def _workspaces(self): + ws_root = PROJECT_ROOT / "workspaces" + if not ws_root.is_dir(): + pytest.skip("无 workspaces/，跳过 demo 守护") + out = [] + for d in sorted(ws_root.iterdir()): + if not d.is_dir() or d.name.startswith("."): + continue + # demo 语义 = raw 不随仓分发（只有 .gitkeep）。raw 在场的是**工作库** + # （dev 仓私人数据），其存量引用质量由 kb-lint 周检 + pre-commit（staged + # 范围）治理，不由本守护全量卡死。 + raw = d / "raw" + has_raw_files = raw.is_dir() and any( + f.is_file() and f.name != ".gitkeep" and not f.name.startswith(".") + for f in raw.rglob("*")) + if not has_raw_files: + out.append(d) + if not out: + pytest.skip("所有 workspace 均为 raw 在场的工作库，demo 守护不适用") + return out + + def test_demo_workspaces_have_no_gate_findings(self): + import sys as _sys + for ws in self._workspaces(): + r = subprocess.run( + [_sys.executable, str(PROJECT_ROOT / "scripts" / "k.py"), + "--workspace", ws.name, "list-cite-mismatches", "--json"], + capture_output=True, text=True, cwd=str(PROJECT_ROOT), + ) + assert r.returncode == 0, f"{ws.name}: list-cite-mismatches 运行失败\n{r.stderr}" + items = json.loads(r.stdout) + gate = [i for i in items if i["issue"] in self.GATE_ISSUES] + assert gate == [], ( + f"demo 库 {ws.name} 存在 {len(gate)} 条引用核对闸门项（高置信错引），" + f"发布前必须整备：{[(i['path'], i['line'], i.get('numbers')) for i in gate]}" + ) diff --git a/scripts/tests/test_retrieval_index.py b/scripts/tests/test_retrieval_index.py new file mode 100644 index 0000000..cbbadd6 --- /dev/null +++ b/scripts/tests/test_retrieval_index.py @@ -0,0 +1,1801 @@ +"""Long-document mechanical evidence-index tests. + +The gold counts in this file are deliberately independent of the SQLite +inventory. A parser change that silently drops a natural unit therefore +breaks the tests instead of merely making ``expected == indexed`` inside the +same implementation. +""" + +from __future__ import annotations + +import hashlib +import json +import sqlite3 +import unicodedata +from pathlib import Path + +import pytest + +import conversion_receipt +from conversion_receipt import build_conversion_receipt +from postprocess import add_anchors +from postprocess import build_outline_data +from retrieval_index import ( + RetrievalIndexError, + coverage_report, + read_evidence_unit, + rebuild_index, + search_evidence, +) + + +GOLD_DOCUMENT = """# Evidence Atlas + +Front matter detail is searchable. + +## 极光流控机制 + +- 第一个列表细节：蓝鲸协议 +- 第二个列表细节 + +| 指标 | 值 | +|---|---| +| 回退窗口 | 17 毫秒 | +| 重试上限 | 4 次 | + +> 这是一条不应丢失的引用内容。 + +```python +timeout_ms = 17 +``` + +![极光流控时序图](aurora-flow.png) +""" + + +def _workspace(tmp_path: Path) -> Path: + workspace = tmp_path / "workspace" + (workspace / "raw" / "papers").mkdir(parents=True) + return workspace + + +def _write_anchored(workspace: Path, body: str, name: str = "manual.md") -> tuple[Path, str]: + path = workspace / "raw" / "papers" / name + anchored, _ = add_anchors(body) + path.write_text(anchored, encoding="utf-8") + return path, anchored + + +def _write_converted( + workspace: Path, + body: str = GOLD_DOCUMENT, + source_name: str = "manual.pdf", +) -> tuple[Path, Path, Path]: + """Create a source→Markdown→outline triple using the real receipt contract.""" + + source = workspace / "raw" / "papers" / source_name + source.write_bytes(b"original-container-v1") + markdown = source.with_suffix(".md") + anchored, _ = add_anchors(body) + markdown.write_text(anchored, encoding="utf-8") + relative_markdown = markdown.relative_to(workspace).as_posix() + outline = build_outline_data(anchored, relative_markdown) + outline["conversion_receipt"] = build_conversion_receipt( + source, workspace, require_raw=True + ) + outline_path = source.with_suffix(".outline.json") + outline_path.write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + return source, markdown, outline_path + + +def _rows(db_path: Path, sql: str) -> list[sqlite3.Row]: + connection = sqlite3.connect(db_path) + connection.row_factory = sqlite3.Row + try: + return list(connection.execute(sql)) + finally: + connection.close() + + +def _gold_multiset_fingerprint(records: list[tuple[str, str, str]]) -> str: + encoded = sorted( + json.dumps(list(record), ensure_ascii=False, separators=(",", ":")).encode("utf-8") + for record in records + ) + digest = hashlib.sha256() + for record in encoded: + digest.update(len(record).to_bytes(8, "big")) + digest.update(record) + return digest.hexdigest() + + +def _gold_content_hash(text: str) -> str: + normalized = " ".join(unicodedata.normalize("NFC", text).split()) + return hashlib.sha256(normalized.encode("utf-8")).hexdigest() + + +def test_gold_natural_unit_and_content_section_coverage_is_100_percent(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + db_path = workspace / ".cache" / "retrieval_index.db" + + rebuilt = rebuild_index(workspace) + report = coverage_report(db_path) + + # Independent gold: 1 paragraph + 2 list items + 2 table data rows + + # 1 blockquote + 1 code block + 1 figure = 8 natural units. + assert rebuilt["natural_units"]["expected"] == 8 + assert rebuilt["natural_units"]["indexed"] == 8 + assert rebuilt["natural_units"]["materialized"] == 8 + assert rebuilt["natural_units"]["unicode61_indexed"] == 8 + assert rebuilt["natural_units"]["trigram_indexed"] == 8 + assert rebuilt["natural_units"]["missing"] == 0 + assert rebuilt["natural_units"]["missing_items"] == [] + assert rebuilt["natural_units"]["coverage"] == 1.0 + assert rebuilt["natural_units"]["coverage_pct"] == 100.0 + expected_kind_counts = { + "blockquote": 1, + "code": 1, + "figure": 1, + "list_item": 2, + "paragraph": 1, + "table_row": 2, + } + assert rebuilt["natural_units"]["expected_kind_counts"] == expected_kind_counts + assert rebuilt["natural_units"]["indexed_kind_counts"] == expected_kind_counts + gold_units = [ + ("paragraph", "Front matter detail is searchable."), + ("list_item", "- 第一个列表细节：蓝鲸协议"), + ("list_item", "- 第二个列表细节"), + ("table_row", "| 回退窗口 | 17 毫秒 |"), + ("table_row", "| 重试上限 | 4 次 |"), + ("blockquote", "> 这是一条不应丢失的引用内容。"), + ("code", "```python\ntimeout_ms = 17\n```"), + ("figure", "![极光流控时序图](aurora-flow.png)"), + ] + gold_fingerprint = _gold_multiset_fingerprint([ + ("raw/papers/manual.md", kind, _gold_content_hash(text)) + for kind, text in gold_units + ]) + fingerprint = rebuilt["natural_units"]["exact_multiset_fingerprint"] + assert fingerprint["algorithm"] == "sha256-lenprefixed-json-v1" + assert fingerprint["fields"] == ["path", "kind", "content_hash"] + assert fingerprint["expected"] == gold_fingerprint + assert fingerprint["indexed"] == gold_fingerprint + assert fingerprint["match"] is True + # A single H1 is the document title; the H2 is the one content section. + assert report["structural_sections"]["expected"] == 1 + assert report["structural_sections"]["registered"] == 1 + assert report["structural_sections"]["coverage_pct"] == 100.0 + assert report["unexpected_empty_sections"] == [] + assert report["content_sections"]["expected"] == 1 + assert report["content_sections"]["registered"] == 1 + assert report["content_sections"]["missing"] == 0 + assert report["content_sections"]["coverage_pct"] == 100.0 + assert report["ok"] is True + assert rebuilt["errors"] == [] + + kinds = [row["kind"] for row in _rows(db_path, "SELECT kind FROM units ORDER BY ordinal")] + assert kinds == [ + "paragraph", + "list_item", + "list_item", + "table_row", + "table_row", + "blockquote", + "code", + "figure", + ] + + +def test_coverage_report_exposes_exact_missing_inventory_item(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT unit_id, path, anchor, kind FROM units WHERE kind='table_row' ORDER BY ordinal LIMIT 1" + ).fetchone() + connection.execute("DELETE FROM units WHERE unit_id=?", (victim[0],)) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["natural_units"]["expected"] == 8 + assert report["natural_units"]["indexed"] == 7 + assert report["natural_units"]["missing"] == 1 + assert report["natural_units"]["coverage_pct"] == 87.5 + assert report["natural_units"]["expected_kind_counts"]["table_row"] == 2 + assert report["natural_units"]["indexed_kind_counts"]["table_row"] == 1 + assert report["natural_units"]["exact_multiset_fingerprint"]["match"] is False + missing = report["natural_units"]["missing_items"][0] + assert (missing["unit_id"], missing["path"], missing["anchor"], missing["kind"]) == victim + assert missing["line_start"] == 12 + assert missing["missing_components"] == ["unit"] + + +def test_coverage_includes_both_fts_materializations(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT unit_id FROM units WHERE kind='blockquote'" + ).fetchone()[0] + connection.execute("DELETE FROM unit_fts_trigram WHERE unit_id=?", (victim,)) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["natural_units"]["materialized"] == 8 + assert report["natural_units"]["unicode61_indexed"] == 8 + assert report["natural_units"]["trigram_indexed"] == 7 + assert report["natural_units"]["indexed"] == 7 + assert report["natural_units"]["missing_items"][0]["missing_components"] == ["trigram"] + + +def test_fts_shadow_content_cannot_mask_broken_inverted_indexes(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT unit_id FROM units WHERE kind='paragraph' LIMIT 1" + ).fetchone()[0] + for table in ("unit_fts_unicode", "unit_fts_trigram"): + row = connection.execute( + f"""SELECT rowid, unit_id, text, heading_path, section_summary, path + FROM {table} WHERE unit_id=?""", + (victim,), + ).fetchone() + connection.execute(f"DELETE FROM {table} WHERE rowid=?", (row[0],)) + # Restore only the external-content shadow row. SELECT now looks + # healthy, but the FTS segment/index has no posting for this unit. + connection.execute( + f"INSERT INTO {table}_content(id,c0,c1,c2,c3,c4) VALUES(?,?,?,?,?,?)", + row, + ) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "incomplete" + assert all( + not row["ok"] for row in report["fts_integrity"].values() + ) + with pytest.raises(RetrievalIndexError) as blocked: + search_evidence(db_path, "searchable detail", limit=5) + assert blocked.value.code == "index-integrity-error" + + +def test_fts_tokenizer_schema_substitution_fails_closed(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + rows = connection.execute( + """SELECT unit_id, normalized_text, heading_path_json, + section_summary, path FROM units ORDER BY ordinal""" + ).fetchall() + connection.execute("DROP TABLE unit_fts_trigram") + connection.execute( + """CREATE VIRTUAL TABLE unit_fts_trigram USING fts5( + unit_id UNINDEXED, text, heading_path, section_summary, path, + tokenize='unicode61 remove_diacritics 2' + )""" + ) + for unit_id, text, heading_json, summary, path in rows: + connection.execute( + "INSERT INTO unit_fts_trigram VALUES(?,?,?,?,?)", + (unit_id, text, " / ".join(json.loads(heading_json)), summary or "", path), + ) + connection.commit() + connection.close() + + with pytest.raises(RetrievalIndexError) as coverage_error: + coverage_report(db_path) + assert coverage_error.value.code == "invalid-index-schema" + with pytest.raises(RetrievalIndexError) as search_error: + search_evidence(db_path, "蓝鲸协议", limit=5) + assert search_error.value.code == "invalid-index-schema" + + +def test_poisoned_unit_text_fails_coverage_search_and_exact_read(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT unit_id FROM units WHERE kind='table_row' ORDER BY ordinal LIMIT 1" + ).fetchone()[0] + poisoned = "| forged metric | 999 years |" + connection.execute( + "UPDATE units SET text=?, normalized_text=? WHERE unit_id=?", + (poisoned, poisoned, victim), + ) + connection.execute( + "UPDATE unit_fts_unicode SET text=? WHERE unit_id=?", (poisoned, victim) + ) + connection.execute( + "UPDATE unit_fts_trigram SET text=? WHERE unit_id=?", (poisoned, victim) + ) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "incomplete" + poisoned_item = next( + item for item in report["natural_units"]["missing_items"] + if item["unit_id"] == victim + ) + assert poisoned_item["missing_components"] == ["unit-metadata-or-content"] + + with pytest.raises(RetrievalIndexError) as search_error: + search_evidence(db_path, "forged metric", limit=5) + assert search_error.value.code == "index-integrity-error" + with pytest.raises(RetrievalIndexError) as read_error: + read_evidence_unit(db_path, victim) + assert read_error.value.code == "index-integrity-error" + + +def test_code_indentation_tamper_fails_exact_text_integrity(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + "# Manual\n\n```python\nif allowed:\n grant()\ndeny()\n```\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + connection = sqlite3.connect(db_path) + victim, original = connection.execute( + "SELECT unit_id, text FROM units WHERE kind='code'" + ).fetchone() + poisoned = original.replace("\ndeny()", "\n deny()") + connection.execute("UPDATE units SET text=? WHERE unit_id=?", (poisoned, victim)) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "incomplete" + assert report["natural_units"]["missing_items"][0]["missing_components"] == [ + "unit-metadata-or-content" + ] + with pytest.raises(RetrievalIndexError) as blocked: + read_evidence_unit(db_path, victim) + assert blocked.value.code == "index-integrity-error" + + +def test_synchronized_unit_deletion_cannot_shrink_manifest_denominator( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + connection = sqlite3.connect(db_path) + victim = connection.execute("SELECT unit_id FROM units LIMIT 1").fetchone()[0] + for table in ("unit_fts_unicode", "unit_fts_trigram", "units", "expected_units"): + connection.execute(f"DELETE FROM {table} WHERE unit_id=?", (victim,)) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "incomplete" + assert report["manifest"]["ok"] is False + issue = next( + item for item in report["manifest"]["issues"] + if item["field"] == "expected_units" + ) + assert issue["declared"] == report["natural_units"]["expected"] + assert issue["observed"] == issue["declared"] - 1 + assert report["natural_units"]["coverage_pct"] < 100.0 + with pytest.raises(RetrievalIndexError) as blocked: + search_evidence(db_path, "回退窗口", limit=5) + assert blocked.value.code == "index-integrity-error" + + +def test_synchronized_section_deletion_cannot_shrink_manifest_denominators( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, "# Manual\n\n## Detail\n\nEvidence.\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT section_id FROM sections WHERE is_content=1 LIMIT 1" + ).fetchone()[0] + connection.execute("DELETE FROM expected_sections WHERE section_id=?", (victim,)) + connection.execute( + "DELETE FROM expected_structural_sections WHERE section_id=?", (victim,) + ) + connection.execute("DELETE FROM sections WHERE section_id=?", (victim,)) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["manifest"]["ok"] is False + assert report["structural_sections"]["coverage_pct"] < 100.0 + assert report["content_sections"]["coverage_pct"] < 100.0 + + +def test_deleted_unit_fails_search_and_exact_read_before_returning_results( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + connection = sqlite3.connect(db_path) + victim = connection.execute( + "SELECT unit_id FROM units WHERE kind='table_row' ORDER BY ordinal LIMIT 1" + ).fetchone()[0] + connection.execute("DELETE FROM units WHERE unit_id=?", (victim,)) + connection.commit() + connection.close() + + with pytest.raises(RetrievalIndexError) as search_error: + search_evidence(db_path, "回退窗口", limit=5) + assert search_error.value.code == "index-integrity-error" + with pytest.raises(RetrievalIndexError) as read_error: + read_evidence_unit(db_path, victim) + assert read_error.value.code == "index-integrity-error" + + +def test_valid_outline_summary_change_marks_index_stale(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, + "# Manual\n\n## Routing\n\nAuthoritative evidence.\n", + ) + outline = build_outline_data(anchored, "raw/papers/manual.md") + routing = outline["sections"][0]["children"][0] + routing["agent_summary"] = "zzqxvplmnbcdfg" + path.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + assert search_evidence(db_path, "zzqxvplmnbcdfg", limit=5)["returned"] >= 1 + + routing["agent_summary"] = "hjkrstuwyeioaa" + path.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + assert report["corpus_freshness"]["outline_summary_changed"] == [ + "raw/papers/manual.md" + ] + with pytest.raises(RetrievalIndexError) as stale: + search_evidence(db_path, "hjkrstuwyeioaa", limit=5) + assert stale.value.code == "index-stale" + + rebuild_index(workspace) + assert search_evidence(db_path, "hjkrstuwyeioaa", limit=5)["returned"] >= 1 + + +def test_synchronized_routing_metadata_poison_fails_expected_inventory_check( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, "# Manual\n\n## Routing\n\nEvidence.\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + poison = "poisonroutingzxq92741" + + connection = sqlite3.connect(db_path) + victim = connection.execute("SELECT unit_id FROM units LIMIT 1").fetchone()[0] + connection.execute( + "UPDATE units SET heading_path_json=? WHERE unit_id=?", + (json.dumps([poison]), victim), + ) + connection.execute( + "UPDATE unit_fts_unicode SET heading_path=? WHERE unit_id=?", + (poison, victim), + ) + connection.execute( + "UPDATE unit_fts_trigram SET heading_path=? WHERE unit_id=?", + (poison, victim), + ) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["coverage_status"] == "incomplete" + assert report["natural_units"]["missing_items"][0]["missing_components"] == [ + "unit-metadata-or-content" + ] + with pytest.raises(RetrievalIndexError) as blocked: + search_evidence(db_path, poison, limit=5) + assert blocked.value.code == "index-integrity-error" + + +def test_frozen_inventory_fingerprint_rejects_fully_synchronized_text_poison( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + poisoned = "forged synchronized evidence 92741" + normalized = " ".join(unicodedata.normalize("NFC", poisoned).split()) + content_hash = hashlib.sha256(normalized.encode("utf-8")).hexdigest() + exact_hash = hashlib.sha256( + unicodedata.normalize("NFC", poisoned).encode("utf-8") + ).hexdigest() + + connection = sqlite3.connect(db_path) + victim = connection.execute("SELECT unit_id FROM units LIMIT 1").fetchone()[0] + connection.execute( + """UPDATE expected_units + SET normalized_text=?, content_hash=?, exact_text_hash=? + WHERE unit_id=?""", + (normalized, content_hash, exact_hash, victim), + ) + connection.execute( + """UPDATE units + SET text=?, normalized_text=?, content_hash=?, exact_text_hash=? + WHERE unit_id=?""", + (poisoned, normalized, content_hash, exact_hash, victim), + ) + for table in ("unit_fts_unicode", "unit_fts_trigram"): + connection.execute( + f"UPDATE {table} SET text=? WHERE unit_id=?", (normalized, victim) + ) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "incomplete" + assert any( + issue["field"] == "inventory_hash" + for issue in report["manifest"]["issues"] + ) + with pytest.raises(RetrievalIndexError) as blocked: + search_evidence(db_path, "forged synchronized evidence", limit=5) + assert blocked.value.code == "index-integrity-error" + + +def test_frozen_inventory_fingerprint_rejects_synchronized_route_poison( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, "# Manual\n\n## Routing\n\nEvidence.\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + poison = "synchronized-route-poison-92741" + encoded = json.dumps([poison]) + + connection = sqlite3.connect(db_path) + victim = connection.execute("SELECT unit_id FROM units LIMIT 1").fetchone()[0] + connection.execute( + "UPDATE expected_units SET heading_path_json=? WHERE unit_id=?", + (encoded, victim), + ) + connection.execute( + "UPDATE units SET heading_path_json=? WHERE unit_id=?", (encoded, victim) + ) + for table in ("unit_fts_unicode", "unit_fts_trigram"): + connection.execute( + f"UPDATE {table} SET heading_path=? WHERE unit_id=?", (poison, victim) + ) + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["ok"] is False + assert any( + issue["field"] == "inventory_hash" + for issue in report["manifest"]["issues"] + ) + with pytest.raises(RetrievalIndexError) as blocked: + search_evidence(db_path, poison, limit=5) + assert blocked.value.code == "index-integrity-error" + + +def test_chinese_trigram_and_heading_only_query_recall_evidence(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + # The phrase only occurs in the H2 title, not in either list item. Units + # remain discoverable because heading_path is an indexed retrieval field. + title_result = search_evidence(db_path, "极光流控机制", limit=20) + assert title_result["returned"] >= 1 + assert all("极光流控机制" in hit["heading_path"] for hit in title_result["hits"]) + assert any( + name.startswith("trigram_routing:") + for hit in title_result["hits"] + for name in hit["channel_ranks"] + ) + + # Chinese has no whitespace segmentation here; trigram/exact channels + # must still find a substring buried inside a longer natural unit. + detail_result = search_evidence(db_path, "蓝鲸协议", limit=20) + assert detail_result["hits"][0]["kind"] == "list_item" + assert "蓝鲸协议" in detail_result["hits"][0]["text"] + assert detail_result["hits"][0]["canonical_ref"].startswith( + "raw/papers/manual.md#^p-" + ) + + +def test_list_items_and_table_rows_keep_exact_source_positions(tmp_path: Path): + workspace = _workspace(tmp_path) + _path, anchored = _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + rows = _rows( + db_path, + """SELECT * FROM units WHERE kind IN ('list_item', 'table_row') + ORDER BY ordinal""", + ) + assert len(rows) == 4 + assert [row["text"] for row in rows] == [ + "- 第一个列表细节：蓝鲸协议", + "- 第二个列表细节", + "| 回退窗口 | 17 毫秒 |", + "| 重试上限 | 4 次 |", + ] + assert all(row["anchor"] for row in rows) + assert rows[0]["anchor"] == rows[1]["anchor"] # canonical parent list block + assert rows[2]["anchor"] == rows[3]["anchor"] # canonical parent table block + for row in rows: + assert anchored[row["char_start"] : row["char_end"]] == row["text"] + assert row["line_start"] <= row["line_end"] + + +def test_read_evidence_unit_preserves_row_identity_while_disclosing_parent_scope( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + row = _rows( + db_path, + "SELECT unit_id FROM units WHERE kind='table_row' ORDER BY subordinal LIMIT 1", + )[0] + + unit = read_evidence_unit(db_path, row["unit_id"]) + assert unit["selection_scope"] == "natural_unit" + assert unit["citation_scope"] == "parent_block" + assert unit["evidence_handle"]["unit_id"] == row["unit_id"] + assert unit["evidence_handle"]["subordinal"] == 1 + assert unit["canonical_ref"].startswith("raw/papers/manual.md#^t-") + + with pytest.raises(RetrievalIndexError) as caught: + read_evidence_unit(db_path, "not-a-unit") + assert caught.value.code == "invalid-unit-id" + + +def test_oversized_natural_unit_requires_explicit_unlimited_read(tmp_path: Path): + workspace = _workspace(tmp_path) + needle = "alpha beta" + body = "alpha " + ("A" * 31_000) + " " + needle + _write_anchored(workspace, f"# Manual\n\n{body}\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + search = search_evidence(db_path, needle, limit=5) + hit = search["hits"][0] + assert hit["text_truncated"] is True + assert hit["text_chars"] == len(body) + assert len(hit["text"]) == 30_000 + assert needle in hit["text"] + assert hit["text_excerpt_start"] > 0 + + with pytest.raises(RetrievalIndexError) as caught: + read_evidence_unit(db_path, hit["unit_id"]) + assert caught.value.code == "evidence-unit-too-large" + assert caught.value.details["canonical_ref"] == hit["canonical_ref"] + + full = read_evidence_unit(db_path, hit["unit_id"], max_chars=0) + assert full["text"] == body + assert full["text_truncated"] is False + + with pytest.raises(RetrievalIndexError) as invalid: + read_evidence_unit(db_path, hit["unit_id"], max_chars=-1) + assert invalid.value.code == "invalid-max-chars" + + +def test_plain_paragraph_cannot_impersonate_a_figure_anchor(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored(workspace, "# Manual\n\nOrdinary evidence.\n") + forged = anchored.replace(" ^p-", " ^f-", 1) + path.write_text(forged, encoding="utf-8") + + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + assert caught.value.code == "rebuild-document-errors" + assert caught.value.details["errors"][0]["code"] == "canonical-anchor-kind-mismatch" + + +@pytest.mark.parametrize( + "header,data_row", + [ + ("name | value", r"A | one \| two"), + ("name | value", "A | `x|y`"), + ("c1 | c2 | c3", "A | B"), + ], +) +def test_common_gfm_rows_remain_exact_table_units( + tmp_path: Path, header: str, data_row: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + f"# Manual\n\nIntro.\n\n## Register\n\n{header}\n--- | ---" + f"{' | ---' if header.count('|') == 2 else ''}\n{data_row}\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + rows = _rows(db_path, "SELECT text FROM units WHERE kind='table_row'") + assert [row["text"] for row in rows] == [data_row] + + +def test_header_only_gfm_table_does_not_abort_document_indexing(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + "# Manual\n\nIntro evidence.\n\n## Empty register\n\nname | value\n--- | ---\n", + ) + report = rebuild_index(workspace) + assert report["ok"] is True + assert report["natural_units"]["expected"] == 1 + assert report["unexpected_empty_sections_count"] == 1 + + +@pytest.mark.parametrize( + "table,data_row", + [ + ("| only |\n|---|\n| value |", "| value |"), + ( + "| c1 | c2 | c3 |\n|---|---|---|\n| onlyone |", + "| onlyone |", + ), + ], +) +def test_single_column_and_short_gfm_rows_keep_table_row_identity( + tmp_path: Path, table: str, data_row: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, f"# Manual\n\nIntro.\n\n## Register\n\n{table}\n" + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + rows = _rows(db_path, "SELECT kind, text FROM units WHERE kind='table_row'") + assert [(row["kind"], row["text"]) for row in rows] == [ + ("table_row", data_row) + ] + + +def test_positions_use_full_file_coordinates_with_frontmatter(tmp_path: Path): + workspace = _workspace(tmp_path) + _path, anchored = _write_anchored( + workspace, + """--- +title: Positioned +--- +# Manual + +Evidence after frontmatter. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + row = _rows(db_path, "SELECT * FROM units")[0] + assert row["line_start"] == 6 + assert anchored[row["char_start"] : row["char_end"]] == "Evidence after frontmatter." + + +def test_unit_identity_survives_position_move_and_retrieval_still_finds_it(tmp_path: Path): + workspace = _workspace(tmp_path) + first = """# Manual + +Opening paragraph. + +## Details + +- stationary rare detail: cobalt-window-771 +- another item + +Closing paragraph. +""" + _write_anchored(workspace, first) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + before = { + row["text"]: row["unit_id"] + for row in _rows(db_path, "SELECT text, unit_id FROM units") + } + + # Move the whole section earlier and add unrelated material. Re-running + # postprocess changes parent anchor sequence/positions, but a natural + # unit's content identity remains stable. + moved = """# Manual + +## Details + +- stationary rare detail: cobalt-window-771 +- another item + +New intervening paragraph. + +Opening paragraph. + +Closing paragraph. +""" + _write_anchored(workspace, moved) + rebuild_index(workspace) + after = { + row["text"]: row["unit_id"] + for row in _rows(db_path, "SELECT text, unit_id FROM units") + } + assert after["- stationary rare detail: cobalt-window-771"] == before[ + "- stationary rare detail: cobalt-window-771" + ] + result = search_evidence(db_path, "cobalt-window-771", limit=20) + assert result["hits"][0]["text"].endswith("cobalt-window-771") + + +def test_explicit_query_expansion_participates_in_rrf(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, GOLD_DOCUMENT) + rebuild_index(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + + result = search_evidence( + db_path, + "fallback latency", + expansions=["回退窗口", "17 毫秒"], + limit=20, + ) + assert result["expansions"] == ["回退窗口", "17 毫秒"] + assert any("回退窗口" in hit["text"] for hit in result["hits"]) + assert any("q1" in channel or "q2" in channel for channel in result["hits"][0]["channel_ranks"]) + + +def test_positive_query_downranks_explicit_wrong_subject_exclusion(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Limits + +The Aster controller has a maximum inlet temperature of 43 C. + +The Boreal relay, not the Aster controller, has a maximum inlet temperature of 43 C; this value must not be applied to the Aster controller. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is the maximum inlet temperature of the Aster controller?", limit=20 + ) + assert result["hits"][0]["text"].startswith("The Aster controller has") + excluded = next(hit for hit in result["hits"] if "Boreal relay" in hit["text"]) + assert excluded["polarity_factor"] == 0.2 + assert excluded["rerank_reasons"] == ["queried-subject-explicitly-excluded"] + + negative_query = search_evidence( + db_path, "Which value must not be applied to the Aster controller?", limit=20 + ) + excluded = next(hit for hit in negative_query["hits"] if "Boreal relay" in hit["text"]) + assert excluded["polarity_factor"] == 1.0 + + +def test_empty_corpus_is_not_misreported_as_100_percent(tmp_path: Path): + workspace = _workspace(tmp_path) + result = rebuild_index(workspace) + assert result["empty_corpus"] is True + assert result["ok"] is False + assert result["coverage_status"] == "empty-corpus" + assert result["natural_units"]["expected"] == 0 + assert result["natural_units"]["coverage"] is None + assert result["natural_units"]["coverage_pct"] is None + assert result["content_sections"]["coverage"] is None + + +def test_document_error_is_structured_and_previous_index_is_preserved(tmp_path: Path): + workspace = _workspace(tmp_path) + path, _ = _write_anchored(workspace, "# Valid\n\nEvidence survives.\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + previous = coverage_report(db_path) + + # A hand-written unconverted raw page has no canonical anchors. It is a + # hard document error, not a silently skipped source. + path.write_text("# Missing anchors\n\nThis must not disappear silently.\n", encoding="utf-8") + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + payload = caught.value.to_dict() + assert payload["error"]["code"] == "rebuild-document-errors" + assert payload["error"]["details"]["errors"][0]["code"] == "missing-canonical-anchor" + assert coverage_report(db_path)["natural_units"] == previous["natural_units"] + + +def test_stale_content_hash_anchor_is_rejected_instead_of_indexed(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, "# Manual\n\nOriginal evidence value is 17.\n" + ) + # Simulate editing the derived Markdown without rerunning conversion. The + # tail anchor still parses, but its content hash no longer represents the + # evidence and therefore cannot be advertised as canonical. + path.write_text(anchored.replace("value is 17", "value is 18"), encoding="utf-8") + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + errors = caught.value.details["errors"] + assert errors[0]["code"] == "stale-canonical-anchor" + + +def test_wrong_anchor_sequence_is_rejected_even_when_content_hash_matches( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, "# Manual\n\nOriginal evidence value is 17.\n" + ) + path.write_text(anchored.replace("^p-1-", "^p-999-"), encoding="utf-8") + + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + errors = caught.value.details["errors"] + assert errors[0]["code"] == "canonical-anchor-coordinate-mismatch" + assert errors[0]["details"]["expected_anchor"].startswith("p-1-") + + +def test_content_heading_without_any_evidence_is_registered_but_not_claimed_as_content(tmp_path: Path): + workspace = _workspace(tmp_path) + body = """# Manual + +## Empty appendix + +## Real section + +Actual evidence. +""" + _write_anchored(workspace, body) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + sections = _rows(db_path, "SELECT title, is_content FROM sections ORDER BY ordinal") + assert [(row["title"], row["is_content"]) for row in sections] == [ + ("Manual", 0), + ("Empty appendix", 0), + ("Real section", 1), + ] + report = coverage_report(db_path) + assert report["structural_sections"]["expected"] == 2 + assert report["structural_sections"]["registered"] == 2 + assert report["structural_sections"]["coverage_pct"] == 100.0 + assert report["unexpected_empty_sections_count"] == 1 + assert [row["title"] for row in report["unexpected_empty_sections"]] == [ + "Empty appendix" + ] + assert report["content_sections"]["expected"] == 1 + assert report["content_sections"]["registered"] == 1 + + +def test_multiple_h1_headings_are_real_content_sections(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Chapter One + +First chapter fact. + +# Chapter Two + +Second chapter fact. +""", + ) + report = rebuild_index(workspace) + assert report["structural_sections"]["expected"] == 2 + assert report["structural_sections"]["registered"] == 2 + assert report["content_sections"]["expected"] == 2 + assert report["content_sections"]["registered"] == 2 + + +def test_structural_coverage_catches_missing_empty_heading_registration(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + "# Manual\n\n## Empty but registered\n\n## Populated\n\nEvidence.\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + connection = sqlite3.connect(db_path) + connection.execute("DELETE FROM sections WHERE title='Empty but registered'") + connection.commit() + connection.close() + + report = coverage_report(db_path) + assert report["content_sections"]["coverage"] == 1.0 + assert report["structural_sections"]["expected"] == 2 + assert report["structural_sections"]["registered"] == 1 + assert report["structural_sections"]["missing"] == 1 + assert report["structural_sections"]["missing_items"][0]["title"] == ( + "Empty but registered" + ) + assert report["coverage_status"] == "incomplete" + assert report["ok"] is False + + +def test_stale_outline_summary_requires_schema_and_full_document_hash(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, + "# Manual\n\n## Routing\n\nAuthoritative evidence.\n", + ) + outline = build_outline_data(anchored, "raw/papers/manual.md") + outline["sections"][0]["children"][0]["agent_summary"] = "STALE-SECRET-SUMMARY" + # Same doc_chars is insufficient: a mismatched content hash must make the + # entire optional summary layer unavailable. + outline["doc_sha256"] = "0" * 64 + path.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + + result = rebuild_index(workspace) + assert any(warning["code"] == "outline-stale" for warning in result["warnings"]) + rows = _rows( + workspace / ".cache" / "retrieval_index.db", + "SELECT section_summary FROM units", + ) + assert {row["section_summary"] for row in rows} == {None} + + +def test_outline_summary_requires_current_section_hash(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, + "# Manual\n\n## Routing\n\nAuthoritative evidence.\n", + ) + outline = build_outline_data(anchored, "raw/papers/manual.md") + routing = outline["sections"][0]["children"][0] + routing["agent_summary"] = "UNVERIFIED-SECTION-SUMMARY" + routing["section_sha256"] = "0" * 64 + path.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + + result = rebuild_index(workspace) + warning = next( + row for row in result["warnings"] + if row["code"] == "outline-summary-unverifiable" + ) + assert "section-hash-mismatch" in warning["reasons"] + rows = _rows( + workspace / ".cache" / "retrieval_index.db", + "SELECT section_summary FROM units", + ) + assert {row["section_summary"] for row in rows} == {None} + + +def test_outline_summary_is_used_only_when_all_hashes_match(tmp_path: Path): + workspace = _workspace(tmp_path) + path, anchored = _write_anchored( + workspace, + "# Manual\n\n## Routing\n\nAuthoritative evidence.\n", + ) + outline = build_outline_data(anchored, "raw/papers/manual.md") + outline["sections"][0]["children"][0]["agent_summary"] = "CURRENT-SUMMARY" + path.with_suffix(".outline.json").write_text( + json.dumps(outline, ensure_ascii=False), encoding="utf-8" + ) + + result = rebuild_index(workspace) + assert not any( + row["code"] == "outline-summary-unverifiable" + for row in result["warnings"] + ) + rows = _rows( + workspace / ".cache" / "retrieval_index.db", + "SELECT section_summary FROM units", + ) + assert {row["section_summary"] for row in rows} == {"CURRENT-SUMMARY"} + + +def test_heading_only_match_is_collapsed_before_top20_fusion(tmp_path: Path): + workspace = _workspace(tmp_path) + paragraphs = "\n\n".join( + f"Unrelated evidence paragraph {index}." for index in range(40) + ) + _write_anchored( + workspace, + f"# Manual\n\n## Flood-prone routing title\n\n{paragraphs}\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "Flood-prone routing title", limit=20) + # Every paragraph inherits the heading, but routing metadata must not fill + # all twenty result slots with clones from one section. + assert 1 <= result["returned"] <= 2 + assert any( + channel.get("routing_collapsed", 0) >= 38 + for channel in result["channels"].values() + if channel["status"] == "ok" + ) + + +def test_question_stopwords_do_not_turn_heading_routes_into_body_hits(tmp_path: Path): + workspace = _workspace(tmp_path) + paragraphs = "\n\n".join("This is unrelated evidence." for _ in range(40)) + _write_anchored( + workspace, + f"# Manual\n\n## Flood routing\n\n{paragraphs}\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "What is Flood routing?", limit=20) + assert 1 <= result["returned"] <= 2 + assert result["channels"]["unicode61_text:q0"]["returned"] == 0 + assert result["channels"]["unicode61_routing:q0"]["routing_collapsed"] >= 38 + + +def test_routing_window_reaches_later_sections_without_offset_rescans( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + first = "\n\n".join(f"Archive paragraph {index}." for index in range(160)) + _write_anchored( + workspace, + "# Manual\n\n" + f"## Flood routing alpha\n\n{first}\n\n" + "## Flood routing omega\n\nFinal independent evidence.\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "Flood routing", limit=20) + owning_sections = {hit["owning_section_anchor"] for hit in result["hits"]} + assert len(owning_sections) == 2 + assert any( + channel.get("scanned", 0) > 100 + for channel in result["channels"].values() + if channel["status"] == "ok" + ) + routing_channels = [ + channel for name, channel in result["channels"].items() + if "_routing:" in name and channel["status"] == "ok" + ] + assert routing_channels + assert all(channel["routing_strategy"] == "window-partition" + for channel in routing_channels) + assert all(channel["query_pages"] == 1 for channel in routing_channels) + + +def test_noncontiguous_multiword_body_hits_are_not_collapsed_as_heading_routes( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + facts = "\n\n".join( + f"alpha fact {index} has a distinct beta condition." for index in range(5) + ) + _write_anchored(workspace, f"# Manual\n\n## Generic chapter\n\n{facts}\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "alpha beta", limit=20) + matching = [hit for hit in result["hits"] if "alpha fact" in hit["text"]] + assert len(matching) == 5 + + +def test_correct_negative_fact_is_not_treated_as_wrong_subject_exclusion( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Compatibility manual + +## Model X + +Model X does not support firmware 4.2. + +## Model Y + +Model Y supports firmware 4.2. + +## Policy A + +Policy A does not govern Project X. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "Does Model X support firmware 4.2?", limit=5) + correct = next(hit for hit in result["hits"] if "does not support" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert correct["rerank_reasons"] == [] + + status_query = search_evidence( + db_path, "What is the support status of firmware 4.2 under Model X?", limit=5 + ) + status_correct = next( + hit for hit in status_query["hits"] if "does not support" in hit["text"] + ) + assert status_correct["polarity_factor"] == 1.0 + + governance_query = search_evidence( + db_path, "What is the governance status of Project X under Policy A?", limit=5 + ) + governance = next( + hit for hit in governance_query["hits"] if "does not govern" in hit["text"] + ) + assert governance["polarity_factor"] == 1.0 + + +def test_unless_condition_does_not_disable_wrong_subject_guard(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Terms + +For the Willow account, the minimum age is 16 years unless local law requires older. + +The Aspen account has a minimum age of 16 years unless local law requires older; the clause does not govern the Willow account. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, + "What is the minimum age for the Willow account unless local law requires older?", + limit=5, + ) + assert result["hits"][0]["text"].startswith("For the Willow account") + excluded = next(hit for hit in result["hits"] if "Aspen account" in hit["text"]) + assert excluded["polarity_factor"] == 0.2 + + +def test_contrast_evidence_is_not_penalized_for_an_unexcluded_entity_query( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + "# Routing\n\nPolicy A, not Policy B, governs Project X.\n", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, "Which policy governs Project X?", limit=5) + assert result["hits"][0]["polarity_factor"] == 1.0 + + +def test_contrast_is_not_downranked_when_excluded_entity_is_the_query_subject( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Routing + +Policy A, not Policy B, governs Project X. + +A superseded draft mistakenly states that Policy B has a role in Project X. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + for query in ( + "Describe Policy B's role in Project X.", + "What is Policy B's role in Project X?", + ): + result = search_evidence(db_path, query, limit=5) + correct = next(hit for hit in result["hits"] if "not Policy B" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert result["hits"][0]["unit_id"] == correct["unit_id"] + + +def test_incidental_year_does_not_turn_qualitative_authority_query_quantitative( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Governance + +Policy A, not Policy B, has authority over Project X in 2026. + +A superseded 2026 draft says Policy B has full authority over Project X. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is Policy B's authority over Project X in 2026?", limit=5 + ) + correct = next(hit for hit in result["hits"] if "not Policy B" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert result["hits"][0]["unit_id"] == correct["unit_id"] + + +@pytest.mark.parametrize( + "query", + ["What is the mass of the Aster unit?", "What is the Aster unit's mass?"], +) +def test_unrelated_exclusion_in_same_natural_unit_does_not_poison_later_fact( + tmp_path: Path, query: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Specifications + +The Boreal bracket, not the Aster unit, uses the legacy mount. The Aster unit has a mass of 17 kg. + +A superseded draft says the Aster unit has a mass of 43 kg. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, query, limit=5) + correct = next(hit for hit in result["hits"] if "17 kg" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert result["hits"][0]["unit_id"] == correct["unit_id"] + + +def test_known_quantitative_query_requires_number_in_exclusion_clause(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Terms + +The Boreal bracket, not the Willow account, uses the legacy mount. The Willow account has a minimum age of 16 years. + +A superseded draft says the Willow account has a minimum age of 43 years. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is the minimum age of the Willow account?", limit=5 + ) + correct = next(hit for hit in result["hits"] if "16 years" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert result["hits"][0]["unit_id"] == correct["unit_id"] + + +def test_sentence_final_number_still_ends_atomic_exclusion_scope(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Specifications + +The Aster unit has a mass of 17. The Boreal bracket, not the Aster unit, uses the legacy mount. + +A superseded draft says the Aster unit has a mass of 43. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is the mass of the Aster unit?", limit=5 + ) + correct = next(hit for hit in result["hits"] if "mass of 17" in hit["text"]) + assert correct["polarity_factor"] == 1.0 + assert result["hits"][0]["unit_id"] == correct["unit_id"] + + +@pytest.mark.parametrize("property_name,correct_value,wrong_value", [ + ("mass", "17 kg", "43 kg"), + ("weight", "17 kg", "43 kg"), + ("speed", "17 km/h", "43 km/h"), + ("price", "$17", "$43"), + ("frequency", "17 Hz", "43 Hz"), +]) +def test_open_quantitative_property_uses_numeric_value_guard( + tmp_path: Path, property_name: str, correct_value: str, wrong_value: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + f"""# Specifications + +The Aster unit has a {property_name} of {correct_value}. + +The Boreal unit, not the Aster unit, has a {property_name} of {wrong_value}. Aster unit {property_name} specifications are repeated here. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, f"What is the {property_name} of the Aster unit?", limit=5 + ) + wrong = next(hit for hit in result["hits"] if "Boreal unit" in hit["text"]) + assert wrong["polarity_factor"] == 0.2 + assert result["hits"][0]["text"].startswith("The Aster unit has") + + +@pytest.mark.parametrize( + "query", + ["What is the Aster unit's mass?", "What is the Aster unit mass?"], +) +def test_open_quantitative_property_supports_possessive_and_telegraphic_queries( + tmp_path: Path, query: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Specifications + +The Aster unit has a mass of 17 kg. + +The Boreal unit, not the Aster unit, has a mass of 43 kg. Aster unit mass specifications are repeated here. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, query, limit=5) + wrong = next(hit for hit in result["hits"] if "Boreal unit" in hit["text"]) + assert wrong["polarity_factor"] == 0.2 + assert result["hits"][0]["text"].startswith("The Aster unit has") + + +@pytest.mark.parametrize( + ("query", "property_name", "correct_value", "wrong_value"), + [ + ("What is the current mass of the Aster unit?", "mass", "17 kg", "43 kg"), + ("What is the voltage of the Aster unit?", "voltage", "-17 V", "-43 V"), + ( + "What is the voltage of the Aster unit?", + "voltage", + "approximately 17 V", + "approximately 43 V", + ), + ], +) +def test_open_property_guard_handles_modifiers_signed_and_approximate_values( + tmp_path: Path, + query: str, + property_name: str, + correct_value: str, + wrong_value: str, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + f"""# Specifications + +The Aster unit has a {property_name} of {correct_value}. + +The Boreal unit, not the Aster unit, has a {property_name} of {wrong_value}. Aster unit {property_name} specifications are repeated here. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence(db_path, query, limit=5) + wrong = next(hit for hit in result["hits"] if "Boreal unit" in hit["text"]) + assert wrong["polarity_factor"] == 0.2 + assert result["hits"][0]["text"].startswith("The Aster unit has") + + +def test_approx_abbreviation_does_not_split_property_value_clause(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Specifications + +The Aster unit has a voltage of 17 V. + +The Boreal unit, not the Aster unit, has a voltage of approx. 43 V. Aster unit voltage specifications are repeated here. +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is the voltage of the Aster unit?", limit=5 + ) + wrong = next(hit for hit in result["hits"] if "Boreal unit" in hit["text"]) + assert wrong["polarity_factor"] == 0.2 + assert result["hits"][0]["text"].startswith("The Aster unit has") + + +def test_table_wrong_subject_row_is_penalized_without_collapsing_parent_anchor( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Register + +| Subject | Metric | Value | Condition | +|---|---|---:|---| +| Quartz-A | precision | 87.5% | validation V4 | +| Quartz-B | precision | 87.5% | validation V4; not applicable to Quartz-A | +""", + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + result = search_evidence( + db_path, "What is the precision of Quartz-A for validation V4?", limit=5 + ) + assert "| Quartz-A |" in result["hits"][0]["text"] + wrong = next(hit for hit in result["hits"] if "| Quartz-B |" in hit["text"]) + assert wrong["polarity_factor"] == 0.2 + assert wrong["canonical_ref"] == result["hits"][0]["canonical_ref"] + assert wrong["unit_id"] != result["hits"][0]["unit_id"] + + +def test_index_path_under_raw_is_rejected(tmp_path: Path): + workspace = _workspace(tmp_path) + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace, workspace / "raw" / "forbidden.db") + assert caught.value.code == "write-protected-path" + + +def test_inventory_covers_parenthesized_lists_and_pipe_less_gfm_tables( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, + """# Manual + +## Alternate Markdown syntax + +1) first ordered requirement +2) second ordered requirement + +Metric | Value +---|--- +latency | 17 ms +retries | 4 +""", + ) + + report = rebuild_index(workspace) + counts = report["natural_units"]["indexed_kind_counts"] + assert counts["list_item"] == 2 + assert counts["table_row"] == 2 + assert report["natural_units"]["exact_multiset_fingerprint"]["match"] is True + + +def test_same_length_raw_rewrite_marks_coverage_and_search_stale(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored( + workspace, "# Manual\n\n## Limits\n\nCurrent value is 17.\n" + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + _write_anchored( + workspace, "# Manual\n\n## Limits\n\nCurrent value is 99.\n" + ) + + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + assert report["corpus_freshness"]["changed"] == ["raw/papers/manual.md"] + with pytest.raises(RetrievalIndexError) as caught: + search_evidence(db_path, "Current value") + assert caught.value.code == "index-stale" + assert caught.value.details["changed"] == ["raw/papers/manual.md"] + + +def test_added_raw_document_marks_coverage_and_search_stale(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_anchored(workspace, "# Manual\n\n## A\n\nEvidence A.\n") + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + _write_anchored( + workspace, "# Added\n\n## B\n\nEvidence B.\n", name="added.md" + ) + + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + assert report["corpus_freshness"]["added"] == ["raw/papers/added.md"] + with pytest.raises(RetrievalIndexError) as caught: + search_evidence(db_path, "Evidence") + assert caught.value.code == "index-stale" + + +def test_removed_raw_document_marks_coverage_and_search_stale(tmp_path: Path): + workspace = _workspace(tmp_path) + path, _ = _write_anchored( + workspace, "# Manual\n\n## A\n\nEvidence A.\n" + ) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + path.unlink() + + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + assert report["corpus_freshness"]["removed"] == ["raw/papers/manual.md"] + with pytest.raises(RetrievalIndexError) as caught: + search_evidence(db_path, "Evidence") + assert caught.value.code == "index-stale" + + +def _conversion_issue_codes(error: RetrievalIndexError) -> list[str]: + codes: list[str] = [] + for item in error.details.get("errors", []): + issue = item.get("details", {}).get("issue", {}) + code = issue.get("code") + if isinstance(code, str): + codes.append(code) + return codes + + +def test_valid_converted_document_freezes_source_binding_in_manifest(tmp_path: Path): + workspace = _workspace(tmp_path) + source, _markdown, _outline = _write_converted(workspace) + report = rebuild_index(workspace) + assert report["ok"] is True + + document = report["documents"][0] + receipt = json.loads( + source.with_suffix(".outline.json").read_text(encoding="utf-8") + )["conversion_receipt"] + assert document["document_origin"] == "converted" + assert document["original_source_path"] == "raw/papers/manual.pdf" + assert document["original_source_sha256"] == receipt["source_sha256"] + assert document["converter_fingerprint"] == receipt["converter_fingerprint"] + assert document["conversion_receipt_hash"] == receipt["receipt_sha256"] + + +@pytest.mark.parametrize("change", ["rewrite", "delete"]) +def test_original_container_drift_blocks_coverage_search_and_read( + tmp_path: Path, change: str +): + workspace = _workspace(tmp_path) + source, _markdown, _outline = _write_converted(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + hit = search_evidence(db_path, "蓝鲸协议")["hits"][0] + + if change == "rewrite": + # Same length defeats size/mtime-style shortcuts; the receipt uses SHA. + source.write_bytes(b"original-container-v2") + expected_code = "conversion-source-sha256-mismatch" + else: + source.unlink() + expected_code = "conversion-source-missing" + + report = coverage_report(db_path) + assert report["ok"] is False + assert report["coverage_status"] == "stale-corpus" + issues = report["corpus_freshness"]["conversion_source_issues"] + assert issues[0]["issue"]["code"] == expected_code + for action in ( + lambda: search_evidence(db_path, "蓝鲸协议"), + lambda: read_evidence_unit(db_path, hit["unit_id"]), + ): + with pytest.raises(RetrievalIndexError) as caught: + action() + assert caught.value.code == "index-stale" + + +@pytest.mark.parametrize("mutation", ["remove-receipt", "wrong-schema", "wrong-path"]) +def test_rebuild_rejects_missing_or_tampered_conversion_receipt( + tmp_path: Path, mutation: str +): + workspace = _workspace(tmp_path) + _source, _markdown, outline_path = _write_converted(workspace) + outline = json.loads(outline_path.read_text(encoding="utf-8")) + if mutation == "remove-receipt": + outline.pop("conversion_receipt") + expected = "conversion-receipt-missing" + elif mutation == "wrong-schema": + outline["conversion_receipt"]["schema_version"] = True + expected = "conversion-receipt-invalid-type" + else: + receipt = outline["conversion_receipt"] + receipt["source_path"] = "raw/papers/other.pdf" + receipt["receipt_sha256"] = conversion_receipt.canonical_receipt_digest(receipt) + expected = "conversion-derived-path-mismatch" + outline_path.write_text(json.dumps(outline), encoding="utf-8") + + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + assert caught.value.code == "rebuild-document-errors" + assert expected in _conversion_issue_codes(caught.value) + + +def test_source_and_receipt_update_with_unchanged_markdown_requires_rebuild( + tmp_path: Path, +): + workspace = _workspace(tmp_path) + source, _markdown, outline_path = _write_converted(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + source.write_bytes(b"replacement-container") + outline = json.loads(outline_path.read_text(encoding="utf-8")) + outline["conversion_receipt"] = build_conversion_receipt( + source, workspace, require_raw=True + ) + outline_path.write_text(json.dumps(outline), encoding="utf-8") + + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + assert report["corpus_freshness"]["binding_changed"] == [ + "raw/papers/manual.md" + ] + assert report["corpus_freshness"]["conversion_source_issues"] == [] + + +def test_current_converter_fingerprint_drift_invalidates_existing_index( + tmp_path: Path, monkeypatch +): + workspace = _workspace(tmp_path) + _write_converted(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + monkeypatch.setattr( + conversion_receipt, "current_converter_fingerprint", lambda _suffix: "0" * 64 + ) + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + issue = report["corpus_freshness"]["conversion_source_issues"][0]["issue"] + assert issue["code"] == "conversion-fingerprint-mismatch" + + +def test_same_stem_conversion_sources_are_rejected_before_index_build(tmp_path: Path): + workspace = _workspace(tmp_path) + _write_converted(workspace) + (workspace / "raw" / "papers" / "manual.docx").write_bytes(b"second-owner") + + with pytest.raises(RetrievalIndexError) as caught: + rebuild_index(workspace) + assert caught.value.code == "rebuild-document-errors" + assert "conversion-source-collision" in _conversion_issue_codes(caught.value) + + +def test_conversion_source_symlink_retarget_marks_existing_index_stale(tmp_path: Path): + workspace = _workspace(tmp_path) + source, _markdown, _outline = _write_converted(workspace) + db_path = workspace / ".cache" / "retrieval_index.db" + rebuild_index(workspace) + + external = workspace / "external.pdf" + external.write_bytes(source.read_bytes()) + source.unlink() + source.symlink_to(external) + report = coverage_report(db_path) + assert report["coverage_status"] == "stale-corpus" + issue = report["corpus_freshness"]["conversion_source_issues"][0]["issue"] + assert issue["code"] == "conversion-source-symlink-refused" diff --git a/scripts/tests/test_section_parser.py b/scripts/tests/test_section_parser.py index aa6dc73..8c74151 100755 --- a/scripts/tests/test_section_parser.py +++ b/scripts/tests/test_section_parser.py @@ -63,6 +63,11 @@ def test_list(self): assert len(blocks) == 1 assert blocks[0].kind == "list" + def test_ordered_list_parenthesis_marker(self): + blocks = split_blocks("1) one\n2) two\n") + assert len(blocks) == 1 + assert blocks[0].kind == "list" + def test_blockquote(self): blocks = split_blocks("> quoted\n> more\n") assert len(blocks) == 1 @@ -79,17 +84,100 @@ def test_code_fence_with_heading_inside_not_recognized(self): assert len(blocks) == 1 assert blocks[0].kind == "code" + def test_indented_code_fence(self): + blocks = split_blocks(" ```text\n# not heading\n ```\n") + assert len(blocks) == 1 + assert blocks[0].kind == "code" + + def test_four_backtick_fence_is_not_closed_by_three(self): + blocks = split_blocks("````text\n```\n# still code\n````\n") + assert len(blocks) == 1 + assert blocks[0].kind == "code" + + def test_backticks_with_trailing_text_are_not_a_closing_fence(self): + blocks = split_blocks("```text\n```junk\n# still code\n```\n") + assert len(blocks) == 1 + assert blocks[0].kind == "code" + + def test_crlf_fence_closes_before_following_heading(self): + blocks = split_blocks("```python\r\n# inside\r\n```\r\n# outside\r\n") + assert [block.kind for block in blocks] == ["code", "heading"] + assert blocks[1].title == "outside" + def test_table(self): text = "| a | b |\n|---|---|\n| 1 | 2 |\n" blocks = split_blocks(text) assert len(blocks) == 1 assert blocks[0].kind == "table" + def test_gfm_table_without_outer_pipes(self): + text = "a | b\n---|---\n1 | 2\n" + blocks = split_blocks(text) + assert len(blocks) == 1 + assert blocks[0].kind == "table" + def test_figure(self): blocks = split_blocks("![caption](image.png)\n") assert len(blocks) == 1 assert blocks[0].kind == "figure" + def test_gfm_table_stops_before_adjacent_list_without_blank_line(self): + blocks = split_blocks( + "metric | value\n--- | ---\nlatency | 17 ms\n- independent detail" + ) + assert [block.kind for block in blocks] == ["table", "list"] + assert blocks[0].text == "metric | value\n--- | ---\nlatency | 17 ms" + assert blocks[1].text == "- independent detail" + + def test_gfm_table_requires_matching_header_and_separator_columns(self): + blocks = split_blocks("metric | value\n--- | --- | ---\nlatency | 17 ms") + assert [block.kind for block in blocks] == ["paragraph"] + + def test_gfm_table_keeps_legacy_standalone_table_anchor(self): + text = "metric | value\n--- | ---\nlatency | 17 ms\n ^t-1-abc123" + blocks = split_blocks(text) + assert [block.kind for block in blocks] == ["table"] + assert blocks[0].text.endswith(" ^t-1-abc123") + + @pytest.mark.parametrize( + "row", + [ + r"A | one \| two", + "A | `x|y`", + "A | B", # a short body row is padded by GFM renderers + ], + ) + def test_gfm_table_accepts_common_body_row_shapes(self, row): + blocks = split_blocks(f"c1 | c2 | c3\n--- | --- | ---\n{row}") + assert [block.kind for block in blocks] == ["table"] + + def test_gfm_table_header_and_delimiter_may_contain_code_or_escaped_pipes(self): + blocks = split_blocks( + r"name | `x|y` | escaped \| label" "\n--- | --- | ---\nA | B | C" + ) + assert [block.kind for block in blocks] == ["table"] + + def test_gfm_single_column_table_and_short_body_row_are_preserved(self): + single = split_blocks("| only |\n|---|\n| value |") + assert [block.kind for block in single] == ["table"] + + short = split_blocks( + "| c1 | c2 | c3 |\n|---|---|---|\n| onlyone |\n- next block" + ) + assert [block.kind for block in short] == ["table", "list"] + assert short[0].text.endswith("| onlyone |") + + @pytest.mark.parametrize( + ("following", "kind"), + [("- list item with A | B", "list"), ("> quoted A | B", "blockquote")], + ) + def test_gfm_table_stops_before_non_table_block_with_pipe(self, following, kind): + blocks = split_blocks( + "| key | value |\n| --- | --- |\n| a | b |\n" + following + ) + assert [block.kind for block in blocks] == ["table", kind] + assert blocks[0].text.endswith("| a | b |") + def test_hr(self): blocks = split_blocks("---\n") assert len(blocks) == 1 diff --git a/scripts/tests/test_stage2_eval.py b/scripts/tests/test_stage2_eval.py new file mode 100644 index 0000000..16ab9b6 --- /dev/null +++ b/scripts/tests/test_stage2_eval.py @@ -0,0 +1,166 @@ +"""Unified Stage-2 public gate terminal-state parsing guards.""" +from __future__ import annotations + +import copy +import json + +import pytest + +from evals import run_stage2 + + +def test_multiline_json_terminal_state_is_accepted(): + payload = {"status": "passed", "nested": {"value": 1}} + rendered = json.dumps(payload, indent=2) + assert run_stage2._parse_terminal_json(rendered, "fixture") == payload + + +def test_prefixed_single_line_terminal_state_is_accepted(): + payload = {"status": "protocol-smoke-passed", "passed": True} + rendered = "CONVERSION_FIDELITY_RESULT " + json.dumps(payload) + assert run_stage2._parse_terminal_json(rendered, "fixture") == payload + + +def test_missing_structured_terminal_state_fails_closed(): + with pytest.raises(run_stage2.Stage2ProtocolError, match="structured JSON"): + run_stage2._parse_terminal_json("looks good\nPASS", "fixture") + + +def _metric_rows(specs, denominators): + return { + name: { + "numerator": 0 if direction == "maximum" else denominators[name], + "denominator": denominators[name], + "value": 0.0 if direction == "maximum" else 1.0, + } + for name, (direction, _threshold) in specs.items() + } + + +def test_answer_terminal_contract_requires_exact_metrics_and_recomputes_rows(): + metrics = _metric_rows( + run_stage2.ANSWER_METRICS, run_stage2.ANSWER_DENOMINATORS + ) + payload = { + "schema_version": run_stage2.ANSWER_SCHEMA_VERSION, + "scope": "public-cc0-protocol-smoke-not-model-capability", + "status": "protocol-smoke-passed", + "passed": True, + "fixture": { + "cases": 6, + "unique_questions": 6, + "answerable_cases": 4, + "unanswerable_cases": 2, + "required_facets": 7, + "public_fixture_sha256": run_stage2.ANSWER_PUBLIC_FIXTURE_SHA256, + "gold_sha256": run_stage2.ANSWER_GOLD_SHA256, + }, + "metrics": metrics, + "thresholds": { + name: { + ("max" if direction == "maximum" else "min"): threshold, + "passed": True, + } + for name, (direction, threshold) in run_stage2.ANSWER_METRICS.items() + }, + } + assert run_stage2._answer_terminal_contract(payload) == (True, []) + + payload["metrics"]["citation_completeness"]["denominator"] = 1 + payload["metrics"]["citation_completeness"]["numerator"] = 1 + passed, errors = run_stage2._answer_terminal_contract(payload) + assert passed is False + assert any("expected denominator 7" in error for error in errors) + + payload["metrics"] = {"dummy": {"passed": True}} + passed, errors = run_stage2._answer_terminal_contract(payload) + assert passed is False + assert any("metric keys differ" in error for error in errors) + + +def test_holdout_terminal_contract_rejects_wrong_direction_and_value_math(): + metrics = _metric_rows( + run_stage2.HOLDOUT_METRICS, run_stage2.HOLDOUT_DENOMINATORS + ) + payload = { + "schema_version": run_stage2.HOLDOUT_SCHEMA_VERSION, + "certification_kind": "public-smoke", + "is_hidden_certification": False, + "runner_verified_independence": False, + "runner_version": "1.0.0", + "bundle_id": "groundmap-public-holdout-smoke", + "bundle_version": "1.0.0", + "top_k": 20, + "evaluation_scope": "retrieval-only exact evidence selection", + "certification_statement": ( + "Public Gold is visible: protocol smoke only, never hidden certification." + ), + "failed_case_ids": [], + "status": "protocol-smoke-passed", + "passed": True, + "preregistered_denominators": copy.deepcopy( + run_stage2.HOLDOUT_PREREGISTERED_DENOMINATORS + ), + "metrics": metrics, + "thresholds": { + name: { + "direction": direction, + "threshold": threshold, + "value": metrics[name]["value"], + "passed": True, + } + for name, (direction, threshold) in run_stage2.HOLDOUT_METRICS.items() + }, + } + assert run_stage2._holdout_terminal_contract(payload) == (True, []) + + contradictory = copy.deepcopy(payload) + contradictory["failed_case_ids"] = ["cannot-coexist-with-51-of-51"] + passed, errors = run_stage2._holdout_terminal_contract(contradictory) + assert passed is False + assert any("contradicts fixed regression" in error for error in errors) + + shrunk = copy.deepcopy(payload) + shrunk["preregistered_denominators"]["cases"] = 1 + passed, errors = run_stage2._holdout_terminal_contract(shrunk) + assert passed is False + assert any("preregistered denominators differ" in error for error in errors) + + payload["thresholds"]["forbidden_selected_rate"]["direction"] = "minimum" + payload["metrics"]["answer_coverage"]["value"] = 0.5 + passed, errors = run_stage2._holdout_terminal_contract(payload) + assert passed is False + assert any("direction changed" in error for error in errors) + assert any("does not equal" in error for error in errors) + + +def test_conversion_terminal_contract_rejects_dummy_or_missing_metrics(): + metrics = _metric_rows( + run_stage2.CONVERSION_METRICS, run_stage2.CONVERSION_DENOMINATORS + ) + for row in metrics.values(): + row.update({"threshold": 1.0, "passed": True}) + payload = { + "gold_schema_version": run_stage2.CONVERSION_GOLD_SCHEMA_VERSION, + "gold_sha256": run_stage2.CONVERSION_GOLD_SHA256, + "fixture_semantic_sha256": run_stage2.CONVERSION_FIXTURE_SEMANTIC_SHA256, + "documents": 3, + "status": "protocol-smoke-passed", + "overall_passed": True, + "metrics": metrics, + } + assert run_stage2._conversion_terminal_contract(payload) == (True, []) + + payload["metrics"]["mutation_sensitivity"].update({ + "numerator": 1, + "denominator": 1, + "value": 1.0, + }) + passed, errors = run_stage2._conversion_terminal_contract(payload) + assert passed is False + assert any("expected denominator 14" in error for error in errors) + + payload["metrics"] = {"dummy": {"passed": True}} + passed, errors = run_stage2._conversion_terminal_contract(payload) + assert passed is False + assert any("metric keys differ" in error for error in errors) diff --git a/tools/cite-audit/README.md b/tools/cite-audit/README.md new file mode 100644 index 0000000..3f23d45 --- /dev/null +++ b/tools/cite-audit/README.md @@ -0,0 +1,48 @@ +# cite-audit — 跨模型引用二审客户端 + +对 wiki 的（论断, 引用）审计对做**换一家模型**的二次审核，判定回写验证台账。 +与 `tools/debug-console` 同类：KB 的**外部客户端**（CLAUDE.md 原则 1 例外区）， +只经 `scripts/k.py` CLI 取数 / 回写，删掉本目录不影响 KB 任何功能。 + +## 为什么存在 + +- **跨模型**：写作与回验若同源（同一个模型 / 同一会话），判定失误存在相关性； + 换模型二审显著降低相关盲区。 +- **独立执行器**：本工具由人工或定时任务触发，自己跑完全流程（取数 → 盲填 → + 反驳 → 机器判分 → 入台账），不依赖写作 agent 的协议自觉。 + +## 双通道判定 + +| 通道 | 机制 | 防什么 | +|---|---|---| +| A 盲填 | 模型只看「挖空论断 + 被引原文」填数字（看不到期望值），`k.py cloze-check` 机器判分 | 判定式审核的附和偏差；数字巧合在场但归属错误 | +| B 反驳 | 模型以「尽力反驳」立场审支撑关系；反驳失败须给原文佐证片段（台账 evidence，k.py 字面校验） | 曲解 / 张冠李戴 / 过度概括 | + +双通道均过 → SUPPORTED 入台账；任一未过 → UNSUPPORTED（打印清单，按 kb-cite-audit +流程落 CAUTION 标注交人裁决——本工具不改 wiki）。 + +## 用法 + +```bash +# API key 走环境变量（不要硬编码）：export DEEPSEEK_API_KEY=... +# 增量审（默认：只审未审过 / 内容漂移的对） +python tools/cite-audit/audit.py --workspace + +# 本次 git 改动 / 指定页面 / 全量交叉复查（忽略台账重审，抽查 agent 自审质量） +python tools/cite-audit/audit.py --workspace --changed +python tools/cite-audit/audit.py --workspace --paths wiki/concepts/x.md +python tools/cite-audit/audit.py --workspace --all --sample 20 --seed 2026-W27 + +# 独立数据根（KB_ROOT 语义同 k.py）；只看判定不写台账 +python tools/cite-audit/audit.py --workspace main --kb-root /path/to/kb-data --dry-run +``` + +默认模型 `deepseek-v4-flash`（便宜、够用）；高难度判定可 `--model deepseek-v4-pro`。 +退出码：0 = 全部通过；1 = 有未通过项；2 = 环境错误。 + +## 约束 + +- 只在 raw 在场的环境有意义（release demo 库的 raw 引用会被 extract-claims 标 + `raw-not-distributed`、自动跳过）。 +- 台账（`.cache/citation_audit.jsonl`）是派生层，本工具的判定同样「删了重审即重建」。 +- 不改任何 wiki 内容；未通过项的 CAUTION 标注由 agent / 人按 kb-cite-audit 流程落。 diff --git a/tools/cite-audit/audit.py b/tools/cite-audit/audit.py new file mode 100644 index 0000000..fa43f2b --- /dev/null +++ b/tools/cite-audit/audit.py @@ -0,0 +1,467 @@ +#!/usr/bin/env python +"""跨模型引用二审客户端（cross-model cite audit）。 + +架构定位：与 tools/debug-console 同类的**外部客户端**（CLAUDE.md 原则 1 例外区）—— +KB 核心（scripts/、web/）零 LLM；本工具只经 `k.py` CLI 取数 / 回写台账，调用 +外部 LLM（默认 DeepSeek）做语义判定。删掉 tools/ 整个目录不影响 KB 任何功能。 + +为什么要「跨模型」：写作 agent 与回验 agent 若同源，存在相关性盲区；换一家模型 +做二审，判定失误的相关性显著降低。同时本工具作为**独立执行器**（人工 / 定时任务 +触发），不依赖写作 agent 的协议自觉——它自己跑完「取数 → 盲填 → 反驳 → 机器判分 +→ 入台账」全流程。 + +双通道判定： + A 盲填（blind cloze）：模型只看「挖空论断 + 被引原文」填数字，全程看不到期望值 + ——从原理上消灭判定式审核的附和偏差；填回值由 `k.py cloze-check` 机器判分。 + B 反驳（refute）：模型以「尽力反驳」立场判断被引原文是否支撑论断——反驳成功 + 即未通过；反驳失败须给出原文佐证片段（写入台账 evidence，k.py 会字面校验）。 + 组合：A 判分失败 或 B 反驳成功 → UNSUPPORTED/CONTRADICTED；双通道均过 → SUPPORTED。 + +用法（在引擎根运行；数据库经 KB_ROOT / --workspace 定位）： + export DEEPSEEK_API_KEY=... # 不要硬编码 key + python tools/cite-audit/audit.py --workspace [--kb-root PATH] \ + [--changed | --unaudited-only | --paths p1 p2 | --all] \ + [--sample N --seed YYYY-WW] [--model deepseek-v4-flash] [--dry-run] + + # 问答草稿：取证、盲填、反驳、入账后再跑 check-draft --strict + python tools/cite-audit/audit.py --workspace --draft /tmp/answer.md + + --dry-run 只打印判定，不写台账 + 默认 --unaudited-only（增量）；--all 忽略台账全量重审（交叉复查用） +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import subprocess +import sys +import time +import unicodedata +import urllib.request +from decimal import Decimal, InvalidOperation + +ENGINE_ROOT = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) +K_PY = os.path.join(ENGINE_ROOT, "scripts", "k.py") + +API_URL = "https://api.deepseek.com/chat/completions" +FULL_EVIDENCE_CHARS = 10 ** 9 +RESULT_PREFIX = "CITE_AUDIT_RESULT " + +FILL_SYSTEM = ( + "你是一名严格的填空员。给你一段【原文】和一句挖了空的【论断】（空位形如 ⟦N1⟧）。" + "只依据原文，把每个空位应填的数值填回去；原文里找不到对应数值的空位填 \"unknown\"。" + "禁止利用你自己的知识猜测。只输出一个 JSON 对象，键是空位名，值是填入的数字串，" + '例如 {"N1": "72.4%", "N2": "unknown"}，不要输出其他任何文字。' +) + +REFUTE_SYSTEM = ( + "你是一名对抗性引用审查员，立场是【尽力反驳】：判断给出的【被引原文】是否真的支撑【论断】。" + "重点攻击：关键事实是否缺失、数字归属的主体是否张冠李戴、有无过度概括或与原文相反的结论。" + "注意：论断块里可能含多条子论断、各挂各的引用；只审与本条被引原文对应的部分，" + "其他子论断挂别的引用不归你管。被引原文是整节时，论断的关键事实分布在节内多处属正常；" + "「见 / 详见 / 参见」类导航措辞、以及数字后显式挂 [需要来源] 的部分，都不构成反驳理由。" + "只输出一个 JSON 对象：" + '{"refuted": true/false, "reason": "一句话", "quote": "未反驳成功时，从原文逐字复制一段最能支撑论断的文字（8-60字），反驳成功时留空"}' + "，不要输出其他任何文字。" +) + + +def emit_result(**payload) -> None: + """输出给 eval runner 的机器可读终态；人类日志不参与判定。""" + print(RESULT_PREFIX + json.dumps(payload, ensure_ascii=False, sort_keys=True)) + + +_NUMERIC_FILL_RE = re.compile( + r"^([+-]?(?:(?:\d{1,3}(?:,\d{3})+)(?:\.\d+)?|\d+(?:\.\d+)?|\.\d+))(.*)$" +) + + +def _norm_fill_value(value: str) -> str: + """盲填值比较用的 canonical form。 + + 数值部分用 Decimal 消除无意义精度和千分位差异；后缀仍保留， + 所以 87.5% == 87.50％、1,000 == 1000，但 87.5% != 87.6%。 + """ + norm = re.sub(r"\s+", "", unicodedata.normalize("NFKC", value)).casefold() + match = _NUMERIC_FILL_RE.fullmatch(norm) + if not match: + return norm + number, suffix = match.groups() + try: + parsed = Decimal(number.replace(",", "")) + except InvalidOperation: + return norm + if parsed == 0: + canonical_number = "0" + else: + canonical_number = format(parsed.normalize(), "f") + if suffix in {"percent", "percentage", "百分比"}: + suffix = "%" + return f"number:{canonical_number}:{suffix}" + + +def merge_cloze_fills(fill_sets: list[dict]) -> tuple[dict[str, str], dict[str, list[str]]]: + """合并同一论断块的多条 evidence 盲填结果。 + + unknown 不参与合并；全角/半角与空白差异视为同值。同一空位出现 + 两个不同非 unknown 值时必须报冲突，不得「first wins」把不确定性藏掉。 + """ + seen: dict[str, dict[str, str]] = {} + for fills in fill_sets: + if not isinstance(fills, dict): + continue + for ph, value in fills.items(): + raw = str(value).strip() + norm = _norm_fill_value(raw) + if not raw or norm == "unknown": + continue + seen.setdefault(str(ph), {}).setdefault(norm, raw) + merged: dict[str, str] = {} + conflicts: dict[str, list[str]] = {} + for ph, values in seen.items(): + if len(values) == 1: + merged[ph] = next(iter(values.values())) + else: + conflicts[ph] = list(values.values()) + return merged, conflicts + + +_ANCHOR_RE_INLINE = re.compile(r"\^([hpcft]-\d+(?:-\d+)?-[a-z0-9]+(?:-\d+)?)") +_ZERO_WIDTH_RE = re.compile(r"[​‌‍]") +_LATEX_SPACING_RE = re.compile(r"\\[,;:!]|\\quad|\\qquad") +_LATEX_WRAP_RE = re.compile(r"\\(?:mathbf|mathrm|mathit|text|textrm|boldsymbol)\{([^{}]*)\}") +_MAG_SUFFIX_ALT = r"[kKMB万亿]|(?i:million|billion|thousand)\b" +_NUM_TAIL_GUARD = r"(?![\d]|\.\d)" +_DECIMAL_REPEAT_RE = re.compile( + r"(? str: + """Mirror ``scripts.k._norm_for_evidence`` for preflight without writing ledger. + + 该客户端仍只经 CLI 与 KB 交互；这里完整复制入账前的纯函数归一化， + 并由测试对两者做契约对账，避免 dry-run 通过而正式入账拒绝。 + """ + value = _ANCHOR_RE_INLINE.sub("", value) + value = unicodedata.normalize("NFKC", value) + value = _ZERO_WIDTH_RE.sub("", value) + value = _LATEX_SPACING_RE.sub("", value) + for _ in range(2): + value = _LATEX_WRAP_RE.sub(r"\1", value) + value = re.sub(r"(?<=\d),(?=\d{3})", "", value) + value = re.sub(r"[*_`~]", "", value) + value = _DECIMAL_REPEAT_RE.sub(r"\1\2", value) + return re.sub(r"\s+", " ", value).strip().casefold() + + +def valid_evidence_quote(quote: str, evidence: str) -> bool: + """本地执行与 k.py 台账入口同方向的 evidence 子串门禁。 + + dry-run 不写台账，因此不能依赖 cite-audit-log 在之后替它拒绝假 quote。 + 这里故意保守：归一化后至少 8 字符，且必须是完整原文的字面子串。 + """ + needle = _norm_evidence(quote) + return len(needle) >= 8 and needle in _norm_evidence(evidence) + + +def packet_issue(pair: dict) -> str | None: + """返回核验包不完整的原因；None 表示可进入语义判定。""" + if "evidence" not in pair: + return "核验包缺少 evidence" + evidence = str(pair.get("evidence", "")) + evidence_length = pair.get("evidence_length") + if (pair.get("evidence_truncated") + or (isinstance(evidence_length, int) and len(evidence) < evidence_length)): + return "evidence 被截断，不能认证整对引用" + if pair.get("claim_text_truncated"): + return "claim_text 被截断，不能认证整个论断块" + claim = str(pair.get("claim_text", "")) + # 兼容旧 k.py：它在 500 字符后追加单字符省略号，但没有 + # claim_text_truncated 字段。新取数契约会返回完整论断与显式状态。 + if ("claim_text_truncated" not in pair + and len(claim) >= 501 and claim.endswith("…")): + return "claim_text 被截断，不能认证整个论断块" + cloze = pair.get("cloze") or {} + cloze_text = str(cloze.get("text", "")) if isinstance(cloze, dict) else "" + if (pair.get("cloze_truncated") + or (len(cloze_text) >= 601 and cloze_text.endswith("…"))): + return "cloze 被截断，不能完成整块盲填复核" + return None + + +def result_counts(returned: int, verdicts: list[dict], failures: list[tuple], + skipped: int) -> dict[str, int]: + """生成互斥的审计终态计数。 + + ``semantic_failed`` 只计已完成语义判定且产生了失败 verdict + 的 pair；截断包、无效 quote 与不可核对目标分别计入 incomplete / + skipped，绝不能通过一个混合 ``failed`` 字段冒充毒化命中。 + """ + incomplete = sum(1 for failure in failures if failure[3] == "incomplete") + semantic_failed = sum( + 1 for verdict in verdicts + if verdict.get("verdict") in {"UNSUPPORTED", "CONTRADICTED"} + ) + return { + "returned": returned, + "judged": len(verdicts), + "semantic_failed": semantic_failed, + "incomplete": incomplete, + "skipped": skipped, + } + + +def call_llm(model: str, system: str, user: str, api_key: str, retries: int = 2) -> dict: + body = json.dumps({ + "model": model, + "messages": [{"role": "system", "content": system}, + {"role": "user", "content": user}], + "temperature": 0.0, + "max_tokens": 2400, # v4 系列是 reasoning 模型：思考也计入，需留足空间 + }).encode("utf-8") + last_err: Exception | None = None + for attempt in range(retries + 1): + try: + req = urllib.request.Request( + API_URL, data=body, + headers={"Content-Type": "application/json", + "Authorization": f"Bearer {api_key}"}) + with urllib.request.urlopen(req, timeout=90) as resp: + data = json.loads(resp.read().decode("utf-8")) + msg = data["choices"][0]["message"] + text = (msg.get("content") or "").strip() + if not text: # reasoning 模型思考未完成（finish_reason=length 等）→ 当失败重试 + raise ValueError(f"content 为空（finish_reason={data['choices'][0].get('finish_reason')}）") + m = re.search(r"\{[\s\S]*\}", text) # 容忍模型包了 ```json 围栏 + return json.loads(m.group(0) if m else text) + except Exception as e: # noqa: BLE001 —— 网络 / 解析失败统一重试 + last_err = e + time.sleep(2 * (attempt + 1)) + raise RuntimeError(f"LLM 调用失败（重试 {retries} 次后）：{last_err}") + + +def run_k(args_list: list[str], env: dict) -> str: + r = subprocess.run([sys.executable, K_PY] + args_list, + capture_output=True, text=True, env=env) + if r.returncode not in (0, 1): # 1 = lint 有 finding，正常 + raise RuntimeError(f"k.py {' '.join(args_list[:3])} 失败: {r.stderr.strip()[:200]}") + return r.stdout + + +def main() -> int: + ap = argparse.ArgumentParser(description="跨模型引用二审（盲填 + 反驳双通道）") + ap.add_argument("--workspace", required=True) + ap.add_argument("--kb-root", help="数据根（含 workspaces/ 的目录；默认引擎根）") + ap.add_argument("--changed", action="store_true") + ap.add_argument("--unaudited-only", action="store_true") + ap.add_argument("--all", action="store_true", help="忽略台账全量重审（交叉复查）") + ap.add_argument("--paths", nargs="+") + ap.add_argument("--draft", help="审计查询/导出草稿；与 --changed/--paths/--all 互斥") + ap.add_argument("--sample", type=int) + ap.add_argument("--seed", default="") + ap.add_argument("--model", default="deepseek-v4-flash") + ap.add_argument("--dry-run", action="store_true") + args = ap.parse_args() + + if args.draft and (args.changed or args.paths or args.all or args.unaudited_only + or args.sample is not None): + ap.error("--draft 不能与 --changed/--paths/--all/--unaudited-only/--sample 同用") + if args.draft and not os.path.isfile(os.path.expanduser(args.draft)): + ap.error(f"草稿文件不存在: {args.draft}") + + api_key = os.environ.get("DEEPSEEK_API_KEY", "") + if not api_key: + print("错误: 环境变量 DEEPSEEK_API_KEY 为空（先 source ~/.zshrc）", file=sys.stderr) + emit_result(status="error", dry_run=args.dry_run, returned=0, judged=0, + semantic_failed=0, incomplete=0, skipped=0, + ledger_errors=0, ledger_written=0, error="DEEPSEEK_API_KEY 为空") + return 2 + + env = dict(os.environ) + if args.kb_root: + env["KB_ROOT"] = args.kb_root + + if args.draft: + ec = ["--workspace", args.workspace, "check-draft", + os.path.expanduser(args.draft), "--with-evidence", + "--max-evidence-chars", str(FULL_EVIDENCE_CHARS), + "--cloze", "--json"] + else: + ec = ["--workspace", args.workspace, "extract-claims", + "--with-evidence", "--max-evidence-chars", str(FULL_EVIDENCE_CHARS), + "--cloze", "--json"] + if args.changed: + ec.append("--changed") + if args.unaudited_only or not (args.changed or args.all or args.paths): + ec.append("--unaudited-only") + if args.paths: + ec += ["--paths"] + args.paths + if args.sample: + ec += ["--sample", str(args.sample), "--seed", args.seed] + data = json.loads(run_k(ec, env)) + pairs = [p for p in data["pairs"] if p["target_status"] == "ok"] + packet_issues = {p["pair_id"]: issue for p in pairs if (issue := packet_issue(p))} + valid_pairs = [p for p in pairs if p["pair_id"] not in packet_issues] + returned = int(data["summary"]["returned"]) + skipped = returned - len(pairs) + print(f"待审 {len(pairs)} 对（跳过不可核对 {skipped}）| 模型: {args.model}") + for p in pairs: + if p["pair_id"] in packet_issues: + print(f" ⚠️ {p['pair_id']} {p['page']}:{p['line']} — {packet_issues[p['pair_id']]}") + + # ── 通道 A：盲填，按「论断块」分组判分 ───────────────────────── + # cloze 按块挖空，而一块常挂多条引用、数字分散在各引用的原文里——单 evidence + # 必然对不归它的空填 unknown。正确语义与确定性 lint 的 union 一致：对块内每条 + # 引用的原文分别盲填，**合并**各空的非 unknown 填值后块级判分一次。 + blocks: dict[tuple, list[dict]] = {} + for p in valid_pairs: + blocks.setdefault((p["page"], p["line"]), []).append(p) + # (passed, detail, incomplete)：incomplete 用于协议异常 / 多 evidence 冲突， + # 与「完整盲填后机器判分未过」的真实语义失败分开。 + block_cloze: dict[tuple, tuple[bool | None, str, bool]] = {} + for key, group in blocks.items(): + rep = group[0] + if not rep.get("cloze"): + block_cloze[key] = (None, "", False) + continue + fill_sets: list[dict] = [] + malformed_fill = False + for p in group: + fills = call_llm(args.model, FILL_SYSTEM, + f"【原文】\n{p.get('evidence', '')}\n\n【挖空论断】\n{rep['cloze']['text']}", + api_key) + if not isinstance(fills, dict): + malformed_fill = True + break + fill_sets.append(fills) + if malformed_fill: + block_cloze[key] = (None, "盲填通道未返回 JSON object", True) + continue + merged, conflicts = merge_cloze_fills(fill_sets) + if conflicts: + detail = "; ".join( + f"{ph}=" + " / ".join(values) for ph, values in sorted(conflicts.items()) + ) + block_cloze[key] = (None, f"多来源盲填冲突: {detail}", True) + continue + cloze_args = ["--workspace", args.workspace, "cloze-check", + "--pair", rep["pair_id"], "--fills", + json.dumps(merged, ensure_ascii=False), "--json"] + if args.draft: + cloze_args += ["--draft", os.path.expanduser(args.draft)] + check = json.loads(run_k(cloze_args, env))[0] + detail = "; ".join( + f"{b['ph']}:{'✓' if b['ok'] else '✗ 填 ' + (b['fill'] or '<空>') + ' 期望 ' + b['expected']}" + for b in check.get("blanks", [])) + block_cloze[key] = (bool(check.get("passed")), detail, False) + + # ── 通道 B：反驳，按 pair ────────────────────────────────────── + verdicts, failures = [], [] + for p in pairs: + pid = p["pair_id"] + loc = f"{p['page']}:{p['line']}" + if pid in packet_issues: + failures.append((pid, loc, packet_issues[pid], "incomplete")) + continue + cloze_pass, cloze_detail, cloze_incomplete = block_cloze[(p["page"], p["line"])] + if cloze_incomplete: + failures.append((pid, loc, cloze_detail, "incomplete")) + print(f" ⚠️ {pid} {loc} — {cloze_detail[:100]}") + continue + ref = call_llm(args.model, REFUTE_SYSTEM, + f"【论断】\n{p['claim_text']}\n\n【被引原文】（本条引用 [[{p['target']}#^{p['anchor']}]] 的目标块）\n{p.get('evidence', '')}", + api_key) + if not isinstance(ref, dict) or not isinstance(ref.get("refuted"), bool): + reason = "反驳通道未返回含 boolean refuted 的 JSON object" + failures.append((pid, loc, reason, "incomplete")) + print(f" ⚠️ {pid} {loc} — {reason}") + continue + refuted = ref["refuted"] + if refuted or cloze_pass is False: + reason = ref.get("reason", "") if refuted else f"盲填未过(块级合并): {cloze_detail}" + verdicts.append({"pair_id": pid, "verdict": "UNSUPPORTED", + "note": f"cross-model:{args.model} | {reason}"[:400]}) + failures.append((pid, loc, reason, "semantic")) + print(f" ❌ {pid} {loc} — {reason[:100]}") + else: + quote = (ref.get("quote") or "").strip() + if not valid_evidence_quote(quote, p.get("evidence", "")): + reason = "反驳通道未返回可在完整被引原文中定位的有效 quote（至少 8 字符）" + failures.append((pid, loc, reason, "incomplete")) + print(f" ⚠️ {pid} {loc} — {reason}") + continue + verdicts.append({"pair_id": pid, "verdict": "SUPPORTED", + "note": f"cross-model:{args.model} | 盲填{'通过' if cloze_pass else '不适用(无数字)'} + 反驳失败", + "evidence": quote}) + print(f" ✅ {pid} {loc}") + + counts = result_counts(returned, verdicts, failures, skipped) + if args.dry_run or not verdicts: + print(f"\n{'[dry-run] ' if args.dry_run else ''}判定 {len(verdicts)} 条" + f"（未通过 {len(failures)}），未写台账") + unfinished = bool(counts["incomplete"] or counts["skipped"]) + emit_result(status="incomplete" if unfinished else "completed", + dry_run=args.dry_run, **counts, + ledger_errors=0, ledger_written=0) + if unfinished: + return 2 + return 1 if counts["semantic_failed"] else 0 + + import tempfile + with tempfile.NamedTemporaryFile("w", suffix=".jsonl", delete=False, + encoding="utf-8") as f: + for v in verdicts: + f.write(json.dumps(v, ensure_ascii=False) + "\n") + batch = f.name + try: + ledger_args = ["--workspace", args.workspace, "cite-audit-log", + "--batch", batch, "--mode", "audit", "--json"] + if args.draft: + ledger_args += ["--draft", os.path.expanduser(args.draft)] + out = run_k(ledger_args, env) + finally: + os.unlink(batch) + result = json.loads(out) + print(f"\n台账写入 {len(result['written'])} 条 / 拒绝 {len(result['errors'])} 条") + for e in result["errors"]: + print(f" ⚠️ {e['pair_id']}: {e['error'][:120]}") + semantic_failures = [f for f in failures if f[3] == "semantic"] + incomplete_failures = [f for f in failures if f[3] == "incomplete"] + if semantic_failures: + print(f"\n语义未通过 {len(semantic_failures)} 条——按 kb-cite-audit 流程落 CAUTION 标注交人裁决：") + for _pid, loc, reason, _kind in semantic_failures: + print(f" {loc}: {reason[:120]}") + if incomplete_failures: + print(f"\n核验未完成 {len(incomplete_failures)} 条（未记 SUPPORTED，也不将执行异常冒充语义错引）：") + for _pid, loc, reason, _kind in incomplete_failures: + print(f" {loc}: {reason[:120]}") + ledger_errors = len(result["errors"]) + unfinished = bool(counts["incomplete"] or counts["skipped"]) + status = "ledger-error" if ledger_errors else "incomplete" if unfinished else "completed" + emit_result(status=status, dry_run=False, **counts, + ledger_errors=ledger_errors, ledger_written=len(result["written"])) + if ledger_errors: + return 2 + if unfinished: + return 2 + return 1 if counts["semantic_failed"] else 0 + + +def cli_main() -> int: + """把网络 / JSON / k.py 协议异常统一映射为执行失败（exit 2）。""" + try: + return main() + except Exception as exc: # noqa: BLE001 —— CLI 边界统一失败协议 + print(f"错误: 引用审计执行失败: {exc}", file=sys.stderr) + emit_result(status="error", dry_run="--dry-run" in sys.argv, + returned=0, judged=0, semantic_failed=0, incomplete=0, + skipped=0, ledger_errors=0, ledger_written=0, + error=str(exc)[:500]) + return 2 + + +if __name__ == "__main__": + sys.exit(cli_main()) diff --git a/tools/debug-console/README.md b/tools/debug-console/README.md index b0c61ff..4fca1a5 100644 --- a/tools/debug-console/README.md +++ b/tools/debug-console/README.md @@ -2,7 +2,7 @@ **可选的独立子项目**：一个带 LLM 的聊天界面，用来调试和演示「外部 agent 如何调用知识库工具回答问题」——每一步工具调用（search / outline / read-section / backlinks…）和返回数据都可视化为流程图。 -> **与 CLAUDE.md 原则 1 的关系**：KB 核心（`scripts/`、`web/`）严禁内嵌 LLM；本子项目**作为 KB 的外部客户端存在**，所以可以引入 LLM SDK。它只通过 HTTP 调主管理台的 `POST /api/agent-tool`（只读工具白名单 + CSRF 防护），不直接读 markdown / `.cache/`。**删掉整个 `tools/` 目录不影响 KB 任何功能。** +> **与 CLAUDE.md 原则 1 的关系**：KB 核心（`scripts/`、`web/`）严禁内嵌 LLM；本子项目**作为 KB 的外部客户端存在**，所以可以引入 LLM SDK。它只通过 HTTP 调主管理台的 `POST /api/agent-tool`（受控白名单 + CSRF 防护），不直接读 markdown / `.cache/`。**删掉整个 `tools/` 目录不影响 KB 任何功能。** ## 启动 @@ -69,11 +69,20 @@ npm run dev # 重启生效（Next.js 启动时读 .env） 唯一耦合面是主 `web/` 的 `POST /api/agent-tool`： -- 请求体 `{ tool: "<白名单工具名>", args: {...} }`，工具名与 `scripts/k.py` 子命令对齐（只读操作：search / outline / read-section / read-block / backlinks / outlinks 等） -- 服务端白名单校验 + CSRF 防护；写操作不在白名单内 +- 请求体 `{ tool: "<白名单工具名>", args: {...} }`；常规只读工具沿用 `search` / `outline` / `read_section` / `read_block` / `backlinks` / `outlinks` 等现有名称。 +- 长文档细节召回新增 `evidence-index-coverage` / `search-evidence` / `read-evidence-unit`；建议先查覆盖与新鲜度，再搜自然证据单元，最后按 `unit_id` 读取精确原文。 +- 服务端白名单校验 + CSRF 防护；真相源写操作仍不在白名单。唯一派生写例外 `rebuild-evidence-index` 只重建 `.cache` 索引（不改 raw/wiki/Git），且明确标记为 high-cost / `derived_write` / `requires_confirmation`；调用时必须传 `confirm: "rebuild-evidence-index"`。 - 返回 k.py 的 JSON 输出，由本控制台渲染为工具调用卡片与流程图 -注意：本控制台的服务端 fetch 不携带浏览器 cookie，查询的 workspace 由主站的 `KB_WORKSPACE` 启动环境决定，与主站 UI 里的 workspace 切换器无关。 +注意：主站打开控制台时会传递经校验的 workspace 上下文；控制台的服务端 fetch 会把它写入 `kb_workspace` cookie，主站再按现有校验规则解析。接口不接受任意 workspace 参数；无合法上下文时仍回退到主站 `KB_WORKSPACE` / 默认库。 + +## 测试 + +```bash +cd tools/debug-console +npm test +npx tsc --noEmit +``` ## 技术栈 diff --git a/tools/debug-console/components/ChatPanel.tsx b/tools/debug-console/components/ChatPanel.tsx index 67783df..045864d 100644 --- a/tools/debug-console/components/ChatPanel.tsx +++ b/tools/debug-console/components/ChatPanel.tsx @@ -53,7 +53,13 @@ type IncomingEvent = error_message?: string; } | { kind: "stream-end" } - | { kind: "ref-validation"; broken: string[]; unread: string[]; downgraded?: string[] }; + | { + kind: "ref-validation"; + broken: string[]; + unread: string[]; + downgraded?: string[]; + uncited?: string[]; + }; export function ChatPanel({ provider, @@ -122,6 +128,9 @@ export function ChatPanel({ const msg = { ...prev[idx], parts: [...prev[idx].parts] }; if (evt.kind === "text-delta") { + // 验证后若又有新文本（如 agent 强制续写），旧结果立即过期。 + // 清空后 MessageBubble 会把所有引用按 pending 去链接化，直到新后验完成。 + msg.refValidation = undefined; const last = msg.parts[msg.parts.length - 1]; if (last && last.kind === "text") { msg.parts[msg.parts.length - 1] = { kind: "text", text: last.text + evt.text }; @@ -196,6 +205,7 @@ export function ChatPanel({ broken: evt.broken, unread: evt.unread, downgraded: evt.downgraded ?? [], + uncited: evt.uncited ?? [], }; } diff --git a/tools/debug-console/components/MessageBubble.tsx b/tools/debug-console/components/MessageBubble.tsx index 19557f5..78fe278 100644 --- a/tools/debug-console/components/MessageBubble.tsx +++ b/tools/debug-console/components/MessageBubble.tsx @@ -18,6 +18,8 @@ import { hrefToRef, inlineWikiRefsNumbered, neutralizeBrokenRefs, + neutralizePendingRefs, + neutralizeUnreadRefs, refDisplayText, refToHref, type WikiRef, @@ -40,8 +42,10 @@ export interface UIMessage { refValidation?: { broken: string[]; unread: string[]; - /** 锚点不支撑论断、已降级为整页链接的块级引用 key（path#anchor） */ + /** 未通过严格后验、已去链接化的引用 key（path#anchor） */ downgraded?: string[]; + /** ANSWER 中没有引用/显式免责标记的数字或事实性行 */ + uncited?: string[]; }; } @@ -101,7 +105,8 @@ export function MessageBubble({ const isError = msg.role === "error"; const isAssistant = msg.role === "assistant"; - // 后验判定"锚点不支撑论断"的块级引用 → 渲染前去掉假精度（剥成整页链接） + const validationPending = isAssistant && msg.refValidation === undefined; + // 后验判定未通过的引用 → 渲染前去链接化 const downgradedSet = useMemo( () => new Set(msg.refValidation?.downgraded ?? []), [msg.refValidation], @@ -111,15 +116,27 @@ export function MessageBubble({ () => new Set(msg.refValidation?.broken ?? []), [msg.refValidation], ); - // 统一的渲染前清洗：先降级不支撑论断的锚点，再去链接化不存在的引用 + const unreadSet = useMemo( + () => new Set(msg.refValidation?.unread ?? []), + [msg.refValidation], + ); + // 统一的渲染前清洗：未通过 / 未读取 / 不存在的引用全部去链接化，禁止扩大到整页洗白 const cleanRefs = useMemo( () => (text: string) => - neutralizeBrokenRefs( - downgradeRefAnchors(text, downgradedSet), - brokenSet, + neutralizePendingRefs( + downgradeRefAnchors( + neutralizeUnreadRefs( + neutralizeBrokenRefs(text, brokenSet, locale), + unreadSet, + locale, + ), + downgradedSet, + locale, + ), + validationPending, locale, ), - [downgradedSet, brokenSet, locale], + [downgradedSet, unreadSet, brokenSet, validationPending, locale], ); const numberedRefs = useMemo(() => { const texts = msg.parts @@ -324,6 +341,24 @@ export function MessageBubble({ )} + {(msg.refValidation.uncited?.length ?? 0) > 0 && ( +
+
+ {t("msg.uncited_claims", { n: msg.refValidation.uncited!.length })} +
+
+ {t("msg.uncited_desc")} +
+
+ {msg.refValidation.uncited!.map((claim, i) => ( +
+ ‼ + {claim} +
+ ))} +
+
+ )} )} diff --git a/tools/debug-console/lib/agent-loop.ts b/tools/debug-console/lib/agent-loop.ts index bc3075e..6f777ff 100644 --- a/tools/debug-console/lib/agent-loop.ts +++ b/tools/debug-console/lib/agent-loop.ts @@ -27,7 +27,12 @@ import { createExploreState, type ExploreState, } from "./mode-augment"; -import { collectRefs } from "./wiki-ref"; +import { + WIKI_REF_RE, + collectRefs, + normalizeBareSlugRefs, + normalizeRefPath, +} from "./wiki-ref"; import { verifyBlockCitations } from "./verify-citations"; export interface AgentRunInput { @@ -62,6 +67,60 @@ function gatherAssistantText(msgs: ChatMessage[]): string { .join("\n"); } +/** + * 只审最后一个【ANSWER】段；没有 ANSWER 时审当前全文（强制续写前也不放过）。 + * 这是轻量确定性 coverage 闸门，不尝试代替 check-draft 的完整语义枚举。 + */ +function answerBody(text: string): string { + const re = /【\s*ANSWER\s*】/gi; + let last: RegExpExecArray | null = null; + let m: RegExpExecArray | null; + while ((m = re.exec(text)) !== null) last = m; + return last ? text.slice((last.index ?? 0) + last[0].length) : text; +} + +/** + * 找出 ANSWER 中没有任何引用/显式免责标记的数字或事实性行。 + * 宁可向用户显示 caution，也不把“没写引用”当成“没有引用问题”。 + */ +function detectUncitedMaterial(text: string): string[] { + const body = answerBody(text).replace(/```[\s\S]*?```/g, ""); + const refsRe = new RegExp(WIKI_REF_RE.source, "g"); + const exemptRe = /\[(?:Agent\s*(?:综合|推断)|知识库未覆盖|需要来源|KB\s*推算[^\]]*)\]/i; + const factRe = /(?:是|属于|包含|包括|采用|达到|显示|表明|证明|发现|发生|提出|发布|增长|下降|高于|低于|优于|支持|反驳|导致|意味着|取决于)|\b(?:is|are|was|were|has|have|shows?|reports?|demonstrates?|increases?|decreases?|causes?|includes?)\b/i; + const numericRe = /\d|[\u4e00-龥]?(?:百分之|[\u4e00二三四五六七八九十百千万亿]+(?:年|月|日|个|项|倍))/; + const out: string[] = []; + const seen = new Set(); + + for (const raw of body.split(/\r?\n/)) { + const normalized = normalizeBareSlugRefs(raw); + refsRe.lastIndex = 0; + // 引用通常紧跟所支撑的论断。行内有引用时，仍检查最后一条引用 + // 之后的尾部，避免“前半句有引用，后半句裸奔”整行洗白。 + let lastRefEnd = 0; + let match: RegExpExecArray | null; + while ((match = refsRe.exec(normalized)) !== null) lastRefEnd = refsRe.lastIndex; + const candidate = lastRefEnd > 0 ? normalized.slice(lastRefEnd) : normalized; + if (exemptRe.test(candidate)) continue; + + let line = candidate + .replace(/^\s*(?:[-*+]\s+|\d+[.)、]\s*)/, "") + .replace(/^\s{0,3}#{1,6}\s*/, "") + .replace(/[|*_`>#]/g, " ") + .replace(/\s+/g, " ") + .trim(); + if (!line || /^[-: —–]+$/.test(line) || line.length < 4) continue; + if (!numericRe.test(line) && !factRe.test(line)) continue; + + line = line.slice(0, 240); + if (!seen.has(line)) { + seen.add(line); + out.push(line); + } + } + return out; +} + export async function* runAgent( input: AgentRunInput, ): AsyncGenerator { @@ -81,106 +140,104 @@ export async function* runAgent( let totalAssistantText = ""; let forcedAnswerOnce = false; - // 来源注册表：本轮对话中所有成功读取过的文件路径（防幻觉白名单） - const availableSources = new Set(); - - /** 从 tool result 提取路径加入注册表 */ - function recordSource(toolName: string, args: Record, data?: unknown) { - const p = - typeof args.path === "string" - ? args.path - : typeof args.file_path === "string" - ? args.file_path - : null; - if (p) availableSources.add(p); - // backlinks / outlinks 的结果里也包含有效路径 - if ((toolName === "backlinks" || toolName === "outlinks") && data) { - const field = toolName === "backlinks" ? "from_path" : "target"; - let arr: unknown = data; - if (!Array.isArray(arr) && typeof arr === "object") { - const obj = arr as Record; - if (Array.isArray(obj.hits)) arr = obj.hits; - else if (Array.isArray(obj.results)) arr = obj.results; - } - if (Array.isArray(arr)) { - for (const item of arr) { - if (item && typeof item === "object") { - const path = (item as Record)[field]; - if (typeof path === "string") availableSources.add(path); - } - } - } - } + // 精确证据注册表：只记录成功返回的 canonical path#anchor。 + // read_page / blocks / 同文件的另一块都不能证明“写作前取回过该证据”。 + const availableEvidence = new Set(); + + /** 只信工具成功返回的实际 path + anchor，不信请求参数。 */ + function recordSource(toolName: string, _args: Record, data?: unknown) { + if (toolName !== "read_block" && toolName !== "read_section") return; + if (!data || typeof data !== "object" || Array.isArray(data)) return; + const obj = data as Record; + if ( + typeof obj.path !== "string" || + typeof obj.anchor !== "string" || + typeof obj.content !== "string" || + !obj.content.trim() + ) return; + const anchor = obj.anchor.replace(/^\^/, ""); + const validAnchor = + toolName === "read_section" + ? /^h-\d+-\d+-[0-9a-f]{6}(?:-\d+)?$/.test(anchor) + : /^[ptcf]-\d+-[0-9a-f]{6}(?:-\d+)?$/.test(anchor); + if (!validAnchor) return; + availableEvidence.add(`${normalizeRefPath(obj.path)}#${anchor}`); } /** ANSWER 后验：扫描所有引用，验证是否有效 */ async function* validateAnswerRefs(text: string): AsyncGenerator { const refs = collectRefs([text]); - if (refs.length === 0) return; - // 先把已读过的标为 valid，未读过的批量调 HTTP 验证 - const toCheck: string[] = []; - const resultMap = new Map(); + const uncited = detectUncitedMaterial(text); + // 路径存在性与“精确证据已读”分开核对。 + const toCheck = new Set(); + const pathExists = new Map(); + const evidencePaths = new Set( + [...availableEvidence].map((key) => key.slice(0, key.lastIndexOf("#"))), + ); for (const r of refs) { - if (availableSources.has(r.ref.path)) { - resultMap.set(r.ref.path, { ok: true, read: true }); - } else { - toCheck.push(r.ref.path); - } + if (evidencePaths.has(r.ref.path)) pathExists.set(r.ref.path, true); + else toCheck.add(r.ref.path); } - if (toCheck.length > 0) { - const exists = await validatePaths(toCheck); + if (toCheck.size > 0) { + const exists = await validatePaths([...toCheck]); for (const [p, ok] of exists) { - resultMap.set(p, { ok, read: false }); + pathExists.set(p, ok); } } - const broken = refs - .filter((r) => { - const res = resultMap.get(r.ref.path); - return !res || !res.ok; - }) - .map((r) => r.ref.path); - const unread = refs - .filter((r) => { - const res = resultMap.get(r.ref.path); - return res && res.ok && !res.read; - }) - .map((r) => r.ref.path); - - // 块级 + 语义核对（基线，与 mode 无关）：对每条块级引用 read_block 验锚点存在性， - // 再用 启发式→LLM判官 两级判断块内容是否支撑论断；不支撑的 key 进 downgraded， - // 由 UI（downgradeRefAnchors）去掉假精度、改成整页链接。 - // 只挑**块级**锚点（p/t/c/f-seq-hash）：按锚点形态判别，兼容模型漏写 ^ 的 [[x#p-3-abc]] - // （isBlock 标志只看 ^，会漏）；排除 h-（heading 指向整节，read_block 只回标题行，不宜按数字降级）。 - const BLOCK_ANCHOR_RE = /^[ptcf]-\d+(?:-\d+)?-[a-z0-9]/; + const broken = [...new Set( + refs.filter((r) => !pathExists.get(r.ref.path)).map((r) => r.ref.path), + )]; + // unread 使用 citation key（path#anchor），避免“读过同文件 A 块”洗白 B 块。 + const unread = [...new Set( + refs + .filter((r) => pathExists.get(r.ref.path) && !availableEvidence.has(r.key)) + .map((r) => r.key), + )]; + + // 严格引用后验：只有 canonical p/t/c/f block 或 h section anchor 能进入语义核对； + // 整页、heading 文本等粗粒度引用直接判未验证。H anchor 用 read_section 取完整节。 + const PRECISE_ANCHOR_RE = /^(?:h-\d+-\d+|[ptcf]-\d+)-[0-9a-f]{6}(?:-\d+)?$/; const blockRefs = refs - .filter((r) => !!r.ref.anchor && BLOCK_ANCHOR_RE.test(r.ref.anchor as string)) + .filter( + (r) => r.ref.isBlock && !!r.ref.anchor && PRECISE_ANCHOR_RE.test(r.ref.anchor), + ) .map((r) => ({ key: r.key, path: r.ref.path, anchor: r.ref.anchor as string })); - let downgraded: string[] = []; + const coarse = refs + .filter( + (r) => !r.ref.isBlock || !r.ref.anchor || !PRECISE_ANCHOR_RE.test(r.ref.anchor), + ) + .map((r) => r.key); + let downgraded: string[] = coarse; if (blockRefs.length > 0) { try { - ({ downgraded } = await verifyBlockCitations(text, blockRefs)); + const verified = await verifyBlockCitations(text, blockRefs); + downgraded = [...new Set([...downgraded, ...verified.downgraded])]; } catch { - downgraded = []; + // 后验自身失败必须 fail-closed：本轮所有精确引用均不可标为已验证。 + downgraded = [...new Set([...downgraded, ...blockRefs.map((r) => r.key)])]; } } - if (broken.length > 0 || unread.length > 0 || downgraded.length > 0) { + if (broken.length > 0 || unread.length > 0 || downgraded.length > 0 || uncited.length > 0) { const segs: string[] = []; if (broken.length) segs.push(`${broken.length} 条文件失效`); if (unread.length) segs.push(`${unread.length} 条未读`); - if (downgraded.length) segs.push(`${downgraded.length} 条锚点不支撑论断→已降级`); + if (downgraded.length) segs.push(`${downgraded.length} 条引用未通过→已去链接化`); + if (uncited.length) segs.push(`${uncited.length} 条实质论断未引用`); yield { kind: "status", text: `引用验证：${segs.join(" / ")}`, level: "warn", } as AgentEvent; - yield { - kind: "ref-validation", - broken, - unread, - downgraded, - } as AgentEvent; } + // 即使全绿也必须发完成事件；UI 在收到前一律按 pending 去链接化。 + yield { + kind: "ref-validation", + broken, + unread, + downgraded, + uncited, + } as AgentEvent; } /** 给一对 trigger call+result 跑 mode augment（emit synthetic 事件） */ @@ -219,6 +276,7 @@ export async function* runAgent( if (evt.kind === "tool-result") { const triggerCall = callsById.get(evt.id); if (triggerCall) { + if (evt.ok) recordSource(triggerCall.name, triggerCall.args, evt.data); for await (const synEvt of runAugment(triggerCall, { ok: evt.ok, data: evt.data, diff --git a/tools/debug-console/lib/citation-judge.ts b/tools/debug-console/lib/citation-judge.ts index 4c7d108..22d68db 100644 --- a/tools/debug-console/lib/citation-judge.ts +++ b/tools/debug-console/lib/citation-judge.ts @@ -1,11 +1,11 @@ /** - * 引用语义判官 —— 两级核对的第 2 级，只用于**灰区**（启发式既非明显匹配也非零重叠）的引用： + * 引用语义判官 —— 两级核对的第 2 级： * 问一次极小的 YES/NO（"这个引用块是否真支撑论断"）。 * * 只用**专用 API 判官**（DeepSeek / OpenAI 兼容），不接 Claude Code CLI： * 实测 `claude -p` 一次性判官对"这一段文字本身是否陈述该事实"把握不可靠（倾向于 - * "出自相关文档就算支撑"地宽容判 YES），会漏掉错配。故灰区判官仅在配了 API key 时启用； - * 没配 → 灰区保守保留（不降级），真正的错配靠启发式的"零重叠→降级"兜住。 + * "出自相关文档就算支撑"地宽容判 YES），会漏掉错配。未配 API key、超时或返回模糊时， + * 严格调用方会把引用标为「无法核验」并去链接化，不会 fail-open。 */ import OpenAI from "openai"; @@ -31,8 +31,8 @@ export function judgeAvailable(): boolean { } /** - * 返回 true=支撑 / false=不支撑 / null=不确定（无 key、失败、回复模糊）。 - * 调用方只对明确的 false 降级，对 null / true 保守不降级。 + * 返回 true=完整支撑 / false=不支撑或仅部分支撑 / null=无法核验。 + * 严格调用方把 false/null 都视为未通过，绝不 fail-open。 */ export async function judgeCitation(claim: string, block: string): Promise { const j = apiClient(); @@ -46,9 +46,9 @@ export async function judgeCitation(claim: string, block: string): Promise **检验你做对了**：用户回看 ANSWER 段，**任何一个具体论断都能立刻点到对应来源**（或看到"这是 Agent 自己说的"的明示标记）——这才是 KB Debug Console 存在的意义。`; diff --git a/tools/debug-console/lib/i18n.ts b/tools/debug-console/lib/i18n.ts index 46993de..db0d9f9 100644 --- a/tools/debug-console/lib/i18n.ts +++ b/tools/debug-console/lib/i18n.ts @@ -112,13 +112,19 @@ export const TRANSLATIONS = { "msg.broken_desc": "这些路径在当前库不存在——模型可能编造了来源，或该问题超出本库范围。正文里它们已被标记为「当前库无此来源」、不可点击；如确属别的库，请切换工作区后重问。", "msg.unverified_refs": "未核实引用 · {n}", - "msg.downgraded_refs": "已降级引用 · {n}", + "msg.downgraded_refs": "未通过引用 · {n}", "msg.downgraded_desc": - "这些块锚点的内容不支撑对应论断，已自动降级为整页链接（去掉假精度）。", + "这些引用不支撑对应论断、无法核验或粒度过粗，正文中已去链接化；不会降级成仍暗示支撑的整页链接。", + "msg.uncited_claims": "未引用实质论断 · {n}", + "msg.uncited_desc": + "ANSWER 中的这些数字或事实性行既没有引用，也没有「需要来源 / Agent 推断」等显式标记；不应视为已验证内容。", "msg.end": "⚠ 结束 · {reason}", // ── 失效引用行内标记（wiki-ref）── "ref.no_source": "当前库无此来源", + "ref.unsupported": "引用未通过", + "ref.not_read": "来源未实际读取", + "ref.pending": "引用验证中", // ── 工具调用卡片（ToolCallCard）── "tool.call_failed": "调用失败", @@ -334,13 +340,19 @@ export const TRANSLATIONS = { "msg.broken_desc": "These paths don't exist in the current knowledge base — the model may have fabricated the source, or the question is outside this base's scope. They've been marked “no such source in the current base” inline and made unclickable; if they belong to another base, switch workspace and ask again.", "msg.unverified_refs": "unverified refs · {n}", - "msg.downgraded_refs": "downgraded refs · {n}", + "msg.downgraded_refs": "rejected refs · {n}", "msg.downgraded_desc": - "The content at these block anchors doesn't support the corresponding claim, so they've been automatically downgraded to whole-page links (removing the false precision).", + "These citations do not support the claim, could not be verified, or are too coarse. They have been neutralized instead of widened into misleading whole-page links.", + "msg.uncited_claims": "uncited material claims · {n}", + "msg.uncited_desc": + "These numeric or factual lines in ANSWER have neither a citation nor an explicit needs-source / agent-inference marker. They must not be treated as verified content.", "msg.end": "⚠ end · {reason}", // ── Broken-ref inline marker (wiki-ref) ── "ref.no_source": "no such source in this base", + "ref.unsupported": "citation not verified", + "ref.not_read": "source not actually read", + "ref.pending": "citation verification pending", // ── Tool call card (ToolCallCard) ── "tool.call_failed": "Call failed", diff --git a/tools/debug-console/lib/kb-http-client.ts b/tools/debug-console/lib/kb-http-client.ts index fe27a43..98eef0b 100644 --- a/tools/debug-console/lib/kb-http-client.ts +++ b/tools/debug-console/lib/kb-http-client.ts @@ -53,18 +53,12 @@ export async function validatePaths( paths: string[], ): Promise> { const results = new Map(); + const uniquePaths = [...new Set(paths)]; await Promise.all( - paths.map(async (p) => { - try { - const res = await fetch(`${kbApiBase()}/api/agent-tool`, { - method: "POST", - headers: buildHeaders(), - body: JSON.stringify({ tool: "read_page", args: { path: p } }), - }); - results.set(p, res.ok); - } catch { - results.set(p, false); - } + uniquePaths.map(async (p) => { + // 复用带 timeout 与响应形状校验的通用客户端；传输失败也按 fail-closed 处理。 + const res = await executeTool("read_page", { path: p }, { timeoutMs: 15_000 }); + results.set(p, res.ok); }), ); return results; diff --git a/tools/debug-console/lib/kb-tools.test.ts b/tools/debug-console/lib/kb-tools.test.ts new file mode 100644 index 0000000..203ec89 --- /dev/null +++ b/tools/debug-console/lib/kb-tools.test.ts @@ -0,0 +1,106 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { + getOpenAITools, + getTool, + KB_TOOLS, + listToolNames, +} from "./kb-tools.ts"; + +const EVIDENCE_TOOL_NAMES = [ + "rebuild-evidence-index", + "evidence-index-coverage", + "search-evidence", + "read-evidence-unit", +] as const; + +function propertySchema( + toolName: string, + propertyName: string, +): Record { + const tool = getTool(toolName); + assert.ok(tool, `missing tool ${toolName}`); + const property = tool.input_schema.properties[propertyName]; + assert.ok(property && typeof property === "object" && !Array.isArray(property)); + return property as Record; +} + +test("KB tool registry has unique names and exposes every evidence tool", () => { + const names = listToolNames(); + assert.equal(new Set(names).size, names.length, "duplicate KB tool name"); + for (const name of EVIDENCE_TOOL_NAMES) { + assert.ok(names.includes(name), `missing ${name}`); + } +}); + +test("evidence index rebuild is an explicit high-cost derived write", () => { + const tool = getTool("rebuild-evidence-index"); + assert.ok(tool); + assert.equal(tool.access, "derived_write"); + assert.equal(tool.cost, "high"); + assert.equal(tool.requires_confirmation, true); + assert.equal(tool.input_schema.additionalProperties, false); + assert.deepEqual(tool.input_schema.required, ["confirm"]); + assert.equal( + propertySchema("rebuild-evidence-index", "confirm").const, + "rebuild-evidence-index", + ); + assert.match(tool.description, /\.cache/); + assert.match(tool.description, /不改 raw\/wiki\/Git/); +}); + +test("evidence search schema bounds every user-controlled CLI value", () => { + const tool = getTool("search-evidence"); + assert.ok(tool); + assert.equal(tool.access, "read"); + assert.equal(tool.cost, "medium"); + assert.equal(tool.input_schema.additionalProperties, false); + assert.deepEqual(tool.input_schema.required, ["query"]); + + assert.equal(propertySchema("search-evidence", "query").minLength, 1); + assert.equal(propertySchema("search-evidence", "query").maxLength, 200); + assert.deepEqual(propertySchema("search-evidence", "limit"), { + type: "integer", + description: "返回候选数，默认 20", + minimum: 1, + maximum: 100, + }); + + const expansions = propertySchema("search-evidence", "expansions"); + assert.equal(expansions.maxItems, 16); + assert.deepEqual(expansions.items, { + type: "string", + minLength: 1, + maxLength: 200, + }); +}); + +test("read-evidence-unit accepts canonical ids and enforces the 30K ceiling", () => { + const tool = getTool("read-evidence-unit"); + assert.ok(tool); + assert.equal(tool.input_schema.additionalProperties, false); + assert.deepEqual(tool.input_schema.required, ["unit_id"]); + assert.equal( + propertySchema("read-evidence-unit", "unit_id").pattern, + "^[0-9a-f]{32}$", + ); + assert.deepEqual(propertySchema("read-evidence-unit", "max_chars"), { + type: "integer", + minimum: 1, + maximum: 30000, + description: "安全读取上限，默认且最大 30000；HTTP 不提供无限制逃生口", + }); + assert.match(tool.description, /unit_id\/evidence_handle/); + assert.match(tool.description, /canonical_ref/); +}); + +test("OpenAI function export preserves exact evidence tool contracts", () => { + const openAITools = getOpenAITools(); + assert.equal(openAITools.length, KB_TOOLS.length); + for (const name of EVIDENCE_TOOL_NAMES) { + const exported = openAITools.find((tool) => tool.function.name === name); + assert.ok(exported, `missing OpenAI function ${name}`); + assert.equal(exported.function.parameters, getTool(name)?.input_schema); + } +}); diff --git a/tools/debug-console/lib/kb-tools.ts b/tools/debug-console/lib/kb-tools.ts index 95e2402..6e14742 100644 --- a/tools/debug-console/lib/kb-tools.ts +++ b/tools/debug-console/lib/kb-tools.ts @@ -13,11 +13,17 @@ export interface KbTool { name: string; description: string; + /** 对 KB 真相源的访问类型；derived_write 仅允许改可重建的 .cache。 */ + access?: "read" | "derived_write"; + /** 供 agent/UI 判断是否应先向用户确认或预留更长超时。 */ + cost?: "low" | "medium" | "high"; + requires_confirmation?: boolean; /** JSON Schema (draft-07 子集) 描述参数 */ input_schema: { type: "object"; properties: Record; required?: string[]; + additionalProperties?: boolean; }; } @@ -43,6 +49,96 @@ export const KB_TOOLS: KbTool[] = [ required: ["query"], }, }, + { + name: "rebuild-evidence-index", + description: + "高成本写操作：从当前 workspace 的 raw/**/*.md 原子重建自然证据单元索引。只改可删除重建的 .cache 派生层，不改 raw/wiki/Git。仅在索引缺失、过期或 raw 已变更时调用，必须显式传入固定确认令牌。", + access: "derived_write", + cost: "high", + requires_confirmation: true, + input_schema: { + type: "object", + properties: { + confirm: { + type: "string", + const: "rebuild-evidence-index", + description: + "显式确认高成本的 .cache 派生索引重建；必须精确填 rebuild-evidence-index。", + }, + }, + required: ["confirm"], + additionalProperties: false, + }, + }, + { + name: "evidence-index-coverage", + description: + "读取当前 workspace 的证据索引覆盖率与 raw 内容新鲜度。长文档细节查询前建议先调用；缺失或 stale 时再显式重建。", + access: "read", + cost: "low", + input_schema: { + type: "object", + properties: {}, + additionalProperties: false, + }, + }, + { + name: "search-evidence", + description: + "在 raw 长文档的自然证据单元（表格行、列表项、段落等）中做精细检索。query 传原问题或一个证据分面；expansions 只传明确别名/译名/改写，系统不隐式调模型。建议先用 evidence-index-coverage 确认 complete，缺失或过期时先重建。", + access: "read", + cost: "medium", + input_schema: { + type: "object", + properties: { + query: { + type: "string", + description: "原问题或单一证据分面（最多 200 字符）", + minLength: 1, + maxLength: 200, + }, + limit: { + type: "integer", + description: "返回候选数，默认 20", + minimum: 1, + maximum: 100, + }, + expansions: { + type: "array", + description: "显式的别名、译名或证据分面扩展（最多 16 项）", + maxItems: 16, + items: { type: "string", minLength: 1, maxLength: 200 }, + }, + }, + required: ["query"], + additionalProperties: false, + }, + }, + { + name: "read-evidence-unit", + description: + "按 search-evidence 返回的 unit_id 精确读取一个自然表格行/列表项/段落。选择与核验用 unit_id/evidence_handle；canonical_ref 是父 block 引用，同一表格或列表的多个单元可共用它。HTTP 客户端不能绕过 30000 字符安全上限。", + access: "read", + cost: "low", + input_schema: { + type: "object", + properties: { + unit_id: { + type: "string", + pattern: "^[0-9a-f]{32}$", + description: "search-evidence 返回的 32 位小写十六进制 unit_id", + }, + max_chars: { + type: "integer", + minimum: 1, + maximum: 30000, + description: "安全读取上限，默认且最大 30000；HTTP 不提供无限制逃生口", + }, + }, + required: ["unit_id"], + additionalProperties: false, + }, + }, { name: "read_page", description: diff --git a/tools/debug-console/lib/providers/types.ts b/tools/debug-console/lib/providers/types.ts index 06bc2f7..acf5df7 100644 --- a/tools/debug-console/lib/providers/types.ts +++ b/tools/debug-console/lib/providers/types.ts @@ -103,10 +103,12 @@ export type AgentEvent = kind: "ref-validation"; /** 不存在的路径（AI 幻觉） */ broken: string[]; - /** 存在但本轮未读取的路径 */ + /** 存在但写作前未取回的精确 citation key（path#anchor） */ unread: string[]; - /** 块级引用 key（path#anchor）——锚点不存在或内容不支撑论断，已降级为整页链接 */ + /** 引用 key（path#anchor）——不存在、不支撑、不可核验或粒度过粗，UI 会去链接化 */ downgraded?: string[]; + /** ANSWER 中无引用且无显式免责标记的数字/事实性行 */ + uncited?: string[]; }; export interface Provider { diff --git a/tools/debug-console/lib/verify-citations.ts b/tools/debug-console/lib/verify-citations.ts index 144c9b9..bd75bb4 100644 --- a/tools/debug-console/lib/verify-citations.ts +++ b/tools/debug-console/lib/verify-citations.ts @@ -3,23 +3,27 @@ * * 对最终答案里每条**块级**引用 `[[path#^p-N-hash]]`： * 1. read_block 真读该锚点。失败要**区分**： - * - 「锚点/文件确实不存在」（error 明确来自 KB 业务层）→ broken → 降级； - * - 「网络/超时/反代故障/DNS」→ **不降级**（unverified，宁可漏也不误伤真实引用）。 - * 2. 语义匹配 —— 只用**硬要素**（日期/数字/百分比/量级，语言中立、可数值归一）做可靠判定： + * - 「锚点/文件确实不存在」（error 明确来自 KB 业务层）→ broken； + * - 「网络/超时/反代故障/DNS」→ unverified。 + * 两类都不允许引用以「已验证」状态出现，调用方会去链接化。 + * 2. 确定性匹配 —— 用**硬要素**（日期/数字/百分比/量级，语言中立、可数值归一）先做闸门： * · 把论断与块都 token 化成 {strong: 具体数字/日期/百分比, years: 裸年份}，用**集合相等**比对 * （而非子串 includes —— 否则 '2024' 会命中 '120240'）；各硬要素**消费即挖空**，不泄漏裸尾数 * （否则 '660万' 泄漏的 '660' 会与 '660亿' 巧合命中 → 万倍量级差异漏降级）； - * · 论断含 strong 要素：块命中任一 strong → 支撑；一个都不命中 → 数字错配（**只命中年份不算支撑**， - * 否则电商语料里年份无处不在会把头条数字错配整体放过）； - * · 论断只有年份 / 无硬要素（纯定性）→ 灰区，启发式不擅自降级，交给判官（有 API key 时）。 - * 唯一的**启发式降级**：某引用存在「有 strong 要素、块全未命中」的出现处，且**没有任何出现处被块 strong 支撑**。 - * 3. 灰区 → 仅当配了可靠 API 判官时并行判一次（YES/NO，带超时）；判 NO → 降级。 + * · 每个出现处的全部 strong 要素都必须在证据里命中；任一数字缺失即拒绝； + * · 只命中年份不算支撑，避免常见年份把数字错配整体放过。 + * 3. 语义判官对每个出现处继续核主体/条件/方向；无判官、超时、模糊、超预算都记未验证并去链接化。 * - * "降级" = 调用方（UI downgradeRefAnchors）去掉块锚点 → 整页链接（去假精度，仍指向正确源页）。 + * 未通过 / 无法核验 = 调用方去链接化并明确标记，绝不扩大为整页引用洗白。 */ import { executeTool } from "./kb-http-client"; import { judgeCitation, judgeAvailable } from "./citation-judge"; -import { WIKI_REF_RE, normalizeBareSlugRefs, parseWikiRef } from "./wiki-ref"; +import { + WIKI_REF_RE, + normalizeBareSlugRefs, + normalizeRefPath, + parseWikiRef, +} from "./wiki-ref"; export interface BlockRefSpec { key: string; // collectRefs 口径 key：path#anchor（anchor 不含 ^） @@ -29,22 +33,20 @@ export interface BlockRefSpec { export interface CitationVerifyResult { brokenAnchors: string[]; // 锚点/文件确实不存在 - downgraded: string[]; // 需降级 - unverified: string[]; // 网络/超时未能核对 —— 不降级 + downgraded: string[]; // 未通过严格后验，需在 UI 去链接化（字段名为兼容保留） + unverified: string[]; // 网络/超时/无判官等导致无法核验（也会去链接化） judged: number; } -const MAX_JUDGE = 12; -// 灰区判官超时：判官是 temperature=0 / max_tokens=4 的 YES/NO，正常亚秒级返回； -// 3s 已是极宽裕的上限。超时 fallback = null = **保守保留该引用**（不降级）， -// 故收紧超时**零质量风险**——只是给「判官卡死」兜个底，避免单条 hung call 把 -// 答案出完后的「引用后验」尾巴拖到 8s（该尾巴计入 stream 关闭、即输入框重新可用的时间）。 +const MAX_JUDGE = 24; +// 判官是 temperature=0 / max_tokens=4 的 YES/NO，正常亚秒级返回。 +// 超时 fallback = null = 无法核验；严格调用方会去链接化，同时避免尾部后验无限阻塞 UI。 const JUDGE_TIMEOUT_MS = 3000; /** - * read_block 失败时：是不是「确定性不存在」（vs 网络/超时/反代/DNS）。 + * read_block / read_section 失败时：是不是「确定性不存在」（vs 网络/超时/反代/DNS）。 * 只命中明确来自 KB 业务层的串；**不**匹配 ENOTFOUND/ECONNREFUSED/http_404/timeout 等传输层故障 - * （那些归 unverified、不降级，避免主站抖动误降级真实引用）。 + * （那些归 unverified，不误报为 broken；但仍然不能作为已验证引用）。 */ function errorIsNotFound(err: string): boolean { return /未找到\s*anchor|未找到\s*文件|文件不存在|no such file|page_not_found|errno\s*2(?!\d)/i.test(err); @@ -56,13 +58,32 @@ function blockContent(data: unknown): string { } return ""; } -function blockKind(data: unknown): string { - if (data && typeof data === "object" && "kind" in data) { - return String((data as { kind: unknown }).kind || ""); + +/** + * 严格证据身份核对。Web 层可能为了预览友好而做 source→raw fallback， + * k.py read-block 也可能按 hash 恢复到另一 canonical anchor。这些都能帮人修正， + * 但不能把原本写错的 path#anchor 判为已验证。 + */ +function exactEvidenceIdentity( + data: unknown, + expectedPath: string, + expectedAnchor: string, +): "exact" | "mismatch" | "unverifiable" { + if (!data || typeof data !== "object" || Array.isArray(data)) return "unverifiable"; + const obj = data as Record; + if ( + Object.prototype.hasOwnProperty.call(obj, "_fallback") || + Object.prototype.hasOwnProperty.call(obj, "recovered_from") + ) { + return "mismatch"; } - return ""; + if (typeof obj.path !== "string" || typeof obj.anchor !== "string") return "unverifiable"; + const actualPath = normalizeRefPath(obj.path); + const actualAnchor = obj.anchor.replace(/^\^/, ""); + return actualPath === normalizeRefPath(expectedPath) && actualAnchor === expectedAnchor + ? "exact" + : "mismatch"; } - /** 归一：全角数字→半角、去千分位逗号、percent→%、小写。 */ function normNum(s: string): string { return s @@ -209,9 +230,10 @@ export async function verifyBlockCitations( const uniq = [...byKey.values()]; const reads = await Promise.all( - uniq.map((r) => - executeTool("read_block", { path: r.path, anchor: `^${r.anchor}` }).then((res) => ({ r, res })), - ), + uniq.map((r) => { + const tool = /^h-/.test(r.anchor) ? "read_section" : "read_block"; + return executeTool(tool, { path: r.path, anchor: `^${r.anchor}` }).then((res) => ({ r, res })); + }), ); const grays: Array<{ key: string; claim: string; content: string }> = []; @@ -222,49 +244,53 @@ export async function verifyBlockCitations( brokenAnchors.push(r.key); downgraded.push(r.key); } else { - unverified.push(r.key); // 网络/超时 → 不降级 + // 网络/超时也不能在严格后验里被当作已验证；保留 unverified 分类并去链接化。 + unverified.push(r.key); + downgraded.push(r.key); } continue; } const content = blockContent(res.data); if (!content.trim()) { unverified.push(r.key); + downgraded.push(r.key); + continue; + } + const identity = exactEvidenceIdentity(res.data, r.path, r.anchor); + if (identity !== "exact") { + if (identity === "unverifiable") unverified.push(r.key); + downgraded.push(r.key); continue; } - if (blockKind(res.data) === "heading") continue; // heading 指整节、只回标题行，豁免 const ctxs = claimContextsFor(answerText, r.key).filter((c) => hasSalient(c)); - if (ctxs.length === 0) continue; + if (ctxs.length === 0) { + unverified.push(r.key); + downgraded.push(r.key); + continue; + } const blockTok = hardNumericTokens(content); - let hardSupported = false; // 某出现处被块的 strong 要素命中 → 整 key 保留 - let numericMismatch = false; // 某出现处有 strong 要素、块全未命中 - let needsJudge = false; // 纯定性 / 仅年份 → 灰区 - let judgeClaim = ""; + let numericMismatch = false; for (const ctx of ctxs) { const ct = hardNumericTokens(ctx); if (ct.strong.size >= 1) { - if (matchHits(ct, blockTok).strongHit >= 1) hardSupported = true; - else numericMismatch = true; - } else { - // 仅年份 或 无硬要素 → 不凭启发式定，交灰区 - needsJudge = true; - if (!judgeClaim) judgeClaim = ctx; + // 必须覆盖该出现处的全部 hard facts；“任一数字命中”会让同句其他错数逃逸。 + if (matchHits(ct, blockTok).strongHit !== ct.strong.size) numericMismatch = true; } } - if (hardSupported) continue; // 任一处被 strong 支撑 → 保守保留 if (numericMismatch) { - // 有确凿数字错配，且无任何 strong 支撑 → 降级（vague/年份处不救活它） downgraded.push(r.key); continue; } - if (needsJudge) grays.push({ key: r.key, claim: judgeClaim, content }); + // 数字全匹配仍可能主体/条件互换；每个出现处都交语义判官，任何一处不通过即拒绝。 + for (const claim of ctxs) grays.push({ key: r.key, claim, content }); } let judged = 0; - if (useJudge && grays.length > 0) { - const slice = grays.slice(0, MAX_JUDGE); + if (grays.length > 0) { + const slice = useJudge ? grays.slice(0, MAX_JUDGE) : []; judged = slice.length; const verdicts = await Promise.all( slice.map((g) => @@ -275,9 +301,23 @@ export async function verifyBlockCitations( ), ); verdicts.forEach((v, i) => { - if (v === false) downgraded.push(slice[i].key); + if (v !== true) { + unverified.push(slice[i].key); + downgraded.push(slice[i].key); + } }); + // 没有可靠判官、或超过受控调用预算的引用，一律保持“未验证”并去链接化。 + const unchecked = useJudge ? grays.slice(MAX_JUDGE) : grays; + for (const g of unchecked) { + unverified.push(g.key); + downgraded.push(g.key); + } } - return { brokenAnchors, downgraded, unverified, judged }; + return { + brokenAnchors: [...new Set(brokenAnchors)], + downgraded: [...new Set(downgraded)], + unverified: [...new Set(unverified)], + judged, + }; } diff --git a/tools/debug-console/lib/wiki-ref.ts b/tools/debug-console/lib/wiki-ref.ts index f7a0a91..a9ec9b1 100644 --- a/tools/debug-console/lib/wiki-ref.ts +++ b/tools/debug-console/lib/wiki-ref.ts @@ -201,24 +201,70 @@ export function collectRefs(texts: string[]): NumberedRef[] { } /** - * 把指定 key（`path#anchor`，anchor 不含 ^）的**块级**引用"降级"为整页链接： - * 去掉 `#^anchor` 只留 `[[path|alias]]`。用于答案后验判定某引用锚点的内容不支撑论断时， - * 去掉假精度（仍指向正确的源页面，只是不再声称精确到那个错块）。 + * 把未通过严格后验的引用去链接化。 + * + * 旧行为会把「不支撑论断」的块引用降成整页链接；这仍会让读者误以为整页来源 + * 支撑该论断，只是精度较低。现在保留可读标签，但移除 wikilink 语法并明确标为 + * 「引用未通过」——错误证据不能靠扩大引用范围洗白。 + * + * 函数名保留以兼容现有 UI import；语义已从 downgrade 改为 neutralize。 */ -export function downgradeRefAnchors(text: string, downgradedKeys: Set): string { +export function downgradeRefAnchors( + text: string, + downgradedKeys: Set, + locale: Locale = DEFAULT_LOCALE, +): string { if (downgradedKeys.size === 0) return text; const norm = normalizeBareSlugRefs(text); const re = new RegExp(WIKI_REF_RE.source, "g"); return norm.replace(re, (full, target: string, anchor?: string, alias?: string) => { const ref = parseWikiRef(target, anchor, alias); const key = `${ref.path}#${ref.anchor || ""}`; - if (ref.anchor && downgradedKeys.has(key)) { - return alias ? `[[${target}|${alias}]]` : `[[${target}]]`; + if (downgradedKeys.has(key)) { + const display = (alias || refDisplayText(ref)).replace(/[[\]]/g, ""); + return `${display}（${t("ref.unsupported", locale)}）`; } return full; }); } +/** + * 存在但写作前未实际取回的精确证据同样不可作为已验证引用。 + * unread 是 citation key（path#anchor），不是 path；读过同文件的另一块不算。 + */ +export function neutralizeUnreadRefs( + text: string, + unreadKeys: Set, + locale: Locale = DEFAULT_LOCALE, +): string { + if (unreadKeys.size === 0) return text; + const norm = normalizeBareSlugRefs(text); + const re = new RegExp(WIKI_REF_RE.source, "g"); + return norm.replace(re, (full, target: string, anchor?: string, alias?: string) => { + const ref = parseWikiRef(target, anchor, alias); + const key = `${ref.path}#${ref.anchor || ""}`; + if (!unreadKeys.has(key)) return full; + const display = (alias || refDisplayText(ref)).replace(/[[\]]/g, ""); + return `${display}（${t("ref.not_read", locale)}）`; + }); +} + +/** 后验完成前一律去链接化，避免流式输出的短暂 fail-open 窗口。 */ +export function neutralizePendingRefs( + text: string, + pending: boolean, + locale: Locale = DEFAULT_LOCALE, +): string { + if (!pending) return text; + const norm = normalizeBareSlugRefs(text); + const re = new RegExp(WIKI_REF_RE.source, "g"); + return norm.replace(re, (_full, target: string, anchor?: string, alias?: string) => { + const ref = parseWikiRef(target, anchor, alias); + const display = (alias || refDisplayText(ref)).replace(/[[\]]/g, ""); + return `${display}（${t("ref.pending", locale)}）`; + }); +} + /** * 把「在当前库根本不存在」的引用（后验 broken 列表）**去链接化**：替换成带标记的纯文本， * 不再渲染为可点链接，也不进编号 / references 列表。 @@ -269,6 +315,10 @@ export function inlineWikiRefsNumbered( /** 根据 ref 决定调哪个 KB 工具 + 工具参数 */ export function refToToolCall(ref: WikiRef): { tool: string; args: Record } { + // ^h 是整节证据，不能像 p/t/c/f 一样只读 heading block。 + if (ref.anchor?.startsWith("h-")) { + return { tool: "read_section", args: { path: ref.path, anchor: ref.anchor } }; + } if (ref.anchor && ref.isBlock) { return { tool: "read_block", args: { path: ref.path, anchor: ref.anchor } }; } diff --git a/tools/debug-console/package.json b/tools/debug-console/package.json index 712e264..97c99a8 100644 --- a/tools/debug-console/package.json +++ b/tools/debug-console/package.json @@ -7,7 +7,8 @@ "dev": "next dev -p 3100", "build": "next build", "start": "next start -p 3100", - "lint": "next lint" + "lint": "next lint", + "test": "node --disable-warning=MODULE_TYPELESS_PACKAGE_JSON --test 'lib/**/*.test.ts'" }, "dependencies": { "@types/dagre": "^0.7.54", diff --git a/tools/debug-console/tsconfig.json b/tools/debug-console/tsconfig.json index 0312134..0200e5c 100644 --- a/tools/debug-console/tsconfig.json +++ b/tools/debug-console/tsconfig.json @@ -11,6 +11,7 @@ "moduleResolution": "bundler", "resolveJsonModule": true, "isolatedModules": true, + "allowImportingTsExtensions": true, "jsx": "preserve", "incremental": true, "plugins": [{ "name": "next" }], diff --git a/web/README.md b/web/README.md index 52c411e..c95eee7 100644 --- a/web/README.md +++ b/web/README.md @@ -55,7 +55,7 @@ npm run dev # 端口 3006 ### 外部 agent 入口 -- `POST /api/agent-tool` — **外部客户端（如 `tools/debug-console`）调用 KB 原子操作的唯一入口**：白名单工具名（search / outline / read-section / backlinks 等只读操作），带 CSRF 防护。它是主管理台向外暴露的 REST 工具接口，`tools/debug-console` 通过它访问 KB 而不直接读 markdown。 +- `POST /api/agent-tool` — **外部客户端（如 `tools/debug-console`）调用 KB 原子操作的唯一入口**：白名单默认只读（search / outline / read_section / backlinks 等），并提供 `evidence-index-coverage` / `search-evidence` / `read-evidence-unit` 三个长文档证据索引读工具。唯一状态变更例外 `rebuild-evidence-index` 只原子重建可删除的 `.cache` 派生索引，不改 markdown/Git；它被标记为高成本写操作，除同源/CSRF 防护外还必须携带 `args.confirm: "rebuild-evidence-index"`。路由不接受任意 workspace 参数；它仅使用经校验的 `kb_workspace` cookie / `KB_WORKSPACE` 解析当前 workspace。 ## 写权限保护 diff --git a/web/app/api/agent-tool/route.ts b/web/app/api/agent-tool/route.ts index 2726a51..82e4655 100644 --- a/web/app/api/agent-tool/route.ts +++ b/web/app/api/agent-tool/route.ts @@ -2,19 +2,21 @@ * POST /api/agent-tool * * 通用 KB 工具调度端点，给外部 LLM agent（如 tools/debug-console/）用。 - * 白名单只读工具；写工具一律拒绝（即便前端被恶意改造也无法越权）。 + * 白名单默认只读。唯一状态变更例外是 rebuild-evidence-index：它只原子重建 + * .cache 派生索引（不改 markdown/Git），但属于高成本操作，必须携带固定确认令牌。 * * 请求体：{ tool: string, args: object } - * 响应：{ ok: true, data: any } | { ok: false, error: string } + * 响应：{ ok: true, data: any } | { ok: false, error: string, error_code?: string } * * 安全约束： * - 工具名白名单（见 TOOL_HANDLERS） * - 所有 path 参数走 isSafeRelPath 校验，限制在 wiki/ raw/ 下 - * - 透传 runKCli 现有的 30s 超时 + 10MB stdout 上限 + * - 透传 runKCli 的 stdout 上限；证据索引查询 60s、显式重建 5min 超时 * - 不内嵌 LLM SDK；本路由是纯 HTTP 桥接 */ import { NextRequest, NextResponse } from "next/server"; import { runKCli } from "@/lib/k-cli"; +import { sanitizeEvidencePayload } from "@/lib/evidence-safety"; import { isSafeRelPath } from "@/lib/kb"; import { getPage } from "@/lib/kb-service"; import { isSameOrigin } from "@/lib/permissions"; @@ -24,9 +26,19 @@ export const dynamic = "force-dynamic"; const MAX_QUERY_LEN = 200; const MAX_SNIPPET_LEN = 500; const MAX_LIMIT = 100; +const MAX_EVIDENCE_EXPANSIONS = 16; +const MAX_EVIDENCE_READ_CHARS = 30_000; +const EVIDENCE_REBUILD_CONFIRMATION = "rebuild-evidence-index"; +const EVIDENCE_UNIT_ID_RE = /^[0-9a-f]{32}$/; type ToolHandler = (args: Record) => Promise< - { ok: true; data: unknown } | { ok: false; error: string; status?: number } + | { ok: true; data: unknown } + | { + ok: false; + error: string; + status?: number; + error_code?: string; + } >; function bad(error: string, status = 400) { @@ -48,6 +60,47 @@ function getInt(args: Record, key: string): number | undefined return undefined; } +function rejectUnknownArgs( + args: Record, + allowed: readonly string[], +): string | undefined { + const allow = new Set(allowed); + const unexpected = Object.keys(args).filter((key) => !allow.has(key)); + return unexpected.length > 0 + ? `unexpected_args: ${unexpected.sort().join(",")}` + : undefined; +} + +function getStringArray( + args: Record, + key: string, +): { ok: true; value: string[] } | { ok: false; error: string } { + const raw = args[key]; + if (raw === undefined) return { ok: true, value: [] }; + if (!Array.isArray(raw)) return { ok: false, error: `${key}_must_be_array` }; + if (raw.length > MAX_EVIDENCE_EXPANSIONS) { + return { ok: false, error: `${key}_too_many` }; + } + const value: string[] = []; + const seen = new Set(); + for (const item of raw) { + if (typeof item !== "string" || !item.trim()) { + return { ok: false, error: `${key}_contains_invalid_string` }; + } + if (item.length > MAX_QUERY_LEN) { + return { ok: false, error: `${key}_item_too_long` }; + } + if (item.includes("\0") || item.trimStart().startsWith("-")) { + return { ok: false, error: `${key}_contains_invalid_cli_value` }; + } + if (!seen.has(item)) { + seen.add(item); + value.push(item); + } + } + return { ok: true, value }; +} + function requirePath(args: Record): string | { error: string } { const p = getStr(args, "path"); if (!p) return { error: "missing args.path" }; @@ -66,6 +119,54 @@ async function callKCli(cliArgs: string[]) { return { ok: true as const, data: result.data }; } +function evidenceErrorStatus(code: string): number { + if (code === "unit-not-found") return 404; + if (code === "index-not-found" || code === "index-stale") return 409; + if (code === "evidence-unit-too-large") return 413; + if (code.startsWith("invalid-")) return 400; + return 500; +} + +/** + * evidence CLI 会在协议错误时 exit=2，但 stdout 仍是结构化 + * `{ok:false,error:{code,message,details}}`。只透传 code/message（details 可能含 + * 本机绝对 db_path），既保留可机读错误类别，也不泄露磁盘结构。 + * coverage/rebuild 的 exit=1 是完整的业务报告（例如 stale/incomplete），仍返回 data。 + */ +async function callEvidenceKCli(cliArgs: string[], timeoutMs = 60_000) { + const result = await runKCli>(cliArgs, { + timeoutMs, + parseStdoutOnNonZero: true, + }); + if (!result.ok) { + return { + ok: false as const, + error: result.error || "evidence_kcli_failed", + status: 500, + }; + } + const payload = result.data; + if (payload && payload.ok === false) { + const rawError = payload.error; + if (rawError && typeof rawError === "object" && !Array.isArray(rawError)) { + const errorObject = rawError as Record; + const code = + typeof errorObject.code === "string" ? errorObject.code : "evidence-index-error"; + const message = + typeof errorObject.message === "string" + ? errorObject.message.slice(0, 300) + : "Evidence index operation failed"; + return { + ok: false as const, + error: `${code}: ${message}`, + error_code: code, + status: evidenceErrorStatus(code), + }; + } + } + return { ok: true as const, data: sanitizeEvidencePayload(payload) }; +} + /** * 锚点查找 fallback：AI 常把一条**真实的 raw 块锚点**（如 [[raw/papers/2024-10-lightrag#^p-54]]） * 误标到它正在读的摘要页路径上（写成 [[wiki/sources/lightrag#^p-54]]）——锚点是对的、路径错了。 @@ -153,6 +254,73 @@ const TOOL_HANDLERS: Record = { return callKCli(["search", q, "--limit", String(limit)]); }, + async "rebuild-evidence-index"(args) { + const unexpected = rejectUnknownArgs(args, ["confirm"]); + if (unexpected) return bad(unexpected); + if (getStr(args, "confirm") !== EVIDENCE_REBUILD_CONFIRMATION) { + return bad( + `confirmation_required: set args.confirm to ${EVIDENCE_REBUILD_CONFIRMATION}`, + 409, + ); + } + return callEvidenceKCli(["rebuild-evidence-index"], 300_000); + }, + + async "evidence-index-coverage"(args) { + const unexpected = rejectUnknownArgs(args, []); + if (unexpected) return bad(unexpected); + return callEvidenceKCli(["evidence-index-coverage"]); + }, + + async "search-evidence"(args) { + const unexpected = rejectUnknownArgs(args, ["query", "limit", "expansions"]); + if (unexpected) return bad(unexpected); + + const query = getStr(args, "query"); + if (!query?.trim()) return bad("missing args.query"); + if (query.length > MAX_QUERY_LEN) return bad("query_too_long"); + if (query.includes("\0") || query.trimStart().startsWith("-")) { + return bad("invalid_query"); + } + + let limit = getInt(args, "limit") ?? 20; + if (limit < 1) limit = 1; + if (limit > MAX_LIMIT) limit = MAX_LIMIT; + + const expansions = getStringArray(args, "expansions"); + if (!expansions.ok) return bad(expansions.error); + + const cliArgs = ["search-evidence", query, "--limit", String(limit)]; + for (const expansion of expansions.value) { + cliArgs.push("--expand", expansion); + } + return callEvidenceKCli(cliArgs); + }, + + async "read-evidence-unit"(args) { + const unexpected = rejectUnknownArgs(args, ["unit_id", "max_chars"]); + if (unexpected) return bad(unexpected); + const unitId = getStr(args, "unit_id"); + if (!unitId) return bad("missing args.unit_id"); + if (!EVIDENCE_UNIT_ID_RE.test(unitId)) return bad("invalid_unit_id"); + const rawMaxChars = args.max_chars; + const maxChars = rawMaxChars === undefined ? MAX_EVIDENCE_READ_CHARS : rawMaxChars; + if ( + typeof maxChars !== "number" || + !Number.isInteger(maxChars) || + maxChars < 1 || + maxChars > MAX_EVIDENCE_READ_CHARS + ) { + return bad("invalid_max_chars"); + } + return callEvidenceKCli([ + "read-evidence-unit", + unitId, + "--max-chars", + String(maxChars), + ]); + }, + async outline(args) { const p = requirePath(args); if (typeof p !== "string") return bad(p.error); @@ -322,8 +490,15 @@ export async function POST(req: NextRequest) { const result = await handler(safeArgs); if (!result.ok) { const status = "status" in result && result.status ? result.status : 400; + const payload: { ok: false; error: string; error_code?: string } = { + ok: false, + error: result.error, + }; + if ("error_code" in result && result.error_code) { + payload.error_code = result.error_code; + } return NextResponse.json( - { ok: false, error: result.error }, + payload, { status }, ); } diff --git a/web/app/api/citations/route.ts b/web/app/api/citations/route.ts new file mode 100644 index 0000000..a79b231 --- /dev/null +++ b/web/app/api/citations/route.ts @@ -0,0 +1,62 @@ +/** + * GET /api/citations + * 透传 `k.py list-suspect-citations --json`：wiki 页面中「> [!CAUTION] + * 引用审计未通过」标注块清单（待人裁决的错引）。 + * 返回 [{ path, title, line, cited, block }]。 + * + * POST /api/citations + * Body: { path: string, line: number, action: "false_positive" | "resolved" } + * 人类裁决入口（见 lib/operations.ts 的 applyCitationVerdict）： + * - false_positive：审计误报，维持引用——CAUTION 块改写为一行 NOTE + * - resolved：论断已修复，移除标注收尾 + */ +import { NextRequest, NextResponse } from "next/server"; +import { runKCli } from "@/lib/k-cli"; +import { applyCitationVerdict, type CitationVerdictAction } from "@/lib/operations"; +import { isSameOrigin } from "@/lib/permissions"; + +export const dynamic = "force-dynamic"; + +export async function GET() { + const result = await runKCli(["list-suspect-citations"]); + if (!result.ok) { + return NextResponse.json({ error: result.error }, { status: 500 }); + } + return NextResponse.json(result.data); +} + +const VALID_ACTIONS: CitationVerdictAction[] = ["false_positive", "resolved"]; + +export async function POST(req: NextRequest) { + // CSRF 兜底：拒绝跨站发起的写请求（本地工具语境，详见 lib/permissions.isSameOrigin） + if (!isSameOrigin(req)) { + return NextResponse.json({ error: "csrf_blocked" }, { status: 403 }); + } + + let body: { path?: string; line?: number; action?: string }; + try { + body = await req.json(); + } catch { + return NextResponse.json({ error: "invalid_json" }, { status: 400 }); + } + + const { path: relPath, line, action } = body; + if (!relPath) { + return NextResponse.json({ error: "missing_path" }, { status: 400 }); + } + if (typeof line !== "number" || !Number.isInteger(line) || line < 1) { + return NextResponse.json({ error: "invalid_line" }, { status: 400 }); + } + if (!action || !VALID_ACTIONS.includes(action as CitationVerdictAction)) { + return NextResponse.json( + { error: "invalid_action", valid: VALID_ACTIONS }, + { status: 400 }, + ); + } + + const result = await applyCitationVerdict(relPath, line, action as CitationVerdictAction); + if (!result.ok) { + return NextResponse.json(result, { status: 500 }); + } + return NextResponse.json(result); +} diff --git a/web/app/health/citations/page.tsx b/web/app/health/citations/page.tsx new file mode 100644 index 0000000..9e93e30 --- /dev/null +++ b/web/app/health/citations/page.tsx @@ -0,0 +1,85 @@ +import Link from "next/link"; +import { runKCli } from "@/lib/k-cli"; +import { Card } from "@/components/ui/card"; +import { CitationVerdictButtons } from "@/components/CitationVerdictButtons"; +import { getServerLocale } from "@/lib/server-locale"; +import { t } from "@/lib/i18n"; + +interface SuspectCitation { + path: string; + title: string; + line: number; + cited: string[]; + block: string; +} + +export const dynamic = "force-dynamic"; + +export default async function CitationsPage() { + const locale = getServerLocale(); + const result = await runKCli(["list-suspect-citations"]); + + return ( +
+
+
+

{t("health.citations.title", locale)}

+

+ {t("health.citations.desc", locale)} +

+
+ + {t("health.return_dashboard", locale)} + +
+ + {!result.ok && ( +
+ {t("common.error_kcli", locale, { err: result.error || "" })} +
+ )} + + {result.ok && result.data && result.data.length === 0 && ( + + {t("health.citations.empty", locale)} + + )} + + {result.ok && result.data && result.data.length > 0 && ( +
+

+ {t("health.citations.count", locale, { n: result.data.length })} +

+ {result.data.map((s, i) => ( + +
+ + {s.title} + + + {s.path}:{s.line} + +
+ 
+                {s.block.replace(/^\n+/, "")}
+
+ {s.cited.length > 0 && ( +
+ + {t("health.citations.cited_label", locale)} + + {s.cited.map((c, j) => ( + + {c} + + ))} +
+ )} + + + ))} +
+ )} +
+ ); +} diff --git a/web/components/CitationVerdictButtons.tsx b/web/components/CitationVerdictButtons.tsx new file mode 100644 index 0000000..bc1de53 --- /dev/null +++ b/web/components/CitationVerdictButtons.tsx @@ -0,0 +1,134 @@ +"use client"; +import { useState } from "react"; +import { useRouter } from "next/navigation"; +import Link from "next/link"; +import { Button, buttonVariants } from "@/components/ui/button"; +import { cn } from "@/lib/utils"; +import { useT } from "@/lib/i18n-client"; +import type { TranslationKey } from "@/lib/i18n"; + +/** + * 引用审计裁决按钮（仿 ResolveButtons 的 confirm + POST 模式）。 + * 与 ResolveButtons 的差异：POST /api/citations，且请求体带 line——裁决是逐条的 + * （同一页可能有多个 CAUTION 块，靠行号定位；后端校验行号防串行）。 + */ + +type CitationVerdictAction = "false_positive" | "resolved"; + +interface VerdictDef { + labelKey: TranslationKey; + confirmKey: TranslationKey; + action: CitationVerdictAction; + variant: "default" | "outline"; +} + +const VERDICTS: VerdictDef[] = [ + { + labelKey: "action.citation_false_positive", + confirmKey: "action.citation_false_positive_confirm", + action: "false_positive", + variant: "default", + }, + { + labelKey: "action.citation_resolved", + confirmKey: "action.citation_resolved_confirm", + action: "resolved", + variant: "outline", + }, +]; + +interface CitationVerdictButtonsProps { + path: string; + line: number; +} + +export function CitationVerdictButtons({ path, line }: CitationVerdictButtonsProps) { + const t = useT(); + const router = useRouter(); + const [busy, setBusy] = useState(false); + const [msg, setMsg] = useState<{ kind: "ok" | "err" | "info"; text: string } | null>(null); + + async function handle(verdict: VerdictDef) { + if (busy) return; + if (!confirm(t(verdict.confirmKey))) return; + + setBusy(true); + setMsg({ kind: "info", text: t("common.processing") }); + try { + const res = await fetch("/api/citations", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ path, line, action: verdict.action }), + }); + const data = await res.json(); + if (res.ok) { + const text = data.commit + ? t("action.commit_msg", { commit: data.commit }) + : t("action.commit_noop"); + setMsg({ kind: "ok", text }); + setTimeout(() => router.refresh(), 600); + } else { + // 已知 error enum 映射成 i18n key——避免中英文 mix + const errorKeyMap: Record = { + invalid_json: "action.error.invalid_json", + missing_path: "action.error.missing_path", + invalid_action: "action.error.invalid_action", + invalid_path: "action.error.invalid_path", + permission_denied: "action.error.permission_denied", + commit_failed: "action.error.commit_failed", + write_failed: "action.error.write_failed", + read_failed: "action.error.read_failed", + }; + const knownKey = errorKeyMap[data.error]; + const text = knownKey ? t(knownKey) : t("action.error.generic"); + setMsg({ kind: "err", text }); + } + } catch (e) { + setMsg({ kind: "err", text: t("common.network_error", { err: String(e) }) }); + } finally { + setBusy(false); + } + } + + return ( +
+ {VERDICTS.map((v) => ( + + ))} + {/* 直接给 Link 套 buttonVariants className，避免 。 */} + + {t("action.go_edit")} + + {msg && ( + + {msg.text} + + )} +
+ ); +} diff --git a/web/components/HealthDashboard.tsx b/web/components/HealthDashboard.tsx index 7b0c536..8890112 100644 --- a/web/components/HealthDashboard.tsx +++ b/web/components/HealthDashboard.tsx @@ -15,6 +15,16 @@ interface HealthData { low_confidence_count: number; stale_drafts_count: number; broken_refs_count: number; + cite_mismatches_count: number; + suspect_citations_count: number; + cite_imprecise_count: number; + unaudited_citations_count: number; + evidence_index?: { + ok: boolean; + coverage_status: string; + natural_units?: { coverage_pct?: number | null }; + corpus_freshness?: { ok?: boolean }; + }; last_check: string; } @@ -34,6 +44,20 @@ export async function HealthDashboard() { } const data = result.data!; + const evidenceStatus = data.evidence_index?.coverage_status ?? "missing"; + const evidenceStatusKeys: Record = { + complete: "health.card.evidence_index.complete", + missing: "health.card.evidence_index.missing", + "stale-corpus": "health.card.evidence_index.stale", + "empty-corpus": "health.card.evidence_index.empty", + incomplete: "health.card.evidence_index.incomplete", + error: "health.card.evidence_index.error", + unknown: "health.card.evidence_index.unknown", + }; + const evidenceStatusLabel = t( + evidenceStatusKeys[evidenceStatus] ?? "health.card.evidence_index.unknown", + locale, + ); const cards = [ { @@ -78,6 +102,18 @@ export async function HealthDashboard() { tone: (data.broken_refs_count ?? 0) > 0 ? "danger" : "ok", href: (data.broken_refs_count ?? 0) > 0 ? "/health/broken-refs" : undefined, }, + { + label: t("health.card.evidence_index", locale), + value: data.evidence_index?.ok + ? `${data.evidence_index.natural_units?.coverage_pct ?? 100}% · ${evidenceStatusLabel}` + : evidenceStatusLabel, + tone: data.evidence_index?.ok + ? "ok" + : evidenceStatus === "missing" + ? "warning" + : "danger", + href: undefined, + }, ] as const; return ( @@ -98,6 +134,7 @@ export async function HealthDashboard() { {cards.map((c) => ( ))} +
@@ -131,7 +168,7 @@ function HealthCard({ href, }: { label: string; - value: number; + value: number | string; tone: "ok" | "warning" | "danger" | "neutral"; href?: string; }) { @@ -164,6 +201,96 @@ function HealthCard({ return href ? {inner} : inner; } +/** + * 引用审计指标卡（kb-cite-audit 体系）——一张卡上四个指标，语义分层： + * - cite_mismatches（闸门项，>0 危险）：引用数字与被引块不符 + * - suspect_citations（待人裁决，>0 链到 /health/citations 工作台） + * - cite_imprecise（观察项）：锚点挂偏 / 引文未逐字命中 + * - unaudited（信息项）：尚未审计的论断数，依赖本地缓存台账（删缓存后回升属预期） + */ +function CitationAuditCard({ data, locale }: { data: HealthData; locale: Locale }) { + const mismatches = data.cite_mismatches_count ?? 0; + const suspects = data.suspect_citations_count ?? 0; + const imprecise = data.cite_imprecise_count ?? 0; + const unaudited = data.unaudited_citations_count ?? 0; + + // 卡片整体色调沿用 HealthCard 惯例：闸门项亮红 > 待裁决亮黄 > 全绿 + const toneClass = + mismatches > 0 + ? "border-destructive/40 bg-destructive/5" + : suspects > 0 + ? "border-amber-500/30 bg-amber-500/5" + : "border-emerald-500/30 bg-emerald-500/5"; + + const rows: { + key: TranslationKey; + value: number; + valueClass: string; + href?: string; + noteKey?: TranslationKey; + }[] = [ + { + key: "health.card.citations.mismatches", + value: mismatches, + valueClass: mismatches > 0 ? "text-destructive" : "text-emerald-700 dark:text-emerald-300", + }, + { + key: "health.card.citations.suspects", + value: suspects, + valueClass: + suspects > 0 ? "text-amber-700 dark:text-amber-300" : "text-emerald-700 dark:text-emerald-300", + href: suspects > 0 ? "/health/citations" : undefined, + }, + { + key: "health.card.citations.imprecise", + value: imprecise, + valueClass: imprecise > 0 ? "text-amber-700 dark:text-amber-300" : "", + }, + { + key: "health.card.citations.unaudited", + value: unaudited, + valueClass: "", + noteKey: "health.card.citations.unaudited_note", + }, + ]; + + return ( + + +
{t("health.card.citations", locale)}
+
    + {rows.map((r) => { + const inner = ( + <> + + {t(r.key, locale)} + {r.noteKey && ( + + {t(r.noteKey, locale)} + + )} + + {r.value} + + ); + return ( +
  • + {r.href ? ( + + {inner} + + ) : ( + {inner} + )} +
    • + ); + })} +
+ + + ); +} + /** 把分布卡片的 key 翻译为对应字段的展示文字（type/status/confidence）。 * 如果 i18n 表里没有对应翻译就回退为原 key——保证未来加新字段值不会显示 "undefined"。 */ diff --git a/web/lib/citation-rewrite.test.ts b/web/lib/citation-rewrite.test.ts new file mode 100644 index 0000000..ab67dda --- /dev/null +++ b/web/lib/citation-rewrite.test.ts @@ -0,0 +1,170 @@ +/** + * citation-rewrite 的回归守护——零依赖，用 Node 内置 node:test 跑（type stripping， + * 直接执行 .ts，无需 jest/vitest）。 + * + * 跑法（Node ≥ 22.6）： + * cd web && npm test + * 或：node --test --experimental-strip-types lib/citation-rewrite.test.ts + * + * 这两个函数按行号对多行 blockquote 做手术，是最容易静默改坏 wiki 正文的逻辑。 + * 下面覆盖：正常改写/删除、行号错位抛错（防串行）、块含空行的多段 callout、 + * 文件首尾边界。日期参数注入固定值断言确定输出。 + */ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { + rewriteCautionAsFalsePositive, + removeCautionBlock, + CITATION_CAUTION_START_RE, +} from "./citation-rewrite.ts"; + +const DAY = "2026-07-02"; + +// 标准单块样本：CAUTION 块在第 5 行（1-based） +const oneCaution = [ + "# 页面标题", + "", + "该方法准确率 95.3%。[[raw/papers/X#^t-77-9e8336]]", + "", + "> [!CAUTION] 引用审计未通过 — 2026-06-30", + "> **论断**：该方法准确率 95.3%（块 ^p-4-34d5b1）", + "> **被引块**：[[raw/papers/X#^t-77-9e8336]]", + "> **审计判定**：UNSUPPORTED — 被引表格里没有 95.3% 这个数", + "> **建议**：改引 ^t-78 或修正数字", + "> **状态**：⏳ 待人类判别", + "", + "结尾一段。", + "", +].join("\n"); +const ONE_LINE = 5; + +// 两个 CAUTION 块：第 3 行与第 9 行 +const twoCautions = [ + "# 标题", + "", + "> [!CAUTION] 引用审计未通过 — 第一处", + "> **状态**：⏳ 待人类判别", + "", + "中间正文。", + "", + "", + "> [!CAUTION] 引用审计未通过 — 第二处", + "> **状态**：⏳ 待人类判别", + "", + "末尾。", + "", +].join("\n"); + +// 多段 callout：块内含空行、空行后仍有 > 行（k.py 同口径视为同一块） +const multiParagraphCaution = [ + "正文。", + "", + "> [!CAUTION] 引用审计未通过 — 2026-06-30", + "> **论断**：…", + ">", + "> **审计判定**：UNSUPPORTED", + "", + "> **建议**：（空行后的 > 行仍属同一块）", + "", + "块后正文。", + "", +].join("\n"); +const MULTI_LINE = 3; + +test("rewriteCautionAsFalsePositive 把块替换为一行 NOTE，前后正文保留", () => { + const out = rewriteCautionAsFalsePositive(oneCaution, ONE_LINE, DAY); + assert.ok(out.includes(`> [!NOTE] 引用审计误报（${DAY} 由人类复核通过）`), "NOTE 行写入 + 日期"); + assert.ok(!out.includes("[!CAUTION]"), "CAUTION 块应被整体替换"); + assert.ok(!out.includes("待人类判别"), "块内行应一并替换"); + assert.ok(out.includes("该方法准确率 95.3%。[[raw/papers/X#^t-77-9e8336]]"), "论断原文（引用）保持不变"); + assert.ok(out.includes("结尾一段。"), "块后正文保留"); +}); + +test("rewriteCautionAsFalsePositive 只改行号指向的块，其他块原样保留", () => { + const out = rewriteCautionAsFalsePositive(twoCautions, 9, DAY); + assert.equal((out.match(/\[!CAUTION\]/g) || []).length, 1, "应只剩第一个 CAUTION 块"); + assert.ok(out.includes("第一处"), "第一个块未被动"); + assert.ok(!out.includes("第二处"), "第二个块已被替换"); + assert.ok(out.includes("引用审计误报"), "NOTE 写入"); +}); + +test("removeCautionBlock 删除整块并折叠空行，前后正文保留", () => { + const out = removeCautionBlock(oneCaution, ONE_LINE); + assert.ok(!out.includes("[!CAUTION]"), "块应被删除"); + assert.ok(!out.includes("审计判定"), "块内行一并删除"); + assert.ok(out.includes("该方法准确率 95.3%。"), "论断保留（修复已发生在论断上）"); + assert.ok(out.includes("结尾一段。"), "块后正文保留"); + assert.ok(!/\n{3,}/.test(out), "不应残留 3+ 连续空行"); +}); + +test("行号错位（普通正文行）→ 两个函数都抛错，不猜块", () => { + assert.throws(() => rewriteCautionAsFalsePositive(oneCaution, 3, DAY), /不是「\[!CAUTION\] 引用审计未通过」/); + assert.throws(() => removeCautionBlock(oneCaution, 3), /不是「\[!CAUTION\] 引用审计未通过」/); +}); + +test("行号指向块的第二行（> 开头但非起始行）→ 抛错", () => { + assert.throws(() => rewriteCautionAsFalsePositive(oneCaution, ONE_LINE + 1, DAY)); + assert.throws(() => removeCautionBlock(oneCaution, ONE_LINE + 1)); +}); + +test("行号指向其他类型 callout（如 WARNING 冲突块）→ 抛错", () => { + const withConflict = ["> [!WARNING] 知识更新冲突 — 2026-01-01", "> **状态**：⏳ 待人类判别", ""].join("\n"); + assert.throws(() => rewriteCautionAsFalsePositive(withConflict, 1, DAY)); + assert.throws(() => removeCautionBlock(withConflict, 1)); +}); + +test("行号越界（0 / 超出文档 / 非整数）→ 抛错", () => { + assert.throws(() => removeCautionBlock(oneCaution, 0), /行号越界/); + assert.throws(() => removeCautionBlock(oneCaution, 999), /行号越界/); + assert.throws(() => rewriteCautionAsFalsePositive(oneCaution, 5.5, DAY), /行号越界/); +}); + +test("多段 callout（块内空行 + 空行后的 > 行）整体视为一块", () => { + const rewritten = rewriteCautionAsFalsePositive(multiParagraphCaution, MULTI_LINE, DAY); + assert.ok(!rewritten.includes("[!CAUTION]"), "起始段被替换"); + assert.ok(!rewritten.includes("空行后的 > 行仍属同一块"), "空行后的 > 续段也属块、一并替换"); + assert.ok(rewritten.includes("块后正文。"), "块后第一个真内容行开始的正文保留"); + + const removed = removeCautionBlock(multiParagraphCaution, MULTI_LINE); + assert.ok(!removed.includes("审计判定") && !removed.includes("> **建议**"), "整块删除"); + assert.ok(removed.includes("正文。") && removed.includes("块后正文。"), "前后正文保留"); + assert.ok(!/\n{3,}/.test(removed), "不残留 3+ 连续空行"); +}); + +test("文件首边界：块在第 1 行", () => { + const atStart = [ + "> [!CAUTION] 引用审计未通过 — 2026-06-30", + "> **状态**：⏳ 待人类判别", + "", + "正文从这里开始。", + "", + ].join("\n"); + const rewritten = rewriteCautionAsFalsePositive(atStart, 1, DAY); + assert.ok(rewritten.startsWith("> [!NOTE] 引用审计误报"), "NOTE 行成为首行"); + assert.ok(rewritten.includes("正文从这里开始。"), "后文保留"); + + const removed = removeCautionBlock(atStart, 1); + assert.ok(removed.startsWith("正文从这里开始。"), "删除后不残留文件头空行"); +}); + +test("文件尾边界：块在末尾（无尾随换行）", () => { + const atEnd = ["正文。", "", "> [!CAUTION] 引用审计未通过 — 2026-06-30", "> **状态**：⏳ 待人类判别"].join("\n"); + const rewritten = rewriteCautionAsFalsePositive(atEnd, 3, DAY); + assert.ok(rewritten.endsWith(`> [!NOTE] 引用审计误报（${DAY} 由人类复核通过）`), "NOTE 行成为末行"); + + const removed = removeCautionBlock(atEnd, 3); + assert.equal(removed, "正文。\n", "删除后尾部收敛为单个换行"); +}); + +test("文件尾边界：块在末尾（有尾随换行）", () => { + const atEndNl = ["正文。", "", "> [!CAUTION] 引用审计未通过 — 2026-06-30", "> **状态**：⏳ 待人类判别", ""].join("\n"); + const removed = removeCautionBlock(atEndNl, 3); + assert.equal(removed, "正文。\n", "尾部空行不堆积"); +}); + +test("CITATION_CAUTION_START_RE 与 k.py 的 SUSPECT_START_RE 同口径", () => { + assert.ok(CITATION_CAUTION_START_RE.test("> [!CAUTION] 引用审计未通过 — 2026-06-30")); + assert.ok(CITATION_CAUTION_START_RE.test(">[!CAUTION]引用审计未通过"), "允许无空格（\\s*）"); + assert.ok(!CITATION_CAUTION_START_RE.test("> [!WARNING] 知识更新冲突"), "不匹配冲突块"); + assert.ok(!CITATION_CAUTION_START_RE.test("> [!CAUTION] 别的警告"), "不匹配其他 CAUTION"); +}); diff --git a/web/lib/citation-rewrite.ts b/web/lib/citation-rewrite.ts new file mode 100644 index 0000000..857baa1 --- /dev/null +++ b/web/lib/citation-rewrite.ts @@ -0,0 +1,84 @@ +/** + * 引用审计 CAUTION 标注块改写——纯字符串 in/out，无 fs/git/next 依赖。 + * + * 与 conflict-rewrite.ts 同模式：kb-cite-audit 流程会在「引用审计未通过」的论断块 + * 下方落 `> [!CAUTION] 引用审计未通过 — YYYY-MM-DD` 标注块，人类在 web 工作台 + * 裁决后由这两个函数改写 markdown。多行 blockquote 手术是最容易静默改坏 wiki + * 正文的一类逻辑，独立成模块后可用 node:test 零依赖回归测试 + * （见 citation-rewrite.test.ts）。 + * + * 与 conflict-rewrite 的关键差异：一页可能有多个 CAUTION 块，裁决是**逐条**的 + * （k.py list-suspect-citations 给出每条的起始行号），所以这里按行号定位块， + * 并校验行号处确实是 CAUTION 起始行——行号错位（页面在裁决前被编辑过）时抛错， + * 绝不"就近猜一个块"改错行。 + * + * 日期通过参数注入（默认 todayISO()）——让测试能传固定日期断言确定输出。 + */ +import { todayISO } from "./conflict-rewrite.ts"; + +/** + * CAUTION 审计标注块起始行。 + * 与 scripts/k.py 的 SUSPECT_START_RE（`^>\s*\[!CAUTION\]\s*引用审计未通过`）对齐。 + */ +export const CITATION_CAUTION_START_RE = /^>\s*\[!CAUTION\]\s*引用审计未通过/; + +/** + * 定位从 line（1-based）起始的 CAUTION 块边界。 + * + * 块的范围与 k.py list_suspect_citations 同口径：起始行之后，连续的 `>` 开头行 + * 或空行都属于块（允许多段 callout 中间夹空行），直到第一个"真内容"行；再回退掉 + * 尾部空行——它们是块与后文的分隔，不属于块本身。 + * + * @throws line 越界或该行不是「[!CAUTION] 引用审计未通过」起始行时抛错（防串行）。 + */ +function locateCautionBlock( + content: string, + line: number, +): { lines: string[]; start: number; end: number } { + const lines = content.split("\n"); + const start = line - 1; + if (!Number.isInteger(line) || start < 0 || start >= lines.length) { + throw new Error(`行号越界：line=${line}（文档共 ${lines.length} 行）`); + } + if (!CITATION_CAUTION_START_RE.test(lines[start])) { + throw new Error( + `第 ${line} 行不是「[!CAUTION] 引用审计未通过」标注起始行（页面可能已被编辑，请刷新后重试）`, + ); + } + let end = start + 1; + while (end < lines.length && (lines[end].startsWith(">") || lines[end].trim() === "")) { + end++; + } + while (end > start + 1 && lines[end - 1].trim() === "") { + end--; + } + return { lines, start, end }; +} + +/** + * false_positive：人类复核后判定审计为误报，引用维持不变。 + * 把 CAUTION 块整体替换为一行 NOTE——删除即标记原则，误报裁决留在真相源， + * 后续读者（与重跑的审计）能看到"此处已被人复核通过"。 + */ +export function rewriteCautionAsFalsePositive( + content: string, + line: number, + date: string = todayISO(), +): string { + const { lines, start, end } = locateCautionBlock(content, line); + const note = `> [!NOTE] 引用审计误报（${date} 由人类复核通过）`; + return [...lines.slice(0, start), note, ...lines.slice(end)].join("\n"); +} + +/** + * resolved：论断本身已修复（修复发生在论断上，不在这里），移除 CAUTION 标注收尾。 + * 块整体删除并折叠多余空行；文件首部残留的空行一并去掉，尾部多余空行压成一个换行。 + */ +export function removeCautionBlock(content: string, line: number): string { + const { lines, start, end } = locateCautionBlock(content, line); + return [...lines.slice(0, start), ...lines.slice(end)] + .join("\n") + .replace(/\n{3,}/g, "\n\n") + .replace(/^\n+/, "") + .replace(/\n{2,}$/, "\n"); +} diff --git a/web/lib/evidence-safety.test.ts b/web/lib/evidence-safety.test.ts new file mode 100644 index 0000000..406b979 --- /dev/null +++ b/web/lib/evidence-safety.test.ts @@ -0,0 +1,31 @@ +import assert from "node:assert/strict"; +import test from "node:test"; + +import { sanitizeEvidencePayload } from "./evidence-safety.ts"; + +test("sanitizeEvidencePayload removes nested machine paths but keeps relative evidence paths", () => { + const sanitized = sanitizeEvidencePayload({ + db_path: "/Users/example/kb/.cache/index.db", + corpus_freshness: { + workspace_root: "/Users/example/kb", + errors: [ + { code: "read-failed", path: "/Users/example/kb/raw/a.md" }, + { code: "windows", path: "C:\\Users\\example\\raw\\b.md" }, + { code: "relative", path: "raw/papers/public.md" }, + ], + }, + hits: [{ path: "raw/papers/public.md", canonical_ref: "raw/papers/public.md#^p-1-a" }], + }) as Record; + + assert.equal("db_path" in sanitized, false); + const freshness = sanitized.corpus_freshness as Record; + assert.equal("workspace_root" in freshness, false); + assert.deepEqual(freshness.errors, [ + { code: "read-failed" }, + { code: "windows" }, + { code: "relative", path: "raw/papers/public.md" }, + ]); + assert.deepEqual(sanitized.hits, [ + { path: "raw/papers/public.md", canonical_ref: "raw/papers/public.md#^p-1-a" }, + ]); +}); diff --git a/web/lib/evidence-safety.ts b/web/lib/evidence-safety.ts new file mode 100644 index 0000000..020a90f --- /dev/null +++ b/web/lib/evidence-safety.ts @@ -0,0 +1,33 @@ +const SENSITIVE_PATH_KEYS = new Set([ + "db_path", + "workspace_root", + "raw_root", + "resolved_path", +]); + +function isAbsoluteLocalPath(value: string): boolean { + return ( + value.startsWith("/") || + value.startsWith("file://") || + /^[A-Za-z]:[\\/]/.test(value) || + value.startsWith("\\\\") + ); +} + +/** Remove machine-local paths while preserving workspace-relative evidence paths. */ +export function sanitizeEvidencePayload(value: unknown): unknown { + if (Array.isArray(value)) return value.map(sanitizeEvidencePayload); + if (!value || typeof value !== "object") return value; + + return Object.fromEntries( + Object.entries(value as Record) + .filter(([key, child]) => { + if (SENSITIVE_PATH_KEYS.has(key)) return false; + if (key === "path" && typeof child === "string") { + return !isAbsoluteLocalPath(child); + } + return true; + }) + .map(([key, child]) => [key, sanitizeEvidencePayload(child)]), + ); +} diff --git a/web/lib/i18n.ts b/web/lib/i18n.ts index f8cbf05..76bd309 100644 --- a/web/lib/i18n.ts +++ b/web/lib/i18n.ts @@ -190,6 +190,20 @@ export const TRANSLATIONS = { "health.card.low_confidence": "低置信度", "health.card.stale_drafts": "陈旧草稿（>30 天未改）", "health.card.broken_refs": "失效引用", + "health.card.evidence_index": "长文档证据索引", + "health.card.evidence_index.complete": "完整且新鲜", + "health.card.evidence_index.missing": "未建立", + "health.card.evidence_index.stale": "原文或摘要已变更", + "health.card.evidence_index.empty": "语料为空", + "health.card.evidence_index.incomplete": "索引不完整", + "health.card.evidence_index.error": "索引校验失败", + "health.card.evidence_index.unknown": "状态未知", + "health.card.citations": "引用审计", + "health.card.citations.mismatches": "数字与被引块不符（闸门）", + "health.card.citations.suspects": "审计未通过待裁决", + "health.card.citations.imprecise": "锚点挂偏（观察）", + "health.card.citations.unaudited": "未审计论断", + "health.card.citations.unaudited_note": "（依赖本地缓存台账）", "health.dist.by_type": "按类型", "health.dist.by_status": "按状态", "health.dist.by_confidence": "按置信度", @@ -226,6 +240,12 @@ export const TRANSLATIONS = { "health.broken_refs.reason.file_missing": "raw 文件不存在", "health.broken_refs.reason.anchor_missing": "anchor 不存在", + "health.citations.title": "引用审计未通过", + "health.citations.desc": "带 > [!CAUTION] 引用审计未通过 块的页面——审计判定被引原文不支撑该论断，待人裁决。误报可一键维持引用；论断已修复后可一键移除标注。重跑审计用 Claude Code /kb-cite-audit。", + "health.citations.empty": "✅ 没有待处理的引用审计标注。", + "health.citations.count": "共 {n} 处待裁决：", + "health.citations.cited_label": "被引锚点：", + "panel.outline": "章节大纲", "outline.empty": "（文档无 heading，无章节树）", "outline.tag.preview": "预览", @@ -276,6 +296,10 @@ export const TRANSLATIONS = { "action.adopt_new_prompt": "请输入新论断的完整段落（将取代冲突主文本，旧观点会保留为历史注释）：", "action.merge": "合并多视角", "action.merge_prompt": "请输入整合后的完整段落（应包含两个视角的对比与你的整合判断；将替换冲突块）：", + "action.citation_false_positive": "误报，维持引用", + "action.citation_false_positive_confirm": "确认判为误报？CAUTION 标注将改写为一行「引用审计误报（由人类复核通过）」备注，引用与论断保持不变。", + "action.citation_resolved": "已修复，移除标注", + "action.citation_resolved_confirm": "确认移除该 CAUTION 标注？（前提：论断本身已修复——删标注只是收尾）", "action.go_edit": "去编辑页", "action.commit_msg": "已 commit {commit}", "action.commit_noop": "已保存（无变更）", @@ -647,6 +671,20 @@ export const TRANSLATIONS = { "health.card.low_confidence": "Low confidence", "health.card.stale_drafts": "Stale draft (>30d unmodified)", "health.card.broken_refs": "Broken refs", + "health.card.evidence_index": "Long-document evidence index", + "health.card.evidence_index.complete": "Complete and fresh", + "health.card.evidence_index.missing": "Not built", + "health.card.evidence_index.stale": "Source or summary changed", + "health.card.evidence_index.empty": "Empty corpus", + "health.card.evidence_index.incomplete": "Index incomplete", + "health.card.evidence_index.error": "Index validation failed", + "health.card.evidence_index.unknown": "Unknown status", + "health.card.citations": "Citation audit", + "health.card.citations.mismatches": "Number vs cited block mismatch (gate)", + "health.card.citations.suspects": "Failed audit, awaiting verdict", + "health.card.citations.imprecise": "Imprecise anchors (watch)", + "health.card.citations.unaudited": "Unaudited claims", + "health.card.citations.unaudited_note": "(relies on local cache ledger)", "health.dist.by_type": "By type", "health.dist.by_status": "By status", "health.dist.by_confidence": "By confidence", @@ -683,6 +721,12 @@ export const TRANSLATIONS = { "health.broken_refs.reason.file_missing": "raw file missing", "health.broken_refs.reason.anchor_missing": "anchor not found", + "health.citations.title": "Failed citation audits", + "health.citations.desc": "Pages with > [!CAUTION] citation-audit blocks — the audit judged that the cited source text does not support the claim; awaiting human verdict. One-click keep the citation if it's a false positive, or remove the marker once the claim itself has been fixed. To re-run the audit, use Claude Code /kb-cite-audit.", + "health.citations.empty": "✅ No pending citation-audit markers.", + "health.citations.count": "{n} awaiting verdict:", + "health.citations.cited_label": "Cited anchors:", + "panel.outline": "Outline", "outline.empty": "(no headings, no outline)", "outline.tag.preview": "preview", @@ -733,6 +777,10 @@ export const TRANSLATIONS = { "action.adopt_new_prompt": "Enter the new claim as a full paragraph (it will replace the conflict's main text; the old view is kept as a historical note):", "action.merge": "Merge perspectives", "action.merge_prompt": "Enter the integrated paragraph (should include both views and your integrated judgment; will replace the conflict block):", + "action.citation_false_positive": "False positive — keep citation", + "action.citation_false_positive_confirm": "Confirm false positive? The CAUTION marker will be rewritten as a one-line \"citation audit false positive (verified by human)\" note; the citation and claim stay unchanged.", + "action.citation_resolved": "Fixed — remove marker", + "action.citation_resolved_confirm": "Remove this CAUTION marker? (Precondition: the claim itself has already been fixed — removing the marker is just the wrap-up.)", "action.go_edit": "Open editor", "action.commit_msg": "Committed {commit}", "action.commit_noop": "Saved (no change)", diff --git a/web/lib/k-cli.ts b/web/lib/k-cli.ts index 6b1244f..99c8ea4 100644 --- a/web/lib/k-cli.ts +++ b/web/lib/k-cli.ts @@ -4,9 +4,11 @@ */ import { spawn } from "node:child_process"; import path from "node:path"; -import { projectRoot, resolveWorkspace } from "./kb"; +import { engineRoot, resolveWorkspace } from "./kb"; -const PY = process.env.KB_PY || "python"; +// 默认 python3：现代 macOS / 多数发行版只装 python3，不带 python（py2 早已移除）； +// k.py/convert.py 都是 Python 3 脚本。需要指定解释器时用 KB_PY 环境变量覆盖。 +const PY = process.env.KB_PY || "python3"; /** 子进程 stdout 上限：超过 → kill 进程并返回错误。 * k.py 输出全是 JSON 结构化数据，10MB 已经远超任何合理 outline / blocks 列表； @@ -46,7 +48,9 @@ export async function runKCli( ): Promise> { const timeoutMs = options.timeoutMs ?? 30000; const parseStdoutOnNonZero = options.parseStdoutOnNonZero ?? false; - const root = projectRoot(); + // 引擎脚本恒在 engineRoot（web 父目录）；KB_ROOT 只搬数据、不搬 scripts/。 + // KB_ROOT 通过下方 spawn 的 env 透传给 k.py 子进程，由 k.py 自己解析数据根。 + const root = engineRoot(); const scriptPath = path.join(root, "scripts", "k.py"); // 与 web 显示层一致：cookie kb_workspace > KB_WORKSPACE env > 默认（resolveWorkspace 解析） const workspace = resolveWorkspace(); diff --git a/web/lib/kb-service.ts b/web/lib/kb-service.ts index 9b333fa..763d983 100644 --- a/web/lib/kb-service.ts +++ b/web/lib/kb-service.ts @@ -4,8 +4,9 @@ * 所有 page / API / component 对**wiki 内容**的读取（页面元数据、完整页面、 * 反链、出链、检查存在性）必须经过本模块。 * - * 当前实现：fs 扫描 + markdown 解析（O(N) 全库扫，适合 < 千文档） - * 未来实现：SQLite (.cache/index.db) 增量索引 + FTS5 全文，调用方 0 改动 + * wiki 当前实现：fs 扫描 + markdown 解析（O(N) 全库扫，适合 < 千文档）。 + * raw 长文档细节检索已由 CLI 的 `.cache/retrieval_index.db` 承担；本服务的 + * wiki 读取边界未来可切换为 `.cache/index.db` 增量索引，调用方无需改动。 * * 例外（允许直接用 lib/kb.ts 原语的场景）： * - raw/ 资产文件读写（不是 markdown 页面） diff --git a/web/lib/kb-symlink-safety.test.ts b/web/lib/kb-symlink-safety.test.ts new file mode 100644 index 0000000..581a936 --- /dev/null +++ b/web/lib/kb-symlink-safety.test.ts @@ -0,0 +1,51 @@ +import assert from "node:assert/strict"; +import fs from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import test from "node:test"; + +import { assertNoSymlinkComponents } from "./path-safety.ts"; + +test("path guard rejects symlinks crossing permission zones or workspace", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "groundmap-kb-symlink-")); + const outside = await fs.mkdtemp(path.join(os.tmpdir(), "groundmap-kb-outside-")); + try { + const workspace = path.join(root, "workspaces", "test"); + await fs.mkdir(path.join(workspace, "wiki"), { recursive: true }); + await fs.mkdir(path.join(workspace, "my_thoughts"), { recursive: true }); + await fs.writeFile( + path.join(workspace, "my_thoughts", "secret.md"), + "private", + "utf8", + ); + + await fs.symlink( + path.join("..", "my_thoughts", "secret.md"), + path.join(workspace, "wiki", "alias.md"), + ); + assert.throws( + () => assertNoSymlinkComponents(path.join(workspace, "wiki", "alias.md"), workspace), + /不允许包含符号链接/, + ); + + await fs.symlink(outside, path.join(workspace, "wiki", "linked-dir")); + assert.throws( + () => + assertNoSymlinkComponents( + path.join(workspace, "wiki", "linked-dir", "created.md"), + workspace, + ), + /不允许包含符号链接/, + ); + await assert.rejects(fs.access(path.join(outside, "created.md"))); + assert.doesNotThrow(() => + assertNoSymlinkComponents( + path.join(workspace, "wiki", "ordinary", "new.md"), + workspace, + ), + ); + } finally { + await fs.rm(root, { recursive: true, force: true }); + await fs.rm(outside, { recursive: true, force: true }); + } +}); diff --git a/web/lib/kb.ts b/web/lib/kb.ts index a4d8b83..b65d8a7 100644 --- a/web/lib/kb.ts +++ b/web/lib/kb.ts @@ -15,15 +15,25 @@ import path from "node:path"; import fs from "node:fs"; import fsp from "node:fs/promises"; import { cookies } from "next/headers"; +import { assertNoSymlinkComponents } from "./path-safety"; /** - * 引擎根目录：web/ 的父目录（scripts/、CLAUDE.md 所在目录）。 + * 引擎根目录：web/ 的父目录（scripts/k.py、CLAUDE.md、wiki/_templates 所在目录）。 * Next.js 启动时 cwd 通常就是 web/，所以 ../ 即可。 - * 也支持 KB_ROOT 环境变量覆盖（便于测试）。 + * **不受 KB_ROOT 影响**——KB_ROOT 只搬「数据」（workspaces/），引擎代码始终在此。 + */ +export function engineRoot(): string { + return path.resolve(process.cwd(), ".."); +} + +/** + * 数据根目录：workspaces/ 所在目录。默认 = 引擎根（数据与代码同库，向后兼容）； + * 设 KB_ROOT 环境变量则指向外置项目的数据目录（含 workspaces/），与 k.py / convert.py + * 的 KB_ROOT 同义。注意：这是「数据根」，解析引擎脚本路径请用 engineRoot()。 */ export function projectRoot(): string { if (process.env.KB_ROOT) return process.env.KB_ROOT; - return path.resolve(process.cwd(), ".."); + return engineRoot(); } export const WORKSPACE_COOKIE = "kb_workspace"; @@ -149,6 +159,7 @@ export function toRelPosix(absPath: string): string { export async function listMarkdownFiles(relDir: string): Promise { const absDir = safeResolve(relDir); if (!fs.existsSync(absDir)) return []; + assertNoSymlinkComponents(absDir, workspaceRoot()); const out: string[] = []; @@ -170,40 +181,10 @@ export async function listMarkdownFiles(relDir: string): Promise { return out.sort(); } -/** Symlink 防护：解析后的真实路径必须仍在 PROJECT_ROOT 内。 - * 否则 `wiki/foo.md → /etc/passwd` 这种 symlink 会让 safeResolve 检查通过、 - * realpath 把读取重定向到项目外。 - * - * 仅当文件存在时检查（lstat 失败说明是新文件，写入路径自然不可能是 symlink）。 - * 失败时抛错——与 safeResolve 风格一致。 - */ -function assertNotSymlinkEscape(abs: string): void { - try { - // realpathSync 跟随所有 symlink；不存在时抛 ENOENT，让 caller 决定 - const real = fs.realpathSync(abs); - const root = fs.realpathSync(workspaceRoot()); - const rel = path.relative(root, real); - if (rel === "" || rel.startsWith("..") || path.isAbsolute(rel)) { - throw new Error(`路径越界（symlink 指向 PROJECT_ROOT 外）：${abs}`); - } - } catch (e: unknown) { - if ( - e && - typeof e === "object" && - "code" in e && - (e as { code: string }).code === "ENOENT" - ) { - // 路径不存在（新文件场景）→ 不可能是 symlink，放行 - return; - } - throw e; - } -} - /** 读单个文件（UTF-8）。把 CRLF 规范化为 LF，避免 SSR/hydration 不一致 */ export async function readFile(relPath: string): Promise { const abs = safeResolve(relPath); - assertNotSymlinkEscape(abs); + assertNoSymlinkComponents(abs, workspaceRoot()); const raw = await fsp.readFile(abs, "utf8"); return raw.replace(/\r\n/g, "\n"); } @@ -211,9 +192,10 @@ export async function readFile(relPath: string): Promise { /** 写单个文件（UTF-8）。调用者负责权限校验。 */ export async function writeFile(relPath: string, content: string): Promise { const abs = safeResolve(relPath); - // 写入前若文件已存在且是 symlink → 拒绝（避免覆盖项目外文件） - assertNotSymlinkEscape(abs); + assertNoSymlinkComponents(abs, workspaceRoot()); await fsp.mkdir(path.dirname(abs), { recursive: true }); + // mkdir 后再核验一次，确保原本不存在的父链没有解析成 symlink。 + assertNoSymlinkComponents(abs, workspaceRoot()); await fsp.writeFile(abs, content, "utf8"); } @@ -222,8 +204,7 @@ export async function fileExists(relPath: string): Promise { const abs = safeResolve(relPath); const st = await fsp.stat(abs); if (!st.isFile()) return false; - // 同时挡 symlink 越界，与 readFile/writeFile 行为对称 - assertNotSymlinkEscape(abs); + assertNoSymlinkComponents(abs, workspaceRoot()); return true; } catch { return false; diff --git a/web/lib/operations.ts b/web/lib/operations.ts index c751cde..ca858be 100644 --- a/web/lib/operations.ts +++ b/web/lib/operations.ts @@ -13,6 +13,11 @@ import { adoptNewFromConflict, mergeConflict, } from "./conflict-rewrite"; +import { + rewriteCautionAsFalsePositive, + removeCautionBlock, + CITATION_CAUTION_START_RE, +} from "./citation-rewrite"; export type ResolveAction = | "set_status_deprecated" @@ -137,6 +142,93 @@ export async function applyResolve( }; } +/** 引用审计裁决：误报（维持引用） / 已修复（移除标注） */ +export type CitationVerdictAction = "false_positive" | "resolved"; + +/** + * 引用审计 CAUTION 标注的人类裁决（与 applyResolve 同链路：权限 → 读 → 纯函数 + * 改写 → frontmatter 收尾 → 写回 → 自动 commit）。 + * + * @param relPath 相对项目根的 wiki 页面路径 + * @param line CAUTION 块起始行号——k.py list-suspect-citations 给出的 1-based + * 行号（相对 **strip 后的正文**：python-frontmatter 会去掉 + * frontmatter 与首尾空白）。这里会换算到 gray-matter 的 content + * 行号（gray-matter 保留 frontmatter 后的前导空行）。 + * @param action false_positive：块改写为一行「引用审计误报」NOTE（删除即标记， + * 误报裁决留在真相源）；resolved：论断已修复，删块收尾。 + */ +export async function applyCitationVerdict( + relPath: string, + line: number, + action: CitationVerdictAction, +): Promise { + const perm = await checkWritePermissionAsync(relPath); + if (!perm.allowed) { + return { ok: false, error: perm.reason || "权限拒绝" }; + } + + let raw: string; + try { + raw = await readFile(relPath); + } catch (e) { + return { ok: false, error: `读取失败: ${e}` }; + } + + const { frontmatter, content } = parseMarkdown(raw); + const newFm = { ...frontmatter }; + + // 行号换算：k.py 的行号基于 strip 后正文；gray-matter 的 content 若带前导空行 + // （frontmatter 结束到正文之间的空行），行号需加上这些前导行数。 + const leadingWs = content.slice(0, content.length - content.trimStart().length); + const leadingLines = (leadingWs.match(/\n/g) || []).length; + + let newContent: string; + try { + newContent = + action === "false_positive" + ? rewriteCautionAsFalsePositive(content, line + leadingLines) + : removeCautionBlock(content, line + leadingLines); + } catch (e) { + return { ok: false, error: e instanceof Error ? e.message : String(e) }; + } + + // 页面已无任何「引用审计未通过」CAUTION 块 → 摘掉 frontmatter 的 citation-suspect 标签 + const anyCautionLeft = new RegExp(CITATION_CAUTION_START_RE.source, "m"); + if (!anyCautionLeft.test(newContent)) { + const rawTags = newFm.tags; + const tags = Array.isArray(rawTags) + ? rawTags.map((t) => String(t)) + : typeof rawTags === "string" + ? [rawTags] + : []; + const kept = tags.filter((t) => t !== "citation-suspect" && t !== "#citation-suspect"); + if (kept.length !== tags.length) { + newFm.tags = kept; + } + } + + // 与冲突决议同规：人类裁决 → last_modified 置今天、last_modified_by 置 Human + newFm.last_modified = todayISO(); + newFm.last_modified_by = "Human"; + + const newRaw = serializeMarkdown(newFm, newContent); + try { + await writeFile(relPath, newRaw); + } catch (e) { + return { ok: false, error: `写入失败: ${e}` }; + } + + const commit = await gitAddAndCommit([relPath], `audit: citation ${action} on ${relPath} via web`); + if (!commit.ok) { + return { ok: false, error: `git commit 失败: ${commit.error}` }; + } + return { + ok: true, + message: commit.noop ? "已保存（无变更）" : `已 commit ${commit.commit}`, + commit: commit.commit, + }; +} + /** * 删除带 `#to-be-updated` 的整个尾部块。 * 通常的写法是： diff --git a/web/lib/path-safety.ts b/web/lib/path-safety.ts new file mode 100644 index 0000000..de77cfd --- /dev/null +++ b/web/lib/path-safety.ts @@ -0,0 +1,37 @@ +import fs from "node:fs"; +import path from "node:path"; + +/** + * Reject any existing symlink component below a trusted root. + * + * Checking only the final realpath is insufficient: a link from `wiki/` to + * `my_thoughts/` stays inside the workspace while crossing its permission + * boundary. A missing leaf is also unsafe when one of its parents is a link. + */ +export function assertNoSymlinkComponents(abs: string, root: string): void { + const rel = path.relative(root, abs); + if (rel === "" || rel.startsWith("..") || path.isAbsolute(rel)) { + throw new Error(`路径越界：${abs}`); + } + + let cursor = root; + for (const component of rel.split(path.sep)) { + cursor = path.join(cursor, component); + try { + if (fs.lstatSync(cursor).isSymbolicLink()) { + throw new Error(`路径不允许包含符号链接：${abs}`); + } + } catch (error: unknown) { + if ( + error && + typeof error === "object" && + "code" in error && + (error as { code: string }).code === "ENOENT" + ) { + // Once a component is absent, no later child can already exist. + break; + } + throw error; + } + } +} diff --git a/workspaces/rag-evolution/log.md b/workspaces/rag-evolution/log.md index 3d1aac1..50094c9 100644 --- a/workspaces/rag-evolution/log.md +++ b/workspaces/rag-evolution/log.md @@ -194,3 +194,23 @@ list-conflicts 仍为 5 处。 - annotate-section 回填 6 个关键章节摘要 - 综合判断:首篇 ingest,无前置冲突;预言后续 CRAG 会走不同路径(evaluator 而非 critic)、Search-R1 / R1-Searcher 用 RL 挑战 reflection tokens 范式 - lint:`list-source-issues` 全部通过 + +## [2026-07-02] update | 引用核对整备(list-cite-mismatches 首跑) + +新引入的确定性引用核对 lint(`k.py list-cite-mismatches`)首次全库扫描,发现并整备 9 处引用质量问题: + +- **错引修复**(数字/引文不在被引块): + - [[wiki/concepts/rag_vs_long_context]]:LC/RAG 绝对分数(49.70 等 6 个)补引 raw 数据表 `^t-33-0c8446`(原只引 sources 摘要块);7B/13B 模型规格补引 [[wiki/sources/self_rag#^p-4-34d5b1]] + - [[wiki/sources/hipporag1]]:头部"closely inspired by HippoRAG"引文在被引块中不存在,改写为被 [[wiki/sources/hipporag2#^p-3-6066c2]] 真实支撑的转述 + - [[wiki/sources/ragas]]:头部"automated RAG metrics"归因于 §VI(⊙ 扫读档,wiki 层无支撑块),去引号转述并挂 [需要来源] 待 partial re-ingest 核实 +- **无来源数字显式化**:[[wiki/concepts/rag_evaluation]] 与 [[wiki/analyses/kb_vs_human_survey_coverage]] 中"NQ 等简单 benchmark 80%+"挂 [需要来源](被引 crag_benchmark 块只支撑 ~40% 部分) +- **强调引号去引**(自述措辞非原文引述,避免逐字核对误报):[[wiki/concepts/self_reflective_rag]] ×2、[[wiki/sources/flare]] ×1 +- 收尾:三个 demo 库 `list-cite-mismatches` 闸门项(mismatch / exempt-missing-basis)全部归零,守护测试 `TestCiteCheckDemoGuard` 钉住该状态 + +## [2026-07-02] update | 条目级引用核对首跑修复 + +- 原子论断分解（list 条目粒度）上线后首跑抓出 1 处块级检查不可见的条目级错引： + [[wiki/concepts/rl_augmented_retrieval]] 中「7-8B 超过 GPT-4o-mini」错引到 + [[wiki/sources/search_r1]] 的局限性清单块，实际出处为 [[wiki/sources/r1_searcher]] + （含 7B/8B/GPT-4o-mini/multi-hop 全部关键事实的 ^p-4-40c58f），已改锚 +- 三库闸门项复归零（TestCiteCheckDemoGuard 守护） diff --git a/workspaces/rag-evolution/wiki/analyses/kb_vs_human_survey_coverage.md b/workspaces/rag-evolution/wiki/analyses/kb_vs_human_survey_coverage.md index 0d7fadc..c3c4e4f 100644 --- a/workspaces/rag-evolution/wiki/analyses/kb_vs_human_survey_coverage.md +++ b/workspaces/rag-evolution/wiki/analyses/kb_vs_human_survey_coverage.md @@ -2,7 +2,7 @@ title: "KB 自动综合 vs Gao 2024 RAG Survey 严格定量对照(Batch 1 升级版)" type: analysis created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -182,7 +182,7 @@ Batch 2-4 完成后,本 analysis 的覆盖度统计**从 14 篇扩到 38 篇**: Batch 4 完成后,KB 通过 38 篇 ingest 识别出 Gao Survey **没预期到的 3 个 2024-2025 重要演化**: 1. **2025 Agentic RAG 三家鼎立**(冲突 #6):o1 派 / R1 派 / IL 派,survey 写作时(2023 末)R1 范式尚不存在 -2. **CRAG benchmark 揭示主流 RAG 真实性能**:**GPT-4 + 主流 RAG 在 CRAG 上仅 ~40% 准确率**,远低于 NQ 等过时 benchmark 的 80%+[[wiki/sources/crag_benchmark#^p-3-3284e5]]。这意味着 **survey Table I 收录的 70+ 方法的"SOTA"宣称大多基于过时 benchmark**,真实场景仍有巨大改进空间 +2. **CRAG benchmark 揭示主流 RAG 真实性能**:**GPT-4 + 主流 RAG 在 CRAG 上仅 ~40% 准确率**,远低于 NQ 等过时 benchmark 的 80%+[需要来源][[wiki/sources/crag_benchmark#^p-3-3284e5]]。这意味着 **survey Table I 收录的 70+ 方法的"SOTA"宣称大多基于过时 benchmark**,真实场景仍有巨大改进空间 3. **Domain-specific RAG 形成独立子方向**:RAFT + KAG 共同主张"通用 RAG 不足以胜任专业领域",这在 survey 三代分类内无对应槽位 ^p-16-a856c5 ## 后续工作 ^h-2-5-7d734f diff --git a/workspaces/rag-evolution/wiki/concepts/rag_evaluation.md b/workspaces/rag-evolution/wiki/concepts/rag_evaluation.md index 3262e26..b7dd5a5 100644 --- a/workspaces/rag-evolution/wiki/concepts/rag_evaluation.md +++ b/workspaces/rag-evolution/wiki/concepts/rag_evaluation.md @@ -2,7 +2,7 @@ title: "RAG Evaluation (评估框架与 Benchmark)" type: concept created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -46,7 +46,7 @@ tags: | **[[wiki/sources/crag_benchmark]]** | 2024-06 | 4.4K | 4 领域 + 8 query 类型 + 3 维度变化 | aggregation / set / temporal / false premise | ^t-2-db7484 ^p-4-c9bb14 -**关键论断**:**主流 RAG(GPT-4 + dense retriever)在 CRAG benchmark 上准确率仅 ~40%** —— 远低于 NQ 等简单 benchmark 上 80%+ 的水平[[wiki/sources/crag_benchmark#^p-3-3284e5]]。**这意味着 RAG 演化的"性能领先"宣称大多基于过时 benchmark,真实场景仍有巨大改进空间**。 ^p-5-f6b9bf +**关键论断**:**主流 RAG(GPT-4 + dense retriever)在 CRAG benchmark 上准确率仅 ~40%** —— 远低于 NQ 等简单 benchmark 上 80%+ 的水平[需要来源][[wiki/sources/crag_benchmark#^p-3-3284e5]]。**这意味着 RAG 演化的"性能领先"宣称大多基于过时 benchmark,真实场景仍有巨大改进空间**。 ^p-5-f6b9bf ## 与本 KB 已 ingest 30 篇论文的关系 ^h-2-3-de05f1 diff --git a/workspaces/rag-evolution/wiki/concepts/rag_vs_long_context.md b/workspaces/rag-evolution/wiki/concepts/rag_vs_long_context.md index 11e81d1..574d9f8 100644 --- a/workspaces/rag-evolution/wiki/concepts/rag_vs_long_context.md +++ b/workspaces/rag-evolution/wiki/concepts/rag_vs_long_context.md @@ -2,7 +2,7 @@ title: "RAG vs Long-Context LLMs (路线之争)" type: concept created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: medium @@ -28,7 +28,7 @@ tags: **LC consistently outperforms RAG**,只要资源足够[[wiki/sources/rag_or_longcontext#^p-2-32eee0]]: - Gemini-1.5-Pro: LC 49.70 vs RAG 37.33(**+7.6%**) - GPT-4O: LC 48.67 vs RAG 32.60(**+13.1%**) -- GPT-3.5-Turbo: LC 32.07 vs RAG 30.33(+3.6%) +- GPT-3.5-Turbo: LC 32.07 vs RAG 30.33(+3.6%)[[raw/papers/2024-07-rag-or-longcontext#^t-33-0c8446]] 但 **RAG 仍有不可替代场景**: - **成本优势**:输入 token 数显著少,直接对应 API 成本 @@ -59,8 +59,8 @@ DeepMind 总结 RAG 输给 LC 的查询类型: ## 对前 3 篇 RAG 工作的隐含挑战 ^h-2-4-698a79 -- **vs [[wiki/concepts/self_reflective_rag]] (Self-RAG)**:Self-RAG 改 generator 控制流以接近 ChatGPT 表现——但若 GPT-4O LC 直接超过 13.1%,Self-RAG 的 7B/13B 模型努力空间在哪?[[wiki/sources/rag_or_longcontext#^p-2-32eee0]] -- **vs [[wiki/concepts/corrective_rag]] (CRAG)**:CRAG 的 web search 兜底解决"static corpus 不够"——但 LC 1M 上下文直接吞下整个 corpus,web search 必要性下降 +- **vs [[wiki/concepts/self_reflective_rag]] (Self-RAG)**:Self-RAG 改 generator 控制流以接近 ChatGPT 表现——但若 GPT-4O LC 直接超过 13.1%,Self-RAG 的 7B/13B 模型努力空间在哪?[[wiki/sources/self_rag#^p-4-34d5b1]][[wiki/sources/rag_or_longcontext#^p-2-32eee0]] +- **vs [[wiki/concepts/corrective_rag]] (CRAG)**:CRAG 的 web search 兜底解决 static corpus 覆盖不足的问题——但 LC 1M 上下文直接吞下整个 corpus,web search 必要性下降 - **vs [[wiki/concepts/graph_rag]] (GraphRAG)**:GraphRAG 的 global sensemaking 优势,LC 长上下文也能做(成本贵但能做) **关键观察**:这 3 篇 RAG 改进与 LC 路径**并非互斥**,但 LC 让"基础 RAG 不够好"的痛点弱化,RAG 改进的价值更多落到**成本敏感场景** + **corpus 大于 context** 两个 specific use case 上。 ^p-5-174837 diff --git a/workspaces/rag-evolution/wiki/concepts/rl_augmented_retrieval.md b/workspaces/rag-evolution/wiki/concepts/rl_augmented_retrieval.md index c7a273d..61587a6 100644 --- a/workspaces/rag-evolution/wiki/concepts/rl_augmented_retrieval.md +++ b/workspaces/rag-evolution/wiki/concepts/rl_augmented_retrieval.md @@ -2,7 +2,7 @@ title: "RL-Augmented Retrieval (2025 Agentic RAG 三家鼎立)" type: concept created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -58,7 +58,7 @@ tags: ## 经验结果摘要 ^h-2-4-6a2356 - **多 multi-hop 大幅领先 GPT-4o-mini**(R1-Searcher 在 HotpotQA +48.22% vs ReARTeR base on GPT-4o-mini)[[wiki/sources/r1_searcher#^p-4-40c58f]] -- **开源 7B 不输闭源大模型**——RL 训出的搜索决策可以让 7-8B 在 multi-hop QA 上超过 GPT-4o-mini[[wiki/sources/search_r1#^p-7-93266d]] +- **开源 7B 不输闭源大模型**——RL 训出的搜索决策可以让 7-8B 在 multi-hop QA 上超过 GPT-4o-mini[[wiki/sources/r1_searcher#^p-4-40c58f]] - **OOD 泛化好**:R1-Searcher 仅训 HotpotQA + 2Wiki,在 Bamboogle online search 也胜 32B Search-o1[[wiki/sources/r1_searcher#^p-4-40c58f]] - **PPO 和 GRPO 都 work**(Search-R1 ablation) ^p-5-913382 diff --git a/workspaces/rag-evolution/wiki/concepts/self_reflective_rag.md b/workspaces/rag-evolution/wiki/concepts/self_reflective_rag.md index 4b41e37..92ae5c7 100644 --- a/workspaces/rag-evolution/wiki/concepts/self_reflective_rag.md +++ b/workspaces/rag-evolution/wiki/concepts/self_reflective_rag.md @@ -2,7 +2,7 @@ title: "Self-Reflective RAG (自反思 RAG)" type: concept created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -114,9 +114,9 @@ Batch 2 ingest 后,本概念页可以梳理 **"自决检索"思想的完整演 ## 与其他范式的关系 ^h-2-6-41a67a -- vs [[wiki/concepts/retrieval_augmented_generation]] (vanilla RAG):本范式解决了 vanilla RAG 的"无差别检索 + 不 grounded 生成"两大缺陷 +- vs [[wiki/concepts/retrieval_augmented_generation]] (vanilla RAG):本范式解决了 vanilla RAG 的无差别检索、不 grounded 生成两大缺陷 - vs [[wiki/concepts/corrective_rag]] (CRAG, 2024-01):**同样做自纠错,但路径根本不同**——CRAG 用外部 lightweight evaluator(T5-large 0.77B)而非内化 critic,且引入 web search 兜底。Self-CRAG > Self-RAG 在 PopQA/Bio/PubHealth 上[[wiki/sources/crag#^p-2-04adb2]](详见上方冲突标注) -- vs [[wiki/concepts/graph_rag]] (GraphRAG / LightRAG / HippoRAG 2):图结构路线 — 改造的是"corpus 的组织形式"而非"检索-生成-评估的控制流" +- vs [[wiki/concepts/graph_rag]] (GraphRAG / LightRAG / HippoRAG 2):图结构路线 — 改造的是 corpus 的组织形式,而非检索-生成-评估的控制流 - vs [[wiki/concepts/rl_augmented_retrieval]] (Search-R1 / R1-Searcher 2025-03):**功能性替代**——RL outcome reward 直接训"检索 → 推理 → 答案"链路,reflection tokens 范式作为通用框架已被替代(详见上方第 2 个冲突标注块) ^p-10-98d609 ## 关联页面 ^h-2-7-e10f36 diff --git a/workspaces/rag-evolution/wiki/sources/flare.md b/workspaces/rag-evolution/wiki/sources/flare.md index 6031a43..890146d 100644 --- a/workspaces/rag-evolution/wiki/sources/flare.md +++ b/workspaces/rag-evolution/wiki/sources/flare.md @@ -2,7 +2,7 @@ title: "FLARE: Active Retrieval-Augmented Generation (Jiang et al. 2023-05)" type: source_summary created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -23,7 +23,7 @@ tags: > **作者**: Zhengbao Jiang, Frank F. Xu et al.(CMU) > **发表**: 2023-05 arXiv preprint(EMNLP 2023) > **arXiv**: [2305.06983](https://arxiv.org/abs/2305.06983) -> **历史地位**:**Self-RAG 之前最有影响力的 adaptive retrieval 工作**;Gao Survey §II-C2 把 FLARE 和 Self-RAG 并列为 "Modular RAG / adaptive retrieval" 模式代表[[wiki/sources/gao_rag_survey#^p-5-3e97fb]] ^p-1-23031a +> **历史地位**:**Self-RAG 之前最有影响力的 adaptive retrieval 工作**;Gao Survey §II-C2 把 FLARE 和 Self-RAG 并列为 Modular RAG / adaptive retrieval 模式代表[[wiki/sources/gao_rag_survey#^p-5-3e97fb]] ^p-1-23031a ## 摘要 ^h-2-1-3ae146 diff --git a/workspaces/rag-evolution/wiki/sources/hipporag1.md b/workspaces/rag-evolution/wiki/sources/hipporag1.md index 2115d9b..7c4720f 100644 --- a/workspaces/rag-evolution/wiki/sources/hipporag1.md +++ b/workspaces/rag-evolution/wiki/sources/hipporag1.md @@ -2,7 +2,7 @@ title: "HippoRAG 1: Neurobiologically Inspired Long-Term Memory (Gutiérrez et al. 2024-05)" type: source_summary created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -24,7 +24,7 @@ tags: > **作者**: Bernal Jiménez Gutiérrez, Yiheng Shu et al.(Ohio State NLP Group) > **发表**: 2024-05 arXiv preprint(NeurIPS 2024) > **arXiv**: [2405.14831](https://arxiv.org/abs/2405.14831) -> **特殊地位**:**[[wiki/sources/hipporag2]] 的直接前作**,HippoRAG 2 在 §3.1 显式说"closely inspired by HippoRAG"[[wiki/sources/hipporag2#^p-3-6066c2]] ^p-1-23fd9a +> **特殊地位**:**[[wiki/sources/hipporag2]] 的直接前作**,HippoRAG 2 在其 OpenIE + Personalized PageRank 框架基础上加 3 处改进(Dense-Sparse / Deeper Contextualization / Recognition Memory)[[wiki/sources/hipporag2#^p-3-6066c2]] ^p-1-23fd9a ## 摘要 ^h-2-1-3ae146 diff --git a/workspaces/rag-evolution/wiki/sources/kag.md b/workspaces/rag-evolution/wiki/sources/kag.md index 18c84ea..fbd63cf 100644 --- a/workspaces/rag-evolution/wiki/sources/kag.md +++ b/workspaces/rag-evolution/wiki/sources/kag.md @@ -30,13 +30,19 @@ tags: 第③档 KAG 处理 — 因 KAG 是**专业领域应用论文**(医疗/金融),技术深度对本 KB demo 的核心论断(RAG 演化范式)贡献度有限,**仅深读概念层章节**: -| Section | 状态 | 选择理由 | -|---|---|---| -| Introduction | ✓ 深读 | KAG 设计目标 + 与 vanilla RAG / GraphRAG 对照 | -| Related Work | ⊙ 扫读 | survey-style 内容,扫读建立映射 | -| KAG Framework | ⊙ 扫读 | 核心方法,但深度细节(domain-specific 设计)对 demo 主线收益小 | -| Domain Applications(医疗/金融) | × 跳过 | domain-specific 案例 | -| Experiments | × 跳过 | 实验细节 | +| Anchor | 原标题 | 状态 | 选择理由与关键词 | +|---|---|---|---| +| ^h-2-1-f8eb4d | 1 Introduction | ✓ 深读 | KAG 设计目标 + 与 vanilla RAG / GraphRAG 对照 | +| ^h-2-2-631edd | 2 Approach | ⊙ 扫读 | 核心方法(登记时曾意译为"KAG Framework"),深度细节对 demo 主线收益小;关键实体:LLMFriSPG / Mutual Indexing / Logical Form Solver / Knowledge Alignment / KAG-Model / semantic chunking | +| ^h-2-3-aa357d | 3 Experiments | × 跳过 | 实验细节;关键词:HotpotQA / 2WikiMultiHopQA / MuSiQue / EM / F1 / ablation | +| ^h-2-4-8d5c18 | 4 Applications | × 跳过 | domain-specific 案例;关键词:E-Health 医疗 / 电信客服;原登记名"Domain Applications" | +| ^h-2-5-cf45fd | 5 Related Works | ⊙ 扫读 | survey-style 内容,扫读建立映射;原登记名"Related Work" | +| ^h-2-6-93f646 | 6 Limitations | × 跳过 | 短小的局限性讨论 | +| ^h-2-7-72b5e1 | 7 Conclusion and Future Work | × 跳过 | 短结论 | +| ^h-2-8-e7caf3 | 8 Acknowledgements | × 跳过 | 元信息 | +| ^h-2-9-44a2dc | References | × 跳过 | 参考文献清单(元信息) | +| ^h-2-10-92f8c2 | Appendix A Example of KAG Solver | × 跳过 | 附录示例 | +| ^h-2-11-028df1 | Appendix B Example of Logical form Reasoner | × 跳过 | 附录示例 | | Discussion / Conclusion | × 跳过 | 短结论 | ^t-1-5d67e9 ^p-2-af89c4 diff --git a/workspaces/rag-evolution/wiki/sources/ragas.md b/workspaces/rag-evolution/wiki/sources/ragas.md index 8b63df4..0aec3f5 100644 --- a/workspaces/rag-evolution/wiki/sources/ragas.md +++ b/workspaces/rag-evolution/wiki/sources/ragas.md @@ -2,7 +2,7 @@ title: "RAGAS: Automated Evaluation of RAG (Es et al. 2023-09)" type: source_summary created_date: 2026-05-26 -last_modified: 2026-05-26 +last_modified: 2026-07-02 last_modified_by: LLM status: draft confidence: high @@ -22,7 +22,7 @@ tags: > **作者**: Shahul Es, Jithin James et al.(Exploding Gradients 创业团队) > **发表**: 2023-09 arXiv preprint(EACL 2024) > **arXiv**: [2309.15217](https://arxiv.org/abs/2309.15217) -> **特殊地位**:**Gao Survey §VI Task and Evaluation 引用 RAGAS 作 "automated RAG metrics" 代表**[[wiki/sources/gao_rag_survey#^p-9-8779ac]];RAGAS Python 库(github.com/explodinggradients/ragas)是工程界事实标准 ^p-1-f54d4c +> **特殊地位**:**Gao Survey §VI Task and Evaluation 将 RAGAS 列为自动化 RAG 评估指标代表**[需要来源](§VI 在 [[wiki/sources/gao_rag_survey]] 中为 ⊙ 扫读档,待 partial re-ingest 核实);RAGAS Python 库(github.com/explodinggradients/ragas)是工程界事实标准 ^p-1-f54d4c ## 摘要 ^h-2-1-811e72 diff --git a/workspaces/rag-evolution/wiki/sources/search_o1.md b/workspaces/rag-evolution/wiki/sources/search_o1.md index 4ae3e4f..7016f2e 100644 --- a/workspaces/rag-evolution/wiki/sources/search_o1.md +++ b/workspaces/rag-evolution/wiki/sources/search_o1.md @@ -31,13 +31,13 @@ tags: 第③档处理(151K 字符)— 因 Search-o1 是 2025 RL/Agentic RAG 范式之争的关键对照基线,深读 Method 部分:[[raw/papers/2025-01-search-o1]] -| Section | 状态 | 选择理由 | -|---|---|---| -| Introduction | ✓ 深读 | 与 R1 派对照的核心动机 | -| Related Work | ⊙ 扫读 | 综述类内容 | -| Method (Search-o1 + Reason-in-Documents) | ✓ 深读 | 核心方法,与 R1 派 RL 对照 | -| Experiments | ⊙ 扫读 | 结果数字,与 R1-Searcher Bamboogle 对比的关键 | -| Discussion / Conclusion | × 跳过 | 短结论 | +| Anchor | 原标题 | 状态 | 选择理由与关键词 | +|---|---|---|---| +| ^h-2-1-f8eb4d | 1 Introduction | ✓ 深读 | 与 R1 派对照的核心动机 | +| ^h-2-2-3cc1d1 | 2 Related Work | ⊙ 扫读 | 综述类内容;关键词:RAG / o1-like reasoning / test-time scaling | +| ^h-2-3-b5857d | 3 Methodology | ✓ 深读 | 核心方法(原登记名"Method");关键实体:Agentic RAG mechanism / Reason-in-Documents / batch inference | +| ^h-2-4-5a99ff | 4 Experiments | ⊙ 扫读 | 结果数字,与 R1-Searcher Bamboogle 对比的关键;关键词:GPQA / HotpotQA / 2WikiMultihopQA / MuSiQue / Bamboogle / QwQ-32B | +| ^h-2-5-* | 5 Conclusion 及之后 | × 跳过 | 短结论与附录(原登记名"Discussion / Conclusion") | ^t-1-5d67e9 ^p-2-af89c4 ## 摘要 ^h-2-1-811e72 diff --git a/workspaces/smb-ecommerce/wiki/sources/businessofapps_shein_2026.md b/workspaces/smb-ecommerce/wiki/sources/businessofapps_shein_2026.md index 9496d59..5ea9c55 100644 --- a/workspaces/smb-ecommerce/wiki/sources/businessofapps_shein_2026.md +++ b/workspaces/smb-ecommerce/wiki/sources/businessofapps_shein_2026.md @@ -19,7 +19,7 @@ tags: # Shein Revenue and Usage Statistics 2026（Business of Apps） ^h-1-1-9ef4db -> Business of Apps 维护的 SHEIN 全球统计数据集合。**关键口径提醒**：页面标题为 "2026"，但所有数据表的时间序列止于 **2023**——这是一份 2023 年口径的历史数据集，并非 2026 实时数据。引用时务必标明年份，避免与 wiki 中更新的 2024/2025 数据混淆。 ^p-1-1f1457 +> Business of Apps 维护的 SHEIN 全球统计数据集合。**关键口径提醒**：页面标题为 "2026"，但所有数据表的时间序列止于 **2023**——这是一份 2023 口径的历史数据集，并非实时数据。引用时务必标明年份，避免与 wiki 中更新的 2024/2025 数据混淆。 ^p-1-1f1457 ## 核心数据（2023 口径）^h-2-1-summary ^h-2-1-b8c3d1 diff --git a/workspaces/smb-ecommerce/wiki/sources/sacra_shein_revenue_valuation.md b/workspaces/smb-ecommerce/wiki/sources/sacra_shein_revenue_valuation.md index b6d4f68..33b1fdb 100644 --- a/workspaces/smb-ecommerce/wiki/sources/sacra_shein_revenue_valuation.md +++ b/workspaces/smb-ecommerce/wiki/sources/sacra_shein_revenue_valuation.md @@ -19,7 +19,7 @@ tags: # Sacra：SHEIN Revenue / Valuation / Funding ^h-1-1-11157b -> Sacra Research 维护的 SHEIN 机构级财务档案（28.1K 字符）。涵盖营收 / 估值 / 融资 / 商业模式 / 竞争 / TAM 扩张 / 风险七大块。私营公司估值常以 Sacra 数据为参考。 ^p-1-cc2306 +> Sacra Research 维护的 SHEIN 机构级财务档案（约三万字符的长文）。涵盖营收 / 估值 / 融资 / 商业模式 / 竞争 / TAM 扩张 / 风险七大块。私营公司估值常以 Sacra 数据为参考。 ^p-1-cc2306 ## 核心财务数据 ^h-2-1-e5f739

%（覆盖率 ；上界解释见命令输出的诚实边界说明） +- 引用审计未通过已标注： 条 → <已标 CAUTION 的页面> ## 待人类判别 <对所有未决冲突列表，每条带链接和摘要> @@ -349,7 +369,8 @@ tags: ## [2026-04-28] lint | 周度健康检查 WXX - 处理积压： 条 - 修复孤儿： 个 -- fact-check：/ 通过 +- cite-mismatch 修复： 条 +- 引用审计： 对（S/P/U = //），UNSUPPORTED 已标注 条 - 待人类判别冲突： 条 - 产出：`wiki/analyses/周报-YYYY-WXX.md` ``` @@ -366,6 +387,7 @@ git commit -m "lint: 周度健康检查 WXX" ## 完成检查清单 - [ ] 跑了 health 拿到全景 +- [ ] evidence_index 为 complete/fresh，三类机械覆盖均为 100%，异常空章节已确认/修复 - [ ] 处理了 to-be-updated 积压（或在周报中说明为什么没处理） - [ ] 处理了孤儿页面（4 种处理方式之一） - [ ] 复核了所有未决冲突 diff --git a/.agents/skills/kb-query/SKILL.md b/.agents/skills/kb-query/SKILL.md index 6aa6028..6bdbcad 100644 --- a/.agents/skills/kb-query/SKILL.md +++ b/.agents/skills/kb-query/SKILL.md @@ -13,7 +13,7 @@ description: 知识库查询工作流——像研究员一样按 root_index → > - `Read` / `Write` 与 `git add` 必须用**带 workspace 的全路径**（如 `Read workspaces/smb-ecommerce/wiki/root_index.md`）。 > - 下文示例为可读性写成裸路径形式（`wiki/...` / `raw/...`），落地执行时一律替换为 `workspaces//...`，k.py 命令加上 `--workspace `。 -**核心原则**（来自 AGENTS.md）： +**核心原则**（来自 CLAUDE.md）： - 永远读完整页面或完整 H2/H3 段，不读 chunk - 优先 wiki/（已编译的综合判断），其次 raw/ - 回答必须附引用清单 @@ -21,33 +21,33 @@ description: 知识库查询工作流——像研究员一样按 root_index → --- -## 第 0 步：模式说明（Codex 中固定走 quick） +## 第 0 步：模式说明（Claude Code 中固定走 quick） -本 skill 设计了 4 个查询深度模式，但**在 Codex 中，默认且唯一行为是 quick 模式**——直接跳到第 1 步执行即可。 +本 skill 设计了 4 个查询深度模式，但**在 Claude Code 中，默认且唯一行为是 quick 模式**——直接跳到第 1 步执行即可。 -**不做关键词自动判别**。原因：基于查询文本的模糊触发词（"合规"、"审计"等）不可靠——同一个词在不同语境含义不同，agent 会误判。可靠的方式是显式 API 字段 / CLI 参数 / Web UI 下拉框，这些都是产品化定制时的工作，不在 Codex 默认行为内。 +**不做关键词自动判别**。原因：基于查询文本的模糊触发词（"合规"、"审计"等）不可靠——同一个词在不同语境含义不同，agent 会误判。可靠的方式是显式 API 字段 / CLI 参数 / Web UI 下拉框，这些都是产品化定制时的工作，不在 Claude Code 默认行为内。 ### 4 个模式定义（设计规范，留待产品化时落地） | 模式 | 工具调用预算 | 典型场景 | 与 quick 的差异 | |---|---|---|---| -| **quick**（Codex 默认 + 唯一）| 5-10 次 | "X 是什么 / 时间 / 数据" | 第 1-8 步完整流程 | +| **quick**（Claude Code 默认 + 唯一）| 5-10 次 | "X 是什么 / 时间 / 数据" | 第 1-8 步完整流程 | | **audit**（产品化预留）| 15-25 次（1.5-3x）| 高 stake 决策：合规 / 合同 / 投资 / 估值 | quick 流程末尾追加"引用核实通道"（详见第 7a 步） | | **explore**（产品化预留）| 20-40 次（3-5x）| 复杂综合：推荐 / 评估 / 行业现状 | quick 流程"决定停止"判据放宽，BFS outlinks 2 层 + 读所有 source_summary（详见第 7b 步） | | **devil**（产品化预留）| 10-20 次（1.5-2x）| 反驳 / 风险点 / 决策审视 | quick 流程末尾追加"反对论构造 + 证据反查"（详见第 7c 步） | ### 高级模式的调用方式（未来产品化） -3 个高级模式不在 Codex 中自动触发，将通过**显式机制**调用： +3 个高级模式不在 Claude Code 中自动触发，将通过**显式机制**调用： - **CLI 参数**：未来可能在 `k.py` 加 `query` 子命令，支持 `--mode={quick,audit,explore,devil}` - **API 字段**：包成 HTTP 服务时，请求体里带 `"mode": "audit"` 字段 - **Web UI 下拉**：产品化 web 端的聊天框旁加模式选择器 - **system prompt 注入**：DeepSeek / Claude API 接入时，在 system prompt 里硬编码 mode 行为 -这些都是后续产品化开发的工作。**当前 Codex 用户**：默认 quick，第 1-8 步走完即可，第 7a/b/c 步当作设计文档参考。 +这些都是后续产品化开发的工作。**当前 Claude Code 用户**：默认 quick，第 1-8 步走完即可，第 7a/b/c 步当作设计文档参考。 -> **重要：第 4.6 步「细节下钻判据」是所有模式共有的核心行为，quick 也执行**——它不属于产品化预留。即 Codex 默认的 quick 模式**会在真正需要原文细节时主动 `read-block` / `read-section` 回查原文**，只是不像 audit 那样对每条引用全量核验。模式差异仅在第 7a/b/c 的追加动作，不在"要不要按需下钻原文"这件事上。 +> **重要：第 4.6 步「细节下钻判据」与第 4.7 步「细节发现」是所有模式共有的核心行为，quick 也执行**——它不属于产品化预留。即 Claude Code 默认的 quick 模式**会在真正需要原文细节时主动 `read-block` / `read-section` 回查原文**，只是不像 audit 那样对每条引用全量核验。模式差异仅在第 7a/b/c 的追加动作，不在"要不要按需下钻原文"这件事上。 --- @@ -119,7 +119,7 @@ Read wiki/sources/.md 3. 触发原因：`query 命中关键词「」但原扫读未深入` 4. 升级完成后回到本流程，**重新** `Read` 已更新的 source_summary 与受影响 wiki 页，再综合回答 -> **不要拿 ⊙ 扫读章节的 outline preview 当真知识使用**——它只是标题级判断，没有论证细节支撑。AGENTS.md "Ingest 操作流程" 反例区已明令。 +> **不要拿 ⊙ 扫读章节的 outline preview 当真知识使用**——它只是标题级判断，没有论证细节支撑。CLAUDE.md "Ingest 操作流程" 反例区已明令。 ## 第 4.6 步：细节下钻判据（所有模式适用，quick 也执行） @@ -152,6 +152,75 @@ python scripts/k.py read-section raw/papers/.md # 取完整 H2/ > **与 audit 模式（第 7a 步）的关系**：本步是日常的"按需精确"——只对真正需要原文级精确的论断下钻；audit 是"全量强化版"——对答案里**每条** anchor 都回查。本步在所有模式（含 quick）默认执行，audit 在此之上把核验扩展到全部引用，两者不冲突。 +## 第 4.7 步：细节发现（原文级检索，wiki 蒸馏层没有时） + +第 4.6 步管「核验」——wiki 已有论断，回原文核对；本步管「发现」——**问题要的细节 +wiki 综合层根本没有**（长文档 ingest 后 90%+ 的内容只在 raw）。此时不要直接回答 +"知识库未涵盖"，**按问题类型选路由**（两路可交替、可都走）： + +| 问题特征 | 首选路径 | +|---|---| +| 有明确术语 / 专名 / 数字（"SCaNN 的查询延迟是多少"） | **自然单元证据索引路**（快而准） | +| 概念性 / 改述性 / 跨文档（"有没有讲过用 RL 替代监督微调这类思想的内容"） | **浏览路**（你阅读目录做语义判断——无 embedding 架构下语义召回的正解） | + +### 自然单元证据索引路（search-evidence） + +0. **先验新鲜度闸门**： + ```bash + python scripts/k.py evidence-index-coverage + ``` + 只有自然单元与章节覆盖为 100%、`corpus_freshness.ok=true` 才能检索。索引 missing/stale/incomplete 时先 `rebuild-evidence-index`；不得在旧索引上把“没搜到”解释为知识库没有。 +1. **先保留原问题，再做显式分面/关键词扩展**：先用用户原问题跑一遍；再把问题拆成必答 facets，逐 facet 补中英同义词、术语/俗称、缩写/全称。扩展只用于召回，不得把第一轮候选中的未知答案或 Gold 信息偷塞进查询： + ```bash + python scripts/k.py search-evidence "<用户原问题>" --limit 20 + python scripts/k.py search-evidence "" --limit 20 \ + --expand "<英文术语/全称>" --expand "<中文别名/缩写>" + ``` + multi-hop 的后续查询只能使用**第一跳实际读到**的新实体，不能预填中间答案。 +2. **按完整证据集而非单一 top-1 收集**：合并各 facet 的 top-20，确认每个必答 facet 至少有一份候选；表格行/list item 虽共享父块 anchor，仍按返回的 `unit_id/content_hash/subordinal` 区分，不能把同锚的错误行当成正确行。先 `read-evidence-unit ` 读取被选中的精确行/条目，再读取其 `canonical_ref` 父块获得上下文。 + - `search-evidence` 对超长自然单元只返回最多 30K、围绕命中词的原文摘录，并标 `text_truncated=true`。`read-evidence-unit` 默认同样在 30K 处 fail-closed；不得由 agent 自动用 `--max-chars 0` 绕过。此时应下钻来源结构/换更细自然块，或将“显式无限制阅读”作为人工审查逃生口。 +3. **精读与选择裁决**：search-evidence 是机械候选生成，命中片段 ≠ 语义支撑——对拟选候选必须先 `read-evidence-unit` 核对精确自然单元，再 `read-block` 打开 canonical parent block，核对主体、指标、值、单位、条件、否定和时间范围。只有核对通过的 unit handle 才能成为 selected evidence；都不对就换措辞、迭代检索或转浏览路。 + + **粒度边界必须披露**：当前 Markdown `[[raw/...#^t/p-...]]` 仍引用整个父 table/list block，精确 row/item 身份存在 `unit_id` 派生索引中。最终写入 wiki 前应尽量把论断拆到单行职责，并保留所选 unit handle 的审计记录；不能声称现有 Markdown 已实现 row-level block anchor，也不能用父表中另一行的值支撑当前行。 +4. **兼容回退**：旧索引不可用且已明确披露降级时，才用 `search-raw` 做纯扫描；它不是与新索引静默混用的第二套排名真相。 + +### 浏览路（corpus-map → outline → read-section） + +像人类研究员翻图书馆目录，三跳定位任何章节，每跳都是你在做语义判断： + +1. **全库地图**（一次调用获得全库视野；~98 篇约 12K token，可 --file 聚焦）： + ```bash + python scripts/k.py corpus-map # 全库 + python scripts/k.py corpus-map --file <路径子串> --sections # 聚焦候选，逐节显示摘要 + ``` + 每篇给出：标题 / 字符数 / 档位 / 顶层章节树（标题 + 摘要或首段预览）/ 深度登记 + 状态（✓⊙×）/ 关联 source_summary（「未 ingest」= 从未蒸馏过的盲区文档，优先怀疑）。 +2. **选篇钻取**：读地图选 2-4 篇候选 → `outline ` 看完整章节树 + preview → + 选章节 → `read-section`（单节 ≤30K；超长节继续下钻完整子节，无子标题时按自然单元定位并读完整父块，不做任意 chunk）。 +3. **精读裁决**：读完由你判断内容是否回答问题；不是 → 回地图换候选。 + +### 两路共享的收尾（缺一不可） + +- **懒回填摘要**：经浏览路 `read-section` 读过的章节，若其 agent_summary 为空， + 顺手 `annotate-section` 回填一句——读都读了，零边际成本；全库摘要密度（目前 + 普遍很低）靠真实使用增长，与懒深化同构，地图会因此越用越准。**但 annotation 会使证据索引的摘要路由指纹过期**：本次所有懒回填完成后，作答/归档前统一重跑 `rebuild-evidence-index` + `evidence-index-coverage`；若还要继续 `search-evidence`，则必须先重建。 +- **deepen 触发**：读到的内容所在章节在登记表为 ⊙ 扫读且被用于作答 → 按第 4.5 步 + 触发 partial re-ingest（search-raw 命中自带 deepen_hint 标注；浏览路看 corpus-map + 的 [⊙] 标注即知）。 +- **作答必须引用读过并通过主体/条件核对的原文块锚点**（`[[raw/...#^p-...]]`）；table/list 候选还必须先用 `read-evidence-unit` 锁定具体行/条目——search-evidence / corpus-map / outline 的 + 标题、摘要、preview 都只是路标，**不许当内容作答**（摘要可能失真，且无块级锚点 + 不可溯源）。 +- 照常过第 7.5 步 check-draft 闸门。 + +**raw 不在场时（release demo / 浅 clone）**：`search-evidence` / `search-raw` / `corpus-map` 会明确提示 +raw/ 未分发——此时原文级检索不可用，基于 wiki 蒸馏层作答并在答案中说明「原文 +不在场，细节无法核验」，不要假装检索过原文（与引用核对的 unverifiable 降级同一 +诚实原则）。 + +**何时放弃**：自然单元索引路（原问题 + 每个 facet 至少 2 组措辞）与浏览路（corpus-map 候选筛过）都落空，才可以 +回答"知识库未涵盖"，并说明尝试过的检索词与翻过的候选文档（诚实透明，也方便用户 +提供正确术语后重试）。 + ## 第 5 步：顺藤摸瓜（backlinks / outlinks） 如果第 4 步发现某页很关键，查它的关系网络： @@ -210,9 +279,31 @@ python scripts/k.py list-conflicts --json # 看有没有相关冲突 - 不要编造未在 wiki 中出现的内容 - 如果 wiki 没说，明确说"知识库中未涵盖这个主题" +## 第 7.5 步：答案落笔前的严格引用闸门（实质性 KB 回答必做） + +只要答案包含从 KB 提取的**实质性事实论断**（不再只限于数字 / 引文），发出前都必须经过完整证据取回、fresh-context 语义审计和 `strict` 总闸门。仅纯导航回应或「知识库未覆盖」的拒答可跳过。 + +```bash +# 1) 把最终答案草稿写到临时文件（如 /tmp/answer-draft.md），先跑零 LLM 确定性层 +python scripts/k.py --workspace check-draft /tmp/answer-draft.md + +# 2) 外部跨模型二审：完整 evidence + 盲填 + 对抗反驳 + 受控入账 +# 需 DEEPSEEK_API_KEY；工具/网络/核验包任一失败都是非 0，不得当作通过 +python tools/cite-audit/audit.py --workspace --draft /tmp/answer-draft.md + +# 3) 最终 fail-closed 总闸门 +python scripts/k.py --workspace check-draft /tmp/answer-draft.md --strict +``` + +`--strict` 必须全绿：它会阻断 broken / raw 不可得 / 非 canonical 锚点 / imprecise / pending `[需要来源]` / bare 或 coarse / 无当前版本 provenance / 未审或非 `SUPPORTED` verdict / 截断核验包。任一 checker 异常或超时同样视为失败，禁止用「删引用」或改成整页链接洗白。 + +如果当前环境无跨模型 API，则必须由**不共享写作上下文**的核验 agent 逐对判定，再用 `cite-audit-log --draft /tmp/answer-draft.md` 受控入账；无法完成这一步时，只能对相应事实拒答 / 明确标为未核验，不得交付「已验证」引用。 + +> 这是第 4.6 步「按需下钻」的机器化收口：quote-first 减少写错，`strict` 证明当前草稿的每条保留引用都已现场取回、语义审计并与现行内容 hash 绑定。 + ## 第 7a 步：audit 模式扩展 — 引用核实通道 -> **产品化预留**——Codex 默认不执行此步。下方流程是为后续产品化（CLI / API / Web UI 显式调用）准备的设计规范，未来 agent 在 mode=audit 时按此执行。 +> **产品化预留**——Claude Code 默认不执行此步。下方流程是为后续产品化（CLI / API / Web UI 显式调用）准备的设计规范，未来 agent 在 mode=audit 时按此执行。 quick 流程组合答案后，**回头校验每条 anchor 引用是否真的支撑论断**。这是 audit 模式的核心价值——捕获 wiki 写错 / AI 综合时偏离原文 / anchor hash 失配。 @@ -256,7 +347,7 @@ quick 流程组合答案后，**回头校验每条 anchor 引用是否真的支 ## 第 7b 步：explore 模式扩展 — 广度扫读 -> **产品化预留**——Codex 默认不执行此步。下方流程是后续产品化时（mode=explore）的执行规范。 +> **产品化预留**——Claude Code 默认不执行此步。下方流程是后续产品化时（mode=explore）的执行规范。 quick 模式优化"最少读"，explore 反过来——主动多读，挖未被 concept/analysis 综合层暴露的细节。 @@ -305,7 +396,7 @@ quick 模式优化"最少读"，explore 反过来——主动多读，挖未被 ## 第 7c 步：devil 模式扩展 — 反对论构造 -> **产品化预留**——Codex 默认不执行此步。下方流程是后续产品化时（mode=devil）的执行规范。 +> **产品化预留**——Claude Code 默认不执行此步。下方流程是后续产品化时（mode=devil）的执行规范。 agent 给完答案后，**主动构造一个反对当前结论的论证**，然后在 wiki 找证据支撑或反驳这个反对论。 @@ -405,18 +496,21 @@ git commit -m "query: <分析主题>" ## 完成检查清单 -**Codex 默认 quick 模式**： +**Claude Code 默认 quick 模式**： - [ ] 走过 root_index → MOC → 具体页的层级（没有直接 Grep） - [ ] 读了完整页面，没有读 chunk - [ ] 检查了相关 source_summary 的「章节深度登记」表；命中扫读章节已触发 partial re-ingest 升级 - [ ] 命中"需要原文级精确"的论断（精确引文/数字/日期/条款，或来源为第③档长文）已按第 4.6 步 `read-block`/`read-section` 核验原文 +- [ ] 原文细节问题已先确认 `evidence-index-coverage` 为 100% 且 fresh；按 required facets 跑 `search-evidence` top-20，拟选引用逐块核验，未把 candidate non-empty 冒充回答正确 - [ ] 至少做了一次 backlinks 或 outlinks 检查（除非问题极简单） - [ ] 做了覆盖度自检 - [ ] 答案的每个论断都有引用 +- [ ] 实质性 KB 回答已完成草稿跨模型审计，并跑 `check-draft --strict` 全绿（第 7.5 步） +- [ ] 细节不在 wiki 时走过第 4.7 步双路由（关键词路 ≥2 组措辞 / 浏览路 corpus-map 筛过候选），没有轻易回答"未涵盖"；读到 ⊙ 扫读章节已触发 partial re-ingest；浏览路读过的无摘要章节已顺手 annotate-section - [ ] （如果有价值）归档到 analyses/ 并 commit -**产品化预留模式**（Codex 不执行，留作未来 CLI / API / Web UI 显式调用时的执行规范）： +**产品化预留模式**（Claude Code 不执行，留作未来 CLI / API / Web UI 显式调用时的执行规范）： - audit：第 7a 步 — 对答案里每条 anchor 引用调 read-block 比对原文；「## 审计报告」节列每条 ✅ / ⚠️ / ❌ - explore：第 7b 步 — 读所有 source_summary + outlinks 跟 2 层 + list-conflicts / list-bare-claims 扫；「## 扩展阅读发现」节 @@ -430,3 +524,5 @@ git commit -m "query: <分析主题>" - ❌ 切片读取（只读某段、想象其他段的内容） - ❌ 把综合分析丢弃在对话历史里（应归档到 analyses/） - ❌ 跳过第 4.5 步——拿 ⊙ 扫读章节的 outline preview 当真知识用（必须先触发 partial re-ingest 升级到 ✓ 深读再综合） +- ❌ wiki 综合层查不到就直接回答"知识库未涵盖"——细节大概率在 raw 里，必须先走第 4.7 步双路由（关键词路 + corpus-map 浏览路）；也不要只试一组关键词就放弃 +- ❌ 拿 corpus-map / outline 的标题、摘要、preview 当内容作答——它们只是路标，作答必须 read 原文块并引用块级锚点 diff --git a/.claude/skills/kb-cite-audit/SKILL.md b/.claude/skills/kb-cite-audit/SKILL.md new file mode 100644 index 0000000..7024cbe --- /dev/null +++ b/.claude/skills/kb-cite-audit/SKILL.md @@ -0,0 +1,129 @@ +--- +name: kb-cite-audit +description: 知识库引用语义审计工作流——确定性枚举（论断, 引用）审计对，逐条以 fresh-context 判定被引原文是否真的支撑论断，判定入验证台账，未通过的落 CAUTION 标注交人裁决。当用户提到"审计引用"、"核对引用"、"引用是否正确"、"cite audit"、"验证来源"、"检查论断与原文一致性"时使用此 skill。 +--- + +# 知识库引用语义审计工作流（kb-cite-audit） + +你现在是知识库的 **引用审计员**。任务：验证 wiki 论断的块级引用在**语义上**成立——被引的那个块是否真的支撑该论断。结构 lint（broken-refs / bare-claims / coarse-citations）只保证「锚点存在、有引用、粒度够细」；本流程补上「引的内容对不对」。 + +> **Workspace 前提（必读）**：数据层按主题隔离在 `workspaces//` 下。本文所有 `wiki/`、`raw/`、`log.md` 路径均相对当前 workspace；`k.py` 命令加 `--workspace `；`Read` / `Edit` / `git add` 用带 workspace 的全路径。 + +> **raw 不在场即降级（必读）**：本仓的 demo 库不分发 `raw/`（版权），协作者浅 clone 同理。`extract-claims` 会把这类引用标为 `raw-not-distributed`——**直接跳过、勿烧 token 强行判定**；语义审计只在 raw 在场的环境（dev 仓 / 用户自己的库）实际执行。 + +**核心原则**： + +- **KB 出数据、agent 出判断**：`extract-claims` / `cite-audit-log` 是确定性工具；「支撑与否」的判定由你（外部 agent）做，这是全流程唯一需要智能的一步。 +- **fresh-context 判定**：只依据 `claim_text` + 现场取回的被引原文下判断。**禁止**凭「我 ingest 时读过这篇」的记忆判定——上下文污染正是错引的成因，不能再用它做裁判。 +- **不默默修正**：发现错引不改论断本身，落 CAUTION 标注 + `citation-suspect` 标签，让人在工作台裁决（与 kb-lint 的冲突处理铁律一致）。 +- **台账是 memoization，不是真相**：`.cache/citation_audit.jsonl` 只记「谁在何时核验过什么」，删了重审即重建；「这条引用有问题」这一知识状态**只以 markdown 标注为准**（`list-suspect-citations` 扫 markdown，不读台账）。 + +--- + +## 第 1 步：枚举待审对 + +```bash +# 增量模式（周检 / 例行）：只审从未审过 + 内容已漂移的对，配额抽样 +python scripts/k.py --workspace extract-claims --unaudited-only --sample 20 --seed --json + +# 全量首审（一次性存量清偿）：去掉 --sample；量大时分多轮 +python scripts/k.py --workspace extract-claims --unaudited-only --json +``` + +- `--seed` 用当周周号（如 `2026-W27`）：同周重跑取样一致（可复现），跨周覆盖累积。 +- 「已审」判定是双 hash pin：claim 内容变 → pair_id 变；被引块内容变（含 ^h- 节**正文**重写而标题不变）→ `target_content_hash` 漂移——任一侧变化都自动回到未审，无需人为盯。 + +## 第 2 步：分诊 + +按 `target_status` 分流，只对 `ok` 的对做语义判定： + +| target_status | 处置 | +|---|---| +| `ok` | 进入第 3 步判定 | +| `raw-not-distributed` | `cite-audit-log --verdict UNVERIFIABLE` 批量入账（不烧 token；k.py 校验：只有目标确实不可得才接受 UNVERIFIABLE） | +| `file-missing` / `anchor-missing` | 归 broken-refs 流程修复，本流程跳过（不双报） | + +## 第 3 步：逐条 fresh-context 判定 + +对每对（建议 20-30 条一批，批间不携带前批内容）： + +1. 取回被引原文（**必须现场取，不用记忆**）： + ```bash + python scripts/k.py --workspace read-block # ^p-/^t-/^c-/^f- + python scripts/k.py --workspace read-section # ^h- + ``` +2. 判定三问： + - ① **关键事实在场**：论断中的数字 / 日期 / 主体是否出现在被引原文？ + - ② **直接支撑**：论断是否被原文直接支撑（不需要额外推理 / 拼接其他来源）？ + - ③ **语义 drift**：有无过度概括（"多数"写成"所有"）、加了原文没有的限定词 / 程度词、因果错置（correlation 写成 causation）？ +3. 映射 verdict：三问全过 → `SUPPORTED`；部分支撑 / 措辞偏移 → `PARTIAL`；被引块与论断无关或关键事实不在 → `UNSUPPORTED`；论断与被引证据**相反** → `CONTRADICTED`。 +4. **盲填复核（含数字的对必做）**：`extract-claims --unaudited-only --cloze` 输出挖空论断（数字→⟦N1⟧）；核验时**只看「挖空论断 + 被引原文」**填空（不看期望值），`python scripts/k.py cloze-check --batch ` 机器判分（块级 union：一块多引用时数字由块内任一引用的原文填出即可）。判分未过按 UNSUPPORTED 处置——盲填从原理上消灭判定式审核的附和偏差，并抓「数字巧合在场但归属错误」。 +5. **跨模型二审（推荐）**：`python tools/cite-audit/audit.py --workspace [--unaudited-only|--all --sample N --seed ]`——外部客户端（tools/ 例外区）调 DeepSeek 自动跑双通道并入台账；`--all --sample` 模式可对已 SUPPORTED 记录换模型交叉复查（防橡皮图章与同源盲区）。问答 / 导出草稿可用 `--draft /tmp/answer.md`，判定会通过草稿专属 pair 受控入账。需 `DEEPSEEK_API_KEY`；无 key 时以 fresh-context 子 agent 反驳式抽查替代。 + + 审计器的结果协议是 fail-closed：exit `0` = 所有实际判定对通过，exit `1` = 完成判定且发现语义未通过，exit `2` = 网络 / 协议 / 核验包 / quote / skipped 等导致审计未完成。`2` 不是「没发现问题」，禁止当作通过；截断 claim/evidence/cloze 和多来源盲填冲突都进 incomplete，不冒充 UNSUPPORTED 或语义查全命中。 + +**防误伤细则**（判定前先过一遍）： + +- **多来源合成论断**（`multi_source: true`，多见于 analysis / comparison 页）：一块多引用时，引用只对其**紧邻的前方分句**负责；单个被引块只支撑论断的一部分是**正常形态** → `PARTIAL` 且**不落标注**。只有「被引块与归属分句无关或矛盾」才 `UNSUPPORTED`。 +- **合法转述白名单**（按 `SUPPORTED` 处理，note 记明）：约数舍入（约 40% ↔ 39.2%）、跨语言日期 / 数字格式转写（August 29 ↔ 2025-08-29）、已标 `[KB 推算: ^锚]` 且现场验算成立的派生算术（差值 / 倍数 / 单位换算——验算不成立则 `UNSUPPORTED`）。 +- **^h- 大节引用**：支撑句埋在整节多处（分布式支撑）属正常 → `PARTIAL` 不落标注，建议 note 记「宜降 ^p- 级锚」。 + +## 第 4 步：判定入台账 + +```bash +# 单条（agent 记 SUPPORTED 必须附 --evidence：被引块原文的一段字面子串，k.py 会校验—— +# 证明确实取回过原文，杜绝橡皮图章） +python scripts/k.py --workspace cite-audit-log --pair --verdict SUPPORTED \ + --evidence "<从 read-block 返回内容里复制的一段>" --note "三问全过" + +# 批量：JSONL 文件每行 {"pair_id": "...", "verdict": "...", "note": "...", "evidence": "..."} +python scripts/k.py --workspace cite-audit-log --batch /tmp/verdicts.jsonl +``` + +k.py 的确定性校验（被拒说明流程有问题，不要绕）：pair 过期（内容已变）拒绝；目标可解析时记 UNVERIFIABLE 拒绝；SUPPORTED 无 / 假 evidence 拒绝。 + +## 第 5 步：UNSUPPORTED 落 CAUTION 标注 + +对 `UNSUPPORTED` / `CONTRADICTED`（以及**实质事实错**的 PARTIAL），`Edit` 在论断块**正下方**追加审计标注（格式固定，`list-suspect-citations` 靠它扫描）： + +```markdown +> [!CAUTION] 引用审计未通过 — YYYY-MM-DD +> **论断**：<论断句摘录>（块 ^p-4-34d5b1） +> **被引块**：[[#^]] +> **审计判定**：UNSUPPORTED — <一句差异说明，如"被引表格中数字为 65.9 非 66.9"> +> **建议**：<改引 [[#^<正确锚>]] / 修正论断数字 / 删除论断> +> **状态**：⏳ 待人类判别 +``` + +同时该页 frontmatter `tags` 追加 `citation-suspect`。**不改论断原文**——修复由人裁决（或人授权后走正规修复 + 删标注，claim 变化会自动触发重审）。 + +## 第 6 步：收尾对账 + 汇报 + commit + +```bash +python scripts/k.py --workspace list-suspect-citations --check-ledger +``` + +- `ledger-unsupported-without-marker` 必须为空——有 = CAUTION 标注被删而论断未改（「删标注蒸发」），恢复标注。 +- log.md 追加 `lint` 类条目：`- 引用审计： 对（S/P/U = x/y/z），UNSUPPORTED 已标注 z 条`。 +- git commit（只含 `wiki/**` 标注改动 + log.md；**.cache 台账不入库**）：`git commit -m "lint: 引用审计 对"`。 + +--- + +## 完成检查清单 + +- [ ] `extract-claims --unaudited-only` 本批返回的 `ok` 对已全部判定并入账 +- [ ] 核验包的 claim / evidence / cloze 未截断，审计器没有 incomplete / skipped / ledger error +- [ ] `raw-not-distributed` 的对已批量记 UNVERIFIABLE（没有烧 token 强判） +- [ ] 每条 SUPPORTED 都带真实 `--evidence`（k.py 校验通过） +- [ ] UNSUPPORTED / CONTRADICTED 已落 CAUTION 标注 + `citation-suspect` 标签，**没有默默修正论断** +- [ ] `list-suspect-citations --check-ledger` 对账一致 +- [ ] log.md 记账、git commit 完成（.cache 不入库） + +## 反例（绝对不要做） + +- ❌ 凭「我 ingest 时读过」的记忆判定，不现场 `read-block` 取回原文——上下文污染不能当裁判 +- ❌ 对可解析的目标记 UNVERIFIABLE 跳过劳动（k.py 会拒绝；被拒就老实取原文） +- ❌ 把多来源合成论断的正常 PARTIAL 当错引落标注（误伤会让人对审计失去信任） +- ❌ 发现错引直接改论断数字（默默修正）——必须落标注走人裁 +- ❌ 删 CAUTION 标注但不修论断（对账会报 ledger-unsupported-without-marker） +- ❌ 把台账 jsonl 提交进 git（它是派生层缓存，删了重审即重建） diff --git a/.claude/skills/kb-edit-source/SKILL.md b/.claude/skills/kb-edit-source/SKILL.md new file mode 100644 index 0000000..c671887 --- /dev/null +++ b/.claude/skills/kb-edit-source/SKILL.md @@ -0,0 +1,261 @@ +--- +name: kb-edit-source +description: 知识库来源编辑工作流——当底层来源（raw 的原文）内容发生变化时，安全地改源头、重转换、把受影响的 wiki 引用与论断同步更新，原子提交。当用户提到"改 raw"、"修改来源"、"原文改了要同步 wiki"、"来源内容更新了"、"edit source"、"改了原始笔记"、"更新引用的原文"时使用此 skill。 +--- + +# 知识库来源编辑工作流（kb-edit-source） + +你现在是知识库的 **来源同步专家**。当一份**原文来源的内容变了**（数字改了、段落重写、章节增删），你要保证引用它的 wiki 页面不悄悄过期。严格遵守以下流程。 + +> **Workspace 前提（必读）**：数据层按主题隔离在 `workspaces//` 下。本文中所有 `wiki/`、`raw/`、`exports/`、`log.md` 路径均**相对于当前 workspace**，实际位于 `workspaces//`（如 `workspaces/smb-ecommerce/wiki/...`）。 +> - 默认 workspace 为 `smb-ecommerce`，不显式指定时即用它（向后兼容）。 +> - `k.py` / `convert.py` 用 `--workspace ` 指定 workspace（参数紧跟脚本名后）。 +> - **跨独立项目（KB_ROOT 外置）时**：数据不在引擎目录，而在 `KB_ROOT` 指向的数据根。此时**每条 `k.py` / `convert.py` / `git` 命令都必须携带 `KB_ROOT`，且 git 必须用 `git -C "$KB_ROOT"` 打到数据仓**（不是引擎 cwd）。`KB_ROOT` 含空格 / 中文时**务必加引号**。 +> - `Read` / `Edit` 与 wiki 路径参数用**带 workspace 的全路径**（如 `workspaces//wiki/sources/.md`）。 + +**核心原则（不可违反）**： + +- **agent 绝不写 `raw/`**。`Edit(raw/**)` / `Write(raw/**)` / `Edit(**/raw/**)` 等被 `.claude/settings.json` 硬 deny（绝对优先、无 ask 回退）。**raw 内容的唯一编辑发生在 raw 之外的「真相源原件」上**（如 Obsidian vault 里的 `.md`）；**raw 的唯一写入者是 `convert.py`（只加锚点）**；把更新后的原件放进 raw 用 `cp`（= ingest「把原始文件放入 raw」的入口，是机械镜像、不是手改内容）。 +- **markdown + Git 是唯一真相源**；锚点是内容的确定性函数——改内容 → 锚点变 → `list-broken-refs` 精确暴露失效引用。你的工作就是把这些失效引用逐条修回，并把**事实真变了**的论断按新原文改写。 +- **删除即标记**：被删/被反驳的论断，改引新锚 / 挂 `[需要来源]` + `#to-be-updated` / 走冲突标注——**绝不静默删引用洗白**。 +- **一次来源同步 = 一个 git commit**（只含 wiki + log.md；raw 是否入库见第 8 步按 `git check-ignore` 决定）。 + +> **本 skill 与 partial re-ingest 的分工**：partial re-ingest 是「**来源没变**、把扫读章节升级为深读」；本 skill 是「**来源内容真的变了**」。别混。 + +--- + +## 第 0 步：先判来源类型（硬分支，决定本 skill 适不适用） + +| 来源类型 | 判据 | 本 skill 怎么走 | +|---|---|---| +| **原生 markdown 来源** | raw 里那个 `.md` 就是内容本体（如 Obsidian 笔记的副本），无同名二进制原件 | ✅ 适用：内容在**真相源原件**（Obsidian `.md`）上改，见下 | +| **二进制派生来源** | raw 里有同名 `.pdf`/`.docx`/`.html`，`.md` 是 `convert.py` 的派生物 | ⚠️ 内容编辑**不适用 agent**：需人改二进制原件再 convert；agent 只做「重转换 + wiki 迁移」（第 4-8 步） | +| **human-only / locked 来源** | 对应 source_summary 或原件带 `#human-only` / `locked: true` | ❌ 拒绝驱动编辑，转交人类 | + +判二进制：`ls workspaces//raw// | grep ` 看有无同名 pdf/docx。**下文以「原生 markdown 来源、真相源在外部（如 Obsidian）」为主线**；二进制来源跳过第 2-3 步（人改原件），其余相同。 + +--- + +## 第 1 步：编辑前——用 backlinks 圈定全部受影响面 + +**改任何东西之前**，先落一份「谁引用了这个来源」的清单——因为改动后锚点会变，事后光靠 broken-refs 可能定位不全扇出。 + +```bash +# 列出所有引用该 raw 的 wiki 页 + 具体锚点（KB_ROOT 外置时加前缀） +python scripts/k.py --workspace backlinks raw/articles/.md +``` + +把命中的 wiki 页、每处 `#^anchor`、上下文抄下来（这是第 5-6 步要逐条修的清单）。 + +- **backlinks 为空**（该来源还没被任何 wiki 引用）：跳过第 5-6 步的引用迁移，但**仍要做第 3-4 步重转换**保持派生层新鲜；若本次无 wiki 变更则无需 KB commit（同步只落原件 + raw 副本），log.md 仍记一条 `update` 备案。 + +**留证（DELETE/重写高危时必做）**：对将被删或大改的被引块，先 `read-block` 取回**旧原文快照**贴进 log.md 或对应 source_summary 的历史 NOTE——KB 仓不留 raw 历史（见第 8 步），旧原文一旦删掉在 KB 内不可恢复。 + +```bash +python scripts/k.py --workspace read-block raw/articles/.md ^p-12-7d8e9a +``` + +--- + +## 第 2 步：在「真相源原件」上改内容（不碰 raw） + +单向同步，方向写死：**改 Obsidian（或其它外部）原件 → 覆盖 raw 副本 → convert**。**永远不要反向把 KB 锚点写回原件**。 + +``` +# 用 Edit 改真相源原件（它在 raw/** deny glob 之外，Edit 允许）： +# $SRC_ROOT/.md +``` + +- 内容编辑一律用 `Edit` 改这个**外部原件**。它不在任何 `raw/**` deny 内，可以改。 +- 三种编辑心里有数（决定第 5 步怎么修引用）：**MODIFY**（改某块正文/数字）/ **ADD**（插入新段/新章节）/ **DELETE**（删段/删章节）。ADD、DELETE 会让**下游所有块的 seq 位移**（见第 5 步）。 + +--- + +## 第 3 步：把更新后的原件镜像进 raw（cp，不是 Edit） + +raw 内容的唯一合法来源是「被 `cp` 进来的真相源原件」。agent **绝不**用 `Edit`/`Write`/`>` 重定向手改 raw；`cp` 是机械镜像（等同 ingest 前置「把原始文件放入 raw」）。 + +```bash +# SRC_ROOT = 真相源原件所在目录（在 raw/** deny 之外的可编辑位置，如你的 Obsidian +# vault 的笔记文件夹）。KB_ROOT = 数据根（含 workspaces/）。二者按你的实际布局设。 +SRC_ROOT="<你的真相源原件目录>" # 例：某个 Obsidian vault 的笔记文件夹 +# 干净原件（无 KB 锚点）整文件覆盖 raw 副本 +cp "$SRC_ROOT/.md" \ + "$KB_ROOT/workspaces//raw/articles/.md" + +# 一致性硬校验：两份必须逐字节相同（不同说明没覆盖成功 / 覆盖错文件） +diff "$SRC_ROOT/.md" \ + "$KB_ROOT/workspaces//raw/articles/.md" && echo "SYNCED" + +# 防「原件里混入了 KB 锚点」导致 convert 静默跳过：grep 到就得处理（见第 4 步硬规则） +grep -nE '\^[hpctf]-[0-9]' "$KB_ROOT/workspaces//raw/articles/.md" && echo "⚠️ 原件含 KB 锚点，需 --force 或先剥锚" || echo "无残留锚点，OK" +``` + +> `cp`/`diff`/`grep` 不在 deny 列表内（deny 的是 `Edit`/`Write`/`rm`/`mv` 对 raw）。若不想让 agent 碰 raw 副本，也可让**人**做这一步 cp，agent 从第 4 步接手——二选一，别让 agent 用 Edit/Write 写 raw。 + +--- + +## 第 4 步：重转换——只重锚被改的那个文件 + +刚 `cp` 进来的原件没有 KB 锚点，所以 `convert.py --workspace` 会重新处理它。对其它文件，`should_convert()` 现在校验 outline schema、全文 SHA-256、章节 SHA-256、canonical anchors 与结构；只有内容地址化契约完整且新鲜才跳过，mtime 不能掩盖同长度改写或坏 outline。 + +```bash +python scripts/convert.py --workspace +``` + +> **硬规则**：仍推荐用无锚干净原件覆盖再 convert；未改块会确定性复现原 anchor，真改块才换锚。不要直接手改已锚 raw（权限上也禁止），也不要裸跑全量 `--force`；新 validator 会拒绝坏/旧 outline，但它不是授权手改 raw 的理由。 + +重转换后必须重建派生证据索引；raw SHA 已变化时旧索引会 fail-closed： + +```bash +python scripts/k.py --workspace rebuild-evidence-index +python scripts/k.py --workspace evidence-index-coverage +``` + +--- + +## 第 5 步：查失效引用 + 三诊断逐条修 + +```bash +python scripts/k.py --workspace list-broken-refs +``` + +它会列出所有指向该 raw、锚点已对不上的 wiki 引用。**注意 raw 目标无 hash6 容错**——任何使块 seq 位移的增删（ADD/DELETE）都会让**下游每一条 raw 引用**被报失效，哪怕内容一字未改。别慌，用**三诊断**逐条判：对每条 broken raw ref，拿它的**旧锚**去 `read-block`： + +```bash +python scripts/k.py --workspace read-block raw/articles/.md ^p-3-7cb619 +``` + +| 诊断 | read-block 结果 | 含义 | 处置 | +|---|---|---|---| +| **(a) 纯 seq 位移** | 成功回收（带 `recovered_from`），preview 与 wiki 论断**一致** | 块只是被上游增删挤动了位置，内容没变（hash6 不变） | **只机械改锚串里的 seq**（保留 hash6），无需重新语义回验 | +| **(b) MODIFY** | 成功回收，但 preview 内容**已变** | 被引块正文真的改了（hash6 变） | 打开新原文，**按原文改写 wiki 论断**（数字按原文改；结论若反转 → 挂 `[!WARNING]` 冲突标注、不覆盖），改锚到新块，重新引用回验 | +| **(c) DELETE** | **回收失败**（hash6 已不存在） | 被引块被整段删了 | 该论断在新原文别处仍有支撑 → 改引新锚；无支撑 → 挂 `[需要来源]` + 页面加 `#to-be-updated`；来源撤稿致论断失真 → 走冲突/降级。**绝不静默删引用** | + +**确定性批量重映射**（应对 seq 级联）：改一处正文没事，但**增删一个块会让下游一片引用失效**。用 `blocks` 建 hash6→当前锚 的映射，机械改 seq： + +```bash +python scripts/k.py --workspace blocks raw/articles/.md --json +# 对每条 broken raw ref：解析其 hash6（⚠️ 锚点可能带碰撞后缀 ^p-2-7cb619-2，别假设结尾就是 6 位 hex） +# → 在 blocks 输出里按 hash6 找当前锚 → 保留 hash6、只把 wiki 引用里的 seq 改成新值 +``` + +用 `find-anchor` 按**新内容**反查新锚（用于 MODIFY 的新块定位）： + +```bash +python scripts/k.py --workspace find-anchor raw/articles/.md "<改后的原文片段>" +``` + +**扇出**：backlinks 清单里的**每一处**都要修；同一个被改数字若散落多个 wiki 页，**所有出现点**都按新原文一致更新。修到 `list-broken-refs` **归零**才算完。 + +--- + +## 第 6 步：连带更新（③ 档登记表 / 摘要 / frontmatter） + +- **③ 档「章节深度登记」表**（无 lint 兜底）：若被编辑的 source_summary 含此表，且编辑增删/改名了 H 标题，表内 `^h-...` 锚点与原标题会集体陈旧、`search-raw` 的 deepen 联动失明。用 `outline` / `blocks` 拉新 H 锚全表，逐行重写 **Anchor 列**与**原标题列**（原标题改了要同步、不许意译）。 +- **agent_summary 摘要**：对内容实变的章节，用 `annotate-section` 重写其摘要（否则摘要层描述旧内容而无告警）： + ```bash + python scripts/k.py --workspace annotate-section raw/articles/.md h-2-3-abc123 "本节现在论证..." + ``` +- **frontmatter**：更新受影响 wiki 页的 `last_modified`（今天）+ `last_modified_by: LLM`；若编辑删掉了某 concept 论断的唯一来源支撑，按需调 `source_count` / `sources` 或补 stub 标记。 +- **ADD 了全新实质章节**：这不是「修引用」，是「新材料进来了」——对新段走 kb-ingest 式**深读 + annotate-section**，按需**新建 / 更新** wiki 论断（区别于 partial re-ingest 的 ⊙→✓ 升级）。 + +--- + +## 第 7 步：提交前质量闸门（用 `--paths` 显式驱动，别用 `--changed`） + +> **为什么不用 `--changed`**：KB_ROOT 外置 + 中文/空格路径 + 独立数据仓时，`--changed` 依赖 `git -C /workspaces/` 判定改动文件；一旦 wiki 未跟踪 / 被 gitignore，改动集为空 → 枚举 **0 对**却报「通过」——**核心闸门假绿**。改用 `--paths <本次改过的 wiki 页...>`，并断言**枚举对数 > 0**（你改了 N 页，就该有 ≥N 对）。 + +```bash +P="wiki/sources/a.md wiki/concepts/b.md ..." # 本次改过的所有 wiki 页 + +python scripts/k.py --workspace list-broken-refs # 必须归零 +python scripts/k.py --workspace list-cite-mismatches # 闸门项（mismatch/exempt-missing-basis）须为空 +python scripts/k.py --workspace list-bare-claims # 空 +python scripts/k.py --workspace list-coarse-citations # 空 +python scripts/k.py --workspace list-source-issues # 空 +# 引用回验：显式 --paths + 断言 pairs>0，再对新增/改写的（论断,引用）对做 fresh-context 判定 +python scripts/k.py --workspace extract-claims --paths $P --with-evidence --json +``` + +验收：`list-broken-refs` = 0；`list-cite-mismatches` 闸门项 = 0；bare/coarse/source-issues 全空；`extract-claims` 返回的对数 > 0（不是 0！）且新增/改写对的语义回验 UNSUPPORTED/CONTRADICTED = 0（判定入 `cite-audit-log --mode ingest`）。**wiki 页无手写块锚点导致 extract-claims 枚举为 0 时**：先 `convert.py --dir workspaces//wiki` 给 wiki 加锚点、再删生成的 `*.outline.json`（wiki 不留 outline.json），然后重跑。 + +--- + +## 第 8 步：原子提交（打到数据仓，raw 是否入库看 check-ignore） + +```bash +# git 必须用 -C 打到「数据仓」（KB_ROOT），不是引擎 cwd +git -C "$KB_ROOT" add workspaces//wiki workspaces//log.md + +# raw 是否入库：由该仓 .gitignore 决定，别猜——用 check-ignore 验 +git -C "$KB_ROOT" check-ignore workspaces//raw/articles/.md \ + && echo "raw 被 ignore：commit 只含 wiki+log（原文可复现性依赖外部原件仓）" \ + || git -C "$KB_ROOT" add "workspaces//raw/articles/.md" # 未 ignore（本库 raw 入库）→ 一并提交 + +git -C "$KB_ROOT" -c user.name="" -c user.email="" \ + commit -m "edit-source: <来源标题> 内容更新，同步 wiki 引用" +``` + +- **raw 被 gitignore 时**（发布版默认）：commit 只含 `wiki/** + log.md`；被引原文的可复现性**依赖外部真相源仓**（如 Obsidian vault 自身受版本控制）——log.md 里记下对应原件路径（+ 若原件仓有版本，记其 commit sha），把两仓这次变更人工挂钩。诚实边界：**raw 内容变更无法从 KB 仓回滚**。 +- **raw 未 ignore 时**（如自建私人库把 raw 纳入版控）：raw 副本一并提交，KB 仓自包含、可回滚。 +- 绝不 `git add -A` / `git add .`。 + +--- + +## 第 9 步：追加 log.md + +`Edit` `log.md`，文件**头部**插入： + +```markdown +## [YYYY-MM-DD] update | <来源标题> 内容更新 +- 来源：`raw/articles/.md`（真相源原件：``） +- 编辑类型：MODIFY / ADD / DELETE（一句话说改了什么） +- 修引用：<改了哪几页的哪几处 anchor>；纯 seq 位移 N 处、实质改写 M 处、DELETE 处置 K 处 +- 连带：<③ 档登记表重对齐 / annotate-section 刷摘要 / frontmatter> +- 闸门：list-broken-refs 归零；cite-mismatch 闸门项 0；extract-claims pairs= +- 溯源挂钩：外部原件仓 sha=<...>（若 raw 未入 KB 仓） +``` + +--- + +## 基线漂移自检（首次在某库用本 skill 前跑一次） + +信「干净覆盖复现锚点」之前，先证它：拿一个**未编辑**的原件，`cp` 覆盖 raw 副本 → `convert` → `list-broken-refs`。**必须零新增失效引用**。若非零，说明外部原件与 KB 里当初 ingest 的版本已漂移（例如原件被塞了 Obsidian 专属语法：原生 `^blockid`、`%%注释%%`、`![[嵌入]]`、dataview 块），需先归一化/剥掉这些再走本流程。 + +--- + +## 完成检查清单 + +- [ ] 第 0 步判明来源类型（原生 md / 二进制派生 / human-only-locked），走对分支 +- [ ] 编辑前 `backlinks` 落全部受影响 wiki 页清单；高危块 `read-block` 留旧原文快照 +- [ ] 内容只在**外部真相源原件**上用 `Edit` 改；**没有对任何 `raw/**` 文件 Edit/Write** +- [ ] `cp` 干净原件覆盖 raw 副本；`diff` 校验逐字节一致；`grep` 确认无残留 KB 锚点 +- [ ] `convert.py --workspace`（覆盖后无锚，免 --force）；未裸跑全量 `--force` +- [ ] `rebuild-evidence-index` 后 natural/content/structural coverage=100%、freshness=true；异常空章节已确认 +- [ ] `list-broken-refs` 逐条三诊断修完，**归零**；扇出全部引用页都修 +- [ ] MODIFY 的数字/结论按新原文改写（结论反转挂冲突标注，不覆盖）；DELETE 未静默删引用 +- [ ] ③ 档「章节深度登记」表 Anchor/原标题重对齐；内容实变章节 `annotate-section` 刷摘要 +- [ ] 受影响 wiki 页 frontmatter（last_modified + last_modified_by + 按需 source_count/sources） +- [ ] 闸门用 `--paths` 显式跑、断言 `extract-claims` pairs>0：broken-refs / cite-mismatch 闸门项 / bare / coarse / source-issues 全过 +- [ ] `git -C "$KB_ROOT"` 提交（不是引擎 cwd）；raw 按 `check-ignore` 决定是否入库；无 `git add -A` +- [ ] log.md 追加 `update` 条目 + 溯源挂钩 + +## 错误恢复 + +- 中途出错**不要 partial commit**。`git -C "$KB_ROOT" status` 查看修改。 +- 撤回单个 wiki 文件：`git -C "$KB_ROOT" checkout `。 +- raw 副本改错了：从外部真相源原件重新 `cp` 覆盖再 convert（原件是源，raw 只是镜像）。 +- 整体撤回未 commit 的 wiki 改动：与用户确认后 `git -C "$KB_ROOT" stash`。 + +## 反例（绝对不要做） + +- ❌ 用 `Edit` / `Write` / `>` 重定向手改 `raw/**` 下任何文件（含原生 md）——被 deny，且违反「raw 只读、唯一写入者是 convert.py」。内容改在**外部真相源原件**上。 +- ❌ 把 KB 锚点（`^p-...`）反向写回 Obsidian 原件——污染真相源，下次覆盖会级联乱套。同步严格单向。 +- ❌ 把 mtime/“文件里已有锚点”当作 outline 新鲜度证明——现行 convert 必须通过 schema + 全文/章节 hash + 结构 validator；推荐仍是无锚干净原件覆盖 +- ❌ 裸跑 `convert --dir raw/... --force` 全量重锚——顺带重锚别的陈旧文件，冒出无关 broken-refs、信号不可归因。 +- ❌ 用 `--changed` 驱动 `extract-claims` / `check-provenance` 闸门——KB_ROOT 外置/未跟踪时会枚举 0 对却报通过（假绿）。用 `--paths` + 断言 pairs>0。 +- ❌ `git` 在引擎 cwd 里跑 commit——会打到引擎仓、漏掉数据仓的 wiki 改动。必须 `git -C "$KB_ROOT"`。 +- ❌ 对 DELETE 掉的被引块「删引用、挂 [需要来源]」当无事发生——要么改引新锚、要么显式 `[需要来源]` + `#to-be-updated` + 留旧原文快照。 +- ❌ 只修 `list-broken-refs` 报的那几条，忘了同一被改数字在别的 wiki 页的其它出现点（扇出漏改）。 +- ❌ 把「来源没变、扫读升深读」的 partial re-ingest 和「来源内容真变了」的本 skill 混为一谈。 diff --git a/.claude/skills/kb-export/SKILL.md b/.claude/skills/kb-export/SKILL.md index 7134341..ed7ce28 100644 --- a/.claude/skills/kb-export/SKILL.md +++ b/.claude/skills/kb-export/SKILL.md @@ -83,7 +83,7 @@ type: source_summary created_date: 2026-04-28 last_modified: 2026-04-28 last_modified_by: LLM -status: reviewed +status: draft # LLM 写入一律 draft；reviewed 仅人类审阅后设（并把 last_modified_by 改 Human） confidence: high source_count: 1 sources: diff --git a/.claude/skills/kb-ingest/SKILL.md b/.claude/skills/kb-ingest/SKILL.md index d1c8a8e..8124448 100644 --- a/.claude/skills/kb-ingest/SKILL.md +++ b/.claude/skills/kb-ingest/SKILL.md @@ -40,6 +40,27 @@ python scripts/convert.py --workspace 如果只想处理某 workspace 的某个子目录，用 `--dir` 显式指定（给出时覆盖 `--workspace`）：`python scripts/convert.py --dir workspaces//raw/papers`。 +### 第 1.5 步：强制建立“全细节证据地图”（不得抽样） + +转换后立即从**全部** `raw/**/*.md` 重建派生证据索引，并核对分母： + +```bash +python scripts/k.py --workspace rebuild-evidence-index +python scripts/k.py --workspace evidence-index-coverage +``` + +两个机械闸门都必须满足： + +- `natural_units.coverage == 1.0`：paragraph、每条 list item、每条 table data row、blockquote、code、figure 全部进入索引； +- `content_sections.coverage == 1.0` 且 `structural_sections.coverage == 1.0`；`unexpected_empty_sections` 逐项确认是故意占位，否则视为转换丢正文并修复； +- `manifest.ok == true`：构建时冻结的完整单位/章节 inventory 指纹、每文档声明分母、物化表与 FTS 对账；保留结构空白的 exact text hash 也必须一致，代码缩进等变化不得被空白折叠掩盖；两张 FTS 的规范 DDL/列序/tokenizer 必须匹配，内部 `quick_check` 也必须为 `ok`，不能只看 shadow content 行； +- `corpus_freshness.ok == true`：当前 raw 文件集合/全文 SHA-256，以及真正参与路由的已验证 `agent_summary` 指纹都与索引一致；`annotate-section` 后必须重建； +- 任一分母为 0、raw 新增/删除/同长度改写、FTS 通道缺行或解析错误都不是“100%”，必须重建或修复后再继续。 + +这是**可发现性层**，与 AI 阅读深度分开：第 ③ 档可以只深读少量章节，但未深读章节的原文自然单元也必须 100% 可检索。机械索引不等于语义理解，不能把 coverage=100% 写成“摘要没有遗漏”或“问答一定正确”。 + +**分母边界**：这里的 100% 是“当前转换后 markdown 的 parser inventory 全部物化”，不是“原 PDF/DOCX 每页、每表、每脚注都转换成功”。高风险来源还要做格式专用 conversion receipt/页表计数/视觉抽检；在该层未认证前，对外只能声明“转换文本内无机械漏索引”。 + ## 第 2 步：看大纲，AI 自动决定阅读策略 ```bash @@ -53,8 +74,8 @@ python scripts/k.py outline raw/papers/.md | 档 | 字数（中文等价） | 策略 | 综合保真度 | |---|---|---|---| | ① 短文 | < 30K | 一次 Read 全文 | 高 | -| ② 中长文 | 30K – 150K（论文 / 报告 / 长文） | 按 H1 切块、每块 ≤ 30K，分段 Read，每段读完调 annotate-section | 高（多步但不漏信息） | -| ③ 整本书规模 | > 150K（专著 / 法规全文 / 长篇手册） | TOC 扫全 + AI 决定深读章节；**全部章节**登记到 source_summary 章节登记表 | 中（透明声明深度差异，保留 partial re-ingest 升级路径） | +| ② 中长文 | 30K – 150K（论文 / 报告 / 长文） | 按标题树选 ≤ 30K 的完整节 Read，超长 H1/H2 继续下钻 H3/H4，每节读完调 annotate-section | 综合层有损；全部原文细节另由机械索引保留 | +| ③ 整本书规模 | > 150K（专著 / 法规全文 / 长篇手册） | TOC 扫全 + AI 决定深读章节；**全部章节**登记到 source_summary 章节登记表 | 综合层显式分级；全部原文细节仍须 100% 可检索 | > 英文文档按 `字符数 × 0.5` 估算中文等价（英文 1 token ≈ 4 char，中文 1 token ≈ 1.5-2 char）。 > 单次 Read 严格控制在 30K 中文字符内——避免 LLM "lost in the middle" 衰减。 @@ -68,7 +89,7 @@ python scripts/k.py outline raw/papers/.md ### 第 ② 档：中长文分段读 -按 H1 章节顺序切块（必要时合并相邻短章节凑近 30K），每块独立 Read： +按标题树顺序选择不超过 30K 的完整节；H1/H2 本身超限时继续下钻 H3/H4，无子标题的超长节则用 `search-evidence → read-evidence-unit → read-block` 按自然块阅读。若**单个**自然块仍超 30K，搜索只返围绕命中词的有界摘录，精确读取 fail-closed；优先修复来源结构/转换分块，`--max-chars 0` 只能作为人工明示的无限制逃生口，不得由 agent 自动绕过： ```bash python scripts/k.py read-section raw/papers/.md ``` @@ -76,7 +97,7 @@ python scripts/k.py read-section raw/papers/.md ```bash python scripts/k.py annotate-section raw/papers/.md h-2-3-abc123 "本节论证..." ``` -最终综合判断（第 3 步）基于**全部章节摘要**，不丢信息。 +最终综合判断（第 3 步）基于已读章节和章节摘要；摘要是有损的，不得声称其“不丢信息”。问答需要摘要未收录的细节时，必须回到全量证据索引发现并核对原文。 ### 第 ③ 档：长篇文档结构化深度选读 @@ -88,13 +109,19 @@ python scripts/k.py annotate-section raw/papers/.md h-2-3-abc123 "本节 3. 对深读章节走第 ② 档流程（read-section + annotate-section） 4. **关键**：source_summary 的「## 章节深度登记」H2 节按 anchor 列出**全部章节**（详见第 5 步模板），扫读 / 跳过的章节**保留 partial re-ingest 升级路径**（详见后文「增量深化」节） -### 精确取段（任何档都可用） +**摘要路由新鲜度回收**：`annotate-section` 会改变参与结构路由的 validated summary 指纹，因此第 1.5 步建的索引会按设计变 stale。完成本次全部 annotation 后，必须**再跑一次** `rebuild-evidence-index` + `evidence-index-coverage`，确认 `outline_summary_changed=[]` 且全部闸门仍绿；不得带旧摘要路由进入后续问答。 + +### 精确取段与 quote-first 写作纪律（任何档都适用） -如果分析中发现需要精确取出某段（比如某个关键数据），调： +取出某段精确原文： ```bash python scripts/k.py read-block raw/papers/.md p-12-7d8e9a ``` +**quote-first 硬规则**：写任何含**数字 / 日期 / 金额 / 百分比 / 精确引文**的论断之前，必须先用 `read-block`（^p-/^t-）或 `read-section`（^h-）打开目标块，**从屏幕上的返回原文抄写**，anchor 也从返回内容行尾复制——**禁止凭「刚才通读时的记忆」写，禁止从 source_summary 二手转抄而不核对**。记忆漂移正是数字抄错与张冠李戴的根源；`list-cite-mismatches` 会把违规兜出来。 + +零成本主路径：②③ 档分段阅读时**「读完一节 → 立即写该节相关论断」**——此刻原文就在上下文里，与 annotate-section 回填并列为「读完即写」双动作。只有离上下文写作（第 5/6 步补数据、更新概念页）才需要回头 read-block。 + ## 第 3 步：基于 wiki 现状做综合判断 读完原文后，**不询问用户**——AI 自行综合"本文相对已有知识库提供了什么"。这一步是后续写作（5-7 步）的信息基础，**不能跳过**。 @@ -212,7 +239,7 @@ tags: |---|---|---|---| | ^h-2-1-... | 摘要 | ✓ 深读 | 已含完整 anchor 引用 | | ^h-2-2-... | 引言 | ✓ 深读 | | -| ^h-2-3-... | 方法 | ⊙ 扫读 | 仅基于 outline preview 概览，本次综合不深入 | +| ^h-2-3-... | 3 Method | ⊙ 扫读 | 仅 preview 概览；关键实体:对比学习 / hard negative / in-batch 采样（从 preview 提取） | | ^h-2-4-... | 实验 | ✓ 深读 | 含数据表 ^t-... | | ^h-2-5-... | 附录 A | × 跳过 | 元信息（参考文献清单） | @@ -220,13 +247,18 @@ tags: - ✓ **深读**：完整 read 该章节，提取了 anchor 级引用，可直接被 wiki 论断引用 - ⊙ **扫读**：仅基于 outline preview / 章节标题做概览判断，未读全文；**保留升级路径**——后续可触发 partial re-ingest 升级到深读 - × **跳过**：与 wiki 主题无关或为元信息（附录 / 致谢 / 索引），不计入价值评估，但仍登记可见，避免"消失" + +**登记表书写硬规范**（`search-raw` 的 deepen_hint 联动与 partial re-ingest 触发都依赖它）： +- **首列必须写真实 `^h-` anchor**（从 `outline` 输出复制），**原标题列必须抄原文标题**、不许意译——意译名（如把 "2 Approach" 写成 "KAG Framework"）会让内容级检索命中后无法联动回登记表，deepen 触发器直接失明 +- **⊙ 扫读行的备注必须点名本节关键实体 / 指标名**（从 outline preview 免费提取，如"关键实体:LLMFriSPG / Mutual Indexing / Logical Form Solver"）——这是扫读章节留给未来检索与人工浏览的唯一索引密度，一行备注换一整章的可发现性 ``` **引用规范**： - 优先用 anchor 形式（`#^h-...` / `#^p-...`）而非 heading 文本 - 整章/整节论证 → `^h-{level}-{seq}-{hash}` -- 关键数据/精确论断 → `^p-{seq}-{hash}` -- 不知道 anchor 时调 `python scripts/k.py find-anchor raw/papers/.md "<原文片段>"` 反查 +- 关键数据/精确论断 → `^p-{seq}-{hash}`（数字论断**必须**锚到 ^p-/^t- 级，不要用 ^h- 大节当支撑） +- 不知道 anchor 时调 `python scripts/k.py find-anchor raw/papers/.md "<原文片段>"` 反查——**返回的 preview 必须与论断核对一致才可采用**；不一致就换 snippet 重查或 read-block 确认，不要拿相邻段落的 anchor 凑数 +- 数字为跨块计算 / 单位换算所得、原文无该字面时，标 `[KB 推算: ^依据锚]`（必须带依据锚，裸 `[KB 推算]` 会被 lint 报 exempt-missing-basis） 校验： @@ -356,6 +388,50 @@ python scripts/k.py --workspace graph - `graph` 能正常输出节点 / 边统计，本次新建 / 更新的页面**出现在节点里且有边相连**；出现意外**孤立节点**说明漏了互链——回第 6 步给它补 `[[...]]`（相关概念 / 实体用标准关系类型 `SUPPORTS` / `EXTENDS` / `PART_OF` 等）。 - web 端 `/graph` 可直接渲染本图谱（节点按 type 染色、边按 link_type 染色），无需额外构建步骤。 +## 第 9.5 步：引用回验（writer/verifier 分离） + +写作者不能自证——刚写完的引用必须经**确定性核对 + 没有写作上下文的核验者**双闸复核，才允许 commit。 + +### 9.5a 确定性核对（零 token） + +```bash +python scripts/k.py --workspace list-cite-mismatches +``` + +验收（accuracy-first）：`mismatch` / `exempt-missing-basis` / `canonical-anchor-mismatch` / `canonical-target-mismatch` / `imprecise-anchor` / `unverifiable` **全部必须为空**。Ingest 是 raw 在场的权威写入环境，「无法核验」不能在这里当信息项放过；hash recovery、`./raw/...` 等非 canonical 写法必须先改成返回的精确 path/anchor。`exempted` 只允许于已标 `[KB 推算: ^依据锚]` 的紧邻**单个值**，依据锚还必须在同一核对单元实际被引。 + +```bash +python scripts/k.py --workspace check-provenance --changed +``` + +验收：**须全空**——每条新引用必须有「取回过被引块当前内容版本」的检索凭证（`read-block` / `read-section` / `blocks` / `extract-claims --with-evidence` 都会自动登记；quote-first 走对了凭证自然齐）。缺凭证 = writer 没真正打开原文（或读的是旧版本），回去 read-block 取回再引用。 + +### 9.5b 语义回验（fresh-context 子 agent） + +1. 枚举本次新增 / 修改的（论断, 引用）对： + ```bash + python scripts/k.py --workspace extract-claims --changed --json + ``` + `summary.broken > 0` 先修引用再回验。核验时必须用 `--with-evidence --max-evidence-chars 1000000000`（或等价的全量取回），claim / evidence / cloze 任一截断都是 incomplete，不得对未见内容写 `SUPPORTED`。事实型 NOTE/TIP/IMPORTANT callout 和 table data row 同样进入枚举；只有知识冲突 / 引用审计的协议外壳排除。 +2. 用 **Task 工具起 fresh-context 子 agent**（**绝不共享写作上下文**——writer 的记忆偏差正是要防的东西；也不得由 writer 自己兼任），每批 ≤ 20 对。子 agent prompt 固定为： + - 身份：「你是**对抗性**引用核验员，立场是尽力反驳，没读过原文全文、不了解写作过程。只依据给你的论断文本与被引块原文判断，禁止用自身领域知识补全证据。」（反驳式立场比中性判定显著降低附和偏差） + - 材料：每对给 `claim_text` + `read-block` / `read-section` 取回的被引块原文（可用 `extract-claims --with-evidence` 组装）。 + - 判定三问：① 关键事实（数字 / 日期 / 主体）是否出现在被引原文？② 论断是否被原文**直接支撑**（不需要额外推理）？③ 有无语义 drift（过度概括 / 加了原文没有的限定词 / 因果错置）？ + - 输出：每对 `{pair_id, verdict: SUPPORTED|PARTIAL|UNSUPPORTED|CONTRADICTED, reason 一句, fix_suggestion}`。 +3. 处置表： + +| verdict | 处置 | +|---|---| +| SUPPORTED | 通过；`cite-audit-log --mode ingest` 入账（附 `--evidence "<被引块原文子串>"`，k.py 会字面校验） | +| PARTIAL | 收窄措辞 / 补 anchor 后复验。**合法转述白名单**（判 SUPPORTED 并在 note 记明）：约数舍入（约 40% ↔ 39.2%）、跨语言日期 / 数字格式转写、已标 `[KB 推算: ^锚]` 的派生算术 | +| UNSUPPORTED | `read-block` 打开原文**按原文改写**，或 `find-anchor` 换正确块，或删掉该论断——**禁止删引用降级 `[需要来源]` 了事**（那是把错误论断洗进 wiki 的通道） | +| CONTRADICTED | 论断与自己引的证据相反 → 按原文改写；若实为「新证据 vs 既有 wiki 论断」的知识冲突 → 按 CLAUDE.md 冲突标注格式写 `[!WARNING]` 块，不覆盖 | + +4. 复验：修完重跑 `extract-claims --changed`，只对上轮非 SUPPORTED 的对再起一次小 Task。**最多 2 轮**——仍不收敛的对保留 PARTIAL、打 `#to-be-updated`、log 记账、交周检跟进；不许反复重掷骰子刷 verdict。 +5. **盲填复核（含数字的对必做）**：`extract-claims --changed --cloze` 输出挖空论断（数字→⟦N1⟧）；给核验 agent **只看「挖空论断 + 被引原文」**填空（绝不给期望值——从原理上消灭附和偏差），填回值经 `python scripts/k.py cloze-check --batch ` 机器判分（数值容差、块级 union 语义）。判分未过按 UNSUPPORTED 处置。 +6. **跨模型二审（发布质量 ingest 必做）**：`python tools/cite-audit/audit.py --workspace --changed`——外部客户端调 DeepSeek 跑「盲填 + 反驳」双通道并自动入台账（独立执行器，不依赖会话内 agent；需 `DEEPSEEK_API_KEY` 环境变量）。换模型降低同源相关盲区。exit `0` 才是全部通过，`1` = 真实语义未通过，`2` = 网络 / 截断 / quote / skipped / 台账等导致核验未完成；后两者都不得 commit。无 API 时必须换一个真正 fresh-context 核验者完成同等双通道，不得由 writer 自审替代。 +7. 验收（与提交前质量闸门同构、可对账）：本次 verifiable 对 **UNSUPPORTED = 0、CONTRADICTED = 0**，全部判定已入台账，log.md 条目含 citation-verify 对账行（见第 10 步）。事后任何人可用 `extract-claims --commit ` 复枚举对账。另有 pre-commit hook 机械兜底：staged wiki 文件带 cite 闸门项时提交直接被拒。 + ## 第 10 步：追加 log.md `Edit` `log.md`，在文件**头部**（最近的 `---` 后）插入： @@ -367,6 +443,7 @@ python scripts/k.py --workspace graph - 更新：`wiki/concepts/.md`、`wiki/concepts/.md` - 标记待更新：<5-10 个文件> - MOC：`wiki/indexes/_index.md` +- citation-verify: pairs= verifiable= SUPPORTED= rewritten= downgraded= unverifiable= - 摘要：<一两句核心收获> ``` @@ -428,11 +505,15 @@ git commit -m "ingest: <来源标题简短>" ## 完成检查清单 - [ ] `convert.py` 已对原始文件生成 `.md` + `.outline.json` +- [ ] **最后一次 `annotate-section` 之后**已重跑 `rebuild-evidence-index`；`evidence-index-coverage` 的自然单元、内容/结构章节覆盖率均为 100%，异常空章节已逐项确认/修复，且 `corpus_freshness.ok=true`、`outline_summary_changed=[]` - [ ] 中长文档（≥30K 字符，即第 ②/③ 档）通过 `outline` → `read-section` 路线分段读取，不是 Read 全文 - [ ] 关键章节已 `annotate-section` 回填精排摘要（②③ 档必经；① 档建议性、非硬性） - [ ] 摘要页 frontmatter 完整且 `validate-frontmatter` 通过 - [ ] 所有实质性论断都有 `[[raw/...#^h-...]]` 或 `[[raw/...#^p-...]]` **块级** anchor 引用，没有"裸论断"、没有"整页引用 `[[raw/X]]` 支撑论断"、没有用 heading 文本作引用——`python scripts/k.py list-bare-claims` / `list-coarse-citations` / `list-source-issues` 三者均须为空 - [ ] `python scripts/k.py list-broken-refs` 没有新增失效引用 +- [ ] `python scripts/k.py list-cite-mismatches` 的 mismatch / exempt-missing-basis / canonical-* / imprecise / unverifiable 全空；推算豁免只绑定紧邻单值 +- [ ] `python scripts/k.py check-provenance --changed` 全空（每条新引用都有当前版本的检索凭证） +- [ ] 第 9.5 步引用回验完成：全量核验包无截断 / incomplete / skipped，`extract-claims --changed` 枚举的 verifiable 对 UNSUPPORTED=0 / CONTRADICTED=0，判定已 `cite-audit-log --mode ingest` 入账，log.md 含 citation-verify 对账行 - [ ] 核心节点（2-3 个）已立即更新 - [ ] 次要节点已标记 `#to-be-updated` - [ ] MOC 索引已更新 @@ -462,7 +543,12 @@ git commit -m "ingest: <来源标题简短>" - ❌ 用 `git add -A` 或 `git add .`（可能误提交无关文件） - ❌ 跳过第 3 步直接进 5-7 步（缺了 wiki 上下文，写出来的摘要页"不知道周围有什么"，会重复造轮子或漏掉冲突） - ❌ 第 3 / 8 步去问用户「这次核心价值是什么」「该建哪个 MOC」（这两步明确改为 agent 自决；用户的修订路径是 web 端审计与冲突工作台，不是 ingest 时实时打断） -- ❌ 单次 Read 超过 30K 中文字符（"lost in the middle" 衰减；中长文必须按 H1 切块到 ≤30K 再分段读） +- ❌ 单次 Read 超过 30K 中文字符（"lost in the middle" 衰减；中长文必须沿标题树下钻到 ≤30K 的完整节，或按自然块读） +- ❌ 只给深读章节建索引、或拿“章节摘要覆盖率”冒充原文细节覆盖率——机械证据地图必须先覆盖全部 raw 自然单元；AI 摘要只是附加路由信号 - ❌ 第 ③ 档长文档把扫读章节当成「已读」用——query 时若命中扫读章节关键词，应**先看 source_summary 章节登记表确认深度**，必要时触发 partial re-ingest，不能直接拿 outline preview 当真知识 - ❌ 把扫读 / 跳过的章节从 source_summary 章节登记表中省略——所有章节必须可见，省略 = 失去升级路径 - ❌ ingest 后忘记跑 `k.py list-broken-refs` 检查新引用是否解析成功 +- ❌ 凭「刚才通读的记忆」写数字 / 日期 / 引文论断而不打开被引块核对——quote-first：写前必须 `read-block` / `read-section` 从返回原文抄写；`list-cite-mismatches` 会报 mismatch +- ❌ 第 9.5 步让 verifier 继承写作上下文、或 writer 自己兼任 verifier——核验者必须 fresh context，否则回验只是自我合理化 +- ❌ 对 UNSUPPORTED 的论断「删引用、挂 [需要来源]」过闸——那是把错误论断洗进 wiki；只能按原文改写、换正确锚点或删掉论断本身 +- ❌ 为过数字核对滥标 `[KB 推算]`（抄错说成推算）——豁免必须带依据锚，且 kb-lint 周检会抽查豁免块、豁免占比异常会被追问 diff --git a/.claude/skills/kb-lint/SKILL.md b/.claude/skills/kb-lint/SKILL.md index a5f0b67..e930cd2 100644 --- a/.claude/skills/kb-lint/SKILL.md +++ b/.claude/skills/kb-lint/SKILL.md @@ -38,9 +38,12 @@ python scripts/k.py health --json - 被引用但缺章节摘要（`unsummarized_sections_count`，被 wiki 章节引用但 outline.json 中 `agent_summary` 为 null 的章节） - 裸论断（`bare_claims_count`，含数字但无引用支撑的段落） - 索引 page_count drift（`index_count_mismatches_count`，type=index 页声明的 page_count 与 scope 实际匹配数不等） +- **引用核对（`cite_mismatches_count`：数字 / 引文与被引块不符 + [KB 推算] 无依据锚——闸门项；`cite_imprecise_count`：锚点挂偏 / 引文未逐字命中——观察项；`cite_unverifiable_count`：被引 raw 不在场——信息项；`cite_exempted_count`：[KB 推算] 豁免块数——环比异常增长要抽查）** +- **引用语义审计（`suspect_citations_count`：待人处理的 CAUTION 审计标注，扫自 markdown；`citation_pairs_count`：审计对总数；`unaudited_citations_count`：未审对数——全指标中唯一依赖 .cache 台账的，新 clone / 删缓存后回升到 verifiable 总数属预期）** - **source_count 一致性问题（`source_issues_count`，六类：count-mismatch / missing-source / analysis-undersourced / source-summary-mismatch / broken-source-link / declared-but-uncited — 详见 CLAUDE.md "source_count 字段约定" 与本文第 6d 步）** - **关系类型问题（`relation_issues_count`，`[[X|RELATION]]` 中非标准关系类型词——拼写错误 / 未在白名单，详见 CLAUDE.md "关系类型语法"）** - **Web i18n 违规（`i18n_violations_count`，web/ 下硬编码中文 UI 字符串，违反 CLAUDE.md "Web 管理台国际化方案"）** +- **长文档证据索引（`evidence_index`：status 必须 complete，natural/content/structural coverage=100%，`corpus_freshness.ok=true`；空结构章节逐项确认。missing/stale/incomplete 时先重建，不能在旧索引上做“知识库没有”判断）** ## 第 2 步：处理 `#to-be-updated` 积压 @@ -108,18 +111,33 @@ python scripts/k.py list-bare-claims --json > **目的**：把"无声的裸论断"逐步转为"有声的占位"，等真正的 raw 入库后能 grep `[需要来源]` 一次性补全。 -### 5b. fact-check 抽查 +### 5b. 引用语义审计（配额制，取代旧「随机抽 3-5 页」） -随机选 3-5 个 `wiki/sources/.md`，做事实抽查： +**先确定性分诊**（零 token）： -1. `Read` 摘要页 -2. 找一条具体数据/论断（如"准确率 95.3%"） -3. `Read` 它引用的 `[[raw/...#^anchor]]` -4. 验证引用对应的原文是否真的支持这个数字 +```bash +python scripts/k.py list-cite-mismatches --json +``` + +对每条 `mismatch` 四选一处置：① 改数字（按被引原文）② `find-anchor` 换正确锚点 ③ 确属「新旧证据打架」→ 建 `[!WARNING]` 知识更新冲突块交人裁 ④ 确属跨块推算 → 标 `[KB 推算: ^依据锚]`。`imprecise-anchor` 批量修锚点。`cite_exempted_count` 环比异常增长时抽查豁免块真伪（防「抄错说成推算」）。 + +**再做语义审计周配额**（协议细节与 CAUTION 标注格式见 CLAUDE.md「引用语义审计规范」）： + +```bash +python scripts/k.py extract-claims --unaudited-only --sample 20 --seed --json +``` -如果发现不一致： -- 手写 `> [!WARNING] 知识更新冲突` 块（详见 CLAUDE.md "冲突处理规范"），不要默默修正 -- 在周报中列出 +- `--seed` 用当周周号：同周重跑取样一致（可复现），跨周覆盖可累积——`unaudited_citations_count` 应逐周下降 +- 对每对：`read-block` / `read-section` 取回被引原文 → **fresh-context 判定**（禁止凭「ingest 时读过」的记忆）三问：关键事实在场？直接支撑？语义 drift？→ `cite-audit-log` 入账（SUPPORTED 必须附 `--evidence "<被引块原文子串>"`） +- **UNSUPPORTED / 实质事实错的 PARTIAL**：在论断块正下方追加 CAUTION 审计标注 + 页面 tags 加 `citation-suspect`——**不默默修正**，让人在工作台看到；修复建议写进标注块 +- **多来源合成论断防误伤**（`multi_source: true`，多见于 analysis / comparison 页）：单块只支撑论断的一部分是正常形态 → PARTIAL 且**不落标注**；只有「被引块与归属分句无关或矛盾」才 UNSUPPORTED。约数舍入 / 跨语言转写 / 已标 `[KB 推算: ^锚]` 的派生算术按 SUPPORTED 处理 +- 收尾对账（堵「删标注蒸发」）： + ```bash + python scripts/k.py list-suspect-citations --check-ledger + ``` + `ledger-unsupported-without-marker` 必须为空（有 = 标注被删而论断未改，恢复标注或走正规修复） +- **agent_summary 抽查**（章节精排摘要的蒸馏忠实度——它被 ingest 综合判断 / 升级判定 / query 深度判断三处消费但无自动核验）：随机抽 3 个被 wiki 引用的章节，`read-section` 对照原文核摘要是否失真；失真则 `annotate-section` 重写并在周报记一笔 +- **SUPPORTED 交叉复查**（防橡皮图章与同源盲区）：每周对已 SUPPORTED 台账记录换模型抽查——`python tools/cite-audit/audit.py --workspace --all --sample 10 --seed `（DeepSeek 盲填+反驳双通道，同 seed 可复现；需 `DEEPSEEK_API_KEY`，无 key 时以 fresh-context 子 agent 反驳式抽查替代）。复查翻案的对按 UNSUPPORTED 流程落 CAUTION ## 第 5c 步：partial re-ingest 升级候选检测 @@ -174,7 +192,7 @@ python scripts/k.py list-i18n-violations --json 扫描 `web/` 下 `.tsx` 文件，找硬编码的中文 UI 字符串（`` / `placeholder="中文"` / `aria-label="中文"` 等）。CLAUDE.md "Web 管理台国际化方案" 明令所有 UI 字符串走 `t()` / `useT()`，硬编码会让英文用户看不懂。 **对每条违规**： -1. 在 `web/lib/i18n.ts` 的 `TRANSLATIONS.zh` 与 `.en` 同时加 key + 翻译（i18n.ts 的 TS 类型会强约束两边对称） +1. 在 `web/lib/i18n.ts` 的 `TRANSLATIONS.zh` 与 `.en` 同时加 key + 翻译（注意：TS 类型只从 zh 侧派生 key、**不强约束 en 侧**，en 缺 key 会静默回退——同步靠约定 + `test_i18n_sync.py` CI 守护） 2. 把 `.tsx` 中的硬编码字符串改为： - Server component：`{t("key", locale)}`（locale 从 `getServerLocale()` 拿） - Client component：`{t("key")}`（t 来自 `useT()`） @@ -322,8 +340,10 @@ tags: ## 本周处理 - 处理 `#to-be-updated`： 条 → <列出页面> - 修复孤儿： 个 → <列出页面与处理方式> -- fact-check 通过： 条 -- fact-check 发现不一致： 条 → <已标注冲突的页面> +- cite-mismatch 修复： 条（改数字 / 换锚 / 建冲突块 / 标推算各 ） +- 引用审计： 对（SUPPORTED/PARTIAL/UNSUPPORTED = //），累计未审 对 +- 统计保证：`python scripts/k.py audit-confidence` → 95% 置信未通过率上界