Daaf dev

.claude/agents/README.md

@@ -589,7 +589,7 @@ code-reviewer returns BLOCKER
 - Breadth-first exploration (scan broadly, then drill into best candidates)
 - Evidence-based findings with mandatory source citations (file:line or URL)
 - Flexible output format adapted to the task (inventories, direct answers, surveys)
-- Web-capable research via WebSearch and WebFetch (unique among read-only agents)
+- Web-capable research via WebSearch (discovery) plus the DAAF fetch protocol (`web-retrieval` skill's `web_fetch.sh`; the built-in WebFetch is blocked) — unique among read-only agents
 - Skill-aware domain knowledge (loads relevant DAAF skills for authoritative context)
 
 **Key distinction from source-researcher:** The source-researcher examines a single known data source in depth using an existing DAAF skill, producing a fixed five-section deliverable. The search-agent explores broadly across any information space with flexible output. If you already know which data source skill to investigate, use source-researcher.

.claude/agents/data-ingest.md

@@ -5,7 +5,7 @@ description: >
   Statistical, Relational, Interpretation), producing detailed findings that feed into
   skill authoring. Invoked by the orchestrator once per profiling part during Data
   Onboarding Mode.
-tools: [Read, Write, Edit, Bash, Glob, Grep, WebSearch, WebFetch, Skill]
+tools: [Read, Write, Edit, Bash, Glob, Grep, WebSearch, Skill]
 skills: data-scientist
 permissionMode: default
 model: opus   # High-judgment tier: interpretive profiling seeds long-lived data-source skills (override per-dispatch allowed)
@@ -63,7 +63,7 @@ You are a **Data Ingest Specialist** -- an agent that performs exhaustive, part-
 | Canonical load pattern | Orchestrator Agent prompt | Conditional | Reuse exact load parameters from script 01 (provided for Parts B/C/D) |
 | File size | Orchestrator Agent prompt | No | Context for sampling decisions and performance expectations |
 | Documentation files | Orchestrator Agent prompt | No | Cross-reference against actual data in Part D |
-| Documentation website URL | Orchestrator Agent prompt | No | Fetch additional context via WebFetch in Part D |
+| Documentation website URL | Orchestrator Agent prompt | No | Fetch additional context via the DAAF fetch protocol (`web_fetch.sh`) in Part D |
 | Priority columns | Orchestrator Agent prompt | No | Columns requiring deeper examination |
 | Access method | Orchestrator Agent prompt | No | "local_file" (default) or "api" — if API, DI-0 provides acquisition details |
 | Acquisition script path | Orchestrator Agent prompt | Conditional | Path to DI-0 acquisition script, if data was fetched via API (for provenance) |
@@ -244,7 +244,7 @@ When invoked, check the `profiling_part` parameter and execute the corresponding
 
 **Execute sequentially:**
 
-1. **Research the API:** Use WebFetch to read API documentation. Use WebSearch if documentation URL is not provided. Identify: base URL, available endpoints, authentication method (query param, header, bearer), response format, pagination method, rate limits.
+1. **Research the API:** Use WebSearch to find the documentation URL if none is provided, then retrieve the docs page with the DAAF fetch protocol — `bash /daaf/scripts/web_fetch.sh <URL> <DEST_DIR>` — and read the `.md` extract (the built-in WebFetch is blocked; see the `web-retrieval` skill for size rules and the untrusted-content rule). Identify: base URL, available endpoints, authentication method (query param, header, bearer), response format, pagination method, rate limits.
 
 2. **Write acquisition script:** Write to `{project_script_dir}/stage5_fetch/00_api-fetch.py` (R: `00_api-fetch.R`)
    - Check `os.environ["{env_var_name}"]` with clear `KeyError` message if missing (R: `Sys.getenv("{env_var_name}")` + `stop()` with a clear message when empty)
@@ -435,7 +435,7 @@ Categories: Access | Data | Method | Perf | Process
 | **Data** | Quality, suppression, distributions | "12% of columns had coded missing as -9 (undocumented)" |
 | **Method** | Methodology edge cases | "FIPS codes stored as float caused join failures" |
 | **Perf** | Performance, memory, runtime | "1.2GB parquet needed chunked profiling" |
-| **Process** | Execution patterns, error patterns | "WebFetch rate-limited after 5 codebook page fetches" |
+| **Process** | Execution patterns, error patterns | "web_fetch.sh rate-limited after 5 codebook page fetches" |
 
 ### Recommendations
 - **Proceed?** [YES -- part complete | NO -- issues block this part | NO -- escalate]
@@ -635,5 +635,6 @@ Load on demand -- do NOT read all at start:
 | `agent_reference/STATE_TEMPLATE_ONBOARDING.md` | When reading or updating STATE.md | Expected STATE.md structure for Data Onboarding projects |
 | `agent_reference/SCRIPT_EXECUTION_REFERENCE.md` | Before writing first script | File-first execution protocol and capture utilities |
 | `agent_reference/INLINE_AUDIT_TRAIL.md` | When writing scripts with transforms | IAT documentation standards |
+| `web-retrieval` skill | When retrieving an API docs page or codebook page to read (Part D-0 API research) | DAAF fetch protocol (`web_fetch.sh`): tool syntax, provenance model, size rules, untrusted-content rule (the built-in WebFetch is blocked) |
 
 **Cross-language annotation skills** are documented in § 4c (Language-Conditional Skill Loading).

.claude/agents/debugger.md

@@ -5,7 +5,7 @@ description: >
   hypothesis-testing methodology. Invoked by orchestrator when errors occur
   during pipeline execution or when code-reviewer identifies complex issues
   requiring root-cause analysis.
-tools: [Read, Write, Edit, Bash, Glob, Grep, Skill, WebFetch, WebSearch]
+tools: [Read, Write, Edit, Bash, Glob, Grep, Skill, WebSearch]
 skills: data-scientist
 permissionMode: default
 model: opus   # High-judgment tier: root-cause hypothesis testing (override per-dispatch allowed)
@@ -622,5 +622,6 @@ Load on demand -- do NOT read all at start:
 | `agent_reference/SCRIPT_EXECUTION_REFERENCE.md` | Before writing first diagnostic script | File-first execution protocol and debug script format |
 | `agent_reference/INLINE_AUDIT_TRAIL.md` | When adding comments to diagnostic code | IAT documentation standards |
 | `agent_reference/ERROR_RECOVERY.md` | When error matches a known recovery pattern | Error type decision trees |
+| `web-retrieval` skill | When retrieving library docs or an issue thread to read during diagnosis | DAAF fetch protocol: use WebSearch to discover, then `bash /daaf/scripts/web_fetch.sh <URL> <DEST_DIR>` to retrieve (the built-in WebFetch is blocked); read the `.md` extract, mind the size rules and untrusted-content rule |
 
 **Language-conditional and cross-language annotation skills** are documented in § 7 (Language-Conditional Skill Loading).

.claude/agents/research-synthesizer.md

@@ -48,7 +48,7 @@ Your mindset is that of a senior analyst conducting a literature review: you eva
 
 | Input | Source | Required | How Used |
 |-------|--------|----------|----------|
-| Stage 2 findings | Domain explorer subagent (e.g., education-data-explorer) | Yes | Baseline: endpoints, variables, coverage, completeness assessment |
+| Stage 2 findings | Domain explorer subagent (e.g., education-data-explorer) | Yes | Baseline: datasets, variables, coverage, completeness assessment |
 | Stage 3 findings (all sources) | source-researcher subagents | Yes | Per-source caveats, coded values, suppression, pitfalls |
 | Research question | Orchestrator Agent prompt | Yes | Anchor for relevance filtering and recommendation framing |
 | Geographic scope | Orchestrator Agent prompt | Yes | Determines cross-state comparability requirements |
@@ -59,10 +59,10 @@ Your mindset is that of a senior analyst conducting a literature review: you eva
 | Section | How You Use It |
 |---------|----------------|
 | `Recommended Data Level` | Primary data level for the analysis |
-| `Candidate Endpoints` table | Data sources available for the research question |
+| `Candidate Datasets` table | Data sources available for the research question |
 | `Key Variables` table | Variables to query, their sources and coverage |
 | `Variables Flagged for Deep-Dive` | Items needing source-specific investigation |
-| `Completeness Assessment` | Confidence in endpoint discovery |
+| `Completeness Assessment` | Confidence in dataset discovery |
 
 **Stage 3 Findings Detail (per source):**
 
@@ -245,7 +245,7 @@ Return synthesis in this structure:
 ## Input Manifest
 | Stage | Source/Skill | Status | Key Findings Count |
 |-------|-------------|--------|-------------------|
-| 2 | [domain explorer skill, e.g., education-data-explorer] | RECEIVED | [N] endpoints, [N] variables |
+| 2 | [domain explorer skill, e.g., education-data-explorer] | RECEIVED | [N] datasets, [N] variables |
 | 3a | [domain source skill, e.g., education-data-source-[name]] | RECEIVED | [N] caveats, [N] coded values |
 | 3b | [domain source skill, e.g., education-data-source-[name]] | RECEIVED | [N] caveats, [N] coded values |
 

.claude/agents/search-agent.md

@@ -5,7 +5,7 @@ description: >
   sources to locate specific information. Invoked by the orchestrator in place
   of generic Plan or Explore subagent types when targeted or broad search is
   needed during any mode or pipeline stage.
-tools: [Read, Bash, Glob, Grep, Skill, WebSearch, WebFetch]
+tools: [Read, Bash, Glob, Grep, Skill, WebSearch]
 permissionMode: plan
 model: sonnet   # Well-specified tier: broad read-only exploration (override per-dispatch allowed)
 ---
@@ -35,7 +35,7 @@ You are comfortable operating across very different contexts: research data expl
 | **Output** | Flexible findings report tailored to the task | Fixed five-section source report (SOURCE_SUMMARY through PITFALLS) | Fixed coverage matrix + structured YAML issues |
 | **Timing** | Any stage, any mode — replaces generic Plan subagent dispatches | Stage 3 (per source) or on-demand deep lookups | Stage 4.5 (after plan creation) |
 | **Scope per invocation** | One or many topics, as needed | Exactly one data source | Exactly one plan document pair |
-| **Web access** | Yes (WebSearch, WebFetch) | No | No |
+| **Web access** | Yes (WebSearch + the DAAF fetch protocol) | No | No |
 
 **Key distinction from source-researcher:** The source-researcher examines a single known data source in depth using an existing DAAF skill, producing a fixed five-section deliverable. The search-agent explores broadly across any information space with flexible output. If you already know which data source skill to investigate, use source-researcher. If you need to survey, discover, or explore across topics, use search-agent.
 
@@ -88,7 +88,14 @@ Unlike specialized agents with fixed deliverable contracts, your output format a
 
 ### 4. Web-Capable Research
 
-You have WebSearch and WebFetch — capabilities no other read-only DAAF agent has. Use them when:
+You have web access — capabilities no other read-only DAAF agent has. Discovery
+uses the built-in **WebSearch** (finding candidate URLs); retrieving a document
+to *read or quote* uses the **DAAF fetch protocol**, `bash
+/daaf/scripts/web_fetch.sh <URL> <DEST_DIR>`, which saves the raw response, a
+deterministic `.md` extract, and a provenance manifest to disk. The built-in
+`WebFetch` tool is blocked (it returns an AI paraphrase, not source text); load
+the `web-retrieval` skill for the full protocol (size rules, exit-code failure
+table, untrusted-content rule, destination conventions). Use web research when:
 - The search prompt asks about external documentation, APIs, or methodologies
 - Local codebase and skills don't contain the needed information
 - The orchestrator explicitly requests web research
@@ -108,7 +115,7 @@ You operate in a subagent context window, not the main orchestrator context. Be
 
 When the search involves a domain covered by DAAF skills (data sources, statistical methods, visualization libraries), load the relevant skill for authoritative context. Skills provide curated knowledge that is more reliable than ad-hoc web searches for framework conventions and structural guidance. The orchestrator may specify which skills to load, or you may identify the right skill based on the search topic.
 
-However, skills' factual claims (URLs, endpoints, variable names, coded values, API parameters) are point-in-time snapshots that can drift. When a skill's `skill-last-updated` frontmatter is more than a few months old, or when skill-sourced details produce unexpected results during a search, cross-reference against authoritative online sources using WebSearch/WebFetch. When reporting findings that extend beyond what a skill explicitly states — filling in details from general knowledge rather than curated content — clearly mark these as "inferred from general knowledge — not from curated skill content" and consider verifying via web search before presenting them as findings.
+However, skills' factual claims (URLs, endpoints, variable names, coded values, API parameters) are point-in-time snapshots that can drift. When a skill's `skill-last-updated` frontmatter is more than a few months old, or when skill-sourced details produce unexpected results during a search, cross-reference against authoritative online sources using WebSearch to find the source and the DAAF fetch protocol (`web-retrieval` skill) to retrieve it. When reporting findings that extend beyond what a skill explicitly states — filling in details from general knowledge rather than curated content — clearly mark these as "inferred from general knowledge — not from curated skill content" and consider verifying via web search before presenting them as findings.
 
 ---
 
@@ -133,7 +140,7 @@ If the request is ambiguous about any of these, make reasonable assumptions and
 | **Code tracing** | Grep for function/class names, Read call sites | Grep, Read |
 | **Framework survey** | Glob for component files, Read structure of each | Glob, Read |
 | **Domain knowledge** | Load relevant skill, extract needed information | Skill, Read |
-| **External documentation** | WebSearch for topic, WebFetch best results | WebSearch, WebFetch |
+| **External documentation** | WebSearch for topic, fetch best results via `web_fetch.sh` | WebSearch, Bash (`web_fetch.sh`) |
 | **Cross-reference audit** | Grep for name/path, verify each reference resolves | Grep, Glob, Read |
 | **Mixed** | Combine strategies as needed | All available |
 
@@ -145,7 +152,7 @@ Execute the chosen strategy iteratively:
 3. Read the most relevant files/sections in full
 4. If initial results are insufficient, refine search terms and repeat
 5. Load skills if domain knowledge is needed
-6. Use WebSearch/WebFetch if local sources are insufficient
+6. Use WebSearch to discover and the DAAF fetch protocol (`web-retrieval` skill) to retrieve, if local sources are insufficient
 
 ### Step 4: Synthesize and Rank Findings
 
@@ -415,4 +422,5 @@ Load on demand — do NOT read all at start:
 |------|-------------|---------|
 | `agent_reference/WORKFLOW_PHASE1_DISCOVERY.md` | When performing Stage 2 data exploration | Discovery protocol specifics and skill invocation patterns |
 | `agent_reference/BOUNDARIES.md` | When encountering scope boundary questions | Deviation rules and boundary specifications |
+| `web-retrieval` skill | When retrieving a web document to read or quote | DAAF fetch protocol (`web_fetch.sh`): tool syntax, provenance model, size rules, failure table, untrusted-content rule |
 | Any `*-data-source-*` skill | When searching involves a specific data domain | Authoritative domain knowledge for the search topic |

.claude/hooks/block-webfetch.sh

@@ -0,0 +1,78 @@
+#!/usr/bin/env bash
+# block-webfetch.sh
+# ---------------------------------------------------------------------------
+# PURPOSE
+#   PreToolUse hook (matcher: WebFetch) that DENIES every WebFetch call and
+#   redirects the agent to DAAF's provenance-preserving fetch utility.
+#
+#   Claude Code's built-in WebFetch is architecturally a summarizer, not a
+#   content retriever: fetched pages are paraphrased by a small model before
+#   the calling agent sees anything (community-reverse-engineered: Haiku
+#   pipeline, <=125-char quote cap, 100KB truncation). "Quotes" from WebFetch
+#   may therefore be paraphrase — which conflicts directly with DAAF's
+#   evidence-graded quoting doctrine (CLAUDE.md § Execution Philosophy). Its
+#   failures are also semi-silent and misleading (Cloudflare-gated preflight
+#   failures misattributed to the user's network; hard domain blocks). DAAF
+#   replaces it with `scripts/web_fetch.sh`, which writes the raw response
+#   plus a deterministic extract and a provenance manifest to disk.
+#
+# WHY A HOOK AND NOT A permissions.deny RULE (recorded decision, 2026-08-08)
+#   Enforcement is an instructive deny hook ONLY. A `permissions.deny` WebFetch
+#   entry would fire BEFORE this hook and suppress the instructional redirect
+#   message below, defeating its teaching purpose. The redirect message is the
+#   higher-value guarantee, so the hook is the single enforcement layer. This
+#   is a deliberate departure from the usual defense-in-depth pattern.
+#
+# FAIL-CLOSED RATIONALE
+#   This is a doctrine/boundary guard: WebFetch must never reach the network,
+#   because its output cannot satisfy DAAF's quoting standard. Unlike the
+#   fail-OPEN dispatch guards (block-nested-dispatch.sh et al.), any unexpected
+#   error here DENIES — the ERR trap emits a deny decision. Blocking a fetch on
+#   error is cheap (the agent retries via web_fetch.sh); allowing an
+#   unattributable paraphrase through is the costly outcome.
+#
+# INPUT   JSON on stdin (standard PreToolUse hook contract).
+# OUTPUT  Deny: JSON permissionDecision=deny (the convention for tool-call
+#         hooks, per enforce-explore-model.sh / deny-claude-code-guide.sh).
+#
+# DEPLOYMENT (human-only — .claude/hooks/ is deny-protected)
+#   1. Rebuild the container FIRST (adds trafilatura; web_fetch.sh needs it for
+#      full extraction — it degrades to raw-only with a loud warning without).
+#   2. Copy this file to .claude/hooks/block-webfetch.sh, chmod +x it.
+#   3. Register it in settings.json under a "WebFetch" PreToolUse matcher
+#      (see settings-registration-snippet.json).
+#   See deploy/DEPLOYMENT.md for the full ordered procedure.
+# ---------------------------------------------------------------------------
+
+# -u catches unset-variable typos; -o pipefail surfaces pipeline failures.
+# Deliberately omit -e: this hook emits its own decision and must not abort
+# into an unintended state. The ERR trap below fails CLOSED (deny).
+set -uo pipefail
+
+# Fail-closed ERR trap: any unexpected error denies the WebFetch call.
+trap 'jq -n "{\"hookSpecificOutput\":{\"hookEventName\":\"PreToolUse\",\"permissionDecision\":\"deny\",\"permissionDecisionReason\":\"block-webfetch hook encountered an unexpected error; denying WebFetch (fail-closed).\"}}" 2>/dev/null; exit 0' ERR
+
+# Drain stdin so the harness pipe closes cleanly (contents are not needed —
+# this hook denies unconditionally for the WebFetch matcher).
+INPUT=$(cat)
+: "${INPUT:=}"
+
+read -r -d '' REASON <<'MSG' || true
+WebFetch is disabled in DAAF: it returns an AI paraphrase of the page (not the source text) and fails opaquely, which conflicts with DAAF's evidence-graded quoting doctrine. Use the DAAF fetch protocol instead:
+
+  bash /daaf/scripts/web_fetch.sh <URL> <dest-dir>
+
+It saves the raw response, a deterministic extract, and a provenance manifest (URL, timestamp, HTTP status, SHA-256s) under the destination directory. Read the .md extract; grep the raw .html when the extract looks thin. See the `web-retrieval` skill for the full protocol, failure-handling table, and size rules.
+
+Note: web_fetch.sh requires the rebuilt container image (it adds trafilatura); pre-rebuild it degrades to a raw-only mode with a loud warning. WebSearch remains available for discovery.
+MSG
+
+jq -n --arg r "$REASON" '{
+  "hookSpecificOutput": {
+    "hookEventName": "PreToolUse",
+    "permissionDecision": "deny",
+    "permissionDecisionReason": $r
+  }
+}'
+
+exit 0