Architecture, intent, invariants, and ADR documentation suite

README.md

@@ -76,8 +76,18 @@ Details: [docs/deploy.md](docs/deploy.md), [docs/hardening-decisions.md](docs/ha
 
 ## Docs
 
+Start at **[docs/README.md](docs/README.md)** (read order: intent → architecture → invariants → data-model → deploy).
+
 | Doc | Contents |
 |---|---|
+| [docs/intent.md](docs/intent.md) | Problem, thesis, users, non-goals |
+| [docs/architecture.md](docs/architecture.md) | Processes, trust boundaries, planes, sequences |
+| [docs/invariants.md](docs/invariants.md) | Numbered guarantees + enforcing code |
+| [docs/data-model.md](docs/data-model.md) | Glossary, tables, relationships |
+| [docs/spec-map.md](docs/spec-map.md) | Spec section → implementation |
+| [docs/threat-model.md](docs/threat-model.md) | Assets, actors, residual risk |
+| [docs/runbooks.md](docs/runbooks.md) | Operator procedures |
+| [docs/adrs/](docs/adrs/) | Architecture decision records |
 | [docs/deploy.md](docs/deploy.md) | Env vars, auth policy, first hunt, gVisor, reverse proxy |
 | [docs/openapi.json](docs/openapi.json) | HTTP API (`GET /api/v1/openapi.json`) |
 | [docs/hardening-decisions.md](docs/hardening-decisions.md) | mTLS, spool crypto, sandbox research |

crates/corpus-core/src/lib.rs

@@ -26,6 +26,13 @@
 //!
 //! [`ENGINE_VERSION`] is folded into rule-bundle digests so engine upgrades
 //! invalidate prior scan caches (spec 14 / 15.4).
+//!
+//! # Design docs (repo `docs/`)
+//!
+//! Product intent, system architecture, numbered invariants, data model,
+//! ADRs, and operator runbooks live under `docs/` (see `docs/README.md`).
+//! Module rustdoc explains local behavior; those pages explain cross-cutting
+//! why.
 
 pub mod agents;
 pub mod analyst;

docs/README.md

@@ -0,0 +1,21 @@
+# Corpus documentation
+
+| Doc | Purpose |
+|-----|---------|
+| [intent.md](intent.md) | Problem, thesis, users, non-goals |
+| [architecture.md](architecture.md) | Processes, trust boundaries, data/control/analysis planes |
+| [invariants.md](invariants.md) | Numbered guarantees the code must not violate |
+| [data-model.md](data-model.md) | Glossary, tables, relationships |
+| [spec-map.md](spec-map.md) | Product-spec section → code location |
+| [threat-model.md](threat-model.md) | Assets, attackers, residual risk |
+| [runbooks.md](runbooks.md) | Operator procedures for common failures |
+| [adrs/](adrs/) | Architecture decision records |
+| [deploy.md](deploy.md) | Env vars, auth policy, first hunt |
+| [semantic-similarity-design.md](semantic-similarity-design.md) | Function-level matching design |
+| [hardening-decisions.md](hardening-decisions.md) | mTLS, spool crypto, scanner tiers (research notes) |
+| [detonation-design.md](detonation-design.md) | CAPE adapter design |
+| [openapi.json](openapi.json) | HTTP surface (`GET /api/v1/openapi.json`) |
+
+Read order for a new engineer: **intent → architecture → invariants → data-model → deploy**.
+
+Decision history: start at [adrs/README.md](adrs/README.md). Milestone research notes (hardening, semantic, detonation) stay as standalone design docs; ADRs capture the cross-cutting choices in short form.

docs/adrs/0001-two-listeners.md

@@ -0,0 +1,34 @@
+# ADR-0001: Separate admin and agent listeners
+
+## Status
+
+Accepted (M6).
+
+## Context
+
+Agent authentication requires mTLS with a deployment-only CA. Admin/CLI
+traffic uses bearer tokens and often runs on loopback without TLS in dev.
+axum-server did not expose peer certificates cleanly for per-route TLS
+policy (see hardening notes).
+
+## Decision
+
+Run **two listeners**:
+
+- Plain HTTP(S) admin/CLI on `CORPUS_LISTEN`
+- mTLS agent listener on `CORPUS_AGENT_LISTEN` with a hand-built
+  `tokio_rustls::TlsAcceptor`
+
+Enrollment (one-time token) remains on the plain listener as the
+documented bootstrap path; renew uses mTLS.
+
+## Consequences
+
+- Route tables are duplicated for agent ingest/heartbeat/gaps.
+- Ops must open/monitor two ports.
+- Clear trust split: agent identity ≠ admin token.
+
+## References
+
+- [hardening-decisions.md](../hardening-decisions.md) §1
+- `corpus-server` dual bind in `main.rs`

docs/adrs/0002-server-owned-writes.md

@@ -0,0 +1,27 @@
+# ADR-0002: Server owns all durable writes
+
+## Status
+
+Accepted (M0).
+
+## Context
+
+Endpoints are untrusted. If agents could invent artifact ids or write
+shared storage, multi-tenant integrity and rehash guarantees collapse.
+
+## Decision
+
+Only `corpus-server` writes Postgres catalog/ledger and CAS objects.
+Agents and `corpusctl` are HTTP clients. Agents may write **local** SQLite
+and spool only.
+
+## Consequences
+
+- All validation (rehash, classify, policy) centralizes on the server.
+- Offline agent work is queue-and-forward, not peer-to-peer CAS.
+- Offline import still goes through announce/finalize.
+
+## References
+
+- [architecture.md](../architecture.md)
+- [invariants.md](../invariants.md) §1, §9

docs/adrs/0003-immutable-rule-bundles.md

@@ -0,0 +1,30 @@
+# ADR-0003: Immutable content-addressed rule bundles
+
+## Status
+
+Accepted (M0 / M3).
+
+## Context
+
+Retro-hunts must name the exact rule set and engine that produced a match
+months later. Mutable “current rules” pointers make historical results
+unverifiable.
+
+## Decision
+
+- Individual rules are compile-validated and stored.
+- A **bundle** freezes sorted sources + `COMPILER_CONFIG` + engine version
+  into a digest.
+- Activation is a pointer for forward coverage; digests never rewrite.
+- Scan cache keys include bundle digest and engine version.
+
+## Consequences
+
+- Rule edits require a new bundle publish.
+- Engine upgrades invalidate cache entries by construction.
+- Hunts pin digests, not “latest”.
+
+## References
+
+- `corpus_core::rules`, `registry`
+- [invariants.md](../invariants.md) §6–7

docs/adrs/0004-typed-edges-not-fuzzy-families.md

@@ -0,0 +1,27 @@
+# ADR-0004: Typed edges; weak never merge groups
+
+## Status
+
+Accepted (M3a); reaffirmed for semantic weak edges.
+
+## Context
+
+ssdeep and weak semantic scores produce useful **leads** but high false
+family membership if they auto-cluster.
+
+## Decision
+
+- Persist **typed** edges with explicit `edge_type` and `model_version`.
+- Only strong types merge variant groups (`merges_groups`).
+- `byte_similar`, `shared_provenance`, `semantic_variant_weak` never merge.
+
+## Consequences
+
+- Analyst UIs must show weak edges as leads.
+- Group membership stays high-precision.
+- Spec 28.5 encoded as a unit test.
+
+## References
+
+- `similarity::model::merges_groups`
+- [invariants.md](../invariants.md) §3

docs/adrs/0005-pure-rust-semantic.md

@@ -0,0 +1,31 @@
+# ADR-0005: Pure-Rust semantic matching (no Ghidra)
+
+## Status
+
+Accepted (M8).
+
+## Context
+
+Spec 16.2/16.5 describes Ghidra BSim-class capability. Embedding a JVM
+Ghidra service adds ops weight and Windows CI friction.
+
+## Decision
+
+Implement function-level matching in-process with **iced-x86** + goblin:
+
+- x86-64 only for v1
+- Heuristic + symbol/pdata boundaries
+- Mnemonic-family tokens + Jaccard; simhash retained for future indexing
+
+Document honest limits (no decompiler CFG, no AArch64 yet, uncalibrated τ).
+
+## Consequences
+
+- Same process as server (resource bounds required).
+- AArch64 and calibration tracked as follow-ups.
+- Design doc thresholds locked to `MODEL_V1` via CI test.
+
+## References
+
+- [semantic-similarity-design.md](../semantic-similarity-design.md)
+- `semantic::*`

docs/adrs/0006-one-to-one-function-matching.md

@@ -0,0 +1,27 @@
+# ADR-0006: One-to-one greedy function matching
+
+## Status
+
+Accepted (similarity investigation foundation).
+
+## Context
+
+Many-to-one assignment lets one popular CRT-like function inflate coverage
+for many peers (false strong edges).
+
+## Decision
+
+- Candidate pairs with Jaccard ≥ τ sorted by score, then offsets.
+- Greedy assignment: each function used at most once.
+- Strong edges require coverage floors **and** min matched pair count.
+
+## Consequences
+
+- Slightly lower scores vs many-to-one inflation (documented in PR notes).
+- Contested/unmatched sets available for explainability.
+- Deterministic ties for stable receipts.
+
+## References
+
+- `semantic::edges::coverage`
+- [semantic-similarity-design.md](../semantic-similarity-design.md)

docs/adrs/0007-filesystem-cas-trait.md

@@ -0,0 +1,28 @@
+# ADR-0007: Filesystem CAS + CasBackend trait
+
+## Status
+
+Accepted (M0 filesystem; trait in investigation foundation).
+
+## Context
+
+Need put-if-absent object storage without requiring S3 for single-node
+homelab/dev. Future object stores should not rewrite ingest.
+
+## Decision
+
+- Default `FsCas` under `CORPUS_CAS_ROOT` (`objects/`, `staging/`).
+- `CasBackend` trait for stage/commit/read/delete.
+- `MemoryCas` + `conformance_suite` for tests.
+- Digest verification remains in ingest (caller), not inside `commit`.
+
+## Consequences
+
+- Ops is directory backup + permissions.
+- S3/MinIO can implement the trait later without changing announce flow.
+- No built-in CAS GC yet (tracked separately).
+
+## References
+
+- `corpus_core::cas`
+- [architecture.md](../architecture.md) storage section

docs/adrs/0008-observe-only-agent.md

@@ -0,0 +1,30 @@
+# ADR-0008: Observe-only agent
+
+## Status
+
+Accepted (M1; product hard rule).
+
+## Context
+
+A control plane that can run arbitrary commands on endpoints is an RCE
+product. IR value must not require that surface.
+
+## Decision
+
+`corpus-agent`:
+
+- Discovers and uploads code-bearing files
+- Reports heartbeats and gaps
+- **Never** receives server-side command payloads for execution
+- **Never** blocks process start
+
+## Consequences
+
+- No remote response actions (kill process, quarantine) in this tree
+- Capture failure modes become gap rows, not silent drops
+- Security review focuses on steal-credentials and DoS, not command inject
+
+## References
+
+- [intent.md](../intent.md) non-goals
+- [invariants.md](../invariants.md) §9–10

docs/adrs/0009-tiered-scan-isolation.md

@@ -0,0 +1,34 @@
+# ADR-0009: Tiered scan isolation
+
+## Status
+
+Accepted (M6).
+
+## Context
+
+YARA on hostile bytes can attack the scanner. Full microVM isolation is
+heavy for default single-node deploy. Need a ladder of controls.
+
+## Decision
+
+Tiers via `CORPUS_SCANNER_TIER`:
+
+| Tier | Mechanism |
+|------|-----------|
+| `inprocess` | Dev only; same process as API |
+| `subprocess` (default) | `corpus-scanner` + seatbelt/landlock where available |
+| `gvisor` | `runsc` when configured |
+
+`CORPUS_MIN_SCANNER_TIER` refuses weaker tiers at startup.
+
+## Consequences
+
+- Default is stronger than in-process, weaker than Kata.
+- Docs must not claim microVM isolation for default installs.
+- Operator can raise the floor on multi-tenant hosts.
+
+## References
+
+- [hardening-decisions.md](../hardening-decisions.md) §3
+- [invariants.md](../invariants.md) §14
+- [threat-model.md](../threat-model.md)

docs/adrs/0010-content-derived-receipts.md

@@ -0,0 +1,30 @@
+# ADR-0010: Content-derived analysis receipts
+
+## Status
+
+Accepted (similarity investigation foundation).
+
+## Context
+
+Analysts need “what analyzer/model saw this artifact?” without re-reading
+samples. Random UUID receipts make concurrent re-analysis noisy and hard
+to upsert.
+
+## Decision
+
+- `AnalysisReceipt` JSON without sample bytes
+- `receipt_id` = truncated SHA-256 over tenant, artifact, analyzer,
+  versions, config digest, input sha256, status, function_count
+- Upsert on id for idempotent concurrent runs
+
+## Consequences
+
+- Same inputs → same receipt row
+- Different function counts → different ids (history preserved)
+- Edge evidence can embed `receipt_id` for join
+
+## References
+
+- `similarity::receipts`
+- [invariants.md](../invariants.md) §19
+- migration `0010_receipts_and_cleanup.sql`

docs/adrs/README.md

@@ -0,0 +1,21 @@
+# Architecture decision records
+
+Short, durable decisions. Milestone research writeups stay in parent docs;
+ADRs capture the choice and consequences.
+
+| ADR | Title | Status |
+|-----|-------|--------|
+| [0001](0001-two-listeners.md) | Separate admin and agent listeners | Accepted |
+| [0002](0002-server-owned-writes.md) | Server owns all durable writes | Accepted |
+| [0003](0003-immutable-rule-bundles.md) | Immutable content-addressed rule bundles | Accepted |
+| [0004](0004-typed-edges-not-fuzzy-families.md) | Typed edges; weak never merge groups | Accepted |
+| [0005](0005-pure-rust-semantic.md) | Pure-Rust semantic matching (no Ghidra) | Accepted |
+| [0006](0006-one-to-one-function-matching.md) | One-to-one greedy function matching | Accepted |
+| [0007](0007-filesystem-cas-trait.md) | Filesystem CAS + CasBackend trait | Accepted |
+| [0008](0008-observe-only-agent.md) | Observe-only agent | Accepted |
+| [0009](0009-tiered-scan-isolation.md) | Tiered scan isolation | Accepted |
+| [0010](0010-content-derived-receipts.md) | Content-derived analysis receipts | Accepted |
+
+## Format
+
+Each ADR: **Context → Decision → Consequences → References**.