feat(security): prompt-injection guard (rag-injection) + red-team probe gate (Step 7.3) - #166
Merged
Merged
Conversation
…be gate (Step 7.3) Turn the governance stack into an adversarial probe gate across four classes (prompt injection / PII egress / ACL bypass / tenant escape) under a `redteam` marker + a first-class `redteam-gate` CI job. ACL + tenant-escape were already gated; this adds prompt injection + PII egress and consolidates them. Close the verified injection gap: trust_level was stored everywhere but nothing enforced it at the LLM adapter, and the OpenAI-chat surface assembled retrieved context as a `role="system"` message (an attacker-controlled system-trust position). New rag-injection package: a pluggable InjectionDetector (heuristic default, attack-grammar regexes) + PromptInjectionGuard.inspect that drops hijack chunks before the LLM, paired with INJECTION_RESISTANT_SYSTEM_PROMPT + build_user_message so untrusted context is fenced data in the user turn, never a system turn. Wired on /v1/query + /v1/chat/completions + MCP; off by default (cfg.injection); PII-free injection.blocked event. Deterministic gate: a 568-known + 630-generated corpus (eval/redteam_v0/) hits 96.6% block (>=95% bar) at 0 false positives, plus the no-system-position invariant end-to-end. Building the corpus hardened the detector (~20 missed phrasings). New PII-egress probe over PiiPolicyEngine (zero leakage, second-detector verified). pip-audit is the CVE gate; external pentest is a documented process item. Injection types stay internal (no attacker signal; dist/schemas + openapi untouched). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Step 7.3 — Red-team / security testing: turns the governance stack into an adversarial probe gate across four classes (prompt injection / PII egress / ACL bypass / tenant escape) and closes the verified prompt-injection gap.
redteamlane + first-class CI gate. All four classes run under aredteammarker (tests/redteam/, auto-marked by a directory-scopedconftest.py) viatask redteamand a dedicatedredteam-gateCI job (alongsideperf-gate). ACL bypass + tenant escape were already gated; this adds prompt injection + PII egress.trust_levelwas stored + filterable everywhere but nothing enforced it at the LLM adapter — and the OpenAI-chat surface assembled retrieved context as arole="system"message (an attacker-controlled system-trust position, which PROBLEM-TRACEABILITY forbids).rag-injectionpackage. A pluggableInjectionDetector(dependency-freeHeuristicInjectionDetector— regexes anchored on the grammar of an attack, so benign prose mentioning "system"/"prompt"/"instructions" isn't flagged) +PromptInjectionGuard.inspectthat drops hijack chunks before the LLM, paired withINJECTION_RESISTANT_SYSTEM_PROMPT+build_user_messageso untrusted context is fenced data in the user turn, never a system turn. Wired on every answer surface (/v1/query,/v1/chat/completions, MCP); off by default (cfg.injection); PII-freeinjection.blockedevent; degrade-open.PiiPolicyEngine:blockdenies,redact/maskstrip every span (verified by an independent second detector), over answer-text +list[Chunk]shapes.Injection types are package-local + internal — no attacker signal on the wire, and
dist/schemas+dist/openapiare untouched.Acceptance (PROBLEM-TRACEABILITY)
A corpus of 568 known + 630 generated payloads (
eval/redteam_v0/) measured against the guard:user_suppliedchunk in a system-trust positionBuilding the corpus hardened the detector — it exposed ~20 missed phrasings (e.g. "ignore your previous instructions"), which is the point of red-teaming. The corpus is kept broader than the detector's patterns (carriers, morphology, evasive payloads) so the rate is honest, not a circular 100 %.
Documentation
docs/adr/ADR-0045-red-team-security.md— the decision (injection guard, trust isolation, the gate, alternatives).docs/guides/red-team.md— runbook for the four probe classes + the external-pentest process item.docs/reference/injection.md— therag-injectionAPI.docs/README.mdindex rows;CLAUDE.mdpackage table + dependency graph;TRACKER.md→ Phase 7 3/10 (77/84), next 7.4.Test plan
task redteam/pytest tests/redteam/ -m redteam— 46 probes green (incl. injection + PII)pytest packages/injection/tests/— 12 package-local unit tests-m "not perf and not redteam"green;perf-gategreenredteam🤖 Generated with Claude Code