Skip to content

feat(security): prompt-injection guard (rag-injection) + red-team probe gate (Step 7.3) - #166

Merged
officialCodeWork merged 1 commit into
mainfrom
build/phase-7/step-7.3-red-team
Jun 8, 2026
Merged

feat(security): prompt-injection guard (rag-injection) + red-team probe gate (Step 7.3)#166
officialCodeWork merged 1 commit into
mainfrom
build/phase-7/step-7.3-red-team

Conversation

@officialCodeWork

Copy link
Copy Markdown
Owner

Summary

Step 7.3 — Red-team / security testing: turns the governance stack into an adversarial probe gate across four classes (prompt injection / PII egress / ACL bypass / tenant escape) and closes the verified prompt-injection gap.

  • redteam lane + first-class CI gate. All four classes run under a redteam marker (tests/redteam/, auto-marked by a directory-scoped conftest.py) via task redteam and a dedicated redteam-gate CI job (alongside perf-gate). ACL bypass + tenant escape were already gated; this adds prompt injection + PII egress.
  • The injection gap was real. trust_level was stored + filterable everywhere but nothing enforced it at the LLM adapter — and the OpenAI-chat surface assembled retrieved context as a role="system" message (an attacker-controlled system-trust position, which PROBLEM-TRACEABILITY forbids).
  • New rag-injection package. A pluggable InjectionDetector (dependency-free HeuristicInjectionDetector — regexes anchored on the grammar of an attack, so benign prose mentioning "system"/"prompt"/"instructions" isn't flagged) + PromptInjectionGuard.inspect that drops hijack chunks before the LLM, paired with INJECTION_RESISTANT_SYSTEM_PROMPT + build_user_message so untrusted context is fenced data in the user turn, never a system turn. Wired on every answer surface (/v1/query, /v1/chat/completions, MCP); off by default (cfg.injection); PII-free injection.blocked event; degrade-open.
  • PII-egress probe over the Step 6.5 PiiPolicyEngine: block denies, redact/mask strip every span (verified by an independent second detector), over answer-text + list[Chunk] shapes.

Injection types are package-local + internal — no attacker signal on the wire, and dist/schemas + dist/openapi are untouched.

Acceptance (PROBLEM-TRACEABILITY)

A corpus of 568 known + 630 generated payloads (eval/redteam_v0/) measured against the guard:

Metric Target Measured
Block rate ≥ 95 % 96.6 %
False-positive rate (benign prose) ≤ 5 % 0 %
user_supplied chunk in a system-trust position never never (end-to-end, both surfaces)

Building the corpus hardened the detector — it exposed ~20 missed phrasings (e.g. "ignore your previous instructions"), which is the point of red-teaming. The corpus is kept broader than the detector's patterns (carriers, morphology, evasive payloads) so the rate is honest, not a circular 100 %.

Documentation

  • docs/adr/ADR-0045-red-team-security.md — the decision (injection guard, trust isolation, the gate, alternatives).
  • docs/guides/red-team.md — runbook for the four probe classes + the external-pentest process item.
  • docs/reference/injection.md — the rag-injection API.
  • docs/README.md index rows; CLAUDE.md package table + dependency graph; TRACKER.md → Phase 7 3/10 (77/84), next 7.4.

Test plan

  • task redteam / pytest tests/redteam/ -m redteam — 46 probes green (incl. injection + PII)
  • pytest packages/injection/tests/ — 12 package-local unit tests
  • Full unit sweep -m "not perf and not redteam" green; perf-gate green
  • ruff, ruff format, mypy --strict (329 files), RAG001, policy-coverage, event/log-schema, schema-drift (config regen) + openapi-drift (no change)
  • caught + fixed a conftest bug that would have marked the whole suite redteam

🤖 Generated with Claude Code

…be gate (Step 7.3)

Turn the governance stack into an adversarial probe gate across four classes
(prompt injection / PII egress / ACL bypass / tenant escape) under a `redteam`
marker + a first-class `redteam-gate` CI job. ACL + tenant-escape were already
gated; this adds prompt injection + PII egress and consolidates them.

Close the verified injection gap: trust_level was stored everywhere but nothing
enforced it at the LLM adapter, and the OpenAI-chat surface assembled retrieved
context as a `role="system"` message (an attacker-controlled system-trust
position). New rag-injection package: a pluggable InjectionDetector (heuristic
default, attack-grammar regexes) + PromptInjectionGuard.inspect that drops
hijack chunks before the LLM, paired with INJECTION_RESISTANT_SYSTEM_PROMPT +
build_user_message so untrusted context is fenced data in the user turn, never a
system turn. Wired on /v1/query + /v1/chat/completions + MCP; off by default
(cfg.injection); PII-free injection.blocked event.

Deterministic gate: a 568-known + 630-generated corpus (eval/redteam_v0/) hits
96.6% block (>=95% bar) at 0 false positives, plus the no-system-position
invariant end-to-end. Building the corpus hardened the detector (~20 missed
phrasings). New PII-egress probe over PiiPolicyEngine (zero leakage,
second-detector verified). pip-audit is the CVE gate; external pentest is a
documented process item. Injection types stay internal (no attacker signal;
dist/schemas + openapi untouched).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@officialCodeWork
officialCodeWork merged commit 5a540fe into main Jun 8, 2026
13 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant