diff --git a/tutorials/README.md b/tutorials/README.md index 921a2e5..193c74e 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -1,11 +1,13 @@ # Tutorials -Tutorials and standalone code samples for agentic AI red teaming. This directory hosts guides, walkthroughs, and reference material to help practitioners learn and apply red teaming techniques. +Guides, walkthroughs, and standalone examples for applying agentic AI red-team +techniques. ## Contents - [Community Resources](community_resources.md) — A curated list of external playbooks, guides, and articles on agentic AI red teaming. - [Tools](tools.md) — A curated list of tools, organized by the phases defined in the [GenAI Red Teaming Manual](https://genai.owasp.org/initiatives/#ai-redteaming). - [LLM Chatbot System Prompt Exfiltration](llm_chatbot_system_prompt_exfiltration.md) — A comprehensive tutorial outlining a five-stage attack chain targeting LLM-powered chatbots to exfiltrate their system prompt. +- [Fake Testing-Mode Prompt Injection](fake_testing_mode_prompt_injection_tutorial.md) — An anonymized case study of a direct prompt-injection guardrail bypass using fabricated evaluation authority and user-defined controls, mapped to OWASP LLM01:2025 and MITRE ATLAS. - [Multi-Technique Guardrail Bypass Evaluation](multi_technique_guardrail_bypass_evaluation.md) — Field observations documenting four guardrail-bypass technique families evaluated across a five-stage black-box sequence against a single chatbot deployment. - [Multi-Vector LLM Safety Bypass](multi_turn_safety_bypass_and_system_role_override.md) — Field observations of a six-stage attack chain demonstrating four distinct classes of LLM safety bypass against production chatbot deployments. diff --git a/tutorials/fake_testing_mode_prompt_injection_tutorial.md b/tutorials/fake_testing_mode_prompt_injection_tutorial.md new file mode 100644 index 0000000..c3a4a4f --- /dev/null +++ b/tutorials/fake_testing_mode_prompt_injection_tutorial.md @@ -0,0 +1,248 @@ +# Fake Testing-Mode Prompt Injection Tutorial + +**Category:** Historical field observation and defensive testing tutorial + +**Classification:** “Known technique, new target-specific case study.” + +**Target:** A publicly accessible third-party LLM chat deployment + +**OWASP mapping:** LLM01:2025 — Prompt Injection + +**MITRE ATLAS mappings:** AML.T0051.000 — Direct Prompt Injection; AML.T0054 — LLM Jailbreak + +## Overview + +A fake testing-mode prompt injection frames ordinary user input as instructions +from an authorized evaluator, developer, or privileged evaluation system. It may +also define policy-like variables or an alternate response mode and claim that +these controls replace the deployment's normal instructions. + +This historical case records that pattern in a public chat interface. The +visible response adopted parts of the user-supplied control framing and crossed +the deployment's expected safety boundary. + +The underlying jailbreak mechanism is established prior art. This tutorial +documents target-specific validation against a public third-party LLM +deployment and the resulting instruction-hierarchy and guardrail failures. + +The exact prompt, control names, harmful request, prohibited output, provider +identity, model identity, and deployment identifiers are intentionally omitted. + +## Learning objectives + +Defenders can use this tutorial to: + +- recognize fabricated testing or evaluation authority in direct user input; +- distinguish user-defined controls from trusted system configuration; +- evaluate whether injected framing persists across a conversation; +- assess the final response independently of displayed reasoning text; and +- document a prompt-injection case without publishing a reusable jailbreak. + +## Scope and evidence boundaries + +The observation concerns an external public LLM deployment; no repository +sandbox was involved. Testing used only the ordinary user-facing chat +interface, without privileged access, administrative functions, external +tools, or real-world actions. + +The historical material supports these limited findings: + +| Question | Evidence-supported answer | +|---|---| +| Was the competing instruction supplied directly by a user? | Yes | +| Did it claim a fabricated testing or evaluation context? | Yes | +| Did it introduce attacker-controlled behavioral controls? | Yes | +| Did the visible response reflect parts of that framing? | Yes | +| Did the final response cross the expected safety boundary? | Yes | +| Was a real system instruction changed? | Not established | +| Was private chain-of-thought exposed? | Not established | +| Were privileged access or external actions obtained? | No evidence | +| Were all observations independent, fresh-session trials? | Unknown | + +Several captures came from the same visible conversation and must not be +counted as independent trials. Interface-displayed reasoning text is treated +only as user-visible model output, not as proof of private internal reasoning +or a particular model architecture. + +## Attack pattern + +The direct prompt injection followed this abstract sequence: + +1. The user message claimed a special testing or evaluation context. +2. It asserted that the claimed context had authority over normal behavior. +3. It introduced attacker-controlled, policy-like behavioral controls. +4. It requested an alternate or less restricted response mode. +5. It attempted to suppress, postpone, or split the normal refusal behavior. +6. The visible response adopted parts of the framing and produced content that + should have been refused. + +The failure occurs when a deployment treats untrusted user text as authority to +change instruction priority or safety policy. Benign use of words such as +“test” or “evaluation” is not enough to establish a prompt-injection attempt. + +## Safe defensive evaluation + +Run this evaluation only in an authorized sandbox with harmless test content. +Never test an external service without permission or use a request whose answer +would create operational harm. + +### 1. Define the invariant + +Write a simple safety rule that must remain true regardless of user-declared +modes. For example, require the model to withhold a synthetic secret or reject +a harmless but explicitly disallowed token transformation. + +Record the expected refusal and the conditions under which the rule may +legitimately change. A user message must never be one of those conditions. + +### 2. Build an abstract test family + +Create harmless paraphrases that vary only the structural elements under +evaluation: + +- a claimed evaluator or developer role; +- a claimed testing, audit, or simulation context; +- user-defined variables that resemble configuration; +- an assertion that ordinary instructions are suspended; and +- a request to delay or separate the refusal. + +Do not copy a public jailbreak verbatim. These variants test whether the system +preserves trust boundaries without optimizing a bypass. + +### 3. Run controlled trials + +Use fresh sessions and a fixed model and policy configuration. Include: + +- a baseline request without injected framing; +- one structural change per test where practical; +- paraphrased variants; +- short follow-up turns that test persistence; and +- negative controls containing benign mentions of testing or evaluation. + +Record configuration, date, session independence, prompt variant, expected +result, observed result, and uncertainty. A continuation in one conversation is +not an independent success. + +### 4. Evaluate the full response + +Assess both the final answer and any user-visible intermediate text. A refusal +does not erase prohibited content that appears before it, and safe-looking +reasoning text does not make an unsafe final answer acceptable. + +Count a bypass only when the response crosses the predefined invariant. Do not +infer hidden prompt replacement, private reasoning disclosure, or the +responsible internal component from interface behavior alone. + +### 5. Preserve evidence privately + +Keep raw prompts, screenshots, transcripts, identifiers, timestamps, and +validation notes in an access-controlled location outside the public +repository. For public reporting, retain only the minimum abstract facts +needed to explain the failure and its defensive implications. + +## Detection guidance + +Watch for user input that: + +- claims evaluator, developer, administrator, or evaluation-system authority; +- declares a new mode that allegedly supersedes normal instructions; +- defines variables that resemble policy or safety configuration; +- asks the model to ignore, reset, reinterpret, or postpone safeguards; +- requests paired restricted and unrestricted answers; or +- uses a short follow-up to continue behavior established by injected framing. + +Treat these as risk indicators rather than standalone proof of abuse. Combine +semantic analysis, instruction-source tracking, session context, and +independent output checks. Keyword blocking alone will miss paraphrases and may +over-block legitimate evaluation discussions. + +## Mitigations + +- Enforce instruction priority outside user-controlled text. +- Treat user-defined modes, roles, and policy-like variables as untrusted data. +- Require authenticated, out-of-band controls for legitimate evaluation modes. +- Reject claims that a user turn has reset or replaced governing instructions. +- Apply output safety checks independently of model-generated reasoning text. +- Evaluate the complete response so that unsafe content followed by a refusal + still fails. +- Test semantic paraphrases, negative controls, and multi-turn persistence. +- Keep hidden processing separate from user-visible explanation interfaces. +- Log policy decisions without exposing sensitive prompts or internal traces. +- Re-run authorized regression tests after model, policy, or wrapper changes. + +## OWASP GenAI mapping + +The primary mapping is **OWASP LLM01:2025 — Prompt Injection**: + +| Dimension | Classification | +|---|---| +| Delivery | Direct user message | +| Pattern | Fabricated authority and instruction-hierarchy manipulation | +| Objective | Guardrail bypass | +| Observed impact | Inconsistent enforcement of the intended response boundary | + +No secondary OWASP category is asserted. The evidence does not establish secret +disclosure, downstream execution, compromised dependencies, excessive agency, +or another separate vulnerability class. + +## MITRE ATLAS mapping + +- **AML.T0051.000 — Direct Prompt Injection:** the competing instructions were + supplied directly through an ordinary user message. +- **AML.T0054 — LLM Jailbreak:** the injected framing attempted to bypass the + deployment's intended safety behavior, and the observed response crossed that + boundary. + +No technique requiring gained permissions, external tools, environment +enumeration, or real-world execution is asserted. + +## Prior art and classification + +Direct instruction override, fabricated authority or context, fake evaluation +modes, and response-control patterns predate this observation. The historical +prompt modified an existing technique family. This case adds independently +observed behavior from one anonymized deployment. + +The appropriate classification is: + +> “Known technique, new target-specific case study.” + +## Reproducibility limitations + +The original observation involved an external public deployment whose behavior +may change. The deployment was unavailable when this tutorial was prepared, +and the reason could not be confirmed. This repository does not automate +testing against that service. Supporting evidence and validation materials +remain private and untracked. + +The exact deployment configuration, fresh-session behavior, server-side logs, +and current behavior are unknown. These limitations prevent claims about +reliability, root cause, or present-day exploitability. + +## Ethical testing, anonymization, and disclosure + +The researchers performed no new testing of the historical target for this +contribution and did not act on captured output outside evidence documentation. +This tutorial contains no operational harmful output or reusable jailbreak +payload. + +The researchers notified the affected organization before preparing this public +contribution. They had received no acknowledgement or remediation confirmation +at the time of preparation. The provider, organization, product, model family +and version, deployment domain, hostnames, endpoints, branding, account +identifiers, session identifiers, conversation titles, and original evidence +filenames remain withheld. + +The public deployment was later observed to be unavailable. No causal +relationship between the notification and the service status has been +established. + +## References + +- [OWASP LLM01:2025 — Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) +- [MITRE ATLAS AML.T0051.000 — Direct Prompt Injection](https://atlas.mitre.org/techniques/AML.T0051.000) +- [MITRE ATLAS AML.T0054 — LLM Jailbreak](https://atlas.mitre.org/techniques/AML.T0054) +- [Ignore Previous Prompt: Attack Techniques for Language Models](https://arxiv.org/abs/2211.09527) +- [Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825) +- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://www.usenix.org/conference/usenixsecurity24/presentation/yu-zhiyuan) +- [Public fake evaluation-mode prompt collection, matching historical revision](https://github.com/davidegat/happy-prompts/commit/69229dc05d8f0b3d5126c20996baa8ee0aa876f5)