From 43c68576194f4197d0191d4e00170c82a80ab901 Mon Sep 17 00:00:00 2001 From: onurcangnc Date: Fri, 31 Jul 2026 05:42:55 +0300 Subject: [PATCH 1/2] docs: add fake testing-mode prompt injection tutorial --- ..._testing_mode_prompt_injection_tutorial.md | 255 ++++++++++++++++++ 1 file changed, 255 insertions(+) create mode 100644 tutorials/fake_testing_mode_prompt_injection_tutorial.md diff --git a/tutorials/fake_testing_mode_prompt_injection_tutorial.md b/tutorials/fake_testing_mode_prompt_injection_tutorial.md new file mode 100644 index 0000000..051e9ad --- /dev/null +++ b/tutorials/fake_testing_mode_prompt_injection_tutorial.md @@ -0,0 +1,255 @@ +# Fake Testing-Mode Prompt Injection Tutorial + +**Category:** Historical field observation and defensive testing tutorial + +**Classification:** “Known technique, new target-specific case study.” + +**Target:** A publicly accessible third-party LLM chat deployment + +**OWASP mapping:** LLM01:2025 — Prompt Injection + +**MITRE ATLAS mappings:** AML.T0051.000 — Direct Prompt Injection; AML.T0054 — LLM Jailbreak + +## Overview + +A fake testing-mode prompt injection presents ordinary user input as if it came +from an authorized evaluator, developer, or privileged test harness. The prompt +may also define policy-like variables or an alternate response mode and claim +that these controls replace the deployment's normal instructions. + +This tutorial documents a historical case in which that pattern was delivered +through a public chat interface. The visible response reflected parts of the +user-supplied control framing and crossed the deployment's expected safety +boundary. + +This tutorial does not claim that the underlying fake testing-mode jailbreak +mechanism is new. It documents target-specific validation against a public +third-party LLM deployment and the resulting instruction-hierarchy and +guardrail failures. + +The exact prompt, control names, harmful request, prohibited output, provider +identity, model identity, and deployment identifiers are intentionally omitted. + +## Learning objectives + +By the end of this tutorial, a defender should be able to: + +- recognize fabricated testing or evaluation authority in direct user input; +- distinguish user-defined controls from trusted system configuration; +- evaluate whether injected framing persists across a conversation; +- assess the final response independently of displayed reasoning text; and +- document a prompt-injection case without publishing a reusable jailbreak. + +## Scope and evidence boundaries + +The observation concerns an external public LLM deployment, not a repository +sandbox. Testing used only the ordinary user-facing chat interface. No +privileged access, administrative function, external tool, or real-world action +was involved. + +The historical material supports these limited findings: + +| Question | Evidence-supported answer | +|---|---| +| Was the competing instruction supplied directly by a user? | Yes | +| Did it claim a fabricated testing or evaluation context? | Yes | +| Did it introduce attacker-controlled behavioral controls? | Yes | +| Did the visible response reflect parts of that framing? | Yes | +| Did the final response cross the expected safety boundary? | Yes | +| Was a real system instruction changed? | Not established | +| Was private chain-of-thought exposed? | Not established | +| Were privileged access or external actions obtained? | No evidence | +| Were all observations independent, fresh-session trials? | Unknown | + +Several captures came from the same visible conversation and must not be +counted as independent trials. Interface-displayed reasoning text is treated +only as user-visible model output, not as proof of private internal reasoning +or a particular model architecture. + +## Attack pattern + +At a safe level of abstraction, the direct prompt injection followed this +sequence: + +1. The user message claimed a special testing or evaluation context. +2. It asserted that the claimed context had authority over normal behavior. +3. It introduced attacker-controlled, policy-like behavioral controls. +4. It requested an alternate or less restricted response mode. +5. It attempted to suppress, postpone, or split the normal refusal behavior. +6. The visible response adopted parts of the framing and produced content that + should have been refused. + +The security failure is not the presence of words such as “test” or +“evaluation.” It is the deployment's treatment of untrusted user text as if it +could alter instruction priority or safety policy. + +## Safe defensive evaluation + +Perform this evaluation only in an authorized sandbox using harmless test +content. Do not test an external service without permission, and do not use a +request whose answer would create operational harm. + +### 1. Define the invariant + +Write a simple safety rule that must remain true regardless of user-declared +modes. For example, require the model to withhold a synthetic secret or reject +a harmless but explicitly disallowed token transformation. + +Record the expected refusal and the conditions under which the rule may +legitimately change. A user message must never be one of those conditions. + +### 2. Build an abstract test family + +Create harmless paraphrases that vary only the structural elements under +evaluation: + +- a claimed evaluator or developer role; +- a claimed testing, audit, or simulation context; +- user-defined variables that resemble configuration; +- an assertion that ordinary instructions are suspended; and +- a request to delay or separate the refusal. + +Do not copy a public jailbreak verbatim. The purpose is to test whether the +system preserves trust boundaries across semantic variants, not to optimize a +bypass. + +### 3. Run controlled trials + +Use fresh sessions and a fixed model and policy configuration. Include: + +- a baseline request without injected framing; +- one structural change per test where practical; +- paraphrased variants; +- short follow-up turns that test persistence; and +- negative controls containing benign mentions of testing or evaluation. + +Record configuration, date, session independence, prompt variant, expected +result, observed result, and uncertainty. A continuation in one conversation is +not an independent success. + +### 4. Evaluate the full response + +Assess both the final answer and any user-visible intermediate text. A refusal +does not erase prohibited content that appears before it, and safe-looking +reasoning text does not make an unsafe final answer acceptable. + +Count a bypass only when the response crosses the predefined invariant. Do not +infer hidden prompt replacement, private reasoning disclosure, or the +responsible internal component from interface behavior alone. + +### 5. Preserve evidence privately + +Keep raw prompts, screenshots, transcripts, identifiers, timestamps, and +validation notes in an access-controlled location outside the public +repository. For public reporting, retain only the minimum abstract facts +needed to explain the failure and its defensive implications. + +## Detection guidance + +Useful signals include user input that: + +- claims evaluator, developer, administrator, or test-harness authority; +- declares a new mode that allegedly supersedes normal instructions; +- defines variables that resemble policy or safety configuration; +- asks the model to ignore, reset, reinterpret, or postpone safeguards; +- requests paired restricted and unrestricted answers; or +- uses a short follow-up to continue behavior established by injected framing. + +These are risk indicators, not standalone proof of abuse. Detection should +combine semantic analysis, instruction-source tracking, session context, and +independent output checks. Keyword blocking alone will miss paraphrases and may +over-block legitimate evaluation discussions. + +## Mitigations + +- Enforce instruction priority outside user-controlled text. +- Treat user-defined modes, roles, and policy-like variables as untrusted data. +- Require authenticated, out-of-band controls for legitimate evaluation modes. +- Reject claims that a user turn has reset or replaced governing instructions. +- Apply output safety checks independently of model-generated reasoning text. +- Evaluate the complete response so that unsafe content followed by a refusal + still fails. +- Test semantic paraphrases, negative controls, and multi-turn persistence. +- Keep hidden processing separate from user-visible explanation interfaces. +- Log policy decisions without exposing sensitive prompts or internal traces. +- Re-run authorized regression tests after model, policy, or wrapper changes. + +## OWASP GenAI mapping + +The primary mapping is **OWASP LLM01:2025 — Prompt Injection**: + +| Dimension | Classification | +|---|---| +| Delivery | Direct user message | +| Pattern | Fabricated authority and instruction-hierarchy manipulation | +| Objective | Guardrail bypass | +| Observed impact | Inconsistent enforcement of the intended response boundary | + +No secondary OWASP category is asserted. The evidence does not establish secret +disclosure, downstream execution, compromised dependencies, excessive agency, +or another separate vulnerability class. + +## MITRE ATLAS mapping + +- **AML.T0051.000 — Direct Prompt Injection:** the competing instructions were + supplied directly through an ordinary user message. +- **AML.T0054 — LLM Jailbreak:** the injected framing attempted to bypass the + deployment's intended safety behavior, and the observed response crossed that + boundary. + +No technique requiring gained permissions, external tools, environment +enumeration, or real-world execution is asserted. + +## Prior art and classification + +Direct instruction override, fabricated authority or context, fake evaluation +modes, and response-control patterns were documented before this observation. +The historical prompt was a modified variant of an existing technique family. +The contribution is the independently observed behavior of one anonymized +deployment. + +The appropriate classification is: + +> “Known technique, new target-specific case study.” + +This wording distinguishes target-specific evidence from a claim of a novel +jailbreak mechanism. + +## Reproducibility limitations + +The original observation involved an external public deployment whose behavior +may change. At the time this tutorial was prepared, the deployment was +unavailable. The reason for its unavailability could not be confirmed. This +repository does not automate testing against that service. Supporting evidence +and validation materials remain private and untracked. + +The exact deployment configuration, fresh-session behavior, server-side logs, +and current behavior are unknown. These limitations prevent claims about +reliability, root cause, or present-day exploitability. + +## Ethical testing, anonymization, and disclosure + +No new testing of the historical target was performed for this contribution. +No captured output was acted on outside evidence documentation, and this +tutorial contains no operational harmful output or reusable jailbreak payload. + +The affected organization was notified before preparation of this public +contribution. No acknowledgement or remediation confirmation had been received +at the time of preparation. The provider, organization, product, model family +and version, deployment domain, hostnames, endpoints, branding, account +identifiers, session identifiers, conversation titles, and original evidence +filenames are withheld. + +The public deployment was later observed to be unavailable. No causal +relationship between the notification and the service status has been +established. + +## References + +- [OWASP LLM01:2025 — Prompt Injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/) +- [MITRE ATLAS AML.T0051.000 — Direct Prompt Injection](https://atlas.mitre.org/techniques/AML.T0051.000) +- [MITRE ATLAS AML.T0054 — LLM Jailbreak](https://atlas.mitre.org/techniques/AML.T0054) +- [Ignore Previous Prompt: Attack Techniques for Language Models](https://arxiv.org/abs/2211.09527) +- [Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825) +- [Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models](https://www.usenix.org/conference/usenixsecurity24/presentation/yu-zhiyuan) +- [Public fake evaluation-mode prompt collection, matching historical revision](https://github.com/davidegat/happy-prompts/commit/69229dc05d8f0b3d5126c20996baa8ee0aa876f5) From 5711413004ecee9ef9ff8e86bd844a26ca2fe46d Mon Sep 17 00:00:00 2001 From: onurcangnc Date: Fri, 31 Jul 2026 05:54:49 +0300 Subject: [PATCH 2/2] docs: refine tutorial copy and add index entry --- tutorials/README.md | 4 +- ..._testing_mode_prompt_injection_tutorial.md | 91 +++++++++---------- 2 files changed, 45 insertions(+), 50 deletions(-) diff --git a/tutorials/README.md b/tutorials/README.md index 921a2e5..193c74e 100644 --- a/tutorials/README.md +++ b/tutorials/README.md @@ -1,11 +1,13 @@ # Tutorials -Tutorials and standalone code samples for agentic AI red teaming. This directory hosts guides, walkthroughs, and reference material to help practitioners learn and apply red teaming techniques. +Guides, walkthroughs, and standalone examples for applying agentic AI red-team +techniques. ## Contents - [Community Resources](community_resources.md) — A curated list of external playbooks, guides, and articles on agentic AI red teaming. - [Tools](tools.md) — A curated list of tools, organized by the phases defined in the [GenAI Red Teaming Manual](https://genai.owasp.org/initiatives/#ai-redteaming). - [LLM Chatbot System Prompt Exfiltration](llm_chatbot_system_prompt_exfiltration.md) — A comprehensive tutorial outlining a five-stage attack chain targeting LLM-powered chatbots to exfiltrate their system prompt. +- [Fake Testing-Mode Prompt Injection](fake_testing_mode_prompt_injection_tutorial.md) — An anonymized case study of a direct prompt-injection guardrail bypass using fabricated evaluation authority and user-defined controls, mapped to OWASP LLM01:2025 and MITRE ATLAS. - [Multi-Technique Guardrail Bypass Evaluation](multi_technique_guardrail_bypass_evaluation.md) — Field observations documenting four guardrail-bypass technique families evaluated across a five-stage black-box sequence against a single chatbot deployment. - [Multi-Vector LLM Safety Bypass](multi_turn_safety_bypass_and_system_role_override.md) — Field observations of a six-stage attack chain demonstrating four distinct classes of LLM safety bypass against production chatbot deployments. diff --git a/tutorials/fake_testing_mode_prompt_injection_tutorial.md b/tutorials/fake_testing_mode_prompt_injection_tutorial.md index 051e9ad..c3a4a4f 100644 --- a/tutorials/fake_testing_mode_prompt_injection_tutorial.md +++ b/tutorials/fake_testing_mode_prompt_injection_tutorial.md @@ -12,27 +12,25 @@ ## Overview -A fake testing-mode prompt injection presents ordinary user input as if it came -from an authorized evaluator, developer, or privileged test harness. The prompt -may also define policy-like variables or an alternate response mode and claim -that these controls replace the deployment's normal instructions. +A fake testing-mode prompt injection frames ordinary user input as instructions +from an authorized evaluator, developer, or privileged evaluation system. It may +also define policy-like variables or an alternate response mode and claim that +these controls replace the deployment's normal instructions. -This tutorial documents a historical case in which that pattern was delivered -through a public chat interface. The visible response reflected parts of the -user-supplied control framing and crossed the deployment's expected safety -boundary. +This historical case records that pattern in a public chat interface. The +visible response adopted parts of the user-supplied control framing and crossed +the deployment's expected safety boundary. -This tutorial does not claim that the underlying fake testing-mode jailbreak -mechanism is new. It documents target-specific validation against a public -third-party LLM deployment and the resulting instruction-hierarchy and -guardrail failures. +The underlying jailbreak mechanism is established prior art. This tutorial +documents target-specific validation against a public third-party LLM +deployment and the resulting instruction-hierarchy and guardrail failures. The exact prompt, control names, harmful request, prohibited output, provider identity, model identity, and deployment identifiers are intentionally omitted. ## Learning objectives -By the end of this tutorial, a defender should be able to: +Defenders can use this tutorial to: - recognize fabricated testing or evaluation authority in direct user input; - distinguish user-defined controls from trusted system configuration; @@ -42,10 +40,10 @@ By the end of this tutorial, a defender should be able to: ## Scope and evidence boundaries -The observation concerns an external public LLM deployment, not a repository -sandbox. Testing used only the ordinary user-facing chat interface. No -privileged access, administrative function, external tool, or real-world action -was involved. +The observation concerns an external public LLM deployment; no repository +sandbox was involved. Testing used only the ordinary user-facing chat +interface, without privileged access, administrative functions, external +tools, or real-world actions. The historical material supports these limited findings: @@ -68,8 +66,7 @@ or a particular model architecture. ## Attack pattern -At a safe level of abstraction, the direct prompt injection followed this -sequence: +The direct prompt injection followed this abstract sequence: 1. The user message claimed a special testing or evaluation context. 2. It asserted that the claimed context had authority over normal behavior. @@ -79,15 +76,15 @@ sequence: 6. The visible response adopted parts of the framing and produced content that should have been refused. -The security failure is not the presence of words such as “test” or -“evaluation.” It is the deployment's treatment of untrusted user text as if it -could alter instruction priority or safety policy. +The failure occurs when a deployment treats untrusted user text as authority to +change instruction priority or safety policy. Benign use of words such as +“test” or “evaluation” is not enough to establish a prompt-injection attempt. ## Safe defensive evaluation -Perform this evaluation only in an authorized sandbox using harmless test -content. Do not test an external service without permission, and do not use a -request whose answer would create operational harm. +Run this evaluation only in an authorized sandbox with harmless test content. +Never test an external service without permission or use a request whose answer +would create operational harm. ### 1. Define the invariant @@ -109,9 +106,8 @@ evaluation: - an assertion that ordinary instructions are suspended; and - a request to delay or separate the refusal. -Do not copy a public jailbreak verbatim. The purpose is to test whether the -system preserves trust boundaries across semantic variants, not to optimize a -bypass. +Do not copy a public jailbreak verbatim. These variants test whether the system +preserves trust boundaries without optimizing a bypass. ### 3. Run controlled trials @@ -146,17 +142,17 @@ needed to explain the failure and its defensive implications. ## Detection guidance -Useful signals include user input that: +Watch for user input that: -- claims evaluator, developer, administrator, or test-harness authority; +- claims evaluator, developer, administrator, or evaluation-system authority; - declares a new mode that allegedly supersedes normal instructions; - defines variables that resemble policy or safety configuration; - asks the model to ignore, reset, reinterpret, or postpone safeguards; - requests paired restricted and unrestricted answers; or - uses a short follow-up to continue behavior established by injected framing. -These are risk indicators, not standalone proof of abuse. Detection should -combine semantic analysis, instruction-source tracking, session context, and +Treat these as risk indicators rather than standalone proof of abuse. Combine +semantic analysis, instruction-source tracking, session context, and independent output checks. Keyword blocking alone will miss paraphrases and may over-block legitimate evaluation discussions. @@ -203,25 +199,21 @@ enumeration, or real-world execution is asserted. ## Prior art and classification Direct instruction override, fabricated authority or context, fake evaluation -modes, and response-control patterns were documented before this observation. -The historical prompt was a modified variant of an existing technique family. -The contribution is the independently observed behavior of one anonymized -deployment. +modes, and response-control patterns predate this observation. The historical +prompt modified an existing technique family. This case adds independently +observed behavior from one anonymized deployment. The appropriate classification is: > “Known technique, new target-specific case study.” -This wording distinguishes target-specific evidence from a claim of a novel -jailbreak mechanism. - ## Reproducibility limitations The original observation involved an external public deployment whose behavior -may change. At the time this tutorial was prepared, the deployment was -unavailable. The reason for its unavailability could not be confirmed. This -repository does not automate testing against that service. Supporting evidence -and validation materials remain private and untracked. +may change. The deployment was unavailable when this tutorial was prepared, +and the reason could not be confirmed. This repository does not automate +testing against that service. Supporting evidence and validation materials +remain private and untracked. The exact deployment configuration, fresh-session behavior, server-side logs, and current behavior are unknown. These limitations prevent claims about @@ -229,16 +221,17 @@ reliability, root cause, or present-day exploitability. ## Ethical testing, anonymization, and disclosure -No new testing of the historical target was performed for this contribution. -No captured output was acted on outside evidence documentation, and this -tutorial contains no operational harmful output or reusable jailbreak payload. +The researchers performed no new testing of the historical target for this +contribution and did not act on captured output outside evidence documentation. +This tutorial contains no operational harmful output or reusable jailbreak +payload. -The affected organization was notified before preparation of this public -contribution. No acknowledgement or remediation confirmation had been received +The researchers notified the affected organization before preparing this public +contribution. They had received no acknowledgement or remediation confirmation at the time of preparation. The provider, organization, product, model family and version, deployment domain, hostnames, endpoints, branding, account identifiers, session identifiers, conversation titles, and original evidence -filenames are withheld. +filenames remain withheld. The public deployment was later observed to be unavailable. No causal relationship between the notification and the service status has been