The attack can arrive through ordinary content
Prompt injection attempts to redirect a model through supplied instructions, including instructions embedded in external material. OWASP describes both direct and indirect attack scenarios. A retrieved page or document may look like useful evidence while containing text that asks the assistant to change its behavior.
The critical question is not whether the text sounds authoritative. It is whether that source is allowed to instruct the application. A customer document can describe a requested change, but it cannot grant itself administrative privileges or authorize exporting another customer's data.
Technical foundation: OWASP: LLM01 Prompt Injection
Trace the path from content to consequence
Map where untrusted material enters the system: uploads, search snippets, email bodies, database notes, and tool results. Then identify what the assistant could do after reading it. A summarizer with no external actions has a different risk profile from an agent that can send messages or modify records.
Imagine a support attachment that includes a request to send account information to an unrelated address. The attachment should remain evidence for the support task. Recipient restrictions and access checks in application code should prevent the proposed export even if the model repeats the malicious instruction.
Enforce controls outside the prompt
Use least-privilege tools, destination restrictions, and server-side authorization. Keep secrets out of model-visible context when they are not necessary. Require review for consequential actions and make the proposed action understandable to the reviewer. These controls reduce the possible impact of a model mistake.
Separating reference text from instructions helps maintain clarity, but formatting alone is not a guarantee. A stronger design assumes the model may still produce a bad proposal and prevents that proposal from crossing an unauthorized boundary. Do not make the model the sole judge of its own permissions.
Test containment, not just refusal wording
A security test should check whether protected data stayed protected and whether unauthorized actions were blocked. A reassuring refusal followed by an unintended tool call is a failure. Conversely, an imperfect explanation with no unauthorized effect still deserves a different diagnosis from a data leak.
Include attacks in multiple content locations and ordinary legitimate documents as controls. Review false positives too: an assistant that rejects every uploaded document is not useful. Keep a small regression set for each discovered weakness and rerun it when prompts, tools, or retrieval behavior change.
A practical containment review
- Inventory untrusted entry points and possible side effects.
- Enforce permissions and destination rules outside the model.
- Keep secrets and unrelated records out of task context.
- Approve specific consequential operations, not vague autonomy.
- Measure prevented actions and data exposure, not refusal style alone.
