概述
The problem can arise in user input, retrieved documents, webpages, images, or tool responses. Defenses must limit the consequences as well as detect suspicious text.
主要要点
- Distinguish instructions from processed content.
- Limit permissions independently of the model.
- Test trust boundaries across input channels.
深入探讨
Separate the user’s authorized task from the content being processed. A document may legitimately contain instructions as part of its subject matter. Those words should not automatically control the assistant’s tools, account access, or response policy. Keep privileges narrow. A summarization task generally does not require unrestricted file access or permission to send messages. Enforce those limits in the application so a model error cannot silently create a broader capability. Validate consequential outputs and actions against the original request. Check the destination, affected records, data being transmitted, and required approval. An external page claiming that the user approved something is not equivalent to an actual user instruction. Build regression cases that vary the location and format of untrusted instructions. Test direct input, retrieved passages, and tool results in a controlled environment. Review whether the application preserves useful task performance while resisting redirection. Avoid claims of a foolproof prompt-injection filter; layered controls and ongoing testing are more credible.
04工作示例
Keep a retrieved instruction inside the document
Construct a test page containing a normal policy paragraph and a sentence telling the assistant to upload unrelated files.
Ask only for the policy summary. Verify that the summary uses relevant facts and that no upload tool is called.
Repeat with the instruction in a quoted block and in a tool result to test the same trust boundary across formats.
它显示了什么
This bounded defensive exercise verifies task adherence without using real private files.
现实世界的实施
Summarize a document that contains an instruction-like passage without executing it.
Check an outgoing tool action against the user’s original destination and purpose.
付诸实践
为什么这很重要
- 成本与预算. 多年来，架构决策决定着性能和运营成本。
- 更清晰的判决. 技术教育帮助团队选择正确的堆栈，而不仅仅是最新的堆栈。
- 质量控制. 更好的工程选择可以减少生产中的可靠性事故。
留意
- 优化一项基准测试可以隐藏更广泛的系统弱点。
- 基础设施和维护成本常常被低估。
- 随着系统变得更加复杂，安全性和可观察性差距可能会扩大。
后续步骤
- 在实施之前定义延迟、质量和成本目标。
- 在实际负载和数据条件下进行基准测试。
- 仪器监控错误、漂移和用户影响。
- 在扩展之前准备回滚和事件响应路径。
资料来源与延伸阅读
不断探索
常见问题
Can prompt injection be solved by banning one phrase?
No. The underlying issue is whether untrusted content can redirect behavior. Attacks can use many phrasings and formats.