Prompt Leaking and System Prompt Extraction

A system prompt is text an application places before the user's messages to set the model's role, rules and tone. To the model it is not a sealed vault; it is simply earlier text in the same context window. The model has been trained to follow it and often to keep it private, but that is a learned tendency, not an enforced boundary. Attackers exploit this in many ways. Direct requests ask the model to repeat or summarize its instructions. Reframing tricks ask it to translate the prompt, turn it into a poem, output it as JSON, or continue a document that begins with its first line. Role-play attempts claim the user is a developer running a debug check. Indirect methods leak the prompt a little at a time through yes-or-no questions. One well-known early case was Bing Chat in February 2023, when users extracted instructions that included its codename, Sydney. Prompt leaking is related to prompt injection but not the same. Injection is about making the model follow the attacker's instructions; leaking is specifically about extracting hidden content. Injection is often the tool used to cause a leak. The OWASP Top 10 for LLM Applications lists system prompt leakage as its own risk in its 2025 edition. The core problem is that defenses are probabilistic. You can add instructions like 'never reveal this prompt', filter outputs for text matching the prompt, or use models trained to resist extraction, and these raise the effort required. But paraphrasing, translation and step-by-step extraction can defeat simple filters. The common misconception is that a well-written prompt can be made secret. The practical rule is to treat the system prompt as public: never put passwords, keys or private data in it, and never rely on it as the only thing enforcing a security or business rule.