Researchers use explainable AI to surgically break LLM safety filters

What happened

Researchers from the University of Pavia and Cochin University of Science and Technology developed XBreaking, a method that uses explainable AI to locate specific transformer layers responsible for safety alignment in open-source LLMs. By injecting calibrated noise into the layer preceding these safety-critical components, the team successfully suppressed refusal mechanisms in models like Llama, Qwen, Gemma, and Mistral while preserving general capabilities. The study, published in Neural Computing and Applications, reports attack success rates exceeding 90% for several models and demonstrates that layer fingerprints identified in small models can be transferred to production-scale systems like Llama 3.1 70B.

A team of computer scientists at the University of Pavia, collaborating with a colleague at Cochin University of Science and Technology, developed a new attack technique named XBreaking. Unlike traditional jailbreaking methods that rely on trial-and-error prompt crafting, XBreaking uses explainable AI tools to map the internal structure of transformer models. The researchers identified specific layers where safety alignment is concentrated and injected calibrated noise into the preceding layer to suppress the refusal mechanism.

The study, published in Neural Computing and Applications, utilized a white-box threat model where the adversary has access to model internals. The team compared censored and uncensored versions of models from the LLaMA, Qwen, Gemma, and Mistral families. They found that safety behavior is localized in specific layers, with fingerprinting accuracy exceeding 90% for most models. For instance, Llama models required identifying four to eight layers, while Mistral 7B required only one.

The attack involved perturbing the weight vector of the layer normalization module following the self-attention mechanism. This approach allowed harmful signals to propagate without globally disrupting attention or degrading generation quality. Evaluation on the JBB-Behaviors dataset showed optimal-balance attack success rates of 95.35% for Llama 3.2 1B, 96% for Llama 3.1 8B, and 90.24% for Gemma 2B. Mistral 7B was the most resistant, with a success rate of 35.56%.

Crucially, the modified models retained substantial benign functionality, with MMLU agreement rates above 68% for Llama models and 92% for Mistral. The researchers also demonstrated that layer fingerprints generalize across benchmarks and can be transferred to larger models. When proportionally mapped onto Llama 3.1 70B and Qwen 2.5 72B, the attack achieved success rates of up to 77.7% and 67.7%, respectively, outperforming traditional methods like GCG and AutoDAN at lower computational cost.

Source details: bioengineer.org ↗

Why it matters

This research reveals that safety alignment in open-source models is not a diffuse property but a localized, fingerprintable pattern concentrated in a small number of layers. This makes alignment more legible and potentially more fragile than previously assumed, as it can be surgically dismantled without degrading the model's core functionality. The findings suggest that current alignment techniques may be vulnerable to weight-space modifications, highlighting the need for more robust architectural defenses and continuous internal security testing.

The study challenges the assumption that safety alignment is a diffuse property spread across the entire network. Instead, it shows that censorship appears as a localized, fingerprintable pattern in a small number of layers. This localization makes alignment more legible but also more fragile, as it can be targeted with minimal impact on the model's overall utility.

The findings have significant implications for the security of open-source models. Since uncensored variants are widely available on platforms like Hugging Face, the white-box assumption is not overly restrictive. The ability to transfer layer fingerprints from small models to production-scale systems suggests that safety mechanisms occupy structurally analogous positions within model families, potentially exposing larger models to similar vulnerabilities.

From a defensive perspective, the same explainability tools used for the attack can be repurposed for automated security testing. Developers can use these methods to probe for failure modes continuously across the software lifecycle. The researchers propose mitigations such as internal activation monitoring, activation obfuscation, and architectural randomization to break the deterministic layer projection that enables cross-scale transfer.

What to watch next

Developers and security teams should monitor for the adoption of internal activation monitoring and architectural randomization as mitigations. The study's demonstration of cross-scale transfer implies that vulnerabilities found in smaller open-source models may indicate risks in larger, proprietary architectures, prompting a re-evaluation of how safety is encoded in transformer networks.

The practical impact of this research depends on the adoption of new defensive strategies. Security teams should look for the implementation of internal activation monitoring to detect anomalies in safety-critical layers. Architectural randomization may become a standard practice to prevent the deterministic mapping of safety mechanisms.

The study's demonstration of cross-scale transfer is particularly concerning for the deployment of open-source models in sensitive environments. Organizations using models like Llama or Qwen should re-evaluate their security posture, assuming that vulnerabilities identified in smaller variants may apply to their larger deployments.

Future research may focus on developing alignment techniques that are robust to weight-space modification. The current findings suggest that existing alignment methods, such as reinforcement learning from human feedback, may need to be supplemented with architectural safeguards to ensure long-term security.

Related guides & quizzes