What happened
George Washington University researchers Neil Johnson and Frank Yingjie Huo published a study on October 8, 2026, introducing a mathematical formula that predicts when AI chatbots may produce harmful or misleading responses. The formula analyzes how models prioritize information internally, identifying shifts that signal an approach toward undesirable outputs. Testing on seven open- models, including GPT-2, Claude Opus 4.5, and Gemini 3 Pro, the method correctly identified harmful response likelihood in 18 of 19 evaluated cases, excluding two borderline instances. The research focuses on smaller, publicly available models to allow examination of internal processing mechanisms, with the goal of informing future safety monitoring systems for AI developers.
Neil Johnson, a physics professor at George Washington University, and Ph.D. student Frank Yingjie Huo published a paper on October 8, 2026, detailing a mathematical formula designed to predict when AI chatbots will generate harmful responses. The formula measures the proximity of a model to producing undesirable outputs, such as misleading information or advice that encourages harmful behavior, by analyzing internal attention mechanisms rather than just the final text.
The study tested seven open- AI models, including GPT-2, Claude Opus 4.5, and Gemini 3 Pro. Open-weight models were selected because their internal numerical settings are publicly available, allowing researchers to examine how the models process information. The formula correctly identified whether a model would produce an undesirable response immediately, after a delay, or not at all in 18 out of 19 evaluated cases, with two cases excluded as borderline.
The researchers found that changing the order of questions on sensitive topics like vaccines, violence, and self-harm could alter whether a chatbot generated appropriate or undesirable answers. Johnson explained that the formula estimates the time until a harmful response by tracking the distance from dangerous behavior and the speed at which the model approaches it, analogous to a car approaching a cliff edge.
The study emphasizes that the formula is currently a research tool and has not been incorporated into real-time monitoring software. Johnson noted that while the mathematical principles may apply to larger commercial models, additional testing is required to confirm accuracy in those more complex systems. The research aims to help developers understand the underlying processes that lead to harmful outputs, providing a basis for creating more effective safety systems.
Why it matters
This research provides a concrete, mathematical approach to predicting failures before they occur, moving beyond reactive content filtering. By identifying internal attention shifts that precede harmful outputs, the formula offers a potential foundation for lightweight, real-time monitoring tools that can supplement existing safeguards. This is particularly significant for mental health and sensitive advice contexts, where technically accurate but dangerous responses pose severe risks. The study supports the argument for independent, evidence-based AI safety standards, suggesting that measurable performance metrics could inform future regulatory requirements for AI developers, ensuring that safety improvements are verifiable rather than self-reported.
The research addresses a critical gap in by offering a predictive metric for harmful behavior. Current safeguards often rely on post-hoc content filtering, which may not catch subtle shifts in model behavior that lead to dangerous advice. By monitoring internal attention shifts, this formula could enable proactive intervention before a user encounters harmful content.
This is particularly relevant for mental health applications, where AI chatbots are increasingly used as affordable alternatives to therapy. A 2025 Stanford study cited in the report found that AI therapy chatbots sometimes provided inappropriate responses to users expressing suicidal thoughts. The GW research provides a technical framework to identify when such models are drifting toward harmful territory, potentially allowing for safer user interactions.
The study supports the broader argument for independent research. Kevin Frazier, director of the AI Innovation and Law Program at the University of Texas School of Law, noted that relying solely on AI labs to self-report safety is insufficient. Independent research like this can inform performance-based legislation, where developers must demonstrate measurable safety improvements rather than adhering to specific, potentially outdated technical mandates.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Why can ethical evaluation not be reduced to one model score?
What to watch next
Researchers will need to test whether this formula applies to larger, more complex commercial AI models that are not open-. Developers may begin integrating similar attention-tracking metrics into their safety evaluation pipelines. Policymakers may reference such independent research when drafting performance-based legislation for , requiring developers to demonstrate measurable reductions in harmful responses. The practical deployment of lightweight monitoring systems based on this formula remains a future possibility, dependent on further validation across diverse model architectures.
The primary limitation is that the formula was tested on open- models. It remains unconfirmed whether this approach will scale effectively to larger, closed-source commercial models like ChatGPT, which have more complex internal architectures. Further testing is required to determine if the mathematical principles hold across different model scales.
Developers may begin experimenting with attention-tracking metrics as part of their safety evaluation processes. If the formula proves robust, it could become a standard for assessing the stability and safety of AI models, particularly in high-stakes domains like healthcare and financial advice.
Policymakers may look to this type of independent research to establish evidence-based standards for AI regulation. The concept of performance-based legislation, where developers must prove their models are becoming safer, could gain traction if independent studies provide clear, measurable indicators of safety performance.