What happened
Researchers at NYU Tandon evaluated 268 responses from GPT-4o, Claude 3.7 Sonnet, and two specialist systems (Aimee and Ruth) against authoritative sources for advice on technology-facilitated abuse. The study found that 86% of responses were inaccurate, incomplete, or unsafe, and 60% omitted critical safety warnings. Specialist systems did not outperform general-purpose models, with Aimee rated unsafe 74% of the time compared to 55% for GPT-4o. The research also noted that all four systems could be induced to answer questions from potential perpetrators.
Researchers at NYU Tandon conducted a study assessing the reliability of AI chatbots in providing advice on technology-facilitated abuse, such as spyware, account takeovers, and location tracking. The team selected 193 questions from academic literature and online discussions to test four systems: GPT-4o, Claude 3.7 Sonnet, and two specialist intimate-partner-violence chatbots, Aimee and Ruth.
The study evaluated 268 chatbot responses against authoritative sources for accuracy, completeness, and safety. The results showed that 86% of the responses fell short in at least one of these categories, while 60% omitted critical safety warnings. The researchers also surveyed 114 people with lived experience of technology-facilitated abuse to gauge the perceived actionability of the advice, though this measured perception rather than real-world outcomes.
Contrary to expectations, the specialist systems did not outperform the general-purpose models. Aimee’s responses were rated unsafe approximately 74% of the time, compared to 65% for Claude, 61% for Ruth, and 55% for GPT-4o. The study noted that the models did not receive equal numbers of expert-evaluated questions, which may affect direct comparability.
Specific failures included recommendations for ineffective tools like radio-frequency detectors or illegal GPS jammers, and advice on password changes that could alert abusers or destroy evidence. None of the systems warned that spyware on a rooted or jailbroken phone could survive a factory reset. Additionally, all four systems could be induced to answer questions from a potential perpetrator, with the specialist systems answering roughly two-thirds of such prompts.
Source details: techtimes.co.uk
Why it matters
The findings highlight a significant safety gap in AI systems used for high-stakes personal security advice, particularly for survivors of technology-facilitated abuse. Because AI advice may lack critical context regarding evidence preservation or abuser alertness, relying on these tools can inadvertently increase risk. This underscores the need for independent evaluation of in support organizations and raises concerns about the reliability of general-purpose LLMs in vulnerable user scenarios.
The study reveals that technically correct AI advice can still be dangerous in high-risk situations if it lacks critical context, such as the risk of alerting an abuser or compromising evidence. This is particularly concerning for survivors of technology-facilitated abuse who may rely on AI for immediate guidance.
The findings challenge the assumption that specialist AI systems are inherently safer or more effective than general-purpose models in sensitive domains. The high rate of unsafe responses from specialist tools like Aimee suggests that domain-specific training alone is insufficient to ensure safety in complex, high-stakes scenarios.
The results have implications for support organizations and policymakers, as they highlight the need for independent evaluation of AI systems before they are used for high-stakes guidance. The study recommends that organizations treat AI-generated safety advice with caution and verify its accuracy against authoritative sources.
The timing of the study is relevant to ongoing efforts by UK authorities and support organizations to address technology-facilitated abuse, with recent data showing significant increases in referrals and online-enabled offenses. The findings underscore the importance of integrating reliable AI tools into broader safety strategies.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Why can ethical evaluation not be reduced to one model score?
What to watch next
Monitor whether AI developers implement specific safety for abuse-related queries, whether support organizations adopt independent evaluation protocols for AI tools, and if regulatory bodies like Ofcom address AI-generated safety advice in their online safety frameworks.
Observe whether AI developers introduce specific safety protocols or for queries related to domestic violence and technology-facilitated abuse, potentially including refusal to answer or redirection to human experts.
Monitor the response of support organizations and NGOs to the study’s recommendations, particularly whether they begin independently evaluating AI tools before integrating them into their service offerings.
Watch for regulatory developments in the UK and US regarding AI-generated safety advice, including whether bodies like Ofcom or the FTC address the reliability of AI in high-stakes personal security contexts.
Track future research that may refine the methodology of this study, such as larger sample sizes, equal distribution of questions across models, or real-world outcome tracking rather than perceived usefulness surveys.