What happened
Researchers led by Kubra Aykac of Hacettepe University evaluated the clinical reasoning capabilities of ChatGPT-5, Gemini 2.5 Flash, and DeepSeek-V3.2-Exp using twelve standardized pediatric infectious disease scenarios. The study, published in Pediatric Research, tasked each model with answering five open-ended questions per scenario, covering etiology, clinical features, diagnostics, treatment, and complications. Two board-certified pediatric infectious disease specialists independently scored the 180 total responses using a blinded, randomized process. The results showed that all three models performed at high levels, with no statistically significant differences in accuracy, completeness, or safety scores across the tested domains.
The study evaluated three models—ChatGPT-5, Gemini 2.5 Flash, and DeepSeek-V3.2-Exp—against twelve pediatric infectious disease scenarios. These scenarios were selected based on clinical relevance, severity, and educational applicability, focusing on conditions like acute bacterial meningitis and common respiratory or gastrointestinal infections.
To ensure an unbiased evaluation, an independent coordinator stripped all model identifiers and artifacts from the responses. Two blinded, board-certified pediatric infectious disease specialists scored the responses based on accuracy, completeness, and safety. The scoring used a purpose-built Likert rubric designed to flag unsafe or insufficient clinical advice.
Statistical analysis using the Friedman test and Cochran's Q test revealed no significant performance gaps between the models. ChatGPT-5 achieved perfect domain scores of 10.0, while DeepSeek-V3.2-Exp and Gemini 2.5 Flash followed closely, with minor variations that did not reach statistical significance (p > 0.05).
The authors noted a significant improvement in performance compared to their previous research involving older models like ChatGPT-3.5, which struggled with age-specific reasoning. This suggests that newer architectures are better at incorporating medical knowledge and clinical context.
Source details: bioengineer.org ↗
Why it matters
This study provides a controlled assessment of how current frontier AI models handle age-specific medical reasoning, a critical requirement for pediatric care. While the models demonstrated high proficiency in structured scenarios, the researchers identified persistent qualitative gaps in areas like vital sign thresholds and precise dosing. The findings suggest that while these tools are becoming increasingly competent for educational support and clinical preparation, they remain best suited for supervised workflows where human judgment can verify outputs. The research highlights a 'ceiling effect' in current evaluation rubrics, indicating that as models improve, more sensitive testing tools are required to distinguish their clinical utility.
The study underscores that high numerical scores in a controlled setting do not equate to fully reliable clinical reasoning. The expert reviewers identified recurring qualitative issues, such as imprecise vital sign cutoffs and hospitalization criteria, which are vital for pediatric safety.
By demonstrating that these models are currently 'competent but supervised assistants,' the research provides a framework for integrating AI into medical education. It suggests that AI can effectively help trainees organize information and outline diagnostic steps, provided the output is reviewed by a qualified clinician.
The research highlights the limitations of current evaluation rubrics, which are primarily designed to catch unsafe advice rather than differentiate between high-performing systems. This indicates a need for more granular, clinically challenging benchmarks as AI models continue to advance.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Which component of an AI application is the machine-learning model itself?
What to watch next
Future research will need to move beyond standardized, predefined scenarios to test how these models perform with fully undifferentiated clinical presentations. Additionally, the study notes that the lack of transparency regarding specific model version updates—as consumer platforms are updated continuously—remains a significant barrier to reproducibility. Clinicians and educators should monitor whether subsequent model iterations can address the identified gaps in age-specific decision thresholds and safety-oriented escalation criteria, which remain essential for reliable pediatric practice.
The researchers acknowledged that the study's reliance on twelve predefined scenarios limits its generalizability to the full spectrum of everyday clinical practice. Future studies are expected to incorporate more complex, undifferentiated cases.
The study highlights the challenge of reproducibility in AI research, as the underlying model checkpoints are not publicly disclosed and are subject to continuous, unannounced updates by the providers.
The authors emphasize that future evaluation tools must be more sensitive to identify meaningful differences in clinical reasoning, as the current generation of models is beginning to hit the top of existing scoring scales.