What happened
Google’s Gemini 4 Carbon model has achieved the highest score on the newly updated AA-Omniscience v1.1 , a tool developed by Artificial Analysis to measure AI knowledge reliability and hallucination rates. The model recorded a Net Score of 54, placing it six points ahead of the second-place Claude Opus 5.5. This update to the benchmark involved refining the evaluation dataset by removing 3,000 potentially saturated or ambiguous questions and replacing them with new, verified content to ensure more accurate testing of model capabilities.
Artificial Analysis released version 1.1 of its AA-Omniscience as part of its Intelligence Index v5. The update reduced the total question count from 6,000 to 3,000, specifically removing questions that were deemed 'saturated'—meaning they were likely present in training data—or contained errors. New questions were added to maintain coverage across six domains, including Science, Engineering, Mathematics, and Law.
Gemini 4 Carbon (high) achieved a Net Score of 54, leading the field. Claude Opus 5.5 (max with fallback) followed with 48, and Claude Fable 5.1 (max with fallback) scored 46. GPT-6 Astra and GPT-6.1 Sol tied for fourth with 42. The highest-scoring open-weights model, Muse Spark Open, achieved a score of 26, trailing the leader by 28 points.
The uses a Net Score metric ranging from -100 to 100, where positive scores indicate reliable knowledge and negative scores suggest a tendency to provide more incorrect answers than correct ones. Several models, including DeepSeek V4.1 Flash and Qwen3.8 27B, recorded negative scores on the updated test.
Artificial Analysis also released a 300-question public set on Hugging Face to allow external verification. The firm reported a 0.99 correlation between the public and private sets, suggesting the public version is a highly accurate proxy for the full .
Source details: officechai.com
Why it matters
The AA-Omniscience v1.1 provides a critical look at how frontier AI models handle factual accuracy versus guessing. By penalizing confident hallucinations and rewarding models that correctly identify their own knowledge gaps, the benchmark offers a more nuanced view of reliability than standard accuracy tests. The significant lead held by Gemini 4 Carbon over both proprietary and open-weights models highlights the persistent performance gap between top-tier frontier systems and the broader ecosystem. This data helps users and developers distinguish between models that provide reliable information and those prone to confident errors.
The shift to v1.1 highlights the ongoing challenge of 'data contamination' in AI benchmarks, where models may perform well simply because they have 'memorized' the test questions during training. By stripping out these questions, Artificial Analysis provides a more rigorous assessment of a model's actual reasoning and knowledge retrieval capabilities.
The performance gap between Gemini 4 Carbon and the rest of the field underscores the difficulty of maintaining high reliability at scale. While hallucination rates improved slightly across the board—with Gemini 4 Argon recording the lowest rate at 4%—the wide variance in Net Scores indicates that many models still struggle to admit uncertainty, a key requirement for enterprise-grade applications.
For users, these results serve as a practical guide for selecting models for knowledge-intensive tasks. The data suggests that while many models are capable of generating fluent text, only a select few demonstrate the factual reliability necessary for high-stakes environments like finance or law.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Which component of an AI application is the machine-learning model itself?
What to watch next
Observers should monitor whether the performance gap between proprietary frontier models and open-weights alternatives continues to narrow in future iterations. Additionally, the industry will likely watch how other labs respond to the v1.1 dataset, as the removal of saturated questions caused notable score drops for several models, such as MiniMax M3.1 Flash and MiMo V2.6 Pro. The high correlation between the new public and the private benchmark suggests that developers now have a more reliable, accessible tool to gauge their models' real-world knowledge performance.
The industry will be watching to see if the 'rank order' of models remains stable as labs update their systems to account for the harder, cleaner dataset. The 0.98 correlation between v1 and v1.1 suggests that while scores dropped, the relative hierarchy of model intelligence remains largely consistent.
The significant score declines for models like MiniMax M3.1 Flash and MiMo V2.6 Pro indicate that some models may have been heavily reliant on memorizing previous questions. Future updates will likely focus on whether these developers can improve their models' capabilities to perform well on unseen, high-difficulty questions.
The availability of the public 300-question set on Hugging Face is a significant development for transparency. It allows researchers and the public to independently verify claims made by AI labs, potentially reducing reliance on proprietary, 'black-box' testing methodologies.