Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
NVIDIA SkillEvaluator measures the impact of verified skills on AI agent performance through a three-tier evaluation process.
毎日更新1793 検証済みのストーリー
製品の発売、政策の変更、安全性研究、業界の動向についてソースチェックされた AI の報道が、非営利教育チームによって平易な英語で説明されます。
すべてのストーリーは、入手可能な最強の証拠、つまり入手可能な場合はオリジナルの情報源、それ以外の場合は明らかに帰属が明記された報道にリンクしています。
What happened, why it matters, and what to watch — without the jargon.
信号が薄い場合は、フィードをパディングするだけで何も公開しません。
Source-checked AI stories, newest first, for people who need to understand AI without chasing hype.
NVIDIA SkillEvaluator measures the impact of verified skills on AI agent performance through a three-tier evaluation process.
A 24-author preprint had frontier AI agents attempt the central research questions of two unpublished NeurIPS 2026 submissions, then had the papers' own authors grade the results. The agents handled the engineering unaided over six days but were unambiguously rejected on the research.
Warp is taking early-access requests for Warp Factories, which defines fleets of coding agents as code — repos, models, permissions and human checkpoints in one YAML file, driven by CLI, API, SDK and MCP. Its automation and cost figures are vendor claims: no pricing, general availability date or independent testing.
A new arXiv paper introduces a test in which models must infer a written word from pen-scratch audio and hand-movement video, with no ink visible. The authors report humans above 80% ordered letter accuracy and leading models below 10% — and that giving models both modalities often made results worse.
A new arXiv preprint describes CacheScout, a layer built on the open-source vLLM server that decides what to keep in a model's key-value cache based on which agent is likely to run next. The authors report double-digit latency and throughput gains; the workloads, models, and hardware are not stated in the abstract.
Two researchers say a lemma proof in Chapter 6 of OpenAI's mathematics document has a polarity error: a test in terms of average success where the next step needs a large conditional failure. They give a counterexample and a corrected proof, and caution that this is not verification of the chapter's main theorem.
A preprint by two researchers reproduces an earlier study on why equal FLOP counts do not mean equal execution time. It confirms the underlying claim but reports that the α-FLOPs correction formula generally underestimates runtime on newer hardware, which shows jumps and oscillations the formula does not capture.
A new arXiv preprint pits four architectures for natural-language querying of enterprise databases against each other on a synthetic bilingual benchmark. None answered more than about a quarter of cases correctly, and the design that scored highest was not the safest or the cheapest.
A new arXiv preprint argues security teams can judge whether a memory- or retrieval-equipped AI agent is learning by measuring how far it closes the gap to a stronger "teacher" model, rather than on labeled benchmarks that are often scarce or stale. Judging by a similarly powered model gave no usable signal.
A 17-author technical report posted to arXiv introduces MobileMem, a benchmark and framework for on-device long-term memory built from a year-scale collection of mobile experiences. The abstract describes the design but reports no scores, and key details about the underlying data remain undisclosed.
A new arXiv preprint says large language models develop functionally specialized internal structure that lines up with distinct human brain networks, based on circuit analyses across 46 tasks in four cognitive domains. The abstract page leaves key methodological details unstated.
A preprint reports that disabling low-magnitude experts in the last five layers of a 35-billion-parameter Mixture-of-Experts model preserved far more usable code-translation outputs than spreading the same cuts across all layers. It covers one model and one benchmark, and the abstract reports no unmasked baseline.
毎週 1 回の有益なブリーフィング
今週の検証済み AI ニュース、オリジナル データ、便利なツール、おすすめの学習情報、最新の AI ジョブを入手します。
AI の専門家を雇いますか、それとも便利な AI 製品を立ち上げますか?それを学び、行動するためにここに来た人々の前に置きます。
AI の仕事を投稿する AIツールを提出する