무슨 일이 일어났나요?
A preprint posted to arXiv on 29 July 2026 and revised on 7 August proposes a new way to measure progress toward automated AI research: hand an agent the central open-ended question of a high-quality unpublished paper and have that paper's original authors grade the output. In two trials, frontier agents completed the engineering without human help but were rejected by the human graders, and the authors identify five recurring failure modes.
A preprint on arXiv, submitted 29 July 2026 and revised on 7 August 2026, asks whether AI agents can conduct open-ended AI research. It carries 24 listed authors, including Peter Kirgis, Sayash Kapoor, Helen Toner, Gillian Hadfield, Seth Lazar, Rishi Bommasani and Arvind Narayanan, and is filed under artificial intelligence, computers and society, and machine learning. Its central contribution is a measurement method the authors call a "shadow evaluation": an agent is given the central, open-ended research question of a high-quality but unpublished paper, and the paper's original authors then grade what the agent produces.
The authors position this as a third option between two existing approaches they describe as inadequate. Current agent evaluations, they write, either test agents on narrow, verifiable tasks — which by construction excludes open-ended research — or submit AI-generated papers to blind peer review, which they characterise as overstretched, stochastic and suffering from poor review quality. Shadow evaluations instead use the people who already know the problem, its literature and its pitfalls as the graders, because they have an unpublished answer to compare against.
The team ran the method on two unpublished submissions to NeurIPS 2026. According to the abstract, frontier agents were given six days and thousands of dollars of compute per attempt. The agents completed all of the engineering without human help — the code, the experiments, the infrastructure — but could not make substantial progress toward answering the research questions themselves. Both resulting papers were, in the authors' words, unambiguously rejected by the original authors acting as reviewers.
From the transcripts the authors extract five recurring failure modes: poor judgment about the bar for publishable research; uncreative responses to shortcomings in the research design; ineffective backtracking from dead ends; poor resource awareness; and instruction drift. They report that a robustness check using a second model and a second scaffold reproduced the same failures, which argues against the result being an artifact of one particular system or harness. The team says it is releasing the expert reviews, survey responses, agent repositories and logs.
Several things the abstract does not establish are worth stating plainly. It does not name which agents, models or scaffolds were used, nor the second system in the robustness check. It does not identify the two NeurIPS submissions or their subfields, give an exact compute figure beyond "thousands of dollars," describe the grading rubric, or say whether graders were blinded to the fact that they were reviewing agent output. The work is a preprint; nothing in the source indicates it has completed peer review. And the sample is two research questions, graded by the people who posed them.
왜 중요한가요?
Arguments that AI progress will accelerate sharply rest on the premise that AI systems can take over AI research itself. This is direct, if small, evidence that the premise does not hold yet, and it separates the part agents can already do — implementation — from the parts they cannot: judging what is worth publishing, redesigning a flawed study, and abandoning dead ends.
The paper's framing goes straight at a load-bearing assumption in current AI forecasting. As the abstract puts it, forecasts of explosive AI progress hinge on AI agents automating AI research — the loop in which better systems build better systems. That loop is frequently asserted and rarely measured, because the thing being automated is exactly the kind of work that resists a scoring script. A method that produces a graded verdict from working researchers on a live, unanswered question is a more direct probe of that assumption than benchmark scores on tasks with known answers.
The result draws a line through the middle of the job rather than around it. Agents ran multi-day experimental programmes unattended and produced working code and results — that is a real capability and a meaningful change from a few years ago. What they did not do is the part that determines whether research is worth publishing: recognising that a result is thin, inventing a way around a design flaw, and knowing when a line of attack is dead. For teams deciding where to put agents in a research or engineering pipeline, that distinction is more actionable than an aggregate capability score.
Two of the five failure modes generalise well beyond research. Poor resource awareness — burning a fixed compute or time budget without adjusting — and instruction drift, where an agent gradually stops doing what it was asked, are recurring problems in any long-horizon autonomous deployment, from multi-day coding agents to operational automation. The six-day runs here are unusually long by evaluation standards, and long horizons are precisely where these failures have room to compound before anyone notices.
The limitations are real and the authors do not hide them. Two case studies at one venue, evaluated on one snapshot of models, cannot support a claim about what agents will be able to do next year. This is a null result with a timestamp. Author-graders also have an obvious stake in the question they chose to work on, and the method partly consumes itself: once a paper is published, its research question can no longer serve as an unseen test. Releasing the reviews, repositories and logs is what makes the finding checkable rather than a claim to be taken on trust.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
crm_get_transaction(id='4092').What is the most accurate way to describe what AI Agents can do today?
다음에 무엇을 볼 것인가
Whether shadow evaluations are repeated at larger scale with blinded, pre-registered grading; whether newer agents and scaffolds close the gap when rerun against the released repositories and logs; and whether the five failure modes hold up under independent scrutiny and peer review.
The first thing to watch is whether shadow evaluations become a repeatable instrument or remain a one-off demonstration. That would mean more papers across more venues and subfields, pre-registered grading rubrics agreed before the agent runs, and grading blinded to whether output came from an agent or a human. It would also mean addressing the supply problem: the method needs a steady flow of high-quality unpublished work whose authors are willing to spend real reviewing effort on an agent's attempt.
The second is whether the gap closes, and how fast. Because the agent repositories and logs are being released, other groups can rerun newer models and scaffolds against the same two research questions and compare directly. Watch for whether improvements show up as better engineering — which the agents already did — or as movement on the judgment failures, which is the part the paper says is missing. A rerun that produces more experiments but the same rejection would be a different signal than one that produces a paper the original authors take seriously.
Third, independent scrutiny of the released artifacts. The five failure modes are the authors' reading of their own transcripts; other researchers reading the same logs may categorise the breakdowns differently, or find that some failures trace to the scaffold and prompting rather than to the models. Whether the preprint clears peer review, and whether its taxonomy survives contact with other people's evaluations, will determine how much weight it can carry.
Finally, uptake outside the research community. The author list includes people who work directly on and policy, and timelines for automated AI R&D feed into frontier-safety frameworks, capability thresholds and national policy debates. Watch whether this evidence is cited in those settings, and whether it is reported accurately: the paper says today's agents struggle with critical parts of the research lifecycle, based on two cases — not that they cannot ever do research, and not that the engineering capability on display is trivial.