What happened
Researchers have introduced the Graph-based Faithful sTep-level credit-assignment framework (GRAFT), a new method designed to improve (RL) for agentic large language models. The framework addresses a known limitation in current group-based RL methods, such as GRPO, which struggle to accurately evaluate the contribution of individual steps within a multi-step reasoning trajectory.
The GRAFT framework addresses the systematic bias found in group-normalized advantage estimation when applied at the step level. While existing methods like GRPO are effective for single-turn tasks, they often fail to accurately reflect the value of individual steps in multi-turn agentic reasoning.
To solve this, GRAFT aggregates multiple rollout trajectories into a single trajectory graph. By performing Bellman iteration on this graph, the framework recovers node state-values and assigns credit to each edge based on the difference in node values. This approach theoretically aligns with the fundamental definition of advantage in .
The authors also introduced 'Graph GAE,' an extension of Generalized Advantage Estimation (GAE) adapted for trajectory graphs, intended to reduce the impact of bias in state-value estimation. Experimental results on multi-turn agentic benchmarks indicate that GRAFT provides consistent performance gains over GRPO and other recent agentic RL algorithms.
Why it matters
Current RL methods for agentic models often rely on coarse-grained, trajectory-level rewards, which can lead to biased evaluations when a successful outcome contains flawed steps or a failed trajectory contains useful intermediate actions. By using trajectory graphs to recover state-values via Bellman iteration, GRAFT provides a more granular and theoretically grounded way to assign credit. This improvement is critical for training agents that perform complex, multi-turn tasks where the quality of individual reasoning steps directly impacts overall performance and reliability.
In agentic AI, the ability to distinguish between a 'lucky' successful trajectory and one that contains high-quality reasoning is essential for effective training. Current methods often treat entire trajectories as a single unit, which can obscure the value of specific, correct intermediate steps.
By enabling more precise credit assignment, GRAFT allows models to learn more effectively from their own reasoning processes. This is a significant step toward more robust agentic systems that can reliably navigate multi-step problems without relying solely on final-outcome feedback.
The framework's reliance on the foundational definition of RL advantage provides a more rigorous mathematical basis for training, potentially reducing the 'noise' that often plagues for complex reasoning tasks.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
What most distinguishes an AI agent from a basic chatbot?
What to watch next
The researchers have announced that code for the GRAFT framework will be made available on GitHub. Future adoption will depend on how effectively this method scales to larger, more complex agentic environments compared to existing industry-standard RL techniques. It remains unknown how the computational overhead of constructing and iterating over trajectory graphs will compare to current training pipelines in production settings.
The research team has committed to releasing the implementation code at https://github.com/xcyao00/GRAFT. The availability of this code will be the primary indicator of the framework's practical utility for the broader research community.
A key unknown is the scalability of the trajectory graph approach. As agentic tasks grow in length and complexity, the memory and requirements for maintaining and iterating over these graphs may become a bottleneck compared to simpler, non-graph-based RL methods.
Industry practitioners should monitor whether this method can be integrated into existing large-scale training pipelines without significantly increasing training time or infrastructure costs.