Abstract
Multi-session agentic reinforcement learning is introduced to handle trajectories across multiple sessions, with Mosaic optimizing session-level contributions and allocation via hierarchical advantage estimation.
Most existing agentic reinforcement learning methods assume single-session trajectories, where interaction proceeds within one continuously accumulated context. However, agentic techniques such as context compaction and subagent forking may produce trajectories spanning multiple sessions, creating a mismatch between the single-session training assumption and inference-time execution. Therefore, we introduce Multi-Session Agentic Reinforcement Learning, a more general setting that supports an arbitrary number of sessions within a trajectory. Under this setting, a widely adopted method is to broadcast trajectory-level advantages to all sessions, but this does not distinguish contributions from various sessions or guide session allocation within trajectories. To address this issue, we propose Mosaic, a multi-session agentic RL method with critic-free hierarchical optimization. At the trajectory level, it incorporates session-usage signals into advantage estimation. At the session level, it shapes each session's advantage according to the estimated marginal contribution. Experiments on BrowseComp with Qwen3.5-4B show that Mosaic outperforms broadcasting in both context compaction and subagent forking scenarios while using fewer sessions on average. Further analyses show that our method achieves improvements in session utility and allocation, which we suggested is a promising target for future advances in multi-session agentic RL.
Community
In this paper, we propose a more general setting for agent training: Multi-Session Agentic Reinforcement Learning. In this setting, each rollout may contain an arbitrary number of sessions. Each session consists of agent–environment interactions within one continuously accumulated context. Each session ultimately corresponds to one training segment with its own assigned advantage.
Beyond that, we propose Mosaic, a general algorithmic improvement for Multi-Session Agentic RL. First, Mosaic adjusts trajectory-level outcome-based advantages using relative session counts within success/failure subgroups, balancing efficiency and exploration. Second, Mosaic measures each session's marginal information gain through changes in the ground-truth answer's log-likelihood, then adds a type-conditioned normalization for different sessions within each prompt group. Finally, trajectory-level advantage and session-level shaping signal are combined together to better estimate the finer-grained advantage for each session. We evaluate on BrowseComp, a benchmark of challenging web-research questions, using Qwen3.5-4B in a ReAct-style agent loop. Training uses 1,000 deep-search tasks from REDSearcher. Training and evaluation use both the context compaction or subagent forking execution loop. Mosaic outperforms the broadcasting baseline in both context compaction and subagent forking scenarios while using fewer sessions on average.
Feel free to visit our project homepage (https://lark-ai-lab.github.io/Mosaic/) to browse our blog, which provides a vivid overview of our research.
Get this paper in your agent:
hf papers read 2610.22174 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper