Abstract
ContextRL enhances long-horizon reasoning and multimodal performance through reinforcement learning that rewards context selection for supporting query-answer pairs, achieving improvements over standard methods on diverse benchmarks.
Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an indirect auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.
Community
Context-Aware RL for Agentic and Multimodal LLMs
π LLMs often fail not because the answer is impossible, but because they miss the one decisive clue hidden in a long trace or image.
π₯ We introduce ContextRL: RL that teaches models to identify which context actually supports an answer.
β
+2.2% on 5 agentic benchmarks
β
+1.8% across 12 VQA benchmarks
β
Works for coding agents & multimodal reasoning
β
Same contrastive data, but better objective β not data augmentation
π§ The key idea: donβt only reward the final answer. Reward the model for grounding it in the right evidence.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents (2026)
- DynFrame: Adaptive Reasoning-Driven Multimodal Framework with Dynamic Frame Augmentation for Complex Video Understanding (2026)
- VISD: Enhancing Video Reasoning via Structured Self-Distillation (2026)
- Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization (2026)
- SSL-R1: Self-Supervised Visual Reinforcement Post-Training for Multimodal Large Language Models (2026)
- Structured Role-Aware Policy Optimization for Multimodal Reasoning (2026)
- AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2606.17053 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper