Incentivizing Self-Learning in Large Language Models via Reinforcement Learning
Abstract
Large language models (LLMs) draw on parametric knowledge and retrieval-augmented generation (RAG) to solve knowledge-intensive tasks. Yet problem solving in unfamiliar domains calls for tightly coupled knowledge acquisition and reasoning, beyond factual retrieval and aggregation or learning solely from a preassembled context. We study self-learning -- autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly. To investigate this capability, we formulate a textbook-based mathematical problem-solving task. LLMs must navigate textbook exposition, learn and apply relevant principles, and construct proofs for exercises, with reference solutions withheld. We introduce a BookNavigator interface to support this human-inspired study process and incentivize self-learning through reinforcement learning (RL) with fine-grained rewards. Experiments show that our method improves performance across in-domain and out-of-domain textbooks and subjects, raising Intern-S2-Preview-35B's overall proof accuracy from 51.5% to 79.5% and outperforming RL-trained textbook-free and fixed-retrieval baselines. These gains extend to complex long-context benchmarks involving different task types, with improvements over the base model of 1.7 and 2.3 percentage points on BrowseComp-Plus and CL-bench, respectively. Together, these findings suggest that RL can foster a domain-agnostic self-learning strategy, highlighting the potential of self-learning from long-form corpora to strengthen broader problem-solving capabilities on complex, long-context tasks.
Community
We study self-learning—autonomously retrieving, studying, and applying knowledge from long-form reference materials on the fly.
Navigating a textbook with the reference solutions withheld is much closer to how people learn a new subject than plain retrieval. I like that the BookNavigator interface rewards the study process in fine-grained steps, not only the final proof. Did the models learn to revisit earlier chapters, or mostly read forward?
Thanks for your kind words! 🤗 Appendix B.4 gives an example of targeted textbook consultation guided by the knowledge needed for a proof. The model encounters a contradiction in its understanding of matroid contraction, locates the relevant section, and checks the definition and rank-function formulas. It then corrects its interpretation and completes the proof. The lookup is driven by a specific gap in its reasoning, rather than by reading forward from the beginning of the book.
Our setup also encourages selective reading: each trajectory allows at most 8 interaction turns and 64k tokens in total, while textbooks average 208k tokens across 56 sections. As a rough comparison to reading forward, the Long-Context baseline in Section 3.1 appends sections in their original textbook order within a 64k-token input budget. After RL, it achieves 72.2% overall proof accuracy, compared with 79.5% for Self-Learning.
Together, the example and results support the value of letting the developing proof guide what to read: identifying missing knowledge, finding the relevant material, and using it to move the reasoning forward.
Really interesting setup, and the ablation is the part I keep thinking about: correctness-only GRPO lifts overall proof accuracy from 51.5% to 74.9%, yet tool-using trajectories drop from 36.9% to 6.7%, so the model improves by consulting the textbook less. The annealed exploration bonus bringing that back to 79.9% while also improving OOD-Book and OOD-Subject accuracy is a nice result. I build study tools for students, so the BookNavigator idea of browsing the table of contents before reading resonates a lot. Did you look at which kinds of sections the trained policy tends to open (definitions, theorems, or proofs of general results), and whether that pattern shifts between in-domain and unseen textbooks?
Thanks for taking a close look at our ablations and for the helpful suggestion! 🙌 We plan to study the model’s learned behavior in more detail, including what kinds of content it tends to consult and how its reading strategy changes across textbooks, subjects, and problems. This could help us understand how models identify gaps in their knowledge, choose what to study, and apply what they learn. We also hope these insights can inform better self-learning systems and study tools for students.
Get this paper in your agent:
hf papers read 2610.39671 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper