Re-Search: Dissecting LLM Ideation Through a Two-Player Questioning Game
Abstract
A Twenty Questions-style benchmark for research ideation shows that large language models make most progress early, can recognize good ideas or questions but often cannot generate them, and may benefit from externally sourced candidates in automated research systems.
Research ideation is iterative: researchers refine their ideas step by step based on feedback. Yet existing evaluations obscure this search process, making it difficult to understand how LLMs explore the space of research ideas and what limits their progress. We introduce Re-Search, which adapts the format of Twenty Questions to research ideation: given only the research problem of a published paper, a model must rediscover the paper's core idea by asking yes-or-no questions. Built from 200 ICML 2026 papers, Re-Search tracks how a model's idea evolves as it asks questions and receives binary oracle feedback, isolating ideation from execution. Across six recent LLMs, nearly all progress happens within the first 30 of 100 questions, which largely decide how the game ends; after that, most models plateau. Through controlled interventions on the candidate questions and the hypothesis space, we find that the bottleneck is generation, not judgment: models recognize a good idea or a good question when one is offered, but often cannot produce it on their own. We further analyze the ideation traces and find that models fail by either over-exploiting one direction or over-exploring too many. These findings point to concrete design principles for autoresearch systems: since models can judge candidates they cannot produce, enriching the search with externally sourced ideas and questions may be more effective than improving any single model's generation ability.
Community
AI is now doing research on its own, but we only ever see the final results, never the ideation behind them. How does an LLM actually search its way to a new idea, and where does it get stuck?
Re-Search turns ideation into a game of Twenty Questions: an LLM asks yes/no questions to rediscover the core idea of a published paper, making every step of the search observable and measurable.
Findings from 6 LLMs on 200 ICML 2026 papers:
Nearly all progress happens in the first 30 of 100 questions, after which models plateau.
Models can judge ideas they cannot produce. When picking from a shared pool of questions, all models score about the same, and weak models gain the most.
Search structure matters. Recovery follows an inverted U in the number of lines of inquiry, peaking at around 8.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation (2026)
- Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation (2026)
- IdeaAnchor: Teaching LLMs to Turn Literature into Research Ideas (2026)
- AlphaIdeaBench: Benchmarking Automated Research Idea Generation under Feasibility Constraints with Peer Review Alignment (2026)
- Your Benchmark Is Not Saturated: Reviving Multiple-Choice Evaluation with Answer Pooling (2026)
- Training Reasoning Models to Generate Diverse Scientific Ideas (2026)
- Distill, Debate, Discover: Grounding Multi-Agent Research Ideation in Graph-Distilled Insights (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.14487 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper