Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
Abstract
OR-Clarify benchmarks clarification before optimization modeling, and InterOPT guides agents to ask questions or stop based on missing formulation-critical details.
Large language models (LLMs) are increasingly used to formulate optimization models from natural-language problem descriptions, yet realistic operations research (OR) requests are often incomplete: missing objectives, constraints, or business rules can change the resulting mathematical program. Existing evaluations largely assume a complete specification and therefore overlook whether an agent knows when clarification is needed before modeling. We introduce OR-Clarify, a benchmark for pre-formulation clarification. Each task presents a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. The benchmark supports both openended and choice-based clarification, and measures slot recovery, stopping behavior, silent assumptions, and interaction cost. We further propose Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to guide whether to ask the next question or to stop. In our choice-based experiments, InterOPT substantially outperforms all baselines in exact slot recovery; in the open-ended setting, it remains competitive with strong prior methods. Together, OR-Clarify and InterOPT reframe OR assistance as a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing.
Community
We introduce InterOpt, an open framework for improving the interaction between large language models and optimization solvers.
While LLMs have shown promising capabilities in mathematical modeling, they often struggle with ambiguous problem descriptions, incomplete specifications, and unreliable formulations. InterOpt explores a solver-in-the-loop paradigm that enables LLMs to iteratively clarify optimization problems, refine formulations, and verify generated solutions through optimization feedback.
We release:
- The InterOpt framework for LLM-based optimization modeling and interaction.
- OR-Clarify, a dataset designed to study ambiguity resolution and clarification in optimization problems.
- Open-source implementations and experiments to facilitate future research on LLMs for Operations Research.
Project page: https://github.com/AIOR-Research/InterOpt
Dataset: https://huggingface.co/datasets/AIOR-Research/OR-Clarify
We hope InterOpt can serve as a step toward more reliable and interactive AI assistants for real-world optimization tasks.
Every agent I've shipped that didn't ask clarifying questions produced something useless — so this benchmark tests the right failure mode. But the real question isn't whether asking helps, it's whether asking costs more than guessing in practice. One extra round-trip per task is cheap when the task is expensive; it's brutal when you're optimizing something trivial. I'd want to see the latency and token overhead of the clarification loop, not just the success rate. If asking adds 30 seconds to a task that would've taken 5 minutes to get wrong, that's a win. If it doubles every call, that's a different story.
Get this paper in your agent:
hf papers read 2609.05258 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper