| --- |
| license: apache-2.0 |
| --- |
| |
|
|
| # MOP-RL-model |
|
|
| ## Introduction |
|
|
| **MOP-RL-model** 是一款专为**多目标混合整数线性规划 (MO-MILP)** 任务打造的大型语言模型。该模型基于 Qwen2.5-7B 架构，并使用新颖的多目标规划强化学习框架 (MOP-RL) 进行了对齐训练。 |
|
|
| 在复杂的资源调度、智能制造与现代物流决策中，传统大模型往往难以平衡存在冲突的多目标，且容易在长序列推理中产生“逻辑幻觉”或“奖励作弊 (Reward Hacking)”。MOP-RL-model 突破了这一瓶颈，不仅掌握了基础的运筹学约束构建能力，更具备了在高维决策空间中捕捉帕累托前沿 (Pareto Front) 的深层权衡智慧。 |
|
|
| 本模型的核心技术亮点包括： |
|
|
| * 两阶段课程学习 (Two-stage Curriculum Learning)：模型经历了从“单目标基础训练 (密集奖励)”到“多目标扩展训练 (稀疏帕累托奖励)”的阶梯式对齐，有效抑制了强化学习中的策略震荡与基础能力崩塌。 |
| * 帕累托感知奖励 (Pareto-Aware Reward Shaping)：摒弃了传统的标量逼近奖励，引入基于底层求解器 (如 Gurobi) 支配性测试的帕累托验证器，将复杂的数学证明转化为精确的绝对物理反馈。 |
| * REINFORCE++ 算法：采用去除了价值网络 (Critic-free) 的改进型策略梯度算法，结合 Batch 内优势函数归一化与概率比裁剪，显著提升了面对上千 Token 的结构化思维链 (Structured CoT) 推理时的收敛稳定性。 |
| * 结构化思维链输出 (Structured CoT)**：模型被强制要求遵循“问题分析——> 建模与标量化——>可执行代码生成”的严谨规范，确保了生成结果的物理可执行性与逻辑自治性。 |
| |
| ## Requirements |
| |
| 运行 MOP-RL-model 的代码与普通 Qwen2.5 模型（详情参见：https://huggingface.co/Qwen/Qwen2.5-7B ）一致，推荐使用最新版本的 `transformers`。 |
| |
| ```bash |
| pip install transformers>=4.37.0 |
| # 如果需要本地运行生成的优化代码，还需要安装求解器 |
| pip install gurobipy |
| |
| ``` |
| |
| > **Warning**: The generated code often imports `gurobipy`. Ensure you have a valid Gurobi license to execute the generated solver scripts in your local environment. |
| |
| ## Quickstart |
| |
| 以下是一段使用 `transformers` 库加载并运行模型进行多目标建模的示例代码： |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_name = "YourOrg/MOP-RL-model" |
| |
| model = AutoModelForCausalLM.from_pretrained( |
| model_name, |
| torch_dtype="auto", |
| device_map="auto" |
| ) |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| |
| # 示例多目标调度问题描述 |
| prompt = """ |
| **[Task]** |
| Based on the [Problem Description] and [Target JSON Format] below, please write a complete Python script. |
| The script must use the `gurobipy` library to model and solve the multi-objective problem, and encapsulate the final solution results into JSON data, writing them to a file named `input.json`. |
| |
| **[Input Information]** |
| 1. **Multi-objective Problem Description**: |
| \"\"\" |
| {problem_desc} |
| \"\"\" |
| 2. **Target Output JSON Format Example**: |
| \"\"\" |
| {json_template} |
| \"\"\" |
| |
| **[Requirements]** |
| 1. **Modeling Logic**: |
| - Clearly define decision variables and constraints. |
| - **Multi-objective Handling**: Choose an appropriate multi-objective processing method based on the context (e.g., weighted sum, hierarchical sequence/lexicographic, or Pareto frontier). If weights are not specified, assume equal weights or provide adjustable parameters. |
| 2. **Code Standards**: |
| - The code must be a complete, runnable Python script. |
| - Include necessary comments explaining the mathematical model. |
| - Must include checks for model solution status (e.g., checking for `GRB.OPTIMAL`). |
| 3. **Data Output**: |
| - After solving, extract variable values. |
| - **Format Matching**: Construct a Python dictionary that strictly matches the [Target Output JSON Format Example]. |
| - **File Writing**: Use `json.dump` to save the result as `input.json`. |
| 4. **Final Output**: |
| - Please wrap your code in a Python markdown block, i.e., starts with ```python and ends with ```. |
| - Provide only the Python code block.""" |
| |
| messages = [ |
| {"role": "system", "content": "You are an algorithm expert proficient in Operations Research and the Python Gurobi solver. You excel at translating complex business scenarios into mathematical models and writing robust code."}, |
| {"role": "user", "content": prompt} |
| ] |
| |
| text = tokenizer.apply_chat_template( |
| messages, |
| tokenize=False, |
| add_generation_prompt=True |
| ) |
| model_inputs = tokenizer([text], return_tensors="pt").to(model.device) |
| |
| generated_ids = model.generate( |
| **model_inputs, |
| max_new_tokens=2048 |
| ) |
| generated_ids = [ |
| output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids) |
| ] |
| |
| response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0] |
| print(response) |
|
|
| ``` |
| |
| ## RL code |
| |
| RL训练代码放在如下链接里： |
| https://github.com/xuebozhang525-alt/MOP_RL |
| |
| ## Evaluation & Performance |
| |
| MOP-RL-model 在具有极高难度的工业级多目标混合整数线性规划测试集（包含 1892 个复杂长尾案例）上进行了严苛的级联指标测评。 |
| |
| 相较于直接在多目标数据上进行零样本推理或单纯进行 SFT 的基线模型，本模型实现了“降维打击”般的性能跃升。 |
| |
| | Metric / Evaluation Level | Description | Score | |
| | --- | --- | --- | |
| | 格式准确率 | 生成代码无格式错误 | 1.000 (100%) | |
| | 代码可执行率 | 无约束遗漏，成功提取变量并求得有效可行解，且求解器语法无报错的概率 | 88.01% | |
| | 条件帕累托率 | 在模型完全正确的前提下，解集通过支配性测试的概率 | 77.43% | |
| | 综合帕累托成功率 | parado解的总概率 | 68.15% | |
| |
| 与base的对比结果如下： |
| |
| | 模型 | 格式准确率 | 代码可执行率 | 条件帕累托率 | 整体帕累托率 | |
| |------|-----------|------------|------------|------------| |
| | ChatGPT 5（闭源） | 0.985 | 0.615 | 0.732 | 45.0% | |
| | DeepSeek-R1(671B) | 0.968 | 0.821 | 0.582 | 47.8% | |
| | Qwen3-Max(1T) | 0.975 | 0.862 | 0.689 | 59.4% | |
| | MOP-RL(7B) (Ours) | 1.000 | 0.880 | 0.774 | 68.1% | |
| |
| ## Citation |
| |
| 如果您在研究中使用了本模型，请引用以下论文： |
| |
| |
| **📢 注：相关数据集和论文将会在近日发布。** |
| |
| --- |
