Add PhAI-IDE-72B model card

README.md

@@ -0,0 +1,134 @@
+---
+model_name: PhAI-IDE-72B
+base_model: Qwen/Qwen2.5-72B-Instruct
+base_model_relation: finetune
+datasets:
+- AItonomy/ScienceIDE
+library_name: transformers
+pipeline_tag: text-generation
+license: other
+license_name: qwen
+license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/495f39366efef23836d0cfae4fbe635880d2be31/LICENSE
+tags:
+- phai-ide
+- science
+- code
+- tool-use
+- sft
+- lora
+- safetensors
+---
+
+# PhAI-IDE-72B
+
+**PhAI-IDE-72B** is a supervised fine-tune of [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) for scientific coding and interaction with tools, trained using [ms-swift](https://github.com/modelscope/ms-swift). The release contains full BF16 weights with the final LoRA adapter merged.
+
+The training data is sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE). This model uses **Codex-only repaired trajectories v3 for 3 epochs**.
+
+## Quick start
+
+Use Transformers 5.16.1, PyTorch and Accelerate. The example loads the model, configuration and tokenizer directly from this repository.
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "leyili6666/PhAI-IDE-72B"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, dtype="bfloat16", device_map="auto",
+)
+inputs = tokenizer.apply_chat_template(
+    [{"role": "user", "content": "Explain how to verify a numerical simulation."}],
+    add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",
+).to(model.device)
+output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
+```
+
+BF16 weights occupy approximately 145.41 GB, plus runtime memory and KV cache.
+
+## Benchmark performance
+
+### Score improvements
+
+Scores (%); gains in percentage points.
+
+| Benchmark | Qwen/Qwen2.5-72B-Instruct | PhAI-IDE-72B | Gain (pp) |
+| --- | ---: | ---: | ---: |
+| ARC-Easy | 83.33 | **84.64** | **+1.30** |
+| ARC-Challenge | 63.05 | **64.42** | **+1.37** |
+| OpenBookQA | 48.80 | **49.20** | **+0.40** |
+| LiveCodeBench execution-v2 | 58.46 | **60.13** | **+1.67** |
+| BIG-bench CS algorithms | 75.76 | **82.58** | **+6.82** |
+
+### Comparison with published models
+
+Scores (%); evaluation settings vary by source.
+
+| Benchmark | Reference model | Parameters | Published score | PhAI-IDE-72B result | Source |
+| --- | --- | ---: | ---: | ---: | --- |
+| GSM8K | Llama-3-70B-Instruct | 70B | 93 | **93.75** | [Meta model card](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct#instruction-tuned-models) |
+| GSM8K | DeepSeek-LLM-67B-Chat | 67B | 84.1 | **93.75** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM#3-evaluation-results) |
+| GSM8K | Qwen2-72B-Instruct | 72B | 93.2 | **93.75** | [Qwen2.5 report, Table 6](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) |
+| GSM8K | SciTulu-70B | 70B | 67.5 | **93.75** | [SciRIFF report, Table 7](https://arxiv.org/html/2406.07835v2#A3) |
+| GSM8K | WizardMath-Llama-RL (Llama 2) | 70B | 92.8 | **93.75** | [WizardMath report, Tables 1 & 15](https://arxiv.org/html/2308.09583v2) |
+| ARC-Easy | DeepSeek-LLM-67B-Chat | 67B | 81.6 | **84.64** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) |
+| ARC-Challenge | DeepSeek-LLM-67B-Chat | 67B | 64.1 | **64.42** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) |
+
+## Training procedure
+
+ScienceIDE demonstrations were collected with **GPT-5.6-sol** and filtered using a **numerical-equivalence verifier**. They capture code inspection, tool use, and responses to execution feedback.
+
+Training uses **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges the final checkpoint's adapter into the base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The Codex-only data retains the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.
+
+
+| Setting | Value |
+| --- | --- |
+| Training subset | Codex-only repaired trajectories v3 |
+| Training examples / tasks | 4,567 segments / 564 tasks |
+| Validation examples / tasks | 544 segments / 81 tasks |
+| Train/validation task overlap | 0 |
+| Training epochs | 3 |
+| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
+| Released weights | LoRA merged into BF16 Safetensors |
+
+Long trajectories are organized into segments. Source partition assignments are preserved, with no task identifiers shared between training and validation.
+
+### Framework versions
+
+The release was validated with the following environment.
+
+| Component | Version |
+| --- | --- |
+| Python | 3.11 |
+| ms-swift | 4.5.3 |
+| Transformers | 5.16.1 |
+| PyTorch | 2.6.0+cu124 |
+| PEFT | 0.20.0 |
+| Datasets | 4.8.4 |
+| Tokenizers | 0.23.2 |
+| Accelerate | 1.14.0 |
+
+## Citations
+
+The training framework is [ms-swift](https://github.com/modelscope/ms-swift), and adapter training uses [LoRA](https://arxiv.org/abs/2106.09685).
+
+```bibtex
+@misc{zhao2024swift,
+  title = {SWIFT: A Scalable lightWeight Infrastructure for Fine-Tuning},
+  author = {Yuze Zhao and others},
+  year = {2024},
+  eprint = {2408.05517},
+  archivePrefix = {arXiv},
+  url = {https://arxiv.org/abs/2408.05517}
+}
+
+@misc{hu2021lora,
+  title = {LoRA: Low-Rank Adaptation of Large Language Models},
+  author = {Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
+  year = {2021},
+  eprint = {2106.09685},
+  archivePrefix = {arXiv},
+  url = {https://arxiv.org/abs/2106.09685}
+}
+```