Update README with verified ScienceAccelBench results

README.md

@@ -19,24 +19,61 @@ tags:
 - safetensors
 ---
 
-# PhAI-IDE-72B
+# PhAI-IDE
 
-**PhAI-IDE-72B** is a supervised fine-tune of [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) for scientific coding and interaction with tools, trained using [ms-swift](https://github.com/modelscope/ms-swift). The release contains full BF16 weights with the final LoRA adapter merged.
+**PhAI-IDE** is a family of models for scientific coding and interaction with tools, available in **4B, 9B, and 72B** sizes. Each model is supervised fine-tuned with [ms-swift](https://github.com/modelscope/ms-swift) and released as full BF16 weights with the final LoRA adapter merged, together with its configuration and tokenizer.
 
-The training data is sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE).
+The **training dataset is Codex trajectories**, sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE).
+
+## Models
+
+| Model | Base model | BF16 weights | License |
+| --- | --- | ---: | --- |
+| [PhAI-IDE-4B](https://huggingface.co/AItonomy/PhAI-IDE-4B) | [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | 9.08 GB | Apache-2.0 |
+| [PhAI-IDE-9B](https://huggingface.co/AItonomy/PhAI-IDE-9B) | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | 18.82 GB | Apache-2.0 |
+| [PhAI-IDE-72B](https://huggingface.co/AItonomy/PhAI-IDE-72B) | [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) | 145.41 GB | [Qwen](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/495f39366efef23836d0cfae4fbe635880d2be31/LICENSE) |
+
+Weight sizes are approximate; inference also requires memory for runtime allocations and the KV cache.
+
+## ScienceAccelBench performance
+
+**PhAI-IDE-9B** improves the task-pooled pass rate from **17.09% to 20.89% (+3.80 percentage points)** across all **27 ScienceAccelBench environments**, using **158 valid paired tasks**.
+
+Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Each row compares the initial model with its corresponding fine-tuned model on identical tasks. Pass rates are percentages; gains are percentage points.
+
+| Model | Environment | Tasks | Initial model | PhAI-IDE | Gain (pp) |
+| --- | --- | ---: | ---: | ---: | ---: |
+| 4B | PLUTO-Particles-Dust | 3 | 0.00 | **33.33** | **+33.33** |
+| 9B | LAPS | 16 | 31.25 | **50.00** | **+18.75** |
+| 9B | MITgcm-biogeo | 8 | 0.00 | **12.50** | **+12.50** |
+| 9B | PLUTO-RMHD | 7 | 0.00 | **28.57** | **+28.57** |
+
+## Comparison with published models
+
+Scores (%), grouped by benchmark and model size. Each reference entry gives its published score and the **PhAI-IDE score difference in percentage points**. Reference models are approximately the same size: 3–4B, 7–9B, and 67–72B, respectively.
+
+| PhAI-IDE | Benchmark | Score | Reference models: score (difference) |
+| --- | --- | ---: | --- |
+| 4B | BBH multistep-arithmetic-two | **97.60** | [Llama-3.2-3B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.21B): 53.2 (**+44.40**); [Phi-3.5-mini-8k-instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.82B): 95.6 (**+2.00**) |
+| 9B | BBH word-sorting | **60.40** | [Llama-3.1-8B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (8.03B): 51.2 (**+9.20**); [Qwen2.5-7B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (7.62B): 15.6 (**+44.80**) |
+| 9B | MATH-500 | **92.20** | [InternLM3-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 83 (**+9.20**); [Qwen2.5-7B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (7B): 72.4 (**+19.80**); [Llama-3.1-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 48.4 (**+43.80**) |
+| 72B | AQuA-RAT | **77.56** | [Llama-2-70B-Chat](https://openreview.net/pdf?id=FvfhHucpLd) (70B): 31.32 (**+46.24**) |
+| 72B | ARC-Easy | **84.64** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 76.5 (**+8.14**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 81.6 (**+3.04**) |
+| 72B | ARC-Challenge | **64.42** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 59.5 (**+4.92**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 64.1 (**+0.32**) |
+
+Reference scores come from the linked publications, model cards, and independent evaluation reports; evaluation settings and sample counts vary by source. Differences describe reported scores across evaluations, rather than matched-protocol head-to-head gains. BBH entries refer to the named tasks.
 
 ## Quick start
 
-Use Transformers 5.16.1, PyTorch and Accelerate. The example loads the model, configuration and tokenizer directly from this repository.
+Use Transformers 5.16.1, PyTorch and Accelerate. Set `model_id` to any model in the table above; the example selects the matching model class.
 
 ```python
-from transformers import AutoTokenizer, AutoModelForCausalLM
+from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForImageTextToText
 
-model_id = "AItonomy/PhAI-IDE-72B"
+model_id = "AItonomy/PhAI-IDE-4B"
+loader = AutoModelForCausalLM if model_id.endswith("72B") else AutoModelForImageTextToText
 tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(
-    model_id, dtype="bfloat16", device_map="auto",
-)
+model = loader.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
 inputs = tokenizer.apply_chat_template(
     [{"role": "user", "content": "Explain how to verify a numerical simulation."}],
     add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",
@@ -45,44 +82,13 @@ output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
 print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
 
-BF16 weights occupy approximately 145.41 GB, plus runtime memory and KV cache.
-
-## Benchmark performance
-
-### Score improvements
-
-Scores (%); gains in percentage points.
-
-| Benchmark | Qwen/Qwen2.5-72B-Instruct | PhAI-IDE-72B | Gain (pp) |
-| --- | ---: | ---: | ---: |
-| ARC-Easy | 83.33 | **84.64** | **+1.30** |
-| ARC-Challenge | 63.05 | **64.42** | **+1.37** |
-| OpenBookQA | 48.80 | **49.20** | **+0.40** |
-| LiveCodeBench execution-v2 | 58.46 | **60.13** | **+1.67** |
-| BIG-bench CS algorithms | 75.76 | **82.58** | **+6.82** |
-
-### Comparison with published models
-
-Scores (%); evaluation settings vary by source.
-
-| Benchmark | Reference model | Parameters | Published score | PhAI-IDE-72B result | Source |
-| --- | --- | ---: | ---: | ---: | --- |
-| GSM8K | Llama-3-70B-Instruct | 70B | 93 | **93.75** | [Meta model card](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct#instruction-tuned-models) |
-| GSM8K | DeepSeek-LLM-67B-Chat | 67B | 84.1 | **93.75** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM#3-evaluation-results) |
-| GSM8K | Qwen2-72B-Instruct | 72B | 93.2 | **93.75** | [Qwen2.5 report, Table 6](https://arxiv.org/html/2412.15115v2#S5.SS2.SSS1) |
-| GSM8K | SciTulu-70B | 70B | 67.5 | **93.75** | [SciRIFF report, Table 7](https://arxiv.org/html/2406.07835v2#A3) |
-| GSM8K | WizardMath-Llama-RL (Llama 2) | 70B | 92.8 | **93.75** | [WizardMath report, Tables 1 & 15](https://arxiv.org/html/2308.09583v2) |
-| ARC-Easy | DeepSeek-LLM-67B-Chat | 67B | 81.6 | **84.64** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) |
-| ARC-Challenge | DeepSeek-LLM-67B-Chat | 67B | 64.1 | **64.42** | [DeepSeek official results](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) |
-
 ## Training procedure
 
 ScienceIDE demonstrations were collected with **GPT-5.6-sol** and filtered using a **numerical-equivalence verifier**. They capture code inspection, tool use, and responses to execution feedback.
 
-Training uses **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges the final checkpoint's adapter into the base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The Codex-only data retains the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.
-
+All three models use **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges each final checkpoint's adapter into its base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The trajectories retain the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.
 
-| Setting | Value |
+| Shared setting | Value |
 | --- | --- |
 | Training dataset | Codex trajectories |
 | Training examples / tasks | 4,567 segments / 564 tasks |
@@ -94,6 +100,18 @@ Training uses **ms-swift** supervised fine-tuning with **LoRA across trainable l
 
 Long trajectories are organized into segments. Source partition assignments are preserved, with no task identifiers shared between training and validation.
 
+For **4B and 9B**, fine-tuning uses text trajectories with the vision tower and aligner frozen. Their additional settings are:
+
+| Setting | 4B and 9B |
+| --- | --- |
+| Maximum training sequence length | 36,864 tokens |
+| Learning rate / schedule | 2e-5 / cosine |
+| Warmup ratio | 0.03 |
+| Batch size per device / accumulation | 1 / 16 |
+| LoRA targets | Linear modules in the unfrozen language model |
+| Optimizer steps | 216 |
+| Packing | Disabled |
+
 ### Framework versions
 
 The release was validated with the following environment.