Simplify published comparisons to complete benchmark configurations

README.md

@@ -17,24 +17,61 @@ tags:
 - safetensors
 ---
 
-# PhAI-IDE-4B
+# PhAI-IDE
 
-**PhAI-IDE-4B** is a supervised fine-tune of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) for scientific coding and interaction with tools, trained using [ms-swift](https://github.com/modelscope/ms-swift). The release contains full BF16 weights with the final LoRA adapter merged.
+**PhAI-IDE** is a family of models for scientific coding and interaction with tools, available in **4B, 9B, and 72B** sizes. Each model is supervised fine-tuned with [ms-swift](https://github.com/modelscope/ms-swift) and released as full BF16 weights with the final LoRA adapter merged, together with its configuration and tokenizer.
 
-The training data is sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE).
+The **training dataset is Codex trajectories**, sourced from [AItonomy/ScienceIDE](https://huggingface.co/datasets/AItonomy/ScienceIDE).
+
+## Models
+
+| Model | Base model | BF16 weights | License |
+| --- | --- | ---: | --- |
+| [PhAI-IDE-4B](https://huggingface.co/AItonomy/PhAI-IDE-4B) | [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) | 9.08 GB | Apache-2.0 |
+| [PhAI-IDE-9B](https://huggingface.co/AItonomy/PhAI-IDE-9B) | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) | 18.82 GB | Apache-2.0 |
+| [PhAI-IDE-72B](https://huggingface.co/AItonomy/PhAI-IDE-72B) | [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) | 145.41 GB | [Qwen](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/495f39366efef23836d0cfae4fbe635880d2be31/LICENSE) |
+
+Weight sizes are approximate; inference also requires memory for runtime allocations and the KV cache.
+
+## ScienceAccelBench performance
+
+**PhAI-IDE-9B** improves the task-pooled pass rate from **17.09% to 20.89% (+3.80 percentage points)** across all **27 ScienceAccelBench environments**, using **158 valid paired tasks**.
+
+Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Each row compares the initial model with its corresponding fine-tuned model on identical tasks. Pass rates are percentages; gains are percentage points.
+
+| Model | Environment | Tasks | Initial model | PhAI-IDE | Gain (pp) |
+| --- | --- | ---: | ---: | ---: | ---: |
+| 4B | PLUTO-Particles-Dust | 3 | 0.00 | **33.33** | **+33.33** |
+| 9B | LAPS | 16 | 31.25 | **50.00** | **+18.75** |
+| 9B | MITgcm-biogeo | 8 | 0.00 | **12.50** | **+12.50** |
+| 9B | PLUTO-RMHD | 7 | 0.00 | **28.57** | **+28.57** |
+
+## Comparison with published models
+
+Scores (%), grouped by benchmark and model size. Each reference entry gives its published score and the **PhAI-IDE score difference in percentage points**. Reference models are approximately the same size: 3–4B, 7–9B, and 67–72B, respectively.
+
+| PhAI-IDE | Benchmark | Score | Reference models: score (difference) |
+| --- | --- | ---: | --- |
+| 4B | BBH multistep-arithmetic-two | **97.60** | [Llama-3.2-3B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.21B): 53.2 (**+44.40**); [Phi-3.5-mini-8k-instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (3.82B): 95.6 (**+2.00**) |
+| 9B | BBH word-sorting | **60.40** | [Llama-3.1-8B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (8.03B): 51.2 (**+9.20**); [Qwen2.5-7B-Instruct](https://huggingface.co/spaces/steampunque/benchlm/blob/d45e8600172857935610426f797a4429f2f136d6/README.md) (7.62B): 15.6 (**+44.80**) |
+| 9B | MATH-500 | **92.20** | [InternLM3-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 83 (**+9.20**); [Qwen2.5-7B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (7B): 72.4 (**+19.80**); [Llama-3.1-8B-Instruct](https://modelscope.cn/models/Shanghai_AI_Laboratory/internlm3-8b-instruct-gptq-int4) (8B): 48.4 (**+43.80**) |
+| 72B | AQuA-RAT | **77.56** | [Llama-2-70B-Chat](https://openreview.net/pdf?id=FvfhHucpLd) (70B): 31.32 (**+46.24**) |
+| 72B | ARC-Easy | **84.64** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 76.5 (**+8.14**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 81.6 (**+3.04**) |
+| 72B | ARC-Challenge | **64.42** | [Llama-2-70B](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (70B): 59.5 (**+4.92**); [DeepSeek-LLM-67B-Chat](https://github.com/deepseek-ai/DeepSeek-LLM/blob/main/evaluation/more_results.md) (67B): 64.1 (**+0.32**) |
+
+Reference scores come from the linked publications, model cards, and independent evaluation reports; evaluation settings and sample counts vary by source. Differences describe reported scores across evaluations, rather than matched-protocol head-to-head gains. BBH entries refer to the named tasks.
 
 ## Quick start
 
-Use Transformers 5.16.1, PyTorch and Accelerate. The example loads the model, configuration and tokenizer directly from this repository.
+Use Transformers 5.16.1, PyTorch and Accelerate. Set `model_id` to any model in the table above; the example selects the matching model class.
 
 ```python
-from transformers import AutoTokenizer, AutoModelForImageTextToText
+from transformers import AutoTokenizer, AutoModelForCausalLM, AutoModelForImageTextToText
 
 model_id = "AItonomy/PhAI-IDE-4B"
+loader = AutoModelForCausalLM if model_id.endswith("72B") else AutoModelForImageTextToText
 tokenizer = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForImageTextToText.from_pretrained(
-    model_id, dtype="bfloat16", device_map="auto",
-)
+model = loader.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
 inputs = tokenizer.apply_chat_template(
     [{"role": "user", "content": "Explain how to verify a numerical simulation."}],
     add_generation_prompt=True, enable_thinking=False, return_dict=True, return_tensors="pt",
@@ -43,25 +80,13 @@ output = model.generate(**inputs, max_new_tokens=128, do_sample=False)
 print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
 ```
 
-BF16 weights occupy approximately 9.08 GB, plus runtime memory and KV cache.
-
-## ScienceAccelBench performance
-
-Task-held-out, localized scientific-code repair on familiar codebases, with original numerical verification. Pass rates (%); gains in percentage points.
-
-| Environment | Tasks | Qwen3.5-4B | PhAI-IDE-4B | Gain (pp) |
-| --- | ---: | ---: | ---: | ---: |
-| pluto-particles-dust | 3 | 0.00 | **33.33** | **+33.33** |
-
 ## Training procedure
 
 ScienceIDE demonstrations were collected with **GPT-5.6-sol** and filtered using a **numerical-equivalence verifier**. They capture code inspection, tool use, and responses to execution feedback.
 
-Training uses **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges the final checkpoint's adapter into the base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The Codex-only data retains the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.
-
-Fine-tuning uses text trajectories with the vision tower and aligner frozen. The merged export supports standard autoregressive Qwen3.5 generation.
+All three models use **ms-swift** supervised fine-tuning with **LoRA across trainable linear layers for three epochs**. The release merges each final checkpoint's adapter into its base model. Retained assistant targets provide the next-token training signal, while conversation history and tool observations provide context. The trajectories retain the native `exec` / `wait` interaction format. Heuristic target masking selects assistant actions for supervision while preserving the surrounding interaction history.
 
-| Setting | Value |
+| Shared setting | Value |
 | --- | --- |
 | Training dataset | Codex trajectories |
 | Training examples / tasks | 4,567 segments / 564 tasks |
@@ -70,6 +95,13 @@ Fine-tuning uses text trajectories with the vision tower and aligner frozen. The
 | Training epochs | 3 |
 | LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
 | Released weights | LoRA merged into BF16 Safetensors |
+
+Long trajectories are organized into segments. Source partition assignments are preserved, with no task identifiers shared between training and validation.
+
+For **4B and 9B**, fine-tuning uses text trajectories with the vision tower and aligner frozen. Their additional settings are:
+
+| Setting | 4B and 9B |
+| --- | --- |
 | Maximum training sequence length | 36,864 tokens |
 | Learning rate / schedule | 2e-5 / cosine |
 | Warmup ratio | 0.03 |
@@ -78,8 +110,6 @@ Fine-tuning uses text trajectories with the vision tower and aligner frozen. The
 | Optimizer steps | 216 |
 | Packing | Disabled |
 
-Long trajectories are organized into segments. Source partition assignments are preserved, with no task identifiers shared between training and validation.
-
 ### Framework versions
 
 The release was validated with the following environment.