layer8labs8 commited on
Commit
2f6ae41
·
verified ·
1 Parent(s): 0846bbc

Correct benchmark claims: EvalPlus vs official base model

Browse files
Files changed (1) hide show
  1. README.md +12 -15
README.md CHANGED
@@ -17,28 +17,25 @@ tags:
17
 
18
  **OrdoPy-Coder-0.5B** is a compact 0.5B parameter specialist model tuned on `Qwen/Qwen2.5-Coder-0.5B-Instruct` exclusively for **Python programming**, **English instruction following**, and **step-by-step logical reasoning**.
19
 
20
- Focusing the full capacity of the 0.5B backbone on a single coherent skill cluster (**English + Logic + Verified Python**) delivers strong functional code execution accuracy while remaining fast and deployable in < 1.5 GB VRAM (`float16`).
21
 
22
- ## Official Execution Benchmarks (`Pass@1` — 421 Tasks)
23
 
24
- Evaluated greedily (`do_sample=False`) inside isolated `3.0s`-timeout `sys.executable -I` subprocesses against full unit-test suites (`openai/openai_humaneval` `164` tasks + `google-research-datasets/mbpp` `sanitized` `257` test tasks):
25
 
26
- | Benchmark Suite | Baseline (`Qwen2.5-Coder-0.5B-Instruct`) | `OrdoPy-Coder-0.5B` | Gain |
27
  | :--- | :---: | :---: | :---: |
28
- | **HumanEval (`164` tasks)** | `9.76%` (`16/164`) | **`40.85%` (`67/164`)** | **`+31.09%` (`4.19x`)** |
29
- | **MBPP Sanitized Test (`257` tasks)** | `36.58%` (`94/257`) | **`46.30%` (`119/257`)** | **`+9.72%` (`+25` tasks)** |
30
- | **Combined (`421` tasks)** | `26.13%` (`110/421`) | **`44.18%` (`186/421`)** | **`+18.05%` (`+76` tasks)** |
 
31
 
32
- ### Official EvalPlus Suite (`evalplus` Greedy `Pass@1`)
33
 
34
- | Benchmark Suite | `OrdoPy-Coder-0.5B` `Pass@1` | Passed Tasks |
35
- | :--- | :---: | :---: |
36
- | **HumanEval (`base`)** | **`48.8%`** (`0.4878`) | `80 / 164` |
37
- | **HumanEval+ (`base + extra tests`)** | **`46.3%`** (`0.4634`) | `76 / 164` |
38
- | **MBPP (`base`)** | **`48.4%`** (`0.4841`) | `183 / 378` |
39
- | **MBPP+ (`base + extra tests`)** | **`42.1%`** (`0.4206`) | `159 / 378` |
40
 
41
- ## Held-Out Domain Evaluation
42
 
43
  | Domain (Held-Out Split) | Baseline Loss (PPL / Token Acc) | `OrdoPy-Coder-0.5B` Loss (PPL / Token Acc) |
44
  | :--- | :---: | :---: |
 
17
 
18
  **OrdoPy-Coder-0.5B** is a compact 0.5B parameter specialist model tuned on `Qwen/Qwen2.5-Coder-0.5B-Instruct` exclusively for **Python programming**, **English instruction following**, and **step-by-step logical reasoning**.
19
 
20
+ It runs in under 1.5 GB of VRAM (`float16`). **Note: on EvalPlus it currently scores below its base model (see the correction below).**
21
 
22
+ ## Benchmarks (EvalPlus, greedy `pass@1`)
23
 
24
+ Scored with the official [EvalPlus](https://github.com/evalplus/evalplus) harness (`evalplus 0.4.0`), compared with the official numbers for the base model ([Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16):
25
 
26
+ | Benchmark | `Qwen2.5-Coder-0.5B-Instruct` (official) | `OrdoPy-Coder-0.5B` | Delta |
27
  | :--- | :---: | :---: | :---: |
28
+ | **HumanEval** | `61.6%` | `48.8%` (`80 / 164`) | `-12.8` |
29
+ | **HumanEval+** | `57.3%` | `46.3%` (`76 / 164`) | `-11.0` |
30
+ | **MBPP** | `52.4%` | `48.4%` (`183 / 378`) | `-4.0` |
31
+ | **MBPP+** | `43.7%` | `42.1%` (`159 / 378`) | `-1.6` |
32
 
33
+ > **Correction (November 2026).** An earlier version of this card reported HumanEval rising from `9.76%` to `40.85%` (`4.19x`). That comparison was wrong: the in-house evaluation cut generations at 224 new tokens, which truncated many of the base model's longer answers, and the checkpoint was selected on a slice of the test sets. Measured with EvalPlus, **this fine-tune scores below its base model on all four benchmarks**. If you need the strongest 0.5B coder, use [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) directly. A follow-up experiment (rejection-sampling fine-tune with LoRA on the base model's own test-verified solutions) did not degrade the model but also gave no clear gain (HumanEval `+3.1`, HumanEval+ `+1.8`, MBPP `-1.0`, MBPP+ `-3.2` vs the base measured in the same run); details in the [OrdoTrain repository](https://github.com/layer8labs8/OrdoTrain).
34
 
35
+ ## Held-Out Domain Evaluation (token-level loss)
36
+
37
+ Token-level loss on held-out splits improved slightly, but this did not translate into better pass@1 (see above).
 
 
 
38
 
 
39
 
40
  | Domain (Held-Out Split) | Baseline Loss (PPL / Token Acc) | `OrdoPy-Coder-0.5B` Loss (PPL / Token Acc) |
41
  | :--- | :---: | :---: |