layer8labs8 commited on
Commit
0028567
·
verified ·
1 Parent(s): 2f6ae41

v2: rejection-sampling LoRA fine-tune (EvalPlus avg 52.2 vs v1 46.4)

Browse files
Files changed (4) hide show
  1. README.md +25 -45
  2. config.json +57 -58
  3. model.safetensors +2 -2
  4. tokenizer.json +2 -2
README.md CHANGED
@@ -9,48 +9,40 @@ tags:
9
  - qwen2
10
  - code-generation
11
  - python
12
- - reasoning
13
- - coding
14
  ---
15
 
16
- # OrdoPy-Coder-0.5B
17
 
18
- **OrdoPy-Coder-0.5B** is a compact 0.5B parameter specialist model tuned on `Qwen/Qwen2.5-Coder-0.5B-Instruct` exclusively for **Python programming**, **English instruction following**, and **step-by-step logical reasoning**.
19
-
20
- It runs in under 1.5 GB of VRAM (`float16`). **Note: on EvalPlus it currently scores below its base model (see the correction below).**
21
 
22
  ## Benchmarks (EvalPlus, greedy `pass@1`)
23
 
24
- Scored with the official [EvalPlus](https://github.com/evalplus/evalplus) harness (`evalplus 0.4.0`), compared with the official numbers for the base model ([Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16):
25
-
26
- | Benchmark | `Qwen2.5-Coder-0.5B-Instruct` (official) | `OrdoPy-Coder-0.5B` | Delta |
27
- | :--- | :---: | :---: | :---: |
28
- | **HumanEval** | `61.6%` | `48.8%` (`80 / 164`) | `-12.8` |
29
- | **HumanEval+** | `57.3%` | `46.3%` (`76 / 164`) | `-11.0` |
30
- | **MBPP** | `52.4%` | `48.4%` (`183 / 378`) | `-4.0` |
31
- | **MBPP+** | `43.7%` | `42.1%` (`159 / 378`) | `-1.6` |
32
 
33
- > **Correction (November 2026).** An earlier version of this card reported HumanEval rising from `9.76%` to `40.85%` (`4.19x`). That comparison was wrong: the in-house evaluation cut generations at 224 new tokens, which truncated many of the base model's longer answers, and the checkpoint was selected on a slice of the test sets. Measured with EvalPlus, **this fine-tune scores below its base model on all four benchmarks**. If you need the strongest 0.5B coder, use [`Qwen/Qwen2.5-Coder-0.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct) directly. A follow-up experiment (rejection-sampling fine-tune with LoRA on the base model's own test-verified solutions) did not degrade the model but also gave no clear gain (HumanEval `+3.1`, HumanEval+ `+1.8`, MBPP `-1.0`, MBPP+ `-3.2` vs the base measured in the same run); details in the [OrdoTrain repository](https://github.com/layer8labs8/OrdoTrain).
 
 
 
 
 
 
34
 
35
- ## Held-Out Domain Evaluation (token-level loss)
36
 
37
- Token-level loss on held-out splits improved slightly, but this did not translate into better pass@1 (see above).
38
 
39
-
40
- | Domain (Held-Out Split) | Baseline Loss (PPL / Token Acc) | `OrdoPy-Coder-0.5B` Loss (PPL / Token Acc) |
41
- | :--- | :---: | :---: |
42
- | **`python_code`** (`MBPP Sanitized Val`) | `1.0226` (`2.78` / `76.44%`) | **`0.9843` (`2.68` / `78.40%`)** |
43
- | **`logic_math`** (`GSM8K Official Test`) | `1.4752` (`4.37` / `73.21%`) | **`1.4096` (`4.09` / `74.68%`)** |
44
- | **`english_debug`** (`Alpaca Held-Out`) | `1.6227` (`5.07` / `59.53%`) | **`1.4409` (`4.22` / `61.58%`)** |
45
 
46
  ## Model Details
47
 
48
- - **Architecture**: `Qwen2ForCausalLM` (`24` layers, `hidden_size=896`, `494M` parameters, `32K` context window)
49
- - **Base Model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
50
- - **Specialization**: Python code generation, algorithmic problem solving, debugging & step-by-step reasoning in English
51
- - **Format**: Safetensors (`model.safetensors`) + Qwen ChatML tokenizer template
52
 
53
- ## Quickstart Usage
54
 
55
  ```python
56
  import torch
@@ -58,25 +50,13 @@ from transformers import AutoModelForCausalLM, AutoTokenizer
58
 
59
  model_id = "layer8labs8/OrdoPy-Coder-0.5B"
60
  tokenizer = AutoTokenizer.from_pretrained(model_id)
61
- model = AutoModelForCausalLM.from_pretrained(
62
- model_id, torch_dtype=torch.float16, device_map="auto"
63
- )
64
 
65
- messages = [
66
- {
67
- "role": "system",
68
- "content": (
69
- "You are OrdoPy, an expert Python software engineer and logical problem solver. "
70
- "Think clearly step-by-step in English and write clean, bug-free Python code."
71
- ),
72
- },
73
- {
74
- "role": "user",
75
- "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` ignoring spaces and punctuation.",
76
- },
77
- ]
78
  text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
79
  inputs = tokenizer([text], return_tensors="pt").to(model.device)
80
- out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
81
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
82
  ```
 
 
 
9
  - qwen2
10
  - code-generation
11
  - python
12
+ - lora
13
+ - rejection-sampling
14
  ---
15
 
16
+ # OrdoPy-Coder-0.5B (v2)
17
 
18
+ **OrdoPy-Coder-0.5B v2** is `Qwen/Qwen2.5-Coder-0.5B-Instruct` fine-tuned with a rejection-sampling recipe: the base model wrote 4 solutions for each of 2,000 [KodCode](https://huggingface.co/datasets/KodCode/KodCode-V1-SFT-4o) problems, every solution was executed against the problem's unit tests, and only passing solutions were kept (max 2 per problem; GPT-4o reference solutions filled in for unsolved problems, capped at 20% of the data). A LoRA adapter (r=32, all linear layers) was trained on these 1,106 examples and merged; the epoch was selected on MBPP `train + validation`, never on test data. It runs in about 1 GB of VRAM (`float16`).
 
 
19
 
20
  ## Benchmarks (EvalPlus, greedy `pass@1`)
21
 
22
+ Base and v2 were scored by the official [EvalPlus](https://github.com/evalplus/evalplus) harness in the **same run** (Kaggle 2x T4, `float16`):
 
 
 
 
 
 
 
23
 
24
+ | Benchmark | Base, official¹ | Base, same run | **v2 (this model)** | v1 (previous upload) |
25
+ | :--- | :---: | :---: | :---: | :---: |
26
+ | HumanEval | `61.6` | `57.9` | **`61.0`** | `48.8` |
27
+ | HumanEval+ | `57.3` | `53.7` | **`55.5`** | `46.3` |
28
+ | MBPP | `52.4` | `52.1` | `51.1` | `48.4` |
29
+ | MBPP+ | `43.7` | `44.2` | `41.0` | `42.1` |
30
+ | **Average** | — | `52.0` | **`52.2`** | `46.4` |
31
 
32
+ ¹ [Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16 (Qwen's own evaluation setup).
33
 
34
+ **Honest summary:** v2 is clearly better than v1 (+5.8 points on average) and **about on par with the base model** (better on HumanEval / HumanEval+, worse on MBPP / MBPP+; single seed, differences of this size are close to run-to-run noise). It does not beat the base model overall.
35
 
36
+ > **About v1.** The first upload is kept under the `v1` tag of this repository (`AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", revision="v1")`). It was a supervised fine-tune that scores below the base model on all four benchmarks. Its card claimed HumanEval `9.76% -> 40.85%`; that comparison was a measurement error (a 224-token generation cap truncated the base model's answers, and the checkpoint was selected on test slices). `eval_benchmarks.json` in this repository is v1 training telemetry. Full v2 report and code: [OrdoTrain](https://github.com/layer8labs8/OrdoTrain) (`results/ordopy_rft_lora_report.json`).
 
 
 
 
 
37
 
38
  ## Model Details
39
 
40
+ - **Architecture**: `Qwen2ForCausalLM` (24 layers, hidden size 896, 494M parameters, 32K context)
41
+ - **Base model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
42
+ - **Fine-tuning**: LoRA r=32 / alpha=64 on all linear layers, lr 1e-4, 1 epoch kept, assistant-only loss, merged into the weights
43
+ - **Format**: `model.safetensors` (`float16`) + Qwen ChatML tokenizer
44
 
45
+ ## Quickstart
46
 
47
  ```python
48
  import torch
 
50
 
51
  model_id = "layer8labs8/OrdoPy-Coder-0.5B"
52
  tokenizer = AutoTokenizer.from_pretrained(model_id)
53
+ model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto")
 
 
54
 
55
+ messages = [{"role": "user", "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` that ignores spaces and punctuation."}]
 
 
 
 
 
 
 
 
 
 
 
 
56
  text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
57
  inputs = tokenizer([text], return_tensors="pt").to(model.device)
58
+ out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
59
  print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
60
  ```
61
+
62
+ On `transformers` < 4.56 use `torch_dtype=` instead of `dtype=`.
config.json CHANGED
@@ -1,58 +1,57 @@
1
- {
2
- "architectures": [
3
- "Qwen2ForCausalLM"
4
- ],
5
- "attention_dropout": 0.0,
6
- "bos_token_id": 151643,
7
- "dtype": "float32",
8
- "eos_token_id": 151645,
9
- "hidden_act": "silu",
10
- "hidden_size": 896,
11
- "initializer_range": 0.02,
12
- "intermediate_size": 4864,
13
- "layer_types": [
14
- "full_attention",
15
- "full_attention",
16
- "full_attention",
17
- "full_attention",
18
- "full_attention",
19
- "full_attention",
20
- "full_attention",
21
- "full_attention",
22
- "full_attention",
23
- "full_attention",
24
- "full_attention",
25
- "full_attention",
26
- "full_attention",
27
- "full_attention",
28
- "full_attention",
29
- "full_attention",
30
- "full_attention",
31
- "full_attention",
32
- "full_attention",
33
- "full_attention",
34
- "full_attention",
35
- "full_attention",
36
- "full_attention",
37
- "full_attention"
38
- ],
39
- "max_position_embeddings": 32768,
40
- "max_window_layers": 24,
41
- "model_type": "qwen2",
42
- "num_attention_heads": 14,
43
- "num_hidden_layers": 24,
44
- "num_key_value_heads": 2,
45
- "pad_token_id": null,
46
- "rms_norm_eps": 1e-06,
47
- "rope_parameters": {
48
- "rope_theta": 1000000.0,
49
- "rope_type": "default"
50
- },
51
- "sliding_window": null,
52
- "tie_word_embeddings": true,
53
- "transformers_version": "5.20.2",
54
- "use_cache": true,
55
- "use_sliding_window": false,
56
- "vocab_size": 151936,
57
- "torch_dtype": "float32"
58
- }
 
1
+ {
2
+ "architectures": [
3
+ "Qwen2ForCausalLM"
4
+ ],
5
+ "attention_dropout": 0.0,
6
+ "bos_token_id": 151643,
7
+ "dtype": "float16",
8
+ "eos_token_id": 151645,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 896,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 4864,
13
+ "layer_types": [
14
+ "full_attention",
15
+ "full_attention",
16
+ "full_attention",
17
+ "full_attention",
18
+ "full_attention",
19
+ "full_attention",
20
+ "full_attention",
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention"
38
+ ],
39
+ "max_position_embeddings": 32768,
40
+ "max_window_layers": 24,
41
+ "model_type": "qwen2",
42
+ "num_attention_heads": 14,
43
+ "num_hidden_layers": 24,
44
+ "num_key_value_heads": 2,
45
+ "pad_token_id": null,
46
+ "rms_norm_eps": 1e-06,
47
+ "rope_parameters": {
48
+ "rope_theta": 1000000.0,
49
+ "rope_type": "default"
50
+ },
51
+ "sliding_window": null,
52
+ "tie_word_embeddings": true,
53
+ "transformers_version": "5.20.2",
54
+ "use_cache": true,
55
+ "use_sliding_window": false,
56
+ "vocab_size": 151936
57
+ }
 
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c2a45803542e6f99d998f8beae68e1e0be97bc02379087d01fd2635cbac9a46c
3
- size 1976163472
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f40b18628c59ed66b26a69b44a2393dffd1dcf36d78baee90d45777951de34b
3
+ size 988097536
tokenizer.json CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:f7f96da3a872b5e901575b2067c744ad336c3a3d77a21584d20024557b1bd7f0
3
- size 11422059
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3fd169731d2cbde95e10bf356d66d5997fd885dd8dbb6fb4684da3f23b2585d8
3
+ size 11421892