File size: 3,710 Bytes
0846bbc
 
 
 
 
 
 
 
 
 
 
0028567
 
0846bbc
 
0028567
0846bbc
0028567
0846bbc
2f6ae41
0846bbc
0028567
0846bbc
0028567
 
 
 
 
 
 
0846bbc
0028567
2f6ae41
0028567
0846bbc
0028567
0846bbc
 
 
0028567
 
 
 
0846bbc
0028567
0846bbc
 
 
 
 
 
 
0028567
0846bbc
0028567
0846bbc
 
0028567
0846bbc
 
0028567
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
language:
  - en
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen2
  - code-generation
  - python
  - lora
  - rejection-sampling
---

# OrdoPy-Coder-0.5B (v2)

**OrdoPy-Coder-0.5B v2** is `Qwen/Qwen2.5-Coder-0.5B-Instruct` fine-tuned with a rejection-sampling recipe: the base model wrote 4 solutions for each of 2,000 [KodCode](https://huggingface.co/datasets/KodCode/KodCode-V1-SFT-4o) problems, every solution was executed against the problem's unit tests, and only passing solutions were kept (max 2 per problem; GPT-4o reference solutions filled in for unsolved problems, capped at 20% of the data). A LoRA adapter (r=32, all linear layers) was trained on these 1,106 examples and merged; the epoch was selected on MBPP `train + validation`, never on test data. It runs in about 1 GB of VRAM (`float16`).

## Benchmarks (EvalPlus, greedy `pass@1`)

Base and v2 were scored by the official [EvalPlus](https://github.com/evalplus/evalplus) harness in the **same run** (Kaggle 2x T4, `float16`):

| Benchmark | Base, official¹ | Base, same run | **v2 (this model)** | v1 (previous upload) |
| :--- | :---: | :---: | :---: | :---: |
| HumanEval | `61.6` | `57.9` | **`61.0`** | `48.8` |
| HumanEval+ | `57.3` | `53.7` | **`55.5`** | `46.3` |
| MBPP | `52.4` | `52.1` | `51.1` | `48.4` |
| MBPP+ | `43.7` | `44.2` | `41.0` | `42.1` |
| **Average** | — | `52.0` | **`52.2`** | `46.4` |

¹ [Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16 (Qwen's own evaluation setup).

**Honest summary:** v2 is clearly better than v1 (+5.8 points on average) and **about on par with the base model** (better on HumanEval / HumanEval+, worse on MBPP / MBPP+; single seed, differences of this size are close to run-to-run noise). It does not beat the base model overall.

> **About v1.** The first upload is kept under the `v1` tag of this repository (`AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", revision="v1")`). It was a supervised fine-tune that scores below the base model on all four benchmarks. Its card claimed HumanEval `9.76% -> 40.85%`; that comparison was a measurement error (a 224-token generation cap truncated the base model's answers, and the checkpoint was selected on test slices). `eval_benchmarks.json` in this repository is v1 training telemetry. Full v2 report and code: [OrdoTrain](https://github.com/layer8labs8/OrdoTrain) (`results/ordopy_rft_lora_report.json`).

## Model Details

- **Architecture**: `Qwen2ForCausalLM` (24 layers, hidden size 896, 494M parameters, 32K context)
- **Base model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
- **Fine-tuning**: LoRA r=32 / alpha=64 on all linear layers, lr 1e-4, 1 epoch kept, assistant-only loss, merged into the weights
- **Format**: `model.safetensors` (`float16`) + Qwen ChatML tokenizer

## Quickstart

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "layer8labs8/OrdoPy-Coder-0.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto")

messages = [{"role": "user", "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` that ignores spaces and punctuation."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

On `transformers` < 4.56 use `torch_dtype=` instead of `dtype=`.