OrdoPy-Coder-0.5B / README.md
layer8labs8's picture
v2: rejection-sampling LoRA fine-tune (EvalPlus avg 52.2 vs v1 46.4)
0028567 verified
|
Raw History Blame Contribute Delete
3.71 kB
---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen2
- code-generation
- python
- lora
- rejection-sampling
---
# OrdoPy-Coder-0.5B (v2)
**OrdoPy-Coder-0.5B v2** is `Qwen/Qwen2.5-Coder-0.5B-Instruct` fine-tuned with a rejection-sampling recipe: the base model wrote 4 solutions for each of 2,000 [KodCode](https://huggingface.co/datasets/KodCode/KodCode-V1-SFT-4o) problems, every solution was executed against the problem's unit tests, and only passing solutions were kept (max 2 per problem; GPT-4o reference solutions filled in for unsolved problems, capped at 20% of the data). A LoRA adapter (r=32, all linear layers) was trained on these 1,106 examples and merged; the epoch was selected on MBPP `train + validation`, never on test data. It runs in about 1 GB of VRAM (`float16`).
## Benchmarks (EvalPlus, greedy `pass@1`)
Base and v2 were scored by the official [EvalPlus](https://github.com/evalplus/evalplus) harness in the **same run** (Kaggle 2x T4, `float16`):
| Benchmark | Base, official¹ | Base, same run | **v2 (this model)** | v1 (previous upload) |
| :--- | :---: | :---: | :---: | :---: |
| HumanEval | `61.6` | `57.9` | **`61.0`** | `48.8` |
| HumanEval+ | `57.3` | `53.7` | **`55.5`** | `46.3` |
| MBPP | `52.4` | `52.1` | `51.1` | `48.4` |
| MBPP+ | `43.7` | `44.2` | `41.0` | `42.1` |
| **Average** | — | `52.0` | **`52.2`** | `46.4` |
¹ [Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16 (Qwen's own evaluation setup).
**Honest summary:** v2 is clearly better than v1 (+5.8 points on average) and **about on par with the base model** (better on HumanEval / HumanEval+, worse on MBPP / MBPP+; single seed, differences of this size are close to run-to-run noise). It does not beat the base model overall.
> **About v1.** The first upload is kept under the `v1` tag of this repository (`AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", revision="v1")`). It was a supervised fine-tune that scores below the base model on all four benchmarks. Its card claimed HumanEval `9.76% -> 40.85%`; that comparison was a measurement error (a 224-token generation cap truncated the base model's answers, and the checkpoint was selected on test slices). `eval_benchmarks.json` in this repository is v1 training telemetry. Full v2 report and code: [OrdoTrain](https://github.com/layer8labs8/OrdoTrain) (`results/ordopy_rft_lora_report.json`).
## Model Details
- **Architecture**: `Qwen2ForCausalLM` (24 layers, hidden size 896, 494M parameters, 32K context)
- **Base model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct`
- **Fine-tuning**: LoRA r=32 / alpha=64 on all linear layers, lr 1e-4, 1 epoch kept, assistant-only loss, merged into the weights
- **Format**: `model.safetensors` (`float16`) + Qwen ChatML tokenizer
## Quickstart
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "layer8labs8/OrdoPy-Coder-0.5B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto")
messages = [{"role": "user", "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` that ignores spaces and punctuation."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```
On `transformers` < 4.56 use `torch_dtype=` instead of `dtype=`.