--- license: apache-2.0 base_model: Qwen/Qwen2.5-Coder-0.5B-Instruct language: - en library_name: transformers pipeline_tag: text-generation tags: - qwen2 - code-generation - python - lora - rejection-sampling --- # OrdoPy-Coder-0.5B (v2) **OrdoPy-Coder-0.5B v2** is `Qwen/Qwen2.5-Coder-0.5B-Instruct` fine-tuned with a rejection-sampling recipe: the base model wrote 4 solutions for each of 2,000 [KodCode](https://huggingface.co/datasets/KodCode/KodCode-V1-SFT-4o) problems, every solution was executed against the problem's unit tests, and only passing solutions were kept (max 2 per problem; GPT-4o reference solutions filled in for unsolved problems, capped at 20% of the data). A LoRA adapter (r=32, all linear layers) was trained on these 1,106 examples and merged; the epoch was selected on MBPP `train + validation`, never on test data. It runs in about 1 GB of VRAM (`float16`). ## Benchmarks (EvalPlus, greedy `pass@1`) Base and v2 were scored by the official [EvalPlus](https://github.com/evalplus/evalplus) harness in the **same run** (Kaggle 2x T4, `float16`): | Benchmark | Base, official¹ | Base, same run | **v2 (this model)** | v1 (previous upload) | | :--- | :---: | :---: | :---: | :---: | | HumanEval | `61.6` | `57.9` | **`61.0`** | `48.8` | | HumanEval+ | `57.3` | `53.7` | **`55.5`** | `46.3` | | MBPP | `52.4` | `52.1` | `51.1` | `48.4` | | MBPP+ | `43.7` | `44.2` | `41.0` | `42.1` | | **Average** | — | `52.0` | **`52.2`** | `46.4` | ¹ [Qwen2.5-Coder Technical Report](https://arxiv.org/abs/2409.12186), Table 16 (Qwen's own evaluation setup). **Honest summary:** v2 is clearly better than v1 (+5.8 points on average) and **about on par with the base model** (better on HumanEval / HumanEval+, worse on MBPP / MBPP+; single seed, differences of this size are close to run-to-run noise). It does not beat the base model overall. > **About v1.** The first upload is kept under the `v1` tag of this repository (`AutoModelForCausalLM.from_pretrained("layer8labs8/OrdoPy-Coder-0.5B", revision="v1")`). It was a supervised fine-tune that scores below the base model on all four benchmarks. Its card claimed HumanEval `9.76% -> 40.85%`; that comparison was a measurement error (a 224-token generation cap truncated the base model's answers, and the checkpoint was selected on test slices). `eval_benchmarks.json` in this repository is v1 training telemetry. Full v2 report and code: [OrdoTrain](https://github.com/layer8labs8/OrdoTrain) (`results/ordopy_rft_lora_report.json`). ## Model Details - **Architecture**: `Qwen2ForCausalLM` (24 layers, hidden size 896, 494M parameters, 32K context) - **Base model**: `Qwen/Qwen2.5-Coder-0.5B-Instruct` - **Fine-tuning**: LoRA r=32 / alpha=64 on all linear layers, lr 1e-4, 1 epoch kept, assistant-only loss, merged into the weights - **Format**: `model.safetensors` (`float16`) + Qwen ChatML tokenizer ## Quickstart ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "layer8labs8/OrdoPy-Coder-0.5B" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.float16, device_map="auto") messages = [{"role": "user", "content": "Write a Python function `is_palindrome_clean(s: str) -> bool` that ignores spaces and punctuation."}] text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) inputs = tokenizer([text], return_tensors="pt").to(model.device) out = model.generate(**inputs, max_new_tokens=512, do_sample=False) print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)) ``` On `transformers` < 4.56 use `torch_dtype=` instead of `dtype=`.