---
base_model: Qwen/Qwen2.5-0.5B-Instruct
library_name: peft
license: apache-2.0
datasets:
- bernabepuente/python-instruction-dataset
language:
- en
pipeline_tag: text-generation
tags:
- lora
- peft
- sft
- python
- code
- qwen2
---

# Qwen2.5-0.5B-Instruct · Python assistant LoRA (learning project)

A LoRA adapter for [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), supervised fine-tuned on the small
[bernabepuente/python-instruction-dataset](https://huggingface.co/datasets/bernabepuente/python-instruction-dataset) dataset to answer Python questions: writing code from instructions,
explaining concepts, and fixing bugs.

**This is a practical learning project, not a state-of-the-art coding model.** It's a 0.5B-parameter model trained on about 259 examples.
Larger coding models (e.g. Qwen2.5-Coder) will do much better. Always run and test the code it produces.

## How to use

```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

repo = "ali4566544/qwen2.5-0.5b-instruct-python-lora"
model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(repo)

messages = [
    {"role": "system", "content": 'You are a helpful Python coding assistant. Write clear, correct, idiomatic Python, explain the code concisely, and point out bugs and how to fix them when asked.'},
    {"role": "user", "content": "Write a function that removes duplicates from a list while keeping order."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
```

The adapter was trained **with the system prompt above**. Use it for coding questions. Without it, the model behaves closer to the base model.
To merge the weights into a standalone model: `model.merge_and_unload().save_pretrained(...)`.

## Training data

* Source: `bernabepuente/python-instruction-dataset`, 462 rows. Fields used: `instruction`, optional `input` (non-empty in only 10 cleaned rows), and `output`.
  **The `thought` field was not used.**
* Cleaning (335 rows kept): dropped 6 empty/very short outputs,
  22 truncated outputs (unbalanced code fences), 61 outputs
  cut off after a trailing `:`, 25 outputs with raw code that couldn't be fenced safely,
  13 outputs whose `python` code blocks fail `ast.parse`, and
  0 exact duplicates. In 68 outputs, raw code that
  parses as Python was wrapped in ```` ```python ```` fences.
* Split (seed 42): instructions were clustered by word overlap (Jaccard ≥ 0.6) so that paraphrased questions never straddle splits, giving
  train 259 / validation 42 / test 34 rows. The exact `row_id`s are in `split.json`.
* **Topic coverage is narrow** (keyword counts on cleaned instructions):

| Topic | Examples |
|---|---|
| exceptions / errors | 88 |
| web (FastAPI/Flask/Django) | 77 |
| pandas / DataFrames | 64 |
| logging / monitoring | 55 |
| Docker / deployment | 51 |
| async / asyncio | 2 |
| debugging / fixing bugs | 2 |
| decorators | 1 |
| testing | 1 |
| type hints | 0 |

Async, decorators, type hints and debugging are barely represented, so the fine-tune can't have learned much about them from this data.
The reference answers were **not** verified for correctness beyond the Python syntax check. The dataset declares **no license**.
Check its terms before reusing this adapter commercially.

## Training procedure

LoRA (r=16, alpha=32, dropout=0.05) on all attention and MLP projections. Loss only on assistant tokens.
lr 0.0001 (cosine, warmup 0.05), effective batch
16, up to 3 epochs with early stopping on validation loss
(best checkpoint kept: eval loss 1.3612 at step 40 of 51).
Max length 1536 tokens, fp16 mixed precision on Tesla T4, 1.6 min.
Software: torch 2.11.0+cu130, transformers 4.57.1, peft 0.17.1, datasets 4.8.5.

![loss curve](loss_curve.png)

## Evaluation

Base = the same model with the adapter disabled. Greedy decoding, same prompts. Full details are in `eval_results.json`.

| Metric | Base | Fine-tuned |
|---|---|---|
| Test loss on reference answers ↓ | 1.490 | 1.241 |
| Test perplexity ↓ | 4.43 | 3.46 |
| ROUGE-L F1 vs reference ↑ | 0.134 | 0.346 |
| Answers containing a fenced code block | 76.5% | 38.2% |
| Generated `python` blocks that parse ↑ | 100.0% of 43 blocks | 100.0% of 12 blocks |
| Answers finished within token limit ↑ | 29.4% | 100.0% |
| Mean answer length (chars) | 2023 | 299 |
| **Executable benchmark (pass@1, greedy)** ↑ | 11/16 | 7/16 |
|   type hints | 2/2 | 2/2 |
|   decorators | 2/2 | 0/2 |
|   async | 0/2 | 0/2 |
|   pandas | 2/3 | 2/3 |
|   exceptions | 2/2 | 2/2 |
|   debugging | 1/3 | 0/3 |
|   basics | 2/2 | 1/2 |
| General probes, auto-checks ↑ | 4/4 | 3/4 |

* **Test loss / ROUGE-L** use the 34 held-out dataset questions. They measure how closely the model matches the dataset's answers and style, *not* whether the code is correct.
  The test questions come from the same narrow topics as training, so they're in-distribution.
* **Executable benchmark**: 16 small hand-written tasks with unit tests (type hints, decorators, async, pandas, exceptions, debugging, basics).
  Each task is worth 6.2 points, so a difference of 1–2 tasks is within noise. This is *not* HumanEval/MBPP.
  For a standard number, run e.g. [EvalPlus](https://github.com/evalplus/evalplus) on both models.
* **General probes**: 6 everyday questions without the coding system prompt, used to spot obvious damage to general chat ability. That's a smoke test, not a benchmark.

## Limitations

* **Tiny, narrow dataset.** About 259 training examples, concentrated on Docker, pandas, exceptions, logging and web frameworks. Expect little or no improvement on other topics.
* **Small model.** 0.5B parameters. It makes logic errors, invents APIs and package options, and can write code that looks right but isn't. Review and test everything.
* **Unverified references.** The training answers come from a public dataset that wasn't checked for correctness or security. The model may reproduce its mistakes,
  e.g. outdated Dockerfile practices.
* **Limited evaluation.** The small test set, a 16-task hand-written benchmark and 6 general probes can't capture regressions on other skills or languages.
* English only. No additional safety tuning beyond what the base model has.

## Reproducibility

The notebook that produced this adapter fixes seed 42, records library versions, and saves the data split (`split.json`) and every metric (`eval_results.json`).