LoRA adapter, tokenizer, model card and evaluation results

5ecafff verified 7 days ago

6.76 kB

---
base_model: Qwen/Qwen2.5-0.5B-Instruct
library_name: peft
license: apache-2.0
datasets:
- bernabepuente/python-instruction-dataset
language:
- en
pipeline_tag: text-generation
tags:
- lora
- peft
- sft
- python
- code
- qwen2
---
# Qwen2.5-0.5B-Instruct · Python assistant LoRA (learning project)

A LoRA adapter for [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), supervised fine-tuned on the small
[bernabepuente/python-instruction-dataset](https://huggingface.co/datasets/bernabepuente/python-instruction-dataset) dataset to answer Python questions: writing code from instructions,
explaining concepts, and fixing bugs.

**This is a practical learning project, not a state-of-the-art coding model.** It's a 0.5B-parameter model trained on about 259 examples.
Larger coding models (e.g. Qwen2.5-Coder) will do much better. Always run and test the code it produces.

## How to use

```python
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer
repo = "ali4566544/qwen2.5-0.5b-instruct-python-lora"
model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(repo)
messages = [
    {"role": "system", "content": 'You are a helpful Python coding assistant. Write clear, correct, idiomatic Python, explain the code concisely, and point out bugs and how to fix them when asked.'},
    {"role": "user", "content": "Write a function that removes duplicates from a list while keeping order."},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
```

The adapter was trained **with the system prompt above**. Use it for coding questions. Without it, the model behaves closer to the base model.
To merge the weights into a standalone model: `model.merge_and_unload().save_pretrained(...)`.

## Training data

* Source: `bernabepuente/python-instruction-dataset`, 462 rows. Fields used: `instruction`, optional `input` (non-empty in only 10 cleaned rows), and `output`.
  **The `thought` field was not used.**
* Cleaning (335 rows kept): dropped 6 empty/very short outputs,
  22 truncated outputs (unbalanced code fences), 61 outputs
  cut off after a trailing `:`, 25 outputs with raw code that couldn't be fenced safely,
  13 outputs whose `python` code blocks fail `ast.parse`, and
  0 exact duplicates. In 68 outputs, raw code that
  parses as Python was wrapped in ```` ```python ```` fences.
* Split (seed 42): instructions were clustered by word overlap (Jaccard ≥ 0.6) so that paraphrased questions never straddle splits, giving
  train 259 / validation 42 / test 34 rows. The exact `row_id`s are in `split.json`.
* **Topic coverage is narrow** (keyword counts on cleaned instructions):

| Topic | Examples |
|---|---|
| exceptions / errors | 88 |
| web (FastAPI/Flask/Django) | 77 |
| pandas / DataFrames | 64 |
| logging / monitoring | 55 |
| Docker / deployment | 51 |
| async / asyncio | 2 |
| debugging / fixing bugs | 2 |
| decorators | 1 |
| testing | 1 |
| type hints | 0 |

Async, decorators, type hints and debugging are barely represented, so the fine-tune can't have learned much about them from this data.
The reference answers were **not** verified for correctness beyond the Python syntax check. The dataset declares **no license**.
Check its terms before reusing this adapter commercially.

## Training procedure

LoRA (r=16, alpha=32, dropout=0.05) on all attention and MLP projections. Loss only on assistant tokens.
lr 0.0001 (cosine, warmup 0.05), effective batch
16, up to 3 epochs with early stopping on validation loss
(best checkpoint kept: eval loss 1.3612 at step 40 of 51).
Max length 1536 tokens, fp16 mixed precision on Tesla T4, 1.6 min.
Software: torch 2.11.0+cu130, transformers 4.57.1, peft 0.17.1, datasets 4.8.5.

![loss curve](loss_curve.png)

## Evaluation

Base = the same model with the adapter disabled. Greedy decoding, same prompts. Full details are in `eval_results.json`.

| Metric | Base | Fine-tuned |
|---|---|---|
| Test loss on reference answers ↓ | 1.490 | 1.241 |
| Test perplexity ↓ | 4.43 | 3.46 |
| ROUGE-L F1 vs reference ↑ | 0.134 | 0.346 |
| Answers containing a fenced code block | 76.5% | 38.2% |
| Generated `python` blocks that parse ↑ | 100.0% of 43 blocks | 100.0% of 12 blocks |
| Answers finished within token limit ↑ | 29.4% | 100.0% |
| Mean answer length (chars) | 2023 | 299 |
| **Executable benchmark (pass@1, greedy)** ↑ | 11/16 | 7/16 |
| &nbsp;&nbsp;type hints | 2/2 | 2/2 |
| &nbsp;&nbsp;decorators | 2/2 | 0/2 |
| &nbsp;&nbsp;async | 0/2 | 0/2 |
| &nbsp;&nbsp;pandas | 2/3 | 2/3 |
| &nbsp;&nbsp;exceptions | 2/2 | 2/2 |
| &nbsp;&nbsp;debugging | 1/3 | 0/3 |
| &nbsp;&nbsp;basics | 2/2 | 1/2 |
| General probes, auto-checks ↑ | 4/4 | 3/4 |

* **Test loss / ROUGE-L** use the 34 held-out dataset questions. They measure how closely the model matches the dataset's answers and style, *not* whether the code is correct.
  The test questions come from the same narrow topics as training, so they're in-distribution.
* **Executable benchmark**: 16 small hand-written tasks with unit tests (type hints, decorators, async, pandas, exceptions, debugging, basics).
  Each task is worth 6.2 points, so a difference of 1–2 tasks is within noise. This is *not* HumanEval/MBPP.
  For a standard number, run e.g. [EvalPlus](https://github.com/evalplus/evalplus) on both models.
* **General probes**: 6 everyday questions without the coding system prompt, used to spot obvious damage to general chat ability. That's a smoke test, not a benchmark.

## Limitations

* **Tiny, narrow dataset.** About 259 training examples, concentrated on Docker, pandas, exceptions, logging and web frameworks. Expect little or no improvement on other topics.
* **Small model.** 0.5B parameters. It makes logic errors, invents APIs and package options, and can write code that looks right but isn't. Review and test everything.
* **Unverified references.** The training answers come from a public dataset that wasn't checked for correctness or security. The model may reproduce its mistakes,
  e.g. outdated Dockerfile practices.
* **Limited evaluation.** The small test set, a 16-task hand-written benchmark and 6 general probes can't capture regressions on other skills or languages.
* English only. No additional safety tuning beyond what the base model has.

## Reproducibility

The notebook that produced this adapter fixes seed 42, records library versions, and saves the data split (`split.json`) and every metric (`eval_results.json`).

	---
	base_model: Qwen/Qwen2.5-0.5B-Instruct
	library_name: peft
	license: apache-2.0
	datasets:
	- bernabepuente/python-instruction-dataset
	language:
	- en
	pipeline_tag: text-generation
	tags:
	- lora
	- peft
	- sft
	- python
	- code
	- qwen2
	---

	# Qwen2.5-0.5B-Instruct · Python assistant LoRA (learning project)

	A LoRA adapter for [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct), supervised fine-tuned on the small
	[bernabepuente/python-instruction-dataset](https://huggingface.co/datasets/bernabepuente/python-instruction-dataset) dataset to answer Python questions: writing code from instructions,
	explaining concepts, and fixing bugs.

	This is a practical learning project, not a state-of-the-art coding model. It's a 0.5B-parameter model trained on about 259 examples.
	Larger coding models (e.g. Qwen2.5-Coder) will do much better. Always run and test the code it produces.

	## How to use

	```python
	from peft import AutoPeftModelForCausalLM
	from transformers import AutoTokenizer

	repo = "ali4566544/qwen2.5-0.5b-instruct-python-lora"
	model = AutoPeftModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
	tokenizer = AutoTokenizer.from_pretrained(repo)

	messages = [
	{"role": "system", "content": 'You are a helpful Python coding assistant. Write clear, correct, idiomatic Python, explain the code concisely, and point out bugs and how to fix them when asked.'},
	{"role": "user", "content": "Write a function that removes duplicates from a list while keeping order."},
	]
	inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
	out = model.generate(inputs, max_new_tokens=512, do_sample=False)
	print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
	```

	The adapter was trained with the system prompt above. Use it for coding questions. Without it, the model behaves closer to the base model.
	To merge the weights into a standalone model: `model.merge_and_unload().save_pretrained(...)`.

	## Training data

	* Source: `bernabepuente/python-instruction-dataset`, 462 rows. Fields used: `instruction`, optional `input` (non-empty in only 10 cleaned rows), and `output`.
	The `thought` field was not used.
	* Cleaning (335 rows kept): dropped 6 empty/very short outputs,
	22 truncated outputs (unbalanced code fences), 61 outputs
	cut off after a trailing `:`, 25 outputs with raw code that couldn't be fenced safely,
	13 outputs whose `python` code blocks fail `ast.parse`, and
	0 exact duplicates. In 68 outputs, raw code that
	parses as Python was wrapped in ```` ```python ```` fences.
	* Split (seed 42): instructions were clustered by word overlap (Jaccard ≥ 0.6) so that paraphrased questions never straddle splits, giving
	train 259 / validation 42 / test 34 rows. The exact `row_id`s are in `split.json`.
	* Topic coverage is narrow (keyword counts on cleaned instructions):

	\| Topic \| Examples \|
	\|---\|---\|
	\| exceptions / errors \| 88 \|
	\| web (FastAPI/Flask/Django) \| 77 \|
	\| pandas / DataFrames \| 64 \|
	\| logging / monitoring \| 55 \|
	\| Docker / deployment \| 51 \|
	\| async / asyncio \| 2 \|
	\| debugging / fixing bugs \| 2 \|
	\| decorators \| 1 \|
	\| testing \| 1 \|
	\| type hints \| 0 \|

	Async, decorators, type hints and debugging are barely represented, so the fine-tune can't have learned much about them from this data.
	The reference answers were not verified for correctness beyond the Python syntax check. The dataset declares no license.
	Check its terms before reusing this adapter commercially.

	## Training procedure

	LoRA (r=16, alpha=32, dropout=0.05) on all attention and MLP projections. Loss only on assistant tokens.
	lr 0.0001 (cosine, warmup 0.05), effective batch
	16, up to 3 epochs with early stopping on validation loss
	(best checkpoint kept: eval loss 1.3612 at step 40 of 51).
	Max length 1536 tokens, fp16 mixed precision on Tesla T4, 1.6 min.
	Software: torch 2.11.0+cu130, transformers 4.57.1, peft 0.17.1, datasets 4.8.5.

	![loss curve](loss_curve.png)

	## Evaluation

	Base = the same model with the adapter disabled. Greedy decoding, same prompts. Full details are in `eval_results.json`.

	\| Metric \| Base \| Fine-tuned \|
	\|---\|---\|---\|
	\| Test loss on reference answers ↓ \| 1.490 \| 1.241 \|
	\| Test perplexity ↓ \| 4.43 \| 3.46 \|
	\| ROUGE-L F1 vs reference ↑ \| 0.134 \| 0.346 \|
	\| Answers containing a fenced code block \| 76.5% \| 38.2% \|
	\| Generated `python` blocks that parse ↑ \| 100.0% of 43 blocks \| 100.0% of 12 blocks \|
	\| Answers finished within token limit ↑ \| 29.4% \| 100.0% \|
	\| Mean answer length (chars) \| 2023 \| 299 \|
	\| Executable benchmark (pass@1, greedy) ↑ \| 11/16 \| 7/16 \|
	\|   type hints \| 2/2 \| 2/2 \|
	\|   decorators \| 2/2 \| 0/2 \|
	\|   async \| 0/2 \| 0/2 \|
	\|   pandas \| 2/3 \| 2/3 \|
	\|   exceptions \| 2/2 \| 2/2 \|
	\|   debugging \| 1/3 \| 0/3 \|
	\|   basics \| 2/2 \| 1/2 \|
	\| General probes, auto-checks ↑ \| 4/4 \| 3/4 \|

	* Test loss / ROUGE-L use the 34 held-out dataset questions. They measure how closely the model matches the dataset's answers and style, not whether the code is correct.
	The test questions come from the same narrow topics as training, so they're in-distribution.
	* Executable benchmark: 16 small hand-written tasks with unit tests (type hints, decorators, async, pandas, exceptions, debugging, basics).
	Each task is worth 6.2 points, so a difference of 1–2 tasks is within noise. This is not HumanEval/MBPP.
	For a standard number, run e.g. [EvalPlus](https://github.com/evalplus/evalplus) on both models.
	* General probes: 6 everyday questions without the coding system prompt, used to spot obvious damage to general chat ability. That's a smoke test, not a benchmark.

	## Limitations

	* Tiny, narrow dataset. About 259 training examples, concentrated on Docker, pandas, exceptions, logging and web frameworks. Expect little or no improvement on other topics.
	* Small model. 0.5B parameters. It makes logic errors, invents APIs and package options, and can write code that looks right but isn't. Review and test everything.
	* Unverified references. The training answers come from a public dataset that wasn't checked for correctness or security. The model may reproduce its mistakes,
	e.g. outdated Dockerfile practices.
	* Limited evaluation. The small test set, a 16-task hand-written benchmark and 6 general probes can't capture regressions on other skills or languages.
	* English only. No additional safety tuning beyond what the base model has.

	## Reproducibility

	The notebook that produced this adapter fixes seed 42, records library versions, and saves the data split (`split.json`) and every metric (`eval_results.json`).