Qwen2.5-Coder-1.5B-Python-OCI-LoRA

A Python code-generation model: Qwen/Qwen2.5-Coder-1.5B-Instruct fine-tuned with QLoRA on 10,000 test-verified Python examples from nvidia/OpenCodeInstruct. The LoRA weights are merged into the base model, so this is a standalone model and does not need PEFT.

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "2003Rohansinha/Qwen2.5-Coder-1.5B-Python-OCI-LoRA"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.float16, device_map="auto")

messages = [
    {"role": "system", "content": "You are an expert Python programmer. Answer with correct, complete, runnable Python code."},
    {"role": "user", "content": "Write a function that merges two sorted lists."},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
enc = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
  • Use the system message above (it was used in training). The chat template is the standard Qwen (ChatML) one.
  • Needs transformers>=4.37. About 3 GB of memory in fp16; on CPU use torch_dtype=torch.float32.
  • Greedy decoding (do_sample=False) is recommended for code.

Example

Prompt: Write a Python function that checks whether a string is a palindrome, ignoring punctuation and case.

```python
import string

def is_palindrome(s):
    """
    Determines if the given string is a palindrome, ignoring punctuation and case.
    
    Parameters:
    s (str): The string to check.
    
    Returns:
    bool: True if the string is a palindrome, False otherwise.
    """
    # Remove punctuation using str.translate and str.maketrans
    translator = str.maketrans('', '', string.punctuation)
    cleaned_string = s.translate(translator).lower()
    
    # Check if the cleaned string is equal to its reverse
    return cleaned_string == cleaned_string[::-1]

# Example usage:
print(is_palindrome("A man, a plan, a canal: Panama"))  # Output: True
print(is_palindrome("No lemon, no melon"))              # Output: True
print(is_palindrome("Hello, World!"))                   # Output: False

## Intended use and limitations

**Intended for:** generating Python functions, classes and small scripts from a natural-language description;
learning, prototyping and research on small code models.

**Not intended for:** unreviewed production use, security-critical code, or other languages (training was Python-only).

- It is a small (1.5B) model trained on synthetic, LLM-generated data. It can produce wrong, inefficient or insecure
  code and may confidently invent APIs. **Always review and test its output.**
- Training sequences were truncated to 1,024 tokens, so very long prompts or answers may degrade.
- It inherits the biases and limitations of the base model and of the synthetic dataset.

## Evaluation

HumanEval pass@1, greedy decoding, same prompt and system message for both models (164 problems):

| Model | HumanEval pass@1 |
|---|---|
| Qwen/Qwen2.5-Coder-1.5B-Instruct (base) | 68.9% |
| **Qwen2.5-Coder-1.5B-Python-OCI-LoRA** (this model) | **67.7%** (-1.2 points) |

This is a single run with my own harness (the model's reply is parsed from a code block and executed against the
official tests), so absolute numbers may differ from other leaderboards. A difference of a few points on 164 problems
is within noise. Final validation loss on 200 held-out training-distribution examples: 0.133.

## Training details

| | |
|---|---|
| Base model | Qwen/Qwen2.5-Coder-1.5B-Instruct |
| Method | QLoRA: 4-bit NF4 base, LoRA r=16, alpha=32, dropout 0.05, on all attention and MLP projections (18.5M trainable parameters, 1.2%) |
| Data | 10,000 Python examples from OpenCodeInstruct, keeping only answers whose generated unit tests all passed (average test score 1.0) and that contain a Python code block; max 1,024 tokens |
| Loss | On the assistant's answer only (prompt tokens masked) |
| Epochs | 1 (about 613 steps) |
| Effective batch size | 16 (2 GPUs x 4 x 2 accumulation) |
| Optimizer / LR | paged AdamW 8-bit, 2e-4, cosine schedule, 40 warmup steps |
| Precision / hardware | fp16, 2 x NVIDIA T4 (Kaggle), about 1 hour |
| Final training loss | about 0.13 to 0.15 (mean over the run: 0.145) |
| Final validation loss | 0.133 |
| Frameworks | transformers, peft, bitsandbytes |

## License and attribution

- **This model:** Apache-2.0, inherited from the base model.
- **Training data:** [nvidia/OpenCodeInstruct](https://huggingface.co/datasets/nvidia/OpenCodeInstruct) by NVIDIA, licensed CC BY 4.0.
  This model was trained on a subset of it.
- **Base model:** Qwen2.5-Coder by the Qwen team, Alibaba Cloud.

## Citation

If you use this model, please also cite the dataset and the base model:

```bibtex
@article{ahmad2025opencodeinstruct,
  title={OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs},
  author={Ahmad, Wasi Uddin and Ficek, Aleksander and Samadi, Mehrzad and Huang, Jocelyn and Noroozi, Vahid and Majumdar, Somshubra and Ginsburg, Boris},
  journal={arXiv preprint arXiv:2504.04030},
  year={2025}
}

@article{hui2024qwen2,
  title={Qwen2.5-Coder Technical Report},
  author={Hui, Binyuan and others},
  journal={arXiv preprint arXiv:2409.12186},
  year={2024}
}
Downloads last month
105
Safetensors
Model size
2B params
Tensor type
F16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for 2003Rohansinha/Qwen2.5-Coder-1.5B-Python-OCI-LoRA

Adapter
(237)
this model

Dataset used to train 2003Rohansinha/Qwen2.5-Coder-1.5B-Python-OCI-LoRA

Papers for 2003Rohansinha/Qwen2.5-Coder-1.5B-Python-OCI-LoRA