README.md

---
language:
- tr
- en
license: apache-2.0
base_model: Qwen/Qwen2.5-1.5B
tags:
- axiom
- qwen
- qwen2
- fine-tuned
- lora
- sft
- trl
- code
- python
- text-generation
pipeline_tag: text-generation
model_type: qwen2
library_name: transformers
---

# Axiom Python 1.5B

**Axiom Python 1.5B** is a text generation (causal language model) fine-tuned on [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) with a focus on Python programming and code generation.

The model was trained using **LoRA + SFT** with the [TRL](https://github.com/huggingface/trl) library on the [CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K) and [PythonCodeInstruct_18K](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca) datasets.

## Model Details

| Property | Value |
|---|---|
| Base Model | [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) |
| Architecture | Qwen2ForCausalLM |
| Parameters | ~1.5B |
| Hidden Layers | 28 |
| Hidden Size | 1536 |
| Attention Heads | 12 |
| KV Heads | 2 |
| Vocabulary Size | 151936 |
| Max Context Length | 131072 |
| Weight Dtype | float16 (FP16) |
| Training Method | LoRA (r=16, alpha=32) + SFT |
| Datasets | CodeAlpaca_20K + PythonCodeInstruct_18K |
| Languages | Turkish and English (code-focused) |

## Installation

Install the following packages to get started:

```bash
pip install transformers torch
```

> If you are using a GPU, make sure you have installed a CUDA-compatible PyTorch version.

## Usage

### 1. Using `pipeline` (Simplest Way)

```python
from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="coderian/axiom-python-1.5B",
    device_map="auto",
    torch_dtype="auto",
)

prompt = """### Instruction:
Write a Python function that reverses the elements of a list.

### Answer:
"""

output = generator(
    prompt,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
)

print(output[0]["generated_text"])
```

### 2. Using `AutoModelForCausalLM`

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "coderian/axiom-python-1.5B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

model.eval()

prompt = """### Instruction:
Write a Python function that adds two numbers.

### Answer:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)
```

### 3. Using the Chat Template

Since the Qwen2.5 tokenizer supports the ChatML format, you can also use the model for chat-style conversations:

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "coderian/axiom-python-1.5B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
)

messages = [
    {"role": "system", "content": "You are Axiom, a helpful Python coding assistant."},
    {"role": "user", "content": "Write a Python function to check if a number is prime."},
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
    )

response = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[1]:],
    skip_special_tokens=True,
)

print(response)
```

### Recommended Generation Parameters

| Parameter | Suggested Value | Description |
|---|---|---|
| `max_new_tokens` | `512` | Maximum number of new tokens to generate |
| `temperature` | `0.7` | Lower values produce more deterministic output |
| `top_p` | `0.9` | Nucleus sampling ratio |
| `do_sample` | `True` | Enable/disable sampling |
| `repetition_penalty` | `1.05` | Reduces repetitive output |

## Training Details

| Setting | Value |
|---|---|
| Base Model | Qwen/Qwen2.5-1.5B |
| LoRA Rank (r) | 16 |
| LoRA Alpha | 32 |
| LoRA Dropout | 0.05 |
| Target Modules | q_proj, v_proj |
| Batch Size | 32 (2 x 4 grad. accumulation) |
| Training Epochs | 1 |
| Learning Rate | 2e-4 |
| Optimizer | AdamW (fused) |
| Precision | FP16 |
| Steps | 4000 |
| Max Sequence Length | 256 |
| Adapter Location | `axiom-python-1.5B/checkpoint-4000` |

After training, the LoRA adapter was merged into the base model and released as a single file. You can also load the adapter directly using the peft library:

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B",
    torch_dtype="auto",
    device_map="auto",
)

model = PeftModel.from_pretrained(base, "path/to/adapter")
```

## Limitations

- It is a small 1.5B parameter model and may make mistakes on very complex and long code generation tasks.
- It was trained only on Python-focused datasets; performance in other languages is limited.
- The training data has a maximum length of 256 tokens; consistency may degrade in very long contexts.
- Generated code may not always be correct or safe. Review it before running.
- It may contain known limitations inherited from the training data regarding bias and harmful content.

## Intended Usage Tips

- It performs best on single-line and medium-complexity Python functions.
- Lower the `temperature` value if you want stable output for code generation.
- Since the model was trained in a completion format, the `### Instruction:` / `### Answer:` template yields the highest quality output.
- For batched inference, remember to set `tokenizer.pad_token = tokenizer.eos_token`.

## License

The base model Qwen2.5 is released under the Apache-2.0 license, and this model is also shared under the **Apache-2.0** license.

## Resources

- Base Model: [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)
- Training Library: [TRL](https://github.com/huggingface/trl)
- Dataset 1: [HuggingFaceH4/CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K)
- Dataset 2: [iamtarun/python_code_instructions_18k_alpaca](https://huggingface.co/datasets/iamtarun/python_code_instructions_18k_alpaca)