README.md

---
language:
  - ar
license: apache-2.0
tags:
  - qwen
  - llama-factory
  - lora
  - arabic
  - question-answering
  - instruction-tuning
  - kaggle
  - transformers
  - fine-tuned
model_name: QWEN_Arabic_Q&A
base_model: Qwen/Qwen2.5-1.5B
pipeline_tag: text-generation
library_name: transformers
datasets:
  - custom
---

# 🧠 Qwen2.5-1.5B - LoRA Fine-Tuned on Arabic Q&A 🕌

This model is a LoRA fine-tuned version of **[Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B)** designed for Arabic Question Answering tasks. It was trained using the **LLaMA-Factory** framework on a custom curated dataset of Arabic Q&A pairs.

## 📚 Training Configuration

- **Base Model**: `Qwen/Qwen2.5-1.5B`
- **Method**: Supervised Fine-Tuning (SFT) with [LoRA](https://arxiv.org/abs/2106.09685)
- **Framework**: [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory)
- **Batch Size**: 1 (gradient accumulation = 16)
- **Epochs**: 3
- **Cutoff Length**: 2048 tokens
- **Learning Rate**: 1e-4
- **Scheduler**: Cosine with warmup ratio 0.1
- **Precision**: bf16
- **LoRA Rank**: 64
- **LoRA Target**: all layers
- **Eval Strategy**: every 200 steps
- **Eval Set Size**: 3020 examples
- **WandB Tracking**: Enabled [`Run Link`](https://wandb.ai/youssefhassan437972-kafr-el-sheikh-university/llamafactory/runs/rdrftts8)

## 📈 Evaluation (Epoch ~1.77)

- **Eval Loss**: 0.4321
- **Samples/sec**: 1.389
- **Steps/sec**: 0.695

## 🚀 Usage

You can use the model via `transformers`:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Youssef/QWEN_Arabic_Q&A")
tokenizer = AutoTokenizer.from_pretrained("Youssef/QWEN_Arabic_Q&A")

prompt = "من هو مؤسس علم الجبر؟"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))


<|user|>
ما هي أركان الإسلام مع ذكر الحديث الذي يذكرها؟
<|assistant|>
أركان الإسلام خمسة، كما جاء في الحديث الصحيح:

عن عبد الله بن عمر رضي الله عنهما قال: قال رسول الله ﷺ: "بني الإسلام على خمس: شهادة أن لا إله إلا الله، وأن محمدًا رسول الله، وإقام الصلاة، وإيتاء الزكاة، وصوم رمضان، وحج البيت لمن استطاع إليه سبيلًا" (رواه البخاري ومسلم).

## 📉 Training Loss Over Epochs

| Epoch | Learning Rate | Loss   |
|-------|------------------------|--------|
| 0.16  | 5.39e-05               | 0.6304 |
| 0.18  | 5.88e-05               | 0.6179 |
| 0.19  | 6.37e-05               | 0.6042 |
| 0.21  | 6.86e-05               | 0.6138 |
| 0.22  | 7.35e-05               | 0.5940 |
| 0.24  | 7.84e-05               | 0.5838 |
| 0.25  | 8.33e-05               | 0.5842 |
| 0.26  | 8.82e-05               | 0.5786 |
| 0.28  | 9.31e-05               | 0.5713 |
| 0.65  | 9.60e-05               | 0.6122 |
| 0.71  | 9.45e-05               | 0.5809 |
| 0.77  | 9.29e-05               | 0.5446 |
| 0.82  | 9.10e-05               | 0.5339 |
| 0.88  | 8.90e-05               | 0.5296 |
| 0.94  | 8.67e-05               | 0.5176 |
| 1.00  | 8.43e-05               | 0.5104 |
| 1.06  | 8.17e-05               | 0.4685 |
| 1.12  | 7.90e-05               | 0.4730 |
| 1.18  | 7.62e-05               | 0.4679 |
| 1.24  | 7.32e-05               | 0.4541 |
| 1.30  | 7.01e-05               | 0.4576 |
| 1.35  | 6.69e-05               | 0.4472 |
| 1.41  | 6.36e-05               | 0.4427 |
| 1.47  | 6.03e-05               | 0.4395 |
| 1.53  | 5.69e-05               | 0.4305 |
| 1.59  | 5.35e-05               | 0.4280 |
| 1.65  | 5.01e-05               | 0.4251 |
| 1.71  | 4.67e-05               | 0.4188 |
| 1.77  | 4.33e-05               | 0.4177 |
| 1.83  | 3.99e-05               | 0.4128 |

**Evaluation Losses:**

- 📍 Epoch 1.18 → `0.4845`
- 📍 Epoch 1.77 → `0.4321`