Training in progress, step 100

README.md

@@ -1,81 +1,59 @@
 ---
 base_model: Qwen/Qwen2.5-3B-Instruct
-library_name: peft
-datasets:
-- shaikhsalman/ZabaanAI-Urdu-3B-data
-language:
-- ur
-pipeline_tag: text-generation
+library_name: transformers
+model_name: ZabaanAI-Urdu-3B
 tags:
-- urdu
-- qlora
-- conversational
-- qwen2.5
-- ml-intern
+- generated_from_trainer
+- trackio:https://shaikhsalman-ml-intern-zabaanai.hf.space?project=zabaanai-urdu-3b&runs=sft_qwen2.5-3b_urdu_v4-clean-urdu-58k_lr0.0001&sidebar=collapsed
+- trl
+- sft
+licence: license
 ---
 
-# ZabaanAI-Urdu-3B (v4)
+# Model Card for ZabaanAI-Urdu-3B
 
-A LoRA adapter fine-tuning **Qwen/Qwen2.5-3B-Instruct** into a conversational **Urdu** AI assistant.
+This model is a fine-tuned version of [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct).
+It has been trained using [TRL](https://github.com/huggingface/trl).
 
-## v4 — Clean Urdu-Script Data (current)
+## Quick start
 
-v3 produced gibberish under sampling because its data contained **Roman-Urdu poetry**
-(Latin script), teaching the model to mix scripts. v4 fixes this:
+```python
+from transformers import pipeline
 
-- **Dataset:** [shaikhsalman/ZabaanAI-Urdu-3B-data](https://huggingface.co/datasets/shaikhsalman/ZabaanAI-Urdu-3B-data) — ~58K train / 1.8K test, all **native Urdu script**, conversational `messages` format.
-  - Core: [large-traversaal/urdu-instruct](https://huggingface.co/datasets/large-traversaal/urdu-instruct) (the *Alif* paper data — reasoning, QA, ethics, generation, classification, sentiment, translation)
-  - Chat turns: [Najaf/urdu_chat_makhzan](https://huggingface.co/datasets/Najaf/urdu_chat_makhzan) (capped at 10K)
-  - Quality gate: rows whose assistant answer is <85% Urdu script were dropped (translation/reasoning handled specially).
-- **Recipe:** QLoRA (4-bit nf4), r=16, alpha=32, dropout=0.05, target_modules=all-linear.
-- **Training:** 2 epochs (~7,286 steps), LR 1e-4 cosine, **assistant-only loss** (trains only on assistant turns), eff. batch 16.
-- **Hardware:** free Kaggle/Colab T4 (fp16, no FlashAttention). ~3h.
+question = "If you had a time machine, but could only go to the past or the future once and never return, which would you choose and why?"
+generator = pipeline("text-generation", model="shaikhsalman/ZabaanAI-Urdu-3B", device="cuda")
+output = generator([{"role": "user", "content": question}], max_new_tokens=128, return_full_text=False)[0]
+print(output["generated_text"])
+```
 
-## Usage
+## Training procedure
 
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
-from peft import PeftModel
+ 
+[<img src="https://raw.githubusercontent.com/gradio-app/trackio/refs/heads/main/trackio/assets/badge.png" alt="Visualize in Trackio" title="Visualize in Trackio" width="150" height="24"/>](https://shaikhsalman-ml-intern-zabaanai.hf.space?project=zabaanai-urdu-3b&runs=sft_qwen2.5-3b_urdu_v4-clean-urdu-58k_lr0.0001&sidebar=collapsed)
 
-tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-3B-Instruct")
-model = AutoModelForCausalLM.from_pretrained(
-    "Qwen/Qwen2.5-3B-Instruct",
-    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16),
-    device_map="auto")
-model = PeftModel.from_pretrained(model, "shaikhsalman/ZabaanAI-Urdu-3B")
-model.eval()
 
-msgs = [
-    {"role": "system", "content": "آپ ایک ذہین اردو AI اسسٹنٹ ہیں۔"},
-    {"role": "user", "content": "پاکستان کا دارالحکومت کیا ہے؟"},
-]
-text = tok.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)
-inputs = tok(text, return_tensors="pt").to(model.device)
-out = model.generate(**inputs, max_new_tokens=200, do_sample=True,
-                     temperature=0.5, top_p=0.9, repetition_penalty=1.2)
-print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
-```
+This model was trained with SFT.
 
-> Tip: use moderate temperature (0.3–0.6) + `repetition_penalty=1.2` for best Urdu coherence.
+### Framework versions
 
-## Training (reproduce on Kaggle/Colab T4)
-
-```python
-!pip install -q trl peft transformers accelerate datasets bitsandbytes trackio
-import os; os.environ["HF_TOKEN"] = "hf_YOUR_TOKEN"
-!rm -f train_colab_gpu.py
-!wget -q https://huggingface.co/shaikhsalman/ZabaanAI-Urdu-3B/resolve/main/train_colab_gpu.py
-!python train_colab_gpu.py
-```
+- TRL: 1.5.1
+- Transformers: 5.0.0
+- Pytorch: 2.10.0+cu128
+- Datasets: 4.8.3
+- Tokenizers: 0.22.2
 
-The script resumes across sessions via `training_progress.json` (target 7,286 steps).
-Healthy training loss is **0.8–1.5**; if loss dives below ~0.4 it is memorizing (the v3 failure mode).
+## Citations
 
-<!-- ml-intern-provenance -->
-## Generated by ML Intern
 
-This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.
 
-- Try ML Intern: https://smolagents-ml-intern.hf.space
-- Source code: https://github.com/huggingface/ml-intern
+Cite TRL as:
+    
+```bibtex
+@software{vonwerra2020trl,
+  title   = {{TRL: Transformers Reinforcement Learning}},
+  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},
+  license = {Apache-2.0},
+  url     = {https://github.com/huggingface/trl},
+  year    = {2020}
+}
+```

adapter_config.json

@@ -18,7 +18,7 @@
   "loftq_config": {},
   "lora_alpha": 32,
   "lora_bias": false,
-  "lora_dropout": 0.1,
+  "lora_dropout": 0.05,
   "megatron_config": null,
   "megatron_core": "megatron.core",
   "modules_to_save": null,
@@ -29,13 +29,13 @@
   "rank_pattern": {},
   "revision": null,
   "target_modules": [
-    "k_proj",
-    "q_proj",
-    "o_proj",
+    "gate_proj",
     "up_proj",
     "v_proj",
     "down_proj",
-    "gate_proj"
+    "o_proj",
+    "q_proj",
+    "k_proj"
   ],
   "target_parameters": null,
   "task_type": "CAUSAL_LM",

adapter_model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4513a109c3529721b87b7cf086ebf3de19e358cb58a9dd0182d150f4695f1ff9
+oid sha256:8fe5e187efa9dc0dc7f4f8df3d1b494124f27e2e2e5c0a77d78b326998805837
 size 59934640

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d3136a8da07614e0a1556a694133ca0ed45b06f6f21c75ee2037ff52965abad0
-size 5713
+oid sha256:236ec4e2a9299c9ce34635495e93ae3121fdd255273453f8782522f42956bc08
+size 5777