soupstick/smollm3-qlora-ft

🧠 SmolLM3 QLoRA - OpenOrca Fine-Tuned

SmolLM3 QLoRA is a lightweight, 3B parameter open-source language model based on SmolLM3-3B, fine-tuned using QLoRA on the OpenOrca Slim dataset (500K examples). It is optimized for retrieval-augmented generation (RAG) use cases and delivers competitive benchmark scores against much larger models like LLaMA-2 7B.

---

✨ Model Highlights

• 🔍 Trained for real-world queries using OpenOrca-style assistant data.
• ⚡ Efficient: 3B parameter model that runs on a single A100 or consumer GPU.
• 🧠 Competent generalist: Performs well on reasoning and knowledge tasks.
• 🔗 RAG-friendly: Ideal for hybrid search setups using BM25 + FAISS.
• 🧪 Evaluated on benchmarks: Outperforms similar-sized models.

---

🧰 Intended Use

SmolLM3 QLoRA is intended to serve as a fast and compact assistant model for:

• 💬 Lightweight RAG pipelines
• 📚 Document and web snippet reasoning
• 🤖 Prototype assistants
• 🧪 AI research in instruction tuning and hybrid retrieval

---

🧪 Evaluation

The model has been evaluated using lm-evaluation-harness on a 500-sample subset of key academic benchmarks.

Task	Accuracy	Normalized Accuracy	LLaMA-2 7B
HellaSwag	51.2%	66.4%	56.7% / 73.2%
ARC-Challenge	49.4%	52.2%	53.7% / 56.9%
BoolQ	81.0%	—	83.1%

👉 Model achieves ~90–95% of LLaMA-2 7B performance at less than half the size.

---

🏗️ Training Configuration

• Base Model: SmolLM3-3B
• Finetuning Method: QLoRA (LoRA rank=8)
• Dataset: Open-Orca/SlimOrca (500K samples)
• Precision: bfloat16
• Epochs: 3
• Max Length: 1024 tokens
• Hardware: 2x A100 80GB
• Framework: 🤗 Transformers + TRL

---

🧠 How to Use

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "soupstick/smollm3-qlora-ft"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto"
)

inputs = tokenizer("Explain retrieval-augmented generation.", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=300)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))