smollm2-135m-squad-lora
This is a LoRA fine-tuned version of the SmolLM2-135M model trained on 5,000 samples from the SQuAD v1.1 question-answering dataset.
The goal of this fine-tune is to improve extractive question answering on short factual passages.
Model Details
- Base model:
HuggingFaceTB/smollm2-135m - Fine-tuning method: LoRA (efficient parameter-efficient fine-tuning)
- Dataset: 5k examples from SQuAD v1.1 (train/dev split)
- Epochs: 2
- Batch size: 4
- Loss after fine-tuning: ~2.21
- Hardware: Google Colab T4 GPU
- Quantization: 4-bit (bitsandbytes)
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "HuggingFaceTB/smollm2-135m"
LORA = "nadaashraff/smollm2-135m-squad-lora"
tokenizer = AutoTokenizer.from_pretrained(BASE)
base_model = AutoModelForCausalLM.from_pretrained(BASE, device_map="auto")
model = PeftModel.from_pretrained(base_model, LORA)
prompt = """Read the following context and answer the question.
Context: Egypt ranks as the fifth worst country in the world for religious freedom. According to a 2010 Pew survey, 84% of Egyptians polled supported the death penalty for those leaving Islam.
Question: What percentage of Egyptians polled supported the death penalty for those leaving Islam?
"""
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))