ayushjaswal/scienceqa-llama32_it1

LLaMA 3.2 3B — ScienceQA Fine-tune

A QLoRA fine-tuned version of unsloth/Llama-3.2-3B-Instruct-bnb-4bit on the ScienceQA dataset, trained for multi-subject science reasoning in a multiple-choice format.

---

Performance

Model	Accuracy
Base LLaMA 3.2 3B Instruct	76.1%
This model (fine-tuned)	93.5%
Improvement	+17.4 percentage points

Evaluated on the ScienceQA test split (4,241 examples) using exact-match accuracy on single-letter predictions (A/B/C/D).

---

Model Details

• Developed by: ayushjaswal
• Base model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit
• Fine-tuning method: QLoRA (Quantized Low-Rank Adaptation)
• Dataset: ayushjaswal/scienceQAcleaned original from derek-thomas/ScienceQA
• Task: Multiple-choice science question answering
• Trainable parameters: ~0.3% of total (LoRA adapters only)

LoRA Configuration

r             = 16
lora_alpha    = 16
lora_dropout  = 0.05
target_modules = q_proj, k_proj, v_proj, o_proj,
                 gate_proj, up_proj, down_proj

---

Dataset & Curation

Trained on a curated subset of derek-thomas/ScienceQA:

• Filtered image-dependent rows (questions requiring visual context)
• Filtered questions referencing visual elements ("diagram", "figure", "image", etc.)
• Resolved integer answer indices to actual choice text
• Formatted as single-letter output (A/B/C/D) for clean, unambiguous supervision

Final splits used:

Train       → 6.51K examples (filtered)
Validation  → 2.14K examples
Test        → 2.22K examples

---

Prompt Format

This model expects inputs in LLaMA 3 chat template format:

<|begin_of_text|><|start_header_id|>user<|end_header_id|>
Subject: {subject} | Category: {category} | Topic: {topic}
Lecture: {lecture}
Hint: {hint}

Question: {question}

Choices:
  A) {choice_0}
  B) {choice_1}
  C) {choice_2}

Answer with only the choice letter (A, B, C, ...).
<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Expected output: A single letter — A, B, or C

---

Note: This model was trained with Unsloth. Load the base model with FastLanguageModel.from_pretrained before attaching the adapter with PeftModel. Loading with vanilla AutoModelForCausalLM will raise an AttributeError due to Unsloth's custom attention patches.

---

Limitations

• Trained on K-12 to early undergraduate science questions. May underperform on graduate-level or highly specialized content
• Image-dependent questions are out of scope; the model has no vision capability
• Outputs a single letter only, no explanation is generated by default
• Subjects covered are limited

---

Training Infrastructure

Trained on Google Colab using Unsloth's optimized QLoRA implementation, which provides 2x faster training and 60% less memory usage compared to standard HuggingFace + Flash Attention 2.

---

Fine-tuned with Unsloth