---
library_name: transformers
base_model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
datasets:
- meta-math/MetaMathQA
- openai/gsm8k
tags:
- lora
- math
- fine-tuned
language:
- en
---

# TinyMathLlama-1.1B

TinyLlama-1.1B fine-tuned on MetaMathQA using a from-scratch LoRA implementation.

- **LoRA config:** r=8, $\alpha$=16, target modules: q_proj + v_proj
- **Training data:** 10k samples from MetaMathQA
- **GSM8K accuracy:** 3.0% (base: 1.5%, 2x improvement)
- **Trainable params:** 1,126,400 / 1,101,174,784 (0.1%)

## Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("schwp/schwp/TinyMathLlama-1.1B")

## Training & Evaluation

The training and evaluation scripts are available on this [github repository](https://github.com/schwp/lora-from-scratch).
The whole LoRA implementation used for the fine-tuning is also on the repository.