metadata
license: apache-2.0
datasets:
- TIGER-Lab/MATH-plus
language:
- en
tags:
- torchtune
- minerva-math
library_name: transformers
pipeline_tag: text-generation
jrc/phi3-mini-math
Phi-3 Mini 4k Instruct model finetuned on math datasets.
Uses
How to Get Started with the Model
Use the code below to get started with the model.
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("jrc/phi3-mini-math", trust_remote_code=True)
Training Details
Phi3 was trained using torchtune and the training script + config file are located in this repository.
CMD:
tune run lora_finetune_distributed.py --config mini_lora.yaml
Training Data
[More Information Needed]
Training Procedure
Evaluation
The finetuned model is evaluated on minerva-math using EleutherAI Eval Harness through torchtune.
CMD:
tune run eleuther_eval --config eleuther_evaluation \
checkpoint.checkpoint_dir=./lora-phi3-math \
tasks=["minerva_math"] \
batch_size=32
RESULTS:
| Tasks |Version|Filter|n-shot| Metric |Value | |Stderr|
|------------------------------------|-------|------|-----:|-----------|-----:|---|-----:|
|minerva_math |N/A |none | 4|exact_match|0.1670|± |0.0051|
| - minerva_math_algebra | 1|none | 4|exact_match|0.2502|± |0.0126|
| - minerva_math_counting_and_prob | 1|none | 4|exact_match|0.1329|± |0.0156|
| - minerva_math_geometry | 1|none | 4|exact_match|0.1232|± |0.0150|
| - minerva_math_intermediate_algebra| 1|none | 4|exact_match|0.0576|± |0.0078|
| - minerva_math_num_theory | 1|none | 4|exact_match|0.1148|± |0.0137|
| - minerva_math_prealgebra | 1|none | 4|exact_match|0.3077|± |0.0156|
| - minerva_math_precalc | 1|none | 4|exact_match|0.0623|± |0.0104|
Technical Specifications [optional]
Hardware
4 x NVIDIA A100 GPUs
Max VRAM used per GPU: 29 GB
Citation [optional]
BibTeX:
[More Information Needed]
APA:
[More Information Needed]
Glossary [optional]
[More Information Needed]
More Information [optional]
[More Information Needed]
Model Card Contact
[More Information Needed]