| # MuMath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models | |
| ## Introduction | |
| We have amalgamated and further refined these strengths while broadening the scope of augmentation methods to construct a multi-perspective augmentation dataset for mathematics—termed [MuMath (μ-Math) Dataset](https://huggingface.co/datasets/weihao1/MuMath). Subsequently, we finetune LLaMA-2 on the MuMath dataset to derive the MuMath model. | |
| | Model | Size | GSM8k | MATH | | |
| |---|---|---|---| | |
| | WizardMath-7B | 7B | 54.9 | 10.7 | | |
| | MetaMath-7B | 7B | 66.3 | 19.7 | | |
| | MuggleMath-7B | 7B | 68.4 | - | | |
| | [MuMath-7B](https://huggingface.co/weihao1/MuMath-7B) | 7B | **79.1** | **30.0** | | |
| || | |
| | WizardMath-13B | 13B | 63.9 | 14 | | |
| | MetaMath-13B | 13B | 72.3 | 22.4 | | |
| | MuggleMath-13B | 13B | 74 | - |s | |
| | [MuMath-13B](https://huggingface.co/weihao1/MuMath-13B) | 13B | **83.6** | **33.3** | | |
| || | |
| | WizardMath-70B | 70B | 81.6 | 22.7 | | |
| | MetaMath-70B | 70B | 82.3 | 26.6 | | |
| | MuggleMath-70B | 70B | 82.3 | - | | |
| | [MuMath-70B](https://huggingface.co/weihao1/MuMath-70B) | 70B | **88.5** | **41.2** | | |
| > The best results are bolded. | |
| ## Augmentation Methods | |
| <p align="center"> | |
| <img src="./images/MuMath.png" width="800"> | |
| <br> | |
| <em>Overview of the augmentation methods our MuMath employs, which can be divided into four categories: (1) Data Reformulation includes solution reorganization and question rephrasing; (2) Backward Creation includes Backward-Forward Transformation (BF-Trans) and FOBAR; (3) Question Alteration includes expression replacement and difficulty enhancement; (4) Nested Multi-task construction includes data of the auxiliary tasks, i.e., Problem Outline and Solution Plan.</em> | |
| </p> | |
| ### Setup | |
| We recommend using [Conda](https://docs.conda.io/projects/miniconda) to manage your environment. We use [vLLM](https://github.com/vllm-project/vllm) to accelerate inference. Run the following commands to setup your environment: | |
| ```sh | |
| conda create -n mumath python=3.10 | |
| conda activate mumath | |
| cd MuMath-src | |
| pip install -r requirements.txt | |
| ``` | |
| ## Training | |
| We also open [MuMath Dataset](https://huggingface.co/datasets/weihao1/MuMath) for the training stage. | |
| To train a model, after specifying `MODEL_PATH`, `SAVE_PATH`, `DATA_PATH`, the conda environment and so on, run the following command: | |
| ```sh | |
| # 7B or 13B | |
| bash train_7b.sh | |
| # 34B | |
| bash train_13b_70b.sh | |
| ``` | |
| ### Inference and Evaluation | |
| We provide scripts for inference and evaluation, which are called in `train_7b.sh` and `train_13b_70b.sh` as mentioned above. | |
| ```sh | |
| python eval_gsm8k.py --model $SAVE_PATH --data_file ./data/test/GSM8K_test.jsonl | |
| python eval_math.py --model $SAVE_PATH --data_path ./data/test/MATH_test.jsonl | |
| ``` | |
| ## Citation | |
| Please cite the paper if you refer to our model, code, data or paper from MuMath. | |
| ``` | |
| @inproceedings{you-etal-2024-mumath, | |
| title = "{M}u{M}ath: Multi-perspective Data Augmentation for Mathematical Reasoning in Large Language Models", | |
| author = "You, Weihao and Yin, Shuo and Zhao, Xudong and Ji, Zhilong and Zhong, Guoqiang and Bai, Jinfeng", | |
| booktitle = "Findings of the Association for Computational Linguistics: NAACL 2024", | |
| month = jun, | |
| year = "2024", | |
| pages = "2932--2958", | |
| } | |
| ``` | |
| ## Credits | |
| This project has adopted the [MeteMath](https://github.com/meta-math/MetaMath) and [MuggleMath](https://github.com/OFA-Sys/gsm8k-ScRel). | |