| | --- |
| | license: apache-2.0 |
| | datasets: |
| | - EleutherAI/muInstruct |
| | - camel-ai/math |
| | language: |
| | - en |
| | tags: |
| | - math |
| | --- |
| | |
| | `llemma_7b_muinstruct_camelmath` is an instruction-following finetune of [Llemma 7B](https://huggingface.co/EleutherAI/llemma_7b), trained on the [μInstruct](https://huggingface.co/datasets/EleutherAI/muInstruct) and [camel-ai/math](https://huggingface.co/datasets/camel-ai/math) datasets. |
| |
|
| | ## Input Formatting |
| | Format input queries as follows: |
| | ``` |
| | input_text = f"Input:{input}\n\nResponse:" |
| | ``` |
| |
|
| | Note that due to an error during training, this model's end-of-sequence token ID is `0` instead of the `2` which is standard for Llama-2 based models. Inference APIs should handle this automatically by reading this repo's `config.json`, but be aware of this difference if you are doing token surgery. |
| |
|
| | ## Evals |
| | ` |
| | llemma_7b_muinstruct_camelmath` compares favorably to other 7B parameter models on the [Hungarian Math Exam](https://huggingface.co/datasets/keirp/hungarian_national_hs_finals_exam/blob/main/README.md). It surpasses the few-shot performance of Llemma 7B whilst being the strongest Llama-2 7B based model. |
| | |
| | | Model | Exam Score | |
| | | ------------------------------------------------------------------------------ | ---------- | |
| | | [Code Llama 7B](https://huggingface.co/codellama/CodeLlama-7b-hf) (few-shot) | 8\% | |
| | | [MetaMath 7B](https://huggingface.co/meta-math/MetaMath-7B-V1.0) | 20\% | |
| | | [MAmmoTH 7B](https://huggingface.co/TIGER-Lab/MAmmoTH-7B) | 17\% | |
| | | [MAmmoTH Coder 7B](https://huggingface.co/TIGER-Lab/MAmmoTH-Coder-7B) | 11\% | |
| | | [Llemma 7B](https://huggingface.co/EleutherAI/llemma_7b) (few-shot) | 23\% | |
| | | Llemma_7B_muinstruct_camelmath | 25\% | |
| | | - | - | |
| | | [Mistral 7B](https://huggingface.co/mistralai/Mistral-7B-v0.1) (few-shot) | 22\% | |
| | | [MetaMath Mistral 7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B) | 29\% | |
| | | [OpenChat 3.5](https://huggingface.co/openchat/openchat_3.5) | 37\% | |
| |
|
| |
|
| |
|
