| --- |
| license: apache-2.0 |
| base_model: mistralai/Mixtral-8x7B-Instruct-v0.1 |
| datasets: |
| - flamiinngo/math-code-qa |
| tags: |
| - math |
| - code |
| - lora |
| - autoscientist |
| - adaption |
| language: |
| - en |
| pipeline_tag: text-generation |
| library_name: peft |
| --- |
| |
| # Math & Code Mixtral-8x7B |
|
|
| A LoRA fine-tune of **Mixtral-8x7B-Instruct-v0.1** for **mathematical word problems |
| and short code generation**, built with **Adaption Labs' AutoScientist** for the |
| AutoScientist Challenge (Math & Code category). |
|
|
| ## Results |
|
|
| Head-to-head win rate against the base model: |
|
|
| | Evaluation | Base | Adapted | |
| |---|---|---| |
| | Training distribution | 33 | **67** | |
| | Math category (all tasks) | 40 | **60** | |
|
|
| These are wins in a paired comparison, not accuracy percentages. |
|
|
| An earlier checkpoint trained on 2,050 rows without augmentation scored **56–44** |
| on the category evaluation. Expanding the training set with domain and |
| general-purpose augmentation, and raising LoRA capacity to rank 64, gained 4 |
| points. The platform's guidance suggested a larger gain from crossing 20,000 |
| datapoints; the observed improvement was smaller than that. |
|
|
| ## Usage |
|
|
| The archive unpacks **flat**, so give it its own directory: |
|
|
| ```bash |
| mkdir -p math-adapter |
| tar --zstd -xf math-code-mixtral-8x7b-weights.tgz -C math-adapter |
| ``` |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| from peft import PeftModel |
| |
| BASE = "mistralai/Mixtral-8x7B-Instruct-v0.1" |
| ADAPTER = "./math-adapter" |
| |
| tokenizer = AutoTokenizer.from_pretrained(ADAPTER) |
| model = AutoModelForCausalLM.from_pretrained( |
| BASE, torch_dtype=torch.bfloat16, device_map="auto" |
| ) |
| model = PeftModel.from_pretrained(model, ADAPTER) |
| model.eval() |
| |
| messages = [{"role": "user", "content": |
| "Jean eats one donut per 2 pages she writes. If she writes 12 pages and each " |
| "donut has 150 calories, how many calories does she eat?"}] |
| inputs = tokenizer.apply_chat_template( |
| messages, add_generation_prompt=True, return_tensors="pt").to(model.device) |
| out = model.generate(inputs, max_new_tokens=256, do_sample=False) |
| print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| **Hardware:** Mixtral-8x7B is a 46.7B-parameter mixture-of-experts model — roughly |
| 94 GB in bf16, or about 25 GB in 4-bit. The adapter is 208 MB. |
|
|
| **Output style:** trained to show brief working then state the answer explicitly, |
| ending with "The answer is X." |
|
|
| ## Training |
|
|
| | Parameter | Value | |
| |---|---| |
| | Base | `mistralai/Mixtral-8x7B-Instruct-v0.1` | |
| | Rank (`r`) | 64 | |
| | `lora_alpha` | 128 | |
| | `lora_dropout` | 0.0 | |
| | Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` | |
| | Peak learning rate | 1e-4 | |
| | Schedule | cosine, warmup 0.1 | |
| | Epochs | 5 (145 steps) | |
|
|
| ### Training curve |
|
|
| | Epoch | Step | Eval loss | |
| |---|---|---| |
| | 1.00 | 29 | 0.7350 | |
| | 2.00 | 58 | 0.7026 | |
| | 3.00 | 87 | 0.6889 | |
| | 4.00 | 116 | 0.6830 | |
| | 5.00 | 145 | **0.6805** | |
|
|
| Training loss fell from 1.248 to 0.764. Evaluation loss was still declining at |
| epoch 5, so this run did not overfit — the last epoch still gained, if marginally. |
|
|
| ## Dataset |
|
|
| [**flamiinngo/math-code-qa**](https://huggingface.co/datasets/flamiinngo/math-code-qa) |
| — 5,200 rows (3,600 math, 1,600 code), every math answer ending in a result |
| verified against the upstream `expected_answer` column. Derived from |
| [nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2) |
| and [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k), |
| both CC-BY-4.0. The training run additionally used AutoScientist domain and |
| general-purpose augmentation on top of this base. |
|
|
| ## Limitations |
|
|
| - **It can produce confident wrong arithmetic.** Fine-tuning improved the base |
| model's working, but this is not a calculator. Check any result that matters. |
| - **Scope is school and early-undergraduate level** — word problems, algebra, |
| combinatorics, geometry. Not competition mathematics. |
| - **Code output is untested.** The code training data was filtered for length, |
| not executed. Treat generated code as a draft. |
| - **Win rate is not accuracy.** It measures preference against one base model on |
| one evaluation, not correctness in absolute terms. |
| - **English only.** |
|
|
| ## License |
|
|
| The adapter is Apache 2.0, matching the base model. The training data is |
| CC-BY-4.0 and requires attribution — see the dataset card. |
|
|
| ## Acknowledgements |
|
|
| - **Adaption Labs** — AutoScientist platform and challenge |
| - **NVIDIA** and **sahil2801** — upstream open datasets |
| - **Mistral AI** — Mixtral-8x7B base model |
|
|
