File size: 4,536 Bytes
9bc4ff5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 | ---
license: apache-2.0
base_model: mistralai/Mixtral-8x7B-Instruct-v0.1
datasets:
- flamiinngo/math-code-qa
tags:
- math
- code
- lora
- autoscientist
- adaption
language:
- en
pipeline_tag: text-generation
library_name: peft
---
# Math & Code Mixtral-8x7B
A LoRA fine-tune of **Mixtral-8x7B-Instruct-v0.1** for **mathematical word problems
and short code generation**, built with **Adaption Labs' AutoScientist** for the
AutoScientist Challenge (Math & Code category).
## Results
Head-to-head win rate against the base model:
| Evaluation | Base | Adapted |
|---|---|---|
| Training distribution | 33 | **67** |
| Math category (all tasks) | 40 | **60** |
These are wins in a paired comparison, not accuracy percentages.
An earlier checkpoint trained on 2,050 rows without augmentation scored **56–44**
on the category evaluation. Expanding the training set with domain and
general-purpose augmentation, and raising LoRA capacity to rank 64, gained 4
points. The platform's guidance suggested a larger gain from crossing 20,000
datapoints; the observed improvement was smaller than that.
## Usage
The archive unpacks **flat**, so give it its own directory:
```bash
mkdir -p math-adapter
tar --zstd -xf math-code-mixtral-8x7b-weights.tgz -C math-adapter
```
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
BASE = "mistralai/Mixtral-8x7B-Instruct-v0.1"
ADAPTER = "./math-adapter"
tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(
BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
messages = [{"role": "user", "content":
"Jean eats one donut per 2 pages she writes. If she writes 12 pages and each "
"donut has 150 calories, how many calories does she eat?"}]
inputs = tokenizer.apply_chat_template(
messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```
**Hardware:** Mixtral-8x7B is a 46.7B-parameter mixture-of-experts model — roughly
94 GB in bf16, or about 25 GB in 4-bit. The adapter is 208 MB.
**Output style:** trained to show brief working then state the answer explicitly,
ending with "The answer is X."
## Training
| Parameter | Value |
|---|---|
| Base | `mistralai/Mixtral-8x7B-Instruct-v0.1` |
| Rank (`r`) | 64 |
| `lora_alpha` | 128 |
| `lora_dropout` | 0.0 |
| Target modules | `q_proj`, `k_proj`, `v_proj`, `o_proj` |
| Peak learning rate | 1e-4 |
| Schedule | cosine, warmup 0.1 |
| Epochs | 5 (145 steps) |
### Training curve
| Epoch | Step | Eval loss |
|---|---|---|
| 1.00 | 29 | 0.7350 |
| 2.00 | 58 | 0.7026 |
| 3.00 | 87 | 0.6889 |
| 4.00 | 116 | 0.6830 |
| 5.00 | 145 | **0.6805** |
Training loss fell from 1.248 to 0.764. Evaluation loss was still declining at
epoch 5, so this run did not overfit — the last epoch still gained, if marginally.
## Dataset
[**flamiinngo/math-code-qa**](https://huggingface.co/datasets/flamiinngo/math-code-qa)
— 5,200 rows (3,600 math, 1,600 code), every math answer ending in a result
verified against the upstream `expected_answer` column. Derived from
[nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)
and [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k),
both CC-BY-4.0. The training run additionally used AutoScientist domain and
general-purpose augmentation on top of this base.
## Limitations
- **It can produce confident wrong arithmetic.** Fine-tuning improved the base
model's working, but this is not a calculator. Check any result that matters.
- **Scope is school and early-undergraduate level** — word problems, algebra,
combinatorics, geometry. Not competition mathematics.
- **Code output is untested.** The code training data was filtered for length,
not executed. Treat generated code as a draft.
- **Win rate is not accuracy.** It measures preference against one base model on
one evaluation, not correctness in absolute terms.
- **English only.**
## License
The adapter is Apache 2.0, matching the base model. The training data is
CC-BY-4.0 and requires attribution — see the dataset card.
## Acknowledgements
- **Adaption Labs** — AutoScientist platform and challenge
- **NVIDIA** and **sahil2801** — upstream open datasets
- **Mistral AI** — Mixtral-8x7B base model
|