README.md

---
license: llama3.3
base_model: meta-llama/Llama-3.3-70B-Instruct
datasets:
- flamiinngo/math-code-qa-v2
tags:
- math
- code
- lora
- peft
- adapter
- autoscientist
- adaption
language:
- en
pipeline_tag: text-generation
library_name: peft
---

# Math & Code — Llama-3.3-70B LoRA

A LoRA adapter for **Llama-3.3-70B-Instruct**, fine-tuned to solve mathematical
problems — arithmetic word problems through algebra, geometry and combinatorics —
and answer short coding questions.

Trained with **Adaption Labs' AutoScientist** for the AutoScientist Challenge
(Math & Code category).

## Result

| Evaluation | Base | Adapted |
|---|---|---|
| Math category | 28 | **72** |
| In-distribution test set | 52 | 48 |

Wins in a paired comparison, not accuracy percentages.

The rows disagree, which is worth explaining. On the narrow in-distribution set a
judge slightly prefers the base model's phrasing. Across the wider category —
including problems well outside the training distribution — the adapted model wins
decisively. The mathematical substance generalised further than the answer style.

## What produced the 12-point gain

An earlier version of this model scored **60–28** on the same category evaluation.
The difference was a single filter in the training data.

v1 capped every solution at 18–75 words. In the upstream corpus, MATH-level
solutions have a **median length of 121–156 words**, while grade-school word
problems sit at 89–101. The cap therefore kept only the shortest, easiest examples
from the hard sources — the model trained almost entirely on arithmetic and was
then evaluated across the full difficulty range.

Setting the word budget per source (30–150 for algebra and geometry, 18–80 for
word problems) raised solution p90 from 77 words to 133, and the category win rate
from 60 to 72.

## Usage

The adapter is stored unpacked and loads directly.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "meta-llama/Llama-3.3-70B-Instruct"
ADAPTER = "flamiinngo/adaption_math_word_problems_solutions"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [{"role": "user", "content":
    "Mrs Thompson has 7 Harry Potter books, 6 Twilight books and 5 Hunger Games "
    "books. Each series must stay together on the shelf. How many orderings are "
    "there?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

out = model.generate(inputs, max_new_tokens=400, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```

**Hardware:** the 70B base needs roughly 140 GB in bf16, or about 40 GB with 4-bit
quantisation. The adapter is 3.3 GB.

**Output style:** brief worked steps, then the result stated explicitly as
"The answer is X."

**Note on the base model name.** `adapter_config.json` records
`togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference`, the base as served during
training. Same architecture — load against `meta-llama/Llama-3.3-70B-Instruct`.

## Training

| Parameter | Value |
|---|---|
| Base | `meta-llama/Llama-3.3-70B-Instruct` |
| Rank (`r`) | 64 |
| `lora_alpha` | 128 |
| Target modules | all-linear |
| Epochs | 3 |
| Peak learning rate | 1e-4, cosine |

## Dataset

[**flamiinngo/math-code-qa-v2**](https://huggingface.co/datasets/flamiinngo/math-code-qa-v2)
— 5,297 rows (4,197 math, 1,100 code), every math answer ending in a result
verified against the upstream `expected_answer` column. Derived from
[nvidia/OpenMathInstruct-2](https://huggingface.co/datasets/nvidia/OpenMathInstruct-2)
and [sahil2801/CodeAlpaca-20k](https://huggingface.co/datasets/sahil2801/CodeAlpaca-20k),
both CC-BY-4.0.

Also on Kaggle:
[model](https://www.kaggle.com/models/flamiinngo/adaption_math-41946c32-256d-4d24-a1ee-6effb91b690c) ·
[dataset](https://www.kaggle.com/datasets/flamiinngo/math-code-qa-v2)

## Limitations

- **It can produce confident wrong reasoning.** The training solutions are
  model-generated upstream; only their final answers were verified. Errors in
  algebraic reasoning exist in the data and this model reproduces that style of
  mistake. Check any result that matters.
- **Not a calculator.** Fine-tuning improved the working, not arithmetic guarantees.
- **Scope is school through early-undergraduate.** Not olympiad or research
  mathematics.
- **Code output is untested.** The code training data was filtered for length,
  not executed. Treat generated code as a draft.
- **Win rate is not accuracy.** It measures preference against one base model on
  one evaluation.
- **English only.**

## License

The adapter is a derivative of Llama-3.3-70B-Instruct and is subject to the
**Llama 3.3 Community License**. The training data is CC-BY-4.0.

## Acknowledgements

- **Adaption Labs** — AutoScientist platform and the challenge
- **NVIDIA** and **sahil2801** — upstream open datasets
- **Meta** — Llama 3.3 base model