Qwen2.5-Math-1.5B-ES-MATH

Qwen/Qwen2.5-Math-1.5B fine-tuned with Evolution Strategies on the MATH level 3–5 training set, following the recipe of ES-at-Scale (Qiu et al., Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning, arXiv:2509.24372).

The whole run happened on one Apple Silicon laptop (Apple M5 Max, PyTorch MPS backend, bf16) with mps_es, a single-process port of the ES loop that replaces vLLM and Ray with plain transformers generation. No gradients, no optimizer state: every one of the model's 1.5 B parameters is perturbed with seeded Gaussian noise, the perturbed model is scored on a batch of problems, and the weights move along the reward-weighted noise.

Results

Held-out evaluation on the first 100 problems of MATH-500 (the upstream math500 split), greedy decoding, up to 3000 new tokens, graded with the upstream boxed-answer grader built on math-verify. The same 100 problems are used at every checkpoint.

checkpoint	accuracy	response ends with \boxed{}	median response (tokens)	ran to the 3000-token cap	echoed the question
Qwen2.5-Math-1.5B (base)	33 %	55 %	970	40 / 100	25 / 100
ES iteration 10	51 %	78 %	621	19 / 100	11 / 100
ES iteration 20	59 %	88 %	528	10 / 100	5 / 100
ES iteration 30 (this model)	64 %	92 %	507	5 / 100	1 / 100

With n = 100 the standard error is about ±5 points, so read the trend, not the last digit. Per problem, 36 were gained, 28 kept, 5 lost and 31 are still wrong. Most of the gain is the base model's degenerate behaviour disappearing: with the Qwen math prompt it often repeats the question or never stops; after 30 iterations it almost always writes a solution and boxes an answer.

Every evaluated response, with its grade, is in training/eval-output/, and the per-iteration training metrics are in training/metrics.jsonl. The other splits of the upstream math suite (AIME, AMC, Minerva, OlympiadBench) were not evaluated.

Training setting

RL-zero style: the base model, not the instruct model, trained directly against a binary outcome reward with the Qwen2.5-Math prompt format.

• Data. The upstream math_lvl3to5_8k set, 8,523 MATH problems of levels 3–5, walked in order 32 problems per iteration, so the 30 iterations saw 960 distinct problems and no repeats.
• Prompt. Qwen2.5-Math chat format with the system message Please reason step by step, and put your final answer within \boxed{}.
• Reward. 1 if the last \boxed{} in the response matches the reference answer under math-verify, else 0. No format or length shaping.
• Decoding. Greedy, at most 3000 new tokens, both for training rollouts and for evaluation.
• ES update. Population of 30 perturbations per iteration, each a full-model Gaussian perturbation θ + σ·ε(seed) applied in place in bf16 and restored from the seed; rewards z-scored across the population; θ ← θ + α/N · Σ zᵢ εᵢ. Same σ, α, population size and update rule as upstream.

Training parameters

parameter	value
perturbation scale σ	0.001
learning rate α	0.0005 (σ / 2)
population size	30
problems per iteration	32
generation mini-batch	128 prompts per forward pass
max response length	3000 tokens
rollout temperature	0 (greedy)
iterations	30
precision / device	bf16 on Apple M5 Max via PyTorch MPS, single process
seed	42
wall time	about 197 h in total, 5–7.5 h per iteration (30 × 32 greedy rollouts of up to 3000 tokens, no batching across population members)

The upstream 14 B run used a population of 32, 1024 problems per iteration and 500 iterations on 8 GPUs; this run keeps the algorithm and shrinks only the budget.

Usage

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "deepsai8/Qwen2.5-Math-1.5B-ES-MATH"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")  # transformers < 5: torch_dtype=

problem = r"Convert the point $(0,3)$ in rectangular coordinates to polar coordinates. Enter your answer in the form $(r,\theta),$ where $r > 0$ and $0 \le \theta < 2 \pi.$"
messages = [
    {"role": "system", "content": "Please reason step by step, and put your final answer within \\boxed{}."},
    {"role": "user", "content": problem},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=3000, do_sample=False)
print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

The bundled chat template reproduces the training prompt exactly. generation_config.json lists both <|endoftext|> and <|im_end|> as end-of-sequence tokens so chat-style generation stops cleanly; the base model's config only had the former.

llama.cpp

Ready-made GGUF files are in gguf/: Q8_0 (near-lossless) and Q4_K_M (small). The chat template is embedded, so llama-server applies it to /v1/chat/completions requests.

# OpenAI-compatible server on http://localhost:8080
llama-server -hf deepsai8/Qwen2.5-Math-1.5B-ES-MATH -hff gguf/Qwen2.5-Math-1.5B-ES-MATH-Q8_0.gguf --temp 0

# one-shot completion with the raw prompt format
llama-completion -hf deepsai8/Qwen2.5-Math-1.5B-ES-MATH -hff gguf/Qwen2.5-Math-1.5B-ES-MATH-Q8_0.gguf --temp 0 -n 3000 -no-cnv \
  -p $'<|im_start|>system\nPlease reason step by step, and put your final answer within \\boxed{}.<|im_end|>\n<|im_start|>user\nWhat is the sum of the first 20 positive odd integers?<|im_end|>\n<|im_start|>assistant\n'

Quantized greedy outputs can differ slightly from the bf16 transformers model; the Q8_0 file solves the example problem above with the same derivation.

-hf downloads the file into llama.cpp's cache on first use; while the repo is private, export HF_TOKEN first.

To build your own GGUF from the safetensors (for example a different quantization):

git clone https://github.com/ggml-org/llama.cpp && pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt
hf download deepsai8/Qwen2.5-Math-1.5B-ES-MATH --local-dir Qwen2.5-Math-1.5B-ES-MATH
python llama.cpp/convert_hf_to_gguf.py Qwen2.5-Math-1.5B-ES-MATH --outfile es-math-bf16.gguf --outtype bf16
llama-quantize es-math-bf16.gguf es-math-q5_k_m.gguf Q5_K_M

Ollama and LM Studio load the same GGUF files. A minimal Ollama Modelfile:

FROM ./Qwen2.5-Math-1.5B-ES-MATH-Q8_0.gguf
PARAMETER temperature 0
SYSTEM "Please reason step by step, and put your final answer within \\boxed{}."
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

Further training

The model is a standard Qwen2 checkpoint, so either trainer can pick it up by name.

On Apple Silicon (or CPU, or one CUDA GPU) with mps_es

git clone -b mps-es-apple-silicon https://github.com/deepsaia/es-at-scale.git
cd es-at-scale
python3 -m venv .venv && source .venv/bin/activate
pip install -r mps_es/requirements.txt          # datasets/ ships with the repository

python -m mps_es.train_math \
  --model-name deepsai8/Qwen2.5-Math-1.5B-ES-MATH \
  --n-iterations 30 --batch-size 32 --mini-batch-size 128 --max-tokens 3000 \
  --eval-splits math500 --eval-samples 100 --eval-freq 10 \
  --experiment-name math-1.5b-continued

python -m mps_es.watch experiments/math-1.5b-continued -f    # live progress; -v for detail

--n-iterations 0 evaluates without training. --eval-splits math500,amc,aime,minerva,olympiad_bench --eval-samples -1 runs the full suite. Starting from the Hub weights begins a fresh seed stream; to continue this run's exact perturbation sequence you would need its checkpoint.pt, which is not published (3.3 GB, redundant with the weights).

On GPUs with the upstream trainer (vLLM + Ray)

Install ES-at-Scale as its README describes, then:

python es_at_scale/train.py --task math \
  --model-name deepsai8/Qwen2.5-Math-1.5B-ES-MATH \
  --sigma 0.001 --population-size 30 --n-iterations 500 --eval-freq 5 \
  --train-dataset datasets/train/math_lvl3to5_8k --eval-dataset datasets/evaluation_suite/math/ \
  --batch-size 1024 --mini-batch-size 1024 --max-tokens 3000 \
  --n-vllm-engines 8 --use-gpus "0,1,2,3,4,5,6,7" \
  --experiment-name es-math-1.5b-gpu

Limitations

• Evaluated on a 100-problem slice of MATH-500 only; the ±5-point error bar is wide, and AIME, AMC, Minerva and OlympiadBench were not run.
• Trained and evaluated with greedy decoding. Sampling behaviour was not tuned.
• 5 of 100 held-out responses still run to the 3000-token cap.
• English competition mathematics only. The base model has no safety alignment and neither does this one.
• The run saw 960 of the 8,523 training problems; this is a small-budget demonstration that the ES recipe works off-cluster, not a state-of-the-art math model.

Files

path	contents
model-*.safetensors, config.json, tokenizer*, chat_template.jinja, generation_config.json	the model, bf16
training_config.json	ES hyperparameters of the run
training/metrics.jsonl, training/run.json	per-iteration training metrics and run settings
training/eval-output/*.json	every held-out response and its grade at baseline and iterations 10, 20, 30
gguf/*.gguf	Q8_0 and Q4_K_M quantizations for llama.cpp
assets/*.png	the figures above, light and dark

Citation

@article{Qiu2026EvolutionStrategies,
  title={Evolution Strategies at Scale: {LLM} Fine-Tuning Beyond Reinforcement Learning},
  author={Qiu, Xin and Gan, Yulu and Hayes, Conor F. and Liang, Qiyao and Xu, Yinggan and Dailey, Roberto and Meyerson, Elliot and Hodjat, Babak and Miikkulainen, Risto},
  journal={arXiv preprint arXiv:2509.24372},
  year={2026},
  eprint={2509.24372},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  doi={10.48550/arXiv.2509.24372}
}