Qwen2.5-0.5B-Instruct-ES-Countdown

Qwen/Qwen2.5-0.5B-Instruct fine-tuned with Evolution Strategies on the Countdown task from ES-at-Scale (Qiu et al., Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning, arXiv:2509.24372): given three or four numbers and a target, write an arithmetic expression that uses each number once and equals the target.

This is the small, fast companion to Qwen2.5-Math-1.5B-ES-MATH. It was trained for 200 iterations on one Apple Silicon laptop (Apple M5 Max, PyTorch MPS, bf16) with mps_es, a single-process port of the ES loop, in a little under eight hours of generation time. It is published as a worked example of the method on a laptop, with every evaluation response included, rather than as a useful assistant: read the limitations before relying on it.

Results

Held-out evaluation on the first 500 puzzles of the upstream 2,000-puzzle Countdown eval split, greedy decoding, up to 256 new tokens. A puzzle counts as solved when the expression in <answer> uses each given number exactly once and evaluates to the target.

checkpoint	puzzles solved	well-formed <think>/<answer>	median response (tokens)
Qwen2.5-0.5B-Instruct (base)	0 %	30 % (training batch)	–
ES iteration 50	3 %	–	–
ES iteration 60	6 %	100 %	45
ES iteration 100	10 %	100 %	41
ES iteration 150	11 %	100 %	41
ES iteration 200 (this model)	12 %	100 %	41

Standard error at n = 500 is about ±1.4 points. ES learns the output format within the first 50 iterations (the format reward is 0.1, the answer reward 1.0, so once format is saturated only solving puzzles moves the score) and then slowly raises the answer rate.

What the model actually learned. The bottom-right panel is the honest part. By iteration 200 half of all held-out answers are literally (a − b + c) with the given numbers in order, and 52 of the 60 correct answers come from that one shape. The model found that a fixed template solves a stable fraction of the three-number puzzles and converged on it: at iteration 200 it solves 52 of the 251 three-number puzzles and 8 of the 249 four-number ones, and the <think> section is one boilerplate sentence. It is a clean illustration of ES climbing an outcome reward with a 0.5 B model and 200 training puzzles, and also of what a reward without diversity pressure selects for.

Every evaluated response is in training/eval-output/ (iterations 60 to 200, every 10), the per-iteration training metrics for iterations 50–200 are in training/metrics.jsonl, and the console log of the second run is training/train.log. The first run (iterations 1–49) kept only its final checkpoint, which the second run resumed from, so its per-iteration file is not included.

Training setting

• Data. The upstream datasets/train/countdown set of 200 puzzles (96 with three numbers, 104 with four), walked in order 32 per iteration, so every puzzle was seen about 32 times over the 200 iterations. Evaluation uses the first 500 of the 2,000 puzzles in datasets/evaluation_suite/countdown (251 three-number, 249 four-number).

• Prompt. The dataset's context field is the complete prompt, passed through as plain text with no chat template. It ends with Let me solve this step by step.\n<think> so the model continues the reasoning block:

  You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [44 19 35], create an equation that equals 98. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.
  <think>
  

• Reward. 0.1 for a well-formed </think> + <answer>…</answer> response, plus 1.0 when the answer expression uses each number once and evaluates to the target (the upstream countdown_reward_fn).

• Decoding. Greedy, at most 256 new tokens, for rollouts and evaluation.

• ES update. Population of 30 full-model Gaussian perturbations θ + σ·ε(seed) per iteration, applied in place in bf16 and restored from the seed; rewards z-scored across the population; θ ← θ + α/N · Σ zᵢ εᵢ. Same σ, α, population size and update rule as upstream.

Training parameters

parameter	value
perturbation scale σ	0.001
learning rate α	0.0005 (σ / 2)
population size	30
puzzles per iteration	32
max response length	256 tokens
rollout temperature	0 (greedy)
iterations	200, as two runs: 50, then resumed from the iteration-50 checkpoint to 200
held-out eval	500 puzzles every 10 iterations
precision / device	bf16 on Apple M5 Max via PyTorch MPS, single process
seed	42
wall time	about 3.9 h for iterations 50–200; 190 s per iteration at the start, 45–85 s at the end as responses got shorter

Usage

Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto")  # transformers < 5: torch_dtype=

numbers, target = [81, 69, 76], 88
prompt = (
    "You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer."
    f"Using the numbers [{' '.join(map(str, numbers))}], create an equation that equals {target}. "
    "You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. "
    "Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>."
    "Let me solve this step by step.\n<think>"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
# -> First, we need to find a way to combine these three numbers into a single equation that equals 88.</think>
#    <answer>(81 - 69 + 76)</answer>

Use the raw prompt above, not apply_chat_template: the model was trained on this exact text.

llama.cpp

GGUF files are in gguf/ (Q8_0 and Q4_K_M).

llama-completion -hf deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown -hff gguf/Qwen2.5-0.5B-Instruct-ES-Countdown-Q8_0.gguf --temp 0 -n 256 -no-cnv \
  -p $'You are a helpful assistant. You first think about the reasoning process in your mind and then provide the user with the answer.Using the numbers [81 69 76], create an equation that equals 88. You can use basic arithmetic operations (+, -, *, /) and each number can only be used once. Show your work in <think> </think> tags. And return the final answer in <answer> </answer> tags, for example <answer> (1 + 2) / 3 </answer>.Let me solve this step by step.\n<think>'

Quantized greedy outputs differ from the bf16 transformers model on individual puzzles: for the puzzle above, Q8_0 under llama.cpp answers (81 + 69 + 76) where bf16 on MPS answers (81 - 69 + 76). On the first 100 held-out puzzles Q8_0 solves 6 and bf16 solves 7, with only 51 of the 100 answers identical. The model sits close to its decision boundaries, so small numeric differences flip many greedy picks without changing the overall rate much.

-hf downloads the file into llama.cpp's cache on first use; while the repo is private, export HF_TOKEN first.

To rebuild or requantize: python llama.cpp/convert_hf_to_gguf.py <local dir> --outfile model-bf16.gguf --outtype bf16, then llama-quantize model-bf16.gguf model-q5_k_m.gguf Q5_K_M.

Further training

On Apple Silicon (or CPU, or one CUDA GPU) with mps_es

git clone -b mps-es-apple-silicon https://github.com/deepsaia/es-at-scale.git
cd es-at-scale
python3 -m venv .venv && source .venv/bin/activate
pip install -r mps_es/requirements.txt          # datasets/ ships with the repository

python -m mps_es.train_countdown \
  --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \
  --n-iterations 100 --experiment-name countdown-continued

python -m mps_es.watch experiments/countdown-continued -f

Defaults match this run (population 30, σ 0.001, 32 puzzles per iteration, 256 tokens, 500 eval puzzles every 10 iterations). Starting from the Hub weights begins a fresh seed stream rather than continuing this run's.

On GPUs with the upstream trainer (vLLM + Ray)

python es_at_scale/train.py --task countdown \
  --model-name deepsai8/Qwen2.5-0.5B-Instruct-ES-Countdown \
  --sigma 0.001 --population-size 30 --n-iterations 1000 --eval-freq 10 \
  --train-dataset datasets/train/countdown --eval-dataset datasets/evaluation_suite/countdown \
  --batch-size 256 --mini-batch-size 256 --max-tokens 1024 \
  --n-vllm-engines 8 --use-gpus "0,1,2,3,4,5,6,7" \
  --experiment-name es-countdown-gpu

Limitations

• Solves 12 % of held-out puzzles (21 % of the three-number ones, 3 % of the four-number ones), mostly with one memorised expression shape; it does not search over the numbers.
• Trained on 200 puzzles with greedy decoding only.
• The base instruct model's general chat ability was not evaluated after fine-tuning and may have degraded.
• No safety alignment beyond what the base model carried.

Files

path	contents
model.safetensors, config.json, tokenizer*, chat_template.jinja, generation_config.json	the model, bf16
training_config.json	ES hyperparameters of the run
training/metrics.jsonl, training/train.log	per-iteration metrics and console log, iterations 50–200
training/eval-output/*.json	every held-out response and its grade, iterations 60–200
gguf/*.gguf	Q8_0 and Q4_K_M quantizations for llama.cpp
assets/*.png	the figures above, light and dark

Citation

@article{Qiu2026EvolutionStrategies,
  title={Evolution Strategies at Scale: {LLM} Fine-Tuning Beyond Reinforcement Learning},
  author={Qiu, Xin and Gan, Yulu and Hayes, Conor F. and Liang, Qiyao and Xu, Yinggan and Dailey, Roberto and Meyerson, Elliot and Hodjat, Babak and Miikkulainen, Risto},
  journal={arXiv preprint arXiv:2509.24372},
  year={2026},
  eprint={2509.24372},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  doi={10.48550/arXiv.2509.24372}
}