Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM
reasoning. However, the optimization behavior of ES remains understudied, making it hard to define
its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy
Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paperfirst identifies a performance advantage of ES over GRPO, theoretically and empirically showing
that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of
pretrained LLMs. Theoretically, we show that verifier-projected Jensen–Shannon diversity across the
ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits
entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop
a sequential GRPO–ES training strategy that combines GRPO’s strength in Pass@1 with ES’s gains in
Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance
gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional
sparsity suggests that large parameter movement need not imply widespread functional change, and
held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally,we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a
smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training
paradigm rather than a less effective, memory-efficient alternative to GRPO.