README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-8B
pipeline_tag: text-generation
library_name: transformers
language:
  - en
tags:
  - spyrl
  - rlsvr
  - self-play
  - reinforcement-learning
  - mathematical-reasoning
  - reasoning
---

# SpyRL-Qwen3-8B-Math

Qwen3-8B trained with SpyRL on mathematical reasoning. In the performing stage each agent designs *and* solves a problem grounded in a math-heavy web document; the spy sees the same document with a contiguous 40% span masked out.

Trained with **SpyRL**, the reference implementation of **RLSVR** (Reinforcement Learning with Self-Verifiable Rewards) from the COLM 2026 paper *From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement*.

RLSVR extends RLVR to open-ended tasks the way self-supervised learning extends supervised learning: instead of approximating a missing reward with a judge or reward model, it *transforms* the task into a proxy environment whose own rules generate the reward. SpyRL instantiates that as a multi-agent self-play game inspired by *Who Is the Spy?* — civilians receive the full input, one spy receives a masked copy, all agents perform the same target task, and then they vote on who the spy is. Because the environment assigns the spy identity up front, the vote is exactly checkable, and avoiding suspicion requires producing genuinely better output.

**No human annotation, no reward model, no LLM judge was used to train this model.**

## Model details

| | |
|---|---|
| **Base model** | [`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) |
| **Target task** | Mathematical reasoning |
| **Self-play corpus** | [`nvidia/Nemotron-CC-Math-v1`](https://huggingface.co/datasets/nvidia/Nemotron-CC-Math-v1) |
| **Algorithm** | GRPO, alternating performing / detection stages |
| **Training** | 100 iterations, 1 node × 8 GPUs |
| **Supervision** | None — reward comes from the game's voting rules |

Exact group size, masking ratio and the rest of the configuration are in the launch script linked
below.

## Results

| Benchmark | Base (Qwen3-8B) | **+ SpyRL** |
|---|---|---|
| GSM8K | 91.8 | **93.5** |
| Math500 | 74.2 | **81.2** |
| AIME 24 | 15.3 | **20.0** |
| AIME 25 | 12.1 | **23.3** |
| Minerva | 49.3 | **56.3** |
| MMLU-Pro | 58.1 | **63.1** |
| GPQA-Diamond | 33.3 | **39.8** |

Average gain over the base model across the seven benchmarks: **+6.16%**.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SpyRL/SpyRL-Qwen3-8B-Math"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))
```

The chat template and tokenizer are inherited unchanged from the base model.

## Training code

Full training code, launch scripts and per-task environments: [https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning](https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning)

```bash
git clone https://github.com/wangqinsi1/SpyRL-Self-PlaY-Reinforcement-Learning.git && cd SpyRL-Self-PlaY-Reinforcement-Learning
conda create -n spyrl python=3.10 -y && conda activate spyrl
bash setup.sh
bash spyrl/train_math_reasoning.sh
```

## Citation

```bibtex
@inproceedings{wang2026spyrl,
    title     = {From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement},
    author    = {Qinsi Wang and Jing Shi and Huazheng Wang and Kun Wan and Yiran Wu and Bo Liu and Qingyun Wu and Hai Helen Li and Yiran Chen and Handong Zhao and Wentian Zhao},
    booktitle = {Conference on Language Modeling (COLM)},
    year      = {2026}
}
```