u-opsd/qwen3-4b-thinking

u-OPSD — Qwen3-4B (thinking)

LoRA adapter for Qwen/Qwen3-4B trained with unsupervised On-Policy Self-Distillation (u-OPSD): a label-free variant of OPSD in which the teacher is conditioned on a majority-vote pseudo-label derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from 74.85 → 77.05 (+2.20) over the base model, in thinking mode.

Results

Five-benchmark evaluation, thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

Model	AIME24	AIME25	HMMT25	MATH500	AMC23	Avg.
Qwen3-4B (base)	74.17	64.72	45.56	94.80	95.00	74.85
OPSD (supervised)	75.28	68.06	43.06	95.20	99.38	76.20
u-OPSD (this adapter)	76.39	68.06	46.94	95.75	98.12	77.05

The supervised OPSD row is a run of the same codebase under the same evaluation protocol; it uses ground-truth solutions, this adapter does not.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-4B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-4b-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True,    # this adapter is trained and evaluated in thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=32768)
print(tok.decode(out[0], skip_special_tokens=True))

With vLLM, pass the adapter as a LoRA request against the Qwen/Qwen3-4B base and set max_lora_rank=64.

Thinking only. Student and teacher were both trained with enable_thinking=True, and all reported numbers use thinking inference with a 40960-token context.

Method

For each prompt, the model samples G = 8 rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label, and a prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold τ = 0.5. This adapter uses the all-agree variant: when every rollout agrees, the shortest one becomes the distillation target and the teacher is conditioned on the longest, so prompts the model already answers consistently still contribute a signal. Training proceeds as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

Training details

Base model	Qwen/Qwen3-4B
Dataset	siyanzhao/Openthoughts_math_30k_opsd (prompts only; solutions unused)
Objective	token-level distribution matching, beta = 0 (forward KL), token loss clip 0.05
Teacher	fixed at initial policy (--fixed_teacher), reference = longest agreeing rollout
Rollouts per prompt	8
Self-consistency threshold	0.5
All-agree distillation target	shortest rollout
Distillation rows	1, selected at random
Max completion length	4096
Sampling (training)	temperature 1.1, top-p 0.95, top-k 20
LoRA	r 64, alpha 128, dropout 0.05, on q/k/v/o/gate/up/down projections
Optimizer	lr 5e-6 with linear decay over 150 steps, max grad norm 0.1
Batch	8 GPUs x 1 per device x 4 grad accum (32 prompts per step)
Precision	bfloat16, FlashAttention-2, gradient checkpointing
Rollout backend	vLLM (colocate)
Released checkpoint	step 25 of 150

Evaluation protocol

vLLM, temperature 1.0, thinking inference, 40960-token context. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with math_verify.

Limitations

• Single seed. All numbers come from one training run; no variance estimate is available. Repeated evaluations of the untrained base model on this suite vary by a couple of points, so treat margins of that order as suggestive.
• Scope. Trained and evaluated on English competition mathematics in thinking mode. Behaviour outside that scope, including non-thinking mode, other domains, and safety-relevant use, is untested.
• Pseudo-label noise. Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold filters low-agreement prompts but does not guarantee correctness.

Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}