u-opsd/qwen3-8b-non-thinking

u-OPSD — Qwen3-8B (non-thinking)

LoRA adapter for Qwen/Qwen3-8B trained with unsupervised On-Policy Self-Distillation (u-OPSD): a label-free variant of OPSD in which the teacher is conditioned on a majority-vote pseudo-label derived from the model's own rollouts instead of a ground-truth solution.

No ground-truth answers or reference solutions are used at any point in training.

On five math benchmarks the adapter improves the five-benchmark average from 43.57 → 54.31 (+10.74) over the base model.

Results

Five-benchmark evaluation, non-thinking inference, temperature 1.0. AIME24 / AIME25 / HMMT25 are avg@12; MATH500 / AMC23 are avg@4.

Model	AIME24	AIME25	HMMT25	MATH500	AMC23	Avg.
Qwen3-8B (base)	27.50	23.33	13.61	84.05	69.38	43.57
SFT	26.94	21.67	11.94	84.10	72.50	43.43
OPSD (supervised)	41.67	28.06	18.33	87.15	85.00	52.04
u-OPSD (this adapter)	45.56	34.72	18.61	89.55	83.12	54.31

The SFT and supervised OPSD rows are runs of the same codebase under the same evaluation protocol, at the same step; both consume ground-truth solutions, this adapter does not.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "Qwen/Qwen3-8B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "u-opsd/qwen3-8b-non-thinking")

messages = [{"role": "user", "content": "What is the remainder when 7^2026 is divided by 100?"}]
text = tok.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,   # this adapter is trained and evaluated in non-thinking mode
)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=4096)
print(tok.decode(out[0], skip_special_tokens=True))

With vLLM, pass the adapter as a LoRA request against the Qwen/Qwen3-8B base and set max_lora_rank=64.

Non-thinking only. Student and teacher were both trained with enable_thinking=False, and all reported numbers use non-thinking inference. Enabling thinking mode is untested.

Method

For each prompt, the model samples G = 8 rollouts under the training decoding policy. The most frequent final answer becomes the pseudo-label. A prompt is kept only if the pseudo-label's share of the rollouts reaches the self-consistency threshold τ = 0.5; otherwise the prompt is dropped for that step. One rollout that disagrees with the pseudo-label is then selected at random as the distillation target, and the teacher is conditioned on an agreeing rollout in place of OPSD's ground-truth solution. Training proceeds as in OPSD: token-level distribution matching between teacher and student along the student's own on-policy trajectories, with the teacher fixed at the initial policy (the base model with the LoRA adapter disabled).

Training details

Base model	Qwen/Qwen3-8B
Dataset	siyanzhao/Openthoughts_math_30k_opsd (prompts only; solutions unused)
Objective	token-level distribution matching, beta = 0 (forward KL)
Teacher	fixed at initial policy (--fixed_teacher)
Rollouts per prompt	8
Self-consistency threshold	0.5
Distillation rows	1, selected at random
Max completion length	4096
Sampling (training)	temperature 1.1, top-p 0.95, top-k 20
LoRA	r 64, alpha 128, dropout 0.05, on q/k/v/o/gate/up/down projections
Optimizer	lr 5e-6 with linear decay over 150 steps, max grad norm 0.1
Precision	bfloat16, FlashAttention-2, gradient checkpointing
Rollout backend	vLLM (colocate)
Released checkpoint	step 150 (final)

Evaluation protocol

vLLM, temperature 1.0, non-thinking inference. AIME24 / AIME25 / HMMT25 at 12 samples per problem, MATH500 / AMC23 at 4. Answers are verified with math_verify.

Limitations

• Single seed. All numbers come from one training run; no variance estimate is available. Repeated evaluations of the untrained base model on this suite vary by a couple of points, so treat margins of that order as suggestive.
• Scope. Trained and evaluated on English competition mathematics in non-thinking mode. Behaviour outside that scope, including thinking mode, other domains, and safety-relevant use, is untested.
• Pseudo-label noise. Supervision comes from the model's own majority vote, which can be confidently wrong; the threshold τ = 0.5 filters low-agreement prompts but does not guarantee correctness.

Citation

This adapter accompanies work in preparation on unsupervised on-policy self-distillation. It builds directly on OPSD:

@article{zhao2026self,
  title={Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models},
  author={Zhao, Siyan and Xie, Zhihui and Liu, Mengchen and Huang, Jing and Pang, Guan and Chen, Feiyu and Grover, Aditya},
  journal={arXiv preprint arXiv:2601.18734},
  year={2026}
}