Instructions to use leizhao7/Qwen3-4B-OPD-RL-Code-Step200 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use leizhao7/Qwen3-4B-OPD-RL-Code-Step200 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="leizhao7/Qwen3-4B-OPD-RL-Code-Step200")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("leizhao7/Qwen3-4B-OPD-RL-Code-Step200")
model = AutoModelForCausalLM.from_pretrained("leizhao7/Qwen3-4B-OPD-RL-Code-Step200", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use leizhao7/Qwen3-4B-OPD-RL-Code-Step200 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "leizhao7/Qwen3-4B-OPD-RL-Code-Step200"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leizhao7/Qwen3-4B-OPD-RL-Code-Step200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/leizhao7/Qwen3-4B-OPD-RL-Code-Step200

SGLang

How to use leizhao7/Qwen3-4B-OPD-RL-Code-Step200 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "leizhao7/Qwen3-4B-OPD-RL-Code-Step200" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leizhao7/Qwen3-4B-OPD-RL-Code-Step200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "leizhao7/Qwen3-4B-OPD-RL-Code-Step200" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "leizhao7/Qwen3-4B-OPD-RL-Code-Step200",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use leizhao7/Qwen3-4B-OPD-RL-Code-Step200 with Docker Model Runner:
```
docker model run hf.co/leizhao7/Qwen3-4B-OPD-RL-Code-Step200
```

Qwen3-4B-OPD-RL-Code-Step200

This is the final student checkpoint after 200 on-policy distillation (OPD) updates from Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals. It starts from Qwen/Qwen3-4B and learns from the fixed teacher Qwen3-4B-Non-Thinking-RL-Code-Step300 (public reference). The uploaded model is the distilled student; the teacher's weights are not included.

Paper · Project page · Code and numerical results

Training

The run uses a reverse-KL OPD objective approximated with the student's top 16 tokens, student-probability weights, and a global mean over valid response tokens. Student-generated responses provide the prefixes for fixed-teacher supervision.

Setting	Value
Initial student	`Qwen/Qwen3-4B`
Teacher training alias	`Qwen3-4B-Non-Thinking-RL-Code-Step300`
Training data	`PRIME-RL/Eurus-2-RL-Data` (code subset)
Final training step	200
Optimizer	AdamW; learning rate 1e-6; constant schedule
Prompt batch / rollouts per prompt	64 / 1
Maximum prompt / response tokens	2048 / 16384
Sampling temperature	1.0
Public recipe	`code_rl`

Qwen3 non-thinking chat template (enable_thinking=False). See the experiment map for launcher provenance and the reproducibility notes for the historical evaluation protocol.

Step-200 evaluation

The following numbers are archived results of the paper run. Accuracy is avg@4, the mean correctness of sampled responses, macro-averaged over the three benchmarks. This is not pass@k, which counts whether any sample succeeds. The maximum validation response length is 16384 tokens, with temperature 1.0 and top-p 1.0.

Benchmark	Final avg@4 (%)
HumanEval+	86.13
MBPP+	70.44
LiveCodeBench	24.62
Macro average	60.39

The initial student scored 52.69%; the step-200 checkpoint scores 60.39% (+7.70 percentage points). The final logged top-16 loss is 0.00069608, a 99.4% reduction from update 1. This is the final reduction, not the minimum loss observed earlier in training, and the logged surrogate is not the full-vocabulary KL.

Completed 2026-09-23 reproduction on 2 H200 GPUs with native checkpoint resumes, not the older 149-step run. Eight responses were generated; avg@4 uses the archived fixed reward-independent four-of-eight selection at every validation checkpoint.
These are archived paper-run metrics; the exported/uploaded checkpoint has not yet been independently re-evaluated during packaging.

Use

Load this repository with AutoTokenizer and AutoModelForCausalLM (Transformers 4.56.2 or a compatible newer version, with PyTorch and Accelerate). Use its saved chat template; the example below is a short inference example, not a reproduction of the benchmark protocol.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "leizhao7/Qwen3-4B-OPD-RL-Code-Step200"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)
messages = [{"role": "user", "content": "Write a Python function that returns the unique elements of a list in order."}]
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,

    enable_thinking=False,

    return_dict=True,
    return_tensors="pt",
).to(model.device)
output = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=True,
    temperature=1.0,
    top_p=1.0,
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Scope and limitations

This is a research checkpoint for reproducing and studying OPD. The paper reports one run per condition under a fixed 200-update horizon. These results do not establish broad general-purpose performance, safety, or that loss plateaus persist under every training configuration. The checkpoint retains the limitations of its initial student and can produce incorrect answers or code.

License and provenance

This OPD checkpoint modifies the Apache-2.0 Qwen3 student. Preserve the upstream Apache-2.0 license and attribution, and state that the weights were modified by 200 OPD updates. No teacher weights are included. See LICENSE and UPSTREAM_ATTRIBUTION.md. The code repository's MIT license is separate from the model's upstream terms.

The linked Keven16 repository is a verified public checkpoint with the same name. The released local training configuration does not establish that this was the historical download origin; no historical HF source ID or revision is asserted. The teacher repository is linked for attribution and reproducibility; it is not listed as the initial student. Training-time upstream revisions are not recoverable from the public numerical release alone. This release includes SHA256SUMS, checkpoint provenance, and validation.json documenting export checks separately from the archived benchmark results.

Citation

@misc{zhao2026opdlearningsignals,
  title={Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals},
  author={Lei Zhao and Qichao Zhao and Bowen Zuo and Qishi Zhan},
  year={2026},
  eprint={2610.11247},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2610.11247}
}