trl/pr_5607/en/paper_index.md

Paper Index

Group Relative Policy Optimization

Papers relating to the GRPOTrainer.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

📜 Paper: https://huggingface.co/papers/2402.03300

Introduces Group Relative Policy Optimization (GRPO) and shows strong math-reasoning gains from math-centric pretraining plus group-relative PPO-style optimization. Used in TRL via GRPOTrainer.

from trl import GRPOConfig, GRPOTrainer

# The paper doesn't specify its hyperparameters, so here we provide hyperparameters from "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning" instead.
training_args = GRPOConfig(
    loss_type="grpo",
    beta=0.001,  # "the KL coefficient to 0.001"
    epsilon=10.0, # "the GRPO clip ratio ϵ to 10"
    num_generations=16,  # "For each question, we sample 16 outputs..."
    max_completion_length=32_768,  # "...with a maximum length of 32,768"
    steps_per_generation=16,  # "To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 minibatches"
    # "resulting in a training batch size of 512". One way to achieve this setting with 1 device is per_device_train_batch_size=4, gradient_accumulation_steps=128
    per_device_train_batch_size=4,
    gradient_accumulation_steps=128,  
)
trainer = GRPOTrainer(
    ...,
    args=training_args,
)

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

📜 Paper: https://huggingface.co/papers/2501.12948

DeepSeek-R1 achieves reasoning performance comparable to OpenAI-o1 through a multi-stage pipeline that transitions from pure reinforcement learning (RL) to a refined, human-aligned model. Unlike its predecessor, DeepSeek-R1-Zero, which used pure RL on a base model, R1 follows a structured four-stage evolution:

1. Cold Start: The base model is fine-tuned on a small set of high-quality, long Chain-of-Thought (CoT) data to provide a stable starting point.
2. Reasoning-Oriented RL: Large-scale RL is applied to enhance performance in math, coding, and logic, using rule-based rewards and a language consistency reward to reduce language mixing.
3. Rejection Sampling & SFT: The RL checkpoint generates 600k reasoning samples via rejection sampling, which are combined with 200k non-reasoning (general) samples to create a new dataset for a second round of Supervised Fine-Tuning.
4. RL for all Scenarios: A final RL stage aligns the model with human preferences (helpfulness and harmlessness) across all domains while maintaining reasoning strength.

Distillation: Empowering Small Models

A key contribution of the paper is demonstrating that reasoning patterns can be distilled from a large model (DeepSeek-R1) into smaller dense models (e.g., Qwen and Llama series). Distillation was found to be more effective for small models than training them with pure RL from scratch.

You can use the GRPOTrainer to replicate the reasoning-heavy stages of this pipeline.

from trl import GRPOConfig, GRPOTrainer

# Example configuration for a reasoning-oriented GRPO stage
# Based on the Open-R1 recipe for Qwen-7B
training_args = GRPOConfig(
    learning_rate=4.0e-5,
    max_prompt_length=4096,
    max_completion_length=32768, # Support for long Chain-of-Thought
    num_generations=16,          # Sample 16 outputs per prompt for group relative advantage
    beta=0.001,                  # KL coefficient
    use_vllm=True,               # Use vLLM backend for accelerated rollout generation
)

trainer = GRPOTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    reward_funcs=[accuracy_reward, format_reward], # R1-Zero used rule-based rewards
)

trainer.train()

Group Sequence Policy Optimization

📜 Paper: https://huggingface.co/papers/2507.18071

GSPO is a GRPO variant that computes importance sampling weights at the sequence level instead of per-token. To reproduce the paper's setting, use this configuration:

from trl import GRPOConfig

training_args = GRPOConfig(
    importance_sampling_level="sequence",
    loss_type="grpo",
    beta=0.0,  # GSPO set KL regularization to zero: https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306 
    epsilon=3e-4,  # GSPO paper (v2), section 5.1
    epsilon_high=4e-4,  # GSPO paper (v2), section 5.1
    gradient_accumulation_steps=1,
    steps_per_generation=4,  # partition rollout batch into 4 mini-batches. GSPO paper (v2), section 5.1. Must be 4 times gradient_accumulation_steps
)

Note that this method only has an effect when training goes slightly off-policy—for example, when steps_per_generation > gradient_accumulation_steps or num_iterations > 1. Otherwise, it is effectively equivalent to no modification.

TRL also provide an experimental implementation of GSPO-token, see Experimental - GSPO-Token.

Policy ratio: GRPO vs. GSPO

In GSPO, the policy ratio is defined at the sequence-level. In other words, it is the ratio between the probability of the current policy generating a sequence over the old policy generating that same sequence.

The sequence likelihood is defined as:

\pi_\theta (o_i | q) = \prod_{t=1}^{|o_i|} \pi_\theta (o_{i,t} | q, o_{i, 0`: ```python from trl import GRPOConfig training_args = GRPOConfig( ..., beta=0.001, # the paper doesn't specify the value used, so we use the value from "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning" use_bias_correction_kl=True, ) ``` - The **Off-Policy Masking**, which stabilizes training by ignoring sequences where the policy performs poorly (negative advantage) **and** has drifted significantly from the old policy (high KL divergence). The off-policy binary mask \\(\textcolor{red}{M_{i,t}}\\) is defined as: \textcolor{red}{M_{i,t}} = \begin{cases} 0 & \text{if } \hat{A}{i,t} \textcolor{blue}{\delta} \ 1 & \text{otherwise} \end{cases} $$ThismaskisthenappliedtotheGRPOlossasfollows:This\; mask\; is\; then\; applied\; to\; the\; GRPO\; loss\; as\; follows:$$ \mathcal{L}{\text{GRPO}}(\theta) = -\frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left[ \min \left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i, 0 (optimism coefficient) and β > 0 (KL regularization) in Algorithm 1 but does not specify numerical values. The following configuration uses TRL defaults:

from trl.experimental.xpo import XPOConfig

training_args = XPOConfig(
    alpha=1e-5,  # α exploration bonus weight, α ≥ 0 where α=0 reduces to online DPO (TRL default)
    beta=0.1,  # β KL regularization coefficient (TRL default)
)

Distillation

Papers relating to training a student model with the help of a teacher model.

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes

📜 Paper: https://huggingface.co/papers/2306.13649

Introduces Generalized Knowledge Distillation (GKD), which addresses distribution mismatch in KD for auto-regressive models by training the student on its own generated outputs with teacher feedback, instead of a fixed set of sequences. GKD supports flexible loss functions (e.g. beyond KL when the student cannot match the teacher) and integrates with RL fine-tuning (RLHF). The paper reports results on summarization, translation, arithmetic reasoning, and instruction-tuning. Used in TRL via experimental.distillation.DistillationTrainer and experimental.gkd.GKDTrainer. To reproduce the paper's setting, use this configuration:

from trl.experimental.distillation import DistillationConfig

# XSum summarization task (Table A.1 of the paper)
training_args = DistillationConfig(
    lmbda=0.5,  # λ student data fraction (Section 3 of the paper)
    beta=0.5,  # β Generalized JSD interpolation, 0=KL, 1=reverse KL (Section 3 of the paper)
    temperature=1.0,  # student training temperature (Appendix A of the paper)
    max_steps=40000,  # training steps (Table A.1 of the paper)
    learning_rate=3e-4,  # learning rate (Table A.1 of the paper)
    per_device_train_batch_size=32,  # batch size (Table A.1 of the paper)
    warmup_steps=2000,  # warm-up steps (Table A.1 of the paper)
    max_completion_length=64,  # max output tokens (Table A.1 of the paper)
)

On-Policy Distillation

📰 Blog: https://thinkingmachines.ai/blog/on-policy-distillation/

On-Policy Distillation involves a student model generating rollouts for each batch of training data. We subsequently obtain the probability distributions for each token of the rollouts from both the student and teacher models. The student model is then optimized to minimize the negative Kullback-Leibler (KL) divergence between its own token distributions and those of the teacher model.

Method	Sampling	Reward signal
Supervised finetuning	off-policy	dense
Reinforcement learning	on-policy	sparse
On-policy distillation	on-policy	dense

On-Policy Distillation has been shown to outperform SFT, GRPO and can be used to restore generalization capabilities lost during SFT.

Additionally on-policy distillation is more compute efficient and is less prone to overfitting when trained with limited data.

To train a model with on-policy distillation using TRL, you can use the following configuration, with the experimental.distillation.DistillationTrainer and experimental.distillation.DistillationConfig:

from trl.experimental.distillation import DistillationConfig

training_args = DistillationConfig(
    lmbda=1.0,  # student produces rollouts for all batches
    beta=1.0,  # to ensure reverse-kl as the loss function
    teacher_model_name_or_path="teacher-model",  # specify the teacher model
)

Alternatively, you can use the experimental.gkd.GKDTrainer and experimental.gkd.GKDConfig:

from trl.experimental.gkd import GKDConfig

training_args = GKDConfig(
    lmbda=1.0,  # student produces rollouts for all batches
    beta=1.0,  # to ensure reverse-kl as the loss function
    teacher_model_name_or_path="teacher-model",  # specify the teacher model
)

You can also use the GOLDTrainer and GOLDConfig to perform on-policy distillation with a similar configuration:

from trl.experimental import GOLDConfig

config = GOLDConfig(
    lmbda=1.0, # student produces rollouts for all batches
    beta=1.0, # to ensure reverse-kl as the loss function
    teacher_model_name_or_path="teacher-model", # specify the teacher model

)

Knowledge Distillation of Large Language Models

📜 Paper: https://huggingface.co/papers/2306.08543

MiniLLM is the first on-policy knowledge distillation method, which minimizes the sequence-level reverse KLD between the teacher and the student model and is optimized by reinforcement learning.

It is a generalized version of Think Machine Lab's On-Policy Distillation, with the option to add distribution-level single-step distillation signals (like GKD when beta=1) and long-context reverse KLD signals.

Alternatively, you can use the experimental.MiniLLMTrainer and experimental.MiniLLMConfig to perform MiniLLM distillation as follows:

from datasets import load_dataset
from trl.experimental.minillm import MiniLLMTrainer

dataset = load_dataset("trl-lib/tldr", split="train")

trainer = MiniLLMTrainer(
    model="Qwen/Qwen3-0.6B",
    teacher_model="Qwen/Qwen3-1.7B",
    train_dataset=dataset,
)
trainer.train()

For more details, see the MiniLLM Trainer documentation documentation.

Reinforcement Learning via Self-Distillation

📜 Paper: https://huggingface.co/papers/2601.20802

Self-Distillation Policy Optimization (SDPO) enhances reinforcement learning with verifiable rewards by converting rich textual feedback (e.g., runtime errors, judge evaluations) into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts.

from trl.experimental.sdpo import SDPOConfig, SDPOTrainer

training_args = SDPOConfig(
    distillation_alpha=0.5,                # Jensen-Shannon divergence (recommended)
    distillation_topk=100,                 # Top-K logit distillation approximation
    full_logit_distillation=True,          # Required for top-K logit-level SDPO
    distillation_is_clip=2.0,              # Importance sampling clipping
    distillation_weight=1.0,               # Weight for self-distillation loss
    sdpo_policy_loss_mode="distillation_only",
    use_successful_as_teacher=True,        # Use successful rollouts as teacher
    teacher_regularization="ema",          # Supported: "ema", "none"
    teacher_update_rate=0.05,              # EMA update rate
    include_environment_feedback=False,    # Use dataset privileged_context when available
)

trainer = SDPOTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    reward_funcs=...,
    args=training_args,
    train_dataset=...,
)
trainer.train()

Expected dataset columns:

• prompt
• privileged_context for optional environment feedback

For more details, see the SDPO Trainer documentation.

Self-Training with On-Policy Self-Distillation for Language Model Alignment

📜 Paper: https://huggingface.co/papers/2601.19897

Self-Distilled Fine-Tuning (SDFT) performs on-policy self-distillation by generating completions during training, then distilling an explicit teacher-conditioned view of those same completions back into the student. In TRL, SDFT uses a shared self-distillation core with SDPO where the teacher is the model itself (base weights with adapter disabled for PEFT, or the same model under no_grad for non-PEFT). The teacher prompt is composed internally from the student prompt plus the dataset privileged_context.

from datasets import Dataset

from trl.experimental.sdft import SDFTConfig, SDFTTrainer

dataset = Dataset.from_dict(
    {
        "prompt": [[{"role": "user", "content": "Solve 2+2."}]],
        "privileged_context": ["Example answer: 4."],
    }
)

training_args = SDFTConfig(
    distillation_alpha=0.5,
    distillation_topk=5,
    max_completion_length=64,
)

trainer = SDFTTrainer(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    args=training_args,
    train_dataset=dataset,
)
trainer.train()

Expected dataset columns:

• prompt
• privileged_context containing only the extra teacher-only information

For more details, see the SDFT Trainer documentation.

Embarrassingly Simple Self-Distillation Improves Code Generation

📜 Paper: https://huggingface.co/papers/2604.01193

Simple Self-Distillation (SSD) improves code generation by sampling completions from the model at a training-time temperature and truncation configuration, then fine-tuning on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or reinforcement learning is needed. SSD reshapes token distributions in a context-dependent way: suppressing distractor tails at "lock" positions (where syntax leaves little ambiguity) while preserving diversity at "fork" positions (where multiple valid continuations exist).

from trl.experimental.ssd import SSDConfig, SSDTrainer

training_args = SSDConfig(
    temperature=0.6,                       # Training-time sampling temperature (T_train)
    top_k=20,                              # Training-time top-k truncation
    top_p=0.95,                            # Training-time top-p truncation
    max_completion_length=65536,
    learning_rate=5e-6,
)

trainer = SSDTrainer(
    model="Qwen/Qwen3-4B-Instruct",
    args=training_args,
    train_dataset=...,
)
trainer.train()

Expected dataset columns:

• prompt

For more details, see the SSD Trainer documentation.

Distributed Training

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

📜 Paper: https://huggingface.co/papers/1910.02054

ZeRO (Zero Redundancy Optimizer) eliminates memory redundancies in data- and model-parallel training by partitioning optimizer states, gradients, and parameters across devices while retaining low communication volume and high computational granularity. This allows for the efficient training of large models that would otherwise not fit in GPU memory.

TRL supports ZeRO via the DeepSpeed integration. To use it, provide a DeepSpeed configuration file with your desired settings,

# config.yaml
distributed_type: DEEPSPEED
num_processes: 2
deepspeed_config:
  zero_stage: 3

and launch the training script using accelerate launch --config_file config_file.

accelerate launch --config_file config.yaml train.py

Proximal Policy Optimization

Papers relating to the experimental.ppo.PPOTrainer

Proximal Policy Optimization Algorithms

📜 Paper: https://huggingface.co/papers/1707.06347

Introduces Proximal Policy Optimization (PPO): policy gradient methods that alternate between collecting rollouts and optimizing a clipped surrogate objective over multiple minibatch epochs. PPO retains benefits of trust-region methods (e.g. TRPO) with simpler implementation and strong empirical sample efficiency, and was validated on robotics and Atari benchmarks. Used in TRL via experimental.ppo.PPOTrainer. To use PPO with TRL, use this configuration:

from trl.experimental.ppo import PPOConfig

training_args = PPOConfig(
    cliprange=0.2,  # ε clipping range (Section 3 and Table 3 of the paper, Mujoco setting)
    num_ppo_epochs=4,  # K epochs of minibatch updates (TRL default; paper uses K=10 Mujoco, K=3 Atari)
    gamma=1.0,  # γ discount factor (TRL default for LLM tasks; paper uses γ=0.99)
    lam=0.95,  # λ GAE parameter (Table 3 of the paper, Mujoco setting)
    kl_coef=0.05,  # KL penalty coefficient (Section 4 of the paper discusses adaptive KL)
    vf_coef=0.1,  # c₁ value function loss weight (Equation 9 of the paper)
)