# Paper Index ## Group Relative Policy Optimization Papers relating to the [`GRPOTrainer`]. ### DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models **๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.03300 Introduces Group Relative Policy Optimization (GRPO) and shows strong math-reasoning gains from math-centric pretraining plus group-relative PPO-style optimization. Used in TRL via [`GRPOTrainer`]. ```python from trl import GRPOConfig, GRPOTrainer # The paper doesn't specify its hyperparameters, so here we provide hyperparameters from "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning" instead. training_args = GRPOConfig( loss_type="grpo", beta=0.001, # "the KL coefficient to 0.001" epsilon=10.0, # "the GRPO clip ratio ฯต to 10" num_generations=16, # "For each question, we sample 16 outputs..." max_completion_length=32_768, # "...with a maximum length of 32,768" steps_per_generation=16, # "To accelerate training, each rollout generates 8,192 outputs, which are randomly split into 16 minibatches" # "resulting in a training batch size of 512". One way to achieve this setting with 1 device is per_device_train_batch_size=4, gradient_accumulation_steps=128 per_device_train_batch_size=4, gradient_accumulation_steps=128, ) trainer = GRPOTrainer( ..., args=training_args, ) ``` ### DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning **๐Ÿ“œ Paper**: https://huggingface.co/papers/2501.12948 DeepSeek-R1 achieves reasoning performance comparable to OpenAI-o1 through a multi-stage pipeline that transitions from pure reinforcement learning (RL) to a refined, human-aligned model. Unlike its predecessor, DeepSeek-R1-Zero, which used pure RL on a base model, R1 follows a structured four-stage evolution: 1. Cold Start: The base model is fine-tuned on a small set of high-quality, long Chain-of-Thought (CoT) data to provide a stable starting point. 2. Reasoning-Oriented RL: Large-scale RL is applied to enhance performance in math, coding, and logic, using rule-based rewards and a language consistency reward to reduce language mixing. 3. Rejection Sampling & SFT: The RL checkpoint generates 600k reasoning samples via rejection sampling, which are combined with 200k non-reasoning (general) samples to create a new dataset for a second round of Supervised Fine-Tuning. 4. RL for all Scenarios: A final RL stage aligns the model with human preferences (helpfulness and harmlessness) across all domains while maintaining reasoning strength. Distillation: Empowering Small Models A key contribution of the paper is demonstrating that reasoning patterns can be distilled from a large model (DeepSeek-R1) into smaller dense models (e.g., Qwen and Llama series). Distillation was found to be more effective for small models than training them with pure RL from scratch. You can use the GRPOTrainer to replicate the reasoning-heavy stages of this pipeline. ```python from trl import GRPOConfig, GRPOTrainer # Example configuration for a reasoning-oriented GRPO stage # Based on the Open-R1 recipe for Qwen-7B training_args = GRPOConfig( learning_rate=4.0e-5, max_prompt_length=4096, max_completion_length=32768, # Support for long Chain-of-Thought num_generations=16, # Sample 16 outputs per prompt for group relative advantage beta=0.001, # KL coefficient use_vllm=True, # Use vLLM backend for accelerated rollout generation ) trainer = GRPOTrainer( model=model, args=training_args, train_dataset=dataset, reward_funcs=[accuracy_reward, format_reward], # R1-Zero used rule-based rewards ) trainer.train() ``` ### Group Sequence Policy Optimization **๐Ÿ“œ Paper**: https://huggingface.co/papers/2507.18071 GSPO is a GRPO variant that computes importance sampling weights at the sequence level instead of per-token. To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig training_args = GRPOConfig( importance_sampling_level="sequence", loss_type="grpo", beta=0.0, # GSPO set KL regularization to zero: https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306 epsilon=3e-4, # GSPO paper (v2), section 5.1 epsilon_high=4e-4, # GSPO paper (v2), section 5.1 gradient_accumulation_steps=1, steps_per_generation=4, # partition rollout batch into 4 mini-batches. GSPO paper (v2), section 5.1. Must be 4 times gradient_accumulation_steps ) ``` Note that this method only has an effect when training goes slightly off-policyโ€”for example, when `steps_per_generation > gradient_accumulation_steps` or `num_iterations > 1`. Otherwise, it is effectively equivalent to no modification. TRL also provide an experimental implementation of GSPO-token, see [Experimental - GSPO-Token](experimental#gspo-token). #### Policy ratio: GRPO vs. GSPO In GSPO, the policy ratio is defined at the sequence-level. In other words, it is the ratio between the probability of the current policy generating a sequence over the old policy generating that same sequence. The sequence likelihood is defined as: $$ \pi_\theta (o_i | q) = \prod_{t=1}^{|o_i|} \pi_\theta (o_{i,t} | q, o_{i, < t} ), $$ where \\( \pi_\theta \\) is the policy \\( \pi \\) with parameters \\(\theta\\), \\( o_i \\) is the \\( i \\)-th output sequence \\( o \\) and \\(o_{i,t}\\) is the \\( t \\)-th token in this sequence, \\( q \\) is the input query. The sequence likelihood ratio \\( s_i (\theta) \\) is defined as: $$ s_i (\theta) = \left(\frac{\pi_\theta (o_i | q)}{\pi_{\theta_{old}} (o_i | q)} \right)^{\frac{1}{|o_i|}} $$ The exponent \\( \frac{1}{|o_i|} \\) represents a sequence-length normalization, minimizing the influence of sequence length in sequence likelihood. In other terms, it computes the geometric mean of token probabilities, ensuring a fair comparison across sequences of varying lengths. While GSPO defines the policy ratio at the sequence level, GRPO operates at the token level. Specifically, GRPO computes an importance ratio for each token in the sequence: $$ w_{i,t}(\theta) = \frac{\pi_\theta (o_{i,t} | q, o_{i,< t})}{\pi_{\theta_{\text{old}}} (o_{i,t} | q, o_{i,< t})} $$ This token-level ratio is then combined with a shared advantage \\( \hat{A}_i \\), and the GRPO objective clips and optimizes each token independently across the sequence. ### DAPO: An Open-Source LLM Reinforcement Learning System at Scale **๐Ÿ“œ Paper**: https://huggingface.co/papers/2503.14476 The DAPO algorithm includes 5 key components: - Overlong Filtering - Clip-Higher - Soft Overlong Punishment - Token-level Loss - Dynamic Sampling (โš ๏ธ Not supported in TRL) To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig, GRPOTrainer training_args = GRPOConfig( # Overlong Filtering mask_truncated_completions=True, # Token-level Loss loss_type="dapo", # Clip-Higher epsilon_high=0.28, # DAPO paper: section 4.1 epsilon=0.2, # DAPO paper: section 4.1 # Other parameters used per_device_train_batch_size=512, # mini-batch size for training in the paper, DAPO paper: section 4.1 num_generations=16, # number of sample responses in the paper, DAPO paper: section 4.1 max_completion_length=20480, # maximum number of tokens for generation in the paper, DAPO paper: section 4.1 beta=0.0, # section 2.3, DAPO paper ) # Soft Overlong Punishment sop_reward = get_soft_overlong_punishment(max_completion_len=20480, soft_punish_cache=4096) # DAPO paper: section 4.1 trainer = GRPOTrainer( ..., args=training_args, reward_funcs=[..., sop_reward], ) ``` ### INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning **๐Ÿ“œ Paper**: https://huggingface.co/papers/2505.07291 INTELLECT-2 is the first globally distributed reinforcement learning training run of a 32 billion parameter language model using fully asynchronous RL across a dynamic, heterogeneous swarm of permissionless compute contributors. The authors propose modifications to the standard GRPO training recipe, including two-sided GRPO clipping for increased training stability. To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig training_args = GRPOConfig( delta=4, # ฮด in section 4.1 of the paper epsilon=0.2, # ฮต in section 4.1 of the paper beta=0.001, # KL divergence coefficient in section 4.1 of the paper num_generations=16, # responses per prompt in section 4.1 of the paper learning_rate=3e-7, # section 4.1 of the paper ) ``` ### Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning **๐Ÿ“œ Paper**: https://huggingface.co/papers/2506.01939 A minority of tokens with high entropy act as reasoning "forks" in the CoT path, driving exploration and performance gains for RLVR, while low-entropy majority tokens contribute little or even impede learning. RLVR mainly adjusts high-entropy tokens, largely preserving the base modelโ€™s overall entropy patterns. Thus landing on the 80/20 rule, training on only 20% of the tokens with the highest entropy is comparable or supasses full-gradient updates for Qwen3 models. The paper's main results use vanilla DAPO (โš ๏ธ Dynamic Sampling is not supported in TRL). To replicate the main results, use the following configuration: ```python from trl import GRPOConfig, GRPOTrainer from trl.rewards import get_soft_overlong_punishment training_args = GRPOConfig( # --- vanilla DAPO parameters (80/20 rule: section 5.2) --- # # Overlong Filtering mask_truncated_completions=True, # Token-level Loss loss_type="dapo", # Clip-Higher epsilon_high=0.28, # DAPO paper: section 4.1 epsilon=0.2, # DAPO paper: section 4.1 # Other parameters used per_device_train_batch_size=512, # mini-batch size for training in the paper, DAPO paper: section 4.1 num_generations=16, # number of sample responses in the paper, DAPO paper: section 4.1 max_completion_length=20480, # maximum number of tokens for generation in the paper, DAPO paper: section 4.1 beta=0.0, # section 2.3, DAPO paper # --- Gradients on the highest entropy tokens --- # top_entropy_quantile=0.2 ) # Soft Overlong Punishment sop_reward = get_soft_overlong_punishment(max_completion_len=20480, soft_punish_cache=4096) # DAPO paper: section 4.1 trainer = GRPOTrainer( ..., args=training_args, reward_funcs=[..., sop_reward], ) ``` ### Dr. GRPO: Understanding R1-Zero-Like Training: A Critical Perspective **๐Ÿ“œ Paper**: https://huggingface.co/papers/2503.20783 A study of R1-Zero training identifies pretraining effects on RL performance and proffers Dr. GRPO to enhance token efficiency, achieving superior accuracy on AIME 2024. To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig training_args = GRPOConfig( loss_type="dr_grpo", per_device_train_batch_size=1, # train_batch_size_per_device in the Training section of the repository num_generations=8, # num_samples in the Training section of the repository max_completion_length=3000, # generate_max_length in the Training section of the repository beta=0.0, # ฮฒ in the Training section of the repository ) ``` ### Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning (Lite PPO) **๐Ÿ“œ Paper**: https://huggingface.co/papers/2508.08221 The authors of this paper find that the combination of: 1. scaling rewards by the standard deviation computed over the entire batch and 2. aggregating loss over the total number of tokens can unlock the learning capability of critic-free policies using vanilla PPO loss. Their results demonstrate that this simple combination consistently improves performance, surpassing strategies like GRPO and [DAPO](https://huggingface.co/papers/2503.14476). TRL supports using these learnings to train a GRPO model by: ```python from trl import GRPOConfig training_args = GRPOConfig( ... scale_rewards="batch", loss_type="dapo", # Other parameters used beta=0.0, # = init_kl_coef in the paper top_p=0.99, top_k=100, temperature=0.99, num_generations=8, # = num_return_sequences in the paper num_iterations=1, # = ppo_epochs in the paper per_device_train_batch_size=4, gradient_accumulation_steps=32, steps_per_generation=8, # (rollout_batch_size*num_return_sequences) / (per_device_train_batch_size*gradient_accumulation_steps) ) ``` Note that when using gradient accumulation, the loss is aggregated over the total number of tokens in the batch, but not over the accumulated batch. For more details, see the [GRPO Trainer - Loss types](grpo_trainer#loss_types). ### Truncated Importance Sampling **๐Ÿ“ฐ Blog**: https://fengyao.notion.site/off-policy-rl Online policy learning methods commonly use an optimized inference framework for rollout generation (e.g vLLM) that is separate from the training backend. This introduces a rollout-training mismatch, exemplified in the following PPO objective: $$ \small{ \mathbb{E}_{a\sim\textcolor{red}{\pi_{\text{inference}}}(\theta_{\mathrm{old}})} \Bigl[ \min\Bigl( \frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})}\,\hat A, \;\mathrm{clip}\bigl(\frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})},\,1-\epsilon,\,1+\epsilon\bigr)\,\hat A \Bigr) \Bigr] } $$ Despite \\( \textcolor{red}{\pi_{\text{inference}}} \\) and \\( \textcolor{blue}{\pi_{\text{training}}} \\) sharing the same model parameters \\( \theta \\), they can produce significantly different token probabilities. This unexpected behavior implicitly breaks the on-policy assumption, and silently turns training off-policy. Truncated Importance Sampling (TIS) addresses this issue by adapting the model update via importance-sampling correction. The gradient computation of the aforementioned PPO objective becomes $$ \small{ \mathbb{E}_{a\sim\textcolor{red}{\pi_{\text{inference}}}(\theta_{\mathrm{old}})} \Bigl[ \underbrace{\min(\frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})}{\textcolor{red}{\pi_{\text{inference}}}(a, \theta_{\mathrm{old}})}, C)}_{\text{truncated importance ratio}} \cdot \nabla_\theta \min\Bigl( \frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})}\,\hat A, \;\mathrm{clip}\bigl(\frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})},\,1-\epsilon,\,1+\epsilon\bigr)\,\hat A \Bigr) \Bigr] } $$ where \\( C \\) is a hyper-parameter. TIS is implemented in GRPO, and is enabled by selecting a `vllm_importance_sampling_mode` variant that includes the term `truncate`, such as `"sequence_truncate"` or `"token_truncate"`. ```python from trl import GRPOConfig training_args = GRPOConfig( ... use_vllm=True, vllm_importance_sampling_correction=True, # default True vllm_importance_sampling_mode="sequence_truncate", # or "token_truncate" vllm_importance_sampling_cap=2.0, # hyper-parameter C ) ``` ### Masked Importance Sampling **๐Ÿ“ฐ Blog**: https://ringtech.notion.site/icepop **๐Ÿ“ฐ Blog**: https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda Masked Importance Sampling (MIS) addresses the same issue as [Truncated Importance Sampling](#truncated-importance-sampling) but replaces clipping with masking. MIS takes a more decisive stance by discarding updates whose discrepancy exceeds a threshold \\( C \\). We apply upper-side masking, so any ratio above \\( C \\) is removed from the update. $$ \small{ \mathbb{E}_{a\sim\textcolor{red}{\pi_{\text{inference}}}(\theta_{\mathrm{old}})} \Bigl[ \underbrace{\mathbf{1}\left[ \frac{\pi_{\text{training}}(a, \theta_{\mathrm{old}})} {\pi_{\text{inference}}(a, \theta_{\mathrm{old}})} \le C \right] \cdot \frac{\pi_{\text{training}}(a, \theta_{\mathrm{old}})} {\pi_{\text{inference}}(a, \theta_{\mathrm{old}})}}_{\text{masked importance ratio}} \cdot \nabla_\theta \min\Bigl( \frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})}\,\hat A, \;\mathrm{clip}\bigl(\frac{\textcolor{blue}{\pi_{\text{training}}}(a, \theta)}{\textcolor{blue}{\pi_{\text{training}}}(a, \theta_{\mathrm{old}})},\,1-\epsilon,\,1+\epsilon\bigr)\,\hat A \Bigr) \Bigr] } $$ MIS is implemented for GRPO, and is enabled by selecting a `vllm_importance_sampling_mode` variant that includes the term `"mask"`, such as `"sequence_mask"` or `"token_mask"`. ```python from trl import GRPOConfig training_args = GRPOConfig( ... use_vllm=True, vllm_importance_sampling_correction=True, # default True vllm_importance_sampling_mode="sequence_mask", # or "token_mask" vllm_importance_sampling_cap=2.0, # hyper-parameter C ) ``` ### Sequence-level Importance Sampling **๐Ÿ“ฐ Blog**: https://yingru.notion.site/When-Speed-Kills-Stability-Demystifying-RL-Collapse-from-the-Training-Inference-Mismatch-271211a558b7808d8b12d403fd15edda The theoretically principled way to correct for the training-inference distribution shift is importance sampling, as introduced in the two papers above [Truncated Importance Sampling](#truncated-importance-sampling) and [Masked Importance Sampling](#masked-importance-sampling). However, the choice of formulation is crucial for keeping the gradient unbiased and ensuring stable training. This work shows that sequence-level importance sampling is the sound approach for addressing the trainingโ€“inference mismatch. Although token-level importance sampling achieves lower variance than a sequence-level ratio, it introduces bias and is therefore argued to be unsuitable for autoregressive models. The token-level gradient estimator is $$ \mathbb{E}_{x\sim\mathcal{D},\, y\sim \pi^{\text{inference}}_\theta(\cdot|x)} \Bigg[ R(x,y)\,\cdot\, \sum_{t=0}^{|y|-1} \frac{\pi^{\text{training}}_\theta(y_t\,|\,x, y_{ 0`: ```python from trl import GRPOConfig training_args = GRPOConfig( ..., beta=0.001, # the paper doesn't specify the value used, so we use the value from "DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning" use_bias_correction_kl=True, ) ``` - The **Off-Policy Masking**, which stabilizes training by ignoring sequences where the policy performs poorly (negative advantage) **and** has drifted significantly from the old policy (high KL divergence). The off-policy binary mask \\(\textcolor{red}{M_{i,t}}\\) is defined as: $$ \textcolor{red}{M_{i,t}} = \begin{cases} 0 & \text{if } \hat{A}_{i,t} < 0 \quad \text{and} \quad \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \log \frac{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i, \textcolor{blue}{\delta} \\ 1 & \text{otherwise} \end{cases} $$ This mask is then applied to the GRPO loss as follows: $$ \mathcal{L}_{\text{GRPO}}(\theta) = -\frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left[ \min \left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,< t})} \hat{A}_{i,t}, \, \text{clip}\left( \frac{\pi_\theta(o_{i,t} \mid q, o_{i,< t})}{\pi_{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,< t})}, 1 - \epsilon, 1 + \epsilon \right) \hat{A}_{i,t} \right) \textcolor{red}{M_{i,t}} - \beta \mathbb{D}_{\text{KL}}\left[\pi_\theta \| \pi_{\text{ref}}\right] \right] $$ To enable this feature, use the `off_policy_mask_threshold` (corresponding to \\( \textcolor{blue}{\delta} \\)) in the [`GRPOConfig`]: ```python from trl import GRPOConfig training_args = GRPOConfig( ..., off_policy_mask_threshold=0.5, ) ``` While the paper doesn't specify a \\( \textcolor{blue}{\delta} \\) value used, a good starting point could be \\( \textcolor{blue}{\delta} = 0.5 \\). If training seems too conservative or too many sequences are masked, you can increase the value. For reference, \\( \textcolor{blue}{\delta} = 1.0 \\) corresponds to an average log-ratio divergence of 1 nat per token, i.e. on sequences where this threshold is exceeded, the old policy was on average \\( e^1 \approx 2.7 \\) times more likely to generate these tokens than the current policy. ### GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization **๐Ÿ“œ Paper**: https://huggingface.co/papers/2601.05242 GDPO is a reinforcement learning optimization method designed for multi-reward training. While existing approaches commonly apply Group Relative Policy Optimization (GRPO) in multi-reward settings, the authors show that this leads to reward advantages collapse, reducing training signal resolution and causing unstable or failed convergence. GDPO resolves this issue by decoupling reward normalization across individual rewards, preserving their relative differences and enabling more faithful preference optimization. To enable GDPO for multi-reward RL training, simply set: For a group of \\( N \\) rewards and \\( G \\) samples per group, GDPO normalizes each reward independently: $$ A_n^{(i,j)} = \frac{r_n^{(i,j)} - \text{mean}\{r_n^{(i,1)}, \ldots, r_n^{(i,G)}\}}{\text{std}\{r_n^{(i,1)}, \ldots, r_n^{(i,G)}\} + \epsilon} $$ The normalized group advantage is then aggregated across rewards: $$ A^{(i,j)} = \sum_{n=1}^{N} w_n A_n^{(i,j)} $$ The final per-batch normalization produces: $$ \hat{A}^{(i,j)} = \frac{A^{(i,j)} - \text{mean}_{i',j'}\{A^{(i',j')}\}}{\text{std}_{i',j'}\{A^{(i',j')}\} + \epsilon} $$ Here, \\( \text{mean}_{i',j'}\{A^{(i',j')}\} \\) and \\( \text{std}_{i',j'}\{A^{(i',j')}\} \\) denote statistics over all groups in the batch. ```python from trl import GRPOConfig training_args = GRPOConfig( ..., multi_objective_aggregation="normalize_then_sum", ) ``` Note that this method only has an effect when training involve more than one reward function. The authors provide a easy-to-use, slurm-free training example that enable the community to quickly validate GDPOโ€™s effectiveness over GRPO, see [Experiment-"Aha" moment](https://github.com/NVlabs/GDPO/tree/main/trl-GDPO). ### Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR **๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.05261 Length-Unbiased Sequence Policy Optimization (LUSPO) modifies GSPO by scaling each sequence's loss by its length. This corrects GSPO's gradient bias that penalizes longer responses. To reproduce the paper's setting, use this configuration: ```python from trl import GRPOConfig training_args = GRPOConfig( loss_type="luspo", importance_sampling_level="sequence", epsilon=2e-3, # section 5.1 of the paper epsilon_high=2.5e-3, # section 5.1 of the paper ) ``` ### VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Training **๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.10693 VESPO addresses training instability in off-policy RL caused by policy staleness, asynchronous updates, and train-inference mismatches. Rather than relying on heuristic token-level clipping (GRPO) or sequence-length normalization (GSPO), VESPO derives a principled reshaping kernel from a variational framework. In practice, this yields a smooth, asymmetric Gamma weighting function that gracefully suppresses extreme sequence-level importance weights without introducing length bias. $$ \mathcal{L}_{\text{VESPO}}(\theta) = - \mathbb{E}_{\tau \sim \mu} \left[ \underbrace{W(\tau)^{k} \cdot \exp\left(\lambda (1 - W(\tau))\right)}_{\phi(W) \text{ detached }} \cdot \mathcal{A}(\tau) \cdot \log \pi_\theta(\tau) \right] $$ with \\( W(\tau) = \frac{\pi_\theta(\tau)}{\mu(\tau)} \\) the sequence level importance ratio, and \\( \phi(W) \\) is detached from the computation graph to serve as a gradient scaling coefficient. ```python from trl import GRPOConfig training_args = GRPOConfig( loss_type="vespo", use_vllm=True, # or False if not using any token-level `vllm_importance_sampling_correction` methods vllm_importance_sampling_mode="token_truncate", # default correction mode for VESPO, `token_mask` also supported vespo_k_pos=2.0, # power exponent (c1 in paper Section 3.4) for positive advantages vespo_lambda_pos=3.0, # decay factor (c2 in paper Section 3.4) for positive advantages vespo_k_neg=3.0, # power exponent (c1 in paper Section 3.4) for negative advantages vespo_lambda_neg=2.0, # decay factor (c2 in paper Section 3.4) for negative advantages ) ``` ### Rethinking the Trust Region in LLM Reinforcement Learning **๐Ÿ“œ Paper**: https://huggingface.co/papers/2602.04879 DPPO replaces PPO/GRPO's heuristic ratio-clipping with a principled trust region based on direct policy divergence estimates. PPO-style clipping masks tokens based on the probability ratio ฯ€/ฮผ, which over-penalizes low-probability tokens and under-penalizes high-probability ones. DPPO instead masks based on direct approximations of policy divergence (TV or KL), ensuring updates stay within a theoretically grounded trust region. Four divergence approximations are supported: `binary_tv`, `binary_kl`, `topk_tv`, and `topk_kl`. ```python from trl.experimental.dppo import DPPOConfig, DPPOTrainer training_args = DPPOConfig( divergence_type="binary_tv", # divergence approximation divergence_topk=20, # K for top-K divergence modes (Section 7 / Appendix G.2 of the paper) epsilon=0.15, # ฮด_low threshold (Appendix F of the paper) epsilon_high=0.15, # ฮด_high threshold (Appendix F of the paper) clip_ratio_c=20.0, # IS ratio upper bound C (Section 5.4 of the paper) beta=0.0, # KL regularization coefficient use_vllm=True, ) trainer = DPPOTrainer( model="your-model", reward_funcs=[...], args=training_args, train_dataset=dataset, ) trainer.train() ``` The official code [sail-sg/Stable-RL](https://github.com/sail-sg/Stable-RL) ## Direct Policy Optimization Papers relating to the [`DPOTrainer`] ### Direct Preference Optimization: Your Language Model is Secretly a Reward Model **๐Ÿ“œ Paper**: https://huggingface.co/papers/2305.18290 Direct Preference Optimization (DPO) fine-tunes language models more efficiently and with better performance compared to reinforcement learning from human feedback (RLHF), by directly optimizing policy training based on human preferences. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="sigmoid", # losses in Appendix B of the paper per_device_train_batch_size=64, # batch size in Appendix B of the paper learning_rate=1e-6, # learning rate in Appendix B of the paper beta=0.1, # ฮฒ in Appendix B of the paper ) ``` ### SLiC-HF: Sequence Likelihood Calibration with Human Feedback **๐Ÿ“œ Paper**: https://huggingface.co/papers/2305.10425 Sequence Likelihood Calibration (SLiC) is shown to be an effective and simpler alternative to Reinforcement Learning from Human Feedback (RLHF) for learning from human preferences in language models. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="hinge", # Section 2 of the paper per_device_train_batch_size=512, # batch size in Section 3.2 of the paper learning_rate=1e-4, # learning rate in Section 3.2 of the paper ) ``` These parameters only appear in the [published version](https://openreview.net/pdf?id=0qSOodKmJaN) ### Statistical Rejection Sampling Improves Preference Optimization **๐Ÿ“œ Paper**: https://huggingface.co/papers/2309.06657 Proposes **RSO**, selecting stronger preference pairs via statistical rejection sampling to boost offline preference optimization; complements DPO/SLiC. They also introduce a new loss defined as: $$ \mathcal{L}_{\text{hinge-norm}}(\pi_\theta) = \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \max\left(0,\; 1 - \left[\gamma \log \frac{\pi_\theta(y_w \mid x)}{\pi_\text{ref}(y_w \mid x)} - \gamma \log \frac{\pi_\theta(y_l \mid x)}{\pi_\text{ref}(y_l \mid x)}\right]\right) \right] $$ To train with RSO-filtered data and the hinge-norm loss, you can use the following code: ```python from trl import DPOConfig, DPOTrainer dataset = ... def rso_accept(example): # replace with your actual filter/score logic return example["rso_keep"] train_dataset = train_dataset.filter(rso_accept) training_args = DPOConfig( loss_type="hinge", beta=0.05, # correspond to ฮณ in the paper ) trainer = DPOTrainer( ..., args=training_args, train_dataset=train_dataset, ) trainer.train() ``` ### Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints **๐Ÿ“œ Paper**: https://huggingface.co/papers/2309.16240 Proposes \(( f \\)-DPO, extending DPO by replacing the usual reverse-KL regularizer with a general \(( f \\)-divergence, letting you trade off mode-seeking vs mass-covering behavior (e.g. forward KL, JS, \(( \alpha \\)-divergences). The only change is replacing the DPO log-ratio margin with an **fโ€ฒ score**: $$ \mathcal{L}_{f\text{-DPO}}(\pi_\theta) = \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ -\log \sigma\left( \beta \textcolor{red}{f'}\textcolor{red}{\Big(}\frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)}\textcolor{red}{\Big)} - \beta \textcolor{red}{f'}\textcolor{red}{\Big(}\frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\textcolor{red}{\Big)} \right) \right] $$ Where \\( f' \\) is the derivative of the convex function defining the chosen \(( f \\)-divergence. To reproduce: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="sigmoid", beta=0.1, f_divergence_type="js_divergence", # or "reverse_kl" (default), "forward_kl", "js_divergence", "alpha_divergence" f_alpha_divergence_coef=0.5, # only used if f_divergence_type="alpha_divergence" ) ``` ### A General Theoretical Paradigm to Understand Learning from Human Preferences **๐Ÿ“œ Paper**: https://huggingface.co/papers/2310.12036 Learning from human preferences can be written as a single KL-regularized objective over pairwise preference probabilities, $$ \max_\pi ;\mathbb{E}\big[\Psi\left(p^*(y \succ y' \mid x)\right)\big] - \tau\mathrm{KL}(\pi||\pi_{\text{ref}}), $$ which reveals RLHF and DPO as special cases corresponding to the logit choice of \\( \Psi \\). The paper shows that this logit transform amplifies near-deterministic preferences and effectively weakens KL regularization, explaining overfitting. Using the **Identity transform (IPO)** avoids this pathology by optimizing preferences directly, without assuming a Bradleyโ€“Terry reward model. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="ipo", # Section 5.1 of the paper per_device_train_batch_size=90, # mini-batch size in Section C.1 of the paper learning_rate=1e-2, # learning rate in Section C.1 of the paper ) ``` These parameters only appear in the [published version](https://proceedings.mlr.press/v238/gheshlaghi-azar24a/gheshlaghi-azar24a.pdf) ### Towards Efficient and Exact Optimization of Language Model Alignment **๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.00856 The paper shows that direct preference methods like DPO optimize the wrong KL direction, leading to blurred preference capture, and proposes EXO as an efficient way to exactly optimize the humanโ€‘preference alignment objective by leveraging reverse KL probability matching rather than forward KL approximations. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="exo_pair", # Section 3.2 of the paper # From Section B of the paper per_device_train_batch_size=64, learning_rate=1e-6, beta=0.1, ) ``` ### Noise Contrastive Alignment of Language Models with Explicit Rewards **๐Ÿ“œ Paper**: https://huggingface.co/papers/2402.05369 The paper reframes language-model alignment as a *noise-contrastive classification* problem, proposing InfoNCA to learn a policy from explicit rewards (or preferences) by matching a reward-induced target distribution over responses, and showing DPO is a special binary case. It then introduces NCA, which adds an absolute likelihood term to prevent the likelihood collapse seen in purely relative (contrastive) objectives. With pairwise preferences, treat the chosen/rejected \\( K=2 \\), define scores \\( r=\beta(\log\pi_\theta-\log\pi_{\text{ref}}) \\), and apply the NCA preference loss \\( -\log\sigma(r_w)-\tfrac12\log\sigma(-r_w)-\tfrac12\log\sigma(-r_l) \\). To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="nca_pair", # From Section C of the paper per_device_train_batch_size=32, learning_rate=5e-6, beta=0.01, ) ``` ### Provably Robust DPO: Aligning Language Models with Noisy Feedback **๐Ÿ“œ Paper**: https://huggingface.co/papers/2403.00409 DPO breaks under noisy human preferences because label flips bias the objective. Robust DPO fixes this by analytically debiasing the DPO loss under a simple noise model, with provable guarantees. $$ \mathcal{L}_{\text{robust}}(\pi_\theta) = \frac{(1-\varepsilon)\mathcal{L}_{\text{DPO}}(y_w, y_l) - \varepsilon\mathcal{L}_{\text{DPO}}(y_l, y_w)} {1-2\varepsilon} $$ Where \\( \mathcal{L}_{\text{DPO}} \\) is the DPO loss defined in [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](#direct-preference-optimization-your-language-model-is-secretly-a-reward-model) and \\( \varepsilon \\) is the probability of a label flip. This single correction turns noisy preference data into an unbiased estimator of the clean DPO objective. ```python from trl import DPOConfig training_args = DPOConfig( loss_type="robust", per_device_train_batch_size=16, # batch size in Section B of the paper learning_rate=1e-3, # learning rate in Section B of the paper beta=0.1, # ฮฒ in Section B of the paper, max_length=512, # max length in Section B of the paper label_smoothing=0.1 # label smoothing $\varepsilon$ in Section 6 of the paper ) ``` ### Binary Classifier Optimization for Large Language Model Alignment **๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.04656 Theoretical analysis and a new algorithm, Binary Classifier Optimization, explain and enhance the alignment of large language models using binary feedback signals. To reproduce the paper's setting, use this configuration: BCO reframes language-model alignment as behavioral cloning from an optimal reward-weighted distribution, yielding simple supervised objectives that avoid RL while remaining theoretically grounded. It supports both unpaired reward data and pairwise preference data, with a reward-shiftโ€“invariant formulation that reduces to a DPO-style loss in the preference setting. For the pairwise preference setting, the BCO loss is defined as: $$ \mathcal{L}_{\text{bco\_pair}}(\pi_\theta) = \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ -\log \sigma\Big( \beta[(\log\pi_\theta-\log\pi_{\text{ref}})(y_w) - (\log\pi_\theta-\log\pi_{\text{ref}})(y_l)] \Big) \right] $$ To reproduce the paper in this setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="bco_pair", # From Section C of the paper per_device_train_batch_size=128, learning_rate=5e-7, beta=0.01, ) ``` For the unpaired version, the user should utilize [`experimental.bco.BCOConfig`] and [`experimental.bco.BCOTrainer`]. ### Learn Your Reference Model for Real Good Alignment **๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.09656 Trust Region DPO (TR-DPO) updates the reference policy during training, demonstrating effectiveness against DPO on the Anthropic HH and TLDR datasets, outperforming DPO by up to 19% measured by automatic evaluation with GPT-4, improving coherence, correctness, level of detail, helpfulness, and harmlessness. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( sync_ref_model=True, # enable TR-DPO (Section 3 of the paper) ref_model_mixup_alpha=0.6, # ฮฑ soft update weight (Table 1 of the paper) ref_model_sync_steps=512, # ฯ„ update frequency in steps (Table 1 of the paper) beta=0.05, # ฮฒ temperature (Table 1 of the paper) learning_rate=1e-6, # learning rate (Table 2 of the paper) num_train_epochs=1, # Table 2 of the paper max_length=1024, # max tokens length (Table 2 of the paper) max_grad_norm=2, # max gradient norm (Table 2 of the paper) warmup_steps=100, # warm-up steps (Table 2 of the paper) ) ``` ### Iterative Reasoning Preference Optimization **๐Ÿ“œ Paper**: https://huggingface.co/papers/2404.19733 Iterative RPO improves reasoning by repeatedly generating chain-of-thought candidates, building preference pairs from correct vs. incorrect answers, and training with a DPO + NLL objective. The extra NLL term is key for learning to actually generate winning traces. TRL can express the DPO + NLL objective by mixing `"sigmoid"` (DPO) with `"sft"` (NLL): ```python from trl import DPOConfig, DPOTrainer training_args = DPOConfig( loss_type=["sigmoid", "sft"], loss_weights=[1.0, 1.0], # alpha in the paper, recommended value is 1.0 ) trainer = DPOTrainer( ..., args=training_args, ) ``` Note that the paper uses an iterative loop: each iteration regenerates CoT candidates with the current model, then retrains on fresh preference pairs. TRL does not automate that loop for you. ### Self-Play Preference Optimization for Language Model Alignment **๐Ÿ“œ Paper**: https://huggingface.co/papers/2405.00675 A self-play method called SPPO for language model alignment achieves state-of-the-art performance by approximating Nash equilibrium policy in a constant-sum game setting, outperforming other approaches with limited data. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="sppo_hard", # From Section 5 of the paper beta=0.001, # ฮฒ = ฮท^โˆ’1 per_device_train_batch_size=64, learning_rate=5e-7, ) ``` ### Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer **๐Ÿ“œ Paper**: https://huggingface.co/papers/2405.16436 Regularized Preference Optimization (RPO) mitigates overoptimization in RLHF by fusing the DPO loss with the SFT loss, provably preventing the policy from choosing actions with spurious high proxy rewards. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type=["sigmoid", "sft"], # RPO loss = DPO + SFT (Section 5 of the paper) loss_weights=[1.0, 0.005], # ฮท=0.005 SFT weight in Appendix E.1 of the paper beta=0.01, # ฮฒ in Appendix E.1 of the paper learning_rate=5e-7, # learning rate in Appendix E.1 of the paper num_train_epochs=1, # Appendix E.1 of the paper ) ``` ### Distributional Preference Alignment of LLMs via Optimal Transport **๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.05882 Alignment via Optimal Transport (AOT) aligns large language models distributionally by penalizing violations of stochastic dominance between positive and negative sample distributions, achieving state-of-the-art performance on alignment benchmarks. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="aot", beta=0.01, # from the caption of Figure 2 ) ``` or, for the unpaired version: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="aot_unpaired", beta=0.01, # from the caption of Figure 2 ) ``` There is no additional hyperparameter in the paper. ### Discovering Preference Optimization Algorithms with and for Large Language Models **๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.08414 An LLM-driven method automatically discovers performant preference optimization algorithms, leading to a new algorithm called DiscoPOP that blends logistic and exponential losses. To reproduce the paper's setting, use this configuration: ```python from trl import DPOConfig training_args = DPOConfig( loss_type="discopop", per_device_train_batch_size=64, # batch size in Section B.1 of the paper learning_rate=5e-7, # learning rate in Section B.1 of the paper beta=0.05, # ฮฒ in Section B.1 of the paper, discopop_tau=0.05 # ฯ„ in Section E of the paper ) ``` ### WPO: Enhancing RLHF with Weighted Preference Optimization **๐Ÿ“œ Paper**: https://huggingface.co/papers/2406.11827 WPO reweights preference pairs by their policy probabilities to reduce the off-policy gap in DPO-style training. The loss is: $$ \mathcal{L}_{\text{WPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \textcolor{red}{w(x, y_w) w(x, y_l)} \log p(y_w \succ y_l \mid x) \right] $$ where the weight \\( w(x, y) \\) is defined as: $$ w(x, y) = \exp\left(\frac{1}{|y|}\sum_{t=1}^{|y|} \log \frac{\pi_\theta(y_t \mid x, y_{ 0 (optimism coefficient) and ฮฒ > 0 (KL regularization) in Algorithm 1 but does not specify numerical values. The following configuration uses TRL defaults: ```python from trl.experimental.xpo import XPOConfig training_args = XPOConfig( alpha=1e-5, # ฮฑ exploration bonus weight, ฮฑ โ‰ฅ 0 where ฮฑ=0 reduces to online DPO (TRL default) beta=0.1, # ฮฒ KL regularization coefficient (TRL default) ) ``` ## Distillation Papers relating to training a student model with the help of a teacher model. ### On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes **๐Ÿ“œ Paper**: https://huggingface.co/papers/2306.13649 Introduces Generalized Knowledge Distillation (GKD), which addresses distribution mismatch in KD for auto-regressive models by training the student on its own generated outputs with teacher feedback, instead of a fixed set of sequences. GKD supports flexible loss functions (e.g. beyond KL when the student cannot match the teacher) and integrates with RL fine-tuning (RLHF). The paper reports results on summarization, translation, arithmetic reasoning, and instruction-tuning. Used in TRL via [`experimental.distillation.DistillationTrainer`] and [`experimental.gkd.GKDTrainer`]. To reproduce the paper's setting, use this configuration: ```python from trl.experimental.distillation import DistillationConfig # XSum summarization task (Table A.1 of the paper) training_args = DistillationConfig( lmbda=0.5, # ฮป student data fraction (Section 3 of the paper) beta=0.5, # ฮฒ Generalized JSD interpolation, 0=KL, 1=reverse KL (Section 3 of the paper) temperature=1.0, # student training temperature (Appendix A of the paper) max_steps=40000, # training steps (Table A.1 of the paper) learning_rate=3e-4, # learning rate (Table A.1 of the paper) per_device_train_batch_size=32, # batch size (Table A.1 of the paper) warmup_steps=2000, # warm-up steps (Table A.1 of the paper) max_completion_length=64, # max output tokens (Table A.1 of the paper) ) ``` ### On-Policy Distillation **๐Ÿ“ฐ Blog**: https://thinkingmachines.ai/blog/on-policy-distillation/ On-Policy Distillation involves a student model generating rollouts for each batch of training data. We subsequently obtain the probability distributions for each token of the rollouts from both the student and teacher models. The student model is then optimized to minimize the negative Kullback-Leibler (KL) divergence between its own token distributions and those of the teacher model. | Method | Sampling | Reward signal | |-------------------------|------------|---------------| | Supervised finetuning | off-policy | dense | | Reinforcement learning | on-policy | sparse | | On-policy distillation | on-policy | dense | On-Policy Distillation has been shown to outperform SFT, GRPO and can be used to restore generalization capabilities lost during SFT. Additionally on-policy distillation is more compute efficient and is less prone to overfitting when trained with limited data. To train a model with on-policy distillation using TRL, you can use the following configuration, with the [`experimental.distillation.DistillationTrainer`] and [`experimental.distillation.DistillationConfig`]: ```python from trl.experimental.distillation import DistillationConfig training_args = DistillationConfig( lmbda=1.0, # student produces rollouts for all batches beta=1.0, # to ensure reverse-kl as the loss function teacher_model_name_or_path="teacher-model", # specify the teacher model ) ``` Alternatively, you can use the [`experimental.gkd.GKDTrainer`] and [`experimental.gkd.GKDConfig`]: ```python from trl.experimental.gkd import GKDConfig training_args = GKDConfig( lmbda=1.0, # student produces rollouts for all batches beta=1.0, # to ensure reverse-kl as the loss function teacher_model_name_or_path="teacher-model", # specify the teacher model ) ``` You can also use the [`GOLDTrainer`] and [`GOLDConfig`] to perform on-policy distillation with a similar configuration: ```python from trl.experimental import GOLDConfig config = GOLDConfig( lmbda=1.0, # student produces rollouts for all batches beta=1.0, # to ensure reverse-kl as the loss function teacher_model_name_or_path="teacher-model", # specify the teacher model ) ``` ### Knowledge Distillation of Large Language Models **๐Ÿ“œ Paper**: https://huggingface.co/papers/2306.08543 MiniLLM is the first on-policy knowledge distillation method, which minimizes the sequence-level reverse KLD between the teacher and the student model and is optimized by reinforcement learning. It is a generalized version of [Think Machine Lab's On-Policy Distillation](https://thinkingmachines.ai/blog/on-policy-distillation/), with the option to add distribution-level single-step distillation signals (like GKD when `beta=1`) and long-context reverse KLD signals. Alternatively, you can use the [`experimental.MiniLLMTrainer`] and [`experimental.MiniLLMConfig`] to perform MiniLLM distillation as follows: ```python from datasets import load_dataset from trl.experimental.minillm import MiniLLMTrainer dataset = load_dataset("trl-lib/tldr", split="train") trainer = MiniLLMTrainer( model="Qwen/Qwen3-0.6B", teacher_model="Qwen/Qwen3-1.7B", train_dataset=dataset, ) trainer.train() ``` For more details, see the [MiniLLM Trainer documentation](minillm) documentation. ### Reinforcement Learning via Self-Distillation **๐Ÿ“œ Paper**: https://huggingface.co/papers/2601.20802 Self-Distillation Policy Optimization (SDPO) enhances reinforcement learning with verifiable rewards by converting rich textual feedback (e.g., runtime errors, judge evaluations) into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. ```python from trl.experimental.sdpo import SDPOConfig, SDPOTrainer training_args = SDPOConfig( distillation_alpha=0.5, # Jensen-Shannon divergence (recommended) distillation_topk=100, # Top-K logit distillation approximation full_logit_distillation=True, # Required for top-K logit-level SDPO distillation_is_clip=2.0, # Importance sampling clipping distillation_weight=1.0, # Weight for self-distillation loss sdpo_policy_loss_mode="distillation_only", use_successful_as_teacher=True, # Use successful rollouts as teacher teacher_regularization="ema", # Supported: "ema", "none" teacher_update_rate=0.05, # EMA update rate include_environment_feedback=False, # Use dataset privileged_context when available ) trainer = SDPOTrainer( model="Qwen/Qwen2.5-1.5B-Instruct", reward_funcs=..., args=training_args, train_dataset=..., ) trainer.train() ``` Expected dataset columns: - `prompt` - `privileged_context` for optional environment feedback For more details, see the [SDPO Trainer documentation](sdpo_trainer). ### Self-Training with On-Policy Self-Distillation for Language Model Alignment **๐Ÿ“œ Paper**: https://huggingface.co/papers/2601.19897 Self-Distilled Fine-Tuning (SDFT) performs on-policy self-distillation by generating completions during training, then distilling an explicit teacher-conditioned view of those same completions back into the student. In TRL, SDFT uses a shared self-distillation core with SDPO where the teacher is the model itself (base weights with adapter disabled for PEFT, or the same model under `no_grad` for non-PEFT). The teacher prompt is composed internally from the student `prompt` plus the dataset `privileged_context`. ```python from datasets import Dataset from trl.experimental.sdft import SDFTConfig, SDFTTrainer dataset = Dataset.from_dict( { "prompt": [[{"role": "user", "content": "Solve 2+2."}]], "privileged_context": ["Example answer: 4."], } ) training_args = SDFTConfig( distillation_alpha=0.5, distillation_topk=5, max_completion_length=64, ) trainer = SDFTTrainer( model="Qwen/Qwen2.5-1.5B-Instruct", args=training_args, train_dataset=dataset, ) trainer.train() ``` Expected dataset columns: - `prompt` - `privileged_context` containing only the extra teacher-only information For more details, see the [SDFT Trainer documentation](sdft_trainer). ### Embarrassingly Simple Self-Distillation Improves Code Generation **๐Ÿ“œ Paper**: https://huggingface.co/papers/2604.01193 Simple Self-Distillation (SSD) improves code generation by sampling completions from the model at a training-time temperature and truncation configuration, then fine-tuning on those raw, unverified samples with standard cross-entropy loss. No reward model, verifier, teacher model, or reinforcement learning is needed. SSD reshapes token distributions in a context-dependent way: suppressing distractor tails at "lock" positions (where syntax leaves little ambiguity) while preserving diversity at "fork" positions (where multiple valid continuations exist). ```python from trl.experimental.ssd import SSDConfig, SSDTrainer training_args = SSDConfig( temperature=0.6, # Training-time sampling temperature (T_train) top_k=20, # Training-time top-k truncation top_p=0.95, # Training-time top-p truncation max_completion_length=65536, learning_rate=5e-6, ) trainer = SSDTrainer( model="Qwen/Qwen3-4B-Instruct", args=training_args, train_dataset=..., ) trainer.train() ``` Expected dataset columns: - `prompt` For more details, see the [SSD Trainer documentation](ssd_trainer). ## Distributed Training ### ZeRO: Memory Optimizations Toward Training Trillion Parameter Models **๐Ÿ“œ Paper**: https://huggingface.co/papers/1910.02054 ZeRO (Zero Redundancy Optimizer) eliminates memory redundancies in data- and model-parallel training by partitioning optimizer states, gradients, and parameters across devices while retaining low communication volume and high computational granularity. This allows for the efficient training of large models that would otherwise not fit in GPU memory. TRL supports ZeRO via the [DeepSpeed integration](deepspeed_integration). To use it, provide a DeepSpeed configuration file with your desired settings, ```yaml # config.yaml distributed_type: DEEPSPEED num_processes: 2 deepspeed_config: zero_stage: 3 ``` and launch the training script using `accelerate launch --config_file config_file`. ```sh accelerate launch --config_file config.yaml train.py ``` ## Proximal Policy Optimization Papers relating to the [`experimental.ppo.PPOTrainer`] ### Proximal Policy Optimization Algorithms **๐Ÿ“œ Paper**: https://huggingface.co/papers/1707.06347 Introduces Proximal Policy Optimization (PPO): policy gradient methods that alternate between collecting rollouts and optimizing a clipped surrogate objective over multiple minibatch epochs. PPO retains benefits of trust-region methods (e.g. TRPO) with simpler implementation and strong empirical sample efficiency, and was validated on robotics and Atari benchmarks. Used in TRL via [`experimental.ppo.PPOTrainer`]. To use PPO with TRL, use this configuration: ```python from trl.experimental.ppo import PPOConfig training_args = PPOConfig( cliprange=0.2, # ฮต clipping range (Section 3 and Table 3 of the paper, Mujoco setting) num_ppo_epochs=4, # K epochs of minibatch updates (TRL default; paper uses K=10 Mujoco, K=3 Atari) gamma=1.0, # ฮณ discount factor (TRL default for LLM tasks; paper uses ฮณ=0.99) lam=0.95, # ฮป GAE parameter (Table 3 of the paper, Mujoco setting) kl_coef=0.05, # KL penalty coefficient (Section 4 of the paper discusses adaptive KL) vf_coef=0.1, # cโ‚ value function loss weight (Equation 9 of the paper) ) ```