docs/source/gspo_token.md

GSPO-token

In the paper Group Sequence Policy Optimization, the authors propose a token-level objective variant to GSPO, called GSPO-token. To use GSPO-token, you can use the GRPOTrainer class in trl.experimental.gspo_token.

Usage

from trl.experimental.gspo_token import GRPOTrainer
from trl import GRPOConfig

training_args = GRPOConfig(
    importance_sampling_level="sequence_token",
    ...
)

To leverage GSPO-token, the user will need to provide the per-token advantage $\widehat{A_{i,t}}\backslash hat\{A\_\{i,t\}\}$ for each token $tt$ in the sequence $ii$ (i.e., make $\widehat{A_{i,t}}\backslash hat\{A\_\{i,t\}\}$ varies with $tt$—which isn't the case here, $\widehat{A_{i,t}}=\widehat{A_i}\backslash hat\{A\_\{i,t\}\}=\backslash hat\{A\_\{i\}\}$). Otherwise, GSPO-Token gradient is just equivalent to the original GSPO implementation.

GRPOTrainer

[[autodoc]] experimental.gspo_token.GRPOTrainer - train - save_model - push_to_hub