| \documentclass[11pt]{article} |
| \usepackage{common} |
| \begin{document} |
| \notehead{0107}{CISPO: clip the importance weight, not the update}{Policy-optimisation lineage, batch M1. Companion notebook: \texttt{notebooks/0107-cispo.ipynb}.} |
|
|
| \begin{problem} |
| \question{Can an off-policy update keep the rare high-ratio tokens that clipping silences, while still bounding the weights?} |
| Off-policy updates on one batch (0103) need the ratio $r_{i,t}=\pi_\theta/\pi_{\rm old}$. PPO and GRPO handle large ratios by \emph{removing the token from the gradient} once it has moved past the clip range in the favourable direction (0104). MiniMax-M1 (arXiv:2506.13585, Sec.~3.1) found that this silences exactly the tokens that matter most for long-CoT learning: rare ``fork'' tokens (\emph{However, Recheck, Wait, Aha}) have low probability under the base model, hence a large ratio after the first update, and are ``clipped out after the first on-policy update, preventing them from contributing to subsequent off-policy gradient updates'' (they run 16 off-policy rounds per generation batch). DAPO's larger $\epsilon_{\rm hi}$ (0106) was ``less effective'' in their setup. CISPO (Clipped IS-weight Policy Optimization) is their fix; it is the RL objective Mistral names for Leanstral~1.5. The announcement gives no equations; the Leanstral technical report (Sec.~2.3) states the objective but not the truncation value $c$. Sections 1--6 below follow the MiniMax paper; the report's variant is derived in the subsection \emph{Leanstral variant}. |
| \end{problem} |
|
|
| \section{Notation} |
| \begin{center}\small\begin{tabular}{@{}ll@{}}\toprule |
| symbol & meaning\\\midrule |
| $r_{i,t}=\pi_\theta(o_{i,t}|s_{i,t})/\pi_{\rm old}(o_{i,t}|s_{i,t})$ & token importance weight\\ |
| $\hat A_i$ & group advantage (0105), shared by all tokens of $o_i$\\ |
| $\sg(\cdot)$ & stop-gradient: value kept, derivative set to $0$\\ |
| $\hat r_{i,t}=\clip(r_{i,t},1-\epsilon^{IS}_{\rm low},1+\epsilon^{IS}_{\rm high})$ & clipped IS weight (Eq.~5)\\ |
| $w(r,A)$ & gradient weight: loss gradient $=w\cdot A\cdot\nabla\log\pi_\theta(o_t|s_t)$\\ |
| $M_{i,t}\in\{0,1\}$ & token mask (Eqs.~6--7)\\ |
| $\epsilon_{\rm hi},\epsilon_{\rm lo}$ & PPO clip bounds (DAPO values $0.28,\,0.2$ used for the tables)\\\bottomrule\end{tabular}\end{center} |
|
|
| \section{Derivation} |
| \subsection{Start from the importance-weighted policy gradient} |
| 0103 gave the exact off-policy gradient with the sequence weight; its token-level form, with the weights treated as constants, is the ``vanilla REINFORCE objective with corrected distribution for offline updates'' (M1 Eq.~3; M1 writes the per-sample normaliser $1/|o_i|$ there and the token-level $1/\sum_i|o_i|$ only in Eq.~4, which we use throughout so that Eqs.~\eqref{eq:reinf} and \eqref{eq:cispo} differ by the clip alone): |
| \begin{equation}\mathcal J_{\rm REINFORCE}=\E\Big[\frac1{\sum_i|o_i|}\sum_{i,t}\sg\big(r_{i,t}\big)\,\hat A_{i}\,\log\pi_\theta(o_{i,t}\mid s_{i,t})\Big],\qquad\nabla\mathcal J=\E\Big[\tfrac1{\sum|o_i|}\textstyle\sum_{i,t}r_{i,t}\hat A_i\nabla\log\pi_\theta\Big].\label{eq:reinf}\end{equation} |
| Since $r\nabla\log\pi=\nabla r$, this is also $\nabla$ of the unclipped surrogate $\E[r\hat A]$ of 0104. The stop-gradient is what makes the \emph{value} of the loss a different function from its gradient: only $\nabla\log\pi$ is differentiated. |
| \ver{at $\theta_{\rm old}$ ($r\equiv1$) the gradient of Eq.~\eqref{eq:reinf} equals the exact $\nabla J$ to $4.2\times10^{-17}$.} |
|
|
| \subsection{PPO/GRPO is Eq.~(\ref{eq:reinf}) times a mask} |
| From 0104 the PPO gradient of one token is $A\,r\,\indb{\rm live}\,\nabla\log\pi$ with $\mathrm{live}=(A>0\wedge r<1+\epsilon_{\rm hi})\vee(A<0\wedge r>1-\epsilon_{\rm lo})$. That is Eq.~\eqref{eq:reinf} multiplied by the mask of M1 Eq.~7: $M=0$ if $A>0$ and $r>1+\epsilon_{\rm hi}$, $M=0$ if $A<0$ and $r<1-\epsilon_{\rm lo}$, else $M=1$ (M1 Eq.~6 is the unified form $\sg(\hat r)\hat A\log\pi\,M$). |
| \ver{unified form with $\hat r=r$ and this mask equals the autograd gradient of the PPO loss to $1.4\times10^{-17}$; the silenced set computed from the weights equals the mask-zero set exactly (symmetric difference $0$).} |
|
|
| \subsection{CISPO: keep every token, clip its weight} |
| Rather than masking, clip the weight and stop its gradient (M1 Eqs.~4--5): |
| \begin{equation}\mathcal J_{\rm CISPO}=\E\Big[\frac1{\sum_i|o_i|}\sum_{i,t}\sg\big(\hat r_{i,t}\big)\hat A_i\log\pi_\theta(o_{i,t}\mid s_{i,t})\Big],\quad\hat r=\clip\big(r,1-\epsilon^{IS}_{\rm low},1+\epsilon^{IS}_{\rm high}\big).\label{eq:cispo}\end{equation} |
| MiniMax ``did not impose a lower bound on the IS weight by setting $\epsilon^{IS}_{\rm low}$ to a large value; instead, we only tuned $\epsilon^{IS}_{\rm high}$'', so $\hat r=\min(r,1+\epsilon_{\rm hi})$ and the gradient of each token is |
| \begin{equation}\nabla\mathcal J_{\rm CISPO}\ \ni\ \min(r_{i,t},\,1+\epsilon_{\rm hi})\ \hat A_i\ \nabla\log\pi_\theta(o_{i,t}\mid s_{i,t}).\label{eq:cgrad}\end{equation} |
| Without clipping ($\epsilon_{\rm hi}=\infty$) it \emph{is} Eq.~\eqref{eq:reinf}, the standard policy gradient. |
| \ver{$\epsilon_{\rm hi}{=}\infty$ gives the token-IS gradient exactly ($0$); autograd of the CISPO, PPO and IS losses equals $\sum_tw(r,A)A\nabla\log\pi$ built from the weight table to $\le1.4\times10^{-17}$.} |
|
|
| \subsection{Gradient comparison table} |
| Weights $w(r,A)$ multiplying $A\nabla\log\pi$ (checked on the grid below; $\epsilon_{\rm hi}=0.28$, $\epsilon_{\rm lo}=0.2$): |
| \begin{center}\small\setlength{\tabcolsep}{4pt}\begin{tabular}{@{}lcccc@{}}\toprule |
| region & PPO/GRPO & CISPO & unclipped token IS & changed by CISPO?\\\midrule |
| $A>0,\ r<1+\epsilon_{\rm hi}$ & $r$ & $r$ & $r$ & no\\ |
| $A>0,\ r>1+\epsilon_{\rm hi}$ & \textbf{0} & $1+\epsilon_{\rm hi}$ & $r$ & \textbf{kept, capped}\\ |
| $A<0,\ r>1+\epsilon_{\rm hi}$ & $r$ (unbounded) & $1+\epsilon_{\rm hi}$ & $r$ & \textbf{bounded}\\ |
| $A<0,\ 1-\epsilon_{\rm lo}<r\le1+\epsilon_{\rm hi}$ & $r$ & $r$ & $r$ & no\\ |
| $A<0,\ r<1-\epsilon_{\rm lo}$ & \textbf{0} & $r$ (no lower clip) & $r$ & \textbf{kept}\\ |
| $A>0,\ r<1-\epsilon_{\rm lo}$ & $r$ & $r$ & $r$ & no\\\bottomrule\end{tabular}\end{center} |
| \textbf{Exactly which tokens PPO/GRPO zero out and CISPO keeps:} $\{A>0,\ r>1+\epsilon_{\rm hi}\}\cup\{A<0,\ r<1-\epsilon_{\rm lo}\}$, the first set being the favourable-direction rare tokens of the problem statement. CISPO additionally caps the weight on $\{A<0,\ r>1+\epsilon_{\rm hi}\}$, where PPO's weight $r$ is unbounded. Numbers on the toy ($\|\theta-\theta_{\rm old}\|=1.5$, ratios in $[0.455,1.992]$): the three sets hold $7$, $63$ and $83$ token slots with probability mass $0.013$, $0.055$, $0.016$ respectively; maximum weight over all tokens: PPO $1.992$, CISPO $1.280$. |
| \ver{the set of tokens with PPO weight $0$ and CISPO weight $>0$ equals the union of the first two cells exactly; CISPO's smallest weight equals the smallest ratio (no token is ever silenced); CISPO weights never exceed $1+\epsilon_{\rm hi}$.} |
|
|
| \paragraph{Observation (mine).} The weight $\min(r,C)$ in Eq.~\eqref{eq:cgrad} is the same function as the token-level \emph{truncated importance sampling} (TIS) weight used against the training--inference mismatch (arXiv:2510.26788 Eq.~6 gives its sequence-level form). The functional form is shared; the ratio is not: here $r=\pi_\theta/\pi_{\rm old}$ (policy lag), there $\rho=\pi_{\rm train}/\pi_{\rm sampler}$ (numerics, 0108). |
|
|
| \subsection{Stop-gradient semantics} |
| The loss in Eq.~\eqref{eq:cispo} has gradient $\hat r\hat A\nabla\log\pi$ only because $\hat r$ is detached. Two look-alikes differ: |
| \begin{itemize} |
| \item \emph{Forgotten $\sg$:} $\nabla[\min(r,c)\,A\log\pi]=A\big(\indb{r<c}\,r\log\pi+\min(r,c)\big)\nabla\log\pi$ (product rule; $\nabla r=r\nabla\log\pi$). On the unclipped set the weight is $r\,(1+\log\pi)$ instead of $r$: an extra term $Ar\log\pi$ proportional to the \emph{log-probability}, which is large and negative, and flips the sign of the update whenever $\pi<1/e$. Beyond the clip the clamp zeroes the extra term, so the error is confined to the live tokens. |
| \item \emph{Ratio-form clip} $\min(r,c)\,A$ (no $\log\pi$ factor): gradient $Ar\,\indb{r<c}\nabla\log\pi$, i.e.\ PPO-like zeroing beyond the clip, which is exactly what CISPO was designed to avoid. |
| \end{itemize} |
| \ver{at $\log\pi_{\rm old}=-2.3$: $r=1,A=+1$ gives CISPO $-1.000$, forgotten-$\sg$ $+1.300$ (sign flipped, $=-Ar(1+\log\pi)$), ratio-form $-1.000$; at $r=2.23$: CISPO $-1.28$, forgotten-$\sg$ $-1.28$ (identical), ratio-form $0$; all six test points match the closed forms to $\le10^{-12}$.} |
|
|
| \subsection{What CISPO costs: bias, and what it does not provide} |
| \paragraph{Bias.} Against the token-IS gradient, the bias is $\E_{\rm old}[\sum_t(\hat r_t-r_t)A_t\nabla\log\pi_t]$ with $\hat r-r=-(r-1-\epsilon_{\rm hi})_+$ (no lower clip), so |
| \begin{equation}\begin{aligned}\|{\rm bias}\|&\le\E_{\rm old}\Big[\sum_t(r_t-1-\epsilon_{\rm hi})_+\,|A_t|\,\|\nabla\log\pi_t\|\Big],\\\E_{a\sim\pi_{\rm old}}(r-1-\epsilon)_+&=\sum_a(\pi-(1{+}\epsilon)\pi_{\rm old})_+\le\TV(\pi,\pi_{\rm old}).\end{aligned}\label{eq:cbias}\end{equation} |
| The last inequality: $(\pi-(1+\epsilon)\pi_{\rm old})_+\le(\pi-\pi_{\rm old})_+$ and $\sum_a(\pi-\pi_{\rm old})_+=\TV$. So the bias is first order in the per-state TV distance (hence $\propto\sqrt{\mathrm{KL}}$) and vanishes with $\theta\to\theta_{\rm old}$ or $\epsilon_{\rm hi}\to\infty$; M1 acknowledges it is ``slightly biased''. PPO's gradient also deviates from the token-IS one (it deletes the silenced tokens' contributions). |
| \ver{$\|{\rm bias}\|=1.4\times10^{-2},\,9.2\times10^{-3},\,1.0\times10^{-3}$ for $\epsilon_{\rm hi}=0.05,\,0.10,\,0.28$, below the bounds $3.8\times10^{-2},\,2.4\times10^{-2},\,4.1\times10^{-3}$ ($\|g_{\rm IS}\|=0.175$); at $\epsilon_{\rm hi}=.28$: $\|g_{\rm PPO}-g_{\rm IS}\|=2.2\times10^{-2}$ vs.\ $\|g_{\rm CISPO}-g_{\rm IS}\|=1.0\times10^{-3}$; $\sum_a(\pi-1.28\pi_{\rm old})_+\le\TV$ at all 40 states.} |
| \paragraph{Variance.} All weights are bounded by $1+\epsilon_{\rm hi}$ (PPO's live $A<0$ tokens are not), which M1 credits for ``helping reduce variance and stabilizing RL training''; their evidence is empirical (Qwen2.5-32B zero-RL, AIME 2024: CISPO beats GRPO and DAPO at equal steps and matches DAPO with $50\%$ of the steps, Fig.~2). We do not re-run it. |
|
|
| \paragraph{No trust region.} Capping the \emph{gradient weight} does not cap the \emph{policy movement}. On a single decision with a rare rewarded token ($\pi_{\rm old}=0.01$; exact expectation, $K=16$ SGD steps on logits with a deliberately large step) PPO's direct weight on that token is $0$ in $15$ of the $16$ steps and its probability ends at $0.017$; CISPO keeps a positive weight at every step ($1.00$ at step 1 where $r=1$, then the cap $1.28$) and the probability reaches $0.862$. Learning from the rare token is exactly what CISPO is for, and the same toy shows the hazard: with no clip-induced brake, stability rests on the learning rate, $K$, and the bounded weights. |
| \ver{PPO silences the token for $15/16$ steps, CISPO for $0/16$; final $p_w$: PPO $0.017$, CISPO $0.862$.} |
|
|
| \subsection{Leanstral variant: sampler ratio, masked tool tokens, one outcome for every token} |
| \label{sec:lean} |
| The Leanstral technical report (Mistral, \texttt{LeanstralReport.pdf}, Sec.~2.3; vault note \texttt{in-this-work-we-present-leanstral-a-series-of-generalist-code-agent-models-for}) writes the objective for tool-using proof agents as |
| \begin{equation}\mathcal L=-\frac1{\sum_i|o_i|}\sum_{i,t}\sg\big(\bar\rho_{i,t}\big)\hat A_i\log\pi_\theta(a_{i,t}\mid s_{i,t},\mathcal T_i),\qquad\bar\rho_{i,t}=\min\Big(\frac{\pi_\theta(a_{i,t}\mid s_{i,t},\mathcal T_i)}{\pi_{\rm sample}(a_{i,t}\mid s_{i,t},\mathcal T_i)},\,c\Big),\label{eq:lean}\end{equation} |
| with $\mathcal T_i$ the interleaved tool calls and tool responses, $c$ a finite truncation threshold (value not reported) and normalised outcome advantages $\hat A_i$; the verifier outcome is propagated to all model-generated tokens, and prompts and tool responses are context, not targets. The stated purpose is to reduce the effect of trainer--generator mismatch on long trajectories. Relative to Eq.~\eqref{eq:cispo} three things change, and each has a precise consequence (the derivations below are mine, checked in notebook section 8 on a two-turn tabular agent enumerated exactly; they are not claims made by the report). |
|
|
| \paragraph{(1) The ratio is against the sampler.} $\bar\rho$ uses $\pi_{\rm sample}$, the distribution that actually produced the token (the inference engine, possibly serving a lagged copy of the weights), not the trainer's own $\pi_{\rm old}$. Then $r=\pi_\theta/\pi_{\rm sample}=(\pi_\theta/\pi_{\rm old})\cdot(\pi_{\rm old}/\pi_{\rm sample})$ contains the policy lag \emph{and} the numerical mismatch of 0108, and the min is the truncated-IS weight of 0108 applied to both at once; no separate $\pi_{\rm old}$ forward pass is needed. With the full sequence weight $\prod_t\rho_t$ the importance-sampled gradient is exact for any sampler with the right support; the per-token weight of Eq.~\eqref{eq:lean} drops the prefix ratios (as in 0103 and above), so it is biased by the mismatch and exact when sampler and policy coincide. The clip bias bound Eq.~\eqref{eq:cbias} carries over with $\pi_{\rm old}$ replaced by $\pi_{\rm sample}$: the proof used only that the expectation is over the base measure of the ratio, so $\sum_a(\pi-c\,\pi_{\rm sample})_+\le\TV(\pi,\pi_{\rm sample})$ and the bias is first order in the sampler--policy distance. A constant baseline still adds exactly zero even under mismatch, because $\E_{\rm sample}[\rho_t s_t\mid\text{context}]=\sum_a\pi(a)\,s(a)=0$ at every context. |
| \ver{toy ($4$ actions per turn, sampler logits perturbed by $\mathcal N(0,0.35^2)$): $\E_\pi[R(s_1{+}s_2)]$ equals the finite-difference $\nabla J$ and the full-sequence sampler-weighted gradient equals $\nabla J$, both to $2.7\times10^{-11}$ (finite-difference accuracy); the per-token sampler-ratio gradient has $\|{\rm bias}\|=8.1\times10^{-3}$ against $\|\nabla J\|=0.139$ and is exact (to $2.7\times10^{-11}$) when the sampler equals the policy; the baseline term is $\le2.8\times10^{-17}$; $\sum_a(\pi-c\pi_{\rm sample})_+\le\TV$ at all five states for $c=1.28$; clipping at $c=1.28$ changes the gradient by $2.0\times10^{-2}$, below the bound $3.2\times10^{-2}$.} |
|
|
| \paragraph{(2) Tool and prompt tokens are masked.} The episode probability factorises as $P_\theta(\tau)=D(x)\prod_t[\pi_\theta(y^t\mid c^t)\Psi(f^t\mid\cdot)]$ with tool responses $f^t$ drawn from the environment $\Psi$, which does not depend on $\theta$; so $\nabla\log P_\theta$ contains only policy-sampled tokens, and the loss mask is not a convenience (0204, its Eq.~for the trajectory score). Scoring the tool tokens adds a reward-weighted supervised term to imitate compiler output (0204: norm $0.68$ against $\|\nabla J\|=0.48$ in its toy). Masked tokens also leave the denominator $\sum_i|o_i|$, and the ratio and clip are applied only to the unmasked tokens. With $\mathcal T_i$ as context the ratios still condition on the tool outputs, which are the same in sampler and trainer because they are inputs, not samples. |
|
|
| \paragraph{(3) One outcome is broadcast to every token, so variance grows with length and with the number of turns.} (a) Score terms are martingale differences: $\E[s_u\mid\text{prefix}_u]=0$, so $\E[s_t\cdot s_u]=0$ for $t<u$ and $\E\|\sum_{t\le L}s_t\|^2=\sum_{t\le L}\E\|s_t\|^2$, growing roughly linearly with the number of policy tokens (toy chain, $L=1..5$: $0.719,1.412,2.055,2.645,3.272$, each equal to the sum of the per-token terms to $5\times10^{-15}$). (b) With one scalar advantage on a two-turn episode, telescoping $R-V=[Q(y^1)-V]+[R-Q(y^1)]$ (notation of 0204) shows the second bracket adds noise to the turn-$1$ coefficient: $\E[(R-V)^2\|s_1\|^2]=\E[(Q-V)^2\|s_1\|^2]+\E[\Var(R\mid y^1)\,\|s_1\|^2]$, because $\E[R-Q\mid y^1]=0$ kills the cross term. The extra term vanishes only if the later turns are deterministic given $y^1$. Both the broadcast and the per-turn estimator are unbiased (0204); only the variance differs. In the toy the turn-$1$ second moment is $0.164=0.038+0.126$ (credit plus noise from turn $2$), and the total gradient variance (trace) is $0.302$ for the broadcast against $0.147$ for the per-turn credit, a factor $2.05$ (a property of this random toy, not a general constant). So an agentic trace with many tool-interleaved turns, one verifier outcome per rollout and a modest group of rollouts is a high-variance gradient; the report's remedies are the pass-rate filter and group normalisation, which reduce but do not remove it. \ver{identities exact to $5.6\times10^{-17}$ (decomposition) and $5.3\times10^{-15}$ (orthogonality); both estimators equal $\nabla J$ to $2.7\times10^{-11}$.} |
|
|
| \subsection{Implementation} |
| \begin{verbatim} |
| r = exp(logp_new - logp_old) # ratio, per token |
| r_hat = clamp(r, max=1 + eps_high).detach() # sg(clip(r)); no lower bound |
| loss = -(r_hat * A[:, None] * logp_new * mask).sum() / mask.sum() # 1/sum|o_i| |
| \end{verbatim} |
| The loss \emph{value} is not an estimate of anything; monitor ratios, clip fraction and entropy instead. |
| \ver{naive per-token Python loop equals this vectorised form (gradient) to $0$.} |
|
|
| \section{Boxed result} |
| \begin{result} |
| \begin{align*} |
| &\nabla\mathcal J_{\rm CISPO}=\E\Big[\tfrac1{\sum_i|o_i|}\textstyle\sum_{i,t}\min(r_{i,t},1+\epsilon_{\rm hi})\,\hat A_i\,\nabla_\theta\log\pi_\theta(o_{i,t}|s_{i,t})\Big]\\ |
| &\qquad(\hat r\ \text{inside }\sg)\\ |
| &\text{PPO/GRPO weight }=r\cdot\indb{(A>0\wedge r<1+\epsilon_{\rm hi})\vee(A<0\wedge r>1-\epsilon_{\rm lo})}\\ |
| &\text{zeroed by PPO/GRPO, kept by CISPO: }\{A>0,r>1+\epsilon_{\rm hi}\}\cup\{A<0,r<1-\epsilon_{\rm lo}\}\\ |
| &\|\text{bias}\|\le\E\textstyle\sum_t(r_t-1-\epsilon_{\rm hi})_+|A_t|\,\|\nabla\log\pi_t\|\\ |
| &\phantom{\|\text{bias}\|}\le(\max|A|\max\|\nabla\log\pi\|)\textstyle\sum_t\E\,\TV_t |
| \end{align*} |
| \end{result} |
|
|
| \section{What breaks} |
| \begin{breaks}\small\begin{itemize} |
| \item \textbf{Biased gradient} (bounded by Eq.~\eqref{eq:cbias}); \emph{violated:} unbiasedness of IS. It grows with how far $\pi_\theta$ moves in $K$ updates. |
| \item \textbf{No trust region.} Weights are bounded but the policy is not; with large $K$ or learning rate nothing stops drift (toy: $0.01\to0.86$). \emph{Violated:} the proximity assumption behind 0104's bound is not enforced at all. |
| \item \textbf{Still a per-token ratio.} The other tokens' ratios $\prod_{s\ne t}r_s$ are dropped exactly as in 0103's token-level approximation; CISPO does not solve long-horizon IS variance, it avoids discarding tokens. |
| \item \textbf{Mismatch contaminates $r$.} If the sampler and trainer disagree (0108), $r\ne1$ even at $\theta=\theta_{\rm old}$; large spurious ratios are now \emph{kept} with weight up to $1+\epsilon_{\rm hi}$ rather than dropped. |
| \item \textbf{Wrong stop-gradient} silently produces a different algorithm (sign errors on live tokens), and the loss value cannot reveal it. |
| \item \textbf{$A<0$ tokens with $r>1+\epsilon_{\rm hi}$} still get weight $1+\epsilon_{\rm hi}$ (strong push-down); PPO gives them weight $r$. Neither limits how often a token that is \emph{both} high-ratio and negative-advantage is penalised. |
| \item \textbf{Leanstral variant.} The sampler ratio is per token and still biased under mismatch (toy: $\|\mathrm{bias}\|=8\times10^{-3}$); one outcome on every token of a multi-turn trace inflates variance relative to per-turn credit (toy: $2.05\times$); $c$ is not reported, so nothing here fixes the clip level. \emph{Violated:} exactness of IS and the absence of cross-turn noise. |
| \item \textbf{Hyper-parameters are paper-specific}: M1 tunes only $\epsilon^{IS}_{\rm high}$; their value is not reported in the passages used here, and they combine CISPO with dynamic sampling and a length penalty (0106) and no KL (0109). |
| \end{itemize}\end{breaks} |
|
|
| \nextnote{CISPO's $r$ assumes the sampler's distribution at $\theta_{\rm old}$ \emph{is} $\pi_{\theta_{\rm old}}$. In practice the inference engine and the training engine disagree numerically, and MiniMax needed an FP32 LM head before reward would grow at all. Note 0108 bounds that bias and the truncated-IS correction.} |
|
|
| \section*{References} |
| {\small\begin{itemize}[leftmargin=1.2em] |
| \item MiniMax, \emph{MiniMax-M1}, arXiv:2506.13585: Sec.~3.1 (Eq.~1 PPO, Eq.~2 group advantage, ``Issues of Token Clipping'', Eq.~3 REINFORCE with $\sg$, Eqs.~4--5 CISPO, Eqs.~6--7 unified mask, no KL, dynamic sampling and length penalty from DAPO), Fig.~2. |
| \item Yu et al., arXiv:2503.14476, Sec.~3.1 (Clip-Higher) for the comparison point. |
| \item Qi et al., arXiv:2510.26788, Sec.~2.1, Eqs.~5--7 (sequence-level IS, TIS, MIS) for the observation on the shared weight form. |
| \item Mistral AI, \emph{Leanstral 1.5} announcement (vault note \texttt{leanstral-15-proof-abundance-for-all}): names mid-training, SFT and RL with CISPO; no equations given. |
| \item Mistral AI, Leanstral technical report (\texttt{LeanstralReport.pdf} in the \texttt{LeanstralSafeVerify} repository; vault notes \texttt{in-this-work-we-present-leanstral-a-series-of-generalist-code-agent-models-for}, \texttt{source-analysis-leanstral-technical-report}), Sec.~2.3: the objective of Eq.~\eqref{eq:lean}, outcome propagated to all model tokens, prompts and tool responses as context; truncation value $c$ not reported. Not on arXiv. |
| \item Notes 0103, 0108 and 0204 of this series for the sequence weight, the sampler--trainer mismatch and the multi-turn masking and credit used in the Leanstral-variant subsection. |
| \item Weight table, bias bound, stop-gradient analysis and the fork-token demonstration are derived/checked here. |
| \end{itemize}} |
| \end{document} |
|
|
