v3/paper_source/03_problem_formulation.tex

This section formulates plug-and-play speedup as CMDP where a scheduler policy optimizes execution efficiency subject to world model-estimated state deviation constraints.

\subsection{Plug-and-Play Speedup via Scheduler Policy}

\begin{figure}[h]
    \centering
\vspace{-2mm}
    \includegraphics[width=0.7\linewidth]{figures/policy_hierarchy.pdf}
    % \includegraphics[width=0.6\linewidth]{./doc/benchmark-demo-v7.pdf}
    \caption{\textbf{Plug-and-Play Speedup via Scheduler Policy.} The scheduler policy $\pi$ predicts a downsampling rate $k$ to downsample the action chunk from the frozen policy into a shorter chunk for acceleration.}
    \label{fig:policy_hierarchy}
    \vspace{-3mm}
\end{figure}



\textbf{Base Policy:} The base policy $\pi_{\text{base}}$ is the policy to be accelerated, which is an visumotor policy capable of predicting action chunk from visual observations and robot states. Specifically, during the inference phase of $\pi_{\text{base}}$, the input consists of the visual observation $I_t$ and robot state $o_t$, and the base policy outputs an action chunk via:
$A_t \sim \pi_{\text{base}}(\cdot | I_t, o_t)$.
Since $\pi_{\text{base}}$ is typically trained on slow demonstration data, the actions predicted by $\pi_{\text{base}}$ are generally inefficient in execution.

\textbf{Scheduler Policy:} We introduce an additional plug-and-play scheduler policy to downsample the action chunk produced by $\pi_{\text{base}}$ for acceleration, as shown in Fig.\ref{fig:policy_hierarchy}. Concretely, the scheduler policy $\pi(\cdot|o_t,A_t)$ is a lightweight policy that predicts a downsample rate $k$ given current state $o_t$ and action chunk $A_t$. The final action executed in the environment is thus the downsampled action chunk $A_t^k$ (Eq. \ref{eq:downsample}). Specifically, when $k=1$, the downsampled action chunk coincides with the original, i.e., $A_t^1 = A_t$. This formulation allows our scheduler $\pi$ to achieve state-dependent execution speedup of $\pi_{\text{base}}$ in a plug-and-play manner.


\subsection{Acceleration via Constrained MDP}
\label{sec:cmdp}

\begin{figure}[!t]
    \centering
    \includegraphics[width=0.95\linewidth]{figures/violation.png}
    \caption{\textbf{Success rate and Violation count.} 
Each subplot (a--d) illustrates the relationship between the cumulative count of violations ($h_\mathcal{E}=1$) and the task success rate across different LIBERO suites. 
The bars represent the conditional success rate for the subset of trajectories containing at least $x$ violations, with the sample size of each subset annotated above the corresponding bar.}
    \label{fig:state_deviation}

\end{figure}
We then formulate speedup learning as a scheduler policy optimization problem within the framework of Constrained Markov Decision Processes (CMDPs), defined by the tuple $(\mathcal{S}, \mathcal{K}, \mathcal{P}, r, c, h, \gamma)$. In this formulation, policy acts as a high-level scheduler that optimizes execution efficiency without compromising task performance. 
$\mathcal{S}$ is the state space augmented to include the current environment observation $o_t$ and the action chunk $A_t$ produced by the base policy $\pi_{\text{base}}$ (i.e., $s_t=(o_t, A_t)$). $\mathcal{K}$ is the action space of our scheduler, defined as a discrete set of downsampling rates $\{k_{\text{min}}, \dots, k_{\text{max}}\}$. $\mathcal{P}$ represents the environment dynamics under the execution of the downsampled action chunk $A_t^k$. To incentivize efficiency, we define the reward function as the acceleration gain: $r(s_t, k_t) = k_t$.

Ideally, the cost function $c$ for acceleration should directly reflect the impact of acceleration on task performance. We define the performance-based cost as:\begin{equation}c_q(s_t, k_t) = Q^{\pi_{\text{base}}}(o_t, A_t^k)-Q^{\pi_{\text{base}}}(o_t, A_t),\end{equation} where $Q^{\pi_{\text{base}}}(o, A)$ represents the expected success rate starting from state $o$ and executing action chunk $A$ from $\pi_{\text{base}}$. The violation function is then $h_q(s_t, k_t) = \mathbb{I}\left[c_{q}(s_t, k_t)<0\right]$. We provide theoretical guarantee as follows:\begin{proposition}\label{prop:performance} Given zero-violation constraint ($h_q(s_t,k_t)=0$) at each state, the scheduler is guaranteed to maintain or improve the success rate of the base policy. See App. \ref{app:proof1} for the proof.\end{proposition}

Therefore, the objective of scheduler is to maximize acceleration gain under a zero-violation constraint:\begin{equation} \begin{aligned} \label{eq:cmdp_final} \max_{\pi} \quad & \mathbb{E}_{\pi} \left[ \sum_{t=0}^{T} \gamma^t r(s_t,k_t) \right] \\ \text{s.t.} \quad & \mathbb{E}_{\pi} \left[ \sum_{t=0}^{T} \gamma^t h_q(s_t, k_t) \right] = 0 \end{aligned} \end{equation}

However, evaluating $c_q$ requires the true value function $Q^{\pi_{\text{base}}}$, which is expensive to estimate online. To enable offline learning, we transition from value-based constraints to state-based constraints using a learned world model.

\subsection{World Model based State Deviation as Cost}

\label{sec:state_deviation}
To evaluate costs offline, we utilize a learned world model $\mathcal{M}_{\theta}$ (Sec. \ref{sec:world_model}) to simulate future trajectories. Our key insight is that the success of $\pi_{\text{base}}$ is tied to its specific motion intent; thus, if the accelerated trajectory remains close to the original one, the task performance is preserved.

Formally, let $\tau_t = \{o_{t+1}, \dots, o_{t+n}\}$ and $\tau^k_t$ be the sequence of states predicted by $\mathcal{M}_\theta$ under the original chunk $A_t$ and the downsampled chunk $A_t^k$. We denote $\hat{\tau}^k_t = \{\hat{o}_{t+1}^k, \dots, \hat{o}_{t+n}^k\}$ as the version of $\tau^k_t$ temporally interpolated to match the length of $\tau_t$. We then define the state deviation $\mathcal{E}$ as the maximum discrepancy between the end-effector (EEF) of these two trajectories: \begin{equation}\label{eq:state_deviation}\mathcal{E}(s_t, k_t) = \max_{i \in [1, n]} d(\text{EEF}(o_{t+i}), \text{EEF}(\hat{o}_{t+i}^k)),\end{equation}where $d(\cdot, \cdot)$ is a distance metric for EEF (App. \ref{app:state_deviation_calc}). 

We empirically validate the reliability of our metric by evaluating the $\pi_{0.5}$ model \cite{intelligence2025pi05} on the LIBERO benchmark with a fixed downsampling rate of $k=2$. As shown in Fig. \ref{fig:state_deviation}, our analysis across four task suites reveals a consistent trend: as the number of violations-defined as $h_\mathcal{E}(s_t, k_t) =\mathbb I\left[\mathcal{E}(s_t,k_t)>\epsilon\right]$—within a trajectory increases, the policy's success rate exhibits an evident decline. This pronounced negative correlation serves as strong evidence that state deviation is a faithful proxy for execution risk, validating its effectiveness as a cost signal.

% We empirically validate this surrogate by evaluating the $\pi_{0.5}$ model \cite{intelligence2025pi05} on the LIBERO benchmark. As shown in Fig. \ref{fig:state_deviation}, our analysis across four task suites reveals a strong negative correlation between the number of violations—defined as $h_\mathcal{E}(s_t, k_t) =\mathbb I\left[\mathcal{E}(s_t,k_t)>\epsilon\right]$—and the final task success rate. This confirms that state deviation $\mathcal{E}$ is a reliable proxy as cost function. Detailed calculations and additional correlation studies are provided in App. [A].





% 跟加速baseline的比较是多维度的： GFLOPS比较，炼丹时长比较，行能比较