Learning to evolve by matching velocities

Can simulation be avoided when learning neural ODE surrogates of dynamical systems?

We introduce Velocity Matching (VM), a simulation-free training objective with guarantees on the learned trajectories.

Neural ODEs are usually trained on trajectories: rollouts backprop through the solver, teacher forcing predicts one step ahead. VM stochastically collocates the field along the piecewise-linear interpolant, regressing it onto finite-diffence velocities. Simulation is not only expensive, it also yields degenerate loss landscapes as the horizon T grows. Already for a scalar linear ODE in continuous time, gradients vanish or explode exponentially in T and the minimum sharpens, hindering convergence of gradient descent. Once the numerical scheme enters the loss, long rollouts in the linear case yield a loss that collapses within the scheme’s absolute stability region and explodes outside it. In the nonlinear setting, this translates into a spectral constraint on the learned Jacobian.

We motivate velocity-based learning by building a theoretical framework based on stability estimates that bound trajectory error by velocity error along true trajectories, so no simulation is needed. It also yields continuous-time universal approximation results. To this mean, we introduce Velocity Matching (VM), an objective which stochastically collocates the learned vector field along the piecewise-linear interpolant of the observed trajectories, relying on a perturbation in the direction of the first order finite-difference velocity estimate.

Experimentally, on nonlinear ODEs test cases, VM reaches lower trajectory error than teacher forcing (TF) and rollouts at the training cost of TF (w/ explicit Euler), and velocity and Jacobian errors drop accordingly (this comes from an implicit regularization effect, explained below). Moreover, VM-learned fields hold a continuous-time approximation property (which we prove via universal approx), namely they can be integrated on finer time grids and with RK schemes other than the training one, whereas rollout-trained models overfit both (here, explicit Euler). On semi-discretized PDEs, rollout training often becomes unstable. With the same architecture and hyperparameters, VM trains stably and attains the lowest test errors, shown here on the Burgers and Korteweg–de Vries equations.

Interpolation-based identification goes back to Bellman in the ’70s (structural system identification), and the same reading connects VM to recent Flow Matching approaches. Recently, CFOs have bridged Neural Operators and Stochastic Interpolants, relying on higher-order splines. Although VM relies on a simpler mechanism, this motivated us to study the role of stochastic collocation along the interpolant, and the perturbation it induces.
We identify three mechanisms behind its effectiveness:

1. The VM objective is a second-order approximation of the velocity risk, with a consistent minimizer, whereas teacher forcing is only first-order.
2. The VM residual coincides with that of an A-stable implicit Runge–Kutta scheme, but without requiring any implicit solve during training.
3. VM implicitly performs a form of Jacobian matching, which proves critical for learning chaotic dynamics and connects to our universal approximation result in Sobolev norms.

As with simulation-based/simulation-free training in flow-based generative modeling, an analogous duality holds for learning dynamical systems. We hope VM and the accompanying theory are a step towards understanding why this is the case.