physicalai-bmi/nano-vla-pixels

A policy trained on real MuJoCo pixels — frames captured live from the Institute's in-browser Forge arm sim, not a synthetic renderer. It watches the actual rendered scene and outputs the joint-delta action, distilling our state-based checkpoint (forge-arm-reach-bc) into a pixel policy on genuine sim renders.

• Input: 3 stacked 48×48 RGB frames of the real MuJoCo render (temporal context = velocity).
• Output: a 3-D joint-delta action.
• Params: 186,099.

How the data was made

The state checkpoint drove the real Forge arm in a headless browser (WebGPU/Metal); 2,500 frames across 128 reaches were captured by screenshotting the WebGPU canvas (a page's own JS cannot read WebGPU pixels — only an external compositor screenshot can), each paired with the policy's action. See v2/tools/nanovla/capture.cjs.

Results (held-out reaches, real pixels)

metric	value
Held-out action MSE	7.26 × 10⁻⁵
Predict-the-mean baseline	4.25 × 10⁻⁴
Variance explained	82.9%
Single-frame (no temporal context)	46.6%

The jump from 46.6% → 82.9% with 3-frame stacking shows the action is velocity-dependent — a single still frame under-determines it. Reading fine control from coarse real renders is genuinely hard (and needs far more data than a state policy), so this is a research artifact, not a production controller.

Honest limitation

It can be run offline on captured frames (there's an in-browser replay at https://physicalai-bmi.org/research/vla), but a live in-page closed loop is impractical today — and it's worth being precise about why. Reading the render back into JS is the catch: the 2D/getImageData/screenshot path reads blank on a WebGPU canvas (it's double-buffered, not preserved after present). WebGPU's proper readback (copyTextureToBuffer + mapAsync) is real, and three.js exposes readRenderTargetPixelsAsync, but that path is currently unreliable in three.js's WebGPU backend when rendering alongside an animation loop (three.js #31658, #31654). So an external capture is the dependable route today. It is impractical, not fundamentally impossible.

Files: model.safetensors, vla.web.json (float32 for in-browser; forward verified vs safetensors), metrics.json, demo/ (real held-out frame triplets). CC-BY-4.0.