Sparse Tokenizer Bench
videoTokenizer · RTX 3090 ·

Sparse Tokenizer Bench

The tokenizer cuts each 128 × 128 frame into an 8 × 8 grid of 16 px cells. A router decides which cells carry an event this frame; a fovea gives a few of them high resolution. A temporal router then groups frames into level-1 and level-2 tokens. This page tracks, run by run, whether each of those choices lands where the video actually changes.

event cell (router) high-res cell (fovea) level-1 temporal token level-2 temporal token pixels changed (ground truth)
Real robot video · held-out episodes

Watch it tokenize

Left to right: frame t with rs_base's event and high-res cells; the same for the pixel-change heuristic rs_heur; rs_base's prediction of frame t+1; the true frame t+1. Strip: the last 96 frames of motion, events per frame and temporal tokens. The clip is the 150-frame window with the most motion among the first 1,500 held-out frames, at the stream's own frame rate.
Probe 1 · temporal router

Does the temporal router fire on real video?

Level-1 tokens per second divided by the heartbeat rate (one forced token every 64 frames). 1× means the router never chose a boundary itself. Bar = all frames; ● = the 25% most moving frames, ○ = the 25% stillest. Log scale.
Probe 2 · spatial router and fovea

Where do the events land?

Each square is one held-out stream (a frame from its middle), tinted by how often each of the 64 cells is changed / gets an event / gets high resolution, over up to 3,000 held-out frames. Numbers: event recall and precision against changed cells.
Probe 3 · what the tokens are good for

Next-frame prediction and actions from vision

Next-frame PSNR minus copying frame t (dB)

Right of the line: better than repeating the last frame.

Action error from vision ÷ predicting zero

Previous actions zeroed, so the model must read the action from the video. Left of the line: better than predicting zero.

All runs, mean over 17 streams

One seed per configuration. rs_d1 and rs_d4 never saw most of these streams in training, so their all-stream means are not a data-scaling result; see the next table.

Data scaling, on the streams the small-data runs were trained on

Action error from vision (lower is better; predicting zero shown for reference) and PSNR gain over copying. All runs: 8,000 steps, B 8, T 256.
Toy stream · JEPA objective

Does a latent objective stop paying for the noisy TV?

Held-out toy stream, 10 fps. Tags: A arm, I item, R rule symbol (shown for 5 frames per flash), TV the noisy TV, which changes every frame but never matters for the task. Strip: events on the TV cell, the rule cell (dark marks = frames where the rule is shown) and the arm + item cells.

Share of all events, by entity

Share of all high-res cells, by entity

High-res rate on the TV cell during training

Latent prediction loss by horizon (JEPA runs)

Notebook

Update log