VideoMAE GGUF (base + Kinetics-400)

GGUF conversion of MCG-NJU/videomae-base (pretraining release) and MCG-NJU/videomae-base-finetuned-kinetics (Kinetics-400 classifier), packaged for the SkyVision.cpp inference engine.

VideoMAE-base is a pure video tower (94M parameters, no text decoder, no tokenizer, no KV cache): 16 uniform frames → bicubic resize + center crop 224 → one full space-time attention pass over 1568 tokens → either a Kinetics-400 label distribution or the pooled 768-d embedding.

Files

File	Size	Notes
`videomae-base.gguf`	169 MB	pretraining release, no classifier head — outputs the 768-d pooled embedding
`videomae-base-ft.gguf`	170 MB	Kinetics-400 fine-tune (F16), `--mode N` prints the top-N labels
`videomae-base-ft-q8.gguf`	93 MB	same fine-tune, Q8_0 weights: ~30% faster on CPU, top-1 probability drift 4e-4

Usage

# video classification (top-5 Kinetics-400 labels)
vqa-cli --model videomae-base-ft.gguf --video clip.mp4 --mode 5

# a still image becomes a clip of identical frames
vqa-cli --model videomae-base-ft.gguf --image photo.jpg --mode 5

# pooled features (768 floats as JSON) from the pretraining release
vqa-cli --model videomae-base.gguf --video clip.mp4 --mode embed

CPU: ~0.85 s per clip (16 frames / 1568 tokens / 12 layers) on an i7-14700K with F16 weights, ~0.6 s with the Q8_0 file. --ask is meaningless for this architecture and is ignored.

Conversion notes

Conv3d(C, 768, kernel=(2,16,16)) patch embedding is folded into a single matmul over the token order used by the official implementation (t', h', w') with element order (c, dt, dh, dw);
q/k/v projections are fused into one wqkv weight; VideoMAE has no key bias, so the combined bias is q_bias | 0 | v_bias;
the position embedding is a fixed sin-cos table (not learned) generated at conversion time;
there is no cls token: pooled is token 0 with a final encoder LayerNorm for the pretraining release, and the mean over tokens + fc_norm (no encoder LayerNorm) for use_mean_pooling fine-tunes like Kinetics;
GELU is the exact erf form (nn.GELU()), not the tanh approximation.

Verification

Checked against an independent numpy recomputation of the whole forward pass straight from the GGUF weights (preprocessing taken from the engine's own patch dump):

stage	max difference
patch embedding + position	3.1e-3
block-0 attention	3.3e-3
12-layer output	~9e-4 relative (values up to 438)
pooled vector	5.5e-4
all 400 Kinetics-400 probabilities	< 9e-5

All within F16 weight noise. The same numbers reproduce on every run, so the conversion can be re-checked with the repo's scripts/ref_videomae.py helper against the official transformers implementation.

License

VideoMAE is released under CC-BY-NC-4.0 (non-commercial). This GGUF conversion inherits that license; commercial use requires the permission of the original authors.