Compresser Decoder (Inverse Perceiver)
Phase 0 pretrained Decoder for the Mamba-3 Semantic Video Compressor.
Architecture
- Type: Inverse Perceiver (cross-attention expansion)
- Input: [B, 64, 512] — Perceiver compressed tokens
- Output: [B, 576, 1664] — reconstructed V-JEPA latents
- Params: ~11.1M
- Details: 576 learned queries, 3 cross-attention layers, 16 heads, FFN 512→2048→512
Training
- Dataset: Vjepa_mamba_dataset_v2 (50 hours video, 384×384, 8fps)
- V-JEPA: Frozen vjepa2_1_vit_gigantic_384 (2.2B params)
- Loss: MSE reconstruction (autoencoder target = V-JEPA latent)
- Optimizer: AdamW, lr=1e-4, cosine to 1e-6
- Hardware: RTX 4090 (48 GB), bf16
Usage
from compressor.decoder import PerceiverDecoder
model = PerceiverDecoder(input_dim=512, output_dim=1664, num_queries=576)
model.load_state_dict(torch.load("decoder_stepX_hrsY.pt"))
# Input: [B, 64, 512] Perceiver output → Output: [B, 576, 1664] V-JEPA latents
Note
Disposable after Phase 0 — only the Encoder (Perceiver) carries forward to the main pipeline.