| # Compresser Decoder (Inverse Perceiver) |
|
|
| Phase 0 pretrained Decoder for the Mamba-3 Semantic Video Compressor. |
|
|
| ## Architecture |
| - **Type**: Inverse Perceiver (cross-attention expansion) |
| - **Input**: [B, 64, 512] — Perceiver compressed tokens |
| - **Output**: [B, 576, 1664] — reconstructed V-JEPA latents |
| - **Params**: ~11.1M |
| - **Details**: 576 learned queries, 3 cross-attention layers, 16 heads, FFN 512→2048→512 |
|
|
| ## Training |
| - **Dataset**: [Vjepa_mamba_dataset_v2](https://huggingface.co/datasets/rookierufus/Vjepa_mamba_dataset_v2) (50 hours video, 384×384, 8fps) |
| - **V-JEPA**: Frozen [vjepa2_1_vit_gigantic_384](https://github.com/facebookresearch/vjepa2) (2.2B params) |
| - **Loss**: MSE reconstruction (autoencoder target = V-JEPA latent) |
| - **Optimizer**: AdamW, lr=1e-4, cosine to 1e-6 |
| - **Hardware**: RTX 4090 (48 GB), bf16 |
|
|
| ## Usage |
| ```python |
| from compressor.decoder import PerceiverDecoder |
| |
| model = PerceiverDecoder(input_dim=512, output_dim=1664, num_queries=576) |
| model.load_state_dict(torch.load("decoder_stepX_hrsY.pt")) |
| # Input: [B, 64, 512] Perceiver output → Output: [B, 576, 1664] V-JEPA latents |
| ``` |
|
|
| ## Note |
| Disposable after Phase 0 — only the Encoder (Perceiver) carries forward to the main pipeline. |
|
|
