| --- |
| license: apache-2.0 |
| tags: [molmo2, codec, adacodec, video, stage2] |
| extra_gated_prompt: "Contains weights derived from an unreleased base (Molmo2-4B-SFT). Research use." |
| --- |
| |
| # molmo2-codec Stage-2 **V4** (trained codec video model) |
|
|
| Stage-2 SFT checkpoints for the AdaCodec-on-Molmo2 video pipeline, **V4** (trainable E_P P-frame |
| ViT branch). Video is ingested as a codec representation — I-frames: 81 tokens, P-frames: 16 tokens |
| via the trained E_P — at **~33% of dense's visual-token budget (3x compression)**. |
|
|
| ## Results — MLVU multiple-choice (n=100, official dense-vs-codec eval, same 100 examples) |
|
|
| | checkpoint | codec acc | token budget | |
| |---|---|---| |
| | v2b (frozen P-tokenizer, 0.49M) | 35.0% | ~25% | |
| | **stage2_v4_step500.pt** | **46.0%** | 33.2% | |
| | **stage2_v4_step1200.pt** | **52.0%** | 33.2% | |
| | dense (per-frame RGB, reference) | 71.0% | 100% | |
|
|
| The trainable ~381M E_P branch (vs the old frozen 0.49M P-tokenizer) is the key change: codec accuracy |
| went 35% -> 46% (500 steps) -> 52% (1200 steps) and was still climbing. |
| |
| ## Files |
| - `stage2_v4_step500.pt`, `stage2_v4_step1200.pt` — consolidated `{model, proj}` state dicts (~19GB fp32, |
| full 4.85B model incl. the trained E_P). Load with `codec/codec_eval.py --ckpt <file>` (E_P auto-attaches |
| because the ckpt carries `codec_ptok.vit_p.*`). |
| |
| Code: https://github.com/weikaih04/molmo2-codec (branch `adacodec`). Stage-1 E_P: |
| https://huggingface.co/weikaih/molmo2-codec-stage1-v4 |
|
|
