| --- |
| license: other |
| tags: [molmo2, codec, adacodec, video, sft] |
| --- |
| |
| # molmo2-codec Stage-2 SFT (step1000) |
|
|
| Stage-2 SFT checkpoint for the **AdaCodec-on-molmo2** video pipeline: the LLM + connector are |
| fine-tuned to consume a **codec-compressed** video representation (I-frames → 81 tokens after 3×3 |
| pool, P-frames → N_P=8 tokens via the Stage-1 P-tokenizer) instead of dense per-frame RGB. |
| |
| - **Base:** Molmo2-4B-SFT (`weikaih/molmo2-codec-base`, gated). Vision tower frozen; LLM + |
| connector + `codec_ptok.proj` trained. Stage-1 P-tokenizer: `weikaih/molmo2-codec-stage1`. |
| - **Codec:** N_I=81, N_P=8 → **~4× token compression** (measured ~25% of the dense token budget). |
| - **Training:** 1000 steps, global batch 128, 8×H100, `VIDEO_ACADEMIC_V2` mixture (cache-only |
| subset of already-codec-cached videos), seq 8192, vision frozen. Loss converged ~0.53. |
|
|
| ## Files |
| - `stage2_step1000.pt` — consolidated `{"model": <full state_dict>, "proj": <codec_ptok.proj>}`. |
| Load via `codec_eval.py --ckpt` (`model.load_state_dict(cd["model"], strict=False)` + |
| `ptok.proj.load_state_dict(cd["proj"])`), together with the Stage-1 P-tokenizer. |
|
|
| ## Evaluation (MLVU multiple-choice, 32 frames, n=100) |
|
|
| | | accuracy | visual-token budget | |
| |---|---|---| |
| | dense (Molmo2-4B-SFT baseline) | **75.0%** | 100% | |
| | **codec (this ckpt, step1000)** | **35.0%** | **25.3% (~4×)** | |
|
|
| **Honest status — this is a small-scale feasibility run, not a competitive model.** The pipeline |
| works end-to-end (codec input active, ~4× compression, stable training, coherent inference, above |
| the ~25% MC chance rate), but at this scale the codec model **converges to ~35% and does not |
| recover dense accuracy** (34.4% @ step660 → 35.0% @ step1000 = essentially flat, i.e. converged). |
| Closing the 35→75 gap is not a matter of training longer at this setup; it needs a larger effort |
| (the AdaCodec paper trains Stage-2 for ~45k steps on ~3.9M examples with 64×H800) and/or a stronger |
| Stage-1 / lower compression ratio. Use as a reproducible **proof-of-pipeline**, not a checkpoint to |
| deploy. |
|
|
