README.md

metadata

license: other
tags:
  - molmo2
  - codec
  - adacodec
  - video
  - sft

molmo2-codec Stage-2 SFT (step1000)

Stage-2 SFT checkpoint for the AdaCodec-on-molmo2 video pipeline: the LLM + connector are fine-tuned to consume a codec-compressed video representation (I-frames → 81 tokens after 3×3 pool, P-frames → N_P=8 tokens via the Stage-1 P-tokenizer) instead of dense per-frame RGB.

• Base: Molmo2-4B-SFT (weikaih/molmo2-codec-base, gated). Vision tower frozen; LLM + connector + codec_ptok.proj trained. Stage-1 P-tokenizer: weikaih/molmo2-codec-stage1.
• Codec: N_I=81, N_P=8 → ~4× token compression (measured ~25% of the dense token budget).
• Training: 1000 steps, global batch 128, 8×H100, VIDEO_ACADEMIC_V2 mixture (cache-only subset of already-codec-cached videos), seq 8192, vision frozen. Loss converged ~0.53.

Files

• stage2_step1000.pt — consolidated {"model": <full state_dict>, "proj": <codec_ptok.proj>}. Load via codec_eval.py --ckpt (model.load_state_dict(cd["model"], strict=False) + ptok.proj.load_state_dict(cd["proj"])), together with the Stage-1 P-tokenizer.

Evaluation (MLVU multiple-choice, 32 frames, n=100)

	accuracy	visual-token budget
dense (Molmo2-4B-SFT baseline)	75.0%	100%
codec (this ckpt, step1000)	35.0%	25.3% (~4×)

Honest status — this is a small-scale feasibility run, not a competitive model. The pipeline works end-to-end (codec input active, ~4× compression, stable training, coherent inference, above the ~25% MC chance rate), but at this scale the codec model converges to ~35% and does not recover dense accuracy (34.4% @ step660 → 35.0% @ step1000 = essentially flat, i.e. converged). Closing the 35→75 gap is not a matter of training longer at this setup; it needs a larger effort (the AdaCodec paper trains Stage-2 for ~45k steps on ~3.9M examples with 64×H800) and/or a stronger Stage-1 / lower compression ratio. Use as a reproducible proof-of-pipeline, not a checkpoint to deploy.