VCompactor

VCompactor learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's input, and the VLM itself is not modified. Each subfolder holds the compactor (config.json + model.safetensors, fp32) for one frozen VLM.

Subfolder	Frozen VLM	Frames	Compact tokens	Accuracy	Uncompressed
llava-onevision-7b-64tok	LLaVA-OneVision-7B	32	64	56.3	58.4
llava-onevision-7b-matryoshka	LLaVA-OneVision-7B	32	32 / 64 / 128 / 320 / 640	54.7 / 55.8 / 57.0 / 57.4 / 57.5	58.4
llava-video-7b-128tok	LLaVA-Video-7B	64	128	58.1	60.7
qwen3.6-35b-a3b-64tok	Qwen3.6-35B-A3B	64	64	64.8	70.6
qwen3.8-27b-64tok	Qwen3.8-27B	64	64	58.7	64.1

Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens, 10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (TOKENS_PER_STEP = 1 / 2 / 4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception 1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first generated token to A-E (ANSWER_LETTERS=ABCDE).

Usage

Install the VCompactor code (requirements.txt for LLaVA, requirements-qwen.txt for Qwen) and pass dingyikang/VCompactor/<subfolder>; only that subfolder is downloaded.

from vcompactor import VCompactor

compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda")

MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh

License

Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from LLaVA-Video-178K; follow the terms of that dataset.