license: apache-2.0
base_model:
- Qwen/Qwen3.6-35B-A3B
- Qwen/Qwen3.8-27B
- lmms-lab/LLaVA-Video-7B-Qwen2
- lmms-lab/llava-onevision-qwen2-7b-ov
base_model_relation: adapter
datasets:
- lmms-lab/LLaVA-Video-178K
tags:
- vcompactor
- vision-token-compression
- video-understanding
VCompactor
VCompactor learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's
input, and the VLM itself is not modified. Each subfolder holds the compactor (config.json + model.safetensors,
fp32) for one frozen VLM.
| Subfolder | Frozen VLM | Frames | Compact tokens | Accuracy | Uncompressed |
|---|---|---|---|---|---|
llava-onevision-7b-64tok |
LLaVA-OneVision-7B | 32 | 64 | 56.3 | 58.4 |
llava-onevision-7b-matryoshka |
LLaVA-OneVision-7B | 32 | 32 / 64 / 128 / 320 / 640 | 54.7 / 55.8 / 57.0 / 57.4 / 57.5 | 58.4 |
llava-video-7b-128tok |
LLaVA-Video-7B | 64 | 128 | 58.1 | 60.7 |
qwen3.6-35b-a3b-64tok |
Qwen3.6-35B-A3B | 64 | 64 | 64.8 | 70.6 |
qwen3.8-27b-64tok |
Qwen3.8-27B | 64 | 64 | 58.7 | 64.1 |
Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens,
10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (TOKENS_PER_STEP = 1 / 2 /
4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception
1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first
generated token to A-E (ANSWER_LETTERS=ABCDE).
Usage
Install the VCompactor code (requirements.txt for LLaVA, requirements-qwen.txt for Qwen) and pass
dingyikang/VCompactor/<subfolder>; only that subfolder is downloaded.
from vcompactor import VCompactor
compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda")
MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh
License
Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from LLaVA-Video-178K; follow the terms of that dataset.