---
license: apache-2.0
base_model:
- Qwen/Qwen3.6-35B-A3B
- Qwen/Qwen3.8-27B
- lmms-lab/LLaVA-Video-7B-Qwen2
- lmms-lab/llava-onevision-qwen2-7b-ov
base_model_relation: adapter
datasets:
- lmms-lab/LLaVA-Video-178K
tags:
- vcompactor
- vision-token-compression
- video-understanding
---

# VCompactor

[VCompactor](https://github.com/DingYikang/longvlm/tree/release) learns compact vision tokens for frozen VLMs: they replace the original vision tokens in the VLM's
input, and the VLM itself is not modified. Each subfolder holds the compactor (`config.json` + `model.safetensors`,
fp32) for one frozen VLM.

| Subfolder | Frozen VLM | Frames | Compact tokens | Accuracy | Uncompressed |
|---|---|:-:|:-:|:-:|:-:|
| `llava-onevision-7b-64tok` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 64 | 56.3 | 58.4 |
| `llava-onevision-7b-matryoshka` | [LLaVA-OneVision-7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov) | 32 | 32 / 64 / 128 / 320 / 640 | 54.7 / 55.8 / 57.0 / 57.4 / 57.5 | 58.4 |
| `llava-video-7b-128tok` | [LLaVA-Video-7B](https://huggingface.co/lmms-lab/LLaVA-Video-7B-Qwen2) | 64 | 128 | 58.1 | 60.7 |
| `qwen3.6-35b-a3b-64tok` | [Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 64 | 64 | 64.8 | 70.6 |
| `qwen3.8-27b-64tok` | [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 64 | 64 | 58.7 | 64.1 |

Accuracy is the average over MVBench, EgoSchema, LongVideoBench and Video-MME (uncompressed: 6,272 vision tokens,
10,816 for LLaVA-Video). The Matryoshka compactor serves all five budgets with one encoder (`TOKENS_PER_STEP` = 1 / 2 /
4 / 10 / 20 tokens per frame) and transfers to images: POPE accuracy 81.11 / 83.89 / 85.63 / 86.12 and MME perception
1081.4 / 1140.0 / 1346.9 / 1375.3 at 2 / 4 / 10 / 20 tokens per image crop. Qwen3.8 results restrict the first
generated token to A-E (`ANSWER_LETTERS=ABCDE`).

## Usage

Install the [VCompactor code](https://github.com/DingYikang/longvlm/tree/release) (`requirements.txt` for LLaVA, `requirements-qwen.txt` for Qwen) and pass
`dingyikang/VCompactor/<subfolder>`; only that subfolder is downloaded.

```python
from vcompactor import VCompactor

compactor = VCompactor.from_pretrained("dingyikang/VCompactor/llava-onevision-7b-64tok", device="cuda")
```

```bash
MODEL=<LLaVA-OneVision-7B directory> COMPACTOR=dingyikang/VCompactor/llava-onevision-7b-64tok bash scripts/eval.sh
```

## License

Apache-2.0. The compactors were trained with the frozen VLMs above (Apache-2.0) on videos and questions from
LLaVA-Video-178K; follow the terms of that dataset.