Compressing visual tokens in vision-language models: 3x more requests per GPU on Qwen2-VL

Gregory Allen

ga11en

Follow

datas3nt

TL;DR

We compress the visual tokens a vision-language model consumes, train a small LoRA adapter to interpret the compressed sequence, and measure the result end-to-end on Qwen2-VL. On Qwen2-VL-7B running on an NVIDIA H200 under vLLM continuous batching, the recipe serves about 3x more requests per GPU at the balanced operating tier (T=2, +7.4pp accuracy), with storage about 5x smaller and accuracy that goes up rather than down (+8.4pp on ScienceQA at the accuracy tier T=3). The same adapter transfers to MMMU and VQAv2 with no retraining.

This is a throughput result, not a per-request latency result: each GPU serves more requests because each request costs less prefill compute, not because a single long generation finishes faster. We measured that distinction and we report it plainly below. The model cards, an interactive demo, and a RAG demo are public; the Polygen SDK is licensed.

Why visual tokens are the cost center

A vision-language model runs an image through a vision encoder, then a merger that down-samples the patch grid, and finally feeds the resulting visual tokens into the LLM alongside the text prompt. For Qwen2-VL on natural-image VQA the visual token count is in the low hundreds; for document images it climbs into the thousands. The LLM forward pass spends most of its prefill attending across those visual tokens.

Two cost pressures fall out of this. At inference time, the LLM's attention is quadratic in sequence length, so shrinking the visual prefix cuts per-request prefill compute more than proportionally. At storage time, multimodal corpora are dominated by raw images, so storing a compact representation of the merger output instead cuts storage substantially.

Polygen gives both at once. Our encoder turns the merger output into a much shorter sequence of polygen tokens, fed to the LLM in place of the raw merger output. The encoder uses a per-tile quality signal, sigma_R, to keep the parts of the image carrying the most information at full fidelity and compress the rest. The math behind the encoder is Datasent's; the rest of this post is how it integrates with Qwen2-VL and what it does end-to-end.

The splice point

[image pixels]
   v vision encoder
[per-patch features]
   v merger (2x2 spatial pooling, 1280 -> 1536)
[merged visual tokens (n_patches, 1536)]    <-- splice here
   v LLM (text + visual tokens, autoregressive generation)
[answer tokens]

We splice right at the merger output. It is the distribution the LLM was trained to consume, the shape is uniform across images modulo n_patches, and the per-patch features upstream of the merger are 4x larger (the merger does 2x2 spatial pooling). Splicing pre-merger amplifies reconstruction error by an order of magnitude in our measurements.

The hybrid policy

The encoder splits each image's merger output into tiles along the patch axis. For each tile it emits a compressed representation plus a sigma_R value, a per-tile signal measuring how much information the compressed form did not capture.

The hybrid policy uses sigma_R to spend compression budget where it matters: tiles ranked highest are kept at full fidelity, tiles ranked lowest are kept compressed. An operating tier knob T (from T=0 most aggressive through T=4 lossless length) sets the cut-off. The intuition is the Pareto principle for visual information: most of the visually meaningful content concentrates in a small fraction of tiles, and the rest compresses with little loss. A LoRA adapter trained on 300 ScienceQA images at random T per step teaches the LLM to interpret the compressed distribution at every tier, and the same adapter generalizes; we never trained on MMMU, VQAv2, or DocVQA.

The headline: throughput density

The point of a shorter visual prefix is that it costs less compute per request, so a fixed GPU serves more requests. We measured this directly on Qwen2-VL-7B on one NVIDIA H200 under vLLM continuous batching, at the balanced tier T=2:

The gain sits in the low-to-mid 3x range across output lengths. Baseline throughput was about 20 requests/second; with Polygen it rose to roughly 70. In hardware terms that is about one-third the GPUs for the same request volume.

The mechanism: prefill attention scales with the square of sequence length, so halving the visual sequence more than halves the per-request prefill cost. Pack more requests into the same GPU-second and throughput rises. This holds for the cached, encode-once / query-many serving pattern on the bypass-eligible path. We have not yet measured a fresh-image-per-request workload that pays the vision tower and encode on every call, and we are not quoting a number for it.

Accuracy and storage across tiers

All numbers are end-to-end on Qwen2-VL with the wrapper. LoRA recipe: r=16, alpha=32, 1000 steps, lr=1e-4, AdamW.

Accuracy peaks at T=3 (+8.4pp over the un-spliced baseline); T=2 gives up a statistically negligible amount of accuracy for more compression and the throughput result above. The gain is consistent rather than a lucky point: it transfers cross-benchmark (+2.50pp on VQAv2 at 7B, +1.57pp on MMMU at 2B, neither trained on), holds across CLIP, SigLIP, and DINOv2 vision encoders, and shows near-zero multi-seed variance.

Where it doesn't win, stated plainly

Single-request wall-clock on long generations is essentially flat. Long-generation latency is decode-bound: the model reloads its weights per output token regardless of how short the visual prefix is, so shortening the visual sequence doesn't speed up a long answer. The levers Polygen moves are throughput, time-to-first-token (2-4x faster at T=2 because prefill is shorter), memory (about 2x more concurrent visual contexts in the KV cache at no accuracy cost), and storage. For long single-stream generation the win is storage and memory; for high-volume serving or retrieval over a fixed corpus, throughput is the headline.

What we learned on DocVQA

The natural splice path bypasses pixel_values after capturing the merger output, then feeds the LLM a shortened inputs_embeds. That bypass works on ScienceQA, MMMU, and VQAv2. On DocVQA it produced an anomalous accuracy drop.

The cause: the bypass omits pixel_values, so Qwen2-VL never computes its multimodal RoPE, the 2D positional encoding that tells the LLM which row and column each visual token came from. For ScienceQA (88 visual tokens) and VQAv2 (345) the layout loss is bearable; the model mostly attends to content. For DocVQA (~4,800 tokens for a document page) the 2D layout is the signal, and the 1D fallback makes "top-right corner" indistinguishable from "bottom-left" up to a constant.

The fix is a second splice path that replaces the merger output in place via a forward hook. pixel_values flows through normally, mRoPE is computed normally, and per-image patch count is unchanged so the standard generate consistency check passes. At T=4 this path matches the baseline within Wilson 95% CI (+0.21pp on DocVQA, n=500). The wrapper auto-dispatches between the bypass (moderate visual length) and the hook (long visual length) from the per-image patch count. Reconstruction on the hook path is GPU-resident, so there is no per-forward CPU round-trip. Note this path keeps the accuracy and storage benefits but not the throughput gain, since it preserves sequence length.

See it run

You can try the recipe right now without any setup:

• Interactive demo: upload an image, ask a question, see baseline and compressed answers side by side: https://huggingface.co/spaces/datas3nt/qwen2vl-polygen
• RAG demo, the encode-once / query-many pattern: https://huggingface.co/spaces/datas3nt/qwen2vl-polygen-rag
• 7B model card with the full per-tier curve and intended-use envelope: https://huggingface.co/datas3nt/qwen2vl-polygen-7b-lora-r16-1000

Integration is a near-drop-in wrapper. At the call site it swaps for stock Qwen2VLForConditionalGeneration, with an operating tier and automatic splice mode:

model = PolygenQwen2VL.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct",
    lora_adapter="datas3nt/qwen2vl-polygen-7b-lora-r16-1000",
    operating_tier="T=2",   # "auto" splice mode by default
)
out = model.generate(**inputs, max_new_tokens=20)

The Polygen encoder and SDK are licensed rather than open, so this shows the integration shape, not a public pip install. If you want to run it against your own workload, get in touch.

What's next

• A fresh-image-per-request throughput measurement, to characterize the non-cached serving case.
• Cross-benchmark coverage on MathVista, a fifth task family.
• Multi-LoRA per-tier, which we expect to lift the last 0.5-1pp at fixed T.

If you're running a VLM workload where visual-token serving cost or multimodal storage is a pain point, the Datasent platform is the contact path.