spectator2026/Infinity-Parser2-Pro-AWQ-W4A16

Infinity-Parser2-Pro — AWQ W4A16 (int4)

An int4 (W4A16) AWQ quantization of infly/Infinity-Parser2-Pro, made to run on NVIDIA A100 / sm80, where the original FP8 path is unsupported. 70 GB bf16 → **21 GB**.

Method

AWQ via llm-compressor, routed experts only (W4A16). Attention, GDN/linear-attention, shared expert, vision tower, and lm_head are kept in bf16. Calibrated on a few hundred diverse document + general-vision samples.

Serving note (important)

vLLM fuses some bf16 layers before consulting the ignore list, which can otherwise yield all-! output. Fix: the saved config.json quantization_config.ignore uses broad regexes matching the fused names. Already applied here.

Quality (VLMEvalKit, AI-judged reproduction)

Benchmark	Published bf16	This int4
MMStar	69.7	66.9
OCRBench	86.2	~89
DocVQA (val)	96.4	96.4

Near-lossless on the cleanly-comparable axes. (MMBench omitted to avoid a circular-vs-vanilla scoring mismatch.)

Usage (vLLM)

vllm serve spectator2026/Infinity-Parser2-Pro-AWQ-W4A16 --dtype bfloat16 --trust-remote-code --reasoning-parser qwen3

Pass chat_template_kwargs={"enable_thinking": false} in requests, or answers land in the reasoning channel.

---

Quantized by @spectator2026. Original model © infly, Apache-2.0.