AutoCompress: Automated Hardware-Guided Model Compression
Abstract
AutoCompress is a hardware-guided compression framework that uses differentiable architecture search and serving-engine cost models to produce heterogeneous LLM architectures meeting latency, throughput, and memory constraints while preserving accuracy.
LLM pruning must preserve quality while meeting hardware- and workload-specific constraints on latency, throughput, and memory. Existing pruning workflows often target parameter counts, requiring practitioners to manually select model sizes that meet latency, throughput, and GPU memory limits. However, parameter count alone does not predict these metrics, which depend on the model architecture, workload, hardware, and serving engine. We introduce AutoCompress, a hardware-guided compression framework that directly incorporates such constraints into differentiable architecture search. AutoCompress jointly optimizes global and per-layer structure to produce heterogeneous architectures, guided by serving-engine cost models validated against end-to-end latency, throughput, and memory measurements. We evaluate AutoCompress on long-context local coding, batched datacenter serving, and latency-sensitive in-game interaction. Compared to the state-of-the-art Minitron pruning framework, AutoCompress improves average accuracy across eight benchmarks by 6.57, 2.35, and 0.57 percentage points on RTX 5070, RTX 5080, and H100, respectively. AutoCompress also compresses Nemotron 3.5 Super from 120B to 74B parameters for single-B200 BF16 serving at 128K context, retaining 95% of its mean benchmark score.
Community
Introducing AutoCompress: Automated Hardware-Guided Model Compression.
Give it a model and a latency and memory constraint for your deployment, and AutoCompress automatically produces a compressed model that meets them. No manual architecture design.
How it works: AutoCompress builds cost models through efficient, accurate profiling of the model on the target GPU. A differentiable architecture search then automatically picks an architecture that fits the constraints. Finally, we prune the model to the chosen architecture and distill it to recover accuracy.
Results:
✅ We compressed models for three diverse real-world use cases: an on-device game NPC (RTX 5070), a long-context coding agent (RTX 5080), and batched datacenter serving (8×H100). AutoCompress improves average accuracy by +6.6 / +2.4 / +0.6 points over the best baseline available (Minitron), with 2–3.3× fewer architecture-search tokens.
✅ At scale, we compressed Nemotron 3.5 Super from 120B to 74B parameters while keeping about 95% of its mean benchmark score.
Get this paper in your agent:
hf papers read 2611.08290 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper