LLM_COMPRESSOR_FEATURES.md

LLM Compressor & vLLM Advanced Features

This document outlines advanced features from LLM Compressor and vLLM that can be leveraged for better performance and optimization.

LLM Compressor Features

1. Quantization Modifiers

LLM Compressor supports multiple quantization methods beyond AWQ:

AWQModifier (Activation-aware Weight Quantization)

from llmcompressor.modifiers.awq import AWQModifier

AWQModifier(
    w_bit=4,              # Weight bits (4 or 8)
    q_group_size=128,     # Quantization group size
    zero_point=True,      # Use zero-point quantization
    version="GEMM"        # Kernel version: "GEMM" or "GEMV"
)

GPTQModifier (GPTQ Quantization)

from llmcompressor.modifiers.quantization import GPTQModifier

GPTQModifier(
    w_bit=4,              # Weight bits
    q_group_size=128,     # Group size
    desc_act=False,       # Whether to use activation order
    sym=True              # Symmetric quantization
)

INT8Modifier (8-bit Quantization)

from llmcompressor.modifiers.quantization import INT8Modifier

INT8Modifier(
    w_bit=8,
    q_group_size=128
)

2. Pruning Modifiers

MagnitudePruningModifier

from llmcompressor.modifiers.pruning import MagnitudePruningModifier

MagnitudePruningModifier(
    sparsity=0.5,         # 50% sparsity
    structured=False      # Unstructured pruning
)

3. Combined Modifiers

You can combine multiple modifiers for maximum compression:

from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier
from llmcompressor.modifiers.pruning import MagnitudePruningModifier

oneshot(
    model="Alovestocode/router-qwen3-32b-merged",
    output_dir="./router-qwen3-compressed",
    modifiers=[
        AWQModifier(w_bit=4, q_group_size=128),
        MagnitudePruningModifier(sparsity=0.1)  # 10% pruning + AWQ
    ]
)

vLLM Advanced Features

1. FP8 Quantization (Latest)

vLLM supports FP8 quantization for even better performance:

from vllm import LLM

llm = LLM(
    model="Alovestocode/router-qwen3-32b-merged",
    quantization="fp8",           # FP8 quantization
    dtype="float8_e5m2",          # FP8 format
    gpu_memory_utilization=0.95
)

Benefits:

• ~2x faster than AWQ
• Lower memory usage
• Better quality retention

2. FP8 KV Cache

Reduce KV cache memory usage with FP8:

llm = LLM(
    model="Alovestocode/router-qwen3-32b-merged",
    quantization="awq",
    kv_cache_dtype="fp8",         # FP8 KV cache
    gpu_memory_utilization=0.90
)

Benefits:

• 50% reduction in KV cache memory
• Enables longer context windows
• Minimal quality impact

3. Chunked Prefill (Already Implemented)

enable_chunked_prefill=True  # ✅ Already in our config

Benefits:

• Better handling of long prompts
• Reduced memory spikes
• Improved throughput

4. Prefix Caching (Already Implemented)

enable_prefix_caching=True  # ✅ Already in our config

Benefits:

• Faster time-to-first-token (TTFT)
• Reuses common prefixes
• Better for repeated prompts

5. Continuous Batching (Already Implemented)

max_num_seqs=256  # ✅ Already in our config

Benefits:

• Dynamic batching
• Better GPU utilization
• Lower latency

6. Tensor Parallelism

For multi-GPU setups:

llm = LLM(
    model="Alovestocode/router-qwen3-32b-merged",
    tensor_parallel_size=2,      # Use 2 GPUs
    pipeline_parallel_size=1      # Pipeline parallelism
)

7. Speculative Decoding

For faster inference with draft models:

llm = LLM(
    model="Alovestocode/router-qwen3-32b-merged",
    speculative_model="small-draft-model",  # Draft model
    num_speculative_tokens=5                # Tokens to speculate
)

8. SGLang Backend

For even better performance with structured outputs:

llm = LLM(
    model="Alovestocode/router-qwen3-32b-merged",
    enable_lora=True,              # LoRA support
    max_lora_rank=16
)

Recommended Optimizations for Our Use Case

Current Setup (Good)

• ✅ AWQ 4-bit quantization
• ✅ Continuous batching (max_num_seqs=256)
• ✅ Prefix caching
• ✅ Chunked prefill
• ✅ FlashAttention-2

Additional Optimizations to Consider

1. FP8 KV Cache (High Impact)

llm_kwargs = {
    "model": repo,
    "quantization": "awq",
    "kv_cache_dtype": "fp8",      # Add this
    "gpu_memory_utilization": 0.95,  # Can increase with FP8 KV
    # ... rest of config
}

Impact: 50% KV cache memory reduction, longer contexts

2. FP8 Quantization (If Available)

llm_kwargs = {
    "model": repo,
    "quantization": "fp8",        # Instead of AWQ
    "dtype": "float8_e5m2",
    # ... rest of config
}

Impact: ~2x faster inference, better quality

3. Optimized Sampling Parameters

sampling_params = SamplingParams(
    temperature=0.2,
    top_p=0.9,
    max_tokens=20000,
    stop=["<|end_of_plan|>"],
    skip_special_tokens=False,   # Keep special tokens for parsing
    spaces_between_special_tokens=False
)

4. Model Warmup with Real Prompts

def warm_vllm_model(llm, tokenizer):
    """Warm up with actual router prompts."""
    warmup_prompts = [
        "You are the Router Agent. Test task: solve 2x+3=7",
        "You are the Router Agent. Test task: implement binary search",
    ]
    for prompt in warmup_prompts:
        outputs = llm.generate(
            [prompt],
            SamplingParams(max_tokens=10, temperature=0)
        )

Implementation Priority

1. High Priority:

   • FP8 KV cache (easy, high impact)
   • Optimized sampling parameters (easy)

2. Medium Priority:

   • FP8 quantization (if models support it)
   • Better warmup strategy

3. Low Priority:

   • Tensor parallelism (requires multi-GPU)
   • Speculative decoding (requires draft model)

References

• vLLM Quantization Docs
• LLM Compressor Docs
• vLLM Performance Guide
• FP8 Quantization Paper