research/benchmark-notes.md

Benchmark Observations & Notes

Chef et Dev | Gemma Speed Challenge

---

Infrastructure Notes

Hardware: NVIDIA A10G 24GB

• Compute capability: 8.9
• Memory bandwidth: ~378 GB/s
• FP16/BF16 performance: ~156 TFLOPS
• FP8 performance: ~313 TFLOPS (sparse)

Known Issues Encountered

1. Volume mount failures on HF Jobs (resolved)
2. rope_scaling compatibility with vLLM 0.7.3 (resolved by upgrading to 0.22.0)
3. --no-enable-log-requests flag removed in newer vLLM (resolved)
4. FlashInfer not installed in harness image (resolved)

---

vLLM 0.22.0 Default Configuration

# Default from harness serve.py:
--model google/gemma-4-E4B-it
--served-model-name gemma-4-e4b-it
--host 0.0.0.0
--port 8000
--dtype bfloat16
--max-model-len 4096
--gpu-memory-utilization 0.90
--trust-remote-code
--no-enable-log-requests
--max-num-batched-tokens 512  # (if provided)

What This Means

• BF16 inference — standard precision, no quantization yet
• 4096 context — focused on single-stream, reasonable context length
• 90% GPU memory — leaves 10% for KV cache + PPL stage
• 512 batched tokens — bounds prefill chunk size for PPL stability

---

Expected Performance Tiers

Tier	TPS	Technique
Tier 1	~300-320	vLLM default BF16 (vanilla)
Tier 2	~350-380	vLLM + FlashInfer + CUDA graphs
Tier 3	~380-400	FP8 quantization + optimized vLLM
Tier 4	~400-416	AWQ/FP8 + kernel optimizations
Frontier	~416+	Aggressive combined optimizations

---

Optimization Research Priorities

High Impact (Quick Wins)

1. FlashInfer installation — should be automatic with vLLM 0.22.0
2. CUDA graph warmup — ensure stable capture before timing
3. PagedAttention tuning — max_seq_len_to_capture parameter

Medium Impact

1. FP8 quantization — native A10G support
2. KV cache quantization — FP8 KV cache
3. Scheduler tuning — delay_factor, num_scheduler_steps

Lower Impact

1. AWQ int4 — requires weight conversion
2. Triton kernels — manual implementation needed
3. FlashAttention-3 — needs custom install

---

PPL Stage Requirements (MUST comply)

• Endpoint must support /v1/completions
• Must accept prompt as list of int token IDs
• Must accept prompt_logprobs
• Must respect add_special_tokens: false
• Must return prompt_logprobs in response
• Must support return_token_ids: true
• Greedy decode must match reference exactly

---

Last updated: 2026-06-11 by chef-et-dev