research/benchmark-notes.md

# Benchmark Observations & Notes

*Chef et Dev | Gemma Speed Challenge*

---

## Infrastructure Notes

### Hardware: NVIDIA A10G 24GB
- Compute capability: 8.9
- Memory bandwidth: ~378 GB/s
- FP16/BF16 performance: ~156 TFLOPS
- FP8 performance: ~313 TFLOPS (sparse)

### Known Issues Encountered
1. Volume mount failures on HF Jobs (resolved)
2. rope_scaling compatibility with vLLM 0.7.3 (resolved by upgrading to 0.22.0)
3. `--no-enable-log-requests` flag removed in newer vLLM (resolved)
4. FlashInfer not installed in harness image (resolved)

---

## vLLM 0.22.0 Default Configuration

```python
# Default from harness serve.py:
--model google/gemma-4-E4B-it
--served-model-name gemma-4-e4b-it
--host 0.0.0.0
--port 8000
--dtype bfloat16
--max-model-len 4096
--gpu-memory-utilization 0.90
--trust-remote-code
--no-enable-log-requests
--max-num-batched-tokens 512  # (if provided)
```

### What This Means
- **BF16 inference** — standard precision, no quantization yet
- **4096 context** — focused on single-stream, reasonable context length
- **90% GPU memory** — leaves 10% for KV cache + PPL stage
- **512 batched tokens** — bounds prefill chunk size for PPL stability

---

## Expected Performance Tiers

| Tier | TPS | Technique |
|------|-----|-----------|
| Tier 1 | ~300-320 | vLLM default BF16 (vanilla) |
| Tier 2 | ~350-380 | vLLM + FlashInfer + CUDA graphs |
| Tier 3 | ~380-400 | FP8 quantization + optimized vLLM |
| Tier 4 | ~400-416 | AWQ/FP8 + kernel optimizations |
| Frontier | ~416+ | Aggressive combined optimizations |

---

## Optimization Research Priorities

### High Impact (Quick Wins)
1. **FlashInfer installation** — should be automatic with vLLM 0.22.0
2. **CUDA graph warmup** — ensure stable capture before timing
3. **PagedAttention tuning** — max_seq_len_to_capture parameter

### Medium Impact
1. **FP8 quantization** — native A10G support
2. **KV cache quantization** — FP8 KV cache
3. **Scheduler tuning** — delay_factor, num_scheduler_steps

### Lower Impact
1. **AWQ int4** — requires weight conversion
2. **Triton kernels** — manual implementation needed
3. **FlashAttention-3** — needs custom install

---

## PPL Stage Requirements (MUST comply)
- Endpoint must support `/v1/completions`
- Must accept `prompt` as list of int token IDs
- Must accept `prompt_logprobs`
- Must respect `add_special_tokens: false`
- Must return `prompt_logprobs` in response
- Must support `return_token_ids: true`
- Greedy decode must match reference exactly

---

*Last updated: 2026-06-11 by chef-et-dev*