Add Performance-under-load section (concurrency sweep, GuideLLM real-data)
Browse files
README.md
CHANGED
|
@@ -141,6 +141,28 @@ context length, batch size, and backend. Notes from the serving campaign:
|
|
| 141 |
`VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a
|
| 142 |
further +13% in my follow-up runs.
|
| 143 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
## Serving (vLLM)
|
| 145 |
|
| 146 |
The architecture is not yet in upstream vLLM
|
|
|
|
| 141 |
`VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a
|
| 142 |
further +13% in my follow-up runs.
|
| 143 |
|
| 144 |
+
|
| 145 |
+
## Performance under load (GuideLLM, real articles, eagle3 head)
|
| 146 |
+
|
| 147 |
+
Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN,
|
| 148 |
+
k=3) on real article prompts. Aggregate throughput scales with concurrency;
|
| 149 |
+
inter-token latency stays essentially flat under load.
|
| 150 |
+
|
| 151 |
+

|
| 152 |
+
|
| 153 |
+
| concurrent streams | aggregate output tok/s | TTFT p50 (ms) | ITL p50 (ms) |
|
| 154 |
+
|---|---|---|---|
|
| 155 |
+
| 4 | 435 | 56 | 7.7 |
|
| 156 |
+
| 8 | 750 | 187 | 8.2 |
|
| 157 |
+
| 16 | 993 | 303 | 8.5 |
|
| 158 |
+
|
| 159 |
+
Measured with [GuideLLM](https://github.com/vllm-project/guidellm) on an RTX 5090,
|
| 160 |
+
vLLM v0.25.1, real multi-kB article prompts. Single-stream decode is the 154-244
|
| 161 |
+
tok/s in the Speed table; this shows the same server saturating to ~1000 tok/s
|
| 162 |
+
aggregate at concurrency 16 with per-token latency near-constant, i.e. it batches
|
| 163 |
+
well for multi-user / parallel-agent workloads. Spec acceptance held over the run
|
| 164 |
+
(165k accepted / 502k drafted).
|
| 165 |
+
|
| 166 |
## Serving (vLLM)
|
| 167 |
|
| 168 |
The architecture is not yet in upstream vLLM
|