NullSense commited on
Commit
f8ec33a
·
verified ·
1 Parent(s): 8290c82

Add Performance-under-load section (concurrency sweep, GuideLLM real-data)

Browse files
Files changed (1) hide show
  1. README.md +22 -0
README.md CHANGED
@@ -141,6 +141,28 @@ context length, batch size, and backend. Notes from the serving campaign:
141
  `VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a
142
  further +13% in my follow-up runs.
143
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
144
  ## Serving (vLLM)
145
 
146
  The architecture is not yet in upstream vLLM
 
141
  `VLLM_ATTENTION_BACKEND=TRITON_ATTN` lifted spec-off short-form decode a
142
  further +13% in my follow-up runs.
143
 
144
+
145
+ ## Performance under load (GuideLLM, real articles, eagle3 head)
146
+
147
+ Concurrency sweep of the production config (FP8 + EAGLE-3 draft, TRITON_ATTN,
148
+ k=3) on real article prompts. Aggregate throughput scales with concurrency;
149
+ inter-token latency stays essentially flat under load.
150
+
151
+ ![Concurrency scaling](https://huggingface.co/NullSense/Nanbeige4.2-3B-FP8-Dynamic/resolve/main/assets/chart_concurrency.png)
152
+
153
+ | concurrent streams | aggregate output tok/s | TTFT p50 (ms) | ITL p50 (ms) |
154
+ |---|---|---|---|
155
+ | 4 | 435 | 56 | 7.7 |
156
+ | 8 | 750 | 187 | 8.2 |
157
+ | 16 | 993 | 303 | 8.5 |
158
+
159
+ Measured with [GuideLLM](https://github.com/vllm-project/guidellm) on an RTX 5090,
160
+ vLLM v0.25.1, real multi-kB article prompts. Single-stream decode is the 154-244
161
+ tok/s in the Speed table; this shows the same server saturating to ~1000 tok/s
162
+ aggregate at concurrency 16 with per-token latency near-constant, i.e. it batches
163
+ well for multi-user / parallel-agent workloads. Spec acceptance held over the run
164
+ (165k accepted / 502k drafted).
165
+
166
  ## Serving (vLLM)
167
 
168
  The architecture is not yet in upstream vLLM