yzhwang commited on
Commit
36bb7bf
·
verified ·
1 Parent(s): de0e2fa

Clarify that 64k context and 8-9 streams are not simultaneous

Browse files
Files changed (1) hide show
  1. README.md +10 -6
README.md CHANGED
@@ -111,12 +111,16 @@ an answer is always produced: see
111
  `gcc --version` still succeeds, and the failure surfaces ~40 s in as a `gcc`
112
  `CalledProcessError` inside `cuda_graph_runner.py` — which reads like a runtime bug and is not.
113
  Preflight in [`sglang/INSTALL.md`](sglang/INSTALL.md#requirements).
114
- - **24 GB VRAM** for the shipped defaults (65,536-token context, ~8–9 concurrent streams) with a
115
- ~10.15 GB model. Note the stream ceiling: `MAMBA_RATIO=0.3` sizes the recurrent-state pool, which
116
- clamps `max_running_requests` to 8–9 on a 24 GB card, so the `12`/`16` entries in the default
117
- `CUDA_GRAPH_BS` are **dropped and never captured**. To serve more streams raise
118
- `MAXREQ`/`MAXMAMBA` with `MEM` — the throughput recipe is in the
119
- [model card](https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2#verified-configurations).
 
 
 
 
120
  16 GB should fit at a reduced context — the cookbook has a recipe, but we have not run it.
121
 
122
  ## Changelog
 
111
  `gcc --version` still succeeds, and the failure surfaces ~40 s in as a `gcc`
112
  `CalledProcessError` inside `cuda_graph_runner.py` — which reads like a runtime bug and is not.
113
  Preflight in [`sglang/INSTALL.md`](sglang/INSTALL.md#requirements).
114
+ - **24 GB VRAM** for the shipped defaults (65,536-token context, ~8–9 concurrent streams *at short
115
+ prompts*) with a ~10.15 GB model. Those two figures are **not simultaneous**: the default pool is
116
+ 68,686 tokens, which is one full-length 64k request or ~8 requests of ~8k. Note also the stream
117
+ ceiling: `MAMBA_RATIO=0.3` sizes the recurrent-state pool, which clamps `max_running_requests` to
118
+ 8–9 on a 24 GB card, so the `12`/`16` entries in the default `CUDA_GRAPH_BS` are **dropped and
119
+ never captured**. To serve more streams raise `MAXREQ`/`MAXMAMBA` with `MEM` — the throughput
120
+ recipe is in the
121
+ [model card](https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2#verified-configurations); for
122
+ long context (128k measured on a 24 GB card) see
123
+ [By VRAM](sglang/INSTALL.md#by-vram).
124
  16 GB should fit at a reduced context — the cookbook has a recipe, but we have not run it.
125
 
126
  ## Changelog