Clarify that 64k context and 8-9 streams are not simultaneous
Browse files
README.md
CHANGED
|
@@ -111,12 +111,16 @@ an answer is always produced: see
|
|
| 111 |
`gcc --version` still succeeds, and the failure surfaces ~40 s in as a `gcc`
|
| 112 |
`CalledProcessError` inside `cuda_graph_runner.py` — which reads like a runtime bug and is not.
|
| 113 |
Preflight in [`sglang/INSTALL.md`](sglang/INSTALL.md#requirements).
|
| 114 |
-
- **24 GB VRAM** for the shipped defaults (65,536-token context, ~8–9 concurrent streams
|
| 115 |
-
~10.15 GB model.
|
| 116 |
-
|
| 117 |
-
`
|
| 118 |
-
`
|
| 119 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 120 |
16 GB should fit at a reduced context — the cookbook has a recipe, but we have not run it.
|
| 121 |
|
| 122 |
## Changelog
|
|
|
|
| 111 |
`gcc --version` still succeeds, and the failure surfaces ~40 s in as a `gcc`
|
| 112 |
`CalledProcessError` inside `cuda_graph_runner.py` — which reads like a runtime bug and is not.
|
| 113 |
Preflight in [`sglang/INSTALL.md`](sglang/INSTALL.md#requirements).
|
| 114 |
+
- **24 GB VRAM** for the shipped defaults (65,536-token context, ~8–9 concurrent streams *at short
|
| 115 |
+
prompts*) with a ~10.15 GB model. Those two figures are **not simultaneous**: the default pool is
|
| 116 |
+
68,686 tokens, which is one full-length 64k request or ~8 requests of ~8k. Note also the stream
|
| 117 |
+
ceiling: `MAMBA_RATIO=0.3` sizes the recurrent-state pool, which clamps `max_running_requests` to
|
| 118 |
+
8–9 on a 24 GB card, so the `12`/`16` entries in the default `CUDA_GRAPH_BS` are **dropped and
|
| 119 |
+
never captured**. To serve more streams raise `MAXREQ`/`MAXMAMBA` with `MEM` — the throughput
|
| 120 |
+
recipe is in the
|
| 121 |
+
[model card](https://huggingface.co/EschaLabs/Qwen3.8-27B-Escha-W2#verified-configurations); for
|
| 122 |
+
long context (128k measured on a 24 GB card) see
|
| 123 |
+
[By VRAM](sglang/INSTALL.md#by-vram).
|
| 124 |
16 GB should fit at a reduced context — the cookbook has a recipe, but we have not run it.
|
| 125 |
|
| 126 |
## Changelog
|