Parsers verified: qwen3 + qwen3_coder
Browse files- README.md +5 -0
- k8s/qwen38-flash-next-nvfp4-sglang.yaml +12 -6
README.md
CHANGED
|
@@ -109,6 +109,11 @@ should leave ~33–35 GB.
|
|
| 109 |
reclaim storm). Avoid large concurrent downloads.
|
| 110 |
- Metrics come with the `sglang:*` prefix (including
|
| 111 |
`sglang:spec_accept_length` to watch the speculative decoding).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 112 |
|
| 113 |
## Sampling and thinking controls (from the official model card)
|
| 114 |
|
|
|
|
| 109 |
reclaim storm). Avoid large concurrent downloads.
|
| 110 |
- Metrics come with the `sglang:*` prefix (including
|
| 111 |
`sglang:spec_accept_length` to watch the speculative decoding).
|
| 112 |
+
- Parsers verified in production: `--reasoning-parser qwen3` and
|
| 113 |
+
`--tool-call-parser qwen3_coder` (the template emits XML tool calls).
|
| 114 |
+
Reasoning arrives in `reasoning_content`, tool calls come structured, and
|
| 115 |
+
sglang#36537 (thinking + qwen3_coder looping) did not reproduce on this
|
| 116 |
+
build — tested with thinking, tools, both combined, and streaming.
|
| 117 |
|
| 118 |
## Sampling and thinking controls (from the official model card)
|
| 119 |
|
k8s/qwen38-flash-next-nvfp4-sglang.yaml
CHANGED
|
@@ -401,10 +401,15 @@ spec:
|
|
| 401 |
- { name: CHUNKED_PREFILL_SIZE, value: "2048" }
|
| 402 |
# Covers 16 running with margin for the NEXTN tree shapes.
|
| 403 |
- { name: CUDA_GRAPH_MAX_BS, value: "32" }
|
| 404 |
-
#
|
| 405 |
-
#
|
| 406 |
-
|
| 407 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 408 |
# Bounds the host-side spike of CUDA graph capture. Without this,
|
| 409 |
# inductor spawns ~1 compile worker per core (20) and the
|
| 410 |
# 20-40 GB spike kills the cgroup 30-90 s after "Capture begin".
|
|
@@ -578,8 +583,9 @@ spec:
|
|
| 578 |
- { name: MAMBA_TRACK_INTERVAL, value: "64" }
|
| 579 |
- { name: CHUNKED_PREFILL_SIZE, value: "2048" }
|
| 580 |
- { name: CUDA_GRAPH_MAX_BS, value: "32" }
|
| 581 |
-
|
| 582 |
-
- { name:
|
|
|
|
| 583 |
# See the head for the rationale of these two.
|
| 584 |
- { name: TORCHINDUCTOR_COMPILE_THREADS, value: "4" }
|
| 585 |
- { name: MAX_JOBS, value: "1" }
|
|
|
|
| 401 |
- { name: CHUNKED_PREFILL_SIZE, value: "2048" }
|
| 402 |
# Covers 16 running with margin for the NEXTN tree shapes.
|
| 403 |
- { name: CUDA_GRAPH_MAX_BS, value: "32" }
|
| 404 |
+
# Verified in production: `qwen3` matches the conditional <think>
|
| 405 |
+
# detector (same as vLLM's `qwen3` parser) and the checkpoint's
|
| 406 |
+
# chat template emits XML tool calls (<function=/<parameter=), so
|
| 407 |
+
# the right tool parser is `qwen3_coder` — the JSON `qwen`/`qwen25`
|
| 408 |
+
# detectors would parse nothing. sglang#36537 (thinking +
|
| 409 |
+
# qwen3_coder looping) did NOT reproduce on this build: thinking,
|
| 410 |
+
# tools, thinking+tools and streaming all tested clean.
|
| 411 |
+
- { name: REASONING_PARSER, value: "qwen3" }
|
| 412 |
+
- { name: TOOL_CALL_PARSER, value: "qwen3_coder" }
|
| 413 |
# Bounds the host-side spike of CUDA graph capture. Without this,
|
| 414 |
# inductor spawns ~1 compile worker per core (20) and the
|
| 415 |
# 20-40 GB spike kills the cgroup 30-90 s after "Capture begin".
|
|
|
|
| 583 |
- { name: MAMBA_TRACK_INTERVAL, value: "64" }
|
| 584 |
- { name: CHUNKED_PREFILL_SIZE, value: "2048" }
|
| 585 |
- { name: CUDA_GRAPH_MAX_BS, value: "32" }
|
| 586 |
+
# See the parser note on the head (same values, verified).
|
| 587 |
+
- { name: REASONING_PARSER, value: "qwen3" }
|
| 588 |
+
- { name: TOOL_CALL_PARSER, value: "qwen3_coder" }
|
| 589 |
# See the head for the rationale of these two.
|
| 590 |
- { name: TORCHINDUCTOR_COMPILE_THREADS, value: "4" }
|
| 591 |
- { name: MAX_JOBS, value: "1" }
|