froggeric

/

Qwen-Fixed-Chat-Templates

like 952

Human Generated Summary

I understand you wanted to fix the KV-cache issue "once and for all" by stripping the extra newlines in the reasoning-bypass block. The catch is that this change is exactly what breaks "no-think mode" under streaming: the model still emits a <think>…</think> block, and streaming clients (e.g. CherryStudio, both Desktop and Mobile) end up showing the answer twice. Non-streaming is clean, which is probably why it slipped through.

For my use I'd prefer the canonical form. Prefix caching never produced a single hit in my setup anyway: Qwopus is a hybrid Mamba/GDN model, so vLLM's prefix caching for it is experimental, and I measured a 0% hit-rate regardless of the template. So the newline optimization buys me nothing, while the doubling is a real problem.

If you'd still rather keep the newline-stripped form, it would make sense to at least note this streaming caveat in the README.

AI Generated Summary

In the unified template (reproduced on v20, v19, v18), the non-thinking branch of the
add_generation_prompt block emits <think>\n</think>\n instead of the canonical
<think>\n\n</think>\n\n. On a Qwen3.6 finetune served by vLLM with --reasoning-parser qwen3,
this non-canonical prefix causes the model to ignore the "thinking suppressed" signal and emit a
<think>…</think>… block anyway. Under streaming, vLLM's streaming reasoning parser does not
strip that block, so the model's reasoning draft + a stray </think> + the final answer all land in
content — the user sees the answer printed twice. Non-streaming is unaffected (the batch
reasoning parser strips it). Restoring the canonical blank-line prefix is a 2-line change that fully
fixes it.

Environment

• Template: froggeric/Qwen-Fixed-Chat-Templates, root chat_template.jinja, v20 (also v19, v18).
• Model: Jackrong/Qwopus3.6-27B-v2-FP8 (Opus-trace finetune of Qwen3.6-27B), FP8, vision.
• Engine: vLLM 0.22.1rc1 (dev), --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml, native MTP speculative decoding.
• Trigger: non-thinking (chat_template_kwargs:{enable_thinking:false} or <|think_off|>) + stream:true, temp 1.0.

Symptom (a single streamed content, non-thinking mode)

Солнце село за холмы, окрашивая небо ...               ← draft
</think>

Солнце скрылось за холмами, окрашивая небо ...          ← final answer

Two complete answers separated by a stray </think>, all inside content.

Root cause (controlled, single variable)

Rendering the same non-thinking request with v20 vs the model's own stock template
(tokenizer_config.json), the only difference is the add_generation_prompt prefix:

v20  : <|im_start|>assistant\n<think>\n</think>\n
stock: <|im_start|>assistant\n<think>\n\n</think>\n\n

Changing only v20's two non-thinking prefix emit lines to the canonical blank-line form:

   {%- if not ns_state.thinking %}
-      {{- '<think>\n</think>\n' }}
+      {{- '<think>\n\n</think>\n\n' }}
   {%- elif ns2.consecutive_failures >= 2 %}
-      {{- '<think>\n</think>\n' }}
+      {{- '<think>\n\n</think>\n\n' }}

eliminates the duplication: streaming non-thinking went from ~5/6 runs duplicated (vanilla v20)
to 0/24 (canonical), across the enable_thinking kwarg, <|think_off|> in a user message, and
<|think_off|> in a system message. Thinking-on and tool-call paths are unchanged.

Why it is streaming-only

vLLM's non-streaming path runs the batch reasoning extractor over the full output and correctly
splits a stray <think>…</think>; the streaming parser, with thinking nominally disabled, does
not — so the stray block stays in content. The template fix prevents the model from emitting the
stray block in the first place, so both paths come out clean.

Suggested fix

Use the canonical <think>\n\n</think>\n\n for the non-thinking branch of add_generation_prompt.
This is the exact form base Qwen3 / the model's stock template use to signal "reasoning skipped,"
and the finetune honors it reliably.

We suspect the single-newline form was a deliberate KV-cache / prefix-stability optimization (cf. the
v18 changelog, "reasoning bypass without stacking newlines"). If so, you likely don't need to
revert the history whitespace normalization — only the live generation-prompt prefix needs the
canonical blank lines. Since that prefix is the trailing segment of the prompt, restoring it does not
affect prefix-cache hits on the (earlier) conversation history.

Minimal repro (vLLM, OpenAI API)

POST /v1/chat/completions
{ "model": "<model>", "temperature": 1, "stream": true,
  "chat_template_kwargs": {"enable_thinking": false},
  "messages": [{"role": "user", "content": "translate a few sentences into Russian"}] }

Run ~6×; with vanilla v20 several responses contain a duplicated answer plus a stray </think> in
the streamed content. With the canonical prefix, all are clean.