reality-interface commited on
Commit
82ae746
·
verified ·
1 Parent(s): e345b64

fix(template): align Qwen reasoning parser state

Browse files

- Make enable_thinking the authoritative reasoning-mode control used by the vLLM Qwen parser.\n- Remove message-level thinking toggles and prevent tools or repeated failures from silently changing parser state.\n- Add ultra-to-xhigh reasoning-effort compatibility while preserving the xhigh default.\n- Reject the unsafe legacy auto-disable option and update the README to use enable_thinking explicitly.\n- Preserve XML tools, OpenAI and Codex history, multimodal input, reasoning history, and agent error guidance.

Files changed (2) hide show
  1. README.md +2 -2
  2. chat_template.jinja +14 -37
README.md CHANGED
@@ -280,7 +280,6 @@ response = client.chat.completions.create(
280
  extra_body={
281
  "chat_template_kwargs": {
282
  "enable_thinking": False,
283
- "auto_disable_thinking_with_tools": True,
284
  }
285
  },
286
  )
@@ -317,9 +316,10 @@ Always use the current `chat_template.jinja`. It is intentionally separate from
317
  | `enable_thinking` | `true` | Enable or disable thinking output |
318
  | `preserve_thinking` | `true` | Preserve earlier thinking in conversation history |
319
  | `tool_call_format` | `xml` | Select `xml` or `json` tool-call output |
320
- | `auto_disable_thinking_with_tools` | `false` | Disable thinking automatically when tools are present |
321
  | `continue_final_message` | `false` | Continue an assistant message instead of opening a new turn |
322
 
 
 
323
  ## Client integrations
324
 
325
  The model and template support developer/system instructions, reasoning history, OpenAI-style tools, parallel calls, tool results, and multimodal placeholders. The serving layer translates the native format for each client.
 
280
  extra_body={
281
  "chat_template_kwargs": {
282
  "enable_thinking": False,
 
283
  }
284
  },
285
  )
 
316
  | `enable_thinking` | `true` | Enable or disable thinking output |
317
  | `preserve_thinking` | `true` | Preserve earlier thinking in conversation history |
318
  | `tool_call_format` | `xml` | Select `xml` or `json` tool-call output |
 
319
  | `continue_final_message` | `false` | Continue an assistant message instead of opening a new turn |
320
 
321
+ Use `enable_thinking=false` explicitly for non-thinking tool calls. The legacy `auto_disable_thinking_with_tools` option is rejected because changing the prompt mode without changing vLLM's Qwen reasoning-parser state would misclassify reasoning as normal content.
322
+
323
  ## Client integrations
324
 
325
  The model and template support developer/system instructions, reasoning history, OpenAI-style tools, parallel calls, tool results, and multimodal placeholders. The serving layer translates the native format for each client.
chat_template.jinja CHANGED
@@ -1,8 +1,8 @@
1
- {%- set template_version = "qwen3.8-agentic-nvfp4-final" -%}
2
  {%- set requested_tool_format = tool_call_format if tool_call_format is defined and tool_call_format is not none else "xml" -%}
3
  {%- set add_vision_id = add_vision_id if add_vision_id is defined else false -%}
4
- {%- set thinking_was_explicit = (enable_thinking is defined) or (thinking is defined) -%}
5
- {%- set enable_thinking = enable_thinking if enable_thinking is defined else (thinking if thinking is defined else true) -%}
6
  {%- if preserve_reasoning is defined and preserve_reasoning is not none -%}
7
  {%- set preserve_history_thinking = preserve_reasoning -%}
8
  {%- elif preserve_thinking is defined and preserve_thinking is not none -%}
@@ -10,7 +10,6 @@
10
  {%- else -%}
11
  {%- set preserve_history_thinking = true -%}
12
  {%- endif -%}
13
- {%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false -%}
14
  {%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 -%}
15
  {%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 -%}
16
  {%- set agent_error_hardening = agent_error_hardening if agent_error_hardening is defined else false -%}
@@ -18,8 +17,11 @@
18
  {%- set image_counter = namespace(value=0) -%}
19
  {%- set video_counter = namespace(value=0) -%}
20
  {%- set state = namespace(thinking=enable_thinking, previous_role="", consecutive_failures=0) -%}
21
- {%- if auto_disable_thinking_with_tools and has_tools -%}
22
- {%- set state.thinking = false -%}
 
 
 
23
  {%- endif -%}
24
 
25
  {%- macro render_json_arguments(value) -%}
@@ -107,7 +109,7 @@
107
  {%- set effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else "xhigh" -%}
108
  {%- set effort = "" -%}
109
  {%- if effort_raw in ("none", "off") -%}
110
- {# Match current vLLM semantics: an explicit enable_thinking/thinking kwarg wins. #}
111
  {%- if not thinking_was_explicit -%}
112
  {%- set state.thinking = false -%}
113
  {%- endif -%}
@@ -115,27 +117,16 @@
115
  {%- set effort = "low" -%}
116
  {%- elif effort_raw == "medium" -%}
117
  {%- set effort = "medium" -%}
118
- {%- elif effort_raw in ("high", "xhigh", "max", "ultracode", "extreme") -%}
119
  {%- set effort = "xhigh" -%}
120
  {%- elif effort_raw -%}
121
  {# Unknown future effort names keep the model's normal thinking behavior and do not hard-fail rendering. #}
122
  {%- set effort = "" -%}
123
  {%- endif -%}
124
 
125
- {# Resolve explicit think on/off controls before building the system prefix. Only trusted
126
- conversation roles are scanned; tool outputs cannot flip reasoning mode. A deliberate
127
- mode change may invalidate the prefix cache for that turn, which is preferable to a
128
- contradictory system prefix and generation scaffold. #}
129
- {%- for control_message in messages -%}
130
- {%- if control_message.role == "system" or control_message.role == "developer" or control_message.role == "user" -%}
131
- {%- set control_text = render_content(control_message.content, false, false) -%}
132
- {%- if "<|think_off|>" in control_text -%}
133
- {%- set state.thinking = false -%}
134
- {%- elif "<|think_on|>" in control_text -%}
135
- {%- set state.thinking = true -%}
136
- {%- endif -%}
137
- {%- endif -%}
138
- {%- endfor -%}
139
 
140
  {# Keep the generation grammar Qwen-native and stable by default. Current vLLM
141
  normalizes OpenAI wire-format argument strings into mappings before rendering.
@@ -165,12 +156,6 @@
165
  {%- set system_acc = namespace(text="") -%}
166
  {%- for message in messages[:head.count] -%}
167
  {%- set part = render_content(message.content, false, true) | trim -%}
168
- {%- if "<|think_off|>" in part -%}
169
- {%- set state.thinking = false -%}
170
- {%- elif "<|think_on|>" in part -%}
171
- {%- set state.thinking = true -%}
172
- {%- endif -%}
173
- {%- set part = part | replace("<|think_off|>", "") | replace("<|think_on|>", "") | trim -%}
174
  {%- if part -%}
175
  {%- if system_acc.text -%}
176
  {%- set system_acc.text = system_acc.text ~ "\n\n" ~ part -%}
@@ -250,14 +235,6 @@
250
  {%- set content = rendered_content if tool_like else (rendered_content | trim) -%}
251
 
252
  {%- set wrapped_tool_response = role == "user" and content[:15] == "<tool_response>" and content[-16:] == "</tool_response>" -%}
253
- {%- if system_like or (role == "user" and not wrapped_tool_response) -%}
254
- {%- if "<|think_off|>" in content -%}
255
- {%- set state.thinking = false -%}
256
- {%- elif "<|think_on|>" in content -%}
257
- {%- set state.thinking = true -%}
258
- {%- endif -%}
259
- {%- set content = content | replace("<|think_off|>", "") | replace("<|think_on|>", "") | trim -%}
260
- {%- endif -%}
261
 
262
  {%- if system_like -%}
263
  {{- "<|im_start|>system\n" ~ content ~ "<|im_end|>\n" -}}
@@ -429,7 +406,7 @@
429
  {%- set continuing_final = continue_final_message is defined and continue_final_message and conversation and conversation[-1].role == "assistant" -%}
430
  {%- if add_generation_prompt and not continuing_final -%}
431
  {{- "<|im_start|>assistant\n" -}}
432
- {%- if state.thinking and (not agent_error_hardening or state.consecutive_failures < 2) -%}
433
  {{- "<think>\n" -}}
434
  {%- else -%}
435
  {# Qwen's own templates use this closed empty think marker to force non-thinking generation. #}
 
1
+ {%- set template_version = "qwen3.8-agentic-nvfp4-v1-stable" -%}
2
  {%- set requested_tool_format = tool_call_format if tool_call_format is defined and tool_call_format is not none else "xml" -%}
3
  {%- set add_vision_id = add_vision_id if add_vision_id is defined else false -%}
4
+ {%- set thinking_was_explicit = enable_thinking is defined -%}
5
+ {%- set enable_thinking = enable_thinking if enable_thinking is defined else true -%}
6
  {%- if preserve_reasoning is defined and preserve_reasoning is not none -%}
7
  {%- set preserve_history_thinking = preserve_reasoning -%}
8
  {%- elif preserve_thinking is defined and preserve_thinking is not none -%}
 
10
  {%- else -%}
11
  {%- set preserve_history_thinking = true -%}
12
  {%- endif -%}
 
13
  {%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 -%}
14
  {%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 -%}
15
  {%- set agent_error_hardening = agent_error_hardening if agent_error_hardening is defined else false -%}
 
17
  {%- set image_counter = namespace(value=0) -%}
18
  {%- set video_counter = namespace(value=0) -%}
19
  {%- set state = namespace(thinking=enable_thinking, previous_role="", consecutive_failures=0) -%}
20
+ {# vLLM's Qwen parser reads only chat_template_kwargs.enable_thinking. A template-only
21
+ auto-disable would desynchronize parser and prompt state, so reject the legacy knob
22
+ instead of silently producing incorrectly parsed reasoning. #}
23
+ {%- if auto_disable_thinking_with_tools is defined and auto_disable_thinking_with_tools -%}
24
+ {{- raise_exception("auto_disable_thinking_with_tools is unsafe with the Qwen vLLM reasoning parser. Set chat_template_kwargs.enable_thinking=false explicitly instead.") -}}
25
  {%- endif -%}
26
 
27
  {%- macro render_json_arguments(value) -%}
 
109
  {%- set effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else "xhigh" -%}
110
  {%- set effort = "" -%}
111
  {%- if effort_raw in ("none", "off") -%}
112
+ {# Match current vLLM semantics: an explicit enable_thinking kwarg wins. #}
113
  {%- if not thinking_was_explicit -%}
114
  {%- set state.thinking = false -%}
115
  {%- endif -%}
 
117
  {%- set effort = "low" -%}
118
  {%- elif effort_raw == "medium" -%}
119
  {%- set effort = "medium" -%}
120
+ {%- elif effort_raw in ("high", "xhigh", "max", "ultra", "ultracode", "extreme") -%}
121
  {%- set effort = "xhigh" -%}
122
  {%- elif effort_raw -%}
123
  {# Unknown future effort names keep the model's normal thinking behavior and do not hard-fail rendering. #}
124
  {%- set effort = "" -%}
125
  {%- endif -%}
126
 
127
+ {# `enable_thinking` is the single source of truth for generation mode. Do not
128
+ mutate reasoning state from magic strings embedded in conversation content: vLLM's
129
+ Qwen parser cannot see those mutations and would parse the output in the wrong state. #}
 
 
 
 
 
 
 
 
 
 
 
130
 
131
  {# Keep the generation grammar Qwen-native and stable by default. Current vLLM
132
  normalizes OpenAI wire-format argument strings into mappings before rendering.
 
156
  {%- set system_acc = namespace(text="") -%}
157
  {%- for message in messages[:head.count] -%}
158
  {%- set part = render_content(message.content, false, true) | trim -%}
 
 
 
 
 
 
159
  {%- if part -%}
160
  {%- if system_acc.text -%}
161
  {%- set system_acc.text = system_acc.text ~ "\n\n" ~ part -%}
 
235
  {%- set content = rendered_content if tool_like else (rendered_content | trim) -%}
236
 
237
  {%- set wrapped_tool_response = role == "user" and content[:15] == "<tool_response>" and content[-16:] == "</tool_response>" -%}
 
 
 
 
 
 
 
 
238
 
239
  {%- if system_like -%}
240
  {{- "<|im_start|>system\n" ~ content ~ "<|im_end|>\n" -}}
 
406
  {%- set continuing_final = continue_final_message is defined and continue_final_message and conversation and conversation[-1].role == "assistant" -%}
407
  {%- if add_generation_prompt and not continuing_final -%}
408
  {{- "<|im_start|>assistant\n" -}}
409
+ {%- if state.thinking -%}
410
  {{- "<think>\n" -}}
411
  {%- else -%}
412
  {# Qwen's own templates use this closed empty think marker to force non-thinking generation. #}