nrrso/Qwen3-Chat-Template-vLLM-Fixes

Enhanced Qwen 3.8 Chat Templates for vLLM

Drop-in Jinja chat templates for Qwen 3.8 (Qwen/Qwen3.8-27B and quantized variants like unsloth/Qwen3.8-27B-NVFP4). The goal is reliable agentic tool calling on vLLM without drifting away from the format the model was trained on.

Most "fixed" templates repair one thing and quietly break another by rewording instructions the model has already internalized. This repo takes a more conservative approach: keep the official template as the skeleton, patch the real bugs, and skip the clever rewrites.

There are two templates:

File	Use it when
qwen3.8-enhanced.jinja	The recommended baseline. Official Qwen 3.8 template plus tool-calling and history-rendering fixes.
qwen3.8-enhanced-extra.jinja	Everything in the baseline, plus extra tolerance for messy client-side history (malformed think tags, alternate field names). Renders byte-identical prompts for well-formed conversations.

⚠️ vLLM only. These templates rely on the from_json filter, which vLLM provides but plain Jinja2 and minijinja (llama.cpp, LM Studio, MLX) don't. If you need something engine-agnostic, use froggeric/Qwen-Fixed-Chat-Templates instead.

---

Quick start

vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --chat-template qwen3.8-enhanced.jinja \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --reasoning-parser qwen3

Notes on the flags:

• --tool-call-parser qwen3_xml matches the XML format these templates emit (<tool_call> → <function=name> → <parameter=x>). Stuck on an older vLLM build? Fall back to qwen3_coder.
• --reasoning-parser qwen3 is required. The generation prompt pre-fills <think>\n, so the model's output starts mid-thought with no opening tag. This parser handles that case and splits reasoning_content from content, which is also what the template needs fed back to it on later turns.
• qwen3_xml leans on your tool JSON schemas for type coercion, so keep those parameters schemas honest. Loose schemas mean arguments come back as strings.

---

What gets fixed

Ported from allanchan339/vLLM-Qwen3-3.5-3.6-chat-template-fix (qwen3.6-enhanced.jinja)

1. JSON-string tool arguments get parsed instead of crashing the server. The official template assumes tool_call.arguments is a mapping and dies with Can only get item pairs from a mapping when an OpenAI-style client echoes back stringified arguments, which most of them do. JSON strings are parsed here into proper <parameter> blocks, nested objects included, so multi-turn history always re-renders in the trained format.
2. Self-healing </think> closure. If echoed history contains a dangling <think> before a <tool_call>, the closing tag gets inserted where it belongs. Tool calls never end up trapped inside reasoning.
3. Inline <think>...</think> extraction. Some clients stuff reasoning into message.content instead of reasoning_content. Rather than double-wrapping it, the template pulls it out and re-renders it canonically.
4. tojson(ensure_ascii=False) for tool definitions and argument values, so CJK text stays readable instead of turning into \uXXXX escapes.
5. Extra agentic rules, appended after the official <IMPORTANT> block rather than replacing it: close </think> before tool calls, prefer tools over answering from memory, reuse ids with update_* tools, keep user-provided strings verbatim, write valid JSON for object and array parameters. There's a worked follow-up-update example too.

Merged from unsloth/Qwen3.8-27B-NVFP4's shipped template

6. developer role support. Leading system and developer messages are accepted and merged into a single system block. Harnesses like Codex won't work without this.
7. reasoning_effort: "high" aliased to xhigh instead of throwing. OpenAI proxies routinely send "high".
8. The no-user-query exception is gone. System-only and tool-continuation prompts from agentic harnesses no longer crash the render.
9. A clear exception when a tool call is missing its function name, rather than a cryptic failure three layers down.

Kept as-is from the official Qwen 3.8 template

• reasoning_effort steering (xhigh default, medium, low) with the official instruction wording, word for word.
• preserve_thinking defaults to true. Qwen 3.8 was trained with retained reasoning context; stripping it by default would fight the model.
• The official tool-instruction wording as the skeleton, since that is the exact phrasing the model was trained against.
• Official </tool_response><|im_end|> spacing and scalar argument rendering (true, null, 5 via tojson; strings verbatim).
• Always-closed <think> blocks, even when reasoning is empty. That's the trained format, empty or not.

---

What the -extra variant adds

Robustness ideas adapted from froggeric/Qwen-Fixed-Chat-Templates:

1. Tolerant think-tag extraction. Beyond the canonical tags, it recognizes <thinking>...</thinking>, </ think>, </think >, content that starts with </think>, and bare reasoning followed by </think>. These are the artifacts you get when a client echoes raw output from the prefilled <think>\n generation prompt. Every variant gets normalized back to a clean <think>...</think> on re-render.
2. message.thinking accepted as an alias of message.reasoning_content. A handful of clients use it; reasoning_content still wins when both are present.
3. preserve_reasoning accepted as an alias of preserve_thinking, matching llama.cpp's --reasoning-preserve naming. Takes precedence if both are passed.

All three only touch the history re-render path, so both templates produce byte-identical prompts for well-formed conversations. A/B them freely; a difference only shows up when a client sends messy history.

What was deliberately left out

Froggeric's template has more features than this one, and leaving them out was a deliberate trade-off. The guiding principle here is to stay on the model's training distribution, so the following features were skipped:

• Rewritten tool instructions forbidding conversational text before tool calls. The official template, and the training behind it, explicitly allows it.
• Tool-response mutation. Injecting "⚠️ SYSTEM WARNING" messages into tool results based on keyword heuristics false-positives easily (a legitimate result containing error: 0 trips it) and puts the whole conversation off-distribution.
• In-template payload truncation (max_tool_arg_chars / max_tool_response_chars). History truncation belongs in the client, where it can happen without silently corrupting replayed turns.
• Raw string-argument passthrough. Dumping unparsed JSON blobs into <function> blocks teaches the model to imitate malformed history. These templates parse instead.
• Omitting empty <think> blocks. The official format always emits the wrapper for preserved turns, so these templates do too.
• tool_call_format="json". Qwen 3.8 is trained on the XML format. Use qwen3_xml.

---

Template kwargs reference

Pass via chat_template_kwargs (vLLM) or apply_chat_template(...):

Kwarg	Default	Description
enable_thinking	true	false pre-fills <think>\n\n</think>\n\n to skip reasoning.
reasoning_effort	"xhigh"	"xhigh", "medium", "low"; "high" is aliased to "xhigh". Injects the official steering instruction into the system block.
preserve_thinking	true	false strips <think> blocks from assistant turns before the last real user query (saves tokens, breaks prefix-cache continuity for those turns).
preserve_reasoning	n/a	(extra only) Alias of preserve_thinking; takes precedence if both are set.
add_vision_id	false	Prefixes Picture N: / Video N: labels to vision inputs.

---

Credits

Role	Author
Qwen 3.8 models & official template	Alibaba Cloud (Qwen team)
Original enhanced-template lineage (3.5/3.6)	allanchan339
Developer-role / effort-alias / tool-name fixes	Unsloth
Robustness ideas adapted in the -extra variant	froggeric
Merge, testing & maintenance	nrrso

License

Apache-2.0, inherited from Qwen.