froggeric

/

Qwen-Fixed-Chat-Templates

like 1.16k

So I asked my local LLM thru CC to take a look at the template because I'm only getting 30-40 tok/s on TP=2 vllm.

This is what it had to say:

Thanks for pasting the chat template. This reveals a critical insight about your performance issue.

Look at this section of the template (lines ~68-100):

{%- if tools and tools is iterable and tools is not mapping %}
{%- set ns_flags.has_tools = true %}
{{- '▁system\n' }}
{{- "# Tools\n\nYou have access to the following functions:\n\n" }}
{%- for tool in tools %}
... builds tool signature ...
{%- endfor %}
{{- "\n" }}
{{- '\n\nIf you choose to call a function ONLY reply in the following format...' }}
{%- if has_system and system_content %}
{{- '\n\n' + system_content }}
{%- endif %}
{{- '▁\n' }}

When tools are present (which they always are with Claude Code), the chat template:

Builds a huge tool definitions block — every single tool (Bash, Read, Edit, Write, Agent, AskUserQuestion, etc.) with all their parameters gets serialized into the system message

Then appends your actual system prompt after the tool definitions

So the ~40k tokens isn't just the Claude Code system prompt — it's the Claude Code system prompt PLUS all the tool definitions serialized by the chat template. This is why you see 25 tok/s through the proxy but 100+ tok/s with a simpler UI — the simpler UI probably doesn't send any tools, so the template doesn't generate that massive block.

This is where system_override helps dramatically. If you override the system prompt to just "You are Qwen, created by Alibaba Cloud. You are a helpful assistant.", you eliminate the ~38k tokens of Claude Code instructions. The tool definitions from the chat template will still be there (since vLLM injects them), but the total system prompt drops significantly.

and also

In the tool call format example, and when the model generates tool calls, it's producing these XML-like tags which are being interpreted as actual tool calls by the proxy or Claude Code harness.

But looking at the actual error the user posted—Error: No such tool available: example_function_name—the real problem is that the model is copying the literal example function name from the template instead of using a real tool. This suggests the model isn't properly grounding itself in the actual available tools when generating responses.

The issue likely stems from the template being too verbose or confusing, combined with a massive system prompt that overwhelms the model. The thinking tags might be exacerbating this by adding noise. Reducing the system prompt with system_override would help the model focus on the actual tool definitions rather than getting distracted by the lengthy instructions.

Does this make sense to you?

Seems to me that Qwen just can't cope, and also not parsing JSON strings/arrays properly (because of the template/format/something else?).

I'm not an expert in templating so can't really offer any suggestions, and I can't compete with the AI's interpretation (which could also be hallucations, LOL).