TOOL CALLING IS BROKEN !
#1
by omshunyaom13 - opened
Permutations Tried & Results
| # | Chat Template | Thinking Kwargs | Reasoning Parser | Tool Parser | Result |
|---|---|---|---|---|---|
| 1 | froggeric (original) | {"enable_thinking": false} | qwen3 | qwen3_coder | ❌ Loop at 1.0 penalty, no tool calls |
| 2 | froggeric | {"enable_thinking": false} | qwen3 | qwen3_xml | ❌ No tool calls, narrates markdown |
| 3 | enhanced.jinja (author's) | {"preserve_thinking": true} | none | qwen3_xml | ❌ Container crash (empty reasoning parser) |
| 4 | enhanced.jinja | {"preserve_thinking": true} | qwen3 | qwen3_xml | ❌ Model thinks, outputs content, no tool calls |
| 5 | native (model dir) | {"enable_thinking": true} | qwen3 | qwen3_coder | ❌ Model thinks, outputs content, no tool calls |
| 6 | native | {"enable_thinking": false} | qwen3 | qwen3_xml | ❌ Markdown code blocks, no tool calls |
| 7 | native | {"enable_thinking": true} | none | qwen3_xml | ❌ Model thinks, outputs content, no tool calls |
| 8 | enhanced.jinja | {"enable_thinking": true, "preserve_thinking": true} | none | qwen3_xml | ❌ Model thinks, outputs content, no tool calls |
Key Observations
| What Worked | What Didn't |
|---|---|
| ✅ Repetition loop fixed by penalty 1.1 (then 1.0) + template fix | ❌ No tool calls in ANY config |
| ✅ Reasoning field populated with native template + qwen3 parser | ❌ Model narrates tool calls as markdown/code blocks |
| ✅ bfloat16 dtype aligned (no cast warning) | ❌ XML tool calls never generated by model |
| ✅ Container stable on all configs | ❌ qwen3_xml parser never receives XML to parse |
Root Cause Identified
The AutoRound W4A16 quantization corrupted the model's JSON tool-calling heads → model now only emits XML tool calls. But:
- Native template prompts for XML but model ignores it
- Enhanced template prompts for XML but model ignores it
- Froggeric template prompts for XML (via tool_call_format: xml) but model ignores it
The model simply doesn't generate tool calls at all — it narrates them in natural language instead.
What Hasn't Been Tried
| Config | Why |
|---|---|
| Froggeric template + tool_call_format: xml + qwen3_xml | This is the 8010 working stack — closest to production |
| Disable thinking entirely (enable_thinking: false) + froggeric + qwen3_xml | May force tool calls without thinking interference |
| Lower temperature (0.3-0.4) | Reduce sampling variance, increase structure adherence |
| Explicit few-shot tool examples in system prompt | Stronger behavioral steering |
TOOL CALLS WORKING - NOT WORKING