v16 failes to parse JSON correctly - Tool Call error
Getting an agent crash after some time with OpenHands.
Qwen3.6-27B iQ4 load with llama.cpp (turboquant fork)
Given parameters: --jinja and --chat-template-file /models/chat_template.jinja
Noteable that it doesnt happen when using no template file at all (so it uses it baked in file in the GGUF)
I get errors like:
Error: Error validating tool 'file_editor': Unterminated string starting at: line 1 column 107 (char 106). Arguments: unparseable JSON
Claude describes the error cause as following:
The core issue is a JSON parsing failure that occurs during tool invocation. This is not related to the LLM's reasoning (Thinking) process, but rather a crash originating from the llama.cpp server itself.
Error Trace:
Failed to parse tool call arguments as JSON: [json.exception.parse_error.101]
parse error at line 1, column 16841: syntax error while parsing value - invalid string: missing closing quote
What Happens:
The model generates a Tool Call where the JSON argument field contains an unterminated string. This occurs because the context code passed to the model by OpenHands (specifically runner_project.py) contains Python Triple-Quotes ("""). The model incorrectly includes these triple-quotes within the JSON string of the Tool Call. Since a double quote (") inside a JSON string must be properly escaped, the triple-quotes cause the JSON parser to fail, as they prematurely close the string field.
Update: Still happening with total plain config + official llama.cpp:server-b9159 Build (released 6 hours ago) + Qwen3.6 Template v16
Since I changed nothing else (no special parameters or something) it must come from the template.
Thank you for your work & effort.
Update2: Same crash with no chat-template given aswell now...
Seems like the baked-in chat template is broken aswell with JSON Escaping.
Im switching to another coding provider now but hopefully this can be worked out.
Might be an idea to let Claude develop a "landmap" of the settings in the chat template you combined from different sources. What do you think?
So any future change to the template can better be pinned to a specific change.
Also I experienced Claude is really not good with unknown Code like chat templates or special start paramters for llama.cpp, its missing context it seems. For example it recommends non-working "--fa" instead of "--fa on".
It would be helpful to have a lookup table or something like that where you once map everything that every parameter/function in the chat template does and can input this into later codingagents or Claude or something each time you need it.
So when someone opens an issue you could automatically pipe this into Claude or similar and it can lookup the landmark file aswell as the chattemplate to make sense out of it without burning through many tokens each time someone experience a new issue.
I think I have finally solved it in v19. So far it has been flawless in 3 long agentic tests in a row. Previously, I had it happen in around 80% of my sessions.
This has been a tough one to crack. To fix it I had to resort to better prompt engineering:
<IMPORTANT>
Reminder:
- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.
- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.
- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.
- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after closing </think>. Do NOT output any conversational text before the tool call.
- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.
- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.
- If you have gathered all necessary data and do not need to call a tool, answer the question like normal and provide your final response to the user IMMEDIATELY after closing </think>.
</IMPORTANT>
It helped a bit, but did not solve it. What I think finally did it, was a complete rewrite of the KV cache handling, by setting preserve_thinking to true as default, and abolishing the empty think injection, which was poisoning the model's in-context learning.