Tool calling broken in Ollama / GGUF: raw JSON returned in content without <tool_call> tags

#2
by a-us - opened

Hi! Thanks for sharing this C# fine-tune of Qwen2.5-Coder.

I tested Qwen2.5-Coder-7B-CSharp-QLoRA-GGUF:Q5_K_M in Ollama for agentic workflows in Visual Studio (Copilot / IDE extensions) and noticed that tool calling (tool_calls) does not work properly, even though ollama show reports tools under Capabilities.

When sending a standard OpenAI/Ollama tool-calling payload (e.g., asking to call a read_file tool), the model outputs a raw JSON object inside the plain text "content" field instead of wrapping it in Qwen's expected ... tags:

{
  "model": "hf.co/testingusera1/Qwen2.5-Coder-7B-CSharp-QLoRA-GGUF:Q5_K_M",
  "message": {
    "role": "assistant",
    "content": "{\"name\": \"read_file\", \"arguments\": {\"path\": \"ValidationJob.cs\"}}"
  },
  "done": true
}

Because the tags are missing, Ollama's parser fails to populate the "tool_calls" array, so IDE clients treat the response as plain text and cannot execute the tool.

Hi Alexey,

Thanks for the detailed report — and sorry for the trouble. This is a known limitation of the current weights, here is what's going on:

Root cause. This is a QLoRA adapter fine-tune (not a full fine-tune), built mainly for Russian-language C# coding assistance. Agentic data was only 8.5% of the training mix (7.9k examples from NousResearch/hermes-function-calling-v1). More importantly, the training pipeline used a flat "instruction → response" format without a system role: the ...<-(here should be tools block, but HF consumed it...) block was embedded inside the user turn. Qwen's native tool-calling template (which Ollama uses) expects the tool schemas in the system message instead. So the adapter never really reinforced Qwen's native protocol — it saw tool-call-like data only in a different context, and at this mixture weight it partially washed out what the base model already knew.

The Capabilities: tools flag from ollama show comes from the chat template embedded in the GGUF, not from the model's behavior — so the flag is there, but the model itself doesn't reliably emit the tags.

What I'd suggest for IDE/agentic use:

For tool calling in Ollama / VS extensions, the safest option is the official Qwen/Qwen2.5-Coder-7B-Instruct GGUF. My own benchmarks actually showed this fine-tune is at parity with the base model on MultiPL-E (humaneval-cs / mbpp-cs), so you lose essentially nothing there and get working tool_calls.
This repo's strength is Russian-language C# Q&A and code writing — on a set of 5 Russian coding tasks (bug fixes, writing an app, LINQ) the fine-tune produced compiling answers 5/5 vs 3/5 for the base model, and noticeably more compact answers. If you don't need tool calling, it works fine as a chat model.

p.s.: sorry if i made any mistakes in answer - I don't realy good in English, so used AI to translate your answer and my answer, so chapter of senses may be lost

Спасибо за ответ, можно и по-русски :) Я модельку как раз и скачал для C# и русского языка. Просто первый раз пробую агента настроить в VS, и с этой моделью не получилось, хотя там писалось что туллинг поддерживается. Теперь понял откуда там туллинг в поддержке написан, спасибо за разъяснения.

Вообще, я пытался сохранить умение использовать инструменты, но, похоже, это умение модели "поехало", т.к. сместился акцент модели на написание C# кода и понимание русского языка :/
(ну и, как и написано в карточке - выборка tool use была <10% от всего обучающего корпуса, это умение деградировало на фоне целевых навыков)

Я как-нибудь займусь этим вопросом, но у меня и это обучение заняло ~неделю(не мог нормально использовать ПК, т.к. вся видеопамять и ОЗУ были забиты), поэтому ничего не могу обещать

Ясно, все равно спасибо :)

a-us changed discussion status to closed

Sign up or log in to comment