Runtime-adaptive tool-call format: XML on vLLM, Hermes JSON on llama.cpp — no config needed
Follow-up to the PR 45 revert — both ecosystems, one template
(Supersedes #52, which could not merge cleanly: chat_template_oneline.txt is a single line, so once main moves it can never auto-merge — this PR is based directly on current main/v21.2.)
The revert was the right call for vLLM (qwen3_coder parser expects XML), but it leaves llama.cpp/ik_llama users with silently-dropped tool calls (their Hermes-style parsers cannot parse the XML body — the <tool_call> opener is consumed by the grammar trigger and the call is dumped into content). Neither format works everywhere. This PR makes the template detect its own runtime and emit the format that engine's parser actually understands:
{%- set tool_call_format = tool_call_format if tool_call_format is defined else ('xml' if cycler is defined else 'json') %}
How detection works: Python Jinja2 (vLLM, SGLang, transformers — all use ImmutableSandboxedEnvironment) defines the built-in globals cycler/joiner/lipsum. minja and llama.cpp's C++ Jinja runtime do not (verified at source: common/jinja/value.cpp registers only raise_exception, namespace, strftime_now). cycler is defined is therefore a zero-cost, side-effect-free engine fingerprint:
- Python Jinja2 → XML (matches
--tool-call-parser qwen3_coder; identical output to current main, including the v21.2<IMPORTANT>wording) - minja/C++ → Hermes JSON (matches llama.cpp-family parsers)
- Explicit override for off-convention setups (e.g. vLLM with the
hermesparser):chat_template_kwargs: {"tool_call_format": "json"}/ llama.cpp--chat-template-kwargs
A nice property on newer llama.cpp: it derives its tool parser by rendering the template with its own runtime, so the detected branch and the learned parser agree by construction.
Both the instruction block and the assistant-history rendering are format-conditional; the XML branch is your current main verbatim (zero behavior change for vLLM users), and max_tool_arg_chars truncation works in both branches.
Verified: dual-runtime render matrix (Jinja2 sandboxed env → XML; same env with cycler/joiner/lipsum removed to emulate minja → JSON; kwarg overrides both directions; truncation both formats; v21.2 fixes intact in both branches) + live end-to-end on ik_llama (auto-selected JSON, structured tool_calls with correct arguments).
Small unrelated note: main's template_version still reads qwen3.6-froggeric-v20 while the README announces v21.2 — left untouched to keep the diff focused.
Thank you for the PR! As discussed in #52, while the runtime-adaptive cycler trick is incredibly clever, we cannot merge it. Forcing Hermes JSON on the entire C++ ecosystem degrades the reasoning capabilities of Qwen models (which are heavily trained on XML) for the vast majority of users who rely on raw text parsers downstream.
However, your underlying need to support llama-server's internal JSON interceptors is completely valid. We have just released v21.3, which implements your tool_call_format kwarg logic as a dedicated opt-in override (without the auto-detect).
You can now explicitly launch with --chat-template-kwargs '{"tool_call_format": "json"}' to safely get the Hermes JSON format for your setup!
Closing this PR in favor of the v21.3 opt-in approach. Thanks again for highlighting the interception issue and proposing the core format logic!