Chat template may re-inject prior-turn reasoning during multi-turn tool use → repetition loops
Summary
The chat_template shipped with this model may be vulnerable to a multi-turn
reasoning re-injection issue that can cause verbatim repetition loops during
agentic / tool-calling use (e.g. OpenCode, llama.cpp --jinja, any harness that sends
back reasoning_content on prior assistant tool-call steps).
It is harmless for single-turn chat — it only triggers in a multi-step tool-calling
sequence — which is why it does not show up in standard single-turn benchmarks.
Mechanism
The template renders each prior assistant turn's thinking back into the prompt:
{%- set thinking_text = message.get('reasoning') or message.get('reasoning_content') -%}
{%- if thinking_text and loop.index0 > ns_turn.last_user_idx and message.get('tool_calls') -%}
{{- '<|channel>thought\n' + thinking_text + '\n<channel|>' -}}
{%- endif -%}
The guard loop.index0 > last_user_idx and message.tool_calls restricts this to the
assistant tool-call steps that occur after the final user message — i.e. exactly the
in-request agentic loop. Each new step is therefore prompted with all of its own previous
private thoughts re-injected as <|channel>thought blocks. As the agentic context grows,
the model is fed an accumulating echo of its own reasoning and can collapse into a
repetition loop.
Scope
Detected by static inspection of the chat_template across the Gemma 4 instruct family
and several third-party requants — the re-injection block is present in all of them. We
have dynamically reproduced and fixed the loop on one pruned-MoE derivative; on other
sizes/quants the same code path is present but we have not measured loop severity directly,
hence "may be vulnerable."
Tests we ran (on our derivative)
Same engine / weights / seeds — only the rendered prompt varied:
| Condition | Multi-turn agentic loop rate | HumanEval+ (Q6_K) | MultiPL-E-100 (Q6_K) |
|---|---|---|---|
| Stock template (re-injection on) | 33% (4/12 seeds) | — | — |
| Fixed template (re-injection off) | 0% | 92.07% | 0.66 |
Single-turn code/instruction scores are unchanged by the fix (the re-injection path is
multi-turn-only), so the fix carries no quality cost.
Fix
Disable the historical-reasoning re-injection (keep the thinking channel only for the
current generation). The minimal change is to make that {%- if ... -%} guard never fire:
{%- if false and thinking_text and loop.index0 > ns_turn.last_user_idx and message.get('tool_calls') -%}
A drop-in corrected template (validated as above) is here:
https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF/blob/main/chat_template.fixed.jinja
For GGUF files the template can be rewritten in-place without re-quantizing, viagguf_new_metadata.py --chat-template-file chat_template.fixed.jinja (llama.cpp gguf-py).
A 1-minute unit test to check any repo / template
Runnable version: https://huggingface.co/ManniX-ITA/gemma-4-A4B-98e-v7-coder-it-GGUF/blob/main/template_loop_unittest.py
from transformers import AutoTokenizer
SENTINEL = "ZZ_HISTORY_THOUGHT_SENTINEL_ZZ"
# One user request, then a multi-step tool-calling sequence (real agentic shape).
conv = [
{"role": "user", "content": "Build a small Three.js website with a rotating cube."},
{"role": "assistant", "content": "", "reasoning_content": SENTINEL,
"tool_calls": [{"type": "function", "function": {"name": "write_file",
"arguments": {"path": "index.html", "content": "<html></html>"}}}]},
{"role": "tool", "content": "index.html written"},
{"role": "assistant", "content": "", "reasoning_content": "Now wire up the JS.",
"tool_calls": [{"type": "function", "function": {"name": "write_file",
"arguments": {"path": "main.js", "content": "//"}}}]},
{"role": "tool", "content": "main.js written"},
]
tok = AutoTokenizer.from_pretrained("<this-repo-or-local-dir>")
rendered = tok.apply_chat_template(conv, tokenize=False, add_generation_prompt=True)
leaks = rendered.count(SENTINEL)
assert leaks == 0, f"MAY BE VULNERABLE: prior-turn reasoning re-injected {leaks}x as a thinking channel"
print("OK: no history-reasoning re-injection")
On the current template this asserts (sentinel re-injected); on the corrected template it
prints OK.
Hope this is useful — feel free to ignore if your harness never replays reasoning_content.
This is an automated message generated from a template audit across the Gemma 4 family
and its requants. It is shared for your awareness; no action is required on our part.
I'm trying to fix that with this PR, @ManniX-ITA - can you take a look if it fixes your scenario? https://huggingface.co/google/gemma-4-12B-it/discussions/35
@lucianommartins
Nope, doesn't seem to fix it.
Did a quick re-run on the same automated test harness with opencode and preserve_thinking = true is just slightly better.
If false it's the same as the original template (expected, same behavior).
Is there a reason for re-emitting the prior-turn reasoning_content as <|channel>thought back into the multi-turn prompt?
I'm running a longer validation now but the first results are already telling.
Completed the validation:
Results (failure = verbatim loop or runaway):
- upstream main: 37.5% (9/24)
- PR#35 as shipped (preserve_thinking=true): 29.2% (7/24) — still loops
- PR#35 preserve_thinking=false: 37.5% (9/24) — identical to main, same failing seeds
- re-injection fully disabled: 0% (0/24)
I have witnessed this behavior personally even on openwebui with web search enabled.
I am new to open models and the tools being used. I am trying to learn how to apply this fix mentioned above. Can you provide any direction for applying the fix with llama.cpp?
It seems that to copy the fix to the models directory, and run llama.cpp gguf-py --chat-template-file chat_template.fixed.jinja. Is this the right direction. I haven't tried it yet.