google

/

gemma-4-31B-it

like 3.61k

Follow

Google 64.9k

Summary

Gemma 4 chat template has two related defects around the
<|channel>thought block:

1. No single convention for who opens the thought channel at generation time.
   On plain thinking turns the prompt ends bare after <|turn>model\n (the model must
   emit its own opener); after tool responses the prompt pre-opens the channel; with
   thinking disabled after tool responses, nothing is emitted at all. No reasoning
   parser can satisfy three contracts at once. In production serving (vLLM ≥ 0.26,
   --reasoning-parser gemma4) any token the model emits before its opener on a plain
   turn is classified as answer content — users see junk fragments fused onto
   replies (observed live: "althi khalid" = a leaked alt + hi khalid).
2. The post-tool pre-open is emitted outside the model turn on the most common tool
   shape. When the tool-calling assistant message also carried prose content
   ("Let me check." + tool call → tool result — the standard OpenAI round shape), the
   template's closure chain emits <turn|> (because has_content is true), but the
   generation block still assumes the turn is open and appends <|channel>thought\n
   after the closed turn, at top level, with no <|turn>model\n — a byte sequence
   that never occurs in a valid transcript. With thinking disabled on the same shape,
   no generation prompt is emitted at all.

Root cause

The add_generation_prompt block infers "the model turn is still open" from
ns.prev_message_type == 'tool_response' — a proxy that disagrees with what the
closure chain actually did whenever the tool-calling message carried prose. And the
thinking on/off prefill is applied in only two of the four reachable states.

Proposed fix (three edits)

One principle: record turn-openness where it is decided instead of inferring it
later, and always pre-fill the thought channel — open when thinking is enabled,
closed-empty when it is not.

Edit 1 — namespace init gains a flag:

{%- set ns = namespace(prev_message_type=None, prev_non_tool_role=None, model_turn_open=false) -%}

Edit 2 — the per-message closure chain records what it did (the {%- set -%}
lines are render-invisible):

        {%- if ns.prev_message_type == 'tool_call' and not ns_tr_out.flag -%}
            {{- '<|tool_response>' -}}
            {%- set ns.model_turn_open = true -%}
        {%- elif continues_into_next -%}
            {%- set ns.model_turn_open = true -%}
        {%- elif not (ns_tr_out.flag and not has_content and not next_nt.found) -%}
            {{- '<turn|>\n' -}}
            {%- set ns.model_turn_open = false -%}
        {%- else -%}
            {%- set ns.model_turn_open = true -%}
        {%- endif -%}

Edit 3 — the generation block reads the record and applies one uniform prefill:

{%- if add_generation_prompt -%}
    {%- if ns.prev_message_type != 'tool_call' -%}
        {%- if not ns.model_turn_open -%}
            {{- '<|turn>model\n' -}}
        {%- endif -%}
        {%- if enable_thinking -%}
            {{- '<|channel>thought\n' -}}
        {%- else -%}
            {{- '<|channel>thought\n<channel|>' -}}
        {%- endif -%}
    {%- endif -%}
{%- endif -%}

Resulting contract, every state: the prompt ends inside an open model turn, with the
thought channel pre-opened when thinking is enabled (the parser starts the
generation in reasoning state — pre-thought leakage becomes structurally impossible)
or pre-closed when it is not (thinking is actually disabled, including after tool
responses). On the prose-carrying tool round, the turn is correctly re-opened with
<|turn>model\n before the prefill — fixing defect 2 for thinking on and off.

Impact

Fixes both defect classes at the source for every client of every serving stack that
splits Gemma 4 reasoning by channel markers. Clients cannot repair this downstream —
once reasoning is mistyped as content, no marker survives to detect it.