# Proposal: fix inconsistent thought-channel convention in `chat_template.jinja` (`add_generation_prompt`)

# Proposal: fix inconsistent thought-channel convention in `chat_template.jinja` (`add_generation_prompt`)

*Target: the `google/gemma-4-31B-it`

---

## Summary

The canonical Gemma 4 chat template uses **three different conventions** for who opens
the `<|channel>thought` block at generation time, depending on the conversation state.
No reasoning parser can be correct against all three, and in production serving
(vLLM ≥ 0.26, `--reasoning-parser gemma4`) this produces two visible failure classes:

1. **Leaked pre-thought fragments rendered as answer text.** On a plain turn with
`enable_thinking=true`, the prompt ends bare after `<|turn>model\n`. If the model
emits any token before opening its thought channel, the parser (which starts in
CONTENT state) types it as answer content. Users see garbage prefixes fused onto
the reply (e.g. `"althi khalid"` — a leaked `alt` glued to `hi khalid`).
2. **Unconstrained thinking after tool responses when thinking is disabled.** With
`enable_thinking=false`, the template pre-fills a closed empty thought on plain
turns — but appends **nothing** after a tool response. The model is free to think
(or mumble degenerate filler that lands in the visible answer) on a turn the
caller explicitly configured as non-thinking.

## The four generation states today

| state | prompt tail emitted by the template | consequence |
|---|---|---|
| plain turn, thinking **on** | *(nothing after `<\|turn>model\n`)* | ✗ model may emit tokens before opening the thought → parser types them as content |
| plain turn, thinking **off** | `<\|channel>thought\n<channel\|>` | ✓ correct — forced non-thinking |
| after tool response, thinking **on** | `<\|channel>thought\n` (pre-opened) | ✓ correct — vLLM's parser supports prompt-side pre-open |
| after tool response, thinking **off** | *(nothing)* | ✗ thinking not actually disabled |

The root defect is not any single state — it is that the template has **no single
answer** to "who emits the thought opener": sometimes the prompt pre-opens, sometimes
the model must. An output-marker-driven parser mishandles the pre-open state; a
reasoning-first parser mishandles the bare state. vLLM's `gemma4` parser resolves the
pre-open case via `adjust_initial_state_from_prompt()` (initial state REASONING when
the prompt ends inside an open `<|channel>` block) — but the bare plain-turn state
still leaves a window in which the model's first tokens are typed as content.

## Proposed fix

Make the convention uniform: **the prompt always pre-fills the thought channel — open
when thinking is enabled, empty-closed when it is not — in every state.** Both prefill
patterns already exist in the current template; the patch only applies them to all
four states instead of two.

```jinja
{%- if add_generation_prompt -%}
{%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
{{- '<|turn>model\n' -}}
{%- endif -%}
{%- if ns.prev_message_type != 'tool_call' -%}
{%- if enable_thinking -%}
{{- '<|channel>thought\n' -}}
{%- else -%}
{{- '<|channel>thought\n<channel|>' -}}
{%- endif -%}
{%- endif -%}
{%- endif -%}
```

(replacing the current final block, which pre-opens only in the
`prev_message_type == 'tool_response' and enable_thinking` arm and emits the closed
prefill only in the non-tool arm.)

After the patch:

| state | prompt tail | change |
|---|---|---|
| plain, thinking on | `<\|channel>thought\n` | **fixed** — generation starts inside reasoning; pre-thought leakage becomes structurally impossible |
| plain, thinking off | `<\|channel>thought\n<channel\|>` | byte-identical to today |
| post-tool, thinking on | `<\|channel>thought\n` | byte-identical to today |
| post-tool, thinking off | `<\|channel>thought\n<channel\|>` | **fixed** — disabled means disabled |

## Why this is safe

- **No new tokens or conventions.** Both prefill shapes are the template's own,
already seen by the model in its canonical formatting (the open prefill in the
post-tool state, the closed prefill on plain non-thinking turns).
- **The two correct states are byte-identical before and after** — behavior there
cannot change, by construction.
- **Parser support is already shipped.** vLLM's `gemma4` reasoning parser
pre-initializes to REASONING when the prompt ends inside an open channel
(`adjust_initial_state_from_prompt`, present since at least v0.26.0), which is
exactly the contract the uniform pre-open relies on.
- **History rendering, tool-call formatting, the system turn, and `strip_thinking`
are untouched** — the patch is confined to the `add_generation_prompt` suffix.


## Impact

Any OpenAI-compatible serving stack that splits Gemma 4 reasoning from answers by
channel markers benefits; clients cannot fix this downstream — by the time the stream
reaches them, reasoning mistyped as content is indistinguishable from a real answer.
The patch closes the failure class at the source for every client at once.

chat_template.jinja

@@ -381,10 +381,12 @@
 {%- if add_generation_prompt -%}
     {%- if ns.prev_message_type != 'tool_response' and ns.prev_message_type != 'tool_call' -%}
         {{- '<|turn>model\n' -}}
-        {%- if not enable_thinking -%}
+    {%- endif -%}
+    {%- if ns.prev_message_type != 'tool_call' -%}
+        {%- if enable_thinking -%}
+            {{- '<|channel>thought\n' -}}
+        {%- else -%}
             {{- '<|channel>thought\n<channel|>' -}}
         {%- endif -%}
-    {%- elif ns.prev_message_type == 'tool_response' and enable_thinking -%}
-        {{- '<|channel>thought\n' -}}
     {%- endif -%}
-{%- endif -%}
+{%- endif -%}