nathanrchn commited on
Commit
2830f02
·
verified ·
1 Parent(s): 20883bb

Fix chat template: render assistant tool_calls

Browse files

Assistant messages carrying `tool_calls` render to nothing under the current
template: the message loop ends in a single `{{- content + "<|im_end|>\n" -}}`
with no branch for tool calls. Two consequences, both reproduced on
llama.cpp `server-rocm-b10088`:

1. llama.cpp probes the template at startup, sees that a tool call renders to
nothing, and reports `supports_tool_calls: false` /
`supports_parallel_tool_calls: false` in `/props`. It then constrains
generation to at most one call, so the second call of a parallel request is
never sampled.
2. Replaying a conversation that already contains an assistant tool call
produces an empty assistant turn, while the following `tool` result is still
rendered -- the model sees an answer to a question it has no record of asking.

**This PR** adds the `format_arg_value` / `render_tool_calls` macros (taken
verbatim from the latest LFM2.5 template) and one branch that emits them, so a
tool call renders as
`<|tool_call_start|>[name(arg='value'), ...]<|tool_call_end|>`.
It also changes `message["content"]` to `message.get("content")`: an assistant
message with `tool_calls` and no `content` key -- the ordinary OpenAI-style
payload -- otherwise raises
`TypeError: Object of type Undefined is not JSON serializable`.

Nothing else changes. The tool-marker convention, thinking handling and
formatting are untouched, and a conversation without tools renders
byte-identically to the current template.

**Verified** on llama.cpp: `chat_template_caps` goes false -> true, parallel
tool calls 1 -> 2, and the replayed assistant turn is preserved. transformers
and minja render identically. Benchmark runs (IFBench, MATH-500, GPQA Diamond,
AA-Omniscience, BFCL v4 multiple/multi_turn_base/irrelevance) are in flight;
please hold merge until those confirm no regression.

Files changed (1) hide show
  1. chat_template.jinja +37 -2
chat_template.jinja CHANGED
@@ -1,4 +1,31 @@
1
  {{- bos_token -}}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2
  {%- set system_prompt = "" -%}
3
  {%- set ns = namespace(system_prompt="") -%}
4
  {%- if messages[0]["role"] == "system" -%}
@@ -23,14 +50,22 @@
23
  {%- endif -%}
24
  {%- for message in messages -%}
25
  {{- "<|im_start|>" + message["role"] + "\n" -}}
26
- {%- set content = message["content"] -%}
27
  {%- if content is not string -%}
28
  {%- set content = content | tojson -%}
29
  {%- endif -%}
30
  {%- if message["role"] == "tool" -%}
31
  {%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%}
32
  {%- endif -%}
33
- {{- content + "<|im_end|>\n" -}}
 
 
 
 
 
 
 
 
34
  {%- endfor -%}
35
  {%- if add_generation_prompt -%}
36
  {{- "<|im_start|>assistant\n" -}}
 
1
  {{- bos_token -}}
2
+ {%- macro format_arg_value(arg_value) -%}
3
+ {%- if arg_value is string -%}
4
+ {{- "'" + (arg_value | replace("\\", "\\\\") | replace("'", "\\'") | replace("\n", "\\n") | replace("\r", "\\r")) + "'" -}}
5
+ {%- elif arg_value is mapping or arg_value is iterable -%}
6
+ {{- arg_value | tojson -}}
7
+ {%- else -%}
8
+ {{- arg_value | string -}}
9
+ {%- endif -%}
10
+ {%- endmacro -%}
11
+ {%- macro render_tool_calls(tool_calls) -%}
12
+ {%- set tool_calls_ns = namespace(tool_calls=[]) -%}
13
+ {%- for tool_call in tool_calls -%}
14
+ {%- set func = tool_call["function"] if "function" in tool_call else tool_call -%}
15
+ {%- set func_name = func["name"] -%}
16
+ {%- set func_args = func.get("arguments") -%}
17
+ {%- set args_ns = namespace(arg_strings=[]) -%}
18
+ {%- if func_args is mapping -%}
19
+ {%- for arg_name, arg_value in func_args.items() -%}
20
+ {%- set args_ns.arg_strings = args_ns.arg_strings + [arg_name + "=" + format_arg_value(arg_value)] -%}
21
+ {%- endfor -%}
22
+ {%- elif func_args is string and (func_args | trim) not in ["", "{}", "null"] -%}
23
+ {{- raise_exception("Tool call arguments must be a mapping, got a JSON-encoded string: parse arguments with json.loads() before applying the chat template") -}}
24
+ {%- endif -%}
25
+ {%- set tool_calls_ns.tool_calls = tool_calls_ns.tool_calls + [func_name + "(" + (args_ns.arg_strings | join(", ")) + ")"] -%}
26
+ {%- endfor -%}
27
+ {{- "<|tool_call_start|>[" + (tool_calls_ns.tool_calls | join(", ")) + "]<|tool_call_end|>" -}}
28
+ {%- endmacro -%}
29
  {%- set system_prompt = "" -%}
30
  {%- set ns = namespace(system_prompt="") -%}
31
  {%- if messages[0]["role"] == "system" -%}
 
50
  {%- endif -%}
51
  {%- for message in messages -%}
52
  {{- "<|im_start|>" + message["role"] + "\n" -}}
53
+ {%- set content = message.get("content") -%}
54
  {%- if content is not string -%}
55
  {%- set content = content | tojson -%}
56
  {%- endif -%}
57
  {%- if message["role"] == "tool" -%}
58
  {%- set content = "<|tool_response_start|>" + content + "<|tool_response_end|>" -%}
59
  {%- endif -%}
60
+ {%- if message["role"] == "assistant" and message.get("tool_calls") -%}
61
+ {%- if content and content != "null" -%}
62
+ {{- content -}}
63
+ {%- endif -%}
64
+ {{- render_tool_calls(message["tool_calls"]) -}}
65
+ {{- "<|im_end|>\n" -}}
66
+ {%- else -%}
67
+ {{- content + "<|im_end|>\n" -}}
68
+ {%- endif -%}
69
  {%- endfor -%}
70
  {%- if add_generation_prompt -%}
71
  {{- "<|im_start|>assistant\n" -}}