Sariel00

/

Ling-3.0-tiny-RKNN

Like 2

This adds OpenAI-style tool calling to POST /v1/chat/completions, so
agent clients (pi, opencode and others) can use the engine directly. Today
the strict request check rejects tools, tool messages and assistant
tool_calls.

Requests. The server now accepts:

• tools;
• tool_choice set to auto or none. required and a named tool return
  400, because there is no constrained decoding;
• parallel_tool_calls;
• tool messages (tool_call_id, name);
• assistant messages with tool_calls (arguments as a JSON string or an
  object), reasoning_content and null content;
• content given as an array of text parts.

The boolean store flag is accepted and ignored. Some clients send
store: false on every request. Unknown keys are still rejected.

Prompt. The prompt follows the official Bailing V3 chat_template.jinja
(inclusionAI/Ling-3.0-tiny@e3a47d5b) branch for branch:

• the tools render as tojson lines in <tools>, keeping the client's key
  order;
• calls use <arg_key>/<arg_value> with raw string values;
• results go in <role>OBSERVATION</role> with one <tool_response> each.

tests/chat_tools_golden.json holds eight renderings produced by
transformers' own render_jinja_template (4.57.1) on the pinned template.
tools/render_chat_tools_golden.py regenerates them. The tests compare
byte for byte. The prompt for requests without tools is unchanged.

Output. Generated <tool_call> blocks are held back from the stream
inside Engine::Run, after TextFilter, so flow_control: ack never
waits on an empty chunk. Parsing follows llama.cpp's ling3 parser
(ggml-org/llama.cpp#28682):

• only declared tool names are parsed as calls;
• string-typed arguments stay raw, and other types are parsed as JSON;
• whitespace between arguments is optional.

Parsed calls come back as message.tool_calls, or as one
delta.tool_calls chunk per call when streaming, with finish_reason: "tool_calls". Text that does not parse is released unchanged as content.
A round-trip test checks that a parsed call re-renders to exactly the
generated text, so the generated-prefix cache keeps hitting on the next
turn.

Tests. -DLING3_WITH_RKNN=OFF host build on Debian 13 (GCC 14,
ICU 76): 10/10 pass. The new test is chat_tools_test, and the tools
rejection case in chat_protocol_test now covers the invalid-type and
tool_choice: required cases instead.

Board check. Run on a FriendlyElec CM3588 (RK3588, rknpu 0.9.8,
librknnrt 2.3.2) with the v6 ling3-tiny-w4.l3r, 64K context, thinking
off:

• A two-stage tool-calling probe passed. In a two-turn read → patch loop
  with the cache on, turn 2 was a generated_prefix_hit (346 tokens
  cached, 23 evaluated).
• Answers without tools were byte-identical to the prebuilt v6 binary.
• At 466 prompt tokens, prefill was 185–187 tok/s, slightly above the
  prebuilt v6 binary's 139–178 on the same board. Decode was 18.8–19.0
  tok/s, inside v6's 18.4–19.4.
• The board build is this branch plus one container-only change to where
  the device probe reads the SoC compatible string. That change is not
  part of this PR.

With thinking on, the reasoning still arrives in content up to
</think>. Splitting it into reasoning_content is left for a separate
change.

If you require higher performance, you can check out Sariel00/Cecilia_srv.There won't be too many updates here for the time being.Thank you for submitting.