docs/runtime-api.md

MiniCPM5 on WebGPU

Two self-contained ES modules that run MiniCPM5-2B on the local GPU and let it call your tools:

File	Size	Exports
minicpm5.js	1.7 MB	MiniCPM5ForCausalLM, which loads the model and runs conversations.
agent.js	6 KB	Agent and runAgent, which run the model as an agent: it calls your tools until it can answer.

Neither file has dependencies. Prompts, replies and tool results stay on the device. The only network traffic is the model download.

Requirements

• WebGPU: a recent Chrome or Edge, or Node with Dawn's WebGPU bindings installed as navigator.gpu.
• Serve the files over HTTP. ES modules and WebGPU do not run from file://.
• The default Q4_K_M file is about a 1.6 GB download. At the default 8,192-token context the model holds about 2 GB of GPU memory.

Quick start

import { MiniCPM5ForCausalLM } from "./minicpm5.js";
import { Agent } from "./agent.js";

const model = await MiniCPM5ForCausalLM.from_pretrained();   // openbmb/MiniCPM5-2B-GGUF, Q4_K_M

const agent = new Agent({
  model,
  instructions: "You are a helpful assistant. Use the tools when they help.",
  tools: [{
    name: "get_weather",
    description: "Current weather for a city.",
    parameters: {
      type: "object",
      properties: { city: { type: "string" }, unit: { type: "string", enum: ["celsius", "fahrenheit"] } },
      required: ["city"],
    },
    async execute({ city, unit = "celsius" }) {
      return JSON.stringify(await lookUpWeather(city, unit));   // your code
    },
  }],
  max_new_tokens: 4096,
});

const run = agent.run("What's the weather in Paris and in Tokyo?");
for await (const event of run) {
  if (event.type === "text") showAnswer(event.text);
}
const { content } = await run.result;

agent.js does not depend on MiniCPM5. It drives any model object with a create_session() method, and any session whose generate works like the one below.

Loading the model

const { ok, reason } = await MiniCPM5ForCausalLM.checkAvailability();
if (!ok) showError(reason);

const model = await MiniCPM5ForCausalLM.from_pretrained("openbmb/MiniCPM5-2B-GGUF", {
  file: "MiniCPM5-2B-Q8_0.gguf",
  maxLength: 16384,
  onProgress: ({ status, fraction }) => showProgress(status, fraction),
});

checkAvailability(id, options) reads only the GGUF header and checks it against this device's GPU. It never throws: if the check itself fails (for example, the network is offline), it resolves { ok: true } so the real load can report the real error.

from_pretrained(id, options) takes a repo id, or a direct URL to a .gguf file. With no id, it loads openbmb/MiniCPM5-2B-GGUF.

Option	Meaning
file	The GGUF file in the repo: MiniCPM5-2B-Q4_K_M.gguf (default), MiniCPM5-2B-Q8_0.gguf or MiniCPM5-2B-F16.gguf.
maxLength	Context size in tokens, prompt and reply together. Default 8192, clamped to what the device supports. Each token of context takes 84 KiB of GPU memory, allocated at load.
onProgress	Called with { status, fraction?, loaded?, total?, message? }. status runs init, config, support, tokenizer, weights, ready.
signal	An AbortSignal that cancels the load and frees whatever it had allocated.
revision	Branch, tag or commit of the repo. Default main.
accessToken	A Hugging Face token, for a private or gated repo. It is used only in request headers and never stored.
cache	false turns off the browser cache described below.

In a browser, the weights are cached as they stream, so a reload downloads only what is missing. Outside a browser nothing is cached, and the file is read over HTTP on every load.

Other repos' copies of MiniCPM5-2B load as well (for example bartowski/MiniCPM5-2B-GGUF, Q4_K_M), as long as every tensor is F32, F16, Q4_K, Q5_K, Q6_K or Q8_0. A file in any other format is refused before its weights download.

A loaded model has:

• create_session(), which starts a conversation.
• generation_config, which is always { do_sample: false } for this model.
• contextLength.
• deviceInfo().
• memoryStats(), the GPU bytes held now and the peak so far.
• lost, a promise that resolves if the GPU device is lost.
• dispose(), which frees everything.

Chatting without tools

const session = model.create_session();
const history = [{ role: "user", content: "Explain recursion in two sentences." }];

const turn = session.generate(history, { max_new_tokens: 2048 });
for await (const event of turn) {
  if (event.type === "reasoning") showThinking(event.text);
  else if (event.type === "text") showAnswer(event.text);
}
const { finish_reason, message, usage, timings } = await turn.result;
history.push(message);   // push it back unchanged

Send the whole history on every turn. The session keeps the conversation's keys and values on the GPU and computes only the part of the prompt it has not seen. To make that work, push each message back exactly as it was returned. Edited or restored histories still work: the session compares tokens, and recomputes from the first difference.

Events arrive in this order:

1. prefill, with { done, total }, while a long prompt is read.
2. reasoning and text deltas. Joined, they equal message.reasoning_content and message.content.
3. tool_call, one per call, after the reply has ended.

finish_reason is one of:

• stop: the model finished.
• length: the reply ran out of room. limit is max_new_tokens or context.
• tool_calls: the reply asked for tools (see below).
• cancelled: the turn was stopped.

usage has prompt_tokens, cached_tokens (not recomputed this turn), completion_tokens and reasoning_tokens. timings has ttft_ms, decode_ms and tokens_per_second.

Turn options:

Option	Meaning
max_new_tokens	Most tokens to generate, reasoning included. Default 1024.
enable_thinking	true or false. Leave it unset to let the model decide (see below).
tools	Functions the model may call.
stop_strings	End the reply at any of these strings. The match is left out of the reply.
signal	An AbortSignal that cancels the turn.

Cancel with turn.cancel(), a signal, or by breaking out of the for await loop. The result is cancelled, with the partial reply. The cache stays valid either way. You may leave the partial reply out of the history, or push it.

Errors. A turn that fails rejects its result, and iterating it throws the same error. The model runs one turn at a time, so a turn started while another is running fails with BusyError. A prompt that cannot fit fails with ContextOverflowError (with required and capacity) before anything runs; trim the history and try again. If the GPU device is lost, the running turn fails with DeviceLostError and model.lost resolves; dispose the model and load it again. All three classes are exported by minicpm5.js.

Sessions. Each session is one conversation with its own cache. The first session uses the cache the model allocated at load, and each further one allocates another. Use session.dispose() to free one. Use session.count_tokens(history, { tools, enable_thinking }) to count a prompt before sending it.

Thinking

With enable_thinking unset, MiniCPM5 decides for itself whether to think first. It usually does, and often at length: a coding question can think for more than 700 tokens. Give such turns a max_new_tokens of a few thousand, or the reply ends on length before it answers. enable_thinking: true always thinks. false answers directly, which is faster but can be weaker.

The reasoning never appears in content.

Decoding

MiniCPM5 decodes greedily (always the most likely token), so the same history gives the same reply. A turn that asks to sample (do_sample: true, temperature, top_p, ...) is refused with an error.

Tool calls without agent.js

If you already have an agent loop, use the session directly. Tools use the format of transformers' chat templates:

const tools = [{ type: "function", function: {
  name: "get_weather",
  description: "Current weather for a city.",
  parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"] },
} }];

const turn = session.generate(history, { tools, max_new_tokens: 4096 });
for await (const event of turn) { /* ... */ }
const { finish_reason, message } = await turn.result;
history.push(message);
if (finish_reason === "tool_calls") {
  for (const call of message.tool_calls) {
    const content = call.errors ? call.errors.join("; ") : await run(call.function.name, call.function.arguments);
    history.push({ role: "tool", tool_call_id: call.id, name: call.function.name, content });
  }
  // generate again with the same tools
}

• call.function.arguments is an object, already parsed and converted to the types in the schema. (OpenAI's API gives a JSON string instead.)
• Never run a call that has errors. The markup did not parse, the tool was not offered, or an argument is missing or has the wrong type. Send the errors back as the call's result, and the model can correct the call.
• Answer every call, in order, with one tool message each. The template pairs results with calls by position.
• Keep tools identical across turns. The tools are part of the prompt's start, so changing them makes the next turn recompute the whole conversation.

The agent (agent.js)

Agent keeps a conversation and runs the loop for you: it generates a reply, runs each tool the reply calls, sends back the results, and repeats until the model answers.

const agent = new Agent({
  model,                      // a loaded model, or a session you created
  instructions: "You are a helpful assistant.",
  tools: [weather, calculator],
  max_new_tokens: 4096,       // and any other turn option: enable_thinking, stop_strings, ...
  max_rounds: 8,              // most replies per run (default 8)
});
await agent.prime();          // optional: read the instructions and tools now, before the first message

const run = agent.run("What is 1234 * 5678 + 91?");
for await (const event of run) {
  switch (event.type) {
    case "reasoning": showThinking(event.text); break;
    case "text": showAnswer(event.text); break;
    case "tool_start": showCall(event.call.function.name, event.call.function.arguments); break;
    case "tool_end": showResult(event.message.content, event.is_error); break;
  }
}
const { finish_reason, content, usage, rounds } = await run.result;

Tools

A tool is a plain object:

Field	Meaning
name	Unique within the agent.
description	What the tool does. The model reads this.
parameters	A JSON Schema object for the arguments.
execute(args, { signal, call })	Runs the call. signal aborts if the run is cancelled.

execute may return a string, or { content, details }. The model sees content, and details reaches only your code, on the tool_end event. Returning nothing sends an empty result. MiniCPM5 reads text only, so return text.

The agent handles these cases for you:

• Errors go to the model. If execute throws, the error message is the call's result, and the model can try something else.
• Bad calls never run. A call that did not parse, names an unknown tool, or has missing or wrong arguments is answered with what is wrong.
• Repeated failures are refused. If the model repeats a call that just failed, with the same arguments and nothing succeeding in between, it gets "already failed" back, and the call does not run again.
• Order. Calls run one at a time, in the order the model wrote them.

Events and results

Every event carries round, which counts the run's replies from 0. In order:

Event	When	Fields
turn_start	A reply begins.
prefill, reasoning, text, tool_call	The model's own events, as from a session.
turn_end	The reply is complete.	result: the turn's finish_reason, message, usage and timings
tool_start	A call is about to run.	call
tool_end	A call has its answer.	call, message (the tool message), is_error, details

run.result resolves with:

Field	Meaning
finish_reason	stop (the model answered), length (out of room; limit says which), cancelled, or max_rounds (it was still calling tools after max_rounds replies)
content	The last reply's answer.
messages	Everything the run added to the conversation.
usage	Token counts summed over every reply.
rounds	Number of replies.

A failure, such as ContextOverflowError or DeviceLostError, rejects run.result, and iterating the run throws the same error.

Cancelling

Call run.cancel(), pass signal to agent.run(input, { signal }), or break out of the loop. A reply in progress stops. A running tool sees its signal abort and is answered "may have partly completed". Calls that had not started are answered "not run". Every call still gets an answer, so the conversation remains valid.

The conversation

agent.messages is the conversation, with the system prompt first. Each reply and each tool result is added as soon as it completes, so it is always a valid history, even after a cancel or an error. A run that produced no reply leaves it unchanged. You can edit it between runs.

The run waits for your loop at the end of each step. It goes on past a turn_end or tool_end only after your loop has handled that event. So you can save the conversation before any tool runs:

for await (const event of run) {
  if (event.type === "turn_end" || event.type === "tool_end") await save(JSON.stringify(agent.messages));
}

To restore a conversation later:

const [system, ...rest] = JSON.parse(await load());
const agent = new Agent({ model, instructions: system.content, tools, messages: rest });

Other methods:

• run(input, options) takes text or a whole message. Its options add to the agent's own.
• running says whether a run is in progress. Only one run can go at a time.
• reset() clears the conversation but keeps the system prompt.
• dispose() cancels any run and frees the session the agent created. A session you passed in stays open.

runAgent: the loop with your own history

runAgent(session, history, options) is the same loop, for code that stores the conversation itself. It never changes the array you pass. Append what the run added:

import { runAgent } from "./agent.js";

const session = model.create_session();
const run = runAgent(session, history, { tools, max_new_tokens: 4096 });
for await (const event of run) render(event);
history.push(...(await run.result).messages);

Push those messages unchanged. The replies are the session's own objects, which it re-renders exactly as the model wrote them, so the next run computes only the new part of the prompt.