docs/runtime-api.md

# MiniCPM5 on WebGPU

Two self-contained ES modules that run [MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF)
on the local GPU and let it call your tools:

| File | Size | Exports |
| --- | --- | --- |
| `minicpm5.js` | 1.7 MB | `MiniCPM5ForCausalLM`, which loads the model and runs conversations. |
| `agent.js` | 6 KB | `Agent` and `runAgent`, which run the model as an agent: it calls your tools until it can answer. |

Neither file has dependencies. Prompts, replies and tool results stay on the device. The only network
traffic is the model download.

**Requirements**

- WebGPU: a recent Chrome or Edge, or Node with Dawn's WebGPU bindings installed as `navigator.gpu`.
- Serve the files over HTTP. ES modules and WebGPU do not run from `file://`.
- The default Q4_K_M file is about a 1.6 GB download. At the default 8,192-token context the model
  holds about 2 GB of GPU memory.

## Quick start

```js
import { MiniCPM5ForCausalLM } from "./minicpm5.js";
import { Agent } from "./agent.js";

const model = await MiniCPM5ForCausalLM.from_pretrained();   // openbmb/MiniCPM5-2B-GGUF, Q4_K_M

const agent = new Agent({
  model,
  instructions: "You are a helpful assistant. Use the tools when they help.",
  tools: [{
    name: "get_weather",
    description: "Current weather for a city.",
    parameters: {
      type: "object",
      properties: { city: { type: "string" }, unit: { type: "string", enum: ["celsius", "fahrenheit"] } },
      required: ["city"],
    },
    async execute({ city, unit = "celsius" }) {
      return JSON.stringify(await lookUpWeather(city, unit));   // your code
    },
  }],
  max_new_tokens: 4096,
});

const run = agent.run("What's the weather in Paris and in Tokyo?");
for await (const event of run) {
  if (event.type === "text") showAnswer(event.text);
}
const { content } = await run.result;
```

`agent.js` does not depend on MiniCPM5. It drives any model object with a `create_session()` method,
and any session whose `generate` works like the one below.

## Loading the model

```js
const { ok, reason } = await MiniCPM5ForCausalLM.checkAvailability();
if (!ok) showError(reason);

const model = await MiniCPM5ForCausalLM.from_pretrained("openbmb/MiniCPM5-2B-GGUF", {
  file: "MiniCPM5-2B-Q8_0.gguf",
  maxLength: 16384,
  onProgress: ({ status, fraction }) => showProgress(status, fraction),
});
```

`checkAvailability(id, options)` reads only the GGUF header and checks it against this device's
GPU. It never throws: if the check itself fails (for example, the network is offline), it resolves
`{ ok: true }` so the real load can report the real error.

`from_pretrained(id, options)` takes a repo id, or a direct URL to a `.gguf` file. With no id, it loads
`openbmb/MiniCPM5-2B-GGUF`.

| Option | Meaning |
| --- | --- |
| `file` | The GGUF file in the repo: `MiniCPM5-2B-Q4_K_M.gguf` (default), `MiniCPM5-2B-Q8_0.gguf` or `MiniCPM5-2B-F16.gguf`. |
| `maxLength` | Context size in tokens, prompt and reply together. Default 8192, clamped to what the device supports. Each token of context takes 84 KiB of GPU memory, allocated at load. |
| `onProgress` | Called with `{ status, fraction?, loaded?, total?, message? }`. `status` runs `init`, `config`, `support`, `tokenizer`, `weights`, `ready`. |
| `signal` | An `AbortSignal` that cancels the load and frees whatever it had allocated. |
| `revision` | Branch, tag or commit of the repo. Default `main`. |
| `accessToken` | A Hugging Face token, for a private or gated repo. It is used only in request headers and never stored. |
| `cache` | `false` turns off the browser cache described below. |

In a browser, the weights are cached as they stream, so a reload downloads only what is missing.
Outside a browser nothing is cached, and the file is read over HTTP on every load.

Other repos' copies of MiniCPM5-2B load as well (for example `bartowski/MiniCPM5-2B-GGUF`, Q4_K_M), as
long as every tensor is F32, F16, Q4_K, Q5_K, Q6_K or Q8_0. A file in any other format is refused
before its weights download.

A loaded model has:

- `create_session()`, which starts a conversation.
- `generation_config`, which is always `{ do_sample: false }` for this model.
- `contextLength`.
- `deviceInfo()`.
- `memoryStats()`, the GPU bytes held now and the peak so far.
- `lost`, a promise that resolves if the GPU device is lost.
- `dispose()`, which frees everything.

## Chatting without tools

```js
const session = model.create_session();
const history = [{ role: "user", content: "Explain recursion in two sentences." }];

const turn = session.generate(history, { max_new_tokens: 2048 });
for await (const event of turn) {
  if (event.type === "reasoning") showThinking(event.text);
  else if (event.type === "text") showAnswer(event.text);
}
const { finish_reason, message, usage, timings } = await turn.result;
history.push(message);   // push it back unchanged
```

**Send the whole history on every turn.** The session keeps the conversation's keys and values on
the GPU and computes only the part of the prompt it has not seen. To make that work, push each
`message` back exactly as it was returned. Edited or restored histories still work: the session
compares tokens, and recomputes from the first difference.

**Events** arrive in this order:

1. `prefill`, with `{ done, total }`, while a long prompt is read.
2. `reasoning` and `text` deltas. Joined, they equal `message.reasoning_content` and `message.content`.
3. `tool_call`, one per call, after the reply has ended.

**`finish_reason`** is one of:

- `stop`: the model finished.
- `length`: the reply ran out of room. `limit` is `max_new_tokens` or `context`.
- `tool_calls`: the reply asked for tools (see below).
- `cancelled`: the turn was stopped.

**`usage`** has `prompt_tokens`, `cached_tokens` (not recomputed this turn), `completion_tokens` and
`reasoning_tokens`. **`timings`** has `ttft_ms`, `decode_ms` and `tokens_per_second`.

**Turn options:**

| Option | Meaning |
| --- | --- |
| `max_new_tokens` | Most tokens to generate, reasoning included. Default 1024. |
| `enable_thinking` | `true` or `false`. Leave it unset to let the model decide (see below). |
| `tools` | Functions the model may call. |
| `stop_strings` | End the reply at any of these strings. The match is left out of the reply. |
| `signal` | An `AbortSignal` that cancels the turn. |

**Cancel** with `turn.cancel()`, a `signal`, or by breaking out of the `for await` loop. The result is
`cancelled`, with the partial reply. The cache stays valid either way. You may leave the partial reply
out of the history, or push it.

**Errors.** A turn that fails rejects its `result`, and iterating it throws the same error. The model
runs one turn at a time, so a turn started while another is running fails with `BusyError`. A prompt
that cannot fit fails with `ContextOverflowError` (with `required` and `capacity`) before anything
runs; trim the history and try again. If the GPU device is lost, the running turn fails with
`DeviceLostError` and `model.lost` resolves; dispose the model and load it again. All three classes
are exported by `minicpm5.js`.

**Sessions.** Each session is one conversation with its own cache. The first session uses the cache
the model allocated at load, and each further one allocates another. Use `session.dispose()` to free
one. Use `session.count_tokens(history, { tools, enable_thinking })` to count a prompt before sending
it.

### Thinking

With `enable_thinking` unset, MiniCPM5 decides for itself whether to think first. It usually does, and
often at length: a coding question can think for more than 700 tokens. Give such turns a
`max_new_tokens` of a few thousand, or the reply ends on `length` before it answers.
`enable_thinking: true` always thinks. `false` answers directly, which is faster but can be weaker.

The reasoning never appears in `content`.

### Decoding

MiniCPM5 decodes greedily (always the most likely token), so the same history gives the same reply.
A turn that asks to sample (`do_sample: true`, `temperature`, `top_p`, ...) is refused with an error.

### Tool calls without agent.js

If you already have an agent loop, use the session directly. Tools use the format of transformers'
chat templates:

```js
const tools = [{ type: "function", function: {
  name: "get_weather",
  description: "Current weather for a city.",
  parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"] },
} }];

const turn = session.generate(history, { tools, max_new_tokens: 4096 });
for await (const event of turn) { /* ... */ }
const { finish_reason, message } = await turn.result;
history.push(message);
if (finish_reason === "tool_calls") {
  for (const call of message.tool_calls) {
    const content = call.errors ? call.errors.join("; ") : await run(call.function.name, call.function.arguments);
    history.push({ role: "tool", tool_call_id: call.id, name: call.function.name, content });
  }
  // generate again with the same tools
}
```

- **`call.function.arguments` is an object**, already parsed and converted to the types in the
  schema. (OpenAI's API gives a JSON string instead.)
- **Never run a call that has `errors`.** The markup did not parse, the tool was not offered, or an
  argument is missing or has the wrong type. Send the errors back as the call's result, and the model
  can correct the call.
- **Answer every call, in order**, with one `tool` message each. The template pairs results with
  calls by position.
- **Keep `tools` identical across turns.** The tools are part of the prompt's start, so changing them
  makes the next turn recompute the whole conversation.

## The agent (agent.js)

`Agent` keeps a conversation and runs the loop for you: it generates a reply, runs each tool the reply
calls, sends back the results, and repeats until the model answers.

```js
const agent = new Agent({
  model,                      // a loaded model, or a session you created
  instructions: "You are a helpful assistant.",
  tools: [weather, calculator],
  max_new_tokens: 4096,       // and any other turn option: enable_thinking, stop_strings, ...
  max_rounds: 8,              // most replies per run (default 8)
});
await agent.prime();          // optional: read the instructions and tools now, before the first message

const run = agent.run("What is 1234 * 5678 + 91?");
for await (const event of run) {
  switch (event.type) {
    case "reasoning": showThinking(event.text); break;
    case "text": showAnswer(event.text); break;
    case "tool_start": showCall(event.call.function.name, event.call.function.arguments); break;
    case "tool_end": showResult(event.message.content, event.is_error); break;
  }
}
const { finish_reason, content, usage, rounds } = await run.result;
```

### Tools

A tool is a plain object:

| Field | Meaning |
| --- | --- |
| `name` | Unique within the agent. |
| `description` | What the tool does. The model reads this. |
| `parameters` | A JSON Schema object for the arguments. |
| `execute(args, { signal, call })` | Runs the call. `signal` aborts if the run is cancelled. |

`execute` may return a string, or `{ content, details }`. The model sees `content`, and `details`
reaches only your code, on the `tool_end` event. Returning nothing sends an empty result. MiniCPM5
reads text only, so return text.

The agent handles these cases for you:

- **Errors go to the model.** If `execute` throws, the error message is the call's result, and the
  model can try something else.
- **Bad calls never run.** A call that did not parse, names an unknown tool, or has missing or wrong
  arguments is answered with what is wrong.
- **Repeated failures are refused.** If the model repeats a call that just failed, with the same
  arguments and nothing succeeding in between, it gets "already failed" back, and the call does not
  run again.
- **Order.** Calls run one at a time, in the order the model wrote them.

### Events and results

Every event carries `round`, which counts the run's replies from 0. In order:

| Event | When | Fields |
| --- | --- | --- |
| `turn_start` | A reply begins. | |
| `prefill`, `reasoning`, `text`, `tool_call` | The model's own events, as from a session. | |
| `turn_end` | The reply is complete. | `result`: the turn's `finish_reason`, `message`, `usage` and `timings` |
| `tool_start` | A call is about to run. | `call` |
| `tool_end` | A call has its answer. | `call`, `message` (the `tool` message), `is_error`, `details` |

`run.result` resolves with:

| Field | Meaning |
| --- | --- |
| `finish_reason` | `stop` (the model answered), `length` (out of room; `limit` says which), `cancelled`, or `max_rounds` (it was still calling tools after `max_rounds` replies) |
| `content` | The last reply's answer. |
| `messages` | Everything the run added to the conversation. |
| `usage` | Token counts summed over every reply. |
| `rounds` | Number of replies. |

A failure, such as `ContextOverflowError` or `DeviceLostError`, rejects `run.result`, and iterating the
run throws the same error.

### Cancelling

Call `run.cancel()`, pass `signal` to `agent.run(input, { signal })`, or break out of the loop. A reply
in progress stops. A running tool sees its `signal` abort and is answered "may have partly completed".
Calls that had not started are answered "not run". Every call still gets an answer, so the conversation
remains valid.

### The conversation

`agent.messages` is the conversation, with the system prompt first. Each reply and each tool result is
added as soon as it completes, so it is always a valid history, even after a cancel or an error. A run
that produced no reply leaves it unchanged. You can edit it between runs.

The run waits for your loop at the end of each step. It goes on past a `turn_end` or `tool_end` only
after your loop has handled that event. So you can save the conversation before any tool runs:

```js
for await (const event of run) {
  if (event.type === "turn_end" || event.type === "tool_end") await save(JSON.stringify(agent.messages));
}
```

To restore a conversation later:

```js
const [system, ...rest] = JSON.parse(await load());
const agent = new Agent({ model, instructions: system.content, tools, messages: rest });
```

Other methods:

- **`run(input, options)`** takes text or a whole message. Its options add to the agent's own.
- **`running`** says whether a run is in progress. Only one run can go at a time.
- **`reset()`** clears the conversation but keeps the system prompt.
- **`dispose()`** cancels any run and frees the session the agent created. A session you passed in
  stays open.

### runAgent: the loop with your own history

`runAgent(session, history, options)` is the same loop, for code that stores the conversation itself.
It never changes the array you pass. Append what the run added:

```js
import { runAgent } from "./agent.js";

const session = model.create_session();
const run = runAgent(session, history, { tools, max_new_tokens: 4096 });
for await (const event of run) render(event);
history.push(...(await run.result).messages);
```

Push those messages unchanged. The replies are the session's own objects, which it re-renders exactly
as the model wrote them, so the next run computes only the new part of the prompt.