| --- |
| library_name: kernels |
| license: apache-2.0 |
| tags: |
| - kernel |
| - webgpu |
| - wgsl |
| --- |
| # ai.onnx.TensorScatter |
|
|
| `ai.onnx` · standard ONNX operator · ONNX opset ≥ 24 |
|
|
| ## Description |
|
|
| Functionally updates a KV cache tensor by scattering an `update` tensor into the `past_cache` along a sequence axis, producing `present_cache` with the same shape. Each batch sample's update is written at the offset given by `write_indices` (zero if omitted), either linearly or in wrap-around `circular` fashion. |
|
|
| See the [ONNX `TensorScatter` spec](https://onnx.ai/onnx/operators/onnx__TensorScatter.html) for the reference semantics. |
|
|
| ## Inputs |
|
|
| | Name | Bind key | Logical dtype | WebGPU storage | Rank | Shape | Description | Presence | |
| | --- | --- | --- | --- | --- | --- | --- | --- | |
| | `past_cache` | `past` | `T` | runtime-selected; narrow integers and bool use 32-bit slots | — | — | Existing cache tensor with shape `(batch_size, ..., max_sequence_length, ...)`. | required | |
| | `update` | `update` | `T` | runtime-selected; narrow integers and bool use 32-bit slots | — | — | New values to scatter in, with the same shape as `past_cache` except the sequence dimension equals `sequence_length`. | required | |
| | `write_indices` | `writeIndices` | `I` | `uint32` | `1` | — | Logical int64 per-sample write offset into the cache sequence dimension; shape `(batch_size,)`, stored as uint32 by WebGPU, and assumed all zeros if absent. | optional | |
|
|
| ## Outputs |
|
|
| | Name | Bind key | Logical dtype | Rank | Shape | Description | Presence | |
| | --- | --- | --- | --- | --- | --- | --- | |
| | `present_cache` | `present` | `T` | same as `past_cache` | same as `past_cache` | Updated cache; same shape as `past_cache`. | required | |
|
|
| ## Attributes |
|
|
| Default values (overridable per request): |
|
|
| | Attribute | Default | Description | |
| | --- | --- | --- | |
| | `axis` | `-2` | Sequence dimension of `past_cache` and `update`; cannot be 0 (the batch dimension). Default is `-2`. | |
| | `mode` | `"linear"` | Write mode: `linear` requires `write_indices + sequence_length <= max_sequence_length`; `circular` wraps the write index modulo `max_sequence_length`. | |
|
|
| ## Type constraints |
|
|
| | Variable | Allowed dtypes | |
| | --- | --- | |
| | `T` | `float32`, `float16`, `int32`, `int16`, `int8`, `uint32`, `uint8`, `bool` | |
| | `I` | `int64` | |
|
|
| ## Files |
|
|
| - [`metadata.json`](build/webgpu/metadata.json) — kernel metadata (id, digests, provenance) |
| - [`manifest.json`](build/webgpu/manifest.json) — the op contract (source of truth) |
| - [`test.json`](build/webgpu/test.json) — correctness cases |
| - [`bench.json`](build/webgpu/bench.json) — benchmark + tuning cases |
| - [`scatter-flat-copy.wgsl.jinja`](build/webgpu/scatter-flat-copy.wgsl.jinja) |
| - [`tensor-scatter.wgsl.jinja`](build/webgpu/tensor-scatter.wgsl.jinja) |
|
|
| ## Use with `@huggingface/kernels` |
|
|
| The loader derives every required output's shape and logical dtype from the manifest contract and this call. |
| It then allocates the result tensors automatically. |
|
|
| The `version: 1` option selects the published kernel contract; it is independent of any operator opset, contrib `since_version`, or model version. |
|
|
| Replace each `*Data` placeholder with a typed array containing the corresponding input data. |
|
|
| ```js |
| import { getKernel } from "@huggingface/kernels"; |
| |
| const kernel = await getKernel("webgpu-kernels/ai.onnx.TensorScatter", { version: 1 }); |
| const { present } = await kernel({ |
| past: { data: pastData, shape: [1, 4, 2] }, |
| update: { data: updateData, shape: [1, 3, 2] }, |
| }); |
| ``` |
|
|
