Xenova HF Staff

sync 2e7068faf55e

4eccb3a verified 7 days ago

4.39 kB

library_name: kernels
license: apache-2.0
tags:
  - kernel
  - webgpu
  - wgsl

ai.onnx.RotaryEmbedding

ai.onnx · standard ONNX operator · ONNX opset ≥ 23

Description

Implements ONNX opset-23 RotaryEmbedding for float16 and float32 tensors. Applies rotary positional embeddings (RoPE) by rotating each head's embedding vector using precomputed cos_cache and sin_cache values. A partial rotation can be applied by setting rotary_embedding_dim to rotate only a prefix of the head dimension. position_ids keeps its standard logical int64 type; valid positions are non-negative and bounded by the WebGPU-addressable cache, so the backend stores them losslessly as uint32. Other ONNX floating-point input types are not yet implemented.

See the ONNX RotaryEmbedding spec for the reference semantics.

Inputs

Name	Bind key	Logical dtype	WebGPU storage	Rank	Shape	Description	Presence
`X`	`x`	`T`	same as logical dtype	—	—	Input token embeddings. Shape is `(batch_size, sequence_length, hidden_size)` for rank 3 or `(batch_size, num_heads, sequence_length, head_size)` for rank 4. `head_size` must be even, and the `num_heads` attribute is required for rank-3 input.	required
`cos_cache`	`cos`	`T`	same as logical dtype	—	—	Precomputed cosine values. Without `position_ids`, shape is `(batch_size, sequence_length, rotary_dim/2)`; with `position_ids`, shape is `(max_sequence_length, rotary_dim/2)`.	required
`sin_cache`	`sin`	`T`	same as logical dtype	—	—	Precomputed sine values with the same shape and type as `cos_cache`.	required
`position_ids`	`positionIds`	`M`	`uint32`	`2`	—	Optional logical int64 per-token position indices of shape `(batch_size, sequence_length)`. Valid positions are non-negative cache-row indices and use uint32 WebGPU storage. When supplied, the 2D cache tables are gathered at these positions.	optional

Outputs

Name	Bind key	Logical dtype	Rank	Shape	Description	Presence
`Y`	`y`	`T`	same as `X`	same as `X`	Rotary-position-encoded tensor with the same shape and type as `X`.	required

Attributes

Attributes and default values (overridable per request):

Attribute	Default	Description
`interleaved`	`0`	Set to 1 to rotate using an interleaved pattern (even/odd elements), or 0 to split the head dimension into two contiguous halves. Default is 0.
`rotary_embedding_dim`	`0`	Number of head-dimension elements to rotate; `0` means rotate the full head dimension. When set, only the leading `rotary_embedding_dim` elements are rotated and the rest are passed through unchanged.
`num_heads`	—	Optional number of attention heads. ONNX requires this attribute when `X` is rank 3; it is unnecessary for rank-4 input because the head count is explicit in the shape.

Type constraints

Variable	Allowed dtypes
`T`	`float32`, `float16`
`M`	`int64`

Files

metadata.json — kernel metadata (id, digests, provenance)
manifest.json — the op contract (source of truth)
test.json — correctness cases
bench.json — benchmark + tuning cases
rotary-embedding.wgsl.jinja

Use with `@huggingface/kernels`

The loader derives every required output's shape and logical dtype from the manifest contract and this call. It then allocates the result tensors automatically.

The version: 1 option selects the published kernel contract; it is independent of any operator opset, contrib since_version, or model version.

Replace each *Data placeholder with a typed array containing the corresponding input data.

import { getKernel } from "@huggingface/kernels";

const kernel = await getKernel("webgpu-kernels/ai.onnx.RotaryEmbedding", { version: 1 });
const { y } = await kernel({
  x: { data: xData, shape: [1, 2, 1, 4] },
  cos: { data: cosData, shape: [16, 2] },
  sin: { data: sinData, shape: [16, 2] },
  positionIds: { data: positionIdsData, shape: [1, 1] },
});