README.md

---
license: mit
base_model: inclusionAI/Ling-3.0-tiny
library_name: transformers.js
pipeline_tag: text-generation
tags:
  - onnx
  - transformers.js
  - webgpu
  - onnxruntime
  - onnxruntime-genai
  - q4
  - q4f16
  - int4
  - cuda
  - moe
  - text-generation
---

# Ling-3.0-tiny ONNX — Q4 WebGPU + CUDA

Community ONNX conversion of
[`inclusionAI/Ling-3.0-tiny`](https://huggingface.co/inclusionAI/Ling-3.0-tiny),
a 7.9B-total / 1.3B-active-parameter hybrid reasoning MoE model.

> [!IMPORTANT]
> This is an independent community conversion, not an official inclusionAI
> release. Read the [original model card](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
> for training, evaluation, intended use, and base-model limitations.

| Target | Files | Runtime | Download |
|---|---|---|---:|
| Browser WebGPU | `onnx/model_q4f16.onnx` + 3 data shards | Transformers.js 4.2+ | 4.849 GB / 4.516 GiB |
| NVIDIA CUDA | `model.onnx` + `model.onnx.data` | ONNX Runtime GenAI | 4.849 GB / 4.516 GiB |

The WebGPU layout follows the standard Transformers.js `q4f16` contract used
by browser-oriented ONNX repositories: the graph is under `onnx/`, external
tensor data is split into three sub-2 GB files, activations and cache are FP16,
and hybrid recurrent-cache names use the Qwen3.5-compatible convention already
supported by Transformers.js.

## Run in the browser with WebGPU

Install Transformers.js 4.2 or newer:

```bash
npm install "@huggingface/transformers@^4.2.0"
```

```javascript
import { pipeline, TextStreamer } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'webbrain-one/Ling-3.0-tiny-ONNX',
  {
    device: 'webgpu',
    dtype: 'q4f16',
  },
);

const messages = [
  { role: 'user', content: 'Explain why the sky is blue in two sentences.' },
];

const output = await generator(messages, {
  max_new_tokens: 128,
  do_sample: false,
  tokenizer_encode_kwargs: { enable_thinking: false },
  streamer: new TextStreamer(generator.tokenizer, {
    skip_prompt: true,
    skip_special_tokens: true,
  }),
});

console.log(output[0].generated_text.at(-1)?.content);
```

For thinking mode, set `enable_thinking: true`. The original model card
recommends `temperature: 1.0`, `top_p: 0.95`, and `top_k: 20` when sampling in
thinking mode.

### Browser requirements

- A current desktop browser with WebGPU enabled; Chrome or Edge is recommended.
- Approximately 4.85 GB of model downloads on first load, plus browser cache.
- Enough GPU memory for all weights, runtime buffers, state, and the requested
  context. The 1.3B active-parameter figure reduces compute, but all 7.9B model
  parameters still need to be stored.
- Start with a short prompt and modest `max_new_tokens`, then increase context
  after confirming memory use on the target device.

The validation machine had two discrete NVIDIA GPUs, and Chromium selected the
display-connected adapter. On multi-GPU systems, check the adapter selected by
the browser rather than assuming it will match a CUDA compute workload.

This graph depends on WebGPU implementations of `MatMulNBits`, `QMoE`,
`LinearAttention`, `CausalConvWithState`, and `GroupQueryAttention`. It is not a
WASM/CPU fallback model.

## Use from WebBrain

Use the same repository ID and standard Transformers.js settings:

```text
model: webbrain-one/Ling-3.0-tiny-ONNX
device: webgpu
dtype: q4f16
task: text-generation
```

WebBrain should select `onnx/model_q4f16.onnx` and fetch the three external-data
files declared by `config.json`.

## Run the CUDA variant with Python

The repository also retains the separately validated CUDA-targeted ONNX Runtime
GenAI graph.

```bash
pip install "onnxruntime-gpu>=1.28.0" \
  "onnxruntime-genai-cuda>=0.15.2" \
  "transformers>=4.57,<5"
```

```python
import numpy as np
import onnxruntime_genai as og
from transformers import AutoTokenizer

model_dir = "Ling-3.0-tiny-ONNX"
model = og.Model(model_dir)
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)

input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain why the sky is blue."}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="np",
    enable_thinking=False,
)

params = og.GeneratorParams(model)
params.set_search_options(
    max_length=int(input_ids.shape[-1]) + 128,
    do_sample=False,
)

generator = og.Generator(model, params)
generator.append_tokens(np.asarray(input_ids[0], dtype=np.int32))
prompt_length = int(input_ids.shape[-1])

while not generator.is_done():
    generator.generate_next_token()

print(tokenizer.decode(generator.get_sequence(0)[prompt_length:], skip_special_tokens=True))
```

## Architecture and quantization

- 18 Kimi Delta Attention layers with recurrent and convolution state
- 6 Multi-Latent Attention layers with KV cache
- 23 sparse MoE layers with 128 routed experts, top-8 group-limited routing,
  expert bias, and one shared expert
- one dense MLP layer at the start of the decoder
- symmetric Q4/block-32 dense and routed-expert weights
- FP16 embeddings, activations, recurrent state, and KV cache
- FP32 MoE router weights and routing math

The ONNX graph contains 235 `MatMulNBits`, 23 `QMoE`, 18 `LinearAttention`,
18 `CausalConvWithState`, and 6 `GroupQueryAttention` nodes.

## Validation

- The complete CUDA artifact generated successfully on an NVIDIA GeForce RTX
  5090 with ONNX Runtime GenAI 0.15.2 and ONNX Runtime GPU 1.28.0.
- The WebGPU repack was verified tensor-for-tensor against the CUDA graph; all
  4,835,749,912 external tensor bytes are identical.
- Transformers.js 4.2.0 loads the config, dispatches `Qwen3_5ForCausalLM`, finds
  all 18 recurrent/conv caches and 6 attention caches, and applies the original
  Ling tokenizer/chat template.
- A full remote-repository test passed in Chrome 150 with Transformers.js 4.2.0
  on an NVIDIA T400 4GB WebGPU adapter using Windows shared-memory
  oversubscription. First load/session creation took 925.87 seconds; a
  deterministic 16-token generation took 90.18 seconds and produced a coherent
  answer. This is a compatibility smoke test, not a performance benchmark.

## Limitations

- Q4 quantization can change outputs and quality relative to the original BF16
  checkpoint. No benchmark parity claim is made here.
- Browser support, GPU limits, shader compilation time, and memory behavior vary
  by operating system, browser version, and GPU driver.
- The advertised 131,072-token context is architectural; practical browser
  context is limited by available GPU memory.
- The WebGPU config uses Transformers.js's existing `qwen3_5_text` hybrid-cache
  adapter solely as a runtime compatibility layer. The underlying graph and
  weights remain Ling/Bailing Hybrid, preserved in
  `config_bailing_original.json`.

## Attribution and license

The model architecture, checkpoint, tokenizer, and chat template are by
[`inclusionAI`](https://huggingface.co/inclusionAI). This conversion retains the
base model's MIT license. Please cite and credit the
[original Ling-3.0-tiny release](https://huggingface.co/inclusionAI/Ling-3.0-tiny)
when using or redistributing this artifact.