Yougori/Ling-mini-2.0-Megakernel

Ling-mini-2.0 Megakernel

Ling-mini-2.0 (a 16B mixture of experts with 1.4B parameters active per token) at 4 bits, running in a CUDA megakernel: one GPU launch runs every layer of up to 256 tokens and chooses each next token itself.

On one NVIDIA GeForce RTX 5090 Laptop GPU it writes 843 tokens per second in one chat, 1.68 times llama.cpp on the same model and GPU. With three chats at once it writes 1,337 tokens per second in total. The design and its measurements are described in the paper, megakernel.pdf.

Speed

Ling-mini-2.0, RTX 5090 Laptop GPU	llama.cpp	Megakernel
One chat	502	843	1.68×
Two chats at once, in total	784	1,096	1.40×
Three chats at once, in total	992	1,337	1.35×
Four chats at once, in total	1,164	1,371	1.18×
Five chats at once, in total	1,320	1,403	1.06×

Tokens per second while writing, greedy decoding, median of warm runs, measured on 6 November 2026 with the laptop GPU at its 175 W power limit. Every chat reads a 32-token prompt and then writes 256 tokens. llama.cpp is the CUDA build b12144 with bartowski's Q4_K_M file and flash attention on: llama-bench for one chat, llama-batched-bench for several. At eight chats llama.cpp reaches 1,616 tokens per second in total, because its batch shares each weight read between the chats; our chats do not share reads, and the provided apps run at most five at once.

What is different

A normal runtime runs each token through the model as hundreds of small GPU programs, one after another, and the GPU sits partly idle between them. This kernel is one program that never stops between layers or tokens.

1. One persistent block of threads sits on every streaming multiprocessor (SM) for the whole launch. When a step needs the result of the step before, the blocks wait at a grid barrier instead of ending and starting again.
2. Steps are merged so that each layer waits only four times. The router's sums are added inside the attention output step, and the next layer's q, k and v inside the expert down step. This works because RMS normalization divides each token's vector by a single number: every block adds its own rows' share at once, and the division follows after the wait.
3. The router picks its 8 experts (and the shared one) inside the launch, and the kernel picks the next token itself, so nothing returns to the CPU between tokens.
4. The word table's 2,048 hidden dimensions are stored in order of importance. The kernel reads the important half of every row, then reads in full only the 4 best rows of each warp. It picks the same word as the full table at 99 to 99.5% of positions; where it differs, the two words were near ties.
5. Several chats run as copies of the kernel side by side, each on its own share of the SMs, with one copy of the model in memory. One chat leaves the GPU waiting at its barriers a good part of the time, and the other chats fill those waits.

Quality

All attention, MLP, expert and word-table matrices are rounded to 4 bits with one fp16 scale per 64 numbers (round to nearest with a per-group clip search). The router, its bias, the normalization gains and the input embeddings keep their original values.

256 positions each	Web text	Python code
Same top token as the same 4-bit weights run in fp32	98.8%	98.8%
KL to the same 4-bit weights in fp32	0.0017	0.0017
Same top token as the original bf16 model	74.6%	91.0%
KL(original ∥ kernel) per token	0.222	0.070
Perplexity, original bf16 model	41.4	6.33
Perplexity, this kernel	45.7	6.83

The first two rows show that the kernel computes what its weights say. The rest is the cost of plain 4-bit rounding. llama.cpp's K-quants keep more of the original model; its quality was not measured here. In our chat tests the model answered simple questions and wrote correct short code.

Requirements

An NVIDIA GeForce RTX 50-series GPU (Blackwell, sm_120) with about 10 GB of free memory for one chat (12 GB for the several-chats app). Windows x64: megakernel_ling.dll is included. Other systems can build the library from megakernel_ling.cu (only Windows was tested). Python 3.10 or newer with PyTorch built for CUDA (install it from pytorch.org first), then the packages in requirements.txt.

Usage

hf download Yougori/Ling-mini-2.0-Megakernel --local-dir Ling-mini-2.0-Megakernel
cd Ling-mini-2.0-Megakernel
pip install -r requirements.txt

Chat in a terminal:

python chat.py

Several chats at once in a browser: run the server, then open http://127.0.0.1:8790. Every pane is its own conversation; while several write, each gets its own copy of the kernel on a share of the GPU.

python server.py

Measure the speed:

python bench.py
python bench.py --chats 3

In Python:

from transformers import AutoTokenizer
from ling_kernel import LingKernel

tok = AutoTokenizer.from_pretrained(".")
mk = LingKernel()
ids = tok.apply_chat_template([{"role": "user", "content": "What is the capital of France?"}],
                              tokenize=True, add_generation_prompt=True)
first = mk.forced(ids)[-1]                       # reads the prompt; its last step chooses the first token
new = [first] + mk.greedy(first, len(ids), 32)   # 32 more tokens
print(tok.decode(new, skip_special_tokens=True))

To rebuild the library with the CUDA Toolkit 12.8 or newer, run build.cmd from an x64 Native Tools Command Prompt for Visual Studio 2022.

Files

File
megakernel.pdf	the paper: design, measurements and limitations
packed/layers.safetensors	the 20 layers in the kernel's layout (8.7 GB)
packed/extras.safetensors	4-bit word table, its dimension order, layer 0's MLP, input embeddings, final norm (0.8 GB)
packed/meta.json	layout numbers and SHA-256 of both files
megakernel_ling.cu, megakernel_ling.dll	the kernel and its Windows build
ling_kernel.py	loads the model and runs the kernel
chat.py, server.py, multichat.html, bench.py	terminal chat, several-chats app, speed test
config.json, generation_config.json, tokenizer*.json, special_tokens_map.json	inclusionAI's, unchanged

Limitations

• Greedy decoding only: the kernel always takes the most likely token. There is no sampling.
• The chat apps use Ling's chat template with thinking off and keep at most 4,096 tokens per conversation.
• Speed depends on the GPU's power limit. On the same laptop with the GPU held to 90 W, one chat runs at about 660 to 720 tokens per second.
• The kernel's floating-point sums run in a varying order, so two runs of the same prompt can choose differently where two tokens are almost equally likely.
• Ling-mini-2.0's own limitations apply. It can be wrong, biased or repetitive. Do not use it for high-impact decisions.

Licenses

The 4-bit weights are derived from Ling-mini-2.0 by inclusionAI, which is licensed under the MIT License (LICENSE-Ling). The kernel and the Python code are licensed under the Apache License 2.0 (LICENSE). Changes from the original are listed in NOTICE. The configuration and tokenizer files are inclusionAI's, unchanged. The paper, megakernel.pdf, is licensed under CC BY 4.0 (PAPER_LICENSE.txt).

Citation

@misc{yougori2026megakernelpaper,
  title  = {A Decoding Megakernel for a 16B Mixture of Experts on a Laptop GPU},
  author = {Yougori},
  year   = {2026},
  url    = {https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel/blob/main/megakernel.pdf}
}

@misc{yougori2026lingmegakernel,
  title  = {Ling-mini-2.0 Megakernel},
  author = {Yougori},
  year   = {2026},
  url    = {https://huggingface.co/Yougori/Ling-mini-2.0-Megakernel}
}

@misc{inclusionai2025lingmini2,
  title  = {Ling-mini-2.0},
  author = {{inclusionAI}},
  year   = {2025},
  url    = {https://huggingface.co/inclusionAI/Ling-mini-2.0}
}