README.md · patrickbdevaney/LLaDA2.2-mini-orin-nano at main

docs(card): add inline provenance citations, verification ledger, and sharded GGUF/128GB memory breakdown

e7ed6ff verified 12 days ago

8.5 kB

---
license: apache-2.0
base_model: inclusionAI/LLaDA2.2-mini
base_model_relation: quantized
pipeline_tag: text-generation
language:
  - en
  - zh
tags:
  - dllm
  - diffusion
  - diffusion-llm
  - llada
  - llada2.2
  - moe
  - quantized
  - gptq
  - hadamard
  - jetson
  - orin
  - orin-nano
  - cuda
  - edge-ai
---
<div align="center">

# LLaDA2.2-mini — Jetson Orin Nano Super 8GB (Packed Mixed 2/3/4-Bit Pure-CUDA Build)

[![Base Model](https://img.shields.io/badge/Base_Model-inclusionAI%2FLLaDA2.2--mini-blue)](https://huggingface.co/inclusionAI/LLaDA2.2-mini)
[![Engine](https://img.shields.io/badge/CUDA_Engine-patrickbdevaney%2Fllada2--orin--cuda-orange)](https://github.com/patrickbdevaney/llada2-orin-cuda)
[![Hardware](https://img.shields.io/badge/Target-Jetson_Orin_Nano_Super_8GB_(32K_Ctx)-76b900)](#use)
[![Speed](https://img.shields.io/badge/Throughput-~150--180_tok%2Fs_(~25ms%2Fstep)-brightgreen)](#use)
[![License](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://opensource.org/licenses/Apache-2.0)

**Mixed 2/3/4-bit Hadamard-rotated GPTQ + 8-bit non-expert pure-CUDA build of `inclusionAI/LLaDA2.2-mini` (16B MoE diffusion LLM, 1.4B active) serving 32K context at ~150–180 tok/s on an 8 GB NVIDIA Jetson Orin Nano Super.**

</div>

---

[`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) (16B MoE diffusion LLM, 1.4B active) quantized and packed for the pure-CUDA engine **[`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda)**, which serves it OpenAI-compatibly on a **Jetson Orin Nano Super 8GB** with **32K context**.

These files are not a `transformers` checkpoint: they are the engine's tile format and need that repo to run.

---

## ✨ Highlights

- **16B MoE Diffusion LLM in 8 GB Unified Memory with 32K Context:** Packs `inclusionAI/LLaDA2.2-mini` (16B total, 1.4B active) + a 32K-token `int4` KV cache into the 8 GB unified memory budget of a **Jetson Orin Nano Super**.
- **~150–180 tok/s Parallel Diffusion Decoding:** Runs a 32-token denoising step in **~25 ms at 1K context** on pinned Orin Nano Super clocks, delivering **~180 tok/s** on math/code/tool-call tasks at the server's default decode settings and **~150 tok/s** at the model card's settings.
- **Hadamard-Rotated Mixed 2/3/4-Bit GPTQ Experts + 8-Bit Non-Expert Path:** Routed experts are quantized with group-128 GPTQ in a Hadamard-rotated basis (`experts_L*.bin`, per-expert bit widths in `experts.json`), while attention, shared experts, layer-0 dense MLP (`nonexpert_L*.bin`), and `lm_head.bin` (with high/low nibble split planes in `lm_head_split.bin` for screened readout) are preserved at 8-bit and host-mapped `embed.bin` at FP16.
- **113–116 / 116 Scored Task Suite:** Achieves **113–116 / 116** across the engine's 96 generated math problems, 12 unit-tested code tasks, and 8 tool calls.

---

## 📋 Model Summary

| Property | Value |
|---|---|
| **Base Model** | [`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) (16B total MoE diffusion LLM, 1.4B active) |
| **Inference Engine** | [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda) (pure-CUDA engine + OpenAI-compatible server) |
| **Target Hardware** | **NVIDIA Jetson Orin Nano Super 8GB** (`32K` context length) |
| **Routed Experts (`experts_L*.bin`)** | Group-128 GPTQ in a Hadamard-rotated basis, mixed 2/3/4-bit per expert (`experts.json`) |
| **Non-Expert Path (`nonexpert_L*.bin`)** | 8-bit GPTQ (attention, shared expert, and layer-0 dense MLP) |
| **Readout & Embeddings** | 8-bit `lm_head.bin` + nibble-split `lm_head_split.bin` (screened readout); host `mmap` FP16 `embed.bin` |
| **KV Cache Format** | Runtime `int4` (keys per channel per 32-token block, values per token) |
| **Measured Throughput** | **~25 ms** / 32-token denoising step at 1K ctx (**~180 tok/s** default server settings, **~150 tok/s** model card settings) |
| **Scored Benchmark** | **113–116 / 116** (96 math problems + 12 unit-tested code tasks + 8 tool calls) |
| **License** | Apache-2.0 |

---

## What is in here

- `experts_L*.bin`: routed experts, GPTQ (group 128) in a Hadamard-rotated basis, per-expert bit widths chosen against
  a 32K-context memory budget (mostly 2-bit, some 3- and 4-bit; [`./experts.json`](./experts.json) lists them).
- `nonexpert_L*.bin`: attention, shared expert and the dense layer-0 MLP, 8-bit GPTQ.
- [`./lm_head.bin`](./lm_head.bin) (8-bit) and [`./lm_head_split.bin`](./lm_head_split.bin) (the same codes split into high/low nibble planes, used by the
  engine's screened readout); [`./embed.bin`](./embed.bin) (fp16 embeddings, memory-mapped on the host); [`./small.bin`](./small.bin) (norms, router,
  rotation signs); [`./engine.idx`](./engine.idx) (the index the engine loads); [`./model.json`](./model.json) (model metadata).
- Tokenizer ([`./tokenizer.json`](./tokenizer.json), [`./tokenization_llada2.py`](./tokenization_llada2.py)), [`./chat_template.jinja`](./chat_template.jinja), and [`./tool_declaration_ts.py`](./tool_declaration_ts.py) from the base model, unchanged, so this directory is also the
  server's `--model` directory.

KV cache at runtime: int4 (keys per channel per 32-token block, values per token).

---

## Use

```bash
git clone https://github.com/patrickbdevaney/llada2-orin-cuda && cd llada2-orin-cuda/engine && make
huggingface-cli download patrickbdevaney/LLaDA2.2-mini-orin-nano --local-dir ~/models/llada2mini-orin
cd ../server && python3 server.py --model ~/models/llada2mini-orin --engine ~/models/llada2mini-orin
```

On the Orin Nano Super with pinned clocks: ~25 ms per 32-token denoising step at 1K context, ~180 tok/s on
math/code/tool-call tasks at the server's default decode settings, ~150 tok/s at the model card's.

---

## Quality

Measured against the BF16 model on held-out text, the quantization costs mostly in the routed experts (see
[`QUANT_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/QUANT_PLAN.md) in the code repo). On the engine's scored test (96 generated math problems, 12 code tasks checked
against tests, 8 tool calls) it scores 113-116 / 116. The misses have correct arithmetic and a repeated final
digit; occasional repetition artifacts like this are the main visible cost of the 2-bit experts.

---

## 🔎 Provenance & Verification Ledger

Every claim, bit-width allocation, and kernel benchmark on this card can be audited directly against the files in this repository and the companion pure-CUDA engine repository:

| Claim / Artifact | Primary Verification Source |
|---|---|
| **Upstream Base Model (`16B` MoE Diffusion LLM, Apache-2.0)** | [`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) |
| **Pure-CUDA Engine & OpenAI Server (`sm_87` Orin Nano Super)** | [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda) (`main` @ `1695e8f`) · [`engine/ENGINE_DESIGN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/ENGINE_DESIGN.md) |
| **Mixed 2/3/4-Bit Hadamard-GPTQ Allocation & Model Selection** | [`QUANT_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/QUANT_PLAN.md) · [`MODEL_SELECTION_V2.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/MODEL_SELECTION_V2.md) · [`RESEARCH_AND_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/RESEARCH_AND_PLAN.md) |
| **Per-Expert Bit-Width Manifest & Engine Index** | [`./experts.json`](./experts.json) · [`./model.json`](./model.json) · [`./engine.idx`](./engine.idx) |
| **Screened Readout, Host-Mapped Embeddings & Small Tensors** | [`./lm_head.bin`](./lm_head.bin) · [`./lm_head_split.bin`](./lm_head_split.bin) · [`./embed.bin`](./embed.bin) · [`./small.bin`](./small.bin) |
| **CUDA Denoising & MoE GEMM Kernels** | [`engine/decode.cu`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/decode.cu) · [`engine/moe_gemm.cu`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/moe_gemm.cu) · [`server/server.py`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/server/server.py) |
---
## License
Apache-2.0, following the base model ([`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini)). Credit for the base model: inclusionAI, LLaDA2.2. Engine & quantization: [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda).

	---
	license: apache-2.0
	base_model: inclusionAI/LLaDA2.2-mini
	base_model_relation: quantized
	pipeline_tag: text-generation
	language:
	- en
	- zh
	tags:
	- dllm
	- diffusion
	- diffusion-llm
	- llada
	- llada2.2
	- moe
	- quantized
	- gptq
	- hadamard
	- jetson
	- orin
	- orin-nano
	- cuda
	- edge-ai
	---

	<div align="center">

	# LLaDA2.2-mini — Jetson Orin Nano Super 8GB (Packed Mixed 2/3/4-Bit Pure-CUDA Build)

	[![Base Model](https://img.shields.io/badge/Base_Model-inclusionAI%2FLLaDA2.2--mini-blue)](https://huggingface.co/inclusionAI/LLaDA2.2-mini)
	[![Engine](https://img.shields.io/badge/CUDA_Engine-patrickbdevaney%2Fllada2--orin--cuda-orange)](https://github.com/patrickbdevaney/llada2-orin-cuda)
	[![Hardware](https://img.shields.io/badge/Target-Jetson_Orin_Nano_Super_8GB_(32K_Ctx)-76b900)](#use)
	[![Speed](https://img.shields.io/badge/Throughput-~150--180_tok%2Fs_(~25ms%2Fstep)-brightgreen)](#use)
	[![License](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://opensource.org/licenses/Apache-2.0)

	Mixed 2/3/4-bit Hadamard-rotated GPTQ + 8-bit non-expert pure-CUDA build of `inclusionAI/LLaDA2.2-mini` (16B MoE diffusion LLM, 1.4B active) serving 32K context at ~150–180 tok/s on an 8 GB NVIDIA Jetson Orin Nano Super.

	</div>

	---

	[`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) (16B MoE diffusion LLM, 1.4B active) quantized and packed for the pure-CUDA engine [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda), which serves it OpenAI-compatibly on a Jetson Orin Nano Super 8GB with 32K context.

	These files are not a `transformers` checkpoint: they are the engine's tile format and need that repo to run.

	---

	## ✨ Highlights

	- 16B MoE Diffusion LLM in 8 GB Unified Memory with 32K Context: Packs `inclusionAI/LLaDA2.2-mini` (16B total, 1.4B active) + a 32K-token `int4` KV cache into the 8 GB unified memory budget of a Jetson Orin Nano Super.
	- ~150–180 tok/s Parallel Diffusion Decoding: Runs a 32-token denoising step in ~25 ms at 1K context on pinned Orin Nano Super clocks, delivering ~180 tok/s on math/code/tool-call tasks at the server's default decode settings and ~150 tok/s at the model card's settings.
	- Hadamard-Rotated Mixed 2/3/4-Bit GPTQ Experts + 8-Bit Non-Expert Path: Routed experts are quantized with group-128 GPTQ in a Hadamard-rotated basis (`experts_L.bin`, per-expert bit widths in `experts.json`), while attention, shared experts, layer-0 dense MLP (`nonexpert_L.bin`), and `lm_head.bin` (with high/low nibble split planes in `lm_head_split.bin` for screened readout) are preserved at 8-bit and host-mapped `embed.bin` at FP16.
	- 113–116 / 116 Scored Task Suite: Achieves 113–116 / 116 across the engine's 96 generated math problems, 12 unit-tested code tasks, and 8 tool calls.

	---

	## 📋 Model Summary

	\| Property \| Value \|
	\|---\|---\|
	\| Base Model \| [`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) (16B total MoE diffusion LLM, 1.4B active) \|
	\| Inference Engine \| [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda) (pure-CUDA engine + OpenAI-compatible server) \|
	\| Target Hardware \| NVIDIA Jetson Orin Nano Super 8GB (`32K` context length) \|
	\| *Routed Experts (`experts_L.bin`)** \| Group-128 GPTQ in a Hadamard-rotated basis, mixed 2/3/4-bit per expert (`experts.json`) \|
	\| *Non-Expert Path (`nonexpert_L.bin`)** \| 8-bit GPTQ (attention, shared expert, and layer-0 dense MLP) \|
	\| Readout & Embeddings \| 8-bit `lm_head.bin` + nibble-split `lm_head_split.bin` (screened readout); host `mmap` FP16 `embed.bin` \|
	\| KV Cache Format \| Runtime `int4` (keys per channel per 32-token block, values per token) \|
	\| Measured Throughput \| ~25 ms / 32-token denoising step at 1K ctx (~180 tok/s default server settings, ~150 tok/s model card settings) \|
	\| Scored Benchmark \| 113–116 / 116 (96 math problems + 12 unit-tested code tasks + 8 tool calls) \|
	\| License \| Apache-2.0 \|

	---

	## What is in here

	- `experts_L*.bin`: routed experts, GPTQ (group 128) in a Hadamard-rotated basis, per-expert bit widths chosen against
	a 32K-context memory budget (mostly 2-bit, some 3- and 4-bit; [`./experts.json`](./experts.json) lists them).
	- `nonexpert_L*.bin`: attention, shared expert and the dense layer-0 MLP, 8-bit GPTQ.
	- [`./lm_head.bin`](./lm_head.bin) (8-bit) and [`./lm_head_split.bin`](./lm_head_split.bin) (the same codes split into high/low nibble planes, used by the
	engine's screened readout); [`./embed.bin`](./embed.bin) (fp16 embeddings, memory-mapped on the host); [`./small.bin`](./small.bin) (norms, router,
	rotation signs); [`./engine.idx`](./engine.idx) (the index the engine loads); [`./model.json`](./model.json) (model metadata).
	- Tokenizer ([`./tokenizer.json`](./tokenizer.json), [`./tokenization_llada2.py`](./tokenization_llada2.py)), [`./chat_template.jinja`](./chat_template.jinja), and [`./tool_declaration_ts.py`](./tool_declaration_ts.py) from the base model, unchanged, so this directory is also the
	server's `--model` directory.

	KV cache at runtime: int4 (keys per channel per 32-token block, values per token).

	---

	## Use

	```bash
	git clone https://github.com/patrickbdevaney/llada2-orin-cuda && cd llada2-orin-cuda/engine && make
	huggingface-cli download patrickbdevaney/LLaDA2.2-mini-orin-nano --local-dir ~/models/llada2mini-orin
	cd ../server && python3 server.py --model ~/models/llada2mini-orin --engine ~/models/llada2mini-orin
	```

	On the Orin Nano Super with pinned clocks: ~25 ms per 32-token denoising step at 1K context, ~180 tok/s on
	math/code/tool-call tasks at the server's default decode settings, ~150 tok/s at the model card's.

	---

	## Quality

	Measured against the BF16 model on held-out text, the quantization costs mostly in the routed experts (see
	[`QUANT_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/QUANT_PLAN.md) in the code repo). On the engine's scored test (96 generated math problems, 12 code tasks checked
	against tests, 8 tool calls) it scores 113-116 / 116. The misses have correct arithmetic and a repeated final
	digit; occasional repetition artifacts like this are the main visible cost of the 2-bit experts.

	---

	## 🔎 Provenance & Verification Ledger

	Every claim, bit-width allocation, and kernel benchmark on this card can be audited directly against the files in this repository and the companion pure-CUDA engine repository:

	\| Claim / Artifact \| Primary Verification Source \|
	\|---\|---\|
	\| Upstream Base Model (`16B` MoE Diffusion LLM, Apache-2.0) \| [`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini) \|
	\| Pure-CUDA Engine & OpenAI Server (`sm_87` Orin Nano Super) \| [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda) (`main` @ `1695e8f`) · [`engine/ENGINE_DESIGN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/ENGINE_DESIGN.md) \|
	\| Mixed 2/3/4-Bit Hadamard-GPTQ Allocation & Model Selection \| [`QUANT_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/QUANT_PLAN.md) · [`MODEL_SELECTION_V2.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/MODEL_SELECTION_V2.md) · [`RESEARCH_AND_PLAN.md`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/RESEARCH_AND_PLAN.md) \|
	\| Per-Expert Bit-Width Manifest & Engine Index \| [`./experts.json`](./experts.json) · [`./model.json`](./model.json) · [`./engine.idx`](./engine.idx) \|
	\| Screened Readout, Host-Mapped Embeddings & Small Tensors \| [`./lm_head.bin`](./lm_head.bin) · [`./lm_head_split.bin`](./lm_head_split.bin) · [`./embed.bin`](./embed.bin) · [`./small.bin`](./small.bin) \|
	\| CUDA Denoising & MoE GEMM Kernels \| [`engine/decode.cu`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/decode.cu) · [`engine/moe_gemm.cu`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/engine/moe_gemm.cu) · [`server/server.py`](https://github.com/patrickbdevaney/llada2-orin-cuda/blob/main/server/server.py) \|

	---

	## License

	Apache-2.0, following the base model ([`inclusionAI/LLaDA2.2-mini`](https://huggingface.co/inclusionAI/LLaDA2.2-mini)). Credit for the base model: inclusionAI, LLaDA2.2. Engine & quantization: [`patrickbdevaney/llada2-orin-cuda`](https://github.com/patrickbdevaney/llada2-orin-cuda).