Akicou/inclusionAI_LLaDA2.2-flash-GGUF

LLaDA2.2-flash — GGUF

GGUF conversions of inclusionAI/LLaDA2.2-flash, a ~103B-parameter (13B active) Mixture-of-Experts diffusion language model (dLLM).

Published at Akicou/inclusionAI_LLaDA2.2-flash-GGUF.

⚠️ These files do not work with llama.cpp. LLaDA2 is a block diffusion model, not an autoregressive one — it denoises a block of masked tokens over several steps instead of predicting one token at a time. Use diffuse-cpp.

---

Files

File	Quant	Size	Notes
LLaDA2.2-flash-Q4_K_S.gguf.part-0000{1,2}-of-00002.gguf	Q4_K_S	59.1 GB	Recommended. Fits in ~64 GB RAM.
LLaDA2.2-flash-F16.gguf.part-0000{1..5}-of-00005.gguf	F16	205.9 GB	Full precision, for re-quantizing.

Hugging Face caps individual files at 50 GB, so each model is uploaded as raw byte-range parts. Rejoin them with cat before use (they are not llama.cpp gguf-split shards):

huggingface-cli download Akicou/inclusionAI_LLaDA2.2-flash-GGUF \
  --include "LLaDA2.2-flash-Q4_K_S.gguf.part-*" --local-dir .
cat LLaDA2.2-flash-Q4_K_S.gguf.part-*-of-00002.gguf > LLaDA2.2-flash-Q4_K_S.gguf

The zero-padded names sort correctly under a shell glob. Verify the result is 59086787040 bytes (Q4_K_S) or 205852119520 bytes (F16), then delete the parts.

---

Usage

./build/diffuse-cli \
  -m LLaDA2.2-flash-Q4_K_S.gguf \
  -p "Explain why the sky appears blue, in two or three sentences." \
  -n 96 -s 32 -t 32

Flag	Default	Description
-n	256	Tokens to generate
-s	32	Denoising steps per block
-t	4	Threads
--temp	0.0	Temperature (0 = argmax)
--threshold	0.95	Confidence needed to commit a token
--remasking	low_confidence	low_confidence or random
--no-editing	off	Disable Levenshtein editing
-ngl	0	GPU layers to offload

---

Quality check (Q4_K_S)

CPU-only, greedy (--temp 0), -n 96 -s 32.

Input

Explain why the sky appears blue, in two or three sentences.

Output

The sky appears blue due to a phenomenon called Rayleigh scattering. When
sunlight enters Earth's atmosphere, it collides with gas molecules and scatters
in all directions. Blue light, with a shorter wavelength, is scattered more
strongly than other colors, making it sky appear blue to our eyes.

59 tokens, stopping on <|role_end|>. The physics and structure are right; note the small grammatical slip ("making it sky appear blue"), the kind of artifact to expect from a 4-bit quant of a diffusion model.

Speed: ~0.1 tok/s on CPU only (no GPU offload). Diffusion decoding runs a full forward pass per denoising step, so throughput is dominated by steps × blocks, not token count. Lower -s to trade quality for speed.

---

Quantization

Source weights are bf16 safetensors (32 shards). Both passes stream tensors one at a time, so peak RAM stays near the largest tensor rather than the model:

1. Convert — tools/convert-llada2-fast.py, bf16 → F16 GGUF. Reads the safetensors shards by raw byte parsing (no PyTorch), converts bf16 → f16 in 4M-element numpy chunks, and writes metadata first then streams tensor data. The 256 per-expert matrices in each MoE layer are stacked into single 3-D tensors ([n_expert, out, in]) so the graph can use ggml_mul_mat_id. The BPE tokenizer (157,153 tokens / 156,635 merges) and chat template are embedded.
2. Quantize — tools/quantize_streaming.py --type q4_k_s, calling ggml's own quantizers through a small C shim (tools/quant_lib.c) via ctypes.

Q4_K_S took 151 min: 254 tensors quantized, 192 copied through unchanged.

Q4_K_S mix

Tensor group	Type	Why
Router (moe_gate, moe_gate_bias)	F32	Routing is winner-take-all; a wrong expert costs more than the bytes saved.
All norms (incl. QK-norm)	F32	Tiny, and scale errors propagate.
token_embd	F16
output.weight	Q6_K	Most quantization-sensitive matmul.
Attention (fused QKV, output proj)	Q4_K
Experts + shared expert + dense FFN	Q4_K	~97% of the weights.

Q4_K_S vs Q4_K_M here: within 30 MB of each other (59.09 vs 59.12 GB). The usual _M upgrades target the first/last FFN layers, but only layer 0 is dense — layers 1–31 are MoE and their expert tensors dominate the file. To get a meaningfully smaller file, drop token_embd from F16 to Q4_K (~0.65 GB).

---

Model details

Parameters	~103B total, 13B active per token
Layers	32 (layer 0 dense, 1–31 MoE)
Hidden size	4096
Attention	32 heads / 4 KV heads (GQA), head_dim 128, QK-norm
RoPE	partial — 64 of 128 dims, θ = 3e6, NeoX-style
Experts	256 routed (top-8) + 1 shared, expert FFN 1024
Routing	sigmoid + expert bias, block routing (capacity 48 per 32-token block)
Diffusion	block length 32, mask token 156895
Context	128K
Vocab	157,184

---

License & credit

Apache-2.0, inherited from the base model. All credit for the model itself goes to the inclusionAI team — see the LLaDA2.X repository and the technical report. This only redistributes converted and quantized weights.

@article{bie2026llada22,
  title={LLaDA2.2: Enabling Agentic Diffusion Language Models via Levenshtein Editing},
  author={Bie, Tiwei and others},
  year={2026}
}