README.md

---
license: other
tags:
- core-ai
- apple
- on-device
- diffusion-llm
- dllm
- llada
base_model:
- GSAI-ML/LLaDA-8B-Instruct
---

# LLaDA-8B dLLM — Core AI (on-device diffusion LLM)

**The model zoo's first diffusion language model (dLLM) for Apple Core AI.** Most on-device LLMs
are autoregressive — they write one token at a time, left to right. This one is a **masked
diffusion** model: it starts from a canvas of `[MASK]` tokens and fills them in **in parallel**,
committing the most-confident positions each step until the answer resolves. A different decoding
paradigm, running on-device on Apple Silicon.

- **Base:** [`GSAI-ML/LLaDA-8B-Instruct`](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct) (LLaMA-dense 8B, **bidirectional**, no causal mask)
- **Distillation:** [`d3LLM/d3LLM_LLaDA`](https://huggingface.co/d3LLM/d3LLM_LLaDA) (hao-ai-lab / NVIDIA) — pseudo-trajectory distillation that cuts the number of denoising steps hard (≈8 tokens committed per forward here)
- **Quantization:** int4 weight-only (per-block-32 body) + int8 head, **≈4.9 GB**
- **Runtime:** Apple Core AI (`coreai-core`), GPU

<!-- gen-cards:use-it begin id=llada-8b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
## Use it

▶️ **Run it (source)** — the [DiffuseChat runner](https://github.com/john-rocky/coreai-kit/tree/main/Examples/DiffuseChat)
(GUI + CLI, one app for every diffusion LM in the catalog):

```bash
git clone https://github.com/john-rocky/coreai-kit
open coreai-kit/Examples/DiffuseChat/DiffuseChat.xcodeproj
# → Run, then pick "LLaDA-8B (diffusion)" in the model picker

# agents / headless (macOS):
cd coreai-kit/Examples/DiffuseChat
swift run diffuse-cli --model llada-8b --prompt "What is the capital of France?"
```

💻 **Build with it** — complete; the glue is kit API, copy-paste runs:

```swift
import CoreAIKit

let dlm = try await KitDiffusionLM(catalog: "llada-8b")
let reply = try await dlm.reply(to: prompt)
// reply: the denoised answer — pass onStep: to watch the canvas fill in per forward
// (still-masked positions as ░), in parallel, not left-to-right
```

The take-home is [`Examples/DiffuseChat/Sources/QuickStart.swift`](https://github.com/john-rocky/coreai-kit/blob/main/Examples/DiffuseChat/Sources/QuickStart.swift)
— this exact code as one typed function, no UI; the CLI is an argument shell over it, and
the GUI renders the same live canvas.
The canvas is fixed (S=256 ≈ 210 generated tokens) and the whole history must fit — no
KV cache. `reply(messages:)` takes role/content turns and drops the oldest first.
Pass `onStep: nil` if you only want the final text.

**Integration checklist**

- SPM: `https://github.com/john-rocky/coreai-kit` → product **CoreAIKit**
- Info.plist: none needed
- Entitlements: none needed
- First run downloads the model — 5.3 GB (Mac) — then it loads from the
  local cache (Application Support; progress via the `downloadProgress` callback)
- Measure in Release — Debug is ~3× slower on per-token host work
<!-- gen-cards:use-it end -->

## What it looks like

The answer doesn't stream left-to-right — the canvas *denoises*. Mid-generation you see tokens pop
in out of order, e.g.:

```
48 + ░4 =░7░ clips░░░   →   48 + 24 = 72 clips altogether.
```

That parallel, out-of-order fill is the signature of a masked-diffusion LM.

## How to run it

Open it in **CoreAIChatMac** (the zoo's macOS chat app) — *Download Models…* → **LLaDA-8B
(diffusion)**, or point the app at a folder containing the `macos/` bundle. The host drives the
denoising loop over a single static, bidirectional forward (`main(input_ids[1,S]) → logits[1,S,vocab]`,
no KV cache).

## Bundle

- `macos/` — Core AI `.aimodel` (GPU) + `tokenizer/` + `metadata.json`
- `metadata.json` exposes the diffusion knobs (no recompile to retune):
  - `seq` — canvas length (`256` ⇒ answers up to ~210 tokens). No KV cache ⇒ the whole prompt+answer
    lives in `S`; per-step cost is ~linear in `S` (a larger canvas = fuller answers but slower steps).
  - `block_size` `32`, `threshold` `1.0` — the entropy threshold trades steps for speed; lower = more gradual, higher = fewer forwards (faster), too high degrades quality

## Performance (M4 Max, GPU)

| metric | value |
|---|---|
| throughput | **≈40 tok/s** (threshold 1.0, S=256) |
| TTFT | ~0.3 s |
| forwards (NFE) | ~22 for a full 210-token answer |
| size | 4.9 GB (int4 + int8 head) |

The per-step cost is a full bidirectional forward over the whole canvas (no KV cache) — the
distillation keeps the step count low. A delayed-KV-cache decode (process only the active region) is
the next speed lever.

## Roadmap

- Longer context (larger canvas) build
- Delayed-KV-cache decode (the d3LLM `generate_multi_block_kv_cache` lever) for higher throughput
- iPhone (h18p) build

## Credits & license

A community port. All credit to the upstream authors — **LLaDA** ([GSAI-ML](https://huggingface.co/GSAI-ML/LLaDA-8B-Instruct))
and **d3LLM** ([hao-ai-lab / NVIDIA](https://huggingface.co/d3LLM/d3LLM_LLaDA)). This bundle inherits
and is bound by their licenses — please review them before use. Core AI conversion only; no
retraining.

Part of the [Core AI model zoo](https://github.com/john-rocky/coreai-model-zoo).