mlboydaisuke/Youtu-LLM-2B-CoreAI

Youtu-LLM-2B — Apple Core AI (.aimodel)

Youtu-LLM-2B (Tencent) converted to Apple Core AI for iOS 27 / macOS 27 (beta) — the zoo's first Multi-head Latent Attention (MLA) model that runs on iPhone, and its first dense MLA (GLM-4.7-Flash brought MLA to the zoo but as a 30B Mac-only MoE).

Youtu-LLM-2B is a dense DeepSeek-V2/V3-style MLA decoder: 1.96B params, 32 layers, kv_lora_rank 512 · q_lora_rank 1536 · qk_nope 128 + decoupled qk_rope 64 (head_dim 192) · v_head_dim 128 · interleaved RoPE θ=1.6e6 · dense SwiGLU FFN (6144) · 128K context · weight-tied head · Llama-3 tokenizer. It has a reasoning ("thinking") mode (<think>…</think>) and native agentic/tool-use ability.

The MLA decode caches only the compressed latent ([512] + [64] rope key per token, 2×[288] halves) instead of a full per-head K/V, and folds the KV up-projection into the query lift / value readout — a tiny KV cache that a custom absorbed-MLA flash-decode Metal kernel attends. Rides Apple's coreai-pipelined GPU engine via the decode-only loop-free export (async encode, on-GPU argmax sampling, on-device KV growth).

surface (int8 ship bundle)	prefill (S=1)	decode	numerics
M4 Max (release llm-runner, greedy)	95.9	102.8 tok/s	16/16 top-1 = HF fp32 oracle
iPhone 17 Pro (PipelinedBench p=128 g=256; in-app warm ~24)	20.5	~19 tok/s	16/16 · device ≡ Mac ≡ HF

Numerics: the authored Core AI model is token-exact to the fp32 HF reference (naive + absorbed forms, prefill cosine 1.000002, greedy 0 flips), and the int8 bundle running on the real GPU engine reproduces the oracle byte-for-byte, 16/16 on both prompts, on Mac and on the iPhone 17 Pro — the zoo ship gate.

Use it

CoreAIKit (SPM) — one line, on-device:

import CoreAIKit

let chat = try await ChatSession(catalog: "youtu-llm-2b")
let reply = try await chat.respond(to: "What can you do, offline?")
// downloads this repo once, then runs fully on-device; the <think> reasoning
// arrives as .thinking events, reply.content is the final answer

Or the ChatDemo runner (GUI + swift run chat-cli --model youtu-llm-2b), or the zoo's CoreAIChat app (pick Youtu-LLM 2B). 2B (≥2 GB) bundles need the host app's com.apple.developer.kernel.increased-memory-limit entitlement.

Engine contract (decode-only static-[1,1] graph): set COREAI_CHUNK_THRESHOLD=1 before engine creation (prefill runs as pipelined S=1 steps); don't call engine.warmup() (it warms query length 256) — a 1-token generate after load is the warmup. Chat template: <|begin_of_text|><|User|>…<|Assistant|> (thinking mode on → <think>…</think> then the answer).

Contents

• gpu-pipelined/youtu_llm_2b_decode_absorbed_msdpa/ — the int8 per-block-32 (body + head) decode bundle with the absorbed-MLA flash-decode Metal kernel: metadata.json, the .aimodel (LanguageBundle), and the tokenizer. ~2.2 GB.
• config.json — the source model config (for reference).

Conversion code and the full engineering notes live in the model zoo (conversion/export_youtu_decode_pipelined.py, models/macos/youtu.py + youtu_absorbed.py; the absorbed-MLA cache + flash-decode kernel are shared with GLM-4.7-Flash).

License

Weights follow Tencent's youtu-llm license (see license_link) — commercial use and redistribution of derivatives permitted with attribution; ⚠️ the license states Youtu-LLM is NOT intended for use within the European Union. The Core AI conversion adds no restrictions.