Upload folder using huggingface_hub

7e0a8c6 verified 2 days ago

10.8 kB

---
license: other
license_name: youtu-llm
base_model: tencent/Youtu-LLM-A7B-Base
tags:
  - llama.cpp
  - gguf
  - patch
---
# llama.cpp support for Youtu-LLM-A7B-Base

Patch that adds `YoutuVLMoEForConditionalGeneration` (`youtu-moe` arch) to
[llama.cpp](https://github.com/ggml-org/llama.cpp) — text **and** vision.

| Item | Value |
|---|---|
| Upstream model | [`tencent/Youtu-LLM-A7B-Base`](https://huggingface.co/tencent/Youtu-LLM-A7B-Base) @ `6b0d02b2912c2975806db5a4922a1e091fa2f5d3` |
| llama.cpp base commit | `e0f142d5ddc1461ce721644d92af33db23545105` |
| GGUF weights | [`gatilin/Youtu-LLM-A7B-Base-GGUF`](https://huggingface.co/gatilin/Youtu-LLM-A7B-Base-GGUF) |
| Status | text + vision verified end-to-end on real 120B weights (macOS / Metal) |

**中文摘要**：本仓库提供一个 llama.cpp 补丁，为 Youtu-LLM-A7B-Base 增加
`youtu-moe` 架构支持（KDA 线性注意力 + GatedMLA 混合、256 专家 QB 路由 MoE、
Qwen3-VL 视觉塔）。已验证文本与图文推理，可导出 mmproj、可量化。下面第 6 节记录了
几个踩过的坑（RoPE 布局、mrope sections、模态感知专家偏置、`</think>` 残留）。

---

## 1. Quick start

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout e0f142d5ddc1461ce721644d92af33db23545105
git apply ../llama.cpp-youtu-moe.patch
cmake -B build -DGGML_METAL=ON -DLLAMA_BUILD_SERVER=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j --target llama-cli llama-server llama-mtmd-cli llama-quantize
```

> On macOS, `cmake` / `ninja` are often only inside a Python venv (`pip install cmake ninja`).

Text inference:

```bash
./build/bin/llama-cli -m youtu-moe-a7b-Q4_K_M.gguf --jinja \
  -p "用一句话解释什么是光合作用。" -st --simple-io -ngl 99 -n 128
```

Image inference:

```bash
./build/bin/llama-mtmd-cli -m youtu-moe-a7b-Q4_K_M.gguf \
  --mmproj youtu-vl-mmproj-f16.gguf --image test.jpeg \
  -p "<|vision_start|><__media__><|vision_end|>描述这张图片的内容。" -n 200 -ngl 99
```

The `<|vision_start|>…<|vision_end|>` wrapper is **required**. `chat_template.jinja`
in the upstream repo has no image branch, so `--jinja` cannot inject the image
placeholder — feed the raw prompt instead. `<__media__>` is mtmd's media marker;
do **not** also write `<|image_pad|>`.

## 2. What's in the patch

| File | Change |
|---|---|
| `src/models/youtu-moe.cpp` | **new** — the arch: graph, KDA layer, MLA layer, QB-MoE, modality input |
| `src/models/models.h` | **new** — `llama_model_youtu_moe` class + graph members |
| `conversion/youtu_moe.py` | **new** — HF → GGUF converter (`YoutuVLMoEForConditionalGeneration`) |
| `conversion/youtu_vl_vision.py` | **new** — visual tower key mapping for `--mmproj` |
| `src/llama-arch.cpp` / `.h` | register `LLM_ARCH_YOUTU_MOE` and its metadata keys |
| `src/llama-model.cpp` | register the arch; route it to the IMROPE group |
| `src/llama-graph.cpp` | `build_inp_mem_hybrid_k()` helper (KDA + MLA hybrid KV cache) |
| `src/llama-context.cpp` | hybrid recurrent state bookkeeping |
| `gguf-py/gguf/constants.py` | tensor/arch/metadata key names |
| `conversion/__init__.py` | register the converter module |

## 3. Conversion

```bash
# put the 24 shards under models/Youtu-LLM-A7B-Base/ first
bash convert_youtu.sh              # 7 steps, idempotent
```

`convert_youtu.sh` runs: weight integrity check → export mmproj → BF16 GGUF →
metadata self-check → Q4_K_M quantize → text smoke test → vision smoke test.

Two flags matter:

* `--use-temp-file` is **mandatory** for BF16. Without it the converter keeps every
  tensor in RAM and a 220 GiB model OOMs on a 128 GiB machine.
* Quantize is done **from the BF16 GGUF**, not from safetensors.

Produced artifacts:

| File | Size |
|---|---|
| `youtu-moe-a7b-bf16.gguf` | 220 GiB (1090 tensors) |
| `youtu-moe-a7b-Q4_K_M.gguf` | 66 GiB |
| `youtu-vl-mmproj-f16.gguf` | 867 MiB |

## 4. Architecture notes

Youtu-LLM-A7B: 48 layers, hidden 3072, ~120B total / ~7B active, vocab 129280,
context 262144, 262144-token training window over 11.5T multimodal tokens.

**Hybrid attention.** 3 KDA (linear attention) layers followed by 1 GatedMLA layer,
repeated 12 times. KDA reuses llama.cpp's gated-delta-net recurrent path
(`llm_build_delta_net_base`, `build_recurrent_attn`, conv state + ssm state,
L2-normalised q/k, gated lower bound −5.0). MLA is the DeepSeek-style low-rank
path with `q_lora_rank=1024`, `kv_lora_rank=512`, `qk_nope_head_dim=128`,
`qk_rope_head_dim=64`, `v_head_dim=128`, plus a **head-wise output gate**
(`[32, 3072]`) applied after the attention output projection.

**MoE.** 256 routed experts, top-8, SwiGLU, width 1024, plus 1 shared expert.
Routing is *quantile-balancing* with a **modality-aware expert bias**
(`qb_beta`, shape `[2, 256]`: row 0 = text, row 1 = image/video):

```
selection = router_logits - qb_beta[modality]
top8      = argmax8(selection)
weights   = sigmoid(router_logits[top8]) / (sum + 1e-20) * 2.83
```

Layer 0 is a dense FFN (12288).

**Vision.** Qwen3-VL-MoE-style ViT + merger (`clip.projector_type=qwen3vl_merger`),
exported as a separate mmproj.

## 5. Gotchas that cost real debugging time

### 5.1 RoPE layout: ggml IMROPE is NEOX, the official model is not

`ggml`'s `MROPE` / `IMROPE` **force NEOX rotation** (first half paired with second
half), while the official MLA implementation does `cat(q[0::2], q[1::2])` +
`rotate_half` — i.e. **adjacent pairs**. The two must be reconciled by
**permuting the weights at conversion time**:

```python
# wq_b: per-head [q_nope(128) | q_pe(64)]
q  = w.reshape(32, 128 + 64, -1)
pe = q[:, 128:]
w  = torch.cat([q[:, :128], pe[:, 0::2], pe[:, 1::2]], dim=1).reshape(32 * 192, -1)
# wkv_a_proj_with_mqa: [kv_lora(512) | k_pe(64)]
pe = w[512:]
w  = torch.cat([w[:512], pe[0::2], pe[1::2]], dim=0)
```

Verified numerically: after the permutation the per-element error vs. the official
implementation is **1e-16**; without it the relative error is **1.33** (garbage).

### 5.2 mrope sections must be `[11, 10, 10, 0]`, not `[12, 10, 10]`

The official config says `mrope_section=[12, 10, 10]` (sum = 32 = `qk_rope_head_dim/2`).
But ggml computes `sector = (freq_index / 2) % sum(sections)`. With `sum = 32`,
`freq 31` lands in the **extra** slot and reads `pos[3]` — which is always 0 for text
tokens, silently corrupting one frequency. Setting `sections[0] = 11` (sum = 31) maps
`freq 31 → sector 0 → T` and leaves every other frequency unchanged, matching the
official `_apply_interleaved_mrope` frequency-by-frequency (verified by exhaustive check).

### 5.3 Modality routing is decided by "is this row an embedding", not by token id

mtmd injects image embeddings via `llama_batch_ext_add_embd`, so for a pure-vision
batch `ubatch->token == nullptr` while `ubatch->type` / embeddings are set. The
graph input `llm_graph_input_youtu_modality` therefore emits `1` for every token of
an embedding-only batch, `1` where `ubatch->type != 0` otherwise, and `0` for plain
text. Two rows of `qb_beta` differ substantially (mean |Δ| = 0.387, max 2.63,
100 % non-zero), so getting this wrong visibly changes expert selection.

Also: when the last layer trims `inpL` with `inp_out_ids`, the modality ids must be
trimmed with the same `ggml_get_rows(...)` call, or the shapes stop matching.

### 5.4 The model always emits a leading `</think>`

This is a **base** model — the upstream README says raw text continuation is the
recommended mode and `chat_template.jinja` is only provided "for further purpose".
It nonetheless emits `</think>` (token **128014**) as its very first assistant-side
token, closing the think block the template pre-opened:

```
prompt:  <｜Assistant｜><think>      <- template
model:   </think>光合作用是…          <- generated
```

It does this even with a bare `<｜User｜>…<｜Assistant｜>` prompt that contains no
think markers at all, and even if the prompt already ends with `</think>`.
`</think>` is token type **USER_DEFINED (4)**, so llama.cpp prints it as ordinary text.
llama.cpp's auto parser (`common/chat-diff-analyzer.cpp`) cannot classify it: the
official template renders `<think>` in thinking mode and `</think>` otherwise, which
is a **replacement** diff, and `compare_thinking_enabled()` only recognises
**append-style** diffs (its 64-char tail anchor never matches on the other side) →
`reasoning_mode` is judged `NONE` and the marker stays in `message.content`.
`conversion/youtu_moe.py` therefore rewrites the non-thinking branch to emit
`<think></think>` (semantically identical — an empty think block), which turns the
diff into an append and makes the analyzer report `reasoning_mode: TAG_BASED`
(`-lv 5`). The parse itself still does not strip the token in current llama.cpp
(`p.prefix(generation_prompt, reasoning_start)` anchors on a prompt literal that is
not present in the generated text), so `strip_think.py` removes it on the output side.
Tried and **not** recommended: banning token 128014 via `logit_bias`. The `</think>`
disappears, but the model then believes it is still inside the think block and
starts emitting quasi-reasoning prose — strictly worse.

### 5.5 Small mtmd-CLI annoyances

* Log timestamps have **four** dot-separated groups (`0.05.132.418 I …`); a
  three-group regex will not filter them.
* `llama-mtmd-cli` prints the generated text with `LOG()`, so `--log-disable`
  swallows the answer too (symptom: empty output).

## 6. Known limitations

* No reasoning extraction into `reasoning_content` (see 5.4) — use `strip_think.py`.
* Only Q4_K_M and BF16 are published; other quant types are untested.
* KDA recurrent path has only been validated on Metal (macOS). CUDA/CPU are expected
  to work but have not been run.
* Small-text OCR is limited by the default 300 image tokens; pass
  `--image-min-tokens 1024` for dense documents.

## 7. Verified output

Text (Q4_K_M, M3 Max 128 GiB, prefill ≈ 38 t/s, decode ≈ 14 t/s):

```
> 用一句话解释什么是光合作用。
光合作用是植物、藻类和某些细菌利用光能将二氧化碳和水转化为有机物（如葡萄糖）并释放氧气的过程。
```

Vision — the 1969-07-21 New York Times front page (mtmd's bundled `test-1.jpeg`):
masthead, date `MONDAY, JULY 21, 1969`, `LATE CITY EDITION`, `10 CENTS`,
`VOL. CXVIII. NO. 46,731`, headline `MEN WALK ON MOON`, subhead
`ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG` — all correct.

## 8. License

The patch follows llama.cpp's MIT license. The weights and the model architecture
follow the upstream **Youtu-LLM license** (`license: other`); see
[`tencent/Youtu-LLM-A7B-Base/LICENSE.txt`](https://huggingface.co/tencent/Youtu-LLM-A7B-Base/blob/main/LICENSE.txt).

	---
	license: other
	license_name: youtu-llm
	base_model: tencent/Youtu-LLM-A7B-Base
	tags:
	- llama.cpp
	- gguf
	- patch
	---

	# llama.cpp support for Youtu-LLM-A7B-Base

	Patch that adds `YoutuVLMoEForConditionalGeneration` (`youtu-moe` arch) to
	[llama.cpp](https://github.com/ggml-org/llama.cpp) — text and vision.

	\| Item \| Value \|
	\|---\|---\|
	\| Upstream model \| [`tencent/Youtu-LLM-A7B-Base`](https://huggingface.co/tencent/Youtu-LLM-A7B-Base) @ `6b0d02b2912c2975806db5a4922a1e091fa2f5d3` \|
	\| llama.cpp base commit \| `e0f142d5ddc1461ce721644d92af33db23545105` \|
	\| GGUF weights \| [`gatilin/Youtu-LLM-A7B-Base-GGUF`](https://huggingface.co/gatilin/Youtu-LLM-A7B-Base-GGUF) \|
	\| Status \| text + vision verified end-to-end on real 120B weights (macOS / Metal) \|

	中文摘要：本仓库提供一个 llama.cpp 补丁，为 Youtu-LLM-A7B-Base 增加
	`youtu-moe` 架构支持（KDA 线性注意力 + GatedMLA 混合、256 专家 QB 路由 MoE、
	Qwen3-VL 视觉塔）。已验证文本与图文推理，可导出 mmproj、可量化。下面第 6 节记录了
	几个踩过的坑（RoPE 布局、mrope sections、模态感知专家偏置、`</think>` 残留）。

	---

	## 1. Quick start

	```bash
	git clone https://github.com/ggml-org/llama.cpp
	cd llama.cpp
	git checkout e0f142d5ddc1461ce721644d92af33db23545105
	git apply ../llama.cpp-youtu-moe.patch

	cmake -B build -DGGML_METAL=ON -DLLAMA_BUILD_SERVER=ON -DCMAKE_BUILD_TYPE=Release
	cmake --build build -j --target llama-cli llama-server llama-mtmd-cli llama-quantize
	```

	> On macOS, `cmake` / `ninja` are often only inside a Python venv (`pip install cmake ninja`).

	Text inference:

	```bash
	./build/bin/llama-cli -m youtu-moe-a7b-Q4_K_M.gguf --jinja \
	-p "用一句话解释什么是光合作用。" -st --simple-io -ngl 99 -n 128
	```

	Image inference:

	```bash
	./build/bin/llama-mtmd-cli -m youtu-moe-a7b-Q4_K_M.gguf \
	--mmproj youtu-vl-mmproj-f16.gguf --image test.jpeg \
	-p "<\|vision_start\|><__media__><\|vision_end\|>描述这张图片的内容。" -n 200 -ngl 99
	```

	The `<\|vision_start\|>…<\|vision_end\|>` wrapper is required. `chat_template.jinja`
	in the upstream repo has no image branch, so `--jinja` cannot inject the image
	placeholder — feed the raw prompt instead. `<__media__>` is mtmd's media marker;
	do not also write `<\|image_pad\|>`.

	## 2. What's in the patch

	\| File \| Change \|
	\|---\|---\|
	\| `src/models/youtu-moe.cpp` \| new — the arch: graph, KDA layer, MLA layer, QB-MoE, modality input \|
	\| `src/models/models.h` \| new — `llama_model_youtu_moe` class + graph members \|
	\| `conversion/youtu_moe.py` \| new — HF → GGUF converter (`YoutuVLMoEForConditionalGeneration`) \|
	\| `conversion/youtu_vl_vision.py` \| new — visual tower key mapping for `--mmproj` \|
	\| `src/llama-arch.cpp` / `.h` \| register `LLM_ARCH_YOUTU_MOE` and its metadata keys \|
	\| `src/llama-model.cpp` \| register the arch; route it to the IMROPE group \|
	\| `src/llama-graph.cpp` \| `build_inp_mem_hybrid_k()` helper (KDA + MLA hybrid KV cache) \|
	\| `src/llama-context.cpp` \| hybrid recurrent state bookkeeping \|
	\| `gguf-py/gguf/constants.py` \| tensor/arch/metadata key names \|
	\| `conversion/__init__.py` \| register the converter module \|

	## 3. Conversion

	```bash
	# put the 24 shards under models/Youtu-LLM-A7B-Base/ first
	bash convert_youtu.sh # 7 steps, idempotent
	```

	`convert_youtu.sh` runs: weight integrity check → export mmproj → BF16 GGUF →
	metadata self-check → Q4_K_M quantize → text smoke test → vision smoke test.

	Two flags matter:

	* `--use-temp-file` is mandatory for BF16. Without it the converter keeps every
	tensor in RAM and a 220 GiB model OOMs on a 128 GiB machine.
	* Quantize is done from the BF16 GGUF, not from safetensors.

	Produced artifacts:

	\| File \| Size \|
	\|---\|---\|
	\| `youtu-moe-a7b-bf16.gguf` \| 220 GiB (1090 tensors) \|
	\| `youtu-moe-a7b-Q4_K_M.gguf` \| 66 GiB \|
	\| `youtu-vl-mmproj-f16.gguf` \| 867 MiB \|

	## 4. Architecture notes

	Youtu-LLM-A7B: 48 layers, hidden 3072, ~120B total / ~7B active, vocab 129280,
	context 262144, 262144-token training window over 11.5T multimodal tokens.

	Hybrid attention. 3 KDA (linear attention) layers followed by 1 GatedMLA layer,
	repeated 12 times. KDA reuses llama.cpp's gated-delta-net recurrent path
	(`llm_build_delta_net_base`, `build_recurrent_attn`, conv state + ssm state,
	L2-normalised q/k, gated lower bound −5.0). MLA is the DeepSeek-style low-rank
	path with `q_lora_rank=1024`, `kv_lora_rank=512`, `qk_nope_head_dim=128`,
	`qk_rope_head_dim=64`, `v_head_dim=128`, plus a head-wise output gate
	(`[32, 3072]`) applied after the attention output projection.

	MoE. 256 routed experts, top-8, SwiGLU, width 1024, plus 1 shared expert.
	Routing is quantile-balancing with a modality-aware expert bias
	(`qb_beta`, shape `[2, 256]`: row 0 = text, row 1 = image/video):

	```
	selection = router_logits - qb_beta[modality]
	top8 = argmax8(selection)
	weights = sigmoid(router_logits[top8]) / (sum + 1e-20) * 2.83
	```

	Layer 0 is a dense FFN (12288).

	Vision. Qwen3-VL-MoE-style ViT + merger (`clip.projector_type=qwen3vl_merger`),
	exported as a separate mmproj.

	## 5. Gotchas that cost real debugging time

	### 5.1 RoPE layout: ggml IMROPE is NEOX, the official model is not

	`ggml`'s `MROPE` / `IMROPE` force NEOX rotation (first half paired with second
	half), while the official MLA implementation does `cat(q[0::2], q[1::2])` +
	`rotate_half` — i.e. adjacent pairs. The two must be reconciled by
	permuting the weights at conversion time:

	```python
	# wq_b: per-head [q_nope(128) \| q_pe(64)]
	q = w.reshape(32, 128 + 64, -1)
	pe = q[:, 128:]
	w = torch.cat([q[:, :128], pe[:, 0::2], pe[:, 1::2]], dim=1).reshape(32 * 192, -1)

	# wkv_a_proj_with_mqa: [kv_lora(512) \| k_pe(64)]
	pe = w[512:]
	w = torch.cat([w[:512], pe[0::2], pe[1::2]], dim=0)
	```

	Verified numerically: after the permutation the per-element error vs. the official
	implementation is 1e-16; without it the relative error is 1.33 (garbage).

	### 5.2 mrope sections must be `[11, 10, 10, 0]`, not `[12, 10, 10]`

	The official config says `mrope_section=[12, 10, 10]` (sum = 32 = `qk_rope_head_dim/2`).
	But ggml computes `sector = (freq_index / 2) % sum(sections)`. With `sum = 32`,
	`freq 31` lands in the extra slot and reads `pos[3]` — which is always 0 for text
	tokens, silently corrupting one frequency. Setting `sections[0] = 11` (sum = 31) maps
	`freq 31 → sector 0 → T` and leaves every other frequency unchanged, matching the
	official `_apply_interleaved_mrope` frequency-by-frequency (verified by exhaustive check).

	### 5.3 Modality routing is decided by "is this row an embedding", not by token id

	mtmd injects image embeddings via `llama_batch_ext_add_embd`, so for a pure-vision
	batch `ubatch->token == nullptr` while `ubatch->type` / embeddings are set. The
	graph input `llm_graph_input_youtu_modality` therefore emits `1` for every token of
	an embedding-only batch, `1` where `ubatch->type != 0` otherwise, and `0` for plain
	text. Two rows of `qb_beta` differ substantially (mean \|Δ\| = 0.387, max 2.63,
	100 % non-zero), so getting this wrong visibly changes expert selection.

	Also: when the last layer trims `inpL` with `inp_out_ids`, the modality ids must be
	trimmed with the same `ggml_get_rows(...)` call, or the shapes stop matching.

	### 5.4 The model always emits a leading `</think>`

	This is a base model — the upstream README says raw text continuation is the
	recommended mode and `chat_template.jinja` is only provided "for further purpose".
	It nonetheless emits `</think>` (token 128014) as its very first assistant-side
	token, closing the think block the template pre-opened:

	```
	prompt: <｜Assistant｜><think> <- template
	model: </think>光合作用是… <- generated
	```

	It does this even with a bare `<｜User｜>…<｜Assistant｜>` prompt that contains no
	think markers at all, and even if the prompt already ends with `</think>`.
	`</think>` is token type USER_DEFINED (4), so llama.cpp prints it as ordinary text.

	llama.cpp's auto parser (`common/chat-diff-analyzer.cpp`) cannot classify it: the
	official template renders `<think>` in thinking mode and `</think>` otherwise, which
	is a replacement diff, and `compare_thinking_enabled()` only recognises
	append-style diffs (its 64-char tail anchor never matches on the other side) →
	`reasoning_mode` is judged `NONE` and the marker stays in `message.content`.

	`conversion/youtu_moe.py` therefore rewrites the non-thinking branch to emit
	`<think></think>` (semantically identical — an empty think block), which turns the
	diff into an append and makes the analyzer report `reasoning_mode: TAG_BASED`
	(`-lv 5`). The parse itself still does not strip the token in current llama.cpp
	(`p.prefix(generation_prompt, reasoning_start)` anchors on a prompt literal that is
	not present in the generated text), so `strip_think.py` removes it on the output side.

	Tried and not recommended: banning token 128014 via `logit_bias`. The `</think>`
	disappears, but the model then believes it is still inside the think block and
	starts emitting quasi-reasoning prose — strictly worse.

	### 5.5 Small mtmd-CLI annoyances

	* Log timestamps have four dot-separated groups (`0.05.132.418 I …`); a
	three-group regex will not filter them.
	* `llama-mtmd-cli` prints the generated text with `LOG()`, so `--log-disable`
	swallows the answer too (symptom: empty output).

	## 6. Known limitations

	* No reasoning extraction into `reasoning_content` (see 5.4) — use `strip_think.py`.
	* Only Q4_K_M and BF16 are published; other quant types are untested.
	* KDA recurrent path has only been validated on Metal (macOS). CUDA/CPU are expected
	to work but have not been run.
	* Small-text OCR is limited by the default 300 image tokens; pass
	`--image-min-tokens 1024` for dense documents.

	## 7. Verified output

	Text (Q4_K_M, M3 Max 128 GiB, prefill ≈ 38 t/s, decode ≈ 14 t/s):

	```
	> 用一句话解释什么是光合作用。
	光合作用是植物、藻类和某些细菌利用光能将二氧化碳和水转化为有机物（如葡萄糖）并释放氧气的过程。
	```

	Vision — the 1969-07-21 New York Times front page (mtmd's bundled `test-1.jpeg`):
	masthead, date `MONDAY, JULY 21, 1969`, `LATE CITY EDITION`, `10 CENTS`,
	`VOL. CXVIII. NO. 46,731`, headline `MEN WALK ON MOON`, subhead
	`ASTRONAUTS LAND ON PLAIN; COLLECT ROCKS, PLANT FLAG` — all correct.

	## 8. License

	The patch follows llama.cpp's MIT license. The weights and the model architecture
	follow the upstream Youtu-LLM license (`license: other`); see
	[`tencent/Youtu-LLM-A7B-Base/LICENSE.txt`](https://huggingface.co/tencent/Youtu-LLM-A7B-Base/blob/main/LICENSE.txt).