hy3-llama-patch/HF_README.md

---
license: apache-2.0
language:
- en
- zh
pipeline_tag: text-generation
tags:
- moe
- hy3
- hy-v3
- huggingface
- llama.cpp
- apple-silicon
- metal
- local-ai
- 295b
base_model: tencent/Hy3
datasets:
- AngelSlim/Hy3-GGUF
---

# Hy3 (hy_v3) — macOS Metal Build

**Run Tencent Hy3 295B MoE on a MacBook via llama.cpp + Metal.**

Not in the cloud. Not on a cluster. On a laptop.

## 🔥 The Numbers

| What | Value |
|------|-------|
| Architecture | Hy3 (hy_v3) — Hunyuan V3 |
| Parameters | **295B** |
| Layers | 81 (80 routed + 1 MTP) |
| Experts | 192 (8 active per token) |
| Quantization | IQ1_M (mixed recipe) |
| File size | ~85 GB |
| RAM needed | **96-128 GB** (Apple Silicon) |
| License | Apache 2.0 |

## 🏆 Why This Matters

A **295B MoE model running on a single MacBook** is a milestone for open local AI:

- ✅ No cloud, no API keys, no subscriptions
- ✅ Total privacy — data never leaves your machine
- ✅ No dedicated GPU — Apple Silicon unified memory is enough
- ✅ Portable — runs on a laptop, not a server rack

## 📥 Download the GGUF

The quantized model comes from [AngelSlim/Hy3-GGUF](https://huggingface.co/AngelSlim/Hy3-GGUF):

```bash
# IQ1_M with MTP (85 GB) — recommended for 128 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M-mtp.gguf

# IQ1_M without MTP (84 GB) — for 96 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M.gguf
```

## 🛠️ Build & Run

### 1. Clone the patched llama.cpp

```bash
git clone https://github.com/RobZombAI/llama.cpp-metal_hyv3
cd llama.cpp-metal_hyv3
```

### 2. Build with Metal

```bash
mkdir build && cd build
cmake .. -DLLAMA_METAL=ON
make -j$(sysctl -n hw.logicalcpu)
```

### 3. Run inference

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 \
    -ngl 99 \
    -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Hello" \
    -n 100 \
    --temp 0.6
```

### With reasoning enabled

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Explain quantum computing" \
    -n 300 --temp 0.6 \
    --reasoning on --reasoning-budget -1
```

### With MTP self-speculative decoding (higher throughput)

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 \
    -ctk q8_0 -ctv q8_0 \
    -ctkd q8_0 -ctvd q8_0 \
    -p "Hello" -n 200 --temp 0.6
```

### OpenAI-compatible server

```bash
./build/bin/llama-server \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.6 --port 8080
```

### For 96 GB Macs (no MTP, compressed KV)

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Hello" -n 100 --temp 0.6
```

## 📊 Hardware Requirements

| Mac | RAM | MTP | Context | Notes |
|-----|-----|:---:|:--------:|-------|
| M5/M4/M3 Max | 128 GB | ✅ | 64K | Everything on |
| MacBook Pro | 128 GB | ✅ | 64K | Runs on a laptop |
| Mac Studio | 96 GB | ❌ | 64K | KV q8_0, no MTP |
| MacBook Pro | 96 GB | ❌ | 64K | Same |

## 🙏 Credits

- **Tencent** — original [Hy3 model](https://huggingface.co/tencent/Hy3) (Apache 2.0)
- **[AngelSlim](https://huggingface.co/AngelSlim)** — GGUF quantization, mixed recipes, importance matrix, base llama.cpp patches, benchmarks. **The real MVP.**
- **RobZombAI** — macOS Metal build, patches integration, testing

## 📚 Resources

- GitHub repo: [github.com/RobZombAI/llama.cpp-metal_hyv3](https://github.com/RobZombAI/llama.cpp-metal_hyv3)
- AngelSlim's GGUF: [huggingface.co/AngelSlim/Hy3-GGUF](https://huggingface.co/AngelSlim/Hy3-GGUF)
- Original model: [huggingface.co/tencent/Hy3](https://huggingface.co/tencent/Hy3)
- llama.cpp: [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)

## 📜 License

Apache 2.0 — same as the original Tencent Hy3 model and AngelSlim's patches.

---

**Long live open local AI. 🎉**