Hugging Face's logo

RobZombAI
/
llama.cpp-metal_hyv3

Model card Files
xet
 Community
llama.cpp-metal_hyv3
File size: 4,099 Bytes
d3ff80a
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
---
license: apache-2.0
language:
- en
- zh
pipeline_tag: text-generation
tags:
- moe
- hy3
- hy-v3
- huggingface
- llama.cpp
- apple-silicon
- metal
- local-ai
- 295b
base_model: tencent/Hy3
datasets:
- AngelSlim/Hy3-GGUF
---

# Hy3 (hy_v3) — macOS Metal Build

**Run Tencent Hy3 295B MoE on a MacBook via llama.cpp + Metal.**

Not in the cloud. Not on a cluster. On a laptop.

## 🔥 The Numbers

| What | Value |
|------|-------|
| Architecture | Hy3 (hy_v3) — Hunyuan V3 |
| Parameters | **295B** |
| Layers | 81 (80 routed + 1 MTP) |
| Experts | 192 (8 active per token) |
| Quantization | IQ1_M (mixed recipe) |
| File size | ~85 GB |
| RAM needed | **96-128 GB** (Apple Silicon) |
| License | Apache 2.0 |

## 🏆 Why This Matters

A **295B MoE model running on a single MacBook** is a milestone for open local AI:

- ✅ No cloud, no API keys, no subscriptions
- ✅ Total privacy — data never leaves your machine
- ✅ No dedicated GPU — Apple Silicon unified memory is enough
- ✅ Portable — runs on a laptop, not a server rack

## 📥 Download the GGUF

The quantized model comes from [AngelSlim/Hy3-GGUF](https://huggingface.co/AngelSlim/Hy3-GGUF):

```bash
# IQ1_M with MTP (85 GB) — recommended for 128 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M-mtp.gguf

# IQ1_M without MTP (84 GB) — for 96 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M.gguf
```

## 🛠️ Build & Run

### 1. Clone the patched llama.cpp

```bash
git clone https://github.com/RobZombAI/llama.cpp-metal_hyv3
cd llama.cpp-metal_hyv3
```

### 2. Build with Metal

```bash
mkdir build && cd build
cmake .. -DLLAMA_METAL=ON
make -j$(sysctl -n hw.logicalcpu)
```

### 3. Run inference

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 \
    -ngl 99 \
    -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Hello" \
    -n 100 \
    --temp 0.6
```

### With reasoning enabled

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Explain quantum computing" \
    -n 300 --temp 0.6 \
    --reasoning on --reasoning-budget -1
```

### With MTP self-speculative decoding (higher throughput)

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    --spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 \
    -ctk q8_0 -ctv q8_0 \
    -ctkd q8_0 -ctvd q8_0 \
    -p "Hello" -n 200 --temp 0.6
```

### OpenAI-compatible server

```bash
./build/bin/llama-server \
    -m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    --temp 0.6 --port 8080
```

### For 96 GB Macs (no MTP, compressed KV)

```bash
./build/bin/llama-cli \
    -m ~/Downloads/Hy3-IQ1_M.gguf \
    -c 65536 -ngl 99 -fa on \
    -ctk q8_0 -ctv q8_0 \
    -p "Hello" -n 100 --temp 0.6
```

## 📊 Hardware Requirements

| Mac | RAM | MTP | Context | Notes |
|-----|-----|:---:|:--------:|-------|
| M5/M4/M3 Max | 128 GB | ✅ | 64K | Everything on |
| MacBook Pro | 128 GB | ✅ | 64K | Runs on a laptop |
| Mac Studio | 96 GB | ❌ | 64K | KV q8_0, no MTP |
| MacBook Pro | 96 GB | ❌ | 64K | Same |

## 🙏 Credits

- **Tencent** — original [Hy3 model](https://huggingface.co/tencent/Hy3) (Apache 2.0)
- **[AngelSlim](https://huggingface.co/AngelSlim)** — GGUF quantization, mixed recipes, importance matrix, base llama.cpp patches, benchmarks. **The real MVP.**
- **RobZombAI** — macOS Metal build, patches integration, testing

## 📚 Resources

- GitHub repo: [github.com/RobZombAI/llama.cpp-metal_hyv3](https://github.com/RobZombAI/llama.cpp-metal_hyv3)
- AngelSlim's GGUF: [huggingface.co/AngelSlim/Hy3-GGUF](https://huggingface.co/AngelSlim/Hy3-GGUF)
- Original model: [huggingface.co/tencent/Hy3](https://huggingface.co/tencent/Hy3)
- llama.cpp: [github.com/ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp)

## 📜 License

Apache 2.0 — same as the original Tencent Hy3 model and AngelSlim's patches.

---

**Long live open local AI. 🎉**