metadata
license: apache-2.0
language:
- en
- zh
pipeline_tag: text-generation
tags:
- moe
- hy3
- hy-v3
- huggingface
- llama.cpp
- apple-silicon
- metal
- local-ai
- 295b
base_model: tencent/Hy3
datasets:
- AngelSlim/Hy3-GGUF
Hy3 (hy_v3) — macOS Metal Build
Run Tencent Hy3 295B MoE on a MacBook via llama.cpp + Metal.
Not in the cloud. Not on a cluster. On a laptop.
🔥 The Numbers
| What | Value |
|---|---|
| Architecture | Hy3 (hy_v3) — Hunyuan V3 |
| Parameters | 295B |
| Layers | 81 (80 routed + 1 MTP) |
| Experts | 192 (8 active per token) |
| Quantization | IQ1_M (mixed recipe) |
| File size | ~85 GB |
| RAM needed | 96-128 GB (Apple Silicon) |
| License | Apache 2.0 |
🏆 Why This Matters
A 295B MoE model running on a single MacBook is a milestone for open local AI:
- ✅ No cloud, no API keys, no subscriptions
- ✅ Total privacy — data never leaves your machine
- ✅ No dedicated GPU — Apple Silicon unified memory is enough
- ✅ Portable — runs on a laptop, not a server rack
📥 Download the GGUF
The quantized model comes from AngelSlim/Hy3-GGUF:
# IQ1_M with MTP (85 GB) — recommended for 128 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M-mtp.gguf
# IQ1_M without MTP (84 GB) — for 96 GB
wget https://huggingface.co/AngelSlim/Hy3-GGUF/resolve/main/Hy3-IQ1_M.gguf
🛠️ Build & Run
1. Clone the patched llama.cpp
git clone https://github.com/RobZombAI/llama.cpp-metal_hyv3
cd llama.cpp-metal_hyv3
2. Build with Metal
mkdir build && cd build
cmake .. -DLLAMA_METAL=ON
make -j$(sysctl -n hw.logicalcpu)
3. Run inference
./build/bin/llama-cli \
-m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
-c 65536 \
-ngl 99 \
-fa on \
-ctk q8_0 -ctv q8_0 \
-p "Hello" \
-n 100 \
--temp 0.6
With reasoning enabled
./build/bin/llama-cli \
-m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
-c 65536 -ngl 99 -fa on \
-ctk q8_0 -ctv q8_0 \
-p "Explain quantum computing" \
-n 300 --temp 0.6 \
--reasoning on --reasoning-budget -1
With MTP self-speculative decoding (higher throughput)
./build/bin/llama-cli \
-m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
-c 65536 -ngl 99 -fa on \
--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-n-min 1 \
-ctk q8_0 -ctv q8_0 \
-ctkd q8_0 -ctvd q8_0 \
-p "Hello" -n 200 --temp 0.6
OpenAI-compatible server
./build/bin/llama-server \
-m ~/Downloads/Hy3-IQ1_M-mtp.gguf \
-c 65536 -ngl 99 -fa on \
-ctk q8_0 -ctv q8_0 \
--temp 0.6 --port 8080
For 96 GB Macs (no MTP, compressed KV)
./build/bin/llama-cli \
-m ~/Downloads/Hy3-IQ1_M.gguf \
-c 65536 -ngl 99 -fa on \
-ctk q8_0 -ctv q8_0 \
-p "Hello" -n 100 --temp 0.6
📊 Hardware Requirements
| Mac | RAM | MTP | Context | Notes |
|---|---|---|---|---|
| M5/M4/M3 Max | 128 GB | ✅ | 64K | Everything on |
| MacBook Pro | 128 GB | ✅ | 64K | Runs on a laptop |
| Mac Studio | 96 GB | ❌ | 64K | KV q8_0, no MTP |
| MacBook Pro | 96 GB | ❌ | 64K | Same |
🙏 Credits
- Tencent — original Hy3 model (Apache 2.0)
- AngelSlim — GGUF quantization, mixed recipes, importance matrix, base llama.cpp patches, benchmarks. The real MVP.
- RobZombAI — macOS Metal build, patches integration, testing
📚 Resources
- GitHub repo: github.com/RobZombAI/llama.cpp-metal_hyv3
- AngelSlim's GGUF: huggingface.co/AngelSlim/Hy3-GGUF
- Original model: huggingface.co/tencent/Hy3
- llama.cpp: github.com/ggml-org/llama.cpp
📜 License
Apache 2.0 — same as the original Tencent Hy3 model and AngelSlim's patches.
Long live open local AI. 🎉