Post
86
Qwen3.8-Flash-Next, a 177B-parameter MoE, now runs on a 16 GB MacBook Air at about 12 tokens/s.
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3.
- The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones.
- Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output.
- It uses 8 of the model's 10 routed experts per token.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6.
- Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant.
- On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build.
Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3.
- The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones.
- Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output.
- It uses 8 of the model's 10 routed experts per token.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6.
- Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant.
- On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build.
Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF