Post
85
Qwen3.6-35B-A3B now runs on a 16 GB MacBook Air at about 35 tokens/s.
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen36-35b.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF, is 16.49 GB: 2.25 trits per weight on average, no retraining.
- On a 16 GB Mac the experts stay on the SSD and an 8,400 MiB cache in memory holds the busy ones. With 24 GB or more the script loads the whole model into memory.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 35 tokens/s (35.1–36.3 over 6 runs on two days, including an independent re-test). Our previous build ran at 23.3–25.2 on the same Mac.
- Up to 12.1 GB wired, so close other large apps first.
- Accuracy in the 16 GB mode: GSM8K 94.0, HumanEval 92.1, MBPP 87.6 vs 95.0 / 92.7 / 87.2 with the model fully in memory, no significant difference (measured on an M5 Max at a 16 GB budget).
Caveats: experimental, and only one 16 GB Mac has been measured. The speed-up costs about 35 % more energy per token. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen36-35b.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF, is 16.49 GB: 2.25 trits per weight on average, no retraining.
- On a 16 GB Mac the experts stay on the SSD and an 8,400 MiB cache in memory holds the busy ones. With 24 GB or more the script loads the whole model into memory.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 35 tokens/s (35.1–36.3 over 6 runs on two days, including an independent re-test). Our previous build ran at 23.3–25.2 on the same Mac.
- Up to 12.1 GB wired, so close other large apps first.
- Accuracy in the 16 GB mode: GSM8K 94.0, HumanEval 92.1, MBPP 87.6 vs 95.0 / 92.7 / 87.2 with the model fully in memory, no significant difference (measured on an M5 Max at a 16 GB budget).
Caveats: experimental, and only one 16 GB Mac has been measured. The speed-up costs about 35 % more energy per token. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF