·
AI & ML interests
None yet
Recent Activity
posted an update 1 day ago Qwen3.6-35B-A3B now runs on a 16 GB MacBook Air at about 35 tokens/s.
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen36-35b.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF, is 16.49 GB: 2.25 trits per weight on average, no retraining.
- On a 16 GB Mac the experts stay on the SSD and an 8,400 MiB cache in memory holds the busy ones. With 24 GB or more the script loads the whole model into memory.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 35 tokens/s (35.1–36.3 over 6 runs on two days, including an independent re-test). Our previous build ran at 23.3–25.2 on the same Mac.
- Up to 12.1 GB wired, so close other large apps first.
- Accuracy in the 16 GB mode: GSM8K 94.0, HumanEval 92.1, MBPP 87.6 vs 95.0 / 92.7 / 87.2 with the model fully in memory, no significant difference (measured on an M5 Max at a 16 GB budget).
Caveats: experimental, and only one 16 GB Mac has been measured. The speed-up costs about 35 % more energy per token. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
https://huggingface.co/Ternii/Qwen3.6-35B-A3B-TR2.25-G-GGUF reacted to theirpost with 🔥 1 day ago Qwen3.8-Flash-Next, a 177B-parameter MoE, now runs on a 16 GB MacBook Air at about 12 tokens/s.
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3.
- The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones.
- Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output.
- It uses 8 of the model's 10 routed experts per token.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6.
- Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant.
- On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build.
Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
https://huggingface.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF reacted to theirpost with ❤️ 1 day ago Qwen3.8-Flash-Next, a 177B-parameter MoE, now runs on a 16 GB MacBook Air at about 12 tokens/s.
One line in Terminal downloads everything and starts a local chat in your browser:
curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash
How it fits in 16 GB:
- The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3.
- The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones.
- Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output.
- It uses 8 of the model's 10 routed experts per token.
Measured on a MacBook Air M5 with 16 GB:
- Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6.
- Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant.
- On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build.
Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it.
https://huggingface.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF View all activity Organizations
None yet