Wai Chung's picture

Wai Chung PRO

Ternii

AI & ML interests

None yet

Recent Activity

reacted to theirpost with 🔥 1 day ago
Qwen3.8-Flash-Next, a 177B-parameter MoE, now runs on a 16 GB MacBook Air at about 12 tokens/s. One line in Terminal downloads everything and starts a local chat in your browser: curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash How it fits in 16 GB: - The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3. - The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones. - Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output. - It uses 8 of the model's 10 routed experts per token. Measured on a MacBook Air M5 with 16 GB: - Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6. - Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant. - On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build. Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it. https://huggingface.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
reacted to theirpost with ❤️ 1 day ago
Qwen3.8-Flash-Next, a 177B-parameter MoE, now runs on a 16 GB MacBook Air at about 12 tokens/s. One line in Terminal downloads everything and starts a local chat in your browser: curl -fsSL https://raw.githubusercontent.com/ternii-ai/tritq-vllm/main/mac/qwen38-16gb.sh | bash How it fits in 16 GB: - The model file, Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF, is 89.8 GB. The routed experts are stored at 2 trits per weight and the other layers at 3. - The experts stay on the SSD. A 6,750 MiB cache in memory holds the busy ones. - Our llama.cpp build (llama.cpp-tr r3) reads the next layer's experts from the SSD while the GPU is still working. A small learned predictor picks which experts to fetch; it never changes the output. - It uses 8 of the model's 10 routed experts per token. Measured on a MacBook Air M5 with 16 GB: - Decode is about 12 tokens/s (11.9-12.6 over six runs, including an independent re-test). Our previous file and build ran at 3.6. - Accuracy with 8 experts vs the model's own 10: GSM8K 96.5 vs 95.0, HumanEval 95.1 vs 95.7, MBPP 91.1 vs 91.1, IFEval 66.0 vs 65.8. None of the differences is significant. - On the same Mac, our Qwen3.6-35B-A3B TR2.25-G runs at about 35 tokens/s with the same build. Caveats: this is experimental and only one Mac has been measured. Prompt processing is slow (about 75 tokens/s). It uses about 13 GB, so close other apps first. It needs our patched llama.cpp; LM Studio and Ollama can't load it. https://huggingface.co/Ternii/Qwen3.8-Flash-Next-TR2.31-G-GGUF
View all activity

Organizations

None yet