| # Changing the number of active experts |
|
|
| ## What was measured |
|
|
| The experiment ran on a real MoE model (OLMoE-1B-7B, 64 experts, 8 active) on |
| this machine: 4 CPU threads, Q4_K_M, 128 tokens, temperature 0. |
|
|
| | Experts active | Throughput | Relative to native | |
| |---|---|---| |
| | 2 | 51.6 tok/s | ×1.71 faster | |
| | **8 (native)** | **30.2 tok/s** | reference | |
| | 15 | 21.1 tok/s | ×1.43 slower | |
| | 64 | crash | memory pool exhausted | |
|
|
| The override was confirmed to reach the compute graph: at 64 experts the |
| process aborted inside `build_moe_ffn` with `ggml_new_object: not enough space |
| in the context's memory pool`. The setting is applied, not ignored. |
|
|
| ## What this means |
|
|
| Raising the number of active experts makes the model **slower**, not smarter. |
| More experts per token means more feed-forward work per layer, so throughput |
| drops and memory grows. It does not add knowledge, and it does not add |
| reasoning ability. |
|
|
| The value is baked in during training. When a model is trained with 6 experts |
| per token, the router learns to distribute work across exactly that budget. |
| Forcing 15 at inference time asks the network to combine expert outputs it was |
| never trained to combine, and past a point the output degrades. |
|
|
| So `flint.expert_used_count = 6` in the Cortex.0 GGUF is not a limiter to |
| remove. It is a trained property. |
|
|
| ## Where the cost actually goes |
|
|
| For Cortex.0: 6 experts out of 256, each with a 512-wide feed-forward, plus 2 |
| shared experts at 1024. Only about 2.4% of expert weights are touched per |
| token. That is the design of the architecture, and it is why a 40B model can |
| run on one GPU. |
|
|
| ## What actually makes the system more capable |
|
|
| Ordered by real effect, for this setup: |
|
|
| 1. **Fine-tuning for tool use.** The agent's quality depends on the model |
| emitting correct tool calls. If that is weak, no serving flag fixes it. |
| This needs GPU training. |
| 2. **Higher-precision weights.** Q8_0 instead of Q4_K_XL recovers quality that |
| quantization removed. Real, measurable, no training needed. |
| 3. **Larger context at inference.** `--ctx-size` controls how much of the 262k |
| window is actually usable. Raising it costs memory but improves long tasks. |
| 4. **Speculative decoding.** A small draft model proposes tokens the big model |
| verifies. This raises speed several-fold with no quality loss. This is the |
| real answer to "make it faster". |
| 5. **Serving engine choice.** SGLang handles MoE batching better than |
| llama.cpp on GPU. Flint ships SGLang deployment scripts upstream. |
| |
| ## How to change it anyway |
| |
| If you want to experiment, the override is supported: |
| |
| ```bash |
| # llama.cpp |
| llama-server --model Cortex.0-Q8_0.gguf --kv-override flint.expert_used_count=15 |
| ``` |
| |
| ```python |
| # llama-cpp-python |
| from llama_cpp import Llama |
| |
| model = Llama( |
| model_path="Cortex.0-Q8_0.gguf", |
| kv_overrides={"flint.expert_used_count": 15}, |
| ) |
| ``` |
| |
| Measure before and after with the benchmark in this repository: |
| |
| ```bash |
| python3 scripts/experiments/bench_experts.py Cortex.0-Q8_0.gguf \ |
| --experts 2,6,15 --tokens 128 |
| ``` |
| |
| If the numbers come out worse, the native value was right. |
|
|
