docs/experts.md

# Changing the number of active experts

## What was measured

The experiment ran on a real MoE model (OLMoE-1B-7B, 64 experts, 8 active) on
this machine: 4 CPU threads, Q4_K_M, 128 tokens, temperature 0.

| Experts active | Throughput | Relative to native |
|---|---|---|
| 2 | 51.6 tok/s | ×1.71 faster |
| **8 (native)** | **30.2 tok/s** | reference |
| 15 | 21.1 tok/s | ×1.43 slower |
| 64 | crash | memory pool exhausted |

The override was confirmed to reach the compute graph: at 64 experts the
process aborted inside `build_moe_ffn` with `ggml_new_object: not enough space
in the context's memory pool`. The setting is applied, not ignored.

## What this means

Raising the number of active experts makes the model **slower**, not smarter.
More experts per token means more feed-forward work per layer, so throughput
drops and memory grows. It does not add knowledge, and it does not add
reasoning ability.

The value is baked in during training. When a model is trained with 6 experts
per token, the router learns to distribute work across exactly that budget.
Forcing 15 at inference time asks the network to combine expert outputs it was
never trained to combine, and past a point the output degrades.

So `flint.expert_used_count = 6` in the Cortex.0 GGUF is not a limiter to
remove. It is a trained property.

## Where the cost actually goes

For Cortex.0: 6 experts out of 256, each with a 512-wide feed-forward, plus 2
shared experts at 1024. Only about 2.4% of expert weights are touched per
token. That is the design of the architecture, and it is why a 40B model can
run on one GPU.

## What actually makes the system more capable

Ordered by real effect, for this setup:

1. **Fine-tuning for tool use.** The agent's quality depends on the model
   emitting correct tool calls. If that is weak, no serving flag fixes it.
   This needs GPU training.
2. **Higher-precision weights.** Q8_0 instead of Q4_K_XL recovers quality that
   quantization removed. Real, measurable, no training needed.
3. **Larger context at inference.** `--ctx-size` controls how much of the 262k
   window is actually usable. Raising it costs memory but improves long tasks.
4. **Speculative decoding.** A small draft model proposes tokens the big model
   verifies. This raises speed several-fold with no quality loss. This is the
   real answer to "make it faster".
5. **Serving engine choice.** SGLang handles MoE batching better than
   llama.cpp on GPU. Flint ships SGLang deployment scripts upstream.

## How to change it anyway

If you want to experiment, the override is supported:

```bash
# llama.cpp
llama-server --model Cortex.0-Q8_0.gguf --kv-override flint.expert_used_count=15
```

```python
# llama-cpp-python
from llama_cpp import Llama

model = Llama(
    model_path="Cortex.0-Q8_0.gguf",
    kv_overrides={"flint.expert_used_count": 15},
)
```

Measure before and after with the benchmark in this repository:

```bash
python3 scripts/experiments/bench_experts.py Cortex.0-Q8_0.gguf \
    --experts 2,6,15 --tokens 128
```

If the numbers come out worse, the native value was right.