docs/experts.md

Changing the number of active experts

What was measured

The experiment ran on a real MoE model (OLMoE-1B-7B, 64 experts, 8 active) on this machine: 4 CPU threads, Q4_K_M, 128 tokens, temperature 0.

Experts active	Throughput	Relative to native
2	51.6 tok/s	×1.71 faster
8 (native)	30.2 tok/s	reference
15	21.1 tok/s	×1.43 slower
64	crash	memory pool exhausted

The override was confirmed to reach the compute graph: at 64 experts the process aborted inside build_moe_ffn with ggml_new_object: not enough space in the context's memory pool. The setting is applied, not ignored.

What this means

Raising the number of active experts makes the model slower, not smarter. More experts per token means more feed-forward work per layer, so throughput drops and memory grows. It does not add knowledge, and it does not add reasoning ability.

The value is baked in during training. When a model is trained with 6 experts per token, the router learns to distribute work across exactly that budget. Forcing 15 at inference time asks the network to combine expert outputs it was never trained to combine, and past a point the output degrades.

So flint.expert_used_count = 6 in the Cortex.0 GGUF is not a limiter to remove. It is a trained property.

Where the cost actually goes

For Cortex.0: 6 experts out of 256, each with a 512-wide feed-forward, plus 2 shared experts at 1024. Only about 2.4% of expert weights are touched per token. That is the design of the architecture, and it is why a 40B model can run on one GPU.

What actually makes the system more capable

Ordered by real effect, for this setup:

1. Fine-tuning for tool use. The agent's quality depends on the model emitting correct tool calls. If that is weak, no serving flag fixes it. This needs GPU training.
2. Higher-precision weights. Q8_0 instead of Q4_K_XL recovers quality that quantization removed. Real, measurable, no training needed.
3. Larger context at inference. --ctx-size controls how much of the 262k window is actually usable. Raising it costs memory but improves long tasks.
4. Speculative decoding. A small draft model proposes tokens the big model verifies. This raises speed several-fold with no quality loss. This is the real answer to "make it faster".
5. Serving engine choice. SGLang handles MoE batching better than llama.cpp on GPU. Flint ships SGLang deployment scripts upstream.

How to change it anyway

If you want to experiment, the override is supported:

# llama.cpp
llama-server --model Cortex.0-Q8_0.gguf --kv-override flint.expert_used_count=15

# llama-cpp-python
from llama_cpp import Llama

model = Llama(
    model_path="Cortex.0-Q8_0.gguf",
    kv_overrides={"flint.expert_used_count": 15},
)

Measure before and after with the benchmark in this repository:

python3 scripts/experiments/bench_experts.py Cortex.0-Q8_0.gguf \
    --experts 2,6,15 --tokens 128

If the numbers come out worse, the native value was right.