Appendix A: Expert-Choice vs. Token-Choice Routing
The MoEP-MSIT architecture requires a routing mechanism to assign tokens to expert branches. Two canonical strategies exist: Token Choice, where each token selects its preferred expert, and Expert Choice \cite{zhou2022mixture}, where each expert selects its preferred tokens. We compare both strategies empirically to justify our selection of Expert-Choice routing.
A.1 Routing Strategies
Token-Choice Top-1. Each token computes a softmax distribution over experts and is dispatched to the expert with the highest affinity. Because tokens choose independently, some experts may receive disproportionately many tokens while others receive few or none—a failure mode known as expert collapse. To mitigate this, token-choice routing requires an auxiliary load-balancing loss. Following MoEP \cite{tapaninaho2025moep}, we use a negative-entropy regulariser $\mathcal{L}_\text{bal} = \sum_i p_i \log p_i$, where $p_i$ is the mean routing probability for expert $i$ across the batch. This term is added to the cross-entropy loss with coefficient $\alpha = 0.01$.
Expert-Choice. Each expert independently selects its top-$k$ tokens ranked by routing affinity, where $k = \lfloor cN/E \rceil$ is determined by the capacity factor $c$. Because every expert receives exactly $k$ tokens by construction, load balancing is guaranteed without any auxiliary loss. The training objective is the standard cross-entropy alone.
A.2 Results
Table A1 reports per-benchmark scores for both routing variants trained under identical conditions (same hyperparameters, tokenizer, data, and number of epochs).
Table A1. Benchmark comparison of routing strategies. All other architectural choices (global block, expert window sizes, SwiGLU, capacity factor $c{=}2.0$) are held constant.
| Benchmark | Expert-Choice (Ours) | Token-Choice Top-1 | $\Delta$ |
|---|---|---|---|
| BLiMP | 64.66 | 64.51 | +0.15 |
| BLiMP Supplement | 61.00 | 64.14 | −3.14 |
| EWoK | 51.34 | 48.57 | +2.77 |
| Entity Tracking | 21.08 | 21.07 | +0.01 |
| COMPS | 49.62 | 49.71 | −0.09 |
| GlobalPIQA (SuperGLUE) | 32.65 | 33.12 | −0.47 |
| Overall Average | 38.42 | — | — |
| NLP Average | 49.04 | — | — |
| Human-like Average | 1.24 | — | — |
A.3 Analysis
The per-benchmark comparison reveals a nuanced picture rather than uniform dominance by either strategy.
Expert-Choice advantages. The most substantial single-benchmark gain is on EWoK (world knowledge), where Expert-Choice outperforms Token-Choice by +2.77 points (51.34 vs. 48.57). We attribute this to the guaranteed uniform utilisation of all four multi-scale expert branches: because every expert processes exactly $k$ tokens regardless of the input distribution, the model develops balanced representations across all contextual scales. World-knowledge tasks, which require integrating both local lexical cues and broader sentential context, benefit most from this multi-scale coverage. Expert-Choice also achieves a modest gain on BLiMP (+0.15).
Token-Choice advantages. Token-Choice top-1 routing achieves higher scores on BLiMP Supplement (+3.14) and marginally on COMPS (+0.09) and GlobalPIQA (+0.47). These gains likely reflect the sharper specialisation that token-choice permits: when tokens freely concentrate on a single expert, that expert can develop stronger representations for the specific phenomena it handles—at the cost of underutilising the remaining experts.
Training simplicity. Beyond benchmark scores, Expert-Choice routing provides an important practical advantage: it eliminates the auxiliary load-balancing loss entirely. Token-choice routing requires careful tuning of the balancing coefficient $\alpha$; too small a value permits expert collapse, while too large a value distorts the primary training signal. Expert-Choice sidesteps this hyperparameter search, yielding a simpler and more robust training procedure—a meaningful benefit under the strict compute budgets of the BabyLM setting.
Selection rationale. We select Expert-Choice routing for the final architecture based on (1) its stronger EWoK performance, indicating better multi-scale knowledge integration; (2) the structural guarantee of balanced expert utilisation, which removes the risk of expert collapse during the volatile early stages of small-corpus pretraining; and (3) the elimination of the auxiliary loss, which simplifies training and removes a sensitive hyperparameter.