metadata
license: cc-by-4.0
language:
- en
- zh
tags:
- research-paper
- mixture-of-experts
- mixture-of-depths
- memory-layers
- architecture
- pretraining
Memory as an Additive Side-Path: Lookup Niches, Combine Mechanisms, and Learned Depth Allocation in a Bilingual MoE
Wei Ciao Wu (independent researcher), with Claude (Anthropic) as AI research assistant.
📄 Read the paper (PDF) · LaTeX source included · 15 pages
Preprint, 2026-07. Part of the circus-0.2 research line (1B-parameter from-scratch bilingual MoE).
TL;DR
At 1B-active-parameter scale, we ask whether a token-keyed static-memory feed-forward module (STEM) — indexed by token identity instead of a learned router — has any regime where it beats an ordinary sparsely-gated MoE layer, by what mechanism, and whether the answer can be turned into a learned per-token depth-allocation architecture. Three pre-registered campaigns share one bilingual (EN/繁中) + code + math wind tunnel (~6.3k steps, single seed):
- Niche mapping (5 hypotheses): 3 of 5 candidate STEM advantages refuted — two in reverse. STEM's genuine quality edge is an early-training low-token-budget regime that dilutes to parity at the endpoint and reverses on code; a systems edge (decode throughput, OOM robustness) holds unconditionally.
- Mechanism factorial (2×2): the addressing axis (token- vs product-key) is second-order; the combine axis is decisive — replacing STEM's multiplicative gate with an additive residual side-path repairs the code reversal (an 11.9-point BPB swing) and yields the only cells that beat MoE on multiple held-out domains, at zero extra FLOPs.
- STEM-Router: turning the additive-combine result into a Mixture-of-Depths-style layer (always-on additive STEM + capacity-gated MoE, expert-choice top-k) beats the all-MoE baseline 3-of-5 domains with zero losses at matched FLOPs, and still wins 3-of-5 at a net-FLOP-saving quarter-capacity point (−8.5% MACs, +4.1% wall-clock). The learned gate is depth-heterogeneous: the final three layers are perfectly frequency-aligned (per-layer Spearman ρ = −1.00, triple-replicated) — MoE capacity concentrates on the most frequent tokens while rare tokens ride the memorized side-path. A per-layer capacity schedule wins iso-FLOP; a post-hoc calibrated per-layer threshold matches the expert-choice oracle within +0.34% (decode needs no auxiliary predictor); the capacity dial degrades gracefully at inference time; and a static bigram-table key axis is dominated from both directions — token-only remains the strongest static key.
Citation
@misc{wu2026stemarchitecture,
title = {Memory as an Additive Side-Path: Lookup Niches, Combine
Mechanisms, and Learned Depth Allocation in a Bilingual MoE},
author = {Wei Ciao Wu},
year = {2026},
note = {Preprint. https://huggingface.co/wcamon/circus-0.2-stem-architecture}
}