---
license: cc-by-4.0
language:
  - en
  - zh
tags:
  - research-paper
  - mixture-of-experts
  - mixture-of-depths
  - memory-layers
  - architecture
  - pretraining
---

# Memory as an Additive Side-Path: Lookup Niches, Combine Mechanisms, and Learned Depth Allocation in a Bilingual MoE

**Wei Ciao Wu** (independent researcher), with Claude (Anthropic) as AI research assistant.

📄 **[Read the paper (PDF)](stem-architecture.pdf)** · LaTeX source included · 15 pages

Preprint, 2026-07. Part of the **circus-0.2** research line (1B-parameter from-scratch bilingual MoE).

## TL;DR

At 1B-active-parameter scale, we ask whether a **token-keyed static-memory feed-forward module (STEM)** — indexed by token identity instead of a learned router — has any regime where it beats an ordinary sparsely-gated MoE layer, by what mechanism, and whether the answer can be turned into a learned per-token depth-allocation architecture. Three pre-registered campaigns share one bilingual (EN/繁中) + code + math wind tunnel (~6.3k steps, single seed):

1. **Niche mapping (5 hypotheses):** 3 of 5 candidate STEM advantages refuted — two *in reverse*. STEM's genuine quality edge is an early-training low-token-budget regime that dilutes to parity at the endpoint and reverses on code; a systems edge (decode throughput, OOM robustness) holds unconditionally.
2. **Mechanism factorial (2×2):** the addressing axis (token- vs product-key) is second-order; the **combine axis is decisive** — replacing STEM's multiplicative gate with an **additive residual side-path** repairs the code reversal (an 11.9-point BPB swing) and yields the only cells that beat MoE on multiple held-out domains, at zero extra FLOPs.
3. **STEM-Router:** turning the additive-combine result into a Mixture-of-Depths-style layer (always-on additive STEM + capacity-gated MoE, expert-choice top-k) **beats the all-MoE baseline 3-of-5 domains with zero losses at matched FLOPs**, and still wins 3-of-5 at a net-FLOP-*saving* quarter-capacity point (−8.5% MACs, +4.1% wall-clock). The learned gate is depth-heterogeneous: the final three layers are perfectly frequency-aligned (per-layer Spearman ρ = −1.00, triple-replicated) — MoE capacity concentrates on the *most frequent* tokens while rare tokens ride the memorized side-path. A per-layer capacity schedule wins iso-FLOP; a post-hoc calibrated per-layer threshold matches the expert-choice oracle within +0.34% (decode needs no auxiliary predictor); the capacity dial degrades gracefully at inference time; and a static bigram-table key axis is dominated from both directions — token-only remains the strongest static key.

## Citation

```bibtex
@misc{wu2026stemarchitecture,
  title  = {Memory as an Additive Side-Path: Lookup Niches, Combine
            Mechanisms, and Learned Depth Allocation in a Bilingual MoE},
  author = {Wei Ciao Wu},
  year   = {2026},
  note   = {Preprint. https://huggingface.co/wcamon/circus-0.2-stem-architecture}
}
```

## Related

- Companion paper: [Binding is Verbatim Copy: diagnosing tool-name binding in a 1B agentic MoE](https://huggingface.co/wcamon/circus-0.2-binding-diagnosis)