How to Loop MoE: Flatten the Experts, Untie the Attention

Looped Transformers reuse the same block across multiple passes, while sparse Mixture-of-Experts models store many experts but activate only a few for each token. These two ideas are naturally complementary: every new pass gives a token another routing decision, allowing it to reach different experts and expert combinations without adding expert parameters. But this raises an open design question: under a fixed parameter and compute budget, how should experts be distributed across layers and passes; and which components should be shared across passes?

Our new preprint, “How to Loop MoE: Flatten the Experts, Untie the Attention,” answers this question with Foil.

𝗢𝘂𝗿 𝗶𝗱𝗲𝗮: Flatten the experts. Loop more. Untie the attention.

At each flattening step, Foil uses:
→ half as many expert layers
→ twice as many experts per layer
→ twice as many recurrent passes
→ separate attention parameters for each pass
while keeping total parameters, expert compute per token, and effective depth fixed.

We go from:
8 experts × 8 layers × 2 passes to:
64 experts × 1 layer × 16 passes

𝗥𝗲𝘀𝘂𝗹𝘁𝘀:
→ Every Foil configuration achieves lower pretraining loss than the baseline at 20B tokens.
→ 𝗔𝘁 𝟭𝟬𝟬𝗕 𝘁𝗼𝗸𝗲𝗻𝘀, 𝗹𝗼𝘀𝘀 𝗶𝗺𝗽𝗿𝗼𝘃𝗲𝘀 𝗺𝗼𝗻𝗼𝘁𝗼𝗻𝗶𝗰𝗮𝗹𝗹𝘆 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲 𝗱𝗲𝗴𝗿𝗲𝗲 𝗼𝗳 𝗳𝗹𝗮𝘁𝘁𝗲𝗻𝗶𝗻𝗴 𝗮𝘁 𝗺𝗮𝘁𝗰𝗵𝗲𝗱 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀 𝗮𝗻𝗱 𝗰𝗼𝗺𝗽𝘂𝘁𝗲. The fully flattened Foil ends 0.012 nat below the baseline, with downstream accuracy on par or better.
→ Untying attention becomes increasingly valuable as the model gets flatter. At the fully flattened shape, it lowers loss by 0.049 nat and improves mean accuracy on three representative downstream tasks by 3.3 points over tied attention at the same shape, with no additional compute.

→ More experts per layer make additional loops more useful; and more loops make additional experts more useful. The two amplify each other.
→ Load balance alone is not enough; it should be read together with routing confidence. In our experiments, routing confidence moves with looping gain, and its per-pass peak may signal diminishing returns from further looping or flattening.