Geonwoohong/GMoE

---

😎 GMoE: Global Mixture of Experts with Logit Propagation 😎

Geonwoo Hong, Taehwan Kim

This repository provides the official GMoE checkpoints, accepted to ACL 2026.

---

Checkpoints

Checkpoint	Size	Experts	Model Parameters	Path
GMoE_Small	Small	16	80M	GMoE_Small/
GMoE_Small_32	Small	32	114M	GMoE_Small_32/
GMoE_Base	Base	16	204M	GMoE_Base/
GMoE_Base_32	Base	32	280M	GMoE_Base_32/
GMoE_Base_64	Base	64	432M	GMoE_Base_64/
GMoE_Large	Large	16	496M	GMoE_Large/
GMoE_Large_32	Large	32	632M	GMoE_Large_32/
GMoE_Large_64	Large	64	903M	GMoE_Large_64/

How to Use

Detailed training and evaluation instructions are available in the GitHub README.

Citation

@inproceedings{hong-kim-2026-gmoe,
    title = "{GM}o{E}: Global Mixture of Experts with Logit Propagation",
    author = "Hong, Geonwoo  and
      Kim, Taehwan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.acl-long.2065/",
    pages = "44599--44614",
    ISBN = "979-8-89176-390-6",
    abstract = "Sparse Mixture of Experts (SMoE) architectures reduce computational cost by activating only a subset of experts per token, yet they often retain large memory footprints and exhibit significant redundancy, both within and across layers. We propose GMoE, a sparse MoE architecture designed to explicitly address these inefficiencies. Instead of maintaining separate expert sets for each layer, GMoE uses Global Experts shared across all layers and adds a Local Expert per layer for layer-specific adaptation. This architecture reuses Global Experts across layers, thereby mitigating inter-layer redundancy while substantially reducing model parameters. In addition, we introduce a Global Router with a GRU-based recurrent component shared across layers and layer-specific routing heads that propagate routing logits across layers. This routing mechanism couples routing decisions across layers, progressively refines routing paths, and helps mitigate intra-layer redundancy. Across diverse language modeling corpora and downstream benchmarks, GMoE remains competitive while using substantially fewer parameters. Routing path analyses and an ablation study show that GMoE reduces cross-layer routing concentration and increases path diversity, with the Global Experts, the Local Expert, and the Global Router all contributing to the gains. The code is available at https://github.com/GEONWOOHONG/GMoE."
}