If You Fix It Like This, It Actually Looks Plausible — A Proposal on LatentMoE Projection and Depth Scaling for the 14T Numbat Architecture.

#39

by Gavin-chen - opened 1 day ago

Fair is fair: the creator did their homework. They used a genuine, geometrically clean $131,072 \times 12,288$ embedding table as bait, shipped reference code that strictly passes strict=True, and aligned every 16-element NVFP4 block scale and absmax code perfectly.

The giveaway, however, is that while the author lifted the trendy "3 GDN + 1 MLA" hybrid attention schedule from recent frontier reports, they ignored basic engineering reality on the MoE side, creating severe dimensional imbalance.

If you're the author, or if you want to turn this phantom spec into something that could actually run on modern hardware, here is a much more sensible blueprint:

I. The Three Fatal Flaws

Under the claimed trunk dimension ($d = 12,288$), the original hyperparameters fall apart:

Exploding Active FFN Width: 1,224 experts ($d_{\text{ff}} = 2,816$) activating Top-48 yields an active FFN width of 137,984 ($11.23d$) per layer. That is 5.6× wider than the model's own dense layers ($24,576 / 2d$).
Attention Diluted to Single Digits: Activating $5.09\text{B}$ FFN parameters per layer dilutes the full-width sequence mixers (GDN/MLA) down to just 6%–9% of active compute, whereas modern MoEs typically allocate 30%–50% to attention.
All-to-All Gridlock: Dispatching 48 full-width ($12,288$) tokens across nodes creates an unsustainable distributed communication bottleneck.

II. Core Fix: LatentMoE & Depth Compensation

No need to start from scratch—just introduce latent projection and rebalance depth versus sparsity:

Architecture Metric	Original numbat	Revised Latent-Numbat	Practical Benefit
Expert Input Dim	d = 12,288 (Full width)	l = 4,096 (1/3d Latent)	Expert footprint reduced by 1/3 to 34.6M
Expert Pool Size	1,224 experts	2,448 experts (2×)	Doubles capacity while saving 33% total MoE layer parameters
Active Routing / Token	Top-48 (1/25.5 sparsity)	Top-32 (1/76.5 sparsity)	Pushes sparsity to 1.3%, matching ultra-sparse frontier design
Active FFN / Layer	5.09B / layer	1.34B / layer	Cuts active FFN compute by 3.8×, realigning with dense layers
All-to-All Comm Payload	48 × 12,288 elements	32 × 4,096 elements	Slashes cross-node communication overhead by 77.8%
Total Backbone Depth	114 layers	138 layers	Converts saved memory into complete 3 GDN + 1 MLA depth blocks
Total / Active Params	~13.7T / ~615B active	~11.2T / ~258B active	Retains 10T+ scale while halving active load; attention share back to ~30%

III. Essential Stability Fixes

To ensure stable training at 138 layers and 10T+ scale, several missing numerical safeguards must be added:

Latent-Space RMSNorm + SiTU-GLU: Insert an RMSNorm before the up-projection $\mathbf{W}^{\uparrow}$ and swap unbounded SwiGLU for a smoothly saturating activation to prevent activation blowup across un-residualized projection chains.
Quantile Balancing: At 2,448 experts with extreme 1/76.5 sparsity, replace fixed-step bias updates with Quantile Balancing to prevent routing oscillations and expert starvation.
Block Attention Residuals: In a 138-layer network, use block-level residual attention (AttnRes) to prevent deep pre-norm layers from washing out early semantic representations.

IV. Takeaway

This revision preserves the massive capacity of an 11T+ class flagship while bringing per-token active compute down from an absurd 615B to a viable 258B. On a GB300 NVL72 rack running MTP speculative decoding, this setup has a genuine shot at delivering a fluid 60–70 tokens/s per user.

Next time you stage an elaborate leak, make sure the architectural ingredients are measured in the right proportions.