11R_Dense_124m_FFTHybrid (nanoMoE v11)
A ~115M-parameter (124M-class) dense hybrid causal language model: a striped stack of spectral FFT-convolution layers and multi-query attention layers. The "nanoMoE" name is historical lineage only — this model contains no MoE.
Status: pretraining complete (run v11-hybrid-r1, 8,750 iters / 4.59B tokens). Gate-probe results below are measured. The full lm-eval harness suite (HellaSwag/ARC/PIQA/etc.) is pending and marked TBD.
Architecture
| Component | Value |
|---|---|
| Layers | 12 (8 spectral + 4 attention) |
| Attention placement | layers 3, 6, 9, 12 (attn_every=3) |
| Embedding dim | 768 |
| Context length | 2048 (trained); extrapolates beyond via spectral kernels + RoPE |
| Attention | RoPE, SDPA flash path, MQA (1 KV head) |
| Spectral mixer | gated causal global convolution via FFT (O(T·logT)); implicit Fourier kernel with per-channel exponential decay; short causal depthwise conv (k=3) |
| Positional info | RoPE on attention layers; spectral layers are position-free (kernel-based). No learned position embeddings. |
| Normalization | LayerNorm (nanoGPT-shaped backbone; no RMSNorm/SwiGLU) |
| Vocab | 50264 (r50k_base + 7 ChatML/tool tokens), padded to 50304 |
| Parameters | 115.03M total / 76.40M non-embedding |
The attention layers are deliberately placed in the first half of the stack so two attention layers (3 and 6) can form a previous-token → induction-head circuit; induction emergence is a tracked metric for this model.
Tokenizer
v9_chatml_enc: GPT-2's r50k_base plus 7 special tokens at IDs 50257–50263:
<|im_start|>, <|im_end|>, <|tool_call|>, <|tool_call_end|>,
<|tool_result|>,