| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - causal-lm |
| - research |
| - bf16 |
| - attention |
| - normalization |
| - neollm |
| datasets: |
| - HuggingFaceFW/fineweb-edu |
| --- |
| |
| # NeoLLM |
|
|
| NeoLLM is a **85.09 M parameter** decoder-only language model trained from scratch on |
| [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) with |
| **BF16** compute, completing training in approximately |
| **~0h 47m** on **NVIDIA RTX PRO 6000 Blackwell Server Edition**. |
| It integrates a collection of recently published attention and normalization techniques |
| into a single architecture, with the goal of studying how they interact during |
| pretraining. The model is actively being developed and the current checkpoint represents |
| an intermediate training state. |
|
|
| > **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X |
| > **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM) |
|
|
| --- |
|
|
| ## Architecture |
|
|
| NeoLLM is a decoder-only transformer with the following configuration: |
|
|
| | Parameter | Value | |
| |---|---| |
| | Hidden size | 512 | |
| | Layers | 12 | |
| | Attention heads | 8 | |
| | KV heads (GQA) | 4 | |
| | Head dim | 64 | |
| | Intermediate size | 1536 | |
| | Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) | |
| | Context length | 512 tokens | |
| | Leviathan | enabled=True; d_seed=128, modes=8, knots=16 | |
| | JTok / JTok-M | enabled=False / False; Torch continuous surfaces, experts=5, top_k=2 | |
| | DAGFormer | enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'} | |
| | Virtual Head Attention | enabled=False; query/KV heads follow the configured geometry | |
| | Copy-Paste LM | enabled=True; causal vocabulary/pointer mixture | |
| |
| ### Parameter breakdown |
| |
| | Parameter bucket | Count | |
| |---|---| |
| | **Total parameters** | 85.09M (85,093,304) | |
| | **Embedding parameters** (tied) | 32.97M (32,973,824) | |
| | **Non-embedding parameters** | 52.12M (52,119,480) | |
| | **Effective trainable parameters** | 85.09M (85,093,304) | |
| |
| > Weight tying is **enabled**: the input embedding matrix and the language-model head |
| > share the same parameters, so the effective trainable budget is |
| > `total − embed = 52.12M`. |
| |
| ### Integrated techniques |
| |
| NeoLLM combines architecture modules, optional auxiliary objectives, and |
| training-time optimizer/stability components from the following papers. |
| |
| **Embedding and token representation** |
| |
| - **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds |
| per-row and per-column learnable scalar parameters to selected matrix layers and, when |
| enabled, embeddings. |
| - **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional |
| continuous token embedding generator that can replace the discrete input lookup table. |
| - **Leviathan-JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) — |
| Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok |
| applies one normalized surface as a multiplicative MLP gate; JTok-M routes a |
| Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled |
| residual. Runtime ``jtok_kernel_backend`` selects the PyTorch reference or the |
| CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion. |
| - **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis |
| reference used by the Leviathan continuous token generator implementation. |
| - **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) — |
| Augments token embeddings with character-level spelling information. |
| - **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) — |
| Reference motivation for treating token embeddings as structured objects rather than |
| unconstrained lookup rows. |
| |
| **Attention, positions, and output projection** |
| |
| - **Virtual Head Attention** ([arXiv:2610.17959](https://arxiv.org/abs/2610.17959)) — Optional |
| group-specific query premix and low-rank head postmix. NeoLLM shares its gated query |
| projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are |
| not automatically folded into standard GQA projections. |
| - **Copy-Paste LM** ([arXiv:2610.19247](https://arxiv.org/abs/2610.19247)) — Optional |
| causal pointer and sink mixed with the vocabulary distribution. CUDA training |
| uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits. |
| - **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks. |
| A portion of the projection channels are dedicated to periodic cosine/sine features. |
| - **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head |
| Attention. Adds small learnable interaction matrices between attention heads for K and V. |
| - **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned |
| lower-triangular preconditioner to V before attention, decorrelating value representations |
| across positions. |
| - **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds |
| two learnable per-head scalars (α and β) to the softmax weights: |
| `[α·softmax(QKᵀ) + β]·V`. |
| - **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention. |
| After computing attention, removes the component of the output aligned with the token's |
| own value vector. |
| - **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head |
| learns K=4 directions in the output space; a learned router suppresses the attention output |
| along each direction per input. |
| - **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate |
| is applied to the attention output before the output projection, introducing non-linearity |
| and preventing attention sinks. |
| - **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q |
| and K by subtracting a fraction of the previous position's Q and K values (causal |
| first-difference). |
| - **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) — |
| Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns |
| per original head. |
| - **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning |
| module that learns contextual position coordinates above a configurable start layer. |
| - **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational |
| position encoding used by the REPO-GRAPE positional path. |
| - **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional |
| factorized attention log-prior channels inspired by trainable attention priors. |
| - **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) — |
| Replaces dense attention output projection with a structured Hadamard transform plus |
| lightweight scaling. |
| |
| **Normalization, residual flow, and MLP** |
| |
| - **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K |
| projections. Dynamically rescales normalization from the input's own statistics. |
| - **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each |
| layer's output is scaled by 1/√ℓ where ℓ is the layer index. |
| - **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving |
| Activation Scaling for residual junctions. |
| - **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard |
| MLP activation with normalized linear, quadratic, and cubic branches. |
| - **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order |
| geometry-inspired normalization strategy applied inside MLP projections. |
| - **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) — |
| Optional differentiable hidden-state stack between decoder layers. |
| - **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) — |
| Optional learned depth-wise aggregation over previous layer outputs or block summaries. |
| - **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned |
| augmented residual layer with residual-weight and low-rank variants. |
| |
| **Training objectives and training-time regularizers** |
| |
| - **Cut Cross Entropy** ([Apple repository](https://github.com/apple/ml-cross-entropy)) — |
| Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary |
| logits tensor. NeoLLM remains compatible with the upstream package when the extensions |
| below are disabled. |
| - **MiLe Loss** ([arXiv:2310.19531](https://arxiv.org/abs/2310.19531)) — Optional detached, |
| mean-normalized predictive-entropy weighting of token losses, implemented inside the |
| extended CCE path. |
| - **Output Embedding Centering / mu-loss** |
| ([arXiv:2601.02031](https://arxiv.org/abs/2601.02031)) — Optional |
| `lambda * ||mean(output_embeddings)||^2` regularizer for output-logit stability. |
| - **MEAP** ([arXiv:2502.07490](https://arxiv.org/abs/2502.07490)) — Optional training-only |
| input corruption that masks a fixed fraction of eligible tokens while preserving clean |
| next-token labels, causal attention, and the inference path. |
| - **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional |
| Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly |
| training. |
| - **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit |
| Token Prediction auxiliary objective using shallow-layer implicit token targets and a |
| cosine loss. |
| - **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent |
| prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL. |
|
|
| ### Optional extended-CCE configuration |
|
|
| | Feature | Enabled | Value | |
| |---|---:|---:| |
| | MiLe Loss | True | gamma=1.0 | |
| | mu-loss | True | lambda=0.0001 | |
| | MEAP | True | ratio=0.15 | |
|
|
| Copy-Paste uses CCE sparse mixture statistics for CUDA training and computes mu-loss in Torch. MEAP and patch training also use the extended cut-cross-entropy package. |
| [`Kitsunp/ml-cross-entropy`](https://github.com/Kitsunp/ml-cross-entropy) supplies the extended kernels. |
| Copy-Paste exposes three compact |
| scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty. |
| Their sum reconstructs `ntp_loss` exactly. MEAP reports its eligible and selected counts from |
| the masking kernel; the trainer logs the selected count and exact fraction. Copy-Paste diagnostics use sparse source-token candidates and CCE vocabulary reductions. |
|
|
| **Optimizer and training stability** |
|
|
| - **Adam-family optimizer mode** — Active variant: **AdEMAMix**. |
| **AdEMAMix** ([arXiv:2409.03137](https://arxiv.org/abs/2409.03137)) contributes the optional |
| beta3 slow-memory branch when `ADEMAMIX=True`. **DeltaMomentum** |
| ([arXiv:2608.19491](https://arxiv.org/abs/2608.19491)) optionally replaces only the fast beta1 |
| first-moment update on hidden Linear weights when `DELTA=True`; beta2 remains Adam's |
| ordinary squared-gradient EMA in every mode. |
| - **BF16 stochastic rounding** ([arXiv:2502.20566](https://arxiv.org/abs/2502.20566)) — |
| Optional master-free BF16 model update with stochastic rather than nearest rounding; the same |
| write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix. |
| - **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) — |
| Sign-selective weight decay variant used by the custom optimizer logic. |
| - **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) — |
| Adapts decoupled weight decay during learning-rate decay. |
| - **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) — |
| Decoupled Huber decay regularization reference used by the optimizer. |
| - **Gradient conditioning** — CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates. |
| References: [CAST](https://arxiv.org/abs/2610.19761), |
| [GradientStabilizer](https://arxiv.org/abs/2502.17055). |
| --- |
|
|
| ## Training |
|
|
| | Setting | Value | |
| |---|---| |
| | Dataset | FineWeb-Edu (sample-10BT) | |
| | Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) | |
| | Precision | BF16 model state (bfloat16: 85,091,128 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled | |
| | Optimizer | AdEMAMix | |
| | Gradient transform | CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates | |
| | Learning rate | 1e-03 with 4,000-step linear warmup | |
| | Weight decay | coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0 | |
| | Training time | ~0h 47m | |
| | Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition | |
|
|
|
|
| ### Training curve — total objective |
|
|
| Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe. |
|
|
| | Step | Train total | Val total | Train without MiLe | Val without MiLe | |
| |---|---|---|---|---| |
| | 1 | 12.2845 | — | 12.2845 | — | |
| | 250 | 7.51328 | — | 7.50593 | — | |
| | 500 | 7.37632 | — | 7.36168 | — | |
| | 750 | 7.33534 | — | 7.31743 | — | |
| | 1,000 | 7.2956 | — | 7.27509 | — | |
| | 1,250 | 7.25757 | — | 7.2354 | — | |
| | 1,500 | 7.22776 | — | 7.20332 | — | |
| | 1,750 | 7.18971 | — | 7.16376 | — | |
| | 2,000 | 7.16679 | — | 7.13879 | — | |
| | 2,250 | 7.15075 | — | 7.12142 | — | |
| | 2,500 | 7.1182 | — | 7.08786 | — | |
| | 2,750 | 7.09542 | — | 7.064 | — | |
| | 3,000 | 7.07595 | — | 7.0437 | — | |
| | 3,250 | 7.05828 | — | 7.02471 | — | |
| | 3,500 | 7.04374 | — | 7.00896 | — | |
| | 3,750 | 7.03011 | — | 6.99452 | — | |
| | 4,000 | 7.01765 | — | 6.98176 | — | |
| | 4,250 | 7.00604 | — | 6.96907 | — | |
| | 4,500 | 6.99175 | — | 6.95386 | — | |
| | 4,750 | 6.97323 | — | 6.93439 | — | |
| | 5,000 | 6.95435 | 7.03772 | 6.91428 | 6.97998 | |
| | 5,250 | 6.94098 | — | 6.90037 | — | |
| | 5,500 | 6.9301 | — | 6.88864 | — | |
| | 5,750 | 6.91726 | — | 6.87426 | — | |
| | 6,000 | 6.905 | — | 6.86125 | — | |
| | 6,250 | 6.89508 | — | 6.85136 | — | |
| | 6,500 | 6.8811 | — | 6.8378 | — | |
| | 6,750 | 6.87753 | — | 6.8327 | — | |
| | 7,000 | 6.86382 | — | 6.81881 | — | |
| | 7,250 | 6.85904 | — | 6.81314 | — | |
| | 7,500 | 6.85565 | — | 6.8089 | — | |
| | 7,750 | 6.84594 | — | 6.79925 | — | |
| | 8,000 | 6.8399 | — | 6.79186 | — | |
| | 8,250 | 6.8271 | — | 6.77829 | — | |
| | 8,500 | 6.82443 | — | 6.77634 | — | |
| | 8,750 | 6.8251 | — | 6.77586 | — | |
| | 9,000 | 6.81961 | — | 6.77061 | — | |
| | 9,250 | 6.80826 | — | 6.75788 | — | |
| | 9,500 | 6.80517 | — | 6.75465 | — | |
| | 9,750 | 6.805 | — | 6.7549 | — | |
| | 10,000 | 6.79291 | 6.90652 | 6.74131 | 6.83695 | |
| | 10,250 | 6.79435 | — | 6.74302 | — | |
| | 10,500 | 6.57804 | — | 6.46818 | — | |
| | 10,750 | 5.64887 | — | 5.30678 | — | |
| | 11,000 | 5.50961 | — | 5.1322 | — | |
| | 11,250 | 5.44243 | — | 5.04573 | — | |
| | 11,500 | 5.39135 | — | 4.97991 | — | |
| | 11,750 | 5.3537 | — | 4.93408 | — | |
| | 12,000 | 5.32865 | — | 4.90204 | — | |
| | 12,250 | 5.29837 | — | 4.86656 | — | |
| | 12,500 | 5.26806 | — | 4.83252 | — | |
| | 12,750 | 5.25518 | — | 4.81263 | — | |
| | 13,000 | 5.22838 | — | 4.78307 | — | |
| | 13,250 | 5.2163 | — | 4.7663 | — | |
| | 13,500 | 5.1947 | — | 4.74032 | — | |
| | 13,750 | 5.19089 | — | 4.73445 | — | |
| | 14,000 | 5.16889 | — | 4.71043 | — | |
| | 14,250 | 5.1559 | — | 4.69624 | — | |
| | 14,500 | 5.14548 | — | 4.68216 | — | |
| | 14,750 | 5.13203 | — | 4.66433 | — | |
| | 15,000 | 5.13145 | 4.7568 | 4.66355 | 4.18981 | |
| | 15,250 | 5.11204 | — | 4.64183 | — | |
| | 15,500 | 5.10473 | — | 4.63283 | — | |
| | 15,625 | 5.09438 | 4.7291 | 4.62103 | 4.15709 | |
|
|
| ### Cross-entropy with and without MiLe |
|
|
| CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. `ntp_loss` includes mu when enabled. |
|
|
| | Step | Train CE without MiLe | Val CE without MiLe | Train CE with MiLe | Val CE with MiLe | |
| |---|---|---|---|---| |
| | 1 | 11.2377 | — | 11.2377 | — | |
| | 250 | 7.33083 | — | 7.33818 | — | |
| | 500 | 7.01992 | — | 7.03457 | — | |
| | 750 | 6.9117 | — | 6.92962 | — | |
| | 1,000 | 6.84122 | — | 6.86173 | — | |
| | 1,250 | 6.78967 | — | 6.81184 | — | |
| | 1,500 | 6.75051 | — | 6.77496 | — | |
| | 1,750 | 6.71397 | — | 6.73992 | — | |
| | 2,000 | 6.68534 | — | 6.71334 | — | |
| | 2,250 | 6.66963 | — | 6.69896 | — | |
| | 2,500 | 6.64192 | — | 6.67227 | — | |
| | 2,750 | 6.621 | — | 6.65242 | — | |
| | 3,000 | 6.60442 | — | 6.63668 | — | |
| | 3,250 | 6.59009 | — | 6.62366 | — | |
| | 3,500 | 6.5758 | — | 6.61058 | — | |
| | 3,750 | 6.55762 | — | 6.59321 | — | |
| | 4,000 | 6.54081 | — | 6.5767 | — | |
| | 4,250 | 6.52894 | — | 6.56591 | — | |
| | 4,500 | 6.51391 | — | 6.5518 | — | |
| | 4,750 | 6.49645 | — | 6.53529 | — | |
| | 5,000 | 6.4769 | 6.35905 | 6.51696 | 6.41679 | |
| | 5,250 | 6.46543 | — | 6.50604 | — | |
| | 5,500 | 6.45592 | — | 6.49738 | — | |
| | 5,750 | 6.441 | — | 6.484 | — | |
| | 6,000 | 6.43025 | — | 6.474 | — | |
| | 6,250 | 6.42147 | — | 6.4652 | — | |
| | 6,500 | 6.40953 | — | 6.45284 | — | |
| | 6,750 | 6.40453 | — | 6.44936 | — | |
| | 7,000 | 6.39054 | — | 6.43555 | — | |
| | 7,250 | 6.38482 | — | 6.43072 | — | |
| | 7,500 | 6.3804 | — | 6.42715 | — | |
| | 7,750 | 6.36999 | — | 6.41668 | — | |
| | 8,000 | 6.36302 | — | 6.41106 | — | |
| | 8,250 | 6.35259 | — | 6.4014 | — | |
| | 8,500 | 6.35024 | — | 6.39833 | — | |
| | 8,750 | 6.34786 | — | 6.3971 | — | |
| | 9,000 | 6.34222 | — | 6.39122 | — | |
| | 9,250 | 6.32834 | — | 6.37873 | — | |
| | 9,500 | 6.32368 | — | 6.37419 | — | |
| | 9,750 | 6.32497 | — | 6.37507 | — | |
| | 10,000 | 6.31065 | 6.20464 | 6.36225 | 6.27421 | |
| | 10,250 | 6.31157 | — | 6.3629 | — | |
| | 10,500 | 6.01485 | — | 6.12471 | — | |
| | 10,750 | 4.89594 | — | 5.23804 | — | |
| | 11,000 | 4.72479 | — | 5.1022 | — | |
| | 11,250 | 4.63385 | — | 5.03055 | — | |
| | 11,500 | 4.56533 | — | 4.97677 | — | |
| | 11,750 | 4.5158 | — | 4.93542 | — | |
| | 12,000 | 4.48125 | — | 4.90787 | — | |
| | 12,250 | 4.44365 | — | 4.87546 | — | |
| | 12,500 | 4.40877 | — | 4.84431 | — | |
| | 12,750 | 4.38846 | — | 4.83101 | — | |
| | 13,000 | 4.35794 | — | 4.80324 | — | |
| | 13,250 | 4.33989 | — | 4.78989 | — | |
| | 13,500 | 4.31309 | — | 4.76746 | — | |
| | 13,750 | 4.30621 | — | 4.76265 | — | |
| | 14,000 | 4.28204 | — | 4.7405 | — | |
| | 14,250 | 4.26676 | — | 4.72641 | — | |
| | 14,500 | 4.2524 | — | 4.71572 | — | |
| | 14,750 | 4.23404 | — | 4.70174 | — | |
| | 15,000 | 4.23243 | 3.78875 | 4.70032 | 4.35574 | |
| | 15,250 | 4.21065 | — | 4.68087 | — | |
| | 15,500 | 4.20151 | — | 4.67341 | — | |
| | 15,625 | 4.18959 | 3.75747 | 4.66293 | 4.32948 | |
|
|
| ### Observed loss components — weighted contributions |
|
|
| Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. `regularization_loss` is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only. |
|
|
| | Step | Phase | mu | TWeo | NITP | Reg. subtotal | |
| |---|---|---|---|---|---| |
| | 1 | train | 3.16275e-10 | 8.85299e-07 | 1.0468 | 8.85615e-07 | |
| | 250 | train | 4.01168e-06 | 0.00101028 | 0.174095 | 0.00101429 | |
| | 500 | train | 8.93938e-06 | 0.000425533 | 0.341321 | 0.000434472 | |
| | 750 | train | 1.49341e-05 | 0.000574217 | 0.405132 | 0.000589151 | |
| | 1,000 | train | 2.21405e-05 | 0.000717058 | 0.433133 | 0.000739198 | |
| | 1,250 | train | 3.00225e-05 | 0.000947325 | 0.444758 | 0.000977348 | |
| | 1,500 | train | 3.7833e-05 | 0.00116823 | 0.451597 | 0.00120607 | |
| | 1,750 | train | 4.49507e-05 | 0.00147651 | 0.448272 | 0.00152146 | |
| | 2,000 | train | 5.20222e-05 | 0.00178034 | 0.451619 | 0.00183236 | |
| | 2,250 | train | 5.97428e-05 | 0.00206258 | 0.449671 | 0.00212232 | |
| | 2,500 | train | 6.78635e-05 | 0.00220622 | 0.443663 | 0.00227409 | |
| | 2,750 | train | 7.61504e-05 | 0.00231168 | 0.440618 | 0.00238783 | |
| | 3,000 | train | 8.58517e-05 | 0.00233333 | 0.436857 | 0.00241919 | |
| | 3,250 | train | 9.77149e-05 | 0.00236563 | 0.432168 | 0.00246334 | |
| | 3,500 | train | 0.000110827 | 0.00231476 | 0.430735 | 0.00242559 | |
| | 3,750 | train | 0.000126875 | 0.00226564 | 0.434512 | 0.00239251 | |
| | 4,000 | train | 0.000145417 | 0.00211525 | 0.438694 | 0.00226067 | |
| | 4,250 | train | 0.000167628 | 0.00196029 | 0.437996 | 0.00212792 | |
| | 4,500 | train | 0.000194782 | 0.00182893 | 0.437925 | 0.00202371 | |
| | 4,750 | train | 0.000227027 | 0.00170113 | 0.436016 | 0.00192816 | |
| | 5,000 | train | 0.000263738 | 0.00154546 | 0.435576 | 0.00180919 | |
| | 5,000 | validation | 0.000283719 | 0.0060813 | 0.614562 | 0.00636502 | |
| | 5,250 | train | 0.000303911 | 0.00140913 | 0.433229 | 0.00171304 | |
| | 5,500 | train | 0.000345929 | 0.00130875 | 0.431072 | 0.00165468 | |
| | 5,750 | train | 0.000391956 | 0.00119768 | 0.431669 | 0.00158963 | |
| | 6,000 | train | 0.000435046 | 0.00113317 | 0.429428 | 0.00156821 | |
| | 6,250 | train | 0.000476276 | 0.00107261 | 0.428334 | 0.00154889 | |
| | 6,500 | train | 0.000520947 | 0.000988251 | 0.426753 | 0.0015092 | |
| | 6,750 | train | 0.000561467 | 0.000925999 | 0.426688 | 0.00148747 | |
| | 7,000 | train | 0.000600642 | 0.000889937 | 0.426782 | 0.00149058 | |
| | 7,250 | train | 0.000640212 | 0.000860261 | 0.426819 | 0.00150047 | |
| | 7,500 | train | 0.000679096 | 0.000852769 | 0.426972 | 0.00153187 | |
| | 7,750 | train | 0.000715778 | 0.000802992 | 0.427743 | 0.00151877 | |
| | 8,000 | train | 0.000753992 | 0.00078903 | 0.4273 | 0.00154302 | |
| | 8,250 | train | 0.000795478 | 0.000771698 | 0.424126 | 0.00156718 | |
| | 8,500 | train | 0.000837118 | 0.000758088 | 0.424502 | 0.00159521 | |
| | 8,750 | train | 0.000873988 | 0.00075022 | 0.426376 | 0.00162421 | |
| | 9,000 | train | 0.000910428 | 0.000754263 | 0.426734 | 0.00166469 | |
| | 9,250 | train | 0.000944036 | 0.000730808 | 0.427862 | 0.00167484 | |
| | 9,500 | train | 0.000975691 | 0.00071411 | 0.429285 | 0.0016898 | |
| | 9,750 | train | 0.00100823 | 0.000682743 | 0.428235 | 0.00169098 | |
| | 10,000 | train | 0.00104292 | 0.000664123 | 0.428943 | 0.00170704 | |
| | 10,000 | validation | 0.00106212 | 0.0022162 | 0.629043 | 0.00327832 | |
| | 10,250 | train | 0.00108166 | 0.000681578 | 0.429688 | 0.00176324 | |
| | 10,500 | train | 0.0011197 | 0.00105885 | 0.451152 | 0.00217854 | |
| | 10,750 | train | 0.00114721 | 0.000800759 | 0.408884 | 0.00194796 | |
| | 11,000 | train | 0.00114271 | 0.000609513 | 0.405662 | 0.00175222 | |
| | 11,250 | train | 0.00112443 | 0.000589441 | 0.410163 | 0.00171387 | |
| | 11,500 | train | 0.00110764 | 0.000626262 | 0.412851 | 0.0017339 | |
| | 11,750 | train | 0.0011031 | 0.0007034 | 0.416477 | 0.0018065 | |
| | 12,000 | train | 0.00110705 | 0.000701879 | 0.418976 | 0.00180893 | |
| | 12,250 | train | 0.00111366 | 0.000731281 | 0.42106 | 0.00184494 | |
| | 12,500 | train | 0.00112138 | 0.000758367 | 0.421869 | 0.00187974 | |
| | 12,750 | train | 0.00113102 | 0.000750959 | 0.422287 | 0.00188198 | |
| | 13,000 | train | 0.00114004 | 0.000732814 | 0.423269 | 0.00187286 | |
| | 13,250 | train | 0.0011468 | 0.000694571 | 0.424568 | 0.00184138 | |
| | 13,500 | train | 0.00115065 | 0.000657363 | 0.425431 | 0.00180801 | |
| | 13,750 | train | 0.00115104 | 0.000615922 | 0.426476 | 0.00176697 | |
| | 14,000 | train | 0.00115097 | 0.000595539 | 0.42664 | 0.00174651 | |
| | 14,250 | train | 0.00115072 | 0.000557399 | 0.427775 | 0.00170812 | |
| | 14,500 | train | 0.00114966 | 0.000522873 | 0.428093 | 0.00167254 | |
| | 14,750 | train | 0.00115023 | 0.000500229 | 0.428635 | 0.00165046 | |
| | 15,000 | train | 0.00115158 | 0.000471158 | 0.429507 | 0.00162274 | |
| | 15,000 | validation | 0.00115209 | 0.000515729 | 0.399388 | 0.00166782 | |
| | 15,250 | train | 0.00115244 | 0.000452634 | 0.429571 | 0.00160508 | |
| | 15,500 | train | 0.00115244 | 0.000431864 | 0.429741 | 0.00158431 | |
| | 15,625 | train | 0.00115257 | 0.000415416 | 0.429879 | 0.00156799 | |
| | 15,625 | validation | 0.00115263 | 0.000468992 | 0.397996 | 0.00162163 | |
|
|
| Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks |
| will improve with more training. |
| - **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that |
| previously captured long-range token correlations. A checkpoint from ~step 38k is expected |
| to have better aggregate benchmark scores. |
| - **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the |
| linear branch. Will be corrected in the next training run. |
| - **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction |
| base model. |
|
|
| --- |
|
|
| ## References |
|
|
| All papers whose techniques are integrated into NeoLLM's architecture, |
| training objective, or training stack: |
|
|
| | Area | Technique | Paper title | Reference | |
| |---|---|---|---| |
| | Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) | |
| | Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) | |
| | Embeddings / residual modulation | JTok / JTok-M | JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) | |
| | Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) | |
| | Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) | |
| | Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) | |
| | Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) | |
| | Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) | |
| | Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) | |
| | Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) | |
| | Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) | |
| | Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) | |
| | Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) | |
| | Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) | |
| | Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) | |
| | Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) | |
| | Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) | |
| | Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) | |
| | Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) | |
| | Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) | |
| | Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) | |
| | Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) | |
| | Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) | |
| | Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) | |
| | Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) | |
| | Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) | |
| | Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) | |
| | Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) | |
| | Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) | |
| | Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) | |
| | Optimizer / training | AdEMAMix | The AdEMAMix Optimizer: Better, Faster, Older | [arXiv:2409.03137](https://arxiv.org/abs/2409.03137) | |
| | Optimizer / training | DeltaMomentum | DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule | [arXiv:2608.19491](https://arxiv.org/abs/2608.19491) | |
| | Optimizer / training | BF16 + SR | Stochastic Rounding for LLM Training: Theory and Practice | [arXiv:2502.20566](https://arxiv.org/abs/2502.20566) | |
| | Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) | |
| | Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) | |
| | Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) | |
| | Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) | |
| |
| --- |
| |
| ## Citation |
| |
| ```bibtex |
| @misc{neollm2026, |
| title = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques}, |
| author = {KitsuVp}, |
| year = {2026}, |
| url = {https://huggingface.co/KitsuVp/NeoLLM} |
| } |
| ``` |
| |
| --- |
| |
| ## Author |
| |
| [@Kyokopom](https://x.com/Kyokopom) on X |
| |
| --- |
| |
| ## License |
| |
| Apache 2.0 |
| |
