---
language: en
license: apache-2.0
tags:
- causal-lm
- research
- bf16
- attention
- normalization
- neollm
datasets:
- HuggingFaceFW/fineweb-edu
---

# NeoLLM

NeoLLM is a **85.09 M parameter** decoder-only language model trained from scratch on
[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) with
**BF16** compute, completing training in approximately
**~0h 47m** on **NVIDIA RTX PRO 6000 Blackwell Server Edition**.
It integrates a collection of recently published attention and normalization techniques
into a single architecture, with the goal of studying how they interact during
pretraining. The model is actively being developed and the current checkpoint represents
an intermediate training state.

> **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
> **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)

---

## Architecture

NeoLLM is a decoder-only transformer with the following configuration:

| Parameter | Value |
|---|---|
| Hidden size | 512 |
| Layers | 12 |
| Attention heads | 8 |
| KV heads (GQA) | 4 |
| Head dim | 64 |
| Intermediate size | 1536 |
| Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
| Context length | 512 tokens |
| Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
| JTok / JTok-M | enabled=False / False; Torch continuous surfaces, experts=5, top_k=2 |
| DAGFormer | enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'} |
| Virtual Head Attention | enabled=False; query/KV heads follow the configured geometry |
| Copy-Paste LM | enabled=True; causal vocabulary/pointer mixture |

### Parameter breakdown

| Parameter bucket | Count |
|---|---|
| **Total parameters** | 85.09M (85,093,304) |
| **Embedding parameters** (tied) | 32.97M (32,973,824) |
| **Non-embedding parameters** | 52.12M (52,119,480) |
| **Effective trainable parameters** | 85.09M (85,093,304) |

> Weight tying is **enabled**: the input embedding matrix and the language-model head
> share the same parameters, so the effective trainable budget is
> `total − embed = 52.12M`.

### Integrated techniques

NeoLLM combines architecture modules, optional auxiliary objectives, and
training-time optimizer/stability components from the following papers.

**Embedding and token representation**

- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
  per-row and per-column learnable scalar parameters to selected matrix layers and, when
  enabled, embeddings.
- **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
  continuous token embedding generator that can replace the discrete input lookup table.
- **Leviathan-JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) —
  Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok
  applies one normalized surface as a multiplicative MLP gate; JTok-M routes a
  Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled
  residual. Runtime ``jtok_kernel_backend`` selects the PyTorch reference or the
  CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
- **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
  reference used by the Leviathan continuous token generator implementation.
- **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
  Augments token embeddings with character-level spelling information.
- **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
  Reference motivation for treating token embeddings as structured objects rather than
  unconstrained lookup rows.

**Attention, positions, and output projection**

- **Virtual Head Attention** ([arXiv:2610.17959](https://arxiv.org/abs/2610.17959)) — Optional
  group-specific query premix and low-rank head postmix. NeoLLM shares its gated query
  projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are
  not automatically folded into standard GQA projections.
- **Copy-Paste LM** ([arXiv:2610.19247](https://arxiv.org/abs/2610.19247)) — Optional
  causal pointer and sink mixed with the vocabulary distribution. CUDA training
  uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits.
- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
  A portion of the projection channels are dedicated to periodic cosine/sine features.
- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
  Attention. Adds small learnable interaction matrices between attention heads for K and V.
- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
  lower-triangular preconditioner to V before attention, decorrelating value representations
  across positions.
- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
  two learnable per-head scalars (α and β) to the softmax weights:
  `[α·softmax(QKᵀ) + β]·V`.
- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
  After computing attention, removes the component of the output aligned with the token's
  own value vector.
- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
  learns K=4 directions in the output space; a learned router suppresses the attention output
  along each direction per input.
- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
  is applied to the attention output before the output projection, introducing non-linearity
  and preventing attention sinks.
- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
  and K by subtracting a fraction of the previous position's Q and K values (causal
  first-difference).
- **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
  Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
  per original head.
- **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
  module that learns contextual position coordinates above a configurable start layer.
- **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
  position encoding used by the REPO-GRAPE positional path.
- **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
  factorized attention log-prior channels inspired by trainable attention priors.
- **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
  Replaces dense attention output projection with a structured Hadamard transform plus
  lightweight scaling.

**Normalization, residual flow, and MLP**

- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
  projections. Dynamically rescales normalization from the input's own statistics.
- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
  layer's output is scaled by 1/√ℓ where ℓ is the layer index.
- **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
  Activation Scaling for residual junctions.
- **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
  MLP activation with normalized linear, quadratic, and cubic branches.
- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
  geometry-inspired normalization strategy applied inside MLP projections.
- **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
  Optional differentiable hidden-state stack between decoder layers.
- **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
  Optional learned depth-wise aggregation over previous layer outputs or block summaries.
- **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
  augmented residual layer with residual-weight and low-rank variants.

**Training objectives and training-time regularizers**

- **Cut Cross Entropy** ([Apple repository](https://github.com/apple/ml-cross-entropy)) —
  Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary
  logits tensor. NeoLLM remains compatible with the upstream package when the extensions
  below are disabled.
- **MiLe Loss** ([arXiv:2310.19531](https://arxiv.org/abs/2310.19531)) — Optional detached,
  mean-normalized predictive-entropy weighting of token losses, implemented inside the
  extended CCE path.
- **Output Embedding Centering / mu-loss**
  ([arXiv:2601.02031](https://arxiv.org/abs/2601.02031)) — Optional
  `lambda * ||mean(output_embeddings)||^2` regularizer for output-logit stability.
- **MEAP** ([arXiv:2502.07490](https://arxiv.org/abs/2502.07490)) — Optional training-only
  input corruption that masks a fixed fraction of eligible tokens while preserving clean
  next-token labels, causal attention, and the inference path.
- **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
  Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
  training.
- **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
  Token Prediction auxiliary objective using shallow-layer implicit token targets and a
  cosine loss.
- **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
  prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.

### Optional extended-CCE configuration

| Feature | Enabled | Value |
|---|---:|---:|
| MiLe Loss | True | gamma=1.0 |
| mu-loss | True | lambda=0.0001 |
| MEAP | True | ratio=0.15 |

Copy-Paste uses CCE sparse mixture statistics for CUDA training and computes mu-loss in Torch. MEAP and patch training also use the extended cut-cross-entropy package.
[`Kitsunp/ml-cross-entropy`](https://github.com/Kitsunp/ml-cross-entropy) supplies the extended kernels.
Copy-Paste exposes three compact
scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty.
Their sum reconstructs `ntp_loss` exactly. MEAP reports its eligible and selected counts from
the masking kernel; the trainer logs the selected count and exact fraction. Copy-Paste diagnostics use sparse source-token candidates and CCE vocabulary reductions.

**Optimizer and training stability**

- **Adam-family optimizer mode** — Active variant: **AdEMAMix**.
  **AdEMAMix** ([arXiv:2409.03137](https://arxiv.org/abs/2409.03137)) contributes the optional
  beta3 slow-memory branch when `ADEMAMIX=True`. **DeltaMomentum**
  ([arXiv:2608.19491](https://arxiv.org/abs/2608.19491)) optionally replaces only the fast beta1
  first-moment update on hidden Linear weights when `DELTA=True`; beta2 remains Adam's
  ordinary squared-gradient EMA in every mode.
- **BF16 stochastic rounding** ([arXiv:2502.20566](https://arxiv.org/abs/2502.20566)) —
  Optional master-free BF16 model update with stochastic rather than nearest rounding; the same
  write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
- **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
  Sign-selective weight decay variant used by the custom optimizer logic.
- **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
  Adapts decoupled weight decay during learning-rate decay.
- **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
  Decoupled Huber decay regularization reference used by the optimizer.
- **Gradient conditioning** — CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates.
  References: [CAST](https://arxiv.org/abs/2610.19761),
  [GradientStabilizer](https://arxiv.org/abs/2502.17055).
---

## Training

| Setting | Value |
|---|---|
| Dataset | FineWeb-Edu (sample-10BT) |
| Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
| Precision | BF16 model state (bfloat16: 85,091,128 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled |
| Optimizer | AdEMAMix |
| Gradient transform | CAST active; history-adaptive spectral/norm policies by optimizer group; 15625 committed updates |
| Learning rate | 1e-03 with 4,000-step linear warmup |
| Weight decay | coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0 |
| Training time | ~0h 47m |
| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |


### Training curve — total objective

Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.

| Step | Train total | Val total | Train without MiLe | Val without MiLe |
|---|---|---|---|---|
| 1 | 12.2845 | — | 12.2845 | — |
| 250 | 7.51328 | — | 7.50593 | — |
| 500 | 7.37632 | — | 7.36168 | — |
| 750 | 7.33534 | — | 7.31743 | — |
| 1,000 | 7.2956 | — | 7.27509 | — |
| 1,250 | 7.25757 | — | 7.2354 | — |
| 1,500 | 7.22776 | — | 7.20332 | — |
| 1,750 | 7.18971 | — | 7.16376 | — |
| 2,000 | 7.16679 | — | 7.13879 | — |
| 2,250 | 7.15075 | — | 7.12142 | — |
| 2,500 | 7.1182 | — | 7.08786 | — |
| 2,750 | 7.09542 | — | 7.064 | — |
| 3,000 | 7.07595 | — | 7.0437 | — |
| 3,250 | 7.05828 | — | 7.02471 | — |
| 3,500 | 7.04374 | — | 7.00896 | — |
| 3,750 | 7.03011 | — | 6.99452 | — |
| 4,000 | 7.01765 | — | 6.98176 | — |
| 4,250 | 7.00604 | — | 6.96907 | — |
| 4,500 | 6.99175 | — | 6.95386 | — |
| 4,750 | 6.97323 | — | 6.93439 | — |
| 5,000 | 6.95435 | 7.03772 | 6.91428 | 6.97998 |
| 5,250 | 6.94098 | — | 6.90037 | — |
| 5,500 | 6.9301 | — | 6.88864 | — |
| 5,750 | 6.91726 | — | 6.87426 | — |
| 6,000 | 6.905 | — | 6.86125 | — |
| 6,250 | 6.89508 | — | 6.85136 | — |
| 6,500 | 6.8811 | — | 6.8378 | — |
| 6,750 | 6.87753 | — | 6.8327 | — |
| 7,000 | 6.86382 | — | 6.81881 | — |
| 7,250 | 6.85904 | — | 6.81314 | — |
| 7,500 | 6.85565 | — | 6.8089 | — |
| 7,750 | 6.84594 | — | 6.79925 | — |
| 8,000 | 6.8399 | — | 6.79186 | — |
| 8,250 | 6.8271 | — | 6.77829 | — |
| 8,500 | 6.82443 | — | 6.77634 | — |
| 8,750 | 6.8251 | — | 6.77586 | — |
| 9,000 | 6.81961 | — | 6.77061 | — |
| 9,250 | 6.80826 | — | 6.75788 | — |
| 9,500 | 6.80517 | — | 6.75465 | — |
| 9,750 | 6.805 | — | 6.7549 | — |
| 10,000 | 6.79291 | 6.90652 | 6.74131 | 6.83695 |
| 10,250 | 6.79435 | — | 6.74302 | — |
| 10,500 | 6.57804 | — | 6.46818 | — |
| 10,750 | 5.64887 | — | 5.30678 | — |
| 11,000 | 5.50961 | — | 5.1322 | — |
| 11,250 | 5.44243 | — | 5.04573 | — |
| 11,500 | 5.39135 | — | 4.97991 | — |
| 11,750 | 5.3537 | — | 4.93408 | — |
| 12,000 | 5.32865 | — | 4.90204 | — |
| 12,250 | 5.29837 | — | 4.86656 | — |
| 12,500 | 5.26806 | — | 4.83252 | — |
| 12,750 | 5.25518 | — | 4.81263 | — |
| 13,000 | 5.22838 | — | 4.78307 | — |
| 13,250 | 5.2163 | — | 4.7663 | — |
| 13,500 | 5.1947 | — | 4.74032 | — |
| 13,750 | 5.19089 | — | 4.73445 | — |
| 14,000 | 5.16889 | — | 4.71043 | — |
| 14,250 | 5.1559 | — | 4.69624 | — |
| 14,500 | 5.14548 | — | 4.68216 | — |
| 14,750 | 5.13203 | — | 4.66433 | — |
| 15,000 | 5.13145 | 4.7568 | 4.66355 | 4.18981 |
| 15,250 | 5.11204 | — | 4.64183 | — |
| 15,500 | 5.10473 | — | 4.63283 | — |
| 15,625 | 5.09438 | 4.7291 | 4.62103 | 4.15709 |

### Cross-entropy with and without MiLe

CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. `ntp_loss` includes mu when enabled.

| Step | Train CE without MiLe | Val CE without MiLe | Train CE with MiLe | Val CE with MiLe |
|---|---|---|---|---|
| 1 | 11.2377 | — | 11.2377 | — |
| 250 | 7.33083 | — | 7.33818 | — |
| 500 | 7.01992 | — | 7.03457 | — |
| 750 | 6.9117 | — | 6.92962 | — |
| 1,000 | 6.84122 | — | 6.86173 | — |
| 1,250 | 6.78967 | — | 6.81184 | — |
| 1,500 | 6.75051 | — | 6.77496 | — |
| 1,750 | 6.71397 | — | 6.73992 | — |
| 2,000 | 6.68534 | — | 6.71334 | — |
| 2,250 | 6.66963 | — | 6.69896 | — |
| 2,500 | 6.64192 | — | 6.67227 | — |
| 2,750 | 6.621 | — | 6.65242 | — |
| 3,000 | 6.60442 | — | 6.63668 | — |
| 3,250 | 6.59009 | — | 6.62366 | — |
| 3,500 | 6.5758 | — | 6.61058 | — |
| 3,750 | 6.55762 | — | 6.59321 | — |
| 4,000 | 6.54081 | — | 6.5767 | — |
| 4,250 | 6.52894 | — | 6.56591 | — |
| 4,500 | 6.51391 | — | 6.5518 | — |
| 4,750 | 6.49645 | — | 6.53529 | — |
| 5,000 | 6.4769 | 6.35905 | 6.51696 | 6.41679 |
| 5,250 | 6.46543 | — | 6.50604 | — |
| 5,500 | 6.45592 | — | 6.49738 | — |
| 5,750 | 6.441 | — | 6.484 | — |
| 6,000 | 6.43025 | — | 6.474 | — |
| 6,250 | 6.42147 | — | 6.4652 | — |
| 6,500 | 6.40953 | — | 6.45284 | — |
| 6,750 | 6.40453 | — | 6.44936 | — |
| 7,000 | 6.39054 | — | 6.43555 | — |
| 7,250 | 6.38482 | — | 6.43072 | — |
| 7,500 | 6.3804 | — | 6.42715 | — |
| 7,750 | 6.36999 | — | 6.41668 | — |
| 8,000 | 6.36302 | — | 6.41106 | — |
| 8,250 | 6.35259 | — | 6.4014 | — |
| 8,500 | 6.35024 | — | 6.39833 | — |
| 8,750 | 6.34786 | — | 6.3971 | — |
| 9,000 | 6.34222 | — | 6.39122 | — |
| 9,250 | 6.32834 | — | 6.37873 | — |
| 9,500 | 6.32368 | — | 6.37419 | — |
| 9,750 | 6.32497 | — | 6.37507 | — |
| 10,000 | 6.31065 | 6.20464 | 6.36225 | 6.27421 |
| 10,250 | 6.31157 | — | 6.3629 | — |
| 10,500 | 6.01485 | — | 6.12471 | — |
| 10,750 | 4.89594 | — | 5.23804 | — |
| 11,000 | 4.72479 | — | 5.1022 | — |
| 11,250 | 4.63385 | — | 5.03055 | — |
| 11,500 | 4.56533 | — | 4.97677 | — |
| 11,750 | 4.5158 | — | 4.93542 | — |
| 12,000 | 4.48125 | — | 4.90787 | — |
| 12,250 | 4.44365 | — | 4.87546 | — |
| 12,500 | 4.40877 | — | 4.84431 | — |
| 12,750 | 4.38846 | — | 4.83101 | — |
| 13,000 | 4.35794 | — | 4.80324 | — |
| 13,250 | 4.33989 | — | 4.78989 | — |
| 13,500 | 4.31309 | — | 4.76746 | — |
| 13,750 | 4.30621 | — | 4.76265 | — |
| 14,000 | 4.28204 | — | 4.7405 | — |
| 14,250 | 4.26676 | — | 4.72641 | — |
| 14,500 | 4.2524 | — | 4.71572 | — |
| 14,750 | 4.23404 | — | 4.70174 | — |
| 15,000 | 4.23243 | 3.78875 | 4.70032 | 4.35574 |
| 15,250 | 4.21065 | — | 4.68087 | — |
| 15,500 | 4.20151 | — | 4.67341 | — |
| 15,625 | 4.18959 | 3.75747 | 4.66293 | 4.32948 |

### Observed loss components — weighted contributions

Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. `regularization_loss` is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.

| Step | Phase | mu | TWeo | NITP | Reg. subtotal |
|---|---|---|---|---|---|
| 1 | train | 3.16275e-10 | 8.85299e-07 | 1.0468 | 8.85615e-07 |
| 250 | train | 4.01168e-06 | 0.00101028 | 0.174095 | 0.00101429 |
| 500 | train | 8.93938e-06 | 0.000425533 | 0.341321 | 0.000434472 |
| 750 | train | 1.49341e-05 | 0.000574217 | 0.405132 | 0.000589151 |
| 1,000 | train | 2.21405e-05 | 0.000717058 | 0.433133 | 0.000739198 |
| 1,250 | train | 3.00225e-05 | 0.000947325 | 0.444758 | 0.000977348 |
| 1,500 | train | 3.7833e-05 | 0.00116823 | 0.451597 | 0.00120607 |
| 1,750 | train | 4.49507e-05 | 0.00147651 | 0.448272 | 0.00152146 |
| 2,000 | train | 5.20222e-05 | 0.00178034 | 0.451619 | 0.00183236 |
| 2,250 | train | 5.97428e-05 | 0.00206258 | 0.449671 | 0.00212232 |
| 2,500 | train | 6.78635e-05 | 0.00220622 | 0.443663 | 0.00227409 |
| 2,750 | train | 7.61504e-05 | 0.00231168 | 0.440618 | 0.00238783 |
| 3,000 | train | 8.58517e-05 | 0.00233333 | 0.436857 | 0.00241919 |
| 3,250 | train | 9.77149e-05 | 0.00236563 | 0.432168 | 0.00246334 |
| 3,500 | train | 0.000110827 | 0.00231476 | 0.430735 | 0.00242559 |
| 3,750 | train | 0.000126875 | 0.00226564 | 0.434512 | 0.00239251 |
| 4,000 | train | 0.000145417 | 0.00211525 | 0.438694 | 0.00226067 |
| 4,250 | train | 0.000167628 | 0.00196029 | 0.437996 | 0.00212792 |
| 4,500 | train | 0.000194782 | 0.00182893 | 0.437925 | 0.00202371 |
| 4,750 | train | 0.000227027 | 0.00170113 | 0.436016 | 0.00192816 |
| 5,000 | train | 0.000263738 | 0.00154546 | 0.435576 | 0.00180919 |
| 5,000 | validation | 0.000283719 | 0.0060813 | 0.614562 | 0.00636502 |
| 5,250 | train | 0.000303911 | 0.00140913 | 0.433229 | 0.00171304 |
| 5,500 | train | 0.000345929 | 0.00130875 | 0.431072 | 0.00165468 |
| 5,750 | train | 0.000391956 | 0.00119768 | 0.431669 | 0.00158963 |
| 6,000 | train | 0.000435046 | 0.00113317 | 0.429428 | 0.00156821 |
| 6,250 | train | 0.000476276 | 0.00107261 | 0.428334 | 0.00154889 |
| 6,500 | train | 0.000520947 | 0.000988251 | 0.426753 | 0.0015092 |
| 6,750 | train | 0.000561467 | 0.000925999 | 0.426688 | 0.00148747 |
| 7,000 | train | 0.000600642 | 0.000889937 | 0.426782 | 0.00149058 |
| 7,250 | train | 0.000640212 | 0.000860261 | 0.426819 | 0.00150047 |
| 7,500 | train | 0.000679096 | 0.000852769 | 0.426972 | 0.00153187 |
| 7,750 | train | 0.000715778 | 0.000802992 | 0.427743 | 0.00151877 |
| 8,000 | train | 0.000753992 | 0.00078903 | 0.4273 | 0.00154302 |
| 8,250 | train | 0.000795478 | 0.000771698 | 0.424126 | 0.00156718 |
| 8,500 | train | 0.000837118 | 0.000758088 | 0.424502 | 0.00159521 |
| 8,750 | train | 0.000873988 | 0.00075022 | 0.426376 | 0.00162421 |
| 9,000 | train | 0.000910428 | 0.000754263 | 0.426734 | 0.00166469 |
| 9,250 | train | 0.000944036 | 0.000730808 | 0.427862 | 0.00167484 |
| 9,500 | train | 0.000975691 | 0.00071411 | 0.429285 | 0.0016898 |
| 9,750 | train | 0.00100823 | 0.000682743 | 0.428235 | 0.00169098 |
| 10,000 | train | 0.00104292 | 0.000664123 | 0.428943 | 0.00170704 |
| 10,000 | validation | 0.00106212 | 0.0022162 | 0.629043 | 0.00327832 |
| 10,250 | train | 0.00108166 | 0.000681578 | 0.429688 | 0.00176324 |
| 10,500 | train | 0.0011197 | 0.00105885 | 0.451152 | 0.00217854 |
| 10,750 | train | 0.00114721 | 0.000800759 | 0.408884 | 0.00194796 |
| 11,000 | train | 0.00114271 | 0.000609513 | 0.405662 | 0.00175222 |
| 11,250 | train | 0.00112443 | 0.000589441 | 0.410163 | 0.00171387 |
| 11,500 | train | 0.00110764 | 0.000626262 | 0.412851 | 0.0017339 |
| 11,750 | train | 0.0011031 | 0.0007034 | 0.416477 | 0.0018065 |
| 12,000 | train | 0.00110705 | 0.000701879 | 0.418976 | 0.00180893 |
| 12,250 | train | 0.00111366 | 0.000731281 | 0.42106 | 0.00184494 |
| 12,500 | train | 0.00112138 | 0.000758367 | 0.421869 | 0.00187974 |
| 12,750 | train | 0.00113102 | 0.000750959 | 0.422287 | 0.00188198 |
| 13,000 | train | 0.00114004 | 0.000732814 | 0.423269 | 0.00187286 |
| 13,250 | train | 0.0011468 | 0.000694571 | 0.424568 | 0.00184138 |
| 13,500 | train | 0.00115065 | 0.000657363 | 0.425431 | 0.00180801 |
| 13,750 | train | 0.00115104 | 0.000615922 | 0.426476 | 0.00176697 |
| 14,000 | train | 0.00115097 | 0.000595539 | 0.42664 | 0.00174651 |
| 14,250 | train | 0.00115072 | 0.000557399 | 0.427775 | 0.00170812 |
| 14,500 | train | 0.00114966 | 0.000522873 | 0.428093 | 0.00167254 |
| 14,750 | train | 0.00115023 | 0.000500229 | 0.428635 | 0.00165046 |
| 15,000 | train | 0.00115158 | 0.000471158 | 0.429507 | 0.00162274 |
| 15,000 | validation | 0.00115209 | 0.000515729 | 0.399388 | 0.00166782 |
| 15,250 | train | 0.00115244 | 0.000452634 | 0.429571 | 0.00160508 |
| 15,500 | train | 0.00115244 | 0.000431864 | 0.429741 | 0.00158431 |
| 15,625 | train | 0.00115257 | 0.000415416 | 0.429879 | 0.00156799 |
| 15,625 | validation | 0.00115263 | 0.000468992 | 0.397996 | 0.00162163 |

Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.

---

## Limitations

- **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
  will improve with more training.
- **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
  previously captured long-range token correlations. A checkpoint from ~step 38k is expected
  to have better aggregate benchmark scores.
- **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
  linear branch. Will be corrected in the next training run.
- **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
  base model.

---

## References

All papers whose techniques are integrated into NeoLLM's architecture,
training objective, or training stack:

| Area | Technique | Paper title | Reference |
|---|---|---|---|
| Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
| Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
| Embeddings / residual modulation | JTok / JTok-M | JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
| Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
| Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
| Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
| Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
| Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
| Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
| Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
| Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
| Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
| Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
| Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
| Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
| Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
| Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
| Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
| Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
| Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
| Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
| Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
| Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
| Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
| Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
| Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
| Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
| Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
| Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
| Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
| Optimizer / training | AdEMAMix | The AdEMAMix Optimizer: Better, Faster, Older | [arXiv:2409.03137](https://arxiv.org/abs/2409.03137) |
| Optimizer / training | DeltaMomentum | DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule | [arXiv:2608.19491](https://arxiv.org/abs/2608.19491) |
| Optimizer / training | BF16 + SR | Stochastic Rounding for LLM Training: Theory and Practice | [arXiv:2502.20566](https://arxiv.org/abs/2502.20566) |
| Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
| Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
| Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
| Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |

---

## Citation

```bibtex
@misc{neollm2026,
  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
  author = {KitsuVp},
  year   = {2026},
  url    = {https://huggingface.co/KitsuVp/NeoLLM}
}
```

---

## Author

[@Kyokopom](https://x.com/Kyokopom) on X

---

## License

Apache 2.0