Model save

README.md

@@ -1,535 +1,84 @@
 ---
-language: en
-license: apache-2.0
+library_name: transformers
 tags:
-- causal-lm
-- research
-- bf16
-- attention
-- normalization
-- neollm
-datasets:
-- HuggingFaceFW/fineweb-edu
+- generated_from_trainer
+model-index:
+- name: NeoLLM
+  results: []
 ---
 
-# NeoLLM
-
-NeoLLM is a **84.57 M parameter** decoder-only language model trained from scratch on
-[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) with
-**BF16** compute, completing training in approximately
-**~1h 05m** on **NVIDIA RTX PRO 6000 Blackwell Server Edition**.
-It integrates a collection of recently published attention and normalization techniques
-into a single architecture, with the goal of studying how they interact during
-pretraining. The model is actively being developed and the current checkpoint represents
-an intermediate training state.
-
-> **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
-> **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
-
----
-
-## Architecture
-
-NeoLLM is a decoder-only transformer with the following configuration:
-
-| Parameter | Value |
-|---|---|
-| Hidden size | 512 |
-| Layers | 12 |
-| Attention heads | 8 |
-| KV heads (GQA) | 4 |
-| Head dim | 64 |
-| Intermediate size | 1536 |
-| Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
-| Context length | 512 tokens |
-| Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
-| JTok / JTok-M | enabled=False / False; Torch continuous surfaces, experts=5, top_k=2 |
-| DAGFormer | enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'} |
-| Linear Depth Memory | enabled=False; groups=8, key dimension=8; shared branch history when combined with SiameseNorm |
-| Virtual Head Attention | enabled=False; query/KV heads follow the configured geometry |
-| Copy-Paste LM | enabled=False; causal vocabulary/pointer mixture |
-
-### Parameter breakdown
-
-| Parameter bucket | Count |
-|---|---|
-| **Total parameters** | 84.57M (84,567,992) |
-| **Embedding parameters** (tied) | 32.97M (32,973,824) |
-| **Non-embedding parameters** | 51.59M (51,594,168) |
-| **Effective trainable parameters** | 84.57M (84,567,992) |
-
-> Weight tying is **enabled**: the input embedding matrix and the language-model head
-> share the same parameters, so the effective trainable budget is
-> `total − embed = 51.59M`.
-
-### Integrated techniques
-
-NeoLLM combines architecture modules, optional auxiliary objectives, and
-training-time optimizer/stability components from the following papers.
-
-**Embedding and token representation**
-
-- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
-  per-row and per-column learnable scalar parameters to selected matrix layers and, when
-  enabled, embeddings.
-- **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
-  continuous token embedding generator that can replace the discrete input lookup table.
-- **Leviathan-JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) —
-  Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok
-  applies one normalized surface as a multiplicative MLP gate; JTok-M routes a
-  Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled
-  residual. Runtime ``jtok_kernel_backend`` selects the PyTorch reference or the
-  CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
-- **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
-  reference used by the Leviathan continuous token generator implementation.
-- **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
-  Augments token embeddings with character-level spelling information.
-- **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
-  Reference motivation for treating token embeddings as structured objects rather than
-  unconstrained lookup rows.
-
-**Attention, positions, and output projection**
-
-- **Virtual Head Attention** ([arXiv:2610.17959](https://arxiv.org/abs/2610.17959)) — Optional
-  group-specific query premix and low-rank head postmix. NeoLLM shares its gated query
-  projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are
-  not automatically folded into standard GQA projections.
-- **Copy-Paste LM** ([arXiv:2610.19247](https://arxiv.org/abs/2610.19247)) — Optional
-  causal pointer and sink mixed with the vocabulary distribution. CUDA training
-  uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits.
-- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
-  A portion of the projection channels are dedicated to periodic cosine/sine features.
-- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
-  Attention. Adds small learnable interaction matrices between attention heads for K and V.
-- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
-  lower-triangular preconditioner to V before attention, decorrelating value representations
-  across positions.
-- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
-  two learnable per-head scalars (α and β) to the softmax weights:
-  `[α·softmax(QKᵀ) + β]·V`.
-- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
-  After computing attention, removes the component of the output aligned with the token's
-  own value vector.
-- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
-  learns K=4 directions in the output space; a learned router suppresses the attention output
-  along each direction per input.
-- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
-  is applied to the attention output before the output projection, introducing non-linearity
-  and preventing attention sinks.
-- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
-  and K by subtracting a fraction of the previous position's Q and K values (causal
-  first-difference).
-- **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
-  Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
-  per original head.
-- **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
-  module that learns contextual position coordinates above a configurable start layer.
-- **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
-  position encoding used by the REPO-GRAPE positional path.
-- **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
-  factorized attention log-prior channels inspired by trainable attention priors.
-- **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
-  Replaces dense attention output projection with a structured Hadamard transform plus
-  lightweight scaling.
-
-**Normalization, residual flow, and MLP**
-
-- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
-  projections. Dynamically rescales normalization from the input's own statistics.
-- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
-  layer's output is scaled by 1/√ℓ where ℓ is the layer index.
-- **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
-  Activation Scaling for residual junctions.
-- **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
-  MLP activation with normalized linear, quadratic, and cubic branches.
-- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
-  geometry-inspired normalization strategy applied inside MLP projections.
-- **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
-  Optional differentiable hidden-state stack between decoder layers.
-- **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
-  Optional learned depth-wise aggregation over previous layer outputs or block summaries.
-- **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
-  augmented residual layer with residual-weight and low-rank variants.
-
-**Training objectives and training-time regularizers**
-
-- **Cut Cross Entropy** ([Apple repository](https://github.com/apple/ml-cross-entropy)) —
-  Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary
-  logits tensor. NeoLLM remains compatible with the upstream package when the extensions
-  below are disabled.
-- **MiLe Loss** ([arXiv:2310.19531](https://arxiv.org/abs/2310.19531)) — Optional detached,
-  mean-normalized predictive-entropy weighting of token losses, implemented inside the
-  extended CCE path.
-- **Output Embedding Centering / mu-loss**
-  ([arXiv:2601.02031](https://arxiv.org/abs/2601.02031)) — Optional
-  `lambda * ||mean(output_embeddings)||^2` regularizer for output-logit stability.
-- **MEAP** ([arXiv:2502.07490](https://arxiv.org/abs/2502.07490)) — Optional training-only
-  input corruption that masks a fixed fraction of eligible tokens while preserving clean
-  next-token labels, causal attention, and the inference path.
-- **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
-  Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
-  training.
-- **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
-  Token Prediction auxiliary objective using shallow-layer implicit token targets and a
-  cosine loss.
-- **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
-  prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.
-
-### Optional extended-CCE configuration
-
-| Feature | Enabled | Value |
-|---|---:|---:|
-| MiLe Loss | True | gamma=1.0 |
-| mu-loss | True | lambda=0.0001 |
-| MEAP | True | ratio=0.15 |
-
-MiLe, mu-loss, and MEAP require the extended cut-cross-entropy package when their corresponding flags are enabled. With all three disabled, NeoLLM calls upstream CCE without extension-specific arguments.
-[`Kitsunp/ml-cross-entropy`](https://github.com/Kitsunp/ml-cross-entropy) supplies the extended kernels.
-When any extension is active, CCE reports three compact
-scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty.
-Their sum reconstructs `ntp_loss` exactly. MEAP reports its eligible and selected counts from
-the masking kernel; the trainer logs the selected count and exact fraction. No diagnostic path materializes full-vocabulary logits or a token mask outside the kernels.
-
-**Optimizer and training stability**
-
-- **Adam-family optimizer mode** — Active variant: **AdEMAMix**.
-  **AdEMAMix** ([arXiv:2409.03137](https://arxiv.org/abs/2409.03137)) contributes the optional
-  beta3 slow-memory branch when `ADEMAMIX=True`. **DeltaMomentum**
-  ([arXiv:2608.19491](https://arxiv.org/abs/2608.19491)) optionally replaces only the fast beta1
-  first-moment update on hidden Linear weights when `DELTA=True`; beta2 remains Adam's
-  ordinary squared-gradient EMA in every mode.
-- **BF16 stochastic rounding** ([arXiv:2502.20566](https://arxiv.org/abs/2502.20566)) —
-  Optional master-free BF16 model update with stochastic rather than nearest rounding; the same
-  write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
-- **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
-  Sign-selective weight decay variant used by the custom optimizer logic.
-- **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
-  Adapts decoupled weight decay during learning-rate decay.
-- **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
-  Decoupled Huber decay regularization reference used by the optimizer.
-- **Gradient conditioning** — CAST active; history-adaptive spectral/norm policies by optimizer group; 5208 committed updates.
-  References: [CAST](https://arxiv.org/abs/2610.19761),
-  [GradientStabilizer](https://arxiv.org/abs/2502.17055).
----
-
-## Training
-
-| Setting | Value |
-|---|---|
-| Dataset | FineWeb-Edu (sample-10BT) |
-| Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
-| Precision | BF16 model state (bfloat16: 84,565,816 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled |
-| Optimizer | AdEMAMix |
-| Gradient transform | CAST active; history-adaptive spectral/norm policies by optimizer group; 5208 committed updates |
-| Learning rate | 1e-03 with 4,000-step linear warmup |
-| Patch transition | Reset optimizer/conditioner at step 10,417; per-phase LR warmup; restarted AdEMAMix warmup fitted to each phase |
-| Weight decay | coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0 |
-| Training time | ~1h 05m |
-| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
-
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-### Training curve — total objective
-
-Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.
-
-| Step | Train total | Val total | Train without MiLe | Val without MiLe |
-|---|---|---|---|---|
-| 1 | 12.1389 | — | 12.1389 | — |
-| 250 | 7.98201 | — | 7.97185 | — |
-| 500 | 7.81187 | — | 7.79567 | — |
-| 750 | 7.68458 | — | 7.66603 | — |
-| 1,000 | 7.60568 | — | 7.58512 | — |
-| 1,250 | 7.55586 | — | 7.5336 | — |
-| 1,500 | 7.52279 | — | 7.4985 | — |
-| 1,750 | 7.49397 | — | 7.46859 | — |
-| 2,000 | 7.46912 | — | 7.442 | — |
-| 2,250 | 7.45625 | — | 7.42709 | — |
-| 2,500 | 7.4223 | — | 7.39181 | — |
-| 2,750 | 7.40078 | — | 7.3695 | — |
-| 3,000 | 7.38591 | — | 7.35374 | — |
-| 3,250 | 7.36801 | — | 7.33416 | — |
-| 3,500 | 7.35368 | — | 7.3189 | — |
-| 3,750 | 7.33075 | — | 7.29491 | — |
-| 4,000 | 7.31505 | — | 7.27878 | — |
-| 4,250 | 7.29996 | — | 7.26238 | — |
-| 4,500 | 7.285 | — | 7.24703 | — |
-| 4,750 | 7.2642 | — | 7.22509 | — |
-| 5,000 | 7.23908 | 7.37795 | 7.19911 | 7.32346 |
-| 5,250 | 7.22418 | — | 7.18371 | — |
-| 5,500 | 7.21138 | — | 7.16986 | — |
-| 5,750 | 7.19068 | — | 7.1479 | — |
-| 6,000 | 7.17464 | — | 7.13095 | — |
-| 6,250 | 7.16397 | — | 7.12018 | — |
-| 6,500 | 7.14833 | — | 7.10474 | — |
-| 6,750 | 7.14106 | — | 7.09624 | — |
-| 7,000 | 7.12509 | — | 7.07998 | — |
-| 7,250 | 7.12046 | — | 7.07473 | ��� |
-| 7,500 | 7.11547 | — | 7.06928 | — |
-| 7,750 | 7.10348 | — | 7.05701 | — |
-| 8,000 | 7.09825 | — | 7.05018 | — |
-| 8,250 | 7.08208 | — | 7.03349 | — |
-| 8,500 | 7.07402 | — | 7.02515 | — |
-| 8,750 | 7.06957 | — | 7.01997 | — |
-| 9,000 | 7.05336 | — | 7.00369 | — |
-| 9,250 | 7.03475 | — | 6.98337 | — |
-| 9,500 | 7.02346 | — | 6.97189 | — |
-| 9,750 | 7.0177 | — | 6.9663 | — |
-| 10,000 | 6.99797 | 7.18215 | 6.94473 | 7.10898 |
-| 10,250 | 6.9905 | — | 6.93698 | — |
-| 10,500 | 6.843 | — | 6.74639 | — |
-| 10,750 | 5.99854 | — | 5.71293 | — |
-| 11,000 | 5.84056 | — | 5.51851 | — |
-| 11,250 | 5.76222 | — | 5.4185 | — |
-| 11,500 | 5.70637 | — | 5.34297 | — |
-| 11,750 | 5.65901 | — | 5.28121 | — |
-| 12,000 | 5.6265 | — | 5.23588 | — |
-| 12,250 | 5.59507 | — | 5.19298 | — |
-| 12,500 | 5.55902 | — | 5.14764 | — |
-| 12,750 | 5.54703 | — | 5.12462 | — |
-| 13,000 | 5.51924 | — | 5.09024 | — |
-| 13,250 | 5.504 | — | 5.06769 | — |
-| 13,500 | 5.4858 | — | 5.04223 | — |
-| 13,750 | 5.48291 | — | 5.03625 | — |
-| 14,000 | 5.46335 | — | 5.01327 | — |
-| 14,250 | 5.45402 | — | 5.00111 | — |
-| 14,500 | 5.4451 | — | 4.98759 | — |
-| 14,750 | 5.42588 | — | 4.96101 | — |
-| 15,000 | 5.41424 | 5.03226 | 4.9464 | 4.46093 |
-| 15,250 | 5.38022 | — | 4.90862 | — |
-| 15,500 | 5.35956 | — | 4.88272 | — |
-| 15,625 | 5.33575 | 4.97132 | 4.85713 | 4.38465 |
-
-### Cross-entropy with and without MiLe
-
-CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. `ntp_loss` includes mu when enabled.
-
-| Step | Train CE without MiLe | Val CE without MiLe | Train CE with MiLe | Val CE with MiLe |
-|---|---|---|---|---|
-| 1 | 11.1797 | — | 11.1797 | — |
-| 250 | 7.70108 | — | 7.71124 | — |
-| 500 | 7.37505 | — | 7.39125 | — |
-| 750 | 7.23219 | — | 7.25074 | — |
-| 1,000 | 7.14555 | — | 7.16612 | — |
-| 1,250 | 7.08309 | — | 7.10535 | — |
-| 1,500 | 7.03722 | — | 7.0615 | — |
-| 1,750 | 6.9924 | — | 7.01777 | — |
-| 2,000 | 6.95729 | — | 6.98441 | — |
-| 2,250 | 6.93361 | — | 6.96278 | — |
-| 2,500 | 6.89767 | — | 6.92817 | — |
-| 2,750 | 6.87287 | — | 6.90415 | — |
-| 3,000 | 6.85143 | — | 6.8836 | — |
-| 3,250 | 6.82904 | — | 6.8629 | — |
-| 3,500 | 6.81154 | — | 6.84631 | — |
-| 3,750 | 6.7845 | — | 6.82033 | — |
-| 4,000 | 6.7686 | — | 6.80487 | — |
-| 4,250 | 6.75048 | — | 6.78806 | — |
-| 4,500 | 6.73372 | — | 6.7717 | — |
-| 4,750 | 6.7133 | — | 6.7524 | — |
-| 5,000 | 6.68841 | 6.6384 | 6.72839 | 6.69289 |
-| 5,250 | 6.67467 | — | 6.71514 | — |
-| 5,500 | 6.66282 | — | 6.70434 | — |
-| 5,750 | 6.64317 | — | 6.68594 | — |
-| 6,000 | 6.63009 | — | 6.67378 | — |
-| 6,250 | 6.62017 | — | 6.66396 | — |
-| 6,500 | 6.60509 | — | 6.64868 | — |
-| 6,750 | 6.59749 | — | 6.64231 | — |
-| 7,000 | 6.58083 | — | 6.62593 | — |
-| 7,250 | 6.5746 | — | 6.62033 | — |
-| 7,500 | 6.56903 | — | 6.61522 | — |
-| 7,750 | 6.55685 | — | 6.60332 | — |
-| 8,000 | 6.54717 | — | 6.59525 | — |
-| 8,250 | 6.53205 | — | 6.58064 | — |
-| 8,500 | 6.52331 | — | 6.57218 | — |
-| 8,750 | 6.5171 | — | 6.5667 | — |
-| 9,000 | 6.50062 | — | 6.55029 | — |
-| 9,250 | 6.48063 | — | 6.53201 | — |
-| 9,500 | 6.46869 | — | 6.52026 | — |
-| 9,750 | 6.46465 | — | 6.51605 | — |
-| 10,000 | 6.44384 | 6.4122 | 6.49709 | 6.48536 |
-| 10,250 | 6.43573 | — | 6.48926 | — |
-| 10,500 | 6.21725 | — | 6.31387 | — |
-| 10,750 | 5.25332 | — | 5.53894 | — |
-| 11,000 | 5.07612 | — | 5.39817 | — |
-| 11,250 | 4.97318 | — | 5.31689 | — |
-| 11,500 | 4.89312 | — | 5.25652 | — |
-| 11,750 | 4.82637 | — | 5.20417 | — |
-| 12,000 | 4.77527 | — | 5.16589 | — |
-| 12,250 | 4.72859 | — | 5.13068 | — |
-| 12,500 | 4.67986 | — | 5.09124 | — |
-| 12,750 | 4.65348 | — | 5.07588 | — |
-| 13,000 | 4.61638 | — | 5.04537 | — |
-| 13,250 | 4.59109 | — | 5.02739 | — |
-| 13,500 | 4.5643 | — | 5.00787 | — |
-| 13,750 | 4.55635 | — | 5.00301 | — |
-| 14,000 | 4.5327 | — | 4.98277 | — |
-| 14,250 | 4.51994 | — | 4.97285 | — |
-| 14,500 | 4.50645 | — | 4.96395 | — |
-| 14,750 | 4.48103 | — | 4.9459 | — |
-| 15,000 | 4.46639 | 4.0033 | 4.93423 | 4.57464 |
-| 15,250 | 4.43047 | — | 4.90207 | — |
-| 15,500 | 4.40561 | — | 4.88245 | — |
-| 15,625 | 4.38098 | 3.93488 | 4.85961 | 4.52155 |
-
-### Observed loss components — weighted contributions
-
-Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. `regularization_loss` is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.
-
-| Step | Phase | mu | TWeo | NITP | Reg. subtotal |
-|---|---|---|---|---|---|
-| 1 | train | 3.18928e-10 | 8.85302e-07 | 0.959145 | 8.8562e-07 |
-| 250 | train | 5.77321e-06 | 0.00087901 | 0.269885 | 0.000884783 |
-| 500 | train | 1.43175e-05 | 0.000500945 | 0.420108 | 0.000515263 |
-| 750 | train | 2.48131e-05 | 0.000619018 | 0.433192 | 0.000643831 |
-| 1,000 | train | 3.74588e-05 | 0.000851164 | 0.438676 | 0.000888622 |
-| 1,250 | train | 5.11827e-05 | 0.00112427 | 0.449326 | 0.00117545 |
-| 1,500 | train | 6.47691e-05 | 0.00153667 | 0.459677 | 0.00160144 |
-| 1,750 | train | 7.67406e-05 | 0.00191231 | 0.474208 | 0.00198905 |
-| 2,000 | train | 8.72836e-05 | 0.0021987 | 0.482433 | 0.00228598 |
-| 2,250 | train | 9.8353e-05 | 0.00237857 | 0.491002 | 0.00247692 |
-| 2,500 | train | 0.000110801 | 0.0025349 | 0.491486 | 0.0026457 |
-| 2,750 | train | 0.000124411 | 0.00264942 | 0.493857 | 0.00277383 |
-| 3,000 | train | 0.000139189 | 0.00260965 | 0.499559 | 0.00274885 |
-| 3,250 | train | 0.000156565 | 0.00285461 | 0.502105 | 0.00301117 |
-| 3,500 | train | 0.000175885 | 0.00258826 | 0.504605 | 0.00276415 |
-| 3,750 | train | 0.000197083 | 0.0024854 | 0.507733 | 0.00268249 |
-| 4,000 | train | 0.000219011 | 0.00235012 | 0.507615 | 0.00256913 |
-| 4,250 | train | 0.000245202 | 0.00226336 | 0.509393 | 0.00250856 |
-| 4,500 | train | 0.000274499 | 0.00235763 | 0.510672 | 0.00263213 |
-| 4,750 | train | 0.000307991 | 0.00194718 | 0.509539 | 0.00225517 |
-| 5,000 | train | 0.000345114 | 0.00177116 | 0.508583 | 0.00211627 |
-| 5,000 | validation | 0.000365396 | 0.00882081 | 0.675876 | 0.0091862 |
-| 5,250 | train | 0.000386967 | 0.0016488 | 0.506997 | 0.00203576 |
-| 5,500 | train | 0.000430528 | 0.00147078 | 0.505146 | 0.00190131 |
-| 5,750 | train | 0.00047628 | 0.00137825 | 0.502882 | 0.00185453 |
-| 6,000 | train | 0.000526616 | 0.00132051 | 0.499018 | 0.00184713 |
-| 6,250 | train | 0.000575462 | 0.00124181 | 0.498193 | 0.00181728 |
-| 6,500 | train | 0.000623088 | 0.00115507 | 0.497874 | 0.00177816 |
-| 6,750 | train | 0.000668097 | 0.0011365 | 0.496952 | 0.0018046 |
-| 7,000 | train | 0.000705407 | 0.00112174 | 0.497326 | 0.00182715 |
-| 7,250 | train | 0.000734961 | 0.00108176 | 0.498316 | 0.00181672 |
-| 7,500 | train | 0.000765161 | 0.00106831 | 0.498418 | 0.00183347 |
-| 7,750 | train | 0.000799848 | 0.00101314 | 0.498347 | 0.00181299 |
-| 8,000 | train | 0.000837485 | 0.00350294 | 0.498661 | 0.00434042 |
-| 8,250 | train | 0.00087259 | 0.000966043 | 0.499607 | 0.00183863 |
-| 8,500 | train | 0.000900815 | 0.000879673 | 0.500062 | 0.00178049 |
-| 8,750 | train | 0.000921038 | 0.000843696 | 0.501104 | 0.00176473 |
-| 9,000 | train | 0.000937911 | 0.000811567 | 0.501322 | 0.00174948 |
-| 9,250 | train | 0.000951546 | 0.000733545 | 0.501052 | 0.00168509 |
-| 9,500 | train | 0.00096614 | 0.00067595 | 0.501561 | 0.00164209 |
-| 9,750 | train | 0.000979469 | 0.000626824 | 0.500042 | 0.00160629 |
-| 10,000 | train | 0.000989486 | 0.000584204 | 0.499306 | 0.00157369 |
-| 10,000 | validation | 0.000992938 | 0.00224879 | 0.693543 | 0.00324173 |
-| 10,250 | train | 0.000994433 | 0.000535531 | 0.499715 | 0.00152996 |
-| 10,500 | train | 0.000998153 | 0.000950822 | 0.527189 | 0.00194898 |
-| 10,750 | train | 0.00100991 | 0.000895329 | 0.457701 | 0.00190524 |
-| 11,000 | train | 0.00100598 | 0.000529556 | 0.440856 | 0.00153554 |
-| 11,250 | train | 0.000994464 | 0.000420516 | 0.443913 | 0.00141498 |
-| 11,500 | train | 0.000980495 | 0.000373817 | 0.448493 | 0.00135431 |
-| 11,750 | train | 0.00096843 | 0.000338886 | 0.453531 | 0.00130732 |
-| 12,000 | train | 0.000955185 | 0.000313992 | 0.459346 | 0.00126918 |
-| 12,250 | train | 0.000938976 | 0.00029757 | 0.463152 | 0.00123655 |
-| 12,500 | train | 0.000923241 | 0.00029515 | 0.466564 | 0.00121839 |
-| 12,750 | train | 0.000910699 | 0.00029568 | 0.469939 | 0.00120638 |
-| 13,000 | train | 0.000901795 | 0.000289766 | 0.472676 | 0.00119156 |
-| 13,250 | train | 0.000895763 | 0.000289525 | 0.475419 | 0.00118529 |
-| 13,500 | train | 0.000890323 | 0.000296688 | 0.476742 | 0.00118701 |
-| 13,750 | train | 0.000883409 | 0.000321699 | 0.478695 | 0.00120511 |
-| 14,000 | train | 0.000875354 | 0.000351263 | 0.479347 | 0.00122662 |
-| 14,250 | train | 0.000868868 | 0.000385058 | 0.479916 | 0.00125393 |
-| 14,500 | train | 0.000864959 | 0.000425899 | 0.479851 | 0.00129086 |
-| 14,750 | train | 0.000864657 | 0.0004665 | 0.478648 | 0.00133116 |
-| 15,000 | train | 0.000866869 | 0.000491292 | 0.478646 | 0.00135816 |
-| 15,000 | validation | 0.000868212 | 0.000531857 | 0.456224 | 0.00140007 |
-| 15,250 | train | 0.000870422 | 0.000489456 | 0.476793 | 0.00135988 |
-| 15,500 | train | 0.000874222 | 0.000472985 | 0.475759 | 0.00134721 |
-| 15,625 | train | 0.000876855 | 0.000457783 | 0.474811 | 0.00133464 |
-| 15,625 | validation | 0.000877582 | 0.000495893 | 0.448396 | 0.00137348 |
-
-Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.
-
----
-
-## Limitations
-
-- **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
-  will improve with more training.
-- **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
-  previously captured long-range token correlations. A checkpoint from ~step 38k is expected
-  to have better aggregate benchmark scores.
-- **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
-  linear branch. Will be corrected in the next training run.
-- **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
-  base model.
-
----
-
-## References
-
-All papers whose techniques are integrated into NeoLLM's architecture,
-training objective, or training stack:
-
-| Area | Technique | Paper title | Reference |
-|---|---|---|---|
-| Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
-| Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
-| Embeddings / residual modulation | JTok / JTok-M | JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
-| Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
-| Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
-| Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
-| Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
-| Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
-| Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
-| Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
-| Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
-| Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
-| Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
-| Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
-| Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
-| Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
-| Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
-| Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
-| Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
-| Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
-| Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
-| Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
-| Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
-| Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
-| Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
-| Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
-| Residual / normalization | Linear Depth Memory | Depth Memory | [arXiv:2610.18809](https://arxiv.org/abs/2610.18809) |
-| Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
-| Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
-| Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
-| Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
-| Optimizer / training | AdEMAMix | The AdEMAMix Optimizer: Better, Faster, Older | [arXiv:2409.03137](https://arxiv.org/abs/2409.03137) |
-| Optimizer / training | DeltaMomentum | DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule | [arXiv:2608.19491](https://arxiv.org/abs/2608.19491) |
-| Optimizer / training | BF16 + SR | Stochastic Rounding for LLM Training: Theory and Practice | [arXiv:2502.20566](https://arxiv.org/abs/2502.20566) |
-| Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
-| Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
-| Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
-| Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |
-
----
-
-## Citation
-
-```bibtex
-@misc{neollm2026,
-  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
-  author = {KitsuVp},
-  year   = {2026},
-  url    = {https://huggingface.co/KitsuVp/NeoLLM}
-}
-```
-
----
-
-## Author
-
-[@Kyokopom](https://x.com/Kyokopom) on X
-
----
-
-## License
+# NeoLLM
 
-Apache 2.0
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 4.7779
+- Ce: 3.8108
+- Valid Targets: 2083162.0
+- Ppl: 45.1865
+- Mile Loss: 4.3798
+- Mile Ppl: 79.8214
+- Top1 Accuracy: 0.2906
+- Top3 Accuracy: 0.4693
+- Topk Valid Targets: 2083162.0
+- Copy Gate Mean: 0.2952
+- Copy Sink Mean: 0.6072
+- Copy Effective Gate Mean: 0.1513
+- Copy Target Responsibility Mean: 0.1461
+- Copy Target In Context Fraction: 0.4607
+- Copy Valid Targets: 2083162.0
+- Ntp Loss: 4.3813
+- Ntp Ce Unweighted: 3.8113
+- Mile Reweighting Delta: 0.5689
+- Mu Loss: 0.0011
+- Tweo Loss: 0.0420
+- Nitp Loss: 0.3962
+- Total Model Loss: 4.7779
+- Tweo Weighted Loss: 0.0004
+- Nitp Weighted Loss: 0.3962
+- Auxiliary Loss: 0.3966
+- Regularization Loss: 0.0015
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 64
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step  | Validation Loss | Ce     | Valid Targets | Ppl      | Mile Loss | Mile Ppl | Top1 Accuracy | Top3 Accuracy | Topk Valid Targets | Copy Gate Mean | Copy Sink Mean | Copy Effective Gate Mean | Copy Target Responsibility Mean | Copy Target In Context Fraction | Copy Valid Targets | Ntp Loss | Ntp Ce Unweighted | Mile Reweighting Delta | Mu Loss | Tweo Loss | Nitp Loss | Total Model Loss | Tweo Weighted Loss | Nitp Weighted Loss | Auxiliary Loss | Regularization Loss |
+|:-------------:|:-----:|:-----:|:---------------:|:------:|:-------------:|:--------:|:---------:|:--------:|:-------------:|:-------------:|:------------------:|:--------------:|:--------------:|:------------------------:|:-------------------------------:|:-------------------------------:|:------------------:|:--------:|:-----------------:|:----------------------:|:-------:|:---------:|:---------:|:----------------:|:------------------:|:------------------:|:--------------:|:-------------------:|
+| 6.8795        | 0.32  | 5000  | 6.9756          | 6.3471 | 2083162.0     | 570.8485 | 6.4034    | 603.8834 | 0.0797        | 0.1682        | 2083162.0          | 0.2414         | 0.3041         | 0.1671                   | 0.1859                          | 0.4607                          | 2083162.0          | 6.4039   | 6.3472            | 0.0563                 | 0.0004  | 0.4454    | 0.5673    | 6.9756           | 0.0045             | 0.5673             | 0.5717         | 0.0048              |
+| 6.6857        | 0.64  | 10000 | 6.8105          | 6.1533 | 2083162.0     | 470.2733 | 6.2220    | 503.6890 | 0.0894        | 0.1861        | 2083162.0          | 0.3464         | 0.3755         | 0.2194                   | 0.1968                          | 0.4607                          | 2083162.0          | 6.2233   | 6.1535            | 0.0687                 | 0.0011  | 0.1092    | 0.5861    | 6.8105           | 0.0011             | 0.5861             | 0.5872         | 0.0022              |
+| 5.2017        | 0.96  | 15000 | 4.8294          | 3.8717 | 2083162.0     | 48.0239  | 4.4302    | 83.9504  | 0.2838        | 0.4616        | 2083162.0          | 0.2732         | 0.5657         | 0.1539                   | 0.1480                          | 0.4607                          | 2083162.0          | 4.4317   | 3.8722            | 0.5585                 | 0.0010  | 0.0424    | 0.3972    | 4.8294           | 0.0004             | 0.3972             | 0.3977         | 0.0015              |
+| 5.1573        | 1.0   | 15625 | 4.7779          | 3.8108 | 2083162.0     | 45.1865  | 4.3798    | 79.8214  | 0.2906        | 0.4693        | 2083162.0          | 0.2952         | 0.6072         | 0.1513                   | 0.1461                          | 0.4607                          | 2083162.0          | 4.3813   | 3.8113            | 0.5689                 | 0.0011  | 0.0420    | 0.3962    | 4.7779           | 0.0004             | 0.3962             | 0.3966         | 0.0015              |
+
+
+### Framework versions
+
+- Transformers 5.22.1
+- Pytorch 2.14.1+cu130
+- Datasets 4.8.5
+- Tokenizers 0.23.3

config.json

@@ -13,7 +13,7 @@
   },
   "bos_token_id": 1,
   "cce_loss_impl": "cce_kahan_full_c",
-  "copy_paste_lexical_keys": false,
+  "copy_paste_lexical_keys": true,
   "dagformer": {
     "input_source": "tokens",
     "local_dim": 128,
@@ -66,11 +66,11 @@
   "mea_component_key_value_heads": 4,
   "mea_groupnorm_eps": 1e-06,
   "meap_exclude_last": true,
-  "meap_exclude_mask_from_copy": false,
+  "meap_exclude_mask_from_copy": true,
   "meap_implementation": "triton",
   "meap_mask_ratio": 0.15,
   "meap_mask_token_id": 16,
-  "meap_protect_structural_tokens": false,
+  "meap_protect_structural_tokens": true,
   "meap_seed": 42,
   "mile_loss_gamma": 1.0,
   "model_type": "neollm",
@@ -132,7 +132,7 @@
   "use_affine_scaled_attention": false,
   "use_attn_res": false,
   "use_cache": false,
-  "use_copy_paste": false,
+  "use_copy_paste": true,
   "use_dagformer": false,
   "use_directional_routing": false,
   "use_embedding_input_norm": true,

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b3f44ecbe7b3eba6e2755602ec7e30f1320657635d8e3c92727fc33c3912d145
-size 169184776
+oid sha256:18b4379a1bb2232108523c44bbe3fd1e0d0364007b5b0add476c8fc2fc372dd6
+size 170301520

optimizer_precision.json

@@ -8,9 +8,9 @@
   "delta_enabled": false,
   "delta_eta": 0.37,
   "fp32_master_active": true,
-  "fp32_master_elements": 84567992,
+  "fp32_master_elements": 85126072,
   "fp32_master_groups": 6,
-  "fp32_master_tensors": 374,
+  "fp32_master_tensors": 380,
   "fp32_passthrough_elements": 0,
   "mode": "fp32_master",
   "mxfp8_active": false,
@@ -19,12 +19,12 @@
   "mxfp8_recipe": null,
   "native_precision_elements": 0,
   "optimizer_variant": "AdEMAMix",
-  "ordinary_linear_layers": 124,
+  "ordinary_linear_layers": 129,
   "parameter_elements_by_dtype": {
-    "bfloat16": 84565816,
+    "bfloat16": 85123896,
     "float32": 2176
   },
-  "parameter_storage_bytes": 169140336,
+  "parameter_storage_bytes": 170256496,
   "runtime_precision_mode": "BF16_FP32_MASTER",
   "stochastic_round_seed": 0
 }

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0c4d44d1cac39f073e0225075f5624d656dbda032765650b44697ac0b291f721
+oid sha256:2fad754c9d42e63a29da766ca7180104fb456d34656654dfd3e5995ee0e698a0
 size 5329