Model save

README.md

@@ -1,525 +1,78 @@
 ---
-language: en
-license: apache-2.0
+library_name: transformers
 tags:
-- causal-lm
-- research
-- bf16
-- attention
-- normalization
-- neollm
-datasets:
-- HuggingFaceFW/fineweb-edu
+- generated_from_trainer
+model-index:
+- name: NeoLLM
+  results: []
 ---
 
-# NeoLLM
-
-NeoLLM is a **84.57 M parameter** decoder-only language model trained from scratch on
-[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) with
-**BF16** compute, completing training in approximately
-**~1h 00m** on **NVIDIA RTX PRO 6000 Blackwell Server Edition**.
-It integrates a collection of recently published attention and normalization techniques
-into a single architecture, with the goal of studying how they interact during
-pretraining. The model is actively being developed and the current checkpoint represents
-an intermediate training state.
-
-> **Author / contact:** [@Kyokopom](https://x.com/Kyokopom) on X
-> **Repository:** [KitsuVp/NeoLLM](https://huggingface.co/KitsuVp/NeoLLM)
-
----
-
-## Architecture
-
-NeoLLM is a decoder-only transformer with the following configuration:
-
-| Parameter | Value |
-|---|---|
-| Hidden size | 512 |
-| Layers | 12 |
-| Attention heads | 8 |
-| KV heads (GQA) | 4 |
-| Head dim | 64 |
-| Intermediate size | 1536 |
-| Vocabulary | LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens) |
-| Context length | 512 tokens |
-| Leviathan | enabled=True; d_seed=128, modes=8, knots=16 |
-| JTok / JTok-M | enabled=False / False; Torch continuous surfaces, experts=5, top_k=2 |
-| DAGFormer | enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'} |
-
-### Parameter breakdown
-
-| Parameter bucket | Count |
-|---|---|
-| **Total parameters** | 84.57M (84,567,992) |
-| **Embedding parameters** (tied) | 32.97M (32,973,824) |
-| **Non-embedding parameters** | 51.59M (51,594,168) |
-| **Effective trainable parameters** | 84.57M (84,567,992) |
-
-> Weight tying is **enabled**: the input embedding matrix and the language-model head
-> share the same parameters, so the effective trainable budget is
-> `total − embed = 51.59M`.
-
-### Integrated techniques
-
-NeoLLM combines architecture modules, optional auxiliary objectives, and
-training-time optimizer/stability components from the following papers.
-
-**Embedding and token representation**
-
-- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Adds
-  per-row and per-column learnable scalar parameters to selected matrix layers and, when
-  enabled, embeddings.
-- **Leviathan** ([arXiv:2601.22040](https://arxiv.org/abs/2601.22040)) — Optional
-  continuous token embedding generator that can replace the discrete input lookup table.
-- **Leviathan-JTok / JTok-M** ([arXiv:2602.00800](https://arxiv.org/abs/2602.00800)) —
-  Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok
-  applies one normalized surface as a multiplicative MLP gate; JTok-M routes a
-  Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled
-  residual. Runtime ``jtok_kernel_backend`` selects the PyTorch reference or the
-  CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
-- **KHRONOS** ([arXiv:2505.13315](https://arxiv.org/abs/2505.13315)) — Kernel/basis
-  reference used by the Leviathan continuous token generator implementation.
-- **Spelling Bee Embeddings** ([arXiv:2601.18030](https://arxiv.org/abs/2601.18030)) —
-  Augments token embeddings with character-level spelling information.
-- **Token Embedding Manifold analysis** ([arXiv:2504.01002](https://arxiv.org/abs/2504.01002)) —
-  Reference motivation for treating token embeddings as structured objects rather than
-  unconstrained lookup rows.
-
-**Attention, positions, and output projection**
-
-- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks.
-  A portion of the projection channels are dedicated to periodic cosine/sine features.
-- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head
-  Attention. Adds small learnable interaction matrices between attention heads for K and V.
-- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Applies a learned
-  lower-triangular preconditioner to V before attention, decorrelating value representations
-  across positions.
-- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Adds
-  two learnable per-head scalars (α and β) to the softmax weights:
-  `[α·softmax(QKᵀ) + β]·V`.
-- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention.
-  After computing attention, removes the component of the output aligned with the token's
-  own value vector.
-- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Each head
-  learns K=4 directions in the output space; a learned router suppresses the attention output
-  along each direction per input.
-- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — A sigmoid gate
-  is applied to the attention output before the output projection, introducing non-linearity
-  and preventing attention sinks.
-- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Modifies Q
-  and K by subtracting a fraction of the previous position's Q and K values (causal
-  first-difference).
-- **Interleaved Head Attention / IHA** ([arXiv:2602.21371](https://arxiv.org/abs/2602.21371)) —
-  Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns
-  per original head.
-- **REPO** ([arXiv:2512.14391](https://arxiv.org/abs/2512.14391)) — Context re-positioning
-  module that learns contextual position coordinates above a configurable start layer.
-- **GRAPE** ([arXiv:2512.07805](https://arxiv.org/abs/2512.07805)) — Group representational
-  position encoding used by the REPO-GRAPE positional path.
-- **GOAT priors** ([arXiv:2601.15380](https://arxiv.org/abs/2601.15380)) — Optional
-  factorized attention log-prior channels inspired by trainable attention priors.
-- **Hadamard output projection** ([arXiv:2603.08343](https://arxiv.org/abs/2603.08343)) —
-  Replaces dense attention output projection with a structured Hadamard transform plus
-  lightweight scaling.
-
-**Normalization, residual flow, and MLP**
-
-- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Applied to Q and K
-  projections. Dynamically rescales normalization from the input's own statistics.
-- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Each
-  layer's output is scaled by 1/√ℓ where ℓ is the layer index.
-- **GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049)) — Gradient-Preserving
-  Activation Scaling for residual junctions.
-- **PolyNorm** ([arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Replaces the standard
-  MLP activation with normalized linear, quadratic, and cubic branches.
-- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order
-  geometry-inspired normalization strategy applied inside MLP projections.
-- **StackMemory / STACKTRANS** ([NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh)) —
-  Optional differentiable hidden-state stack between decoder layers.
-- **Attention Residuals / AttnRes** ([arXiv:2603.15031](https://arxiv.org/abs/2603.15031)) —
-  Optional learned depth-wise aggregation over previous layer outputs or block summaries.
-- **LAUREL** ([arXiv:2411.07501](https://arxiv.org/abs/2411.07501)) — Optional learned
-  augmented residual layer with residual-weight and low-rank variants.
-
-**Training objectives and training-time regularizers**
-
-- **Cut Cross Entropy** ([Apple repository](https://github.com/apple/ml-cross-entropy)) —
-  Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary
-  logits tensor. NeoLLM remains compatible with the upstream package when the extensions
-  below are disabled.
-- **MiLe Loss** ([arXiv:2310.19531](https://arxiv.org/abs/2310.19531)) — Optional detached,
-  mean-normalized predictive-entropy weighting of token losses, implemented inside the
-  extended CCE path.
-- **Output Embedding Centering / mu-loss**
-  ([arXiv:2601.02031](https://arxiv.org/abs/2601.02031)) — Optional
-  `lambda * ||mean(output_embeddings)||^2` regularizer for output-logit stability.
-- **MEAP** ([arXiv:2502.07490](https://arxiv.org/abs/2502.07490)) — Optional training-only
-  input corruption that masks a fixed fraction of eligible tokens while preserving clean
-  next-token labels, causal attention, and the inference path.
-- **TWEO** ([arXiv:2511.23225](https://arxiv.org/abs/2511.23225)) — Optional
-  Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly
-  training.
-- **NITP** ([arXiv:2605.24956](https://arxiv.org/abs/2605.24956)) — Optional Next Implicit
-  Token Prediction auxiliary objective using shallow-layer implicit token targets and a
-  cosine loss.
-- **NextLat** ([arXiv:2511.05963](https://arxiv.org/abs/2511.05963)) — Optional next-latent
-  prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.
-
-### Optional extended-CCE configuration
-
-| Feature | Enabled | Value |
-|---|---:|---:|
-| MiLe Loss | True | gamma=1.0 |
-| mu-loss | True | lambda=0.0001 |
-| MEAP | True | ratio=0.15 |
-
-MiLe, mu-loss, and MEAP require the extended
-[`Kitsunp/ml-cross-entropy`](https://github.com/Kitsunp/ml-cross-entropy) package only when
-their corresponding flags are enabled. With all flags disabled, NeoLLM calls upstream CCE
-without extension-specific arguments. When any extension is active, CCE reports three compact
-scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty.
-Their sum reconstructs `ntp_loss` exactly. MEAP reports its eligible and selected counts from
-the masking kernel; the trainer logs the selected count and exact fraction. No diagnostic path
-materializes full-vocabulary logits or a token mask outside the kernels.
-
-**Optimizer and training stability**
-
-- **Adam-family optimizer mode** — Active variant: **AdEMAMix**.
-  **AdEMAMix** ([arXiv:2409.03137](https://arxiv.org/abs/2409.03137)) contributes the optional
-  beta3 slow-memory branch when `ADEMAMIX=True`. **DeltaMomentum**
-  ([arXiv:2608.19491](https://arxiv.org/abs/2608.19491)) optionally replaces only the fast beta1
-  first-moment update on hidden Linear weights when `DELTA=True`; beta2 remains Adam's
-  ordinary squared-gradient EMA in every mode.
-- **BF16 stochastic rounding** ([arXiv:2502.20566](https://arxiv.org/abs/2502.20566)) —
-  Optional master-free BF16 model update with stochastic rather than nearest rounding; the same
-  write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
-- **Cautious Weight Decay** ([arXiv:2510.12402](https://arxiv.org/abs/2510.12402)) —
-  Sign-selective weight decay variant used by the custom optimizer logic.
-- **Correction of Decoupled Weight Decay** ([arXiv:2512.08217](https://arxiv.org/abs/2512.08217)) —
-  Adapts decoupled weight decay during learning-rate decay.
-- **AdamHD** ([arXiv:2511.14721](https://arxiv.org/abs/2511.14721)) —
-  Decoupled Huber decay regularization reference used by the optimizer.
-- **Gradient conditioning** — Active; repository per-parameter tensor norms; bias correction=True; no stabilizer warmup; EMA state stored in optimizer checkpoints.
-  References: [CAST](https://arxiv.org/abs/2610.19761),
-  [GradientStabilizer](https://arxiv.org/abs/2502.17055).
----
-
-## Training
-
-| Setting | Value |
-|---|---|
-| Dataset | FineWeb-Edu (sample-10BT) |
-| Tokens seen | ~0.51B (15,625 steps × batch 64 × length 512) |
-| Precision | BF16 model state (bfloat16: 84,565,816 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled |
-| Optimizer | AdEMAMix |
-| Gradient transform | Active; repository per-parameter tensor norms; bias correction=True; no stabilizer warmup; EMA state stored in optimizer checkpoints |
-| Learning rate | 1e-03 with 4,000-step linear warmup |
-| Weight decay | coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0 |
-| Training time | ~1h 00m |
-| Hardware | NVIDIA RTX PRO 6000 Blackwell Server Edition |
-
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
 
-### Training curve — total objective
-
-Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.
-
-| Step | Train total | Val total | Train without MiLe | Val without MiLe |
-|---|---|---|---|---|
-| 1 | 12.1628 | — | 12.1628 | — |
-| 250 | 7.60646 | — | 7.43483 | — |
-| 500 | 7.11046 | — | 6.83781 | — |
-| 750 | 6.78445 | — | 6.49071 | — |
-| 1,000 | 6.56215 | — | 6.25472 | — |
-| 1,250 | 6.41043 | — | 6.0883 | — |
-| 1,500 | 6.3068 | — | 5.97184 | — |
-| 1,750 | 6.21515 | — | 5.86749 | — |
-| 2,000 | 6.14136 | — | 5.77925 | — |
-| 2,250 | 6.08889 | — | 5.70973 | — |
-| 2,500 | 6.03413 | — | 5.64105 | — |
-| 2,750 | 5.98894 | — | 5.58622 | — |
-| 3,000 | 5.95055 | — | 5.54003 | — |
-| 3,250 | 5.91021 | — | 5.48942 | — |
-| 3,500 | 5.87788 | — | 5.4486 | — |
-| 3,750 | 5.83239 | — | 5.40089 | — |
-| 4,000 | 5.79932 | — | 5.36159 | — |
-| 4,250 | 5.76789 | — | 5.32479 | — |
-| 4,500 | 5.7337 | — | 5.28573 | — |
-| 4,750 | 5.69838 | — | 5.24759 | — |
-| 5,000 | 5.65781 | 5.23543 | 5.2039 | 4.67604 |
-| 5,250 | 5.63635 | — | 5.17947 | — |
-| 5,500 | 5.61476 | — | 5.15386 | — |
-| 5,750 | 5.5869 | — | 5.1214 | — |
-| 6,000 | 5.56535 | — | 5.09997 | — |
-| 6,250 | 5.54592 | — | 5.07789 | — |
-| 6,500 | 5.52431 | — | 5.05509 | — |
-| 6,750 | 5.51545 | — | 5.04329 | — |
-| 7,000 | 5.4953 | — | 5.02273 | — |
-| 7,250 | 5.48169 | — | 5.00525 | — |
-| 7,500 | 5.47765 | — | 4.99956 | — |
-| 7,750 | 5.45981 | — | 4.98085 | — |
-| 8,000 | 5.44806 | — | 4.9653 | — |
-| 8,250 | 5.43501 | — | 4.95204 | — |
-| 8,500 | 5.428 | — | 4.9437 | — |
-| 8,750 | 5.42458 | — | 4.93817 | — |
-| 9,000 | 5.41423 | — | 4.9277 | — |
-| 9,250 | 5.39804 | — | 4.91027 | — |
-| 9,500 | 5.3914 | — | 4.90269 | — |
-| 9,750 | 5.38942 | — | 4.89884 | — |
-| 10,000 | 5.37553 | 4.91535 | 4.8827 | 4.32289 |
-| 10,250 | 5.37482 | — | 4.88179 | — |
-| 10,500 | 5.3682 | — | 4.87639 | — |
-| 10,750 | 5.36149 | — | 4.86526 | — |
-| 11,000 | 5.35708 | — | 4.86357 | — |
-| 11,250 | 5.35302 | — | 4.85766 | — |
-| 11,500 | 5.35091 | — | 4.85158 | — |
-| 11,750 | 5.34317 | — | 4.84358 | — |
-| 12,000 | 5.34214 | — | 4.84396 | — |
-| 12,250 | 5.33081 | — | 4.83187 | — |
-| 12,500 | 5.31501 | — | 4.81569 | — |
-| 12,750 | 5.31292 | — | 4.80983 | — |
-| 13,000 | 5.29299 | — | 4.78959 | — |
-| 13,250 | 5.28632 | — | 4.77953 | — |
-| 13,500 | 5.27245 | — | 4.76257 | — |
-| 13,750 | 5.27055 | — | 4.76017 | — |
-| 14,000 | 5.2511 | — | 4.74102 | — |
-| 14,250 | 5.23908 | — | 4.72851 | — |
-| 14,500 | 5.22941 | — | 4.71561 | — |
-| 14,750 | 5.21514 | — | 4.69735 | — |
-| 15,000 | 5.21468 | 4.74511 | 4.69776 | 4.12708 |
-| 15,250 | 5.19578 | — | 4.67791 | — |
-| 15,500 | 5.18821 | — | 4.66907 | — |
-| 15,625 | 5.17338 | 4.72462 | 4.65385 | 4.09783 |
-
-### Cross-entropy with and without MiLe
-
-CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. `ntp_loss` includes mu when enabled.
-
-| Step | Train CE without MiLe | Val CE without MiLe | Train CE with MiLe | Val CE with MiLe |
-|---|---|---|---|---|
-| 1 | 11.1858 | — | 11.1858 | — |
-| 250 | 7.01026 | — | 7.18189 | — |
-| 500 | 6.21718 | — | 6.48982 | — |
-| 750 | 5.84786 | — | 6.1416 | — |
-| 1,000 | 5.60969 | — | 5.91713 | — |
-| 1,250 | 5.44332 | — | 5.76544 | — |
-| 1,500 | 5.32873 | — | 5.66369 | — |
-| 1,750 | 5.2266 | — | 5.57426 | — |
-| 2,000 | 5.14025 | — | 5.50236 | — |
-| 2,250 | 5.07082 | — | 5.44999 | — |
-| 2,500 | 5.00553 | — | 5.39861 | — |
-| 2,750 | 4.95387 | — | 5.35659 | — |
-| 3,000 | 4.91284 | — | 5.32336 | — |
-| 3,250 | 4.87043 | — | 5.29122 | — |
-| 3,500 | 4.8399 | — | 5.26919 | — |
-| 3,750 | 4.80168 | — | 5.23318 | — |
-| 4,000 | 4.77362 | — | 5.21135 | — |
-| 4,250 | 4.74767 | — | 5.19077 | — |
-| 4,500 | 4.71803 | — | 5.166 | — |
-| 4,750 | 4.68754 | — | 5.13833 | — |
-| 5,000 | 4.65231 | 4.20722 | 5.10622 | 4.76661 |
-| 5,250 | 4.63333 | — | 5.09021 | — |
-| 5,500 | 4.61365 | — | 5.07455 | — |
-| 5,750 | 4.58672 | — | 5.05222 | — |
-| 6,000 | 4.5698 | — | 5.03518 | — |
-| 6,250 | 4.55087 | — | 5.0189 | — |
-| 6,500 | 4.53138 | — | 5.00059 | — |
-| 6,750 | 4.52143 | — | 4.99358 | — |
-| 7,000 | 4.50296 | — | 4.97553 | — |
-| 7,250 | 4.4874 | — | 4.96384 | — |
-| 7,500 | 4.48267 | — | 4.96075 | — |
-| 7,750 | 4.46623 | — | 4.94519 | — |
-| 8,000 | 4.45225 | — | 4.93501 | — |
-| 8,250 | 4.44049 | — | 4.92346 | — |
-| 8,500 | 4.43286 | — | 4.91715 | — |
-| 8,750 | 4.42866 | — | 4.91507 | — |
-| 9,000 | 4.41742 | — | 4.90394 | — |
-| 9,250 | 4.40077 | — | 4.88854 | — |
-| 9,500 | 4.39266 | — | 4.88136 | — |
-| 9,750 | 4.38946 | — | 4.88003 | — |
-| 10,000 | 4.37345 | 3.91333 | 4.86628 | 4.50578 |
-| 10,250 | 4.37077 | — | 4.8638 | — |
-| 10,500 | 4.36409 | — | 4.8559 | — |
-| 10,750 | 4.35384 | — | 4.85006 | — |
-| 11,000 | 4.35064 | — | 4.84415 | — |
-| 11,250 | 4.34393 | — | 4.83929 | — |
-| 11,500 | 4.33768 | — | 4.83701 | — |
-| 11,750 | 4.3293 | — | 4.82889 | — |
-| 12,000 | 4.32781 | — | 4.826 | — |
-| 12,250 | 4.31572 | — | 4.81466 | — |
-| 12,500 | 4.2998 | — | 4.79911 | — |
-| 12,750 | 4.29454 | — | 4.79763 | — |
-| 13,000 | 4.27499 | — | 4.77838 | — |
-| 13,250 | 4.26461 | — | 4.77139 | — |
-| 13,500 | 4.24787 | — | 4.75775 | — |
-| 13,750 | 4.24555 | — | 4.75592 | — |
-| 14,000 | 4.22651 | — | 4.7366 | — |
-| 14,250 | 4.21505 | — | 4.72561 | — |
-| 14,500 | 4.20286 | — | 4.71666 | — |
-| 14,750 | 4.18628 | — | 4.70407 | — |
-| 15,000 | 4.18707 | 3.70576 | 4.70399 | 4.32378 |
-| 15,250 | 4.16733 | — | 4.68519 | — |
-| 15,500 | 4.15842 | — | 4.67756 | — |
-| 15,625 | 4.14494 | 3.67859 | 4.66447 | 4.30539 |
-
-### Observed loss components — weighted contributions
-
-Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. `regularization_loss` is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.
-
-| Step | Phase | mu | TWeo | NITP | Reg. subtotal |
-|---|---|---|---|---|---|
-| 1 | train | 3.18928e-10 | 1.73896e-06 | 0.976953 | 1.73928e-06 |
-| 250 | train | 6.84891e-06 | 0.000676235 | 0.42389 | 0.000683084 |
-| 500 | train | 1.96041e-05 | 6.16063e-05 | 0.620559 | 8.12103e-05 |
-| 750 | train | 3.3879e-05 | 4.86311e-05 | 0.642769 | 8.25101e-05 |
-| 1,000 | train | 4.82082e-05 | 6.36086e-05 | 0.644915 | 0.000111817 |
-| 1,250 | train | 6.0768e-05 | 0.00010342 | 0.644818 | 0.000164188 |
-| 1,500 | train | 7.04269e-05 | 0.000192415 | 0.642843 | 0.000262841 |
-| 1,750 | train | 7.75483e-05 | 0.000342753 | 0.64047 | 0.000420301 |
-| 2,000 | train | 8.38168e-05 | 0.000578622 | 0.63834 | 0.000662439 |
-| 2,250 | train | 9.03597e-05 | 0.00085802 | 0.637955 | 0.00094838 |
-| 2,500 | train | 0.000100004 | 0.00115163 | 0.634271 | 0.00125163 |
-| 2,750 | train | 0.000113281 | 0.0014239 | 0.63081 | 0.00153718 |
-| 3,000 | train | 0.000121862 | 0.00165795 | 0.625408 | 0.00177981 |
-| 3,250 | train | 0.000132248 | 0.00182268 | 0.617035 | 0.00195493 |
-| 3,500 | train | 0.000148375 | 0.00187566 | 0.606676 | 0.00202403 |
-| 3,750 | train | 0.000165467 | 0.00187988 | 0.597164 | 0.00204534 |
-| 4,000 | train | 0.000180491 | 0.00183098 | 0.58596 | 0.00201147 |
-| 4,250 | train | 0.000198139 | 0.00178194 | 0.57514 | 0.00198007 |
-| 4,500 | train | 0.000220956 | 0.00169323 | 0.565785 | 0.00191419 |
-| 4,750 | train | 0.000241636 | 0.00160502 | 0.558204 | 0.00184665 |
-| 5,000 | train | 0.000259585 | 0.00144827 | 0.549893 | 0.00170785 |
-| 5,000 | validation | 0.000273011 | 0.00140449 | 0.467142 | 0.0016775 |
-| 5,250 | train | 0.000285291 | 0.00137516 | 0.544479 | 0.00166045 |
-| 5,500 | train | 0.000303946 | 0.00127061 | 0.538641 | 0.00157455 |
-| 5,750 | train | 0.000329422 | 0.00121415 | 0.533132 | 0.00154357 |
-| 6,000 | train | 0.000353627 | 0.00118309 | 0.52863 | 0.00153672 |
-| 6,250 | train | 0.000376571 | 0.00113181 | 0.525507 | 0.00150838 |
-| 6,500 | train | 0.00040026 | 0.00103695 | 0.522277 | 0.00143721 |
-| 6,750 | train | 0.000426527 | 0.000991961 | 0.520443 | 0.00141849 |
-| 7,000 | train | 0.000443904 | 0.000980956 | 0.518338 | 0.00142486 |
-| 7,250 | train | 0.000466515 | 0.000963704 | 0.516422 | 0.00143022 |
-| 7,500 | train | 0.000493662 | 0.00095226 | 0.515451 | 0.00144592 |
-| 7,750 | train | 0.000522439 | 0.000951694 | 0.513145 | 0.00147413 |
-| 8,000 | train | 0.000548451 | 0.000962954 | 0.511537 | 0.00151141 |
-| 8,250 | train | 0.000573295 | 0.000961034 | 0.510014 | 0.00153433 |
-| 8,500 | train | 0.000590981 | 0.000977169 | 0.509274 | 0.00156815 |
-| 8,750 | train | 0.000606377 | 0.00095807 | 0.507948 | 0.00156445 |
-| 9,000 | train | 0.000622443 | 0.000957854 | 0.508708 | 0.0015803 |
-| 9,250 | train | 0.000648756 | 0.000963291 | 0.50789 | 0.00161205 |
-| 9,500 | train | 0.000669239 | 0.000969349 | 0.508395 | 0.00163859 |
-| 9,750 | train | 0.000691675 | 0.000935606 | 0.507755 | 0.00162728 |
-| 10,000 | train | 0.000712 | 0.000931068 | 0.507608 | 0.00164307 |
-| 10,000 | validation | 0.000721167 | 0.0009688 | 0.407873 | 0.00168997 |
-| 10,250 | train | 0.000736315 | 0.000914907 | 0.50937 | 0.00165122 |
-| 10,500 | train | 0.000777281 | 0.00088052 | 0.510645 | 0.0016578 |
-| 10,750 | train | 0.000806279 | 0.000859968 | 0.509759 | 0.00166625 |
-| 11,000 | train | 0.000826967 | 0.000838481 | 0.511263 | 0.00166545 |
-| 11,250 | train | 0.00083419 | 0.000837565 | 0.512058 | 0.00167176 |
-| 11,500 | train | 0.000834836 | 0.000822191 | 0.512248 | 0.00165703 |
-| 11,750 | train | 0.000863479 | 0.000848665 | 0.512566 | 0.00171214 |
-| 12,000 | train | 0.00089745 | 0.000850088 | 0.514401 | 0.00174754 |
-| 12,250 | train | 0.000905931 | 0.000829532 | 0.514414 | 0.00173546 |
-| 12,500 | train | 0.000906045 | 0.000873171 | 0.51412 | 0.00177922 |
-| 12,750 | train | 0.000912162 | 0.000857579 | 0.513524 | 0.00176974 |
-| 13,000 | train | 0.000919919 | 0.000819745 | 0.512863 | 0.00173966 |
-| 13,250 | train | 0.000935974 | 0.00078068 | 0.513211 | 0.00171665 |
-| 13,500 | train | 0.000945851 | 0.000747928 | 0.513005 | 0.00169378 |
-| 13,750 | train | 0.000948116 | 0.000699751 | 0.512978 | 0.00164787 |
-| 14,000 | train | 0.000957065 | 0.000670298 | 0.512876 | 0.00162736 |
-| 14,250 | train | 0.000960524 | 0.000637824 | 0.511865 | 0.00159835 |
-| 14,500 | train | 0.000959794 | 0.000615776 | 0.511177 | 0.00157557 |
-| 14,750 | train | 0.000965124 | 0.000594786 | 0.509515 | 0.00155991 |
-| 15,000 | train | 0.000968428 | 0.000552157 | 0.509166 | 0.00152059 |
-| 15,000 | validation | 0.000970282 | 0.000559313 | 0.419799 | 0.00152959 |
-| 15,250 | train | 0.000970819 | 0.000518389 | 0.509097 | 0.00148921 |
-| 15,500 | train | 0.000968686 | 0.000494455 | 0.509181 | 0.00146314 |
-| 15,625 | train | 0.000967579 | 0.00047286 | 0.507476 | 0.00144044 |
-| 15,625 | validation | 0.000967973 | 0.000520825 | 0.417745 | 0.0014888 |
-
-Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.
-
----
-
-## Limitations
-
-- **Token budget** — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks
-  will improve with more training.
-- **Gradient spike at step 40k** — Reorganized the attention pattern in layer 9 that
-  previously captured long-range token correlations. A checkpoint from ~step 38k is expected
-  to have better aggregate benchmark scores.
-- **PolyNorm exclusivity** — The quadratic branch has become partially redundant with the
-  linear branch. Will be corrected in the next training run.
-- **Base model only** — Not instruction-tuned or aligned; purely a next-token-prediction
-  base model.
-
----
-
-## References
-
-All papers whose techniques are integrated into NeoLLM's architecture,
-training objective, or training stack:
-
-| Area | Technique | Paper title | Reference |
-|---|---|---|---|
-| Embeddings | Learnable Multipliers | Freeing the Scale of Language Model Matrix Layers | [arXiv:2601.04890](https://arxiv.org/abs/2601.04890) |
-| Embeddings | Leviathan | A Separable Architecture for Continuous Token Representation in Language Models | [arXiv:2601.22040](https://arxiv.org/abs/2601.22040) |
-| Embeddings / residual modulation | JTok / JTok-M | JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation | [arXiv:2602.00800](https://arxiv.org/abs/2602.00800) |
-| Embeddings | KHRONOS | KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation | [arXiv:2505.13315](https://arxiv.org/abs/2505.13315) |
-| Embeddings | Spelling Bee | Spelling Bee Embeddings for Language Modeling | [arXiv:2601.18030](https://arxiv.org/abs/2601.18030) |
-| Embeddings | Token embedding analysis | Token Embeddings Violate the Manifold Hypothesis | [arXiv:2504.01002](https://arxiv.org/abs/2504.01002) |
-| Attention / positions | FAN | Fourier Analysis Networks | [arXiv:2502.21309](https://arxiv.org/abs/2502.21309) |
-| Attention / positions | MEA | Explicit Multi-head Attention for Inter-head Interaction in Large Language Models | [arXiv:2601.19611](https://arxiv.org/abs/2601.19611) |
-| Attention / positions | LUCID | Attention with Preconditioned Representations | [arXiv:2602.10410](https://arxiv.org/abs/2602.10410) |
-| Attention / positions | Affine-Scaled Attention | Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention | [arXiv:2602.23057](https://arxiv.org/abs/2602.23057) |
-| Attention / positions | XSA | Exclusive Self Attention | [arXiv:2603.09078](https://arxiv.org/abs/2603.09078) |
-| Attention / positions | Directional Routing | Directional Routing in Transformers | [arXiv:2603.14923](https://arxiv.org/abs/2603.14923) |
-| Attention / positions | Gated Attention | Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free | [arXiv:2505.06708](https://arxiv.org/abs/2505.06708) |
-| Attention / positions | Momentum Attention | Momentum Attention | [arXiv:2411.03884](https://arxiv.org/abs/2411.03884) |
-| Attention / positions | IHA | Interleaved Head Attention | [arXiv:2602.21371](https://arxiv.org/abs/2602.21371) |
-| Attention / positions | REPO | Language Models with Context Re-Positioning | [arXiv:2512.14391](https://arxiv.org/abs/2512.14391) |
-| Attention / positions | GRAPE | Group Representational Position Encoding | [arXiv:2512.07805](https://arxiv.org/abs/2512.07805) |
-| Attention / positions | GOAT priors | You Need Better Attention Priors | [arXiv:2601.15380](https://arxiv.org/abs/2601.15380) |
-| Attention / positions | Hadamard o_proj | Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers | [arXiv:2603.08343](https://arxiv.org/abs/2603.08343) |
-| Residual / normalization | SeeDNorm | Self-Rescaled Dynamic Normalization | [arXiv:2510.22777](https://arxiv.org/abs/2510.22777) |
-| Residual / normalization | LNS | The Curse of Depth in LLMs | [arXiv:2502.05795](https://arxiv.org/abs/2502.05795) |
-| Residual / normalization | GPAS | Gradient-Preserving Activation Scaling | [arXiv:2506.22049](https://arxiv.org/abs/2506.22049) |
-| Residual / normalization | PolyNorm | PolyNorm / PolyCom | [arXiv:2602.04902](https://arxiv.org/abs/2602.04902) |
-| Residual / normalization | SimpleGPT | SimpleGPT | [arXiv:2602.01212](https://arxiv.org/abs/2602.01212) |
-| Residual / normalization | StackMemory / STACKTRANS | Recursive Transformer: Boosting Reasoning Ability with State Stack | [NeurIPS 2025](https://openreview.net/forum?id=2bbDg587uh) |
-| Residual / normalization | Attention Residuals | Attention Residuals | [arXiv:2603.15031](https://arxiv.org/abs/2603.15031) |
-| Residual / normalization | LAUREL | LAUREL: Learned Augmented Residual Layer | [arXiv:2411.07501](https://arxiv.org/abs/2411.07501) |
-| Objectives | TWEO | Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies | [arXiv:2511.23225](https://arxiv.org/abs/2511.23225) |
-| Objectives | NITP | Next Implicit Token Prediction for LLM Pre-training | [arXiv:2605.24956](https://arxiv.org/abs/2605.24956) |
-| Objectives | NextLat | Next-Latent Prediction Transformers Learn Compact World Models | [arXiv:2511.05963](https://arxiv.org/abs/2511.05963) |
-| Optimizer / training | AdEMAMix | The AdEMAMix Optimizer: Better, Faster, Older | [arXiv:2409.03137](https://arxiv.org/abs/2409.03137) |
-| Optimizer / training | DeltaMomentum | DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule | [arXiv:2608.19491](https://arxiv.org/abs/2608.19491) |
-| Optimizer / training | BF16 + SR | Stochastic Rounding for LLM Training: Theory and Practice | [arXiv:2502.20566](https://arxiv.org/abs/2502.20566) |
-| Optimizer / training | CWD | Cautious Weight Decay | [arXiv:2510.12402](https://arxiv.org/abs/2510.12402) |
-| Optimizer / training | WD correction | Correction of Decoupled Weight Decay | [arXiv:2512.08217](https://arxiv.org/abs/2512.08217) |
-| Optimizer / training | AdamHD | AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training | [arXiv:2511.14721](https://arxiv.org/abs/2511.14721) |
-| Optimizer / training | GradientStabilizer | GradientStabilizer | [arXiv:2502.17055](https://arxiv.org/abs/2502.17055) |
-
----
-
-## Citation
-
-```bibtex
-@misc{neollm2026,
-  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
-  author = {KitsuVp},
-  year   = {2026},
-  url    = {https://huggingface.co/KitsuVp/NeoLLM}
-}
-```
-
----
-
-## Author
-
-[@Kyokopom](https://x.com/Kyokopom) on X
-
----
-
-## License
+# NeoLLM
 
-Apache 2.0
+This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 4.7393
+- Ce: 3.6698
+- Valid Targets: 2083162.0
+- Ppl: 39.2448
+- Mile Loss: 4.3028
+- Mile Ppl: 73.9062
+- Top1 Accuracy: 0.3349
+- Top3 Accuracy: 0.4953
+- Topk Valid Targets: 2083162.0
+- Ntp Loss: 4.3042
+- Ntp Ce Unweighted: 3.6703
+- Mile Reweighting Delta: 0.6329
+- Mu Loss: 0.0010
+- Tweo Loss: 0.0455
+- Nitp Loss: 0.4346
+- Total Model Loss: 4.7393
+- Tweo Weighted Loss: 0.0005
+- Nitp Weighted Loss: 0.4346
+- Auxiliary Loss: 0.4350
+- Regularization Loss: 0.0015
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 0.001
+- train_batch_size: 64
+- eval_batch_size: 64
+- seed: 42
+- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: linear
+- num_epochs: 1
+
+### Training results
+
+| Training Loss | Epoch | Step  | Validation Loss | Ce     | Valid Targets | Ppl     | Mile Loss | Mile Ppl | Top1 Accuracy | Top3 Accuracy | Topk Valid Targets | Ntp Loss | Ntp Ce Unweighted | Mile Reweighting Delta | Mu Loss | Tweo Loss | Nitp Loss | Total Model Loss | Tweo Weighted Loss | Nitp Weighted Loss | Auxiliary Loss | Regularization Loss |
+|:-------------:|:-----:|:-----:|:---------------:|:------:|:-------------:|:-------:|:---------:|:--------:|:-------------:|:-------------:|:------------------:|:--------:|:-----------------:|:----------------------:|:-------:|:---------:|:---------:|:----------------:|:------------------:|:------------------:|:--------------:|:-------------------:|
+| 5.6020        | 0.32  | 5000  | 5.2087          | 4.1859 | 2083162.0     | 65.7549 | 4.7563    | 116.3132 | 0.2851        | 0.4336        | 2083162.0          | 4.7569   | 4.1864            | 0.5703                 | 0.0002  | 0.1707    | 0.4501    | 5.2087           | 0.0017             | 0.4501             | 0.4518         | 0.0020              |
+| 5.3530        | 0.64  | 10000 | 4.9332          | 3.9013 | 2083162.0     | 49.4689 | 4.5039    | 90.3670  | 0.3120        | 0.4672        | 2083162.0          | 4.5050   | 3.9018            | 0.6025                 | 0.0007  | 0.0826    | 0.4274    | 4.9332           | 0.0008             | 0.4274             | 0.4282         | 0.0015              |
+| 5.1984        | 0.96  | 15000 | 4.7608          | 3.6975 | 2083162.0     | 40.3448 | 4.3240    | 75.4908  | 0.3318        | 0.4919        | 2083162.0          | 4.3255   | 3.6979            | 0.6265                 | 0.0010  | 0.0516    | 0.4348    | 4.7608           | 0.0005             | 0.4348             | 0.4353         | 0.0015              |
+| 5.1708        | 1.0   | 15625 | 4.7393          | 3.6698 | 2083162.0     | 39.2448 | 4.3028    | 73.9062  | 0.3349        | 0.4953        | 2083162.0          | 4.3042   | 3.6703            | 0.6329                 | 0.0010  | 0.0455    | 0.4346    | 4.7393           | 0.0005             | 0.4346             | 0.4350         | 0.0015              |
+
+
+### Framework versions
+
+- Transformers 5.20.2
+- Pytorch 2.14.1+cu130
+- Datasets 4.8.5
+- Tokenizers 0.23.3

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4910e940a8b4c99e5efeb5ac241d7c30d65f42dd964ddb11fb246e52ead76e3
+oid sha256:4fc07cc3b132ccb80a7f7a7456ad2de82ee8e3464929c0164a41dcf53529c284
 size 169184776

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:43965fb73dc2d925bceee6982d939c9058b6c1831af2c457ff5265b2b6d3ba41
+oid sha256:277269add51017ca892f475f645345b7b1b93b48b9a5a94a24da38f8c8f74192
 size 5329