Update README.md

README.md

@@ -5,61 +5,180 @@ tags:
 model-index:
 - name: NeoLLM
   results: []
+license: apache-2.0
+datasets:
+- HuggingFaceFW/fineweb-edu
+language:
+- en
 ---
 
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment. -->
-
 # NeoLLM
 
-This model is a fine-tuned version of [](https://huggingface.co/) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 2.3449
+NeoLLM is a 135M parameter language model trained from scratch on [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) in FP8. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying their combined behavior during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.
+
+---
+
+## Architecture
+
+NeoLLM is a decoder-only transformer with the following configuration:
+
+| Parameter | Value |
+|---|---|
+| Hidden size | 512 |
+| Layers | 12 |
+| Attention heads | 8 |
+| KV heads (GQA) | 2 |
+| Head dim | 64 |
+| Intermediate size | 1536 |
+| Parameters | ~135M (77M tied embeddings, ~58M effective) |
+| Vocabulary | Qwen3 tokenizer |
+| Context length | 512 |
+
+### Components
+
+Each layer integrates the following mechanisms simultaneously:
+
+**Normalization**
+- **SeeDNorm** ([arXiv:2510.22777](https://arxiv.org/abs/2510.22777)) — Self-rescaled dynamic normalization applied to Q and K projections
+- **PolyNorm / GPAS** ([arXiv:2506.22049](https://arxiv.org/abs/2506.22049), [arXiv:2602.04902](https://arxiv.org/abs/2602.04902)) — Polynomial composition activations with three branches (linear, quadratic, cubic) and gradient-preserving activation scaling on residual connections
+- **LayerNorm Scaling / LNS** ([arXiv:2502.05795](https://arxiv.org/abs/2502.05795)) — Per-layer scaling by 1/√ℓ to address the Curse of Depth
+
+**Attention**
+- **FAN** ([arXiv:2502.21309](https://arxiv.org/abs/2502.21309)) — Fourier Analysis Networks: periodic channels in input projections for improved periodicity modeling
+- **MEA** ([arXiv:2601.19611](https://arxiv.org/abs/2601.19611)) — Explicit Multi-head Attention with learnable inter-head interaction matrices for K and V
+- **LUCID** ([arXiv:2602.10410](https://arxiv.org/abs/2602.10410)) — Lower-triangular preconditioner on V for attention decorrelation
+- **Affine-Scaled Attention** ([arXiv:2602.23057](https://arxiv.org/abs/2602.23057)) — Learnable per-head α and β scaling of softmax weights: `[α·softmax(QKᵀ)+β]V`
+- **XSA** ([arXiv:2603.09078](https://arxiv.org/abs/2603.09078)) — Exclusive Self Attention: removes the auto-position component from attention output
+- **Directional Routing** ([arXiv:2603.14923](https://arxiv.org/abs/2603.14923)) — Learned suppression of attention output along K=4 directions per head
+- **Gated Attention** ([arXiv:2505.06708](https://arxiv.org/abs/2505.06708)) — Sigmoid gate applied to attention output before o_proj
+- **Momentum Attention** ([arXiv:2411.03884](https://arxiv.org/abs/2411.03884)) — Causal first-difference shear on Q and K
+
+**MLP**
+- **Learnable Multipliers** ([arXiv:2601.04890](https://arxiv.org/abs/2601.04890)) — Per-row and per-column learnable scale parameters on linear layers
+- **SimpleGPT** ([arXiv:2602.01212](https://arxiv.org/abs/2602.01212)) — Second-order geometry normalization applied inside MLP projections
+
+---
+
+## Training
+
+| Setting | Value |
+|---|---|
+| Dataset | FineWeb-Edu |
+| Tokens | ~1.5B (45k steps × batch 64 × length 512) |
+| Precision | FP8 (E4M3/E5M2) native |
+| Optimizer | AdamW fused (β₁=0.9, β₂=0.999, ε=1e-8) |
+| Learning rate | 6e-4 with linear warmup (10%) |
+| Weight decay | 0.1 |
+| Training time | ~6 hours |
+| Hardware | NVIDIA RTX 5090 |
+
+### Training curve
+
+| Step | Train Loss | Val Loss |
+|---|---|---|
+| 5,000 | 3.701 | 3.601 |
+| 10,000 | 3.286 | 3.169 |
+| 15,000 | 3.074 | 2.954 |
+| 20,000 | 2.935 | 2.808 |
+| 25,000 | 2.832 | 2.695 |
+| 30,000 | 2.747 | 2.605 |
+| 35,000 | 2.686 | 2.544 |
+| 40,000 | 2.658 | 2.505 |
+| 45,000 | 2.493 | 2.345 |
+
+
+---
+
+## Benchmarks
+
+Evaluated with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), 0-shot.
+
+| Task | Metric | Score |
+|---|---|---|
+| MMLU | acc | 23.02% |
+| BoolQ | acc | 54.43% |
+| ARC-Easy | acc_norm | 34.55% |
+| ARC-Challenge | acc_norm | 22.44% |
+| PIQA | acc_norm | 53.59% |
+| HellaSwag | acc_norm | 26.78% |
+| OpenBookQA | acc_norm | 29.00% |
+| Winogrande | acc | 46.96% |
+| SCIQ | acc | 58.30% |
+| CommonsenseQA | acc | 20.88% |
+| Lambada (OpenAI) | acc | 9.02% |
+| TruthfulQA MC2 | acc | 46.45% |
+| COPA | acc | 57.00% |
+| ANLI R1 | acc | 35.90% |
+| ANLI R2 | acc | 33.90% |
+| ANLI R3 | acc | 33.58% |
+| Wikitext | word_ppl | 40.41 |
+| **Average (acc tasks)** | | **36.61%** |
 
-## Model description
+---
+
+## Interpretability notes
+
+A set of custom interpretability tools was developed alongside training to analyze the internal dynamics of the model. The following observations correspond to the 45k step checkpoint.
 
-More information needed
+**Layer hierarchy.** The model develops a functional hierarchy across layers. Early layers (L0–L3) focus on local structure processing, with XSA auto-position removal active at frac > 1.5 throughout. Middle layers (L4–L6) capture periodic and sequential patterns — L5 explicitly attends to preceding elements in geometric sequences. Layers L7–L8 perform global integration with cross-head similarity above 0.87. Layers L9–L11 perform synthesis toward the vocabulary head, with L11 MLP contributing rho_mlp = 0.81 to the residual stream.
 
-## Intended uses & limitations
+**Affine-Scaled Attention.** Learned α values range from 0.23 to 0.54 across layers. Lower α appears in layers with high XSA activity, where affine defers to global integration via the β·V_cumsum term. ΔH (attention entropy increase from affine) is consistently positive across all layers and inputs.
 
-More information needed
+**Periodicity.** FAN R_FAN grows monotonically with depth in long sequences, reaching 0.18 in L11 at 130 tokens. This indicates that periodic channels are activated in proportion to available sequential structure in the input.
 
-## Training and evaluation data
+**Numerical condition.** MLP outlier risk (simulated via the colinearity mechanism described in TWEO, arXiv:2511.23225) is 1.96 mean (vs 500–5000 typical for standard transformers), reflecting near-orthogonal gate_proj and up_proj dominant directions. The only component with elevated condition numbers is o_proj (κ from 2,009 to 105,616), partially compensated by the learnable multipliers.
 
-More information needed
+---
+
+## Limitations
 
-## Training procedure
+- The model is trained for a single epoch on FineWeb-Edu and has not reached its estimated token optimum. Knowledge-intensive tasks (MMLU, factual retrieval) remain limited.
+- A gradient spike near step 40k degraded the long-range correlation structure in layer 9, affecting tasks that require integrating information across long distances (Lambada, SCIQ, BoolQ). An earlier checkpoint around step 38k is expected to have better aggregate benchmark performance.
+- PolyNorm exclusivity for the quadratic branch (delta_12) is negative across all layers at 45k steps, indicating that the exclusive_logits parameters did not converge to effective orthogonalization. This will be addressed in subsequent runs by including exclusive_logits in the checkpoint from the start of training.
+- The model has not been instruction-tuned or aligned in any way. It is a base language model intended for research.
 
-### Training hyperparameters
+---
 
-The following hyperparameters were used during training:
-- learning_rate: 0.0006
-- train_batch_size: 64
-- eval_batch_size: 64
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 0.1
-- num_epochs: 1
+## Intended use
 
-### Training results
+NeoLLM is released for research purposes. It is intended as a testbed for studying the combined behavior of recent attention and normalization techniques during pretraining at small scale. It is not intended for production deployment in its current state.
+
+---
+
+## References
+
+| Paper | arXiv |
+|---|---|
+| SeeDNorm: Self-Rescaled Dynamic Normalization | [2510.22777](https://arxiv.org/abs/2510.22777) |
+| Explicit Multi-head Attention (MEA) | [2601.19611](https://arxiv.org/abs/2601.19611) |
+| Learnable Multipliers | [2601.04890](https://arxiv.org/abs/2601.04890) |
+| Directional Routing in Transformers | [2603.14923](https://arxiv.org/abs/2603.14923) |
+| Exclusive Self Attention (XSA) | [2603.09078](https://arxiv.org/abs/2603.09078) |
+| Gated Attention for LLMs | [2505.06708](https://arxiv.org/abs/2505.06708) |
+| Affine-Scaled Attention | [2602.23057](https://arxiv.org/abs/2602.23057) |
+| The Curse of Depth in LLMs (LNS) | [2502.05795](https://arxiv.org/abs/2502.05795) |
+| LUCID: Attention with Preconditioned Representations | [2602.10410](https://arxiv.org/abs/2602.10410) |
+| FAN: Fourier Analysis Networks | [2502.21309](https://arxiv.org/abs/2502.21309) |
+| SimpleGPT | [2602.01212](https://arxiv.org/abs/2602.01212) |
+| GPAS: Gradient-Preserving Activation Scaling | [2506.22049](https://arxiv.org/abs/2506.22049) |
+| PolyNorm / PolyCom | [2602.04902](https://arxiv.org/abs/2602.04902) |
+| Momentum Attention | [2411.03884](https://arxiv.org/abs/2411.03884) |
+
+---
 
-| Training Loss | Epoch  | Step  | Validation Loss |
-|:-------------:|:------:|:-----:|:---------------:|
-| 3.7011        | 0.1067 | 5000  | 3.6012          |
-| 3.2863        | 0.2133 | 10000 | 3.1686          |
-| 3.0744        | 0.32   | 15000 | 2.9543          |
-| 2.9347        | 0.4267 | 20000 | 2.8076          |
-| 2.8322        | 0.5333 | 25000 | 2.6952          |
-| 2.7471        | 0.64   | 30000 | 2.6047          |
-| 2.6859        | 0.7467 | 35000 | 2.5439          |
-| 2.6580        | 0.8533 | 40000 | 2.5045          |
-| 2.4929        | 0.96   | 45000 | 2.3449          |
+## License
 
+Apache 2.0
 
-### Framework versions
+## Citation
 
-- Transformers 5.3.0
-- Pytorch 2.10.0+cu130
-- Datasets 4.8.4
-- Tokenizers 0.22.2
+```bibtex
+@misc{neollm2026,
+  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
+  author = {KitsuVp},
+  year   = {2026},
+  url    = {https://huggingface.co/KitsuVp/NeoLLM}
+}
+```