| --- |
| library_name: transformers |
| tags: |
| - hyper-efficient |
| - long-context |
| - randnla |
| - matryoshka |
| - sub-quadratic |
| - muon |
| - research |
| license: mit |
| language: |
| - en |
| metrics: |
| - perplexity |
| --- |
| |
|
|
| # MaximusLLM |
|
|
| MaximusLLM is a long-context language model designed for hyper-efficient architecture and training. It introduces a new paradigm for to long context while reducing training VRAM by ~40% and increasing throughput by over 17x compared to optimized standard Cross-Entropy baselines. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| - **Developed by:** Yousef Gamaleldin (Independent Researcher) |
| - **Model type:** Transformer with Bifurcated Latent Attention |
| - **Language(s) (NLP):** English |
| - **License:** MIT |
| - **Finetuned from model:** Trained from scratch (Base) followed by Instruction Pre-training. |
| - **Tokenizer:** Gemma 3 (262,144 vocab size) |
|
|
| ### Model Sources |
|
|
| - **Repository:** [yousefg/MaximusLLM](https://github.com/yousefg/MaximusLLM) |
| - **Technical Reports:** |
| - *MAXIS: A Hyper-Efficient Paradigm for Scalable Long-Context LLM Training* |
| - *Bifurcated Latent Attention: Scaling LLMs to Infinite Context via Asymmetric Causal RandNLA* |
|
|
| ## Bias, Risks, and Limitations |
|
|
| MaximusLLM (190M) is an architectural proof-of-concept. While it demonstrates extreme efficiency, its absolute knowledge capacity is limited by its parameter count. Users should expect hallucinations. |
|
|
| ## How to Get Started with the Model |
|
|
| ```python |
| from src.model import Model, Config |
| from src.lora import blockswap_attention_layers |
| from src.infer import general_generate_fn |
| |
| config = Config.from_pretrained("yousefg/MaximusLLM") |
| model = Model(config, device="cuda") |
| blockswap_attention_layers(model) |
| |
| prompt = "<start_of_turn>user\nWhat is the capital of France?<end_of_turn>\n<start_of_turn>model\n" |
| inputs = tokenizer(prompt, return_tensors="pt").to("cuda") |
| output = general_generate_fn(model, inputs, tokenizer, max_new_tokens=50) |
| print(tokenizer.decode(output[0])) |
| ``` |
|
|
| ## Training Details |
|
|
| ### Training Data |
|
|
| 1. **Pre-training:** A high-quality subset of `HuggingFaceFW/fineweb-edu`. |
| 2. **Narrative Alignment:** `roneneldan/TinyStories` to stabilize linguistic fluidity. |
| 3. **Instruction Alignment:** `HuggingFaceH4/ultrachat_200k` using a multi-turn conversational format. |
|
|
| ### Training Procedure |
|
|
| Maximus utilizes a specialized training pipeline to maintain FP32 master weight stability while achieving FP16 throughput. |
|
|
| #### Training Hyperparameters |
|
|
| - **Optimizers:** |
| - **Muon:** Applied to all 2D weight matrices (Attention/MLP) with LR 0.02 (Pre-train) and 0.005 (SFT). |
| - **AdamW:** Applied to Embeddings, Head, and Norms (LR 4e-4). |
| - **Loss Function:** **MAXIS Loss** (Unnormalized Ghost Logits + Matryoshka Auxiliary loss). |
| - **Precision:** FP32 Master Weights, FP16 Mixed Precision (Autocast). |
| - **Effective Batch Size:** 64 to 256 (via Gradient Accumulation). |
| - **Context Length:** Scaled from 2,048 to 8,192 native (Long-context phase). |
|
|
| #### Speeds, Sizes, Times |
|
|
| - **Throughput:** 2.81 updates/sec (17.5x faster than Liger-fused Cross-Entropy). |
| - **VRAM Savings:** 38.7% reduction in peak memory usage. |
| - **Scaling:** $O(N \cdot K)$ complexity achieved via Query Chunking and KV-compression. |
|
|
| ## Technical Specifications |
|
|
| ### Model Architecture and Objective |
|
|
| MaximusLLM utilizes three core innovations: |
| 1. **MAXIS Loss:** A Matryoshka-structured loss using **Dynamic Variance Ghost Logits** to simulate the full-vocabulary distribution, preventing the "premature saturation" common in sampled softmax. |
| 2. **RandNLA Attention:** Bifurcates the KV-cache into a **Top-K Detail Path** (lossless) and a **Causal Kronecker Sketch Path** (compressed background). It uses an **Asymmetric Causal Mask** to remain strictly autoregressive. |
| 3. **Fisher SVD:** Leverages the Fisher Information Matrix ($\sum (\frac{\partial L}{\partial W})^2$) to optimally initialize latent spaces, preserving pre-trained intelligence during architectural transitions. |
|
|
| ### Compute Infrastructure |
|
|
| #### Hardware |
| - **Primary:** NVIDIA Tesla T4 (16GB VRAM) / 2x Tesla T4 via Kaggle/Cloud. |
| - **Secondary:** Benchmarked on NVIDIA L4 (24GB VRAM). |
|
|
| #### Software |
| - **Framework:** PyTorch 2.5+ or 2.9+ for training |
| - **Compiler:** `torch.compile` (Hollow-compilation of inner blocks for stability). |
|
|
| ## Citation |
|
|
| **MAXIS Loss:** |
| ```bibtex |
| @article{gamaleldin2026maxis, |
| title={MAXIS: A Hyper-Efficient Paradigm for Scalable Long-Context LLM Training}, |
| author={Gamaleldin, Yousef}, |
| journal={SSRN: Artificial Intelligence eJournal}, |
| year={2026} |
| } |
| ``` |
|
|
| **RandNLA Attention:** |
| ```bibtex |
| @article{gamaleldin2026randnla, |
| title={Bifurcated Latent Attention: Scaling LLMs to Infinite Context via Asymmetric Causal RandNLA}, |
| author={Gamaleldin, Yousef}, |
| journal={SSRN: Artificial Intelligence eJournal}, |
| year={2026} |
| } |
| ``` |
|
|
| ## Model Card Contact |
| Yousef Gamaleldin - [yrafat38@gmail.com] |
