🚀 MCR-Attention V2.0 — 66.9M (FineWeb-Edu)
MCR-Attention V2.0 (66.9M) is a custom-built, lightweight language model powered by a Multi-Scale Context Router (MCR) attention mechanism. Trained on HuggingFace's FineWeb-Edu (200k samples / ~212.7M effective tokens), this architecture achieves constant-time $O(1)$ per-step generation speed regardless of sequence length, making it ultra-fast and memory-efficient for real-time streaming.
📊 Key Highlights & Metrics
| Metric | Value |
|---|---|
| Total Parameters | 66,878,464 (~66.9M) |
| Vocabulary Size | 24,576 (BPE, 128-aligned) |
| Evaluation Loss | 4.8983 |
| Evaluation Perplexity | 134.07 |
| Final Training Loss (avg200) | 4.9295 |
| Peak Streaming Throughput | ~369 tokens/sec (2.71 ms/token) |
| Inference Step Complexity | $O(1)$ Constant Time |
| Training Time | 97.8 minutes (5,869s) on 2x NVIDIA T4 |
🏗️ Architecture & Parameter Breakdown
The MCR-Attention mechanism uses a dynamic Attention Router that allocates compute across $K=4$ explicit memory scales: [32, 128, 512, 2048].
Parameter Distribution
| Component | Parameters | Percentage |
|---|---|---|
| Embedding Layer | 12,582,912 | 18.8% |
| Memory Scales | 1,576,960 | 2.4% |
| Attention Router | 1,311,744 | 2.0% |
| Language Model Head | 51,406,848 | 76.8% |
| TOTAL | 66,878,464 | 100% |
⚡ Generation Speed Benchmark
Due to fixed state-compression across the memory blocks, generation latency remains constant as the sequence grows:
| Generated Tokens | Total Time (s) | Latency (ms/token) | Throughput (tok/s) | Step Complexity |
|---|---|---|---|---|
| 50 | 0.167 | 3.34 | 299 | $O(1)$ |
| 100 | 0.297 | 2.97 | 336 | $O(1)$ |
| 200 | 0.573 | 2.87 | 349 | $O(1)$ |
| 500 | 1.401 | 2.80 | 357 | $O(1)$ |
| 1000 | 2.724 | 2.72 | 367 | $O(1)$ |
| 2000 | 5.421 | 2.71 | 369 | $O(1)$ Constant |
Key Takeaway: As shown above, time per token asymptotically approaches 2.71 ms/token, confirming true $O(1)$ step-wise inference.
📈 Training Progress & Hyperparameters
- Dataset:
HuggingFaceFW/fineweb-edu(200,000 samples, ~212.66M tokens processed) - Sequence Length: 2048 tokens (Sample packing enabled)
- Hardware: 2x NVIDIA T4 GPUs (Kaggle)
- Precision:
fp16mixed precision with gradient scaling - Batch Size: Physical batch 4 per GPU $\rightarrow$ Effective batch 32 (65,504 tokens/step)
- Optimizer:
AdamW($\beta_1=0.9, \beta_2=0.95, \epsilon=10^{-8}$) - Learning Rate Schedule: Warmup 100 steps $\rightarrow$ Cosine Decay (
5.0e-04$\rightarrow$5.0e-05) - Total Training Throughput: ~36,237 tokens/sec average
📝 Sample Generation Output
Prompt: "The theory of evolution explains how"
The theory of evolution explains how species change over time through mechanisms such as natural selection. Organisms with traits better suited to their environment are more likely to survive and reproduce, passing these advantageous traits to the next generation...
Copyright (c) 2026 Gerson Fabian Buenahora Ormaza (BUEORM)
Licensed under the BUEORM MCR-Attention Community & Research License v1.0.
- Downloads last month
- 210