| # QuantMobileLLM — Lightweight GPT-Style Language Model | |
| MobileLLM is a **lightweight GPT-style language model** designed for efficiency, fast inference, and small deployment environments. | |
| It’s trained on **FineWeb-MINI** and optimized with **modern attention techniques**. | |
| --- | |
| ## 🚀 Model Highlights | |
| - **Architecture**: Decoder-only GPT-style transformer | |
| - **Parameters**: ~17M (6 layers, 8 heads, 256 embedding dim) | |
| - **Context Length**: 512 tokens | |
| - **Vocabulary Size**: 50,304 tokens | |
| - **Precision**: Supports both `fp16` and `bf16` | |
| - **Optimized for**: Small GPUs, mobile inference, | |
| --- | |
| ## 🧠 Architecture Details | |
| | **Component** | **Value** | | |
| |--------------------|-----------| | |
| | Layers | 6 | | |
| | Attention Heads | 8 | | |
| | KV Heads | 4 | | |
| | Embedding Dim | 256 | | |
| | Context Length | 512 | | |
| | Vocab Size | 50,304 | | |
| | Attention Type | Multi-Query Attention | | |
| | Norm Type | RMSNorm | | |
| | Position Encoding | Rotary Position Embeddings (RoPE) | | |
| | FFN Activation | SwiGLU (`silu`) | | |
| ### 🔹 Key Optimizations | |
| - **RMSNorm** → Improves training stability over LayerNorm. | |
| - **Multi-Query Attention** → Reduces KV-cache size → lower memory footprint. | |
| - **Rotary Embeddings (RoPE)** → Better handling of long context windows. | |
| - **`safetensors` checkpoints** → Faster & safer loading. | |
| --- | |
| ## 📊 Training Setup | |
| | **Property** | **Value** | | |
| |------------------------|-----------| | |
| | Dataset | [FineWeb-MINI](https://huggingface.co/datasets/AryanNsc/FineWeb-Mini) | | |
| | Tokens Trained | ~100M | | |
| | Optimizer | AdamW | | |
| | Learning Rate | 6e-4 (cosine decay) | | |
| | Warmup Steps | 100 | | |
| | Batch Size | 64 × 2 grad accum | | |
| | Effective Batch Size | 128 | | |
| | Mixed Precision | `fp16` / `bf16` (auto-detect) | | |
| | Distributed Training | DDP | | |
| | Logging | Weights & Biases (`wandb`) | | |
| | Checkpoint Format | `.safetensors` | | |
| --- | |
| ## 🧩 Model Checkpoints | |
| | **Step** | **Filename** | **Format** | | |
| |----------|------------|------------| | |
| | Final | `mobile_llm_final.safetensors` | safetensors | | |
| | Intermediate | `checkpoints/mobile_llm_step_<step>.safetensors` | safetensors | | |
| --- | |
| ## 🔮 Roadmap | |
| - [x] Train **MobileLLM** on **FineWeb-MINI** | |
| - [x] Add **multi-query attention** | |
| - [x] Export **safetensors** checkpoints | |
| - [ ] Quantized **int8** & **int4** inference | |
| - [ ] Expand training on **FineWeb-1B** | |
| --- | |
| ## 📜 License | |
| This model is licensed under the [MIT License](LICENSE). | |
| --- | |
| ## 🌐 Links | |
| - **Github** → [MobileLLM training code](https://github.com/Guney-olu/Quantgpt) | |