--- language: - en license: apache-2.0 pipeline_tag: text-generation library_name: gguf tags: - llm - transformer - causal-language-model - gguf - pytorch - text-generation - decoder-only - rope - rmsnorm - swiglu --- # SwitLM

**A Lightweight Decoder-Only Transformer Language Model Built from Scratch** * RoPE • RMSNorm • SwiGLU • GGUF*

--- ## Overview **SwitLM** is a family of lightweight decoder-only Transformer language model implemented entirely from scratch using **PyTorch**. The project was created to explore every stage of modern LLM development, including architecture design, training, checkpointing, and conversion to the **GGUF** format for efficient inference. Unlike models built on existing Hugging Face architectures, this implementation includes custom implementations of: - Transformer Blocks - Multi-Head Self Attention - Rotary Position Embeddings (RoPE) - RMSNorm - SwiGLU Feed Forward Networks - Causal Language Modeling - GGUF Export Pipeline The model is intended primarily for **research, experimentation, and educational purposes**. --- # Features - ✅ Decoder-only Transformer - ✅ Rotary Position Embeddings (RoPE) - ✅ RMSNorm Normalization - ✅ SwiGLU Feed Forward Network - ✅ Weight Tied Embeddings - ✅ GPT-2 Tokenizer - ✅ Mixed Precision Training - ✅ Gradient Accumulation - ✅ Exported in GGUF Format --- # Model Architecture ```mermaid flowchart TD A[Input Text] B[GPT-2 Tokenizer] C[Token Embeddings] A --> B B --> C subgraph Transformer["12 × Transformer Decoder Blocks"] D1[RMSNorm] D2[Multi-Head Self Attention] D3[Rotary Position Embeddings] D4[Residual Connection] D5[RMSNorm] D6[SwiGLU Feed Forward] D7[Residual Connection] D1 --> D2 D2 --> D3 D3 --> D4 D4 --> D5 D5 --> D6 D6 --> D7 end C --> Transformer Transformer --> E[Final RMSNorm] E --> F[Language Modeling Head] F --> G[Softmax] G --> H[Next Token Prediction] ``` # Model Architecture ``` SwitLM Decoder-Only Transformer Architecture ┌──────────────────────────────────────────────────────────────────────────────┐ │ Input Text │ └───────────────────────────────┬──────────────────────────────────────────────┘ │ ▼ GPT-2 Byte Pair Tokenizer │ ▼ Token & Embedding Layer │ ▼ ┌──────────────────────────────────────────────────────────────────────────────┐ │ Transformer Decoder Block × 12 │ │ │ │ ┌────────────────────────────────────────────────────────────┐ │ │ │ RMSNorm │ │ │ └──────────────────────┬─────────────────────────────────────┘ │ │ ▼ │ │ Multi-Head Self Attention (12 Heads) │ │ │ │ │ Rotary Position Embeddings (RoPE) │ │ │ │ │ Residual Connection (+) │ │ │ │ │ ┌──────────────────────▼─────────────────────────────────────┐ │ │ │ RMSNorm │ │ │ └──────────────────────┬─────────────────────────────────────┘ │ │ ▼ │ │ SwiGLU Feed Forward Network │ │ │ │ │ Residual Connection (+) │ │ │ │ └─────────────────────────────┼────────────────────────────────────────────────┘ │ ▼ Final RMSNorm │ ▼ Language Modeling Head (Weight Tied Output) │ ▼ Softmax Next Token Prediction ``` --- ## Decoder Block ``` ┌──────────────────────────┐ │ Input (x) │ └────────────┬─────────────┘ │ ▼ RMSNorm │ ▼ Multi-Head Self Attention + Rotary Position Embeddings │ ▼ Residual Add (+) │ ▼ RMSNorm │ ▼ SwiGLU Feed Forward │ ▼ Residual Add (+) │ ▼ Output ``` --- ## Architecture Summary | Component | Specification | |-----------|---------------| | Architecture | Decoder-only Transformer | | Hidden Size | 768 | | Transformer Layers | 12 | | Attention Heads | 12 | | Feed Forward Network | SwiGLU (3072 Hidden Units) | | Position Encoding | Rotary Position Embeddings (RoPE) | | Normalization | RMSNorm | | Context Length | 2048 Tokens | | Tokenizer | GPT-2 BPE | | Weight Tying | Enabled | | Output | GGUF | # Model Specifications | Property | Value | |-----------|--------| | Architecture | Decoder-only Transformer | | Parameters | ~100M | | Hidden Size | 768 | | Transformer Layers | 12 | | Attention Heads | 12 | | Feed Forward Size | 3072 | | Context Length | 2048 Tokens | | Positional Encoding | Rotary Position Embeddings (RoPE) | | Normalization | RMSNorm | | Activation | SwiGLU | | Weight Tying | Yes | | Tokenizer | GPT-2 | | Output Format | GGUF | --- # Training Configuration The model was trained entirely in **PyTorch** using a custom training pipeline. ### Optimizer - AdamW ### Hyperparameters | Parameter | Value | |-----------|--------| | Learning Rate | 3e-4 | | Weight Decay | 0.01 | | Betas | (0.9, 0.95) | | Learning Rate Scheduler | Linear Warmup + Linear Decay | | Mixed Precision | Enabled | | Gradient Accumulation | Enabled | | Gradient Clipping | 1.0 | --- # Training Datasets The model was trained sequentially on multiple publicly available datasets. | Dataset | Description | |----------|-------------| | WikiText | General language modeling | | TinyStories | Story generation | | AG News | News articles | | IMDb Reviews | Long-form text | All datasets were tokenized using the GPT-2 tokenizer. --- # Tokenizer The model uses the **GPT-2 Byte Pair Encoding (BPE)** tokenizer. - GPT-2 Vocabulary - Byte Pair Encoding - EOS token used as padding token --- # Design Choices ## Rotary Position Embeddings (RoPE) Instead of learned positional embeddings, SwitLM uses Rotary Position Embeddings to improve positional awareness and enable better handling of longer contexts. --- ## RMSNorm RMSNorm replaces LayerNorm to reduce computational overhead while maintaining stable training dynamics. --- ## SwiGLU Feed Forward Network The standard GELU feed-forward network has been replaced with SwiGLU, improving model expressiveness and training efficiency. --- ## Weight Tying The input embedding matrix is shared with the output language modeling head, reducing parameter count and improving generalization. --- # Repository Contents ``` SmallLLM-100M.gguf README.md LICENSE ``` --- # Running the Model The model is provided in **GGUF** format and is compatible with several inference engines. Supported runtimes include: - llama.cpp - LM Studio - KoboldCpp - GPT4All - Jan - Ollama (after conversion if required) Example: ```bash ./llama-cli \ -m SmallLLM-100M.gguf \ -p "The future of artificial intelligence is" ``` --- # Intended Use This model is suitable for: - Educational purposes - Transformer architecture research - Learning how LLMs work - Fine-tuning experiments - Small-scale text generation - GGUF inference experiments --- # Limitations It is a relatively small language model compared to modern large language models. Users should expect: - Limited reasoning capability - Limited factual knowledge - Reduced instruction-following performance - Hallucinations - Lower performance on complex tasks This model should **not** be used for: - Medical advice - Legal advice - Financial decisions - Safety-critical applications --- # Hardware The training pipeline was designed for consumer GPUs. Primary development hardware: - NVIDIA RTX 3070 (8 GB VRAM) Inference can be performed on either CPU or GPU depending on the inference backend. --- # Citation If you use this model in your research, please cite: ```bibtex @misc{SwitLM, title={SwitLM-100M}, author={Avijit Paul}, year={2026}, publisher={Hugging Face}, url={https://huggingface.co/AvijitPaul/SwitLM} } ``` --- # License This project is released under the **Apache 2.0 License**.