README.md

---
language:
- en
license: apache-2.0
pipeline_tag: text-generation
library_name: gguf

tags:
- llm
- transformer
- causal-language-model
- gguf
- pytorch
- text-generation
- decoder-only
- rope
- rmsnorm
- swiglu
---

# SwitLM

<p align="center">

**A Lightweight Decoder-Only Transformer Language Model Built from Scratch**

* RoPE • RMSNorm • SwiGLU • GGUF*

</p>

---

## Overview

**SwitLM** is a family of lightweight decoder-only Transformer language model implemented entirely from scratch using **PyTorch**. The project was created to explore every stage of modern LLM development, including architecture design, training, checkpointing, and conversion to the **GGUF** format for efficient inference.

Unlike models built on existing Hugging Face architectures, this implementation includes custom implementations of:

- Transformer Blocks
- Multi-Head Self Attention
- Rotary Position Embeddings (RoPE)
- RMSNorm
- SwiGLU Feed Forward Networks
- Causal Language Modeling
- GGUF Export Pipeline

The model is intended primarily for **research, experimentation, and educational purposes**.

---

# Features

- ✅ Decoder-only Transformer
- ✅ Rotary Position Embeddings (RoPE)
- ✅ RMSNorm Normalization
- ✅ SwiGLU Feed Forward Network
- ✅ Weight Tied Embeddings
- ✅ GPT-2 Tokenizer
- ✅ Mixed Precision Training
- ✅ Gradient Accumulation
- ✅ Exported in GGUF Format

---

# Model Architecture

```mermaid
flowchart TD

A[Input Text]
B[GPT-2 Tokenizer]
C[Token Embeddings]

A --> B
B --> C

subgraph Transformer["12 × Transformer Decoder Blocks"]

D1[RMSNorm]
D2[Multi-Head Self Attention]
D3[Rotary Position Embeddings]
D4[Residual Connection]
D5[RMSNorm]
D6[SwiGLU Feed Forward]
D7[Residual Connection]

D1 --> D2
D2 --> D3
D3 --> D4
D4 --> D5
D5 --> D6
D6 --> D7

end

C --> Transformer
Transformer --> E[Final RMSNorm]
E --> F[Language Modeling Head]
F --> G[Softmax]
G --> H[Next Token Prediction]
```

# Model Architecture

```
                                SwitLM
                   Decoder-Only Transformer Architecture
┌──────────────────────────────────────────────────────────────────────────────┐
│                              Input Text                                     │
└───────────────────────────────┬──────────────────────────────────────────────┘
                                │
                                ▼
                     GPT-2 Byte Pair Tokenizer
                                │
                                ▼
                     Token & Embedding Layer
                                │
                                ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                     Transformer Decoder Block × 12                           │
│                                                                              │
│      ┌────────────────────────────────────────────────────────────┐          │
│      │ RMSNorm                                                    │          │
│      └──────────────────────┬─────────────────────────────────────┘          │
│                             ▼                                                │
│              Multi-Head Self Attention (12 Heads)                            │
│                             │                                                │
│             Rotary Position Embeddings (RoPE)                                │
│                             │                                                │
│                   Residual Connection (+)                                    │
│                             │                                                │
│      ┌──────────────────────▼─────────────────────────────────────┐          │
│      │ RMSNorm                                                    │          │
│      └──────────────────────┬─────────────────────────────────────┘          │
│                             ▼                                                │
│                   SwiGLU Feed Forward Network                                │
│                             │                                                │
│                   Residual Connection (+)                                    │
│                             │                                                │
└─────────────────────────────┼────────────────────────────────────────────────┘
                              │
                              ▼
                         Final RMSNorm
                              │
                              ▼
                     Language Modeling Head
                       (Weight Tied Output)
                              │
                              ▼
                   Softmax Next Token Prediction
```

---

## Decoder Block

```
                  ┌──────────────────────────┐
                  │        Input (x)         │
                  └────────────┬─────────────┘
                               │
                               ▼
                          RMSNorm
                               │
                               ▼
                  Multi-Head Self Attention
                     + Rotary Position
                        Embeddings
                               │
                               ▼
                     Residual Add (+)
                               │
                               ▼
                          RMSNorm
                               │
                               ▼
                   SwiGLU Feed Forward
                               │
                               ▼
                     Residual Add (+)
                               │
                               ▼
                           Output
```

---

## Architecture Summary

| Component | Specification |
|-----------|---------------|
| Architecture | Decoder-only Transformer |
| Hidden Size | 768 |
| Transformer Layers | 12 |
| Attention Heads | 12 |
| Feed Forward Network | SwiGLU (3072 Hidden Units) |
| Position Encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm |
| Context Length | 2048 Tokens |
| Tokenizer | GPT-2 BPE |
| Weight Tying | Enabled |
| Output | GGUF |

# Model Specifications

| Property | Value |
|-----------|--------|
| Architecture | Decoder-only Transformer |
| Parameters | ~100M |
| Hidden Size | 768 |
| Transformer Layers | 12 |
| Attention Heads | 12 |
| Feed Forward Size | 3072 |
| Context Length | 2048 Tokens |
| Positional Encoding | Rotary Position Embeddings (RoPE) |
| Normalization | RMSNorm |
| Activation | SwiGLU |
| Weight Tying | Yes |
| Tokenizer | GPT-2 |
| Output Format | GGUF |

---

# Training Configuration

The model was trained entirely in **PyTorch** using a custom training pipeline.

### Optimizer

- AdamW

### Hyperparameters

| Parameter | Value |
|-----------|--------|
| Learning Rate | 3e-4 |
| Weight Decay | 0.01 |
| Betas | (0.9, 0.95) |
| Learning Rate Scheduler | Linear Warmup + Linear Decay |
| Mixed Precision | Enabled |
| Gradient Accumulation | Enabled |
| Gradient Clipping | 1.0 |

---

# Training Datasets

The model was trained sequentially on multiple publicly available datasets.

| Dataset | Description |
|----------|-------------|
| WikiText | General language modeling |
| TinyStories | Story generation |
| AG News | News articles |
| IMDb Reviews | Long-form text |

All datasets were tokenized using the GPT-2 tokenizer.

---

# Tokenizer

The model uses the **GPT-2 Byte Pair Encoding (BPE)** tokenizer.

- GPT-2 Vocabulary
- Byte Pair Encoding
- EOS token used as padding token

---

# Design Choices

## Rotary Position Embeddings (RoPE)

Instead of learned positional embeddings, SwitLM uses Rotary Position Embeddings to improve positional awareness and enable better handling of longer contexts.

---

## RMSNorm

RMSNorm replaces LayerNorm to reduce computational overhead while maintaining stable training dynamics.

---

## SwiGLU Feed Forward Network

The standard GELU feed-forward network has been replaced with SwiGLU, improving model expressiveness and training efficiency.

---

## Weight Tying

The input embedding matrix is shared with the output language modeling head, reducing parameter count and improving generalization.

---

# Repository Contents

```
SmallLLM-100M.gguf
README.md
LICENSE
```

---

# Running the Model

The model is provided in **GGUF** format and is compatible with several inference engines.

Supported runtimes include:

- llama.cpp
- LM Studio
- KoboldCpp
- GPT4All
- Jan
- Ollama (after conversion if required)

Example:

```bash
./llama-cli \
-m SmallLLM-100M.gguf \
-p "The future of artificial intelligence is"
```

---

# Intended Use

This model is suitable for:

- Educational purposes
- Transformer architecture research
- Learning how LLMs work
- Fine-tuning experiments
- Small-scale text generation
- GGUF inference experiments

---

# Limitations

It is a relatively small language model compared to modern large language models.

Users should expect:

- Limited reasoning capability
- Limited factual knowledge
- Reduced instruction-following performance
- Hallucinations
- Lower performance on complex tasks

This model should **not** be used for:

- Medical advice
- Legal advice
- Financial decisions
- Safety-critical applications

---

# Hardware

The training pipeline was designed for consumer GPUs.

Primary development hardware:

- NVIDIA RTX 3070 (8 GB VRAM)

Inference can be performed on either CPU or GPU depending on the inference backend.


---

# Citation

If you use this model in your research, please cite:

```bibtex
@misc{SwitLM,
  title={SwitLM-100M},
  author={Avijit Paul},
  year={2026},
  publisher={Hugging Face},
  url={https://huggingface.co/AvijitPaul/SwitLM}
}
```

---

# License

This project is released under the **Apache 2.0 License**.