---
license: mit
language:
- en
tags:
- text-generation
- causal-lm
- gpt
- tinygpt
- transformer
- llm
library_name: transformers
pipeline_tag: text-generation
base_model: gpt2
widget:
- text: Once upon a time, a little boy named Sam
  example_title: Story start
- text: The cat and the dog decided to
  example_title: Animals
---

# TinyGPT

A small-scale, **trained-from-scratch** GPT language model. Based on the GPT-2 architecture and fully compatible with `transformers`; downloadable via the Hub, with model source code included in this repo.

- **Architecture:** Decoder-only Transformer (custom, Chapter-1 style)
- **Parameters:** ~29M (`28,974,161`)
- **Context length (`max_seq_len`):** 128 tokens
- **Tokenizer:** GPT-2 (vocab size = 50,257, BPE)
- **Training data:** First 50,000 stories from [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories)
- **Inference memory (fp32 weights):** ~116MB -- runs comfortably on a single GPU or even CPU

---

## Installation

```bash
pip install transformers torch
```

---

## Usage (with transformers)

TinyGPT uses a custom architecture, so the model code is loaded from this repo at runtime. This requires the **`trust_remote_code=True`** flag.

### 1) Simplest approach: Auto classes

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "coderian/TinyGPT"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,   # loads custom architecture code from repo
)

model.eval()

# Generation (sampling)
prompt = "Once upon a time, a little girl named Lily found"
inputs = tokenizer(prompt, return_tensors="pt")

output = model.generate(
    **inputs,
    max_new_tokens=50,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    top_p=0.95,
)

text = tokenizer.decode(output[0], skip_special_tokens=True)
print(text)
```

### 2) Direct class (code-based usage)

If you copied the source code into your own project, `trust_remote_code` is not needed:

```python
from models.tinygpt import TinyGPT

model = TinyGPT.from_pretrained("coderian/TinyGPT", device="cpu")
```

### 3) GPU placement and half-precision

```python
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype="auto",          # uses config dtype (fp32)
    device_map="auto",           # places model on GPU if available
)

# Lower memory usage with bf16/fp16:
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype="bf16",          # or "float16"
    device_map="auto",
)
```

### 4) Simple greedy chat loop

```python
def reply(prompt, max_new_tokens=40):
    inputs = tokenizer(prompt, return_tensors="pt")
    out = model.generate(**inputs, max_new_tokens=max_new_tokens)
    return tokenizer.decode(out[0], skip_special_tokens=True)

print(reply("One rainy day, Ben and his dog Rex decided to"))
```

> **Note:** The model was trained exclusively on **English** TinyStories data. It works best with simple English story openings.

---

## Repo contents

| File | Description |
|---|---|
| `model.safetensors` | Trained weights (~116MB, fp32) |
| `config.json` | Model configuration (`TinyGPTConfig`) |
| `generation_config.json` | Default `generate()` parameters |
| `tokenizer.json`, `tokenizer_config.json` | GPT-2 tokenizer |
| `models/` | Open-source model code (config, tinygpt, transformer_block, attention) |

---

## Architecture

TinyGPT is a **decoder-only Transformer**. Input tokens pass through `token + position embedding`, are processed by `num_layers` Transformer blocks, and finally a LayerNorm + linear head produces logits over the vocabulary.

```
Input tokens (B, T)
      |
      v
+-----------------------------------+
|  token_embedding  (50,257 -> d)    |  nn.Embedding
|  position_embedding (T -> d)       |  nn.Embedding
|  x = token_emb + pos_emb           |
+------------------+-----------------+
                   v
   +-----------------------------+  x num_layers (4)
   |  LN1 -> CausalSelfAttn      |  Pre-LN, residual
   |       +                     |
   |  LN2 -> FFN (x4)            |  GELU activation
   |       +                     |
   +-----------------------------+
                   v
            ln_f (LayerNorm)
                   v
        lm_head (d -> 50,257) --> logits (B, T, V)
```

### 1) Embedding layers

- **`token_embedding`**: maps each token ID to a `d=256` dimensional vector. `W_e` in R^(50257 x 256)
- **`position_embedding`**: adds positional information. Consecutive positions use the same learned embedding; `W_p` in R^(128 x 256)

Input vector: `x = W_e[input_ids] + W_p[positions]`

### 2) Causal Self-Attention (masked scaled dot-product)

Each position can only attend to **previous** tokens (causal mask).

```
Q = x W_q ,  K = x W_k ,  V = x W_v          W in R^(256x256)

scores     = (Q K^T) / sqrt(d)                 d = 256
scores     = masked_fill(upper_triangle, -inf)  <- blocks future tokens
attn       = softmax(scores, dim=-1)
output     = attn @ V ,  then  output W_o
```

- Scaling by `1/sqrt(d)`: keeps dot-product variance stable at high dimensions,
  preventing softmax saturation.
- `nn.Embedding` and `nn.Linear` weights are initialized from `N(0, 0.02)`.

### 3) Residual connections and Pre-LayerNorm

LayerNorm is applied before each sublayer, followed by a residual add:

```
x = x + attention(ln1(x))
x = x + ffn(ln2(x))
```

Pre-LN keeps training stable in deeper models.

### 4) FFN (Feed-Forward Network)

```
FFN(x) = W2 * GELU(W1 * x + b1) + b2

W1: 256 -> 1024   (expansion x4)
GELU: activation
W2: 1024 -> 256   (contraction)
```

### 5) Output

```
logits = lm_head(ln_f(x))          ln_f: LayerNorm(256)
                                    lm_head: Linear(256 -> 50,257)
```

During training, `CrossEntropyLoss(logits, shift_right(y))` is used for the language modeling objective.
During inference, the `lm_head` logits directly represent the next-token distribution.

### Parameter breakdown

| Component | Parameters | Notes |
|---|---|---|
| `token_embedding` | 12,865,792 | 50,257 x 256 |
| `position_embedding` | 32,768 | 128 x 256 |
| 1 Transformer block | 789,760 | LN + attn + FFN |
| 4 Transformer blocks | 3,159,040 | total body |
| `ln_f` + `lm_head` | 12,916,561 | language head |

**Total: ~28.97M.** Most parameters reside in the embedding/head due to the large vocabulary; the actual reasoning capacity (transformer body) is ~3.1M.

---

## Training details

| Metric | Value |
|---|---|
| Data | TinyStories (first 50,000 stories, ~11.1M tokens) |
| Training samples | 86,636 blocks x 128 tokens |
| `embed_dim` | 256 |
| `num_layers` | 4 |
| `max_seq_len` | 128 |
| Optimizer | AdamW, lr `3e-4` |
| Batch size | 24 |
| Epochs | 2 |
| Duration | Short training run on CPU |

---

## Limitations & Notes

- English only; produces nonsensical output for non-English input.
- 128-token context window; insufficient for tasks requiring long-range context.
- Due to its small size, the model does not perform logic, recall factual knowledge, or produce coherent long-form text.
- Requires `trust_remote_code=True` to run; the code is open for inspection in this repo's `models/` directory.
- No weight decay or other regularization was applied during training.

---

## License

MIT -- model weights and code are freely usable.

**HF Hub:** [huggingface.co/coderian/TinyGPT](https://huggingface.co/coderian/TinyGPT)