---
license: mit
tags:
  - gpt
  - language-model
  - text-generation
  - tinystories
  - pytorch
  - from-scratch
  - slm
library_name: pytorch
pipeline_tag: text-generation
datasets:
  - roneneldan/TinyStories
---

# scribe

A **~30M-parameter small language model built from scratch in PyTorch** and trained on
[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). It's a decoder-only GPT —
hand-written multi-head causal attention, pre-LayerNorm blocks, GELU MLP, tied LM head — with no
`transformers`/`nanoGPT` modelling code. The architecture is deliberately **GPT-2-compatible**, so
these weights load directly into [`ember`](https://github.com/mbsdeepak/ember), a from-scratch
LLM inference server.

📦 **Code, training, and full write-up (incl. the maths):** https://github.com/mbsdeepak/scribe

## What it does

Give it a prompt and it continues it as a simple, coherent children's story:

> **"Once upon a time"** → *there was a little girl named Lily. She loved to read magazines. One
> day, she found a magazine in her mom's room... "Mommy, look at that magazine!" Lily said... Her
> mom looked at the magazine and smiled. "That's very nice, Lily. But be careful with it, it might
> break."*
>
> **"One day, a boy named Deepak"** → *went to the park every day. He saw a large tree and wanted to
> climb it. He started to climb it, but it was too heavy for him. He tried to jump, but he fell.*

## Model

| | |
| :-- | :-- |
| Architecture | decoder-only transformer (GPT-2 family) |
| Parameters | ~30.0M (19.3M of it the tied 50257-token embedding) |
| Layers / heads / width | 6 / 6 / 384 |
| Context length | 256 tokens |
| Tokenizer | GPT-2 BPE (`tiktoken`) |
| Training | TinyStories, AdamW + warmup/cosine, ~3.4h on Apple-Silicon MPS |
| Validation loss | **1.74** (cross-entropy) |

## Files

- `model.safetensors` — the weights (fp32)
- `config.json` — architecture config to rebuild the model

## Usage

```python
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
import json, torch

# get the code: git clone https://github.com/mbsdeepak/scribe
from config import GPTConfig
from src.model import GPT
from src.tokenizer import Tokenizer

cfg_json = json.load(open(hf_hub_download("mbsdeepak/scribe", "config.json")))
gc = GPTConfig(n_layer=cfg_json["n_layer"], n_head=cfg_json["n_head"],
               n_embd=cfg_json["n_embd"], block_size=cfg_json["max_position"])
model = GPT(gc).eval()
model.load_state_dict(load_file(hf_hub_download("mbsdeepak/scribe", "model.safetensors")))

tok = Tokenizer()
ids = torch.tensor([tok.encode("Once upon a time")])
out = model.generate(ids, max_new_tokens=120, temperature=0.8, top_k=200)[0].tolist()
print(tok.decode(out))
```

Or serve it with [ember](https://github.com/mbsdeepak/ember) for a streaming, OpenAI-compatible API.

## Limitations

A 30M model on TinyStories writes simple, coherent children's-story English — not general-purpose
text. It's a learning/portfolio project: the goal is understanding an LLM end-to-end (architecture,
training, serving), not competing with production models. Context is 256 tokens; architecture is
vanilla GPT-2 (no RoPE/RMSNorm/SwiGLU).

## License

MIT