Add scribe: 30M from-scratch GPT (TinyStories), val 1.74

README.md

@@ -0,0 +1,92 @@
+---
+license: mit
+tags:
+  - gpt
+  - language-model
+  - text-generation
+  - tinystories
+  - pytorch
+  - from-scratch
+  - slm
+library_name: pytorch
+pipeline_tag: text-generation
+datasets:
+  - roneneldan/TinyStories
+---
+
+# scribe
+
+A **~30M-parameter small language model built from scratch in PyTorch** and trained on
+[TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories). It's a decoder-only GPT —
+hand-written multi-head causal attention, pre-LayerNorm blocks, GELU MLP, tied LM head — with no
+`transformers`/`nanoGPT` modelling code. The architecture is deliberately **GPT-2-compatible**, so
+these weights load directly into [`ember`](https://github.com/mbsdeepak/ember), a from-scratch
+LLM inference server.
+
+📦 **Code, training, and full write-up (incl. the maths):** https://github.com/mbsdeepak/scribe
+
+## What it does
+
+Give it a prompt and it continues it as a simple, coherent children's story:
+
+> **"Once upon a time"** → *there was a little girl named Lily. She loved to read magazines. One
+> day, she found a magazine in her mom's room... "Mommy, look at that magazine!" Lily said... Her
+> mom looked at the magazine and smiled. "That's very nice, Lily. But be careful with it, it might
+> break."*
+>
+> **"One day, a boy named Deepak"** → *went to the park every day. He saw a large tree and wanted to
+> climb it. He started to climb it, but it was too heavy for him. He tried to jump, but he fell.*
+
+## Model
+
+| | |
+| :-- | :-- |
+| Architecture | decoder-only transformer (GPT-2 family) |
+| Parameters | ~30.0M (19.3M of it the tied 50257-token embedding) |
+| Layers / heads / width | 6 / 6 / 384 |
+| Context length | 256 tokens |
+| Tokenizer | GPT-2 BPE (`tiktoken`) |
+| Training | TinyStories, AdamW + warmup/cosine, ~3.4h on Apple-Silicon MPS |
+| Validation loss | **1.74** (cross-entropy) |
+
+## Files
+
+- `model.safetensors` — the weights (fp32)
+- `config.json` — architecture config to rebuild the model
+
+## Usage
+
+```python
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+import json, torch
+
+# get the code: git clone https://github.com/mbsdeepak/scribe
+from config import GPTConfig
+from src.model import GPT
+from src.tokenizer import Tokenizer
+
+cfg_json = json.load(open(hf_hub_download("mbsdeepak/scribe", "config.json")))
+gc = GPTConfig(n_layer=cfg_json["n_layer"], n_head=cfg_json["n_head"],
+               n_embd=cfg_json["n_embd"], block_size=cfg_json["max_position"])
+model = GPT(gc).eval()
+model.load_state_dict(load_file(hf_hub_download("mbsdeepak/scribe", "model.safetensors")))
+
+tok = Tokenizer()
+ids = torch.tensor([tok.encode("Once upon a time")])
+out = model.generate(ids, max_new_tokens=120, temperature=0.8, top_k=200)[0].tolist()
+print(tok.decode(out))
+```
+
+Or serve it with [ember](https://github.com/mbsdeepak/ember) for a streaming, OpenAI-compatible API.
+
+## Limitations
+
+A 30M model on TinyStories writes simple, coherent children's-story English — not general-purpose
+text. It's a learning/portfolio project: the goal is understanding an LLM end-to-end (architecture,
+training, serving), not competing with production models. Context is 256 tokens; architecture is
+vanilla GPT-2 (no RoPE/RMSNorm/SwiGLU).
+
+## License
+
+MIT

config.json

@@ -0,0 +1,10 @@
+{
+  "name": "scribe",
+  "n_layer": 6,
+  "n_head": 6,
+  "n_kv_head": 6,
+  "n_embd": 384,
+  "vocab_size": 50257,
+  "max_position": 256,
+  "dtype": "float32"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5f9fe3effaf126382fb5a2d0d1e9454e662aa1959d55f96953e89ffa5a5924fa
+size 120184776