| --- |
| license: unknown |
| language: |
| - en |
| tags: |
| - text-generation |
| - from-scratch |
| - transformer |
| - gpt |
| - numpy |
| - cupy |
| - tinystories |
| datasets: |
| - roneneldan/TinyStories |
| pipeline_tag: text-generation |
| --- |
| |
| # DGPT v1-base |
|
|
| **A small, from-scratch base language model. Not an instruction-tuned assistant.** |
|
|
| DGPT v1-base is a 13,049,856-parameter decoder-only Transformer trained from |
| scratch (manual forward pass, manual backward pass, manual AdamW — no |
| autograd, no PyTorch/JAX/TensorFlow) on the TinyStories dataset. It generates |
| short, simple, TinyStories-style children's narratives and nothing more. |
|
|
| **Do not expect:** instruction following, multi-turn conversation, reasoning, |
| factual world knowledge, or ChatGPT-comparable capability of any kind. This |
| model was never trained or tuned for any of those. |
|
|
| ## Model description |
|
|
| - **Model type:** decoder-only Transformer, Pre-LN, GELU (tanh approx), tied |
| token embedding / LM head (no output bias), learned positional embeddings. |
| - **Parameters:** 13,049,856 |
| - **Context length:** 256 tokens |
| - **Vocabulary:** 6,000 (locked byte-level BPE, `bpe_6000.json`) |
| - **Framework:** none — hand-implemented NumPy/CuPy. Every layer's backward |
| pass was independently verified against finite-difference gradient checks |
| before training. |
|
|
| ## Architecture |
|
|
| | param | value | |
| |---|---| |
| | vocab_size | 6000 | |
| | block_size | 256 | |
| | d_model | 384 | |
| | n_layer | 6 | |
| | n_head | 6 | |
| | head_dim | 64 | |
| | d_ff | 1536 | |
| | activation | GELU (tanh approx) | |
| | norm | Pre-LN | |
| | positions | learned | |
| | lm_head | tied to token embedding, no bias | |
|
|
| ## Training data |
|
|
| [TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) |
| (`TinyStoriesV2-GPT4-train.txt`): 2,717,495 synthetically generated (GPT-3.5/ |
| GPT-4) short stories using a deliberately small vocabulary |
| ([Eldan & Li, 2023](https://arxiv.org/abs/2305.07759)), tokenized to |
| 371,525,259 tokens. Licensed by its authors under CDLA-Sharing-1.0 — this |
| model card does not redistribute the dataset itself. |
|
|
| ## Training procedure |
|
|
| - **Optimizer:** manually implemented AdamW (lr, betas, weight decay applied |
| only to matrix params — biases/LayerNorm params excluded from decay). |
| - **Stage 2 (validation run):** 50k-story subset, 2000 steps, batch size 64, |
| LR 3e-4 with warmup, used to gate correctness before full training. |
| - **Full run:** resumed from the Stage 2 checkpoint, continued on the full |
| 371.5M-token corpus at LR 3e-5, to a **final checkpoint at step 5000**. |
| - **Tokenizer:** locked, pre-trained externally, never retrained during model |
| training. |
|
|
| ## Hardware |
|
|
| - 1x NVIDIA Tesla T4 (Turing, SM75, 16 GB VRAM), Kaggle. |
| - CuPy 14.0.1 as the GPU numerical execution backend (no autograd usage). |
| - Measured throughput: ~3,300–3,440 tokens/sec at batch size 64 (directly |
| measured, not extrapolated). |
|
|
| ## Intended use |
|
|
| - Educational reference for from-scratch Transformer implementation |
| (manual forward/backward/AdamW) at small scale. |
| - Generating short, TinyStories-style children's narratives from a prompt. |
| - Portfolio / ML-engineering demonstration. |
|
|
| ## Out-of-scope use |
|
|
| - Any production or consumer-facing assistant use case. |
| - Instruction following, chat, question answering, factual retrieval, |
| reasoning tasks, or code generation. |
| - Anything requiring broad world knowledge — the model's effective knowledge |
| is bounded by TinyStories' simplified vocabulary and narrative style. |
| - Any use that assumes safety alignment or content filtering — **none was |
| performed.** |
|
|
| ## Evaluation |
|
|
| From the training notebook (full-data run, step 5000): |
|
|
| - Train loss ≈ 3.3 |
| - Val loss ≈ 3.3–3.4 |
| - Val perplexity ≈ 27–29 |
|
|
| No held-out benchmark suite (e.g. downstream NLP tasks) was run — TinyStories |
| train/val loss and perplexity are the only reported metrics. Treat any |
| numbers as approximate; see the training notebook's step-by-step log for the |
| exact source values. |
|
|
| ## Known generation issues |
|
|
| - Occasional run-on or abruptly concatenated sentences (short stories |
| sometimes blend into the next without a clean boundary). |
| - Repetition of simple phrases/character names across generations. |
| - No factual grounding — names, objects, and events are generated freely and |
| are not to be treated as accurate about anything. |
| - Context is capped at 256 tokens; longer prompts are truncated from the |
| left before generation. |
|
|
| ## How to use |
|
|
| ```python |
| from src.generate import load_dgpt, generate_text |
| from src.tokenizer import BPETokenizer |
| |
| tok = BPETokenizer("tokenizer/bpe_6000.json") |
| model, _ = load_dgpt("model.npz") |
| |
| print(generate_text(model, tok, "Once upon a time", max_new_tokens=150)) |
| ``` |
|
|
| ## Licensing |
|
|
| License is marked `unknown` above deliberately. See this repository's main |
| `README.md` → "Licensing" for the full breakdown across code, weights, |
| tokenizer, and the TinyStories dataset — the weights and tokenizer do not |
| have an established license and none is invented here. |
|
|
