| --- |
| license: mit |
| library_name: pytorch |
| tags: |
| - char-lm |
| - transformer |
| - kv-cache |
| - educational |
| datasets: |
| - tiny_shakespeare |
| pipeline_tag: text-generation |
| --- |
| |
| # tinygpt-shakespeare (kv-cache-throughput-bench) |
|
|
| A 3.307M-parameter char-level decoder-only transformer trained from scratch on |
| Tiny Shakespeare. It exists to drive the benchmarks in the |
| [kv-cache-throughput-bench](https://github.com/narinzar/kv-cache-throughput-bench) |
| repo: a from-scratch KV cache, a cached-vs-uncached throughput sweep, and an |
| eviction-policy study. |
|
|
| ## Architecture |
|
|
| - 4 layers, 4 heads, 256 embedding dim, block size 512, vocab 65 (characters) |
| - weight-tied token embeddings, pre-norm blocks, learned absolute positions |
|
|
| ## Training |
|
|
| - 2000 steps, batch 48, block 512, AdamW lr 3e-4 |
| - 134 s on a single RTX 5090 Laptop GPU (CUDA 12.8) |
| - final validation loss 1.543, perplexity 4.68 |
|
|
| ## Files |
|
|
| - `tinygpt.pt`: state dict plus the config and char->id map |
|
|
| ## Usage |
|
|
| Clone the repo and load with the provided code: |
|
|
| ```python |
| from src.train import load_model |
| model, stoi, itos = load_model("tinygpt.pt") |
| ``` |
|
|
| ## Intended use and limits |
|
|
| Educational. A char model this small produces locally-coherent but repetitive |
| text under greedy decoding. It is not meant for downstream language tasks. |
|
|
