File size: 2,076 Bytes
2741fc9 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
language:
- en
license: mit
library_name: custom
tags:
- from-scratch
- storytelling
- creative-writing
- cpu-trained
- transformer
- pytorch
base_model: []
pipeline_tag: text-generation
---
# AetherStory
> A from-scratch, CPU-trained storyteller transformer. ~863,492 parameters.
AetherStory is a tiny decoder-only transformer (GPT-style) trained **entirely
on CPU** on the procedurally generated [AetherStory dataset](wincode/aetherstory-data).
It writes short fantasy fables given an opening prompt.
This is **not** a fine-tune of a larger model and **not** a wrapper around
`transformers` — every layer is implemented by hand in plain PyTorch.
## Architecture
```
token + position embeddings
|
+----------------+
| Transformer x4 |
| causal MHA |
| GELU FFN |
+----------------+
|
LayerNorm
|
tied output head
```
| hyperparameter | value |
|---|---|
| vocab size | 10000 |
| d_model | 128 |
| layers | 4 |
| heads | 4 |
| ffn dim | 512 |
| max seq len | 64 |
| parameters | 863,492 |
| tied embeddings| True |
## Training
Trained with AdamW (lr 3e-4, cosine schedule, warmup 200) for 4 epochs on a
4-core CPU. Best validation loss: **0.3103**, trained in unknown (recovered) on CPU.
## Usage
```python
# files needed next to this script:
# model.safetensors, config.json, tokenizer.json, modeling_aetherstory.py
from modeling_aetherstory import StoryTeller
teller = StoryTeller.from_dir(".")
print(teller("In the Glasslands there lived", max_tokens=100, temperature=0.9))
```
## Limitations
A ~2M-parameter model trained on synthetic fables will **not** produce
literature. It will produce charming, sometimes incoherent, fairy-tale-flavoured
text — which is the point. It is a demonstration that a small, fully
custom model can be trained, evaluated, and shipped end-to-end on commodity
hardware.
## License
MIT.
|