Add README with architecture details and usage

README.md

@@ -0,0 +1,75 @@
+---
+license: apache-2.0
+language: en
+tags:
+  - tiny
+  - slm
+  - small-language-model
+  - from-scratch
+  - gqa
+  - swiglu
+  - rope
+  - training-script
+pipeline_tag: text-generation
+metrics:
+  - accuracy
+  - perplexity
+---
+
+# ram-18m Training Script
+
+Training script for **ram-18m**: an 18,290,304-param LLaMA-style language model.
+
+## Architecture
+
+| Parameter | Value |
+|-----------|-------|
+| d_model | 384 |
+| n_heads | 6 |
+| n_kv_heads | 2 (GQA) |
+| head_dim | 64 |
+| n_layers | 7 |
+| FFN | SwiGLU, 4x (1536) |
+| Norm | RMSNorm |
+| Positional | RoPE (θ=10000) |
+| Vocab | 8192 (BPE) |
+| Tied embed/head | yes |
+| **Total params** | **18,290,304** |
+
+## Default Training Config
+
+- Data: FineWeb-Edu L3 (sample-100BT), ~2B tokens
+- Optimizer: AdamW (β=0.9/0.95, wd=0)
+- LR: 2e-4, cosine decay to 2e-5, warmup 200 steps
+- Batch: 32, seq_len 512
+- Steps: 12,207 (~2B tokens)
+- Grad clip: 1.0
+
+## Usage
+
+```bash
+# 1. Prepare data (trains BPE tokenizer + tokenizes 2B tokens)
+python3 train_ram_18m.py --stage prepare
+
+# 2. Train
+python3 train_ram_18m.py --stage train
+
+# 3. Or do both
+python3 train_ram_18m.py --stage all
+
+# 4. Eval a checkpoint
+python3 train_ram_18m.py --stage eval --ckpt ckpt_step12207.pt
+```
+
+## Requirements
+
+```
+pip install torch transformers datasets numpy tokenizers
+```
+
+## Notes
+
+- Requested by GGUFGuy in [model-requests #30](https://huggingface.co/spaces/Compactbot/model-requests/discussions/30)
+- The model is NOT trained yet — this is the script only.
+- GPU recommended (RTX 3090+ for reasonable speed); CPU works but is ~50x slower.
+- Checkpoints saved every 500 steps to the script directory.