---
license: apache-2.0
language: en
tags:
  - tiny
  - slm
  - small-language-model
  - from-scratch
  - gqa
  - swiglu
  - rope
  - training-script
pipeline_tag: text-generation
metrics:
  - accuracy
  - perplexity
---

# ram-18m Training Script

Training script for **ram-18m**: an 18,290,304-param LLaMA-style language model.

## Architecture

| Parameter | Value |
|-----------|-------|
| d_model | 384 |
| n_heads | 6 |
| n_kv_heads | 2 (GQA) |
| head_dim | 64 |
| n_layers | 7 |
| FFN | SwiGLU, 4x (1536) |
| Norm | RMSNorm |
| Positional | RoPE (θ=10000) |
| Vocab | 8192 (BPE) |
| Tied embed/head | yes |
| **Total params** | **18,290,304** |

## Default Training Config

- Data: FineWeb-Edu L3 (sample-100BT), ~2B tokens
- Optimizer: AdamW (β=0.9/0.95, wd=0)
- LR: 2e-4, cosine decay to 2e-5, warmup 200 steps
- Batch: 32, seq_len 512
- Steps: 12,207 (~2B tokens)
- Grad clip: 1.0

## Usage

```bash
# 1. Prepare data (trains BPE tokenizer + tokenizes 2B tokens)
python3 train_ram_18m.py --stage prepare

# 2. Train
python3 train_ram_18m.py --stage train

# 3. Or do both
python3 train_ram_18m.py --stage all

# 4. Eval a checkpoint
python3 train_ram_18m.py --stage eval --ckpt ckpt_step12207.pt
```

## Requirements

```
pip install torch transformers datasets numpy tokenizers
```

## Notes

- Requested by GGUFGuy in [model-requests #30](https://huggingface.co/spaces/Compactbot/model-requests/discussions/30)
- The model is NOT trained yet — this is the script only.
- GPU recommended (RTX 3090+ for reasonable speed); CPU works but is ~50x slower.
- Checkpoints saved every 500 steps to the script directory.