# Eather-500M — custom language model from scratch

A real, randomly initialized ~505M-parameter decoder-only language model, **not pretrained weights**. Includes tokenizer training, dataset conversion, pretraining, chat SFT, and inference.

## Architecture
- 26 transformer blocks; 1280 hidden width; 20 query heads / 5 KV heads (GQA)
- 3584 SwiGLU intermediate width; RoPE; RMSNorm; Flash/SDPA causal attention
- 32,000-token byte-level BPE vocabulary; tied embedding/output weights
- 2,048-token initial context. Context extension requires further training/testing.
- PyTorch implementation; compatible with ROCm PyTorch when properly installed

## Install (Linux)

Use Python 3.11+ in an environment with an appropriate GPU-enabled PyTorch build. On AMD RX 7800 XT, **install the ROCm PyTorch build appropriate to your exact ROCm, GPU, and OS** per official documentation first. Generic `pip install torch` does not guarantee ROCm support.

```bash
python -m venv .venv
source .venv/bin/activate
# Install your appropriate ROCm PyTorch build first.
pip install numpy tokenizers
python model.py --device meta
python -c "import torch; print('GPU available:',torch.cuda.is_available()); print('Backend:',torch.version.hip)"
```

The full `requirements.txt` is a conventional generic environment requirement list; it does not configure ROCm.

## 1. Train a tokenizer

Put licensed or self-created `.txt` files in `data/raw/` (a substantial corpus is necessary).

```bash
python tokenizer_train.py --input data/raw --output tokenizer.json
```

**Tokenizer training requires enough text to reach 32,000 learned tokens.** For very small toy data, either add more text or lower both `--vocab-size` and `config.json` `vocab_size` to the same value.

## 2. Pre-tokenize the text

```bash
python prepare.py --input data/raw --tokenizer tokenizer.json --output-dir data/pretrain --format text
```

`prepare.py` creates `train.bin` (uint32 token IDs) and `mask.bin` (uint8 loss mask), reading text in chunks rather than holding the entire corpus in RAM.

## 3. Pretrain from zero

```bash
python train.py --data-dir data/pretrain --out-dir runs/pretrain \
  --seq-len 256 --batch-size 1 --grad-accum 16 \
  --steps 20000 --lr 0.0003 --save-every 1000
```

Effective tokens/update = `seq-len * batch-size * grad-accum`. On 16 GB VRAM, start conservatively at 128–256 sequence length and 1 microbatch; gradient checkpointing defaults ON. Actual feasibility and speed depend on ROCm/PyTorch versions, available VRAM, and kernel support. If you get OOM, reduce sequence length and ensure other GPU apps are closed. Optimizer checkpoints are large. The 20,000 steps above are merely an example, *not* a promise of an intelligent model.

## 4. Chat fine-tuning

Training data format, one JSON object per line in `data/chat.jsonl`:

```json
{"messages":[{"role":"user","content":"Hi"},{"role":"assistant","content":"Hello!"}]}
```

```bash
python prepare.py --input data/chat.jsonl --tokenizer tokenizer.json --output-dir data/chat --format chat
python train.py --data-dir data/chat --out-dir runs/chat \
  --resume runs/pretrain/step_0020000.pt --fine-tune \
  --seq-len 256 --batch-size 1 --grad-accum 16 \
  --steps 2000 --lr 0.00002 --save-every 500
```

Assistant tokens (plus the assistant end-of-turn token) receive SFT loss; user/system tokens are masked. Avoid mixing unrelated conversations without clear boundaries in an SFT dataset; this minimal streaming approach concatenates conversations and can occasionally sample across boundaries.

## 5. Generate

```bash
python generate.py --checkpoint runs/chat/step_0002000.pt --tokenizer tokenizer.json \
  --prompt "Write a Python function to reverse a string" --chat
```

Or for raw pretraining checkpoints:

```bash
python generate.py --checkpoint runs/pretrain/step_0020000.pt --prompt "Once upon a time"
```

## Important limitations

- All weights start random; this download is **code, not a trained chatbot**. Reaching useful coding and conversational ability needs high-quality data, substantial compute, evaluation, and repeated tuning.
- A 500M model is fairly small for advanced coding. Start with broad text pretraining, then carefully curated Python/Luau and instruction/chat examples.
- This project is an educational/reference trainer, not a production-scale distributed training stack. It does not include KV caching, packing-aware attention, validation/perplexity curves, distributed training, quantization, or model exports.
- Increasing `max_seq_len` alone does not grant reliable longer context.
- Only train on material you have the rights to use.


## Unsloth Studio / Hugging Face compatibility

See [`UNSLOTH_STUDIO.md`](UNSLOTH_STUDIO.md) and run `export_for_unsloth.py` to create a **fresh randomly initialized** standard Llama-format model using your trained 32k Eather tokenizer. This is not a conversion of custom weights; Unsloth Studio training support still depends on your installed version.