---
{}
---

(v2 train scripts : RoPE Postitional Encoding)
# SpiceeChat — Train From Scratch via Flash3 Kernels

A custom TinyGPT trained from scratch using a BPE tokenizer and causal LM pipeline.

## Model Details

| Property | Value |
|---|---|
| Architecture | TinyGPT (custom GPT-style) |
| Layers | 4 |
| Heads | 4 |
| Hidden size | 384 |
| Context length | 128 |
| Vocab size | 32,768 |
| Attention | Torch (T4 compatible) |

## Files

- `checkpoint_step_*.pt` — model weights
- `tokenizer/` — BPE tokenizer trained on the same data
- `config.json` — model hyperparameters

## Load

```python
import torch
import json
from tokenizers import Tokenizer

# Load tokenizer
tok = Tokenizer.from_file("tokenizer/tokenizer.json")

# Load model (requires train.py in same directory)
from train import TinyGPT, GPTConfig
cfg = GPTConfig(vocab_size=32768, ctx_len=128, n_layer=4, n_head=4, n_embd=384, attention_backend="torch")
model = TinyGPT(cfg)
ckpt = torch.load("latest.pt", map_location="cpu")
model.load_state_dict(ckpt["model"])
model.eval()
```