gpt-scratch-dual-t4

A ~100M decoder-only Transformer trained from scratch on Kaggle dual-T4: RoPE + RMSNorm + SwiGLU + Grouped-Query Attention, FP16 mixed precision, native DDP. Pretrained on FineWeb-Edu, then fine-tuned on the GSM8K + ARC train splits.

Files

• model_fp16.safetensors - fp16 weights (primary)
• model_int8.pt - INT8 dynamic-quantized (portable CPU inference)
• tokenizer.json - custom 32k byte-level BPE
• model.py, config.json - architecture + config

Usage

import json, torch
from tokenizers import Tokenizer
from model import GPT, GPTConfig
from safetensors.torch import load_file
cfg = GPTConfig(**json.load(open("config.json")))
m = GPT(cfg); m.load_state_dict(load_file("model_fp16.safetensors"), strict=False); m.eval()
tok = Tokenizer.from_file("tokenizer.json")
ids = torch.tensor([tok.encode("The history of AI").ids])
print(tok.decode(m.generate(ids, 60)[0].tolist()))