---
language: en
tags:
  - gpt
  - language-model
  - gpu
  - cuda
  - ai-systems
  - pytorch
license: mit
---

# KernelGPT — GPU/AI Systems Performance

A GPT-style decoder-only transformer trained from scratch on GPU/AI systems performance engineering text.

## Model Specs

| Property | Value |
|----------|-------|
| Parameters | ~125M |
| Architecture | Decoder-only Transformer |
| Embedding dim | 768 |
| Attention heads | 12 |
| Layers | 8 |
| Context length | 512 tokens |
| Vocab size | 32,000 (SentencePiece BPE) |

## Training

| Setting | Value |
|---------|-------|
| Training steps | 162,000 |
| Val loss | 4.3889 |
| Optimizer | AdamW |
| Learning rate | 3e-4 (cosine decay) |
| Batch size | 1 (effective 4 with grad accum) |

## Training Data

- **FineWeb** (general web text)
- **arXiv papers** (cs.DC, cs.AR, cs.LG, cs.PF categories — GPU/AI/systems)
- **Wikipedia** (ML/systems filtered articles)
- **GPU-specific crawl** (NVIDIA docs, GitHub READMEs, arXiv abstracts)

Topics cover all 20 chapters of *AI Performance Engineering* including CUDA internals,
KV cache tuning, LLM inference, distributed training, and GPU cluster scaling.

## Usage

```python
import torch
import sentencepiece as spm
from huggingface_hub import hf_hub_download

# Download files
ckpt_path = hf_hub_download("saiakula/KernelGPT", "pytorch_model.pt")
tok_path  = hf_hub_download("saiakula/KernelGPT", "tokenizer.model")

# Load tokenizer
sp = spm.SentencePieceProcessor(model_file=tok_path)

# Load model
# (requires TinyGPT src — clone https://github.com/your-username/TinyGPT)
checkpoint = torch.load(ckpt_path, map_location="cpu")
```

## Acknowledgments

- Inspired by Andrej Karpathy's nanoGPT
- Training topics based on *AI Performance Engineering* by Chris Fregly