smjain/cuda2llm-trained

CUDA2LLM Trained Checkpoints

This repository contains checkpoints produced by the pure CUDA C implementation from cuda2llm. The code path uses hand-written CUDA kernels for the transformer forward pass, backward pass, AdamW update, checkpoint loading, and text generation.

What This Run Proves

• The chapter CUDA programs compile and pass their checks.
• The full GPT program compiles with CUDA 12.4.
• prepare_data, grad_check, train_gpt, and generate_gpt all run on an NVIDIA RTX A4000.
• A 5000-step training run completed and generated text from the final checkpoint.

Important Quality Note

The final sample is not coherent Shakespeare. It is a small character-level model trained on a small corpus with an intentionally explicit educational implementation. It learns character labels, line breaks, archaic cadence, and local word shapes, but it does not reliably learn plot, speaker intent, or sentence-level meaning.

The validation curve shows overfitting after about step 2000. The final checkpoint is useful as proof that the scheduled run completed; the step-2000 region is the better demo checkpoint by validation loss.

Loss Curve

step  1000 | train loss 1.6028 | val loss 1.7558
step  2000 | train loss 1.2770 | val loss 1.5911
step  3000 | train loss 1.1062 | val loss 1.6381
step  4000 | train loss 0.7658 | val loss 1.8825
step  5000 | train loss 0.7931 | val loss 2.0483

Final Sample Excerpt

ROMEO:
Ay not sustom's master and by earor-woo's love;
She damned life, lest have had some removed,
Lest in neither slander man, and might have
Because the moon his commission.

This is best described as Shakespeare-like texture with weak semantics.

Files

model.bin
vocab.bin
train.bin
val.bin
full_run.log
checkpoints/model-interim-20260706-011839.bin
checkpoints/model-interim-20260706-020538.bin
samples/interim_sample_20260706-011839.txt
samples/interim_sample_20260706-020538.txt

Final Artifact Hashes

75eb8311e2bdec22b0c0e40084fb39fb1ed190662d3cd51f26e9f4b37ee06691  model.bin
464eb51d66b5b643889064b82c75607e2d756df11ec960e5e6aaaf03c71734be  vocab.bin
a3f591a5c0dfa7f54a2ec0ab39c4561d6475f1af28264f38076932ca425d5037  train.bin
836220b36a012c19cd285816001700117edef4f8e1098130b234d4b8fa766d86  val.bin