Tiny-DGM-1.5M

Tiny-DGM-1.5M is a 1,500,100-parameter causal Transformer pretrained from scratch on a staged subset of FineWeb-Edu data. The run used an NVIDIA RTX 5060 Ti and PyTorch CUDA.

Training

• Data: HuggingFaceFW/fineweb-edu, sample-10BT, pretokenized with the GPT-2 BPE vocabulary
• Tokens available on the cluster: 2.7B
• Tokens consumed: approximately 81.9M
• Context length: 256 GPT-2 tokens
• Batch size: 64
• Optimizer: AdamW, betas (0.9, 0.95), weight decay 0.1
• Steps: 5,000
• Parameter count: 1,500,100

The Descendant Gradient Matching schedule used 1,000 steps at each learning rate:

0.003 → 0.001 → 0.0003 → 0.0001 → 0.00003

Validation loss was evaluated after every phase. The selected checkpoint is from step 4,000, with validation loss 5.886. The final step-5,000 validation loss was 5.890.

Architecture

• GPT-2 BPE vocabulary: 50,257 tokens
• Hidden size: 28
• Attention heads: 4
• Transformer blocks: 9
• Tied input/output embeddings
• Pre-layer-normalized causal self-attention

Loading

tiny-dgm-1p5m.pt is a PyTorch checkpoint containing model, config, and training metadata. The model uses a custom compact architecture and is not directly loadable with AutoModelForCausalLM without a wrapper.

import torch

checkpoint = torch.load("tiny-dgm-1p5m.pt", map_location="cpu")
print(checkpoint["config"])
print(checkpoint["params"], checkpoint["best_val_loss"])

Limitations

This is a small research model trained for a short run. Its validation loss and generations should not be compared directly with large language models. The LR=0.1 continuation was tested separately from the selected checkpoint; it initially spiked and plateaued near validation loss 6.34 after 2,000 steps, so it is not the published checkpoint.

Files

• tiny-dgm-1p5m.pt: selected PyTorch checkpoint
• metrics.json: run metrics
• sample.txt: sample generated during validation