Tiny-DGM-1M

Tiny-DGM-1M is a 105,984-parameter character-level causal Transformer trained locally on Apple Silicon with PyTorch MPS.

Method

The model uses Descendant Gradient Matching (DGM), a staged learning-rate schedule. It was first trained for 100 steps at each rate:

1.0 → 0.1 → 0.01 → 0.001 → 0.0001

The best checkpoint was then continued with:

0.2 → 0.03 → 0.004

Each stage used 100 steps with AdamW. The final reported held-out loss in the published run was 0.515.

Model details

• Architecture: causal character-level Transformer
• Parameters: 105,984
• Embedding dimension: 64
• Attention heads: 4
• Transformer layers: 2
• Context length: 64 characters
• Vocabulary: 28 characters
• Hardware: Apple Silicon MPS

Limitations

This is a small research demonstration trained on a repetitive synthetic corpus. It is intended for experimentation with optimization schedules, not for general-purpose language generation. The results are from a single short run and have not been compared systematically with baseline schedules.

Files

tiny-dgm-1m.pt contains the PyTorch state dictionary, vocabulary, and model configuration.