## Model Description
MzModelV1 is a decoder-only Transformer language model built from scratch using PyTorch and Hugging Face Transformers. It was created as an experiment to explore language model training on consumer hardware in a home lab environment.
The project demonstrates that it is possible to train a language model with approximately 110 million parameters on a single consumer GPU.
## Model Details
| Property | Value |
|---|---|
| Model name | MzModelV1 |
| Architecture | Decoder-only Transformer |
| Model type | `mz_model_v1` |
| Parameters | 109.65 million |
| Hidden size | 768 |
| Number of layers | 12 |
| Attention heads | 12 |
| Feed-forward dimension | 3,072 |
| Vocabulary size | 32,000 |
| Maximum sequence length | 512 tokens |
| Positional encoding | Learned positional embeddings |
| Attention mechanism | Causal multi-head self-attention |
| Activation function | GELU |
| Normalization | LayerNorm |
| Weight tying | Input embeddings and output projection |
| Traini…
