| language: en | |
| tags: | |
| - causal-lm | |
| - gqa | |
| - rope | |
| - swiglu | |
| license: apache-2.0 | |
| # Experimenting | |
| Small language model (9.9M parameters) trained from scratch. | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Layers | 12 | | |
| | Hidden size | 256 | | |
| | Intermediate size | 704 | | |
| | Attention heads | 4 (GQA kv=2) | | |
| | Max sequence length | 1024 | | |
| | Vocab size | 4096 | | |
| | Tied embeddings | True | | |
| | Total parameters | 9.902M | | |
| ## Training | |
| - Tokens seen: 2,979,215,382 | |
| - Val loss: 2.5222 | |
| - Val PPL: 12.46 | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| tokenizer = AutoTokenizer.from_pretrained("GODELEV/Experimenting") | |
| model = AutoModelForCausalLM.from_pretrained("GODELEV/Experimenting") | |
| inputs = tokenizer("Hello", return_tensors="pt") | |
| output = model.generate(**inputs, max_new_tokens=50) | |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) | |
| ``` | |