| language: en | |
| tags: | |
| - causal-lm | |
| - gqa | |
| - rope | |
| - swiglu | |
| license: apache-2.0 | |
| # Exp-4 | |
| Small language model (3.0M parameters) trained from scratch. | |
| ## Architecture | |
| | Property | Value | | |
| |---|---| | |
| | Layers | 5 | | |
| | Hidden size | 192 | | |
| | Intermediate size | 512 | | |
| | Attention heads | 6 (GQA kv=6) | | |
| | Max sequence length | 1024 | | |
| | Vocab size | 4096 | | |
| | Tied embeddings | True | | |
| | Total parameters | 3.000M | | |
| ## Training | |
| - Tokens seen: 15,605,382,144 | |
| - Val loss: 2.3172 | |
| - Val PPL: 10.15 | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| tokenizer = AutoTokenizer.from_pretrained("CyanMonkey/Exp-4") | |
| model = AutoModelForCausalLM.from_pretrained("CyanMonkey/Exp-4") | |
| inputs = tokenizer("Hello", return_tensors="pt") | |
| output = model.generate(**inputs, max_new_tokens=50) | |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) | |
| ``` | |