| # Test-shakespeare |
|
|
| Trained with **transformer-toolkit**. |
|
|
| ## Architecture |
| | param | value | |
| |---|---| |
| | `vocab_size` | `32000` | |
| | `dim` | `512` | |
| | `n_layers` | `12` | |
| | `n_heads` | `8` | |
| | `max_seq` | `512` | |
| | `attn` | `gqa` | |
| | `n_kv_heads` | `2` | |
| | `latent_dim` | `64` | |
| | `ffn` | `swiglu` | |
| | `hidden_dim` | `1536` | |
| | `n_experts` | `8` | |
| | `top_k` | `2` | |
| | `moe_aux_weight` | `0.01` | |
| | `moe_capacity` | `1.0` | |
| | `moe_n_shared` | `2` | |
| | `moe_n_routed` | `6` | |
| | `norm` | `rmsnorm` | |
| | `eps` | `1e-06` | |
| | `pos_enc` | `rope` | |
| | `dropout` | `0.1` | |
| | `tie_weights` | `False` | |
|
|
| ## Metrics |
| | metric | value | |
| |---|---| |
| | `val_loss` | `3.949221694469452` | |
