| --- |
| language: |
| - en |
| license: mit |
| library_name: transformers |
| tags: |
| - gemma |
| - text-generation |
| - transformer |
| datasets: |
| - karpathy/tiny_shakespeare |
| metrics: |
| - cross_entropy |
| --- |
| |
| scaled down version of the **Gemma** architecture trained on the **Tiny Shakespeare** dataset. |
|
|
| ## Model |
|
|
| - **Architecture**: Gemma (Transformer Decoder) |
| - **Attention**: Multi Query Attention (MQA) |
| - **Hidden Size**: 768 |
| - **Number of Layers**: 12 |
| - **Number of Query Heads**: 2 |
| - **Number of KV Heads**: 1 |
| - **Sequence Length**: 128 (Block Size) |
| - **Vocabulary Size**: 65 (Character-level encoding) |
| - **Total Training Steps**: 3,500 |
|
|
| ## Architecture |
|
|
| 1. **RMSNorm** |
| 2. **GeGLU** |
| 3. **RoPE** |
| 4. **Embedding Scaling** |
|
|
| ## Usage |
|
|
| You can load this model directly using the Hugging Face `transformers` library: |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("parkneurals/Gemma") |
| |
| # Note: This model uses a custom character-level tokenizer. |
| # You can use the provided char_map.json for encoding/decoding. |
| ``` |
| ### This model has slow inference due to rotation matrix calcuation on every layer for each token(as I made it only for learning purposes; please bear if anyone using) |
