File size: 1,220 Bytes
aa2bdaa 4fffdf4 33c8c48 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 | ---
language:
- en
license: mit
library_name: transformers
tags:
- gemma
- text-generation
- transformer
datasets:
- karpathy/tiny_shakespeare
metrics:
- cross_entropy
---
scaled down version of the **Gemma** architecture trained on the **Tiny Shakespeare** dataset.
## Model
- **Architecture**: Gemma (Transformer Decoder)
- **Attention**: Multi Query Attention (MQA)
- **Hidden Size**: 768
- **Number of Layers**: 12
- **Number of Query Heads**: 2
- **Number of KV Heads**: 1
- **Sequence Length**: 128 (Block Size)
- **Vocabulary Size**: 65 (Character-level encoding)
- **Total Training Steps**: 3,500
## Architecture
1. **RMSNorm**
2. **GeGLU**
3. **RoPE**
4. **Embedding Scaling**
## Usage
You can load this model directly using the Hugging Face `transformers` library:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("parkneurals/Gemma")
# Note: This model uses a custom character-level tokenizer.
# You can use the provided char_map.json for encoding/decoding.
```
### This model has slow inference due to rotation matrix calcuation on every layer for each token(as I made it only for learning purposes; please bear if anyone using) |