File size: 1,582 Bytes
d993048 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
language:
- hy
license: apache-2.0
tags:
- armenian
- gpt
- text-generation
- causal-lm
pipeline_tag: text-generation
---
# ArmGPT (110M)
A GPT language model trained on Armenian text data.
## Model Details
| Property | Value |
|----------|-------|
| Parameters | 109,923,072 (110M) |
| Architecture | Transformer (RMSNorm, SwiGLU, RoPE) |
| Layers | 12 |
| Heads | 12 |
| Embedding dim | 768 |
| Context length | 512 tokens |
| Vocab size | 16,000 |
| Tokenizer | BPE (SentencePiece) |
| Training steps | 24,000 |
## Training Data
Trained on Armenian text from multiple sources:
- Armenian Wikipedia
- CC-100 Armenian
- CulturaX Armenian
- OSCAR Armenian
- HPLT Armenian
- Glot500 Armenian
## Usage
```python
import torch
import json
from core.model import GPT
from core.bpe_tokenizer import BPETokenizer
# Load model
checkpoint = torch.load("model.pt", map_location="cpu")
with open("config.json") as f:
cfg = json.load(f)
tokenizer = BPETokenizer.load("tokenizer.json")
model = GPT(
vocab_size=cfg["vocab_size"],
n_layer=cfg["n_layer"],
n_head=cfg["n_head"],
n_embd=cfg["n_embd"],
block_size=cfg["block_size"],
dropout=0.0,
)
model.load_state_dict(checkpoint)
model.eval()
# Generate text
prompt = "Հայաստան"
ids = tokenizer.encode(prompt)
context = torch.tensor([ids], dtype=torch.long)
output = model.generate(context, max_new_tokens=200, temperature=0.8, top_k=40)
print(tokenizer.decode(output[0].tolist()))
```
## License
Apache 2.0
|