| ---
|
| language:
|
| - hy
|
| license: apache-2.0
|
| tags:
|
| - armenian
|
| - gpt
|
| - text-generation
|
| - causal-lm
|
| pipeline_tag: text-generation
|
| ---
|
|
|
| # ArmGPT (110M)
|
|
|
| A GPT language model trained on Armenian text data.
|
|
|
| ## Model Details
|
|
|
| | Property | Value |
|
| |----------|-------|
|
| | Parameters | 109,923,072 (110M) |
|
| | Architecture | Transformer (RMSNorm, SwiGLU, RoPE) |
|
| | Layers | 12 |
|
| | Heads | 12 |
|
| | Embedding dim | 768 |
|
| | Context length | 512 tokens |
|
| | Vocab size | 16,000 |
|
| | Tokenizer | BPE (SentencePiece) |
|
| | Training steps | 24,000 |
|
|
|
| ## Training Data
|
|
|
| Trained on Armenian text from multiple sources:
|
| - Armenian Wikipedia
|
| - CC-100 Armenian
|
| - CulturaX Armenian
|
| - OSCAR Armenian
|
| - HPLT Armenian
|
| - Glot500 Armenian
|
|
|
| ## Usage
|
|
|
| ```python
|
| import torch
|
| import json
|
| from core.model import GPT
|
| from core.bpe_tokenizer import BPETokenizer
|
|
|
| # Load model
|
| checkpoint = torch.load("model.pt", map_location="cpu")
|
| with open("config.json") as f:
|
| cfg = json.load(f)
|
|
|
| tokenizer = BPETokenizer.load("tokenizer.json")
|
|
|
| model = GPT(
|
| vocab_size=cfg["vocab_size"],
|
| n_layer=cfg["n_layer"],
|
| n_head=cfg["n_head"],
|
| n_embd=cfg["n_embd"],
|
| block_size=cfg["block_size"],
|
| dropout=0.0,
|
| )
|
| model.load_state_dict(checkpoint)
|
| model.eval()
|
|
|
| # Generate text
|
| prompt = "Հայաստան"
|
| ids = tokenizer.encode(prompt)
|
| context = torch.tensor([ids], dtype=torch.long)
|
| output = model.generate(context, max_new_tokens=200, temperature=0.8, top_k=40)
|
| print(tokenizer.decode(output[0].tolist()))
|
| ```
|
|
|
| ## License
|
|
|
| Apache 2.0
|
|
|
