farefaine/tigrinya-pretraining
Viewer β’ Updated β’ 52.1k β’ 82
Qal is a small GPT-style language model for Tigrinya, trained from scratch. This is the large tier (55.2M parameters), one of five sizes trained with the same data, tokenizer and recipe so they can be compared.
Built as part of the Building Qal series: https://farefaine.substack.com. Code: https://github.com/farefaine/qal.
| Split | Loss | Perplexity |
|---|---|---|
| Validation | 3.016 | 20.4 |
| Test | 3.029 | 20.7 |
| Parameters | 55,196,160 |
| Layers | 6 |
| Heads | 12 |
| Embedding dim | 768 |
| Context length | 128 tokens |
| Vocabulary | 8,192 (BPE) |
| Position embeddings | learned, absolute |
The weights load with the model class from the Qal repository:
import torch
from tokenizers import Tokenizer
from model.transformer import GPT # from https://github.com/farefaine/qal
model = GPT.from_pretrained("farefaine/qal-large").eval()
tokenizer = Tokenizer.from_pretrained("farefaine/qal-large")
ids = tokenizer.encode("αα α£αα© α°α΅ αα₯α").ids
x = torch.tensor([ids])
with torch.no_grad():
for _ in range(40):
logits = model(x[:, -128:])
next_id = torch.multinomial(torch.softmax(logits[0, -1] / 0.8, dim=-1), 1)
x = torch.cat([x, next_id.view(1, 1)], dim=1)
print(tokenizer.decode(x[0].tolist()))
A small model trained on about 79M tokens of mostly news text. It produces fluent-looking Tigrinya but is not reliable for facts and has not been tuned to follow instructions. It inherits the biases of its training sources.
The training text comes from public Tigrinya datasets. Please credit them: