Qal xlarge
Qal is a small GPT-style language model for Tigrinya, trained from scratch. This is the xlarge tier (69.4M parameters), one of five sizes trained with the same data, tokenizer and recipe so they can be compared.
Built as part of the Building Qal series: https://farefaine.substack.com. Code: https://github.com/farefaine/qal.
Results
| Split | Loss | Perplexity |
|---|---|---|
| Validation | 2.786 | 16.2 |
| Test | 2.805 | 16.5 |
Model
| Parameters | 69,367,296 |
| Layers | 8 |
| Heads | 12 |
| Embedding dim | 768 |
| Context length | 128 tokens |
| Vocabulary | 8,192 (BPE) |
| Position embeddings | learned, absolute |
Training
- 120 epochs (385,116 steps). The first 20 epochs come from the 30-epoch run (revision
epoch-20); training then continued from that checkpoint under a 120-epoch schedule, holding the learning rate until about epoch 96 and decaying it to the end - Data: a cleaned version of farefaine/tigrinya-pretraining, about 79M tokens per epoch
- Batch 192 sequences of 128 tokens
- AdamW (betas 0.9, 0.99), weight decay 0.1, dropout 0.1
- Peak learning rate 0.0015, warmup-stable-decay schedule
- BF16 mixed precision, single GPU
Revisions
Besides the final model on main, this repo keeps checkpoints from other points in training:
| Revision | What it is | Validation loss |
|---|---|---|
main |
Final model after 120 epochs | 2.786 |
pre-decay |
Epoch 90 of the 120-epoch run, before the learning rate decays | about 3.21 |
30ep-final |
Final model of the 30-epoch run, same recipe as the other tiers | 2.935 |
epoch-20 |
Epoch 20 of the 30-epoch run; the 120-epoch run continued from here | about 3.32 |
Most of the gain from 120 epochs comes in the decay: validation loss is about 3.20 when the decay starts and 2.79 at the end.
Load a revision with GPT.from_pretrained("farefaine/qal-xlarge", revision="pre-decay").
Usage
The weights load with the model class from the Qal repository:
import torch
from tokenizers import Tokenizer
from model.transformer import GPT # from https://github.com/farefaine/qal
model = GPT.from_pretrained("farefaine/qal-xlarge").eval()
tokenizer = Tokenizer.from_pretrained("farefaine/qal-xlarge")
ids = tokenizer.encode("αα α£αα© α°α΅ αα₯α").ids
x = torch.tensor([ids])
with torch.no_grad():
for _ in range(40):
logits = model(x[:, -128:])
next_id = torch.multinomial(torch.softmax(logits[0, -1] / 0.8, dim=-1), 1)
x = torch.cat([x, next_id.view(1, 1)], dim=1)
print(tokenizer.decode(x[0].tolist()))
Limitations
A small model trained on about 79M tokens of mostly news text. It produces fluent-looking Tigrinya but is not reliable for facts and has not been tuned to follow instructions. It inherits the biases of its training sources.
Data sources
The training text comes from public Tigrinya datasets. Please credit them:
- TLMD: Gaim, Yang & Park (2021), https://zenodo.org/records/5139094
- Tigrinya NLP Corpus, SeReT Foundation Language Team, https://github.com/TigrinyaNLP/corpus
- TiQuAD: Gaim, Yang, Park & Park (2023), https://github.com/fgaim/TiQuAD
- Downloads last month
- 62