Qal xlarge

Qal is a small GPT-style language model for Tigrinya, trained from scratch. This is the xlarge tier (69.4M parameters), one of five sizes trained with the same data, tokenizer and recipe so they can be compared.

Built as part of the Building Qal series: https://farefaine.substack.com. Code: https://github.com/farefaine/qal.

Results

Split Loss Perplexity
Validation 2.786 16.2
Test 2.805 16.5

Model

Parameters 69,367,296
Layers 8
Heads 12
Embedding dim 768
Context length 128 tokens
Vocabulary 8,192 (BPE)
Position embeddings learned, absolute

Training

  • 120 epochs (385,116 steps). The first 20 epochs come from the 30-epoch run (revision epoch-20); training then continued from that checkpoint under a 120-epoch schedule, holding the learning rate until about epoch 96 and decaying it to the end
  • Data: a cleaned version of farefaine/tigrinya-pretraining, about 79M tokens per epoch
  • Batch 192 sequences of 128 tokens
  • AdamW (betas 0.9, 0.99), weight decay 0.1, dropout 0.1
  • Peak learning rate 0.0015, warmup-stable-decay schedule
  • BF16 mixed precision, single GPU

Revisions

Besides the final model on main, this repo keeps checkpoints from other points in training:

Revision What it is Validation loss
main Final model after 120 epochs 2.786
pre-decay Epoch 90 of the 120-epoch run, before the learning rate decays about 3.21
30ep-final Final model of the 30-epoch run, same recipe as the other tiers 2.935
epoch-20 Epoch 20 of the 30-epoch run; the 120-epoch run continued from here about 3.32

Most of the gain from 120 epochs comes in the decay: validation loss is about 3.20 when the decay starts and 2.79 at the end.

Load a revision with GPT.from_pretrained("farefaine/qal-xlarge", revision="pre-decay").

Usage

The weights load with the model class from the Qal repository:

import torch
from tokenizers import Tokenizer
from model.transformer import GPT  # from https://github.com/farefaine/qal

model = GPT.from_pretrained("farefaine/qal-xlarge").eval()
tokenizer = Tokenizer.from_pretrained("farefaine/qal-xlarge")

ids = tokenizer.encode("ሎሚ αŠ£α‹α‹© α‹°αˆ΅ α‹˜α‰₯ል").ids
x = torch.tensor([ids])
with torch.no_grad():
    for _ in range(40):
        logits = model(x[:, -128:])
        next_id = torch.multinomial(torch.softmax(logits[0, -1] / 0.8, dim=-1), 1)
        x = torch.cat([x, next_id.view(1, 1)], dim=1)
print(tokenizer.decode(x[0].tolist()))

Limitations

A small model trained on about 79M tokens of mostly news text. It produces fluent-looking Tigrinya but is not reliable for facts and has not been tuned to follow instructions. It inherits the biases of its training sources.

Data sources

The training text comes from public Tigrinya datasets. Please credit them:

Downloads last month
62
Safetensors
Model size
69.5M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train farefaine/qal-xlarge

Collection including farefaine/qal-xlarge