Qal small

Qal is a small GPT-style language model for Tigrinya, trained from scratch. This is the small tier (13.4M parameters), one of five sizes trained with the same data, tokenizer and recipe so they can be compared.

Built as part of the Building Qal series: https://farefaine.substack.com. Code: https://github.com/farefaine/qal.

Results

Split Loss Perplexity
Validation 3.481 32.5
Test 3.492 32.9

Model

Parameters 13,434,624
Layers 4
Heads 6
Embedding dim 384
Context length 128 tokens
Vocabulary 8,192 (BPE)
Position embeddings learned, absolute

Training

  • 30 epochs (96,420 steps), learning rate held for 80% of the run, then decayed
  • Data: a cleaned version of farefaine/tigrinya-pretraining, about 79M tokens per epoch
  • Batch 192 sequences of 128 tokens
  • AdamW (betas 0.9, 0.99), weight decay 0.1, dropout 0.1
  • Peak learning rate 0.003, warmup-stable-decay schedule
  • BF16 mixed precision, single GPU

Usage

The weights load with the model class from the Qal repository:

import torch
from tokenizers import Tokenizer
from model.transformer import GPT  # from https://github.com/farefaine/qal

model = GPT.from_pretrained("farefaine/qal-small").eval()
tokenizer = Tokenizer.from_pretrained("farefaine/qal-small")

ids = tokenizer.encode("ሎሚ αŠ£α‹α‹© α‹°αˆ΅ α‹˜α‰₯ል").ids
x = torch.tensor([ids])
with torch.no_grad():
    for _ in range(40):
        logits = model(x[:, -128:])
        next_id = torch.multinomial(torch.softmax(logits[0, -1] / 0.8, dim=-1), 1)
        x = torch.cat([x, next_id.view(1, 1)], dim=1)
print(tokenizer.decode(x[0].tolist()))

Limitations

A small model trained on about 79M tokens of mostly news text. It produces fluent-looking Tigrinya but is not reliable for facts and has not been tuned to follow instructions. It inherits the biases of its training sources.

Data sources

The training text comes from public Tigrinya datasets. Please credit them:

Downloads last month
41
Safetensors
Model size
13.5M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train farefaine/qal-small

Collection including farefaine/qal-small