dawidkielbasa/qielbas-tiny-9m-m1

Qielbas Tiny 9M M1

A 9,557,716-parameter English causal language model pretrained on 6.7B tokens.

Architecture

Five blocks: four GatedDeltaNet2 layers and one causal attention layer. Width 320, five heads of dimension 64, SwiGLU hidden size 1048, RMSNorm, RoPE attention, and tied input/output embeddings. Context length 256; BPE vocabulary 4096.

Training

Pretraining uses Muon for eligible hidden weight matrices and AdamW for embeddings and remaining parameters. Peak learning rate 0.001; seed 7. Seven late checkpoints were averaged in FP32, selected using held-out language loss.

Pretraining data	Tokens
FineWeb-Edu	2.7B
Cosmopedia v2	0.3B
FinePhrase FAQ	0.5B
FinePhrase Tutorial	0.5B
DCLM-Edu	1.0B
FinePDFs-Edu, English	1.2B
FineWiki, English	0.5B

ARC adaptation uses 3345 Easy/Challenge training questions for five epochs, with AdamW, peak learning rate 0.00005, and general-text replay. Epoch selection uses 867 held-out ARC validation questions with a general-language loss guard. ARC test questions are excluded from adaptation.

Benchmarks

Metric	Result
Tiny-ML Efficiency	80.1426
Raw Overall	71.8696
ARC-Easy test accuracy	48.2744%
BLiMP accuracy	75.3224%
WikiText-2 byte perplexity	2.90791
WikiText-2 token perplexity	25.87719

Measured on 2376 ARC-Easy questions and 67000 BLiMP pairs. The frozen Glint-1.3 protocol uses unnormalized answer likelihood and 256-token contexts. Efficiency uses the retained Tiny-ML normalization; these are reproduced results, not an official leaderboard evaluation. Full reports and predictions are included.

Usage

Native PyTorch loader; requires a BF16-capable CUDA GPU and the pinned dependencies.

pip install -r requirements.txt

import torch
from load_model import load_model

model, tokenizer = load_model(".")
ids = tokenizer.encode("The Earth revolves around").ids
inputs = torch.tensor([ids], device="cuda")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    next_token = model(inputs)[0, -1].argmax().item()
print(tokenizer.decode([next_token]))