icelandic-prior-151m

A 151.67M-parameter GPT trained from scratch on Icelandic text and nothing else. It is a plain next-token language model: no instruction tuning, no chat format, no alignment. Its job is to be a cheap Icelandic fluency prior, a model whose per-token probabilities say how natural a piece of Icelandic is, so it can score or rerank candidate text. It is not a knowledge base and not an assistant.

Developed by Hafsteinn Einarsson at the University of Iceland, as part of the TrustLLM project.

The weights ship as a plain PyTorch checkpoint (ckpt.pt) holding the tensors and a small model_args config, not a 🤗 Transformers model. Loading it takes a few lines of code (see below) together with the bespoke tokenizer shipped alongside.

The playground

This repo also carries a browser playground for the model, with four modes: inline continuation with ghost text, word-at-a-time diverse beam search, a per-token surprisal heat map, and a two-text comparison. It runs entirely on your own machine — there are no external requests and no webfonts, and nothing you type is sent anywhere.

It needs uv and about 610 MB of disk for the checkpoint:

hf download hafsteinn/icelandic-prior-151m --local-dir icelandic-prior-151m
cd icelandic-prior-151m && sh run.sh
# then open http://127.0.0.1:8008

git clone works too, as long as git-lfs is installed so ckpt.pt arrives as the real 610 MB file rather than a pointer stub; without it run.sh notices and downloads the weights instead.

The first launch is slow, because uv resolves torch and the checkpoint has to load. After that a continuation takes 100–300 ms on Apple MPS and scoring runs at roughly 1500 tokens/second. Options pass straight through to the server, e.g. sh run.sh --device cpu --port 9000. The four modes and every control in them are documented in playground/README.md.

What it does well

The model was trained only to predict the next token of Icelandic, and it learned Icelandic grammar from raw text in the process. On a grammar minimal-pair test, where the task is to rank a correct sentence above a minimally wrong one, it prefers the correct form about 90% of the time or better wherever the pair is a real grammaticality contrast: case government, adjective and participle agreement, V2 word order, and verb and preposition government.

Because it is a fluency judge rather than a rule engine, it can be misled when the wrong form is simply the more common one out of context. It also holds no facts: prompted with "The capital of Iceland is", it produces fluent Icelandic, not necessarily "Reykjavík".

Architecture

Parameters	151.67M
Layers	16
Attention heads	12
Embedding dimension	768
Context length	1024 tokens
Bias	none
Vocabulary	50,000
Weight precision	fp32 in the checkpoint (trained in bf16)

A standard decoder-only transformer in the GPT-2 mould, with learned positional embeddings and tied input and output embeddings.

Tokenizer

A bespoke 50,000-token byte-level BPE trained on Icelandic, shipped as tokenizer.json in the 🤗 tokenizers format. It is deliberately low-fertility for Icelandic. On a 30,000-line held-out sample it uses 1.23 tokens per word (4.89 characters per token), against 2.71 for Llama-3 and 3.11 for GPT-2 on the same text. That is roughly 2.2 times fewer tokens than Llama-3 per unit of Icelandic, which is why a small model can afford a 1024-token context and why training was cheap. The end-of-text token is id 0.

The efficiency is bought with a deliberate trade-off. Because the vocabulary is Icelandic-specific and shared with no other model, this tokenizer suits sequence-level scoring rather than token-level steering of a different model.

Training data

Icelandic only. The corpus is the CC-BY portion of the Icelandic Gigaword Corpus (IGC, Risamálheild), taken from the LLM-ready release IGC1_jsonl, CLARIN handle 334 (CC-BY, open access). This covers the CC-BY subcorpora of the IGC (social media, news, parliamentary speech, law, journals, Wikipedia), without the Twitter part that CLARIN cannot redistribute.

Source	IGC1 JSONL (CLARIN handle 334), CC-BY
Raw size	1,312,369,457 words
Documents kept	1,935,876 (0.16% dropped by language-ID filtering)
Tokenized (bespoke 50k)	1,698,981,066 training tokens (uint16)
Held-out validation	6,267,401 tokens (0.5%)
Split	leakage-safe, assigned per document by sha256(source, id) before any dedup

Training on Icelandic alone is a deliberate choice: a monolingual prior treats non-Icelandic text as unnatural, which is what a fluency judge should do. Language filtering was a heuristic pass. A fastText language-ID pass and cross-split MinHash near-deduplication are noted for future work.

Training

Objective	next-token cross-entropy
Epochs	4 (13,000 iterations of 524,288 tokens, about 6.82B tokens seen over the 1.70B-token corpus)
Hardware	one NVIDIA H200
Precision	bfloat16
Optimizer	AdamW (betas 0.9 and 0.95, weight decay 0.1)
Learning rate	6e-4 cosine-decayed to 6e-5, 300-step warmup
Batch	64 sequences of 1024 tokens with 8 gradient-accumulation steps (524,288 tokens per step)
Gradient clipping	1.0
Final validation loss	2.975

Four epochs was a deliberate bet drawn from data-constrained scaling, where repeating a corpus a few times is nearly as useful as fresh tokens up to about four passes. It held here: validation loss was still falling at four epochs (3.115 at two epochs, 2.975 at four), so the model had not saturated on this corpus.

Evaluation

On a 4,340-pair Icelandic grammar minimal-pair test the model ranks the correct sentence above the wrong one 80.3% of the time overall, and 85 to 100% on every category that poses a genuine grammaticality contrast (word order 99%, case government 94%, agreement 92%, framed noun-case inflection 83%). The two low categories, bare definiteness swaps and subjunctive mood, are limits of the test rather than demonstrated gaps in the model: a neutral frame cannot make one definiteness or one mood the only grammatical option, so neither can be posed as a fair contrast.

How to load

The checkpoint holds the weights and a model_args config. To instantiate it you need a GPT-2-style decoder-only model definition exposing GPT and GPTConfig classes, plus the tokenizers library.

import torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
from model import GPT, GPTConfig   # a GPT-2-style decoder-only definition

repo = "hafsteinn/icelandic-prior-151m"
ckpt = torch.load(hf_hub_download(repo, "ckpt.pt"), map_location="cpu", weights_only=True)

model = GPT(GPTConfig(**ckpt["model_args"]))
state = {k.removeprefix("_orig_mod."): v for k, v in ckpt["model"].items()}
model.load_state_dict(state)
model.eval()

tok = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))

# score the log-probability of a piece of Icelandic (higher means more natural)
import torch.nn.functional as F
def mean_logprob(text, eot=0):
    ids = [eot] + tok.encode(text).ids
    idx = torch.tensor(ids)[None]
    with torch.no_grad():
        logits, _ = model(idx, targets=idx)
    lp = F.log_softmax(logits[0].float(), -1)[:-1]
    return lp.gather(1, idx[0, 1:, None]).mean().item()

print(mean_logprob("Ísland er fallegt land."))

Intended use and limitations

The model is meant for sequence-level fluency scoring and reranking of Icelandic, and for research on small monolingual language models. It is a 151M model trained on a news-heavy and forum-heavy corpus. It is not factually reliable, not safety-tuned, and not an instruction or chat model. Its probabilities track what is common in the training text, so it can rate a frequent but wrong form above a rare but correct one. Generated text can carry the biases present in Icelandic web and news writing.

Citation and attribution

Developed by Hafsteinn Einarsson (University of Iceland) as part of the TrustLLM project (trustllm.eu). Released under CC-BY-4.0.

The training corpus, the CC-BY subset of the Icelandic Gigaword Corpus (CLARIN handle 334), is CC-BY and requires attribution. Please credit the IGC and CLARIN-IS when using this model or its outputs.