KnegisTokenizerV1

A compact 2,048-piece SentencePiece unigram tokenizer optimized for lowercase Swedish and English. It was built for speech transcripts, such as Swedish workplace conversation that mixes in English, and suits the output vocabulary of a speech recognition model.

• Swedish and English share one vocabulary: common words in both languages are single pieces, and rarer words split into a few subword pieces.
• Works on lowercase, normalized text. A matching normalizer is included.
• No special tokens besides <unk>.

Usage

import sys

import sentencepiece as spm
from huggingface_hub import snapshot_download

path = snapshot_download("Knegis/KnegisTokenizerV1")
sys.path.insert(0, path)
from knegis_normalize import normalize  # plain Python; needs the `regex` package

sp = spm.SentencePieceProcessor(model_file=f"{path}/tokenizer.model")
text = normalize("Vi ses på mötet i morgon kl. 14:30, okej?")
print(text)  # vi ses på mötet i morgon kl 14:30 okej
print(sp.encode(text, out_type=str))
# ['▁vi', '▁se', 's', '▁på', '▁mö', 't', 'et', '▁i', '▁morgon', '▁kl', '▁', '1', '4', ':', '3', '0', '▁okej']

Always run normalize first. It lowercases, applies Unicode NFC, unifies quotes and dashes, and removes punctuation except inside words, numbers, e-mail addresses, paths and URLs. The tokenizer applies no normalization of its own, so capital letters and any character outside its 85 supported characters become <unk>. For normalized text, sp.decode(sp.encode(text)) == text.

Performance

Tokens per word on held-out text that was not used for training, after normalization:

Text	Words	Tokens per word
Swedish	842,868	1.81
English	506,554	1.57
Swedish professional podcast transcripts (39 shows)	525,266	1.49

Unknown tokens are rare: 77 in 1.53 million Swedish tokens, and none in English. These are text measurements; speech recognition accuracy has not been measured.

Settings

Setting	Value
Model type	Unigram, 2,048 pieces including <unk> (id 0)
Special tokens	None besides <unk>: no BOS, EOS or padding ids
Characters	85: a–z, å ä ö and other Latin letters, digits and # $ % & ' + , - . / : @ \ _ ~ £ €; no byte fallback
Splitting	By whitespace, by Unicode script and into single digits
Maximum piece length	32 characters
Training text	16.8 million words, about half Swedish and half English

Trained with SentencePiece 0.2.2; the model loads with the standard sentencepiece package. train_config.json has the full training settings.

Limitations

• Lowercase only, and no byte fallback: unnormalized text produces <unk>.
• With 2,048 pieces, rarer words take several pieces, and numbers are always spelled digit by digit.
• English contractions take three tokens (let's → ▁let ' s).

Files

File	Contents
tokenizer.model	SentencePiece model (SHA-256 4f92b7ba3a29fc2aaa4a4b8449c1ad1fe35a8718025926e5fd5a981fe8cf5045)
tokenizer.vocab	The 2,048 pieces with their scores
knegis_normalize.py	The matching text normalizer, version 1.0.0
config.json	Summary: format, vocabulary size, files and normalizer version
train_config.json	Training settings

Licence

The tokenizer is released under CC BY-NC 4.0. The normalizer is MIT-licensed, Copyright (c) 2026 Knegis.