Micka-65k

A 65,536-token byte-level BPE tokenizer for Slovenian-focused models, with English and Croatian support. The package includes a chat template and contains no model weights.

  • Vocabulary: 65,408 ordinary tokens and 128 special tokens.
  • Text handling: full byte coverage; preserves whitespace, case and Unicode composition. Digits are tokenized individually.
  • Author: Marko Kokol / klokedm.
  • License: CC-BY-4.0; see attribution.

Usage

pip install transformers jinja2
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("klokedm/Micka-65k")
text = "Slovenščina, English in hrvatski."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text

messages = [{"role": "user", "content": "Kaj pomeni beseda prijateljstvo?"}]
chat_ids = tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True
)

Raw encoding does not insert BOS/EOS. The chat template supports string-valued system, user, assistant and tool messages. See special_token_schema.json for control tokens and IDs.

Tested with Transformers 4.57.6 and 5.16.1. Using this tokenizer with an existing pretrained model requires embedding/output-head adaptation and further training because token IDs differ.

Training and evaluation

Trained on 198,223 records (515 MB of UTF-8 text), predominantly source-labelled Slovenian (73.94%) and English (18.73%), with Croatian, Bosnian, Serbian Latin and mixed Slovenian/code. Sources, revisions and contributions are listed in training_sources.json.

On FLORES-200 devtest (1,012 sentences per language), token counts compared with Gemma 3/4 were:

Language Token-count difference
Slovenian −31.2%
Croatian −15.3%
English +11.6%

All 24,276 evaluated texts reconstructed exactly, with zero unknown tokens. Compression is less efficient for many other languages, especially Greek and Cyrillic scripts. Token counts do not measure model quality. Full results and sampling details are in evaluation.json.

See training_receipt.json for build details, vocabulary_filter_audit.json for vocabulary filtering, and SHA256SUMS for file checksums.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train klokedm/Micka-65k