Instructions to use klokedm/Micka-65k with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use klokedm/Micka-65k with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("klokedm/Micka-65k", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Micka-65k
A 65,536-token byte-level BPE tokenizer for Slovenian-focused models, with English and Croatian support. The package includes a chat template and contains no model weights.
- Vocabulary: 65,408 ordinary tokens and 128 special tokens.
- Text handling: full byte coverage; preserves whitespace, case and Unicode composition. Digits are tokenized individually.
- Author: Marko Kokol / klokedm.
- License: CC-BY-4.0; see attribution.
Usage
pip install transformers jinja2
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("klokedm/Micka-65k")
text = "Slovenščina, English in hrvatski."
ids = tokenizer.encode(text, add_special_tokens=False)
assert tokenizer.decode(ids, clean_up_tokenization_spaces=False) == text
messages = [{"role": "user", "content": "Kaj pomeni beseda prijateljstvo?"}]
chat_ids = tokenizer.apply_chat_template(
messages, tokenize=True, add_generation_prompt=True
)
Raw encoding does not insert BOS/EOS. The chat template supports string-valued system, user, assistant and tool messages. See special_token_schema.json for control tokens and IDs.
Tested with Transformers 4.57.6 and 5.16.1. Using this tokenizer with an existing pretrained model requires embedding/output-head adaptation and further training because token IDs differ.
Training and evaluation
Trained on 198,223 records (515 MB of UTF-8 text), predominantly source-labelled Slovenian (73.94%) and English (18.73%), with Croatian, Bosnian, Serbian Latin and mixed Slovenian/code. Sources, revisions and contributions are listed in training_sources.json.
On FLORES-200 devtest (1,012 sentences per language), token counts compared with Gemma 3/4 were:
| Language | Token-count difference |
|---|---|
| Slovenian | −31.2% |
| Croatian | −15.3% |
| English | +11.6% |
All 24,276 evaluated texts reconstructed exactly, with zero unknown tokens. Compression is less efficient for many other languages, especially Greek and Cyrillic scripts. Token counts do not measure model quality. Full results and sampling details are in evaluation.json.
See training_receipt.json for build details, vocabulary_filter_audit.json for vocabulary filtering, and SHA256SUMS for file checksums.