Kecal

A compact language model (~37.5M parameters) that writes natural, casual Czech chat messages: one person sending a few messages in a row to a friend, the way people actually type on Messenger, WhatsApp or Discord.

Kecal was created for the Chat cipher on encrypt.click, where it turns encrypted data into ordinary-looking cover chat messages and back. It is a specialized, deterministic model for that job, not a chatbot or a general-knowledge assistant.

to je pravda
tak jsem ti psal
abys vedela ze to vim
jinak bych ti mohl pomoct neco zjistit
treba ti muzu poslat odkaz na nejaky obchod
ale to jeste nevim
tak ti napisu az budu vedet
jinak jak se mas
uz dlouho jsme se nevideli

Model

Property	Value
Parameters	37.5 M
Architecture	Decoder-only transformer, 8 layers, 8 heads, width 512
Tokens	24,000 words (lowercase, without diacritics) plus <pad>, <eol>, <unk>, <turn>
Context	128 words

It is a standard GPT-2 compatible model with a word-level tokenizer. Words are stored folded (lowercase, no diacritics); vocab.json maps each one back to its most common written form. <eol> ends a message, <turn> marks a change of writer.

Usage

With the helper script in this repository, which keeps to one writer and restores diacritics:

from kecal import Kecal

kecal = Kecal.load('danielsebesta/kecal')
print(kecal.generate(60))

Or with standard transformers (output stays folded):

from transformers import pipeline

kecal = pipeline('text-generation', model='danielsebesta/kecal')
print(kecal('hele zitra', max_new_tokens=30, do_sample=True, top_k=40)[0]['generated_text'])

Training

1. Pretraining on full Czech subtitles (OpenSubtitles via OPUS, ~149 million lines, ~812 million words).
2. Fine-tuning on authentic conversational Czech:
   • CNC ORTOFON (informal spoken Czech transcripts).
   • CNC KSK (Corpus of Private Correspondence, letters, notes and personal messages).
   • Vystadial Telephony (real conversational telephone dialogues between young Czech speakers).
   • Facebook comments and Reddit posts from r/czech.
3. Targeted fine-tuning on synthetic casual bursts:
   • Hundreds of thousands of one-writer chat bursts generated across diverse everyday topics (errands, meetings, complaints, small talk).

Validation perplexity on held-out one-writer chat: 14.3 (down from 36 before fine-tuning). Steganographic encoding rate: ~3.56 bits per word with 100% bit-exact round-trip recovery.

Limitations

• Output has no punctuation or capital letters; diacritics follow the most common written form of each word.
• Built for natural-sounding casual filler text, not facts. It invents scenarios and can drift between topics.
• Trained on informal language, so it may occasionally produce colloquial slang.

Credits

Kecal builds on these excellent resources:

• ORTOFON & KSK - Corpora of informal spoken Czech and private correspondence, Institute of the Czech National Corpus, Faculty of Arts, Charles University, Prague (korpus.cz). Via CZLC/CNC_oral_ortofon and CZLC/CNC_KSK (CC BY-NC-SA 4.0).
• Vystadial Telephony - Transcripts of real conversational telephone speech, Charles University (LINDAT/CLARIAH-CZ). Via BrunoHays/vystadial-telephony-cs (CC BY-SA 3.0).
• Czech Facebook Sentiment Corpus - I. Habernal, T. Ptáček, J. Steinberger (2013), University of West Bohemia. Via CZLC/fb_sentiment_balanced (CC BY-NC-SA 4.0).
• Reddit-MultiGEC - Czech posts from r/czech in lang-uk/Reddit-MultiGEC (MIT).
• OpenSubtitles - P. Lison and J. Tiedemann (2016), OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles, via OPUS.
• Synthetic Chats - Large share of the fine-tuning mix; generated for this project with LLMs.

License

CC BY-NC-SA 4.0, following the licenses of the training data: attribution, non-commercial use, share alike.