---
language:
- tt
license: cc-by-nc-sa-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- tatar
- turkic
- qypchaq
- from-scratch
- literary
- base-model
---

# TATlit — a native literary Tatar base language model

TATlit is a 478M-parameter, from-scratch base language model for literary Tatar (ISO 639-3 `tat`), a Qypchaq Turkic language written in Cyrillic. It is trained without adapting any English- or Russian-dominant base, so that it produces native Tatar morphosyntax by default rather than acting as a translation layer over a dominant language. This is a base model, not instruction-tuned.

## Usage

```python
# pip install sentencepiece
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("yasalma/TATlit")
tok   = AutoTokenizer.from_pretrained("yasalma/TATlit", use_fast=False)
```

The tokenizer is SentencePiece-based; pass `use_fast=False` so tokenization matches training exactly (the auto-converted fast tokenizer differs on the leading token). The weights are a standard Llama-architecture checkpoint in safetensors.

## Model
- Llama-style decoder, 24 layers, hidden size 1280, 16 attention heads with 4-way grouped-query attention, SwiGLU, RoPE, RMSNorm, tied embeddings, context 4096.
- 48K SentencePiece Unigram tokenizer, Tatar fertility 4.2-4.5 chars/token, 1.5-3x more efficient than general tokenizers.
- 478M parameters, bf16.

## Training
Two stages. Stage 1 pretrains on a Cyrillic-Qypchaq pool (Tatar 43 / Kazakh 30 / Kyrgyz 16 / Bashkir 10 / smaller siblings ~1 percent) under target-capped sampling. Stage 2 specializes on pure, register-weighted literary Tatar with an anti-memorization regime that keeps verbatim reproduction low.

## Results

TATlit is compared against three Tatar-specialized baselines (Goldfish-tat 125M, Tweety-7B-tatar, mGPT-1.3B-tatar) and four frontier models from 31B to 120B (Gemma-4-31B, Llama-4-Scout, Qwen3-32B, gpt-oss-120b), each measured with the same harness per benchmark.

**Byte-per-byte on external public sets** (FLORES, BOUQuET, UD), lower is better.

| Model | Params | FLORES | BOUQuET | UD |
|---|---|---|---|---|
| **TATlit** | **478M** | **.740** | **.777** | **.502** |
| Tweety-7B | 7B | .757 | .800 | .544 |
| Gemma-4-31B | 31B | .772 | .939 | .691 |
| Goldfish-tat | 125M | .778 | .801 | **.502** |
| mGPT-1.3B | 1.3B | .950 | 1.186 | .880 |
| gpt-oss-120b | 120B | 1.092 | 1.164 | 1.099 |
| Qwen3-32B | 32B | 1.144 | 1.423 | 1.159 |
| Llama-4-Scout | 109B | 1.222 | 1.498 | 1.080 |

**Byte-per-byte on in-house held-out sets** (literary, periodical), lower is better.

| Model | Params | Held-out lit | Held-out per |
|---|---|---|---|
| **TATlit** | **478M** | **.732** | **.701** |
| Gemma-4-31B | 31B | .790 | .705 |
| Tweety-7B | 7B | .810 | .778 |
| Goldfish-tat | 125M | .929 | .874 |
| Llama-4-Scout | 109B | 1.084 | .995 |
| Qwen3-32B | 32B | 1.105 | .979 |
| mGPT-1.3B | 1.3B | 1.199 | 1.094 |
| gpt-oss-120b | 120B | 2.181 | 1.857 |

**TatBLiMP, morphological acceptability**, higher is better.

| Model | Params | acc | acc_norm |
|---|---|---|---|
| **TATlit** | **478M** | **.975** | **.958** |
| Goldfish-tat | 125M | .974 | **.958** |
| Tweety-7B | 7B | .956 | .915 |
| Gemma-4-31B | 31B | .924 | .839 |
| Llama-4-Scout | 109B | .889 | .806 |
| Qwen3-32B | 32B | .811 | .708 |
| gpt-oss-120b | 120B | .803 | .677 |
| mGPT-1.3B | 1.3B | .736 | .639 |

On the school-knowledge benchmark TUMLU-mini the model sits near chance (0.297 against a 0.25 random baseline), which is expected for a base model with no encyclopedic training.

## Limitations
Base model, no instruction following, no encyclopedic knowledge. Softest axis is the person/possessive system inside izafet.

## Authors
Ilshat Saetov, Dmitry Gaynullin.

## Data
The training corpus is withheld on stewardship grounds for a minoritized language's literary heritage; a full datasheet with per-document provenance is available on reasonable request. The tokenizer and the model weights are released.

## License
The model weights and the tokenizer are released under **CC BY-NC-SA 4.0**. [TatBLiMP](https://huggingface.co/datasets/yasalma/TatBLiMP) benchmark is released under **CC BY-NC 4.0**. The training corpus is not released.

## Citation

```bibtex
@misc{saetov2026tatlit,
  title  = {Thanks to the Siblings: Bootstrapping a Native Tatar Language Model from a Cyrillic Qypchaq Pool},
  author = {Saetov Ilshat, Gaynullin Dmitry},
  year   = {2026}
}
```