Til-0.5B-multilingual-base

A 470M-parameter, from-scratch multilingual base language model with a deliberately token-balanced pretraining corpus. Kazakh · Russian · English · Code · Math — each language contributes an equal share of the training tokens, so no single language dominates the model's capacity.

This is a base (foundation) model — it is not instruction-tuned. It is the multilingual foundation of the TilQazyna scaling ladder (0.5B → 1B → 2B) and the base for downstream Kazakh task models (e.g. grammatical error correction).

---

Why "token-balanced"?

Kazakh documents are short (350 tokens), while code and math documents are long (3000 tokens). Balancing a corpus by document count therefore leaves it badly token-imbalanced — one language can silently consume 5–6× the model's training budget of another. This model instead balances by tokens: every language receives the same token budget (~5B), set equal to the total available high-quality Kazakh (so all Kazakh data is used, none wasted).

The payoff is visible in the evaluation below: bits-per-byte (BPB) is low and even across all five domains, instead of excellent on Kazakh and poor everywhere else.

---

Key results — bits-per-byte (BPB, lower is better)

Evaluated on a frozen held-out set (kk 600, ru 322, en 290, math 274, code 179 texts), same tokenizer, same eval for both models.

Model	ru	kk	en	code	math	macro
Til-0.5B-multilingual-base (this model)	0.45	0.55	0.70	0.70	0.72	0.624
Kazakh-first 0.5B (same arch, kk-dominant corpus)	0.50	0.52	1.52	2.07	1.77	1.275

• Balanced and low across every language — the per-language spread is only 0.27 BPB (0.45 → 0.72).
• ~2× better macro-BPB than a Kazakh-first model of identical architecture (0.624 vs 1.275).
• Trade-off: Kazakh BPB is marginally higher (0.55 vs 0.52) because Kazakh is one-fifth of the corpus rather than the majority; in exchange English, code and math improve 2–3×. This is the intended effect of balancing.

---

Model details

Architecture	DeepseekV3ForCausalLM (dense decoder-only + Multi-head Latent Attention)
Parameters	470.1 M
Hidden size	1024
Layers	24
Attention heads	8
MLA	q_lora_rank 256, kv_lora_rank 192, qk_nope/qk_rope/v_head = 64 / 32 / 64
Intermediate size	4096 (SiLU)
Context length	4096
Positional encoding	RoPE, θ = 100 000
Tie embeddings	yes
Vocabulary	131 072 (Til-Tokenizer-128k)
Optimizer	MUON
Precision	bfloat16

Training data

Total 24.96 B tokens, balanced by token count across five domains:

Domain	Tokens	Share
English	5.00 B	20.0 %
Russian	5.03 B	20.1 %
Code	5.00 B	20.0 %
Math	5.00 B	20.0 %
Kazakh (incl. digitized KazNEB books)	4.93 B	19.8 %

All documents are quality-scored (q1–q5); only q≥4 is kept, and Kazakh additionally requires a native-fluency check to exclude calque/translationese. Kazakh uses all available high-quality data, which sets the per-language budget the other languages are capped to.

Curriculum (NOSHUF). Training is ordered, not shuffled: lower-quality-but-plentiful q4 data fills the early stable-LR phase, and the premium q5 data is placed at the end, in the learning-rate decay phase (~85 % of tokens), so the highest-quality data is imprinted last at low LR.

Training procedure

• Hardware: 8 × NVIDIA H200 (80 GB), manual DDP launcher (NCCL_IB_DISABLE=1).
• Schedule: 1 epoch over the 24.96 B-token corpus, ~47.6 k steps, WSD-style LR (warmup → stable → decay) spanning the full run.
• Objective: standard next-token cross-entropy.
• Final training loss ≈ 1.88.

Note on scale. This is a single-epoch 0.5B research checkpoint intended to validate the balanced corpus by the numbers before scaling. Larger siblings (1B, 2B) trained on the same corpus are planned.

Intended use & limitations

• Intended: research on multilingual/Kazakh language modeling, a strong base for fine-tuning (Kazakh GEC, instruction tuning, domain adaptation), and BPB/perplexity studies.
• Not intended: direct use as a chat/assistant model — it is not instruction-tuned and will simply continue text.
• Limitations: 0.5B parameters and a single epoch; no safety/alignment tuning; may produce factually wrong or biased text. The 4096-token context limit applies.

How to use

Requires a transformers build with deepseek_v3 support.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TilQazyna/Til-0.5B-multilingual-base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).eval()

prompt = "Қазақстанның астанасы —"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation details

BPB = negative log-likelihood in bits divided by the number of UTF-8 bytes of the text; it is tokenizer-agnostic and directly comparable across models. Numbers above use the frozen evaluation set described in Key results, max_length 1024.

Model family

Part of the Til family by TilQazyna. Related: Til-Tokenizer-128k, Til-Core-0.5B (Kazakh-first), and the GEC lineage (*-GEC).

Citation

@misc{tilqazyna_til_05b_multilingual_base,
  title  = {Til-0.5B-multilingual-base: a token-balanced multilingual base language model},
  author = {TilQazyna},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/TilQazyna/Til-0.5B-multilingual-base}}
}