Til-1B-multilingual-base

A 941M-parameter, from-scratch multilingual base language model with a deliberately token-balanced pretraining corpus. Kazakh · Russian · English · Code · Math — each language contributes an equal share of the training tokens, so no single language dominates the model's capacity.

This is a base (foundation) model — it is not instruction-tuned. It is the 1B rung of the TilQazyna scaling ladder (0.5B → 1B → 2B), trained on the same token-balanced corpus as its 0.5B sibling, and serves as the multilingual base for downstream Kazakh task models (e.g. grammatical error correction).

---

Why "token-balanced"?

Kazakh documents are short (350 tokens), while code and math documents are long (3000 tokens). Balancing a corpus by document count therefore leaves it badly token-imbalanced — one language can silently consume 5–6× the model's training budget of another. This model instead balances by tokens: every language receives the same token budget (~5B), set equal to the total available high-quality Kazakh (so all Kazakh data is used, none wasted).

The payoff is visible in the evaluation below: bits-per-byte (BPB) is low and even across all five domains, instead of excellent on Kazakh and poor everywhere else.

---

Key results — bits-per-byte (BPB, lower is better)

Evaluated on a frozen held-out set (kk 600, ru 322, en 290, math 274, code 179 texts), same tokenizer, same eval for all models.

Model	ru	kk	en	code	math	macro
Til-1B-multilingual-base (this model)	0.43	0.54	0.67	0.66	0.68	0.597
Til-0.5B-multilingual-base (0.5B sibling, same corpus)	0.45	0.55	0.70	0.70	0.72	0.624
Kazakh-first base (same tokenizer, kk-dominant corpus)	0.50	0.52	1.52	2.07	1.77	1.275

• Scaling helps uniformly. Going 0.5B → 1B on the same balanced corpus lowers macro-BPB by 0.027 (0.624 → 0.597) and improves every domain — ru −0.02, kk −0.01, en −0.03, code −0.04, math −0.04.
• Balanced and low across every language — the per-language spread is only 0.25 BPB (0.43 → 0.68).
• ~2× better macro-BPB than a Kazakh-first model on the same tokenizer (0.597 vs 1.275): balancing buys a 2–3× improvement on English, code and math for only a marginal Kazakh trade-off.

---

Model details

Architecture	DeepseekV3ForCausalLM (dense decoder-only + Multi-head Latent Attention)
Parameters	941.1 M
Hidden size	1536
Layers	24
Attention heads	12
MLA	q_lora_rank 256, kv_lora_rank 192, qk_nope/qk_rope/v_head = 64 / 32 / 64
Intermediate size	6144 (SiLU)
Context length	4096
Positional encoding	RoPE, θ = 100 000
Tie embeddings	yes
Vocabulary	131 072 (Til-Tokenizer-128k)
Optimizer	MUON (matrix params) + AdamW (rest)
Precision	bfloat16

Training data

Total 24.96 B tokens, balanced by token count across five domains (identical corpus to the 0.5B sibling):

Domain	Tokens	Share
English	5.00 B	20.0 %
Russian	5.03 B	20.1 %
Code	5.00 B	20.0 %
Math	5.00 B	20.0 %
Kazakh (incl. digitized KazNEB books)	4.93 B	19.8 %

All documents are quality-scored (q1–q5); only q≥4 is kept, and Kazakh additionally requires a native-fluency check to exclude calque/translationese. Kazakh uses all available high-quality data, which sets the per-language budget the other languages are capped to.

Curriculum (NOSHUF). Training is ordered, not shuffled: lower-quality-but-plentiful q4 data fills the early stable-LR phase, and the premium q5 data is placed at the end, in the learning-rate decay phase, so the highest-quality data is imprinted last at low LR.

Training procedure

• Hardware: 8 × NVIDIA H200 (80 GB), manual DDP launcher (NCCL_IB_DISABLE=1).
• Schedule: 1 epoch over the 24.96 B-token corpus, 47 607 steps, WSD-style LR (warmup → stable → decay) spanning the full run; ~23.5 h wall-clock.
• Objective: standard next-token cross-entropy.
• Final training loss ≈ 1.8.

Note on scale. This is a single-epoch 1B research checkpoint, the second rung of a scaling ladder (0.5B → 1B → 2B) built to measure how the token-balanced corpus scales. It is trained on the exact same corpus and recipe as the 0.5B, so the two are directly comparable.

Intended use & limitations

• Intended: research on multilingual/Kazakh language modeling, a strong base for fine-tuning (Kazakh GEC, instruction tuning, domain adaptation), and BPB/perplexity studies.
• Not intended: direct use as a chat/assistant model — it is not instruction-tuned and will simply continue text.
• Limitations: 941M parameters and a single epoch; no safety/alignment tuning; may produce factually wrong or biased text. The 4096-token context limit applies.

How to use

Requires a transformers build with deepseek_v3 support.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "TilQazyna/Til-1B-multilingual-base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16).eval()

prompt = "Қазақстанның астанасы —"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=40, do_sample=True, temperature=0.8, top_p=0.95)
print(tok.decode(out[0], skip_special_tokens=True))

Evaluation details

BPB = negative log-likelihood in bits divided by the number of UTF-8 bytes of the text; it is tokenizer-agnostic and directly comparable across models. Numbers above use the frozen evaluation set described in Key results, max_length 1024.

Model family

Part of the Til family by TilQazyna. Related: Til-Tokenizer-128k, Til-0.5B-multilingual-base (0.5B sibling), Til-Core-1B (Kazakh-first), and the GEC lineage (*-GEC).

Citation

@misc{tilqazyna_til_1b_multilingual_base,
  title  = {Til-1B-multilingual-base: a token-balanced multilingual base language model},
  author = {TilQazyna},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/TilQazyna/Til-1B-multilingual-base}}
}