AslanZamaev/tau-krc-mt-v1

Karachay-Balkar ↔ Russian MT (NLLB-200-600M + DoRA)

Aslan Zamaev

A bidirectional Karachay-Balkar ↔ Russian translation model. It is a DoRA adapter on top of facebook/nllb-200-distilled-600M, trained on ~730k Karachay-Balkar sentence, phrase and word pairs.

Karachay-Balkar (krc, къарачай-малкъар тил) is a Kipchak Turkic language of the North Caucasus with roughly 300k speakers. It is not one of the 200 languages in NLLB.

chrF (sentences, KB↔RU, held-out)	56.41
Base model	facebook/nllb-200-distilled-600M
Adaptation	DoRA, r=64, α=64, dropout 0.2
Adapter size	133 MB
Training pairs	729,650 (1,445,188 direction-doubled rows)
Directions	krc→rus and rus→krc in one model

---

⚠️ Read this before running it

Two things about this model are non-obvious, and it will produce poor output if you skip them.

1. Karachay-Balkar rides the Bashkir language slot. NLLB has no krc token, so KB was trained into bak_Cyrl. Bashkir is the closest Kipchak relative available in the tokenizer, which is a large part of why 730k pairs are enough to get useful quality. Set src_lang / forced_bos_token_id to bak_Cyrl when you mean Karachay-Balkar.

2. Every source string must carry a direction tag prefix. Training used a literal text prefix on the source side. Reproduce it exactly, including the trailing space:

KB → RU :  "<2ru> "          + karachay_balkar_text
RU → KB :  "<2kb_balkar> "   + russian_text

These are plain text, not special tokens. Omitting them degrades output noticeably.

---

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

BASE = "facebook/nllb-200-distilled-600M"
ADAPTER = "AslanZamaev/tau-krc-mt-v1"   # this repo
RUS, KBB = "rus_Cyrl", "bak_Cyrl"   # KB uses the Bashkir slot

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForSeq2SeqLM.from_pretrained(BASE)
model = PeftModel.from_pretrained(model, ADAPTER).merge_and_unload().eval()
if torch.cuda.is_available():
    model.cuda()

def translate(text, direction="kb-ru", beams=4, maxlen=384):
    if direction == "kb-ru":
        tok.src_lang, tgt, tagged = KBB, RUS, f"<2ru> {text}"
    else:
        tok.src_lang, tgt, tagged = RUS, KBB, f"<2kb_balkar> {text}"
    enc = tok(tagged, return_tensors="pt", truncation=True, max_length=maxlen)
    enc = {k: v.to(model.device) for k, v in enc.items()}
    with torch.no_grad():
        out = model.generate(**enc,
                             forced_bos_token_id=tok.convert_tokens_to_ids(tgt),
                             max_length=maxlen, num_beams=beams,
                             no_repeat_ngram_size=3, early_stopping=True)
    return tok.batch_decode(out, skip_special_tokens=True)[0].strip()

print(translate("Мен школгъа барама", "kb-ru"))
print(translate("Как тебя зовут?",   "ru-kb"))

Recommended generation settings

Setting	Value	Why
num_beams	4–5	Evaluation used 2 for speed; 4–5 is better for real use
max_length	384	Training used 128 (96.8% of pairs are ≤64 tokens), but a 128-token inference cap silently truncates long inputs
no_repeat_ngram_size	3	Suppresses the usual seq2seq loop on long inputs

The model saw very few targets longer than 128 tokens, so very long outputs may still be weak — but with max_length=384 they will at least not be cut off.

Detecting the direction automatically

If you need to auto-detect which language you were handed, the reliable signal is the presence of the digraphs къ or гъ — these do not occur in Russian orthography. Do not use дж as a signal: Russian has it in loanwords (джентльмен, бюджет, Джордж), and using it as a KB marker routes Russian input into KB→RU, which silently returns a paraphrase of the input rather than a translation.

---

Orthography and dialect

Written Karachay and Balkar differ essentially in one thing: the phoneme /dʒ/ is spelled дж in Karachay and ж in Balkar. The training corpus was normalised to the ж form, with one guard: дж → ж was applied only when дж was absent from the Russian side of the pair, so Russian loanwords keeping дж were not corrupted.

The dative pronouns манга / санга / анга are normative for both varieties and are what the model produces. меннге / сеннге is a tolerated Karachay dialect form (~1% of Balkar literature, 30–40% of Karachay) and was normalised to манга / санга with word-boundary anchoring — an unanchored regex corrupts ordinary datives such as тирменнге (to the mill) or экзаменнге (to the exam). анга is never rewritten: its stem has no final -н, so аннге cannot exist.

The tsokayushchiy (ц/дз/з, Cherek) variety is spoken but written as standard, so it is effectively absent from text and from this model.

---

Training data

729,650 pairs:

Tier	Pairs	Share of training rows
Sentences	480,858	66.1%
Words	136,645	18.6%
Phrases	112,147	15.2%

Sources: digitised Karachay-Balkar books and literature, a merged lexicon built from 12 dictionaries, and 171 issues (one year, 2007) of the newspaper Заман.

Provenance disclosure. The Karachay-Balkar side of every pair is human-authored — this was a hard project rule, and no synthetic KB was ever admitted into training. A substantial share of the Russian side of the sentence tier is machine-generated (LLM translation of the KB source) rather than a published human translation. Treat Russian output fluency as better attested than Russian output fidelity on rare or literary material.

Gates applied to every batch of data before it entered training: orthographic normalisation on both sides, exact deduplication, and a hard benchmark-leak check. Rejected rows were parked, never deleted.

---

Training recipe

base        facebook/nllb-200-distilled-600M
method      DoRA (PEFT), r=64, alpha=64, dropout=0.2
targets     q_proj k_proj v_proj out_proj fc1 fc2
epochs      4   (of 5 run — epoch 5 overfits, see below)
batch       32
lr          2e-4
max_length  128
seed        42
hardware    2x H100 SXM, ~11 h
final loss  0.90

The published checkpoint is step 180652 = epoch 4.

Held-out split: 2,981 sentence keys / 1,461 phrase keys / 684 word keys held out by Russian headword, with zero train↔test headword overlap. Reported chrF is sentence-only, both directions, on a 2,000-row subset of that holdout.

---

How the score got to ~56

The two score columns below come from different holdouts and are not comparable to each other: an early all-tier score (which mixes in single words, where chrF is inherently low) and the later sentence-only score. Compare within a column, not across.

Stage	Corpus	all-tier chrF	sentence chrF
v1 — first attempt	5,849 sentences	—	unusable
v10 — full fine-tune baseline	small	43.98	—
KB2 — corpus rebuild + DoRA	690,652 pairs	49.05	54.67
This model — + Заман 2007	729,650 pairs	—	56.41

What actually moved the number:

1. Corpus size. Going from 5,849 to 441,860 KB sentences was the whole ballgame. At v1 scale the model was not usable at any hyperparameter setting. The bottleneck was data, not capacity — 600M was nowhere near saturated.
2. DoRA over OFT. Head-to-head at this data scale, DoRA beat OFT by 3.2 chrF. OFT was theoretically attractive (it better preserves the pretrained Turkic manifold) but lost on effective capacity.
3. Domain diversity, not just volume. Adding one year of newspapers — 38,998 sentences, an 8.8% increase — bought +1.74 chrF. The model's remaining failures were concentrated exactly where books have no coverage: news register, place names, modern vocabulary. This was the single best return on effort in the project.
4. Four epochs, not five. Both training arms peaked at epoch 4 and regressed at epoch 5.

What was tried and did not help

• Generated morphological forms as bulk training data. A validated KB morphological generator (checked cell-by-cell against the apertium-krc FST, 99.17% agreement on 26,396 cells) produced 674k inflected KB/RU word pairs, aligned by shared case. Adding them nearly doubled the corpus and changed sentence chrF by −0.05 — noise. This was run as a controlled A/B against the present model on an identical held-out set. This release is the arm without them. The generator remains useful as an independent output validator; it is not useful as training data.
• Raising training max_length from 128 to 256. 0.1% of pairs exceed 128 tokens.
• Turkish-pretrained models as a base. Turkish is Oghuz, Karachay-Balkar is Kipchak; the documented failure mode is Turkish interference and hallucination.
• Adding English to the main training mix. Dilution. For English, pivot at inference through Russian with a separate model instead.

A methodological note

chrF misled this project three times: a 47.29 that corresponded to poor output; an evaluation bug that split sentence references on commas and scored against the best-matching fragment, systematically penalising the more complete model; and a 256-row smoke checkpoint that scored garbage and was mislabelled as epoch 1. Every headline number here was re-derived after those fixes, and every model decision was confirmed by reading actual output. If you evaluate this model, read the output.

---

Limitations

• Not comparable to a high-resource MT system. ~56 chrF on a low-resource pair is a usable draft, not a publication-grade translation.
• Weakest on the word tier (isolated lexicon lookups) and on very long inputs.
• Trained overwhelmingly on literary and news prose. Conversational, technical, medical and legal registers are out of distribution.
• Russian-side fidelity is limited by the partly machine-generated Russian in the corpus (see Training data).
• The tsokayushchiy (Cherek) spoken variety is not represented.
• No safety or toxicity filtering beyond what facebook/nllb-200-distilled-600M carries.

License

CC-BY-NC-4.0, inherited from facebook/nllb-200-distilled-600M. Non-commercial use only. An adapter cannot be more permissive than its base.

Citation

@misc{zamaev2026krcru,
  title        = {Karachay-Balkar to Russian machine translation with NLLB-200 and DoRA},
  author       = {Zamaev, Aslan},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/AslanZamaev/tau-krc-mt-v1}},
  note         = {DoRA adapter on facebook/nllb-200-distilled-600M, 729,650 KB-RU pairs}
}

Contact

Corrections from Karachay-Balkar speakers are the most useful feedback this model can get — especially cases where the output is fluent but wrong. Open a discussion on this repo, or reach Aslan Zamaev through the links on his profile.

The training corpus, the extraction pipeline and the morphological generator are not published. Only the model weights are released here.

Built on NLLB-200 (NLLB Team et al., 2022). Morphological validation used apertium-krc.