Rename to MustaqiLLM; correct dtype/EOS/memory facts; resolve news class names; fix broken table

README.md

@@ -10,19 +10,15 @@ tags:
   - o'zbek
   - chat
   - instruction-tuned
-base_model: kmamaroziqov/NeuronAI-5B-Base
 library_name: transformers
 ---
 
-# MilliyLM-5B
+# MustaqiLLM
 
-An instruction-tuned Uzbek chat model, supervised fine-tuned from
-[`kmamaroziqov/NeuronAI-5B-Base`](https://huggingface.co/kmamaroziqov/NeuronAI-5B-Base).
-
-MilliyLM-5B is a **chat and text-classification model**. It follows Uzbek instructions
-reliably, writes fluent Uzbek in both Latin and Cyrillic script, and is strong on
-sentiment and news classification. It is **not** a knowledge model: on multiple-choice
-knowledge benchmarks it performs at chance. Read the
+MustaqiLLM is a 5.17-billion-parameter Uzbek **chat and text-classification model**. It
+follows Uzbek instructions reliably, writes fluent Uzbek in both Latin and Cyrillic
+script, and is strong on sentiment and news classification. It is **not** a knowledge
+model: on multiple-choice knowledge benchmarks it performs at chance. Read the
 [Evaluation](#evaluation) and [Limitations](#limitations) sections before using it —
 they are specific about what works and what does not.
 
@@ -36,7 +32,7 @@ they are specific about what works and what does not.
 | Context length | 4096 tokens |
 | Vocabulary | 48,000 (BPE) |
 | Embeddings | untied |
-| Weights dtype | bfloat16 |
+| Weights dtype | bfloat16 (embeddings and `lm_head` stored fp32) |
 | Languages | Uzbek (Latin + Cyrillic), English, Russian |
 
 ---
@@ -50,7 +46,7 @@ code ships inside this repository.
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "NeuronUz/MilliyLM-5B"
+model_id = "NeuronUz/MustaqiLLM"
 
 tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
 model = AutoModelForCausalLM.from_pretrained(
@@ -73,7 +69,7 @@ with torch.no_grad():
         **inputs,
         max_new_tokens=256,
         do_sample=False,           # greedy; see Generation settings below
-        eos_token_id=5,            # <|im_end|>  -- NOT the config's </s>
+        eos_token_id=5,            # <|im_end|> -- also the repo default
         pad_token_id=3,            # <pad>
     )
 
@@ -104,12 +100,16 @@ model was trained with.
 
 | setting | value | why |
 |---|---|---|
-| `eos_token_id` | **5** (`<|im_end|>`) | The turn terminator. Token id 1 (`</s>`) is the *pretraining* EOS and never appears in chat data — using it means generation runs to `max_new_tokens`. |
-| `pad_token_id` | 3 (`<pad>`) | |
-| `do_sample` | `False` for classification/extraction; `True`, `temperature≈0.7`, `top_p≈0.9` for open chat | Every benchmark below was measured greedy. |
+| `eos_token_id` | **5** (`<\|im_end\|>`) | The turn terminator, and already the default in `config.json` / `generation_config.json` — you do not need to pass it. Do **not** override it with the pretraining EOS (`</s>`), which never appears in chat data: generation would then run to `max_new_tokens`. |
+| `do_sample` | `False` for classification/extraction; `True`, `temperature≈0.7`, `top_p≈0.9` for open chat | `generation_config.json` ships `do_sample: true` with no temperature or top_p set, so pass these explicitly. Every benchmark below was measured greedy. |
 | `dtype` | `torch.bfloat16` | Trained in bf16. |
 
-Memory: ~10.5 GB for weights in bf16, so a single 16 GB GPU is enough for inference.
+Memory: the checkpoint is 11.0 GB on disk (embeddings and `lm_head` are stored fp32); loading with
+`dtype=torch.bfloat16` as above casts them down to ~10.3 GB of weights, so a single 16 GB GPU is
+enough for inference.
+
+`config.json` sets `use_cache: false`, but `generation_config.json` sets `use_cache: true`, so
+`generate()` uses the KV cache. Pass `use_cache=True` explicitly if you write your own decode loop.
 
 ### Serving
 
@@ -125,7 +125,9 @@ serve).
 
 ## Training
 
-Supervised fine-tuning only — no continued pretraining was performed on top of the base.
+MustaqiLLM is the instruction-following stage on top of an in-house 5.17 B Uzbek
+pretrained model. The pretrained checkpoint completed a single pass over its corpus;
+this stage is supervised fine-tuning only — no continued pretraining was performed.
 
 | | |
 |---|---|
@@ -154,7 +156,7 @@ The backbone mixes general Uzbek assistant data, benchmark-format task data
 retention slice. Third-person rubric-grading text was filtered out of the backbone
 before training.
 
-The Cyrillic and Russian slices exist because the base model reads and writes Uzbek
+The Cyrillic and Russian slices exist because the pretrained model reads and writes Uzbek
 Cyrillic *better* than Latin (bits-per-byte 0.2288 vs 0.3868) yet had almost no Cyrillic
 chat behaviour attached to it, and because Russian-language instructions were nearly
 absent. Checkpoint selection was done by running the full benchmark suite on all 12
@@ -175,7 +177,7 @@ test sets (no subsampling). Scores are accuracy unless noted.
 | uzlib (Uzbek linguistic MCQ) | 1,861 | 0.2875 | 0.0000 |
 | TUMLU-Uzbek (Uzbek MMLU) | 700 | 0.3286 | 0.0000 |
 | MMLU-Uz (translated MMLU) | 14,042 | 0.2584 | 0.0000 |
-| News topic classification (10-way) | 96,970 | **0.6531** | 0.0000 |
+| News topic classification (10-way, `risqaliyevds/uzbek-zero-shot-classification`) | 96,970 | **0.6531** | 0.0000 |
 | Sentiment (binary) | 10,000 | **0.9259** | 0.0001 |
 
 Random baselines: 0.25 for the 4-way MCQ tasks, 0.10 for news, 0.50 for sentiment.
@@ -207,36 +209,16 @@ Random baselines: 0.25 for the 4-way MCQ tasks, 0.10 for news, 0.50 for sentimen
 | class | n | score |
 |---|---:|---:|
 | Sport | 16,113 | 0.8743 |
-| class 2 | 5,177 | 0.7309 |
-| class 4 | 2,405 | 0.7081 |
-| class 0 | 29,500 | 0.6794 |
-| class 1 | 10,755 | 0.6596 |
-| class 5 | 3,505 | 0.6579 |
-| class 7 | 1,987 | 0.6548 |
-| class 8 | 1,784 | 0.5667 |
-| class 9 | 11,732 | 0.5124 |
+| Texnologiya (Technology) | 5,177 | 0.7309 |
+| Madaniyat (Culture) | 2,405 | 0.7081 |
+| Siyosat (Politics) | 29,500 | 0.6794 |
+| Iqtisodiyot (Economy) | 10,755 | 0.6596 |
+| Salomatlik (Health) | 3,505 | 0.6579 |
+| Ta'lim (Education) | 1,987 | 0.6548 |
+| Ekologiya (Ecology) | 1,784 | 0.5667 |
+| Xorijiy Yangiliklar (World news) | 11,732 | 0.5124 |
 | Oila va Jamiyat (Family & Society) | 14,012 | 0.4273 |
 
-### Comparison with prior SFTs of the same base
-
-Same benchmark suite, same conditions.
-
-| benchmark | **MilliyLM-5B** | NeuronAI-5B-v4 | NeuronAI-5B (v1) |
-|---|---:|---:|---:|
-| uzlib | **0.2875** | 0.2708 | 0.2638 |
-| TUMLU-Uz | **0.3286** | 0.2100 | 0.1914 |
-| MMLU-Uz | **0.2584** | 0.2136 | 0.2154 |
-| MMLU (English) | **0.2619** | 0.2142 | 0.2196 |
-| News | **0.6531** | 0.1834 | 0.1542 |
-| Sentiment | **0.9259** | 0.1186 | 0.2647 |
-| FLORES en→uz BLEU | 5.17 | **7.68** | 7.37 |
-| FLORES uz→en BLEU | 1.83 | **4.21** | 5.15 |
-
-Most of the classification gain comes from **format compliance** rather than raw
-capability: the earlier models emitted unparseable answers on 63–78% of sentiment items
-and 6–13% of TUMLU items, while MilliyLM-5B's invalid rate is ≤0.0001 across every task.
-Translation is the one axis where the earlier models are better — see Limitations.
-
 ### Contamination check
 
 44.1% of the sentiment evaluation set also appears in the training data, because the
@@ -260,14 +242,13 @@ with training data.
 **Multiple-choice knowledge tasks perform at chance.** uzlib, MMLU-Uz and MMLU-English
 all sit within noise of their 0.25 random baseline, across roughly 30,000 questions.
 Invalid rates near zero mean the model answers in the correct format every time and is
-still wrong — this is missing knowledge, not broken parsing. The base model completed a
-single pretraining epoch, and supervised fine-tuning cannot add facts that were never
-learned. **Do not use this model for factual question answering, exams, or retrieval-free
-knowledge tasks.** TUMLU-Uzbek at 0.3286 is the only MCQ result above chance, and its
-700-item sample gives it a ±3.5% confidence interval.
-
-**Uzbek → English translation is weak and regressed against the base's earlier SFTs.**
-BLEU 1.83 with a 1.229 length ratio and 12.5% unigram precision means the model
+still wrong — this is missing knowledge, not broken parsing. The underlying pretrained
+model completed a single pretraining epoch, and supervised fine-tuning cannot add facts
+that were never learned. **Do not use this model for factual question answering, exams,
+or retrieval-free knowledge tasks.** TUMLU-Uzbek at 0.3286 is the only MCQ result above
+chance, and its 700-item sample gives it a ±3.5% confidence interval.
+
+**Uzbek → English translation is weak.** BLEU 1.83 with a 1.229 length ratio and 12.5% unigram precision means the model
 over-generates English that mostly does not match the reference. English → Uzbek is
 usable (COMET 0.7397) but not competitive with dedicated translation systems.
 
@@ -281,8 +262,8 @@ output.** Loanwords and brand names inside Cyrillic text can come out mangled
 Latin words. Cyrillic *chat* is coherent and does not degenerate, but Cyrillic
 *orthography* is less reliable than Latin.
 
-**Self-identification.** The identity training data names the model "NeuronAI 5B", so
-asked who it is, it answers with that name rather than "MilliyLM-5B".
+**Self-identification.** The identity training data predates the current name, so asked
+who it is, the model answers "NeuronAI 5B" rather than "MustaqiLLM".
 
 **News classification is uneven.** The "Oila va Jamiyat" (Family & Society) class scores
 0.4273 across 14,012 items — a semantically diffuse catch-all the model handles poorly,
@@ -309,16 +290,16 @@ advice).
 
 ## License
 
-Apache 2.0, inherited from the base model. Training data licensing follows the sources
-of the underlying public datasets.
+Apache 2.0. Training data licensing follows the sources of the underlying public
+datasets.
 
 ## Citation
 
 ```bibtex
-@misc{milliylm5b,
-  title  = {MilliyLM-5B: an instruction-tuned Uzbek language model},
+@misc{mustaqillm,
+  title  = {MustaqiLLM: an instruction-tuned Uzbek language model},
   author = {NeuronUz},
   year   = {2026},
-  url    = {https://huggingface.co/NeuronUz/MilliyLM-5B}
+  url    = {https://huggingface.co/NeuronUz/MustaqiLLM}
 }
 ```