Add tokenizer fertility and cross-model benchmark comparison

Tokenizer fertility on uz-crawl (2.41 vs 3.43 tokens/word against stock Qwen3.5-2B) and a full eight-task suite comparison against alloma-1B, alloma-3B, and stock Qwen3.5-2B, with all four runs rescored under one COMET-primary weighting. Also corrects the lineage step: the tokenizer is a retrofit at unchanged vocabulary size, not an extension.

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/suite_by_task.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -18,7 +18,7 @@ tags:
 
 # NeuronAI 2B
 
-`NeuronUz/qwen3.5-2b-fine-tuned` is an Uzbek-first, text-only Qwen3.5 2B
+`NeuronUz/NeuronAI-2B` is an Uzbek-first, text-only Qwen3.5 2B
 assistant model with English capability retained. The published weights are a
 merged checkpoint: no adapter is required at inference time.
 
@@ -29,7 +29,7 @@ replaces the previous full-parameter SFT checkpoint.
 ## Model lineage
 
 1. `Qwen/Qwen3.5-2B-Base`
-2. Uzbek tokenizer extension and embedding initialization
+2. Uzbek tokenizer retrofit (same vocabulary size) and embedding re-initialization
 3. Uzbek continued pretraining
 4. Annealing
 5. Assistant-only supervised fine-tuning with LoRA r64
@@ -66,6 +66,73 @@ Checkpoint 1188 scored higher on UzLiB (0.54326), but checkpoint 4749 was
 selected because it was stronger overall, particularly in translation,
 classification, and sentiment.
 
+## Tokenizer fertility
+
+The Uzbek tokenizer work in this model is a **retrofit, not an extension**: the
+vocabulary stays at Qwen3.5's 248,044 entries, and Uzbek merges replace low-value
+slots rather than being appended. The embedding matrix
+therefore does not grow, and the parameter count is unchanged from the base
+model -- the gain is purely in how Uzbek text segments.
+
+![Uzbek tokenizer fertility on uz-crawl](assets/fertility_uzcrawl.png)
+
+| Tokenizer | Vocabulary | Tokens per word (uz-crawl) |
+| --- | ---: | ---: |
+| NeuronAI-2B | 248,044 | **2.407** |
+| Qwen3.5-2B (stock) | 248,044 | 3.426 |
+
+Measured on 20,000 documents (4.8M whitespace words) reservoir-sampled with seed 42
+from uz-crawl (tahrirchi web crawl), keeping documents of at least 250 characters.
+Stock Qwen3.5 needs **1.42x** as many tokens for the same Uzbek text, so the retrofit
+is worth about **30%** of the context window, the prefill cost, and the decode steps
+on Uzbek input.
+
+Caveat: uz-crawl is part of the Uzbek continued-pretraining corpus, so it is
+in-domain for the tokenizer that was fitted on it. The fertility gap on unseen
+Uzbek web text is expected to be somewhat smaller.
+
+## Comparison with other models
+
+![Uzbek public benchmark suite, weighted score](assets/weighted_score.png)
+
+Compared against the two open Uzbek instruction models in this size class
+(`alloma-1B-Instruct` and `alloma-3B-Instruct`, both Llama-3.2 fine-tunes) and
+against the stock base model, `Qwen/Qwen3.5-2B`.
+
+![Uzbek public benchmark suite, per task](assets/suite_by_task.png)
+
+| Benchmark | Metric | Weight | NeuronAI-2B | Qwen3.5-2B | alloma-3B | alloma-1B |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| uzlib (uz MCQ) | accuracy | 0.20 | 0.4960 | 0.2869 | 0.3208 | 0.2332 |
+| TUMLU-uz | accuracy | 0.20 | 0.3257 | 0.3129 | 0.2771 | 0.2200 |
+| FLORES en->uz (COMET) | COMET | 0.15 | 0.8762 | 0.7010 | 0.8673 | 0.7383 |
+| News classification | accuracy | 0.10 | 0.7855 | 0.3675 | 0.1360 | 0.2541 |
+| MMLU (en) | accuracy | 0.10 | 0.5407 | 0.5239 | 0.3873 | 0.2198 |
+| MMLU-uz | accuracy | 0.10 | 0.4685 | 0.3710 | 0.3274 | 0.2111 |
+| FLORES uz->en (COMET) | COMET | 0.05 | 0.8535 | 0.8072 | 0.7954 | 0.7636 |
+| Sentiment (binary) | accuracy | 0.05 | 0.9550 | 0.7687 | 0.3885 | 0.7954 |
+| **Weighted score** | | 0.95 | **0.5657** | **0.4301** | **0.3940** | **0.3478** |
+
+**On score comparability.** These four runs were produced by three different
+suite revisions. The alloma runs used the APST suite variant, stock Qwen used the
+plain variant, and NeuronAI-2B used the COMET-primary variant; the first two weight
+translation by BLEU and the third by COMET, so their archived weighted scores are
+not on one scale. Every run passed `--compute-comet`, so the table above recomputes
+all four from the per-task outputs under a single weighting (the COMET-primary
+weights, translation scored by COMET). The NeuronAI-2B weighted score therefore
+reads 0.5657 here versus 0.5954 in the section above, which normalizes over the 0.95 total weight; the ranking is unaffected.
+
+Both alloma models were evaluated with the apostrophe-to-`APST` substitution their
+model card requires. Their runs tripped the suite's invalid-output gate on several
+MCQ tasks (alloma-1B on uzlib and TUMLU, alloma-3B on TUMLU and MMLU-en), meaning
+part of their MCQ gap is format compliance rather than knowledge.
+
+NeuronAI-2B leads the weighted score and every one of the eight tasks, including
+against alloma-3B at over half again its parameter count. The margin is narrowest
+on FLORES+ en->uz (0.8762 against alloma-3B's 0.8673) and on TUMLU-Uzbek, which is
+the weakest task for every model tested, ours included -- at 0.3257 it is close
+enough to the 0.25 four-way chance level to treat as unsolved at this scale.
+
 ## Training summary
 
 - Base: Uzbek CPT + anneal checkpoint
@@ -96,7 +163,7 @@ not distributed in this model repository.
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
-model_id = "NeuronUz/qwen3.5-2b-fine-tuned"
+model_id = "NeuronUz/NeuronAI-2B"
 
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
@@ -137,7 +204,7 @@ The model emits `<|im_end|>` and the generation metadata registers both
 from vllm import LLM, SamplingParams
 
 llm = LLM(
-    model="NeuronUz/qwen3.5-2b-fine-tuned",
+    model="NeuronUz/NeuronAI-2B",
     max_model_len=4096,
 )
 params = SamplingParams(temperature=0.0, max_tokens=512)