Adapt model card to NeuronAI-4B style

README.md

@@ -16,149 +16,39 @@ tags:
 - lora
 ---
 
-# NeuronAI 2B
+# NeuronAI-2B
 
-`NeuronUz/NeuronAI-2B` is an Uzbek-first, text-only Qwen3.5 2B
-assistant model with English capability retained. The published weights are a
-merged checkpoint: no adapter is required at inference time.
+**NeuronAI-2B** is an Uzbek-first, bilingual assistant model built from
+Qwen3.5-2B-Base. It combines an Uzbek tokenizer retrofit, continued pretraining,
+annealing, and assistant-only supervised fine-tuning. The published weights are
+fully merged—no LoRA adapter is needed.
 
-This release updates the repository to the best overall checkpoint from the
-2026-08-18 LoRA r64 checkpoint sweep (`checkpoint-4749`, one SFT epoch). It
-replaces the previous full-parameter SFT checkpoint.
+![Strict eight-task benchmark comparison](assets/weighted_score.png)
 
-## Model lineage
+## Quick start
 
-1. `Qwen/Qwen3.5-2B-Base`
-2. Uzbek tokenizer retrofit (same vocabulary size) and embedding re-initialization
-3. Uzbek continued pretraining
-4. Annealing
-5. Assistant-only supervised fine-tuning with LoRA r64
-6. LoRA merged into the base weights for deployment
-
-The model is called **NeuronAI 2B** in its identity training examples.
-
-## Evaluation
-
-The four retained checkpoints were evaluated on the same strict eight-task
-public suite using full test splits, greedy vLLM decoding, and COMET-primary
-translation scoring. Checkpoint 4749 achieved the best normalized weighted
-score. All format-scored tasks had a 0% invalid-output rate.
-
-| Benchmark | Metric | Checkpoint 4749 |
-| --- | --- | ---: |
-| UzLiB | accuracy | 0.49597 |
-| TUMLU-Uzbek | accuracy | 0.32571 |
-| FLORES+ en→uz | COMET | 0.87615 |
-| FLORES+ en→uz | BLEU | 12.00 |
-| Uzbek news classification | accuracy | 0.78549 |
-| MMLU (English) | accuracy | 0.54066 |
-| MMLU (Uzbek) | accuracy | 0.46852 |
-| FLORES+ uz→en | COMET | 0.85347 |
-| FLORES+ uz→en | BLEU | 27.21 |
-| Uzbek sentiment (binary) | accuracy | 0.95500 |
-| **Normalized weighted score** | COMET-primary | **0.59542** |
-
-Suite weights were UzLiB 0.20, TUMLU 0.20, en→uz COMET 0.15, news 0.10,
-MMLU-en 0.10, MMLU-uz 0.10, uz→en COMET 0.05, and sentiment 0.05, normalized
-over their total weight of 0.95.
-
-Checkpoint 1188 scored higher on UzLiB (0.54326), but checkpoint 4749 was
-selected because it was stronger overall, particularly in translation,
-classification, and sentiment.
-
-## Tokenizer fertility
-
-The Uzbek tokenizer work in this model is a **retrofit, not an extension**: the
-vocabulary stays at Qwen3.5's 248,044 entries, and Uzbek merges replace low-value
-slots rather than being appended. The embedding matrix
-therefore does not grow, and the parameter count is unchanged from the base
-model -- the gain is purely in how Uzbek text segments.
-
-![Uzbek tokenizer fertility on uz-crawl](assets/fertility_uzcrawl.png)
-
-| Tokenizer | Vocabulary | Tokens per word (uz-crawl) |
-| --- | ---: | ---: |
-| NeuronAI-2B | 248,044 | **2.407** |
-| Qwen3.5-2B (stock) | 248,044 | 3.426 |
-
-Measured on 20,000 documents (4.8M whitespace words) reservoir-sampled with seed 42
-from uz-crawl (tahrirchi web crawl), keeping documents of at least 250 characters.
-Stock Qwen3.5 needs **1.42x** as many tokens for the same Uzbek text, so the retrofit
-is worth about **30%** of the context window, the prefill cost, and the decode steps
-on Uzbek input.
-
-Caveat: uz-crawl is part of the Uzbek continued-pretraining corpus, so it is
-in-domain for the tokenizer that was fitted on it. The fertility gap on unseen
-Uzbek web text is expected to be somewhat smaller.
-
-## Comparison with other models
-
-![Uzbek public benchmark suite, weighted score](assets/weighted_score.png)
-
-Compared against the two open Uzbek instruction models in this size class
-(`alloma-1B-Instruct` and `alloma-3B-Instruct`, both Llama-3.2 fine-tunes) and
-against the stock base model, `Qwen/Qwen3.5-2B`.
-
-![Uzbek public benchmark suite, per task](assets/suite_by_task.png)
-
-| Benchmark | Metric | Weight | NeuronAI-2B | Qwen3.5-2B | alloma-3B | alloma-1B |
-| --- | --- | ---: | ---: | ---: | ---: | ---: |
-| uzlib (uz MCQ) | accuracy | 0.20 | 0.4960 | 0.2869 | 0.3208 | 0.2332 |
-| TUMLU-uz | accuracy | 0.20 | 0.3257 | 0.3129 | 0.2771 | 0.2200 |
-| FLORES en->uz (COMET) | COMET | 0.15 | 0.8762 | 0.7010 | 0.8673 | 0.7383 |
-| News classification | accuracy | 0.10 | 0.7855 | 0.3675 | 0.1360 | 0.2541 |
-| MMLU (en) | accuracy | 0.10 | 0.5407 | 0.5239 | 0.3873 | 0.2198 |
-| MMLU-uz | accuracy | 0.10 | 0.4685 | 0.3710 | 0.3274 | 0.2111 |
-| FLORES uz->en (COMET) | COMET | 0.05 | 0.8535 | 0.8072 | 0.7954 | 0.7636 |
-| Sentiment (binary) | accuracy | 0.05 | 0.9550 | 0.7687 | 0.3885 | 0.7954 |
-| **Weighted score** | | 0.95 | **0.5657** | **0.4301** | **0.3940** | **0.3478** |
-
-**On score comparability.** These four runs were produced by three different
-suite revisions. The alloma runs used the APST suite variant, stock Qwen used the
-plain variant, and NeuronAI-2B used the COMET-primary variant; the first two weight
-translation by BLEU and the third by COMET, so their archived weighted scores are
-not on one scale. Every run passed `--compute-comet`, so the table above recomputes
-all four from the per-task outputs under a single weighting (the COMET-primary
-weights, translation scored by COMET). The NeuronAI-2B weighted score therefore
-reads 0.5657 here versus 0.5954 in the section above, which normalizes over the 0.95 total weight; the ranking is unaffected.
-
-Both alloma models were evaluated with the apostrophe-to-`APST` substitution their
-model card requires. Their runs tripped the suite's invalid-output gate on several
-MCQ tasks (alloma-1B on uzlib and TUMLU, alloma-3B on TUMLU and MMLU-en), meaning
-part of their MCQ gap is format compliance rather than knowledge.
-
-NeuronAI-2B leads the weighted score and every one of the eight tasks, including
-against alloma-3B at over half again its parameter count. The margin is narrowest
-on FLORES+ en->uz (0.8762 against alloma-3B's 0.8673) and on TUMLU-Uzbek, which is
-the weakest task for every model tested, ours included -- at 0.3257 it is close
-enough to the 0.25 four-way chance level to treat as unsolved at this scale.
-
-## Training
+Install a recent Transformers build with Qwen3.5 support:
 
-One epoch of assistant-only LoRA fine-tuning (rank 64, alpha 128) on 152,152
-Uzbek-first examples, using a 2,048-token context and effective batch size 32.
-The adapter was merged into the Uzbek CPT + anneal checkpoint for release.
-
-## Usage
+```bash
+pip install -U "transformers>=5.1" accelerate torch
+```
 
 ```python
 import torch
 from transformers import AutoModelForCausalLM, AutoTokenizer
 
 model_id = "NeuronUz/NeuronAI-2B"
-
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
     model_id,
     dtype=torch.bfloat16,
     device_map="cuda:0",
-)
+).eval()
 
 messages = [
-    {"role": "system", "content": "Siz foydali AI yordamchisiz."},
-    {"role": "user", "content": "O'zbekiston haqida qisqacha ma'lumot bering."},
+    {"role": "system", "content": "Siz foydali va aniq AI yordamchisiz."},
+    {"role": "user", "content": "Alisher Navoiy haqida qisqacha aytib bering."},
 ]
-
 inputs = tokenizer.apply_chat_template(
     messages,
     add_generation_prompt=True,
@@ -168,42 +58,113 @@ inputs = tokenizer.apply_chat_template(
 ).to(model.device)
 
 with torch.inference_mode():
-    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
 
 reply = tokenizer.decode(
-    output[0][inputs["input_ids"].shape[-1]:],
+    output[0, inputs["input_ids"].shape[1]:],
     skip_special_tokens=True,
 ).strip()
 print(reply)
 ```
 
-The model emits `<|im_end|>` and the generation metadata registers both
-`<|im_end|>` and `<|endoftext|>` as end-of-sequence tokens.
+The generation metadata already registers `<|im_end|>` and `<|endoftext|>` as
+end-of-sequence tokens.
 
-### vLLM
+### Serve with vLLM
 
-```python
-from vllm import LLM, SamplingParams
-
-llm = LLM(
-    model="NeuronUz/NeuronAI-2B",
-    max_model_len=4096,
-)
-params = SamplingParams(temperature=0.0, max_tokens=512)
+```bash
+pip install -U vllm
+vllm serve NeuronUz/NeuronAI-2B \
+  --dtype bfloat16 \
+  --max-model-len 4096 \
+  --language-model-only \
+  --mamba-block-size 16 \
+  --mamba-cache-mode align
+```
 
-messages = [
-    {"role": "system", "content": "Siz foydali AI yordamchisiz."},
-    {"role": "user", "content": "O'zbekiston haqida qisqacha ma'lumot bering."},
-]
-print(llm.chat(messages, params)[0].outputs[0].text)
+```bash
+curl http://localhost:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{"model":"NeuronUz/NeuronAI-2B","messages":[{"role":"user","content":"O‘zbekiston haqida uchta fakt ayting."}]}'
 ```
 
-## Notes and limitations
+## Benchmarks
+
+The results below are recomputed from archived full-suite runs using the same
+eight tasks, COMET-primary scoring, and normalized task weights. Classification
+and multiple-choice tasks use accuracy; FLORES+ translation uses COMET. All
+format-scored NeuronAI-2B tasks had a 0% invalid-output rate.
+
+![Per-task comparison](assets/suite_by_task.png)
+
+| Benchmark | Metric | Weight | **NeuronAI-2B** | Qwen3.5-2B | alloma-3B | alloma-1B |
+| --- | --- | ---: | ---: | ---: | ---: | ---: |
+| UzLiB | accuracy | 0.20 | **49.60%** | 28.69% | 32.08% | 23.32% |
+| TUMLU-Uzbek | accuracy | 0.20 | **32.57%** | 31.29% | 27.71% | 22.00% |
+| FLORES+ en→uz | COMET | 0.15 | **0.8762** | 0.7010 | 0.8673 | 0.7383 |
+| Uzbek news | accuracy | 0.10 | **78.55%** | 36.75% | 13.60% | 25.41% |
+| MMLU English | accuracy | 0.10 | **54.07%** | 52.39% | 38.73% | 21.98% |
+| MMLU Uzbek | accuracy | 0.10 | **46.85%** | 37.10% | 32.74% | 21.11% |
+| FLORES+ uz→en | COMET | 0.05 | **0.8535** | 0.8072 | 0.7954 | 0.7636 |
+| Uzbek sentiment | accuracy | 0.05 | **95.50%** | 76.87% | 38.85% | 79.54% |
+| **Normalized weighted score** | | 1.00 | **0.5954** | 0.4528 | 0.4147 | 0.3661 |
+
+The archived runs came from three suite revisions, but every run computed COMET,
+allowing the table to use one scoring method. Alloma runs used the `APST`
+apostrophe preprocessing required by their model cards.
+
+## Uzbek tokenizer efficiency
+
+The tokenizer is an in-place retrofit rather than a vocabulary extension:
+Uzbek merges replace low-value slots while the 248,044-token vocabulary and
+parameter count remain unchanged.
+
+![Uzbek tokenizer fertility](assets/fertility_uzcrawl.png)
+
+| Tokenizer | Vocabulary | Tokens per word |
+| --- | ---: | ---: |
+| NeuronAI-2B | 248,044 | **2.407** |
+| Qwen3.5-2B | 248,044 | 3.426 |
+
+This measurement used 20,000 documents (4.8M whitespace words) sampled with
+seed 42 from `uz-crawl`. The retrofit uses about 30% fewer tokens on this
+in-domain Uzbek text. Fertility measures tokenization efficiency, not model
+quality or measured inference speed.
+
+## Model lineage and training
+
+1. `Qwen/Qwen3.5-2B-Base`
+2. Uzbek tokenizer retrofit with embedding re-initialization
+3. Uzbek continued pretraining
+4. Instruction-data annealing
+5. Assistant-only SFT with LoRA r64
+6. LoRA merged into the model weights
+
+One epoch of assistant-only LoRA fine-tuning (rank 64, alpha 128) on 152,152
+Uzbek-first examples, using a 2,048-token context and effective batch size 32.
+Training data is not distributed with this model repository.
+
+## Intended use
+
+Good fits include Uzbek research, education, prototyping, translation
+experiments, writing assistance, retrieval-augmented generation, and
+local/offline applications.
+
+## Limitations
+
+- TUMLU-Uzbek remains close to the 0.25 random baseline and should not be
+  treated as solved.
+- The model can hallucinate, repeat biases in its data, or produce unsafe or
+  outdated content. It has not been comprehensively safety-evaluated.
+- Do not rely on it without expert review for medical, legal, financial, public
+  safety, or other high-stakes decisions.
+- SFT used sequences up to 2,048 tokens; serving at longer inherited context
+  lengths has not been validated here.
+- Use a recent Transformers or vLLM release with Qwen3.5 support. Keep the
+  hybrid model on one device unless multi-device Gated-DeltaNet inference has
+  been validated in your serving stack.
+
+## License
 
-- Use a Transformers/vLLM release with Qwen3.5 support.
-- Keep the hybrid Qwen3.5 model on one device unless the serving stack has been
-  validated for multi-device Gated-DeltaNet inference.
-- TUMLU-Uzbek remains close to the 0.25 random baseline, so specialist factual
-  knowledge should be independently verified.
-- The model may produce inaccurate, biased, or fabricated information and has
-  not been comprehensively evaluated for safety or high-stakes use.
+NeuronAI-2B is released under the
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).