Release NeuronAI 2B LoRA r64 checkpoint 4749

Replace the previous full-parameter SFT release with the merged CPT+anneal, grouped-v4 LoRA r64 checkpoint 4749. Updates model card with the completed eight-task COMET-primary suite and BLEU results.

README.md

@@ -12,117 +12,83 @@ tags:
 - text-generation
 - conversational
 - axolotl
+- peft
+- lora
 ---
 
-# Qwen3.5 2B Uzbek Fine-Tuned
+# NeuronAI 2B
 
-A text-only Qwen3.5 2B checkpoint for Uzbek instruction following and
-conversation, with English retained. Uzbek continued pretraining and annealing,
-then a full-parameter supervised fine-tune.
+`NeuronUz/qwen3.5-2b-fine-tuned` is an Uzbek-first, text-only Qwen3.5 2B
+assistant model with English capability retained. The published weights are a
+merged checkpoint: no adapter is required at inference time.
 
-This release replaces the earlier LoRA-broad checkpoint. It scores higher on
-every benchmark below and, unlike its predecessor, **stops generating on its
-own** — the previous checkpoints never learned to emit the turn terminator and
-needed sentence counters and repetition penalties to produce usable output. The
-usage example below is correspondingly plain.
+This release updates the repository to the best overall checkpoint from the
+2026-08-18 LoRA r64 checkpoint sweep (`checkpoint-4749`, one SFT epoch). It
+replaces the previous full-parameter SFT checkpoint.
 
 ## Model lineage
 
 1. `Qwen/Qwen3.5-2B-Base`
-2. Uzbek tokenizer extension (vocabulary 248,320) and embedding initialization
+2. Uzbek tokenizer extension and embedding initialization
 3. Uzbek continued pretraining
 4. Annealing
-5. Full-parameter supervised fine-tuning (no LoRA — these are the trained weights)
-
-## Benchmarks
-
-### Comparison with alloma-1B and alloma-3B
-
-Run with the benchmark protocol published in the **alloma model card** — its
-tasks, datasets and prompts, used verbatim — so the comparison does not depend
-on prompt formats chosen by us. alloma's required APST apostrophe preprocessing
-is applied to the alloma models. Full test splits, greedy decoding. COMET is
-`Unbabel/wmt22-comet-da`. Higher is better throughout.
-
-| Benchmark | **This model** (1.88B) | alloma-1B (1.24B) | alloma-3B (3.21B) |
-| --- | ---: | ---: | ---: |
-| Uzbek sentiment analysis | **93.09** | 58.05 | 81.53 |
-| Uzbek news classification | **56.47** | 15.85 | 33.78 |
-| MMLU (English) | **52.81** | 28.61 | 45.34 |
-| BLEU uz→en | 21.04 | 16.10 | **24.62** |
-| COMET uz→en | 81.06 | 80.08 | **85.14** |
-| BLEU en→uz | 9.85 | 8.70 | **14.54** |
-| COMET en→uz | 83.82 | 81.20 | **86.80** |
-
-Read this honestly, in both directions:
-
-- This model beats **alloma-1B on every metric**, and beats the ~1.7x larger
-  **alloma-3B on classification and English knowledge** by wide margins
-  (sentiment +11.6, news +22.7, MMLU +7.5).
-- **alloma-3B is clearly better at translation**, in both directions and on both
-  BLEU and COMET. If translation quality is your priority, alloma-3B is the
-  stronger choice, and alloma-8B stronger still.
-
-### Full public suite
-
-Our own eight-task Uzbek suite, vLLM backend, full test splits, greedy decoding.
-Shown against this model's predecessor and the stock instruct model.
-
-| Benchmark | Metric | **This model** | Previous release (LoRA broad) | Qwen3.5-2B-Instruct (stock) |
-| --- | --- | ---: | ---: | ---: |
-| UzLiB | accuracy | **0.4863** | 0.4782 | 0.2880 |
-| TUMLU-Uzbek | accuracy | **0.3214** | 0.3686 | 0.3129 |
-| News classification | accuracy | **0.7948** | 0.7355 | 0.3675 |
-| Sentiment (binary) | accuracy | **0.9626** | 0.9348 | 0.7676 |
-| MMLU (English) | accuracy | **0.5422** | 0.5300 | 0.5241 |
-| MMLU (Uzbek) | accuracy | **0.4707** | 0.4640 | 0.3711 |
-| FLORES+ en→uz | BLEU | **9.90** | 4.05 | 4.16 |
-| FLORES+ en→uz | COMET | **0.8496** | 0.7413 | 0.6790 |
-| FLORES+ uz→en | BLEU | **23.07** | 5.93 | 17.13 |
-| FLORES+ uz→en | COMET | **0.8314** | 0.6056 | 0.8091 |
-| **Weighted score** | | **0.4388** | 0.4190 | 0.3154 |
-
-The weighted score combines all eight tasks (UzLiB 0.20, TUMLU 0.20, en→uz 0.15,
-news 0.10, MMLU-en 0.10, MMLU-uz 0.10, uz→en 0.05, sentiment 0.05), with BLEU
-scaled to a 0–1 range.
-
-The alloma models are deliberately **not** listed in this second table. They were
-measured on it (alloma-1B 0.2242, alloma-3B 0.2666), but those runs fail the
-suite's own output-validity gate: alloma-3B produced unparseable answers on 44.1%
-of sentiment items and 28.7% of TUMLU items under our prompt formats, versus
-0.0% invalid for this model. Those scores measure compliance with our prompt
-templates, not model capability, so quoting them as a head-to-head result would
-overstate this model. The alloma-protocol table above is the fair comparison.
-
-Notes on reading these numbers honestly:
-
-- **Translation is where the gain over the previous release is largest** (en→uz
-  BLEU 4.05 → 9.90, uz→en 5.93 → 23.07). Much of that is the terminator fix: the
-  previous checkpoint ran past the end of its answer, which BLEU punishes
-  severely.
-- **TUMLU-Uzbek regressed** (0.3686 → 0.3214) and is this model's weakest task.
-  It is also near the 0.25 random baseline for 4-choice questions, so treat
-  Uzbek multi-subject knowledge as unreliable.
-- Invalid-output rate was 0.0000 on all format-scored tasks.
-- The checkpoint published here is the one that scored best on this suite
-  (1.5 epochs), selected across all 12 training checkpoints. It is not the final
-  epoch-3 weights, which scored 0.4310.
+5. Assistant-only supervised fine-tuning with LoRA r64
+6. LoRA merged into the base weights for deployment
+
+The model is called **NeuronAI 2B** in its identity training examples.
+
+## Evaluation
+
+The four retained checkpoints were evaluated on the same strict eight-task
+public suite using full test splits, greedy vLLM decoding, and COMET-primary
+translation scoring. Checkpoint 4749 achieved the best normalized weighted
+score. All format-scored tasks had a 0% invalid-output rate.
+
+| Benchmark | Metric | Checkpoint 4749 |
+| --- | --- | ---: |
+| UzLiB | accuracy | 0.49597 |
+| TUMLU-Uzbek | accuracy | 0.32571 |
+| FLORES+ en→uz | COMET | 0.87615 |
+| FLORES+ en→uz | BLEU | 12.00 |
+| Uzbek news classification | accuracy | 0.78549 |
+| MMLU (English) | accuracy | 0.54066 |
+| MMLU (Uzbek) | accuracy | 0.46852 |
+| FLORES+ uz→en | COMET | 0.85347 |
+| FLORES+ uz→en | BLEU | 27.21 |
+| Uzbek sentiment (binary) | accuracy | 0.95500 |
+| **Normalized weighted score** | COMET-primary | **0.59542** |
+
+Suite weights were UzLiB 0.20, TUMLU 0.20, en→uz COMET 0.15, news 0.10,
+MMLU-en 0.10, MMLU-uz 0.10, uz→en COMET 0.05, and sentiment 0.05, normalized
+over their total weight of 0.95.
+
+Checkpoint 1188 scored higher on UzLiB (0.54326), but checkpoint 4749 was
+selected because it was stronger overall, particularly in translation,
+classification, and sentiment.
 
 ## Training summary
 
-- Framework: Axolotl / Transformers
-- Method: full-parameter supervised fine-tuning
-- Data: 169,919 conversational examples (Uzbek-first, with task-format and
-  English retention data)
-- Sequence length: 2,048, one sample per sequence (no packing)
-- Epochs: 3, best checkpoint by benchmark score taken at 1.5 epochs
+- Base: Uzbek CPT + anneal checkpoint
+- Method: LoRA, merged for release
+- LoRA rank/alpha/dropout: 64 / 128 / 0.05
+- LoRA targets: all linear layers
+- Trainable adapter parameters: 67,276,800
+- Training data: 152,152 grouped and globally deduplicated examples
+- Validation data: 1,537 grouped examples
+- Train/dev prompt-group overlap: 0
+- Sequence length: 2,048
+- Packing: disabled
+- Epochs: 1
 - Effective batch size: 32
-- Learning rate: 1e-5, cosine schedule, 3% warmup
-- Optimizer: AdamW, β₂ = 0.95, gradient clipping 1.0
-- Precision: bf16
-- Loss on assistant turns only; `<|im_end|>` trained as the turn terminator
+- Learning rate: 1e-4, cosine schedule, 3% warmup
+- Optimizer: AdamW, beta2 0.95, weight decay 0.01
+- Precision: bf16 with selected recurrent parameters retained in fp32
+- Loss: assistant turns only; `<|im_end|>` explicitly supervised
 
-The training data is not included in this repository.
+The training mixture is Uzbek-first and includes clean general assistant data,
+translation, Uzbek language and literature, spelling and orthography, MMLU-style
+tasks, classification, math, and English-retention examples. Training data is
+not distributed in this model repository.
 
 ## Usage
 
@@ -136,7 +102,7 @@ tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
     model_id,
     dtype=torch.bfloat16,
-    device_map="cuda:0",  # keep this hybrid model on a single device
+    device_map="cuda:0",
 )
 
 messages = [
@@ -147,6 +113,7 @@ messages = [
 inputs = tokenizer.apply_chat_template(
     messages,
     add_generation_prompt=True,
+    enable_thinking=False,
     return_tensors="pt",
     return_dict=True,
 ).to(model.device)
@@ -155,73 +122,39 @@ with torch.inference_mode():
     output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
 
 reply = tokenizer.decode(
-    output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
+    output[0][inputs["input_ids"].shape[-1]:],
+    skip_special_tokens=True,
 ).strip()
 print(reply)
 ```
 
-No stopping criteria, repetition penalty, or `no_repeat_ngram_size` are needed —
-the model emits `<|im_end|>`, and `generation_config.json` already registers it
-as an end-of-sequence token.
-
-### Multi-turn chat
-
-```python
-messages = [{"role": "system", "content": "Siz foydali AI yordamchisiz."}]
-
-while True:
-    user = input("> ").strip()
-    if user in {"", "exit", "quit"}:
-        break
-    messages.append({"role": "user", "content": user})
-
-    inputs = tokenizer.apply_chat_template(
-        messages,
-        add_generation_prompt=True,
-        return_tensors="pt",
-        return_dict=True,
-    ).to(model.device)
-
-    with torch.inference_mode():
-        output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
-
-    reply = tokenizer.decode(
-        output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
-    ).strip()
-    print(reply)
-    messages.append({"role": "assistant", "content": reply})
-```
+The model emits `<|im_end|>` and the generation metadata registers both
+`<|im_end|>` and `<|endoftext|>` as end-of-sequence tokens.
 
 ### vLLM
 
 ```python
 from vllm import LLM, SamplingParams
 
-llm = LLM(model="NeuronUz/qwen3.5-2b-fine-tuned", max_model_len=4096)
+llm = LLM(
+    model="NeuronUz/qwen3.5-2b-fine-tuned",
+    max_model_len=4096,
+)
 params = SamplingParams(temperature=0.0, max_tokens=512)
 
 messages = [
     {"role": "system", "content": "Siz foydali AI yordamchisiz."},
-    {"role": "user", "content": "Bugungi ob-havo haqida nima deya olasiz?"},
+    {"role": "user", "content": "O'zbekiston haqida qisqacha ma'lumot bering."},
 ]
 print(llm.chat(messages, params)[0].outputs[0].text)
 ```
 
-### Notes
-
-- Requires a Transformers release with Qwen3.5 support.
-- Greedy decoding (`do_sample=False`) was used for all benchmark numbers above.
-  For sampling, a reasonable starting point is `temperature=0.7`, `top_p=0.8`,
-  `top_k=20`.
-- Avoid `device_map="auto"` when several GPUs are visible: current
-  Accelerate/Transformers releases may split the Qwen3.5 hybrid layers across
-  devices and produce invalid text. Pin the model to one device as shown.
-
-## Limitations
+## Notes and limitations
 
-The model may produce inaccurate, biased, or fabricated information. Uzbek
-multi-subject knowledge (TUMLU) is close to the random baseline, so factual
-answers in specialist domains should not be trusted. It has not been
-comprehensively evaluated for safety or high-stakes use. Verify outputs
-independently before relying on them in medical, legal, financial, or other
-consequential settings.
+- Use a Transformers/vLLM release with Qwen3.5 support.
+- Keep the hybrid Qwen3.5 model on one device unless the serving stack has been
+  validated for multi-device Gated-DeltaNet inference.
+- TUMLU-Uzbek remains close to the 0.25 random baseline, so specialist factual
+  knowledge should be independently verified.
+- The model may produce inaccurate, biased, or fabricated information and has
+  not been comprehensively evaluated for safety or high-stakes use.

config.json

@@ -7,7 +7,10 @@
   "attn_output_gate": true,
   "bos_token_id": null,
   "dtype": "float32",
-  "eos_token_id": 248044,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
   "full_attention_interval": 4,
   "head_dim": 256,
   "hidden_act": "silu",

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d39d981c03f63bb711d44ab9a7fb963dc118f57eef0cc00bff9d39543e086f03
+oid sha256:7b3667d6eb7841e22ebf2c8290e8fdf02c6cafdb9cb913c73d944abf31b2161a
 size 4781022144

tokenizer_config.json

@@ -10,7 +10,7 @@
   "errors": "replace",
   "image_token": "<|image_pad|>",
   "is_local": true,
-  "local_files_only": true,
+  "local_files_only": false,
   "model_max_length": 262144,
   "model_specific_special_tokens": {
     "audio_bos_token": "<|audio_start|>",