Replace with full-FT v3 checkpoint (weighted 0.4388, was 0.4190); update card with benchmarks + clean usage

README.md

@@ -12,93 +12,98 @@ tags:
 - text-generation
 - conversational
 - axolotl
-- lora
 ---
 
-# Qwen3.5 2B Uzbek Fine-Tuned (LoRA Broad)
+# Qwen3.5 2B Uzbek Fine-Tuned
 
-This is a merged, text-only Qwen3.5 2B checkpoint fine-tuned primarily for
-Uzbek instruction following and conversational use. It is the `lora-broad`
-experiment: a broad supervised mixture intended to improve general Uzbek
-assistant capability while retaining task-format and English examples.
+A text-only Qwen3.5 2B checkpoint for Uzbek instruction following and
+conversation, with English retained. Uzbek continued pretraining and annealing,
+then a full-parameter supervised fine-tune.
+
+This release replaces the earlier LoRA-broad checkpoint. It scores higher on
+every benchmark below and, unlike its predecessor, **stops generating on its
+own** — the previous checkpoints never learned to emit the turn terminator and
+needed sentence counters and repetition penalties to produce usable output. The
+usage example below is correspondingly plain.
 
 ## Model lineage
 
 1. `Qwen/Qwen3.5-2B-Base`
-2. Local Uzbek continued-pretraining and annealing checkpoint
-3. Supervised fine-tuning with LoRA
-4. LoRA weights merged into the model for direct inference
-
-This repository contains the merged model, so PEFT is not required to load it.
+2. Uzbek tokenizer extension (vocabulary 248,320) and embedding initialization
+3. Uzbek continued pretraining
+4. Annealing
+5. Full-parameter supervised fine-tuning (no LoRA — these are the trained weights)
+
+## Benchmarks
+
+Public Uzbek evaluation suite, vLLM backend, full test splits, greedy decoding.
+Identical harness and settings for all three models. Translation is FLORES+ with
+sacreBLEU and COMET (`Unbabel/wmt22-comet-da`).
+
+| Benchmark | Metric | **This model** | Previous release (LoRA broad) | Qwen3.5-2B-Instruct (stock) |
+| --- | --- | ---: | ---: | ---: |
+| UzLiB | accuracy | **0.4863** | 0.4782 | 0.2880 |
+| TUMLU-Uzbek | accuracy | **0.3214** | 0.3686 | 0.3129 |
+| News classification | accuracy | **0.7948** | 0.7355 | 0.3675 |
+| Sentiment (binary) | accuracy | **0.9626** | 0.9348 | 0.7676 |
+| MMLU (English) | accuracy | **0.5422** | 0.5300 | 0.5241 |
+| MMLU (Uzbek) | accuracy | **0.4707** | 0.4640 | 0.3711 |
+| FLORES+ en→uz | BLEU | **9.90** | 4.05 | 4.16 |
+| FLORES+ en→uz | COMET | **0.8496** | 0.7413 | 0.6790 |
+| FLORES+ uz→en | BLEU | **23.07** | 5.93 | 17.13 |
+| FLORES+ uz→en | COMET | **0.8314** | 0.6056 | 0.8091 |
+| **Weighted score** | | **0.4388** | 0.4190 | 0.3154 |
+
+The weighted score combines all eight tasks (UzLiB 0.20, TUMLU 0.20, en→uz 0.15,
+news 0.10, MMLU-en 0.10, MMLU-uz 0.10, uz→en 0.05, sentiment 0.05), with BLEU
+scaled to a 0–1 range.
+
+Notes on reading these numbers honestly:
+
+- **Translation is where the gain is largest** (en→uz BLEU 4.05 → 9.90, uz→en
+  5.93 → 23.07). Much of that is the terminator fix: the previous checkpoint ran
+  past the end of its answer, which BLEU punishes severely.
+- **TUMLU-Uzbek regressed** (0.3686 → 0.3214) and is this model's weakest task.
+  It is also near the 0.25 random baseline for 4-choice questions, so treat
+  Uzbek multi-subject knowledge as unreliable.
+- Invalid-output rate was 0.0000 on all format-scored tasks.
+- The checkpoint published here is the one that scored best on this suite
+  (1.5 epochs), selected across all 12 training checkpoints. It is not the final
+  epoch-3 weights, which scored 0.4310.
 
 ## Training summary
 
 - Framework: Axolotl / Transformers
-- Training data: 269,467 conversational examples
-- Languages: primarily Uzbek, with English retention data
-- Context length during SFT: 2,048 tokens
-- Epochs: 1
-- LoRA rank: 32
-- LoRA alpha: 64
-- LoRA dropout: 0.05
-- Learning rate: 1e-4 with cosine scheduling
-- Validation split: 2%
-- Final reported training loss: 1.381
-
-The training mixture contained broad conversational data, task-formatted
-examples, and Uzbek knowledge/language material. The underlying dataset is not
-included in this repository.
+- Method: full-parameter supervised fine-tuning
+- Data: 169,919 conversational examples (Uzbek-first, with task-format and
+  English retention data)
+- Sequence length: 2,048, one sample per sequence (no packing)
+- Epochs: 3, best checkpoint by benchmark score taken at 1.5 epochs
+- Effective batch size: 32
+- Learning rate: 1e-5, cosine schedule, 3% warmup
+- Optimizer: AdamW, β₂ = 0.95, gradient clipping 1.0
+- Precision: bf16
+- Loss on assistant turns only; `<|im_end|>` trained as the turn terminator
+
+The training data is not included in this repository.
 
 ## Usage
 
 ```python
-import re
 import torch
-from transformers import (
-    AutoModelForCausalLM,
-    AutoTokenizer,
-    StoppingCriteria,
-    StoppingCriteriaList,
-)
-
-
-class SentenceLimitCriteria(StoppingCriteria):
-    """Stop after a fixed number of complete generated sentences."""
-
-    def __init__(self, tokenizer, prompt_length, max_sentences=4):
-        self.tokenizer = tokenizer
-        self.prompt_length = prompt_length
-        self.max_sentences = max_sentences
-
-    def __call__(self, input_ids, scores, **kwargs):
-        generated = self.tokenizer.decode(
-            input_ids[0, self.prompt_length:], skip_special_tokens=True
-        )
-        endings = re.findall(r'[.!?](?:["\'’”)]*)?\s+', generated)
-        return len(endings) >= self.max_sentences
-
+from transformers import AutoModelForCausalLM, AutoTokenizer
 
 model_id = "NeuronUz/qwen3.5-2b-fine-tuned"
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-max_sentences = 4
 
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
     model_id,
-    torch_dtype="auto",
-    # Keep this hybrid model on one device. See the note below.
-    device_map=device,
+    dtype=torch.bfloat16,
+    device_map="cuda:0",  # keep this hybrid model on a single device
 )
 
 messages = [
-    {
-        "role": "system",
-        "content": (
-            "Siz foydali AI yordamchisiz. Javoblarni qisqa va aniq yozing. "
-            "Agar foydalanuvchi batafsil javob so'ramasa, odatda 2-4 ta "
-            "to'liq gap bilan javob bering."
-        ),
-    },
+    {"role": "system", "content": "Siz foydali AI yordamchisiz."},
     {"role": "user", "content": "O'zbekiston haqida qisqacha ma'lumot bering."},
 ]
 
@@ -109,58 +114,77 @@ inputs = tokenizer.apply_chat_template(
     return_dict=True,
 ).to(model.device)
 
-im_end_id = tokenizer.convert_tokens_to_ids("<|im_end|>")
-eos_ids = [tokenizer.eos_token_id, im_end_id]
-stopping_criteria = StoppingCriteriaList(
-    [
-        SentenceLimitCriteria(
-            tokenizer,
-            prompt_length=inputs["input_ids"].shape[-1],
-            max_sentences=max_sentences,
-        )
-    ]
-)
-
 with torch.inference_mode():
-    output = model.generate(
-        **inputs,
-        max_new_tokens=256,
-        do_sample=False,
-        repetition_penalty=1.15,
-        no_repeat_ngram_size=3,
-        eos_token_id=eos_ids,
-        pad_token_id=tokenizer.eos_token_id,
-        stopping_criteria=stopping_criteria,
-    )
-
-prompt_length = inputs["input_ids"].shape[-1]
+    output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+
 reply = tokenizer.decode(
-    output[0][prompt_length:], skip_special_tokens=True
+    output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
 ).strip()
+print(reply)
+```
 
-# A token can contain the final period and the start of the next word, so trim
-# the displayed output back to the fourth complete sentence.
-sentence_end_re = re.compile(r'[.!?](?:["\'’”)]*)?(?=\s|$)')
-sentence_endings = list(sentence_end_re.finditer(reply))
-if len(sentence_endings) >= max_sentences:
-    reply = reply[:sentence_endings[max_sentences - 1].end()].strip()
+No stopping criteria, repetition penalty, or `no_repeat_ngram_size` are needed —
+the model emits `<|im_end|>`, and `generation_config.json` already registers it
+as an end-of-sequence token.
 
-print(reply)
+### Multi-turn chat
+
+```python
+messages = [{"role": "system", "content": "Siz foydali AI yordamchisiz."}]
+
+while True:
+    user = input("> ").strip()
+    if user in {"", "exit", "quit"}:
+        break
+    messages.append({"role": "user", "content": user})
+
+    inputs = tokenizer.apply_chat_template(
+        messages,
+        add_generation_prompt=True,
+        return_tensors="pt",
+        return_dict=True,
+    ).to(model.device)
+
+    with torch.inference_mode():
+        output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
+
+    reply = tokenizer.decode(
+        output[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True
+    ).strip()
+    print(reply)
+    messages.append({"role": "assistant", "content": reply})
+```
+
+### vLLM
+
+```python
+from vllm import LLM, SamplingParams
+
+llm = LLM(model="NeuronUz/qwen3.5-2b-fine-tuned", max_model_len=4096)
+params = SamplingParams(temperature=0.0, max_tokens=512)
+
+messages = [
+    {"role": "system", "content": "Siz foydali AI yordamchisiz."},
+    {"role": "user", "content": "Bugungi ob-havo haqida nima deya olasiz?"},
+]
+print(llm.chat(messages, params)[0].outputs[0].text)
 ```
 
-Use a recent Transformers release with Qwen3.5 support.
+### Notes
 
-When multiple GPUs are visible, avoid `device_map="auto"` with this checkpoint.
-Current Accelerate/Transformers releases may split the Qwen3.5 hybrid layers
-across GPUs and produce invalid text. Pin the complete model to one GPU as shown
-above. The example uses greedy decoding (`do_sample=False`, equivalent to
-temperature 0 in the local chat script) and limits normal answers to four
-complete sentences. If sampling is desired, a tested starting point is
-`temperature=0.7`, `top_p=0.8`, and `top_k=20`.
+- Requires a Transformers release with Qwen3.5 support.
+- Greedy decoding (`do_sample=False`) was used for all benchmark numbers above.
+  For sampling, a reasonable starting point is `temperature=0.7`, `top_p=0.8`,
+  `top_k=20`.
+- Avoid `device_map="auto"` when several GPUs are visible: current
+  Accelerate/Transformers releases may split the Qwen3.5 hybrid layers across
+  devices and produce invalid text. Pin the model to one device as shown.
 
 ## Limitations
 
-The model may produce inaccurate, biased, or fabricated information. It has not
-been comprehensively evaluated for safety or high-stakes domains. Outputs should
-be independently verified before use in medical, legal, financial, or other
+The model may produce inaccurate, biased, or fabricated information. Uzbek
+multi-subject knowledge (TUMLU) is close to the random baseline, so factual
+answers in specialist domains should not be trusted. It has not been
+comprehensively evaluated for safety or high-stakes use. Verify outputs
+independently before relying on them in medical, legal, financial, or other
 consequential settings.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d41fe7acbe94f42dde8dd12e48c791628a1a9a22343e3eaf910b9d4b03813ee1
+oid sha256:d39d981c03f63bb711d44ab9a7fb963dc118f57eef0cc00bff9d39543e086f03
 size 4781022144

tokenizer_config.json

@@ -10,7 +10,7 @@
   "errors": "replace",
   "image_token": "<|image_pad|>",
   "is_local": true,
-  "local_files_only": false,
+  "local_files_only": true,
   "model_max_length": 262144,
   "model_specific_special_tokens": {
     "audio_bos_token": "<|audio_start|>",