MustaqiLLM-Qwen3

MustaqiLLM-Qwen3 is a native Qwen3-compatible conversion of NeuronUz/MustaqiLLM, a 5.17B-parameter language model focused primarily on Uzbek.

This checkpoint does not represent a retrained or fine-tuned model. The original MustaqiLLM weights were converted from the custom NeuronLMForCausalLM layout into the standard Hugging Face Qwen3ForCausalLM layout.

The main purpose of this conversion is to make MustaqiLLM directly compatible with inference engines that natively support Qwen3, including:

• Hugging Face Transformers
• vLLM
• SGLang
• other Qwen3-compatible runtimes

No trust_remote_code=True is required for the converted model.

Original model: NeuronUz/MustaqiLLM

---

Model Details

Parameter	Value
Parameters	5.17B
Architecture	Qwen3ForCausalLM
Original architecture	NeuronLMForCausalLM
Layers	36
Hidden size	3584
Intermediate size	9728
Attention	GQA
Query heads	28
KV heads	4
Head dimension	128
QK normalization	Yes
Position encoding	RoPE
RoPE θ	500000
Context length	4096 tokens
Vocabulary	48,000 BPE
Embeddings	Untied
Original weights	BF16, with embeddings and lm_head stored as FP32
Languages	Uzbek Latin, Uzbek Cyrillic, English, Russian

---

What Was Converted?

The original MustaqiLLM architecture is functionally equivalent to Qwen3 but stores several projections in fused tensors.

The conversion performs the following mappings:

Attention

NeuronLM                         Qwen3
------------------------------------------------------
qkv_proj.weight        ->       q_proj.weight
                                k_proj.weight
                                v_proj.weight

out_proj.weight        ->       o_proj.weight

q_norm.weight          ->       q_norm.weight
k_norm.weight          ->       k_norm.weight

The original fused QKV projection has:

Q: 28 × 128 = 3584
K:  4 × 128 =  512
V:  4 × 128 =  512

Total QKV output size = 4608

Therefore:

qkv_proj
  ├── q_proj: first 3584 rows
  ├── k_proj: next 512 rows
  └── v_proj: final 512 rows

MLP

NeuronLM                         Qwen3
------------------------------------------------------
gate_up_proj.weight    ->       gate_proj.weight
                                up_proj.weight

down_proj.weight       ->       down_proj.weight

gate_up_proj is split equally into the gate and up projections.

All other compatible tensors are copied directly.

---

Important

This conversion changes the checkpoint layout, not the learned model behavior.

No additional:

• training
• fine-tuning
• distillation
• pruning
• merging

was performed as part of the Qwen3 conversion.

Minor numerical differences may occur depending on inference backend, dtype, GPU architecture, batching, and attention implementation.

---

Usage

Transformers

Install:

pip install -U transformers accelerate torch

Then:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mrmuminov/MustaqiLLM-Qwen3"

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.float16,
    device_map="auto",
).eval()

messages = [
    {
        "role": "user",
        "content": "O'zbekistonning poytaxti qaysi shahar?"
    }
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
).to(model.device)

with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

generated = output[0][inputs["input_ids"].shape[1]:]

print(
    tokenizer.decode(
        generated,
        skip_special_tokens=True,
    )
)

Unlike the original custom architecture, this checkpoint does not require:

trust_remote_code=True

---

vLLM

One of the main purposes of this conversion is native vLLM compatibility.

Install vLLM:

pip install -U vllm

Start an OpenAI-compatible API server:

vllm serve mrmuminov/MustaqiLLM-Qwen3 \
    --dtype float16 \
    --max-model-len 4096

For GPUs with native BF16 support, BF16 may also be used.

For older GPUs such as NVIDIA Turing cards, including the Quadro RTX 8000, use:

--dtype float16

Test the API

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "mrmuminov/MustaqiLLM-Qwen3",
        "messages": [
            {
                "role": "user",
                "content": "O'\''zbekiston haqida qisqacha gapirib ber."
            }
        ],
        "temperature": 0.7,
        "top_p": 0.9,
        "max_tokens": 512
    }'

---

SGLang

The native Qwen3 layout should also allow the model to be loaded by Qwen3-compatible SGLang versions.

Example:

python -m sglang.launch_server \
    --model-path mrmuminov/MustaqiLLM-Qwen3 \
    --host 0.0.0.0 \
    --port 30000 \
    --dtype float16

---

Chat Template

MustaqiLLM uses a ChatML-style conversation format:

<|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>

Using tokenizer.apply_chat_template() is recommended instead of manually constructing this format.

A system message is optional.

For general conversation, the original model authors recommend omitting a generic system prompt when it is unnecessary.

---

Recommended Generation Settings

For open-ended chat, the original MustaqiLLM evaluation recommends approximately:

output = model.generate(
    **inputs,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.05,
    use_cache=True,
)

Recommended range:

Setting	Recommended value
temperature	0.7
top_p	0.9
repetition_penalty	1.05–1.10
do_sample	True for chat
use_cache	True
context	up to 4096 tokens

For classification, extraction, or very short deterministic answers, greedy decoding can be used.

The original model authors found repetition control particularly important for longer generations.

---

Languages

MustaqiLLM primarily supports:

Uzbek

• Latin script
• Cyrillic script

Additional languages

• English
• Russian

The model was designed primarily around Uzbek language capabilities.

---

Evaluation

The following results come from the original MustaqiLLM model card. They have not been independently re-evaluated specifically for this Qwen3-layout conversion.

Because this checkpoint is intended as a layout-preserving conversion of the same model weights, these numbers are included for reference only.

Uzbek

Benchmark	Samples	Score
uzlib	1,861	0.2875
TUMLU-Uzbek	700	0.3286
MMLU-Uz	14,042	0.2584
Uzbek news classification	96,970	0.6531
Sentiment classification	10,000	0.9259

English

Benchmark	Samples	Score
MMLU	14,042	0.2619

FLORES+ Translation

Direction	BLEU	COMET
English → Uzbek	5.17	0.7397
Uzbek → English	1.83	0.5376

See the original model card for the complete evaluation methodology and per-category results:

NeuronUz/MustaqiLLM — Evaluation

---

Limitations

The limitations of the original MustaqiLLM model remain applicable to this converted checkpoint.

Factual knowledge

MustaqiLLM is not primarily a knowledge model.

The original evaluation found performance close to random baseline on several multiple-choice knowledge benchmarks.

It should therefore not be relied upon for:

• factual question answering without verification
• exams
• high-stakes knowledge tasks
• retrieval-free factual systems

Using retrieval or external verification is recommended when factual accuracy matters.

Translation

Uzbek → English translation is relatively weak according to the original evaluation.

English → Uzbek performs better, but the model should not be considered a replacement for a dedicated machine translation system.

Cyrillic

The original model card notes artifacts in Uzbek Cyrillic because some training data was machine-transliterated.

Latin-script Uzbek is generally the safer choice when orthographic accuracy matters.

Repetition

Long generations may enter repetition loops.

For chat, using:

repetition_penalty = 1.05–1.10

is recommended.

Model identity

The original training data may cause the model to identify itself using an older model or project name.

---

Qwen3 Conversion vs. Original MustaqiLLM

Feature	Original	This checkpoint
Model class	NeuronLMForCausalLM	Qwen3ForCausalLM
Custom modeling code	Required	Not required
trust_remote_code	Required	Not required
Fused QKV	Yes	Split
Fused Gate/Up	Yes	Split
Transformers	✅	✅
vLLM native loading	❌	✅
SGLang native loading	❌	✅
Learned weights	MustaqiLLM	MustaqiLLM
Additional training	—	None

---

Quantization

This repository contains the converted native Qwen3 checkpoint.

Quantized variants can be produced separately, for example:

• INT8 W8A8
• FP8 on supported hardware
• AWQ
• GPTQ
• GGUF variants

Quantized checkpoints should be published separately so that this repository remains the reference Qwen3-compatible conversion.

---

Intended Use

Suitable uses include:

• Uzbek conversational applications
• Uzbek text generation
• sentiment classification
• news classification
• experimentation with Uzbek LLM inference
• vLLM/SGLang deployment
• research involving Uzbek language models

The model should not be treated as an authoritative factual source.

---

Attribution

This checkpoint is derived from:

NeuronUz/MustaqiLLM

Original model:

https://huggingface.co/NeuronUz/MustaqiLLM

All model capability claims and benchmark results above originate from the original MustaqiLLM project unless explicitly stated otherwise.

This repository primarily provides a Qwen3-compatible checkpoint conversion for broader inference-engine compatibility.

---

License

The original MustaqiLLM model is released under the Apache License 2.0.

This converted checkpoint follows the same license.

Please review the original repository and license before redistribution or commercial use:

https://huggingface.co/NeuronUz/MustaqiLLM

---

Acknowledgements

Thanks to NeuronUz / NeuronAI.uz for developing and releasing MustaqiLLM and making an Uzbek-focused language model publicly available.

The Qwen3 conversion preserves the original model while making the checkpoint easier to deploy using standard inference infrastructure.