laryme/MustaqiLLM-GGUF

MustaqiLLM — GGUF

GGUF conversions of NeuronUz/MustaqiLLM, a 5.17 B Uzbek chat model. All credit for the model itself goes to NeuronUz; this repo only contains format conversions and quantizations. Apache-2.0, same as the original.

Uzbek text must be normalized before it reaches llama.cpp, and de-normalized on the way out. This is not optional and not cosmetic — see below. Use the included mustaqi_proxy.py, or apply mustaqi_norm.py yourself.

Files

file	size	notes
MustaqiLLM-Q8_0.gguf	5.12 GiB	effectively lossless; recommended if it fits
MustaqiLLM-Q6_K.gguf	3.95 GiB	imatrix; quality indistinguishable in practice
MustaqiLLM-Q5_K_M.gguf	3.41 GiB	imatrix
MustaqiLLM-Q4_K_M.gguf	2.91 GiB	imatrix; smallest sensible
MustaqiLLM-F16.gguf	9.63 GiB	unquantized, for requantizing
imatrix-uz.gguf	4.3 MiB	importance matrix used for the K-quants

The K-quants were built with an importance matrix computed over 400 × 512-token chunks of normalized Uzbek Wikipedia plus some Russian and English. Q8_0 does not use one.

Most published GGUF quants are calibrated on English; this one is not, so the channels the quantizer protects are the ones that matter for Uzbek. imatrix-uz.gguf is included so the K-quants can be reproduced exactly, or new types derived on the same calibration footing:

llama-quantize --imatrix imatrix-uz.gguf \
    MustaqiLLM-F16.gguf MustaqiLLM-IQ4_XS.gguf IQ4_XS

Perplexity across the ladder has not been measured — treat the ordering above as the usual size/quality ordering, not as a measurement of this model.

The normalization requirement

MustaqiLLM's tokenizer folds the Uzbek digraphs oʻ and gʻ (in any of their five apostrophe spellings) into the single code points ō and ḡ before BPE, using an HF Normalizer, and reverses it in its decoder. llama.cpp implements neither normalizers nor decoder replacements for BPE vocabularies, so both steps have to happen outside it.

Measured over 400 KB of Uzbek Wikipedia:

text fed to llama.cpp	token agreement with the HF tokenizer	chunks matching exactly
raw	83.86 %	0 / 100
normalized first	99.99 %	96 / 100

Oʻzbekiston is a single token (1421) when normalized and three (60, 14017, 25198) when not. Skipping this also makes the model emit Ōzbekiston instead of Oʻzbekiston.

Folding the mapping into the vocabulary instead — so a stock llama.cpp would need no help — does not work. The normalizer only fires before a letter, so word-final togʻ and word-internal togʻa must tokenize differently, and BPE cannot condition on what follows. That approach was implemented and measured: it disagreed with the reference tokenizer on 18.5 % of chunks.

Usage

# backend on loopback, normalizing proxy on the public port
llama-server -m MustaqiLLM-Q8_0.gguf --host 127.0.0.1 --port 4115 \
    --ctx-size 4096 -ngl 999 -fa on --jinja --alias mustaqi-llm \
    --temp 0.7 --top-p 0.9 --repeat-penalty 1.05

pip install regex
python mustaqi_proxy.py --host 0.0.0.0 --port 4114 --upstream http://127.0.0.1:4115

Then talk to http://127.0.0.1:4114/v1 with any OpenAI-compatible client. Streaming is handled. Or do it yourself:

from mustaqi_norm import normalize, denormalize
prompt = normalize("Oʻzbekiston togʻlari haqida yozing.")
reply  = denormalize(raw_model_output)

The remaining 0.01 % gap is the pre-tokenizer: the model splits on ' ?\p{L}+(?:['’‘ʻʼ]\p{L}+)*', and these GGUFs are tagged gpt-2, which lacks the apostrophe group. It only affects ASCII ', ’ and ‘ — ʻ (U+02BB) and ʼ (U+02BC) are category Lm and already match \p{L}.

Conversion notes

The original architecture is NeuronLMForCausalLM (custom, trust_remote_code). It is mathematically identical to Qwen3 — pre-norm decoder, GQA 28:4 with per-head QK-RMSNorm applied before RoPE, SwiGLU, untied embeddings, RoPE θ = 500000. Three things differ, all checkpoint layout rather than semantics:

1. fused qkv_proj → q_proj / k_proj / v_proj
2. fused gate_up_proj → gate_proj / up_proj
3. RoPE convention — NeuronLM rotates adjacent channel pairs (GPT-J style), Qwen3 and llama.cpp rotate half-split pairs (NeoX style)

(3) is the one that matters: relabelling the config as Qwen3 loads without error and produces fluent-looking nonsense. The fix is a permutation of each head's projection rows to [0,2,4,…,1,3,5,…]. q·k is unchanged because q and k are permuted identically, v/o_proj are untouched because RoPE never sees them, and q_norm/k_norm are permuted with them because they run before the rotation.

Verified against the original checkpoint in fp32: max |Δlogit| ≈ 1e-5, 100 % argmax agreement across test prompts.

Sampling

From the original model card, measured over 8 400 generations — repeat-penalty is not optional; verbatim sentence repetition is 4.0 % at 1.00 vs 1.3 % at 1.05.

--temp 0.7 --top-p 0.9 --repeat-penalty 1.05

Leave the system prompt out for open chat; a generic one measurably increases repetition. Task-specific Uzbek system prompts work well.

Limitations

Inherited from the base model: this is a chat and text-classification model, not a knowledge model — it performs at chance on multiple-choice knowledge benchmarks. Read the original model card's Evaluation and Limitations sections before using it for anything.