Qolda-AVL-5B-OptimizedTokenizer
An optimized new version of ISSAI's audio-vision-language model Qolda-AVL-5B.
Developed by Almubdi Mutaikhan with Batyr Arystanbekov (ISSAI), under the supervision of Prof. Huseyin Atakan Varol (ISSAI). See Citation.
ISSAI earlier released issai/Qolda-AVL-5B: a 5B audio-vision-language model for Kazakh, Russian and English. It extends Qwen3-VL-4B-Thinking with an audio branch: a fine-tuned Whisper encoder and an audio projection. All three modalities are adapted to Kazakh in staged training.
This model is its next version. It keeps Qolda-AVL's audio encoder and audio training data, and changes three things:
- A new language model: issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer (Qwen3.5, a Kazakh + Russian tokenizer, Russian in pretraining). Kazakh text needs 1.98ร fewer tokens than with Qwen3.5.
- The language model is never changed by audio training. Audio is added by a small connector and an audio-only LoRA adapter that is switched on only when the prompt contains audio. Text and image answers are therefore exactly those of the chat model (checked bit for bit).
- 4ร fewer audio tokens: 12.5 per second of audio (125 for a 10-second clip). The connector also reads a middle Whisper layer (layer 18), which carries speaker information.
Result: stronger text and vision than Qolda-AVL-5B, better Russian and English speech recognition and better speech translation. On audio question answering (SAKURA, SpokenMQA) it is 2.7 points behind Qolda-AVL-5B. The numbers are in Comparison. How it was built is in Implementation.
The name "Qolda" reflects both the design and the purpose of the model family in Kazakh: "in hand" (าะพะปะดะฐ) for its compact accessibility, and "to support" (าะพะปะดะฐั) for its assistive nature.
Contents: Model details ยท Implementation ยท Comparison ยท How to use ยท Limitations ยท Citation ยท Contributions
Model details
| type | chat model with thinking (<think> โฆ </think>); input: text, image, audio; output: text |
| language model + vision | issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, unchanged (4.58 B) |
| audio encoder | Whisper-large-v3-turbo encoder, ISSAI's Kazakh fine-tune from Qolda-AVL-5B, frozen (0.64 B) |
| audio connector | trained, 13 M |
| audio-only LoRA | r = 16, ฮฑ = 32, 30.5 M; on only for prompts with audio |
| parameters | 5.26 B in total |
| audio input | one clip per prompt, 0.3โ30 s, any sample rate (resampled to 16 kHz) |
| precision | bfloat16 |
| languages | Kazakh, Russian, English |
Implementation
1. Audio encoder
Qolda-AVL-5B's Whisper-large-v3-turbo encoder (ISSAI's Kazakh fine-tune), copied unchanged and kept frozen. It gives 50 frames per second of audio.
2. Connector
average-pool ร4 โ LayerNorm โ Linear โ GELU โ Linear: 4 Whisper frames become one audio token, so one second of
audio is 12.5 tokens in the language model. A second input reads Whisper's layer 18 through its own LayerNorm and a
projection that starts at zero. Middle layers of Whisper keep more speaker information than its last layer: a
linear probe tells speaker gender on SAKURA with 93% accuracy from layer 18, but only 51% from the last layer.
3. Audio-only LoRA
A LoRA adapter (r = 16) on the attention, linear-attention and MLP projections of all 32 layers of the language model. It is switched on for prompts with audio and off otherwise. With it off, the model is the chat model itself: text and image logits are bit-identical (maximum difference 0.0 on Kazakh, Russian, English and image prompts).
4. Training
The language model's own weights and the audio encoder stay frozen throughout.
| stage | trained | data | settings |
|---|---|---|---|
| 1 | connector | one pass over 2.22 M examples | learning rate 1e-4, cosine; 50% of answers with thinking |
| 2 | connector + layer-18 input + LoRA | 425 k examples | learning rate 5e-5 (connector), 1e-4 (LoRA) |
Training data
ISSAI's Qolda-AVL audio training set, plus our own self-distilled spoken instructions. One pass:
| group | examples | share |
|---|---|---|
| speech recognition, English / Russian / Kazakh (Kazakh ร2) | 652 k / 558 k / 132 k | 29% / 25% / 12% |
| language identification (17 languages) | 244 k | 11% |
| speech translation, English โ Kazakh | 134 k | 6% |
| spoken instructions (answers written by the chat model from the transcript) | 134 k | 6% |
| everyday sounds, music | 63 k (ร2), 44 k | 6%, 2% |
| gender, emotion, animal sounds (ร3) | 30 k | 4% |
1% of the rows were held out for testing. Training clips that also occur in the SAKURA test set were removed.
Comparison
Three models: Qwen3.5-4B (no audio), ISSAI's earlier Qolda-AVL-5B, and this model.
At a glance
| Qwen3.5-4B | Qolda-AVL-5B | this model | |
|---|---|---|---|
| language model | Qwen3.5-4B | Qwen3-VL-4B-Thinking, trained further | issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, frozen |
| tokens per word, Kazakh / Russian / English | 3.48 / 1.89 / 1.25 | โ | 1.76 / 1.72 / 1.25 |
| audio tokens per second | โ | 50 | 12.5 |
| Kazakh text average (7 benchmarks, below) | 76.44 | 70.88 | 78.33 |
| vision average (9 benchmarks, below) | 74.2 | 68.2 | 74.2 |
| audio question answering (20 tasks, below) | โ | 68.3 | 65.6 |
Audio question answering
SAKURA (Kazakh and Russian, single- and multi-hop) and SpokenMQA (Kazakh spoken maths), from ISSAI's audio benchmark collection. All items (SAKURA 497โ500 per task, SpokenMQA 2,168), our runs of both models with the same prompts and settings: thinking on, temperature 0.7, top-p 0.95, top-k 20, up to 2,048 new tokens (4,096 for SpokenMQA), answer in JSON. An answer that is missing or cannot be parsed counts as wrong. Accuracy in %.
| category | tasks | Qolda-AVL-5B | this model |
|---|---|---|---|
| language | 4 | 90.5 | 90.0 |
| emotion | 4 | 37.0 | 37.5 |
| gender | 4 | 73.8 | 72.7 |
| animal | 4 | 53.8 | 48.1 |
| spoken maths (SpokenMQA) | 4 | 86.3 | 79.7 |
| average of 20 tasks | 68.3 | 65.6 |
Click for all 20 tasks
| task | language | hops | Qolda-AVL-5B | this model |
|---|---|---|---|---|
| SAKURA ยท Language | Kazakh | single | 95.2 | 94.0 |
| Kazakh | multi | 85.4 | 84.4 | |
| Russian | single | 95.2 | 95.6 | |
| Russian | multi | 86.0 | 86.0 | |
| SAKURA ยท Emotion | Kazakh | single | 41.4 | 38.2 |
| Kazakh | multi | 32.2 | 34.2 | |
| Russian | single | 38.4 | 41.8 | |
| Russian | multi | 35.8 | 35.8 | |
| SAKURA ยท Gender | Kazakh | single | 74.8 | 75.0 |
| Kazakh | multi | 71.8 | 71.0 | |
| Russian | single | 74.4 | 75.6 | |
| Russian | multi | 74.0 | 69.2 | |
| SAKURA ยท Animal | Kazakh | single | 57.7 | 49.7 |
| Kazakh | multi | 49.1 | 45.5 | |
| Russian | single | 57.5 | 46.5 | |
| Russian | multi | 51.1 | 50.7 | |
| SpokenMQA ยท short digit | Kazakh | 92.0 | 81.0 | |
| SpokenMQA ยท long digit | Kazakh | 86.0 | 77.9 | |
| SpokenMQA ยท single-step reasoning | Kazakh | 90.9 | 86.3 | |
| SpokenMQA ยท multi-step reasoning | Kazakh | 76.3 | 73.7 |
Note on the sound tasks: the Qolda-AVL audio training set contains 459 of the 500 SAKURA AnimalQA clips and 282 of the 500 EmotionQA clips (from public sound and emotion datasets). We removed them from our training, so for this model they are unseen.
Speech recognition and translation
The 1% hold-out of the audio training data (500 clips per language; 300 for translation), never used in our training. Same prompt for both models. This model: thinking off (the recommended mode for transcription). Qolda-AVL-5B: its default thinking mode. Scored on the clips where Qolda-AVL-5B gave an answer within 2,048 new tokens (4,096 for translation): all 500 Kazakh, 494 English, 467 Russian and 263 translation clips.
| task | Qolda-AVL-5B | this model |
|---|---|---|
| Kazakh speech recognition, WER โ | 10.8% | 10.9% |
| Russian speech recognition, WER โ | 11.3% | 4.7% |
| English speech recognition, WER โ | 11.6% | 6.1% |
| English speech โ Kazakh text, chrF โ | 40.3 | 48.0 |
Vision benchmarks
Rule-scored benchmarks, our runs of all three models with the same harness and settings (thinking on, temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B was run as its vision-language part. Accuracy in %.
| benchmark | lang | n | Qwen3.5-4B | Qolda-AVL-5B | this model |
|---|---|---|---|---|---|
| MMBench | kk | 4,329 | 87.3 | 87.3 | 88.5 |
| OCRBench | kk | 441 | 31.5 | 23.6 | 32.0 |
| AI2D | kk | 3,088 | 78.2 | 72.8 | 79.9 |
| MMStar | kk | 1,499 | 71.4 | 64.4 | 70.7 |
| MMStar | ru | 1,500 | 62.4 | 56.1 | 62.2 |
| MMBench | en | 4,377 | 89.7 | 88.3 | 89.6 |
| OCRBench | en | 1,000 | 82.3 | 74.5 | 82.0 |
| AI2D | en | 3,088 | 87.5 | 79.0 | 87.4 |
| MMStar | en | 1,500 | 77.1 | 67.5 | 75.2 |
These settings are Qwen3.5's recommended ones. Qolda-AVL-5B's own card reports its results with its own settings, and some of them are higher than in our run (for example MMStar-kk 67.5, OCRBench-kk 30.6).
Text benchmarks
Thinking on, full test sets, Kazakh benchmarks from ISSAI's Qolda language suite. Qwen3.5-4B and this model: our runs (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B: from its model card, which does not state its settings, so this part is for reference. Accuracy in %.
| benchmark | lang | Qwen3.5-4B | Qolda-AVL-5B | this model |
|---|---|---|---|---|
| KazMMLU | kk | 72.5 | 69.3 | 78.4 |
| MMLU | kk | 77.7 | 73.1 | 79.2 |
| MMLU-Pro | kk | 70.0 | 62.2 | 70.0 |
| MMLU-Redux | kk | 80.6 | 76.2 | 81.1 |
| GPQA | kk | 61.7 | 47.9 | 62.1 |
| GSM8K | kk | 88.0 | 85.8 | 90.3 |
| Belebele | kk | 84.6 | 81.7 | 87.2 |
| Kazakh average | 76.44 | 70.88 | 78.33 | |
| MMLU | en | 85.9 | 79.1 | 84.5 |
| MMLU-Pro | en | 79.4 | 71.3 | 77.6 |
| GSM8K | en | 96.2 | 95.2 | 96.1 |
All text results of this model, with more benchmarks, are on the card of issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer. They hold here unchanged, because text answers are identical.
What it shows
- Text and vision: the gains of the new chat model carry over unchanged: +7.5 points over Qolda-AVL-5B on the Kazakh text average (from its card), and +6.0 on the vision average in our runs (on par with Qwen3.5-4B).
- Speech recognition and translation: on par in Kazakh; clearly lower error in Russian and English, and better English โ Kazakh translation.
- Audio question answering: 2.7 points behind. The gap is mostly spoken maths (โ6.6) and animal sounds (โ5.7); language, emotion and gender are within about 1 point. Qolda-AVL-5B trains its whole language model on audio; this model keeps it frozen so that text and vision stay exactly as they are.
How to use
Requirements: transformers 5.x (tested with 5.12.1), torch, peft, soundfile, scipy, pillow.
One GPU with about 13 GB of memory.
pip install torch "transformers>=5" peft soundfile scipy pillow
Download the model and load it with the provided loader:
import sys
from huggingface_hub import snapshot_download
path = snapshot_download("issai/Qolda-AVL-5B-OptimizedTokenizer")
sys.path.insert(0, path)
from load_avl import load, ask, ask_audio
model = load()
The model thinks by default (<think> โฆ </think> before the answer); pass think=False to answer directly.
Language:
print(ask(model, "y = (lnx)^2 ััะฝะบัะธัััะฝัาฃ ัััะฝะดัััะฝ ัะฐะฟ. JSON ัะพัะผะฐััะฝะดะฐ ะถะฐัะฐะฟ ะฑะตั: {'answer': '...'}"))
Vision-Language:
print(ask(model, "ะกััะตััั ะตะณะถะตะน-ัะตะณะถะตะนะปั ัะธะฟะฐััะฐะฟ ะฑะตั. ะะตัะต ะถัะปาั ะบำฉััะฟ ัาฑัััาฃ ะถำะฝะต ะพะปะฐัะดัาฃ ัาฏััะตัั าะฐะฝะดะฐะน?",
image=f"{path}/assets/sample_image.jpg"))
Audio-Language:
Give a text instruction together with the audio, describing what to do with it.
prompt = """ะะฐัะตะผะฐัะธะบะฐะปัา ะตัะตะฟัั ัะตั.
Respond ONLY with this JSON format: {"explanation": "<your step-by-step reasoning>", "answer": <integer or float number>}
The answer must be a number (integer or float). No text, no units, just the number.
"""
print(ask_audio(model, f"{path}/assets/sample_audio.wav", prompt))
# transcription: thinking off is faster
print(ask_audio(model, f"{path}/assets/sample_audio.wav", "ะัั ะฐัะดะธะพะดะฐาั ัำฉะนะปะตัะดั ะผำััะฝะณะต ะฐะนะฝะฐะปะดัััาฃัะท.", think=False))
From the shell: python load_avl.py clip.wav "Transcribe this audio."
Serving text and images with vLLM: the folder is also a normal Qwen3.5 checkpoint (the audio files are ignored),
so vllm serve issai/Qolda-AVL-5B-OptimizedTokenizer --reasoning-parser qwen3 serves text and image prompts. Audio
needs load_avl.py.
Files
| file | what |
|---|---|
model.safetensors, config.json, tokenizer*, chat_template.jinja, *preprocessor_config.json |
the chat model with vision, unchanged |
audio_encoder/ |
Whisper encoder weights and feature-extractor settings |
audio_connector.safetensors |
the connector |
audio_lora.safetensors |
the audio-only LoRA |
avl_config.json, load_avl.py, avl_code/ |
how the parts fit together, and the loader |
assets/ |
the architecture figure and the sample image and audio used above |
Limitations
- Audio question answering is 2.7 points behind Qolda-AVL-5B; spoken numbers (SpokenMQA digit tasks) and animal sounds are the weakest.
- One audio clip per prompt, at most 30 s. The loader does not take audio and an image in the same prompt. No audio output.
- Audio needs the provided loader (
load_avl.py); vLLM serves only text and images. - The vision encoder was not trained on Kazakh data: reading Kazakh text in images (OCRBench-kk โ 32%) is much weaker than in English (โ 82%).
- Not measured: Russian text benchmarks, audio captioning (WavCaps).
Contributions
- Almubdi Mutaikhan: language model, audio connector, audio-only LoRA, training, evaluation, release.
- Batyr Arystanbekov: ISSAI's training pipeline and data, including the Qolda-AVL audio encoder and audio training data.
- Huseyin Atakan Varol: supervision.
License
Apache-2.0. Built on Qwen/Qwen3.5-4B and the audio encoder of issai/Qolda-AVL-5B. The training data have their own licenses.
Citation
@misc{mutaikhan2026qoldaavl5bopt,
title = {Qolda-AVL-5B-OptimizedTokenizer},
author = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/issai/Qolda-AVL-5B-OptimizedTokenizer}}
}
The audio encoder and audio training data come from Qolda-AVL. Please also cite:
@article{qolda-avl-bdcc,
AUTHOR = {Arystanbekov, Batyr and Maxutov, Akylbek and Nurimanov, Aspandiyar and Varol, Huseyin Atakan},
TITLE = {Extending a VisionโLanguage Model with Audio Understanding: Introducing Qolda-AVL for the Kazakh Language},
JOURNAL = {Big Data and Cognitive Computing},
VOLUME = {10},
YEAR = {2026},
NUMBER = {6},
ARTICLE-NUMBER = {192},
URL = {https://www.mdpi.com/2504-2289/10/6/192},
ISSN = {2504-2289},
DOI = {10.3390/bdcc10060192}
}
- Downloads last month
- 56