Qolda-AVL-5B-OptimizedTokenizer

An optimized new version of ISSAI's audio-vision-language model Qolda-AVL-5B.

Developed by Almubdi Mutaikhan with Batyr Arystanbekov (ISSAI), under the supervision of Prof. Huseyin Atakan Varol (ISSAI). See Citation.

ISSAI earlier released issai/Qolda-AVL-5B: a 5B audio-vision-language model for Kazakh, Russian and English. It extends Qwen3-VL-4B-Thinking with an audio branch: a fine-tuned Whisper encoder and an audio projection. All three modalities are adapted to Kazakh in staged training.

This model is its next version. It keeps Qolda-AVL's audio encoder and audio training data, and changes three things:

  1. A new language model: issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer (Qwen3.5, a Kazakh + Russian tokenizer, Russian in pretraining). Kazakh text needs 1.98ร— fewer tokens than with Qwen3.5.
  2. The language model is never changed by audio training. Audio is added by a small connector and an audio-only LoRA adapter that is switched on only when the prompt contains audio. Text and image answers are therefore exactly those of the chat model (checked bit for bit).
  3. 4ร— fewer audio tokens: 12.5 per second of audio (125 for a 10-second clip). The connector also reads a middle Whisper layer (layer 18), which carries speaker information.

Result: stronger text and vision than Qolda-AVL-5B, better Russian and English speech recognition and better speech translation. On audio question answering (SAKURA, SpokenMQA) it is 2.7 points behind Qolda-AVL-5B. The numbers are in Comparison. How it was built is in Implementation.

The name "Qolda" reflects both the design and the purpose of the model family in Kazakh: "in hand" (า›ะพะปะดะฐ) for its compact accessibility, and "to support" (า›ะพะปะดะฐัƒ) for its assistive nature.

Contents: Model details ยท Implementation ยท Comparison ยท How to use ยท Limitations ยท Citation ยท Contributions

Model details

type chat model with thinking (<think> โ€ฆ </think>); input: text, image, audio; output: text
language model + vision issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, unchanged (4.58 B)
audio encoder Whisper-large-v3-turbo encoder, ISSAI's Kazakh fine-tune from Qolda-AVL-5B, frozen (0.64 B)
audio connector trained, 13 M
audio-only LoRA r = 16, ฮฑ = 32, 30.5 M; on only for prompts with audio
parameters 5.26 B in total
audio input one clip per prompt, 0.3โ€“30 s, any sample rate (resampled to 16 kHz)
precision bfloat16
languages Kazakh, Russian, English

Implementation

architecture

1. Audio encoder

Qolda-AVL-5B's Whisper-large-v3-turbo encoder (ISSAI's Kazakh fine-tune), copied unchanged and kept frozen. It gives 50 frames per second of audio.

2. Connector

average-pool ร—4 โ†’ LayerNorm โ†’ Linear โ†’ GELU โ†’ Linear: 4 Whisper frames become one audio token, so one second of audio is 12.5 tokens in the language model. A second input reads Whisper's layer 18 through its own LayerNorm and a projection that starts at zero. Middle layers of Whisper keep more speaker information than its last layer: a linear probe tells speaker gender on SAKURA with 93% accuracy from layer 18, but only 51% from the last layer.

3. Audio-only LoRA

A LoRA adapter (r = 16) on the attention, linear-attention and MLP projections of all 32 layers of the language model. It is switched on for prompts with audio and off otherwise. With it off, the model is the chat model itself: text and image logits are bit-identical (maximum difference 0.0 on Kazakh, Russian, English and image prompts).

4. Training

The language model's own weights and the audio encoder stay frozen throughout.

stage trained data settings
1 connector one pass over 2.22 M examples learning rate 1e-4, cosine; 50% of answers with thinking
2 connector + layer-18 input + LoRA 425 k examples learning rate 5e-5 (connector), 1e-4 (LoRA)

Training data

ISSAI's Qolda-AVL audio training set, plus our own self-distilled spoken instructions. One pass:

group examples share
speech recognition, English / Russian / Kazakh (Kazakh ร—2) 652 k / 558 k / 132 k 29% / 25% / 12%
language identification (17 languages) 244 k 11%
speech translation, English โ†’ Kazakh 134 k 6%
spoken instructions (answers written by the chat model from the transcript) 134 k 6%
everyday sounds, music 63 k (ร—2), 44 k 6%, 2%
gender, emotion, animal sounds (ร—3) 30 k 4%

1% of the rows were held out for testing. Training clips that also occur in the SAKURA test set were removed.

Comparison

Three models: Qwen3.5-4B (no audio), ISSAI's earlier Qolda-AVL-5B, and this model.

At a glance

Qwen3.5-4B Qolda-AVL-5B this model
language model Qwen3.5-4B Qwen3-VL-4B-Thinking, trained further issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer, frozen
tokens per word, Kazakh / Russian / English 3.48 / 1.89 / 1.25 โ€” 1.76 / 1.72 / 1.25
audio tokens per second โ€” 50 12.5
Kazakh text average (7 benchmarks, below) 76.44 70.88 78.33
vision average (9 benchmarks, below) 74.2 68.2 74.2
audio question answering (20 tasks, below) โ€” 68.3 65.6

Audio question answering

SAKURA (Kazakh and Russian, single- and multi-hop) and SpokenMQA (Kazakh spoken maths), from ISSAI's audio benchmark collection. All items (SAKURA 497โ€“500 per task, SpokenMQA 2,168), our runs of both models with the same prompts and settings: thinking on, temperature 0.7, top-p 0.95, top-k 20, up to 2,048 new tokens (4,096 for SpokenMQA), answer in JSON. An answer that is missing or cannot be parsed counts as wrong. Accuracy in %.

category tasks Qolda-AVL-5B this model
language 4 90.5 90.0
emotion 4 37.0 37.5
gender 4 73.8 72.7
animal 4 53.8 48.1
spoken maths (SpokenMQA) 4 86.3 79.7
average of 20 tasks 68.3 65.6
Click for all 20 tasks
task language hops Qolda-AVL-5B this model
SAKURA ยท Language Kazakh single 95.2 94.0
Kazakh multi 85.4 84.4
Russian single 95.2 95.6
Russian multi 86.0 86.0
SAKURA ยท Emotion Kazakh single 41.4 38.2
Kazakh multi 32.2 34.2
Russian single 38.4 41.8
Russian multi 35.8 35.8
SAKURA ยท Gender Kazakh single 74.8 75.0
Kazakh multi 71.8 71.0
Russian single 74.4 75.6
Russian multi 74.0 69.2
SAKURA ยท Animal Kazakh single 57.7 49.7
Kazakh multi 49.1 45.5
Russian single 57.5 46.5
Russian multi 51.1 50.7
SpokenMQA ยท short digit Kazakh 92.0 81.0
SpokenMQA ยท long digit Kazakh 86.0 77.9
SpokenMQA ยท single-step reasoning Kazakh 90.9 86.3
SpokenMQA ยท multi-step reasoning Kazakh 76.3 73.7

Note on the sound tasks: the Qolda-AVL audio training set contains 459 of the 500 SAKURA AnimalQA clips and 282 of the 500 EmotionQA clips (from public sound and emotion datasets). We removed them from our training, so for this model they are unseen.

Speech recognition and translation

The 1% hold-out of the audio training data (500 clips per language; 300 for translation), never used in our training. Same prompt for both models. This model: thinking off (the recommended mode for transcription). Qolda-AVL-5B: its default thinking mode. Scored on the clips where Qolda-AVL-5B gave an answer within 2,048 new tokens (4,096 for translation): all 500 Kazakh, 494 English, 467 Russian and 263 translation clips.

task Qolda-AVL-5B this model
Kazakh speech recognition, WER โ†“ 10.8% 10.9%
Russian speech recognition, WER โ†“ 11.3% 4.7%
English speech recognition, WER โ†“ 11.6% 6.1%
English speech โ†’ Kazakh text, chrF โ†‘ 40.3 48.0

Vision benchmarks

Rule-scored benchmarks, our runs of all three models with the same harness and settings (thinking on, temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B was run as its vision-language part. Accuracy in %.

benchmark lang n Qwen3.5-4B Qolda-AVL-5B this model
MMBench kk 4,329 87.3 87.3 88.5
OCRBench kk 441 31.5 23.6 32.0
AI2D kk 3,088 78.2 72.8 79.9
MMStar kk 1,499 71.4 64.4 70.7
MMStar ru 1,500 62.4 56.1 62.2
MMBench en 4,377 89.7 88.3 89.6
OCRBench en 1,000 82.3 74.5 82.0
AI2D en 3,088 87.5 79.0 87.4
MMStar en 1,500 77.1 67.5 75.2

These settings are Qwen3.5's recommended ones. Qolda-AVL-5B's own card reports its results with its own settings, and some of them are higher than in our run (for example MMStar-kk 67.5, OCRBench-kk 30.6).

Text benchmarks

Thinking on, full test sets, Kazakh benchmarks from ISSAI's Qolda language suite. Qwen3.5-4B and this model: our runs (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5). Qolda-AVL-5B: from its model card, which does not state its settings, so this part is for reference. Accuracy in %.

benchmark lang Qwen3.5-4B Qolda-AVL-5B this model
KazMMLU kk 72.5 69.3 78.4
MMLU kk 77.7 73.1 79.2
MMLU-Pro kk 70.0 62.2 70.0
MMLU-Redux kk 80.6 76.2 81.1
GPQA kk 61.7 47.9 62.1
GSM8K kk 88.0 85.8 90.3
Belebele kk 84.6 81.7 87.2
Kazakh average 76.44 70.88 78.33
MMLU en 85.9 79.1 84.5
MMLU-Pro en 79.4 71.3 77.6
GSM8K en 96.2 95.2 96.1

All text results of this model, with more benchmarks, are on the card of issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer. They hold here unchanged, because text answers are identical.

What it shows

  • Text and vision: the gains of the new chat model carry over unchanged: +7.5 points over Qolda-AVL-5B on the Kazakh text average (from its card), and +6.0 on the vision average in our runs (on par with Qwen3.5-4B).
  • Speech recognition and translation: on par in Kazakh; clearly lower error in Russian and English, and better English โ†’ Kazakh translation.
  • Audio question answering: 2.7 points behind. The gap is mostly spoken maths (โˆ’6.6) and animal sounds (โˆ’5.7); language, emotion and gender are within about 1 point. Qolda-AVL-5B trains its whole language model on audio; this model keeps it frozen so that text and vision stay exactly as they are.

How to use

Requirements: transformers 5.x (tested with 5.12.1), torch, peft, soundfile, scipy, pillow. One GPU with about 13 GB of memory.

pip install torch "transformers>=5" peft soundfile scipy pillow

Download the model and load it with the provided loader:

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("issai/Qolda-AVL-5B-OptimizedTokenizer")
sys.path.insert(0, path)
from load_avl import load, ask, ask_audio

model = load()

The model thinks by default (<think> โ€ฆ </think> before the answer); pass think=False to answer directly.

Language:

print(ask(model, "y = (lnx)^2 ั„ัƒะฝะบั†ะธััั‹ะฝั‹าฃ ั‚ัƒั‹ะฝะดั‹ัั‹ะฝ ั‚ะฐะฟ. JSON ั„ะพั€ะผะฐั‚ั‹ะฝะดะฐ ะถะฐัƒะฐะฟ ะฑะตั€: {'answer': '...'}"))

Vision-Language:

print(ask(model, "ะกัƒั€ะตั‚ั‚ั– ะตะณะถะตะน-ั‚ะตะณะถะตะนะปั– ัะธะฟะฐั‚ั‚ะฐะฟ ะฑะตั€. ะะตัˆะต ะถั‹ะปา›ั‹ ะบำฉั€ั–ะฟ ั‚าฑั€ัั‹าฃ ะถำ™ะฝะต ะพะปะฐั€ะดั‹าฃ ั‚าฏัั‚ะตั€ั– า›ะฐะฝะดะฐะน?",
          image=f"{path}/assets/sample_image.jpg"))

Audio-Language:

Give a text instruction together with the audio, describing what to do with it.

prompt = """ะœะฐั‚ะตะผะฐั‚ะธะบะฐะปั‹า› ะตัะตะฟั‚ั– ัˆะตัˆ.
Respond ONLY with this JSON format: {"explanation": "<your step-by-step reasoning>", "answer": <integer or float number>}
The answer must be a number (integer or float). No text, no units, just the number.
"""
print(ask_audio(model, f"{path}/assets/sample_audio.wav", prompt))

# transcription: thinking off is faster
print(ask_audio(model, f"{path}/assets/sample_audio.wav", "ะžัั‹ ะฐัƒะดะธะพะดะฐา“ั‹ ัำฉะนะปะตัƒะดั– ะผำ™ั‚ั–ะฝะณะต ะฐะนะฝะฐะปะดั‹ั€ั‹าฃั‹ะท.", think=False))

From the shell: python load_avl.py clip.wav "Transcribe this audio."

Serving text and images with vLLM: the folder is also a normal Qwen3.5 checkpoint (the audio files are ignored), so vllm serve issai/Qolda-AVL-5B-OptimizedTokenizer --reasoning-parser qwen3 serves text and image prompts. Audio needs load_avl.py.

Files

file what
model.safetensors, config.json, tokenizer*, chat_template.jinja, *preprocessor_config.json the chat model with vision, unchanged
audio_encoder/ Whisper encoder weights and feature-extractor settings
audio_connector.safetensors the connector
audio_lora.safetensors the audio-only LoRA
avl_config.json, load_avl.py, avl_code/ how the parts fit together, and the loader
assets/ the architecture figure and the sample image and audio used above

Limitations

  • Audio question answering is 2.7 points behind Qolda-AVL-5B; spoken numbers (SpokenMQA digit tasks) and animal sounds are the weakest.
  • One audio clip per prompt, at most 30 s. The loader does not take audio and an image in the same prompt. No audio output.
  • Audio needs the provided loader (load_avl.py); vLLM serves only text and images.
  • The vision encoder was not trained on Kazakh data: reading Kazakh text in images (OCRBench-kk โ‰ˆ 32%) is much weaker than in English (โ‰ˆ 82%).
  • Not measured: Russian text benchmarks, audio captioning (WavCaps).

Contributions

  • Almubdi Mutaikhan: language model, audio connector, audio-only LoRA, training, evaluation, release.
  • Batyr Arystanbekov: ISSAI's training pipeline and data, including the Qolda-AVL audio encoder and audio training data.
  • Huseyin Atakan Varol: supervision.

License

Apache-2.0. Built on Qwen/Qwen3.5-4B and the audio encoder of issai/Qolda-AVL-5B. The training data have their own licenses.

Citation

@misc{mutaikhan2026qoldaavl5bopt,
  title        = {Qolda-AVL-5B-OptimizedTokenizer},
  author       = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/issai/Qolda-AVL-5B-OptimizedTokenizer}}
}

The audio encoder and audio training data come from Qolda-AVL. Please also cite:

@article{qolda-avl-bdcc,
  AUTHOR = {Arystanbekov, Batyr and Maxutov, Akylbek and Nurimanov, Aspandiyar and Varol, Huseyin Atakan},
  TITLE = {Extending a Visionโ€“Language Model with Audio Understanding: Introducing Qolda-AVL for the Kazakh Language},
  JOURNAL = {Big Data and Cognitive Computing},
  VOLUME = {10},
  YEAR = {2026},
  NUMBER = {6},
  ARTICLE-NUMBER = {192},
  URL = {https://www.mdpi.com/2504-2289/10/6/192},
  ISSN = {2504-2289},
  DOI = {10.3390/bdcc10060192}
}
Downloads last month
56
Safetensors
Model size
5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for issai/Qolda-AVL-5B-OptimizedTokenizer

Adapter
(1)
this model