KE-Team
/

Ke-Omni-R-3B

 base_model:
 - Qwen/Qwen2.5-Omni-3B
 pipeline_tag: audio-text-to-text
+---
+# Ke-Omni-R: Achieving Advanced Audio Reasoning with a Concise 50-Words Think Process
+If you wish to train or perform inference with the model, please visit the GitHub repository: [https://github.com/shuaijiang/Ke-Omni-R/](https://github.com/shuaijiang/Ke-Omni-R/).
+If you find this model helpful, please like this model and star our GitHub.
+Ke-Omni-R is an advanced audio reasoning model built upon [Qwen2.5-Omni-3B](https://github.com/QwenLM/Qwen2.5-Omni). With only 10k post-training samples, Ke-Omni-R has achieved state-of-the-art performance on the MMAU *Test-mini* and *Test* benchmarks. Key insights from its development include:
+- **GRPO Algorithm**: The GRPO algorithm significantly enhances the performance of the already strong base model (Qwen2.5-Omni-7B), demonstrating superior generalization even in unseen speech domains.
+- **Think Process**: Incorporating a concise think process (less than 50 words) plays a crucial role in improving reasoning capabilities.
+- **KL Divergence**: Slight improvements were observed during GRPO training by leveraging KL divergence.
+- **Domain Ratio vs. Data Volume**: Domain diversity outweighs data volume. We utilized only 10k samples, with 5k randomly selected from AVQA and another 5k from MusicBench.
+## Performance: Accuracies (%) on MMAU Test-mini and Test benchmark
+| Model                                 | Method                | Sound (Test-mini) | Sound (Test)  | Music (Test-mini) | Music (Test)  | Speech (Test-mini) | Speech (Test)  | Average (Test-mini) | Average (Test)  |
+|---------------------------------------|-----------------------|-----------|-------|-----------|-------|-----------|------|------------|-------|
+| -                                     | Human\*               | 86.31     | -     | 78.22     | -     | 82.17     | -     | 82.23     | -     |
+| Gemini Pro 2.0 Flash                  | Direct Inference\*    | 56.46     | 61.73 | 58.68     | 56.53 | 51.65     | 61.53 | 55.60     | 59.93 |
+| Audio Flamingo 2                      | Direct Inference\*    | 61.56     | 65.10 | **73.95** |**72.90**| 30.93     | 40.26 | 55.48     | 59.42 |
+| GPT4o + Strong Cap.                   | Direct Inference\*    | 57.35     | 55.83 | 49.70     | 51.73 | 64.86     | **68.66** | 57.30     | 58.74 |
+| Llama-3-8B-Instruct + Strong Cap.     | Direct Inference\*    | 50.75     | 49.10 | 48.93     | 48.93 | 55.25     | 62.70 | 52.10     | 53.57 |
+| Qwen2-Audio-7B-Instruct               | Direct Inference\*    | 54.95     | 45.90 | 50.98     | 53.26 | 42.04     | 45.90 | 49.20     | 52.50 |
+| SALAMONN                              | Direct Inference\*    | 41.00     | 40.30 | 34.80     | 33.76 | 25.50     | 24.24 | 33.70     | 32.77 |
+| Audio-Reasoner(Qwen2-Audio-7B-Instruct) | \[1\]               | 60.06     | -     | 64.30     | -     | 60.70     | -     | 61.71     | -     |
+| Audio-Cot(Qwen2-Audio-7B-Instruct)    | \[2\]                 | 61.86     | -     | 56.29     | -     | 55.26     | -     | 57.80     | -     |
+| R1-AQA(Qwen2-Audio-7B-Instruct)       | \[3\]                 | 68.77     | 69.76 | 64.37     | 61.40 | 63.66     | 62.70 | 65.60     | 64.36 |
+| Qwen2.5-Omni-7B                       | \[4\]                 | 67.87     | -     | 69.16     | -     | 59.76     | -     | 65.60     | -     |
+| Ke-Omni-R-3B(Qwen2.5-Omni-3B)         | GRPO(ours)            | - | - | - | - |-  | - | - |- |
+| Ke-Omni-R(Qwen2.5-Omni-7B)            | GRPO(ours)            | **69.37** | **71.90** | 69.46 | 67.13 |**67.87**  | 67.10 | **68.90** |**68.71** |
+Note:
+- \* The data are sourced from the [MMAU leaderboard](https://sakshi113.github.io/mmau_homepage/#leaderboard).
+- \[1\] Xie, Zhifei, et al. "Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models." arXiv preprint arXiv:2503.02318.
+- \[2\] Ma, Ziyang, et al. "Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model." arXiv preprint arXiv:2501.07246.
+- \[3\] Li, Gang, et al. "Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering." arXiv preprint arXiv:2503.11197
+- \[4\] Xu, Jin, et al. "Qwen2.5-Omni Technical Report." arXiv preprint arXiv:2503.20215
+## Usage
+```python
+from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
+from qwen_omni_utils import process_mm_info
+# You can directly insert a local file path, a URL, or a base64-encoded audio into the position where you want in the text.
+messages = [
+  # Audio
+    ## Local audio path
+    [{"role": "system", "content":[{"type": "text", "text": "You are Qwen, a virtual human developed by the Qwen Team, Alibaba Group, capable of perceiving auditory and visual inputs, as well as generating text and speech."}]},
+     {"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBtBeR6B00_000000.wav"}, {"type": "text", "text": "Please describe this audio."}]}],
+    [{"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBtBeR6B00_000000.wav"}, {"type": "text", "text": "What is the main source of sound in the audio? ['aircraft', 'Car', 'Tank', 'Missile'] Output the thinking process (less than 50 words) in <think> </think> and final answer in <answer> </answer>."}]}],
+    [{"role": "user", "content": [{"type": "audio", "audio": "/path_to_avqa_wavs/-IBXTktoom8_000030.wav"}, {"type": "text", "text": "What animal is the main source of sound in the video? ['dog', 'wasp', 'honeybee', 'dragonfly'] Output the thinking process (less than 50 words) in <think> </think> and final answer in <answer> </answer>."}]}],
+]
+model = Qwen2_5OmniForConditionalGeneration.from_pretrained('KE-Team/Ke-Omni-R-3B')
+processor = Qwen2_5OmniProcessor.from_pretrained(model_path)
+text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+print(text)
+audios, images, videos = process_mm_info(messages, use_audio_in_video=False)
+inputs = processor(text=text, images=images, videos=videos, audio=audios, padding=True, return_tensors="pt")
+generation = model.generate(**inputs, thinker_temperature=0, thinker_do_sample=False)
+generated_ids = generation[:, inputs.input_ids.size(1):]
+completions = processor.batch_decode(generated_ids, skip_special_tokens=True)
+print(completions)
+```
+the output should be
+```
+["Well, it sounds like there's a car accelerating. You can hear the engine revving up, and there's a bit of a thump or thud sound too. It might be the car hitting something or just a part of the acceleration process. It gives off a sense of speed and power. What do you think about it? Do you have any other audio samples you want to talk about?", '<think>The audio features a vehicle accelerating and revving, which is characteristic of a car. The sound is consistent with a car engine, not an aircraft, tank, or missile.</think>\n<answer>Car</answer>', "<think>The main source of sound is a buzzing insect, which is consistent with the size and sound of a honeybee. The other options don't match the sound or context.</think>\n<answer>honeybee</answer>"]
+```
+## Acknowledgements
+We express our gratitude to the following projects and teams for their contributions:
+- **R1-AQA**: Referenced the GRPO-based training implementation from [R1-AQA](https://github.com/xiaomi-research/r1-aqa).
+- **Qwen Team**: Special thanks to the [Qwen2.5-Omni](https://github.com/QwenLM/Qwen2.5-Omni) model for providing a robust foundation.
+- **Datasets**:
+  - [AVAQ](https://mn.cs.tsinghua.edu.cn/avqa/)
+  - [MusicBench](https://amaai-lab.github.io/mustango/)
+  - [MMAU](https://github.com/Sakshi113/MMAU/)
+## Citation
+```bib
+@misc{zhao2025keomnir,
+  author = {Zhao, Shuaijiang and Guo, Tingwei and Wen, Cheng and Xiang, Bajian and Zou, Wei},
+  title = {Ke-Omni-R: Achieving Advanced Audio Reasoning with a Concise 50-Words Think Process},
+  year = {2025},
+  publisher = {GitHub},
+  journal = {GitHub Repository},
+  howpublished = {\url{https://github.com/shuaijiang/Ke-Omni-R}},
+}
+```