--- license: apache-2.0 language: - en - zh tags: - audio - speech - multimodal - audio-language-model - asr - speech-recognition library_name: transformers pipeline_tag: audio-text-to-text base_model: - Qwen/Qwen3-1.7B-Base - openai/whisper-large-v3 ---

Eureka-Audio-Instruct

HuggingFace   ModelScope   Paper

We present **Eureka-Audio**, a compact yet high-performance audio language model that achieves competitive performance against models that are **4 to 18 times larger** across a broad range of audio understanding benchmarks. Despite containing only **1.7B parameters**, Eureka-Audio demonstrates strong performance on automatic speech recognition (ASR), audio understanding, and dense audio captioning, matching or surpassing multiple 7B to 30B audio and omni-modal baselines. ## News * Feb 25, 2026: We release the inference code and model weights of [Eureka-Audio-Instruct](https://huggingface.co/cslys1999/Eureka-Audio-Instruct). * Feb 17, 2026: We release the technical report of [Eureka-Audio](https://arxiv.org/abs/2602.13954). ## Table of Contents - [Introduction](#introduction) - [Architecture Overview](#architecture-overview) - [Quick Start](#quick-start) - [Evaluation](#evaluation) - [Automatic Speech Recognition](#automatic-speech-recognition-asr) - [Audio Understanding](#audio-understanding) - [Dense Audio Captioning](#dense-audio-captioning) - [License](#license) - [Acknowledgements](#acknowledgements) - [Citation](#citation) ## Introduction Eureka-Audio is designed as a lightweight yet powerful audio foundation model capable of handling a wide variety of audio understanding tasks within a single unified framework. Key features include: * **Lightweight yet Powerful:** Achieve competitive results with only **1.7B parameters**, delivering up to **3.7x faster** decoding speed compared to larger models. * **Universal Audio Understanding:** Handle diverse tasks like automatic speech recognition (ASR), audio question answering, audio captioning, speech emotion recognition, and sound event classification. * **Competitive Performance:** Achieve competitive performance against models that are 4 to 18 times larger, matching or surpassing multiple 7B-30B audio and omni-modal baselines. * **DataFlux Pipeline:** A closed-loop audio instruction data synthesis and verification pipeline that constructs high-quality, logically consistent supervision from raw audio. * **Sparse MoE Adapter:** A novel sparsely activated Mixture-of-Experts adapter that explicitly accounts for audio heterogeneity and alleviates cross-modal optimization conflicts. * **Open-Source:** Release the code and model checkpoints for community research and development. ## Architecture Overview

Eureka-Audio consists of three main components: 1. **Audio Encoder:** A Whisper-based audio encoder that encodes raw waveforms into high-temporal-resolution acoustic representations, capturing fine-grained perceptual and semantic information present in the audio signal. 2. **Sparse MoE Adapter:** A Mixture-of-Experts adapter that maps audio representations into the embedding space of the language model. This design explicitly models the heterogeneity of audio signals at both the semantic and acoustic levels, mitigating optimization conflicts while improving representational efficiency. 3. **Language Model Backbone:** Qwen3-1.7B-base serves as the language backbone. After alignment via the MoE Adapter, audio embeddings are concatenated with text token embeddings and jointly modeled by the backbone in a standard autoregressive manner. ## Getting Started ### Installation ```bash git clone https://github.com/Alittleegg/Eureka-Audio.git cd Eureka-Audio pip install -r requirements.txt ``` ## Quick Start This example demonstrates basic usage for generating text from audio. ```python """ Eureka-Audio Local Inference Script Usage: python infer_local.py --audio_path test_wav/0.wav --prompt "Descript The audio." """ import os import sys import argparse from eureka_infer.api import EurekaAudio def main(): parser = argparse.ArgumentParser(description="Eureka-Audio Local Inference") parser.add_argument("--model_path", type=str, default="Eureka-Audio-Instruct", help="Path to the model checkpoint") parser.add_argument("--audio_path", type=str, required=True, help="Path to the audio file") parser.add_argument("--prompt", type=str, default="Descript The audio.", help="User prompt") parser.add_argument("--max_new_tokens", type=int, default=512, help="Maximum number of new tokens to generate") parser.add_argument("--device", type=str, default="cuda:0", help="Device to use (cuda:0/cpu)") args = parser.parse_args() print(f"Loading model from {args.model_path}...") model = EurekaAudio(model_path=args.model_path, device=args.device) # Build messages messages = [ { "role": "user", "content": [ {"type": "audio_url", "audio_url": {"url": args.audio_path}}, {"type": "text", "text": args.prompt} ] } ] print(f"Processing audio: {args.audio_path}") print(f"Prompt: {args.prompt}") print("Generating response...") response = model.generate( messages, max_new_tokens=args.max_new_tokens, temperature=0.0, top_p=0.0, top_k=0, do_sample=False, ) print("\n" + "="*50) print(f"Response:\n{response}") print("="*50) if __name__ == "__main__": main() ``` ### Using HuggingFace Transformers ```python from transformers import AutoModelForCausalLM, AutoTokenizer import torch # Load model with trust_remote_code model = AutoModelForCausalLM.from_pretrained( "cslys1999/Eureka-Audio-Instruct", torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True, ) ``` ## Evaluation Eureka-Audio achieves competitive performance across a wide range of audio benchmarks despite having only 1.7B parameters.

### Automatic Speech Recognition (ASR)
Datasets Type Model Size WER/CER ↓
LibriSpeech
test-clean | test-other		 Omni Qwen3-Omni-Instruct 30B-A3B 1.60 | 2.93
Ming-Lite-Omni-1.5 19B-A2.8B 1.90 | 3.54
MiniCPM-o 9B 2.01 | 4.87
Qwen2.5-Omni-7B 7B 1.53 | 3.19
Qwen2.5-Omni-3B 3B 1.68 | 3.90
Audio Step-Audio-2-mini 8B 1.41 | 2.76
Audio Flamingo 3 8B 1.39 | 2.96
Qwen2-Audio 7B 1.74 | 4.01
Kimi-Audio-7B-Instruct 7B 1.33 | 2.57
Ours Eureka-Audio-Base 1.7B 1.59 | 3.34
Eureka-Audio-Instruct 1.7B 1.46 | 3.24
Fleurs-en Omni Qwen3-Omni-Instruct 30B-A3B 5.04
Ming-Lite-Omni-1.5 19B-A2.8B 5.82
MiniCPM-o 9B 6.18
Qwen2.5-Omni-7B 7B 5.49
Qwen2.5-Omni-3B 3B 5.65
Audio Step-Audio-2-mini 8B 4.51
Audio Flamingo 3 8B 6.30
Qwen2-Audio 7B 6.92
Kimi-Audio-7B-Instruct 7B 6.11
Ours Eureka-Audio-Base 1.7B 5.73
Eureka-Audio-Instruct 1.7B 5.39
AISHELL-2 ios Omni Qwen3-Omni-Instruct 30B-A3B 2.63
Ming-Lite-Omni-1.5 19B-A2.8B 2.66
MiniCPM-o 9B 3.42
Qwen2.5-Omni-7B 7B 2.58
Qwen2.5-Omni-3B 3B 2.77
Audio Step-Audio-2-mini 8B 2.33
Qwen2-Audio 7B 3.08
Kimi-Audio-7B-Instruct 7B 2.80
Ours Eureka-Audio-Base 1.7B 3.17
Eureka-Audio-Instruct 1.7B 3.10
WenetSpeech
test-meeting | test-net		 Omni Qwen3-Omni-Instruct 30B-A3B 6.12 | 5.29
Ming-Lite-Omni-1.5 19B-A2.8B 5.96 | 6.26
MiniCPM-o 9B 15.53 | 7.68
Qwen2.5-Omni-7B 7B 8.43 | 7.10
Qwen2.5-Omni-3B 3B 8.53 | 7.14
Audio Step-Audio-2-mini 8B 5.43 | 5.50
Qwen2-Audio 7B 8.40 | 8.00
Kimi-Audio-7B-Instruct 7B 6.38 | 7.17
Ours Eureka-Audio-Base 1.7B 10.37 | 8.63
Eureka-Audio-Instruct 1.7B 9.14 | 7.55
### Audio Understanding
Datasets Type Model Size Performance ↑
Knowledge
MMSU | OpenBookQA		 Omni Qwen3-Omni-Instruct 30B-A3B 77.00 | 92.31
Ming-Lite-Omni-1.5 19B-A2.8B 47.00 | 69.67
MiniCPM-o 9B 54.55 | 79.12
Qwen2.5-Omni-7B 7B 61.22 | 81.53
Qwen2.5-Omni-3B 3B 53.41 | 77.36
Audio Step-Audio-2-mini 8B 55.14 | 75.60
Audio Flamingo 3 8B 47.07 | 61.54
Qwen2-Audio 7B 35.75 | 49.67
Kimi-Audio-7B-Instruct 7B 61.26 | 84.18
Ours Eureka-Audio-Base 1.7B 38.03 | 52.53
Eureka-Audio-Instruct 1.7B 55.63 | 69.23
Safety
AdvBench		 Omni Qwen3-Omni-Instruct 30B-A3B 99.61
Ming-Lite-Omni-1.5 19B-A2.8B 99.23
MiniCPM-o 9B 95.76
Qwen2.5-Omni-7B 7B 96.92
Qwen2.5-Omni-3B 3B 89.80
Audio Step-Audio-2-mini 8B 93.08
Audio Flamingo 3 8B 98.26
Qwen2-Audio 7B 98.84
Kimi-Audio-7B-Instruct 7B 100.00
Ours Eureka-Audio-Instruct 1.7B 99.81
Instruction
IFEval		 Omni Qwen3-Omni-Instruct 30B-A3B 81.17
Ming-Lite-Omni-1.5 19B-A2.8B 53.68
MiniCPM-o 9B 41.72
Qwen2.5-Omni-7B 7B 39.84
Qwen2.5-Omni-3B 3B 32.97
Audio Step-Audio-2-mini 8B 43.54
Audio Flamingo 3 8B 32.27
Qwen2-Audio 7B 26.24
Kimi-Audio-7B-Instruct 7B 47.91
Ours Eureka-Audio-Instruct 1.7B 53.21
Paralinguistic
MMAU | MMAR		 Omni Qwen3-Omni-Instruct 30B-A3B 74.57 | 67.10
Ming-Lite-Omni-1.5 19B-A2.8B 63.52 | 45.40
MiniCPM-o 9B 64.92 | 47.90
Qwen2.5-Omni-7B 7B 66.23 | 49.60
Qwen2.5-Omni-3B 3B 62.91 | 43.40
Audio Step-Audio-2-mini 8B 71.96 | 61.57
Audio Flamingo 3 8B 74.77 | 61.00
Qwen2-Audio 7B 59.80 | 37.90
Kimi-Audio-7B-Instruct 7B 72.86 | 57.40
Ours Eureka-Audio-Base 1.7B 63.42 | 46.80
Eureka-Audio-Instruct w/o DataFlux 1.7B 66.93 | 50.70
Eureka-Audio-Instruct 1.7B 74.67 | 56.20
### Dense Audio Captioning
Datasets Model Size MMAU | MMAR ↑
Dense Captioning Qwen3-Omni-Captioner 30B-A3B 56.68 | 46.40
Qwen3-Omni-Instruct 30B-A3B 48.24 | 36.90
Eureka-Audio-Instruct (Ours) 1.7B 52.96 | 41.70
## Audio Input Formats The model supports multiple audio input formats: - **Local file path**: `"path/to/audio.wav"` - **HTTP URL**: `"https://example.com/audio.wav"` - **Base64 data URI**: `"data:audio/wav;base64,"` Supported audio formats: WAV, MP3, FLAC, OGG, etc. (via torchaudio) ## License The model is based and modified from [Qwen3](https://github.com/QwenLM/Qwen3). Code derived from Qwen3 is licensed under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). Other parts of the code are licensed under the [MIT License](https://opensource.org/licenses/MIT). ## Acknowledgements We would like to thank the following projects for their contributions: * [Whisper](https://github.com/openai/whisper) - Audio encoder * [Qwen3](https://github.com/QwenLM/Qwen3) - Language model backbone * [Transformers](https://github.com/huggingface/transformers) - Model framework Thank you to all the open-source projects for their contributions! ## Citation If you find Eureka-Audio useful in your research or applications, please cite our technical report: ```bibtex @misc{zhang2026eurekaaudiotriggeringaudiointelligence, title={Eureka-Audio: Triggering Audio Intelligence in Compact Language Models}, author={Dan Zhang and Yishu Lei and Jing Hu and Shuwei He and Songhe Deng and Xianlong Luo and Danxiang Zhu and Shikun Feng and Rui Liu and Jingzhou He and Yu Sun and Hua Wu and Haifeng Wang}, year={2026}, eprint={2602.13954}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2602.13954}, } ``` ```bibtex @misc{lei2026moeadapterlargeaudio, title={MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free}, author={Yishu Lei and Shuwei He and Jing Hu and Dan Zhang and Xianlong Luo and Danxiang Zhu and Shikun Feng and Rui Liu and Jingzhou He and Yu Sun and Hua Wu and Haifeng Wang}, year={2026}, eprint={2601.02967}, archivePrefix={arXiv}, primaryClass={cs.SD}, url={https://arxiv.org/abs/2601.02967}, } ``` ## Contact Us For questions, issues, or collaboration inquiries, please feel free to open an issue on GitHub.