cslys1999/Eureka-Audio-Instruct

Eureka-Audio-Instruct

   

We present Eureka-Audio, a compact yet high-performance audio language model that achieves competitive performance against models that are 4 to 18 times larger across a broad range of audio understanding benchmarks. Despite containing only 1.7B parameters, Eureka-Audio demonstrates strong performance on automatic speech recognition (ASR), audio understanding, and dense audio captioning, matching or surpassing multiple 7B to 30B audio and omni-modal baselines.

News

• Feb 25, 2026: We release the inference code and model weights of Eureka-Audio-Instruct.
• Feb 17, 2026: We release the technical report of Eureka-Audio.

Table of Contents

• Introduction
• Architecture Overview
• Quick Start
• Evaluation
  • Automatic Speech Recognition
  • Audio Understanding
  • Dense Audio Captioning
• License
• Acknowledgements
• Citation

Introduction

Eureka-Audio is designed as a lightweight yet powerful audio foundation model capable of handling a wide variety of audio understanding tasks within a single unified framework. Key features include:

• Lightweight yet Powerful: Achieve competitive results with only 1.7B parameters, delivering up to 3.7x faster decoding speed compared to larger models.
• Universal Audio Understanding: Handle diverse tasks like automatic speech recognition (ASR), audio question answering, audio captioning, speech emotion recognition, and sound event classification.
• Competitive Performance: Achieve competitive performance against models that are 4 to 18 times larger, matching or surpassing multiple 7B-30B audio and omni-modal baselines.
• DataFlux Pipeline: A closed-loop audio instruction data synthesis and verification pipeline that constructs high-quality, logically consistent supervision from raw audio.
• Sparse MoE Adapter: A novel sparsely activated Mixture-of-Experts adapter that explicitly accounts for audio heterogeneity and alleviates cross-modal optimization conflicts.
• Open-Source: Release the code and model checkpoints for community research and development.

Architecture Overview

Eureka-Audio consists of three main components:

1. Audio Encoder: A Whisper-based audio encoder that encodes raw waveforms into high-temporal-resolution acoustic representations, capturing fine-grained perceptual and semantic information present in the audio signal.

2. Sparse MoE Adapter: A Mixture-of-Experts adapter that maps audio representations into the embedding space of the language model. This design explicitly models the heterogeneity of audio signals at both the semantic and acoustic levels, mitigating optimization conflicts while improving representational efficiency.

3. Language Model Backbone: Qwen3-1.7B-base serves as the language backbone. After alignment via the MoE Adapter, audio embeddings are concatenated with text token embeddings and jointly modeled by the backbone in a standard autoregressive manner.

Getting Started

Installation

git clone https://github.com/Alittleegg/Eureka-Audio.git
cd Eureka-Audio
pip install -r requirements.txt

Quick Start

This example demonstrates basic usage for generating text from audio.

"""
Eureka-Audio Local Inference Script

Usage:
    python infer_local.py --audio_path test_wav/0.wav --prompt "Descript The audio."
"""

import os
import sys
import argparse

from eureka_infer.api import EurekaAudio


def main():
    parser = argparse.ArgumentParser(description="Eureka-Audio Local Inference")
    parser.add_argument("--model_path", type=str, default="Eureka-Audio-Instruct",
                        help="Path to the model checkpoint")
    parser.add_argument("--audio_path", type=str, required=True,
                        help="Path to the audio file")
    parser.add_argument("--prompt", type=str, default="Descript The audio.",
                        help="User prompt")
    parser.add_argument("--max_new_tokens", type=int, default=512,
                        help="Maximum number of new tokens to generate")
    parser.add_argument("--device", type=str, default="cuda:0",
                        help="Device to use (cuda:0/cpu)")
    args = parser.parse_args()

    print(f"Loading model from {args.model_path}...")
    model = EurekaAudio(model_path=args.model_path, device=args.device)

    # Build messages
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "audio_url", "audio_url": {"url": args.audio_path}},
                {"type": "text", "text": args.prompt}
            ]
        }
    ]

    print(f"Processing audio: {args.audio_path}")
    print(f"Prompt: {args.prompt}")
    print("Generating response...")

    response = model.generate(
        messages,
        max_new_tokens=args.max_new_tokens,
        temperature=0.0,
        top_p=0.0,
        top_k=0,
        do_sample=False,
    )

    print("\n" + "="*50)
    print(f"Response:\n{response}")
    print("="*50)


if __name__ == "__main__":
    main()

Using HuggingFace Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model with trust_remote_code
model = AutoModelForCausalLM.from_pretrained(
    "cslys1999/Eureka-Audio-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

Evaluation

Eureka-Audio achieves competitive performance across a wide range of audio benchmarks despite having only 1.7B parameters.

Automatic Speech Recognition (ASR)

Datasets	Type	Model	Size	WER/CER ↓
LibriSpeech test-clean | test-other	Omni	Qwen3-Omni-Instruct	30B-A3B	1.60 | 2.93
		Ming-Lite-Omni-1.5	19B-A2.8B	1.90 | 3.54
		MiniCPM-o	9B	2.01 | 4.87
		Qwen2.5-Omni-7B	7B	1.53 | 3.19
		Qwen2.5-Omni-3B	3B	1.68 | 3.90
	Audio	Step-Audio-2-mini	8B	1.41 | 2.76
		Audio Flamingo 3	8B	1.39 | 2.96
		Qwen2-Audio	7B	1.74 | 4.01
		Kimi-Audio-7B-Instruct	7B	1.33 | 2.57
	Ours	Eureka-Audio-Base	1.7B	1.59 | 3.34
		Eureka-Audio-Instruct	1.7B	1.46 | 3.24
Fleurs-en	Omni	Qwen3-Omni-Instruct	30B-A3B	5.04
		Ming-Lite-Omni-1.5	19B-A2.8B	5.82
		MiniCPM-o	9B	6.18
		Qwen2.5-Omni-7B	7B	5.49
		Qwen2.5-Omni-3B	3B	5.65
	Audio	Step-Audio-2-mini	8B	4.51
		Audio Flamingo 3	8B	6.30
		Qwen2-Audio	7B	6.92
		Kimi-Audio-7B-Instruct	7B	6.11
	Ours	Eureka-Audio-Base	1.7B	5.73
		Eureka-Audio-Instruct	1.7B	5.39
AISHELL-2 ios	Omni	Qwen3-Omni-Instruct	30B-A3B	2.63
		Ming-Lite-Omni-1.5	19B-A2.8B	2.66
		MiniCPM-o	9B	3.42
		Qwen2.5-Omni-7B	7B	2.58
		Qwen2.5-Omni-3B	3B	2.77
	Audio	Step-Audio-2-mini	8B	2.33
		Qwen2-Audio	7B	3.08
		Kimi-Audio-7B-Instruct	7B	2.80
	Ours	Eureka-Audio-Base	1.7B	3.17
		Eureka-Audio-Instruct	1.7B	3.10
WenetSpeech test-meeting | test-net	Omni	Qwen3-Omni-Instruct	30B-A3B	6.12 | 5.29
		Ming-Lite-Omni-1.5	19B-A2.8B	5.96 | 6.26
		MiniCPM-o	9B	15.53 | 7.68
		Qwen2.5-Omni-7B	7B	8.43 | 7.10
		Qwen2.5-Omni-3B	3B	8.53 | 7.14
	Audio	Step-Audio-2-mini	8B	5.43 | 5.50
		Qwen2-Audio	7B	8.40 | 8.00
		Kimi-Audio-7B-Instruct	7B	6.38 | 7.17
	Ours	Eureka-Audio-Base	1.7B	10.37 | 8.63
		Eureka-Audio-Instruct	1.7B	9.14 | 7.55

Audio Understanding

Datasets	Type	Model	Size	Performance ↑
Knowledge MMSU | OpenBookQA	Omni	Qwen3-Omni-Instruct	30B-A3B	77.00 | 92.31
		Ming-Lite-Omni-1.5	19B-A2.8B	47.00 | 69.67
		MiniCPM-o	9B	54.55 | 79.12
		Qwen2.5-Omni-7B	7B	61.22 | 81.53
		Qwen2.5-Omni-3B	3B	53.41 | 77.36
	Audio	Step-Audio-2-mini	8B	55.14 | 75.60
		Audio Flamingo 3	8B	47.07 | 61.54
		Qwen2-Audio	7B	35.75 | 49.67
		Kimi-Audio-7B-Instruct	7B	61.26 | 84.18
	Ours	Eureka-Audio-Base	1.7B	38.03 | 52.53
		Eureka-Audio-Instruct	1.7B	55.63 | 69.23
Safety AdvBench	Omni	Qwen3-Omni-Instruct	30B-A3B	99.61
		Ming-Lite-Omni-1.5	19B-A2.8B	99.23
		MiniCPM-o	9B	95.76
		Qwen2.5-Omni-7B	7B	96.92
		Qwen2.5-Omni-3B	3B	89.80
	Audio	Step-Audio-2-mini	8B	93.08
		Audio Flamingo 3	8B	98.26
		Qwen2-Audio	7B	98.84
		Kimi-Audio-7B-Instruct	7B	100.00
	Ours	Eureka-Audio-Instruct	1.7B	99.81
Instruction IFEval	Omni	Qwen3-Omni-Instruct	30B-A3B	81.17
		Ming-Lite-Omni-1.5	19B-A2.8B	53.68
		MiniCPM-o	9B	41.72
		Qwen2.5-Omni-7B	7B	39.84
		Qwen2.5-Omni-3B	3B	32.97
	Audio	Step-Audio-2-mini	8B	43.54
		Audio Flamingo 3	8B	32.27
		Qwen2-Audio	7B	26.24
		Kimi-Audio-7B-Instruct	7B	47.91
	Ours	Eureka-Audio-Instruct	1.7B	53.21
Paralinguistic MMAU | MMAR	Omni	Qwen3-Omni-Instruct	30B-A3B	74.57 | 67.10
		Ming-Lite-Omni-1.5	19B-A2.8B	63.52 | 45.40
		MiniCPM-o	9B	64.92 | 47.90
		Qwen2.5-Omni-7B	7B	66.23 | 49.60
		Qwen2.5-Omni-3B	3B	62.91 | 43.40
	Audio	Step-Audio-2-mini	8B	71.96 | 61.57
		Audio Flamingo 3	8B	74.77 | 61.00
		Qwen2-Audio	7B	59.80 | 37.90
		Kimi-Audio-7B-Instruct	7B	72.86 | 57.40
	Ours	Eureka-Audio-Base	1.7B	63.42 | 46.80
		Eureka-Audio-Instruct w/o DataFlux	1.7B	66.93 | 50.70
		Eureka-Audio-Instruct	1.7B	74.67 | 56.20

Dense Audio Captioning

Datasets	Model	Size	MMAU | MMAR ↑
Dense Captioning	Qwen3-Omni-Captioner	30B-A3B	56.68 | 46.40
	Qwen3-Omni-Instruct	30B-A3B	48.24 | 36.90
	Eureka-Audio-Instruct (Ours)	1.7B	52.96 | 41.70

Audio Input Formats

The model supports multiple audio input formats:

• Local file path: "path/to/audio.wav"
• HTTP URL: "https://example.com/audio.wav"
• Base64 data URI: "data:audio/wav;base64,<base64_string>"

Supported audio formats: WAV, MP3, FLAC, OGG, etc. (via torchaudio)

License

The model is based and modified from Qwen3. Code derived from Qwen3 is licensed under the Apache 2.0 License. Other parts of the code are licensed under the MIT License.

Acknowledgements

We would like to thank the following projects for their contributions:

• Whisper - Audio encoder
• Qwen3 - Language model backbone
• Transformers - Model framework

Thank you to all the open-source projects for their contributions!

Citation

If you find Eureka-Audio useful in your research or applications, please cite our technical report:

@misc{zhang2026eurekaaudiotriggeringaudiointelligence,
      title={Eureka-Audio: Triggering Audio Intelligence in Compact Language Models}, 
      author={Dan Zhang and Yishu Lei and Jing Hu and Shuwei He and Songhe Deng and Xianlong Luo and Danxiang Zhu and Shikun Feng and Rui Liu and Jingzhou He and Yu Sun and Hua Wu and Haifeng Wang},
      year={2026},
      eprint={2602.13954},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2602.13954}, 
}

@misc{lei2026moeadapterlargeaudio,
      title={MoE Adapter for Large Audio Language Models: Sparsity, Disentanglement, and Gradient-Conflict-Free}, 
      author={Yishu Lei and Shuwei He and Jing Hu and Dan Zhang and Xianlong Luo and Danxiang Zhu and Shikun Feng and Rui Liu and Jingzhou He and Yu Sun and Hua Wu and Haifeng Wang},
      year={2026},
      eprint={2601.02967},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2601.02967}, 
}

Contact Us

For questions, issues, or collaboration inquiries, please feel free to open an issue on GitHub.