lysanderism/TimeAudio

🚀🚀 TimeAudio: Bridging Temporal Gaps in Large Audio-Language

Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data.

To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance.

Method

TimeAudio is based on the fundamental architecture of SALMONN. Specifically, TimeAudio is consists of four components: a sliding audio encoder, a window Q-former, a segment-level token merging module, and an LLM to process raw audio.

Compare

Compared with traditional speech and audio processing tasks such as speech recognition and audio caption, Example of failed cases by Qwen2-Audio and Qwen2-Audio-R1 on fine-grained tasks that require both semantics and timestamps as output.

How to inference in CLI

You need to use the following dependencies:

1. Our environment: The python version is 3.10.16, and other required packages can be installed with the following command: pip install -r requirements.txt.
2. Download whisper large v2 to whisper_path.
3. Download Fine-tuned BEATs_iter3+ (AS2M) (cpt2) to beats_path.
4. Download vicuna 7B v1.5 to vicuna_path.
5. Download timeaudio to ckpt_path.
6. Running with python3 cli_inference.py --cfg-path "configs/infer_config.yaml" to start cli inference. Please make sure your GPU has more than 40G of memory. If your GPU does not have enough memory (e.g. only 24G), you can quantize the model using the --low_resource parameter to reduce the memory usage.

Batch inference

Same as How to inference in CLI: 1-5.
torchrun --nproc_per_node=8 inference.py --cfg-path "configs/infer_config.yaml"

Citation

If you find TimeAudio great and useful, please cite our paper:

@article{,
  title={TimeAudio: Bridging Temporal Gaps in Large Audio-Language Models},
  author={Hualei Wang, Yiming Li, Shuo Ma, Hong Liu, Xiangdong Wang },
  journal={arXiv preprint arXiv:2511.11039},
  year={2025},
  url={https://arxiv.org/abs/2511.11039}
}

🙏 Acknowledgments

We gratefully acknowledge the creators of:

• SALMONN(Tang et al.)
• Qwen2-Audio(Chu et al.)