SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection

Runbang Wang, Zining Liang, Yin Cao, and Qiuqiang Kong

Models · DCASE Results · Demos · Evaluation · Training · Documentation

Overview

SAID predicts a separate labeled acoustic map for every active sound source from four-channel spatial audio. Each 180 × 360 map represents the source region, its acoustic energy, and its sound-event class. Audio2Sph organizes audio features by direction, and Sph2Imaging decodes these panoramic features into a variable set of source maps. The release includes the complete inference, evaluation, training, and Online Scene Generation workflows described in the paper.

Details: Model Architecture.

Demos

The following 20-second examples show SAID (PaSST) and Audio2Sph + Panoramic Decoder outputs. Ground Truth is shown on the left and Prediction on the right with a shared timeline, colors, and display scale. The SAID videos also include the sound-event class legend. All four MP4 files include synchronized audio.

SAID (PaSST) — TAU scene

SAID (PaSST) — Sony scene

Audio2Sph + Panoramic Decoder — TAU scene

Audio2Sph + Panoramic Decoder — Sony scene

Details: Inference.

Quick start

SAID supports Python 3.10 through 3.12. Create an environment, enter the project directory, and install the package:

conda create -n said python=3.10 -y
conda activate said
cd /path/to/SAID
pip install -e .

FFmpeg must be available on PATH to generate demos or inference visualizations.

Run the four packaged scenes with the default complete model:

said demo --model said_passt

Run SAID on an Eigenmike recording:

said infer recording.wav --model said_passt

The CLI downloads the selected checkpoint from Hugging Face into the repository-root checkpoints/ directory on first use. recording.wav may contain the complete 32-channel Eigenmike signal or the four selected signals in the order of the 1-based Eigenmike capsule numbers [6, 10, 26, 22]. The default output is readable DCASE JSON. Select the AudioMAE variant with --model said_audiomae.

Details: Inference.

Results

DCASE2026 Task 3 Track A evaluation set

System	Rank	Macro mAP	Macro Pearson r
CUHK (SAID)	1	0.1080	0.3962

These are the official challenge results for the submitted audio-only system.

Full-recording development test set

(4.2) denotes the official evaluator dated 2026-04-02 (84b2cd1); (6.30) denotes the official evaluator dated 2026-06-30 (d4df662). The table evaluates all 78 full development-test recordings.

Paper checkpoint	Macro mAP (4.2)	Mask AP (4.2)	Macro mAP (6.30)	Class-agnostic AP (6.30)	Macro Pearson r (6.30)	Macro Class-F1
SAID (PaSST)	0.120150	0.237972	0.124514	0.197483	0.426790	0.388488
SAID (AudioMAE)	0.113441	0.228684	0.124763	0.193225	0.430736	0.395145

The (4.2) AP columns reproduce the evaluator revision used by the paper; (6.30) applies the current official evaluator to the same predictions.

Details: Evaluation.

Evaluation

Download and extract the official STAIRS26 development audio and labels as described in Evaluation, then run:

said evaluate "$HOME/datasets/dcase2026_task3" \
  --model said_passt \
  --add-previous-metrics

The default metrics follow the current (6.30) official definition; --add-previous-metrics adds the two (4.2) AP values reported by the paper. The evaluator is acquired from the official DCASE2026 Task 3 repository.

Details: Evaluation.

Models and checkpoints

Model	Role and output	Checkpoint
SAID (PaSST)	Complete class-aware SAID; default inference model	said_passt.ckpt · card
SAID (AudioMAE)	Complete class-aware SAID with AudioMAE Class Features	said_audiomae.ckpt · card
Audio2Sph + Panoramic Decoder	Class-agnostic panoramic acoustic maps	audio2sph.ckpt · card

All published checkpoints are verified by filename, size, tensor schema, dtype, and SHA256 before loading. They are released for non-commercial research use only under the SAID Model Weights Non-Commercial Research License 1.0 and all applicable upstream terms.

Details: SAID (PaSST), SAID (AudioMAE), and Audio2Sph + Panoramic Decoder.

Training and data

The paper training workflow is:

Audio2Sph pretraining
-> complete SAID training with SourceBank and Online Scene Generation
-> DCASE fine-tuning

The public recipes cover the Audio2Sph, SAID (PaSST), and SAID (AudioMAE) routes. For example, after configuring the DCASE data path, fine-tune the published PaSST model with:

said train --config configs/training/dcase_passt.yaml

Readers obtain the datasets from their rights holders and connect them through the included adapters:

• DCASE2026 Task 3 recordings and labels for fine-tuning and evaluation;
• VCTK v0.80 for Audio2Sph pretraining;
• an authorized SourceBank manifest for class-labeled Online Scene Generation.

Details: Training, Data, and Configuration.

Documentation

• Inference
• Evaluation
• Training
• Configuration
• Data
• Model Architecture
• Compression, an optional DCASE JSON post-processing utility

Command-specific options are available through said COMMAND --help.

License

Original SAID software and documentation are released under the MIT License. The three published checkpoints are distributed for non-commercial research. Third-party implementations, pretrained components, datasets, demo media, and the official evaluator retain their corresponding terms. The complete attribution and license boundaries are recorded in Third-party notices and the model and asset license summary.

Commercial applications can train new models with the MIT-licensed SAID code and independently obtained components and data whose licenses permit the intended use. Commercial use of these checkpoints, including modified or fine-tuned derivatives, is not permitted under the weights license.

Citation

If you use SAID, please cite:

@inproceedings{wang2026said,
  title     = {{SAID}: Semantic Acoustic Imaging Detector for Sound Event
               Localization and Detection},
  author    = {Wang, Runbang and Liang, Zining and Cao, Yin and Kong, Qiuqiang},
  booktitle = {Proceedings of the Detection and Classification of Acoustic
               Scenes and Events 2026 Workshop},
  year      = {2026}
}

Machine-readable citation metadata is available in CITATION.cff.

Acknowledgements

The Online Scene Generation components build on the NESD release_v1.0 renderer. The rigid-sphere implementation credits Yin Cao and Qiuqiang Kong; the renderer and shoebox image-source implementation credit Qiuqiang Kong and CUHK. Detailed source revisions and notices are provided in THIRD_PARTY_NOTICES.md.

This work was supported by the Innovation and Technology Fund (ITF), Hong Kong, under Project ITS/301/24.