Melati-AudioSet-EAT-Large

An AudioSet tagger: one model that names which of AudioSet's 527 sounds are in a clip, scoring 0.5105 mAP on AudioSet's evaluation set, ahead of every released single model we scored on the same clips (the best, CED-base, 0.4997). It is EAT-large (309M parameters) distilled from the average of three released taggers (CED-base + EAT-large + SSLAM; 0.5166 averaged) on 730,841 AudioSet clips (2,011 hours), none of them from its evaluation set; its last stage also trained a frame-level head beside it on the hand-timed events of AudioSet's strongly labelled clips. Part of Melati, Scicom's speech-data pipeline.

model	mAP
this model	0.5105
this model after stage 2	0.5059
this model after stage 1	0.5033
CED-base	0.4997	Dinkel et al., ICASSP 2024
EAT-large	0.4947	Chen et al., IJCAI 2024
SSLAM	0.4944	Alex et al., ICLR 2025
EAT-base	0.4901	Chen et al., IJCAI 2024
AST	0.4572	Gong et al., Interspeech 2021

All on the same 17,141 clips: AudioSet's evaluation set as agkphysics/AudioSet holds it today (YouTube removes clips over time, so every paper's copy differs and published numbers are not comparable across papers), each model in fp32 with its own preprocessing, mAP the mean over the 527 classes of their average precision.

Use

import torch, torchaudio, soundfile as sf
from transformers import AutoModel

model = AutoModel.from_pretrained("Scicom-intl/Melati-AudioSet-EAT-Large", trust_remote_code=True).eval()

wav, sr = sf.read("clip.wav", dtype="float32", always_2d=True)
wav = torch.from_numpy(wav.mean(1))
if sr != 16000:
    wav = torchaudio.functional.resample(wav, sr, 16000)
wav = wav - wav.mean()
fbank = torchaudio.compliance.kaldi.fbank(wav[None], htk_compat=True, sample_frequency=16000, use_energy=False,
                                          window_type="hanning", num_mel_bins=128, dither=0.0, frame_shift=10)
fbank = torch.nn.functional.pad(fbank, (0, 0, 0, max(0, 1024 - len(fbank))))[:1024]     # 10.24 s
x = ((fbank + 4.268) / (4.569 * 2))[None, None]                                          # [1, 1, 1024, 128]

with torch.no_grad():
    probs = torch.sigmoid(model(x))[0]
for i in probs.topk(5).indices.tolist():
    print(model.config.id2label[i], round(probs[i].item(), 3))

The input is EAT's: 16 kHz mono, 128-bin Kaldi filterbanks every 10 ms, 1024 frames (a longer clip in 10 s windows).

How it was trained

The clip head is trained to the average of the teachers' probabilities (binary cross-entropy against soft targets; AudioSet's own clip labels are not used) on AudioSet's strongly labelled training clips (102,123, every event timed by hand, from CLAPv2/audioset_strong), its balanced set (11,397) and 617,321 clips of its unbalanced set, any clip of the evaluation set left out, in 3 stages:

1. From EAT-large fine-tuned on AudioSet-2M (0.4942 on these clips in bf16), 10 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5027 at its best (bf16).
2. From stage 1, 3 epochs on the strongly labelled set, the balanced set and 617,321 clips of the unbalanced set (730,841 clips, 2,011 hours), learning rate 1e-5: 0.5054 at its best (bf16).
3. From stage 2, with a second head beside the first that places each sound in time, every 160 ms, 5 epochs on the strongly labelled set and the balanced set (113,520 clips, 311 hours), learning rate 1e-5: 0.5098 where the frame head was best (bf16).

Stage 3's frame head learns the hand-timed events of the 102,123 strongly labelled clips (binary cross-entropy every 160 ms). The stage raised the clip head as well, 0.5059 after stage 2 and 0.5105 after it on these clips in fp32; which part of it did so (the timed events, a second learning-rate cycle, no mixup) is not yet separated. Every stage: SpecAugment, warm-up and cosine decay, bf16 with gradient checkpointing, and an exponential moving average of the weights (0.9995) whose best evaluation is kept; stages 1 and 2 also mixed half their batches (mixup). The code is audioset/ in Melati.

Limitations

• One evaluation set: AudioSet's own. On speech-heavy material (podcasts) it tags what AudioSet tags, which is not a measure of how well it times or places an event inside a long recording.
• Clip-level: a probability for each class over the whole input. The frame head trained beside it in stage 3 is not part of this release: on AudioSet's 16,896 strongly labelled test clips it places sounds in time less well than PretrainedSED (frame mAP 0.4102 against 0.4261 on the 364 classes both have), which is what Melati's own pipeline uses for that.
• The training clips are 730,841 of AudioSet's 2 million; more of its unbalanced set is the obvious lever.

Licence

MIT, as EAT's code and weights it starts from. The teachers it learned from are released under Apache-2.0 (CED-base) and MIT (EAT-large, SSLAM). AudioSet's labels are CC BY 4.0 (Google); its audio is YouTube's and is not redistributed here.

Citation

@misc{zolkepli2026melati,
  title  = {Melati: Multilingual Expressive speech, Labelled for Affect, Tags and Interjections},
  author = {Husein Zolkepli},
  year   = {2026},
  note   = {Scicom (MSC) Berhad}
}

We thank Scicom (MSC) Berhad for the compute.