wav2taste — trained taste-prediction heads

Predict the five basic tastes — sweet, bitter, salty, sour, spicy, each in [0, 1] — that a piece of audio evokes. These are the trained heads behind the paper Taste-aware music retrieval from audio embeddings (Matteo Spanio, Antonio Rodà — University of Padua, CBMI 2026).

Each head is a small MLP (single encoder) or a gated late-fusion module (multiple encoders) sitting on top of frozen audio encoders. The heads are tiny and downloaded from this repo; the frozen encoders they sit on are pulled from their own upstream repos at load time. Inference needs no access to the training dataset.

Usage

pip install git+https://github.com/CSCPadova/wav2taste

from wav2taste import load_taste_model

model = load_taste_model("vggish+mule")     # strongest configuration
model.predict("song.wav")
# {'sweet': 0.34, 'bitter': 0.71, 'salty': 0.12, 'sour': 0.58, 'spicy': 0.09}

load_taste_model(name) downloads heads/<name>.pt from this repo and rebuilds the encoder(s) + head. Audio is decoded to mono, resampled to each encoder's rate, split into 15-second windows that the encoder pools, then averaged across windows — exactly as at training time. Outputs are a sigmoid, so every value is in [0, 1].

Available heads

Single encoders (10): mfcc, clap, mert, ast, encodec, vggish, hubert, panns, omar-rq, mule.

Gated fusions (7): ast+vggish, ast+mule, vggish+mule, clap+mule, clap+vggish, ast+vggish+mule, clap+ast+vggish+mule.

mfcc is the untrained-DSP floor; mule uses the matteospanio/mule weights (CC-BY-NC). The full list is also exposed as wav2taste.PRETRAINED.

Performance

On the 40-item held-out test set (five-seed means):

model	macro Pearson r	macro RMSE
vggish+mule (best fusion)	0.724 ± 0.020	0.134
best single encoder (~`vggish/ast`)	≈ 0.666	≈ 0.134
prior per-taste-AST baseline	0.556	0.219

• On absolute error the encoders are statistically flat — a single vggish matches the best fusion (macro RMSE ≈ 0.134); the fusion advantage is confined to rank correlation (0.724 vs 0.666), and the lift over strong single encoders is within bootstrap confidence intervals.
• On held-out real music, model error is less than half an average human rater's deviation from the group consensus (RMSE 0.13 vs 0.28) — the model tracks the consensus more closely than a typical individual rater.
• Each head was selected on validation loss from a single seed; the headline numbers above are five-seed means. See the paper for per-target, per-source, and CI tables.

Training data

Trained on csc-unipd/sonic-seasoning (269 train / 68 val / 40 test clips) with masked multi-task MSE over the five tastes — one model, five outputs, and each row contributes loss only on the tastes it was actually rated for (no imputation). The 40-item test split is a real-music + generated-music generalization probe held out entirely from training.

Intended use & limitations

• Intended use: research on crossmodal sound↔taste correspondences, content-based music retrieval by taste profile, and as baselines/reference heads for the wav2taste benchmark. Non-commercial only.
• Taste ratings are perceptual consensus values, not ground-truth chemistry — they encode how listeners associate a sound with a taste. Cross-source comparability is approximate (different elicitation procedures; see the dataset card).
• spicy is the sparsest target (fewest training rows); its predictions are the least reliable. Report confidence intervals when comparing models on it.
• Trained on short music/sound clips at mono/44.1 kHz; behavior on speech, very long recordings, or out-of-distribution audio is unvalidated.

License

CC-BY-NC-4.0 (non-commercial): the heads derive from a non-commercial training corpus and from MULE's CC-BY-NC weights. The training/inference code is Apache-2.0 at github.com/CSCPadova/wav2taste. The frozen upstream encoders keep their own licenses.

Citation

@inproceedings{spanio2026taste,
  title     = {Taste-aware music retrieval from audio embeddings},
  author    = {Spanio, Matteo and Rod{\`a}, Antonio},
  booktitle = {Proceedings of the International Conference on Content-Based Multimedia Indexing (CBMI)},
  year      = {2026},
}