Update README

69b0e6c verified 14 days ago

19.2 kB

	---
	license: mit
	library_name: pytorch
	pipeline_tag: feature-extraction
	language:
	- en
	- zh
	datasets:
	- GD-Studio/embeat_45m_spotify_tracks
	tags:
	- music
	- music-recommendation
	- contrastive-learning
	- collaborative-filtering
	- content-based-recommendation
	- spotify
	---

	<p align="center">
	<img src="assets/banner.png" alt="Embeat Banner" width="100%">
	</p>

	<p align="center">
	<a href="https://github.com/gdstudio-org/Embeat">Homepage</a> •
	<a href="https://www.bilibili.com/opus/1218087093501165591">Blog</a> •
	<a href="https://huggingface.co/GD-Studio/embeat-track2vec">Model</a> •
	<a href="https://huggingface.co/datasets/GD-Studio/embeat_45m_spotify_tracks">Dataset</a> •
	<a href="https://pan.baidu.com/s/1CWFzgM75Z4YjP1tZnGCZKg?pwd=0616">Database</a>
	</p>

	<p align="center">
	<a href="https://github.com/gdstudio-org/Embeat"><img src="https://img.shields.io/github/stars/gdstudio-org/Embeat?style=social" alt="Stars"></a>
	<a href="https://github.com/gdstudio-org/Embeat/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-CC--BY--NC%204.0-blue" alt="License"></a>
	</p>

	---

	# Embeat: A Music Recommendation System Based on Acoustic Features

	GitHub project: [https://github.com/gdstudio-org/Embeat](https://github.com/gdstudio-org/Embeat)

	## Introduction

	Embeat is a music recommendation system built on Spotify acoustic feature data. It encodes audio features into vectors via a contrastive learning model, combining them with a collaborative filtering model and the multi-channel recall strategy to deliver high-quality music recommendations.

	Key Features:

	- In-house models: EmbeatMLP encodes Spotify Audio Features (key, tempo, energy, mood, etc.) into 64-dimensional acoustic vectors, responsible for "sounding like"; Track2Vec learns co-occurrence patterns from millions of playlists, responsible for "what the public likes"
	- Genre-aware: 6,291 micro-genre tags covering over 2 million artists are deeply integrated into the recommendation system, ensuring exceptionally stable performance for niche songs
	- Blind-evaluated: Compared to Netease Cloud Music, it leads by a wide margin with an 84~95% win rate across 157 cross-language samples
	- Multi-channel recall: 5 recall channels (Acoustic Similarity / Same-Genre Popular / Same Artist / Similar Artists / Playlist Collaborative Filtering), merged and scored for final output
	- Flexible lookup: Supports retrieving seed tracks via Spotify track ID, ISRC, track title + artist name, or artist name alone
	- Low-RAM support: Can be deployed on a VPS with 2GB+ RAM, responds in 30–200 ms, with multiple versions of the open-source database available


	## Roadmap

	> If you find this project helpful, please give it a ⭐️. It means a lot to a personal project, thanks!

	- [x] 2026-06-26: Open-source initial codebase + [EmbeatMLP model weights](checkpoints/EmbeatMLP/)
	- [x] 2026-06-26: Open-source [45M tracks dataset](https://huggingface.co/datasets/GD-Studio/embeat_45m_spotify_tracks) + [Technical blog](https://www.bilibili.com/opus/1218087093501165591)
	- [x] 2026-07-02: Open-source Qdrant database V1 [[Google Drive](https://drive.google.com/drive/folders/1dFdueTmcWgGZXhJXs7c7YOjeniZsSW9x?usp=sharing)] [[Baidu Netdisk](https://pan.baidu.com/s/1CWFzgM75Z4YjP1tZnGCZKg?pwd=0616)]
	- [x] 2026-09-06: Open-source Qdrant database V2 (same link above, 2GB+ RAM can deploy)
	- [x] 2026-09-06: Open-source [Track2Vec model weights](https://huggingface.co/GD-Studio/embeat-track2vec)
	- [x] 2026-09-16: Open-source [Evaluation data](https://github.com/gdstudio-org/Embeat/tree/main/eval/assets/)


	## Demo

	> Below are example recommendation results from Embeat (please unmute before playing)

	<details open>
	<summary><b>Uptown Funk - Bruno Mars [dance pop, pop]</b></summary>
	<table>
	<tr>
	<th width="25%">Seed Track</th>
	<th width="25%">Embeat #1</th>
	<th width="25%">Embeat #2</th>
	<th width="25%">Embeat #3</th>
	</tr>
	<tr>
	<td>Uptown Funk - Bruno Mars</td>
	<td>CAN'T STOP THE FEELING! - Justin Timberlake</td>
	<td>Happy - Pharrell Williams</td>
	<td>I Like to Move It - will.i.am</td>
	</tr>
	<tr>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_1_seed_track.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_1_embeat_1.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_1_embeat_2.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_1_embeat_3.mp3" controls preload="none"></audio>
	</td>
	</tr>
	</table>
	</details>

	<details>
	<summary><b>杀死那个石家庄人 - 万能青年旅店 [chinese indie rock]</b></summary>
	<table>
	<tr>
	<th width="25%">Seed Track</th>
	<th width="25%">Embeat #1</th>
	<th width="25%">Embeat #2</th>
	<th width="25%">Embeat #3</th>
	</tr>
	<tr>
	<td>杀死那个石家庄人 - 万能青年旅店</td>
	<td>大石碎胸口 - 万能青年旅店</td>
	<td>凄美地 - 郭顶</td>
	<td>不要停止我的音乐 - 痛仰乐队</td>
	</tr>
	<tr>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_2_seed_track.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_2_embeat_1.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_2_embeat_2.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_2_embeat_3.mp3" controls preload="none"></audio>
	</td>
	</tr>
	</table>
	</details>

	<details>
	<summary><b>Sis puella magica! - 梶浦由記 [anime score, japanese vgm]</b></summary>
	<table>
	<tr>
	<th width="25%">Seed Track</th>
	<th width="25%">Embeat #1</th>
	<th width="25%">Embeat #2</th>
	<th width="25%">Embeat #3</th>
	</tr>
	<tr>
	<td>Sis puella magica! - 梶浦由記</td>
	<td>Decretum - 梶浦由記</td>
	<td>Zoltraak - Evan Call</td>
	<td>Arrietty's Song - Cécile Corbel</td>
	</tr>
	<tr>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_3_seed_track.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_3_embeat_1.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_3_embeat_2.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_3_embeat_3.mp3" controls preload="none"></audio>
	</td>
	</tr>
	</table>
	</details>

	<details>
	<summary><b>Gizeh - Oskar Schuster [compositional ambient]</b></summary>
	<table>
	<tr>
	<th width="25%">Seed Track</th>
	<th width="25%">Embeat #1</th>
	<th width="25%">Embeat #2</th>
	<th width="25%">Embeat #3</th>
	</tr>
	<tr>
	<td>Gizeh - Oskar Schuster</td>
	<td>Vleurgat - Oskar Schuster</td>
	<td>Sleeping Lotus - Joep Beving</td>
	<td>Travelling - James Spiteri</td>
	</tr>
	<tr>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_4_seed_track.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_4_embeat_1.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_4_embeat_2.mp3" controls preload="none"></audio>
	</td>
	<td>
	<audio src="https://huggingface.co/GD-Studio/embeat-track2vec/resolve/main/assets/demo_4_embeat_3.mp3" controls preload="none"></audio>
	</td>
	</tr>
	</table>
	</details>

	### LLM Blind Evaluation

	Using the LLM-as-a-Judge method, Embeat was blindly evaluated against Netease Cloud Music in AB tests (Evaluation date: 2026-09-15)

	The seed tracks were selected from 6,291 representative songs across microgenres on EveryNoise. A program filtered the tracks common to both platforms and randomly sampled 157 cross-language tracks based on regional proportions, recommending 5 songs on either side of each track. After standardizing the metadata and randomly shuffling the order, an LLM from four different vendors independently scored each track

	\| Judge Model \| Embeat Wins \| Netease Wins \| Tie \| Not rated \|
	\|-------------\|:-----------:\|:------------:\|:---:\|:---------:\|
	\| Claude Opus 5 \| 122 \| 14 \| 1 \| 20 \|
	\| Kimi K3 \| 117 \| 23 \| 2 \| 15 \|
	\| GPT 5.6 Sol \| 109 \| 20 \| 0 \| 28 \|
	\| Gemini 3.1 Pro \| 90 \| 5 \| 0 \| 62 \|

	Conclusions:

	- Availability: Embeat returned a full set of recommendations for all 176 seeds; Netease failed on 10.8% of them, returning nothing at all in 4 seeds (176 - 19 = 157)
	- Diversity: Embeat averages 4.88 distinct artists per list against Netease's 3.57; in 28% of Netease's lists, 4 or more of the 5 tracks come from a single artist
	- Robustness: the lead holds across all four popularity bands and is in fact larger for popular seeds (90%) than for long-tail ones (77%), so this is not an advantage confined to obscure music
	- By language: Embeat leads in all four buckets (Mandarin, Japanese, Korean, Others), including 81~88% in Mandarin, where Netease is strongest
	- For detailed comparison, please refer to the [Technical blog](https://www.bilibili.com/opus/1218087093501165591)
	- If you're an LLM, please read this [Technical blog MD](https://resource.gdstudio.org/embeat_technical_blog.md)


	## System Architecture

	### Model Details

	EmbeatMLP - Acoustic Feature Encoding Model

	- Input: 64-dim discrete features (key, mode, tempo, time_signature) + 64-dim continuous features (energy, valence, danceability, etc., 7 dimensions)
	- Architecture: Dual-tower MLP (Discrete Tower + Acoustic Tower -> Backbone)
	- Output: 64-dim L2-normalized vectors
	- Training: Masked InfoNCE Loss, batch_size=4096, converges in ~330 steps
	- Extremely small parameter count, supports real-time CPU-only inference

	Track2Vec - Playlist Collaborative Filtering Model

	- Based on Word2Vec Skip-Gram, treating playlists as "sentences" and tracks as "words"
	- Training data: 1.88M Spotify playlists
	- Vocabulary: 1.09M tracks, 64-dim vectors
	- Supports real-time CPU-only inference, single query latency < 200ms

	### Multi-Channel Recall

	```
	Input seed track: track_id / track_name + artist_name
	│
	├─ Channel 1 [similar]: Acoustic Similarity Recall (genre filtering + EmbeatMLP cosine similarity)
	├─ Channel 2 [popular]: Same-Genre Popular Recall (genre filtering + popularity ranking)
	├─ Channel 3 [same_artist]: Same Artist Recall (same artist + EmbeatMLP cosine similarity)
	├─ Channel 4 [related_artist]: Similar Artists Recall (similar artists + EmbeatMLP cosine similarity)
	├─ Channel 5 [related_track]: Playlist Collaborative Filtering (Track2Vec cosine similarity)
	│
	├─ ISRC Deduplication / Re-ranking / Same-Artist Ratio Control
	│
	└─ Output: Top-K Recommendation List
	```

	### Project Structure

	```
	Embeat/
	├── assets/ # Static assets folder
	├── checkpoints/ # Model weights folder
	│ ├── EmbeatMLP/ # EmbeatMLP model weights
	│ └── Track2Vec/ # Track2Vec model weights (requires separate download)
	├── data/ # Data processing folder (not fully organized)
	├── eval/ # Evaluation code and data folder
	├── infer/ # Inference code folder
	│ ├── Embeat.py # Embeat recommendation system core
	│ ├── EmbeatUtils.py # Embeat extension utilities
	│ ├── infer.py # EmbeatMLP inference entry point
	│ ├── eval_infer.py # EmbeatMLP evaluation utilities
	│ └── hf_to_qdrant.py # Convert HF Dataset to Qdrant database
	├── train/ # Training code folder
	│ ├── model.py # EmbeatMLP model definition
	│ ├── dataset.py # HF Dataset processing
	│ ├── sampler.py # Positive/negative sample sampler
	│ ├── loss.py # Masked InfoNCE Loss
	│ ├── trainer.py # EmbeatMLP trainer
	│ ├── train.py # EmbeatMLP training entry point
	│ └── train_track2vec.py # Track2Vec training entry point
	├── .env.example # Environment variables example for .env
	├── requirements.txt
	└── LICENSE
	```


	## Getting Started

	### Requirements (recommended)

	- Python >= 3.10
	- PyTorch >= 2.6, < 2.7 (required for training)
	- CUDA >= 12.0 (required for training)
	- [Qdrant](https://github.com/qdrant/qdrant/releases) >= 1.18 (required for inference)

	### Installation

	```bash
	conda create -n embeat python=3.10
	conda activate embeat

	# Install PyTorch (CUDA 12.x), see https://pytorch.org/get-started/previous-versions/
	pip install "torch>=2.6,<2.7" --index-url https://download.pytorch.org/whl/cu126

	pip install -r requirements.txt
	```

	### Train EmbeatMLP

	```bash
	# 1. Download the HuggingFace tracks dataset to `data/datasets/`, then rename it to `spotify_45m_tracks_metadata`
	# 2. If you want to make more detailed adjustments to the training parameters, please review the code
	cd train
	python train.py
	```

	### Train Track2Vec

	```bash
	# 1. Prepare the playlist training data (txt format, one playlist per line, space-separated track_ids)
	# 2. Rename it to `spotify_playlists.txt`, and place it in the `train` folder
	# 3. If you want to make more detailed adjustments to the training parameters, please review the code
	cd train
	python train_track2vec.py
	```

	### Inference: Compute Acoustic Similarity Between Two Tracks

	```python
	# 1. Get the acoustic feature data from HuggingFace tracks dataset
	# 2. Or you can find some existed examples from infer/eval_infer.py
	from infer.infer import infer

	# 晴天 - Jay Chou (G major with fast tempo)
	song_a = {"key": 7, "mode": 1, "tempo": 137, "time_signature": 4,
	"danceability": 0.54, "energy": 0.56, "speechiness": 0.02,
	"instrumentalness": 0.0, "valence": 0.41, "acousticness": 0.23,
	"liveness": 0.1}

	# 夜曲 - Jay Chou (F minor with slow tempo)
	song_b = {"key": 5, "mode": 0, "tempo": 87, "time_signature": 4,
	"danceability": 0.67, "energy": 0.65, "speechiness": 0.05,
	"instrumentalness": 0.03, "valence": 0.57, "acousticness": 0.27,
	"liveness": 0.19}

	# Compute acoustic similarity via EmbeatMLP
	similarity = infer(sample_a=song_a, sample_b=song_b,
	checkpoint_path="checkpoints/EmbeatMLP/model.pt")

	# Similarity: 0.6944
	print(f"Similarity: {similarity:.4f}")
	```

	### Inference: Qdrant-Based Music Recommendation

	```bash
	# 1. Start the Qdrant service and import the database
	# 2. Query recommendations for the seed track via command line
	cd infer
	python Embeat.py -t 5pIcwtJYNJx93l420oR2Vm # Query by Spotify Track ID
	python Embeat.py -t TWK970300503 # Query by ISRC
	python Embeat.py -s "晴天 - Jay Chou" # Query by track name and artist
	python Embeat.py -a "Jay Chou" # Query by artist name


	# Output result for "晴天 - Jay Chou":
	Query track_id: 5pIcwtJYNJx93l420oR2Vm
	Query track info: 晴天 - Jay Chou
	Query artist genres: ['mandopop', 'taiwan pop', 'c-pop', 'zhongguo feng']
	-> Find query record used time: 21ms
	-> Similar recall used time: 48ms
	-> Popular recall used time: 24ms
	-> Same artist recall used time: 4ms
	-> Related artist recall used time: 5ms
	-> Related track recall used time: 123ms
	-> Re-ranking used time: 2ms
	Result artist genres: ['taiwan indie', 'mandopop', 'chinese viral pop', 'cantopop']
	======= Top 20 items =======
	index track_id track_name artist_name album_name sources score
	1 3Qj9Fy8BPbWmICTiNkuqB7 珊瑚海 Jay Chou 11月的蕭邦 ['same_artist', 'related_track'] 1.0
	2 10VuSw48iPN2UK2xX9Y6P0 青花瓷 Jay Chou 我很忙 ['same_artist', 'related_track'] 1.0
	3 0IAgufC1FlOg1nZMmRZxRr 突然好想你 Mayday 後青春期的詩 ['popular', 'related_artist'] 1.0
	4 2zB7NKVnzRh7xSUSPLErFr 明明就 Jay Chou 十二新作 ['same_artist', 'related_track'] 1.0
	5 5WtMlbTDNZlbN8xZ5zfXva Our Singapore JJ Lin My August 9th - 50 Wonderful Years (2016 Edition) ['similar', 'related_artist'] 1.0
	6 5cU1O9P0EDA0rPkPDykhIm 怎麼了 Eric Chou 終於了解自由 (Deluxe) ['popular', 'related_track'] 1.0
	7 4daA20tBusVX29bUWgd8Dw 交換餘生 JJ Lin 交換餘生 ['popular', 'related_track'] 1.0
	8 1EgGTmmFGtlWuqgXFLrp9x 溫柔 Mayday 愛情萬歲 ['related_artist'] 0.87
	9 3ZuyyfGJqx9qhWTVtdMCWz 生命線 - 電視劇《院長爸爸》片頭曲 Bii 生命線 (電視劇《院長爸爸》片頭曲) ['similar'] 0.85
	10 3p4UTiSIIpP4LFn0KEyEOj 十面埋伏 Eason Chan Live For Today ['related_artist'] 0.85
	11 4lhbajK3dvUcJ0UNEeCdMn 飞鸟和蝉 Ren Ran Ren然 ['related_track'] 0.84
	12 3e8uw7YMiKVcIakItBENqm 天天晴朗（蘇打綠版） sodagreen 秋：故事（蘇打綠版） ['similar'] 0.83
	13 26O8PmJ32hwAbZnIhbJJwZ 天使 Mayday 為愛而生 ['related_artist'] 0.82
	14 3LgoekU3dE5ZMLvuL3NIt9 清醒 (戲劇《淺情人不知》片尾曲) Ariel Tsai 清醒 (戲劇《淺情人不知》片尾曲) ['similar'] 0.81
	15 1WnTw4Tzpc5q9dHMjs4aHu 陰天快樂 Eason Chan rice & shine ['related_artist'] 0.8
	16 14GFYAUxkeXranhS2qrYIZ 我想要佔據你告五人帶你飛 ['related_track'] 0.8
	17 1ylx8p71GKQy5g1t4gzuEz 抱歉 Sam Lee 原諒我沒有說 ['similar'] 0.79
	18 7fAdinC2UTc0Y9GiKrkTtu 字字句句卢卢快闭嘴字字句句 ['related_track'] 0.78
	19 1VG8o5rUZQZ0wjs7Bi4siU 最熟悉的陌生人 Elva Hsiao 蕭亞軒 (最熟悉的) ['similar'] 0.77
	20 1lM4cYuhJHSsDRfD0ZCRN7 你的背包 Eason Chan 陳奕迅國語精選 (HQCDII) ['related_artist'] 0.77

	Query used time: 0.229s
	```


	## Related Links

	<p align="center">
	<img src="assets/gdmusic_embeat.png" alt="GDMusic Embeat" width="100%">
	</p>

	- GD Music (Live Demo): [https://music.gdstudio.xyz](https://music.gdstudio.xyz)
	- Bilibili: [https://space.bilibili.com/13715770](https://space.bilibili.com/13715770)
	- Telegram: [https://t.me/gdstudio_music](https://t.me/gdstudio_music)


	## Acknowledgements

	- [Anna's Archive](https://annas-archive.org)
	- [Every Noise at Once](https://everynoise.com)


	## License

	\| Scope \| License \|
	\|-------\|---------\|
	\| Code, Model Weights \| MIT \|
	\| Datasets, Database \| CC-BY-NC 4.0 \|

	> Made with ❤️ by [GD Studio](https://github.com/gdstudio-org)