README.md

metadata

title: OpenMusic Analysis API
emoji: 🎵
colorFrom: blue
colorTo: purple
sdk: docker
pinned: false

OpenMusic Music Analysis API

Versioned server-side analysis for separate global audio, temporal audio and lyrics representations. The service deliberately does not fuse these embedding spaces.

Architecture

multipart HTTP request
        ↓
validation + temporary-file lifetime
        ↓
MusicAnalysisService
├── GlobalAudioAnalyzer ───┐
├── TemporalAudioAnalyzer ─┼── shared CLAP audio encoder
├── LyricsAnalyzer ─────────── BGE-M3 text encoder
└── ModelRegistry
        ↓
versioned typed response

The HTTP layer validates, calls the application service and serializes. Analyzer implementations own preprocessing, inference, aggregation and normalization. Model objects are lazy singletons behind an inference semaphore; they are never created per request or exposed to the HTTP layer.

AnalysisContext decodes an upload once to mono float32 PCM at 48 kHz and caches additional deterministic in-memory resamples by sample rate. Source-file decoding is centralized in ffmpeg and supports mp3, m4a, flac, wav, aac, ogg and opus.

Models

Representation	Model	Exact revision	Dimension	Normalization
audio.global	laion/clap-htsat-unfused	8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a	512	L2 per window, mean, L2 final
audio.temporal	laion/clap-htsat-unfused	8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a	512 per segment	L2 per segment
lyrics.global	BAAI/bge-m3	5617a9f61b028005a4858fdac845db406aefb181	1024	L2 per chunk, token-weighted mean, L2 final

CLAP is the existing backend's baseline, now pinned instead of following Hugging Face main. BGE-M3 was selected for its MIT license, English/Russian and broader multilingual support, semantic retrieval training, 1024-dimensional dense space and long-document support up to 8192 tokens. Lyrics still use structural chunks so section information is not discarded and section embeddings can be exposed later. Compared alternatives were multilingual E5 large (good multilingual quality but a 512-token limit), multilingual MPNet (older, 128-token sentence/paragraph setup), and GTE multilingual base (smaller, long-context, but requires repository remote code).

Every registry item declares model_id, exact model_version, modality, dimension, dtype, normalization, license, preprocessing version and full representation config. The preprocessing version contains a deterministic hash of all result-affecting config, so changing window/hop/chunk settings changes cache identity.

Audio preprocessing

The decoder runs ffprobe validation, then one deterministic ffmpeg decode (-threads 1) to mono little-endian float32 PCM at 48,000 Hz. Future model sample rates are produced from that request-scoped PCM with deterministic polyphase resampling.

audio.global defaults:

• 10-second analysis windows;
• adaptive count: one for short tracks, up to four as duration grows;
• window starts are uniformly distributed over 10%–90% of the valid start range;
• short tracks are repeated deterministically to a full model window;
• every CLAP vector is L2-normalized, vectors are averaged, then normalized again.

This prevents a global track vector from effectively describing only the intro. Windows longer than CLAP's native 10 seconds are deterministically divided into evenly placed 10-second encoder subwindows and aggregated; CLAP never receives a long input from which its processor could choose a random crop.

audio.temporal defaults:

• 10-second window and 10-second hop;
• explicit tail window so the ending is represented;
• at most 24 segments, with deterministic uniform selection if the candidate count is larger;
• chronological millisecond timestamps and one normalized CLAP vector per segment.

The temporal summary reports only representation geometry: number_of_segments, cosine mean_adjacent_distance, max_adjacent_distance, trajectory_variance, and largest_transition_index. It does not claim to measure emotion, climax or tension.

Lyrics preprocessing

LyricsPreprocessor applies Unicode NFKC, normalizes line endings, removes control characters, LRC metadata/timestamps and pure technical URL lines, while preserving punctuation, paragraphs and repeated choruses. It recognizes English and Russian section markers such as Verse/Chorus/Bridge and Куплет/Припев/Бридж.

Sections are the first chunk boundary. Oversized sections are split by lines and then by token IDs, never by arbitrary character count. The default chunk limit is 512 tokens. Repeated sections remain repeated and therefore retain their weight. BGE-M3 uses its document-side CLS dense representation; chunk vectors are normalized and aggregated by token count. The internal chunk/section pipeline is ready for a later lyrics.sections or lyrics.temporal response without changing lyrics.global.

Installation and deployment

The production image uses Python 3.11. A system ffmpeg/ffprobe is required.

python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
uvicorn main:app --host 0.0.0.0 --port 7860

Or:

docker compose up --build

Docker Compose exposes http://localhost:8000; Hugging Face Spaces uses container port 7860. Model files are cached in the hf_cache volume.

Device selection is centralized. OPENMUSIC_DEVICE=auto chooses CUDA, then Apple MPS, then CPU. Set OPENMUSIC_DEVICE=cpu for a forced CPU fallback. Output embeddings remain float32 regardless of device. OPENMUSIC_INFERENCE_CONCURRENCY=1 is the safe default that avoids duplicate loading and unbounded concurrent GPU work.

API

Models

curl http://localhost:7860/v1/models

GET /v1/models is the client source of truth. loaded distinguishes an available lazy analyzer from one whose model is currently resident.

Analyze a track

curl -X POST http://localhost:7860/v1/tracks/analyze \
  -F 'audio=@song.flac' \
  -F 'lyrics=[Verse]
Hello world' \
  -F 'track_id=local:42' \
  -F 'content_identity=sha256:...' \
  -F 'requested_representations=audio.global' \
  -F 'requested_representations=lyrics.global'

Repeated fields, a comma-separated value, or a JSON array are accepted for requested_representations. With no field, both audio representations are calculated, plus lyrics.global only when non-empty lyrics are supplied. Explicitly requesting lyrics.global without lyrics is an error.

Abbreviated response:

{
  "schema_version": "1",
  "track": {"track_id": "local:42", "content_identity": "sha256:..."},
  "representations": {
    "audio.global": {
      "representation": "audio.global",
      "model_id": "laion/clap-htsat-unfused",
      "model_version": "8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a",
      "preprocessing_version": "audio-clap-global-v1.<config-hash>",
      "modality": "audio.global",
      "dimension": 512,
      "dtype": "float32",
      "normalized": true,
      "configuration": {},
      "embedding": [0.01, -0.02],
      "analysis": {"duration_ms": 213000, "windows_used": 4}
    },
    "audio.temporal": {
      "representation": "audio.temporal",
      "model_id": "laion/clap-htsat-unfused",
      "model_version": "8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a",
      "preprocessing_version": "audio-clap-temporal-v1.<config-hash>",
      "modality": "audio.temporal",
      "dimension": 512,
      "dtype": "float32",
      "normalized": true,
      "configuration": {},
      "segments": [{"start_ms": 0, "end_ms": 10000, "embedding": [0.01]}],
      "summary": {"number_of_segments": 1, "mean_adjacent_distance": 0.0,
        "max_adjacent_distance": 0.0, "trajectory_variance": 0.0,
        "largest_transition_index": null}
    }
  }
}

Errors never contain stack traces:

{"error": {"code": "AUDIO_DECODE_FAILED", "message": "The uploaded audio could not be decoded."}}

GET /v1/status reports the selected device and loaded/available representations without environment details. GET /health is a lightweight container probe.

Configuration

Variable	Default
OPENMUSIC_GLOBAL_WINDOW_SECONDS	10
OPENMUSIC_GLOBAL_WINDOWS	4
OPENMUSIC_TEMPORAL_WINDOW_SECONDS	10
OPENMUSIC_TEMPORAL_HOP_SECONDS	10
OPENMUSIC_TEMPORAL_MAX_SEGMENTS	24
OPENMUSIC_CLAP_BATCH_SIZE	4
OPENMUSIC_LYRICS_CHUNK_TOKENS	512
OPENMUSIC_MAX_UPLOAD_BYTES	104857600
OPENMUSIC_MAX_LYRICS_CHARACTERS	100000
OPENMUSIC_MAX_AUDIO_SECONDS	1800
OPENMUSIC_REQUEST_TIMEOUT_SECONDS	300

Changing result-affecting representation configuration changes its generated preprocessing_version and benchmark cache key.

Benchmark tools

Place optional UTF-8 lyrics beside audio using the same stem (song.flac + song.txt).

python tools/nearest_neighbors.py ./tracks \
  --representation audio.global --top-k 10 --output reports/audio-global.json

python tools/nearest_neighbors.py ./tracks \
  --representation lyrics.global --output reports/lyrics-global.json

python tools/nearest_neighbors.py ./tracks \
  --representation audio.temporal --output reports/audio-temporal.json

The tool writes human-readable JSON and CSV. Its cache identity includes audio SHA-256, lyrics SHA-256, model ID, exact revision, preprocessing version, representation and full configuration. Unchanged embeddings are reused. Temporal reports contain two explicitly experimental metrics: normalized-time interpolation with mean cosine, and classic DTW over cosine distance. Neither is treated as the final similarity definition.

Measure the current machine with a representative track:

python tools/profile_analysis.py song.flac --lyrics song.txt

It reports total model startup, each representation latency, peak process RSS and peak CUDA allocation when CUDA is available.

Tests

pip install -r requirements-dev.txt
pytest

Tests use deterministic fake encoders and do not download model weights. They cover API selection/errors, global aggregation, temporal segmentation/limits, request-scoped decode reuse, English/Russian structured lyrics and long chunking, model registry consistency, inference failure sanitization, and both temporal similarity baselines.

Extension points and limitations

A future MERT analyzer should implement the audio analyzer protocol, request its sample rate from AnalysisContext, declare a separate registry item and use explicit names such as audio.mert.global and audio.mert.temporal. It must not be averaged with CLAP. It is not enabled in V1 to avoid adding a second large model/dependency path before comparative benchmarks exist.

• CLAP temporal vectors are a trajectory through CLAP's audio space, not a trained music emotion representation.
• BGE-M3 semantic lyrics similarity is not lyrics emotional similarity.
• Embeddings from CLAP and BGE-M3 (and future MERT models) are different spaces and must never be compared directly with cosine or added together.
• Temporal interpolation and DTW are baselines for experiments, not validated music structure metrics.
• Exact revisions and preprocessing config make inference reproducible, but very small floating-point differences can still occur across PyTorch/device/hardware versions.