--- title: OpenMusic Analysis API emoji: 🎡 colorFrom: blue colorTo: purple sdk: docker pinned: false --- # OpenMusic Music Analysis API Versioned server-side analysis for separate global audio, temporal audio and lyrics representations. The service deliberately does not fuse these embedding spaces. ## Architecture ```text multipart HTTP request ↓ validation + temporary-file lifetime ↓ MusicAnalysisService β”œβ”€β”€ GlobalAudioAnalyzer ───┐ β”œβ”€β”€ TemporalAudioAnalyzer ─┼── shared CLAP audio encoder β”œβ”€β”€ LyricsAnalyzer ─────────── BGE-M3 text encoder └── ModelRegistry ↓ versioned typed response ``` The HTTP layer validates, calls the application service and serializes. Analyzer implementations own preprocessing, inference, aggregation and normalization. Model objects are lazy singletons behind an inference semaphore; they are never created per request or exposed to the HTTP layer. `AnalysisContext` decodes an upload once to mono float32 PCM at 48 kHz and caches additional deterministic in-memory resamples by sample rate. Source-file decoding is centralized in ffmpeg and supports mp3, m4a, flac, wav, aac, ogg and opus. ## Models | Representation | Model | Exact revision | Dimension | Normalization | |---|---|---|---:|---| | `audio.global` | `laion/clap-htsat-unfused` | `8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a` | 512 | L2 per window, mean, L2 final | | `audio.temporal` | `laion/clap-htsat-unfused` | `8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a` | 512 per segment | L2 per segment | | `lyrics.global` | `BAAI/bge-m3` | `5617a9f61b028005a4858fdac845db406aefb181` | 1024 | L2 per chunk, token-weighted mean, L2 final | CLAP is the existing backend's baseline, now pinned instead of following Hugging Face `main`. BGE-M3 was selected for its MIT license, English/Russian and broader multilingual support, semantic retrieval training, 1024-dimensional dense space and long-document support up to 8192 tokens. Lyrics still use structural chunks so section information is not discarded and section embeddings can be exposed later. Compared alternatives were multilingual E5 large (good multilingual quality but a 512-token limit), multilingual MPNet (older, 128-token sentence/paragraph setup), and GTE multilingual base (smaller, long-context, but requires repository remote code). Every registry item declares `model_id`, exact `model_version`, modality, dimension, dtype, normalization, license, preprocessing version and full representation config. The preprocessing version contains a deterministic hash of all result-affecting config, so changing window/hop/chunk settings changes cache identity. ## Audio preprocessing The decoder runs ffprobe validation, then one deterministic ffmpeg decode (`-threads 1`) to mono little-endian float32 PCM at 48,000 Hz. Future model sample rates are produced from that request-scoped PCM with deterministic polyphase resampling. `audio.global` defaults: - 10-second analysis windows; - adaptive count: one for short tracks, up to four as duration grows; - window starts are uniformly distributed over 10%–90% of the valid start range; - short tracks are repeated deterministically to a full model window; - every CLAP vector is L2-normalized, vectors are averaged, then normalized again. This prevents a global track vector from effectively describing only the intro. Windows longer than CLAP's native 10 seconds are deterministically divided into evenly placed 10-second encoder subwindows and aggregated; CLAP never receives a long input from which its processor could choose a random crop. `audio.temporal` defaults: - 10-second window and 10-second hop; - explicit tail window so the ending is represented; - at most 24 segments, with deterministic uniform selection if the candidate count is larger; - chronological millisecond timestamps and one normalized CLAP vector per segment. The temporal summary reports only representation geometry: `number_of_segments`, cosine `mean_adjacent_distance`, `max_adjacent_distance`, `trajectory_variance`, and `largest_transition_index`. It does not claim to measure emotion, climax or tension. ## Lyrics preprocessing `LyricsPreprocessor` applies Unicode NFKC, normalizes line endings, removes control characters, LRC metadata/timestamps and pure technical URL lines, while preserving punctuation, paragraphs and repeated choruses. It recognizes English and Russian section markers such as Verse/Chorus/Bridge and ΠšΡƒΠΏΠ»Π΅Ρ‚/ΠŸΡ€ΠΈΠΏΠ΅Π²/Π‘Ρ€ΠΈΠ΄ΠΆ. Sections are the first chunk boundary. Oversized sections are split by lines and then by token IDs, never by arbitrary character count. The default chunk limit is 512 tokens. Repeated sections remain repeated and therefore retain their weight. BGE-M3 uses its document-side CLS dense representation; chunk vectors are normalized and aggregated by token count. The internal chunk/section pipeline is ready for a later `lyrics.sections` or `lyrics.temporal` response without changing `lyrics.global`. ## Installation and deployment The production image uses Python 3.11. A system ffmpeg/ffprobe is required. ```bash python3.11 -m venv .venv source .venv/bin/activate pip install -r requirements.txt uvicorn main:app --host 0.0.0.0 --port 7860 ``` Or: ```bash docker compose up --build ``` Docker Compose exposes `http://localhost:8000`; Hugging Face Spaces uses container port 7860. Model files are cached in the `hf_cache` volume. Device selection is centralized. `OPENMUSIC_DEVICE=auto` chooses CUDA, then Apple MPS, then CPU. Set `OPENMUSIC_DEVICE=cpu` for a forced CPU fallback. Output embeddings remain float32 regardless of device. `OPENMUSIC_INFERENCE_CONCURRENCY=1` is the safe default that avoids duplicate loading and unbounded concurrent GPU work. ## API ### Models ```bash curl http://localhost:7860/v1/models ``` `GET /v1/models` is the client source of truth. `loaded` distinguishes an available lazy analyzer from one whose model is currently resident. ### Analyze a track ```bash curl -X POST http://localhost:7860/v1/tracks/analyze \ -F 'audio=@song.flac' \ -F 'lyrics=[Verse] Hello world' \ -F 'track_id=local:42' \ -F 'content_identity=sha256:...' \ -F 'requested_representations=audio.global' \ -F 'requested_representations=lyrics.global' ``` Repeated fields, a comma-separated value, or a JSON array are accepted for `requested_representations`. With no field, both audio representations are calculated, plus `lyrics.global` only when non-empty lyrics are supplied. Explicitly requesting `lyrics.global` without lyrics is an error. Abbreviated response: ```json { "schema_version": "1", "track": {"track_id": "local:42", "content_identity": "sha256:..."}, "representations": { "audio.global": { "representation": "audio.global", "model_id": "laion/clap-htsat-unfused", "model_version": "8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a", "preprocessing_version": "audio-clap-global-v1.", "modality": "audio.global", "dimension": 512, "dtype": "float32", "normalized": true, "configuration": {}, "embedding": [0.01, -0.02], "analysis": {"duration_ms": 213000, "windows_used": 4} }, "audio.temporal": { "representation": "audio.temporal", "model_id": "laion/clap-htsat-unfused", "model_version": "8fa0f1c6d0433df6e97c127f64b2a1d6c0dcda8a", "preprocessing_version": "audio-clap-temporal-v1.", "modality": "audio.temporal", "dimension": 512, "dtype": "float32", "normalized": true, "configuration": {}, "segments": [{"start_ms": 0, "end_ms": 10000, "embedding": [0.01]}], "summary": {"number_of_segments": 1, "mean_adjacent_distance": 0.0, "max_adjacent_distance": 0.0, "trajectory_variance": 0.0, "largest_transition_index": null} } } } ``` Errors never contain stack traces: ```json {"error": {"code": "AUDIO_DECODE_FAILED", "message": "The uploaded audio could not be decoded."}} ``` `GET /v1/status` reports the selected device and loaded/available representations without environment details. `GET /health` is a lightweight container probe. ## Configuration | Variable | Default | |---|---:| | `OPENMUSIC_GLOBAL_WINDOW_SECONDS` | `10` | | `OPENMUSIC_GLOBAL_WINDOWS` | `4` | | `OPENMUSIC_TEMPORAL_WINDOW_SECONDS` | `10` | | `OPENMUSIC_TEMPORAL_HOP_SECONDS` | `10` | | `OPENMUSIC_TEMPORAL_MAX_SEGMENTS` | `24` | | `OPENMUSIC_CLAP_BATCH_SIZE` | `4` | | `OPENMUSIC_LYRICS_CHUNK_TOKENS` | `512` | | `OPENMUSIC_MAX_UPLOAD_BYTES` | `104857600` | | `OPENMUSIC_MAX_LYRICS_CHARACTERS` | `100000` | | `OPENMUSIC_MAX_AUDIO_SECONDS` | `1800` | | `OPENMUSIC_REQUEST_TIMEOUT_SECONDS` | `300` | Changing result-affecting representation configuration changes its generated `preprocessing_version` and benchmark cache key. ## Benchmark tools Place optional UTF-8 lyrics beside audio using the same stem (`song.flac` + `song.txt`). ```bash python tools/nearest_neighbors.py ./tracks \ --representation audio.global --top-k 10 --output reports/audio-global.json python tools/nearest_neighbors.py ./tracks \ --representation lyrics.global --output reports/lyrics-global.json python tools/nearest_neighbors.py ./tracks \ --representation audio.temporal --output reports/audio-temporal.json ``` The tool writes human-readable JSON and CSV. Its cache identity includes audio SHA-256, lyrics SHA-256, model ID, exact revision, preprocessing version, representation and full configuration. Unchanged embeddings are reused. Temporal reports contain two explicitly experimental metrics: normalized-time interpolation with mean cosine, and classic DTW over cosine distance. Neither is treated as the final similarity definition. Measure the current machine with a representative track: ```bash python tools/profile_analysis.py song.flac --lyrics song.txt ``` It reports total model startup, each representation latency, peak process RSS and peak CUDA allocation when CUDA is available. ## Tests ```bash pip install -r requirements-dev.txt pytest ``` Tests use deterministic fake encoders and do not download model weights. They cover API selection/errors, global aggregation, temporal segmentation/limits, request-scoped decode reuse, English/Russian structured lyrics and long chunking, model registry consistency, inference failure sanitization, and both temporal similarity baselines. ## Extension points and limitations A future MERT analyzer should implement the audio analyzer protocol, request its sample rate from `AnalysisContext`, declare a separate registry item and use explicit names such as `audio.mert.global` and `audio.mert.temporal`. It must not be averaged with CLAP. It is not enabled in V1 to avoid adding a second large model/dependency path before comparative benchmarks exist. - CLAP temporal vectors are a trajectory through CLAP's audio space, not a trained music emotion representation. - BGE-M3 semantic lyrics similarity is not lyrics emotional similarity. - Embeddings from CLAP and BGE-M3 (and future MERT models) are different spaces and must never be compared directly with cosine or added together. - Temporal interpolation and DTW are baselines for experiments, not validated music structure metrics. - Exact revisions and preprocessing config make inference reproducible, but very small floating-point differences can still occur across PyTorch/device/hardware versions.