| | ---
|
| | license: mit
|
| | library_name: transformers
|
| | tags:
|
| | - music
|
| | - audio
|
| | ---
|
| |
|
| | <a href="https://arxiv.org/abs/2602.00744">Tech Report</a>
|
| |
|
| | # ACE-Step Captioner
|
| |
|
| | ## Description
|
| |
|
| | ACE-Step Captioner is the annotation model used by **ACE-Step v1.5** for training data labeling. It is a professional-grade music captioning model that generates detailed, structured descriptions of audio content.
|
| |
|
| | ### Performance
|
| |
|
| | 🏆 **Accuracy surpasses Gemini Pro 2.5** in music description tasks
|
| |
|
| | ### Key Features
|
| |
|
| | - 🎼 **Musical Style Analysis** - Identifies genres, sub-genres, and stylistic influences
|
| | - 🎸 **Instrument Recognition** - Detects and describes 1000+ instrument types and combinations
|
| | - 🎭 **Structure & Progression** - Analyzes musical arrangement including intro, verse, chorus, bridge, climax, and outro
|
| | - 🔊 **Timbre Description** - Captures tonal qualities, textures, and sonic characteristics
|
| | - 📝 **Rich Vocabulary** - Supports 1000+ descriptive terms for comprehensive music annotation
|
| |
|
| | ## Usage
|
| |
|
| | The usage is the same as [Qwen2.5 Omni-7B](https://huggingface.co/Qwen/Qwen2.5-Omni-7B).
|
| |
|
| | ### Prompt Format
|
| |
|
| | Use the following prompt to caption audio:
|
| |
|
| | ```
|
| | *Task* Describe this audio in detail
|
| | <audio>
|
| | ```
|
| |
|
| | ### Output Format
|
| |
|
| | The model generates natural language descriptions covering multiple aspects of the music.
|
| |
|
| | ### Example Output
|
| |
|
| | ```
|
| | A melancholic indie folk track featuring fingerpicked acoustic guitar
|
| | as the primary instrument. The song opens with a sparse, contemplative
|
| | intro before the vocals enter with a breathy, intimate delivery.
|
| | The arrangement gradually builds through the verse, adding subtle
|
| | string pads and a gentle kick drum. The chorus lifts with layered
|
| | harmonies and a warmer, fuller texture. The bridge introduces a
|
| | key change and emotional climax before returning to the stripped-down
|
| | acoustic arrangement for the outro.
|
| | ```
|
| |
|
| | ## Descriptive Capabilities
|
| |
|
| | ### Musical Styles (Examples)
|
| |
|
| | | Category | Styles |
|
| | |----------|--------|
|
| | | **Electronic** | Ambient, Techno, House, Drum & Bass, Synthwave, IDM, Downtempo |
|
| | | **Rock** | Alternative, Indie, Post-Rock, Progressive, Psychedelic, Grunge |
|
| | | **Pop** | Synth-pop, Electropop, Dream Pop, Art Pop, Indie Pop |
|
| | | **Classical** | Orchestral, Chamber, Minimalist, Neo-Classical, Cinematic |
|
| | | **World** | Latin, African, Middle Eastern, Asian Traditional, Celtic |
|
| | | **Jazz** | Fusion, Smooth, Bebop, Modal, Free Jazz |
|
| | | **Hip-Hop** | Trap, Boom Bap, Lo-fi, Instrumental, Cloud Rap |
|
| |
|
| | ### Instruments (1000+ Supported)
|
| |
|
| | | Category | Examples |
|
| | |----------|----------|
|
| | | **Strings** | Acoustic Guitar, Electric Guitar, Violin, Cello, Bass, Harp, Mandolin |
|
| | | **Keys** | Piano, Synthesizer, Organ, Rhodes, Wurlitzer, Mellotron |
|
| | | **Percussion** | Drums, Electronic Drums, Congas, Bongos, Timpani, Vibraphone |
|
| | | **Wind** | Saxophone, Trumpet, Flute, Clarinet, Oboe, French Horn |
|
| | | **Electronic** | Synth Bass, Pad, Lead, Arpeggiator, Sampler, 808, 303 |
|
| |
|
| | ### Structure Analysis
|
| |
|
| | - **Intro / Outro** - Opening and closing sections
|
| | - **Verse / Pre-Chorus / Chorus** - Main song structure
|
| | - **Bridge / Break** - Transitional sections
|
| | - **Build-up / Drop / Climax** - Dynamic progression
|
| | - **Interlude / Solo** - Instrumental passages
|
| |
|
| | ### Timbre Descriptions
|
| |
|
| | | Dimension | Descriptors |
|
| | |-----------|-------------|
|
| | | **Texture** | Warm, Bright, Dark, Crisp, Muddy, Clean, Distorted, Saturated |
|
| | | **Space** | Reverberant, Dry, Spacious, Intimate, Cavernous, Tight |
|
| | | **Dynamics** | Punchy, Soft, Aggressive, Gentle, Compressed, Dynamic |
|
| | | **Character** | Ethereal, Gritty, Smooth, Raw, Polished, Organic, Synthetic |
|
| |
|
| | ## Use Cases
|
| |
|
| | - **Music AI Training** - Generate high-quality captions for music generation models
|
| | - **Music Information Retrieval** - Create searchable metadata for audio databases
|
| | - **Content Moderation** - Analyze and categorize music content
|
| | - **Music Education** - Provide detailed analysis for learning purposes
|
| | - **Audio Production** - Document and describe sound design elements |
