Spaces:
Sleeping
Sleeping
| title: VoiceKit MCP | |
| emoji: 🎤 | |
| colorFrom: purple | |
| colorTo: indigo | |
| sdk: gradio | |
| sdk_version: "6.0.0" | |
| app_file: app.py | |
| pinned: false | |
| tags: | |
| - building-mcp-track-creative | |
| - mcp-server | |
| # 🎤 VoiceKit MCP | |
| > **Professional voice analysis as MCP tools — extract embeddings, compare voices, transcribe speech, and more.** | |
| 6 powerful MCP tools for voice processing, all accepting base64-encoded audio. | |
| 📢 **Social Post:** [View on X](https://x.com/dahee_pk/status/1994389505898582442)<br> | |
| 🎬 **Demo Video:** [Watch on YouTube](https://www.youtube.com/watch?v=1VIqvpwfyWU)<br> | |
| 👥 **Team:** [@EricYoun](https://huggingface.co/EricYoun), [@NickEo](https://huggingface.co/NickEo), [@HYENA-WON](https://huggingface.co/HYENA-WON), [@jjin6573](https://huggingface.co/jjin6573), [@cocoajoa](https://huggingface.co/cocoajoa) | |
| --- | |
| ## 📋 Submission Info | |
| | | | | |
| |---|---| | |
| | **Track** | Building MCP — Creative | | |
| | **MCP Endpoint** | `https://mcp-1st-birthday-voicekit.hf.space/gradio_api/mcp/sse` | | |
| | **Framework** | Gradio 6.0 | | |
| --- | |
| ## ✅ Track 1 Requirements | |
| | Requirement | How We Fulfill It | | |
| |-------------|-------------------| | |
| | **Functioning MCP Server** | 6 MCP tools exposed via Gradio's `mcp_server=True` | | |
| | **MCP Client Demo** | Video shows integration with Claude Desktop / MCP client | | |
| | **Documented Tools** | Full API documentation with inputs/outputs below | | |
| | **Gradio App** | Interactive demo UI + hidden MCP tool interfaces | | |
| --- | |
| ## 🛠️ MCP Tools (6 Tools) | |
| All tools accept **base64-encoded audio** as input. | |
| ### 1. `extract_embedding` <img src="icons/extract_embedding.svg" width="20" height="20"> | |
| Extract voice embeddings using Wav2Vec2 model. | |
| | | | | |
| |---|---| | |
| | **Input** | `audio_base64` (base64-encoded audio) | | |
| | **Output** | `embedding_preview` (first 5 values), `embedding_length` (768) | | |
| | **Use Case** | Speaker identification, voice fingerprinting | | |
| <img src="imgs/extract_embedding.jpg" height="300"> | |
| ### 2. `match_voice` <img src="icons/match_voice.svg" width="20" height="20"> | |
| Compare similarity between two voices. | |
| | | | | |
| |---|---| | |
| | **Inputs** | `audio1_base64`, `audio2_base64` | | |
| | **Output** | `similarity` (0-1), `tone_score` (0-100) | | |
| | **Use Case** | Voice cloning verification, speaker matching | | |
| <img src="imgs/match_voice.jpg" height="300"> | |
| ### 3. `analyze_acoustics` <img src="icons/analyze_acoustics.svg" width="20" height="20"> | |
| Extract detailed acoustic characteristics. | |
| | | | | |
| |---|---| | |
| | **Input** | `audio_base64` | | |
| | **Output** | Pitch, energy, rhythm, tempo, spectral info | | |
| | **Use Case** | Emotional tone detection, voice profiling | | |
| <img src="imgs/analyze_acoustics.jpg" height="300"> | |
| ### 4. `transcribe_audio` <img src="icons/transcribe_audio.svg" width="20" height="20"> | |
| Convert speech to text (multilingual). | |
| | | | | |
| |---|---| | |
| | **Inputs** | `audio_base64`, `language` (default: "en") | | |
| | **Output** | Transcribed text, detected language | | |
| | **Model** | ElevenLabs Scribe v1 | | |
| | **Languages** | English, Korean, Japanese, and 15+ more | | |
| <img src="imgs/transcribe_audio.jpg" height="300"> | |
| ### 5. `isolate_voice` <img src="icons/isolate_voice.svg" width="20" height="20"> | |
| Remove background music/noise and extract clean voice. | |
| | | | | |
| |---|---| | |
| | **Input** | `audio_base64` (audio with background sounds) | | |
| | **Output** | Isolated audio (base64), BGM detection status | | |
| | **Use Case** | Audio cleanup for memes, songs, movies | | |
| <img src="imgs/isolate_voice.jpg" height="300"> | |
| ### 6. `grade_voice` <img src="icons/grade_voice.svg" width="20" height="20"> | |
| Comprehensive voice comparison with multi-metric scoring. | |
| | | | | |
| |---|---| | |
| | **Inputs** | `user_audio_base64`, `reference_audio_base64`, `reference_text` (optional), `category` (meme\|song\|movie) | | |
| | **Output** | Pitch, rhythm, energy, pronunciation scores (0-100), overall score, user transcription | | |
| | **Use Case** | Voice mimicry evaluation, pronunciation games | | |
| <img src="imgs/grade_voice.jpg" height="300"> | |
| --- | |
| ## 🏗️ Architecture | |
| ``` | |
| ┌─────────────────────────────────────────────────────────────────┐ | |
| │ VoiceKit MCP │ | |
| ├─────────────────────────────────────────────────────────────────┤ | |
| │ │ | |
| │ ┌────────────────────────────────────────────────────────────┐ │ | |
| │ │ MCP Client (Claude) │ │ | |
| │ │ base64 audio → SSE endpoint │ │ | |
| │ └──────────────────────────┬─────────────────────────────────┘ │ | |
| │ ↓ │ | |
| │ ┌────────────────────────────────────────────────────────────┐ │ | |
| │ │ Gradio MCP Server (app.py) │ │ | |
| │ │ mcp_server=True • 6 tool interfaces │ │ | |
| │ └──────────────────────────┬─────────────────────────────────┘ │ | |
| │ ↓ │ | |
| │ ┌────────────────────────────────────────────────────────────┐ │ | |
| │ │ Modal GPU Container (T4) │ │ | |
| │ │ Wav2Vec2 • librosa • ElevenLabs APIs • DTW │ │ | |
| │ └──────────────────────────┬─────────────────────────────────┘ │ | |
| │ ↓ │ | |
| │ ┌────────────────────────────────────────────────────────────┐ │ | |
| │ │ JSON Response │ │ | |
| │ │ embeddings • scores • transcripts • audio │ │ | |
| │ └────────────────────────────────────────────────────────────┘ │ | |
| │ │ | |
| └─────────────────────────────────────────────────────────────────┘ | |
| ``` | |
| --- | |
| ## 🔌 How to Connect | |
| ### Claude Desktop / MCP Client | |
| Add to your MCP configuration: | |
| ```json | |
| { | |
| "mcpServers": { | |
| "voicekit": { | |
| "url": "https://mcp-1st-birthday-voicekit.hf.space/gradio_api/mcp/sse" | |
| } | |
| } | |
| } | |
| ``` | |
| ### Example Usage | |
| ```python | |
| # 1. Encode audio to base64 | |
| import base64 | |
| with open("audio.wav", "rb") as f: | |
| audio_base64 = base64.b64encode(f.read()).decode() | |
| # 2. Call MCP tool | |
| result = mcp_client.call("extract_embedding", {"audio_base64": audio_base64}) | |
| # 3. Use the 768-dim embedding | |
| embedding = result["embedding"] | |
| ``` | |
| --- | |
| ## 🛠️ Tech Stack | |
| | Component | Technology | | |
| |-----------|------------| | |
| | MCP Server | Gradio 6.0 (`mcp_server=True`) | | |
| | GPU Compute | Modal (T4 GPU) | | |
| | Embeddings | Wav2Vec2 (facebook/wav2vec2-base-960h) | | |
| | Speech-to-Text | ElevenLabs Scribe v1 | | |
| | Voice Isolation | ElevenLabs Voice Isolator | | |
| | Acoustic Analysis | librosa + scipy | | |
| --- | |
| ## ⚡ Performance | |
| | Metric | Value | | |
| |--------|-------| | |
| | Response Time (warm) | <200ms | | |
| | Cold Start | 1-3s (memory snapshot optimized) | | |
| | Embedding Dimensions | 768 | | |
| | Supported Audio | Any format (auto-converts to WAV) | | |
| | Max Duration | Tested up to 10 minutes | | |
| --- | |
| ## 🎯 Why VoiceKit MCP? | |
| | Criteria | Our Approach | | |
| |----------|--------------| | |
| | **Functionality** | 6 production-ready tools covering full voice analysis pipeline | | |
| | **Innovation** | First MCP server for comprehensive voice analysis | | |
| | **Documentation** | Complete API docs with inputs/outputs/use cases | | |
| | **Real-world Impact** | Powers Voice Sementle game; applicable to voice cloning, accessibility, language learning | | |
| --- | |
| ## 🎮 Interactive Demo | |
| 👆 **Click the interface above to try each tool!** | |
| 1. Upload or record audio | |
| 2. Select a tool to test | |
| 3. View JSON results with scores and analysis | |
| 4. Copy embeddings or transcripts for your app | |
| --- | |
| ## 🔗 Related Projects | |
| - **[Voice Sementle](https://huggingface.co/spaces/MCP-1st-Birthday/Voice-Sementle)** — Daily voice puzzle game powered by VoiceKit MCP | |
| --- | |
| **Built for [MCP's 1st Birthday Hackathon](https://huggingface.co/MCP-1st-Birthday)** 🎂 | |
| *Celebrating one year of Model Context Protocol!* |