Download index.html from MiniMaxAI/MiniMax-Speech-Tech-Report: direct link, hf CLI and curl.
- Browser
- Download file 41.7 kB
-
https://huggingface.co/spaces/MiniMaxAI/MiniMax-Speech-Tech-Report/resolve/main/index.html
- Command line
-
hf download hf://spaces/MiniMaxAI/MiniMax-Speech-Tech-Report/index.html
-
curl -L -o index.html https://huggingface.co/spaces/MiniMaxAI/MiniMax-Speech-Tech-Report/resolve/main/index.html
41.7 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8" /> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |
| <title>MiniMax-Speech Tech Report | Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder</title> | |
| <meta name="description" | |
| content=" MiniMax-Speech, an autoregressive Transformer-based Text-to-Speech (TTS) model that generates high-quality speech" /> | |
| <meta name="keywords" content="latex.css,css library,class-less css,latex css" /> | |
| <meta property="og:title" | |
| content="MiniMax-Speech Tech Report | Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder" /> | |
| <meta property="og:url" content="https://minimax-ai.github.io/tts_tech_report" /> | |
| <meta property="og:description" | |
| content=" MiniMax-Speech, an autoregressive Transformer-based Text-to-Speech (TTS) model that generates high-quality speech" /> | |
| <meta property="og:type" content="website" /> | |
| <link rel="stylesheet" href="style.css" /> | |
| </head> | |
| <body id="top" class="text-justify"> | |
| <header | |
| style="background-image: url('assets/images/header-bg.jpeg'); background-size: cover; background-position: center; padding: 1rem 0; border-radius: 1rem;"> | |
| <h1>MiniMax-Speech</h1> | |
| <h4 style="font-size: 1.3rem; line-height: 1; text-align: center;">Intrinsic Zero-Shot Text-to-Speech | |
| with a | |
| Learnable Speaker | |
| Encoder</h4> | |
| <p class="author"> | |
| MiniMax Team <span class="date">May 2025</span><br /> | |
| <a style="font-size: 1.1rem;" target="_blank" href="https://arxiv.org/abs/2505.07916">[Tech | |
| Report]</a> | |
| <a style="font-size: 1.1rem; margin-left: 1rem;" target="_blank" | |
| href="https://huggingface.co/datasets/MiniMaxAI/TTS-Multilingual-Test-Set">[Multilingual Test Set]</a> | |
| <a style="font-size: 1.1rem; margin-left: 1rem;" target="_blank" href="https://github.com/MiniMax-AI">[GitHub]</a> | |
| </p> | |
| </header> | |
| <div class="abstract"> | |
| <h2>Abstract</h2> | |
| <p style="text-align: left;"> | |
| We introduce MiniMax-Speech, an autoregressive Transformer-based Text-to-Speech (TTS) model that generates | |
| high-quality | |
| speech. A key innovation is our learnable speaker encoder, which extracts timbre features from a reference audio | |
| without | |
| requiring its transcription. This enables MiniMax-Speech to produce highly expressive speech with timbre | |
| consistent with | |
| the reference in a zero-shot manner, while also supporting one-shot voice cloning with exceptionally high | |
| similarity to | |
| the reference voice. In addition, the overall quality of the synthesized audio is enhanced through the proposed | |
| Flow-VAE. Our model supports 32 languages and demonstrates excellent performance across multiple objective and | |
| subjective evaluations metrics. Notably, it achieves state-of-the-art (SOTA) results on objective voice cloning | |
| metrics | |
| (Word Error Rate and Speaker Similarity) and has secured the top position on the public TTS Arena leaderboard. | |
| Another | |
| key strength of MiniMax-Speech, granted by the robust and disentangled representations from the speaker encoder, | |
| is its | |
| extensibility without modifying the base model, enabling various applications such as: arbitrary voice emotion | |
| control | |
| via LoRA; text to voice (T2V) by synthesizing timbre features directly from text description; and professional | |
| voice | |
| cloning (PVC) by fine-tuning timbre features with additional data. | |
| </p> | |
| </div> | |
| <nav role="navigation" class="toc"> | |
| <h2>Explore MiniMax-Speech</h2> | |
| <p>Welcome to visit | |
| <a href="https://www.minimax.io/audio">MiniMax Audio</a> and | |
| explore our powerful TTS features. | |
| </p> | |
| <h2>Contents</h2> | |
| <ol> | |
| <li> | |
| <a href="#architecture-overview">Architecture Overview</a> | |
| </li> | |
| <li> | |
| <a href="#expressiveness-demonstrations">Expressiveness Demonstrations</a> | |
| <ol> | |
| <li><a href="#showcase-with-high-versatility">Showcase with High Versatility</a></li> | |
| <li><a href="#showcase-with-multiple-generation-attempts">Showcase with Multiple Generation Attempts</a></li> | |
| </ol> | |
| </li> | |
| <li><a href="#zero-shot-vs-one-shot-demonstrations">Zero-Shot vs. One-Shot Demonstrations</a></li> | |
| <li><a href="#multilingual-and-cross-lingual-capabilities-demonstrations">Multilingual and Cross-Lingual | |
| Capabilities Demonstrations</a></li> | |
| <li><a href="#flow-vae-vs-vae-comparisons">Flow-VAE vs. VAE Comparisons</a></li> | |
| <li><a href="#professional-voice-clone-pvc-demonstrations">Professional Voice Clone (PVC) Demonstrations</a></li> | |
| <li><a href="#emotion-control-demonstrations">Emotion Control Demonstrations</a></li> | |
| <li><a href="#text-prompted-voice-generation-demonstrations">Text-Prompted Voice Generation Demonstrations</a> | |
| </li> | |
| <li><a href="#comparison-of-voice-naturalness">Comparison of voice | |
| naturalness with the previous generation products</a></li> | |
| <li><a href="#citation">Citation</a></li> | |
| </ol> | |
| </nav> | |
| <main> | |
| <article> | |
| <div class="article-block"> | |
| <h2 id="architecture-overview">Architecture Overview</h2> | |
| <figure> | |
| <img src="assets/images/system-overview.jpg" loading="lazy" alt="System Architecture" width="100%" | |
| height="auto" /> | |
| <figcaption> | |
| An overview of the architecture of MiniMax-Speech. | |
| </figcaption> | |
| </figure> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="expressiveness-demonstrations">Expressiveness Demonstrations</h2> | |
| <h3 id="showcase-with-high-versatility">Showcase with High Versatility</h3> | |
| <div class="scroll-wrapper"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col" style="width: 40%;">Description</th> | |
| <th scope="col" style="width: 30%; text-align: center;">Source Audio</th> | |
| <th scope="col" style="width: 30%; text-align: center;">Generated Audio</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Compelling and Persuasive Speaker Voice | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Marketing_Voice_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Compelling%20and%20Persuasive.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Clear and Explanatory Voice with Broad Emotional Dynamics Across Different Texts | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Science_Voice_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Explanatory%20Broad%20Emotional.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| Another Explanatory Voice with Supernatural Prosody, <br> | |
| Featuring Distinct Ethnic and Age Characteristics | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Sociology_Sourse.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Explanatory Supernatural Prosody.MP3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Warm and Magnetic Voice that Brings Comfort | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Warm%20and%20Magnetic_Sourse.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Warm%20and%20Magnetic.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| An ASMR Whispering Voice with Generated Breathing and Sound Effects | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Breathy%20ASMR_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Breathy%20ASMR.MP3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Robotic Voice with Rich Bass Resonance and Spatial Presence | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Lucky%20Robot_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Lucky%20Robot.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Sardonic Mature Female Voice | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Onee-san_Sourse.MP3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Onee-san.wav" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <h3 id="showcase-with-multiple-generation-attempts">Showcase with Multiple Generation Attempts, Post-Processing | |
| Audio Effects and Added Sound Effects</h3> | |
| <div class="scroll-wrapper"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col" style="width: 50%;">Description</th> | |
| <th scope="col" style="width: 50%; text-align: center;">Generated Audio</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| A Husky Male Voice: From Soft Murmur to Excitement to Anger, then to Whispers | |
| </td> | |
| <td> | |
| <audio class="audio-lg" src="assets/audios/Murmur-Excitement-Anger-%20Whispers.MP3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| An Angry Female Voice: From Soft Murmur to Rage to Reminiscence, then to Weeping | |
| </td> | |
| <td> | |
| <audio class="audio-lg" src="assets/audios/Neutral-Rage-Reminiscence-Weeping.MP3" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="zero-shot-vs-one-shot-demonstrations">Zero-Shot vs. One-Shot Demonstrations</h2> | |
| <p> | |
| ZeroShot maintains speaker identity while generating more natural emotions, pauses, and other expressive | |
| features based | |
| on the text content, whereas OneShot adheres more strictly to the speaker characteristics (prosody, speech | |
| rate, | |
| emotions, etc.). For details of Zero-Shot and One-Shot, refer to the <a | |
| href="https://arxiv.org/abs/2505.07916" target="_blank">technical report</a>. | |
| </p> | |
| <div class="scroll-wrapper" style="margin-top: 2rem;"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Source Audio</th> | |
| <th scope="col">Text</th> | |
| <th scope="col">Zero-Shot Version</th> | |
| <th scope="col">One-Shot Version</th> | |
| <th scope="col">Elevenlabs Multilingual_v2</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Lyrical%20Cantonese_Prompt.WAV" controls></audio> | |
| </td> | |
| <td> | |
| 命运就算颠沛流离,<br> | |
| 命运就算曲折离奇,<br> | |
| 命运就算恐吓着你,<br> | |
| 做人没趣味。<br> | |
| 别流泪,心酸,更不应舍弃。<br> | |
| 我愿能,一生永远陪伴你。 | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Lyrical%20Cantonese_ZeroShot.mp3" controls></audio> | |
| Preserving Distinctive Voice<br> | |
| Timbre and Expressive <br> | |
| Prosody with Regularized <br> | |
| Pausing and Speech Rate | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Lyrical%20Cantonese_Oneshot.mp3" controls></audio> | |
| Better Reproduction of<br> | |
| Prompt's Exaggerated Speech<br> | |
| Rate and Characteristic<br> | |
| Phrase-Initial Pauses | |
| </td> | |
| <td> | |
| Cantonese not supported | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Breaking%20Down%20Mandarin_Prompt.WAV" controls></audio> | |
| </td> | |
| <td> | |
| 你们这些躲在道德高地的懦夫,<br> | |
| 敢承认自己对本我的恐惧吗?<br> | |
| 回答我!嗯?你回答我!<br> | |
| Look in my eyes!<br> | |
| 老子写梦的解析时<br> | |
| 你们还在玩泥巴,<br> | |
| 我精神分析引论每个字母都能<br> | |
| 刺穿文明社会的虚伪面具,<br> | |
| 我解剖潜意识就像<br> | |
| 外科医生划开皮肤。<br> | |
| 是不是啊?说话! | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Breaking%20Down%20Mandarin_ZeroShot.mp3" controls></audio> | |
| Capable of Generating<br> | |
| Relatively Calmer Emotions<br> | |
| while Preserving Voice<br> | |
| Identity | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Breaking%20Down%20Mandarin_OneShot.mp3" controls></audio> | |
| Consistently Reproducing the<br> | |
| Angry Emotion from Prompt<br> | |
| in Every Utterance | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Breaking Down Mandarin.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Quirky%20Female%20English_Prompt.MP3" controls></audio> | |
| </td> | |
| <td> | |
| Would you believe what happened at the<br> | |
| grocery store today? My goodness! The<br> | |
| avocados were on sale - half price! Half<br> | |
| price! I bought twenty of them! | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Quirky%20Female%20English_ZeroShot.MP3" controls></audio> | |
| Effectively follows textual cues<br> | |
| for both longer and shorter<br> | |
| inter-sentence pauses | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Quirky%20Female%20English_OneShot.MP3" controls></audio> | |
| Better reproduces the<br> | |
| exaggerated high pitch<br> | |
| characteristic of anime voices<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Quirky%20Female%20English.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Neurotic%20Teenage%20English_Prompt.MP3" controls></audio> | |
| </td> | |
| <td> | |
| Oh my gosh, like, I literally can't believe<br> | |
| what just happened! Um, so basically, I was,<br> | |
| you know, just sitting there in class,<br> | |
| right? And then, ugh, this totally weird<br> | |
| thing happened - like, seriously weird! Wait,<br> | |
| wait... Should I even be talking about this?<br> | |
| Ugh, whatever. | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Neurotic%20Teenage%20English_ZeroShot.MP3" | |
| controls></audio> | |
| Effectively follows textual cues<br> | |
| for both longer and shorter<br> | |
| inter-sentence pauses | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Neurotic%20Teenage%20English_OneShot.MP3" controls></audio> | |
| Better reproduces the<br> | |
| exaggerated high pitch<br> | |
| characteristic of anime voices<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Neurotic%20Teenage%20English.mp3" | |
| controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="multilingual-and-cross-lingual-capabilities-demonstrations">Multilingual and Cross-Lingual Capabilities | |
| Demonstrations</h2> | |
| <p>Speech-02-HD maintains high naturalness in less common languages while demonstrating significant advantages | |
| in | |
| Standard | |
| Chinese pronunciation accuracy.</p> | |
| <div class="scroll-wrapper" style="margin-top: 2rem;"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Languages</th> | |
| <th scope="col">Source Audio</th> | |
| <th scope="col">Text</th> | |
| <th scope="col">MiniMax<br>Speech_02_HD</th> | |
| <th scope="col">ElevenLabs<br>Multilingual_v2</th> | |
| <th scope="col">OpenAI<br>TTS_1_HD<br>(*not cloned voice)</th> | |
| </tr> | |
| <!-- Thai --> | |
| <tr class="border-bottom-thin"> | |
| <th>Thai</th> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Thai_Male_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| สวัสดีค่ะ วันนี้อากาศดีมากเลย<br> | |
| คุณจะไปทานอาหารกลางวันที่ไหนคะ<br> | |
| ฉันกำลังคิดว่าจะไปร้านอาหารไทยแถวนี้<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Thai.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Thai not perfectly supported | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Thai.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/OpenAI_Thai.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <!-- Vietnamese --> | |
| <tr class="border-bottom-thin"> | |
| <th>Vietnamese</th> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Vietnamese_Female_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| Tôi đang đọc một cuốn sách rất hay về lịch sử Việt Nam.<br> | |
| Những câu chuyện về văn hóa truyền<br> | |
| thống thật sự rất thú vị.<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Vietnamese.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Vietnamese not perfectly supported | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Vietnamese.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/OpenAI_Vietnamese.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <!-- Czech --> | |
| <tr class="border-bottom-thin"> | |
| <th>Czech</th> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Czech_Female_Sourse.wav" controls></audio> | |
| </td> | |
| <td> | |
| Ranní mlha se pomalu zvedá nad řekou,<br> | |
| zatímco první paprsky slunce prosvítají mezi stromy.<br> | |
| Ptáci začínají svůj ranní koncert.<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Czech.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Czech.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/OpenAI_Czech.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <!-- Polish --> | |
| <tr class="border-bottom-thin"> | |
| <th>Polish</th> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Polish_Male_Sourse.wav" controls></audio>、 | |
| </td> | |
| <td> | |
| Młoda sowa siedzi cicho na gałęzi sosny,<br> | |
| obserwując leśną polanę w świetle księżyca.<br> | |
| Wiatr delikatnie porusza liśćmi drzew.<br> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Polish.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Polish.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/OpenAI_Polish.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <!-- Japanese --> | |
| <tr class="border-bottom-thin"> | |
| <th>Japanese</th> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Japanese_DominantMan_Sourse.mp3" controls></audio> | |
| </td> | |
| <td> | |
| 電車が遅延している影響で、渋谷駅がとても混雑<br> | |
| しています。次の山手線は約10分後に到着<br> | |
| 予定です。お急ぎのお客様は、他の路線も<br> | |
| ご利用ください。 | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Japanese.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ElevenLabs_Japanese_Dominant_Man.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/OpenAI_Japanese.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p style="margin-top: 4rem;">Speech-02-HD has superior performance in zero-shot cross-lingual scenarios.</p> | |
| <div class="scroll-wrapper" style="margin-top: 2rem;"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Original Language</th> | |
| <th scope="col">Source Audio</th> | |
| <th scope="col">Mixed Language</th> | |
| <th scope="col">Text</th> | |
| <th scope="col">MiniMax<br>Speech_02_HD</th> | |
| <th scope="col">ElevenLabs<br>Multilingual_v2</th> | |
| <th scope="col">OpenAI<br>TTS_1_HD<br>(*not cloned voice)</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td>English</td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Wong_Sourse.mp3" controls></audio> | |
| </td> | |
| <td>English + Mandarin</td> | |
| <td> | |
| Kiddo! Come come come, 学如逆水行舟,不进则退。<br> | |
| I see you're using AI tools already - so smart!<br> | |
| But eh, cannot just rely on tools only lah!<br> | |
| The future belongs to those who can work alongside AI,<br> | |
| not those scared of it. | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/English-Mandarin.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/ElevenLabs_English-Mandarin.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/OpenAI_English-Mandarin.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td>Mandarin</td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ShiBanYu_Sourse.mp3" controls></audio> | |
| </td> | |
| <td>Mandarin + Cantonese</td> | |
| <td> | |
| 老铁啊,多谢晒你送我呢本,广州话正音字典,咁好嘢喎!<br> | |
| 我呢个大老爷们儿学广州话真系好难㗎!成日都分唔清声调啊。<br> | |
| 嗱,而家有咗呢本书,什么都好啦。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Mandarin-Cantonese.MP3" controls></audio> | |
| </td> | |
| <td> | |
| Cantonese not supported | |
| </td> | |
| <td> | |
| Cantonese not supported | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td>Mandarin</td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/ShuanQ_Sourse.mp3" controls></audio> | |
| </td> | |
| <td>Mandarin + English</td> | |
| <td> | |
| The people said, 桂林's scenery is the first under heaven.<br> | |
| Yet in my opinion, 阳朔 scenery is better than 桂林。<br> | |
| 群峰倒影山浮水,无水无山不入神。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Mandarin-English.WAV" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/ElevenLabs_Mandarin-English.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/OpenAI_Mandarin-English.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td>English</td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/CoCo_Sourse.mp3" controls></audio> | |
| </td> | |
| <td>English + Spanish</td> | |
| <td> | |
| Mi abuelita always told me "el que persevera, alcanza".<br> | |
| If you persevere, you'll achieve your dreams!<br> | |
| Guess what! They choose me to play the lead role in our BIG show! | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/English-Spanish.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/ElevenLabs_English-Spanish.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/OpenAI_English-Spanish.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td>Japanese</td> | |
| <td> | |
| <audio class="audio-sm" src="assets/audios/Powerful_Girl_Sourse.mp3" controls></audio> | |
| </td> | |
| <td>Japanese + Korean</td> | |
| <td> | |
| 最近の天気予報によりますと、今週末は桜の開花に最適<br> | |
| な気温になる予定です。<br> | |
| 東京都内の各公園では花見客で賑わうことが予想されますが、<br> | |
| 서울에서도 벚꽃이 피기 시작했다고 하네요.<br> | |
| 이번 주말에는 여의도 공원에서 벚꽃 축제가 열린다고 하니<br> | |
| 많은 분들이 찾아오실 것 같습니다. | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Japanese-Korean.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/ElevenLabs_Japanese-Korean.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/OpenAI_Japanese-Korean.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| <p>*Although OpenAI currently does not support voice cloning functionality, we still wish to conduct comparative | |
| listening | |
| tests with its excellent naturalness as a reference.</p> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="flow-vae-vs-vae-comparisons">Flow-VAE vs. VAE Comparison</h2> | |
| <p>Flow-VAE is less likely to produce the following instabilities.</p> | |
| <div class="scroll-wrapper" style="margin-top: 2rem;"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col" style="text-align: center;">Source Audio</th> | |
| <th scope="col" style="text-align: center;">Flow-VAE</th> | |
| <th scope="col" style="text-align: center;">VAE</th> | |
| <th scope="col" style="text-align: center;">Differences</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/Condition1.wav" controls></audio> | |
| </td> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/FlowVAE1.wav" controls></audio> | |
| </td> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/VAE1.wav" controls></audio> | |
| </td> | |
| <td> | |
| Flow-VAE reproduces more continuous<br> | |
| and natural reverberation | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio src="assets/audios/Condition2.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/FlowVAE2.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/VAE2.wav" controls></audio> | |
| </td> | |
| <td> | |
| VAE introduces unwanted<br> | |
| high-frequency components | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| <audio src="assets/audios/Conditon3.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/FlowVAE3.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/VAE3.wav" controls></audio> | |
| </td> | |
| <td> | |
| VAE produces electronic-sounding<br> | |
| artifacts at the beginning | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="professional-voice-clone-pvc-demonstrations">Professional Voice Clone (PVC) Demonstrations</h2> | |
| <p>For more complex dialectal accents and tonal characteristics, PVC can reproduce these features while | |
| maintaining high | |
| naturalness based on the text content.</p> | |
| <div class="scroll-wrapper" style="margin-top: 2rem;"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col" style="text-align: center;">Source Audio</th> | |
| <th scope="col" style="text-align: center;">Zero-Shot</th> | |
| <th scope="col" style="text-align: center;">PVC</th> | |
| <th scope="col" style="text-align: center;">Differences</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/JosephBrodsky_Source.wav" controls></audio> | |
| </td> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/JosephBrodsky_Fast.mp3" controls></audio> | |
| </td> | |
| <td style="width: 25%"> | |
| <audio src="assets/audios/JosephBrodsky_PVC.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Like the ZeroShot version, the PVC<br> | |
| version has rising sentence-final intonation,<br> | |
| but distinctively sustains this<br> | |
| elevated pitch instead of the typical<br> | |
| pitch declination found in common<br> | |
| declarative sentences | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio src="assets/audios/TianJin_Source.wav" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/TianJin_Fast.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio src="assets/audios/TianJin_PVC.mp3" controls></audio> | |
| </td> | |
| <td> | |
| With more materials, the model not only<br> | |
| reproduces the speaker's voice characteristics<br> | |
| but also accurately captures more<br> | |
| dialectal features | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="emotion-control-demonstrations">Emotion Control Demonstrations</h2> | |
| <h3>Source Audio for Refreshing Young Man</h3> | |
| <audio src="assets/audios/Mandarin_Refreshing_Young_Man_Sourse.mp3" controls></audio> | |
| <h3>DEMO</h3> | |
| <div class="scroll-wrapper"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Neutral</th> | |
| <th scope="col" style="min-width: 120px;">Emotion</th> | |
| <th scope="col">Text</th> | |
| <th scope="col">Emotion Control Audio</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral1.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Surprised | |
| </td> | |
| <td> | |
| 天哪!我完全没想到会在这里遇见你,<br> | |
| 都过去这么多年了,你一点都没变! | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Surprised.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral2.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Disgusted | |
| </td> | |
| <td> | |
| 这个地方实在太脏乱了,到处都是垃圾和难闻的气味儿,<br> | |
| 我一秒钟都不想多待。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Disgusted.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral3.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Fearful | |
| </td> | |
| <td> | |
| 深夜回家的路上,我清楚地听见身后有脚步声在跟着我,<br> | |
| 可是回头却什么都看不见。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Fearful.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral4.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Angry | |
| </td> | |
| <td> | |
| 我付出了这么多,换来的却是这样的背叛!<br> | |
| 你怎么可以这样对待我的信任! | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Angry.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral5.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Sad | |
| </td> | |
| <td> | |
| 躺在床上翻来覆去,心里压着说不出的难过和沮丧,<br> | |
| 昨天晚上又失眠了。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Sad.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Neutral6.mp3" controls></audio> | |
| </td> | |
| <td> | |
| Happy | |
| </td> | |
| <td> | |
| 和好朋友一起在院子里烧烤,聊着有趣的故事,<br> | |
| 享受着美食和欢乐的时光。 | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Happy.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="text-prompted-voice-generation-demonstrations">Text-Prompted Voice Generation Demonstrations</h2> | |
| <div class="scroll-wrapper"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Prompt</th> | |
| <th scope="col">Text</th> | |
| <th scope="col" style="text-align: center;">Audio</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| 男性中年声音,说中文,音色浑厚醇厚,带有自然的磁性,<br> | |
| 语速偏慢,音量适中,音调偏低沉。声音整体给人沉稳可靠的感觉,<br> | |
| 在深度访谈场景中表现出专业性和亲和力,音质清晰,吐字规整有力。 | |
| </td> | |
| <td> | |
| 在这个安静的夜晚,让我们一起走进《人生笔记》这本书。<br> | |
| 作者用平实的文字记录下生活中的点点滴滴,<br> | |
| 让我们看到平凡中的真善美。<br> | |
| 今天,我们先来读第一章:'生活的痕迹'...... | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/深度访谈男中年.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| 说中文的女青年,音色偏甜美,语速比较快,<br> | |
| 说话时带着一种轻快的感觉,整体音调较高,像是在直播带货,<br> | |
| 整体氛围比较活跃,声音清晰,听起来很有亲和力。 | |
| </td> | |
| <td> | |
| 亲爱的宝宝们,等了好久的神仙面霜终于到货啦!<br> | |
| 你们看这个包装是不是超级精致?<br> | |
| 我自己已经用了一个月了,效果真的绝绝子!<br> | |
| 而且这次活动价真的太划算了,错过真的会后悔的哦~ | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/直播带货女青年.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| 中国男性声音,听着像是青年,音色清亮,语速比较快,<br> | |
| 说话很有激情,像是在解说比赛,声音中带着紧张和兴奋的感觉。 | |
| </td> | |
| <td> | |
| 漂亮!这个进攻太精彩了!张伟突破防线,<br> | |
| 一个漂亮的转身,球传到禁区,王超跟上,射门!<br> | |
| 球进了!难以置信的精彩配合,现场观众都沸腾了! | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/体育解说男青年.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| 中国女青年的声音,音色清脆,说话速度偏快,语调活泼,<br> | |
| 像是在做游戏直播,声音中带着愉快的感觉,整体音调较高,<br> | |
| 整体氛围比较轻松。 | |
| </td> | |
| <td> | |
| 啊!这里有个宝箱!让我们看看里面是什么~<br> | |
| 哇!是传说中的紫色装备!运气也太好了吧!<br> | |
| 谢谢小伙伴们的打赏,我们继续往前探索...... | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/游戏主播女青年.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| English-speaking female voice, sounding relatively young,<br> | |
| with a sweet and pleasant tone. Speaking at a moderate pace<br> | |
| with a touch of energy, similar to someone narrating a<br> | |
| beauty/makeup tutorial video. The overall atmosphere is<br> | |
| relaxed and cheerful. | |
| </td> | |
| <td> | |
| Hi everyone! Today I'll be sharing a soft, romantic<br> | |
| makeup look that's perfect for dates. Many of you have <br> | |
| been asking how to apply this eyeshadow naturally - the<br> | |
| key is using gentle techniques. Let's go through the<br> | |
| steps together... | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/美妆女博主.wav" controls></audio> | |
| </td> | |
| </tr> | |
| <tr> | |
| <td> | |
| English-speaking middle-aged male voice, slightly husky, <br> | |
| speaking at a moderate-to-slow pace with a deep tone. Like<br> | |
| someone telling an old story, conveying a nostalgic feeling,<br> | |
| with a relaxed and composed manner of speaking. | |
| </td> | |
| <td> | |
| That was back in the late 1970s. I remember when our <br> | |
| village first got electricity - everyone was so excited. <br> | |
| In theevenings, people would bring their stools and <br> | |
| gather under the big banyan tree by the village committee <br> | |
| office to watch movies projected on the wall. Even now, <br> | |
| thinking back to those moments still fills me with warmth. | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/回忆男中年.wav" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="comparison-of-voice-naturalness">Comparison of voice naturalness | |
| with the previous generation products</h2> | |
| <p>The new model demonstrates significant advantages in naturalness compared to the previous version.</p> | |
| <h3 style="margin-top: 2rem;">Source Audio for Radiant_Girl</h3> | |
| <audio src="assets/audios/English_Radiant_Girl_Sourse.wav" controls></audio> | |
| <h3>DEMO</h3> | |
| <div class="scroll-wrapper"> | |
| <table style="width: 100%;"> | |
| <tbody> | |
| <tr class="border-bottom-thin"> | |
| <th scope="col">Text</th> | |
| <th scope="col" style="text-align: center;">MiniMax<br>Speech_02_HD</th> | |
| <th scope="col" style="text-align: center;">Microsoft<br>Azure TTS</th> | |
| <th scope="col" style="text-align: center;">AWS<br>Polly</th> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| I sat alone in the empty room, staring at the old photographs,<br> | |
| wondering how everything could change so quickly,<br> | |
| how a lifetime of memories could fade away just like that. | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Radiant_Girl_1.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Emma_1.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Joanna_1.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| <tr class="border-bottom-thin"> | |
| <td> | |
| The moment I held my acceptance letter, my heart burst with joy - <br> | |
| all those sleepless nights finally paid off, and I couldn't stop<br> | |
| dancing around the room, calling everyone I knew to share this amazing news! | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Radiant_Girl_2.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Emma_2.mp3" controls></audio> | |
| </td> | |
| <td> | |
| <audio class="audio-md" src="assets/audios/Joanna_2.mp3" controls></audio> | |
| </td> | |
| </tr> | |
| </tbody> | |
| </table> | |
| </div> | |
| </div> | |
| <div class="article-block"> | |
| <h2 id="citation">Citation</h2> | |
| <div> | |
| <pre> | |
| <code> | |
| @misc{minimax2025minimaxspeechintrinsiczeroshottexttospeech, | |
| title={MiniMax-Speech: Intrinsic Zero-Shot Text-to-Speech with a Learnable Speaker Encoder}, | |
| author={Bowen Zhang, Congchao Guo, Geng Yang, Hang Yu, Haozhe Zhang, Heidi Lei, Jialong Mai, Junjie Yan, Kaiyue Yang, | |
| Mingqi Yang, Peikai Huang, Ruiyang Jin, Sitan Jiang, Weihua Cheng, Yawei Li, Yichen Xiao, Yiying Zhou, Yongmao Zhang, | |
| Yuan Lu, Yucen He}, | |
| year={2025}, | |
| eprint={2505.07916}, | |
| archivePrefix={arXiv}, | |
| primaryClass={eess.AS}, | |
| url={https://arxiv.org/abs/2505.07916}, | |
| }</code> | |
| </pre> | |
| </div> | |
| </div> | |
| </article> | |
| </main> | |
| <script> | |
| MathJax = { | |
| tex: { | |
| inlineMath: [['$', '$'],], | |
| }, | |
| } | |
| const darkModeToggle = document.getElementById('dark-mode-toggle') | |
| darkModeToggle.addEventListener('click', () => { | |
| document.body.classList.toggle('latex-dark') | |
| }) | |
| </script> | |
| <script id="MathJax-script" async src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js"></script> | |
| </body> | |
| </html> |