File size: 43,132 Bytes
f24a0a3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 | {
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# 🎙️ VoxCPM2 - Interactive TTS Web App (Gradio UI)\n",
"**Launch your own voice cloning & synthesis web app in Google Colab!**\n",
"\n",
"[Model: openbmb/VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) | [Paper](https://arxiv.org/abs/2509.24650) | [Docs](https://voxcpm.readthedocs.io/)\n",
"\n",
"### 🚀 MAIN FEATURE: Interactive Gradio Web UI\n",
"Run a beautiful web interface with tabs for:\n",
"- 🔊 **Basic TTS** — Type text, get 48kHz studio-quality speech\n",
"- 🎨 **Voice Design** — Describe any voice with natural language\n",
"- 👤 **Voice Clone** — Upload a 3-10s voice sample, clone it!\n",
"- 🌐 **Multilingual** — Type in any language → auto-detects & speaks it\n",
"- 🌐👤 **Multilingual + Clone** — Clone a voice & speak ANY language\n",
"- 🌐🎨 **Multilingual + Design** — Design a voice & speak ANY language\n",
"\n",
"**You'll get a public URL** to share or use from your phone!\n",
"\n",
"---\n",
"\n",
"### ⚙️ Quick Setup (2 steps then launch UI)\n",
"1. Run **Step 1** → Install packages\n",
"2. Run **Step 2** → Load model\n",
"3. Run **Step 3 (MAIN)** → Launch Gradio UI 🎯\n",
"\n",
"*Optional: Scroll down for individual mode demos if you want to experiment with code.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 📦 Step 1: Install Dependencies (Run First)\n",
"*Note: First run downloads ~4.6GB model weights. Takes ~3-5 minutes.*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!pip install -q voxcpm soundfile gradio\n",
"\n",
"import torch\n",
"print(f\"PyTorch: {torch.__version__}\")\n",
"print(f\"CUDA available: {torch.cuda.is_available()}\")\n",
"if torch.cuda.is_available():\n",
" print(f\"GPU: {torch.cuda.get_device_name(0)}\")\n",
" print(f\"VRAM: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB\")\n",
"else:\n",
" print(\"\\n⚠️ WARNING: No GPU detected! VoxCPM2 needs a GPU runtime.\")\n",
" print(\"➡️ Run: Runtime -> Change runtime type -> T4 GPU -> Save -> Restart session\")\n",
" print(\"Then re-run all cells from the top.\")\n",
" raise SystemExit(\"Please switch to GPU runtime and restart.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 🔄 Step 2: Load Model (Run Second)\n",
"**⚠️ Must use GPU runtime.** (Runtime → Change runtime type → T4 GPU)\n",
"\n",
"*Torch compilation (`optimize=True`) is disabled on T4 because bfloat16 is not natively supported.*\n",
"*This saves ~50-100 seconds of failed compilation during warmup.*\n",
"\n",
"*If CPU is detected, model auto-converts to float32 (slower but works).*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from voxcpm import VoxCPM\n",
"import time\n",
"\n",
"print(\"\\n\" + \"=\"*60)\n",
"print(\"LOADING VoxCPM2...\")\n",
"print(\"=\"*60 + \"\\n\")\n",
"\n",
"import torch\n",
"device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
"print(f\"Detected device: {device}\")\n",
"\n",
"start = time.time()\n",
"\n",
"# ⚡ SPEED TIP: T4 GPU does NOT support bfloat16 torch.compile natively.\n",
"# Setting optimize=False saves ~50-100 seconds of failed compilation during warmup.\n",
"model = VoxCPM.from_pretrained(\n",
" \"openbmb/VoxCPM2\",\n",
" load_denoiser=False, # Disable to save VRAM (~500MB)\n",
" optimize=False, # torch.compile fails on T4 bfloat16, skip it\n",
")\n",
"\n",
"elapsed = time.time() - start\n",
"print(f\"\\n✅ Model loaded in {elapsed:.1f}s\")\n",
"print(f\"📢 Output sample rate: {model.tts_model.sample_rate} Hz\")\n",
"print(f\"🖥️ Device: {device}\")\n",
"\n",
"# CPU fallback: bfloat16 crashes on CPU, convert to float32\n",
"if not torch.cuda.is_available():\n",
" print(\"\\n⚠️ CPU detected! Converting model from bfloat16 to float32...\")\n",
" model.tts_model = model.tts_model.to(torch.float32)\n",
" print(\"✅ Model converted to float32 (slower but works on CPU)\")\n",
"else:\n",
" print(f\"🎮 GPU VRAM used: {torch.cuda.memory_allocated() / 1024**3:.1f} GB\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 🎯 Step 3 (MAIN): Launch Interactive Gradio Web UI\n",
"**This is the primary way to use VoxCPM2.** Run this cell and you'll get a public URL!\n",
"\n",
"### What's inside the UI:\n",
"| Tab | What it does |\n",
"|-----|-------------|\n",
"| 🔊 **Basic TTS** | Type any text → get 48kHz speech |\n",
"| 🎨 **Voice Design** | Describe a voice (e.g. \"warm female narrator\") → get custom voice |\n",
"| 👤 **Voice Clone** | Upload 3-10s audio → clone that voice |\n",
"| 🌐 **Multilingual** | Type in any language → auto-detects & speaks it |\n",
"| 🌐👤 **Multilingual + Clone** | Clone a voice AND speak in ANY language |\n",
"| 🌐🎨 **Multilingual + Design** | Design a voice AND speak in ANY language |\n",
"| ℹ️ **About** | Tips, capabilities, speed/quality guide |\n",
"\n",
"**Controls in every tab:**\n",
"- **Timesteps slider** (4-30): Lower = faster, Higher = better quality\n",
"- **CFG Scale slider** (1.0-3.0): Lower = more natural, Higher = tighter control\n",
"\n",
"*The UI will generate a public `gradio.live` URL you can open anywhere — even your phone!*"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import gradio as gr\n",
"import soundfile as sf\n",
"import numpy as np\n",
"\n",
"print(\"\\n\" + \"=\"*60)\n",
"print(\"🚀 LAUNCHING VoxCPM2 INTERACTIVE WEB UI\")\n",
"print(\"=\"*60)\n",
"print(\"\\nTabs: 🔊 Basic | 🎨 Design | 👤 Clone | 🌐 Multilingual | 🌐👤 Multi+Clone | 🌐🎨 Multi+Design\")\n",
"print(\"Controls: Timesteps (4-30) | CFG Scale (1.0-3.0)\")\n",
"print(\"\\n⏳ Starting Gradio server...\\n\")\n",
"\n",
"def tts_basic(text, timesteps, cfg):\n",
" \"\"\"Basic text-to-speech generation.\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" wav = model.generate(\n",
" text=text.strip(),\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_basic.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" return output, f\"✅ Generated! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s\"\n",
"\n",
"def tts_design(text, description, timesteps, cfg):\n",
" \"\"\"Voice design via natural language description.\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" full_text = f\"({description.strip()}) {text.strip()}\"\n",
" wav = model.generate(\n",
" text=full_text,\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_design.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" return output, f\"✅ Generated with voice design! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s\"\n",
"\n",
"def tts_clone(text, audio_file, timesteps, cfg):\n",
" \"\"\"Zero-shot voice cloning from reference audio.\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" if audio_file is None:\n",
" return None, \"⚠️ Please upload a reference audio file (3-10s WAV or MP3).\"\n",
" wav = model.generate(\n",
" text=text.strip(),\n",
" reference_wav_path=audio_file,\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_clone.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" return output, f\"✅ Voice cloned! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s\"\n",
"\n",
"def tts_multilingual(text, timesteps, cfg):\n",
" \"\"\"Multilingual TTS — just type in any supported language.\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" wav = model.generate(\n",
" text=text.strip(),\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_multilingual.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" return output, f\"✅ Multilingual audio generated! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s\"\n",
"\n",
"def tts_multilingual_clone(text, audio_file, timesteps, cfg):\n",
" \"\"\"Clone a voice and speak in ANY language!\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" if audio_file is None:\n",
" return None, \"⚠️ Please upload a reference audio file (3-10s). The cloned voice will speak your text in any language you type!\"\n",
" wav = model.generate(\n",
" text=text.strip(),\n",
" reference_wav_path=audio_file,\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_multilingual_clone.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" lang_note = \"🌍 The cloned voice now speaks your text in whatever language you typed!\"\n",
" return output, f\"✅ Multilingual voice cloned! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s | {lang_note}\"\n",
"\n",
"def tts_multilingual_design(text, description, timesteps, cfg):\n",
" \"\"\"Design a voice and speak in ANY language!\"\"\"\n",
" if not text or not text.strip():\n",
" return None, \"Please enter some text.\"\n",
" full_text = f\"({description.strip()}) {text.strip()}\"\n",
" wav = model.generate(\n",
" text=full_text,\n",
" cfg_value=float(cfg),\n",
" inference_timesteps=int(timesteps),\n",
" )\n",
" output = \"/content/gradio_multilingual_design.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
" lang_note = \"🌍 The designed voice speaks your text in whatever language you typed!\"\n",
" return output, f\"✅ Multilingual voice designed! Duration: {len(wav)/model.tts_model.sample_rate:.1f}s | {lang_note}\"\n",
"\n",
"# ───────────────────────────────────────────────\n",
"# BUILD THE GRADIO INTERFACE\n",
"# ───────────────────────────────────────────────\n",
"with gr.Blocks() as demo:\n",
"\n",
" gr.Markdown(\"\"\"\n",
" <div align=\"center\">\n",
" <h1>🎙️ VoxCPM2 — Voice Synthesis & Cloning</h1>\n",
" <b>Tokenizer-Free TTS • Voice Design • Zero-Shot Voice Cloning • 30+ Languages • Multilingual Cloning</b><br>\n",
" <a href=\"https://huggingface.co/openbmb/VoxCPM2\" target=\"_blank\">🤗 Model</a> |\n",
" <a href=\"https://arxiv.org/abs/2509.24650\" target=\"_blank\">📄 Paper</a> |\n",
" <a href=\"https://voxcpm.readthedocs.io/\" target=\"_blank\">📖 Docs</a>\n",
" </div><br>\n",
" \"\"\")\n",
"\n",
" # ── Shared controls component ──\n",
" with gr.Row():\n",
" with gr.Column(scale=1):\n",
" shared_steps = gr.Slider(\n",
" minimum=4, maximum=30, value=10, step=1,\n",
" label=\"⏱️ Timesteps (quality vs speed)\",\n",
" info=\"Lower = faster draft | Higher = better quality\"\n",
" )\n",
" with gr.Column(scale=1):\n",
" shared_cfg = gr.Slider(\n",
" minimum=1.0, maximum=3.0, value=2.0, step=0.1,\n",
" label=\"🎚️ CFG Scale (control vs naturalness)\",\n",
" info=\"Lower = more natural | Higher = tighter control\"\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 1: Basic TTS\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"🔊 Basic TTS\"):\n",
" gr.Markdown(\"Type any text and generate studio-quality 48kHz speech instantly.\")\n",
" with gr.Row():\n",
" basic_text = gr.Textbox(\n",
" label=\"Text to Speak\",\n",
" value=\"Hello! This is VoxCPM2, a powerful tokenizer-free text-to-speech model.\",\n",
" lines=4,\n",
" placeholder=\"Enter text in any of 30+ languages...\"\n",
" )\n",
" with gr.Row():\n",
" basic_btn = gr.Button(\"🔊 Generate Speech\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" basic_audio = gr.Audio(label=\"Generated Audio\", type=\"filepath\", autoplay=False)\n",
" basic_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" basic_btn.click(\n",
" tts_basic,\n",
" [basic_text, shared_steps, shared_cfg],\n",
" [basic_audio, basic_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 2: Voice Design\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"🎨 Voice Design\"):\n",
" gr.Markdown(\"Describe any voice with natural language — the model will create it!\")\n",
" with gr.Row():\n",
" with gr.Column(scale=2):\n",
" design_text = gr.Textbox(\n",
" label=\"Text to Speak\",\n",
" value=\"Welcome to the presentation. Today we explore the frontiers of artificial intelligence.\",\n",
" lines=3,\n",
" )\n",
" with gr.Column(scale=1):\n",
" design_desc = gr.Textbox(\n",
" label=\"Voice Description\",\n",
" value=\"A warm, professional female narrator with a calm and reassuring tone\",\n",
" lines=3,\n",
" placeholder=\"e.g., A deep male voice, an enthusiastic child, a robot...\"\n",
" )\n",
" with gr.Row():\n",
" gr.Examples(\n",
" examples=[\n",
" [\"Hello! How can I help you today?\", \"A young woman, gentle and soothing voice\"],\n",
" [\"Welcome to the nightly news.\", \"A deep male narrator, professional and authoritative\"],\n",
" [\"Hey there! Ready to build something amazing?\", \"An energetic young man, enthusiastic and upbeat\"],\n",
" [\"System initialized. All parameters nominal.\", \"A robot, monotone synthetic voice\"],\n",
" [\"In a world where technology knows no bounds...\", \"A dramatic movie trailer voice, deep and intense\"],\n",
" ],\n",
" inputs=[design_text, design_desc],\n",
" label=\"🎨 Quick Voice Presets\"\n",
" )\n",
" with gr.Row():\n",
" design_btn = gr.Button(\"🎨 Design Voice\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" design_audio = gr.Audio(label=\"Designed Voice\", type=\"filepath\")\n",
" design_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" design_btn.click(\n",
" tts_design,\n",
" [design_text, design_desc, shared_steps, shared_cfg],\n",
" [design_audio, design_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 3: Voice Clone\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"👤 Voice Clone\"):\n",
" gr.Markdown(\"Upload a 3-10 second audio clip of any voice. The model will clone it!\")\n",
" with gr.Row():\n",
" with gr.Column(scale=2):\n",
" clone_text = gr.Textbox(\n",
" label=\"Text to Speak in Cloned Voice\",\n",
" value=\"Hello! This is my voice cloned by artificial intelligence. The quality is remarkably accurate.\",\n",
" lines=3,\n",
" )\n",
" with gr.Column(scale=1):\n",
" clone_audio_in = gr.Audio(\n",
" label=\"📤 Upload Reference Voice (3-10s)\",\n",
" type=\"filepath\"\n",
" )\n",
" with gr.Row():\n",
" clone_btn = gr.Button(\"👤 Clone Voice\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" clone_audio_out = gr.Audio(label=\"🔊 Cloned Output\", type=\"filepath\")\n",
" clone_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" clone_btn.click(\n",
" tts_clone,\n",
" [clone_text, clone_audio_in, shared_steps, shared_cfg],\n",
" [clone_audio_out, clone_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 4: Multilingual\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"🌐 Multilingual\"):\n",
" gr.Markdown(\"Type in ANY language — no language tags needed! Supports 30+ languages.\")\n",
" with gr.Row():\n",
" multi_text = gr.Textbox(\n",
" label=\"Text in Any Language\",\n",
" value=\"你好！这是VoxCPM2的多语言语音合成演示。Hello! This is multilingual too. ¡Hola! Esto es español.\",\n",
" lines=3,\n",
" placeholder=\"English, 中文, Español, Français, Deutsch, 日本語, 한국어, हिन्दी, and more...\"\n",
" )\n",
" with gr.Row():\n",
" gr.Examples(\n",
" examples=[\n",
" [\"The quick brown fox jumps over the lazy dog.\"],\n",
" [\"你好，这是中文语音合成演示。\"],\n",
" [\"¡Hola! VoxCPM2 es un modelo de síntesis de voz multilingüe.\"],\n",
" [\"Bonjour! VoxCPM2 est un modèle de synthèse vocale multilingue.\"],\n",
" [\"こんにちは、VoxCPM2は多言語対応の高品質な音声合成モデルです。\"],\n",
" [\"안녕하세요, VoxCPM2는 고품질 다국어 음성 합성 모델입니다.\"],\n",
" ],\n",
" inputs=[multi_text],\n",
" label=\"🌍 Language Examples\"\n",
" )\n",
" with gr.Row():\n",
" multi_btn = gr.Button(\"🌐 Generate Multilingual Speech\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" multi_audio = gr.Audio(label=\"Multilingual Output\", type=\"filepath\")\n",
" multi_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" multi_btn.click(\n",
" tts_multilingual,\n",
" [multi_text, shared_steps, shared_cfg],\n",
" [multi_audio, multi_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 5: Multilingual + Voice Clone ⭐ NEW\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"🌐👤 Multilingual + Clone\"):\n",
" gr.Markdown(\"\"\"\n",
" **Clone a voice and make it speak ANY language!** 🌍\n",
" \n",
" Upload a voice sample (e.g., an English speaker), then type text in Chinese, Spanish, French, Japanese, Hindi — anything!\n",
" The cloned voice will naturally speak your text in that language.\n",
" \"\"\")\n",
" with gr.Row():\n",
" with gr.Column(scale=2):\n",
" mc_text = gr.Textbox(\n",
" label=\"Text to Speak (Any Language!)\",\n",
" value=\"你好！这是我克隆的声音在说中文。¡Hola! Esta es mi voz clonada hablando español.\",\n",
" lines=3,\n",
" placeholder=\"Type in ANY language: English, 中文, Español, Français, 日本語, 한국어, हिन्दी...\"\n",
" )\n",
" with gr.Column(scale=1):\n",
" mc_audio_in = gr.Audio(\n",
" label=\"📤 Upload Voice to Clone (3-10s)\",\n",
" type=\"filepath\"\n",
" )\n",
" with gr.Row():\n",
" gr.Examples(\n",
" examples=[\n",
" [\"你好！这是克隆的声音在说中文。\", None],\n",
" [\"¡Hola! Esta es mi voz clonada hablando español.\", None],\n",
" [\"Bonjour! C'est ma voix clonée parlant français.\", None],\n",
" [\"こんにちは、これは克隆された声が日本語を話しています。\", None],\n",
" [\"नमस्ते! यह मेरी क्लोन की आवाज़ हिंदी में बोल रही है।\", None],\n",
" ],\n",
" inputs=[mc_text, mc_audio_in],\n",
" label=\"🌍 Multilingual Clone Examples (upload your own voice!)\"\n",
" )\n",
" with gr.Row():\n",
" mc_btn = gr.Button(\"🌐👤 Clone Voice + Speak Any Language\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" mc_audio_out = gr.Audio(label=\"🔊 Multilingual Cloned Output\", type=\"filepath\")\n",
" mc_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" mc_btn.click(\n",
" tts_multilingual_clone,\n",
" [mc_text, mc_audio_in, shared_steps, shared_cfg],\n",
" [mc_audio_out, mc_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 6: Multilingual + Voice Design ⭐ NEW\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"🌐🎨 Multilingual + Design\"):\n",
" gr.Markdown(\"\"\"\n",
" **Design a voice with words and make it speak ANY language!** 🎨🌍\n",
" \n",
" Describe the voice you want (e.g., \"warm elderly British woman\"), then type text in any language.\n",
" The designed voice will speak your text in that language.\n",
" \"\"\")\n",
" with gr.Row():\n",
" with gr.Column(scale=2):\n",
" md_text = gr.Textbox(\n",
" label=\"Text to Speak (Any Language!)\",\n",
" value=\"你好！这是用设计的温暖女声说的中文。Bonjour! C'est une voix française chaleureuse et élégante.\",\n",
" lines=3,\n",
" placeholder=\"Type in ANY language...\"\n",
" )\n",
" with gr.Column(scale=1):\n",
" md_desc = gr.Textbox(\n",
" label=\"Voice Description\",\n",
" value=\"A warm, gentle female voice, clear and articulate\",\n",
" lines=3,\n",
" placeholder=\"e.g., A deep British male, a cheerful child, a wise elderly woman...\"\n",
" )\n",
" with gr.Row():\n",
" gr.Examples(\n",
" examples=[\n",
" [\"你好！这是一个温暖的中文女声。\", \"A warm young Chinese woman, gentle and friendly\"],\n",
" [\"¡Hola! Esta es una voz española profesional y clara.\", \"A confident Spanish female narrator, professional\"],\n",
" [\"Bonjour! C'est une voix française élégante et douce.\", \"A sophisticated French woman, elegant and soft-spoken\"],\n",
" [\"こんにちは、これは日本語の丁寧な女性の声です。\", \"A polite Japanese woman, soft and respectful\"],\n",
" [\"नमस्ते! यह एक गर्मजोशी भारतीय महिला की आवाज़ है।\", \"A warm Indian woman, friendly and expressive\"],\n",
" ],\n",
" inputs=[md_text, md_desc],\n",
" label=\"🎨🌍 Multilingual Design Examples\"\n",
" )\n",
" with gr.Row():\n",
" md_btn = gr.Button(\"🌐🎨 Design Voice + Speak Any Language\", variant=\"primary\", size=\"lg\")\n",
" with gr.Row():\n",
" md_audio_out = gr.Audio(label=\"🔊 Multilingual Designed Output\", type=\"filepath\")\n",
" md_status = gr.Textbox(label=\"Status\", interactive=False)\n",
" md_btn.click(\n",
" tts_multilingual_design,\n",
" [md_text, md_desc, shared_steps, shared_cfg],\n",
" [md_audio_out, md_status]\n",
" )\n",
"\n",
" # ═══════════════════════════════════════════\n",
" # TAB 7: About / Tips\n",
" # ═══════════════════════════════════════════\n",
" with gr.Tab(\"ℹ️ About & Tips\"):\n",
" gr.Markdown(\"\"\"\n",
" ## ⚡ Speed vs Quality Guide\n",
"\n",
" | Timesteps | Quality | Speed | Best For |\n",
" |-----------|---------|-------|----------|\n",
" | **4-5** | Draft | ⚡ Fast | Quick testing |\n",
" | **8-10** | Good | 🚀 Normal | Default, balanced |\n",
" | **15-20** | High | 🐢 Slow | Voice cloning |\n",
" | **25-30** | Best | 🐌 Very Slow | Audiobooks, production |\n",
"\n",
" ## 🎚️ CFG Scale Guide\n",
" | CFG Value | Effect |\n",
" |-----------|--------|\n",
" | **1.0-1.5** | More natural, relaxed |\n",
" | **2.0** | Balanced (default) |\n",
" | **2.5-3.0** | Tighter control, more precise |\n",
"\n",
" ## 📝 Model Info\n",
" - **2B parameters** — tokenizer-free diffusion TTS\n",
" - **48kHz** studio-quality output\n",
" - **30+ languages** — no language tags needed\n",
" - **Zero-shot voice cloning** — 3-10s samples\n",
" - **Voice design** — describe any voice in words\n",
" - **Multilingual voice cloning** — clone voice & speak any language ⭐ NEW\n",
" - **Multilingual voice design** — design voice & speak any language ⭐ NEW\n",
" - **Apache-2.0 license** — free for commercial use\n",
"\n",
" ## 💡 Pro Tips\n",
" - **Multilingual + Clone**: Upload an English voice, type in Chinese → the English voice speaks Chinese!\n",
" - **Multilingual + Design**: Describe \"a warm French woman\" and type in Japanese → a French-accented voice speaks Japanese!\n",
" - For voice cloning, upload **clean 3-10s WAV/MP3** of a single speaker\n",
" - For voice design, be descriptive: *\"A young British woman, soft and caring\"*\n",
" - Lower timesteps for quick drafts, higher for final quality\n",
" - The model auto-detects language — just type in any language!\n",
" - **No language tags needed** — just type naturally in any language\n",
" \"\"\")\n",
"\n",
"# ── LAUNCH ──\n",
"demo.launch(share=True, debug=True, theme=gr.themes.Soft())\n",
"\n",
"print(\"\\n✅ Gradio UI is running!\")\n",
"print(\"\\n💡 Use the public URL above to access from any device.\")\n",
"print(\"💡 The URL is active as long as this Colab session runs.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 📥 Download All Generated Audio\n",
"After using the UI (or any optional mode below), download all your audio files as a ZIP."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import glob\n",
"import os\n",
"from google.colab import files\n",
"import zipfile\n",
"\n",
"wav_files = glob.glob(\"/content/*.wav\")\n",
"print(f\"Found {len(wav_files)} WAV files:\")\n",
"for f in wav_files:\n",
" size = os.path.getsize(f) / 1024**2\n",
" print(f\" - {os.path.basename(f)} ({size:.1f} MB)\")\n",
"\n",
"if wav_files:\n",
" zip_path = \"/content/voxcpm2_outputs.zip\"\n",
" with zipfile.ZipFile(zip_path, 'w') as zf:\n",
" for f in wav_files:\n",
" zf.write(f, os.path.basename(f))\n",
" print(f\"\\n📦 ZIP created: {zip_path}\")\n",
" files.download(zip_path)\n",
"else:\n",
" print(\"No WAV files found. Use the Gradio UI above to generate audio first.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"## 🧹 Cleanup (Free GPU Memory)\n",
"Run this when done to release GPU memory."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import gc\n",
"\n",
"if 'model' in globals():\n",
" del model\n",
"if 'demo' in globals():\n",
" del demo\n",
"\n",
"gc.collect()\n",
"\n",
"if torch.cuda.is_available():\n",
" torch.cuda.empty_cache()\n",
" torch.cuda.ipc_collect()\n",
" print(f\"🧹 GPU memory cleared.\")\n",
"\n",
"print(\"✅ Cleanup complete. Reload the model if you want to use it again.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"---\n",
"## 🔬 OPTIONAL: Individual Mode Experiments\n",
"*The cells below let you experiment with each VoxCPM2 capability directly in code.*\n",
"*These are completely optional — the Gradio UI above covers all of this already.*\n",
"\n",
"*Skip to the bottom if you only need the UI.*"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🔊 [OPTIONAL] Mode 1: Basic Text-to-Speech (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Basic TTS via code\n",
"import soundfile as sf\n",
"import IPython.display as ipd\n",
"\n",
"text = \"Hello! VoxCPM2 produces studio-quality 48 kilohertz audio.\"\n",
"print(f\"Text: {text}\")\n",
"\n",
"start = time.time()\n",
"wav = model.generate(text=text, cfg_value=2.0, inference_timesteps=10)\n",
"elapsed = time.time() - start\n",
"\n",
"output_path = \"/content/optional_basic_tts.wav\"\n",
"sf.write(output_path, wav, model.tts_model.sample_rate)\n",
"\n",
"duration = len(wav) / model.tts_model.sample_rate\n",
"print(f\"✅ Saved: {output_path}\")\n",
"print(f\"⏱️ Time: {elapsed:.1f}s | Duration: {duration:.1f}s\")\n",
"ipd.Audio(output_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎨 [OPTIONAL] Mode 2: Voice Design (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Voice design via code\n",
"description = \"A deep male narrator, professional and authoritative\"\n",
"text = \"Welcome to today's presentation. We will cover the fundamentals of machine learning.\"\n",
"full_text = f\"({description}) {text}\"\n",
"\n",
"print(f\"🎨 Voice Design: {description}\")\n",
"\n",
"start = time.time()\n",
"wav = model.generate(text=full_text, cfg_value=2.0, inference_timesteps=12)\n",
"elapsed = time.time() - start\n",
"\n",
"output_path = \"/content/optional_voice_design.wav\"\n",
"sf.write(output_path, wav, model.tts_model.sample_rate)\n",
"\n",
"duration = len(wav) / model.tts_model.sample_rate\n",
"print(f\"✅ Saved: {output_path}\")\n",
"print(f\"⏱️ Time: {elapsed:.1f}s | Duration: {duration:.1f}s\")\n",
"ipd.Audio(output_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 👤 [OPTIONAL] Mode 3: Voice Cloning (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Voice cloning via code — upload your own audio file first\n",
"from google.colab import files\n",
"\n",
"print(\"📤 Upload a short voice sample (3-10s WAV or MP3):\")\n",
"uploaded = files.upload()\n",
"\n",
"if uploaded:\n",
" ref_filename = list(uploaded.keys())[0]\n",
" ref_path = f\"/content/{ref_filename}\"\n",
" print(f\"\\n✅ Reference ready: {ref_path}\")\n",
"\n",
" clone_text = \"Hello! This is my cloned voice speaking through artificial intelligence.\"\n",
" print(f\"🎙️ Cloning: {clone_text}\")\n",
"\n",
" start = time.time()\n",
" wav = model.generate(\n",
" text=clone_text,\n",
" reference_wav_path=ref_path,\n",
" cfg_value=2.0,\n",
" inference_timesteps=15,\n",
" )\n",
" elapsed = time.time() - start\n",
"\n",
" output_path = \"/content/optional_voice_clone.wav\"\n",
" sf.write(output_path, wav, model.tts_model.sample_rate)\n",
"\n",
" duration = len(wav) / model.tts_model.sample_rate\n",
" print(f\"\\n✅ Cloned! Saved: {output_path}\")\n",
" print(f\"⏱️ Time: {elapsed:.1f}s | Duration: {duration:.1f}s\")\n",
" ipd.Audio(output_path)\n",
"else:\n",
" print(\"❌ No file uploaded.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🌍 [OPTIONAL] Mode 4: Multilingual Voice Clone (Code) ⭐ NEW"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Clone a voice and speak in ANY language!\n",
"from google.colab import files\n",
"\n",
"print(\"📤 Upload a voice sample to clone (3-10s):\")\n",
"uploaded = files.upload()\n",
"\n",
"if uploaded:\n",
" ref_filename = list(uploaded.keys())[0]\n",
" ref_path = f\"/content/{ref_filename}\"\n",
" print(f\"\\n✅ Reference ready: {ref_path}\")\n",
"\n",
" # Type in ANY language — the cloned voice will speak it!\n",
" multilingual_text = \"你好！这是我克隆的声音在说中文。非常自然和流畅！\"\n",
" print(f\"🌍 Multilingual Clone: {multilingual_text}\")\n",
"\n",
" start = time.time()\n",
" wav = model.generate(\n",
" text=multilingual_text,\n",
" reference_wav_path=ref_path,\n",
" cfg_value=2.0,\n",
" inference_timesteps=15,\n",
" )\n",
" elapsed = time.time() - start\n",
"\n",
" output_path = \"/content/optional_multilingual_clone.wav\"\n",
" sf.write(output_path, wav, model.tts_model.sample_rate)\n",
"\n",
" duration = len(wav) / model.tts_model.sample_rate\n",
" print(f\"\\n✅ Multilingual clone saved: {output_path}\")\n",
" print(f\"⏱️ Time: {elapsed:.1f}s | Duration: {duration:.1f}s\")\n",
" print(\"🎉 The cloned voice now speaks Chinese! Try Spanish, French, Japanese, Hindi, etc.\")\n",
" ipd.Audio(output_path)\n",
"else:\n",
" print(\"❌ No file uploaded.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 🎵 [OPTIONAL] Mode 5: Hi-Fi Ultimate Cloning (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Hi-Fi cloning — needs prompt audio + prompt text + reference audio\n",
"print(\"📤 Upload PROMPT audio (with known text) first:\")\n",
"up1 = files.upload()\n",
"if up1:\n",
" prompt_file = list(up1.keys())[0]\n",
" !cp /content/{prompt_file} /content/prompt.wav\n",
" print(\"\\n📤 Upload REFERENCE audio (clean voice) now:\")\n",
" up2 = files.upload()\n",
" if up2:\n",
" ref_file = list(up2.keys())[0]\n",
" !cp /content/{ref_file} /content/reference.wav\n",
" prompt_text = input(\"\\nEnter EXACT transcript of prompt audio: \")\n",
" hifi_text = \"This is the ultimate high fidelity cloned voice.\"\n",
"\n",
" print(\"\\n🎵 Hi-Fi Cloning...\")\n",
" start = time.time()\n",
" wav = model.generate(\n",
" text=hifi_text,\n",
" prompt_wav_path=\"/content/prompt.wav\",\n",
" prompt_text=prompt_text,\n",
" reference_wav_path=\"/content/reference.wav\",\n",
" cfg_value=2.5,\n",
" inference_timesteps=20,\n",
" )\n",
" elapsed = time.time() - start\n",
"\n",
" output_path = \"/content/optional_hifi_clone.wav\"\n",
" sf.write(output_path, wav, model.tts_model.sample_rate)\n",
" print(f\"\\n✅ Hi-Fi clone saved: {output_path}\")\n",
" print(f\"⏱️ Time: {elapsed:.1f}s\")\n",
" ipd.Audio(output_path)\n",
" else:\n",
" print(\"❌ No reference file.\")\n",
"else:\n",
" print(\"❌ No prompt file.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 📡 [OPTIONAL] Mode 6: Streaming Generation (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Streaming generation for long texts\n",
"import numpy as np\n",
"\n",
"long_text = \"This is a demonstration of streaming text-to-speech generation. \" * 5\n",
"long_text += \"The model processes text in chunks, useful for long-form content.\"\n",
"\n",
"print(f\"📝 Text length: {len(long_text)} chars\")\n",
"print(\"📡 Streaming...\\n\")\n",
"\n",
"all_chunks = []\n",
"chunk_num = 0\n",
"start = time.time()\n",
"\n",
"for chunk in model.generate_streaming(text=long_text, cfg_value=2.0, inference_timesteps=10):\n",
" chunk_num += 1\n",
" all_chunks.append(chunk)\n",
" if chunk_num <= 3:\n",
" print(f\"📦 Chunk #{chunk_num}: {len(chunk)} samples\")\n",
" elif chunk_num == 4:\n",
" print(\"... (more chunks) ...\")\n",
"\n",
"elapsed = time.time() - start\n",
"full_audio = np.concatenate(all_chunks)\n",
"output_path = \"/content/optional_streaming.wav\"\n",
"sf.write(output_path, full_audio, model.tts_model.sample_rate)\n",
"\n",
"duration = len(full_audio) / model.tts_model.sample_rate\n",
"print(f\"\\n✅ Streaming complete! {chunk_num} chunks | {duration:.1f}s audio\")\n",
"print(f\"⏱️ Total time: {elapsed:.1f}s\")\n",
"ipd.Audio(output_path)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ⚡ [OPTIONAL] Mode 7: Speed vs Quality Comparison (Code)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# OPTIONAL: Compare different timesteps (quality vs speed tradeoff)\n",
"test_text = \"Quality versus speed tradeoff demonstration.\"\n",
"timesteps_to_test = [5, 10, 20]\n",
"\n",
"for steps in timesteps_to_test:\n",
" print(f\"\\n⚙️ Testing timesteps={steps}\")\n",
" start = time.time()\n",
" wav = model.generate(text=test_text, cfg_value=2.0, inference_timesteps=steps)\n",
" elapsed = time.time() - start\n",
"\n",
" output = f\"/content/optional_quality_{steps}steps.wav\"\n",
" sf.write(output, wav, model.tts_model.sample_rate)\n",
"\n",
" print(f\"✅ Saved: {output}\")\n",
" print(f\"⏱️ Time: {elapsed:.1f}s\")\n",
" display(ipd.Audio(output))\n",
" print(\"-\"*50)"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python",
"version": "3.10.0"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
|