Faithful-cloning defaults + sliders + optional background-audio removal (demucs-onnx)

README.md

@@ -19,14 +19,23 @@ which beats ElevenLabs in independent blind preference tests.
 
 ## How to use (manual A/B)
 1. Upload a **reference audio** clip of the voice to clone (5–20 s of clean speech is ideal).
-2. Pick the **language** (default: English).
-3. Type the **text** to speak (long scripts are auto-chunked at sentence boundaries).
-4. Click **Clone & Speak** → you get audio in the cloned voice.
+2. (Optional) Tick **🧹 Remove background audio from reference** to isolate the voice
+   (HT-Demucs) before cloning if the clip has music/noise. Use **Preview cleaned reference**
+   to hear the isolated result first.
+3. Pick the **language** (default: English).
+4. Type the **text** to speak (long scripts are auto-chunked at sentence boundaries).
+5. Click **Clone & Speak** → you get audio in the cloned voice.
 
 Tip: leave the reference empty to hear a built-in sample voice for the selected language.
 
+### Cloning defaults (tuned for faithful cloning)
+Tuned for **speaker similarity**, not expressiveness:
+`exaggeration=0.4` (neutral), `cfg_weight=0.5` (balanced; ~0.3 faster pace, 0.0 cross-lingual),
+`temperature=0.7` (consistent). All knobs are exposed as sliders.
+
 ## API (for bot integration later)
-Gradio exposes a programmatic endpoint named **`clone`**:
+Gradio exposes a programmatic endpoint named **`clone`** (plus **`isolate_voice`** for
+standalone background-audio removal):
 
 ```python
 from gradio_client import Client, handle_file
@@ -36,13 +45,20 @@ sr_path = client.predict(
     text="Hey, it's good to finally hear your voice.",
     language_id="en",
     audio_prompt_path=handle_file("reference.wav"),
-    exaggeration=0.5,
+    exaggeration=0.4,
     cfg_weight=0.5,
-    temperature=0.8,
+    temperature=0.7,
     seed=0,
+    clean_reference=False,   # True = strip background music/noise first
+    repetition_penalty=2.0,
+    min_p=0.05,
+    top_p=1.0,
     api_name="/clone",
 )
 print(sr_path)  # path to generated wav
+
+# Just clean a reference clip (returns isolated-voice wav):
+cleaned = client.predict(handle_file("noisy_reference.wav"), api_name="/isolate_voice")
 ```
 
 ## Notes

app.py

@@ -9,10 +9,13 @@ Mirrors the official ResembleAI/Chatterbox-Multilingual-TTS inference path, with
   - long-text sentence chunking (so JOI-length scripts work, not just 300 chars),
   - a clean programmatic endpoint (api_name="/clone") for later bot integration.
 """
+import os
 import random
 import re
+import tempfile
 
 import numpy as np
+import soundfile as sf
 import torch
 import gradio as gr
 import spaces
@@ -24,6 +27,22 @@ print(f"Running on device: {DEVICE}")
 
 MODEL = None
 
+# ── Faithful-cloning defaults ────────────────────────────────────────────────
+# Tuned for SPEAKER SIMILARITY (clean identity match), not expressiveness.
+# Rationale (Resemble AI Chatterbox guidance + community cloning presets):
+#   - exaggeration LOW (~0.4): keeps delivery neutral/professional so the model
+#     reproduces the reference identity instead of "acting" it.
+#   - cfg_weight 0.5: balanced default; lower (~0.3) speeds pacing, 0.0 helps
+#     cross-lingual transfer avoid inheriting the reference-language accent.
+#   - temperature 0.7: slightly below the 0.8 default for steadier, more
+#     consistent output across chunked long scripts (less random drift).
+DEFAULT_EXAGGERATION = 0.4
+DEFAULT_CFG_WEIGHT = 0.5
+DEFAULT_TEMPERATURE = 0.7
+DEFAULT_REPETITION_PENALTY = 2.0
+DEFAULT_MIN_P = 0.05
+DEFAULT_TOP_P = 1.0
+
 # Built-in sample reference voices per language (used when no reference is uploaded).
 LANGUAGE_CONFIG = {
     "en": {
@@ -76,6 +95,69 @@ def set_seed(seed: int):
     np.random.seed(seed)
 
 
+# ── Audio cleanup (background-audio removal) ─────────────────────────────────
+# Optional preprocessing: isolate the spoken voice from a noisy/musical
+# reference clip BEFORE cloning, so the speaker conditionals are built from
+# clean speech. Uses HT-Demucs (htdemucs_ft vocals stem, #1 open-source vocal
+# SDR) via the pure-numpy + onnxruntime `demucs-onnx` package — no torch/
+# torchaudio dependency, so it can't disturb the pinned Chatterbox stack.
+# Runs on CPU so it does NOT consume the ZeroGPU budget. Designed as the first
+# member of a future "audio cleanup" feature group (denoise, trim, normalize…).
+_SEPARATOR_READY = None
+
+
+def _ensure_separator():
+    """Lazy-import demucs-onnx. Returns the callable or None if unavailable."""
+    global _SEPARATOR_READY
+    if _SEPARATOR_READY is None:
+        try:
+            from demucs_onnx import separate_stem  # noqa: PLC0415
+            _SEPARATOR_READY = separate_stem
+        except Exception as e:  # noqa: BLE001
+            print(f"WARNING: demucs-onnx unavailable, voice isolation disabled: {e}")
+            _SEPARATOR_READY = False
+    return _SEPARATOR_READY or None
+
+
+def isolate_voice(audio_path: str) -> str:
+    """Return a path to a cleaned WAV with background music/noise removed.
+
+    Falls back to the original clip (and warns) if separation is unavailable
+    or fails, so cloning never hard-breaks on a cleanup error.
+    """
+    if not audio_path:
+        return audio_path
+    separate_stem = _ensure_separator()
+    if separate_stem is None:
+        raise gr.Error("Voice isolation is unavailable (demucs-onnx not installed).")
+
+    try:
+        sr = sf.info(audio_path).samplerate
+    except Exception:  # noqa: BLE001
+        sr = 44100
+
+    # htdemucs_ft vocals specialist (CPU keeps this off the ZeroGPU budget).
+    vocals = separate_stem(audio_path, "vocals", providers="cpu")  # (channels, samples)
+    vocals = np.asarray(vocals, dtype=np.float32)
+    if vocals.ndim == 2:
+        vocals = vocals.mean(axis=0)  # downmix to mono for the speaker encoder
+    peak = float(np.max(np.abs(vocals))) if vocals.size else 0.0
+    if peak > 1.0:
+        vocals = vocals / peak
+
+    out_path = os.path.join(tempfile.gettempdir(), f"isolated_{random.randint(0, 1_000_000)}.wav")
+    sf.write(out_path, vocals, sr)
+    print(f"Isolated voice -> {out_path} ({len(vocals)/sr:.1f}s @ {sr}Hz)")
+    return out_path
+
+
+def isolate_voice_ui(audio_path: str):
+    """UI/endpoint wrapper: preview the cleaned reference (api_name=/isolate_voice)."""
+    if not audio_path:
+        raise gr.Error("Upload a reference clip first.")
+    return isolate_voice(audio_path)
+
+
 def default_audio_for_ui(lang: str):
     return LANGUAGE_CONFIG.get(lang, {}).get("audio")
 
@@ -111,15 +193,30 @@ def split_into_chunks(text: str, max_chars: int = CHUNK_CHARS):
     return [c for c in chunks if c]
 
 
+def _maybe_clean_reference(ref: str, clean_reference: bool) -> str:
+    """Optionally strip background music/noise from a user-supplied reference."""
+    if not (clean_reference and ref):
+        return ref
+    try:
+        return isolate_voice(ref)
+    except Exception as e:  # noqa: BLE001
+        gr.Warning(f"Background-audio removal failed, using raw reference: {e}")
+        return ref
+
+
 @spaces.GPU(duration=120)
 def clone_and_speak(
     text: str,
     language_id: str = "en",
     audio_prompt_path: str = None,
-    exaggeration: float = 0.5,
-    cfg_weight: float = 0.5,
-    temperature: float = 0.8,
+    exaggeration: float = DEFAULT_EXAGGERATION,
+    cfg_weight: float = DEFAULT_CFG_WEIGHT,
+    temperature: float = DEFAULT_TEMPERATURE,
     seed: int = 0,
+    clean_reference: bool = False,
+    repetition_penalty: float = DEFAULT_REPETITION_PENALTY,
+    min_p: float = DEFAULT_MIN_P,
+    top_p: float = DEFAULT_TOP_P,
 ):
     """
     Clone the voice in `audio_prompt_path` and speak `text` in language `language_id`.
@@ -129,10 +226,16 @@ def clone_and_speak(
         language_id: language code (en, fr, de, es, it, pt, hi, ja, zh, ...).
         audio_prompt_path: path/URL to a reference voice clip. If omitted, a
             built-in sample voice for the language is used.
-        exaggeration: expressiveness (0.25-2.0, neutral 0.5).
-        cfg_weight: CFG / pacing (0.2-1.0; set ~0.3 for cross-lingual transfer).
-        temperature: sampling randomness (0.05-5.0).
+        exaggeration: expressiveness (0.25-2.0; ~0.4 = neutral/faithful clone).
+        cfg_weight: CFG / pacing (0.0-1.0; lower ~0.3 = faster pace, 0.0 for
+            cross-lingual transfer; 0.5 = balanced default).
+        temperature: sampling randomness (0.05-2.0; lower = more consistent).
         seed: 0 for random, otherwise reproducible.
+        clean_reference: if True, isolate the voice (remove background music/
+            noise) from the uploaded reference before cloning.
+        repetition_penalty: discourages repeated tokens (model default 2.0).
+        min_p: min-p nucleus floor (model default 0.05).
+        top_p: top-p nucleus threshold (model default 1.0).
 
     Returns:
         (sample_rate, waveform) tuple consumable by gr.Audio.
@@ -151,9 +254,14 @@ def clone_and_speak(
     if not ref:
         raise gr.Error("Upload a reference audio clip to clone (or pick a language with a built-in sample).")
 
+    # Optional preprocessing: clean the reference so conditionals are built from
+    # isolated speech (only applies to a user-uploaded clip, not built-in samples).
+    if audio_prompt_path:
+        ref = _maybe_clean_reference(ref, clean_reference)
+
     lang = (language_id or "en").lower()
     chunks = split_into_chunks(text)
-    print(f"Cloning voice | lang={lang} | chunks={len(chunks)} | ref={ref}")
+    print(f"Cloning voice | lang={lang} | chunks={len(chunks)} | clean_ref={clean_reference} | ref={ref}")
 
     # Prepare speaker conditionals ONCE from the reference, then reuse across chunks
     # so the cloned identity stays consistent for the whole script.
@@ -170,6 +278,9 @@ def clone_and_speak(
             exaggeration=exaggeration,
             cfg_weight=cfg_weight,
             temperature=temperature,
+            repetition_penalty=repetition_penalty,
+            min_p=min_p,
+            top_p=top_p,
         )
         arr = wav.squeeze(0).detach().cpu().numpy().astype(np.float32)
         pieces.append(arr)
@@ -201,6 +312,13 @@ with gr.Blocks(title="Voice Clone Bench") as demo:
                 label="① Reference voice to clone (5–20s clean speech). Empty = built-in sample.",
                 value=default_audio_for_ui("en"),
             )
+            clean_reference = gr.Checkbox(
+                value=False,
+                label="🧹 Remove background audio from reference (isolate voice before cloning)",
+                info="Strips music/noise with HT-Demucs so the clone is built from clean speech.",
+            )
+            preview_btn = gr.Button("🧹 Preview cleaned reference", size="sm")
+            cleaned_preview = gr.Audio(label="Isolated voice (preview)", visible=True)
             language_id = gr.Dropdown(
                 choices=list(ChatterboxMultilingualTTS.get_supported_languages().keys()),
                 value="en",
@@ -212,20 +330,46 @@ with gr.Blocks(title="Voice Clone Bench") as demo:
                 lines=5,
                 max_lines=20,
             )
-            with gr.Accordion("Advanced", open=False):
-                exaggeration = gr.Slider(0.25, 2.0, step=0.05, value=0.5, label="Exaggeration (neutral 0.5)")
-                cfg_weight = gr.Slider(0.2, 1.0, step=0.05, value=0.5, label="CFG / Pace (≈0.3 for cross-lingual)")
-                temperature = gr.Slider(0.05, 2.0, step=0.05, value=0.8, label="Temperature")
+            with gr.Accordion("Cloning controls (tuned for faithful voice cloning)", open=True):
+                exaggeration = gr.Slider(
+                    0.0, 2.0, step=0.05, value=DEFAULT_EXAGGERATION,
+                    label="Exaggeration — lower = more neutral/faithful (≈0.4); 0.7+ = expressive",
+                )
+                cfg_weight = gr.Slider(
+                    0.0, 1.0, step=0.05, value=DEFAULT_CFG_WEIGHT,
+                    label="CFG / Pace — 0.5 balanced; ~0.3 faster; 0.0 for cross-lingual",
+                )
+                temperature = gr.Slider(
+                    0.05, 2.0, step=0.05, value=DEFAULT_TEMPERATURE,
+                    label="Temperature �� lower = more consistent/faithful (≈0.7)",
+                )
                 seed = gr.Number(value=0, label="Seed (0 = random)")
+                with gr.Accordion("Sampling (advanced)", open=False):
+                    repetition_penalty = gr.Slider(
+                        1.0, 2.5, step=0.05, value=DEFAULT_REPETITION_PENALTY,
+                        label="Repetition penalty (default 2.0)",
+                    )
+                    min_p = gr.Slider(0.0, 0.5, step=0.01, value=DEFAULT_MIN_P, label="min_p (default 0.05)")
+                    top_p = gr.Slider(0.1, 1.0, step=0.05, value=DEFAULT_TOP_P, label="top_p (default 1.0)")
             run_btn = gr.Button("Clone & Speak", variant="primary")
         with gr.Column():
             audio_output = gr.Audio(label="Cloned speech output")
 
     language_id.change(fn=on_language_change, inputs=[language_id], outputs=[ref_wav, text], show_progress=False)
 
+    preview_btn.click(
+        fn=isolate_voice_ui,
+        inputs=[ref_wav],
+        outputs=[cleaned_preview],
+        api_name="isolate_voice",
+    )
+
     run_btn.click(
         fn=clone_and_speak,
-        inputs=[text, language_id, ref_wav, exaggeration, cfg_weight, temperature, seed],
+        inputs=[
+            text, language_id, ref_wav, exaggeration, cfg_weight, temperature, seed,
+            clean_reference, repetition_penalty, min_p, top_p,
+        ],
         outputs=[audio_output],
         api_name="clone",
     )

requirements.txt

@@ -14,6 +14,11 @@ silero-vad==5.1.2
 conformer==0.3.2
 safetensors
 
+# Audio cleanup: HT-Demucs (htdemucs_ft vocals stem) via pure numpy + onnxruntime.
+# No torch/torchaudio at inference, so it can't disturb the pinned Chatterbox stack.
+# Used for optional background-audio removal from reference clips before cloning.
+demucs-onnx==0.3.4
+
 # Optional language-specific normalizers (disabled for build reliability — English-first prototype).
 # Re-enable only if you need advanced zh / ja / ru text normalization:
 # spacy_pkuseg          # Chinese text segmentation