runtime/kokoclone-python/kokoclone-source/README.md

🎙️ KokoClone

KokoClone is a fast, real-time compatible multilingual voice cloning system built on top of Kokoro-ONNX, one of the fastest open-source neural TTS engines available today.

It allows you to:

• Text → Clone: Type text in multiple languages, provide a short reference audio clip, and instantly generate speech in that same voice.
• Audio → Clone: Re-voice an existing audio recording to sound like any reference speaker — no transcription needed.

Features

Multilingual Speech Generation

Generate native speech in English (en), Hindi (hi), French (fr), Japanese (ja), Chinese (zh), Italian (it), Portuguese (pt), and Spanish (es).

Zero-Shot Voice Cloning

Upload a 3–10 second voice sample and KokoClone instantly transfers its vocal characteristics to the generated speech.

Audio-to-Audio Voice Conversion

Upload any existing speech recording and re-voice it to sound like a reference speaker. The pipeline skips TTS entirely and runs purely through the Kanade voice-conversion model. Works on recordings of any length thanks to automatic VRAM-aware chunking!

Automatic Model Handling

On the first run, the required model weights (.onnx and .bin files) are automatically downloaded from Hugging Face and placed in the correct directories.

Real-Time Friendly

Built on Kokoro's efficient ONNX runtime pipeline, KokoClone detects your hardware and runs smoothly on both standard laptops (CPU) and workstations (GPU).

Live Demo

Try it instantly without installing anything:
👉 KokoClone on Hugging Face Spaces

Installation

You can set up KokoClone using either Conda (Recommended) or uv.

1. Clone the Repository

git clone https://github.com/Ashish-Patnaik/kokoclone.git
cd kokoclone

2. Set Up the Environment & Install Dependencies

Option A: Using Conda (Recommended)

conda create -n kokoclone python=3.12.12 -y
conda activate kokoclone

For CPU Users (Mac / Standard Laptops):

pip install torch torchaudio --index-url [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu)
pip install -r requirements.txt

For GPU Users (Nvidia GPUs):

pip install -r requirements.txt
pip install kokoro-onnx[gpu]

Option B: Using uv

If you prefer uv for fast package management:

# For CPU Users
uv sync

# For GPU Users (Nvidia)
uv sync --extra gpu

# Activate the environment
source .venv/bin/activate  # Linux/macOS
.venv\Scripts\activate     # Windows

Usage

KokoClone is highly flexible and can be used via Web UI, CLI, or Python API.

1. Web Interface (Gradio)

Launch the interactive web app:

python app.py

• Tab 1 (Text → Clone): Enter text, pick a language, upload a reference voice, and generate.
• Tab 2 (Audio → Clone): Upload source audio and a reference voice, and get back re-voiced audio.

2. Command Line Interface (CLI)

Generate speech directly from your terminal.

Text to cloned speech (default mode):

python cli.py --text "Hello from KokoClone" --lang en --ref reference.wav --out output.wav

Audio to re-voiced speech:

python cli.py --mode convert --source original_speech.wav --ref target_voice.wav --out revoiced.wav

Argument	Default	Description
--mode	tts	tts (text → speech) or convert (audio → re-voiced audio)
--text	—	Text to synthesize (required for tts mode)
--lang	en	Language code: en hi fr ja zh it es pt
--source	—	Path to source audio (required for convert mode)
--ref	—	Path to reference voice audio (always required)
--out	output.wav	Output file path

3. Python API

Integrate KokoClone into your own Python applications.

Text to Cloned Speech:

from core.cloner import KokoClone

cloner = KokoClone()
cloner.generate(
    text="This voice is cloned using KokoClone.",
    lang="en",
    reference_audio="reference.wav",
    output_path="output.wav"
)

Audio-to-Audio Voice Conversion:

import soundfile as sf
from kanade_tokenizer import load_audio
from core.cloner import KokoClone
from core.chunked_convert import chunked_voice_conversion

cloner = KokoClone()

# Load audio tensors
source_wav = load_audio("source_speech.wav", sample_rate=cloner.sample_rate).to(cloner.device)
ref_wav = load_audio("target_voice.wav", sample_rate=cloner.sample_rate).to(cloner.device)

# Convert using VRAM-aware chunking
converted = chunked_voice_conversion(
    kanade=cloner.kanade,
    vocoder_model=cloner.vocoder,
    source_wav=source_wav,
    ref_wav=ref_wav,
    sample_rate=cloner.sample_rate,
)

sf.write("revoiced_output.wav", converted.numpy(), cloner.sample_rate)

Memory Management for Long Audio

The chunked_voice_conversion function in core/chunked_convert.py handles memory automatically when converting long audio recordings:

• VRAM Budget: On CUDA, chunks are sized so each forward pass uses at most 50% of total GPU memory (configurable via the vram_fraction parameter).
• RoPE Ceiling: The Kanade mel_decoder Transformer has positional embeddings precomputed for 1,024 mel frames. Chunk windows are hard-capped below this limit (≈ 8.9s of source audio per chunk) with a 10% safety margin to prevent recomputation and quality degradation.
• Overlap Smoothing: Each chunk includes a 0.5s overlap on both sides to suppress boundary artifacts.
• Single-Pass Vocoding: The full reassembled mel spectrogram is passed to the vocoder in one shot for clean waveform reconstruction.

Project Structure

app.py                → Gradio Web Interface (two-tab UI)
cli.py                → Command-line tool (tts and convert modes)
inference.py          → Example API usage script
core/
 ├── cloner.py        → Core TTS + voice cloning engine
 └── chunked_convert.py → VRAM-aware chunked audio conversion
model/                → Downloaded Kokoro model weights (Auto-populates)
voice/                → Downloaded Kokoro voice bins (Auto-populates)

Star History

Acknowledgments

This project builds upon the incredible open-source work of:

• Kokoro-ONNX — for fast and efficient neural speech synthesis.
• Kanade Tokenizer — for the brilliant zero-shot voice conversion architecture.

License

Licensed under the Apache 2.0 License.