runtime/kokoclone-python/kokoclone-source/README.md

<p align="center">
  <img width="1050" height="450" alt="KokoClone Banner" src="https://github.com/user-attachments/assets/26fbb00c-220e-435a-8f54-431781449c76" />
</p>

<h1 align="center">🎙️ KokoClone</h1>

<p align="center">
  <a href="https://huggingface.co/spaces/PatnaikAshish/kokoclone">
    <img src="https://img.shields.io/badge/🤗%20Hugging%20Face-Live%20Demo-blue" alt="Hugging Face Space" />

  </a>

  <a href="https://huggingface.co/PatnaikAshish/kokoclone">

    <img src="https://img.shields.io/badge/🤗%20Models-Repository-orange" alt="Hugging Face Models" />

  </a>

  <img src="https://img.shields.io/badge/Python-3.10%20to%203.12-3776AB.svg?logo=python&logoColor=white" alt="Python" />

  <a href="https://opensource.org/licenses/Apache-2.0">

    <img src="https://img.shields.io/badge/License-Apache_2.0-green.svg" alt="License" />

  </a>

</p>


**KokoClone** is a fast, real-time compatible multilingual voice cloning system built on top of **Kokoro-ONNX**, one of the fastest open-source neural TTS engines available today.

It allows you to:
* **Text → Clone:** Type text in multiple languages, provide a short reference audio clip, and instantly generate speech in that same voice.
* **Audio → Clone:** Re-voice an existing audio recording to sound like any reference speaker — *no transcription needed*.


## Features

### Multilingual Speech Generation
Generate native speech in English (`en`), Hindi (`hi`), French (`fr`), Japanese (`ja`), Chinese (`zh`), Italian (`it`), Portuguese (`pt`), and Spanish (`es`).

### Zero-Shot Voice Cloning
Upload a 3–10 second voice sample and KokoClone instantly transfers its vocal characteristics to the generated speech.

### Audio-to-Audio Voice Conversion
Upload any existing speech recording and re-voice it to sound like a reference speaker. The pipeline skips TTS entirely and runs purely through the Kanade voice-conversion model. Works on recordings of any length thanks to automatic VRAM-aware chunking!

### Automatic Model Handling
On the first run, the required model weights (`.onnx` and `.bin` files) are automatically downloaded from Hugging Face and placed in the correct directories.

### Real-Time Friendly
Built on Kokoro's efficient ONNX runtime pipeline, KokoClone detects your hardware and runs smoothly on both standard laptops (CPU) and workstations (GPU).


## Live Demo
Try it instantly without installing anything:  
👉 **[KokoClone on Hugging Face Spaces](https://huggingface.co/spaces/PatnaikAshish/kokoclone)**


## Installation

You can set up KokoClone using either **Conda** (Recommended) or **uv**.

### 1. Clone the Repository
```bash

git clone https://github.com/Ashish-Patnaik/kokoclone.git

cd kokoclone



```

### 2. Set Up the Environment & Install Dependencies

#### Option A: Using Conda (Recommended)

```bash

conda create -n kokoclone python=3.12.12 -y

conda activate kokoclone



```

**For CPU Users (Mac / Standard Laptops):**

```bash

pip install torch torchaudio --index-url [https://download.pytorch.org/whl/cpu](https://download.pytorch.org/whl/cpu)

pip install -r requirements.txt



```

**For GPU Users (Nvidia GPUs):**

```bash

pip install -r requirements.txt

pip install kokoro-onnx[gpu]



```

#### Option B: Using `uv`

If you prefer [uv](https://docs.astral.sh/uv/) for fast package management:

```bash

# For CPU Users

uv sync



# For GPU Users (Nvidia)

uv sync --extra gpu



# Activate the environment

source .venv/bin/activate  # Linux/macOS

.venv\Scripts\activate     # Windows



```



##  Usage

KokoClone is highly flexible and can be used via Web UI, CLI, or Python API.

### 1. Web Interface (Gradio)

Launch the interactive web app:

```bash

python app.py



```

* **Tab 1 (Text → Clone):** Enter text, pick a language, upload a reference voice, and generate.
* **Tab 2 (Audio → Clone):** Upload source audio and a reference voice, and get back re-voiced audio.

### 2. Command Line Interface (CLI)

Generate speech directly from your terminal.

**Text to cloned speech (default mode):**

```bash

python cli.py --text "Hello from KokoClone" --lang en --ref reference.wav --out output.wav



```

**Audio to re-voiced speech:**

```bash

python cli.py --mode convert --source original_speech.wav --ref target_voice.wav --out revoiced.wav



```

| Argument | Default | Description |
| --- | --- | --- |
| `--mode` | `tts` | `tts` (text → speech) or `convert` (audio → re-voiced audio) |
| `--text` | — | Text to synthesize *(required for `tts` mode)* |
| `--lang` | `en` | Language code: `en hi fr ja zh it es pt` |
| `--source` | — | Path to source audio *(required for `convert` mode)* |
| `--ref` | — | Path to reference voice audio *(always required)* |
| `--out` | `output.wav` | Output file path |

### 3. Python API

Integrate KokoClone into your own Python applications.

**Text to Cloned Speech:**

```python

from core.cloner import KokoClone



cloner = KokoClone()

cloner.generate(

    text="This voice is cloned using KokoClone.",

    lang="en",

    reference_audio="reference.wav",

    output_path="output.wav"

)



```

**Audio-to-Audio Voice Conversion:**

```python

import soundfile as sf

from kanade_tokenizer import load_audio

from core.cloner import KokoClone

from core.chunked_convert import chunked_voice_conversion



cloner = KokoClone()



# Load audio tensors

source_wav = load_audio("source_speech.wav", sample_rate=cloner.sample_rate).to(cloner.device)

ref_wav = load_audio("target_voice.wav", sample_rate=cloner.sample_rate).to(cloner.device)



# Convert using VRAM-aware chunking

converted = chunked_voice_conversion(

    kanade=cloner.kanade,

    vocoder_model=cloner.vocoder,

    source_wav=source_wav,

    ref_wav=ref_wav,

    sample_rate=cloner.sample_rate,

)



sf.write("revoiced_output.wav", converted.numpy(), cloner.sample_rate)



```


## Memory Management for Long Audio

The `chunked_voice_conversion` function in `core/chunked_convert.py` handles memory automatically when converting long audio recordings:

* **VRAM Budget:** On CUDA, chunks are sized so each forward pass uses at most 50% of total GPU memory (configurable via the `vram_fraction` parameter).
* **RoPE Ceiling:** The Kanade `mel_decoder` Transformer has positional embeddings precomputed for 1,024 mel frames. Chunk windows are hard-capped below this limit (≈ 8.9s of source audio per chunk) with a 10% safety margin to prevent recomputation and quality degradation.
* **Overlap Smoothing:** Each chunk includes a 0.5s overlap on both sides to suppress boundary artifacts.
* **Single-Pass Vocoding:** The full reassembled mel spectrogram is passed to the vocoder in one shot for clean waveform reconstruction.


## Project Structure

```text

app.py                → Gradio Web Interface (two-tab UI)

cli.py                → Command-line tool (tts and convert modes)

inference.py          → Example API usage script

core/

 ├── cloner.py        → Core TTS + voice cloning engine

 └── chunked_convert.py → VRAM-aware chunked audio conversion

model/                → Downloaded Kokoro model weights (Auto-populates)

voice/                → Downloaded Kokoro voice bins (Auto-populates)



```
## Star History

<a href="https://www.star-history.com/#Ashish-Patnaik/kokoclone&Date">
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=Ashish-Patnaik/kokoclone&type=Date&theme=dark" />
   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/svg?repos=Ashish-Patnaik/kokoclone&type=Date" />
   <img alt="Star History Chart" src="https://api.star-history.com/svg?repos=Ashish-Patnaik/kokoclone&type=Date" />
 </picture>
</a>


## Acknowledgments

This project builds upon the incredible open-source work of:

* **[Kokoro-ONNX](https://github.com/thewh1teagle/kokoro-onnx)** — for fast and efficient neural speech synthesis.
* **[Kanade Tokenizer](https://github.com/frothywater/kanade-tokenizer)** — for the brilliant zero-shot voice conversion architecture.

## License

Licensed under the [Apache 2.0 License](https://www.google.com/search?q=LICENSE).

```