MeanVC (code, models, paper)

.gitattributes

@@ -33,3 +33,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+MeanVC.[[:space:]]Lightweight[[:space:]]and[[:space:]]Streaming[[:space:]]Zero-Shot[[:space:]]Voice[[:space:]]Conversion[[:space:]]via[[:space:]]Mean[[:space:]]Flows.pdf filter=lfs diff=lfs merge=lfs -text
+models/en,zh/MeanVC/figs/model.png filter=lfs diff=lfs merge=lfs -text
+models/en,zh/MeanVC/figs/npu@aslp.jpeg filter=lfs diff=lfs merge=lfs -text

MeanVC. Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bfef007767b7af2c7d0cd0d12a3069518568fdb17394fad19df0eba85cece251
+size 852776

code/MeanVC [Zemacs] +1 Streaming ONNX export.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79e94feed1289797fe18e501452897893748ca29be12681063c41597abde1a05
+size 5401663

code/MeanVC [sutungpo] +5 -2.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83412d4c866b0c7410651fd26e88790599bf639999af6a09c2a07fd1d389e5dc
+size 28387187

code/MeanVC [y12rf] +6.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:18d821f32793c9c16206d5f2b7780b392457202c745f2a938ee425ec6f330d0a
+size 5434841

code/MeanVC-experiments [benhsampson] +1 -1 Notebook.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6fc1771c4a78f11307e61d18d4baa1b2e30d8c121f90873bf02d789ece5a8b0c
+size 6456970

code/MeanVC.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b9bb04097804f772c2ce104b31418cb5cc7333b61edf48ccad4830c1bd334b58
+size 29525745

models/en,zh/MeanVC/.gitattributes

@@ -0,0 +1,37 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text

models/en,zh/MeanVC/README.md

@@ -0,0 +1,109 @@
+---
+language:
+- en
+- zh
+license: apache-2.0
+pipeline_tag: audio-to-audio
+---
+
+# MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows
+
+<div align="center">
+
+[![Paper](https://img.shields.io/badge/arXiv-2510.08392-b31b1b.svg)](https://arxiv.org/pdf/2510.08392)
+[![Github](https://img.shields.io/badge/Github-Page-green)](https://github.com/ASLP-lab/MeanVC)
+[![Demo Page](https://img.shields.io/badge/Demo-Audio%20Samples-green)](https://aslp-lab.github.io/MeanVC/)
+
+</div>
+
+**MeanVC** is a lightweight and streaming zero-shot voice conversion system that enables real-time timbre transfer from any source speaker to any target speaker while preserving linguistic content. The system introduces a diffusion transformer with a chunk-wise autoregressive denoising strategy and mean flows for efficient single-step inference.
+
+![img](https://huggingface.co/ASLP-lab/MeanVC/resolve/main/figs/model.png)
+
+## ✨ Key Features
+
+-   **🚀 Streaming Inference**: Real-time voice conversion with chunk-wise processing.
+-   **⚡ Single-Step Generation**: Direct mapping from start to endpoint via mean flows for fast generation.
+-   **🎯 Zero-Shot Capability**: Convert to any unseen target speaker without re-training.
+-   **💾 Lightweight**: Significantly fewer parameters than existing methods.
+-   **🔊 High Fidelity**: Superior speech quality and speaker similarity.
+
+## 💻 Sample Usage
+
+### 1. Environment Setup
+First, follow these steps to clone the repository and install the required environment.
+
+```bash
+# Clone the repository and enter the directory
+git clone https://github.com/ASLP-lab/MeanVC.git
+cd MeanVC
+
+# Create and activate a Conda environment
+conda create -n meanvc python=3.11 -y
+conda activate meanvc
+
+# Install dependencies
+pip install -r requirements.txt
+```
+
+### 2. Download Pre-trained Models
+Run the provided script to automatically download all necessary pre-trained models.
+
+```bash
+python download_ckpt.py
+```
+
+This will download the main VC model, vocoder, and ASR model into the `src/ckpt/` directories.
+The speaker verification model (`wavlm_large_finetune.pth`) must be downloaded manually from Google Drive. Download the file from [this link](https://drive.google.com/file/d/1-aE1NfzpRCLxA4GUxX9ITI3F9LlbtEGP/view). Place the downloaded `wavlm_large_finetune.pth` file into the `src/runtime/speaker_verification/ckpt/` directory.
+
+### 3. Real-Time Voice Conversion
+This script captures audio from your microphone and converts it in real-time to the voice of a target speaker.
+
+```bash
+python src/runtime/run_rt.py --target-path "path/to/target_voice.wav"
+```
+
+-   `--target-path`: Path to a clean audio file of the target speaker. This voice will be used as the conversion target. An example file is provided at `src/runtime/example/test.wav`.
+
+When you run the script, you will be prompted to select your audio input (microphone) and output (speaker) devices from a list.
+
+### 4. Offline Voice Conversion
+For batch processing or converting pre-recorded audio files, use the offline conversion script.
+
+```bash
+bash scripts/infer_ref.sh
+```
+
+Before running the script, you need to configure the following paths in `scripts/infer_ref.sh`:
+
+-   `source_path`: Path to the source audio file or directory containing multiple audio files to be converted
+-   `reference_path`: Path to a clean audio file of the target speaker (used as voice reference)
+-   `output_dir`: Directory where converted audio files will be saved (default: `src/outputs`)
+-   `steps`: Number of denoising steps (default: 2)
+
+## 📜 License & Disclaimer
+
+MeanVC is released under the Apache License 2.0. This open-source license allows you to freely use, modify, and distribute the model, as long as you include the appropriate copyright notice and disclaimer.
+
+MeanVC is designed for research and legitimate applications in voice conversion technology. Users must obtain proper consent from individuals whose voices are being converted or used as references. We strongly discourage any malicious use including impersonation, fraud, or creating misleading audio content. Users are solely responsible for ensuring their use cases comply with ethical standards and legal requirements.
+
+## 📄 Citation
+
+If you find our work helpful, please cite our paper:
+
+```bibtex
+@article{ma2025meanvc,
+  title={MeanVC: Lightweight and Streaming Zero-Shot Voice Conversion via Mean Flows},
+  author={Ma, Guobin and Yao, Jixun and Ning, Ziqian and Jiang, Yuepeng and Xiong, Lingxin and Xie, Lei and Zhu, Pengcheng},
+  journal={arXiv preprint arXiv:2510.08392},
+  year={2025}
+}
+```
+
+## 📧 Contact
+
+If you are interested in leaving a message to our research team, feel free to email guobin.ma@mail.nwpu.edu.cn
+
+<p align="center">
+    <img src="https://huggingface.co/ASLP-lab/MeanVC/resolve/main/figs/npu@aslp.jpeg" width="500"/>
+</p>

models/en,zh/MeanVC/config.json

@@ -0,0 +1,14 @@
+{
+    "model_type": "DiT",
+    "model": {
+        "dim": 512,
+        "depth": 4,
+        "heads": 2,
+        "ff_mult": 2,
+        "bn_dim": 256,
+        "conv_layers": 4,
+        "chunk_size": 20,
+        "dropout": 0.0,
+        "qk_norm": "rms_norm"
+    }
+}

models/en,zh/MeanVC/fastu2++.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d6bc4290c4d489ed50b6ffbbcda33bd3ba9551506852c7f2fa683f9fe9512a1
+size 91972768

models/en,zh/MeanVC/meanvc_200ms.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17a234944c7e63bfc94e71eea7de8dfb9f7f2e990cde9fd8df12ddad5237c68f
+size 56355402

models/en,zh/MeanVC/model_200ms.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c2d9ed6c8c149d4fdf9ba6f17ebbc675784010585344448136261c874decb0f
+size 56271424

models/en,zh/MeanVC/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/ASLP-lab/MeanVC

models/en,zh/MeanVC/vocos.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9e8aba28aa9cea0813e571a25ee33ef35bd74c803da34a65e683d9b0f7e2f281
+size 33223980

models/wavlm_large_finetune.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51f07e3b94d9e0262a6a675ef5a087be3dd09e8c62e9d886827f44f82fe7f94b
+size 1301926579