VoiceSculptor (models: X-Codec-2.0-25TPS-24k)

models/xcodec2/code/X-Codec-2.0-25TPS-24k.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:515c72147c6c44af49408f3cbb0ae318a1b2c8f08ce4eec6b0b5ddfd584620e8
+size 13680614

models/xcodec2/models/xcodec2-25TPS-24k/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

models/xcodec2/models/xcodec2-25TPS-24k/README.md

@@ -0,0 +1,67 @@
+---
+base_model:
+- HKUSTAudio/xcodec2
+datasets:
+- malaysia-ai/common_voice_17_0
+- mesolitica/Malaysian-STT-Whisper-Stage2
+- malaysia-ai/Multilingual-TTS
+- mesolitica/Malaysian-Emilia-v2
+library_name: transformers
+pipeline_tag: audio-to-audio
+---
+
+# xcodec2-25TPS-24k
+
+This repository contains the improved X-Codec-2.0 model as described in the paper [Improving X-Codec-2.0 for Multi-Lingual Speech: 25 Hz Latent Rate and 24 kHz Sampling](https://huggingface.co/papers/2601.20185).
+
+[![Preprint](https://img.shields.io/badge/technical-preprint-<COLOR>.svg)](https://github.com/Scicom-AI-Enterprise-Organization/X-Codec-2.0-25TPS-24k/blob/main/preprint/neurips_2023.pdf)
+
+Improve https://huggingface.co/HKUSTAudio/xcodec2 from 50 TPS to become 25 TPS and upscale output to 24k sample rate.
+
+WanDB at https://wandb.ai/huseinzol05/xcodec2-24k-25tps, we also pushed all checkpoints in [checkpoint](checkpoint).
+
+## Dataset
+
+1. https://huggingface.co/datasets/malaysia-ai/common_voice_17_0, train set only.
+2. https://huggingface.co/datasets/mesolitica/Malaysian-STT-Whisper-Stage2, except `noise` and `audioset_0.5s`.
+3. https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS, specific commit [2421a13e07226d96ac7009d5327d96a84672768c](https://huggingface.co/datasets/malaysia-ai/Multilingual-TTS/commit/2421a13e07226d96ac7009d5327d96a84672768c) except `cml-tts` and `libritts_r_filtered`
+4. https://huggingface.co/datasets/mesolitica/Malaysian-Emilia-v2, only `sg_podcast` and `malaysian_podcast`
+
+## How to use
+
+1. Git clone,
+
+```bash
+git clone https://github.com/Scicom-AI-Enterprise-Organization/X-Codec-2.0-25TPS-24k
+cd X-Codec-2.0-25TPS-24k
+```
+
+2. Load the model,
+
+```python
+from modeling_xcodec2 import XCodec2Model
+model = XCodec2Model.from_pretrained("Scicom-intl/xcodec2-25TPS-24k")
+```
+
+3. Encode,
+
+```python
+import librosa
+import torch
+
+y, sr = librosa.load('259041.mp3', sr = 16000)
+wav_tensor = torch.from_numpy(y).float().unsqueeze(0)
+codes = model.encode_code(wav_tensor)
+```
+
+4. Decode,
+
+```python
+import IPython.display as ipd
+
+ipd.Audio(model.decode_code(codes)[0, 0].cpu(), rate = 24000)
+```
+
+## Source code
+
+Source code at https://github.com/Scicom-AI-Enterprise-Organization/X-Codec-2.0-25TPS-24k

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D1045000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:49aa2f3b1ebb5fed0a70448a769c3f203036e72e0a5e239c63e63391718b17ca
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D1150000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c221501f69f89fcc108802e9b40247a62a75e0969930a17bac7f979da07cc30
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D1407500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4fa986876b642cb82094a757f603488c0a8c138b270e9951a9c3eb7411745a2d
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D1712500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e65d797086e1275da3ca16d1f68b0f260b9edcfc1a8434cd375b72e6c197301b
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D2005000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bd6174a68ba30e2825c27898b9f2b502cb9ce390cfe6b61a1512f8bf904c9e11
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D2325000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2f6942e06f0a6f41d459dbe71436ace8eeafd14c0e2c7ca4f9b8f5fcad297d65
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D2702500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a970479564fe3772866f396adcb4609a63dbd42a35bab4936cd9594ac704601
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D2902500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9fb5ae95e7ea6e9ee529f190d2631aec862758602b6c52559863a26fd95b1266
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D2997500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb7266c15b36760aaf6e64eef608be0bfaa6694c2b5ddf6adb6da7057d8574c9
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D3000000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb4f8312f961b067c500536127fa560da93eec8d4b37af9967a03ac85a2a0025
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D592500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:96d1aa0700a6d13aa107a5b50d5baa22561c8ec4df07e5087d5d095e8641c161
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D762500.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:049f3ada32131dfb998378d807ebb99e2cabe8a053bf41f858e999222ebe79c3
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/checkpoint/epoch3D880000.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13824baa9b2538d0ad261267e93cc16627dbec5316a3359446add74fbfc8da27
+size 5144889455

models/xcodec2/models/xcodec2-25TPS-24k/config.json

@@ -0,0 +1,15 @@
+{
+  "architectures": [
+    "XCodec2Model"
+  ],
+  "auto_map": {
+    "AutoModel": "modeling_xcodec2.XCodec2Model"
+  },
+  "codec_decoder_hidden_size": 1024,
+  "codec_encoder_hidden_size": 1024,
+  "dtype": "float32",
+  "model_type": "xcodec",
+  "semantic_hidden_size": 1024,
+  "transformers_version": "4.56.2",
+  "use_vocos": true
+}

models/xcodec2/models/xcodec2-25TPS-24k/configuration_bigcodec.py

@@ -0,0 +1,19 @@
+from transformers import PretrainedConfig
+
+class BigCodecConfig(PretrainedConfig):
+    model_type = "xcodec"
+
+    def __init__(
+        self,
+        # 下面这些只是示例超参
+        semantic_hidden_size=1024,
+        codec_encoder_hidden_size=1024,
+        codec_decoder_hidden_size=1024,
+        use_vocos=True,
+        **kwargs
+    ):
+        super().__init__(**kwargs)
+        self.semantic_hidden_size = semantic_hidden_size
+        self.codec_encoder_hidden_size = codec_encoder_hidden_size
+        self.codec_decoder_hidden_size = codec_decoder_hidden_size
+        self.use_vocos = use_vocos

models/xcodec2/models/xcodec2-25TPS-24k/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a8598048127f24161e69f0f83a4a3b1bcc83e4ebcf20d4ad6259b528090e997
+size 3301612648

models/xcodec2/models/xcodec2-25TPS-24k/modeling_xcodec2.py

@@ -0,0 +1,165 @@
+import torch
+import torch.nn as nn
+from transformers import PreTrainedModel
+from configuration_bigcodec import BigCodecConfig
+
+# 请确保这些模块路径是正确的
+from vq.codec_encoder import CodecEncoder_Transformer
+from vq.codec_decoder_vocos import CodecDecoderVocos
+from vq.module import SemanticEncoder
+from transformers import AutoFeatureExtractor, Wav2Vec2BertModel
+import torch.nn.functional as F
+class XCodec2Model(PreTrainedModel):
+    config_class = BigCodecConfig
+
+    def __init__(self, config: BigCodecConfig):
+        super().__init__(config)
+
+        # 1) 语义模型
+        self.semantic_model = Wav2Vec2BertModel.from_pretrained(
+            "facebook/w2v-bert-2.0",
+            output_hidden_states=True
+        )
+        self.semantic_model.eval()
+
+        self.SemanticEncoder_module = SemanticEncoder(
+             config.semantic_hidden_size,
+             config.semantic_hidden_size,
+             config.semantic_hidden_size
+        )
+
+        # 2) Codec Encoder
+        self.CodecEnc = CodecEncoder_Transformer()
+
+        # 3) Codec Decoder
+        self.generator = CodecDecoderVocos()  
+
+        # 4) 两个全连接层
+        self.fc_prior = nn.Linear(2048, 2048)
+        self.fc_post_a = nn.Linear(2048, 1024)
+        feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/w2v-bert-2.0")
+        self.feature_extractor = feature_extractor
+        self.avg_pooler = nn.AvgPool1d(2, stride=2)
+
+    def forward(self, input_waveform, input_features=None, sample_rate=16000):
+        """
+        这里的 forward 不一定要叫 forward，也可以拆成别的方法；
+        但是如果想兼容 pipeline，需要在 forward 里给出核心逻辑。
+        
+        参数：
+          input_waveform: [batch_size, waveform_length]
+          sample_rate: 默认 16000
+        返回：
+          重构后的语音音频 (Tensor)
+        """
+        # 1) 特征提取
+        # 如果需要 padding，可以在这里做
+        wav = input_waveform 
+        with torch.no_grad():
+          if input_features is None:
+              pad_for_wav = (320 - (wav.shape[1] % 320))
+              wav = torch.nn.functional.pad(wav, (0, pad_for_wav))
+              padded = F.pad(wav, (160, 160, 0, 0))
+              input_features = self.feature_extractor(
+                  padded.cpu().numpy(),
+                  sampling_rate=sample_rate, 
+                  return_tensors="pt"
+              ).input_features.to(self.device)
+
+        # 2) 语义层
+        semantic_output = self.semantic_model(input_features)
+        semantic_hidden_16 = semantic_output.hidden_states[16]  # 取第16层
+        semantic_hidden_16 = semantic_hidden_16.transpose(1, 2)  # [batch, hidden_dim, frames]
+        semantic_encoded = self.SemanticEncoder_module(semantic_hidden_16)
+
+        # 3) codec encoder
+        wav = wav.to(self.device)  # shape: [batch, 1, time]
+        vq_emb = self.CodecEnc(wav.unsqueeze(1))  # [batch, time//down, 1024] 只是示例
+        vq_emb = vq_emb.transpose(1, 2)  # -> [batch, 1024, frames]
+
+        concat_emb = torch.cat([semantic_encoded, vq_emb], dim=1)  # [batch, 1024 + 1024, frames]
+
+        # 5) fc_prior
+        concat_emb = self.fc_prior(concat_emb.transpose(1, 2)).transpose(1, 2)
+        concat_emb = self.avg_pooler(concat_emb)
+
+        # 6) decoder 的量化部分
+        _, vq_code, _ = self.generator(concat_emb, vq=True)
+        vq_post_emb = self.generator.quantizer.get_output_from_indices(vq_code.transpose(1, 2))
+        vq_post_emb = vq_post_emb.transpose(1, 2)
+
+        # 7) fc_post_a
+        vq_post_emb = self.fc_post_a(vq_post_emb.transpose(1, 2)).transpose(1, 2)
+
+        # 8) 最后解码成波形
+        recon_audio = self.generator(vq_post_emb.transpose(1, 2), vq=False)[0]
+        # recon_audio: [batch, time]
+        return recon_audio
+
+    def encode_code(self, input_waveform, sample_rate=16000):
+        """
+        将输入的音频编码为代码表示。
+
+        参数：
+          input_waveform: [batch_size, waveform_length]
+          sample_rate: 默认 16000
+        返回：
+          编码后的代码 (Tensor)
+        """
+        with torch.no_grad():
+ 
+    
+            wav = input_waveform 
+            pad_for_wav = (320 - (wav.shape[1] % 320))
+    
+            wav = torch.nn.functional.pad(wav, (0, pad_for_wav))
+
+            input_features = self.feature_extractor(
+                F.pad(wav[0,:].cpu(), (160, 160)), 
+                sampling_rate=sample_rate, 
+                return_tensors="pt"
+            ).input_features.to(self.device)  # [batch, frames, feat_dim]
+
+            # 2) 语义层
+            semantic_output = self.semantic_model(input_features)
+            semantic_hidden_16 = semantic_output.hidden_states[16]  # 取第16层
+            semantic_hidden_16 = semantic_hidden_16.transpose(1, 2)  # [batch, hidden_dim, frames]
+            semantic_encoded = self.SemanticEncoder_module(semantic_hidden_16)
+
+            # 3) codec encoder
+            wav = wav.to(self.device)  # shape: [batch, 1, time]
+            vq_emb = self.CodecEnc(wav.unsqueeze(1))  # [batch, time//down, 1024] 只是示例
+            vq_emb = vq_emb.transpose(1, 2)  # -> [batch, 1024, frames]
+
+            # 4) 拼接
+            concat_emb = torch.cat([semantic_encoded, vq_emb], dim=1)  # [batch, 2048, frames]
+
+            # 5) fc_prior
+            concat_emb = self.fc_prior(concat_emb.transpose(1, 2)).transpose(1, 2)
+
+            # 6) decoder 的量化部分，获取code
+            concat_emb = self.avg_pooler(concat_emb)
+            _, vq_code, _ = self.generator(concat_emb, vq=True)
+            # vq_code: [batch, frames]
+            return vq_code
+
+    def decode_code(self, vq_code):
+        """
+        将编码后的代码解码回音频。
+
+        参数：
+          vq_code: 编码后的代码 (Tensor) [batch, frames]
+        返回：
+          解码后的音频 (Tensor) [batch, waveform_length]
+        """
+        with torch.no_grad():
+            # 获取量化后的嵌入
+            vq_post_emb = self.generator.quantizer.get_output_from_indices(vq_code.transpose(1, 2))
+            vq_post_emb = vq_post_emb.transpose(1, 2)  # [batch, 1024, frames]
+
+            # 7) fc_post_a
+            vq_post_emb = self.fc_post_a(vq_post_emb.transpose(1, 2)).transpose(1, 2)  # [batch, 1024, frames]
+
+            # 8) 最后解码成波形
+            recon_audio = self.generator(vq_post_emb.transpose(1, 2), vq=False)[0]  # [batch, time]
+            return recon_audio

models/xcodec2/models/xcodec2-25TPS-24k/source.txt

@@ -0,0 +1 @@
+https://huggingface.co/Scicom-intl/xcodec2-25TPS-24k

models/xcodec2/models/xcodec2-25TPS-24k/test-set/sample-common-voice-17-test-set.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0447dbcc5fd06bdabd7f72161d992fb472c57672f612c26c456fb9110185e0a2
+size 2311734364

models/xcodec2/models/xcodec2-25TPS-24k/test-set/sample-common-voice-22-test-set.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd2f01dcef5cc75a5921e4ed94ef345ba768b3a9c2c1e1782c3582c3bcccb667
+size 2657635349