Hugging Face's logo

hf-audio
/
xcodec-hubert-librispeech

Feature Extraction
Transformers
Safetensors
xcodec
Model card Files
xet
Community
xcodec-hubert-librispeech
File size: 6,183 Bytes 
058074e
 
1ee9a72
 
 
058074e
 
1ee9a72
058074e
bdc0d0b
 
 
 
2344bd0
bdc0d0b
 
 
 
058074e
e17193c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
 
---
library_name: transformers
license: cc-by-4.0
datasets:
- openslr/librispeech_asr
---

# X-Codec (speech, HuBERT)

This codec is part of the X-Codec family of codecs as shown below:

| Model checkpoint                                  | Semantic Model                                                        | Domain        | Training Data                 |
|--------------------------------------------|-----------------------------------------------------------------------|---------------|-------------------------------|
| **[xcodec-hubert-librispeech](https://huggingface.co/hf-audio/xcodec-hubert-librispeech) (this model)**                | [facebook/hubert-base-ls960](https://huggingface.co/facebook/hubert-base-ls960)   | Speech        | Librispeech                   |
| [xcodec-wavlm-mls](https://huggingface.co/hf-audio/xcodec-wavlm-mls)  | [microsoft/wavlm-base-plus](https://huggingface.co/microsoft/wavlm-base-plus)| Speech        | MLS English                   |
| [xcodec-wavlm-more-data](https://huggingface.co/hf-audio/xcodec-wavlm-more-data) | [microsoft/wavlm-base-plus](https://huggingface.co/microsoft/wavlm-base-plus)| Speech        | MLS English + Internal data   |
| [xcodec-hubert-general](https://huggingface.co/hf-audio/xcodec-hubert-general)                 | [ZhenYe234/hubert_base_general_audio](https://huggingface.co/ZhenYe234/hubert_base_general_audio) | General audio | 200k hours internal data      |
| [xcodec-hubert-general-balanced](https://huggingface.co/hf-audio/xcodec-hubert-general-balanced) | [ZhenYe234/hubert_base_general_audio](https://huggingface.co/ZhenYe234/hubert_base_general_audio) | General audio | More balanced data            |

Original model is `xcodec_hubert_librispeech` from [this table](https://github.com/zhenye234/xcodec?tab=readme-ov-file#available-models).

## Example usage

The example below applies the codec over all possible bandwidths.

```python

from datasets import Audio, load_dataset
from transformers import XcodecModel, AutoFeatureExtractor
import torch
import os
from scipy.io.wavfile import write as write_wav


model_id = "hf-audio/xcodec-hubert-librispeech"
torch_device = "cuda" if torch.cuda.is_available() else "cpu"
available_bandwidths = [0.5, 1, 1.5, 2, 4]

# load model
model = XcodecModel.from_pretrained(model_id, device_map=torch_device)
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

# load audio example
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
librispeech_dummy = librispeech_dummy.cast_column(
    "audio", Audio(sampling_rate=feature_extractor.sampling_rate)
)
audio_array = librispeech_dummy[0]["audio"]["array"]
inputs = feature_extractor(
    raw_audio=audio_array, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt"
).to(model.device)
audio = inputs["input_values"]

for bandwidth in available_bandwidths:
    print(f"Encoding with bandwidth: {bandwidth} kbps")
    # encode
    audio_codes = model.encode(audio, bandwidth=bandwidth, return_dict=False)
    print("Codebook shape", audio_codes.shape)
    # 0.5 kbps -> torch.Size([1, 1, 293])
    # 1.0 kbps -> torch.Size([1, 2, 293])
    # 1.5 kbps -> torch.Size([1, 3, 293])
    # 2.0 kbps -> torch.Size([1, 4, 293])
    # 4.0 kbps -> torch.Size([1, 8, 293])

    # decode
    input_values_dec = model.decode(audio_codes).audio_values

    # save audio to file
    write_wav(f"{os.path.basename(model_id)}_{bandwidth}.wav", feature_extractor.sampling_rate, input_values_dec.squeeze().detach().cpu().numpy())

write_wav("original.wav", feature_extractor.sampling_rate, audio.squeeze().detach().cpu().numpy())
```

### 🔊 Audio Samples

**Original**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/original.wav" type="audio/wav">
</audio>

**0.5 kbps**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/xcodec-hubert-librispeech_0.5.wav" type="audio/wav">
</audio>

**1 kbps**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/xcodec-hubert-librispeech_1.wav" type="audio/wav">
</audio>

**1.5 kbps**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/xcodec-hubert-librispeech_1.5.wav" type="audio/wav">
</audio>

**2 kbps**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/xcodec-hubert-librispeech_2.wav" type="audio/wav">
</audio>

**4 kbps**
<audio controls>
  <source src="https://huggingface.co/datasets/bezzam/xcodec_samples/resolve/main/xcodec-hubert-librispeech_4.wav" type="audio/wav">
</audio>

## Batch example

```python

from datasets import Audio, load_dataset
from transformers import XcodecModel, AutoFeatureExtractor
import torch


model_id = "hf-audio/xcodec-hubert-librispeech"
torch_device = "cuda" if torch.cuda.is_available() else "cpu"
bandwidth = 4
n_audio = 2  # number of audio samples to process in a batch

# load model
model = XcodecModel.from_pretrained(model_id, device_map=torch_device)
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)

# load audio example
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
ds = ds.cast_column(
    "audio", Audio(sampling_rate=feature_extractor.sampling_rate)
)
audio = [audio_sample["array"] for audio_sample in ds[-n_audio:]["audio"]]
print(f"Input audio shape: {[_sample.shape for _sample in audio]}")
# Input audio shape: [(113840,), (71680,)]
inputs = feature_extractor(
    raw_audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt"
).to(model.device)
audio = inputs["input_values"]
print(f"Padded audio shape: {audio.shape}")
# Padded audio shape: torch.Size([2, 1, 113920])

# encode
audio_codes = model.encode(audio, bandwidth=bandwidth, return_dict=False)
print("Codebook shape", audio_codes.shape)
# Codebook shape torch.Size([2, 8, 356])

# decode
decoded_audio = model.decode(audio_codes).audio_values
print("Decoded audio shape", decoded_audio.shape)
# Decoded audio shape torch.Size([2, 1, 113920])
```