|
|
--- |
|
|
license: mit |
|
|
tags: |
|
|
- audio |
|
|
- speech-enhancement |
|
|
- denoising |
|
|
- generative |
|
|
pipeline_tag: audio-to-audio |
|
|
--- |
|
|
|
|
|
# DAC-SE1: High-Fidelity Speech Enhancement via Discrete Audio Tokens |
|
|
|
|
|
This checkpoint has been trained to specifically reflect real-world denoising scenarios. It utilizes discrete audio tokens to perform high-fidelity speech enhancement, treating audio restoration as a sequence modeling task to generate clean audio tokens from noisy input sequences. |
|
|
|
|
|
## Usage |
|
|
|
|
|
To use this model, you need the inference tools and tokenizers provided in the official GitHub repository. |
|
|
|
|
|
### 1. Setup Environment |
|
|
First, clone the repository to get the necessary helper scripts (`DACTools`, `DACTokenizer`, etc.) and navigate into the folder: |
|
|
|
|
|
```sh |
|
|
git clone https://github.com/ETH-DISCO/DAC-SE1.git |
|
|
cd DAC-SE1 |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
### 2. Inference |
|
|
You can run the following Python script to denoise an audio file. |
|
|
|
|
|
```python |
|
|
import torch |
|
|
from transformers import LlamaForCausalLM, LogitsProcessorList |
|
|
from inference import DACTools, DACTokenizer, DACConstrainedLogitsProcessor |
|
|
import re |
|
|
from huggingface_hub import login |
|
|
|
|
|
|
|
|
# Initialize DAC tools for audio encoding/decoding |
|
|
dac_tools = DACTools() |
|
|
tokenizer = DACTokenizer(num_layers=9, codebook_size=1024) |
|
|
|
|
|
# Load denoiser model |
|
|
model_path = "disco-eth/DAC-SE1" |
|
|
model = LlamaForCausalLM.from_pretrained(model_path) |
|
|
model = model.to('cuda') |
|
|
model.eval() |
|
|
|
|
|
# Load noisy audio and convert to tokens |
|
|
noisy_tokens = dac_tools.audio_to_tokens('input.wav') |
|
|
|
|
|
# Prepare input for model |
|
|
token_ids = tokenizer.encode(noisy_tokens, add_special_tokens=False) |
|
|
input_ids = [tokenizer.bos_token_id] + token_ids + [tokenizer.start_clean_token_id] |
|
|
input_tensor = torch.tensor([input_ids]).cuda() |
|
|
|
|
|
# Generate clean tokens |
|
|
num_tokens = len(re.findall(r'<\|s\d+_c\d\|>', noisy_tokens)) |
|
|
logits_processor = LogitsProcessorList([ |
|
|
DACConstrainedLogitsProcessor(tokenizer=tokenizer, min_tokens=num_tokens) |
|
|
]) |
|
|
|
|
|
with torch.no_grad(): |
|
|
outputs = model.generate( |
|
|
input_tensor, |
|
|
max_new_tokens=num_tokens, |
|
|
min_new_tokens=num_tokens, |
|
|
logits_processor=logits_processor, |
|
|
pad_token_id=tokenizer.pad_token_id, |
|
|
eos_token_id=tokenizer.eos_token_id, |
|
|
do_sample=False, |
|
|
) |
|
|
|
|
|
# Extract generated tokens |
|
|
generated_ids = outputs[0, len(input_ids):].tolist() |
|
|
generated_output = tokenizer.decode(generated_ids, skip_special_tokens=True) |
|
|
|
|
|
# Convert tokens back to audio |
|
|
valid_tokens = re.findall(r'<\|s\d+_c\d\|>', generated_output) |
|
|
if valid_tokens: |
|
|
remainder = len(valid_tokens) % 9 |
|
|
if remainder != 0: |
|
|
valid_tokens = valid_tokens[:len(valid_tokens) - remainder] |
|
|
denoised_tokens = "".join(valid_tokens) |
|
|
tokens = dac_tools.string_to_tokens(denoised_tokens) |
|
|
clean_audio = dac_tools.tokens_to_audio(tokens) |
|
|
|
|
|
# Save denoised audio |
|
|
import soundfile as sf |
|
|
sf.write('output.wav', clean_audio, dac_tools.sample_rate) |
|
|
|
|
|
``` |
|
|
|
|
|
## Citation |
|
|
If you use this model, please cite our paper: |
|
|
|
|
|
```bibtex |
|
|
@misc{lanzendörfer2025highfidelityspeechenhancementdiscrete, |
|
|
title={High-Fidelity Speech Enhancement via Discrete Audio Tokens}, |
|
|
author={Luca A. Lanzendörfer and Frédéric Berdoz and Antonis Asonitis and Roger Wattenhofer}, |
|
|
year={2025}, |
|
|
eprint={2510.02187}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.SD}, |
|
|
url={https://arxiv.org/abs/2510.02187}, |
|
|
} |
|
|
``` |
