speech
codec
tokenizer
Metacebertrunk commited on
Commit
5184a76
Β·
verified Β·
1 Parent(s): ed9445c

Update README.md

Browse files

Push first version of hcodec-2.0

Files changed (1) hide show
  1. README.md +92 -1
README.md CHANGED
@@ -4,4 +4,95 @@ tags:
4
  - speech
5
  - codec
6
  - tokenizer
7
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
  - speech
5
  - codec
6
  - tokenizer
7
+ ---
8
+ ---
9
+ # QuarkAudio-HCodec: A Unified Discrete Audio Tokenizer for High-Fidelity, Multitask Audio Generation
10
+
11
+ <p align="center">
12
+ <a href="https://arxiv.org/pdf/2512.20151">
13
+ <img src="https://img.shields.io/badge/Paper-ArXiv-red.svg" alt="Paper">
14
+ </a>
15
+ <a href="https://github.com/alibaba/unified-audio/tree/main/QuarkAudio-HCodec/HCodec-2.0/">
16
+ <img src="https://img.shields.io/badge/GitHub-Code-green.svg" alt="GitHub">
17
+ </a>
18
+ <a href="https://huggingface.co/QuarkAudio/QuarkAudio-HCodec/">
19
+ <img src="https://img.shields.io/badge/Model-Hugging%20Face-yellow.svg" alt="Hugging Face">
20
+ </a>
21
+ <a href="https://www.modelscope.cn/models/QuarkAudio/QuarkAudio-HCodec/">
22
+ <img src="https://img.shields.io/badge/Model-%20%E9%AD%94%E6%90%AD-orange.svg" alt="ModelScope">
23
+ </a>
24
+ </p>
25
+
26
+ <p align="center">
27
+ <a href="https://arxiv.org/pdf/2512.20151"><img src="HCodec.jpg" width="70%" /></a>
28
+ </p>
29
+
30
+ > πŸ”Š **H-Codec**: *A Unified, Dual-Stream Neural Audio Codec with Adaptive Frame Rate and 48kHz Support*
31
+ > Enabling high-fidelity, efficient, and semantically rich audio tokenization for next-generation LLM-based audio generation.
32
+
33
+ πŸš€ **Key Highlights**:
34
+ - βœ… **Dual-Stream Tokenization**: Separately quantizes acoustic and semantic features into independent codebooks β€” preserving both signal fidelity and linguistic content.
35
+ - πŸ”„ **Dynamic Frame Rate (H-Codec-1.5)**: Introduces an adaptive temporal resolution mechanism built upon H-Codec-1.0, enabling variable frame rates based on content complexity.
36
+ - βš™οΈ **Multi-Sampling Rate (H-Codec-2.0)**: Extends the sampling rate from **16kHz to 48kHz** under a fixed frame rate, significantly improving audio fidelity and high-frequency detail preservation.
37
+ - 🌍 **Unified Foundation**: Designed as a core component for multimodal LLMs, supporting diverse downstream tasks: TTS, VC, Editing, TTA, SE, and more.
38
+
39
+ πŸ“„ **Paper**: [arXiv:2510.26372](https://arxiv.org/pdf/2512.20151) | πŸ€— **Model**: [Hugging Face Spaces](https://huggingface.co/QuarkAudio/QuarkAudio-HCodec/)
40
+
41
+ ---
42
+
43
+ ## πŸ“¦ Overview
44
+
45
+ This project introduces **H-Codec**, a unified discrete audio tokenizer that integrates self-supervised learning (SSL) representations into the codec architecture to enable **dual-stream (acoustic + semantic) tokenization**. Unlike prior work that fuses modalities before quantization (e.g., X-Codec), H-Codec employs **separate codebooks** for acoustic and semantic streams, allowing independent optimization and better reconstruction quality.
46
+
47
+ We extend the original H-Codec (*aka* H-Codec-1.0) in *UniTok-Audio (Liu et al., 2025)* into two advanced variants:
48
+
49
+ | Version | Key Feature | Sampling Rate | Frame Rate |
50
+ |---------------|----------------------------------|---------------|----------------|
51
+ | **H-Codec-1.0** | Dual-stream quantization | 16 kHz | Fixed |
52
+ | **H-Codec-1.5** | Dynamic frame rate adaptation | 16 kHz | Adaptive |
53
+ | **H-Codec-2.0** | Full-bandwidth 48kHz support | 48 kHz | Fixed |
54
+
55
+ These improvements significantly enhance **audio fidelity**, **temporal efficiency**, and **applicability** across speech, music, and general audio.
56
+
57
+ πŸ”§ **Architecture Core Components**:
58
+ 1. **Encoder**: Extracts continuous representations from waveform and SSL model (e.g., WavLM).
59
+ 2. **Quantizer Module**: Two independent codebooks β€” one for acoustic details, one for semantic meaning.
60
+ 3. **Decoder**: Reconstructs high-quality audio from discrete token sequences.
61
+
62
+ πŸ’‘ H-Codec is designed as a foundational module for **LLM-based audio generation**, seamlessly integrating with autoregressive language models for end-to-end training and inference.
63
+
64
+ <!-- ---
65
+
66
+ ## 🧰 Installation
67
+
68
+ ### Option 1: Using pip
69
+
70
+ ```bash
71
+ pip install -r requirements.txt -->
72
+
73
+
74
+ ---
75
+
76
+ ## 🎯 Quick Start: Run Inference in 3 Minutes
77
+
78
+ ### 1. Clone Repository
79
+
80
+ ```bash
81
+ git clone https://github.com/alibaba/unified-audio.git
82
+ cd QuarkAudio-HCodec-2.0
83
+ ```
84
+
85
+ ### 2. Create a Conda environment and install dependencies
86
+
87
+ ```bash
88
+ conda create -n unise python=3.10
89
+ conda activate unise
90
+ pip install -r requirements.txt
91
+ ```
92
+
93
+ ## 3. Tokenizer
94
+
95
+ ```bash
96
+ #!/bin/bash
97
+ python audio_tokenizer.py
98
+ ```