XY_Tokenizer/README.md

# XY Tokenizer

XY Tokenizer is a speech codec that simultaneously models both semantic and acoustic aspects of speech, converting audio into discrete tokens and decoding them back to high-quality audio. It achieves efficient speech representation at only 1kbps with RVQ8 quantization at 12.5Hz frame rate.

## Features

- **Dual-channel modeling**: Simultaneously captures semantic meaning and acoustic details
- **Efficient representation**: 1kbps bitrate with RVQ8 quantization at 12.5Hz
- **High-quality audio tokenization**: Convert speech to discrete tokens and back with minimal quality loss
- **Long audio support**: Process audio files longer than 30 seconds using chunking with overlap
- **Batch processing**: Efficiently process multiple audio files in batches
- **24kHz output**: Generate high-quality 24kHz audio output

## Installation

```bash
# Create and activate conda environment
conda create -n xy_tokenizer python=3.10 -y && conda activate xy_tokenizer

# Install dependencies
pip install -r requirements.txt
```

## Usage

### Basic Inference

To tokenize audio files and reconstruct them:

```bash
python inference.py \
  --config_path ./config/xy_tokenizer_config.yaml \
  --checkpoint_path ./weights/xy_tokenizer.ckpt \
  --input_dir ./input_wavs/ \
  --output_dir ./output_wavs/
```

### Parameters

- `--config_path`: Path to the model configuration file
- `--checkpoint_path`: Path to the pre-trained model checkpoint
- `--input_dir`: Directory containing input WAV files
- `--output_dir`: Directory to save reconstructed audio files
- `--device`: Device to run inference on (default: "cuda")
- `--debug`, `--debug_ip`, `--debug_port`: Debugging options (disabled by default)

## Project Structure

- `xy_tokenizer/`: Core model implementation
  - `model.py`: Main XY_Tokenizer model class
  - `nn/`: Neural network components
- `config/`: Configuration files
- `utils/`: Utility functions
- `weights/`: Pre-trained model weights
- `input_wavs/`: Directory for input audio files
- `output_wavs/`: Directory for output audio files

## Model Architecture

XY Tokenizer uses a dual-channel architecture that simultaneously models:
1. **Semantic Channel**: Captures high-level semantic information and linguistic content
2. **Acoustic Channel**: Preserves detailed acoustic features including speaker characteristics and prosody

The model processes audio through several stages:
1. Feature extraction (mel-spectrogram)
2. Parallel semantic and acoustic encoding
3. Residual Vector Quantization (RVQ8) at 12.5Hz frame rate (1kbps)
4. Decoding and waveform generation

## License

[Specify your license here]