Wendy9805 commited on
Commit
8feb7a1
·
verified ·
1 Parent(s): f00909c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +79 -25
README.md CHANGED
@@ -76,58 +76,112 @@ Trained on [REAL-PS4](https://huggingface.co/datasets/TaurenMountain/REAL-PS4),
76
 
77
  ## Usage
78
 
79
- ### Load the Model
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  ```python
82
  import torch
 
83
 
84
- # Load checkpoint
85
- ckpt = torch.load("checkpoint_epoch037.pt", map_location="cpu")
 
 
 
 
 
 
 
 
 
86
 
87
- # The checkpoint contains the full TSE model state dict
88
- # Compatible with the wesep BSRNN + ECAPA-TDNN framework
89
- model_state = ckpt["model"] if "model" in ckpt else ckpt
90
  ```
91
 
92
- ### Inference Example
93
 
94
  ```python
95
  import torch
96
  import torchaudio
97
- from wesep.models import get_model
98
 
99
- # Initialize model (must match training config)
100
- model = get_model("BSRNN")(
101
  feat_type="consistent",
102
  feature_dim=128,
103
  num_repeat=6,
104
  spk_emb_dim=192,
105
  spk_fuse_type="multiply",
 
106
  spk_model="ECAPA_TDNN_GLOB_c512",
107
- sr=16000,
108
- win=512,
109
- stride=128,
 
 
 
 
110
  )
111
-
112
- # Load PS4 weights
113
- ckpt = torch.load("checkpoint_epoch037.pt", map_location="cpu")
114
- model.load_state_dict(ckpt["model"] if "model" in ckpt else ckpt)
115
  model.eval()
116
 
117
- # Load mixture and enrollment audio (16 kHz mono)
118
- mixture, sr = torchaudio.load("mixture.wav")
119
- enrollment, sr = torchaudio.load("enrollment.wav")
 
120
 
121
- # Run extraction
 
 
122
  with torch.no_grad():
123
- extracted = model(mixture, enrollment)
124
  ```
125
 
126
- > **Note:** The speaker encoder is frozen during training (`spk_model_freeze: true`). For inference, use the same ResNet34-LM speaker encoder as the evaluation pipeline: `voxceleb_resnet34_LM` for English, `cnceleb_resnet34_LM` for Chinese.
127
 
128
- ## Pretrained Backbone
 
 
 
 
129
 
130
- PS4 is fine-tuned from `bsrnn_ecapa_vox1`, a BSRNN + ECAPA-TDNN model pretrained on VoxCeleb1. The proxy-supervised fine-tuning on REAL-PS4 adapts the model to real far-field multi-speaker meeting conditions.
 
 
131
 
132
  ## Citation
133
 
 
76
 
77
  ## Usage
78
 
79
+ ### Quick Start (recommended)
80
+
81
+ Use the included [`inference.py`](inference.py) — a self-contained script with absolutely no external dependencies beyond `torch`, `torchaudio`, and `numpy`:
82
+
83
+ ```bash
84
+ # Install dependencies
85
+ pip install torch torchaudio numpy
86
+
87
+ # Single file extraction
88
+ python inference.py \
89
+ --checkpoint checkpoint_epoch037.pt \
90
+ --mix mix.wav \
91
+ --enroll target_speaker.wav \
92
+ --output result.wav
93
+
94
+ # Use GPU
95
+ python inference.py \
96
+ --checkpoint checkpoint_epoch037.pt \
97
+ --mix mix.wav \
98
+ --enroll target.wav \
99
+ --output result.wav \
100
+ --device cuda
101
+
102
+ # Batch mode (process all .wav files in a directory)
103
+ python inference.py \
104
+ --checkpoint checkpoint_epoch037.pt \
105
+ --mix-dir ./mixtures/ \
106
+ --enroll-dir ./enrollments/ \
107
+ --output-dir ./results/ \
108
+ --device cuda
109
+
110
+ # List available CUDA devices
111
+ python inference.py --list-devices
112
+ ```
113
+
114
+ ### Python API
115
 
116
  ```python
117
  import torch
118
+ from inference import BSRNN, load_audio, extract_speaker, load_checkpoint, build_model
119
 
120
+ # Build model and load weights
121
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
122
+ model = build_model(device)
123
+ load_checkpoint("checkpoint_epoch037.pt", model, device)
124
+
125
+ # Load audio (16 kHz mono)
126
+ mix = load_audio("mixture.wav")
127
+ enroll = load_audio("enrollment.wav")
128
+
129
+ # Run extraction
130
+ extracted = extract_speaker(model, mix, enroll, device)
131
 
132
+ # Save result
133
+ torchaudio.save("result.wav", extracted, 16000)
 
134
  ```
135
 
136
+ ### Advanced: Manual Model Loading
137
 
138
  ```python
139
  import torch
140
  import torchaudio
141
+ from inference import BSRNN
142
 
143
+ # Build model with exact training config
144
+ model = BSRNN(
145
  feat_type="consistent",
146
  feature_dim=128,
147
  num_repeat=6,
148
  spk_emb_dim=192,
149
  spk_fuse_type="multiply",
150
+ multi_fuse=False,
151
  spk_model="ECAPA_TDNN_GLOB_c512",
152
+ sr=16000, win=512, stride=128,
153
+ spk_args={"feat_dim": 80, "embed_dim": 192, "pooling_func": "ASTP"},
154
+ spk_model_freeze=True,
155
+ use_spk_transform=False,
156
+ joint_training=True,
157
+ multi_task=False,
158
+ spk_feat=False,
159
  )
 
 
 
 
160
  model.eval()
161
 
162
+ # Load checkpoint
163
+ ckpt = torch.load("checkpoint_epoch037.pt", map_location="cpu")
164
+ state_dict = ckpt["model"] if "model" in ckpt else ckpt
165
+ model.load_state_dict(state_dict, strict=False)
166
 
167
+ # Run inference
168
+ mix, sr = torchaudio.load("mixture.wav")
169
+ enroll, sr = torchaudio.load("enrollment.wav")
170
  with torch.no_grad():
171
+ extracted, _ = model(mix, enroll)
172
  ```
173
 
174
+ ## What's in this repository
175
 
176
+ | File | Description |
177
+ |------|-------------|
178
+ | `checkpoint_epoch037.pt` | PS4 model weights (epoch 37, proxy-supervised fine-tuned) |
179
+ | [`inference.py`](inference.py) | Self-contained inference script (no external ML libs required) |
180
+ | `README.md` | This file |
181
 
182
+ The training code, dataset, and full evaluation pipeline are available at:
183
+ - **Training code:** [GitHub - TaurenMountain/real-t](https://github.com/TaurenMountain/real-t) (see `opensource/code/`)
184
+ - **Dataset:** [TaurenMountain/REAL-PS4](https://huggingface.co/datasets/TaurenMountain/REAL-PS4)
185
 
186
  ## Citation
187