README.md

---
library_name: transformers
tags:
  - audio
  - speaker-recognition
  - speaker-embedding
  - speaker-verification
  - wav2vec2
  - vocbulwark
license: cc-by-4.0
---

# VocBulwark speaker encoder

Standalone speaker encoder: turns a raw waveform into a fixed **768-dimensional** speaker embedding. It is the conditioning front-end for the companion VocBulwark vocoder — compute an embedding once from a reference clip of a speaker, then pass it to the vocoder to synthesize in that voice. The embedding is also usable on its own for speaker verification / similarity.

Wav2Vec2-based encoder with attentive statistics pooling, trained with a GE2E objective. Self-contained: loads with `trust_remote_code=True`, no training repo required.

## Companion Models

This speaker encoder is part of a set of 6 repositories:

| Repo | Role |
|------|------|
| `mlr2000/vocoder-large` | Large vocoder |
| `mlr2000/vocoder-large-watermark-detector` | Watermark detector for the large model |
| `mlr2000/vocoder-large-speaker-encoder` | Speaker encoder for the large model |
| `mlr2000/vocoder-small` | Small vocoder (generates the watermarked audio) |
| `mlr2000/vocoder-small-watermark-detector` | Watermark detector for the small model |
| `mlr2000/vocoder-small-speaker-encoder` | Speaker encoder (this repo) |


## Usage

```python
import torchaudio, torchaudio.functional as AF
from transformers import AutoModel

enc = AutoModel.from_pretrained("mlr2000/vocoder-small-speaker-encoder", trust_remote_code=True).eval()

wav, sr = torchaudio.load("reference.wav")             # [C, T]
wav = wav.mean(0, keepdim=True)                        # mono [1, T]
if sr != enc.config.raw_sample_rate:                   # encoder expects 22.05 kHz
    wav = AF.resample(wav, sr, enc.config.raw_sample_rate)

emb = enc.embed(wav)          # [1, 768]  — feed as speaker_embedding to the vocoder
```

See **`example_roundtrip.ipynb`** in this repo for the full pipeline (reference clip → embedding → vocode → detect watermark).

## Notes

- Input: mono waveform at **22.05 kHz** (`config.raw_sample_rate`). Resample first if your audio differs.
- Output: `[B, 768]` L2-comparable speaker embeddings.
- Pair with the VocBulwark small vocoder trained *jointly* with this encoder. An embedding from a different encoder will not condition the vocoder correctly.
- This encoder is paired with `mlr2000/vocoder-small`. For the large vocoder (`mlr2000/vocoder-large`) use the companion large speaker encoder (`mlr2000/vocoder-large-speaker-encoder`).
- Not intended for speaker identification or surveillance of individuals without consent.

## Citation

If you use this model, please cite:

```bibtex
@misc{muletta2026,
  title  = {Training a Discriminator-Free Foundation Vocoder 
             with Integrated Audio Watermarking},
  author = {Muletta, Romolo and Deriu, Jan},
  year   = {2026},
  note   = {VT2 Project Report, ZHAW School of Engineering}
}
```

## License

`cc-by-4.0`. Trained on MLS (CC-BY-4.0) and Common Voice (CC0). Builds on BigVGAN (MIT) and wav2vec 2.0 (Apache-2.0). Please retain attribution when redistributing or building on this model.