Initial v0.1 alpha release

README.md

@@ -0,0 +1,177 @@
+---
+license: apache-2.0
+language:
+  - en
+tags:
+  - audio
+  - speech
+  - neural-audio-codec
+  - speech-codec
+  - speech-llm
+  - speech-to-speech
+  - zero-shot-voice-cloning
+  - speech-enhancement
+  - asr
+  - pytorch
+library_name: pytorch
+pipeline_tag: audio-to-audio
+---
+
+# SoviaMate-Codec
+
+Pretrained weights for **SoviaMate-Codec**, a neural audio codec designed from the ground up for integration with speech-aware large language models.
+
+SoviaMate-Codec is the first released component of [**SoviaMate**](https://github.com/samson-ailabs/SoviaMate) — an open research effort building toward end-to-end spoken dialogue systems.
+
+> 🚧 **Status**: alpha research release. APIs are not stable; evaluation numbers are preliminary.
+
+## What's in this repository
+
+```
+samson-ailabs/SoviaMate-Codec
+├── neural_audio_codec/
+│   ├── audio_codec_base.ckpt   # reconstruction codec
+│   └── audio_codec_spk.ckpt    # voice-conversion codec (+ ASR head)
+└── speaker_verification/
+    ├── campplus.bin            # CAM++ speaker verifier
+    ├── eres2netv2.ckpt         # ERes2Net-v2 speaker verifier
+    └── wavlm_ecapa.pth         # WavLM + ECAPA-TDNN speaker verifier
+```
+
+| Asset | Purpose | Size |
+|---|---|---|
+| `neural_audio_codec/audio_codec_base.ckpt` | **Reconstruction codec.** Encoder + quantizer + decoder, trained as a standard compress / reconstruct codec without the speaker-adaptation objective. Use for low-bitrate speech coding and feature extraction. (No ASR head.) | ~753 MB |
+| `neural_audio_codec/audio_codec_spk.ckpt` | **Voice-conversion codec.** Adds the integrated ASR head and the post-quantization speaker adapter trained for zero-shot voice swapping from a 3–5 s reference. Always pass a speaker prompt — running it without one under-conditions the decoder and degrades quality. Use `base` for plain reconstruction. | ~939 MB |
+| `speaker_verification/*` | Pretrained speaker-embedding extractors. `campplus.bin` and `eres2netv2.ckpt` are interchangeable backbones for the speaker adapter — whichever was used at training is also required at inference time for that `spk` checkpoint (this release uses `campplus.bin`). `wavlm_ecapa.pth` is for evaluation only (e.g., SECS-style speaker-similarity scoring). | ~1.3 GB total |
+
+Each codec checkpoint is a portable export containing `model_weights` (per-module `state_dict`) and `hyper_parameters` (architecture config), produced by `AudioCodecTask.export_model()`. Optimizer state, discriminators, and other training-only components are excluded.
+
+## Architecture at a glance
+
+Four design choices distinguish SoviaMate-Codec from EnCodec / SoundStream / DAC:
+
+1. **ASR decoder *before* quantization** *(spk checkpoint only)* — A lightweight ASR head reads the encoder's continuous features. Its gradient forces linguistic content into the representation, so semantic fidelity is directly measurable (WER), not assumed.
+2. **Continuous features for LLM input** — Discrete tokens are used only for compression/transmission. The downstream LLM consumes the *pre-quantization* continuous features, avoiding quantization loss in the LLM input path.
+3. **Speech enhancement as a training paradigm** — The codec is trained noisy-in → clean-out, so the encoder learns to discard noise rather than encode it.
+4. **Post-quantization speaker adapter** *(spk checkpoint only)* — A hybrid AdaLN + cross-attention adapter injects voice identity after quantization. This decouples "what is said" from "who says it" and enables zero-shot voice swapping from a 3–5 s reference.
+
+Full architecture write-up: [SoviaMate repository](https://github.com/samson-ailabs/SoviaMate). A technical report is in preparation.
+
+## Load in Python
+
+Download just what you need:
+```bash
+# Reconstruction only (base checkpoint)
+hf download samson-ailabs/SoviaMate-Codec \
+    --include "neural_audio_codec/audio_codec_base.ckpt" \
+    --local-dir checkpoints
+
+# Voice conversion (spk checkpoint + the campplus speaker verifier it depends on)
+hf download samson-ailabs/SoviaMate-Codec \
+    --include "neural_audio_codec/audio_codec_spk.ckpt" \
+    --include "speaker_verification/campplus.bin" \
+    --local-dir checkpoints
+```
+
+Then, after installing SoviaMate (see [Getting started](https://github.com/samson-ailabs/SoviaMate#getting-started)), load a checkpoint into an `AudioCodecBundle`. Pick the checkpoint that matches the task — they are **not** interchangeable.
+
+### Reconstruction — use the `base` checkpoint
+```python
+from soviamate.bundles import AudioCodecBundle
+
+reconstructor = AudioCodecBundle.from_checkpoint(
+    "checkpoints/neural_audio_codec/audio_codec_base.ckpt",
+    device="cuda",  # or "cpu"
+)
+
+# Compress → decode
+reconstructed, _ = reconstructor(source_audio)
+```
+
+### Voice conversion (+ optional ASR transcript) — use the `spk` checkpoint
+```python
+voice_converter = AudioCodecBundle.from_checkpoint(
+    "checkpoints/neural_audio_codec/audio_codec_spk.ckpt",
+    device="cuda",
+)
+
+# Convert source speech to a target speaker via a 3–5 s reference
+converted, _ = voice_converter(source_audio, prompt_audios=target_speaker_audio)
+
+# Voice conversion with an ASR transcript as a by-product
+converted, transcript = voice_converter(
+    source_audio, prompt_audios=target_speaker_audio, return_text=True
+)
+```
+
+> ⚠️ Do not call the `spk` bundle without `prompt_audios` — the speaker adapter expects a prompt at inference time; calling it without one leaves the decoder under-conditioned and audio quality drops.
+
+### Streaming (low-latency inference)
+
+Both bundles expose the same streaming API; the call signature differs only in whether you pass a speaker prompt and whether a transcript comes back.
+
+```python
+# Reconstruction streaming (base checkpoint)
+state = reconstructor.init_stream(chunk_size=8)
+for chunk in audio_chunks:
+    waveform_chunk, _, state = reconstructor.stream_chunk(chunk, state)
+
+# Voice-conversion streaming (spk checkpoint)
+state = voice_converter.init_stream(
+    chunk_size=8,
+    prompt_audio=target_speaker_audio,
+    return_text=True,  # optional incremental transcript
+)
+for chunk in audio_chunks:
+    waveform_chunk, text_chunk, state = voice_converter.stream_chunk(chunk, state)
+```
+
+See [`soviamate/bundles/codec.py`](https://github.com/samson-ailabs/SoviaMate/blob/main/soviamate/bundles/codec.py) for the full API.
+
+## Training data
+
+The released checkpoints were trained on publicly available English speech corpora (LibriHeavy and derivatives). Multilingual checkpoints are not yet available — contributions of multilingual training pipelines are welcome at the [project repository](https://github.com/samson-ailabs/SoviaMate).
+
+## Intended use
+
+- **Research** on neural audio codecs, speech LLMs, and end-to-end spoken dialogue systems.
+- **Educational** exploration of ASR-constrained codec training and zero-shot speaker adaptation.
+- **Engineering experimentation** as a building block for downstream speech-to-speech systems.
+
+## Out-of-scope / responsible-use note
+
+The post-quantization speaker adapter supports **zero-shot voice cloning** from a few seconds of reference audio. These weights **must not** be used for:
+- impersonation, fraud, or any form of non-consensual voice synthesis;
+- producing audio attributed to a real person without their explicit, informed consent;
+- deceptive, harassing, or otherwise harmful generation.
+
+Outputs may reflect biases in the training data. Users are responsible for compliance with applicable law and platform policies.
+
+## Limitations
+
+- English-only training data; performance on other languages is untested.
+- Preliminary checkpoint — comprehensive objective benchmarks (PESQ / ViSQOL / WER / SECS vs. EnCodec / SoundStream / DAC) have not yet been published.
+- Streaming inference is implemented (`init_stream` / `stream_chunk`) but has not yet been benchmarked end-to-end for production-grade latency or multi-session throughput.
+
+## License
+
+Apache License 2.0 — see [LICENSE](https://github.com/samson-ailabs/SoviaMate/blob/main/LICENSE).
+
+The speaker-verification weights under `speaker_verification/` are redistributed for convenience from their original authors; please consult and respect the licenses of those individual upstream projects (CAM++, ERes2Net-v2, WavLM, ECAPA-TDNN) when using or redistributing them.
+
+## Citation
+
+A technical report is in preparation. For now, please cite:
+
+```bibtex
+@misc{soviamate2026,
+  author       = {Son Dang Dinh (Samson)},
+  title        = {SoviaMate: Toward End-to-End Spoken Dialogue Systems},
+  year         = {2026},
+  howpublished = {\url{https://github.com/samson-ailabs/SoviaMate}},
+}
+```
+
+## Contact
+
+For research collaboration, dataset partnerships, or compute grants: **samson.ailabs@gmail.com** (subject line: `SoviaMate collaboration`). For code-level discussion, open an issue or discussion on the [GitHub repository](https://github.com/samson-ailabs/SoviaMate/issues).

neural_audio_codec/audio_codec_base.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb3650f96718e620e5f2cf37e676046c7274f07142723a2ba9fdbe04fdea3252
+size 747544365

neural_audio_codec/audio_codec_spk.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bf1c41f429cb58d46aa24e12246cba0c788a3362591c84b48f656f9379ba72fa
+size 984911111

speaker_verification/campplus.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3388cf5fd3493c9ac9c69851d8e7a8badcfb4f3dc631020c4961371646d5ada8
+size 28036335

speaker_verification/eres2netv2.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0eb4057106b2573dd7b132cf0c36273ab29afd192c1610f80baa9c556dbb963c
+size 71768231

speaker_verification/wavlm_ecapa.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:51f07e3b94d9e0262a6a675ef5a087be3dd09e8c62e9d886827f44f82fe7f94b
+size 1301926579