| ---
|
| pretty_name: thelarryproject larry-speech-encoder v1
|
| license: cc-by-4.0
|
| language: [ en ]
|
| pipeline_tag: automatic-speech-recognition
|
| library_name: larry
|
| datasets:
|
| - thelarryproject/scotus-index-asr
|
| - thelarryproject/vctk-normalized-asr
|
| - thelarryproject/librispeech-asr-index-asr-validation
|
| - thelarryproject/libriheavy-large-index-asr
|
| - thelarryproject/libriheavy-test-clean-large-normalized-asr
|
| - thelarryproject/libriheavy-test-other-large-normalized-asr
|
| - thelarryproject/hifi-tts-all-normalized-asr
|
| - thelarryproject/lj-speech-normalized-asr
|
| - thelarryproject/common-voice-27-index-asr
|
| ---
|
|
|
| # thelarryproject larry-speech-encoder v1
|
|
|
| 
|
|
|
| The first model trained under thelarryproject in phase 1, larry-speech-encoder is primarily a speech unit ID encoder
|
| with a CTC proxy task head for training speech feature alignment.
|
|
|
| As you can see from the above diagram, this model was trained with only PD, CC0, or CC-BY (no ND, NC, or SA) data. This
|
| data was additionally never processed, filtered, or evaluated using a model that was trained on anything other than
|
| PD, CC0, or CC-BY data. Essentially, this model is verifiably both by legal requirements and intent requirements to not
|
| be a derivative of *ANY* NC, ND, SA, or similarly licensed content.
|
|
|
| Note: explicit consent for AI training specifically *was not* obtained from the original license holders, and
|
| PD/CC0/CC-BY is *assumed* consent.
|
|
|
| This model features in order:
|
|
|
| - a log mel frontend
|
| - conv subsampling frontend to downsample input 100hz mel features to 25hz (4x downsampling)
|
| - an encoder1 stack of 6 non-causally attending SAN-M style transformer encoder layers
|
| - contains another subsampling merge down to 12.5hz after encoder layer 3
|
| - SAN-M: uses FSMN attention layers instead of traditional MHA; uses a convolutional memory path in addition to MHA
|
| - FSMN uses kernel size of 31 prior to the merge and 15 after the merge
|
| - an FSQ layer for quantizing into a fixed set of codes
|
| - codebook size for this model is 15360
|
| - an encoder2 stack of 6 more non-causally attending traditional (MHA) transformer encoder layers
|
| - a CTC head over the vocabulary of the packaged SentencePiece tokenizer (plus a blank token) trained specifically on
|
| the same weighted distribution of data that the model trains on
|
|
|
| For a total of `163,618,694` parameters.
|
|
|
| The codes output by the FSQ layer are the real reason for this model to exist, and the CTC head is not intended to be
|
| used for real inference for CTC alignment. Post-quantization CTC alignment is not the intended task for this model, with
|
| it only achieving ~5.52% WER (using greedy decoding and the whisper standard for transcript normalization on the model
|
| output and ground-truth labels) on the validation set. CTC is only a proxy task, and using this model for plain ASR or
|
| CTC alignment is not supported explicitly, as the quantization layer likely hampers performance measurably. As yet, no
|
| study has been done on this specific model for those effects.
|
|
|
| The unit ID vocabulary is meant to encode speech features required for speech
|
| reconstruction by a downstream decoder that takes unit IDs, speaker representation, F0/harmonics conditioning, and
|
| optionally prepended sequence conditioning to produce plausible speaker-cloned speech.
|
|
|
| This model's code is an approximation of the code used in CosyVoice2 to train a similar speech tokenizer, but is not
|
| directly copied from them. Their code is licensed as apache-2.0 at https://github.com/QwenAudio/CosyVoice.
|
|
|
| ## Construction
|
|
|
| This model used the following training and validation splits, which is configured in more detail in the model's own
|
| config yamls in the repo for thelarryproject at https://github.com/Mekadrom/larry:
|
|
|
| | Dataset | Training Hours | Validation Hours |
|
| |------------------------|----------------|------------------|
|
| | libriheavy-large | 37,000 | 166.8 |
|
| | Common Voice 27.0 | 1,800 | 25.4 |
|
| | thelarryproject/scotus | 1,100 | 1.8 |
|
| | VCTK | 39.3 | 2 |
|
| | LJSpeech | 23.7 | N/a |
|
| | HiFi-TTS v1 | 289.8 | 0.6 |
|
| | librispeech-asr | N/a | 10.1 |
|
| | Total | 40,252.8 | 206.7 |
|
|
|
| Where CV27, scotus, vctk, and librispeech are all speaker-disjoint from their train splits (where a train split exists).
|
| The Libriheavy and HiFi-TTS validation splits share speakers with their train splits.
|
|
|
| ## Train Statistics
|
|
|
| - Tokens seen: 3,658,563,934 (~3.7B) CTC SentencePiece vocab tokens
|
|
|
| ## Eval Statistics
|
|
|
| - wer_normalized (whisper-style transcript normalization on ground-truth labels and model greedy CTC decoding outputs)
|
| - Weighted average of all datasets: ~5.52%
|
| - wer_normalized/cv27: ~13.4%
|
| - wer_normalized/hifi-tts-all: ~0.510%
|
| - wer_normalized/libriheavy-test-clean-large: ~3.39%
|
| - wer_normalized/libriheavy-test-other-large: ~6.56%
|
| - wer_normalized/librispeech-asr: ~5.00%
|
| - wer_normalized/scotus: ~4.33%
|
| - wer_normalized/vctk: ~1.27%
|
| - measures CTC proxy task performance only
|
| - Codebook (FSQ) perplexity: 5827
|
| - intended to represent how rich the vocabulary/codebook space is; a measure of how diffuse the codebook space is by
|
| approximate frequency of occurrence. For example, if every code was used the exact same amount, the perplexity
|
| would be 15360. If only one single code was ever used, the perplexity would be 1. This is different from LLM
|
| perplexity, though they are both proxy metrics for model health and represent exp(entropy) of the output space.
|
| - Codebook usage: 0.999
|
| - 99.9% of FSQ codes are seen in a full pass of the validation set; 99.9% of 15360 is 15345, leaving ~15 codes that
|
| happened to not appear in the FSQ layer's output at eval.
|
|
|
| ## Citation
|
|
|
| If you use this model, please cite the model that it is based on as well:
|
|
|
| ```
|
| @article{du2024cosyvoice,
|
| title={Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens},
|
| author={Du, Zhihao and Chen, Qian and Zhang, Shiliang and Hu, Kai and Lu, Heng and Yang, Yexin and Hu, Hangrui and Zheng, Siqi and Gu, Yue and Ma, Ziyang and others},
|
| journal={arXiv preprint arXiv:2407.05407},
|
| year={2024}
|
| }
|
|
|
| @article{du2024cosyvoice2,
|
| title={Cosyvoice 2: Scalable streaming speech synthesis with large language models},
|
| author={Du, Zhihao and Wang, Yuxuan and Chen, Qian and Shi, Xian and Lv, Xiang and Zhao, Tianyu and Gao, Zhifu and Yang, Yexin and Gao, Changfeng and Wang, Hui and others},
|
| journal={arXiv preprint arXiv:2412.10117},
|
| year={2024}
|
| }
|
|
|
| @article{du2025cosyvoice,
|
| title={CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training},
|
| author={Du, Zhihao and Gao, Changfeng and Wang, Yuxuan and Yu, Fan and Zhao, Tianyu and Wang, Hao and Lv, Xiang and Wang, Hui and Shi, Xian and An, Keyu and others},
|
| journal={arXiv preprint arXiv:2505.17589},
|
| year={2025}
|
| }
|
|
|
| @inproceedings{lyu2025build,
|
| title={Build LLM-Based Zero-Shot Streaming TTS System with Cosyvoice},
|
| author={Lyu, Xiang and Wang, Yuxuan and Zhao, Tianyu and Wang, Hao and Liu, Huadai and Du, Zhihao},
|
| booktitle={ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
|
| pages={1--2},
|
| year={2025},
|
| organization={IEEE}
|
| }
|
| ```
|
|
|
| As well as thelarryproject:
|
|
|
| ```
|
| @misc{larry-speech-encoder-2026,
|
| author = {{thelarryproject}},
|
| title = {larry-speech-encoder: a 12.5 Hz FSQ speech encoder},
|
| year = {2026},
|
| publisher = {Hugging Face},
|
| howpublished = {\url{https://huggingface.co/thelarryproject/larry-speech-encoder}},
|
| note = {Phase 1 release}
|
| }
|
| ```
|
|
|
| Training pipeline code is at https://github.com/Mekadrom/larry, and this model is fully reproducible (RNG state
|
| permitting) with a single command in that repo given the following presumptions:
|
|
|
| - A preprocessed normalized Common Voice 27.0 distribution stored locally
|
| - See [thelarryproject/common-voice-27-index-asr](https://huggingface.co/datasets/thelarryproject/common-voice-27-index-asr)
|
| for the command to create this and optionally the index locally
|
| - Configure the training script to point at these datasets
|
| - The provided `larry-ctc-sp512.model` as `tokenizer_model_path`
|
| - a speaker mapping from `uv run speaker-mapping` as `speaker_mapping_path`
|
|
|
| ```
|
| uv run training -f configs/training/voice/ctc/larry-speech-encoder.yaml -f configs/training/voice/mixtures/asr.yaml
|
| ```
|
|
|
| This works best on commits later than the one on `provenance.json`, as the default config values as of that commit have
|
| been replaced with the values that produced the model distributed here. The commit that changed the default values is
|
| `3b12f6b`.
|
|
|
| ## Usage
|
|
|
| Input is mono 16khz audio waveforms, and the output is one unit ID in the range 0-15,359 per 80 ms of input audio.
|
|
|
| Install the larry package locally: `pip install git+https://github.com/Mekadrom/larry` (requires Python 3.14+)
|
|
|
| ```python
|
| from larry.voice.model.speech_encoder import SpeechEncoderModel
|
|
|
| model = SpeechEncoderModel.from_pretrained("thelarryproject/larry-speech-encoder")
|
|
|
| # assuming waveforms is a tensor of [batch, samples] and waveform_lengths is a tensor of [batch]
|
| unit_ids, unit_lengths = model.tokenize(waveforms, waveform_lengths)
|
| ```
|
|
|
| ## Other
|
|
|
| - Speaker information in the codes: linear and non-linear probes trained to extract speaker identity on
|
| the static set of speakers in the training corpus only achieve 14.6% and 12.7% validation accuracy respectively when
|
| receiving the codes as input.
|
| - When receiving the raw features prior to the quantization, the same linear and non-linear probes achieve 93.3% and
|
| 89.1% speaker identification accuracy respectively, and this drop is taken to mean that quantization removes most
|
| speaker information.
|
| - Trained on 233 speakers, chance is 0.43%, so some speaker leakage still occurs.
|
| - Per-dataset codebook usage is more nuanced than 99.9% of codes being seen at eval:
|
| - ~98% for CV27
|
| - ~73% for VCTK
|
| - ~50% for hifi-tts
|
| - this seems to mostly be an artifact of the size of these validation datasets
|
| |
