wav2vec2-lv-60-espeak-cv-ft (fp16 ONNX) + CMU Pronouncing Dictionary, mirrored for Rostrum

Meta's wav2vec2 phoneme recogniser fine-tuned on CommonVoice (espeak phones), as the fp16 ONNX export by onnx-community, with its vocabulary; plus the CMU Pronouncing Dictionary (cmudict-7479086/cmudict.dict) for English spellings.

This repository is a pinned, public copy of the exact files the Rostrum apps download. Rostrum fetches each file at a fixed commit of this repository and checks its SHA-256 before use.

What Rostrum uses it for

2D's lip sync: it hears the speech sounds in a recorded voice and when each is said, so the mouth shapes are keyed to them. It only listens.

Rostrum uses it for analysis only, never generation: it reads what is in a recording or a picture and makes no new sound, voice or image.

Files

File Bytes SHA-256
model_fp16.onnx 632,331,247 7ff53b55abe98dc26da93f445e39d33b95491d4f0292e1fef5bcbfa40aa22932
vocab.json 4,637 d732ab2456c0c017930001dc9af0b41b3b93d25b2eb9740bf9d925508d7d87d0
cmudict-7479086/cmudict.dict 3,618,488 81917843c7f44ce2b094ac63873c2c7a4cf802040792c455ba3ca406891c3d22

In Rostrum's models folder they go under wav2vec2-lv60-espeak-fp16/ (and cmudict-7479086/ for the dictionary).

Source

What Where Commit / version
Exact model files (unchanged copy) https://huggingface.co/onnx-community/wav2vec2-lv-60-espeak-cv-ft-ONNX c69750f5043e5e1f8a71ab95dd3b98338c280c92 (onnx/model_fp16.onnx, vocab.json)
Original model, Meta https://huggingface.co/facebook/wav2vec2-lv-60-espeak-cv-ft ae45363bf3413b374fecd9dc8bc1df0e24c3b7f4 (main when mirrored)
CMU Pronouncing Dictionary (unchanged copy) https://github.com/cmusphinx/cmudict 74790861f652b15e4ac49015a90074ad62a27690

Authors

Qiantong Xu, Alexei Baevski, Michael Auli (Meta AI, "Simple and Effective Zero-shot Cross-lingual Phoneme Recognition", arXiv:2109.11680); onnx-community (ONNX export); Carnegie Mellon University (CMU Pronouncing Dictionary).

Licence

The model and its export are Apache-2.0 (LICENSE). The CMU Pronouncing Dictionary is BSD-2-Clause, Copyright (C) 1993-2015 Carnegie Mellon University (cmudict-7479086/LICENSE). Files redistributed unchanged.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for rostrumstudios/lipsync-phones

Quantized
(10)
this model

Collection including rostrumstudios/lipsync-phones

Paper for rostrumstudios/lipsync-phones