| --- |
| license: mit |
| tags: |
| - lip-reading |
| - visual-speech-recognition |
| - vsr |
| - silent-speech |
| - auto-avsr |
| library_name: pytorch |
| pipeline_tag: video-classification |
| --- |
| |
| # Silent Lip Reader — VSR weights |
|
|
| The visual-speech-recognition (lip-reading) model weights used by the |
| [**Silent Lip Reader** Space](https://huggingface.co/spaces/aaahmet/silent-lip-reader). |
| Re-hosted here so that the open-source Space is **self-contained** and does not break |
| if upstream repos move. |
|
|
| - **Architecture:** Auto-AVSR — ResNet-3D + Conformer encoder, Transformer decoder, |
| joint CTC/attention. Input: 88×88 grayscale mouth crops @ 25fps. Output: text via a |
| 5000-unit SentencePiece (`unigram5000`) vocabulary. **Video-only** (no audio path). |
| - **Files:** `pytorch_model.pt` (state dict), `unigram5000.model`, `unigram5000_units.txt`. |
|
|
| ## Credits / provenance (please read) |
|
|
| This checkpoint is **not** trained by the re-host. Honest attribution: |
|
|
| - **Model architecture & training:** **Auto-AVSR** (Pingchuan Ma et al., *"Auto-AVSR: |
| Audio-Visual Speech Recognition with Automatic Labels"*). All model credit to the |
| original authors. |
| - **Checkpoint source:** mirrored from `AD1TEYA/lip-reading-model` on the Hub. |
| - **Re-host + the surrounding system, demo, visual-VAD pipeline, evaluation and research:** |
| **[Ahmet Dedeler](https://ahmetdedeler.com)** ([🤗 aaahmet](https://huggingface.co/aaahmet)). |
|
|
| ## Intended use |
|
|
| Research and demos of silent visual speech recognition. The weights were trained on |
| LRS3-derived data; treat as **research use**. Best on clear, frontal, well-articulated |
| English. ~25–30% WER on clean speech, higher on casual speech (lip reading is inherently |
| ambiguous — many phonemes look identical on the lips). |
|
|
| ## Usage |
|
|
| Used by the Silent Lip Reader Space — record a (silent) video, it crops your mouth, |
| chunks utterances by lip motion, and decodes text. See the |
| [Space](https://huggingface.co/spaces/aaahmet/silent-lip-reader) for the full pipeline |
| and research log. |
|
|
| --- |
| Built / curated by **Ahmet Dedeler** — https://ahmetdedeler.com. A credit/link back is |
| appreciated if you use this. License MIT (follows the upstream Space). |
|
|
