| --- |
| language: |
| - en |
| license: cc-by-nc-4.0 |
| tags: |
| - ultrasound |
| - silent-speech-interface |
| - speech-synthesis |
| - mel-spectrogram |
| - pytorch |
| - tal-corpus |
| datasets: |
| - TaL80 |
| library_name: pytorch |
| pipeline_tag: other |
| --- |
| |
| # ult2mel_2DCNN — Ultrasound-to-Mel (TaL80, 4 speakers) |
| |
| Speaker-dependent 2D CNN that maps a single ultrasound (ULT) tongue-imaging frame to one 80-band log-mel frame. This repo bundles **four per-speaker checkpoints** trained on the TaL80 corpus. |
| |
| ## Files |
| |
| One checkpoint + scaler pair per speaker. |
| |
| ``` |
| 01fi/model.ckpt 01fi/scaler.pkl |
| 02fe/model.ckpt 02fe/scaler.pkl |
| 03mn/model.ckpt 03mn/scaler.pkl |
| 04me/model.ckpt 04me/scaler.pkl |
| ``` |
| |
| ## Usage |
| |
| For training, inference, and a HuggingFace-tailored predict script, see the GitHub repository: |
| |
| **[`ibrahimkhaliloglu/ult-to-speech-pytorch`](https://github.com/ibrahimkhaliloglu/ult-to-speech-pytorch)** |
| |
| ## Intended use & limitations |
| |
| Research baseline for ultrasound-to-speech conversion and silent speech interfaces; downstream vocoders (e.g. HiFi-GAN) can synthesize audio from the predicted mel. |
| |
| - **Speaker-dependent** — each checkpoint works only for its own speaker. |
| - **Frame-wise** — no temporal context modeled. |
| |
| ## License |
| |
| The TaL80 corpus has its own license - users must comply with it. |
| |
| ## Citation |
| |
| ```bibtex |
| @inproceedings{ibrahimov25_interspeech, |
| title = {{Conformer-based Ultrasound-to-Speech Conversion}}, |
| author = {Ibrahim Ibrahimov and Csaba Zainkó and Gábor Gosztolya}, |
| year = {2025}, |
| booktitle = {{Interspeech 2025}}, |
| pages = {5578--5582}, |
| doi = {10.21437/Interspeech.2025-2147}, |
| issn = {2958-1796}, |
| } |
| ``` |
| |
| Contact: Ibrahim Ibrahimov — <ibrahimkhaliloglu@gmail.com> |
| |
