---
language:
  - en
license: cc-by-nc-4.0
tags:
  - ultrasound
  - silent-speech-interface
  - speech-synthesis
  - mel-spectrogram
  - pytorch
  - tal-corpus
datasets:
  - TaL80
library_name: pytorch
pipeline_tag: other
---

# ult2mel_2DCNN — Ultrasound-to-Mel (TaL80, 4 speakers)

Speaker-dependent 2D CNN that maps a single ultrasound (ULT) tongue-imaging frame to one 80-band log-mel frame. This repo bundles **four per-speaker checkpoints** trained on the TaL80 corpus.

## Files

One checkpoint + scaler pair per speaker.

```
01fi/model.ckpt   01fi/scaler.pkl
02fe/model.ckpt   02fe/scaler.pkl
03mn/model.ckpt   03mn/scaler.pkl
04me/model.ckpt   04me/scaler.pkl
```

## Usage

For training, inference, and a HuggingFace-tailored predict script, see the GitHub repository:

**[`ibrahimkhaliloglu/ult-to-speech-pytorch`](https://github.com/ibrahimkhaliloglu/ult-to-speech-pytorch)**

## Intended use & limitations

Research baseline for ultrasound-to-speech conversion and silent speech interfaces; downstream vocoders (e.g. HiFi-GAN) can synthesize audio from the predicted mel.

- **Speaker-dependent** — each checkpoint works only for its own speaker.
- **Frame-wise** — no temporal context modeled.

## License

The TaL80 corpus has its own license - users must comply with it.

## Citation

```bibtex
@inproceedings{ibrahimov25_interspeech,
  title     = {{Conformer-based Ultrasound-to-Speech Conversion}},
  author    = {Ibrahim Ibrahimov and Csaba Zainkó and Gábor Gosztolya},
  year      = {2025},
  booktitle = {{Interspeech 2025}},
  pages     = {5578--5582},
  doi       = {10.21437/Interspeech.2025-2147},
  issn      = {2958-1796},
}
```

Contact: Ibrahim Ibrahimov — <ibrahimkhaliloglu@gmail.com>