fiato
A tiny voice that knows how to sigh. fiato is Italian for breath. It is an English text-to-speech model with two voices and 4.5 million parameters, waveform generator included: 727 times smaller than Maya-1, the model it learned from by listening to 23.7 hours of its speech.
Dataset | Browser demo | Training script | Cost of every run
| Parameters | 4,537,867 |
| Voices | F1 (female), M1 (male), American English |
| Expressions | <sigh>, <exhale>, <gasp> |
| Output | 24 kHz mono |
| Speed | About 150x real time on 4 CPU threads of an Apple M3 Max laptop |
| Total cost to make | $18.27 |
| License | Apache-2.0 |
Samples
| Text | F1 | M1 |
|---|---|---|
| Hello! This is fiato, a small text to speech model that runs on a laptop. | ||
| The meeting moved to 3:30 pm on March 14th, so we have about 45 minutes left. | ||
| Did you really think I would forget your birthday? | ||
| She had never seen the sea before, and for a long while she stood at the edge of the water without saying a word. | ||
Well, <sigh> I guess we should start over from the beginning. |
||
Wait, <gasp> you are telling me the tickets were free? |
||
That was the last box, <exhale> and now we can finally sit down. |
Usage
pip install git+https://github.com/n4ze3m/dizione torch soundfile
hf download cfahlgren1/fiato --local-dir fiato-model
export DIZIONE_DATA=fiato-model/phonemizer FIATO_FRONTEND=own PYTHONPATH=fiato-model
python -m fiato.infer \
--checkpoint fiato-model/fiato.pt \
--voice F1 \
--text "Well, <sigh> I guess we should start over." \
--output hello.wav
| Option | Default | Meaning |
|---|---|---|
--voice |
F1 |
F1 or M1 |
--speed |
1.0 | Speaking rate |
--pitch-shift |
0 | Added to log F0. 0.058 is about one semitone. |
--pitch-range |
1.0 | Scales pitch movement around the utterance mean. |
Write an expression where the sound should be: after a comma, at the start of a sentence or at its end. Text of any length works; it is read one sentence at a time.
onnx/model.onnx is the same model for ONNX Runtime. It takes phoneme ids, which fiato.text.phonemize
produces; config.json lists the vocabulary, the voices and the graph's inputs.
Try it in your browser
The demo runs the model in a browser tab, with the dataset's clips to compare against. Nothing you type is sent anywhere.
The page is in this repository under demo/, so it also runs from a download:
hf download cfahlgren1/fiato --local-dir fiato-model
python -m http.server --directory fiato-model 8000
Then open http://localhost:8000/demo/.
How it works
Text becomes phonemes with a stress mark each. A transformer encoder reads them, a predictor gives each one a length, and two more transformers predict pitch, loudness and the settings of a small source-filter synthesizer that makes the sound: a train of pulses at the predicted pitch plus noise, each shaped by a filter. There is no neural vocoder, which is why the model is small and fast.
An expression such as <sigh> is one more token between the words. In the training data, the
sound between the silences around it belongs to that token, so the model learns how long the sound
lasts and what it sounds like.
The design and the training code are a fork of tiny-voca by Muhammed Nazeem. What is new here is the teacher, the two voices and the expression tokens.
Training
| Data | cfahlgren1/fiato: 20,665 clips, 25.7 hours |
| Used for training | 19,086 clips, 23.7 hours, after alignment checks |
| Steps | 104,000 |
| Hardware | One Nvidia L40S on Hugging Face Jobs |
| Time | 6.2 hours of training, 1.1 hours preparing the data |
| Losses | Multi-resolution spectral losses; adversarial loss after step 8,000 |
Phoneme lengths come from forced alignment with facebook/wav2vec2-lv-60-espeak-cv-ft, which is
used only to label the training data. No pretrained weights are part of the model.
To train it again, run train_job.py on Hugging Face Jobs:
hf jobs uv run --flavor l40sx1 --timeout 12h --secrets HF_TOKEN train_job.py \
--dataset cfahlgren1/fiato --model cfahlgren1/fiato
generate.py makes the dataset, and costs.py adds up the cost of every job in jobs.json.
Evaluation
Word error rate of the model's speech, transcribed with Whisper small.en. The teacher's recordings of the same sentences are scored the same way.
| Sentences | Clips | fiato | Teacher recordings |
|---|---|---|---|
| Validation | 8 | 0.0% | 0.0% |
| Test | 8 | 0.6% | 0.0% |
| New sentences, not in the dataset | 16 | 1.0% |
These sets are small: a sanity check, not a benchmark. No listening test has been run.
Are the expressions there?
The audio event classifier that screened the dataset, run on the samples above. It gives the probability that the expression's sound is in the clip.
| Sample | F1 | M1 |
|---|---|---|
<sigh> |
0.146 | 0.062 |
<exhale> |
0.228 | 0.051 |
<gasp> |
0.167 | 0.014 |
| Plain samples, highest | 0.006 | 0.004 |
The teacher's clips in the dataset score 0.24 to 0.49 on average, so the model's expressions are there but fainter than Maya-1's, and fainter in the male voice, where the gasp scores little above plain speech. The model gives a sigh or an exhale about 1.9 seconds and a gasp about 1.3, pauses around it included.
Cost
$18.27 of GPU time on Hugging Face Jobs, counting every run including failed tries.
| Stage | Cost |
|---|---|
| Voice spike | $0.30 |
| Dataset | $3.83 |
| Expression spike | $0.62 |
| Training smoke test | $0.32 |
| Training | $13.20 |
| Total | $18.27 |
costs.md lists every job.
Limitations
- English only, two fixed voices, no voice cloning.
- Three expressions. Laughs are not supported: the teacher did not render them reliably.
- The same text is always read the same way.
- The training data is synthetic, so the model cannot sound more natural than Maya-1 does.
- Foreign names and unusual spellings are often mispronounced.
- No streaming.
License
Apache-2.0 for the weights and code. fiato/ is a fork of tiny-voca (Apache-2.0); see
fiato/NOTICE. The files in phonemizer/ come from dizione and the CMU Pronouncing Dictionary;
their licenses are next to them. This project is not affiliated with Maya Research.
- Downloads last month
- 216

