fiato

fiato

A tiny voice that knows how to sigh. fiato is Italian for breath. It is an English text-to-speech model with two voices and 4.5 million parameters, waveform generator included: 727 times smaller than Maya-1, the model it learned from by listening to 23.7 hours of its speech.

Dataset  |  Browser demo  |  Training script  |  Cost of every run

Parameters 4,537,867
Voices F1 (female), M1 (male), American English
Expressions <sigh>, <exhale>, <gasp>
Output 24 kHz mono
Speed About 150x real time on 4 CPU threads of an Apple M3 Max laptop
Total cost to make $18.27
License Apache-2.0

Samples

Text F1 M1
Hello! This is fiato, a small text to speech model that runs on a laptop.
The meeting moved to 3:30 pm on March 14th, so we have about 45 minutes left.
Did you really think I would forget your birthday?
She had never seen the sea before, and for a long while she stood at the edge of the water without saying a word.
Well, <sigh> I guess we should start over from the beginning.
Wait, <gasp> you are telling me the tickets were free?
That was the last box, <exhale> and now we can finally sit down.

Usage

pip install git+https://github.com/n4ze3m/dizione torch soundfile
hf download cfahlgren1/fiato --local-dir fiato-model
export DIZIONE_DATA=fiato-model/phonemizer FIATO_FRONTEND=own PYTHONPATH=fiato-model

python -m fiato.infer \
  --checkpoint fiato-model/fiato.pt \
  --voice F1 \
  --text "Well, <sigh> I guess we should start over." \
  --output hello.wav
Option Default Meaning
--voice F1 F1 or M1
--speed 1.0 Speaking rate
--pitch-shift 0 Added to log F0. 0.058 is about one semitone.
--pitch-range 1.0 Scales pitch movement around the utterance mean.

Write an expression where the sound should be: after a comma, at the start of a sentence or at its end. Text of any length works; it is read one sentence at a time.

onnx/model.onnx is the same model for ONNX Runtime. It takes phoneme ids, which fiato.text.phonemize produces; config.json lists the vocabulary, the voices and the graph's inputs.

Try it in your browser

The demo runs the model in a browser tab, with the dataset's clips to compare against. Nothing you type is sent anywhere.

The page is in this repository under demo/, so it also runs from a download:

hf download cfahlgren1/fiato --local-dir fiato-model
python -m http.server --directory fiato-model 8000

Then open http://localhost:8000/demo/.

How it works

Maya-1 and fiato drawn to scale by parameter count, with the dataset between them

Text becomes phonemes with a stress mark each. A transformer encoder reads them, a predictor gives each one a length, and two more transformers predict pitch, loudness and the settings of a small source-filter synthesizer that makes the sound: a train of pulses at the predicted pitch plus noise, each shaped by a filter. There is no neural vocoder, which is why the model is small and fast.

An expression such as <sigh> is one more token between the words. In the training data, the sound between the silences around it belongs to that token, so the model learns how long the sound lasts and what it sounds like.

The design and the training code are a fork of tiny-voca by Muhammed Nazeem. What is new here is the teacher, the two voices and the expression tokens.

Training

Data cfahlgren1/fiato: 20,665 clips, 25.7 hours
Used for training 19,086 clips, 23.7 hours, after alignment checks
Steps 104,000
Hardware One Nvidia L40S on Hugging Face Jobs
Time 6.2 hours of training, 1.1 hours preparing the data
Losses Multi-resolution spectral losses; adversarial loss after step 8,000

Phoneme lengths come from forced alignment with facebook/wav2vec2-lv-60-espeak-cv-ft, which is used only to label the training data. No pretrained weights are part of the model.

To train it again, run train_job.py on Hugging Face Jobs:

hf jobs uv run --flavor l40sx1 --timeout 12h --secrets HF_TOKEN train_job.py \
  --dataset cfahlgren1/fiato --model cfahlgren1/fiato

generate.py makes the dataset, and costs.py adds up the cost of every job in jobs.json.

Evaluation

Word error rate of the model's speech, transcribed with Whisper small.en. The teacher's recordings of the same sentences are scored the same way.

Sentences Clips fiato Teacher recordings
Validation 8 0.0% 0.0%
Test 8 0.6% 0.0%
New sentences, not in the dataset 16 1.0%

These sets are small: a sanity check, not a benchmark. No listening test has been run.

Are the expressions there?

The audio event classifier that screened the dataset, run on the samples above. It gives the probability that the expression's sound is in the clip.

Sample F1 M1
<sigh> 0.146 0.062
<exhale> 0.228 0.051
<gasp> 0.167 0.014
Plain samples, highest 0.006 0.004

The teacher's clips in the dataset score 0.24 to 0.49 on average, so the model's expressions are there but fainter than Maya-1's, and fainter in the male voice, where the gasp scores little above plain speech. The model gives a sigh or an exhale about 1.9 seconds and a gasp about 1.3, pauses around it included.

Cost

$18.27 of GPU time on Hugging Face Jobs, counting every run including failed tries.

Stage Cost
Voice spike $0.30
Dataset $3.83
Expression spike $0.62
Training smoke test $0.32
Training $13.20
Total $18.27

costs.md lists every job.

Limitations

  • English only, two fixed voices, no voice cloning.
  • Three expressions. Laughs are not supported: the teacher did not render them reliably.
  • The same text is always read the same way.
  • The training data is synthetic, so the model cannot sound more natural than Maya-1 does.
  • Foreign names and unusual spellings are often mispronounced.
  • No streaming.

License

Apache-2.0 for the weights and code. fiato/ is a fork of tiny-voca (Apache-2.0); see fiato/NOTICE. The files in phonemizer/ come from dizione and the CMU Pronouncing Dictionary; their licenses are next to them. This project is not affiliated with Maya Research.

Downloads last month
216
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train cfahlgren1/fiato

Space using cfahlgren1/fiato 1

Collection including cfahlgren1/fiato