eeh-1 / README.md
ctrltokyo's picture
Card: front-end speed
9d94d59 verified
|
Raw History Blame Contribute Delete
12.2 kB
metadata
language: ja
license: apache-2.0
pipeline_tag: text-to-speech
library_name: eeh
tags:
  - text-to-speech
  - japanese
  - rust
  - webassembly
  - tiny

eeh

A tiny Japanese voice that fits in a browser tab. eeh is a Japanese text-to-speech model with two voices and 4.5 million parameters, waveform generator included. It reads kanji, numbers and Tokyo pitch accent, and runs on a CPU or fully inside a browser, with nothing sent anywhere.

There is no neural vocoder. A transformer reads OpenJTalk's phonemes with their pitch-accent tones, predicts how long each one lasts, then predicts pitch, loudness and the settings of a small source-filter synthesizer, which makes the waveform from a pulse train and noise. That is why it is small and fast. It was trained from scratch on 18.6 hours of speech by Irodori-TTS v4.1 Small Yomi.

Parameters 4,534,219
Voices F1 (female, late 20s, calm and clear), M1 (male, early 30s, warm), Tokyo Japanese
Output 24 kHz mono
Input Japanese text (kanji, kana, numbers), read with OpenJTalk
Speed 120x real time on one CPU thread, 280x on four (Apple M4 Max); about 50x in a browser tab. First sound after 9 ms (median) on the CPU, 30–50 ms in a browser
Pretrained weights in the model None
License Apache-2.0

Samples

Text F1 M1
こんにちは!これは、ノートパソコンでも動く小さな音声合成モデルです。
会議は三月十四日の午後三時半に変わったので、あと四十五分ほどあります。
本当に、あなたの誕生日を忘れると思ったの?
彼女は海を見たことがなかった。しばらくの間、何も言わずに波打ち際に立っていた。
恐れ入りますが、お名前とご連絡先をこちらにご記入いただけますか。

Files

Path Contents
eeh.safetensors Weights, float32 (18.2 MB)
eeh-dict.bin The Japanese dictionary the front end reads (36 MB)
python/ Python wheels for macOS (arm64) and Linux (aarch64, x86_64)
bin/<platform>/eeh Command-line program
lib/<platform>/, include/eeh.h C library and header, for any language with a C FFI
demo/ The browser demo: the whole engine compiled to WebAssembly, the weights and the dictionary
samples/ The audio on this page
eval/report.json, eval/metrics.jsonl Evaluation report and the training log
LICENSE Apache-2.0

Usage

eeh runs on its own engine, written in Rust: a Japanese front end that reads text exactly as OpenJTalk does (identical on more than 500,000 test texts) and the model. The same engine is behind the Python package, the command-line program, the C library and the browser demo.

hf download ctrltokyo/eeh-1 --local-dir eeh
cd eeh

Python

pip install python/eeh-0.1.0-py3-none-<your platform>.whl
import eeh, soundfile

synth = eeh.Synthesizer("eeh.safetensors", "eeh-dict.bin")
audio = synth.speak("こんにちは。今日はいい天気ですね。", voice="F1")   # float32 array, 24 kHz
soundfile.write("hello.wav", audio, synth.sample_rate)

speak takes voice (F1 or M1), speed (1.0), pitch_shift (0; 0.058 is about one semitone), pitch_range (1.0) and seed. speak_each(text, callback) hands over each sentence as soon as it is made, for streaming. The package has no dependencies; it returns a NumPy array when NumPy is installed. The synthesizer can be shared between threads.

Command line

chmod +x bin/macos-arm64/eeh     # the Hub does not keep the executable bit
bin/macos-arm64/eeh speak --weights eeh.safetensors --dictionary eeh-dict.bin \
  --voice M1 --text "こんにちは。今日はいい天気ですね。" --out hello.wav

C and other languages

lib/<platform>/libeeh_ffi with include/eeh.h: eeh_synthesizer_new takes the weights and dictionary as bytes, eeh_speak returns the audio and eeh_speak_each streams it sentence by sentence. Errors come back as status codes, never as crashes. Go, Node, Java, Ruby and Swift can call it through their C FFI (all five are tested).

Text of any length works. It is split into sentences at 。!? and read one sentence at a time; a quote followed by と or って stays with its sentence, and a sentence too long for one call is cut after 、. A text without a final mark gets 。. The audio never peaks above 0.99.

In a browser

python -m http.server --directory demo 8000

Then open http://localhost:8000. The engine runs in a Web Worker and plays long text sentence by sentence as it is made. The first visit downloads about 26 MB (gzip): 17 MB of weights, 9 MB of dictionary and 0.4 MB of program, all kept by the browser afterwards.

How it works

Front end. eeh's own Rust front end reads the text the way OpenJTalk does with the NAIST Japanese Dictionary: it splits it into words, picks each kanji's reading, expands numbers, joins words into accent phrases and assigns their accent types. It turns the result into tokens:

  • one token per phoneme (k, sh, a, N for ん, cl for っ); a long vowel is two tokens, one per mora;
  • devoiced vowels (the u of です, the i of した) are their own tokens A I U E O;
  • <sp> marks a boundary between accent phrases, and punctuation is kept;
  • each token carries the Tokyo pitch-accent tone of its mora, low or high, computed from the phrase's accent type and the mora's position.

It is fast: the 8,650 training sentences, 11.8 hours of speech, are read in 0.29 s on one CPU thread (0.03 ms per sentence, about 150,000x real time), 5.5x faster than pyopenjtalk. Its dictionary loads in 3 ms.

Model. A transformer encoder (4 blocks) reads the tokens and tones, a predictor gives each token a length in frames, and two frame-level transformers (2 and 5 blocks) predict pitch, voicing and loudness and then the settings of the synthesizer: a band-limited pulse train at the predicted pitch plus noise, each shaped by a filter predicted every frame (256 samples, 10.7 ms). All blocks use 192 channels, 4 heads and rotary position embeddings.

Component Parameters
Embeddings (token, tone, voice) 13,632
Text encoder 1,588,864
Duration and pause predictor 123,906
Frame transformer A, pitch/voicing/energy head 795,971
Frame transformer B 1,985,984
Source-filter waveform head 25,862
Total 4,534,219

Training data

The speech is synthetic. Two voices were designed once from Japanese descriptions with Irodori-TTS v4.1 Small (落ち着いた20代後半の女性。明瞭で自然な東京の話し言葉。 and 30代前半の男性。温かく親しみやすい、自然な東京の話し言葉。), then every sentence was read by Irodori-TTS v4.1 Small Yomi (MIT) cloning those two references. No recording of a real speaker is involved.

On the 300 test sentences in both voices, the teacher's speech has 1.7% kana reading error, 0.70 pitch-accent agreement and a UTMOS of 3.38.

The 8,650 sentences (8,000 train, 350 validation, 300 test, split by sentence) were chosen to cover as many morae and mora pairs as possible; 25% are questions and 10% exclamations.

Every clip was screened before training:

  • Reading. kana-whisper transcribed it to katakana, compared with OpenJTalk's reading; more than 10% kana error was rejected.
  • Alignment. Phoneme durations come from forced alignment with facebook/wav2vec2-lv-60-espeak-cv-ft, used only to label the data. Its labels were recalibrated on the teacher's audio, and clips it aligned with low confidence were rejected.
  • Speaking rate and length.
Clips Hours
Generated (sentences x voices) 17,300
Passed screening 15,684
Used for training (F1 7,301, M1 6,783) 14,084 18.6
Validation / test 614 / 527 1.3 / 1.0

Training

Steps 104,000
Batches about 9,000 frames (96 s of audio) per step
Losses Multi-resolution spectral losses; adversarial loss with multi-period and multi-resolution discriminators after step 8,000
Hardware One NVIDIA GB10 (Lenovo ThinkStation PGX) for training; an Apple M4 Max MacBook generated and scored half of the corpus
Time 11.5 hours of training; 5.7 hours to generate the corpus on both machines

Two changes to the training step made the adversarial phase twice as fast without changing the gradients: the discriminators' gradient at the waveform is computed once instead of twice, and their convolutions run in bfloat16 with autotuned kernels, on batches padded to a few fixed shapes.

Evaluation

Speech from the final checkpoint was scored the same way as the teacher's recordings of the same sentences: kana-whisper transcribes it to katakana (reading error against OpenJTalk's reading), the pitch tracker checks each expected rise and fall of the pitch accent, and UTMOS rates naturalness.

Sentences Clips Reading error, eeh Reading error, teacher Pitch-accent agreement, eeh Pitch-accent agreement, teacher UTMOS, eeh UTMOS, teacher
Test 300 0.8% 1.2% 0.70 0.69 2.94 3.38
Validation 300 1.0% 1.7% 0.71 0.68 2.94 3.34
New sentences, not in the dataset 16 0.0% 0.69 2.88

eeh reads and accents as well as the teacher it learned from; it sounds less natural (UTMOS about 0.4 lower), which is the cost of the small synthesizer. No listening test has been run.

Limitations

  • Japanese only, two fixed voices, no voice cloning.
  • Readings come from OpenJTalk's dictionary. Rare kanji readings, names and new words can be read wrong, and the accent of compound phrases is predicted by rules. Whatever OpenJTalk gets wrong, eeh says wrong.
  • Latin letters are spelled out one by one (eeh is read イー・イー・エイチ).
  • The same text is always read the same way.
  • The training data is synthetic, so eeh cannot sound more natural than the teacher; the small source-filter synthesizer sounds more buzzy than it.

License

Apache-2.0 for the weights and code.

Citation

@software{nicholson2026eeh,
  author = {Nicholson, Alexander},
  title  = {eeh: a 4.5M-parameter two-voice Japanese text-to-speech model},
  year   = {2026},
  url    = {https://huggingface.co/ctrltokyo/eeh-1}
}

Acknowledgements

Inspired by tiny-voca, trained on speech from Irodori-TTS, and reading Japanese with OpenJTalk.