|
Download README.md from ctrltokyo/eeh-1: direct link, hf CLI and curl.
- Browser
- Download file 12.2 kB
-
https://huggingface.co/ctrltokyo/eeh-1/resolve/main/README.md
- Command line
-
hf download hf://ctrltokyo/eeh-1/README.md
-
curl -L -o README.md https://huggingface.co/ctrltokyo/eeh-1/resolve/main/README.md
12.2 kB
| language: ja | |
| license: apache-2.0 | |
| pipeline_tag: text-to-speech | |
| library_name: eeh | |
| tags: | |
| - text-to-speech | |
| - japanese | |
| - rust | |
| - webassembly | |
| - tiny | |
| # eeh | |
| **A tiny Japanese voice that fits in a browser tab.** eeh is a Japanese | |
| text-to-speech model with two voices and 4.5 million parameters, waveform | |
| generator included. It reads kanji, numbers and Tokyo pitch accent, and runs on | |
| a CPU or fully inside a browser, with nothing sent anywhere. | |
| There is no neural vocoder. A transformer reads OpenJTalk's phonemes with their | |
| pitch-accent tones, predicts how long each one lasts, then predicts pitch, | |
| loudness and the settings of a small source-filter synthesizer, which makes the | |
| waveform from a pulse train and noise. That is why it is small and fast. It was | |
| trained from scratch on 18.6 hours of speech by | |
| [Irodori-TTS v4.1 Small Yomi](https://huggingface.co/takuma104/Irodori-TTS-v4.1-Small-Yomi). | |
| <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-hello.wav"></audio> | |
| | | | | |
| |---|---| | |
| | Parameters | 4,534,219 | | |
| | Voices | F1 (female, late 20s, calm and clear), M1 (male, early 30s, warm), Tokyo Japanese | | |
| | Output | 24 kHz mono | | |
| | Input | Japanese text (kanji, kana, numbers), read with OpenJTalk | | |
| | Speed | 120x real time on one CPU thread, 280x on four (Apple M4 Max); about 50x in a browser tab. First sound after 9 ms (median) on the CPU, 30–50 ms in a browser | | |
| | Pretrained weights in the model | None | | |
| | License | Apache-2.0 | | |
| ## Samples | |
| | Text | F1 | M1 | | |
| |---|---|---| | |
| | こんにちは!これは、ノートパソコンでも動く小さな音声合成モデルです。 | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-hello.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/M1-hello.wav"></audio> | | |
| | 会議は三月十四日の午後三時半に変わったので、あと四十五分ほどあります。 | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-numbers.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/M1-numbers.wav"></audio> | | |
| | 本当に、あなたの誕生日を忘れると思ったの? | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-question.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/M1-question.wav"></audio> | | |
| | 彼女は海を見たことがなかった。しばらくの間、何も言わずに波打ち際に立っていた。 | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-story.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/M1-story.wav"></audio> | | |
| | 恐れ入りますが、お名前とご連絡先をこちらにご記入いただけますか。 | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/F1-polite.wav"></audio> | <audio controls preload="none" src="https://huggingface.co/ctrltokyo/eeh-1/resolve/main/samples/M1-polite.wav"></audio> | | |
| ## Files | |
| | Path | Contents | | |
| |---|---| | |
| | `eeh.safetensors` | Weights, float32 (18.2 MB) | | |
| | `eeh-dict.bin` | The Japanese dictionary the front end reads (36 MB) | | |
| | `python/` | Python wheels for macOS (arm64) and Linux (aarch64, x86_64) | | |
| | `bin/<platform>/eeh` | Command-line program | | |
| | `lib/<platform>/`, `include/eeh.h` | C library and header, for any language with a C FFI | | |
| | `demo/` | The browser demo: the whole engine compiled to WebAssembly, the weights and the dictionary | | |
| | `samples/` | The audio on this page | | |
| | `eval/report.json`, `eval/metrics.jsonl` | Evaluation report and the training log | | |
| | `LICENSE` | Apache-2.0 | | |
| ## Usage | |
| eeh runs on its own engine, written in Rust: a Japanese front end that reads | |
| text exactly as OpenJTalk does (identical on more than 500,000 test texts) and | |
| the model. The same engine is behind the Python package, the command-line | |
| program, the C library and the browser demo. | |
| ```bash | |
| hf download ctrltokyo/eeh-1 --local-dir eeh | |
| cd eeh | |
| ``` | |
| ### Python | |
| ```bash | |
| pip install python/eeh-0.1.0-py3-none-<your platform>.whl | |
| ``` | |
| ```python | |
| import eeh, soundfile | |
| synth = eeh.Synthesizer("eeh.safetensors", "eeh-dict.bin") | |
| audio = synth.speak("こんにちは。今日はいい天気ですね。", voice="F1") # float32 array, 24 kHz | |
| soundfile.write("hello.wav", audio, synth.sample_rate) | |
| ``` | |
| `speak` takes `voice` (`F1` or `M1`), `speed` (1.0), `pitch_shift` (0; 0.058 | |
| is about one semitone), `pitch_range` (1.0) and `seed`. `speak_each(text, | |
| callback)` hands over each sentence as soon as it is made, for streaming. The | |
| package has no dependencies; it returns a NumPy array when NumPy is installed. | |
| The synthesizer can be shared between threads. | |
| ### Command line | |
| ```bash | |
| chmod +x bin/macos-arm64/eeh # the Hub does not keep the executable bit | |
| bin/macos-arm64/eeh speak --weights eeh.safetensors --dictionary eeh-dict.bin \ | |
| --voice M1 --text "こんにちは。今日はいい天気ですね。" --out hello.wav | |
| ``` | |
| ### C and other languages | |
| `lib/<platform>/libeeh_ffi` with `include/eeh.h`: `eeh_synthesizer_new` takes | |
| the weights and dictionary as bytes, `eeh_speak` returns the audio and | |
| `eeh_speak_each` streams it sentence by sentence. Errors come back as status | |
| codes, never as crashes. Go, Node, Java, Ruby and Swift can call it through | |
| their C FFI (all five are tested). | |
| Text of any length works. It is split into sentences at 。!? and read one | |
| sentence at a time; a quote followed by と or って stays with its sentence, and | |
| a sentence too long for one call is cut after 、. A text without a final mark | |
| gets 。. The audio never peaks above 0.99. | |
| ### In a browser | |
| ```bash | |
| python -m http.server --directory demo 8000 | |
| ``` | |
| Then open `http://localhost:8000`. The engine runs in a Web Worker and plays | |
| long text sentence by sentence as it is made. The first visit downloads about | |
| 26 MB (gzip): 17 MB of weights, 9 MB of dictionary and 0.4 MB of program, all | |
| kept by the browser afterwards. | |
| ## How it works | |
| **Front end.** eeh's own Rust front end reads the text the way OpenJTalk does | |
| with the NAIST Japanese Dictionary: it splits it into words, picks each kanji's | |
| reading, expands numbers, joins words into accent phrases and assigns their | |
| accent types. It turns the result into tokens: | |
| - one token per phoneme (`k`, `sh`, `a`, `N` for ん, `cl` for っ); | |
| a long vowel is two tokens, one per mora; | |
| - devoiced vowels (the u of です, the i of した) are their own tokens `A I U E O`; | |
| - `<sp>` marks a boundary between accent phrases, and punctuation is kept; | |
| - each token carries the Tokyo pitch-accent tone of its mora, low or high, | |
| computed from the phrase's accent type and the mora's position. | |
| It is fast: the 8,650 training sentences, 11.8 hours of speech, are read in | |
| 0.29 s on one CPU thread (0.03 ms per sentence, about 150,000x real time), 5.5x | |
| faster than pyopenjtalk. Its dictionary loads in 3 ms. | |
| **Model.** A transformer encoder (4 blocks) reads the tokens and tones, a | |
| predictor gives each token a length in frames, and two frame-level transformers | |
| (2 and 5 blocks) predict pitch, voicing and loudness and then the settings of | |
| the synthesizer: a band-limited pulse train at the predicted pitch plus noise, | |
| each shaped by a filter predicted every frame (256 samples, 10.7 ms). All blocks | |
| use 192 channels, 4 heads and rotary position embeddings. | |
| | Component | Parameters | | |
| |---|---:| | |
| | Embeddings (token, tone, voice) | 13,632 | | |
| | Text encoder | 1,588,864 | | |
| | Duration and pause predictor | 123,906 | | |
| | Frame transformer A, pitch/voicing/energy head | 795,971 | | |
| | Frame transformer B | 1,985,984 | | |
| | Source-filter waveform head | 25,862 | | |
| | Total | 4,534,219 | | |
| ## Training data | |
| The speech is synthetic. Two voices were designed once from Japanese | |
| descriptions with Irodori-TTS v4.1 Small (`落ち着いた20代後半の女性。明瞭で自然な東京の話し言葉。` | |
| and `30代前半の男性。温かく親しみやすい、自然な東京の話し言葉。`), then every | |
| sentence was read by [Irodori-TTS v4.1 Small Yomi](https://huggingface.co/takuma104/Irodori-TTS-v4.1-Small-Yomi) | |
| (MIT) cloning those two references. No recording of a real speaker is involved. | |
| On the 300 test sentences in both voices, the teacher's speech has 1.7% kana | |
| reading error, 0.70 pitch-accent agreement and a UTMOS of 3.38. | |
| The 8,650 sentences (8,000 train, 350 validation, 300 test, split by sentence) | |
| were chosen to cover as many morae and mora pairs as possible; 25% are | |
| questions and 10% exclamations. | |
| Every clip was screened before training: | |
| - **Reading.** [kana-whisper](https://huggingface.co/sbintuitions/kana-whisper) | |
| transcribed it to katakana, compared with OpenJTalk's reading; more than 10% | |
| kana error was rejected. | |
| - **Alignment.** Phoneme durations come from forced alignment with | |
| `facebook/wav2vec2-lv-60-espeak-cv-ft`, used only to label the data. Its | |
| labels were recalibrated on the teacher's audio, and clips it aligned with | |
| low confidence were rejected. | |
| - **Speaking rate and length.** | |
| | | Clips | Hours | | |
| |---|---:|---:| | |
| | Generated (sentences x voices) | 17,300 | | | |
| | Passed screening | 15,684 | | | |
| | Used for training (F1 7,301, M1 6,783) | 14,084 | 18.6 | | |
| | Validation / test | 614 / 527 | 1.3 / 1.0 | | |
| ## Training | |
| | | | | |
| |---|---| | |
| | Steps | 104,000 | | |
| | Batches | about 9,000 frames (96 s of audio) per step | | |
| | Losses | Multi-resolution spectral losses; adversarial loss with multi-period and multi-resolution discriminators after step 8,000 | | |
| | Hardware | One NVIDIA GB10 (Lenovo ThinkStation PGX) for training; an Apple M4 Max MacBook generated and scored half of the corpus | | |
| | Time | 11.5 hours of training; 5.7 hours to generate the corpus on both machines | | |
| Two changes to the training step made the adversarial phase twice as fast | |
| without changing the gradients: the discriminators' gradient at the waveform is | |
| computed once instead of twice, and their convolutions run in bfloat16 with | |
| autotuned kernels, on batches padded to a few fixed shapes. | |
| ## Evaluation | |
| Speech from the final checkpoint was scored the same way as the teacher's | |
| recordings of the same sentences: kana-whisper transcribes it to katakana | |
| (reading error against OpenJTalk's reading), the pitch tracker checks each | |
| expected rise and fall of the pitch accent, and UTMOS rates naturalness. | |
| | Sentences | Clips | Reading error, eeh | Reading error, teacher | Pitch-accent agreement, eeh | Pitch-accent agreement, teacher | UTMOS, eeh | UTMOS, teacher | | |
| |---|---:|---:|---:|---:|---:|---:|---:| | |
| | Test | 300 | 0.8% | 1.2% | 0.70 | 0.69 | 2.94 | 3.38 | | |
| | Validation | 300 | 1.0% | 1.7% | 0.71 | 0.68 | 2.94 | 3.34 | | |
| | New sentences, not in the dataset | 16 | 0.0% | | 0.69 | | 2.88 | | | |
| eeh reads and accents as well as the teacher it learned from; it sounds less | |
| natural (UTMOS about 0.4 lower), which is the cost of the small synthesizer. No | |
| listening test has been run. | |
| ## Limitations | |
| - Japanese only, two fixed voices, no voice cloning. | |
| - Readings come from OpenJTalk's dictionary. Rare kanji readings, names and | |
| new words can be read wrong, and the accent of compound phrases is predicted | |
| by rules. Whatever OpenJTalk gets wrong, eeh says wrong. | |
| - Latin letters are spelled out one by one (`eeh` is read イー・イー・エイチ). | |
| - The same text is always read the same way. | |
| - The training data is synthetic, so eeh cannot sound more natural than the | |
| teacher; the small source-filter synthesizer sounds more buzzy than it. | |
| ## License | |
| Apache-2.0 for the weights and code. | |
| ## Citation | |
| ```bibtex | |
| @software{nicholson2026eeh, | |
| author = {Nicholson, Alexander}, | |
| title = {eeh: a 4.5M-parameter two-voice Japanese text-to-speech model}, | |
| year = {2026}, | |
| url = {https://huggingface.co/ctrltokyo/eeh-1} | |
| } | |
| ``` | |
| ## Acknowledgements | |
| Inspired by [tiny-voca](https://github.com/n4ze3m/tiny-voca), trained on speech from [Irodori-TTS](https://github.com/Aratako/Irodori-TTS), and reading Japanese with [OpenJTalk](https://open-jtalk.sourceforge.net/). | |