README.md

---
language:
- en
license: apache-2.0
tags:
- text-to-speech
- tts
- matcha-tts
- flow-matching
- tflite
- onnx
- audio
- on-device
- expressive-speech
datasets:
- ljspeech
- libritts_r
metrics:
- loss
- wer
pipeline_tag: text-to-speech
---

# Sonora

**Sonora** is a lightweight, non-autoregressive text-to-speech (TTS) model line built for
real-time, on-device, and resource-constrained environments — the voice engine of
**Project Prosodia**. Architecture: Matcha-TTS-family Optimal-Transport Conditional Flow
Matching (OT-CFM) acoustic model + HiFi-GAN vocoder, with a growing set of **expressive
conditioning channels** (valence / arousal-energy / tension via FiLM) aimed at directable,
story-driven narration.

This repository is the **model registry**: each top-level directory is one promoted,
audited milestone artifact with its own README, eval report, and provenance.

## Registry index

> **Current trunk:** `derisk-energy-24k` (acoustic) + `vocoder-24k-hifigan` (vocoder) —
> the 24 kHz multi-speaker line. Next up: the v1.1 VAT continuation (story-driven valence
> corpus), warm-starting from `vat3-24k`.
> Machine-readable lineage: [`registry.json`](./registry.json).


| Directory | Training pass | What it is | Status |
|---|---|---|---|
| [`baseline-ljspeech-22k/`](./baseline-ljspeech-22k) | **Phase 0** | Single-speaker LJSpeech fine-tune, 22.05 kHz, end-to-end graphs (ONNX + TFLite) and the LiteRT split-graph mobile lane. *(Renamed from `v1-ljspeech` on 2026-07-22 — update old deep links.)* | Published 2026-07-12 |
| [`vocoder-24k-hifigan/`](./vocoder-24k-hifigan) | 24 kHz trunk — vocoder | HiFi-GAN vocoder fine-tune, 24 kHz / 80-band — the vocoder pairing for the multi-speaker line. Converged + human-audited (copy-synthesis A/B indistinguishable). | Published 2026-07-16 |
| [`derisk-energy-24k/`](./derisk-energy-24k) | **Expressive de-risk** (north-star §7) | First trained expressive channel: energy/arousal FiLM conditioning, multi-speaker (247 spk, LibriTTS-R), 24 kHz. Controllability ρ≈1.0, identity-preserving, WER-safe; includes the VAT-ready LiteRT split-graph export. | Published 2026-07-16 |
| [`vat3-24k/`](./vat3-24k) | **Full 3-channel VAT** (milestone 3) | First all-three-channels training (valence/energy/tension FiLM), warm-started from `derisk-energy-24k`. **Mixed verdict**: energy PASS, tension near-pass, valence FAIL — a corpus-label limitation, not a training one (see its README). Warm-start seed for the v1.1 continuation. | Published 2026-07-22 |

Directory names are deliberately **descriptive, not phase-numbered** — artifacts keep their
identity even as the roadmap's phase labels evolve; the "Training pass" column carries the
roadmap mapping.

**Developed by:** Artificial Humanity · **License:** Apache-2.0 · **Languages:** English

## Model line at a glance

* **Phase 0 (`baseline-ljspeech-22k`)** proved the on-device lane: ~18.2M-parameter acoustic
  model, single forward pass text→PCM, running on phones via TFLite/LiteRT.
* **The 24 kHz multi-speaker line** (`vocoder-24k-hifigan` + `derisk-energy-24k`) is the
  current trunk: native 24 kHz, 247 speakers, and validated FiLM conditioning — the
  de-risk experiment demonstrated a continuous, monotonic, speaker-preserving loudness/
  energy control learned from weak labels in ~35 GPU-hours on consumer hardware.
* **Full 3-channel VAT conditioning** (valence/arousal/tension) and story-driven expressive
  training data (directed teacher synthesis + aligned public-domain audiobooks) are in
  active development; artifacts land here as they pass their gates.

## Provenance & audit culture

Every artifact directory documents: training corpus + license wall, warm-start lineage,
pre-registered convergence criteria, automated eval-harness results (controllability /
identity leakage / intelligibility), and human audition verdicts. Checkpoints are
distributed as **safetensors** (weights-only); full resumable training checkpoints are
retained privately.

## Companion dataset

The **`sonora-expressive-registers`** dataset (CC-BY-4.0) — certified keeps from our
directed teacher-synthesis pipeline, with exact intended-VAT labels and full render
provenance — is published separately under this organization.