AuroraTTS-100K

AuroraTTS-100K is a very small English TTS model with 95,638 inference parameters. It was trained from scratch on LJSpeech, without distillation, and targets the sub-100K parameter regime.

The model has no neural vocoder. The network predicts per-frame synthesis parameters (normalized log-F0, voicing, a 40-band harmonic spectral envelope and a 24-band noise spectral envelope), and a differentiable harmonic+noise DSP synthesizer renders the waveform. The DSP block has only 145 learned parameters and is included in the total inference parameter count.

Inference parameters	95,638
Sample rate	22,050 Hz
Frame hop	256 samples (~11.6 ms)
Input	79-symbol ARPAbet phoneme vocabulary (via g2p_en)
Speaker	single English speaker
Training data	LJSpeech-1.1
Checkpoint	EMA weights, 150k steps
Synthesis	differentiable harmonic + noise DSP

---

Usage

import soundfile as sf
from transformers import AutoModel

# variant: omit for FP32 (374 KB), "fp16" (187 KB), or "int8" (111 KB)
model = AutoModel.from_pretrained(
    "AuroraAI-Research/AuroraTTS-100K",
    trust_remote_code=True,
    variant="fp16",
).eval()

wav, sr = model.synthesize("The quick brown fox jumps over the lazy dog.")
sf.write("out.wav", wav, sr)

Requirements: PyTorch, torchaudio, NumPy, soundfile, g2p_en, and nltk. The text frontend downloads the required NLTK data on first use. Batch size is 1.

---

Evaluation

Held-out set: 24 sentences from the SanoTTS English diverse held-out set. UTMOS: SpeechMOS utmos22_strong, reported as mean ± std over sentences.

AuroraTTS-100K

Precision	Size	UTMOS	Δ vs FP32
FP32	374 KB	3.530 ± 0.267	—
FP16	187 KB	3.501 ± 0.269	noise
INT8, per-channel	111 KB	3.557 ± 0.256	noise

For the INT rows, weights are rounded to the target grid and dequantized to fp32 for compute. This measures rounding damage, not the speed of a real integer kernel. Biases, norms, and DSP parameters remain fp32.

External baselines

Model	Params	UTMOS
AuroraTTS-100K	95.6K	3.530 ± 0.267
Sano TTS (amy)	1.46M	4.047 ± 0.230
Sano TTS (heart-nano)	294.3K	2.351 ± 0.290

Speed

Real-time factor (RTF), lower is faster. FP32.

Device	RTF	Speed (x real time)
CPU, 1 core	0.0270	37.03×
CPU, 4 cores	0.0139	71.84×
GPU	0.0009	1120×

---

Architecture

AuroraTTS-100K is a non-autoregressive TTS model with learned alignment and a parametric waveform synthesizer. The design is close to FastPitch / RAD-TTS, but replaces the neural vocoder with a tiny differentiable DSP synthesizer.

Component	Setting
Token embedding	79 symbols → 56 channels
Encoder	One conv-GLU + FFN block (kernel 5), applied 4 times with shared weights and per-pass LayerNorms
Context	Packed BiGRU, 28 units per direction
Duration / pitch predictors	Depthwise conv followed by two 1×1 layers (32 channels)
Pitch embedding	Added to encoder output (teacher-forced token pitch in training)
Length regulator	Gaussian soft upsampling with a learned width, initialized to 2 frames
Decoder	9 dilated depthwise-conv layers (dilations 1, 2, 4 cycling), 3 FFNs shared across layers
Output heads	log-F0, voicing logit, 40 harmonic log-envelope bands, 24 noise log-envelope bands

During training, durations and token-level pitch are teacher-forced from the alignment. At inference, the duration and pitch predictors are used directly.

DSP synthesizer

• Up to 80 harmonics, with amplitudes sampled from the predicted 40-band mel-spaced envelope.
• Noise is shaped by the predicted 24-band envelope in the STFT domain (1024-point FFT, hop 256).
• Learned per-harmonic phase offsets (80 parameters).
• Learned static output FIR filter (65 taps).
• Phase is accumulated in float64 during both training and inference to avoid drift over long utterances.

---

Training methodology

Text frontend

• English text is converted to phonemes with g2p_en.
• Vowels carry ARPAbet stress markers.
• The vocabulary has 79 symbols: padding, silence, phonemes, punctuation, and space.
• Repeated spaces are collapsed, and silence tokens are added at utterance boundaries.

Data preparation

• Dataset: LJSpeech-1.1, 22.05 kHz, 13,100 clips. A fixed split (seed 1234) gives 128 test clips, 64 validation clips, and the remaining 12,908 for training.
• Clips longer than 12 s are excluded, as are clips with fewer than 5 voiced frames or too few frames for their transcript.
• F0 is extracted with PyWorld (harvest by default, dio optional) and refined with stonemask, with a floor of 70 Hz and a ceiling of 450 Hz.
• Unvoiced frames are filled by interpolating log-F0 from voiced frames for training targets.
• Global F0 mean and standard deviation (log-Hz) are computed from voiced training frames and used for normalization.

Alignment

Alignment is learned during training and is not used at inference.

• A train-time text/mel aligner produces frame-token alignment scores from 80-band log-mels.
• A beta-binomial prior encourages monotonic, near-diagonal attention. It is computed on the GPU with lgamma.
• Forward-sum loss enforces monotonic alignment.
• Monotonic alignment search (MAS), compiled with numba, extracts hard token durations.
• The first 3,000 steps train only the aligner. The binary alignment loss then ramps in linearly over the following 10,000 steps.

Losses

The acoustic model is trained with:

• duration loss (MSE in log space)
• token-level pitch loss
• frame-level F0 L1 loss, weighted toward voiced frames
• voiced/unvoiced binary cross-entropy
• multi-resolution STFT loss (FFT sizes 256 to 2048)
• log-mel L1 loss

Waveform losses are computed on random 128-frame crops (about 1.5 s), using teacher-forced F0 and voicing passed through the DSP synthesizer. Loss weights: STFT 1.0, log-mel 2.0, F0 2.0, voicing 0.5.

From step 90,000, a multi-resolution spectrogram discriminator (FFT sizes 512, 1024, 2048) adds least-squares adversarial and feature-matching losses (weights 0.5 and 2.0). The discriminator is train-time only.

Optimization

Setting	Value
Steps / batch size	150,000 / 48 utterances
Generator optimizer	AdamW, betas (0.9, 0.98), weight decay 0.01 on weight matrices (none on biases and norms)
Learning rate	Peak 1e-3, 2,000-step linear warmup, cosine decay to 2e-5
Gradient clipping	Global norm 1.0
EMA	Decay 0.999, started after the aligner-only phase; released weights are EMA
Discriminator optimizer	AdamW, lr 2e-4, betas (0.8, 0.99)
Precision	bf16 autocast on CUDA; DSP and losses in fp32
Batching	Length-bucketed random batches; the training split is held on the GPU