weights-and-wires/audio8-tts-from-scratch

A DualAR text-to-speech language model trained from scratch on 440 clips (72.6 minutes) of LibriSpeech clean/train.100, 4 speaker(s).

⚠️ This is a training-pipeline experiment, not a production TTS model. The ~601 M-parameter transformer was randomly re-initialized and trained for 280 optimizer steps. Expect undertrained, partly unintelligible speech. LibriSpeech is 16 kHz upsampled to the codec's 44.1 kHz, which caps quality further.

What was trained vs. reused

Component	Status
Slow AR transformer (24 layers, dim 896)	trained from random init
Fast AR transformer (4 layers)	trained from random init
Neural codec (codec.pth, 10x4096 @ 44100 Hz)	reused frozen from Audio8/Audio8-TTS-Preview-0.6b
Tokenizer / architecture config	reused from the same checkpoint

Training configuration

Setting	Value
epochs	10
effective batch size	16
learning rate	0.0002 (cosine, 0.05 warmup)
weight decay / grad clip	0.01 / 1.0
precision	fp32 weights + bf16 autocast
max sequence length	2048
train / eval examples	440 / 23
hardware	1x NVIDIA A100-SXM4-40GB

Final metrics

Metric	Value
final train slow_loss	3.9706
final train fast_loss	5.4779
final train slow_accuracy	0.4759
final train fast_accuracy	0.1876
final eval slow_loss	7.6749
final eval fast_loss	6.9021

Random-guess baselines for reference: slow_loss ln(vocab) = 11.96, fast_loss ln(4096) = 8.32.

Usage

import soundfile as sf, torch
from transformers import AutoModel, AutoProcessor

repo = "weights-and-wires/audio8-tts-from-scratch"
processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16).eval().cuda()

inputs = processor(text=["Hello world."], return_tensors="pt")
inputs = {k: v.cuda() for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=512, temperature=0.8, top_p=0.95,
                     top_k=50, do_sample=True, return_dict_in_generate=True)
audio, lengths = model.decode_audio(out.codes)
sf.write("out.wav", audio[0, :int(lengths[0])].float().cpu().numpy(), 44100)

The model was trained on the no-reference prompt template only — pass text= alone; voice cloning via reference_audio= was never trained and will not work.

License

Apache-2.0, inherited from the base checkpoint. LibriSpeech is CC-BY-4.0.