Pitchy

A pitch (f0) tracker for the audio speech systems actually get: phone lines, noisy rooms, reverberant microphones. One ONNX graph, 16 kHz audio in, a pitch salience every 10 ms out, on one CPU thread; the PyTorch model it was exported from is here too.

Work in progress. Pitchy is still being trained, and this repository is updated as better models pass our development-set rules; every upload is its own revision, so pin one. In progress now: clean studio speech, where FCNF0++ (trained on PTDB-TUG's speakers) still leads on PTDB-TUG by 0.33 points of frame error; pitch precision, where the lead is in voicing rather than pitch (86.6 % of voiced frames within 50 cents over every condition, RMVPE 87.3 %); and music and pitch above 800 Hz, which the training speech does not reach (MDB-stem-synth: 27.2 % frame error, SwiftF0 13.0 %). The numbers below are the model in this revision.

Revisions

• w7_rv_hold10 (2026-10-28): voices held vowels and codec-decoded TTS speech. The first upload called a held vowel unvoiced (music under its training speech had taught it that a held harmonic sound is not a voice) and dropped voicing for up to 1.84 s of loud TTS speech; this one also learns voicing from the WORLD resyntheses, a tenth of them with one vowel held 0.5 to 2 s. On a TTS stability study's 69 passages, the longest stretch of loud speech that RMVPE, FCPE and CREPE all call voiced and Pitchy does not is 4 frames at most (the first upload: 184); a held 1.5 s vowel is 100 % voiced (was 52 %); voicing holds through a sudden +3 semitone step on 99 % of frames (was 79 %). It gives a little back on natural speech: frame error over every condition 4.17 % on PTDB-TUG's test speakers (the first upload 4.02 %) and 8.79 % on Keele (7.57 %), where it misses more voiced frames in heavy noise; on music it is much better (27.2 % on MDB-stem-synth, was 54.1 %). The first upload stays available as revision first-release.
• w4_mel_rmvpev_norv (2026-10-27): the first upload.

Use

ONNX, with numpy and onnxruntime only (pitchy_onnx.py):

from pitchy_onnx import Pitchy
m = Pitchy('pitchy.onnx')
f0, voiced, confidence = m(audio_16k)

PyTorch, with torch, numpy and safetensors (pitchy_torch.py, the network code; pitchy.safetensors, the weights):

import torch, pitchy_torch
m = pitchy_torch.load('.')                 # config.json + pitchy.safetensors
salience, voicing = m(torch.from_numpy(audio_16k)[None])

Contract: audio [1, N] float32 at 16 kHz -> salience [1, T, 360] (CREPE's grid, 20 cents from 32.70 Hz; its peak is the voicing probability) and voicing [1, T]; T = N // 160 + 1, frame i centred at i x 10 ms. Voiced where the peak exceeds 0.869, the threshold chosen on PTDB-TUG's development speakers.

Benchmark

F0 frame error (FFE, the headline), gross pitch error (GPE) and voicing decision error (VDE), % and lower is better; every threshold and frame lag chosen on PTDB-TUG's development speakers only. The best published tracker is the lowest FFE of 14 trackers, each with its better decoder, per row. Four of them were trained on PTDB-TUG; Keele and FDA are free of that.

Set	Conditions	Pitchy FFE	GPE	VDE	Best published	FFE	GPE	VDE
PTDB-TUG test speakers	Clean	3.37	1.38	3.06	FCNF0++	3.04	1.49	2.70
PTDB-TUG test speakers	Real noise, 5 and 0 dB	4.80	1.88	4.40	RMVPE	12.78	1.51	12.49
PTDB-TUG test speakers	Real rooms	4.66	2.08	4.22	RMVPE	8.92	1.33	8.69
PTDB-TUG test speakers	Phone line (8 kHz codecs)	4.13	1.63	3.77	RAPT	6.98	7.45	5.38
PTDB-TUG test speakers	Room, noise and line	5.05	2.27	4.58	SwiftF0	14.26	2.07	13.99
PTDB-TUG test speakers	Every condition	4.17	1.72	3.80	RMVPE	9.61	1.47	9.34
Keele	Clean	5.33	0.44	5.12	RAPT	5.04	2.12	3.98
Keele	Real noise, 5 and 0 dB	12.36	0.82	12.03	RMVPE	16.50	1.22	16.02
Keele	Real rooms	11.54	1.22	11.03	RMVPE	21.02	1.02	20.66
Keele	Phone line (8 kHz codecs)	7.03	1.04	6.54	YAAPT	8.99	3.88	7.18
Keele	Room, noise and line	11.80	1.21	11.29	YAAPT	21.44	8.74	17.41
Keele	Every condition	8.79	0.76	8.46	RMVPE	14.86	0.90	14.51
FDA	Clean	4.02	0.40	3.88	RMVPE	4.25	0.21	4.17
FDA	Real noise, 5 and 0 dB	6.47	1.17	6.08	RMVPE	11.72	0.66	11.52
FDA	Real rooms	6.26	0.92	5.96	RMVPE	13.70	0.68	13.52
FDA	Phone line (8 kHz codecs)	4.65	0.66	4.42	RAPT	6.74	3.85	5.45
FDA	Room, noise and line	6.56	1.03	6.23	RMVPE	16.84	0.85	16.64
FDA	Every condition	5.20	0.72	4.95	RMVPE	9.32	0.42	9.20

Every tracker on PTDB-TUG's test speakers by condition family (log scale; each row sorted by its error over every condition):

Cost

3.78M parameters, 19.6 MB, 0.010 s of one CPU thread per second of audio, peak memory 239 MB on 30 s. The graph matches PyTorch to 0.0002 cents on speech and 0.0102 cents on a 30 s pure tone, with every voicing decision equal (our budget is 0.01 cents: this revision misses it on the tone by 0.0002); the standalone PyTorch module here matches the graph to 0.0102 cents (100 % voicing agreement).

Training data and licences

LibriTTS-R train-clean-100 (CC BY 4.0). Half the training audio is that speech, with pitch targets where RMVPE (RVC weights: MIT with a research-use clause), FCPE (MIT) and CREPE (MIT) agree within 50 cents and voicing targets from RMVPE; half is the same speech resynthesised with WORLD, whose f0 is exact (pitch targets only). RMVPE's weights were trained on PTDB-TUG, so its voicing convention may reach Pitchy through those targets; Keele and FDA are free of this. Noise from MUSAN and DEMAND (its Hugging Face mirror, CC BY 4.0), simulated rooms from OpenSLR 28. No speaker of any evaluation set is in the training data. The licence of these weights is not set yet.

The preprint (Pitchy) explains how it was made and measured.