ThorBaller/Kokoro_Jarvis

Kokoro JARVIS

A fine-tuned Kokoro-82M voice, trained with the kikiri-tts StyleTTS2 recipe (Stage 1 reconstruction → Stage 2 GAN + WavLM SLM adversarial) and exported to Kokoro's two runtime artifacts.

This is a fine-tune of Kokoro's architecture, not a model in HF's standard pipeline format. There is no from_pretrained path — loading it requires the Kokoro source tree plus the key remapping described under Loading the fine-tuned weights.

Listen

Rendered from the voicepack in this repo, 24 kHz mono, all English.

ep00000_0.wav — "At last, sir, all systems are functional."

ep00000_1.wav — "I have completed the diagnostics as requested."

ep00000_2.wav — "Would you care for a cup of tea, sir?"

ep00000_3.wav — "The probability of success is approximately 90%."

Clip	Duration	Peak	RMS
ep00000_0.wav	3.10 s	0.394	0.058
ep00000_1.wav	3.40 s	0.370	0.054
ep00000_2.wav	2.40 s	0.408	0.053
ep00000_3.wav	3.95 s	0.412	0.055

Contents

Path	Size	What it is
jarvis_ep00000_kokoro.pth	312 MiB	The fine-tuned weights — 5 module state_dicts
voices/jarvis_ep00000.pt	511 KiB	The voicepack — a (510, 1, 256) style vector
output/ep00000_{0..3}.wav	~150–190 KiB each	Rendered samples
training/config.json	2.4 KiB	Kokoro-82M architecture config + phoneme vocab
configs/config_german_ft.yml	7.4 KiB	A German single-speaker fine-tuning config

voices/jarvis_ep00000.pt was averaged over the 258 training WAVs.

You need both artifacts

This is the single most expensive mistake with this repo:

Feeding the voicepack into stock Kokoro weights produces a plausible, transcribable voice that is not the fine-tune at all — stock timbre with one style vector swapped. Nothing errors. The audio just sounds wrong.

You load	You get
stock Kokoro + this voicepack	almost-stock voice, fine-tune thrown away
this .pth + this voicepack	the actual JARVIS voice

Loading the fine-tuned weights

The checkpoint was saved by torch ≥ 2.1 under DataParallel with weight-norm parametrization, so its keys do not match what KModel expects. Two independent mismatches:

1. module. prefix — training wrapped modules in MyDataParallel, so keys are module.<name>; KModel expects <name>.
2. Weight-norm parametrization — saved as <conv>.parametrizations.weight.original0 / original1; torch 2.2.2 expects weight_g / weight_v. This is the bigger one: fixing only the prefix leaves the decoder at 0/491 keys matched.

These weights do not load under torch 2.2.x, and they fail silently. KModel wraps every load_state_dict in a bare except: and retries with strict=False. When nothing matches, the module keeps its random initialization and the pipeline still runs, emitting flat noise-like audio with a valid waveform and plausible duration.

Always assert key overlap

Never trust the audio, and never trust the absence of an error:

live = set(model.decoder.state_dict())
fk = set(converted["decoder"])
print(len(live & fk), "of", len(live))   # 0 means nothing loaded

A correct load reads bert 25/25, bert_encoder 2/2, text_encoder 24/24, predictor 122/146, decoder 375/491. The ~116 unmatched decoder keys are norm parameters absent from the fine-tune; they keep stock values, which is expected.

Converter used for this checkpoint: ~/.hermes/venvs/kokoro/bin/jarvis-weights-fix.py --input <raw.pth> --output <fixed.pth> --config <training/config.json> — it prints per-module match counts and exits non-zero if any module matches zero keys.

Telling a silent random-init load from real speech

• Whisper returns (machine whirring) or BLANK_AUDIO.
• RMS is suspiciously uniform — near-constant across every quarter-second — where real speech has an envelope that starts low and rises past 0.11.

The samples in output/ show healthy envelopes (quarter-to-quarter spread of 0.023–0.041, peaks 0.37–0.41, RMS ~0.055), so these are real synthesis, not silent failures.

GGUF conversion (kokoro.cpp)

The upstream converter takes a local directory holding config.json (with its vocab block) plus a .pth:

python models/convert-kokoro-to-gguf.py --input <dir> --output model.gguf --outtype f16

It strips module. and pre-fuses weight_norm — the transform torch 2.2.x cannot do. A correct run prints Mapped: 459, skipped: 0; an orphan unfused weight_g/weight_v makes it abort. Feed it the raw checkpoint keys, not the already-remapped file the PyTorch path needs.

Voicepack conversion is a separate script (models/convert-kokoro-voices-to-gguf.py) and needs no custom work — a Kokoro voicepack is (510, 1, 256), the same shape as stock voices/*.pt.

Training config notes

training/config.json is Kokoro-82M's own architecture block: istftnet decoder (upsample_rates [10, 6], gen_istft_n_fft 20, gen_istft_hop_size 5), hidden_dim 512, style_dim 128, n_token 178, multispeaker true, plus the 178-entry phoneme vocab the GGUF converter needs.

configs/config_german_ft.yml targets German single-speaker fine-tuning from the pre-trained checkpoint — istftnet (no diffusion), lambda_diff 0.0, diff_epoch 999, TMA_epoch 0. It carries a dead training: block that neither train_first.py nor train_second.py reads; the live values are the top-level keys. See kikiri-tts#14.

Checksums

47a2ed18ff7389e005500d65adb880d642ad8ec6b56f7be81edca6f422e0d5f4  jarvis_ep00000_kokoro.pth
ca07d81ba363d43f13613b4d3058b049923e5fe1dc0c58a26b1931f8ff2398cd  voices/jarvis_ep00000.pt

Verify a download by size, not by presence — a zip round-trip can land most files intact while zeroing a few, and ls shows all the names either way. A 0-byte .pth means absent.