AGIWSVoice

Research and engineering on a TTS / voice-conversion engine built around a single audio decoder: a latent of shape [T, 256] at 25 fps is mapped to 24 kHz audio.

Motivation

OmniVoice consists of a Qwen3 backbone, a discrete 8-codebook head, and a DAC decoder. The backbone learns text -> latent, but it is not streamable: it is bidirectional and runs 32 full MaskGIT passes per chunk (142-199 ms per step, about 5-6 s per chunk). This project does not use the backbone or the discrete codebooks. Only the decoder is kept; the latent is produced and edited by our own tooling.

phonemes + durations + curves (F0 / energy)  ->  latent [T,256]  ->  DAC decoder  ->  24 kHz
                                                    ^
                                       inversion of real audio

Decoder weights come from the original PyTorch checkpoint; the GGUF copy is kept only as a cross-check that both hold the same tensors.

What decoder inversion enables

  • artifact detection β€” residual audio - decode(latent); clicks, clipping, hum, dropouts and compression damage are not explained by the decoder and land in the residual, so no labels are needed;
  • voice cloning β€” timbre from the mean latent of a reference utterance, without any speaker encoder;
  • speech-to-speech β€” content and curves from the source, timbre from the target speaker;
  • prosody control β€” sensitivity directions d(F0, energy) / d(latent) and counterfactual stress edits;
  • ASR add-ons β€” emotion from curve coordinates, stress placement, and rescoring of external ASR hypotheses by residual.

Reference blocks

The upstream PyTorch implementation is exposed block by block, so both forward and backward passes can be run through a single block or the whole pipeline:

RVQ dequant -> fc2 (1024 -> 256) -> conv1 -> decoder blocks -> conv2
DAC encoder / HuBERT / quantizer   (input side)

Structure

  • docs/ β€” internal plan and work journal;
  • src/agiws_voice/ β€” library code;
  • scripts/ β€” publishing, job submission, extraction jobs;
  • tests/ β€” tests.

External dependencies

  • agiws/OmniVoice-0.6B β€” PyTorch reference implementation and codec weights;
  • agiws/OmniVoice-0.6B-Q8-GGUF-llama.cpp β€” GGUF copy of the codec weights;
  • agiws/NeuralQuant β€” quantized modules, used only by later phases;
  • a Colab T4 compute worker behind the AGIWS Gateway for heavy runs.
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support