aoiandroid/neutts-jp-150m-onnx-float32

neuTTS-JP-150m ONNX float32

This is the ONNX export of the neuTTS-JP-150m model, a Japanese text-to-speech model with voice cloning capabilities.

Model Details

• Original Model: aoiandroid/neuTTS-JP-150m
• Architecture: LlamaForCausalLM (151.8M parameters)
• File Size: ~609 MB (1.75 MB ONNX + 607 MB data)
• Format: ONNX (Opset 18)
• Language: Japanese
• Sampling Rate: 24 kHz

Repository Structure

aoiandroid/neutts-jp-150m-onnx/
├── neutts-jp-150m-original.onnx          # ONNX model structure (1.75 MB)
├── neutts-jp-150m-original.onnx.data      # ONNX model weights (607 MB)
└── README.md                              # Model card

File Descriptions:

• neutts-jp-150m-original.onnx: ONNX model file containing the graph structure and metadata
• neutts-jp-150m-original.onnx.data: External data file containing the model weights (large tensor data)
• README.md: Model documentation and usage instructions

Usage

Installation

pip install onnxruntime transformers torchaudio neucodec soundfile

Python Example

import onnxruntime as ort
from transformers import AutoTokenizer
from neucodec import NeuCodec
import torch
import torchaudio
from torchaudio import transforms as T
import soundfile as sf

# Load ONNX model
session = ort.InferenceSession(
    "neutts-jp-150m-original.onnx",
    providers=['CPUExecutionProvider']
)

# Load tokenizer and codec
tokenizer = AutoTokenizer.from_pretrained(
    "aoiandroid/neuTTS-JP-150m",
    trust_remote_code=True
)
codec = NeuCodec.from_pretrained("neuphonic/neucodec")
codec.eval()

# Load and process reference audio
waveform, sr = torchaudio.load("reference_audio.wav")
if waveform.shape[0] > 1:
    waveform = waveform.mean(dim=0, keepdim=True)
if sr != 16_000:
    waveform = T.Resample(sr, 16_000)(waveform)
waveform = waveform.unsqueeze(0)

# Encode reference audio
with torch.inference_mode():
    ref_codes = codec.encode_code(waveform).flatten().tolist()

# Tokenize text
text = "こんにちは、これはテストです。"
text_ids = tokenizer.encode(text, add_special_tokens=False)
eos_id = int(tokenizer.eos_token_id)
input_ids = ref_codes + [eos_id] + text_ids + [eos_id]
input_ids = torch.tensor([input_ids], dtype=torch.long)

# Generate using ONNX model
input_ids_np = input_ids.numpy()
outputs = session.run(None, {'input_ids': input_ids_np})
logits = outputs[0]

# Get next token (greedy decoding)
next_token = int(logits[0, -1].argmax())
# (Repeat generation loop for full sequence)

# Decode audio
gen_ids = [generated_tokens]  # Your generated tokens
gen_ids_tensor = torch.tensor(gen_ids, dtype=torch.long).unsqueeze(0).unsqueeze(0)
with torch.inference_mode():
    audio_data = codec.decode_code(gen_ids_tensor).cpu()

# Save output
sf.write("output.wav", audio_data[0][0].cpu().numpy(), 24_000)

Model Architecture

This model is based on LlamaForCausalLM with the following configuration:

• hidden_size: 512
• num_hidden_layers: 12
• num_attention_heads: 8
• intermediate_size: 2048
• vocab_size: 99111

Requirements

• onnxruntime >= 1.25.0
• transformers >= 4.29.0
• torch >= 2.11.0
• torchaudio >= 2.11.0
• neucodec
• soundfile

Original Model

aoiandroid/neuTTS-JP-150m

License

MIT License

Citation

If you use this model, please cite the original neuTTS-JP-150m model.