Kiseki-TTS

A small, fast Japanese TTS model with a Mamba2 state-space decoder, built on top of Qwen/Qwen3-TTS-Tokenizer-12Hz.

Kiseki-TTS generates discrete neural audio codec tokens at 12.5 Hz and decodes them to waveform with the Qwen3 TTS codec. Because the acoustic decoder is a linear-time SSM rather than a self-attention stack, generation cost is constant per frame — memory does not grow with utterance length, and there is no KV cache to manage.

The same checkpoint also performs ASR (Japanese speech → text), since TTS and ASR were trained jointly in a single multi-task run.

Example

Model Details

Model Description

Developed by: telecomadm1145
Model type: Encoder–decoder (Transformer encoder + cross-attention/Mamba2 decoder)
Language: Japanese (ja)
License: MIT
Finetuned from: telecomadm1145/Kiseki-1.1-0.3B (a seq2seq translation model)
Audio codec: Qwen/Qwen3-TTS-Tokenizer-12Hz

Architecture

Component	Spec
Encoder	12 layers, bidirectional self-attention + RoPE + SwiGLU
Decoder	6 layers, cross-attention → Mamba2 SSM (no self-attention)
Hidden size	1024
Attention heads	8
SSM state size / head dim	128 / 64 (32 SSM heads)
Conv kernel / expand	4 / 2
Norm	RMSNorm (pre-norm), gated RMSNorm inside the SSM mixer
Text vocab	65,792 (shared encoder embed / decoder embed / LM head)
Params	≈0.33 B backbone + ≈78 M audio branch ≈ 0.41 B total

The decoder deliberately has no causal self-attention. Temporal context is carried entirely by the Mamba2 recurrent state; text conditioning enters through cross-attention whose K/V are computed once from the encoder and reused for every frame.

Audio tokenization

Property	Value
Frame rate	12.5 Hz (80 ms per frame)
Quantizer layers (Q)	16
Codebook size	2048 per layer
Effective token vocab	2176 (2048 codes + EOS/BOS/PAD, padded to a multiple of 128)
Reserved IDs	`EOS = 2048`, `BOS = 2049`, `PAD = 2050`
Nominal bitrate	16 × 12.5 × log₂(2048) = 2.2 kbps
Max trained length	512 frames ≈ 41 seconds

Depth modelling (MTP head). Each frame's 16 codebook layers are predicted by a shared "multi-token prediction" head rather than 16 separate decoder passes. Layer q sees the decoder hidden state plus the exclusive prefix sum of the embeddings of layers 0 … q-1:

logits_q = W_q · Block( h_t + (1/√Q) · Σ_{j<q} E_j(c_t^j) ) + b_q

where Block is a small RMSNorm → SwiGLU(×2) → RMSNorm residual body shared across all 16 layers. This means one trunk evaluation per frame and 16 cheap head evaluations, instead of 16 full autoregressive steps.

Why it's fast

1. 12.5 Hz is the headline number. One second of speech is 12.5 decoder steps. Codecs running at 50 Hz or 75 Hz need 4–6× more autoregressive steps for the same audio. Concretely:

Audio duration	Decoder trunk steps	Codebook head evals
1 s	12.5	200
5 s	63	1,000
10 s	125	2,000
30 s	375	6,000

A 10-second utterance is 125 recurrent steps. For comparison, a token-level LLM TTS at 50 Hz × 8 codebooks would be pushing ~500 trunk steps for the same clip.

2. O(1) state, not O(T) cache. The Mamba2 decoder carries a fixed (32 heads × 128 state × 64 dim) tensor plus a 3-frame conv window per layer. Generating 40 seconds costs exactly as much per step as generating 1 second — no attention matrix, no KV cache reallocation, no quadratic blowup. Long-form synthesis degrades gracefully instead of falling off a memory cliff.

3. Cross-attention K/V is computed once. Encoder output is projected to per-layer K/V a single time during prefill. Every subsequent frame does one small Q·Kᵀ against a fixed-length text sequence.

4. A shallow decoder. Only 6 decoder layers sit in the autoregressive loop. The 12-layer encoder runs exactly once, fully parallel over the input text.

5. The depth loop is cheap. The 16 codebook layers are resolved sequentially (layer q conditions on layers <q), but each step is one ×2 SwiGLU block at d=1024 — small enough that batch-1 generation is memory-bandwidth-bound rather than compute-bound.

How to Get Started

Setup

import torch
from transformers import AutoModelForSeq2SeqLM, PreTrainedTokenizerFast

repo = "telecomadm1145/Kiseki-TTS"
tok = PreTrainedTokenizerFast.from_pretrained(repo)
m = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval().cuda()

lang = tok.convert_tokens_to_ids("<|2ja|>")

TTS — text to audio codes

ids = m.build_tts_input_ids(tok.encode("こんにちは"), lang, device="cuda")
out = m.generate_audio(ids, max_new_frames=250, temperature=0.9, top_k=50)
codes = out["audio_codes"][0][out["valid_mask"][0]]   # (T, Q)

build_tts_input_ids assembles [TTS] <|2ja|> …text… <eos>. max_new_frames=250 ≈ 20 seconds of audio at 12.5 Hz.

Decode codes to waveform

import soundfile as sf
from qwen_tts import Qwen3TTSTokenizer

tokenizer = Qwen3TTSTokenizer.from_pretrained(
    "Qwen/Qwen3-TTS-Tokenizer-12Hz",
    device_map="cuda:0",
)

wavs, sr = tokenizer.decode({"audio_codes": codes})
sf.write("decode_output.wav", wavs[0], sr)

ASR — audio codes to text

ASR consumes only the first quantizer layer (q0) of the codec output.

enc = m.build_asr_encoder_codes(q0_codes, device="cuda")
txt_ids = m.generate_transcription(enc, decoder_start_tokens=[m.config.bos_token_id, lang])
print(tok.decode(txt_ids[0], skip_special_tokens=True))

Sampling parameters

Argument	Default	Notes
`max_new_frames`	512	Divide by 12.5 for seconds
`temperature`	0.9	Lower → flatter, more monotone prosody
`top_k`	50
`top_p`	0.95
`temperature_q0` / `top_k_q0`	inherit	Tune layer 0 separately — it carries most of the semantic content; residual layers tolerate more randomness

Generation stops when layer 0 emits EOS (2048). BOS and PAD are masked out of the logits, so they can never be sampled.

Training Details

Initialization: all non-audio weights restored from the Kiseki-1.1-0.3B translation checkpoint; the audio embedding tables and MTP head were randomly initialized and trained with a 3× learning-rate multiplier relative to the backbone.
Objective: joint TTS + ASR, sampled at roughly 70 % TTS / 30 % ASR per step.
TTS supervision: teacher forcing in both directions — along time (previous frames) and along depth (ground-truth prefix of lower codebook layers).
Packing: multiple utterances are packed per row with segment-ID masking; attention, the depthwise conv, and the SSM recurrence are all reset at segment boundaries so packed samples never leak into one another.
Optimizer: AdamW, cosine schedule with warmup, gradient clipping at 1.0, weight decay applied only to ≥2-D parameters.
Data: telecomadm1145/asmr_archive_qwentts_encoded.
Task tokens: [TTS] and [ASR] reuse the last two UL2 sentinel IDs in the 65,792-token vocabulary, so the tokenizer is unchanged from the base model.

Limitations and Bias

Japanese only. No other language was trained; the <|2ja|> tag is the only supported language token for speech tasks.
Single-domain voice. Training data is ASMR-style Japanese speech, so timbre, pacing, and recording character are strongly biased toward that domain. There is no speaker conditioning or voice cloning — output voice is not controllable.
~41 s ceiling. The model saw at most 512 frames during training. Longer requests will run (the SSM state is length-agnostic) but quality beyond ~40 s is untested.
ASR is a byproduct. It reads only quantizer layer 0 and was trained as an auxiliary task; do not expect dedicated-ASR accuracy.
Sampling sensitivity. Discrete codec TTS can occasionally loop or emit early EOS. Lower temperature_q0 if you observe repetition.
No safety filtering was applied to the training corpus.

Special Token Reference

Token	ID	Purpose
`<bos>`	1	Text decoder start
`<eos>`	2	Text end
`<pad>`	3	Text padding
`<\|2ja\|>`	—	Target-language tag (`tok.convert_tokens_to_ids`)
`[TTS]`	`vocab_size - 1`	TTS task prefix
`[ASR]`	`vocab_size - 2`	ASR task prefix
audio `EOS`	2048	Stop condition (layer 0)
audio `BOS`	2049	Audio decoder start
audio `PAD`	2050	Audio padding

Citation

@misc{kiseki-tts,
  title  = {Kiseki-TTS: A Fast Japanese TTS Model with a Mamba2 Decoder},
  author = {telecomadm1145},
  year   = {2026},
  url    = {https://huggingface.co/telecomadm1145/Kiseki-TTS}
}