Mamba-CoT-TTS: pure Mamba-2, hybrid and transformer TTS models that write a prosody plan before they speak

This repository holds the models, evaluation results, training corpora and code of the final-year project [Paper title] (Maximilian Wong Junlin, Nanyang Technological University, 2026). The code is also on GitHub at maxpicy/FYP.

Each model reads the text and writes a think block, then speaks. The think block has three parts:

• a short description of the delivery (the C3 models only);
• tags for emotion, pace and pitch;
• a word-level prosody plan of pitch, duration and energy.

The model then generates speech as Fish S2 codec tokens. Three backbones were trained on the same corpus with the same tokens, tokenizer and recipe:

Arm	Backbone	Parameters (with the depth module, speaker table and plan injector)
pure	Mamba-2 1.3B (state-spaces/mamba2-1.3b), 48 SSM layers	1.540 B
hybrid	the same, with the top 4 Mamba mixers replaced by causal attention (RoPE)	1.503 B
transformer	Pythia-1.4B (EleutherAI/pythia-1.4b)	1.507 B

All three share the rest of the model:

• The backbone generates codebook 0 frame by frame at 21.5 Hz.
• A 4-layer Mamba-2 depth module (width 1024) generates codebooks 1 to 9 inside each frame.
• A speaker table is added to the backbone input, so the voice is a speaker id, not prompt audio.
• A plan injector embeds the plan bins into the input as the frames are generated.

Contents

Folder	What it is	Licence
two_stage_v2/{pure,hybrid,transformer}_c3	The report's main models. Two-stage recipe (Stage 1′ then Stage 2 v2), C3 tier: description + tags + plan. Training seed 1.	CC BY-NC 4.0
two_stage_v2/{pure,hybrid,transformer}_c2	The same recipe, C2 tier: tags + plan, no description. Trained on the identical Stage 2 file.	CC BY-NC 4.0
two_stage_v2/{pure,hybrid,transformer}_c3_seed2	C3 retrained with training seed 2. Same data and steps, different data order.	CC BY-NC 4.0
stage1p_bases/{pure,hybrid,transformer}	The Stage 1′ models every Stage 2 run started from (tags + plan, 120K steps, the full 2.36M-row corpus).	CC BY-NC 4.0
stage_a/{pure,hybrid,transformer}_c3	The one-stage recipe that came before the two-stage one (details below).	CC BY-NC 4.0
path_b_renderer/	Path B's acoustic stage: Fish Audio's openaudio-s1-mini, fine-tuned on our data, with the voice 4200 / 4201 reference clips. See path_b_renderer/README.md.	CC BY-NC-SA 4.0
eval/	Every generated row, score and statistic behind the decode-H results, and a paired audio sample. See eval/README.md.	CC BY-NC 4.0
corpus/	The Stage 1′ and Stage 2 v2 training corpora, tokenized, as parquet. See corpus/README.md.	CC BY-NC-SA 4.0
code/	The inference, scoring and training code, the decode recipe (decode.sh) and the exact environments. See code/README.md.	MIT

Each model folder holds model.safetensors and config.json:

• model.safetensors: fp32 weights only, bit-identical to the training checkpoint's model state.
• config.json: the flags that build the model before the weights load, the step, the speaker-table size, the weight hash and the parameter count.

Training

Stage 1′ trained each tokens-only base on the full corpus, with the prose stripped at runtime:

• the starting point was the Fish-codec Stage 1 base of that backbone, with its speaker table grown to 32,768 rows;
• 2.36 M rows, about 4,970 h of audio (corpus/stage1p);
• 120K steps.

Stage 2 v2 continued for 40K steps on a rebalanced file of 1.43 M rows (corpus/stage2_v2):

• Emilia is capped at 25 % of the file by DNSMOS rank (it ends up at 17 %);
• expressive rows are up-weighted, and 40 % of the file is replay from Stage 1′;
• the pause-exit loss weight is 3.0.

C3 learns the description at loss weight 1.0. C2 learns from the same file with the description removed.

Stage A (stage_a/) was the earlier, one-stage recipe. Each C3 model trained for 120K steps from the same Fish Stage 1 base (8,192-row speaker table) on an earlier plan corpus, in which only about 5 % of rows carry a description. Their decode-H cells are in eval/ under the runs p2cotA_<arm>_full_dense_*.

Results (decode H, Path A, the field's protocol)

The protocol is text-only with one fixed voice (4200). WER is computed with whisper-large-v3 and the Whisper English normaliser. Naturalness is UTMOS22. All cells are self-plan C3 and seed 1, with WER / UTMOS in each cell:

List (rows)	pure	hybrid	transformer
Seed-TTS test-en (1,088)	0.088 / 3.42	0.080 / 3.49	0.035 / 4.11
LibriSpeech-PC test-clean (1,127)	0.088 / 3.32	0.078 / 3.42	0.039 / 4.07
EmergentTTS-Eval, Emotions + Paralinguistics (560)	0.437 / 2.87	0.456 / 2.91	0.280 / 3.76
EmergentTTS-Eval, the other four categories (1,085)	0.458 / 3.15	0.450 / 3.19	0.274 / 4.13

The variants on Seed-TTS:

Variant (Seed-TTS WER)	pure	hybrid	transformer
C2	0.095	0.081	0.032
C3, training seed 2	0.074	0.073	0.031

What the results support:

• The transformer leads when the models write their own plans. This holds on every list, at both tiers and at both seeds. Every such contrast survives Holm correction over all 124 paired comparisons.
• The hybrid shows no reliable difference from the pure model.
• At training seed 1, the pure model's no-plan cells are 0.019 lower in WER than its self-plan cells. The Holm p is 1.00 on Seed-TTS and 0.005 on LibriSpeech-PC. This is an observation, not a finding: the two-stage models never trained a row without a plan, and the difference is absent at training seed 2.

The full tables, the Path B numbers and every interval are in eval/stats/numbers_stats_H.json. Each interval there comes with its sign test and Holm p.

Use

See code/README.md. In short:

hf download maxpicy/mamba-cot-tts --include "two_stage_v2/pure_c3/*" --local-dir mamba-cot-tts
hf download maxpicy/mamba-cot-tts --include "code/*" --local-dir mamba-cot-tts
hf download maxpicy/mamba-cot-tts --include "eval/prompt_sets/*" --local-dir mamba-cot-tts
bash mamba-cot-tts/code/decode.sh mamba-cot-tts/two_stage_v2/pure_c3 mamba-cot-tts/eval/prompt_sets/ext_seedtts_en.jsonl out selfplan

The codes become audio through the Fish S2 codec: codec.pth from fishaudio/openaudio-s1-mini (CC BY-NC-SA 4.0). It is not redistributed here.

Limitations

• English only and one voice per speaker id. There is no cloning from prompt audio, so speaker similarity (SIM) does not apply.
• Codebooks 1 to 9 come from our own depth module. Path A naturalness (UTMOS 2.9 to 4.1) is below current open TTS systems. Path B (path_b_renderer) raises the Mamba models' UTMOS to about 4.0 but leaves their WER where it was.
• This repository holds the WER, UTMOS and completion results. The listening tests and the EmergentTTS-Eval judge are reported in the paper.

Licence

Each part carries the most open licence its sources allow, and every one of them only asks for credit and citation. The weights and eval/ are CC BY-NC 4.0, because part of the training data (Emilia, EARS) is non-commercial. corpus/ and path_b_renderer/ are CC BY-NC-SA 4.0, because the corpus contains ParaSpeechCaps captions and the renderer is fine-tuned from openaudio-s1-mini, both share-alike. The code is MIT. LICENSE lists every part and every source.

Citation

Maximilian Wong Junlin, "[Paper title]," Final Year Project report, Nanyang Technological University, 2026. Available: https://www.ntu.edu.sg/

@misc{wong2026mambacot,
  author       = {{Maximilian Wong Junlin}},
  title        = {[Paper title]},
  howpublished = {Final Year Project report, Nanyang Technological University},
  year         = {2026},
  url          = {https://www.ntu.edu.sg/}
}