ModelsLab/midashenglm-gen-wer-lora

A LoRA adapter for mispeech/midashenglm-gen, trained across all five of the model's capabilities — speech, sound effects, music, ambience and mixed scenes — rather than trading one for another.

Hear it

Same prompt, same seed in both arms — generate() seeds the global RNG, so the solver draws identical noise and every audible difference is the adapter.

Speech, 4-word line — "The river remembers everything." The bucket that went 13.9% → 0.0% WER; the base arm garbles it.

base	scene-v8

Speech, 16-word line — both arms intelligible; the adapter is cleaner and tighter.

base	scene-v8

SFX — footsteps on gravel. Texture and caption match improve; the adapter clip runs shorter, the one real remaining cost.

base	scene-v8

Music — slow solo piano. The adapter plays longer and closer to the caption at this seed.

base	scene-v8

Ambience — steady rain on a metal roof. Near-identical by design: this arm held.

base	scene-v8

Why

Two weak spots in the base model, measured rather than assumed.

Short lines. A four-word line scores around 42% WER where a sixteen-word one scores near zero. The failures are onsets: the model needs a moment to settle and a short line does not give it one. Prompt-side remedies make it worse, so it is a property of the weights.

Sound effects. The model's worst benchmark — AudioCaps FAD 5.01 against TangoFlux's 2.26 — which the paper attributes to training on mixed scenes rather than dedicated sound-effect corpora.

Results

Same prompt, same seed, base weights against these adapters. generate() seeds the global RNG, so the flow-matching solver draws identical noise in both arms and every difference is the adapters.

capability	metric	base	tuned	change
speech	WER	5.6%	0.2%	-5.4%
sfx	CLAP text	0.4064	0.3837	-0.0226
sfx	CLAP real	0.4152	0.4201	+0.0049
music	CLAP text	0.4736	0.4673	-0.0063
music	CLAP real	0.5928	0.6012	+0.0084
ambience	CLAP text	0.2442	0.2070	-0.0371
ambience	CLAP real	0.3667	0.3732	+0.0065
mixed	WER	4.8%	4.7%	-0.1%
mixed	CLAP text	0.1708	0.1745	+0.0037
mixed	CLAP real	0.7014	0.6943	-0.0072
speech	WER, 4-word lines	13.9%	0.0%	-13.9%
speech	WER, 9-word lines	3.2%	0.0%	-3.2%
speech	WER, 16-word lines	1.9%	0.6%	-1.2%
speech	WER, 27-word lines	3.4%	0.3%	-3.1%
speech	mean length	4.73s	4.47s	-0.26s
sfx	mean length	8.08s	6.93s	-1.15s
music	mean length	9.42s	9.19s	-0.23s
ambience	mean length	10.43s	10.29s	-0.14s
mixed	mean length	4.97s	4.47s	-0.50s

CLAP text is "does it match the caption". CLAP real is cosine to the centroid of genuine MECAT audio of that category — "does it sound like the real thing", which text similarity alone cannot see. WER cannot see either: a clip with no words in it scores 0% however badly the room tone came out.

WER improved by 5.4 points, and that part is solid. The scene arms are not.

Caption adherence or realism fell on: sfx clap_text -0.023, ambience clap_text -0.037.

Use this for speech. For beds, effects and mixed scenes, A/B it against the base model on your own prompts first — the held-out flow loss improved on every capability, and on the scene arms that did not translate.

Training

Method	LoRA r=32, alpha=64, on LLM attention/MLP and the flow-matching DiT
Trainable	72876032 of 2.89 B
Precision	float32 with TF32 matmuls, matching how the base model is served
Optimiser	AdamW-8bit, lr 5e-05, cosine decay, grad clip 1.0
Steps	2500 at an effective batch of 16
Capability mix	sfx 22%, ambience 20%, speech 20%, mixed 20%, music 18%
Speech data	LibriTTS-R + MECAT S00; clips under 3 s oversampled 2.5x
Scene data	MECAT-Caption, all eight categories
Anchor	vector field pulled towards the frozen base, weight 0.5 on 35% of batches

Base-model hyperparameters follow arXiv:2608.11804 §3.3 where they apply (lambda_stop = 0.01, grad clip 1.0).

Avoiding catastrophic forgetting

Four things, because one is not enough:

1. LoRA rather than a full fine-tune. Rank 32 bounds how far the weights can move at all.
2. A capability-weighted mix. The sampler draws by capability, not by pool. At natural frequency the speech buckets are ~80% of the clips and the model would simply get better at reading aloud.
3. An anchor loss. The tuned vector field is pulled towards the frozen base's on identical noise and timesteps — a trust region around the base model, stated directly rather than hoped for.
4. A per-capability gate. Held-out flow loss is tracked for each capability separately, and this checkpoint was only saved because the mean improved and no single capability regressed past 2%.

Use

from peft import PeftModel
from transformers import AutoModel
import soundfile as sf

model = AutoModel.from_pretrained("mispeech/midashenglm-gen", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "ModelsLab/midashenglm-gen-wer-lora").merge_and_unload().cuda().eval()

result = model.generate(
    "<|caption|> A close, clean recording of a single narrator, with nothing behind it. "
    "<|asr|> The river remembers everything. "
    "<|speech|> A woman in her forties, warm and even, speaking clearly. "
    "<|sfx|> <|unknown|> <|music|> <|unknown|> <|env|> <|unknown|>",
    seed=11,
)
sf.write("out.wav", result["audio"], result["sample_rate"])

Tag order matters more than anything else in the prompt: <|asr|> comes before <|speech|>. Reversed, the model produces fluent unrelated speech — 14.2% mean WER against 373%.

Licences

Apache 2.0, following the base model. Training data: LibriTTS-R (CC-BY-4.0) and MECAT-Caption (CC-BY-3.0), both attribution-only.

Training code: audio-scenegen/training.