MOSS Voice-Acting β 40 Emotion LoRAs (v3)
40 PEFT/LoRA adapters, one per emotion, for
laion/moss-tts-local-transformer-4.55b-voice-acting-v2.
Rank 32, alpha 64, single-phase training.
v3 vs v2: Got-Talent data is capped at β€ 25 % per bucket, so β₯ ~60 % of each mix is natural/expressive non-GT material (EmoLia, DACVAE, gemini-adult, mitermix). This removes the synthetic "Got-Talent" character that v2 had.
These are the adapters used to produce the DramaBox reinterpretation corpus β ~20,000 acting prompts re-performed 64Γ each and reward-ranked.
Where everything lives
| 𧩠Base model (required) | laion/moss-tts-local-transformer-4.55b-voice-acting-v2 β these adapters are trained against v2 and will misbehave on the earlier β¦-voice-acting checkpoint |
| π¦ Model home, demos, prompting guide | github.com/LAION-AI/laion-moss-local-1.5-voice-acting-4.55b |
| π Prompting manual | projects.laion.ai/moss-voiceacting-manual Β· emotions chapter |
| π¬ Pipeline, training & measured learnings | github.com/LAION-AI/Voice-Acting-Pipeline-WIP Β· docs/17 |
| ποΈ Vocal-burst adapters (64 classes) | laion/vocal-burst-lora-adapters β stack with these for bursts inside emotional speech |
| π§ Hear them | 40-emotion best-of-64 grid Β· reinterpretations |
Quickstart
import torch, numpy as np, soundfile as sf
from transformers import AutoProcessor, AutoModel
from peft import PeftModel
BASE = "laion/moss-tts-local-transformer-4.55b-voice-acting-v2"
CODEC = "OpenMOSS-Team/MOSS-Audio-Tokenizer-v2"
EMO = "Pain" # any directory name in this repo
# NOTE: AutoModel, NOT AutoModelForCausalLM -- MossTTSLocalConfig is not registered for the
# CausalLM auto-class and from_pretrained raises "Unrecognized configuration class".
proc = AutoProcessor.from_pretrained(BASE, trust_remote_code=True, codec_path=CODEC)
model = AutoModel.from_pretrained(
BASE, trust_remote_code=True, dtype=torch.bfloat16,
attn_implementation="sdpa", # flash-attn 2.x is incompatible with this model
).cuda().eval()
pm = PeftModel.from_pretrained(
model, "TTS-AGI/moss-emotion-loras-v3", subfolder=EMO, adapter_name=EMO
).eval()
# `instruction` is the whole director's note; `text` is ONLY the spoken words.
# Empty fields render as the literal string "None", so fill them deliberately.
instruction = (
"GENERAL: A natural adult voice, clean studio capture, genuine unperformed delivery; "
"clearly carrying pain.\n"
"SCRIPT:\n"
'(pained, strained) "I told you I could handle it, but I really cannot."'
)
text = "I told you I could handle it, but I really cannot."
conv = [[proc.build_user_message(text=text, instruction=instruction,
language="English", tokens=len(text.split()))]]
batch = proc(conv, mode="generation")
with torch.no_grad():
out = pm.generate(
input_ids=batch["input_ids"].cuda(),
attention_mask=batch["attention_mask"].cuda(),
max_new_frames=220, do_sample=True,
text_temperature=0.7, text_top_k=50, text_top_p=1.0,
audio_temperature=1.0, audio_top_k=30, audio_top_p=0.95,
audio_repetition_penalty=1.1,
)
msg = proc.decode(out)[0]
# `audio_codes_list` already holds a DECODED waveform -- see "Traps" below.
w = msg.audio_codes_list[0].cpu().float().numpy()
if w.ndim > 1:
w = w.mean(0) # stereo -> mono
sf.write("out.wav", w, 48000)
audio_lm_heads.* and text_lm_head.weight reported MISSING when the base model loads is
benign β those heads are weight-tied. Do not try to "fix" it.
Controlling emotion strength (merge scale)
The adapter delta is added to the base weights, scaled by alpha / r β here 64 / 32 = 2.0.
Multiply that by a dose Ξ» to dial the emotion up or down:
from peft.tuners.lora import LoraLayer
# Capture the untouched scaling ONCE, right after loading. If you instead read the current
# value and multiply, the scale compounds every time you change it and silently drifts.
base_scaling = {n: dict(m.scaling) for n, m in pm.named_modules()
if isinstance(m, LoraLayer)}
def set_dose(adapter: str, lam: float):
for n, m in pm.named_modules():
if isinstance(m, LoraLayer) and adapter in m.scaling:
m.scaling[adapter] = base_scaling[n][adapter] * lam
set_dose(EMO, 0.5) # half strength
Ξ» = 1.5 destroys speaker identity β measured
If you are voice-cloning from a reference, the dose is not free. ECAPA speaker similarity between the generation and its reference clip, measured on a cross-lingual test:
| dose Ξ» | speaker similarity |
|---|---|
| 0.0 (no adapter) | 0.62 |
| 0.5 | 0.57 |
| 1.0 | 0.50 |
| 1.5 | β0.03 |
Anchors from the same encoder: a reference against itself scores 1.000, and two different speakers score 0.105. So at Ξ» = 1.5 the output voice has fallen below the unrelated-speaker floor β no relationship to the reference at all.
Guidance: if reference fidelity matters, keep Ξ» β€ 0.5. If you only care about the emotion and not the voice, 0.75β1.0 is the usable band. Higher is not better on this model.
Switching and stacking adapters
Swapping the active adapter costs ~0.021 s across 268 modules, so hot-swapping between emotions mid-batch is essentially free relative to generation. Load several by name and activate the combination you want:
pm.load_adapter("TTS-AGI/moss-emotion-loras-v3", subfolder="Fear", adapter_name="Fear")
pm.base_model.set_adapter(["Pain", "Fear"]) # both active
set_dose("Pain", 0.6); set_dose("Fear", 0.3)
The same mechanism stacks an emotion with a
vocal-burst adapter β e.g. Pain +
Pain Moan, or Fear + Scream.
Generate more than one take
Emotion strength, burst placement and prosody are seed-dependent. Best-of-N with a reward is how the reinterpretation corpus was built, and it is worth doing:
t_genis flat from batch 16 to 32 (20.34 s β 20.54 s on a GH200) because the decode loop is latency-bound, so 32 candidates cost the same wall time as 16. Batch 16 wastes half the GPU. Peak VRAM at batch 32 is ~23 GB.- Rank with
(blend + genuineness + target emotion) Γ (1 β min(WER, 1)). Never divide by(1 + WER)β most candidates have a negative core score, so the division form increases the reward as transcription gets worse. On a 1,000-clip control set a literalWER Γ qualityfilter put 602 of 1,000 candidates at exactly 0 and selected the half with worse WER.
The 40 emotions
seen = training samples the adapter saw; val_loss = final validation loss.
β οΈ val_loss is not comparable across emotions β each was trained and validated on its own
data, so a lower number does not mean a better adapter. Use it only to compare a run against
itself. seen is the number to watch: it ranges from 456 to 6,000, so
some adapters saw an order of magnitude less material than others and should be expected to be
weaker.
| emotion | seen | val_loss |
|---|---|---|
Longing |
6,000 | 4.583 |
Doubt |
2,424 | 4.621 |
Infatuation |
6,000 | 4.696 |
Contemplation |
6,000 | 4.752 |
Shame |
2,688 | 4.800 |
Bitterness |
752 | 4.855 |
Confusion |
6,000 | 4.862 |
Fatigue_Exhaustion |
6,000 | 4.947 |
Pain |
6,000 | 4.969 |
Awe |
1,144 | 4.992 |
Malevolence_Malice |
2,816 | 4.996 |
Disappointment |
1,696 | 5.032 |
Hope_Enthusiasm_Optimism |
6,000 | 5.039 |
Emotional_Numbness |
456 | 5.048 |
Relief |
6,000 | 5.106 |
Fear |
6,000 | 5.151 |
Sadness |
2,496 | 5.164 |
Interest |
6,000 | 5.177 |
Concentration |
6,000 | 5.181 |
Thankfulness_Gratitude |
6,000 | 5.188 |
Triumph |
6,000 | 5.190 |
Teasing |
5,424 | 5.217 |
Amusement |
6,000 | 5.243 |
Jealousy_and_Envy |
6,000 | 5.251 |
Pleasure_Ecstasy |
1,960 | 5.252 |
Astonishment_Surprise |
6,000 | 5.268 |
Distress |
6,000 | 5.274 |
Contempt |
2,008 | 5.282 |
Disgust |
3,096 | 5.285 |
Embarrassment |
1,776 | 5.310 |
Sexual_Lust |
6,000 | 5.310 |
Anger |
6,000 | 5.324 |
Helplessness |
2,400 | 5.350 |
Pride |
1,480 | 5.350 |
Affection |
6,000 | 5.399 |
Contentment |
2,136 | 5.412 |
Sourness |
752 | 5.448 |
Impatience_and_Irritability |
6,000 | 5.453 |
Intoxication_Altered_States_of_Consciousness |
6,000 | 5.518 |
Elation |
6,000 | 5.645 |
Aggregate: val_loss mean 5.148 (min 4.583, max 5.645);
seen median 6,000.
Caveats
- Emotion coverage is uneven. Adapters with only a few hundred training samples exist in this set; treat them as experimental rather than production-ready.
- Not every emotion transfers equally. An earlier 40-emotion sweep found roughly a quarter reached a strong lift; the rest are subtler. Listen before committing to one.
- These are v2-base adapters. Loading them onto the v1
β¦-voice-actingcheckpoint will produce degraded output rather than an error. - Scores quoted here come from model-based evaluators (VoiceCLAP heads, ECAPA, ASR), not human raters.
Citation / provenance
Trained by LAION as part of the MOSS voice-acting line. Base model and full experimental record: LAION-AI/laion-moss-local-1.5-voice-acting-4.55b.
- Downloads last month
- -