laion/vocal-burst-lora-adapters

Vocal-Burst LoRA Adapters (Gemini-Confirmed, 64 classes)

64 PEFT/LoRA adapters for laion/moss-tts-local-transformer-4.55b-voice-acting-v2, one per non-verbal vocal-burst class (laughs, sighs, gasps, whistles, breathing, grunts, moans, etc.), trained on classifier-filtered DramaBox audio and selected from a pool of 77 trained classes using a listening evaluation, not the automatic metric alone.

Each class ships as a single adapter directory (its own adapter_model.safetensors + adapter_config.json) at the rank and training epoch that scored best. classes.json has the full per-class metadata; inference.py is a runnable example of how to load the base model and one adapter and generate a clip.

Where everything lives

🎧 Listen — all 77 classes	projects.laion.ai/…/vocal_burst_loras.html — real recordings vs the no-LoRA baseline vs the top 3 per rank, every clip with its scores. The 13 rejected classes and the 26 the automatic metric cannot see are flagged individually.
🧩 Base model (required)	laion/moss-tts-local-transformer-4.55b-voice-acting-v2 — these adapters are trained against v2 and will not behave correctly on the earlier …-voice-acting checkpoint.
📦 Model home / demos	github.com/LAION-AI/laion-moss-local-1.5-voice-acting-4.55b — model card, prompting guide, and all listening grids.
🔬 Pipeline & learnings	github.com/LAION-AI/Voice-Acting-Pipeline-WIP — training/eval code and the measured write-up: docs/17_lora_and_reinterpretation_learnings.md (merge-dose vs speaker identity, best-of-N returns, the metric-blindness finding, and the silent-failure traps).
📖 Prompting manual	projects.laion.ai/moss-voiceacting-manual

One result worth knowing before you pick a merge scale

Merging an adapter harder does not monotonically improve things. Measured on cross-lingual voice cloning, ECAPA speaker similarity to the reference falls as the emotion-adapter merge dose rises — 0.62 at dose 0, 0.57 at 0.5, 0.50 at 1.0, and −0.03 at 1.5, against an unrelated-speaker floor of 0.105. At dose 1.5 the output voice has no relationship to the reference at all. If reference-voice fidelity matters, keep emotion merges ≤ 0.5. (Burst adapters on burst-only clips behaved differently — full dose won every class there — so treat the two cases separately.)

Why 64 of 77, and why a listening judge

The training run produced 77 class adapters, each at 3 LoRA ranks (r16/r32/r64) and up to 6 epochs — about 186 GB in total. Before publishing anything, every configuration was scored two different ways:

1. An automatic evaluator (a burst locator + a vocal-burst classifier v2, blended into a composite score) run over held-out prompts.
2. A gemini-3-flash listening judge, which listened to the top-3 generated clips per class per rank and rated match-to-description on a 0/1/2 scale — with real recordings of that class and the no-LoRA base model included as blind controls, so a class scoring 2.0 could be checked against something.

The controls are the reason the judge scores can be trusted:

control	mean judge score
real recordings	1.874
base model (no LoRA)	0.411
LoRA adapters (all ranks)	1.755

The judge clearly discriminates — it does not rate everything a 2 — and LoRA clips land much closer to real recordings than to the unadapted base model.

The automatic evaluator cannot recognise 26 of the 77 classes even in real human recordings — their anchor composite (the automatic score computed on real, held-out recordings of that class) is below 0.10, and six are exactly 0.000. These are mostly quiet mouth/breath sounds (whistles, hiccups, breathing, lip/kissing sounds) that the locator or classifier front-end simply doesn't register as bursts. For those 26 classes, the automatic "gain" from training measures the metric's blindness, not the adapter's quality — a class that scores worse or unchanged on the automatic metric could still be a well-trained, audibly correct adapter, and there is no way to tell from the automatic number alone.

19 of those 26 metric-blind classes were confirmed by the listening judge and are included below — including every whistle class (Wolf Whistle, Sharp Whistle, Soft Whistle, Person Whistling Playfully, Person Whistling to Get Attention), all of which the automatic metric scored as regressions relative to the base model. Listening to them tells a different story, which is the entire reason this repo uses a listening judge instead of shipping (or rejecting) classes on the automatic score alone.

Selection rule

Publish only classes with a mean LoRA judge score lora >= 1.5 (out of 2.0, averaged across all rank/epoch clips judged for that class). This produced exactly 64 of 77 classes (verified directly against gem/vb_judge.json, not assumed) — the same 64 shipped here.

For each of the 64, we ship one rank and one epoch:

• Rank: whichever of r16/r32/r64 got the higher listening-judge score for that class (by_rank in the judge data); ties are broken by the automatic composite score.
• Epoch: whichever epoch the automatic evaluator identified as best for that chosen rank (max composite over the epochs trained for that rank/class).

This means the repo ships neither "all ranks for all classes" (which would triple size for no benefit) nor a single global rank/epoch — each class gets its own best configuration.

The 13 rejected classes

These scored below 1.5 on the listening judge and are not included in this repo. They are dominated by tongue clicks, lip smacks, licking and sucking sounds — extremely short, low-energy mouth noises the model struggled to reproduce distinctly. Purr is the most severe case: it failed with only 13 usable training items, and its real recordings also score 0 on the automatic anchor metric — i.e. the source data for that class appears to be bad or mislabeled, not merely the adapter.

Class	Slug	Mean LoRA judge score	Anchor (real) composite	Clips judged
Purr	purr	0.000	0.000	12
Tongue Click	tongue_click	0.667	0.000	12
Smacks Lips	smacks_lips	0.750	0.000	12
Chewing Noises	chewing_noises	0.833	0.013	12
Clicks Tongue	clicks_tongue	0.833	0.003	12
Nervous Gulp	nervous_gulp	0.833	0.095	12
Licking Sound	licking_sound	0.909	0.000	11
Click One's Tongue	click_one_s_tongue	1.000	0.000	12
Whispered Mumble	whispered_mumble	1.083	0.055	12
Sucking Noise	sucking_noise	1.250	0.011	12
Tsk	tsk	1.250	0.008	12
Kissing Sounds	kissing_sounds	1.333	0.062	12
Ahem	ahem	1.455	0.319	11

The 64 published classes

rank/epoch are the ones actually shipped in this repo. judge score is the Gemini listening-judge score (0–2) for that specific rank; judge mean (all ranks) is the class's overall mean used for the >=1.5 cutoff. automatic composite is the automatic evaluator's score for the shipped rank/epoch; anchor composite is the automatic evaluator's score on real recordings of that class (a low anchor means the automatic metric can't reliably "see" this class at all — see metric_blind). n train items is the number of classifier-agreed training examples actually used for that class (see caveats below).

Class	Slug	Rank	Epoch	Judge score	Judge mean (all ranks)	Automatic composite	Anchor composite	n train items	metric-blind
Affirmative Grunt	affirmative_grunt	r32	2	2.00	2.00	0.159	0.314	176
Breathy Giggle	breathy_giggle	r64	1	2.00	2.00	0.683	0.705	1300
Cackle	cackle	r64	2	2.00	2.00	0.239	0.437	249
Childlike Giggle	childlike_giggle	r64	2	2.00	2.00	0.640	0.644	250
Chuckle	chuckle	r64	1	2.00	2.00	0.595	0.716	1396
Clears Throat	clears_throat	r32	3	2.00	2.00	0.223	0.306	193
Contented Sigh	contented_sigh	r16	2	2.00	2.00	0.482	0.463	475
Convulsive Sob	convulsive_sob	r32	1	2.00	2.00	0.201	0.272	186
Cough	cough	r32	2	2.00	1.83	0.219	0.304	199
Coughing	coughing	r32	3	2.00	1.67	0.188	0.288	202
Deep Breath	deep_breath	r64	1	2.00	2.00	0.136	0.027	117	yes
Deep Breathing	deep_breathing	r16	1	2.00	2.00	0.113	0.018	86	yes
Displeased Grunt	displeased_grunt	r32	2	2.00	1.83	0.150	0.269	186
Drinking Noises	drinking_noises	r64	2	2.00	2.00	0.205	0.096	137	yes
Effort Grunt	effort_grunt	r16	1	2.00	1.73	0.175	0.250	187
Exasperated Sigh	exasperated_sigh	r16	3	2.00	2.00	0.171	0.262	188
Exhausted Groan	exhausted_groan	r64	4	2.00	2.00	0.489	0.560	656
Fast Breathing	fast_breathing	r32	2	2.00	2.00	0.225	0.174	157
Fearful Gasp	fearful_gasp	r64	2	2.00	2.00	0.202	0.242	178
Frustrated Groan	frustrated_groan	r64	3	2.00	1.92	0.131	0.254	191
Growl	growl	r64	2	2.00	2.00	0.130	0.149	172
Guffaw	guffaw	r64	2	2.00	2.00	0.355	0.307	202
Gulps	gulps	r32	2	2.00	2.00	0.096	0.113	139
Gurgling	gurgling	r16	2	2.00	1.83	0.033	0.010	171	yes
Heavy Breathing	heavy_breathing	r64	3	2.00	2.00	0.231	0.210	144
Hiccup	hiccup	r16	3	2.00	1.82	0.081	0.071	135	yes
Hiccups	hiccups	r64	1	2.00	1.82	0.103	0.045	126	yes
Hiss	hiss	r16	1	2.00	2.00	0.086	0.007	155	yes
Humming	humming	r64	3	2.00	2.00	0.276	0.353	193
Kissing Noises	kissing_noises	r64	2	2.00	1.83	0.055	0.051	94	yes
Lip Smack	lip_smack	r16	1	2.00	1.83	0.100	0.004	81	yes
Low Mumble	low_mumble	r16	2	2.00	1.92	0.426	0.269	197
Mournful Wail	mournful_wail	r64	3	2.00	1.83	0.177	0.206	169
Nervous Giggle	nervous_giggle	r32	2	2.00	2.00	0.263	0.335	188
Normal Breathing	normal_breathing	r64	3	2.00	2.00	0.151	0.053	111	yes
Pain Moan	pain_moan	r64	3	2.00	1.83	0.109	0.160	176
Panting	panting	r16	2	2.00	2.00	0.225	0.190	156
Person Whistling Playfully	person_whistling_playfully	r16	1	2.00	2.00	0.060	0.011	120	yes
Person Whistling to Get Attention	person_whistling_to_get_attention	r32	3	2.00	2.00	0.080	0.032	156	yes
Pleasure Moan	pleasure_moan	r32	2	2.00	2.00	0.236	0.242	184
Quiet Sob	quiet_sob	r16	3	2.00	2.00	0.245	0.254	156
Relief Sigh	relief_sigh	r32	2	2.00	1.83	0.224	0.192	149
Resonant Hum	resonant_hum	r16	1	2.00	2.00	0.623	0.609	179
Scream	scream	r32	5	2.00	2.00	0.313	0.405	185
Sharp Inhale	sharp_inhale	r32	2	2.00	2.00	0.283	0.279	370
Sharp Whistle	sharp_whistle	r64	2	2.00	2.00	0.086	0.078	155	yes
Shriek	shriek	r64	1	2.00	1.83	0.155	0.180	174
Slow Breathing	slow_breathing	r32	2	2.00	1.91	0.129	0.016	84	yes
Slurping Noises	slurping_noises	r16	2	2.00	1.50	0.022	0.000	113
Smack One's Lips	smack_one_s_lips	r16	1	2.00	2.00	0.108	0.003	102	yes
Snicker	snicker	r64	3	2.00	2.00	0.272	0.334	192
Sniff	sniff	r16	2	2.00	2.00	0.187	0.172	198
Snort	snort	r32	1	2.00	1.82	0.207	0.174	190
Snorting Giggle	snorting_giggle	r64	2	2.00	2.00	0.227	0.323	180
Sobs	sobs	r32	2	2.00	1.83	0.218	0.298	178
Soft Hum	soft_hum	r64	3	2.00	2.00	0.468	0.527	192
Soft Whistle	soft_whistle	r16	2	2.00	2.00	0.063	0.017	111	yes
Spitting	spitting	r32	2	2.00	1.50	0.132	0.082	139	yes
Surprised Gasp	surprised_gasp	r64	2	2.00	1.67	0.621	0.576	1351
Swallows	swallows	r32	1	2.00	1.50	0.064	0.041	75	yes
Trembling Whimper	trembling_whimper	r16	2	2.00	2.00	0.243	0.268	177
Wistful Sigh	wistful_sigh	r64	1	2.00	2.00	0.496	0.485	598
Wolf Whistle	wolf_whistle	r64	2	2.00	2.00	0.094	0.050	172	yes
Yawn	yawn	r64	1	2.00	1.83	0.468	0.338	202

(sorted by judge score, descending; ties broken arbitrarily)

Caveats — read before relying on these adapters

• Training data is small and heavily filtered. Across all 77 classes: 97,246 raw DramaBox items → 16,689 usable (17.2%). An item was kept only if an independent classifier agreed with its label. 15 classes have under 100 usable training items (several published classes are in this group — see the n train items column above, e.g. Swallows at 75, Lip Smack at 81, Slow Breathing at 84, Deep Breathing at 86, Kissing Noises at 94). Adapters for these classes are trained on very little data and should be treated as lower-confidence than the headline numbers suggest.
• The training-data filter is circular. Items were kept when a classifier from the same model family used elsewhere in this pipeline agreed with the human/source label. This self-reinforcing filter could systematically exclude atypical-but-correct examples of a class and/or retain examples that merely look classifiable rather than being representative, which is part of why a separate, independent (listening) judge was used for the final publish/reject decision rather than trusting the automatic pipeline end-to-end.
• The Gemini judge is a single model, not a human panel. It rated 0/1/2 on at most 9 clips per class (top-3 clips x 3 ranks), which is a strong, control-validated signal but not a substitute for human evaluation at scale. Treat class-level scores as indicative, not as ground truth.
• Not all adapters are converged. Training ran for 3 epochs by default (a handful of classes were extended to 6). Roughly 40% of classes were still improving on the automatic metric when training stopped — for those classes, the shipped checkpoint is the best one produced so far, not necessarily the best one that further training could produce.
• The automatic evaluator is not a reliable arbiter on its own for roughly a third of these classes (see metric_blind column) — those numbers are reported here for completeness and reproducibility, but the actual publish/reject and rank/epoch choices for those classes leaned on the listening judge, since the automatic composite is close to meaningless when the anchor (real-recording) score is itself near zero.

Inference

See inference.py for a complete, runnable example. Key points it documents in detail (all learned during training/eval of these adapters):

• Load the base model with attn_implementation="sdpa" — flash-attn 2.x is incompatible.
• audio_lm_heads.* and text_lm_head.weight reported MISSING on load is benign (the heads are weight-tied internally).
• Build prompts with the processor's build_user_message(text=..., instruction=..., tokens=..., language=...), not chat_template.jinja. instruction is the full "GENERAL: ...\nSCRIPT:\n..." caption; text is only the spoken words (None for vocal bursts, which the processor renders as the literal string "None" — this is expected and matches training).
• The 0.16s trap: AssistantMessage.audio_codes_list already holds decoded waveforms on the output side (Processor.decode() calls decode_audio_codes() internally). Calling decode_audio_codes() on it again silently returns 7,680 samples = 0.16s for every clip, regardless of true length. Use the tensor directly and mean over the channel axis.

from transformers import AutoProcessor, AutoModel
from peft import PeftModel
import torch

proc = AutoProcessor.from_pretrained(
    "laion/moss-tts-local-transformer-4.55b-voice-acting-v2",
    trust_remote_code=True, codec_path="OpenMOSS-Team/MOSS-Audio-Tokenizer-v2")
proc.audio_tokenizer = proc.audio_tokenizer.to("cuda").eval()
model = AutoModel.from_pretrained(
    "laion/moss-tts-local-transformer-4.55b-voice-acting-v2",
    trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa").cuda().eval()

adapter = PeftModel.from_pretrained(model, "./wolf_whistle", adapter_name="wolf_whistle")

msg = proc.build_user_message(
    text=None,
    instruction="GENERAL: A person's voice. Clean studio recording.\nSCRIPT:\n(Wolf Whistle, 0.8 seconds long)",
    tokens=10, language="English")
batch = proc([[msg]], mode="generation")
out = adapter.generate(input_ids=batch["input_ids"].cuda(),
                        attention_mask=batch["attention_mask"].cuda(),
                        max_new_frames=90, do_sample=True,
                        text_temperature=0.7, audio_temperature=1.0,
                        audio_top_k=30, audio_top_p=0.95, audio_repetition_penalty=1.1)
wav = proc.decode(out)[0].audio_codes_list[0].float().cpu().numpy().mean(0)  # already decoded -- do not re-decode

See inference.py for the full, runnable version with error handling and prompt-building helpers.

Repository layout

<class_slug>/adapter_model.safetensors
<class_slug>/adapter_config.json     # one directory per class, 64 total
classes.json                          # full per-class metadata (this README's tables, machine-readable)
inference.py                          # runnable load + generate example
README.md                             # this file

License

CC-BY-4.0.

Base model / acknowledgements

Base model: laion/moss-tts-local-transformer-4.55b-voice-acting-v2. Adapters trained with PEFT/LoRA on classifier-filtered DramaBox audio. Automatic evaluation used an internal burst locator + laion/voiceclap-commercial classifier/blend/genuineness heads. Listening evaluation used gemini-3-flash as an audio judge with real-recording and base-model controls.