ChristianYang/Fine-Tuned-WAV-VAE

Fine-Tuned LongCat WavVAE (decoder conv-LoRA, env-controllable)

Decoder of the LongCat-AudioDiT-1B WavVAE (24 kHz, 64-ch latent, ~11.7 Hz), fine-tuned with conv-LoRA (rank 16) to better render reverberant / noisy speech for the Env-TTS task.

• Base: meituan-longcat/LongCat-AudioDiT-1B WavVAE
• Encoder: frozen (unchanged from LongCat — latent geometry preserved, DiT-safe)
• Decoder: conv-LoRA on all 37 weight_norm Conv1d/ConvTranspose1d layers, merged into base weights
• Data: ChristianYang/Env-TTS-Clean speech, 50/25/25 clean / noise / noise+reverb (DNS-Challenge noise + RIR)
• Recipe: GAN (EncodecDiscriminator) + multi-res STFT + mel + L1 + KL; disc-warmup -> LoRA lr ramp 0->1e-3 -> cosine decay
• Steps: 16k

Files

• fine_tuned_wavvae.pt — full VAE state_dict (encoder.* + decoder.*, 363 keys) for env-tts AudioDiTVae
• wavvae_decoder_merged_lora.pt — decoder-only (layers.*, 182 keys)

Usage (env-tts)

import torch
model.vae.load_state_dict(torch.load("fine_tuned_wavvae.pt"))                 # full VAE
model.vae.decoder.load_state_dict(torch.load("wavvae_decoder_merged_lora.pt"))  # decoder only

Note

Aggregate training loss stayed ~flat (GAN equilibrium + frozen-encoder reconstruction floor); evaluate the reverb/noise gain via reverb LSD (env3 fidelity), not the training loss.