TK3105/UniAVC

Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling

Kien T. Pham¹, Yuhao Liu², Chong Hou Choi², Zhenqi He¹, Yanghao Wang¹, Chenyang Lei², Rui Liu², Qifeng Chen¹, Long Chen¹

¹The Hong Kong University of Science and Technology    ²Huawei Research

tkpham@connect.ust.hk  ·  longchen@ust.hk

---

🎬 Demo

---

Overview

UniAVC is a unified model that natively handles text-conditioned joint audio-video generation, bidirectional cross-modal synthesis, and instruction-guided editing of either or both modalities without relying on task-specific adapters. To achieve this versatility, the model utilizes a progressive training paradigm that first masters unimodal generation and editing capabilities before dedicating scarce paired datasets entirely to establishing cross-modal correspondence and temporal synchronization. The framework further introduces an asymmetric noise-level scheduling technique that derives decoupled per-branch noise trajectories and explicitly supervises extreme boundary states to enable intent-aware editing and post-inference self-refinement. Extensive empirical evaluations demonstrate that this unified approach delivers performance comparable to specialized, state-of-the-art baselines across a wide range of generation and editing benchmarks.

Highlights

Comparison with the state of the art

One sample per editing family, applying the same instruction with UniAVC and with InstructAV2AV and MiniMax-H3.

Subject Editing

Transform the man into a cyberpunk painting style with high-saturation neon colors and sharp digital illustration strokes, while keeping all characters, actions, and composition unchanged.

Input	UniAVC
InstructAV2AV	MiniMax-H3

Subject Addition

Add a person wearing white clothes, standing still in the center of the frame, looking at the camera with a blank expression.

Input	UniAVC
InstructAV2AV	MiniMax-H3

Subject Removal

Remove the woman wearing a white lab coat, a stethoscope, and looking confused or distressed from the source video, while preserving the background of a blurred office environment, a desk lamp, the wall calendar, and the shelving behind her.

Input	UniAVC
InstructAV2AV	MiniMax-H3

Speech Editing

Alter the character's dialogue to: "We were in the drum circle like, wow, this is where real musical magic happens right now!" while keeping visual elements and background sounds identical to the original scene.

Input	UniAVC
InstructAV2AV	MiniMax-H3

Background Editing

Replace the background with a private study at late night, with quiet city night lights visible outside the window, while keeping the person's image, clothing, actions, and contemplative head-down expression in the foreground completely consistent.

Input	UniAVC
InstructAV2AV	MiniMax-H3

Editing versatility

Each grid holds the source clip and the results of three different instructions applied to that same clip.

Subject Editing

Restyle the subject while preserving background, lighting and posture.

Source	1. Racing suit instead of the T-shirt
2. Astronaut pressure suit instead of the T-shirt	3. Black wetsuit with blue shoulder panels

Subject Addition

Insert a new person into the shot.

Source	1. Add a man with short brown hair
2. Add a woman with long dark hair	3. Add an older man with white hair and glasses

Subject Removal

Erase a person and reconstruct what was behind them. This family ships two examples with one instruction each.

Source	1. Remove the man in the white shirt and dark tie
Source	2. Remove the woman with the blue patterned scarf

Speech Editing

Replace a line of dialogue with new words and matched lip-sync.

Source	1. "Scroll back to the log from Tuesday afternoon."
2. "That value should never have gone negative."	3. "Look at the screen, the error code matches the original test."

Background Editing

Relocate the background scene, keeping subject appearance and expression intact.

Source	1. A tranquil beach at dusk
2. A rooftop at dusk, laundry and swifts	3. A railway platform at dusk, a train pulling away

Joint generation

Installation

conda create -n uniavc python=3.10 -y
conda activate uniavc

# Install a torch build matching your CUDA, then the rest.
pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0 --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt
pip install -e .

Tested environment: Python 3.10, CUDA 12.8, PyTorch 2.8, on a single 80 GB GPU.

Model weights

bash scripts/download_weights.sh fetches the weights, plus Gemma-3 for text conditioning:

bash scripts/download_weights.sh

Or by hand:

pip install -U huggingface_hub

# The released model.
hf download TK3105/UniAVC model.safetensors \
    --local-dir models/UniAVC

# Gemma-3 text encoder.
hf download google/gemma-3-12b-it-qat-q4_0-unquantized \
    --local-dir models/gemma-3-12b-it-qat-q4_0-unquantized

Once downloaded, the weights are stored as follows:

models/
├── UniAVC/
│   └── model.safetensors                  # the DiT + the VAEs, vocoder, post-modules
└── gemma-3-12b-it-qat-q4_0-unquantized/

Usage

Joint audio-video editing

Edit the video and its soundtrack from a single instruction. Each of the five families gets one clip and one instruction:

# Subject Editing
python inference/infer_tav2av.py \
    --video samples/sample_1.mp4 \
    --prompt "Replace the man's dark brown T-shirt with a blue cycling jersey with white stripes on the sleeves. Keep the pull-up bar, the beige wall and the lighting unchanged." \
    -o sample_1_out.mp4

# Subject Addition
python inference/infer_tav2av.py \
    --video samples/sample_2.mp4 \
    --prompt "Include a man in a grey coat seated in the front seat, saying: \"We should have left an hour ago. Nobody checked the traffic or the weather.\"" \
    -o sample_2_out.mp4

# Subject Removal
python inference/infer_tav2av.py \
    --video samples/sample_3.mp4 \
    --prompt "Remove the table lamp from the foreground and fill in the window and curtains behind it. Keep the man and his dialogue unchanged." \
    -o sample_3_out.mp4

# Speech Editing
python inference/infer_tav2av.py \
    --video samples/sample_4.mp4 \
    --prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \
    -o sample_4_out.mp4

# Background Editing
python inference/infer_tav2av.py \
    --video samples/sample_5.mp4 \
    --prompt "Replace the background with a warm, sunlit kitchen with wooden cabinets. Keep the man, his clothing and his speech unchanged." \
    -o sample_5_out.mp4

The two headline modes are extra flags on this script.

Post-inference self-refinement: re-denoise only one branch against the other, held as an accepted reference.

python inference/infer_tav2av.py \
    --video samples/sample_4.mp4 \
    --prompt "Have her say: \"I did not expect the city to be this quiet at dawn.\"" \
    --post-refine audio --refine-lambda 0.5 \
    -o sample_4_refined.mp4

At the default --refine-lambda 0.5 this runs round(0.5 * 30) = 15 refinement steps. The count scales with lambda so the refine pass keeps the main pass's per-step resolution; --refine-steps overrides it.

Intent-aware editing: hold one branch intact by conditioning, and let it constrain the other.

python inference/infer_tav2av.py \
    --video samples/sample_5.mp4 \
    --prompt "The room is now a quiet library, with faint page turns and distant footsteps." \
    --intent-aware video \
    -o sample_5_audio_edit.mp4

Joint audio-video generation

No input media: both branches are generated from the prompt alone.

python inference/infer_t2av.py \
    --prompt "A man in a grey shirt sits at a wooden table in a quiet cafe, a window with soft daylight behind him and a few empty chairs further back. He says: \"We can start whenever you are ready, but let me finish this first.\" Static medium shot." \
    -o sample_t2av.mp4

Audio → video generation

Given a soundtrack, generate the matching video; the input audio is preserved.

python inference/infer_ta2v.py \
    --audio samples/sample_4.mp4 \
    --prompt "Medium close-up of a woman with long dark hair speaking directly to camera, standing in a sunlit kitchen with pale oak cabinets and a steaming kettle on the counter behind her, warm late-afternoon light through a side window, shallow depth of field, static shot." \
    -o sample_4_ta2v.mp4

Video → audio generation

Given a video, generate the matching soundtrack; the input video is preserved.

python inference/infer_tv2a.py \
    --video samples/sample_1.mp4 \
    --prompt "The man is out of breath after a set of pull-ups. A barbell is racked nearby in a bare gym, with voices and movement echoing off a hard wall." \
    -o sample_1_tv2a.mp4

Options

Every option has a working default, and the commands above need nothing beyond --video / --audio / --prompt.

Output

Option	Default	Notes
--height, --width	704, 1280	Both must be multiples of 32.
--num-frames	121	Must be 8k+1.
--frame-rate	24
-o, --output	<task>_output.mp4

Sampling and memory

Option	Default	Notes
--num-inference-steps	30	Denoising steps.
--seed	42
--offload	on	--no-offload keeps all weights resident; see Hardware.
--device	cuda
--no-tiled	off	Decode the video without VAE tiling. Needs more VRAM.

Guidance, set per branch.

Task	video-text	audio-text	video-reference	audio-reference
tav2av	3.0	2.0	1.5	3.0
t2av, ta2v, tv2a	6.0	6.0	1.0	1.0

The flags are --video-text-cfg-scale, --audio-text-cfg-scale, --video-reference-cfg-scale and --audio-reference-cfg-scale.

Demo

A Gradio app is included:

python app.py

Hardware

The released checkpoint is a 19B-parameter DiT in bfloat16, and text conditioning adds Gemma-3 12B. The two together are ~65 GB, so weights are offloaded by default: each phase loads only what it needs (the text encoder while encoding, the DiT while denoising, the VAEs while decoding) and the rest waits on CPU. Peak residency is the DiT alone, ~39 GB, plus activations. --no-offload keeps everything on the card for speed and needs substantially more VRAM than a single 80 GB GPU provides at the default 720p resolution.

Acknowledgements

UniAVC is built on DiffSynth-Studio, whose model-loading framework and LTX-2.3 implementation the vendored uniavc/ package derives from. The base generative model is LTX-2.3 from Lightricks. Text conditioning uses Gemma-3 from Google. We thank JAVEdit for the useful training data and their benchmark for evaluation; MiniMax-H3, InstructAV2AV for the comparison baselines; and the wider open-source audio-video community that make this work complete.

Citation

If you find UniAVC useful for your research, please consider giving a star ⭐ and citation 📝 :D

@article{pham2026uniavc,
  title   = {UniAVC: Unified Audio-Video Creation via Progressive Training and Asymmetric Scheduling},
  author  = {Pham, Kien T. and Liu, Yuhao and Choi, Chong Hou and He, Zhenqi and Wang, Yanghao
             and Lei, Chenyang and Liu, Rui and Chen, Qifeng and Chen, Long},
  journal = {arXiv preprint arXiv:2610.24791},
  year    = {2026}
}

License

LTX-2 Community License.