Upload 9 files

.gitignore

@@ -0,0 +1,144 @@
+run_*.sh
+log/
+saves
+saves/
+output/
+output
+pretrained/
+workspace
+workspace/
+.checkpoints/
+weights/
+ext_weights/
+*.pth
+*.pt
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

README.md

@@ -1,12 +1,129 @@
 ---
-title: Automation
-emoji: 🌖
-colorFrom: red
+title: VEO3 Free
+emoji: 🔊
+colorFrom: blue
 colorTo: indigo
 sdk: gradio
-sdk_version: 6.20.0
+sdk_version: 5.35.0
 app_file: app.py
 pinned: false
+short_description: Wan2.1-T2V-14B + Fast 4-step with NAG + Automatic Audio
+models:
+  - VIDraft/Gemma-3-R1984-4B
+  - google/gemma-3-4b-it
+  - Wan-AI/Wan2.1-T2V-14B-Diffusers
+  - vrgamedevgirl84/Wan14BT2VFusioniX
+  - Kijai/WanVideo_comfy  
 ---
+## English Explanation
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+### Overview
+This is a **VEO3 Free** application - an advanced AI video generation system that combines Wan2.1-T2V-14B model with automatic audio generation capabilities. It creates videos from text descriptions and automatically generates matching audio using MMAudio technology.
+
+### Key Features
+
+1. **Text-to-Video Generation**
+   - Uses Wan2.1-T2V-14B Diffusion model (14 billion parameters)
+   - Fast 4-step generation with NAG (Noise-Augmented Generation)
+   - Supports various resolutions from 128x128 to 896x896
+   - Duration: 1-8 seconds at 16 FPS
+   - Cinema-quality output with professional camera movements
+
+2. **Automatic Audio Generation**
+   - MMAudio integration for synchronized sound effects
+   - Uses the same text prompt for both video and audio
+   - Configurable audio quality and guidance strength
+   - Optional feature - can be disabled if needed
+
+3. **Advanced Controls**
+   - **NAG Scale**: Controls guidance strength (1.0-20.0)
+   - **Inference Steps**: Balances quality vs speed (1-8 steps)
+   - **Seed Control**: For reproducible results
+   - **Negative Prompts**: Specify what to avoid in generation
+
+### How It Works
+1. **Input**: Enter a detailed scene description
+2. **Video Generation**: The AI creates video frames based on your prompt
+3. **Audio Synthesis**: Automatically generates matching sound effects
+4. **Output**: Combined video with synchronized audio
+
+### Example Use Cases
+- Film previews and concept visualization
+- Music video creation
+- Advertising content
+- Creative storytelling
+- Game cinematics
+
+### Technical Details
+- **GPU Acceleration**: Uses CUDA for fast processing
+- **Model Architecture**: Transformer-based diffusion model
+- **Audio Model**: Flow-matching based audio synthesis
+- **Processing Time**: ~30-70 seconds depending on settings
+
+### Tips for Best Results
+- Use detailed, cinematic descriptions
+- Include camera movements and visual style
+- Specify lighting, colors, and atmosphere
+- Add sound descriptions for better audio matching
+- Higher NAG scale = more prompt adherence
+
+---
+
+## 한글 설명
+
+### 개요
+**VEO3 Free**는 Wan2.1-T2V-14B 모델과 자동 오디오 생성 기능을 결합한 고급 AI 비디오 생성 시스템입니다. 텍스트 설명으로부터 비디오를 생성하고 MMAudio 기술을 사용해 자동으로 일치하는 오디오를 생성합니다.
+
+### 주요 기능
+
+1. **텍스트-비디오 변환**
+   - Wan2.1-T2V-14B Diffusion 모델 사용 (140억 파라미터)
+   - NAG(노이즈 증강 생성)를 통한 빠른 4단계 생성
+   - 128x128부터 896x896까지 다양한 해상도 지원
+   - 지속 시간: 16 FPS로 1-8초
+   - 전문적인 카메라 움직임을 포함한 영화 품질 출력
+
+2. **자동 오디오 생성**
+   - 동기화된 사운드 효과를 위한 MMAudio 통합
+   - 비디오와 오디오 모두 동일한 텍스트 프롬프트 사용
+   - 오디오 품질과 가이던스 강도 조절 가능
+   - 선택적 기능 - 필요시 비활성화 가능
+
+3. **고급 제어 기능**
+   - **NAG 스케일**: 가이던스 강도 제어 (1.0-20.0)
+   - **추론 단계**: 품질 대 속도 균형 조절 (1-8단계)
+   - **시드 제어**: 재현 가능한 결과를 위한 설정
+   - **네거티브 프롬프트**: 생성에서 피할 요소 지정
+
+### 작동 방식
+1. **입력**: 상세한 장면 설명 입력
+2. **비디오 생성**: AI가 프롬프트 기반 비디오 프레임 생성
+3. **오디오 합성**: 자동으로 일치하는 사운드 효과 생성
+4. **출력**: 동기화된 오디오가 포함된 비디오 출력
+
+### 활용 사례
+- 영화 프리뷰 및 컨셉 시각화
+- 뮤직 비디오 제작
+- 광고 콘텐츠 생성
+- 창의적 스토리텔링
+- 게임 시네마틱
+
+### 기술 사양
+- **GPU 가속**: 빠른 처리를 위한 CUDA 사용
+- **모델 아키텍처**: 트랜스포머 기반 확산 모델
+- **오디오 모델**: 플로우 매칭 기반 오디오 합성
+- **처리 시간**: 설정에 따라 약 30-70초
+
+### 최상의 결과를 위한 팁
+- 상세하고 영화적인 설명 사용
+- 카메라 움직임과 시각적 스타일 포함
+- 조명, 색상, 분위기 명시
+- 더 나은 오디오 매칭을 위해 사운드 설명 추가
+- 높은 NAG 스케일 = 프롬프트에 더 충실한 생성
+
+### 특별 기능
+- **영화급 프롬프트 예제**: 전문적인 촬영 기법이 포함된 3가지 예제 ��공
+- **실시간 진행 표시**: 생성 과정을 실시간으로 확인
+- **원클릭 예제 적용**: 예제를 클릭하면 자동으로 설정값 적용
+
+이 도구는 전문가 수준의 비디오 콘텐츠를 쉽게 생성할 수 있도록 설계되었으며, 창의적인 아이디어를 빠르게 시각화하는 데 이상적입니다.

app.py

@@ -0,0 +1,595 @@
+import types
+import random
+import spaces
+import logging
+import os
+from pathlib import Path
+from datetime import datetime
+import re
+
+import torch
+import numpy as np
+import torchaudio
+from diffusers import AutoencoderKLWan, UniPCMultistepScheduler
+from diffusers.utils import export_to_video
+from diffusers import AutoModel
+import gradio as gr
+import tempfile
+from huggingface_hub import hf_hub_download
+import traceback
+
+# Patch for scaled_dot_product_attention to fix enable_gqa issue
+import torch.nn.functional as F
+
+original_sdpa = F.scaled_dot_product_attention
+
+def patched_scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None, enable_gqa=None):
+   # enable_gqa 파라미터를 무시하고 나머지 파라미터만 전달
+   kwargs = {}
+   if attn_mask is not None:
+       kwargs['attn_mask'] = attn_mask
+   if dropout_p != 0.0:
+       kwargs['dropout_p'] = dropout_p
+   if is_causal:
+       kwargs['is_causal'] = is_causal
+   if scale is not None:
+       kwargs['scale'] = scale
+   
+   return original_sdpa(query, key, value, **kwargs)
+
+# 패치 적용
+F.scaled_dot_product_attention = patched_scaled_dot_product_attention
+
+from src.pipeline_wan_nag import NAGWanPipeline
+from src.transformer_wan_nag import NagWanTransformer3DModel
+
+# MMAudio imports
+try:
+   import mmaudio
+except ImportError:
+   os.system("pip install -e .")
+   import mmaudio
+
+from mmaudio.eval_utils import (ModelConfig, all_model_cfg, generate as mmaudio_generate, 
+                               load_video, make_video, setup_eval_logging)
+from mmaudio.model.flow_matching import FlowMatching
+from mmaudio.model.networks import MMAudio, get_my_mmaudio
+from mmaudio.model.sequence_config import SequenceConfig
+from mmaudio.model.utils.features_utils import FeaturesUtils
+
+# NAG Video Settings
+MOD_VALUE = 32
+DEFAULT_DURATION_SECONDS = 4
+DEFAULT_STEPS = 4
+DEFAULT_SEED = 2025
+DEFAULT_H_SLIDER_VALUE = 480
+DEFAULT_W_SLIDER_VALUE = 832
+NEW_FORMULA_MAX_AREA = 480.0 * 832.0
+
+SLIDER_MIN_H, SLIDER_MAX_H = 128, 896
+SLIDER_MIN_W, SLIDER_MAX_W = 128, 896
+MAX_SEED = np.iinfo(np.int32).max
+
+FIXED_FPS = 16
+MIN_FRAMES_MODEL = 8
+MAX_FRAMES_MODEL = 129
+
+DEFAULT_NAG_NEGATIVE_PROMPT = "Static, motionless, still, ugly, bad quality, worst quality, poorly drawn, low resolution, blurry, lack of details"
+DEFAULT_AUDIO_NEGATIVE_PROMPT = "music, speech, voice, singing, narration"
+
+# NAG Model Settings
+MODEL_ID = "Wan-AI/Wan2.1-T2V-14B-Diffusers"
+SUB_MODEL_ID = "vrgamedevgirl84/Wan14BT2VFusioniX"
+SUB_MODEL_FILENAME = "Wan14BT2VFusioniX_fp16_.safetensors"
+LORA_REPO_ID = "Kijai/WanVideo_comfy"
+LORA_FILENAME = "Wan21_CausVid_14B_T2V_lora_rank32.safetensors"
+
+# MMAudio Settings
+torch.backends.cuda.matmul.allow_tf32 = True
+torch.backends.cudnn.allow_tf32 = True
+log = logging.getLogger()
+device = 'cuda'
+dtype = torch.bfloat16
+audio_model_config: ModelConfig = all_model_cfg['large_44k_v2']
+audio_model_config.download_if_needed()
+setup_eval_logging()
+
+# Initialize NAG Video Model
+try:
+   vae = AutoencoderKLWan.from_pretrained(MODEL_ID, subfolder="vae", torch_dtype=torch.float32)
+   wan_path = hf_hub_download(repo_id=SUB_MODEL_ID, filename=SUB_MODEL_FILENAME)
+   transformer = NagWanTransformer3DModel.from_single_file(wan_path, torch_dtype=torch.bfloat16)
+   pipe = NAGWanPipeline.from_pretrained(
+       MODEL_ID, vae=vae, transformer=transformer, torch_dtype=torch.bfloat16
+   )
+   pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config, flow_shift=5.0)
+   pipe.to("cuda")
+
+   pipe.transformer.__class__.attn_processors = NagWanTransformer3DModel.attn_processors
+   pipe.transformer.__class__.set_attn_processor = NagWanTransformer3DModel.set_attn_processor
+   pipe.transformer.__class__.forward = NagWanTransformer3DModel.forward
+   print("NAG Video Model loaded successfully!")
+except Exception as e:
+   print(f"Error loading NAG Video Model: {e}")
+   pipe = None
+
+# Initialize MMAudio Model
+def get_mmaudio_model() -> tuple[MMAudio, FeaturesUtils, SequenceConfig]:
+   seq_cfg = audio_model_config.seq_cfg
+   
+   net: MMAudio = get_my_mmaudio(audio_model_config.model_name).to(device, dtype).eval()
+   net.load_weights(torch.load(audio_model_config.model_path, map_location=device, weights_only=True))
+   log.info(f'Loaded MMAudio weights from {audio_model_config.model_path}')
+   
+   feature_utils = FeaturesUtils(tod_vae_ckpt=audio_model_config.vae_path,
+                                 synchformer_ckpt=audio_model_config.synchformer_ckpt,
+                                 enable_conditions=True,
+                                 mode=audio_model_config.mode,
+                                 bigvgan_vocoder_ckpt=audio_model_config.bigvgan_16k_path,
+                                 need_vae_encoder=False)
+   feature_utils = feature_utils.to(device, dtype).eval()
+   
+   return net, feature_utils, seq_cfg
+
+try:
+   audio_net, audio_feature_utils, audio_seq_cfg = get_mmaudio_model()
+   print("MMAudio Model loaded successfully!")
+except Exception as e:
+   print(f"Error loading MMAudio Model: {e}")
+   audio_net = None
+
+# 비디오 프롬프트를 오디오 프롬프트로 변환하는 함수
+def extract_audio_description(video_prompt):
+   """비디오 프롬프트에서 오디오 관련 설명 추출/변환"""
+   
+   # 키워드 매핑
+   audio_keywords = {
+       'car': 'car engine sound, vehicle noise',
+       'porsche': 'sports car engine roar, exhaust sound',
+       'guitar': 'electric guitar playing, guitar music',
+       'concert': 'crowd cheering, live music, applause',
+       'motorcycle': 'motorcycle engine sound, motor rumble',
+       'highway': 'traffic noise, road ambience',
+       'rain': 'rain sounds, water drops',
+       'wind': 'wind blowing sound',
+       'ocean': 'ocean waves, water sounds',
+       'city': 'urban ambience, city traffic sounds',
+       'singer': 'singing voice, vocals',
+       'crowd': 'crowd noise, people talking',
+       'flames': 'fire crackling sound',
+       'pyro': 'fire whoosh, flame burst sound',
+       'explosion': 'explosion sound, blast',
+       'countryside': 'nature ambience, birds chirping',
+       'wheat fields': 'wind through grass, rural ambience',
+       'engine': 'motor sound, mechanical noise',
+       'flat-six engine': 'sports car engine sound',
+       'roaring': 'loud engine roar',
+       'thunderous': 'loud booming sound',
+       'child': 'children playing sounds',
+       'running': 'footsteps sound',
+       'woman': 'ambient sounds',
+       'phone': 'subtle electronic ambience',
+       'advertisement': 'modern ambient sounds'
+   }
+   
+   # 간단한 키워드 기반 변환
+   audio_descriptions = []
+   lower_prompt = video_prompt.lower()
+   
+   for key, value in audio_keywords.items():
+       if key in lower_prompt:
+           audio_descriptions.append(value)
+   
+   # 기본값 설정
+   if not audio_descriptions:
+       # 프롬프트에 명시적인 오디오 설명이 있는지 확인
+       if 'sound' in lower_prompt or 'audio' in lower_prompt or 'noise' in lower_prompt:
+           # 프롬프트에서 오디오 관련 부분만 추출
+           audio_pattern = r'([^.]*(?:sound|audio|noise|music|voice|roar|rumble)[^.]*)'
+           matches = re.findall(audio_pattern, lower_prompt, re.IGNORECASE)
+           if matches:
+               return ', '.join(matches)
+       
+       # 기본 ambient sound
+       return "ambient environmental sounds matching the scene"
+   
+   return ', '.join(audio_descriptions)
+
+# Audio generation function
+@torch.inference_mode()
+def add_audio_to_video(video_path, prompt, audio_custom_prompt, audio_negative_prompt, audio_steps, audio_cfg_strength, duration):
+   """Generate and add audio to video using MMAudio"""
+   if audio_net is None:
+       print("MMAudio model not loaded, returning video without audio")
+       return video_path
+       
+   try:
+       # 커스텀 오디오 프롬프트가 있으면 사용, 없으면 비디오 프롬프트에서 추출
+       if audio_custom_prompt and audio_custom_prompt.strip():
+           audio_prompt = audio_custom_prompt.strip()
+       else:
+           audio_prompt = extract_audio_description(prompt)
+       
+       print(f"Original prompt: {prompt}")
+       print(f"Audio prompt: {audio_prompt}")
+       
+       rng = torch.Generator(device=device)
+       rng.manual_seed(random.randint(0, 2**32 - 1))  # 더 명확한 랜덤 시드
+       fm = FlowMatching(min_sigma=0, inference_mode='euler', num_steps=audio_steps)
+       
+       video_info = load_video(video_path, duration)
+       clip_frames = video_info.clip_frames
+       sync_frames = video_info.sync_frames
+       duration = video_info.duration_sec
+       clip_frames = clip_frames.unsqueeze(0)
+       sync_frames = sync_frames.unsqueeze(0)
+       audio_seq_cfg.duration = duration
+       audio_net.update_seq_lengths(audio_seq_cfg.latent_seq_len, audio_seq_cfg.clip_seq_len, audio_seq_cfg.sync_seq_len)
+       
+       # 향상된 네거티브 프롬프트
+       enhanced_negative = f"{audio_negative_prompt}, distortion, static noise, silence, random beeps"
+       
+       audios = mmaudio_generate(clip_frames,
+                                 sync_frames, [audio_prompt],  # 변환된 오디오 프롬프트 사용
+                                 negative_text=[enhanced_negative],
+                                 feature_utils=audio_feature_utils,
+                                 net=audio_net,
+                                 fm=fm,
+                                 rng=rng,
+                                 cfg_strength=audio_cfg_strength)
+       audio = audios.float().cpu()[0]
+       
+       # Create video with audio
+       video_with_audio_path = tempfile.NamedTemporaryFile(delete=False, suffix='.mp4').name
+       make_video(video_info, video_with_audio_path, audio, sampling_rate=audio_seq_cfg.sampling_rate)
+       
+       return video_with_audio_path
+   except Exception as e:
+       print(f"Error in audio generation: {e}")
+       traceback.print_exc()
+       return video_path
+
+# Combined generation function
+def get_duration(prompt, nag_negative_prompt, nag_scale, height, width, duration_seconds, 
+                steps, seed, randomize_seed, enable_audio, audio_custom_prompt,
+                audio_negative_prompt, audio_steps, audio_cfg_strength):
+   # Calculate total duration including audio processing if enabled
+   video_duration = int(duration_seconds) * int(steps) * 2.25 + 5
+   audio_duration = 30 if enable_audio else 0  # Additional time for audio processing
+   return video_duration + audio_duration
+
+@spaces.GPU(duration=get_duration)
+def generate_video_with_audio(
+       prompt,
+       nag_negative_prompt, nag_scale,
+       height=DEFAULT_H_SLIDER_VALUE, width=DEFAULT_W_SLIDER_VALUE, duration_seconds=DEFAULT_DURATION_SECONDS,
+       steps=DEFAULT_STEPS,
+       seed=DEFAULT_SEED, randomize_seed=False,
+       enable_audio=True, audio_custom_prompt="",
+       audio_negative_prompt=DEFAULT_AUDIO_NEGATIVE_PROMPT,
+       audio_steps=30, audio_cfg_strength=4.5,
+):
+   if pipe is None:
+       return None, DEFAULT_SEED
+       
+   try:
+       # Generate video first
+       target_h = max(MOD_VALUE, (int(height) // MOD_VALUE) * MOD_VALUE)
+       target_w = max(MOD_VALUE, (int(width) // MOD_VALUE) * MOD_VALUE)
+       
+       num_frames = np.clip(int(round(int(duration_seconds) * FIXED_FPS) + 1), MIN_FRAMES_MODEL, MAX_FRAMES_MODEL)
+       
+       current_seed = random.randint(0, MAX_SEED) if randomize_seed else int(seed)
+       
+       print(f"Generating video with: prompt='{prompt}', resolution={target_w}x{target_h}, frames={num_frames}")
+       
+       with torch.inference_mode():
+           nag_output_frames_list = pipe(
+               prompt=prompt,
+               nag_negative_prompt=nag_negative_prompt,
+               nag_scale=nag_scale,
+               nag_tau=3.5,
+               nag_alpha=0.5,
+               height=target_h, width=target_w, num_frames=num_frames,
+               guidance_scale=0.,
+               num_inference_steps=int(steps),
+               generator=torch.Generator(device="cuda").manual_seed(current_seed)
+           ).frames[0]
+       
+       # Save initial video without audio
+       with tempfile.NamedTemporaryFile(suffix=".mp4", delete=False) as tmpfile:
+           temp_video_path = tmpfile.name
+       export_to_video(nag_output_frames_list, temp_video_path, fps=FIXED_FPS)
+       print(f"Video saved to: {temp_video_path}")
+       
+       # Add audio if enabled
+       if enable_audio:
+           try:
+               print("Adding audio to video...")
+               final_video_path = add_audio_to_video(
+                   temp_video_path, 
+                   prompt,
+                   audio_custom_prompt,
+                   audio_negative_prompt,
+                   audio_steps,
+                   audio_cfg_strength,
+                   duration_seconds
+               )
+               # Clean up temp video
+               if os.path.exists(temp_video_path) and final_video_path != temp_video_path:
+                   os.remove(temp_video_path)
+               print(f"Final video with audio: {final_video_path}")
+           except Exception as e:
+               log.error(f"Audio generation failed: {e}")
+               final_video_path = temp_video_path
+       else:
+           final_video_path = temp_video_path
+       
+       return final_video_path, current_seed
+   except Exception as e:
+       print(f"Error in video generation: {e}")
+       return None, current_seed
+
+# Example generation function - simplified
+def set_example(prompt, nag_negative_prompt, nag_scale):
+   """Set example values in the UI without triggering generation"""
+   return (
+       prompt,
+       nag_negative_prompt,
+       nag_scale,
+       DEFAULT_H_SLIDER_VALUE,
+       DEFAULT_W_SLIDER_VALUE,
+       DEFAULT_DURATION_SECONDS,
+       DEFAULT_STEPS,
+       DEFAULT_SEED,
+       True,  # randomize_seed
+       True,  # enable_audio
+       "",    # audio_custom_prompt
+       DEFAULT_AUDIO_NEGATIVE_PROMPT,
+       30,    # audio_steps
+       4.5    # audio_cfg_strength
+   )
+
+# Examples with audio descriptions
+examples = [
+   ["Midnight highway outside a neon-lit city. A black 1973 Porsche 911 Carrera RS speeds at 120 km/h. Inside, a stylish singer-guitarist sings while driving, vintage sunburst guitar on the passenger seat. Sodium streetlights streak over the hood; RGB panels shift magenta to blue on the driver. Camera: drone dive, Russian-arm low wheel shot, interior gimbal, FPV barrel roll, overhead spiral. Neo-noir palette, rain-slick asphalt reflections, roaring flat-six engine blended with live guitar.", DEFAULT_NAG_NEGATIVE_PROMPT, 11],
+   ["Arena rock concert packed with 20 000 fans. A flamboyant lead guitarist in leather jacket and mirrored aviators shreds a cherry-red Flying V on a thrust stage. Pyro flames shoot up on every downbeat, CO₂ jets burst behind. Moving-head spotlights swirl teal and amber, follow-spots rim-light the guitarist's hair. Steadicam 360-orbit, crane shot rising over crowd, ultra-slow-motion pick attack at 1 000 fps. Film-grain teal-orange grade, thunderous crowd roar mixes with screaming guitar solo.", DEFAULT_NAG_NEGATIVE_PROMPT, 11],
+   ["Golden-hour countryside road winding through rolling wheat fields. A man and woman ride a vintage café-racer motorcycle, hair and scarf fluttering in the warm breeze. Drone chase shot reveals endless patchwork farmland; low slider along rear wheel captures dust trail. Sun-flare back-lights the riders, lens blooms on highlights. Soft acoustic rock underscore; engine rumble mixed at –8 dB. Warm pastel color grade, gentle film-grain for nostalgic vibe.", DEFAULT_NAG_NEGATIVE_PROMPT, 11],
+]
+
+# CSS styling - Fixed for better layout
+css = """
+/* Right column - video output */
+.video-output {
+   border-radius: 15px;
+   overflow: hidden;
+   box-shadow: 0 10px 30px rgba(0, 0, 0, 0.2);
+   width: 100% !important;
+   height: auto !important;
+   min-height: 400px;
+}
+
+/* Ensure video container is responsive */
+.video-output video {
+   width: 100% !important;
+   height: auto !important;
+   max-height: 600px;
+   object-fit: contain;
+   display: block;
+}
+
+/* Remove any overlay or background from video container */
+.video-output > div {
+   background: transparent !important;
+   padding: 0 !important;
+}
+
+/* Remove gradio's default video player overlay */
+.video-output .wrap {
+   background: transparent !important;
+}
+
+/* Ensure no gray overlay on video controls */
+.video-output video::-webkit-media-controls-enclosure {
+   background: transparent;
+}
+"""
+
+# Gradio interface - Fixed structure
+with gr.Blocks(css=css, theme=gr.themes.Soft()) as demo:
+   gr.HTML("""
+       <div class="container">
+           <h1 class="main-title">🎬 VEO3 Free</h1>
+           <p class="subtitle">Wan2.1-T2V-14B + Fast 4-step with NAG + Automatic Audio Generation</p>
+       </div>
+   """)
+   
+   gr.HTML("""
+       <div class='container' style='display:flex; justify-content:center; gap:12px; margin-bottom: 20px;'>
+           <a href="https://huggingface.co/spaces/openfree/Best-AI" target="_blank">
+               <img src="https://img.shields.io/static/v1?label=OpenFree&message=BEST%20AI%20Services&color=%230000ff&labelColor=%23000080&logo=huggingface&logoColor=%23ffa500&style=for-the-badge" alt="OpenFree badge">
+           </a>
+
+           <a href="https://discord.gg/openfreeai" target="_blank">
+               <img src="https://img.shields.io/static/v1?label=Discord&message=Openfree%20AI&color=%230000ff&labelColor=%23800080&logo=discord&logoColor=white&style=for-the-badge" alt="Discord badge">
+           </a>
+       </div>
+   """)
+   
+   with gr.Row(equal_height=True):
+       with gr.Column(scale=5):
+           with gr.Group(elem_classes="prompt-container"):
+               prompt = gr.Textbox(
+                   label="✨ Video Prompt (also used for audio generation)",
+                   placeholder="Describe your video scene in detail...",
+                   lines=3,
+                   elem_classes="prompt-input"
+               )
+               
+               with gr.Accordion("🎨 Advanced Video Settings", open=False):
+                   nag_negative_prompt = gr.Textbox(
+                       label="Video Negative Prompt",
+                       value=DEFAULT_NAG_NEGATIVE_PROMPT,
+                       lines=2,
+                   )
+                   nag_scale = gr.Slider(
+                       label="NAG Scale",
+                       minimum=1.0,
+                       maximum=20.0,
+                       step=0.25,
+                       value=11.0,
+                       info="Higher values = stronger guidance"
+                   )
+           
+           with gr.Group(elem_classes="settings-panel"):
+               gr.Markdown("### ⚙️ Video Settings")
+               
+               with gr.Row():
+                   duration_seconds_input = gr.Slider(
+                       minimum=1,
+                       maximum=8,
+                       step=1,
+                       value=DEFAULT_DURATION_SECONDS,
+                       label="📱 Duration (seconds)",
+                       elem_classes="slider-container"
+                   )
+                   steps_slider = gr.Slider(
+                       minimum=1,
+                       maximum=8,
+                       step=1,
+                       value=DEFAULT_STEPS,
+                       label="🔄 Inference Steps",
+                       elem_classes="slider-container"
+                   )
+               
+               with gr.Row():
+                   height_input = gr.Slider(
+                       minimum=SLIDER_MIN_H,
+                       maximum=SLIDER_MAX_H,
+                       step=MOD_VALUE,
+                       value=DEFAULT_H_SLIDER_VALUE,
+                       label=f"📐 Height (×{MOD_VALUE})",
+                       elem_classes="slider-container"
+                   )
+                   width_input = gr.Slider(
+                       minimum=SLIDER_MIN_W,
+                       maximum=SLIDER_MAX_W,
+                       step=MOD_VALUE,
+                       value=DEFAULT_W_SLIDER_VALUE,
+                       label=f"📐 Width (×{MOD_VALUE})",
+                       elem_classes="slider-container"
+                   )
+               
+               with gr.Row():
+                   seed_input = gr.Slider(
+                       label="🌱 Seed",
+                       minimum=0,
+                       maximum=MAX_SEED,
+                       step=1,
+                       value=DEFAULT_SEED,
+                       interactive=True
+                   )
+                   randomize_seed_checkbox = gr.Checkbox(
+                       label="🎲 Random Seed",
+                       value=True,
+                       interactive=True
+                   )
+           
+           with gr.Group(elem_classes="audio-settings"):
+               gr.Markdown("### 🎵 Audio Generation Settings")
+               
+               enable_audio = gr.Checkbox(
+                   label="🔊 Enable Automatic Audio Generation",
+                   value=True,
+                   interactive=True
+               )
+               
+               with gr.Column(visible=True) as audio_settings_group:
+                   audio_custom_prompt = gr.Textbox(
+                       label="Custom Audio Prompt (Optional)",
+                       placeholder="Leave empty to auto-generate from video prompt, or specify custom audio description (e.g., 'car engine sound, traffic noise')",
+                       value="",
+                   )
+                   audio_negative_prompt = gr.Textbox(
+                       label="Audio Negative Prompt",
+                       value=DEFAULT_AUDIO_NEGATIVE_PROMPT,
+                       placeholder="Elements to avoid in audio",
+                   )
+                   
+                   with gr.Row():
+                       audio_steps = gr.Slider(
+                           minimum=10,
+                           maximum=50,
+                           step=5,
+                           value=30,
+                           label="🎚️ Audio Steps",
+                           info="More steps = better quality"
+                       )
+                       audio_cfg_strength = gr.Slider(
+                           minimum=1.0,
+                           maximum=10.0,
+                           step=0.5,
+                           value=4.5,
+                           label="🎛️ Audio Guidance",
+                           info="Strength of prompt guidance"
+                       )
+               
+               # Toggle audio settings visibility
+               enable_audio.change(
+                   fn=lambda x: gr.update(visible=x),
+                   inputs=[enable_audio],
+                   outputs=[audio_settings_group]
+               )
+           
+           generate_button = gr.Button(
+               "🎬 Generate Video with Audio",
+               variant="primary",
+               elem_classes="generate-btn"
+           )
+       
+       with gr.Column(scale=5):
+           video_output = gr.Video(
+               label="Generated Video with Audio",
+               autoplay=True,
+               interactive=False,
+               elem_classes="video-output",
+               height=600
+           )
+           
+           gr.HTML("""
+               <div style="text-align: center; margin-top: 20px; color: #6b7280;">
+                   <p>💡 Tip: For better audio, use Custom Audio Prompt with sound descriptions!</p>
+                   <p>🎧 Examples: "car engine sound", "crowd cheering", "nature ambience"</p>
+               </div>
+           """)
+   
+   # Examples section moved outside of columns
+   with gr.Row():
+       gr.Markdown("### 🎯 Example Prompts")
+   
+   gr.Examples(
+       examples=examples,
+       inputs=[prompt, nag_negative_prompt, nag_scale],
+       outputs=None,  # Don't connect outputs to avoid index issues
+       cache_examples=False
+   )
+   
+   # Connect UI elements
+   ui_inputs = [
+       prompt,
+       nag_negative_prompt, nag_scale,
+       height_input, width_input, duration_seconds_input,
+       steps_slider,
+       seed_input, randomize_seed_checkbox,
+       enable_audio, audio_custom_prompt, audio_negative_prompt, 
+       audio_steps, audio_cfg_strength,
+   ]
+   
+   generate_button.click(
+       fn=generate_video_with_audio,
+       inputs=ui_inputs,
+       outputs=[video_output, seed_input],
+   )
+
+if __name__ == "__main__":
+   demo.queue().launch()

demo.py

@@ -0,0 +1,135 @@
+import logging
+from argparse import ArgumentParser
+from pathlib import Path
+
+import torch
+import torchaudio
+
+from mmaudio.eval_utils import (ModelConfig, all_model_cfg, generate, load_video, make_video,
+                                setup_eval_logging)
+from mmaudio.model.flow_matching import FlowMatching
+from mmaudio.model.networks import MMAudio, get_my_mmaudio
+from mmaudio.model.utils.features_utils import FeaturesUtils
+
+torch.backends.cuda.matmul.allow_tf32 = True
+torch.backends.cudnn.allow_tf32 = True
+
+log = logging.getLogger()
+
+
+@torch.inference_mode()
+def main():
+    setup_eval_logging()
+
+    parser = ArgumentParser()
+    parser.add_argument('--variant',
+                        type=str,
+                        default='large_44k_v2',
+                        help='small_16k, small_44k, medium_44k, large_44k, large_44k_v2')
+    parser.add_argument('--video', type=Path, help='Path to the video file')
+    parser.add_argument('--prompt', type=str, help='Input prompt', default='')
+    parser.add_argument('--negative_prompt', type=str, help='Negative prompt', default='')
+    parser.add_argument('--duration', type=float, default=8.0)
+    parser.add_argument('--cfg_strength', type=float, default=4.5)
+    parser.add_argument('--num_steps', type=int, default=25)
+
+    parser.add_argument('--mask_away_clip', action='store_true')
+
+    parser.add_argument('--output', type=Path, help='Output directory', default='./output')
+    parser.add_argument('--seed', type=int, help='Random seed', default=42)
+    parser.add_argument('--skip_video_composite', action='store_true')
+    parser.add_argument('--full_precision', action='store_true')
+
+    args = parser.parse_args()
+
+    if args.variant not in all_model_cfg:
+        raise ValueError(f'Unknown model variant: {args.variant}')
+    model: ModelConfig = all_model_cfg[args.variant]
+    model.download_if_needed()
+    seq_cfg = model.seq_cfg
+
+    if args.video:
+        video_path: Path = Path(args.video).expanduser()
+    else:
+        video_path = None
+    prompt: str = args.prompt
+    negative_prompt: str = args.negative_prompt
+    output_dir: str = args.output.expanduser()
+    seed: int = args.seed
+    num_steps: int = args.num_steps
+    duration: float = args.duration
+    cfg_strength: float = args.cfg_strength
+    skip_video_composite: bool = args.skip_video_composite
+    mask_away_clip: bool = args.mask_away_clip
+
+    device = 'cuda'
+    dtype = torch.float32 if args.full_precision else torch.bfloat16
+
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    # load a pretrained model
+    net: MMAudio = get_my_mmaudio(model.model_name).to(device, dtype).eval()
+    net.load_weights(torch.load(model.model_path, map_location=device, weights_only=True))
+    log.info(f'Loaded weights from {model.model_path}')
+
+    # misc setup
+    rng = torch.Generator(device=device)
+    rng.manual_seed(seed)
+    fm = FlowMatching(min_sigma=0, inference_mode='euler', num_steps=num_steps)
+
+    feature_utils = FeaturesUtils(tod_vae_ckpt=model.vae_path,
+                                  synchformer_ckpt=model.synchformer_ckpt,
+                                  enable_conditions=True,
+                                  mode=model.mode,
+                                  bigvgan_vocoder_ckpt=model.bigvgan_16k_path,
+                                  need_vae_encoder=False)
+    feature_utils = feature_utils.to(device, dtype).eval()
+
+    if video_path is not None:
+        log.info(f'Using video {video_path}')
+        video_info = load_video(video_path, duration)
+        clip_frames = video_info.clip_frames
+        sync_frames = video_info.sync_frames
+        duration = video_info.duration_sec
+        if mask_away_clip:
+            clip_frames = None
+        else:
+            clip_frames = clip_frames.unsqueeze(0)
+        sync_frames = sync_frames.unsqueeze(0)
+    else:
+        log.info('No video provided -- text-to-audio mode')
+        clip_frames = sync_frames = None
+
+    seq_cfg.duration = duration
+    net.update_seq_lengths(seq_cfg.latent_seq_len, seq_cfg.clip_seq_len, seq_cfg.sync_seq_len)
+
+    log.info(f'Prompt: {prompt}')
+    log.info(f'Negative prompt: {negative_prompt}')
+
+    audios = generate(clip_frames,
+                      sync_frames, [prompt],
+                      negative_text=[negative_prompt],
+                      feature_utils=feature_utils,
+                      net=net,
+                      fm=fm,
+                      rng=rng,
+                      cfg_strength=cfg_strength)
+    audio = audios.float().cpu()[0]
+    if video_path is not None:
+        save_path = output_dir / f'{video_path.stem}.flac'
+    else:
+        safe_filename = prompt.replace(' ', '_').replace('/', '_').replace('.', '')
+        save_path = output_dir / f'{safe_filename}.flac'
+    torchaudio.save(save_path, audio, seq_cfg.sampling_rate)
+
+    log.info(f'Audio saved to {save_path}')
+    if video_path is not None and not skip_video_composite:
+        video_save_path = output_dir / f'{video_path.stem}.mp4'
+        make_video(video_info, video_save_path, audio, sampling_rate=seq_cfg.sampling_rate)
+        log.info(f'Video saved to {output_dir / video_save_path}')
+
+    log.info('Memory usage: %.2f GB', torch.cuda.max_memory_allocated() / (2**30))
+
+
+if __name__ == '__main__':
+    main()

gitignore

@@ -0,0 +1,6 @@
+.idea/
+
+__pycache__/
+*.py[cod]
+*$py.class
+

pyproject.toml

@@ -0,0 +1,52 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.yapf]
+based_on_style = "pep8"
+indent_width = 4
+column_limit = 100
+
+[project]
+name = "mmaudio"
+version = "1.0.0"
+authors = [{ name = "Rex Cheng", email = "hkchengrex@gmail.com" }]
+description = ""
+readme = "README.md"
+requires-python = ">=3.9"
+classifiers = [
+  "Programming Language :: Python :: 3",
+  "Operating System :: OS Independent",
+]
+dependencies = [
+  'torch >= 2.5.1',
+  'python-dotenv',
+  'cython',
+  'gitpython >= 3.1',
+  'tensorboard >= 2.11',
+  'numpy >= 1.21, <2.1',
+  'Pillow >= 9.5',
+  'opencv-python >= 4.8',
+  'scipy >= 1.7',
+  'tqdm >= 4.66.1',
+  'gradio >= 3.34',
+  'einops >= 0.6',
+  'hydra-core >= 1.3.2',
+  'requests',
+  'torchdiffeq',
+  'librosa >= 0.8.1',
+  'nitrous-ema',
+  'safetensors',
+  'auraloss',
+  'hydra_colorlog',
+  'tensordict',
+  'colorlog',
+  'open_clip_torch',
+  'soundfile',
+]
+
+[tool.hatch.build.targets.wheel]
+packages = ["mmaudio"]

requirements.txt

@@ -0,0 +1,35 @@
+git+https://github.com/huggingface/diffusers.git
+transformers
+accelerate
+safetensors
+sentencepiece
+peft
+ftfy
+imageio-ffmpeg
+opencv-python
+
+torch == 2.4.0
+torchvision
+torchaudio
+python-dotenv
+cython
+gitpython >= 3.1
+tensorboard >= 2.11
+numpy >= 1.21, <2.1
+Pillow >= 9.5
+scipy >= 1.7
+tqdm >= 4.66.1
+gradio >= 3.34
+einops >= 0.6
+hydra-core >= 1.3.2
+requests
+torchdiffeq
+librosa >= 0.8.1
+nitrous-ema
+auraloss
+hydra_colorlog
+tensordict
+colorlog
+open_clip_torch
+soundfile
+av