Add lip-sync app code and configuration

README.md

@@ -1,14 +1,54 @@
----
-title: Lip Sync Generator
-emoji: 👀
-colorFrom: gray
-colorTo: yellow
-sdk: gradio
-sdk_version: 5.49.1
-app_file: app.py
-pinned: false
-license: mit
-short_description: A Gradio app to lip-sync an avatar image or video with an au
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Lip-Sync Video Generator
+
+This project contains a simple web application that allows you to upload a single frame (or short clip) of an avatar and an audio file and then produce a lip‑synchronised video.  Internally the app uses the open‑source **Wav2Lip** model to animate the avatar so that the mouth movements follow the uploaded audio.  Everything runs on a free cloud platform – there is no need to install anything locally.
+
+## Features
+
+- ✅ **Upload your own avatar:** any static image (PNG, JPG) or existing video clip can be used as the source face.
+- ✅ **Upload an audio track:** accepts common audio formats (MP3, WAV, M4A) from 1 – 10 minutes long.
+- ✅ **Fully automatic:** the application clones the Wav2Lip repository and downloads the necessary model weights on first run, so no manual setup is required.
+- ✅ **Runs on free cloud hardware:** designed to be deployed to [Hugging Face Spaces](https://huggingface.co/spaces), which provide free CPU/GPU resources for open‑source projects.  You only need a free Hugging Face account to host the app.
+- ✅ **Extensible:** advanced users can tweak padding, super‑resolution and segmentation options by modifying the arguments passed to the inference script.
+
+## How it works
+
+When the “Generate video” button is pressed, the application performs the following steps:
+
+1. If the Wav2Lip repository is not already present, it clones it from GitHub.  The clone is shallow (only the latest commit) to minimise the download size.
+2. If the pre‑trained weights (``wav2lip_gan.pth`` and the face segmentation model) are not present, it downloads them from publicly available sources.  The downloads are large (~436 MB), so the first run can take some time.
+3. The uploaded image/video and audio files are saved into a temporary folder.  Basic validation ensures that the audio duration is between 1 and 10 minutes; otherwise an error is shown.
+4. The application calls the official ``inference.py`` script from Wav2Lip in a subprocess.  The script reads the avatar frame and audio, applies the lip‑sync model, and writes an MP4 video to the ``outputs`` directory.
+5. Once the script completes, the resulting video is returned to the web UI and can be played or downloaded.
+
+The heavy lifting is done by Wav2Lip; this project simply wraps it in a clean user interface with sensible defaults and handles all setup.
+
+## Deploying to Hugging Face Spaces
+
+1. **Create a free account:** go to [huggingface.co](https://huggingface.co) and create a free account if you don’t already have one.
+2. **Create a new Space:** from your dashboard, click “New Space”, choose the **Gradio** SDK and give your space a name (e.g. ``lip-sync-app``).  Set it to “Public” so that it can use free hardware.
+3. **Upload the project files:** clone or download this repository, then upload the contents of the ``lipsync_app`` folder (``app.py``, ``requirements.txt`` and this ``README.md``) into your new space.  The file structure should look like this:
+
+```
+├── app.py
+├── requirements.txt
+└── README.md
+```
+
+4. **Commit and build:** once the files are uploaded, commit them to the Space.  Hugging Face will install the dependencies and build the application automatically.  The first build may take a few minutes because it has to download the Wav2Lip model weights.
+5. **Use the app:** after the build succeeds, open the “App” tab of your space.  You can now upload an image and an audio file and click “Generate video” to produce a lip‑synchronised output.  Longer audio clips (up to ten minutes) will take longer to process.
+
+## Limitations
+
+- **Large downloads on first run:** the Wav2Lip weights are hundreds of megabytes.  The application caches them in the working directory so subsequent runs are faster.  If the environment is reset (e.g. if the space times out), the weights will be downloaded again.
+- **Processing time:** running Wav2Lip on CPU is slow (several minutes for a 1‑minute clip).  For best performance, enable GPU hardware in your Hugging Face space settings.  GPU hours are free on public spaces but limited, so plan accordingly.
+- **Avatar quality:** Wav2Lip works best with clear, front‑facing images where the mouth is visible.  Complex backgrounds or occlusions can degrade the result.
+
+## Troubleshooting
+
+- *Error cloning repository:* if cloning GitHub fails due to network restrictions, you can manually upload the Wav2Lip repository into the space.  Download it locally from <https://github.com/Rudrabha/Wav2Lip> and add it as a folder named ``Wav2Lip`` next to ``app.py``.
+- *Model download fails:* check the logs of your space for errors.  You can upload the ``wav2lip_gan.pth`` and ``face_segmentation.pth`` files manually into a ``checkpoints`` folder to avoid automatic downloads.
+- *App times out or runs out of memory:* try reducing the audio length or using CPU hardware.  Free GPU instances provide limited memory (T4/8 GB), which may not handle extremely high resolution inputs.
+
+## Acknowledgements
+
+This project would not be possible without the original [Wav2Lip](https://github.com/Rudrabha/Wav2Lip) authors and their permissive MIT licence.  It also makes use of [Gradio](https://gradio.app/) for the web interface.

app.py

@@ -0,0 +1,183 @@
+"""
+Gradio web application for generating lip‑synchronised videos from an avatar image
+and an audio clip.  On the first run it clones the Wav2Lip repository and
+downloads the required model checkpoints.  Subsequent runs reuse the cached
+repository and weights.  The heavy inference is executed by spawning the
+original `inference.py` script provided by Wav2Lip.
+
+To deploy on Hugging Face Spaces:
+    1. Create a new Gradio space.
+    2. Upload this file along with ``requirements.txt`` and ``README.md``.
+    3. Commit and wait for the space to build.  Enable GPU hardware for best performance.
+
+Author: ChatGPT (2025)
+License: MIT (same as Wav2Lip)
+"""
+
+import os
+import shutil
+import subprocess
+import tempfile
+from pathlib import Path
+
+import gradio as gr
+import numpy as np
+from pydub import AudioSegment
+
+
+# Constants for repository and checkpoint names
+REPO_URL = "https://github.com/Rudrabha/Wav2Lip.git"
+REPO_DIR = Path("Wav2Lip")
+CHECKPOINTS_DIR = REPO_DIR / "checkpoints"
+WAV2LIP_MODEL = "wav2lip_gan.pth"
+FACE_SEG_MODEL = "face_segmentation.pth"
+
+# Direct download links for model weights.  These links point to publicly
+# accessible mirrors.  If they stop working, you can upload the files into
+# the checkpoints folder manually and the app will skip downloading.
+MODEL_URLS = {
+    WAV2LIP_MODEL: "https://huggingface.co/Non-playing-Character/Wave2lip/resolve/main/wav2lip_gan.pth",
+    FACE_SEG_MODEL: "https://huggingface.co/Non-playing-Character/Wave2lip/resolve/main/face_segmentation.pth",
+}
+
+
+def clone_repository() -> None:
+    """Clone the Wav2Lip GitHub repository if it does not already exist."""
+    if REPO_DIR.exists():
+        return
+    # Use shallow clone to reduce download size
+    subprocess.run([
+        "git",
+        "clone",
+        "--depth",
+        "1",
+        REPO_URL,
+        str(REPO_DIR),
+    ], check=True)
+
+
+def download_model_weights() -> None:
+    """Download the Wav2Lip and face segmentation checkpoints if missing."""
+    CHECKPOINTS_DIR.mkdir(parents=True, exist_ok=True)
+    for filename, url in MODEL_URLS.items():
+        dest = CHECKPOINTS_DIR / filename
+        if dest.exists():
+            continue
+        # Stream download using requests to avoid loading the full file into memory
+        import requests
+        with requests.get(url, stream=True) as r:
+            r.raise_for_status()
+            with open(dest, "wb") as f:
+                for chunk in r.iter_content(chunk_size=8192):
+                    if chunk:
+                        f.write(chunk)
+
+
+def ensure_setup() -> None:
+    """Ensure that the repository and model weights are available."""
+    clone_repository()
+    download_model_weights()
+
+
+def validate_audio_length(audio_path: str) -> None:
+    """Raise a ValueError if the audio duration is not between 1 and 10 minutes."""
+    audio = AudioSegment.from_file(audio_path)
+    duration_sec = len(audio) / 1000.0
+    if duration_sec < 60:
+        raise ValueError("Audio is too short: please provide at least 1 minute of audio.")
+    if duration_sec > 600:
+        raise ValueError("Audio is too long: please limit the audio to 10 minutes.")
+
+
+def run_inference(image_path: Path, audio_path: Path) -> Path:
+    """Run Wav2Lip inference and return the path to the generated video."""
+    ensure_setup()
+
+    # Prepare output directory and file name
+    outputs_dir = Path("outputs")
+    outputs_dir.mkdir(exist_ok=True)
+    output_path = outputs_dir / f"result_{image_path.stem}.mp4"
+
+    # Build command to run inference
+    cmd = [
+        "python", "inference.py",
+        "--checkpoint_path", str(CHECKPOINTS_DIR / WAV2LIP_MODEL),
+        "--segmentation_path", str(CHECKPOINTS_DIR / FACE_SEG_MODEL),
+        "--face", str(image_path),
+        "--audio", str(audio_path),
+        "--outfile", str(output_path),
+        "--pads", "0", "10", "0", "0",  # default padding
+    ]
+    # Execute inside repository directory
+    subprocess.run(cmd, cwd=str(REPO_DIR), check=True)
+    return output_path
+
+
+def generate_video(avatar_file, audio_file):
+    """
+    Gradio callback to generate a lip‑synced video.
+
+    Parameters
+    ----------
+    avatar_file : file-like
+        Uploaded image or video containing the face.
+    audio_file : file-like
+        Uploaded audio file.
+
+    Returns
+    -------
+    str
+        Path to the generated MP4 file (relative to Gradio working directory).
+    """
+    if avatar_file is None or audio_file is None:
+        return None
+
+    # Save uploaded files to temporary directory
+    with tempfile.TemporaryDirectory() as tmpdir:
+        avatar_path = Path(tmpdir) / "avatar"
+        audio_path = Path(tmpdir) / "audio"
+        # Write binary contents
+        avatar_path.write_bytes(avatar_file.read())
+        audio_path.write_bytes(audio_file.read())
+        # Validate audio length
+        try:
+            validate_audio_length(str(audio_path))
+        except Exception as e:
+            raise gr.Error(str(e))
+        # Run inference
+        try:
+            result_path = run_inference(avatar_path, audio_path)
+        except subprocess.CalledProcessError as e:
+            raise gr.Error(f"Inference failed: {e}")
+        return str(result_path)
+
+
+def build_interface():
+    """Construct the Gradio interface."""
+    with gr.Blocks(title="Lip‑Sync Video Generator") as demo:
+        gr.Markdown(
+            """
+            # Lip‑Sync Video Generator
+
+            Upload an image (PNG/JPG) or short video of an avatar and an audio file (MP3/WAV/M4A) between 1 – 10 minutes.  Click **Generate video** to create a new video where the avatar lip‑syncs to the audio.  The first generation may take several minutes because the model needs to be downloaded.
+            """
+        )
+        with gr.Row():
+            avatar_input = gr.File(label="Avatar image/video", file_count="single")
+            audio_input = gr.File(label="Audio (1–10 min)", file_count="single")
+        generate_btn = gr.Button("Generate video")
+        result_video = gr.Video(label="Output video", interactive=False)
+
+        generate_btn.click(
+            fn=generate_video,
+            inputs=[avatar_input, audio_input],
+            outputs=result_video,
+            show_progress=True,
+        )
+    return demo
+
+
+if __name__ == "__main__":
+    demo = build_interface()
+    # Queue requests so multiple users can use the model concurrently
+    demo.queue().launch()

requirements.txt

@@ -0,0 +1,15 @@
+gradio==4.43.0
+numpy
+opencv-python
+scipy
+pydub
+moviepy
+torch
+torchvision
+gitpython
+ffmpeg-python
+tqdm
+requests
+gfpgan
+realesrgan
+basicsr