Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+demonstration/demo.mp4 filter=lfs diff=lfs merge=lfs -text

.github/workflows/update_space.yml

@@ -0,0 +1,28 @@
+name: Run Python script
+
+on:
+  push:
+    branches:
+      - main
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+    - name: Checkout
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: '3.9'
+
+    - name: Install Gradio
+      run: python -m pip install gradio
+
+    - name: Log in to Hugging Face
+      run: python -c 'import huggingface_hub; huggingface_hub.login(token="${{ secrets.hf_token }}")'
+
+    - name: Deploy to Spaces
+      run: gradio deploy

.gitignore

@@ -0,0 +1,3 @@
+.env
+__pycache__
+.gradio

README.md

@@ -1,14 +1,140 @@
 ---
 title: SyncDub
-emoji: 🔥
-colorFrom: green
-colorTo: blue
+app_file: gradio_app.py
 sdk: gradio
 sdk_version: 5.29.0
-app_file: app.py
-pinned: false
-license: mit
-short_description: Synchronised dubbing with voice cloning
 ---
+# SyncDub
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+SyncDub is a Python application designed to automatically translate and dub videos into various languages. It leverages speech recognition, speaker diarization, machine translation, and text-to-speech (TTS) technologies to create dubbed versions of input videos while attempting to preserve background audio and optionally clone speaker voices.
+
+## Features
+
+*   **Video Input:** Accepts YouTube URLs or local video file uploads.
+*   **Audio Extraction & Separation:** Extracts audio from video and separates speech from background noise using Demucs.
+*   **Speech Recognition:** Transcribes the spoken content using Whisper.
+*   **Speaker Diarization:** Identifies different speakers in the audio using `pyannote.audio`.
+*   **Machine Translation:** Translates the transcribed text into multiple target languages using `deep-translator` (with options for batch, iterative, or Groq API methods).
+*   **Text-to-Speech (TTS):**
+    *   **Simple Dubbing:** Generates dubbed audio using Microsoft Edge TTS, allowing gender selection per speaker.
+    *   **Voice Cloning:** Uses Coqui XTTS to clone the original speakers' voices for a more natural dub (requires reference audio extraction).
+*   **Audio Mixing:** Combines the generated dubbed speech with the original background audio.
+*   **Video Reassembly:** Creates the final dubbed video file.
+*   **Subtitle Generation:** Outputs translated subtitles in `.srt` format.
+*   **Web Interface:** Provides an easy-to-use Gradio interface for processing videos.
+*   **Command-Line Interface:** Includes a `demo.py` script for terminal-based usage.
+
+## Requirements
+
+*   Python 3.8+
+*   FFmpeg (must be installed and accessible in your system's PATH)
+*   Key Python packages (see `requirements.txt`):
+    *   `yt-dlp`
+    *   `moviepy`
+    *   `pyannote.audio`
+    *   `transformers`
+    *   `torch` & `torchaudio` (often dependencies of the above)
+    *   `deep-translator`
+    *   `TTS` (Coqui TTS for voice cloning)
+    *   `edge-tts`
+    *   `gradio`
+    *   `python-dotenv`
+    *   `demucs`
+*   **API Keys:**
+    *   `HUGGINGFACE_TOKEN`: Required for `pyannote.audio` speaker diarization models. Obtain from [Hugging Face](https://huggingface.co/settings/tokens).
+    *   `GROQ_API_KEY` (Optional): Required if using the "groq" translation method. Obtain from [GroqCloud](https://console.groq.com/keys).
+
+## Setup
+
+1.  **Clone the repository:**
+    ```bash
+    git clone https://github.com/PranavInani/SyncDub
+    cd SyncDub
+    ```
+2.  **Create and activate a virtual environment (recommended):**
+    ```bash
+    python -m venv venv
+    source venv/bin/activate  # On Windows use `venv\Scripts\activate`
+    ```
+3.  **Install FFmpeg:** Follow instructions for your operating system ([https://ffmpeg.org/download.html](https://ffmpeg.org/download.html)). Ensure it's added to your system's PATH.
+4.  **Install Python dependencies:**
+    ```bash
+    pip install -r requirements.txt
+    # Depending on your system, you might need to install PyTorch separately
+    # See https://pytorch.org/get-started/locally/
+    ```
+5.  **Create a `.env` file:** In the root directory of the project, create a file named `.env` and add your Hugging Face token:
+    ```dotenv
+    # filepath: .env
+    HUGGINGFACE_TOKEN=hf_YOUR_HUGGINGFACE_TOKEN_HERE
+    # Add GROQ_API_KEY if you plan to use the Groq translation method
+    # GROQ_API_KEY=gsk_YOUR_GROQ_API_KEY_HERE
+    ```
+    Replace `hf_YOUR_HUGGINGFACE_TOKEN_HERE` with your actual token.
+
+## Usage
+
+### Gradio Web Interface
+
+Launch the web application:
+
+```bash
+python gradio_app.py
+```
+
+This will start a local web server. Open the provided URL (usually `http://127.0.0.1:7860`) in your browser.
+
+1.  **Input:** Enter a video URL (e.g., YouTube) or upload a local video file.
+2.  **Target Language:** Select the desired output language.
+3.  **TTS Method:** Choose between "Simple dubbing (Edge TTS)" or "Voice cloning (XTTS)".
+4.  **Translation Method:** Select "batch", "iterative", or "groq".
+5.  **Maximum Speakers:** Optionally specify the number of speakers to detect. Click "Update Speaker Options" to configure genders if needed (especially for Edge TTS).
+6.  **Process:** Click "Process Video".
+7.  **Output:** Download links for the dubbed video and subtitle file will appear upon completion. Use the "Reset Everything" button to clear temporary files before processing a new video.
+
+### Command-Line Demo
+
+Run the demo script:
+
+```bash
+python demo.py
+```
+
+The script will prompt you for:
+
+1.  Video URL or local file path.
+2.  Target language code (e.g., `en`, `es`, `hi`).
+3.  TTS engine choice (1 for Edge TTS, 2 for XTTS).
+4.  Maximum number of speakers (optional).
+5.  Speaker genders (if using Edge TTS or as fallback for XTTS).
+
+The processed files will be saved in the `temp` directory, with the final video typically named `output_video.mp4`.
+
+## Configuration
+
+*   **API Keys:** Configure `HUGGINGFACE_TOKEN` and optionally `GROQ_API_KEY` in the `.env` file.
+*   **Models:** Model sizes and specific checkpoints can be adjusted within the Python scripts (`speech_recognition.py`, `speech_diarization.py`, etc.) if needed.
+
+## Directory Structure
+
+*   `temp/`: Stores intermediate files like downloaded video, extracted audio, separated sources, final output video.
+*   `audio/`: Often used for initial audio extraction outputs.
+*   `audio2/`: Stores the generated TTS audio segments and the final mixed dubbed audio.
+*   `reference_audio/`: Stores extracted audio snippets for each speaker when using XTTS voice cloning.
+*   `outputs/`: Stores the final video and subtitle files made available for download in the Gradio interface.
+
+## Troubleshooting
+
+*   **Errors during processing:**
+    *   Ensure `ffmpeg` is installed correctly and accessible in your PATH.
+    *   Verify that the `HUGGINGFACE_TOKEN` in your `.env` file is correct and valid.
+    *   Check if you have sufficient disk space and memory, especially for large videos or voice cloning.
+    *   Ensure all dependencies from `requirements.txt` are installed correctly in your virtual environment.
+*   **Voice cloning issues:** XTTS quality depends heavily on the quality and duration of the extracted reference audio for each speaker. If diarization is poor or speakers have very little unique speech, cloning may fail or produce poor results. Consider using "Simple dubbing" as an alternative.
+*   **Slow processing:** Video processing, especially diarization, translation, and TTS (XTTS), can be computationally intensive and time-consuming.
+*   **Reset:** Use the "Reset Everything" button in the Gradio app to clear temporary directories if you encounter persistent issues or before starting a new video.
+
+
+## Contributing
+
+Everyone is encouraged to contribute.

audio_to_video.py

@@ -0,0 +1,77 @@
+import os
+import subprocess
+from IPython.display import FileLink, display
+
+def create_video_with_mixed_audio(
+    main_video_path, 
+    background_music_path, 
+    main_audio_path, 
+    temp_dir="temp",  # Directory for temporary files
+    bg_volume=0.3,
+    main_audio_volume=1.0
+):
+    """
+    Create a video with mixed audio (main audio + background music)
+    
+    Parameters:
+        main_video_path (str): Path to the main video file
+        background_music_path (str): Path to the background music file
+        main_audio_path (str): Path to the main audio (dubbed speech)
+        temp_dir (str): Directory for temporary files
+        bg_volume (float): Volume level for background music (0.0-1.0)
+        main_audio_volume (float): Volume level for main audio (0.0-1.0)
+        
+    Returns:
+        bool: True if successful, False otherwise
+    """
+    
+    try:
+        # Ensure the temporary directory exists
+        os.makedirs(temp_dir, exist_ok=True)
+        
+        # Define paths for temporary and output files
+        temp_audio_path = os.path.join(temp_dir, "mixed_audio.wav")
+        output_video_path = os.path.join(temp_dir, "output_video.mp4")
+        
+        # Step 1: Mix the background audio and main audio with volume control
+        print("Step 1: Mixing audio tracks...")
+        mix_command = f'''ffmpeg -i "{background_music_path}" -i "{main_audio_path}" -filter_complex \
+            "[0:a]volume={bg_volume}[bg]; \
+             [1:a]volume={main_audio_volume}[main]; \
+             [bg][main]amix=inputs=2:duration=first:dropout_transition=2" \
+            "{temp_audio_path}" -y'''
+        
+        subprocess.run(mix_command, shell=True, check=True)
+        
+        # Step 2: Replace the original audio in the video with mixed audio
+        print("Step 2: Creating final video with mixed audio...")
+        video_command = f'''ffmpeg -i "{main_video_path}" -i "{temp_audio_path}" \
+            -c:v copy -map 0:v:0 -map 1:a:0 -shortest -c:a aac \
+            "{output_video_path}" -y'''
+        
+        subprocess.run(video_command, shell=True, check=True)
+        
+        # Check if output file exists and has a reasonable size
+        if os.path.exists(output_video_path) and os.path.getsize(output_video_path) > 1000:
+            print(f"✅ Success! Video created at: {output_video_path}")
+            print(f"File size: {os.path.getsize(output_video_path) / (1024*1024):.2f} MB")
+            
+            # Display download link in Jupyter/Colab
+            try:
+                display(FileLink(output_video_path))
+            except:
+                pass
+                
+            return True
+        else:
+            print("❌ Something went wrong. Output file is missing or too small.")
+            return False
+            
+    except Exception as e:
+        print(f"❌ Error during processing: {str(e)}")
+        return False
+    finally:
+        # Optional: Clean up temporary files
+        # if os.path.exists(temp_audio_path):
+        #     os.remove(temp_audio_path)
+        pass

demo.py

@@ -0,0 +1,166 @@
+import os
+import sys
+import logging
+from dotenv import load_dotenv
+import re
+
+# Add the current directory to path to help with imports
+current_dir = os.path.dirname(os.path.abspath(__file__))
+if current_dir not in sys.path:
+    sys.path.append(current_dir)
+
+# Import the required modules
+from media_ingestion import MediaIngester
+from speech_recognition import SpeechRecognizer
+from speech_diarization import SpeakerDiarizer
+from translate import translate_text, generate_srt_subtitles
+from text_to_speech import generate_tts  # Import both TTS functions
+from audio_to_video import create_video_with_mixed_audio
+
+def create_directories(dirs):
+    """Create necessary directories"""
+    for directory in dirs:
+        os.makedirs(directory, exist_ok=True)
+
+def main():
+    # Load environment variables
+    load_dotenv()
+    
+    # Configure logging
+    logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+    logger = logging.getLogger(__name__)
+    
+    # Create necessary directories
+    create_directories(["temp", "audio", "audio2", "reference_audio"])
+    
+    # Get API tokens
+    hf_token = os.getenv("HUGGINGFACE_TOKEN")
+    
+    if not hf_token:
+        logger.error("Error: HUGGINGFACE_TOKEN not found in .env file")
+        return
+    
+    # Get input from user
+    media_source = input("Enter video URL or local file path: ")
+    target_language = input("Enter target language code (e.g., en, es, fr, de): ")
+    
+    # Choose TTS engine
+    print("\nSelect TTS engine:")
+    print("1. Simple dubbing (Edge TTS)")
+    print("2. Voice cloning (XTTS)")
+    tts_choice = input("Enter choice (1/2): ").strip()
+    use_voice_cloning = tts_choice == "2"
+    
+    # Initialize components
+    logger.info("Initializing pipeline components...")
+    ingester = MediaIngester(output_dir="temp")
+    recognizer = SpeechRecognizer(model_size="base")
+    diarizer = SpeakerDiarizer(hf_token=hf_token)
+    
+    # Step 1: Process input and extract audio
+    logger.info("Processing media source...")
+    video_path = ingester.process_input(media_source)
+    audio_path = ingester.extract_audio(video_path)
+    clean_audio_path, bg_audio_path = ingester.separate_audio_sources(audio_path)
+    logger.info("Extracted audio: %s", audio_path)
+    logger.info("Cleaned audio: %s", clean_audio_path)
+    logger.info("Background audio: %s", bg_audio_path)
+    logger.info("Audio processing completed.")
+    
+    # Step 2: Perform speech recognition
+    logger.info("Transcribing audio...")
+    segments = recognizer.transcribe(clean_audio_path)
+    
+    # Step 3: Perform speaker diarization
+    logger.info("Identifying speakers...")
+    
+    # Add user input for max speakers
+    max_speakers_str = input("Maximum number of speakers to detect (leave blank for auto): ")
+    max_speakers = int(max_speakers_str) if max_speakers_str.strip() else None
+
+    # Then call diarize with this parameter
+    speakers = diarizer.diarize(clean_audio_path, max_speakers=max_speakers)
+    
+    # Step 4: Assign speakers to segments
+    logger.info("Assigning speakers to segments...")
+    final_segments = diarizer.assign_speakers_to_segments(segments, speakers)
+    
+    # Step 5: Translate the segments
+    logger.info(f"Translating to {target_language}...")
+    translated_segments = translate_text(
+        final_segments, 
+        target_lang=target_language,
+        translation_method="batch"  # Can be "batch" or "iterative" or "groq"
+    )
+
+    # Print translated segments for debugging
+    subtitle_file = f"temp/{os.path.basename(video_path).split('.')[0]}_{target_language}.srt"
+    generate_srt_subtitles(translated_segments, output_file=subtitle_file)
+    logger.info(f"Generated subtitle file: {subtitle_file}")
+    
+    # Step 6: Configure voice characteristics for speakers
+    voice_config = {}  # Map of speaker_id to gender or voice config
+
+    # Detect number of unique speakers
+    unique_speakers = set()
+    for segment in translated_segments:
+        if 'speaker' in segment:
+            unique_speakers.add(segment['speaker'])
+    
+    logger.info(f"Detected {len(unique_speakers)} speakers")
+    
+    if use_voice_cloning:
+        # Extract reference audio for voice cloning
+        logger.info("Extracting speaker reference audio for voice cloning...")
+        reference_files = diarizer.extract_speaker_references(
+            clean_audio_path, 
+            speakers, 
+            output_dir="reference_audio"
+        )
+        
+        # Create voice config for XTTS
+        for speaker in sorted(list(unique_speakers)):
+            match = re.search(r'SPEAKER_(\d+)', speaker)
+            if match:
+                speaker_id = int(match.group(1))
+                if speaker in reference_files:
+                    voice_config[speaker_id] = {
+                        'engine': 'xtts',
+                        'reference_audio': reference_files[speaker],
+                        'language': target_language
+                    }
+                    logger.info(f"Using voice cloning for Speaker {speaker_id+1} with reference file: {os.path.basename(reference_files[speaker])}")
+                else:
+                    # Fallback to Edge TTS if no reference audio
+                    logger.warning(f"No reference audio found for Speaker {speaker_id+1}, falling back to Edge TTS")
+                    gender = input(f"Select voice gender for Speaker {speaker_id+1} (m/f): ").lower()
+                    voice_config[speaker_id] = {
+                        'engine': 'edge_tts',
+                        'gender': "female" if gender.startswith("f") else "male"
+                    }
+    else:
+        # Standard Edge TTS configuration - keeping your current approach
+        if len(unique_speakers) > 0:
+            for speaker in sorted(list(unique_speakers)):
+                match = re.search(r'SPEAKER_(\d+)', speaker)
+                if match:
+                    speaker_id = int(match.group(1))
+                    gender = input(f"Select voice gender for Speaker {speaker_id+1} (m/f): ").lower()
+                    voice_config[speaker_id] = {
+                        'engine': 'edge_tts',
+                        'gender': "female" if gender.startswith("f") else "male"
+                    }
+    
+    # Step 7: Generate speech in target language
+    logger.info("Generating speech...")
+    
+    dubbed_audio_path = generate_tts(translated_segments, target_language, voice_config, output_dir="audio2")
+
+    # Step 8: Create video with mixed audio
+    logger.info("Creating video with translated audio...")
+    create_video_with_mixed_audio(video_path, bg_audio_path, dubbed_audio_path)
+    
+    logger.info("Process completed successfully!")
+
+if __name__ == "__main__":
+    main()

demonstration/demo.mp4

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5cbbcaef260aaf1256756f6ffa67d7a6fc8b25a71daeba4a22052d4773f8e767
+size 13703532

gradio_app.py

@@ -0,0 +1,661 @@
+import os
+import sys
+import logging
+import tempfile
+import re
+import gradio as gr
+from dotenv import load_dotenv
+import threading
+import shutil
+
+# Add the current directory to path to help with imports
+current_dir = os.path.dirname(os.path.abspath(__file__))
+if current_dir not in sys.path:
+    sys.path.append(current_dir)
+
+# Import the required modules
+from media_ingestion import MediaIngester
+from speech_recognition import SpeechRecognizer
+from speech_diarization import SpeakerDiarizer
+from translate import translate_text, generate_srt_subtitles
+from text_to_speech import generate_tts
+from audio_to_video import create_video_with_mixed_audio
+
+# Load environment variables
+load_dotenv()
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Create necessary directories
+os.makedirs("temp", exist_ok=True)
+os.makedirs("audio", exist_ok=True)
+os.makedirs("audio2", exist_ok=True)
+os.makedirs("reference_audio", exist_ok=True)
+os.makedirs("outputs", exist_ok=True)  # Add directory for downloadable outputs
+
+# Global variables for process tracking
+processing_status = {}
+
+def create_session_id():
+    """Create a unique session ID for tracking progress"""
+    import uuid
+    return str(uuid.uuid4())[:8]
+
+def reset_application():
+    """
+    Reset the application state by thoroughly cleaning all temporary files and directories
+    """
+    import os
+    import shutil
+    import time
+    
+    # Directories to completely clean
+    directories_to_clean = ["temp", "audio", "audio2", "reference_audio", "outputs"]
+    
+    try:
+        # First attempt - delete individual files
+        for directory in directories_to_clean:
+            if os.path.exists(directory):
+                logger.info(f"Cleaning directory: {directory}")
+                for filename in os.listdir(directory):
+                    file_path = os.path.join(directory, filename)
+                    try:
+                        if os.path.isfile(file_path):
+                            os.unlink(file_path)
+                            logger.info(f"Deleted file: {file_path}")
+                        elif os.path.isdir(file_path):
+                            shutil.rmtree(file_path)
+                            logger.info(f"Deleted subdirectory: {file_path}")
+                    except Exception as e:
+                        logger.warning(f"Failed to delete {file_path}: {e}")
+                        
+        # Double-check if any files remain (for stubborn files)
+        for directory in directories_to_clean:
+            if os.path.exists(directory) and any(os.scandir(directory)):
+                # Try more aggressive approach - recreate the directory
+                try:
+                    # Rename the old directory
+                    temp_dir = f"{directory}_old_{int(time.time())}"
+                    os.rename(directory, temp_dir)
+                    # Create a new empty directory
+                    os.makedirs(directory, exist_ok=True)
+                    # Try to remove the old directory in background
+                    try:
+                        shutil.rmtree(temp_dir)
+                    except:
+                        pass  # Ignore if we can't delete it immediately
+                except Exception as e:
+                    logger.warning(f"Failed to recreate directory {directory}: {e}")
+        
+        # Clear out individual files that might be in the root directory
+        root_files_to_check = [
+            "dubbed_conversation.wav", 
+            "downloaded_video.mp4",
+            "downloaded_video_hi.srt",
+            "extracted_audio.wav",
+            "mixed_audio.wav",
+            "output_video.mp4"
+        ]
+        
+        for filename in root_files_to_check:
+            if os.path.exists(filename):
+                try:
+                    os.unlink(filename)
+                    logger.info(f"Deleted root file: {filename}")
+                except Exception as e:
+                    logger.warning(f"Failed to delete root file {filename}: {e}")
+                        
+        # Reset the global processing status
+        global processing_status
+        processing_status = {}
+        
+        # Generate a new session ID
+        new_session_id = create_session_id()
+        
+        # For visual confirmation to the user, return timestamp of the reset
+        timestamp = time.strftime("%H:%M:%S", time.localtime())
+        success_message = f"⚠️ Application reset complete at {timestamp}. All temporary files cleared. Ready for new processing."
+        
+        return {
+            "new_status": success_message,
+            "session_id": new_session_id,
+            "media_input": None,  # Clear media input field
+            "output": None,       # Clear output file
+            "subtitle": None,     # Clear subtitle file
+            "message": ""         # Clear output message
+        }
+        
+    except Exception as e:
+        logger.exception("Error during application reset")
+        timestamp = time.strftime("%H:%M:%S", time.localtime())
+        return {
+            "new_status": f"⚠️ Reset attempted at {timestamp} but encountered errors: {str(e)}",
+            "session_id": None,
+            "media_input": None,
+            "output": None,
+            "subtitle": None,
+            "message": ""
+        }
+
+def process_video(media_source, target_language, tts_choice, max_speakers, speaker_genders, session_id, translation_method="batch", progress=gr.Progress()):
+    """Main processing function that handles the complete pipeline"""
+    global processing_status
+    processing_status[session_id] = {"status": "Starting", "progress": 0}
+    
+    try:
+        # Get API tokens
+        hf_token = os.getenv("HUGGINGFACE_TOKEN")
+        
+        if not hf_token:
+            return {"error": True, "message": "Error: HUGGINGFACE_TOKEN not found in .env file"}
+        
+        # Determine if input is URL or file
+        is_url = media_source.startswith(("http://", "https://"))
+        
+        # Initialize components
+        progress(0.05, desc="Initializing components")
+        processing_status[session_id] = {"status": "Initializing components", "progress": 0.05}
+        
+        ingester = MediaIngester(output_dir="temp")
+        recognizer = SpeechRecognizer(model_size="base")
+        diarizer = SpeakerDiarizer(hf_token=hf_token)
+        
+        # Step 1: Process input and extract audio
+        progress(0.1, desc="Processing media source")
+        processing_status[session_id] = {"status": "Processing media source", "progress": 0.1}
+        
+        video_path = ingester.process_input(media_source)
+        audio_path = ingester.extract_audio(video_path)
+        
+        progress(0.15, desc="Separating audio sources")
+        processing_status[session_id] = {"status": "Separating audio sources", "progress": 0.15}
+        
+        clean_audio_path, bg_audio_path = ingester.separate_audio_sources(audio_path)
+        
+        # Step 2: Perform speech recognition
+        progress(0.2, desc="Transcribing audio")
+        processing_status[session_id] = {"status": "Transcribing audio", "progress": 0.2}
+        
+        segments = recognizer.transcribe(clean_audio_path)
+        
+        # Step 3: Perform speaker diarization
+        progress(0.3, desc="Identifying speakers")
+        processing_status[session_id] = {"status": "Identifying speakers", "progress": 0.3}
+        
+        # Convert max_speakers to int or None
+        max_speakers_val = int(max_speakers) if max_speakers and max_speakers.strip() else None
+        
+        # Diarize audio to identify speakers
+        speakers = diarizer.diarize(clean_audio_path, max_speakers=max_speakers_val)
+        
+        # Step 4: Assign speakers to segments
+        progress(0.4, desc="Assigning speakers to segments")
+        processing_status[session_id] = {"status": "Assigning speakers to segments", "progress": 0.4}
+        
+        final_segments = diarizer.assign_speakers_to_segments(segments, speakers)
+        
+        # Step 5: Translate the segments
+        progress(0.5, desc=f"Translating to {target_language}")
+        processing_status[session_id] = {"status": f"Translating to {target_language}", "progress": 0.5}
+        
+        # Validate target language
+        valid_languages = ["en", "es", "fr", "de", "it", "ja", "ko", "pt", "ru", "zh", "hi"]
+        if target_language not in valid_languages:
+            logger.warning(f"Unsupported language: {target_language}, falling back to English")
+            target_language = "en"
+        
+        translated_segments = translate_text(
+            final_segments, 
+            target_lang=target_language,
+            translation_method=translation_method
+        )
+        
+        # Generate subtitle file
+        subtitle_file = f"temp/{os.path.basename(video_path).split('.')[0]}_{target_language}.srt"
+        generate_srt_subtitles(translated_segments, output_file=subtitle_file)
+        
+        # Step 6: Configure voice characteristics for speakers
+        progress(0.6, desc="Configuring voices")
+        processing_status[session_id] = {"status": "Configuring voices", "progress": 0.6}
+        
+        # Detect number of unique speakers
+        unique_speakers = set()
+        for segment in translated_segments:
+            if 'speaker' in segment:
+                unique_speakers.add(segment['speaker'])
+        
+        logger.info(f"Detected {len(unique_speakers)} speakers")
+        
+        # Use provided speaker genders
+        use_voice_cloning = tts_choice == "Voice cloning (XTTS)"
+        voice_config = {}  # Map of speaker_id to gender or voice config
+        
+        if use_voice_cloning:
+            # Extract reference audio for voice cloning
+            logger.info("Extracting speaker reference audio for voice cloning...")
+            reference_files = diarizer.extract_speaker_references(
+                clean_audio_path, 
+                speakers, 
+                output_dir="reference_audio"
+            )
+            
+            # Create voice config for XTTS
+            for speaker in sorted(list(unique_speakers)):
+                match = re.search(r'SPEAKER_(\d+)', speaker)
+                if match:
+                    speaker_id = int(match.group(1))
+                    if speaker in reference_files:
+                        voice_config[speaker_id] = {
+                            'engine': 'xtts',
+                            'reference_audio': reference_files[speaker],
+                            'language': target_language  # Use the validated target language
+                        }
+                        logger.info(f"Using voice cloning for Speaker {speaker_id+1} with reference file: {os.path.basename(reference_files[speaker])}")
+                    else:
+                        # Fallback to Edge TTS if no reference audio
+                        logger.warning(f"No reference audio found for Speaker {speaker_id+1}, falling back to Edge TTS")
+                        gender = "female"  # Default fallback
+                        if str(speaker_id) in speaker_genders and speaker_genders[str(speaker_id)]:
+                            gender = speaker_genders[str(speaker_id)]
+                        
+                        voice_config[speaker_id] = {
+                            'engine': 'edge_tts',
+                            'gender': gender
+                        }
+        else:
+            # Standard Edge TTS configuration
+            if len(unique_speakers) > 0:
+                for speaker in sorted(list(unique_speakers)):
+                    match = re.search(r'SPEAKER_(\d+)', speaker)
+                    if match:
+                        speaker_id = int(match.group(1))
+                        gender = "female" if speaker_id % 2 == 0 else "male"  # Default fallback
+                        
+                        # Use selected gender if available
+                        if str(speaker_id) in speaker_genders and speaker_genders[str(speaker_id)]:
+                            gender = speaker_genders[str(speaker_id)]
+                            
+                        voice_config[speaker_id] = gender
+        
+        # Step 7: Generate speech in target language
+        progress(0.7, desc=f"Generating speech in {target_language}")
+        processing_status[session_id] = {"status": f"Generating speech in {target_language}", "progress": 0.7}
+        
+        dubbed_audio_path = generate_tts(translated_segments, target_language, voice_config, output_dir="audio2")
+        
+        # Step 8: Create video with mixed audio
+        progress(0.85, desc="Creating final video")
+        processing_status[session_id] = {"status": "Creating final video", "progress": 0.85}
+        
+        success = create_video_with_mixed_audio(
+            main_video_path=video_path, 
+            background_music_path=bg_audio_path, 
+            main_audio_path=dubbed_audio_path
+        )
+        
+        if not success:
+            raise RuntimeError("Failed to create final video with audio")
+        
+        # Use known output path since function returns boolean
+        output_video_path = os.path.join("temp", "output_video.mp4")
+        
+        # Verify the output video exists
+        if not os.path.exists(output_video_path):
+            raise FileNotFoundError(f"Output video not found at expected path: {output_video_path}")
+        
+        # Create downloadable copies with unique names
+        file_basename = os.path.basename(video_path).split('.')[0]
+        downloadable_video = f"outputs/{file_basename}_{target_language}_{session_id}.mp4"
+        downloadable_subtitle = f"outputs/{file_basename}_{target_language}_{session_id}.srt"
+        
+        # Copy files to outputs directory for download
+        shutil.copy2(output_video_path, downloadable_video)
+        shutil.copy2(subtitle_file, downloadable_subtitle)
+            
+        # Complete
+        progress(1.0, desc="Process completed")
+        processing_status[session_id] = {"status": "Completed", "progress": 1.0}
+        
+        return {
+            "error": False,
+            "video": downloadable_video,
+            "subtitle": downloadable_subtitle,
+            "message": "Process completed successfully! Click on the files to download."
+        }
+        
+    except Exception as e:
+        logger.exception("Error in processing pipeline")
+        processing_status[session_id] = {"status": f"Error: {str(e)}", "progress": -1}
+        return {"error": True, "message": f"Error: {str(e)}"}
+
+def get_processing_status(session_id):
+    """Get the current processing status for the given session"""
+    global processing_status
+    if session_id in processing_status:
+        return processing_status[session_id]["status"]
+    return "No status available"
+
+def check_api_tokens():
+    """Check if required API tokens are set"""
+    missing_tokens = []
+    
+    if not os.getenv("HUGGINGFACE_TOKEN"):
+        missing_tokens.append("HUGGINGFACE_TOKEN")
+    
+    if missing_tokens:
+        return f"Warning: Missing API tokens: {', '.join(missing_tokens)}. Please set them in your .env file."
+    else:
+        return "All required API tokens are set."
+
+def check_system_state():
+    """
+    Check the state of all directories and report what files still exist
+    """
+    import os
+    
+    # Directories to check
+    directories_to_check = ["temp", "audio", "audio2", "reference_audio", "outputs"]
+    report = []
+    
+    for directory in directories_to_check:
+        if os.path.exists(directory):
+            files = os.listdir(directory)
+            if files:
+                report.append(f"Directory '{directory}' contains {len(files)} files: {', '.join(files[:5])}")
+                if len(files) > 5:
+                    report.append(f"... and {len(files) - 5} more files")
+            else:
+                report.append(f"Directory '{directory}' is empty")
+        else:
+            report.append(f"Directory '{directory}' does not exist")
+    
+    # Check root files
+    root_files = [f for f in os.listdir('.') if os.path.isfile(f)]
+    if root_files:
+        report.append(f"Root directory contains {len(root_files)} files")
+    
+    return "\n".join(report)
+
+# Define the Gradio interface
+def create_interface():
+    with gr.Blocks(title="SyncDub - Video Translation and Dubbing") as app:
+        gr.Markdown("# SyncDub - Video Translation and Dubbing")
+        gr.Markdown("Translate and dub videos to different languages with speaker diarization")
+        
+        session_id = create_session_id()  # Create a session ID for tracking progress
+        
+        with gr.Tab("Process Video"):
+            with gr.Row():
+                with gr.Column(scale=2):
+                    media_input = gr.Textbox(label="Video URL or File Upload", placeholder="Enter a YouTube URL or upload a video file")
+                    
+                    with gr.Row():
+                        # Enhanced language dropdown with full language names
+                        target_language = gr.Dropdown(
+                            choices=[
+                                ("English", "en"), 
+                                ("Spanish", "es"), 
+                                ("French", "fr"), 
+                                ("German", "de"), 
+                                ("Hindi", "hi"),
+                                ("Italian", "it"), 
+                                ("Japanese", "ja"), 
+                                ("Korean", "ko"), 
+                                ("Portuguese", "pt"), 
+                                ("Russian", "ru"), 
+                                ("Chinese", "zh")
+                            ],
+                            label="Target Language",
+                            value="hi"
+                        )
+                        tts_choice = gr.Radio(
+                            choices=["Simple dubbing (Edge TTS)", "Voice cloning (XTTS)"],
+                            label="TTS Method",
+                            value="Simple dubbing (Edge TTS)"
+                        )
+                    
+                    # Add translation method selection
+                    with gr.Row():
+                        translation_method = gr.Radio(
+                            choices=["batch", "iterative", "groq"],
+                            label="Translation Method",
+                            value="batch",
+                            info="Batch: Faster for longer content. Iterative: May be more accurate for short content. Groq: Uses Groq LLM API."
+                        )
+                    
+                    # Speaker count input and update button
+                    with gr.Row():
+                        max_speakers = gr.Textbox(label="Maximum number of speakers", placeholder="Leave blank for auto")
+                        update_speakers_btn = gr.Button("Update Speaker Options")
+                    
+                    # Speaker gender container
+                    with gr.Group(visible=False) as speaker_genders_container:
+                        gr.Markdown("### Speaker Gender Selection")
+                        speaker_genders = {}
+                        for i in range(8):  # Support up to 8 speakers
+                            speaker_genders[str(i)] = gr.Radio(
+                                choices=["male", "female"],
+                                value="male" if i % 2 == 1 else "female",
+                                label=f"Speaker {i} Gender",
+                                visible=False  # Initially hidden
+                            )
+                    
+                    process_btn = gr.Button("Process Video", variant="primary")
+                    status_text = gr.Textbox(label="Status", value="Ready", interactive=False)
+                
+                with gr.Column(scale=3):
+                    # Replace video display with file downloads
+                    gr.Markdown("### Output Files")
+                    output_message = gr.Textbox(label="Status", interactive=False)
+                    with gr.Row():
+                        output = gr.File(label="Download Video")
+                        subtitle_output = gr.File(label="Download Subtitles")
+            
+            # Function to update speaker gender options
+            def update_speaker_options(max_speakers_value):
+                updates = {}
+                
+                try:
+                    num_speakers = int(max_speakers_value) if max_speakers_value.strip() else 0
+                    
+                    if num_speakers > 0:
+                        # Show the speaker gender container
+                        updates[speaker_genders_container] = gr.Group(visible=True)
+                        
+                        # Show only the relevant number of speaker options
+                        for i in range(8):
+                            updates[speaker_genders[str(i)]] = gr.Radio(
+                                visible=(i < num_speakers)
+                            )
+                    else:
+                        # Hide all if no valid number
+                        updates[speaker_genders_container] = gr.Group(visible=False)
+                except ValueError:
+                    # Hide all if invalid number
+                    updates[speaker_genders_container] = gr.Group(visible=False)
+                
+                return updates
+            
+            # Connect the update button to show/hide speaker options
+            update_speakers_btn.click(
+                fn=update_speaker_options,
+                inputs=[max_speakers],
+                outputs=[speaker_genders_container] + [speaker_genders[str(i)] for i in range(8)]
+            )
+            
+            # Function to actually pass the gender values to the process_video function
+            def process_with_genders(media_source, target_language, tts_choice, max_speakers, translation_method, *gender_values):
+                # Convert the gender values into a dictionary to pass to process_video
+                speaker_genders_dict = {str(i): gender for i, gender in enumerate(gender_values) if gender}
+                result = process_video(media_source, target_language, tts_choice, max_speakers, 
+                                      speaker_genders_dict, session_id, translation_method=translation_method)
+                
+                # Return the output values based on whether there was an error
+                if result.get("error", False):
+                    return None, None, result.get("message", "An error occurred")
+                else:
+                    return result.get("video"), result.get("subtitle"), result.get("message")
+            
+            # Connect the process button
+            process_btn.click(
+                fn=process_with_genders, 
+                inputs=[
+                    media_input, 
+                    target_language, 
+                    tts_choice, 
+                    max_speakers,
+                    translation_method,  # Add translation method to inputs
+                    # Pass individual radio components, not a Group
+                    *[speaker_genders[str(i)] for i in range(8)]
+                ],
+                outputs=[output, subtitle_output, output_message]
+            )
+            
+            # Update status periodically
+            status_timer = gr.Timer(2, lambda: get_processing_status(session_id), None, status_text)
+            
+            # Create a more compatible approach for status updates
+            def start_status_updates(session_id):
+                def update_status_thread():
+                    import time
+                    while session_id in processing_status and processing_status[session_id]["progress"] < 1.0:
+                        try:
+                            time.sleep(1)  # Update status every second
+                            # This is a workaround since we can't use JavaScript directly
+                        except:
+                            break
+                
+                thread = threading.Thread(target=update_status_thread)
+                thread.daemon = True  # Thread will exit when main program exits
+                thread.start()
+                return "Processing started"
+            
+            # Status checking function
+            def check_status(session_id):
+                status = get_processing_status(session_id)
+                return status
+            
+            # Define the handle_reset function here
+            def handle_reset():
+                """Handle the reset button click by calling reset_application()"""
+                try:
+                    # Assuming reset_application is defined elsewhere in your code
+                    result = reset_application()
+                    # Inform user that reset is in progress
+                    gr.Info("Resetting application...")
+                    # Return values in the order expected by the outputs list
+                    return (
+                        "Application reset successful. Ready for new video processing.",
+                        None,  # media_input
+                        None,  # output
+                        None,  # subtitle_output
+                        ""     # output_message
+                    )
+                except Exception as e:
+                    logger.exception("Error in reset handler")
+                    return (
+                        f"Reset failed: {str(e)}",
+                        None, None, None, ""
+                    )
+            
+            # Replace multiple button rows with a single row containing both buttons
+            with gr.Row():
+                reset_btn = gr.Button("🗑️ Reset Everything", variant="stop")
+                refresh_btn = gr.Button("Refresh Status")
+            
+            # Connect the refresh button to check status
+            refresh_btn.click(
+                fn=check_status,
+                inputs=[gr.State(session_id)],
+                outputs=[status_text]
+            )
+            
+            # Now use the defined handle_reset function
+            reset_btn.click(
+                fn=handle_reset,
+                inputs=[],
+                outputs=[
+                    status_text,
+                    media_input,
+                    output,
+                    subtitle_output,
+                    output_message
+                ]
+            )
+            
+            # Create a simple auto-refresh component using a Textbox with a timer
+            gr.HTML("""
+            <script>
+            // Simple poller to update status
+            document.addEventListener('DOMContentLoaded', function() {
+                let refreshInterval;
+                
+                // Look for the primary button (Process Video)
+                const processButton = document.querySelector('button.primary');
+                
+                if (processButton) {
+                    // When process starts, begin polling
+                    processButton.addEventListener('click', function() {
+                        if (refreshInterval) clearInterval(refreshInterval);
+                        
+                        // Find the refresh button
+                        const refreshButtons = Array.from(document.querySelectorAll('button'));
+                        const refreshButton = refreshButtons.find(btn => btn.textContent.includes('Refresh Status'));
+                        
+                        if (refreshButton) {
+                            // Start auto-polling every 2 seconds
+                            refreshInterval = setInterval(function() {
+                                refreshButton.click();
+                            }, 2000);
+                            
+                            // Stop polling after 30 minutes (safety)
+                            setTimeout(function() {
+                                if (refreshInterval) clearInterval(refreshInterval);
+                            }, 30*60*1000);
+                        }
+                    });
+                }
+            });
+            </script>
+            """)
+            
+            # Add a debug button to the interface
+            with gr.Row():
+                check_btn = gr.Button("Check System State", variant="secondary")
+                check_output = gr.Textbox(label="System State")
+
+            check_btn.click(fn=check_system_state, inputs=[], outputs=[check_output])
+            
+        with gr.Tab("Help"):
+            gr.Markdown("""
+            ## How to use SyncDub
+            
+            1. **Input**: Enter a YouTube URL or path to a local video file, or upload a video
+            2. **Target Language**: Select the language you want to translate and dub into
+            3. **TTS Engine**: 
+               - **Simple dubbing**: Uses Edge TTS (faster but less natural sounding)
+               - **Voice cloning**: Uses XTTS to clone the original speakers' voices (slower but more natural)
+            4. **Maximum Speakers**: Optionally specify the maximum number of speakers to detect
+            5. **Translation Method**: Choose the translation method (Batch, Iterative, or Groq)
+            6. **Process**: Click the Process Video button to start
+            
+            ## Requirements
+            
+            Make sure you have the following API tokens in your `.env` file:
+            - `HUGGINGFACE_TOKEN`: Required for speech diarization
+            
+            ## Troubleshooting
+            
+            - If you encounter errors, check that all API tokens are set correctly
+            - For large videos, the process may take several minutes
+            - If voice cloning doesn't sound right, try simple dubbing instead
+            """)
+    
+    return app
+
+# Launch the interface
+if __name__ == "__main__":
+    app = create_interface()
+    app.launch(share=True)

media_ingestion.py

@@ -0,0 +1,135 @@
+import yt_dlp
+import os
+from moviepy.editor import VideoFileClip
+import subprocess
+import shutil
+
+class MediaIngester:
+    def __init__(self, output_dir="temp"):
+        self.output_dir = output_dir
+        os.makedirs(output_dir, exist_ok=True)
+        
+    def process_input(self, source):
+        """Process various input types and extract audio"""
+        if source.startswith(("http://", "https://")):
+            # Handle URL (including YouTube)
+            return self.download_from_url(source)
+        elif os.path.isfile(source):
+            # Handle local file
+            return self.process_local_file(source)
+        else:
+            raise ValueError("Input source not recognized")
+    
+    def download_from_url(self, url):
+        """Download media from URL (including YouTube)"""
+        output_path = os.path.join(self.output_dir, "downloaded_video.mp4")
+        ydl_opts = {
+            'format': 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]',
+            'outtmpl': output_path,
+            # Add additional options to make downloads more robust
+            'retries': 10,
+            'fragment_retries': 10,
+            'ignoreerrors': False,
+            'no_warnings': False,
+            'geo_bypass': True,
+        }
+        with yt_dlp.YoutubeDL(ydl_opts) as ydl:
+            ydl.download([url])
+        return output_path
+    
+    def process_local_file(self, file_path):
+        """Process local video or audio file"""
+        # For simplicity, just return the path if it's a valid file
+        return file_path
+        
+    def extract_audio(self, video_path):
+        """Extract audio from video file"""
+        audio_path = os.path.join(self.output_dir, "extracted_audio.wav")
+        video = VideoFileClip(video_path)
+        video.audio.write_audiofile(audio_path)
+        return audio_path
+    
+
+
+    def separate_audio_sources(self, audio_path):
+        """
+        Separate voice and background music from an audio file using Demucs
+        
+        Parameters:
+            audio_path (str): Path to the input audio file
+            
+        Returns:
+            tuple: (voice_audio_path, background_music_path)
+        """
+        # Create output directory for separated audio
+        separation_dir = os.path.join(self.output_dir, "separated")
+        os.makedirs(separation_dir, exist_ok=True)
+        
+        # Final output paths
+        voice_path = os.path.join(separation_dir, "voice.wav")
+        music_path = os.path.join(separation_dir, "music.wav")
+        
+        try:
+            # Method 1: Using Demucs as a command-line tool
+            cmd = [
+                "demucs", "--two-stems=vocals",
+                "-o", separation_dir,
+                audio_path
+            ]
+            
+            print(f"Separating audio sources from {os.path.basename(audio_path)}...")
+            result = subprocess.run(cmd, check=True, capture_output=True, text=True)
+            print("Separation complete.")
+            
+            # Demucs creates a subdirectory with model name and then the base name
+            base_name = os.path.splitext(os.path.basename(audio_path))[0]
+            model_name = "htdemucs"  # default model
+            demucs_output_dir = os.path.join(separation_dir, model_name, base_name)
+            
+            # Get the paths to the separated files
+            actual_voice_path = os.path.join(demucs_output_dir, "vocals.wav") 
+            actual_music_path = os.path.join(demucs_output_dir, "no_vocals.wav")
+            
+            # Move files to their final locations
+            shutil.copy2(actual_voice_path, voice_path)
+            shutil.copy2(actual_music_path, music_path)
+            
+            # Clean up if needed
+            shutil.rmtree(os.path.join(separation_dir, model_name))
+            
+            return voice_path, music_path
+            
+        except Exception as e:
+            print(f"Error during audio separation: {e}")
+            
+            # Method 2: Fall back to Python API
+            try:
+                print("Attempting separation using Python API...")
+                import torch
+                from demucs.pretrained import get_model
+                from demucs.apply import apply_model
+                import torchaudio
+                
+                # Load audio
+                audio, sr = torchaudio.load(audio_path)
+                
+                # Convert to mono if needed
+                if audio.shape[0] > 1:
+                    audio = audio.mean(0, keepdim=True)
+                
+                # Load model
+                model = get_model('htdemucs')
+                model.to(torch.device("cuda" if torch.cuda.is_available() else "cpu"))
+                
+                # Apply separation
+                sources = apply_model(model, audio, device=torch.device("cuda" if torch.cuda.is_available() else "cpu"))
+                
+                # Sources is a dictionary with keys "vocals" and "no_vocals"
+                torchaudio.save(voice_path, sources[0].cpu(), sr)
+                torchaudio.save(music_path, sources[1].cpu(), sr)
+                
+                return voice_path, music_path
+            
+            except Exception as e2:
+                print(f"Python API separation also failed: {e2}")
+                return None, None

requirements.txt

@@ -0,0 +1,21 @@
+yt-dlp==2025.3.25
+moviepy==1.0.3
+pyannote.audio==3.3.2
+librosa==0.10.0
+soundfile==0.12.1
+scipy==1.11.4
+numpy==1.22.0
+pydub==0.25.1
+ffmpeg-python==0.2.0
+python-dotenv==1.1.0
+transformers==4.36.2
+deep-translator==1.11.4
+langchain-groq==0.3.1
+tqdm==4.66.1
+edge-tts==7.0.0
+openai==1.12.0
+gtts==2.5.4
+demucs==4.0.1
+TTS==0.22.0
+langchain==0.3.21
+logger==1.4

speech_diarization.py

@@ -0,0 +1,223 @@
+from pyannote.audio import Pipeline
+import torch
+import os
+import time
+
+class SpeakerDiarizer:
+    def __init__(self, hf_token, device=None):
+        """Initialize speaker diarization with HuggingFace token"""
+        self.diarization_pipeline = None
+        try:
+            print("Loading diarization pipeline...")
+            # Check available devices
+            if device is None:
+                device = "cuda:0" if torch.cuda.is_available() else "cpu"
+            print(f"Using device: {device}")
+            
+            # Use the newer version that's compatible with your libraries
+            self.diarization_pipeline = Pipeline.from_pretrained(
+                "pyannote/speaker-diarization-3.1",
+                use_auth_token=hf_token
+            )
+            self.diarization_pipeline.to(torch.device(device))
+            print("Diarization model loaded successfully!")
+        except Exception as e:
+            print(f"Error loading diarization model: {e}")
+    
+    def diarize(self, audio_path, min_speakers=1, max_speakers=None, device=None):
+        """Identify speakers in audio file"""
+        if not self.diarization_pipeline:
+            print("Diarization pipeline not available")
+            return []
+        
+        try:
+            print("Starting speaker diarization (this may take several minutes for longer files)...")
+            start_time = time.time()
+            
+            # Set parameters for diarization
+            params = {}
+            if min_speakers is not None:
+                params["min_speakers"] = min_speakers
+            if max_speakers is not None:
+                params["max_speakers"] = max_speakers
+            
+            # Set device if specified (cuda:0, cpu, etc.)
+            if device:
+                print(f"Using device: {device}")
+                self.diarization_pipeline.to(torch.device(device))
+            
+            # Add progress updates
+            print("Running diarization model...")
+            print("This process may take several minutes with no visible progress...")
+            print("Consider using a smaller audio segment for testing")
+            
+            # Use the diarization pipeline
+            diarization = self.diarization_pipeline(audio_path, **params)
+            
+            print("Processing diarization results...")
+            speakers = []
+            for turn, _, speaker in diarization.itertracks(yield_label=True):
+                speakers.append({'start': turn.start, 'end': turn.end, 'speaker': speaker})
+            
+            duration = time.time() - start_time
+            print(f"Diarization completed in {duration:.1f} seconds")
+            print(f"Detected {len(set(s['speaker'] for s in speakers))} unique speakers")
+            
+            return speakers
+        except Exception as e:
+            print(f"Error during diarization: {e}")
+            return []
+    
+    def assign_speakers_to_segments(self, segments, speakers):
+        """
+        Assign speaker labels to transcript segments based on timing overlap
+        
+        Args:
+            segments: List of transcript segments with start/end times
+            speakers: List of speaker segments from diarization
+            
+        Returns:
+            Updated segments with speaker information
+        """
+        # If no speakers found, assign everything to SPEAKER_0
+        if not speakers:
+            for segment in segments:
+                segment["speaker"] = "SPEAKER_0"
+            return segments
+        
+        # For single speaker, optimize by assigning all to same speaker
+        if len(set(s["speaker"] for s in speakers)) == 1:
+            speaker_id = speakers[0]["speaker"]
+            for segment in segments:
+                segment["speaker"] = speaker_id
+            return segments
+        
+        # Process each segment
+        for segment in segments:
+            segment_start = segment.get("start", 0)
+            segment_end = segment.get("end", 0)
+            segment_duration = segment_end - segment_start
+            
+            # Find overlapping speakers
+            speaker_overlaps = []
+            
+            for speaker_turn in speakers:
+                # Fast check for any overlap
+                if not (speaker_turn["end"] <= segment_start or speaker_turn["start"] >= segment_end):
+                    # Calculate overlap duration
+                    overlap_start = max(speaker_turn["start"], segment_start)
+                    overlap_end = min(speaker_turn["end"], segment_end)
+                    overlap_duration = overlap_end - overlap_start
+                    
+                    # Calculate overlap percentage relative to segment duration
+                    overlap_percentage = overlap_duration / segment_duration if segment_duration > 0 else 0
+                    
+                    speaker_overlaps.append((speaker_turn["speaker"], overlap_duration, overlap_percentage))
+            
+            # Assign speaker with the most overlap
+            if speaker_overlaps:
+                # Sort by overlap duration (descending)
+                speaker_overlaps.sort(key=lambda x: x[1], reverse=True)
+                segment["speaker"] = speaker_overlaps[0][0]
+                
+                # Add confidence score if desired
+                # segment["speaker_confidence"] = speaker_overlaps[0][2]
+            else:
+                # Find nearest speaker if no overlap
+                segment_mid = (segment_start + segment_end) / 2
+                
+                closest_speaker = min(
+                    speakers,
+                    key=lambda s: min(
+                        abs(s["start"] - segment_mid),
+                        abs(s["end"] - segment_mid)
+                    )
+                )
+                segment["speaker"] = closest_speaker["speaker"]
+                
+                # You can log this if logging is set up
+                # print(f"No speaker overlap found for segment at {segment_start:.2f}s, using nearest speaker")
+        
+        return segments
+    
+    def extract_speaker_references(self, audio_path, speakers, output_dir="reference_audio", min_duration=3.0, max_duration=10.0):
+        """
+        Extract reference audio clips for each unique speaker.
+        
+        Args:
+            audio_path: Path to the original audio file
+            speakers: List of speaker segments from diarization
+            output_dir: Directory to save reference audio clips
+            min_duration: Minimum duration for a reference clip (seconds)
+            max_duration: Maximum duration for a reference clip (seconds)
+            
+        Returns:
+            Dictionary mapping speaker IDs to reference audio file paths
+        """
+        import os
+        from pydub import AudioSegment
+        
+        # Ensure output directory exists
+        os.makedirs(output_dir, exist_ok=True)
+        
+        # Load the original audio file
+        try:
+            full_audio = AudioSegment.from_file(audio_path)
+        except Exception as e:
+            print(f"Error loading audio file: {e}")
+            return {}
+        
+        # Get unique speaker IDs
+        unique_speakers = set(segment["speaker"] for segment in speakers)
+        reference_files = {}
+        
+        print(f"Extracting reference audio for {len(unique_speakers)} speakers...")
+        
+        for speaker in unique_speakers:
+            # Find all segments for this speaker
+            speaker_segments = [s for s in speakers if s["speaker"] == speaker]
+            
+            # Sort segments by duration (descending)
+            speaker_segments.sort(key=lambda s: s["end"] - s["start"], reverse=True)
+            
+            # Find a segment with suitable duration
+            selected_segment = None
+            for segment in speaker_segments:
+                duration = segment["end"] - segment["start"]
+                if duration >= min_duration:
+                    # If longer than max_duration, trim it
+                    if duration > max_duration:
+                        mid_point = (segment["start"] + segment["end"]) / 2
+                        half_max = max_duration / 2
+                        segment = {
+                            "start": mid_point - half_max,
+                            "end": mid_point + half_max,
+                            "speaker": speaker
+                        }
+                    selected_segment = segment
+                    break
+            
+            # If no segment is long enough, take the longest one
+            if selected_segment is None and speaker_segments:
+                selected_segment = speaker_segments[0]
+                
+            # Extract the audio segment
+            if selected_segment:
+                start_ms = int(selected_segment["start"] * 1000)
+                end_ms = int(selected_segment["end"] * 1000)
+                
+                # Extract audio segment
+                speaker_audio = full_audio[start_ms:end_ms]
+                
+                # Save to file
+                speaker_id = speaker.replace("SPEAKER_", "")
+                output_path = os.path.join(output_dir, f"speaker_{speaker_id}_reference.wav")
+                speaker_audio.export(output_path, format="wav")
+                
+                reference_files[speaker] = output_path
+                
+                print(f"  Extracted {selected_segment['end'] - selected_segment['start']:.2f}s reference audio for {speaker}")
+            else:
+                print(f"  No suitable audio segment found for {speaker}")
+        
+        return reference_files

speech_recognition.py

@@ -0,0 +1,19 @@
+import whisper
+import os
+
+class SpeechRecognizer:
+    def __init__(self, model_size="base"):
+        self.model = whisper.load_model(model_size)
+        
+    def transcribe(self, audio_path, language="en"):
+        """Transcribe audio file with timestamps"""
+        # Get results with word timestamps
+        result = self.model.transcribe(
+            audio_path,
+            language=language,
+            word_timestamps=False,
+            verbose=False
+        )
+        
+        # Return segments with timestamps
+        return result["segments"]

text_to_speech.py

@@ -0,0 +1,619 @@
+import numpy as np
+import os
+import re
+import tempfile
+import logging
+import torch
+from pydub import AudioSegment
+from pathlib import Path
+import subprocess
+import librosa
+import soundfile as sf
+
+
+# Set up basic logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Create directory structure
+def ensure_directories():
+    """Ensure the required directories exist"""
+    directories = ["audio", "audio2", "reference_audio"]
+    for directory in directories:
+        os.makedirs(directory, exist_ok=True)
+    
+ensure_directories()  # Call immediately to ensure directories exist
+
+# Setup audio effects for pydub
+def setup_audio_effects():
+    """Setup custom audio effects"""
+    from pydub import effects
+    
+    # Add speedup if it's missing
+    if not hasattr(AudioSegment, "speedup"):
+        def speedup(audio_segment, playback_speed=1.5):
+            if playback_speed <= 0 or playback_speed == 1.0:
+                return audio_segment
+            new_frame_rate = int(audio_segment.frame_rate * playback_speed)
+            adjusted = audio_segment._spawn(audio_segment.raw_data, 
+                                          overrides={'frame_rate': new_frame_rate})
+            return adjusted.set_frame_rate(audio_segment.frame_rate)
+        AudioSegment.speedup = speedup
+    
+    # Add time_stretch if it's missing
+    if not hasattr(effects, "time_stretch"):
+        def time_stretch(audio_segment, stretch_factor):
+            if stretch_factor <= 0 or stretch_factor == 1.0:
+                return audio_segment
+            original_frame_rate = audio_segment.frame_rate
+            new_frame_rate = int(original_frame_rate / stretch_factor)
+            stretched = audio_segment._spawn(
+                audio_segment.raw_data,
+                overrides={'frame_rate': new_frame_rate}
+            )
+            return stretched.set_frame_rate(original_frame_rate)
+        effects.time_stretch = time_stretch
+    
+    return effects
+
+effects = setup_audio_effects()
+
+def adjust_audio_duration(audio_segment, target_duration):
+    """Adjust audio to target duration by adding silence or trimming"""
+    current_duration = len(audio_segment) / 1000  # ms to seconds
+    
+    if current_duration < target_duration:
+        silence_duration_ms = int((target_duration - current_duration) * 1000)
+        silence = AudioSegment.silent(duration=silence_duration_ms)
+        return audio_segment + silence
+    else:
+        return audio_segment[:int(target_duration * 1000)]
+
+# XTTS Model Loader (Singleton pattern)
+class XTTSModelLoader:
+    _instance = None
+    model = None
+    
+    @classmethod
+    def get_model(cls):
+        """Get or initialize the XTTS model"""
+        if cls.model is None:
+            try:
+                from TTS.api import TTS
+                
+                # Determine device
+                device = "cuda" if torch.cuda.is_available() else "cpu"
+                logger.info(f"Loading XTTS model on {device}...")
+                
+                # Load the model
+                cls.model = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to(device)
+                logger.info("XTTS model loaded successfully")
+            except Exception as e:
+                logger.error(f"Error loading XTTS model: {e}")
+                return None
+                
+        return cls.model
+
+def smooth_speed_change(audio_path, target_duration):
+    """
+    Adjust audio speed with instantaneous time stretching to match target duration
+    
+    Args:
+        audio_path: Path to audio file to adjust
+        target_duration: Target duration in seconds
+        
+    Returns:
+        Path to adjusted audio file (temporary file)
+    """
+    try:
+        # Debug start
+        print(f"\n[DEBUG] Starting audio speed adjustment:")
+        print(f"[DEBUG] Input file: {audio_path}")
+        print(f"[DEBUG] Target duration: {target_duration:.2f}s")
+        
+        # Load audio with librosa
+        y, sr = librosa.load(audio_path, sr=None)
+        
+        # Calculate current duration and speed factor
+        current_duration = librosa.get_duration(y=y, sr=sr)
+        speed_factor = current_duration / target_duration
+        
+        print(f"[DEBUG] Current duration: {current_duration:.2f}s")
+        print(f"[DEBUG] Calculated speed factor: {speed_factor:.3f}")
+        
+        # If the difference is minimal, return original path
+        if abs(speed_factor - 1) < 0.05:
+            print(f"[DEBUG] Speed factor {speed_factor:.3f} is within 5% threshold, skipping adjustment")
+            return audio_path
+        
+        # Dynamic speed factor limits based on audio duration
+        # Allow more aggressive speed factors for short audio
+        if current_duration < 10.0:  # Short audio under 10 seconds
+            max_speed = 3.0  # More aggressive for short segments
+        else:
+            max_speed = 2.7  # Standard limit for longer audio
+            
+        min_speed = 0.5  # Allow more slowdown when needed
+        
+        # Check if extreme speed change is needed
+        extreme_adjustment = (speed_factor > max_speed)
+        
+        # Limit speed factor to reasonable range
+        original_speed_factor = speed_factor
+        speed_factor = min(max(speed_factor, min_speed), max_speed)
+        
+        if original_speed_factor != speed_factor:
+            print(f"[DEBUG] Speed factor clamped from {original_speed_factor:.3f} to {speed_factor:.3f}")
+            if extreme_adjustment:
+                print(f"[DEBUG] Extreme adjustment needed - will apply max speed and then trim")
+        
+        # Track processing time
+        import time
+        start_time = time.time()
+        
+        # SIMPLIFIED: Apply direct time stretching to the entire audio at once
+        print(f"[DEBUG] Applying instantaneous time stretching with factor {speed_factor:.3f}")
+        stretched_audio = librosa.effects.time_stretch(y=y, rate=speed_factor)
+        
+        # Calculate new duration
+        expected_duration = len(stretched_audio) / sr
+        
+        # Save to temporary file
+        temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+        sf.write(temp_file.name, stretched_audio, sr)
+        
+        # Calculate processing time
+        process_time = time.time() - start_time
+        
+        # Verify the actual duration after processing
+        y_check, sr_check = librosa.load(temp_file.name, sr=None)
+        actual_duration = librosa.get_duration(y=y_check, sr=sr_check)
+        
+        method = "direct"
+        
+        # For extreme cases, perform additional trimming
+        if extreme_adjustment and actual_duration > target_duration:
+            print(f"[DEBUG] Performing additional trim for extreme case")
+            # Calculate how many samples to keep
+            samples_to_keep = int(target_duration * sr_check)
+            
+            # Apply a small fade out to avoid clicks
+            fade_samples = min(int(0.1 * sr_check), samples_to_keep // 4)  # 100ms fade or less
+            
+            # Keep only the needed samples
+            trimmed_audio = y_check[:samples_to_keep]
+            
+            # Apply fade out to avoid clicks
+            if fade_samples > 0:
+                fade_env = np.linspace(1.0, 0.0, fade_samples)
+                trimmed_audio[-fade_samples:] *= fade_env
+            
+            # Save the trimmed version
+            sf.write(temp_file.name, trimmed_audio, sr_check)
+            
+            # Update actual duration
+            actual_duration = librosa.get_duration(y=trimmed_audio, sr=sr_check)
+            method += "+trim"
+        
+        print(f"[DEBUG] Method used: {method}")
+        print(f"[DEBUG] Processing completed in {process_time:.2f} seconds")
+        print(f"[DEBUG] Expected new duration: {expected_duration:.2f}s")
+        print(f"[DEBUG] Actual new duration: {actual_duration:.2f}s")
+        print(f"[DEBUG] Target was: {target_duration:.2f}s")
+        print(f"[DEBUG] Difference from target: {abs(actual_duration - target_duration):.3f}s")
+        print(f"[DEBUG] Output file: {temp_file.name}")
+        
+        return temp_file.name
+        
+    except Exception as e:
+        import traceback
+        print(f"[DEBUG ERROR] Audio speed adjustment failed: {e}")
+        print(traceback.format_exc())
+        logger.warning(f"Audio speed adjustment failed: {e}")
+        return audio_path
+
+def create_segmented_edge_tts(text, pitch, voice, output_path, target_duration=None):
+    """Create voice clone with specific characteristics and timing using Edge TTS"""
+    # Create a temporary file
+    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.mp3')
+    temp_filename = temp_file.name  # Store filename before closing
+    temp_file.close()
+    
+    # Fix pitch formatting
+    pitch_param = f"+{pitch}Hz" if pitch >= 0 else f"{pitch}Hz"
+    
+    command = [
+        "edge-tts",
+        f"--pitch={pitch_param}",
+        "--voice", voice,
+        "--text", text,
+        "--write-media", temp_filename
+    ]
+    subprocess.run(command, check=True)
+    # Load audio
+    audio = AudioSegment.from_file(temp_filename, format="mp3")
+    
+    # Time constraint adjustment
+    if target_duration is not None:
+        current_duration = len(audio) / 1000  # ms to seconds
+        
+        if abs(current_duration - target_duration) > 0.1:  # 100ms threshold
+            speed_factor = current_duration / target_duration
+            speed_factor = min(max(speed_factor, 0.7), 3)  # Keep within bounds
+            
+            logger.info(f"  Adjusting timing: {current_duration:.2f}s → {target_duration:.2f}s (factor: {speed_factor:.2f})")
+            
+            # Apply time adjustment
+            # Instead of speed adjustments after generation, use Edge TTS rate parameter
+            if speed_factor < 1:
+                rate_adjustment = f"-{int((1 - speed_factor) * 100)}%"
+            else:
+                rate_adjustment = f"+{int((speed_factor - 1) * 100)}%"
+            
+            # Regenerate with adjusted rate
+            os.unlink(temp_file.name)  # Remove the previous temp file
+            
+            # Create new command with rate parameter and fixed pitch formatting
+            command = [
+                "edge-tts",
+                f"--pitch={pitch_param}",
+                f"--rate={rate_adjustment}",
+                "--voice", voice,
+                "--text", text,
+                "--write-media", temp_filename
+            ]
+            subprocess.run(command, check=True)
+            
+            # Reload audio with rate adjustment
+            audio = AudioSegment.from_file(temp_filename, format="mp3")
+            
+            # Fine-tune if needed
+            new_duration = len(audio) / 1000
+            if abs(new_duration - target_duration) > 0.1:
+                audio = adjust_audio_duration(audio, target_duration)
+    
+    # Save the modified audio
+    audio.export(output_path, format="wav")
+    
+    # Clean up temporary file
+    os.unlink(temp_file.name)
+    
+    # Log final duration
+    final_audio = AudioSegment.from_file(output_path)
+    final_duration = len(final_audio) / 1000
+    logger.info(f"  Final duration: {final_duration:.2f}s (target: {target_duration if target_duration else 'None'}s)")
+    
+    return output_path
+
+def create_segmented_xtts(text, reference_audio, language, output_path, target_duration=None):
+    """Create voice-cloned speech using XTTS with speaker's reference audio and duration control"""
+    # Get the model (will be loaded on first call)
+    tts_model = XTTSModelLoader.get_model()
+    
+    if tts_model is None:
+        raise RuntimeError("XTTS model could not be loaded. Ensure TTS is installed.")
+    
+    # Verify reference audio exists
+    if not os.path.exists(reference_audio):
+        raise FileNotFoundError(f"Reference audio file not found: {reference_audio}")
+    
+    # Generate speech
+    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+    temp_filename = temp_file.name
+    temp_file.close()
+    
+    logger.info(f"Generating XTTS speech using reference: {os.path.basename(reference_audio)}")
+
+    # Step 1: Try to optimize the generation parameters based on text length and target duration
+    # Short text might need special handling to avoid excessive padding
+    is_short_text = len(text.strip()) < 10
+    
+    # XTTS generation options
+    generation_kwargs = {}
+    
+    # Add text length information for very short text to help the model
+    # Note: These are example parameters - actual parameter support depends on the XTTS version
+    if is_short_text and target_duration is not None and target_duration < 2.0:
+        logger.info(f"  Short text detected, attempting to minimize padding")
+        # These parameters may or may not be supported by the TTS model being used
+        generation_kwargs = {
+            'enable_text_splitting': False,  # Avoid splitting short text
+            'no_silence_end': True,          # Reduce trailing silence 
+        }
+        # Some models may support 'speed' parameter
+        if hasattr(tts_model, 'tts_with_speed'):
+            generation_kwargs['speed'] = 1.2  # Slightly faster for short text
+    
+    try:
+        # Try generating with optional parameters if supported
+        if generation_kwargs:
+            try:
+                tts_model.tts_to_file(
+                    text=text,
+                    speaker_wav=reference_audio,
+                    language=language,
+                    file_path=temp_filename,
+                    **generation_kwargs
+                )
+            except (TypeError, ValueError):
+                # If parameters aren't supported, fall back to standard call
+                logger.info("  Advanced parameters not supported, using standard generation")
+                tts_model.tts_to_file(
+                    text=text,
+                    speaker_wav=reference_audio,
+                    language=language,
+                    file_path=temp_filename
+                )
+        else:
+            # Standard generation
+            tts_model.tts_to_file(
+                text=text,
+                speaker_wav=reference_audio,
+                language=language,
+                file_path=temp_filename
+            )
+        
+        # Load generated audio
+        audio = AudioSegment.from_file(temp_filename)
+        
+        # Step 2: Apply duration adjustment if needed
+        if target_duration is not None:
+            current_duration = len(audio) / 1000  # ms to seconds
+            
+            if abs(current_duration - target_duration) > 0.1:  # 100ms threshold
+                # Calculate speed factor - inverse of duration ratio
+                speed_factor = current_duration / target_duration
+                speed_factor = min(max(speed_factor, 0.7), 3)  # Allow wider range for better adjustment
+                
+                logger.info(f"  Adjusting timing: {current_duration:.2f}s → {target_duration:.2f}s (speed factor: {speed_factor:.2f})")
+                
+                try:
+                    # Always attempt smooth speed change since regeneration doesn't work
+                    logger.info("  Applying smooth speed adjustment...")
+                    adjusted_path = smooth_speed_change(temp_filename, target_duration)
+                    
+                    if adjusted_path != temp_filename:  # If path is different, adjustment was done
+                        # Load the adjusted audio
+                        audio = AudioSegment.from_file(adjusted_path)
+                        
+                        # Check if adjustment was successful
+                        new_duration = len(audio) / 1000
+                        if abs(new_duration - target_duration) <= 0.15:  # 150ms tolerance
+                            logger.info(f"  Smooth adjustment successful: {new_duration:.2f}s")
+                            
+                            # Clean up original file and use the adjusted one
+                            os.unlink(temp_filename)
+                            temp_filename = adjusted_path
+                        else:
+                            # Clean up adjusted file and just use duration adjustment
+                            logger.info(f"  Smooth adjustment not precise enough ({new_duration:.2f}s), will fine-tune with duration adjustment")
+                            os.unlink(adjusted_path)
+                            # We'll fall through to the final duration adjustment step
+                except Exception as e:
+                    logger.warning(f"  Smooth speed adjustment failed: {str(e)}")
+                    # We'll fall through to the final duration adjustment step
+                
+                # Always perform final duration adjustment to ensure exact timing
+                new_duration = len(audio) / 1000
+                if abs(new_duration - target_duration) > 0.1:
+                    logger.info(f"  Fine-tuning with duration adjustment: {new_duration:.2f}s → {target_duration:.2f}s")
+                    audio = adjust_audio_duration(audio, target_duration)
+        
+        # Save the final audio
+        audio.export(output_path, format="wav")
+        
+        # Clean up
+        os.unlink(temp_filename)
+        
+        # Log final duration
+        final_audio = AudioSegment.from_file(output_path)
+        final_duration = len(final_audio) / 1000
+        logger.info(f"  Final duration: {final_duration:.2f}s (target: {target_duration if target_duration else 'None'}s)")
+        
+        return output_path
+        
+    except Exception as e:
+        logger.error(f"XTTS generation failed: {e}")
+        if os.path.exists(temp_filename):
+            os.unlink(temp_filename)
+        raise
+
+def process_voice_config(voice_config):
+    """
+    Process voice configuration to support both Edge TTS and XTTS
+    
+    Args:
+        voice_config: Dict with speaker_id keys and configuration values
+            For Edge TTS: {'engine': 'edge_tts', 'gender': 'male'/'female'} or simply 'male'/'female'
+            For XTTS: {'engine': 'xtts', 'reference_audio': '/path/to/audio.wav', 'language': 'hi'}
+    
+    Returns:
+        Processed configuration dictionary
+    """
+    processed_config = {}
+    
+    # Handle empty config
+    if not voice_config:
+        return {0: {'engine': 'edge_tts', 'voice': "hi-IN-MadhurNeural", 'pitch': 0}}
+    
+    # Track Edge TTS speaker counts for pitch variations
+    edge_male_count = 0
+    edge_female_count = 0
+    
+    # Pitch variations for multiple Edge TTS speakers of same gender
+    male_pitches = [0, -30, 40]  # Default, deeper, higher
+    female_pitches = [0, 25, -25]  # Default, higher, deeper
+    
+    for speaker_id, config in voice_config.items():
+        # Convert string speaker_id to int if needed
+        if isinstance(speaker_id, str) and speaker_id.isdigit():
+            speaker_id = int(speaker_id)
+        
+        # Determine which engine to use (default is edge_tts)
+        if isinstance(config, dict):
+            engine = config.get('engine', 'edge_tts')
+        else:
+            # Handle simple gender strings for backwards compatibility
+            engine = 'edge_tts'
+            config = {'gender': config} if config in ['male', 'female'] else {'gender': 'male'}
+        
+        if engine == 'xtts':
+            # XTTS configuration - each speaker needs their own reference audio
+            if 'reference_audio' not in config:
+                logger.warning(f"No reference audio provided for XTTS speaker {speaker_id}, falling back to Edge TTS")
+                # Fall back to Edge TTS if no reference audio
+                engine = 'edge_tts'
+                gender = config.get('gender', 'male')
+            else:
+                # Valid XTTS configuration
+                processed_config[speaker_id] = {
+                    'engine': 'xtts',
+                    'reference_audio': config['reference_audio'],
+                    'language': config.get('language', 'hi')  # Default to Hindi
+                }
+                continue  # Skip the Edge TTS processing below
+        
+        # Edge TTS configuration (if engine is edge_tts or XTTS fallback)
+        gender = config.get('gender', 'male')
+        
+        if gender == 'male':
+            # Assign male voice and pitch
+            pitch = male_pitches[edge_male_count % len(male_pitches)]
+            processed_config[speaker_id] = {
+                'engine': 'edge_tts',
+                'voice': "hi-IN-MadhurNeural",
+                'pitch': pitch
+            }
+            edge_male_count += 1
+        else:
+            # Assign female voice and pitch
+            pitch = female_pitches[edge_female_count % len(female_pitches)]
+            processed_config[speaker_id] = {
+                'engine': 'edge_tts',
+                'voice': "hi-IN-SwaraNeural", 
+                'pitch': pitch
+            }
+            edge_female_count += 1
+    
+    return processed_config
+
+def generate_tts(segments, target_language, voice_config=None, output_dir="audio2"):
+    """
+    Generate speech for all segments using appropriate TTS engine per speaker
+    
+    Args:
+        segments: List of segments with text, speaker, start and end times
+        target_language: Language code for TTS
+        voice_config: Dictionary with speaker configurations
+                     - For Edge TTS: {'gender': 'male'/'female'} or just 'male'/'female'
+                     - For XTTS: {'engine': 'xtts', 'reference_audio': '/path/to/audio.wav'}
+        output_dir: Directory to save the final audio
+        
+    Returns:
+        Path to the final combined audio file
+    """
+    # Ensure output directory exists
+    os.makedirs(output_dir, exist_ok=True)
+    
+    # Generate the full audio
+    output_path = os.path.join(output_dir, "dubbed_conversation.wav")
+    max_end_time = max(segment['end'] for segment in segments)
+    
+    # Create a silent audio of the total duration
+    combined = AudioSegment.silent(duration=int(max_end_time * 1000) + 100) 
+    ensure_directories()
+    audio_files = []
+    
+    # Process voice configuration
+    processed_config = process_voice_config(voice_config or {})
+    print(processed_config)
+    
+    # Process each segment
+    for i, segment in enumerate(segments):
+        # Extract speaker ID
+        speaker = segment.get('speaker', 'SPEAKER_00')
+        match = re.search(r'SPEAKER_(\d+)', speaker)
+        speaker_id = int(match.group(1)) if match else 0
+        
+        # Get speaker configuration
+        speaker_config = processed_config.get(speaker_id, 
+                                             {'engine': 'edge_tts', 'voice': "hi-IN-SwaraNeural", 'pitch': 0})
+        
+        # Get text and timing information
+        text = segment['text']
+        start = segment['start']
+        end = segment['end']
+        duration = end - start
+        
+        # Create output filename
+        output_file = f"audio/{start}.wav"
+        
+        logger.info(f"Processing segment {i+1} (Speaker {speaker_id}, Engine: {speaker_config['engine']}):")
+        logger.info(f"  Text: {text[:50]}{'...' if len(text) > 50 else ''}")
+        logger.info(f"  Duration: {duration:.2f}s")
+        
+        # Choose appropriate TTS engine
+        if speaker_config['engine'] == 'xtts':
+            # XTTS generation with speaker's reference audio
+            try:
+                create_segmented_xtts(
+                    text=text,
+                    reference_audio=speaker_config['reference_audio'],
+                    language=speaker_config.get('language', target_language),
+                    output_path=output_file,
+                    target_duration=duration,
+                )
+            except Exception as e:
+                logger.error(f"Error using XTTS for speaker {speaker_id}: {e}")
+                logger.warning(f"Falling back to Edge TTS for this segment")
+                # Fallback to Edge TTS
+                create_segmented_edge_tts(
+                    text=text,
+                    pitch=0, 
+                    voice="hi-IN-SwaraNeural",
+                    output_path=output_file,
+                    target_duration=duration,
+                )
+        else:
+            # Edge TTS generation
+            create_segmented_edge_tts(
+                text=text,
+                pitch=speaker_config.get('pitch', 0),
+                voice=speaker_config.get('voice', "hi-IN-SwaraNeural"),
+                output_path=output_file,
+                target_duration=duration,
+            )
+        
+        audio_files.append(output_file)
+
+        # Add segment to combined audio at the exact timestamp
+        segment_audio = AudioSegment.from_file(output_file)
+        position_ms = int(segment['start'] * 1000)
+        combined = combined.overlay(segment_audio, position=position_ms)
+    
+    # Export the final combined audio
+    combined.export(output_path, format="wav")
+    logger.info(f"  Final combined duration: {len(combined) / 1000:.2f}s")
+    
+    # Clean up segment files
+    for file in audio_files:
+        try:
+            os.remove(file)
+        except:
+            pass
+    
+    # Verify the final duration
+    final_audio = AudioSegment.from_file(output_path)
+    final_duration_sec = len(final_audio) / 1000
+    
+    print(f"\nTarget duration: {max_end_time:.2f} seconds")
+    print(f"Actual duration: {final_duration_sec:.2f} seconds")
+    
+    # If the final audio is still too long, trim it
+    if final_duration_sec > max_end_time + 0.1:  # Allow 100ms grace
+        trimmed = final_audio[:int(max_end_time * 1000)]
+        trimmed.export(output_path, format="wav")
+        print(f"Trimmed to exactly {max_end_time:.2f} seconds")
+    
+    return output_path
+

translate.py

@@ -0,0 +1,391 @@
+import json
+import copy
+import logging
+import time
+from typing import List, Dict, Optional, Any, Union
+from itertools import chain
+from tqdm import tqdm
+from deep_translator import GoogleTranslator
+from dotenv import load_dotenv
+
+# Add this at the top of your translate.py file
+# Load environment variables from .env file
+load_dotenv()
+
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+# Import Groq
+from langchain_groq import ChatGroq
+from langchain_core.prompts import ChatPromptTemplate
+from langchain.chains import LLMChain
+
+# Language code constants - simplified for now
+ISO_LANGUAGE_CODES = {
+    "en": "english",
+    "es": "spanish",
+    "fr": "french",
+    "de": "german",
+    "it": "italian",
+    "pt": "portuguese",
+    "ru": "russian",
+    "zh": "chinese",
+    "ja": "japanese",
+    "ko": "korean",
+    "hi": "hindi",
+    "ar": "arabic",
+}
+
+def fix_language_code(language_code: Optional[str]) -> str:
+    """Convert language code to format compatible with translator."""
+    if not language_code:
+        return "auto"
+    
+    # Clean up language code (remove region specifiers)
+    language_code = language_code.lower().split('-')[0]
+    
+    # Return the cleaned code if it's in our list, otherwise default to auto
+    return language_code if language_code in ISO_LANGUAGE_CODES else "auto"
+
+def translate_iterative(segments: List[Dict[str, Any]], 
+                        target_lang: str, 
+                        source_lang: Optional[str] = None) -> List[Dict[str, Any]]:
+    """
+    Translate text segments individually to the specified language.
+
+    Args:
+        segments: List of dictionaries with 'text' key containing the text to translate
+        target_lang: Target language code
+        source_lang: Source language code (defaults to auto-detect)
+
+    Returns:
+        List of segments with translated text
+    """
+    segments_copy = copy.deepcopy(segments)
+    source = fix_language_code(source_lang)
+    target = fix_language_code(target_lang)
+    
+    logger.info(f"Translating {len(segments)} segments from {source} to {target} (iterative)")
+    translator = GoogleTranslator(source=source, target=target)
+
+    for i, segment in enumerate(tqdm(segments_copy, desc="Translating")):
+        text = segment["text"].strip()
+        try:
+            translated_text = translator.translate(text)
+            segments_copy[i]["text"] = translated_text
+        except Exception as error:
+            logger.error(f"Error translating segment {i}: {error}")
+            # Keep original text if translation fails
+            segments_copy[i]["text"] = text
+
+    return segments_copy
+
+def verify_translation(original_segments: List[Dict[str, Any]],
+                      segments_copy: List[Dict[str, Any]],
+                      translated_lines: List[str],
+                      target_lang: str,
+                      source_lang: Optional[str] = None) -> List[Dict[str, Any]]:
+    """
+    Verify translation integrity and assign translated text to segments.
+    Falls back to iterative translation if segment counts don't match.
+    """
+    if len(original_segments) == len(translated_lines):
+        for i in range(len(segments_copy)):
+            segments_copy[i]["text"] = translated_lines[i].replace("\t", " ").replace("\n", " ").strip()
+        return segments_copy
+    else:
+        logger.error(
+            f"Translation failed: segment count mismatch. Original: {len(original_segments)}, "
+            f"Translated: {len(translated_lines)}. Switching to iterative translation."
+        )
+        return translate_iterative(original_segments, target_lang, source_lang)
+
+def translate_batch(segments: List[Dict[str, Any]], 
+                   target_lang: str, 
+                   chunk_size: int = 4000,
+                   source_lang: Optional[str] = None) -> List[Dict[str, Any]]:
+    """
+    Translate a batch of text segments in chunks to respect API limits.
+
+    Args:
+        segments: List of dictionaries with 'text' key
+        target_lang: Target language code
+        chunk_size: Maximum character count per chunk (default: 4000)
+        source_lang: Source language code (defaults to auto-detect)
+
+    Returns:
+        List of segments with translated text
+    """
+    segments_copy = copy.deepcopy(segments)
+    source = fix_language_code(source_lang)
+    target = fix_language_code(target_lang)
+    
+    logger.info(f"Translating {len(segments)} segments from {source} to {target} (batch)")
+
+    # Extract text from segments
+    text_lines = [segment["text"].strip() for segment in segments]
+    
+    # Create chunks respecting character limit
+    text_chunks = []
+    current_chunk = ""
+    chunk_segments = []
+    segment_tracking = []
+    
+    for line in text_lines:
+        line = " " if not line else line
+        if (len(current_chunk) + len(line) + 7) <= chunk_size:  # 7 for separator
+            if current_chunk:
+                current_chunk += " ||||| "
+            current_chunk += line
+            chunk_segments.append(line)
+        else:
+            text_chunks.append(current_chunk)
+            segment_tracking.append(chunk_segments)
+            current_chunk = line
+            chunk_segments = [line]
+    
+    if current_chunk:
+        text_chunks.append(current_chunk)
+        segment_tracking.append(chunk_segments)
+
+    # Translate chunks
+    translator = GoogleTranslator(source=source, target=target)
+    translated_segments = []
+    progress_bar = tqdm(total=len(segments), desc="Translating")
+    
+    try:
+        for chunk_text, chunk_segments in zip(text_chunks, segment_tracking):
+            translated_chunk = translator.translate(chunk_text.strip())
+            split_translations = translated_chunk.split("|||||")
+            
+            # Verify chunk integrity
+            if len(split_translations) == len(chunk_segments):
+                progress_bar.update(len(split_translations))
+                translated_segments.extend([t.strip() for t in split_translations])
+            else:
+                logger.warning(
+                    f"Chunk translation mismatch. Expected {len(chunk_segments)}, "
+                    f"got {len(split_translations)}. Translating segment by segment."
+                )
+                for segment in chunk_segments:
+                    translated_text = translator.translate(segment.strip())
+                    translated_segments.append(translated_text.strip())
+                    progress_bar.update(1)
+        
+        progress_bar.close()
+        
+        # Verify and return
+        return verify_translation(segments, segments_copy, translated_segments, target_lang, source_lang)
+    
+    except Exception as error:
+        progress_bar.close()
+        logger.error(f"Batch translation failed: {error}")
+        return translate_iterative(segments, target_lang, source_lang)
+
+def translate_with_groq(segments: List[Dict[str, Any]],
+                       target_lang: str,
+                       model_name: str = "llama-3.3-70b-versatile",
+                       source_lang: Optional[str] = None,
+                       batch_size: int = 10) -> List[Dict[str, Any]]:
+    """
+    Translate text segments using Groq API.
+    
+    Args:
+        segments: List of dictionaries with 'text' key
+        target_lang: Target language code
+        model_name: Groq model to use (default: "llama-3.3-70b-versatile")
+        source_lang: Source language code (optional)
+        batch_size: Number of segments to process in each API call
+        
+    Returns:
+        List of segments with translated text
+    """
+    segments_copy = copy.deepcopy(segments)
+    
+    # Get language names instead of codes for clarity in prompting
+    target_language = ISO_LANGUAGE_CODES.get(fix_language_code(target_lang), "the target language")
+    source_language = "auto-detected language"
+    if source_lang:
+        source_language = ISO_LANGUAGE_CODES.get(fix_language_code(source_lang), "the source language")
+    
+    logger.info(f"Translating {len(segments)} segments from {source_language} to {target_language} using Groq")
+    
+    # Set up Groq LLM
+    llm = ChatGroq(model_name=model_name, temperature=0.2)
+    
+    # Process segments in batches
+    translated_segments = []
+    total_batches = (len(segments) + batch_size - 1) // batch_size
+    
+    for batch_idx in tqdm(range(total_batches), desc="Translating batches"):
+        start_idx = batch_idx * batch_size
+        end_idx = min(start_idx + batch_size, len(segments))
+        batch = segments[start_idx:end_idx]
+        
+        # Extract text from segments
+        batch_texts = [segment["text"].strip() for segment in batch]
+        
+        # Create numbered text array for the prompt
+        numbered_texts = [f"{i+1}. {text}" for i, text in enumerate(batch_texts)]
+        batch_content = "\n".join(numbered_texts)
+        
+        # Create a prompt template for translation
+        template = """
+        You are a professional translator. Translate the following text segments from {source_language} to {target_language}.
+        
+        IMPORTANT INSTRUCTIONS:
+        1. Preserve the meaning, tone, and style of the original text
+        2. Only respond with JSON in the exact format shown below
+        3. Each numbered segment should be translated separately
+        4. Maintain the original numbering in your response
+        5. Translated Segments should be short and concise
+        6. the translated segment should be of similar size as input.
+        
+        Text to translate:
+        {text_segments}
+        
+        The response should be ONLY a JSON array with this exact structure:
+        [
+          "translated segment 1",
+          "translated segment 2",
+          ...
+        ]
+        """
+        
+        prompt = ChatPromptTemplate.from_messages([("system", template)])
+        
+        try:
+            # Create a chain and execute
+            chain = LLMChain(llm=llm, prompt=prompt)
+            response = chain.run(
+                source_language=source_language,
+                target_language=target_language,
+                text_segments=batch_content
+            )
+            
+            # Parse the response
+            # First try to find JSON in the response using regex
+            import re
+            json_match = re.search(r'\[.*\]', response.strip(), re.DOTALL)
+            
+            if json_match:
+                try:
+                    translated_texts = json.loads(json_match.group(0))
+                except:
+                    # If regex json extraction fails, try direct parsing
+                    translated_texts = json.loads(response.strip())
+            else:
+                # If no JSON array found, try to parse directly
+                translated_texts = json.loads(response.strip())
+            
+            # Verify correct count
+            if len(translated_texts) != len(batch):
+                logger.warning(
+                    f"Translation count mismatch. Expected {len(batch)}, "
+                    f"got {len(translated_texts)}. Falling back to Google Translate for this batch."
+                )
+                # Fall back to Google for this batch
+                fallback_translations = translate_iterative(batch, target_lang, source_lang)
+                translated_texts = [segment["text"] for segment in fallback_translations]
+            
+            # Add translations to the result
+            translated_segments.extend(translated_texts)
+            
+            # Avoid hitting rate limits
+            time.sleep(0.5)
+            
+        except Exception as error:
+            logger.error(f"Groq translation error for batch {batch_idx+1}/{total_batches}: {error}")
+            logger.warning("Falling back to Google Translate for this batch")
+            
+            # Fall back to Google for this batch
+            fallback_translations = translate_iterative(batch, target_lang, source_lang)
+            batch_translations = [segment["text"] for segment in fallback_translations]
+            translated_segments.extend(batch_translations)
+    
+    # Verify and update segments
+    return verify_translation(segments, segments_copy, translated_segments, target_lang, source_lang)
+
+def translate_text(segments: List[Dict[str, Any]],
+                  target_lang: str,
+                  translation_method: str = "batch",
+                  chunk_size: int = 4000,
+                  source_lang: Optional[str] = None,
+                  groq_model: str = "llama-3.3-70b-versatile",
+                  groq_batch_size: int = 10) -> List[Dict[str, Any]]:
+    """
+    Main translation function that handles different translation methods.
+    
+    Args:
+        segments: List of dictionaries with 'text' key
+        target_lang: Target language code
+        translation_method: "batch", "iterative", or "groq" (default: "batch")
+        chunk_size: Maximum character count per chunk for batch translation
+        source_lang: Source language code (defaults to auto-detect)
+        groq_model: Model name for Groq translation
+        groq_batch_size: Batch size for Groq translation
+        
+    Returns:
+        List of segments with translated text
+    """
+    if not segments:
+        logger.warning("No segments to translate")
+        return segments
+    
+    if translation_method == "batch":
+        return translate_batch(segments, target_lang, chunk_size, source_lang)
+    elif translation_method == "iterative":
+        return translate_iterative(segments, target_lang, source_lang)
+    elif translation_method == "groq":
+        return translate_with_groq(
+            segments, 
+            target_lang, 
+            model_name=groq_model,
+            source_lang=source_lang,
+            batch_size=groq_batch_size
+        )
+    else:
+        logger.error(f"Unknown translation method: {translation_method}")
+        return translate_batch(segments, target_lang, chunk_size, source_lang)
+    
+def generate_srt_subtitles(segments, output_file="output.srt"):
+    """
+    Generate an SRT subtitle file from translated segments.
+    
+    Args:
+        segments: List of dictionaries with 'start', 'end', and 'text' keys
+        output_file: Path to the output SRT file
+        
+    Returns:
+        Path to the created SRT file
+    """
+    logger.info(f"Generating SRT subtitle file: {output_file}")
+    
+    # Format time as HH:MM:SS,mmm
+    def format_time(seconds):
+        hours = int(seconds // 3600)
+        minutes = int((seconds % 3600) // 60)
+        seconds = seconds % 60
+        milliseconds = int((seconds - int(seconds)) * 1000)
+        return f"{hours:02d}:{minutes:02d}:{int(seconds):02d},{milliseconds:03d}"
+    
+    with open(output_file, "w", encoding="utf-8") as f:
+        for i, segment in enumerate(segments, 1):
+            # Extract timing information
+            start_time = segment.get("start", 0)
+            end_time = segment.get("end", 0)
+            text = segment.get("text", "").strip()
+            
+            # Skip empty segments
+            if not text:
+                continue
+                
+            # Write subtitle entry
+            f.write(f"{i}\n")
+            f.write(f"{format_time(start_time)} --> {format_time(end_time)}\n")
+            f.write(f"{text}\n\n")
+    
+    logger.info(f"SRT subtitle file created successfully: {output_file}")
+    return output_file