fix

AI_Transformers_Audio_Processing_Guide.md

@@ -0,0 +1,431 @@
+# 🎤 Complete Guide to AI Transformers in Audio Processing
+
+## Table of Contents
+1. [Introduction](#introduction)
+2. [Transformer Architecture Fundamentals](#transformer-architecture-fundamentals)
+3. [Audio Transformers: From Sound Waves to Text](#audio-transformers-from-sound-waves-to-text)
+4. [Model Architectures Implementation](#model-architectures-implementation)
+5. [Audio Processing Pipeline](#audio-processing-pipeline)
+6. [Technical Implementation Deep Dive](#technical-implementation-deep-dive)
+7. [Performance Optimization](#performance-optimization)
+8. [Model Comparison and Benchmarks](#model-comparison-and-benchmarks)
+9. [Code Examples and Usage Patterns](#code-examples-and-usage-patterns)
+10. [Best Practices and Production Deployment](#best-practices-and-production-deployment)
+
+---
+
+## Introduction
+
+This comprehensive guide explores the application of AI transformer models to audio processing, specifically focusing on speech-to-text systems for Indian languages. The project demonstrates practical implementation of multiple transformer architectures including Whisper, Wav2Vec2, SeamlessM4T, and SpeechT5.
+
+### Project Overview
+- **Multi-model speech-to-text application** supporting 13 Indian languages
+- **Transformer architectures**: Whisper, Wav2Vec2, SeamlessM4T, SpeechT5
+- **Technology stack**: PyTorch, TensorFlow, Transformers library, Gradio UI
+- **Processing modes**: Real-time and batch processing
+- **Commercial license**: All models free for commercial use
+
+---
+
+## Transformer Architecture Fundamentals
+
+### What are Transformers?
+
+Transformers are a revolutionary neural network architecture introduced in the "Attention Is All You Need" paper (2017). They've transformed not just NLP, but also audio processing, computer vision, and more.
+
+#### Key Components
+
+1. **Self-Attention Mechanism**
+   - Allows the model to focus on different parts of the input sequence
+   - Computes attention weights for each position relative to all other positions
+   - Formula: `Attention(Q,K,V) = softmax(QK^T/√d_k)V`
+
+2. **Multi-Head Attention**
+   - Multiple attention mechanisms running in parallel
+   - Each head learns different types of relationships
+   - Concatenated and linearly transformed
+
+3. **Positional Encoding**
+   - Provides sequence order information (transformers have no inherent notion of order)
+   - Uses sinusoidal functions: `PE(pos,2i) = sin(pos/10000^(2i/d_model))`
+
+4. **Feed-Forward Networks**
+   - Process attended information through dense layers
+   - Applied to each position separately and identically
+
+5. **Layer Normalization**
+   - Stabilizes training and improves convergence
+   - Applied before each sub-layer (Pre-LN) or after (Post-LN)
+
+### Why Transformers Excel at Audio Processing?
+
+1. **Sequence Modeling**: Audio is inherently sequential data with temporal dependencies
+2. **Long-Range Dependencies**: Can capture relationships across entire audio sequences
+3. **Parallel Processing**: Unlike RNNs, transformers can process all time steps simultaneously
+4. **Attention to Relevant Features**: Focus on important audio segments for transcription
+5. **Scalability**: Performance improves with model size and data
+
+---
+
+## Audio Transformers: From Sound Waves to Text
+
+### Audio Processing Pipeline in Transformers
+
+#### Step 1: Audio Preprocessing
+```python
+# From audio_utils.py
+def preprocess_audio(self, audio_input: Union[str, np.ndarray]) -> np.ndarray:
+    """Preprocess audio for optimal speech recognition."""
+    
+    # Load and resample to 16kHz (standard for speech models)
+    if isinstance(audio_input, str):
+        audio, sr = librosa.load(audio_input, sr=self.target_sr)
+    else:
+        audio = audio_input
+    
+    # Resample if needed
+    if sr != self.target_sr:
+        audio = librosa.resample(audio, orig_sr=sr, target_sr=self.target_sr)
+    
+    # Normalize amplitude
+    audio = librosa.util.normalize(audio)
+    
+    # Trim silence from beginning/end
+    audio, _ = librosa.effects.trim(audio, top_db=20)
+    
+    # Basic noise reduction
+    if noise_reduction:
+        audio = self._reduce_noise(audio)
+    
+    return audio
+```
+
+#### Step 2: Feature Extraction
+- **Mel-spectrograms**: Convert audio waveform to frequency domain representation
+- **Log-mel features**: Logarithmic scaling for better perceptual representation
+- **Windowing**: Short-time analysis with overlapping windows
+- **Positional encoding**: Add temporal information to features
+
+#### Step 3: Transformer Processing
+- **Encoder**: Processes audio features with self-attention layers
+- **Decoder**: Generates text tokens sequentially (for encoder-decoder models)
+- **Cross-attention**: Links audio features to text generation
+
+### Audio-Specific Transformer Adaptations
+
+1. **Convolutional Front-end**: Extract local audio features before transformer layers
+2. **Relative Positional Encoding**: Better handling of variable-length audio sequences
+3. **Chunked Processing**: Handle long audio sequences efficiently
+4. **Multi-scale Features**: Process audio at different temporal resolutions
+
+---
+
+## Model Architectures Implementation
+
+### A. Whisper Models (OpenAI)
+
+**Architecture**: Encoder-Decoder Transformer with Cross-Attention
+
+```python
+# From speech_to_text.py
+def _load_whisper_model(self) -> None:
+    """Load Whisper-based models with optimization."""
+    self.pipe = pipeline(
+        "automatic-speech-recognition",
+        model=self.model_id,  # e.g., "openai/whisper-large-v3"
+        dtype=self.torch_dtype,
+        device=self.device,
+        model_kwargs={"cache_dir": self.cache_dir, "use_safetensors": True},
+        return_timestamps=True
+    )
+```
+
+#### How Whisper Works:
+1. **Audio Encoder**: 
+   - Processes 80-channel log-mel spectrogram
+   - 6 convolutional layers followed by transformer blocks
+   - Self-attention across time and frequency dimensions
+
+2. **Text Decoder**: 
+   - Generates text tokens autoregressively
+   - Cross-attention to audio encoder outputs
+   - Language identification and task specification
+
+3. **Training Strategy**:
+   - Trained on 680,000 hours of multilingual data
+   - Multitask learning: transcription, translation, language ID
+   - Zero-shot capability for new languages
+
+### B. Wav2Vec2 Models (Meta/Facebook)
+
+**Architecture**: Self-Supervised Transformer with CTC Head
+
+```python
+def _load_wav2vec2_model(self) -> None:
+    """Load Wav2Vec2 models."""
+    self.model = Wav2Vec2ForCTC.from_pretrained(
+        self.model_id,  # e.g., "ai4bharat/indicwav2vec-hindi"
+        cache_dir=self.cache_dir
+    ).to(self.device)
+    
+    self.processor = Wav2Vec2Processor.from_pretrained(
+        self.model_id,
+        cache_dir=self.cache_dir
+    )
+```
+
+#### How Wav2Vec2 Works:
+1. **Self-Supervised Pre-training**:
+   - Learns audio representations without transcription labels
+   - Contrastive learning: distinguish true vs. false audio segments
+   - Masked prediction: predict masked audio segments
+
+2. **Architecture Components**:
+   - **Feature Encoder**: 7 convolutional layers (raw audio → latent features)
+   - **Transformer**: 12-24 layers with self-attention
+   - **Quantization Module**: Discretizes continuous representations
+
+3. **Fine-tuning for ASR**:
+   - Add CTC (Connectionist Temporal Classification) head
+   - Train on labeled speech data
+   - Language-specific optimization possible
+
+4. **CTC Decoding Process**:
+   ```python
+   def _transcribe_wav2vec2(self, audio_input: Union[str, np.ndarray]) -> str:
+       # Preprocess audio
+       audio, sr = librosa.load(audio_input, sr=16000)
+       
+       # Convert to model input format
+       input_values = self.processor(
+           audio, 
+           return_tensors="pt", 
+           sampling_rate=16000
+       ).input_values.to(self.device)
+       
+       # Forward pass through transformer
+       with torch.no_grad():
+           logits = self.model(input_values).logits
+       
+       # CTC decoding: collapse repeated tokens and remove blanks
+       prediction_ids = torch.argmax(logits, dim=-1)
+       transcription = self.processor.batch_decode(prediction_ids)[0]
+       
+       return transcription
+   ```
+
+---
+
+## Audio Processing Pipeline
+
+### Advanced Audio Preprocessing
+
+#### Noise Reduction Using Spectral Subtraction
+```python
+def _reduce_noise(self, audio: np.ndarray, noise_factor: float = 0.1) -> np.ndarray:
+    """Simple noise reduction using spectral subtraction."""
+    try:
+        # Compute Short-Time Fourier Transform
+        stft = librosa.stft(audio)
+        magnitude = np.abs(stft)
+        phase = np.angle(stft)
+        
+        # Estimate noise from first few frames
+        noise_frames = min(10, magnitude.shape[1] // 4)
+        noise_profile = np.mean(magnitude[:, :noise_frames], axis=1, keepdims=True)
+        
+        # Spectral subtraction
+        clean_magnitude = magnitude - noise_factor * noise_profile
+        clean_magnitude = np.maximum(clean_magnitude, 0.1 * magnitude)
+        
+        # Reconstruct audio
+        clean_stft = clean_magnitude * np.exp(1j * phase)
+        clean_audio = librosa.istft(clean_stft)
+        
+        return clean_audio
+        
+    except Exception as e:
+        self.logger.warning(f"Noise reduction failed: {e}")
+        return audio
+```
+
+---
+
+## Performance Optimization
+
+### GPU Acceleration and Mixed Precision
+
+```python
+# From speech_to_text.py - Device and precision configuration
+def __init__(self, model_type: str = "distil-whisper", language: str = "hindi"):
+    self.device = "cuda" if torch.cuda.is_available() and os.getenv("ENABLE_GPU", "True") == "True" else "cpu"
+    self.torch_dtype = torch.float16 if self.device == "cuda" else torch.float32
+```
+
+### TensorFlow Integration
+
+```python
+# From tensorflow_integration.py
+def _configure_tensorflow(self):
+    """Configure TensorFlow for optimal performance."""
+    try:
+        # Enable mixed precision for faster inference
+        tf.keras.mixed_precision.set_global_policy('mixed_float16')
+        
+        # Configure GPU memory growth to avoid OOM
+        gpus = tf.config.experimental.list_physical_devices('GPU')
+        if gpus:
+            for gpu in gpus:
+                tf.config.experimental.set_memory_growth(gpu, True)
+                
+    except Exception as e:
+        self.logger.warning(f"TensorFlow configuration warning: {e}")
+```
+
+---
+
+## Model Comparison and Benchmarks
+
+### Performance Metrics Table
+
+| Model | RTF | Memory (GPU) | WER (Hindi) | Languages | Best Use Case |
+|-------|-----|--------------|-------------|-----------|---------------|
+| **Distil-Whisper** | 0.17 | ~2GB | 8.5% | 99 | Production deployment |
+| **Whisper Large** | 1.0 | ~4GB | 8.1% | 99 | Best accuracy |
+| **Whisper Small** | 0.5 | ~1GB | 10.2% | 99 | CPU deployment |
+| **Wav2Vec2 Hindi** | 0.3 | ~1GB | 12% | 1 | Hindi specialization |
+| **SeamlessM4T** | 1.5 | ~6GB | 9.8% | 101 | Multilingual tasks |
+
+---
+
+## Code Examples and Usage Patterns
+
+### Basic Usage
+
+```python
+# Initialize the speech-to-text system
+from src.models.speech_to_text import FreeIndianSpeechToText
+
+# Single model usage
+asr = FreeIndianSpeechToText(model_type="distil-whisper")
+
+# Transcribe audio file
+result = asr.transcribe("hindi_audio.wav", language_code="hi")
+print(f"Transcription: {result['text']}")
+print(f"Processing time: {result['processing_time']:.2f}s")
+
+# Switch models dynamically
+asr.switch_model("wav2vec2-hindi")
+result = asr.transcribe("hindi_audio.wav", language_code="hi")
+```
+
+### Batch Processing
+
+```python
+def batch_transcribe(self, audio_paths: List[str], language_code: str = "hi") -> List[Dict]:
+    """Enhanced batch transcription with progress tracking."""
+    results = []
+    total_files = len(audio_paths)
+    
+    for i, audio_path in enumerate(audio_paths):
+        progress = (i + 1) / total_files * 100
+        self.logger.info(f"Processing file {i+1}/{total_files} ({progress:.1f}%): {audio_path}")
+        
+        try:
+            result = self.transcribe(audio_path, language_code)
+            result["file"] = audio_path
+            results.append(result)
+        except Exception as e:
+            results.append({
+                "file": audio_path, 
+                "error": str(e),
+                "success": False
+            })
+    
+    return results
+```
+
+---
+
+## Best Practices and Production Deployment
+
+### Environment Configuration
+
+```python
+# .env.local configuration
+APP_ENV=local
+DEBUG=True
+MODEL_CACHE_DIR=./models
+GRADIO_SERVER_NAME=127.0.0.1
+GRADIO_SERVER_PORT=7860
+DEFAULT_MODEL=distil-whisper
+ENABLE_GPU=True
+```
+
+### Docker Deployment
+
+```dockerfile
+# From Dockerfile
+FROM python:3.9-slim
+
+WORKDIR /app
+COPY requirements.txt .
+RUN pip install -r requirements.txt
+
+COPY . .
+EXPOSE 7860
+
+CMD ["python", "app.py"]
+```
+
+### Model Selection Guidelines
+
+1. **Production**: Use Distil-Whisper for best speed-accuracy balance
+2. **Accuracy**: Use Whisper Large for highest quality transcription
+3. **Hindi-specific**: Use Wav2Vec2 Hindi for specialized Hindi processing
+4. **CPU deployment**: Use Whisper Small for resource-constrained environments
+5. **Multilingual**: Use SeamlessM4T for 101 language support
+
+### Error Handling and Monitoring
+
+```python
+def transcribe_with_error_handling(self, audio_input, language_code="hi"):
+    """Robust transcription with comprehensive error handling."""
+    try:
+        # Validate input
+        if not audio_input:
+            return {"error": "No audio input provided", "success": False}
+        
+        # Check model status
+        if not self.current_model:
+            return {"error": "No model loaded", "success": False}
+        
+        # Perform transcription
+        result = self.transcribe(audio_input, language_code)
+        
+        # Log success metrics
+        if result["success"]:
+            self.logger.info(f"Transcription successful: {result['processing_time']:.2f}s")
+        
+        return result
+        
+    except Exception as e:
+        self.logger.error(f"Transcription failed: {str(e)}")
+        return {"error": str(e), "success": False}
+```
+
+---
+
+## Conclusion
+
+This guide provides a comprehensive understanding of AI transformers in audio processing, demonstrating practical implementation through a production-ready speech-to-text system for Indian languages. The combination of theoretical knowledge and hands-on code examples makes it an excellent resource for understanding modern audio AI systems.
+
+### Key Takeaways
+
+1. **Transformers revolutionized audio processing** through attention mechanisms and parallel processing
+2. **Multiple architectures serve different purposes**: Whisper for general use, Wav2Vec2 for specialization
+3. **Performance optimization is crucial** for production deployment
+4. **Proper preprocessing enhances accuracy** significantly
+5. **Model selection depends on specific requirements** and constraints
+
+The project showcases best practices in AI system design, from environment configuration to production deployment, making it a valuable reference for audio AI development.

app.py

@@ -3,11 +3,16 @@
 Hugging Face Spaces optimized version of the Indian Speech-to-Text application.
 This version is specifically configured for deployment on Hugging Face Spaces.
 """
-
 import os
 import sys
 import logging
 from pathlib import Path
+from dotenv import load_dotenv
+
+# Explicitly load .env from ./config/env/.env
+env_path = Path(__file__).parent / "config" / "env" / ".env"
+load_dotenv(dotenv_path=env_path, override=True)
+
 
 # Set up environment for Spaces
 os.environ['APP_ENV'] = 'prod'
@@ -16,6 +21,7 @@ os.environ['GRADIO_SERVER_PORT'] = '7860'
 os.environ['MODEL_CACHE_DIR'] = '/app/models'
 os.environ['HF_HOME'] = '/app/models'
 os.environ['TRANSFORMERS_CACHE'] = '/app/models'
+os.environ['HF_TOKEN'] = os.getenv('HF_TOKEN') or os.getenv('HUGGINGFACE_HUB_TOKEN') or ""
 
 # Add src to Python path
 src_path = Path(__file__).parent / "src"

requirements_spaces.txt

@@ -1,15 +0,0 @@
-torch>=2.0.0
-transformers>=4.35.0
-gradio>=4.0.0
-librosa>=0.10.0
-datasets>=2.14.0
-accelerate>=0.24.0
-safetensors>=0.4.0
-soundfile>=0.12.0
-numpy>=1.24.0
-scipy>=1.11.0
-python-dotenv>=1.0.0
-pydub>=0.25.0
-ffmpeg-python>=0.2.0
-huggingface-hub>=0.19.0
-psutil>=5.9.0