Spaces:
Running
Running
| title: ClearSpeech API | |
| emoji: 🎙️ | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: docker | |
| pinned: false | |
| license: mit | |
| # 🎙️ ClearSpeech | |
| **AI-Powered Speech Enhancement & Transcription System** | |
| ClearSpeech uses a custom U-Net deep learning model to remove background noise from audio, then transcribes the enhanced audio using OpenAI's Whisper. Perfect for cleaning up voice recordings, meeting audio, podcasts, or any noisy speech. | |
| **🌐 Live Website (will be updated)**: https://clearspeech.yourdomain.com | |
| ## 🌟 Features | |
| - 🧹 **AI-Powered Noise Reduction**: Custom U-Net model trained to remove background noise | |
| - 📝 **Automatic Transcription**: Whisper integration for accurate speech-to-text | |
| - ⚡ **Fast Processing**: Optimized pipeline with GPU support | |
| - 🌐 **REST API**: Easy-to-use FastAPI backend | |
| - 🎯 **High Quality**: Val loss of 0.031 | |
| - 🔧 **Flexible**: Enhancement-only, transcription-only, or both | |
| ## 📋 Table of Contents | |
| - Installation | |
| - Quick Start | |
| - API Documentation | |
| - Project Structure | |
| - Contributing | |
| ## 🚀 Installation | |
| ### Prerequisites | |
| - Python 3.8+ | |
| - pip | |
| - Optional CUDA GPU | |
| ### Step 1: Clone Repository | |
| ``` | |
| git clone https://github.com/yourusername/ClearSpeech.git | |
| cd ClearSpeech | |
| ``` | |
| ### Step 2: Create Virtual Environment | |
| ``` | |
| # Create environment | |
| python3.10 -m venv venv | |
| # Activate (macOS/Linux) | |
| source venv/bin/activate | |
| # Activate (Windows) | |
| venv\Scripts\activate | |
| ``` | |
| ### Step 3: Install Dependencies | |
| ``` | |
| # Install dependencies | |
| pip install -r requirements.txt | |
| ``` | |
| ### Step 4: Download Pretrained Model | |
| ``` | |
| # Download model | |
| python -c " | |
| from huggingface_hub import hf_hub_download | |
| hf_hub_download( | |
| repo_id='thecodeworm/clearspeech-unet', | |
| filename='best_model.pt', | |
| local_dir='enhancement_model/checkpoints/' | |
| ) | |
| " | |
| ``` | |
| ### Step 5: Generate Noisy Samples | |
| 1. Make your own WAV sample | |
| 2. Run the generate_noisy_samples.py file on the sample to make the audio noisier to test the model | |
| ``` | |
| # Generate all noise types at multiple SNR levvels | |
| python generate_noisy_samples.py \ | |
| --input my_clean_voice.wav \ | |
| --output test_samples/ | |
| ``` | |
| ## ⚡ Quick Start | |
| ### Method 1: Using the API (Recommended) | |
| **Start the server:** | |
| ``` | |
| python -m backend.app | |
| ``` | |
| Server starts at `http://localhost:8000` | |
| **Start the server:** | |
| ``` | |
| cd frontend | |
| python -m http.server 3000 | |
| ``` | |
| Frontend starts at `http://localhost:3000` | |
| **Process audio:** | |
| ``` | |
| # Full pipeline (enhance + transcribe) | |
| curl -X POST "http://localhost:8000/process" \ | |
| -F "file=@your_audio.wav" \ | |
| | jq . | |
| # Enhance only | |
| curl -X POST "http://localhost:8000/enhance" \ | |
| -F "file=@your_audio.wav" \ | |
| -o enhanced_output.wav | |
| # Transcribe only | |
| curl -X POST "http://localhost:8000/transcribe" \ | |
| -F "file=@your_audio.wav" \ | |
| -F "enhance=true" \ | |
| | jq . | |
| ``` | |
| **Method 2: Using Python** | |
| ``` | |
| from backend.inference_pipeline import EnhancementPipeline | |
| # Initialize pipeline | |
| pipeline = EnhancementPipeline( | |
| cnn_checkpoint_path="enhancement_model/checkpoints/best_model.pt", | |
| whisper_model_name="base", | |
| device="cpu" # or "cuda" or "mps" | |
| ) | |
| # Process audio | |
| result = pipeline.process("path/to/noisy_audio.wav") | |
| print(f"Transcript: {result['transcript']}") | |
| print(f"Duration: {result['duration']:.2f}s") | |
| # Save enhanced audio | |
| import soundfile as sf | |
| sf.write("enhanced.wav", result['enhanced_audio'], result['sample_rate']) | |
| ``` | |
| **Method 3: Command Line** | |
| ``` | |
| # Enhance audio file | |
| python enhancement_model/infer.py \ | |
| --checkpoint enhancement_model/checkpoints/best_model.pt \ | |
| --input noisy_audio.wav \ | |
| --output enhanced_audio.wav \ | |
| --comparison # Creates stereo comparison file | |
| ``` | |
| ## 📚 API Documentation | |
| ### Interactive Docs | |
| Once the server is running, visit: | |
| - **Swagger UI**: [http://localhost:8000/docs](http://localhost:8000/docs) | |
| - **ReDoc**: [http://localhost:8000/redoc](http://localhost:8000/redoc) | |
| ### Endpoints | |
| #### `POST /process` | |
| Process audio with enhancement and transcription. | |
| **Request:** | |
| ```curl -X POST "http://localhost:8000/process" \ | |
| -F "file=@audio.wav" \ | |
| -F "language=en" \ | |
| -F "skip_enhancement=false" | |
| ``` | |
| **Response:** | |
| ``` | |
| { | |
| "success": true, | |
| "transcript": "Transcribed text here", | |
| "duration": 3.5, | |
| "language": "en", | |
| "enhanced_audio_url": "/download/enhanced_123.wav", | |
| "segments": [...], | |
| "processing_time": 2.3 | |
| } | |
| ``` | |
| #### `POST /enhance` | |
| Enhance audio only (no transcription). | |
| **Request:** | |
| ``` | |
| curl -X POST "http://localhost:8000/enhance" \ | |
| -F "file=@audio.wav" \ | |
| -o enhanced.wav | |
| ``` | |
| **Response:** Enhanced audio file (WAV) | |
| #### `POST /transcribe` | |
| Transcribe audio with optional enhancement. | |
| **Request:** | |
| ``` | |
| curl -X POST "http://localhost:8000/transcribe" \ | |
| -F "file=@audio.wav" \ | |
| -F "language=en" \ | |
| -F "enhance=true" | |
| ``` | |
| **Response:** | |
| ``` | |
| { | |
| "success": true, | |
| "transcript": "Transcribed text", | |
| "duration": 3.5, | |
| "language": "en", | |
| "segments": [...] | |
| } | |
| ``` | |
| #### `GET /download/{filename}` | |
| Download enhanced audio file. | |
| #### `GET /health` | |
| Health check endpoint. | |
| ## 📁 Project Structure | |
| ``` | |
| ClearSpeech/ | |
| ├── backend/ # FastAPI backend | |
| │ ├── app.py # Main API server | |
| │ ├── inference_pipeline.py # Processing pipeline | |
| │ └── requirements.txt | |
| ├── enhancement_model/ # U-Net model | |
| │ ├── model.py # U-Net architecture | |
| │ ├── dataset.py # PyTorch dataset | |
| │ ├── train.py # Training script | |
| │ ├── infer.py # Inference script | |
| │ ├── checkpoints/ # Trained models | |
| │ │ └── best_model.pt | |
| │ └── requirements.txt | |
| ├── data/ # Training/test data | |
| │ ├── audio_clean/ # Clean audio | |
| │ ├── audio_raw/ # Noisy audio | |
| │ ├── metadata/ | |
| │ │ └── metadata.json # Dataset metadata | |
| │ └── spectrograms/ # Mel-spectrograms | |
| │ ├── clean/ | |
| │ └── noisy/ | |
| ├── frontend/ # Web interface (optional) | |
| │ ├── index.html | |
| │ └── script.js | |
| ├── tests/ # Test files | |
| │ └── test_backend.py | |
| ├── README.md | |
| └── requirements.txt | |
| ``` | |
| ## 🤝 Contributing | |
| We welcome contributions! Here's how: | |
| 1. **Fork the repository** | |
| 2. **Create a feature branch**: `git checkout -b feature/amazing-feature` | |
| 3. **Commit changes**: `git commit -m 'Add amazing feature'` | |
| 4. **Push to branch**: `git push origin feature/amazing-feature` | |
| 5. **Open a Pull Request** | |
| **Development Setup** | |
| ``` | |
| # Install dev dependencies | |
| pip install -r requirements-dev.txt | |
| # Run tests before committing | |
| python -m pytest tests/ | |
| # Format code | |
| black backend/ enhancement_model/ | |
| ``` | |
| ## 🙏 Acknowledgments | |
| - **U-Net Architecture**: Inspired by [Ronneberger et al.](https://arxiv.org/abs/1505.04597) | |
| - **Whisper**: [OpenAI Whisper](https://github.com/openai/whisper) | |
| - **Training Data**: [LibriSpeech](http://www.openslr.org/12/), [MS-SNSD](https://github.com/microsoft/MS-SNSD) | |
| ## 📧 Contact | |
| **Project Maintainers**: Aditya Chanda, Josh Pal, Advik Kumar Singh | |
| **Project Link**: [https://github.com/thecodeworm/ClearSpeech](https://github.com/thecodeworm/ClearSpeech) | |
| ## ⭐ Show Your Support | |
| Give a ⭐️ if this project helped you! | |
| ---------- | |
| **Built with ❤️ using PyTorch and FastAPI** | |