--- title: ClearSpeech API emoji: ๐ŸŽ™๏ธ colorFrom: blue colorTo: purple sdk: docker pinned: false license: mit --- # ๐ŸŽ™๏ธ ClearSpeech **AI-Powered Speech Enhancement & Transcription System** ClearSpeech uses a custom U-Net deep learning model to remove background noise from audio, then transcribes the enhanced audio using OpenAI's Whisper. Perfect for cleaning up voice recordings, meeting audio, podcasts, or any noisy speech. **๐ŸŒ Live Website (will be updated)**: https://clearspeech.yourdomain.com ## ๐ŸŒŸ Features - ๐Ÿงน **AI-Powered Noise Reduction**: Custom U-Net model trained to remove background noise - ๐Ÿ“ **Automatic Transcription**: Whisper integration for accurate speech-to-text - โšก **Fast Processing**: Optimized pipeline with GPU support - ๐ŸŒ **REST API**: Easy-to-use FastAPI backend - ๐ŸŽฏ **High Quality**: Val loss of 0.031 - ๐Ÿ”ง **Flexible**: Enhancement-only, transcription-only, or both ## ๐Ÿ“‹ Table of Contents - Installation - Quick Start - API Documentation - Project Structure - Contributing ## ๐Ÿš€ Installation ### Prerequisites - Python 3.8+ - pip - Optional CUDA GPU ### Step 1: Clone Repository ``` git clone https://github.com/yourusername/ClearSpeech.git cd ClearSpeech ``` ### Step 2: Create Virtual Environment ``` # Create environment python3.10 -m venv venv # Activate (macOS/Linux) source venv/bin/activate # Activate (Windows) venv\Scripts\activate ``` ### Step 3: Install Dependencies ``` # Install dependencies pip install -r requirements.txt ``` ### Step 4: Download Pretrained Model ``` # Download model python -c " from huggingface_hub import hf_hub_download hf_hub_download( repo_id='thecodeworm/clearspeech-unet', filename='best_model.pt', local_dir='enhancement_model/checkpoints/' ) " ``` ### Step 5: Generate Noisy Samples 1. Make your own WAV sample 2. Run the generate_noisy_samples.py file on the sample to make the audio noisier to test the model ``` # Generate all noise types at multiple SNR levvels python generate_noisy_samples.py \ --input my_clean_voice.wav \ --output test_samples/ ``` ## โšก Quick Start ### Method 1: Using the API (Recommended) **Start the server:** ``` python -m backend.app ``` Server starts at `http://localhost:8000` **Start the server:** ``` cd frontend python -m http.server 3000 ``` Frontend starts at `http://localhost:3000` **Process audio:** ``` # Full pipeline (enhance + transcribe) curl -X POST "http://localhost:8000/process" \ -F "file=@your_audio.wav" \ | jq . # Enhance only curl -X POST "http://localhost:8000/enhance" \ -F "file=@your_audio.wav" \ -o enhanced_output.wav # Transcribe only curl -X POST "http://localhost:8000/transcribe" \ -F "file=@your_audio.wav" \ -F "enhance=true" \ | jq . ``` **Method 2: Using Python** ``` from backend.inference_pipeline import EnhancementPipeline # Initialize pipeline pipeline = EnhancementPipeline( cnn_checkpoint_path="enhancement_model/checkpoints/best_model.pt", whisper_model_name="base", device="cpu" # or "cuda" or "mps" ) # Process audio result = pipeline.process("path/to/noisy_audio.wav") print(f"Transcript: {result['transcript']}") print(f"Duration: {result['duration']:.2f}s") # Save enhanced audio import soundfile as sf sf.write("enhanced.wav", result['enhanced_audio'], result['sample_rate']) ``` **Method 3: Command Line** ``` # Enhance audio file python enhancement_model/infer.py \ --checkpoint enhancement_model/checkpoints/best_model.pt \ --input noisy_audio.wav \ --output enhanced_audio.wav \ --comparison # Creates stereo comparison file ``` ## ๐Ÿ“š API Documentation ### Interactive Docs Once the server is running, visit: - **Swagger UI**: [http://localhost:8000/docs](http://localhost:8000/docs) - **ReDoc**: [http://localhost:8000/redoc](http://localhost:8000/redoc) ### Endpoints #### `POST /process` Process audio with enhancement and transcription. **Request:** ```curl -X POST "http://localhost:8000/process" \ -F "file=@audio.wav" \ -F "language=en" \ -F "skip_enhancement=false" ``` **Response:** ``` { "success": true, "transcript": "Transcribed text here", "duration": 3.5, "language": "en", "enhanced_audio_url": "/download/enhanced_123.wav", "segments": [...], "processing_time": 2.3 } ``` #### `POST /enhance` Enhance audio only (no transcription). **Request:** ``` curl -X POST "http://localhost:8000/enhance" \ -F "file=@audio.wav" \ -o enhanced.wav ``` **Response:** Enhanced audio file (WAV) #### `POST /transcribe` Transcribe audio with optional enhancement. **Request:** ``` curl -X POST "http://localhost:8000/transcribe" \ -F "file=@audio.wav" \ -F "language=en" \ -F "enhance=true" ``` **Response:** ``` { "success": true, "transcript": "Transcribed text", "duration": 3.5, "language": "en", "segments": [...] } ``` #### `GET /download/{filename}` Download enhanced audio file. #### `GET /health` Health check endpoint. ## ๐Ÿ“ Project Structure ``` ClearSpeech/ โ”œโ”€โ”€ backend/ # FastAPI backend โ”‚ โ”œโ”€โ”€ app.py # Main API server โ”‚ โ”œโ”€โ”€ inference_pipeline.py # Processing pipeline โ”‚ โ””โ”€โ”€ requirements.txt โ”œโ”€โ”€ enhancement_model/ # U-Net model โ”‚ โ”œโ”€โ”€ model.py # U-Net architecture โ”‚ โ”œโ”€โ”€ dataset.py # PyTorch dataset โ”‚ โ”œโ”€โ”€ train.py # Training script โ”‚ โ”œโ”€โ”€ infer.py # Inference script โ”‚ โ”œโ”€โ”€ checkpoints/ # Trained models โ”‚ โ”‚ โ””โ”€โ”€ best_model.pt โ”‚ โ””โ”€โ”€ requirements.txt โ”œโ”€โ”€ data/ # Training/test data โ”‚ โ”œโ”€โ”€ audio_clean/ # Clean audio โ”‚ โ”œโ”€โ”€ audio_raw/ # Noisy audio โ”‚ โ”œโ”€โ”€ metadata/ โ”‚ โ”‚ โ””โ”€โ”€ metadata.json # Dataset metadata โ”‚ โ””โ”€โ”€ spectrograms/ # Mel-spectrograms โ”‚ โ”œโ”€โ”€ clean/ โ”‚ โ””โ”€โ”€ noisy/ โ”œโ”€โ”€ frontend/ # Web interface (optional) โ”‚ โ”œโ”€โ”€ index.html โ”‚ โ””โ”€โ”€ script.js โ”œโ”€โ”€ tests/ # Test files โ”‚ โ””โ”€โ”€ test_backend.py โ”œโ”€โ”€ README.md โ””โ”€โ”€ requirements.txt ``` ## ๐Ÿค Contributing We welcome contributions! Here's how: 1. **Fork the repository** 2. **Create a feature branch**: `git checkout -b feature/amazing-feature` 3. **Commit changes**: `git commit -m 'Add amazing feature'` 4. **Push to branch**: `git push origin feature/amazing-feature` 5. **Open a Pull Request** **Development Setup** ``` # Install dev dependencies pip install -r requirements-dev.txt # Run tests before committing python -m pytest tests/ # Format code black backend/ enhancement_model/ ``` ## ๐Ÿ™ Acknowledgments - **U-Net Architecture**: Inspired by [Ronneberger et al.](https://arxiv.org/abs/1505.04597) - **Whisper**: [OpenAI Whisper](https://github.com/openai/whisper) - **Training Data**: [LibriSpeech](http://www.openslr.org/12/), [MS-SNSD](https://github.com/microsoft/MS-SNSD) ## ๐Ÿ“ง Contact **Project Maintainers**: Aditya Chanda, Josh Pal, Advik Kumar Singh **Project Link**: [https://github.com/thecodeworm/ClearSpeech](https://github.com/thecodeworm/ClearSpeech) ## โญ Show Your Support Give a โญ๏ธ if this project helped you! ---------- **Built with โค๏ธ using PyTorch and FastAPI**