README.md

metadata

title: SongLift LyrGen2
emoji: 🎵
colorFrom: indigo
colorTo: purple
sdk: streamlit
sdk_version: 1.41.0
app_file: app.py
pinned: false

SongLift LyrGen2 🎵

An AI-powered lyrics generation system that uses semantic understanding of existing lyrics to generate new, contextually relevant song lyrics. Built with LangChain, RAG (Retrieval-Augmented Generation), and OpenAI's GPT-4.

🚀 Deploy Your Own

This app is designed to be easily deployed on HuggingFace Spaces. Follow the setup instructions below to create your own instance.

✨ Features

• Semantic Lyrics Generation: Uses vector embeddings of 234K+ lyrics for contextual understanding
• RAG Technology: Retrieval-Augmented Generation finds similar lyrics to inform new creations
• Modern Sensibilities: Trained on contemporary pop and hip-hop lyrics
• Interactive Web Interface: Clean Streamlit interface for easy use
• Source Attribution: Shows which lyrics influenced the generation

🏗️ Architecture

Core Components

• Vector Database: ChromaDB with OpenAI Ada-002 embeddings
• AI Models: GPT-4 for generation, Ada-002 for embeddings
• Data Pipeline: Automated processing of raw lyrics into searchable embeddings
• Dual Deployment: Local development + HuggingFace Spaces production

Workflow

Raw Lyrics → Data Cleaning → Text Chunking → Embeddings → ChromaDB → Generation

🛠️ Local Development

Prerequisites

• Python 3.8+
• OpenAI API key
• HuggingFace token (optional, for dataset access)

Setup

# Clone the repository
git clone <your-repo-url>
cd SongLift_LyrGen2

# Create virtual environment
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env with your API keys

Environment Variables

Create a .env file with:

OPENAI_API_KEY=your_openai_api_key_here
HF_TOKEN=your_huggingface_token_here
DEPLOYMENT_MODE=local

Run Locally

streamlit run app.py

Visit http://localhost:8501

🧪 Testing & Validation

# Test your environment setup
python scripts/test_environment.py

# Test OpenAI connection
python scripts/test_openai_connection.py

# Validate embeddings database
python scripts/test_embeddings.py

📊 Data Processing

The system processes lyrics through a sophisticated pipeline:

1. Raw Data Loading (scripts/process_lyrics.py)

   • Multi-encoding support (UTF-8, Latin-1, CP1252)
   • Section detection ([Verse], [Chorus], etc.)
   • Metadata preservation

2. Text Processing

   • Recursive text splitting (300 chars, 75 overlap)
   • Batch processing with rate limiting
   • Automatic retry on API limits

3. Vector Storage

   • ChromaDB collection: "lyrics_v1"
   • ~234K embedded documents
   • Metadata tracking (artist, song title)

🚀 Deployment

HuggingFace Spaces

The app auto-deploys to HuggingFace Spaces via GitHub sync:

• Space: SongLift/LyrGen2
• Dataset: SongLift/LyrGen2_DB

Configure secrets in HF Spaces settings:

• OPENAI_API_KEY
• HF_TOKEN

Local to Production Sync

# Process and upload embeddings
python scripts/process_lyrics.py
python scripts/upload_embeddings.py

🔧 Configuration

Key configuration in config/settings.py:

• Models: GPT-4 for generation, Ada-002 for embeddings
• Paths: Auto-detects local vs HuggingFace environment
• Database: ChromaDB with persistent storage

📁 Project Structure

SongLift_LyrGen2/
├── app.py                 # Main Streamlit application
├── config/
│   └── settings.py        # Central configuration
├── src/
│   ├── generator/         # Core generation logic
│   └── utils/            # Utility functions
├── scripts/              # Data processing & testing
├── data/
│   ├── raw/lyrics/       # Place your lyrics files here (organized by artist folders)
│   └── processed/        # Generated embeddings & ChromaDB files
└── .env.example          # Environment variables template

📂 Data Directory Setup

The data/ directory structure is preserved for you to add your own lyrics:

data/raw/lyrics/
├── artist1/
│   ├── song1.txt
│   └── song2.txt
├── artist2/
│   ├── song1.txt
│   └── song2.txt
└── ...

After adding lyrics, run the processing pipeline:

python scripts/process_lyrics.py

🔍 Browser Compatibility

⚠️ Recommended: Chrome or Chromium-based browsers for optimal performance. Some features may not work correctly in Safari.

� HouggingFace Spaces Setup

Deploy Your Own Space

1. Create a HuggingFace Space:

   • Go to HuggingFace Spaces
   • Click "Create new Space"
   • Choose "Streamlit" as SDK
   • Set app_file: app.py

2. Configure Secrets:

   • In your Space settings, add these secrets:
     • OPENAI_API_KEY: Your OpenAI API key
     • HF_TOKEN: Your HuggingFace token (for dataset access)

3. Upload Your Dataset:

   # Process and upload embeddings to HF dataset
   python scripts/process_lyrics.py
   python scripts/upload_embeddings.py
   

4. Sync with GitHub (optional):

   • Connect your Space to a GitHub repo for automatic deployments
   • Push changes to GitHub → auto-deploys to HF Spaces

Running HuggingFace Locally

You can test the HuggingFace environment locally:

# Set HuggingFace mode
export DEPLOYMENT_MODE=huggingface

# Run locally (will use HF dataset paths)
streamlit run app.py

This helps debug HF-specific issues before deploying.

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

📄 License

MIT License

Copyright (c) 2024 SongLift

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

🙏 Acknowledgments

• Built with LangChain and Streamlit
• Powered by OpenAI and HuggingFace
• Vector storage by ChromaDB