Spaces:
Sleeping
Sleeping
| license: mit | |
| title: Generate Knowledge Graphs | |
| sdk: streamlit | |
| emoji: 📉 | |
| colorFrom: indigo | |
| colorTo: pink | |
| short_description: Use LLM to generate a knowledge graph from your input data. | |
| # 🕸️ Knowledge Graph Extraction App | |
| A complete knowledge graph extraction application using LLMs via OpenRouter, available in both Gradio and Streamlit versions. | |
| ## 🚀 Features | |
| - **Multi-format Document Support**: PDF, TXT, DOCX, JSON files up to 10MB | |
| - **LLM-powered Extraction**: Uses OpenRouter API with free models (Gemma-2-9B, Llama-3.1-8B) | |
| - **Smart Entity Detection**: Automatically identifies people, organizations, locations, concepts, events, and objects | |
| - **Importance Scoring**: LLM evaluates entity importance from 0.0 to 1.0 | |
| - **Interactive Visualization**: Multiple graph layout algorithms with filtering options | |
| - **Batch Processing**: Optional processing of multiple documents together | |
| - **Export Capabilities**: JSON, GraphML, and GEXF formats | |
| - **Real-time Statistics**: Graph metrics and centrality analysis | |
| ## 📁 Project Structure | |
| ``` | |
| knowledge-graphs/ | |
| ├── app.py # Main Gradio application (legacy) | |
| ├── app_streamlit.py # Main Streamlit application (recommended) | |
| ├── run_streamlit.py # Simple launcher script | |
| ├── requirements.txt # Python dependencies | |
| ├── README.md # Project documentation | |
| ├── .env.example # Environment variables template | |
| ├── config/ | |
| │ └── settings.py # Configuration management | |
| └── src/ | |
| ├── document_processor.py # Document loading and chunking | |
| ├── llm_extractor.py # LLM-based entity extraction | |
| ├── graph_builder.py # NetworkX graph construction | |
| └── visualizer.py # Graph visualization and export | |
| ``` | |
| ## 🔧 Installation & Setup | |
| ### Option 1: Streamlit Version (Recommended) | |
| The Streamlit version is more stable and has better file handling. | |
| **Quick Start:** | |
| ```bash | |
| python run_streamlit.py | |
| ``` | |
| **Manual Setup:** | |
| 1. **Install dependencies**: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| 2. **Run the Streamlit app**: | |
| ```bash | |
| streamlit run app_streamlit.py --server.address 0.0.0.0 --server.port 8501 | |
| ``` | |
| The app will be available at `http://localhost:8501` | |
| ### Option 2: Gradio Version (Legacy) | |
| The Gradio version may have some file caching issues but is provided for compatibility. | |
| 1. **Install dependencies**: | |
| ```bash | |
| pip install -r requirements.txt | |
| ``` | |
| 2. **Set up environment variables** (optional): | |
| ```bash | |
| cp .env.example .env | |
| # Edit .env and add your OpenRouter API key | |
| ``` | |
| 3. **Run the application**: | |
| ```bash | |
| python app.py | |
| ``` | |
| The app will be available at `http://localhost:7860` | |
| ### HuggingFace Spaces Deployment | |
| For **Streamlit deployment**: | |
| 1. Create a new Space on [HuggingFace Spaces](https://huggingface.co/spaces) | |
| 2. Choose "Streamlit" as the SDK | |
| 3. Upload `app_streamlit.py` as `app.py` (HF Spaces expects this name) | |
| 4. Upload all other project files maintaining directory structure | |
| For **Gradio deployment**: | |
| 1. Create a new Space with "Gradio" as the SDK | |
| 2. Upload `app.py` and all other files | |
| 3. Note: May experience file handling issues | |
| ## 🔑 API Configuration | |
| ### Getting OpenRouter API Key | |
| 1. Visit [OpenRouter.ai](https://openrouter.ai) | |
| 2. Sign up for a free account | |
| 3. Navigate to API Keys section | |
| 4. Generate a new API key | |
| 5. Copy the key and use it in the application | |
| ### Free Models Used | |
| - **Primary**: `google/gemma-2-9b-it:free` | |
| - **Backup**: `meta-llama/llama-3.1-8b-instruct:free` | |
| These models are specifically chosen to minimize API costs while maintaining quality. | |
| ## 📖 Usage Guide | |
| ### Basic Workflow | |
| 1. **Upload Documents**: | |
| - Select one or more files (PDF, TXT, DOCX, JSON) | |
| - Toggle batch mode for multiple document processing | |
| 2. **Configure API**: | |
| - Enter your OpenRouter API key | |
| - Key is stored temporarily for the session | |
| 3. **Customize Settings**: | |
| - Choose graph layout algorithm | |
| - Toggle label visibility options | |
| - Set minimum importance threshold | |
| - Select entity types to include | |
| 4. **Extract Knowledge Graph**: | |
| - Click "Extract Knowledge Graph" button | |
| - Monitor progress through the status updates | |
| - View results in multiple tabs | |
| 5. **Explore Results**: | |
| - **Graph Visualization**: Interactive graph with colored nodes by entity type | |
| - **Statistics**: Detailed metrics about the graph structure | |
| - **Entities**: Complete list of extracted entities with details | |
| - **Central Nodes**: Most important entities based on centrality measures | |
| 6. **Export Data**: | |
| - Choose export format (JSON, GraphML, GEXF) | |
| - Download structured graph data | |
| ### Advanced Features | |
| #### Entity Types | |
| - **PERSON**: Individuals mentioned in the text | |
| - **ORGANIZATION**: Companies, institutions, groups | |
| - **LOCATION**: Places, addresses, geographical entities | |
| - **CONCEPT**: Abstract ideas, theories, methodologies | |
| - **EVENT**: Specific occurrences, meetings, incidents | |
| - **OBJECT**: Physical items, products, artifacts | |
| #### Relationship Types | |
| - **works_at**: Employment relationships | |
| - **located_in**: Geographical associations | |
| - **part_of**: Hierarchical relationships | |
| - **causes**: Causal relationships | |
| - **related_to**: General associations | |
| #### Filtering Options | |
| - **Importance Threshold**: Show only entities above specified importance score | |
| - **Entity Types**: Filter by specific entity categories | |
| - **Layout Algorithms**: Spring, circular, shell, Kamada-Kawai, random | |
| ## 🛠️ Technical Details | |
| ### Architecture Components | |
| 1. **Document Processing**: | |
| - Multi-format file parsing | |
| - Intelligent text chunking with overlap | |
| - File size validation | |
| 2. **LLM Integration**: | |
| - OpenRouter API integration | |
| - Structured prompt engineering | |
| - Error handling and fallback models | |
| 3. **Graph Processing**: | |
| - NetworkX-based graph construction | |
| - Entity deduplication and standardization | |
| - Relationship validation | |
| 4. **Visualization**: | |
| - Matplotlib-based static graphs | |
| - Interactive HTML visualizations | |
| - Multiple export formats | |
| ### Configuration Options | |
| All settings can be modified in `config/settings.py`: | |
| - **Chunk Size**: Default 2000 characters | |
| - **Chunk Overlap**: Default 200 characters | |
| - **Max File Size**: Default 10MB | |
| - **Max Entities**: Default 100 per extraction | |
| - **Max Relationships**: Default 200 per extraction | |
| - **Importance Threshold**: Default 0.3 | |
| ### Differences Between Versions | |
| **Streamlit Version Advantages:** | |
| - More reliable file handling | |
| - Better progress indicators | |
| - Cleaner UI with sidebar configuration | |
| - More stable caching system | |
| - Built-in download functionality | |
| **Gradio Version Advantages:** | |
| - Simpler deployment to HF Spaces | |
| - More compact interface | |
| - Familiar for ML practitioners | |
| ## 🔒 Security & Privacy | |
| - API keys are not stored permanently | |
| - Files are processed temporarily and discarded | |
| - No data is retained between sessions | |
| - All processing happens server-side | |
| ## 🐛 Troubleshooting | |
| ### Common Issues | |
| 1. **"OpenRouter API key is required"**: | |
| - Ensure you've entered a valid API key | |
| - Check the key has sufficient credits | |
| 2. **"No entities extracted"**: | |
| - Document may be too short or unstructured | |
| - Try lowering the importance threshold | |
| - Check if the document contains meaningful text | |
| 3. **File upload issues (Gradio version)**: | |
| - Known issue with Gradio's file caching system | |
| - Try the Streamlit version instead | |
| - Ensure files are valid and not corrupted | |
| 4. **Segmentation fault (local development)**: | |
| - Usually related to matplotlib backend | |
| - Try setting `MPLBACKEND=Agg` environment variable | |
| - Install GUI toolkit if running locally with display | |
| 5. **Module import errors**: | |
| - Ensure all requirements are installed: `pip install -r requirements.txt` | |
| - Check Python version compatibility (3.8+) | |
| ### Performance Tips | |
| - Use batch mode for related documents | |
| - Adjust chunk size for very long documents | |
| - Lower importance threshold for sparse documents | |
| - Use simpler layout algorithms for large graphs | |
| ## 🤝 Contributing | |
| 1. Fork the repository | |
| 2. Create a feature branch | |
| 3. Make your changes | |
| 4. Test with both Streamlit and Gradio versions if applicable | |
| 5. Add tests if applicable | |
| 6. Submit a pull request | |
| ## 📄 License | |
| This project is licensed under the MIT License - see the LICENSE file for details. | |
| ## 🙏 Acknowledgments | |
| - [OpenRouter](https://openrouter.ai) for LLM API access | |
| - [Streamlit](https://streamlit.io) for the modern web interface framework | |
| - [Gradio](https://gradio.app) for the ML-focused web interface | |
| - [NetworkX](https://networkx.org) for graph processing | |
| - [HuggingFace Spaces](https://huggingface.co/spaces) for hosting |