Spaces:
Running
Running
| title: Scene Graph Generator | |
| emoji: 🧠 | |
| colorFrom: blue | |
| colorTo: purple | |
| sdk: gradio | |
| app_file: app.py | |
| pinned: false | |
| # 🧠 Scene Graph Generator (Multimodal AI System) | |
| A multimodal computer vision system that takes an input image, detects objects, predicts relationships between them, constructs a structured scene graph, and generates a natural language description of the scene. | |
| 🔗 **Live Demo (Hugging Face Spaces):** | |
| https://huggingface.co/spaces/<your-username>/scene-graph-generator | |
| --- | |
| # 🚀 Features | |
| - 🖼️ Object Detection using DETR (ResNet-50) | |
| - 🔗 Relationship Prediction (Custom Trained Model) | |
| - 📐 Spatial Reasoning (Hybrid AI with Geometry Rules) | |
| - 🧩 Scene Graph Construction (Directed Graph) | |
| - 📊 Graph Visualization (NetworkX + Matplotlib) | |
| - 🧠 Graph-to-Text Generation (FLAN-T5) | |
| - 🌐 Interactive UI (Gradio) | |
| - ☁️ Deployed on Hugging Face Spaces (CPU) | |
| --- | |
| # 🧠 How It Works (End-to-End Pipeline) | |
| ### 1. Input | |
| - User uploads an image (JPG/PNG) via Gradio UI | |
| - Image is converted from PIL → OpenCV format | |
| --- | |
| ### 2. Object Detection | |
| - Uses `facebook/detr-resnet-50` from Hugging Face | |
| - Outputs: | |
| - Object labels (COCO classes) | |
| - Bounding boxes | |
| - Confidence scores | |
| - Applies threshold (≥ 0.7) to filter noise | |
| --- | |
| ### 3. Pairwise Object Processing | |
| - Generates object pairs using `itertools.combinations` | |
| - Extracts bounding boxes for each pair | |
| - Creates union region for relation inference | |
| - Filters duplicate object pairs | |
| --- | |
| ### 4. Relationship Prediction | |
| - Custom-trained classifier on Visual Genome subset (~10K samples) | |
| - Predicts semantic relations: | |
| - `on`, `holding`, `behind`, etc. | |
| - Trained using PyTorch (10 epochs) | |
| --- | |
| ### 5. Spatial Reasoning (Hybrid AI) | |
| - Uses bounding box geometry to compute: | |
| - `left_of`, `right_of`, `above`, `below`, `near` | |
| - Hybrid logic: | |
| - Semantic relations from model (if confident) | |
| - Otherwise fallback to spatial rules | |
| - Reduces bias (e.g., “everything = on”) | |
| --- | |
| ### 6. Graph Construction | |
| - Builds a **directed graph (NetworkX DiGraph)** | |
| - Nodes → objects | |
| - Edges → relationships | |
| - Removes duplicates and limits edges for clarity | |
| --- | |
| ### 7. Graph Visualization | |
| - Uses NetworkX + Matplotlib | |
| - Displays: | |
| - Directed edges with labels | |
| - Clean layout for readability | |
| --- | |
| ### 8. Graph → Text (NLP) | |
| - Uses `google/flan-t5-small` | |
| - Converts structured triples into natural language | |
| Example: | |
| laptop → on → table | |
| mouse → next_to → laptop | |
| Output: | |
| "A laptop is placed on a table with a mouse next to it." | |
| --- | |
| ### 9. UI (Gradio) | |
| - Upload image | |
| - View: | |
| - Scene graph | |
| - Generated description | |
| - Fully interactive and browser-based | |
| --- | |
| # 🏗️ Tech Stack | |
| ``` | |
| - **Computer Vision:** DETR (Hugging Face Transformers) | |
| - **Deep Learning:** PyTorch | |
| - **Graph Processing:** NetworkX | |
| - **NLP:** FLAN-T5 | |
| - **Image Processing:** OpenCV | |
| - **Frontend/UI:** Gradio | |
| - **Deployment:** Hugging Face Spaces | |
| ``` | |
| --- | |
| # 📁 Project Structure | |
| scene-graph-generator/ | |
| │ | |
| ├── app.py | |
| ├── requirements.txt | |
| ├── README.md | |
| │ | |
| ├── src/ | |
| │ ├── pipeline.py | |
| │ ├── detection.py | |
| │ ├── spatial_rules.py | |
| │ ├── relationship_infer.py | |
| │ ├── scene_graph.py | |
| │ ├── visualization.py | |
| │ ├── text_generation.py | |
| --- | |
| # ⚙️ Installation (Local Setup) | |
| ```bash | |
| git clone https://github.com/<your-username>/scene-graph-generator.git | |
| cd scene-graph-generator | |
| pip install -r requirements.txt | |
| python app.py |