README.md

---
title: ForgeSight
emoji: 🏗️
colorFrom: red
colorTo: gray
sdk: docker
pinned: true
license: mit
short_description: "Multimodal Civil QC Copilot on AMD MI300X + ROCm"
tags:
  - amd
  - rocm
  - mi300x
  - qwen
  - vllm
  - civil-engineering
  - quality-control
  - agents
---

# 🏗️ ForgeSight — Multimodal QC Copilot on AMD Instinct™ MI300X

ForgeSight is a production-ready **Agentic Quality Control (QC) Pipeline** designed for civil engineering, construction, and infrastructure projects. Built exclusively for the **AMD + lablab.ai Developer Hackathon**, it leverages the massive 192GB VRAM of the **AMD Instinct MI300X** to run a state-of-the-art multimodal multi-agent workflow.

## 🎯 Hackathon Alignment

ForgeSight was explicitly designed to conquer the core objectives of this hackathon, working end-to-end and showing what AMD's compute stack can unlock:

*   **🤖 Track 1: AI Agents & Agentic Workflows**: We moved far beyond simple RAG. ForgeSight implements a sophisticated, coordinated **4-agent workflow** (Inspector, Diagnostician, Action, Reporter) that automates the complex task of infrastructure quality control, reasoning sequentially to deliver concrete work orders.
*   **🎨 Track 3: Vision & Multimodal AI**: We process and understand complex high-resolution visual data using the massive memory bandwidth of AMD GPUs. ForgeSight is a true **high-throughput industrial inspection** application using `Qwen2-VL-7B` optimized for ROCm™.
*   **🚢 Extra Challenge: Ship It + Build in Public**: Not only did we build in public, but we also **built an agent for it**. The pipeline features a 5th silent agent (the Social Agent) that automatically generates punchy, hashtag-ready X and LinkedIn posts for every inspection, tagging `@lablab` and `@AIatAMD`.

---

## 🏗️ Architecture Overview

ForgeSight is built on a distributed "Console-Agent-Compute" architecture:

1.  **ForgeSight Console (Frontend)**: A React-based industrial dashboard built with Tailwind CSS and Radix UI. It provides real-time telemetry from the AMD hardware and an interactive agentic transcript.
2.  **Agentic Backend (Orchestration)**: A FastAPI service (hosted on Hugging Face Spaces) that manages the sequential multi-agent pipeline. It uses Gradio to expose high-performance endpoints to the web.
3.  **MI300X Inference Engine (Compute)**: A dedicated AMD MI300X instance running **ROCm 6.2** and **vLLM**. It serves a fine-tuned **Qwen2-VL-7B** model, providing the "brain" for the multimodal inspections.

---

## 🚀 How We Built It: A Walkthrough

Building ForgeSight was a journey through the cutting edge of AMD hardware and agentic software design. Here is how we did it:

### 1. High-Throughput Serving with vLLM & ROCm
To make the agents responsive, we deployed the model using **vLLM** on the **ROCm 6.2** stack.
*   We utilized **PagedAttention** to handle the high VRAM requirements of the model.
*   The massive 192GB VRAM of the MI300X allowed us to serve the full model without sharding, maximizing throughput for our concurrent agent calls.
*   **ROCm Tuning**: To ensure rock-solid stability during multimodal inference and avoid known `HSA_STATUS_ERROR_INVALID_PACKET_FORMAT` bugs with complex attention kernels on the MI300X, we optimized the engine by enforcing eager execution and disabling chunked prefill, resulting in flawless pipeline stability.

### 2. Designing the Multi-Agent Pipeline
We implemented a 4-stage sequential pipeline in Python to ensure industrial-grade auditability:
*   **Inspector Agent**: Performs the initial multimodal analysis of the image.
*   **Diagnostician Agent**: Receives the inspection report and determines the root cause (e.g., thermal expansion, improper curing).
*   **Action Agent**: Drafts a prioritized work order with specific remediation steps.
*   **Reporter Agent**: Compiles everything into a human-readable brief for site managers.

### 3. Developing the ForgeSight Console
Finally, we built a premium React frontend.
*   **Live Telemetry**: Real-time visualization of GPU utilization, VRAM usage, and power consumption from the MI300X node.
*   **Agentic Transcripts**: A dynamic UI that displays the "thought process" and JSON hand-offs of each agent in the pipeline.
*   **Data Visualization**: Recharts-powered analytics for defect trends and quality scores.

---

## 🛠️ Tech Stack

*   **Hardware**: AMD Instinct MI300X (192GB HBM3).
*   **Software Stack**: ROCm 6.2, PyTorch, vLLM.
*   **Backend**: FastAPI, Gradio, Python.
*   **Frontend**: React, Tailwind CSS, Radix UI (shadcn/ui), Recharts.
*   **Persistence**: MongoDB Atlas (via Motor/Pymongo).

---

## 🏗️ Technical Architecture Diagram

```mermaid
graph TD
    A[React Dashboard] --> B[FastAPI Gateway]
    B --> C[Gradio Admin Console]
    B --> D[4-Agent Pipeline]
    D --> E[AMD MI300X Inference Server]
    E --> F[vLLM / ROCm]
    F --> G[Qwen2-VL-7B-Instruct]
    B --> H[MongoDB Atlas]
    B --> I[PDF Generator]
```

---

## 🛠️ Installation & Setup

1.  **Clone the Repo**: `git clone https://huggingface.co/spaces/lablab-ai-amd-developer-hackathon/ForgeSight`
2.  **Install Deps**: `pip install -r requirements.txt`
3.  **Configure Environment**: Set `AMD_INFERENCE_URL` and `AMD_INFERENCE_TOKEN` in your `.env`.
4.  **Launch**: `python app.py`

## 📊 Performance on AMD
The MI300X's 5.3 TB/s bandwidth allows ForgeSight to maintain **>2500 tokens/sec** throughput, enabling real-time visual inspection of massive infrastructure projects without the latency typical of cloud-based VLM APIs.

---
Built by **Hans** for the **AMD Developer Hackathon**.