methodology.md

# Methodology

The chatbot integrates multiple AI workflows into a single Gradio UI. The process follows these main stages:

## Input Handling

Users interact via a multimodal text box (supports text, image, and audio).

The chatbot determines whether the query contains:

Text only

Image file

Audio file

## Intent Classification

Text queries are processed through a rule-based intent classifier (intents.json).

Example intents:

"chat" → Send to hosted chatbot LLM.

"search_local_image" → Trigger local semantic image search.

"request_image_analysis" → Ask user to upload an image.

"request_audio_analysis" → Ask user to upload audio.

## Local Semantic Search

Metadata from image.json provides descriptions for images in /images/.

Each description is encoded using SentenceTransformers (all-MiniLM-L6-v2).

Query embeddings are compared with stored embeddings using cosine similarity.

If similarity > threshold (0.4), best match image is returned.

## Image Analysis Workflow

Uploaded images are passed to the vision model (via gradio_client).

Raw AI output (JSON) is summarized with Groq API (LLaMA-3.3-70B).

Final user-facing response is a friendly explanation.

## Audio Analysis Workflow

Uploaded audio is processed via the audio model (Gradio client).

Returns prediction text (e.g., transcription or classification).

Packaged as a human-readable response.

## Groq Summarization

Any complex JSON output (e.g., image analysis) is summarized.

A system prompt guides Groq to produce short, user-friendly summaries.

Ensures technical data is explained in simple language.

## Conversation Management

All interactions are stored in Chatbot history.

User query + bot response pairs are maintained for continuity.

Multimodal interactions (e.g., image + explanation) are rendered in chat.

## Architecture at a Glance

User Input (Text / Image / Audio)


        │
        
        ▼
        
Intent Classifier ──► Rule-based (intents.json)

        │
        
        ├─ Chat → Chatbot Client (LLM)
        
        ├─ Search Local Image → Embedding Match
        
        ├─ Image Analysis → Vision Client + Groq Summary
        
        └─ Audio Analysis → Audio Client
        
        ▼
        
Response Generator (Groq Narrative + History)

        ▼
        
Gradio Chat UI