DOCUMENTATION.md

AI Copilot Backend & Pipeline Management

Multi-Modal Continuous Learning Pipeline

This repository handles the intensive GPU backend for the Diagnostic Co-Pilot web application.

Rather than sending 100% of our patient inputs to generic APIs (like Gemini or OpenAI), this ecosystem trains a tailored Medical Vision-Language Model efficiently on limited memory (like Kaggle's T4x2 GPU).

Architecture Components:

1. app.py: A Hugging Face Space Gradio inference API. It hosts our specialized code and safely answers WebSocket requests from our main React frontend with heavily formatted Multi-Modal Diagnostics.
2. train_multimodal.py: The fully automated Python training pipeline. It clones this repository inside a runtime environment (like Kaggle), dynamically loads Qwen2-VL into an explosive 4-bit footprint, slices off a tiny Low-Rank Adaptation (LoRA) layer (adapter_model.safetensors), trains it on medical images (flaviagiammarino/vqa-rad), and automatically pushes the weights back out to the Hugging Face Hub.
3. .github/workflows/sync_to_hub.yml: Automates pushing GitHub commits straight to Hugging Face Spaces seamlessly!

Setting Up Your Ecosystem:

Part 1: Linking GitHub to Hugging Face

Any time you update the app.py code here, you want it to push to Hugging Face instantly:

1. Log into your Hugging Face Settings -> Access Tokens and generate a Write Token (e.g. hf_...).
2. Go to your GitHub repository -> Settings -> Secrets and Variables -> Actions -> New Repository Secret. Name it HF_TOKEN and paste the token from Step 1.
3. Create an empty Hugging Face Space: Head over to Hugging Face Spaces, Create a New Space called diagnostic-copilot-api, specify Gradio SDK, and leave it blank. Wait 1 min, and GitHub will auto-sync.

Part 2: Training the Multimodal AI (Kaggle)

Because this relies on heavy GPU access and requires careful navigation around Kaggle's internal root, use the exact sequence of code below inside your Notebook.

1. Add a Secret to your Kaggle Notebook named HF_TOKEN and input the write token you generated above. toggle it ON.
2. Add a T4x2 GPU accelerator to the environment.
3. Run this exact sequence of Python inside a single code cell:

import os
import shutil

# 1. ALWAYS start at the absolute global root of the container
os.chdir('/kaggle/working')

# 2. Clean out old attempts safely (forces a fresh sync with GitHub updates)
if os.path.exists('diagnostic-copilot-model'):
    shutil.rmtree('diagnostic-copilot-model')

# 3. Clone fresh repository
!git clone https://github.com/hssling/diagnostic-copilot-model.git

# 4. Install requirements from the newly downloaded folder
!pip install -r /kaggle/working/diagnostic-copilot-model/requirements.txt

# 5. Enter directory and run the fully automated training script!
os.chdir('/kaggle/working/diagnostic-copilot-model')
!python train_multimodal.py

What to expect from Kaggle

The notebook will execute the training run. You will see progress bars from PyTorch indicating loss reduction over max_steps bounds. Upon hitting 100%, the automated python script will intercept your Kaggle Secret, package the med-qwen-vl-adapter module, and silently upload it directly to your live Hugging Face Account.

Wait about 1 Minute. Connect the React frontend Settings to hf-space:hssling/diagnostic-copilot-api. Your bespoke medical multimodal model is now receiving diagnostic scans securely from the web!