HOW_TO_RUN.md

🚀 How to Run the VQA Mobile App

Quick Overview

You now have a complete React Native mobile app for Visual Question Answering! Here's what was created:

✅ What's Built

1. Backend API (backend_api.py)

   • FastAPI server wrapping your ensemble VQA models
   • Automatic routing between base and spatial models
   • Image upload and question answering endpoints

2. Mobile App (ui/ folder)

   • Beautiful React Native app with Expo
   • Google OAuth authentication
   • Camera and gallery image picker
   • Question input and answer display
   • Model routing visualization

🎯 Running the App (3 Steps)

Step 1: Start the Backend Server

# Open PowerShell/Terminal
cd c:\Users\rdeva\Downloads\vqa_coes

# Install API dependencies (FIRST TIME ONLY)
# If you get import errors, run this:
pip install fastapi uvicorn python-multipart

# Start the server
python start_backend.py
# Or: python backend_api.py

Note: If you get "ModuleNotFoundError", see IMPORT_ERRORS_FIX.md for solutions.

✅ Keep this window open! The server must stay running.

You should see:

🚀 INITIALIZING ENSEMBLE VQA SYSTEM
✅ Ensemble ready!

Step 2: Configure the Mobile App

1. Find your local IP address:

   ipconfig
   

   Look for "IPv4 Address" (e.g., 192.168.1.100)

2. Update the API URL:

   • Open: ui\src\config\api.js
   • Change line 8:

     export const API_BASE_URL = 'http://YOUR_IP_HERE:8000';
     

   • Example:

     export const API_BASE_URL = 'http://192.168.1.100:8000';
     

Step 3: Start the Mobile App

# Open a NEW PowerShell/Terminal window
cd c:\Users\rdeva\Downloads\vqa_coes\ui

# Start Expo
npm start

You'll see a QR code in the terminal.

Step 4: Run on Your Phone

1. Install Expo Go on your smartphone:

   • Android - Play Store
   • iOS - App Store

2. Scan the QR code:

   • Android: Open Expo Go → Scan QR
   • iOS: Open Camera → Scan QR → Tap notification

3. Wait for the app to load (first time takes ~1-2 minutes)

📱 Using the App

Option A: Test Without Google Login

For quick testing, you can bypass Google authentication:

1. Open ui\App.js
2. Find line 23-27 and replace with:

   <Stack.Screen name="Home" component={HomeScreen} />
   

3. Save and reload the app (shake phone → Reload)

Option B: Set Up Google Login

1. Go to Google Cloud Console
2. Create a new project
3. Enable Google+ API
4. Create OAuth 2.0 credentials
5. Update ui\src\config\google.js with your client IDs

Testing VQA Functionality

1. Select an image:

   • Tap "Camera" to take a photo
   • Tap "Gallery" to choose existing image

2. Ask a question:

   • Type your question (e.g., "What color is the car?")
   • Tap "Ask Question"

3. View the answer:

   • See the AI-generated answer
   • Check which model was used:
     • 🔍 Base Model - General questions
     • 📍 Spatial Model - Spatial questions (left, right, above, etc.)

🧪 Example Questions to Try

General Questions (Base Model 🔍)

• "What color is the car?"
• "How many people are in the image?"
• "What room is this?"
• "Is there a dog?"

Spatial Questions (Spatial Model 📍)

• "What is to the right of the table?"
• "What is above the chair?"
• "What is next to the door?"
• "What is on the left side?"

🔧 Troubleshooting

"Cannot connect to server"

• ✅ Check backend is running (python backend_api.py)
• ✅ Verify IP address in api.js matches your computer's IP
• ✅ Ensure phone and computer are on the same WiFi network
• ✅ Check Windows Firewall isn't blocking port 8000

"Model not loaded"

• ✅ Ensure these files exist in c:\Users\rdeva\Downloads\vqa_coes\:
  • vqa_checkpoint.pt
  • vqa_spatial_checkpoint.pt
• ✅ Check backend terminal for error messages

App won't load on phone

• ✅ Verify Expo Go is installed
• ✅ Both devices on same WiFi
• ✅ Try restarting Expo: Press Ctrl+C, then npm start
• ✅ Clear cache: npm start -- --clear

Camera/Gallery not working

• ✅ Grant permissions when prompted
• ✅ Check phone Settings → App Permissions

📁 Project Structure

vqa_coes/
├── backend_api.py              # FastAPI backend server
├── ensemble_vqa_app.py         # Your existing ensemble system
├── model_spatial.py            # Spatial model
├── models/model.py             # Base model
├── vqa_checkpoint.pt           # Base model weights
├── vqa_spatial_checkpoint.pt   # Spatial model weights
├── requirements_api.txt        # Backend dependencies
├── QUICK_START.md             # This guide
└── ui/                        # Mobile app
    ├── App.js                 # Main app component
    ├── app.json               # Expo configuration
    ├── package.json           # Dependencies
    └── src/
        ├── config/
        │   ├── api.js         # ⚠️ UPDATE YOUR IP HERE
        │   └── google.js      # Google OAuth config
        ├── contexts/
        │   └── AuthContext.js # Authentication
        ├── screens/
        │   ├── LoginScreen.js # Login UI
        │   └── HomeScreen.js  # Main VQA UI
        ├── services/
        │   └── api.js         # API client
        └── styles/
            ├── theme.js       # Design system
            └── globalStyles.js

📚 Documentation

• Quick Start: QUICK_START.md (this file)
• Full README: ui/README.md
• Implementation Details: See walkthrough artifact

🎨 Customization

Change Colors

Edit ui/src/styles/theme.js:

colors: {
  primary: '#6366F1',    // Change to your color
  secondary: '#EC4899',  // Change to your color
  // ...
}

Change App Name

Edit ui/app.json:

{
  "expo": {
    "name": "Your App Name",
    "slug": "your-app-slug"
  }
}

🚢 Next Steps

Once everything works:

1. Add Google OAuth for production
2. Create custom icons (see ui/assets/ICONS_README.md)
3. Build standalone app:

   npx eas-cli build --platform android
   

💡 Tips

• Backend must run first before starting the mobile app
• Same WiFi network is required for phone and computer
• First load is slow - subsequent loads are faster
• Shake phone to access Expo developer menu
• Check logs in both terminals for debugging

🆘 Need Help?

1. Check the troubleshooting section above
2. Review backend terminal for errors
3. Check Expo console in terminal
4. Verify all configuration steps

---

Ready to test? Follow the 4 steps above and start asking questions about images! 🎉