--- title: Room Visualizer API emoji: 🛋️ colorFrom: indigo colorTo: blue sdk: docker app_port: 7860 pinned: false --- # Room Visualizer AI Upload room photos, pick **one**, browse a local furniture catalog, select items, and get a **realistic AI-generated image of that room with the selected real catalog furniture correctly placed and scaled into it.** The placement is not a guess: a Shapely + perspective pipeline decides *where* an item goes and *how big* it should be from its real-world centimetre dimensions, then an image-conditioned inpaint composites the actual catalog product image into that region. --- ## Two ways to generate Reference-image placement needs **image-conditioned inpainting** (paint a *specific product image* into a *masked region* of a *specific room photo*). Generation is hidden behind one interface (`ImageGenProvider`) with two implementations: | Provider | What it does | Needs | |---|---|---| | **`mock`** (default) | Composites the catalog image onto the room at the computed mask, scaled & bottom-aligned, with a "MOCK" stamp. Deterministic. | **Nothing** — no API key, no network, no GPU | | **`replicate`** | Real SDXL inpaint + IP-Adapter on cloud GPUs. | `REPLICATE_API_TOKEN` + a model id + an international payment method | Switching is a **one-line env change** (`IMAGE_PROVIDER=mock|replicate`) — no code changes anywhere else. If `replicate` is selected but unavailable, the app falls back to `mock` (never to a different placement strategy). > **`IMAGE_PROVIDER=mock` runs the entire app end-to-end with no API key and no external > network.** That is the recommended path for local dev and review. --- ## Repository layout ``` room-visualizer-ai/ ├── backend/ # FastAPI + the analysis/placement pipeline │ ├── app/ │ │ ├── main.py # routes, CORS, static mounts │ │ ├── config.py db.py models.py schemas.py pipeline.py │ │ ├── modules/ # input_preprocess, detection, depth, segmentation, │ │ │ # floor_plane, spatial (Shapely), scaling, masking, recommend │ │ └── providers/ # base (interface), mock_provider, replicate_provider, factory │ ├── data/catalog.json # 36 items, exact schema │ ├── scripts/generate_catalog_images.py │ ├── static/{catalog,uploads,outputs}/ │ ├── requirements.txt # LIGHT core (runs the whole app in mock mode) │ └── requirements-ml.txt # OPTIONAL heavy stack (real YOLO/MiDaS/SAM) ├── frontend/ # Next.js (App Router) WEB client + TypeScript │ ├── app/ components/ lib/api.ts ├── mobile/ # Expo (React Native) MOBILE client + TypeScript │ ├── App.tsx src/api.ts src/components/ └── README.md ``` Two clients ship against the **same** FastAPI backend: a Next.js web app and an Expo (React Native) mobile app. Pick whichever you want to run — the backend and the whole analysis/placement pipeline are identical for both. --- ## Prerequisites - **Python 3.11** (recommended — best wheel compatibility for the optional ML stack). - **Node.js 18+** (any modern LTS; Node 20/22/25 all fine). - That's it for `mock` mode. Real generation additionally needs a Replicate token; real ML *analysis* additionally needs the heavy stack in `requirements-ml.txt`. - **For the mobile app:** the **Expo Go** app on your phone (iOS/Android), with the phone on the **same Wi-Fi** as your computer. (An Android emulator works too.) --- ## 1) Backend — setup & run ```bash cd backend # create + activate a virtual environment python -m venv .venv # Windows (PowerShell): .venv\Scripts\Activate.ps1 # macOS/Linux: # source .venv/bin/activate # install the LIGHT core deps (enough for full mock-mode flow) pip install -r requirements.txt # render the placeholder catalog product images into static/catalog/ python scripts/generate_catalog_images.py # configure (mock is the default even without a .env) cp .env.example .env # Windows: copy .env.example .env # run the API uvicorn app.main:app --reload --port 8000 ``` Backend is now at **http://127.0.0.1:8000** (open `/docs` for interactive API docs). > On Windows, if `python` opens the Microsoft Store, use the full interpreter path to > create the venv, e.g. > `C:\Users\\AppData\Local\Programs\Python\Python311\python.exe -m venv .venv`. > After activating the venv, `python`/`pip` resolve correctly. ## 2) Frontend — setup & run ```bash cd frontend npm install cp .env.local.example .env.local # Windows: copy .env.local.example .env.local npm run dev ``` Frontend is now at **http://localhost:3000**. It talks to the backend at `NEXT_PUBLIC_API_BASE` (defaults to `http://127.0.0.1:8000`). ## 3) Mobile app (Expo / React Native) — setup & run The mobile client uses the **same backend**. The only extra requirement is that your phone and computer are on the **same Wi-Fi** and the backend is reachable from the phone. ```bash cd mobile npm install npx expo start # then scan the QR code with Expo Go ``` **Make the backend reachable from your phone:** 1. Bind the backend to all interfaces (not just localhost): ```bash cd backend uvicorn app.main:app --host 0.0.0.0 --port 8000 ``` 2. The app **auto-detects your computer's LAN IP** from the Expo dev server, so it targets `http://:8000` with no config. The resolved URL is shown at the top of the app. To override (e.g. a deployed backend), set `EXPO_PUBLIC_API_BASE`: ```bash # mobile/.env EXPO_PUBLIC_API_BASE=http://192.168.1.50:8000 ``` 3. If your OS firewall prompts, allow Python/uvicorn on the **private** network. Run targets: scan with **Expo Go** (Android/iOS), or press `a` for an Android emulator (`i`/iOS simulator needs macOS), or `npm run web` for a browser preview. The app requests photo-library/camera permissions (to pick room photos) and photo-save permission (to save the generated room); these are declared in `mobile/app.json`. > Same five steps as the web app: upload 3–10 photos → pick one → analyze → choose > furniture → generate. Mock mode works identically on mobile. ### Build a standalone APK (Android) The project is already prebuilt for native Android (`mobile/android/`) and the release variant is signed with the debug keystore (installable; not a Play-Store release). **Requirement — build with JDK 17.** Expo SDK 56 ships a Gradle 9.3.1 wrapper whose Java toolchain auto-download is broken; building on JDK 21 crashes (`NoSuchFieldError: IBM_SEMERU`). A JDK 17 is installed at `C:\Users\Hanzala\jdk17\jdk-17.0.19+10`, and `org.gradle.java.installations.auto-download=false` is set in `android/gradle.properties`. 1. A standalone APK can't auto-detect the backend, so the URL is baked in at build time via `EXPO_PUBLIC_API_BASE`. Find your PC's Wi-Fi IPv4 with `ipconfig`. 2. Build (PowerShell): ```powershell $env:JAVA_HOME="C:\Users\Hanzala\jdk17\jdk-17.0.19+10" $env:ANDROID_HOME="C:\Users\Hanzala\AppData\Local\Android\Sdk" $env:EXPO_PUBLIC_API_BASE="http://192.168.51.176:8000" # your PC IP cd C:\Users\Hanzala\room-visualizer-ai\mobile\android .\gradlew.bat assembleRelease -x lint ``` 3. APK output: `mobile/android/app/build/outputs/apk/release/app-release.apk` (~71 MB, all ABIs). Or in **Android Studio**: open `mobile/android`, set Settings → Build, Execution, Deployment → Build Tools → Gradle → **Gradle JDK = 17**, choose the **release** build variant, then Build → Build APK(s). **Install & run:** copy the APK to the phone and open it (allow "install from unknown sources"), or `adb install app-release.apk`. Then run the backend with `uvicorn app.main:app --host 0.0.0.0 --port 8000` and keep the phone on the **same Wi-Fi**. If your PC's IP changes, rebuild with the new `EXPO_PUBLIC_API_BASE` (fast — Gradle is cached). --- ## End-to-end walkthrough 1. **Upload** 3–10 room photos (drag/drop or browse). 2. **Choose** exactly one photo to redesign. 3. **Analyze** — see detected objects, the Shapely usable-floor polygon, blocked areas, and the free-space percentage overlaid on the photo. 4. **Pick furniture** from the catalog (filter by category/style; tag-matched items are ranked first). Select one or more items. 5. **Generate** — synchronous; a loading state shows during the call, then the final image appears with a **Download** button. --- ## Environment variables | Var | Default | Purpose | |---|---|---| | `IMAGE_PROVIDER` | `mock` | `mock` or `replicate` | | `REPLICATE_API_TOKEN` | — | required for `replicate` | | `REPLICATE_MODEL` | — | `owner/model:version` of an SDXL-inpaint + IP-Adapter model | | `REPLICATE_INPUT_*_KEY` | sensible defaults | override a model's input field names | | `ENABLE_ML` | `auto` | `auto` (ML if `torch` present), `true`, or `false` | | `YOLO_MODEL` / `MIDAS_MODEL` | `yolov8n.pt` / `MiDaS_small` | lightweight CPU-friendly variants | | `SAM_CHECKPOINT` / `SAM_MODEL_TYPE` | — / `vit_b` | optional real SAM floor isolation | | `ROOM_FLOOR_WIDTH_CM` | `360` | real floor width assumed at the image bottom (drives cm→px scaling) | | `HORIZON_FRAC`, `DEPTH_FORESHORTEN` | `0.5`, `0.55` | perspective heuristics | | `MIN_IMAGES` / `MAX_IMAGES` / `MAX_FILE_MB` / `MAX_EDGE_PX` | `3` / `10` / `15` / `1536` | upload limits & normalization | | `HOST` / `PORT` / `FRONTEND_ORIGIN` | `127.0.0.1` / `8000` / `http://localhost:3000` | server & CORS | ## Where files land - Uploaded (normalized) photos → `backend/static/uploads/` - Generated rooms + depth previews → `backend/static/outputs/` - Catalog product images → `backend/static/catalog/` - Session/image/generation metadata → `backend/app.db` (SQLite) All are served under `/static`. --- ## Optional: real ML room analysis The analysis modules (detection/depth/segmentation) **lazy-load** their models and degrade gracefully to OpenCV/geometric fallbacks. To enable the real models: ```bash pip install -r requirements-ml.txt # torch (CPU by default), ultralytics, timm, segment-anything # ENABLE_ML=auto already turns them on once torch is importable ``` - **YOLOv8** detection and **MiDaS** depth then run for real (CPU is fine, just slower). - **SAM** floor isolation activates only if you also set `SAM_CHECKPOINT` to a downloaded checkpoint (e.g. `sam_vit_b_01ec64.pth`); otherwise a robust geometric floor estimate is used. The Shapely placement + perspective scaling + mask construction always run for real regardless of ML. ## Optional: real Replicate generation ```bash # in backend/.env IMAGE_PROVIDER=replicate REPLICATE_API_TOKEN=r8_xxx REPLICATE_MODEL=owner/sdxl-inpaint-ip-adapter: ``` Pick a model that accepts an image + mask + reference (IP-Adapter) image. If its input field names differ, set the `REPLICATE_INPUT_*_KEY` vars — no code changes needed. --- ## Honest environment notes - **Self-hosting the generation model (SDXL + IP-Adapter) needs an NVIDIA GPU** with ~8–12 GB VRAM. On a laptop with integrated graphics it is not viable — use the `replicate` provider (cloud GPU) for real output, or `mock` for everything else. - Real ML *analysis* (YOLO/MiDaS/SAM) runs on CPU; it's correct, just slower. Use the configured lightweight variants. - Pay-per-call providers bill in **USD** and need a working international payment method. The `mock` path means development is never blocked by this. --- ## API endpoints | Method | Path | Purpose | |---|---|---| | `POST` | `/upload-room` | accept 3–10 images, preprocess (OpenCV), store, return session + image ids | | `POST` | `/analyze-room` | `{session_id, image_id}` → detected objects, usable-floor polygon, blocked areas, free space | | `GET` | `/get-catalog` | catalog JSON; `?category=&styles=a,b` filtering + recommendation ordering | | `POST` | `/generate-room` | `{session_id, image_id, item_ids[]}` → generated image URL (synchronous) | There is **no `/remove-object`** endpoint — object removal is intentionally out of scope. ## Testing pipeline modules independently Every stage is importable on its own, e.g.: ```python from app.modules.floor_plane import estimate_floor_plane from app.modules.spatial import compute_spaces, select_placement from app.modules.scaling import footprint_quad fp = estimate_floor_plane("static/uploads/.jpg") ``` --- ## Scope (locked) **In:** furniture *addition* via reference-image placement; gallery upload with one redesign target; Shapely + perspective placement; rule-based recommendations; swappable provider with mock + Replicate; SQLite metadata; synchronous generation. **Out (by design):** object removal (any form); multi-view fusion / 3D reconstruction / photogrammetry / NeRF; ML-trained recommendations; text-prompt invention of furniture.