Spaces:
Sleeping
Sleeping
| title: Room Visualizer API | |
| emoji: 🛋️ | |
| colorFrom: indigo | |
| colorTo: blue | |
| sdk: docker | |
| app_port: 7860 | |
| pinned: false | |
| # Room Visualizer AI | |
| Upload room photos, pick **one**, browse a local furniture catalog, select items, and | |
| get a **realistic AI-generated image of that room with the selected real catalog | |
| furniture correctly placed and scaled into it.** | |
| The placement is not a guess: a Shapely + perspective pipeline decides *where* an item | |
| goes and *how big* it should be from its real-world centimetre dimensions, then an | |
| image-conditioned inpaint composites the actual catalog product image into that region. | |
| --- | |
| ## Two ways to generate | |
| Reference-image placement needs **image-conditioned inpainting** (paint a *specific | |
| product image* into a *masked region* of a *specific room photo*). Generation is hidden | |
| behind one interface (`ImageGenProvider`) with two implementations: | |
| | Provider | What it does | Needs | | |
| |---|---|---| | |
| | **`mock`** (default) | Composites the catalog image onto the room at the computed mask, scaled & bottom-aligned, with a "MOCK" stamp. Deterministic. | **Nothing** — no API key, no network, no GPU | | |
| | **`replicate`** | Real SDXL inpaint + IP-Adapter on cloud GPUs. | `REPLICATE_API_TOKEN` + a model id + an international payment method | | |
| Switching is a **one-line env change** (`IMAGE_PROVIDER=mock|replicate`) — no code changes | |
| anywhere else. If `replicate` is selected but unavailable, the app falls back to `mock` | |
| (never to a different placement strategy). | |
| > **`IMAGE_PROVIDER=mock` runs the entire app end-to-end with no API key and no external | |
| > network.** That is the recommended path for local dev and review. | |
| --- | |
| ## Repository layout | |
| ``` | |
| room-visualizer-ai/ | |
| ├── backend/ # FastAPI + the analysis/placement pipeline | |
| │ ├── app/ | |
| │ │ ├── main.py # routes, CORS, static mounts | |
| │ │ ├── config.py db.py models.py schemas.py pipeline.py | |
| │ │ ├── modules/ # input_preprocess, detection, depth, segmentation, | |
| │ │ │ # floor_plane, spatial (Shapely), scaling, masking, recommend | |
| │ │ └── providers/ # base (interface), mock_provider, replicate_provider, factory | |
| │ ├── data/catalog.json # 36 items, exact schema | |
| │ ├── scripts/generate_catalog_images.py | |
| │ ├── static/{catalog,uploads,outputs}/ | |
| │ ├── requirements.txt # LIGHT core (runs the whole app in mock mode) | |
| │ └── requirements-ml.txt # OPTIONAL heavy stack (real YOLO/MiDaS/SAM) | |
| ├── frontend/ # Next.js (App Router) WEB client + TypeScript | |
| │ ├── app/ components/ lib/api.ts | |
| ├── mobile/ # Expo (React Native) MOBILE client + TypeScript | |
| │ ├── App.tsx src/api.ts src/components/ | |
| └── README.md | |
| ``` | |
| Two clients ship against the **same** FastAPI backend: a Next.js web app and an | |
| Expo (React Native) mobile app. Pick whichever you want to run — the backend and | |
| the whole analysis/placement pipeline are identical for both. | |
| --- | |
| ## Prerequisites | |
| - **Python 3.11** (recommended — best wheel compatibility for the optional ML stack). | |
| - **Node.js 18+** (any modern LTS; Node 20/22/25 all fine). | |
| - That's it for `mock` mode. Real generation additionally needs a Replicate token; real | |
| ML *analysis* additionally needs the heavy stack in `requirements-ml.txt`. | |
| - **For the mobile app:** the **Expo Go** app on your phone (iOS/Android), with the phone | |
| on the **same Wi-Fi** as your computer. (An Android emulator works too.) | |
| --- | |
| ## 1) Backend — setup & run | |
| ```bash | |
| cd backend | |
| # create + activate a virtual environment | |
| python -m venv .venv | |
| # Windows (PowerShell): | |
| .venv\Scripts\Activate.ps1 | |
| # macOS/Linux: | |
| # source .venv/bin/activate | |
| # install the LIGHT core deps (enough for full mock-mode flow) | |
| pip install -r requirements.txt | |
| # render the placeholder catalog product images into static/catalog/ | |
| python scripts/generate_catalog_images.py | |
| # configure (mock is the default even without a .env) | |
| cp .env.example .env # Windows: copy .env.example .env | |
| # run the API | |
| uvicorn app.main:app --reload --port 8000 | |
| ``` | |
| Backend is now at **http://127.0.0.1:8000** (open `/docs` for interactive API docs). | |
| > On Windows, if `python` opens the Microsoft Store, use the full interpreter path to | |
| > create the venv, e.g. | |
| > `C:\Users\<you>\AppData\Local\Programs\Python\Python311\python.exe -m venv .venv`. | |
| > After activating the venv, `python`/`pip` resolve correctly. | |
| ## 2) Frontend — setup & run | |
| ```bash | |
| cd frontend | |
| npm install | |
| cp .env.local.example .env.local # Windows: copy .env.local.example .env.local | |
| npm run dev | |
| ``` | |
| Frontend is now at **http://localhost:3000**. It talks to the backend at | |
| `NEXT_PUBLIC_API_BASE` (defaults to `http://127.0.0.1:8000`). | |
| ## 3) Mobile app (Expo / React Native) — setup & run | |
| The mobile client uses the **same backend**. The only extra requirement is that your | |
| phone and computer are on the **same Wi-Fi** and the backend is reachable from the phone. | |
| ```bash | |
| cd mobile | |
| npm install | |
| npx expo start # then scan the QR code with Expo Go | |
| ``` | |
| **Make the backend reachable from your phone:** | |
| 1. Bind the backend to all interfaces (not just localhost): | |
| ```bash | |
| cd backend | |
| uvicorn app.main:app --host 0.0.0.0 --port 8000 | |
| ``` | |
| 2. The app **auto-detects your computer's LAN IP** from the Expo dev server, so it targets | |
| `http://<your-computer-ip>:8000` with no config. The resolved URL is shown at the top of | |
| the app. To override (e.g. a deployed backend), set `EXPO_PUBLIC_API_BASE`: | |
| ```bash | |
| # mobile/.env | |
| EXPO_PUBLIC_API_BASE=http://192.168.1.50:8000 | |
| ``` | |
| 3. If your OS firewall prompts, allow Python/uvicorn on the **private** network. | |
| Run targets: scan with **Expo Go** (Android/iOS), or press `a` for an Android emulator | |
| (`i`/iOS simulator needs macOS), or `npm run web` for a browser preview. The app requests | |
| photo-library/camera permissions (to pick room photos) and photo-save permission (to save | |
| the generated room); these are declared in `mobile/app.json`. | |
| > Same five steps as the web app: upload 3–10 photos → pick one → analyze → choose | |
| > furniture → generate. Mock mode works identically on mobile. | |
| ### Build a standalone APK (Android) | |
| The project is already prebuilt for native Android (`mobile/android/`) and the release | |
| variant is signed with the debug keystore (installable; not a Play-Store release). | |
| **Requirement — build with JDK 17.** Expo SDK 56 ships a Gradle 9.3.1 wrapper whose Java | |
| toolchain auto-download is broken; building on JDK 21 crashes (`NoSuchFieldError: IBM_SEMERU`). | |
| A JDK 17 is installed at `C:\Users\Hanzala\jdk17\jdk-17.0.19+10`, and | |
| `org.gradle.java.installations.auto-download=false` is set in `android/gradle.properties`. | |
| 1. A standalone APK can't auto-detect the backend, so the URL is baked in at build time via | |
| `EXPO_PUBLIC_API_BASE`. Find your PC's Wi-Fi IPv4 with `ipconfig`. | |
| 2. Build (PowerShell): | |
| ```powershell | |
| $env:JAVA_HOME="C:\Users\Hanzala\jdk17\jdk-17.0.19+10" | |
| $env:ANDROID_HOME="C:\Users\Hanzala\AppData\Local\Android\Sdk" | |
| $env:EXPO_PUBLIC_API_BASE="http://192.168.51.176:8000" # your PC IP | |
| cd C:\Users\Hanzala\room-visualizer-ai\mobile\android | |
| .\gradlew.bat assembleRelease -x lint | |
| ``` | |
| 3. APK output: `mobile/android/app/build/outputs/apk/release/app-release.apk` (~71 MB, all ABIs). | |
| Or in **Android Studio**: open `mobile/android`, set Settings → Build, Execution, Deployment → | |
| Build Tools → Gradle → **Gradle JDK = 17**, choose the **release** build variant, then | |
| Build → Build APK(s). | |
| **Install & run:** copy the APK to the phone and open it (allow "install from unknown | |
| sources"), or `adb install app-release.apk`. Then run the backend with | |
| `uvicorn app.main:app --host 0.0.0.0 --port 8000` and keep the phone on the **same Wi-Fi**. | |
| If your PC's IP changes, rebuild with the new `EXPO_PUBLIC_API_BASE` (fast — Gradle is cached). | |
| --- | |
| ## End-to-end walkthrough | |
| 1. **Upload** 3–10 room photos (drag/drop or browse). | |
| 2. **Choose** exactly one photo to redesign. | |
| 3. **Analyze** — see detected objects, the Shapely usable-floor polygon, blocked areas, | |
| and the free-space percentage overlaid on the photo. | |
| 4. **Pick furniture** from the catalog (filter by category/style; tag-matched items are | |
| ranked first). Select one or more items. | |
| 5. **Generate** — synchronous; a loading state shows during the call, then the final | |
| image appears with a **Download** button. | |
| --- | |
| ## Environment variables | |
| | Var | Default | Purpose | | |
| |---|---|---| | |
| | `IMAGE_PROVIDER` | `mock` | `mock` or `replicate` | | |
| | `REPLICATE_API_TOKEN` | — | required for `replicate` | | |
| | `REPLICATE_MODEL` | — | `owner/model:version` of an SDXL-inpaint + IP-Adapter model | | |
| | `REPLICATE_INPUT_*_KEY` | sensible defaults | override a model's input field names | | |
| | `ENABLE_ML` | `auto` | `auto` (ML if `torch` present), `true`, or `false` | | |
| | `YOLO_MODEL` / `MIDAS_MODEL` | `yolov8n.pt` / `MiDaS_small` | lightweight CPU-friendly variants | | |
| | `SAM_CHECKPOINT` / `SAM_MODEL_TYPE` | — / `vit_b` | optional real SAM floor isolation | | |
| | `ROOM_FLOOR_WIDTH_CM` | `360` | real floor width assumed at the image bottom (drives cm→px scaling) | | |
| | `HORIZON_FRAC`, `DEPTH_FORESHORTEN` | `0.5`, `0.55` | perspective heuristics | | |
| | `MIN_IMAGES` / `MAX_IMAGES` / `MAX_FILE_MB` / `MAX_EDGE_PX` | `3` / `10` / `15` / `1536` | upload limits & normalization | | |
| | `HOST` / `PORT` / `FRONTEND_ORIGIN` | `127.0.0.1` / `8000` / `http://localhost:3000` | server & CORS | | |
| ## Where files land | |
| - Uploaded (normalized) photos → `backend/static/uploads/` | |
| - Generated rooms + depth previews → `backend/static/outputs/` | |
| - Catalog product images → `backend/static/catalog/` | |
| - Session/image/generation metadata → `backend/app.db` (SQLite) | |
| All are served under `/static`. | |
| --- | |
| ## Optional: real ML room analysis | |
| The analysis modules (detection/depth/segmentation) **lazy-load** their models and | |
| degrade gracefully to OpenCV/geometric fallbacks. To enable the real models: | |
| ```bash | |
| pip install -r requirements-ml.txt # torch (CPU by default), ultralytics, timm, segment-anything | |
| # ENABLE_ML=auto already turns them on once torch is importable | |
| ``` | |
| - **YOLOv8** detection and **MiDaS** depth then run for real (CPU is fine, just slower). | |
| - **SAM** floor isolation activates only if you also set `SAM_CHECKPOINT` to a downloaded | |
| checkpoint (e.g. `sam_vit_b_01ec64.pth`); otherwise a robust geometric floor estimate is | |
| used. The Shapely placement + perspective scaling + mask construction always run for real | |
| regardless of ML. | |
| ## Optional: real Replicate generation | |
| ```bash | |
| # in backend/.env | |
| IMAGE_PROVIDER=replicate | |
| REPLICATE_API_TOKEN=r8_xxx | |
| REPLICATE_MODEL=owner/sdxl-inpaint-ip-adapter:<version-hash> | |
| ``` | |
| Pick a model that accepts an image + mask + reference (IP-Adapter) image. If its input | |
| field names differ, set the `REPLICATE_INPUT_*_KEY` vars — no code changes needed. | |
| --- | |
| ## Honest environment notes | |
| - **Self-hosting the generation model (SDXL + IP-Adapter) needs an NVIDIA GPU** with | |
| ~8–12 GB VRAM. On a laptop with integrated graphics it is not viable — use the | |
| `replicate` provider (cloud GPU) for real output, or `mock` for everything else. | |
| - Real ML *analysis* (YOLO/MiDaS/SAM) runs on CPU; it's correct, just slower. Use the | |
| configured lightweight variants. | |
| - Pay-per-call providers bill in **USD** and need a working international payment method. | |
| The `mock` path means development is never blocked by this. | |
| --- | |
| ## API endpoints | |
| | Method | Path | Purpose | | |
| |---|---|---| | |
| | `POST` | `/upload-room` | accept 3–10 images, preprocess (OpenCV), store, return session + image ids | | |
| | `POST` | `/analyze-room` | `{session_id, image_id}` → detected objects, usable-floor polygon, blocked areas, free space | | |
| | `GET` | `/get-catalog` | catalog JSON; `?category=&styles=a,b` filtering + recommendation ordering | | |
| | `POST` | `/generate-room` | `{session_id, image_id, item_ids[]}` → generated image URL (synchronous) | | |
| There is **no `/remove-object`** endpoint — object removal is intentionally out of scope. | |
| ## Testing pipeline modules independently | |
| Every stage is importable on its own, e.g.: | |
| ```python | |
| from app.modules.floor_plane import estimate_floor_plane | |
| from app.modules.spatial import compute_spaces, select_placement | |
| from app.modules.scaling import footprint_quad | |
| fp = estimate_floor_plane("static/uploads/<id>.jpg") | |
| ``` | |
| --- | |
| ## Scope (locked) | |
| **In:** furniture *addition* via reference-image placement; gallery upload with one | |
| redesign target; Shapely + perspective placement; rule-based recommendations; swappable | |
| provider with mock + Replicate; SQLite metadata; synchronous generation. | |
| **Out (by design):** object removal (any form); multi-view fusion / 3D reconstruction / | |
| photogrammetry / NeRF; ML-trained recommendations; text-prompt invention of furniture. | |