Hugging Face's logo

Spaces:
Johnntirs
/
room-visualizer-api
Sleeping

App Files Community
room-visualizer-api / README.md
Johnntirs's picture
Johnntirs
Room Visualizer backend (Docker)
0cdb4e8
preview code
|
Raw
History Blame Contribute Delete
13 kB
metadata 
title: Room Visualizer API
emoji: 🛋️
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

Room Visualizer AI

Upload room photos, pick one, browse a local furniture catalog, select items, and get a realistic AI-generated image of that room with the selected real catalog furniture correctly placed and scaled into it.

The placement is not a guess: a Shapely + perspective pipeline decides where an item goes and how big it should be from its real-world centimetre dimensions, then an image-conditioned inpaint composites the actual catalog product image into that region.

Two ways to generate

Reference-image placement needs image-conditioned inpainting (paint a specific product image into a masked region of a specific room photo). Generation is hidden behind one interface (ImageGenProvider) with two implementations:

Provider What it does Needs
mock (default) Composites the catalog image onto the room at the computed mask, scaled & bottom-aligned, with a "MOCK" stamp. Deterministic. Nothing — no API key, no network, no GPU
replicate Real SDXL inpaint + IP-Adapter on cloud GPUs. REPLICATE_API_TOKEN + a model id + an international payment method

Switching is a one-line env change (IMAGE_PROVIDER=mock|replicate) — no code changes anywhere else. If replicate is selected but unavailable, the app falls back to mock (never to a different placement strategy).

IMAGE_PROVIDER=mock runs the entire app end-to-end with no API key and no external network. That is the recommended path for local dev and review.

Repository layout 

room-visualizer-ai/
├── backend/                      # FastAPI + the analysis/placement pipeline
│   ├── app/
│   │   ├── main.py               # routes, CORS, static mounts
│   │   ├── config.py  db.py  models.py  schemas.py  pipeline.py
│   │   ├── modules/              # input_preprocess, detection, depth, segmentation,
│   │   │                         # floor_plane, spatial (Shapely), scaling, masking, recommend
│   │   └── providers/            # base (interface), mock_provider, replicate_provider, factory
│   ├── data/catalog.json         # 36 items, exact schema
│   ├── scripts/generate_catalog_images.py
│   ├── static/{catalog,uploads,outputs}/
│   ├── requirements.txt          # LIGHT core (runs the whole app in mock mode)
│   └── requirements-ml.txt       # OPTIONAL heavy stack (real YOLO/MiDaS/SAM)
├── frontend/                     # Next.js (App Router) WEB client + TypeScript
│   ├── app/  components/  lib/api.ts
├── mobile/                       # Expo (React Native) MOBILE client + TypeScript
│   ├── App.tsx  src/api.ts  src/components/
└── README.md

Two clients ship against the same FastAPI backend: a Next.js web app and an Expo (React Native) mobile app. Pick whichever you want to run — the backend and the whole analysis/placement pipeline are identical for both.

Prerequisites

  • Python 3.11 (recommended — best wheel compatibility for the optional ML stack).
  • Node.js 18+ (any modern LTS; Node 20/22/25 all fine).
  • That's it for mock mode. Real generation additionally needs a Replicate token; real ML analysis additionally needs the heavy stack in requirements-ml.txt.
  • For the mobile app: the Expo Go app on your phone (iOS/Android), with the phone on the same Wi-Fi as your computer. (An Android emulator works too.)

1) Backend — setup & run 

cd backend

# create + activate a virtual environment
python -m venv .venv
# Windows (PowerShell):
.venv\Scripts\Activate.ps1
# macOS/Linux:
# source .venv/bin/activate

# install the LIGHT core deps (enough for full mock-mode flow)
pip install -r requirements.txt

# render the placeholder catalog product images into static/catalog/
python scripts/generate_catalog_images.py

# configure (mock is the default even without a .env)
cp .env.example .env        # Windows: copy .env.example .env

# run the API
uvicorn app.main:app --reload --port 8000

Backend is now at http://127.0.0.1:8000 (open /docs for interactive API docs).

On Windows, if python opens the Microsoft Store, use the full interpreter path to create the venv, e.g. C:\Users\<you>\AppData\Local\Programs\Python\Python311\python.exe -m venv .venv. After activating the venv, python/pip resolve correctly.

2) Frontend — setup & run 

cd frontend
npm install
cp .env.local.example .env.local    # Windows: copy .env.local.example .env.local
npm run dev

Frontend is now at http://localhost:3000. It talks to the backend at NEXT_PUBLIC_API_BASE (defaults to http://127.0.0.1:8000).

3) Mobile app (Expo / React Native) — setup & run

The mobile client uses the same backend. The only extra requirement is that your phone and computer are on the same Wi-Fi and the backend is reachable from the phone.

cd mobile
npm install
npx expo start          # then scan the QR code with Expo Go

Make the backend reachable from your phone:

  1. Bind the backend to all interfaces (not just localhost):
    cd backend
uvicorn app.main:app --host 0.0.0.0 --port 8000
  2. The app auto-detects your computer's LAN IP from the Expo dev server, so it targets http://<your-computer-ip>:8000 with no config. The resolved URL is shown at the top of the app. To override (e.g. a deployed backend), set EXPO_PUBLIC_API_BASE:
    # mobile/.env
EXPO_PUBLIC_API_BASE=http://192.168.1.50:8000
  3. If your OS firewall prompts, allow Python/uvicorn on the private network.

Run targets: scan with Expo Go (Android/iOS), or press a for an Android emulator (i/iOS simulator needs macOS), or npm run web for a browser preview. The app requests photo-library/camera permissions (to pick room photos) and photo-save permission (to save the generated room); these are declared in mobile/app.json.

Same five steps as the web app: upload 3–10 photos → pick one → analyze → choose furniture → generate. Mock mode works identically on mobile.

Build a standalone APK (Android)

The project is already prebuilt for native Android (mobile/android/) and the release variant is signed with the debug keystore (installable; not a Play-Store release).

Requirement — build with JDK 17. Expo SDK 56 ships a Gradle 9.3.1 wrapper whose Java toolchain auto-download is broken; building on JDK 21 crashes (NoSuchFieldError: IBM_SEMERU). A JDK 17 is installed at C:\Users\Hanzala\jdk17\jdk-17.0.19+10, and org.gradle.java.installations.auto-download=false is set in android/gradle.properties.

  1. A standalone APK can't auto-detect the backend, so the URL is baked in at build time via EXPO_PUBLIC_API_BASE. Find your PC's Wi-Fi IPv4 with ipconfig.
  2. Build (PowerShell):
    $env:JAVA_HOME="C:\Users\Hanzala\jdk17\jdk-17.0.19+10"
$env:ANDROID_HOME="C:\Users\Hanzala\AppData\Local\Android\Sdk"
$env:EXPO_PUBLIC_API_BASE="http://192.168.51.176:8000"   # your PC IP
cd C:\Users\Hanzala\room-visualizer-ai\mobile\android
.\gradlew.bat assembleRelease -x lint
  3. APK output: mobile/android/app/build/outputs/apk/release/app-release.apk (~71 MB, all ABIs).

Or in Android Studio: open mobile/android, set Settings → Build, Execution, Deployment → Build Tools → Gradle → Gradle JDK = 17, choose the release build variant, then Build → Build APK(s).

Install & run: copy the APK to the phone and open it (allow "install from unknown sources"), or adb install app-release.apk. Then run the backend with uvicorn app.main:app --host 0.0.0.0 --port 8000 and keep the phone on the same Wi-Fi. If your PC's IP changes, rebuild with the new EXPO_PUBLIC_API_BASE (fast — Gradle is cached).

End-to-end walkthrough

  1. Upload 3–10 room photos (drag/drop or browse).
  2. Choose exactly one photo to redesign.
  3. Analyze — see detected objects, the Shapely usable-floor polygon, blocked areas, and the free-space percentage overlaid on the photo.
  4. Pick furniture from the catalog (filter by category/style; tag-matched items are ranked first). Select one or more items.
  5. Generate — synchronous; a loading state shows during the call, then the final image appears with a Download button.

Environment variables

Var Default Purpose
IMAGE_PROVIDER mock mock or replicate
REPLICATE_API_TOKEN — required for replicate
REPLICATE_MODEL — owner/model:version of an SDXL-inpaint + IP-Adapter model
REPLICATE_INPUT_*_KEY sensible defaults override a model's input field names
ENABLE_ML auto auto (ML if torch present), true, or false
YOLO_MODEL / MIDAS_MODEL yolov8n.pt / MiDaS_small lightweight CPU-friendly variants
SAM_CHECKPOINT / SAM_MODEL_TYPE — / vit_b optional real SAM floor isolation
ROOM_FLOOR_WIDTH_CM 360 real floor width assumed at the image bottom (drives cm→px scaling)
HORIZON_FRAC, DEPTH_FORESHORTEN 0.5, 0.55 perspective heuristics
MIN_IMAGES / MAX_IMAGES / MAX_FILE_MB / MAX_EDGE_PX 3 / 10 / 15 / 1536 upload limits & normalization
HOST / PORT / FRONTEND_ORIGIN 127.0.0.1 / 8000 / http://localhost:3000 server & CORS

Where files land

  • Uploaded (normalized) photos → backend/static/uploads/
  • Generated rooms + depth previews → backend/static/outputs/
  • Catalog product images → backend/static/catalog/
  • Session/image/generation metadata → backend/app.db (SQLite)

All are served under /static.

Optional: real ML room analysis

The analysis modules (detection/depth/segmentation) lazy-load their models and degrade gracefully to OpenCV/geometric fallbacks. To enable the real models:

pip install -r requirements-ml.txt   # torch (CPU by default), ultralytics, timm, segment-anything
# ENABLE_ML=auto already turns them on once torch is importable
  • YOLOv8 detection and MiDaS depth then run for real (CPU is fine, just slower).
  • SAM floor isolation activates only if you also set SAM_CHECKPOINT to a downloaded checkpoint (e.g. sam_vit_b_01ec64.pth); otherwise a robust geometric floor estimate is used. The Shapely placement + perspective scaling + mask construction always run for real regardless of ML.

Optional: real Replicate generation 

# in backend/.env
IMAGE_PROVIDER=replicate
REPLICATE_API_TOKEN=r8_xxx
REPLICATE_MODEL=owner/sdxl-inpaint-ip-adapter:<version-hash>

Pick a model that accepts an image + mask + reference (IP-Adapter) image. If its input field names differ, set the REPLICATE_INPUT_*_KEY vars — no code changes needed.

Honest environment notes

  • Self-hosting the generation model (SDXL + IP-Adapter) needs an NVIDIA GPU with ~8–12 GB VRAM. On a laptop with integrated graphics it is not viable — use the replicate provider (cloud GPU) for real output, or mock for everything else.
  • Real ML analysis (YOLO/MiDaS/SAM) runs on CPU; it's correct, just slower. Use the configured lightweight variants.
  • Pay-per-call providers bill in USD and need a working international payment method. The mock path means development is never blocked by this.

API endpoints

Method Path Purpose
POST /upload-room accept 3–10 images, preprocess (OpenCV), store, return session + image ids
POST /analyze-room {session_id, image_id} → detected objects, usable-floor polygon, blocked areas, free space
GET /get-catalog catalog JSON; ?category=&styles=a,b filtering + recommendation ordering
POST /generate-room {session_id, image_id, item_ids[]} → generated image URL (synchronous)

There is no /remove-object endpoint — object removal is intentionally out of scope.

Testing pipeline modules independently

Every stage is importable on its own, e.g.:

from app.modules.floor_plane import estimate_floor_plane
from app.modules.spatial import compute_spaces, select_placement
from app.modules.scaling import footprint_quad
fp = estimate_floor_plane("static/uploads/<id>.jpg")

Scope (locked)

In: furniture addition via reference-image placement; gallery upload with one redesign target; Shapely + perspective placement; rule-based recommendations; swappable provider with mock + Replicate; SQLite metadata; synchronous generation.

Out (by design): object removal (any form); multi-view fusion / 3D reconstruction / photogrammetry / NeRF; ML-trained recommendations; text-prompt invention of furniture.