deploy.md

CineMatch — Deployment Guide

Everything needed to go from a fresh clone to a live API on HuggingFace Spaces with Upstash Redis caching.

---

Architecture

Browser / Frontend
      │
      ▼
HuggingFace Space (Docker)          ← this guide
  uvicorn api.main:app :7860
      │
      ├── artifacts/                ← committed to Space via git-lfs
      │     two_tower.pt, deepfm_best.pt,
      │     faiss.index, preprocessor.pkl, …
      │
      └── Upstash Redis (TLS)       ← secret injected at runtime
            user embeddings cache

---

Prerequisites

Tool	Version	Install
Python	≥ 3.11	python.org
git	any	git-scm.com
git-lfs	≥ 3.0	git lfs install
HuggingFace account	—	huggingface.co
Upstash account	—	upstash.com

---

Part 1 — Train the Models Locally

Skip this part if you already have the artifacts/ directory populated.

1.1 Create and activate a virtual environment

cd recommender/
python -m venv .venv

# Windows
.venv\Scripts\activate

# macOS / Linux
source .venv/bin/activate

1.2 Install dependencies

pip install -r requirements.txt

1.3 Run the training pipeline

# Quick dev run — 10% sample (~2M ratings, ~5 min on CPU)
./run_training.sh

# Full 20M dataset run
SAMPLE=1.0 EPOCHS=20 ./run_training.sh

This produces the following files in recommender/artifacts/:

File	Size	Purpose
preprocessor.pkl	~21 MB	ID maps, feature scalers
movie_meta.csv	~1.5 MB	Movie titles, genres, years
two_tower.pt	~40 MB	Two-Tower model weights
deepfm_best.pt	~12 MB	DeepFM ranker weights
faiss.index	~7 MB	ANN index for candidate retrieval
item_embeddings.npy	~7 MB	Item vectors (for MMR diversity)
item_features.npy	~6 MB	Item feature matrix
user_features.npy	~12 MB	User feature matrix

---

Part 2 — Upstash Redis Setup

Upstash provides a free-tier serverless Redis that works without a persistent server.

2.1 Create a database

1. Go to upstash.com → Sign in → Create Database
2. Choose Redis → Region closest to your HF Space region → Create
3. On the database detail page, copy two values:
   • Redis URL (TCP / TLS) — looks like:

     rediss://default:TOKEN@your-db-name.upstash.io:6379
     

   • REST URL — looks like:

     https://your-db-name.upstash.io
     

   • REST Token

2.2 Verify the connection locally

python -c "
import redis, os
r = redis.from_url('rediss://default:TOKEN@your-db.upstash.io:6379', decode_responses=True)
r.ping()
print('Upstash connected')
"

2.3 Add credentials to your local .env

# recommender/.env
REDIS_URL=rediss://default:TOKEN@your-db.upstash.io:6379
UPSTASH_REDIS_URL=https://your-db.upstash.io
UPSTASH_REDIS_TOKEN=your-token

---

Part 3 — HuggingFace Spaces Deployment

3.1 Create the Space

1. Go to huggingface.co/new-space
2. Fill in:
   • Space name: Cinewatch-recommender
   • SDK: Docker
   • Visibility: Public (or Private)
3. Click Create Space

3.2 Install git-lfs (if not already done)

# Windows (via winget)
winget install GitHub.GitLFS

# macOS
brew install git-lfs

# Linux
sudo apt install git-lfs

# After installing:
git lfs install

3.3 Clone the Space locally

git clone https://YOUR_USERNAME:YOUR_HF_TOKEN@huggingface.co/spaces/YOUR_USERNAME/Cinewatch-recommender
cd Cinewatch-recommender

Replace YOUR_HF_TOKEN with your HuggingFace write token (Settings → Access Tokens → New token → Write).

3.4 Copy backend files into the Space

Run from inside the Cinewatch-recommender folder:

# macOS / Linux
cp -r ../recommender/api      ./api
cp -r ../recommender/serving  ./serving
cp -r ../recommender/models   ./models
cp -r ../recommender/data     ./data
cp -r ../recommender/artifacts ./artifacts

# Windows (PowerShell)
xcopy /E /I ..\recommender\api      api
xcopy /E /I ..\recommender\serving  serving
xcopy /E /I ..\recommender\models   models
xcopy /E /I ..\recommender\data     data
xcopy /E /I ..\recommender\artifacts artifacts

3.5 Create requirements.txt

Create a file called requirements.txt in the root of the Space folder:

torch>=2.2.0
numpy>=1.26.0
pandas>=2.2.0
scikit-learn>=1.4.0
scipy>=1.12.0
faiss-cpu>=1.8.0
fastapi>=0.109.0
uvicorn[standard]>=0.27.0
pydantic>=2.6.0
python-multipart>=0.0.9
redis>=5.0.0
kafka-python>=2.0.2
python-dotenv>=1.0.0
tqdm>=4.66.0

3.6 Create Dockerfile

Create a file called Dockerfile in the root of the Space folder:

FROM python:3.11-slim

RUN useradd -m -u 1000 user
USER user

ENV PATH="/home/user/.local/bin:$PATH"
ENV PYTHONPATH="/app"
ENV ARTIFACT_DIR="/app/artifacts"

WORKDIR /app
COPY --chown=user . /app

RUN pip install --no-cache-dir -r requirements.txt

EXPOSE 7860

CMD ["uvicorn", "api.main:app", "--host", "0.0.0.0", "--port", "7860"]

3.7 Create .gitignore

__pycache__/
*.pyc
*.pyo
.env

# Training-only — not needed for serving
artifacts/splits.pkl
artifacts/two_tower_best.pt
artifacts/eval_results.json

# Runtime state — generated on first run
artifacts/events.db
artifacts/kafka_fallback.db
artifacts/feature_store.db

3.8 Configure git-lfs to track large binaries

git lfs track "*.pt" "*.pkl" "*.npy" "*.index"

HuggingFace's default .gitattributes already covers most of these, but running the command is safe (it only adds missing entries).

3.9 Commit and push to HuggingFace

git add .
git commit -m "Deploy CineMatch backend: Two-Tower + DeepFM + MMR + Upstash Redis"
git push origin main

git-lfs uploads the large model files (~109 MB total) before pushing the regular commit. Expect ~30–60 seconds depending on your connection.

3.10 Set the REDIS_URL secret in the Space

1. Open your Space on HuggingFace
2. Go to Settings → Variables and Secrets
3. Click New Secret and add:

Name	Value
REDIS_URL	rediss://default:TOKEN@your-db.upstash.io:6379

Why a secret and not a variable? The Redis URL contains an auth token. HF Secrets are encrypted at rest and never shown in logs.

---

Part 4 — Watch the Build

After git push completes:

1. Open https://huggingface.co/spaces/YOUR_USERNAME/Cinewatch-recommender
2. Click the Logs tab — you will see the Docker build progress
3. Build takes 3–8 minutes on first deploy (installs PyTorch + all deps)
4. When the status badge turns green, the API is live

---

Part 5 — Verify the Deployment

Replace YOUR_USERNAME with your HuggingFace username:

# Health check
curl https://your-username-cinewatch-recommender.hf.space/api/health

# Recommendations for user 1
curl "https://your-username-cinewatch-recommender.hf.space/api/recommendations/1?limit=5"

# Search
curl "https://your-username-cinewatch-recommender.hf.space/api/search?q=inception"

# Interactive docs
open https://your-username-cinewatch-recommender.hf.space/docs

Expected health response:

{
  "status": "ok",
  "models_loaded": true,
  "faiss_index_size": 27278,
  "num_users": 135626,
  "num_movies": 27278
}

---

Part 6 — Updating the Deployment

Whenever you retrain models or update code:

cd Cinewatch-recommender

# Pull latest from HF first
git pull origin main

# Overwrite changed files
cp -r ../recommender/artifacts ./artifacts   # new model weights
cp -r ../recommender/api       ./api         # code changes

git add .
git commit -m "Update: retrained Two-Tower with full 20M dataset"
git push origin main

HF Space auto-rebuilds on every push.

---

Part 7 — API Endpoint Reference

Method	Path	Description
GET	/api/health	System health + index size
GET	/api/recommendations/{user_id}?limit=20	Personalised picks
GET	/api/movies/{movie_id}	Movie detail + similar movies
GET	/api/movies/{movie_id}/similar?limit=10	Similar movies
GET	/api/movies/popular?limit=20	Trending (cold-start)
GET	/api/search?q={query}&limit=20	Title / genre search
GET	/api/taste-profile/{user_id}	Genre preferences + stats
POST	/api/feedback	Submit thumbs / rating / mood
POST	/api/events	Log a user interaction

Full interactive docs: https://your-space-url.hf.space/docs

---

Troubleshooting

Build fails — torch takes too long to install

HF free-tier builds have a 30-minute timeout. PyTorch is ~800 MB. If the build times out, try pinning to a CPU-only wheel in requirements.txt:

torch==2.2.0+cpu --index-url https://download.pytorch.org/whl/cpu

FileNotFoundError: artifacts/preprocessor.pkl

The artifacts were not pushed correctly. Check:

cd Cinewatch-recommender
git lfs ls-files   # should list .pt, .pkl, .npy files

If the list is empty, re-run git lfs track "*.pt" "*.pkl" "*.npy" "*.index" and re-push.

Redis unavailable — falling back to SQLite

The REDIS_URL secret is missing or wrong in the Space settings. Go to Space → Settings → Secrets and verify the value starts with rediss://.

ModuleNotFoundError: No module named 'api'

PYTHONPATH is not set to /app in the Dockerfile. Verify the Dockerfile contains:

ENV PYTHONPATH="/app"

Space stuck on Building

Click Factory reboot in Space Settings to force a clean rebuild.

---

Environment Variables Reference

Variable	Where to set	Required	Description
REDIS_URL	HF Space Secret	Yes (for caching)	Full rediss:// Upstash connection URL
ARTIFACT_DIR	Dockerfile ENV	Auto-set	Path to model artifacts (/app/artifacts)
PYTHONPATH	Dockerfile ENV	Auto-set	Module search path (/app)
DEVICE	HF Space Variable	No	cpu (default) or cuda
CORS_ORIGINS	HF Space Variable	No	Comma-separated extra allowed origins