chore: remove deploy script; add persistent model storage & diagnostics

README.md

@@ -147,6 +147,8 @@ Pipeline stats (subset) from swap pipeline:
 | `MIRAGE_TURN_TLS_ONLY` | Filter TURN to TLS/TCP | `1` |
 | `MIRAGE_PREFER_H264` | Prefer H264 codec in SDP munging | `0` |
 | `MIRAGE_VOICE_ENABLE` | Enable voice processor stub | `0` |
+| `MIRAGE_PERSIST_MODELS` | Persist models under /data and symlink /app/models | `1` |
+| `MIRAGE_PERSIST_MODELS` | Persist models in /data (HF Space) via symlink | `1` |
 
 CodeFormer fidelity example:
 ```bash
@@ -230,6 +232,70 @@ If problems persist, capture the Container log stack trace and open an issue.
 ## Model Auto-Download
 `model_downloader.py` manages required weights with atomic file locks. It supports overriding sources via env variables and gracefully continues if optional enhancers fail to download.
 
+### Persistent Storage Strategy (Hugging Face Spaces)
+
+By default (`MIRAGE_PERSIST_MODELS=1`), the container will:
+
+1. Create (if missing) a persistent directory: `/data/mirage_models`.
+2. Migrate any existing files from an earlier ephemeral `/app/models` (first run only, if the persistent dir is empty).
+3. Symlink `/app/models -> /data/mirage_models`.
+4. Run integrity checks each startup: if the sentinel `.provisioned` exists but any required model (currently `inswapper/inswapper_128_fp16.onnx`) is missing, the downloader is re-invoked automatically.
+
+Disable persistence with:
+```bash
+MIRAGE_PERSIST_MODELS=0
+```
+This forces models to re-download on each cold start (not recommended for production latency / rate-limits).
+
+Sentinel files:
+- `.provisioned` – marker indicating a successful prior provisioning.
+- `.provisioned_meta.json` – sizes and metadata of required models at provisioning time (informational).
+
+If you set `MIRAGE_PROVISION_FRESH=1`, the sentinel is removed and a full re-download is attempted (useful when updating model versions or clearing partial/corrupt files).
+
+Troubleshooting missing models:
+- Call `/debug/models` – it now reports symlink status, sentinel presence, and sizes.
+- If `inswapper` is missing but sentinel present, integrity logic should already trigger re-provision. If not, trigger manually with `MIRAGE_PROVISION_FRESH=1`.
+
+
+### Persistence Strategy (Hugging Face Spaces)
+By default (`MIRAGE_PERSIST_MODELS=1`) the container will:
+1. Create a persistent directory at `/data/mirage_models`.
+2. Symlink `/app/models -> /data/mirage_models` so model downloads survive restarts.
+3. Maintain a sentinel file `.provisioned` plus a meta JSON summarizing required model sizes.
+4. On startup, if the sentinel exists but required files are missing (stale / manual deletion), it forces a re-download.
+
+Disable persistence (always ephemeral) with:
+```bash
+MIRAGE_PERSIST_MODELS=0
+```
+
+You can also force a fresh provisioning ignoring the sentinel:
+```bash
+MIRAGE_PROVISION_FRESH=1
+```
+
+Debug current model status:
+```bash
+curl -s https://<space-subdomain>.hf.space/debug/models | jq
+```
+
+Example response:
+```json
+{
+	"inswapper": {"exists": true, "size": 87916544},
+	"codeformer": {"exists": true, "size": 178140560},
+	"sentinel": {"exists": true, "meta_exists": true},
+	"storage": {
+		"root_is_symlink": true,
+		"root_path": "/app/models",
+		"target": "/data/mirage_models",
+		"persist_mode_env": "1"
+	},
+	"pipeline_initialized": false
+}
+```
+
 ### Endpoints Recap
 See Metrics Endpoints section above. Typical usage examples:
 

app.py

@@ -161,9 +161,27 @@ async def debug_models():
     root = Path(__file__).parent / 'models'
     ins = root / 'inswapper' / 'inswapper_128_fp16.onnx'
     codef = root / 'codeformer' / 'codeformer.pth'
+    sentinel = root / '.provisioned'
+    meta = root / '.provisioned_meta.json'
+    # Detect symlink & target
+    is_symlink = root.is_symlink()
+    target = None
+    if is_symlink:
+        try:
+            target = root.resolve()
+        except Exception:
+            target = None
+    storage_mode = os.environ.get('MIRAGE_PERSIST_MODELS', '1')
     return {
         'inswapper': {'exists': ins.exists(), 'size': ins.stat().st_size if ins.exists() else 0},
         'codeformer': {'exists': codef.exists(), 'size': codef.stat().st_size if codef.exists() else 0},
+        'sentinel': {'exists': sentinel.exists(), 'meta_exists': meta.exists()},
+        'storage': {
+            'root_is_symlink': is_symlink,
+            'root_path': str(root),
+            'target': str(target) if target else None,
+            'persist_mode_env': storage_mode
+        },
         'pipeline_initialized': pipeline_initialized
     }
 

deploy.sh

@@ -1,130 +0,0 @@
-#!/bin/bash
-# Deployment script for Mirage Real-time AI Avatar System
-
-set -e
-
-echo "🎭 Mirage Real-time AI Avatar - Deployment Script"
-echo "=================================================="
-
-# Check if we're deploying to HuggingFace Spaces
-if [[ "${SPACE_ID}" ]]; then
-    echo "📡 Deploying to HuggingFace Spaces: ${SPACE_ID}"
-    DEPLOYMENT_TARGET="huggingface"
-else
-    echo "🐳 Local Docker deployment"
-    DEPLOYMENT_TARGET="local"
-fi
-
-# Set environment variables for optimal A10G performance
-export CUDA_VISIBLE_DEVICES=0
-export TORCH_CUDA_ARCH_LIST="8.6"  # A10G architecture
-export CUDA_LAUNCH_BLOCKING=0
-export MIRAGE_VOICE_ENABLE=1
-export MIRAGE_CHUNK_MS=160
-export MIRAGE_VIDEO_MAX_FPS=20
-
-echo "🔧 Environment configured for A10G GPU"
-
-# Download required models
-echo "📥 Downloading AI models..."
-
-# Create model directories
-mkdir -p models/{liveportrait,rvc,hubert,rmvpe}
-mkdir -p checkpoints
-
-# Function to download from HuggingFace with retry
-download_hf_model() {
-    local repo=$1
-    local filename=$2
-    local output_dir=$3
-    local max_retries=3
-    local retry_count=0
-    
-    while [ $retry_count -lt $max_retries ]; do
-        if python3 -c "
-from huggingface_hub import hf_hub_download
-import os
-try:
-    hf_hub_download('$repo', '$filename', local_dir='$output_dir', local_dir_use_symlinks=False)
-    print('✅ Downloaded $filename')
-except Exception as e:
-    print(f'❌ Failed to download $filename: {e}')
-    exit(1)
-        "; then
-            break
-        fi
-        
-        retry_count=$((retry_count + 1))
-        echo "⏳ Retry $retry_count/$max_retries for $filename"
-        sleep 2
-    done
-    
-    if [ $retry_count -eq $max_retries ]; then
-        echo "❌ Failed to download $filename after $max_retries retries"
-        return 1
-    fi
-}
-
-# Download LivePortrait models (if available)
-if python3 -c "from huggingface_hub import HfApi; api = HfApi(); print('✅ HuggingFace available')" 2>/dev/null; then
-    echo "🎨 Attempting to download LivePortrait models..."
-    # Note: These would be the actual model files when available
-    # download_hf_model "KwaiVGI/LivePortrait" "appearance_feature_extractor.pth" "models/liveportrait"
-    # download_hf_model "KwaiVGI/LivePortrait" "motion_extractor.pth" "models/liveportrait"
-    # download_hf_model "KwaiVGI/LivePortrait" "warping_module.pth" "models/liveportrait"
-    # download_hf_model "KwaiVGI/LivePortrait" "spade_generator.pth" "models/liveportrait"
-    echo "ℹ️ LivePortrait models will be downloaded on first use"
-else
-    echo "⚠️ HuggingFace Hub not available, models will be downloaded at runtime"
-fi
-
-# Verify GPU availability
-echo "🔍 Checking GPU configuration..."
-python3 -c "
-import torch
-print(f'PyTorch version: {torch.__version__}')
-print(f'CUDA available: {torch.cuda.is_available()}')
-if torch.cuda.is_available():
-    print(f'GPU: {torch.cuda.get_device_name(0)}')
-    print(f'CUDA version: {torch.version.cuda}')
-    print(f'GPU memory: {torch.cuda.get_device_properties(0).total_memory / 1024**3:.1f} GB')
-else:
-    print('⚠️ GPU not available - running in CPU mode')
-"
-
-# Setup virtual camera (Linux only)
-if [[ "$OSTYPE" == "linux-gnu"* ]]; then
-    echo "📹 Setting up virtual camera (v4l2loopback)..."
-    
-    # Check if v4l2loopback is available
-    if modprobe v4l2loopback devices=1 video_nr=10 card_label="Mirage Virtual Camera" 2>/dev/null; then
-        echo "✅ Virtual camera device created: /dev/video10"
-    else
-        echo "⚠️ Could not create virtual camera device (requires sudo)"
-        echo "💡 Run: sudo modprobe v4l2loopback devices=1 video_nr=10 card_label='Mirage Virtual Camera'"
-    fi
-fi
-
-# Start the application
-echo "🚀 Starting Mirage AI Avatar System..."
-
-if [[ "${DEPLOYMENT_TARGET}" == "huggingface" ]]; then
-    # HuggingFace Spaces deployment
-    echo "🤗 Running on HuggingFace Spaces with A10G GPU"
-    exec python3 -u app.py
-else
-    # Local deployment
-    echo "💻 Running locally"
-    
-    # Check if port 7860 is available
-    if lsof -Pi :7860 -sTCP:LISTEN -t >/dev/null; then
-        echo "⚠️ Port 7860 is already in use"
-        PORT=7861
-    else
-        PORT=7860
-    fi
-    
-    echo "🌐 Server will be available at: http://localhost:${PORT}"
-    export PORT=${PORT}
-    exec python3 -u app.py
-fi

entrypoint.sh

@@ -4,32 +4,102 @@ set -euo pipefail
 echo "[entrypoint] Starting Mirage container..."
 echo "[entrypoint] Python: $(python3 --version 2>&1)"
 
+#############################################
+# Model persistence & provisioning strategy  #
+#############################################
+# We support persistent storage on Hugging Face Spaces by symlinking /app/models
+# to /data/mirage_models (HF persists /data across restarts). Controlled by
+# MIRAGE_PERSIST_MODELS (default=1). If disabled, we keep ephemeral /app/models.
+
+PERSIST_DEFAULT=1
+if [[ "${MIRAGE_PERSIST_MODELS:-}" =~ ^(0|false|no|off)$ ]]; then
+  PERSIST_DEFAULT=0
+fi
+
+if [[ $PERSIST_DEFAULT -eq 1 ]]; then
+  PERSIST_ROOT="/data/mirage_models"
+  mkdir -p "$PERSIST_ROOT" || true
+  # If /app/models exists and is not a symlink, migrate its contents (first run)
+  if [[ -d /app/models && ! -L /app/models ]]; then
+    # Only migrate if persistent dir is empty (avoid overwriting existing cache)
+    if [[ -z "$(ls -A "$PERSIST_ROOT" 2>/dev/null)" ]]; then
+      echo "[entrypoint] Migrating existing /app/models/* -> $PERSIST_ROOT (first persistent run)"
+      shopt -s dotglob nullglob
+      for f in /app/models/*; do
+        mv "$f" "$PERSIST_ROOT/" 2>/dev/null || true
+      done
+      shopt -u dotglob nullglob
+    fi
+    rm -rf /app/models
+    ln -s "$PERSIST_ROOT" /app/models
+    echo "[entrypoint] Symlinked /app/models -> $PERSIST_ROOT"
+  elif [[ -L /app/models ]]; then
+    echo "[entrypoint] /app/models already symlinked"
+  else
+    # Create then symlink if missing
+    rm -rf /app/models 2>/dev/null || true
+    ln -s "$PERSIST_ROOT" /app/models
+    echo "[entrypoint] Initialized persistent symlink /app/models -> $PERSIST_ROOT"
+  fi
+  STORAGE_MODE="persistent"
+  STORAGE_PATH="$PERSIST_ROOT"
+else
+  mkdir -p /app/models || true
+  STORAGE_MODE="ephemeral"
+  STORAGE_PATH="/app/models"
+fi
+
 # Updated provisioning for face swap pipeline (InSwapper + optional CodeFormer)
 MODEL_ROOT="/app/models"
 SENTINEL="${MODEL_ROOT}/.provisioned"
+SENTINEL_META="${MODEL_ROOT}/.provisioned_meta.json"
 AUDIT_FILE="${MODEL_ROOT}/_download_audit.jsonl"
 REQ_FILES=("inswapper/inswapper_128_fp16.onnx")
 DL_TAG="${MIRAGE_DL_TAG:-startup}"
 
+echo "[entrypoint] Storage mode: ${STORAGE_MODE} (root=${STORAGE_PATH})"
 echo "[entrypoint] Ensuring model directory exists: ${MODEL_ROOT}"
-mkdir -p "${MODEL_ROOT}"
+mkdir -p "${MODEL_ROOT}" || true
 
+# Function to validate required model presence
+validate_models() {
+  local missing=0
+  for f in "${REQ_FILES[@]}"; do
+    if [[ ! -f "${MODEL_ROOT}/$f" ]]; then
+      missing=1
+    fi
+  done
+  return $missing
+}
+
+# Decide whether to (re)download. Conditions forcing provisioning:
+#  - Sentinel missing
+#  - MIRAGE_PROVISION_FRESH set
+#  - Sentinel exists but required files missing (stale sentinel)
+#  - Sentinel meta hash mismatch (future extension; placeholder now)
 should_download=0
+
+SENTINEL_REASON="skip"
 if [[ ! -f "${SENTINEL}" ]]; then
   should_download=1
-  echo "[entrypoint] Sentinel missing; provisioning required"
+  SENTINEL_REASON="no_sentinel"
+elif [[ "${MIRAGE_PROVISION_FRESH:-0}" =~ ^(1|true|yes|on)$ ]]; then
+  echo "[entrypoint] MIRAGE_PROVISION_FRESH set; forcing fresh provisioning"
+  rm -f "${SENTINEL}" "${SENTINEL_META}" || true
+  should_download=1
+  SENTINEL_REASON="forced_fresh"
 else
-  if [[ "${MIRAGE_PROVISION_FRESH:-0}" =~ ^(1|true|yes|on)$ ]]; then
-    echo "[entrypoint] MIRAGE_PROVISION_FRESH set; forcing fresh provisioning"
-    rm -f "${SENTINEL}" || true
+  if ! validate_models; then
+    echo "[entrypoint] Sentinel present but required model(s) missing; will re-provision"
     should_download=1
-  else
-    echo "[entrypoint] Sentinel present; skipping download (idempotent)"
-    # Audit skip
-    ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
-    echo "{\"ts\":\"${ts}\",\"event\":\"skip_provision\",\"tag\":\"${DL_TAG}\"}" >> "${AUDIT_FILE}" 2>/dev/null || true
+    SENTINEL_REASON="stale_sentinel"
   fi
 fi
+if [[ $should_download -eq 0 ]]; then
+  echo "[entrypoint] Sentinel OK; skipping download (reason=${SENTINEL_REASON})"
+  ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
+  echo "{\"ts\":\"${ts}\",\"event\":\"skip_provision\",\"reason\":\"${SENTINEL_REASON}\",\"tag\":\"${DL_TAG}\"}" >> "${AUDIT_FILE}" 2>/dev/null || true
+fi
 
 if [[ $should_download -eq 1 ]]; then
   if [[ "${MIRAGE_DOWNLOAD_MODELS:-1}" =~ ^(1|true|TRUE|yes|on)$ ]]; then
@@ -39,7 +109,27 @@ if [[ $should_download -eq 1 ]]; then
     if python3 /app/model_downloader.py; then
       ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
       echo "{\"ts\":\"${ts}\",\"event\":\"provision_success\",\"tag\":\"${DL_TAG}\"}" >> "${AUDIT_FILE}" 2>/dev/null || true
+      # Write sentinel + meta summary (sizes of required models)
       touch "${SENTINEL}" || true
+      {
+        echo '{'
+        echo '  "ts": "'"${ts}"'",'
+        echo '  "storage_mode": "'"${STORAGE_MODE}"'",'
+        echo '  "required_models": {'
+        comma=0
+        for f in "${REQ_FILES[@]}"; do
+          size=0
+          if [[ -f "${MODEL_ROOT}/$f" ]]; then
+            size=$(stat -c%s "${MODEL_ROOT}/$f" 2>/dev/null || wc -c <"${MODEL_ROOT}/$f")
+          fi
+          if [[ $comma -eq 1 ]]; then echo ','; fi
+          printf '    "%s": {"size": %s}' "$f" "$size"
+          comma=1
+        done
+        echo ''
+        echo '  }'
+        echo '}'
+      } > "${SENTINEL_META}" 2>/dev/null || true
     else
       ts=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
       echo "{\"ts\":\"${ts}\",\"event\":\"provision_error\",\"tag\":\"${DL_TAG}\"}" >> "${AUDIT_FILE}" 2>/dev/null || true

model_downloader.py

@@ -285,14 +285,25 @@ def maybe_download() -> bool:
     codef_dest = CODEFORMER_DIR / 'codeformer.pth'
     if not codef_dest.exists():
         try:
-            print('[downloader] Downloading CodeFormer model...')
+            print('[downloader] Downloading CodeFormer model (optional)...')
             codef_dest.parent.mkdir(parents=True, exist_ok=True)
             with _FileLock(codef_dest):
                 if not codef_dest.exists():
                     _attempt_urls(codeformer_urls, codef_dest)
-            print(f'[downloader] ✅ CodeFormer ready: {codef_dest}')
-            _audit('download_ok', model='codeformer', path=str(codef_dest))
-        except Exception as e:
+            if codef_dest.exists() and codef_dest.stat().st_size > 1048576:  # >1MB sanity threshold
+                print(f'[downloader] ✅ CodeFormer ready: {codef_dest}')
+                _audit('download_ok', model='codeformer', path=str(codef_dest))
+            else:
+                if codef_dest.exists():
+                    try:
+                        size = codef_dest.stat().st_size
+                        print(f'[downloader] ⚠️ CodeFormer file too small ({size} bytes); removing partial')
+                        codef_dest.unlink()
+                    except Exception:
+                        pass
+                print('[downloader] ⚠️ CodeFormer unavailable after attempts (continuing without enhancer)')
+                _audit('download_incomplete', model='codeformer')
+        except Exception as e:  # pragma: no cover - best effort optional
             print(f'[downloader] ⚠️ CodeFormer download failed (continuing): {e}')
             _audit('download_error', model='codeformer', error=str(e))
     else:

models/liveportrait/README.md

@@ -1,16 +0,0 @@
-# LivePortrait (ONNX) Models
-
-Place the following ONNX files in this directory:
-
-- appearance_feature_extractor.onnx (required)
-- motion_extractor.onnx (required)
-- generator.onnx (optional, improves fidelity; if missing we use motion-based warping or landmark fallback)
-
-Environment variables for runtime downloader (optional):
-
-- MIRAGE_DOWNLOAD_MODELS=1
-- MIRAGE_LP_APPEARANCE_URL
-- MIRAGE_LP_MOTION_URL
-- MIRAGE_LP_GENERATOR_URL (optional)
-
-At startup, the app will try to download models to this folder if URLs are configured. You can also drop files here manually. The /debug/models endpoint reports presence and sizes.