docs/staging-runbook.md

Staging Deployment Runbook

Purpose: Validate the LDV stack on a staging server before promoting to production.
Audience: The engineer running the deploy.
Gate: Every checkbox in this document must pass before flipping LDV_PRODUCTION=1 on the real server.

---

1. Prerequisites

Requirement	Check
Python 3.10+	python3 --version
libmagic (libmagic1)	apt install libmagic1
rsync (for backups)	apt install rsync
NVIDIA driver + CUDA (optional, speeds up L2/L4)	nvidia-smi
Port 5000 open on the staging host	firewall / security group
A second machine or directory for backup rsync target	—

---

2. Clone and Install

git clone https://github.com/vadhh/cra.git /opt/ldv
cd /opt/ldv/ldv-backend
pip install -r requirements.txt

Verify pinned deps installed cleanly — no version conflicts in pip output.

---

2a. Alternative: Docker Compose (skips steps 3–5)

docker-compose.yml runs the full stack (app + redis + nginx/TLS) in one command. Faster for local/staging smoke-testing than the bare-metal path above.

git clone https://github.com/vadhh/cra.git /opt/ldv && cd /opt/ldv

# nginx needs a cert — deploy/certs/ is gitignored (holds a private key),
# so a fresh clone has none. Generate a self-signed one first:
bash deploy/gen-cert.sh          # writes deploy/certs/server.{crt,key}

export LDV_SECRET_KEY=$(python3 -c "import secrets; print(secrets.token_hex(32))")
export LDV_ENCRYPTION_KEY=$(cd ldv-backend && python3 manage.py gen-key)

docker compose up --build

Skipping gen-cert.sh is why nginx fails with cannot load certificate "/etc/nginx/certs/server.crt" — the mount exists, the file doesn't.

Seed the admin account (once containers are up):

docker compose exec app python3 manage.py seed-admin

Serves on https://localhost (self-signed — browser will warn). Continue at section 6 for the health check.

---

3. Environment Variables

Create /opt/ldv/ldv-backend/.env (never commit this file):

# --- REQUIRED ---
LDV_SECRET_KEY=<generate: python3 -c "import secrets; print(secrets.token_hex(32))">
LDV_ENCRYPTION_KEY=<generate: python3 manage.py gen-key>
LDV_PRODUCTION=1

# --- SECURITY ---
LDV_COOKIE_SECURE=1          # set to 0 only if staging has no HTTPS
LDV_CORS_ORIGINS=https://staging.example.com

# --- STORAGE ---
LDV_DB_PATH=/opt/ldv/data/sydeco.db
LDV_RETENTION_DAYS=30

# --- BACKUP ---
LDV_BACKUP_DIR=/var/backups/ldv
LDV_BACKUP_REMOTE=user@backup-host:/backups/ldv   # optional

# --- OPTIONAL ---
LDV_MAX_UPLOAD_MB=10
LDV_DOWNLOAD_LINK_TTL=900
LDV_REMOTE_TRANSLATION=0     # 1=Google, local=Helsinki-NLP offline
LDV_DEBUG=0                  # never 1 in staging/prod

Load before any command:

set -a && source /opt/ldv/ldv-backend/.env && set +a

Harden the file:

chmod 600 /opt/ldv/ldv-backend/.env

---

4. Database Initialisation

cd /opt/ldv/ldv-backend
mkdir -p /opt/ldv/data

# Init schema + run all migrations
python3 -c "import database; database.init_db()"

# Seed the first admin account
LDV_ADMIN_EMAIL=admin@example.com \
LDV_ADMIN_PASSWORD=<strong-password> \
python3 manage.py seed-admin

# Create a test org and users
python3 manage.py create-org "Test Org"
python3 manage.py create-user analyst@example.com "Test Org" --role analyst
python3 manage.py create-user reviewer@example.com "Test Org" --role reviewer

---

5. Start the Server

cd /opt/ldv/ldv-backend
gunicorn -w 2 -b 0.0.0.0:5000 \
  --timeout 120 \
  --access-logfile /var/log/ldv-access.log \
  --error-logfile /var/log/ldv-error.log \
  app:app

Use -w 1 if GPU memory is tight — each worker loads its own model copy.

---

6. Health Check

curl -s http://localhost:5000/health | python3 -m json.tool

Expected — all fields must be non-degraded:

{
  "status": "ok",
  "encryption": "enabled",
  "layer2_available": true,
  "layer4_available": false,
  "db": "ok",
  "datasets": "ok"
}

Block on: encryption: degraded (means LDV_ENCRYPTION_KEY not set), db: error.

---

7. Automated Test Suite

cd /opt/ldv/ldv-backend

# Generate fixtures (once)
python3 tests/create_fixtures.py

# Quick regression (~2 min)
python3 tests/run_validation.py

# Full checklist (~10 min, skips L4 PENDING sections)
python3 tests/run_full_validation.py

Expected: ≥60 PASS · 0 FAIL. WARN on legal_mlp.pkl is acceptable. Any FAIL = stop.

---

8. Auth & MFA Smoke Test

BASE=http://localhost:5000

# 1. Login as analyst
curl -sc cookies.txt -X POST $BASE/login \
  -H "Content-Type: application/json" \
  -d '{"email":"analyst@example.com","password":"<password>"}'
# Expect: {"mfa_enroll_required": true}  OR  session cookie

# 2. Skip MFA as analyst (must succeed — analyst role not mandatory)
curl -sb cookies.txt -X POST $BASE/api/v1/mfa/skip
# Expect: {"ok": true}

# 3. Set org mfa_required and retry (must be blocked)
# sqlite3 /opt/ldv/data/sydeco.db \
#   "UPDATE organizations SET mfa_required=1 WHERE name='Test Org';"
curl -sb cookies.txt -X POST $BASE/api/v1/mfa/skip
# Expect: 403 {"error": "MFA is mandatory for this account"}

# 4. Reviewer role — skip must always be 403 (role is mandatory)
curl -sc cookies2.txt -X POST $BASE/login \
  -H "Content-Type: application/json" \
  -d '{"email":"reviewer@example.com","password":"<password>"}'
curl -sb cookies2.txt -X POST $BASE/api/v1/mfa/skip
# Expect: 403

---

9. Upload & Analysis Smoke Test

# Upload a contract (requires auth cookie from step 8)
curl -sb cookies.txt -X POST $BASE/upload \
  -F "file=@/path/to/test-contract.pdf"
# Expect: {"id": "<uuid>", "status": "queued"}

UUID=<uuid from above>

# Poll until completed
curl -sb cookies.txt $BASE/api/v1/result/$UUID
# Expect: {"status": "completed", "progress_pct": 100, "_meta": {"encryption_enabled": true}, ...}

---

10. Cross-Org Access Control

# Create a second org + user
python3 manage.py create-org "Other Org"
python3 manage.py create-user other@example.com "Other Org" --role analyst

# Login as other@example.com, try to fetch the UUID from step 9
# Expect: 403 Forbidden

---

11. Backup Smoke Test

mkdir -p /var/backups/ldv

# Dry run first
python3 manage.py backup --dry-run

# Real run
python3 manage.py backup

# Confirm timestamped dir exists
ls /var/backups/ldv/
# Expect: 20260701T020000Z/sydeco.db  20260701T020000Z/uploads/

Install the nightly cron:

sudo cp /opt/ldv/deploy/ldv-backup.cron /etc/cron.d/ldv-backup
sudo chmod 644 /etc/cron.d/ldv-backup
# Edit the file: update /opt/ldv path and www-data user if different

---

12. Manual QA — Lawyer Spot-Check

Upload 5–10 real contracts (mix of FR/ID/NL, invoice + service agreement) and verify:

• governing_law detected correctly
• jurisdiction detected correctly
• document_type matches the actual contract type
• red_flags fire on genuinely risky clauses, not on clean contracts
• Citations shown are verified only (no draft in output)
• Risk score feels calibrated
• Non-contract documents (invoices) score low / get non-contract routing

This is the hardest gate. Code cannot substitute for a lawyer reading the output.

---

13. Production Promotion Checklist

Only proceed when all sections above pass:

• Section 6: encryption: enabled, db: ok
• Section 7: 0 FAIL
• Section 8: MFA skip returns 403 when mandatory
• Section 9: encryption_enabled: true in result, progress_pct: 100
• Section 10: 403 on cross-org UUID
• Section 11: backup dir created, nightly cron installed
• Section 12: lawyer spot-check signed off
• LDV_COOKIE_SECURE=1 confirmed (HTTPS in place)
• LDV_DEBUG=0 confirmed
• .env permissions: chmod 600
• Gunicorn running as non-root user (www-data or dedicated ldv user)
• Log rotation configured (/etc/logrotate.d/ldv)

---

Known Acceptable Gaps at Staging

These are tracked but do not block the staging gate:

Gap	Status
legal_mlp.pkl missing — clause_tags returns []	Acceptable, sydeco_engine decoupled
L4 (Qwen) not loaded — layer4_available: false	Acceptable unless ?explain=1 is tested
FR/BE citation rows draft	Output suppresses drafts — acceptable
Bootstrap MLP risk scorer	Deterministic fallback active — acceptable