README.md

metadata

title: LDV Pilot
emoji: ⚖️
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860

Contract Risk Analyzer (CRA) — LDV Pilot

Automated contract risk assessment for legal professionals. Upload a PDF, DOCX, or TXT contract and receive a structured risk score, clause-gap analysis, red-flag detection, and a downloadable PDF report — all offline-capable and multilingual.

---

Features

• 3-layer analysis pipeline — rule-based clause detection (L1) → NLI document classifier (L2) → deterministic risk scorer with versioned policy (L3)
• 56 contract type profiles — employment, lease, NDA, service, loan, construction, government procurement, and more; covers Indonesian, French, and Dutch contracts
• Red-flag detection — leonine clauses, abusive penalties, unilateral modification, rights waivers
• Explain Mode — per-finding annotations with legal citations and replacement clause suggestions
• Multilingual — detects and processes contracts in Indonesian, French, Dutch, English; offline translation via lightml-translator
• OCR fallback — scanned PDFs handled via Tesseract (EN / FR / ID / NL)
• Job durability — queued → processing → completed / retryable; crashed jobs auto-recover after 30 min; max 3 retries enforced
• Professional review workflow — draft citations suppressed in customer mode; reviewer audit trail
• Multi-tenant auth — org isolation, MFA (TOTP), per-user download controls, API token access
• Encrypted at rest — document text and results encrypted with LDV_ENCRYPTION_KEY
• Offline deployment — no outbound network required; all models ship locally

---

Repository Structure

.
├── ldv-backend/            # Flask API + ML pipeline
│   ├── app.py              # Routes, auth, upload, result, admin
│   ├── worker.py           # Background thread-pool job executor
│   ├── database.py         # SQLite schema, migrations, audit log
│   ├── auth.py             # Session + API token auth, MFA
│   ├── crypto.py           # Fernet encryption at rest
│   ├── pdf_report.py       # PDF report generator (ReportLab)
│   ├── translator.py       # In-process offline translation
│   ├── translator_client.py# External microservice translation (disabled by default)
│   ├── detector/
│   │   ├── detector_rules.py       # L1 rule-based clause & red-flag detection
│   │   ├── detector_distilbert.py  # L2 NLI document classifier
│   │   ├── detector_scorer.py      # L3 risk scorer (versioned policy)
│   │   ├── detector_explain.py     # Explain Mode annotations
│   │   ├── clause_db.py            # Clause keyword definitions
│   │   ├── citation_db.py          # Legal citation database
│   │   ├── risk_clause_db.py       # Red-flag keyword database
│   │   ├── profile_registry.py     # File-driven contract profile loader
│   │   ├── profiles/
│   │   │   └── registry_v1.json    # 56 contract type profiles (single source of truth)
│   │   └── policies/
│   │       └── default_v1.json     # Default scoring policy (weights & thresholds)
│   ├── tests/              # Unit, API, and integration tests (51 tests)
│   └── manage.py           # CLI — seed, migrate, backup, purge
│
├── ldv-frontend/           # Single-page app (Alpine.js + Vanilla CSS)
│   ├── index.html          # Upload, result, retry, explain mode
│   ├── admin.html          # User / org management
│   ├── result.html         # Result permalink page
│   └── account.html        # MFA setup, API token
│
├── datasets/               # Rule and citation CSV databases
│   ├── dangerous_clauses_MASTER.csv
│   ├── legal_citations.csv
│   ├── leonine_clauses.csv
│   └── abusive_clauses.csv
│
├── deploy/                 # Production deployment helpers
│   ├── nginx.conf          # Nginx reverse proxy + TLS config
│   ├── setup.sh            # Fresh-machine install script
│   ├── ldv-backup.cron     # Automated backup cron job
│   └── gen-cert.sh         # Self-signed cert generator
│
├── docs/                   # Engineering docs and reports
├── docker-compose.yml      # Full stack: app + lightml-translator + nginx
└── Dockerfile              # Multi-stage production image

---

Quick Start

Docker (recommended)

# 1. Clone and configure
git clone https://github.com/vadhh/cra.git && cd cra

# 2. Set required secrets
cp .env.example .env          # then edit LDV_SECRET_KEY and LDV_ENCRYPTION_KEY

# 3. Run
docker compose up -d

App is available at http://localhost:7860. Default admin: admin@example.com / password — change immediately.

Local Development

cd ldv-backend
pip install -r requirements.txt
python app.py

---

Environment Variables

Variable	Required	Default	Description
LDV_SECRET_KEY	✅	(random, non-persistent)	Flask session secret
LDV_ENCRYPTION_KEY	✅	(none — plaintext mode)	Fernet key for at-rest encryption
LDV_DB_PATH		./sydeco.db	SQLite database path
LDV_ADMIN_EMAIL		admin@example.com	Auto-seeded admin email
LDV_ADMIN_PASSWORD		password	Auto-seeded admin password
EXTERNAL_TRANSLATION_DISABLED		1	Set to 0 to enable external microservice translation
LIGHTML_TRANSLATOR_URL		(none)	URL of the lightml-translator service
LDV_FORCE_MFA_TESTING		(none)	Set 1 to bypass MFA in test environments

⚠️ Always set LDV_SECRET_KEY and LDV_ENCRYPTION_KEY before any real deployment.

---

Analysis Pipeline

Upload (PDF / DOCX / TXT)
    │
    ├── OCR fallback (Tesseract) if scanned PDF
    ├── Language detection (langdetect)
    └── Translation to EN if needed
            │
            ▼
    Layer 1 — Rule-Based Detection (detector_rules.py)
    ├── Governing law & venue detection
    ├── Clause presence check (43 clause types)
    └── Red-flag detection (leonine, abusive, penalty, waiver)
            │
            ▼
    Layer 2 — NLI Document Classifier (detector_distilbert.py)
    ├── Zero-shot NLI classification (DistilBERT-based)
    ├── Keyword pre-filter (4 languages)
    └── Document type → profile lookup (56 types, registry_v1.json)
            │
            ▼
    Layer 3 — Risk Scorer (detector_scorer.py)
    ├── Versioned scoring policy (policies/default_v1.json)
    ├── Score breakdown persisted per analysis
    └── Risk score 0–100 + label (LOW / MEDIUM / HIGH / CRITICAL)
            │
            ▼
    Result stored in SQLite (encrypted)
    ├── profile_id, profile_version, detection_source, detection_confidence
    ├── score_breakdown (JSON), policy_version
    └── Served via REST API + PDF report

---

Contract Type Support

The profile registry (detector/profiles/registry_v1.json) covers 56 contract types. See docs/p7_catalogue_reconciliation.md for the full breakdown:

Status	Count	Description
✅ Fully auto-detected	11	NLI hypothesis + keyword patterns + profile + scorer
🟡 Manual override works	45	Profile and scoring correct; auto-detection unreliable

---

Running Tests

cd ldv-backend
python -m pytest tests/ -q          # 51 tests
python tests/validate_profiles.py   # clause reference validator
python tests/test_retry.py          # job recovery (B2/B3/B4)

---

Tech Stack

Layer	Technology
API	Flask 3.1, Gunicorn, Flask-Limiter
Auth	Session + TOTP MFA (pyotp), API tokens
ML	PyTorch 2.11, HuggingFace Transformers 5.4
OCR	Tesseract + PyMuPDF
DB	SQLite (via database.py)
Encryption	Fernet (cryptography 48)
PDF	ReportLab
Frontend	Alpine.js, Vanilla CSS
Deployment	Docker, Nginx, Docker Compose
Translation	In-process (offline) + lightml-translator (optional)

---

License

Private — PT Sydeco / LDV Pilot. Not for public distribution.