File size: 8,561 Bytes
82f53e7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 | # Contract Risk Analyzer (CRA) Stack — Current State Specification
This document presents the detailed architectural blueprint, directory layout, subsystem specifications, and runtime configuration of the current **SYDECO Contract Risk Analyzer (CRA)** system.
---
## 1. System Architecture Overview
The Contract Risk Analyzer is a secure, containerized, multi-tenant contract auditing platform. It is engineered with a **local-first, privacy-respecting posture** that operates fully offline, preventing legal document assets from traversing external networks.
```mermaid
graph TD
Client[Client Browser / Frontend] -->|HTTPS / Port 443| Nginx[Nginx Reverse Proxy]
Nginx -->|Port 5000| Flask[Flask Backend / Gunicorn]
Flask -->|Memory/TCP| Redis[Redis Rate Limiter]
Flask -->|Disk Reads/Writes| SQLite[(SQLite DB: sydeco.db)]
Flask -->|Loopback| Worker[In-Process ThreadPool Worker]
Flask -->|HTTP / Port 8000| Translator[Translation Service]
Translator -->|CTranslate2 INT8 / MarianMT| Models[(Local NMT Model Cache)]
```
The system consists of four primary nodes:
1. **Ingress & Reverse Proxy (Nginx)**: Terminates TLS, enforces security headers, and routes traffic.
2. **App Server (Flask/Gunicorn)**: Orchestrates the analysis pipeline, handles user authentication, multi-tenant databases, audit logging, and document encryption/decryption.
3. **Background Worker**: In-process single-thread queue (`ThreadPoolExecutor`) that processes heavy ML classification and reasoning tasks without stalling Flask workers.
4. **Translation Microservice (MarianMT)**: Direct and pivoted translation service using optimized CTranslate2 INT8 model formats.
---
## 2. Directory Structure
Below is the recursive map of active code directories and configuration assets:
* [datasets/](file:///mnt/c/Users/ADVAN/cra/datasets) — Reference databases compiled by legal experts:
* `abusive_clauses.csv`: Abusive clause keywords, impact scores, and recommendations.
* `dangerous_clauses.csv`: Dangerous clause indicators.
* `dangerous_clauses_MASTERv2.csv`: Fully merged set of dangerous clauses and additions.
* `illegal_clauses.csv`: Mandatory criminal/public policy violations.
* `leonine_clauses.csv`: Overbearing or one-sided contract clauses.
* `required_clauses_MASTER.csv`: Global reference library for expected clauses by contract type.
* `legal_citations.csv`: Localized legal code articles (BE, FR, ID, NL, US, EN&W, generic).
* `risk_levels.csv`: Score ranges mapping to LOW/MEDIUM/HIGH/CRITICAL categories.
* [deploy/](file:///mnt/c/Users/ADVAN/cra/deploy) — System configuration and orchestration templates:
* `nginx.conf`: Hardware headers, SSL certificate routing, and Gunicorn proxy configuration.
* `ldv-backup.cron`: Nightly database and upload volume backup cron schedule.
* `setup.sh` & `gen-cert.sh`: SSL provisioning script hooks.
* [ldv-backend/](file:///mnt/c/Users/ADVAN/cra/ldv-backend) — Primary Flask application package:
* `app.py`: REST routes, file ingestion, OCR checks, and process controls.
* `auth.py`: Cryptographic password hashing, user roles, bearer token authentication, and MFA hooks.
* `crypto.py`: Dynamic symmetric encryption (Fernet / AES-256) at rest for raw text and database variables.
* `database.py`: SQLite engine mapping tenant limits, retention times, download links, and durably written audit logs.
* `worker.py`: Background job consumer utilizing Python thread-pools.
* `pdf_report.py`: Document summary report generator utilizing ReportLab.
* `translator.py` & `translator_client.py`: Language gateways coordinating remote Google translate or local Translation Service.
* `sydeco_engine.py`: Direct rule-based backup clause classification interface.
* [detector/](file:///mnt/c/Users/ADVAN/cra/ldv-backend/detector) — Document scanning engine layers:
* `clause_db.py` & `risk_clause_db.py`: Database adapters mapping CSV rows to internal rules.
* `citation_db.py`: In-memory index matching citations to flagged clauses.
* `detector_rules.py`: Layer 1 deterministic regex rule set.
* `detector_distilbert.py`: Layer 2 zero-shot DistilBERT classifier.
* `detector_scorer.py`: Layer 3 risk scoring formulas.
* `detector_explain.py`: Layer 4 opt-in Qwen explanation prompts.
* `detector_profiles.py`: JSON contract profile manager.
* [scripts/](file:///mnt/c/Users/ADVAN/cra/ldv-backend/scripts) — Utility scripts:
* `import_datasets.py` & `train_mlp.py`: Machine learning training pipeline tools.
* `backup.py`: DB backup utility.
* [tests/](file:///mnt/c/Users/ADVAN/cra/ldv-backend/tests) — Integration and unit test cases.
* [ldv-frontend/](file:///mnt/c/Users/ADVAN/cra/ldv-frontend) — Static single-page dashboard application:
* `index.html`: Client upload portal and stepper-guided analysis.
* `admin.html`: Organizations and User management dashboard.
* `citations.html`: Citation verification portal.
* `account.html`: User account, password, and MFA configuration page.
* [lightml-translator/](file:///mnt/c/Users/ADVAN/cra/lightml-translator) — Offline translation service container:
* `app/`: Cleaner modules, PII masking, glossary lookups, and routing engines.
* `config/settings.py`: Port, cache sizes, and HuggingFace directories.
* `download_models.py`: Dependency downloader for MarianMT models.
* `tests/`: Marian and CTranslate2 stress test suite.
* [uiux/](file:///mnt/c/Users/ADVAN/cra/uiux) — HTML mockups and design assets.
---
## 3. Subsystem Specifications
### 3.1. Ingestion & Security Controls
* **Upload Limit**: Enforced at 10 MB. Files are read in chunks up to 10 MB + 1 byte; if the limit is exceeded, an HTTP 413 error is returned.
* **MIME Validation**: Validated via `python-magic` on the first 4096 bytes. ZIP files mimicking `.docx` are resolved correctly, while malicious or renamed extensions (e.g. `.png` as `.pdf`) are rejected with HTTP 400.
* **Rate Limiting**: Configured using `Flask-Limiter` with Redis. Capped at 10 requests/minute on `/login`, 20 requests/minute on `/upload`/`/analyze`, and 60 requests/minute default.
### 3.2. ML Core Pipeline (4 Layers)
* **Layer 1 (Rules)**: Executes regex rules matching 7 jurisdictions and 11 clause types, followed by keyword corroboration in `risk_clause_db.py`.
* **Layer 2 (DistilBERT)**: Translates non-English texts to English, splits them into paragraphs, and feeds them to `typeform/distilbert-base-uncased-mnli`. Labels require a confidence threshold of `0.70`.
* **Semantic Backfill**: Promotes false negatives. Re-checks missing required clauses with semantic NLI; if text matches above `0.65`, the clause is marked present, avoiding missing-clause penalties.
* **Layer 3 (Scorer)**: Adjusts scores using contract profiles. Critical omissions drop scores by up to 20 points, and red flags trigger deductions of 25 (HIGH) or 10 (MEDIUM) points.
* **Layer 4 (Qwen)**: Generates detailed explanation summaries using Qwen3-1.7B, triggered with the `explain=1` parameter.
### 3.3. Isolation, Security at Rest, and Auditing
* **Data Encryption**: AES-256 encryption via Fernet keys (`LDV_ENCRYPTION_KEY`). File uploads, extracted text, and results are encrypted before being written to disk.
* **Purging & Retention**: Analyses are assigned an expiration date based on organizational policies. Purging is executed via `manage.py purge` (vacuuming the database to completely wipe data).
* **Durable Audit Trails**: Key system events write to the SQLite `audit_log` table and are double-written to `audit_durable.log` in append-only mode.
### 3.4. Offline NMT Pipeline
* **Translator microservice**: Uses Helsinki-NLP MarianMT models converted to CTranslate2 INT8 format, caching translated segments using a thread-safe LRU cache. Non-English language pairs translate by pivoting through English.
---
## 4. Hardware and Environment Configurations
* **Host OS**: Ubuntu (WSL2 / Linux).
* **Container Limits**:
* `cra-app-1`: 2.0 CPU cores, 1500 MB RAM limit.
* `cra-lightml-translator-1`: 2.0 CPU cores, 1000 MB RAM limit.
* `cra-redis-1`: 0.5 CPU cores, 256 MB RAM limit.
* **GPU availability**: NVIDIA RTX 4050 Laptop (5 GB VRAM) is present and available via CUDA, though CPU execution is currently set as the default for compatibility.
|