docs/lightml/MIGRATION_PLAN.md

Contract Risk Analyzer (CRA) — Migration Plan

This plan details the step-by-step steps for migrating historical datasets and resolving remaining functionality gaps in the Contract Risk Analyzer (CRA).

---

Migration Roadmap

chronology
    title Gap Resolution Timeline
    2026-07-15 : Retrain MLP Classifier
    2026-07-16 : Cache Qwen LLM Weights
    2026-07-17 : Remove Deprecated Datasets
    2026-07-18 : Prepare PostgreSQL Migration

---

1. Action Items

Step 1: Retrain the MLP Clause Classifier (risk_scorer.pkl / legal_mlp.pkl)

• Rationale: The MLP classifier file is currently missing, causing sydeco_engine.py to fall back to rule-based classification. Retraining on current master CSVs will restore AI clause tag matching.
• Execution: Run these commands inside the app container:

  # Step A: Compile training data from datasets directory
  python3 scripts/import_datasets.py
  
  # Step B: Retrain classifier and generate risk_scorer.pkl
  python3 scripts/train_mlp.py
  

Step 2: Cache Qwen LLM weights for Layer 4 Explanations

• Rationale: The prompt runner is ready, but tests are currently PENDING because the Qwen3-1.7B weights are not cached locally.
• Execution:
  1. Temporarily enable downloading by setting LDV_DOWNLOAD_MODELS=1 in docker-compose.yml.
  2. Restart the container to download and cache the model:

     docker compose up --build -d app
     

  3. Once downloaded, revert LDV_DOWNLOAD_MODELS to 0 to enforce strict offline execution.

Step 3: Remove Deprecated Datasets

• Rationale: Deleting historical files prevents configuration conflicts and reduces confusion.
• Execution: Safely delete the following files from /datasets:
  • dangerous_clauses.csv
  • dangerous_clauses_ADDITIONS.csv
  • dangerous_clauses_MASTER.csv
  • required_clauses.csv
  • required_clauses_final_OPTIMAL_with_Legal_Reference.csv

Step 4: Prepare PostgreSQL Schema Migration

• Rationale: Upgrading from SQLite to PostgreSQL avoids database write-locking issues in multi-worker production environments.
• Execution: Configure target database endpoints by injecting LDV_DB_PATH=postgresql://user:pass@host:port/dbname into Gunicorn environment variables.