migrate data

DEPLOYMENT.md

@@ -0,0 +1,237 @@
+# Guide de déploiement - CyberLegalAI Knowledge Graph sur Hugging Face
+
+## Vue d'ensemble
+
+Ce guide explique comment déployer CyberLegalAI en utilisant un dataset Hugging Face pour stocker le knowledge graph, libérant ainsi 842 Mo dans le repo principal Space.
+
+## Avantages de cette solution
+
+✅ **842 Mo libérés** dans le repo Space (limite de 1 Go respectée)  
+✅ **Téléchargement intelligent** avec cache local (pas de re-download)  
+✅ **Démarrage rapide** après premier téléchargement  
+✅ **Multi-juridictions** supportées nativement  
+✅ **Maintenabilité** facile d'ajouter de nouvelles juridictions  
+✅ **Robustesse** données sauvegardées dans dataset séparé  
+
+## Architecture
+
+```
+Hugging Face Space (Repo principal)
+├── Code application
+└── Configuration
+
+Hugging Face Dataset (Séparé)
+└── data/rag_storage/
+    ├── romania/ (~267 Mo)
+    └── bahrain/ (~575 Mo)
+```
+
+## Étapes de déploiement
+
+### 1. Prérequis
+
+- Compte Hugging Face avec accès aux Spaces
+- Token d'accès Hugging Face avec permissions de lecture sur les datasets
+- CLI Hugging Face installée: `pip install huggingface-hub`
+
+### 2. Création du Dataset Hugging Face
+
+```bash
+# Créer le dataset
+huggingface-cli repo create Cyberlgl/CyberLegalAI-knowledge-graph --type dataset
+
+# Uploader les données du knowledge graph
+cd /Users/cgrdj/Documents/Code/Cyberlgl/CyberlegalAI
+huggingface-cli upload Cyberlgl/CyberLegalAI-knowledge-graph data/rag_storage/
+```
+
+Vérifiez que le dataset est bien créé: https://huggingface.co/datasets/Cyberlgl/CyberLegalAI-knowledge-graph
+
+### 3. Configuration du Space
+
+Ajoutez les variables d'environnement suivantes dans votre Hugging Face Space:
+
+```
+HF_TOKEN=your_hf_token_here
+JURISDICTIONS=romania,bahrain
+HF_KNOWLEDGE_GRAPH_DATASET=Cyberlgl/CyberLegalAI-knowledge-graph
+HF_HOME=/data/.huggingface
+```
+
+### 4. Nettoyage du repo principal
+
+Une fois les données transférées vers le dataset:
+
+```bash
+# Supprimer les données du repo principal (conservez localement si nécessaire)
+rm -rf data/rag_storage/
+
+# Ajouter au .gitignore pour éviter de re-ajouter ces fichiers
+echo "data/rag_storage/" >> .gitignore
+
+# Commiter les changements
+git add .gitignore
+git commit -m "Exclude knowledge graph from repo - now served from Hugging Face dataset"
+git push
+```
+
+### 5. Redéploiement du Space
+
+Le Space va automatiquement:
+1. Télécharger le knowledge graph depuis le dataset
+2. Le mettre en cache dans `/data/.huggingface`
+3. Copier les fichiers vers `data/rag_storage/`
+4. Démarrer les serveurs LightRAG et l'API
+
+## Vérification du déploiement
+
+### Logs de démarrage
+
+Vous devriez voir dans les logs du Space:
+
+```
+📥 Checking for knowledge graph data...
+🚀 Knowledge graph not found, downloading from Hugging Face...
+================================================================================
+🚀 Starting Knowledge Graph Download
+================================================================================
+📦 Dataset: Cyberlgl/CyberLegalAI-knowledge-graph
+🌍 Jurisdictions: romania, bahrain
+💾 HF Cache: /data/.huggingface
+📁 Target Directory: data/rag_storage
+================================================================================
+
+📥 Processing jurisdiction: romania
+...
+✅ romania: 18 files copied (267.0 MB)
+
+📥 Processing jurisdiction: bahrain
+...
+✅ bahrain: 18 files copied (575.0 MB)
+
+================================================================================
+🎉 Knowledge Graph Download Complete!
+================================================================================
+📊 romania: 267.0 MB
+📊 bahrain: 575.0 MB
+
+💾 Total size: 842.0 MB
+================================================================================
+```
+
+### Redémarrage ultérieur
+
+Aux redémarrages suivants, vous verrez:
+
+```
+📥 Checking for knowledge graph data...
+✅ Knowledge graph data already present, skipping download
+```
+
+## Maintenance
+
+### Mettre à jour le knowledge graph
+
+1. Mettez à jour les données localement
+2. Uploadez les modifications vers le dataset Hugging Face:
+
+```bash
+huggingface-cli upload Cyberlgl/CyberLegalAI-knowledge-graph data/rag_storage/ --repo-type dataset
+```
+
+3. Redémarrez le Space pour appliquer les modifications
+
+### Ajouter une nouvelle juridiction
+
+1. Ajoutez les données dans `data/rag_storage/nouvelle_juridiction/`
+2. Uploadez vers le dataset Hugging Face:
+
+```bash
+huggingface-cli upload Cyberlgl/CyberLegalAI-knowledge-graph data/rag_storage/nouvelle_juridiction/ --repo-type dataset
+```
+
+3. Mettez à jour la variable `JURISDICTIONS` dans le Space:
+   ```
+   JURISDICTIONS=romania,bahrain,nouvelle_juridiction
+   ```
+
+4. Redémarrez le Space
+
+## Dépannage
+
+### Erreur "Dataset not found"
+
+**Symptôme:** `Repo not found: Cyberlgl/CyberLegalAI-knowledge-graph`
+
+**Solution:**
+- Vérifiez que le dataset existe: https://huggingface.co/datasets/Cyberlgl/CyberLegalAI-knowledge-graph
+- Vérifiez que l'ID du dataset dans `HF_KNOWLEDGE_GRAPH_DATASET` est correct
+
+### Erreur "Invalid token"
+
+**Symptôme:** `Invalid token passed`
+
+**Solution:**
+- Vérifiez que `HF_TOKEN` est correctement configuré dans le Space
+- Créez un nouveau token avec les permissions de lecture (read) sur les datasets: https://huggingface.co/settings/tokens
+
+### Téléchargement lent
+
+**Symptôme:** Le téléchargement prend beaucoup de temps
+
+**Solution:**
+- Le premier téléchargement peut prendre plusieurs minutes pour 842 Mo
+- Les téléchargements suivants seront instantanés grâce au cache
+- Vérifiez que les permissions réseau du Space sont correctes
+
+### Erreur "Permission denied" lors du téléchargement
+
+**Symptôme:** `PermissionError: [Errno 13] Permission denied: '/data/.huggingface'`
+
+**Solution:**
+- Le script devrait créer automatiquement le répertoire avec les bonnes permissions
+- Si l'erreur persiste, vérifiez les permissions dans le Dockerfile
+
+## Variables d'environnement
+
+| Variable | Description | Défaut |
+|----------|-------------|--------|
+| `HF_TOKEN` | Token d'accès Hugging Face | (requis) |
+| `JURISDICTIONS` | Liste des juridictions à télécharger (séparées par virgules) | `romania,bahrain` |
+| `HF_KNOWLEDGE_GRAPH_DATASET` | ID du dataset Hugging Face | `Cyberlgl/CyberLegalAI-knowledge-graph` |
+| `HF_HOME` | Répertoire de cache Hugging Face | `/data/.huggingface` |
+
+## Scripts inclus
+
+### `scripts/download_knowledge_graph.py`
+
+Script principal gérant le téléchargement du knowledge graph.
+
+**Fonctionnalités:**
+- Téléchargement automatique depuis le dataset Hugging Face
+- Support du cache persistant pour éviter les re-téléchargements
+- Téléchargement sélectif par juridiction
+- Logs détaillés du processus de téléchargement
+- Copie des fichiers vers le répertoire d'application
+
+Pour plus de détails, voir: `scripts/README.md`
+
+## Support
+
+Pour toute question ou problème:
+1. Consultez `scripts/README.md` pour les détails des scripts
+2. Consultez les logs du Space pour les erreurs spécifiques
+3. Ouvrez une issue sur GitHub: https://github.com/Cgrandjean/CyberLegalAI
+
+## Migration depuis l'ancien système
+
+Si vous migrez depuis un système où les données étaient dans le repo principal:
+
+1. **Sauvegardez localement** les données existantes
+2. **Créez le dataset** (étape 2)
+3. **Uploadez les données** (étape 2)
+4. **Configurez le Space** (étape 3)
+5. **Nettoyez le repo** (étape 4)
+6. **Redéployez** (étape 5)
+
+Les données resteront accessibles et le système fonctionnera comme avant, mais avec les avantages du nouveau système.

Dockerfile

@@ -11,6 +11,8 @@ ENV PYTHONIOENCODING=utf-8
 ENV LIGHTRAG_HOST=127.0.0.1
 ENV LIGHTRAG_PORT=9621
 ENV API_PORT=8000
+ENV HF_HOME=/data/.huggingface
+ENV JURISDICTIONS=romania,bahrain
 
 # Install system dependencies
 RUN apt-get update && apt-get install -y \

data/lawyers.json

@@ -1,202 +0,0 @@
-[
-  {
-    "name": "Nader Bakri",
-    "experience_years": 8,
-    "specialty": "Cyber Law",
-    "presentation": "Experienced lawyer focusing on complex legal matters at the intersection of technology, business, and regulatory compliance. Provides practical and solution-oriented legal advice tailored to modern digital challenges.",
-    "areas_of_practice": [
-      "Criminal Law",
-      "Commercial Law",
-      "Civil Law",
-      "Administrative Law",
-      "Family Law",
-      "Cyber Law",
-      "IT Law",
-      "AI Law",
-      "Data Protection"
-    ]
-  },
-  {
-    "name": "Andrei Popescu",
-    "experience_years": 12,
-    "specialty": "Commercial & Corporate Law",
-    "presentation": "Seasoned legal professional with extensive experience advising companies on corporate governance, contracts, and commercial disputes at both national and international levels.",
-    "areas_of_practice": [
-      "Commercial Law",
-      "Corporate Law",
-      "Civil Law",
-      "Contract Law",
-      "Commercial Litigation",
-      "Arbitration"
-    ]
-  },
-  {
-    "name": "Maria Ionescu",
-    "experience_years": 9,
-    "specialty": "Data Protection & Privacy Law",
-    "presentation": "Specialized in data protection and privacy compliance, assisting organizations in aligning their operations with GDPR and international data protection standards.",
-    "areas_of_practice": [
-      "Data Protection",
-      "GDPR",
-      "Cyber Law",
-      "IT Law",
-      "Civil Law",
-      "Commercial Law",
-      "Compliance"
-    ]
-  },
-  {
-    "name": "Karim Al-Hassan",
-    "experience_years": 15,
-    "specialty": "International Business Law",
-    "presentation": "International lawyer advising multinational clients on cross-border transactions, regulatory frameworks, and international commercial contracts.",
-    "areas_of_practice": [
-      "International Commercial Law",
-      "Civil Law",
-      "Contract Law",
-      "Arbitration",
-      "Customs Law"
-    ]
-  },
-  {
-    "name": "Elena Radu",
-    "experience_years": 7,
-    "specialty": "Civil & Family Law",
-    "presentation": "Dedicated legal professional providing assistance in sensitive civil and family matters, with a strong focus on ethics, discretion, and client trust.",
-    "areas_of_practice": [
-      "Civil Law",
-      "Family Law",
-      "Matrimonial Law",
-      "Inheritance Law",
-      "Litigation"
-    ]
-  },
-  {
-    "name": "Victor Marinescu",
-    "experience_years": 14,
-    "specialty": "Criminal Law",
-    "presentation": "Experienced criminal defense lawyer representing clients in complex investigations and high-stakes criminal proceedings.",
-    "areas_of_practice": [
-      "Criminal Law",
-      "Criminal Procedure",
-      "Related Civil Claims",
-      "Litigation"
-    ]
-  },
-  {
-    "name": "Sophia Klein",
-    "experience_years": 10,
-    "specialty": "IT & Technology Law",
-    "presentation": "Technology-focused legal advisor assisting startups and technology companies with contracts, compliance, and intellectual property matters.",
-    "areas_of_practice": [
-      "IT Law",
-      "Cyber Law",
-      "AI Law",
-      "Intellectual Property Law",
-      "Commercial Law"
-    ]
-  },
-  {
-    "name": "Mihai Dumitrescu",
-    "experience_years": 18,
-    "specialty": "Administrative & Public Law",
-    "presentation": "Legal expert in administrative disputes and public procurement, representing both private entities and public authorities.",
-    "areas_of_practice": [
-      "Administrative Law",
-      "Public Law",
-      "Public Procurement",
-      "Administrative Litigation",
-      "Constitutional Law"
-    ]
-  },
-  {
-    "name": "Laura Petrescu",
-    "experience_years": 6,
-    "specialty": "Employment & Labor Law",
-    "presentation": "Advises employers and employees on labor relations, regulatory compliance, and employment-related dispute resolution.",
-    "areas_of_practice": [
-      "Employment Law",
-      "Labor Law",
-      "Civil Law",
-      "Employment Litigation",
-      "Compliance"
-    ]
-  },
-  {
-    "name": "Omar Khaled",
-    "experience_years": 11,
-    "specialty": "Cybercrime & Digital Evidence",
-    "presentation": "Specialist in cybercrime cases and digital investigations, with strong expertise in electronic evidence and forensic collaboration.",
-    "areas_of_practice": [
-      "Cyber Law",
-      "Criminal Law",
-      "Cybercrime",
-      "IT Law",
-      "Digital Evidence"
-    ]
-  },
-  {
-    "name": "Ana-Maria Stoica",
-    "experience_years": 13,
-    "specialty": "Intellectual Property Law",
-    "presentation": "Provides strategic legal protection for brands, software, and creative works in both domestic and international markets.",
-    "areas_of_practice": [
-      "Intellectual Property Law",
-      "Commercial Law",
-      "IT Law",
-      "Copyright",
-      "Trademarks"
-    ]
-  },
-  {
-    "name": "Daniel Weiss",
-    "experience_years": 16,
-    "specialty": "Arbitration & Litigation",
-    "presentation": "Experienced litigator representing clients in complex commercial disputes before courts and arbitral tribunals.",
-    "areas_of_practice": [
-      "Litigation",
-      "Arbitration",
-      "Commercial Law",
-      "Civil Law",
-      "Private International Law"
-    ]
-  },
-  {
-    "name": "Raluca Neagu",
-    "experience_years": 8,
-    "specialty": "Compliance & Regulatory Law",
-    "presentation": "Advises companies on regulatory compliance, internal governance, and risk management frameworks.",
-    "areas_of_practice": [
-      "Compliance",
-      "Regulatory Law",
-      "Commercial Law",
-      "Administrative Law",
-      "Data Protection"
-    ]
-  },
-  {
-    "name": "Hassan Farouk",
-    "experience_years": 20,
-    "specialty": "Banking & Financial Law",
-    "presentation": "Senior legal advisor with extensive experience in banking regulation, financial transactions, and risk mitigation.",
-    "areas_of_practice": [
-      "Banking Law",
-      "Financial Law",
-      "Compliance",
-      "Commercial Law"
-    ]
-  },
-  {
-    "name": "Ioana Vasilescu",
-    "experience_years": 5,
-    "specialty": "AI & Emerging Technologies Law",
-    "presentation": "Focused on legal challenges related to artificial intelligence, automation, and emerging technologies, supporting innovation-driven organizations.",
-    "areas_of_practice": [
-      "AI Law",
-      "Cyber Law",
-      "IT Law",
-      "Data Protection",
-      "Commercial Law"
-    ]
-  }
-]

requirements.txt

@@ -26,3 +26,4 @@ langchain-tavily>=0.2.16
 resend>=0.8.0
 beautifulsoup4>=4.12.0
 httpx>=0.24.0
+huggingface-hub>=0.20.0

scripts/download_knowledge_graph.py

@@ -0,0 +1,118 @@
+#!/usr/bin/env python3
+"""
+Download knowledge graph from Hugging Face dataset at container startup
+"""
+import os
+import shutil
+import logging
+from pathlib import Path
+from huggingface_hub import snapshot_download
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv(dotenv_path=".env", override=False)
+
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
+logger = logging.getLogger(__name__)
+
+
+def download_knowledge_graph():
+    """
+    Download knowledge graph from Hugging Face dataset and copy to application directory
+    """
+    # Configure Hugging Face cache to use persistent storage
+    hf_home = "/data/.huggingface"
+    os.environ["HF_HOME"] = hf_home
+    os.makedirs(hf_home, exist_ok=True)
+    
+    # Get jurisdictions to download
+    jurisdictions_str = os.getenv("JURISDICTIONS", "romania,bahrain")
+    jurisdictions = [j.strip() for j in jurisdictions_str.split(",")]
+    
+    # Dataset configuration
+    dataset_id = os.getenv("HF_KNOWLEDGE_GRAPH_DATASET", "Cyberlgl/CyberLegalAI-knowledge-graph")
+    hf_token = os.getenv("HF_TOKEN")
+    
+    # Target directory
+    target_base_dir = "data/rag_storage"
+    os.makedirs(target_base_dir, exist_ok=True)
+    
+    logger.info("=" * 80)
+    logger.info("🚀 Starting Knowledge Graph Download")
+    logger.info("=" * 80)
+    logger.info(f"📦 Dataset: {dataset_id}")
+    logger.info(f"🌍 Jurisdictions: {', '.join(jurisdictions)}")
+    logger.info(f"💾 HF Cache: {hf_home}")
+    logger.info(f"📁 Target Directory: {target_base_dir}")
+    logger.info("=" * 80)
+    
+    try:
+        for jurisdiction in jurisdictions:
+            logger.info(f"\n📥 Processing jurisdiction: {jurisdiction}")
+            
+            # Download from dataset with filtering
+            local_path = snapshot_download(
+                repo_id=dataset_id,
+                repo_type="dataset",
+                allow_patterns=[f"{jurisdiction}/*"],
+                cache_dir=hf_home,
+                token=hf_token
+            )
+            
+            logger.info(f"✅ Downloaded to cache: {local_path}")
+            
+            # Copy to application directory
+            dest_dir = os.path.join(target_base_dir, jurisdiction)
+            os.makedirs(dest_dir, exist_ok=True)
+            
+            src_dir = os.path.join(local_path, jurisdiction)
+            
+            if os.path.exists(src_dir):
+                files_copied = 0
+                total_size = 0
+                
+                for file in os.listdir(src_dir):
+                    src_file = os.path.join(src_dir, file)
+                    dest_file = os.path.join(dest_dir, file)
+                    
+                    # Copy file
+                    shutil.copy2(src_file, dest_file)
+                    file_size = os.path.getsize(dest_file)
+                    total_size += file_size
+                    files_copied += 1
+                    
+                    logger.info(f"📄 Copied: {file} ({file_size / (1024*1024):.1f} MB)")
+                
+                logger.info(f"✅ {jurisdiction}: {files_copied} files copied ({total_size / (1024*1024):.1f} MB)")
+            else:
+                logger.warning(f"⚠️  Jurisdiction directory not found in dataset: {src_dir}")
+        
+        logger.info("\n" + "=" * 80)
+        logger.info("🎉 Knowledge Graph Download Complete!")
+        logger.info("=" * 80)
+        
+        # Print summary
+        total_size = 0
+        for jurisdiction in jurisdictions:
+            jur_dir = os.path.join(target_base_dir, jurisdiction)
+            if os.path.exists(jur_dir):
+                jur_size = sum(os.path.getsize(os.path.join(jur_dir, f)) for f in os.listdir(jur_dir))
+                total_size += jur_size
+                logger.info(f"📊 {jurisdiction}: {jur_size / (1024*1024):.1f} MB")
+        
+        logger.info(f"\n💾 Total size: {total_size / (1024*1024):.1f} MB")
+        logger.info("=" * 80)
+        
+        return True
+        
+    except Exception as e:
+        logger.error("\n" + "=" * 80)
+        logger.error(f"❌ Error downloading knowledge graph: {e}")
+        logger.error("=" * 80)
+        return False
+
+
+if __name__ == "__main__":
+    success = download_knowledge_graph()
+    exit(0 if success else 1)

startup.sh

@@ -1,6 +1,22 @@
 #!/usr/bin/env bash
 set -euo pipefail
 
+# Step 1: Download knowledge graph from Hugging Face
+echo "📥 Checking for knowledge graph data..."
+if [ ! -d "data/rag_storage/romania" ] || [ ! -d "data/rag_storage/bahrain" ]; then
+    echo "🚀 Knowledge graph not found, downloading from Hugging Face..."
+    python scripts/download_knowledge_graph.py
+    if [ $? -ne 0 ]; then
+        echo "❌ Failed to download knowledge graph. Exiting."
+        exit 1
+    fi
+    echo "✅ Knowledge graph download complete"
+else
+    echo "✅ Knowledge graph data already present, skipping download"
+fi
+
+echo ""
+
 HOST="${LIGHTRAG_HOST:-127.0.0.1}"
 ROOT="${LIGHTRAG_STORAGE_ROOT:-data/rag_storage}"
 GRAPHS="${LIGHTRAG_GRAPHS:-romania:9621}"