CLAUDE.md

# 🏛️ CIDADÃO.AI - CONTEXTO GERAL DO PROJETO

**⚠️ HEADER UNIVERSAL - NÃO REMOVER - Atualizado: Janeiro 2025**

## 🎯 VISÃO GERAL DO ECOSSISTEMA

O **Cidadão.AI** é um ecossistema de **4 repositórios especializados** que trabalham em conjunto para democratizar a transparência pública brasileira através de IA avançada:

### 📦 REPOSITÓRIOS DO ECOSSISTEMA
- **cidadao.ai-backend** → API + Sistema Multi-Agente + ML Pipeline
- **cidadao.ai-frontend** → Interface Web + Internacionalização
- **cidadao.ai-docs** → Hub de Documentação + Landing Page
- **cidadao.ai-models** → Modelos IA + Pipeline MLOps (ESTE REPOSITÓRIO)

### 🤖 SISTEMA MULTI-AGENTE (17 Agentes)
1. **MasterAgent (Abaporu)** - Orquestração central com auto-reflexão
2. **InvestigatorAgent** - Detecção de anomalias em dados públicos
3. **AnalystAgent** - Análise de padrões e correlações
4. **ReporterAgent** - Geração inteligente de relatórios
5. **SecurityAuditorAgent** - Auditoria e compliance
6. **CommunicationAgent** - Comunicação inter-agentes
7. **CorruptionDetectorAgent** - Detecção de corrupção
8. **PredictiveAgent** - Análise preditiva
9. **VisualizationAgent** - Visualizações de dados
10. **BonifacioAgent** - Contratos públicos
11. **DandaraAgent** - Diversidade e inclusão
12. **MachadoAgent** - Processamento de linguagem natural
13. **SemanticRouter** - Roteamento inteligente
14. **ContextMemoryAgent** - Sistema de memória
15. **ETLExecutorAgent** - Processamento de dados
16. **ObserverAgent** - Monitoramento
17. **ValidatorAgent** - Validação de qualidade

### 🏗️ ARQUITETURA TÉCNICA
- **Score Geral**: 9.3/10 (Classe Enterprise)
- **Backend**: FastAPI + Python 3.11+ + PostgreSQL + Redis + ChromaDB
- **Frontend**: Next.js 15 + React 19 + TypeScript + Tailwind CSS 4
- **Deploy**: Docker + Kubernetes + SSL + Monitoring
- **IA**: LangChain + Transformers + OpenAI/Groq + Vector DBs

### 🛡️ SEGURANÇA E AUDITORIA
- **Multi-layer security** com middleware especializado
- **JWT + OAuth2 + API Key** authentication
- **Audit trail** completo com severity levels
- **Rate limiting** + **CORS** + **SSL termination**

### 🎯 MISSÃO E IMPACTO
- **Democratizar** acesso a análises de dados públicos
- **Detectar anomalias** e irregularidades automaticamente
- **Empoderar cidadãos** com informação clara e auditável
- **Fortalecer transparência** governamental via IA ética

### 📊 STATUS DO PROJETO
- **Versão**: 1.0.0 (Production-Ready)
- **Score Técnico**: 9.3/10
- **Cobertura de Testes**: 23.6% (Target: >80%)
- **Deploy**: Kubernetes + Vercel + HuggingFace Spaces

---

# CLAUDE.md - MODELOS IA

Este arquivo fornece orientações para o Claude Code ao trabalhar com os modelos de IA e pipeline MLOps do Cidadão.AI.

## 🤖 Visão Geral dos Modelos IA

**Cidadão.AI Models** é o repositório responsável pelos modelos de machine learning, pipeline MLOps e infraestrutura de IA que alimenta o sistema multi-agente. Este repositório gerencia treinamento, versionamento, deploy e monitoramento dos modelos especializados em transparência pública.

**Status Atual**: **Pipeline MLOps em Desenvolvimento** - Infraestrutura para modelos personalizados, integração com HuggingFace Hub e pipeline de treinamento automatizado.

## 🏗️ Análise Arquitetural Modelos IA

### **Score Geral dos Modelos: 7.8/10** (Pipeline em Construção)

O **Repositório de Modelos Cidadão.AI** representa uma **base sólida para MLOps** especializado em análise de transparência pública. O sistema está preparado para hospedar modelos customizados e integrar-se com o ecossistema de agentes.

### 📊 Métricas Técnicas Modelos
- **Framework**: PyTorch + Transformers + HuggingFace
- **MLOps**: MLflow + DVC + Weights & Biases
- **Deploy**: HuggingFace Spaces + Docker containers
- **Monitoring**: Model performance tracking + drift detection
- **Storage**: HuggingFace Hub + cloud storage integration
- **CI/CD**: Automated training + testing + deployment

### 🚀 Componentes Planejados (Score 7-8/10)
- **Model Registry**: 7.8/10 - HuggingFace Hub integration
- **Training Pipeline**: 7.5/10 - Automated training workflow
- **Model Serving**: 7.7/10 - FastAPI + HuggingFace Spaces
- **Monitoring**: 7.3/10 - Performance tracking system
- **Version Control**: 8.0/10 - Git + DVC + HuggingFace

### 🎯 Componentes em Desenvolvimento (Score 6-7/10)
- **Custom Models**: 6.8/10 - Domain-specific fine-tuning
- **Data Pipeline**: 6.5/10 - ETL for training data
- **Evaluation**: 6.7/10 - Automated model evaluation
- **A/B Testing**: 6.3/10 - Model comparison framework

## 🧠 Arquitetura de Modelos

### **Modelos Especializados Planejados**
```python
# Taxonomy dos Modelos Cidadão.AI
models_taxonomy = {
    "corruption_detection": {
        "type": "classification",
        "base_model": "bert-base-multilingual-cased",
        "specialization": "Brazilian Portuguese + government documents",
        "use_case": "Detect corruption indicators in contracts"
    },
    "anomaly_detection": {
        "type": "regression + classification", 
        "base_model": "Custom ensemble",
        "specialization": "Financial data patterns",
        "use_case": "Identify unusual spending patterns"
    },
    "entity_extraction": {
        "type": "NER",
        "base_model": "roberta-large",
        "specialization": "Government entities + Brazilian names",
        "use_case": "Extract companies, people, organizations"
    },
    "sentiment_analysis": {
        "type": "classification",
        "base_model": "distilbert-base-uncased",
        "specialization": "Public opinion on transparency",
        "use_case": "Analyze citizen feedback sentiment"
    },
    "summarization": {
        "type": "seq2seq",
        "base_model": "t5-base",
        "specialization": "Government reports + legal documents",
        "use_case": "Generate executive summaries"
    }
}
```

### **Pipeline MLOps Architecture**
```yaml
# MLOps Workflow
stages:
  data_collection:
    - Portal da Transparência APIs
    - Government databases
    - Public procurement data
    - Historical investigations
    
  data_preprocessing:
    - Data cleaning & validation
    - Privacy anonymization
    - Feature engineering
    - Data augmentation
    
  model_training:
    - Hyperparameter optimization
    - Cross-validation
    - Ensemble methods
    - Transfer learning
    
  model_evaluation:
    - Performance metrics
    - Fairness evaluation
    - Bias detection
    - Interpretability analysis
    
  model_deployment:
    - HuggingFace Spaces
    - Container deployment
    - API endpoints
    - Model serving
    
  monitoring:
    - Model drift detection
    - Performance degradation
    - Data quality monitoring
    - Usage analytics
```

## 🔬 Modelos de IA Especializados

### **1. Corruption Detection Model**
```python
# Modelo especializado em detecção de corrupção
class CorruptionDetector:
    base_model: "bert-base-multilingual-cased"
    fine_tuned_on: "Brazilian government contracts + known corruption cases"
    
    features:
        - Contract language analysis
        - Pricing anomaly detection
        - Vendor relationship patterns
        - Temporal irregularities
    
    metrics:
        - Precision: >85%
        - Recall: >80% 
        - F1-Score: >82%
        - False Positive Rate: <5%
```

### **2. Anomaly Detection Ensemble**
```python
# Ensemble para detecção de anomalias financeiras
class AnomalyDetector:
    models:
        - IsolationForest: "Outlier detection"
        - LSTM: "Temporal pattern analysis"
        - Autoencoder: "Reconstruction error"
        - Random Forest: "Feature importance"
    
    features:
        - Amount deviation from median
        - Vendor concentration
        - Seasonal patterns
        - Geographic distribution
    
    output:
        - Anomaly score (0-1)
        - Confidence interval
        - Explanation vector
        - Risk category
```

### **3. Entity Recognition (NER)**
```python
# NER especializado para entidades governamentais
class GovernmentNER:
    base_model: "roberta-large"
    entities:
        - ORGANIZATION: "Ministérios, órgãos, empresas"
        - PERSON: "Servidores, políticos, empresários"
        - LOCATION: "Estados, municípios, endereços"
        - CONTRACT: "Números de contratos, licitações"
        - MONEY: "Valores monetários, moedas"
        - DATE: "Datas de contratos, vigências"
    
    brazilian_specialization:
        - CPF/CNPJ recognition
        - Brazilian address patterns
        - Government terminology
        - Legal document structure
```

## 🚀 HuggingFace Integration

### **Model Hub Strategy**
```python
# HuggingFace Hub Organization
organization: "cidadao-ai"
models:
    - "cidadao-ai/corruption-detector-pt"
    - "cidadao-ai/anomaly-detector-financial"
    - "cidadao-ai/ner-government-entities"
    - "cidadao-ai/sentiment-transparency"
    - "cidadao-ai/summarization-reports"

spaces:
    - "cidadao-ai/corruption-demo"
    - "cidadao-ai/anomaly-dashboard"
    - "cidadao-ai/transparency-analyzer"
```

### **Model Cards Template**
```markdown
# Model Card: Cidadão.AI Corruption Detector

## Model Description
- **Developed by**: Cidadão.AI Team
- **Model type**: BERT-based binary classifier
- **Language**: Portuguese (Brazil)
- **License**: MIT

## Training Data
- **Sources**: Portal da Transparência + curated corruption cases
- **Size**: 100K+ government contracts
- **Preprocessing**: Anonymization + cleaning + augmentation

## Evaluation
- **Test Set**: 10K held-out contracts
- **Metrics**: Precision: 87%, Recall: 83%, F1: 85%
- **Bias Analysis**: Evaluated across regions + contract types

## Ethical Considerations
- **Intended Use**: Transparency analysis, not legal evidence
- **Limitations**: May have bias toward certain contract types
- **Risks**: False positives could damage reputations
```

## 🛠️ MLOps Pipeline

### **Training Infrastructure**
```yaml
# training-pipeline.yml
name: Model Training Pipeline
on:
  schedule:
    - cron: '0 2 * * 0'  # Weekly retraining
  workflow_dispatch:

jobs:
  data_preparation:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch latest data
      - name: Validate data quality
      - name: Preprocess & augment
      
  model_training:
    runs-on: gpu-runner
    steps:
      - name: Hyperparameter optimization
      - name: Train model
      - name: Evaluate performance
      
  model_deployment:
    runs-on: ubuntu-latest
    if: model_performance > threshold
    steps:
      - name: Upload to HuggingFace Hub
      - name: Update model registry
      - name: Deploy to production
```

### **Model Monitoring Dashboard**
```python
# Métricas de monitoramento
monitoring_metrics = {
    "performance": {
        "accuracy": "Real-time accuracy tracking",
        "latency": "Response time monitoring", 
        "throughput": "Requests per second",
        "error_rate": "Failed prediction rate"
    },
    "data_drift": {
        "feature_drift": "Input distribution changes",
        "label_drift": "Output distribution changes",
        "concept_drift": "Relationship changes"
    },
    "business": {
        "investigations_triggered": "Anomalies detected",
        "false_positive_rate": "User feedback tracking",
        "citizen_satisfaction": "User experience metrics"
    }
}
```

## 🧪 Experimentação e Avaliação

### **Experiment Tracking**
```python
# MLflow + Weights & Biases integration
import mlflow
import wandb

def train_model(config):
    with mlflow.start_run():
        wandb.init(project="cidadao-ai", config=config)
        
        # Log hyperparameters
        mlflow.log_params(config)
        wandb.config.update(config)
        
        # Training loop
        for epoch in range(config.epochs):
            metrics = train_epoch(model, train_loader)
            
            # Log metrics
            mlflow.log_metrics(metrics, step=epoch)
            wandb.log(metrics)
        
        # Log model artifacts
        mlflow.pytorch.log_model(model, "model")
        wandb.save("model.pt")
```

### **A/B Testing Framework**
```python
# Framework para testes A/B de modelos
class ModelABTest:
    def __init__(self, model_a, model_b, traffic_split=0.5):
        self.model_a = model_a
        self.model_b = model_b
        self.traffic_split = traffic_split
        
    def predict(self, input_data, user_id):
        # Route traffic based on user_id hash
        if hash(user_id) % 100 < self.traffic_split * 100:
            result = self.model_a.predict(input_data)
            self.log_prediction("model_a", result, user_id)
        else:
            result = self.model_b.predict(input_data)
            self.log_prediction("model_b", result, user_id)
        
        return result
```

## 📊 Datasets e Treinamento

### **Datasets Especializados**
```python
# Datasets para treinamento
datasets = {
    "transparency_contracts": {
        "source": "Portal da Transparência API",
        "size": "500K+ contracts",
        "format": "JSON + PDF text extraction",
        "labels": "Manual annotation + expert review"
    },
    "corruption_cases": {
        "source": "Historical investigations + court records",
        "size": "10K+ labeled cases",
        "format": "Structured data + documents",
        "labels": "Binary classification + severity"
    },
    "financial_anomalies": {
        "source": "Government spending data",
        "size": "1M+ transactions", 
        "format": "Tabular data",
        "labels": "Statistical outliers + domain expert"
    }
}
```

### **Data Preprocessing Pipeline**
```python
# Pipeline de preprocessamento
class DataPreprocessor:
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
        self.anonymizer = GovernmentDataAnonymizer()
        
    def preprocess_contract(self, contract_text):
        # 1. Anonymize sensitive information
        anonymized = self.anonymizer.anonymize(contract_text)
        
        # 2. Clean and normalize text
        cleaned = self.clean_text(anonymized)
        
        # 3. Tokenize for model input
        tokens = self.tokenizer(
            cleaned,
            max_length=512,
            truncation=True,
            padding=True,
            return_tensors="pt"
        )
        
        return tokens
```

## 🔄 Integração com Backend

### **Model Serving API**
```python
# FastAPI endpoints para servir modelos
from fastapi import FastAPI
from transformers import pipeline

app = FastAPI()

# Load models
corruption_detector = pipeline(
    "text-classification",
    model="cidadao-ai/corruption-detector-pt"
)

anomaly_detector = joblib.load("models/anomaly_detector.pkl")

@app.post("/analyze/corruption")
async def detect_corruption(contract_text: str):
    result = corruption_detector(contract_text)
    return {
        "prediction": result[0]["label"],
        "confidence": result[0]["score"],
        "model_version": "v1.0.0"
    }

@app.post("/analyze/anomaly")
async def detect_anomaly(financial_data: dict):
    features = extract_features(financial_data)
    anomaly_score = anomaly_detector.predict(features)
    return {
        "anomaly_score": float(anomaly_score),
        "is_anomaly": anomaly_score > 0.7,
        "explanation": generate_explanation(features)
    }
```

### **Agent Integration**
```python
# Integração com sistema multi-agente
class ModelService:
    def __init__(self):
        self.models = {
            "corruption": self.load_corruption_model(),
            "anomaly": self.load_anomaly_model(),
            "ner": self.load_ner_model()
        }
    
    async def analyze_for_agent(self, agent_name: str, data: dict):
        if agent_name == "InvestigatorAgent":
            return await self.detect_anomalies(data)
        elif agent_name == "CorruptionDetectorAgent":
            return await self.detect_corruption(data)
        elif agent_name == "AnalystAgent":
            return await self.extract_entities(data)
```

## 🔒 Ética e Governança

### **Responsible AI Principles**
```python
# Princípios de IA Responsável
class ResponsibleAI:
    principles = {
        "transparency": "Explicabilidade em todas as decisões",
        "fairness": "Avaliação de viés em grupos demográficos",
        "privacy": "Anonimização de dados pessoais",
        "accountability": "Auditoria e rastreabilidade",
        "robustness": "Teste contra adversarial attacks"
    }
    
    def evaluate_bias(self, model, test_data, protected_attributes):
        """Avalia viés do modelo em grupos protegidos"""
        bias_metrics = {}
        for attr in protected_attributes:
            group_metrics = self.compute_group_metrics(model, test_data, attr)
            bias_metrics[attr] = group_metrics
        return bias_metrics
```

### **Model Interpretability**
```python
# Ferramentas de interpretabilidade
from lime.lime_text import LimeTextExplainer
from shap import Explainer

class ModelExplainer:
    def __init__(self, model):
        self.model = model
        self.lime_explainer = LimeTextExplainer()
        self.shap_explainer = Explainer(model)
    
    def explain_prediction(self, text, method="lime"):
        if method == "lime":
            explanation = self.lime_explainer.explain_instance(
                text, self.model.predict_proba
            )
        elif method == "shap":
            explanation = self.shap_explainer(text)
        
        return explanation
```

## 📋 Roadmap Modelos IA

### **Curto Prazo (1-2 meses)**
1. **Setup MLOps Pipeline**: MLflow + DVC + CI/CD
2. **Corruption Detection Model**: Fine-tune BERT para português
3. **HuggingFace Integration**: Upload initial models
4. **Basic Monitoring**: Performance tracking dashboard

### **Médio Prazo (3-6 meses)**
1. **Anomaly Detection Ensemble**: Multiple algorithms
2. **NER Government Entities**: Custom entity recognition
3. **Model A/B Testing**: Production experimentation
4. **Advanced Monitoring**: Drift detection + alerting

### **Longo Prazo (6+ meses)**
1. **Custom Architecture**: Domain-specific model architectures
2. **Federated Learning**: Privacy-preserving training
3. **AutoML Pipeline**: Automated model selection
4. **Edge Deployment**: Local model inference

## ⚠️ Áreas para Melhoria

### **Priority 1: Data Pipeline**
- **Data Collection**: Automated data ingestion
- **Data Quality**: Validation + cleaning pipelines
- **Labeling**: Active learning + human-in-the-loop
- **Privacy**: Advanced anonymization techniques

### **Priority 2: Model Development**
- **Custom Models**: Domain-specific architectures
- **Transfer Learning**: Portuguese government domain
- **Ensemble Methods**: Model combination strategies
- **Optimization**: Model compression + acceleration

### **Priority 3: MLOps Maturity**
- **CI/CD**: Automated testing + deployment
- **Monitoring**: Comprehensive drift detection
- **Experimentation**: A/B testing framework
- **Governance**: Model audit + compliance

## 🎯 Métricas de Sucesso

### **Technical Metrics**
- **Model Performance**: F1 > 85% for all models
- **Inference Latency**: <200ms response time
- **Deployment Success**: >99% uptime
- **Data Pipeline**: <1% data quality issues

### **Business Metrics**
- **Anomalies Detected**: 100+ monthly
- **False Positive Rate**: <5%
- **User Satisfaction**: >80% positive feedback
- **Investigation Success**: >70% actionable insights

## 🔧 Comandos de Desenvolvimento

### **Model Training**
```bash
# Train corruption detection model
python train_corruption_detector.py --config configs/corruption_bert.yaml

# Evaluate model performance  
python evaluate_model.py --model corruption_detector --test_data data/test.json

# Upload to HuggingFace Hub
python upload_to_hub.py --model_path models/corruption_detector --repo_name cidadao-ai/corruption-detector-pt
```

### **Monitoring**
```bash
# Check model drift
python monitor_drift.py --model corruption_detector --window 7d

# Generate performance report
python generate_report.py --models all --period monthly
```

## 📝 Considerações Técnicas

### **Compute Requirements**
- **Training**: GPU-enabled instances (V100/A100)
- **Inference**: CPU instances sufficient for most models
- **Storage**: Cloud storage for datasets + model artifacts
- **Monitoring**: Real-time metrics collection

### **Security**
- **Model Protection**: Encrypted model artifacts
- **API Security**: Authentication + rate limiting
- **Data Privacy**: LGPD compliance + anonymization
- **Audit Trail**: Complete lineage tracking

### **Scalability**
- **Horizontal Scaling**: Load balancer + multiple instances
- **Model Versioning**: Backward compatibility
- **Cache Strategy**: Redis for frequent predictions
- **Batch Processing**: Async inference for large datasets

---

**Models Status**: Pipeline em desenvolvimento com infraestrutura sólida para modelos especializados.
**Next Update**: Implementação do primeiro modelo de detecção de corrupção e pipeline MLOps completo.