docs/CONFLUENCE_SETUP_GUIDE.md

# Confluence Integration Setup Guide

This guide walks you through setting up Confluence integration for the Fraud Model Explainability Assistant.

---

## Prerequisites

- Access to a Confluence instance (Atlassian Cloud or Data Center)
- Permission to create API tokens
- Permission to read the Confluence spaces you want to search

---

## Step 1: Get Confluence API Credentials

### Option A: Atlassian Cloud (Most Common)

1. **Get your Confluence URL**:
   - Go to your Confluence instance
   - Copy the URL (e.g., `https://yourcompany.atlassian.net`)

2. **Create an API Token**:
   - Go to https://id.atlassian.com/manage-profile/security/api-tokens
   - Click **"Create API token"**
   - Give it a label (e.g., "Fraud Assistant")
   - Copy the token (you won't be able to see it again!)

3. **Get your email**:
   - Use the email address associated with your Atlassian account
   - Recommendation: Create a service account (e.g., `fraud-bot@yourcompany.com`)

### Option B: Confluence Data Center (Self-Hosted)

1. **Get your Confluence URL**:
   - Your internal Confluence URL (e.g., `https://confluence.yourcompany.com`)

2. **Create a Personal Access Token**:
   - Go to Confluence → Profile → Personal Access Tokens
   - Click **"Create token"**
   - Give it a name and select appropriate permissions
   - Copy the token

3. **Authentication**:
   - For Data Center, you may use username/password or PAT
   - See confluence-ingestor documentation for details

---

## Step 2: Configure Environment Variables

### Create .env File

```bash
cd fraud_model_explainability_assistant

# Copy the example file
cp .env.example .env

# Edit with your credentials
nano .env  # or use your preferred editor
```

### Edit .env File

```bash
# Confluence Configuration
CONFLUENCE_URL=https://yourcompany.atlassian.net  # YOUR Confluence URL
CONFLUENCE_EMAIL=your.email@company.com           # YOUR email
CONFLUENCE_API_TOKEN=ATATT3xFfGF0...              # YOUR API token

# Embedding Provider (free, local)
EMBEDDING_PROVIDER=huggingface

# Vector Store
VECTOR_STORE_TYPE=chroma
CHROMA_PERSIST_DIRECTORY=./chroma_db

# Optional: OpenAI (if not using Bedrock)
OPENAI_API_KEY=sk-...  # YOUR OpenAI key (optional)
```

**Important**: Never commit `.env` to git! It contains secrets.

---

## Step 3: Install Dependencies

```bash
# Make sure you're in the fraud assistant directory
cd fraud_model_explainability_assistant

# Install confluence-ingestor with Strands adapter
pip install confluence-ingestor[strands,huggingface,chroma]

# Verify installation
python -c "from confluence_ingestor import ConfluenceRAG; print('✅ Installation successful')"
```

---

## Step 4: Identify Confluence Spaces to Ingest

### Find Your Spaces

1. Go to Confluence → Spaces → View All Spaces
2. Identify spaces with relevant documentation:
   - Model governance documentation
   - Compliance policies
   - Fraud investigation procedures
   - Training materials
   - Regulatory guidelines

### Recommended Spaces for Fraud Assistant

| Space Purpose | Example Space Key | Priority |
|---------------|-------------------|----------|
| Model Governance | `fraud-model-governance` | ⭐⭐⭐⭐⭐ |
| Compliance Policies | `compliance-policies` | ⭐⭐⭐⭐⭐ |
| Investigation Playbooks | `fraud-investigation` | ⭐⭐⭐⭐ |
| Regulatory Prep | `regulatory-compliance` | ⭐⭐⭐⭐ |
| Team Procedures | `fraud-analytics-team` | ⭐⭐⭐ |

### Get Space Keys

1. Go to a Confluence space
2. Look at the URL: `https://yourcompany.atlassian.net/wiki/spaces/SPACEKEY/`
3. The `SPACEKEY` is what you need

---

## Step 5: Test Confluence Connection

### Quick Connection Test

```bash
cd fraud_model_explainability_assistant

# Test Confluence connection
python -c "
from confluence_ingestor import ConfluenceClient

client = ConfluenceClient.from_env()
spaces = client.list_spaces()

print('✅ Connected to Confluence!')
print(f'Found {len(spaces)} spaces')
for space in spaces[:5]:
    print(f'  - {space.key}: {space.name}')
"
```

**Expected Output**:
```
✅ Connected to Confluence!
Found 15 spaces
  - FMG: Fraud Model Governance
  - CP: Compliance Policies
  - FI: Fraud Investigation
  ...
```

**If you see errors**:
- `401 Unauthorized`: Check your email and API token
- `404 Not Found`: Check your Confluence URL
- `403 Forbidden`: You don't have permission to access Confluence

---

## Step 6: Ingest Confluence Spaces

### Initial Ingestion (One-Time Setup)

This downloads your Confluence content and creates a searchable vector database.

```bash
cd fraud_model_explainability_assistant

python << 'EOF'
from confluence_ingestor import ConfluenceRAG

# Initialize RAG pipeline
print("Initializing Confluence RAG...")
rag = ConfluenceRAG.from_env(
    embedding_provider="huggingface",
    vector_store_type="chroma"
)

# Ingest spaces (this may take 5-10 minutes)
spaces = {
    "fraud-model-governance": 100,  # Adjust space keys and limits
    "compliance-policies": 50,
    "fraud-investigation": 75,
}

for space_key, max_pages in spaces.items():
    print(f"\nIngesting {space_key} (max {max_pages} pages)...")
    try:
        stats = rag.ingest_space(space_key, max_pages=max_pages, force=False)

        if stats.get("skipped"):
            print(f"  ⊙ {stats['reason']}")
        else:
            print(f"  ✅ Ingested {stats['pages']} pages")
            print(f"     Created {stats['chunks']} chunks")
    except Exception as e:
        print(f"  ❌ Error: {e}")

print("\n✅ Ingestion complete!")
print("Vector database saved to: ./chroma_db/")
EOF
```

**Expected Output**:
```
Initializing Confluence RAG...
Downloading model... (first time only, ~400MB)

Ingesting fraud-model-governance (max 100 pages)...
  ✅ Ingested 87 pages
     Created 1,245 chunks

Ingesting compliance-policies (max 50 pages)...
  ✅ Ingested 42 pages
     Created 628 chunks

✅ Ingestion complete!
Vector database saved to: ./chroma_db/
```

**Time**: 5-10 minutes for ~300 pages (first time only)

---

## Step 7: Test the Integration

### Test Search Functionality

```bash
python << 'EOF'
from confluence_ingestor import ConfluenceRAG

rag = ConfluenceRAG.from_env()

# Test search
results = rag.search("fair lending policy synthetic ID", k=3)

print(f"Found {len(results)} results:\n")
for i, result in enumerate(results, 1):
    print(f"{i}. {result.title}")
    print(f"   Space: {result.space_key}")
    print(f"   Score: {result.score:.4f}")
    print(f"   Content: {result.content[:150]}...")
    print(f"   Source: {result.source}\n")
EOF
```

**Expected Output**:
```
Found 3 results:

1. Fair Lending Policy - Synthetic Identity Detection
   Space: compliance-policies
   Score: 0.8523
   Content: Our fair lending policy addresses synthetic identity detection through the following guidelines...
   Source: https://yourcompany.atlassian.net/wiki/spaces/CP/pages/12345

2. Model Validation Report - XGBoost v3.2
   Space: fraud-model-governance
   Score: 0.7891
   Content: The model validation for XGBoost v3.2 includes fair lending compliance testing...
   Source: https://yourcompany.atlassian.net/wiki/spaces/FMG/pages/67890

3. Investigation Playbook - Synthetic ID Fraud
   Space: fraud-investigation
   Score: 0.7654
   Content: When investigating synthetic identity fraud, follow these procedures...
   Source: https://yourcompany.atlassian.net/wiki/spaces/FI/pages/11111
```

---

## Step 8: Run the Enhanced Fraud Assistant

```bash
# Run the Confluence-integrated version
python confluence_integration_example.py
```

**The app will**:
1. Initialize Confluence RAG (reads from `./chroma_db/`)
2. Create the Confluence search tool
3. Add it to the Strands agent
4. Launch the Gradio UI

**Test it**:
1. Open http://localhost:7860
2. Ask: "What does our fair lending policy say about phone type features?"
3. The agent should search Confluence and cite specific policy documents!

---

## Troubleshooting

### Error: "3 validation errors for ConfluenceConfig"

**Cause**: Environment variables not loaded

**Fix**:
```bash
# Check if .env exists
ls -la .env

# If missing, create it from .env.example
cp .env.example .env

# Edit with your credentials
nano .env

# Verify variables are set
python -c "import os; from dotenv import load_dotenv; load_dotenv(); print('URL:', os.getenv('CONFLUENCE_URL'))"
```

### Error: "401 Unauthorized"

**Cause**: Invalid email or API token

**Fix**:
1. Re-generate your API token at https://id.atlassian.com/manage-profile/security/api-tokens
2. Double-check your email is correct
3. Make sure there are no extra spaces in `.env`

### Error: "404 Not Found"

**Cause**: Incorrect Confluence URL

**Fix**:
1. Check your Confluence URL format
2. For Atlassian Cloud: `https://yourcompany.atlassian.net` (no `/wiki` suffix)
3. For Data Center: `https://confluence.yourcompany.com`

### Error: "403 Forbidden"

**Cause**: User doesn't have permission to access Confluence

**Fix**:
1. Log into Confluence with the same email/token
2. Verify you can view the spaces you're trying to ingest
3. Contact your Confluence admin to grant read access

### Error: "Module not found: confluence_ingestor"

**Cause**: Package not installed

**Fix**:
```bash
pip install confluence-ingestor[strands,huggingface,chroma]
```

### Error: "Slow search queries"

**Cause**: Large vector database or slow embeddings

**Fix**:
1. Reduce `max_pages` when ingesting
2. Consider using OpenAI embeddings (faster but costs money)
3. Filter searches to specific spaces

### Error: "Out of disk space"

**Cause**: Vector database and model cache are large

**Fix**:
1. Vector database: ~150 MB per 100 pages
2. HuggingFace model: ~400 MB (one-time)
3. Free up space or use a smaller corpus

---

## Maintenance

### Update Confluence Content (Daily/Weekly)

```bash
# Re-ingest spaces to pick up new content
python << 'EOF'
from confluence_ingestor import ConfluenceRAG

rag = ConfluenceRAG.from_env()

# Force re-ingestion
for space in ["fraud-model-governance", "compliance-policies"]:
    print(f"Updating {space}...")
    stats = rag.ingest_space(space, max_pages=100, force=True)
    print(f"  ✅ Updated {stats['pages']} pages")
EOF
```

### Automated Updates (Optional)

Add to your deployment:

```python
from apscheduler.schedulers.background import BackgroundScheduler

def refresh_confluence():
    rag = ConfluenceRAG.from_env()
    for space in ["fraud-model-governance", "compliance-policies"]:
        rag.ingest_space(space, max_pages=100, force=True)

scheduler = BackgroundScheduler()
scheduler.add_job(refresh_confluence, 'cron', hour=2)  # 2 AM daily
scheduler.start()
```

---

## Security Best Practices

### 1. Use a Service Account

Create a dedicated Confluence user:
- Email: `fraud-bot@yourcompany.com`
- Name: "Fraud Model Assistant"
- Purpose: Read-only access to specific spaces

### 2. Limit Permissions

Grant minimal access:
- ✅ Read access to relevant spaces
- ❌ No write access
- ❌ No admin access

### 3. Secure Credentials

- ✅ Store in `.env` (never commit to git)
- ✅ Use environment variables in production
- ✅ Rotate API tokens regularly
- ❌ Never hard-code credentials

### 4. Add to .gitignore

```bash
# Add to .gitignore
echo ".env" >> .gitignore
echo "chroma_db/" >> .gitignore
echo "*.log" >> .gitignore
```

---

## Next Steps

Once setup is complete:

1. ✅ Test with simple queries
2. ✅ Add more Confluence spaces as needed
3. ✅ Customize space filters for different use cases
4. ✅ Share with your team
5. ✅ Monitor usage and performance
6. ✅ Set up automated updates

---

## Support

### Documentation

- **confluence-ingestor**: See `/confluence-ingestor/README.md`
- **Strands Agents**: https://strandsagents.com/latest/documentation/docs/

### Common Issues

Check `CONFLUENCE_INTEGRATION_ANALYSIS.md` for:
- Architecture details
- Performance optimization
- Advanced configuration
- ROI analysis

---

## Summary Checklist

- [ ] Create Confluence API token
- [ ] Create `.env` file with credentials
- [ ] Install `confluence-ingestor[strands,huggingface,chroma]`
- [ ] Test Confluence connection
- [ ] Identify relevant Confluence spaces
- [ ] Ingest spaces (5-10 minutes)
- [ ] Test search functionality
- [ ] Run enhanced fraud assistant
- [ ] Add to `.gitignore`
- [ ] Share with team

**Estimated Time**: 30-60 minutes (including ingestion)

---

**Need Help?** Check the troubleshooting section or review the detailed analysis in `CONFLUENCE_INTEGRATION_ANALYSIS.md`.