backend/RAG_MODERNIZATION.md

# RMI RAG Modernization — 2026 Standards
# ======================================
# Design document for upgrading RMI's RAG system to production-grade
# modern standards. Based on audit of all 40+ endpoints, 4 pipelines,
# 9 collections, and 3 embedders.

## Current State Audit

### Collections (crypto_embeddings.py)
wallet_profiles, token_analysis, scam_patterns, forensic_reports,
market_intel, contract_audits, known_scams, news_articles,
transaction_patterns

### Embedders (INCONSISTENT — 3 different models)
- nomic-embed-text (768d) — rag_engine.py, smart_ai_engine.py
- bge-m3 (1024d) — rag_ingestion.py, rag_supreme.py
- bge-small-en-v1.5 (384d) — crypto_embeddings.py (primary)

### Pipelines (4 separate, overlapping)
- rag_engine.py — Qdrant REST API, nomic-embed-text, 5 collections
- rag_service.py — FAISS ANN, bge-small, 9 collections, 3-pillar search
- rag_supreme.py — 15-win pipeline, bge-m3, 5 Qdrant collections
- rag_firehose.py — continuous ingestion engine (designed, not fully wired)

### Gaps Identified
1. NO historical scam ingestion (Rekt DB, Chainabuse, DeFi hacks)
2. NO structured chunking — raw text embedding, no overlap
3. NO evaluation running (RAGAS mentioned, not active)
4. Embedding model inconsistency across pipelines
5. Firehose sources not wired (cadences defined, fetchers missing)
6. NO query transformation in production path
7. NO feedback loop active
8. Redis SCARD bug (FIXED 2026-06-17)
9. FAISS disk indexes exist but Redis backing data evicted for 7/9 collections

## Modern Standards (2025-2026 Industry Consensus)

### 1. Chunking Strategy
- DEFAULT: Recursive character splitting, 512 tokens, 15% overlap
- For code: add class/function boundary separators
- For news: sentence-based chunking preserves coherence
- For scam reports: semantic chunking on topic boundaries
- Overlap: 10-20% (test for your domain — some studies show no benefit)

### 2. Embedding Models
- STANDARDIZE on bge-m3 (1024d) — best open-source, multilingual
- Fallback: bge-small-en-v1.5 (384d) for fast/local
- Multi-head: different dims for different content types
  - Contract code: 128d structural features (already in crypto_embeddings.py)
  - Scam patterns: 384d behavioral embedding
  - News/articles: 1024d semantic (bge-m3)
  - Wallet profiles: 64d behavioral fingerprint

### 3. Retrieval Architecture
- HYBRID: Dense (70%) + BM25/Sparse (30%) — 5-15% recall improvement
- RRF fusion (Reciprocal Rank Fusion) — proven best for hybrid
- Cross-encoder rerank: top-20 → rerank → top-5
- MMR dedup: remove near-duplicate results
- Query expansion: generate 3 variants, fuse results

### 4. Ingestion Pipeline (UNIFIED)
- SINGLE entry point: POST /api/v1/rag/ingest
- Pipeline: Parse → Chunk → Dedup → Classify → Embed → Store → Index
- Dedup: content hash in Redis (MD5 of normalized text)
- Quality filter: skip docs below quality threshold
- Rate limiting: per-collection docs/minute
- Batch embedding: groups of 25-50, async

### 5. Historical Data Sources (NEW)
- Rekt DB (de.fi/rekt-database) — 3,000+ DeFi hacks since 2020
- Chainabuse — scam reports with addresses
- TRM Labs Crypto Crime Report — annual typologies
- Elliptic State of Crypto Scams — annual report
- Chainalysis Crypto Crime Report — annual trends
- SlowMist Hacked Archive — detailed exploit analysis
- Immunefi Bug Bounty Reports — vulnerability patterns
- CertiK Audit Findings — smart contract vulnerabilities
- Solana Compromised Accounts — known drained wallets
- Etherscan Labels — 115K+ labeled addresses (already have)

### 6. Evaluation Framework
- RAGAS metrics: faithfulness, answer_relevancy, context_precision, context_recall
- Golden test set: 50 known scam queries with expected answers
- Run weekly, alert on regression
- Track: Hit@5, MRR, NDCG@10

### 7. Feedback Loop
- Scanner hits → boost source weight
- False positives → penalize
- User corrections → update embeddings
- Track helpful docs, boost in future searches

## Implementation Plan

### Phase 1: Standardize & Consolidate (NOW)
1. Standardize embedder: bge-m3 (1024d) primary, bge-small (384d) fallback
2. Add recursive chunking to ingest pipeline
3. Wire firehose sources (Rekt DB, Chainabuse, Etherscan labels)
4. Add content hash dedup to all ingestion paths

### Phase 2: Historical Data Ingestion (THIS WEEK)
5. Build Rekt DB scraper → forensic_reports collection
6. Build Chainabuse scraper → known_scams collection
7. Ingest TRM/Elliptic/Chainalysis annual reports → market_intel
8. Ingest SlowMist/Immunefi/CertiK findings → contract_audits

### Phase 3: Evaluation & Feedback (NEXT WEEK)
9. Activate RAGAS evaluation pipeline
10. Build golden test set (50 queries)
11. Wire feedback loop (scanner hits → boost)
12. Add query transformation (HyDE, expansion)

### Phase 4: Advanced Retrieval (ONGOING)
13. Cross-encoder reranking (bge-reranker-v2-m3)
14. Parent-child retrieval for long documents
15. Multi-modal: code + text + transaction patterns
16. Streaming response for agentic investigation

## New Unified Ingestion Pipeline

```
POST /api/v1/rag/ingest
{
  "documents": [...],
  "collection": "known_scams",
  "source": "rekt_db",
  "chunking": "recursive"  // or "semantic", "sentence", "none"
}

Pipeline:
1. PARSE — extract text, metadata, entities
2. CHUNK — recursive split (512 tokens, 15% overlap)
3. DEDUP — MD5 hash check against Redis
4. QUALITY — score content, skip if < threshold
5. CLASSIFY — route to correct collection
6. EMBED — batch embed via bge-m3 (Ollama)
7. STORE — Redis (hot) + FAISS (index) + R2 (cold)
8. INDEX — update ANN index version
```

## New Collections to Add

| Collection | Source | Dims | Purpose |
|-----------|--------|------|---------|
| defi_hacks | Rekt DB, SlowMist | 1024d | Historical DeFi exploits |
| rug_timeline | Chainabuse, SENTINEL | 1024d | Rug pull chronology |
| vuln_patterns | Immunefi, CertiK | 1024d | Smart contract vulnerabilities |
| crime_reports | TRM, Elliptic, Chainalysis | 1024d | Annual crime typologies |
| compromised_wallets | Solana, Etherscan | 384d | Known drained addresses |
| exploit_techniques | All sources | 1024d | How hacks were executed |

## Success Metrics

- RAG total_docs: 2,473 → 50,000+ (20x)
- Collections with data: 2/9 → 9/9 + 6 new
- Embedding consistency: 3 models → 1 primary + 1 fallback
- Ingestion cadence: ad-hoc → continuous (firehose)
- Evaluation: none → weekly RAGAS
- Chunking: none → recursive 512-token
- Dedup: none → content hash
- Cold storage: partial → full R2 permanence