README.md

metadata

title: SafeRAG Demo
emoji: 🤖
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.0.0
app_file: app.py
pinned: false
license: apache-2.0

SafeRAG: High-Performance Calibrated RAG

A production-ready Retrieval-Augmented Generation (RAG) system with risk calibration, built on the Hugging Face ecosystem.

🚀 Key Features

• Risk Calibration: Multi-layer risk assessment with adaptive strategies
• High Performance: Optimized for 2-3.5x throughput improvement
• Hugging Face Native: Built on HF Datasets, Models, and Spaces
• Production Ready: Complete pipeline with error handling and monitoring

🏗️ Architecture

HF Datasets → Embedding (BGE/E5) → FAISS Index
Query → Batched Retrieval → Evidence Selector → Generator (vLLM + gpt-oss-20b)
→ Risk Calibration → Adaptive Strategy → Output (Answer + Citations + Risk Score)

📊 Performance Targets

• QA Accuracy: EM/F1 improvements over vanilla RAG
• Attribution: +8-12pt improvement in citation precision/recall
• Calibration: 30-40% reduction in ECE (Expected Calibration Error)
• Throughput: 2-3.5x improvement with vLLM

🛠️ Quick Start

Run Tests

python3 simple_e2e_test.py

Start Demo

python3 app.py

📈 Evaluation

The system has been tested with comprehensive end-to-end tests:

• ✅ Text processing and sentence extraction
• ✅ Embedding creation and similarity calculation
• ✅ Passage retrieval and reranking
• ✅ Risk feature extraction and prediction
• ✅ Risk-aware answer generation
• ✅ Evaluation metrics (EM, F1, ROUGE)
• ✅ Complete end-to-end RAG pipeline

🔧 Configuration

Key parameters in config.yaml:

• Risk Thresholds: τ₁ = 0.3, τ₂ = 0.7
• Retrieval: k = 20, rerank_k = 10
• Generation: max_tokens = 512, temperature = 0.7
• Calibration: 16 features, logistic regression

🎯 Risk Calibration

Risk Features (16-dimensional)

1. Retrieval Statistics: Similarity scores, variance, diversity
2. Coverage Features: Token/entity overlap between Q&A
3. Consistency Features: Semantic similarity between passages
4. Diversity Features: Topic variance, passage diversity

Adaptive Strategies

• Low Risk (r < τ₁): Normal generation
• Medium Risk (τ₁ ≤ r < τ₂): Conservative generation + citations
• High Risk (r ≥ τ₂): Very conservative or refuse

📚 Datasets

• HotpotQA: Multi-hop reasoning with supporting facts
• TriviaQA: Open-domain QA for general knowledge
• Wikipedia: Knowledge base via HF Datasets

📄 Citation

@article{safrag2024,
  title={SafeRAG: High-Performance Calibrated RAG with Risk Assessment},
  author={Your Name},
  journal={arXiv preprint},
  year={2024}
}

📝 License

Apache 2.0 License - see LICENSE file for details.

---

SafeRAG: A production-ready RAG system with risk calibration, built on Hugging Face ecosystem.