fix: Improve DuckDuckGo error handling + deployment tools

- Better error handling for DecodeError and rate limits
- Graceful fallback to database when live search fails
- Added upgrade_dependencies.sh script
- Added DEPLOYMENT_FIXES.md guide

Run ./upgrade_dependencies.sh to fix reranker

DEPLOYMENT_FIXES.md

@@ -0,0 +1,151 @@
+# Deployment Fixes for RAG API v2.4
+
+## Issues Fixed
+
+### 1. ✅ Qdrant `order_by` Error (FIXED IN CODE)
+- **Status:** Fixed in commit `7110008`
+- **Action:** Already deployed, no action needed
+
+### 2. ✅ DuckDuckGo Decode Error (FIXED IN CODE)
+- **Status:** Fixed with better error handling
+- **Action:** Deploy latest code
+- **Details:** System now gracefully handles decode errors and falls back to database
+
+### 3. ⚠️ Reranker Tokenizer Error (NEEDS DEPLOYMENT)
+- **Status:** Fix ready, needs dependency upgrade
+- **Action:** Run upgrade script (see below)
+
+---
+
+## Deployment Steps
+
+### Option A: Automatic Upgrade (Recommended)
+
+```bash
+cd services/apis/rag-api
+chmod +x upgrade_dependencies.sh
+./upgrade_dependencies.sh
+```
+
+### Option B: Manual Upgrade
+
+```bash
+pip install --upgrade FlagEmbedding>=1.2.11
+pip install --upgrade transformers>=4.45.0
+```
+
+### Option C: Fresh Install
+
+```bash
+cd services/apis/rag-api
+pip install -r requirements.txt --upgrade
+```
+
+---
+
+## Verification
+
+After deployment, check logs for:
+
+### ✅ Success Indicators:
+```
+# No more Qdrant errors
+✓ No "order_by not supported" warnings
+
+# No more reranker errors  
+✓ No "XLMRobertaTokenizer has no attribute prepare_for_model" errors
+
+# DuckDuckGo graceful handling
+✓ "DuckDuckGo decode error (likely rate limit)" → Falls back to DB
+✓ System continues working with database results
+```
+
+### ❌ If Still Broken:
+```bash
+# Check installed versions
+pip show FlagEmbedding transformers
+
+# Should show:
+# FlagEmbedding: 1.2.11 or higher
+# transformers: 4.45.0 or higher
+
+# If not, force reinstall:
+pip uninstall FlagEmbedding transformers -y
+pip install FlagEmbedding==1.2.11 transformers==4.45.0
+```
+
+---
+
+## Testing
+
+### Test 1: Check Reranker
+```bash
+# Query the API and check logs
+curl -X POST http://localhost:7860/api/v1/rag/chat/stream \
+  -H "Content-Type: application/json" \
+  -d '{"query": "Ethiopia news today"}'
+
+# Check logs - should NOT see:
+# ❌ "Reranker scoring failed"
+
+# Should see:
+# ✅ "DEBUG: After hybrid ranking: 15 results"
+```
+
+### Test 2: Check DuckDuckGo
+```bash
+# Query with temporal intent
+curl -X POST http://localhost:7860/api/v1/rag/chat/stream \
+  -H "Content-Type: application/json" \
+  -d '{"query": "breaking news today"}'
+
+# Check logs:
+# ✅ "DEBUG: Hybrid search returned X DB + Y live results"
+# If Y=0, check for:
+# ✅ "DuckDuckGo decode error (likely rate limit)" → Graceful fallback
+```
+
+### Test 3: Check News Endpoint
+```bash
+# Test browse endpoint
+curl http://localhost:7860/api/v1/news/latest?limit=10
+
+# Should return:
+# ✅ 200 OK with recent articles
+# ✅ No 400 errors in logs
+```
+
+---
+
+## Rollback Plan
+
+If issues occur after deployment:
+
+```bash
+# Rollback to previous versions
+pip install FlagEmbedding==1.2.5 transformers==4.40.0
+
+# Or rollback code
+git revert 7110008
+git push
+```
+
+---
+
+## Support
+
+If issues persist:
+1. Check logs for specific error messages
+2. Verify Python version (should be 3.10+)
+3. Check if running in virtual environment
+4. Try fresh install in clean environment
+
+---
+
+## Version History
+
+- **v2.4** (2026-05-04): Fixed Qdrant order_by, DuckDuckGo errors, upgraded reranker
+- **v2.3** (2026-05-04): Interactive thinking indicator with Lucide icons
+- **v2.2** (2026-05-03): Time-based filtering for news endpoint
+- **v2.1** (2026-05-03): Query enhancements and flexible prompting
+- **v2.0** (2026-05-02): Hybrid RAG with live search

src/infrastructure/adapters/duckduckgo_adapter.py

@@ -153,8 +153,19 @@ class DuckDuckGoAdapter:
                     results.append(normalized)
             
         except Exception as e:
-            logger.error(f"DuckDuckGo API error: {e}")
-            raise
+            # Handle specific DuckDuckGo errors gracefully
+            error_msg = str(e)
+            if "DecodeError" in error_msg or "Body collection error" in error_msg:
+                logger.warning(f"DuckDuckGo decode error (likely rate limit or API issue): {e}")
+                # Return empty results instead of raising - system will use database fallback
+                return []
+            elif "No results found" in error_msg:
+                logger.debug(f"DuckDuckGo: No results for query '{query[:50]}'")
+                return []
+            else:
+                logger.error(f"DuckDuckGo API error: {e}")
+                # Return empty results for graceful degradation
+                return []
         
         return results
     

upgrade_dependencies.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+# Upgrade script for fixing reranker tokenizer issue
+# Run this on HF Spaces or production server
+
+echo "=========================================="
+echo "Upgrading RAG API Dependencies"
+echo "=========================================="
+
+echo ""
+echo "📦 Upgrading FlagEmbedding (reranker fix)..."
+pip install --upgrade FlagEmbedding>=1.2.11
+
+echo ""
+echo "📦 Upgrading transformers (tokenizer fix)..."
+pip install --upgrade transformers>=4.45.0
+
+echo ""
+echo "✅ Dependencies upgraded successfully!"
+echo ""
+echo "🔄 Please restart the service for changes to take effect:"
+echo "   - HF Spaces: Will auto-restart"
+echo "   - Docker: docker-compose restart rag-api"
+echo "   - Systemd: systemctl restart rag-api"
+echo ""
+echo "=========================================="