refactor: Reorganize PDFs and improve error handling

- Move arXiv research papers to pdf_research/ directory
- Add IEA energy reports and publications to pdf_energy/
- Update README to reflect new document content (clean energy, renewables, EVs, CCUS)
- Create pdf symlink for default document source
- Improve security by hiding internal error details from users in stream output
- Replace print statements with proper logger.warning/error calls
- Remove TEST_REVIEW.md and .DS_Store files
- Add consistent trailing newlines to source files

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.coveragerc

@@ -21,3 +21,5 @@ exclude_lines =
     if TYPE_CHECKING:
     @abstractmethod
 
+
+

.gitignore

@@ -35,3 +35,5 @@ Thumbs.db
 vectorstore/
 *.log
 
+
+

README.md

@@ -34,20 +34,15 @@ This application provides an interactive chat interface that allows users to ask
 - ChromaDB for vector storage
 - Supports PDF document processing
 
-## Included Papers
-The following research papers are included in the `pdf` directory:
-
-- [arXiv:2306.13649v3](https://arxiv.org/abs/2306.13649)
-- [arXiv:2309.16609v1](https://arxiv.org/abs/2309.16609)
-- [arXiv:2312.06795v1](https://arxiv.org/abs/2312.06795)
-- [arXiv:2403.19522v1](https://arxiv.org/abs/2403.19522)
-- [arXiv:2405.04434v5](https://arxiv.org/abs/2405.04434)
-- [arXiv:2406.11617v1](https://arxiv.org/abs/2406.11617)
-- [arXiv:2410.21228v1](https://arxiv.org/abs/2410.21228)
-- [arXiv:2411.05059v2](https://arxiv.org/abs/2411.05059)
-- [arXiv:2501.09223v1](https://arxiv.org/abs/2501.09223)
-- [arXiv:2501.12948v1](https://arxiv.org/abs/2501.12948)
-- [arXiv:2503.04872v1](https://arxiv.org/abs/2503.04872)
+## Included Documents
+The `pdf` directory contains IEA (International Energy Agency) reports and publications covering topics such as:
+
+- Clean energy transitions and net zero pathways
+- Renewable energy capacity and projections
+- Electric vehicles and battery technologies
+- Carbon capture, utilisation and storage (CCUS)
+- Energy efficiency and critical minerals
+- Regional energy profiles and policy recommendations
 
 ## Deployment
 This application is hosted as a Hugging Face Space. Configuration details can be found in the [spaces config reference](https://huggingface.co/docs/hub/spaces-config-reference).

models.py

@@ -41,3 +41,5 @@ def create_embeddings():
         encode_kwargs={"normalize_embeddings": True},
     )
 
+
+

qa_chain.py

@@ -103,7 +103,7 @@ Rewritten query (keywords and key phrases only, be concise):"""
 
             return rewritten
         except Exception as e:
-            print(f"Query rewriting failed: {e}")
+            logger.warning(f"Query rewriting failed: {e}")
             return question
 
     def rerank_documents(self, query, documents, top_k=RETRIEVER_K):
@@ -138,7 +138,7 @@ Rewritten query (keywords and key phrases only, be concise):"""
 
             return [(doc, score) for doc, score in scored_docs[:top_k]]
         except Exception as e:
-            print(f"Re-ranking failed: {e}")
+            logger.warning(f"Re-ranking failed: {e}")
             # Fall back to original order
             return [(doc, None) for doc in documents[:top_k]]
 
@@ -363,8 +363,9 @@ Rewritten query (keywords and key phrases only, be concise):"""
                     "hybrid_scores": hybrid_scores,
                 }
         except Exception as e:
+            logger.error(f"Stream error: {e}")
             yield {
-                "chunk": f"\n\n[Error: {e}]",
+                "chunk": "\n\n[An error occurred. Please try again.]",
                 "source_documents": docs,
                 "docs_with_scores": docs_with_scores,
                 "rewritten_query": rewritten_query,

tests/TEST_REVIEW.md

@@ -1,370 +0,0 @@
-# Critical Test Suite Review
-
-## Executive Summary
-
-**Current Status:** 98.28% code coverage with 119 tests passing ✅
-
-**Overall Assessment:** The test suite is comprehensive and well-structured, but several important workflows and edge cases are missing that could impact production reliability.
-
----
-
-## ✅ Well-Covered Areas
-
-1. **Core Functionality**
-   - Unit tests for all major modules (config, models, utils, vectorstore, retrievers, qa_chain)
-   - Integration tests for RAG flow
-   - Edge cases for individual components
-
-2. **Search Strategies**
-   - MMR, Similarity, and Hybrid search types
-   - Query rewriting and re-ranking
-   - Document filtering
-
-3. **Error Handling**
-   - LLM errors
-   - Retriever errors
-   - Empty document scenarios
-
----
-
-## ❌ Missing Critical Workflows
-
-### 1. **Multi-Turn Conversations** ⚠️ HIGH PRIORITY
-**Issue:** No tests for chat history context in RAG queries
-
-**Missing Scenarios:**
-- RAG query with previous conversation context
-- Follow-up questions that reference previous answers
-- Context accumulation across multiple turns
-- Chat history limit enforcement (CHAT_HISTORY_LIMIT)
-
-**Impact:** Users rely on conversational context for follow-up questions. Without testing, context may be lost or incorrectly formatted.
-
-**Test Needed:**
-```python
-def test_rag_with_chat_history_context():
-    """Test RAG query with multi-turn conversation."""
-    # First question
-    # Follow-up question that references first answer
-    # Verify context is properly included
-```
-
----
-
-### 2. **Mode Switching Workflows** ⚠️ HIGH PRIORITY
-**Issue:** No tests for switching between RAG and Vanilla modes
-
-**Missing Scenarios:**
-- Start with RAG, switch to Vanilla mid-conversation
-- Start with Vanilla, switch to RAG mid-conversation
-- Chat history preservation across mode switches
-- UI state consistency during switches
-
-**Impact:** Users may lose context or get confused when switching modes.
-
-**Test Needed:**
-```python
-def test_mode_switching_preserves_history():
-    """Test that chat history is preserved when switching modes."""
-```
-
----
-
-### 3. **Empty Vectorstore Scenarios** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of empty/initial state
-
-**Missing Scenarios:**
-- First-time user (no vectorstore exists)
-- Vectorstore exists but is empty (no documents)
-- User queries before any documents are loaded
-- Graceful degradation when no documents available
-
-**Impact:** Application may crash or show confusing errors to new users.
-
-**Test Needed:**
-```python
-def test_rag_query_with_empty_vectorstore():
-    """Test RAG query when no documents are available."""
-    # Should return empty results or helpful message
-```
-
----
-
-### 4. **Document Update Workflow** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of adding new documents to existing vectorstore
-
-**Missing Scenarios:**
-- Adding new PDFs to existing vectorstore
-- Updating vectorstore with duplicate documents
-- Partial document loading failures
-- Concurrent document additions
-
-**Impact:** Users may not see new documents or experience data corruption.
-
-**Test Needed:**
-```python
-def test_vectorstore_update_with_new_documents():
-    """Test adding new documents to existing vectorstore."""
-    # Verify new documents are indexed
-    # Verify old documents remain
-    # Verify no duplicates
-```
-
----
-
-### 5. **Network/API Failure Scenarios** ⚠️ HIGH PRIORITY
-**Issue:** Limited testing of external service failures
-
-**Missing Scenarios:**
-- LLM server unavailable/timeout
-- Embedding model download failure
-- ChromaDB connection failures
-- Network interruptions during streaming
-- Retry logic for transient failures
-
-**Impact:** Application may hang or crash when external services fail.
-
-**Test Needed:**
-```python
-def test_llm_server_unavailable():
-    """Test graceful handling when LLM server is down."""
-    # Should show user-friendly error
-    # Should not crash application
-```
-
----
-
-### 6. **Input Validation & Edge Cases** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of invalid inputs
-
-**Missing Scenarios:**
-- Very long queries (>10k characters)
-- Empty queries (whitespace only)
-- Special characters in queries
-- Invalid document filter selections
-- Malformed chat history
-- Unicode/emoji in queries
-
-**Impact:** Application may crash or behave unexpectedly with edge case inputs.
-
-**Test Needed:**
-```python
-def test_very_long_query():
-    """Test handling of extremely long queries."""
-    
-def test_empty_whitespace_query():
-    """Test handling of whitespace-only queries."""
-    
-def test_invalid_document_filter():
-    """Test handling of invalid document filter selection."""
-```
-
----
-
-### 7. **Hybrid Search Parameter Variations** ⚠️ LOW PRIORITY
-**Issue:** Limited testing of hybrid alpha parameter
-
-**Missing Scenarios:**
-- Hybrid search with alpha=0.0 (pure keyword)
-- Hybrid search with alpha=1.0 (pure semantic)
-- Hybrid search with various alpha values (0.1, 0.5, 0.9)
-- Score fusion correctness at boundaries
-
-**Impact:** Users may not get optimal results with different alpha settings.
-
-**Test Needed:**
-```python
-def test_hybrid_search_alpha_boundaries():
-    """Test hybrid search with extreme alpha values."""
-```
-
----
-
-### 8. **Streaming Edge Cases** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of streaming failures
-
-**Missing Scenarios:**
-- Stream interruption mid-response
-- Partial stream completion
-- Multiple concurrent streams
-- Stream timeout handling
-- Memory leaks during long streams
-
-**Impact:** Users may experience incomplete responses or resource issues.
-
-**Test Needed:**
-```python
-def test_stream_interruption():
-    """Test handling of interrupted streams."""
-    
-def test_concurrent_streams():
-    """Test multiple concurrent streaming requests."""
-```
-
----
-
-### 9. **PDF Loading & Processing** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of PDF processing edge cases
-
-**Missing Scenarios:**
-- Corrupted PDF files
-- Password-protected PDFs
-- Very large PDF files (>100MB)
-- PDFs with no extractable text
-- PDFs with images only
-- Multiple PDFs with same filename
-
-**Impact:** Application may fail silently or crash when processing problematic PDFs.
-
-**Test Needed:**
-```python
-def test_corrupted_pdf_handling():
-    """Test handling of corrupted PDF files."""
-    
-def test_large_pdf_processing():
-    """Test processing of very large PDF files."""
-```
-
----
-
-### 10. **Vectorstore Persistence & Recovery** ⚠️ LOW PRIORITY
-**Issue:** No tests for persistence and recovery
-
-**Missing Scenarios:**
-- Vectorstore corruption detection
-- Recovery from corrupted vectorstore
-- Backup and restore workflows
-- Migration between vectorstore versions
-
-**Impact:** Users may lose their indexed documents or experience data corruption.
-
-**Test Needed:**
-```python
-def test_vectorstore_corruption_recovery():
-    """Test recovery from corrupted vectorstore."""
-```
-
----
-
-### 11. **UI/UX Workflows** ⚠️ MEDIUM PRIORITY
-**Issue:** Limited testing of UI interaction flows
-
-**Missing Scenarios:**
-- Clear button functionality
-- Example question selection
-- RAG toggle state persistence
-- Search type change during active query
-- Document filter change during active query
-- Context panel visibility toggling
-
-**Impact:** Users may experience UI bugs or confusion.
-
-**Test Needed:**
-```python
-def test_ui_clear_functionality():
-    """Test clear button resets all state correctly."""
-    
-def test_ui_state_consistency():
-    """Test UI state remains consistent during operations."""
-```
-
----
-
-### 12. **Performance & Scalability** ⚠️ LOW PRIORITY
-**Issue:** No performance tests
-
-**Missing Scenarios:**
-- Query latency with large document sets
-- Memory usage with many documents
-- Concurrent user handling
-- Large chat history performance
-
-**Impact:** Application may become slow or unresponsive with scale.
-
-**Test Needed:**
-```python
-def test_query_performance_large_dataset():
-    """Test query performance with large document sets."""
-```
-
----
-
-## 🔍 Additional Observations
-
-### Test Organization
-- ✅ Good separation of unit, integration, and edge case tests
-- ✅ Clear naming conventions
-- ⚠️ Some test files could be consolidated (multiple qa_chain test files)
-
-### Test Quality
-- ✅ Good use of fixtures and mocks
-- ✅ Comprehensive assertions
-- ⚠️ Some tests are too focused on implementation details rather than behavior
-
-### Coverage Gaps
-- **qa_chain.py lines 269-274:** Similarity search document matching (edge case)
-- **retrievers.py lines 66, 128, 179-185:** Hybrid search edge cases
-
----
-
-## 📋 Recommended Action Items
-
-### Immediate (Before Production)
-1. ✅ Add multi-turn conversation tests
-2. ✅ Add mode switching tests
-3. ✅ Add empty vectorstore tests
-4. ✅ Add network failure tests
-5. ✅ Add input validation tests
-
-### Short-term (Next Sprint)
-6. ✅ Add document update workflow tests
-7. ✅ Add streaming edge case tests
-8. ✅ Add PDF processing edge case tests
-9. ✅ Add UI workflow tests
-
-### Long-term (Future Enhancements)
-10. ✅ Add performance tests
-11. ✅ Add vectorstore persistence tests
-12. ✅ Add hybrid search parameter variation tests
-
----
-
-## 🎯 Priority Matrix
-
-| Workflow | Priority | Impact | Effort | Status |
-|----------|----------|--------|--------|--------|
-| Multi-turn conversations | HIGH | HIGH | MEDIUM | ❌ Missing |
-| Mode switching | HIGH | HIGH | LOW | ❌ Missing |
-| Network failures | HIGH | HIGH | MEDIUM | ⚠️ Partial |
-| Empty vectorstore | MEDIUM | MEDIUM | LOW | ⚠️ Partial |
-| Input validation | MEDIUM | MEDIUM | LOW | ⚠️ Partial |
-| Document updates | MEDIUM | MEDIUM | MEDIUM | ⚠️ Partial |
-| Streaming edge cases | MEDIUM | MEDIUM | MEDIUM | ⚠️ Partial |
-| PDF processing | MEDIUM | MEDIUM | MEDIUM | ⚠️ Partial |
-| UI workflows | MEDIUM | LOW | LOW | ⚠️ Partial |
-| Performance | LOW | LOW | HIGH | ❌ Missing |
-| Vectorstore persistence | LOW | LOW | HIGH | ❌ Missing |
-
----
-
-## 💡 Recommendations
-
-1. **Focus on User Journeys:** Add end-to-end tests that simulate real user workflows
-2. **Error Resilience:** Expand error handling tests to cover all failure modes
-3. **Integration Testing:** Add more integration tests that test multiple components together
-4. **Performance Baseline:** Establish performance benchmarks for critical paths
-5. **Test Documentation:** Document test scenarios and their business value
-
----
-
-## Conclusion
-
-The test suite provides excellent code coverage (98.28%) and covers most unit-level functionality well. However, several critical user workflows and edge cases are missing, particularly around:
-
-- **Multi-turn conversations** (critical for RAG use case)
-- **Mode switching** (core feature)
-- **Error resilience** (production readiness)
-- **Input validation** (user experience)
-
-Addressing these gaps will significantly improve production reliability and user experience.
-

tests/__init__.py

@@ -1,2 +1,4 @@
 """Test suite for the RAG application."""
 
+
+

tests/conftest.py

@@ -131,3 +131,5 @@ def mock_hybrid_results():
         },
     ]
 
+
+

tests/test_cli.py

@@ -215,3 +215,5 @@ def test_cli_main_rag_error(
     captured = capsys.readouterr()
     assert "Error" in captured.out
 
+
+

tests/test_config.py

@@ -54,3 +54,5 @@ def test_environment_variables():
     assert isinstance(os.getenv("CHROMA_PATH", "vectorstore"), str)
     assert isinstance(os.getenv("PDF_PATH", "pdf"), str)
 
+
+

tests/test_models.py

@@ -43,3 +43,5 @@ def test_create_embeddings(mock_embeddings):
     assert call_kwargs["encode_kwargs"]["normalize_embeddings"] is True
     assert result == mock_instance
 
+
+

tests/test_qa_chain.py

@@ -241,7 +241,7 @@ def test_stream_error(mock_format_history, mock_create_llm, qa_chain_wrapper, mo
 
     results = list(qa_chain_wrapper.stream(inputs))
     assert len(results) > 0
-    assert "Error" in results[0]["chunk"]
+    assert "error occurred" in results[0]["chunk"]
 
 
 def test_create_qa_chain(mock_vectorstore):

tests/test_qa_chain_final.py

@@ -64,3 +64,5 @@ def test_stream_similarity_exact_match(mock_format_history, mock_create_llm, qa_
     # Verify that the matching logic was executed (lines 269-274)
     assert results[0].get("docs_with_scores") is not None
 
+
+

tests/test_retrievers.py

@@ -137,3 +137,5 @@ def test_hybrid_search_semantic_error(mock_vectorstore, sample_documents):
     # Should still return results from BM25
     assert isinstance(results, list)
 
+
+

tests/test_retrievers_edge_cases.py

@@ -77,3 +77,5 @@ def test_hybrid_search_negative_distance(mock_vectorstore):
 
         assert len(results) > 0
 
+
+

tests/test_utils.py

@@ -131,3 +131,5 @@ def test_format_context_with_hybrid_scores(sample_documents):
     assert "s:" in result  # Semantic score
     assert "k:" in result  # Keyword score
 
+
+

tests/test_utils_edge_cases.py

@@ -43,3 +43,5 @@ def test_format_context_no_scores():
     assert "Test content" in result
     assert "⭐" in result  # First chunk should be highlighted by default
 
+
+

tests/test_vectorstore.py

@@ -182,3 +182,5 @@ def test_create_new_vectorstore(mock_chroma, mock_loader, mock_get_pdfs, mock_em
         assert result == mock_vectorstore
         mock_chroma.from_documents.assert_called_once()
 
+
+

utils.py

@@ -143,3 +143,5 @@ def format_context_with_highlight(
 
     return query_info + sources_header + "\n\n".join(formatted_chunks)
 
+
+