Enhance RAG processing and UI components

1. Updated `run_retrieval_step` to accept an optional `original_query` parameter for improved document retrieval.
2. Refactored `retrieve_documents` to be asynchronous, enhancing performance during document queries.
3. Introduced a new `format_source_html` function for consistent HTML formatting of source documents in the chat UI.
4. Improved mixed language text handling in the chat display, ensuring proper RTL support for Hebrew content.
5. Added logging for better debugging and error handling in the RAG processing pipeline.
6. Enhanced sidebar functionality with improved language and font selection management.
7. Updated task tracking guidelines in `memorybank/tasks.md` to reflect recent project developments.
8. Removed outdated `rules_compliance.md` file to streamline documentation.

.cursor/rules/task-tracking.mdc

@@ -0,0 +1,118 @@
+---
+description: 
+globs: 
+alwaysApply: true
+---
+# Task Tracking Guidelines
+
+## Overview
+This rule provides guidelines for maintaining and updating [memorybank/tasks.md](mdc:memorybank/tasks.md) throughout the development process. The task list should be updated after each significant process step to maintain accurate project status.
+
+## Update Requirements
+
+### After Each Process Step
+1. Update "Last Updated" date in Executive Summary
+2. Adjust task counts in Executive Summary
+3. Move completed tasks to appropriate sections
+4. Update progress indicators (✓, [ ], [x])
+5. Add any new tasks discovered during the process
+
+### Section-Specific Updates
+
+#### Executive Summary
+- Update Project Status if phase changes
+- Recalculate Pending and Completed task counts
+- Update Complexity Level if assessment changes
+
+#### Branch Achievements
+- Add new achievements with ✓ prefix
+- Group achievements by category
+- Include specific implementation details
+
+#### Active Development Tasks
+- Update priority levels if changed
+- Add new technical debt items discovered
+- Update security improvements needed
+- Add new UX improvement requirements
+- Maintain clear issue references (e.g., RET-DYNAMIC-IMPORTS)
+
+#### Project Notes
+- Add timestamped entries for significant changes
+- Update rules compliance status
+- Document new issues identified
+- Track implementation progress
+
+#### Task Archive
+- Move completed tasks with completion timestamps
+- Maintain categorization of archived tasks
+- Include brief completion notes if relevant
+
+## Update Process
+
+### Pre-Update Checks
+1. Review current task status
+2. Identify completed tasks
+3. List new tasks discovered
+4. Note any priority changes
+
+### Update Steps
+1. Update Executive Summary metrics
+2. Add new achievements if applicable
+3. Update Active Development Tasks
+   - Mark completed tasks
+   - Add new tasks
+   - Update priorities
+4. Add relevant Project Notes
+5. Move completed tasks to Archive
+
+### Post-Update Validation
+1. Verify all sections are properly formatted
+2. Check timestamp accuracy
+3. Validate task counts match
+4. Ensure proper categorization
+
+## Format Requirements
+
+### Task Format
+```markdown
+- [ ] **Task Name** (ISSUE-REFERENCE)
+  - File: `filename.py` (Line XX)
+  - Issue: Brief description
+  - Solution: Proposed solution
+```
+
+### Achievement Format
+```markdown
+- ✓ Achievement description with specific details
+```
+
+### Project Note Format
+```markdown
+- Note description (YYYY-MM-DD HH:MM:SS)
+```
+
+## Automation Hooks
+
+### Pre-Commit
+- Validate task.md format
+- Update task counts
+- Verify timestamps
+
+### Post-Process
+- Check for completed tasks
+- Update achievement list
+- Recalculate metrics
+
+## Error Prevention
+- Maintain consistent formatting
+- Use proper timestamp format
+- Keep issue references consistent
+- Preserve existing categories
+- Maintain proper indentation
+
+## Best Practices
+1. Update tasks.md immediately after process completion
+2. Include specific file references when relevant
+3. Maintain clear issue tracking references
+4. Keep achievement descriptions specific and measurable
+5. Use consistent formatting throughout the document

components/chat.py

@@ -3,113 +3,40 @@ from typing import Dict, Any, List
 import asyncio
 import logging
 import traceback
+import nest_asyncio
+
+# Apply nest_asyncio once during module import
+nest_asyncio.apply()
 
 # Setup logger
 logger = logging.getLogger(__name__)
 
-def display_chat_message(message: Dict[str, Any]):
-    """
-    Display a chat message with proper formatting.
-    
-    Args:
-        message (Dict[str, Any]): Message object with role, content, and optional metadata
-    """
-    # Import here to avoid circular imports
-    from i18n import get_direction, get_text
-    from utils.sanitization import sanitize_html
-    from utils import clean_source_text
-    
-    text_direction = get_direction()
-    role = message.get("role", "assistant")
-    
-    # Ensure the current font is used
-    hebrew_font = st.session_state.hebrew_font
-    
-    with st.chat_message(role):
-        # Sanitize the message content for HTML rendering
-        content = message.get('content', '')
-        if isinstance(content, str):
-            content = sanitize_html(content)
-            
-            # Add font-family style if not already present
-            if 'font-family' not in content and role == "assistant" and text_direction == "rtl":
-                content = f"<div style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important; direction: {text_direction};'>{content}</div>"
-        
-        st.markdown(content, unsafe_allow_html=True)
-        
-        if role == "assistant" and message.get("final_docs"):
-            docs = message["final_docs"]
-            # Use a simple text title for the expander
-            with st.expander(f"{get_text('sources_title')} ({len(docs)})", expanded=False):
-                # Add the rich HTML content inside the expander (static HTML is safe)
-                expander_title = f"<div class='expander-title {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{get_text('sources_text').format(len(docs))}</div>"
-                st.markdown(expander_title, unsafe_allow_html=True)
-                st.markdown(f"<div dir='{text_direction}' class='expander-content' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>", unsafe_allow_html=True)
-                for i, doc in enumerate(docs, start=1):
-                    source = doc.get('source_name', '') or get_text('unknown_source')
-                    # Sanitize the source text
-                    source = sanitize_html(source)
-                    
-                    # Clean and sanitize the Hebrew text
-                    text = doc.get('hebrew_text', '')
-                    text = clean_source_text(text)
-                    text = sanitize_html(text)
-                    
-                    source_html = f"<div class='source-info {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'><strong>{get_text('source_label').format(i)}</strong> {source}</div>"
-                    text_html = f"<div class='hebrew-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{text}</div>"
-                    
-                    st.markdown(source_html, unsafe_allow_html=True)
-                    st.markdown(text_html, unsafe_allow_html=True)
-                st.markdown("</div>", unsafe_allow_html=True)
-
-
-def display_status_updates(status_log: List[str]):
-    """
-    Display processing status log in an expander.
-    
-    Args:
-        status_log (List[str]): List of status update messages
-    """
-    # Import here to avoid circular imports
-    from i18n import get_direction, get_text
-    
-    text_direction = get_direction()
-    hebrew_font = st.session_state.hebrew_font
-    
-    if status_log:
-        # Use a simple text title for the expander
-        with st.expander(get_text('processing_details'), expanded=False):
-            # Add the rich HTML content inside the expander
-            st.markdown(f"<div class='expander-title {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{get_text('processing_log')}</div>", unsafe_allow_html=True)
-            for u in status_log:
-                st.markdown(
-                    f"<code class='status-update {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>- {u}</code>",
-                    unsafe_allow_html=True
-                )
-
+# Import our refactored modules
+from ui.hebrew import handle_mixed_language_text
+from ui.chat_render import display_chat_message, display_status_updates, format_source_html
+from pipeline.rag import process_rag_request, create_async_execution_context, extract_citations
 
 def process_prompt(prompt: str, rag_params: Dict[str, Any]):
     """
     Process a user prompt and generate a response.
     
     Args:
-        prompt (str): User input prompt
+        prompt (str): User input prompt (may contain template)
         rag_params (Dict[str, Any]): RAG parameters from sidebar
     """
     # Import here to avoid circular imports
     from i18n import get_direction, get_text
     from utils.sanitization import sanitize_html
-    from utils import clean_source_text
-    from services.openai_service import extract_citations_with_openai
-    from rag_processor import execute_validate_generate_pipeline, PIPELINE_VALIDATE_GENERATE_GPT4O
-    import nest_asyncio
+    from rag_processor import PIPELINE_VALIDATE_GENERATE_GPT4O
+    
+    # Initialize session state if needed
+    if 'messages' not in st.session_state:
+        st.session_state.messages = []
     
     text_direction = get_direction()
     hebrew_font = st.session_state.hebrew_font
     
-    # Ensure we can run async code
-    nest_asyncio.apply()
-    
+    # Add the visible prompt to chat history (what the user sees)
     st.session_state.messages.append({"role": "user", "content": prompt})
     display_chat_message(st.session_state.messages[-1])
 
@@ -120,21 +47,23 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
         try:
             def status_cb(m): status_container.update(label=f"{get_text('processing_step')} {m}")
             def stream_cb(c):
-                # Sanitize the chunk before appending and displaying
+                # Sanitize the chunk before appending
                 if isinstance(c, str):
                     c = sanitize_html(c)
                 chunks.append(c)
-                # Sanitize the entire response for rendering
-                sanitized_response = sanitize_html(''.join(chunks))
-                msg_placeholder.markdown(
-                    f"<div dir='{text_direction}' class='{text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{sanitized_response}▌</div>",
-                    unsafe_allow_html=True
-                )
+                # Process the entire response with mixed language handler first
+                joined_text = ''.join(chunks) + "▌"  # Add cursor
+                display_html = handle_mixed_language_text(joined_text, hebrew_font)
+                # Sanitize the final HTML to prevent injection
+                safe_html = sanitize_html(display_html)
+                msg_placeholder.markdown(safe_html, unsafe_allow_html=True)
 
             try:
-                loop = asyncio.get_event_loop()
+                # Get the current event loop or create a new one
+                loop = create_async_execution_context()
+                
                 final_rag = loop.run_until_complete(
-                    execute_validate_generate_pipeline(
+                    process_rag_request(
                         history=st.session_state.messages,
                         params=rag_params,
                         status_callback=status_cb,
@@ -142,21 +71,23 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                     )
                 )
                 
-                # Try to get cited IDs after getting the response
+                # Extract citations after getting the response
                 cited_ids = []
                 if isinstance(final_rag, dict):
                     raw = final_rag.get("final_response", "")
                     # Only attempt to extract citations if we have a valid response
-                    if raw:
-                        try:
-                            cited_ids = loop.run_until_complete(extract_citations_with_openai(raw))
-                        except Exception as citation_err:
-                            # Log citation extraction error but don't fail the entire request
-                            st.error(f"Citation extraction failed: {citation_err}", icon="⚠️")
-                            cited_ids = []
+                    if raw and raw.strip():
+                        cited_ids = extract_citations(raw)
             except (RuntimeError, asyncio.CancelledError, asyncio.TimeoutError) as loop_err:
                 st.error(f"{get_text('error_async')} {loop_err}", icon="⚠️")
-                err_html = f"<div dir='{text_direction}' class='{text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'><strong>{get_text('request_error')}</strong><br>{get_text('error_async')}<br>{type(loop_err).__name__}</div>"
+                # Format error message with RTL support
+                err_html = f"""
+                <div dir='{text_direction}' class='{text_direction}-text hebrew-font'>
+                    <strong>{get_text('request_error')}</strong><br>
+                    {get_text('error_async')}<br>
+                    {type(loop_err).__name__}
+                </div>
+                """
                 # Sanitize error HTML
                 err_html = sanitize_html(err_html)
                 msg_placeholder.markdown(err_html, unsafe_allow_html=True)
@@ -181,15 +112,14 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                 docs = final_rag.get("generator_input_documents", [])
                 pipeline = final_rag.get("pipeline_used", PIPELINE_VALIDATE_GENERATE_GPT4O)
 
-                # wrap result
-                final = raw
-                if not (err and final.strip().startswith("<div")) and not final.strip().startswith((
-                    '<div', '<p', '<ul', '<ol', '<strong'
-                )):
-                    final = f"<div dir='{text_direction}' class='{text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{final or get_text('no_response')}</div>"
+                # Process final content with mixed language handler
+                if not (err and raw.strip().startswith("<div")):
+                    final = handle_mixed_language_text(raw, hebrew_font)
+                    # Final sanitization after processing
+                    final = sanitize_html(final)
+                else:
+                    final = sanitize_html(raw)
                 
-                # Final sanitization before rendering
-                final = sanitize_html(final)
                 msg_placeholder.markdown(final, unsafe_allow_html=True)
 
                 # Use the citations extracted earlier instead of making another async call
@@ -202,25 +132,25 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                 if docs_to_show:
                     # Use a simple text title for the expander
                     with st.expander(f"{get_text('sources_title')} ({len(docs_to_show)})", expanded=False):
-                        # Add the rich HTML content inside the expander
-                        expander_title = f"<div class='expander-title {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{get_text('sources_text').format(len(docs_to_show))}</div>"
-                        st.markdown(sanitize_html(expander_title), unsafe_allow_html=True)
-                        st.markdown(sanitize_html(f"<div dir='{text_direction}' class='expander-content' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>"), unsafe_allow_html=True)
+                        # Add RTL Hebrew wrapper with embed
+                        expander_title = f"""
+                        <div class='expander-title rtl-text hebrew-font' dir='rtl' lang="he">
+                            {get_text('sources_text').format(len(docs_to_show))}
+                        </div>
+                        """
+                        st.markdown(expander_title, unsafe_allow_html=True)
+                        
+                        # Container for all sources with RTL direction
+                        st.markdown(f"""
+                        <div dir='rtl' lang="he" class='expander-content rtl-text hebrew-font'>
+                        """, unsafe_allow_html=True)
+                        
+                        # Format each source consistently using our helper function
                         for idx, doc in docs_to_show:
-                            source = doc.get('source_name', '') or get_text('unknown_source')
-                            # Sanitize the source text
-                            source = sanitize_html(source)
-                            
-                            # Clean and sanitize the Hebrew text
-                            text = doc.get('hebrew_text', '')
-                            text = clean_source_text(text)
-                            text = sanitize_html(text)
-                            
-                            source_html = f"<div class='source-info {text_direction}-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'><strong>{get_text('source_label').format(idx)}</strong> {source}</div>"
-                            text_html = f"<div class='hebrew-text' style='font-family: \"{hebrew_font}\", \"Open Sans Hebrew\", \"Alef Hebrew\", \"Arial Hebrew\", sans-serif !important;'>{text}</div>"
-                            
+                            source_html, text_html = format_source_html(doc, idx, hebrew_font, get_text)
                             st.markdown(source_html, unsafe_allow_html=True)
                             st.markdown(text_html, unsafe_allow_html=True)
+                        
                         st.markdown("</div>", unsafe_allow_html=True)
 
                 # store message
@@ -239,10 +169,14 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                 else:
                     status_container.update(label=get_text('processing_complete'), state="complete", expanded=False)
             else:
-                msg_placeholder.markdown(
-                    f"<div dir='{text_direction}' class='{text_direction}-text'><strong>{get_text('communication_error')}</strong></div>",
-                    unsafe_allow_html=True
-                )
+                # Format communication error message with proper RTL support
+                err_msg = f"""
+                <div dir='{text_direction}' class='{text_direction}-text hebrew-font'>
+                    <strong>{get_text('communication_error')}</strong>
+                </div>
+                """
+                msg_placeholder.markdown(sanitize_html(err_msg), unsafe_allow_html=True)
+                
                 st.session_state.messages.append({
                     "role": "assistant",
                     "content": get_text('communication_error'),
@@ -254,12 +188,19 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                 status_container.update(label=f"{get_text('error')}!", state="error", expanded=False)
 
         except Exception as e:
-            logger.error(f"Unhandled exception in RAG processing: {e}", exc_info=True)
+            logger.exception("Unhandled exception in RAG processing")
             traceback.print_exc()
-            err_html = (
-                f"<div dir='{text_direction}' class='{text_direction}-text'><strong>{get_text('critical_error')}</strong><br>{get_text('reload')}"  
-                f"<details><summary>{get_text('details')}</summary><pre>{sanitize_html(traceback.format_exc())}</pre></details></div>"
-            )
+            # Format critical error with RTL support
+            err_html = f"""
+            <div dir='{text_direction}' class='{text_direction}-text hebrew-font'>
+                <strong>{get_text('critical_error')}</strong><br>
+                {get_text('reload')}
+                <details>
+                    <summary>{get_text('details')}</summary>
+                    <pre>{sanitize_html(traceback.format_exc())}</pre>
+                </details>
+            </div>
+            """
             # Sanitize error HTML
             err_html = sanitize_html(err_html)
             msg_placeholder.error(err_html, icon="🔥")
@@ -271,4 +212,4 @@ def process_prompt(prompt: str, rag_params: Dict[str, Any]):
                 "status_log": [f"Critical: {type(e).__name__}"],
                 "error": str(e)
             })
-            status_container.update(label=str(e), state="error", expanded=False) 
+            status_container.update(label=get_text('processing_error'), state="error", expanded=False) 

components/sidebar.py

@@ -1,6 +1,9 @@
 import streamlit as st
 from typing import Dict, Any
-import time
+import logging
+
+# Setup logger
+logger = logging.getLogger(__name__)
 
 def display_sidebar() -> Dict[str, Any]:
     """
@@ -17,89 +20,85 @@ def display_sidebar() -> Dict[str, Any]:
     
     text_direction = get_direction()
     
-    # Use Streamlit's native sidebar instead of custom toggle
+    # Use Streamlit's native sidebar
     with st.sidebar:
-        st.markdown(f"<h3 class='{text_direction}-text'>{get_text('settings_title')}</h3>", unsafe_allow_html=True)
-        
-        # Language selection
-        st.markdown(f"<h4 class='{text_direction}-text'>{get_text('display_settings')}</h4>", unsafe_allow_html=True)
+        # All sidebar header items
+        st.header(get_text('settings_title'))
+        st.subheader(get_text('display_settings'))
         
-        # Convert language codes to display names for the selector
+        # Language selector
         language_options = {code: name for code, name in LANGUAGES.items()}
+        current_language_index = list(language_options.keys()).index(st.session_state.language) if st.session_state.language in language_options else 0
         
-        # Language selector
         selected_language = st.selectbox(
             get_text('language_setting'),
             options=list(language_options.keys()),
             format_func=lambda x: language_options.get(x, x),
-            index=list(language_options.keys()).index(st.session_state.language) if st.session_state.language in language_options else 0
+            index=current_language_index
         )
         
-        # Update language in session state when changed
+        # Update language if changed
         if selected_language != st.session_state.language:
+            # Update session state and clear template
             st.session_state.language = selected_language
-            st.rerun()  # Rerun to apply the new language
+            if "active_template" in st.session_state:
+                st.session_state.active_template = None
+                
+            # Notification and rerun
+            st.success(f"Language changed to {language_options.get(selected_language, selected_language)}", icon="✅")
+            st.rerun()
         
-        # Get appropriate font options for current language
+        # Font selector
         font_options = get_font_options()
+        font_index = list(font_options.keys()).index(st.session_state.hebrew_font) if st.session_state.hebrew_font in font_options else 0
         
-        # Font selector
         selected_font = st.selectbox(
             get_text('font_setting'),
             options=list(font_options.keys()),
             format_func=lambda x: font_options.get(x, x),
-            index=list(font_options.keys()).index(st.session_state.hebrew_font) if st.session_state.hebrew_font in font_options else 0
+            index=font_index
         )
         
-        # Update font in session state when changed
+        # Update font if changed
         if selected_font != st.session_state.hebrew_font:
-            # Store the new font in session state
             st.session_state.hebrew_font = selected_font
-            # Add a reload flag to force CSS regeneration - use a random number to ensure it's always different
-            st.session_state.css_reload_trigger = int(time.time() * 1000)  # Use milliseconds for better uniqueness
-            # Show notification about font change
-            st.success(f"Font changed to {font_options.get(selected_font, selected_font)}. Applying changes...", icon="✅")
-            # Force a complete rerun to apply the new CSS with the updated font
+            st.success(f"Font changed to {font_options.get(selected_font, selected_font)}", icon="✅")
             st.rerun()
         
-        # Show font preview with improved styling
+        # Font preview
         font_preview_text = get_text('font_preview').format(font_options.get(selected_font, selected_font))
-        preview_style = f"""
-            font-family: "{selected_font}", "Open Sans Hebrew", "Alef Hebrew", "Arial Hebrew", sans-serif !important;
-            direction: {text_direction};
-            text-align: {("right" if text_direction == "rtl" else "left")};
-            padding: 10px;
-            border: 1px solid #ddd;
-            border-radius: 4px;
-            margin-top: 10px;
-            font-size: 1.2em;
-            line-height: 1.5;
-            background-color: #f8f8f8;
-        """
-        st.markdown(f"<div style='{preview_style}'>{font_preview_text}</div>", unsafe_allow_html=True)
+        special_class = "rashi-text" if selected_font == "Noto Rashi Hebrew" else ""
         
-        # Separator
-        st.markdown("<hr>", unsafe_allow_html=True)
+        # Use streamlit container instead of custom HTML for preview
+        preview = st.container(border=True)
+        with preview:
+            st.markdown(f"<div class='{special_class}' style='font-family: \"{selected_font}\", sans-serif;direction:{text_direction};text-align:{'right' if text_direction=='rtl' else 'left'}'>{font_preview_text}</div>", unsafe_allow_html=True)
         
-        # Service status
+        st.divider()  # Native divider instead of HTML hr
+        
+        # Service status with native components
         retriever_ready, _ = get_retriever_status()
         openai_ready, _ = get_openai_status()
         
-        st.markdown(
-            f"<p class='{text_direction}-text'><strong>{get_text('retriever_status')}</strong> {'✅' if retriever_ready else '❌'}</p>",
-            unsafe_allow_html=True
-        )
+        status_col1, status_col2 = st.columns(2)
+        with status_col1:
+            st.write(f"**{get_text('retriever_status')}**")
+        with status_col2:
+            st.write("✅" if retriever_ready else "❌")
+            
         if not retriever_ready:
-            st.sidebar.error(get_text('retriever_error'), icon="🛑")
+            st.error(get_text('retriever_error'), icon="🛑")
             st.stop()
+        
+        status_col3, status_col4 = st.columns(2)
+        with status_col3:
+            st.write(f"**{get_text('openai_status')}**")
+        with status_col4:
+            st.write(f"{'✅' if openai_ready else '❌'}")
             
-        st.markdown(
-            f"<p class='{text_direction}-text'><strong>{get_text('openai_status')}</strong> {'✅' if openai_ready else '❌'} ({config.OPENAI_VALIDATION_MODEL} / {config.OPENAI_GENERATION_MODEL})</p>",
-            unsafe_allow_html=True
-        )
         if not openai_ready:
-            st.sidebar.error(get_text('openai_error'), icon="⚠️")
-            
+            st.warning(get_text('openai_error'), icon="⚠️")
+        
         # RAG parameters
         n_retrieve = st.slider(get_text('retrieval_count'), 1, 300, config.DEFAULT_N_RETRIEVE)
         max_validate = min(n_retrieve, 100)
@@ -112,19 +111,21 @@ def display_sidebar() -> Dict[str, Any]:
         )
         st.info(get_text('validation_info'), icon="ℹ️")
         
-        # Prompt editors
+        # Prompt editors in expander
         with st.expander(get_text('edit_prompts'), expanded=False):
-            gen_prompt = st.text_area(
+            config.OPENAI_SYSTEM_PROMPT = st.text_area(
                 get_text('system_prompt'),
                 value=config.OPENAI_SYSTEM_PROMPT,
                 height=200
             )
-            val_prompt = st.text_area(
+            config.VALIDATION_PROMPT_TEMPLATE = st.text_area(
                 get_text('validation_prompt'),
                 value=config.VALIDATION_PROMPT_TEMPLATE,
                 height=200
             )
-        config.OPENAI_SYSTEM_PROMPT = gen_prompt
-        config.VALIDATION_PROMPT_TEMPLATE = val_prompt
 
-    return {"n_retrieve": n_retrieve, "n_validate": n_validate, "services_ready": (retriever_ready and openai_ready)} 
+    return {
+        "n_retrieve": n_retrieve, 
+        "n_validate": n_validate, 
+        "services_ready": (retriever_ready and openai_ready)
+    } 

memorybank/projectDocs/rules_compliance.md

@@ -1,169 +0,0 @@
-# Rules Compliance Report Guide
-
-## Overview
-The `rules_compliance_report.md` file documents the evaluation of Python files in the Divrei Yoel AI Chat project against established coding rules and best practices. This report serves as a quality assessment tool to identify potential issues, anti-patterns, and areas for improvement across the codebase.
-
-## Purpose and Value
-- **Quality Assurance**: Identifies potential code quality issues before they impact production
-- **Consistency**: Ensures adherence to project-wide coding standards
-- **Documentation**: Provides a record of technical debt and improvement opportunities
-- **Onboarding**: Helps new developers understand code quality expectations
-
-## Report Structure
-
-### File Organization
-The report evaluates 19 Python files across multiple directories:
-- Main application files (`app.py`, `config.py`, `i18n.py`, `rag_processor.py`)
-- Component modules (`components/*.py`)
-- CSS modules (`css/*.py`)
-- Prompt templates (`prompts/*.py`)
-- Service integrations (`services/*.py`)
-- Utility functions (`utils/*.py`)
-
-### Evaluation Categories
-Each file is evaluated against several rule categories:
-- **`python.mdc`**: General Python best practices
-- **`streamlit.mdc`**: Streamlit-specific guidelines
-- **`openai.mdc`**: OpenAI API interaction standards
-- **`asyncio.mdc`**: Asynchronous programming patterns
-- **`langsmith.mdc`**: LangSmith tracing and monitoring
-- **`langchain.mdc`**: LangChain development patterns
-- **`pydantic.mdc`**: Pydantic usage recommendations
-
-### Issue Documentation Format
-For each identified issue, the report provides:
-- **Issue ID**: Unique identifier (e.g., `APP-CSS-HACK`)
-- **File**: Location of the issue
-- **Lines**: Line numbers where the issue appears
-- **Description**: Explanation of the problem
-- **Rule Violation**: Reference to the specific rule being violated
-- **Severity**: Impact level (Critical, Medium, Minor, Informational)
-- **Suggestion**: Recommended fix (when applicable)
-
-## Key Findings Summary
-
-### Critical Issues
-- **Dynamic Imports (RET-DYNAMIC-IMPORTS, UTIL-DYNAMIC-CONFIG-IMPORT)**:
-  - Issue in `services/retriever.py` (lines 13-22) and `utils/__init__.py` (lines 9-13)
-  - Using `sys.path.insert` and `importlib.util` for importing modules
-  - Impact: Breaks static analysis and standard Python import behaviors
-  - Recommendation: Restructure imports to use standard Python import system
-
-- **Global State (OPENAI-GLOBAL-CLIENT, RET-GLOBAL-CLIENT)**:
-  - Issue in `services/openai_service.py` (lines 15-17) and `services/retriever.py` (lines 28-31)
-  - Managing service clients using global variables
-  - Impact: Makes testing and configuration management less flexible
-  - Recommendation: Use class-based services or dependency injection
-
-- **Config Modification (SBAR-CONFIG-MODIFICATION)**:
-  - Issue in `components/sidebar.py` (lines 128-129)
-  - Directly modifies `config.OPENAI_SYSTEM_PROMPT` and `config.VALIDATION_PROMPT_TEMPLATE`
-  - Impact: Creates unpredictable application state
-  - Recommendation: Store modified values in session state instead
-
-### Medium Issues
-- **Async/Sync Mixing (RAG-ASYNC-SYNC-RETRIEVER, UTIL-SYNC-EMBEDDING)**:
-  - Issue in `rag_processor.py` (line 20) and `utils/__init__.py` (in `get_embedding` function)
-  - Calling synchronous functions with blocking I/O from async contexts
-  - Impact: Blocks the event loop, degrading performance
-  - Recommendation: Make retrieval functions async or use thread pool executors
-
-- **LangSmith Tracing (RAG-LS-MISSING-TAGS-META, OPENAI-LS-MISSING-TAGS-META)**:
-  - Issue in multiple `@traceable` decorators across files
-  - Lack of tags (e.g., `model:gpt-4o`) and metadata in traces
-  - Impact: Reduced observability and filtering capabilities
-  - Recommendation: Add consistent tags and metadata to all traced functions
-
-- **Input Sanitization (RAG-LS-INPUT-SANITIZATION, OPENAI-LS-INPUT-SANITIZATION)**:
-  - Issue in traced functions receiving user input
-  - Potential exposure of sensitive information in traces
-  - Impact: Privacy and security concerns
-  - Recommendation: Sanitize inputs before tracing and add "sanitized: True" to metadata
-
-- **CSS in Python (CSS-IN-PYTHON)**:
-  - Issue in `css/styles.py` and UI components
-  - Large blocks of CSS generated as Python f-strings
-  - Impact: Makes CSS editing, linting, and maintenance difficult
-  - Recommendation: Use separate .css files when possible
-
-### Minor Issues
-- **Print vs Logging (CFG-PRINT-LOGGING, RAG-PRINT-LOGGING, etc.)**:
-  - Issue in multiple files using `print()` for status updates and errors
-  - Impact: Inconsistent logging and reduced control over output
-  - Recommendation: Use the `logging` module consistently throughout
-
-- **Long Functions (RAG-LONG-FUNCTION)**:
-  - Issue in `rag_processor.py` (`execute_validate_generate_pipeline`, lines 107-202)
-  - Impact: Reduced readability and maintainability
-  - Recommendation: Refactor into smaller, focused helper functions
-
-- **Pydantic Settings (CFG-PYDANTIC-SETTINGS)**:
-  - Issue in `config.py` (manual environment variable handling)
-  - Impact: Less robust configuration management
-  - Recommendation: Use Pydantic's `BaseSettings` for validation and typing
-
-- **Local Imports (CHAT-LOCAL-IMPORTS, PGALLERY-LOCAL-IMPORTS, SBAR-LOCAL-IMPORTS)**:
-  - Issue in component modules to avoid circular dependencies
-  - Impact: Reduces clarity and can mask design issues
-  - Recommendation: Restructure dependencies for top-level imports
-
-- **HTML Generation (CHAT-HTML-IN-PYTHON, PGALLERY-HTML-IN-PYTHON, SBAR-HTML-IN-PYTHON)**:
-  - Issue in UI components generating HTML with inline styles
-  - Impact: Reduced maintainability and separation of concerns
-  - Recommendation: Extract styling to CSS and use simpler markup generation
-
-## How to Use This Report
-
-### Prioritizing Issues
-1. **Critical Issues**: Must be addressed immediately as they affect stability or security
-2. **Medium Issues**: Should be scheduled for remediation in the next development cycle
-3. **Minor Issues**: Can be addressed during regular maintenance or refactoring
-
-### Implementing Fixes
-For each issue category, consider the following approaches:
-
-#### Dynamic Imports
-- Restructure imports to use standard Python import system
-- Fix PYTHONPATH issues rather than manipulating `sys.path`
-
-#### Global State
-- Refactor to use dependency injection or service classes
-- Create proper initialization and shutdown procedures
-
-#### Logging
-- Replace all `print()` calls with appropriate `logging` module usage
-- Implement centralized logging configuration
-
-#### CSS Management
-- Extract CSS to dedicated CSS files
-- Consider using Streamlit's theming capabilities
-
-#### LangSmith Tracing
-- Add consistent tags, metadata, and sanitization
-- Follow the patterns outlined in `langsmith.mdc`
-
-## Integration with Development Workflow
-
-### Code Review
-Use this report as a reference during code reviews to ensure new code doesn't introduce known issues.
-
-### Continuous Improvement
-- Update the report regularly as the codebase evolves
-- Use it to track progress in resolving identified issues
-- Reference issue IDs in commit messages when fixing problems
-
-### Documentation
-Keep the report synchronized with other project documentation to maintain a consistent view of code quality standards.
-
-## Relationship to Other Documentation
-This report complements other project documentation:
-- **Technical Documentation**: Provides a quality assessment perspective
-- **Development Workflow Guide**: Informs development standards
-- **Module Guides**: Identifies specific areas for improvement within modules
-
-## Recent Updates
-- Comprehensive evaluation of all Python files
-- Addition of severity ratings for each issue
-- Enhanced guidance for resolving critical issues
-- Improved issue categorization and naming conventions
-- Detailed examples of specific code locations needing attention 

memorybank/tasks.md

@@ -1,12 +1,114 @@
-# Tasks
+# DivreiYoelChezky Project Task Tracker
 
-## Current Status
+## 📋 Executive Summary
+- **Project Status**: Phase 2 (Implementation)
+- **Complexity Level**: 2 (Moderate)
+- **Last Updated**: 2025-05-08
+- **Pending Tasks**: 8
+- **Completed Tasks**: 43
+
+## 🏆 Branch Achievements Summary
+
+### Core Application Development
+- ✓ Built Streamlit-based chat application for Rabbi Yoel Teitelbaum's teachings using RAG
+- ✓ Implemented bilingual support (Hebrew/English) with RTL text handling
+- ✓ Created customizable Hebrew font system with Google Fonts integration
+- ✓ Developed modular component architecture with reusable UI elements
+- ✓ Added source citation functionality with OpenAI-based extraction
+- ✓ Optimized bidi text handling with proper unicode-bidi embedding
+
+### RAG Pipeline Implementation
+- ✓ Designed three-stage RAG pipeline (Retrieve → Validate → Generate)
+- ✓ Integrated Pinecone for vector retrieval of relevant documents
+- ✓ Added OpenAI GPT-4o validation to filter relevant passages
+- ✓ Implemented streaming responses for better user experience
+- ✓ Added tracing with LangSmith for observability and debugging
+
+### Code Quality & Architecture
+- ✓ Refactored codebase into modular components and services
+- ✓ Implemented comprehensive error handling throughout application
+- ✓ Created centralized configuration system for environment variables
+- ✓ Added HTML sanitization for security with bleach
+- ✓ Improved session state management for Streamlit
+- ✓ Refactored UI components for better separation of concerns
+- ✓ Moved CSS to external files to reduce duplication
+
+### Documentation & Best Practices
+- ✓ Created extensive module guides for all components
+- ✓ Developed comprehensive rules compliance report
+- ✓ Documented development workflow with best practices
+- ✓ Added internationalization system with translation dictionaries
+- ✓ Created product documentation with user experience flow
+
+### Process & Project Management
+- ✓ Set up Memory Bank system with VAN mode initialization
+- ✓ Completed technical validation of all components
+- ✓ Executed PLAN mode with optimization priorities
+- ✓ Created detailed implementation plans
+- ✓ Fixed 4 of 8 identified issues during review
+
+## 🚀 Active Development Tasks
+
+### 🔧 Technical Debt (Critical)
+
+### ⚡ Performance Optimization (High Priority)
+
+### 🔒 Security Improvements (High Priority)
+- [ ] **Add Input Sanitization for Tracing** (OPENAI-LS-INPUT-SANITIZATION, RET-LS-QUERY-SANITIZATION)
+  - Files: `openai_service.py`, `retriever.py`
+  - Issue: Missing input sanitization for LangSmith tracing of potentially sensitive user queries
+  - Solution: Add sanitization before tracing for security
+
+### 🌐 User Experience Improvements
+- [ ] **Custom Prompt Templates** (UI-PROMPT-TEMPLATES)
+  - Feature: Add system for custom prompt template management
+  - Priority: Medium
+  - Components: Template editor, save/load functionality, template validation
+
+- [ ] **Enhance Service Reliability**
+  - Feature: Add retry mechanism for service initialization failures
+  - Priority: Medium
+
+### 📈 Future Enhancements
+- [ ] **Add Comprehensive API Retries** (OPENAI-NO-RETRIES)
+  - Feature: Enhanced retry mechanism for API calls with exponential backoff
+  - Priority: Low
+
+- [ ] **Improve Observability** 
+  - Feature: Add tags to LangSmith tracing decorators
+  - Files: `rag_processor.py`, `openai_service.py`
+  - Priority: Low
+
+
+## 📝 Project Notes
+- Memory Bank system initialized via VAN mode
+- VAN mode technical validation completed
+- Rules compliance analysis completed (2025-05-06)
+- Product documentation enhanced (2025-05-06 16:35:20)
+- Final assessment confirms moderate complexity (Level 2)
+- Core RAG functionality is well-implemented with good architecture
+- PLAN mode completed with successful implementation of enhancements
+- Project has robust error handling and good separation of concerns 
+- New analysis (2025-05-06) identified 8 issues that require addressing
+- Implementation completed for 4 of 8 issues (2025-05-06)
+- Comprehensive rules compliance report added with 15 prioritized tasks (2025-05-07)
+- UI components refactored for better separation of concerns (2025-05-08)
+- Moved CSS to external files and reduced duplication (2025-05-08)
+- Improved HTML sanitization and Unicode bidirectional text handling (2025-05-08)
+
+---
+
+# 📚 Task Archive (Completed)
+
+## ✅ System Initialization (2025-05-06)
 - [x] VAN mode initialization (TIMESTAMP: 2025-05-06 16:31:19)
 - [x] System files verification
 - [x] Project structure evaluation
 - [x] Technology stack assessment
 - [x] Memory Bank initialization
 - [x] Creation of core Memory Bank files
+
+## ✅ Technical Validation
 - [x] Initial technical validation
 - [x] Detailed code review of app.py
 - [x] Detailed code review of rag_processor.py
@@ -17,21 +119,17 @@
 - [x] Performance optimization assessment
 - [x] Complexity assessment finalization
 - [x] Rules compliance analysis completed
-- [x] Product documentation enhancement (2025-05-06 16:35:20)
-
-## System Implementation Tasks
 - [x] Complete VAN mode validation
+
+## ✅ Project Management
 - [x] Determine appropriate next mode based on complexity
 - [x] Execute mode transition
 - [x] Update Memory Bank with comprehensive project analysis
-
-## Next Mode Preparation
-- [ ] If Level 1 complexity → Prepare for IMPLEMENT mode
 - [x] If Level 2-4 complexity → Prepare for PLAN mode
 - [x] Document evaluation results in Memory Bank
 - [x] Check for missing dependencies or environment variables
 
-## PLAN Mode Tasks (Current Phase)
+## ✅ Planning Phase
 - [x] Define optimization priorities
 - [x] Identify potential enhancements
 - [x] Draft architectural improvements
@@ -39,47 +137,33 @@
 - [x] Schedule task execution
 - [x] Document technical specifications
 
-## Implementation Tasks (Completed)
+## ✅ Implementation Achievements
 - [x] Improved citation detection using OpenAI instead of regex
 - [x] Implemented grid-based prompt gallery with Hebrew examples
 - [x] Enhanced CSS for RTL text display
 - [x] Updated UI labels for better Hebrew support
 - [x] Added collapsible sidebar with settings button
 - [x] Improved overall user experience
+- [x] Completed dropdown menu reorganization and renaming (UI-DROPDOWN-RENAME)
+- [x] Implemented advanced citation filtering system (UI-CITATION-FILTER)
+- [x] Optimized RTL citation display with proper Hebrew support (UI-RTL-CITATIONS)
+- [x] Fixed dynamic imports in retriever.py (RET-DYNAMIC-IMPORTS)
+- [x] Fixed dynamic imports in utils/__init__.py (UTIL-DYNAMIC-CONFIG-IMPORT)
+- [x] Converted retriever.retrieve_documents to async with asyncio.to_thread for Pinecone queries
+- [x] Updated embedding function to use AsyncOpenAI client and asyncio.sleep
+- [x] Completed Streamlit UI Enhancement (UI-STREAMLIT-ENHANCE) - Refactored components, improved CSS, optimized RTL handling (2025-05-08)
 
-## Newly Identified Issues (Current Analysis)
+## ✅ Bug Fixes
 - [x] Fix missing input variable handling in app.py (Line 171: Using `prompt` without initializing)
 - [x] Implement error handling for asyncio loop failures
 - [x] Enhance exception handling in the main RAG processing block
 - [x] Review and secure HTML rendering (multiple instances of `unsafe_allow_html=True`)
-- [ ] Implement proper internationalization for Hebrew text
 - [x] Remove debug print statements from production code
-- [ ] Optimize performance by making UI rendering calls asynchronous
-- [ ] Add retry mechanism for service initialization failures
-
-## Code Quality Improvement Tasks (From Rules Compliance Report)
-### Critical Priority
-- [ ] Fix dynamic imports in retriever.py (RET-DYNAMIC-IMPORTS) - Lines 13-22
-- [ ] Fix dynamic imports in utils/__init__.py (UTIL-DYNAMIC-CONFIG-IMPORT) - Lines 9-13
-- [ ] Refactor global client state in openai_service.py (OPENAI-GLOBAL-CLIENT) - Lines 15-17
-- [ ] Refactor global client state in retriever.py (RET-GLOBAL-CLIENT) - Lines 28-31
-- [ ] Fix runtime config modification in sidebar.py (SBAR-CONFIG-MODIFICATION) - Lines 128-129
-
-### Medium Priority
-- [ ] Fix async/sync mixing in rag_processor.py (RAG-ASYNC-SYNC-RETRIEVER) - Line 20
-- [ ] Update get_embedding() in utils/__init__.py to use async (UTIL-SYNC-EMBEDDING)
-- [ ] Add tags and metadata to traced functions (RAG-LS-MISSING-TAGS-META) in multiple files
-- [ ] Implement input sanitization for LangSmith tracing (RAG-LS-INPUT-SANITIZATION)
-- [ ] Refactor CSS generation in css/styles.py (CSS-IN-PYTHON) to use external files
-
-### Minor Priority
-- [ ] Replace print() with logging in all files (CFG-PRINT-LOGGING, RAG-PRINT-LOGGING, etc.)
-- [ ] Refactor long function in rag_processor.py (RAG-LONG-FUNCTION) - Lines 107-202
-- [ ] Use Pydantic BaseSettings in config.py (CFG-PYDANTIC-SETTINGS)
-- [ ] Fix local imports in component files (CHAT-LOCAL-IMPORTS, PGALLERY-LOCAL-IMPORTS)
-- [ ] Extract HTML generation to separate functions or templates (CHAT-HTML-IN-PYTHON)
-
-## Documentation Enhancements
+- [x] Fixed citation extraction unnecessary API calls with empty responses (2025-05-08)
+- [x] Added unicode bidirectional text optimization to prevent browser cursor jumps (2025-05-08)
+- [x] Improved HTML sanitization to prevent XSS with better timing (2025-05-08)
+
+## ✅ Documentation
 - [x] Improve product description in projectbrief.md
 - [x] Expand user experience workflow in productContext.md
 - [x] Add detailed corpus and knowledge base information
@@ -87,17 +171,4 @@
 - [x] Enhance technical architecture documentation
 - [x] Clarify core functionality and capabilities
 - [x] Update activeContext.md with documentation improvement tracking
-- [x] Create comprehensive rules compliance report with specific issue tracking
-
-## Notes
-- Memory Bank system initialized via VAN mode
-- VAN mode technical validation completed
-- Rules compliance analysis completed (2025-05-06)
-- Product documentation enhanced (2025-05-06 16:35:20)
-- Final assessment confirms moderate complexity (Level 2)
-- Core RAG functionality is well-implemented with good architecture
-- PLAN mode completed with successful implementation of enhancements
-- Project has robust error handling and good separation of concerns 
-- New analysis (2025-05-06) identified 8 issues that require addressing
-- Implementation completed for 4 of 8 issues (2025-05-06)
-- Comprehensive rules compliance report added with 15 prioritized tasks (2025-05-07) 
+- [x] Create comprehensive rules compliance report with specific issue tracking 

pipeline/__init__.py

@@ -0,0 +1,5 @@
+# Pipeline components package
+"""
+Contains pipeline processing components:
+- rag.py: RAG pipeline wrapper and processing
+""" 

pipeline/rag.py

@@ -0,0 +1,105 @@
+import asyncio
+import logging
+import traceback
+from typing import Dict, Any, List, Callable, Optional
+
+import streamlit as st
+
+# Setup logger
+logger = logging.getLogger(__name__)
+
+async def process_rag_request(
+    history: List[Dict[str, Any]], 
+    params: Dict[str, Any], 
+    status_callback: Optional[Callable[[str], None]] = None,
+    stream_callback: Optional[Callable[[str], None]] = None
+) -> Dict[str, Any]:
+    """
+    Process a RAG request asynchronously.
+    
+    Args:
+        history (List[Dict[str, Any]]): Message history
+        params (Dict[str, Any]): RAG parameters
+        status_callback (Optional[Callable]): Callback for status updates
+        stream_callback (Optional[Callable]): Callback for streaming response chunks
+        
+    Returns:
+        Dict[str, Any]: Response data including final response, documents, and logs
+    """
+    from i18n import get_text
+    from rag_processor import execute_validate_generate_pipeline
+    
+    try:
+        return await execute_validate_generate_pipeline(
+            history=history,
+            params=params,
+            status_callback=status_callback,
+            stream_callback=stream_callback
+        )
+    except asyncio.CancelledError:
+        logger.warning("RAG request was cancelled")
+        return {
+            "final_response": get_text('request_cancelled'),
+            "error": "Request cancelled",
+            "status_log": ["Request cancelled by user or system"],
+            "generator_input_documents": [],
+            "pipeline_used": "Cancelled"
+        }
+    except asyncio.TimeoutError:
+        logger.error("RAG request timed out")
+        return {
+            "final_response": get_text('request_timeout'),
+            "error": "Request timed out",
+            "status_log": ["Request exceeded maximum allowed time"],
+            "generator_input_documents": [],
+            "pipeline_used": "Timeout"
+        }
+    except Exception as e:
+        logger.exception("Error in RAG processing")
+        return {
+            "final_response": f"{get_text('processing_error')}: {type(e).__name__}",
+            "error": str(e),
+            "status_log": [f"Error: {type(e).__name__}", traceback.format_exc()],
+            "generator_input_documents": [],
+            "pipeline_used": "Error"
+        }
+
+def create_async_execution_context():
+    """
+    Create or get the appropriate asyncio execution context.
+    
+    Returns:
+        asyncio.AbstractEventLoop: The event loop to use
+    """
+    try:
+        # Try to get the current running loop
+        loop = asyncio.get_running_loop()
+    except RuntimeError:
+        # No running loop, create a new one
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+    
+    return loop
+
+def extract_citations(response: str) -> List[str]:
+    """
+    Extract citation IDs from a response text.
+    
+    Args:
+        response (str): Response text with potential citations
+        
+    Returns:
+        List[str]: List of citation IDs
+    """
+    # Early return if empty response
+    if not response or not response.strip():
+        return []
+        
+    try:
+        from services.openai_service import extract_citations_with_openai
+        loop = create_async_execution_context()
+        return loop.run_until_complete(extract_citations_with_openai(response))
+    except Exception as e:
+        logger.exception("Failed to extract citations")
+        st.error(f"Citation extraction failed: {e}", icon="⚠️")
+        return [] 

prompts/__init__.py

@@ -4,4 +4,5 @@ Provides access to system and validation prompts.
 """
 
 from .system_prompt import OPENAI_SYSTEM_PROMPT
-from .validation_prompt import VALIDATION_PROMPT_TEMPLATE 
+from .validation_prompt import VALIDATION_PROMPT_TEMPLATE
+from .templates import get_templates_for_language, get_template_by_id 

prompts/templates.py

@@ -0,0 +1,140 @@
+"""
+Prompt templates module for the DivreiYoel app.
+Provides structured templates for common query patterns.
+"""
+from typing import Dict, Any, List
+import logging
+
+# Setup logger
+logger = logging.getLogger(__name__)
+
+# Template structure:
+# - id: Unique identifier for the template
+# - name: Display name for the template (localized)
+# - template: The template text that prefixes user input
+# - isolate_query: Whether to isolate the user's query when searching Pinecone
+# - description: Optional description of what the template does
+# - language_code: The language code this template belongs to (added for verification)
+
+PROMPT_TEMPLATES = {
+    "he": [
+        {
+            "id": "dvar_torah_support",
+            "name": "גיבוי לרעיון לדבר תורה",
+            "template": """תפקידך הוא לעזור לי לגבות את רעיונות דברי התורה שלי ולתמוך בהם עם מקורות שיכולים לתמוך במושג או במושגים שאני מבסס עליהם את דבר התורה שלי.
+
+עליך להעריך בקפידה כל דבר תורה כדי לראות אם וכיצד הוא יכול לעזור לגבות את דבר התורה שלי, עליך להשתמש ביצירתיות והיגיון כדי להגיע לכך.
+
+הנה הרעיון שאני מחפש לגבות:
+""",
+            "isolate_query": True,
+            "description": "מציאת מקורות לתמיכה ברעיון דבר תורה",
+            "language_code": "he"
+        },
+        {
+            "id": "html_template_he",
+            "name": "תבנית HTML - מציאת מקורות",
+            "template": """<div dir="rtl" lang="he" class="rtl-text mixed-content hebrew-font">
+תפקידך הוא לעזור לי לגבות את רעיונות דברי התורה שלי ולתמוך בהם עם מקורות שיכולים לתמוך במושג או במושג שאני מבסס עליו את הדרשה.
+
+<ul>
+  <li>יש לאתר <strong>מקורות רלוונטיים</strong> מהחומר הנלמד</li>
+  <li>יש להביא <strong>ציטוטים מדויקים</strong> מהמקורות</li>
+  <li>יש להסביר את <strong>הקשר בין המקורות</strong> לרעיון המרכזי</li>
+</ul>
+
+הנה הרעיון שאני מחפש לגבות:
+</div>""",
+            "isolate_query": True,
+            "description": "תבנית עם תגיות HTML",
+            "language_code": "he"
+        }
+    ],
+    "en": [
+        {
+            "id": "dvar_torah_support",
+            "name": "Dvar Torah Support",
+            "template": """Your job is to help me back up my dvar torah ideas that I come up with and back it up with sources that can support the concept or concepts that I'm basing my dvar torah on.
+
+You have to carefully evaluate each dvar torah to see if and how it can help back up this dvar torah of mine and how, you have to use some creativity and reasoning in order to get to that.
+
+Here's the idea that I'm looking to backup:
+""",
+            "isolate_query": True,
+            "description": "Find sources to support a dvar torah concept",
+            "language_code": "en"
+        },
+        {
+            "id": "html_template_en",
+            "name": "HTML Template - Finding Sources",
+            "template": """<div dir="ltr" lang="en" class="ltr-text">
+Your task is to help me support my Torah insights by finding relevant sources that can back up the concepts I'm developing in my teaching.
+
+<ul>
+  <li>Find <strong>relevant sources</strong> from the studied material</li>
+  <li>Provide <strong>exact quotations</strong> from these sources</li>
+  <li>Explain the <strong>connection between the sources</strong> and the central idea</li>
+</ul>
+
+Here's the concept I'm looking to support:
+</div>""",
+            "isolate_query": True,
+            "description": "Template with HTML tags",
+            "language_code": "en"
+        }
+    ]
+}
+
+def get_templates_for_language(language_code: str) -> List[Dict[str, Any]]:
+    """
+    Get all prompt templates for a specific language.
+    
+    Args:
+        language_code (str): Language code ('he' or 'en')
+        
+    Returns:
+        List[Dict[str, Any]]: List of template objects for the language
+    """
+    # Log the requested language code to help debug
+    logger.info(f"Getting templates for language code: {language_code}")
+    
+    # Strict language matching - only return templates for the exact language
+    templates = PROMPT_TEMPLATES.get(language_code, [])
+    
+    # Fallback to English if no templates found for the requested language
+    if not templates and language_code != "en":
+        logger.warning(f"No templates found for {language_code}, falling back to English templates")
+        templates = PROMPT_TEMPLATES.get("en", [])
+    
+    # Add language_code to each template if not already present
+    for template in templates:
+        if "language_code" not in template:
+            template["language_code"] = language_code
+    
+    # Log the number of templates found
+    logger.info(f"Found {len(templates)} templates for language: {language_code}")
+    
+    return templates
+
+def get_template_by_id(template_id: str, language_code: str) -> Dict[str, Any]:
+    """
+    Get a specific template by ID for a language.
+    
+    Args:
+        template_id (str): Template identifier
+        language_code (str): Language code ('he' or 'en')
+        
+    Returns:
+        Dict[str, Any]: Template object or empty dict if not found
+    """
+    # Log the requested template lookup
+    logger.info(f"Looking up template ID '{template_id}' for language: {language_code}")
+    
+    templates = get_templates_for_language(language_code)
+    for template in templates:
+        if template.get("id") == template_id:
+            logger.info(f"Found template: {template.get('name', 'Unnamed')}")
+            return template
+    
+    logger.warning(f"Template with ID '{template_id}' not found for language: {language_code}")
+    return {} 

rag_processor.py

@@ -18,10 +18,29 @@ StatusCallback = Callable[[str], None]
 # --- Step Functions ---
 
 @traceable(name="rag-step-retrieve")
-async def run_retrieval_step(query: str, n_retrieve: int, update_status: StatusCallback) -> List[Dict]:
+async def run_retrieval_step(query: str, n_retrieve: int, update_status: StatusCallback, original_query: str = None) -> List[Dict]:
+    """
+    Retrieve documents from the vector store.
+    
+    Args:
+        query (str): The full query text (may include template)
+        n_retrieve (int): Number of documents to retrieve
+        update_status (StatusCallback): Status update callback function
+        original_query (str, optional): The original user query without template
+        
+    Returns:
+        List[Dict]: List of retrieved documents
+    """
+    # Import inside function to avoid circular imports
+    from i18n import get_text
+    from services.retriever import retrieve_documents
+    
+    # Use original query for Pinecone search if provided
+    search_query = original_query if original_query else query
+    
     update_status(get_text("retrieving_docs").format(n_retrieve))
     start_time = time.time()
-    retrieved_docs = retriever.retrieve_documents(query_text=query, n_results=n_retrieve)
+    retrieved_docs = await retrieve_documents(query_text=search_query, n_results=n_retrieve)
     retrieval_time = time.time() - start_time
     update_status(get_text("retrieved_docs").format(len(retrieved_docs), f"{retrieval_time:.2f}"))
     if not retrieved_docs:
@@ -143,9 +162,12 @@ async def execute_validate_generate_pipeline(
         return result
 
     try:
+        # Extract original query for search if present
+        original_query = params.get('original_query')
+        
         # 1. Retrieval
         retrieved_docs = await run_retrieval_step(
-            current_query_text, params['n_retrieve'], update_status_and_log
+            current_query_text, params['n_retrieve'], update_status_and_log, original_query
         )
         if not retrieved_docs:
             result["error"] = get_text("no_docs_found")

services/retriever.py

@@ -3,33 +3,21 @@
 # It correctly uses config and utils.
 import time
 import traceback
-import sys
 import os
-import importlib.util
+import asyncio
 from typing import List, Dict, Optional, Tuple
 from pinecone import Pinecone, Index
 from langsmith import traceable
 
-try:
-    # Add parent directory to path to ensure imports work
-    parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
-    sys.path.insert(0, parent_dir)
-    
-    # Import config.py directly from parent directory
-    print(f"Loading config from: {os.path.join(parent_dir, 'config.py')}")
-    spec = importlib.util.spec_from_file_location("config", os.path.join(parent_dir, "config.py"))
-    config = importlib.util.module_from_spec(spec)
-    spec.loader.exec_module(config)
-    
-    # Import from utils package
-    print(f"Importing from utils package")
-    from utils import clean_source_text, get_embedding
-    
-except Exception as e:
-    print(f"Error loading modules: {e}")
-    print(f"Current working directory: {os.getcwd()}")
-    traceback.print_exc()
-    raise SystemExit("Failed to load required modules in retriever.py")
+# Change relative imports to absolute imports
+import config
+from config import (
+    PINECONE_API_KEY,
+    OPENAI_API_KEY,
+    PINECONE_INDEX_NAME,
+    EMBEDDING_MODEL
+)
+from utils import clean_source_text, get_embedding
 
 # --- Globals ---
 pinecone_client: Optional[Pinecone] = None
@@ -42,16 +30,16 @@ def init_retriever() -> Tuple[bool, str]:
     """Initializes the Pinecone client and index connection."""
     global pinecone_client, pinecone_index, is_retriever_ready, retriever_status_message
     if is_retriever_ready: return True, retriever_status_message
-    if not config.PINECONE_API_KEY:
+    if not PINECONE_API_KEY:
         retriever_status_message = "Error: PINECONE_API_KEY not found in Secrets."
         is_retriever_ready = False; return False, retriever_status_message
-    if not config.OPENAI_API_KEY:
+    if not OPENAI_API_KEY:
         retriever_status_message = "Error: OPENAI_API_KEY not found (needed for query embeddings)."
         is_retriever_ready = False; return False, retriever_status_message
     try:
         print("Retriever: Initializing Pinecone client...")
-        pinecone_client = Pinecone(api_key=config.PINECONE_API_KEY)
-        index_name = config.PINECONE_INDEX_NAME
+        pinecone_client = Pinecone(api_key=PINECONE_API_KEY)
+        index_name = PINECONE_INDEX_NAME
         print(f"Retriever: Checking for Pinecone index '{index_name}'...")
         available_indexes = [idx.name for idx in pinecone_client.list_indexes().indexes]
         if index_name not in available_indexes:
@@ -64,7 +52,7 @@ def init_retriever() -> Tuple[bool, str]:
         if stats.total_vector_count == 0:
             retriever_status_message = f"Retriever connected, but index '{index_name}' is empty."
         else:
-            retriever_status_message = f"Retriever ready (Index: {index_name}, Embed Model: {config.EMBEDDING_MODEL})."
+            retriever_status_message = f"Retriever ready (Index: {index_name}, Embed Model: {EMBEDDING_MODEL})."
         is_retriever_ready = True
         return True, retriever_status_message
     except Exception as e:
@@ -78,16 +66,22 @@ def get_retriever_status() -> Tuple[bool, str]:
 
 # --- Core Function ---
 @traceable(name="pinecone-retrieve-documents")
-def retrieve_documents(query_text: str, n_results: int) -> List[Dict]:
+async def retrieve_documents(query_text: str, n_results: int) -> List[Dict]:
     global pinecone_index
     ready, message = get_retriever_status()
     if not ready or pinecone_index is None:
         print(f"Retriever not ready: {message}"); return []
     print(f"Retriever: Retrieving top {n_results} docs for query: '{query_text[:100]}...'"); start_time = time.time()
     try:
-        query_embedding = get_embedding(query_text, model=config.EMBEDDING_MODEL)
+        query_embedding = await get_embedding(query_text, model=EMBEDDING_MODEL)
         if query_embedding is None: print("Retriever: Failed query embedding."); return []
-        response = pinecone_index.query(vector=query_embedding, top_k=n_results, include_metadata=True)
+        # Run Pinecone query in a thread to avoid blocking
+        response = await asyncio.to_thread(
+            pinecone_index.query,
+            vector=query_embedding,
+            top_k=n_results,
+            include_metadata=True
+        )
         formatted_results = []
         if not response or not response.matches: print("Retriever: No results found."); return []
         for match in response.matches:

ui/__init__.py

@@ -0,0 +1,6 @@
+# UI components package
+"""
+Contains UI components for the app:
+- hebrew.py: Hebrew text handling
+- chat_render.py: Chat UI rendering functions
+""" 

ui/chat_render.py

@@ -0,0 +1,164 @@
+import streamlit as st
+from typing import Dict, Any, List
+import logging
+
+# Setup logger
+logger = logging.getLogger(__name__)
+
+def format_source_html(doc: Dict[str, Any], i: int, hebrew_font: str, get_text: callable) -> tuple:
+    """
+    Format a single source document as HTML with proper RTL styling.
+    
+    Args:
+        doc (Dict): Source document
+        i (int): Source index
+        hebrew_font (str): Hebrew font to use
+        get_text (callable): Function to get translated text
+        
+    Returns:
+        tuple: (source_html, text_html) formatted HTML strings
+    """
+    from utils.sanitization import sanitize_html
+    from utils import clean_source_text
+    
+    source = doc.get('source_name', '') or get_text('unknown_source')
+    source = sanitize_html(source)
+    
+    # Clean and sanitize the Hebrew text
+    text = doc.get('hebrew_text', '')
+    if text is None:
+        text = get_text('no_text_available')
+    text = clean_source_text(text)
+    text = sanitize_html(text)
+    
+    # Force RTL and proper Hebrew font styling for sources
+    source_html = f"""
+    <div class='source-info rtl-text hebrew-font' dir='rtl' lang="he">
+        <strong>{get_text('source_label').format(i)}</strong> {source}
+    </div>
+    """
+    
+    text_html = f"""
+    <div class='hebrew-text rtl-text hebrew-font' dir='rtl' lang="he">
+        {text}
+    </div>
+    """
+    
+    return source_html, text_html
+
+def display_chat_message(message: Dict[str, Any]):
+    """
+    Display a chat message with proper formatting.
+    
+    Args:
+        message (Dict[str, Any]): Message object with role, content, and optional metadata
+    """
+    # Import here to avoid circular imports
+    from i18n import get_direction, get_text
+    from utils.sanitization import sanitize_html, escape_html
+    from ui.hebrew import handle_mixed_language_text
+    
+    text_direction = get_direction()
+    role = message.get("role", "assistant")
+    
+    # Ensure we have a valid font setting
+    if 'hebrew_font' not in st.session_state:
+        st.session_state.hebrew_font = "Noto Rashi Hebrew"
+        
+    # Use the current font
+    hebrew_font = st.session_state.hebrew_font
+    
+    with st.chat_message(role):
+        # Sanitize the message content for HTML rendering
+        content = message.get('content', '')
+        if isinstance(content, str):
+            # Escape any HTML tags in the original content
+            content = escape_html(content)
+            content = sanitize_html(content)
+            
+            # Process with the mixed language handler
+            content = handle_mixed_language_text(content, hebrew_font)
+            # Final sanitization after processing
+            content = sanitize_html(content)
+            
+            # If this is a user message (prompt) and it's longer than a threshold, show it in an expandable section
+            if role == "user" and len(content) > 150:
+                # Create a preview of the content
+                preview_content = content[:147] + "..."
+                preview_content = handle_mixed_language_text(preview_content, hebrew_font)
+                preview_content = sanitize_html(preview_content)
+                
+                # Show the preview
+                st.markdown(preview_content, unsafe_allow_html=True)
+                
+                # Show the full content in an expander
+                with st.expander(get_text('show_full_prompt'), expanded=False):
+                    st.markdown(f"""<div dir="{text_direction}" 
+                        class="prompt-full-text {text_direction}-text">
+                        {content}
+                    </div>""", unsafe_allow_html=True)
+            else:
+                # Display normal content
+                st.markdown(content, unsafe_allow_html=True)
+        
+        if role == "assistant" and message.get("final_docs"):
+            docs = message["final_docs"]
+            # Use a simple text title for the expander
+            with st.expander(f"{get_text('sources_title')} ({len(docs)})", expanded=False):
+                # Add the rich HTML content inside the expander (static HTML is safe)
+                expander_title = f"""
+                <div class='expander-title rtl-text hebrew-font' dir="rtl" lang="he">
+                    {get_text('sources_text').format(len(docs))}
+                </div>
+                """
+                st.markdown(expander_title, unsafe_allow_html=True)
+                st.markdown(f"""
+                <div dir='rtl' lang="he" class='expander-content rtl-text hebrew-font'>
+                """, unsafe_allow_html=True)
+                
+                for i, doc in enumerate(docs, start=1):
+                    source_html, text_html = format_source_html(doc, i, hebrew_font, get_text)
+                    st.markdown(source_html, unsafe_allow_html=True)
+                    st.markdown(text_html, unsafe_allow_html=True)
+                    
+                st.markdown("</div>", unsafe_allow_html=True)
+
+
+def display_status_updates(status_log: List[str]):
+    """
+    Display processing status log in an expander.
+    
+    Args:
+        status_log (List[str]): List of status update messages
+    """
+    # Import here to avoid circular imports
+    from i18n import get_direction, get_text
+    
+    text_direction = get_direction()
+    
+    # Ensure we have a valid font setting
+    if 'hebrew_font' not in st.session_state:
+        st.session_state.hebrew_font = "Noto Rashi Hebrew"
+        
+    hebrew_font = st.session_state.hebrew_font
+    
+    # Text alignment based on direction
+    text_align = "right" if text_direction == "rtl" else "left"
+    
+    if status_log:
+        # Use a simple text title for the expander
+        with st.expander(get_text('processing_details'), expanded=False):
+            # Add the rich HTML content inside the expander
+            st.markdown(f"""<div 
+                class='expander-title {text_direction}-text hebrew-font' 
+                dir="{text_direction}"
+            >{get_text('processing_log')}</div>""", unsafe_allow_html=True)
+            
+            for u in status_log:
+                st.markdown(
+                    f"""<code 
+                        class='status-update {text_direction}-text hebrew-font' 
+                        dir="{text_direction}"
+                    >- {u}</code>""",
+                    unsafe_allow_html=True
+                ) 

ui/hebrew.py

@@ -0,0 +1,80 @@
+import re
+from typing import Optional
+
+def contains_hebrew(text: str) -> bool:
+    """
+    Check if text contains Hebrew characters.
+    
+    Args:
+        text (str): Text to check
+        
+    Returns:
+        bool: True if text contains Hebrew characters
+    """
+    # Hebrew Unicode range (1,424–1,535) plus some additional ranges for Hebrew characters
+    hebrew_pattern = re.compile(r'[\u0590-\u05FF\uFB1D-\uFB4F]')
+    return bool(hebrew_pattern.search(text))
+
+def contains_english(text: str) -> bool:
+    """
+    Check if text contains English characters.
+    
+    Args:
+        text (str): Text to check
+        
+    Returns:
+        bool: True if text contains English characters
+    """
+    english_pattern = re.compile(r'[a-zA-Z]')
+    return bool(english_pattern.search(text))
+
+def handle_mixed_language_text(text: str, hebrew_font: str) -> str:
+    """
+    Process text that may contain both Hebrew and English to display correctly.
+    
+    Args:
+        text (str): The text to process
+        hebrew_font (str): The Hebrew font to use
+        
+    Returns:
+        str: Processed HTML with proper bidirectional handling
+    """
+    has_hebrew = contains_hebrew(text)
+    has_english = contains_english(text)
+    
+    if has_hebrew and has_english:
+        # For mixed language content, we need to identify and wrap English words
+        # Split by whitespace to find words
+        words = text.split()
+        processed_words = []
+        
+        for word in words:
+            # If word contains English but not Hebrew, wrap it in LTR span
+            if contains_english(word) and not contains_hebrew(word):
+                processed_words.append(f'<span dir="ltr">{word}</span>')
+            else:
+                processed_words.append(word)
+        
+        # Join words back with spaces
+        processed_text = ' '.join(processed_words)
+        
+        # Wrap everything in RTL container
+        return f"""
+        <div dir="rtl" lang="he" class="rtl-text mixed-content hebrew-font">
+            {processed_text}
+        </div>
+        """
+    elif has_hebrew:
+        # For Hebrew-only content, use RTL direction
+        return f"""
+        <div dir="rtl" lang="he" class="rtl-text hebrew-font">
+            {text}
+        </div>
+        """
+    else:
+        # For English-only content, use LTR direction
+        return f"""
+        <div dir="ltr" lang="en" class="ltr-text hebrew-font">
+            {text}
+        </div>
+        """ 

utils/__init__.py

@@ -1,18 +1,13 @@
 from .sanitization import sanitize_html
 import re
-import sys
 import os
-import time
-import openai
+import asyncio
 from typing import List, Dict, Optional
+from openai import AsyncOpenAI
 
-# Add config to the path if needed
-sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
-try:
-    import config
-except ImportError:
-    print("Error: Failed to import config in utils/__init__.py")
-    raise
+# Change relative imports to absolute imports
+import config
+from config import OPENAI_API_KEY, EMBEDDING_MODEL
 
 def clean_source_text(text: str) -> str:
     """
@@ -30,9 +25,9 @@ def clean_source_text(text: str) -> str:
     text = re.sub(r'\s+', ' ', text).strip()
     return text
 
-def get_embedding(text: str, model: str = None, max_retries: int = 3) -> Optional[List[float]]:
+async def get_embedding(text: str, model: str = None, max_retries: int = 3) -> Optional[List[float]]:
     """
-    Get embedding for text using OpenAI's API
+    Get embedding for text using OpenAI's API asynchronously
     
     Args:
         text (str): Text to get embedding for
@@ -43,9 +38,9 @@ def get_embedding(text: str, model: str = None, max_retries: int = 3) -> Optiona
         List[float]: Embedding vector or None if failed
     """
     if model is None:
-        model = config.EMBEDDING_MODEL
+        model = EMBEDDING_MODEL
         
-    openai_client = openai.OpenAI(api_key=config.OPENAI_API_KEY)
+    openai_client = AsyncOpenAI(api_key=OPENAI_API_KEY)
     
     if not text or not isinstance(text, str):
         print("Error: Invalid input text for embedding.")
@@ -59,16 +54,16 @@ def get_embedding(text: str, model: str = None, max_retries: int = 3) -> Optiona
     attempt = 0
     while attempt < max_retries:
         try:
-            response = openai_client.embeddings.create(input=[cleaned_text], model=model)
+            response = await openai_client.embeddings.create(input=[cleaned_text], model=model)
             return response.data[0].embedding
         except openai.RateLimitError as e:
             wait_time = (2 ** attempt)
             print(f"Rate limit embedding. Retrying in {wait_time}s...")
-            time.sleep(wait_time)
+            await asyncio.sleep(wait_time)
             attempt += 1
         except openai.APIConnectionError as e:
             print(f"Connection error embedding. Retrying...")
-            time.sleep(2)
+            await asyncio.sleep(2)
             attempt += 1
         except Exception as e:
             print(f"Error generating embedding (Attempt {attempt + 1}/{max_retries}): {type(e).__name__}")

utils/sanitization.py

@@ -1,31 +1,83 @@
 import bleach  # For HTML sanitization
+from bleach.css_sanitizer import CSSSanitizer
 
 def sanitize_html(html_content: str) -> str:
     """
-    Sanitize HTML content to prevent XSS attacks but preserve necessary tags
-    for Hebrew text and RTL formatting.
+    Sanitize HTML content to prevent XSS attacks while preserving Hebrew text and RTL support.
     """
     if not isinstance(html_content, str):
         return str(html_content)
-        
+    
+    # Allowed HTML tags organized by purpose
     allowed_tags = [
-        'div', 'p', 'span', 'br', 'hr', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
-        'ul', 'ol', 'li', 'dl', 'dt', 'dd', 'img', 'a', 'strong', 'em', 
-        'b', 'i', 'u', 'code', 'pre', 'blockquote', 'table', 'thead', 
-        'tbody', 'tr', 'th', 'td', 'sup', 'sub', 'details', 'summary'
+        # Structure elements
+        'div', 'p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6',
+        'ul', 'ol', 'li', 'blockquote', 'pre',
+        
+        # Inline formatting
+        'span', 'strong', 'em', 'b', 'i', 'u', 'code', 'sup', 'sub',
+        
+        # Interactive elements
+        'a', 'img', 'button',
+        
+        # Layout elements
+        'br', 'hr', 'table', 'thead', 'tbody', 'tr', 'th', 'td',
+        
+        # Style
+        'style'
     ]
+    
+    # Allowed HTML attributes
     allowed_attributes = {
-        '*': ['class', 'style', 'dir'],
+        # Global attributes
+        '*': ['class', 'style', 'dir', 'lang'],
+        
+        # Specific elements
         'a': ['href', 'title', 'target'],
         'img': ['src', 'alt', 'width', 'height'],
-        'div': ['dir', 'class', 'style'],
-        'span': ['dir', 'class', 'style'],
+        'div': ['data-testid'],
+        'button': ['kind']
     }
     
-    # Use bleach to sanitize the HTML - only using supported parameters
+    # Essential CSS properties for proper text direction and sidebar positioning
+    allowed_css_properties = [
+        # Text formatting
+        'font-family', 'font-size', 'font-weight', 'font-style',
+        'text-align', 'direction', 'line-height', 'color',
+        
+        # Basic layout
+        'display', 'margin', 'padding', 'width', 'height', 
+        'border', 'border-radius', 'background-color',
+        
+        # Positioning (needed for sidebar)
+        'position', 'top', 'right', 'bottom', 'left', 'z-index'
+    ]
+    
+    # Create CSS sanitizer
+    css_sanitizer = CSSSanitizer(allowed_css_properties=allowed_css_properties)
+    
+    # Sanitize and return the HTML
     return bleach.clean(
         html_content,
         tags=allowed_tags,
         attributes=allowed_attributes,
+        css_sanitizer=css_sanitizer,
         strip=True
-    ) 
+    )
+
+def escape_html(text: str) -> str:
+    """
+    Escape HTML characters in a string to safely embed it within HTML.
+    This is used for user inputs that should not contain HTML.
+    
+    Args:
+        text (str): Text that might contain HTML characters
+        
+    Returns:
+        str: Text with HTML characters escaped
+    """
+    if not isinstance(text, str):
+        text = str(text)
+    
+    # Replace HTML special characters with their entities
+    return text.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;').replace('"', '&quot;').replace("'", '&#39;')