Fix latency: switch to bge-small and bge-reranker-base for CPU optimization

PROJECT_OVERVIEW.md

@@ -11,9 +11,9 @@ The assistant uses a hybrid LLM approach:
 ## 2. How it Works (The "RAG" Process)
 Instead of just relying on general knowledge, this bot "reads" your documents to give specific answers.
 1. **Reading**: It looks at your files in the `docs/` folder (PDFs and Text files).
-2. **Memorizing**: It breaks the text into small chunks and converts them into mathematical "vectors" (using the `bge-large-en-v1.5` model).
+2. **Memorizing**: It breaks the text into small chunks and converts them into mathematical "vectors" (using the `bge-small-en-v1.5` model).
 3. **Searching**: When you ask a question, it expands the query using Llama 3.1 8B, then performs a **Hybrid Search** combining Dense vectors (**FAISS**) and Keyword search (**BM25**).
-4. **Reranking**: It deeply evaluates the top retrieved chunks using `bge-reranker-large` to ensure maximum relevance.
+4. **Reranking**: It deeply evaluates the top retrieved chunks using `bge-reranker-base` to ensure maximum relevance.
 5. **Answering**: It sends your question along with the most relevant document parts to the Qwen 2.5 AI, which then writes a highly precise, formatted reply.
 
 ## 3. The Architecture

app/core/config.py

@@ -14,8 +14,8 @@ class Settings(BaseSettings):
     groq_rewrite_model: str = Field(default="llama-3.1-8b-instant", alias="GROQ_REWRITE_MODEL")
     hf_api_key: str = Field(default="", alias="HF_API_KEY")
     hf_model: str = Field(default="meta-llama/Llama-3.1-8B-Instruct", alias="HF_MODEL")
-    embedding_model: str = Field(default="BAAI/bge-large-en-v1.5", alias="EMBEDDING_MODEL")
-    reranker_model: str = Field(default="BAAI/bge-reranker-large", alias="RERANKER_MODEL")
+    embedding_model: str = Field(default="BAAI/bge-small-en-v1.5", alias="EMBEDDING_MODEL")
+    reranker_model: str = Field(default="BAAI/bge-reranker-base", alias="RERANKER_MODEL")
     docs_dir: Path = Field(default=Path("docs"), alias="DOCS_DIR")
     @model_validator(mode='after')
     def set_persistent_paths(self) -> 'Settings':

app/services/llm.py

@@ -89,7 +89,7 @@ class LLMService:
             # Ensure we include the original query too
             if query not in queries:
                 queries.append(query)
-            return queries[:4] # Original + 3 variations
+            return queries[:3] # Original + 2 variations
         except Exception:
             return [query]