try n_thread=4

.gitignore

@@ -0,0 +1,4 @@
+
+.vscode/
+
+*.ipynb

app/core/config.py

@@ -12,6 +12,6 @@ class Settings(BaseSettings):
 
     # CPU constraints for HF Free Space
     N_CTX: int = 2048
-    N_THREADS: int = 2
+    N_THREADS: int = 4
 
 settings = Settings()

app/services/engine.py

@@ -18,7 +18,7 @@ class InferenceEngine:
                 model_path=settings.MODEL_PATH,
                 n_ctx=settings.N_CTX,
                 n_threads=settings.N_THREADS,
-                use_mmap=False,
+                use_mmap=False,    # it force to load all the weights in the RAM (if True then store in the Disk)
                 verbose=False
             )
             

test.ipynb

@@ -18,7 +18,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
+   "execution_count": 4,
    "id": "7a86f8cc",
    "metadata": {},
    "outputs": [
@@ -27,7 +27,7 @@
      "output_type": "stream",
      "text": [
       "--- LangChain Pipeline Stream Output ---\n",
-      "How can I assist you today?\n",
+      "I'm doing well! How about you? How can I assist you today?\n",
       "\n"
      ]
     }
@@ -43,14 +43,14 @@
     ")\n",
     "\n",
     "print(\"--- LangChain Pipeline Stream Output ---\")\n",
-    "for chunk in llm.stream([HumanMessage(content=\"Hi!\")]):\n",
+    "for chunk in llm.stream([HumanMessage(content=\"Hello how are you \")]):\n",
     "    print(chunk.content, end=\"\", flush=True)\n",
     "print(\"\\n\")"
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 21,
+   "execution_count": 2,
    "id": "6ecc6c66",
    "metadata": {},
    "outputs": [
@@ -59,21 +59,21 @@
      "output_type": "stream",
      "text": [
       "--- LangChain Pipeline Stream Output ---\n",
-      "LLM stands for Large Language Model. These are advanced AI models designed for natural language understanding and generation. LLMs are trained on vast corpora of text data, enabling them to generate human-like text, answer questions, and perform various language-related tasks.\n",
-      "\n",
-      "Some notable LLMs include:\n",
+      "As of my knowledge cutoff in October 2023, LLM typically refers to \"Large Language Models.\" These are advanced machine learning models designed to understand, generate, and sometimes translate human language with remarkable accuracy. They are built using deep learning techniques, particularly architectures like Transformers. Some well-known examples of large language models include:\n",
       "\n",
-      "1. **GPT-3 (Generative Pre-trained Transformer 3):** Developed by Microsoft, GPT-3 is one of the most powerful LLMs available. With 175 billion parameters, it can generate coherent and contextually relevant text, answer questions, and even create code snippets.\n",
+      "1. **GPT (Generative Pre-trained Transformer) Series**:\n",
+      "   - **GPT-3**: Released by Microsoft in June 2020, it features 175 billion parameters and is known for its ability to generate coherent and contextually relevant text.\n",
+      "   - **GPT-4**: Microsoft released this iteration in March 2023, building upon GPT-3's architecture and features to further enhance language understanding and generation capabilities.\n",
       "\n",
-      "2. **BERT (Bidirectional Encoder Representations from Transformers):** Developed by Google, BERT is designed for better understanding the context of words in search queries. It is particularly useful for tasks like question answering and text classification.\n",
+      "2. **BERT (Bidirectional Encoder Representations from Transformers)**: Developed by Google, BERT is designed to understand the context of words in search queries by considering the words that come before and after each word in the sentence. BERT has various versions like BERT-large, BERT-base, etc.\n",
       "\n",
-      "3. **T5 (Text-To-Text Transfer Transformer):** Developed by Google, T5 treats every NLP problem as a text-to-text problem, where the input and output are both strings. This makes it versatile for various tasks like translation, summarization, and question answering.\n",
+      "3. **T5 (Text-to-Text Transfer Transformer)**: Created by Google, T5 treats all NLP tasks as text-to-text problems, transforming them into a unified framework. It is known for its versatility and efficiency.\n",
       "\n",
-      "4. **XLNet:** Developed by researchers at Microsoft, XLNet is an LLM that combines the benefits of both autoregressive models like GPT and non-autoregressive models, making it highly effective for next-word prediction tasks.\n",
+      "4. **PaLM (Pathways Language Model)**: Developed by Google, PaLM is a smaller but highly efficient version of their original model. It was released in October 2023 and has demonstrated impressive performance across a wide range of language tasks, including math and science reasoning.\n",
       "\n",
-      "5. **RoBERTa (A Robustly Optimized BERT Pretraining Approach):** Developed by Facebook AI Research, RoBERTa builds upon BERT by using a larger dataset, removing the next sentence prediction task, and increasing the batch size during training.\n",
+      "5. **Phi (Phi-2)**: Developed by Microsoft, Phi-2 is a smaller and more efficient language model compared to some of its predecessors. It is designed to provide high performance while using significantly less computational power.\n",
       "\n",
-      "These models have revolutionized the way we interact with machines, enabling applications in areas such as chatbots, virtual assistants, content creation, language translation, and more. They continue to evolve and improve, pushing the boundaries of what's possible in natural language processing.\n",
+      "These large language models have revolutionized fields such as natural language processing (NLP), conversational AI, content generation, and many others. They are trained on vast datasets comprising diverse forms of text, enabling them to understand and generate human-like text with high accuracy. As research and development in this area continue, we can expect even more powerful and versatile language models in the future.\n",
       "\n"
      ]
     }