Final Commit v1

README.md

@@ -1,12 +1,179 @@
+<h1 align="center">Welcome to Ekatra QnA System 👋</h1>
+<p>
+  <a href="." target="_blank">
+    <img alt="Documentation" src="https://img.shields.io/badge/documentation-yes-brightgreen.svg" />
+  </a>
+  <a href="#" target="_blank">
+    <img alt="License: MIT" src="https://img.shields.io/badge/License-MIT-yellow.svg" />
+  </a>
+  <a href="https://twitter.com/ekatraone" target="_blank">
+    <img alt="Twitter: ekatraone" src="https://img.shields.io/twitter/follow/ekatraone.svg?style=social" />
+  </a>
+</p>
+
+> Ekatra QnA is a student-focused intelligent search engine that enables them to find answers across various sources.
+
+### 🏠 [Homepage](https://github.com/ekatraone/Ekatra-QnA-System)
+---
+### 🎥 Check out our [demo video](https://www.awesomescreenshot.com/video/15118083?key=c9f98dd8fcc835f6f97c1cefd7ac6317)
+---
+## Abstract
+
+- The objective of this project is to create a smart search engine named [Ekatra](www.ekatra.one) QnA, which is intended to assist students in finding answers from various sources such as web links and PDF documents in a prompt and effective manner. The search engine employs NLP techniques, including text extraction, preprocessing, vectorization, and semantic similarity, to accurately and quickly identify the most suitable answers to students' queries. The intended target audience for this project is students worldwide who face difficulties in finding pertinent information from a vast amount of data.
+
+
+- Ekatra QnA employs cutting-edge technologies such as OpenAI and Sentence Transformers to filter out irrelevant information and offer students the most useful and appropriate content. The system is built to boost productivity and save time for students as it efficiently identifies the required information, allowing them to concentrate more on learning and less on searching.
+
+- Furthermore, it should be noted that all information in the search engine comes from a validated source. If the answer to a question is not present in the documents, the system is equipped to recognize this and prevent sharing misleading information.
+
+*Learn more about Ekatra here: https://www.ekatra.one/*
+
+---
+
+## Features
+Our proposed intelligent search engine for students offers several features that address the common challenges students face when searching for information. Here are more details on each of these features:
+
+1. **Multiple File Search:**  The system allows students to search for answers across multiple files, including PDFs. This feature ensures that students have access to a broader range of information sources, enabling them to find answers to their queries faster and more efficiently.
+
+
+1.	**Simultaneous Text Extraction:** The system can extract text from multiple sources simultaneously, enabling faster and more efficient searches.
+
+2.	**Accurate and Condensed Presentation:** 
+The system presents the extracted text in the most accurate and condensed form, making it easier for students to read and understand.
+
+4. **Relevant Content:**
+- One of the most important features is its capability to detect questions that are not relevant to the provided documents. This functionality helps prevent the system from providing erroneous or misleading answers to the student.
+- The system filters out irrelevant information and selects the most appropriate answer by analyzing the given data, providing students with the most useful and pertinent content.
+
+---
+## Workflow
+
+The proposed workflow for the intelligent search engine includes the following steps:
+
+1. Users are able to upload a PDF document to the system.
+2. The system employs the PyPDF2 library to extract the text from the uploaded PDF.
+3. The extracted text undergoes a preprocessing step to remove noise such as stop words and punctuation.
+4. A pre-trained model from Hugging Face is used to vectorize the preprocessed text and the user's question
+
+6. The resulting embeddings of the text and question are stored for future use.
+7. The stored embeddings are used to locate similar sentences from the text.
+8. The similar sentences are then combined to form a concise and accurate answer to the user's question.
+9. The system presents the final answer in the most appropriate format within the provided document. If the answer cannot be found within the document, the system displays "I don't know." to the user.
+----
+## Tech Stack
+
+### Programming Language
+
+- [**Python**](https://www.python.org/doc/): A popular, high-level programming language known for its simplicity, readability, and versatility.
+
+### Data Processing and Analysis
+
+- [**Numpy**](https://numpy.org): A library for scientific computing and numerical analysis in Python. It provides support for large, multi-dimensional arrays and matrices, as well as a collection of mathematical functions.
+
+### Natural Language Processing
+
+- [**NLTK**](https://www.nltk.org): A suite of libraries and programs for natural language processing tasks such as tokenization, stemming, and part-of-speech tagging.
+
+### PDF Text Extraction
+
+- [**PyPDF2**](https://pypi.org/project/PyPDF2/): A pure-python library for extracting text from PDF documents. It can also be used to extract metadata and other information from PDF files.
+
+### Web App Framework
+
+- [**Streamlit**](https://pypi.org/project/PyPDF2/): An open-source web app framework for building interactive, customizable, and data-driven apps using Python. It provides an intuitive UI and fast feedback, making it easy to create and iterate on apps.
+
+### AI and Machine Learning Frameworks:
+
+- [**OpenAI**](https://openai.com): OpenAI is known for its advanced language models such as GPT-3, which provide developers with powerful natural language processing capabilities for a wide range of applications. Their language models can perform tasks such as text completion, translation, and question-answering with impressive accuracy.
+
+- [**Huggingface**](https://huggingface.co): A startup that develops state-of-the-art natural language processing technologies and tools, including pre-trained models, fine-tuning tools, and language generation tools.
+
+- [**Sentence Transformers Library**](https://huggingface.co/sentence-transformers): The Sentence Transformers library is a core component of the tech stack as it provides pre-trained models for sentence embedding generation, as well as tools for fine-tuning and training new models.
+
+----
+## Architecture
+
+![System Architecture](./system-architecture.png)
+
+- **Text extraction** refers to the process of extracting relevant text data from a given source, such as a PDF document, web page, or image.
+
+- **Preprocessing** is the initial step in natural language processing, which involves cleaning and transforming raw text data into a format that is suitable for further analysis. This includes removing stop words, punctuations, converting all text to lowercase, and stemming.
+
+- **Vectorization** is the process of converting text data into a numerical vector representation, allowing it to be processed and analyzed using mathematical algorithms. This involves converting each word or sentence into a high-dimensional vector using techniques such as Word2Vec or BERT.
+
+- A **vector database** is a database system that stores and retrieves vectors efficiently. Vector databases are used in various applications, such as search engines, recommendation systems, and natural language processing.
+
+- **Semantic similarity** refers to the degree of similarity between two pieces of text in terms of their meaning or semantic content. It is often measured by comparing the vector representations of the text using algorithms such as cosine similarity.
+
+- **Answer synthesis** using OpenAI involves using pre-trained language models and natural language generation techniques to generate coherent and relevant responses to a user's query. The models use deep learning algorithms and large amounts of training data to generate human-like responses to natural language queries.
+
 ---
-title: Ekatra QnA
-emoji: 💻
-colorFrom: red
-colorTo: blue
-sdk: streamlit
-sdk_version: 1.17.0
-app_file: app.py
-pinned: false
+## Open-source and Digital Public Goods leveraged/used:
+
+1.	Python
+2.	Numpy
+3.	NLTK
+4.	Streamlit
+5.	Sentence Transformers
+
 ---
+## User's Guide 👥
+Here's a quick guide to help you get started:
+
+1. Visit [Ekatra QnA](https://ramsha-ekatra-ekatra-qna.hf.space/)
+
+2. Once you're on our website, simply upload your PDF files using the easy upload feature.
+
+3. Once you have uploaded your files, simply wait for the **"Process Completed"** message before you start asking any questions related to the content of your PDF files. 
+
+4. Our search engine will then be ready to provide you with the most relevant answers to your queries.
+
+
+---
+## Developer's Guide 💻
+
+### **Prerequisites: Obtain required OpenAPI Keys 🔑**
+
+1. Go to the [OpenAI](https://openai.com) website and click on the "Sign Up"/ "Login" button
+
+2. Once you have successfully registered/ logged in, go to the API dashboard and click on "Create API Key".
+
+3. Select the API plan that best suits your needs and click "Continue".
+
+4. Review the API agreement and accept the terms and conditions.
+
+5. Your API key will now be displayed on the dashboard. Copy the key and use it in your project.
+
+*Note: Keep your API key confidential and do not share it with anyone. Also, make sure to follow the OpenAI API usage guidelines to avoid any misuse of the API.*
+
+### **Run on local server**
+1. Clone our github [repo](https://github.com/ekatraone/Ekatra-QnA-System.git)
+
+```bash
+git clone https://github.com/ekatraone/Ekatra-QnA-System.git
+```
+2. Install dependencies
+
+```sh
+pip install -r requirements.txt
+```
+
+3. Run app.py
+```
+streamlit run app.py
+```
+
+---
+## Authors
+
+👤 **Ekatra Team**
+
+* [Abheejit](https://github.com/iabheejit) 
+* [Ramsha](https://github.com/ramshashaikh)
+* [Rushikesh](https://github.com/shaharao-rushikesh)
+
+🏢 **Ekatra Learning, Inc.**
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+* Website: [Ekatra](www.ekatra.one)
+* Twitter: [@ekatraone](https://twitter.com/ekatraone)
+* LinkedIn: [@ekatraone](https://www.linkedin.com/company/ekatraone/)

app.py

@@ -3,27 +3,18 @@ import textwrap
 # from dotenv import load_dotenv
 import streamlit as st
 import os
-import PyPDF2
 import nltk
 from nltk.tokenize import sent_tokenize
-from sentence_transformers import SentenceTransformer
-import numpy as np
-from numpy.linalg import norm
 import openai
 nltk.download('punkt')
-# load_dotenv()
 
 
 openai.organization = os.environ['org']
 openai.api_key = os.environ['api_key']
 
 
-
-url = "https://www.independentschoolparent.com/wp-content/uploads/2018/01/AI.jpg"
-
 st.title("Ekatra QnA")
 st.write("AI Powered Smart Search System")
-# st.image(url)
 
 st.markdown('_Welcome to Question Answering System 🧠 🤖_')
 
@@ -32,108 +23,94 @@ top_match_sentences = []
 
 @st.cache(allow_output_mutation=True)
 def process_pdf_data(uploaded_files):
+    """
+    The function accepts an uploaded PDF file as input and then proceeds to extract, preprocess, and vectorize the text. It ultimately returns a list of filtered sentences and their respective embeddings.
+
+    Parameters: 
+    uploaded_files (list): List of uploaded files
+
+    Returns: 
+    filt1_list (list): List of filtered sentences
+    embeddings (list): List of embeddings of the sentences
+
+    """
     filt1_list = []
     embeddings = []
-    cosine_lis = []
 
     text_ext = []
     for i in uploaded_files:
         if i.type == "application/pdf":
-            # reader = PyPDF2.PdfReader(i)
-
+            # Reading the pdf file and extracting the text
             text_ext += op.read_pdf(i)
 
+    # Applying sent_tokenize to the text and storing the result in a list
     sent_toks = []
     for i in text_ext:
         sent_toks.append(sent_tokenize(i))
 
-    # print("len(sent_toks) ", sent_toks)
-
     concat_list = [j for i in sent_toks for j in i]
 
+    # Removing the new line characters from the list
     for i in concat_list:
         a = (i.replace('\n', ' '))
         filt1_list.append(a)
 
-    # model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
-
-    embeddings = op.create_content_embeddings(concat_list)
+    # Creating embeddings for the sentences
+    embeddings = op.create_content_embeddings(filt1_list)
     return filt1_list, embeddings
 
 
-# if st.sidebar.radio("SELECT -", ['PDF']) == 'PDF':
+# Streamlit code to upload files
 uploaded_files = st.file_uploader(
     "Upload files - ", accept_multiple_files=True, type=['pdf'])
 
-# print(len(uploaded_files) == 0)
 if st.button("Process!"):
     if len(uploaded_files) != 0:
-        # print(len(uploaded_files))
+        # Calling the function process_pdf_data to process the uploaded files
         filt1_list, embeddings = process_pdf_data(uploaded_files)
         st.write("Process Completed")
 
     else:
         st.warning("Please upload a PDF file.")
 
+# Streamlit code to take user input after vectorization of the documents
 query = st.text_input('Ask me anything!', placeholder='Type.....')
-if st.button("Confirm!"):
-    model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
-    cached_data = process_pdf_data(uploaded_files)
-    filt1_list = cached_data[0]
-    embeddings = cached_data[1]
-    query_embedding = op.create_query_embeddings(query)
-    cosine_lis = op.calculate_cosine(
-        query_embedding, embeddings, filt1_list)
-# print("cosine ", cosine_lis)
-
-    indexes_final = op.fetch_top_rank_ans(cosine_lis, 15)
-
-    most_relevant = op.fetch_most_relevant(
-        indexes_final, filt1_list, cosine_lis, query)
-
-# print("most_relevant ", most_relevant)
-
-    COMPLETIONS_API_PARAMS = {
-        "temperature": 0.0,
-        "max_tokens": 300,
-        "model": "text-davinci-003",
-    }
-
-    response = openai.Completion.create(
-        prompt=most_relevant,
-        **COMPLETIONS_API_PARAMS
-    )
-    print("\n\n", textwrap.fill(
-        response["choices"][0]["text"].strip(" \n")))
-
-    # prompt = f"Q: {question}\n\nA: Please give an honest answer, only based on given data. If you don't know the answer, simply say 'I don't know'.\n\n"
-    # # add the sentences to the prompt
-    # for sentence in top_15_sentences:
-    #         prompt += f"{sentence}\n"
-
-    # header = """Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text below, say "I don't know."\n\nContext:\n"""
-    # chosen_sections = "".join(top_15_sentences)
-    # prompt = header + chosen_sections + "\n\n Q: " + query + "\n A:"
-
-    # # add the top 15 sentences to the prompt
-    # for sentence in top_15_sentences:
-    #     prompt += f"\n{sentence}"
-
-    # # generate the answer using the OpenAI API
-    # response = openai.Completion.create(
-    #     engine="davinci", prompt=prompt, max_tokens=300, n=1, stop=None, temperature=0.001
-    # )
-    # # answer = response.choices[0].text.strip()
-
-    # COMPLETIONS_API_PARAMS = {
-    #     "temperature": 0.0,
-    #     "max_tokens": 300,
-    #     "model": "text-davinci-003",
-    # }
-
-    # response = openai.Completion.create(
-    #     prompt=prompt,
-    #     **COMPLETIONS_API_PARAMS
-    # )
-
-    st.write(textwrap.fill(response["choices"][0]["text"].strip(" \n")))
+try:
+    if st.button("Confirm!"):
+
+        cached_data = process_pdf_data(uploaded_files)
+        filt1_list = cached_data[0]
+        embeddings = cached_data[1]
+
+        # Creating embeddings for the query
+        query_embedding = op.create_query_embeddings(query)
+
+        # Calculating cosine similarity between the query and the sentences
+        cosine_lis = op.calculate_cosine(
+            query_embedding, embeddings, filt1_list)
+
+        # Fetching the top 15 sentences with the highest cosine similarity
+        indexes_final = op.fetch_top_rank_ans(cosine_lis, 15)
+
+        # Fetching the most relevant sentence from the top 15 sentences, and providing it as the context to the GPT-3 model
+        most_relevant = op.fetch_most_relevant(
+            indexes_final, filt1_list, cosine_lis, query)
+
+        # Calling the GPT-3 model to generate the answer
+        COMPLETIONS_API_PARAMS = {
+            "temperature": 0.0,
+            "max_tokens": 300,
+            "model": "text-davinci-003",
+        }
+
+        response = openai.Completion.create(
+            prompt=most_relevant,
+            **COMPLETIONS_API_PARAMS
+        )
+        # print("\n\n", textwrap.fill(
+        #     response["choices"][0]["text"].strip(" \n")))
+
+        # Displaying the answer to the user
+        st.write(textwrap.fill(response["choices"][0]["text"].strip(" \n")))
+except:
+    st.warning("Something went wrong. Please try again.")

operations.py

@@ -3,40 +3,57 @@ import numpy as np
 from sentence_transformers import SentenceTransformer
 import PyPDF2
 from nltk.tokenize import sent_tokenize
-import nltk
-# nltk.download('punkt')
-import sys
-# from dotenv import load_dotenv
-# load_dotenv('./env')
-
-sys.setrecursionlimit(5500)
 
 
 def read_pdf(fname):
+    """
+    This function reads the pdf file and extracts the text from it.
+
+    Parameters:
+    fname (str): Name of the pdf file
+
+    Returns:
+    text_ext (list): List of extracted text from the pdf file
+    """
     reader = PyPDF2.PdfReader(fname)
     text_ext = []
     for i in range(len(reader.pages)):
         pageObj = reader.pages[i]
         # extracting text from page
         text_ext.append(pageObj.extract_text())
-        # text_ext.append(pageObj.extract_text())
-    # print("len(text_ext) ", len(text_ext))
+
     return text_ext
 
 
 def sent_tokenize(text_ext):
+    """
+    This function apply sent_tokenize to the text and stores the result in a list.
+
+    Parameters:
+    text_ext (list): List of extracted text from the pdf file
+
+    Returns:
+    sent_toks (list): List of tokenized sentences
+    """
     sent_toks = []
 
     for i in text_ext:
         sent_toks.append(sent_tokenize(i))
     print("len(sent_toks) ", len(sent_toks))
 
-    # concat_list = [j for i in sent_toks for j in i]
-    # print(len(concat_list))
     return sent_toks
 
 
 def create_content_embeddings(concat_list):
+    """
+    This function creates embeddings for the document sentences.
+
+    Parameters:
+    concat_list (list): List of tokenized sentences
+
+    Returns:
+    embeddings (list): List of embeddings of the sentences
+    """
     model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
     embeddings = model.encode(concat_list)
 
@@ -44,17 +61,33 @@ def create_content_embeddings(concat_list):
 
 
 def create_query_embeddings(query_text):
+    """
+    This function creates embeddings for the query.
+    Parameters:
+    query_text (str): Query entered by the user
+
+    Returns:
+    query_embedding (list): List of embeddings of the query
+    """
+
     model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')
     query_embedding = model.encode(query_text)
     return query_embedding
 
 
 def calculate_cosine(query_embedding, embeddings, concat_list):
-    # cosine = np.dot(
-    #     query_embedding, embeddings[350]) / (norm(query_embedding)*norm(embeddings[350]))
+    """
 
-    # print("cosine ", cosine)
+    This function calculates cosine similarity between the query and the sentences.
 
+    Parameters:
+    query_embedding (list): List of embeddings of the query
+    embeddings (list): List of embeddings of the sentences
+    concat_list (list): List of tokenized sentences
+
+    Returns:
+    cosine_lis (list): List of cosine similarity values
+    """
     cosine_lis = []
 
     for i in range(len(concat_list)):
@@ -67,6 +100,17 @@ def calculate_cosine(query_embedding, embeddings, concat_list):
 
 
 def fetch_top_rank_ans(cosine_lis, N):
+    """
+    This function fetches the top N ranked sentences.
+
+    Parameters:
+    cosine_lis (list): List of cosine similarity values
+    N (int): Number of sentences to be ranked
+
+    Returns:
+    indexes_final (list): List of top N ranked sentences
+    """
+
     list1 = cosine_lis
     indexes_final = sorted(
         range(len(list1)), key=lambda i: list1[i], reverse=True)[:N]
@@ -84,14 +128,25 @@ def fetch_top_rank_ans(cosine_lis, N):
 
 
 def fetch_most_relevant(indexes_final, concat_list, list1, query):
+    """
+    This function fetches the most relevant sentences, pass it as a context to GPT-3 prompt along with user's query.
+
+    Parameters:
+    indexes_final (list): List of top N ranked sentences
+    concat_list (list): List of tokenized sentences
+    list1 (list): List of cosine similarity values
+    query (str): Query entered by the user
+
+    Returns:
+    prompt (str): GPT-3 prompt
+    """
+
     dicts = {}
 
     keys = indexes_final
     for i in keys:
         dicts[i] = concat_list[i]
 
-    # print("[dicts] ", [dicts])
-
     most_relevant_document_sections = [dicts]
 
     len(most_relevant_document_sections)
@@ -99,13 +154,10 @@ def fetch_most_relevant(indexes_final, concat_list, list1, query):
     chosen_sections = []
     chosen_sections_len = 0
     chosen_sections_indexes = []
-    document_section = []
 
     indices = range(len(list1))
     sorted_indices = sorted(indices, key=lambda i: list1[i], reverse=True)
-    print(len(indexes_final))
-
-    # indexes_final = []
+    # print(len(indexes_final))
 
     for section_index in range(len(indexes_final)):
 
@@ -117,7 +169,6 @@ def fetch_most_relevant(indexes_final, concat_list, list1, query):
 
     # Useful diagnostic information
     print(f"Selected {len(chosen_sections)} document sections:")
-    # print("\n".join(chosen_sections_indexes))
 
     header = """Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text below, say "I don't know."\n\nContext:\n"""