add api to update the pinecone database

sifars.md

@@ -0,0 +1,257 @@
+# Sifars
+
+## Brand Positioning
+- Whether you’re a start-up or an established enterprise looking to outshine with innovative ideas, our Enterprise Software Development Services helps business swiftness and efficiency with mobile and web solutions.
+- We engineer delightful exceptional and customer-grade experiences.
+- We love the work we do. We work closely with our clients to deliver the best possible solutions for their needs
+- We tailor products that serve as fuel to your business.
+- Driving Growth through Digital Innovation
+- Sifars synergizes talent and experience. We deliver the best Web, UX & Software development solutions for startups & enterprises.
+
+
+
+## Technologies Used by Sifars
+
+1. Artificial Intelligence
+    - **Generative AI**: Imagine an AI that can create new ideas for you. Our generative AI solutions can generate code, write content, or even compose music. The possibilities are endless.
+    - **AI Consultation**: We'll help you navigate the AI landscape. Our experts will guide you in choosing the right AI solution for your business.
+    - **Model Fine-Tuning**: We'll take your AI model to the next level. Our team of experts will fine-tune your model to achieve better results.
+    - **Conversational Chatbots**: Create a better customer experience with our conversational chatbots. We'll help you build a chatbot that can understand and respond to customer queries in a human-like way.
+    - **LLM Optimization**: Get the most out of your large language model (LLM). We'll help you optimize your LLM to achieve better performance and reduce costs.
+    - **Computer Vision**: Analyze and understand visual data with our computer vision solutions. We'll help you build a system that can detect objects, recognize faces, and more.
+    - **Predictive Analytics**: Make better decisions with our predictive analytics solutions. We'll help you build a system that can predict customer behavior, detect anomalies, and more.
+    - **Machine Learning**: Build a better future with machine learning. We'll help you build a system that can learn from data and make decisions on its own.
+    - **Natural Language Processing**: Build a better customer experience with natural language processing. We'll help you build a system that can understand and respond to customer queries in a human-like way.
+    - **Deep Learning**: Build a better AI model with deep learning. We'll help you build a model that can learn from data and make decisions on its own.
+    - **Reinforcement Learning**: Build a better AI model with reinforcement learning. We'll help you build a model that can learn from data and make decisions on its own.
+    - **Transfer Learning**: Build a better AI model with transfer learning. We'll help you build a model that can learn from data and make decisions on its own.
+    - **And more**: We have expertise in many other AI technologies, including neural networks, decision trees, random forests, and more.
+
+2. Web Development
+    - **Laravel**: Build a better web application with Laravel. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **UI/UX**: Create a better user experience with our UI/UX solutions. We'll help you design a user interface that is intuitive and easy to use.
+    - **Node Js**: Build a better web application with Node.js. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **React Js**: Build a better web application with React.js. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Vue Js**: Build a better web application with Vue.js. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Angular**: Build a better web application with Angular. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Python**: Build a better web application with Python. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Django**: Build a better web application with Django. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Ruby on Rails**: Build a better web application with Ruby on Rails. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **PHP**: Build a better web application with PHP. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Cloud App**: Build a better web application with cloud app. Our team of experts will help you build a web application that is fast, secure, and scalable.
+    - **Automation**: Automate your web application with our automation solutions. We'll help you build a system that can automate tasks and workflows.
+    - **Bot**: Build a better web application with bots. We'll help you build a system that can automate tasks and workflows.
+    - **Data Services**: Get the most out of your data with our data services. We'll help you build a system that can collect, store, and analyze data.
+    - **And more**: We have expertise in many other web development technologies, including HTML, CSS, JavaScript, and more.
+
+3. Mobile App Development:
+    - **Mobility**: Build a better mobile app with our mobility solutions. We'll help you build a mobile app that is fast, secure, and scalable.
+    - **Android Apps**: Build a better Android app with our Android app development solutions. We'll help you build a mobile app that is fast, secure, and scalable.
+    - **iOS Apps**: Build a better iOS app with our iOS app development solutions. We'll help you build a mobile app that is fast, secure, and scalable.
+    - **React Native**: Build a better mobile app with React Native. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **Ionic**: Build a better mobile app with Ionic. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **Flutter**: Build a better mobile app with Flutter. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **Kotlin**: Build a better Android app with Kotlin. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **Swift**: Build a better iOS app with Swift. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **Java**: Build a better Android app with Java. Our team of experts will help you build a mobile app that is fast, secure, and scalable.
+    - **And more**: We have expertise in many other mobile app development technologies, including Objective-C, C#, and more.
+
+4. DevOps
+    - **Docker**: Build a better application with Docker. Our team of experts will help you use Docker to build, ship, and run your application.
+    - **Kubernetes**: Build a better application with Kubernetes. Our team of experts will help you use Kubernetes to build, ship, and run your application.
+    - **Jenkins**: Build a better application with Jenkins. Our team of experts will help you use Jenkins to build, ship, and run your application.
+    - **GitLab CI/CD**: Build a better application with GitLab CI/CD. Our team of experts will help you use GitLab CI/CD to build, ship, and run your application.
+    - **Azure DevOps**: Build a better application with Azure DevOps. Our team of experts will help you use Azure DevOps to build, ship, and run your application.
+    - **Cloud Foundry**: Build a better application with Cloud Foundry. Our team of experts will help you use Cloud Foundry to build, ship, and run your application.
+    - **Google Cloud Build**: Build a better application with Google Cloud Build. Our team of experts will help you use Google Cloud Build to build, ship, and run your application.
+    - **CircleCI**: Build a better application with CircleCI. Our team of experts will help you use CircleCI to build, ship, and run your application.
+    - **Travis CI**: Build a better application with Travis CI. Our team of experts will help you use Travis CI to build, ship, and run your application.
+    - **And more**: We have expertise in many other DevOps technologies, including Ansible, Puppet, Chef, and more.
+
+5. Cyber Security
+    - **Penetration Testing**: Protect your business from cyber threats with our penetration testing solutions. We'll help you identify vulnerabilities in your system and fix them before they can be exploited.
+    - **Vulnerability Assessment**: Get a comprehensive view of your system's vulnerabilities with our vulnerability assessment solutions. We'll help you identify vulnerabilities and fix them before they can be exploited.
+    - **Compliance and Risk Management**: Get a comprehensive view of your system's compliance and risk management with our compliance and risk management solutions. We'll help you identify vulnerabilities and fix them before they can be exploited.
+    - **Incident Response**: Protect your business from cyber threats with our incident response solutions. We'll help you respond quickly and effectively to cyber threats.
+    - **Security Orchestration**: Protect your business from cyber threats with our security orchestration solutions. We'll help you automate and streamline your security processes.
+    - **Identity and Access Management**: Protect your business from cyber threats with our identity and access management solutions. We'll help you manage access to your system and ensure that only authorized users can access your data.
+    - **Network Security**: Protect your business from cyber threats with our network security solutions. We'll help you secure your network and ensure that only authorized users can access your data.
+    - **Cryptography**: Protect your business from cyber threats with our cryptography solutions. We'll help you encrypt your data and ensure that only authorized users can access it.
+    - **And more**: We have expertise in many other cyber security technologies, including encryption, access control, and more.
+
+6. Database Management
+    - **MySQL**: Store your data securely with MySQL. Our team of experts will help you design, deploy, and manage a MySQL database that is optimized for performance and scalability.
+    - **PostgreSQL**: Store your data securely with PostgreSQL. Our team of experts will help you design, deploy, and manage a PostgreSQL database that is optimized for performance and scalability.
+    - **MongoDB**: Store your data securely with MongoDB. Our team of experts will help you design, deploy, and manage a MongoDB database that is optimized for performance and scalability.
+    - **Cassandra**: Store your data securely with Cassandra. Our team of experts will help you design, deploy, and manage a Cassandra database that is optimized for performance and scalability.
+    - **Oracle**: Store your data securely with Oracle. Our team of experts will help you design, deploy, and manage an Oracle database that is optimized for performance and scalability.
+    - **Microsoft SQL Server**: Store your data securely with Microsoft SQL Server. Our team of experts will help you design, deploy, and manage a Microsoft SQL Server database that is optimized for performance and scalability.
+    - **IBM DB2**: Store your data securely with IBM DB2. Our team of experts will help you design, deploy, and manage an IBM DB2 database that is optimized for performance and scalability.
+    - **Amazon DynamoDB**: Store your data securely with Amazon DynamoDB. Our team of experts will help you design, deploy, and manage an Amazon DynamoDB database that is optimized for performance and scalability.
+    - **Google Cloud SQL**: Store your data securely with Google Cloud SQL. Our team of experts will help you design, deploy, and manage a Google Cloud SQL database that is optimized for performance and scalability.
+    - **And more**: We have expertise in many other database management technologies, including IBM Informix, Apache Cassandra, and more.
+
+7. Cloud Services
+    - **AWS**: Build a better cloud infrastructure with AWS. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **Azure**: Build a better cloud infrastructure with Azure. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **Google Cloud Platform**: Build a better cloud infrastructure with Google Cloud Platform. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **IBM Cloud**: Build a better cloud infrastructure with IBM Cloud. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **Rackspace**: Build a better cloud infrastructure with Rackspace. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **Cloud Foundry**: Build a better cloud infrastructure with Cloud Foundry. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **Heroku**: Build a better cloud infrastructure with Heroku. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **OpenStack**: Build a better cloud infrastructure with OpenStack. Our team of experts will help you design, deploy, and manage a cloud infrastructure that is optimized for performance and scalability.
+    - **And more**: We have expertise in many other cloud services, including Oracle Cloud, SAP Cloud Platform, and more.
+
+## Testimonials
+
+**Great Work Speaks for Itself**
+
+**Gabriela Bazan**  
+*Founder of GLB Design & Development*  
+"The Sifars Team is an arsenal of talented developers that I admire and trust. They’re a dream team, so fun to work with, and at the same time completely professional."
+
+---
+
+**Siamak Mehraghdam**  
+*Co-Founder of Qcentic*  
+"Sifars is an extended team to us. The level of their expertise, consultancy, and professionalism are commendable. Really responsive and good listeners, deliver exactly what one expects with a level of expertise."
+
+---
+
+**Santiago Sanches**  
+*Development Manager at Savage Global Marketing*  
+"The level of support during the whole development cycle was remarkable. I must say the team is highly professional, competent, and responsive."
+
+---
+
+**Barrister Naveed Khan**  
+*CEO of Askwakeel*  
+"They are very competent and reasonable with their pricing. The platform functions well without any bugs. Sifars is a highly deadline-oriented, communicative group."
+
+
+## Projects
+**Our Portfolio Embraces Apex and Extraordinary Projects**
+
+### Hoora
+*Introducing Hoora, the avant-garde multi-tenant employee management app redefining workplace efficiency. Hoora comes equipped with an advanced AI chatbot that not only provides information but actively assists employees across various organizational aspects. Seamlessly integrating with your database and organizational files, Hoora's AI chatbot streamlines tasks, offers real-time information, and ensures a more productive and engaged workforce.*
+
+---
+
+### Elevate.AI
+*Welcome to Elevate.AI, where cutting-edge technology meets personalized real estate solutions. Elevate.AI is a revolutionary project designed to seamlessly connect with potential real estate clients through the familiar and convenient platform of WhatsApp. Acting as your virtual human assistant, Elevate.AI employs advanced artificial intelligence to engage, generate leads, and schedule appointments with clients, making your real estate business more efficient and customer-centric.*
+
+---
+
+### Adamo
+*Personalized nutrition, wellness tips, travel insights, and more. Upcoming features include prompt and image generators, seamless social media integration, and a media center. Boost business productivity with AI integration. Welcome to Adamo – where possibilities abound.*
+
+---
+
+### SalesBod
+*A Chrome extension empowered with AI capabilities that helps salespeople gain information about companies and individuals. It enables salespeople to compose emails with a single prompt and generate POV. Additionally, it provides access to a user admin panel where users can manage their histories and company information related to executed prompts.*
+
+---
+
+### Travelidu
+*A custom web application that scrapes data from multiple providers and displays suitable vacations for users. It is empowered with an admin panel to manage all its bots and scrapers, along with data management functionalities. It provides various filters for holidays and locations to facilitate search. The admin panel empowers administrators to manage complex data with a user-friendly interface.*
+
+---
+
+### Online Donation Form Management Web Application
+*A custom web application that allows multiple clients to manage online donation forms. A robust and flexible application with technology integrated to seamlessly allow future enhancement and growth to the online donation processing system.*
+
+---
+
+### Indian Wedding & Event Planning Platform
+*A dedicated website for easy event planning and hiring resources for an event. Specially designed and curated for Indian weddings and events.*
+
+---
+
+### CatsBill
+*A Sifars initiative, an elegant and easy-to-use online portal to generate GST invoicing in India with many advanced features like inventory, stock transfer, and invoice management.*
+
+---
+
+### Instagram Auto Posting Platform
+*An interactive Instagram auto-posting platform engaging multiple profiles. A custom automation tool with features like scheduled follow, unfollow, and likes to organically grow Instagram followers.*
+
+---
+
+### iTunes Traffic Booster
+*An automation tool to increase the traffic of your song on iTunes.*
+
+---
+
+### Domotron
+*An interactive design made using PicoCMS and Pattern Labs.*
+
+---
+
+### Labonics WooCommerce Theme
+*A custom theme for a WooCommerce site to shop Labonics products integrated with an ERP system.*
+
+---
+
+### Get Connect
+*A cloud-based ETL software solution specifically designed for the banking sector to automate manual ETL processes into a design-enriched automated software program. The program is highly flexible and integrated with many administrative controls.*
+
+---
+
+### Advertorials.ai
+*Struggling to create ads that actually convert? Meet Advertorials.ai. Our AI-powered platform crafts top-performing advertorials in minutes, based on the best in the business. Plus, we streamline your workflow with project management tools, making collaboration a breeze. Test more, stress less, and watch your ad game soar with Advertorials.ai.*
+
+---
+
+### Forecasty.ai
+*Forecasty.ai is a self-service forecasting platform powered by advanced AI/ML technology, enabling you to generate forecasts easier and faster. It generates high-quality forecasts and provides predictive insights to make smarter business decisions. In-depth forecasts for commodity prices to prepare better in a volatile commodity market.*
+
+
+##  About Sifars
+
+Sifars, a pioneering web service provider, emerged onto the tech landscape in 2018 with a vision to revolutionize the digital sphere. Founded by visionary entrepreneurs Jatin Sethi, Munish Kumar, and Sukhwinder Singh, Sifars set its sights on empowering businesses worldwide with cutting-edge technology solutions. With its global headquarters nestled in the vibrant city of Patiala, Punjab, India, Sifars quickly garnered recognition as a leading application development company, committed to propelling businesses towards success in the ever-evolving tech landscape. 
+
+**Mission and Vision**
+
+At Sifars, we pride ourselves on our unwavering commitment to excellence and innovation. Our mission is simple yet profound: to tailor products that serve as the lifeblood of businesses, fueling their growth and success in the digital age. With a steadfast focus on driving growth through digital innovation, we strive to empower businesses of all sizes to establish their identity and thrive in the competitive tech world. Our vision is to be the catalyst for transformation, leveraging technology to unlock new possibilities and propel businesses towards unprecedented success.
+
+**Core Values**
+
+At the heart of Sifars lie our core values, which serve as the guiding principles in everything we do:
+
+* **Innovation:** We embrace innovation as the cornerstone of progress, constantly pushing boundaries and exploring new frontiers in technology. 
+* **Quality:** We are committed to delivering excellence in every endeavor, ensuring that our products and solutions meet the highest standards of quality and reliability. 
+* **Collaboration:** We foster a culture of collaboration and teamwork, recognizing that the collective genius of our team fuels our success. 
+* **Customer-Centricity:** We prioritize the needs and aspirations of our clients, striving to exceed their expectations and deliver unparalleled value. 
+* **Integrity:** We conduct ourselves with integrity and honesty, upholding the highest ethical standards in all our interactions and endeavors. 
+
+## Approach to Clients
+
+At Sifars, we believe in building strong, long-lasting relationships with our clients. Our success is intrinsically tied to theirs. That's why we take a highly personalized and client-centric approach to every project. 
+
+**We begin by truly understanding your vision:**
+
+* **We listen attentively:** We don't just hear your words, we actively listen to grasp the underlying needs and challenges driving your project. We ask questions to ensure we fully comprehend your goals and expectations.
+* **We leverage our industry expertise:** Our team is well-versed in various industries and can quickly tailor solutions to your specific needs.  Whether you're in healthcare, finance, or retail, we can understand your unique challenges and develop solutions that address them directly.
+* **We collaborate with you:** We believe in a collaborative approach to ideation. We brainstorm together to develop innovative and creative solutions that push the boundaries of what's possible.
+
+**Once we've gained a deep understanding, we focus on crafting a tailor-made solution:**
+
+* **Custom development is our specialty:**  We don't believe in one-size-fits-all solutions.  We create custom software that perfectly addresses your unique requirements and challenges, ensuring the final product is exactly what you need. 
+* **We embrace agile development:** This iterative process allows us to incorporate your feedback at every stage, ensuring that the final product meets your evolving needs. It's about constant collaboration and adaptation.
+* **We stay at the forefront of technology:** We are always learning and adapting, using the latest tools and frameworks to create the most efficient and cutting-edge solutions. We have the expertise to choose the right technology for your project, guaranteeing the best possible outcome.
+
+**But we don't stop at just delivering a product.**
+
+* **Transparency is key:** We believe in open and honest communication. We keep you informed every step of the way, providing regular updates on timelines, costs, and any potential challenges. We value transparency and trust in our client relationships.
+* **We're obsessed with quality:** Our rigorous testing and quality assurance processes guarantee a bug-free and highly functional product. We want you to be confident in the quality and reliability of our work.
+* **We're here for the long haul:**  We don't just deliver a product and move on.  We provide ongoing support and maintenance to ensure your solution continues to function smoothly and meets your evolving needs. We are committed to your long-term success.
+
+**At Sifars, we are passionate about building innovative solutions that help our clients achieve their goals.**  Our client-centric approach, combined with our expertise, commitment to quality, and dedication to long-term partnerships,  allows us to create truly impactful and transformative solutions. 
+
+## Work-Life Balance
+
+At Sifars, we understand the importance of maintaining a healthy work-life balance. In addition to our commitment to delivering excellence, we prioritize the well-being of our team members. To foster a positive work environment and promote team cohesion, we organize weekly strengthening bond calls, where team members come together to unwind and engage in recreational activities. These calls serve as a respite from the demands of work, allowing team members to recharge and rejuvenate, thereby enhancing overall productivity and morale. 
+
+
+## COntact Details
+

src/app.py

@@ -5,7 +5,7 @@ from fastapi.middleware.cors import CORSMiddleware
 
 from src.utils import logger
 
-from src.controllers import ws_router
+from src.controllers import ws_router, api_router
 
 
 @asynccontextmanager
@@ -37,4 +37,5 @@ app.add_middleware(
 async def check_health():
     return {"response": "Service is healthy!"}
 
-app.include_router(ws_router)
+app.include_router(ws_router)
+app.include_router(api_router, prefix="/api/v1")

src/controllers/__init__.py

@@ -1,9 +1,12 @@
 from ._chat_controller import ChatController
+from ._database_controller import DatabaseController
 
 ws_router = ChatController().router
+api_router = DatabaseController().router
 
 __all__ = [
     "ws_router",
+    "api_router",
 ]
 __version__ = "0.1.0"
 __author__ = "Aryan Jain"

src/controllers/_database_controller.py

@@ -0,0 +1,19 @@
+from fastapi import APIRouter, HTTPException
+
+from src.utils import logger
+
+from src.services import DatabaseService
+
+class DatabaseController:
+    def __init__(self):
+        self.database_service = DatabaseService
+        self.router = APIRouter(prefix="/update_database", tags=["database"])
+        self.router.add_api_route("/", self.update_database, methods=["POST"])
+
+    async def update_database(self, url: str = "https://sifars.com/"):
+        try:
+            async with self.database_service() as database_service:
+                await database_service._update_database(url)
+        except Exception as e:
+            logger.error(e)
+            raise HTTPException(status_code=500, detail=str(e))

src/crawler/__init__.py

@@ -0,0 +1,9 @@
+from ._web_crawler import WebCrawler
+from ._database_updater import DatabaseUpdater
+
+__all__ = [
+    "WebCrawler"
+    "DatabaseUpdater"
+    ]
+__version__ = "0.1.0"
+__author__ = "Aryan Jain"

src/crawler/_database_updater.py

@@ -0,0 +1,57 @@
+import aiofiles
+from src.utils import PineconeClient
+from src.crawler import WebCrawler
+import tiktoken
+
+class DatabaseUpdater:
+    def __init__(self):
+        self.model_name = 'gpt-3.5-turbo'
+        self.pinecone_client = PineconeClient
+        self.web_crawler = WebCrawler
+
+    async def __aenter__(self):
+        self.tokenizer = tiktoken.encoding_for_model(self.model_name)
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        pass
+
+    async def _split_text_with_overlap(self, text, token_limit, overlap_size):
+        tokens = self.tokenizer.encode(text)
+        chunks = []
+        i = 0
+        while i < len(tokens):
+            chunk = tokens[i:i + token_limit]
+            chunks.append(chunk)
+            i += token_limit - overlap_size
+        text_chunks = [self.tokenizer.decode(chunk) for chunk in chunks]
+        return text_chunks
+    
+    async def _update_database(self, sentences):
+        async with self.pinecone_client() as pinecone_client:
+            for i in range(0, len(sentences), 95):
+                await pinecone_client._upsert(sentences[i:i+95])
+        return
+    
+    async def _delete_old_database(self):
+        async with self.pinecone_client() as pinecone_client:
+            await pinecone_client._delete_index()
+        return
+    
+    async def _extract_new_data(self, url):
+        async with self.web_crawler(url) as crawler:
+            await crawler.crawl(url)
+        return
+    
+    async def _update_database_from_file(self, file_paths):
+        for file_path in file_paths:
+            async with aiofiles.open(file_path, 'r', encoding='utf-8') as f:
+                content = await f.read()
+            sentences = await self._split_text_with_overlap(content, 1024, 200)
+            await self._update_database(sentences)
+        return
+    
+    async def _clear_old_scrapped_file(self):
+        async with aiofiles.open("scrapped.txt", 'w', encoding='utf-8') as f:
+            await f.write("")
+        return

src/crawler/_web_crawler.py

@@ -0,0 +1,83 @@
+import time
+from selenium import webdriver
+from selenium.webdriver.chrome.service import Service
+from webdriver_manager.chrome import ChromeDriverManager
+from bs4 import BeautifulSoup
+import aiofiles
+
+from src.utils import logger
+from selenium.webdriver.common.by import By
+
+from selenium.common.exceptions import StaleElementReferenceException
+from selenium.webdriver.support.ui import WebDriverWait
+from selenium.webdriver.support import expected_conditions as EC
+
+
+
+class WebCrawler:
+    def __init__(self, base_url):
+        self.base_url = base_url
+        self.visited_links = set()
+        self.file_path = "scrapped.txt"
+        options = webdriver.ChromeOptions()
+        options.add_argument("--headless")
+        options.add_argument("--no-sandbox")
+        options.add_argument("--disable-dev-shm-usage")
+        self.driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()), options=options)
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        await self.close()
+
+    async def crawl(self, url):
+        if url in self.visited_links:
+            return
+        self.visited_links.add(url)
+        self.driver.get(url)
+        time.sleep(3) 
+        structured_text = await self.extract_structured_text()
+        if "404 Not Found" in structured_text:
+            return
+        async with aiofiles.open(self.file_path, 'a+', encoding='utf-8') as f:
+            await f.write(f'Page: {url}\n')
+            await f.write(f'{structured_text}\n')
+            await f.write("-" * 200 + "\n")
+        links = await self.get_links()
+        for link in links:
+            await self.crawl(link)
+
+    async def get_links(self):
+        links = set()
+        try:
+            WebDriverWait(self.driver, 10).until(
+                EC.presence_of_element_located((By.TAG_NAME, "a"))
+            )
+            anchors = self.driver.find_elements(By.TAG_NAME, "a")
+            for anchor in anchors:
+                try:
+                    href = anchor.get_attribute("href")
+                    if href and await self.is_valid_url(href):
+                        links.add(href)
+                except StaleElementReferenceException:
+                    logger.warning("Stale element encountered while getting href. Skipping...")
+        except StaleElementReferenceException as e:
+            logger.error(f"Failed to retrieve elements due to stale reference: {str(e)}")
+        return links
+
+    async def extract_structured_text(self):
+        soup = BeautifulSoup(self.driver.page_source, 'html.parser')
+        structured_content = []
+        for element in soup.find_all(['h1', 'h2', 'h3', 'p', 'ul', 'ol']):
+            if element.name in ['h1', 'h2', 'h3']:
+                structured_content.append(f"{element.name.upper()}: {element.get_text(strip=True)}")
+            else:
+                structured_content.append(f"TEXT: {element.get_text(strip=True)}")
+        return '\n'.join(structured_content)
+
+    async def is_valid_url(self, url):
+        return url.startswith(self.base_url) and not url.startswith("#")
+
+    async def close(self):
+        self.driver.quit()

src/services/__init__.py

@@ -1,9 +1,11 @@
 from ._connection_service import ConnectionService
 from ._chat_service import ChatService
+from ._database_service import DatabaseService
 
 __all__ = [
     "ConnectionService",
     "ChatService",
+    "DatabaseService"
 ]
 __version__ = "0.1.0"
 __author__ = "Aryan Jain"

src/services/_chat_service.py

@@ -25,5 +25,5 @@ class ChatService:
         message = f"Context is: {context} and User Query is: {messages[-1]['content']}"
         messages[-1]["content"] = message
         async with self.chat_client() as chat_client:
-            async for response in chat_client.chat(messages[2:]):
+            async for response in chat_client.chat(messages[-1:]):
                 yield response

src/services/_database_service.py

@@ -0,0 +1,21 @@
+from src.crawler import DatabaseUpdater
+
+from src.utils import logger
+
+class DatabaseService:
+    def __init__(self):
+        self.database_updater = DatabaseUpdater
+        self.file_paths = ["scrapped.txt", "sifars.md"]
+
+    async def __aenter__(self):
+        return self
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        pass
+
+    async def _update_database(self, url: str = ""):
+        async with self.database_updater() as database_updater:
+            await database_updater._clear_old_scrapped_file()
+            await database_updater._extract_new_data(url)
+            await database_updater._delete_old_database()
+            await database_updater._update_database_from_file(file_paths=self.file_paths)

src/utils/_chat_client.py

@@ -4,33 +4,25 @@ from huggingface_hub import InferenceClient
 INSTRUCTIONS="""
 You are an AI assistant designed by Sifars, a web development company, who gives answers to queries regarding Sifars, to the best of your ability.
 You will be provided a context along with each user query which you will use to answer the queries.
+Match the question with the context and according to that answer the user query.
 <RULES>
-- Keep answers as short and to the point as possible.
+- Keep your responses as short and to the point as possible. They should be concise, engaging, and easy to read.
+- Avoid including extra information not asked for in the user query. 
 - Use plural first person pronouns when talking about sifars.
-- Keep the answers short and engaging so that they are interesting and understandable to read.
-- Direct the user to the following links according to their interest:
-    - https://www.sifars.com/en/ Sifars's website
-    - https://www.sifars.com/en/about/ if user asks something about Sifars or if you do not know an answer regarding Sifars.
-    - https://www.sifars.com/en/services/ for more about the services we provide
-    - https://www.sifars.com/en/portfolio/ for more about our portfolio and projects
-    - https://www.sifars.com/en/technology/ for more about our technology stack
-    - https://www.sifars.com/en/career/ for more about our career opportunities
-- If user asks to reach out to us prompt them to do so by:
-    - Providing their contact details in the chat, and our experts can assist them with their query.
-    - Our email to [connect@sifars.com](mailto:connect@sifars.com)
-    - Visiting our contact page at [this link](https://www.sifars.com/en/contact/)
-    - Our address: "SCO 6, First Floor, Phulkian Enclave, Near Mini Secretariat, Patiala, Punjab 147001"
-    - Our phone number: +91 8106 455 950, +91 8008 296 463, +91 8896 720 000
-- Answer only using the context provided. Do not make up any responses.
-- If the provided context cannot be used at all to answer the query, send this exact message with no changes or precursors: "Thank you for reaching out! Our experts are ready to assist you with any queries you may have. Please provide your contact details, and we'll connect with you shortly. You can also reach us directly at [contact@sifars.com](mailto:contact@sifars.com) or visit our contact page at [this link](https://www.sifars.com/en/contact/). We look forward to helping you!". Do not add anything else to the message, whether before it or after.
-- Do not, for any reason whatsoever, mention the context provided, or the contents of the context, in your response.
+- Try to answer the query from the context provided. Even if the context does not directly answer the question, try to relate it to the question and formulate an answer.
+- Do not mention the context provided, or the contents of the context, for any reason in your message.
+- If someone asks about sifars, give them only the necessary details and then ask them to visit our about page at [this link](https://www.sifars.com/en/about/).
+- If someone asks about the services we provide, give them a bit of idea according to the interest of their query and then direct them to our services page at [this link](https://www.sifars.com/en/services/).
+- If someone asks about our projects or portfolio, provide them details of any three projects with minimal overview of those projects, according to the interest of their query and then direct them to our portfolio page at [this link](https://www.sifars.com/en/portfolio/).
+- If someone asks about our technology or tech stack that we use, give them a bit of idea according to the interest of their query and then direct them to our technology page at [this link](https://www.sifars.com/en/technology/).
+- If someone asks about career opportunities, direct them to our career page at [this link](https://www.sifars.com/en/career/).
 </RULES>
 """
 
 class ChatClient:
     def __init__(
         self,
-        model: str="meta-llama/Meta-Llama-3-8B-Instruct",
+        model: str="meta-llama/Llama-3.2-11B-Vision-Instruct",
         max_tokens: int=4096,
         stream: bool=True,
         system_message: str=INSTRUCTIONS

src/utils/_pinecone_client.py

@@ -1,4 +1,5 @@
 import os
+import uuid
 
 from pinecone.grpc import PineconeGRPC as Pinecone
 from pinecone import ServerlessSpec
@@ -71,7 +72,7 @@ class PineconeClient:
             index.upsert(
                 vectors=[
                     {
-                        "id": str(i),
+                        "id": str(uuid.uuid1()),
                         "values": embeddings.data[i]['values'],
                         "metadata": {
                             "text": text
@@ -88,9 +89,15 @@ class PineconeClient:
         index = await self._fetch_index(index_name)
         results = index.query(
             vector=embeddings.data[0]['values'],
-            top_k=top_k
+            top_k=top_k,
+            include_metadata=True
         )
-        ids = [result['id'] for result in results['matches']]
-        vectors = index.fetch(ids)['vectors']
-        for id in ids:
-            yield vectors[id]['metadata']['text']
+        for metadata in results['matches']:
+            yield metadata["metadata"]["text"]
+
+    async def _delete_index(self, index_name=None):
+        if not index_name:
+            index_name = self.index_name
+        if await self._is_index_exist(index_name):
+            self.pc.delete_index(index_name)
+        return