{ "cells": [ { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from langchain.document_loaders import DirectoryLoader,PyPDFLoader\n", "from langchain.text_splitter import RecursiveCharacterTextSplitter" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def load_docs(directory_path):\n", " loader = DirectoryLoader(directory_path, glob=\"**/*.pdf\", loader_cls=PyPDFLoader)\n", " documents = loader.load()\n", " return documents" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "doc = load_docs(\"data\")" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'source': 'data\\\\a16-ahn.pdf', 'page': 10}\n", "16:10 J. Ahn et al.\n", "Our system uses a commercial image labeling Web service, called IQEngines\n", "[IQEngines 2013], to determine the user’s initial starting position when using our\n", "grocery shopping application. IQEngines uses a combination of computer vision and\n", "crowdsourcing to tag a photo with a label describing the content of the image. For\n", "example, an image of a box of Frosted Cheerios cereal might be labeled “General Mills\n", "Frosted Cheerios”. When an image is submitted to IQEngines, the image is first pro-\n", "cessed by a computer vision system in an effort to provide an accurate label. If the\n", "computer vision system cannot identify the image, then IQEngines passes the image to\n", "its crowdsourcing network for analysis and tagging. According to IQEngines, the time\n", "to return a label for an image varies from a few seconds for the computer vision sys-\n", "tem, to a few minutes for the crowdsourcing system. To ensure fast image labeling in\n", "our experiments, we have pretrained IQEngines with specified images and associated\n", "labels for each of the food items in our test environment.\n", "To locate a user within the indoor shopping environment, our mobile application\n", "prompts the user to take a picture of the nearest food item using the smartphone. After\n", "this image is submitted to our cloud server, the server submits the image to IQEngines\n", "for labeling. Upon receiving the item label for the image, our server looks up the loca-\n", "tion for this item using a spatial database. This spatial database contains the name,\n", "location, and other associated metadata for each item found in the shopping environ-\n", "ment. In our grocery shopping application, the coordinate system for item locations\n", "is expressed using the following dimensions: aisle number, aisle side (left or right),\n", "division number, shelf number, and item sequence number. Based on our conversations\n", "with local supermarkets, we have found that this coordinate system is representative\n", "of item databases found at some establishments. In this coordinate system, aisles are\n", "separated into 4-foot divisions, and shelves in each aisle are numbered from bottom\n", "to top. Items in each location specified by a tuple of “aisle number, aisle side, division\n", "number, shelf number” are ordered according to item sequence number. Figure 2 shows\n", "a graphical representation of this coordinate system for a typical grocery store aisle.\n", "4.2. Localizing the User within the Grocery Aisle\n", "As mentioned earlier, our approach is to apply prior work in personalized pedometry\n", "to the aisle navigation problem. However, some adaptation is needed for the grocery\n", "store scenario. We observed shoppers’ behavioral patterns as they used our application\n", "in grocery stores. We found that the users did not always hold the phone consistently,\n", "upright, pointed forward down the aisle. The mobile phone’s orientation was often\n", "changed, whenever they moved towards food items recommended by the AR tags of\n", "our application. When they moved towards the products they wished to purchase, they\n", "usually changed the mobile phone’s orientation, such as holding it down by their side\n", "while walking or in a strange angle while holding a basket or operating a cart. When-\n", "ever this happened, the accelerometer sensor incorrectly detected a stride. To avoid\n", "these false strides, we modified the pedometry algorithm to ignore sudden changes in\n", "acceleration if they also corresponded with sudden changes in the orientation sensor.\n", "Another modification we made to the pedometry algorithm was to limit motion to\n", "the component in the direction parallel to the long axis of the aisle. This is essentially\n", "1D map matching, wherein the walls of the aisle form a map that confines the travels\n", "of the user to a set of acceptable paths, or in this case a single path. In this way, our\n", "algorithm cannot misestimate the user as being located within a shelf/wall, and thus\n", "our location error is limited to lie only along the long axis of the aisle.\n", "To achieve this, we construct a bounding box around each aisle, where we bound\n", "the range of the x-axis by the width of a regular aisle in the grocery. When the user\n", "approaches the edge of the bounding box, for example, the shelves, then we only take\n", "the component of the motion along the axis parallel to the bounding edge, and ignore\n", "ACM Trans. Multimedia Comput. Commun. Appl., Vol. 12, No. 1s, Article 16, Publication date: October 2015.\n" ] } ], "source": [ "print(doc[10].metadata)\n", "print(doc[10].page_content)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def make_chunks(docs):\n", " text_splitter = RecursiveCharacterTextSplitter(\n", " chunk_size=1000,\n", " chunk_overlap=200,\n", " length_function=len,\n", " add_start_index=True\n", " )\n", "\n", " chunks = text_splitter.split_documents(docs)\n", "\n", " return chunks" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Document(metadata={'source': 'data\\\\a16-ahn.pdf', 'page': 3, 'start_index': 0}, page_content='Supporting Healthy Grocery Shopping via Mobile Augmented Reality 16:3\\nAR tag grows in size. The tags when clicked reveal nutritional information about the\\nproduct. The tags are also colored, for example, green to indicate products that are\\nnutritionally preferable (e.g., low-calorie, gluten-free), and red to indicate products to\\navoid (e.g., high cholesterol or peanut content). Further, shoppers can specify health\\nprofiles which may impact their food purchase choices, such as weight control, heart\\ndisease, food allergies, etc. The recommended products shown via AR tags will then\\nchange depending on what health condition/concern is indicated by the user. We believe\\nour system is the first to integrate augmented reality tagging and pedometry-based\\nlocalization with a back-end server to provide health-based grocery recommendations\\nat the point-of-purchase. We evaluated the effectiveness of our system in a real gro-')" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "chunks = make_chunks(doc)\n", "chunks[10]" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Document(metadata={'source': 'data\\\\a16-ahn.pdf', 'page': 3, 'start_index': 754}, page_content='localization with a back-end server to provide health-based grocery recommendations\\nat the point-of-purchase. We evaluated the effectiveness of our system in a real gro-\\ncery store aisle with 15 actual grocery shopping subjects to determine how easy and\\nfast the subjects reported it was to locate healthy food products and avoid unhealthy\\nones, using AR tagging with our application. We also evaluated our application’s func-\\ntionality and performance by analyzing data we collected from 104 online application\\ndemonstration/survey participants.\\n2. RELATED WORK\\nAugmented reality has been recently applied in the mobile health arena in a variety\\nof applications. For example, AR tags are overlaid in a home environment to pro-\\nvide instructions to the elderly for tasks like taking medication, cooking, washing,\\netc. [Herv´as et al. 2011]. TriggerHunter is an AR-based game that overlays tags on\\npotential asthma triggers in the local environment [Hong et al. 2010]. Neither project')" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "chunks[11]" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# from langchain.embeddings import HuggingFaceEmbeddings,OpenAIEmbeddings\n", "from langchain.embeddings.huggingface import HuggingFaceInferenceAPIEmbeddings\n", "# import chromadb\n", "# from chromadb.utils import embedding_functions\n", "from langchain_chroma import Chroma\n", "from chromadb import Client\n", "from chromadb.config import Settings\n", "\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "from dotenv import load_dotenv\n", "import os\n", "\n", "load_dotenv()\n", "\n", "HF_TOKEN = os.getenv('HF_TOKEN')" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "def save_embeddings(chunks, embedding_function,CHROMA_PATH=\"chroma_db\"):\n", " chroma = Chroma.from_documents(\n", " documents=chunks,\n", " embedding=embedding_function,\n", " persist_directory=CHROMA_PATH,\n", " collection_name=\"my_collection\"\n", " )\n", " # chroma.persist()\n", " print(f\"Saved {len(chunks)} chunks to {CHROMA_PATH}.\")\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "\n", "# Define Chroma database path\n", "CHROMA_PATH = \"chroma\"\n", "\n", "# Initialize Hugging Face embedding\n", "hugging_face_ef = HuggingFaceInferenceAPIEmbeddings(\n", " api_key=HF_TOKEN,\n", " model_name=\"sentence-transformers/all-MiniLM-L6-v2\"\n", ")\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Saved 162 chunks to chroma_db.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "C:\\Users\\Dell\\AppData\\Local\\Temp\\ipykernel_22236\\452922040.py:8: LangChainDeprecationWarning: Since Chroma 0.4.x the manual persistence method is no longer supported as docs are automatically persisted.\n", " chroma.persist()\n" ] } ], "source": [ "# save_embeddings(chunks, hugging_face_ef,CHROMA_PATH)" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "db = Chroma(\n", " persist_directory=CHROMA_PATH,\n", " embedding_function=hugging_face_ef\n", " )" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'ids': ['e34aab38-8069-440e-8723-d1cedfe48b93',\n", " '3fa98ab3-e075-4b88-8165-48096c60ca70',\n", " '1e771030-2db0-4cea-97ad-9d8afdbff741',\n", " '3fbb6928-c00a-4a6b-8aa0-de2e7bbca3e4',\n", " '056aba92-2c01-410c-be0e-424b2d52167e',\n", " '5fac6c9a-fc34-479a-bda9-a8ea03876e68',\n", " '992ad3e3-15e9-4ba1-a9c8-7e7db6fb8907',\n", " '056f4dce-6e4e-4801-8601-e664382060e2',\n", " 'e5841c9e-c287-45c2-a0af-7a76c1a5dd98',\n", " 'e926c90d-9e5f-4b57-8531-b2bb96134716',\n", " '2bf05434-f855-4234-bccb-bb33931e06c8',\n", " '9380898c-140d-46e5-955a-2d415d911c98',\n", " 'e6f6fd3f-6bb8-4118-9b43-516e1ffbc550',\n", " 'f13fa58c-16d0-41dd-8c3d-f3cba6d637c1',\n", " 'dd52bec5-97bb-4ef1-b0bd-8a7d7acc190a',\n", " '45e50f14-f4b6-4eb1-8fdd-a177c897cbfe',\n", " 'cad78450-a41c-4f3b-bf4f-370a05d8d1b9',\n", " 'c33cadd3-dcfd-4c24-b990-5c96f41e55d3',\n", " 'f6b9b2cb-a955-49f5-b14e-93a25e27f432',\n", " 'da6ad6f4-87b5-423a-afa6-8d7d30f6bd58',\n", " 'f9453981-a5ab-4aa4-bede-2ce6c52d5d19',\n", " '540db486-e8dd-4606-9c1d-f132e3ab8be1',\n", " '03604131-da4e-432b-a9fa-c0f0dbd95489',\n", " '649f67a5-2cd3-419a-a8a8-b4f07353d1f7',\n", " '8288e8cc-3183-4cae-812c-b5d1f393d337',\n", " '987957fd-abe6-40ad-aa37-9aa327ecd8ed',\n", " '98d4884b-ca95-442f-ba70-98888168583b',\n", " '33b7f9a6-75fb-4085-9ed5-8261cd05cb42',\n", " '9fbde644-be9d-4a5c-a0be-8316c84334fc',\n", " '786d5a04-f953-4f20-9da1-411ab145df60',\n", " '793b0b96-573a-4475-88ea-d062ece260f8',\n", " 'a75bef1b-42c3-49ad-b302-ac9654c605ed',\n", " '4020f743-2b45-4e64-ae39-e1dd67f12615',\n", " '7e40de2c-1f1d-4327-850d-45bc6450f503'],\n", " 'embeddings': array([[-0.11553047, -0.05751311, -0.08627735, ..., -0.00305746,\n", " 0.00041176, 0.0331987 ],\n", " [-0.11386822, -0.01693425, -0.00576754, ..., -0.01278613,\n", " -0.01054305, 0.05054566],\n", " [-0.09244151, -0.04838978, -0.007405 , ..., 0.04828871,\n", " 0.01121346, 0.02583535],\n", " ...,\n", " [-0.07942896, -0.00332908, -0.02907163, ..., 0.01292658,\n", " 0.04750381, -0.0322074 ],\n", " [-0.09497212, 0.00745012, -0.01084985, ..., -0.04577165,\n", " -0.0279846 , -0.00024868],\n", " [-0.00399996, 0.02088188, -0.04878428, ..., -0.03420498,\n", " -0.0191393 , 0.04296514]]),\n", " 'documents': None,\n", " 'uris': None,\n", " 'data': None,\n", " 'metadatas': None,\n", " 'included': []}" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "db.get(include=['embeddings'])" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "def get_top_docs(query,k=4):\n", " # load db\n", " db = Chroma(\n", " persist_directory=CHROMA_PATH,\n", " embedding_function=hugging_face_ef\n", " )\n", " \n", " # search db\n", " results = db.similarity_search_with_relevance_scores(query,k=k)\n", " \n", " return results\n", " " ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "\n", "# get query input\n", "# query = input(\"Enter your query: \")\n", "query = \"what is FoodKG how it uses AR ?\"\n", "\n", "result = get_top_docs(query)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "[(Document(metadata={'file_id': '0', 'session_id': 't-Sdfp5aqCsRH4hrsbTNRZ5EvskxunVBspFm9fU_xRM', 'source': 'uploads\\\\ar_influence_on_market.md', 'start_index': 3201}, page_content='Sources admit that, despite the problems in AR, it has uses and future directions.\\n\\nAR can be used to create interactive menus in restaurants so that customers can view three-dimensional visualizations of the dishes and the lists of ingredients in detail [4]. It may enhance customer engagement and possibly your customer choice.\\n\\nAR can be embedded in food packaging so that consumers get a rich interaction experience [4]. Scanning the packaging shall unlock for users information regarding the nutritional data, the origin, and the process of manufacture along with virtual representations of the food. Such enhanced transparency and engagement might impact the purchasing decision.\\n\\nAR applications can be further developed to aid in new food product development [4]. Developers can easily identify possible problems and then optimize a product for actual production by virtue of virtualization of the product followed by its analysis in real environments or simulated environments.'),\n", " 0.28280806368958833),\n", " (Document(metadata={'file_id': '0', 'session_id': 't-Sdfp5aqCsRH4hrsbTNRZ5EvskxunVBspFm9fU_xRM', 'source': 'uploads\\\\food_ingredient_dataset.txt', 'start_index': 3508}, page_content='assistance. The interface of the application focused on simplicity in daily use. [3] \\nTherefore, developers have to face the specific functional-related issues pertaining to such applications. For\\nexample, food scanner applications require one to scan the barcodes of each article purchased in the\\nmarket. It is cumbersome and time-consuming in the process. These features, combined with overall\\ninformation regarding nutritional content usually available, pose a burdensome job for the user to go through\\nthe information and take the right decisions. [3] \\nOn the contrary, FOP labels present a much less complicated approach. FOP labeling acquires the real\\nsimplified nutritional information on product packaging. This ease of access and processing becomes a\\ngood reason why FOP labels are more effective in determining consumer choices compared to food scanner\\napps. [11] \\nAnother great UI/UX is restaurant AR menus. With restaurant AR, the wish of the customers can be'),\n", " 0.2103979693981065),\n", " (Document(metadata={'file_id': '0', 'session_id': 't-Sdfp5aqCsRH4hrsbTNRZ5EvskxunVBspFm9fU_xRM', 'source': 'uploads\\\\food_ingredient_dataset.txt', 'start_index': 886}, page_content=\"they liked and wanted by preference and need[5].\\nAnother application is regarding ingredient analysis especially concerning food allergies. This is done using\\na barcode scanner and OCR by scanning the ingredients against a food and health database having known\\nallergens. Ingredients scanned will alert the users on those materials that can cause allergic reactions. [5] \\nARFusion is the grocery shopping application enabled with AR based on a health-based model of nutrition,\\nand has personal and family profiles filled into it. It requires relating data from product database tracking\\ninformation of every product's ingredient, nutrient, and the location in the grocery, which information allows\\nthe application to make real-time personalized recommendations on healthy products. [9] \\nMoreover, the substitution of ingredients is performed using knowledge graphs. One such system was\\ncreated based on a knowledge graph called FoodKG which interconnected the ingredients with the food\"),\n", " 0.20021785362753386),\n", " (Document(metadata={'file_id': '0', 'session_id': 't-Sdfp5aqCsRH4hrsbTNRZ5EvskxunVBspFm9fU_xRM', 'source': 'uploads\\\\ar_influence_on_market.md', 'start_index': 2383}, page_content=\"Reasons for Differences in Performance:\\n\\nThere are several reasons that can explain differences in performance. FOPs give consumers a look from the front of the package nutrition information directly accessible and easy to process. With AR applications, especially if they require scanning the barcode, their use will require more effort from the user—which may interfere with the proper processing of the information or even reduce the making of an informed choice.\\n\\nSource [13] concludes that food scanner apps are ineffective as substitutes for FOP labels since the information is far more extensive and multiple scans are required for comparison of packaged products. Such multilevel processing of information might, in turn, sub-optimally affect consumers' choices.\\n\\nPotential Applications and Future Directions\\n\\nSources admit that, despite the problems in AR, it has uses and future directions.\"),\n", " 0.18317890961800543)]" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "result" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "d:\\Coding\\RAG using langchain\\venv\\Lib\\site-packages\\tqdm\\auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", " from .autonotebook import tqdm as notebook_tqdm\n" ] } ], "source": [ "import google.generativeai as genai\n", "import os\n", "\n", "genai.configure(api_key=os.environ[\"GOOGLE_API_KEY\"])\n", "model = genai.GenerativeModel(\"gemini-1.5-flash\")" ] }, { "cell_type": "code", "execution_count": 26, "metadata": {}, "outputs": [], "source": [ "def generate_response(query):\n", " PROMPT_TEMPLATE = \"\"\"\n", "Answer the question given based only on the context given below.\n", "context:\n", "{context}\n", "\n", "---\n", "based on the context above answer the following query:\n", "{query}\n", "\"\"\"\n", " top_3_results = get_top_docs(query,k=3)\n", " # if top result is not releavent that is score less than 0.6\n", " if top_3_results[0][1] < 0.6:\n", " print(\"Query is not relevant to the provided material. So the output might not be accurate.\")\n", " \n", " context = \"\\n\".join([result[0].page_content for result in top_3_results])\n", " prompt = PROMPT_TEMPLATE.format(context=context,query=query)\n", " response = model.generate_content(prompt)\n", " print(\"Query:\",query)\n", " print(\"Response:\" ,end=\" \")\n", " for chunk in model.generate_content(prompt, stream=True):\n", " print(chunk.text, end=\"\", flush=True) \n", " print()\n", " print(\"Sources:\" + \",\".join([result[0].metadata['source'] + f\" [Page:{int(result[0].metadata['page']) + 1}]\" if 'page' in result[0].metadata else \"\" for result in top_3_results]))\n", "\n", " " ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Query is not relevant to the provided material. So the output might not be accurate.\n", "Query: how ar affect consumer behaivour\n", "Response: Based on the provided text, AR's influence on consumer behavior is mixed. While studies show potential for AR to encourage healthier food choices through personalized suggestions and ingredient substitutions, its effectiveness is lower than traditional methods like FOP labeling. A gap exists between intentions to buy healthier products (indicated by AR apps) and actual behavior in real-world shopping situations. The complexity and ease of use of AR applications are significant hurdles to overcome in order to maximize their impact on consumer behavior.\n", "\n", "Sources:data\\review_paper.pdf [Page:7],data\\review_paper.pdf [Page:6],data\\review_paper.pdf [Page:6]\n" ] } ], "source": [ "\n", "generate_response(input())\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.6" } }, "nbformat": 4, "nbformat_minor": 2 }