day2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e06c733a-1124-44b5-a634-37d0887fdfe6",
   "metadata": {},
   "source": [
    "# The Price is Right\n",
    "\n",
    "## Week 8 Order of Play\n",
    "\n",
    "Day 1: Modal.com and SpecialistAgent  \n",
    "Day 2: RAG, FrontierAgent, Ensemble Agent  \n",
    "Day 3: ScannerAgent, MessengerAgent  \n",
    "Day 4: AutonomousPlannerAgent and DealAgentFramework  \n",
    "Day 5: The Price Is Right Finale\n",
    "\n",
    "## RAG (Retrieval Augmented Generation) based on a dataset of 800,000 scraped Amazon products\n",
    "\n",
    "#### For our 2nd agent, we will be asking OpenAI to estimate the price of one of our deals - and we will give it a hand.\n",
    "\n",
    "We discovered that LLMs are really good at this, out of the box.\n",
    "\n",
    "And we discovered that we can beat a frontier LLM by fine-tuning an open-source LLM.\n",
    "\n",
    "Now we are going to try **inference time** techniques instead of training -- by using RAG!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2db71ba5-55a8-48b7-97d5-9db8dc872837",
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "\n",
    "import os\n",
    "import logging\n",
    "from dotenv import load_dotenv\n",
    "from huggingface_hub import login\n",
    "import numpy as np\n",
    "import re\n",
    "from sentence_transformers import SentenceTransformer\n",
    "import chromadb\n",
    "from sklearn.manifold import TSNE\n",
    "import plotly.graph_objects as go\n",
    "from litellm import completion\n",
    "from tqdm.notebook import tqdm\n",
    "from agents.evaluator import evaluate\n",
    "from agents.items import Item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b044d040-e467-4463-a3a5-119939ca8199",
   "metadata": {},
   "outputs": [],
   "source": [
    "# environment\n",
    "\n",
    "load_dotenv(override=True)\n",
    "DB = \"products_vectorstore\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c1cb7f1-41f7-4df8-95fa-f3143b4ce312",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Log in to HuggingFace\n",
    "# If you don't have a HuggingFace account, you can set one up for free at www.huggingface.co\n",
    "# And then add the HF_TOKEN to your .env file as explained in the project README\n",
    "\n",
    "hf_token = os.environ['HF_TOKEN']\n",
    "login(token=hf_token, add_to_git_credential=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15ec0e43",
   "metadata": {},
   "outputs": [],
   "source": [
    "LITE_MODE = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5b8f790",
   "metadata": {},
   "outputs": [],
   "source": [
    "username = \"ed-donner\"\n",
    "dataset = f\"{username}/items_lite\" if LITE_MODE else f\"{username}/items_full\"\n",
    "\n",
    "train, val, test = Item.from_hub(dataset)\n",
    "\n",
    "print(f\"Loaded {len(train):,} training items, {len(val):,} validation items, {len(test):,} test items\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf43f181-8b51-43c8-9763-599220cf6e66",
   "metadata": {},
   "source": [
    "# Now create a Chroma Datastore\n",
    "\n",
    "Now we will use the free, open-source Vector database Chroma.  \n",
    "We will create a Chroma datastore with 400,000 products from our training dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ba77914-ea9a-4b92-9280-863ee07ca8d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "client = chromadb.PersistentClient(path=DB)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1744c683-847a-4151-b6e0-56066f1fe4b0",
   "metadata": {},
   "source": [
    "# Introducing the SentenceTransformer Encoding LLM\n",
    "\n",
    "The all-MiniLM is a very useful model from HuggingFace that maps sentences & paragraphs to 384 dimensional vectors and is ideal for tasks like semantic search.\n",
    "\n",
    "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2\n",
    "\n",
    "It can run pretty quickly locally.\n",
    "\n",
    "As an alternative, OpenAI provides a closed-source Embeddings model. Benefits compared to OpenAI embeddings:\n",
    "1. It's free and fast!\n",
    "3. We can run it locally, so the data never leaves our box - might be useful if you're building a personal RAG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af2545a0-e160-41db-8914-f77b1c7eff26",
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32abb023-64b5-40a4-bfc1-e22c3ec31221",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pass in a list of texts, get back a numpy array of vectors\n",
    "\n",
    "vector = encoder.encode([\"A proficient AI engineer who has almost reached the finale of AI Engineering Core Track!\"])[0]\n",
    "print(vector.shape)\n",
    "vector"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa837b98-17ff-486e-ac30-a4b4f794af7b",
   "metadata": {},
   "source": [
    "## With that background, let's populate our Chroma database\n",
    "\n",
    "### By calculating vectors for 800,000 scraped products\n",
    "\n",
    "This takes 30 minutes on my machine on my GPU - it might take longer for you - feel free to use the Lite dataset!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1c8f101-9c81-462d-be2e-9b479831857f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if the collection exists; if not, create it\n",
    "\n",
    "collection_name = \"products\"\n",
    "existing_collection_names = [collection.name for collection in client.list_collections()]\n",
    "\n",
    "if collection_name not in existing_collection_names:\n",
    "    collection = client.create_collection(collection_name)\n",
    "    for i in tqdm(range(0, len(train), 1000)):\n",
    "        documents = [item.summary for item in train[i: i+1000]]\n",
    "        vectors = encoder.encode(documents).astype(float).tolist()\n",
    "        metadatas = [{\"category\": item.category, \"price\": item.price} for item in train[i: i+1000]]\n",
    "        ids = [f\"doc_{j}\" for j in range(i, i+1000)]\n",
    "        ids = ids[:len(documents)]\n",
    "        collection.add(ids=ids, documents=documents, embeddings=vectors, metadatas=metadatas)\n",
    "\n",
    "collection = client.get_or_create_collection(collection_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c65375e1-a8eb-4203-b8f1-dfff69a693cc",
   "metadata": {},
   "source": [
    "# Let's visualize the vectorized data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "202c3b08-dc89-4995-a25c-041417ec9b9b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# It is very fun turning this up to 800_000 and seeing the full dataset visualized,\n",
    "# but it almost crashes my box every time so do that at your own risk!! 10_000 is safe!\n",
    "\n",
    "MAXIMUM_DATAPOINTS = 10_000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c653565a-6405-4c5a-b925-7e14a17bf2da",
   "metadata": {},
   "outputs": [],
   "source": [
    "CATEGORIES = ['Appliances', 'Automotive', 'Cell_Phones_and_Accessories', 'Electronics','Musical_Instruments', 'Office_Products', 'Tools_and_Home_Improvement', 'Toys_and_Games']\n",
    "COLORS = ['cyan', 'blue', 'brown', 'orange', 'yellow', 'green' , 'purple', 'red']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a754334-69ef-4b4f-92c7-d7da89457f7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prework\n",
    "result = collection.get(include=['embeddings', 'documents', 'metadatas'], limit=MAXIMUM_DATAPOINTS)\n",
    "vectors = np.array(result['embeddings'])\n",
    "documents = result['documents']\n",
    "categories = [metadata['category'] for metadata in result['metadatas']]\n",
    "colors = [COLORS[CATEGORIES.index(c)] for c in categories]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a30b5e9-7dd9-45c1-a9a7-74cb22cdef2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try a 2D chart\n",
    "# TSNE stands for t-distributed Stochastic Neighbor Embedding - it's a common technique for reducing dimensionality of data\n",
    "\n",
    "tsne = TSNE(n_components=2, random_state=42)\n",
    "reduced_vectors = tsne.fit_transform(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7a97fc5-9f44-4f1d-a253-8c8f0bcd9ec9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the 2D scatter plot\n",
    "fig = go.Figure(data=[go.Scatter(\n",
    "    x=reduced_vectors[:, 0],\n",
    "    y=reduced_vectors[:, 1],\n",
    "    mode='markers',\n",
    "    marker=dict(size=4, color=colors, opacity=0.7),\n",
    "    text=[f\"Category: {c}<br>Text: {d[:50]}...\" for c, d in zip(categories, documents)],\n",
    "    hoverinfo='text'\n",
    ")])\n",
    "\n",
    "fig.update_layout(\n",
    "    title='2D Chroma Vectorstore Visualization',\n",
    "    scene=dict(xaxis_title='x', yaxis_title='y'),\n",
    "    width=1200,\n",
    "    height=800,\n",
    "    margin=dict(r=20, b=10, l=10, t=40)\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0bdb35e3-bd0d-4569-872b-34bea8316675",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's try 3D!\n",
    "\n",
    "tsne = TSNE(n_components=3, random_state=42)\n",
    "reduced_vectors = tsne.fit_transform(vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4361c151-9f1b-4652-9204-695baf3860d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the 3D scatter plot\n",
    "fig = go.Figure(data=[go.Scatter3d(\n",
    "    x=reduced_vectors[:, 0],\n",
    "    y=reduced_vectors[:, 1],\n",
    "    z=reduced_vectors[:, 2],\n",
    "    mode='markers',\n",
    "    marker=dict(size=2, color=colors, opacity=0.7),\n",
    "    text=[f\"Category: {c}<br>Text: {d[:50]}...\" for c, d in zip(categories, documents)],\n",
    "    hoverinfo='text'\n",
    ")])\n",
    "\n",
    "fig.update_layout(\n",
    "    title='3D Chroma Vector Store Visualization',\n",
    "    scene=dict(xaxis_title='x', yaxis_title='y', zaxis_title='z'),\n",
    "    width=1200,\n",
    "    height=800,\n",
    "    margin=dict(r=20, b=10, l=10, t=40)\n",
    ")\n",
    "\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "caa8e99f",
   "metadata": {},
   "outputs": [],
   "source": [
    "test[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4b72f1b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def vector(item):\n",
    "    return encoder.encode(item.summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5dc3419",
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_similars(item):\n",
    "    vec = vector(item)\n",
    "    results = collection.query(query_embeddings=vec.astype(float).tolist(), n_results=5)\n",
    "    documents = results['documents'][0][:]\n",
    "    prices = [m['price'] for m in results['metadatas'][0][:]]\n",
    "    return documents, prices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98de579a",
   "metadata": {},
   "outputs": [],
   "source": [
    "find_similars(test[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c74a25b1-f93c-4a75-9999-09e262f9abc9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We need to give some context to GPT-5.1 by selecting 5 products with similar descriptions\n",
    "\n",
    "def make_context(similars, prices):\n",
    "    message = \"For context, here are some other items that might be similar to the item you need to estimate.\\n\\n\"\n",
    "    for similar, price in zip(similars, prices):\n",
    "        message += f\"Potentially related product:\\n{similar}\\nPrice is ${price:.2f}\\n\\n\"\n",
    "    return message"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b10dd77",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents, prices = find_similars(test[0])\n",
    "print(make_context(documents, prices))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "05b57490-060d-47ff-9cf0-2b61b455bcd8",
   "metadata": {},
   "outputs": [],
   "source": [
    "def messages_for(item, similars, prices):\n",
    "    message = f\"Estimate the price of this product. Respond with the price, no explanation\\n\\n{item.summary}\\n\\n\"\n",
    "    message += make_context(similars, prices)\n",
    "    return [{\"role\": \"user\", \"content\": message}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a81ff141",
   "metadata": {},
   "outputs": [],
   "source": [
    "documents, prices = find_similars(test[0])\n",
    "print(messages_for(test[0], documents, prices)[0]['content'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "721ab130-b6d8-4356-9704-687c9bc2636f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The function for gpt-5-mini\n",
    "\n",
    "def gpt_5__1_rag(item):\n",
    "    documents, prices = find_similars(item)\n",
    "    response = completion(model=\"gpt-5.1\", messages=messages_for(item, documents, prices), reasoning_effort=\"none\", seed=42)\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "613a6278-2da5-4bf8-b733-2947736feb63",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How much does our favorite distortion pedal cost?\n",
    "\n",
    "test[0].price"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d30586f2-6b84-4750-acf5-a113ac9ccb48",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's do this!!\n",
    "\n",
    "gpt_5__1_rag(test[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "846067a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate(gpt_5__1_rag, test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb035c28",
   "metadata": {},
   "outputs": [],
   "source": [
    "import modal\n",
    "Pricer = modal.Cls.from_name(\"pricer-service\", \"Pricer\")\n",
    "pricer = Pricer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "715300c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def specialist(item):\n",
    "    return pricer.price.remote(item.summary)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3acd1a24",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_price(reply):\n",
    "    reply = reply.replace(\"$\", \"\").replace(\",\", \"\")\n",
    "    match = re.search(r\"[-+]?\\d*\\.\\d+|\\d+\", reply)\n",
    "    return float(match.group()) if match else 0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "944a9b15",
   "metadata": {},
   "source": [
    "## Download the Neural Network weights from Week 6 into this directory\n",
    "\n",
    "The file `deep_neural_network.pth` here:\n",
    "\n",
    "https://drive.google.com/drive/folders/1uq5C9edPIZ1973dArZiEO-VE13F7m8MK?usp=drive_link"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b45acd2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from agents.deep_neural_network import DeepNeuralNetworkInference\n",
    "\n",
    "runner = DeepNeuralNetworkInference()\n",
    "runner.setup()\n",
    "runner.load(\"deep_neural_network.pth\")\n",
    "\n",
    "def deep_neural_network(item):\n",
    "    return runner.inference(item.summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6327b3bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def ensemble(item):\n",
    "    price1 = get_price(gpt_5__1_rag(item))\n",
    "    price2 = specialist(item)\n",
    "    price3 = deep_neural_network(item)\n",
    "    return price1 * 0.8 + price2 * 0.1 + price3 * 0.1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9624bce1",
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate(ensemble, test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2684714-e6d0-47d6-bf31-87fe349fc15c",
   "metadata": {},
   "outputs": [],
   "source": [
    "root = logging.getLogger()\n",
    "root.setLevel(logging.INFO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c293f0a8-7097-4744-a2e9-7da5268406a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from agents.frontier_agent import FrontierAgent\n",
    "\n",
    "agent = FrontierAgent(collection)\n",
    "agent.price(\"Quadcast HyperX condenser mic, connects via usb-c to your computer for crystal clear audio\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8043d5e4-ec42-4fa0-9344-28a856d4f6d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.price(\"Shure MV7+ professional podcaster microphone with usb-c and XLR outputs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39799a17-20ad-45ef-8c04-4712f189c9d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from agents.neural_network_agent import NeuralNetworkAgent\n",
    "agent = NeuralNetworkAgent()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "029d5dd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.price(\"Shure MV7+ professional podcaster microphone with usb-c and XLR outputs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "de6ace7d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from agents.ensemble_agent import EnsembleAgent\n",
    "agent = EnsembleAgent(collection)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72533425",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent.price(\"Shure MV7+ professional podcaster microphone with usb-c and XLR outputs\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c22c03a4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}