src/lmms-eval/tools/live_bench/example.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/pufanyi/anaconda3/anacondabin/envs/live_bench/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from random import sample\n",
    "\n",
    "from live_bench.websites.website import DefaultWebsite\n",
    "from live_bench.websites import load_websites\n",
    "\n",
    "# website = load_websites()\n",
    "# website = sample(website, 1)\n",
    "# website[0].url\n",
    "website = [DefaultWebsite(url=\"https://www.asahi.com/\")]  # , DefaultWebsite(url=\"https://www.bbc.com/sport\"), DefaultWebsite(url=\"https://www.bbc.com/business\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "could not detect version_main.therefore, we are assuming it is chrome 108 or higher\n"
     ]
    }
   ],
   "source": [
    "from live_bench.data_generator.utils.extract_information import InfomationExtractor\n",
    "from live_bench.screen_shoter import get_shoter\n",
    "from live_bench.driver import load_driver\n",
    "\n",
    "shoter = get_shoter(\"single_screen\")\n",
    "driver = load_driver()\n",
    "w = shoter(driver, website[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "extractor = InfomationExtractor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = extractor.extract_information(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "**Here is something you can take as reference.**\n",
      "\n",
      "## Text Extracted in the HTML\n",
      "\n",
      "Below is the text extracted from the website for you to take reference:\n",
      "BBC Home - Breaking News, World News, US News, Sports, Business, Innovation, Climate, Culture, Travel, Video & Audio\n",
      "\n",
      "What son's conviction means for President Biden\n",
      "The guilty verdict is unlikely to change voters' minds, but it will be a personal blow for the US president.\n",
      "1 hr ago | US & Canada\n",
      "\n",
      "Hunter Biden found guilty on all counts in gun trial\n",
      "The US president's son is found guilty of lying about his drug use when buying a handgun in 2018.\n",
      "- The struggles and scandals of Hunter Biden\n",
      "\n",
      "Blinken says fate of ceasefire plan down to Hamas\n",
      "The US diplomat says Israel's prime minister \"reaffirmed his commitment\" to a Gaza ceasefire plan.\n",
      "2 hrs ago | Middle East\n",
      "\n",
      "Ukraine 'hits missile launch sites in Russia'\n",
      "The mayor of the city of Kharkiv says the situation there is \"calmer\" as Russia has been shelling less.\n",
      "1 hr ago | Europe\n",
      "\n",
      "Four US college instructors stabbed in public park in China\n",
      "The instructors were on a daytime visit to a public park when they were attacked, Cornell College says.\n",
      "4 hrs ago | Asia\n",
      "\n",
      "Animal-rights protesters attack portrait of King\n",
      "Animal-rights protesters attack a portrait of King Charles III, in a London gallery.\n",
      "2 hrs ago | UK\n",
      "\n",
      "Warning shots from South as NK soldiers cross border\n",
      "The incident at the DMZ comes at a time of heightened tensions between the two Koreas.\n",
      "11 hrs ago | Asia\n",
      "\n",
      "## Image Features\n",
      "\n",
      "From the screenshot of the news website you provided, here is the analysis based on the images displayed alongside the corresponding news headings and text:\n",
      "\n",
      "1. **Image Associated with Hunter Biden's Conviction**:\n",
      "   - **Description**: The image depicts Hunter Biden escorted by, possibly, security personnel or aides. He seems to be dressed in a formal dark suit and appears to be descending stairs or possibly exiting a vehicle. He carries an air of seriousness, likely reflective of the gravity of his legal situation.\n",
      "   - **Relevance**: This image effectively captures the serious, personal, and public nature of the judicial proceedings against the President's son, making the situation more relatable to the audience. It directly ties to the news confirming Hunter Biden’s guilty verdict in a gun trial related to lying about drug use.\n",
      "\n",
      "2. **Image Accompanying the Article on Biden's Supporters**:\n",
      "   - **Description**: The accompanying image shows a group of enthusiastic supporters holding signs, with one prominently reading \"Say Yes to Biden,\" suggesting a political rally or campaign event. The participants display expressions of support and enthusiasm.\n",
      "   - **Relevance**: This image provides a visual contrast to the first, highlighting the ongoing support for the Biden family or campaign despite the legal issues faced by Hunter Biden. It serves to illustrate the political backdrop and public opinion dynamic mentioned in the news headlines.\n",
      "\n",
      "These images serve different purposes:\n",
      "- The first image personalizes the news story, putting a face to the name in a high-stakes legal controversy. It underlines the personal and public challenges faced by the Biden family due to the conviction.\n",
      "- The second image contextualizes the broader political support for the Biden family, suggesting that despite personal legal woes, there is a segment of the populace fervently supporting them.\n",
      "\n",
      "The clear connection between the images and the corresponding textual content on the news site helps readers visualize and better understand the unfolding events, enhancing the impact of the news storytelling.\n",
      "\n",
      "## Interesting Points\n",
      "\n",
      "The BBC news website, as demonstrated through the detailed examination of its content, offers a dynamic and visually engaging approach to news presentation. Here’s a deeper analysis of how it distinguishes itself:\n",
      "\n",
      "1. **Comprehensive and Geographically Diverse News Coverage**:\n",
      "   - The content spans a wide range of geographical locations including the US, Middle East, Europe, Asia, and the UK. Each news piece targets a major recent event, reflecting the website’s commitment to global news coverage. This expansive geographic focus ensures that readers have access to a broad spectrum of significant, impactful news.\n",
      "\n",
      "2. **Varied Content Themes**: \n",
      "   - The news themes are diverse, covering political, social, and cultural issues. From the legal troubles of a high-profile political figure’s son in the US to a ceasefire plan in the Middle East and violent incidents in Asia, the website covers a wide array of topics. This variety meets different readers' interests and keeps the content engaging.\n",
      "\n",
      "3. **Immediate Relevance**:\n",
      "   - The website's content is timely, as indicated by timestamps such as “1 hr ago” and “2 hrs ago.” This reflects the website’s commitment to providing the latest news, which is crucial for maintaining reader engagement and trust in a digital age where current information is highly valued.\n",
      "\n",
      "4. **Stylistic and Engaging Visual Design**:\n",
      "   - The use of compelling images alongside the news articles plays a critical role in storytelling. For instance, the image of Hunter Biden descending steps with a serious demeanor visually reinforces the gravity of the news about his conviction. \n",
      "   - Meanwhile, the image of supporters holding \"Say Yes to Biden\" signs juxtaposed with Hunter Biden's legal news offers a visual narrative of continued support amidst political strife, underscoring the complexity and depth of public and personal life in politics.\n",
      "\n",
      "5. **Interactive and Multimedia Features**:\n",
      "   - The use of tags such as \"OLIVE\" beside the breaking story of Hunter Biden indicates an interactive or breaking news feature that likely offers real-time updates and extensive coverage. This kind of multimedia integration enhances user interaction and engagement.\n",
      "\n",
      "In summary, the BBC news website sets itself apart through a combination of up-to-date, visually engaging, and comprehensively covered news items that cater to a global audience with varied interests. The effective use of images not only contextualizes the stories but also adds a layer of emotional and visual impact, making the news relatable and striking.\n"
     ]
    }
   ],
   "source": [
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This screenshot from a news website contains several images corresponding to different news stories. Let's examine each image and extract relevant details:\n",
      "\n",
      "1. **Image associated with the Michael Mosley news story:**\n",
      "   - The image depicts a middle-aged couple, smiling warmly at each other in a sunny, natural outdoor setting. This photo likely portrays Dr. Michael Mosley with his wife, representing a human interest angle to the story about Dr. Mosley's disappearance on the Greek island of Symi. The caption, \"'We will not lose hope,' says Michael Mosley's wife,\" implies a context of hope and determination amidst difficult circumstances.\n",
      "\n",
      "2. **Image linked to the news about the hostages freed in Gaza:**\n",
      "   - This image features a group of soldiers with one individual in civilian clothing at the center, being lifted or celebrated, possibly right after a rescue scenario. The setting appears to be a rugged outdoor area, suggestive of a conflict or military zone, which aligns with the news story about hostages being freed in Gaza. The inclusion of armed personnel and a jubilant expression on the civilian's face highlights the relief and successful outcome of a dangerous operation.\n",
      "\n",
      "3. **Image for the Nova festival hostages news:**\n",
      "   - This image depicts a motorboat on clear water under bright skies, possibly implying the geographic setting related to Michael Mosley’s disappearance near the Greek island of Symi. The serene environment contrasts starkly with the concerning news of his disappearance during what might have been a routine outing or travel.\n",
      "\n",
      "These images serve as visual supplements to the written content, providing readers with a clearer, more immediate understanding of the stories. They help bridge the emotional and contextual gaps that pure text might leave, allowing readers to engage more deeply with the news events. Each image is carefully selected to evoke specific sentiments and to provide visual context to the news headlines and summaries.\n"
     ]
    }
   ],
   "source": [
    "print(response[\"features\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from live_bench import LiveBench"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = LiveBench(force_clear=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2024-06'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "could not detect version_main.therefore, we are assuming it is chrome 108 or higher\n",
      "Capturing websites:   0%|          | 0/1 [00:00<?, ?it/s]Failed to generate response: 429 Resource has been exhausted (e.g. check quota).\n",
      "Failed to generate response: 429 Resource has been exhausted (e.g. check quota).\n",
      "Capturing websites: 100%|██████████| 1/1 [03:53<00:00, 233.51s/it]\n"
     ]
    }
   ],
   "source": [
    "dataset.capture(websites=website, driver_kwargs={\"headless\": True}, screen_shoter=\"single_screen\", shoter_kwargs={\"screen_size\": (1024, 1024)}, qa_generator=\"gpt4v\", scorer=\"gpt4v\", checker=\"gemini\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset.hf_data[0][\"images\"][0].save(\"img.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['id', 'images', 'website', 'question', 'answer', 'subtask', 'data_generator', 'checker', 'date_time', 'screen_shoter', 'screen_size', 'score', 'reason', 'scorer_name'],\n",
       "    num_rows: 8\n",
       "})"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.hf_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 9/9 [00:00<00:00, 243.32 examples/s]?, ?it/s]\n",
      "Creating parquet from Arrow format: 100%|██████████| 1/1 [00:00<00:00, 194.00ba/s]\n",
      "Uploading the dataset shards: 100%|██████████| 1/1 [00:02<00:00,  2.07s/it]\n"
     ]
    }
   ],
   "source": [
    "dataset.upload()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/data/pufanyi/anaconda3/anacondabin/envs/live_bench/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading readme: 100%|██████████| 2.40k/2.40k [00:00<00:00, 4.53MB/s]\n",
      "Downloading data: 100%|██████████| 60.6M/60.6M [00:13<00:00, 4.37MB/s]\n",
      "Downloading data: 100%|██████████| 119M/119M [00:06<00:00, 17.7MB/s] \n",
      "Generating test split: 100%|██████████| 787/787 [00:02<00:00, 284.48 examples/s]\n"
     ]
    }
   ],
   "source": [
    "data = load_dataset(\"lmms-lab/LiveBench\", \"2024-06\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datasets\n",
    "\n",
    "data = datasets.load_dataset(\"lmms-lab/LiveBench\", \"2024-07\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    test: Dataset({\n",
       "        features: ['id', 'images', 'website', 'question', 'answer', 'criteria', 'subtask', 'data_generator', 'checker', 'date_time', 'screen_shoter', 'screen_size', 'score', 'reason', 'scorer_name'],\n",
       "        num_rows: 250\n",
       "    })\n",
       "    train: Dataset({\n",
       "        features: ['id', 'images', 'website', 'question', 'answer', 'criteria', 'subtask', 'data_generator', 'checker', 'date_time', 'screen_shoter', 'screen_size', 'score', 'reason', 'scorer_name'],\n",
       "        num_rows: 250\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = datasets.DatasetDict({\"test\": data[\"train\"]})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DatasetDict({\n",
       "    test: Dataset({\n",
       "        features: ['id', 'images', 'website', 'question', 'answer', 'criteria', 'subtask', 'data_generator', 'checker', 'date_time', 'screen_shoter', 'screen_size', 'score', 'reason', 'scorer_name'],\n",
       "        num_rows: 250\n",
       "    })\n",
       "})"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Map: 100%|██████████| 250/250 [00:00<00:00, 305.90 examples/s]it/s]\n",
      "Creating parquet from Arrow format: 100%|██████████| 3/3 [00:00<00:00,  5.01ba/s]\n",
      "Uploading the dataset shards: 100%|██████████| 1/1 [00:02<00:00,  2.10s/it]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "CommitInfo(commit_url='https://huggingface.co/datasets/lmms-lab/LiveBench/commit/e7641058fcfc9d4a3039db5b995803e492ee5c4e', commit_message='Upload dataset', commit_description='', oid='e7641058fcfc9d4a3039db5b995803e492ee5c4e', pr_url=None, pr_revision=None, pr_num=None)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.push_to_hub(\"lmms-lab/LiveBench\", \"2024-07\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "live_bench",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}