Colab Notebooks/Web-Scraping-Lab.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"P1c-xykrqg8c"},"source":["<p style=\"text-align:center\">\n","    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\" target=\"_blank\">\n","    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n","    </a>\n","</p>\n"]},{"cell_type":"markdown","metadata":{"id":"0AdMDArqqg8e"},"source":["# **Hands-on Lab : Web Scraping**\n"]},{"cell_type":"markdown","metadata":{"id":"9aZhOx_Bqg8f"},"source":["Estimated time needed: **30 to 45** minutes\n"]},{"cell_type":"markdown","metadata":{"id":"MopEv3Mmqg8f"},"source":["## Objectives\n"]},{"cell_type":"markdown","metadata":{"id":"ZgFKmY0Uqg8g"},"source":["In this lab you will perform the following:\n"]},{"cell_type":"markdown","metadata":{"id":"ltgXjniVqg8g"},"source":["* Extract information from a given web site\n","* Write the scraped data into a csv file.\n"]},{"cell_type":"markdown","metadata":{"id":"nou-6a2Sqg8g"},"source":["## Extract information from the given web site\n","You will extract the data from the below web site: <br>\n"]},{"cell_type":"code","execution_count":1,"metadata":{"id":"7crnkHy3qg8h","executionInfo":{"status":"ok","timestamp":1735698039704,"user_tz":-60,"elapsed":289,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#this url contains the data you need to scrape\n","url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html\""]},{"cell_type":"markdown","metadata":{"id":"yHlccnqVqg8i"},"source":["The data you need to scrape is the **name of the programming language** and **average annual salary**.<br> It is a good idea to open the url in your web broswer and study the contents of the web page before you start to scrape.\n"]},{"cell_type":"markdown","metadata":{"id":"wdI10MuCqg8i"},"source":["Import the required libraries\n"]},{"cell_type":"code","execution_count":3,"metadata":{"id":"qYHrSGXIqg8i","executionInfo":{"status":"ok","timestamp":1735698217536,"user_tz":-60,"elapsed":299,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["# Your code here\n","import requests\n","from bs4 import BeautifulSoup"]},{"cell_type":"markdown","metadata":{"id":"Q9qTtYNwqg8j"},"source":["Download the webpage at the url\n"]},{"cell_type":"code","execution_count":4,"metadata":{"id":"DmcM5qicqg8j","executionInfo":{"status":"ok","timestamp":1735698316347,"user_tz":-60,"elapsed":300,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#your code goes here\n","data=requests.get(url).text"]},{"cell_type":"markdown","metadata":{"id":"VGsNp_XNqg8j"},"source":["Create a soup object\n"]},{"cell_type":"code","execution_count":5,"metadata":{"id":"cS0NEpi2qg8j","executionInfo":{"status":"ok","timestamp":1735698367358,"user_tz":-60,"elapsed":326,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#your code goes here\n","soup=BeautifulSoup(data,'html.parser')"]},{"cell_type":"markdown","metadata":{"id":"bYpFVz8Sqg8j"},"source":["Scrape the `Language name` and `annual average salary`.\n"]},{"cell_type":"code","execution_count":7,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"GvA7aenPqg8k","executionInfo":{"status":"ok","timestamp":1735698907874,"user_tz":-60,"elapsed":281,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"36df5d25-acb1-4fda-f3a2-b6191d4ec57b"},"outputs":[{"output_type":"stream","name":"stdout","text":["Language--->Average Annual Salary\n","Python--->$114,383\n","Java--->$101,013\n","R--->$92,037\n","Javascript--->$110,981\n","Swift--->$130,801\n","C++--->$113,865\n","C#--->$88,726\n","PHP--->$84,727\n","SQL--->$84,793\n","Go--->$94,082\n"]}],"source":["#your code goes here\n","table=soup.find('table')\n","for row in table.find_all('tr'): # in html table row is represented by the tag <tr>\n","    # Get all columns in each row.\n","    cols = row.find_all('td') # in html a column is represented by the tag <td>\n","    color_name = cols[1].getText() # store the value in column 3 as color_name\n","    color_code = cols[3].getText() # store the value in column 4 as color_code\n","    print(\"{}--->{}\".format(color_name,color_code))\n"]},{"cell_type":"markdown","metadata":{"id":"zlmZsGajqg8k"},"source":["Save the scrapped data into a file named *popular-languages.csv*\n"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"ThuQO7nfqg8k","executionInfo":{"status":"ok","timestamp":1735699302270,"user_tz":-60,"elapsed":913,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"6c1cbcc9-86a3-4573-db7e-ca501aa38a33"},"outputs":[{"output_type":"stream","name":"stdout","text":["Data saved to popular-languages.csv\n"]}],"source":["# your code goes here\n","#this url contains the data you need to scrape\n","url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/datasets/Programming_Languages.html\"\n","\n","# Your code here\n","import requests\n","from bs4 import BeautifulSoup\n","import csv\n","\n","# Download the webpage at the url\n","data  = requests.get(url).text\n","\n","# Create a soup object\n","soup = BeautifulSoup(data,\"html5lib\")\n","\n","# Scrape the `Language name` and `annual average salary`.\n","table = soup.find('table')\n","popular_languages = []\n","\n","for row in table.find_all('tr')[1:]:  # Skip the header row\n","    cols = row.find_all('td')\n","    language = cols[1].getText()\n","    salary = cols[3].getText()\n","    popular_languages.append([language, salary])\n","\n","# Save the scrapped data into a file named *popular-languages.csv*\n","with open('popular-languages.csv', 'w', newline='', encoding='utf-8') as csvfile:\n","    writer = csv.writer(csvfile)\n","    writer.writerow(['Language', 'Average Annual Salary'])  # Write header row\n","    writer.writerows(popular_languages)\n","\n","print(\"Data saved to popular-languages.csv\")"]},{"cell_type":"markdown","metadata":{"id":"jqySvIHsqg8k"},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{"id":"8iYQW9Qvqg8k"},"source":["Ramesh Sannareddy\n"]},{"cell_type":"markdown","metadata":{"id":"EkjycvrKqg8l"},"source":["### Other Contributors\n"]},{"cell_type":"markdown","metadata":{"id":"ORQ2xdYWqg8l"},"source":["Rav Ahuja\n"]},{"cell_type":"markdown","metadata":{"id":"MT0LZPnQqg8l"},"source":["## Change Log\n"]},{"cell_type":"markdown","metadata":{"id":"xeyWSHQrqg8l"},"source":["|  Date (YYYY-MM-DD) |  Version | Changed By  |  Change Description |\n","|---|---|---|---|\n","| 2020-10-17  | 0.1  | Ramesh Sannareddy  |  Created initial version of the lab |\n"]},{"cell_type":"markdown","metadata":{"id":"E0zcMzAWqg8l"},"source":[" Copyright &copy; 2020 IBM Corporation. This notebook and its source code are released under the terms of the [MIT License](https://cognitiveclass.ai/mit-license/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01).\n"]}],"metadata":{"kernelspec":{"display_name":"Python","language":"python","name":"conda-env-python-py"},"language_info":{"name":""},"colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}