Colab Notebooks/Collecting_job_data_using_APIs-Lab.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"id":"57gs3igfjqbX"},"source":["<p style=\"text-align:center\">\n","    <a href=\"https://skills.network\" target=\"_blank\">\n","    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\">\n","    </a>\n","</p>\n"]},{"cell_type":"markdown","metadata":{"id":"qAgAIG49jqbZ"},"source":["# **Collecting Job Data Using APIs**\n"]},{"cell_type":"markdown","metadata":{"id":"hbJWUZhGjqba"},"source":["Estimated time needed: **30** minutes\n"]},{"cell_type":"markdown","metadata":{"id":"BvJm9l36jqba"},"source":["## Objectives\n"]},{"cell_type":"markdown","metadata":{"id":"b89a9kfPjqba"},"source":["After completing this lab, you will be able to:\n"]},{"cell_type":"markdown","metadata":{"id":"JwVU4mmXjqbb"},"source":["*   Collect job data using Jobs API\n","*   Store the collected data into an excel spreadsheet.\n"]},{"cell_type":"markdown","metadata":{"id":"98dJhLTnjqbb"},"source":["><strong>Note: Before starting with the assignment make sure to read all the instructions and then move ahead with the coding part.</strong>\n"]},{"cell_type":"markdown","metadata":{"id":"KjvZsQJjjqbb"},"source":["#### Instructions\n"]},{"cell_type":"markdown","metadata":{"id":"7DILwUINjqbb"},"source":["To run the actual lab, firstly you need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link. The file contains flask code which is required to run the Jobs API data.\n","\n","Now, to run the code in the file that opens up follow the below steps.\n","\n","Step1: Download the file.\n","\n","Step2: Upload the file into your current Jupyter environment using the upload button in your Jupyter interface. Ensure that the file is in the same folder as your working .ipynb file.\n","\n","Step 2: If working in a local Jupyter environment, use the \"Upload\" button in your Jupyter interface to upload the Jobs_API notebook into the same folder as your current .ipynb file.\n","\n","<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Upload.PNG\">\n","\n","Step3:  Open the Jobs_API notebook, and run all the cells to start the Flask application. Once the server is running, you can access the API from the URL provided in the notebook.\n","\n","If you want to learn more about flask, which is optional, you can click on this link [here](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/FLASK_API.md.html).\n","\n","Once you run the flask code, you can start with your assignment.\n"]},{"cell_type":"markdown","metadata":{"id":"4hD0PdbZjqbc"},"source":["## Dataset Used in this Assignment\n","\n","The dataset used in this lab comes from the following source: https://www.kaggle.com/promptcloud/jobs-on-naukricom under the under a **Public Domain license**.\n","\n","> Note: We are using a modified subset of that dataset for the lab, so to follow the lab instructions successfully please use the dataset provided with the lab, rather than the dataset from the original source.\n","\n","The original dataset is a csv. We have converted the csv to json as per the requirement of the lab.\n"]},{"cell_type":"markdown","metadata":{"id":"nTEsKPudjqbc"},"source":["## Warm-Up Exercise\n"]},{"cell_type":"markdown","metadata":{"id":"FDNZVXOQjqbc"},"source":["Before you attempt the actual lab, here is a fully solved warmup exercise that will help you to learn how to access an API.\n"]},{"cell_type":"markdown","metadata":{"id":"R2o0Ynw6jqbc"},"source":["Using an API, let us find out who currently are on the International Space Station (ISS).<br> The API at [http://api.open-notify.org/astros.json](http://api.open-notify.org/astros.json?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ) gives us the information of astronauts currently on ISS in json format.<br>\n","You can read more about this API at [http://open-notify.org/Open-Notify-API/People-In-Space/](http://open-notify.org/Open-Notify-API/People-In-Space?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2021-01-01&cm_mmc=Email_Newsletter-_-Developer_Ed%2BTech-_-WW_WW-_-SkillsNetwork-Courses-IBM-DA0321EN-SkillsNetwork-21426264&cm_mmca1=000026UJ&cm_mmca2=10006555&cm_mmca3=M12345678&cvosrc=email.Newsletter.M12345678&cvo_campaign=000026UJ)\n"]},{"cell_type":"code","execution_count":1,"metadata":{"tags":[],"id":"HAkhyNM4jqbc","executionInfo":{"status":"ok","timestamp":1735696253119,"user_tz":-60,"elapsed":808,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["import requests # you need this module to make an API call\n","import pandas as pd"]},{"cell_type":"code","execution_count":2,"metadata":{"tags":[],"id":"hVrEijjVjqbd","executionInfo":{"status":"ok","timestamp":1735696253498,"user_tz":-60,"elapsed":10,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["api_url = \"http://api.open-notify.org/astros.json\" # this url gives use the astronaut data"]},{"cell_type":"code","execution_count":3,"metadata":{"tags":[],"id":"7D2FNvj7jqbd","executionInfo":{"status":"ok","timestamp":1735696253498,"user_tz":-60,"elapsed":9,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["response = requests.get(api_url) # Call the API using the get method and store the\n","                                # output of the API call in a variable called response."]},{"cell_type":"code","execution_count":4,"metadata":{"tags":[],"id":"4vec3IRPjqbd","executionInfo":{"status":"ok","timestamp":1735696253498,"user_tz":-60,"elapsed":8,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["if response.ok:             # if all is well() no errors, no network timeouts)\n","    data = response.json()  # store the result in json format in a variable called data\n","                            # the variable data is of type dictionary."]},{"cell_type":"code","execution_count":5,"metadata":{"tags":[],"colab":{"base_uri":"https://localhost:8080/"},"id":"Hr5jRnoejqbd","executionInfo":{"status":"ok","timestamp":1735696253499,"user_tz":-60,"elapsed":8,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"806f7e2d-64ca-45ab-e26d-4e2528d99ae0"},"outputs":[{"output_type":"stream","name":"stdout","text":["{'people': [{'craft': 'ISS', 'name': 'Oleg Kononenko'}, {'craft': 'ISS', 'name': 'Nikolai Chub'}, {'craft': 'ISS', 'name': 'Tracy Caldwell Dyson'}, {'craft': 'ISS', 'name': 'Matthew Dominick'}, {'craft': 'ISS', 'name': 'Michael Barratt'}, {'craft': 'ISS', 'name': 'Jeanette Epps'}, {'craft': 'ISS', 'name': 'Alexander Grebenkin'}, {'craft': 'ISS', 'name': 'Butch Wilmore'}, {'craft': 'ISS', 'name': 'Sunita Williams'}, {'craft': 'Tiangong', 'name': 'Li Guangsu'}, {'craft': 'Tiangong', 'name': 'Li Cong'}, {'craft': 'Tiangong', 'name': 'Ye Guangfu'}], 'number': 12, 'message': 'success'}\n"]}],"source":["print(data)   # print the data just to check the output or for debugging"]},{"cell_type":"markdown","metadata":{"id":"RXv2YvNZjqbe"},"source":["Print the number of astronauts currently on ISS.\n"]},{"cell_type":"code","execution_count":6,"metadata":{"tags":[],"colab":{"base_uri":"https://localhost:8080/"},"id":"Foxw1biqjqbe","executionInfo":{"status":"ok","timestamp":1735696259888,"user_tz":-60,"elapsed":342,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"9e92d5a1-dcb3-4faa-e838-42b4d96fe2f8"},"outputs":[{"output_type":"stream","name":"stdout","text":["12\n"]}],"source":["print(data.get('number'))"]},{"cell_type":"markdown","metadata":{"id":"Urm05yc5jqbe"},"source":["Print the names of the astronauts currently on ISS.\n"]},{"cell_type":"code","execution_count":7,"metadata":{"tags":[],"colab":{"base_uri":"https://localhost:8080/"},"id":"YaBDws1zjqbe","executionInfo":{"status":"ok","timestamp":1735696265315,"user_tz":-60,"elapsed":276,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"2f7b5b3a-d223-4db0-8b33-04ec7022282b"},"outputs":[{"output_type":"stream","name":"stdout","text":["There are 12 astronauts on ISS\n","And their names are :\n","Oleg Kononenko\n","Nikolai Chub\n","Tracy Caldwell Dyson\n","Matthew Dominick\n","Michael Barratt\n","Jeanette Epps\n","Alexander Grebenkin\n","Butch Wilmore\n","Sunita Williams\n","Li Guangsu\n","Li Cong\n","Ye Guangfu\n"]}],"source":["astronauts = data.get('people')\n","print(\"There are {} astronauts on ISS\".format(len(astronauts)))\n","print(\"And their names are :\")\n","for astronaut in astronauts:\n","    print(astronaut.get('name'))"]},{"cell_type":"markdown","metadata":{"id":"-kCI-RaZjqbf"},"source":["Hope the warmup was helpful. Good luck with your next lab!\n"]},{"cell_type":"markdown","metadata":{"id":"6xNDZeYmjqbf"},"source":["## Lab: Collect Jobs Data using Jobs API\n"]},{"cell_type":"markdown","metadata":{"id":"0cRso4G7jqbf"},"source":["### Objective: Determine the number of jobs currently open for various technologies  and for various locations\n"]},{"cell_type":"markdown","metadata":{"id":"Q4Rk46d8jqbf"},"source":["Collect the number of job postings for the following locations using the API:\n","\n","* Los Angeles\n","* New York\n","* San Francisco\n","* Washington DC\n","* Seattle\n","* Austin\n","* Detroit\n"]},{"cell_type":"code","execution_count":8,"metadata":{"tags":[],"id":"j6R6QzCfjqbf","executionInfo":{"status":"ok","timestamp":1735696286613,"user_tz":-60,"elapsed":308,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#Import required libraries\n","import pandas as pd\n","import json\n"]},{"cell_type":"markdown","metadata":{"id":"K4JA2OlZjqbf"},"source":["https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json#### Write a function to get the number of jobs for the Python technology.<br>\n","> Note: While using the lab you need to pass the **payload** information for the **params** attribute in the form of **key** **value** pairs.\n","  Refer the ungraded **rest api lab** in the course **Python for Data Science, AI & Development**  <a href=\"https://www.coursera.org/learn/python-for-applied-data-science-ai/ungradedLti/P6sW8/hands-on-lab-access-rest-apis-request-http?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork928-2022-01-01\">link</a>\n","  \n"," ##### The keys in the json are\n"," * Job Title\n","\n"," * Job Experience Required\n","\n"," * Key Skills\n","\n"," * Role Category\n","\n"," * Location\n","\n"," * Functional Area\n","\n"," * Industry\n","\n"," * Role\n","\n","You can also view  the json file contents  from the following <a href = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json\">json</a> URL.\n","\n"]},{"cell_type":"code","execution_count":11,"metadata":{"tags":[],"id":"7XKXJ8ayjqbf","executionInfo":{"status":"ok","timestamp":1735696974583,"user_tz":-60,"elapsed":251,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["api_url=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json\"\n","\n","def get_number_of_jobs_T(technology):\n","    response = requests.get(api_url)\n","    payload={\"Key Skills\":technology}\n","    response=requests.get(api_url,params=payload)\n","    number_of_jobs=len(response.json())\n","\n","\n","    return technology,number_of_jobs\n"]},{"cell_type":"markdown","metadata":{"id":"EWqHeeHejqbf"},"source":["Calling the function for Python and checking if it works.\n"]},{"cell_type":"code","execution_count":12,"metadata":{"tags":[],"colab":{"base_uri":"https://localhost:8080/"},"id":"PAgqyZmfjqbg","executionInfo":{"status":"ok","timestamp":1735696983257,"user_tz":-60,"elapsed":3390,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"d05ca82e-1b43-4338-9f48-f8e2fc3e5459"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["('Python', 27005)"]},"metadata":{},"execution_count":12}],"source":["get_number_of_jobs_T(\"Python\")"]},{"cell_type":"markdown","metadata":{"id":"1FQr8ue9jqbg"},"source":["#### Write a function to find number of jobs in US for a location of your choice\n"]},{"cell_type":"code","execution_count":13,"metadata":{"tags":[],"id":"DoB3oYcJjqbg","executionInfo":{"status":"ok","timestamp":1735697115974,"user_tz":-60,"elapsed":272,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":[" def get_number_of_jobs_L(location):\n","    #your coe goes here\n","    response = requests.get(api_url)\n","    payload={\"Location\":location}\n","    response=requests.get(api_url,params=payload)\n","    number_of_jobs=len(response.json())\n","    return location,number_of_jobs"]},{"cell_type":"markdown","metadata":{"id":"xse9pWMMjqbg"},"source":["Call the function for Los Angeles and check if it is working.\n"]},{"cell_type":"code","execution_count":14,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"QMtXTSdIjqbg","executionInfo":{"status":"ok","timestamp":1735697133606,"user_tz":-60,"elapsed":1394,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"da2a0955-d313-4a8a-acdb-2a62c5f47d25"},"outputs":[{"output_type":"execute_result","data":{"text/plain":["('Los Angeles', 27005)"]},"metadata":{},"execution_count":14}],"source":["#your code goes here\n","get_number_of_jobs_L(\"Los Angeles\")\n"]},{"cell_type":"markdown","metadata":{"id":"uBLLP-K2jqbg"},"source":["### Store the results in an excel file\n"]},{"cell_type":"markdown","metadata":{"id":"BIhPu9dMjqbh"},"source":["Call the API for all the given technologies above and write the results in an excel spreadsheet.\n"]},{"cell_type":"markdown","metadata":{"id":"vA1YwY97jqbh"},"source":["If you do not know how create excel file using python, double click here for **hints**.\n","\n","<!--\n","\n","from openpyxl import Workbook        # import Workbook class from module openpyxl\n","wb=Workbook()                        # create a workbook object\n","ws=wb.active                         # use the active worksheet\n","ws.append(['Country','Continent'])   # add a row with two columns 'Country' and 'Continent'\n","ws.append(['Eygpt','Africa'])        # add a row with two columns 'Egypt' and 'Africa'\n","ws.append(['India','Asia'])          # add another row\n","ws.append(['France','Europe'])       # add another row\n","wb.save(\"countries.xlsx\")            # save the workbook into a file called countries.xlsx\n","\n","\n","-->\n"]},{"cell_type":"markdown","metadata":{"id":"Gvzt_Mndjqbh"},"source":["Create a python list of all technologies for which you need to find the number of jobs postings.\n"]},{"cell_type":"code","execution_count":15,"metadata":{"id":"cmDWDoowjqbi","executionInfo":{"status":"ok","timestamp":1735697241011,"user_tz":-60,"elapsed":294,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#your code goes here\n","technologies=['C','C#','C++','Java','JavaScript','Python','Scala','Oracle','SQL Server','MySQL Server','PostgreSQL','MongoDB']"]},{"cell_type":"markdown","metadata":{"id":"caQrgjLtjqbi"},"source":["Import libraries required to create excel spreadsheet\n"]},{"cell_type":"code","execution_count":16,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"id":"eNH176bOjqbi","executionInfo":{"status":"ok","timestamp":1735697270815,"user_tz":-60,"elapsed":3679,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}},"outputId":"6ed2b2a9-6d5d-46c3-93f6-5e27c719e9b3"},"outputs":[{"output_type":"stream","name":"stdout","text":["Requirement already satisfied: openpyxl in /usr/local/lib/python3.10/dist-packages (3.1.5)\n","Requirement already satisfied: et-xmlfile in /usr/local/lib/python3.10/dist-packages (from openpyxl) (2.0.0)\n"]}],"source":["# your code goes here\n","!pip install openpyxl\n","from openpyxl import Workbook"]},{"cell_type":"markdown","metadata":{"id":"CO-CP_d4jqbi"},"source":["Create a workbook and select the active worksheet\n"]},{"cell_type":"code","execution_count":18,"metadata":{"id":"IPFUjMBpjqbi","executionInfo":{"status":"ok","timestamp":1735697314133,"user_tz":-60,"elapsed":291,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["# your code goes here\n","wb=Workbook()\n","ws=wb.active\n"]},{"cell_type":"markdown","metadata":{"id":"9UP20P7Qjqbi"},"source":["Find the number of jobs postings for each of the technology in the above list.\n","Write the technology name and the number of jobs postings into the excel spreadsheet.\n"]},{"cell_type":"code","execution_count":20,"metadata":{"id":"QSvFJ47Gjqbj","executionInfo":{"status":"ok","timestamp":1735697366247,"user_tz":-60,"elapsed":12583,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#your code goes here\n","ws.append(['Technology','Number of Jobs'])\n","for technology in technologies:\n","    ws.append(get_number_of_jobs_T(technology))\n"]},{"cell_type":"markdown","metadata":{"id":"9xpKeha8jqbj"},"source":["Save into an excel spreadsheet named **job-postings.xlsx**.\n"]},{"cell_type":"code","execution_count":22,"metadata":{"id":"rkMcKvtQjqbj","executionInfo":{"status":"ok","timestamp":1735697396821,"user_tz":-60,"elapsed":268,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["#your code goes here\n","wb.save(\"job-postings.xlsx\")"]},{"cell_type":"markdown","metadata":{"id":"Q2PfvN1Qjqbj"},"source":["#### In the similar way, you can try for below given technologies and results  can be stored in an excel sheet.\n"]},{"cell_type":"markdown","metadata":{"id":"rb-qJb18jqbj"},"source":["Collect the number of job postings for the following languages using the API:\n","\n","*   C\n","*   C#\n","*   C++\n","*   Java\n","*   JavaScript\n","*   Python\n","*   Scala\n","*   Oracle\n","*   SQL Server\n","*   MySQL Server\n","*   PostgreSQL\n","*   MongoDB\n"]},{"cell_type":"code","execution_count":23,"metadata":{"id":"_gQrfa8sjqbj","executionInfo":{"status":"ok","timestamp":1735697477075,"user_tz":-60,"elapsed":13474,"user":{"displayName":"Tongue Kevin","userId":"13768078595560270101"}}},"outputs":[],"source":["# your code goes here\n","technologies=['C','C#','C++','Java','JavaScript','Python','Scala','Oracle','SQL Server','MySQL Server','PostgreSQL','MongoDB']\n","wb=Workbook()\n","ws=wb.active\n","ws.append(['Technology','Number of Jobs'])\n","for technology in technologies:\n","    ws.append(get_number_of_jobs_T(technology))\n","wb.save(\"job-postings.xlsx\")\n","\n"]},{"cell_type":"markdown","metadata":{"id":"lO_XXzKKjqbj"},"source":["## Authors\n"]},{"cell_type":"markdown","metadata":{"id":"vDhGvJxsjqbk"},"source":["Ayushi Jain\n"]},{"cell_type":"markdown","metadata":{"id":"APDAJJjhjqbk"},"source":["### Other Contributors\n"]},{"cell_type":"markdown","metadata":{"id":"LueYIIKojqbk"},"source":["Rav Ahuja\n","\n","Lakshmi Holla\n","\n","Malika\n"]},{"cell_type":"markdown","metadata":{"id":"BIyMznXHjqbk"},"source":["Copyright © IBM Corporation.\n"]},{"cell_type":"markdown","metadata":{"id":"Ssg6NOhBjqbk"},"source":["<!--## Change Log\n"]},{"cell_type":"markdown","metadata":{"id":"39-Li1i1jqbk"},"source":["<!--| Date (YYYY-MM-DD) | Version | Changed By        | Change Description                 |\n","| ----------------- | ------- | ----------------- | ---------------------------------- |\n","| 2022-01-19        | 0.3     | Lakshmi Holla        | Added changes in the markdown      |\n","| 2021-06-25        | 0.2     | Malika            | Updated GitHub job json link       |\n","| 2020-10-17        | 0.1     | Ramesh Sannareddy | Created initial version of the lab |--!>\n"]}],"metadata":{"kernelspec":{"display_name":"Python","language":"python","name":"conda-env-python-py"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.12"},"prev_pub_hash":"61a35a07ad98492b710274ae0e52a0fdce2221e88e366133dd4a20370680fa8f","colab":{"provenance":[]}},"nbformat":4,"nbformat_minor":0}