Pipeline de Dados

.gitignore

@@ -0,0 +1,3 @@
+# EddyGiusepe
+venv_pipeData/
+

1_Data_Pipelines/exemplo_Data_Pipeline.ipynb

@@ -0,0 +1,565 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<h1 align=\"center\"><font color=\"gree\">Data Pipelines</font></h1>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font color=\"yellow\">Data Scientist.: Dr. Eddy Giusepe Chirinos Isidro</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Aqui vamos a executar um `Pipeline` que carregue os dados do [PokéAPI](https://pokeapi.co/) em um Bando de Dados [DuckDB](https://duckdb.org/) usando a Biblioteca [dlt](https://dlthub.com/docs/intro)."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# <font color=\"gree\">Contextualizando</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">O que é `dlt`</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`dlt` é uma biblioteca de código aberto que você pode adicionar aos seus scripts Python para carregar dados de várias fontes de dados, muitas vezes confusas, em conjuntos de dados ativos e bem estruturados. Para começar, instale-o com:\n",
+    "\n",
+    "```\n",
+    "pip install dlt\n",
+    "```\n",
+    "\n",
+    "\n",
+    "Ao contrário de outras soluções, com `dlt` não há necessidade de usar `back-ends` ou `contêineres`. Basta importar `dlt` um arquivo Python ou uma célula do Jupyter Notebook e criar um pipeline para carregar dados em qualquer um dos destinos suportados. Você pode carregar dados de qualquer fonte que produza estruturas de dados `Python`, incluindo `APIs`, `arquivos`, `bancos de dados` e muito mais. `dlt` também suporta a construção de um [destino personalizado](https://dlthub.com/docs/dlt-ecosystem/destinations/destination), que você pode usar como `ETL` reverso.\n",
+    "\n",
+    "A biblioteca criará ou atualizará tabelas, inferirá tipos de dados e manipulará dados aninhados automaticamente."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# <font color=\"pink\">Exemplo 1:</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">Instalação</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture\n",
+    "%pip install \"dlt[duckdb]\" # Instale o dlt com todas as dependências necessárias do DuckDB"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!dlt --version"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">Importe `dlt` e inicialize o pipeline</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import dlt\n",
+    "\n",
+    "pipeline = dlt.pipeline(pipeline_name=\"Eddy_pipeline_pokemon\",\n",
+    "                        destination=\"duckdb\",\n",
+    "                        dataset_name=\"Eddy_dataset_pokemon\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "pipeline"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">Obtenha dados da fonte</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font color=\"orange\">Carregamos a lista de dados de pokémons usando url.</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from dlt.sources.helpers import requests\n",
+    "\n",
+    "POKEMON_URL = \"https://pokeapi.co/api/v2/pokemon/\"\n",
+    "\n",
+    "data = requests.get(POKEMON_URL).json()[\"results\"]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "data"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">Execute o pipeline</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "%%capture\n",
+    "\n",
+    "# Normalize e carregue os dados no banco de dados duckdb criado localmente 'pokemon_pipeline.duckdb'\n",
+    "pipeline.run(data, table_name='Eddy_Table_pokemon')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## <font color=\"red\">Query nos dados carregados 🦆</font>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<font color=\"orange\">Para acessar os dados carregados, conecte-se ao Banco de Dados `DuckDB` usando o conector Python `DuckDB`.</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import duckdb\n",
+    "#from google.colab import data_table\n",
+    "#data_table.enable_dataframe_formatter()\n",
+    "\n",
+    "# a database 'chess_pipeline.duckdb' was created in working directory so just connect to it\n",
+    "conn = duckdb.connect(f\"{pipeline.pipeline_name}.duckdb\")\n",
+    "\n",
+    "# Listar todas as tabelas\n",
+    "display(conn.sql(\"DESCRIBE\"))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Isso nos permite consultar dados sem adicionar prefixo de esquema aos nomes de tabelas:\n",
+    "conn.sql(f\"SET search_path = '{pipeline.dataset_name}'\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# listar todas as tabelas\n",
+    "display(conn.sql(\"DESCRIBE\"))\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "stats_table = conn.sql(\"SELECT * FROM Eddy_Table_pokemon\").df()\n",
+    "\n",
+    "display(stats_table)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# <font color=\"pink\">Exemplo 2:</font>"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import dlt\n",
+    "from dlt.sources.helpers import requests\n",
+    "\n",
+    "# Crie um pipeline dlt que será carregado.\n",
+    "# Dados do jogador de xadrez para o destino DuckDB\n",
+    "pipeline = dlt.pipeline(pipeline_name=\"chess_pipeline\",\n",
+    "                        destination=\"duckdb\",\n",
+    "                        dataset_name=\"player_data\"\n",
+    "                       )\n",
+    "\n",
+    "\n",
+    "# Obtenha alguns dados de jogadores da API do Chess.com:\n",
+    "data = []\n",
+    "for player in [\"magnuscarlsen\", \"rpragchess\"]:\n",
+    "    response = requests.get(f\"https://api.chess.com/pub/player/{player}\")\n",
+    "    response.raise_for_status()\n",
+    "    data.append(response.json())\n",
+    "\n",
+    "# Extraia, normalize e carregue os dados:\n",
+    "load_info = pipeline.run(data, table_name=\"player\")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "Pipeline chess_pipeline load step completed in 0.19 seconds\n",
+      "1 load package(s) were loaded to destination duckdb and into dataset player_data\n",
+      "The duckdb destination used duckdb:////home/eddygiusepe/1_Eddy_Giusepe/6_REPO_HuggingFace/Data_Pipelines/1_Data_Pipelines/chess_pipeline.duckdb location to store data\n",
+      "Load package 1712029347.285123 is LOADED and contains no failed jobs\n"
+     ]
+    }
+   ],
+   "source": [
+    "print(load_info)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "┌────────────────┬─────────────┬─────────────────────┬──────────────────────┬──────────────────────────────┬───────────┐\n",
+       "│    database    │   schema    │        name         │     column_names     │         column_types         │ temporary │\n",
+       "│    varchar     │   varchar   │       varchar       │      varchar[]       │          varchar[]           │  boolean  │\n",
+       "├────────────────┼─────────────┼─────────────────────┼──────────────────────┼──────────────────────────────┼───────────┤\n",
+       "│ chess_pipeline │ player_data │ _dlt_loads          │ [load_id, schema_n…  │ [VARCHAR, VARCHAR, BIGINT,…  │ false     │\n",
+       "│ chess_pipeline │ player_data │ _dlt_pipeline_state │ [version, engine_v…  │ [BIGINT, BIGINT, VARCHAR, …  │ false     │\n",
+       "│ chess_pipeline │ player_data │ _dlt_version        │ [version, engine_v…  │ [BIGINT, BIGINT, TIMESTAMP…  │ false     │\n",
+       "│ chess_pipeline │ player_data │ player              │ [avatar, player_id…  │ [VARCHAR, BIGINT, VARCHAR,…  │ false     │\n",
+       "└────────────────┴─────────────┴─────────────────────┴──────────────────────┴──────────────────────────────┴───────────┘"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "import duckdb\n",
+    "\n",
+    "# a database 'chess_pipeline.duckdb' was created in working directory so just connect to it\n",
+    "conn1 = duckdb.connect(f\"{pipeline.pipeline_name}.duckdb\")\n",
+    "\n",
+    "# Listar todas as tabelas:\n",
+    "display(conn1.sql(\"DESCRIBE\"))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Isso nos permite consultar dados sem adicionar prefixo de esquema aos nomes de tabelas:\n",
+    "conn1.sql(f\"SET search_path = '{pipeline.dataset_name}'\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>avatar</th>\n",
+       "      <th>player_id</th>\n",
+       "      <th>aid</th>\n",
+       "      <th>url</th>\n",
+       "      <th>name</th>\n",
+       "      <th>username</th>\n",
+       "      <th>title</th>\n",
+       "      <th>followers</th>\n",
+       "      <th>country</th>\n",
+       "      <th>location</th>\n",
+       "      <th>last_online</th>\n",
+       "      <th>joined</th>\n",
+       "      <th>status</th>\n",
+       "      <th>is_streamer</th>\n",
+       "      <th>verified</th>\n",
+       "      <th>league</th>\n",
+       "      <th>_dlt_load_id</th>\n",
+       "      <th>_dlt_id</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>https://images.chesscomfiles.com/uploads/v1/us...</td>\n",
+       "      <td>3889224</td>\n",
+       "      <td>https://api.chess.com/pub/player/magnuscarlsen</td>\n",
+       "      <td>https://www.chess.com/member/MagnusCarlsen</td>\n",
+       "      <td>Magnus Carlsen</td>\n",
+       "      <td>magnuscarlsen</td>\n",
+       "      <td>GM</td>\n",
+       "      <td>189570</td>\n",
+       "      <td>https://api.chess.com/pub/country/NO</td>\n",
+       "      <td>Norway</td>\n",
+       "      <td>1712007974</td>\n",
+       "      <td>1282856720</td>\n",
+       "      <td>premium</td>\n",
+       "      <td>False</td>\n",
+       "      <td>False</td>\n",
+       "      <td>Champion</td>\n",
+       "      <td>1712028961.2599978</td>\n",
+       "      <td>lcRtKfLWXvYMIA</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>https://images.chesscomfiles.com/uploads/v1/us...</td>\n",
+       "      <td>28692936</td>\n",
+       "      <td>https://api.chess.com/pub/player/rpragchess</td>\n",
+       "      <td>https://www.chess.com/member/rpragchess</td>\n",
+       "      <td>Praggnanandhaa Rameshbabu</td>\n",
+       "      <td>rpragchess</td>\n",
+       "      <td>GM</td>\n",
+       "      <td>6906</td>\n",
+       "      <td>https://api.chess.com/pub/country/IN</td>\n",
+       "      <td>CHENNAI</td>\n",
+       "      <td>1711993442</td>\n",
+       "      <td>1466301035</td>\n",
+       "      <td>premium</td>\n",
+       "      <td>False</td>\n",
+       "      <td>False</td>\n",
+       "      <td>Crystal</td>\n",
+       "      <td>1712028961.2599978</td>\n",
+       "      <td>+erswyz4f2/xMQ</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>https://images.chesscomfiles.com/uploads/v1/us...</td>\n",
+       "      <td>3889224</td>\n",
+       "      <td>https://api.chess.com/pub/player/magnuscarlsen</td>\n",
+       "      <td>https://www.chess.com/member/MagnusCarlsen</td>\n",
+       "      <td>Magnus Carlsen</td>\n",
+       "      <td>magnuscarlsen</td>\n",
+       "      <td>GM</td>\n",
+       "      <td>189571</td>\n",
+       "      <td>https://api.chess.com/pub/country/NO</td>\n",
+       "      <td>Norway</td>\n",
+       "      <td>1712007974</td>\n",
+       "      <td>1282856720</td>\n",
+       "      <td>premium</td>\n",
+       "      <td>False</td>\n",
+       "      <td>False</td>\n",
+       "      <td>Champion</td>\n",
+       "      <td>1712029347.285123</td>\n",
+       "      <td>opkc5HKMEFbeuw</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>https://images.chesscomfiles.com/uploads/v1/us...</td>\n",
+       "      <td>28692936</td>\n",
+       "      <td>https://api.chess.com/pub/player/rpragchess</td>\n",
+       "      <td>https://www.chess.com/member/rpragchess</td>\n",
+       "      <td>Praggnanandhaa Rameshbabu</td>\n",
+       "      <td>rpragchess</td>\n",
+       "      <td>GM</td>\n",
+       "      <td>6906</td>\n",
+       "      <td>https://api.chess.com/pub/country/IN</td>\n",
+       "      <td>CHENNAI</td>\n",
+       "      <td>1711993442</td>\n",
+       "      <td>1466301035</td>\n",
+       "      <td>premium</td>\n",
+       "      <td>False</td>\n",
+       "      <td>False</td>\n",
+       "      <td>Crystal</td>\n",
+       "      <td>1712029347.285123</td>\n",
+       "      <td>XvJq9hHcwg80rg</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "                                              avatar  player_id  \\\n",
+       "0  https://images.chesscomfiles.com/uploads/v1/us...    3889224   \n",
+       "1  https://images.chesscomfiles.com/uploads/v1/us...   28692936   \n",
+       "2  https://images.chesscomfiles.com/uploads/v1/us...    3889224   \n",
+       "3  https://images.chesscomfiles.com/uploads/v1/us...   28692936   \n",
+       "\n",
+       "                                              aid  \\\n",
+       "0  https://api.chess.com/pub/player/magnuscarlsen   \n",
+       "1     https://api.chess.com/pub/player/rpragchess   \n",
+       "2  https://api.chess.com/pub/player/magnuscarlsen   \n",
+       "3     https://api.chess.com/pub/player/rpragchess   \n",
+       "\n",
+       "                                          url                       name  \\\n",
+       "0  https://www.chess.com/member/MagnusCarlsen             Magnus Carlsen   \n",
+       "1     https://www.chess.com/member/rpragchess  Praggnanandhaa Rameshbabu   \n",
+       "2  https://www.chess.com/member/MagnusCarlsen             Magnus Carlsen   \n",
+       "3     https://www.chess.com/member/rpragchess  Praggnanandhaa Rameshbabu   \n",
+       "\n",
+       "        username title  followers                               country  \\\n",
+       "0  magnuscarlsen    GM     189570  https://api.chess.com/pub/country/NO   \n",
+       "1     rpragchess    GM       6906  https://api.chess.com/pub/country/IN   \n",
+       "2  magnuscarlsen    GM     189571  https://api.chess.com/pub/country/NO   \n",
+       "3     rpragchess    GM       6906  https://api.chess.com/pub/country/IN   \n",
+       "\n",
+       "  location  last_online      joined   status  is_streamer  verified    league  \\\n",
+       "0   Norway   1712007974  1282856720  premium        False     False  Champion   \n",
+       "1  CHENNAI   1711993442  1466301035  premium        False     False   Crystal   \n",
+       "2   Norway   1712007974  1282856720  premium        False     False  Champion   \n",
+       "3  CHENNAI   1711993442  1466301035  premium        False     False   Crystal   \n",
+       "\n",
+       "         _dlt_load_id         _dlt_id  \n",
+       "0  1712028961.2599978  lcRtKfLWXvYMIA  \n",
+       "1  1712028961.2599978  +erswyz4f2/xMQ  \n",
+       "2   1712029347.285123  opkc5HKMEFbeuw  \n",
+       "3   1712029347.285123  XvJq9hHcwg80rg  "
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "table = conn1.sql(\"SELECT * FROM player\").df()\n",
+    "\n",
+    "\n",
+    "table.head()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(4, 18)"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "table.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv_pipeData",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.10.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

requirements.txt

@@ -0,0 +1,37 @@
+astunparse==1.6.3
+certifi==2024.2.2
+charset-normalizer==3.3.2
+click==8.1.7
+dlt==0.4.7
+duckdb==0.9.2
+fsspec==2024.3.1
+gitdb==4.0.11
+GitPython==3.1.43
+giturlparse==0.12.0
+hexbytes==1.2.0
+humanize==4.9.0
+idna==3.6
+install==1.3.5
+jsonpath-ng==1.6.1
+makefun==1.15.2
+orjson==3.9.10
+packaging==24.0
+pathvalidate==3.2.0
+pendulum==3.0.0
+ply==3.11
+python-dateutil==2.9.0.post0
+pytz==2024.1
+PyYAML==6.0.1
+requests==2.31.0
+requirements-parser==0.7.0
+semver==3.0.2
+simplejson==3.19.2
+six==1.16.0
+smmap==5.0.1
+tenacity==8.2.3
+time-machine==2.14.1
+tomlkit==0.12.4
+types-setuptools==69.2.0.20240317
+typing_extensions==4.10.0
+tzdata==2024.1
+urllib3==2.2.1